Today I gave a tutorial at PyCon
2014 entitled Search 101: An Introduction
to Information Retrieval.
It was an experiment of sorts: the first workshop I've run primarily by
myself, my first tutorial at PyCon, my first paid teaching gig. It was
an opportunity to take some of the lessons I learned from teaching the
Boston Python Workshop and apply
them to a new situation.
The material itself is a distillation of many hours of frustration with
the documentation for various open source search engine libraries,
frustration that they didn't tell me where to start or about the big
picture, they just jumped straight into the details.
Here's what worked:
pyzmq, which requires compilation. I
don't know what the current landscape is for Python distribution, but
installing these libraries through pip is still a pain. I've heard
rumour that more ubiquitous wheels may
solve this in the future.virtualenv everywhere. Even seeing
that, I still think it's worth the confusion to put it forth as the
recommended setup method.
And thanks as well to my helpers: Leo, the tutorial host, Eben,
my TA, and Roberto, on AV. It's impossible to pay adequate attention to
20+ people as a single person, couldn't have done a decent job without
y'all. 
First of all, good news, MirBSD is not vulnerable to The Heartbleed Bug due to my
deliberate choice to stick to an older OpenSSL version. My inquiry
(in various places) as to what precisely could leak when
a vulnerable client connected to a n n-vulnerable server has yet
to be answered, though we can assume private key material is safe.
Now the bad news: while the CA
I use and a CA I don t use
offer free rekeying (in general), a
CA I also use occasionally refuses to do that. The ugly: they
will not even revoke the certificates, so any attacker who gained
your key, for example when you have been using a certificate of
theirs on a Debian system, will be able to use it (e.g. to MITM your
visitors traffic) unless you shell over lots of unreasonable money
per certificate. (Someone wrote they got the fee waived, but others don t, nor do
I. (There s also a great Twitter discussion-thingy about this
involving Zugschlus, but I won t link Twitter because they are not
accessible to Lynx users
like me and other Planet Debian authors.)
While trying to debug a bandwidth problem on a 3G connection, I tried speedtest.net, which ranks fairly high when one searches for bandwidth test on various search engines. I was getting very strange results, so I started wondering if my ISP might be bandwidth-throttling all traffic except the one from speedtest.net tests. After all, that s on a 3G network, and another french 3G ISP (SFR) apparently uses Citrix ByteMobile to optimize the QoE by minifying HTML pages and recompressing images on-the-fly (amongst other things).
So, I fired wireshark, and discovered that no, it s just speedtest being a bit naive. Speedtest uses its own text-based protocol on port 8080. Here is an excerpt of a download speed test:
> HI
< HELLO 2.1 2013-08-14.01
> DOWNLOAD 1000000
< DOWNLOAD JABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFG
I create my local passphrase using pwget 50 or similar, but any sensible way to create a fairly random password should do it. Armed with these details, it is now time to run mkfs, entering the API details and password to create it:[s3c] storage-url: s3c://s.greenqloud.com:443/bucket-name backend-login: API-login backend-password: API-password fs-passphrase: local-password
The next step is mounting the file system to make the storage available.# mkdir -m 700 /var/lib/s3ql-cache # mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ --ssl s3c://s.greenqloud.com:443/bucket-name Enter backend login: Enter backend password: Before using S3QL, make sure to read the user's guide, especially the 'Important Rules to Avoid Loosing Data' section. Enter encryption password: Confirm encryption password: Generating random encryption key... Creating metadata tables... Dumping metadata... ..objects.. ..blocks.. ..inodes.. ..inode_blocks.. ..symlink_targets.. ..names.. ..contents.. ..ext_attributes.. Compressing and uploading metadata... Wrote 0.00 MB of compressed metadata. #
The file system is now ready for use. I use rsync to store my backups in it, and as the metadata used by rsync is downloaded at mount time, no network traffic (and storage cost) is triggered by running rsync. To unmount, one should not use the normal umount command, as this will not flush the cache to the cloud storage, but instead running the umount.s3ql command like this:# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql Using 4 upload threads. Downloading and decompressing metadata... Reading metadata... ..objects.. ..blocks.. ..inodes.. ..inode_blocks.. ..symlink_targets.. ..names.. ..contents.. ..ext_attributes.. Mounting filesystem... # df -h /s3ql Filesystem Size Used Avail Use% Mounted on s3c://s.greenqloud.com:443/bucket-name 1.0T 0 1.0T 0% /s3ql #
There is a fsck command available to check the file system and correct any problems detected. This can be used if the local server crashes while the file system is mounted, to reset the "already mounted" flag. This is what it look like when processing a working file system:# umount.s3ql /s3ql #
Thanks to the cache, working on files that fit in the cache is very quick, about the same speed as local file access. Uploading large amount of data is to me limited by the bandwidth out of and into my house. Uploading 685 MiB with a 100 MiB cache gave me 305 kiB/s, which is very close to my upload speed, and downloading the same Debian installation ISO gave me 610 kiB/s, close to my download speed. Both were measured using dd. So for me, the bottleneck is my network, not the file system code. I do not know what a good cache size would be, but suspect that the cache should e larger than your working set. I mentioned that only one machine can mount the file system at the time. If another machine try, it is told that the file system is busy:# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name Using cached metadata. File system seems clean, checking anyway. Checking DB integrity... Creating temporary extra indices... Checking lost+found... Checking cached objects... Checking names (refcounts)... Checking contents (names)... Checking contents (inodes)... Checking contents (parent inodes)... Checking objects (reference counts)... Checking objects (backend)... ..processed 5000 objects so far.. ..processed 10000 objects so far.. ..processed 15000 objects so far.. Checking objects (sizes)... Checking blocks (referenced objects)... Checking blocks (refcounts)... Checking inode-block mapping (blocks)... Checking inode-block mapping (inodes)... Checking inodes (refcounts)... Checking inodes (sizes)... Checking extended attributes (names)... Checking extended attributes (inodes)... Checking symlinks (inodes)... Checking directory reachability... Checking unix conventions... Checking referential integrity... Dropping temporary indices... Backing up old metadata... Dumping metadata... ..objects.. ..blocks.. ..inodes.. ..inode_blocks.. ..symlink_targets.. ..names.. ..contents.. ..ext_attributes.. Compressing and uploading metadata... Wrote 0.89 MB of compressed metadata. #
The file content is uploaded when the cache is full, while the metadata is uploaded once every 24 hour by default. To ensure the file system content is flushed to the cloud, one can either umount the file system, or ask S3QL to flush the cache and metadata using s3qlctrl:# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql Using 8 upload threads. Backend reports that fs is still mounted elsewhere, aborting. #
If you are curious about how much space your data uses in the cloud, and how much compression and deduplication cut down on the storage usage, you can use s3qlstat on the mounted file system to get a report:# s3qlctrl upload-meta /s3ql # s3qlctrl flushcache /s3ql #
I mentioned earlier that there are several possible suppliers of storage. I did not try to locate them all, but am aware of at least Greenqloud, Google Drive, Amazon S3 web serivces, Rackspace and Crowncloud. The latter even accept payment in Bitcoin. Pick one that suit your need. Some of them provide several GiB of free storage, but the prize models are quite different and you will have to figure out what suits you best. While researching this blog post, I had a look at research papers and posters discussing the S3QL file system. There are several, which told me that the file system is getting a critical check by the science community and increased my confidence in using it. One nice poster is titled "An Innovative Parallel Cloud Storage System using OpenStack s SwiftObject Store and Transformative Parallel I/O Approach" by Hsing-Bung Chen, Benjamin McClelland, David Sherrill, Alfred Torrez, Parks Fields and Pamela Smith. Please have a look. Given my problems with different file systems earlier, I decided to check out the mounted S3QL file system to see if it would be usable as a home directory (in other word, that it provided POSIX semantics when it come to locking and umask handling etc). Running my test code to check file system semantics, I was happy to discover that no error was found. So the file system can be used for home directories, if one chooses to do so. If you do not want a locally file system, and want something that work without the Linux fuse file system, I would like to mention the Tarsnap service, which also provide locally encrypted backup using a command line client. It have a nicer access control system, where one can split out read and write access, allowing some systems to write to the backup and others to only read from it. As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.# s3qlstat /s3ql Directory entries: 9141 Inodes: 9143 Data blocks: 8851 Total data size: 22049.38 MB After de-duplication: 21955.46 MB (99.57% of total) After compression: 21877.28 MB (99.22% of total, 99.64% of de-duplicated) Database size: 2.39 MB (uncompressed) (some values do not take into account not-yet-uploaded dirty blocks in cache) #
Zoe slept in even later this morning. I'm liking this colder weather. We had
nothing particular happening first thing today, so we just snuggled in bed for
a bit before we got started.
Tumble Tastics were offering free
trial classes this week, so I signed Zoe up for one today. She really enjoyed
going to Gold Star Gymnastics in the
US, and has asked me about finding a gym class over here every now and then.
Tumble Tastics is a much smaller affair than Gold Star, but at 300 metres from
home on foot, it's awesomely convenient. Zoe scootered there this morning.
It seems to be physically part of what I'm guessing used to be the Church of
Christ's church hall, so it's not big at all, but the room that Zoe had her
class in still had plenty of equipment in it. There were 8 kids in her class,
all about her size. I peeked around the door and watched.
Most of the class was instructor led and mainly mat work, but then part way
through, the parents were invited in, and the teacher walked us all through a
course around the room, using the various equipment, and the parents had to
spot for their kids.
The one thing that cracked me up was when the kids were supposed to be tucking
into a ball and rocking on their backs. Zoe instead did a Brazilian Jiu-Jitsu
break-fall and fell backwards slapping the mat instead. It was good to see that
some of what she learned in those classes has kicked in reflexively.
She really enjoyed the rope swing and hanging upside down on the uneven bars.
The class ran for 50 minutes (I was only expecting it to last 30 minutes) and
Zoe did really well straight off. I think we'll make this her 4th term
extra-curricular activity.
We scootered home the longer way, because we were in no particular hurry. Zoe
did some painting when we got home, and then we had lunch.
After lunch we goofed off for a little bit, and then we did quiet time. Zoe
napped for about two and a half hours, and then we did some plaster play.
I'd picked up a fish ice cube tray from IKEA on the weekend for 99 cents (queue
Thrift Shop), and I
bought a bag of plaster of Paris a while back, but haven't had a chance to do
anything with it yet. I bribed Zoe into doing quiet time by telling her we'd do
something new with the ice cube tray I'd bought.
We mixed up a few paper cups with plaster of Paris in them and then I squirted
some paint in. I'm not sure if the paint caused a reaction, or the plaster was
already starting to set by the time the paint got mixed in, but it became quite
viscous as soon as the paint was mixed in. We did three different colours and
used tongue depressers to jam it into the tray. Zoe seemed to twig that it was
the same stuff as the impressions of her baby feet, which I thought was a
pretty clever connection to make.
After that, there was barely enough time to watch a tiny bit of TV before Sarah
arrived to pick Zoe up. I told her that her plaster would be set by the time
she got dropped off in the morning.
I procrastinated past the point of no return and didn't go for a run. Instead I
decided to go out to Officeworks and print out some photos to stick in the
photo frame I bought from IKEA on the weekend.
A new release of the
BH package
is now on CRAN and its mirrors. BH
provides (a sizeable subset of) the Boost
library for C++, particularly (large) parts delivered as pure template
headers not requiring linking. See the
BH page for more details.
This release provides our first update relative to the Boost tarballs we
started with. It moves us from 1.51.0 (which was getting a little long in
the tooth) to 1.54.0. This is just about the first time ever that I didn't
package something straight from the current release (now 1.55.0). My aim was to
to balance the oh, shiny, new aspect with some stability. Comments
welcome--maybe I will go to the bleeding edge next time.
As before, the CRAN is created by running bcp over a number of
selected components of Boost. If you'd like to see additional ones include,
please do get in touch too. Before uploading, I also tested against all of
these sixteen CRAN dependents I could quickly test on my server given the
installed dependencies there.
The complete list changes follows below.
Changes in version 1.54.0-1 (2014-04-07)Courtesy of CRANberries, there is also a diffstat report for the most recent release. Comments and suggestions are welcome via the mailing list or issue tracker at the GitHub repo.
- Upgraded to Boost 1.54.0
- Adjust build script
local/script/CreateBoost.shaccordingly- Renamed
generation_runge_kutta_cash_karp54_classic.hpptogeneration_runge_kutta_cash_karp54_cl.hppto remain within 100-character limit fortar
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Using address extension
Postfix (and many other mail servers) offers one nice address
extension feature: addresses like
<user+whaterver@> are implicit aliases to
<user@>. This allows users to implement a simple measure to
fight spam:
<user+somecompany@>;
<user+evilcorp@> is
being spammed. In /etc/postfix/main.cf:
smtpd_recipient_restrictions =
check_recipient_access hash:/etc/postfix/recipients,
[ ]
Then, create /etc/postfix/recipients containing the
addresses to disable:
user+evilcorp@example.com 553 5.7.1 I did not subscribe to receive spam, go awayOf course, the error codes and message can be freely configured, just make sure you are using a permanent error code so senders do not retry. Hash that table, reload Postfix and it is done:
# postmap /etc/postfix/recipients # service postfix reloadAfter that, your mail server will reject messages sent to these addresses. And it will do so at the
RCPT TO step, saving
your bandwidth for more useful things.
| Publisher: | Anchor |
| Copyright: | November 2013 |
| ISBN: | 0-307-74381-0 |
| Format: | Trade paperback |
| Pages: | 244 |
Short version:
grep -l 'libssl.*deleted' /proc/*/maps tr -cd 0-9\\n xargs -r ps u Some people seem to indicate that the 64kB leak can enable an attacker to get pretty much anything from the process memory space, including the certificate private key. While we weren't able to confirm that yet, that's not really impossible, so you might also want to regenerate those private keys, although that's not something you should do in a rush either.
Some times it is desired to load arbitrary python files or pre-
installed python modules during application run time.I had encountered
2 such usecases, one is in SILPA application
and other is dictionary-bot which I was
refactoring recently.
from __future__ import print_function
import sys
import importlib
def load_module(modulename):
mod = None
try:
mod = importlib.import_module(modulename)
except ImportError:
print("Failed to load module ".format(module=modulename),
file=sys.stderr)
return mod
pluginloader.py
plugins
__ aplugin.py
__ bplugin.py
pluginloader.py is the file which needs to load python files under
plugins directory. This was again done using importlib module as
shown below.
import os
import sys
import re
import importlib
def load_plugins():
pysearchre = re.compile('.py$', re.IGNORECASE)
pluginfiles = filter(pysearchre.search,
os.listdir(os.path.join(os.path.dirname(__file__),
'plugins')))
form_module = lambda fp: '.' + os.path.splitext(fp)[0]
plugins = map(form_module, pluginfiles)
# import parent module / namespace
importlib.import_module('plugins')
modules = []
for plugin in plugins:
if not plugin.startswith('__'):
modules.append(importlib.import_module(plugin, package="plugins"))
return modules
As announced previously, I ve been hacking on a migration tool allowing to import into the MediaWiki of a FusionForge project, a conversion of the contents of a TWiki wiki.
I ve succesfully imported a first project (from PicoForge to FusionForge) using the tool, so I ve decided to tag a first release and make the Git repo accessible.
More details at : https://fusionforge.int-evry.fr/projects/pytwiki2mediawi/
Feel free to ask here in the comments or by email, in case of need.
And, yes, my Python is most likely awful, but at least, this works, and much more featureful than existing tools I could test.
Today was a really good day, right up until the end, when it wasn't so good,
but could have been a whole lot worse, so I'm grateful for that.
I've been wanting to walk
out to King Island at low tide with Zoe for a while, but it's taken about a
month to get the right combination of availability, weather and low tide timing
to make it possible.
Today, there was a low tide at about 10:27am, which I thought would work out
pretty well. I wasn't sure if the tide needed to be dead low to get to King
Island, so I thought we could get there a bit early and possibly follow the
tide out. I invited Megan and Jason to join us for the day and make a picnic of
it.
It turned out that we didn't need a really low tide, the sand bar connecting
King Island to Wellington Point was well and truly accessible well before low
tide was reached, so we headed out as soon as we arrived.
I'd brought Zoe's water shoes, but from looking at it, thought it would be
walkable in bare feet. We got about 10 metres out on the sand and Zoe started
freaking out about crabs. I think that incident with the mud
crab on Coochiemudlo Island has left her slightly phobic of crabs.
So I went back to Jason's car and got her water shoes. I tried to allay her
fears a bit by sticking my finger in some of the small holes in the sand, and
even got her to do it too.
I'm actually glad that I did get her water shoes, because the shell grit got a
bit sharp and spiky towards King Island, so I probably would have needed to
carry her more than I did otherwise.
Along the way to the island we spotted a tiny baby mud crab, and Zoe was brave
enough to hold it briefly, so that was good.
We walked all the way out and partially around the island and then across it
before heading back. The walk back was much slower because where was a massive
headwind. Zoe ran out of steam about half way back. She didn't like the sand
getting whipped up and stinging her legs, and the wind was forcing the brim of
her hat down, so I gave her a ride on my shoulders for the rest of the way
back.
We had some lunch after we got back to Wellington Point, and Zoe found her
second wind chasing seagulls around the picnic area.
After an ice cream, we went over to the playground and the girls had a great
time playing. It was a pretty good park. There was this huge tree with a really
big, thick, horizontal branch only about a metre or two off the ground. All the
kids were climbing on it and then shimmying along the branch to the trunk.
Zoe's had a few climbs in trees and seems not afraid of it, so she got up and
had a go. She did really well and did a combination of scooting along,
straddling the branch and doing a Brazilian Jiu-Jitsu-style "bear crawl" along
the branch.
It was funny seeing different kids' limits. Zoe was totally unfazed by climbing
the tree. Megan was totally freaking out. But when it came to walking in bare
feet in an inch of sea water, Zoe wanted to climb up my leg like a rat up a
rope, in case there were crabs. Each to their own.
Zoe wanted to have a swim in the ocean, so I put her into her swimsuit, but had
left the water shoes back in the car. Once again, she freaked out about crabs
as soon as we got ankle deep in the water, and was freaking out Megan as well,
so the girls elected to go back to playing in the park.
After a good play in the park, we headed back home. We'd carpooled in Jason's
truck, with both girls in the back. I'd half expected Zoe to fall asleep on the
way back, but the girls were very hyped up and had a great time playing games
and generally being silly in the back.
When we got back to our place, Jason was in need of a coffee, so we walked to
the Hawthorne Garage and had coffee and babyccinos, before Megan and Jason went
home.
It was about 3:30pm at this point, and I wanted to make a start on dinner. I
was making a wholemeal
pumpkin quiche, which I've made a few times before, and I discovered we
were low on linseed. I thought I'd push things and see if Zoe was up for a
scooter ride to the health food shop to get some more and kill some time.
She was up for it, but ran out of steam part way across Hawthorne Park.
Fortunately she was okay with walking and didn't want me to carry her and the
scooter. It took us about an hour to get to the health food shop.
Zoe immediately remembered the place from the previous week where we'd had to
stop for a
post-meltdown pit stop and declared she needed to go to the toilet again.
We finally made it out of the shop. I wasn't looking forward to the long walk
back home, but there were a few people waiting for a bus at the bus stop near
the health food shop, and on checking the timetable, the bus was due in a
couple of minutes, so we just waited for the bus. That drastically shortened
the trip back.
Zoe managed to drop the container of linseed on the way home from the bus stop,
but miraculously the way it landed didn't result in the loss of too much of the
contents, it just split the container. So I carefully carried the container
home the rest of the way.
By this stage it was quite a bit later than I had really wanted to be starting
dinner, but we got it made, and Zoe really liked the pumpkin quiche, and ate a
pretty good dinner.
It was after dinner when things took a turn for the worse.
Zoe was eating an ice block for dessert, and for whatever reason, she'd decided
to sit in the corner of the kitchen next to the dishwasher, while I was loading
it. I was carrying over one of the plates, and the knife managed to fall off
the plate, bounce off the open dishwasher door and hit her in the mouth,
splitting her lip.
Zoe was understandably upset, and I was appalled that the whole thing had
happened. She never sits on the kitchen floor, let alone in the corner where
the dishwasher is. And this knife came so close to her eye.
Fortunately the lip didn't look too bad. It stopped bleeding quickly, and we
kept some ice on it and the swelling went down.
I hate it when accidents happen on my watch. I feel like I'm fighting the
stigma of the incompetent single Dad, or the abusive single Dad, so when Zoe
sustains an injury to the face like a fat lip, which could be misinterpreted,
I, well, really hate it. This was such a freak accident, and it could have gone
so much worse. I'm just so glad she's okay.
Zoe recovered pretty well from it, and I was able to brush her teeth without
aggravating her lip. She went to bed well, and I suspect she's going to sleep
really well. It's a bit cooler tonight, so I'm half-expecting a sleep in in the
morning with any luck.
| Editor: | Gordon van Gelder |
| Issue: | Volume 121, No. 3 & 4 |
| ISSN: | 1095-8258 |
| Pages: | 258 |
When I started to write the landing page for
The Hacker's Guide to Python, I wanted
to try new things at the same time. I read about A/B testing a while ago,
and I figured it was a good opportunity to test it out.
A/B testing
If you do not know what A/B testing is about, take a quick look at the
Wikipedia page on that subject.
Long story short, the idea is to serve two different version of a page to
your visitors and check which one is getting the most success. When you
found which version is better, you can definitely switch to it.
In the case of my book, I used that technique on the pre-launch page where
people were able to subscribe to the newsletter. I didn't have a lot of
things I wanted to test out on that page, so I just used that approach on
the subtitle, being either "Learn everything you need to build a successful
Python project" or "It's time you make the most out of Python".
Statistically, each version would be served half of the time, so both would
get the same number of view. I then would build statistics about which page
was attracting the most subscribers. With the results I would be able to
switch definitively to that version of the landing page.
Technical design
My Web site, this Web site, is entirely static and served by
Apache httpd. I didn't want to use any dynamic
page, language or whatever. Mainly because I didn't want to have something
else to install and maintain just for that on my server.
It turns out that Apache httpd is powerful enough to implement such a
feature. There are different ways to build it, and I'm going to describe my
choices here.
The first thing to pick is a way to balance the display of the page. You
need to find a way so that if you get 100 visitors, around 50 will see the
version A of your page, and around 50 will see the version B of the page.
You could use a random number generator, pick a random number for each
visitor, and decides which page he's going to see. But it turns out that I
didn't find a way to do that with Apache httpd at first sight.
My second thought was to pick the client IP address. But it's not such a
good idea, because if you got visitors from, for example, people behind a
company firewall, they are all going to be served the same page, so that
kind of kills the statistics.
Finally, I picked time based balancing: if you visit the page on a second
that is even, you get version A of the page, and if you visit the page on a
second that is odd, you get version B. Simple, and so far nothing proves
there are more visitors on even than odd seconds, or vice-versa.
The next thing is to always serve the same page to a returning visitor. I
mean that if the visitor comes back later and get a different version,
that's cheating. I decided the system should always serve the same page once
a visitor "picked" a version. To do that, it's simple enough, you just have
to use cookies to store the page the visitor has been attributed, and then
use that cookie if he comes back.
Implementation
To do that in Apache httpd, I used the powerful
mod_rewrite
that is shipped with it. I put 2 files in the books directory, named either
"the-hacker-guide-to-python-a.html" and "the-hacker-guide-to-python-b.html"
that got served when you requested "/books/the-hacker-guide-to-python".
RewriteEngine On
RewriteBase /books
# If there's a cookie called thgtp-pre-version set,
# use its value and serve the page
RewriteCond % HTTP_COOKIE thgtp-pre-version=([^;])
RewriteRule ^the-hacker-guide-to-python$ % REQUEST_FILENAME -%1.html [L]
# No cookie yet and
RewriteCond % HTTP_COOKIE !thgtp-pre-version=([^;]+)
# the number of seconds of the time right now is even
RewriteCond % TIME_SEC [02468]$
# Then serve the page A and store "a" in a cookie
RewriteRule ^the-hacker-guide-to-python$ % REQUEST_FILENAME -a.html [cookie=thgtp-pre-version:a:julien.danjou.info,L]
# No cookie yet and
RewriteCond % HTTP_COOKIE !thgtp-pre-version=([^;]+)
# the number of seconds of the time right now is odd
RewriteCond % TIME_SEC [13579]$
# Then serve the page B and store "b" in a cookie
RewriteRule ^the-hacker-guide-to-python$ % REQUEST_FILENAME -b.html [cookie=thgtp-pre-version:b:julien.danjou.info,L]
The other week I was toying with using GNU stow to build an NFS-share, which would allow remote machines to boot from it.
It worked. It worked well. (Standard stuff, PXE booting with an NFS-root.)
Then I started wondering about distributions, since in one sense what I'd built was a minimal distribution.
On that basis yesterday I started hacking something more minimal:
$ cat > t.c <>EOF
int main ( int argc, char *argv[] )
printf("OK\n" );
return 1;
EOF
$ /opt/tcc/bin/tcc t.c
tcc: error: file 'crt1.o' not found
tcc: error: file 'crti.o' not found
..
Attempting to fix this up resulted in nothing much better:
$ /opt/tcc/bin/tcc t.c -I/opt/musl/include -L/opt/musl/lib/And because I don't have a full system I cannot compile t.c to t.o and use ld to link (because I have no ld.) I had a brief flirt with the portable c-compiler, pcc, but didn't get any further with that. I suspect the real solution here is to install gcc onto my host system, with something like --prefix=/opt/gcc, and then rsync that into my (suddenly huge) intramfs image. Then I have all the toys.
archive.d.o into
sources.d.n. In both cases the respective maintainers
(Guillem and Ganneff, kudos) have been positive about and helpful
in improving the situation, despite the low impact of the bugs I've
found on the average user. That's quite important for the
long-term preservation of digital information in
general, and for the perennity of access to Free Software in the
specific case of Debian.
While we are it, I'm now maintaining a list of
bugs affecting sources.d.n but belonging to other
packages, in case you fancy helping out but are not a Python
hacker. Interestingly enough, quite a bit of those bugs are related
to the fact that tools debsources uses (e.g. ctags, sloccount) are
also starting to show their age.
You might wander why buzz, rex, and bo are still missing from
sources.d.n. That's in fact for similar reasons.
Before hamm Debian didn't have complete archive coverage in terms
of Sources indexes and .dsc files. Given
that debsources rely on both to extract source packages, it first
needs to grow an additional abstraction layer that can cope with
their absence. It's SMOP, and planned.
And now let's have fun with ctags
bombs.
Yours truly,#debian-ftp)
Java Sound on Linux
I m often in the position of having my favorite Java program (AltosUI)
unable to make any sounds. Here s a history of the various adventures
I ve had.
Java and PulseAudio ALSA support
When we started playing with Java a few years ago, we discovered that
if PulseAudio were enabled, Java wouldn t make any sound. Presumably,
that was because the ALSA emulation layer offered by PulseAudio wasn t
capable of supporting Java.
The fix for that was to make sure pulseaudio would never run. That s
harder than it seems; pulseaudio is like the living dead; rising from
the grave every time you kill it. As it s nearly impossible to install
any desktop applications without gaining a bogus dependency on
pulseaudio, the solution that works best is to make sure dpkg never
manages to actually install the program with dpkg-divert:
# dpkg-divert --rename /usr/bin/pulseaudio
$ cat /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/sound.properties
...
#javax.sound.sampled.Clip=org.classpath.icedtea.pulseaudio.PulseAudioMixerProvider
#javax.sound.sampled.Port=org.classpath.icedtea.pulseaudio.PulseAudioMixerProvider
#javax.sound.sampled.SourceDataLine=org.classpath.icedtea.pulseaudio.PulseAudioMixerProvider
#javax.sound.sampled.TargetDataLine=org.classpath.icedtea.pulseaudio.PulseAudioMixerProvider
javax.sound.sampled.Clip=com.sun.media.sound.DirectAudioDeviceProvider
javax.sound.sampled.Port=com.sun.media.sound.PortMixerProvider
javax.sound.sampled.SourceDataLine=com.sun.media.sound.DirectAudioDeviceProvider
javax.sound.sampled.TargetDataLine=com.sun.media.sound.DirectAudioDeviceProvider
$ cat /etc/asound.conf
pcm.!default
type plug
slave.pcm "dmixer"
pcm.dmixer
type dmix
ipc_key 1024
slave
pcm "hw:1,0"
period_time 0
period_size 1024
buffer_size 4096
rate 44100
bindings
0 0
1 1
ctl.dmixer
type hw
card 1
ctl.!default
type hw
card 1
Next.