In today s Linux distributions, there are usually two to three levels of package management. In this blog post, I will explain the three levels of package management. 1. Local (dpkg, rpm) The first level of package management is the local level. This level consists of package management tools that install and/or remove packages via package archives such as .deb or .rpm files and a database of the local s system state. 2. Connected (APT, YUM, Smart) The second level of package management is the connected level. In this level, tools are fetching packages and information about them from (remote) locations known as repositories or sources . A level 2 package manager is usually built on top of a level 1 package manager, for example, APT uses dpkg but there are also cases where one tool is both, such as opkg. 3. Abstract (PackageKit, Smart) A level 3 package manager is a tool that supports different level 1 package managers via one generic interface. It may be implemented on top of a level 2 package manager (PackageKit), or may be implemented directly in level 2 (Smart). Conclusion: Merge Level 2 and 3 Most level 2 package managers share a great deal of tasks, such as solving dependency problems. Merging level 2 package management and level 3 package management would enable us to reduce the number of dependency problem solvers and combine our efforts into a few package managers. Such an implementation would need to be written in C and support all common level 1 package managers in order to be successful. As some might have guessed, this is what I plan to do with the project codenamed APT2, although work is not progressing very fast. PS: Back from vacation I spent the more than the complete last week in (mostly rainy) Corfu, Greece and am now getting back to my usual development stuff. I have already processed my news & blogs backlog (1000+ messages) and will now start with my email backlog, so don t be angry if the answer to your email takes some time.
Filed under: APT2, General

We all know the good old Emacs mode-line you got under every window. Since the beginning (a long time ago), it starts and ends with dashes. I've proposed a patch to remove them. Before: After: This has been merged in Emacs 24. You won't see any more ugly dashes in graphical mode.

I like to think that in the Ubuntu project, we re pragmatic about technology. This means keeping an open mind, considering alternatives, and evaluating them objectively. It means bearing in mind the needs of the user, and measuring ourselves based on how well we solve their problems (not merely our own). It is in this spirit that I have been thinking about Qt recently. We want to make it fast, easy and painless to develop applications for Ubuntu, and Qt is an option worth exploring for application developers. In thinking about this, I ve realized that there is quite a bit of commonality between the strengths of Qt and some of the new directions in Ubuntu:

• Qt has a long history of use on ARM as well as x86, by virtue of being popular on embedded devices. Consumer products have been built using Qt on ARM for over 10 years. We ve been making Ubuntu products available for ARM for nearly two years now, and 10.10 supports more ARM boards than ever, including reference boards from Freescale, Marvell and TI. Qt is adding ARMv7 optimizations to benefit the latest ARM chips. We do this in order to offer OEMs a choice of hardware, without sacrificing software choice. Qt preserves this same choice for application developers.
• Qt is a cross-platform application framework, with official ports for Windows, MacOS and more, and experimental community ports to Android, the iPhone and WebOS. Strong cross-platform support was one of the original principles of Qt, and it shows in the maturity of the official ports. With Ubuntu Light being installed on computers with Windows, and Ubuntu One landing on Android and the iPhone, we need interoperability with other platforms. There is also a large population of developers who already know how to target Windows, who can reach Ubuntu users as well by choosing Qt.
• Qt has a fairly mature touch input system, which now has support for multi-touch and gestures (including QML), though it s only complete on Windows 7 and Mac OS X 10.6. Meanwhile, Canonical has been working with the community to develop a low-level multi-touch framework for Linux and X11, for the benefit of Qt and other toolkits. These efforts will eventually meet in the middle.

Overall, I think Qt has a lot to offer people who want to develop applications for (and on) Ubuntu, particularly now. It already powers popular cross-platform applications like VLC, not to mention the entire Kubuntu distribution. I missed it when this happened last year, but Qt is now available under either the LGPL 2.1 or the GPL 3.0, which should make it suitable for virtually any Ubuntu application. It has strong commercial backing as well as a large developer community. No single solution will meet all developers needs, of course, and Ubuntu supports multiple toolkits and frameworks for this reason, but Qt seems like a great tool to have in our toolbox for the road ahead.

I've just fnished another step for new Gammu manual - it includes also documentation for gammu utility. IIt includes all content from the man page, reformatted and improved in some sections. So now the documentation does cover all important parts and can contain correct cross references. The only problematic thing was to handle non dash starting commands in Sphinx. Unfortunately it does not seem to like this idea, so I had to invent some ugly hacks, which workaround that (just changing the regexps which sphinx.domains.std uses). The man page will be from now on generated from this source as well.

Filed under: Gammu 0 comments

Since a few versions now, dpkg has been made safer in that it makes sure everything is written to disk after unpacking a package. It does so with sync(), which means all kernel buffer caches are being flushed. Between every package install. All kernel buffer caches. Imagine what it does when you have two 2TB hard drives (though they are far from being full). Update: The use of btrfs might not be helping either. I have 2 concurrent pbuilder instances that have been installing iceweasel build dependencies for more than 40 minutes, and still not finished yet. I don t expect the builds themselves, including test suites, to take more than 20 minutes on that machine I should really start using eatmydata Update: with eatmydata, the package installation is almost instantaneous Update 2: It might just be btrfs. And the best part yet is that since sync() is global, even when running pbuilder over, say, tmpfs, it doesn t change a damn thing.

If you re running a 64bit system, and you ve got users with access to a video device (/dev/video*), then be sure you update your kernels for CVE-2010-2963. I ve been slowly making my way through auditing the many uses in the Linux kernel of the copy_from_user() function, and ran into this vulnerability. Here s the kernel code from drivers/media/video/v4l2-compat-ioctl32.c:

static int get_microcode32(struct video_code *kp, struct video_code32 __user *up)
 
        if (!access_ok(VERIFY_READ, up, sizeof(struct video_code32))  
                copy_from_user(kp->loadwhat, up->loadwhat, sizeof(up->loadwhat))  
                get_user(kp->datasize, &up->datasize)  
                copy_from_user(kp->data, up->data, up->datasize))
                        return -EFAULT;
        return 0;
 

Note that kp->data is being used as the target for up->data in the final copy_from_user() without actually verifying that kp->data is pointing anywhere safe. Here s the caller of get_microcode32:

static long do_video_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 
        union  
                struct video_tuner vt;
                struct video_code vc;
...
          karg;
        void __user *up = compat_ptr(arg);
...
        switch (cmd)  
...
        case VIDIOCSMICROCODE:
                err = get_microcode32(&karg.vc, up);
...

So, the contents of up are totally under control of the caller, and the contents of karg (in our case, the video_code structure) are not initialized at all. So, it seems like a call for VIDIOCSMICROCODE would write video_code->datasize bytes from video_code->data into some random kernel address, just causing an Oops, since we don t control what is on the kernel s stack. But wait, who says we can t control the contents of the kernel s stack? In fact, this compat function makes it extremely easy. Let s look back at the union. Notice the struct video_tuner? That gets populated from the caller s up memory via this case of the switch (cmd) statement:

...
        case VIDIOCSTUNER:
        case VIDIOCGTUNER:
                err = get_video_tuner32(&karg.vt, up);
...

So, to control the kernel stack, we just need to call this ioctl twice in a row: once to populate the stack via VIDIOCSTUNER with the contents we want (including the future address for video_code->data, which starts at the same location as video_tuner->name[20]), and then again with VIDIOCSMICROCODE. Tricks involved here are: the definition of the VIDIOCSMICROCODE case in the kernel is wrong, and calling the ioctls without any preparation can trigger other kernel work (memory faults, etc) that may destroy the stack contents. First, we need the real value for the desired case statement. This turns out to be 0 4020761b. Next, we just repeatedly call the setup ioctl in an attempt to get incidental kernel work out of the way so that our last ioctl doing the stack preparation will stick, and then we call the buggy ioctl to trigger the vulnerability. Since the ioctl already does a multi-byte copy, we can now copy arbitrary lengths of bytes into kernel memory. One method of turning an arbitrary kernel memory write into a privilege escalation is to overwrite a kernel function pointer, and trigger that function. Based on the exploit for CVE-2010-3081, I opted to overwrite the security_ops function pointer table. Their use of msg_queue_msgctl wasn t very good for the general case since it s near the end of the table and its offset would depend on kernel versions. Initially I opted for getcap, but in the end used ptrace_traceme, both of which are very near the top the security_ops structure. (Though I need share credit here with Dan Rosenberg as we were working together on improving the reliability of the security_ops overwrite method. He used the same approach for his excellent RDS exploit.) Here are the steps for one way of taking an arbitrary kernel memory write and turning it into a root escalation:

• overwrite security_ops with default_security_ops, which will revert the LSM back to the capabilities-only security operations. This, however, means we can calculate where cap_ptrace_traceme is.
• overwrite default_security_ops->ptrace_traceme to point to our supplied function that will actually perform the privilege escalation (thanks to Brad Spengler for his code from Enlightenment).
• trigger the function (in this case, call ptrace(PTRACE_TRACEME, 0, NULL, NULL)).
• restore default_security_ops->ptrace_traceme to point to cap_ptrace_traceme so the next caller doesn t Oops the system (since userspace memory will be remapped).

Here s the source for Vyakarana as seen running in Enlightenment using cap_getcap (which is pretty unstable, so you might want to switch it to use ptrace_traceme), and as a stand-alone memory writer. Conclusions: Keep auditing the kernel for more arbitrary writes; I think there are still many left. Reduce the exploitation surface within the kernel itself (which PaX and grsecurity have been doing for a while now), specifically:

• Block userspace memory access while in kernel mode. This would stop the ability to make the kernel start executing functions that live in userspace a clear privilege violation. This protection would stop the current exploit above, but the exploit could be adjusted to use kernel memory instead.
• Keep function pointers read-only. There is no reason for these function pointer tables (fops, IDT, security_ops, etc) to be writable. These should all be marked correctly, with inline code exceptions being made for updating the global pointers to those tables, leaving the pointer read-only after it gets set. This would stop this particular exploit above, but there are still plenty more targets.
• Randomize the kernel stack location on a per-syscall basis. This will stop exploits that depend on a stable kernel stack location (as this exploit does).

Buying a ThinkPad? Don't buy one with this wireless chipset. "ThinkPad wireless" is apparently code for "poorly supported RealTek chip". If you do end up with one of these chips (perhaps you failed to notice the addition of a new option on the website before the time you spec'ed out your new laptop and the time you ordered it; perhaps you assumed it was iwlagn by default; who can say?), you may be treated to such issues as a lack of rfkill interface support, a driver that needs to be hard reset about once an hour to continue to be able to see APs, a kernel panic if you try to run powertop... or maybe a mysterious bug that prevents the wireless card from transmitting packets to a VoIP server, with 100% reliability. And after you wear yourself out from laughing at the Lenovo sales rep's suggestion that you could pay a 15% restocking fee and reorder another laptop that costs $40 more to get the Intel Centrino option when that same Centrino card can be found on-line for $40 by itself, you might decide to order an Intel Centrino Ultimate-N 6300 from a third party, the same card that's advertised on Lenovo's website as an option for the X201, and install it yourself. And that's when the fun would start. Did you know that it's common practice for laptop OEMs to hard-code a whitelist of authorized wifi (and 3G) cards in the BIOS, and refuse to boot with an unrecognized card - nominally to ensure no one can combine a wireless adapter with their hardware antennas to create a transmission device that violates FCC regulations? And that these same OEMs may have different PCI subsystem IDs for the cards they ship than are used for the ones you can buy retail from third parties? I did not know that. Apparently some people knew that because they had tried this before, but I missed that memo, having never before needed to combine a laptop and a wifi card from different vendors. Here, gentle reader, is a memo. You're welcome. Hacking the ThinkPad BIOS So although this feature was news to me, it was not news to the Internet at large. A quick search on the BIOS error message quickly turned up a relevant description of the problem, as well as pointers to some options for fixing (in the veterinary sense) the whitelist checking in the ThinkPad BIOS. It's great to be part of such a large community of ThinkPad-using Free Software hackers looking out for each other (and not taking "no" for an answer from their BIOS)! But wait, none of the methods described on that page for modifying your BIOS (or bypassing the check by moving the card to a different slot, or taping over one of the pins, or...) work on the X201! As Matthew summarized it, things are more difficult for "anything with three digits or starting with a six". Fortunately, another hint from Matthew took me to the website of a guy who's been providing fixes for this BIOS issue for the past couple of years. Hmm, but the X201 is a bleeding-edge new model and isn't on his list of models that he's patched the BIOS for. Well, fortunately there's a step-by-step howto here that I can follow here to do it myself... grab the latest BIOS image from the vendor's website; decompress the image with phcomp.exe (oh Wine, you pair well with more than cheese); unpack the individual BIOS modules with phnxsplit; run phnxpatch to apply the binary patches, finding that the only one that applies is "wwan1a" when the issue is not with a WWAN card at all, so hand-hack the whitelist in the binary instead; use the fi.exe and fp.exe tools to add the necessary header back onto the BIOS module (after deciphering the example batch script accompanying the BIOS update and translating it into a reusable script in a real scripting language); use phnxmod to inject the newly-packaged BIOS module into the original image; call phnxcksm to update the BIOS's internal checksum; ... Uh? Phnxcksum fails, saying that there's no extended checksum field. Just how deep is this rabbit hole? How much was that restocking fee again, and how much is my time worth? But no; now it's a matter of principle. Besides, I'd rather spend my time fighting with a BIOS than having to reinstall the OS on another new machine. Which is to say that I'd rather beg the local reverse engineering experts for their time fighting with a BIOS. So now we have a BIOS image whose code we're confident has been correctly patched to bypass the whitelist, but we can't find the checksum field that our patching tools expect. Maybe we can flash this to the BIOS as is? No such luck; sure enough, we get a checksum error. Ah, but WinPhlash gives us a logfile, and shows that the failure is a per-module checksum mismatch. So where is that checksum? Oh, there it is; it's the byte labeled "data_checksum" in the module header description in phnxsplit.h, which phnxmod doesn't copy when it splices an updated module into the BIOS image. Well, could it really be as easy as updating a single byte? Yes, the BIOS utility takes it and flashes it without complaint... and the system boots afterwards! Thanks to Paul and Matthew for their great work documenting this problem in general; Zender, for tools that provide a 99% solution; and Kees and Jesse for a dazzling display of impromptu reverse-engineering. Now, to figure out how to get these tools packaged and into Debian and Ubuntu. Posted from my Centrino-wireless-using Lenovo X201 laptop.

I finally bought myself an eBook reader. To be precise, I got the Sony PRS-600 Reader Touch Edition. Two reasons:

• I bought a lot of technical literature in digital (PDF) format and want to read it without looking at a CRT while on a train.
• The idea of having a lot of reference material with me appeals me.
• Many of my books are read only once (especially Star Trek etc.) and I don t want to waste paper and the energy for shipping.

First of all, the display is only really great with a powerful light source. The reflection is a bit of a nuisance but I can live with it. What I don t like is the PDF support of the device. Viewing a normal page, the font faces are so tiny that it is not readable in any way. The zoom feature works around that, but you can t switch pages from there. Instead you have to unzoom, go to the next page and zoom again. Crappy! There is also the reflow option, which does not mix with source code figures in my books. I am glad that most of my electronic books are from The Pragmatic Bookshelf, which means that I can download them in Epub as well, all without DRM. Thanks, guys! For the PDFs I would like to carry, I googled a while and stumbled across this blog post. The example to work with source documents worked fine for The Little Book of Semaphores at least. But even with BRISS I did not (yet) manage to crop books I bought from APress. Most likely I will try doing it via a bit of python scripting and the poppler bindings for python. I also bought my first DRM content: Star Trek Vanguard: Harbinger. I really enjoyed the book, but the download was a PITA. After buying, you only get a link to a ASCM file, which is not useful by itself. One needs Adobe Digital Editions to download it. So I installed it into a virtual machine which has XP installed. After that, I was able to read it on the PC, which wasn t the goal of the exercise. To put it on the reader, one needs to authorize it from the same Windows software and upload the file. All that stuff took almost an hour and of course the digital edition of the book is more expensive than getting it from amazon germany. But at least I did not have to wait for delivery. Having finished the book I wanted to buy the second part, but of course it is only available for canadian residents on ebooks.com. I researched a number of ebook sites which mostly sell it only in the US. Funny enough, the first hits when searching where pirated versions of the book in full text. Why do they make it so hard to get stuff legally? *sigh*

Recent Emacs versions have a wonderful capacity to hide the mouse pointer as soon as you type and insert characters in a buffer. This is controlled by the make-pointer-invisible' variable, which is set to t by default. However that does not hide the pointer when simply moving the cursor on screen. Therefore, I've started to use mouse-avoidance-mode', which make the mouse pointer jump if your cursor hits it. Unfortunately, if your cursor hits the invisible mouse pointer, mouse-avoidance-mode' makes it jump too, because it does not know it is invisible. Well, it did not know. Now it does, thanks to my patches which have been merged in Emacs 24. Using the new function frame-pointer-invisible-p', one can know if the mouse pointer has been hidden by Emacs. Therefore I enhanced mouse-avoidance-mode' to use it, and everything is alright now. :-)

The Gnash project is the most promising solution for a Free Software Flash implementation. It has done great so far, but there is still far to go, and recently its funding has dried up. I believe AVM2 support in Gnash is vital to the continued progress of the project, as more and more sites show up with AVM2 flash files. To try to get funding for developing such support, I have started a pledge with the following text:

"I will pay 100$ to the Gnash project to develop AVM2 support but only if 10 other people will do the same." - Petter Reinholdtsen, free software developer Deadline to sign up by: 24th December 2010 The Gnash project need to get support for the new Flash file format AVM2 to work with a lot of sites using Flash on the web. Gnash already work with a lot of Flash sites using the old AVM1 format, but more and more sites are using the AVM2 format these days. The project web page is available from http://www.getgnash.org/ . Gnash is a free software implementation of Adobe Flash, allowing those of us that do not accept the terms of the Adobe Flash license to get access to Flash sites. The project need funding to get developers to put aside enough time to develop the AVM2 support, and this pledge is my way to try to get this to happen. The project accept donations via the OpenMediaNow foundation, http://www.openmedianow.org/?q=node/32 .

I hope you will support this effort too. I hope more than 10 people will participate to make this happen. The more money the project gets, the more features it can develop using these funds. :)

Recently I've been producing simple Flash animations. Mostly these are simple "Show an image, slide it around a bit, show another..". But I still feel vaguely unclean and non-free. I started off using the SWF Perl binding, but soon realised that wasn't much fun. So I wrote a mini intepretter such that I can script creation:

#!/usr/bin/flash-scripter
# create the movie
create 640 480
# background == black
clear 0, 0, 0
# load image at 0,0
load 0, 0, foo.png
# move it about a bit
move 0, -1
move 0, -1
move 0, -1
# movie-time is over now.
stop
# finally save the movie
save foo.swf
# all done
exit

I see I'm not alone in doing such a thing as swftools (not available for Debian) includes swfc "A tool for creating SWF files from simple script files.". Sadly swftools fails to build for me on Squeeze so I couldn't try it out. My little tool is called SWF::Scripter and uses plugins to implement each "command" so I should probably upload it somewhere public. On the other hand it was a quick hack to produce a mini-story from a bunch of images with no complex transitions so I'm not sure it is worth the effort. ObQuote: "I honestly think I'd give up smoking if he asked me." - Breakfast at Tiffany's

Another week has passed, where is Squeeze? Sadly, I did even less than the week before, but here is the log anyways:

• #599782 pyopencl: FTBFS: ImportError: No module named pyopencl
  the build process imports the just built module to generate the docs but the build-path guessing in debian/rules is broken, after some testing Jakub has posted the right snippet needed. RT has to decide whether maintainer can upload a slightly updated upstream version with the fix, or the on in the archive (with the fix too ;))
• #598202 update-manager-gnome: will not start
  this one is from the last week neither me nor the submitter can reproduce it any longer, thus closed
• #599523 update-manager-gnome: wants to downgrade packages without any notice
  submitters pinning seems broken, waiting for a reply

That s all. Sorry, I had a busy week.

I travel across the Sedgemoor/North Somerset/Bristol council borders, so I m a big fan of FixMyStreet which puts fault reporting for things like potholes, flytipping and broken lights behind a common interface. Credit to the MySociety charity project, once again. So I m quite happy that something similar is starting for website faults. If you know disabled people who are willing to report web accessibility issues for volunteers to pick up, please let them know they can currently:

• email
  
  post(at)fixtheweb.net
• tweet #fixtheweb #fail url problem

They will very shortly be able to use a browser toolbar too, which should make things even easier. The core team are intending to launch a website later in November, but there are the usual teething problems.

Currently, the main product of the Debian project is its stable release^[1]. Those release come out approximately every 18-24 months. This article gives a short overview of the process leading to the next stable release. Creating a new distribution Immediately after a stable release, a new distribution is created in the Debian archive. Its initial content is a copy of the (just released) stable distribution. Its codename is decided by the release managers and there s a tradition of picking a character s name from the Toy Story movie. As an example, the wheezy distribution will be created once squeeze (aka Debian 6.0) is out. For simplicity there s a generic name to refer to the distribution used to prepare the next stable release: it s testing. In the Debian archive, testing is just a symbolic link pointing to the right directory (squeeze currently). Updating packages, working on release goals During most of the cycle, developers work on packaging new upstream versions and implementing release goals. They upload their packages in the unstable distribution. From there packages trickle to the testing distribution once they satisfy some quality checks: they must not have new release-critical bugs, they must have been built on all architectures that were previously supported, they must not break any dependency in testing, and they must have spent at least 10 days in unstable. This minimal period ensures the package has been tested and gives enough time to users to file bugs if the package is suffering from problems. If the problems are deemed release-critical, they will block the migration of the package to testing. During this part of the release cycle, the main work of the release team is ensuring that updated packages flow from unstable to testing. It can be a tricky task: package dependencies frequently tie packages together so that they can only migrate to testing together. If only one of the tied packages is not ready (for example if a new revision has been uploaded and has not spent 10 days in unstable yet), then none of them can migrate. Stabilizing, polishing, fixing release-critical issues The constant churn of new packages makes it very difficult to build a very polished release. That s why, at some point, release managers freeze the testing distribution: automatic updates are stopped and they vet every single update made to testing. They have strong requirements, the goal is to only allow updates fixing release critical bugs, or those which are low-risk and bring significant value to the user experience (like new translations, updated documentation, etc.). During freeze, some packages are also removed because the current upstream version can t be supported for the lifetime of the stable release. The freeze tends to slow down the pace of changes in unstable. Many maintainers opt to push new upstream versions in experimental instead so that if they need to update their packages in testing, they can still do it through unstable. This procedure is recommended by the release managers because it means that updates that they unblock have been tested as usual. It s not the case for updates uploaded directly to testing (through testing-proposed-updates). This behavior is rather annoying for the bleeding-edge users that use testing or unstable like a rolling release. Release time Once release managers are satisfied of the quality of the new distribution, some last minute work is needed, like generating the CD images. In the Debian archive, the release is made official by pointing the stable symbolic link to the new distribution (and the oldstable one to the previous distribution). Now it s party time, the cycle is over, and a new one can start. ^[1] The Constantly Usable Testing project aims to make testing a first-class product like stable but with a very different update policy.

Follow me on Identi.ca, Twitter and Facebook.

3 comments Support my work

When talking about free culture or free software, many people suggest that they would love to support free models, but that they don't see how to make it all work. Until they have an alternate model in front of them, they cannot bring themselves to argue for a more ethical alternative. I disagree with this approach. Instead, I say, "this is the world I want to live in and, even though I don't know exactly how to get to there from here, I'm going to refuse to settle for anything short of this ideal." Most people dismiss such thinking as "impractical" and "unrealistic." I think most people are being unimaginative. Robot jockeys are one recent illustrative reason, among many, that I feel comfortable taking this position. Some background is necessary for those that are unfamiliar with the example. A decade ago, several Gulf emirates used thousands of young boys from Sudan and South Asia as jockeys for camel racing. Human rights groups campaigned against the practice and suggested that these boys were at sometimes held as slaves and intentionally underfed to keep their weight low. Despite criticism, camel racers resisted moving away from young boys as jockeys. If they moved to heavier adults instead of young boys, they reasoned, the camels would be much slower. Of course, they were right. But they were being unimaginative in the alternatives they were considering. As the young jockeys became a increasingly unjustifiable public relations disaster for the states that supported it, law-makers in several Gulf states gave in to calls from UNICEF and others and created laws to outlaw the practice. Within three years of UAE passing strict laws against child jockeys, Swiss engineers, funded by racers desperate for an alternative, had created the first robotic camel jockey. Within several years these jockeys were lighter, cheaper, more responsive to the owner, and well on their way to being more effective than any young boy. When forced, by law and by an ethical prerogative, to come up with an alternative to young boys, racers created a solution that was superior, along nearly every axis, to the system they had fought to keep. Although the costs to society of proprietary software cannot be compared to slavery and abuse, the basic same pattern of solution-seeking can be seen in the example of free software. Early free software advocates suggested that most programmers would likely need to take a paycut. As it turned out, vibrant and successful economic models to support free software have supported a large and growing free software industry. But we have free software business models only because a small group of principled individuals refused to settle for what they knew, came up with creative ethical business models that "just might work," and put their own paychecks on the line to try them out. As open source has shown, some of these creative solutions offered models superior to what we had before. In the world of software development, free software redefined "practical" and "realistic." One can think of solving human problems as like searching for the highest point in hilly terrain in thick fog. It's easy to get stuck on the top of the first little hill you walk up (i.e., a local maximum) and then conclude you can never do better. If we refuse to compromise and force ourselves to leave that first little hill, chances are pretty good we'll find a "higher" peak. Of course, it is also possible that we will find the global maximum or the best possible solution to a given problem. In those cases, any change will mean a sacrifice. But when dealing with most most social and legal dilemmas, there are enough variables involved that this seems very unlikely. Indeed, most big problems can be thought of as having many interacting dimensions -- and only some of these will be ethical concerns. In other words, most social problems are more like the problem of child camel jockeys than they are like trying to transcend the laws of physics. Business models and laws for the regulation of technology and knowledge are extremely complicated human creations. Do we really think we cannot create ethical systems to compensate cultural creators that are at least as good as what we have now? If we never force ourselves to be "impractical" and "unrealistic", we will never find out.

One more personal record broken for my 6th marathon (including those I ran alone). This time, it was a small marathon, the Seine-Eure marathon, in Normandy, between Paris and Rouen. But a small marathon is small only by number of runners (here, 500, plus about 160 teams of Ekiden runners -relay marathon-). It still has 42.195km. Despite a non optimal preparation (too many runs in September and no run over 25km since June), it went fairly well and I could benefit from the optimal conditions: ultra-flat race, few turns, few runners, goo dweather, though very cold for the season. Again, I started too fast (around 5'/km) and had trouble finishing with pace down to 5'30. But I'm still very happy for breaking my PB for the 4th time in a row with my results now being:

• 4h12'56" (Paris 2009)
• 4h15'10" (Rambouillet forest, 2009, ran alone)
• 4h00'49" (Cologne, 2009)
• 3h49'29" (Paris 2010)
• 3h43'14" (Maurepas 2010, ran alone)
• 3h40'20" (Val de Rueil)

Next marathon will be Paris marathon, on April 11th 2011. Target: 3h35 For french-speaking readers: full report on the Kikourou website.

Within the next few days there will be the openSUSE conference held in Nuremberg/Germany. The topic of the conference was set to Collaboration Across Borders and along that theme they invited other distributions to submit talks. Given my involvement with the Derivatives Frontdesk zack asked me whether I'd like to submit a talk proposal. I usually have a hard time to refuse such questions this will actually happen. My talk will be in the Distributions Track on Thursday afternoon, the topic is Debian The Project and its Resources. The resources that I will cover are those that can be very useful to other distributions too, and last Thursday I tested the talk at our local Debienna Meeting, gathered further ideas and feedback to cover in the talk. Thanks to you, guys! Unfortunately contrary to what we are used to in Debian (extraordinary thanks to the Video Team!!!) there will be no streaming or recording of the talks from what I was told, so you have either to hurry to be there, or be satisfied with the slides I'll put up after the talk. It's the best I can offer. See you around!

/debian permanent link Comments: 5

Before I upgraded from Lenny to Squeeze, all my CGI scripts were working properly. The CGI scripts has been executed by Apache and resulted in rendered webpage. After the upgrade to Squeeze all those CGI scripts stopped been executed but instead started to get displayed as a plain text. Common to all those CGI scripts is that they have .phtml as suffix, but the bang path/shebang consists of "#!/usr/bin/python". As an example you can have a look at Buildd.Net and its scripts like this one. The section in Apaches config looks like this:

<Directory /home/builddnet/unstable/WWW/cgi/>
Options -Indexes +ExecCGI +FollowSymLinks
AddHandler cgi-script .cgi .sh .pl .py .phtml
Order allow,deny
Allow from all
</Directory>

So, I would expect that the CGI script handler will get executed when loading a *.phtml file and that the shebang would be honoured. Funny enough: when I rename that script to *.cgi it works. I haven't figured out yet, what causes this beahviour or what have changed during the upgrade - and how to revert to the old behaviour. So, dear lazyweb, can you give me some hints and pointers?

Award winning code Me and Yuwei had a fun day at hhhmcr (#hhhmcr) and even managed to put together a prototype that won the first prize \o/ We played with the gmp24 dataset kindly extracted from Twitter by Michael Brunton-Spall of the Guardian into a convenient JSON dataset. The idea was to find ways of making it easier to look at the data and making sense of it. This is the story of what we did, including the code we wrote. The original dataset has several JSON files, so the first task was to put them all together:

#!/usr/bin/python
# Merge the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import os
res = []
for f in os.listdir("."):
    if not f.startswith("gmp24"): continue
    data = open(f).read().strip()
    if data == "[]": continue
    parsed = simplejson.loads(data)
    res.extend(parsed)
print simplejson.dumps(res)

The results however were not ordered by date, as GMP had to use several accounts to twit because Twitter was putting Greather Manchester Police into jail for generating too much traffic. There would be quite a bit to write about that, but let's stick to our work. Here is code to sort the JSON data by time:

#!/usr/bin/python
# Sort the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import sys
import datetime as dt
all_recs = simplejson.load(sys.stdin)
all_recs.sort(key=lambda x: dt.datetime.strptime(x["created_at"], "%a %b %d %H:%M:%S +0000 %Y"))
simplejson.dump(all_recs, sys.stdout)

I then wanted to play with Tf-idf for extracting the most important words of every tweet:

#!/usr/bin/python
# tfifd - Annotate JSON elements with Tf-idf extracted keywords
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import sys, math
import simplejson
import re
# Read all the twits
records = simplejson.load(sys.stdin)
# All the twits by ID
byid = dict(((x["id"], x) for x in records))
# Stopwords we ignore
stopwords = set(["by", "it", "and", "of", "in", "a", "to"])
# Tokenising engine
re_num = re.compile(r"^\d+$")
re_word = re.compile(r"(\w+)")
def tokenise(tweet):
    "Extract tokens from a tweet"
    for tok in tweet["text"].split():
        tok = tok.strip().lower()
        if re_num.match(tok): continue
        mo = re_word.match(tok)
        if not mo: continue
        if mo.group(1) in stopwords: continue
        yield mo.group(1)
# Extract tokens from tweets
tokenised = dict(((x["id"], list(tokenise(x))) for x in records))
# Aggregate token counts
aggregated =  
for d in byid.iterkeys():
    for t in tokenised[d]:
        if t in aggregated:
            aggregated[t] += 1
        else:
            aggregated[t] = 1
def tfidf(doc, tok):
    "Compute TFIDF score of a token in a document"
    return doc.count(tok) * math.log(float(len(byid)) / aggregated[tok])
# Annotate tweets with keywords
res = []
for name, tweet in byid.iteritems():
    doc = tokenised[name]
    keywords = sorted(set(doc), key=lambda tok: tfidf(doc, tok), reverse=True)[:5]
    tweet["keywords"] = keywords
    res.append(tweet)
simplejson.dump(res, sys.stdout)

I thought this was producing a nice summary of every tweet but nobody was particularly interested, so we moved on to adding categories to tweet. Thanks to Yuwei who put together some useful keyword sets, we managed to annotate each tweet with a place name (i.e. "Stockport"), a social place name (i.e. "pub", "bank") and a social category (i.e. "man", "woman", "landlord"...) The code is simple; the biggest work in it was the dictionary of keywords:

#!/usr/bin/python
# categorise - Annotate JSON elements with categories
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
# Copyright (C) 2010  Yuwei Lin <yuwei@ylin.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import sys, math
import simplejson
import re
# Electoral wards from http://en.wikipedia.org/wiki/List_of_electoral_wards_in_Greater_Manchester
placenames = ["Altrincham", "Sale West",
"Altrincham", "Ashton upon Mersey", "Bowdon", "Broadheath", "Hale Barns", "Hale Central", "St Mary", "Timperley", "Village",
"Ashton-under-Lyne",
"Ashton Hurst", "Ashton St Michael", "Ashton Waterloo", "Droylsden East", "Droylsden West", "Failsworth East", "Failsworth West", "St Peter",
"Blackley", "Broughton",
"Broughton", "Charlestown", "Cheetham", "Crumpsall", "Harpurhey", "Higher Blackley", "Kersal",
"Bolton North East",
"Astley Bridge", "Bradshaw", "Breightmet", "Bromley Cross", "Crompton", "Halliwell", "Tonge with the Haulgh",
"Bolton South East",
"Farnworth", "Great Lever", "Harper Green", "Hulton", "Kearsley", "Little Lever", "Darcy Lever", "Rumworth",
"Bolton West",
"Atherton", "Heaton", "Lostock", "Horwich", "Blackrod", "Horwich North East", "Smithills", "Westhoughton North", "Chew Moor", "Westhoughton South",
"Bury North",
"Church", "East", "Elton", "Moorside", "North Manor", "Ramsbottom", "Redvales", "Tottington",
"Bury South",
"Besses", "Holyrood", "Pilkington Park", "Radcliffe East", "Radcliffe North", "Radcliffe West", "St Mary", "Sedgley", "Unsworth",
"Cheadle",
"Bramhall North", "Bramhall South", "Cheadle", "Gatley", "Cheadle Hulme North", "Cheadle Hulme South", "Heald Green", "Stepping Hill",
"Denton", "Reddish",
"Audenshaw", "Denton North East", "Denton South", "Denton West", "Dukinfield", "Reddish North", "Reddish South",
"Hazel Grove",
"Bredbury", "Woodley", "Bredbury Green", "Romiley", "Hazel Grove", "Marple North", "Marple South", "Offerton",
"Heywood", "Middleton",
"Bamford", "Castleton", "East Middleton", "Hopwood Hall", "Norden", "North Heywood", "North Middleton", "South Middleton", "West Heywood", "West Middleton",
"Leigh",
"Astley Mosley Common", "Atherleigh", "Golborne", "Lowton West", "Leigh East", "Leigh South", "Leigh West", "Lowton East", "Tyldesley",
"Makerfield",
"Abram", "Ashton", "Bryn", "Hindley", "Hindley Green", "Orrell", "Winstanley", "Worsley Mesnes",
"Manchester Central",
"Ancoats", "Clayton", "Ardwick", "Bradford", "City Centre", "Hulme", "Miles Platting", "Newton Heath", "Moss Side", "Moston",
"Manchester", "Gorton",
"Fallowfield", "Gorton North", "Gorton South", "Levenshulme", "Longsight", "Rusholme", "Whalley Range",
"Manchester", "Withington",
"Burnage", "Chorlton", "Chorlton Park", "Didsbury East", "Didsbury West", "Old Moat", "Withington",
"Oldham East", "Saddleworth",
"Alexandra", "Crompton", "Saddleworth North", "Saddleworth South", "Saddleworth West", "Lees", "St James", "St Mary", "Shaw", "Waterhead",
"Oldham West", "Royton",
"Chadderton Central", "Chadderton North", "Chadderton South", "Coldhurst", "Hollinwood", "Medlock Vale", "Royton North", "Royton South", "Werneth",
"Rochdale",
"Balderstone", "Kirkholt", "Central Rochdale", "Healey", "Kingsway", "Littleborough Lakeside", "Milkstone", "Deeplish", "Milnrow", "Newhey", "Smallbridge", "Firgrove", "Spotland", "Falinge", "Wardle", "West Littleborough",
"Salford", "Eccles",
"Claremont", "Eccles", "Irwell Riverside", "Langworthy", "Ordsall", "Pendlebury", "Swinton North", "Swinton South", "Weaste", "Seedley",
"Stalybridge", "Hyde",
"Dukinfield Stalybridge", "Hyde Godley", "Hyde Newton", "Hyde Werneth", "Longdendale", "Mossley", "Stalybridge North", "Stalybridge South",
"Stockport",
"Brinnington", "Central", "Davenport", "Cale Green", "Edgeley", "Cheadle Heath", "Heatons North", "Heatons South", "Manor",
"Stretford", "Urmston",
"Bucklow-St Martins", "Clifford", "Davyhulme East", "Davyhulme West", "Flixton", "Gorse Hill", "Longford", "Stretford", "Urmston",
"Wigan",
"Aspull New Springs Whelley", "Douglas", "Ince", "Pemberton", "Shevington with Lower Ground", "Standish with Langtree", "Wigan Central", "Wigan West",
"Worsley", "Eccles South",
"Barton", "Boothstown", "Ellenbrook", "Cadishead", "Irlam", "Little Hulton", "Walkden North", "Walkden South", "Winton", "Worsley",
"Wythenshawe", "Sale East",
"Baguley", "Brooklands", "Northenden", "Priory", "Sale Moor", "Sharston", "Woodhouse Park"]
# Manual coding from Yuwei
placenames.extend(["City centre", "Tameside", "Oldham", "Bury", "Bolton",
"Trafford", "Pendleton", "New Moston", "Denton", "Eccles", "Leigh", "Benchill",
"Prestwich", "Sale", "Kearsley", ])
placenames.extend(["Trafford", "Bolton", "Stockport", "Levenshulme", "Gorton",
"Tameside", "Blackley", "City centre", "Airport", "South Manchester",
"Rochdale", "Chorlton", "Uppermill", "Castleton", "Stalybridge", "Ashton",
"Chadderton", "Bury", "Ancoats", "Whalley Range", "West Yorkshire",
"Fallowfield", "New Moston", "Denton", "Stretford", "Eccles", "Pendleton",
"Leigh", "Altrincham", "Sale", "Prestwich", "Kearsley", "Hulme", "Withington",
"Moss Side", "Milnrow", "outskirt of Manchester City Centre", "Newton Heath",
"Wythenshawe", "Mancunian Way", "M60", "A6", "Droylesden", "M56", "Timperley",
"Higher Ince", "Clayton", "Higher Blackley", "Lowton", "Droylsden",
"Partington", "Cheetham Hill", "Benchill", "Longsight", "Didsbury",
"Westhoughton"])
# Social categories from Yuwei
soccat = ["man", "woman", "men", "women", "youth", "teenager", "elderly",
"patient", "taxi driver", "neighbour", "male", "tenant", "landlord", "child",
"children", "immigrant", "female", "workmen", "boy", "girl", "foster parents",
"next of kin"]
for i in range(100):
    soccat.append("%d-year-old" % i)
    soccat.append("%d-years-old" % i)
# Types of social locations from Yuwei
socloc = ["car park", "park", "pub", "club", "shop", "premises", "bus stop",
"property", "credit card", "supermarket", "garden", "phone box", "theatre",
"toilet", "building site", "Crown court", "hard shoulder", "telephone kiosk",
"hotel", "restaurant", "cafe", "petrol station", "bank", "school",
"university"]
extras =   "placename": placenames, "soccat": soccat, "socloc": socloc  
# Normalise keyword lists
for k, v in extras.iteritems():
    # Remove duplicates
    v = list(set(v))
    # Sort by length
    v.sort(key=lambda x:len(x), reverse=True)
# Add keywords
def add_categories(tweet):
    text = tweet["text"].lower()
    for field, categories in extras.iteritems():
        for cat in categories:
            if cat.lower() in text:
                tweet[field] = cat
                break
    return tweet
# Read all the twits
records = (add_categories(x) for x in simplejson.load(sys.stdin))
simplejson.dump(list(records), sys.stdout)

All these scripts form a nice processing chain: each script takes a list of JSON records, adds some bit and passes it on. In order to see what we have so far, here is a simple script to convert the JSON twits to CSV so they can be viewed in a spreadsheet:

#!/usr/bin/python
# Convert the JSON twits to CSV
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import sys
import csv
rows = ["id", "created_at", "text", "keywords", "placename"]
writer = csv.writer(sys.stdout)
for rec in simplejson.load(sys.stdin):
    rec["keywords"] = " ".join(rec["keywords"])
    rec["placename"] = rec.get("placename", "")
    writer.writerow([rec[row] for row in rows])

At this point we were coming up with lots of questions: "were there more reports on women or men?", "which place had most incidents?", "what were the incidents involving animals?"... Time to bring Xapian into play. This script reads all the JSON tweets and builds a Xapian index with them:

#!/usr/bin/python
# toxapian - Index JSON tweets in Xapian
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import simplejson
import sys
import os, os.path
import xapian
DBNAME = sys.argv[1]
db = xapian.WritableDatabase(DBNAME, xapian.DB_CREATE_OR_OPEN)
stemmer = xapian.Stem("english")
indexer = xapian.TermGenerator()
indexer.set_stemmer(stemmer)
indexer.set_database(db)
data = simplejson.load(sys.stdin)
for rec in data:
    doc = xapian.Document()
    doc.set_data(str(rec["id"]))
    indexer.set_document(doc)
    indexer.index_text_without_positions(rec["text"])
    # Index categories as categories
    if "placename" in rec:
        doc.add_boolean_term("XP" + rec["placename"].lower())
    if "soccat" in rec:
        doc.add_boolean_term("XS" + rec["soccat"].lower())
    if "socloc" in rec:
        doc.add_boolean_term("XL" + rec["socloc"].lower())
    db.add_document(doc)
db.flush()
# Also save the whole dataset so we know where to find it later if we want to
# show the details of an entry
simplejson.dump(data, open(os.path.join(DBNAME, "all.json"), "w"))

And this is a simple command line tool to query to the database:

#!/usr/bin/python
# xgrep - Command line tool to query the GMP24 tweet Xapian database
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import simplejson
import sys
import os, os.path
import xapian
DBNAME = sys.argv[1]
db = xapian.Database(DBNAME)
stem = xapian.Stem("english")
qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
qp.add_boolean_prefix("place", "XP")
qp.add_boolean_prefix("soc", "XS")
qp.add_boolean_prefix("loc", "XL")
query = qp.parse_query(sys.argv[2],
    xapian.QueryParser.FLAG_BOOLEAN  
    xapian.QueryParser.FLAG_LOVEHATE  
    xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE  
    xapian.QueryParser.FLAG_WILDCARD  
    xapian.QueryParser.FLAG_PURE_NOT  
    xapian.QueryParser.FLAG_SPELLING_CORRECTION  
    xapian.QueryParser.FLAG_AUTO_SYNONYMS)
enquire = xapian.Enquire(db)
enquire.set_query(query)
count = 40
matches = enquire.get_mset(0, count)
estimated = matches.get_matches_estimated()
print "%d/%d results" % (matches.size(), estimated)
data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))
for m in matches:
    rec = data[m.document.get_data()]
    print rec["text"]
print "%d/%d results" % (matches.size(), matches.get_matches_estimated())
total = db.get_doccount()
estimated = matches.get_matches_estimated()
print "%d results over %d documents, %d%%" % (estimated, total, estimated * 100 / total)

Neat! Now that we have a proper index that supports all sort of cool things, like stemming, tag clouds, full text search with complex queries, lookup of similar documents, suggest keywords and so on, it was just fair to put together a web service to share it with other people at the event. It helped that I had already written similar code for apt-xapian-index and dde before. Here is the server, quickly built on bottle. The very last line starts the server and it is where you can configure the listening interface and port.

#!/usr/bin/python
# xserve - Make the GMP24 tweet Xapian database available on the web
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import bottle
from bottle import route, post
from cStringIO import StringIO
import cPickle as pickle
import simplejson
import sys
import os, os.path
import xapian
import urllib
import math
bottle.debug(True)
DBNAME = sys.argv[1]
QUERYLOG = os.path.join(DBNAME, "queries.txt")
data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))
prefixes =   "place": "XP", "soc": "XS", "loc": "XL"  
prefix_desc =   "place": "Place name", "soc": "Social category", "loc": "Social location"  
db = xapian.Database(DBNAME)
stem = xapian.Stem("english")
qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
for k, v in prefixes.iteritems():
    qp.add_boolean_prefix(k, v)
def make_query(qstring):
    return qp.parse_query(qstring,
        xapian.QueryParser.FLAG_BOOLEAN  
        xapian.QueryParser.FLAG_LOVEHATE  
        xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE  
        xapian.QueryParser.FLAG_WILDCARD  
        xapian.QueryParser.FLAG_PURE_NOT  
        xapian.QueryParser.FLAG_SPELLING_CORRECTION  
        xapian.QueryParser.FLAG_AUTO_SYNONYMS)
@route("/")
def index():
    query = urllib.unquote_plus(bottle.request.GET.get("q", ""))
    out = StringIO()
    print >>out, '''
<html>
<head>
<title>Query</title>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
<script type="text/javascript">
$(function() 
    $("#queryfield")[0].focus()
 )
</script>
</head>
<body>
<h1>Search</h1>
<form method="POST" action="/query">
Keywords: <input type="text" name="query" value="%s" id="queryfield">
<input type="submit">
<a href="http://xapian.org/docs/queryparser.html">Help</a>
</form>''' % query
    print >>out, '''
<p>Example: "car place:wigan"</p>

<p>Available prefixes:</p>

<ul>
'''
    for pfx in prefixes.keys():
        print >>out, "<li><a href='/catinfo/%s'>%s - %s</a></li>" % (pfx, pfx, prefix_desc[pfx])
    print >>out, '''
</ul>
'''
    oldqueries = []
    if os.path.exists(QUERYLOG):
        total = db.get_doccount()
        fd = open(QUERYLOG, "r")
        while True:
            try:
                q = pickle.load(fd)
            except EOFError:
                break
            oldqueries.append(q)
        fd.close()
        def print_query(q):
            count = q["count"]
            print >>out, "<li><a href='/query?query=%s'>%s (%d/%d %.2f%%)</a></li>" % (urllib.quote_plus(q["q"]), q["q"], count, total, count * 100.0 / total)
        print >>out, "<p>Last 10 queries:</p><ul>"
        for q in oldqueries[:-10:-1]:
            print_query(q)
        print >>out, "</ul>"
        # Remove duplicates
        oldqueries = dict(((x["q"], x) for x in oldqueries)).values()
        print >>out, "<table>"
        print >>out, "<tr><th>10 queries with most results</th><th>10 queries with least results</th></tr>"
        print >>out, "<tr><td>"
        print >>out, "<ul>"
        oldqueries.sort(key=lambda x:x["count"], reverse=True)
        for q in oldqueries[:10]:
            print_query(q)
        print >>out, "</ul>"
        print >>out, "</td><td>"
        print >>out, "<ul>"
        nonempty = [x for x in oldqueries if x["count"] > 0]
        nonempty.sort(key=lambda x:x["count"])
        for q in nonempty[:10]:
            print_query(q)
        print >>out, "</ul>"
        print >>out, "</td></tr>"
        print >>out, "</table>"
    print >>out, '''
</body>
</html>'''
    return out.getvalue()
@route("/query")
@route("/query/")
@post("/query")
@post("/query/")
def query():
    query = bottle.request.POST.get("query", bottle.request.GET.get("query", ""))
    enquire = xapian.Enquire(db)
    enquire.set_query(make_query(query))
    count = 40
    matches = enquire.get_mset(0, count)
    estimated = matches.get_matches_estimated()
    total = db.get_doccount()
    out = StringIO()
    print >>out, '''
<html>
<head><title>Results</title></head>
<body>
<h1>Results for "<b>%s</b>"</h1>
''' % query
    if estimated == 0:
        print >>out, "No results found."
    else:
        # Give as results the first 30 documents; also use them as the key
        # ones to use to compute relevant terms
        rset = xapian.RSet()
        for m in enquire.get_mset(0, 30):
            rset.add_document(m.document.get_docid())
        # Compute the tag cloud
        class NonTagFilter(xapian.ExpandDecider):
            def __call__(self, term):
                return not term[0].isupper() and not term[0].isdigit()
        cloud = []
        maxscore = None
        for res in enquire.get_eset(40, rset, NonTagFilter()):
            # Normalise the score in the interval [0, 1]
            weight = math.log(res.weight)
            if maxscore == None: maxscore = weight
            tag = res.term
            cloud.append([tag, float(weight) / maxscore])
        max_weight = cloud[0][1]
        min_weight = cloud[-1][1]
        cloud.sort(key=lambda x:x[0])
        def mklink(query, term):
            return "/query?query=%s" % urllib.quote_plus(query + " and " + term)
        print >>out, "<h2>Tag cloud</h2>"
        print >>out, "<blockquote>"
        for term, weight in cloud:
            size = 100 + 100.0 * (weight - min_weight) / (max_weight - min_weight)
            print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(query, term), size, term)
        print >>out, "</blockquote>"
        print >>out, "<h2>Results</h2>"
        print >>out, "<p><a href='/'>Search again</a></p>"
        print >>out, "<p>%d results over %d documents, %.2f%%</p>" % (estimated, total, estimated * 100.0 / total)
        print >>out, "<p>%d/%d results</p>" % (matches.size(), estimated)
        print >>out, "<ul>"
        for m in matches:
            rec = data[m.document.get_data()]
            print >>out, "<li><a href='/item/%s'>%s</a></li>" % (rec["id"], rec["text"])
        print >>out, "</ul>"
        fd = open(QUERYLOG, "a")
        qinfo = dict(q=query, count=estimated)
        pickle.dump(qinfo, fd)
        fd.close()
    print >>out, '''
<a href="/">Search again</a>

</body>
</html>'''
    return out.getvalue()
@route("/item/:id")
@route("/item/:id/")
def show(id):
    rec = data[id]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Result %s</title></head>
<body>
<h1>Raw JSON record for twit %s</h1>
<pre>''' % (rec["id"], rec["id"])
    print >>out, simplejson.dumps(rec, indent=" ")
    print >>out, '''
</pre>
</body>
</html>'''
    return out.getvalue()
@route("/catinfo/:name")
@route("/catinfo/:name/")
def catinfo(name):
    prefix = prefixes[name]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Values for %s</title></head>
<body>
''' % name
    terms = [(x.term[len(prefix):], db.get_termfreq(x.term)) for x in db.allterms(prefix)]
    terms.sort(key=lambda x:x[1], reverse=True)
    freq_min = terms[0][1]
    freq_max = terms[-1][1]
    def mklink(name, term):
        return "/query?query=%s" % urllib.quote_plus(name + ":" + term)
    # Build tag cloud
    print >>out, "<h1>Tag cloud</h1>"
    print >>out, "<blockquote>"
    for term, freq in sorted(terms[:20], key=lambda x:x[0]):
        size = 100 + 100.0 * (freq - freq_min) / (freq_max - freq_min)
        print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(name, term), size, term)
    print >>out, "</blockquote>"
    print >>out, "<h1>All terms</h1>"
    print >>out, "<table>"
    print >>out, "<tr><th>Occurrences</th><th>Name</th></tr>"
    for term, freq in terms:
        print >>out, "<tr><td>%d</td><td><a href='/query?query=%s'>%s</a></td></tr>" % (freq, urllib.quote_plus(name + ":" + term), term)
    print >>out, "</table>"
    print >>out, '''
</body>
</html>'''
    return out.getvalue()
# Change here for bind host and port
bottle.run(host="0.0.0.0", port=8024)

...and then we presented our work and ended up winning the contest. This was the story of how we wrote this set of award winning code.

If you are (like I do) experiencing issues with Google Chrome, in particular when it comes to applications such as Google Maps, try using the --disable-accelerated-compositing option.For me, the issues started with version 6.0 and resulted in various pages hanging at random, in particular graphics-heavy sites such as maps. For one of my own sites, markers started disappearing on maps at certain zoom levels. Only when it also occurred on the main Google Maps I stopped looking for bugs in my own site and started focusing on the browser ...