Some time ago, I asked what are the common solutions for services remote logging. It was suggested scribe, an application log server developed at Facebook and later released as an open source project, and we finally managed to get this ready for Debian squeeze.

While I have spent countless hours working on the new source format known as 3.0 (quilt) , I ve just realized that I have never blogged about its features and the reasons that lead me to work on it. Let s fix this. The good old 1.0 format Up to 2008, dpkg-source was only able to cope with a single source format (now named 1.0 ). That format was used since the inception of the project. While it worked fine for most cases, it suffered from a number of limitations mainly because it stored the Debian packaging files as a patch to apply on top of the upstream source tarball. This patch can have two functions: creating the required files in the debian sub-directory and applying changes to the upstream sources. Over time, if the maintainer made several modifications to the upstream source code, they would end up entangled (and undocumented) in this single patch. In order to solve this problem, patch systems were created (dpatch, quilt, simple-patchsys, dbs, ) and many maintainers started using them. Each implementation is slightly different but the basic principle is always the same: store the upstream changes as multiple patches in the debian/patches/ directory and apply them at build-time (and remove them during cleanup). Design goals for the new formats When I started working on the new source package format, I set out to get rid of all the known limitations and to integrate a patch system in dpkg-source. I wanted to clear up the situation so that learning packaging only requires to learn one patch system and would not require modifying debian/rules to use it. I picked quilt because it was popular, came with a large set of features, and was not suffering from NIH syndrome. This lead to the 3.0 (quilt) source format. I also created 3.0 (native) as a distinct format. 1.0 was able to generate two types of source packages (native and non-native) but I did not want to continue with this mistake of mixing both in a single format. The KISS principle dictated that the user should pick the format of his choice, put it in debian/source/format and be done with it. Now the build can rightfully fail when the requirements are not met instead of doing something unexpected as a fallback. Features of 3.0 (quilt) This is the format that replaces the non-native variant of the 1.0 source format. The features below are specific to the new format and differentiate it from its ancestor:

• Supports compression formats other than gzip: bzip2, lzma, xz.
• Can use multiple upstream tarballs.
• Can include binary files in the debian packaging.
• Automatically replaces the debian directory present in the upstream tarball (no repacking required).
• Creates a new quilt-managed patch in debian/patches/ when it finds changes to the upstream files.

Features of 3.0 (native) This format is very similar to the native variant of the 1.0 source format except for two things:

• it supports compression formats other than gzip: bzip2, lzma, xz.
• it excludes by default a bunch of files that should usually not be part of the tarball (VCS specific files, vim backup files, etc.)

Timeline Looking back at the history is interesting. This project already spans multiple years and is not really over until a majority of packages have switched to the new formats.

• January 2008: the discussion how to cope with patches sanely rages on debian-devel@lists.debian.org. My initial decisions are the result of this discussion.
• March 2008: I have implemented the new formats and I request feedback. dpkg 1.14.17 (uploaded to experimental) is the first release supporting them.
• April 2008: I ask ftpmasters to support the new source packages in #457345.
• June 2008: Lenny freeze. dpkg is not supposed to change anymore. Several changes concerning the new source formats are still accepted in the following months given that this code is not yet used in production and must only be present so that lenny can cope with new source packages once squeeze starts using them.
• February 2009: Lenny release.
• March 2009: Work on squeeze has started, ftpmasters have done nothing to support new source formats, I submit a patch in #457345 to speed things up. I start a wiki page to track the project s progress and to answer common questions of maintainers.
• November 2009: After an ftpmaster sprint, it s now possible to upload new source packages in unstable. This draws massive attention to the new format and some people start complaining about some design decisions. The implementation of 3.0 (quilt) changes a lot during this month. dpkg in lenny is even updated to keep up with those changes.
• March 2010: Up to now, I was planning to let dpkg-source build new source packages by default at some point in the future. After several rounds of discussions, I agree that it s not the best course of action and decides instead to make debian/source/format mandatory. The maintainer must be explicit about the source format that s/he wants to use.
• October 2010: The new source formats are relatively popular, a third of the source packages have already switched: see the graph. The squeeze freeze in August clearly stopped the trend, hopefully it will continue once squeeze is released.
• June 2013: Project is finished?

As you can see this project is not over yet, although the most difficult part is already behind me. For my part, the biggest lesson is that you won t ever get enough review until your work is used within unstable. So if you have a Debian project that impacts a lot of people, make sure to organize an official review process from the start. And specifying your project through a Debian Enhancement Proposal is probably the best way to achieve this. If you appreciate the work that I put into this project, feel free to join Flattr and to flattr dpkg from time to time. Or check out my page Support my work .

12 comments Support my work

After getting a flood of spam, I became suspicious that there was an exploit in my blog software allowing easy robo-posts. Despite a code audit I couldn't see anything, and thus wanted to log the incoming POST requests before any local processing at all. It took me a while to figure out how to do this, hopefully this helps someone else. Firstly install libapache-mod-security, then the magic incarnation is So, to break it down a little, the default action says to do nothing during phase 2 (when the body is available for inspection); the allow means that we're indicating that nothing further will happen in any of the remaining phases, so the module can shortcut through them. The two SecRules work together -- the first says that any POST requests should be tested by the next rule (i.e. the chained rule), which in this case says that any request should be sent to the audit log. After that, the similar allow/phase argument again says that nothing further is going to happen in any of the subsequent phases mod_security can work on. As per the parts between A and Z, we'll log the headers, the request body, the final response and trailer. So, as it turns out, there is no exploit; it seems most likely there is an actual human behind the spam that gets through, because every time they take a guess it is correct. So I guess I'll take a glass-half-full kind of approach and rather than being annoyed at removing the spam, I'll just convince myself that I made a small donation from some spam overlord to one of their poor minions!

The big advantage of Unison on Windows is that it allows quite easily to synchronize between Windows and Linux. For those who need to work on Windows with the same set of files as on Linux, this is a big plus. Other tools do it as well, but the 2-way sync of Unison is quite nice. When you need to compile a software on Linux and Windows, you can modify both side at the same time and (almost) don't have problems. On Windows, the .unison and unison.log are located into your %HOMEPATH%, which the upper directory of the classic Documents folder. In the directory .unison, you will find the .prf files that describe your unison profiles. As usual, default.prf in this directory is the default profile. Basic The basic tips are:

• use fastcheck = true in your default.prf
• disable directory indexing

You can also disable virus live scan -- if you think it is safe!!!! SSH Using ssh under Windows is always a challenge. As a matter of fact, this tool doesn't match Windows context and it is not as integrated as in Linux/BSD. There is Putty which can help you. It has a good support for remote shell but it is not very easy to setup with Unison. Putty and OpenSSH doesn't have precisely the same set of options and Unison relies on some not available in Putty. There is a script called ssh2plink.bat that can help you using Putty's plink as with Unison. I used it for a while, but this didn't give the expected throughput. The best option is to use the ssh command provided by Cygwin. In this case you have at the same time good throughput and unison integration. I explain here how to configure you Cygwin's ssh to use a SSH key. You can bypass the following steps, if you wish to use a password or if you have already setup your ssh to connect to the target computer. Launch Cygwin's setup.exe and select openssh for installation. To add a SSH key, launch the cygwin shell:

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
[...]

Copy the file .ssh/id_rsa.pub to your target computer's .ssh/authorized_keys. You should be aware that the file format can be Windows EOL style (in this case use dos2unix to convert the file) and if you copy/paste from a dos box, some end of lines are added and you should remove them from the authorized_keys, to have a single line key. Once, you have installed your ssh key into target computer, you should try to connect directly from the cygwin shell.

$ ssh XXX

Now, you can configure sshcmd = c:\cygwin\bin\ssh.exe to your default.prf. Using Cygwin's ssh allow you to get ~2MB/s (or more) when you only get ~100KB/s using ssh2plink.bat. If you have any other tips to improve Unison on Windows, I will be happy to test them and post it here.

In today s Linux distributions, there are usually two to three levels of package management. In this blog post, I will explain the three levels of package management. 1. Local (dpkg, rpm) The first level of package management is the local level. This level consists of package management tools that install and/or remove packages via package archives such as .deb or .rpm files and a database of the local s system state. 2. Connected (APT, YUM, Smart) The second level of package management is the connected level. In this level, tools are fetching packages and information about them from (remote) locations known as repositories or sources . A level 2 package manager is usually built on top of a level 1 package manager, for example, APT uses dpkg but there are also cases where one tool is both, such as opkg. 3. Abstract (PackageKit, Smart) A level 3 package manager is a tool that supports different level 1 package managers via one generic interface. It may be implemented on top of a level 2 package manager (PackageKit), or may be implemented directly in level 2 (Smart). Conclusion: Merge Level 2 and 3 Most level 2 package managers share a great deal of tasks, such as solving dependency problems. Merging level 2 package management and level 3 package management would enable us to reduce the number of dependency problem solvers and combine our efforts into a few package managers. Such an implementation would need to be written in C and support all common level 1 package managers in order to be successful. As some might have guessed, this is what I plan to do with the project codenamed APT2, although work is not progressing very fast. PS: Back from vacation I spent the more than the complete last week in (mostly rainy) Corfu, Greece and am now getting back to my usual development stuff. I have already processed my news & blogs backlog (1000+ messages) and will now start with my email backlog, so don t be angry if the answer to your email takes some time.
Filed under: APT2, General

If you re running a 64bit system, and you ve got users with access to a video device (/dev/video*), then be sure you update your kernels for CVE-2010-2963. I ve been slowly making my way through auditing the many uses in the Linux kernel of the copy_from_user() function, and ran into this vulnerability. Here s the kernel code from drivers/media/video/v4l2-compat-ioctl32.c:

static int get_microcode32(struct video_code *kp, struct video_code32 __user *up)
 
        if (!access_ok(VERIFY_READ, up, sizeof(struct video_code32))  
                copy_from_user(kp->loadwhat, up->loadwhat, sizeof(up->loadwhat))  
                get_user(kp->datasize, &up->datasize)  
                copy_from_user(kp->data, up->data, up->datasize))
                        return -EFAULT;
        return 0;
 

Note that kp->data is being used as the target for up->data in the final copy_from_user() without actually verifying that kp->data is pointing anywhere safe. Here s the caller of get_microcode32:

static long do_video_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 
        union  
                struct video_tuner vt;
                struct video_code vc;
...
          karg;
        void __user *up = compat_ptr(arg);
...
        switch (cmd)  
...
        case VIDIOCSMICROCODE:
                err = get_microcode32(&karg.vc, up);
...

So, the contents of up are totally under control of the caller, and the contents of karg (in our case, the video_code structure) are not initialized at all. So, it seems like a call for VIDIOCSMICROCODE would write video_code->datasize bytes from video_code->data into some random kernel address, just causing an Oops, since we don t control what is on the kernel s stack. But wait, who says we can t control the contents of the kernel s stack? In fact, this compat function makes it extremely easy. Let s look back at the union. Notice the struct video_tuner? That gets populated from the caller s up memory via this case of the switch (cmd) statement:

...
        case VIDIOCSTUNER:
        case VIDIOCGTUNER:
                err = get_video_tuner32(&karg.vt, up);
...

So, to control the kernel stack, we just need to call this ioctl twice in a row: once to populate the stack via VIDIOCSTUNER with the contents we want (including the future address for video_code->data, which starts at the same location as video_tuner->name[20]), and then again with VIDIOCSMICROCODE. Tricks involved here are: the definition of the VIDIOCSMICROCODE case in the kernel is wrong, and calling the ioctls without any preparation can trigger other kernel work (memory faults, etc) that may destroy the stack contents. First, we need the real value for the desired case statement. This turns out to be 0 4020761b. Next, we just repeatedly call the setup ioctl in an attempt to get incidental kernel work out of the way so that our last ioctl doing the stack preparation will stick, and then we call the buggy ioctl to trigger the vulnerability. Since the ioctl already does a multi-byte copy, we can now copy arbitrary lengths of bytes into kernel memory. One method of turning an arbitrary kernel memory write into a privilege escalation is to overwrite a kernel function pointer, and trigger that function. Based on the exploit for CVE-2010-3081, I opted to overwrite the security_ops function pointer table. Their use of msg_queue_msgctl wasn t very good for the general case since it s near the end of the table and its offset would depend on kernel versions. Initially I opted for getcap, but in the end used ptrace_traceme, both of which are very near the top the security_ops structure. (Though I need share credit here with Dan Rosenberg as we were working together on improving the reliability of the security_ops overwrite method. He used the same approach for his excellent RDS exploit.) Here are the steps for one way of taking an arbitrary kernel memory write and turning it into a root escalation:

• overwrite security_ops with default_security_ops, which will revert the LSM back to the capabilities-only security operations. This, however, means we can calculate where cap_ptrace_traceme is.
• overwrite default_security_ops->ptrace_traceme to point to our supplied function that will actually perform the privilege escalation (thanks to Brad Spengler for his code from Enlightenment).
• trigger the function (in this case, call ptrace(PTRACE_TRACEME, 0, NULL, NULL)).
• restore default_security_ops->ptrace_traceme to point to cap_ptrace_traceme so the next caller doesn t Oops the system (since userspace memory will be remapped).

Here s the source for Vyakarana as seen running in Enlightenment using cap_getcap (which is pretty unstable, so you might want to switch it to use ptrace_traceme), and as a stand-alone memory writer. Conclusions: Keep auditing the kernel for more arbitrary writes; I think there are still many left. Reduce the exploitation surface within the kernel itself (which PaX and grsecurity have been doing for a while now), specifically:

• Block userspace memory access while in kernel mode. This would stop the ability to make the kernel start executing functions that live in userspace a clear privilege violation. This protection would stop the current exploit above, but the exploit could be adjusted to use kernel memory instead.
• Keep function pointers read-only. There is no reason for these function pointer tables (fops, IDT, security_ops, etc) to be writable. These should all be marked correctly, with inline code exceptions being made for updating the global pointers to those tables, leaving the pointer read-only after it gets set. This would stop this particular exploit above, but there are still plenty more targets.
• Randomize the kernel stack location on a per-syscall basis. This will stop exploits that depend on a stable kernel stack location (as this exploit does).

Nice tip, I didn t know about it.

State of X Server 1.9 The push of X Server 1.9 mentioned previously continued past weekend, with an upload of 1.9.1 rc2 this time, which is likely to become 1.9.1 in a few days. To keep it company in experimental, almost all X video drivers were uploaded as well, with a few exceptions:

• intel: We would like the current version in sid to migrate to squeeze now that a suitable kernel ( 2.6.32-24) is available there. Additional patches from experimental or from upstream might be added to sid then. For now, I kept 2.13 (built against X Server 1.7) in experimental, since I asked users a couple of times to try 2.13 from there.
• nouveau and openchrome: There were entirely new codebases in experimental for both of them, so I didn t touch anything there.
• nv: We plan to drop it entirely in favor of nouveau.

If my package manager isn t lying, the xserver-xorg-video-all meta package pulling everything could almost be installed, if one were to ignore those drivers. The same goes for xserver-xorg-input-all: almost alright, except for xserver-xorg-input-wacom, which is maintained outside debian-x. Autobuilding X packages So we have mostly this situation: 1.7 in sid, 1.9 in experimental. But we may want to build experimental packages (new upstream releases, or packages with patches cherry-picked from upstream) against either of them, without interfering with what is in the archive. We could ask users to try some patches. Actually, we do that already, and while many of them are usually responsive and willing to do so, not everyone knows how to build a package, or has time to. That s why I spent some time setting up http://autobuild.ikibiki.org/, where I can push source packages in the autobuild-unstable or autobuild-experimental suites, so that they get built on amd64 and on i386. It supports multiple versions in a given suite, which may help narrowing down when a fix appeared. That s part of a let s try to stick as closely as possible to upstream plan. We regularly get blamed for not doing so, but there are several reasons (or excuses? You decide.) for that:

• Spending time stabilizing stuff is the established standard in Debian, with that stuff called freeze. This means that rushing for the very last upstream releases isn t usually a good idea.
• X is some kind of a beast. Many libraries, many drivers, and a server. That means plenty of time to spend. And we re only a handful (and that includes Ubuntu folks who are helping us preparing stuff in git, thanks guys).

So if anyone thinks we re willingly holding back any kind of move towards newer upstream releases or the like, rest assured that s not the case. There s just too few of us to closely track upstream (which we do by reporting bugs and trying to cherry-pick patches anyway) in addition to preparing stuff for the next stable release. Now that things are settling down, that sounds feasible somehow, but remember: your help is more than welcome. Speaking of those Ubuntu guys, Robert Hooker is currently updating a bunch of drivers in experimental, yay.

Neil asks about removing debconf old values when replacing a package with a different one. Usually, you shouldn't need to worry about this; when the old package is finally purged all its debconf values will be cleaned up. In case that doesn't happen for some reason, you do not need to include the templates in the new package in order to get them removed. Actually, due to debconf's reference counting, that shouldn't work anyway. What you need is debconf-communicate(1). Specifically, something like this in a maintainer script of the new package, before it sources debconf's confmodule:

echo purge   debconf-communicate oldpackage

(Updated: Fixed incorrect syntax!)

---

Meh, databases add such complexity.. If I were redesigning debconf today, it would use flat files on disk and those files would be removed by dpkg on purge.

time flies when you're (busy) having fun If you could excuse me for an off-by-one (day) error, exactly 6 months ago I cried OH MY! in reaction to DPL election results. Time flies when you're (busy) having fun. 6 months it's a long time, lots of stuff can happen in such a time frame. But 6 months is also a relatively short time frame to get up to speed in a new job. 6 months, finally, also corresponds to the midterm of any given DPL term. While I've been doing a sort of personal "self assessment" of my DPL activities (and sooner or later I will take the time to put that down into public writing), this seems to be the perfect moment to ask others for feedback. 6 more months of the current term are ahead of us, and there is still time to change or adapt what I've been doing and how I've been doing it. So, in case you care about what a DPL can do to make your staying in Debian more fun and useful for the Project, then this is your turn to let me know. Feel free to mention not only what you think I've been doing right, but also and in fact more importantly what you think I've been doing wrong. (Tip of the day: as usual in volunteer communities, lingering on the side of constructive criticism will maximize the chances of having your suggestions implemented.) I would very much appreciate if you could take a moment to think about it and get back to me, via the medium you prefer: leave a comment here, mail me, or whatever else you please. Remember that a good deal of DPL day to day activities is about representing the Debian Project and it's entirely up to you, project members, to say how that should best be done. So many thanks in advance for your invaluable feedback, I'm looking forward to it!

Currently, the main product of the Debian project is its stable release^[1]. Those release come out approximately every 18-24 months. This article gives a short overview of the process leading to the next stable release. Creating a new distribution Immediately after a stable release, a new distribution is created in the Debian archive. Its initial content is a copy of the (just released) stable distribution. Its codename is decided by the release managers and there s a tradition of picking a character s name from the Toy Story movie. As an example, the wheezy distribution will be created once squeeze (aka Debian 6.0) is out. For simplicity there s a generic name to refer to the distribution used to prepare the next stable release: it s testing. In the Debian archive, testing is just a symbolic link pointing to the right directory (squeeze currently). Updating packages, working on release goals During most of the cycle, developers work on packaging new upstream versions and implementing release goals. They upload their packages in the unstable distribution. From there packages trickle to the testing distribution once they satisfy some quality checks: they must not have new release-critical bugs, they must have been built on all architectures that were previously supported, they must not break any dependency in testing, and they must have spent at least 10 days in unstable. This minimal period ensures the package has been tested and gives enough time to users to file bugs if the package is suffering from problems. If the problems are deemed release-critical, they will block the migration of the package to testing. During this part of the release cycle, the main work of the release team is ensuring that updated packages flow from unstable to testing. It can be a tricky task: package dependencies frequently tie packages together so that they can only migrate to testing together. If only one of the tied packages is not ready (for example if a new revision has been uploaded and has not spent 10 days in unstable yet), then none of them can migrate. Stabilizing, polishing, fixing release-critical issues The constant churn of new packages makes it very difficult to build a very polished release. That s why, at some point, release managers freeze the testing distribution: automatic updates are stopped and they vet every single update made to testing. They have strong requirements, the goal is to only allow updates fixing release critical bugs, or those which are low-risk and bring significant value to the user experience (like new translations, updated documentation, etc.). During freeze, some packages are also removed because the current upstream version can t be supported for the lifetime of the stable release. The freeze tends to slow down the pace of changes in unstable. Many maintainers opt to push new upstream versions in experimental instead so that if they need to update their packages in testing, they can still do it through unstable. This procedure is recommended by the release managers because it means that updates that they unblock have been tested as usual. It s not the case for updates uploaded directly to testing (through testing-proposed-updates). This behavior is rather annoying for the bleeding-edge users that use testing or unstable like a rolling release. Release time Once release managers are satisfied of the quality of the new distribution, some last minute work is needed, like generating the CD images. In the Debian archive, the release is made official by pointing the stable symbolic link to the new distribution (and the oldstable one to the previous distribution). Now it s party time, the cycle is over, and a new one can start. ^[1] The Constantly Usable Testing project aims to make testing a first-class product like stable but with a very different update policy.

Follow me on Identi.ca, Twitter and Facebook.

3 comments Support my work

When talking about free culture or free software, many people suggest that they would love to support free models, but that they don't see how to make it all work. Until they have an alternate model in front of them, they cannot bring themselves to argue for a more ethical alternative. I disagree with this approach. Instead, I say, "this is the world I want to live in and, even though I don't know exactly how to get to there from here, I'm going to refuse to settle for anything short of this ideal." Most people dismiss such thinking as "impractical" and "unrealistic." I think most people are being unimaginative. Robot jockeys are one recent illustrative reason, among many, that I feel comfortable taking this position. Some background is necessary for those that are unfamiliar with the example. A decade ago, several Gulf emirates used thousands of young boys from Sudan and South Asia as jockeys for camel racing. Human rights groups campaigned against the practice and suggested that these boys were at sometimes held as slaves and intentionally underfed to keep their weight low. Despite criticism, camel racers resisted moving away from young boys as jockeys. If they moved to heavier adults instead of young boys, they reasoned, the camels would be much slower. Of course, they were right. But they were being unimaginative in the alternatives they were considering. As the young jockeys became a increasingly unjustifiable public relations disaster for the states that supported it, law-makers in several Gulf states gave in to calls from UNICEF and others and created laws to outlaw the practice. Within three years of UAE passing strict laws against child jockeys, Swiss engineers, funded by racers desperate for an alternative, had created the first robotic camel jockey. Within several years these jockeys were lighter, cheaper, more responsive to the owner, and well on their way to being more effective than any young boy. When forced, by law and by an ethical prerogative, to come up with an alternative to young boys, racers created a solution that was superior, along nearly every axis, to the system they had fought to keep. Although the costs to society of proprietary software cannot be compared to slavery and abuse, the basic same pattern of solution-seeking can be seen in the example of free software. Early free software advocates suggested that most programmers would likely need to take a paycut. As it turned out, vibrant and successful economic models to support free software have supported a large and growing free software industry. But we have free software business models only because a small group of principled individuals refused to settle for what they knew, came up with creative ethical business models that "just might work," and put their own paychecks on the line to try them out. As open source has shown, some of these creative solutions offered models superior to what we had before. In the world of software development, free software redefined "practical" and "realistic." One can think of solving human problems as like searching for the highest point in hilly terrain in thick fog. It's easy to get stuck on the top of the first little hill you walk up (i.e., a local maximum) and then conclude you can never do better. If we refuse to compromise and force ourselves to leave that first little hill, chances are pretty good we'll find a "higher" peak. Of course, it is also possible that we will find the global maximum or the best possible solution to a given problem. In those cases, any change will mean a sacrifice. But when dealing with most most social and legal dilemmas, there are enough variables involved that this seems very unlikely. Indeed, most big problems can be thought of as having many interacting dimensions -- and only some of these will be ethical concerns. In other words, most social problems are more like the problem of child camel jockeys than they are like trying to transcend the laws of physics. Business models and laws for the regulation of technology and knowledge are extremely complicated human creations. Do we really think we cannot create ethical systems to compensate cultural creators that are at least as good as what we have now? If we never force ourselves to be "impractical" and "unrealistic", we will never find out.

After a somewhat long hiatus, RPostgreSQL version 0.1-7 has now been released to CRAN. RPostgreSQL connects R to PostgreSQL database systems using the standard DBI interface. This version fixes a number of issues that had been compiled in the issue tracker on the project site at Google Code. Tomoaki Nishiyama, who joined our small development group for his package a few weeks ago, was instrumental in a number of these fixes, with assistance from Joe Conway. The relevant NEWS file entry follows below:

Version 0.1-7 -- 2010-10-17
    o   Several potential buffer overruns were fixed
    o   dbWriteTable now writes a data.frame to database through a network
        connection rather than a temporary file. Note that row_names may be
        changed in future releases.  Also, passing in filenames instead of
        data.frame is not supported at this time. 
    o   When no host is specified, a connection to the PostgreSQL server 
        is made via UNIX domain socket (just like psql does)
    o   Table and column names are case sensitive, and identifiers are escaped
        or quoted appropriately, so that any form of table/column names can be
        created, searched, or removed, including upper-, lower- and mixed-case.
    o   nullOk in dbColumnInfo has a return value of NA when the column does
        not correspond to a column in the table. The utility of nullOk is
        doubtful but not removed at this time.
    o   Correct Windows getpid() declaration (with thanks to Brian D. Ripley)
    o   A call of as.POSIXct() with a time format string wrongly passed to TZ
        has been corrected; this should help with intra-day timestamps (with
        thanks to Steve Eick)
    o   Usage of tmpdir has been improved on similarly to Linux (with thanks
        to Robert McGehee)

More information is on the my RPostgreSQL page, and on project site at Google Code.

I m currently working on something which will eventually amount to a re-bootstrapping of sorts of the Debian/m68k (Linux) port patches to the Linux kernel, gcc, etc. are prepared (some have been accepted into upstream or the packages already). I will probably have more, once the compile processes finish, anyway (even emulated, it s slow). I think that, once I get past that TLS (thread-local storage, needed by eglibc) migration, I will try to find out a list of packages needed for debootstrap (AFAICT: all packages marked Essential, or of priority important, and all marked Build-Essential (for the *-builder variant), and their dependencies (although I ll substitute sysv-rc with file-rc, which is better and needs less deps)) and pull arch:all from sid, then build the rest myself using a consistent snapshot of sid possibly with patches going to unreleased. Then I can use cowbuilder to make cleaner packages, which can eventually be uploaded (once I get enough to get a buildd running kernel, bootloader, etc) binNMUs are way to go here I suppose. I will only upload once it s self-hosted, installable (seen by edos-debcheck), clean, etc. (i.e. I ll rebuild all binaries) and probably keep a bootstrap repo around (until m68k caught up) so that unstable (possibly amended by unreleased for a while) will not again become uninstallable, e.g. if arch:all packages change their dependencies (Python, gcc-defaults are some I ve seen). That bootstrap repository is needed anyway because debootstrap can t install from two separate repositories (unstable+unreleased for example). Progress is slow because I try to keep as close to official packages as possible, refuse to cross-compile, and try to produce uploadable if possible packages all the time. Getting patches into packages, so that I can build from unstable, instead of debian-ports.org unreleased, has proven time-consuming and occasionally frustrating as well. Although I would like to thank the people who helped me on the way already. (I am not naming any in fear of forgetting some, but you know who you are ) They are among the Debian (gcc, kernel, m68k) and Linux-68k crowd. (Why does genattrtab in gcc-4.4 take 3 hours when it took less than half an hour in gcc-4.3 anyway?) I m also still working on mksh and some Python ISO hacks for mika and some minor stuff, and further cleaning up MirBSD.

Award winning code Me and Yuwei had a fun day at hhhmcr (#hhhmcr) and even managed to put together a prototype that won the first prize \o/ We played with the gmp24 dataset kindly extracted from Twitter by Michael Brunton-Spall of the Guardian into a convenient JSON dataset. The idea was to find ways of making it easier to look at the data and making sense of it. This is the story of what we did, including the code we wrote. The original dataset has several JSON files, so the first task was to put them all together:

#!/usr/bin/python
# Merge the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import os
res = []
for f in os.listdir("."):
    if not f.startswith("gmp24"): continue
    data = open(f).read().strip()
    if data == "[]": continue
    parsed = simplejson.loads(data)
    res.extend(parsed)
print simplejson.dumps(res)

The results however were not ordered by date, as GMP had to use several accounts to twit because Twitter was putting Greather Manchester Police into jail for generating too much traffic. There would be quite a bit to write about that, but let's stick to our work. Here is code to sort the JSON data by time:

#!/usr/bin/python
# Sort the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import sys
import datetime as dt
all_recs = simplejson.load(sys.stdin)
all_recs.sort(key=lambda x: dt.datetime.strptime(x["created_at"], "%a %b %d %H:%M:%S +0000 %Y"))
simplejson.dump(all_recs, sys.stdout)

I then wanted to play with Tf-idf for extracting the most important words of every tweet:

#!/usr/bin/python
# tfifd - Annotate JSON elements with Tf-idf extracted keywords
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import sys, math
import simplejson
import re
# Read all the twits
records = simplejson.load(sys.stdin)
# All the twits by ID
byid = dict(((x["id"], x) for x in records))
# Stopwords we ignore
stopwords = set(["by", "it", "and", "of", "in", "a", "to"])
# Tokenising engine
re_num = re.compile(r"^\d+$")
re_word = re.compile(r"(\w+)")
def tokenise(tweet):
    "Extract tokens from a tweet"
    for tok in tweet["text"].split():
        tok = tok.strip().lower()
        if re_num.match(tok): continue
        mo = re_word.match(tok)
        if not mo: continue
        if mo.group(1) in stopwords: continue
        yield mo.group(1)
# Extract tokens from tweets
tokenised = dict(((x["id"], list(tokenise(x))) for x in records))
# Aggregate token counts
aggregated =  
for d in byid.iterkeys():
    for t in tokenised[d]:
        if t in aggregated:
            aggregated[t] += 1
        else:
            aggregated[t] = 1
def tfidf(doc, tok):
    "Compute TFIDF score of a token in a document"
    return doc.count(tok) * math.log(float(len(byid)) / aggregated[tok])
# Annotate tweets with keywords
res = []
for name, tweet in byid.iteritems():
    doc = tokenised[name]
    keywords = sorted(set(doc), key=lambda tok: tfidf(doc, tok), reverse=True)[:5]
    tweet["keywords"] = keywords
    res.append(tweet)
simplejson.dump(res, sys.stdout)

I thought this was producing a nice summary of every tweet but nobody was particularly interested, so we moved on to adding categories to tweet. Thanks to Yuwei who put together some useful keyword sets, we managed to annotate each tweet with a place name (i.e. "Stockport"), a social place name (i.e. "pub", "bank") and a social category (i.e. "man", "woman", "landlord"...) The code is simple; the biggest work in it was the dictionary of keywords:

#!/usr/bin/python
# categorise - Annotate JSON elements with categories
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
# Copyright (C) 2010  Yuwei Lin <yuwei@ylin.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import sys, math
import simplejson
import re
# Electoral wards from http://en.wikipedia.org/wiki/List_of_electoral_wards_in_Greater_Manchester
placenames = ["Altrincham", "Sale West",
"Altrincham", "Ashton upon Mersey", "Bowdon", "Broadheath", "Hale Barns", "Hale Central", "St Mary", "Timperley", "Village",
"Ashton-under-Lyne",
"Ashton Hurst", "Ashton St Michael", "Ashton Waterloo", "Droylsden East", "Droylsden West", "Failsworth East", "Failsworth West", "St Peter",
"Blackley", "Broughton",
"Broughton", "Charlestown", "Cheetham", "Crumpsall", "Harpurhey", "Higher Blackley", "Kersal",
"Bolton North East",
"Astley Bridge", "Bradshaw", "Breightmet", "Bromley Cross", "Crompton", "Halliwell", "Tonge with the Haulgh",
"Bolton South East",
"Farnworth", "Great Lever", "Harper Green", "Hulton", "Kearsley", "Little Lever", "Darcy Lever", "Rumworth",
"Bolton West",
"Atherton", "Heaton", "Lostock", "Horwich", "Blackrod", "Horwich North East", "Smithills", "Westhoughton North", "Chew Moor", "Westhoughton South",
"Bury North",
"Church", "East", "Elton", "Moorside", "North Manor", "Ramsbottom", "Redvales", "Tottington",
"Bury South",
"Besses", "Holyrood", "Pilkington Park", "Radcliffe East", "Radcliffe North", "Radcliffe West", "St Mary", "Sedgley", "Unsworth",
"Cheadle",
"Bramhall North", "Bramhall South", "Cheadle", "Gatley", "Cheadle Hulme North", "Cheadle Hulme South", "Heald Green", "Stepping Hill",
"Denton", "Reddish",
"Audenshaw", "Denton North East", "Denton South", "Denton West", "Dukinfield", "Reddish North", "Reddish South",
"Hazel Grove",
"Bredbury", "Woodley", "Bredbury Green", "Romiley", "Hazel Grove", "Marple North", "Marple South", "Offerton",
"Heywood", "Middleton",
"Bamford", "Castleton", "East Middleton", "Hopwood Hall", "Norden", "North Heywood", "North Middleton", "South Middleton", "West Heywood", "West Middleton",
"Leigh",
"Astley Mosley Common", "Atherleigh", "Golborne", "Lowton West", "Leigh East", "Leigh South", "Leigh West", "Lowton East", "Tyldesley",
"Makerfield",
"Abram", "Ashton", "Bryn", "Hindley", "Hindley Green", "Orrell", "Winstanley", "Worsley Mesnes",
"Manchester Central",
"Ancoats", "Clayton", "Ardwick", "Bradford", "City Centre", "Hulme", "Miles Platting", "Newton Heath", "Moss Side", "Moston",
"Manchester", "Gorton",
"Fallowfield", "Gorton North", "Gorton South", "Levenshulme", "Longsight", "Rusholme", "Whalley Range",
"Manchester", "Withington",
"Burnage", "Chorlton", "Chorlton Park", "Didsbury East", "Didsbury West", "Old Moat", "Withington",
"Oldham East", "Saddleworth",
"Alexandra", "Crompton", "Saddleworth North", "Saddleworth South", "Saddleworth West", "Lees", "St James", "St Mary", "Shaw", "Waterhead",
"Oldham West", "Royton",
"Chadderton Central", "Chadderton North", "Chadderton South", "Coldhurst", "Hollinwood", "Medlock Vale", "Royton North", "Royton South", "Werneth",
"Rochdale",
"Balderstone", "Kirkholt", "Central Rochdale", "Healey", "Kingsway", "Littleborough Lakeside", "Milkstone", "Deeplish", "Milnrow", "Newhey", "Smallbridge", "Firgrove", "Spotland", "Falinge", "Wardle", "West Littleborough",
"Salford", "Eccles",
"Claremont", "Eccles", "Irwell Riverside", "Langworthy", "Ordsall", "Pendlebury", "Swinton North", "Swinton South", "Weaste", "Seedley",
"Stalybridge", "Hyde",
"Dukinfield Stalybridge", "Hyde Godley", "Hyde Newton", "Hyde Werneth", "Longdendale", "Mossley", "Stalybridge North", "Stalybridge South",
"Stockport",
"Brinnington", "Central", "Davenport", "Cale Green", "Edgeley", "Cheadle Heath", "Heatons North", "Heatons South", "Manor",
"Stretford", "Urmston",
"Bucklow-St Martins", "Clifford", "Davyhulme East", "Davyhulme West", "Flixton", "Gorse Hill", "Longford", "Stretford", "Urmston",
"Wigan",
"Aspull New Springs Whelley", "Douglas", "Ince", "Pemberton", "Shevington with Lower Ground", "Standish with Langtree", "Wigan Central", "Wigan West",
"Worsley", "Eccles South",
"Barton", "Boothstown", "Ellenbrook", "Cadishead", "Irlam", "Little Hulton", "Walkden North", "Walkden South", "Winton", "Worsley",
"Wythenshawe", "Sale East",
"Baguley", "Brooklands", "Northenden", "Priory", "Sale Moor", "Sharston", "Woodhouse Park"]
# Manual coding from Yuwei
placenames.extend(["City centre", "Tameside", "Oldham", "Bury", "Bolton",
"Trafford", "Pendleton", "New Moston", "Denton", "Eccles", "Leigh", "Benchill",
"Prestwich", "Sale", "Kearsley", ])
placenames.extend(["Trafford", "Bolton", "Stockport", "Levenshulme", "Gorton",
"Tameside", "Blackley", "City centre", "Airport", "South Manchester",
"Rochdale", "Chorlton", "Uppermill", "Castleton", "Stalybridge", "Ashton",
"Chadderton", "Bury", "Ancoats", "Whalley Range", "West Yorkshire",
"Fallowfield", "New Moston", "Denton", "Stretford", "Eccles", "Pendleton",
"Leigh", "Altrincham", "Sale", "Prestwich", "Kearsley", "Hulme", "Withington",
"Moss Side", "Milnrow", "outskirt of Manchester City Centre", "Newton Heath",
"Wythenshawe", "Mancunian Way", "M60", "A6", "Droylesden", "M56", "Timperley",
"Higher Ince", "Clayton", "Higher Blackley", "Lowton", "Droylsden",
"Partington", "Cheetham Hill", "Benchill", "Longsight", "Didsbury",
"Westhoughton"])
# Social categories from Yuwei
soccat = ["man", "woman", "men", "women", "youth", "teenager", "elderly",
"patient", "taxi driver", "neighbour", "male", "tenant", "landlord", "child",
"children", "immigrant", "female", "workmen", "boy", "girl", "foster parents",
"next of kin"]
for i in range(100):
    soccat.append("%d-year-old" % i)
    soccat.append("%d-years-old" % i)
# Types of social locations from Yuwei
socloc = ["car park", "park", "pub", "club", "shop", "premises", "bus stop",
"property", "credit card", "supermarket", "garden", "phone box", "theatre",
"toilet", "building site", "Crown court", "hard shoulder", "telephone kiosk",
"hotel", "restaurant", "cafe", "petrol station", "bank", "school",
"university"]
extras =   "placename": placenames, "soccat": soccat, "socloc": socloc  
# Normalise keyword lists
for k, v in extras.iteritems():
    # Remove duplicates
    v = list(set(v))
    # Sort by length
    v.sort(key=lambda x:len(x), reverse=True)
# Add keywords
def add_categories(tweet):
    text = tweet["text"].lower()
    for field, categories in extras.iteritems():
        for cat in categories:
            if cat.lower() in text:
                tweet[field] = cat
                break
    return tweet
# Read all the twits
records = (add_categories(x) for x in simplejson.load(sys.stdin))
simplejson.dump(list(records), sys.stdout)

All these scripts form a nice processing chain: each script takes a list of JSON records, adds some bit and passes it on. In order to see what we have so far, here is a simple script to convert the JSON twits to CSV so they can be viewed in a spreadsheet:

#!/usr/bin/python
# Convert the JSON twits to CSV
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import sys
import csv
rows = ["id", "created_at", "text", "keywords", "placename"]
writer = csv.writer(sys.stdout)
for rec in simplejson.load(sys.stdin):
    rec["keywords"] = " ".join(rec["keywords"])
    rec["placename"] = rec.get("placename", "")
    writer.writerow([rec[row] for row in rows])

At this point we were coming up with lots of questions: "were there more reports on women or men?", "which place had most incidents?", "what were the incidents involving animals?"... Time to bring Xapian into play. This script reads all the JSON tweets and builds a Xapian index with them:

#!/usr/bin/python
# toxapian - Index JSON tweets in Xapian
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import simplejson
import sys
import os, os.path
import xapian
DBNAME = sys.argv[1]
db = xapian.WritableDatabase(DBNAME, xapian.DB_CREATE_OR_OPEN)
stemmer = xapian.Stem("english")
indexer = xapian.TermGenerator()
indexer.set_stemmer(stemmer)
indexer.set_database(db)
data = simplejson.load(sys.stdin)
for rec in data:
    doc = xapian.Document()
    doc.set_data(str(rec["id"]))
    indexer.set_document(doc)
    indexer.index_text_without_positions(rec["text"])
    # Index categories as categories
    if "placename" in rec:
        doc.add_boolean_term("XP" + rec["placename"].lower())
    if "soccat" in rec:
        doc.add_boolean_term("XS" + rec["soccat"].lower())
    if "socloc" in rec:
        doc.add_boolean_term("XL" + rec["socloc"].lower())
    db.add_document(doc)
db.flush()
# Also save the whole dataset so we know where to find it later if we want to
# show the details of an entry
simplejson.dump(data, open(os.path.join(DBNAME, "all.json"), "w"))

And this is a simple command line tool to query to the database:

#!/usr/bin/python
# xgrep - Command line tool to query the GMP24 tweet Xapian database
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import simplejson
import sys
import os, os.path
import xapian
DBNAME = sys.argv[1]
db = xapian.Database(DBNAME)
stem = xapian.Stem("english")
qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
qp.add_boolean_prefix("place", "XP")
qp.add_boolean_prefix("soc", "XS")
qp.add_boolean_prefix("loc", "XL")
query = qp.parse_query(sys.argv[2],
    xapian.QueryParser.FLAG_BOOLEAN  
    xapian.QueryParser.FLAG_LOVEHATE  
    xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE  
    xapian.QueryParser.FLAG_WILDCARD  
    xapian.QueryParser.FLAG_PURE_NOT  
    xapian.QueryParser.FLAG_SPELLING_CORRECTION  
    xapian.QueryParser.FLAG_AUTO_SYNONYMS)
enquire = xapian.Enquire(db)
enquire.set_query(query)
count = 40
matches = enquire.get_mset(0, count)
estimated = matches.get_matches_estimated()
print "%d/%d results" % (matches.size(), estimated)
data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))
for m in matches:
    rec = data[m.document.get_data()]
    print rec["text"]
print "%d/%d results" % (matches.size(), matches.get_matches_estimated())
total = db.get_doccount()
estimated = matches.get_matches_estimated()
print "%d results over %d documents, %d%%" % (estimated, total, estimated * 100 / total)

Neat! Now that we have a proper index that supports all sort of cool things, like stemming, tag clouds, full text search with complex queries, lookup of similar documents, suggest keywords and so on, it was just fair to put together a web service to share it with other people at the event. It helped that I had already written similar code for apt-xapian-index and dde before. Here is the server, quickly built on bottle. The very last line starts the server and it is where you can configure the listening interface and port.

#!/usr/bin/python
# xserve - Make the GMP24 tweet Xapian database available on the web
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import bottle
from bottle import route, post
from cStringIO import StringIO
import cPickle as pickle
import simplejson
import sys
import os, os.path
import xapian
import urllib
import math
bottle.debug(True)
DBNAME = sys.argv[1]
QUERYLOG = os.path.join(DBNAME, "queries.txt")
data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))
prefixes =   "place": "XP", "soc": "XS", "loc": "XL"  
prefix_desc =   "place": "Place name", "soc": "Social category", "loc": "Social location"  
db = xapian.Database(DBNAME)
stem = xapian.Stem("english")
qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
for k, v in prefixes.iteritems():
    qp.add_boolean_prefix(k, v)
def make_query(qstring):
    return qp.parse_query(qstring,
        xapian.QueryParser.FLAG_BOOLEAN  
        xapian.QueryParser.FLAG_LOVEHATE  
        xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE  
        xapian.QueryParser.FLAG_WILDCARD  
        xapian.QueryParser.FLAG_PURE_NOT  
        xapian.QueryParser.FLAG_SPELLING_CORRECTION  
        xapian.QueryParser.FLAG_AUTO_SYNONYMS)
@route("/")
def index():
    query = urllib.unquote_plus(bottle.request.GET.get("q", ""))
    out = StringIO()
    print >>out, '''
<html>
<head>
<title>Query</title>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
<script type="text/javascript">
$(function() 
    $("#queryfield")[0].focus()
 )
</script>
</head>
<body>
<h1>Search</h1>
<form method="POST" action="/query">
Keywords: <input type="text" name="query" value="%s" id="queryfield">
<input type="submit">
<a href="http://xapian.org/docs/queryparser.html">Help</a>
</form>''' % query
    print >>out, '''
<p>Example: "car place:wigan"</p>

<p>Available prefixes:</p>

<ul>
'''
    for pfx in prefixes.keys():
        print >>out, "<li><a href='/catinfo/%s'>%s - %s</a></li>" % (pfx, pfx, prefix_desc[pfx])
    print >>out, '''
</ul>
'''
    oldqueries = []
    if os.path.exists(QUERYLOG):
        total = db.get_doccount()
        fd = open(QUERYLOG, "r")
        while True:
            try:
                q = pickle.load(fd)
            except EOFError:
                break
            oldqueries.append(q)
        fd.close()
        def print_query(q):
            count = q["count"]
            print >>out, "<li><a href='/query?query=%s'>%s (%d/%d %.2f%%)</a></li>" % (urllib.quote_plus(q["q"]), q["q"], count, total, count * 100.0 / total)
        print >>out, "<p>Last 10 queries:</p><ul>"
        for q in oldqueries[:-10:-1]:
            print_query(q)
        print >>out, "</ul>"
        # Remove duplicates
        oldqueries = dict(((x["q"], x) for x in oldqueries)).values()
        print >>out, "<table>"
        print >>out, "<tr><th>10 queries with most results</th><th>10 queries with least results</th></tr>"
        print >>out, "<tr><td>"
        print >>out, "<ul>"
        oldqueries.sort(key=lambda x:x["count"], reverse=True)
        for q in oldqueries[:10]:
            print_query(q)
        print >>out, "</ul>"
        print >>out, "</td><td>"
        print >>out, "<ul>"
        nonempty = [x for x in oldqueries if x["count"] > 0]
        nonempty.sort(key=lambda x:x["count"])
        for q in nonempty[:10]:
            print_query(q)
        print >>out, "</ul>"
        print >>out, "</td></tr>"
        print >>out, "</table>"
    print >>out, '''
</body>
</html>'''
    return out.getvalue()
@route("/query")
@route("/query/")
@post("/query")
@post("/query/")
def query():
    query = bottle.request.POST.get("query", bottle.request.GET.get("query", ""))
    enquire = xapian.Enquire(db)
    enquire.set_query(make_query(query))
    count = 40
    matches = enquire.get_mset(0, count)
    estimated = matches.get_matches_estimated()
    total = db.get_doccount()
    out = StringIO()
    print >>out, '''
<html>
<head><title>Results</title></head>
<body>
<h1>Results for "<b>%s</b>"</h1>
''' % query
    if estimated == 0:
        print >>out, "No results found."
    else:
        # Give as results the first 30 documents; also use them as the key
        # ones to use to compute relevant terms
        rset = xapian.RSet()
        for m in enquire.get_mset(0, 30):
            rset.add_document(m.document.get_docid())
        # Compute the tag cloud
        class NonTagFilter(xapian.ExpandDecider):
            def __call__(self, term):
                return not term[0].isupper() and not term[0].isdigit()
        cloud = []
        maxscore = None
        for res in enquire.get_eset(40, rset, NonTagFilter()):
            # Normalise the score in the interval [0, 1]
            weight = math.log(res.weight)
            if maxscore == None: maxscore = weight
            tag = res.term
            cloud.append([tag, float(weight) / maxscore])
        max_weight = cloud[0][1]
        min_weight = cloud[-1][1]
        cloud.sort(key=lambda x:x[0])
        def mklink(query, term):
            return "/query?query=%s" % urllib.quote_plus(query + " and " + term)
        print >>out, "<h2>Tag cloud</h2>"
        print >>out, "<blockquote>"
        for term, weight in cloud:
            size = 100 + 100.0 * (weight - min_weight) / (max_weight - min_weight)
            print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(query, term), size, term)
        print >>out, "</blockquote>"
        print >>out, "<h2>Results</h2>"
        print >>out, "<p><a href='/'>Search again</a></p>"
        print >>out, "<p>%d results over %d documents, %.2f%%</p>" % (estimated, total, estimated * 100.0 / total)
        print >>out, "<p>%d/%d results</p>" % (matches.size(), estimated)
        print >>out, "<ul>"
        for m in matches:
            rec = data[m.document.get_data()]
            print >>out, "<li><a href='/item/%s'>%s</a></li>" % (rec["id"], rec["text"])
        print >>out, "</ul>"
        fd = open(QUERYLOG, "a")
        qinfo = dict(q=query, count=estimated)
        pickle.dump(qinfo, fd)
        fd.close()
    print >>out, '''
<a href="/">Search again</a>

</body>
</html>'''
    return out.getvalue()
@route("/item/:id")
@route("/item/:id/")
def show(id):
    rec = data[id]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Result %s</title></head>
<body>
<h1>Raw JSON record for twit %s</h1>
<pre>''' % (rec["id"], rec["id"])
    print >>out, simplejson.dumps(rec, indent=" ")
    print >>out, '''
</pre>
</body>
</html>'''
    return out.getvalue()
@route("/catinfo/:name")
@route("/catinfo/:name/")
def catinfo(name):
    prefix = prefixes[name]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Values for %s</title></head>
<body>
''' % name
    terms = [(x.term[len(prefix):], db.get_termfreq(x.term)) for x in db.allterms(prefix)]
    terms.sort(key=lambda x:x[1], reverse=True)
    freq_min = terms[0][1]
    freq_max = terms[-1][1]
    def mklink(name, term):
        return "/query?query=%s" % urllib.quote_plus(name + ":" + term)
    # Build tag cloud
    print >>out, "<h1>Tag cloud</h1>"
    print >>out, "<blockquote>"
    for term, freq in sorted(terms[:20], key=lambda x:x[0]):
        size = 100 + 100.0 * (freq - freq_min) / (freq_max - freq_min)
        print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(name, term), size, term)
    print >>out, "</blockquote>"
    print >>out, "<h1>All terms</h1>"
    print >>out, "<table>"
    print >>out, "<tr><th>Occurrences</th><th>Name</th></tr>"
    for term, freq in terms:
        print >>out, "<tr><td>%d</td><td><a href='/query?query=%s'>%s</a></td></tr>" % (freq, urllib.quote_plus(name + ":" + term), term)
    print >>out, "</table>"
    print >>out, '''
</body>
</html>'''
    return out.getvalue()
# Change here for bind host and port
bottle.run(host="0.0.0.0", port=8024)

...and then we presented our work and ended up winning the contest. This was the story of how we wrote this set of award winning code.

Warning: rant ahead Someone decided that doing a Windows-specific protocol would be a good idea. Whoever this person is, I hope he rots in hell. But that's not what this post is about. Someone else decided that doing a free software implementation of that Windows-specific protocol would be a good idea. Hence samba was born. And all was good. For the next few years, Samba kept implementing more and more features of that particular Windows-specific protocol, which kept getting changed and renamed over the years. Originally, samba could just export files, and you could specify whether users could or could not write to your files. Perhaps it already had passwords at that point not sure. Later, usernames were added, and the ability to join a Windows domain, and then another type of Windows domain, and eventually a number of other differently deficient difficulties. As these things were implemented, the ways of configuring samba to properly do all these things was made more and more complex, to the point where it now no longer seems possible to install samba without major repercussions on the 'plain' Linux installation. A bit like having an apache requiring you to add apache-specific usernames through an apache-specific NSS module (winbind anyone?) just so you could export ACLs. Oh, and forget about retaining your existing usernames. Now I'm not sure whether this is related to a) me having tried winbind at some point in the past, deciding that I didn't like the effects it had, and removing it again (with as a result that it left some traces on the system), b) the system being joined to an ADS domain which might have some special requirements, or c) this particular system having a bit of a weird history that might have fucked up some of the internal state files of samba; but at any rate, it sucks. It seems to me that somehow, over the years, samba has lost the ability to just function as a minor something that just happens to be one service on a machine much like apache on my laptop and now requires one to do a complete overhaul of the entire system. That's just stupid; I am not going to migrate users to their winbind counterparts, thankyouverymuch. Stupid ******** Of course, though unlikely, it's not impossible that I've missed something. But at any rate, even if that is the case, the hoops one needs to jump through to get to the above situations are far too numerous and complex.

I love titles with plenty of buzz-words :) After a bit of google-ing I finally found the right combination of tools that fit my needs. Luckily they are all packaged for debian and well documented. Auto-configure your network is an old topic. There are plenty of articles and howtos. Moreover the gnome NetworkManager already solves many problems alone. But since I don't use NetworkManager I tried to find a solution that works well with wicd. The "right" combination for me so far is to use guessnet and ifplugd to handle my wired connection and use Wicd for the wireless network. In theory, you can go for a fully automated solution using guessnet and wpa_supplicant, but I find much better to hide all the complexity of wpa_supplicant and just use a simple - and unobtrusive - UI to handle my wireless. First thing is to convince Wicd to leave alone my wired connection. This is pretty easy since Wicd wont touch your wired interface if it finds an entry in /etc/network/interfaces for it. Moreover to selecting always switch to a wired connection when available will make Wicd to leave alone you routing tables even if there is a wireless connection available that is marked as automatically connect to this network. Second you need to install guessnet and ifplugd I've noticed that the default settings for ifplugd work pretty well except for the -q option that sometimes lives my interface configured even if I've been disconnected for quite a while. Using ifplugd default, I re-obtain an address every time the cable is plugged-in, maybe causing a small delay in some occasion, but I prefer waiting that de-configuring the interface by hand. Guessnet needs a tiny bit of work. The debian package has already nice defaults that you can copy directly in your /etc/network/interfaces. These will make sure you avoid long timeouts and you leave the interface configured even if no cable is connected. Now you have to add one entry per network that you want to configure: For example to use dhcp on your home network (and of course run any if-up / if-down script as you like). Or : to set a static address at work. Guessnet is nice and well integrated with ifupdown. Very flexible and documented. I had however the impression that from time to time the documentation I found on the net was not in sync with the latest release... I've also noticed that a component of guessnet is not in the package anymore. Guessnet-scan is a small program that will output a stanza to be used with ifupdown based on values sniffed on the wire. This is a kinda of extreme / hackish way to get connected but i reckon a nice tool to have. Hopefully it will make it back in the package sometimes soon... ps: first post on debian planet !

I wrote the following as a foreword for the great German book Open Source Projektmanagement that Michael Prokop wrote. Unfortunately the foreword didn't manage it in time into the book because of various circumstances. As I'm not a person to throw away already written material, here is it for your reading pleasure. Maybe it makes you consider obtaining a copy of the German language book for yourself!

The Whole is more than the Sum of the Parts When I first met Mika at a Linux conference some years ago I quickly noticed that he had potential. The way he asked questions and tackled issues did impress me. We did meet again at various events, joined forces in various projects and knew that the result would be good. Back in the year 2004 the only real live CD was Knoppix. It always was too sluggish for me because I never really liked KDE and also OpenOffice.org was too bloated for me. I was working since a year at an Internet provider, felt comfortable in the shell and was missing the tools on Knoppix which I used daily. Often enough the frustration with the status quo starts the best projects. And so I had the idea spinning around to start a sysadmin live CD. There was just one problem here: Exactly at the same time Mika had already started such a project. And I knew one thing: it would had been a lost race to compete with him in creating the better live CD. So I joined his team and helped with the best of my knowledge. The name choice alone showed that he was the right person for the job: Grml an expression of the frustration that even he felt which spoke directly to the heart of so many. But it wasn't just the chosen name that showed that Mika was the right person for the job of the project leader. It's always the sum of the parts, no matter how minor they might look. He quickly managed a first release and in light of that also created an event which was fitting for the release of the first version: the OS04 an Open Source event which managed to get Jon Maddog Hall as keynote speaker. Mika managed through his welcoming way to lure more people into helping out. The team grew over the years, further regular releases increased the fan base, not only through the creative release names which are surely one of the parts that helped create the success. He always was open to suggestions for new tools that helped to extend the project. The lived openness did lead among other things to the case that other live CDs started to use the grml build system. JUXlala 2.0 (the system for preschool kids) is just one example. Why do I write primarily about Grml, one could get subliminal advertising probably cheaper? I do it because Grml is an extremely good example of the experience revealed in this book. And even though every project is different, it is still exactly the sum of the parts that leads to the success of a project. And exactly these parts are covered in this book and can be considered for the project at hand and get implemented accordingly. Good luck whether in large or small.
Rhonda

/debian permanent link Comments: 0

When I took over the developement of xen-tools, I looked around for an appropriate git hosting. I especially had a look at GitHub and Gitorious. If you just regard the features, GitHub is definitely more targetted on single developers and Gitorious more towards projects: At GitHub, every repository has its URL under the URL of a user page which makes it nearly impossible to have user independent, official repositories for projects which have more than one official developer. At Gitorious, every hosted repository needs to belong to a project, even if it s only a published configuration. But a project can have more than one git repository. You only seem to be able to have personal repositories if you clone some existing Gitorious repository. So from a feature point of view, the xen-tools git repositories fit way better to Gitorious hosting while the git repositories with zshrc, conkerorrc and desktop configuration files defintely have a more fitting addresses on GitHub, in my case http://github.com/xtaran/$repository. On Gitorious, they are now together under a project called Axel s configuration files at http://gitorious.org/abe which contains git repositories of my on grml s .zshrc based .zshrc, my configuration for Conkeror and all the files necessary for my ratpoison/xmobar based netbook/laptop desktop. I though feel a little bad for giving the project the very short slug name abe instead of abe-config (as I did initially) since abe is IMHO not a proper project name for my configuration files and possibly other projects would have a more reasonable claim for that project name on Gitorious. But that way it suites more its purpose: Gather some of my git repositories which don t belong to a proper project. But there s another important point when comparing Gitorious and GitHub: Free Software needs free tools as Benjamin Mako Hill posted recently on Planet Debian. Despite my (probably well known) distrust against Google and therefore also Google Code, and despite knowing the history of SourceForge becoming non-free, I was not that much aware that GitHub s software is only partially open source and therefore also not free software while Gitorious is both as it s licensed under the GNU Affero General Public License (like StatusNet/identi.ca for example) which is basically GPLv3, but its ideas applied to hosted web applications scenario (where the GPL itself doesn t grasp), too. Initially I just had the xen-tools git repositories on Gitorious and all my small one-repository projects as copies of the repositories on my own git server on GitHub to get some more publicity for them and allow social cloning . After reading Mako s article, I decided to at least have repository clones on Gitorious of all repositories I mirror at GitHub, too. That way I force nobody to use the non-free tools on GitHub for social cloning one of my git repositories. And of course I have copies of my code somewhere on the net as backup. Or to say it with Linus Torvalds (slightly updated) words: Only wimps use tape backup: real men just upload their important stuff on git, and let the rest of the world clone it. ;-) But isn t that tedious to always push your code to three repositories? No, it isn t. I just push my code to git.noone.org where I have configured the according remotes and the following post-receive hook:

#!/bin/sh
                                                                     
read oldrev newrev refname
git push gitorious $ refname:t 
git push github $ refname:t 

The only other thing necessary is to use ssh-agent and SSH agent forwarding to at least the host you re pushing to.

A little while back, as part of a discussion on debian-devel, I spent some time bad-mouthing pkg-config, and then later realized (with the help of an IRC conversation) that I was doing so unfairly. So as penance here is some documentation for how to add a pkg-config configuration file to a typical Automake project. This information is also in my build system coding style and will be maintained there (so if you see this some time after I wrote it, check that page for more up-to-date information). pkg-config is a tool to provide the flags required to compile or link against a shared library. It's a replacement for complex Autoconf probes and for all of the incompatible and occasionally buggy *-config scripts. To install a pkg-config configuration file, provide a template pkg-config configuration in a file named after the library but with a .pc.in extension. Here is an example:

    prefix=@prefix@
    exec_prefix=@exec_prefix@
    includedir=@includedir@
    libdir=@libdir@
    Name: <library>
    Description: <short description>
    URL: <upstream url>
    Version: @PACKAGE_VERSION@
    Cflags: -I$ includedir 
    Libs: -L$ libdir  -l<library>
    Libs.private: @KRB5_LDFLAGS@ @KRB5_LIBS@

Replace <library> with the official name of the library (which isn't necessarily the library name with lib removed). This is only informational, may contain spaces, and isn't used to locate the file. Its filename is used for that. Replace <short description> with a short description of the purpose of the library. Often the short description of the package from the start of the README file works well here. Replace <upstream url> with the URL of the package's distribution page. Normally, all Cflags needs to contain is what's indicated above, which lets the compiler find the header files if installed in a non-standard location. For Libs, note that pkg-config distinguishes between the shared libraries that must be linked against on a system with proper transitive dependency support and the libraries that must be linked for a static build or on a system with poor support for shared libraries. For a well-designed shared library, one should rarely need anything in Libs other than what's listed above. Libs.private, on the other hand, needs all the LDFLAGS and LIBS of everything the library itself links with external to this package. In this example, the shared library is linked with the Kerberos libraries and therefore needs the Kerberos LDFLAGS and LIBS; modify as needed for your example. If the dependencies of the shared library also use pkg-config, just list the other pkg-config file names in Requires or Requires.private instead of using Libs or Libs.private for them. Finally, add the Automake code to build and install the pkg-config configuration file. It should look like this:

    pkgconfigdir = $(libdir)/pkgconfig
    nodist_pkgconfig_DATA = lib/libfoo.pc
    lib/libfoo.pc: $(srcdir)/lib/libfoo.pc.in
            sed -e 's![@]prefix[@]!$(prefix)!g' \
                -e 's![@]exec_prefix[@]!$(exec_prefix)!g' \
                -e 's![@]includedir[@]!$(includedir)!g' \
                -e 's![@]libdir[@]!$(libdir)!g' \
                -e 's![@]PACKAGE_VERSION[@]!$(PACKAGE_VERSION)!g' \
                -e 's![@]KRB5_LDFLAGS[@]!$(KRB5_LDFLAGS)!g' \
                -e 's![@]KRB5_LIBS[@]!$(KRB5_LIBS)!g' \
                $(srcdir)/lib/libfoo.pc.in > $@

Replace libfoo with the actual name of the library and change its path in the source package as needed. Note the pattern above and use it for any Autoconf substitution variables that you need to replace in the pkg-config template. The above example adds KRB5_LDFLAGS and KRB5_LIBS to the variables; you may have other ones you need, particularly to construct Libs.private. The above installs the pkg-config configuration file under $(libdir)/pkgconfig. $(datadir)/pkgconfig is appropriate instead if one is certain that the flags required to link with this library will never vary by architecture, but that's not the case for most libraries due to Libs.private. Note that the Automake documentation recommends using comma as the sed delimiter for doing this sort of trick, but LDFLAGS frequently contains a comma since it's the separator for parts of a linker-specific option. I use an exclamation point instead.