I've recently found myself explaining polkit (formerly PolicyKit) to one of Collabora's clients, and thought that blogging about the same topic might be useful for other people who are confused by it; so, here is why udisks2 and polkit are the way they are. As always, opinions in this blog are my own, not Collabora's. Privileged actions Broadly, there are two ways a process can do something: it can do it directly (i.e. ask the kernel directly), or it can use inter-process communication to ask a service to do that operation on its behalf. If it does it directly, the components that say whether it can succeed are the Linux kernel's normal permissions checks (DAC), and if configured, AppArmor, SELinux or a similar MAC layer. All very simple so far. Unfortunately, the kernel's relatively coarse-grained checks are not sufficient to express the sorts of policies that exist on a desktop/laptop/mobile system. My favourite example for this sort of thing is mounting filesystems. If I plug in a USB stick with a FAT filesystem, it's reasonable to expect my chosen user interface to either mount it automatically, or let me press a button to mount it. Similarly, to avoid data loss, I should be able to unmount it when I'm finished with it. However, mounting and unmounting a USB stick is fundamentally the same system call as mounting and unmounting any other filesystem - and if ordinary users can do arbitrary mount system calls, they can cause all sorts of chaos, for instance by mounting a filesystem that contains setuid executables (privilege escalation), or umounting a critical OS filesystem like /usr (denial of service). Something needs to arbitrate: you can mount filesystems, but only under certain conditions . The kernel developer motto for this sort of thing is mechanism, not policy : they are very keen to avoid encoding particular environments' policies (the sort of thing you could think of as business rules ) in the kernel, because that makes it non-generic and hard to maintain. As a result, direct mount/unmount actions are only allowed by privileged processes, and it's up to user-space processes to arrange for a privileged process to make the desired mount syscall. Here are some other privileged actions which laptop/desktop users can reasonably expect to just work , with or without requiring a sysadmin-like (root-equivalent) user:

• reconfiguring networking (privileged because, in the general case, it's an availability and potentially integrity issue)
• installing, upgrading or removing packages (privileged because, in the general case, it can result in arbitrary root code execution)
• suspending or shutting down the system (privileged because you wouldn't want random people doing this on your server, but should normally be allowed on e.g. laptops for people with physical access, because they could just disconnect the power anyway)

In environments that use a MAC framework like AppArmor, actions that would normally be allowed can become privileged: for instance, in a framework for sandboxed applications, most apps shouldn't be allowed to record audio. This prevents carrying out these actions directly, again resulting in the only way to achieve them being to ask a service to carry out the action. Ask a system service to do it On to the next design, then: I can submit a request to a privileged process, which does some checks to make sure I'm not trying to break the system (or alternatively, that I have enough sysadmin rights that I'm allowed to break the system if I want to), and then does the privileged action for me. You might think I'm about to start discussing D-Bus and daemons, but actually, a prominent earlier implementation of this was mount(8), which is normally setuid root:

% ls -l /bin/mount
-rwsr-xr-x 1 root root 40000 May 22 11:37 /bin/mount

If you look at it from an odd angle, this is inter-process communication across a privilege boundary: I run the setuid executable, creating a process. Because the executable has the setuid bit set, the kernel makes the process highly privileged: its effective uid is root, and it has all the necessary capabilities to mount filesystems. I submit the request by passing it in the command-line arguments. mount does some checks - specifically, it looks in /etc/fstab to see whether the filesystem I'm trying to mount has the user or users flag - then carries out the mount system call. There are a few obvious problems with this:

• When machines had a static set of hardware devices (and a sysadmin who knew how to configure them), it might have made sense to list them all in /etc/fstab; but this is not a useful solution if you can plug in any number of USB drives, or if you are a non-expert user with Linux on your laptop. The decision ought to be based on general attributes of devices, such as is removable? , and on the role of the machine.
• Setuid executables are alarmingly easy to get wrong so it is not necessarily wise to assume that mount(8) is safe to be setuid.
• One fact that a reasonable security policy might include is users who are logged in remotely should have less control over physically present devices than those who are physically present - but that sort of thing can't be checked by mount(8) without specifically teaching the mount binary about it.

Ask a system service to do it, via D-Bus or other IPC To avoid the issues of setuid, we could use inter-process communication in the traditional sense: run a privileged daemon (on boot or on-demand), make it listen for requests, and use the IPC channel as our privilege boundary. udisks2 is one such privileged daemon, which uses D-Bus as its IPC channel. D-Bus is a commonly-used inter-process system; one of its intended/designed uses is to let user processes and system services communicate, especially this sort of communication between a privileged daemon and its less-privileged clients. People sometimes criticize D-Bus as not doing anything you couldn't do yourself with some AF_UNIX sockets. Well, no, of course it doesn't - the important bit of the reference implementation and the various interoperable reimplementations consists of a daemon and some AF_UNIX sockets, and the rest is a simple matter of programming. However, it's sufficient for most uses in its problem space, and is usually better than inventing your own. The advantage of D-Bus over doing your own thing is precisely that you are not doing your own thing: good IPC design is hard, and D-Bus makes some structural decisions so that fewer application authors have to think about them. For instance, it has a central hub daemon (the dbus-daemon, or message bus ) so that n communicating applications don't need O(n ) sockets; it uses the dbus-daemon to provide a total message ordering so you don't have to think about message reordering; it has a distributed naming model (which can also be used as a distributed mutex) so you don't have to design that; it has a serialization format and a type system so you don't have to design one of those; it has a framework for activating" run-on-demand daemons so they don't have to use resources initially, implemented using a setuid helper and/or systemd; and so on. If you have religious objections to D-Bus, you can mentally replace D-Bus with AF_UNIX or something and most of this article will still be true. Is this OK? In either case - exec'ing a privileged helper, or submitting a request to a privileged daemon via IPC - the privileged process has two questions that it needs to answer before it does its work:

• what am I being asked to do?
• should I do it?

It needs to make some sort of decision on the latter based on the information available to it. However, before we even get there, there is another layer:

• did the request get there at all?

In the setuid model, there is a simple security check that you can apply: you can make /bin/mount only executable by a particular group, or only executable by certain AppArmor profiles, or similar. That works up to a point, but cannot distinguish between physically-present and not-physically-present users, or other facts that might be interesting to your local security policy. Similarly, in the IPC model, you can make certain communication channels impossible, for instance by using dbus-daemon's ability to decide which messages to deliver, or AF_UNIX sockets' filesystem permissions, or a MAC framework like AppArmor. Both of these are quite coarse-grained checks which don't really understand the finer details of what is going on. If the answer to "is this safe? is something of the form maybe, it depends on... , then they can't do the right thing: they must either let it through and let the domain-specific privileged process do the check, or deny it and lose potentially useful functionality. For instance, in an AppArmor environment, some applications have absolutely no legitimate reason to talk to udisks2, so the AppArmor policy can just block it altogether. However, once again, this is a coarse-grained check: the kernel has mechanism, not policy, and it doesn't know what the service does or why. If the application does need to be able to talk to the service at all, then finer-grained access control (obeying some, but not all, requests) has to be the service's job. dbus-daemon does have the ability to match messages in a relatively fine-grained way, based on the object path, interface and member in the message, as well as the routing information that it uses itself (i.e. the source and destination). However, it is not clear that this makes a great deal of sense conceptually: these are facts about the mechanics of the IPC, not facts about the domain-specific request (because the mechanics of the IPC are all that dbus-daemon understands). For instance, taking the udisks2 example again, dbus-daemon can't distinguish between an attempt to adjust mount options for a USB stick (probably fine) and an attempt to adjust mount options for /usr (not good). To have a domain-specific security policy, we need a domain-specific component, for instance udisks2, to get involved. Unlike dbus-daemon, udisks2 knows that not all disks are equal, knows which categories make sense to distinguish, and can identify which categories a particular disk is in. So udisks2 can make a more informed decision. So, a naive approach might be to write a function in udisks2 that looks something like this pseudocode:

may_i_mount_this_disk (user, disk, mount options)   boolean
 
    if (user is root   user is root-equivalent)
        return true;
    if (disk is not removable)
       return false;
    if (mount options are scary)
       return false;
    if (user is in  manipulate non-local disks  group)
        return true;
    if (user is not logged-in locally)
       return false;
    # https://en.wikipedia.org/wiki/Multiseat_configuration
    if (user is not logged-in on the same seat where the disk is
            plugged in)
       return false;
    return true;
 

Delegating the security policy to something central The pseudocode security policy outlined above is reasonably complicated already, and doesn't necessarily cover everything that you might want to consider. Meanwhile, not every system is the same. A general-purpose Linux distribution like Debian might run on server/mainframe systems with only remote users, personal laptops/desktops with one root-equivalent user, locked-down corporate laptops/desktops, mobile devices and so on; these systems should not necessarily all have the same security policy. Another interesting factor is that for some privileged operations, you might want to carry out interactive authorization: ask the requesting user to confirm that the action (which might have come from a background process) should take place (like Windows' UAC), or to prove that the person currently at the keyboard is the same as the person who logged in by giving their password (like sudo). We could in principle write code for all of this in udisks2, and in NetworkManager, and in systemd, ... - but that clearly doesn't scale, particularly if you want the security policy to be configurable. Enter polkit (formerly PolicyKit), a system service for applying security policies to actions. The way polkit works is that the application does its domain-specific analysis of the request - in the case of udisks2, whether the device to be mounted is removable, whether the mount options are reasonable, etc. - and converts it into an action. The action gives polkit a way to distinguish between things that are conceptually different, without needing to know the specifics. For instance, udisks2 currently divides up filesystem-mounting into org.freedesktop.udisks2.filesystem-mount, org.freedesktop.udisks2.filesystem-mount-fstab, org.freedesktop.udisks2.filesystem-mount-system and org.freedesktop.udisks2.filesystem-mount-other-seat. The application also finds the identity of the user making the request. Next, the application sends the action, the identity of the requesting user, and any other interesting facts to polkit. As currently implemented, polkit is a D-Bus service, so this is an IPC request via D-Bus. polkit consults its database of policies in order to choose one of several results:

• yes, allow it
• no, do not allow it
• ask the user to either authenticate as themselves or as a privileged (sysadmin) user to allow it, or cancel authentication to not allow it
• ask the user to authenticate the first time, but if they do, remember that for a while and don't ask again

So how does polkit decide this? The first thing is that it reads the machine-readable description of the actions, in /usr/share/polkit-1/actions, which specifies a default policy. Next, it evaluates a local security policy to see what that says. In the current version of polkit, the local security policy is configured by writing JavaScript in /etc/polkit-1/rules.d (local policy) and /usr/share/polkit-1/rules.d (OS-vendor defaults). In older versions such as the one currently shipped in Debian unstable, there was a plugin architecture; but in practice nobody wrote plugins for it, and instead everyone used the example local authority shipped with polkit, which was configured via files in /etc/polkit-1/localauthority and /etc/polkit-1/localauthority.d. These policies can take into account useful facts like:

• what is the action we're talking about?
• is the user logged-in locally? are they active, i.e. they are not just on a non-current virtual console?
• is the user in particular groups?

For instance, gnome-control-center on Debian installs this snippet:

polkit.addRule(function(action, subject)  
        if ((action.id == "org.freedesktop.locale1.set-locale"  
             action.id == "org.freedesktop.locale1.set-keyboard"  
             action.id == "org.freedesktop.hostname1.set-static-hostname"  
             action.id == "org.freedesktop.hostname1.set-hostname"  
             action.id == "org.gnome.controlcenter.datetime.configure") &&
            subject.local &&
            subject.active &&
            subject.isInGroup ("sudo"))  
                    return polkit.Result.YES;
             
 );

which is reasonably close to being pseudocode for active local users in the sudo group may set the system locale, keyboard layout, hostname and time, without needing to authenticate . A system administrator could of course override that by dropping a higher-priority policy for some or all of these actions into /etc/polkit-1/rules.d. Summary

• Kernel-based permission checks are not sufficiently fine-grained to be able to express some quite reasonable security policies
• Fine-grained access control needs domain-specific understanding
• The kernel doesn't have that information (and neither does dbus-daemon)
• The privileged service that does the domain-specific thing can provide the domain-specific understanding to turn the request into an action
• polkit evaluates a configurable policy to determine whether privileged services should carry out requested actions

It was in my mind to open a new series of articles with topic selfhosting , because I really believe in free software based network services and since long time I want to plug a machine 24 7 at home to host my blog, microblog, MediaGoblin, XMPP server, mail, and, in conclusion, all the services that now I trust to very kind third parties that run them with free software, but I know I could run myself (and offer them to my family and friends). Last September I bought the domain larjona.net (curious, they say buy but it s a rent, for 1,2,3 years never yours. Pending another post about my adventures with the domain name, dynamic DNS, and SSL certs!) and I bought an HP Microserver G7 N54L, with 2 GB RAM. It had a 250GB SATA harddisk and I bought 2 more SATA harddisks, 1 TB each, to setup a RAID 1 (mirror). Total cost (with keyboard and mouse), 300 . A friend gave me a TFT monitor that was too old for him (1024 768) but it serves me well, (it s a server, no graphical interface, and I will connect remotely most of the times). Installing Debian stable (wheezy) I decided to install Debian stable. Jessie was not frozen yet, and since it was my first non-LAMP server install, I wanted to make sure that errors and problems would be my errors, not issues of the non-released-yet distro. I thought to install YunoHost or some other distro prepared for selfhosting, but I ve never tried them, and I have not much free time, so I decided to stick on Debian, my beloved distro, because it s the one that I know best and I m part of its awesome community. And maybe I could contribute back some bug reports or documentation. I wanted to try a crypto setup (just for fun, just for learn, for its benefits, and to be one more freecrypto-tester in the world) so after reading a bit: https://wiki.debian.org/DebianInstaller/SataRaid
https://wiki.archlinux.org/index.php/disk_encryption
http://madduck.net/docs/cryptdisk/
http://linuxgazette.net/140/kapil.html
http://smcv.pseudorandom.co.uk/2008/09/cryptroot/
http://www.linuxquestions.org/questions/linux-security-4/lvm-before-and-after-encryption-871379/ and some other pages, and try some different things, this is the setup that I managed to configure:

• A rescue system with /boot and / partitions, both in the 250 GB disk.
• A RAID 1 system of the two 1TB disks, setup in the BIOS of the machine (so the motherboard handles the RAID and the OS is focused in other things).
• Inside the Debian installer, I went to manual partition, then I put my /boot in the 250GB disk (yes, a 2nd /boot there), and then selected the 1TB disk (since the RAID was already made, it appeared a single 1TB disk) as physical device to be encrypted.
• After that, still in the Debian installer, I setup LVM there: configured a volume group, then, two volumes, one for / and the other one for swap.
• Then I saved the changes and go on installing my system.

Everything went well. Yay! Some doubts and one problem Everything went quite well except some doubts:

• I m still not sure if this BIOS RAID ( Fake RAID ) is better than a software RAID or not. I suppose it s better since I delegate in the motherboard to do it, and leave the OS to care about other things (transcode my videos yeah!). But I don t know how to measure performance and which metrics and results should I expect. The disks (cheap disks) are a bit noisy (just a bit! or maybe it s the fan that it s very quiet! poor Laura, never saw/had a luxury machine like this one :)
• I had to install firmware-linux-nonfree in order to properly use the graphics card (Mobility Radeon HD 4225/4250). I have no graphical environment there, only a console, so I was not sure if installing the firmware or not (without the firmware, the letters of the console were bigger, but I just don t mind since I most of the time I log in remotely from my laptop). Then, two questions arised to my ignorant mind:
  1. Do I need the driver for better performance (aka is the graphics card used for rendering/transcoding/showing images and videos in my MediaGoblin site or just when it s needed to display them in local (and subsequently, never)?
  2. If I leave the system like that, and forget about the firmware warning at boot time, can the hardware be damaged by the default (free) driver? (for example, due to fan controlling malfunction or something like that).

After talking about this issues with friends (and in debian-women IRC channel), I decided to install the non-free driver, just in case, with the same reasoning as with the RAID: let the card do the job, so the CPU can care about other things. Again, I notice that learning a bit about benchmarking (and having some time to do some tests) would be nice And now, the problem:

• I noticed something strange in my setup. Sometimes, after a system reboot, cryptsetup was not accepting the password to unlock the encrypted disk. And believe me, I was typing it carefully. But when I completely shutdown the computer, unplug the cable, replug the cable, and start again, the password was accepted. The keyboard is USB and this machine does not accept other connection for the keyboard. The keyboard configuration, language and so, was all correct. No Non-ASCII symbols in my password. My password would need to press the same keys in a Spanish and an English keyboard.
• I thought that maybe something in my RAID was failing. I tried to disconnect one of the disks, and see if (1) the bug was solved (no) and (2) the RAID was working (yes). I made the same with the other disk. I was happy that I could reconstruct my RAID when plugging the disk again. But still I had the problem of the password.

I left this problem apart and go on installing the software. I would think later what to do. Installing MediaGoblin The most urgent selfhosting service, for me, was GNU MediaGoblin, because I wanted to show my server to my family in Christmas, and upload the pictures of the babies and kids of the family. And it s a project where I contribute translations and I am a big fan, so I would be very proud of hosting my own instance. I followed the documentation to setup 2 instances of GNU MediaGoblin 0.7 (the stable release in the moment), with their corresponding PostgreSQL databases. Why two instances? Well, I want an instance to host and show my videos, images, and replicate videos that I like, and a private one for sharing photos and videos with my family. MediaGoblin has no privacy settings yet, so I installed separate instances, and the private one I put it in a different port, with a self-signed SSL cert, and enabled http-authorization in Nginx, so only authorized Linux users of my machine can accesss the website. Installing MediaGoblin was easier than what I thought. I only had some small doubts about the documentation, and they were solved in the IRC channel. You can access, for example, my user profile in my public instance, and see some different files that I already uploaded. I m very happy!! Face to face with the bug, again I had to solve the problem of the password not accepted in reboots. I began to think that it could be a bug in cryptsetup. Should I upgrade the package to the version in wheezy-backports? Jessie was almost frozen, maybe it was time to upgrade the whole system, to see if the problem was solved (and to see how my MediaGoblin was working or not in Jessie. It should work, it s almost packaged! But who knows). And if it didn t work, maybe it was time to file a bug So I upgraded my system to Debian Jessie. And after upgrade, the system didn t boot. But that s the story of another blog post (that I still need to finish to write don t worry, it has happy end, as you could see accessing my Mediagoblin site!). Comments? You can comment in this pump.io thread.
Filed under: My experiences and opinion Tagged: Debian, encryption, English, libre software, MediaGoblin, Moving into free software, N54L, selfhosting, sysadmin

---

By now, it is well known that Debian Jessie will not be using sysvinit as its boot system by default. But how can one keep using sysvinit in Jessie? It is fairly easy, and here are a few recipes, courtesy of Erich Schubert and Simon McVittie. If you already are using Wheezy and want to upgrade to Jessie and keep sysvinit as your boot system, create a file /etc/apt/preferences.d/use-sysvinit with this content before you upgrade:

Package: systemd-sysv
Pin: release o=Debian
Pin-Priority: -1

This file content will tell apt and aptitude to not consider installing systemd-sysv as part of any installation and upgrade solution when resolving dependencies, and thus tell it to avoid systemd as a default boot system. The end result should be that the upgraded system keep using sysvinit. If you are installing Jessie for the first time, there is no way to get sysvinit installed by default (debootstrap used by debian-installer have no option for this), but one can tell the installer to switch to sysvinit before the first boot. Either by using a kernel argument to the installer, or by adding a line to the preseed file used. First, the kernel command line argument:

preseed/late_command="in-target apt-get install --purge -y sysvinit-core"

Next, the line to use in a preseed file:

d-i preseed/late_command string in-target apt-get install -y sysvinit-core

One can of course also do this after the first boot by installing the sysvinit-core package. I recommend only using sysvinit if you really need it, as the sysvinit boot sequence in Debian have several hardware specific bugs on Linux caused by the fact that it is unpredictable when hardware devices show up during boot. But on the other hand, the new default boot system still have a few rough edges I hope will be fixed before Jessie is released. Update 2014-11-26: Inspired by a blog post by Torsten Glaser, added --purge to the preseed line.

Debian's latest round of angry mailing list threads have been about some combination of init systems, future direction and project governance. The details aren't particularly important here, and pretty much everything worthwhile in favour of or against each position has already been said several times, but I think this bit is important enough that it bears repeating: the reason I voted "we didn't need this General Resolution" ahead of the other options is that I hope we can continue to use our normal technical and decision-making processes to make Debian 8 the best possible OS distribution for everyone. That includes people who like systemd, people who dislike systemd, people who don't care either way and just want the OS to work, and everyone in between those extremes. I think that works best when we do things, and least well when a lot of time and energy get diverted into talking about doing things. I've been trying to do my small part of the former by fixing some release-critical bugs so we can release Debian 8. Please join in, and remember to write good unblock requests so our hard-working release team can get through them in a finite time. I realise not everyone will agree with my idea of which bugs, which features and which combinations of packages are highest-priority; that's fine, there are plenty of bugs to go round! Regarding init systems specifically, Debian 'jessie' currently works with at least systemd-sysv or sysvinit-core as pid 1 (probably also Upstart, but I haven't tried that) and I'm confident that Debian developers won't let either of those regress before it's released as Debian 8. I expect the freeze for Debian 'stretch' (presumably Debian 9) to be a couple of years away, so it seems premature to say anything about what will or won't be supported there; that depends on what upstream developers do, and what Debian developers do, between now and then. What I can predict is that the components that get useful bug reports, active maintenance, thorough testing, careful review, and similar help from contributors will work better than the things that don't; so if you like a component and want it to be supported in Debian, you can help by, well, supporting it.

---

PS. If you want the Debian 8 installer to leave you running sysvinit as pid 1 after the first reboot, here's a suitable incantation to add to the kernel command-line in the installer's bootloader. This one certainly worked when KiBi asked for testing a few days ago:

preseed/late_command="in-target apt-get install -y sysvinit-core"

I think that corresponds to this line in a preseeding file, if you use those:

d-i preseed/late_command string in-target apt-get install -y sysvinit-core

A similar apt-get command, without the in-target prefix, should work on an installed system that already has systemd-sysv. Depending on other installed software, you might need to add systemd-shim to the command line too, but when I tried it, apt-get was able to work that out for itself. If you use aptitude instead of apt-get, double-check what it will do before saying "yes" to this particular switchover: its heuristic for resolving conflicts seems to be rather more trigger-happy about removing packages than the one in apt-get.

Most of my recent Debian work has been the usual pkg-telepathy stuff (mainly in experimental while we're frozen), and hacking on the Quake III engine. Having started working on OpenArena DFSG-freeness as random bug-squashing a year ago, I've accidentally ended up taking over the package. The situation for Debian 6.0 (squeeze) looks something like this:

• openarena contains both the engine (a modified ioquake3, which I modified further for Debian) and the game logic compiled to native code
• openarena-data has had the bytecode game logic stripped out, since no Free compiler can produce it; the bytecode has been replaced with stub files that direct our modified engine to load native code

The plan for Quake III in Debian 7.0 (wheezy) has already started in experimental:

• ioquake3 contains the ioquake3.org port of the Quake III engine, which I've modified to be able to run either Quake III Arena or OpenArena based on runtime options
• openarena contains the game logic compiled to native code (in experimental it still contains source code for the engine, but it's no longer patched or compiled, and I'll hopefully drop it entirely after the next upstream release), and launcher scripts to run it under the ioquake3 engine
• openarena-data is much like it is now, although it might move to data.debian.org if that happens
• quake3 (currently in the NEW queue) contains launcher scripts to run Quake III Arena under the ioquake3 engine, requiring a non-distributable quake3-data package
• game-data-packager version 23 or later can build the quake3-data package for local use, if you give it pak0.pk3 (from the CD-ROM) and the latest Quake III patch

I haven't been doing as much QA work as the RCBW crowd, but here's some halfway-recent bug squashing in the hope that it motivates others:

• alsa-lib Debian bug #589896 upstream patch backported, NMU'd and herded into testing
• atftpd Debian bug #598474 downgraded
• courier-authlib Debian bug #554788 patch submitted, uploaded
• isc-dhcp Debian bug #592361 patch submitted, uploaded by maintainer by maintainer
• mixxx Debian bug #587110 patch submitted, acked by maintainer
• tar Debian bug #602184, Debian bug #602209 upstream patches NMU'd
• witty Debian bug #591209 patch submitted, although it doesn't list copyright holders so probably isn't suitable for upload

I also started looking at the series of bugs regarding Flash not compiled from source, but I mostly got distracted by all the webapps having a contrib/ directory containing a million embedded code copies...

gfcombinefs is a side project I did some work on a few weeks ago. It combines several "shares" of a secret file previously split using Shamir secret sharing, to produce the original secret, and presents it as a file in a FUSE filesystem. It uses libgfshare for the actual mathematics, and expects its input to have been split with the gfsplit tool shipped with libgfshare. At this stage of development, I suggest not trusting it with important data, like the GnuPG secret keyrings it's designed for. However, I hope that with some feedback from others I can get it into a state where it's ready to be released and packaged (perhaps I'm being unnecessarily conservative, but for something that deals with GnuPG keys, it seems wise to be careful). Source code is available in a git repository, and I'd welcome contributions, bug reports (other than the limitations that are already listed in the documentation) and in particular, a systematic code audit from someone (the good news is that there isn't very much code, so that shouldn't actually take long).

More release critical bug squashing of the "week" (fortnight, really):

• openarena-data Debian bug #523323 + patch (the necessary changes in openarena are extensive, so I don't plan to NMU this unless there's no response for a long time)
• fuse Debian bug #550334, Debian bug #553015, Debian bug #557143

Removals from testing and proposed removals from unstable:

• alien-arena Debian bug #559725 (removal from testing only)
• commit-tool Debian bug #558592 (removal from testing only)
• gadfly Debian bug #559375
• gaim-otr Debian bug #552761 (transitional package)
• levee Debian bug #558604
• libroxen-adbanner Debian bug #559251
• libroxen-diary Debian bug #559250
• libroxen-imho Debian bug #559249 (removed from archive)
• libroxen-tokenfs Debian bug #559248
• mailreader Debian bug #559252
• openc++ Debian bug #559111
• overkill Debian bug #558669
• p3nfs Debian bug #559105
• portslave Debian bug #559254
• skyutils2 Debian bug #558605 (removed from archive)
• whirlpool Debian bug #559255
• ziproxy Debian bug #559627

Possible candidates for removal:

• bfm (unmaintained upstream since 2004, declining popcon score)

Other drive-by QA:

• tse3 Debian bug #524726 marked as done (by a binNMU)
• joy2key Debian bug #554521 downgraded to important
• nvidia-graphics-drivers Debian bug #525414 noted as probably fixed, submitter/maintainer pinged for confirmation
• Bugs filed about the packages keeping zope3 around in testing: Debian bug #558204, Debian bug #558205
• mail-notification-evolution Debian bug #559106 upgraded to serious

RCBW - week #12 Here is this week squashes, by yours truly:

• 23/11/2009 bug #553122 - libtcl-chiark-1 - add missing ldconfig invocation, for *.so in linker locations
• 24/11/2009 bug #528152 - tulip - replace ttf-bistream-vera w/ dejavusans (patch from James Westby)
• 25/11/2009 bug #533993 - solfege - remove net need for xmllint (patch from Ubuntu by Alessio Treglia)
• 26/11/2009 bug #531220 - nut - fix ntp.conf postinst processing to avoid overwriting user settings
• 27/11/2009 bug #523912 - mit-scheme - add missing build-dep + tricky build due to bootstrap
• 28/11/2009 bug #521964 - libapache2-mod-lisp - fix APR version detection, patch by Antonio Radici
• 29/11/2009 bug #553257 - pidgin-hotkeys - remove unappropriate e-Xclusion in dh_shlibdebs

The funniest part of this RCBW issue is probably that Luk has started handing over RC bugs to me: from time to time I just get a /query from him with bug numbers In fact, this is more useful than one can imagine, because a considerable amount of time in periodic RC squashing is taken by triaging the RC bug list to choose your targets. Of course there is no guarantee that I'll be willing/able to fix handed-over bugs but, as a RM, Luk has evidently well-trained feelings about how hard would be to fix a given bug. All in all, for this week it worked well: 4 of the above 7 came from SchindlerLuk's list. ... and of course, let's welcome Simon on board! He is definitely doing a better job at spotting packages that should be removed from the archive rather than resurrected (potential negative effect that a NMU can indeed have on a package, de facto "saving" it from testing/release exclusion). In fact, the dilemma of "NMU vs Request Removal" raises various tricky questions. For instance, while we currently have pretty clear guidelines on how and, more importantly, when doing NMUs, we are more stringent on "Non-Maintainer Removals". Quoting from the Package Removal Requests wiki page:

    In all cases, if there is a maintainer and it's not you, mention the
    maintainer's opinion or, if you don't know it, mention how and when you
    tried to contact him.

It is of course reasonable to be more stringent on a measure that is way more invasive than a "simple" NMU. But if we start to rely on NMU-time as an important trigger for removal requests, then we could probably use a more aggressive and more distributed process for removals.

I've been intermittently prodding at the Debian release-critical bug list for some time, but inspired by Zack and Tim I've decided to start keeping track, if only for my own interest.

• Monday: openipmi Debian bug #474087 + patch (not uploaded since I have no way to test IPMI libraries, not having the appropriate hardware...)
• Monday: potential candidates for removal:
  • skyutils (RC bug, library with no reverse dependencies since smssend was removed in 2008)
  • libhid (RC bug, RFA since August 2008, mainly exists to support libphidgets)
  • libphidgets (RC bug, RFA since 2007)
  • libnoise (RC bug, library with no reverse dependencies or popcon votes)
• Thursday: hercules Debian bug #553110 uploaded to DELAYED/7, with various bonus changes including making the copyright file sufficient to satisfy Policy 12.5
• Friday: spider Debian bug #537631 fixed by a QA upload, fast-forwarding through 8 years of Debian policy and upgrading from debhelper 3 (!) to 7 in the process

In general I've been trying to avoid resurrecting packages that I don't think should be in the archive, even if the fix is trivial. I'm not sure whether that applies to spider; according to popcon, it still has a substantial number of users, so it may be worth keeping even if there are no upstream releases (also, it amused me to convert such an old package to Debhelper 7 and dpkg source format v3). In the process I've discovered that git-buildpackage makes a great NMU tool. I'll probably be putting all future NMU diffs in my users/smcv directory on git.debian.org (at least until the maintainer acknowledges or rejects the NMU), just because it's a convenient way to give the maintainer a nice queue of individual changes rather than a monolithic diff; if any maintainers decide they'll use git as a result, that's a bonus :-)

This afternoon Andre released telepathy-qt4 0.2.0, the first version that builds a shared library, and I've uploaded it to Debian (it'll get into unstable as soon as it clears the NEW queue). This is a major milestone for telepathy-qt4: over the course of 16 months' development it's gone from a collection of auto-generated constants (Olli Salli's initial commit, back in July 2008) to a shared library with a frozen API and ABI. We encourage Qt/KDE application developers to treat telepathy-qt4 as the preferred Qt 4 binding for Telepathy. Andre and the rest of the Telepathy project will continue to add high-level bindings for more Telepathy features over the course of the 0.3.x series; we plan to keep the API and ABI backwards-compatible until shortly before the next milestone, 0.4.0, at which point we'll consider breaking ABI to drop deprecated functionality.

(The much-delayed part 2 in an ongoing series: shiny things in telepathy-glib) Over a year ago I explained my first major round of work on telepathy-glib, with which it could be used to implement D-Bus services. I started to write this post shortly after, but then got distracted by a bee or something; recent discussion about having a D-Bus binding in GLib has finally prompted me to finish it. The second major round of work on telepathy-glib dealt with implementing client code, and went sufficiently well that telepathy-qt4 (currently under development) uses essentially the same model. Again, while telepathy-glib is intended for Telepathy implementors, I think TpProxy's ideas are generically useful, particularly for "large" D-Bus APIs.

Some background: dbus-glib and libtelepathy

The life-cycle of a DBusGProxy (PD clipart credit: karderio and Andy)

The API provided by dbus-glib for client code is, er, rather less than ideal. The basic object is DBusGProxy, which has a few problems. There's one per interface, rather than one per object; this leads to a maze of twisty GObjects, all suspiciously similar. Our old client library, libtelepathy, got round this by subclassing DBusGProxy to make helper objects called TpChan, TpConn etc., which were simultaneously the proxy for the "main" interface (Channel, Connection etc.), and a factory for proxies for the other interfaces (like Channel.Type.Text and Channel.Interface.Password). This still wasn't ideal, but it was a start. If you use a DBusGProxy in any way after it has emitted the destroy signal, this is considered to be invalid use of the API, and dbus-glib asserts all over the place. We'd rather it didn't do that - remote objects becoming invalid is a fact of life, and if you don't immediately discard all your references to those objects, remote operations on them should just give you a runtime error. You have to be prepared to handle such runtime errors already, because any IPC call can fail at any time. The main API for calling methods uses varargs and is rather subtle. Notably, the calling convention for variant parameters (of D-Bus signature 'v') is not consistent with the calling convention for any other container. For any other container, for instance a GPtrArray *, you pass in a GPtrArray ** pointing to a GPtrArray * variable containing NULL; on success, a pointer to a new GPtrArray is written into that variable. For variant parameters, though, you must pass in a GValue * pointing to a zero-filled GValue, and the result will be written into that GValue. dbus-binding-tool generates thin wrappers around this method-calling API which provide type-safety, although they can't directly specify a non-default timeout. As for signals, before binding to a signal, you have to tell dbus-glib about its signature. This tells dbus-glib how it should push the signal arguments onto the C stack (in GObject terminology: how it should marshal them), to match the signature of the signal-handler function. The documentation claims that this isn't necessary for objects supporting the Introspect method, but that's untrue - on the other hand, if the documented behaviour actually happened, I would consider this to be a critical bug in dbus-glib, since it would enable remote objects to crash dbus-glib clients (by causing signal arguments to be pushed onto the stack in a way that does not match the signature of the signal handler). Of course, the signal-adding function takes varargs, and has to be called exactly once per signal per object. If you don't, dbus-glib asserts when you connect to the signal; if you call it twice, dbus-glib asserts anyway.

TpProxy TpProxy is a GObject subclass vaguely inspired by Python's dbus.proxy.ProxyObject. It isn't a subclass of DBusGProxy, which lets us encapsulate dbus-glib almost completely - in fact, all the references to DBusGProxy are in a separate header, proxy-subclass.h, only used by subclasses of TpProxy (as opposed to normal library API users). TpProxy instances are per-remote-object (per-object-path) like in dbus-python, rather than per-interface like in dbus-glib or QtDBus. As explained below, I believe that this is the most natural object mapping for D-Bus interfaces; in practice it tends to lead to clearer code for us, since one remote object often maps nicely to one UI element (of course, if your objects only have one interface, this approach is no better or worse than dbus-glib's). TpProxy provides support for the following common D-Bus things:

• Calling methods
• Connecting to signals
• Objects becoming invalid
• Objects with optional interfaces
• Extensibility

It doesn't currently provide any particular help with Properties - the Properties interface is just another interface as far as we're concerned. We find that, in practice, that's not a problem (even though we use D-Bus properties extensively). In telepathy-qt4 we basically reinvented TpProxy in C++, as Tp::DBusProxy. I've put in some comments below illustrating how the general ideas we used in TpProxy translate into a different object mapping.

Calling methods Method calls on TpProxy are done through auto-generated wrapper functions, much like DBusGProxy. These look something like this:

typedef void (*tp_cli_channel_callback_for_close) (TpChannel *proxy,
    /* any 'out' arguments would go here - Close() doesn't have any */
    const GError *error, gpointer user_data,
    GObject *weak_object);
TpProxyPendingCall *tp_cli_channel_call_close (TpChannel *proxy,
    gint timeout_ms,
    /* any 'in' arguments would go here - Close() doesn't have any */
    tp_cli_channel_callback_for_close callback,
    gpointer user_data,
    GDestroyNotify destroy,
    GObject *weak_object);

Methods for some interfaces, like DBus.Properties, are available on TpProxy itself; others, like Telepathy's Channel.anything, are only available on a particular subclass, like TpChannel. Things to note here:

• It's an asynchronous call with a callback. General design principle: "this is IPC, get over it". We don't try to hide the fact that IPC is taking place by doing pseudo-blocking calls.
• The timeout is explicit (although you can always pass -1 to let libdbus pick a "reasonable" default for you). This is because the remote service might not respond, and API users should be at least vaguely aware of this: "this is IPC, get over it".
• The callback isn't a GCallback; it has an explicit signature, for some semblance of type-safety (it's not much, but it's better than nothing).
• The callback takes a GError, because any IPC call can fail for any reason.
• A TpProxyPendingCall * is returned. This is a pointer to an object that only exists until just after the callback is called (so you can ignore it if you don't want it), with a method you can call to cancel the method call (with hindsight this is not ideal, and if I was writing this API now, I'd use gio's GCancellable). "Cancel" is perhaps slightly misleading: the method call still takes place (it was already in libdbus's outgoing queue) but the result gets ignored and the callback isn't called. This is mostly so you an object can forget all the calls it was busy making at the time that the results become irrelevant (e.g. the object is destroyed).
• There is an extra weak_object argument, which is weakly referenced and is passed to the callback: in practice, most users of telepathy-glib use this as well as or instead of user_data. If the weak_object runs out of references, the callback is automatically cancelled, which means that in practice, explicit cancellation is almost never needed.
• The generated code has as its first argument a suitable subclass of TpProxy, in this case TpChannel (we tell the code generator about subclasses).
• You can't see it in the API, but the TpProxy gains an extra ref while a method call is in progress, so you never have to worry about what happens if the proxy gets unreffed during a method call.
• There's an explicit destructor for the user_data, which is called even if you "cancel" the method call.

In telepathy-qt4, these methods are on small generated helper classes similar to dbus-python's dbus.Interface, one per interface (we provide accessors for them on the non-generated DBusProxy subclass). Instead of using callbacks, we return a temporary QObject per call, which emits a Qt signal on success or failure (that's how all async calls in QtDBus work).

Connecting to signals Similarly, signals have some generated functions:

typedef void (*tp_cli_channel_signal_callback_closed) (TpChannel *proxy,
    /* any arguments would go here - Closed doesn't have any */
    gpointer user_data, GObject *weak_object);
TpProxySignalConnection *tp_cli_channel_connect_to_closed (TpChannel *proxy,
    tp_cli_channel_signal_callback_closed callback,
    gpointer user_data,
    GDestroyNotify destroy,
    GObject *weak_object,
    GError **error);

API notes for these:

• The callback doesn't take a GError any more, because signals can't fail; the weak_object and the user_data are the same as for method calls
• This isn't actually a GObject signal at all: those aren't particularly typesafe, and lack namespacing. This might not be very binding-friendly, we haven't tried doing that yet... it might be worth introducing a signal with details, like in dbus-glib, for which these functions are "C bindings".
• The returned TpProxySignalConnection lets you disconnect from the signal, just like the TpProxyPendingCall above; it can safely be ignored
• Similarly, if the weak_object dies, then the signal automatically disconnects
• It is possible for connecting to a signal to fail: this happens if the proxy doesn't actually have the requested interface. If this happens, it's graceful, not a crash. In practice, the flow of code is almost always such that error can safely be NULL because the interfaces are already known.

In telepathy-qt4 the per-interface generated helper classes emit Qt signals. In many cases, we end up proxying these signals by responding to them in the hand-written Tp::DBusProxy subclass by emitting a different signal, in order to give them a nicer representation.

Becoming invalid This is IPC, and any error can happen to you at any time. We found that it was useful to have a general concept of "this object is no longer useful", so we introduced the concept of an object becoming invalidated. The invalidation reason is stored on the object as a GError. If this GError is NULL (as it is initially), then the object is still expected to be valid; if it's non-NULL, the remote object has vanished and the local proxy no longer works. There are several ways a TpProxy can become invalid:

• its bus name falls off the bus (the process exited or crashed)
• application-specific reasons (usually triggered by a D-Bus signal in a subclass, like the Telepathy Channel's Closed signal)
• the GObject gets disposed

Method calls on an invalidated object always fail, with the invalidation reason as the error. This means that calling a method on a TpProxy should never crash your process. The method's callback has to be prepared to handle an error in any case, because this is Sparta^WIPC, so there's no loss in giving it another error. Connecting to signals on an invalidated object always fails, and any existing signal connections are disconnected when it becomes invalidated. This means that if a new remote object appears, and it happens to have the same object path, bus name etc. as the old one, you won't get its signals. A TpProxy can either be bound to a unique bus name or a well-known bus name. In the equivalent code in telepathy-qt4, we call this "stateful" or "stateless" - these names don't capture the intention perfectly, but they're the best we could come up with. A proxy for a stateful API like Telepathy's Channel should always bind to a unique name - when the unique name falls off the bus, the TpProxy becomes invalidated automatically, representing the fact that the Channel you had no longer exists, and nothing can be a perfect replacement for it (the VoIP call has already been terminated, you've already left the chatroom, or whatever - if you make another Channel, that's e.g. a distinct VoIP call). A proxy for a stateless API like Telepathy's ConnectionManager, where an exiting process can disappear, be service-activated again, and still have exactly the same API, should bind to the well-known name. Proxies for well-known names aren't invalidated when the process exits. Qt doesn't have a natural equivalent for GError, so the invalidation reasons in telepathy-qt4 are just a pair of strings, the namespaced name and the message - basically a libdbus DBusError. The principle is the same, though.

Optional interfaces As mentioned above, TpProxy instances are per-remote-object (per-object-path) like in dbus-python, rather than per-interface like in dbus-glib or QtDBus. I believe that this is the most natural object mapping for the D-Bus object model, because the D-Bus interfaces on a remote object can behave like any of these:

• classes and subclasses (Telepathy.Channel.Type.Text is a Telepathy.Channel)
• shared interfaces (some Telepathy.Channel objects, of many Types, implement Telepathy.Channel.Interface.Group; any D-Bus object can implement DBus.Properties)
• optional features (some Telepathy.Channel.Type.Text objects implement Telepathy.Channel.Interface.ChatStates, some do not)
• discoverable extensions (some Telepathy.Channel.Type.Text objects implement Telepathy.Channel.Interface.Messages, which is more or less Text 2.0)

The way in which interfaces are discovered is also variable. The "classic" D-Bus way to discover interfaces would be to call Introspect. Telepathy services still support that style of introspection, but we don't use it for anything beyond d-feet:

• Introspect returns a blob of XML, which you have to parse into something useful; if it was in a D-Bus data structure, you'd already have some sensible data structure.
• Finding out more information about the supported interfaces than their names isn't very useful: in a static language like C you already need to know the methods, signals, properties and their signatures in order to write and compile your code, and in a dynamic language like Python consumers of Introspect end up being remotely crashable by services with unexpected method signatures (this is one of dbus-python's big mistakes in my opinion).
• If a method that you want to call turns out to be missing, the worst that can happen (in a competently written service) is that it fails with a D-Bus error - and you have to handle D-Bus errors anyway, because this is IPC and anything could fail.
• Static bindings like dbus-glib have poor support for removing interfaces for which C code exists at runtime, whereas Telepathy needs features like an IM connection that might or might not support avatars (we don't get to find out until we've successfully connected to the server).

Instead, our older interfaces have a method on the "base class" (Channel, Connection etc.) called GetInterfaces, which just returns an array of strings. Newer interfaces (like Account), and older interfaces that have been ported (like Channel), have a property called Interfaces which is, again, an array of strings - this lets us combine the download of the interfaces list with downloading other basic information in a GetAll call. To cater for all these ways to use interfaces, the TpProxy base class has very basic support for interface discovery - you can ask it whether it supports an interface, and you can tell it that it does, in fact, support an interface. Asking which interfaces are supported is directly useful for library users; it's also used as a check by the generated method-call and signal-connection stubs, which return a canned error (Telepathy.Error.NotImplemented in Telepathy's case) if the object isn't known to have the interface. Telling TpProxy that it does support an interface is intended for use by subclasses, and might have been protected if we were writing C++: what happens in practice is that a concrete subclass like TpChannel knows how to discover the fully supported interfaces for this particular object, does so, and calls methods on the TpProxy to tell it which interfaces can work.

Extensibility The API stubs used by TpProxy to call methods and bind to signals are generated from XML documents containing an augmented form of D-Bus introspection data. These XML documents contain various Telepathy-specific extensions, but none of the extensions are language-specific - telepathy-glib, telepathy-python and telepathy-qt4 all operate from exactly the same XML specification. telepathy-qt4's different object-mapping did require us to tighten up some rules that were previously only conventions, but even so, the format is the same. I think this is vitally important - it just doesn't scale to have one set of language-specific annotations in a D-Bus API for each language that will have bindings. One thing that we make heavy use of is what I call "Ugly_Case", which is camel-case with underscores at word boundaries. This gives us a simple, unambiguous and foolproof rule to use in code generators whenever we generate any of the popular conventions for identifiers:

• CamelCase: delete underscores
• javaCamelCase: force everything before the first underscore to lower case, then delete underscores
• lower_case: force everything to lower case, retain underscores
• UPPER_CASE: force everything to upper case, retain underscores

dbus-glib, by contrast, applies a complex and subtly buggy algorithm to the CamelCase D-Bus names, which results in our SendDTMF method being mapped into GLib as send_dt_mf, and requires that telepathy-glib's Python reimplementation of dbus-binding-tool uses a bug-for-bug compatible implementation (we ended up reimplementing dbus-binding-tool in order to generate the TpProxy-based bindings). Our code generation tools are still rather ad-hoc, because so is our spec format, but for draft interfaces it's possible to copy them into individual projects and use them to generate additional methods for TpProxy or any subclass. This is how we implement unfinished APIs like geolocation in Empathy, for instance. The Interfaces or GetInterfaces hook described above is entirely usable for these extension interfaces, and in fact that's how they're set up. In telepathy-qt4, the generated method stubs are genuine C++ methods, so we can't just append methods to an existing class: this is why we have one helper class per interface, so individual clients can generate a helper class for their particular version of an unfinished API, and instantiate an object of this class attached to a particular Tp::DBusProxy. When an interface becomes stable, it can be added to the set of interfaces for which code is generated in telepathy-glib or telepathy-qt4, at which point any client or service that was already using the final draft of that interface can easily be ported to the library version using sed (the API remains the same).

Becoming ready The concept of being "ready" does not directly exist in TpProxy, but is implemented in TpChannel, TpConnection and TpConnectionManager. The idea is that a freshly created proxy object isn't really fully usable yet - you have to connect to basic signals and recover the initial state of the remote object, as well as checking which extension interfaces are supported. In many well-designed D-Bus APIs you can do this initial setup with a few signal connections and a DBus.Properties.GetAll call. After doing that initial setup (which has to be done asynchronously, because it's IPC), the TpProxy subclass has a local cache of the remote object's state (not necessarily in the same format as the representation on the bus), which can be accessed synchronously at any time, and will be updated in response to change-notification signals. TpChannel and subclasses like it have explicit support for checking for readiness, and having a callback called when the object is ready or invalidated (whichever happens first). The "ready" property also supports the GObject notify signal. telepathy-qt4 took this concept and ran with it - there is library support for objects that can become ready, including a mixin. If I had as much time to work on telepathy-glib as I wish I had, this would be one of the first telepathy-qt4 features to be added to telepathy-glib.

Features becoming ready Going beyond that, in an extensible framework like Telepathy it's probable that not every client understands (or wants to understand) every feature of every object, so it's highly inefficient for every proxy object to subscribe to all the change notification signals and cache all the remote state on the off-chance that the library user wants them. In the API for TpContact (which is not actually a TpProxy for various reasons) we introduced the concept of optional features; telepathy-qt4 extended this idea throughout the library. The idea is that each library user knows which of the optional features are "interesting" to it, and makes a single asynchronous call (which expands into a series of D-Bus calls inside the library) to download the state for core functionality plus all of the selected features, and subscribe to change notification for all of those. This is another telepathy-qt4 thing that telepathy-glib still needs to catch up with; until someone works out how to make the clone() syscall apply to programmers, I fear all libraries are doomed to lag behind how their designers want them to look :-) (license: CC-BY-3.0 / LGPL-2.1+)

At some point I might document my current e-mail setup, but for now, here's the bit that caused plenty of frustration on Friday. Many howtos for postfix and dspam will tell you that if you embed user IDs in the signature with PgSQLUIDInSignature on (or the MySQL equivalent, you can do something like this for retraining, and set it up to receive spam@example.com:

# master.cf
dspamretrain unix -     n       n       -       10      pipe
  flags=R user=dspam argv=/usr/bin/dspam --user dspam
      --class=$ nexthop  --source=error
# mailbox_transport map
spam    dspamretrain:spam
notspam dspamretrain:innocent

in which --user dspam names an arbitrary user (dspam will extract the uid from the signature and switch from "dspam" to the correct user, but the command line argument is still required, for no particular reason). I said it named an arbitrary user, but actually that's a lie. The things to be careful of are:

• the dspam user must be listed in your database (e.g. in dspam_virtual_uids for postgres users with the default setup)
• if you're running dspam in opt-in mode, dspam must have opted in (so create /var/spool/dspam/opt-in/local/dspam.dspam)

Having failed at both of those, my retrain address wasn't working, and sadness ensued.

Developers seem to do a lot of fighting with bleeding-edge libraries installed in a --prefix, so I thought I'd share how I develop telepathy-glib (a library) and telepathy-gabble (a program using that library). ~/Collabora/telepathy/tpglib on my laptop is a git clone of telepathy-glib.git, branched using the process I documented on http://telepathy.freedesktop.org/wiki/Git, which is basically:

• clone the upstream repository (git+ssh://git.collabora.co.uk/git/telepathy-glib.git in this case) into your home directory on the server (git+ssh://people.collabora.co.uk/home/smcv/public_html/git/telepathy-glib-smcv.git in this case)
• clone that onto your laptop
• on your laptop, add "upstream" as a remote, tracking the upstream repository
• this reduces mistakes, because "git push" pushes to origin (which is in my home directory) for review, and "git push upstream" is necessary when I actually want to commit to the upstream repository

~/Collabora/telepathy/gabble is a git clone of telepathy-gabble.git, using the same process. In my Gabble checkout, I used to configure like this:

cd ~/Collabora/telepathy/gabble
./autogen.sh \
TP_GLIB_CFLAGS="-I$ HOME /Collabora/telepathy/tpglib" \
TP_GLIB_LIBS="$ HOME /Collabora/telepathy/tpglib/telepathy-glib/libtelepathy-glib.la"

This would make libtool link Gabble against the copy of telepathy-glib built by my bleeding-edge git checkout, without ever needing to install it. This allowed me to try out my latest telepathy-glib changes with Gabble's more extensive regression tests, which was a great win. It also meant I could develop Gabble branches that required a not-yet-released telepathy-glib, and all without touching my system copy of telepathy-glib (the one I use to run Empathy and talk to people - which is the packaged version from Debian unstable or experimental). The reason I could do this is that I laid out the telepathy-glib source tree in The Right Way , which is: if a header is designed to be used via #include <my-library/foo.h>, then it goes in a my-library/ directory in the source tree, and includes other headers (even from the same project) with #include <my-library/other.h>. This means you can just add the top directory of the source tree to your include path (which automake does by default) and the right thing will happen. (Actually, this is a slight simplification of the truth: because telepathy-glib contains generated headers, you have to add the top directory of the build tree to your include path too, if you want to support out-of-tree builds. Which, of course, you do.) Other projects that do this correctly include GLib and Avahi. Avahi is a particularly interesting example because it has numerous libraries in the same tarball (including avahi-common, avahi-core, avahi-glib and avahi-gobject, which form a dependency chain). Projects that don't do this correctly include telepathy-mission-control (which is on my hit list for further swamp-draining) and libtelepathy (which is obsolete). Today I committed a patch to telepathy-glib, inspired by GStreamer, to generate telepathy-glib-uninstalled.pc. The presence of this patch simplifies the process further, to:

./autogen.sh PKG_CONFIG_PATH=$HOME/Collabora/tpglib/telepathy-glib

This has the same benefits as setting TP_GLIB_CFLAGS and TP_GLIB_LIBS, but is easier to remember, and is more future-proof against changes to telepathy-glib. It only works because pkg-config has a special case for this situation - if you ask for the package "foo", pkg-config will always look for foo-uninstalled.pc before foo.pc. (Edited 2009-01-13: putting environment variables on the ./configure or ./autogen.sh command line like ./autogen.sh PKG_CONFIG_PATH=foo is better than putting them in the environment like PKG_CONFIG_PATH=foo ./autogen.sh, since it works around Debian bug #398901 and bugs like it)

I recently converted most of the Debian packaging for Telepathy and related projects from bzr to git, while changing the repository contents from just the debian/ directory to the whole source tree. Here's the recipe, using the latest one to be converted (pymsn) as an example.

Create a repository

mkdir pymsn
cd pymsn
git init

Mass tarball import

• download all the tarballs into ../tarballs
• rename all the tarballs to *.orig.tar.gz:

  rename 's/^pymsn-/pymsn_/' *.tar.gz
  rename 's/\.tar\.gz$/.orig.tar.gz/' *.tar.gz
  

• use git-import-orig to import them:

  cd ../pymsn
  ls ../tarballs/pymsn_*.orig.tar.gz
  for v in 0.2 0.2.1 0.2.2 0.3.0 0.3.1 0.3.2 0.3.3; do \
  git-import-orig --no-merge --no-dch --pristine-tar \
      -u $v ../tarballs/pymsn_$v.orig.tar.gz; done
  

• throw away the resulting master branch in favour of using one we convert from bzr later:

  git checkout upstream
  git branch -D master
  

Converting the Debian packaging This has the slight complication that in some (but not all) of the bzr commits, the repository contained only the contents of the debian directory, in the root of the repository. So, I needed to rewrite history, with this script:

#!/bin/sh
# index-filter.sh
if git ls-files -s   grep debian; then
    :
else
    git ls-files -s  
        sed -e "s-\t-&debian/-"  
        GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &&
    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE
fi

• Download unstable and experimental packaging into ../pymsn-experimental and ../pymsn-unstable
• Use bzr-fast-export and git-fast-import to import the two branches:

  ~/.bazaar/plugins/fastimport/exporters/bzr-fast-export.py \
      ../pymsn-unstable   git fast-import
  git branch -m master debian-lenny
  ~/.bazaar/plugins/fastimport/exporters/bzr-fast-export.py \
      ../pymsn-experimental   git fast-import
  git branch -m master debian
  git branch -D tmp
  

• Rewrite history so changelog, control etc. are consistently inside a debian directory, with the script shown above:

  git filter-branch --index-filter $(pwd)/../index-filter.sh debian-lenny
  rm -r .git/refs/original
  git filter-branch --index-filter $(pwd)/../index-filter.sh debian
  rm -r .git/refs/original
  

• Merge the appropriate upstream versions into each branch:

  git checkout debian
  git merge upstream/0.3.3
  vim debian/control        # and change Vcs-Bzr to Vcs-Git
  dch "Move packaging to git"
  debcommit -a
  git checkout debian-lenny
  git merge upstream/0.3.1
  vim debian/control        # and change Vcs-Bzr to Vcs-Git
  dch "Move packaging to git"
  debcommit -a
  

• Rescue Debian patches and put them on a debian-patches branch, which is rebased with each upstream release. For pymsn, there aren't any patches in the debian branch at the moment:

  git branch debian-patches upstream/0.3.3
  

  but there is one in the debian-lenny branch:

  git branch debian-lenny-patches upstream/0.3.1
  git checkout debian-lenny
  cp debian/patches/00-fix-dns-resolution.diff ..
  git checkout debian-lenny-patches
  patch -p1 < ../00-fix-dns-resolution.diff
  git commit -a
  

• Make a repository on alioth:

  cd /git/pkg-telepathy
  GIT_DIR=pymsn.git git init --bare --shared
  GIT_DIR=pymsn.git git config hooks.mailinglist ...
  vim pymsn.git/description
  

• Push to the repository:

  git gc --aggressive
  git remote add origin git+ssh://git.debian.org/git/pkg-telepathy/pymsn.git
  git push --all
  

• On alioth, do some more setup after the initial push:

  cat > pymsn.git/hooks/post-receive <<EOF
  #!/bin/sh
  exec /usr/local/bin/git-commit-notice
  EOF
  chmod +x pymsn.git/hooks/post-receive
  GIT_DIR=pymsn.git git symbolic-ref HEAD refs/heads/debian
  

• Mark the old repositories as no longer used:

  cd ../pymsn-unstable
  dch $'This repository is no longer used, instead please use git:\n'\
  "git://git.debian.org/git/pkg-telepathy/pymsn.git" -c changelog
  debcommit -c changelog
  bzr push
  cd ../pymsn-experimental
  dch $'This repository is no longer used, instead please use git:\n'\
  "git://git.debian.org/git/pkg-telepathy/pymsn.git" -c changelog
  debcommit -c changelog
  bzr push
  cd ../pymsn
  

The Debian Python Modules Team is discussing which DVCS to switch to from SVN. Ondrej Certik asked how to generate a list of commiters to the team s repository, so I looked at it and got this:
emilio@saturno:~/deb/python-modules$ svn log egrep "^r[0-9]+ cut -f2 -d sed s/-guest// sort uniq -c sort -n -r
865 piotr
609 morph
598 kov
532 bzed
388 pox
302 arnau
253 certik
216 shlomme
212 malex
175 hertzog
140 nslater
130 kobold
123 nijel
121 kitterma
106 bernat
99 kibi
87 varun
83 stratus
81 nobse
81 netzwurm
78 azatoth
76 mca
73 dottedmag
70 jluebbe
68 zack
68 cgalisteo
61 speijnik
61 odd_bloke
60 rganesan
55 kumanna
52 werner
50 haas
48 mejo
45 ucko
43 pabs
42 stew
42 luciano
41 mithrandi
40 wardi
36 gudjon
35 jandd
34 smcv
34 brettp
32 jenner
31 davidvilla
31 aurel32
30 rousseau
30 mtaylor
28 thomasbl
26 lool
25 gaspa
25 ffm
24 adn
22 jmalonzo
21 santiago
21 appaji
18 goedson
17 toadstool
17 sto
17 awen
16 mlizaur
16 akumar
15 nacho
14 smr
14 hanska
13 tviehmann
13 norsetto
13 mbaldessari
12 stone
12 sharky
11 rainct
11 fabrizio
10 lash
9 rodrigogc
9 pcc
9 miriam
9 madduck
9 ftlerror
8 pere
8 crschmidt
7 ncommander
7 myon
7 abuss
6 jwilk
6 bdrung
6 atehwa
5 kcoyner
5 catlee
5 andyp
4 vt
4 ross
4 osrevolution
4 lamby
4 baby
3 sez
3 joss
3 geole
2 rustybear
2 edmonds
2 astraw
2 ana
1 twerner
1 tincho
1 pochu
1 danderson
As it s likely that the Python Applications Packaging Team will switch too to the same DVCS at the same time, here are the numbers for its repo:

emilio@saturno:~/deb/python-apps$ svn log egrep "^r[0-9]+ cut -f2 -d sed s/-guest// sort uniq -c sort -n -r
401 nijel
288 piotr
235 gothicx
159 pochu
76 nslater
69 kumanna
68 rainct
66 gilir
63 certik
52 vdanjean
52 bzed
46 dottedmag
41 stani
39 varun
37 kitterma
36 morph
35 odd_bloke
29 pcc
29 gudjon
28 appaji
25 thomasbl
24 arnau
20 sc
20 andyp
18 jalet
15 gerardo
14 eike
14 ana
13 dfiloni
11 tklauser
10 ryanakca
10 nxvl
10 akumar
8 sez
8 baby
6 catlee
4 osrevolution
4 cody-somerville
2 mithrandi
2 cjsmo
1 nenolod
1 ffm
Here I m the 4th most committer And while I was on it, I thought I could do the same for the GNOME and GStreamer teams:
emilio@saturno:~/deb/pkg-gnome$ svn log egrep "^r[0-9]+ cut -f2 -d sed s/-guest// sort uniq -c sort -n -r
5357 lool
2701 joss
1633 slomo
1164 kov
825 seb128
622 jordi
621 jdassen
574 manphiz
335 sjoerd
298 mlang
296 netsnipe
291 grm
255 ross
236 ari
203 pochu
198 ondrej
190 he
180 kilian
176 alanbach
170 ftlerror
148 nobse
112 marco
87 jak
84 samm
78 rfrancoise
75 oysteigi
73 jsogo
65 svena
65 otavio
55 duck
54 jcurbo
53 zorglub
53 rtp
49 wasabi
49 giskard
42 tagoh
42 kartikm
40 gpastore
34 brad
32 robtaylor
31 xaiki
30 stratus
30 daf
26 johannes
24 sander-m
21 kk
19 bubulle
16 arnau
15 dodji
12 mbanck
11 ruoso
11 fpeters
11 dedu
11 christine
10 cpm
7 ember
7 drew
7 debotux
6 tico
6 emil
6 bradsmith
5 robster
5 carlosliu
4 rotty
4 diegoe
3 biebl
2 thibaut
2 ejad
1 naoliv
1 huats
1 gilir

emilio@saturno:~/deb/pkg-gstreamer$ svn log egrep "^r[0-9]+ cut -f2 -d sed s/-guest// sort uniq -c sort -n -r
891 lool
840 slomo
99 pnormand
69 sjoerd
27 seb128
21 manphiz
8 he
7 aquette
4 elmarco
1 fabian
Conclusions:
- Why do I have the full python-modules and pkg-gstreamer trees, if I have just one commit to DPMT, and don t even have commit access to the GStreamer team?
- If you don t want to seem like you have done less commits than you have actually done, don t change your alioth name when you become a DD (hint: pox-guest and piotr in python-modules are the same person)
- If the switch to a new VCS was based on a vote where you have one vote per commit, the top 3 commiters in pkg-gnome could win the vote if they chosed the same! For python-apps it s the 4 top commiters, and the 7 ones for python-modules. pkg-gstreamer is a bit special

I've found myself writing this several times recently in the context of Telepathy, so I thought I'd make a blog post that I can refer people to. D-Bus is defined (by the wire protocol spec) to be an asynchronous message-passing system, and libdbus behaves accordingly; there are no blocking calls at the wire protocol level. However, libdbus provides a "blocking" API (dbus_do_something_and_block), via a mechanism I'll refer to as "pseudo-blocking" for the purposes of this post. Most D-Bus bindings (at least dbus-glib, dbus-python and QtDBus) expose this pseudo-blocking behaviour to library users. These pseudo-blocking calls work like this:

• send a method call message (call it M)
• while a reply to M has not been received:
  • select() or poll() on the D-Bus socket, ignoring all other I/O
  • whenever a whole message has been received:
  • check whether the message is a reply to M
  • if it is (call it R), stop
  • if it is not, put it on the incoming message queue

The messages received between M and R are delivered when the main loop is next entered. This can cause a number of problems:

• Messages are re-ordered: messages received between M and R aren't delivered until after the reply, violating the ordering guarantee that the D-Bus daemon usually provides. (This causes practical problems if a signal indicating object destruction is delayed - the client gets a method reply "UnknownMethod", has to guess that this is because the object has vanished, and can't know why it vanished until the signal indicating its destruction arrives with more details.)
• The client is completely unresponsive until the service replies - if the service has somehow got wedged (e.g. telepathy-gabble is meant to be purely non-blocking and asynchronous, but there are cases where it will do blocking I/O on SSL connections due to Loudmouth bugs), the client will be unresponsive for (by default) 25 seconds until the call times out. (Clients shouldn't crash or lock up, whatever happens to the services they depend on.)
• The client can't parallelize calls - if a signal (e.g. an incoming Text message in Telepathy) causes method calls to be made, a client that uses pseudo-blocking calls can't even start processing the next message until those method calls return
• If two processes make pseudo-blocking calls on each other, deadlock occurs. This is particularly tricky in the presence of a plugin architecture and shared D-Bus connections - a plugin that "knows" it's a client and not a service, and a plugin in the same process that "knows" it's a service and not a client, can end up sharing a connection, resulting in a process that is both a service and a client (and hence deadlock-prone). (We've seen this happen in the OLPC Sugar environment and on Nokia internet tablets; it's not just a theoretical concern.)

As a result, telepathy-glib's code generation mechanism does not generate any pseudo-blocking code. There are several alternative modes of operation that we do support:

• Fully asynchronous: call a method now, pass it a callback, get your callback called from the main loop later. This can be awkward to program with, but is the only way forward for most non-trivial projects.
• Re-entrant main loop: re-enter the main loop, processing all messages (D-Bus, GUI, network, anything), and run it until the reply has been received. This is useful for trivial projects (like telepathy-glib's regression tests), but dangerous for non-trivial projects (like Empathy), and I'm somewhat regretting implementing this.
• Ignoring the reply: when appropriate (it rarely is), you can make an asynchronous call with callback = NULL and the reply will be ignored completely

We make a couple of narrowly targeted exceptions to the "no pseudo-blocking" policy in internal code, by allowing telepathy-glib internals to make a small number of pseudo-blocking method calls to the dbus-daemon (which is the one component we can definitely trust to return results promptly). Here are some examples of the same strategies in other libraries:

• Pseudo-blocking: dbus-python method calls with no special keyword arguments; dbus-glib dbus_g_proxy_call; QDBus calls with mode QDBus::Block
• Fully asynchronous: dbus-python method calls with reply_callback and error_callback keyword arguments; dbus-glib dbus_g_proxy_begin_call; QDBusConnection::callWithCallback
• Re-entrant: QDBus calls with mode QDBus::BlockWithGui
• Ignoring reply: dbus-python method calls with ignore_reply keyword argument; QDBus calls with mode QDBus::NoWaitForReply

(Written on Friday 16th, posted on Monday)

As I write this, I'm on a plane off the coast of the Netherlands, on the way back from a couple of days designing APIs with Rob McQueen and the RTCom people at NRC Helsinki (who we work with on the chat and VoIP functionality for Nokia internet tablets).

We've had a very useful couple of days, mainly designing the next-generation channel requesting and dispatch API for Telepathy (the same APIs I've been doing preparatory work for on the Telepathy mailing list in the last couple of weeks). API sketches for review and comment are either on are on Merge Monkey or on the way there.

Before even reaching Nokia, Rob and I did a lot of design on the plane over to Helsinki, and I took some notes: here they are, by way of a glimpse at how the Telepathy design process works when we don't have a whiteboard[1] :-)

• Geoloc looks good in principle, needs some more work
  • add more docstrings
  • add a way to ask people for uncached info (RequestLocations)
  • move access control to Co.I.Presence, call it RichPresenceAccessControl?
• DeliveryReporting looks good, ish
  • improve wording of delivery-echo and rationale
  • when sending a delivery report, add a reason for the status (using Channel_Text_Send_Error?)
• Uniqueness requirement for (channel, handle_type != 0, handle) is in the way - let's drop it
• MSN should use "authentic" 1-1 chats, and use Ch.I.UpgradeableToRoom (name tbd) to migrate the underlying switchboard to be a room
• XMPP should use Ch.I.UpgradeableToRoom too, but the 1-1 channel may also stay open
• Mutable handle => handle 0 must stay (for requestotron's benefit)
• bundles ftw
• renaming: we want SelfHandleChanged in the core
• TpC* - add ability to subscribe to ifaces ay construct time (blocks Ready) or later
• Dispatching: when registering a c'handler, specify everything it can handle
• If a new bundle can be handled in its entirety by the same handler (call it 'the bundle handler'), do it. Otherwise split it up completely (there is no 'bundle handler').
• If a channel in an existing bundle can be handled by the bundle handler, send it there, else send to the default handler.
• Group - add SelfHandleChanged, maybe? Also Connection
• Add HandleOwners: prop a uu , HandleOwnersChanged(a uu , au)
• Use TargetHandleType, TargetHandle
• Stringified initiator handle as a separate property
• Maybe even a FirstMessage property, although this stretches the definition of "immutable"
• org.freedesktop.Telepathy.Client interface

Nokia's Mikhail Zabaluev kindly took notes for some further discussion we had with him and Alberto Mardegan about the dispatch API, which he already posted to the mailing list.

We hope this API will make it possible to integrate Telepathy into the desktop better - it should make it a lot easier to share connections between processes in a useful way and hand out incoming channels to the appropriate handlers, which was always a major goal for Telepathy. It should also let us and others implement exciting new notification mechanisms (hi to the people on IRC and the mailing list wanting to patch their media players to integrate with Telepathy! :-)

[1]	When we do have a whiteboard, the design process is: "Robot101 scribbles on a whiteboard, smcv writes down <tp:rationale> elements"

I'm now a full Debian developer! Slightly too late to vote for my landlord as project leader, but it's OK, he got in anyway :-)

Thanks to the NM team, the newly delegated keyring maintainer, and the previous DPL's eleventh-hour redelegation for making this happen!