I recently bought this lovely Canon EOS 350D camera from a friend who bought the new 400D instead. It's a nice DSLR camera, which I'm convinced is able to take magnificent pictures when used appropriately (my friend's pics are a good example). So since I like things to Just Work, I updated my image downloading script. It used to just do an rsync since my previous camera was available as an USB mass storage device, which I could simply mount with autofs. But the Canon guys still don't make cameras that work that way, and one has to use the PTP protocol instead. Which means rsync isn't an option, and one has to do shell and awk magic to retrieve all the pictures. Because the camera doesn't even put all images in the same folder, but it changes folders once every 100 shots, so you have to parse the output from gphoto2 and get the pics one by one. Anyway. It works, and it even registers the pictures in the F-Spot database, with a tag on them (with a semantic of "TODO" or "unsorted"). If anyone wants the scripts, just give me a shout. Speaking of F-Spot: the new 0.2.2 version wasn't in Debian unstable yet, and the boolean logic queries were very appealing to me. So I took the liberty of preparing an unofficial package, which you'll find in my APT repository until official packages enter Sid. Oh, and since my friend's new EOS 400D isn't supported by the latest released libgphoto2 packages, I also patched them (one-line patch, to add a pair of USB ids) and uploaded them to the same place. Same thing, unofficial packages, use at your own risk, etc. He tells me there's a versioned dependency on dbus that can't be satisfied in testing, but that should be a temporary problem. That was the easy part. Now, the hard part is: learning how to do good-looking photos. I expect that'll take much longer.

A nice example of everyday sexism, I only noticed it when passing by the third or fourth time.



(Note to non-german speaking readers: -in is a female ending)

Dear old Mutt, ... My MUA of choice is Mutt. I've been using it for something like 8 years now and I like it: it's quick, powerful and well integrated with the Unix way of handling mail (e.g. procmail). But we all know that. My feeling today is that sooner or later I will be forced to abandon it. The main reason is that it seems to be a dead product, upstream development is not dead but I fail to see an overall plan for my beloved MUA. Other text based application have understood that they should be integrated in some way with the desktop and/or windowing environment in which they are run: Vim for example is able to deal with the mouse wheel event when run in an xterm or even has developed a GUI which is well integrated with the Gnome session mechanism. Why Mutt is not? Such feelings of today has been originated by Tomboy, a wonderful note taking tool with Wiki-like capabilities, integrated with Gnome. It has a plugin for drag and dropping Evolution mails on notes automatically providing links from the notes to your mailbox: that's ... wow, great! I'm already using Evolution as my organizer and I already have Gaim integrated with it for contacts. I fear the moment I will try it for mails because I'm pretty sure it also has an UI which can be easily used with keyboard without the need of the mouse. Since I'm using all this on my laptop which is always with me (and this means that the argument "Mutt can be used remotely using a 9600 GSM connection" is bogus), it seems I'm running out of arguments for sticking to Mutt. Oh, right, there is the procmail stuff, my last feeble chance of keep on using Mutt ...

I'm working on an application that needs an idle callback (do something after the user has been idle for N seconds). It was pretty easy to hack up once I found some example javascript code (hopefully it works in IE too...):

def on_idle(timeout, options)
  timeout *= 1000
  remote = remote_function(options)
  javascript_tag("
var alarm =  
  remind: function()  
    # remote 
    delete this.timeoutID;
    this.setup(this.ms);
   ,
  setup: function(ms)  
    this.cancel();
    var self = this;
    self.ms = ms;
    this.timeoutID = window.setTimeout(function(msg)  self.remind(); , ms);
   ,
  cancel: function()  
    if(typeof this.timeoutID == \"number\")  
      window.clearTimeout(this.timeoutID);
      delete this.timeoutID;
     
   
 ;
alarm.setup(# timeout );
window.onblur = function()   alarm.setup(# timeout )  ;
window.onfocus = function()   alarm.setup(# timeout )  ;
window.onmousemove = function()   alarm.setup(# timeout )  ;
")
end

I use it like so, after putting it into my application_helper.rb file.

<%= on_idle 600, :url =>   :action => 'on_idle'   %>

It could probably improved by making it take some options to determine which events cause un-idleness and perhaps call a no-longer-idle hook. I'll update this page with those changes should I get around to them. I wonder if there's a good, centralized place to submit these kinds of little hacks to? The Rails trac system seems to be full of spam, indicating that people aren't triaging bugs at all.

Interestingly, Simon Richter write about rerunning maintainer scripts for changes in related packages just when I thought that it would really be nice to get SpamPD (a SMTP-proxy in Perl which scans mails for spam using the Mail::SpamAssassin Perl module) restarted when SpamAssassin gets updated. Or to get spamd (which is part of SpamAssassin) restarted when a plugin is updated, removed or installed. Of course the second one is easier since the plugin package “knows” that SpamAssassin is affected by the change and can thus reload/restart spamd.
One idea I had for this is to create a hierarchy to which an admin or maintainer could add scripts (effectively providing hooks), like this:
/var/lib/dpkg/hooks/<package>/(removal update install)/(pre post)/<hook-package>
Where <hook-package> would be the package installing the hook while <package> is the package for which the action is to be “monitored”.So that I as the SpamPD maintainer could make the spampd package install a script in
/var/lib/dpkg/hooks/spamassassin/update/post/spampd which would get called whenever spamassassin is updated, after the update, to be exact.
An alternative on per-package basis would of course be:
/var/lib/<package>/hooks/(pre post)inst/<hook-package> or /var/lib/spamassassin/hooks/postinst/spampd for the above example. I would be quite intested to hear what other devs think about this.

It’s been over a year now since my last proper upload to Debian, and nine months since I announced my intent to put aside working on Debian for a while. With Matthew Garrett’s resignation from Debian, several people have compared it to my own “resignation” from Debian. It has got my thinking about whether I currently intend to ever end my “Sabbatical” and return as an active Debian Developer. I think that the end of my love-affair with Debian started at Debconf last year where several developers treated those of us who also worked on Ubuntu quite rudely. Someone was attacked for wearing an Ubuntu t-shirt at the conference, while someone else was applauded for wearing a “Fuck Ubuntu” t-shirt. That’s where I realised that maybe I didn’t have as much in common with these people as I thought I did. I still don’t understand why Debian singles Ubuntu out for this kind of treatment, we’re still the only derviative distribution that makes all of our patches to Debian available, yet Ubuntu is claimed to not do anything at all. Another example is that Ubuntu is being asked to change the Maintainer field of every package, something no other derivative is being asked to do or has ever been asked to do. Martin Krafft has had some interesting things to say about this strange relationship in the past. If that was the start of my falling out with Debian, I think that Debian considering removing documentation and firmware from the distribution, especially the documentation, was another point I started wondering whether I shared anything in common with the project anymore. Call me strange, but I think that one of the fundamental purposes of a Linux distribution is to be useful to its users. If nobody can use the distribution because it doesn’t support their hardware, and even if it did, all the documentation has been stripped out; I started to wonder what it’s aims are. It became increasingly apparent that the only users Debian was considering a priority were its own developer. And the third thing is simply a matter of Fun. Fun for me, at the moment at least, has been to build a system that fits together extremely nicely with each component doing the right thing. For Ubuntu this has meant being a driving force for it being Linux 2.6 only, with reliance on udev for hardware, etc. Upstart is just a continuation of these goals, getting a system which all “just works”, even if it means throwing out a few things people were previously fond of. All of these things would have been impossible to do in Debian itself. Getting upstart installable required changes to twelve different packages, including sysvinit itself; at a worst case, this would have required the agreement of twelve different maintainers in Debian. It’s often exhausting just persuading one of the reason for the change, persuading a dozen would have been a herculian effort. Perhaps Matthew is right, what Debian lacks is a single leadership. There’s nobody in Debian I could have gone to for approval over the changes I wanted to make, whose decision developers cannot overrule. Ubuntu has such people (the Technical Board and sabdfl), which gives the project an obvious direction instead of a couple of hundred people all pulling in different directions. In a way, I don’t feel that I’ve left Debian. I feel like I was happily going along, only to realise the mob had gone in a different direction and with no easy way of rejoining the group.

Today I reached another milestone in the development of upstart, the packages in universe can now replace the existing sysvinit package. Before trying this, make sure your installation is up to date as we’ve had to split out some parts of sysvinit into a new sysvutils package. If you’re up to date, and want to try it out, install the upstart and upstart-compat-sysv packages from universe. Note that the first reboot after you’ve installed the packages (from sysvinit to upstart) will be a little tricky … use reboot -f. If your system boots and shuts down normally, everything’s working just fine. Note that both will be somewhat more quiet than you’re used to, unless you have usplash running. Throughout the rest of this entry, I’ll try to answer some of the questions and comments that I’ve received since the last post. Events As I talked about previously, upstart is an event-based init daemon. Events are the primary force for having your services and tasks started and stopped at the appropriate time and in the appropriate order. So what are events and where do they come from? (Note that this part is under development, so may change in later releases). Events are just simple strings that may be sent by any process when something it is tracking the state of changes. They have no state or longevity, and if, when queued, they do not cause any job state changes, then they have no effect unless they are sent again. Jobs can list which events cause them to be started if they are not already running and which events cause them to be stopped if they are running. Multiple start and stop events may be listed, in which case the first to occur changes the job until the next one occurs. upstart itself generates the following system events:

• “startup”, on system boot.
• “shutdown”, when the system is about to be shut down.
• “stalled”, when there are no jobs running and no events in the queue.

The shutdown tool included in the package also causes one of the following events to be sent once the “shutdown” event has been handled:

• “reboot”,
• “halt”,
• “poweroff”,
• “maintenance” (aka. going into “single user” mode),
• any user-defined event with shutdown -e event.

Jobs also generate events whenever they change state, this is the primary source of events for ordering:

• “jobname/start”, when the job is first started.
• “jobname/started”, once the job is running.
• “jobname/stop”, when the job is first stopped.
• “jobname/stopped”, once the job has stopped.
• “jobname”, for services this is generated once it is running; for tasks this is generated once it has finished.

And as mentioned, any other process on the system may send events through the control socket or just by using initctl trigger EVENT. For now this is just the event string, however it’s intended that the event may include other details including environment variables and even file descriptors. Typical example To clarify how it all hangs together, here’s an example (using fictional names) of how the tasks and events can be arranged to provide race-free mounting of filesystems.

• “udev” service started on the “startup” event.
• udev daemon is configured to send a “new-block-device” event whenever a new block device is added to the system.
• “checkfs” task is started on the “new-block-device” event to check the filesystem.
• “mountfs” task is started when the “checkfs” task has finished and mounts the filesystem if listed in /etc/fstab
• “filesystem-mounted” event is generated whenever a filesystem is mounted.
• “fstab” task is started on the “filesystem-mounted” event, it checks the list of mounted filesystems against /etc/fstab and if all are mounted, generates the “writable-filesystem” event.
• other services and tasks would be started on the “writable-filesystem” event.

By breaking this job into these small tasks, we can see how the pieces fit together. Because everything is now done on events, there are no race conditions; we know that any filesystem listed in /etc/fstab will be checked and mounted. The only reason they wouldn’t be is if there’s an error of some kind, and that means you have larger problems anyway and the system administrator would have a shell to fix it. Of course, the moment they finish checking the filesystem and mount it, the boot process would carry on. There’s no reason that any of these events need to be generated by the upstart daemon itself, it can receive them from any other daemon on the system such as udev, acpid, etc. This keeps the focus of the init daemon narrow. A large part of the future development will be working out exactly what kinds of events we want init itself to generate, what kinds we want to come from elsewhere, and what the contents of an event can be. Getting Involved If you want to get involved with trying to nudge the direction of upstart development, you can join the upstart-devel mailing list at http://lists.netsplit.com/ Or if you just want to grab the source code, tarballs are published at http://people.ubuntu.com/\_{scott/software/upstart/ and the bzr archive is at http://bazaar.launchpad.net/\}keybuk/upstart/main

upstart is a replacement for the init daemon, the process spawned by the kernel that is responsible for starting, supervising and stopping all other processes on the system. The existing daemon is based on the one found in UNIX System V, and is thus known as sysvinit. It separates jobs into different “run levels” and can either run a job when particular run levels are entered (e.g. /etc/init.d/rc 2) or continually during a particular run level (e.g. /sbin/getty). The /etc/init.d/rc script is also based on the System V one (and is in the sysv-rc package), it simply executes the stop then start scripts found in /etc/rcN.d (where N is the run level) in numerical order. Why change it? Running a fixed set of scripts, one after the other, in a particular order has served us reasonably well until now. However as Linux has got better and better at dealing with modern computing (arguably Linux’s removable device support is better than Windows’ now) this approach has begun to have problems. The old approach works as long as you can guarantee when in the boot sequence things are available, so you can place your init script after that point and know that it will work. Typical ordering requirements are:

• Hard drive devices must have been discovered, initialised and partitions detected before we try and mount from /etc/fstab.

• Network devices must have been discovered and initialised before we try and start networking.

This worked ten years ago, why doesn’t it work now? The simple answer is that our computer has become far more flexible:

• Drives can be plugged in and removed at any point, e.g. USB drives.

• Storage buses allow more than a fixed number of drives, so they must be scanned for; this operation frequently does not block.

• To reduce power consumption, the drive may not actually be spun up until the bus scan so will not appear for an even longer time.

• Network devices can be plugged in and removed at any point.

• Firmware may need to be loaded after the device has been detected, but before it is usable by the system.

• Mounting a partition in /etc/fstab may require tools in /usr which is a network filesystem that cannot be mounted until after networking has been brought up.

We’ve been able to hack the existing system to make much of this possible, however the result is chock-full of race conditions and bugs. It was time to design a new system that can cope with all of these things without any problems. What we needed was an init system that could dynamically order the start up sequence based on the configuration and hardware found as it went along. Design of upstart upstart is an event-based init daemon; events generated by the system cause jobs to be started and running jobs to be stopped. Events can include things such as:

• the system has started,

• the root filesystem is now writable,

• a block device has been added to the system,

• a filesystem has been mounted,

• at a certain time or repeated time period,

• another job has begun running or has finished,

• a file on the disk has been modified,

• there are files in a queue directory,

• a network device has been detected,

• the default route has been added or removed.

In fact, any process on the system may send events to the init daemon over its control socket (subject to security restrictions, of course) so there is no limit. Each job has a life-cycle which is shown in the graph below: The two states shown in red (“waiting” and “running”) are rest states, normally we expect the job to remain in these states until an event comes in, at which point we need to take actual to get the job into the next state. The other states are temporary states; these allow a job to run shell script to prepare for the job itself to be run (“starting”) and clean up afterwards (“stopping”). For services that should be respawned if they terminate before an event that stops them is received, they may run shell script before the process is started again (“respawning”). Jobs leave a state because the process associated with them terminates (or gets killed) and move to the next appropriate state, following the green arrow if the job is to be started or the red arrow if it is to be stopped. When a script returns a non-zero exit status, or is killed, the job will always be stoped. When the main process terminates and the job should not be respawned, the job will also always be stopped. As already covered, events generated by the init daemon or received from other processes cause jobs to be started or stopped; also manual requests to start or stop a job may be received. The communication between the init daemon and other processes is bi-directional, so the status of jobs may be queries and even changes of state to all jobs be received. How does it differ from launchd? launchd is the replacement init system used in MacOS X developed as an “Open Source” project by Apple. For much of its life so far, the licence has actually been entirely non-free and thus it has only become recently interesting with the licence change. Much of the goal of both systems appears initially to be the same; they both start jobs based on system events, however the launchd system severly limits the events to only the following:

• system startup,

• file modified or placed in queue directory,

• particular time (cron replacement),

• connection on a particular port (inetd replacement).

Therefore it does not actually allow us to directly solve the problems we currently have; we couldn’t mount filesystems once the “filesystem checked” event has been recived, we couldn’t check filesystems when the block device is added and we certainly couldn’t start daemons once the complete filesystem (as described by /etc/fstab) is available and writable. The launchd model expects the job to “sit and wait” if it is unable to start, rather than provide a mechanism for the job to only be started when it doesn’t need to wait. Jobs that need /usr to be mounted would need to spin in a loop waiting for /usr to be available before continuing (or use a file in a tmpfs to indicate it’s available, and use that modification as the event). This is not especially surprising given that Apple have a high degree of control over both their hardware and the actual underlying operating system; they don’t need to deal with the wide array of different configurations that we have in the Linux world. Had the licence been sufficiently free at the point we began development of our own system, we would probably have extended launchd rather than implement our own. At the point Apple changed the licence, our own system was already more suitable for our purposes. How does it differ from initng? Initng by Jimmy Wennlund is another replacement init daemon intended to replace the sysvinit system used by Linux. It is a dependency-based system, where upstart is an event-based system. The notion of a dependency-based system is interesting to talk about at this point. Jobs declare dependencies on other jobs that need to happen before the job itself can be started. Starting the job causes its dependencies to be started first, and their dependencies, and so on. When jobs are stopped, if running jobs have no dependencies, they themselves can be stopped. It’s a neat solution to the problem of ordering a fixed boot sequence and the problem of keeping the number of running processes to a minimum needed. However this means that you need to have goals in mind when you boot the system, you need to have decided that you want gdm to be started in order for it, and its dependencies, to be started. Initng uses run levels to ensure this happens, where a run level is a list of goal jobs that should be running in that run level. It’s also not clear how the dependencies interact with the different types of job, a dependency on Apache would need the daemon to be running where a dependency on “checkroot” would need the script to have finished running. Upstart handles this by using different events (“apache running” vs. “checkroot stopping”). Again while interesting, Initng does not solve the problems that we wanted to solve. It can reorder a fixed set of jobs, but cannot dynamically determine the set of jobs needed for that particular boot. A typical example would be that if the only dependency on the job that configures networking is the mount network filesystems job, then should that job fail or notbe a goal (e.g. because there are no network filesystems to be mounted) the result is that network devices themselves will not be configured. You could make everything a goal, and just use the dependencies to determine the order, however this is less efficient than just ordering the existing sysv-rc scripts (which can be done at install time). Another example is that often you simply don’t know whether something is a dependency or not without reading other configuration, for example the mount network filesystems may be a dependency of everything under /usr or may just be a dependency of anything allowing the user to login if it just mounts /home. The difference in model can be summed up as “initng starts with a list of goals and works out how to get there, upstart starts with nothing and finds out where it gets to.” How does it differ from Solaris SMF? SMF is another approach to replacing init developed by Sun for the Solaris operating system. Like initng it’s a dependency-based system, so see above for the differences between those systems and upstart. SMF’s main focus is serive management; making sure that once services are running, they stay running, and allowing the system administrator to query and modify the states of jobs on the system. Upstart provides the same set of functionality in this regard, services are respawned when they fail and system administrators can at any time query the state of running services and adjust the state to their liking. Will it replace cron, inetd, etc? The goal of upstart is to replace those daemons, so that there is only one place (/etc/event.d) where system administrators need to configure when and how jobs should be run. In fact, the goal is that upstart should also replace the “run event scripts” functionality of any daemon on the system. Daemons such as acpid, apmd and Network Manager would send events to init instead of running scripts themselves with their own perculiar configuration and semantics. A system administrator who only wanted a particular daemon to be run while the computer was on AC power would simply need to edit /etc/event.d/daemon and change “on startup” to “on ac power”. What about compatibility? There’s a lot of systems administrators out there who have learned how Linux works already and will not want to learn again immediately, there’s also a large number of books that cover the existing software and won’t cover upstart for at least a couple of years. For this reason, compatibility is very important. upstart will continue to run the existing scripts for the forseeable future so that packages will not need to be updated until the author wants. Compatibility command-line tools that behave like their existing equivalents will also be implemented, a system administrator would never need to know that crontab -e is actually changing upstart jobs. Does it use D-BUS?

“To D-BUS people, every problem seems like a D-BUS problem.” —Erik Troan

The UNIX philosophy is that something should do just one job, and do it very well. upstart’s one job is starting, supervising and stopping other jobs; D-BUS’s one job is passing messages between other jobs. D-BUS does provide a mechanism for services to be activated when the first message is sent to them, thereby starting other jobs. Some people have taken this idea and extended it to suggest that all a replacement init system need do is register jobs with D-BUS and turn booting into a simple matter of message parsing. This seems wrong to me, D-BUS would need to be extended to supervise these services, provide means for them to be restarted and stopped; as well as deal with being process #1 which means cleaning up after children whose parent’s have died, etc. It seems far simpler to arrange for D-BUS to send an event to init when it needs a service to be started, and focus on being a very good message passing system. The IPC mechanism used by upstart is not currently D-BUS because of various problems, however it’s always been expected that even if init itself doesn’t communicate with D-BUS directly, there would be a D-BUS proxy that would ensure messages about all init jobs and events would be given to D-BUS and D-BUS clients could send messages to init to query and change the state of jobs. What is the implementation plan? Because this is process #1 we are changing, we want to make sure that we get it right. Therefore instead of releasing a fully-featured daemon and configuration to the world, we’re developing it in the following stages:

1. Principal development; at the end of this stage the daemon has been implemented and can manage jobs as described.
2. Replacement of /sbin/init while running the existing sysv-rc scripts. This is the shake-down test of the daemon, can it perform the same job as the existing sysvinit daemon without any regressions?
3. /etc/rcS.d scripts replaced by upstart jobs. These consitute the majority of tasks for booting the system into at least single-user mode, and contain many of the current ordering problems and race conditions. If the daemon solves the problems here, it will be a success.
4. Other daemon’s scripts replaced by upstart jobs on a package-by-package basis; this will be an ongoing effort during which upstart will continue running the existing sysv-rc scripts as well as its own jobs. During this time the event system may be tweaked to ensure it truly solves the problems we need.
5. Replcement of cron, atd, anacron and inetd. This will happen alongside the above and result in a single place to configure system jobs.
6. Modification of other daemons and processes to send events to init instead of trying to run things themselves.

The current plan is that we will be at least part of the way into stage #3 by the time edgy is released, with that release shipping with upstart as the init daemon and the most critical rcS scripts being run by it to correct the major problems For edgy+1 we hope to have completed stage #5 and be at least part of the way into the implementation of stage #6. From the start of development of edgy+2, no new packages will be accepted unless they provide upstart jobs instead of init scripts and init scripts will be considered deprecated. What state is it in now? The init daemon has been written and is able to manage jobs as described above, receiving events on the control socket to start and stop them. This has now been uploaded to the Ubuntu universe component in the upstart package for testing before it becomes the init daemon. We welcome any experienced users who want to help test this; install the package and follow the instructions in /usr/share/doc/upstart/README.Debian to add a boot option that will use upstart instead of init. If your system boots and shut downs normally (other than a slightly more verbose boot without usplash running) then it is working correctly. Other types of events will be added as required during development and testing. Currently only a basic client tool (initctl) has been written, compatibility tools such as shutdown will be written over the next week or two before it replaces our sysvinit package.

Here at Canonical Towers we have several staff who worship at the altar of Extreme Programming, and as such many of the methodologies and rituals prescribed by that religion find their way into our day-to-day working practices. A few of these came together in an interesting way a few weeks ago, and it was suggested by a cow-orker that I blog about it so he could give the URL to people. The first ritual is that of the sprint, I don’t think this is orthodox XP but rather something inspired from it that we picked up from the Python community. With all of the Canonical developers scattered across the globe mostly working from their own homes, it’s become a useful tool; particularly for the Launchpad team. The basic idea for those that haven’t heard of it is simple, and perhaps obvious; you get a selection of the team together in one place, sit them around the same table with particular goals to complete. In effect it’s highly compressed facetime and high bandwidth interaction to nail those tasks that need it, before you head off again and work at a more sedate pace. The second is directly from XP doctrine and is that peculiar observance known as pair programming, something that is often coupled with sprints. For anybody who hasn’t encountered this before, it’s something I’ve always found rather odd. You get two programmers together, both itching to code, and you take one of them’s laptop away; and you don’t just force them to fight over the keyboard either, the laptop-less soul isn’t allowed to touch it. The theory here is that it frees the deprived individual of the hassle of coding and allows them to direct the programmer in ways that may not be immediately obbious, or alternately think about the next bit of work that needs doing. Another interesting side-effect is it can be an interesting way to learn code you’ve not worked with before, if you’re the one doing the coding and you’re being guided what to do by a sage who already knows it. I have a funny story here, so I’m going to digress from the main stream for a moment to tell it. A few months ago I was working in the London office, Mark had been up all night coding and had then given me responsibility to get his changes landed onto the Launchpad mainline. I’m not that hot on quite a bit of Launchpad, and fade to utterly clueless the nearer the code gets to the web application itself; and when trying to merge the two there were conflicts which I needed Mark’s help to resolve. Now as anybody who’s met Mark knows, he’s a pretty good example of a Type-A personality, yet he suggested that we do a spot of pair programming to get the code in and I’d be the one with the laptop as it’d help me get to grips with the affected parts. This hilighted something I’ve always seen as a problem with pair programming, dealing with it when the a person with the keyboard has a totally different way of working to you. In particular, I’m an emacs user, whereas Mark is a die-hard vim user; but also right down to the working directory I would work from, how I test results, etc. When you’re the one without the laptop, this can be quite infuriating, as you know exactly how to solve a problem and the idiot with the keyboard is dithering about doing things you don’t understand just to get there. If you’re a Type-A personality, you generally snatch the laptop away at this point and do it yourself; or at least you would, if you could drive the strange editor the other person uses. I swear Mark was sulking as he gave the laptop back and let me do it. Anyway, there is some relevance to that and we’ll come to it in a moment. The third and final methodology is the concept of unit tests and more specifically test driven development. For anyone who hasn’t come across this before, I highly recommend it; I was suspicious too when Robert Collins took it upon himself to teach me the true way, but now I’m sold. Simply the idea is that if you have any code that doesn’t have another piece of code in a test suite that checks that it’s working correctly, that code is broken. More particularly unit testing involves checking just one feature or requirement at a time, and stacking them together to test all of the code paths. I’ve actually found that this improves the APIs I design, as I write the code in small blocks and functions to make them easier to test. Test driven development takes it even further, you write the unit tests first, before you write the code they’re supposed to test; usually one or two at a time. Obviously these tests will fail at first, it’s then your job to write as little code as possible to make them pass. If your code isn’t right, you add a unit test that will fail, and modify the code to make it pass. This really comes into its own when you’re fixing bugs later, once you’ve identified the bug you write a test case or two that cause the problem; these will of course fail. You can then modify the code to make them pass, and at the same time be sure you’ve not broken other functionality because of the existing test suite. I’ve also found it really useful for particularly complicated or tricky pieces of code, especially those intricate algorithms that do particularly heavy lifting. So onto the event itself, this was a sprint in our London office to indoctrinate Gustavo Niemeyer with the various projects he’d be working on. The goal for this sprint was decided to start the conversion of HCT to Bazaar-NG, and to do this Gustavo and I would pair program. Now my last experience of pair programming had been that story with Mark (see, it had some relevance) and I knew I’d be just as bad if I wasn’t allowed to have the keyboard. I was also acutely aware that if Gustavo was just sitting by and watching, he wouldn’t get much benefit from it either, so he needed to be actively involved in the coding so he could learn the things he’d be working on. I came to what I thought was a pretty neat, and obvious, solution to these problems. We set up my laptop with the code we’d be working on, and on my display were two side-by-side terminals both running screen sessions. Gustavo then set up two same-sized terminals on his, ssh’d into my laptop from both of them and joined the screen sessions. In the left-hand one he ran vi, and in the right-hand one I ran emacs. Thus we both had keyboards, and both had editors, yet could see each other working and even steal the keyboard without danger of violence. We didn’t just sit side-by-side and code on different things though, that’s not pair programming and is just ordinary programming with a bit of a voyeuristic twist. What we did was: in my terminal I started writing the test cases for the code we needed, it was Gustavo’s job to write the code that would make them pass. We actually added a third terminal in which we could run the test cases themselves; so I could run them when I’d added something that would fail, and Gustavo could run them when he thought he’d made them pass again. This turned out to be a rather fun way to work, and at one point I was almost able to try and convince myself that the code was writing itself to pass the test cases I was writing. It also got me thinking that it’d be really neat to use genetic algorithms to breed code to pass test cases.

Now that edgy has been frozen for the beta release in a week’s time, and the next developer summit, where the plans for the next release will be discussed, has been announced, I think it’s the perfect time to start thinking about the kinds of features we want to see in that release. At our recent sprint in Wiesbaden, Mark reminded us that when Ubuntu first released we were leading the field in many of the components we installed by default. Many users came to us because we provided Linux 2.6, hotplug/udev and Project Utopia by default; along with the latest GNOME release and development framework. Now everybody does that, butwe led the way. With the dapper release, we produced something that could be supported for a long time. With edgy we’ve scratched our itches and fixed things under the hood that have been causing us problems. edgy+1 seems a good time to integrate the hottest new technologies and lead the way again. So what do I think is exciting, and what do I want in edgy+1? telepathy, farsight and galago Anybody who has used the 2006 software release for the Nokia 770 Internet Tablet will have seen these in action already. They’re, in my opinion, some of the most exciting projects currently being worked on; and ripe for integration into Ubuntu. Telepathy is a communication framework built on DBus that allows any application to establish communication with users all around the world, using any protocol from IRC to MSN to VoIP. It uses Farsight to provide the codecs and protocol support on top of the (GStreamer)[http://gstreamer.freedesktop.org/] media framework. And Galago glues it all together; providing presence information about contacts and allowing users to specify their own status. This means that you’ll be able to contact your contacts. Your calendar alarm tells you that it’s a colleague’s birthday, you’d be able to click their name and see a selection of different ways that you can contact that person right now. From e-mail, opening an MSN or Jabber window or even establishing a VoIP or SIP call to them. The most exciting thing is that because every application on the desktop has easy access to this, people are coming up with fantastic ideas for using it! zeroconf By this, I mean zero configuration networking. Something the OLPC project appears to be going to do very well indeed. Where no existing network infrastructure exists, your computer should still be able to communicate with other nearby computers. It should use ad-hoc wireless networks, link-local IP addresses and server-free DNS resolution with libnss-mdns. Play games with your friends on the train without touching a button. Chat with other people in a lecture hall or at a conference where the existing network is unavailable. Hassle-free home and office networking without any complicated set up. Networking that “just works”. avahi Whether your computer is on an infrastructure or ad-hoc network, it should be able to locate resources on the network and share its own to other people. Avahi allows applications to, via DBus, do this by using a mulicast form of DNS; compatible with Apple’s Bonjour protocol. The immediately obvious benefits of this are that one can locate such things as file servers without any complicated directory; more fun things can be done too, such as sharing your music from RhythmBox with other users on the network. Combine it with telepathy, etc. and you can establish communication with anyone else, with a wide range of different protocols. The immediate problem here is that we currently have a “no open ports” policy, which if we’re going to give the users the opportunity to establish ad-hoc networks is almost certainly a very good thing. We already have exceptions for that policy for DNS and DHCP, it may be that Avahi (mDNS) joins that list. Another option is a “mobile-phone like” interface for enabling and disabling discovery and sharing. Bluetooth I don’t think we should stop discovery and sharing at 802.* networking either. Linux actually has a pretty decent bluetooth stack, it’s something that we should start taking advantage of. Your computer should be able to discover paired or known bluetooth devices when in range without intervention and communicate with them. If I attempt to establish a network connection, and my phone is nearby, that should be offered as a method for getting on the Internet. If possible, it should also be offered through telepathy as a method for contacting people in my address book! Synchronisation These technologies together give us a really big possibility, automatic synchronisation of user’s information. When I bring my laptop home, it should automatically update my desktop with any changes I’ve made to my address book or calendar, and any shared files, etc. Likewise it should be automatically be updated with any changes made on the network, to a company address book on a server, for example. And why stop there? Don’t use share your music, synchronise it so that your laptop, desktop and iPod have the same selection available at all times. Hassle-free backups is an obvious win here. But we’re stopping again. Why is my mobile phone, and PDA excluded from this? Via bluetooth, my address book and calendar should be automatically updated from these and these should be automatically updated themselves. If I add a new number to my phone’s memory, it should appear in my computer address book; and my computer should offer me the ability to contact that person (using VoIP, for example).

I just wrapped up version 0.0.8 of r (pronounced littler), put it here and sent it to Debian's incoming queue. This version has a few small changes and fixes over 0.0.7. Most notably is that all of R's environment variables are now set for r as well (at compile time). This permits it to call JGR, for example. Other than that, we made the build process more robust, extended or clarified some of the documentation and added two more example scripts.

I've been working on yet another rails project lately, and, as usual, having a lot of fun with it. I bumped into a bit of a limitation though, in the default setup. Namely, that it's difficult to pass much information between draggables and droppables. For example, if you look at the scriptaculous demo here: http://demo.script.aculo.us/shop You'll see that Rails receives and deals with the element that was dragged and dropped in sort of a hacky way, in the add method:

product_id = params[:id].split("_")[1]

In other words to get the product ID, you have to hack the string! Well, that's certainly not very elegant. I'm not sure I would consider my own solution "beautiful" either, but it does have the potential to let you pass multiple values, and do so in a way that doesn't involve chopping up strings in the controller portion of the code (which ought to be dealing with logic, not how to pick strings apart). Most importantly, you can pass values that are decoupled from the html element id used as a draggable element.

def draggable_dest()
  return "'sourceid=' + element.sourceid + '&sourceobj=' + element.sourceobj"
end
def draggable_source(htmlid, obj)
  javascript_tag "e = $('# htmlid ') ; e.sourceid = '# htmlid '; e.sourceobj = # obj.id "
end

I created these in the application_helper.rb file, so that I can use them whereever I need. They are used like so:

<%= drop_receiving_element('current', :url =>   :action => "replace_current",
                         :destid => 'current', :destobj => @current.id  ,
                         :with => draggable_dest()) %>

This sets up the action of the receiving element with the destination's html element id, and an ActiveRecord object id, as well as the :with tag that knows how to grab the information we pass from the draggable element:

<%= draggable_element("mytask", :revert => true) %>
<%= draggable_source("mytask", t) %>

What's happening here is that first, we create a draggable element with the regular Rails method, but then we apply the secret sauce, defined above, that loads the html element id, and the object id into the draggable element, so that when it lands on the receiving element, the parameters we set up get passed on. This is a rough idea at this point, as I only figured out how to do what I wanted yesterday. Potential improvements include passing arbitrary parameters, as well as integrating the whole thing somewhere "upstream", which is something I wanted to avoid for my initial attempt, so as not to lock myself into a hacked, local version of Rails. In any case, I hope you enjoy it.

Are blogs and conference speeches inherently about self promotion? If so is that a bad thing?

Recently I mentioned my Planet configuration on a mailing list where most people don't track new technology. Some people viewed blog entries for the first time as a result of this and then claimed that blogs appeared to be mostly about self-promotion.

It seems to me that people offer conference speeches to promote technology that they believe in (which often equates to self-promotion as such people are well known to be associated with the technology in question), for promoting themself or their own business, or to promote the company that employes them. Of these categories of talk the worst ones are those which are given to promote a company, it's especially bad when a talk starts with "the guy who was supposed to give this talk was called to a client so I'm doing it instead" - this indicates how much the company cares for the quality of the talk. Someone who is promoting themself will care about doing a reasonable job. Someone who promotes their favourite technology will usually give a great talk! But corporations rarely get the idea that a good quality talk which makes little mention of their products is the most effective advertisement. Of recent times Google seems to be the best example of a company which gets this idea, at many conferences there are Google employees giving talks about various technologies not directly related to Google operations without any direct sales pitch. Everyone who attends such talks gets the message - Google has hired many smart people and has them working on cool things.

It seems that blogs are often written with similar motivations to conference presentations but with no control over the topic and less quality control. The difference of course is that a blog doesn't get a forum the way a talk which is accepted by a conference will. So a corporate blog has to be really good to get readers.

PS I've added the day15 beard picture to this entry, it was taken on the 22nd of October, but I had only just got around to GIMPing it and uploading it.

A while ago I read Steve Yegge’s rant about Agile development, though I’ve forgotten who linked to it. The thing that struck me as interesting was the bit about “work queues”:

With a priority queue, you have a dumping-ground for any and all ideas (and bugs) that people suggest as the project unfolds. No engineer is ever idle, unless the queue is empty, which by definition means the project has launched. Tasks can be suspended and resumed simply by putting them back in the queue with appropriate notes or documentation. You always know how much work is left, and if you like, you can make time estimates based on the remaining tasks. You can examine closed work items to infer anything from bug regression rates to (if you like) individual productivity. You can see which tasks are often passed over, which can help you discover root causes of pain in the organization. A work queue is completely transparent, so there is minimal risk of accidental duplication of work.

Sadly, googling for “work queue” doesn’t come up with any sort of todo list stuff, but rather a multiprocessing scheduling tool which is cool, but not immediately relevant for me. As far as I can see, whatever was actually being talked about either isn’t public, or is more of a concept than an actual tool. The only Google Todo thing I could find was an applet thing for the personalised google homepage, which just lets you make todo items and set them as high/medium/low priority. And while that might be all that Steve Yegge was talking about, it doesn’t really feel terribly inspiring to me. I guess what I’d really like is to have todo items get assigned to a project (so that I can ignore all the todos for projects I don’t want to worry about atm), and also to be able to give them a deadline (so I can treat them with a bit more urgency when necessary) and a priority (so that I can easily spot things I’m willing to defer or ignore completely when I find I don’t have time to do everything I’d like). I suspect I’ll probably just stick with writing notes in vi to keep track of things, though, same as I have been for years.

Last week, the Free Standards Group launched the LSB Developer Network, a community resource for developers building portable Linux applications via the LSB (read the press release and the introduction from LDN editor Martin Streicher). The LSB Developer Network (or LDN for short) aims to provide a well known starting point for developers looking to target the variety of Linux distributions available today without requiring a separate version for each distribution. It is the perfect complement to the LSB, which already provides a “highest common denominator” across the major distributions; and LSB Certification, which allows ISVs to indicate that their products will work with LSB Certified distributions. The big themes of LDN are “decentralized” and “bottom up”—to paraphrase Darrin Thompson, it’s a developer network that’s actually a network. In other words, we’re not taking the usual path of hiring a bunch of people to construct a centralized “network” from the top down—after all, Linux is a fundamentally decentralized phenomenon so shouldn’t a developer network for Linux be decentralized as well? Like the LSB, we see LDN as essential to the ultimate success of the Linux platform, an answer to similar programs for the centralized, proprietary platforms Linux competes with but built using the very techniques that make Linux what it is. Our fundamental belief is that “decentralized” doesn’t have to mean “fragmented”, and LSB and LDN are both key steps toward that end goal. We’re off to a good start too: LDN has the backing of a broad range of Linux platform stakeholders ranging from distribution vendors (Novell, Red Hat, Ubuntu) to OEMs (HP, IBM) and ISVs (MySQL, RealNetworks) to tools and content providers (O’Reilly, SlickEdit). One upcoming feature I’m particularly excited about is the integration of Safari Books Online. Soon, you’ll be able to type, for example, a function name into the LDN search box and get not just results from the link directory but also results from O’Reilly titles and others you know and love. We’re moving rapidly to add community content as well, including man pages and canonical reference documentation; and we’ll be hooking LDN into the LSB database, so you’ll be able to query the status of interfaces too, including whether or not those interfaces are in the LSB and what their status is in each of the major Linux distributions, to help make your applications as portable as possible. If you’re a Linux developer, and particularly if you use del.icio.us to bookmark resources on the web related to software development on Linux, I urge you to contribute to the LSB Developer Network. It’s easy: Simply create an account and link your account to your del.icio.us account (you don’t need to provide your del.icio.us password, just your username), and as you bookmark, tag, and annotate pages, your bookmarks, tags, and annotations will be pulled into the LDN directory. Help us build a developer network for Linux, bottom-up style!

The blog post Drupal vs. Joomla - Fight! pointed me to a discussion on the drupal-devel mailing list about the ohloh.net website which tries to gather some statistics/metrics about the code of Free Software projects. Their slogan:

Explore Open Source Mapping the open source world by collecting
objective information on open source projects.

You might want to read their Terms of Use, though, which don't sound very open... Anyway, their stats about Drupal and about Joomla! are flawed (for example) because they seem to include the whole contrib CVS tree of Drupal (not just the core Drupal), which is huuuge. But there's no need to use any fancy website anyway, there are Free Software tools out there which can produce some metrics, too. The following data is generated using David A. Wheeler's SLOCCount (for the respective tarballs of the current stable releases): Drupal 4.7.4

Total Physical Source Lines of Code (SLOC)                = 8,012
Development Effort Estimate, Person-Years (Person-Months) = 1.78 (21.34)
 (Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
Schedule Estimate, Years (Months)                         = 0.67 (8.00)
 (Basic COCOMO model, Months = 2.5 * (person-months**0.38))
Estimated Average Number of Developers (Effort/Schedule)  = 2.67
Total Estimated Cost to Develop                           = $ 240,198
 (average salary = $56,286/year, overhead = 2.40).

Joomla 1.0.11

Total Physical Source Lines of Code (SLOC)                = 65,880
Development Effort Estimate, Person-Years (Person-Months) = 16.25 (194.94)
 (Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
Schedule Estimate, Years (Months)                         = 1.54 (18.54)
 (Basic COCOMO model, Months = 2.5 * (person-months**0.38))
Estimated Average Number of Developers (Effort/Schedule)  = 10.51
Total Estimated Cost to Develop                           = $ 2,194,486
 (average salary = $56,286/year, overhead = 2.40).

Not that I think these numbers mean anything ;-) Lines of Code is a very, very unreliable indicator for code quality (or anything else, for that matter)...

Just a few weeks ago, the world heard the news of the tragic school shooting at an Amish school in rural Pennsylvania. A deranged man entered the schoolhouse, bound and gagged female hostages, brought along torture equipment, and shot 10 of them. 5 died, and the remaining 5 are believed to still be hospitalized.

Back in 1990, a deranged man committed a series of murders near the University of Florida campus in Gainesville. The story mentions 5 people that were killed.

Both were tragic situations. Both men killed people that had their whole lives in front of them. Both shook an entire community.

But look at how the communities responded. The Amish responded like this:

CNN reported a grandfather of one of the murdered Amish girls said of the killer on the day of the murder: "We must not think evil of this man."

Jack Meyer, a member of the Brethren community living near the Amish in Lancaster County, explained: "I don't think there's anybody here that wants to do anything but forgive and not only reach out to those who have suffered a loss in that way but to reach out to the family of the man who committed these acts," he told CNN.

The Amish have reached out to Roberts' family. Dwight Lefever, a Roberts family spokesman said an Amish neighbor comforted the Roberts family hours after the shooting and extended forgiveness to them.

An article in a Canadian newspaper the National Post stated that the Amish have set up a charitable fund for the family of the shooter. (Wikipedia)

In addition, the Amish invited the Roberts family to attend the funerals for the Amish girls he killed.

Gainesville reacted this way:

Dianna Hoyt, Christa Hoyt's stepmother, said Rolling's execution has been eagerly awaited by the victims' families. Some will be inside the prison to witness it. . .

Sadie Darnell, who was the police department's media spokeswoman at the time and developed enduring friendships with the victims' families, said Rolling's execution still matters, even if it also provides him more of the notoriety he sought.

"Retribution . . . is important because it represents that our society is holding that person accountable," said Darnell, now a candidate for Alachua County sheriff. (CNN)

We've all heard of murders that have taken place lately. Usually they are accompanied by calls by politicians, victim's family, and sometimes even clergy to kill the perpetrator. In the days after 9/11, there were reports of anybody that looked Middle Eastern being attacked in several different places around the country.

I have never understood this great desire for revenge. How does that help anyone?

What the Amish did was right religiously and morally. They truly followed the New Testament call to love your enemies and forgive. It is not easy to follow all of Jesus' teachings, and nobody said it would be. But they are doing it, and they have already begun healing. Reports are that the Roberts family has become friends with several of the Amish in the area, and they are working to help each other out after this horrible tragedy.

Even putting religion aside for a moment, the Amish actions are quite simply the right thing to do. By spreading love instead of hate, and friendship instead of revenge, they have succeeded in making sure that no cycle of violence starts there.

In contrast, 16 years later, the families of the victims in Florida still aren't healing. They are still angry and bitter. They are still seeking revenge. They hope that their lives will get back to normal after the murderer is killed. But after 16 years of stewing about it, will they really? And what about the family of the murderer, whose lives certainly must have been a mess for the past 16 years? They will now lose a family member. Does anyone care about them, or will they now turn angry at society and possibly spread the pain more?

Imagine what would happen if so many more people around the world took the Amish perspective -- to forgive those that wronged us. How long must it be before we can forgive? How far back do we spread our hate? Do we still hate those that were involved in 9/11, or can we forgive them? Do we still hate the Germans for what their ancestors did in World War II, or can we forgive them? Do we hate politicians with whom we strongly disagree, or think are liars? Do we still hate all those that have wronged us personally -- someone that stole something from us or the sadistic boss?

Knives, electric chairs, and bombs do not buy reconciliation. They can not "win over" the hearts of others. They do not make our lives easier. Hate brings more hate, and more resentment.

Forgiveness is not easy. We all hope that we will never be involved in such a tragedies as these. But let us follow the example the Amish have shown -- forgive for all things, big or small, important or not, painful or not.

Only then will we be at peace with ourselves, and only then will we have the chance to be at peace with our neighbors.

This morning I got back to an article that has been on and off my work-plate for a year now, one that I have to turn in to a discussant for my 4S panel as well for an edited collection of articles on the intersection between art, activism, and biopoltics. Part of my efforts meant attacking a virtual “stack” of articles and one of them was one of best journalistic articles I have read in a long time. Exceedingly clear prose is combined with good references, hard numbers and just the right amount of passionate verve to make reading actually fun and not just another nameless, faceless cog in the academic research wheel. The article mentioned a couple of interesting studies that while sadly not linked from the article (but I am going to take that was an editorial and not authorial decision), they were pretty much cake to find by doing a little poking and prodding on the Internet. So for example, one of them was published on PLoS Serotonin and Depression: A Disconnect between the Advertisements and the Scientific Literature and stands pretty strongly on its own right. But what is even better are the long list of citations (many with links) that would otherwise be frankly, a *total* time consuming bitch to find like the correspondence between the FDA and pharmaceutical companies: # Food and Drug Administration Division of Drug Marketing Advertising and Communications (1997) Effexor warning letter. Rockville (Maryland): Food and Drug Administration. Available: http://www.fda.gov/cder/warn/june97/effexor.pdf. Accessed 14 October 2005. Total gems given to you right there on a silver plate. That is what I love about Internet based research… On the other hand, this extreme access is not always so peachy. The amount of data now available can be experienced as a totally mind-numbing, frightening, chaotic infolanche (as one of my old adivsors liked to call it), which translates into a lot of academic hunting and gathering not to mention all the sorting, sifting, cataloging, and then of course trying to remember what the heck you have just amassed in the first place, even with the aid of tags and all I am not sure if things have gotten easier or harder because this is all I personally know but what I do know is that there are times when I just love the hunt, losing myself for hours, following links, gathering articles, enlivened, exicted about research and there are others when the woolly mammoth of infomartion gets the better of me.