Very quietly, the Ubuntu community reached a major milestone today when we held a Community Council meeting, like it does fortnightly. The only thing different was that the council included five new members -- Mike Basinger, Corey Burger, Matthew East, Jerome S. Gotangco and Daniel Holbach. These members are, with the exception of Holbach, not employed by Canonical and were each confirmed by a vote of the full Ubuntu membership. Before the recent elections, I was the only member who was not a Canonical employee -- and I used to be one. From a technical perspective, the founding Ubuntu team was able to benefit from everything that Debian had built -- a running start if there ever was one. From a community perspective though, we had to start from scratch and had to deal with the very difficult situation that paid labor and closely entangled corporate interests. Working with the rest of the team, I drafted a set of community norms (the Code of Conduct) and governance structures designed to keep both the community and Canonical under control. They seemed like good ideas but, because we didn't have a community yet, only reflected the sensibilities of Mark Shuttleworth, myself, and the rest of the early Ubuntu team. The highest Ubuntu governance board, the Community Council was initially filled with people that were in the room in Oxford when we came up with the idea: myself, Mark, James Troup, and Colin Watson. We decided that the council members should, and would, be approved by a vote of the membership. With no members though, we faced a bit of a bootstrapping problem. Three years later, Ubuntu has a vibrant community with hundreds of enfranchised members who have an up-or-down say on the members of the council itself. When we looked for new potential council members to propose to the community, we tried to pick the most active, most level-headed, and most representative group we could find. It was pleasing to see that only one member of the new CC board works for Canonical; Canonical employees are now outnumbered. It has been interesting to see announcements by Fedora, FreeSpire, OpenSuSE over the last few years proposing systems of more inclusive community governance structures that, perhaps not entirely coincidentally, look a bit like what Ubuntu has built in its attempts to empower users in that sometimes awkward community/company environment. Whatever the reasons, I think it means there's more pressure on us at Ubuntu to keep raising the bar. I see today as a great example of how we've done just that.

A few months ago, my good friend Kris gave me an OldWorld Powermac (a model 8500/150, even though the processor really runs at 120Mhz) as a donation to the Debian project. I arranged to hand it to Sven Luther, who is very much into Debian on PowerPC, at FOSDEM in Brussels. Due to some misunderstandings, however, Sven did not take the mac with him; so it had been left at the office. And since I don't go to an awful lot of meetings (FOSDEM is probably going to be the only one this year), the mac is likely going to stay at our office. So I thought I'd put it to some good use. I saw last week that the PowerPC dailies hadn't been built for about a week or so. When I asked around, I found out this was because Colin Watson, who usually builds them, does so on his laptop. Since he was not directly available, however (due to him being on vacation or something similar), they were not getting built for about a week. An understandable situation, but rather suboptimal. So, since I had this unused PowerPC machine anyway, I installed Debian on it, called it "ragtime" in accordance with my usual machine naming scheme, did a checkout of the d-i subversion tree, and added the daily-build script to cron. As of yesterday, Frans Pop changed the different configuration items to point to my dailies instead of Colin's; so I guess it's now rather official. Ish. Update: NOW. Not NOT. Aargh.

More on the Google Summer of Code: as well as the project I'm mentoring for Debian, I'm mentoring Evan Dandrea (no blog yet?), writing a migration assistant for Ubiquity. I haven't talked much about Ubiquity, mostly because I've been far too busy writing it. Ubuntu has needed an installer for its live CD for a while, partly because, well, loads of users want it, and partly because it will cut Canonical's costs quite a lot if we only have to send out half the number of CDs in shipit (apparently single-CD packaging is significantly cheaper than double-CD packaging). I'd been resisting doing something from scratch because I love d-i and I think it would be a really bad idea to end up maintaining two installer implementations from the ground up (live CDs are nice, but they don't cut it for everyone). So when I was given the task of doing a shiny live CD installer with a custom-designed UI, while I started with a more-or-less from-scratch implementation put together by Guadalinex (thanks!), I fairly quickly diverged from that and morphed it into something that uses d-i code for as much of the backend operation and logic as it sensibly can. I'd call it a sort of debconf frontend except that its design is almost opposite to how a debconf frontend should work, in that it's highly specialised to react to particular question names. This had a lot of advantages in terms of being fairly quick to write, although in the long term I think I might prefer something closer to cdebconf plugins for the job; we'll see how things turn out. Evan wanted to write an extension to this to automatically migrate settings and documents from an existing Windows installation, which I think would be an absolutely excellent thing to have: automatic migration is a real killer feature in an installer. In fact, d-i already has a start at this, namely os-prober, which in conjunction with some bootloader installer code magically sets up boot menu entries for other operating systems on your disk. So, I suggested to Evan that he might want to put most of the clever logic in a udeb, so that it can be used in d-i too, and to my relief he seemed quite enthused by the idea. He's starting on the preliminary work now and I look forward to seeing his progress. Best of luck, Evan!

I'm mentoring Matheus Morais in the Google Summer of Code, porting d-i to the Hurd. We've exchanged a few mails and he has in hand all the preliminary (but not yet functional; wouldn't want to make it too easy :-)) patches I've put together in the past. I think I should be reasonably well-placed to judge his progress. Best of luck, Matheus!

I'm mentoring Matheus Morais in the Google Summer of Code, porting d-i to the Hurd. We've exchanged a few mails and he has in hand all the preliminary (but not yet functional; wouldn't want to make it too easy :-)) patches I've put together in the past. I think I should be reasonably well-placed to judge his progress. Best of luck, Matheus!

Following up on my earlier post:

• Mark Hillebrand pointed me to some prior art on vidir, his mvedit is the same idea, with a different set of features including recursive directory lists and file name swapping, but not deletion.
• I noticed Mark also wrote a matchedit which "allows editing of regions of files that match a regexp in a single session"
• In the meantime, I've updated vidir, adding support for file swapping.
• Enrico Zini pointed out that his tagcoll is a true unix tool, that can process its own output and can probably be used for all sorts of things besides working with debtags. A good example of how to generalize an originally very specific problem.
• Colin Watson wrote about sponge, a tool no more complex than cat, that I am sure I have written in a line of perl at least twice, but never thought to save, and whose absence from the standard toolkit I've probably worked around hundreds of times.

This has only whetted my interest. Maybe the problem isn't that no-one is writing them, or that the unix toolspace is covered except for specialised tools, but that the most basic tools fall through the cracks and are never noticed by people who could benefit from them. So, to provide some more incentive, I'd like to start a collection of new unix tools. I've worked on this kind of thing before with my collection of biff-type filters and Debian's devscripts. Sometimes just bundling up a set of programs consistently can add to their value. I think it can here too, once a critical mass of tools is assembled. So, write more unix tools, and tell me about them so I can collect them into a package.

Joey writes about the lack of new tools that fit into the Unix philosophy. My favourite of such things I've written is sponge. It addresses the problem of editing files in-place with Unix tools, namely that if you just redirect output to the file you're trying to edit then the redirection takes effect (clobbering the contents of the file) before the first command in the pipeline gets round to reading from the file. Switches like sed -i and perl -i work around this, but not every command you might want to use in a pipeline has such an option, and you can't use that approach with multiple-command pipelines anyway. I normally use sponge a bit like this:

sed '...' file   grep '...'   sponge file

Since it's so trivial I imagine lots of other people have written something similar (another common name for it seems to be inplace; my name indicates soaking up all the input and then squeezing it all out again); but I do keep meaning to try to get a rewritten version into coreutils at some point.

Joey has been campaigning for a while to get everything in the archive changed to depend on debconf debconf-2.0 or similar rather than just debconf, in order that we can start rolling out cdebconf as its replacement. Like most jobs that involve touching the bulk of the archive, this looks set to take quite a while, as the list of bugs should indicate. Recently it occurred to me that we didn't necessarily have to do it that way round. In a bout of late-night hacking while staying awake to look after a sick child (he seems mostly OK now, although the rushed trip to the hospital earlier was a bit on the nerve-wracking side), I've shuffled things around in the cdebconf package so that it no longer has any file conflicts with debconf or debconf-doc, and changed the debconf confmodule to fire up the cdebconf frontend rather than its own if the DEBCONF_USE_CDEBCONF environment variable is non-empty. (The details of this may change before it actually gets uploaded, as I'd like to get Joey to look it over and approve it first.) This allows you to install cdebconf, set that environment variable, and play around with cdebconf with relative ease; when we come to switch to cdebconf for real, instead of a huge conflicting mess that apt will probably have trouble resolving, it'll just be a matter of changing a couple of lines in /usr/share/debconf/confmodule. Of course, don't expect cdebconf to be a complete working replacement for debconf just yet; if you try using it for a dist-upgrade run it'll fall over. Due to its d-i heritage, it doesn't yet load templates automatically; that has to be done by hand. Frontend names differ from debconf's, which will need some migration code. At the moment it can only handle UTF-8 templates, which are mandated in the installer but only optional in the rest of the system. It doesn't have all of debconf's rich array of database modules. I haven't adapted the Perl or Python confmodules yet. The list goes on. However, I think we at least stand a chance of getting a handle on the problem now. (I'll post this article to debian-devel once the changes have been reviewed and uploaded.)

Working on free software has made me fairly revision control system-agnostic; I can't afford to get too wedded to any one system because as soon as I do somebody will invent something new and I'll have to convert again, so I just work with whatever other people on the same project are using. Even CVS doesn't make a lot of difference to the way I work as long as I'm working online and have cvsps handy. And of course I usually don't bother with revision control if I'm just tweaking somebody else's Debian source package a bit (in which case I just use debdiff for paranoia). Using bzr at work, though, I think I just found my killer app in Michael Ellerman's shelve plugin. My working style generally involves alternating between doing lots and lots of stuff in the one working copy and (after testing) going through and committing it in logical chunks. This is fine if everything's in separate files (most revision control systems let you commit just some files), but if several of the chunks are in the one file then I'm reduced to saving diffs and manually editing out the bits I don't want to commit yet, which is obviously pretty tedious and error-prone. bzr shelve presents each diff hunk in your working copy to you in turn and asks you whether you want to keep it. If you say no, that hunk gets unapplied and goes into a "shelf", where bzr unshelve can later reapply it. In the meantime commits act as though the shelved hunks didn't exist. This doesn't help if you want to defer only one of two immediately adjacent changes that end up in the same hunk, of course, but it vastly reduces the scale of the problem. I suppose it would be easy enough to write a shelve-a-like for any other system; it's just that I haven't seen it for any other system yet. If working with systems that lack it really starts to annoy me, I may have to rip out the guts of shelve and figure out how to make it generic.

Sometimes following up on a bug takes you a lot further than you expected. Debian bug #337041 looked like it was going to be fairly straightforward once I upgraded coreutils to figure out what the new IUTF8 flag actually did, since the SSH2 protocol already supports transferring termios flags around. Unfortunately, since IUTF8 is relatively new, it doesn't have a number assigned in the draft connection protocol. Moreover, that Internet-Draft is in the last stages before becoming an RFC and can't be modified any more, and it doesn't include any facility for private-use extensions. D'oh. To add further complication, since IUTF8 is Linux-specific, it's not hard to imagine that some other OS might introduce something with the same name but subtly different semantics, and so the SSH protocols can't just defer to POSIX for the definition but instead have to spell out exactly what they mean. As a result of all of this, it looks like the best way to make progress might be for me to write an I-D myself that creates a channel extension to set or clear IUTF8, and attempt to enlist support from some upstream implementors. I didn't expect bug triage to lead me into the Internet standardisation process quite so quickly!

Hot on the heels of Joey's tale of getting rid of base-config (the second stage of the installer) in Debian, we've now pretty much got rid of it in Ubuntu Dapper too. The upshot of this is that rather than asking a bunch of questions, installing the base system, and rebooting to install everything else, we now just install everything in one go and reboot into a completed system. This does mean that, if your system doesn't boot, you don't get to find out about it for a bit longer. However, it has lots of advantages in terms of speed (the much-maligned archive-copier mostly goes away), reducing code duplication (base-config had a bunch of infrastructure of its own which was done better in the core installer anyway), comprehensibility, and killing off some annoying bugs like #13561 (duplicate mirror questions in netboot installs), #15213 (second stage hangs if you skip archive-copier in the first stage), and #19571 (kernel messages scribble over base-config's UI). To go with Joey's Debian timeline, the Ubuntu history looks a bit like this:

• 2004 (Jul): First base-config modifications for Ubuntu; we need to install the default desktop rather than dropping into tasksel.
• 2004 (Aug): Mark phones me up and asks if I can make the installer not need the CD in the second stage by copying all the packages across beforehand. Although it's a bit awkward, I can see the UI advantages in that, so I write archive-copier at the Canonical conference in Oxford.
• 2004 (Sep): Mark asks me if we can ask the timezone, user/password, and apt configuration questions before the first reboot. With less than a month to go until our first release, I have a heart-attack at how much needs to be done, and it eventually gets deferred to Hoary.
• 2005 (Jan): Matt fixes up debconf's passthrough frontend for use on the live CD, and we realise that this is an obvious way to run bits of base-config before the first reboot. It's rather messy and takes until March or so before it really works right, but we get there in the end.
• 2005 (Apr): I get "put a progress bar in front of the dpkg output in the second stage" as a goal for Breezy. Naïvely, I think it's a simple matter of programming, since I'd already done something similar for debootstrap and base-installer the previous year.
• 2005 (May): I hack progress bar support into debconf. Nothing actually uses it for anything yet, except as a convenient passthrough stub.
• 2005 (Jul/Aug): I actually try to implement the second-stage progress bar and realise that it's about an order of magnitude harder than I thought, requiring a whole load of extra infrastructure in apt. Fortunately Michael Vogt saves the day here by writing lots of working code, and the progress bar works by early August.
• 2005 (Sep-Dec): Upstream d-i development ramps back up again, with tzsetup, clock-setup, apt-setup, and user-setup all being cranked out in short order and the corresponding pieces removed from base-config. I merge these as they mature, and manage to get agreement on including the Ubuntu debconf template changes in upstream apt-setup, which helps the diff size a lot.
• 2005 (Nov/Dec): Joey and I chat one evening about the Ubuntu second-stage progress bar work, and we end up designing and writing debconf-apt-progress based on its ideas, after which Joey knocks up pkgsel in no time flat.
• 2006 (Jan): The rest of the pieces land in Ubuntu, and we drop base-config out of the installer. To my surprise, nearly everything still just works.

Although it caused some friction, I'm glad that we did the first cuts of many of these things outside Debian and got to try things out before landing version-2-quality code in Debian. The end result is much nicer than the intermediate ones ever were.

Over the past months we've been working on a big change in the debian installer, removing base-config from the installation process. Doing this has required a great many changes, some of them user-visible. The installer now configures the timezone, users and passwords, and apt all in the first stage instead of after rebooting. Some preseed values have been removed or changed, including base-config/early_command and base-config/late_command. The biggest change of all, the the hardest to get working, has been moving of the selection of tasks and the installation of all packages into the first stage of the installer. Now tasksel pops up a debconf question that looks like any other question d-i asks, and the installation of packages is hidden behind a nice clean progress bar, and if any packages use debconf to ask questions, d-i will display those questions using its frontend too. I'd include a screenshot, but really, it's just another progress bar, how boring is that? By the way, if you maintain a debian package and it prompts without using debconf (when debconf is available), then your package will obviously break this, and it's well past time to fix it. I think that all the packages d-i installs do use debconf except for possibly a few like libc during upgrades. Upgrades are theoretically possible at this step for any packages debootstrap installs, so those will need to be fixed too. This has been a long, long time coming. Some milestones include:

• 1998: Wichert Akkerman and others do the design of debconf, and first draft of its specification. Without debconf, none of this would be possible.
• 1999: At linuxworld, I see Stormix's Debian-based install. I remember discussing with them about how debconf could be used to clean up the install process. I don't remember what kind of hack they used to avoid the long dpkg log interspersed with the occsional question that a Debian install involved back then.
• 1999: I begin writing the debconf program. And promoting its use.
• 2000: Randolph Chung begins writing cdebconf; d-i begins.
• 2000: Randolph writes the passthrough frontend for debconf. At the time this was expected to be used by tools like aptitude (deity?) to integrate debconf into their GUIs, but it mostly just rotted for a long time unused. Looking back this was a contribution of unexpected significance.
• 2001: I visit Progeny, they show me their implementation of a progress bar running over a Debian installation. It's a special-purpose hack, but it works.
• 2001: debconf enters Debian policy
• 2003: debconf becomes the policy-accepted standard way to do installation prompting
• 2003: Adam Heath adds --status-fd to dpkg.
• 2005 (Jan): Matt Zimmerman dusts of the passthrough frontend and makes it usable to pass debconf questions from debconf running in a chroot up to cdebconf running in d-i.
• 2005 (Jan): Michael Vogt makes apt support status fds too.
• 2005 (Fall): Ubuntu pulls most of the above peices together and fixes some bugs, gets a progress bar running over apt with debconf passthrough.
• 2005 (Dec): Colin Watson and I develop debconf-apt-progress, rest of the pieces fall into place for using this stuff in Debian.

So yeah, this has been in er, progress for 8 years, and at least three Debian derived distributions have come up with thier own approaches in between with only the last one being quite similar to the end result, and the other two being rather dead. I don't know whether this is a study in how Debian is slow, or a study in how we do eventually come up with infrastructure that is done right and ends up being used by everyone. Or both.

Two Sundays ago, Jim and I walked down to Ubuntu Love Day. The Ubuntu Below Zero conference is being held at the Holiday Inn in Chinatown. That's the one with the fake Chinese tiles on the roof, which is sort of tacky. We found our way downstairs to the conference room and sat down. The previous night's revelries still had quite the effect, so I'm afraid I drifted in and out while listening to people speak. I thought I had rid myself of this habit, but apparently university has trained me to consider digital projectors as sleeping aids. I'll have to do something about this one day. I got to see some fairly interesting people: Mattias Klaus, whom I met earlier at Jeff Bailey's place; Fabio Di Nitto, who is incredibly jolly; Chris Peterman, who was with his classmates from Clarkson; Colin Watson, whom I thanked for working so hard on Debian; Jeff Waugh, who dressed up as an inmate; and plenty of others. Oh, I managed to meet Andrew Mitchell in person, but we failed to have a proper CHUG meeting while we were all in town. We broke for lunch, which was held in the hotel restaurant. A buffet, with some Chinese-style dishes, was being served. There, we had to negotiate with the unilingual (Cantonese) staff to find out what Jeff could eat. We had a special order of plain white rice brought for him and we were informed that it was safe for him to eat the stir-fried vegetables. I suggested to Jeff that he ought to keep a phrasebook on him whenever he travels. On each page, one per language, there should be a written description of what he can and cannot eat. Then, he just has to show this to waiters who don't speak English or French. For reference, the characters that mean "vegetarian" in Chinese are and . Most of the time, they mean "vegan", but there are some places where eggs and oysters are considered vegetarian. After lunch, Mark Shuttleworth gave Jeff a very nice birthday speech. He also gave Jeff a nice bottle of wine, which was very nice. And Matthias came up to give Jeff a bottle of fish-oil pills, which was very odd. Then, there were two Birds Of a Feather sessions, involving how to make an Ubuntu package and how to be an Ubuntu advocate. Being upstream to both these issues, I sat in the middle of the room and listened to both presentations at once. This was quite interesting, and slightly taxing. After this was done, we broke up once more. The plan for the evening was to meet for dinner in the hotel lobby. This transformed to meeting outside the front door. Jim, wlach, and I showed up and were greeted by this mass of people. Since there were stragglers and I knew where dinner was to be held, Jeff took a group of nineteen people in the metro. I followed about ten minutes later with a group of eighteen. We met up on St-Denis, and by this time we had lost some of the more impatient, and more hungry, people. Eventually, a group of twenty-three were milling about outside Le Nil Bleu. Angie had made arrangements for us to be served a set meal, and they had agreed to seat twenty-five of us. I went inside to let the staff know that we had arrived. When 20:30 came around, we were let into the restaurant and seated. I think the kitchen was still preparing food, because we had to wait about an hour before being served. Unlike my previous experience there with Jeff, the food was quite good this time and the service was very attentive. We enjoyed ourselves very much, and Jeff seemed to be having a very excellent birthday dinner. Some of the guests were not so happy, because the food was a little spicy for their European palates. Since gratuities are not standardised around the world, we discovered that some people had not tipped. To cover the embarassment, Angie and I made sure that the wait-staff were adequately compensated for their efforts. I'll be sure to go back to this restaurant again, but with a more significant crowd to ensure good service. After eating, a group of us including Matthias and Tollef went up to B l K , which was just a little north on Mont-Royal. That's a really interesting bar with beautiful tiled floors and ostrich heads on the walls. We enjoyed the lovely music, had a very good conversation there, and laughed until we were all exhausted. Since the Canonical people had to be awake for the 09:00 keynote speech the next day, they left early. Jim and I bid goodnight to them before we staggered home.

So this week’s project was working on dak, in particular getting the tiffani implemention included. What’s tiffani, you ask? It lets you just download the changes to Sources.gz and Packages.gz files, instead of the whole damn thing – if you have main, contrib, and non-free for unstable in your sources.list, this means your apt-get update only needs to download 27.5 KiB instead of 2.85 MiB (for bz2) or 3.75 MiB (for gz). Two orders of magnitude saving’s not bad. I’ve mentioned this previously a few times if you want to see the history, in January I noted some quick hacks which have now been obsoleted by Michael Vogt’s integration into apt (which is in a personal repository, not uploaded to unstable yet…), back in August last year I did a little promotion of the concept, referring back to my post in December 2003 that summarised the analysis done previously, which takes us back to March and April 2002. The server-side implementation, tiffani, is thus a little python script that groks through the list of suites and components dak knows abouts, looks for Packages, Sources and Contents files, and creates a directory names something like Packages.diff which in turn contains an Index file and a number of gzipped patches, named after the time they were created. The process for updating an existing Packages file is to look for the matching SHA1-History entry in the Index file, download the corresponding patch, apply that patch to your existing Packages file, and repeat until your Packages file matches the one you were looking for (which you can determine by looking at the checksum in either the Release or Index file). Anyway, tiffani’s been co-developed with Andreas Barth (aka Andi) after the following exchange back in November 2003 on the RM channel:

<aba> elmo: If I find some time to take a look at katies TODO-file, what would be most appreciated? Deleting udebs with melanie? “Right” handling of orig.tar.gz when jumping out of NEW / changing of the archive section? Something else?
<elmo> aba: the TODO list is a bit of a mess to be honest
<aba> elmo: So, what should I do if I want to help the ftp-masters with katie?
<elmo> aba: I wouldn’t necessarily do anything based off that unless it’s obviously correct/wanted. none of the really urgent stuff is even on there (e.g. testing-security changes)
<aba> elmo: If you tell me something urgent, I can put some time into it.
<aj> hrm, surely i have some wishlists
<aba> urgent wishlist? :)
<aj> actually, you could critique something for me if you’ve got half an hour free now?
<aba> “critique” means?
<elmo> tell him how crap his ideas are. as brutually as possible
<elmo> ;) … <aj> ooo!
<aj> aba: around?
<aj> aba: one useful thing you could work on that’s kinda katie related is implementing pdiffs into katie/apt-ftparchive/apt/whatever
<Kamion> aj: mvo’s been working on that over the last couple of days, see #128818 logs
<aj> apt side or katie side?
<Kamion> apt side
<aj> cool
<aj> aba: hacking up a katie-side implementation for that would probably be interesting, worthwhile and straightforward

(aj = me, aba = Andi, elmo = James Troup, Kamion = Colin Watson) That ended up with me finding an old script I made up called update.py (from Dec 2003 apparently) that did the core patch and Index generation, Andi updating that to some more sane pythonic style and making it work on an actual archive, then both of us cleaning it up so it would actually work sensibly without handholding. (For those playing along at home, the idea needing critiquing was trying to fix up dak’s idea of what should happen if a new version of a package is uploaded to stable or testing, when it hasn’t changed in unstable. At the time, dak would REJECT it, because it violated the requirement that you can’t downgrade packages between stable and testing or testing and unstable. That’s still somewhat broken, unfortunately) TTBOMK, Tiffani’s named after Tiffani Amber Thiessen, apparently of Saved by the Bell and Beverly Hills 90210 fame. I’m not sure who of aba or elmo is more to blame for that one, though I guess it is at least a little more distinctive than update…