I just found out by chance that Fedora 14 switched from their old man package to man-db. This is great news: it should now be the beginning of the end of the divergence of man implementations that happened way back in the mid-1990s, when two different people took John W. Eaton's man package and developed it in different directions without being aware of each other's existence. For a while it looked as though man-db was stuck on just the Debian family and openSUSE, but a number of distributions have switched over in the last few years. As of now, the only remaining major distribution not using man-db is Gentoo, and they have a bug for switching which I think should be unblocked fairly soon. In some ways man-db's package name didn't help it; people thought that the main difference was that man-db had a database backend stuck around apropos. These days, the database is one of the least important parts of man-db as far as I'm concerned. Other ways in which it's very significantly superior to anything man could do without years of equivalent effort include correct encoding support, robust child process handling, and use of more modern development facilities (dear catgets: you belong to a previous millennium, so please go away). I'm glad that Fedora has recognised this.

Colin Watson is not a high-profile Debian figure, you rarely see him on mailing lists but he cares a lot about Debian and you will see him on Debconf videos sharing many thoughtful comments. I have the pleasure to work with him on dpkg as he maintains the package in Ubuntu, but he does a lot of more interesting things. I also took the opportunity to ask some Ubuntu specific questions since he s worked for Canonical since the start. Read on. My questions are in bold, the rest is by Colin. Who are you? Hi. I m 32 years old, grew up in Belfast in Northern Ireland, but have been living in Cambridge, England, since I was 18. I m married with a stepson and a daughter. I became interested in Debian due to the critical mass of Debian work happening in Cambridge at the time (and perhaps more immediately because my roommate was running Debian: hey, what s that? ), started doing random bits of development in 2000, and joined as a developer in 2001 (a really exciting time, with lots of new people joining who became integral parts of the project). I d only really been intending to do QA work and various bits of packaging around the edges, and maybe some work on the BTS, but then Fabrizio Polacco died and I took over man-db from him, and it sort of snowballed from there. I graduated from university shortly before becoming a Debian developer. I worked for a web server company (Zeus), then a hardware cryptography company (nCipher), before moving to work for Canonical in 2004, since when I ve been working full-time on Ubuntu. By this point, I suspect that going back to work in an office every day would be pretty tough. What s your biggest achievement within Debian or Ubuntu? One thing I should say: I rarely start projects. Firstly I don t think I m very good at it, and secondly I much prefer coming on to an existing project and worrying away at all the broken bits, often after other people have got bored and wandered off to the next new and shiny thing. That s probably why I ended up in the GNU/Linux distribution world in the first place, rather than doing lots of upstream development from the start I like being able to polish things into a finished product that we can give to end users. So, I ve had my fingers in a lot of pies over the years, doing ongoing maintenance and fixing lots of bugs. I think the single project I m most proud of would have to be my work on the Debian installer. I joined that team in early 2004 (a few months before Canonical started up) partly because I was a release assistant at the time and it was an obvious hot-spot, and partly because I thought it d be a good idea to make sure it worked well on the shiny new G4 PowerBook I d just treated myself to. I ended up as one of the powerpc d-i port maintainers for a while (no longer, as that machine is dead), but I ve done a lot of core work as well: much of the work to put progress bars in front of absolutely everything that used to have piles of text output, rescue mode, the current kernel selection framework, a good deal of udev support, several significant debconf extensions, lots of os-prober work, and I think I can claim to be one of the few people who understands the partitioner almost top to bottom. d-i is the very first thing many of our users see, and has a huge range of uses, from simple desktop installs to massive corporate deployments; it s unspeakably important that it works well, and it s a testament to its design that it s been able to trundle along without actually very much serious refactoring for the best part of five years now. I have a soft spot for man-db too. It was my first major project in Debian, starting out from an embarrassingly broken state, and is now nice and stable to the point where I recently had time to spawn a useful generic library out of it (libpipeline). What are your plans for Debian Wheezy? d-i has a lot of code to deal with disks and partitions. Of course a lot of it is in the partitioner, and for that we use libparted so we don t have to worry very much about the minutiae of device naming. But there are several other cases where we do need to care about naming, mainly before the partitioner when detecting disks, and after the partitioner when installing the boot loader. Back in etch, we introduced list-devices , which abstracted away the disk naming assumptions involved in hardware detection. In wheezy, I would like to take all the messy, duplicated, and error-prone code that handles disk naming in the boot loader installers, and design a simple interface to cover all of them. This has only got more important following the addition of the kFreeBSD and Hurd d-i ports in squeeze, but it bites us every time we notice that, say, CCISS arrays aren t handled consistently, and it s a pain to test all that duplicated code. I d also like to spread the use of libpipeline through C programs in the archive, which I think has potential to eliminate a class of security vulnerabilities in a much simpler way than was previously available. If you could spend all your time on Debian, what would you work on? I would love to systematically reduce the need for the current mass of boot loaders. There s a significant cost to having so much variation across architectures here: it s work that needs to be done in N different places, the wildly differing configuration means that d-i has to have huge piles of code to manage them all differently, and there are a bunch of strange arbitrary limitations on what you can do. The reason I m working on GRUB 2 is that, in my view, it s the project with the best chance of centralising all this duplicated work into a single place, and making it easier to bring up new hardware in future (in a way that doesn t compromise software freedom, as many proprietary boot loaders of the kind often found on phones do). Of course, with flexibility tends to come complexity, and some people have a natural objection to that and prefer something simpler. The things I don t quite have time to do here are to figure out a coherent way to address the specific over-complexity problems people have with the configuration framework while still keeping the flexibility we need, and to do enough QA and porting work to be able to roll out GRUB 2 at installation time to all the Debian architectures it theoretically supports. What s the biggest problem of Debian? Backbiting, and too much playing the man rather than the ball. With one or two honourable exceptions, I ve largely stopped reading most Debian mailing lists since it just never seems a productive way to spend time compared to writing code and fixing bugs; and yet I m conscious that they re one of the primary means of communication for the project and I m derelict in not taking part in them. I do find it a bit frustrating that people are seen primarily in terms of their affiliations. I suppose it s natural for people to see me as an Ubuntu guy , but I don t really see myself that way: I ve been working on Debian for nearly twice as long as I ve been working on Ubuntu, and, while I care a great deal about both projects, I ve put far more of my own personal time into Debian and I try to make sure that a decent number of the things I m involved with there aren t to do with work. Work/life separation is a good thing, not that I m very good at it. Generally speaking, when I m working on Debian, I m doing so as a Debian developer, because I want Debian to be better. When that s not the case, if it matters, I try to indicate it explicitly. You re working for Canonical since Ubuntu s inception. If you were Mark Shuttleworth, is there something that you would have done differently? We had many good intentions when we founded Ubuntu. We also had a huge amount of work to deliver, to the point where it wasn t at all clear whether it would be possible (the warty release was named based on the expectations of it, after all, and came out much more usable than we d dared to hope). In hindsight, it might have helped to be quieter about our good intentions, so that we could exceed expectations rather than in some cases failing to meet them. That might have set a very different tone early on. (Personally, I m happy I m not Mark. The decisions in my office are much easier to take.) It seems to me that the community part of Ubuntu is much more eager to cooperate with Debian than the corporate part. It s probably just that more and more Canonical employees are not former Debian contributors. Do you also have this feeling? Are there processes in place to ensure everybody at Canonical is trying to do the right thing towards Debian cooperation? Just to be clear, I m wearing my own hat here which, ironically, is a fedora rather than a company hat. It makes sense for Canonical to be taking on more non-Debian folks; after all, we can t simply hire from the Debian community forever, and a variety of backgrounds is healthy. As you say, it may well be natural that Ubuntu developers who don t work for Canonical are more likely to have a Debianish background, as it tends to take something significant to get people to switch to a very different family of GNU/Linux distributions, and changing jobs is one of the most obvious of those things. Certainly, there was a definite sense among the early developers that we were all part of the Debian family and cared about the success of Debian as well. As Ubuntu has developed its own identity, people involved in it now tend to care primarily about the success of Ubuntu. At the same time, pragmatically, it s still true that getting code changes into Debian is one of the most economical ways to land them; changes made in Debian or upstream land once and tend to stay in place, while changes made only in Ubuntu incur an ongoing merge overhead, which is not at all trivial. In many ways it s human nature to try to fulfil your immediate goals in the most direct way possible. If your goal is to deliver changes to Ubuntu users, then it s natural to concentrate on that rather than looking at the bigger picture (which takes experience). Debian developers often fail to send changes upstream for much the same reason, although there s more variation there because they re normally working on Debian of their own volition and thus tend to have wider goals; the economics are more or less parallel though. Thus, I think the best way to improve things is to make it the path of least resistance for Ubuntu developers to send changes to Debian. We re already seeing how this works with the Ubuntu MOTU group; if you send a patch for review, or work on merging a package from Debian, very often the response includes have you sent these changes to Debian? . We re working on both streamlining our code review through a regular patch pilot programme and requiring more code review for changes in general, so I think this will be a good opportunity to ask more people to work with Debian when they propose changes to Ubuntu. For myself, this may be obvious, but I notice that I m much better at getting changes into Debian when I already have commit access to the Debian package in question. All the work on improving collaborative maintenance in Debian can only help, for Ubuntu as well as for everyone else. It doesn t make so much difference for large changes that require extensive discussion, but there are lots of small changes too. Canonical is upstream of many software projects (unity, indicators infrastructure, etc.). Why aren t those software immediately packaged in Debian? Do you think we can get this to change? I m not sure what the right approach is here, particularly as I haven t been involved with much of that on the Ubuntu side. I suspect it would be helpful to look at this in a similar way to Ubuntu changes in general. It s understandable that those developers have getting changes into Ubuntu as their first goal. And yet, having code in Debian offers a wider, and often technically adept, audience, and most developers like having their code reach a wider audience even if it s not their first priority, particularly if that audience is likely to be able to help with finding problems and fixing bugs. It should be seen as something beneficial to both distributions. The hardest problems will be with things that aren t merely optional add-ons (which should generally be fairly non-controversial in Debian, given the breadth of the archive in general the existence of things like bzr and germinate as Debian packages was never a hard question), but which require changes in established packages. For example, gnome-power-manager in Ubuntu is built with application indicator support, and that s an important part of having a good indicator-based panel: a lot of the point of indicators is consistency. Since I do very little desktop work myself, I don t know exactly what would be involved in making it possible to choose this system based on a Debian desktop, but I think it s probably a bit more complicated than just making sure all the new packages exist in Debian too. Obviously you have to start somewhere. Is there someone in Debian that you admire for his contributions? Christian Perrier is absolutely tireless and has done superb things for the state of translations in Debian. And Russ Allbery, even aside from his fine ongoing work on policy, Lintian, and Kerberos, is a constant voice of sanity and calmness. Release management is incredibly hard work, as I know from my own experience, and anyone who can sustain involvement in it for a long period is somebody pretty special. Steve Langasek and I got involved at about the same time but he outlasted me by quite a few years. He deserves some kind of medal for everything he s done there.

---

Thank you to Colin for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

11 comments Liked this article? Click here. My blog is Flattr-enabled.

Last week I was on my first Ubuntu Developer Summit (UDS-N), I wanted to attended this summit for couple of reasons: 1. Developer, I m part of Debian Java, working on Eclipse IDE, and as you re developing Debian, you re automatically developing Ubuntu, no? 2. DebConf11 main organizer, I really wanted to see how it s all done on corporate level, for instance we re going with hotels instead of student dorms and so on. 3. Migration to Linux infrastructure, make contacts and have some talks on this topic, because after DebConf11 in Bosnia a lot of institution will most probably open doors to open solutions such as Linux. I went to this summit without any expectations or plans, just said I ll play it by ear. This years UDS was held in Orlando, Florida before Orlando I was planning to stay in NYC for couple of days, therefor my whole plan was to get to NYC, then Orlando then head back home I NYC I spent couple of days here with my cousins and friends, as always NYC never fails to surprise me, and I ll always stop by even if it s only for couple of days I wanted to meet up with couple of Debian people as well, but unfortunately my schedule was too tight so Caribe Royale, Orlando Please note that this whole trip was crazy (in a good way) starting from Sarajevo; on my flight from NYC to Orlando I see a guy that s sitting in front of me is wearing Canonical Landscape tshirt, I approach him and it turns he was Jamu Kakar working on Landscape. Since my luggage was late we had conversations from Debian over Ubuntu to Java to everything pretty much. While we were still at the airport, we bump into two more people going to UDS. As we got to hotel, same moment we walked out of cab there he was Jorge Castro, was he there just to smoke a cigarette or whatever doesn t even matter, what matters is that even before walking into hotel building I knew where reception, my tower and bar was, everything I needed to know at that moment Hotel where everything was held was Caribe Royale and even though I looked at their website I thought there was no way it s all going to be that nice, to my delight it was exact replica. Place is All-Suite Hotel and Convention Center , which meant that both accomodation, venue and everything else was in one place which is 15 minutes later after I checked in, I was at a bar talking to Jim Baker about Java, Jython and Python, exact things I wanted to talk about at that given moment. What I said in these last couple of paragraphs sounds pretty cool eh? Well to me this whole UDS was just like this, everybody was so accessible and open for discussion, from Jono Bacon walking around offering candy to UDS reception staff Michelle, Marianna and others who were always there whether you had a complaint, problem or a praise. I really should stop mentioning names, because I ll forget somebody, and since it s just not fair because I had a feeling everybody was just there for you, whatever it was you needed. Ubuntu Developer Summit (UDS-N) Ubuntu, Debian, DebConf, differences? Even though I said I ll play it by ear, from very beginning I thought I d be complete outsider, being part of Debian and all. Quite contrary, I was everything but outsider, and even more there was plenty of Debian people there, including current DPL Stefano Zacchiroli, Colin Watson, Riku Voipio just to name few. There was also a Debian Healt Check plenary. But overall, there are differences between DebConf and UDS. One of the things is that you can attend UDS remotely I m not sure how much is this doable on DebConf, but then again vast differences are that UDS is filled with BOF s (as we would call it in Debian) meetings of different teams, I attended a lot of these, and most of them end with some kind of conclusion. While each day there was (only?) an hour of lectures on various topics, which on DebConf we would characterize maybe even as speed talks . However, everything was high paced, and it all ended up on You think that s a problem? How do you think we could solve it? . I had a feeling even if I didn t say something out loud that someone would hear me and approach me to discuss it. Everything was straight forward, all in goal to resolve certain issue or a problem, which I absolutely loved. Fun It wasn t all work, even though Orlando doesn t seem to be my kind of city from what I saw, it s way too magical for my taste There was a lot of places where you could go to enjoy yourself. Good people at Canonical even offered me free ticket to Disney World, there was a lot of destinations to visit and transpiration was also organized by Canonical. Unfortunately I missed the notorious UDS party on Friday because I had to head home early, but possibly I ll make it next year Even though, hotel was that great and to be honest you didn t have to leave to look for party. My favorite was Universal City Walk, one night I decided to go to see Blue Man Group, best part of it was that I got pulled up on stage by one of the members and was part of one of the acts! This is definitely something everyone should see during his life, one moment you re laughing like a lunatic, while other moment you re just having goose bumps from the performance itself. Amazing stuff. After all I wrote here along with the pictures, is there even anything else I need to say? I believe that even if I didn t play it by ear, this whole trip would exceed all my expectations in every possible sense. Because that s really what happened, those three reasons I was going to UDS for? Far more happened than that. All I can say in the end is that I ll move closer to having a part in Ubuntu development, and yea, I ll try to be regular on UDS from now on See you in Budapest? If all these photos weren t enough, you can find official group photo as well as personal set of photos can be found on: UDS-N photos by 2010 Sean Sosik-Hamor

In my previous post, I described the pipeline library from man-db and asked whether people were interested in a standalone release of it. Several people expressed interest, and so I've now released libpipeline version 1.0.0. It's in the Debian NEW queue, and my PPA contains packages of it for Ubuntu lucid and maverick. I gave a lightning talk on this at UDS in Orlando, and my slides are available. I hope there'll be a video at some point which I can link to. Thanks to Scott James Remnant for code review (some time back), Ian Jackson for an extensive design review, and Kees Cook and Matthias Klose for helpful conversations.

When I took over man-db in 2001, one of the major problems that became evident after maintaining it for a while was the way it handled subprocesses. The nature of man and friends means that it spends a lot of time calling sequences of programs such as zsoelim < input-file tbl nroff -mandoc -Tutf8. Back then, it was using C library facilities such as system and popen for all this, and I had to deal with several bugs where those functions were being called with untrusted input as arguments without properly escaping metacharacters. Of course it was possible to chase around every such call inserting appropriate escaping functions, but this was always bound to be error-prone and one of the tasks that rapidly became important to me was arranging to start subprocesses in a way that was fundamentally immune to this kind of bug. In higher-level languages, there are usually standard constructs which are safer than just passing a command line to the shell. For example, in Perl you can use system([$command, $arg1, $arg2, ...]) to invoke a program with arguments without the interference of the shell, and perlipc(1) describes various facilities for connecting them together. In Python, the subprocess module allows you to create pipelines easily and safely (as long as you remember the SIGPIPE gotcha). C has the fork and execve primitives, but assembling these to construct full-blown pipelines correctly is difficult and error-prone, so many programmers don't bother and use the simple but unsafe library facilities instead. I wrote a couple of thousand lines of library code in man-db to address this problem, loosely and now quite distantly based on code in groff. In the following examples, function names starting with command_, pipeline_, or decompress_ are real functions in the library, while any other function names are pseudocode. Constructing the simplified example pipeline from my first paragraph using this library looks like this:

pipeline *p;
int status;
p = pipeline_new ();
p->want_infile = "input-file";
pipeline_command_args (p, "zsoelim", NULL);
pipeline_command_args (p, "tbl", NULL);
pipeline_command_args (p, "nroff", "-mandoc", "-Tutf8", NULL);
pipeline_start (p);
status = pipeline_wait (p);
pipeline_free (p);

You might want to construct a command more dynamically:

command *manconv = command_new_args ("manconv", "-f", from_code,
                                     "-t", "UTF-8", NULL);
if (quiet)
	command_arg (manconv, "-q");
pipeline_command (p, manconv);

Perhaps you want an environment variable set only while running a certain command:

command *less = command_new ("less");
command_setenv (less, "LESSCHARSET", lesscharset);

You might find yourself needing to pass the output of one pipeline to several other pipelines, in a "tee" arrangement:

pipeline *source, *sink1, *sink2;
source = make_source ();
sink1 = make_sink1 ();
sink2 = make_sink2 ();
pipeline_connect (source, sink1, sink2, NULL);
/* Pump data among these pipelines until there's nothing left. */
pipeline_pump (source, sink1, sink2, NULL);
pipeline_free (sink2);
pipeline_free (sink1);
pipeline_free (source);

Maybe one of your commands is actually an in-process function, rather than an external program:

command *inproc = command_new_function ("in-process", &func, NULL, NULL);
pipeline_command (p, inproc);

Sometimes your program needs to consume the output of a pipeline, rather than sending it all to some other subprocess:

pipeline *p = make_pipeline ();
const char *line;
line = pipeline_peekline (p);
if (!strstr (line, "coding: UTF-8"))
	printf ("Unicode text follows:\n");
while (line = pipeline_readline (p))
	printf ("  %s", line);
pipeline_free (p);

man-db deals with compressed files a lot, so I wrote an add-on library for opening compressed files (which is somewhat man-db-specific, but the implementation wasn't difficult given the underlying library):

pipeline *decomp_file = decompress_open (compressed_filename);
pipeline *decomp_stdin = decompress_fdopen (fileno (stdin));

This library has been in production in man-db for over five years now. The very careful signal handling code has been reviewed independently and the whole thing has been run through multiple static analysis tools, although I would always welcome more review; in particular I have no idea what it would take to make it safe for use in threaded programs since I generally avoid threading wherever possible. There have been a handful of bugs, which I've fixed promptly, and I've added various new features to support particular requirements of man-db (though in as general a way as possible). Every so often I see somebody asking about subprocess handling in C, and I wonder if I should split this library out into a standalone package so that it can be used elsewhere. Web searches for things like "pipeline library" and "libpipeline" don't reveal anything that's a particularly close match for what I have. The licensing would be GPLv2 or later; this isn't likely to be negotiable since some of the original code wasn't mine and in any case I don't feel particularly bad about giving an advantage to GPLed programs. For more details on the interface, the header file is well-commented. Is there enough interest in this to make the effort of producing a separate library package worthwhile? As well as the general effort of creating a new package, I'd need to do some work to disentangle it from a few bits and pieces specific to man-db. If you maintain a specific package that could use this and you're interested, please contact me with details, mentioning any extensions you think you'd need. I intentionally haven't enabled comments on my blog for various reasons, but you can e-mail me at cjwatson at debian.org or man-db-devel at nongnu.org.

Hello fellow developers, The summer is over but I m happy to announce that this year s Summer of Code at Debian has been better than ever! This is indeed the 4th time we had the privilege of participating in the Google Summer of Code and each year has been a little different. This year, 8 of our 10 students succeeded in our (very strict!) final evaluations, but we have reasons to believe that they will translate into more long-term developers than ever, all thank to you. The highlight this year has been getting almost all of our students at DebConf10. Thanks again this year to generous Travel Grants from the Google Open Source Team, we managed to fly in 7 of our students (up from 3!). You certainly saw them, presenting during DebianDay, hacking on the grass of Columbia, hacking^Wcheering our Debian Project Leader throwing the inaugural pitch of a professional baseball game or hacking^Wsun-tanning on the tr s kitsch Coney Island beach. Before I give the keyboard to our Students, I d like to tell you that it will be the pleasure and honor of Obey Arthur Liu (yours truly, as Administrator) and Bastian Venthur (as Mentor) to represent Debian at the Summer of Code 2010 Mentors Summit on 23-24 October 2010, at the Google Headquarters in Mountain View. Like last year, we expect many other DDs to be present under other hats. We will be having 2 days of unconference on GSoC and free software related topics. We all look forward to reporting from California on Planet and soc-coordination@l.a.d.o! All of our students had a wonderful experience, even if they couldn t come to DebConf, that is best shared in their own voice, so without further ado, our successful projects: Multi-Arch support in APT by David Kalnischkies, mentored by Michael Vogt apt-get install MultiArch does mostly work now as most code is already merged in squeeze, but if not complain about us at deity@l.d.o! Still, a lot left on the todo list not only in APT so let us all add MultiArch again to the Release Goals and work hard on squeezing it into wheezy. Debbugs Bug Reporting and Manipulation API by David Wendt Jr., mentored by Bastian Venthur Hello, I m David Wendt, and I went to Debconf10 to learn more about the development side of Debian. Having used it since the 9th grade, I ve been intimately familiar with many of Debian s internals. However, I wanted to see the developers and other Debian users. At DebConf, I was able to see a variety of talks from Debian and Ubuntu developers. I also got to meet with my mentor as well as the maintainer of Debbugs. Content-aware Config Files Upgrading by Krzysztof Tyszecki, mentored by Dominique Dumont Config::Model is now capable of manipulating files using shorter and easier to write models. Thanks to that, packagers may start experiment with creating upgrade models. Further work is needed to support more complicated config files Dominique Dumont is working on DEP-5 parser, I ll shortly start working on a cupsd config file parser.
The best thing about DebConf10 is that every person I talked with knew what I was doing. I had a mission to get some feedback on my project. Everybody liked the idea of making upgrades less cumbersome. On the other side, it was my first visit to United States, so I decided to go on a daytrip on my own (instead of staying inside the building, despite heat warnings). I had a chance to visit many interesting places like Ground Zero, the UN headquarters, Grand Central Terminal, Times square and Rockefeller Center that was a great experience. Hurd port and de-Linux-ization of Debian-Installer by J r mie Koenig, mentored by Samuel Thibault Debconf10 was great! Among other people working on the installer, I met Aur lien Jarno from the Debian/kFreeBSD team and we worked together on a cross-platform busybox package. Besides, the talks were very interesting and I ve filled my TODO-list for the year.
For instance I learned about the Jigsaw project of OpenJDK, and how Debian would be the ideal platform to experiment with it. More generally, some people think Debian could push Java 7 forward and I d like to see this happen. Smart Upload Server for FTP Master by Petr Jasek, mentored by Joerg Jaspert I must say that it was great time for me in NY, I ve met and talked and coded with people from ftp-master team like Torsten Werner who helped me to push the project a bit further and with some other people who were looking forward to release of the tool which I hope they will use quite soon. Everybody interested, everybody excited, really cool place and time. And I can t forget the Coney Island beach and stuff, lot of fun, lot of sun;) Aptitude Qt by Piotr Galiszewski, mentored by Sune Vuorela Currently, development branches support full features searching, viewing extended package s informations, performing cache and packages operations. Code and GUI still require a lot of work which will be continued. Informations about further progress could be found on aptitude mailing list and repository rss channel. Debian-Installer on Neo FreeRunner and Handheld Devices by Thibaut Girka, mentored by Gaudenz Steinlin For me, DebConf 10 started at the airport, where Sylvestre Ledru (whom I didn t know of before) was wearing a GSoC 2007 t-shirt, that is, given the circumstances, almost equivalent to say I m a hacker, I m going to DebConf 10 .
I ve spent my time at the conference attending various talks, hacking, meeting DDs and other hackers (amongst others, my co-mentor Per Andersson, Paul Wise, Julien Cristau, Christian Perrier, Cyril Brulebois, Martin Michlmayr, Colin Watson and Otavio Salvadores who I have to thank for his patience while dealing with my questions), chatting, cross-signing keys, rushing to finish eating before 7pm, getting sunburnt, sightseeing (thanks, Arthur, for the lightning-fast tour of Manhattan!), and so on. Debian Developers and community, we count on you. See you next year! (cross-posted to debian-devel-announce@l.d.o and soc-coordination@l.a.d.o)

If you find that running Windows makes a GRUB 2-based system unbootable (Debian bug, Ubuntu bug), then I'd like to hear from you. This is a bug in which some proprietary Windows-based software overwrites particular sectors in the gap between the master boot record and the first partition, sometimes called the "embedding area". GRUB Legacy and GRUB 2 both normally use this part of the disk to store one of their key components: GRUB Legacy calls this component Stage 1.5, while GRUB 2 calls it the core image (comparison). However, Stage 1.5 is less useful than the core image (for example, the latter provides a rescue shell which can be used to recover from some problems), and is therefore rather smaller: somewhere around 10KB vs. 24KB for the common case of ext[234] on plain block devices. It seems that the Windows-based software writes to a sector which is after the end of Stage 1.5, but before the end of the core image. This is why the problem appears to be new with GRUB 2. At least some occurrences of this are with software which writes a signature to the embedding area which hangs around even after uninstallation (even with one of those tools that tracks everything the installation process did and reverses it, I gather), so that you cannot uninstall and reinstall the application to defeat a trial period. This seems like a fine example of an antifeature, especially given its destructive consequences for free software, and is in general a poor piece of engineering; what happens if multiple such programs want to use the same sector, I wonder? They clearly aren't doing much checking that the sector is unused, not that that's really possible anyway. While I do not normally think that GRUB should go to any great lengths to accommodate proprietary software, this is a case where we need to defend ourselves against the predatory practices of some companies making us look bad: a relatively small number of people do enough detective work to realise that it's the fault of a particular Windows application, but many more simply blame our operating system because it won't start any more. I believe that it may be possible to assemble a collection of signatures of such software, and arrange to avoid the disk sectors they have stolen. Indeed, I have a first draft of the necessary code. This is not a particularly pleasant solution, but it seems to be the most practical way around the problem; I'm hoping that several of the programs at fault are using common "licence manager" code or something like that, so that we can address most of the problems with a relatively small number of signatures. In order to do this, I need to hear from as many people as possible who are affected by this problem. If you suffer from this problem, then please do the following:

• Save the output of fdisk -lu to a file. In this output, take note of the start sector of the first partition (usually 63, but might also be 2048 on recent installations, or occasionally something else). If this is something other than 63, then replace 63 in the following items with your number.
• Save the contents of the embedding area to a file (replace /dev/sda with your disk device if it's something else): dd if=/dev/sda of=sda.1 count=63
• Do whatever you do to make GRUB unbootable (presumably starting Windows), then boot into a recovery environment. Before you reinstall GRUB, save the new contents of the embedding area to a different file: dd if=/dev/sda of=sda.2 count=63
• Follow up to either the Debian or the Ubuntu bug with these three files (the output of fdisk -lu, and the embedding area before and after making GRUB unbootable.

I hope that this will help me to assemble enough information to fix this bug at least for most people, and of course if you provide this information then I can make sure to fix your particular version of this problem. Thanks in advance!

Apropos of my previous post, I see that dh has now overtaken CDBS as the most popular rules helper system of its kind in Debian unstable, and shows no particular sign of slowing its rate of uptake any time soon. The resolution of the graph is such that you can't see it yet, but dh drew dead level with CDBS on Thursday, and today 3836 packages are using dh as opposed to 3823 using CDBS.

I m a Debian developer since 1998 and have been fostering cooperation between Debian and Ubuntu since its inception with various levels of success. I recently decided to go one step further and join Ubuntu as an official member. I wanted to be able to vote and to promote best cooperation practices from the Ubuntu side as well (and not only in Debian). Joining as a Debian developer should be relatively easy, it is well known in the Debian community that Mark Shuttleworth once said:

Every Debian developer is also an Ubuntu developer, because one way to contribute to Ubuntu is to contribute to Debian.

So I sought to verify that too! Here s the result. I prepared my application page collecting 2 testimonials from Lucas Nussbaum and Colin Watson, I registered for the next EMEA membership meeting (which happened yesterday, irc logs here), and I attended. (I just followed the instructions on the wiki) During the meeting, when it was my turn, I pasted my short introduction and got immediately +1 votes from several board members, they didn t even want to hear from me, they were already convinced. I still responded to a few questions and elaborated on my goals and was promptly accepted even if I made it clear that my technical contributions will continue to be done within Debian. A few minutes later I was added to the ubuntumembers launchpad group and this morning I followed the instructions to add my feed to Planet Ubuntu and here I am The whole process has been very smooth and the meeting shows a very welcoming attitude. Even someone who was denied the Ubuntu member status was cheered for his work and encouraged to try again when he can present some testimonials from other members. Hello Planet Ubuntu! 11 comments Support my work

... this version, or something not too far away from it, might actually stand a chance of getting into testing. I've just uploaded grub2 1.98+20100702-1. The most significant set of changes in this release is that it switches /boot/grub/device.map and the grub-pc/install_devices debconf question over to stable device names under /dev/disk/by-id (on Linux kernels). The code implementing this is reasonably careful, and it should make it quite difficult for people to accidentally fail to upgrade their installed GRUB core image; I explained the problems that tends to cause in the previous post in this series. There will probably be a few small glitches I need to clear up, but I've given this much more extensive testing than usual so I hope I won't break too many people's computers (again). I did this work first in Ubuntu as one of my major goals for 10.04 LTS, which exposed a few problems that I wanted to fix before inflicting it on Debian as well (fixes for those are now under testing for 10.04.1). Most significantly, I felt it was necessary to start offering partitions in the select list for grub-pc/install_devices, but I went a bit overboard and offered all partitions in a giant list. This seemed like a good idea at the time, but it tended to confuse people into just selecting everything in the list, which in particular tended to make Windows unbootable! So I dialled that back a bit, and in the version I just merged it will only offer the partitions mounted on /, /boot, and /boot/grub (de-duplicating if necessary). This seems like a reasonable compromise between confusing people too much and forcing them to install only to MBRs. My next priority will be making whatever fixes are necessary to get this version into testing, since the problems with /dev/mapper symlinks in testing aren't getting any less urgent, and this is finally a version that shouldn't break for most people due to the kernel's switch to libata. I expect that I'll try to get mdadm 1.x metadata sorted out immediately after that. Other improvements since my last entry have included:

• Further documentation work. Thanks to Vladimir Serbinenko (and to Jordan Uggla for hosting it temporarily), there's now an HTML version of the GRUB manual from trunk online, which includes new sections on embedded configuration files, the various GRUB image files, device.map, and (shortly) a summary of changes from GRUB Legacy.
• Video improvements: among other things, UEFI systems whose firmware uses the Graphics Output Protocol should now work rather better, and GRUB now includes specific support for some cards often used with minimal firmware support under emulation.
• A fix to handle large memory maps exposed by some UEFI firmware.
• Automatic configuration support for Fedora 13. You may need os-prober 1.39 from unstable as well.
• Automatic configuration support for Linux on Xen.
• Skip LVM snapshots rather than failing when they're present.

(This is partly a repost of material I've posted to bug reports and to debian-release, put together with some more detail for a wider audience.) You could be forgiven for looking at the RC bug activity on grub2 over the last couple of days and thinking that it's all gone to hell in a handbasket with recent uploads. In fact, aside from an interesting case which turned out to be due to botched handling of the GRUB Legacy to GRUB 2 chainloading setup (which prompted me to fix three other RC bugs along the way), all the recent problems people have been having have been duplicates of one of these bugs which have existed essentially forever:

• #554790 - grub-pc/install_devices uses unstable device names
• #583271 - device.map uses unstable device names

When GRUB boots, its boot sector first loads its "core image", which is usually embedded in the gap between the boot sector and the first partition on the same disk as the boot sector. This core image then figures out where to find /boot/grub, and loads grub.cfg from it as well as more GRUB modules. The thing that tends to go wrong here is that the core image must be from the same version of GRUB as any modules it loads. /boot/grub/*.mod are updated only by grub-install, so this normally works OK. However, for various reasons (deliberate or accidental) some people install GRUB to multiple disks. In this case, grub-install might update /boot/grub/*.mod along with the core image on one disk, but your BIOS might actually be booting from a different disk. The effect of this will be that you'll have an old core image and new modules, which will probably blow up in any number of possible ways. Quite often, this problem lies dormant for a while because GRUB happens not to change in a way that causes incompatibility between the core image and modules, but then we get massive spikes of bug reports any time the interface does change. Since these bugs sometimes bite people upgrading from testing to unstable, they get interpreted as regressions from the version in testing even though that isn't strictly true (but it tends not to be very productive to argue this line; after all, people's computers suddenly don't boot!). Any problem that causes the core image to be installed to a disk other than the one actually being booted from, or not to be installed at all, will show up this way sooner or later. On 2010-06-10, there was a substantial upstream change to the handling of list iterators (to reduce core image size and make code clearer and faster) which introduced an incompatibility between old core images and newer modules. This caused a bunch of dormant problems to flare up again, and so there was a flood of reports of booting problems with 1.98+20100614-1 and newer, often described as "the unaligned pointer bug" due to how it happened to manifest this time round. In previous cases, GRUB reported undefined symbols on boot, but it's all essentially the same problem even though there are different symptoms. The confusing bit when handling bug reports is that not only are there different symptoms with the same cause, but there are also multiple causes for the same symptom! This takes a certain amount of untangling, especially when lots of people have thought "ooh, that bug looks a bit like mine" and jumped in with their own comments. Working through this was a worthwhile exercise, as it came up with an entirely new cause for a problem I thought was fairly well-understood (thanks to debugging assistance from Sedat Dilek). If you had set up GRUB 2 to be automatically chainloaded from GRUB Legacy (which happens automatically on upgrade from the latter to the former), never got round to running upgrade-from-grub-legacy once you confirmed it worked, and then later ran grub-install by hand for one reason or another, then the core image you installed by hand would never be updated and would eventually fall over the next time the core/modules interface changed. Fixing future cases of this was easy enough, but fixing existing cases involved figuring out how to detect whether an installed GRUB boot sector came from GRUB Legacy or GRUB 2, which isn't as easy as you might think. Fortunately, it turns out that there are a limited number of jump offsets that have ever been used in the second byte of the boot sector, and none of the GRUB 2 values clash with the only value ever used in GRUB Legacy; so, if you still have /boot/grub/stage2 et al on upgrade, we scan all disks for a GRUB 2 boot sector, and if we find one then we offer to complete the upgrade to GRUB 2. Unless anything new shows up, that just leaves the problems that were already understood. Today, I posted a patch to generate stable device names in device.map by default. If this is accepted, then we can do something or other to fix up device.map on upgrade, switch over to /dev/disk/by-id names in grub-pc/install_devices at the same time, and that should take care of the vast majority of this kind of upgrade bug. I think at that point it should be feasible to get a new version into testing, and we should be down from 18 RC bugs towards the end of last month to around 6. We can then start attacking things like the lack of support for mdadm 1.x metadata. Since my last blog entry on GRUB 2, improvements have included:

• Substantial work on info grub, with, among other things, new sections on /etc/default/grub and on configuring authentication.
• A workaround for GRUB's inability to probe dm-crypt devices, thanks to Marc Haber.
• Several build fixes for architectures I wasn't testing, and a fix for broken nested partition handling on Debian GNU/kFreeBSD. I'm now testing GNU/kFreeBSD locally.
• Rather less cruft in fs.lst, partmap.lst, and video.lst, which should speed up booting a bit by e.g. avoiding unnecessary filesystem probing.
• upgrade-from-grub-legacy actually now installs GRUB 2 to the boot sector (!).
• Ask for confirmation if grub-pc/install_devices is left empty.

The next upstream snapshot will bring several improvements to EFI video support, mainly thanks to Vladimir Serbinenko. I've been working on making grub-install actually work on UEFI systems as one of my goals for the next Ubuntu release, and I hope to get this landed in the not-too-distant future.

Various people observed in a long thread on debian-devel that the grub2 package was in a bit of a mess in terms of its release-critical bug count, and Jordi and Stefano both got in touch with me directly to gently point out that I probably ought to be doing something about it as one of the co-maintainers. Actually, I don't think grub2 was in quite as bad a state as its 18 RC bugs suggested. Of course every boot loader failure is critical to the person affected by it, not to mention that GRUB 2 offers more complex functionality than any other boot loader (e.g. LVM and RAID), and so it tends to accumulate RC bugs at rather a high rate. That said, we'd been neglecting its bug list for some time; Robert and Felix have both been taking some time off, Jordi mostly only cared about PowerPC and can't do that any more due to hardware failure, and I hadn't been able to pick up the slack. Most of my projects at work for the next while involve GRUB in one way or another, so I decided it was a perfectly reasonable use of work time to do something about this; I was going to need fully up-to-date snapshots anyway, and practically all the Debian grub2 bugs affect Ubuntu too. Thus, with the exception of some other little things like releasing the first Maverick alpha, I've spent pretty much the last week and a half solidly trying to get the grub2 package back into shape, with four uploads so far. The RC issues that remain are:

• upgrade-from-grub-legacy problems (#547944, #550477):
  I think this has just been traditionally undertested. I'm setting up a KVM image now with GRUB Legacy which I can snapshot just before and after running upgrade-from-grub-legacy, and I should be able to unpick the bugs this way.
• LVM snapshots break GRUB's LVM module (#574863):
  Sean has been working on this and seems to be nearly there. Yay.
• RAID metadata version 1.x not supported (#492897):
  This became rather more of an issue recently since mdadm switched its default from the old 0.90 format which GRUB understood. Felix put together a branch implementing the hard parts of this a while back, and I've been trying to finish it off. The hard bit is dealing with device naming, especially as the new-format and rather more useful names under /dev/md/ don't show up during d-i after creating RAID volumes; I think this is because we always create them as /dev/md0 etc. It's looking tractable, though.
• Another odd problem probing RAID (#548648):
  Not sure about this one, and I'll need to work with Josip on it as soon as I get a chance.
• Stable device naming #554790) and consequential problems due to grub-install not being properly run (#557425 and many other sub-RC bugs):
  Ubuntu's been carrying a patch to rearrange device presentation in the postinst, which Robert OKed in principle ages ago and so I've been intending to merge it for a while, but there are a few known problems with it that I need to fix first. One known unfixable problem is that it will have to ask some people which devices they want GRUB to be installed on, even if they'd answered that question before: this will be one-time, and it's because it recorded the answer using unstable device names and so has in some sense forgotten. Simple cases (e.g. single-disk) can be handled without needing to ask again, though.
• Alignment errors on SPARC (#560823):
  I have no idea what's going on here, I'm afraid. I'll try to trace it, but may have to downgrade it at some point since after all we don't install GRUB by default on SPARC yet.
• Fonts not shown in gfxmenu (#564844):
  Apparently fixed upstream, but I couldn't find the responsible commit so I want to make sure I can get gfxmenu working before closing this.
• Sensitivity to out-of-date device.map files (#575076 and other sub-RC bugs):
  We're trying to get rid of device.map in general. It was fine in the 1990s but it's hopeless now. Unfortunately there are still a small number of problems with running entirely without one, and one of my patches to help is controversial upstream, so we probably won't get to that for squeeze. In the meantime we'll probably just need some extra sanity-checking and robustness in the event that there's an incorrect or out-of-date device.map lying around, which we may just be able to do in the maintainer scripts or something if necessary.
• Seriously weird failures to load initramfs (#582342):
  If anyone can produce a reproduction recipe for this, that would really help me out. There are too many reports to discount as user error, but I haven't seen this myself yet.
• Build failure on sparc (unfiled):
  We've been discussing this upstream, but for the time being I'm just going to stop building grub-emu on sparc as a workaround.

If we can fix that lot, or even just the ones that are reasonably well-understood, I think we'll be in reasonable shape. I'd also like to make grub-mkconfig a bit more robust in the event that the root filesystem isn't one that GRUB understands (#561855, #562672), and I'd quite like to write some more documentation. On the upside, progress has been good. We have multiple terminal support thanks to a new upstream snapshot (#506707), update-grub runs much faster (#508834, #574088), we have DM-RAID support with a following wind (#579919), the new scheme with symlinks under /dev/mapper/ works (#550704), we have basic support for btrfs / as long as you have something GRUB understands properly on /boot (#540786), we have full info documentation covering all the user-adjustable settings in /etc/default/grub, and a host of other smaller fixes. I'm hoping we can keep this up. If you'd like to help, contact me, especially if there's something particular that isn't being handled that you think you could work on. GRUB 2 is actually quite a pleasant codebase to work on once you get used to its layout; it's certainly much easier to fix bugs in than GRUB Legacy ever was, as far as I'm concerned. Thanks to tools like grub-probe and grub-fstest, it's very often possible to fix problems without needing to reboot for anything other than a final sanity check (although KVM certainly helps), and you can often debug very substantial bits of the boot loader - the bits that actually go wrong - using standard tools such as strace and gdb. Upstream is helpful and I've been able to get many of the problems above fixed directly there. If you have a sound knowledge of C and a decent level of understanding of the environment a boot loader needs to operate in - or for that matter specialist knowledge of interesting device types - then you should be able to find something to do.

For various reasons, I chose to leave Ubuntu 10.04 LTS using OpenSSH 5.3p1. The new features in 5.4p1 such as certificate authentication, the new smartcard handling, netcat mode, and tab-completion in sftp are great, but unfortunately it was available just a little bit too late for me to be able to land it for 10.04 LTS. I realise that many Lucid users want to make use of these features for one reason or another, though, so as a compromise here's a PPA containing OpenSSH 5.5p1 for Lucid. I intend to keep this up to date for as long as I reasonably can, and I'm happy to accept bug reports on it in the usual place.

The overdue dash sync from 0.5.3 took a month to be done, but now klibc is shipping newer dash 0.5.6 then actual unstable dash. Fixes for this release include fstype support for btrfs and ext4 without journal. Moving README's around so that they can be shipped for avid readers: README.ipconfig go :) The goodie from this time is the sh4 build fix form the very active Debian sh4 porters. ipconfig, nfsmount and kinit have now simpler DEBUG build. ipconfig build warnings got shot by a Google patch. If your patch hasn't made it yet, please ping me for next queue: klibc git repo, Unofficial patch queue. P.S.: Ubuntu Lucid pushes out klibc 1.5.17 thanks to Colin Watson.

Note: I wrote most of this before Neil Williams' recent comments on the 3.0 family of formats, so despite the timing this isn't really a reaction to that although I do have a couple of responses. On the whole I think I agree that the Lintian message is a bit heavy-handed and I'm not sure I'm thrilled about the idea of the default source format being changed (though I can see why the dpkg maintainers are interested in that). That said, as far as I personally am concerned, there is a vast cognitive benefit to me in having as much as possible be common to all my packages. Once I have more than a couple of packages that require patching and benefit from the 3.0 (quilt) format as a result, I find it in my interest to use it for all my non-native packages even if they're patchless right now, so that for instance if they need patches in the future I can handle them the same way. It's not unheard of for me to apply temporary patches even to packages I actively maintain upstream, so I don't discount those either. I haven't decided what to do with my native packages yet; unless they're big enough for bzip2 compression to be worthwhile, there doesn't seem to be much immediate advantage to 3.0 (native). Anyway, on to the main body of this post: I've been one of the holdouts resisting use of patch systems for a long time, on the basis that I felt strongly that dpkg-source -x ought to give you the source that's actually built, rather than having to mess around with debian/rules targets in order to see it. Now that the 3.0 (quilt) format is available to fix this bug, I felt that I ought to revisit my resistance and start trying to use it. Migrating to it from monolithic diffs is of course a bit more work than migrating to it from other patch systems, so it's taken me a little while to get round to it. I'd been thinking about holding off until there was better integration with revision control (e.g. bzr looms), as I feel that patch files really ought to be an export format, but I eventually decided that I shouldn't let the perfect be the enemy of the good. I have enough experience with co-maintaining packages that use build-time patch systems to be able to compare my reactions. After experimenting with a couple of small packages, I moved over to the deep end and converted openssh a few weekends ago, since quite a few people have requested over the years that the Debian changes to openssh be easier to audit. This was a substantial job - over 6000 lines of upstream patches - but not actually as much work as I expected. I took a fairly simplistic approach: first, I unapplied all the upstream patches from my tree; then I ran bzr di interdiff -q /dev/stdin /dev/null >x, reduced it to a single logically-discrete patch, applied it to a new quilt patch using quilt fold, and repeated until x was empty. This was maybe an hour or two of work, and then I went through and tagged all the patches according to DEP-3, which took another few hours. After the first pass, I ended up with 38 patches and a much clearer idea of what has been forwarded upstream and what hasn't; I currently have 5 patches to forward or eliminate, down from 18. Good things:

• I don't lose any of my history. Since all the patches remain applied to the tree in revision control (this is what dpkg-source -x gives you, so it's the natural representation in revision control too), bzr blame works just as you'd expect and displays both upstream and Debian changes at once. I rely on tools like blame a lot, and I really hate the way build-time patch systems make it hard to use revision control when the tree is in a built state, so this was a hard requirement for me.
• I've used patch tagging before, so I was expecting some benefits, but viscerally I feel much more in control. It's so much less laborious now to see what I need to do by way of forwarding. I don't regret waiting for 3.0 (quilt) to become available, but I hadn't realised quite how much I was being held back beforehand.
• Adding new patches is pretty natural, much more so than with build-time patch systems. You can create and apply the patch, test-build, and commit when it works. I much prefer this over having to clean the tree before committing (or commit just part of the tree, which is error-prone). The more that committing to a Debian package feels like committing to an upstream project, the better.
• There's definitely something to be said for patch-tracker being more useful. It deals with DEP-3 to the extent of linkifying URLs, although it might be nice if patch descriptions were displayed on the overview page for each version.

Bad things:

• It's a bit awkward to set things up when checking out from revision control; I didn't really want to check in the .pc directory, and the tree checks out in the patched state (as it should), so I needed some way for developers to get quilt working easily after a checkout. This is sort of the reverse of the previous problem, where users had to do something special after dpkg-source -x, and I consider it less serious so I'm willing to put up with it. I ended up with a rune in debian/rules that ought to live somewhere more common.
• Everything ends up represented twice in revision control: the patch files, plus the changes to the patched files themselves. I'm OK with this although it is a little inelegant.
• Although I haven't had to do it yet, I expect that merging new upstream releases will be a bit harder. bzr will deal with resolving conflicts in the patched files themselves, and that's why I use a revision control system after all, but then I'll have to go and refresh all the patches and will probably end up doing some of the same conflict resolution a second time. I think the best answer right now is to quilt pop -a, force a merge despite the modified working tree, and then quilt push && quilt refresh -pab until I get back to the top of the stack, modulo slight fiddliness when a patch disappears entirely; thus effectively using quilt's conflict resolution rather than bzr's. I suppose this will serve as additional incentive to reduce my patch count. I know that people have been working on making this work nicely with topgit, although I'm certainly not going to put up with the rest of git due to that; I'm happy to wait for looms to become usable and integrated. :-)
• It would be nice if there were some standard DEP-3 way to note that a patch has been accepted or rejected upstream, beyond just putting it in the description. In particular, it seems to me that listing patches accepted upstream could be used to speed up the process of merging new upstream releases.

On the whole I'm satisfied with this, and the benefits definitely outweigh the costs. Thanks to the dpkg team for all their work on this!

Note: I wrote most of this before Neil Williams' recent comments on the 3.0 family of formats, so despite the timing this isn't really a reaction to that although I do have a couple of responses. On the whole I think I agree that the Lintian message is a bit heavy-handed and I'm not sure I'm thrilled about the idea of the default source format being changed (though I can see why the dpkg maintainers are interested in that). That said, as far as I personally am concerned, there is a vast cognitive benefit to me in having as much as possible be common to all my packages. Once I have more than a couple of packages that require patching and benefit from the 3.0 (quilt) format as a result, I find it in my interest to use it for all my non-native packages even if they're patchless right now, so that for instance if they need patches in the future I can handle them the same way. It's not unheard of for me to apply temporary patches even to packages I actively maintain upstream, so I don't discount those either. I haven't decided what to do with my native packages yet; unless they're big enough for bzip2 compression to be worthwhile, there doesn't seem to be much immediate advantage to 3.0 (native). Anyway, on to the main body of this post: I've been one of the holdouts resisting use of patch systems for a long time, on the basis that I felt strongly that dpkg-source -x ought to give you the source that's actually built, rather than having to mess around with debian/rules targets in order to see it. Now that the 3.0 (quilt) format is available to fix this bug, I felt that I ought to revisit my resistance and start trying to use it. Migrating to it from monolithic diffs is of course a bit more work than migrating to it from other patch systems, so it's taken me a little while to get round to it. I'd been thinking about holding off until there was better integration with revision control (e.g. bzr looms), as I feel that patch files really ought to be an export format, but I eventually decided that I shouldn't let the perfect be the enemy of the good. I have enough experience with co-maintaining packages that use build-time patch systems to be able to compare my reactions. After experimenting with a couple of small packages, I moved over to the deep end and converted openssh a few weekends ago, since quite a few people have requested over the years that the Debian changes to openssh be easier to audit. This was a substantial job - over 6000 lines of upstream patches - but not actually as much work as I expected. I took a fairly simplistic approach: first, I unapplied all the upstream patches from my tree; then I ran bzr di interdiff -q /dev/stdin /dev/null >x, reduced it to a single logically-discrete patch, applied it to a new quilt patch using quilt fold, and repeated until x was empty. This was maybe an hour or two of work, and then I went through and tagged all the patches according to DEP-3, which took another few hours. After the first pass, I ended up with 38 patches and a much clearer idea of what has been forwarded upstream and what hasn't; I currently have 5 patches to forward or eliminate, down from 18. Good things:

• I don't lose any of my history. Since all the patches remain applied to the tree in revision control (this is what dpkg-source -x gives you, so it's the natural representation in revision control too), bzr blame works just as you'd expect and displays both upstream and Debian changes at once. I rely on tools like blame a lot, and I really hate the way build-time patch systems make it hard to use revision control when the tree is in a built state, so this was a hard requirement for me.
• I've used patch tagging before, so I was expecting some benefits, but viscerally I feel much more in control. It's so much less laborious now to see what I need to do by way of forwarding. I don't regret waiting for 3.0 (quilt) to become available, but I hadn't realised quite how much I was being held back beforehand.
• Adding new patches is pretty natural, much more so than with build-time patch systems. You can create and apply the patch, test-build, and commit when it works. I much prefer this over having to clean the tree before committing (or commit just part of the tree, which is error-prone). The more that committing to a Debian package feels like committing to an upstream project, the better.
• There's definitely something to be said for patch-tracker being more useful. It deals with DEP-3 to the extent of linkifying URLs, although it might be nice if patch descriptions were displayed on the overview page for each version.

Bad things:

• It's a bit awkward to set things up when checking out from revision control; I didn't really want to check in the .pc directory, and the tree checks out in the patched state (as it should), so I needed some way for developers to get quilt working easily after a checkout. This is sort of the reverse of the previous problem, where users had to do something special after dpkg-source -x, and I consider it less serious so I'm willing to put up with it. I ended up a rune in debian/rules that ought to live somewhere more common.
• Everything ends up represented twice in revision control: the patch files, plus the changes to the patched files themselves. I'm OK with this although it is a little inelegant.
• Although I haven't had to do it yet, I expect that merging new upstream releases will be a bit harder. bzr will deal with resolving conflicts in the patched files themselves, and that's why I use a revision control system after all, but then I'll have to go and refresh all the patches and will probably end up doing some of the same conflict resolution a second time. I think the best answer right now is to quilt pop -a, force a merge despite the modified working tree, and then quilt push && quilt refresh -pab until I get back to the top of the stack, modulo slight fiddliness when a patch disappears entirely; thus effectively using quilt's conflict resolution rather than bzr's. I suppose this will serve as additional incentive to reduce my patch count. I know that people have been working on making this work nicely with topgit, although I'm certainly not going to put up with the rest of git due to that; I'm happy to wait for looms to become usable and integrated. :-)
• It would be nice if there were some standard DEP-3 way to note that a patch has been accepted or rejected upstream, beyond just putting it in the description. In particular, it seems to me that listing patches accepted upstream could be used to speed up the process of merging new upstream releases.

On the whole I'm satisfied with this, and the benefits definitely outweigh the costs. Thanks to the dpkg team for all their work on this!

I've started the transition of parted 2.2 to unstable. This is a major update needed for sensible support of newer hard disks with alignment requirements different from the archaic cylinder alignment tradition. I posted to debian-boot with a summary of the partman changes involved.

I don't know if anyone else has been tracking this recently, but a while back I got curious about the relative proportions of dh(1) and CDBS in the archive, and started running some daily analysis on the Lintian lab. Apologies for my poor graphing abilities, but the graph is here (occasionally updated): Although dh is still a bit behind CDBS, the steady upward trend is quite striking - it looks set to break 20% soon, up from under 13% in September - compared with CDBS which has been sitting within half a percentage point of 25% the whole time. Incidentally, was that an ftpmaster trying to sign his name in the graph over Christmas or something? :-)

I did a bit of catching up on my Debian backlog over the last week or so. Among the things I got round to:

• I released man-db 2.5.7. This was mostly an "I've been meaning to do this for ages" kind of thing to reduce the bug list a bit, closing ten Debian bugs, but there were a few interesting things in there as well, such as always saving cat pages in UTF-8 and recoding to the user's locale at display time (long overdue), adjusting the search order for localised manual pages by request of quite a few non-native English speakers to prefer a page in the right section over a page in the right language, and a cute gimmick to make things like man /usr/bin/time display the appropriate manual page rather than the text of the executable. See the NEWS file for more details.
• binfmt-support now installs cleanly on non-Linux systems, even if it doesn't do anything useful yet.
• I fixed a couple of shell bugs in groff.
• halibut now complies with the Debian Vim policy, even though I can't say I entirely agree with it in this case.
• I fixed a really odd build failure in troffcvt. Yay imake, or something.
• All Debian patches to putty are now upstream, or will be once I upload a new snapshot. Thanks to Simon Tatham and Jacob Nevins.
• I did a few bits and pieces of packaging cleanup with an eye on my DDPO list, and added some watch files where they were missing.
• Responded to an offer to take over icoutils maintenance.

So nothing really earth-shaking, and as ever openssh could use some attention, but I feel a bit better about my backlog now. I do still have a critical bug in makepasswd to fix, and a sponsored upload of parrot; those are the next two things on my to-do list.

In case it isn't obvious, in "Ubuntu 9.10 SP1 coming in spring 2010", "Ubuman" is blatantly lying in attributing a number of statements to me. None of the text there was written by me, and if you thought any of it was true then you should probably make sure your troll radar is working properly. Nice joke, but try harder next time - it doesn't even look like my writing style. (I wouldn't normally bother to respond, since I'm probably just giving it more publicity, but apparently one or two people may already have been taken in by it. One person was sensible enough to write to me and check the facts.)