Recently, I ve been watching the Debian Schadenfreude related to init systems. For those who do not know, Debian is trying to decide whether Systemd or Upstart will be used to start software on Debian. There are two other options, although I think Systemd and Upstart are the main contenders. Currently the Technical Committee is considering the issue, although there s some very real chance that the entire project will get dragged through this swamp with the general resolution process. This is one of those discussions that proves that Debian truly is a community: if we didn t have something really strong holding us together we d never put up with something this painful. Between the accusations that our users now know the persecution European pagans faced at the hands of Christians as the Systemd priests drive all before them, a heated discussion of how the Debian voting system interacts with this issue, two failed calls for votes, a discussion of conflicts of interests, and a last-minute discussion of whether the matter had been appropriately brought before the Technical committee (and if so, under what authority), there s certainly schadenfreude to go around if you re into that sort of thing. However, through all this, the Technical Committee has been doing great work understanding the issues; it has been a pleasure to watch their hard work from the side lines. I d like to focus on one key question they ve found: how tightly can other software depend on the init system. Each init system offers some nice features. Upstart has an event triggering model. Systemd manages login sessions and at least in my opinion has a really nice service file format. Can I take advantage of these in my software. If I do, then users of my software might need to use a particular init system. Ian Jackson argues that we should give our users the choice of what init system to run. He reminds us that Debian is a community that supports diversity and diverse goals. We support multiple desktop systems, web browsers, etc. Wherever we can, we support people working on their goals and give developers the opportunity to make it work. When developers succeed, we give our users the opportunity to choose from among the options that have been developed. I think of Ian s argument as an appeal to part of our Social Contract. Clause 4 of the social contract begins:

Our priorities are our users and free software.
We will be guided by the needs of our users and the free software community. We will place their interests first in our priorities. We will support the needs of our users for operation in many different kinds of computing environments.

Ian is right that our users will be served by choice in init systems. However, this trade off needs to be balanced against the needs of the free software community. Diversity is an important goal but it should not come at the price of stagnation and innovation. If I want to avoid using init scripts because they don t provide restart on failure, because they are hard to write correctly, and because they don t provide access to modern security isolation techniques, I should be able to do that. If Systemd service files provide a superior solution, I should be able to work toward using them. If the desire for init system diversity shuts down my ability to find like-minded people who want to take advantage of improvements in init systems and work towards that goal, then we ve significantly damaged Debian as a forum for technical cooperation. Early in my work as a Debian Developer, Anthony Towns taught me an interesting principle. Those who do the work get significant influence over how and what gets done. Ian s right that we need to enable people to work towards choice in init systems. Those like-minded people need to be given a chance to find each other and pursue their goal. However the cost of success should be on the shoulders of those who value choice in init systems. It should not come at the cost of preventing people from depending on improvements in init systems. The best proposal I ve seen for balancing this is to enumerate stable interfaces that people want to use. Things like the logind interface or my favorite the service file interface. It needs to be possible to make another implementation of the interface. The interface needs to be stable enough that a dedicated team could have a chance of catching up with an implementation. At some point during the release cycle such interfaces would need to be frozen. However, I don t think it is reasonable to mandate that there are multiple implementations of the interface, only that there could be. The point is to give people a chance to work towards diversity in init systems, not to force it down peoples throats kicking and screaming until they leave the project or ignore our processes. Steve Langasek and Colin Watson seem to be working towards this. It s possible there s another approach besides interfaces. My main concern is the same as Ian s: maintain Debian as a forum for people to pursue their goals and work together. I suspect we see the conflict in these goals differently. I hope that we as a project can explore this conflict and figure out where we have common ground. I commit to exploring how we can work towards init choice in my frameworks; I ask those who prioritize init choice to commit to explore how we can take advantage of new features in their framework.

The Debian Technical Committee has concluded that all of the available init systems were missing a key technical feature, and, in the spirit of free software, has gone ahead and fixed that lack. All init systems currently packaged for Debian can now double as test case generators for Condorcet vote analysis. I'm not sure if this counts as an NMU. In the event that the Debian project votes on a General Resolution on the default init system, this blog post should be considered striken in its entirety and replaced by the most complex email message from Anthony Towns that the reader can find.

Sam Hartman is a Debian developer since 2000. He has never taken any sort of official role within Debian (that is besides package maintainer), yet I know him for his very thoughtful contributions to discussions both on mailing lists and IRL during Debconf. Until I met him at Debconf, I didn t know that he was blind, and the first reaction was to be impressed because it must be some tremendous effort to read the volume of information that Debian generates on mailing lists. In truth he s at ease with his computer much like I am although he uses it in a completely different way. Read on to learn more, my questions are in bold, the rest is by Sam. Raphael: Who are you? Sam: I m Sam Hartman. I m a 35-year-old software engineer. I am a principal consultant and co-owner of a small consulting company, called Painless Security. I started using Debian in the mid 1990 s around the time of the bo release. I ended up deciding to join the project as a developer in 2000. Raphael: You re blind and yet you re using your computer as effectively as I am. Can you explain us how you setup your computer ? Sam: I gave a talk at Debconf9 on how my computer is set up; you can watch the video for full details. My main laptop runs Debian. I use the gnome-orca package as my primary screen reader. It speaks the Gnome desktop. It does a relatively good job of speaking Iceweasel/Firefox and Libreoffice. While it does speak gnome-terminal, it s not really good enough at speaking terminal programs that I am comfortable using it. So, I run Emacs with the Emacspeak package. Within that, I run the Emacs terminal emulator, and within that, I tend to run Screen. For added fun, I often run additional instances of Emacs within the inner screens. Raphael: Are there important problems in Debian in terms of accessibility to blind people? KDE documentation talks a lot about accessibility, but at least for blind users, the code completely fails to deliver. That means there are a lot of good packages a blind user cannot use. The non-free Adobe Flash player has some accessibility, but it could be a lot better. The free alternatives have none. The free PDF readers have basically no accessibility. You can use pdftotext, but you cannot actually read a PDF in a graphical application. It s way too easy for a misbehaving program to lock up the entire accessibility infrastructure. Gnash is a big culprit here: if my Iceweasel starts Gnash, there s a good chance that either Iceweasel or the entire desktop will appear to hang from an accessibility standpoint. Other programs, including gksu tend to fail in this way. Some of the more dynamic website features like pop up menus or selection lists are really difficult to find and click without causing them to disappear. This gets better and worse over time as the accessibility support in Iceweasel changes and as websites change. Raphael: What s your biggest achievement within Debian? Sam: I decided to be a Debian developer because I thought that in 2000, Debian support for using enterprise security and infrastructure software was lacking. Back then, any software that included crypto functionality was segregated into a special non-us archive. Some of the software was missing; I started by packaging MIT Kerberos for Debian. Other software had security or enterprise features disabled in the packages. At the height of working on this, I was maintaining krb5, some SASL modules, PAM, some PAM modules,OpenAFS, a version of and Ssh with Kerberos support. I was also involved in the effort to resolve the legal issues so that we could move security software into the main archive. I think this work has been a huge success. In fact, it s been such a success that other people are now doing most of the work. I still maintain the krb5 package. When I started I felt like I was pushing against the flow trying to get people to add patches, sometimes even having to fork a package. However, now, I can maintain just one component and there are enough others who shared my original goal that the work continues. These days, I m working on something that s an evolution of this enterprise security work. I m packaging Project Moonshot for Debian and Ubuntu. Project Moonshot is a response to the wide variety of identity management systems such as OpenID, Oauth, SAML, and the like. Moonshot works to create an identity management approach that works well both for the web and for client-server and other applications. The project is a lot of fun, but the role of Debian in the project is also interesting. One of the things we want to show with Moonshot is how our technology can be effective when it is integrated throughout a platform. To really show the potential it needs to work with as many applications in a given platform as possible. The open and collaborative nature of the Debian community makes it possible to introduce a new technology and have that technology evaluated based on its merits. We don t need huge business relationships or marketing deals to integrate our technology: we need some combination of doing the work ourselves and showing others the benefits of working with us. For someone trying to do innovative work, the Debian model is powerful. Raphael: If you could spend all your time on Debian, what would you work on? Sam: I d really love to work with the embedded Debian folks. The vision of a single source base that could be the stack from devices from small embedded devices all the way up to high-end servers is very appealing. Doing that with Debian involves a number of challenges. However with the right people working on meeting these challenges full-time, I think we could offer something promising. I d also love to have the time to contribute to project infrastructure: working on the release team, helping ftpmaster, that sort of thing. However I don t. I m just happy that so much of my consulting practice involves working on open-source software. Raphael: What s the biggest problem of Debian? Sam: There s something really not right about how we transition libraries from unstable to testing. Every time I get involved in a library transition I m shocked at how complicated it is and how disruptive it is both to testing and unstable. We need to look at technology and processes to break up the dependency snarles. For example we don t have good archive tools for keeping old versions of libraries around to ease transitions. If you haven t thought about this issue you ll probably say that I m being overly picky and this can t be the major problem for Debian. However, if you think about how much this impacts our ability to introduce things into unstable around the times of the freeze or about how much it slows the release process, you may begin to appreciate how big of an issue this is. Raphael: Is there someone in Debian that you admire for their contributions? Sam: There are a number of people who have been role models for me over the years. Anthony Towns really helped me understand a lot of what drives free-software projects and what needs to be true for positive motivation. Joey Hess showed us all that sometimes, social problems do have technical solutions. If the tools are so good that doing the right thing is far easier than any other course of action, quality improves.

---

Thank you to Sam for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook.

5 comments Liked this article? Click here. My blog is Flattr-enabled.

Out of curiosity, and because it is Sunday morning and I have a cold and can't get my brain to do anything tricky, I counted the number of candidates in each year's DPL elections.

Year	Count	Names
1999	4	Joseph Carter, Ben Collins, Wichert Akkerman, Richard Braakman
2000	4	Ben Collins, Wichert Akkerman, Joel Klecker, Matthew Vernon
2001	4	Branden Robinson, Anand Kumria, Ben Collins, Bdale Garbee
2002	3	Branden Robinson, Rapha l Hertzog, Bdale Garbee
2003	4	Moshe Zadka, Bdale Garbee, Branden Robinson, Martin Michlmayr
2004	3	Martin Michlmayr, Gergely Nagy, Branden Robinson
2005	6	Matthew Garrett, Andreas Schuldei, Angus Lees, Anthony Towns, Jonathan Walther, Branden Robinson
2006	7	Jeroen van Wolffelaar, Ari Pollak, Steve McIntyre, Anthony Towns, Andreas Schuldei, Jonathan (Ted) Walther, Bill Allombert
2007	8	Wouter Verhelst, Aigars Mahinovs, Gustavo Franco, Sam Hocevar, Steve McIntyre, Rapha l Hertzog, Anthony Towns, Simon Richter
2008	3	Marc Brockschmidt, Rapha l Hertzog, Steve McIntyre
2009	2	Stefano Zacchiroli, Steve McIntyre
2010	4	Stefano Zacchiroli, Wouter Verhelst, Charles Plessy, Margarita Manterola
2011	1	Stefano Zacchiroli (no vote yet)

Winner indicate by boldface. I expect Zack to win over "None Of The Above", so I went ahead and boldfaced him already, even if there has not been a vote for this year. Median number of candidates is 4.

AltOS 0.8 New Software and Firmware for Altus Metrum Devices Bdale and I are pleased to announce the release of AltOS version 0.8. AltOS is the core of the software for all of the Altus Metrum products. It consists of cc1111-based microcontroller firmware and Java-based ground station software. AltosUI New Features in the AltusMetrum Ground Station AltOS version 0.8 contains significant upgrades to the ground station software, AltosUI:

• Post-flight graphing tool. This lets you explore the behaviour of your rocket after flight with a scroll-able and zoom-able chart showing the altitude, speed and acceleration of the airframe along with events recorded by the flight computer. You can export graphs to PNG files, or print them directly.

• Real-time moving map which overlays the in-progress flight on satellite imagery fetched from Google Maps. This lets you see in pictures where your rocket has landed, allowing you to plan recovery activities more accurately.

• Wireless recovery system testing. Prep your rocket for flight and test fire the deployment charges to make sure things work as expected. All without threading wires through holes in your airframe.
• Optimized flight status displays. Each flight state now has it s own custom tab in the flight monitoring window so you can focus on the most important details. Pre-flight, the system shows a set of red/green status indicators for battery voltage, apogee/main igniter continutity and GPS reception. Wait until they re all green and your rocket is ready for flight. There are also tabs for ascent, descent and landing along with the original tabular view of the data.
• Monitor multiple flights simultaneously. If you have more than one TeleDongle, you can monitor a flight with each one on the same computer.
• Automatic flight monitoring at startup. Plug TeleDongle into the machine before starting AltosUI and it will automatically connect to it and prepare to monitor a flight.
• Exports Google Earth flight tracks. Using the Keyhole Markup Language (.kml) file format, this provides a 3D view of your rocket flight through the Google Earth program.

Continuing Features AltOS version 0.8 continues to provide the following features:

• Receive and log telemetry from a connected TeleDongle device. All data received is saved to log files named with the current date and the connected rocket serial and flight numbers. There is no mode in which telemetry data will not be saved.
• Download logged data from TeleMetrum devices, either through a direct USB connection or over the air through a TeleDongle device.
• Configure a TeleMetrum device, setting the radio channel, callsign, apogee delay and main deploy height. This can be done through either a USB connection or over a radio link via a TeleDongle device.
• Replay a flight in real-time. This takes a saved telemetry log or eeprom download and replays it through the user interface so you can relive your favorite rocket flights.
• Reprogram Altus Metrum devices. Using an Altus Metrum device connected via USB, another Altus Metrum device can be reprogrammed using the supplied programming cable between the two devices.
• Export Flight data to a comma-separated-values file. This takes either telemetry or on-board flight data and generates data suitable for use in external applications. All data is exported using standard units so that no device-specific knowledge is needed to handle the data.
• Speak to you during the flight. Instead of spending the flight hunched over your laptop looking at the screen, enjoy the view while the computer tells you what s going on up there. During ascent, you hear the current flight state and altitude information. During descent, you get azimuth, elevation and range information to try and help you find your rocket in the air. Once on the ground, the direction and distance are reported.

AltOS Firmware Update AltOS version 0.8 contains a minor firmware update for TeleMetrum to resolve an issue with main deployment. A mis-feature in the igniter firing code would delay main deployment by 2 seconds in some cases. Thanks to our contributors! We had a lot of help with this release:

• Anthony Towns wrote both the new data graphing interface and the moving map display.
• Bob Finch helped clean up our documentation, and provided flight testing for the firmware updates.

Future Plans A number of features are implemented or in process in the sources available in our publicly visible repository that are not part of the current stable release.

• A Kalman-filter based approach to apogee detection using more than just the baro sensor, so that we can safely control apogee ejection on flights to altitudes beyond the range of our baro sensor alone. Unlike the other items on the list, this will be a significant change to the in-rocket TeleMetrum firmware. It may therefore be a while before this becomes part of a stable firmware release.
• Motor characterization. Because TeleMetrum contains a high-resolution, high-frequency accelerometer, it is possible to take the data from that and compute an accurate thrust curve for the motor.
• Comprehensive PDF and/or HTML -based flight report. Construct a complete report of the flight suitable for publication on the web that includes graphs of the flight and details about motor performance etc.
• Publish flight data to the Altus Metrum web site. This will allow you to share your flight data with others, and let you download flights published by others.

There are any number of additions that could be made to this list; feel free to send along ideas that you ve got. Of course, all of this software is licensed under the GNU General Public License, so you can get the source and hack on it in the comfort of your own home.

I decided recently to publish interviews from Debian contributors and I picked Joey Hess as my first target. He s one of the few who have heavily influenced Debian by creating software that have become building blocks of the project, like the debian-installer (Joey uses the shorthand d-i to refer to it). My questions are in bold, the rest is by Joey (except for the additional information that I inserted in italics). Who are you? Hello, I m Joey Hess. I m one of the oldtimers in Debian. Actually, I just checked, and there are still up to nineteen active Debian Developers who joined the project before I did, in 1996. I got started fairly young, and am just 34 years old. I spend part of my time working with Lars Wirzenius, another Debian oldtimer, on Branchable. It makes it dead-easy for anyone to make a website that is built from Git, using my Ikiwiki engine to do wiki and blog style things. These days I spend the rest of my time working on free software, when I should really be looking for work to pay the bills. What s your biggest achievement within Debian or Ubuntu? I guess I m mostly known by Debian developers for writing debhelper and perhaps debconf. Probably founding the Debian Installer project has been a bigger impact for users. I m fairly equally proud of all three projects. But while it might sound corny, I am more proud of the accumulation of all the smaller things done in the context of Debian. It s more of a deep connection to the project. All the bugs fixed, and filed, and packages uploaded, and late night discussions, and just being a part of the larger project. What are your plans for Debian Wheezy? Hmm, I stopped thinking of Debian releases by code names back around Slink. So, few specific plans for Wheezy. The main thing I would like to help make happen in Debian next is Constantly Usable Testing, and it transcends releases anyway. The only specific plans I have for Wheezy are that there will probably be a new debhelper compat level, and I hope d-i will finally switch to using git. RH: You can learn more about Constantly Usable Testing in my article Can Debian offer a Constantly Usable Testing distribution?. If you could spend all your time on Debian, what would you work on? Getting Debian on all these computers that we carry around in our pockets and can t easily run apt-get on and hack on. Phones that is. It s a bigger problem than just Debian, but I think Debian needs to find a way to be part of the eventual solution. The FreedomBox concept at least hints at a way around the current situation with embedded computers in general. What s the biggest problem of Debian? I believe that the biggest problem is institutional, social and technological inertia. Every specific case of something that frustrates me about Debian today can be traced back to that. You contribute regularly to Debian mailing lists, yet I don t remember any aggressive/frustrated mail of you. How do you manage that? Are you avoiding heated discussions? Rats, sounds like all my wonderful flames of years past have been forgotten! Seriously though, after I noticed thread patterns, I started trying to avoid participating in the bad patterns myself. Now I limit myself to one expression of an opinion, and I don t care who gets the last word. If people can t be convinced, it s time to find another approach to the problem. Also, code talks. Most of the programs you wrote for Debian are in Perl. Do you regret this choice? I love that question! No, no regrets. My only concern is whether the language limits contributors or users. I have not seen Perl significantly limiting the use of anything except for debconf (we had to rewrite it in C for d-i). And I do not notice fewer contributions to my Perl-based code than to other code. I do sometimes regret when someone tells me they had to learn or re-learn Perl to work on something I wrote. But while I m enjoying writing new things in Haskell now, and while I hope it will mean less maintenance burden later, I don t think using Haskell will make it easier for others to contribute to my programs. Anyway, for me the interesting thing about writing a program is the problem it solves and the decisions made doing it. Choice of language is one of the less interesting decisions. Is there someone in Debian that you admire for his contributions? Well, lots. You re tempting me to throw a dart at a map. So arbitrarily, I ll say Anthony Towns. We could all learn something from how he s approached making big, fundamental changes, like introducing Testing, and making Debian Maintainers happen.

---

Thank you to Joey for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Subscribe to my newsletter and don t miss further interviews. You can also follow along on Identi.ca, Twitter and Facebook. PS: If you want to suggest me someone to interview, leave a comment or mail me.

3 comments Liked this article? Click here. My blog is Flattr-enabled.

Bernd Zeimetz:

From the other candidates I'd like to know their opinion and plans (if there are any) about license/copyright requirements in Debian.

I believe that intellectual property laws are wrong and should be abolished, so I am not particularly in favor of giving copyrights any more respect than necessary. However, I do not think that Debian is the place to undermine copyright, and that we should attempt to maintain standards which not only protect people from legal liability, but are also in the best interests of our developers and users. I do not know what level of nitpicking best serves our developers and users. A GR seems like an interesting way of determining at part of that. Anthony Towns:

What's your estimate of the current number of Debian users?

I do not have one. While I do acknowledge that the number of users does affect us in terms of potential contributor base, and scaling issues (including bug reporting), I do not think that the exact number is important at all to what we do. Also I would hope that whichever DPL is elected chooses to focus on practical matters rather than academic exercises. Kurt Roeckx:

I think that one of issues we have is that there is alot of work to be done by some teams, some of them even regularaly mail that they need more members, but they seem to have a hard time keeping the numbers up, burning the other team members out. What are your ideas to make sure those teams keep running?

• Diffusion of power throughout the rest of the project
• Delegation of people who do not seek to be core team members if they have the time, are willing, and would make good additions.

Note that I do not think it appropriate either for core teams to choose their own members or to reserve the privilege of demanding that a new person be able to work with them in a certain way. An ideal team member should be able to work with anyone. If the existing delegates are unable to work with new people, perhaps they are not ideal team members. Charles Plessy:

During the discussions that started after the GR, I suggested that the GR proposer should have more control about the options put to the vote. In particular, it would be useful if he can refuse an option that would disequilibrate the voting system. That would make him responsible for the success of the GR: discarding a popular option is taking the risk that the whole GR is refused and the option is accepted as a separate GR, which is the kind of public failure that I expect that people will avoid.

I do not think GR proposers should have any special control over general resolutions beyond what is Constitutionally granted at present. In fact, I would prefer if people did not try to propose multiple options or strategically craft ballots for any purpose. To me, the only honest method is to propose the outcome one genuinely wants to win, and let our strange and special brand of parliamentary procedure take care of the rest. As for the supermajority issue, I am not a huge fan of political word games. Debian distributes non-free software. We have a non-free section within our normal infrastructure, and while it is segregated and does not receive quite the same love that main does, it is still very much part of what we work on and distribute from our official infrastructure. So there is no amount of word lawyering, cognitive dissonance, or NM indoctrination that can make me believe that it is an honest statement to say that non-free is not part of Debian. Likewise I do not believe in redefining success or the meanings of Foundation Documents. I do not think that we should allow ourselves to pass any kind of magical resolution that contravenes the DFSG or Social Contract without actually modifying them. That seems utterly absurd to me, and I have seen what has become of the United States government.

Guess I got your attention with that title? :) Anyways, britney is no longer. And with her testing died too. Ok, not really, but todays work was hand over all of testing and put it where it should have been for years - in the hands of the release team. For those not following the “inner” workings of stuff:

• Testing is the place where our next stable release gets prepared. It is managed by a software called britney, which was written by Anthony Towns.
• Britney was, until today, run by the ftpmasters. The release team had no way to change anything, besides their hint files. They couldn’t change code, couldn’t rerun it themselve if needed, nothing. (Somewhere in the past, for some period, some of the release team was able to do extra britney runs, AFAIK, but thats long ago).

As the above is surely not nice, especially as Anthony (and as such the one in ftpmaster who really fully knew britney) left the ftpmaster group, all of the code managing testing now lives with the release team. They now just prepare a file in a special format and then tell us (via a ssh trigger) to read that in. Which simply sets testing to the new state. This new way of doing things has multiple advantages:

• The release team can run britney whenever they want to. It no longer needs to wait for a cronjob at some externally defined time or ping an ftpmaster to do an extra run. That should be pretty helpful, especially when we are near to a release again.
• They can do code changes or even replace the current britney with a totally new one. (There seems to be a new one available that does the same as the current one, just way faster. Or even more. Which never seems to have got any comment from our side. Badbad. Well, now they can play with it if they want to.). As long as they output valid data for us to import, it is now completly their decision how it gets generated.

The above action did need surprisingly few code changes in dak. It basically was

• disable the cronjob for britney as user dak
• Add a script that the release team now pushes via ssh. This script reads in the above mentioned special file and sets the database.
• Change process-accepted to write the urgencies to a different directory. And no longer try to mkdir this directory, but warn if it doesn’t exist (or is not writeable by dak) and then use a temporary one for the run.
• Move the transitions.yaml to a different location and add a redirect in apache, so everyone can still access it from outside. (It has to live in a ftpmaster controlled directory, thats all).
• Eliminate a britney call from our daily cronjob.

Comments: 8

I’ve noted in the past that Debian has deliberately enshrined in its constitution some rather serious principal-agent problems. By and large this isn’t a bad thing, since there isn’t the consensus within the Debian community to support the “benevolent dictator for life” model of decision-making—if you want that, well that’s what Ubuntu and Daddy Warbucks is for. But it does mean that sometimes the caca hits the fan when a Debian project leader does exercise his powers, as our now-former DPL did earlier this week just before the end of his term of office (by my estimate, just over one hour and 27 minutes before). John Adams would be proud. So we have three related issues in my mind:

• As a matter of general principle, lame-duck DPLs shouldn’t be making appointments. This issue is ameliorated somewhat because DPL delegations—unlike “midnight judges”—can be revoked at any time, but it strikes me that whatever legitimacy a DPL has from the developers evaporates once a new DPL-elect has been designated. I can only speculate why this happened in this case, so I won’t bother.
• Second, while Debian has a very strong tradition of developer sovereignty, with many aspects of the project being self-organized rather than originating with appointments from upon high, it seems to me that certain aspects of core infrastructure can’t be managed in this way.
• Third, the appointment does little to relieve the excessive concentration of power in the core of Debian; if anything, Anthony Towns’ apparent resignation in the wake of Jörg’s appointment worsens the situation. Ensuring there is proper vetting of people with access to important infrastructure is important, but at the same time I find it difficult to believe that there are only a half-dozen or so Debian developers who are trustworthy enough to be system administrators, account managers, or archive maintainers (several of them occupying overlapping positions). That, rather than a lack of technical tools, has been a problem of note within Debian since, oh, the days of my youthful vigor within Debian (which are long since past).

In any event, congratulations to all the new Debian developers—and I’ll avoid pondering for too long why one person’s appointment to an unrelated group would suddenly break the logjam of developer application approvals.

One of the freedoms I value is the freedom to choose what you spend your time on and who you spend it with. And while I’ve spent a lot of time arguing that people in key roles in Debian still have those freedoms (hey, 2.1(1), don’t you know), reality these days seems to be otherwise. But hey, solving that quandry just requires a mail to DSA. To folks on the core teams I’ve been involved with: it’s been a pleasure and an honour working with you; if not always, at least mostly. Best of luck, and I hope y’all accept patches.

Anthony Towns writes about using an improved version of jigdo to download CD/DVD images [1]. His improvement is basically to pipeline operation for better performance. Jigdo (the Jigsaw download) is a tool to download a set of files and then use them to create a CD or DVD image [2]. The idea is that most web sites that have CD or DVD images also have a collection of files which comprise the DVD image available. This removes the need to store the data twice (wasting disk space on mirrors and in some situations breaking web caching). I have never used jigdo, and for all Debian installations in recent times (the last few years at least) I download a small net-inst CD image and then have it download the other files from the net. I have Squid set up to cache large objects so this doesn’t waste too much of my precious network bandwidth (which is limited and expensive in Australia). Now I’m thinking about what the optimum method for doing installs might be. One thing that would be good would be to support multiple repositories, the packages have unique file names and checksums so it should be possible to check one repository and then check another if it’s not working. I don’t mean multiple “deb” lines in the APT configuration. What I would like to do is to have an NFS file server or web server with an archive of downloaded packages and have APT check there first before downloading a file. So APT could get a list of packages from the net and then get the actual files locally if they are available. The next thing that would be good is the ability to create a CD or DVD image dynamically and to store all temporary files. So I could download files from the repository and create a DVD image with just the packages that I need. Every time I create a DVD image my sub-set of the Debian archives would increase and the number of files actually downloaded in the creation process would be reduced. The effect would be to incrementally create a local mirror of the Debian repository. Then I would like to see a two-stage DVD install process. I would like to boot from a CD or DVD and start the install and then have it give a list of files needed (which could be stored on a USB device or floppy) to create further CDs or DVDs for installation. One situation where this could have been beneficial was when I was doing an emergency install of CentOS. I did the first part of the install (selecting packages etc) to determine which CDs were needed. It turned out that almost all CDs were needed even though some of the CDs had only a few files that I was installing. If the installer could have written a list of packages to a USB device then I could have downloaded just those packages and got the install working a lot sooner. It seems to me that it’s fairly common to do one test install and then do some dozens of other installs with the same settings. So the ability to create a DVD of exactly the needed files for the other dozens of installs would be a great benefit. Now this is just random commentary, unfortunately don’t have time to do any coding on this. But it seems obvious that something has to be done to improve the situation for developers and IT staff who need some degree of mirroring of the Debian package pool but who can’t do a full mirror. Back in 1996 I was able to mirror Debian over a 28K8 modem link and fit it on what was a reasonable hard drive by the standards of the day (incredibly tiny by today’s standards). Now I can’t practically mirror Debian over an Australian cable broadband connection and even by the standards of about 4 years ago (the age of most of my hard disks) the space requirements are significant. I hope this post helps inspire some interest in developing these features. As delays in joining Debian [3] is the topic of the day it should be noted that work on preparing DVD images can easily be done by people who are not DDs. Such software should work from Debian archives without requiring any changes to them, and thus nothing special is needed from the Debian project to start work.

• [1] http://azure.humbug.org.au/~aj/blog/2008/04/15#2008-04-15-jigdodl


• [2] http://atterer.net/jigdo/


• [3] http://etbe.coker.com.au/2008/04/17/motivation-and-perspective/

Share This

Last month we had a brief discussion on debian-devel about what images would be good to have for lenny – we’re apparently up to about 30 CDs or 4 DVDs per architecture, which over 12 architectures adds to about 430GB in total. That’s a lot, given it’s only one release, and meanwhile the entire Debian archive is only 324GB. The obvious way to avoid that is to make use of jigdo – which lets you recreate an iso from a small template and the existing Debian mirror network. I’ve personally never used jigdo much, half because I don’t usually use isos anyway, but also because the few times I have tried jigdo it always seemed really unnecessarily slow. So the other day I tried writing my own jigdo download tool focussed on making sure it was as fast as possible. The official jigdo download tool, ttbomk, is jigdo-lite – which you give a .jigdo file, and the url of a local mirror. It then downloads the first ten files using wget, and once they’re all downloaded, it calls jigdo-file to get them merged into the output image. This gets repeated until all the files have been downloaded. By doing the download in sequence like this, you miss out on using your full network connection in two ways: one during the connection setup latency when starting to download the next package, and also while jigdo-lite stops downloading to run jigdo-file. And if you’ve got a fast download link, but a slower CPU or disk, you can also find yourself constrained in that you’re maxing those out while running jigdo-file, but leaving them more or less idle while downloading. To avoid this, you want to do multiple things at once: most importantly, to be writing data to the image at the same time as you’re downloading more data. With jigdodl (the name I’ve given to my little program), I went a little bit overboard, and made it not only do that, but also manage four downloads and the decompression of the raw data from the template. That’s partly due to not being entirely sure what needed to be done to get a speedy jigdo program, and partly because the communicate module I’d just written to deal with this sort of parallelism making that somewhat natural. In the end, it works: from wireless over ADSL to my ISP’s Debian mirror, I get the following output:

Jigsaw download:
  Filename: debian-40r3-amd64-CD-1.iso
  Length:   675477504
  MD5sum:   d3924cdaceeb6a3706a6e2136e5cfab2
Total: 679 s; d/l: 586 MB at 883 kB/s; dump: 57 MB at 57 MB/s          
Finished!

which is only slightly short of maxing out my downstream bandwidth, taking a total of about 11m20s. Running jigdodl with a closer mirror works pretty well too, though evidently some of my more recent changes weren’t so great, because I’ve gone from 9153 kB/s on a 100 Mbps link down to 7131 kB/s or lower. The CPU usage also seems a bit high, hovering at between five to ten percent at 900 kB/s. For comparison, running jigdo-lite on the same file took 17m41s, which is about 566 kB/s, with the overhead being about 6m20s. What that means is if I doubled my bandwidth to about 20Mbps, jigdodl would halve its time for the download to about 5m50s, while jigdo-lite would still have about the same non-download overhead, and thus take 12m10, which is still 69% of its original speed. Going from 10Mbps ADSL speed to 100Mbps LAN gets jigdodl down to 1m31s (13% of the time, with optimal being 10%), while jigdo-lite would be expected to still be about 7m51s (43% of its original time). I suspect the next thing to do is to rewrite the downloading code to use python-curl instead of running curl, and thus downloading multiple files with a single connection, and tweaking the code so that it writes the file in order, rather than updating whichever parts are ready first. Anyway, debs are available for anyone who wants to try it out, along with source in the new git source package format.

In a couple of days, DPL-elect Steve McIntyre takes over as DPL, after being elected by around four hundred of his peers… Because I can’t help myself, I thought I might poke at election numbers and see if anything interesting fell out. First the basics: I get the same results as the official ones when recounting the vote. Using first-past-the-post, Steve wins with 147 first preference votes against Raphael’s 124, Marc’s 90 and NOTA’s 19 (with votes that specify a tie for first dropped). Using instant-runoff / single transferable vote, the winner is also Steve, with NOTA elimited first and Marc collecting collecting 5 votes, Steve 4 and Raphael 2, followed by Marc getting eliminated with Steve collecting 50 votes, against Raphael’s 26. So, as usual, different voting systems would have given the same result, presuming people voted in basically the same way. NOTA really didn’t fare well at all in this election, with a majority of voters ranking it beneath all candidates (268 of 401, 53.5%). For comparison, only 18 voters ranked all candidates beneath NOTA, with 9 of those voters then ranking all candidates equally. (For comparison, in 2007, 312 of 482 voters (about 65%) ranked some candidate below NOTA, though that drops to 225 voters (47%) if you ignore voters that just left some candidates unranked. Only 98 voters (20%) voted every candidate above NOTA) With NOTA excluded from consideration, things simplify considerably, with only 13 possible different votes remaining. Those come in four categories: ranking everyone equal (17 votes, 9 below NOTA as mentioned above, and 8 above NOTA), ranking one candidate below the others (13 votes total, 7 ranking Raphael last, 3 each for Steve and Marc), ranking one candidate above the others (66 votes; 30 ranking Steve first, 18 each ranking Raphael and Marc first), and the remainder with full preferences between the candidates:

     70 V: 213
     63 V: 123
     56 V: 132
     52 V: 231
     38 V: 312
     26 V: 321

The most interesting aspect of that I can see is that of the people who ranked Raphael first, there was a 1.8:1 split in preferring Steve to Marc, and for those who preferred Marc first, there was a 2:1 split preferring Steve to Raphael. For those who preferred Steve, there was only a 1.1:1 split favouring Raphael over Marc. I think it’s fair to infer from that that not only was Steve the preferred candidate overall, but that he’s considered a good compromise canidate for supporters of both the alternative candidates (though if all the people who ended up supporting Steve hadn’t been voting, Raphael would have won by something like 26 votes (129:103) with a 1.25:1 majority; if they had been voting, but Steve hadn’t been a candidate, Raphael’s margin would’ve increased absolutely to 33 votes (192:159) but decreased in ratio to a 1:1.2 majority.

One of the loveliest things about Unix is the select() function (or its replacement, poll()), and the way it lets a single thread handle a host of concurrent tasks efficiently by just using file descriptors as work queues. Unfortunately, it can be a nuisance to use – you end up having to structure your program as a state machine around the select() invocation, rather than the actual procedure you want to have happen. You can avoid that by not using select() and instead just having a separate thread/process for every task you want to do – but that creates a bunch of tedious overhead for the OS (and admin) to worry about. But magically making state machines is what Python’s generators are all about; so for my little pet project that involves forking a bunch of subprocesses to do the interesting computational work my python program wants done, I thought I’d see if I could use that to make my code more obvious. What I want to achieve is to have a bunch of subprocesses accepting some setup data, then a bunch of two byte ids, terminated by two bytes of 0xFF, and for each of the two byte inputs to output a line of text giving the calculation result. For the time being at least, I want the IO to be asynchronous: so I’ll give it as many inputs as I can, rather than waiting for the result before sending the next input. So basically, I want to write something like:

def send_inputs(f, s, n):
	f.write(s) # write setup data
	for i in xrange(n):
		f.write(struct.pack("!H", i))
	f.write(struct.pack("!H", 0xFFFF))
def read_output(f):
	for line in f:
		if is_interesting(line):
			print line

Except of course, that doesn’t work directly because writing some data or reading a line can block, and when it does, I want it to be doing something else (reading instead of writing or vice-versa, or paying attention to another process). Generators are the way to do that in Python, with the “yield” keyword passing control flow and some information back somewhere else, so adopting the theory that: (a) I’ll only resume from a “yield” when it’s okay to write some more data, (b) if I “yield None” there’s probably no point coming back to me unless you’ve got some more data for me to read, and (c) I’ll provide a single parameter which is an iterator that will give me input when it’s available and None when it’s not, I can code the above as:

def send_inputs(_):
	# s, n declared in enclosing scope
	yield s
	for i in xrange(n):
		yield struct.pack("!H", i))
	yield struct.pack("!H", 0xFFFF)
def read_output(f):
	for line in f:
		if line is None: yield None; continue
		if is_interesting(line):
			print line

There’s a few complications there. For one, I could be yielding more data than can actually be written, so I might want to buffer there to avoid blocking. (I haven’t bothered; just as I haven’t worried about “print” possibly blocking) Likewise, I might only receive part of a line, or I might receive more than one line at once, and afaics a buffer there is unavoidable. If I were doing fixed size reads (instead of line at a time), that might be different. So far, the above seems pretty pleasant to me – those functions describe what I want to have happen in a nice procedural manner (almost as if they had a thread all to themselves) with the only extra bit the “None, None, continue” line, which I’m willing to accept in order not to use threads. Making that actually function does need a little grunging around, but happily we can hide that away in a module – so my API looks like:

p = subprocess.Popen(["./helper"], stdin=PIPE, stdout=PIPE, close_fds=True)
comm = communicate.Communication()
comm.add(send_inputs, p.stdin, None)
comm.add(read_output, None, p.stdout, communicate.ByLine())
comm.communicate()

The comm.add() function takes a generator function, an output fd (ie, the subprocess’s stdin), an input fd (the subprocess’s output), and an (optional) iterator. The generator gets created when communication starts, with the iterator passed as the argument. The iterator needs to have an “add” function (which gets given the bytes received), a “waiting” function, which returns True or False depending on whether it can provide any more input for the generator, and a “finish” function that gets called once EOF is hit on the input. (Actually, it doesn’t strictly need to be an iterator, though it’s convenient for the generator if it is) The generator functions once “executed” return an object with a next() method that’ll run the function you defined until the next “yield” (in which case next() will return the value yielded), or a “return” is hit (in which case the StopIteration exception is raised). So what we then want to do to have this all work then, is this: (a) do a select() on all the files we’ve been given; (b) for the ones we can read from, read them and add() to the corresponding iterators; (c) for the generators that don’t have an output file, or whose output file we can write to, invoke next() until either: they raise StopIteration, they yield a value for us to output, or they yield None and their iterator reports that it’s waiting. Add in some code to ensure that reads from the file descriptors don’t block, and you get:

def communicate(self):
    readable, writable = [], []
    for g,o,i,iter in self.coroutines:
        if i is not None:
            fcntl.fcntl(i, fcntl.F_SETFL, 
                        fcntl.fcntl(i, fcntl.F_GETFL)   os.O_NONBLOCK)
            readable.append(i)
        if o is not None:
            writable.append(o)
    
    while readable != [] or writable != []:
        read, write, exc = select.select(readable, writable, [])
        for g,o,i,iter in self.coroutines:
            if i in read:
                x = i.read()
                if x == "": # eof
                    iter.finish()
                    readable.remove(i)
                else:
                    iter.add(x)
            if o is None or o in write:
                x = None
                try:
                    while x is None and not iter.waiting():
                        x = g.next()
                    if x is not None:
                        o.write(x)
                except StopIteration:
                    if o is not None:
                        writable.remove(o)
    return

You can break it by: (a) yielding more than you can write without blocking (it’ll block rather than buffer, and you might get a deadlock), (b) yielding a value from a generator that doesn’t have a file associated with it (None.write(x) won’t work), (c) having generators that don’t actually yield, and (d) probably some other ways. And it would’ve been nice if I could have somehow moved the “yield None” into the iterator so that it was implicit in the “for line in f”, rather than explicit. But even so, I quite like it.

One of the challenges maintaining the Debian archive kit (dak) is dealing with Debian-specific requirements: fundamentally because there are a lot of them, and they can get quite hairy – yet at the same time, you want to keep them as separate as possible both so dak can be used elsewhere, and just so you can keep your head around what’s going on. You can always add in hooks, but that tends to make the code even harder to understand, and it doesn’t really achieve much if you hadn’t already added the hook. However, dak’s coded in python, and being an interpreted language with lots of support for introspection, that more or less means there’s already hooks in place just about everywhere. For example, if you don’t like the way some function in some other module/class works, you can always change it (other_module.function = my_better_function). Thus, with some care and a bit of behind the scenes kludging, you can have python load a module from a file specified in dak.conf that can both override functions/variables in existing modules, and be called directly from other modules where you’ve already decided a configurable hook would be a good idea. So, at the moment, as a pretty simple example there’s an init() hook invoked from the main dak.py script, which simply says if userext.init is not None: userext.init(cmdname). But more nifty is the ability to replace functions, simply by writing something like:

# Replace process_unchecked.py's check_signed_by_key
@replace_dak_function("process-unchecked", "check_signed_by_key")
def check_signed_by_key(old_chk_key):
    changes = dak_module.changes
    reject = dak_module.reject
    ...
    old_chk_key()

That’s made possible mostly by the magic of python decorators – the little @-sign basically passes the new check_signed_by_key function to replace_dak_function (or, more accurately, the function replace_dak_function(...) returns), which does the dirty work replacing the function in the real module. To be just a little bit cleverer, it doesn’t replace it with the function we define, but its own function with simply invokes our function with an additional argument to whatever the caller supplied, so we can invoke the original function if we choose (the old_chk_key parameter – the original function takes no arguments, so our function only takes one). Right now, we don’t do much interesting with it; but that should change once Ganneff’s little patch is finished, which should be RSN… Hopefully, this might start making it easier to keep dak maintained in a way that’s useful for non-Debian installs – particularly if we can get to the point where hacking it for Debian generally just implies changing configuration and extension stuff – then we can treat updating all the real scripts as a regular software upgrade, just like it is outside Debian.

Continuing from where we left off… The lower bound for me becoming a DD was 8th Feb ‘98 when I applied; for comparison, the upper bound as best I can make out was 23rd Feb, when I would have received this mail through the debian-private list:

Resent-Date: 23 Feb 1998 18:18:57 -0000
From: Martin Schulze 
To: Debian Private 
Subject: New accepted maintainers
Hi folks,
I wish you a pleasant beginning of the week.  Here are the first good
news of the week (probably).
This is the weekly progress report about new-maintainers.  These people
have been accepted as new maintainer for Debian GNU/Linux within the
last week.
[...]
Anthony Towns <ajt@debian.org>
    Anthony is going to package the personal proxy from
    distributed.net - we don't have the source... He may adopt the
    transproxy package, too.
Regards,
        Joey

I never did adopt transproxy – apparently Adam Heath started fixing bugs in it a few days later anyway, and it was later taken over by Bernd Eckenfels (ifconfig upstream!) who’s maintained it ever since. Obviously I did do other things instead, which brings us back to where we left off…

---

• Crypto in main: It’s getting hard to remember the days when strong crypto software was classed as munitions and even getting a web browser to do SSL properly was a legal challenge; but for a long while that’s how it was, and as a result we had to keep all our security tools hosted separately to our main, US-based archive. The laws for that changed in early 2000, introducing a new exemption to the prohibition for software whose source code was available royalty-free (I think they called it “public source”). Unfortunately there were still drawbacks: it wouldn’t apply to some stuff in non-free, it wasn’t clear how far you had to go with the source code (is libssl’s source enough, or do you need the source for the apps that use ssl too?), and the biggest problem was you had to notify the export bureau and the NSA for everything you wanted to distribute. And all of that’s couched in the sort of confusing language you’d expect to find in, well, export regulations. So we ended up getting nowhere on merging crypto into the main archive for a long while because we couldn’t be confident of what we actually needed to do. We’d more or less already missed the opportunity to do the merge for the potato (2.2) release because it was frozen more or less exactly when the new regulations came out, and according to my mail archives Ben Collins (DPL at the time) got a team organised to review the legal issues by mid-May 2001. By July 2001, though, we still weren’t really getting anywhere; the lawyer we were talking to was looking at the ENC exception (which was for closed source stuff, and requires an active review by the BXA), instead of the TSU exception (which just requires notification). While it wasn’t terrible legal advice – following the ENC exception had a lower risk of misunderstandings on our part leading to problems later – it wasn’t useful in achieving our aims, ie, getting crypto stuff into main with a minimal level of ongoing effort, as well as a minimal level of risk. So moving onto August, we’d moved on from the lawyer we’d been using for this and other issues, to talking to a lawyer who specialised in export issues (and whose time was being covered by HP). That actually got us somewhere, with the end results being a matter of public record – we got the advice we wanted by November, we got a new incoming system so we could make sure we didn’t publish anything we hadn’t advised the NSA and BXA about, and most of all, we got cryptographic software in main for woody’s release. ssh at standard priority, apt by default checking archive signatures before downloading code to run as root, iceweasel coping with https:// urls out of the box all suddenly became possible, rather than artificially blocked due to pointless legal constraints. Yay! The way we notified BXA/NSA was somewhat interesting – we ended up notifying once for each package, when it passed through NEW: which is easy to automate, and limits the amount of software we block from review by other developers as best we can. OTOH, it also means a lot of notifications: the initial notification was a huge box of paper that got faxed off (there should be pictures somewhere on the net, but I don’t know where), and the vague hope that the ongoing reporting would not only comply with the regs, but manage to be something of a denial-of-service by compliance. As it turns out it apparently got noticed, with the following mail arriving in September 2005:

  From: Encryption 
  Date: Fri, 23 Sep 2005 12:51:02 -0400
  Subject: Addition to Debian Source Code updates
  Dear Mr. Collins,
  Your diligent reporting of updates to the Debian source code to the Bureau
  of Industry and Security has been an excellent example of perspicacity.
  However, you no longer need to do so per the following update to the EAR:
  [...]
  

  (The mails have always been sent with Ben's name attached) That mail was followed by another suggesting a phone conversation; unfortunately nobody seemed to notice the mail, at least until a year later (to the day!) when Joerg pointed out he was getting bounces form the BXA email address we'd been using... (Since then, based on the above mail, we've been recording notifications, but not passing them on; I sent a reply to the mails above indicating that, but didn't receive a response. There's been about 5000 notifications since then)
• Security and the architecture explosion: with crypto-in-main out of the way, along with lots of RC bug squashing, woody was looking good to come out in April or May, but unfortunately that all fell apart because of a lack of automation for handling security updates. Taking a step back, woody managed to introduce five new architectures compared to the previous release -- HP's pets, hppa and ia64; the IBM mainfram arch, s390, and the SGI/embedded mips and mipsel chips. That came just short of doubling our supported architectures (which had been i386, m68k, alpha, arm, sparc and powerpc), and was working really impressively well: testing was watching that they kept in sync, and the buildd network was keeping them all up to date with the latest source uploads, and the toolchain maintainers were making sure new kernels and compilers didn't cause massive breakages. It was great fun to be a part of, and I'm not sure how much is just blurry memories, but from here, it sure feels like it was a golden age on that score... Anyway, all the extra architectures freaked the security team out, given they were already having to worry about ssh'ing to half a dozen machines and manually do builds, and fix them when they broke, and to deal with that there'd been plans afoot for "rbuilder" to get rolled out, which never actually happened. All of which means it escalated from being the security team's problem, to a problem for everyone, particularly the release manager... The result of which meant knuckling down and coming up with a new security build infrastructure, which involved doing a dak install for the security archive, then building on the changes just rolled out for crypto-in-main to allow for packages to be autobuilt before actually being included in the archive, then updating official buildds for all the various architectures to target stable-security and testing-security and prioritise them and upload to the right place. But as stressful and last minute as that was, at least it worked, and woody made it out the door on July 20th 2002 -- and it's a good thing it worked, given woody ended up getting security support until June 30th 2006 -- the usual year after it's successor is released -- for four years support. (The same rule applied to sarge has resulted in two years and ten months' security support; eighteen month release cycles in general should result in two years and six months of support) Since then, with security managed by dak, there's been a few more bits I've been involved in. The most interesting of which (to my mind) was splitting the security upload queue from being completely secret and restricted to folks who are allowed to access updates prior to public announcement, into separate secret and non-secret queues. That was December 2005, and since then we've gone from having Joey Schulze basically being the sole person doing security updates to two separate teams, with ten new members. It's not without it's problems -- particularly for packages that get (mis)classified as NEW, or that get lost on their way for inclusion in the next stable point update, but it's nevertheless lightyears from where we were in April 2002, or even November 2005.
• Playing other people for suckers: with woody released, the whole "redo to release process" thing was essentially done -- it had been implemented, it had worked all the way to getting a release out, and in a fair few respects had improved things. Instead of the eight months of hand-approved uploads for the potato freeze, things had gone it automatically, albeit with gradually stricter constraints, right up until April; and even when the security problems delayed things, the manual updates only lasted a bit under three months. So at that point I was pretty happy, and looking forward to making things work a bit better and basically passing the buck to someone else. The first bit of buck passing that worked exceptionally well was jumping on Colin Watson, who'd been offering some debbugs patches on IRC, and after a delightfully brief discussion with some of the other debbugs folks, got getting him added to the group and able to make changes directly on the site. His advogato post the next day was:

  Yesterday I was invited onto the group that administers the Debian bug tracking system. I can't decide whether to be intimidated or excited. There's a great deal to do: my first couple of major projects look set to be integrating some of the existing QA interfaces to release-critical bug tracking, and developing a tool to help us edit spam out of bug histories.

  There's a lot to be said for grabbing people at the point where they're still intimidated by what they're joining; sure there's a definite risk that they'll break something you'll have to clean up, but that's pretty easy to deal with, and well worth all the extra excitement and energy that comes with a little bit of nerves. I did a talk on debbugs a few years later, at the 2005 DebConf, at which point we managed to entice Pascal Hakim and Don Armstrong into signing up, if I'm remembering right. At the same time Colin rolled out his implementation of version tracking support for debbugs (which I'd specced up a while ago, and helped design, but cannily avoided actually coding). Shortly after, Pasc put together bug subscriptions, and these days Don's pretty much the primary maintainer of debbugs. How cool is that? Passing on release managership was a bit less fortuitous though; it's a big job, and it's not something that you can offer "patches" for, generally. So after a bit of cogitating on just how to do it, I went with an open recruitment call in March 2003, and then a bunch of practical assignments to get the volunteers used to what RM work actually entailed and to see whether they actually like it. The recruitment call was mostly focussed on all the negatives, half to avoid people volunteering to be RM for the power (to decide which release policy wins, to tell maintainers what to do, to drop packages, or to just decide what name/number the next release gets) instead of wanting to actually do the work to get the next release out and keep developers happy; and the other half was the theory that if you make a job look hard, technical, but not terribly risky, it starts looking interesting to the right kind of people. The assignments generally were two parters -- first to introduce some stuff about release management that they mightn't be familiar with, and the second a bunch of problems that needed resolving that'd use those skills. The first set was RC bug fixing, followed by working out why packages aren't getting into testing, followed by dealing with packages with mutual dependencies and removing packages. At some point that evolved into the testing scripts accepting "hints" from the release team as to what packages to try in combination, or which packages to force out of testing. That turned out pretty successful, with Steve Langasek, Colin Watson and Joey Hess becoming official release assistants in August, and Colin and Steve replacing me as RM in mid-2004. It was also pretty satisfying to see Andi Barth do some more release assistant recruiting in 2005 following much the same model, and indeed using much of my mail verbatim. You can't really ask for better than that, I figure.

Geez, this was meant to be briefer...

So, sometime over the past few weeks I clocked up ten years as a Debian developer:

From: Anthony Towns <aj@humbug.org.au>
Subject: Wannabe maintainer.
Date: Sun, 8 Feb 1998 18:35:28 +1000 (EST)
To: new-maintainer@debian.org
Hello world,
I'd like to become a debian maintainer.
I'd like an account on master, and for it to be subscribed to the
debian-private list.
My preferred login on master would have been aj, but as that's taken
ajt or atowns would be great.
I've run a debian system at home for half a year, and a system at work
for about two months. I've run Linux for two and a half years at home,
two years at work. I've been active in my local linux users' group for
just over a year. I've written a few programs, and am part way through
packaging the distributed.net personal proxy for Debian (pending
approval for non-free distribution from distributed.net).
I've read the Debian Social Contract.
My PGP public key is attached, and also available as
<http://azure.humbug.org.au/~aj/aj_key.asc>.
If there's anything more you need to know, please email me.
Thanks in advance.
Cheers,
aj
-- 
Anthony Towns <aj@humbug.org.au> <http://azure.humbug.org.au/~aj/>
I don't speak for anyone save myself. PGP encrypted mail preferred.
On Netscape GPLing their browser:  How can you trust a browser that
ANYONE can hack? For the secure choice, choose Microsoft.''
        -- <oryx@pobox.com> in a comment on slashdot.org

Apparently that also means I’ve clocked up ten and a half years as a Debian user; I think my previous two years of Linux (mid-95 to mid-97) were split between Slackware and Red Hat, though I couldn’t say for sure at this point. There’s already been a few other grand ten-year reviews, such as Joey Hess’s twenty-part serial, or LWN’s week-by-week review, or ONLamp’s interview with Bruce Perens, Eric Raymond and Michael Tiemann on ten years of “open source”. I don’t think I’m going to try matching that sort of depth though, so here are some of my highlights (after the break).

---

• Starting small: my first package was distributed-net-pproxy, which would claim a bunch of work units which could then be distributed over the local LAN. Useful if you’re in Australia in the mid-90s and being charged by the minute for Internet access. distributed.net didn’t allow general distribution, so I asked for (and received) explicit permission to include it in Debian. (distributed.net had a ten year anniversary last year too)
• Diving straight in: I got sucked in to the mailing lists pretty quickly, and within a couple of months was up to my neck trying to redesign the way we did releases; oddly enough that didn’t turn out quite so easy, but at least it ended up with some concrete proposals within a couple of months, pretty much synchronously with the hamm (2.0) release going out (and, I guess, about a year after I’d started using Debian…). At around the same time was, I think, my first recorded comment about trade offs regarding freeness…
• cruft: My first bit of Debian specific code was cruft (that its name, not it’s value…), which worked okay as a prototype, but got snagged up in debian-policy trying to get packages to make a note of files they’ll put on the system without telling dpkg (eg, /etc/passwd, /var/cache/apt/archives/*.deb, etc). That pretty much sapped my interest in it, and cruft stagnated until Marcin Owsiany picked it up in 2005.
• Played for a sucker, part one: a few months later I was messing around doing QA stuff and fixing bugs and as part of that did an NMU of netbase (which at the time directly contained all sorts of important tools like ping and inetd directly). That went something like:

  From: Anthony Towns
  Date: Sat, Nov 21, 1998
  There are a few bugs accumulating against the netbase package which
  you're maintaining. I was wondering if you'd mind if I made an NMU
  to fix some of them for the upcoming slink release?
  

  From: Peter Tobias
  Date: Sat, Nov 21, 1998
  No, please go ahead ... I'm quite busy right now and I would really
  appreciate any help. Please let me know if you need additional information
  about the package.
  

  From: Anthony Towns
  Date: Sun, Dec 6, 1998
  There's an NMU sitting in Incoming now. It fixes a few bugs, viz:
  [...]
  

  From: Peter Tobias
  Date: Fri, Dec 25, 1998
  due to my current job I don't have much time to work on my debian
  packages. In order to have more time for my other debian packages
  I would like to give away the netbase package. Are you interested
  in maintaining this package?
  

  From: Anthony Towns
  Date: Fri, Dec 25, 1998
  Ummm. Sure. I guess. (or, iow, *Eeeeeeeeeeeeek*!!!)
  
  

• ifupdown: As part of maintaining netbase I tried to figure out a way of fixing bugs like 31745 and 39118. Basically, Debian used to do networking by generating a script on install that you could modify by hand, and that was it – so when the commands needed changed between 2.0 and 2.2 (no more “route add -net”), you couldn’t make that upgrade “just work”. Red Hat handled it with “ifup” and “ifdown” commands that would look at a whole bunch of shell-format variables in /etc, which worked but wasn’t elegant, so I came up with something I thought was actually pleasant. It’s fundamental attitude is to be a parser – the networking commands are specifed as compile-time configuration, the description of your network is your runtime configuration, and ifupdown just puts those all together without trying to actually understand what’s going on. To some degree, that works really well – it keeps all the knowledge separate so when you’re writing an /etc/network/interfaces, you’re not worried about how DHCP works; in others it breaks down – in particular if bringing up an interface fails part way through, is it up, or down, or something else entirely?
• bugs.debian.org: I’m not entirely sure why, but in August ‘99 I decided to start running a script to stop old bugs from being permanently deleted:

  From: Anthony Towns
  Subject: BTS and old bugs
  Date: Tue, 24 Aug 1999 22:33:16 +1000
  To: debian-private@lists.debian.org
  ObPrivate: Erm. I'm not sure. It is only even vaguely relevant to
  developers.
  Since bugs #9705 and #36727 don't seem like being fixed any time soon
  and Darren hasn't managed to convert the BTS to using debbugs.deb yet,
  I've made a little script to stop us from continuing to lose bug
  reports, and am running it in my crontab on master.
  ~ajt/debian-bugs/archive/ contains hardlinked copies of the bugs in
  ~iwj/debian-bugs/spool/db (except split into sub-directories). When
  the bugs get expired from the BTS, the hardlink in ~ajt remains, so
  the file doesn't get lost forever.
  In the week or so I've been running it, some 500 odd bugs expired [0].
  

  Next month I sent Darren Benham a first version of bugreport.cgi, and at some point around then must’ve sent off a pkreport.cgi too; by the month after (October) I’d evidently been added to the debbugs group, because I was merging my archived bugs into the official debbugs directories.
• testing: So a year and a bit later and there had been some more discussions about making major changes to our release process, and since I’d done some more algorithms subjects at university by this point, dove in a bit more seriously. For me, the main challenge seemed to be keeping dependencies consistent, and it turns out validating dependencies and conflicts is an NP-complete problem, and solving that reasonably efficiently seemed like a good first step. By October ‘99 I’d come up with a first pass solution, which I think was still in Perl at the time. A couple more rounds of playing with consistency checking stuff ensued, resulting in some regularly updated lists being generated by December, and by March 2000 or so that had graduated to a simulation of testing as we know and love it today.
• Played for a sucker, part two: so after a few months of maintaining a testing suite while potato (Debian 2.2) was getting finalised for release I figured it’d be useful to get broader release experience, so asked Richard Braakman if I could help out with the last bits of the potato release. At which point he said “sure”, gave me James Troup’s email address for accepting/rejecting packages etc, then buzzed off to a mobile phone related junket in the US, and laid low until the release was actually out… A cunning plan, for sure. Happily, that didn’t take too long – I think I started around June, and potato was finished baking for release at LinuxWorld Expo in August, though it did involve about 2000 lines of archive changes and stuff mailed to James over the period, along with half a dozen mails to -devel-announce.
• Junket: In the middle of that was my first ever debconf – or technically the Debian portion of the 2000 LSM/RMLL in Bordeaux, aka Debconf 0. It was awesome – met heaps of cool people, got to practice my high school French, and enjoy some lovely red wine while gossipping about free software. Also heard RMS sing the free software song live. But the wine was great!
• katie/dak: The biggest problem with the “simulation of testing” mentioned above was it was held together with paperclips and sticky-tape; or less metaphorically, shell one-liners and hardlinks. Since packages would originally get uploaded to unstable, and then move into testing, and then get deleted from unstable, you’d end up at least doubling the bandwidth required to mirror Debian, because each package would get sent to mirrors once for unstable and once for testing. Which is fine for a simulation, but for deployment, well, we needed package pools, which in turned required a rewrite of dinstall. James did all the initial work, apart from some of the schema design and the scarier SQL, and by November had it deployed for the non-US archive, and got it onto ftp-master in December. At which point the time was ripe for hooking up testing into the archive proper, which is about the point that the testing scripts got named “britney” so as to fit in with all silly names in the da-katie suite. I’m pretty sure the rationale was that I did the final coordination of the potato release (archive changes, cd images, announcement, etc), I spent from about 3am to 10am listening to a handful of Britney Spears songs on repeat. The phrase “scarred for life” might come to mind. Since that point, there’s been lots of evolutionary changes to both dak and testing (including a rewrite of the perl parts of testing into python), but it’s all built fairly naturally from that point.
• debootstrap: I’m pretty sure my motivation for writing debootstrap was a mix of just wanting to avoid having to worry about out of date base tarballs for the purposes of keeping testing in-sync, and wondering if it was actually possible to write a script to bootstrap a Debian system from the sort of minimally functioanl environment the installer has to put up with. In the end, it turned out pretty cool – it was used by the final boot-floppies, it’s part of d-i, it makes tools like pbuilder possible, I think it helps embedded folks heaps, it’s spawned some imitators, and generally been a pretty cool exercise in hackery.
• Freedom and purity: About six months earlier, there was a big flamewar and an attempted vote on whether to remove non-free from the archive. I’d written a rebuttal and counter-proposal, but it all ended up collapsing in a heap, due to disputes about the constitution. For some reason I feel like this mail was more interesting than all the previous ones combined… That ended up needing a whole bunch of changes to the constitution: one to make it clear under what circumstances the social contract can be changed, and another on what happens when one option on a vote requires a supermajority while another doesn’t, and working that out ended up taking from October 2000 until October 2003. It was actually kind-of interesting; trying to analyse a good voting system that works in the real world has all sorts of interesting maths to it, and there’s a whole bunch of election methods guys that have studied it in a bunch of depth. And since its elections, you can have fun with all sorts of scenarious of corruption – like what if the secretary’s trying to change the results. In practice, the difference between good and bad election methods seems to turn out to be negligible, but why wouldn’t you have the best one you possibly can? A couple of months after those changes were done, dropping non-free got reproposed, and I did another counter-proposal; this time they were actually voted on, with the counter-proposal winning, by a little under a two-to-one margin. Which means we get to get rid of non-free by making it completely superfluous, rather than just near enough.

Hrm, this is going on longer than I’d hoped. Oh well, to be continued!

The story seems to have started with Damog supporting sexism and xenophobia in Debian, followed by Bernhard’s comment about that not being tolerable under Debian’s flag. Aigarius replied kinda shocked that stopping verbal abuses and violence would be censorship. AJ clearly explained the reasons why that shouldn’t be tolerated in Debian, but Aigarius seems to keep thinking that poor jerks are being discriminated when they’re not allowed to insult us. Wouter seems to be tired of the whole topic. Am I missing any chapter? Now my turn: We are on the right way. Since the Debian-Women project started, we have being able to achieve a friendly and non-sexist environment inside Debian, which is something that I’d really like to keep. It doesn’t really mind that a couple of people try to go back to a hostile environment. Trolls just do exist, I guess, but they are not representative of the general feeling of the people inside Debian. Damog, I won’t get the fuck out of here, like it or not. It seems to me that it’s just a couple of you who are happy to be in such an aggressive environment as the one you’re promoting. For what I know, most of Debian people are happy to live in peace. You don’t have social life? You don’t like it? That’s your problem, dude, but don’t ask us to leave ours. I respect you, you know I have always done. I’m just very disappointed with your comment. Really. Making fun of other people’s problems and traumas is simply bullying. I hope you think about it. For the rest of Debian, keep up the good job. Right now my feeling is that the level of sexism, racism, ageism, homophobia, hostility and unfriendliness in general inside Debian development’s world is below the average in the Free Software world. It’s nice to be there, and I’m really proud of belonging here, which is something I cannot say about every group in Free Software. I’d like to keep it that way. Sorry for the aggressiveness in this weblog entry.

Oh yay, another argument about sexism. I thought we were over this. Aigars writes:

Trying to restrict what words people can or can not use (by labeling them sexist, racist or obscene) is the bread and butter of modern day media censorship. It is censorship and not “just political correctness”. While I would not want people trying to limit contributions to Debian only to “smart and educated white people” (racism) or “logically thinking males” (sexism), going the other way and excluding people from Debian because their remarks or way of thinking might offend someone is just offensive to me.

One: it’s not censorship for Debian to limit discussion of various things on Debian channels. It’s censorship when you prevent discussion of something anywhere. Two: if you think excluding people is bad, then supporting jerks whose mysogyny repulses people isn’t compatible with that. Three: doing something in someone else’s name, that isn’t supported by them, is wrong. If you’re in a channel called “debian-something” don’t act in ways that don’t match Debian’s goals. If you want to be free to go against the principles of the DFSG by, eg, discriminating against people or groups, make up your own name for a channel.