Now that my Debian packages are in a better shape, I’m focusing on closing as much upstream bugs as I can in the BackupManager source tree. The more I close, the more I think it’s time to freeze the code for targetting a new stable release. I’d like very much to manage to release Backup Manager 0.8 before Debian enter its frozen-prerelase phase; providing a mature version of BM within “etch” would rock.

Yesterday, I held the first meeting of the Computer Club that I founded here at the college in Tema, Ghana. I didn't go into very technical things right now, to not scare anyone away, but gave an introduction to wikis and blogs, and showed the students how to use the wiki and blog I set up for the club. You can have a look at my LaTeX-created beamer slides.I think the club went quite well. About 20 students seem to be interested, and a few more might come with the 11-graders who start school next week. There seems to be little geeky knowledge present, so I'm not sure if I can go into the more technical things of geekdom, like weird programming language, fixing bugs in Free Software etc. But we'll see, maybe I underestimate them, and one year is a long time for an interested geek to get into new stuff.Besides one box that I can fiddle around with, and which hosts the club's wiki and blog, there is no Linux yet at the school. I did help a bit with setting up a few windows boxes, and already I'm happy to know that at least I have an alternative. Walking from computer to computer to manually deinstall software preinstalled by the vendor, manually set hostnames, manually write down MAC adresses for the WLAN MAC-filter, manually joining the domain and manually install all other software does not seem like the way I would want to administrate more than two computers. In Germany, we call that "Turnschuh-Administration" (sneaker or running shoe administration).

Unfortunately, that is not an argument to bring to make the school use more Linux, as a dual boot setup is the most I can hope fore at the moment, and that does not ease the pain of Windows administration. I'll keep you up to date on that.

A new multifunction device (a Samsung SCX-4200) was recently bought at home. It “supports” Linux via a proprietary, binary-only driver that does not let you scan except as root. A normal user receives a Segmentation Fault. Of course, the installer “fixes” it by adding a setuid wrapper to xsane… As I didn’t like that “solution” very much, I investigated a bit and learned that the Segfault happened after “scanimage -L” called the iopl() syscall. That call is used to be able to access I/O ports directly, and can only be called by a process running as root. That driver’s name was “Unified Driver blah, blah, blah". That means that it “supports” a lot of multifunction devices, among them several that run plugged into the parallel port, and the driver was trying to detect a parallel port device by accessing the parallel port directly, and that’s why it called iopl(). After a couple of hours (I don’t do it every day, mind you) trying objdump and gdb and a hex editor (biew, which helpfully includes a disassembler too to guide you in your hex editing adventures) I managed to disable the parallel port check and make the driver a bit less insane (now I only need to make users who need to use the scanner members of the “lp” group). I guess that it means that you don’t really need source code to modify your drivers and firmware (picture yourself a tongue well stuck to my cheek). To be helpful I made the modified version, with instructions and HOWTO, available at http://jacobo.tarrio.org/Samsung_SCX-4200_on_Debian. Copyrights be damned, I shouldn’t have had to do it.

upstart is a replacement for the init daemon, the process spawned by the kernel that is responsible for starting, supervising and stopping all other processes on the system. The existing daemon is based on the one found in UNIX System V, and is thus known as sysvinit. It separates jobs into different “run levels” and can either run a job when particular run levels are entered (e.g. /etc/init.d/rc 2) or continually during a particular run level (e.g. /sbin/getty). The /etc/init.d/rc script is also based on the System V one (and is in the sysv-rc package), it simply executes the stop then start scripts found in /etc/rcN.d (where N is the run level) in numerical order. Why change it? Running a fixed set of scripts, one after the other, in a particular order has served us reasonably well until now. However as Linux has got better and better at dealing with modern computing (arguably Linux’s removable device support is better than Windows’ now) this approach has begun to have problems. The old approach works as long as you can guarantee when in the boot sequence things are available, so you can place your init script after that point and know that it will work. Typical ordering requirements are:

• Hard drive devices must have been discovered, initialised and partitions detected before we try and mount from /etc/fstab.

• Network devices must have been discovered and initialised before we try and start networking.

This worked ten years ago, why doesn’t it work now? The simple answer is that our computer has become far more flexible:

• Drives can be plugged in and removed at any point, e.g. USB drives.

• Storage buses allow more than a fixed number of drives, so they must be scanned for; this operation frequently does not block.

• To reduce power consumption, the drive may not actually be spun up until the bus scan so will not appear for an even longer time.

• Network devices can be plugged in and removed at any point.

• Firmware may need to be loaded after the device has been detected, but before it is usable by the system.

• Mounting a partition in /etc/fstab may require tools in /usr which is a network filesystem that cannot be mounted until after networking has been brought up.

We’ve been able to hack the existing system to make much of this possible, however the result is chock-full of race conditions and bugs. It was time to design a new system that can cope with all of these things without any problems. What we needed was an init system that could dynamically order the start up sequence based on the configuration and hardware found as it went along. Design of upstart upstart is an event-based init daemon; events generated by the system cause jobs to be started and running jobs to be stopped. Events can include things such as:

• the system has started,

• the root filesystem is now writable,

• a block device has been added to the system,

• a filesystem has been mounted,

• at a certain time or repeated time period,

• another job has begun running or has finished,

• a file on the disk has been modified,

• there are files in a queue directory,

• a network device has been detected,

• the default route has been added or removed.

In fact, any process on the system may send events to the init daemon over its control socket (subject to security restrictions, of course) so there is no limit. Each job has a life-cycle which is shown in the graph below: The two states shown in red (“waiting” and “running”) are rest states, normally we expect the job to remain in these states until an event comes in, at which point we need to take actual to get the job into the next state. The other states are temporary states; these allow a job to run shell script to prepare for the job itself to be run (“starting”) and clean up afterwards (“stopping”). For services that should be respawned if they terminate before an event that stops them is received, they may run shell script before the process is started again (“respawning”). Jobs leave a state because the process associated with them terminates (or gets killed) and move to the next appropriate state, following the green arrow if the job is to be started or the red arrow if it is to be stopped. When a script returns a non-zero exit status, or is killed, the job will always be stoped. When the main process terminates and the job should not be respawned, the job will also always be stopped. As already covered, events generated by the init daemon or received from other processes cause jobs to be started or stopped; also manual requests to start or stop a job may be received. The communication between the init daemon and other processes is bi-directional, so the status of jobs may be queries and even changes of state to all jobs be received. How does it differ from launchd? launchd is the replacement init system used in MacOS X developed as an “Open Source” project by Apple. For much of its life so far, the licence has actually been entirely non-free and thus it has only become recently interesting with the licence change. Much of the goal of both systems appears initially to be the same; they both start jobs based on system events, however the launchd system severly limits the events to only the following:

• system startup,

• file modified or placed in queue directory,

• particular time (cron replacement),

• connection on a particular port (inetd replacement).

Therefore it does not actually allow us to directly solve the problems we currently have; we couldn’t mount filesystems once the “filesystem checked” event has been recived, we couldn’t check filesystems when the block device is added and we certainly couldn’t start daemons once the complete filesystem (as described by /etc/fstab) is available and writable. The launchd model expects the job to “sit and wait” if it is unable to start, rather than provide a mechanism for the job to only be started when it doesn’t need to wait. Jobs that need /usr to be mounted would need to spin in a loop waiting for /usr to be available before continuing (or use a file in a tmpfs to indicate it’s available, and use that modification as the event). This is not especially surprising given that Apple have a high degree of control over both their hardware and the actual underlying operating system; they don’t need to deal with the wide array of different configurations that we have in the Linux world. Had the licence been sufficiently free at the point we began development of our own system, we would probably have extended launchd rather than implement our own. At the point Apple changed the licence, our own system was already more suitable for our purposes. How does it differ from initng? Initng by Jimmy Wennlund is another replacement init daemon intended to replace the sysvinit system used by Linux. It is a dependency-based system, where upstart is an event-based system. The notion of a dependency-based system is interesting to talk about at this point. Jobs declare dependencies on other jobs that need to happen before the job itself can be started. Starting the job causes its dependencies to be started first, and their dependencies, and so on. When jobs are stopped, if running jobs have no dependencies, they themselves can be stopped. It’s a neat solution to the problem of ordering a fixed boot sequence and the problem of keeping the number of running processes to a minimum needed. However this means that you need to have goals in mind when you boot the system, you need to have decided that you want gdm to be started in order for it, and its dependencies, to be started. Initng uses run levels to ensure this happens, where a run level is a list of goal jobs that should be running in that run level. It’s also not clear how the dependencies interact with the different types of job, a dependency on Apache would need the daemon to be running where a dependency on “checkroot” would need the script to have finished running. Upstart handles this by using different events (“apache running” vs. “checkroot stopping”). Again while interesting, Initng does not solve the problems that we wanted to solve. It can reorder a fixed set of jobs, but cannot dynamically determine the set of jobs needed for that particular boot. A typical example would be that if the only dependency on the job that configures networking is the mount network filesystems job, then should that job fail or notbe a goal (e.g. because there are no network filesystems to be mounted) the result is that network devices themselves will not be configured. You could make everything a goal, and just use the dependencies to determine the order, however this is less efficient than just ordering the existing sysv-rc scripts (which
can be done at install time). Another example is that often you simply don’t know whether something is a dependency or not without reading other configuration, for example the mount network filesystems may be a dependency of everything under /usr or may just be a dependency of anything allowing the user to login if it just mounts /home. The difference in model can be summed up as “initng starts with a list of goals and works out how to get there, upstart starts with nothing and finds out where it gets to.” How does it differ from Solaris SMF? SMF is another approach to replacing init developed by Sun for the Solaris operating system. Like initng it’s a dependency-based system, so see above for the differences between those systems and upstart. SMF’s main focus is serive management; making sure that once services are running, they stay running, and allowing the system administrator to query and modify the states of jobs on the system. Upstart provides the same set of functionality in this regard, services are respawned when they fail and system administrators can at any time query the state of running services and adjust the state to their liking. Will it replace cron, inetd, etc? The goal of upstart is to replace those daemons, so that there is only one place (/etc/event.d) where system administrators need to configure when and how jobs should be run. In fact, the goal is that upstart should also replace the “run event scripts” functionality of any daemon on the system. Daemons such as acpid, apmd and Network Manager would send events to init instead of running scripts themselves with their own perculiar configuration and semantics. A system administrator who only wanted a particular daemon to be run while the computer was on AC power would simply need to edit /etc/event.d/daemon and change “on startup” to “on ac power”. What about compatibility? There’s a lot of systems administrators out there who have learned how Linux works already and will not want to learn again immediately, there’s also a large number of books that cover the existing software and won’t cover upstart for at least a couple of years. For this reason, compatibility is very important. upstart will continue to run the existing scripts for the forseeable future so that packages will not need to be updated until the author wants. Compatibility command-line tools that behave like their existing equivalents will also be implemented, a system administrator would never need to know that crontab -e is actually changing upstart jobs. Does it use D-BUS?

“To D-BUS people, every problem seems like a D-BUS problem.”
—Erik Troan

The UNIX philosophy is that something should do just one job, and do it very well. upstart’s one job is starting, supervising and stopping other jobs; D-BUS’s one job is passing messages between other jobs. D-BUS does provide a mechanism for services to be activated when the first message is sent to them, thereby starting other jobs. Some people have taken this idea and extended it to suggest that all a replacement init system need do is register jobs with D-BUS and turn booting into a simple matter of message parsing. This seems wrong to me, D-BUS would need to be extended to supervise these services, provide means for them to be restarted and stopped; as well as deal with being process #1 which means cleaning up after children whose parent’s have died, etc. It seems far simpler to arrange for D-BUS to send an event to init when it needs a service to be started, and focus on being a very good message passing system. The IPC mechanism used by upstart is not currently D-BUS because of various problems, however it’s always been expected that even if init itself doesn’t communicate with D-BUS directly, there would be a D-BUS proxy that would ensure messages about all init jobs and events would be given to D-BUS and D-BUS clients could send messages to init to query and change the state of jobs. What is the implementation plan? Because this is process #1 we are changing, we want to make sure that we get it right. Therefore instead of releasing a fully-featured daemon and configuration to the world, we’re developing it in the following stages:

1. Principal development; at the end of this stage the daemon has been implemented and can manage jobs as described.
2. Replacement of /sbin/init while running the existing sysv-rc scripts. This is the shake-down test of the daemon, can it perform the same job as the existing sysvinit daemon without any regressions?
3. /etc/rcS.d scripts replaced by upstart jobs. These consitute the majority of tasks for booting the system into at least single-user mode, and contain many of the current ordering problems and race conditions. If the daemon solves the problems here, it will be a success.
4. Other daemon’s scripts replaced by upstart jobs on a package-by-package basis; this will be an ongoing effort during which upstart will continue running the existing sysv-rc scripts as well as its own jobs. During this time the event system may be tweaked to ensure it truly solves the problems we need.
5. Replcement of cron, atd, anacron and inetd. This will happen alongside the above and result in a single place to configure system jobs.
6. Modification of other daemons and processes to send events to init instead of trying to run things themselves.

The current plan is that we will be at least part of the way into stage #3 by the time edgy is released, with that release shipping with upstart as the init daemon and the most critical rcS scripts being run by it to correct the major problems For edgy+1 we hope to have completed stage #5 and be at least part of the way into the implementation of stage #6. From the start of development of edgy+2, no new packages will be accepted unless they provide upstart jobs instead of init scripts and init scripts will be considered deprecated. What state is it in now? The init daemon has been written and is able to manage jobs as described above, receiving events on the control socket to start and stop them. This has now been uploaded to the Ubuntu universe component in the upstart package for testing before it becomes the init daemon. We welcome any experienced users who want to help test this; install the package and follow the instructions in /usr/share/doc/upstart/README.Debian to add a boot option that will use upstart instead of init. If your system boots and shut downs normally (other than a slightly more verbose boot without usplash running) then it is working correctly. Other types of events will be added as required during development and testing. Currently only a basic client tool (initctl) has been written, compatibility tools such as shutdown will be written over the next week or two before it replaces our sysvinit package.

As I mentioned in my first post the amount of effort I'm prepared to invest in posting to a small group of people is limited. I don't think that I am the only person with this opinion.<br /><br />I also believe that the number of people who refuse to post to open lists is quite small, and that on many lists they aren't the people who contribute much. I believe that they are outweighed in both number and contributions by the people who want open lists and who are unwilling to spend a large effort on posting to a closed list.<br /><br />When posting to an open list you have to be concerned about your online reputation. Some lists are closed because of having NSFW content that people don't want known by their colleagues and managers, I guess that this makes sense for some lists.<br /><br />IMHO the only good reason for closed lists is for discussion of truly sensitive information. This ranges from security problems in software that have not yet been fixed to medical and psychiatric problems. There are many lists which should not be publicly archived, but for general discussion of computers there is no such motivation.<br /><br />For a list with a primarily technical focus on answering basic questions secrecy does no good, it merely protects people who want to post off-topic messages and create pointless arguments about issues that they don't understand.<br /><br />My solution to some of these problems is to use this blog to comment on such things. I expect that my solution will also be adopted by other people on some of the closed lists that I use.<br /><br />Also it has occurred to me that blogging about issues may improve the quality of list discussion. If instead of responding to a message in point-form you write an article about the general issue then it may reduce the level of personal dispute. I think it would be difficult to have a flame-war by blog.<br /><br />Finally while on the topic I have to mention that I don't believe in anonymous posting to technical forums. Any content that is worth having should come with someone's name attached. IRC nicks etc are OK, but the person writing the content should be identifiable.

We're still in wt:Copenhagen today, but we head out this afternoon for our tour of the Baltic Sea. We've had a great time in the city so far, mostly due to staying with longtime Wikitraveller wt:User:Elgaard, who generously offered us the extra room in his apartment for the time we're here. There's something about staying in an actual human being's home while you're travelling to really recharge the batteries. After a week in hotels and restaurants, a little time in a real house can make you feel a lot more human. We took our time getting to Copenhagen from wt:Odense at the end of the Wikisym conference, mostly because I was really hung over. I'd spent the last night of the event drinking at the Irish pub in the pedestrian area, and I probably had 1-5 pints too many. Paul Yount calls Irish pubs "the first Open Source franchise", and it's true: you can expect certain things at a pub no matter where it is in the world. Although we started out with about 30 people, the crowd thinned through the night. I had a good time talking with Brion Vibber and Chuck Smith about the state of Esperanto-language wikis on the Web -- in a mish-mash of English and Esperanto, of course. (In case you were wondering: the state is "fine".) I also chatted Wikimedia Foundation politics with Angela Beesley; we both think Erik Moeller is going to make a good addition to the WMF board. After the pub shut down, I headed to the deathly silent casino in the sub-basement of the Radisson of Odense. The bar was eerily silent, and the gaming tables were intense and serious; I don't think people could have been more somber if they were organ harvesters gambling for body parts. The last hangers-on were Andrea Forte of Georgia Tech, Alex Schroeder of CommunityWiki, Sunir Shah of MeatballWiki, and Eugene Eric Kim, of Blue Oxen and HyperCore. In other words, some of the smartest people in the wiki world today -- we talked wiki politics and theory until the wee hours of the morning. They all switched to mineral water while I was still slugging down pints of Carlsberg Special, which is probably where I went wrong. Wikisym, for me, was one of the best conferences I've been to this year, and I've been to a lot. It capped off a long summer of conference-going, and this fall should be relatively tranquil. Which is great: I have a lot of work to do, based on the ideas I've picked up this season. tags: wikisym2006 odense copenhagen wiki

sterbro All of which is to say that I was feeling pretty crufty the next morning as we got on the train. Kindly Maj got us a late checkout and let me sleep in late, which helped a lot; a falafel sandwich around 11AM also cleared my head a bit. The train ride from Odense to Copenhagen was great, and by the time we got to Niels Elgaard Larsen's house in sterbro (a neighborhood of Copenhagen) I was pretty refreshed. Niels took us out to an early dinner at a restaurant called Pixie in a square by his house. Copenhagen has some pretty strict traffic laws, and the number of cars on the street is possibly the least I've ever seen for a major metropolis, leaving a lot of room for sidewalks, bike paths, and open parks and plazas. We got to chat with Niels over a fresh pasta dinner, and thence home to his house for beers and more talk. One great thing about visiting Wikitravellers is that they tend to be a) very smart, b) well-travelled, and c) interested in Free Software and Free Culture. Niels is no exception; he's active in the Danish and European Free Software and information freedom communities, and he's extremely interesting to talk to on the subject. We shared travel stories and talked about Open Source and Open Content later than I would have thought, based on my earlier-morning hangover. Probably the high point of the evening, for me, was when Niels shared a brilliant idea for a way to distribute Wikitravel guides in digital form for a very wide array of users. We've had a hard time settling on a mobile document format for Wikitravel guides, since mobile devices like PDAs and cell phones support standards so poorly. It's one of the most requested features for Wikitravel, but it's going to take a huge amount of work to cover even a fraction of the mobile device owner community. Except Niels had a genius idea for how to distribute guides in a simple, standard format that works on a mobile digital device many if not most travellers carry with them every time they travel. The idea: make guides that are downloadable JPEG images which can be copied to a SD card or Compact Flash or Memory stick or whatever, then stuck into a digital camera. If the camera supports zooming on images, and has a decent screen for review, the guides might be useful and readable. I'm going to try to do some experiments with the idea -- I think it could be a real winner. Adding 30-40 images to a camera today is pretty cheap, and if they're just text I think some aggressive compression can work well. tag: jpeg sterbro niels copenhagen wikitravel mobile

Copenhagen on foot Yesterday we did a big tour of the city on our own. In the morning, we had some breakfast at the Laundromat Cafe right next door to Niels's house, and I took the opportunity to clean a week's worth of dirty laundry. Nothing like a stack of clean clothes to make you feel more like a decent citizen. I was down to a bathing suit and the ICANNWiki t-shirt Ray King gave me at Wikisym. Then we headed out on the town to do some sight-seeing. First we hit the Kastellet, a fortress in northeast Copenhagen that's the oldest operational military base in Europe. It's got the distinctive star-shaped outline typical of old fortifications, and is surrounded by a lovely park with ducks and swans. We had lunch at a nearby caf , with large beers. (Gee, I'm getting worried about the pattern here.) After lunch we made an unscheduled stop at den lille havfrue -- the famous Litle Mermaid statue in the harbour. It was miniscule and surrounded by perhaps 100 onlookers, crammed into a tiny swatch of sidewalk. We stopped about half a mile away, took some pictures of the crowd, and moved on. We then hoofed it down through the harbour and across Nyhavn and a bridge to Christianhavn and Christiania. We'd both heard about the city-within-a-city run on anarchist principles -- kind of like a giant squat -- but neither of us had been there before. It was an interesting place -- packed with people, lots of food cooking a beer-drinking, caf s and restaurants all over the place, interspersed with art installations and little gardens. But it had the unkempt scumminess of a big squat, too -- especially on the main drag -- and there were a few too many pitbulls and gutterpunks for my tastes. Overall a good experience but I don't know if I'll go back. After our long walk we took the metro back to Niels's, where we napped until he and wife Chiquita got home. Chiquita is an investment banker from wt:Toronto who does a lot of business in Europe, where she met Niels a few years ago. They were married last year and have had a long-distance relationship -- visiting each other a week or so out of each month -- for the whole time. She's moving here this year, which I think will be nice for both of them. We had a good dinner at a place called Thomas's -- a buffet, which seems to be very popular here -- and then back home for a nightcap and bed. Amita June was overtired by the time we got her to bed, and she let us know that we'd pushed her too hard with a quarter-hour of heavy crying. But we all got to sleep pretty well afterwards, and at least the parents know that we should take it easy in the future. tags: copenhagen kastellet christiania little mermaid

So earlier today (or yesterday, if you want to be technical) I final dragged myself down to Stratton to pick up my MIT student ID and tell the orientation dudes that I’m here, even though I’ve already been here for two months. I ended up being tackled by a rep from Bank of America as soon as I stepped into the building, who assumed I was Not A Freshman™ when I mentioned that I in fact am already the possessor of not one but two Bank of America bank accounts. The wearing of the bike helmet probably helped there, since I doubt many other freshmen are biking to orientation. The ID itself is, to put it shortly, kind of rediculous looking. What with the bubbly background and the shiny “MIT” holograms all over it. It’s definitely better than my currently-used-as-ID New York State driver’s permit though, which has “UNDER 21″ plastered all over it in rather large letters and a rediculously old photo. Which can get pretty annoying sometimes. Now, I just hope I can get the card activated for access to the Media Lab pretty quickly so I’ll stop locking myself out trying to go to the bathroom by the elevator after 6 PM or so. I don’t know about you, but I found that pretty annoying. And I never did manage to figure out exactly what time they started locking the doors, so I did it at least three or so times. Even though my last day of work was technically Friday, I’ll probably go in for a couple hours on a day or two next week. I plan on being a real UROP there during term anyway. Other than the ID, the other major things I picked up were “The Hitchhiker’s Guide to Orientation” which actually does mention some things that may end up being fun during orientation, a t-shirt that just screams frosh, and a book called How to Get Around MIT which actually isn’t really a whole lot about getting around MIT, and looks like it may even have some useful info in it. I blame it on the fact that the book’s written by students. Orientation starts Sunday, but I’m skipping that day to go on a 50-ish mile bike ride on a rather sexy borrowed touring bike. Word.

It's strange that the FSF/UNESCO Free Software Directory project has managed to remain hidden from my eye until today...

The Free Software Directory is a project of the Free Software Foundation (FSF) and United Nations Education, Scientific and Cultural Organization (UNESCO). We catalog useful free software that runs under free operating systems particularly the GNU operating system and its GNU/Linux variants.

The main difference to similar software directories (Freshmeat, SourceForge, ...) is that "licenses are verified for each and every program listed in this directory", which is a good thing. If you've got too much time on your hands, here's an idea how to get rid of it and at the same time help the Free Software community...

Ryanair finally sued the UK government for 3 million pounds for the air traffic disruptions, just like I called it!
Now they only need to team up with British Airways (who demand their money from goverment’s BAA) and all the other affected airlines, so that the court doesn’t just dismiss it right away. Also it would be nice if Ryanair rallied the people to halp them - demonstration in front of the courthouse and bumper stickers saying “Keep Britain Flying!” and “Keep US Flying!” (along with t-shirts, umbrellas and sports style water bottles) would also help to rally public support.

Amused to see a mail from Radu asking me if I were the Steve Kemp mentioned in relation to osCommerce - I’m not. I know that names aren’t unique because I’ve noticed collisions in the past, but this is the first time that I’ve seen it in relation to myself and software. I remember a few months back an utterly surreal situtation where I checked into The Tower Hotel in London, and after being in my room for a short while a guy tried to open the door. I opened it said “Hi, this is my room”. (My response might have been a little less polite. I was tired..) He replied my “My name is Steve Kemp, and this is my room.” Turns out that the hotel had the two of us there that night and managed to believe that the two of us were the same person - so we’d both been given keys to the same room.

Introductory notes about tagging A group of nice Swiss researchers are doing some research on Debian, and as part of their research they are about to tag around 400 libraries using Debtags. They feel that being able to distinguish projects by categories will help us to improve their analysis of component (library) reuse. They asked me if I had useful Debtags information for them besides the website and the Debconf5 paper, so I did some research and I share it here. The list archives have various useful posts:

• Thirteen tips from effective tagging, a commented link to an article about tagging websites on Del.icio.us tags.
• Many discussions about vocabulary structure.
• Tips collected while approving LOTS of tags

Also, the vocabulary itself has short and long comments for each tag, and the long comments sometimes have useful instructions. In order to contribute properly reviewed tags, I suggested to start by posting tag patches on the mailing list: after we discussed some of them and we trust each others, I'll be happy to give commit access to the svn repository. To create a tag patch, one can use tagcoll:

svn cat svn://svn.debian.org/debtags/tagdb/tags > tags
cp tags tags.edited
[...edit tags.edited...]
tagcoll diff tags tags.edited

Note that there are probably many new tags in the not-yet-revied database, so one may want to proceed this way:

1. svn cat svn://svn.debian.org/debtags/tagdb/tags > tags
2. remove from tags all the lines corresponding to the packages you're not interested in
3. wget http://debtags.alioth.debian.org/tags/tags-current.gz
4. remove from tags-current all the lines corresponding to the packages you're not interested in
5. tagcoll diff tags tags-current > changes
6. edit changes removing those changes that make no sense you can review the edits you made to the changes patch this way: svn cat svn://svn.debian.org/debtags/tagdb/tags tagcoll --patch-from=changes.orig > tmp1 svn cat svn://svn.debian.org/debtags/tagdb/tags tagcoll --patch-from=changes.edited > tmp2 tagcoll diff tmp1 tmp2
7. apply the reviewed patch to the svn repository: svn cat svn://svn.debian.org/debtags/tagdb/tags tagcoll --patch-from=changes copy > tags-patched
8. work from there

Everyone is of course free to use the list for any questions.

(and what we want to do to avoid similar problems in the future) A few days ago, a broken xserver-xorg-core was uploaded to dapper-updates. This caused, unsurprisingly, a large amount of bug reports. A fixed package was uploaded about 12 hours after the first bug report came in. We also pointed all the $countrycode.archive.ubuntu.com (se.archive.ubuntu.com, au.archive.ubuntu.com, etc) to archive.ubuntu.com to speed up the propagation of the fix. Even so, it hit far, far too many users. Incidentially, the distro team is currently in a sprint in Wiesbaden, Germany and we had a large discussion this morning about both how to handle such situations when they happen (and recover from them, mostly in the technical sense) and how to prevent them from happening in the first place. The latter is obviously more important as we won't have to recover if there is not a problem in the first place. Note that those ideas listed below were ideas and not finished procedures and we are going to write up a proper policy document. Ideas for prevention ranged form using $distro-proposed and explicit call for testers of that to getting more code review for updates. The current review process for updates to $distro-updates is a review by the release manager who accepts it into $distro-updates. This update passed that review even though the update was faulty, so even if we had another reviewer, it might have passed that review too. Using $distro-proposed does not prevent the problem from affecting anybody, it just changes the group affected with the goal that the group choosing to enable $distro-proposed will hopefully be able to recover more easily. Recovery ideas ranged from being able to put an update onto a user's machine whether he wanted it or not to snapshot/rollback support and having some kind of a "safe mode" where it would give you a kdrive-based VESA X server and tools to fix your system. I'm not sure what we would do if we managed to break one of those tools (or the "safe-mode" X server), though. We all agreed that being open about the problem, the cause of the problem and how we are working to solve it is important. Downplaying the severity or making jokes or sarcastic comments about the fix ("Ooops, we did it again") is bad and something we shouldn't do. (And I don't think we did it this time either.) No response is equally bad and something we should work hard to avoid. Hopefully, we will have a procedure in place in a little while which tells us how to handle such an emergency when it happens and we will be deploying safeguards to prevent it from happening again while not ending up paralysing us and making us unable to deploy updates to $distro-updates as this is something we need to be able to. While this post is fairly critical of the current set of policies and procedures, I have tried hard to avoid pointing fingers at anybody in particular. I would much rather have us have a positive and constructive discussion about how to avoid similar problems in the future than a discussion on who is to blame. The points and views in this post is also those of my own and not any kind of official communication from Canonical or Ubuntu.

On the whole I agree with Erich. Subprojects are key to preserving positive energy as a free software project scales. But I wouldn’t say the ill effects suffered by projects as they age is just “a matter of size”. And I wouldn’t de-emphasize the importance of guidelines like the CoC in countering those effects, though I’d agree it does not provide immunity by itself. Ubuntu benefits from the Debian social experiment having started first. So the people involved have the opportunity to learn from and improve on earlier efforts while their project is still young. The CoC is no panacea, but it is a clue that lessons are being learned; deliberate effort needs to be made to keep work fun, especially as a project grows and people begin to lose touch with each other. Putting these lessons into practice isn’t always easy, but I think it would help if as individual developers we’d reflect more on how we interact within our projects and work on making improvements. I appreciate the personal guidelines Erich shared and will adapt them to make them my own.

Short summary: Server down.

Isn't Murphy absolutely lovely? Hardware will, of course, break on weekends.

The light in my kitchen's ceiling stopped working a month ago. I changed the light bulb, a week ago, but it didn't help, so on Friday I asked H ctor to take a look at it. Absolutely low priority. The cable was literally roasted, so he asked me to cut the power off, in order to replace the cable. I did, without remembering to propperly power off the server. What are journalling filesystems for anyway? The UPS battery used to perform much better, oh the decay, so the server is abruptly shut down after a couple of minutes. As H ctor was not able to "quickly" fix the kitchen light, and I was not in a hurry, and do not really care about the damn light, we just didn't look at it any further. So far so good.

Some hours later, at midnight (maybe that's technically Saturday, but it really felt like a Friday, I swear to the allmighty vegetarian spaghetty monster with no meat balls whatsoever), I try to power the machine up. And a sadly familiar ritual is about to be performed. Since I moved the server into "the run down attic I call home" I dreaded the moment I had to reboot the machine. It would come up, you could see the blue case fans light up and spin, the hard drives and CD drives be initialized, but no beep, no POST, no nada.

The ritual dance would consist of me submissively kneeling down to power on and off in a random sequence the back switch, the front one, the UPS, the reset buttom, curse, turn it completely off for random amounts of time, pull the cable in and out, curse some more... never really finding out when or what would make it fully come to life. But eventually it did, after 5, 10, 15 minutes, without any reproduceable pattern, it did. Not on Friday night. Or saturday. I spent most of the night (until 6 am) performing the ritual, googling like crazy, fustrated and eventually plain angry. All I could find was "the mobo is b0rked, exchange it". But I knew it was not. I knew the CPU was fine, the MoBO was fine, the memory was fine, because when I finally managed to get it to power up, all worked well!

But Miry came home and helped me out, coming up with a valuable clue: The mobo was not getting enough power form the PSU, so I got a spare PSU from H ctor and swapped them. The baby came up instantly. So I was very pissed off.

The good thing about it all is:
- I learned that power supply units sometimes fail. Even my wonderful, 100 EUR BQuiet! PSU.
- I missed completely all of the udev fun. Well, at least on my machines, but there's a couple of people I needed to help out today.

On Monday I got a new power supply, it is a joke of a plastic, for SM's sake! plastic!?, PSU with a couple blue fans. Pics of old nice one and crappy new one here. My sitting room looks like a space craft, and sounds like a plane is about to take off. The cats have not complained yet. They have not disconnected my land line (yet).
Update: The reason I bought this PSU is I have quite a big case, a case I can ride! So I needed Log cables. And I had to go for a cheapo modder plstic thingy, simply because I could not afford anything better.

I hate hardware.

Okay, so the Xen host I've been setting up is nearing the "full production" phase (so far, only emdebian has been running production, the other hosts are being copied as I type). What is still unsolved is the problem of kernel upgrades, as I'm using the stock Debian Xen kernel for now, I need an initrd in order to start a system. While the normal mkinitramfs generated one usually manages to bring the system up, I think it is rather pointless to start the domU systems with anything more than what yaird would build; however that only works from inside, with the kernel and modules installed, which I'd rather not do, and requires me to copy the image outside. So I wonder whether it would make sense to build a set of standard initrds, probably in separate packages, that contain the necessary drivers to start a domU (variants: root on virtual block device, root on LVM, others?)
I guess tonight is going to be "sending patches" night.

LWN has an article up about how RPM is forked, with no single agreed-upon upstream and lots of distros going off and patching it on their own:

Fedora currently ships version 4.4.2, and even the Fedora development repository has not gone beyond that. SUSE remains at 4.4.2, and the current RHEL offerings have rather older versions.

(And Debian's at 4.4.1+a year of patches.) rpm is not the only peice of software that is a major component of many linux distributions but has no agreed-upon upstream maintainer, and thus effectively one fork per distro. Another such peice of software is cron. Vixie cron was last released in '93. In many distributions it's still used, but in eg, Debian, the package is the result of 13 years of patches on top of that release. The debian version number is 3.0pl1-96. That's ninty-six versions based on 3.0pl1. The Red Hat sub-version is in the fourtys. There's no standard version anymore, just this pile of patches, and other piles of patches in other distributions. If we're lucky, they at least share the same security fixes. Another example is makedev, which was had its last upstream release in '98 and is at patchlevel 82 in Debian. Or netcat, last released in '96, patchlevel 32 in Debian, and a fork/rewrite at netcat.sourceforge.net. Or xgalaga, whose upstream author has fallen off the net and last released it in '98. The Debian package, which I maintain, is at patchlevel 37, and seeme to be the only place a lot of people can find to send patches to. But I don't want to be its upstream maintainer. This is the kind of thing that makes me gibber in horror when people at Ubuntu talk about Debian and other distros being part of an "ecosystem of software". I don't want to be part of an ecosystem; I'd rather be part of something that works not something that muddles through via massive trial-and-error and redundancy. Biomass is another word for "shit". But I digress.. The most productive thing I've seen done to try to address this general problem is the Unmaintained Free Software wiki, which tries to track this software and find new maintainers. That has managed to match up some software with interested people (though it failed with xgalaga), but I think it actually works less well for big and important software that ends up forked and maintained separately by the major linux distributions. I, myself, am beginning to use the degree of divergence from upstream as an indicator of how well a distribution maintains a given peice of software, with more divergence == bad. A distro that's doing a good job will a) get their patches integrated upstream and b) should work with other distros to find a new developer if upstream goes missing or insane. This is the inverse of what people might naively consider a good metric of how much work a distro is doing, but I think it leads to better software in the long run.