Ajax isn't progress, it's actually a step back. Most people have been happy to actually use some sane advanced language for doing web sites. There is Java, JSP, PHP, Ruby, Python, Perl, ... And then there is Ajax. One of the main components is JavaScript. That language most of us would have loved to forget. Forever. Ajax brings you back into the dark ages of internet development, with browser incompatibilities, it features memory leaks, and many Ajax apps (such as live.com) will run unbearably slow on older computers. Such as my 1.8 GHz P4M laptop. If they work at all - live.com just shows "Loading..." for me right now. Have you ever looked at some of the actual Ajax code? This isn't the future, this is the dark ages coming back! [MJ Ray replies that I didn't mention accessibility issues with Ajax. That's true; I'm aware of them, but this post focuses on Ajax doing away with all the advances in programming languages and software engineering... The in-accessibility of Ajax stuff is worth an own blog post sometime]

Now that my Debian packages are in a better shape, I’m focusing on closing as much upstream bugs as I can in the BackupManager source tree. The more I close, the more I think it’s time to freeze the code for targetting a new stable release. I’d like very much to manage to release Backup Manager 0.8 before Debian enter its frozen-prerelase phase; providing a mature version of BM within “etch” would rock.

upstart is a replacement for the init daemon, the process spawned by the kernel that is responsible for starting, supervising and stopping all other processes on the system. The existing daemon is based on the one found in UNIX System V, and is thus known as sysvinit. It separates jobs into different “run levels” and can either run a job when particular run levels are entered (e.g. /etc/init.d/rc 2) or continually during a particular run level (e.g. /sbin/getty). The /etc/init.d/rc script is also based on the System V one (and is in the sysv-rc package), it simply executes the stop then start scripts found in /etc/rcN.d (where N is the run level) in numerical order. Why change it? Running a fixed set of scripts, one after the other, in a particular order has served us reasonably well until now. However as Linux has got better and better at dealing with modern computing (arguably Linux’s removable device support is better than Windows’ now) this approach has begun to have problems. The old approach works as long as you can guarantee when in the boot sequence things are available, so you can place your init script after that point and know that it will work. Typical ordering requirements are:

• Hard drive devices must have been discovered, initialised and partitions detected before we try and mount from /etc/fstab.

• Network devices must have been discovered and initialised before we try and start networking.

This worked ten years ago, why doesn’t it work now? The simple answer is that our computer has become far more flexible:

• Drives can be plugged in and removed at any point, e.g. USB drives.

• Storage buses allow more than a fixed number of drives, so they must be scanned for; this operation frequently does not block.

• To reduce power consumption, the drive may not actually be spun up until the bus scan so will not appear for an even longer time.

• Network devices can be plugged in and removed at any point.

• Firmware may need to be loaded after the device has been detected, but before it is usable by the system.

• Mounting a partition in /etc/fstab may require tools in /usr which is a network filesystem that cannot be mounted until after networking has been brought up.

We’ve been able to hack the existing system to make much of this possible, however the result is chock-full of race conditions and bugs. It was time to design a new system that can cope with all of these things without any problems. What we needed was an init system that could dynamically order the start up sequence based on the configuration and hardware found as it went along. Design of upstart upstart is an event-based init daemon; events generated by the system cause jobs to be started and running jobs to be stopped. Events can include things such as:

• the system has started,

• the root filesystem is now writable,

• a block device has been added to the system,

• a filesystem has been mounted,

• at a certain time or repeated time period,

• another job has begun running or has finished,

• a file on the disk has been modified,

• there are files in a queue directory,

• a network device has been detected,

• the default route has been added or removed.

In fact, any process on the system may send events to the init daemon over its control socket (subject to security restrictions, of course) so there is no limit. Each job has a life-cycle which is shown in the graph below: The two states shown in red (“waiting” and “running”) are rest states, normally we expect the job to remain in these states until an event comes in, at which point we need to take actual to get the job into the next state. The other states are temporary states; these allow a job to run shell script to prepare for the job itself to be run (“starting”) and clean up afterwards (“stopping”). For services that should be respawned if they terminate before an event that stops them is received, they may run shell script before the process is started again (“respawning”). Jobs leave a state because the process associated with them terminates (or gets killed) and move to the next appropriate state, following the green arrow if the job is to be started or the red arrow if it is to be stopped. When a script returns a non-zero exit status, or is killed, the job will always be stoped. When the main process terminates and the job should not be respawned, the job will also always be stopped. As already covered, events generated by the init daemon or received from other processes cause jobs to be started or stopped; also manual requests to start or stop a job may be received. The communication between the init daemon and other processes is bi-directional, so the status of jobs may be queries and even changes of state to all jobs be received. How does it differ from launchd? launchd is the replacement init system used in MacOS X developed as an “Open Source” project by Apple. For much of its life so far, the licence has actually been entirely non-free and thus it has only become recently interesting with the licence change. Much of the goal of both systems appears initially to be the same; they both start jobs based on system events, however the launchd system severly limits the events to only the following:

• system startup,

• file modified or placed in queue directory,

• particular time (cron replacement),

• connection on a particular port (inetd replacement).

Therefore it does not actually allow us to directly solve the problems we currently have; we couldn’t mount filesystems once the “filesystem checked” event has been recived, we couldn’t check filesystems when the block device is added and we certainly couldn’t start daemons once the complete filesystem (as described by /etc/fstab) is available and writable. The launchd model expects the job to “sit and wait” if it is unable to start, rather than provide a mechanism for the job to only be started when it doesn’t need to wait. Jobs that need /usr to be mounted would need to spin in a loop waiting for /usr to be available before continuing (or use a file in a tmpfs to indicate it’s available, and use that modification as the event). This is not especially surprising given that Apple have a high degree of control over both their hardware and the actual underlying operating system; they don’t need to deal with the wide array of different configurations that we have in the Linux world. Had the licence been sufficiently free at the point we began development of our own system, we would probably have extended launchd rather than implement our own. At the point Apple changed the licence, our own system was already more suitable for our purposes. How does it differ from initng? Initng by Jimmy Wennlund is another replacement init daemon intended to replace the sysvinit system used by Linux. It is a dependency-based system, where upstart is an event-based system. The notion of a dependency-based system is interesting to talk about at this point. Jobs declare dependencies on other jobs that need to happen before the job itself can be started. Starting the job causes its dependencies to be started first, and their dependencies, and so on. When jobs are stopped, if running jobs have no dependencies, they themselves can be stopped. It’s a neat solution to the problem of ordering a fixed boot sequence and the problem of keeping the number of running processes to a minimum needed. However this means that you need to have goals in mind when you boot the system, you need to have decided that you want gdm to be started in order for it, and its dependencies, to be started. Initng uses run levels to ensure this happens, where a run level is a list of goal jobs that should be running in that run level. It’s also not clear how the dependencies interact with the different types of job, a dependency on Apache would need the daemon to be running where a dependency on “checkroot” would need the script to have finished running. Upstart handles this by using different events (“apache running” vs. “checkroot stopping”). Again while interesting, Initng does not solve the problems that we wanted to solve. It can reorder a fixed set of jobs, but cannot dynamically determine the set of jobs needed for that particular boot. A typical example would be that if the only dependency on the job that configures networking is the mount network filesystems job, then should that job fail or notbe a goal (e.g. because there are no network filesystems to be mounted) the result is that network devices themselves will not be configured. You could make everything a goal, and just use the dependencies to determine the order, however this is less efficient than just ordering the existing sysv-rc scripts (which
can be done at install time). Another example is that often you simply don’t know whether something is a dependency or not without reading other configuration, for example the mount network filesystems may be a dependency of everything under /usr or may just be a dependency of anything allowing the user to login if it just mounts /home. The difference in model can be summed up as “initng starts with a list of goals and works out how to get there, upstart starts with nothing and finds out where it gets to.” How does it differ from Solaris SMF? SMF is another approach to replacing init developed by Sun for the Solaris operating system. Like initng it’s a dependency-based system, so see above for the differences between those systems and upstart. SMF’s main focus is serive management; making sure that once services are running, they stay running, and allowing the system administrator to query and modify the states of jobs on the system. Upstart provides the same set of functionality in this regard, services are respawned when they fail and system administrators can at any time query the state of running services and adjust the state to their liking. Will it replace cron, inetd, etc? The goal of upstart is to replace those daemons, so that there is only one place (/etc/event.d) where system administrators need to configure when and how jobs should be run. In fact, the goal is that upstart should also replace the “run event scripts” functionality of any daemon on the system. Daemons such as acpid, apmd and Network Manager would send events to init instead of running scripts themselves with their own perculiar configuration and semantics. A system administrator who only wanted a particular daemon to be run while the computer was on AC power would simply need to edit /etc/event.d/daemon and change “on startup” to “on ac power”. What about compatibility? There’s a lot of systems administrators out there who have learned how Linux works already and will not want to learn again immediately, there’s also a large number of books that cover the existing software and won’t cover upstart for at least a couple of years. For this reason, compatibility is very important. upstart will continue to run the existing scripts for the forseeable future so that packages will not need to be updated until the author wants. Compatibility command-line tools that behave like their existing equivalents will also be implemented, a system administrator would never need to know that crontab -e is actually changing upstart jobs. Does it use D-BUS?

“To D-BUS people, every problem seems like a D-BUS problem.”
—Erik Troan

The UNIX philosophy is that something should do just one job, and do it very well. upstart’s one job is starting, supervising and stopping other jobs; D-BUS’s one job is passing messages between other jobs. D-BUS does provide a mechanism for services to be activated when the first message is sent to them, thereby starting other jobs. Some people have taken this idea and extended it to suggest that all a replacement init system need do is register jobs with D-BUS and turn booting into a simple matter of message parsing. This seems wrong to me, D-BUS would need to be extended to supervise these services, provide means for them to be restarted and stopped; as well as deal with being process #1 which means cleaning up after children whose parent’s have died, etc. It seems far simpler to arrange for D-BUS to send an event to init when it needs a service to be started, and focus on being a very good message passing system. The IPC mechanism used by upstart is not currently D-BUS because of various problems, however it’s always been expected that even if init itself doesn’t communicate with D-BUS directly, there would be a D-BUS proxy that would ensure messages about all init jobs and events would be given to D-BUS and D-BUS clients could send messages to init to query and change the state of jobs. What is the implementation plan? Because this is process #1 we are changing, we want to make sure that we get it right. Therefore instead of releasing a fully-featured daemon and configuration to the world, we’re developing it in the following stages:

1. Principal development; at the end of this stage the daemon has been implemented and can manage jobs as described.
2. Replacement of /sbin/init while running the existing sysv-rc scripts. This is the shake-down test of the daemon, can it perform the same job as the existing sysvinit daemon without any regressions?
3. /etc/rcS.d scripts replaced by upstart jobs. These consitute the majority of tasks for booting the system into at least single-user mode, and contain many of the current ordering problems and race conditions. If the daemon solves the problems here, it will be a success.
4. Other daemon’s scripts replaced by upstart jobs on a package-by-package basis; this will be an ongoing effort during which upstart will continue running the existing sysv-rc scripts as well as its own jobs. During this time the event system may be tweaked to ensure it truly solves the problems we need.
5. Replcement of cron, atd, anacron and inetd. This will happen alongside the above and result in a single place to configure system jobs.
6. Modification of other daemons and processes to send events to init instead of trying to run things themselves.

The current plan is that we will be at least part of the way into stage #3 by the time edgy is released, with that release shipping with upstart as the init daemon and the most critical rcS scripts being run by it to correct the major problems For edgy+1 we hope to have completed stage #5 and be at least part of the way into the implementation of stage #6. From the start of development of edgy+2, no new packages will be accepted unless they provide upstart jobs instead of init scripts and init scripts will be considered deprecated. What state is it in now? The init daemon has been written and is able to manage jobs as described above, receiving events on the control socket to start and stop them. This has now been uploaded to the Ubuntu universe component in the upstart package for testing before it becomes the init daemon. We welcome any experienced users who want to help test this; install the package and follow the instructions in /usr/share/doc/upstart/README.Debian to add a boot option that will use upstart instead of init. If your system boots and shut downs normally (other than a slightly more verbose boot without usplash running) then it is working correctly. Other types of events will be added as required during development and testing. Currently only a basic client tool (initctl) has been written, compatibility tools such as shutdown will be written over the next week or two before it replaces our sysvinit package.

Wow, haven't had time to post in a while. Some major highlights over the past couple of weeks:

(longish entry, hidding behind an lj-cut to avoid flooding various aggregators...)



Opus and ConCentric

Angie and I went to Opus and ConCentric. For those not familiar, it's an annual UU Young Adult (18-35yr olds) campout and spritual retreat. When we're there, we do activities, workshops, worship, cook, clean, sit around the fire, swim, talk, laugh, cry, and all the things that come when you throw people together in a space for 10 days. Every time that I go, I both miss it and am glad when I leave. Over the course of the year, I start to think about only the good parts and when I go back. But when I'm there, there's tempers, hurt feelings, things go wrong. At the same time, when I'm there, I meet some of the people I'll connect with the strongest over the course of the year. Sometimes just a feeling or a song from the camp will help me reconnect and push through in the year in between.

This Opus in particular was interesting. I'd been away for two years, after having been on the planning committee for two years. There were things I loved: There were new people, new songs, new games. I learned to crochet, swam naked under the stars, reconnected with many old and new friends. This particular camp in Maryland is the one at which I proposed to Angie in 2000. There were things that sucked: The cooking staff had trouble getting food out for the first few days, which resulted in one of them quitting; a few hospital visits for people onsite stressed my friend who was the first aid attendant. Fortunately, Angie and I have worked in camp kitchens before, and she was able to step in and fill that roll. A ...

I need a collective for a group of people that doesn't sound like 90's corporate psychobabble. 'team's are things that compete. Maybe I'll steal the word from the current Ubuntu release.

... knot of 4 of us are applying to be kitchen staff next year. I have to be careful what I commit to this year, since in general all commitments I'm making to people officially expire in February. I can't tell what life will be like on the other side of having a child!

It's also hard going to a camp where we've been before, I think. My memories of that camp are mostly centred around people who weren't there. One of whom is dead, many have aged out, many just didn't come this time. I miss all of you.

For the drive down, Angie and I rented a Smart Car from Avis. (Photo by Simon Law) Driving down to Maryland was fun. These things aren't for sale in the US until 2008, and there were many pictures of us taken on the drive and at gas stations. At least a dozen SUV drivers walked over to us to ask the mileage. It sustains 120km/h with no problems and doesn't get blown around too much on the highway. A few times I took it up to 130km/h, but at that point it starts to blow around alot. It *is* tiny, though. Packing enough for Angie and me to travel for 10 days was a bit tight, and driving a Yarris when I got back felt *huge*. On the way down, including Angie going to work once, and us taking a detour, the Diesel cost us $35, and the tolls cost us $40.

We did the drive down in two days. The first day making sure that we crossed the border and getting from Montr al to Albany. On the drive down on the second day, we didn't want to hit the NJ Turnpike at 5pm on a Friday, so we stopped for lunch at a place called Eduardo's Pizza in Boonton, NJ. They had some of the best pizza we've ever had, and the nicest staff. They gave us directions to a waterpark nearby which had a neat ride called High Anxiety (Picture 2 Picture 3). I prefered to call it Funnel of Doom. The park was overpriced for a full day for what you'd get, but the half-day discount prices were worth it and the park was mostly empty because of a state fair.

The drive back was mostly a day of hard driving. Being at camp was wonderful, but we wanted our own *beds*. We got a bit lost coming back, and somehow paid less than $10 in tolls, despite travelling in the same amount of time. I like adventures like that.

So what's next? Re-entry into the world is almost done, but now there's re-entry into the mainstream UU communities. This is the part I find the hardest. Opus is such an idealised world, where we talk about anti-oppression work, changing the world, changing ourselves to help create this new world. The worships are spritually alive (Elandria, YOU ROCK MY WORLD!) and the group doesn't stay long enough for alot of politicking to set in. Our jobs leaving there are always to take a part of that community and recreate it elsewhere. I think it's the toughest job I've ever had.

Go out into the highways and byways. Give the people something of your new vision. You may possess only a small light, but uncover it, let it shine, use it in order to bring more light and understanding to the hearts and minds of men and women. Give them not Hell, but hope and courage. Do not push them deeper into their theological despair, but preach the kindness and everlasting love of God.
- John Murray

For whatever God means to you of course. Blessed be. =)

It's often the case with technology that serious changes occur at a particular price or performance point in development. Something has small use until it can be developed to a certain combination of low price and high performance that everyone demands.<br /><br />I believe that USB flash devices are going to be used for many interesting things starting about now. The reason is that 2G flash devices are now on sale for under $100. To be more precise 1G costs $45AU and<br />2G costs $85AU.<br /><br /><a href="http://www.coker.com.au/hardware/usb.html">http://www.coker.com.au/hardware/usb.html</a><br /><a href="http://www.coker.com.au/hardware/usb.html"><span class="" style="display: block;" id="formatbar_CreateLink" title="Link" onmouseover="ButtonHoverOn(this);" onmouseout="ButtonHoverOff(this);" onmouseup="" onmousedown="CheckFormatting(event);FormatbarButton('richeditorframe', this, 8);ButtonMouseDown(this);"></span></a><br />The above page on my web site has some background information on the performance of USB devices and the things that people are trying to do with them (including MS attempting to use them as cache).<br /><br />One thing that has not been done much is to use USB for the main storage of a system. The <a href="http://laptop.media.mit.edu/">OLPC </a>machines have been designed to use only flash for storage as has the <a href="http://familiar.handhelds.org/">Familiar</a> distribution for iPaQ PDAs (and probably several other Linux distributions of which I am not aware). But there are many other machines that could potentially use it. Firewall and router machines would work well. With 2G of storage you could even have a basic install of a workstation!<br /><br />Some of the advantages of Flash for storage are that it uses small amounts of electricity, has no moving parts (can be dropped without damage), and has very low random access times. These are good things for firewalls and similar embedded devices.<br /><br />An independent advantage of USB Flash is that it can be moved between machines with ease. Instead of moving a flash disk with your data files you can move a flash disk with your complete OS and applications!<br /><br />The next thing I would like to do with USB devices is to install systems. Currently a CentOS or Red Hat Enterprise Linux install is just over 2G (I might be able to make a cut-down version that fits on a 2G flash device) and Fedora Core is over 3G. As Flash capacity goes up in powers of two I expect that soon the 4G flash devices will appear on the market and I will be able to do automated installs from Flash. This will be really convenient for my SE Linux hands-on training sessions as I like to have a quick way of re-installing a machine for when a student breaks it badly - I tell the students "play with things, experiment, break things now when no-one cares so that you can avoid breaking things at work".<br /><br />The final thing I would like to see is PCs shipped with the ability to boot from all manner of Flash devices (not just USB). I recently bought myself a new computer and it has a built-in capacity to read four different types of Flash modules for cameras etc. Unfortunately it was one of the few recent machines I've seen that won't boot from USB Flash (the BIOS supported it but it didn't work for unknown reasons). Hopefully the vendors will soon make machines that can boot from CF and other flash formats (the more format choices we have the better the prices will be).

There is ongoing discussion about whether outsourcing is good or bad. The general assumptions seem to be that it is bad for people who work in the computer industry (more competition for jobs and thus lower pay) and good for employers (more work done for less money).<br /><br />I am not convinced that employers can get any benefit from outsourcing. The problem is that the pay rates for computer work are roughly proportional to the logarithm of the productivity of the person (at a rough estimation - it's certainly not linear). Therefore if you get an employee on twice the base salary you might expect ten times the productivity, and an employee on three times the base salary could be expected to deliver one hundred times the productivity. These numbers may sound incredible to someone who has not done any technical work in the computer industry, but actually aren't that exciting to people who regularly do the work. Someone who knows nothing may perform a repetitive task manually and waste a lot of time, someone who knows a little will write a program to automate it, and someone who knows a lot will write a program to automate it that won't crash...<br /><br />Programmers in Indian outsourcing companies are paid reasonably well by Indian standards, but they know that it's possible to do a lot better. So all the best Indian programmers end up either migrating to a first-world country or running their own outsourcing company (there are a lot of great Indian programmers out there, but they aren't working in sweat-shops). The Indians who actually end up doing the coding are not the most skilled Indian programmers.<br /><br />It might be better to hire cheap Indian programmers of average skill than cheap first-world programmers of average skill. But hiring a single skilled programmer (from any country) rather than a team of average programmers will be a significant benefit (both in terms of price and productivity). In addition to this there are the communication problems that you experience with different time zones (the idea that one team can solve a problem while the team on another continent is asleep is a myth) and with different cultures.<br /><br />I am not convinced that outsourcing does any real harm to good programmers in first-world countries. If someone does computer work strictly 9-5 and never does it for fun then they are not a serious programmer. People who aren't serious about computers will probably be just as happy working in another industry if they get the same pay. Moving a few of the average computer programmer positions to India isn't going to hurt anyone, especially as the industry is continually growing and therefore there is little risk of any given programmer being forced out of the industry. The people who are serious about computers (the ones who program for fun and would do it even if they weren't paid to do so) are the most skilled programmers, they will always be able to find jobs. Will outsourcing reduce the income for such people? Maybe, but earning 5* the average income instead of 6* shouldn't hurt them much.<br /><br />The final question is whether outsourcing is a good thing. I think it is good even though it's bad for first-world companies and not particularly good for programmers in first-world countries. Outsourcing benefits developing countries by injecting money into their economies and driving the development of a modern communications infrastructure (telephones, mobile phones, fast Internet access, reliable couriers, etc). I believe that the good which is being done in India by outsourcing money greatly exceeds the damage done to companies that use outsourcing services. Therefore I want this to continue and I also want to see outsourcing in other developing countries too. There is already a trend in outsourcing to eastern-European countries such as Russia, this is a good thing and I hope that it will continue.

So earlier today (or yesterday, if you want to be technical) I final dragged myself down to Stratton to pick up my MIT student ID and tell the orientation dudes that I’m here, even though I’ve already been here for two months. I ended up being tackled by a rep from Bank of America as soon as I stepped into the building, who assumed I was Not A Freshman™ when I mentioned that I in fact am already the possessor of not one but two Bank of America bank accounts. The wearing of the bike helmet probably helped there, since I doubt many other freshmen are biking to orientation. The ID itself is, to put it shortly, kind of rediculous looking. What with the bubbly background and the shiny “MIT” holograms all over it. It’s definitely better than my currently-used-as-ID New York State driver’s permit though, which has “UNDER 21″ plastered all over it in rather large letters and a rediculously old photo. Which can get pretty annoying sometimes. Now, I just hope I can get the card activated for access to the Media Lab pretty quickly so I’ll stop locking myself out trying to go to the bathroom by the elevator after 6 PM or so. I don’t know about you, but I found that pretty annoying. And I never did manage to figure out exactly what time they started locking the doors, so I did it at least three or so times. Even though my last day of work was technically Friday, I’ll probably go in for a couple hours on a day or two next week. I plan on being a real UROP there during term anyway. Other than the ID, the other major things I picked up were “The Hitchhiker’s Guide to Orientation” which actually does mention some things that may end up being fun during orientation, a t-shirt that just screams frosh, and a book called How to Get Around MIT which actually isn’t really a whole lot about getting around MIT, and looks like it may even have some useful info in it. I blame it on the fact that the book’s written by students. Orientation starts Sunday, but I’m skipping that day to go on a 50-ish mile bike ride on a rather sexy borrowed touring bike. Word.

Evaluating programming languages for playing with Debtags Since having workable bindings for the C++ Debtags libraries seems to be still a bit in the future, I'm planning to build a bit of native infrastructure in some higher level language. First step is seeing what language I could start playing with. The problem At the most basic level, in Debtags we have a number of packages, each of which have a set of tags. The way I usually save tags is a file with the format:

package1, package2: tag1, tag2, tag3
package3: tag1, tag2

That is, every line has a list of packages with the same tags, and the list of their tags. Since any script I'm going to write has to at least be able to parse the data into something like a package -> tags hash, then print it out. Let's see how perl, python and ruby perform. Tests C++ The reference point for the experiment will be the C++ implementation, tagcoll:

$ time tagcoll copy package-tags > /dev/null
real    0m0.421s
user    0m0.412s
sys     0m0.000s

Perl First attempt is with Perl, creating the script that parses into a hash of package => set of tags and prints the result. There are set modules for Perl on CPAN, but I have none handy at the moment. However, since they are implemented using hashes, I can approximate them by using a hash. Note that I also want to have a different copy of the tag set for every package, so that I can manipulate them in the future without unwanted side effects. Here is the code:

#!/usr/bin/perl -w
use strict;
my %db;
# Read the tag database
while (<>)
 
    chop();
    my ($pkgs, $tags) = split(': ');
    # Create the tagset using keys of a hash
    my %tags = map   $_ => undef   split(', ', $tags);
    for my $p (split(', ', $pkgs))
     
        # Make a copy of the tagset
        $db $p  =  %tags ;
     
 
# Write the tag database
while (my ($pkg, $tags) = each %db)
 
    print $pkg, join(', ', keys %$tags), "\n";
 

Here is the running time:

$ time ./parse.pl package-tags > /dev/null
real    0m0.448s
user    0m0.436s
sys     0m0.008s

Not so bad, comparable with tagcoll. Python Then comes Python. I'm not much of a Python fancier, but I'm rather attracted by the new set native type introduced with Python 2.4, which seems to have most of what I need nice and done. Here is the script:

#!/usr/bin/python
import sys
input = sys.stdin
if len(sys.argv) > 1:
    input = open(sys.argv[1],"r")
# Read the tag database
db =  
for line in input:
    # Is there a way to remove the last character of a line that does not
    # make a copy of the entire line?
    line = line.rstrip("\n")
    pkgs, tags = line.split(": ")
    # Create the tag set using the native set
    tags = set(tags.split(", "))
    for p in pkgs.split(", "):
        db[p] = tags.copy()
# Write the tag database
for pkg, tags in db.items():
    # Using % here seems awkward to me, but if I use calls to
    # sys.stdout.write it becomes a bit slower
    print "%s:" % (pkg), ", ".join(tags)

Here is the running time:

$ time ./parse.py  package-tags  > /dev/null
real    0m0.418s
user    0m0.376s
sys     0m0.036s

I'm pleased, very pleased. Using the native set seems to be not only handy, but efficient. Ruby Finally, Ruby. I like to use Ruby. In this case, however, it lacks a native set implementation, although it has a set module which is implemented using a hash. Here is the script:

#!/usr/bin/ruby
require 'set'
infile = ARGV[0] ? File.new(ARGV[0]) : $stdin
# Read the tag database
db =  
infile.each_line do  line 
    line.chop()
    pkgs, tags = line.split(": ")
    # Create the set using the Set module
    tags = Set.new(tags.split(", "))
    pkgs.split(", ").each do  p 
        # Is this a copy or a reference?  I need to find out.
        db[p] = tags
    end
end
# Write the tag database
db.each do  key, tags 
    # Ouch, Set does not do join by itself
    print key, ": ", tags.to_a.join(", ")
end

Here is the running time:

$ time ./parse.rb package-tags > /dev/null
real    0m1.637s
user    0m1.572s
sys     0m0.052s

I hope I got something wrong in the script, but I can't see what. Results As much as I don't fancy Python, it looks like it's currently the best choice for playing around with Debtags. I hope the native sets will bring me joy. If in the future I'll be asked "how come you chose Python for this Debtags thing?", I can point to this page.

Welcome to this 2nd issue of the DeFuBu contest, the monthly championship of the funniest bug reported to the Debian BTS. The challengers

• Julien Danjou with #360285 (ITP: libjoey -- forks itself and does several things at the same time), #102434 (gpm: reverse marking) and #178290 (grave problems when badly mixed with irda)
• Pierre Habouzit with #379912 (kodo: Incorrectly described as useless)
• Julien Goodwin with #318229 (XOrg package is controlled by a woman disrespectful of men (which make up the vast majority of debian developers))
• Frans Pop with #258723 (vim: replace $ shlibs:Depends] in debian/control)
• Daniel Burrows with #331210 (aptitude maintainer should do bug DB clean-up and/or bug merging)
• Andres Salomon with #236874 (xscreensaver: pacman screensaver plays game incorrectly)
• Loic Minier with #382014 (can't fly to girlfriend) and #375789 (gnome-osd: information leakage when screensaver is active)
• Florent Bayle with #381978 (debian-installer: etch d-i installs an antique mouse cursor)
• Julien Louis with #382181 (nmap is a clairvoyant)
• Cyril Lacoux with #373142 (apt-get install spamc on testing removes sarge kernel image)

How the vote has been done Five Debian related people voted for these bugs: Roland Mas, Alexis Sukrieh, Cl ment Stenac, kolter and Yves-Alexis Perez. Full ranking Bugs

• #360285 (ITP: libjoey -- forks itself and does several things at the same time) (52 points)
• #102434 (gpm: reverse marking) (44 points)
• #236874 (xscreensaver: pacman screensaver plays game incorrectly) (43 points)
• #258723 (vim: replace $ shlibs:Depends] in debian/control) (37 points)
• #331210 (aptitude maintainer should do bug DB clean-up and/or bug merging) (33 points)
• #382014 (can't fly to girlfriend) (31 points)
• #373142 (apt-get install spamc on testing removes sarge kernel image) (29 points)
• #379912 (kodo: Incorrectly described as useless) and #382181 (nmap is a clairvoyant) (26 points)
• #381978 (debian-installer: etch d-i installs an antique mouse cursor) (15 points)
• #318229 (XOrg package is controlled by a woman disrespectful of men (which make up the vast majority of debian developers)) (22 points)
• #178290 (grave problems when badly mixed with irda) and #375789 (gnome-osd: information leakage when screensaver is active) (13 points)

Challengers

• Julien Danjou (109 points)
• Loic Minier (44 points)
• Andres Salomon (43 points)
• Frans Pop (37 points)
• Daniel Burrows (33 points)
• Cyril Lacoux (29 points)
• Julien Louis and Pierre Habouzit (26 points)
• Julien Goodwin (22 points)
• Florent Bayle (15 points)

The winners

• Bug: #360285 (ITP: libjoey -- forks itself and does several things at the same time)
• Challenger: Julien Danjou
• Bug reporter: Alexander Wirt
• Voter: Roland Mas with the correct reverse ordered tierc : [1] #236874, [2] #102434 and [3] #360285.

Notes To participate, simply drop me an email with a bug number. About DeFuBu

Just when I was thinking that the Unmaintained Free Software wiki was finally dead, Joey Hess mentioned it in his blog (thanks!), which (it seems) has resulted in a few new contributors (thanks!). In addition, I got motivated to work on it again; I had neglected the wiki for quite a while, partly because I was busy with tons of other projects (which I always am, so I guess that doesn't count), and partly also because it was a one-man project for a long time. Even after being mentioned on Slashdot, Newsforge and a bunch of other sites it never reached a state of halfway constant contributions from other people. But, it somehow survived while one or two similar projects I was aware of died away. As it's the only project of its kind which exists at the moment (to the best of my knowledge) and I really think it's a worthwhile cause, I'll continue searching and adding more projects , in the hope that some interested programmers take over maintainance of one or the other... Any help and contributions are of course very welcome! Joey: As for Xgalaga, it was not yet re-added after I switched from a custom-made CMS to a wiki, but it's there now. That'll hopefully increase it's chances to find a new maintainer ;)

On the whole I agree with Erich. Subprojects are key to preserving positive energy as a free software project scales. But I wouldn’t say the ill effects suffered by projects as they age is just “a matter of size”. And I wouldn’t de-emphasize the importance of guidelines like the CoC in countering those effects, though I’d agree it does not provide immunity by itself. Ubuntu benefits from the Debian social experiment having started first. So the people involved have the opportunity to learn from and improve on earlier efforts while their project is still young. The CoC is no panacea, but it is a clue that lessons are being learned; deliberate effort needs to be made to keep work fun, especially as a project grows and people begin to lose touch with each other. Putting these lessons into practice isn’t always easy, but I think it would help if as individual developers we’d reflect more on how we interact within our projects and work on making improvements. I appreciate the personal guidelines Erich shared and will adapt them to make them my own.

Opensource has some scalability limits. A recent blog post shows that e.g. Gentoo is suffering from similar effects as Debian. I believe this is a matter of size, so I doubt that Ubuntu will be immune to this just by it's Code of Conduct and similar things. There is this well-known idiom: "too many cooks spoil the broth" - I think this applies to Opensource work, too. Given that we all want "democratic" work, without having a cabal do the important decisions, I see only one way to resolve this. Subprojects. Just like a distribution doesn't inherit all the software package communities problems, we should try to avoid "aggregating" all the issues at a distribution level. And if you take for example Gnome, many developers are involved only in a few applications, and the communities in each application are probably not of the "critical mass". Examples of such subprojects in Debian include:

• installer work
• architecture porters
• team maintainance of large package sets

Maybe we can find a way to make more use of these effects, that smaller groups usually work together much better. I'm not sure yet how to add extra "sub-communities". Some like team maintainance oranize themselves, which is probably best. Some parts can move to freedesktop, becoming "upstream", and joining efforts with other distributions. Some were created on "alioth". So I think Debian is already on the right track here. Anyway, I'm not a "key player" in Debian, and I'm probably not going to change this. But I'll personally try to follow some simple guidelines to make life nicer for everyone.

• I try to not participate in flame wars. If a mailing list is frequently ridden by flame wars, it's best to leave it alone. There will just be new flamewars coming up again and again. Instead, I try to use mailinglists with fewer people, that maybe are more on-topic.
• I don't read threads that obviously got out of hand.
• I don't read debian-devel. It's a waste of time.
• I try to keep out of other peoples work. Either I become a "serious" contributor, or I let them do what they think it's best. I'm likely less capable of seeing how it all works together if I'm not closely involved. Pinging or filing "wishlist" bugs is of course okay, but you have to accept the decision of those doing the actual work.
• I try to keep stuff away from the full community. Discussing it within a smaller group (the people who actually do the job) is more productive, and you probably won't get much extra results from asking a larger group - but you'll end up with lots more fighting.

I spent the first 12 days of August at the German Junior Bridge camp^[*] in Rieneck. The reason I'm blogging about that in the "debian" section is that it felt very much like DebConf - lots of weird just-like-you people. We played about two tournaments per day, always using different scoring methods or other ways to make up pairs of players - we were not supposed to play with the same partner twice, though Frederic, my flatmate, and I were casted together for a second time in a young-plays-with-old tourney. In this sport, you are a junior until well past 30, and he was just above the cut of 35 of this event. We also bribed the good fairy to let Eva and me play together in the final masterpoint tourney. We didn't score especially well since we were quite tired, but hey :* The site is an old castle, now operated by a German Scouts organization. Somehow, the castle operates in a different time zone, breakfast was served at 12am, lunch at 7pm, and dinner at 11pm. DebConf should consider adopting that :) It would also be interesting to see if we could have a "performance" at the end, and who would run for Miss and Mister DebConf... The pictures I took are in my gallery. Sadly I didn't take a picture of the coffee tally list where one guy made two ticks in the "flatrate" column 8-) [*] beware, this is ugly html Update: Next year's bridge camp will be from July 26th to August 5th - please pick a disjoint range for DebConf...

I uploaded a new perl package to the archive today Net::HTTPServer a stunningly easy to use HTTP-server module. The last time I did so it was rejected by the ftpmasters who suggested that I used the wrong license. I didn’t so I’m hoping this upload will get approved! (Understandable mistake; the package contains LICENSE.LGPL but is dual-licensed like perl as the code and readme says. It was rejected because I said it was “duel perl” and the reviewer believed it was LGPL.) I’ve been using this module a lot recently at home and work and like it. I might write about using it for random purposes in my copious free time …

In 2003, I needed to write a PHP site that would need to be internationalized, and decided to use gettext() for this purpose. While PHP has proper support for gettext, it unfortunately does not have any builtin way to easily convert the data in the Accept-Language and Accept-Charset HTTP headers into something that can be understood by gettext. So I sat down, and over the course of an afternoon, I wrote 83 lines of PHP code (excluding comments) that would do just that: parse the above two headers, and return a gettext string which represents the best match out of a set of strings specified. It's pretty easy in use, and was written quite generically, so I put a GPL tag on top of it, put it online, added a comment to the documentation of the gettext stuff in the PHP manual advertising the URL, and stopped thinking of it. The project that I wrote this file originally for eventually wasn't even finished; today, I don't even recall what it was. In early 2005, I received a mail from Matthew Palmer thanking me for the code, and telling me that he'd incorporated it in IRM of which he's part of the development team. Which reminded me of its existance. For a short while. Today, I was cleaning out my INBOX, and stumbled upon this old email again. Which got me curious, so I started googling for those 83 lines of code that I'd entrusted to the Free Software world three years ago. I must say I'm pleasantly surprised. My file accept-to-gettext.inc seems to be used in a variety of free software projects, including of course IRM, but also the GnuCash website, or in this french educational thing called SLIS. It seems to be quoted in its entirety on some turkish PHP-related forum. It has been adapted to support different translation systems. And so on... the list is rather long. Can't say I'm not proud. And yet, the code contains a rather silly bug: the assumption that character set encodings are somehow linked to translations. They are not; gettext is perfectly capable of transparently transcribing one character set to another. Oh well...

LWN has an article up about how RPM is forked, with no single agreed-upon upstream and lots of distros going off and patching it on their own:

Fedora currently ships version 4.4.2, and even the Fedora development repository has not gone beyond that. SUSE remains at 4.4.2, and the current RHEL offerings have rather older versions.

(And Debian's at 4.4.1+a year of patches.) rpm is not the only peice of software that is a major component of many linux distributions but has no agreed-upon upstream maintainer, and thus effectively one fork per distro. Another such peice of software is cron. Vixie cron was last released in '93. In many distributions it's still used, but in eg, Debian, the package is the result of 13 years of patches on top of that release. The debian version number is 3.0pl1-96. That's ninty-six versions based on 3.0pl1. The Red Hat sub-version is in the fourtys. There's no standard version anymore, just this pile of patches, and other piles of patches in other distributions. If we're lucky, they at least share the same security fixes. Another example is makedev, which was had its last upstream release in '98 and is at patchlevel 82 in Debian. Or netcat, last released in '96, patchlevel 32 in Debian, and a fork/rewrite at netcat.sourceforge.net. Or xgalaga, whose upstream author has fallen off the net and last released it in '98. The Debian package, which I maintain, is at patchlevel 37, and seeme to be the only place a lot of people can find to send patches to. But I don't want to be its upstream maintainer. This is the kind of thing that makes me gibber in horror when people at Ubuntu talk about Debian and other distros being part of an "ecosystem of software". I don't want to be part of an ecosystem; I'd rather be part of something that works not something that muddles through via massive trial-and-error and redundancy. Biomass is another word for "shit". But I digress.. The most productive thing I've seen done to try to address this general problem is the Unmaintained Free Software wiki, which tries to track this software and find new maintainers. That has managed to match up some software with interested people (though it failed with xgalaga), but I think it actually works less well for big and important software that ends up forked and maintained separately by the major linux distributions. I, myself, am beginning to use the degree of divergence from upstream as an indicator of how well a distribution maintains a given peice of software, with more divergence == bad. A distro that's doing a good job will a) get their patches integrated upstream and b) should work with other distros to find a new developer if upstream goes missing or insane. This is the inverse of what people might naively consider a good metric of how much work a distro is doing, but I think it leads to better software in the long run.

Argh... trying to get caught up on this journal in the wee hours of the morning here in wt:Odense. I've had pretty bad jet lag this trip -- I've never been particularly good with time changes, and the North-America-to-Europe switcheroo really throws me off. So by 2PM I'm falling down on my feet, but at 4AM I'm rarin' to go. Odense has been a particularly nice place to visit. We got into wt:Copenhagen a few days ago, and crashed at the Airport Hilton there. It's actually attached to the airport -- you can just walk right in. Maj got a good deal on a room, and we figured it would make sense to sleep with the baby right there, rather than fight the train system to find another hotel in the city centre. After a free brunch including many different kinds of pickled fish, and a looong nap, we took the train into the Central Station in Copenhagen, right by Tivoli Gardens. There were lines around the block for the amusement park, but we skirted them and headed into the pedestrian area near City Hall. Based on a recommendation in Wikitravel, we went to the line of caf s and restaurants off of Nyhavn, a sliver of water poking into the heart of the town. It was a great walk, and we had a casual sidewalk dinner at a place called Carl's Corner. The canal was lined with boats that had been converted to restaurants -- we may try one when we go back to Copenhagen this week. tags: travel copenhagen jetlag nyhaven

Wikisym The next morning I got up at dawn and got on a train to Odense for Wikisym. I missed my connection at the Central Station, and all the trains were running late, so I got real confused about where to go. The fact that I was carrying all of our luggage didn't help matters. But a few false starts later, I managed to get myself oriented and on my way. The train ride was beautiful -- wt:Denmark is a very pretty country, with rolling grassy hills and clusters of small, peak-rooved houses by the tracks. The trip from Copenhagen to Odense takes you from wt:Zealand to wt:Funen, across a long rail bridge. Very impressive. Once I'd landed in Odense and got myself organized, I slipped into the back of the room for Wikisym to catch the last few minutes of Angela Beesley's keynote address. Angela might be the hardest-working person in Wikidom -- she's a tireless community organizer not only for the Wikimedia Foundation sites like Wikipedia but also for Wikia's sites. After her talk, we did an wp:Open Space session, which continued throughout the day, interlaced with the academic presentations. I'm getting to be a big fan of Open Space; it was really effective for RecentChangesCamp, and I think it works well for organizing a conference. Ted Ernst led the sessions, with the help of Gerard Muller of the Danish Open Space Institute. Probably the most productive Open Space session for me was the one about non-text wikis -- using wiki techniques for video, audio, image and visual language editing. I'm fascinated by this stuff, and it's great to hear that some people are working with SVG and SMIL to do in-browser multimedia editing. The consensus is that the technologies Aren't There Yet but are around the corner. The other major thing I think I picked up from this session was Eugene Eric Kim's discussion of Wiki Ohana. Ohana is the Hawaiian word for family; the Wiki Ohana is a project to develop some interoperability and a transmission of our Wiki principles to new members of the family. I think it's a great idea and I'm going to try to get involved further. tags: wikisym2006 wikisym open space wiki ohana

Odense We had a great walk around Odense's core pedestrian area. It's not exceptional for Northern European towns -- it reminds me of the Netherlands, and Maj of Switzerland -- but the car-free cobble-stone walking areas are a real relief after years in North America. We have some pedestrian areas in Montreal, of course, but not in the same way. Odense is the childhood home of wp:Hans Christian Anderson, and the city is decorated with statues of HCA, and of characters from his stories (the Tin Soldier, Thumbelina, etc.) It gives a kind of fairy-tale air to the town. I'm not sure they should feel so proud of being Anderson's childhood home, though -- from everything I've read, his childhood was nightmarishly cruel, and his abysmally poor family was harshly treated by the society here. But, of course, that's in the past. Now, we had a real nice time walking around. Amita June likes tottering along on the cobblestones, and shouting really loud in the small streets to hear the echos. We ended up having dinner in a place called the Cuckoo Caf -- great food under a spreading tree in a little square. This morning -- no, yesterday morning -- Maj got up and took the first sessions, and Amita June and I slept in. We went off into town again to find a bakery, and shared a chunk of brunsviger, a sugary pastry that will probably rot out her tiny teeth. I had a cup of tea, which was less than satisfying. I've been trying to cut caffeine out of my life, so we've been having decaf at home, but they don't seem to serve it in caf s here. I'm a creature of habit, and I like the taste of coffee even if it doesn't have the caffeine oomph. tags: odense amitajune decaf hca

CAPS LOCK SUCKS In response to Lars's defense of CAPS LOCK: Lars, you ignorant slut. Caps lock has been the bane of online culture since the invention of mixed case in the late 1960s. It is a hallmark of clueless newbie-ism for people who are too analphabetic to remember the exceedingly simple English capitalization rules. The only legitimate purpose for CAPS LOCK is for doing C macros and defines, but hey, who writes in C any more? Anyways, that's just lazy programming. A real programmer would write a macro to capitalize the names of other macros where used. That's just good programming practice. Another important point is that CAPS LOCK is Eurocentric and imperialistic. What value does CAPS LOCK have for someone writing in Eritrean, Malayalam or Ojibwa? None at all. Yet dead white males like Lars Wirzenius want to force their CAPS LOCK cultural hegemony on an increasingly global culture. For shame. The worst thing about CAPS LOCK is that it squats in the middle of the most important letters on the keyboard, like a rat trap waiting to spring. Big and ugly, it always catches your left pinky when you're reaching for Ctl or Shift or Tab or A or whatever. Key-chording is here to stay, and CAPS LOCK not only ignores this important UI development but actively interferes with it. Modal interfaces are rightfully extinct, and CAPS LOCK needs to go the way of the dinosaur. It's time to eliminate this obsolete key once and for all. (This message brought to you by The Taxpayers Against Caps Lock Coalition. I, Evan Prodromou, endorse this message.) tags: caps lock ignorant slut

People writing about how the "new graphical installer" looks the same as the one "in text" don't understand debian installer. If you want to know the truth, 'get the facts':<br /><ul><li>There's a (not so) new frontend (GTK+ based) - read 'skin', not a entirely new installer</li><li>The questions will never be totally different than the dialog frontend (read 'blue' background) or readline frontend (read 'text mode'). It makes no sense<br /></li><li>The partitioner can look better in the GTK+ frontend and hopefully will, after Etch release<br /></li><li>If in doubt ask in debian-boot (http://lists.debian.org/debian-boot) for technical issues or debian-publicity (http://lists.debian.org/debian-publicity) for press related stuff, before post your shiny new article about beta quality software that you've no idea what really does and contains behind the scenes</li><li>debian-installer development page is http://www.debian.org/devel/debian-installer wiki.debian.org contains useful information to make a good or poor article too, but it's like the pile of papers in our desk, nothing more.<br /></li></ul>

<span style="font-weight: bold;">Debian Python Modules Team</span> <span style="font-weight: bold;"><span style="font-weight: bold;"><br /></span></span>The group is receiving new members, old packages from these new members and of course brand new packages every week, to say the least. I would like to say thanks for those who joined and for buxy, fatalerror and arnaud. We're working on a lot of interesting stuff and QA in the packages, you will notice the difference in Etch. Btw, i hope to see just Python 2.4 (and not 2.3 too, since we already dropped 2.1 and 2.2) in our upcoming release and a lot of interesting modules already packaged and well supported (dpmt is working on this, right now).<br /><br /><span style="font-weight: bold;">pkg-gnome</span><br />I'm a newbie in the group, i'm cleaning up bugs (pinging submitters) and already did some work on the packages on our svn repository. Unfortunately my first upload (totem) was rejected due to some naming and compatibility issues with the browsers plugins (mozilla vs libxul). seb128 jumped in and saved us. totem 1.4 is still almost ready to go and the rest of the gnome 2.14 is entering the archive.<br /><br />Well, i'm counting the days to the debcamp. Hopefully i'll participate in: ltsp modularization (pkg-ltsp), cleaning up bugs (pkg-gnome) and do some utnubu (read desktop) related work on tasksel. Since utnubu-meta was rejected. :/<span style="font-weight: bold;"><br /><br /></span>