I’m just sitting in the night venue of Debconf7 and do several tasks in the same moment, that includes: * Getting kind of a spanish course on IRC in #debian-qt-kde * Get some news from friends in .ch via MSN (there were several unweathers and other troubles) * Think about tdbackup which is the topic of this blog entry. So what is tdbackup? It is just another backup implementation but it should work kind of distributed. I manage several servers and in that farm are 3 backup servers with a tape drive, every sunday i mount parts of the other machines via NFS and write them with tar to the tape. Until I did that by hand because there was no implementation which fits my needs unfortunately. This is the point where tdbackup jumps in. I want to create a GUI which edits an ini file in the form of: [backupmachine 1] mount = bla1:/home mount = bla2:/home tdbackup will then distribute a specifically generated configfile to each backuphost and a little python script which is called by cron and mounts the file systems, write some information into mysql and generate warning messages via e-mail if the tape is almost full. So, I will be able to sleep better because the stuff is auto-documented in a way. I’m still thinking about how exactly i want to implement that, if there are any interested people who would like to join that project, please just tell me.

So, I'm back from Holidays, to find we now have a new Government in Ireland. (An unexpected choice of government, of which more later.) We were camping in Fouesnant, in Brittany; Family & Friends might like to see the holiday snaps here). Camping in France went well. With three children aged 7, 4 and 2, there is only so much you can expect from a family holiday. Like it or not, you have to mind the toddler fairly consistently, and the older kids would like your full attention too. This is a holiday ''for'' the kids; plan also a break ''from'' the kids later (a long-weekend, with kids at grandparents, for example). This was our first time camping abroad, and its well recommended. While we went touring with a camper van with the two older children two years ago, this is different in that we were at the campsite for almost two weeks: the kids got to meet and play with others in a way they couldn't before. Also, there were good facilities at the site in Fouesnant, and the swimming pools in particular were appreciated (Caveat emptor though: there were no lifeguards or supervision. We went off-peak, and there were apparently two accidents in the time we were there: a broken kneecap and chin when people came down the slides too soon after others and collided). It was the first time the children (a) had pocket money to spend, and (b) were able to walk to a (campsite) shop to spend it. Lessons were learnt in the finiteness of pocket money, especially by Caoimhe . The weather was fairly good: two wet days (though not overly so, compared to what we're used to in Galway); enough to give the 'authentic camping experience'. And for the children especially, just warm enough. Long drives South were not an option with young kids, and we wanted to see Carnac and Brittany in general. We went touring to two places: Mont St. Michel, on the way down, Carnac, and locally around Quimper and Concarneau. At seven, Cormac is beginning to be old enough to enjoy this, but Sadhbh at 2 is too young, and easily aggravated. Caoimhe can appreciate some of it, such as the well recommended Breton museum in Quimper, seeing the local costumes and clothing is just her style at the moment. We're looking forward to going back when the kids are older, or we can enjoy touring on our own. The camp site is dominated by English / Dutch / Irish. It is still term-time in Britain and Ireland, which meant the children on the site was full of 2-4 year-olds, with a shortage of playmates for Cormac. Next time, synchronise with English school times, etc. We went during Cormacs term-time because it was much cheaper (1000 Euro) than high-season. If Cormac was having difficulty in school things would have been different, as it is, we had no qualms in doing so. Things will probably be different next year with Caoimhe also in school at the same time, and we would go later. Suffice it to say I think synchonisation of school timetables is a bad idea, as it has led to these huge price differentials and parents taking their kids from school. As a friend with experience of the French school system points out, they've spent the last 40 years desynchronising their timetables ... So would we go again, or buy our own tents, etc? possibly. The brochure for next year arrived already, with large discounts for booking early ... Tags holiday, family, france, brittany, cormac, coimhe, sadhbh

Since the release team update got the arm architecture names mixed, chances are that others are confused as well.

* arm - This is the current in-Debian, little-endian hard-float old-abi port.

This is somewhat inefficent port, as the hard-float code needs to be emulated
in the kernel. It is also depreciated by upstream.

* armeb - This was a effort to create an big-endian old-abi port.

Since the Linksys NSLU-2 got it's Ethernet driver reverse engineered and thus it became possible to run little-endian Debian on it, interest on this port has been weak. If interest in bigendian arm returns, it will probably be re-ported using EABI and softfloat.

* armel - This the shining new little-endian EABI (and thus soft-float) based architecture.

THIS IS SPARTA^W THE FUTURE.

A month ago I asked for help to localize reportbug-ng. To my surprise translations came in pretty quick. Junichi Uekawa was the first one, he sent his Japanese translation only a few hours after my request for help. Currently we have rng translated into the following languages: Catalan, Czech, English, German, Japanese, Spanish and Swedish. Thanks David Planella, Miroslav Kure, Junichi Uekawa, Javier Fern ndez-Sanguino Pe a and Daniel Nylander for your translations! But that’s not enough. If your language is missing, please consider translating rng to your language. Just take the source of the current version in unstable, grab the .pot and translate it. There isn’t really that much text to translate. The .pot contains only 42 messages and most of them are very short. When you’re done, please file a l10n+patch bug against rng with the translation attached. I’ll include your translation as soon as possible, thank you in the changelog and will probably name my first pony after you Thanks!

The source-searching system I was talking about previously is progressing slowly. So far I've synced the source to Etch to my local machine, total size 29Gb, and this evening I've started unpacking all the source. I'm still in the "a" section at the moment, but thanks to caching I should be able to re-sync the source archive and unpack newer revisions pretty speedily. The big problem at the moment is that the unpacking of all the archives is incredibly slow. Still I do have one new bug to report aatv: Buffer overflow in handling environmental variables.. That was found with:

rgrep getenv /mnt/mirror/unpacked   grep sprintf

(A very very very slow pair of greps. Hopefully once the unpacking has finished it will become faster. ha!) The only issue I see at the moment is that I might not have the disk space to store an unpacked tree. I've got 100Gb allocated, with 29Gb comprised of the source. I'll just have to hope that the source is less than 70Gb unpacked or do this in stages.) I've been working on a list of patterns and processes to run, I think pscan, rats, and its should be the first tools to run on the archive. Then after that some directed use of grep. If anybody else with more disk space and connectivity than myself be interested I can post the script(s) I'm using to sync and unpack .. Failing that I'll shut up now.

One feature I’ve always found terribly useful for team maintained packages are commit emails. They help you know what’s going on without need for extra communication (eg. that somebody started packaging a new releases), and they are useful when there are new maintainers, because their work can be easily reviewed. Centralized version control systems like CVS or Subversion make possible and easy to have such messages sent automatically upon commit, so all fits nicely. Enter distributed version control systems; particularly those like Bazaar that can push over dumb protocols like SFTP: it is not possible to have the server do something (send an email) when new revisions are commited. One has to resort to solutions like bzr-email, where each individual commiter configures a plugin to send out the email from their machines when they commit. The result IME is that, either nobody does that, or not everybody does. Either way, no cookie. So I decided to bite the bullet here at DebConf, and write a tool to solve this problem: a program that monitors the branches in the machine where they are hosted, and sends emails for new revisions. The result is here. If you’d like to use it for an Alioth project, you just need to do:

 alioth% /var/lib/gforge/chroot/home/users/adeodato/bzr-hookless-email/alioth_helper.sh $PROJECT_NAME -d

(And ensure that the process does not die, or gets restarted when it does. I guess I could add code for that in the helper script, hm.)

I've a Toy Story to tell you today and to make a long story a short one: debian-br is proud to announce the birth of a new baby during debconf 7, smile and say hello world to: get.Debian.net

If you don't want to figure out by yourself which one of the hundreds images (ISOs) available is the best for you to install Debian GNU/Linux. We do our best with get.Debian.net to figure out that automagically. You just need to use one of these javascript enabled browsers - yeah, nothing is perfect.

What it does?

• List the best images available to your architecture based on your browser information
• List Win32 loader for those browsing the web site using Windows
• get.Debian.net/bittorrent gives you link for torrents
• Make Debian looks more sexy. Good enough, Sam? :-P

TODO

• Use GeoIP for mirror selection - code in place, need list of mirrors
• i18n support
• List Live CD image where possible (i386 only?)
• Add get.Debian.net/kde, /xfce and /gnome support
• Figure out how to include usb and other images on the 1-click soup

Download Debian: http://get.debian.net/
Download Debian using Bittorrent: http://get.debian.net/bittorrent

Special thanks to faw and valessio.

My laptop's power supply just died, and I'm running on borrowed time/power. Gonna suck until I get my replacement. :-( I have a sneaking suspicion this might be due to British power, though of course the power supply is rated to work over here. (Update: If anyone here has a spare 16v / 3.75 amp supply that I can cut the cord off of and canabalise, that would be appreciated..)

So, I'm happy to see that commercial wiki provider WetPaint now supports OpenID for logging into their site. It's great to have more wiki services supporting this single sign-on standard. A couple of nits, though: first, I logged in with my normal OpenID, http://evan.prodromou.name/ , which delegates to http://certifi.ca/evan . WetPaint's OpenID service used the delegated URL, which is just wrong. They need to fix that. The delegate is the "backend", and users should be able to change it over their lifetime. If relying parties store that backend URL instead of the "main" OpenID URL, that undermines the process. Second, they have a signup page after the validation, and it has a lot of fields that could be pre-populated using the OpenID Simple Registration extension. I shouldn't have to type in my preferred user name, real name, or email address. That information is already available through my OpenID provider. Other than that, it seems to work OK. I'm glad that it exists at all, but I wish it was a smoother user experience. tags: wetpaint wiki openid sso

Wikispaces and SourceForge I haven't already mentioned the announcement that SourceForge has implemented Wikispaces wikis for every project on their site. I think that's great news for the wiki community, and probably a good thing for SourceForge users. I feel churlish mentioning it, but it has to be said: with hundreds of available WikiEngines, the vast majority of which are Open Source, and many of which are hosted on SourceForge, why did SourceForge go with a commercial service provider? It doesn't seem like they've justified that decision very well. tags: wikispaces sourceforge wiki announcement

Some time ago, I needed a small but helpful tool which could start some shell command with bounded time and kill spawned process if it was timed out. There is timeout package in Debian but it has very serious bug #187092. In fact, this tool can't intercept the proper status of called command and can't kill processes spawned by called command. I've tried to fix the Debian's timeout package but the bug is still opened over 4 years. So I decided to code the own Perl implementation. The result was attached to bug report. I've noticed that my script is currently used by Exim packagers. It's nice to see that my work is still useful. I think more people can find this script helpful so I set the Google Code project: timeout-pl. You can download the newest version of this script from Google or CPAN. Have a fun.

There is some effort here at Debconf to improve Debian’s game stuff. I want to contribute to that by announcing my first upload of the game “Feeding Frenzy!” to Debian Sid. The game is about dwarfs (or sheep) that have to collect food to survive which becomes increasingly scarce during the game. Besides collecting food, the dwarfs can also can choose to throw the food at competing dwarfs to temporarily disable them. Four years ago, a group consisting of some of my friends and myself created this game as an assignment for a project in our bachelor computer science. A few weeks ago, we undusted the game, Bram fixed some compiler warnings and I fixed up the package. It is ready for Debian now and has a pretty good gameplay, though it can probably still be improved graphic- and level-wise. I uploaded it two days ago. So, if our FTP-masters are no longer busy with Debconf, I hope they can get the game through the NEW queue soon.

In the continuing effort to clear some space and pay some bills, I'm selling my collection of Babylon 5 DVDs. I have the boxed sets of series 1 through to 5, in great condition: they've been watched once and then put on the shelf, where they've stayed. Each series retails at £57, I'm asking for £90 for the complete set (plus shipping, or we meet in Cambridge or London for the cost of a beer). If anyone is interested, please mail me. Update: I've got a buyer!

I finally decided myself to switch my laptop to using a LVM over LUKS setup instead of LVM alone. The problem it involves is that there is currently no way that I know to do this in-place. I do have an external disk on which I could pvmove everything and back, but that would not be very challenging ;). So I started wondering how doable that would be. LUKS has a 1032 sectors (a sector being 512 bytes) overhead in which it store the LUKS header. Following the header, encrypted data just takes as much space as its decrypted counterpart, linearly. So all it would take to do in-place LUKS conversion would be to move data by 1032 sectors, and encrypt it. Which means the main problem is to have 1032 sectors free after the current volume. Fortunately, LVM doesn’t take up all the space on the partition a physical volume is setup, except if you’re unlucky or created partitions specifically so that it uses all the space. Because LVM physical volumes use the space by big chunks, 4MB by default), if the size of the partition is not 4MB aligned (modulo the LVM header), you have some free space between the physical volume end and the partition end. We thus need this free space to be greater than 1032 sectors. On my laptop, there is more than 1.8MB free there, so there would be no problem.
Note the debian-installer seems to do more than a simple cryptsetup luksFormat, as LUKS volumes it creates have an overhead of 2056 sectors instead of the 1032 sectors I got creating a volume by hand. Next problem is how to move the data and encrypt it in-place. Bonus points if that can be done without unmounting the filesystems. My first guess was to play around with the device mapper and a specially crafted FUSE filesystem to create a fake LUKS volume that could be half encrypted and half not, and would evolve, with time, into a fully encrypted LUKS volume. Since I didn’t know much about the device mapper, I looked at it a bit, and discovered a FUSE intermediate would be unnecessary: not only can you remap on-the-fly (which I already guessed was somehow possible, considering how LVM can do it, though I didn’t know much about the interaction between LVM and the d-m), but you can also mix dm-crypt and linear chunks in the same d-m table. It also appears LVM over LUKS has an overhead that is actually not necessary. How sad both use the device mapper but yet can’t do an efficient setup with it. When you setup LVM over LUKS, you open a LUKS volume, which will create a dm-crypt device mapper, and use this device mapper as the physical volume for LVM. Logical volumes created with LVM are actually device mappers using the physical volume as destination device. When you access a block on an LVM logical volume, your access is first mapped through the LV device mapper and then mapped again through the dm-crypt mapper. So it goes through 2 device mappers, but it could do with only one: the linear mapping used in most of the cases with LVM could be replaced by crypt mappings, that are just as linear, but also handle the encryption/decryption part. Playing around under qemu with a freshly installed etch system, I was able to implement this and came up with the following script. It should work with most setups, but use at your own risk, it comes with no warranty.


MAJOR=$(awk '/device-mapper/ print $1;exit ' /proc/devices)
dmsetup ls --target linear while read map extra; do
  TABLE=$(dmsetup table $map while IFS=" :" read start length linear major minor offset; do
    map2=
    if [ "$major" -eq "$MAJOR" ]; then
      IFS=":" read map2 extra <<EOF
$(dmsetup info -c --noheadings -j $major -m $minor)
EOF
    fi
    if [ -z "$map2" ] [ $(dmsetup table "$map2" wc -l) -gt 1 ] [ $(dmsetup table --target crypt "$map2" wc -l) -eq 0 ]; then
      echo $start $length $linear $major:$minor $offset
    else
      IFS=" " read cstart clength crypt format key IVoff dev coffset <<EOF
$(dmsetup table "$map2")
EOF
      echo $start $length crypt $format $key $(expr $IVoff + $offset) $dev $(expr $coffset + $offset)
    fi
  done)
  if (dmsetup table $map; echo "$TABLE") sort uniq -c grep -v "^ *2 " > /dev/null 2>&1;
  then
    echo Diverting $map
    dmsetup reload $map <<EOF
$TABLE
EOF
    dmsetup resume $map
  fi
done


[ Update: There was a dmsetup resume missing in the script ] After running this script, you should be able to run cryptsetup luksClose on your LUKS volume. Once you’ve done it, pvdisplay and other LVM tools will show you nothing, because the physical volume will just have disappeared… but everything will still be in place. Note that if you leave the LUKs volume open, you’d better be careful NOT to do operations with LVM (such as pvmove) because it could break everything. Note that on sid, you would need to add a –showkey option into the $(dmsetup table "$map2") line. This option is necessary for dmsetup to display the encryption keys. I wonder if LVM upstream would consider supporting LUKS volumes directly… Anyways, back to the in-place conversion to LVM over LUKs. The idea would be to:

• Create a file the size of the LUKS headers + one block.
• Create a loopback device for this file.
• luksFormat the loopback device.
• luksOpen the loopback device, so that dmsetup can give us the key to use. I wonder if there are ways to get the key not involving the creation of the device mapper.
• luksClose it, because we actually don’t need it
• Create a device mapper that linearly maps the whole LVM physical volume (i.e the real partition, but without the empty tail).
• Remap the LVM device mappers so that they use our own device mapper instead of the partition as physical volume.
• Here is the trickiest part, and I still have to take a look at how pvmove does to know exactly how this can be done (I guess it will require an additional device mapper): move data chunk by chunk (offsetting with the size of the LUKS headers), starting from the end of the LVM physical volume, and encrypt at the same time. Once a chunk is moved, adjust our device mapper so that it maps to the linear unmoved data followed by the moved encrypted data. As I said earlier, this is a setup that works.
• Once everything is moved, we are in a situation where our device mapper is a single crypt mapping, in the same shape as a luksOpen would have made it. We can now copy the LUKS headers at the beginning of the partition, and add the appropriate configuration to /etc/crypttab.

If I got it all right, both LUKS and LVM should be believing they have done this setup themselves, so unmounting, lvchange, etc. should just be fine. It might be necessary to regenerate the initramfs, though. I’ll tell you when I fuck up my hard disk try it. To be continued…

So, yeah, I’m in Edinburgh right now. For those of you who have asked, here is a little track of my adventures in the previous days:

• Saturday morning. I leave my apartment in Mexico City at around 4:30 am. Raquel takes me to the airport. We both realize how we could not work properly without each other.
• Saturday, 1 pm. Arriving in New York City. Given the time and the stupidly long line for passport/visa control, I drop my plans of visiting the city.
• Saturday, 3 pm. After getting used to the AirTrain at JFK, I finally find Terminal 4, in order to check whether FlyGlobeSpan is really there. I realize that screen in the terminal announce my flight as cancelled.
• Saturday 5 pm. We receive an explanation, that the company representing the airline in the airport did not receive the proper aircraft (!), so the flight was cancelled. We all get accommodation at the DoubleTree hotel near the airport. I blame and say bad words about the situation. My flight was being rescheduled to Sunday or Monday, upon availability.
• Saturday 6:30 pm. I was chatting with my people from the hotel room. I really love wifi.
• Saturday 7 pm. I take the hotel shuttle to get to the AirTrain and then get to the subway system in New York.
• Saturday 8:30 pm. Getting to Chambers St. in Little Italy, Manhattan. Walk a few blocks and have some pizza with some latinos. Realizing that they also sell pirate DVDs in NY’s metro and pirate lenses and hats on street markets. Have my first Budweiser.
• Saturday, 9 pm. After calling Raquel, I realize the time and in order to not miss time or whatever and given how tired I was, I decide to go back. I go up the subway on Essex station.
• Saturday 11:30 pm. On my way to Jamaica Station, which connecs to the AirTrain, they took us out of the train because it was their last stop (missing like 5 stations to get to the real last one). We get on a free shuttle. While waiting for it, I realize that most of the people there are Caribbeans and Mexicans, most people speak Spanish or a Caribbean English. Get on the shuttle. A black lady gets on and start telling Jesus is coming and we are all sinners, she gives people flyers on the holy things and the evil things. People (specially youngs, wearing sportswear) make fun at her, I only feel sorry. A latina rubbs a latino’s arms and admires his tattoos (both complete strangers that eventually met on the shuttle), while being invited to their party. Just as I said to Biella, it’s a bit outstanding to me since it’s the first time I’m in NYC and happen to be real what neighborhoods seem in the movies
• Saturday, 12:30 pm. Finally got to the hotel, after being waiting for a taxi in some terminal at JFK. The hotel shuttle showed up and they brought me there.
• Sunday, 11:00 am. I get a phone call from the reception hotel saying that checkout time is at 12. Hurry to shower, do some stuff and look for some other flights. I find a good price for a flight from NYC to Dublin, for that same day. Consider it, talk to Raquel about it and try to buy it. Not possible, my credit card is denied for some reason. Take the shuttle to the airport.
• Sunday, 1:00 pm. The AerLingus ticket sales are opened until 2:30 pm (what kind of timetable is that?). Call their sales line, talk to a representative and I get the flight.
• Sunday, 2:00 pm. I’m waiting to check-in my luggage. Plane leaves at 17:40.
• Sunday, 3:00 pm. Having a Budweiser at some bar in the terminal. Have some pizza.
• Sunday, 4:00 pm. Buy a flight from Dublin to Edinburgh on RyanAir. The flight from NYC arrives at 6:00 AM on Monday and the flight to Edinburgh departes at 8:40 PM.
• Sunday, 6:00 pm. I take off from NYC and get on my way to Dublin, Ireland.
• Monday, 6:30 am. I land at Dublin.
• Monday, 8:00 am. After having a coffee and a donut I realize that the UK and Ireland does not use the EU power adapters, so I’m unable to get online or something because battery was empty. I sit on a couch and try to get some sleep.
• Monday, 12:00. I wake up with a terrible neck and pack pain and realize that 8 hours are still missing. Start to read.
• Monday, 15:00. After three hours of reading I’m tired and consider going to the city.
• Monday, 16:00. Dropping the idea of going to the city I simply wait and wait and wait. I realize they drive from the right side of the car.
• Monday, 18:00. I had overweight on my luggage. I took the bottles from the checked-in luggage and was pretending to take them with me. They tell me I’m not allowed to carry them anymore. They tell me that the cost of the extra weight would be 40. Decide to drop them. The lady tells me that I could bring one to the luggage. I give her a tequila Azul. I drop the third bottle on the toilet. The mezcal worm goes in circles until it’s gone.
• Monday, 21:00 pm. I take off from Dublin.
• Monday, 21:45 pm. I land at Edinburgh.
• Monday, 22:15 pm. I’m on my way to Teviot. I find Edinburgh to be just another reality of life, with all its incredible landscapes, buildings and streets. I realize that people indeed use kilts (Scottish skirts, for those like me unfamiliar with the term) without underwear.
• Monday, 22:40 pm. I find all people coming out from Teviot on Bristo Square. I join some Venezuelan buddies. Everybody heads to the Chesse and Wine party. Go to the hostel with Orlando and leave my luggage on the room.
• Monday, 23:30 pm. Join everybody in the party and have some nice red wines, Chinese (yes, not Japanese) sake, tequila, Cuban rum.

After all, I had made it to DebConf7.

Debconf7 continues very well, with a nice trip to a local sauna this evening. Unfortunately our party had to split into two, with the ladies going to one section and the gentlemen to another. Still it was fun, and I'm glad I went. It was probably just as well that Megan didn't get to see me being terrified of the cold water! Apart from that things went well today. I saw half a talk on Xen, sufficient to see my name in lights, then half a talk on security, again managing to see my name on the big screen. If people, at debconf7, would like to learn more about security I've volunteered myself on the "skills exchange" page of the wiki to demonstrate the process which Moritz described as "manual and complicated". (ie. releasing a DSA and updating the webpage.) I think this will happen sometime on Friday. I also need to track down AJ and talk about debootstrap work. One more thing, before I go to sleep. During the security talk Moritz did mention the idea of being able to grep through the source code of the entire archive. This is a topic which has been raised before. Right now I'm keen to make this possible, so overnight I'm syncing the latest sid archive and I think I have a plan to make it work.

• Sync the sources of the given distribution.
• Once that has happened recursively unpack any archive we can understand. (.tar.bz2, tar.gz, etc).
• Either:
  • Write a simple script with "grep".
  • or import the unpacked root into a text indexing system.
  • Also it would be useful to recognize common files via SHA + MD5 checksums.
• make some simple GUI.

I think that the sync will take a while for me on my home connection, and I also believe that the unpacking will be grossly CPU-intensive. Still it should be a worthwhile job even if I can't get it done, because it will tell us the kind of machine which is required to actually do it. I'd like to use a project machine because I'm not entirely sure I have the necessary space but that should become apparent fairly quickly. (The archive I can handle, the unpacked tress might be a little bit much for me. Still I have 100Gb free and I guess that is a good starting point). If anybody has any tips for full-text-indexers that would be appropriate for fast queries against a large directory tree of source code then I'd be interested in hearing about them.

This is my desk. I’m planning on blogging about working at TiVo. I’m also testing phone-Flickr-blog.

This started as an Emdebian idea but it is growing into a more all-purpose option. The requirement is to be able to filter packages during installation or cross-building so that manpages, infopages, documentation (including changelogs, copyright and readme etc.) and translation files for unsupported locales are omitted. When installing, the files would simply not appear on the filesystem or in the dpkg status information. When building, the files would disappear from the package completely (hence the role of filtering in Emdebian). The intention is that embedded systems can have empty /usr/share/man/ /usr/share/info/ and /usr/share/doc directories and a much reduced number of translation files in /usr/share/locale - the default would be only the translation files for languages that are supported on that machine, via 'dpkg-reconfigure locales'. Initially, only the narrow filters needed by Emdebian were considered - discussions at DebConf7 have shown how this can be easily extended to support arbitrary filters - necessarily the risks of using such optional filters would be made clear to the users in the manpage etc. - so that any file in any package could be omitted from the install process and the package file lists as well as during the preparation of packages for embedded devices.

Filters would be glob patterns held in a file in /etc/ (possibly /etc/dpkg/filters.d/) and would include the option to exclude files en-masse and specifically include some files that may otherwise be excluded in specific circumstances. The same filters could be used when building packages for embedded usage as well as installing Debian packages on systems where space is a problem. The main benefit of this filtering for Emdebian is that it makes it much easier to isolate the package changes needed to make the package cross-compilable from the changes needed to remove stuff that embedded devices don't need but which Debian Policy requires. This in turn makes Emdebian patches more useful to Debian maintainers when bugs are filed on their packages about a failed cross-build.

So far, it's taken up most of my hacking time in DebConf. Ian Jackson, Enrico Zini and Wookey have all helped improve the original ideas during that time. Prototype code is mostly complete but the extended filtering support will now have to wait until after DebConf. I'm hopeful that the patch will be accepted, once it is ready.

If this seems strange or unnecessary, just compare the disc usage of /usr/share/doc (which you can control) with /usr/share/locale/ (which you cannot currently control) or view the baobab images from my current laptop installation.

For some days at Debconf, I've been somewhat stressed and frustrated. You see, the first DebCamp/DebConf I took part in (Oslo), I was really productive - Besides meeting lots and lots of people and having a great time, during DebConf3 I learnt a scary lot about the project in many aspects and, although I was already a DD, it really decided me to get more involved.
When we were at Brazil, I had a great time as well, but (code-wise, bugs-wise and so on) I was much less productive - I started peeking at the processes that involve running such a strange and intensive conference, I was more active talking with other Latin Americans, and so on.
So, for DebConf5, I was a full orga team member - Man, did that make me feel powerful! Of course, as there was so much orga work to be done, I basically didn't hack into anything. During the day it was organizational work, and during the night time-after-22:00 (how do you people live without a night!?) it was sauna. What a great invention of the Finns! After getting relaxed by sauna, nights were socially intensive
About DebConf 6... Well, it was hectic and chaotic. It is probably the most stressful two-week period (two week? Two months at least! Well, anyway) period of my life. It is the only Debconf so far I don't even remember having had much social interaction. But still, for reasons I fail to understand, I am really happy we held DebConf in Mexico. It is one of the best, most exciting and (hopefully!) unrepeatable experiences in life.
What about this year? Well, I took a couple of steps back, and while I'm formally still part of the always-great orga team, there are way too many things I don't know or can answer. Partially, I blame that impossible dialect of English they speak in the UK - Please, next time you feel a bit tired, lock yourself in a room with six hurried English and Scottish people, add a couple of Germans to the mix (no, I don't have problem with the other accents spoken in there), and try to get the general lines discussed. Impossible.
As I took a couple of steps back, I thought I would be able to hack a bit. There were some great ideas of things to work on - But so far, I've practically not started with any! I wanted to do some QA rounds in the pkg-perl repository, kill a couple of long-standing bugs, and then there is thismetainit idea we have been discussing... So we held the MetaInit BoF, and Nomeata has been actively hacking it - But I must recognize I have done little besides documenting and cleaning a bit a minor module of it. The second day I was here, I started working on something that was completely outside of my plan (hacking up a subsystem for Pentabarf to help the video team, as I previously complained). Just after we finished with it, DebConf properly started - So I've been basically taping the talks, reviewing the files, in the orga and video meetings... Until yesterday, I was quite frustrated not to have been able to work a bit on the couple of things I wanted to do. I have not yet been able even to work on three things I need for my Real Life, which still counts, even at DebConf.
But then, yesterday I read a post by Wouter - Man, you are completely right, and I must keep this in mind at all times: DebCamp to me isn't about being productive, so much as it is about meeting people and getting to know the environment. Well, I've terribly even failed to get to know the environment (I'll reserve some time on Thursday or Friday, I hope), but so far this has been quite fun, and completely out of what I usually do. And, even without me considering this, DebConf has been quite productive for me in ways I didn't expect - I've met and had a couple of beers with the pkg-perl group (and today I'll meet with the pkg-ruby-extras group), have plotted several things with the Spanish-Speaking group (which, by the way, has tremendously grown since we were at Oslo), I even started hacking a bit my way into Pentabarf, which will be very useful once I get back to work!
Anyway, in short: Wouter, you made me reconsider, you made my day - Thanks! And now, I'm a happy man again :)

This post is a sequel to my “What I want in edgy+1” post, which was written when the developer summit was first announced. Now that the summit (and the following company All Hands meeting) is over, and we’re all back home, this seems as good a time as any to review what was discussed and get a good idea of what feisty might look like. I’ve touched on the problem of predicting time-based releases before. It’s both the gift and the curse of a time-based release schedule that work not completed in time can be deferred to a later release. So take the following with a pinch of salt, some of this may still not make it. General Themes The general theme of dapper was to be a release that could be supported for a long term, conservatism was the goal. We did do some quite exciting work under the hood, such as the switch away from hotplug to a fully udev based system, but in general it wasn’t innovative or ground-breaking. Edgy was intended to be more ground-breaking, but the practical matter of having only a few months to develop it and our own pride in shipping something that still worked meant that it turned out as a shinier, improved dapper. So what’s feisty going to be like? Judging from the discussions at UDS, and the specifications that have been written, the general theme of feisty is to lead the way again with new technologies. The Desktop For the users, perhaps the most obvious change will be the active use of 3D acceleration to draw the desktop where hardware can support it (the issue of binary drivers has not yet been resolved). Windows are more visually distinct from each other through shadows behind them, and transparency for the non-active windows. The relationship between different workspaces/viewports is much clearer as the transition is animated on a cube or sliding pane. And for the bling crowd, window s can wobble, burn, explode or dissolve. There are two different compositors being considered at this point, compiz and beryl; we’re likely to decide which to use at Feature Freeze based on how well they’ve been fixed, developed and supported until that point. Underneath the hood, the configuration of the X server will be simpler and more robust; so even the worst case will not leave you confined to a console without any help. Networking Networking in feisty should be a much more pleasurable experience. The Network Manager project, which has been waiting on the side lines for a couple of releases, may finally get a shot at being in the default installation. For the average user, this makes switching between wired and wireless networks, including setting up WEP and WPA much, much easier. And what if there’s no network infrastructure around? Out of the box support for RFC 3927 link-local networks, and multi-cast DNS resolution (aka. Zeroconf), means that you just need to agree on a network name with others around you to be able to communicate. Of course, once you’re on a network, you still need to be able to share files and access local services. The integration of the Avahi project gives you one-click access to other people’s shared music or files; and lets you share your own, should you choose to do so. Customisations One of the most encountered problems with edgy was it being difficult to install various common packages that aren’t part of the default installation, especially codecs. Projects such as Automatix attempt to tackle this, but can cause problems with upgrading to later releases. Some effort will be going into feisty to make performing these common customisations much simpler, including being able to install codecs or viewers by just trying to open the file. Boot Sequence A long-running project within Ubuntu has been to get the boot and shutdown sequences as fast and efficient as possible. At the time we started, it was common for a Linux distribution to boot in a mere two or three minutes. If you thought edgy booted fast, wait until you see feisty. Feisty is the release where we take full advantage of Upstart, not only bringing the system up as fast as possible but also more robustly than we can do today. And if that weren’t enough, it should look slicker too; without some of the nasty flickering and mode changes that happen today.

For the last couple of months, both at the Ubuntu Developer Summit in Mountain View and on the #upstart IRC channel, we’ve been discussing the changes we want to make to upstart for the Feisty Fawn release of Ubuntu. This will ship with a version of upstart based on the 0.3 series (it may end up getting called 0.5 before release); the primary goal for this are to have an init system that is suitable for general standalone list in any Linux distribution. I’ll be giving a talk at linux.conf.au 2007 in Sydney with that aim, I hope to persuade at least one other major Linux distribution that it’s the right solution. A complete list of the specifications and bugs being targeted for the 0.3 release can be found in Launchpad. The rest of this post will introduce some of the shiniest new things. Writing Jobs Upstart takes care of starting, supervising and stopping daemons itself; unlike in the init script system where you have to write code to do that yourself, often using a helper like start-stop-daemon. All you need to is give the path to, and arguments for, the binary you wish to be started.

exec /usr/bin/dbus-daemon

Some jobs, especially quick tasks, will usually be written as shell scripts. To save having to write a separate file and invoke it, you can include shell script code directly in the job file instead of using the exec stanza.

script
    echo /usr/share/apport/apport > /proc/sys/kernel/crashdump-helper
end script

Usually it’s not sufficient to just start a binary and wish it well; you frequently need something to be run before it is started to prepare the system, and sometimes something after it terminates to clean up again. For these purposes, additional snippets of shell code can be given – to be run before the binary is started, and after it has finished. Unlike init scripts, these do not need to start or stop the daemon itself; that’s done automatically based on the exec stanza.

pre-start script
    mkdir -p /var/run/dbus
    chown messagebus:messagebus /var/run/dbus
end script
post-stop script
    rm -f /var/run/dbus/pid
end script

For consistency, executables may be specified with pre-start exec and post-start exec instead of shell scripts as above. It’s sometimes useful to be able to run something after the binary has been started; for example, you may wish to attempt to connect to the daemon to determine whether it is ready to serve requests. post-start script or post-start exec can be used to this.

post-start script
    # wait for listen on port 80
    while ! nc -q0 localhost 80 </dev/null >/dev/null 2>&1; do
        sleep 1;
    done
end script

It’s also useful to be able to notify a daemon that it may be about to be stopped, or delay it for a while. pre-stop script or pre-stop exec can be used for this.

pre-stop script
    # disable the queue, wait for it to become empty
    fooctl disable
    while fooq >/dev/null; do
        sleep 1
    done
end script

Events Events are now quite a bit more detailed than in previous versions; they’re still named with simple strings that are up to the system sending the event, but they can now include arguments and environment variables which are passed through to jobs being started or stopped as a result.

initctl emit network-interface-up eth0 -DIFADDR=00:11:D8:98:1B:37

This command will now output all of the effects of this event, and will not terminate until the event has been fully handled inside upstart. Events such as the above can be used by jobs that examine the event arguments and environment within their script:

start on network-interface-up
script
    [ $1 = lo ] && exit 0
    grep -q $IFADDR /etc/network/blacklist && exit 0
    # etc.
 end script

or matched directly in the start on and stop on stanzas:

start on block-device-added sda*

The events generated by job state changes have also changed. Previously both jobs and events shared the same namespace, which not only caused confusion but actually caused some problems when one accidentally named a job after an event. The two primary events generated are now simply called started and stopped; they inform you that a job is fully up and running, or fully shut down again. The name of the job is received as an argument to this event.

start on started dbus

The started event is not emitted until the post-start task (described above) has finished; so the post-start task can delay other jobs from starting because they can’t yet connect to the daemon. Likewise the stopped event is not emitted until after the post-stop task has finished. The other two events emitted by a job are special; they are the starting and stopping events. The reason they are special is that the job is not permitted to start or stop until the event has been handled. This means that if you have a task to perform when your database server is stopped, but before it’s actually terminated, it’s as simple as:

start on stopping mysql
exec /usr/bin/backup-db.py

MySQL won’t be terminated until the backup has finished. This is especially useful for daemons that depend on each other, for example HAL needs DBUS, it shouldn’t be started until DBUS is running and DBUS should not be stopped until HAL has been terminated. All the HAL job needs is:

start on started dbus
stop on stopping dbus

Likewise if tomcat is installed, Apache should not be started until tomcat is running; and tomcat should not be stopped until apache has been terminated. All the tomcat job needs is:

start on starting apache
stop on stopped apache

Failure Nothing goes smoothly all of the time, sometimes tasks the job runs will fail, or the daemon itself will die. As well as providing the ability for a crashed daemon to be automatically restarted, upstart ensured that other jobs are notified with a special failed argument to the stopping and stopped events.

start on stopped typo failed
script
    echo "typo failed again :-("   mail -s "typo failed" root
end script

And if any job started or stopped by an event fails, it’s possible to discover that the event itself failed.

start on network-interface-up/failed

States While tasks such as configuring a network interface, or checking and mounting a block device are usually performed as a result of events; services are more complicated. Services normally need to be running while the system is in a certain state, not just when a particular event occurs. Therefore upstart allows you to describe arbitrarily complex system states by referring to events that define their changes. For example, many services should be running only while the filesystem is mounted, and at least one network device is up. We have events to indicate the changes into and out of these dates, we just need to combine them:

from fhs-filesystem-mounted until fhs-filesystem-unmounted
and from network-up until network-down

The until operator defines a period between two events, the and operator ensures we’re within both of these periods. Perhaps we need to be running while any display manager is:

from started gdm until stopping gdm
or started kdm until stopping kdm

Or maybe we only want to be run if a network interface comes up before bind9 has been started:

on network-interface-up and from startup until started bind9

These “complex event configurations” can appear in any job file; and any job file itself can serve as a reference for other jobs. They will be started and stopped at the same time as the named job:

with apache

Omitting the exec or script stanza from a job file means that it simply defines a state that can serve as a reference for others. As such, the multiuser state is simply a job file that defines it. As an added bonus, these states can still have pre-start, post-stop, etc. scripts.