I received a SheevaPlug this week, an intriguing device that packs incredible power and functionality into a tiny package. As many of you know, I've been doing a lot of work on Debian for the Linksys NSLU2 in the last few years. The NSLU2 is a key reason why ARM has become the third most popular architecture in Debian (after 32 and 64 bit x86), and I believe a main reason is that the NSLU2 is so incredibly cheap. At a price under $100, most people don't think too long and simply buy a device and do something cool with it. The SheevaPlug is being offered at the same price range but offers considerably more. Riku Voipio asked the right question: "What would you do with something approximately 10x more powerful with same prize/size range?" I believe the SheevaPlug is a killer replacement for the NSLU2 and here's why:

• Marvell Kirkwood 6281 CPU at 1.2 GHz with 256 KB L2 cache (compared to a 266 MHz CPU)
• 512 MB RAM (compared to lousy 32 MB)
• 512 MB flash, enough for a basic Debian installation (compared to 8 MB)
• Gigabit Ethernet (compared to 100 Mb) and no proprietary microcode required
• 1x USB and 1x SD (compared to 2x USB: the only advantage the NSLU2 has)
• ... and finally: mini-USB with serial console and JTAG: while Debian on the NSLU2 works well and is stable, we regularly had users whose system would no longer boot and there was no good way to figure out why. The serial console on the SheevaPlug actually allows users to figure out what's going on.

I'm incredibly excited about the SheevaPlug and the first thing I did was to take the device apart and look at the inside. The results can be found in the SheevaPlug image gallery. My next project will be slightly more productive: porting Debian. As I see it, we should support the following three installation variants for the SheevaPlug:

• To an external USB drive or stick.
• To an SD card.
• To the 512 MB flash memory on the device.

The first two should be relatively straight forward, but of course installing to the internal flash memory is particularly interesting given that 512 MB (plus compression) is enough for a basic installation of Debian. Unfortunately, installations to MTD flash are currently not supported in the Debian installer but I hope we can find a volunteer who wants to implement this functionality. My next steps are to put a kernel for the SheevaPlug into the archive and to get a basic installation going. From there we can look at more sophisticated installation options and other functionality.

This morning it was once more time for the annual March Madness Half Marathon in Cary. This race is basically the start of the running season in Chicagoland. And we could not have asked for better weather. After a really cold and long winter, and a short snapback to really cold temperatures this week, it started to warm up a little yesterday with expectations of more of the same today. So while it was still cold at the start at around 37 degrees, most runners opted for shorts and by the finish temperatures were in the high 40s. Coupled with clear blue skies and no wind, it was a really nice morning for a run. So how did it go? Well, I had pretty low expectations. Training has been difficult with too much snow, rain and plain cold weather. So like most of my running friends, motivation ran pretty low of late. I was really only trying to set a modest pace, and to hope to hang on to it and run steadily. That worked: I didn't walk a single water stop, and while my legs were getting really tired and sore I carried through to the end. Final time was 1:36:08.57 per my stop watch. That's a tad faster than last year, a lot slower than 2007, just a tad slower than 2006, and quite a bit faster than 2005. Next stop: my second Boston Marathon in five weeks and given the underwhelming training this winter, it will be a challenge.

Warning: Don t upgrade root partition (/) or the partition in which /boot is placed unless you know what you are doing. More on this later.
Warning2: Make backups of your data. Last Friday I went on upgrading to ext4 my laptop. I decided to go first with /home partition, as / has also /boot and can give some problems if not handled with care. Upgrading it is quite easy, and fast, as the old data is not upgraded, only new files will be added using new features. First, I installed latest linux-image package to start using kernel version 2.6.28, which includes production ready ext4 support. After rebooting, I logged in as root so I could umount /home. The steps to upgrade are as follows:

# uname -a
Linux gimli 2.6.28-1-686 #1 SMP Mon Feb 23 03:13:24 UTC 2009 i686 GNU/Linux
# umount /home
# tune2fs -O extents,uninit_bg,dir_index /dev/home_partition
# fsck -pfD /dev/home_partition
# mount /home

You MUST run fsck. If not, ext4 won t mount your filesystem, so don t forgive this step if you re upgrading your / partition. You will see some checksums errors appear. Don t be afraid, that s expected, and that s why -p is added to fsck command, you it is not asking which action to perform in every error. Upgrading partition containing /boot
For upgrading your / partition you have to take into account what grub and klibc version you have installed. The former is need so grub can read your kernel image from filesystem, and the second is needed due a bug that made initramfs detect ext4 filesystems as ext3, and passing incorrect option to mount, which failed. Versions in Debian known to work are libklibc >=1.5.15-1 and grub2 (packaged as grub-pc). Make sure you have these versions installed before trying to go to ext4.
Also, as you have to upgrade having your partition unmounted, you will have to get a console before root partition is mounted. You can do that by passing mount=break break=mount to your boot options in grub. ENJOY!

I m in the middle of migrating a mail server away from the Cyrus mail store [1]. Cyrus provides a POP and IMAP server, a local delivery agent (accepting mail via LMTP). It is widely believed that Cyrus will give better performance than other mail stores, but according to a review by linux-magazin.de Dovecot and Courier deliver comparable (and sometimes better) performance [2]. The biggest problem with Cyrus is that it is totally incompatible with the Unix way. This wouldn t be a problem if it would just work and if it would display reasonable error messages when it failed, but it doesn t. It often refuses to work as desired, gives no good explanation, and it s data structures can t be easily manipulated. Dovecot [3] and Courier [4] use the Maildir++ format [5] (as well as many other programs). I have set up a system with Courier Maildrop and Dovecot for the IMAP server [6] and it works well - it s good to have a choice! But also Maildir++ is reasonably well documented and is an extension to the well known Maildir format. This means that it s easy to manipulate things if necessary, I can use mv to rename folders and rm to remove them. Cyrus starts with a database (Berkeley DB file) of all folders in all mailboxes. Therefore it is not possible to move a user from one back-end server to another by merely copying all the files across and changing the LDAP (or whatever else contains the primary authentication data) to point to the new one. It also makes it impossible to add or remove folders by using maildirmake or rm -rf. The defined way of creating, deleting, and modifying mailboxes is through IMAP. One of the problems with this is that copying a mailbox from one server to another requires writing a program to open IMAP connections to both servers at once (tar piped through netcat is much faster and easier). Also if you need to rename a mailbox that contains many gigabytes of mail then it will be a time consuming process (as opposed to a fraction of a second for mv). Cyrus has a tendency to break while Dovecot is documented as being self-healing and Cyrus also seems to cope well in the fact of a corrupted mail store. Even manually repairing problems with Cyrus is a painful exercise. The Cyrus mail store is also badly designed - and it s design was worse for older filesystems (which were common when it was first released) than it is for modern ones. The top level of a Cyrus maildir contains all the messages in the INBOX stored one per file, as well as three files containing Cyrus indexes and sub-directories for each of the sub-folders. So if I want to discover what sub-folders a mailbox has then I can run ls and wait for it to stat every file in the directory or I can use an IMAP client (which takes more configuration time). As opposed to a Maildir++ store where every file that contains a message is stored in a folder subdirectory named new , cur , or tmp which means that I can run ls on the main directory of the mail store and get a short (and quick) result. Using tools such as ls to investigate the operation of a server is standard practice for a sysadmin, it should work well! A finall disadvantage of Cyrus seems to have many small and annoying bugs (such as the reconstruct program not correctly recursing the sub folders). I guess it s because not many people use Cyrus that such things don t get fixed. One trivial advantage of Cyrus is that by default it splits users into different sub-directories for the first letter of the account name. Dovecot supports using a hash of the user-name this is better than splitting by first-letter for performance (it gives a more equal distribution) but will make it slightly more difficult to manipulate the mail store by script. Ext3 can give decent performance without a two level directory structure for as many as 31,998 sub-directories (the maximum that it will support) due to directory indexing and Linux caching of dentries. There may be some other advantages of Cyrus, but I can t think of them at the moment. Here is a script I wrote to convert Cyrus mail boxes to Maildir++. To make this usable for a different site would require substituting a different domain name for example.com (or writing extra code to handle multiple domains) and inserting commands to modify a database or directory with the new server name. There is no chance of directly using this script on another system, but it should give some ideas for people performing similar tasks.

#!/bin/bash -e
# $1 is the user to move # BASEDIR is the directory for temporary files
BASEDIR=/mail/new
rm -rf $BASEDIR/$1 archive.tgz archive2.tgz ORIGSRV=$(echo "MYSQL_COMMAND_TO_GET_SERVER" mysql dbname)
echo "original server was $ORIGSRV" if [ "$ORIGSRV" = "" ]; then
echo "No server for account, strange error."
exit 1
fi ORIGPARENT="/mail/domain/e/example.com/$(echo $1 cut -c1)/user" # copy the data to a local archive
# USERDIR is for using ^ instead of . as Cyrus does.
mkdir -p $BASEDIR
cd $BASEDIR
USERDIR=$(echo $1 sed -e "s/\./^/g")
ssh cyrus@$ORIGSRV "(cd $ORIGPARENT ; tar czf - $USERDIR)" > archive.tgz
tar xzf archive.tgz
if [ "$USERDIR" != "$1" ]; then
mv $USERDIR $1
fi DEPTH=$(find . -mindepth 3 -type d wc -l)
if [ "$DEPTH" != "0" ]; then
echo "too deep"
exit 1
fi # remove cyrus index files
find $1 -name "cyrus.*" -exec rm \;
cd $1 # "mail" is a directory with sub-folders, every other name starting with
# [a-zA-Z] is a regular folder that needs to become a Maildir
for n in [a-zA-Z]* ; do
if [ "$n" != "mail" ]; then
cd "$n" ; mkdir new cur tmp ; touch maildirfolder
mv [0-9]* new true
cd .. ; mv "$n" ".$n"
fi
done # move the inbox files to Maildir storage
mkdir new cur tmp
mv [0-9]* new true
mv mail/[0-9]* cur true # move sub-folders of "mail/" and merge ones with duplicate names
cd mail
for n in * ; do
if [ -e "../.$n" ]; then
cd "$n"
mv [0-9]* "../../.$n/cur" true
cd .. ; rmdir "$n"
else
cd "$n" ; mkdir new cur tmp ; touch maildirfolder
mv [0-9]* new true
cd .. ; mv "$n" "../.$n"
fi
done
cd ..
rmdir mail # get a list of all names so we can recognise duplicates
find . -type d egrep -v "new cur tmp" cd $BASEDIR ssh root@localhost "mv /mail/new/$1 /mail/example.com ; chown -R mail:mail /mail/example.com/$1"
echo "MYSQL_COMMAND_TO_SET_SERVER" mysql dbname

• [1] http://cyrusimap.web.cmu.edu/


• [2] http://www.linux-magazin.de/heft_abo/ausgaben/2007/06/auf_der_teststrecke/


• [3] http://www.dovecot.org/


• [4] http://www.courier-mta.org/imap/


• [5] http://wiki.dovecot.org/MailboxFormat/Maildir


• [6] http://etbe.coker.com.au/2009/03/10/maildrop-imap-postfixadmin/

Earlier today I stumbled across a recording of a work I had not heard in a few years by the obscure Swiss-German composer Joachim Raff. The composition is an orchestral transcription of the Chaconne from Bach's Partita in D minor, without doubt the finest work ever written for solo violin. This piece has been transcribed countless times; the source material is clearly a masterpiece, but what makes the movement really attractive is its implicit nature - dispite the violin's limited facilities for counterpoint and polyphony in general, Bach succeeds in alluding to a multitude of voices within the same line. Larry Solomon provides a brief overview of what to look out for. Raff's transcription thus endeavours to render the implicit explicit; for example, where there is the implication of a held note, it can played for its implied duration. Indeed, Raff believed that Chaconne was in itself a "reduction" from an orchestrated original. Whilst I don't share that literal belief, it is certainly good working model for a transcription. And what a transcription! I've often thought of the romantic transcriptions such as these to be a guilty pleasures; they clearly violate any sensibilities I have about performing early music. However, what is most surprising (and guilt-relieving) about Raff's effort is the sheer amount of novel material involved. Here is an example starting at bar 216 - I have overlayed Bach's original line with a melody of Raff's own invention:

There are countless other examples which, following Bach's example, vary greatly in their subtlety. No prizes for guessing why I chose this excerpt though. However, dispite these additions I feel some crucial aspects are actually lost in translation. Firstly, we can muse over whether Bach would have approved of such a rendering - not wishing to dwell too deeply in well-tread arguments, many would point to Bach's re-use of his own material (as well as his transcriptions of other composers) to illustrate that he was not against the practice, but I find that argument difficult to apply to the Chaconne - the dualism between the implicit and the explicit would have greatly appealed to Bach, so to rob the work of it would seem to be doing him a serious discourtesy. Furthermore, the implications in Bach's work are certainly not followed to the same conclusions by all - a truly solo musician has the advantage of being able to make their own decisions about the music, perhaps even on the spur of the moment. In contrast, an ensemble is effectively forced to obey Hr. Raff's inferences. This lack of spontaneity is particularly damning; I am sure one would tire of differing interpretations of Raff's transcription quicker than one would of Bach's original. Lastly (and certainly most subjectively) there is something disarmingly solitary about the original work which is only amplified by being played solo - dispite my inability to provide a satisfactory rendition I would find performing this piece with (or even to) others rather discomforting. In conclusion, I would recommend listening to this work (and the rest of the CD). I have found it extremely illuminating, if only about Bach's original than of Raff's.

On off the biggest problems with developing servlets under a container like Tomcat is the amount of time taken to build your code, deploy it to the container and restart it to pick up any changes. Maven and the Jetty plugin allow you to cut down on this cycle considerably. The first step is to allow you to start your application in maven by running:

mvn jetty:run

We do this by configuring the jetty plugin inside our pom.xml:

<plugin>
   <groupId>org.mortbay.jetty</groupId>
   <artifactId>maven-jetty-plugin</artifactId>
   <version>6.1.10</version>
</plugin>

Now when you run mvn jetty:run your application will start up. But we can improve on this. The Jetty plugin can be configured to scan your project every so often and rebuild it and reload it if anything changes. We do this by changing our pom.xml to read:

<plugin>
   <groupId>org.mortbay.jetty</groupId>
   <artifactId>maven-jetty-plugin</artifactId>
   <version>6.1.10</version>
   <configuration>
      <scanIntervalSeconds>10</scanIntervalSeconds>
   </configuration>
</plugin>

Now when you save a file in your IDE, by the time you've switched to your web browser, Jetty is already running your updated code. Your development cycle is almost up to the same speed as Perl or PHP. You can find more information at the plugin page.

Tonight begins the end of a great ( IMHO ) Sci-Fi story. I just hope that someday it may be continued in written/novel form. Ahh the things dreams are made of. I'll leave it to the reader to translate the title ...

For some time ago I ve been pondering about this question. As long as GSoC 2009 is about to start and people are looking for project ideas, I m posting here a very preliminar draft of my findings and an idea for a posible software project. It s pretty written on-the-fly while I ve managed to have some minutes between work and uni. Rigurous wording isn t P1. Debian s social contract 4th item states:

Our priorities are our users and free software We will be guided by the needs of our users and the free software community. We will place their interests first in our priorities.

When we make decisions, either technical, legal and others regarding the operating system we deliver, this is one of the most referred argument. Altough that s seen as high priority often we don t have ways to properly know what our users want. While working on translations on the Spanish team we ve faced this situation. Do our users find them useful? they are able to understand them? they find that a much familiar wording would be bette than high technical details? We didn t know. Then, the question still exists. Altough Debian has ways for users to provide feedback (BTS, mailing lists, IRC, popcon, etc) none of them is designed to offer a way for the user to provide feedback on features, development roadmap and other non-bug aspects that developers and the project can tally and use for prioritize on releases. It s rather amusing to note that one of the most valuable assets free software community has is user involvement and contribution. But before we enter in details let s start on the basics. For using this information we first need to know who our users are? do we? From my findings and interaction with the community I ve identified two kinds of Debian users who can be clearly named. 1) The derivatives or pure blends, who use the Debian base and framework to build niche distributions. 2) The lead users (using E. Von Hippel s definition), who are the developers, contributors and a group of users. Debian is, if not the only, one of the projects who is best for fostering lead users. Most of them at some point involve themselves in the development process and some, later become developers. There is a more diverse group who are end-users but it s unclear to me how we can group them. Despite that they probably represent the biggest part of the pie. So, how we do please everyone? can we? In my opinion we can, at some degree. By implementing tools for first: gather user feedback, make statistical data, tally, we later can take informed decisions. In Debian decisions are voted and the set of people who votes are only developers with their own constraints. I think this would ease the constant that release-decisions-regarding-foo-tech-legal-issue represented on each release iteration, saving time, health and bits. It could be useful for knowing more about who are the ones we develop things for. I m thinking of some sort of Dell s IdeaStorm.com for software. What do you think? I m willing to co-mentor this if someone finds that is an interesting project for GSoC. Let me know!.

I ve been very impressed several times in the past few months when I ve discovered awesome new top-like utilities. I m probably being slow on the uptake and everyone else but me knows about these, but in case its not just me thats been stuck in the 70s:

htop

A much-needed refresh of oldschool top, this still works on your beloved console but gives you visual bar-graphs of CPU, RAM and swap, lets you scroll through the processes and deliver signals/renicing without having to copy the PID off the moving target. Its like the future!

iftop

One of those things I use so often now I have no idea how I even survived without it. Why is this server lagging, who s hogging the wireless/DSL, which VM is chewing all of the upstream bandwidth? iftop shows you at a glance how much traffic is being used by which host pairs on a given interface, and you can toggle port numbers on and off with simple key-presses. Absolutely indispensable.

iotop

Does this box feel slow to anyone else? Is it swapping, or is it the database server chewing all the IO? Why does my drive keep seeking? It s amazing top for IO bandwidth usage!

A passing mention is deserved for apachetop too, which is pretty neat, but when a server is being hammered it s not something I found too hard to get a feel for just by tailing the log for a while, so it s not been as life-changing as the others. Maybe that just means my servers don t see enough traffic.

Eheh, I'm getting older... 22 today! I just wanted to put down some of the important things happened to me this year...

• sentimental life: epic fail... not really in the mood of talking about it;
• University life: I finally passed two exams which blocked me for all the previous year: yay! Now I'm being stuck with other things -- not too much enthusiasm in some courses...
• Debian life: I finally had my GPG key signed by TWO Debian Developers: I became DM, and had upload rights on non-NEW packages. Great! We had Lenny released. Uber-Great!
• misc life: I just tried getting more people-friendly, more active (meaning being less lazy), less nerd (I probably failed in this particular aspect), better, in a word.

Wishes for the 22-23 development cycle :

• finally becoming a DD! (I'm NM since Dec-2007..)
• trying to pass many exams as possible at University
• seriously fall in love (I think I've already started this with the girl I'm with now...)
• misc improvements (sorry, you can't send patches... or yes?)

Happy birthday to me!

I ve had smart quotes envy for a long time now. I particularly like them in contractions, and I think they make the text more pleasant to read. Of course, I ve been using them on my web pages and blog for some time now, since they are well-supported by browsers. TeX/LaTeX deals with them fine as well, even by just typing ASCII in the source document. Mostly only mail was missing. I ve delayed using smart quotes in mail for a long time, since they imply a lot of UTF-8, and there was/is still quite a lot of software and setups not ready for it. But we re in the 10th year of the 21st century already, and I was getting envious of the increasing number of people that are using them for e-mail. So I ve recently switched to smart quotes in my outgoing mail. I realize this goes against the Be strict in what you emit and liberal in what you accept principle, which is a fine one, but I must confess aesthetics have won the battle for me in this case. I ve already received one complaint from a person using broken software, asking me to send more comprehensible and correct messages , and referring to UTF-8 as WTF-8. I wonder how many people to whom my mails are targetted at will have trouble reading them now. I don t have comments enabled in this blog yet I m afraid, but feel free to participate in the poll.

So, Kodak announced the availability of Linux drivers for some of its scanners, together with a frontend application. While the effort is laudable, they re not quite there yet. Let s have a look. First of all, and contrary to what a few people believe after reading the press release, the drivers themselves aren t OpenSource, let alone Free Software. They re all proprietary; read better, the press release clearly mentions that only the frontend application is GPLv2. Now let s look at what Kodak offers in its drivers packages. I ve downloaded LinuxSoftware_i1200_v3.5.tar.gz from the Kodak website, these are the drivers for the Kodak i1200 series. The archive contains a set of RPM packages, one Debian package corresponding to one of the RPM packages, and a setup script. One of the packages contains the OpenUSB library, which is released under the LGPL; sources aren t there, I haven t asked for the sources, don t plan on doing so, and am not implying anything regarding license compliance whatsoever. I ll note that I haven t seen any written offer for the sources, though, be it in the archive (there s not even a README in there) or on the website. So, although Ubuntu is listed as supported, we can say that no Debian packages are provided. The setup script actually relies on alien to convert them if it thinks it s running on a Debian-based system, which is determined by the availability of dpkg in the PATH. Using alien is far from ideal. The packages ship libraries and components under /usr/lib, /usr/local/lib and even /opt, pretty randomly it seems. Let s repeat here that /usr/local and /opt are strictly reserved for the local administrator and packages must not install anything here. There are other oddities in the paths used by the software stack, like /var/kodak. It looks like the SANE backend is only a bridge to Kodak s TWAIN data source, so you d actually need the whole stack to use it. All in all, Kodak does no better, I d even venture to say it does worse, than Epson (Avasys). At least part of Epson s backend is Free Software, even if it relies on proprietary libraries for some scanners, though unfortunately most if not all of the recent ones. I also have to mention that Epson is offering both RPM and Debian packages, and by that I mean proper Debian packages. Kodak are far from that, even the RPM aren t that great (I think the RPM packages even lack dependency information, based on what I ve read in the setup script, though I haven t checked). What Kodak offers here is a GPL frontend for TWAIN that not many people will be using, while keeping the drivers proprietary. Unfortunately their SANE backend isn t a backend so to speak, and even if it was OpenSource it d still rely on the rest of the TWAIN software stack. So, in a nutshell, while I m very happy to see yet another vendor go down the road of Linux support for its products, I am immensely disappointed by the proprietary nature of the drivers and the low quality of the packages. I was, of course, investigating the drivers to possibly package them for Debian; as you ve probably understood, it s not possible, both technically and legally. And I kept a little gem from the setup script for the end:

# We need to create a link for the libdbus-1 requirement for openusb,
# but only on Fedora.
if [ -f /etc/fedora-release ]; then
    ln -s /lib/libdbus-1.so.3 /lib/libdbus-1.so.2 2>> /dev/null
fi

As we all know, sonames and soversions are useless and they just keep getting in the way. Duh.

Just a quick update to let you know how the Debian bid for the Google Summer of Code is going. We re doing great right now. Our application is sent out, you can consult the public part here. This year, we re committed to doing a great Summer of Code performance, starting first by improving communication with the Debian community. We opened a Twitter feed for you Twitter freaks: http://twitter.com/DebianGSoC. It will be updated with live information relevant to students and devs alike during the whole summer. UPDATE: We re now also on identi.ca : http://identi.ca/debiangsoc We now have 12 applications on our wiki. Come, add your ideas and discuss them on our IRC channel (#debian-soc on irc.debian.org) and mailing-list. Packaging

• Large dataset manager: download and update local copies of public datasets, and integrate them in Debian with the tools we packaged.
• Mergemaster: perform 3-way merge of config files on upgrades by keeping a copy of original config files.
• Aptitude packages query UI and language: improve the GTK+ search UI and language for complex packages queries
• Aptitude search ranking and presentation: improve the presentation of search results with substring highlighting, relevancy ranking
• Aptitude history tracking: extend and build on the packages actions history tracking features of Aptitude

Debian Installer

• MTD support in d-i: add MTD installation support to the Debian Installer
• Debian-Installer support for GNU/kFreeBSD: replace the currently used FreeBSD sysinstall with Debian-Installer running possibly under a FreeBSD kernel

Blends

• blends webtools: enhancing DebianPureBlends webtools by using UltimateDebianDatabase

Bug tracking

• Debbugs Read/Write Soap Interface: overhaul the existing read-only SOAP Debbugs interface with submission and manipulation capabilities
• Debbugs Web UI: Modern web interface for submitting and manipulating bugs.debian.org s bugs
• Mylyn plugin for debbugs: Plugin for Mylyn to support debbugs, allowing the management of Debian bug reports from the inside of Eclipse.

Building infrastructure

• Building a new interface for Wanna-build using PostgreSQL: replace the existing setup of multiple MLDBM databases with an extensible and flexible relational database for use both by wanna-build and other tools such as web CGI scripts

With any luck, someone has seen this problem before and can fix it easily. I don t seem to be able to find the fix by searching, but the search terms feel like they re either too general or too specific. I was trying to install some perl software on a debian lenny system that was upgraded from etch (and previously from sarge). After installing the dependencies, I started getting errors like this:-

$ perl -MYAML::Syck -e print
Can't locate YAML/Syck.pm in @INC (@INC contains:
/usr/local/lib/perl5/5.10.0/i686-linux /usr/local/lib/perl5/5.10.0
/usr/local/lib/perl5/site_perl/5.10.0/i686-linux
/usr/local/lib/perl5/site_perl/5.10.0 .).
BEGIN failed--compilation aborted.

Update: Well done to Raphael who spotted the local that I d overlooked. D oh! Now to see if I can discover where that came from. Update 2: Also thanks to Florian who emailed in at about the same time (I don t read my email as much as my website dashboard, usually).

Then I discovered that the debian mirror being used (which is in the same city, and in a nearby network) is marked as bad on the health check. I switched mirrors to a known-good one that I ve used for other systems recently, but no joy. I think a key symptom is this bit near the end of the output of perl -V:

Compiled at Aug 4 2008 09:48:59
@INC:
/usr/local/lib/perl5/5.10.0/i686-linux
/usr/local/lib/perl5/5.10.0
/usr/local/lib/perl5/site_perl/5.10.0/i686-linux
/usr/local/lib/perl5/site_perl/5.10.0

Our good systems show a December compiled at date and a much longer @INC list. I ve tried reinstalling pretty much every package I think might be connected, including libc6, libperl5.10, perl, perl-base and perl-modules (those last four are all version 5.10.0-19). I really don t want to reinstall the whole system because it s behind an irritating firewall that will make it a 10-hour round-trip site visit. I m currently hunting and removing obsolete packages to try to make sure they re not causing problems, because I m running out of straws to clutch at. How would you recover a debian system from this?

A recent Ubuntu bug has gotten slashdotted, and has started raising a lot of questions about the safety of using ext4. I ve actually been meaning to blog about this for a week or so, but between a bout of the stomach flu and a huge todo list at work, I simply haven t had the time. The essential problem is that ext4 implements something called delayed allocation. Delayed allocation isn t new to Linux; xfs has had delayed allocation for years. Pretty much all modern file systems have delayed allocation, according to the Wikipedia Allocate-on-flush article, this includes HFS+, Reiser4, and ZFS; btrfs has this property as well. Delayed allocation is a major win for performance, both because it allows writes to be streamed more efficiently to disk, and because it can reduce file fragmentation so that later on they can be read more efficiently from disk. This sounds like a good thing, right? It is, except for badly written applications that don t use fsync() or fdatasync(). Application writers had gotten lazy, because ext3 by default has a commit interval of 5 seconds, and and uses a journalling mode called data=ordered. What does this mean? The journalling mode data=ordered means that before the commit takes place, any data blocks are
associated with inodes that are about to be committed in that transaction will be forced out to disk. This is primarily done for security reasons; if this is not done (which would be the case if the disk is mounted with the mount option data=writeback), then any newly allocated blocks might still contain previous data belonging to some other file or user, and after a crash, accessing that file might result in a user seeing unitialized data that had previously belonged to another user (say, their e-mail or their p0rn), which would be a Bad Thing from a security perspective. However, this had the side effect of essentially guaranteeing that anything that had been written was guaranteed to be on disk after 5 seconds. (This is somewhat modified if you are running on batteries
and have enabled laptop mode, but we ll ignore that for the purposes of this discussion.) Since ext3 became the dominant filesystem for Linux, application writers and users have started depending on this, and so they become shocked and angry when their system locks up and they lose data even though POSIX never really made any such guarantee. This become especially noticeable on Ubuntu, which uses many proprietary, binary-only drivers, which caused some Ubuntu systems to become highly unreliable, especially for Alpha releases of Ubuntu Jaunty, with the net result that some Ubuntu users have become used to their machines regularly crashing. (I use bleeding edge kernels, and I don t see the kind of unreliability that apparently at least some Ubuntu users are seeing, which came as quite a surprise to me.) So what are the solutions to this? One thing is that the applications could simply be rewritten to properly use fsync() and fdatasync(). This is what is required by POSIX, if you want to be sure that data has gotten written to stable storage. Some folks have resisted this suggestions, on two grounds; first, that it s too hard to fix all of the applications out there, and second, that fsync() is too slow. This perception that fsync() is too slow was most recently caused by a problem with Firefox 3.0. As Mike Shaver put it:

On some rather common Linux configurations, especially using the ext3 filesystem in the data=ordered mode, calling fsync doesn t just flush out the data for the file it s called on, but rather on all the buffered data for that filesystem.

Fundamentally, the problem is caused by data=ordered mode. This problem can be avoided by mounting the filesystem using data=writeback or by using a filesystem that supports delayed allocation such as ext4. This is because if you have a small sqllite database which you are fsync(), and in another process you are writing a large 2 megabyte file, the 2 megabyte file won t be be allocated right away, and so the fsync operation will not force the dirty blocks of that 2 megabyte file to disk; since the blocks haven t been allocated yet, there is no security issue to worry about with the previous contents of newly allocated blocks if the system were to crash at that point. Another solution is a set of patches to ext4 that has been queued for 2.6.30 merge window. These three patches (with git id s bf1b69c0, f32b730a, and 8411e347) will cause a file to have any delayed allocation blocks to be allocated immediately when a file is replaced. This gets done for files which were truncated using ftruncate() or opened via O_TRUNC when the file is closed, and when a file is renamed on top of an existing file. This solves the most annoying set of problems where an existing file gets rewritten, and thanks to the delayed allocation semantics, that existing file gets replaced with a zero-length file. However, it will not solve the problem for newly created files, of course, which would have delayed allocation semantics. Yet another solution would be to mount ext4 volumes with the nodelalloc mount option. This will cause a significant performance hit, but apparently some Ubuntu users are happy using proprietary Nvidia drivers, even if it means that when they are done playing World of Goo, quitting the game causes the system to hang and they must hard-reset the system. For those users, it may be that nodelalloc is the right solution for now personally, I would consider that kind of system instability to be completely unacceptable, but I guess gamers have very different priorities than I do. A final solution which might not be that hard to implement would be a new mount option, data=alloc-on-commit. This would work much like data=ordered, with the additional constraint that all blocks that had delayed allocation would be allocated and forced out to disk before a commit takes place. This would probably give slightly better performance compared to mounting with nodelalloc, but it shares many of the disadvantages of nodelalloc, including making fsync() to be potentially very slow because it would force all dirty blocks to be forced out to disk once again. What s the best path forward? For now, what I would recommend to Ubuntu gamers whose systems crash all the time and who want to use ext4, to use the nodelalloc mount option. I haven t quantified what the performance penalty will be for this mode of operation, but the performance will be better than ext3, and at least this way they won t have to worry about files getting lost as a result of delayed allocation. Long term, application writers who are worried about files getting lost on an unclena shutdown really should use fsync. Modern filesystems are all going to be using delayed allocation, because of its inherent performance benefits, and whether you think the future belongs to ZFS, or btrfs, or XFS, or ext4 all of these filesystems used delayed allocation. What do you think? Do you think all of these filesystems have gotten things wrong, and delayed allocation is evil? Should I try to implement a data=alloc-on-commit mount option for ext4? Should we try to fix applications to properly use fsync() and fdatasync()? Related posts (automatically generated):

1. Don t fear the fsync! After reading the comments on my earlier post, Delayed allocation...
2. SSD s, Journaling, and noatime/relatime On occasion, you will see the advice that the ext3...
3. Ext4 is now the primary filesystem on my laptop Over the weekend, I converted my laptop to use the...

If you code for long enough on x86-64, you'll eventually hit an error such as: Here's a little example that might help you figure out what you've done wrong. Consider the following code: In case it's not clear, that would look something like: Let's build that code, and see what it looks like We can see that the mov instruction has only allocated 4 bytes (00 00 00 00) for the linker to put in the address of foovar. If we check the relocations: The R_X86_64_32S relocation is indeed only a 32-bit relocation. Now we can tickle this error. Consider the following linker script, which puts the foo section about 5 gigabytes away from the code. This now means that we can not fit the address of foovar inside the space allocated by the relocation. When we try it: What this means is that the full 64-bit address of foovar, which now lives somewhere above 5 gigabytes, can't be represented within the 32-bit space allocated for it. For code optimisation purposes, the default immediate size to the mov instructions is a 32-bit value. This makes sense because, for the most part, programs can happily live within a 32-bit address space, and people don't do things like keep their data so far away from their code it requires more than a 32-bit address to represent it. Defaulting to using 32-bit immediates therefore cuts the code size considerably, because you don't have to make room for a possible 64-bit immediate for every mov. So, if you want to really move a full 64-bit immediate into a register, you want the movabs instruction. Try it out with the code above - with movabs you should get a R_X86_64_64 relocation and 64-bits worth of room to patch up the address, too. If you're seeing this and you're not hand-coding, you probably want to check out the -mmodel argument to gcc.

I've just uploaded Policy 3.8.1.0 with the first batch of post-lenny changes. There isn't anything that earth-shattering in this set, but I'm hoping that we'll be able to get a lot more done during the squeeze release cycle. Policy has well over a hundred open bugs, quite a few of which we should be able to resolve. I also uploaded Lintian 2.2.8, which is synchronized against Policy 3.8.1 and also reverts some changes around install-menu due to a reverted change in debhelper 7.2.5, and fixes diagnosis of diversions. The last few versions of Lintian have all been a little bit buggy due mostly to the current pace of change. Hopefully this version will have fewer unexpected problems. Here is the summary of normative Policy changes, taken from upgrading-checklist.txt.gz in the debian-policy package.

• Care should be taken when adding functionality to essential and such additions create an obligation to support that functionality in essential forever unless significant work is done.
• Changelog files must be encoded in UTF-8.
• Tighten some format requirements for changelog files from a should to a must.
• Remove alternative changelog formats. Debian only supports one changelog format for the Debian Archive.
• New nocheck option for DEB_BUILD_OPTIONS indicating any build-time test suite provided by the package should not be run.
• All control files must be encoded in UTF-8.
• debian/control allows comment lines starting with # with no preceding whitespace.
• Init scripts ending in .sh are not handled specially. They are not sourced and are not guaranteed to be run by /bin/sh regardless of the #! line. This brings Policy in line with the long-standing behavior of the init system in Debian.
• The start action of an init script must exit successfully and not start the daemon again if it's already running.
• /var/run and /var/lock may be mounted as temporary filesystems, and init scripts must therefore create any necessary subdirectories dynamically.
• /bin/sh scripts may assume that local can take multiple variable arguments and supports assignment.
• User mailboxes may be mode 600 and owned by the user rather than mode 660, owned by user, and group mail.

Wouter and Christian have started bragging about their travel arrangements to attend DebConf, with probably insane amounts of toxic substances aimed at making DebConf9 the smelliest DebConf ever. So, it is time for me to announce my intentions as well. I have booked my ticket to travel MEX-MAD, leaving on Thursday 16/07/2009 and arriving to Spain on Friday 17. That means I will be getting to C ceres Friday evening. And I am flying back on Sunday 02/08/2009, which only means that I'll be jetlagged on Monday at work :-P Buying plane tickets when your country's currency has lost 50% of its worth against any other major currency in the world (US dollars are now at ~MX$15.50, and Euros are quickly approaching MX$20), let me tell you, is no fun. Even less if the plane tickets have increased their cost in absolute terms as well, due to the decrease in demand. But, yes, the good news do stand: See you in C ceres!

I finally got a chance to overhaul my backporting helper script to add support for lenny. The new version bases the default chroot basepath on the target distribution, chooses the right version suffix for backports.org based on the distribution, and tweaks a few other things. One should be able to use it for both stable and oldstable backports, although I've not tested it all that thoroughly yet. The default target is now lenny. There are, as yet, no transforms applied for lenny backports, but I'll be adding them as I run into necessary changes for the packages that I backport. You can get the latest version from my scripts page.