I have just been asked for advice about secure filesystem and decided to blog my answers. The first issue is what is meant by secure filesystem, that could either mean the ability to restrict file access (EG by supporting SE Linux security contexts and using SE Linux for file access control) or the ability to encrypt data in case the machine is stolen. For access control I recommend SE Linux of course. For encryption on a local machine I mostly use dm-crypt which is configured with the cryptsetup utility. I encrypt at the LVM logical volume level as it is common that there are some LVs that don t need to be encrypted. For files that need extra encryption or files that are shared between machines I use GPG. A question was asked about kernel vs user-space filesystem encryption. AES is in the kernel so there is no lack in terms of strong encryption there. Also performance is pretty good (in most cases the CPU is fast enough that the hard drive is the bottleneck). For fine grained encryption (such as some of the experimental filesystems that encrypt data separately for each user) user-space is probably the only way to go. If you want servers to be high-security level and protected from hackers or unauthorised people then it s difficult to offer any advice that is smaller than a text book. I suggest that if you have such questions then you should do one of two things. If you are running a corporate IT department then hire an expert who can help with determine your specific requirements and meet them. If you want to learn about computer security and run your own systems in the best way possible then read as much from the experts as possible. If you are looking for a project to contribute to related to security then if you choose SE Linux I could offer some specific advice on things that need work. I suggest not deciding on whether to do kernel level or user level work up front, but decide first which area of security you want to work on and then select a project which fits - then you should be able to determine whether your skills are best suited to kernel or user space coding. As for whether developing a new filesystem is necessary, I will note that SE Linux works well on Ext3 and XFS, it has just become usable on JFFS2, and it will work on other newer filesystems in the near future. Adding SE Linux support to a filesystem is not a difficult task if the filesystem supports XATTRs. I believe that there is a lot of scope for other access control systems to be developed which use XATTRs for security labels. I can t advise on e-books. I generally don t read books, I read blogs and papers. Anything that I read which I consider to be worth recommending will probably have a link from my blog.

I received a SheevaPlug this week, an intriguing device that packs incredible power and functionality into a tiny package. As many of you know, I've been doing a lot of work on Debian for the Linksys NSLU2 in the last few years. The NSLU2 is a key reason why ARM has become the third most popular architecture in Debian (after 32 and 64 bit x86), and I believe a main reason is that the NSLU2 is so incredibly cheap. At a price under $100, most people don't think too long and simply buy a device and do something cool with it. The SheevaPlug is being offered at the same price range but offers considerably more. Riku Voipio asked the right question: "What would you do with something approximately 10x more powerful with same prize/size range?" I believe the SheevaPlug is a killer replacement for the NSLU2 and here's why:

• Marvell Kirkwood 6281 CPU at 1.2 GHz with 256 KB L2 cache (compared to a 266 MHz CPU)
• 512 MB RAM (compared to lousy 32 MB)
• 512 MB flash, enough for a basic Debian installation (compared to 8 MB)
• Gigabit Ethernet (compared to 100 Mb) and no proprietary microcode required
• 1x USB and 1x SD (compared to 2x USB: the only advantage the NSLU2 has)
• ... and finally: mini-USB with serial console and JTAG: while Debian on the NSLU2 works well and is stable, we regularly had users whose system would no longer boot and there was no good way to figure out why. The serial console on the SheevaPlug actually allows users to figure out what's going on.

I'm incredibly excited about the SheevaPlug and the first thing I did was to take the device apart and look at the inside. The results can be found in the SheevaPlug image gallery. My next project will be slightly more productive: porting Debian. As I see it, we should support the following three installation variants for the SheevaPlug:

• To an external USB drive or stick.
• To an SD card.
• To the 512 MB flash memory on the device.

The first two should be relatively straight forward, but of course installing to the internal flash memory is particularly interesting given that 512 MB (plus compression) is enough for a basic installation of Debian. Unfortunately, installations to MTD flash are currently not supported in the Debian installer but I hope we can find a volunteer who wants to implement this functionality. My next steps are to put a kernel for the SheevaPlug into the archive and to get a basic installation going. From there we can look at more sophisticated installation options and other functionality.

This morning it was once more time for the annual March Madness Half Marathon in Cary. This race is basically the start of the running season in Chicagoland. And we could not have asked for better weather. After a really cold and long winter, and a short snapback to really cold temperatures this week, it started to warm up a little yesterday with expectations of more of the same today. So while it was still cold at the start at around 37 degrees, most runners opted for shorts and by the finish temperatures were in the high 40s. Coupled with clear blue skies and no wind, it was a really nice morning for a run. So how did it go? Well, I had pretty low expectations. Training has been difficult with too much snow, rain and plain cold weather. So like most of my running friends, motivation ran pretty low of late. I was really only trying to set a modest pace, and to hope to hang on to it and run steadily. That worked: I didn't walk a single water stop, and while my legs were getting really tired and sore I carried through to the end. Final time was 1:36:08.57 per my stop watch. That's a tad faster than last year, a lot slower than 2007, just a tad slower than 2006, and quite a bit faster than 2005. Next stop: my second Boston Marathon in five weeks and given the underwhelming training this winter, it will be a challenge.

This weekend I mostly fiddled around migrating machines from Xen hosting to KVM hosting. Ultimately it was largely a waste of time, due to various other factors. Still with a bit of luck it will be possible to move the machiens next week. That aside I spent a while updating my blogspam detection site. As a brief recap this site offers a simple XML-RPC service which allows you to test whether incoming blog comments are spam or not. Originally this was put together to fight an invasion of comments submited to the Debian Administration website: The site currently shows:

Site	Spam	Non-Spam	% spam
debian-administration.org	238	372	60.98% spam

Depressing. But not as depressing as the real live stats which show since I last reset the counters 36,995 spam comments vs. 1,206 non-spam comments. (live updating counters here) Anyway I updated the service today to add two new plugins, both of which are a little reactionary. The first new plugin is called "multilink" and is based upon the observation that spammers rarely know the markup of the site they are submitting comments to. This means you can frequently see submitted comments like this:

 <a href="http://spam.com">buy viagra</a>
 [url=http://spam.com]buy viagra[/url]
 [link=http://spam.com]buy me[/link]

Here we have three different styles of links - "a href", "link=", and "url=". I figure this is a clear indicator of a confused mind, or more likely a spammer. The second new plugin is designed to stop people who enter "<strong>" words. It is a little coarse but actuall zero false positives in the real world so I'm going to leave it live to see how it works out. In happier news I'm just back from a trip to the beach. Sand rocks. Even if it wasn't windy enough for my kite .. ObFilm: Dracula ("Bram Stoker's Dracula" - 1992)

Warning: Don t upgrade root partition (/) or the partition in which /boot is placed unless you know what you are doing. More on this later.
Warning2: Make backups of your data. Last Friday I went on upgrading to ext4 my laptop. I decided to go first with /home partition, as / has also /boot and can give some problems if not handled with care. Upgrading it is quite easy, and fast, as the old data is not upgraded, only new files will be added using new features. First, I installed latest linux-image package to start using kernel version 2.6.28, which includes production ready ext4 support. After rebooting, I logged in as root so I could umount /home. The steps to upgrade are as follows:

# uname -a
Linux gimli 2.6.28-1-686 #1 SMP Mon Feb 23 03:13:24 UTC 2009 i686 GNU/Linux
# umount /home
# tune2fs -O extents,uninit_bg,dir_index /dev/home_partition
# fsck -pfD /dev/home_partition
# mount /home

You MUST run fsck. If not, ext4 won t mount your filesystem, so don t forgive this step if you re upgrading your / partition. You will see some checksums errors appear. Don t be afraid, that s expected, and that s why -p is added to fsck command, you it is not asking which action to perform in every error. Upgrading partition containing /boot
For upgrading your / partition you have to take into account what grub and klibc version you have installed. The former is need so grub can read your kernel image from filesystem, and the second is needed due a bug that made initramfs detect ext4 filesystems as ext3, and passing incorrect option to mount, which failed. Versions in Debian known to work are libklibc >=1.5.15-1 and grub2 (packaged as grub-pc). Make sure you have these versions installed before trying to go to ext4.
Also, as you have to upgrade having your partition unmounted, you will have to get a console before root partition is mounted. You can do that by passing mount=break break=mount to your boot options in grub. ENJOY!

I m in the middle of migrating a mail server away from the Cyrus mail store [1]. Cyrus provides a POP and IMAP server, a local delivery agent (accepting mail via LMTP). It is widely believed that Cyrus will give better performance than other mail stores, but according to a review by linux-magazin.de Dovecot and Courier deliver comparable (and sometimes better) performance [2]. The biggest problem with Cyrus is that it is totally incompatible with the Unix way. This wouldn t be a problem if it would just work and if it would display reasonable error messages when it failed, but it doesn t. It often refuses to work as desired, gives no good explanation, and it s data structures can t be easily manipulated. Dovecot [3] and Courier [4] use the Maildir++ format [5] (as well as many other programs). I have set up a system with Courier Maildrop and Dovecot for the IMAP server [6] and it works well - it s good to have a choice! But also Maildir++ is reasonably well documented and is an extension to the well known Maildir format. This means that it s easy to manipulate things if necessary, I can use mv to rename folders and rm to remove them. Cyrus starts with a database (Berkeley DB file) of all folders in all mailboxes. Therefore it is not possible to move a user from one back-end server to another by merely copying all the files across and changing the LDAP (or whatever else contains the primary authentication data) to point to the new one. It also makes it impossible to add or remove folders by using maildirmake or rm -rf. The defined way of creating, deleting, and modifying mailboxes is through IMAP. One of the problems with this is that copying a mailbox from one server to another requires writing a program to open IMAP connections to both servers at once (tar piped through netcat is much faster and easier). Also if you need to rename a mailbox that contains many gigabytes of mail then it will be a time consuming process (as opposed to a fraction of a second for mv). Cyrus has a tendency to break while Dovecot is documented as being self-healing and Cyrus also seems to cope well in the fact of a corrupted mail store. Even manually repairing problems with Cyrus is a painful exercise. The Cyrus mail store is also badly designed - and it s design was worse for older filesystems (which were common when it was first released) than it is for modern ones. The top level of a Cyrus maildir contains all the messages in the INBOX stored one per file, as well as three files containing Cyrus indexes and sub-directories for each of the sub-folders. So if I want to discover what sub-folders a mailbox has then I can run ls and wait for it to stat every file in the directory or I can use an IMAP client (which takes more configuration time). As opposed to a Maildir++ store where every file that contains a message is stored in a folder subdirectory named new , cur , or tmp which means that I can run ls on the main directory of the mail store and get a short (and quick) result. Using tools such as ls to investigate the operation of a server is standard practice for a sysadmin, it should work well! A finall disadvantage of Cyrus seems to have many small and annoying bugs (such as the reconstruct program not correctly recursing the sub folders). I guess it s because not many people use Cyrus that such things don t get fixed. One trivial advantage of Cyrus is that by default it splits users into different sub-directories for the first letter of the account name. Dovecot supports using a hash of the user-name this is better than splitting by first-letter for performance (it gives a more equal distribution) but will make it slightly more difficult to manipulate the mail store by script. Ext3 can give decent performance without a two level directory structure for as many as 31,998 sub-directories (the maximum that it will support) due to directory indexing and Linux caching of dentries. There may be some other advantages of Cyrus, but I can t think of them at the moment. Here is a script I wrote to convert Cyrus mail boxes to Maildir++. To make this usable for a different site would require substituting a different domain name for example.com (or writing extra code to handle multiple domains) and inserting commands to modify a database or directory with the new server name. There is no chance of directly using this script on another system, but it should give some ideas for people performing similar tasks.

#!/bin/bash -e
# $1 is the user to move # BASEDIR is the directory for temporary files
BASEDIR=/mail/new
rm -rf $BASEDIR/$1 archive.tgz archive2.tgz ORIGSRV=$(echo "MYSQL_COMMAND_TO_GET_SERVER" mysql dbname)
echo "original server was $ORIGSRV" if [ "$ORIGSRV" = "" ]; then
echo "No server for account, strange error."
exit 1
fi ORIGPARENT="/mail/domain/e/example.com/$(echo $1 cut -c1)/user" # copy the data to a local archive
# USERDIR is for using ^ instead of . as Cyrus does.
mkdir -p $BASEDIR
cd $BASEDIR
USERDIR=$(echo $1 sed -e "s/\./^/g")
ssh cyrus@$ORIGSRV "(cd $ORIGPARENT ; tar czf - $USERDIR)" > archive.tgz
tar xzf archive.tgz
if [ "$USERDIR" != "$1" ]; then
mv $USERDIR $1
fi DEPTH=$(find . -mindepth 3 -type d wc -l)
if [ "$DEPTH" != "0" ]; then
echo "too deep"
exit 1
fi # remove cyrus index files
find $1 -name "cyrus.*" -exec rm \;
cd $1 # "mail" is a directory with sub-folders, every other name starting with
# [a-zA-Z] is a regular folder that needs to become a Maildir
for n in [a-zA-Z]* ; do
if [ "$n" != "mail" ]; then
cd "$n" ; mkdir new cur tmp ; touch maildirfolder
mv [0-9]* new true
cd .. ; mv "$n" ".$n"
fi
done # move the inbox files to Maildir storage
mkdir new cur tmp
mv [0-9]* new true
mv mail/[0-9]* cur true # move sub-folders of "mail/" and merge ones with duplicate names
cd mail
for n in * ; do
if [ -e "../.$n" ]; then
cd "$n"
mv [0-9]* "../../.$n/cur" true
cd .. ; rmdir "$n"
else
cd "$n" ; mkdir new cur tmp ; touch maildirfolder
mv [0-9]* new true
cd .. ; mv "$n" "../.$n"
fi
done
cd ..
rmdir mail # get a list of all names so we can recognise duplicates
find . -type d egrep -v "new cur tmp" cd $BASEDIR ssh root@localhost "mv /mail/new/$1 /mail/example.com ; chown -R mail:mail /mail/example.com/$1"
echo "MYSQL_COMMAND_TO_SET_SERVER" mysql dbname

• [1] http://cyrusimap.web.cmu.edu/


• [2] http://www.linux-magazin.de/heft_abo/ausgaben/2007/06/auf_der_teststrecke/


• [3] http://www.dovecot.org/


• [4] http://www.courier-mta.org/imap/


• [5] http://wiki.dovecot.org/MailboxFormat/Maildir


• [6] http://etbe.coker.com.au/2009/03/10/maildrop-imap-postfixadmin/

Today, I was sitting on the couch. Jacob crawled up, and pushed me forward, saying I go there. He crawled behind me. Then, ready for one of his favorite games, yelled out: QUISH, daddy! So I leaned back gently a bit, and said squish! Jacob yelled, laughing, QUISH! Quish again, daddy! So I d lean back a bit, gently, again, this time reaching behind me to tickle him a bit as I leaned. Squash! Louder laughing. QUASH!!! Right in my ear ouch, but I didn t mind. Do it again, daddy! Right now, being a dad seems complicated enough. You ve got to have the right touch to squish a 2-year-old without really squishing him. Or have the presence of mind to realize that when Jacob was happily playing outside, then suddenly comes running over, very upset, saying Go inside! it means he probably needs to use the potty urgently, or just had an accident. (Or both, as it happened today.) Or to recognize that little walk that means he really does need to use the potty even though he d rather not. And, of course, there s figuring out what he s saying, when his words can still be a bit garbled. But these all seem simple to me, compared to what will come. How will we help Jacob to grow as a person of good character? How will we meet his need to be challenged intellectually? Will we be able to maintain a good relationship, and yet still have the judgment to have the right set of rules, when he gets to high school? Will I have a good relationship with him as an adult? And how am I going to react when the day comes when he tells me I ought to move into the nursing home? Jacob, of course, doesn t care about any of that right now. Each night, when I put him down for the night, he wants me to cover him up with blankets. Once I ve done that, he peeks out and says, Have a good night, dad! I always reply with You have a good night too, Jacob! If we can get along that well for the next 60 years, I guess we ll do all right.

Edited to add: Sorry, it turns out that ext4 does now have a heuristic to deal with this case. However, it seems that btrfs doesn't. My point remains, though - designing a filesystem in such a way that a useful behaviour is impossible is unhelpful for the vast majority of application writers, even if it does improve performance in some other use cases.

Further edit: It's mentioned in the comments that btrfs will force a flush of data when a rename is performed as of 2.6.30. That prevents the data loss, but it means that we're still stuck with disk access when we'd be happy with that being batched up for the future. It feels like we're optimising for the wrong case here.

Original text:

There's been a certain amount of discussion about behavioural differences between ext3 and ext4[1], most notably due to ext4's increased window of opportunity for files to end up empty due to both a longer commit window and delayed allocation of blocks in order to obtain a more pleasing on-disk layout. The applications that failed hardest were doing open("foo", O_TRUNC), write(), close() and then being surprised when they got zero length files back after a crash. That's fine. That was always stupid. Asking the filesystem to truncate a file and then writing to it is an invitation to failure - there's clearly no way for it to intuit the correct answer here. In the end this has been avoided by avoiding delayed allocation when writing to a file that's just been truncated, so everything's fine.

However, there's another case that also breaks. A common way of saving files is to open("foo.tmp"), write(), close() and then rename("foo.tmp", "foo"). The mindset here is that a crash will either result in foo.tmp being zero length, foo still being the original file or foo being your new data. The important aspect of this is that the desired behaviour of this code is that foo will contain either the original data or the new data. You may suffer data loss, but you won't suffer complete data loss - the application state will be consistent.

When used with its (default) data=ordered journal option, ext3 provided these semantics. ext4 doesn't. Instead, if you want to ensure that your data doesn't get trampled, it's necessary to fsync() before closing in order to make sure it hits disk. Otherwise the rename can occur before the data is written, and you're back to a zero length file. ext4 doesn't make guarantees about whether data will be flushed before metadata is written.

Now, POSIX says this is fine, so any application that expected this behaviour is already broken by definition. But this is rules lawyering. POSIX says that many things that are not useful are fine, but doesn't exist for the pleasure of sadistic OS implementors. POSIX exists to allow application writers to write useful applications. If you interpret POSIX in such a way that gains you some benefit but shafts a large number of application writers then people are going to be reluctant to use your code. You're no longer a general purpose filesystem - you're a filesystem that's only suitable for people who write code with the expectation that their OS developers are actively trying to fuck them over. I'm sure Oracle deals with this case fine, but I also suspect that most people who work on writing Oracle on a daily basis have very, very unfulfilling lives.

But anyway. We can go and fix every single piece of software that saves files to make sure that it fsync()s, and we can avoid this problem. We can probably even do it fairly quickly, thanks to us having the source code to all of it. A lot of this code lives in libraries and can be fixed up without needing to touch every application. It's not the end of the world.

So why do I still think it's a bad idea?

It's simple. open(),write(),close(),rename() and open(),write(),fsync(),close(),rename(), are not semantically equivalent. One is "give me either the original data or the new data"[2]. The other is "always give me the new data". This is an important distinction. fsync() means that we've sent the data to the disk[3]. And, in general, that means that we've had to spin the disk up.

So, on the one hand, we're trying to use things like relatime to batch data to reduce the amount of time a disk has to be spun up. And on the other hand, we're moving to filesystems that require us to generate more io in order to guarantee that our data hits disk, which is a guarantee we often don't want anyway! Users will be fine with losing their most recent changes to preferences if a machine crashes. They will not be fine with losing the entirity of their preferences. Arguing that applications need to use fsync() and are otherwise broken is ignoring the important difference between these use cases. It's no longer going to be possible to spin down a disk when any software is running at all, since otherwise it's probably going to write something and then have to fsync it out of sheer paranoia that something bad will happen. And then probably fsync the directory as well, because what if someone writes an even more pathological filesystem. And the disks sit there spinning gently and chitter away as they write tiny files[4] and never spin down and the polar bears all drown in the bitter tears of application developers who are forced to drink so much to forget that they all die of acute liver failure by the age of 35 and where are we then oh yes we're screwed.

So. I said we could fix up applications fairly easily. But to do that, we need an interface that lets us do the right thing. The behaviour application writers want is one which ext4 doesn't appear to provide. Can that be fixed, please?

[1] xfs behaves like ext4 in this respect, so the obvious argument is that all our applications have been broken for years and so why are you complaining now. To which the obvious response is "Approximately anyone who ever used xfs expected their data to vanish if their machine crashed so nobody used it by default and seriously who gives a shit". xfs is a wonderful filesystem for all sorts of things, but it's lousy for desktop use for precisely this reason.

[2] Yes, ok, we've just established that it actually isn't that in the same way that GMT isn't UTC and battery refers to a collection of individual cells and so you don't usually put multiple batteries in your bike lights, but the point is that this is, for all practical intents and purposes, an unimportant distinction and not one people should have to care about in their daily lives.

[3] The disk is free to sit there bored for arbitrary periods of time before it does anything, but that's fine, because the OS is behaving correctly. Sigh.

[4] Dear filesystem writers - application developers like writing lots of tiny files, because it makes a large number of things significantly easier. This is fine because sheer filesystem performance is not high on the list of priorities of a typical application developer. The answer is not "Oh, you should all use sqlite". If the only effective way to use your filesystem is to use a database instead, then that indicates that you have not written a filesystem that is useful to typical application developers who enjoy storing things in files rather than binary blobs that end up with an entirely different set of pathological behaviours. If I wanted all my data to be in oracle then I wouldn't need a fucking filesystem in the first place, would I?

Excellent summary of where we find the newspaper business today:

Round and round this goes, with the people committed to saving newspapers demanding to know If the old model is broken, what will work in its place? To which the answer is: Nothing. Nothing will work. There is no general model for newspapers to replace the one the internet just broke.

Oops 3/16/09. Shirky, not Sharky.

Luk had a nice idea today in the release channel: Is there anyone attending this years DebCamp willing to teach us (at least Luk and me) R?

On off the biggest problems with developing servlets under a container like Tomcat is the amount of time taken to build your code, deploy it to the container and restart it to pick up any changes. Maven and the Jetty plugin allow you to cut down on this cycle considerably. The first step is to allow you to start your application in maven by running:

mvn jetty:run

We do this by configuring the jetty plugin inside our pom.xml:

<plugin>
   <groupId>org.mortbay.jetty</groupId>
   <artifactId>maven-jetty-plugin</artifactId>
   <version>6.1.10</version>
</plugin>

Now when you run mvn jetty:run your application will start up. But we can improve on this. The Jetty plugin can be configured to scan your project every so often and rebuild it and reload it if anything changes. We do this by changing our pom.xml to read:

<plugin>
   <groupId>org.mortbay.jetty</groupId>
   <artifactId>maven-jetty-plugin</artifactId>
   <version>6.1.10</version>
   <configuration>
      <scanIntervalSeconds>10</scanIntervalSeconds>
   </configuration>
</plugin>

Now when you save a file in your IDE, by the time you've switched to your web browser, Jetty is already running your updated code. Your development cycle is almost up to the same speed as Perl or PHP. You can find more information at the plugin page.

For some time ago I ve been pondering about this question. As long as GSoC 2009 is about to start and people are looking for project ideas, I m posting here a very preliminar draft of my findings and an idea for a posible software project. It s pretty written on-the-fly while I ve managed to have some minutes between work and uni. Rigurous wording isn t P1. Debian s social contract 4th item states:

Our priorities are our users and free software We will be guided by the needs of our users and the free software community. We will place their interests first in our priorities.

When we make decisions, either technical, legal and others regarding the operating system we deliver, this is one of the most referred argument. Altough that s seen as high priority often we don t have ways to properly know what our users want. While working on translations on the Spanish team we ve faced this situation. Do our users find them useful? they are able to understand them? they find that a much familiar wording would be bette than high technical details? We didn t know. Then, the question still exists. Altough Debian has ways for users to provide feedback (BTS, mailing lists, IRC, popcon, etc) none of them is designed to offer a way for the user to provide feedback on features, development roadmap and other non-bug aspects that developers and the project can tally and use for prioritize on releases. It s rather amusing to note that one of the most valuable assets free software community has is user involvement and contribution. But before we enter in details let s start on the basics. For using this information we first need to know who our users are? do we? From my findings and interaction with the community I ve identified two kinds of Debian users who can be clearly named. 1) The derivatives or pure blends, who use the Debian base and framework to build niche distributions. 2) The lead users (using E. Von Hippel s definition), who are the developers, contributors and a group of users. Debian is, if not the only, one of the projects who is best for fostering lead users. Most of them at some point involve themselves in the development process and some, later become developers. There is a more diverse group who are end-users but it s unclear to me how we can group them. Despite that they probably represent the biggest part of the pie. So, how we do please everyone? can we? In my opinion we can, at some degree. By implementing tools for first: gather user feedback, make statistical data, tally, we later can take informed decisions. In Debian decisions are voted and the set of people who votes are only developers with their own constraints. I think this would ease the constant that release-decisions-regarding-foo-tech-legal-issue represented on each release iteration, saving time, health and bits. It could be useful for knowing more about who are the ones we develop things for. I m thinking of some sort of Dell s IdeaStorm.com for software. What do you think? I m willing to co-mentor this if someone finds that is an interesting project for GSoC. Let me know!.

Since this January, I ve been doing upstream work for Liferea. This is a great oportunity to learn C and to contribute more to an upstream project! And the atmosphere around Liferea is great. Today I ve published my first post in the Liferea blog. If you are a Liferea user, you may want to subscribe to it! There is a lot of work going on in Liferea. We are working hard to release 1.6 (which will use WebKit as the rendering backend) without any known regressions. Most of the performance work will likely go into the next series though, but 1.6 shouldn t be any worse than 1.4. If you would like to contribute, don t be shy and join #liferea on Freenode! We have some blockers for 1.6 and some other things to do, and we will appreciate any contributions. Testing is also appreciated. We are mostly interested in the unstable series, so if you find a bug in the latest unstable release or in trunk, file a bug report!

I ve had smart quotes envy for a long time now. I particularly like them in contractions, and I think they make the text more pleasant to read. Of course, I ve been using them on my web pages and blog for some time now, since they are well-supported by browsers. TeX/LaTeX deals with them fine as well, even by just typing ASCII in the source document. Mostly only mail was missing. I ve delayed using smart quotes in mail for a long time, since they imply a lot of UTF-8, and there was/is still quite a lot of software and setups not ready for it. But we re in the 10th year of the 21st century already, and I was getting envious of the increasing number of people that are using them for e-mail. So I ve recently switched to smart quotes in my outgoing mail. I realize this goes against the Be strict in what you emit and liberal in what you accept principle, which is a fine one, but I must confess aesthetics have won the battle for me in this case. I ve already received one complaint from a person using broken software, asking me to send more comprehensible and correct messages , and referring to UTF-8 as WTF-8. I wonder how many people to whom my mails are targetted at will have trouble reading them now. I don t have comments enabled in this blog yet I m afraid, but feel free to participate in the poll.

So, Kodak announced the availability of Linux drivers for some of its scanners, together with a frontend application. While the effort is laudable, they re not quite there yet. Let s have a look. First of all, and contrary to what a few people believe after reading the press release, the drivers themselves aren t OpenSource, let alone Free Software. They re all proprietary; read better, the press release clearly mentions that only the frontend application is GPLv2. Now let s look at what Kodak offers in its drivers packages. I ve downloaded LinuxSoftware_i1200_v3.5.tar.gz from the Kodak website, these are the drivers for the Kodak i1200 series. The archive contains a set of RPM packages, one Debian package corresponding to one of the RPM packages, and a setup script. One of the packages contains the OpenUSB library, which is released under the LGPL; sources aren t there, I haven t asked for the sources, don t plan on doing so, and am not implying anything regarding license compliance whatsoever. I ll note that I haven t seen any written offer for the sources, though, be it in the archive (there s not even a README in there) or on the website. So, although Ubuntu is listed as supported, we can say that no Debian packages are provided. The setup script actually relies on alien to convert them if it thinks it s running on a Debian-based system, which is determined by the availability of dpkg in the PATH. Using alien is far from ideal. The packages ship libraries and components under /usr/lib, /usr/local/lib and even /opt, pretty randomly it seems. Let s repeat here that /usr/local and /opt are strictly reserved for the local administrator and packages must not install anything here. There are other oddities in the paths used by the software stack, like /var/kodak. It looks like the SANE backend is only a bridge to Kodak s TWAIN data source, so you d actually need the whole stack to use it. All in all, Kodak does no better, I d even venture to say it does worse, than Epson (Avasys). At least part of Epson s backend is Free Software, even if it relies on proprietary libraries for some scanners, though unfortunately most if not all of the recent ones. I also have to mention that Epson is offering both RPM and Debian packages, and by that I mean proper Debian packages. Kodak are far from that, even the RPM aren t that great (I think the RPM packages even lack dependency information, based on what I ve read in the setup script, though I haven t checked). What Kodak offers here is a GPL frontend for TWAIN that not many people will be using, while keeping the drivers proprietary. Unfortunately their SANE backend isn t a backend so to speak, and even if it was OpenSource it d still rely on the rest of the TWAIN software stack. So, in a nutshell, while I m very happy to see yet another vendor go down the road of Linux support for its products, I am immensely disappointed by the proprietary nature of the drivers and the low quality of the packages. I was, of course, investigating the drivers to possibly package them for Debian; as you ve probably understood, it s not possible, both technically and legally. And I kept a little gem from the setup script for the end:

# We need to create a link for the libdbus-1 requirement for openusb,
# but only on Fedora.
if [ -f /etc/fedora-release ]; then
    ln -s /lib/libdbus-1.so.3 /lib/libdbus-1.so.2 2>> /dev/null
fi

As we all know, sonames and soversions are useless and they just keep getting in the way. Duh.

Just a quick update to let you know how the Debian bid for the Google Summer of Code is going. We re doing great right now. Our application is sent out, you can consult the public part here. This year, we re committed to doing a great Summer of Code performance, starting first by improving communication with the Debian community. We opened a Twitter feed for you Twitter freaks: http://twitter.com/DebianGSoC. It will be updated with live information relevant to students and devs alike during the whole summer. UPDATE: We re now also on identi.ca : http://identi.ca/debiangsoc We now have 12 applications on our wiki. Come, add your ideas and discuss them on our IRC channel (#debian-soc on irc.debian.org) and mailing-list. Packaging

• Large dataset manager: download and update local copies of public datasets, and integrate them in Debian with the tools we packaged.
• Mergemaster: perform 3-way merge of config files on upgrades by keeping a copy of original config files.
• Aptitude packages query UI and language: improve the GTK+ search UI and language for complex packages queries
• Aptitude search ranking and presentation: improve the presentation of search results with substring highlighting, relevancy ranking
• Aptitude history tracking: extend and build on the packages actions history tracking features of Aptitude

Debian Installer

• MTD support in d-i: add MTD installation support to the Debian Installer
• Debian-Installer support for GNU/kFreeBSD: replace the currently used FreeBSD sysinstall with Debian-Installer running possibly under a FreeBSD kernel

Blends

• blends webtools: enhancing DebianPureBlends webtools by using UltimateDebianDatabase

Bug tracking

• Debbugs Read/Write Soap Interface: overhaul the existing read-only SOAP Debbugs interface with submission and manipulation capabilities
• Debbugs Web UI: Modern web interface for submitting and manipulating bugs.debian.org s bugs
• Mylyn plugin for debbugs: Plugin for Mylyn to support debbugs, allowing the management of Debian bug reports from the inside of Eclipse.

Building infrastructure

• Building a new interface for Wanna-build using PostgreSQL: replace the existing setup of multiple MLDBM databases with an extensible and flexible relational database for use both by wanna-build and other tools such as web CGI scripts

With any luck, someone has seen this problem before and can fix it easily. I don t seem to be able to find the fix by searching, but the search terms feel like they re either too general or too specific. I was trying to install some perl software on a debian lenny system that was upgraded from etch (and previously from sarge). After installing the dependencies, I started getting errors like this:-

$ perl -MYAML::Syck -e print
Can't locate YAML/Syck.pm in @INC (@INC contains:
/usr/local/lib/perl5/5.10.0/i686-linux /usr/local/lib/perl5/5.10.0
/usr/local/lib/perl5/site_perl/5.10.0/i686-linux
/usr/local/lib/perl5/site_perl/5.10.0 .).
BEGIN failed--compilation aborted.

Update: Well done to Raphael who spotted the local that I d overlooked. D oh! Now to see if I can discover where that came from. Update 2: Also thanks to Florian who emailed in at about the same time (I don t read my email as much as my website dashboard, usually).

Then I discovered that the debian mirror being used (which is in the same city, and in a nearby network) is marked as bad on the health check. I switched mirrors to a known-good one that I ve used for other systems recently, but no joy. I think a key symptom is this bit near the end of the output of perl -V:

Compiled at Aug 4 2008 09:48:59
@INC:
/usr/local/lib/perl5/5.10.0/i686-linux
/usr/local/lib/perl5/5.10.0
/usr/local/lib/perl5/site_perl/5.10.0/i686-linux
/usr/local/lib/perl5/site_perl/5.10.0

Our good systems show a December compiled at date and a much longer @INC list. I ve tried reinstalling pretty much every package I think might be connected, including libc6, libperl5.10, perl, perl-base and perl-modules (those last four are all version 5.10.0-19). I really don t want to reinstall the whole system because it s behind an irritating firewall that will make it a 10-hour round-trip site visit. I m currently hunting and removing obsolete packages to try to make sure they re not causing problems, because I m running out of straws to clutch at. How would you recover a debian system from this?

I received a sample of QNAP's new TS-219 Turbo NAS today. The TS-119 and TS-219 devices are an upgrade to QNAP's TS-109 and TS-209 devices and were announced earlier this month at CeBIT. The new TS-219 uses the same robust case as the TS-209 but offers much more performance:

• Marvell Kirkwood 6281 at 1.2 GHz (instead of Orion 5182 at 500 MHz)
• 512 MB RAM (instead of 128 or 256 MB)
• 16 MB flash (instead of 8 MB)

The numbers I've seen about the Kirkwood CPU suggest that the TS-219 will indeed give a very significant performance boost. So far, the TS-219 looks very nice. My only complaint is that QNAP didn't export the second Ethernet port the Kirkwood chip offers, but I suspect this is because they wanted to stay as close to the original design of the TS-209 as possible. In any case, the TS-219 is a nice machine and I look forward to porting Debian to it. As a first step, this means getting the mainline kernel to run on the device and adding a kernel for Kirkwood to the archive (the latter is needed for a number of other devices based on Kirkwood anyway). Hopefully, the remaining porting work and integration into the Debian installer should be fairly straightforward given that the TS-209 is already supported in Debian. I'm sure I'll find out as I dig into the details... I'll give periodic updates of my progress. In the meantime, I've created an image gallery of the TS-219.

A recent Ubuntu bug has gotten slashdotted, and has started raising a lot of questions about the safety of using ext4. I ve actually been meaning to blog about this for a week or so, but between a bout of the stomach flu and a huge todo list at work, I simply haven t had the time. The essential problem is that ext4 implements something called delayed allocation. Delayed allocation isn t new to Linux; xfs has had delayed allocation for years. Pretty much all modern file systems have delayed allocation, according to the Wikipedia Allocate-on-flush article, this includes HFS+, Reiser4, and ZFS; btrfs has this property as well. Delayed allocation is a major win for performance, both because it allows writes to be streamed more efficiently to disk, and because it can reduce file fragmentation so that later on they can be read more efficiently from disk. This sounds like a good thing, right? It is, except for badly written applications that don t use fsync() or fdatasync(). Application writers had gotten lazy, because ext3 by default has a commit interval of 5 seconds, and and uses a journalling mode called data=ordered. What does this mean? The journalling mode data=ordered means that before the commit takes place, any data blocks are
associated with inodes that are about to be committed in that transaction will be forced out to disk. This is primarily done for security reasons; if this is not done (which would be the case if the disk is mounted with the mount option data=writeback), then any newly allocated blocks might still contain previous data belonging to some other file or user, and after a crash, accessing that file might result in a user seeing unitialized data that had previously belonged to another user (say, their e-mail or their p0rn), which would be a Bad Thing from a security perspective. However, this had the side effect of essentially guaranteeing that anything that had been written was guaranteed to be on disk after 5 seconds. (This is somewhat modified if you are running on batteries
and have enabled laptop mode, but we ll ignore that for the purposes of this discussion.) Since ext3 became the dominant filesystem for Linux, application writers and users have started depending on this, and so they become shocked and angry when their system locks up and they lose data even though POSIX never really made any such guarantee. This become especially noticeable on Ubuntu, which uses many proprietary, binary-only drivers, which caused some Ubuntu systems to become highly unreliable, especially for Alpha releases of Ubuntu Jaunty, with the net result that some Ubuntu users have become used to their machines regularly crashing. (I use bleeding edge kernels, and I don t see the kind of unreliability that apparently at least some Ubuntu users are seeing, which came as quite a surprise to me.) So what are the solutions to this? One thing is that the applications could simply be rewritten to properly use fsync() and fdatasync(). This is what is required by POSIX, if you want to be sure that data has gotten written to stable storage. Some folks have resisted this suggestions, on two grounds; first, that it s too hard to fix all of the applications out there, and second, that fsync() is too slow. This perception that fsync() is too slow was most recently caused by a problem with Firefox 3.0. As Mike Shaver put it:

On some rather common Linux configurations, especially using the ext3 filesystem in the data=ordered mode, calling fsync doesn t just flush out the data for the file it s called on, but rather on all the buffered data for that filesystem.

Fundamentally, the problem is caused by data=ordered mode. This problem can be avoided by mounting the filesystem using data=writeback or by using a filesystem that supports delayed allocation such as ext4. This is because if you have a small sqllite database which you are fsync(), and in another process you are writing a large 2 megabyte file, the 2 megabyte file won t be be allocated right away, and so the fsync operation will not force the dirty blocks of that 2 megabyte file to disk; since the blocks haven t been allocated yet, there is no security issue to worry about with the previous contents of newly allocated blocks if the system were to crash at that point. Another solution is a set of patches to ext4 that has been queued for 2.6.30 merge window. These three patches (with git id s bf1b69c0, f32b730a, and 8411e347) will cause a file to have any delayed allocation blocks to be allocated immediately when a file is replaced. This gets done for files which were truncated using ftruncate() or opened via O_TRUNC when the file is closed, and when a file is renamed on top of an existing file. This solves the most annoying set of problems where an existing file gets rewritten, and thanks to the delayed allocation semantics, that existing file gets replaced with a zero-length file. However, it will not solve the problem for newly created files, of course, which would have delayed allocation semantics. Yet another solution would be to mount ext4 volumes with the nodelalloc mount option. This will cause a significant performance hit, but apparently some Ubuntu users are happy using proprietary Nvidia drivers, even if it means that when they are done playing World of Goo, quitting the game causes the system to hang and they must hard-reset the system. For those users, it may be that nodelalloc is the right solution for now personally, I would consider that kind of system instability to be completely unacceptable, but I guess gamers have very different priorities than I do. A final solution which might not be that hard to implement would be a new mount option, data=alloc-on-commit. This would work much like data=ordered, with the additional constraint that all blocks that had delayed allocation would be allocated and forced out to disk before a commit takes place. This would probably give slightly better performance compared to mounting with nodelalloc, but it shares many of the disadvantages of nodelalloc, including making fsync() to be potentially very slow because it would force all dirty blocks to be forced out to disk once again. What s the best path forward? For now, what I would recommend to Ubuntu gamers whose systems crash all the time and who want to use ext4, to use the nodelalloc mount option. I haven t quantified what the performance penalty will be for this mode of operation, but the performance will be better than ext3, and at least this way they won t have to worry about files getting lost as a result of delayed allocation. Long term, application writers who are worried about files getting lost on an unclena shutdown really should use fsync. Modern filesystems are all going to be using delayed allocation, because of its inherent performance benefits, and whether you think the future belongs to ZFS, or btrfs, or XFS, or ext4 all of these filesystems used delayed allocation. What do you think? Do you think all of these filesystems have gotten things wrong, and delayed allocation is evil? Should I try to implement a data=alloc-on-commit mount option for ext4? Should we try to fix applications to properly use fsync() and fdatasync()? Related posts (automatically generated):

1. Don t fear the fsync! After reading the comments on my earlier post, Delayed allocation...
2. SSD s, Journaling, and noatime/relatime On occasion, you will see the advice that the ext3...
3. Ext4 is now the primary filesystem on my laptop Over the weekend, I converted my laptop to use the...