As some of you may know, I am currently in Ireland, spending most of the summer at the Irish Software Engineering Research Centre to get a start on my Ph.D. research. This is the story of how I got here and how it's been so far. In the middle of July, I left my beloved city of Zurich and headed for Dublin. I commenced my stay in "the green country" with a three-day Debian workshop for the Google Europe sysops, which was intensive and yet enjoyable. I had between 4 and 10 participants, and the syllabus was to cover most everything, from APT basics to advanced concepts. That's a lot for three days, and I usually advise clients to consider a five-day or even two-week course instead, but after convincing the organiser to plan for three days rather than just two, I figured I'd be dealing with pretty good people anyway, given Google's hiring standards. I was not to be disappointed. We started the course with installation problems, because the 2.6 kernel install which I suggested could not deal with the SATA driver in the new IBM T43p notebooks which the participants were using. While one guy decided to try a 2.4 install with the IDE-sata driver, I opted to go down the debootstrap route instead, and was especially delighted -- at first -- to learn about a working netboot setup in the Google LAN. My joy vanished when I found the Ubuntu rescue image not to contain debootstrap, so I didn't have much choice but to prepare a root tarball and offer it for wget download. This worked, and after manual tweaking and making the system bootable, running into some issues after inadvertedly upgrading the entire system to the backports.org archive (which still does not offer a proper default pin), we decided to just go with testing and be done with it. I have to say that I really like courses where stuff does not work, as I believe those to be far more valuable to the participants. This time around, it was also the first time (after having taught maybe 60 or 70 courses in total) that I did not let myself get stressed out. Realising this put me in a great mood and made those days even more enjoyable. The rest of the workshop went without further complications, but judging from feedback over cigarettes, lunch, or the evening beer(s), the guys and girls still appreciated it. I have to thank Google and all participants for a great three days of hospitality and challenges, interesting discussions and fun. I really ought to reconsider my decision not to accept their offer for a sysop position... The last day saw me getting sick though, it must have been the air-conditioning. When I reached the hotel after the course, I definitely knew I had high fever and was developing a bad headache and cough, so I didn't get to enjoy a Friday night out in Dublin. Instead, I slept from 18:00 hours until the late morning, when I had to drag myself to the train station to catch the train to Limerick... which was painful, but I managed. Some time in between, I tried out nstx and managed to get a decent SSH connection to the outside world from the hotel room (just don't forget the -a -x flags!). Arriving in Limerick, I registered for the student apartment I had rented and found myself in a small room with a bed and desk, my own bathroom, a common room and kitchen I share with my room mate (whom I barely saw), including a TV, fridge, oven, microwave, and enough "little stuff" to make me conclude that this place was livable -- for 60 /week not a bad deal. I spent most of the weekend sleeping, but it was not until Tuesday that my state had returned to normal. The place also has an Ethernet jack, but that turned out to be unusable as it was (a) behind a fascist firewall, and (b) on the same network as about 40 other users, 17 of which seemed to be running peer-to-peer filesharing clients all hours of the day. In my office at the university, I found myself in the same situation as (a), but at least I managed to install OpenVPN, and dsniff and was thus ready to take back the net in my apartment. I wrote a little script which would (a) spoof ARP replies from the router for all hosts for a few minutes, (b) analyse the traffic it got to identify obvious P2P machines, and (c) proceed to poison their ARP cache every few seconds, effectively shaping their connection. I had to go this way, because tools like tcpnice (part of dsniff) do not work in a switched network very well, and neither did macof. Unfortunately, I lost the script in the recent XFS fiasco, but the major P2P offenders seem to have left anyway. Thanks to OpenVPN, I also got VoIP working and am happily tunneling through the university's proxy (which otherwise would require me to use the SOCKS4 proxy protocol with proxychains, which would be a nuisance). I really don't understand why a university blocks outgoing traffic. Oh well... It's been three weeks since I've come to Limerick, and my feelings are mixed. I've been spending most time in my (temporary) office (pretending to be) doing work, I have not really acquainted myself with the food here, the university campus is filled with hundreds of 14 year old summer school students but nobody of my age, and, well, research is coming along slower than I had hoped. I guess that's normal in a sense, but I also know for sure that I have a lot of work left to do on my discipline. I am a master of procrastination, you see, and Debian is a dangerous drug, a black hole that sucks in your time while it excites your synapses and splashes endorphines all around. But I've not lost my hope at the very least... Thanks for caring.

Erich complains: "If the file is damaged, just fill it up with zeroes?" (in response to my recent post on filesystem problems). Update: I rewrote the second sentence of the next paragraph (up until and including the footnote) to get things right. Thanks to Russell Cattelan for providing the info. This must be the most misunderstood feature of XFS. What happens is that XFS logs all metadata changes to the journal, except for the inode size, which gets flushed to disk immediately for performance reasons [*]. At this point, the file will actually be a sparse file, which is nothing more than a file whose metadata lists a file as being of a size different than it currently is (I realise the "sparse" does not really apply when the file is "overfull", i.e. when the metadata lists it as smaller than it really is, but I am lacking a good word for that). The disk extents get allocated only when the data actually hits the disk (that's XFS's famous delayed allocation mechanism). If the power fails before the data was flushed to disk and the journal entry cleared, XFS will serve zeroes, rather than the potentially random or sensitive data that is actually on disk. This is a good thing.

[*]	since almost every write() changes the file size, it would be a massive performance hit if every size change was logged. However, XFS actually violates its own journaling rules by doing this.

You can run into more or less the same problem with any journaling filesystem; the others just don't serve zeroes. Instead, they give you the data that's physically on the medium. Imagine the situation when the corrupt /etc/motd suddenly becomes a window to your previous /etc/shadow contents... I really prefer how XFS handles that. Sometimes you do get the old data back with the other filesystems, but this is because the filesystems may reuse the blocks of the old file. So it's a trade-off, and your choice between security and, uh, convenience. The only way to protect against this is to use "physical-block journaling" (as opposed to "logical journaling"), which is only supported by ext3 as far as I know (option data=ordered), at a massive performance loss. See this mailing list post by Theodore Ts'o for more info. Thanks to Alceste Scalas for sharing it with me. PS: Thanks to Wessel Dankers and Eric Sandeen for their help on #xfs. Update: Again thanks to Russell Cattelan, I just received word that code has been added to the 2.6.17 kernel which will flush an inode at close if it was truncated at some previous point. This should take care of many of the zeroed-file issues. Update: RazorFS also supports data=ordered. Thanks to Hans Kratz for pointing this out. He also adds some more information, parts of which I've written about previously:

One of the major problems of all journalled filesystems especially on notebooks/desktop machines leading to filesystem corruption is the harddisk writeback cache (enabled by default). With the writeback cache some writes are not immediately written to disk Now if a crash happens some of the data may not have hit the disk. The journalling filesystems however need a way to rely that the certain writes have hit the disk to ensure filesystem consistency. There are two solutons: Enable write barrier support in the filesystems (avalable for ext3 since around 2.6.8, mount option: barrier=1, available for the other filesystems as well now probably) or disabling the write cache altogether with hdparm/sdparm.

XFS supports barriers as well, but unfortunately not yet on non-physical media, like RAID or dm-crypt. ext3 seems to work okay on both. Enable your barriers today!

After some recent filesystem fun, I now have a laptop with encrypted partitions, using dm-crypt and LUKS, conveniently installed with partman-crypto. Pre-Scriptum: I have updated my previous post about XFS troubles in response to Ingo's statements. Just in case you care... The following post poses a question and a meta-question: As I was cycling home today, I thought about what would happen if my laptop lost power now in the middle of a write operation. Sure, ext3's journal would make sure the filesystem was consistent, but what about the actual data on the disk? Journaling works by writing a change to the journal, then writing it to disk, then deleting the enty from the journal. If the filesystem is mounted and it has entries in the journal, those are "played back", meaning they just get written to disk again, overwriting potentially incomplete data written as the power was lost. Yes, this is simplified, but it'll do for now. With encrypted filesystems, there's an important layer between the filesystem and the physical storage: the encryption algorithm. An encryption algorithm like AES works on blocks, by default 64 bytes in size. 64 bytes are not a whole lot, but given the nature of hard drives, it's not possible to write 64 bytes to the drive atomically [*]. Thus, conceivably, it could happen that your power cuts out after only 32 bytes have been written. I don't believe dm-crypt does journaling for the encrypted data it writes, so effectively, a power loss on systems with encrypted filesystems almost certainly means data loss? Please correct me if I'm wrong! And the meta-question would be: where would I go to discuss such issues? I've abused debian-security previously, the issue seems too general for the dm-crypt mailing list, there is cryptography t metzdowd.com but somehow this does not seem to be a crypto problem... so where would you take this? Let me know...

[*]

In a computer architecture course in college, the prof raised the interesting point that between the loss of power and the actual disappearance of all current in the circuits, several million operations are possible on modern machines. I have not yet heard of any computer or hardware device making use of those precious cycles. Update: actually, several manufacturers are making use of the precious cycles. See for instance this post, which describes some stuff done by SGI. Hanspeter Kunz also adds that harddrives use this time to park their heads.

I am through with XFS, once and for all. Well, at least for laptops. I still think it's a good filesystem when you can ensure that the power never goes, and your hardware is reliable, but it's just not adequate for laptops or even desktops. I ran into some serious problems a while ago, but managed to recover. Two nights ago, however, three XFS filesystems on my laptop decided to blow up and left my system thoroughly broken. I guess as the hibernate maintainer, I should really start doing my tests somewhere else than my main system... It all started out with a dist-upgrade and this output:

dpkg: error processing /var/cache/apt/archives/dpkg_1.13.22_i386.deb
(--unpack): unable to make backup link of
'./usr/share/man/man1/dpkg-deb.1.gz' before installing new version: Unknown
error 990

Looking at /usr/share/man/man1, I started to anticipate the apocalypse:

# ls -l /usr/share/man/man1
total 7956
?????????? ? ? ?    ? ? ? 7zr.1.gz
?????????? ? ? ?    ? ? ? 822-date.1.gz
?????????? ? ? ?    ? ? ? CA.pl.1ssl.gz
?????????? ? ? ?    ? ? ? Defoma::Common.1.gz

So I look at the log, and amidst kernel oops notices, there's this lovely cookie:

Filesystem "hda6": Corruption of in-memory data detected.  Shutting down
filesystem: hda6

Filesystem hda6 is /usr, so at that time I figured "it could have been worse", booted to single user, and remade the filesystem with the intention to simply reinstall all packages... when I found /var/lib/dpkg/info to be in similar condition. The rest of /var seemed fine, but I resolved then that there was no hope in reviving this system. Fortunately I brought an external drive that had just enough free space to hold my /home and some other stuff, but since USB is really slow when it comes to shifting large amounts of data, I decided to do something productive in the mean time and to answer some outstanding mails. It wasn't difficult to get SSH back up, so I started to work on a remote machine and used the time efficiently. Some time later, though, I got confused in the mist of screen sessions and was browsing my home directory on the laptop, thinking I was elsewhere (my home directories are mostly synchronised), when I noticed a directory in similar condition as the above. Oh shit. Imagine my pain and fear as I first thought my remote machine was also dying, imagine the sigh when I found out I was on the local filesystem, and imagine the shock when I realised that /home was also affected by the XFS breakage... A glance around /var confirmed that the XFS breakage was actually spreading and had now affected three filesystems on this machine. Fortunately, by that time, I had copied everything to the external drive, and decided to put my laptop and myself to sleep. I woke the next morning to the task of reinstalling the thing and decided to be optimistic about it. After all, a reinstall would mean I could finally try partman-crypto and encrypt my laptop's data to protect against leaking sensitive stuff in the case of loss or theft of my laptop. The installation was not as painless as I had hoped, but that was mainly because I ran into a known problem with the graphical installer and partman-crypto, which does not allow to set up volumes with random encryption keys (e.g. swap; see the forthcoming announcement for the beta3 release of the installer), and a bunch of smaller bugs. I had to restart the installation with the traditional frontend to get what I wanted, but other than that, I was very impressed with what our installer development team has accomplished! And a special round of gratitude to Frans Pop for not losing his patience while helping me on several occasions throughout the process. Now, 24 hours after the incident, I am back to normal with a fresh laptop and no data lost (except for one directory which I pulled from a mirrored remote machine; it had no local modifications (so why did XFS screw it up anyway?)). The fonts are all jaggy, so there's something I have to figure out. All things considered, I am sad to have lost 24 hours, but I can also relax more now, without fear of further XFS breakage or loss of private data. Update: Oh, and despite this, I did choose ext3 for all my laptop's filesystems. JFS was really cumbersome and slow last time I tried it, and I surely would not touch RazorFS after experiencing serious data loss on numerous occasions. Update: Two responses so far. Full ack for Julien (except for him laughing at me), Ingo's post warrants a reply though. First, ext3 is also journaled, and if you're about to say "yeah, but it's a hack on top of ext2, well... ext2 is damn mature, and journaling isn't really rocket science, so that "hack" isn't going to be too complicated. In fact, I like the idea of journaling being an option rather than a built-in feature. Second, of course you're supposed to keep backups. But since you keep backups, my top requirement of a filesystem is not "how to get the data back", but "how to ensure it does not break. If it breaks, I can reinstall and restore from backup, but that's a certain amount of time lost. If it doesn't break, well, that's like stealing a little something back from death then, isn't it? Third, I do follow the linux-xfs mailing list, but so what? I did not have write cache enabled, and I was running the 2.6.17.7 kernel at the time of the mishap. Lastly, you point to "excellent tools" to recover the filesystem. I am not sure how excellent xfs_repair really is when it reports "bad magic number 0x0 on dir inode 4696727" during the run, claims to have fixed it, I mount the filesystem, unmount it, run xfs_repair again and get the same message. No filesystem is perfect, and as we know from Biella's problems (among many others, ext3 is no exception. But we did get her data off! So then it's really an open field again, crap filesystem against crap filesystem. I guess at this point it helps to know that ext3 actually follows VFS semantics, while on XFS, a completed sync() syscall does not actually mean it has written the data to disk (see e.g. #317479). And then there are bugs like #239111... ext3 it is for now. If that let's me down, I'll try JFS. If that fails and noone has actually implemented a proper filesystem, I might have a go myself. Haha. Update: Alceste Scalas adds:

Ingo is right when he says that every filesystem has bugs --- but bugs apart, the design of Ext3 (i.e. its physical-block journaling) makes it a far more reliable choice for desktop and laptop PCs, expecially for people without an UPS. An Ext3 filesystem could only crash because of a bug or an hardware failure, while an XFS filesystem can be trashed even without bugs or hardware failures, due to the unavoidable consequences of a power loss on PC-class hardware.

He also alerted me to this mailing list post, which compares data=ordered journaling of ext3 (which almost noone does for performance reasons) with XFS and RazorFS. Update: You may also be interested in this post. Update: Otavio Salvador points me to this FAQ entry, which SGI must have added very lately. It explains how to deal with the directory corruption that was part of my problem. I guess I would have liked to know earlier, but I consider the outcome with dm-crypt + ext3 a win anyhow.

I've successfully avoided microwave food for at least as long as I've cooked my own meals, which makes it now almost 10 years. My rationaly has always been that I deserve better than some brute-force heated, tasteless crap. Today I broke with my principles and popped in a "Chicken Tika Masala" by "Patek's Original". They've had a special at the Dunnes Store where I do my shopping, and I went for it. The pack was still on the expensive side, almost 4 , but oh well... ... and I am excited by it. It actually tastes really good. I am not sure whether that's good or bad for my future. PS: oh, and when they write "spicy", they mean spicy.

If I remember correctly, then at 12 years of age, I was making my first attempts at chasing girls (mostly to play pranks on them), causing a hell of a lot of trouble in school, smoking and acquiring my first alcohol experiences, hacking on my Mac SE/30, and taking my wonderful mountain bike to places noone had ever gone before. Unfortunately, I did not pay any serious attention to the art of percussion back then, or else I would have maybe... ah, nevermind, no way I would have ever done this (videodownloader.net has links to other formats -- and there is a Firefox extension). I miss my drum kit, which is at this moment a mere 1872.45 kilometres away (or so says map24.com anyway, which is crap and does not give me a way to link to the route from Limerick to Munich, it seems).

I sometimes wonder how amazingly b0rken the X window system is, and most of the time it's me swearing about cut-and-paste. Or is it by design that I open a terminal, select some text, close the terminal, and find the paste buffer to be empty? I can't imagine... I'll leave it at that for now and won't bitch about Firefox, OpenOffice.org, and most GTK apps with respect to pasting what I previously copied by selecting stuff with the mouse. Update: many people responded, pointing me to clipboard managers that work around the basic design limitation, because X "cut-n-paste" is actually more of a "drag-n-drop" implementation. Chris Siebenmann explained it best:

The short answer is 'by (bad) design'. The X people were reacting to the problem of not wanting to shove (potentially) large amounts of data into the X server and require the X server to hold onto things, so they designed cut & paste (the second time around) so that the program that created the cut had to hold onto the data.

The longer answer, with some history, is at

http://utcc.utoronto.ca/~cks/space/blog/sysadmin/XCutAndPasteHistory

In retrospect it's probably an over-engineered solution, and it does have the big flaw you ran into. (Arguably everything from that era of X is very complicated and overengineered.)

Other than that, I've also been pointed to the FreeDesktop ClipBoard standard. Somehow I doubt this will have any influence on Fluxbox in the short run.

If you have an XFS filesystem on hardware without barriers (or with broken barriers as was the case on my X40's Hitachi drive) and with write-caching enabled, disable the latter NOW (the following will work for PATA drives, and SATA if you have a [rd]ecent kernel):

apt-get install hdparm
df -T   egrep ' xfs '   sed -rne 's,(/dev/[sh]d[[:alpha:]]).*,\1,p' \
    sort -u   while read disk; do cat <<_eof
$disk  
  write_cache = off
 
_eof
done >> /etc/hdparm.conf
/etc/init.d/hdparm start

Also see http://oss.sgi.com/projects/xfs/faq.html#wcache . Anyway, damage mitigated. xfs_repair -d /dev/md1 feels funny when /dev/md1 is the root partition of your laptop and you're about to move to Ireland for six weeks, fully dependent on that machine. But yeah, XFS on a laptop is asking for trouble anyway. Call me masochist.