Many people have responded to my post on encrypted filesystems and power losses; thank you! To clear up the confusion, I decided to write a new entry rather than updating the previous one... I learnt two important things about encrypted block devices, which make me conclude that the addition of encryption does not put your filesystem at jeopardy any more than if it lived directly on the medium. First of all, AES works on blocks of size 128 bits, not 64 bytes, but that's mostly a cosmetic issue with respect to the original blog post. What's more important is that a power loss while writing out a cipher block effectively is the same as a power loss while writing out a physical block to the storage medium: the cipher block is usually smaller than the disk block, and an incompletely written disk block is considered to be invalid. So any incomplete cipher block results in an incomplete disk block, which in turn causes the filesystem to consider the block invalid and try to recover from the journal. What confused me for a long time was the use of chaining block ciphers (see the excellent Wikipedia article for more info). Essentially, block chaining adds security to encryption by making each (cipher) block depend on its predecessor. Great, I thought, so if the first block on disk is corrupted, it'll percolate all the way through the disk and render it all unusable... Well, fortunately, this is not the case... by that logic, a write to the first block would require all remaining blocks on the disk to be recalculated. It is true that cipher blocks are chained, but only within a physical block. This means that a read error on the medium may corrupt a cipher block, which then renders all following cipher blocks contained in the physical block unusable (for they cannot be decrypted anymore). But since the filesystem would already consider the entire block invalid, this does not add to problem. PS: Thanks to Peter Samuelson for (passively) leading me to the light.

As some of you may know, I am currently in Ireland, spending most of the summer at the Irish Software Engineering Research Centre to get a start on my Ph.D. research. This is the story of how I got here and how it's been so far. In the middle of July, I left my beloved city of Zurich and headed for Dublin. I commenced my stay in "the green country" with a three-day Debian workshop for the Google Europe sysops, which was intensive and yet enjoyable. I had between 4 and 10 participants, and the syllabus was to cover most everything, from APT basics to advanced concepts. That's a lot for three days, and I usually advise clients to consider a five-day or even two-week course instead, but after convincing the organiser to plan for three days rather than just two, I figured I'd be dealing with pretty good people anyway, given Google's hiring standards. I was not to be disappointed. We started the course with installation problems, because the 2.6 kernel install which I suggested could not deal with the SATA driver in the new IBM T43p notebooks which the participants were using. While one guy decided to try a 2.4 install with the IDE-sata driver, I opted to go down the debootstrap route instead, and was especially delighted -- at first -- to learn about a working netboot setup in the Google LAN. My joy vanished when I found the Ubuntu rescue image not to contain debootstrap, so I didn't have much choice but to prepare a root tarball and offer it for wget download. This worked, and after manual tweaking and making the system bootable, running into some issues after inadvertedly upgrading the entire system to the backports.org archive (which still does not offer a proper default pin), we decided to just go with testing and be done with it. I have to say that I really like courses where stuff does not work, as I believe those to be far more valuable to the participants. This time around, it was also the first time (after having taught maybe 60 or 70 courses in total) that I did not let myself get stressed out. Realising this put me in a great mood and made those days even more enjoyable. The rest of the workshop went without further complications, but judging from feedback over cigarettes, lunch, or the evening beer(s), the guys and girls still appreciated it. I have to thank Google and all participants for a great three days of hospitality and challenges, interesting discussions and fun. I really ought to reconsider my decision not to accept their offer for a sysop position... The last day saw me getting sick though, it must have been the air-conditioning. When I reached the hotel after the course, I definitely knew I had high fever and was developing a bad headache and cough, so I didn't get to enjoy a Friday night out in Dublin. Instead, I slept from 18:00 hours until the late morning, when I had to drag myself to the train station to catch the train to Limerick... which was painful, but I managed. Some time in between, I tried out nstx and managed to get a decent SSH connection to the outside world from the hotel room (just don't forget the -a -x flags!). Arriving in Limerick, I registered for the student apartment I had rented and found myself in a small room with a bed and desk, my own bathroom, a common room and kitchen I share with my room mate (whom I barely saw), including a TV, fridge, oven, microwave, and enough "little stuff" to make me conclude that this place was livable -- for 60 /week not a bad deal. I spent most of the weekend sleeping, but it was not until Tuesday that my state had returned to normal. The place also has an Ethernet jack, but that turned out to be unusable as it was (a) behind a fascist firewall, and (b) on the same network as about 40 other users, 17 of which seemed to be running peer-to-peer filesharing clients all hours of the day. In my office at the university, I found myself in the same situation as (a), but at least I managed to install OpenVPN, and dsniff and was thus ready to take back the net in my apartment. I wrote a little script which would (a) spoof ARP replies from the router for all hosts for a few minutes, (b) analyse the traffic it got to identify obvious P2P machines, and (c) proceed to poison their ARP cache every few seconds, effectively shaping their connection. I had to go this way, because tools like tcpnice (part of dsniff) do not work in a switched network very well, and neither did macof. Unfortunately, I lost the script in the recent XFS fiasco, but the major P2P offenders seem to have left anyway. Thanks to OpenVPN, I also got VoIP working and am happily tunneling through the university's proxy (which otherwise would require me to use the SOCKS4 proxy protocol with proxychains, which would be a nuisance). I really don't understand why a university blocks outgoing traffic. Oh well... It's been three weeks since I've come to Limerick, and my feelings are mixed. I've been spending most time in my (temporary) office (pretending to be) doing work, I have not really acquainted myself with the food here, the university campus is filled with hundreds of 14 year old summer school students but nobody of my age, and, well, research is coming along slower than I had hoped. I guess that's normal in a sense, but I also know for sure that I have a lot of work left to do on my discipline. I am a master of procrastination, you see, and Debian is a dangerous drug, a black hole that sucks in your time while it excites your synapses and splashes endorphines all around. But I've not lost my hope at the very least... Thanks for caring.

Erich complains: "If the file is damaged, just fill it up with zeroes?" (in response to my recent post on filesystem problems). Update: I rewrote the second sentence of the next paragraph (up until and including the footnote) to get things right. Thanks to Russell Cattelan for providing the info. This must be the most misunderstood feature of XFS. What happens is that XFS logs all metadata changes to the journal, except for the inode size, which gets flushed to disk immediately for performance reasons [*]. At this point, the file will actually be a sparse file, which is nothing more than a file whose metadata lists a file as being of a size different than it currently is (I realise the "sparse" does not really apply when the file is "overfull", i.e. when the metadata lists it as smaller than it really is, but I am lacking a good word for that). The disk extents get allocated only when the data actually hits the disk (that's XFS's famous delayed allocation mechanism). If the power fails before the data was flushed to disk and the journal entry cleared, XFS will serve zeroes, rather than the potentially random or sensitive data that is actually on disk. This is a good thing.

[*]	since almost every write() changes the file size, it would be a massive performance hit if every size change was logged. However, XFS actually violates its own journaling rules by doing this.

You can run into more or less the same problem with any journaling filesystem; the others just don't serve zeroes. Instead, they give you the data that's physically on the medium. Imagine the situation when the corrupt /etc/motd suddenly becomes a window to your previous /etc/shadow contents... I really prefer how XFS handles that. Sometimes you do get the old data back with the other filesystems, but this is because the filesystems may reuse the blocks of the old file. So it's a trade-off, and your choice between security and, uh, convenience. The only way to protect against this is to use "physical-block journaling" (as opposed to "logical journaling"), which is only supported by ext3 as far as I know (option data=ordered), at a massive performance loss. See this mailing list post by Theodore Ts'o for more info. Thanks to Alceste Scalas for sharing it with me. PS: Thanks to Wessel Dankers and Eric Sandeen for their help on #xfs. Update: Again thanks to Russell Cattelan, I just received word that code has been added to the 2.6.17 kernel which will flush an inode at close if it was truncated at some previous point. This should take care of many of the zeroed-file issues. Update: RazorFS also supports data=ordered. Thanks to Hans Kratz for pointing this out. He also adds some more information, parts of which I've written about previously:

One of the major problems of all journalled filesystems especially on notebooks/desktop machines leading to filesystem corruption is the harddisk writeback cache (enabled by default). With the writeback cache some writes are not immediately written to disk Now if a crash happens some of the data may not have hit the disk. The journalling filesystems however need a way to rely that the certain writes have hit the disk to ensure filesystem consistency. There are two solutons: Enable write barrier support in the filesystems (avalable for ext3 since around 2.6.8, mount option: barrier=1, available for the other filesystems as well now probably) or disabling the write cache altogether with hdparm/sdparm.

XFS supports barriers as well, but unfortunately not yet on non-physical media, like RAID or dm-crypt. ext3 seems to work okay on both. Enable your barriers today!