Edited to add: Sorry, it turns out that ext4 does now have a
heuristic to deal with this case. However, it seems that
btrfs doesn't. My point remains, though - designing a filesystem in such a way that a useful behaviour is impossible is unhelpful for the vast majority of application writers, even if it does improve performance in some other use cases.
Further edit: It's mentioned
in the comments that btrfs will force a flush of data when a rename is performed as of 2.6.30. That prevents the data loss, but it means that we're still stuck with disk access when we'd be happy with that being batched up for the future. It feels like we're optimising for the wrong case here.
Original text:
There's been a
certain amount of
discussion about behavioural differences between ext3 and ext4[1], most notably due to ext4's increased window of opportunity for files to end up empty due to both a longer commit window and delayed allocation of blocks in order to obtain a more pleasing on-disk layout. The applications that failed hardest were doing open("foo", O_TRUNC), write(), close() and then being surprised when they got zero length files back after a crash. That's fine. That was always stupid. Asking the filesystem to truncate a file and then writing to it is an invitation to failure - there's clearly no way for it to intuit the correct answer here. In the end this has been avoided by avoiding delayed allocation when writing to a file that's just been truncated, so everything's fine.
However, there's another case that also breaks. A common way of saving files is to open("foo.tmp"), write(), close() and then rename("foo.tmp", "foo"). The mindset here is that a crash will either result in foo.tmp being zero length, foo still being the original file or foo being your new data. The important aspect of this is that the desired behaviour of this code is that foo will contain either the original data or the new data. You may suffer data loss, but you won't suffer
complete data loss - the application state will be consistent.
When used with its (default) data=ordered journal option, ext3 provided these semantics. ext4 doesn't. Instead, if you want to ensure that your data doesn't get trampled, it's necessary to fsync() before closing in order to make sure it hits disk. Otherwise the rename can occur before the data is written, and you're back to a zero length file. ext4 doesn't make guarantees about whether data will be flushed before metadata is written.
Now, POSIX says this is fine, so any application that expected this behaviour is already broken by definition. But this is rules lawyering. POSIX says that many things that are not useful are fine, but doesn't exist for the pleasure of sadistic OS implementors. POSIX exists to allow application writers to write useful applications. If you interpret POSIX in such a way that gains you some benefit but shafts a large number of application writers then people are going to be reluctant to use your code. You're no longer a general purpose filesystem - you're a filesystem that's only suitable for people who write code with the expectation that their OS developers are actively trying to fuck them over. I'm sure Oracle deals with this case fine, but I also suspect that most people who work on writing Oracle on a daily basis have very, very unfulfilling lives.
But anyway. We can go and fix every single piece of software that saves files to make sure that it fsync()s, and we can avoid this problem. We can probably even do it fairly quickly, thanks to us having the source code to all of it. A lot of this code lives in libraries and can be fixed up without needing to touch every application. It's not the end of the world.
So why do I still think it's a bad idea?
It's simple. open(),write(),close(),rename() and open(),write(),fsync(),close(),rename(), are
not semantically equivalent. One is "give me either the original data or the new data"[2]. The other is "always give me the new data". This is an important distinction. fsync() means that we've sent the data to the disk[3]. And, in general, that means that we've had to spin the disk up.
So, on the one hand, we're trying to use things like relatime to batch data to reduce the amount of time a disk has to be spun up. And on the other hand, we're moving to filesystems that require us to generate
more io in order to guarantee that our data hits disk, which is a guarantee we often don't want anyway! Users will be fine with losing their most recent changes to preferences if a machine crashes. They will not be fine with losing the entirity of their preferences. Arguing that applications need to use fsync() and are otherwise broken is ignoring the important difference between these use cases. It's no longer going to be possible to spin down a disk when any software is running at all, since otherwise it's probably going to write something and then have to fsync it out of sheer paranoia that something bad will happen. And then probably fsync the directory as well, because what if someone writes an even more pathological filesystem. And the disks sit there spinning gently and chitter away as they write tiny files[4] and never spin down and the polar bears all drown in the bitter tears of application developers who are forced to drink so much to forget that they all die of acute liver failure by the age of 35 and where are we then oh yes we're screwed.
So. I said we could fix up applications fairly easily. But to do that, we need an interface that lets us do the right thing. The behaviour application writers want is one which ext4 doesn't appear to provide. Can that be fixed, please?
[1] xfs behaves like ext4 in this respect, so the obvious argument is that all our applications have been broken for years and so why are you complaining now. To which the obvious response is "Approximately anyone who ever used xfs expected their data to vanish if their machine crashed so nobody used it by default and seriously who gives a shit". xfs is a wonderful filesystem for all sorts of things, but it's lousy for desktop use for precisely this reason.
[2] Yes, ok, we've just established that it actually isn't that in the same way that GMT isn't UTC and battery refers to a collection of individual cells and so you don't usually put multiple batteries in your bike lights, but the point is that this is, for all practical intents and purposes, an unimportant distinction and not one people should have to care about in their daily lives.
[3] The disk is free to sit there bored for arbitrary periods of time before it does anything, but that's fine, because the OS is behaving correctly. Sigh.
[4] Dear filesystem writers - application developers like writing lots of tiny files, because it makes a large number of things significantly easier. This is fine because sheer filesystem performance is not high on the list of priorities of a typical application developer. The answer is not "Oh, you should all use sqlite". If the only effective way to use your filesystem is to use a database instead, then that indicates that you have not written a filesystem that is useful to typical application developers who enjoy storing things in files rather than binary blobs that end up with an entirely different set of pathological behaviours. If I wanted all my data to be in oracle then I wouldn't need a fucking filesystem in the first place, would I?
Mark Purcell reported bug
#520000 on Monday March 16th, against the gsm-utils package.
Bug #510000 was reported as of Dec 28th 2008. So, it took us 2.5 months for
10,000 bugs. This is still slower than our rate of 10,000 bugs/2 months but
more than the pace we had for 500000-510000.
Of course, in the meantime, we released lenny on Feb 14th and, therefore, the development started again and, with development, new bugs come in.
To illustrate this, one will notice that bug #515000 was reported on Feb 12th. So, it took 46 days for the first 5,000 bugs and then only 32 for the remainder 5,000. In short, we're probably back to our pace of 10,000 bugs in two months.
In case someone cares, the first bug reported after the release of Lenny is #515247, reported by Robin Kluth against the gvfs package. It followed the release of lenny by 25 minutes and 14 seconds. That bug isn't fixed yet but is already reported upstream.
Excellent summary of 
Just a quick update to let you know how the Debian bid for the Google Summer of Code is going.
We re doing great right now. Our application is sent out, you can consult the public part 
We now have 12 applications on 
With any luck, someone has seen this problem before and can fix it easily. I don t seem to be able to find the fix by searching, but the search terms feel like they re either too general or too specific.
I was trying to install some perl software on a debian lenny system that was upgraded from etch (and previously from sarge). After installing the dependencies, I started getting errors like this:-
I received a sample of QNAP's new TS-219 Turbo NAS today. The TS-119 and
TS-219 devices are an upgrade to QNAP's TS-109 and TS-209 devices and were
announced earlier this month at CeBIT. The new TS-219 uses the same robust
case as the TS-209 but offers much more performance:
Yesterday was my birthday, and it was full of cookies, pies, magical pixie dust and things made entirely of sugar and spice!
The remainder of the day was spent re-installing Debian Lenny upon my EEE PC - Somehow I managed to completely screw the system.
Because the EEE PC is one of those ultra-portable machines I mostly used it when I was travelling, or outdoors. That mean I was generally receiving poor connectivity and the system packages weren't up to date.
While I was in bed I figured I'd dist-upgrade it to the recently released Lenny. Unfortunately I started the dist-upgrade inside X.org, once I realised this I figured I'd cancel the operation via Ctrl-c.
Bad news everbody: I think I was unlucky enough to interrupt an upgrade of libc, or something equally critical. Every single application gave segfaults afterward.
I had two open root terminals and I could navigate around via cd .., and "echo *", but all other commands such as sudo, dpkg, strace just gave segfaults. (Even static commands gave errors - so it might have been the dynamic loader that was borked, I admit I didn't look too closely.)
I figured reinstalling would be a good solution since the machine has a 4Gb root partition and /home was stored on a separate 16Gb volume. Unfortunately I managed to misjudge the installer's partitioning step and nuke the partition table on the external volume so I ended up losing the whole system.
Happily reinstallation was a breeze as my home network is setup to allow 
A cautionary tale: Verizon's DSL salespeople are completely incompetent.
Several weeks ago, I ordered residential service. I was unable to get
the prequalification app on their web site to work for my address, so
I called in. I was told (1) I could get a modem for free, and (2) that
I didn't need to do anything for the install except plug it in --
they'd just be turning something on at the CO, perhaps even before the
date my service was scheduled to begin.
These were both, of course, pure fiction. On the install date, while at
work, I got a call from a tech who had shown up at my house to rewire
something. Luckily, I was able to rendezvous with him later in the day.
The service worked briefly and then went down. Over the next few days,
I called in, was told to wait at home for 8 hours, no one showed up, I
called in again, was told "oh, sorry, it's going to be tomorrow", waited
all day the next day, no one showed up again... a week and two actual
visits later, I finally had connectivity.
I can't really blame them for whatever the line problem was, but to
repeatedly treat my time as worthless by making me wait at home for an
entire day just in case a tech can get there, even though it's highly
unlikely they will, is just offensive.
Then, I received my first bill and... was charged for the modem. Calling
in again, I was told that the deal I was offered was for online orders
only, and even if I specifically confirmed with the sales person that he
could get me the same deal since the web site wouldn't let me complete
an order (and I already had an old, power-sucking modem anyway),
there was no way I could get a refund. So I got an RMA and planned to
go shopping for a replacement (people are selling the same model on
Craigslist for $10). The RMA came in an email which also told me that if I did
not return the equipment within X days I would be charged $100.
This is all utter bullshit. Do not use Verizon. If your only other
option is Comcast, as it is here, just steal your neighbor's wireless or
tether your 3G.
They could have left me reasonably happy with the whole botched
situation if they had provided a honest estimate of how long it
would really take to get someone out (even if it was "days"), or
given me access to the same information without having to go through
a call center of people only trained to read a script about how to
power-cycle your router or whatever. And if they actually apologized for
a saleperson convincing me to sign up by lying about what I would be
charged. But why would they? What are you going to do, switch to an ISP
that treats you like a human being and doesn't fuck with your traffic?
Ha. Ha ha ha. Good luck with that.