Edited to add: Sorry, it turns out that ext4 does now have a heuristic to deal with this case. However, it seems that btrfs doesn't. My point remains, though - designing a filesystem in such a way that a useful behaviour is impossible is unhelpful for the vast majority of application writers, even if it does improve performance in some other use cases.

Further edit: It's mentioned in the comments that btrfs will force a flush of data when a rename is performed as of 2.6.30. That prevents the data loss, but it means that we're still stuck with disk access when we'd be happy with that being batched up for the future. It feels like we're optimising for the wrong case here.

Original text:

There's been a certain amount of discussion about behavioural differences between ext3 and ext4[1], most notably due to ext4's increased window of opportunity for files to end up empty due to both a longer commit window and delayed allocation of blocks in order to obtain a more pleasing on-disk layout. The applications that failed hardest were doing open("foo", O_TRUNC), write(), close() and then being surprised when they got zero length files back after a crash. That's fine. That was always stupid. Asking the filesystem to truncate a file and then writing to it is an invitation to failure - there's clearly no way for it to intuit the correct answer here. In the end this has been avoided by avoiding delayed allocation when writing to a file that's just been truncated, so everything's fine.

However, there's another case that also breaks. A common way of saving files is to open("foo.tmp"), write(), close() and then rename("foo.tmp", "foo"). The mindset here is that a crash will either result in foo.tmp being zero length, foo still being the original file or foo being your new data. The important aspect of this is that the desired behaviour of this code is that foo will contain either the original data or the new data. You may suffer data loss, but you won't suffer complete data loss - the application state will be consistent.

When used with its (default) data=ordered journal option, ext3 provided these semantics. ext4 doesn't. Instead, if you want to ensure that your data doesn't get trampled, it's necessary to fsync() before closing in order to make sure it hits disk. Otherwise the rename can occur before the data is written, and you're back to a zero length file. ext4 doesn't make guarantees about whether data will be flushed before metadata is written.

Now, POSIX says this is fine, so any application that expected this behaviour is already broken by definition. But this is rules lawyering. POSIX says that many things that are not useful are fine, but doesn't exist for the pleasure of sadistic OS implementors. POSIX exists to allow application writers to write useful applications. If you interpret POSIX in such a way that gains you some benefit but shafts a large number of application writers then people are going to be reluctant to use your code. You're no longer a general purpose filesystem - you're a filesystem that's only suitable for people who write code with the expectation that their OS developers are actively trying to fuck them over. I'm sure Oracle deals with this case fine, but I also suspect that most people who work on writing Oracle on a daily basis have very, very unfulfilling lives.

But anyway. We can go and fix every single piece of software that saves files to make sure that it fsync()s, and we can avoid this problem. We can probably even do it fairly quickly, thanks to us having the source code to all of it. A lot of this code lives in libraries and can be fixed up without needing to touch every application. It's not the end of the world.

So why do I still think it's a bad idea?

It's simple. open(),write(),close(),rename() and open(),write(),fsync(),close(),rename(), are not semantically equivalent. One is "give me either the original data or the new data"[2]. The other is "always give me the new data". This is an important distinction. fsync() means that we've sent the data to the disk[3]. And, in general, that means that we've had to spin the disk up.

So, on the one hand, we're trying to use things like relatime to batch data to reduce the amount of time a disk has to be spun up. And on the other hand, we're moving to filesystems that require us to generate more io in order to guarantee that our data hits disk, which is a guarantee we often don't want anyway! Users will be fine with losing their most recent changes to preferences if a machine crashes. They will not be fine with losing the entirity of their preferences. Arguing that applications need to use fsync() and are otherwise broken is ignoring the important difference between these use cases. It's no longer going to be possible to spin down a disk when any software is running at all, since otherwise it's probably going to write something and then have to fsync it out of sheer paranoia that something bad will happen. And then probably fsync the directory as well, because what if someone writes an even more pathological filesystem. And the disks sit there spinning gently and chitter away as they write tiny files[4] and never spin down and the polar bears all drown in the bitter tears of application developers who are forced to drink so much to forget that they all die of acute liver failure by the age of 35 and where are we then oh yes we're screwed.

So. I said we could fix up applications fairly easily. But to do that, we need an interface that lets us do the right thing. The behaviour application writers want is one which ext4 doesn't appear to provide. Can that be fixed, please?

[1] xfs behaves like ext4 in this respect, so the obvious argument is that all our applications have been broken for years and so why are you complaining now. To which the obvious response is "Approximately anyone who ever used xfs expected their data to vanish if their machine crashed so nobody used it by default and seriously who gives a shit". xfs is a wonderful filesystem for all sorts of things, but it's lousy for desktop use for precisely this reason.

[2] Yes, ok, we've just established that it actually isn't that in the same way that GMT isn't UTC and battery refers to a collection of individual cells and so you don't usually put multiple batteries in your bike lights, but the point is that this is, for all practical intents and purposes, an unimportant distinction and not one people should have to care about in their daily lives.

[3] The disk is free to sit there bored for arbitrary periods of time before it does anything, but that's fine, because the OS is behaving correctly. Sigh.

[4] Dear filesystem writers - application developers like writing lots of tiny files, because it makes a large number of things significantly easier. This is fine because sheer filesystem performance is not high on the list of priorities of a typical application developer. The answer is not "Oh, you should all use sqlite". If the only effective way to use your filesystem is to use a database instead, then that indicates that you have not written a filesystem that is useful to typical application developers who enjoy storing things in files rather than binary blobs that end up with an entirely different set of pathological behaviours. If I wanted all my data to be in oracle then I wouldn't need a fucking filesystem in the first place, would I?

The following information is based on Anomos website. Funnily, when you go to that website, the default page is a blog that doesn t answer the What s anomos? question. The answer to that question is available on the About page, on this blog post, and those slides. So, Anomos seems like a good answer to the it s to easy to determine who is donwloading a file with BitTorrent problem. It:

uses a mix network (like Tor) to transfer all the content.

encrypts the file being transferred.

It is based on BitTorrent for performance, however, performance might be seriously affected by the mix network (which probably don t resist too well to heavy load since most internet connections have asymmetric upload/download bandwidths). A strange point is that it uses a central tracker. I m not sure why the central tracker couldn t be dropped, to simply use a DHT instead. (And traffic to the DHT could go through the mix network as well).

I ve been very impressed several times in the past few months when I ve discovered awesome new top-like utilities. I m probably being slow on the uptake and everyone else but me knows about these, but in case its not just me thats been stuck in the 70s:

htop

A much-needed refresh of oldschool top, this still works on your beloved console but gives you visual bar-graphs of CPU, RAM and swap, lets you scroll through the processes and deliver signals/renicing without having to copy the PID off the moving target. Its like the future!

iftop

One of those things I use so often now I have no idea how I even survived without it. Why is this server lagging, who s hogging the wireless/DSL, which VM is chewing all of the upstream bandwidth? iftop shows you at a glance how much traffic is being used by which host pairs on a given interface, and you can toggle port numbers on and off with simple key-presses. Absolutely indispensable.

iotop

Does this box feel slow to anyone else? Is it swapping, or is it the database server chewing all the IO? Why does my drive keep seeking? It s amazing top for IO bandwidth usage!

A passing mention is deserved for apachetop too, which is pretty neat, but when a server is being hammered it s not something I found too hard to get a feel for just by tailing the log for a while, so it s not been as life-changing as the others. Maybe that just means my servers don t see enough traffic.

I ve had smart quotes envy for a long time now. I particularly like them in contractions, and I think they make the text more pleasant to read. Of course, I ve been using them on my web pages and blog for some time now, since they are well-supported by browsers. TeX/LaTeX deals with them fine as well, even by just typing ASCII in the source document. Mostly only mail was missing. I ve delayed using smart quotes in mail for a long time, since they imply a lot of UTF-8, and there was/is still quite a lot of software and setups not ready for it. But we re in the 10th year of the 21st century already, and I was getting envious of the increasing number of people that are using them for e-mail. So I ve recently switched to smart quotes in my outgoing mail. I realize this goes against the Be strict in what you emit and liberal in what you accept principle, which is a fine one, but I must confess aesthetics have won the battle for me in this case. I ve already received one complaint from a person using broken software, asking me to send more comprehensible and correct messages , and referring to UTF-8 as WTF-8. I wonder how many people to whom my mails are targetted at will have trouble reading them now. I don t have comments enabled in this blog yet I m afraid, but feel free to participate in the poll.

I received a sample of QNAP's new TS-219 Turbo NAS today. The TS-119 and TS-219 devices are an upgrade to QNAP's TS-109 and TS-209 devices and were announced earlier this month at CeBIT. The new TS-219 uses the same robust case as the TS-209 but offers much more performance:

• Marvell Kirkwood 6281 at 1.2 GHz (instead of Orion 5182 at 500 MHz)
• 512 MB RAM (instead of 128 or 256 MB)
• 16 MB flash (instead of 8 MB)

The numbers I've seen about the Kirkwood CPU suggest that the TS-219 will indeed give a very significant performance boost. So far, the TS-219 looks very nice. My only complaint is that QNAP didn't export the second Ethernet port the Kirkwood chip offers, but I suspect this is because they wanted to stay as close to the original design of the TS-209 as possible. In any case, the TS-219 is a nice machine and I look forward to porting Debian to it. As a first step, this means getting the mainline kernel to run on the device and adding a kernel for Kirkwood to the archive (the latter is needed for a number of other devices based on Kirkwood anyway). Hopefully, the remaining porting work and integration into the Debian installer should be fairly straightforward given that the TS-209 is already supported in Debian. I'm sure I'll find out as I dig into the details... I'll give periodic updates of my progress. In the meantime, I've created an image gallery of the TS-219.

A recent Ubuntu bug has gotten slashdotted, and has started raising a lot of questions about the safety of using ext4. I ve actually been meaning to blog about this for a week or so, but between a bout of the stomach flu and a huge todo list at work, I simply haven t had the time. The essential problem is that ext4 implements something called delayed allocation. Delayed allocation isn t new to Linux; xfs has had delayed allocation for years. Pretty much all modern file systems have delayed allocation, according to the Wikipedia Allocate-on-flush article, this includes HFS+, Reiser4, and ZFS; btrfs has this property as well. Delayed allocation is a major win for performance, both because it allows writes to be streamed more efficiently to disk, and because it can reduce file fragmentation so that later on they can be read more efficiently from disk. This sounds like a good thing, right? It is, except for badly written applications that don t use fsync() or fdatasync(). Application writers had gotten lazy, because ext3 by default has a commit interval of 5 seconds, and and uses a journalling mode called data=ordered. What does this mean? The journalling mode data=ordered means that before the commit takes place, any data blocks are
associated with inodes that are about to be committed in that transaction will be forced out to disk. This is primarily done for security reasons; if this is not done (which would be the case if the disk is mounted with the mount option data=writeback), then any newly allocated blocks might still contain previous data belonging to some other file or user, and after a crash, accessing that file might result in a user seeing unitialized data that had previously belonged to another user (say, their e-mail or their p0rn), which would be a Bad Thing from a security perspective. However, this had the side effect of essentially guaranteeing that anything that had been written was guaranteed to be on disk after 5 seconds. (This is somewhat modified if you are running on batteries
and have enabled laptop mode, but we ll ignore that for the purposes of this discussion.) Since ext3 became the dominant filesystem for Linux, application writers and users have started depending on this, and so they become shocked and angry when their system locks up and they lose data even though POSIX never really made any such guarantee. This become especially noticeable on Ubuntu, which uses many proprietary, binary-only drivers, which caused some Ubuntu systems to become highly unreliable, especially for Alpha releases of Ubuntu Jaunty, with the net result that some Ubuntu users have become used to their machines regularly crashing. (I use bleeding edge kernels, and I don t see the kind of unreliability that apparently at least some Ubuntu users are seeing, which came as quite a surprise to me.) So what are the solutions to this? One thing is that the applications could simply be rewritten to properly use fsync() and fdatasync(). This is what is required by POSIX, if you want to be sure that data has gotten written to stable storage. Some folks have resisted this suggestions, on two grounds; first, that it s too hard to fix all of the applications out there, and second, that fsync() is too slow. This perception that fsync() is too slow was most recently caused by a problem with Firefox 3.0. As Mike Shaver put it:

On some rather common Linux configurations, especially using the ext3 filesystem in the data=ordered mode, calling fsync doesn t just flush out the data for the file it s called on, but rather on all the buffered data for that filesystem.

Fundamentally, the problem is caused by data=ordered mode. This problem can be avoided by mounting the filesystem using data=writeback or by using a filesystem that supports delayed allocation such as ext4. This is because if you have a small sqllite database which you are fsync(), and in another process you are writing a large 2 megabyte file, the 2 megabyte file won t be be allocated right away, and so the fsync operation will not force the dirty blocks of that 2 megabyte file to disk; since the blocks haven t been allocated yet, there is no security issue to worry about with the previous contents of newly allocated blocks if the system were to crash at that point. Another solution is a set of patches to ext4 that has been queued for 2.6.30 merge window. These three patches (with git id s bf1b69c0, f32b730a, and 8411e347) will cause a file to have any delayed allocation blocks to be allocated immediately when a file is replaced. This gets done for files which were truncated using ftruncate() or opened via O_TRUNC when the file is closed, and when a file is renamed on top of an existing file. This solves the most annoying set of problems where an existing file gets rewritten, and thanks to the delayed allocation semantics, that existing file gets replaced with a zero-length file. However, it will not solve the problem for newly created files, of course, which would have delayed allocation semantics. Yet another solution would be to mount ext4 volumes with the nodelalloc mount option. This will cause a significant performance hit, but apparently some Ubuntu users are happy using proprietary Nvidia drivers, even if it means that when they are done playing World of Goo, quitting the game causes the system to hang and they must hard-reset the system. For those users, it may be that nodelalloc is the right solution for now personally, I would consider that kind of system instability to be completely unacceptable, but I guess gamers have very different priorities than I do. A final solution which might not be that hard to implement would be a new mount option, data=alloc-on-commit. This would work much like data=ordered, with the additional constraint that all blocks that had delayed allocation would be allocated and forced out to disk before a commit takes place. This would probably give slightly better performance compared to mounting with nodelalloc, but it shares many of the disadvantages of nodelalloc, including making fsync() to be potentially very slow because it would force all dirty blocks to be forced out to disk once again. What s the best path forward? For now, what I would recommend to Ubuntu gamers whose systems crash all the time and who want to use ext4, to use the nodelalloc mount option. I haven t quantified what the performance penalty will be for this mode of operation, but the performance will be better than ext3, and at least this way they won t have to worry about files getting lost as a result of delayed allocation. Long term, application writers who are worried about files getting lost on an unclena shutdown really should use fsync. Modern filesystems are all going to be using delayed allocation, because of its inherent performance benefits, and whether you think the future belongs to ZFS, or btrfs, or XFS, or ext4 all of these filesystems used delayed allocation. What do you think? Do you think all of these filesystems have gotten things wrong, and delayed allocation is evil? Should I try to implement a data=alloc-on-commit mount option for ext4? Should we try to fix applications to properly use fsync() and fdatasync()? Related posts (automatically generated):

1. Don t fear the fsync! After reading the comments on my earlier post, Delayed allocation...
2. SSD s, Journaling, and noatime/relatime On occasion, you will see the advice that the ext3...
3. Ext4 is now the primary filesystem on my laptop Over the weekend, I converted my laptop to use the...

If you code for long enough on x86-64, you'll eventually hit an error such as: Here's a little example that might help you figure out what you've done wrong. Consider the following code: In case it's not clear, that would look something like: Let's build that code, and see what it looks like We can see that the mov instruction has only allocated 4 bytes (00 00 00 00) for the linker to put in the address of foovar. If we check the relocations: The R_X86_64_32S relocation is indeed only a 32-bit relocation. Now we can tickle this error. Consider the following linker script, which puts the foo section about 5 gigabytes away from the code. This now means that we can not fit the address of foovar inside the space allocated by the relocation. When we try it: What this means is that the full 64-bit address of foovar, which now lives somewhere above 5 gigabytes, can't be represented within the 32-bit space allocated for it. For code optimisation purposes, the default immediate size to the mov instructions is a 32-bit value. This makes sense because, for the most part, programs can happily live within a 32-bit address space, and people don't do things like keep their data so far away from their code it requires more than a 32-bit address to represent it. Defaulting to using 32-bit immediates therefore cuts the code size considerably, because you don't have to make room for a possible 64-bit immediate for every mov. So, if you want to really move a full 64-bit immediate into a register, you want the movabs instruction. Try it out with the code above - with movabs you should get a R_X86_64_64 relocation and 64-bits worth of room to patch up the address, too. If you're seeing this and you're not hand-coding, you probably want to check out the -mmodel argument to gcc.

Samba.grep.be, my main server (which occasionally also hosts a domU for p2) was upgraded to lenny this monday evening. I wish I could say there were no issues, but unfortunately that's not true. At first, everything seemed to run smoothly; of course I did have to recompile a few locally written programs to run against some SONAME changes (programs that are too trivial to package, really, and too ugly too), but other than that, everything was fine. It went wrong sometime during the night. Apparently something didn't like the new kernel or the new xen, because everything started to lock up. Since I only noticed this when I was at a customer the next day, I couldn't fix it until I got home again in the evening (meaning, downtime for a full day); and unfortunately, the exact failure mode meant that not much was written to the log -- certainly not enough to figure out what went wrong:

Mar 10 04:04:41 samba kernel: [ 7033.011523] INFO: task cron:21006 blocked for more than 120 seconds.
Mar 10 06:43:46 samba kernel: [ 7033.011614] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 17:35:04 samba kernel: [ 7033.011701] cron          D 7fffffffffffffff     0 21006   4147

After that, nothing before the reboot, which happened at around 21:00. Take good note of the timings on those messages, and the apparent discrepancies between the syslog timestamp and the kernel timestamp. 7033 seconds after boot was around 3:30 AM... When I arrived at the physical server, this kind of message was rolling over the console several times a second, with just the name and PID of the blocking process changed. And with several more lines following the message, of course. As a result, the system was not responsive at all. I didn't file a bug since, frankly, there's so little information that I wouldn't even know what to file a bug against (the kernel? Xen? Some rogue process that runs as root and changes settings which cause a driver to block? I wouldn't know). Instead, all I could do was to reboot the server and hope for the best. Not that I like that kind of situation. Fortunately, the problem didn't reproduce itself today; here's for hoping that it never will.

The recent problem of the Portuguese translation of GCompris shows how important a good translation is, especially as far as free software are concerned.
I have not been able to find any news in English relating this story, and think it is essential to spread it so that people better appreciate the role translators play in the free software development process. The Portuguese Magalh es Project lead to provide 300,000 netbooks to Portuguese children aged of 6 to 10, for a fee varying from 0 to 50 , depending on the earnings of the family. These netbooks have a dual boot, with Windows and Linux (a distribution based on Mandriva called Caixa M gica - Magic Box in English).

The problem has begun when some opponents to this project pointed out that the Portuguese translation of GCompris, a free educational software for children from 2 to 10, contained some grammar, syntax and spelling faults, and did not reach the level of quality expected to be distributed to Portuguese children. This story was even published by some major Portuguese newspaper, which is a very bad publicity not only for GCompris, but also for free software in general. Caixa M gica Software have also published a press release explaining the situation. The Portuguese Government have even published a document explaining how to uninstall GCompris, whereas work for correcting the translation was running, and automatic updates being set up on all Magalh es netbooks. The GNOME Portuguese translation team have made some hard work during the week-end to review and correct the existing translation, following to a call for help raised by GCompris lead developer. The new version released yesterday already include an updated Portuguese translation. I have noticed that this problem was already reported on launchpad 2 years ago, but people got lost as the translation was not made through Rosetta (a closed source platform for open source application translation). This should have been reported upstream or at least to the main translator! I take the occasion to thank all translators, l10n and i18n coordinators of the various free software projects I enjoy using in French!

It took us a couple of weeks to organise, but we had a small Lenny release party in Cambridge last weekend. We had the usual crowd of Cambridge folks, plus Noodles and codehelp. Jason Clifford from UKFSN even threw some cash our way to help cover the costs - Thanks Jason! :-) We started at the Regal pub in town, then headed back to my place and drank until late. Quite a number of the revellers also bought some of our shiny new Lenny release T-shirts! If you'd like one, look at the details here and mail me! Update: Fixed the URL to the T-shirt photo. Doh!

If you are managing a Free Software project, you may eventually be confronted with a security vulnerability. Normally bugs in the Open Source world are discussed in a transparent way on public forums. In the case of security bugs however, there are benefits to temporarily withholding these details from the public.

Some people describe this approach as responsible disclosure. It boils down to this:

if the vulnerability is not publicly known, warn the vendors first and give them some time to fix it before making the details public.

If the vulnerability is already public knowledge, then you can focus on fixing it as soon as possible and maybe let reporters know about a better way to report their findings.
Sample Security PolicyHere's the procedure we now follow in the Mahara project for security bugs we found ourselves or which have been privately disclosed to us through security@mahara.org:

1. Figure out the extent of the problem: which versions of Mahara are affected by this problem?
2. Fix the problem on all supported branches of the project in a private source control repository.
3. Share the vulnerability information with vendor-sec and request a CVE identifier.
   
4. Prepare release tarballs and packages.
5. Draft a security advisory for the Security forum.
6. Wait until the embargo date to push the release out.
   

Emails to vendor-sec should include:

• a description of the problem
• any proof of concept code demonstrating the vulnerability
  
• the list of affected versions
• any patches you have prepared
• a proposed embargo date

Benefits of properly handling security bugs
First and foremost, it protects end-users by giving them a chance to download and install fixed versions of your software before widespread exploitation of the security flaws.

Secondly, it may increase your project credibility. While you are admitting that your software has flaws, you are also demonstrating that your project is committed to dealing with the most serious ones promptly and in a responsible manner.

Finally, sharing security flaws and having your fixes reviewed by a select group of experts may reveal extra vulnerabilities you missed while preparing your patches. This gives you an opportunity to improve your fixes before they are released and to avoid having to issue yet another security advisory a few days later.

Big thanks to Steffen Joeris for his help in shaping the Mahara policy!

A cautionary tale: Verizon's DSL salespeople are completely incompetent. Several weeks ago, I ordered residential service. I was unable to get the prequalification app on their web site to work for my address, so I called in. I was told (1) I could get a modem for free, and (2) that I didn't need to do anything for the install except plug it in -- they'd just be turning something on at the CO, perhaps even before the date my service was scheduled to begin. These were both, of course, pure fiction. On the install date, while at work, I got a call from a tech who had shown up at my house to rewire something. Luckily, I was able to rendezvous with him later in the day. The service worked briefly and then went down. Over the next few days, I called in, was told to wait at home for 8 hours, no one showed up, I called in again, was told "oh, sorry, it's going to be tomorrow", waited all day the next day, no one showed up again... a week and two actual visits later, I finally had connectivity. I can't really blame them for whatever the line problem was, but to repeatedly treat my time as worthless by making me wait at home for an entire day just in case a tech can get there, even though it's highly unlikely they will, is just offensive. Then, I received my first bill and... was charged for the modem. Calling in again, I was told that the deal I was offered was for online orders only, and even if I specifically confirmed with the sales person that he could get me the same deal since the web site wouldn't let me complete an order (and I already had an old, power-sucking modem anyway), there was no way I could get a refund. So I got an RMA and planned to go shopping for a replacement (people are selling the same model on Craigslist for $10). The RMA came in an email which also told me that if I did not return the equipment within X days I would be charged $100. This is all utter bullshit. Do not use Verizon. If your only other option is Comcast, as it is here, just steal your neighbor's wireless or tether your 3G. They could have left me reasonably happy with the whole botched situation if they had provided a honest estimate of how long it would really take to get someone out (even if it was "days"), or given me access to the same information without having to go through a call center of people only trained to read a script about how to power-cycle your router or whatever. And if they actually apologized for a saleperson convincing me to sign up by lying about what I would be charged. But why would they? What are you going to do, switch to an ISP that treats you like a human being and doesn't fuck with your traffic? Ha. Ha ha ha. Good luck with that.

I am working on a system where a front-end mail server sends mail to what it considers to be a LDA (Local Delivery Agent) which actually sends mail to a back-end server via LMTP. I can t remove that fake LDA from the design because it does a bunch of business specific processing along the way. I am working on changing the back-end from Cyrus to Dovecot. Currently the mail goes from the fake LDA to the Cyrus LMTP server. What I would like to do is to have an LMTP server run on the back-end machine that launches the Dovecot deliver program immediately and then returns an appropriate code. So far I have been experimenting with having Postfix run on the back-end to use deliver as the real LDA. The first problem with this is that the mail will be written to the Postfix queue and then written to the mail store. Doubling the number of writes is a real problem for a system that is going to be write-bottlenecked - it would significantly increase the hardware costs. The second problem is that when an account goes over quota the back-end server would be generating a bounce message. I would prefer the front-end server to generate the bounce on an un-munged message. Basically all I need is a simple daemon (which could even be launched from inetd) that talks LMTP (a very simple cut-down version of SMTP) and executes a single command to receive the data. It might be necessary to serialise running the delivery process, in which case the mail data would have to be stored in memory and there would need to be a semaphore around executing the delivery program. Does anyone know of such a program? If not then I ll have to write it myself (which shouldn t be difficult) and GPL it. If I have to do that then I need a suitable name for it. Any suggestions would be appreciated.

This release had more annoying globalish changes than any release that I remember recently. There were lots of little changes across all files, lots of internal reorganization, and lots of tweaks and little changes to match behavior changes elsewhere in the infrastructure. It's all movement towards a more maintainable code base, but it's not as much fun as fixing lots of bugs. This release should fix the occasional problems that people saw with checking source packages (sorry about that, my fault) and somewhat faster performance due to a bunch of work by Raphael Geissert. It also incorporates changes from debhelper 7.2.3, including removing the requirement that one call install-docs and update-menus in maintainer scripts. That work is now handled by triggers (except for menu-methods files). This was supposed to be the Policy 3.8.1 release, but I ran out of time today. I'm still hoping to get to that very soon (before 2.2.7 propagates into testing), but there were enough important fixes in this that I didn't want to delay it further. I'm running into a bit of a problem with Lintian at the moment. It's a lot of fun, and we're making a ton of progress, but it's taking me roughly a day a week to really keep up with the incoming bugs and patch submissions and do a proper job of merging them (which is more than just applying them). This really isn't sustainable. I have a bunch of other things that I want to do, both inside Debian and outside, and I'm going to have to reduce the amount of time that I'm spending on Lintian soon. I hate to do that, since it's great to stay this up-to-date and see this high pace of change, but things may start backing up a bit in the near future. As usual, thank you very much to Adam D. Barratt and Raphael Geissert for all of their work, without which this release wouldn't have been possible.

So I am going to do something that nobody on the Internet is doing lately: post a review of the Kindle 2 after having only used it for three days. Shocking, yes, I know. I had never even seen a Kindle of either model before getting the Kindle 2. I had, though, thought about getting an eInk device for some time. The $359 Kindle 2 price tag caused me significant pause, though in the end I went for it due to the 30-day return policy. On the surface, I thought that it would be weird to have a Kindle. After all, how often am I away from the computer? And there s a small local library a few blocks from where I work. But I had a hunch it might turn out for me like my iPod did: something that didn t sound all that useful from reading about it, but turned out to be tremendously so after having actually used it. Turtleback Delivery I ordered my Kindle 2 with standard shipping, which meant that it went by FedEx Smart Post. Here is my SmartPost rant. There are two words in Smart Post that are misleading. I once had an item take literally a week to make it from St. Louis to Kansas. That is, I kid you not, slower than the Pony Express ran in 1860. This time, my Kindle made it from Kentucky to Kansas in a mere five days. Oh, and it spent more than 24 hours just sitting in St. Louis. The Device Overall, the device is physically quite nice. It is larger and thinner than I had imagined, and the screen also is a bit smaller. It is usually easier to hold than a paperback, due to not having to prevent it from closing too far at the binding edge. The buttons are easy to press, though I could occasionally wish for them to be easier, but that s a minor nit. The Screen The most important consideration for me was the screen. The eInk display as both stunningly awesome and disappointing. This is not the kind of display you get on your computer. Or, for that matter, any other device. It isn t backlit. It reacts to light as paper does. It can be viewed from any angle. And it consumes no power to sustain an image; only to change it. Consequently, it puts up a beautiful portrait of a famous author on the screen when it is put to sleep, and consumes no power to maintain it. The screen s response time isn t anywhere near as good as you d expect from a regular LCD. It flashes to black when you turn a page, and there is no scrolling. On the other hand, this is not really a problem. I found the page turning speed to be more than adequate, and probably faster than I d turn the page on a real book. The resolution of the display has the feeling of being incredible. The whole thing provides a far different, and in my eyes superior, experience to reading on an LCD or CRT screen. My nit is the level of contrast. The background is not really a pure white, but more of a light gray. This results in a contrast level that is quite clearly poorer than that of the printed page. At first I thought this would be a serious problem, though I am growing somewhat more used to it as I read more. Reading Experience Overall, I ve got to say that it is a great device. You can easily get lost in a book reading it on the Kindle. I m reading David Copperfield for the first time, and have beat a rather rapid path through the first five chapters on the Kindle already. And that, I think, is the best thing that could be said about an ebook reader. It stays out of the way and lets you immerse yourself in your reading. The Kindle s smartly-integrated Oxford-American Dictionary was useful too. One thing about a novel written 150 years ago is that there are some words I just haven t ever heard. Nosegay, for instance. You can move a cursor to a word to see a brief pop-up definition appear, or press Enter to see the entire entry. This is nice and so easy that I m looking up words I wouldn t have bothered to if I were reading the book any other way. A nosegay, by the way, is a bouquet of showy flowers. Buying Experience The Kindle has a wireless modem tied to the Sprint network on it. The data charges for this, whatever they may be, are absorbed by Amazon in the cost of the device and/or the books you buy for it. This turned out to be a very smart piece of engineering. I discovered on Amazon s Kindle Daily Post that Random House is offering five mostly highly-rated sci-fi books for free on the Kindle for a limited time. So I went over to the page for each, and made my purchase . It was only a click or two, and I saw a note saying it was being delivered. A few minutes later, I picked up the Kindle off the kitchen counter. Sure enough, my purchases were there ready to read. Impressive. This level of ease of use smells an awful lot like Apple. Actually, I think it s surpassed them. You can delete books from the Kindle and re-download them at any time. You can initiate that operation from either the PC or the Kindle. And you can also browse Amazon s Kindle store directly from the device itself. I haven t subscribed to any magazines or newspapers, but I gather that they deliver each new issue automatically the moment it s released by the publisher, in the middle of the night. I pre-ordered the (free to Kindle) Cook s Illustrated How-to-Cook Library. It makes me way happier than it should to see This item will be auto-delivered to your Kindle on March 26 in the order status. Free Books Amazon s Kindle library has a number of completely free Kindle books as well. These are mostly out-of-copyright books, probably sourced from public etext places like Project Gutenberg, and converted to the Mobipocket format that is the Kindle s native format with a minimum of human intervention. As they are free, you can see them in Amazon s library if you sort by price. And, of course, Amazon will transfer them to the Kindle wirelessly, and maintain a copy of them in your amazon.com account. Unfortunately, as with free etexts in general on the Internet, the quality of these varies. I was very annoyed to find that many free etexts look like they were done on a typewriter, rather than professionally printed. They don t use smart quotes; only the straight ones. When reading on a device that normally shows you a faithful print experience, this is jarring. And I spent an inordinate amount of time trying to find a copy of Return of Sherlock Holmes that actually had the graphic figures in Dancing Men. Ah well. Your Own Content Amazon operates a mail server, username@kindle.com. You can email stuff to it, and it will convert it to the Kindle format and wirelessly upload it to your kindle for a fee of $0.10. Alternatively, you can use username@free.kindle.com, which does the same thing at no charge, but emails you back a link to download the converted work to install via USB yourself. I tried it with a number of PDFs. It rejected about a dozen times from only my single mail message a PDF containing graphic images only. However, it does quite well with most text-heavy PDFs notably doing an excellent job with Return of Sherlock Holmes from bookstacks.org the only source I found that was both beautifully typeset and preserved the original figures. Unfortunately, the PDF converter occasionally has troubles identifying what should be a paragraph, particularly in sections of novels dealing with brief dialog. I have also sent it some HTML files to convert, which it also does a great job with. You can also create Mobipocket files yourself and upload them directly. There is a Mobipocket creator, or you can use mobiperl if you are Windows-impaired or prefer something scriptable on Linux. The device presents itself as a USB mass storage device, so you can see it under any OS. There s a documents folder to put your files in. You can back it up with your regular backup tools, too. And it charges over USB. Web Browser I haven t tried it much. It usually works, but seems to be completely down on occasion. It would get by in a pinch, but is not suitable for any serious work. The guys over at XKCD seem to love it; in fact, their blog post was what finally convinced me to try the Kindle in the first place. Final Thoughts I ve ordered a clip-on light and a leather case for the Kindle. The light, I believe, will completely resolve my contrast complaint. The leather case to protect it, of course. I can t really see myself returning the Kindle anymore. It s way too much fun, and it s making it easier to read more again. And really, if Amazon manages to reach out to a whole generation of people and make it easy and fun for them to read again and make a profit doing it, of course they may move up a notch or two from being an evil patent troll company to a positive social force company. Wow, never thought I d say that one.

As you might have guessed (or not), I ve been playing with the idea of improving my Debian rebuild setup by building on tmpfs: disk I/O take a signifiant amount of time, that could easily be reduced by building (mostly) in memory. Most packages build fine on tmpfs: only 8 packages fail only on tmpfs. Apparently, most of the failures are caused by old versions of autoconf: this version used /tmp to store temporary files ; for example, when generating Makefile, it would create it on /tmp, then move it to the current directory. The problem arises when /tmp is an ext3 filesystem, while the current directory is on tmpfs: when moving the file back to the current directory, the mtime of the file is truncated. This can lead make to think that a given target needs to be rebuilt. A nice example of that is the smuxi package, that just loops when building. Demo:

# cat Makefile
f2: f1
        echo "Need to rebuild f2!"

First case: f1 and f2 are stored in the current directory. Make agrees that
f2 was created after f1:

# rm f1 f2 ; touch f1 ; touch f2 ; make
make:  f2' is up to date.

Second case: f2 is created on /tmp, then moved to the current directory. Make thinks that f2 was created before f1 (and is right, according to stat).

# rm f1 f2 ; touch f1 ; touch /tmp/f2 ; mv /tmp/f2 f2 ; make
echo "Need to rebuild f2!"
Need to rebuild f2!
# stat f1 f2
  File:  f1'
[..]
Modify: 2009-03-08 18:21:08.900237000 +0100
  File:  f2 
[..]
Modify: 2009-03-08 18:21:08.000000000 +0100

Today is the International Woman's Day (IWD) so I will take advantage of my blog to congratulate all women either into the Debian Project and in General.

The women's history, from my point of view, has been involved of many difficult situations. Women have had to face the inequality in many ways and levels. On the other hand, it has changed a lot but there still happening some cases nowadays.

I got surprised, when I read the article in Wikipedia, by some controversial facts. Sometimes it is a shame our "human" behavior...

The listeners of yesterday s episode of Venty s Hackerfunk radio show on Radio LoRa already know and heard it: We made identi.ca talk. And we did it with help of other microbloggers. (The podcast version of this Hackerfunk episode will be online in a few days, too. Will link it here and either Venty or me will post it on identi.ca, too, as soon as it s published.) A few weeks ago we thought about how we could show microblogging on the radio. With identi.ca s Jabber (XMPP) interface we have real time access, and so the idea was born to pipe all incoming dents into a speech synthesis system. Then we tried to figure out which tools would be appropriate. Quite fast, people on identi.ca as well as on the LUGS IRC (e.g. bones0) pointed us to festival and espeak. We found no support for German in festival, so we went for espeak although festival would have had the advantage of the existence of a festival plugin for the popular multiprotocol messenger Pidgin. Next step was more difficult than expected: How to make a tail -f of XMPP incoming messages? Something like rsstail, just for XMPP. Although using the IM to IRC gateway Bitlbee (as I use it myself) and using tail -f (or better inotail -f ) on the IRC client s log file (ii comes to my mind for such purposes) would have been an option, nobody had the idea at that time. And since @deepspawn conjured xmpptail in less than two hours we happily took it. xmpptail (tar.gz) is written in Python and uses Twisted Words (Debian package python-twisted-words) as XMPP libraries. I had to patch xmpptail slightly for unbuffered I/O, Unicode support and for removing things we don t want to hear on the radio as follows, but it worked more or less out of the box.

--- xmmptail.py 2009-02-25 20:47:48.000000000 +0100
+++ xmpptail.py 2009-03-07 18:48:57.000000000 +0100
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/python -u
 # -*- coding: utf-8 -*-
 # author: Carlos A. Perilla 
 # This file is part of Jance bot.
@@ -65,7 +65,8 @@
          body = unicode(e.__str__())
          break
 
-    print("%s: %s" % (from_id,body))
+#    print("%s: %s" % (from_id,body))
+    print("%s" % (body.encode('utf-8')))
 
 
 def authfailedEvent(xmlstream):
@@ -80,9 +81,9 @@
   dprint('Got something: %s -> %s' % (el.name, str(el.attributes)))
 
 if __name__ == '__main__':
-    print "Starting"
+    #print "Starting"
     execfile('tailconf')
-    print USER_HANDLE
+    #print USER_HANDLE
     me = USER_HANDLE + "/xmpptail"
     myJid = jid.JID(me)
     server = USER_HANDLE[USER_HANDLE.find('@')+1:]

So after configuring xmpptail to use the hackerfunk Jabber account, we successfully ran the following script during the radio show:

./xmpptail.py   while read LINE; do
        if [ "$LINE" = "empty" ]; then
                continue;
        fi;
        echo $LINE
        echo $LINE   tee -a xmpp-espeak.log   espeak --stdin -v de;
done

At the end of the show, @rebugger found this howto which describes very detailed how to get festival working together with the non-free ( non-free as in DFSG) MBROLA project which offers also the appropriate files for German. But because of how much work this would be to get it running, I currently prefer to stay with espeak for German speech synthesis . Next step would be to use mnoGosearch s mguesser to detect the language of a dent and run espeak (or whatever text-to-speech system is appropriate for the guessed language) with the appropiate options for that language, because otherwise many dents sound really funny. ;-) Update, 15:02: Venty gave the whole system the name Identibla .

So, fanatical readers of my blog (are there any?) may remember a long time back that I got bored and wrote a little script that amused me. While testing something out yesterday, I needed to find a domain that I wasn t using in production so I could test on it. I stumbled across Cheese Nibbles which had a lovely We ll be back with something cool soon message on it. So, I did my tests, and they worked. Everything s good. I d actually forgotten what was on Cheese Nibbles in the first place, so I started rooting round in the code. I found the old site, and I decided to resurrect it. I posted in a couple of IRC channels, to see if other people found the site amusing, and got the following back in response

<Daviey> neat
<Daviey> Mez: rss feed needed
<Mez> Daviey: *chuckles*
<Mez> but then it d be twitter
<Daviey> sure, but a shared account
<Daviey> Or.. make it twitter when changed

So, well, thanks to Dave Walker s suggestion, I did a quick google, and within 6 minutes, had it working. You can find the twitter feed at http://twitter.com/cheesenibbles Also, over the last couple of weeks I ve been working with Zeth from my local LUG (and the tech, who shall remain anonymous, like The Stig ) on a podcast, which we are, for now calling The Podcast (imaginative I know). I was interviewed in the first podcast (which Zeth did alone, and I (apparently) provided the comedy element for) and presented with Zeth for the second one. Also, as the second one was recorded during the Ubuntu Global Bug Jam, of which I was running one of the venues, I actually allowed myself to be interviewed for it aswell. The podcast is very rough around the edges at the moment, and well, we haven t got a website, or an RSS feed yet, but you can find the show notes here, and download the podcast episode in either MP3 Format, or OGG Format All comments/thoughts/suggestions welcome!

I ve been working as of late with psycopg2, the Python bindings for PostgreSQL. Documentation is a bit, erm, lacking. You can get the basic stuff working by reading Python s DBAPI-2.0 specification, since psycopg2 follows it, but (as I ve only recently realized) discovering the extra features that exist needs you to poke the source. The other day I was wanting to pass to the execute method of a cursor a sequence to fill the value list in a SQL IN statement. The DBAPI spec does not mention this usage at all, and a cursory question in #python told me it wasn t supported (required?) by the spec. It is, in any case, supported in psycopg2. You just need to

  import psycopg2.extensions

and it just works: you can pass tuples, and they ll be converted to a value list. If you want to be able to pass lists as well:

  from psycopg2.extensions import SQL_IN, register_adapter
  register_adapter(list, SQL_IN)

There is also a row factory in psycopg2.extras that offers a dictionary interface, instead of tuple-based. Finally, I have no idea why, but when I started with psycopg2, I came to get the idea that only named parameter substitution was supported (INSERT ... VALUES (%(var1)s, %(var2)s)), but simple %s substitution is also supported: just pass (of course) a tuple as the second parameter to execute, instead of a dictionary.