I have just been informed that syfy is the new SMS abbreviation for syphilis .

Bruce Schneier writes about the risks involving children abandoned in cars and cites an article about the tragic deaths of children in hot cars [1]. One unfortunate error that he made was to not cite the following from the end of the last page of the Washington post article he cited [2]:
In hyperthermia cases, he believes, the parents are demonized for much the same reasons. We are vulnerable, but we don t want to be reminded of that. We want to believe that the world is understandable and controllable and unthreatening, that if we follow the rules, we ll be okay. So, when this kind of thing happens to other people, we need to put them in a different category from us. We don t want to resemble them, and the fact that we might is too terrifying to deal with. So, they have to be monsters. I believe that similar thought processes are used in relation to many other situations, and that such thought processes prevent people from taking appropriate actions to minimise the risk. If someone considers that forgetting a child in the back seat to be an accident that could happen to anyone then they would be inclined to take action to minimise the risk (such as spending some money on a sensor). If however they consider such forgetfulness to be proof of being a bad parent , then as they are a good parent they would have to avoid buying a monitor. I m surprised that Bruce didn t draw an analogy between this and the forgetful losses of laptops and guns by people who work for law enforcement agencies (which he has written about before). I wonder how expensive it would be to make a sensor for heart-rate, breathing, and temperature integrated with a GSM modem and a GPS? If it could be small enough to be attached to clothes then the child could wear it at all times. If such a sensor was to detect a sign of a problem it wouldn t matter whether the child was forgotten in a car, at day-care, or even being actively supervised. The data would be sent to the monitoring agency along with GPS data. The monitoring agency could then phone the parents. If the parents don t answer or don t know where the child is then the police could track down the GPS location. Probably most calls would be due to parents leaving a child too close to an air-conditioner or playing outside in the sun in summer which are unlikely to give a fatal result and a phone call would get a quick fix for what would only be a minor health problem. If the device was marketed as monitoring for sleep apnia then parents could buy it without admitting to the possibility that they might do anything wrong. The causes of SIDS are a topic of ongoing research and parents can admit to being worried about their children suffering from it without admitting any possibility that they might make a mistake.

• [1] http://www.schneier.com/blog/archives/2009/03/leaving_infants.html


• [2] http://www.washingtonpost.com/wp-dyn/content/article/2009/02/27/AR2009022701549.html

Various people were wondering how much power the SheevaPlug consumes exactly. Laurent Guerby, who runs the GCC Compile Farm, posted some data today. According to his measurements with an Energy Monitor 3000, the SheevaPlug uses:

3.1W	just on, Linux booted
3.3W	serial
4.3W	serial + Ethernet
5.4W	serial + Ethernet + 100% CPU
6.0W	serial + Ethernet + USB disk
7.0W	serial + Ethernet + USB disk + 100% CPU

However, these figures should be seen as preliminary and relatively high values since power management for Kirkwood has not been implemented in the kernel yet. In particular, the Kirkwood chip has SATA and PCI-E but they could be turned off since they're not used by the SheevaPlug. This is currently not done in the kernel, so you can expect some improvements in this area.

I am not aware of any Linux Users Group (LUG) being active in informing it s members of how the policies of the various political parties compare with regard to free software and the other issues that are of interest to most members. I believe that this is a grave mistake. Shortly before an election there are many social groups that send lists of questions to all the parties. They ask about the policies the parties have in regard to the issues that they care about, and helpfully mention the number of members that will receive the response. This of course doesn t mean that every member of the group in question will cast their vote in the same way, merely that they will take note of the answers. The committee members of the parties in question will then decide how to answer the questions and whether policy should be tweaked to allow answers that the lobby groups will like. So this process not only helps members of a group make informed voting decisions related to issues that they care about, but it also helps political parties choose policies that are least offensive to the group in question. Here is a draft of a list of questions that I think should be asked of all political parties on behalf of Linux users:

1. It is important for all citizens to access all government data without being forced to buy new software or hardware, open standards allow everyone to access the data with free software. Do you support the use of open standards for data on government web sites and other forms of electronic communication between government agencies and citizens?


2. For long term archival of records it is important that file formats remain readable. The only effective way of doing this is to use open file formats that are implemented in free software. Do you support mandating that all data submitted to government agencies (by citizens or corporations) be in open file formats wherever possible?


3. In these difficult economic times there is a great interest in keeping jobs in the country instead of sending money overseas. To what extent do you support the use of free software that is installed and managed by locals (keeping the money in the economy) instead of importing software at great taxpayer expense?


4. Commercial software has a limited support period, after that time has elapsed there is no further support and systems become increasingly unreliable. Do you support mandating that all systems relating to the emergency services run on free software to allow quality long-term support by local citizens?


5. There has been a lot of concern recently about the spread of child-porn. The best available evidence shows that insecure home PCs that run Trojan Horse programs are a key part of distributing it and other illegal material. Do you support the introduction of government programs to train parents in installing one of the more secure free operating systems on their home PC to protect their children?

This is just a rough draft. Obviously there needs to be local differences (EG don t use point 3 in the US because MS brings money into the US economy). Does anyone have any suggestions for other questions?

I'm reading the d-i IRC team meeting logfile at the moment, and I must say, I'm impressed by the clear communication and seeing people openly discussing problems (while not forgetting the good stuff) and how to solve them. I think that all those real life meetings we had in the last years really help. Personalities are visible and respected, problems are mentioned but the focus is on solutions. This meeting was mostly about release management and debian-installer team issues, the next meeting is planned to happen in two weeks and should focus on the technical details. Feels good to be watching this :-)

Horse Gym USA has a picture of a Canadian at the top of the page.

San Francisco looks like a small place from the top, i.e. the top of Golden Gate Heights. It also seems like all the views of the city face northwards. Mt Davidson does have some views south, but they aren't as pretty. San Francisco is sand and rocks, the trees that do well here are not from here. The cloud forest plants should do well, the botanical garden has lots of them.

This post is for Akash. You'll have to ask him why.

The book Expert Python Programming comes with subtitle Best practices for designing, coding, and distributing your Python software. Well this seems to be quite broad area to be covered by a single book, but the book really tries to cover all topics at least in some form of overview and pointing user to other documentation sources. In first chapter you can find information about installing Python on various platforms. As the book seems to be targeted to more experienced users, this chapter seems to be a bit useless, but author decided to give a chance also to users, who do not know the basics. Next chapters describe best practices for writing Python code, which was the most interesting part for me. I use Python several years, but I never really found time time to properly look at using decorators, generators and other "recently" introduced language features. Second chapter nicely covers these and I really appreciated that I got idea for what decorators can be used. Third chapter into class level and describes how to use metaclasses, attributes, constructors and other things you can face on class level. Fourth moves the topic to even more abstract way and in short summarises PEP 8 best coding practices. Here we leave solid ground of writing code and we move to more abstract areas of managing projects, testing, documenting and distributing software. It begins with description of distutils and setuptools in chapters five and six, where you will find basic information about distributing your software. Seventh chapter covers zc.buildout, which can help you with setting up and deploying applications in way Zope and Plone do. Next chapter is not really related to Python, it just describes advantages of version control. The most Python related part is using Mercurial VCS, which is written in Python, but you can of course use it for managing any project. Next tree chapters are devoted to project development in general, describing management of the project, generating documentation for project using Sphinx (what actually made me into starting documentation for python-gammu in it) and test-driven development practices. Now comes again two more interesting chapters describing optimization of Python code, giving the best advice not to optimize unless you are sure it is needed ;-). But it also goes more deeper and describes benchmarking procedures, how to debug memory usage and some basic tips for general things which might improve performance. Last chapter is some kind of extra in the book and to me it looks it was added just to use buzzwords design patters, but well it is there and it covers few design patterns which might be useful in Python like singleton or more advanced version of it called borg. In general this book covers quite wide range of topics, what can be seen as biggest advantage and disadvantage. You will get basic overview of creating Python project using modern methods (test driven development, design patters, best practices for Python code), but you can not expect it will go into much deep details. The book can extend your knowledge of development in Python, but you should carefully check examples with errata, because otherwise several of them will fail. I'm not that good at spotting problems in written English (as you can probably see in this post), but I spotted several errors in code and I think more careful editor work on the book would improve my feelings a bit. Also I personally would prefer much deeper look into some topics and skipping the others, but one book can not fit all ;-).

After reading the comments on my earlier post, Delayed allocation and the zero-length file problem as well as some of the comments on the Slashdot story as well as the Ubuntu bug, it s become very clear to me that there are a lot of myths and misplaced concerns about fsync() and how best to use it. I thought it would be appropriate to correct as many of these misunderstandings about fsync() in one comprehensive blog posting. As the Eat My Data presentation points out very clearly, the only safe way according that POSIX allows for requesting data written to a particular file descriptor be safely stored on stable storage is via the fsync() call. Linux s close(2) man page makes this point very clearly:

A successful close does not guarantee that the data has been successfully saved to disk, as the kernel defers writes. It is not common for a file system to flush the buffers when the stream is closed. If you need to be sure that the data is physically stored use fsync(2).

Why don t application programmers follow these sage words? These three reasons are most often given as excuses:

1. (Perceived) performance problems with fsync()
2. The application only needs atomicity, but not durability
3. The fsync() causing the hard drive to spin up unnecessarily in laptop_mode

Let s examine each of these excuses one at a time, to see how valid they really are. (Perceived) performance problems with fsync() Most of the bad publicity with fsync() originated with the now infamous problem with Firefox 3.0 that showed up about a year ago in May, 2008. What happened with Firefox 3.0 was that the primary user interface thread called the sqllite library each time the user clicked on a link to go to a new page. The sqllite library called fsync(), which in ext3 s data=ordered mode, caused a large, visible latency which was visible to the user if there was a large file copy happening by another process. Nearly all of the reported delays was a few seconds, which would be expected; normally there isn t that much dirty data that needs to be flushed out on a Linux system, even if it is even very busy. For example, consier the example of a laptop downloading an .iso image from a local file server; if the laptop has the exclusive link of a 100 megabit/second ethernet link, and the server has the .iso file in cache, or has a nice fast RAID array so it is not the bottleneck, then in the best case, the laptop will be able to download data at the rate of 10-12 MB/second. Assuming the default 5 second commit interval, that means that in the worst case, there will be at most 60 megabytes which must be written out before the commit can proceed. A reasonably modern 7200 rpm laptop drive can write between 60 and 70 MB/second. (The Seagate Momentus 7200.4 laptop drive is reported to be able to deliver 85-104 MB/second, but I can t find it for sale anywhere for love or money.) In this example, an fsync() will trigger a commit and might need to take a second while the download is going on; perhaps half a second if you have a really fast 7200 rpm drive, and maybe 2-3 seconds if you have a slow 5400 rpm drive. (Jump to Sidebar: What about those 30 second fsync reports?) Obviously, you can create workloads that aren t bottlenecked on the maximum ethernet download speed, or the speed of reading from a local disk drive; for example, dd if=/dev/zero of=big-zero-file will create a very large number of dirty pages that must be written to the hard drive at the next commit or fsync() call. It s important to remember though, fsync() doesn t create any extra I/O (although it may remove some optimization opportunities to avoid double writes); fsync() just pushes around when the I/O gets done, and whether it gets done synchronously or asynchronously. If you create a large number of pages that need to be flushed to disk, sooner or later it will have a significant and unfortunate effect on your system s performance. Fsync() might make things more visible, but if the fsync() is done off the main UI thread, the fact that fsync() triggers a commit won t actually disturb other processes doing normal I/O; in ext3 and ext4, we start a new transaction to take care of new file system operations while the committing transction completes. The final observation I ll make is that part of the problem is that Firefox as an application wants to make a huge number of updates to state files and was concerned about not losing that information even in the face of a crash. Every application writer should be asking themselves whether this sort of thing is really necessary. For example, doing some quick measurements using ext4, I determined that Firefox was responsible for 2.54 megabytes written to the disk for each web page visited by the user (and this doesn t include writes to the Firefox cache; I symlinked the cache directory to a tmpfs directory mounted on /tmp to reduce the write load to my SSD). So these 2.54 megabytes is just for Firefox s cookie cache and Places database to maintain its Awesome bar . Is that really worth it? If you visit 400 web pages in a day, that s 1GB of writes to your SSD, and if you write more than 20GB/day, the Intel SSD will enable its write endurance management feature which slows down the performance of the drive. In light of that, exactly how important is it to update those darned sqllite databases after every web click? What if Firefox saved a list of URL s that has been visited, and only updated every 30 or 60 minutes, instead? Is it really that every last web page that you browse be saved if the system crashes? An fsync() call every 15, 30, or 60 minutes, done by a thread which doesn t block the application s UI, would have never been noticed and would have not started the firestorm on Firefox s bugzilla #421482. Very often, after a little thinking, a small change in the application is all that s necessary for to really optimize the application s fsync() usage. (Skip over the sidebar if you ve already read it). Sidebar: What about those 30 second fsync reports? If you read through the Firefox s bugzilla entry, you ll find reports of fsync delays of 30 seconds or more. That tale has grown in the retelling, and I ve seen some hyperbolic claims of five minute delays. Where did that come from? Well, if you look that those claims, you ll find they were using a very read-heavy workload, and/or they were using the ionice command to set a real-time I/O priority. For example, something like ionice -c 1 -n 0 tar cvf /dev/null big-directory . This will cause some significant delays, first of all because ionice -c 1 causes the process to have a real-time I/O priority, such that any I/O requests issued by that process will be serviced before all others. Secondly, even without the real-time I/O priority, the I/O scheduler naturally prioritizes reads as higher priority than writes because normally processes are waiting for reads to complete, but writes are normally asynchronous. This is not at all realistic workload, and it is even more laughable that some people thought this might be an accurate representation of the I/O workload of a kernel compile. These folks had never tried the experiment, or measured how much I/O goes on during a kernel compile. If you try it, you ll find that a kernel compile sucks up a lot of CPU, and doesn t actually do that much I/O. (In fact, that s why an SSD only speeds up a kernel compile by about 20% or so, and that s in a completely cold cache case. If the commonly used include files are already in the system s page cache, the performance improvement of the SSD is much less.) Jump back to reading Performance problems with fsync. The atomicity not durability argument One argument that has commonly been made on the various comment streams is that when replacing a file by writing a new file and the renaming file.new to file , most applications don t need a guarantee that new contents of the file are committed to stable store at a certain point in time; only that either the new or the old contents of the file will be present on the disk. So the argument is essentially that the sequence:

• fd = open( foo.new , O_WRONLY);
• write(fd, buf, bufsize);
• fsync(fd);
• close(fd);
• rename( foo.new , foo );

is too expensive, since it provides atomicity and durability , when in fact all the application needed was atomicity (i.e., either the new or the old contents of foo should be present after a crash), but not durability (i.e., the application doesn t need to need the new version of foo now, but rather at some intermediate time in the future when it s convenient for the OS). This argument is flawed for two reasons. First of all, the squence above exactly provides desired atomicity without durability . It doesn t guarantee which version of the file will appear in the event of an unexpected crash; if the application needs a guarantee that the new version of the file will be present after a crash, it s necessary to fsync the containing directory. Secondly, as we discussed above, fsync() really isn t that expensive, even in the case of ext3 and data=ordered; remember, fsync() doesn t create extra I/O s, although it may introduce latency as the application waits for some of the pending I/O s to complete. If the application doesn t care about exactly when the new contents of the file will be committed to stable store, the simplest thing to do is to execute the above sequence (open-write-fsync-close-rename) in a separate, asynchronous thread. And if the complaint is that this is too complicated, it s not hard to put this in a library. For example, there is currently discussion on the gtk-devel-list on adding the fsync() call to g_file_set_contents(). Maybe if someone asks nicely, the glib developers will add an asynchronous version of this function which runs g_file_set_contents() in a separate thread. Voila! Avoiding hard drive spin-ups with laptop_mode Finally, as Nathaniel Smith said in Comment #111 of of my previous post:

The problem is that I don t, really, want to turn off fsync s, because I like my data. What I want to do is to spin up the drive as little as possible while maintaining data consistency. Really what I want is a knob that says I m willing to lose up to minutes of work, but no more . We even have that knob (laptop mode and all that), but it only works in simple cases.

This is a reasonable concern, and the way to fix this is to enhance laptop_mode in the Linux kernel. Bart Samwel, the author and maintainer of laptop_mode, actually discussed this idea with me last month at FOSDEM. Laptop_mode already adjusts /proc/sys/vm/dirty_expire_centisecs and /proc/sys/vm/dirty_writeback_centisecs based on the configuration parameter MAX_LOST_WORK_SECONDS, and it also adjusts the file system commit time (for ext3; it needs to be taught to do the same thing for ext4, which is a simple patch) to MAX_LOST_WORK_SECONDS as well. All that is necessary is a kernel patch to allow laptop_mode to disable fsync() calls, since the kernel knows that it is in laptop_mode, and it notices that the disk has spun up, it will sync out everything to disk, since once the energy has been spent to spin up the hard drive, we might as well write everything in memory that needs to be written out right away. Hence, a patch which allows fsync() calls to be disabled while in laptop_mode should do pretty much everything Nate has asked. I need to check to see if laptop_mode does this already, but if it doesn t force a file system commit when it detects that the hard drive has been spun up, it should obviously do this as well. (In addition to having a way to globally disable fsync() s, it may also be useful to have a way to selectively disable fsync() s on a per-process basis, or on the flip side, exempt some process from a global fsync-disable flag. This may be useful if there are some system daemons that really do want to wake up the hard drive and once the hard drive is spinning, naturally everything else that needs to pushed out to stable store should be immediately written.) With this relatively minor change to the kernel s support of laptop_mode, it should be possible to achieve the result that Nate desires, without needing force applications to worry about this issue; applications should be able to just simply use fsync() without fear. Summary As we ve seen, the reasons most people think fsync() should be avoided really don t hold water. The fsync() call really is your friend, and it s really not the villain that some have made it out to be. If used intelligently, it can provide your application with a portable way of assuring that your data has been safely written to stable store, without causing a user-visible latency in your application. The problem is getting people to not fear fsync(), understand fsync(), and then learning the techniques to use fsync() optimally. So just as there has been a Don t fear the penguin campaign, maybe we also need to have a Don t fear the fsync() campaign. All we need is a friendly mascot and logo for a Don t fear the fsync() campaign. Anybody want to propose an image? We can make some T-shirts, mugs, bumper stickers Related posts (automatically generated):

1. Delayed allocation and the zero-length file problem A recent Ubuntu bug has gotten slashdotted, and has started...

And there we have done it - new sections in the archive. Read more about it in my d-d-a post and the whole set of affected packages is also available - and updated slightly, as some small things slipped through. The whole process was less painful than feared. After some discussion, started a few weeks ago we finally went and coordinated the change today. There was no reliable and good way for changes that large, so we had to create our own way. (The last section changes had been done ages ago). Mike provided a little python script which was able to parse our input file with the section matches against our database, so we ended up with a list we could possibly feed back into the database. Except, not really. Thanks to various things the list generated needed some manual work (which I also managed to fuck up a little), but after that was done we were ready to go. And while Mark told the wanna-build admins about the new sections (for some reason our buildd network cares about sections) and prepared a patch for our policy, I prepared the mail for the d-d-a list. Then we all had a break, as a dinstall was still running, and we did not want to break that. The actual changes then took only a few seconds, sending the mail to d-d-a took longer, thankyou gnupg. :) I wonder how much afterwork this will generate. There are at least the policy changes, lintian, the packages control files to adjust. Also some websites, where Frank already changed packages.debian.org to know about the new names. (Before he fixed up me forgetting some PHP packages to move. This way all of the active ftpteam members did something in it :) ).

---

In other news, we are currently stress-testing our ftp-master host a little. Well, not really, it could take more, but right now we are scanning the full archive once, importing all filenames from all our Debian packages. The goal is to get projectb larger. Err, no, actually we do want to get rid of apt-ftparchive, as that is one of the worst tools in our dinstall runs. It is slow as hell in the mode it has to run for us, making up a large part of our runtime. Step 1 to get rid of it is to generate the content files on our own, and before we can do that we have to import them all Which means we have to unpack every binary package once, reading in all filenames to store them in our database. Thats a lot of data. Currently the associations table, which is nothing more than 4 integers per row, takes up 972MB, and import is still running. (The index on the binary pkg id in this table alone has 700MB). And this table does nothing more than linking the binary package to the filename and filepath. Which are two extra tables, with the filenames having only 87MB, the path entries only taking up 30MB. Yes, anyone using databases will tell me thats not much, and it really isn t that much. It s just that the largest table before this import had 120MB in it. And some of the others used more space for their indexes than for the actual data. :) Anyway, this is only step one on our way to get rid of the slowness apt-ftparchive is. When the contents all work as we want them, the next step will be to get rid of it generating Packages/Sources files. Which means we have some more data to store in projectb. And after that - we also intend to do a Contents file for our source. Each and every source file listed. Guess we really will end up with some more large tables. :) The final goal and main reason why we do this is to save us at least 60 minutes runtime per dinstall run. My own guess is that Contents alone will get us about 20 minutes down. No, before someone wants to yell at me - apt-ftparchive can t be changed as much to make this possible and keep using it. Not in the way the generate option works. It has the disadvantage that it scans the files after they are installed in their pool/ location and stores that in its own database . This approach has to negative points:

• first, it is using another database (BerkeleyDB). We do have one database already, projectb, thank you.
• second, and more important, the files only end up in pool/ on dinstall time. Which means the whole scanning is done during dinstall. Which eats time. The new way will surely use the same amount of cpu time in total - but it will spread it over the whole day. 4 runs each hour. Not 4 runs each day. And during dinstall it will boil down to one sql query per installed package moving data from a pending table into the real contents table. Which is a cheap operation.

---

The title says some more, so here is some more: We aren t only working on what I wrote above. There is also work on changing our database access. Historically dak used the pg module to do database work. While the module itself does work, the way its used isn t really the best one. Way too much groundwork everywhere. Not nearly as much of the nice things used people say python could have. So another largish work area is getting rid of pg. Migrating to psycopg2. And putting a proper OO class around it, which in the end hopefully hides those dark details from the application. And to achieve even more sanity, we (well, honestly, Mark) are looking into SQL Alchemy usage. (We also considered storm from Canonical, but that sucks when you take a look at its documentation. (Which documentation?). SQLA won there easily.) While I m ranting: WTF are people neglecting their docs that much? Ever tried looking at the documentation for psycopg2? Forget it. No, a db api compliance test suite is NOT documentation. If there would be any better python module to connect to postgres, I would veto the usage of psycopg2, but there doesn t seem to be one. Sucks.

There is a particular issue about which I am incapable of expressing my opinion. I do have an opinion about it, and I have developed it quite a bit in my own head over the years. However, if I try to voice it, something in me tells me not to. There is another issue about which I am capable of expressing my opinion, but I choose not to. It is a highly controversial topic, and it is very likely that if I expressed my opinion about it, it would be interpreted as being another opinion, which I do not hold. I do not know whether to call that Wittgenstein s curse or not.

DO NOT send your query for more information only to nnn@bugs.debian.org. That address (almost) never reaches the submitter. (The correct address is nnn-submitter@bugs.debian.org or you can CC the submitter directly.) DO NOT close a bug just because your query can you still reproduce it has not been promptly answered. And, actually: DO NOT close a bug if you do not have the maintainer s permission to do so. You may, if you wish, state in the bug logs that you think the bug should be closed. This ends today s public service announcement. Thank you for your attention.

I m in the middle of migrating a mail server away from the Cyrus mail store [1]. Cyrus provides a POP and IMAP server, a local delivery agent (accepting mail via LMTP). It is widely believed that Cyrus will give better performance than other mail stores, but according to a review by linux-magazin.de Dovecot and Courier deliver comparable (and sometimes better) performance [2]. The biggest problem with Cyrus is that it is totally incompatible with the Unix way. This wouldn t be a problem if it would just work and if it would display reasonable error messages when it failed, but it doesn t. It often refuses to work as desired, gives no good explanation, and it s data structures can t be easily manipulated. Dovecot [3] and Courier [4] use the Maildir++ format [5] (as well as many other programs). I have set up a system with Courier Maildrop and Dovecot for the IMAP server [6] and it works well - it s good to have a choice! But also Maildir++ is reasonably well documented and is an extension to the well known Maildir format. This means that it s easy to manipulate things if necessary, I can use mv to rename folders and rm to remove them. Cyrus starts with a database (Berkeley DB file) of all folders in all mailboxes. Therefore it is not possible to move a user from one back-end server to another by merely copying all the files across and changing the LDAP (or whatever else contains the primary authentication data) to point to the new one. It also makes it impossible to add or remove folders by using maildirmake or rm -rf. The defined way of creating, deleting, and modifying mailboxes is through IMAP. One of the problems with this is that copying a mailbox from one server to another requires writing a program to open IMAP connections to both servers at once (tar piped through netcat is much faster and easier). Also if you need to rename a mailbox that contains many gigabytes of mail then it will be a time consuming process (as opposed to a fraction of a second for mv). Cyrus has a tendency to break while Dovecot is documented as being self-healing and Cyrus also seems to cope well in the fact of a corrupted mail store. Even manually repairing problems with Cyrus is a painful exercise. The Cyrus mail store is also badly designed - and it s design was worse for older filesystems (which were common when it was first released) than it is for modern ones. The top level of a Cyrus maildir contains all the messages in the INBOX stored one per file, as well as three files containing Cyrus indexes and sub-directories for each of the sub-folders. So if I want to discover what sub-folders a mailbox has then I can run ls and wait for it to stat every file in the directory or I can use an IMAP client (which takes more configuration time). As opposed to a Maildir++ store where every file that contains a message is stored in a folder subdirectory named new , cur , or tmp which means that I can run ls on the main directory of the mail store and get a short (and quick) result. Using tools such as ls to investigate the operation of a server is standard practice for a sysadmin, it should work well! A finall disadvantage of Cyrus seems to have many small and annoying bugs (such as the reconstruct program not correctly recursing the sub folders). I guess it s because not many people use Cyrus that such things don t get fixed. One trivial advantage of Cyrus is that by default it splits users into different sub-directories for the first letter of the account name. Dovecot supports using a hash of the user-name this is better than splitting by first-letter for performance (it gives a more equal distribution) but will make it slightly more difficult to manipulate the mail store by script. Ext3 can give decent performance without a two level directory structure for as many as 31,998 sub-directories (the maximum that it will support) due to directory indexing and Linux caching of dentries. There may be some other advantages of Cyrus, but I can t think of them at the moment. Here is a script I wrote to convert Cyrus mail boxes to Maildir++. To make this usable for a different site would require substituting a different domain name for example.com (or writing extra code to handle multiple domains) and inserting commands to modify a database or directory with the new server name. There is no chance of directly using this script on another system, but it should give some ideas for people performing similar tasks.

#!/bin/bash -e
# $1 is the user to move # BASEDIR is the directory for temporary files
BASEDIR=/mail/new
rm -rf $BASEDIR/$1 archive.tgz archive2.tgz ORIGSRV=$(echo "MYSQL_COMMAND_TO_GET_SERVER" mysql dbname)
echo "original server was $ORIGSRV" if [ "$ORIGSRV" = "" ]; then
echo "No server for account, strange error."
exit 1
fi ORIGPARENT="/mail/domain/e/example.com/$(echo $1 cut -c1)/user" # copy the data to a local archive
# USERDIR is for using ^ instead of . as Cyrus does.
mkdir -p $BASEDIR
cd $BASEDIR
USERDIR=$(echo $1 sed -e "s/\./^/g")
ssh cyrus@$ORIGSRV "(cd $ORIGPARENT ; tar czf - $USERDIR)" > archive.tgz
tar xzf archive.tgz
if [ "$USERDIR" != "$1" ]; then
mv $USERDIR $1
fi DEPTH=$(find . -mindepth 3 -type d wc -l)
if [ "$DEPTH" != "0" ]; then
echo "too deep"
exit 1
fi # remove cyrus index files
find $1 -name "cyrus.*" -exec rm \;
cd $1 # "mail" is a directory with sub-folders, every other name starting with
# [a-zA-Z] is a regular folder that needs to become a Maildir
for n in [a-zA-Z]* ; do
if [ "$n" != "mail" ]; then
cd "$n" ; mkdir new cur tmp ; touch maildirfolder
mv [0-9]* new true
cd .. ; mv "$n" ".$n"
fi
done # move the inbox files to Maildir storage
mkdir new cur tmp
mv [0-9]* new true
mv mail/[0-9]* cur true # move sub-folders of "mail/" and merge ones with duplicate names
cd mail
for n in * ; do
if [ -e "../.$n" ]; then
cd "$n"
mv [0-9]* "../../.$n/cur" true
cd .. ; rmdir "$n"
else
cd "$n" ; mkdir new cur tmp ; touch maildirfolder
mv [0-9]* new true
cd .. ; mv "$n" "../.$n"
fi
done
cd ..
rmdir mail # get a list of all names so we can recognise duplicates
find . -type d egrep -v "new cur tmp" cd $BASEDIR ssh root@localhost "mv /mail/new/$1 /mail/example.com ; chown -R mail:mail /mail/example.com/$1"
echo "MYSQL_COMMAND_TO_SET_SERVER" mysql dbname

• [1] http://cyrusimap.web.cmu.edu/


• [2] http://www.linux-magazin.de/heft_abo/ausgaben/2007/06/auf_der_teststrecke/


• [3] http://www.dovecot.org/


• [4] http://www.courier-mta.org/imap/


• [5] http://wiki.dovecot.org/MailboxFormat/Maildir


• [6] http://etbe.coker.com.au/2009/03/10/maildrop-imap-postfixadmin/

We managed to watch the multi-award winning "Slumdog Millionaire" this weekend. It is difficult for me to understand why people (in India or PIO's) are so upset about the movie. It is a good movie (I do not know enough about cinema to pronounce it a great movie).

• the narrative style is very nice --- question and answer.
• the editing is tight; the movie doesn't drag at any point.
• there are some very nice sequences and panoramic views.
• none of the actors stands out, but then none bombs out either!

As a Rahman fan since "Kadal Desam" (1993 or so), my opinion on the music is a bit biased. His work on this movie is more than competent though not his best. A. R. Rahman has written better songs than "Jai Ho!" and I didn't notice anything spectacular about the rest of the music. Still, it is worth noting that his best work is in the context of Indian-style films, where there are a lot of songs; the background score of those films only stands out if it is bad! So, I suppose that the criticism of this movie by Indians is largely based on its story-line and setting. This is strange, since poverty in a Mumbai slum (to the extent that I am acquainted with it) is depicted quite accurately here. The plot (like any Dickensian story) drives home the terrible nature of poverty in cities but still manages to inject a note of optimism.

If you are lucky/strong enough to survive with your humanity intact after growing up in a Mumbai slum, then the luck required to win a TV show is nothing in comparison.

Edited to add: Sorry, it turns out that ext4 does now have a heuristic to deal with this case. However, it seems that btrfs doesn't. My point remains, though - designing a filesystem in such a way that a useful behaviour is impossible is unhelpful for the vast majority of application writers, even if it does improve performance in some other use cases.

Further edit: It's mentioned in the comments that btrfs will force a flush of data when a rename is performed as of 2.6.30. That prevents the data loss, but it means that we're still stuck with disk access when we'd be happy with that being batched up for the future. It feels like we're optimising for the wrong case here.

Original text:

There's been a certain amount of discussion about behavioural differences between ext3 and ext4[1], most notably due to ext4's increased window of opportunity for files to end up empty due to both a longer commit window and delayed allocation of blocks in order to obtain a more pleasing on-disk layout. The applications that failed hardest were doing open("foo", O_TRUNC), write(), close() and then being surprised when they got zero length files back after a crash. That's fine. That was always stupid. Asking the filesystem to truncate a file and then writing to it is an invitation to failure - there's clearly no way for it to intuit the correct answer here. In the end this has been avoided by avoiding delayed allocation when writing to a file that's just been truncated, so everything's fine.

However, there's another case that also breaks. A common way of saving files is to open("foo.tmp"), write(), close() and then rename("foo.tmp", "foo"). The mindset here is that a crash will either result in foo.tmp being zero length, foo still being the original file or foo being your new data. The important aspect of this is that the desired behaviour of this code is that foo will contain either the original data or the new data. You may suffer data loss, but you won't suffer complete data loss - the application state will be consistent.

When used with its (default) data=ordered journal option, ext3 provided these semantics. ext4 doesn't. Instead, if you want to ensure that your data doesn't get trampled, it's necessary to fsync() before closing in order to make sure it hits disk. Otherwise the rename can occur before the data is written, and you're back to a zero length file. ext4 doesn't make guarantees about whether data will be flushed before metadata is written.

Now, POSIX says this is fine, so any application that expected this behaviour is already broken by definition. But this is rules lawyering. POSIX says that many things that are not useful are fine, but doesn't exist for the pleasure of sadistic OS implementors. POSIX exists to allow application writers to write useful applications. If you interpret POSIX in such a way that gains you some benefit but shafts a large number of application writers then people are going to be reluctant to use your code. You're no longer a general purpose filesystem - you're a filesystem that's only suitable for people who write code with the expectation that their OS developers are actively trying to fuck them over. I'm sure Oracle deals with this case fine, but I also suspect that most people who work on writing Oracle on a daily basis have very, very unfulfilling lives.

But anyway. We can go and fix every single piece of software that saves files to make sure that it fsync()s, and we can avoid this problem. We can probably even do it fairly quickly, thanks to us having the source code to all of it. A lot of this code lives in libraries and can be fixed up without needing to touch every application. It's not the end of the world.

So why do I still think it's a bad idea?

It's simple. open(),write(),close(),rename() and open(),write(),fsync(),close(),rename(), are not semantically equivalent. One is "give me either the original data or the new data"[2]. The other is "always give me the new data". This is an important distinction. fsync() means that we've sent the data to the disk[3]. And, in general, that means that we've had to spin the disk up.

So, on the one hand, we're trying to use things like relatime to batch data to reduce the amount of time a disk has to be spun up. And on the other hand, we're moving to filesystems that require us to generate more io in order to guarantee that our data hits disk, which is a guarantee we often don't want anyway! Users will be fine with losing their most recent changes to preferences if a machine crashes. They will not be fine with losing the entirity of their preferences. Arguing that applications need to use fsync() and are otherwise broken is ignoring the important difference between these use cases. It's no longer going to be possible to spin down a disk when any software is running at all, since otherwise it's probably going to write something and then have to fsync it out of sheer paranoia that something bad will happen. And then probably fsync the directory as well, because what if someone writes an even more pathological filesystem. And the disks sit there spinning gently and chitter away as they write tiny files[4] and never spin down and the polar bears all drown in the bitter tears of application developers who are forced to drink so much to forget that they all die of acute liver failure by the age of 35 and where are we then oh yes we're screwed.

So. I said we could fix up applications fairly easily. But to do that, we need an interface that lets us do the right thing. The behaviour application writers want is one which ext4 doesn't appear to provide. Can that be fixed, please?

[1] xfs behaves like ext4 in this respect, so the obvious argument is that all our applications have been broken for years and so why are you complaining now. To which the obvious response is "Approximately anyone who ever used xfs expected their data to vanish if their machine crashed so nobody used it by default and seriously who gives a shit". xfs is a wonderful filesystem for all sorts of things, but it's lousy for desktop use for precisely this reason.

[2] Yes, ok, we've just established that it actually isn't that in the same way that GMT isn't UTC and battery refers to a collection of individual cells and so you don't usually put multiple batteries in your bike lights, but the point is that this is, for all practical intents and purposes, an unimportant distinction and not one people should have to care about in their daily lives.

[3] The disk is free to sit there bored for arbitrary periods of time before it does anything, but that's fine, because the OS is behaving correctly. Sigh.

[4] Dear filesystem writers - application developers like writing lots of tiny files, because it makes a large number of things significantly easier. This is fine because sheer filesystem performance is not high on the list of priorities of a typical application developer. The answer is not "Oh, you should all use sqlite". If the only effective way to use your filesystem is to use a database instead, then that indicates that you have not written a filesystem that is useful to typical application developers who enjoy storing things in files rather than binary blobs that end up with an entirely different set of pathological behaviours. If I wanted all my data to be in oracle then I wouldn't need a fucking filesystem in the first place, would I?

Riddle: Totus illae has venio pro , quod is mos totus venio iterum Answer: All they /h s/ I come for , because he custom all I come again

For some time ago I ve been pondering about this question. As long as GSoC 2009 is about to start and people are looking for project ideas, I m posting here a very preliminar draft of my findings and an idea for a posible software project. It s pretty written on-the-fly while I ve managed to have some minutes between work and uni. Rigurous wording isn t P1. Debian s social contract 4th item states:

Our priorities are our users and free software We will be guided by the needs of our users and the free software community. We will place their interests first in our priorities.

When we make decisions, either technical, legal and others regarding the operating system we deliver, this is one of the most referred argument. Altough that s seen as high priority often we don t have ways to properly know what our users want. While working on translations on the Spanish team we ve faced this situation. Do our users find them useful? they are able to understand them? they find that a much familiar wording would be bette than high technical details? We didn t know. Then, the question still exists. Altough Debian has ways for users to provide feedback (BTS, mailing lists, IRC, popcon, etc) none of them is designed to offer a way for the user to provide feedback on features, development roadmap and other non-bug aspects that developers and the project can tally and use for prioritize on releases. It s rather amusing to note that one of the most valuable assets free software community has is user involvement and contribution. But before we enter in details let s start on the basics. For using this information we first need to know who our users are? do we? From my findings and interaction with the community I ve identified two kinds of Debian users who can be clearly named. 1) The derivatives or pure blends, who use the Debian base and framework to build niche distributions. 2) The lead users (using E. Von Hippel s definition), who are the developers, contributors and a group of users. Debian is, if not the only, one of the projects who is best for fostering lead users. Most of them at some point involve themselves in the development process and some, later become developers. There is a more diverse group who are end-users but it s unclear to me how we can group them. Despite that they probably represent the biggest part of the pie. So, how we do please everyone? can we? In my opinion we can, at some degree. By implementing tools for first: gather user feedback, make statistical data, tally, we later can take informed decisions. In Debian decisions are voted and the set of people who votes are only developers with their own constraints. I think this would ease the constant that release-decisions-regarding-foo-tech-legal-issue represented on each release iteration, saving time, health and bits. It could be useful for knowing more about who are the ones we develop things for. I m thinking of some sort of Dell s IdeaStorm.com for software. What do you think? I m willing to co-mentor this if someone finds that is an interesting project for GSoC. Let me know!.

Just a quick update to let you know how the Debian bid for the Google Summer of Code is going. We re doing great right now. Our application is sent out, you can consult the public part here. This year, we re committed to doing a great Summer of Code performance, starting first by improving communication with the Debian community. We opened a Twitter feed for you Twitter freaks: http://twitter.com/DebianGSoC. It will be updated with live information relevant to students and devs alike during the whole summer. UPDATE: We re now also on identi.ca : http://identi.ca/debiangsoc We now have 12 applications on our wiki. Come, add your ideas and discuss them on our IRC channel (#debian-soc on irc.debian.org) and mailing-list. Packaging

• Large dataset manager: download and update local copies of public datasets, and integrate them in Debian with the tools we packaged.
• Mergemaster: perform 3-way merge of config files on upgrades by keeping a copy of original config files.
• Aptitude packages query UI and language: improve the GTK+ search UI and language for complex packages queries
• Aptitude search ranking and presentation: improve the presentation of search results with substring highlighting, relevancy ranking
• Aptitude history tracking: extend and build on the packages actions history tracking features of Aptitude

Debian Installer

• MTD support in d-i: add MTD installation support to the Debian Installer
• Debian-Installer support for GNU/kFreeBSD: replace the currently used FreeBSD sysinstall with Debian-Installer running possibly under a FreeBSD kernel

Blends

• blends webtools: enhancing DebianPureBlends webtools by using UltimateDebianDatabase

Bug tracking

• Debbugs Read/Write Soap Interface: overhaul the existing read-only SOAP Debbugs interface with submission and manipulation capabilities
• Debbugs Web UI: Modern web interface for submitting and manipulating bugs.debian.org s bugs
• Mylyn plugin for debbugs: Plugin for Mylyn to support debbugs, allowing the management of Debian bug reports from the inside of Eclipse.

Building infrastructure

• Building a new interface for Wanna-build using PostgreSQL: replace the existing setup of multiple MLDBM databases with an extensible and flexible relational database for use both by wanna-build and other tools such as web CGI scripts

I've been on Twitter for over a year and a half now. If I had to explain why I like it, and why it's so popular now, I would use an analogy:

Twitter : IRC :: Blogs : Usenet

(This applies equally to other "micro-blogging" services, but I am about to explain why I believe that's not the right metaphor. You may also substitute mailing lists for Usenet.) With the older media, you have a place -- a newsgroup, or a channel, that people went to, with a distinct culture, and that (mostly) weren't "owned" by anyone, but rather by the community. With the new ones, we are all sole proprietors of our own streams, and we "tune in" to the subset of people we find interesting, rather than topics we invest in. So, instead of bumping into the same person in a couple different groups, or never reading their words at all, you might find that your feeds overlap a bit more than they do with most people. This is how I find people to "follow" and blogs to read, in fact -- as my network expands, more people become loosely joined to it, and as I notice ones worth reading I add them. Is one model better than the other? Probably not. I could make an analogy to music. In theory, I'd rather read what my favorite critics have to say about a wide variety of new releases -- some of which I'd never know about otherwise -- than keep up with the discussion of bands and genres I really like, even if most people writing about them are terrible (remember, 90% of everything is crap). But I also lose that sense of community of being a "fan" of something; I no longer have a deep connection to what's going on in the fandom or the scene, which I also would never know about otherwise (some of it is just too obscure for my favorite writers to cover). In practice, it seems the new approach has been more popular, but maybe that's because more people are on the net now, and both kinds of communication are/were shaped by the technology available at the time (destinations make much more sense with limited/centralized computing resources, and aggregation makes much more sense with powerful clients and a wider, less specialized user base). Anyway. This post is not actually about social media theory or whatever you want to call it; it's about some software I have packaged. Because of all the above, I have always wished that I could use Twitter from something more like my IRC client. Like, say, my IRC client. One could abuse the concepts of IRC to make an "on the fly" channel of whoever happens to be in my feed. (I once read a blog comment somewhere complaining that Twitter could easily be implemented on any existing IRC server using one +m channel for everyone and some client-side direction of messages from all such channels to a single window. +1 for cleverness, but they did sort of miss the point of why normal people sign up for web sites rather than installing and configuring clients for obscure chat protocols.) So, I had grand plans to write a Twitter client which was an IRC server, with some clever mapping of IRC concepts and commands to their equivalents over there. I never got around to it, as I barely have enough time for anything I do now. But last month I noticed that someone else had implemented such a thing: tircd. I had seen Twitter/IRC services before, but like the official Twiter XMPP service, they were all implemented as bots, which I detest for this sort of application. Bitlbee, for example, translates various IM protocols to IRC, but only halfway -- for anything else you have to use a bot as a sort of poor man's command line. If I want a command line for Twitter, I already have several; IRC can do better. And tircd really does! It's great. You're not required to edit the config file, and there's no extra layer on top of IRC for things like logging in or adding people. I've finally packaged the latest release, which is waiting in NEW currently. I got a request for sneak preview packages, so if you want to install some unapproved .debs check this repository. I may still pick my own project back up, as Twitter itself, being a centralized service, feels like a stopgap solution on the way to to a more generalized 140-character equivalent of the, er, blogosphere as envisioned by open-source projects like Identi.ca. In the future, perhaps, when we use a certain micro-blogging "service" we might be randomly connected to one of any number of servers run by different individuals but all mirroring messages back and forth to each other. Which, now that I think of it, sounds vaguely like some obscure, obsolete chat and news-posting protocols I know.

Nearly a month has elapsed since Debian released Lenny. The interesting thing for me about this release was that I failed to get to the point where running etch was causing me pain. This became even more evident when Katherine commented on my Lenny T-Shirt - I'd forgotten to mention our release to her! Previously there have been a bunch of features coming that I really wanted, sometimes to the point of using backports. It's probably something to do with the way I run Debian; I use stable for remote machines and while in the past this would have meant quite a few work boxes these days it's just a box at my parents' and my colo. Where I felt the pain during this release cycle was on the boxes where I run unstable (my home desktop) or testing (my work box and my laptops). Development on new shininess slows down significantly while we're preparing for release (not unsurprisingly). And there are things coming up I quite like the look of, largely aimed at the desktop arena. I've been running Network Manager 0.7 from experimental on my EEE, for example - I need the HSO 3G support to avoid unnecessary faff on a daily basis. I've got ATI cards in my work and home desktops, so I'm watching the Mesa developments with interest - roll on OpenGL 2.0 support hopefully. The Intel KMS support that doesn't seem to be quite there yet but RSN, no, really, will be nice for my laptops. Experimental VDRs are looking interesting and perhaps worth upgrading my PVR to.

For a long time I haven't really understood the point of view that Debian shouldn't release. Or rather, I haven't understood why people would think that was a serious option. The first Debian box I installed went into a colo facility shortly afterwards. This is the first release I've not been running Debian on a commercial colo box and while I treat my personal colo in a similar fashion it's not quite the same in terms of priorities. However I still think regular releases are hugely important to the project. Some pain for those of us more interested in new shiny is a small price to pay for producing a flexible, reliable distribution, be they server administrators (though I see a worrying number of people decide to install testing on their production servers and then rarely/never update it), derivers or appliance vendors.

Mind you, now we've released I'm going to have to do all those things I've been putting off. I should probably start with moving my Perl bits into pkg-perl. And maybe I should switch over some sources.list files to point at squeeze instead of lenny...