Since I already shared it elsewhere here is my KVM-launcher, and the mercurial repository it lives in. I'll add my kvm-shell program later - the tools I've written so far is mostly standalone, rather than a package. This is almost a content-free post, but I can pretend it isn't because I'm testing a new theme on my blog. The theme is included in the new release of my chronicle blog compiler which was released yesterday. ObFilm: Heathers

In one of the comments to my last blog entry, an anonymous commenter writes:

You seem to be taking a different perspective to linus on the adapting to the the disk technology front (Linus seems to against having to have the OS know about disk boundaries and having to do levelling itself)

That s an interesting question, and I figure it s worth its own top-level entry, as opposed to a reply in the comment stream. One of the interesting design questions in any OS or Computer Architecture is where the abstraction boundaries should be drawn and which side of an abstraction boundary should various operations be pushed. Linus s arguments is that there a flash controller can do a better job of wear leveling, including detecting how worn a particular flash cell might be (for example, perhaps by looking at the charge levels at an analog level and knowing when the last time the cell was programmed), and so it doesn t make sense to try to do wear leveling in a flash file system. Some responsibilities of flash management, such as coalescing newly written blocks into erase blocks to avoid write amplification can be done either on the SSD or in the file system for example, by using a log-structured file system, or some other copy-on-write file system, instead of a rewrite-in-place style file system, you can essentially solve the write amplification problem. In some cases, it s necessary let additional information leak across the abstraction for example, the ATA TRIM command is a way for the file system to let the disk know that certain blocks no longer need to be used. If too much information needs to be pushed across the abstraction, one way or another, then maybe we need to rethink whether the abstraction barrier is in the right place. In addition, if the abstraction has been around for a long time, changing it also has costs, which has to be taken into account. The 512 byte sector LBA abstraction has been around long time, and therefore dislodging it is difficult and costly. For example, the same argument which says that because the underlying hardware details are changing between different generations of SSD is all of these details should be hidden in hardware, was also used to justify something that has been a complete commercial failure for years if not decades: Object Based Disks. One of the arguments of OBD s was that the hard drive has the best knowledge of how and where to store an contiguous stream of bytes, and so perhaps filesystems should not be trying to decide where on disk an inode should be stored, but instead tell the hard drive, I have this object, which is 134 kilobytes long; please store it somewhere on the disk . At least in theory the HDD or SSD could handle all of the details of knowing the best place to store the object on the spinning magnetic media or flash media, taking into account how worn the flash is and automatically move the object around in the case of an SSD, and in the case of the HDD, the drive could know about (real) cylinder and track boundaries, and store the object in the most efficient way possible, since the drive has intimate knowledge about the low-level details of how data is stored on the disk. This theory makes a huge amount of sense; but there s only one problem. Object Based Disks have been proposed in academia and advanced R&D shops of companies like Seagate et.al. have been proposing them for over a decade, with absolutely nothing to show for it. Why? There have been two reasons proposed. One is that OBD vendors were too greedy, and tried to charge too much money for OBD s. Another explanation is that the interface abstraction for OBD s was too different, and so there wasn t enough software or file systems or OS s that could take advantage of OBD s. Both explanations undoubtedly contributed to the commercial failure of OBD s, but the question is which is the bigger reason. And the reason why it is particularly important here is because at least as far as Intel s SSD strategy is concerned, its advantage is that (modulo implementation shortcomings such as the reported internal LBA remapping table fragmentation problem and the lack of ATA TRIM support) filesystems don t need to change (much) in order to take advantage of the Intel SSD and get at least decent performance. However, if the price delta is a stronger reason for its failure, then the X25-M may be in trouble. Currently the 80GB Intel X25-M has a street price of $400, so it costs roughly $5 per gigabyte. Dumb MLC SATA SSD s are available for roughly half the cost/gigabyte (64 GB for $164). So what does the market look like 12-18 months from now? If dumb SSD s are still available at 50% of the cost of smart SSD s, it would probably be worth it to make a copy-on-write style filesystem that attempts to do the wear leveling and write amplification reduction in software. Sure, it s probably more efficient to do it in hardware, but a 2x price differential might cause people will settle for a cheaper solution even if isn t the absolutely best technical choice. On the hand, if prices drop significantly, and/or dumb SSD s completely disappear from the market, then time spent now optimizing for dumb SSD s will be completely wasted. So for Linus to make the proclamation that it s completely stupid to optimize for dumb SSD s seems to be a bit premature. Market externalities for example, does Intel have patents that will prevent competing smart SSD s from entering the market and thus forcing price drops? could radically change the picture. It s not just a pure technological choice, which is what makes projections and prognostications difficult. As another example, I don t know whether or not Intel will issue a firmware update that adds ATA TRIM support to the X25-M, or how long it will take before such SSD s become available. Until ATA TRIM support becomes available, it will be advantageous to add support in ext4 for a block allocator option that aggressively reuses blocks above all else, and avoids using blocks that have never been allocated or used before, even if it causes more in-file system fragmentation and deeper extent allocation trees. The reason for this is at the moment, once a block is used by the file system, at least today, the X25-M has absolutely no idea whether we still care about the contents of that block, or whether the block has since been released when the file was deleted. However, if 20% of the SSD s blocks have never been used, the X25-M can use 20% of the flash for better garbage collection and defragmentation algorithms. And if Intel never releases a firmware update to add ATA TRIM support, then I will be out $400 out of my own pocket for an SSD that lacks this capability, and so adding a block allocator which works around limitations of the X25-M probably makes sense. If it turns out that it takes two years before disks that have ATA TRIM support show up, then it will definitely make sense to add such an optimization. (Hard drive vendors have been historically S-L-O-W to finish standardizing new features and then letting such features enter the market place, so I m not necessarily holding my breath; after all, the Linux block device layer and and file systems have been ready to send ATA TRIM support for about six months; what s taking the ATA committees and SSD vendors so long? <grin> On the other hand, if Intel releases ATA TRIM support next month, then it might not be worth my effort to add such a mount option to ext4. Or maybe Sandisk will make an ATA TRIM capable SSD available soon, and which is otherwise competitive with Intel, and I get a free sample, but it turns out another optimization on Sandisk SSD s will give me an extra 10% performance gain under some workloads. Is it worth it in that case? Hard to tell, unless I know whether such a tweak addresses an optimization problem which is fundamental, and whether or not such a tweak will either be unnecessary, or perhaps actively unhelpful in the next generation. As long as SSD manufacturers force us treat these devices as black boxes, there may be a certain amount of cargo cult science which may be forced upon us file system designers or I guess I should say, in order to be more academically respectable, we will be forced to rely more on empirical measurements leading to educated engineering estimations about what the SSD is doing inside the black box . Heh. Related posts (automatically generated):

1. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...

In one of the comments to my last blog entry, an anonymous commenter writes:

You seem to be taking a different perspective to linus on the adapting to the the disk technology front (Linus seems to against having to have the OS know about disk boundaries and having to do levelling itself)

That s an interesting question, and I figure it s worth its own top-level entry, as opposed to a reply in the comment stream. One of the interesting design questions in any OS or Computer Architecture is where the abstraction boundaries should be drawn and which side of an abstraction boundary should various operations be pushed. Linus s arguments is that there a flash controller can do a better job of wear leveling, including detecting how worn a particular flash cell might be (for example, perhaps by looking at the charge levels at an analog level and knowing when the last time the cell was programmed), and so it doesn t make sense to try to do wear leveling in a flash file system. Some responsibilities of flash management, such as coalescing newly written blocks into erase blocks to avoid write amplification can be done either on the SSD or in the file system for example, by using a log-structured file system, or some other copy-on-write file system, instead of a rewrite-in-place style file system, you can essentially solve the write amplification problem. In some cases, it s necessary let additional information leak across the abstraction for example, the ATA TRIM command is a way for the file system to let the disk know that certain blocks no longer need to be used. If too much information needs to be pushed across the abstraction, one way or another, then maybe we need to rethink whether the abstraction barrier is in the right place. In addition, if the abstraction has been around for a long time, changing it also has costs, which has to be taken into account. The 512 byte sector LBA abstraction has been around long time, and therefore dislodging it is difficult and costly. For example, the same argument which says that because the underlying hardware details are changing between different generations of SSD is all of these details should be hidden in hardware, was also used to justify something that has been a complete commercial failure for years if not decades: Object Based Disks. One of the arguments of OBD s was that the hard drive has the best knowledge of how and where to store an contiguous stream of bytes, and so perhaps filesystems should not be trying to decide where on disk an inode should be stored, but instead tell the hard drive, I have this object, which is 134 kilobytes long; please store it somewhere on the disk . At least in theory the HDD or SSD could handle all of the details of knowing the best place to store the object on the spinning magnetic media or flash media, taking into account how worn the flash is and automatically move the object around in the case of an SSD, and in the case of the HDD, the drive could know about (real) cylinder and track boundaries, and store the object in the most efficient way possible, since the drive has intimate knowledge about the low-level details of how data is stored on the disk. This theory makes a huge amount of sense; but there s only one problem. Object Based Disks have been proposed in academia and advanced R&D shops of companies like Seagate et.al. have been proposing them for over a decade, with absolutely nothing to show for it. Why? There have been two reasons proposed. One is that OBD vendors were too greedy, and tried to charge too much money for OBD s. Another explanation is that the interface abstraction for OBD s was too different, and so there wasn t enough software or file systems or OS s that could take advantage of OBD s. Both explanations undoubtedly contributed to the commercial failure of OBD s, but the question is which is the bigger reason. And the reason why it is particularly important here is because at least as far as Intel s SSD strategy is concerned, its advantage is that (modulo implementation shortcomings such as the reported internal LBA remapping table fragmentation problem and the lack of ATA TRIM support) filesystems don t need to change (much) in order to take advantage of the Intel SSD and get at least decent performance. However, if the price delta is a stronger reason for its failure, then the X25-M may be in trouble. Currently the 80GB Intel X25-M has a street price of $400, so it costs roughly $5 per gigabyte. Dumb MLC SATA SSD s are available for roughly half the cost/gigabyte (64 GB for $164). So what does the market look like 12-18 months from now? If dumb SSD s are still available at 50% of the cost of smart SSD s, it would probably be worth it to make a copy-on-write style filesystem that attempts to do the wear leveling and write amplification reduction in software. Sure, it s probably more efficient to do it in hardware, but a 2x price differential might cause people will settle for a cheaper solution even if isn t the absolutely best technical choice. On the hand, if prices drop significantly, and/or dumb SSD s completely disappear from the market, then time spent now optimizing for dumb SSD s will be completely wasted. So for Linus to make the proclamation that it s completely stupid to optimize for dumb SSD s seems to be a bit premature. Market externalities for example, does Intel have patents that will prevent competing smart SSD s from entering the market and thus forcing price drops? could radically change the picture. It s not just a pure technological choice, which is what makes projections and prognostications difficult. As another example, I don t know whether or not Intel will issue a firmware update that adds ATA TRIM support to the X25-M, or how long it will take before such SSD s become available. Until ATA TRIM support becomes available, it will be advantageous to add support in ext4 for a block allocator option that aggressively reuses blocks above all else, and avoids using blocks that have never been allocated or used before, even if it causes more in-file system fragmentation and deeper extent allocation trees. The reason for this is at the moment, once a block is used by the file system, at least today, the X25-M has absolutely no idea whether we still care about the contents of that block, or whether the block has since been released when the file was deleted. However, if 20% of the SSD s blocks have never been used, the X25-M can use 20% of the flash for better garbage collection and defragmentation algorithms. And if Intel never releases a firmware update to add ATA TRIM support, then I will be out $400 out of my own pocket for an SSD that lacks this capability, and so adding a block allocator which works around limitations of the X25-M probably makes sense. If it turns out that it takes two years before disks that have ATA TRIM support show up, then it will definitely make sense to add such an optimization. (Hard drive vendors have been historically S-L-O-W to finish standardizing new features and then letting such features enter the market place, so I m not necessarily holding my breath; after all, the Linux block device layer and and file systems have been ready to send ATA TRIM support for about six months; what s taking the ATA committees and SSD vendors so long? <grin> On the other hand, if Intel releases ATA TRIM support next month, then it might not be worth my effort to add such a mount option to ext4. Or maybe Sandisk will make an ATA TRIM capable SSD available soon, and which is otherwise competitive with Intel, and I get a free sample, but it turns out another optimization on Sandisk SSD s will give me an extra 10% performance gain under some workloads. Is it worth it in that case? Hard to tell, unless I know whether such a tweak addresses an optimization problem which is fundamental, and whether or not such a tweak will either be unnecessary, or perhaps actively unhelpful in the next generation. As long as SSD manufacturers force us treat these devices as black boxes, there may be a certain amount of cargo cult science which may be forced upon us file system designers or I guess I should say, in order to be more academically respectable, we will be forced to rely more on empirical measurements leading to educated engineering estimations about what the SSD is doing inside the black box . Heh. Related posts (automatically generated):

1. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...

As if there weren't already enough reasons not to use out-of-office replies, here's a new one I haven't seen before (from a web design company, I think):

I am out of the office until Monday, 9 March. For LargeCorpA, please contact Person A at person.a@example.com. For LargeCorpB, SmallCompany1, or SmallCompany2, please contact Person B at person.b@example.com.

Well, I now know several companies who are clients of this web design company that I didn't know before. So, there's a nice corporate info gathering trick -- send e-mail to known addresses at your competitors on a regular basis (especially during holiday-prone times, like school holidays and Christmas/New Year) and see if they'll give you lists of their clients for free, like this helpful person has.

My message Intent to package a Debian control file grepper to WNPP and debian-devel is today a decade old. Apparently, the message predates the invention of the acronym ITP for Intent to Package (the first instance I can find is from May 1999). The changelog reveals that the first upload was on March 1st, 1999. There is unfortunately no record of when the package hit unstable the first time, since dinstall did not send Installed (or even the later Accepted) mails to an archived mailing list at that time. I suppose I might have it in some old private email archive, but more likely it s just gone. Similarly, the first fixed bug (#35527) predates BTS archiving and is now lost. The grep-dctrl program came out of repeated awkward grepping of the dpkg available and status files. Eventually I decided there must be a better way, and failed to find any canned solutions (I was later pointed to sgrep, which didn t look useful enough, and even later to dpkg-awk, but I had already committed to my own solution by then). I wrote a simple C program that processed these files as a sequence of records and did simple substring searches in each record. I rapidly added support for field selection, regular expressions and output field selection. By version 1.3a of March 2000 (which was released with Debian 2.2 potato ), the program was as good as it was going to get with one exception. Make disjunctive searches possible, said my TODO file those days. Conjunctive searches (that is, AND-searches) were possible even then by using more than one grep-dctrl command in a pipeline. Disjunctive (OR) was not possible. The problem was not so much that it would be hard to program (although the program s internal structure wasn t very good, to be honest, making extensive changes difficult), it was more a problem of coming up with a good command line syntax. Another thing that bothered me with the old grep-dctrl was how to implement Ben Armstrong s feature request. Again, the programming part wasn t the problem, the problem was coming up with a good, clean semantics for the feature. It was finally the appearance of ara in 2003 that got me moving again. Ara s author proudly compared eir program to grep-dctrl, claiming that my program did not do disjunctive searches while ara does. Competition being good for the soul, I took it as a challenge. In April 2003 I announced a complete rewrite of grep-dctrl, which was completed in January 2004 (the 2.0 release). The rewrite changed the way the command line was handled even though the usual Unix switch style is still used, the command line is regarded as a language with a parser (first an operator-precedence parser, then a recursive-descent one). The command line is transformed into an interpreted stack-based language which drives the actual grepping. The rewrite also generalised the internal data structures into an internal library which could be used to write other tools. The first such tool was sort-dctrl (introcuded in 2.7, June 2005), which was soon followed by tbl-dctrl (2.8, July 2005). The later appearance of join-dctrl (2.11, August 2007) finally allowed me to close Ben Armstrong s longstanding feature request mentioned above. The unpronounceable part of the names, dctrl , is an abbreviation for Debian control , which I decided to call the file format used by dpkg. Some people call it a RFC-822 format, but that really is a misnomer, since the differences between dctrl and RFC-822 outweigh the mainly superficial similarities (and the historical connection). I did consider calling my program dpkg-grep, but I didn t feel like I had the right to invade the dpkg namespace. The later rename to dctrl-tools reflects the fact that there are now several tools, grep-dctrl being just the oldest. I have several plans for the dctrl-tools suite, but my time and energy are mostly claimed by other responsibilities. The suite is currently team-maintained, but unfortunately the team is not very active. I would love it if I weren t the most active one with my busy schedule! Feel free to pop in on the dctrl-tools-devel mailing list, and to look at the Git repository and the todo list. If you decide to participate, please follow the rules.

It s no secret for those who follow me on Twitter/identi.ca that I have been toying with virtualization technologies for some time now. I have been using Xen in production for some time and, apart from not thinking it s the way to go for future virtualization, it s quite usable right now, specially if you are going to use Debian Lenny as the dom0, as it now has a more modern kernel than the dreaded 2.6.18 we Xen users were forced to keep using. However, I have been keeping my eyes on KVM as well, as I always liked how much simpler and well integrated into the Linux kernel it is. I m even subscribed to the KVM development mailing list, even not understanding most of the things the developers are talking about there. It s no problem for me, I just want to know what s being worked on and what s on the pipeline. As a KVM fan, I m also using it intensively as a tool for prototyping servers. It s easy enough to set up a new Debian server to test things on and learn new technologies, as well as troubleshoot problems without intefering with production environments. Althought KVM is not as good as Xen when it comes to performance, it s quickly improving every single day. And, for those of you who still believes KVM is only about full virtualization, I m happy to say that it has come a long way and paravirtualization has started to infiltrate KVM land as well. Today, KVM already sports paravirtualized clock (pvclock), paravirtualized memory management unit (pv MMU) and VirtIO drivers. Actually, it seems that KVM these days is the biggest user of VirtIO, maybe only loosing the leadership to Rusty s lguest. By using VirtIO s disk and network drivers (virtio_blk and virtio_net, respectively, and its associated modules), KVM can deliver a much improved I/O experience than when using QEMU s emulated drivers. For this reason, it s always preferable to use VirtIO drivers whenever possible when setting up your KVM guests. During the Etch lifetime, one will need to do some tricks in order to use VirtIO when using Etch as a guest under KVM. However, when preparing Lenny s d-i, the developers were smart enough to add to it virtual disk detection support. What it means is that now, starting with Lenny, d-i will recognize that it s being given a VirtIO block device and automatically load the needed kernel modules to support it. Also, the disk detection and partition modules (partman et all) were modified to show a detected virtual disk (i.e. /dev/vdX) and let the user partition it, as well as grub-installer was changed to allow GRUB to be installed onto a virtual disk s MBR, effectively making d-i a really powerful virtualization aware installer. Here you can see Lenny s d-i showing a detected virtual disk, named vda : Brazilian Portuguese readers could see the version using brazilian portuguese texts on the screen here. Sure, d-i also received a lot of improvements and special support for installing Lenny as a Xen domU was added as well, but I haven t played with it yet, so I won t comment on that right now. And, hey, one can even use virt-install and virt-manager to deploy KVM guests under Lenny And you know what gives me even more confidence that KVM will be a first class citizen inside Debian ? The fact that Steve Kemp is toying with the idea to change his xen-hosting.org project so it would become a new kvm-hosting.org project. Maybe I m praying for the preacher here, but Steve is very well well know for being the author of xen-tools and xen-shell, as well as being the creator of a number of other nice free softwares. And, judging by the comments on his post about the future kvm-hosting.org project, it seems that Steve maybe will need to update xen-tools and xen-shell to account for KVM or create a new set of tools dedicated exclusively to KVM based on his past experience creating the current tools for Xen. Good times ahead for those of us who are surfing the Debian Virtualization wave, indeed

I ve been using git this last few days and I am still working on a workflow for my projects. Unfortunately, as others have noticed, git violates POLS is so many ways, it ends up being hard to get. Creating a remote repository seems to be the first thing to bite a developer switching to git (mainly if coming from a centralized SCM). I have not decided which is the best way of doing it, but I ve been using git-daemon via inetd and a path in my remote hosting holding all my repositories for public pulling, and ssh for pushing. Here is how to create it:

local$ ssh spectra@remotehost
remotehost$ mkdir /var/git/myproject.git
remotehost$ cd /var/git/myproject.git
remotehost$ git --bare init 
remotehost$ logout
local$ cd myproject
local$ git remote add remotehost spectra@remotehost:/var/git/myproject.git
local$ git push remotehost master
...
otherlocal$ git clone git://remotelocal/myproject.git
otherlocal$ cd myproject
otherlocal$ git remote add remotehost spectra@remotehost:/var/git/myproject.git
(hack hack hack)
otherlocal$ git push remotehost master

Now everybody can pull your repository at git://remotehost/myproject.git and you can push and pull to it via ssh. Note that you have to setup git-daemon, which is pretty straight forward. I am using it as an inetd daemon, but you can use it as a standalone one. Debian has a package which does just that. Now, some people think logging in a remote server just to create an empty repository is too much. Well repositories are just .git directories. It happens that you can push for the first time by rsyncing your .git with a remote host:

local$ cd myproject
local$ rsync -a .git/ spectra@remotehost:/var/git/myproject.git
local$ git remote add remotehost spectra@remotehost:/var/git/myproject.git
(hack hack hack)
local$ git push remotehost master

(Apparently you can push using rsync every time, but it s regarded as wiser to keep your probably crappy local repository commits separated from the public repository otherwise commit messages like Please, don t use this code are likely to pop up everywhere :) ). Now, I don t know if this have side effects, but it works :-) Another thing to notice is that git is more directed at pulling than pushing. This may be because of its designer: the way Linus works is by pulling changes from others repositories and not by letting others push em into his one. And this is another violation of POLS for most of the people, who is used to commit their changes into some remote repository. Rather than that, people using git would expose their own repositories in order to have it pulled by others. I also agree with most of the git critics wrt git commands There s a lot of examples and I am not going deeper in this but I think it was a bad choice calling checkout what git does when told to checkout , for instance. Yes, I know different tools, different ways of seeing it but everyone was already used to what centralized SCMs call their operations, and I think it would only help git adopting same names for the same operations, and inventing new ones for those proper of decentralized operations. Anyway, once you get it (and I have not completely got it yet), it seems all flow as expected. If this adaptation fails, there s still Easy Git to the rescue! One last thing that I think contribute to the git sucks effect: git-svn. This is a great tool, but it was built from git s point of view Given it s intended as a glue for Subversion newcomers, it would benefit more from being built from svn s point of view. This was mentioned by a colleague developer in my company, when he just couldn t understand why one have to git svn rebase instead of a simple update. So git-svn also suffers of the bad command naming habit git do. Of course, that given that you came from this environment (I am sure git and git-svn makes perfect sense for Linus & cia :-) ). I have not tried yet, but yap seems to be targeted on providing an alternative to git-svn.

The recent issue of Frontline has an interview of Stallman. It is indeed a nice thing to see that nationally circulated news magazines in India are talking about Mukta Software. However, it is disappointing to see the FSF take one more pot-shot at Linus Torvalds.

Torvalds said he rejects GNU GPL Version 3 because he approves of tivoization, because he does not agree that users deserve the freedom to change their own copies of software.

By choosing the GPL for his own code Linus Torvalds gave users the freedom to change their own copies of Linux. This certainly seems at odds with what is quoted. There are many different points of view about how the cause of mukta software and its users can be strengthened. I think we can agree that it is certainly not strengthened by shooting each other in the foot. The GPLv3 bolsters-up one aspect of this cause: use the legal system to make if difficult (if not impossible) to take free software and put it in proprietary bottles. Linus Torvalds and all his "show me the code" buddies are addressing another aspect; to ensure that there are enough people accomplished in the task of reading, improving and writing high quality free software. The West has traditionally had more success with the law. When free programs was written in the 70's and 80's by people who did not care to preserve its freedom with a free license, this code was gobbled up and re-gurgitated as proprietary software. Once free licenses became commonplace in the 90's, the West has had a resurgence of high quality public code and coders. Unfortunately, some of the latter tend to look down on "mere users". Over here, in India, we have little faith in the legal system. (For reasons, see this report and a more recent one). We know that freedom is hard fought even when the law is technically on your side. At the same time, the brahminical approach of knowledge being safe-guarded by the self-proclaimed elite was what rang the death knell for Indian knowledge systems in the past. So it is important that the breed of skeptical people in India who tinker with their computers and software (and everything else) grows.

We have far too many cattle who will follow any flute-playing good-looking young person who is willing to fight their demons for them.

Ok, my last version was a pure Bash solution: working, but slow. There were some comments how to improve the performance and I decided finally to reimplement the second script as Python script.
The Bash script didn't finish within a day. The Python script ends after 1-2 hours in my test scenario. So, here are the scripts again:

make-lsLR.sh - call this regularly (cron) to create the needed files that are stored in /root/. Of course you can alter the location easily and exclude other directories from being scanned. [download make-lsLR.sh]


check_lost+found.py - The second script is to be run when your fsck managed to mess up with your files and stored them into lost+found directory. It takes 3 arguments: 1) the source directory where your messed up lost+found directory is, 2) the target directory to which the data will be saved and 3) a switch to actually make it happen instead of a dry-run. [download check_lostfound.py.sh]


I've chosen to copy the files to a different place instead of moving them within the same filesystem to their original place for safety reasons. Primary goal is to retrieve the files from lost+found, not to replace a full featured backup and restore application. Because of this the script doesn't handle hard- nor symlinks correctly. It just copy files.

Of course there's still room for improvements, like handling hard-/symlinks correctly or using inode number instead of md5sums to move data back to its prior location. But it works for me[tm] well enough in this way, so I'm satisfied so far. You're welcome, though, to improve this piece of ugliness if you like.

Maybe someone else finds this usefull as well. Use it on your own risk, of course. :)

The website for Tourism Queensland's extremely popular promotion to find someone to be paid $150,000 to be an island caretaker for 6 months is hosted by a Canberra company on a server in Canberra. So much for supporting local business. (Small world, one of the Executive Directors was briefly my manager at CyberTrust, but that's the Canberra IT scene for you)

Can you tell there's a whole linux machine loaded with hundreds of albums in there? Well, the USB ports on the front panel, and inside the cassete tape deck are a bit of a giveaway, as are all the blinkenlights. Details here. I'm still suprised I got it all squeezed in, put together, and working. Some things fit better than I had any right to expect:

tiny airgap between the QNAP computer and front panel control board

I may be wrong on this but it looks like Microsoft SMTP clients (at least Windows Mail and Outlook) don t like being sent a large volume of SSL certificate information when opening a TLS connection. They appear to assume that the data they are being sent is malformed and assume that STARTTLS failed, continuing with an unencrypted SMTP dialogue. This can be triggered relatively easily on a Debian system by telling Exim to use all the certificates provided by the ca-certificates package (which is the default configuration). The Windows clients will give an unhelpful the remote end dropped the connection style error, caused by the server getting upset by the unexpected fallback to unencrypted SMTP. The server logs will show something like this:

2009-02-17 21:32:55 TLS error on connection from client.example.com (Client) [192.168.192.168] (gnutls_handshake): A TLS packet with unexpected length was received.
2009-02-17 21:33:00 SMTP protocol synchronization error (input sent without waiting for greeting): rejected connection from H=client.example.com [192.168.192.168] input="EHLO Client\r\n"

Configuring the MAIN_VERIFY_TLS_CERTIFICATES option in the Debian Exim configuration (which sets the tls_verifiy_certificates option in the actual Exim configuration) to point to something with less certificates in should avoid the issue. On the bright side, at least they re making an effort to avoid overflows.

Yesterday, I posted part 1 of how to think about compression. If you haven t read it already, take a look now, so this post makes sense. Introduction In the part 1 test, I compressed a 6GB tar file with various tools. This is a good test if you are writing an entire tar file to disk, or if you are writing to tape. For part 2, I will be compressing each individual file contained in that tarball individually. This is a good test if you back up to hard disk and want quick access to your files. Quite a few tools take this approach rdiff-backup, rdup, and backuppc are among them. We can expect performance to be worse both in terms of size and speed for this test. The compressor tool will be executed once per file, instead of once for the entire group of files. This will magnify any startup costs in the tool. It will also reduce compression ratios, because the tools won t have as large a data set to draw on to look for redundancy. To add to that, we have the block size of the filesystem 4K on most Linux systems. Any file s actual disk consumption is always rounded up to the next multiple of 4K. So a 5-byte file takes up the same amount of space as a 3000-byte file. (This behavior is not unique to Linux.) If a compressor can t shrink enough space out of a file to cross at least one 4K barrier, it effectively doesn t save any disk space. On the other hand, in certain situations, saving one byte of data could free 4K of disk space. So, for the results below, I use du to calculate disk usage, which reflects the actual amount of space consumed by files on disk. The Tools Based on comments in part 1, I added tests for lzop and xz to this iteration. I attempted to test pbzip2, but it would have taken 3 days to complete, so it is not included here more on that issue below. The Numbers Let s start with the table, using the same metrics as with part 1:

Tool	MB saved	Space vs. gzip	Time vs. gzip	Cost
gzip	3081	100.00%	100.00%	0.41
gzip -1	2908	104.84%	82.34%	0.36
gzip -9	3091	99.72%	141.60%	0.58
bzip2	3173	97.44%	201.87%	0.81
bzip2 -1	3126	98.75%	182.22%	0.74
lzma -1	3280	94.44%	163.31%	0.63
lzma -2	3320	93.33%	217.94%	0.83
xz -1	3270	94.73%	176.52%	0.68
xz -2	3309	93.63%	200.05%	0.76
lzop -1	2508	116.01%	77.49%	0.39
lzop -2	2498	116.30%	76.59%	0.39

As before, in the MB saved column, higher numbers are better; in all other columns, lower numbers are better. I m using clock seconds here on a dual-core machine. The cost column is clock seconds per MB saved. Let s draw some initial conclusions:

• lzma -1 continues to be both faster and smaller than bzip2. lzma -2 is still smaller than bzip2, but unlike the test in part 1, is now a bit slower.
• As you ll see below, lzop ran as fast as cat. Strangely, lzop -3 produced larger output than lzop -1.
• gzip -9 is probably not worth it it saved less than 1% more space and took 42% longer.
• xz -1 is not as good as lzma -1 in either way, though xz -2 is faster than lzma -2, at the cost of some storage space.
• Among the tools also considered for part 1, the difference in space and time were both smaller. Across all tools, the difference in time is still far more significant than the difference in space.

The Pretty Charts Now, let s look at an illustration of this. As before, the sweet spot is the lower left, and the worst spot is the upper right. First, let s look at the compression tools themselves: At the extremely fast, but not as good compression, end is lzop. gzip is still the balanced performer, bzip2 still looks really bad, and lzma -1 is still the best high-compression performer. Now, let s throw cat into the mix: Here s something notable, that this graph makes crystal clear: lzop was just as fast as cat. In other words, it is likely that lzop was faster than the disk, and using lzop compression would be essentially free in terms of time consumed. And finally, look at the cost: What happened to pbzip2? I tried the parallel bzip2 implementation just like last time, but it ran extremely slow. Interestingly, pbzip2 < notes.txt > notes.txt.bz2 took 1.002 wall seconds, but pbzip2 notes.txt finished almost instantaneously. This 1-second startup time for pbzip2 was a killer, and the test would have taken more than 3 days to complete. I killed it early and omitted it from my results. Hopefully this bug can be fixed. I didn t expect pbzip2 to help much in this test, and perhaps even to see a slight degradation, but not like THAT. Conclusions As before, the difference in time was far more significant than the difference in space. By compressing files individually, we lost about 400MB (about 7%) space compared to making a tar file and then combining that. My test set contained 270,101 files. gzip continues to be a strong all-purpose contender, posting fast compression time and respectable compression ratios. lzop is a very interesting tool, running as fast as cat and yet turning in reasonable compression though 25% worse than gzip on its default settings. gzip -1 was almost as fast, though, and compressed better. If gzip weren t fast enough with -6, I d be likely to try gzip -1 before using lzop, since the gzip format is far more widely supported, and that s important to me for backups. These results still look troubling for bzip2. lzma -1 continued to turn in far better times and compression ratios that bzip2. Even bzip2 -1 couldn t match the speed of lzma -1, and compressed barely better than gzip. I think bzip2 would be hard-pressed to find a comfortable niche anywhere by now. As before, you can download my spreadsheet with all the numbers behind these charts and the table.

If you re reading this post via the website rather than a feed/planet then you will notice that the site has gone completely black in support of the Creative Freedom Foundation s campaign against S92A of the NZ Copyright Amendment Act which is due to come into effect on 28th February 2009. I ve also joined the wave of people blacking out their avatar on Facebook/Jabber/MSN, etc. S92A introduces Guilt Upon Accusation whereby if you are accused of copyright infringement (downloading music and movies, etc) repeatedly (likely 3 or more times) you are at risk of being disconnected from the Internet by your ISP. The law does not require any proof or substantiation of the accusations and the entire process would occur outside of the courts and the established legal system. Not only does it place every user at risk, the wording is very unclear on exactly what type of organisation is considered an ISP and there is significant concern that schools, businesses, libraries and hospitals will be placed in the difficult position of determining whether their users have broken the law and require disconnection. Opposition to the law is not an attack on copyright, or a statement that we should be free to download all the movies and music that we desire. Those sorts of activities are clearly wrong, and I don t have any issue with copyright holders wanting to enforce their rights when their content is illegally copied. However, disconnecting people upon accusation, with no proof or formal legal process to prove guilt is not the right way to go about it. The fact that the law does not require proof of guilt is only the tip of the iceberg in terms of problems with it. For further background on the problems it causes for ISPs by placing them as the middle-man in copyright disputes you should refer to the following posts:

• Russel Brown of Public Address, points out that among other faults the law stipulates a penalty that no court would impose in adjudicating a copyright complaint
• David Farrar of Kiwiblog, has a guest post at the NZ Center for Political Research explaining how the law affects businesses
• Creative Freedom Foundation Background on S92

Finally, I think it is worth pointing out that S92A was removed from the proposed Amendment at the select committee stage, but was later reintroduced by Judith Tizard during the final reading of the bill. Mark Harris has an excellent post on the history of the amendment which includes facts such as the official report on the amendment also recommended removing S92A as it was unecessary given existing ISP terms and conditions which forbid illegal activity. The fact that the select committee (based on public submissions) recognised the problems with S92A and removed it, only to have it added back in again at the last stage when we no longer had any say on it really hacks me off and I cant help but feel the influence of the big money American media companies pressuring our politicians to pass a law that they don t really understand the full consequences of. So what is to be done? The Blacked Out campaign, being run by the Creative Freedom Foundation is gathering steam and international attention. Peter Dunne of United Future (who originally voted for the amendment) has declared that the amendment is wrong, and doesn t do what they thought they were voting to do, we need to convince National and the rest of the house of the same. Time is running out for this to happen before the amendment comes into effect on Feb 28th, but there is still time to write to your local MP and sign the petition against S92A Guilt Upon Accusation . The Creative Freedom Foundation site has a nice easy list of what you can do to register your protest.

After dropping all (pretense at) regular exercise after leaving high/grammar/secondary school (the german word is Gymnasium) and doing more or less nothing for more than 10 years, I've taken up weight training (after a not very successful attempt at volleyball; I like the sport but the team fell apart) two years ago. Since I actually managed to work out regularly (twice a week) in the first year, and since membership in a gym is not cheap, and since I moved and thus had to find a different solution anyway, I decided to buy a Bowflex Sport home gym. I can't compare it to other home equipment, and it apparently it has been discontinued by Nautilus, but nonetheless, a few thoughts: The conclusion first: Would I buy it again? No. Was it a waste of money? No, it's not that bad. In detail:

• The free-running seat that should allow cardio rowing is not very useful because there's no good foot-rest, and because the range of motion is too limited. This is my main grievance because being able to do cardio was one of the main reasons for chosing this engine over other offers.
• Go for the 310lbs extension at least (I did, luckily.) With leg exercises, 210lbs are just not enough (I'm 70kg/180cm and have been doing weight training two years. Certainly no muscle freak...)
• It's slightly too small. For many exercises, I'd wish for a bigger range of motion. Especially the lat tower could be 10cm higher.
• While the famous power rods haven't shown signs of wearing out after this first year, the resistance is not consistent. The 310lbs is split in rods of 5, 10, 10, 30, 50, 50lbs per side, and I've noticed that 30 + 10 + 10lbs is noticeably lighter than 50lbs. (I haven't measured at which point of the range of motion the nominal resistance applies. Resistance goes from very light at the start of the motion to hard towards the end, so the numbers don't really compare to training with real weights anyway. I haven't had any problem with this, though.)
• The documentation is lacking. Besides quality issues (wrong images, spelling), some of the stuff delivered in the box as well as some of the possible exercises are simply not used/described. (Example: it is possible to do squats, but the manual doesn't describe them.) So don't get this machine if you don't have an idea about what kinds of exercises you want to do.
• Assembling it is quite a project. But the documentation is good enough on the assembly if you have ever managed to assemble a Kinder Surprise toy, and you only do it once anyway, so it's not really an issue.

Closing remarks: I'll probably keep it for quite a while. I've thought about spending much less money and just getting a bench and a few weights, but I really like to have a lat tower. Overall I can do what I want to do with the exception of cardio workouts. The other angle is that there is quite a bit of bad press around about the founder of Nautilus, Arhur Jones. I can't comment on how trustworthy these articles are, and I can't comment on Nautilus' business practices today (Mr. Jones seems to be dead), so take this with a large pinch of salt before making a buying decision based on moral/ethical standards.

Yay! This blog post is not strictly speaking news anymore - But for those who don't know it yet, three days ago Debian 5.0 Lenny was released, after 22 months of work (plus fun, plus flamewars, plus everything that makes a Free Software lover tick). And, of course, that makes us all very happy and proud. As always, upgrading is a breeze. Hats off specially to Wolfgang and everybody who worked towards the great release notes - No, it is not a simple task, by far. And I _do_ feel ashamed I didn't even beep that way :-( So, many of us are in the middle of planning/executing our servers' migrations. So far, I'm amazed for good. Even hairy issues such as firmware removal are magically and beautifully taken care of - i.e. my firewall kindly informed me during update that I would need to install the non-free firmware for my BNX2 (Broadcom) network interfaces, and I was just a package away from absolute happiness. Lets see what happens next Friday, as I will be upgrading our storage+application server (which is _way_ more complex than the servers I've dealt with so far). Anyway... But what good is a blog post if you are not ranting? Many people recognize me as one of the most Debian-connected people in Mexico, and that's very good. And yes, besides being a Debian Developer, I am a co-sysadmin for the main Debian mirror in Mexico. Some people have already asked me for CD-ROMs and DVDs. Of course, if I had a BluRay drive, I'm sure I'd also get requests for it. People: Do you really want such media? Think again... Do you really want 31 CDs or five DVDs for your favorite architecture? Ok, maybe many will say "nah, just give me the first one" - Then, do you want to limit your Debian experience to just the software that lives on the first 1/31th (or 1/5th) of it all? Of course there are many situations where it is desirable. Low bandwidth users, or people with no regular connectivity, will be much better served by suitable media from which to install. However, most of us (computer geeks living in Mexico City - Yes, that's the people contacting me) have at least a 1Mbps connection at home. People, just get the Netinst or Businesscard (180 or 40MB) images and download whatever is left via the network. Debian is extremely network-friendly. And, believe me, even if packages are sorted by popularity (and that's why most people will be happy with the first DVD if needed), you never know if you will want precisely a package that sits towards the lonely tail of popularity. And, yes, I did have my CD images handy for the Potato, Woody and Sarge cycles - but increasingly, it became easier for me just to ask apt-get to fetch stuff from the web (which is done without me moving from my comfy chair while I do anything else) than asking apt-get to ask me to go search for the f.*ing CD which I left dont-remember-where. Anyway... I'm not saying I won't burn your CDs - If you want them, please tell me in advance and come to my office, I'll be glad to give you some Debian disks. But spare the environment. We don't need to burn more and more disks. Use the network, be a better human being!

On my Sansa Clip, a playlist that worked fine before suddenly stopped working. I couldn't find anything wrong with the list: it had DOS line separators, DOS directory separators and relative paths (all stuff that Sansa needs, unfortunately). I had generated proper EXTINF meta data lines, the list was called .m3u8 to singal it's UTF-8. And after all: it had worked before.After deleting a leftover playlist, suddenly I noticed with another playlist that had worked the day before that it was now empty. Ouch. So I investigated a bit more...At some point, I ran across a post on a similar Sansa device titled

Firmware Bug - Playlists numbers 43, 46, and 49 are always empty

This is also what I'm seeing on my device. Certain 'slots' in the playlist list don't work.No, I'm not kidding.The way I 'fixed' my non-working playlist was this: I duplicated the file. Now the duplicate is on the broken slot, and the old copy is working again. Unfortunately, this also has shifted the later playlists into other slots, so I'll have to repeat this for the other affected playlists. I guess I'll just write a script to keep duplicate copies of all my playlists, since apparently there are no two consequent slots affected.Broken slots on my Sansa Clip v2: 22, 25, 28, 34, 37, 40 (I have 59 lists currently on the player, so there might be more broken 'blocks' somewhere)

I think a desireable property of the helper stuff that debian packages wrap around the underlying program (such as mailservers, webservers, etc.) is that you are still exposed to some extend to the configuration of the underlying service. The exim4 debian "value-add" layer is so thick that despite using exim4 in Debian for more than 4 years, I have been totally insulated from the configuration and have no idea how to do even the simplest tasks. Upgrading to lenny and exim4-config will not install due to DEBCONF templates in my config file. Yes, they are there, the older exim4 packages put them there. When prompted by debconf during the initial upgrade attempt, I said "do not overwrite my local exim4 configuration, it's changed a little". Reading the resulting error messages, that was the wrong answer: Now I need domain knowledge that the debian package has kept from me for all these years to proceed at all, and the tone of NEWS.gz is that it's my own damn fault. Thanks to Brett Parker for the solution: cp /var/lib/exim4/config.autogenerated /etc/exim4/exim4.conf and finally put a stop to this poisonous relationship that the exim4 packaging and I have had for all this time.

So, after some time contributing to WebKit/GTK+ I got invited to be an official committer, and today (yesterday, technically) my account was setup. I have thus been able to commit one of my patches which were already reviewed myself. It is indeed pretty simple. I didn t want to fuck up big if I did, you know =). At least it is much more meaningful than my first patch that got committed. Being a committer means I can help speed things up a little by freeing reviewers from the work involved in commiting some of the patches, giving them more time to review new patches instead, so I am very excited about this.