In one of the comments to my last blog entry, an anonymous commenter writes:

You seem to be taking a different perspective to linus on the adapting to the the disk technology front (Linus seems to against having to have the OS know about disk boundaries and having to do levelling itself)

That s an interesting question, and I figure it s worth its own top-level entry, as opposed to a reply in the comment stream. One of the interesting design questions in any OS or Computer Architecture is where the abstraction boundaries should be drawn and which side of an abstraction boundary should various operations be pushed. Linus s arguments is that there a flash controller can do a better job of wear leveling, including detecting how worn a particular flash cell might be (for example, perhaps by looking at the charge levels at an analog level and knowing when the last time the cell was programmed), and so it doesn t make sense to try to do wear leveling in a flash file system. Some responsibilities of flash management, such as coalescing newly written blocks into erase blocks to avoid write amplification can be done either on the SSD or in the file system for example, by using a log-structured file system, or some other copy-on-write file system, instead of a rewrite-in-place style file system, you can essentially solve the write amplification problem. In some cases, it s necessary let additional information leak across the abstraction for example, the ATA TRIM command is a way for the file system to let the disk know that certain blocks no longer need to be used. If too much information needs to be pushed across the abstraction, one way or another, then maybe we need to rethink whether the abstraction barrier is in the right place. In addition, if the abstraction has been around for a long time, changing it also has costs, which has to be taken into account. The 512 byte sector LBA abstraction has been around long time, and therefore dislodging it is difficult and costly. For example, the same argument which says that because the underlying hardware details are changing between different generations of SSD is all of these details should be hidden in hardware, was also used to justify something that has been a complete commercial failure for years if not decades: Object Based Disks. One of the arguments of OBD s was that the hard drive has the best knowledge of how and where to store an contiguous stream of bytes, and so perhaps filesystems should not be trying to decide where on disk an inode should be stored, but instead tell the hard drive, I have this object, which is 134 kilobytes long; please store it somewhere on the disk . At least in theory the HDD or SSD could handle all of the details of knowing the best place to store the object on the spinning magnetic media or flash media, taking into account how worn the flash is and automatically move the object around in the case of an SSD, and in the case of the HDD, the drive could know about (real) cylinder and track boundaries, and store the object in the most efficient way possible, since the drive has intimate knowledge about the low-level details of how data is stored on the disk. This theory makes a huge amount of sense; but there s only one problem. Object Based Disks have been proposed in academia and advanced R&D shops of companies like Seagate et.al. have been proposing them for over a decade, with absolutely nothing to show for it. Why? There have been two reasons proposed. One is that OBD vendors were too greedy, and tried to charge too much money for OBD s. Another explanation is that the interface abstraction for OBD s was too different, and so there wasn t enough software or file systems or OS s that could take advantage of OBD s. Both explanations undoubtedly contributed to the commercial failure of OBD s, but the question is which is the bigger reason. And the reason why it is particularly important here is because at least as far as Intel s SSD strategy is concerned, its advantage is that (modulo implementation shortcomings such as the reported internal LBA remapping table fragmentation problem and the lack of ATA TRIM support) filesystems don t need to change (much) in order to take advantage of the Intel SSD and get at least decent performance. However, if the price delta is a stronger reason for its failure, then the X25-M may be in trouble. Currently the 80GB Intel X25-M has a street price of $400, so it costs roughly $5 per gigabyte. Dumb MLC SATA SSD s are available for roughly half the cost/gigabyte (64 GB for $164). So what does the market look like 12-18 months from now? If dumb SSD s are still available at 50% of the cost of smart SSD s, it would probably be worth it to make a copy-on-write style filesystem that attempts to do the wear leveling and write amplification reduction in software. Sure, it s probably more efficient to do it in hardware, but a 2x price differential might cause people will settle for a cheaper solution even if isn t the absolutely best technical choice. On the hand, if prices drop significantly, and/or dumb SSD s completely disappear from the market, then time spent now optimizing for dumb SSD s will be completely wasted. So for Linus to make the proclamation that it s completely stupid to optimize for dumb SSD s seems to be a bit premature. Market externalities for example, does Intel have patents that will prevent competing smart SSD s from entering the market and thus forcing price drops? could radically change the picture. It s not just a pure technological choice, which is what makes projections and prognostications difficult. As another example, I don t know whether or not Intel will issue a firmware update that adds ATA TRIM support to the X25-M, or how long it will take before such SSD s become available. Until ATA TRIM support becomes available, it will be advantageous to add support in ext4 for a block allocator option that aggressively reuses blocks above all else, and avoids using blocks that have never been allocated or used before, even if it causes more in-file system fragmentation and deeper extent allocation trees. The reason for this is at the moment, once a block is used by the file system, at least today, the X25-M has absolutely no idea whether we still care about the contents of that block, or whether the block has since been released when the file was deleted. However, if 20% of the SSD s blocks have never been used, the X25-M can use 20% of the flash for better garbage collection and defragmentation algorithms. And if Intel never releases a firmware update to add ATA TRIM support, then I will be out $400 out of my own pocket for an SSD that lacks this capability, and so adding a block allocator which works around limitations of the X25-M probably makes sense. If it turns out that it takes two years before disks that have ATA TRIM support show up, then it will definitely make sense to add such an optimization. (Hard drive vendors have been historically S-L-O-W to finish standardizing new features and then letting such features enter the market place, so I m not necessarily holding my breath; after all, the Linux block device layer and and file systems have been ready to send ATA TRIM support for about six months; what s taking the ATA committees and SSD vendors so long? <grin> On the other hand, if Intel releases ATA TRIM support next month, then it might not be worth my effort to add such a mount option to ext4. Or maybe Sandisk will make an ATA TRIM capable SSD available soon, and which is otherwise competitive with Intel, and I get a free sample, but it turns out another optimization on Sandisk SSD s will give me an extra 10% performance gain under some workloads. Is it worth it in that case? Hard to tell, unless I know whether such a tweak addresses an optimization problem which is fundamental, and whether or not such a tweak will either be unnecessary, or perhaps actively unhelpful in the next generation. As long as SSD manufacturers force us treat these devices as black boxes, there may be a certain amount of cargo cult science which may be forced upon us file system designers or I guess I should say, in order to be more academically respectable, we will be forced to rely more on empirical measurements leading to educated engineering estimations about what the SSD is doing inside the black box . Heh. Related posts (automatically generated):

1. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...

In one of the comments to my last blog entry, an anonymous commenter writes:

You seem to be taking a different perspective to linus on the adapting to the the disk technology front (Linus seems to against having to have the OS know about disk boundaries and having to do levelling itself)

That s an interesting question, and I figure it s worth its own top-level entry, as opposed to a reply in the comment stream. One of the interesting design questions in any OS or Computer Architecture is where the abstraction boundaries should be drawn and which side of an abstraction boundary should various operations be pushed. Linus s arguments is that there a flash controller can do a better job of wear leveling, including detecting how worn a particular flash cell might be (for example, perhaps by looking at the charge levels at an analog level and knowing when the last time the cell was programmed), and so it doesn t make sense to try to do wear leveling in a flash file system. Some responsibilities of flash management, such as coalescing newly written blocks into erase blocks to avoid write amplification can be done either on the SSD or in the file system for example, by using a log-structured file system, or some other copy-on-write file system, instead of a rewrite-in-place style file system, you can essentially solve the write amplification problem. In some cases, it s necessary let additional information leak across the abstraction for example, the ATA TRIM command is a way for the file system to let the disk know that certain blocks no longer need to be used. If too much information needs to be pushed across the abstraction, one way or another, then maybe we need to rethink whether the abstraction barrier is in the right place. In addition, if the abstraction has been around for a long time, changing it also has costs, which has to be taken into account. The 512 byte sector LBA abstraction has been around long time, and therefore dislodging it is difficult and costly. For example, the same argument which says that because the underlying hardware details are changing between different generations of SSD is all of these details should be hidden in hardware, was also used to justify something that has been a complete commercial failure for years if not decades: Object Based Disks. One of the arguments of OBD s was that the hard drive has the best knowledge of how and where to store an contiguous stream of bytes, and so perhaps filesystems should not be trying to decide where on disk an inode should be stored, but instead tell the hard drive, I have this object, which is 134 kilobytes long; please store it somewhere on the disk . At least in theory the HDD or SSD could handle all of the details of knowing the best place to store the object on the spinning magnetic media or flash media, taking into account how worn the flash is and automatically move the object around in the case of an SSD, and in the case of the HDD, the drive could know about (real) cylinder and track boundaries, and store the object in the most efficient way possible, since the drive has intimate knowledge about the low-level details of how data is stored on the disk. This theory makes a huge amount of sense; but there s only one problem. Object Based Disks have been proposed in academia and advanced R&D shops of companies like Seagate et.al. have been proposing them for over a decade, with absolutely nothing to show for it. Why? There have been two reasons proposed. One is that OBD vendors were too greedy, and tried to charge too much money for OBD s. Another explanation is that the interface abstraction for OBD s was too different, and so there wasn t enough software or file systems or OS s that could take advantage of OBD s. Both explanations undoubtedly contributed to the commercial failure of OBD s, but the question is which is the bigger reason. And the reason why it is particularly important here is because at least as far as Intel s SSD strategy is concerned, its advantage is that (modulo implementation shortcomings such as the reported internal LBA remapping table fragmentation problem and the lack of ATA TRIM support) filesystems don t need to change (much) in order to take advantage of the Intel SSD and get at least decent performance. However, if the price delta is a stronger reason for its failure, then the X25-M may be in trouble. Currently the 80GB Intel X25-M has a street price of $400, so it costs roughly $5 per gigabyte. Dumb MLC SATA SSD s are available for roughly half the cost/gigabyte (64 GB for $164). So what does the market look like 12-18 months from now? If dumb SSD s are still available at 50% of the cost of smart SSD s, it would probably be worth it to make a copy-on-write style filesystem that attempts to do the wear leveling and write amplification reduction in software. Sure, it s probably more efficient to do it in hardware, but a 2x price differential might cause people will settle for a cheaper solution even if isn t the absolutely best technical choice. On the hand, if prices drop significantly, and/or dumb SSD s completely disappear from the market, then time spent now optimizing for dumb SSD s will be completely wasted. So for Linus to make the proclamation that it s completely stupid to optimize for dumb SSD s seems to be a bit premature. Market externalities for example, does Intel have patents that will prevent competing smart SSD s from entering the market and thus forcing price drops? could radically change the picture. It s not just a pure technological choice, which is what makes projections and prognostications difficult. As another example, I don t know whether or not Intel will issue a firmware update that adds ATA TRIM support to the X25-M, or how long it will take before such SSD s become available. Until ATA TRIM support becomes available, it will be advantageous to add support in ext4 for a block allocator option that aggressively reuses blocks above all else, and avoids using blocks that have never been allocated or used before, even if it causes more in-file system fragmentation and deeper extent allocation trees. The reason for this is at the moment, once a block is used by the file system, at least today, the X25-M has absolutely no idea whether we still care about the contents of that block, or whether the block has since been released when the file was deleted. However, if 20% of the SSD s blocks have never been used, the X25-M can use 20% of the flash for better garbage collection and defragmentation algorithms. And if Intel never releases a firmware update to add ATA TRIM support, then I will be out $400 out of my own pocket for an SSD that lacks this capability, and so adding a block allocator which works around limitations of the X25-M probably makes sense. If it turns out that it takes two years before disks that have ATA TRIM support show up, then it will definitely make sense to add such an optimization. (Hard drive vendors have been historically S-L-O-W to finish standardizing new features and then letting such features enter the market place, so I m not necessarily holding my breath; after all, the Linux block device layer and and file systems have been ready to send ATA TRIM support for about six months; what s taking the ATA committees and SSD vendors so long? <grin> On the other hand, if Intel releases ATA TRIM support next month, then it might not be worth my effort to add such a mount option to ext4. Or maybe Sandisk will make an ATA TRIM capable SSD available soon, and which is otherwise competitive with Intel, and I get a free sample, but it turns out another optimization on Sandisk SSD s will give me an extra 10% performance gain under some workloads. Is it worth it in that case? Hard to tell, unless I know whether such a tweak addresses an optimization problem which is fundamental, and whether or not such a tweak will either be unnecessary, or perhaps actively unhelpful in the next generation. As long as SSD manufacturers force us treat these devices as black boxes, there may be a certain amount of cargo cult science which may be forced upon us file system designers or I guess I should say, in order to be more academically respectable, we will be forced to rely more on empirical measurements leading to educated engineering estimations about what the SSD is doing inside the black box . Heh. Related posts (automatically generated):

1. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...

On a deployment I m currently working on, I ve seen two different cases of proprietary software use leading to both madness and sadness, which are just so typical that I can t resist to tell you. Keep in mind, for the rest of this reading, that the whole platform is running under Solaris 10 x86-64. The first case is a data quality management software that we will keep anonymous. The editor promised before the deployment began that they had a version of the software for the operating system that we would be using. Well, it turned out they do have a Solaris version for sparc, and a x86-64 version for Linux. No Solaris x86-64 version, and no way to get a rebuild in a timely fashion. The second case is a content management software that we will keep anonymous. It comes in the form of a java web application and a java application container. Part of integrating this software involves a proprietary plugin for Apache HTTPd that acts as a mix of mod_proxy_balancer, mod_disk_cache, and htcacheclean, as well as a cache invalidator. Originally, the java web application was supposed to be installed within a JBoss Application Server instead of the editor provided container, and Apache HTTPd would reverse proxy requests to the JBoss server. This means we already had an Apache HTTPd in place (latest version ; 2.2.11 at the time), and since we have x86-64 processors, it was built as a 64-bits binary. Contrary to the first case, this time, we had a Solaris x86 binary. Yes, you read correctly: x86 ; 32-bits only. After going through the pain of rebuilding Apache HTTPd in 32-bits (there are various reasons why we don t use sunfreeware software), it turned out the module was a 2.0 ABI module not compatible with the 2.2 ABI. It also turned out there was a 2.2 ABI version of the module for Solaris sparc. It finally worked after another build of Apache HTTPd, a 2.0.63 release, this time. The more you get used to free software, the more these kind of things get frustrating.

Some guy from New Zealand blogged about a vulnerability in .desktop files as used by Gnome and KDE, and claims you can write a virus with them for Linux. He actually meant a Trojan rather than a virus, but still. I'm not saying the vulnerability does not exist. It does, and the trojan he describes should work in theory. However, in practice, I believe that the real reason why there aren't any Linux viruses is not the fact that Linux is somehow 'safer', but really the fact that Linux isn't a monoculture like Windows is. If you want to write a Windows virus, you have to deal with, at most, four to five different versions of windows that are currently still in active use with most users. Target that, and your virus will live like there's no tomorrow. If you want to write a Linux virus, the range of target system is much, much larger. In his very blog entry that describes the vulnerability, 'foobar' describes already two special cases he has to consider: the fact that KDE and Gnome write .desktop entries that get executed on startup to different locations, and the fact that, while most distributions ship either curl or wget, none of them has standardized on either of those, requiring the virus writer to account for that, too. While his blog entry stops at that, there's way, way more he has to deal with. If our supposed trojan writer wants his trojan to proliferate, he will have to deal with distributions that come with or without SE Linux; distributions that ship with a /tmp mounted noexec; distributions that ship with python 2.3, 2.4, 2.5, or 3.0; users who use Thunderbird, Evolution, Kmail, mutt, pine, or another mail client; and possibly much, much more (this only describes what's required as described in the referenced blog post). All these differences would make writing some malware that would exploit the described vulnerability tedious, at best, and rather impractical in most cases. Again, I'm not saying the vulnerability does not exist; it does. But I think this lack of monoculture under Linux, not the (perceived) strength of the platform, is what helps users defend against malware on Linux.

Ok, my last version was a pure Bash solution: working, but slow. There were some comments how to improve the performance and I decided finally to reimplement the second script as Python script.
The Bash script didn't finish within a day. The Python script ends after 1-2 hours in my test scenario. So, here are the scripts again:

make-lsLR.sh - call this regularly (cron) to create the needed files that are stored in /root/. Of course you can alter the location easily and exclude other directories from being scanned. [download make-lsLR.sh]


check_lost+found.py - The second script is to be run when your fsck managed to mess up with your files and stored them into lost+found directory. It takes 3 arguments: 1) the source directory where your messed up lost+found directory is, 2) the target directory to which the data will be saved and 3) a switch to actually make it happen instead of a dry-run. [download check_lostfound.py.sh]


I've chosen to copy the files to a different place instead of moving them within the same filesystem to their original place for safety reasons. Primary goal is to retrieve the files from lost+found, not to replace a full featured backup and restore application. Because of this the script doesn't handle hard- nor symlinks correctly. It just copy files.

Of course there's still room for improvements, like handling hard-/symlinks correctly or using inode number instead of md5sums to move data back to its prior location. But it works for me[tm] well enough in this way, so I'm satisfied so far. You're welcome, though, to improve this piece of ugliness if you like.

Maybe someone else finds this usefull as well. Use it on your own risk, of course. :)

Can you tell there's a whole linux machine loaded with hundreds of albums in there? Well, the USB ports on the front panel, and inside the cassete tape deck are a bit of a giveaway, as are all the blinkenlights. Details here. I'm still suprised I got it all squeezed in, put together, and working. Some things fit better than I had any right to expect:

tiny airgap between the QNAP computer and front panel control board

The Open Source Initiative and the Free Software Foundation are probably the two authorities on what is free and open source software that are better respected than the debian project. It s fairly easy to see that the famous licences (GNU GPL) appear on both lists, but how similar are their approvals? There was a comparison at Asheeshworld Notes you will like: OSI vs FSF in 2007, but I ve noticed more and more duplicates in that listing and the source code didn t work for me. I ve captured the current differences between their actual approvals in a pseudo-diff. The generating code and intermediate steps are splattered around the same folder. Many of the differences look like a result of process differences, but I m not sure because OSI doesn t make any comments and their licence review process archive seems awkward to search. As I understand it, a lawyer advocates a licence in the OSI process, so it requires the licensor to contribute (and many licensors couldn t care less about OSI); but FSF does an independent review, so FSF has to decide it s worth reviewing. In case it s not obvious, I think FSF s independent foundation-led review is much the better of those two. There are two licences that OSI approved but FSF lists as non-free: NASA Open Source Agreement and the Reciprocal Public License. Both of these are send-back-ware where changes have to be sent back or made available to the upstream developer. Many debian developers agree such payments-in-kind are non-free (OSI s OSDefinition is based on the Debian Free Software Guidelines), but I think an early OSI advisor thought they were a good idea, so those approvals look like an OSI bug to me. It s not obvious if there are licences that FSF approves but OSI rejects. OSI doesn t seem to list licences to avoid. Is it helpful to see the differences? Would a comparison with the debian archive be useful? Are there other curiousities captured in the differences? Are people interested in reconciling the differences?

It's time to drive some buzz this way ;-) Although this post will only be a pointer towards the Spanish post I made on Planeta EDUSOL, for reasons soon to become obvious. In any case, the information I'm posting here is not exactly the same. Can you read Spanish? Please go on to the invitation for the first videoconference for the EDUSOL Seminar. This year, we the organizers of the On-line Encounter of Education and Free Software (EDUSOL) are aiming higher - we are not "just" having a two-week encounter at the end of the year - We are having an all-year-long Seminar, focusing on the collaborative construction of knowledge. People from quite distinct backgrounds will be part of this project, and we aim to drive it towards the publication of a book. We (mainly, Alejandro Miranda and me) have been quite busy bootstrapping this seminar, getting the proposed authors, thinking over the intended communication channels and ways, and setting up the needed infrastructure) and are ready to start the public-facing activities. We will be having monthly videoconferencing sessions, the last Thursday each month, 16:00-18:00 Mexico Central time (currently GMT-6; GMT-5 after the beginning of April). The VC sessions will be also relayed through Ogg streams, and we will have an IRC channel available to offer full interactivity for those who do not have access to a H.323 VC setup. This first session will be moderated by Victor Manuel Mart nez; the speakers will be Alejandro Miranda and myself - The topics we will present are:

• Short project presentation, delineating the list of invited authors and tematic lines we will pursue
• Description of the collaboration scheme we expect to hold, including how everybody (not just invited authors) can participate
• Presentation of one of the topics we will work into in the Seminar: Free Software and the Democratic Construction of the Society

If you have access to videoconferencing facilities, please get in touch with Carlos Cruz, the Videoconferences Coordinator at the Economic Research Institute, as soon as possible for all the needed coordination.

If you're not reading The Daily WTF, today's soapbox article, Programming Sucks! Or At Least, It Ought To is a good reason to at least give it a glance now and then. Essentially, it's a wake-up call to all those programmers who think that their working life isn't complete unless every project is implemented in a new language, with a new framework, or at the very least using a bunch of the latest whizz-bang techniques that they read about on the back of their weeties blog this morning. My favourite rule, and one that I really need to re-learn myself, is:

Serve the Business. Every tradesman wants to use the latest, greatest, and most powerful tools, but rarely are they appropriate for the job. Likewise, there's hardly ever a business case to immediately upgrade all platforms/libraries/languages. That 10-year old "Classic ASP" code hasn't gotten worn out, just much less fun to maintain.

The rest of his rules are also useful, and should probably be printed out and stuck up around the offices of every IS development shop.

Scott has several posts about git (1, 2, 3), and the I worked on this locally, now I want to push this to a remote place where others will be able to fetch it . I ve been using the following snippet for that. YMMV.

REPO=reponame
ROOT=/wheremygitreposare/
SERV=remoteserver
git clone --bare . /tmp/$ REPO .git && \
rsync -avzP /tmp/$ REPO .git $ SERV :$ROOT && \
git remote add origin $ SERV :$ ROOT $ REPO .git && \
git config branch.master.remote origin && \
git config branch.master.merge refs/heads/master (Please post comments if you can improve it!)
But I totally agree that this common use case is not well addressed in git.

Three things I really like have released recently:-

1. Debian GNU/Linux 5.0 ( Lenny ): I installed it on a laptop the weekend before going to a conference and I m really impressed. It installed fairly neatly and was good-looking (gtk installer from USB stick, default desktop task, in case anyone cares). I found one or two small glitches while travelling, which I ll reportbug soon, but they re very minor. Well done to everyone involved in stabilising this release!
2. The co-operative officially launched its new brand with a 2.5-minute whole-break advert using a Bob Dylan track, both of which are unusual. OK, so I feel it should have been available in a FOSS-friendly format and not done on the same day as announcing a sale of opticians or a big closure, but it s still good to see the co-operative getting big TV exposure.
3. Videos of Debian at FOSDEM and the Business of Open Source mini-conference at Linux.conf.au were released, which I ll watch soon. The current trend of recording key conferences isn t a substitute for being there in-person, but it s still invaluable and I thank all those involved.

Okay, again delay :-). I wanted to push this version out yesterday, but real life has caught me and I did not manage to do it. So this time Gammu is released on Tuesday, but I want to get back to Monday releases next week. If all things go without problems, this is last testing release before 1.23.0. Besides changing files structure a lot (I hope current one better matches current state of code), there are only some minor fixes and cleanups. Full list of changes follows:

• Reorganize directory structure to match current state of things.
• Completely switch DBI driver to utf-8.
• Fixed date off by 1900 error in some cases.
• Cleanup a bit phonet driver.
• Handle properly requests for network code (bug #819).
• Allow to enter more security codes through SMSD (bug #819).
• Fixed restoring to specific folder on AT (bug #815).

You can download from usual place: http://cihar.com/gammu/, Debian users will find packages in experimental soon, SUSE, Fedora and Mandriva users can check experimental repository in OpenSuse Build Service.

After the release of Lenny I've upgraded all my servers to it. Earlier than I planned. The upgrade went mostly smooth, but there were some glitches:

1. somehow the roundcube webmailer was gone and uninstallable. I could fix that by building the package from Sids sources and installing it.


2. Squirrelmail was broken in that regard that you got an error message after logging in, stating something about the compatibility plugin. A Google search confimed that there are problems with the compatibility plugin. But sadly it didn't help to just deactivate the plugin in the config. Instead I needed to rename/remove the old directory from Squirrelmails plugin directory and installed the appropriate package again. Works again.


3. OpenLDAP has changed its method how replicas are being configured. But as this is just a change in configuration, it shouldn't be difficult, although I haven't found the time yet to do the changes.

I guess I'll have more "fun" in upgrade a Xen domU at work that is running Ubuntu dapper... sigh ;-)

So, I've had this plan for ages to follow up with what I did with my nslu2, and hide a Debian box inside a working stereo. Finally I've started work on this little project. I bought a cheap stereo with CD/tape/radio, mostly on the criteria:

• It needed to be big enough to stand a chance of fitting my qnap board and a full size SATA drive in it. But not too big..
• It needed to have line in. Luckily, most stereos do, thanks to ipods.

I got lucky.. after removing the CD drive, ripping out the battery compartment, and making other modifications to the interior of the case, I had just enough space to fit the qnap board, with centimeters to spare in some places, and millimeters in others. It fits perfectly. The board's LEDs and reset button will be accessible via the CD slot. I was able to keep the radio and amp working. Wiring up a USB sound card to that was easy. I could have kept the cassette deck working, but since I don't have any more of that media, I stripped it, and am planning to put a USB hub in there, accessible by pressing Stop/Eject. ;-) It seems I will even have a fair bit of empty space. Pondering adding a small UPS. Or, I might toss in an access point.

and the case against bzip2 Compression is with us all the time. I want to talk about general-purpose lossless compression here. There is a lot of agonizing over compression ratios: the size of output for various sizes of input. For some situations, this is of course the single most important factor. For instance, if you re Linus Torvalds putting your code out there for millions of people to download, the benefit of saving even a few percent of file size is well worth the cost of perhaps 50% worse compression performance. He compresses a source tarball once a month maybe, and we are all downloading it thousands of times a day. On the other hand, when you re doing backups, the calculation is different. Your storage media costs money, but so does your CPU. If you have a large photo collection or edit digital video, you may create 50GB of new data in a day. If you use a compression algorithm that s too slow, your backup for one day may not complete before your backup for the next day starts. This is even more significant a problem when you consider enterprises backing up terabytes of data each day. So I want to think of compression both in terms of resulting size and performance. Onward Starting Point I started by looking at the practical compression test, which has some very useful charts. He has charted savings vs. runtime for a number of different compressors, and with the range of different settings for each. If you look at his first chart, you ll notice several interesting things:

• gzip performance flattens at about -5 or -6, right where the manpage tells us it will, and in line with its defaults.
• 7za -2 (the LZMA algorithm used in 7-Zip and p7zip) is both faster and smaller than any possible bzip2 combination. 7za -3 gets much slower.
• bzip2 s performance is more tightly clustered than the others, both in terms of speed and space. bzip2 -3 is about the same speed as -1, but gains some space.

All this was very interesting, but had one limitation: it applied only to the gimp source tree, which is something of a best-case scenario for compression tools. A 6GB Test
I wanted to try something a bit more interesting. I made an uncompressed tar file of /usr on my workstation, which comes to 6GB of data. My /usr contains highly compressible data such as header files and source code, ELF binaries and libraries, already-compressed documentation files, small icons, and the like. It is a large, real-world mix of data. In fact, every compression comparison I saw was using data sets less than 1GB in size hardly representative of backup workloads. Let s start with the numbers:

Tool	MB saved	Space vs. gzip	Time vs. gzip	Cost
gzip	3398	100.00%	100.00%	0.15
bzip2	3590	92.91%	333.05%	0.48
pbzip2	3587	92.99%	183.77%	0.26
lzma -1	3641	91.01%	195.58%	0.28
lzma -2	3783	85.76%	273.83%	0.37

In the MB saved column, higher numbers are better; in all other columns, lower numbers are better. I m using clock seconds here on a dual-core machine. The cost column is clock seconds per MB saved. What does this tell us?

• bzip2 can do roughly 7% better than gzip, at a cost of a compression time more than 3 times as long.
• lzma -1 compresses better than bzip2 -9 in less than twice the time of gzip. That is, it is significantly faster and marginally smaller than bzip2.
• lzma -2 is significantly smaller and still somewhat faster than bzip2.
• pbzip2 achieves better wall clock performance, though not better CPU time performance, than bzip2 though even then, it is only marginally better than lzma -1 on a dual-core machine.

Some Pretty Charts First, let s see how the time vs. size numbers look: Like the other charts, the best area is the lower left, and worst is upper right. It s clear we have two outliers: gzip and bzip2. And a cluster of pretty similar performers. This view somewhat magnifies the differences, though. Let s add cat to the mix: And finally, look at the cost: Conclusions First off, the difference in time is far larger than the difference in space. We re talking a difference of 15% at the most in terms of space, but orders of magnitude for time. I think this pretty definitively is a death knell for bzip2. lzma -1 can achieve better compression in significantly less time, and lzma -2 can achieve significantly better compression in a little less time. pbzip2 can help even that out in terms of clock time on multicore machines, but 7za already has a parallel LZMA implementation, and it seems only a matter of time before /usr/bin/lzma gets it too. Also, if I were to chart CPU time, the numbers would be even less kind to pbzip2 than to bzip2. bzip2 does have some interesting properties, such as resetting everything every 900K, which could provide marginally better safety than any other compressor here though I don t know if lzma provides similar properties, or could. I think a strong argument remains that gzip is most suitable for backups in the general case. lzma -1 makes a good contender when space is at more of a premium. bzip2 doesn t seem to make a good contender at all now that we have lzma. I have also made my spreadsheet (OpenOffice format) containing the raw numbers and charts available for those interested. Update Part 2 of this story is now available, which considers more compression tools, and looks at performance compressing files individually rather than the large tar file.

Last june, I went to Cologne to visit a friend I'd first met Heike at Debconf7 in Edinburg, where we had much fun playing some music together. So the next time we met, which was at FOSDEM '08, she told me that she'd been playing in a band. As in, the kind of people who make music profesionally. Which was really cool. She also told me that this band, which consists mainly of a Canadian couple, would be playing in Cologne in early june, and that perhaps it'd be nice if I could come over to watch. Considering that'd be a nice excuse for a short holiday, I went, and had a great week in Cologne. Since Heike and the others had to practice, I only saw them in the evening; but Cologne is a great city, with interesting museums and other things, so I would spend the day sightseeing in Cologne (and Bonn on one day), and would be at the particular bar or whatever where the concert would be in the evening. I had a great time. Before going home, I bought one of their CDs, since I really liked the music. Anyway, at this year's FOSDEM, Heike told me that there was a new CD out (actually MP3 downloads), made from a recording that was done at one of the concerts. So, today, I bought a copy. Come to think of it, this is the first time I bought an album of a concert that I actually went to, and where I did not perform myself. After all, I don't really go to concerts all that often unless I'll be on stage. What's strange is that the music sounds slightly different from how I remember it. The mind can do strange things... or is it the case that a recording is not the real thing? Probably. In any case, the music is great, so go check it out!

This is a followup to the earlier design of dh. After converting many packages to minimal rules files using dh, I find that it works very well for the 3 line rules file case, which works for 80% of packages. But getting from there to a slightly more complex rules file that 20% of packages need can be non-obvious. So, let's make it obvious. Suppose I want to run dh_compress -Xsomefile. But my rules file as it currently stands doesn't offer an easy place to run that command:

#!/usr/bin/make -f
%:
    dh $@

To use dh in this situation you had to go look up rather a lot of boilerplate, and to insert an explicit version of the right target, with the right dependencies, that called dh before and after your command:

#!/usr/bin/make -f
%:
    dh $@
binary: binary-indep
binary-indep:
    dh $@ --before dh_compress
    dh_compress -Xsomefile
    dh $@ --after dh_compress

That took me half an hour to come up with, and I wrote this stuff! Also, if I want to override a second command, I have to do it in the right order. So, the full flexability of debhelper is there, but it's too hard to get at. Here's the new new thing.

#!/usr/bin/make -f
%:
    dh $@
override_dh_compress:
        dh_compress -Xsomefile

What this does should be obvious. These override targets can be written for any debhelper command that you need to change the default behavior of, or even avoid running at all.

override_dh_auto_configure:
    ./configure --with-kitchen-sink
override_dh_auto_test:
    # The test suite for this package is busted on s390.
    if [ $DEB_BUILD_ARCH != s390 ]; then dh_auto_test; fi

The overrides can also be used to run additional stuff before, or after any debhelper command:

override_dh_clean:
    dh_clean
    rm -rf build_dir

So, I think that's beautiful. Simple, clean, no boilerplate, and intuitively obvious if you are familiar with debhelper.

---

FWIW, the implementation, which will be in the next debhelper release unless someone screams loudly, is not beautiful. Mostly because I sorta shot myself in the foot with the cute implicit "%:" rules. But, I got around it. :-)

In a nice coincedence with the release of Lenny (cheers!!1), the low-quality versions of the videos from the Debian DevRoom at FOSDEM 2009 are available now!

Better quality versions will be encoded and uploaded by Ben over the next days. I'd also like to collect the slides from the talks there, so if you had a talk there, please send them to Wouter and/or me, and Wouter, please send me the slides you collected on the presetation computer ;-)

Basically I'm very happy with our results, excpet for two things: sadly the computer running dvswitch wasn't powerful enough to do picture in picture (and due to time constraints we weren't able to switch to a more powerful one), so we'll have to plan this better next year. And second, I have no real clue how useful our streaming efforts are, I believe the recordings are useful, but in both cases feedback from people who find them useful is appreciated! Covering FOSDEM is certainly a lot of fun, but it's also a lot of work and leaves not much room for seeing other parts of the conference. So a short reply (I'll post this to d-d-a with reply-to set to -project) will be very much appreciated! :)

Big thanks go to Womble2, edrz, xerrako, Q_, danielsan, DaCa, tokkee, luca, Yoe, Ganneff, franklin, huhn, luk, CarlFK, daven and some more volunteers I cannot remember right now! Also not to be forgotten the FOSDEM crew & volunteers for an awesome event and the people from CCC Hamburg who came by car and took a lot of audio/video/computer equipment for us (and left 90min later than planned! Thanks again!)! And also Debian (via the DPL) for some 120 Euros to rent equiqment and for buying batteries and the always needed duct-tape!

If you are following this blog, you probably remember per-process namespaces. Today, I m going to tell you how I did use them in the process of preparing this server to be upgraded to Lenny. I must say I was not using the most recent stuff on this server, and this is why I needed such preparation. First, this server is still running with php4. And well, the following line in the apt-get dist-upgrade output got my full attention (emphasis mine):

The following packages will be REMOVED:
cacti libapache2-mod-php4 libarchive-tar-perl libcurl3-openssl-dev
libgssapi2 libpci2 librpm4 libssp0 linux-kernel-headers modutils php4-mysql


While this server is not important enough not to be broken for a couple hours, I do like to test procedures that could help on servers that are. The first thing I needed to do on this server was obviously to upgrade php. But we all know how php applications are not fully compatible with all versions of php, so I also needed to test the upgrade was not breaking anything. On a server you don t care much about, you can just upgrade, test if all is okay, and be done with it. Obviously, if all is not okay, your visitors will see it, and you may also have a bad time downgrading. Another way to perform the upgrade is to have a similarly installed server on the side, test and validate the upgrade there, and replicate the upgrade on the production server if everything is fine. However, that does require additional resources, and possibly to setup them if they don t already exist. A cheaper way to do the above is to do it on the production server, both in-place and on the side (you ll see what I mean), using unionfs and per-process namespaces. Full containers could be used instead of per-process namespaces (such as openvz, vserver or lxc), but they still require much more setup, especially when you don t use them in the first place. Chroots could work just as well as per-process namespaces, but one of the ideas here is to expose the per-process namespaces feature, and allow for improvement of this procedure with pid and network namespaces, which are not available in Etch (where I m starting from), but are in Lenny. Unionfs allows to merge several directories into a single one, accessing some read-only, and others read-write. Installing unionfs is as easy as running the following command:

apt-get install unionfs-modules- uname -r

I won t describe all the kinds of setups that are possible with unionfs, but only one typical use case, which is what we will be using here:

# mkdir /tmp/root-cow
# mount -t unionfs -o dirs=/tmp/root-cow:/=ro none /mnt


The first thing we do here is to create an empty directory. Next, we merge it with the root filesystem (/), that we will keep read-only (meaning unionfs won t allow itself to write there), and mount the merged filesystem under /mnt. none could just be anything, as there is no device to be mounted. The result, in /mnt, is just something that looks like the root filesystem:

# ls /
bin boot dev etc home initrd lib media mnt opt proc root sbin srv sys tmp usr var
# ls /mnt
bin boot dev etc home initrd lib media mnt opt proc root sbin srv sys tmp usr var


But creating or modifying a file will do so in /tmp/root-cow:

# echo a > /mnt/a
# cat /a
cat: /a: No such file or directory
# cat /tmp/root-cow/a
a
# echo foo.com > /mnt/etc/mailname
# cat /etc/mailname
glandium.org
# cat /tmp/root-cow/etc/mailname
foo.com
# find /tmp/root-cow
/tmp/root-cow
/tmp/root-cow/etc
/tmp/root-cow/etc/mailname
/tmp/root-cow/a


Keep in mind that it doesn t include submounts, though:

# ls /var
backups cache lib local lock log mail opt run spool tmp www
# ls /mnt/var


This means we ll have to also mount /var, /proc, /dev, and /sys. That s enough testing for now, and we ll first do some cleanup before going after the real job:

# umount /mnt
# rm -rf /tmp/root-cow


Using the newns utility from my first post on per-process namespaces, let s create a new namespace to keep our testing private, and populate it with the necessary mount points:

# newns
# umount /tmp
# mount -t tmpfs tmpfs /tmp
# mkdir /tmp/root-cow /tmp/var-cow
# mount -t unionfs -o dirs=/tmp/root-cow:/=ro none /mnt
# mount -t unionfs -o dirs=/tmp/var-cow:/var=ro none /mnt/var
# cd /mnt
# pivot_root . mnt
# mount --move /mnt/proc /proc
# mount --move /mnt/sys /sys
# mount --move /mnt/lib/init/rw /lib/init/rw
# mount --move /mnt/tmp /tmp
# mount --move /mnt/dev /dev


The second and third statements are useful to avoid sharing /tmp with the main namespace, which means the directories we create in the fourth statement won t be visible in /tmp outside this namespace. The fifth and sixth statements put the union filesystems in place. As my system only has a separate /var filesystem (no /usr, and I don t care about /home here), I only need to setup these two. Feel free to add more union filesystems as necessary. The pivot_root call allows to switch to the unionfs ed root: everything under /mnt (/mnt and /mnt/var in our case) will be remounted under /, while what was under / is remounter under /mnt under the new root.
This means, in our case, that we have / and /var as union filesystems, while the old / and /var are respectively in /mnt and /mnt/var. It also means /dev, /proc, /sys and other filesystems are remounted as /mnt/dev, /mnt/proc, /mnt/sys, etc., which is why we next mount --move all of them to a better place in our namespace. Once all this setup is done, we are ready to do our php upgrade test. As Etch doesn t have support for neither PID namespaces nor network namespaces, we ll still have some conflicts with the main namespace for TCP port binding and process handling, so we need to be a little careful:

# rm /var/run/apache2.pid
# echo Listen 8080 > /etc/apache2/ports.conf


With these changes, we can now safely start apache2 in this namespace, at the same time the main apache2 runs in the main namespace:

# /etc/init.d/apache2 start


Now, there are actually 2 problems showing up in apache in this setup. The first one is that displaying static files doesn t work. At all. It appears unionfs doesn t support sendfile(). Which is fine. But apache doesn t check for sendfile() s error cases and doesn t fallback to a working solution when sendfile() doesn t work. So we have to manually disable it:

# echo EnableSendfile off > /etc/apache2/conf.d/sendfile.conf


The second one is that access to the mysql socket doesn t work properly under the unionfs. I didn t want to investigate further, so I only worked around the issue by forcing to use the socket outside the unionfs:

# mount --bind /mnt/var/run/mysqld /var/run/mysqld


Finally, we can test and validate our php upgrade:

# apt-get install php5-mysql php5-cli php5-snmp libapache2-mod-php5
# /etc/init.d/apache2 restart
# apt-get remove --purge php4-common


Note that installing libapache2-mod-php5 will remove libapache2-mod-php4, and apache2 gets restarted, taking into account this change. But the php modules (php5-mysql and php5-snmp) install is only going to happen after that, and no apache2 restart is triggered then, which leaves a half working php setup
Also note that the cacti setup in etch supposes php4 is installed, using a IfModule statement against mod_php4.c and none for mod_php5.c, which means a part of its setup doesn t work out of the box (most notably, the DirectoryIndex). Once all is validated, all we need to do is to stop apache2 and exit from the namespace. The union filesystems and temporary filesystems are then automagically cleaned-up and the namespace disappears as well as all modifications we just did, since all processes using the namespace ended. We are now free to upgrade on the production server, as we know all the side effects. I ll now be able to upgrade to Lenny without php getting removed.