In one of the comments to my last blog entry, an anonymous commenter writes:

You seem to be taking a different perspective to linus on the adapting to the the disk technology front (Linus seems to against having to have the OS know about disk boundaries and having to do levelling itself)

That s an interesting question, and I figure it s worth its own top-level entry, as opposed to a reply in the comment stream. One of the interesting design questions in any OS or Computer Architecture is where the abstraction boundaries should be drawn and which side of an abstraction boundary should various operations be pushed. Linus s arguments is that there a flash controller can do a better job of wear leveling, including detecting how worn a particular flash cell might be (for example, perhaps by looking at the charge levels at an analog level and knowing when the last time the cell was programmed), and so it doesn t make sense to try to do wear leveling in a flash file system. Some responsibilities of flash management, such as coalescing newly written blocks into erase blocks to avoid write amplification can be done either on the SSD or in the file system for example, by using a log-structured file system, or some other copy-on-write file system, instead of a rewrite-in-place style file system, you can essentially solve the write amplification problem. In some cases, it s necessary let additional information leak across the abstraction for example, the ATA TRIM command is a way for the file system to let the disk know that certain blocks no longer need to be used. If too much information needs to be pushed across the abstraction, one way or another, then maybe we need to rethink whether the abstraction barrier is in the right place. In addition, if the abstraction has been around for a long time, changing it also has costs, which has to be taken into account. The 512 byte sector LBA abstraction has been around long time, and therefore dislodging it is difficult and costly. For example, the same argument which says that because the underlying hardware details are changing between different generations of SSD is all of these details should be hidden in hardware, was also used to justify something that has been a complete commercial failure for years if not decades: Object Based Disks. One of the arguments of OBD s was that the hard drive has the best knowledge of how and where to store an contiguous stream of bytes, and so perhaps filesystems should not be trying to decide where on disk an inode should be stored, but instead tell the hard drive, I have this object, which is 134 kilobytes long; please store it somewhere on the disk . At least in theory the HDD or SSD could handle all of the details of knowing the best place to store the object on the spinning magnetic media or flash media, taking into account how worn the flash is and automatically move the object around in the case of an SSD, and in the case of the HDD, the drive could know about (real) cylinder and track boundaries, and store the object in the most efficient way possible, since the drive has intimate knowledge about the low-level details of how data is stored on the disk. This theory makes a huge amount of sense; but there s only one problem. Object Based Disks have been proposed in academia and advanced R&D shops of companies like Seagate et.al. have been proposing them for over a decade, with absolutely nothing to show for it. Why? There have been two reasons proposed. One is that OBD vendors were too greedy, and tried to charge too much money for OBD s. Another explanation is that the interface abstraction for OBD s was too different, and so there wasn t enough software or file systems or OS s that could take advantage of OBD s. Both explanations undoubtedly contributed to the commercial failure of OBD s, but the question is which is the bigger reason. And the reason why it is particularly important here is because at least as far as Intel s SSD strategy is concerned, its advantage is that (modulo implementation shortcomings such as the reported internal LBA remapping table fragmentation problem and the lack of ATA TRIM support) filesystems don t need to change (much) in order to take advantage of the Intel SSD and get at least decent performance. However, if the price delta is a stronger reason for its failure, then the X25-M may be in trouble. Currently the 80GB Intel X25-M has a street price of $400, so it costs roughly $5 per gigabyte. Dumb MLC SATA SSD s are available for roughly half the cost/gigabyte (64 GB for $164). So what does the market look like 12-18 months from now? If dumb SSD s are still available at 50% of the cost of smart SSD s, it would probably be worth it to make a copy-on-write style filesystem that attempts to do the wear leveling and write amplification reduction in software. Sure, it s probably more efficient to do it in hardware, but a 2x price differential might cause people will settle for a cheaper solution even if isn t the absolutely best technical choice. On the hand, if prices drop significantly, and/or dumb SSD s completely disappear from the market, then time spent now optimizing for dumb SSD s will be completely wasted. So for Linus to make the proclamation that it s completely stupid to optimize for dumb SSD s seems to be a bit premature. Market externalities for example, does Intel have patents that will prevent competing smart SSD s from entering the market and thus forcing price drops? could radically change the picture. It s not just a pure technological choice, which is what makes projections and prognostications difficult. As another example, I don t know whether or not Intel will issue a firmware update that adds ATA TRIM support to the X25-M, or how long it will take before such SSD s become available. Until ATA TRIM support becomes available, it will be advantageous to add support in ext4 for a block allocator option that aggressively reuses blocks above all else, and avoids using blocks that have never been allocated or used before, even if it causes more in-file system fragmentation and deeper extent allocation trees. The reason for this is at the moment, once a block is used by the file system, at least today, the X25-M has absolutely no idea whether we still care about the contents of that block, or whether the block has since been released when the file was deleted. However, if 20% of the SSD s blocks have never been used, the X25-M can use 20% of the flash for better garbage collection and defragmentation algorithms. And if Intel never releases a firmware update to add ATA TRIM support, then I will be out $400 out of my own pocket for an SSD that lacks this capability, and so adding a block allocator which works around limitations of the X25-M probably makes sense. If it turns out that it takes two years before disks that have ATA TRIM support show up, then it will definitely make sense to add such an optimization. (Hard drive vendors have been historically S-L-O-W to finish standardizing new features and then letting such features enter the market place, so I m not necessarily holding my breath; after all, the Linux block device layer and and file systems have been ready to send ATA TRIM support for about six months; what s taking the ATA committees and SSD vendors so long? <grin> On the other hand, if Intel releases ATA TRIM support next month, then it might not be worth my effort to add such a mount option to ext4. Or maybe Sandisk will make an ATA TRIM capable SSD available soon, and which is otherwise competitive with Intel, and I get a free sample, but it turns out another optimization on Sandisk SSD s will give me an extra 10% performance gain under some workloads. Is it worth it in that case? Hard to tell, unless I know whether such a tweak addresses an optimization problem which is fundamental, and whether or not such a tweak will either be unnecessary, or perhaps actively unhelpful in the next generation. As long as SSD manufacturers force us treat these devices as black boxes, there may be a certain amount of cargo cult science which may be forced upon us file system designers or I guess I should say, in order to be more academically respectable, we will be forced to rely more on empirical measurements leading to educated engineering estimations about what the SSD is doing inside the black box . Heh. Related posts (automatically generated):

1. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...

In one of the comments to my last blog entry, an anonymous commenter writes:

You seem to be taking a different perspective to linus on the adapting to the the disk technology front (Linus seems to against having to have the OS know about disk boundaries and having to do levelling itself)

That s an interesting question, and I figure it s worth its own top-level entry, as opposed to a reply in the comment stream. One of the interesting design questions in any OS or Computer Architecture is where the abstraction boundaries should be drawn and which side of an abstraction boundary should various operations be pushed. Linus s arguments is that there a flash controller can do a better job of wear leveling, including detecting how worn a particular flash cell might be (for example, perhaps by looking at the charge levels at an analog level and knowing when the last time the cell was programmed), and so it doesn t make sense to try to do wear leveling in a flash file system. Some responsibilities of flash management, such as coalescing newly written blocks into erase blocks to avoid write amplification can be done either on the SSD or in the file system for example, by using a log-structured file system, or some other copy-on-write file system, instead of a rewrite-in-place style file system, you can essentially solve the write amplification problem. In some cases, it s necessary let additional information leak across the abstraction for example, the ATA TRIM command is a way for the file system to let the disk know that certain blocks no longer need to be used. If too much information needs to be pushed across the abstraction, one way or another, then maybe we need to rethink whether the abstraction barrier is in the right place. In addition, if the abstraction has been around for a long time, changing it also has costs, which has to be taken into account. The 512 byte sector LBA abstraction has been around long time, and therefore dislodging it is difficult and costly. For example, the same argument which says that because the underlying hardware details are changing between different generations of SSD is all of these details should be hidden in hardware, was also used to justify something that has been a complete commercial failure for years if not decades: Object Based Disks. One of the arguments of OBD s was that the hard drive has the best knowledge of how and where to store an contiguous stream of bytes, and so perhaps filesystems should not be trying to decide where on disk an inode should be stored, but instead tell the hard drive, I have this object, which is 134 kilobytes long; please store it somewhere on the disk . At least in theory the HDD or SSD could handle all of the details of knowing the best place to store the object on the spinning magnetic media or flash media, taking into account how worn the flash is and automatically move the object around in the case of an SSD, and in the case of the HDD, the drive could know about (real) cylinder and track boundaries, and store the object in the most efficient way possible, since the drive has intimate knowledge about the low-level details of how data is stored on the disk. This theory makes a huge amount of sense; but there s only one problem. Object Based Disks have been proposed in academia and advanced R&D shops of companies like Seagate et.al. have been proposing them for over a decade, with absolutely nothing to show for it. Why? There have been two reasons proposed. One is that OBD vendors were too greedy, and tried to charge too much money for OBD s. Another explanation is that the interface abstraction for OBD s was too different, and so there wasn t enough software or file systems or OS s that could take advantage of OBD s. Both explanations undoubtedly contributed to the commercial failure of OBD s, but the question is which is the bigger reason. And the reason why it is particularly important here is because at least as far as Intel s SSD strategy is concerned, its advantage is that (modulo implementation shortcomings such as the reported internal LBA remapping table fragmentation problem and the lack of ATA TRIM support) filesystems don t need to change (much) in order to take advantage of the Intel SSD and get at least decent performance. However, if the price delta is a stronger reason for its failure, then the X25-M may be in trouble. Currently the 80GB Intel X25-M has a street price of $400, so it costs roughly $5 per gigabyte. Dumb MLC SATA SSD s are available for roughly half the cost/gigabyte (64 GB for $164). So what does the market look like 12-18 months from now? If dumb SSD s are still available at 50% of the cost of smart SSD s, it would probably be worth it to make a copy-on-write style filesystem that attempts to do the wear leveling and write amplification reduction in software. Sure, it s probably more efficient to do it in hardware, but a 2x price differential might cause people will settle for a cheaper solution even if isn t the absolutely best technical choice. On the hand, if prices drop significantly, and/or dumb SSD s completely disappear from the market, then time spent now optimizing for dumb SSD s will be completely wasted. So for Linus to make the proclamation that it s completely stupid to optimize for dumb SSD s seems to be a bit premature. Market externalities for example, does Intel have patents that will prevent competing smart SSD s from entering the market and thus forcing price drops? could radically change the picture. It s not just a pure technological choice, which is what makes projections and prognostications difficult. As another example, I don t know whether or not Intel will issue a firmware update that adds ATA TRIM support to the X25-M, or how long it will take before such SSD s become available. Until ATA TRIM support becomes available, it will be advantageous to add support in ext4 for a block allocator option that aggressively reuses blocks above all else, and avoids using blocks that have never been allocated or used before, even if it causes more in-file system fragmentation and deeper extent allocation trees. The reason for this is at the moment, once a block is used by the file system, at least today, the X25-M has absolutely no idea whether we still care about the contents of that block, or whether the block has since been released when the file was deleted. However, if 20% of the SSD s blocks have never been used, the X25-M can use 20% of the flash for better garbage collection and defragmentation algorithms. And if Intel never releases a firmware update to add ATA TRIM support, then I will be out $400 out of my own pocket for an SSD that lacks this capability, and so adding a block allocator which works around limitations of the X25-M probably makes sense. If it turns out that it takes two years before disks that have ATA TRIM support show up, then it will definitely make sense to add such an optimization. (Hard drive vendors have been historically S-L-O-W to finish standardizing new features and then letting such features enter the market place, so I m not necessarily holding my breath; after all, the Linux block device layer and and file systems have been ready to send ATA TRIM support for about six months; what s taking the ATA committees and SSD vendors so long? <grin> On the other hand, if Intel releases ATA TRIM support next month, then it might not be worth my effort to add such a mount option to ext4. Or maybe Sandisk will make an ATA TRIM capable SSD available soon, and which is otherwise competitive with Intel, and I get a free sample, but it turns out another optimization on Sandisk SSD s will give me an extra 10% performance gain under some workloads. Is it worth it in that case? Hard to tell, unless I know whether such a tweak addresses an optimization problem which is fundamental, and whether or not such a tweak will either be unnecessary, or perhaps actively unhelpful in the next generation. As long as SSD manufacturers force us treat these devices as black boxes, there may be a certain amount of cargo cult science which may be forced upon us file system designers or I guess I should say, in order to be more academically respectable, we will be forced to rely more on empirical measurements leading to educated engineering estimations about what the SSD is doing inside the black box . Heh. Related posts (automatically generated):

1. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...

It s no secret for those who follow me on Twitter/identi.ca that I have been toying with virtualization technologies for some time now. I have been using Xen in production for some time and, apart from not thinking it s the way to go for future virtualization, it s quite usable right now, specially if you are going to use Debian Lenny as the dom0, as it now has a more modern kernel than the dreaded 2.6.18 we Xen users were forced to keep using. However, I have been keeping my eyes on KVM as well, as I always liked how much simpler and well integrated into the Linux kernel it is. I m even subscribed to the KVM development mailing list, even not understanding most of the things the developers are talking about there. It s no problem for me, I just want to know what s being worked on and what s on the pipeline. As a KVM fan, I m also using it intensively as a tool for prototyping servers. It s easy enough to set up a new Debian server to test things on and learn new technologies, as well as troubleshoot problems without intefering with production environments. Althought KVM is not as good as Xen when it comes to performance, it s quickly improving every single day. And, for those of you who still believes KVM is only about full virtualization, I m happy to say that it has come a long way and paravirtualization has started to infiltrate KVM land as well. Today, KVM already sports paravirtualized clock (pvclock), paravirtualized memory management unit (pv MMU) and VirtIO drivers. Actually, it seems that KVM these days is the biggest user of VirtIO, maybe only loosing the leadership to Rusty s lguest. By using VirtIO s disk and network drivers (virtio_blk and virtio_net, respectively, and its associated modules), KVM can deliver a much improved I/O experience than when using QEMU s emulated drivers. For this reason, it s always preferable to use VirtIO drivers whenever possible when setting up your KVM guests. During the Etch lifetime, one will need to do some tricks in order to use VirtIO when using Etch as a guest under KVM. However, when preparing Lenny s d-i, the developers were smart enough to add to it virtual disk detection support. What it means is that now, starting with Lenny, d-i will recognize that it s being given a VirtIO block device and automatically load the needed kernel modules to support it. Also, the disk detection and partition modules (partman et all) were modified to show a detected virtual disk (i.e. /dev/vdX) and let the user partition it, as well as grub-installer was changed to allow GRUB to be installed onto a virtual disk s MBR, effectively making d-i a really powerful virtualization aware installer. Here you can see Lenny s d-i showing a detected virtual disk, named vda : Brazilian Portuguese readers could see the version using brazilian portuguese texts on the screen here. Sure, d-i also received a lot of improvements and special support for installing Lenny as a Xen domU was added as well, but I haven t played with it yet, so I won t comment on that right now. And, hey, one can even use virt-install and virt-manager to deploy KVM guests under Lenny And you know what gives me even more confidence that KVM will be a first class citizen inside Debian ? The fact that Steve Kemp is toying with the idea to change his xen-hosting.org project so it would become a new kvm-hosting.org project. Maybe I m praying for the preacher here, but Steve is very well well know for being the author of xen-tools and xen-shell, as well as being the creator of a number of other nice free softwares. And, judging by the comments on his post about the future kvm-hosting.org project, it seems that Steve maybe will need to update xen-tools and xen-shell to account for KVM or create a new set of tools dedicated exclusively to KVM based on his past experience creating the current tools for Xen. Good times ahead for those of us who are surfing the Debian Virtualization wave, indeed

Alexander Wirt just wrote a mail to our mailinglist to announce that lenny-backports suite is ready for uploads. We will of course continue supporting etch-backports with security updates as long as etch will be supported by the Debian security team.

Happy backporting!

I start to move some servers when lenny was freeze, and did not have any complain, the update was really smooth; but I still let some servers to be migrated just at the end of the release cycle, and sure, they fail I actually have some fun migrating freeradius and cacti, and note that I actually will have more fun migrating asterisk and openser servers, but for the sake of this release, is the fun that every BOFH should expect, and also, cheers and thanks to everyone who work on this release

On a deployment I m currently working on, I ve seen two different cases of proprietary software use leading to both madness and sadness, which are just so typical that I can t resist to tell you. Keep in mind, for the rest of this reading, that the whole platform is running under Solaris 10 x86-64. The first case is a data quality management software that we will keep anonymous. The editor promised before the deployment began that they had a version of the software for the operating system that we would be using. Well, it turned out they do have a Solaris version for sparc, and a x86-64 version for Linux. No Solaris x86-64 version, and no way to get a rebuild in a timely fashion. The second case is a content management software that we will keep anonymous. It comes in the form of a java web application and a java application container. Part of integrating this software involves a proprietary plugin for Apache HTTPd that acts as a mix of mod_proxy_balancer, mod_disk_cache, and htcacheclean, as well as a cache invalidator. Originally, the java web application was supposed to be installed within a JBoss Application Server instead of the editor provided container, and Apache HTTPd would reverse proxy requests to the JBoss server. This means we already had an Apache HTTPd in place (latest version ; 2.2.11 at the time), and since we have x86-64 processors, it was built as a 64-bits binary. Contrary to the first case, this time, we had a Solaris x86 binary. Yes, you read correctly: x86 ; 32-bits only. After going through the pain of rebuilding Apache HTTPd in 32-bits (there are various reasons why we don t use sunfreeware software), it turned out the module was a 2.0 ABI module not compatible with the 2.2 ABI. It also turned out there was a 2.2 ABI version of the module for Solaris sparc. It finally worked after another build of Apache HTTPd, a 2.0.63 release, this time. The more you get used to free software, the more these kind of things get frustrating.

The recent issue of Frontline has an interview of Stallman. It is indeed a nice thing to see that nationally circulated news magazines in India are talking about Mukta Software. However, it is disappointing to see the FSF take one more pot-shot at Linus Torvalds.

Torvalds said he rejects GNU GPL Version 3 because he approves of tivoization, because he does not agree that users deserve the freedom to change their own copies of software.

By choosing the GPL for his own code Linus Torvalds gave users the freedom to change their own copies of Linux. This certainly seems at odds with what is quoted. There are many different points of view about how the cause of mukta software and its users can be strengthened. I think we can agree that it is certainly not strengthened by shooting each other in the foot. The GPLv3 bolsters-up one aspect of this cause: use the legal system to make if difficult (if not impossible) to take free software and put it in proprietary bottles. Linus Torvalds and all his "show me the code" buddies are addressing another aspect; to ensure that there are enough people accomplished in the task of reading, improving and writing high quality free software. The West has traditionally had more success with the law. When free programs was written in the 70's and 80's by people who did not care to preserve its freedom with a free license, this code was gobbled up and re-gurgitated as proprietary software. Once free licenses became commonplace in the 90's, the West has had a resurgence of high quality public code and coders. Unfortunately, some of the latter tend to look down on "mere users". Over here, in India, we have little faith in the legal system. (For reasons, see this report and a more recent one). We know that freedom is hard fought even when the law is technically on your side. At the same time, the brahminical approach of knowledge being safe-guarded by the self-proclaimed elite was what rang the death knell for Indian knowledge systems in the past. So it is important that the breed of skeptical people in India who tinker with their computers and software (and everything else) grows.

We have far too many cattle who will follow any flute-playing good-looking young person who is willing to fight their demons for them.

I recently purchased a new toy, an Intel X25-M SSD, and when I was setting it up initially, I decided I wanted to make sure the file system was aligned on an erase block boundary. This is a generally considered to be a Very Good Thing to do for most SSD s available today, although there s some question about how important this really is for Intel SSD s more on that in a moment. It turns out this is much more difficult than you might first think most of Linux s storage stack is not set up well to worry about alignment of partitions and logical volumes. This is surprising, because it s useful for many things other than just SSD s. This kind of alignment is important if you are using any kind of hardware or software RAID, for example, especially RAID 5, because if writes are done on stripe boundaries, it can avoid a read-modify-write overhead. In addition, the hard drive industry is planning on moving to 4096 byte sectors instead of the way-too-small 512 byte sectors at some point in the future. Linux s default partition geometry of 255 heads and 63 sectors/track means that there are 16065 (512 byte) sectors per cylinder. The initial round of 4k sector disks will emulate 512 byte disks, but if the partitions are not 4k aligned, then the disk will end up doing a read/modify/write on two internal 4k sectors for each singleton 4k file system write, and that would be unfortunate. Vista has already started working around this problem, since it uses a default partitioning geometry of 240 heads and 63 sectors/track. This results in a cylinder boundary which is divisible by 8, and so the partitions (with the exception of the first, which is still misaligned unless you play some additional tricks) are 4k aligned. So this is one place where Vista is ahead of Linux . unfortunately the default 255 heads and 63 sectors is hard coded in many places in the kernel, in the SCSI stack, and in various partitioning programs; so fixing this will require changes in many places. However, with SSD s (remember SSD s? This is a blog post about SSD s ) you need to align partitions on at least 128k boundaries for maximum efficiency. The best way to do this that I ve found is to use 224 (32*7) heads and 56 (8*7) sectors/track. This results in 12544 (or 256*49) sectors/cylinder, so that each cylinder is 49*128k. You can do this by doing starting fdisk with the following options when first partitioning the SSD: # fdisk -H 224 -S 56 /dev/sdb The first partition will only be aligned on a 4k boundary, since in order to be compatible with MS-DOS, the first partition starts on track 1 instead of track 0, but I didn t worry too much about that since I tend to use the first partition for /boot, which tends not to get modified much. You can go into expert mode with fdisk and force the partition to begin on an 128k alignment, but many Linux partition tools will complain about potential compatibility problems (which are obsolete warnings, since the systems that would have booting systems with these issues haven t been made in about ten years), but I didn t needed to do that, so I didn t worry about it. So I created a 1 gigabyte /boot partition as /dev/sdb1, and allocated the rest of the SSD for use by LVM as /dev/sdb2. And that s where I ran into my next problem. LVM likes to allocate 192k for its header information, and 192k is not a multiple of 128k. So if you are creating file systems as logical volumes, and you want those volume to be properly aligned you have to tell LVM that it should reserve slightly more space for its meta-data, so that the physical extents that it allocates for its logical volumes are properly aligned. Unfortunately, the way this is done is slightly baroque: # pvcreate metadatasize 250k /dev/sdb2
Physical volume /dev/sdb2 successfully created Why 250k and not 256k? I can t tell you sometimes the LVM tools aren t terribly intuitive. However, you can test to make sure that physical extents start at the proper offset by using: # pvs /dev/sdb2 -o+pe_start
PV VG Fmt Attr PSize PFree 1st PE
/dev/sdb2 lvm2 73.52G 73.52G 256.00K If you use a metadata size of 256k, the first PE will be at 320k instead of 256k. There really ought to be an pe-align option to pvcreate, which would be far more user-friendly, but, we have to work with the tools that we have. Maybe in the next version of the LVM support tools . Once you do this, we re almost done. The last thing to do is to create the file system. As it turns out, if you are using ext4, there is a way to tell the file system that it should try to align files so they match up with the RAID stripe width. (These techniques can be used for RAID disks as well). If your SSD has an 128k erase block size, and you are creating the file system with the default 4k block size, you just have to specify a strip width when you create the file system, like so: # mke2fs -t ext4 -E stripe-width=32,resize=500G /dev/ssd/root (The resize=500G limits the number of blocks reserved for resizing this file system so that the guaranteed number size that the file system can be grown via online resize is 500G. The default is 1000 times the initial file system size, which is often far too big to be reasonable. Realistically, the file system I am creating is going to be used for a desktop device, and I don t foresee needing to resize it beyond 500G, so this saves about a 50 megabytes or so. Not a huge deal, but waste not, want not , as the saying goes.) With e2fsprogs 1.41.4, the journal will be 128k aligned, as will the start of the file system, and with the stripe-width specified, the ext4 allocator will try to align block writes to the stripe width where that makes sense. So this is as good as it gets without kernel changes to make the block and inode allocators more SSD aware, something which I hope to have a chance to look at.

All of this being said, it s time to revisit this question is all of this needed for a smart , better by design next-generation SSD such as Intel s? Aligning your file system on an erase block boundary is critical on first generation SSD s, but the Intel X25-M is supposed to have smarter algorithms that allow it to reduce the effect of write-amplification. The details are a little bit vague, but presumably there is a mapping table which maps sectors (at some internal sector size we don t know for sure whether it s 512 bytes or some larger size) to individual erase blocks. If the file system sends a series of 4k writes for file system blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, 99, followed by a barrier operation, a traditional SSD might do read/modify/write on four 128k erase blocks one covering the blocks 0-31, another for the blocks 32-63, and so on. However, the Intel SSD will simply write a single 128k block that indicates where the latest versions of blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, and 99 can be found. This technique tends to work very well. However, over time, the table will get terribly fragmented, and depending on whether the internal block sector size is 512 or 4k (or something in between), there could be a situation where all but one or two of the internal sectors on the disks have been mapped away to other erase blocks, leading to fragmentation of the erase blocks. This is not just a theoretical problem; there are reports from the field that this happens relatively easy. For example, see Allyn Malventano s Long-term performance analysis of Intel Mainstream SSDs and Marc Prieur s report from BeHardware.com which includes an official response from Intel regarding this phenomenon. Laurent Gilson posted on the Linux-Thinkpad mailing list that when he tried using the X25-M to store commit journals for a database, that after writing 170% of the capacity of an Intel SSD, the small writes caused the write performance to go through the floor. More troubling, Allyn Malventano indicated that if the drive is abused for too long with a mixture of small and large writes, it can get into a state where the performance degredation is permanent, and even a series of large writes apparently does not restore the drive s function only an ATA SECURITY ERASE command to completely reset the mapping table seems to help. So, what can be done to prevent this? Allyn s review speculates that aligning writes to erase write boundaries can help I m not 100% sure this is true, but without detailed knowledge of what is going on under the covers in Intel s SSD, we won t know for sure. It certainly can t hurt, though, and there is a distinct possibility that the internal sector size is larger than 512 bytes, which means the default partitioning scheme of 255 heads/63 sectors is probably not a good idea. (Even Vista has moved to a 240/63 scheme, which gives you 8k alignment of partitions; I prefer 224/56 partitioning, since the days when BIOS s used C/H/S I/O are long gone.) The Ext3 and Ext4 file system tend to defer meta-data writes by pinning them until a transaction commit; this definitely helps, and ext4 allows you to configure an erase block boundary, which should also be helpful. Enabling laptop mode will discourage writing to the disk except in large blocks, which probably helps significantly as well. And avoiding fsync() in applications will also be helpful, since a cache flush operation will force the SSD to write to an erase block even if it isn t completely filled. Beyond that, clearly some experimentation will be needed. My current thinking is to use a standard file system aging workload, and then performing an I/O benchmark to see if there has been any performance degradation. I can then vary various file system tuning parameters and algorithms, confirm whether or not a heavy fsync workload makes the performance worse. In the long term, hopefully Intel will release a firmware update which adds support for ATA TRIM/DISCARD commands, which will allow the file system to inform the SSD that various blocks have been deleted and no longer need to be preserved by the SSD. I suspect this will be a big help, if the SSD knows that certain sectors no longer need to be preserved, it can avoid copying them when trying to defragment the SSD. Given how expensive the X25-M SSD s are, I hope that there will be a firmware update to support this, and that Intel won t leave its early adopters high and dry by only offering this functionality in newer models of the SSD. If they were to do that, it would leave many of these early adopters, especially your humble writer (who paid for his SSD out of his own pocket), to be quite grumpy indeed. Hopefully, though, it won t come to that. Update: I ve since penned a follow-up post Should Filesystems Be Optimized for SSD s? Related posts (automatically generated):

1. Should Filesystems Be Optimized for SSD s? In one of the comments to my last blog entry,...
2. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...

I recently purchased a new toy, an Intel X25-M SSD, and when I was setting it up initially, I decided I wanted to make sure the file system was aligned on an erase block boundary. This is a generally considered to be a Very Good Thing to do for most SSD s available today, although there s some question about how important this really is for Intel SSD s more on that in a moment. It turns out this is much more difficult than you might first think most of Linux s storage stack is not set up well to worry about alignment of partitions and logical volumes. This is surprising, because it s useful for many things other than just SSD s. This kind of alignment is important if you are using any kind of hardware or software RAID, for example, especially RAID 5, because if writes are done on stripe boundaries, it can avoid a read-modify-write overhead. In addition, the hard drive industry is planning on moving to 4096 byte sectors instead of the way-too-small 512 byte sectors at some point in the future. Linux s default partition geometry of 255 heads and 63 sectors/track means that there are 16065 (512 byte) sectors per cylinder. The initial round of 4k sector disks will emulate 512 byte disks, but if the partitions are not 4k aligned, then the disk will end up doing a read/modify/write on two internal 4k sectors for each singleton 4k file system write, and that would be unfortunate. Vista has already started working around this problem, since it uses a default partitioning geometry of 240 heads and 63 sectors/track. This results in a cylinder boundary which is divisible by 8, and so the partitions (with the exception of the first, which is still misaligned unless you play some additional tricks) are 4k aligned. So this is one place where Vista is ahead of Linux . unfortunately the default 255 heads and 63 sectors is hard coded in many places in the kernel, in the SCSI stack, and in various partitioning programs; so fixing this will require changes in many places. However, with SSD s (remember SSD s? This is a blog post about SSD s ) you need to align partitions on at least 128k boundaries for maximum efficiency. The best way to do this that I ve found is to use 224 (32*7) heads and 56 (8*7) sectors/track. This results in 12544 (or 256*49) sectors/cylinder, so that each cylinder is 49*128k. You can do this by doing starting fdisk with the following options when first partitioning the SSD: # fdisk -H 224 -S 56 /dev/sdb The first partition will only be aligned on a 4k boundary, since in order to be compatible with MS-DOS, the first partition starts on track 1 instead of track 0, but I didn t worry too much about that since I tend to use the first partition for /boot, which tends not to get modified much. You can go into expert mode with fdisk and force the partition to begin on an 128k alignment, but many Linux partition tools will complain about potential compatibility problems (which are obsolete warnings, since the systems that would have booting systems with these issues haven t been made in about ten years), but I didn t needed to do that, so I didn t worry about it. So I created a 1 gigabyte /boot partition as /dev/sdb1, and allocated the rest of the SSD for use by LVM as /dev/sdb2. And that s where I ran into my next problem. LVM likes to allocate 192k for its header information, and 192k is not a multiple of 128k. So if you are creating file systems as logical volumes, and you want those volume to be properly aligned you have to tell LVM that it should reserve slightly more space for its meta-data, so that the physical extents that it allocates for its logical volumes are properly aligned. Unfortunately, the way this is done is slightly baroque: # pvcreate metadatasize 250k /dev/sdb2
Physical volume /dev/sdb2 successfully created Why 250k and not 256k? I can t tell you sometimes the LVM tools aren t terribly intuitive. However, you can test to make sure that physical extents start at the proper offset by using: # pvs /dev/sdb2 -o+pe_start
PV VG Fmt Attr PSize PFree 1st PE
/dev/sdb2 lvm2 73.52G 73.52G 256.00K If you use a metadata size of 256k, the first PE will be at 320k instead of 256k. There really ought to be an pe-align option to pvcreate, which would be far more user-friendly, but, we have to work with the tools that we have. Maybe in the next version of the LVM support tools . Once you do this, we re almost done. The last thing to do is to create the file system. As it turns out, if you are using ext4, there is a way to tell the file system that it should try to align files so they match up with the RAID stripe width. (These techniques can be used for RAID disks as well). If your SSD has an 128k erase block size, and you are creating the file system with the default 4k block size, you just have to specify a strip width when you create the file system, like so: # mke2fs -t ext4 -E stripe-width=32,resize=500G /dev/ssd/root (The resize=500G limits the number of blocks reserved for resizing this file system so that the guaranteed number size that the file system can be grown via online resize is 500G. The default is 1000 times the initial file system size, which is often far too big to be reasonable. Realistically, the file system I am creating is going to be used for a desktop device, and I don t foresee needing to resize it beyond 500G, so this saves about a 50 megabytes or so. Not a huge deal, but waste not, want not , as the saying goes.) With e2fsprogs 1.41.4, the journal will be 128k aligned, as will the start of the file system, and with the stripe-width specified, the ext4 allocator will try to align block writes to the stripe width where that makes sense. So this is as good as it gets without kernel changes to make the block and inode allocators more SSD aware, something which I hope to have a chance to look at.

All of this being said, it s time to revisit this question is all of this needed for a smart , better by design next-generation SSD such as Intel s? Aligning your file system on an erase block boundary is critical on first generation SSD s, but the Intel X25-M is supposed to have smarter algorithms that allow it to reduce the effect of write-amplification. The details are a little bit vague, but presumably there is a mapping table which maps sectors (at some internal sector size we don t know for sure whether it s 512 bytes or some larger size) to individual erase blocks. If the file system sends a series of 4k writes for file system blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, 99, followed by a barrier operation, a traditional SSD might do read/modify/write on four 128k erase blocks one covering the blocks 0-31, another for the blocks 32-63, and so on. However, the Intel SSD will simply write a single 128k block that indicates where the latest versions of blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, and 99 can be found. This technique tends to work very well. However, over time, the table will get terribly fragmented, and depending on whether the internal block sector size is 512 or 4k (or something in between), there could be a situation where all but one or two of the internal sectors on the disks have been mapped away to other erase blocks, leading to fragmentation of the erase blocks. This is not just a theoretical problem; there are reports from the field that this happens relatively easy. For example, see Allyn Malventano s Long-term performance analysis of Intel Mainstream SSDs and Marc Prieur s report from BeHardware.com which includes an official response from Intel regarding this phenomenon. Laurent Gilson posted on the Linux-Thinkpad mailing list that when he tried using the X25-M to store commit journals for a database, that after writing 170% of the capacity of an Intel SSD, the small writes caused the write performance to go through the floor. More troubling, Allyn Malventano indicated that if the drive is abused for too long with a mixture of small and large writes, it can get into a state where the performance degredation is permanent, and even a series of large writes apparently does not restore the drive s function only an ATA SECURITY ERASE command to completely reset the mapping table seems to help. So, what can be done to prevent this? Allyn s review speculates that aligning writes to erase write boundaries can help I m not 100% sure this is true, but without detailed knowledge of what is going on under the covers in Intel s SSD, we won t know for sure. It certainly can t hurt, though, and there is a distinct possibility that the internal sector size is larger than 512 bytes, which means the default partitioning scheme of 255 heads/63 sectors is probably not a good idea. (Even Vista has moved to a 240/63 scheme, which gives you 8k alignment of partitions; I prefer 224/56 partitioning, since the days when BIOS s used C/H/S I/O are long gone.) The Ext3 and Ext4 file system tend to defer meta-data writes by pinning them until a transaction commit; this definitely helps, and ext4 allows you to configure an erase block boundary, which should also be helpful. Enabling laptop mode will discourage writing to the disk except in large blocks, which probably helps significantly as well. And avoiding fsync() in applications will also be helpful, since a cache flush operation will force the SSD to write to an erase block even if it isn t completely filled. Beyond that, clearly some experimentation will be needed. My current thinking is to use a standard file system aging workload, and then performing an I/O benchmark to see if there has been any performance degradation. I can then vary various file system tuning parameters and algorithms, confirm whether or not a heavy fsync workload makes the performance worse. In the long term, hopefully Intel will release a firmware update which adds support for ATA TRIM/DISCARD commands, which will allow the file system to inform the SSD that various blocks have been deleted and no longer need to be preserved by the SSD. I suspect this will be a big help, if the SSD knows that certain sectors no longer need to be preserved, it can avoid copying them when trying to defragment the SSD. Given how expensive the X25-M SSD s are, I hope that there will be a firmware update to support this, and that Intel won t leave its early adopters high and dry by only offering this functionality in newer models of the SSD. If they were to do that, it would leave many of these early adopters, especially your humble writer (who paid for his SSD out of his own pocket), to be quite grumpy indeed. Hopefully, though, it won t come to that. Update: I ve since penned a follow-up post Should Filesystems Be Optimized for SSD s? Related posts (automatically generated):

1. Should Filesystems Be Optimized for SSD s? In one of the comments to my last blog entry,...
2. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...

Got myself a new laptop this week. Not that the other one broke down or some such, but it was getting old; and its single-core 1.3Ghz PowerPC was starting to be somewhat slow in comparison to what's available on the market today. Additionally, while it was still functional, there were some issues with wear-out; for instance, sometimes the battery would just fall out while I was walking around with it, or the network cable connection would not fit entirely, and similar minor problems The new one is quite a bargain; 2G of RAM, 2Ghz Core2Duo, one of those newfangled 'n' wireless network interfaces, plenty of diskspace, and other interesting things; and all that for only 600ish. You'd say it's a netbook, but it isn't although I guess the low price of those netbook systems is pulling down the bottom line of these machines. Anyway. After using it for slightly more than a day, I'm quite happy with it. I did have to upgrade a few packages in order to make all the hardware work properly; for instance, kernel 2.6.26 does not yet support the wireless (so I'm running 2.6.28 now), and the X.org in unstable can only get at the screen in vesa mode, which means I'm running the 7.4 X.org packages from experimental. But overall, most of it (that I care about) works flawlessly. With a single exception: I have no audio. The weird thing is, the audio chipset is recognized, and I can play around with alsamixer; but no matter, there is no sound. Which is kindof annoying. It doesn't appear that I'm the only one with this problem (witness this debugging page on the alsa wiki), so I'm sure I'll get there. Eventually. Oh wel. While fetching the laptop, I was told that the NSLU2 which I had on backorder had arrived, so I took that with me, too. Add to that the 160G disk that I'd fetched earlier, and the USB NIC that I also got, and the Thecus N2100 that's still running at my parents' place will soon be moved to my home again and its 500G RAID1 with it. Whee.

If you're interested in free software --- and free network services in particular --- and should try to join me in Boston for the weekend of March 21st and 22nd. The FSF is organizing its annual members meeting again. This year the model is very different. For a start, the audience isn't limited to FSF members and the conference is not just about FSF projects and work. Instead, the meeting has been rebranded LibrePlanet and has been broken up into a two-day event that is going to talk about and then try to tackle some of the biggest problems facing the world of free and open source software. Saturday March 21st will feature a series of talks about major issues facing free software. Sunday March 22nd will be focused on an unconference attempt to tackle and explore several of the key themes or tracks: network services, high priority projects, and the nascent LibrePlanet activism network. I'll be focused on the track around free network services which I'm helping organize in part through Autonomo.us. For more information on that angle, please take a look at my blog post over at Autonomo.us. We're going to have a great group of people at the track and I'm excited by the idea that that we'll be able to make some real progress on the issues. I encourage anyone who thinks they might be able to make it to consider doing so. There are details including travel, location, hotel information and much more on the event web page and wiki (login is required to RSVP). Please spread the word!

I have been cleaning my ~/ and noticed a lot of files in ~/.metacity/sessions/ and ~/.config/metacity/sessions/ (800+) and in ~/.nautilus/ (600+). Nautilus and Metacity are to balem for these behaviours. Bugs have already been reported in the Debian BTS[1][2] and upstream [3][4]. I hope this will be fixed shortly.

1. http://bugs.debian.org/443665
2. http://bugs.debian.org/469267
3. http://bugzilla.gnome.org/show_bug.cgi?id=412755
4. http://bugzilla.gnome.org/show_bug.cgi?id=513881

Some guy from New Zealand blogged about a vulnerability in .desktop files as used by Gnome and KDE, and claims you can write a virus with them for Linux. He actually meant a Trojan rather than a virus, but still. I'm not saying the vulnerability does not exist. It does, and the trojan he describes should work in theory. However, in practice, I believe that the real reason why there aren't any Linux viruses is not the fact that Linux is somehow 'safer', but really the fact that Linux isn't a monoculture like Windows is. If you want to write a Windows virus, you have to deal with, at most, four to five different versions of windows that are currently still in active use with most users. Target that, and your virus will live like there's no tomorrow. If you want to write a Linux virus, the range of target system is much, much larger. In his very blog entry that describes the vulnerability, 'foobar' describes already two special cases he has to consider: the fact that KDE and Gnome write .desktop entries that get executed on startup to different locations, and the fact that, while most distributions ship either curl or wget, none of them has standardized on either of those, requiring the virus writer to account for that, too. While his blog entry stops at that, there's way, way more he has to deal with. If our supposed trojan writer wants his trojan to proliferate, he will have to deal with distributions that come with or without SE Linux; distributions that ship with a /tmp mounted noexec; distributions that ship with python 2.3, 2.4, 2.5, or 3.0; users who use Thunderbird, Evolution, Kmail, mutt, pine, or another mail client; and possibly much, much more (this only describes what's required as described in the referenced blog post). All these differences would make writing some malware that would exploit the described vulnerability tedious, at best, and rather impractical in most cases. Again, I'm not saying the vulnerability does not exist; it does. But I think this lack of monoculture under Linux, not the (perceived) strength of the platform, is what helps users defend against malware on Linux.

Ok, my last version was a pure Bash solution: working, but slow. There were some comments how to improve the performance and I decided finally to reimplement the second script as Python script.
The Bash script didn't finish within a day. The Python script ends after 1-2 hours in my test scenario. So, here are the scripts again:

make-lsLR.sh - call this regularly (cron) to create the needed files that are stored in /root/. Of course you can alter the location easily and exclude other directories from being scanned. [download make-lsLR.sh]


check_lost+found.py - The second script is to be run when your fsck managed to mess up with your files and stored them into lost+found directory. It takes 3 arguments: 1) the source directory where your messed up lost+found directory is, 2) the target directory to which the data will be saved and 3) a switch to actually make it happen instead of a dry-run. [download check_lostfound.py.sh]


I've chosen to copy the files to a different place instead of moving them within the same filesystem to their original place for safety reasons. Primary goal is to retrieve the files from lost+found, not to replace a full featured backup and restore application. Because of this the script doesn't handle hard- nor symlinks correctly. It just copy files.

Of course there's still room for improvements, like handling hard-/symlinks correctly or using inode number instead of md5sums to move data back to its prior location. But it works for me[tm] well enough in this way, so I'm satisfied so far. You're welcome, though, to improve this piece of ugliness if you like.

Maybe someone else finds this usefull as well. Use it on your own risk, of course. :)

I ve decided I m going to blog in Spanish from time to time (1, 2). These will be posts personal in nature, and I don t think I ll be sending them to Planeta Debian. I ve also reworked (with some pain) the list of feeds in my blog. Apart from the feed with all content, there is now:

• a feed of what is sent to Planet Debian.
• a feed for entries about life, many of which will be sent to Planet Debian as well, and which will include all entries in Spanish.
• a feed for Spanish-only content.

Red Hat has just announced a deal with MS for support of RHEL virtual machines on Windows Server and Windows virtual machines on RHEL [1]. It seems that this deal won t deliver anything before calendar H2 2009 so nothing will immediately happen - but the amount of testing to get these things working correctly is significant. Red Hat has stated that the agreements contain no patent or open source licensing components and the agreements contain no financial clauses, other than industry-standard certification/validation testing fees so it seems that there is nothing controversial in this. Of course that hasn t stopped some people from getting worked up about it. I think that this deal is a good thing. I have some clients who run CentOS and RHEL servers (that I installed and manage) as well as some Windows servers. Some of these clients have made decisions about the Windows servers that concern me (such as not using ECC RAM, RAID, or backups). It seems to me that if I was to use slightly more powerful hardware for the Linux servers I could run Windows virtual machines for those clients, manage all the backups at the block device level (without bothering the Windows sysadmins). This also has the potential to save the client some costs in terms of purchasing hardware and managing it. When this deal with MS produces some results (maybe in 6 months time) I will recommend that some of my clients convert CentOS machines to RHEL to take advantage of it. If my clients take my advice in this regard then it will result in a small increase in revenue and market share for RHEL. So Red Hat s action in this regard seems to be a good business decision for them. If my clients take my advice and allow me to use virtualisation to better protect their critical data that is on Windows servers then it will be a significant benefit for the users.

• [1] http://www.redhat.com/promo/svvp/?intcmp=70160000000HiHHAA0

I think I've made the decision that at some point in the next few months the xen-hosting.org setup I maintain will be going away, and will be replaced with kvm-hosting(.org). The only issue I need to ponder is handling the migration with the minimum downtime. The plan would probably involve upgrading the host machine to Lenny, then installing KVM and fiddling with filesystems until the guests boot. I suspect it wouldn't be a huge job, but there are a few issues that will need to be planned. Most notably I expect that most of the current guests don't have grub installed, etc, so we'd be in the position to use an external kernel + initrd. That's not an insurmountable problem, but I know that externally supplied kernels have caused me problem in the past with KVM. Perhaps the actual plan would be to wait until September at which point I could order a new machine and cancel the current one. That would mean another increase in spec and the migration process would be a lot simpler - instead of everybody being offline for a few hours I could migrate guests individually from the old host to the new. Anyway decisions decisions .. ObFilm: Buffy - But we'll pretend the TV series counts as a film, kthxbye?

I may be wrong on this but it looks like Microsoft SMTP clients (at least Windows Mail and Outlook) don t like being sent a large volume of SSL certificate information when opening a TLS connection. They appear to assume that the data they are being sent is malformed and assume that STARTTLS failed, continuing with an unencrypted SMTP dialogue. This can be triggered relatively easily on a Debian system by telling Exim to use all the certificates provided by the ca-certificates package (which is the default configuration). The Windows clients will give an unhelpful the remote end dropped the connection style error, caused by the server getting upset by the unexpected fallback to unencrypted SMTP. The server logs will show something like this:

2009-02-17 21:32:55 TLS error on connection from client.example.com (Client) [192.168.192.168] (gnutls_handshake): A TLS packet with unexpected length was received.
2009-02-17 21:33:00 SMTP protocol synchronization error (input sent without waiting for greeting): rejected connection from H=client.example.com [192.168.192.168] input="EHLO Client\r\n"

Configuring the MAIN_VERIFY_TLS_CERTIFICATES option in the Debian Exim configuration (which sets the tls_verifiy_certificates option in the actual Exim configuration) to point to something with less certificates in should avoid the issue. On the bright side, at least they re making an effort to avoid overflows.

The Open Source Initiative and the Free Software Foundation are probably the two authorities on what is free and open source software that are better respected than the debian project. It s fairly easy to see that the famous licences (GNU GPL) appear on both lists, but how similar are their approvals? There was a comparison at Asheeshworld Notes you will like: OSI vs FSF in 2007, but I ve noticed more and more duplicates in that listing and the source code didn t work for me. I ve captured the current differences between their actual approvals in a pseudo-diff. The generating code and intermediate steps are splattered around the same folder. Many of the differences look like a result of process differences, but I m not sure because OSI doesn t make any comments and their licence review process archive seems awkward to search. As I understand it, a lawyer advocates a licence in the OSI process, so it requires the licensor to contribute (and many licensors couldn t care less about OSI); but FSF does an independent review, so FSF has to decide it s worth reviewing. In case it s not obvious, I think FSF s independent foundation-led review is much the better of those two. There are two licences that OSI approved but FSF lists as non-free: NASA Open Source Agreement and the Reciprocal Public License. Both of these are send-back-ware where changes have to be sent back or made available to the upstream developer. Many debian developers agree such payments-in-kind are non-free (OSI s OSDefinition is based on the Debian Free Software Guidelines), but I think an early OSI advisor thought they were a good idea, so those approvals look like an OSI bug to me. It s not obvious if there are licences that FSF approves but OSI rejects. OSI doesn t seem to list licences to avoid. Is it helpful to see the differences? Would a comparison with the debian archive be useful? Are there other curiousities captured in the differences? Are people interested in reconciling the differences?

Given the fact that patients have been shamelessly and copiously sharing information on the web since, well, it first sprung up, it is nice to see that the medical establishment has gotten on board with the idea of collaboration. Medpedia is firmly an expert project (though it looks like anyone can provide suggestions) and I am not bothered by this limitation. What I do hope to see is a recognition of the controversies that abound in the realm of medicine (Lyme disease, fibromylagia, and a real hot rod issue = autism). There is much about diagnosis and therapy in medicine that exists in the I am not sure (how to diagnose/treat category and I wonder if this will be reflected in this new project. I certainly hope so.