Search Results: "ag"

22 February 2009

Andre Luis Lopes: Debian : a virtualization friendly platform

It s no secret for those who follow me on Twitter/identi.ca that I have been toying with virtualization technologies for some time now. I have been using Xen in production for some time and, apart from not thinking it s the way to go for future virtualization, it s quite usable right now, specially if you are going to use Debian Lenny as the dom0, as it now has a more modern kernel than the dreaded 2.6.18 we Xen users were forced to keep using. However, I have been keeping my eyes on KVM as well, as I always liked how much simpler and well integrated into the Linux kernel it is. I m even subscribed to the KVM development mailing list, even not understanding most of the things the developers are talking about there. It s no problem for me, I just want to know what s being worked on and what s on the pipeline. As a KVM fan, I m also using it intensively as a tool for prototyping servers. It s easy enough to set up a new Debian server to test things on and learn new technologies, as well as troubleshoot problems without intefering with production environments. Althought KVM is not as good as Xen when it comes to performance, it s quickly improving every single day. And, for those of you who still believes KVM is only about full virtualization, I m happy to say that it has come a long way and paravirtualization has started to infiltrate KVM land as well. Today, KVM already sports paravirtualized clock (pvclock), paravirtualized memory management unit (pv MMU) and VirtIO drivers. Actually, it seems that KVM these days is the biggest user of VirtIO, maybe only loosing the leadership to Rusty s lguest. By using VirtIO s disk and network drivers (virtio_blk and virtio_net, respectively, and its associated modules), KVM can deliver a much improved I/O experience than when using QEMU s emulated drivers. For this reason, it s always preferable to use VirtIO drivers whenever possible when setting up your KVM guests. During the Etch lifetime, one will need to do some tricks in order to use VirtIO when using Etch as a guest under KVM. However, when preparing Lenny s d-i, the developers were smart enough to add to it virtual disk detection support. What it means is that now, starting with Lenny, d-i will recognize that it s being given a VirtIO block device and automatically load the needed kernel modules to support it. Also, the disk detection and partition modules (partman et all) were modified to show a detected virtual disk (i.e. /dev/vdX) and let the user partition it, as well as grub-installer was changed to allow GRUB to be installed onto a virtual disk s MBR, effectively making d-i a really powerful virtualization aware installer. Here you can see Lenny s d-i showing a detected virtual disk, named vda :
lenny-vdisk-partitioning-english

lenny-vdisk-partitioning-english

Brazilian Portuguese readers could see the version using brazilian portuguese texts on the screen here. Sure, d-i also received a lot of improvements and special support for installing Lenny as a Xen domU was added as well, but I haven t played with it yet, so I won t comment on that right now. And, hey, one can even use virt-install and virt-manager to deploy KVM guests under Lenny :-) And you know what gives me even more confidence that KVM will be a first class citizen inside Debian ? The fact that Steve Kemp is toying with the idea to change his xen-hosting.org project so it would become a new kvm-hosting.org project. Maybe I m praying for the preacher here, but Steve is very well well know for being the author of xen-tools and xen-shell, as well as being the creator of a number of other nice free softwares. And, judging by the comments on his post about the future kvm-hosting.org project, it seems that Steve maybe will need to update xen-tools and xen-shell to account for KVM or create a new set of tools dedicated exclusively to KVM based on his past experience creating the current tools for Xen. Good times ahead for those of us who are surfing the Debian Virtualization wave, indeed :-)

21 February 2009

Uwe Hermann: Debian Lenny 5.0 released

Debian Lenny banner Yes, just when you thought the spamming of Planet Debian with "Lenny released" blog posts had finally stopped, here comes another one :-) Let me join the crowd by saying a great "thank you!" to all the people who made this release possible, especially so the release team who organized everything, as well as the thousands of contributors (in one form or another) who helped shape the new release! Personally, I'm eager to try out the new Linux 2.6.28 kernel package in unstable now (which have been uploaded today or so, but haven't yet reached my mirror), since they contain mainline wireless drivers for my One A110 netbook, among many other things. Also, in the next few days I'll probably re-install my NSLU2 ARM box using the latest Lenny installer, following the HOWTOs by Martin Michlmayr (I'll probably write about the experience later). This re-install is long overdue, as I'm currently running the box from an 1GB thumb drive, which works ok, but I'm slowly running out of space. So I'll re-install on a 4 GB (or bigger) thumb drive.

Norbert Tretkowski: First backports for lenny available

Alexander Wirt just wrote a mail to our mailinglist to announce that lenny-backports suite is ready for uploads. We will of course continue supporting etch-backports with security updates as long as etch will be supported by the Debian security team.

Happy backporting!

Pablo Lorenzzoni: Git basics: reversing the 'git sucks' effect

I ve been using git this last few days and I am still working on a workflow for my projects. Unfortunately, as others have noticed, git violates POLS is so many ways, it ends up being hard to get. Creating a remote repository seems to be the first thing to bite a developer switching to git (mainly if coming from a centralized SCM). I have not decided which is the best way of doing it, but I ve been using git-daemon via inetd and a path in my remote hosting holding all my repositories for public pulling, and ssh for pushing. Here is how to create it:

local$ ssh spectra@remotehost
remotehost$ mkdir /var/git/myproject.git
remotehost$ cd /var/git/myproject.git
remotehost$ git --bare init 
remotehost$ logout
local$ cd myproject
local$ git remote add remotehost spectra@remotehost:/var/git/myproject.git
local$ git push remotehost master
...
otherlocal$ git clone git://remotelocal/myproject.git
otherlocal$ cd myproject
otherlocal$ git remote add remotehost spectra@remotehost:/var/git/myproject.git
(hack hack hack)
otherlocal$ git push remotehost master
Now everybody can pull your repository at git://remotehost/myproject.git and you can push and pull to it via ssh. Note that you have to setup git-daemon, which is pretty straight forward. I am using it as an inetd daemon, but you can use it as a standalone one. Debian has a package which does just that. Now, some people think logging in a remote server just to create an empty repository is too much. Well repositories are just .git directories. It happens that you can push for the first time by rsyncing your .git with a remote host:

local$ cd myproject
local$ rsync -a .git/ spectra@remotehost:/var/git/myproject.git
local$ git remote add remotehost spectra@remotehost:/var/git/myproject.git
(hack hack hack)
local$ git push remotehost master
(Apparently you can push using rsync every time, but it s regarded as wiser to keep your probably crappy local repository commits separated from the public repository otherwise commit messages like Please, don t use this code are likely to pop up everywhere :) ). Now, I don t know if this have side effects, but it works :-) Another thing to notice is that git is more directed at pulling than pushing. This may be because of its designer: the way Linus works is by pulling changes from others repositories and not by letting others push em into his one. And this is another violation of POLS for most of the people, who is used to commit their changes into some remote repository. Rather than that, people using git would expose their own repositories in order to have it pulled by others. I also agree with most of the git critics wrt git commands There s a lot of examples and I am not going deeper in this but I think it was a bad choice calling checkout what git does when told to checkout , for instance. Yes, I know different tools, different ways of seeing it but everyone was already used to what centralized SCMs call their operations, and I think it would only help git adopting same names for the same operations, and inventing new ones for those proper of decentralized operations. Anyway, once you get it (and I have not completely got it yet), it seems all flow as expected. If this adaptation fails, there s still Easy Git to the rescue! One last thing that I think contribute to the git sucks effect: git-svn. This is a great tool, but it was built from git s point of view Given it s intended as a glue for Subversion newcomers, it would benefit more from being built from svn s point of view. This was mentioned by a colleague developer in my company, when he just couldn t understand why one have to git svn rebase instead of a simple update. So git-svn also suffers of the bad command naming habit git do. Of course, that given that you came from this environment (I am sure git and git-svn makes perfect sense for Linus & cia :-) ). I have not tried yet, but yap seems to be targeted on providing an alternative to git-svn.

René Mayorga: It is already too late to say this, but welcome to Lenny \o/

I start to move some servers when lenny was freeze, and did not have any complain, the update was really smooth; but I still let some servers to be migrated just at the end of the release cycle, and sure, they fail I actually have some fun migrating freeradius and cacti, and note that I actually will have more fun migrating asterisk and openser servers, but for the sake of this release, is the fun that every BOFH should expect, and also, cheers and thanks to everyone who work on this release :)

20 February 2009

Mike Hommey: The niceties of proprietary software

On a deployment I m currently working on, I ve seen two different cases of proprietary software use leading to both madness and sadness, which are just so typical that I can t resist to tell you. Keep in mind, for the rest of this reading, that the whole platform is running under Solaris 10 x86-64. The first case is a data quality management software that we will keep anonymous. The editor promised before the deployment began that they had a version of the software for the operating system that we would be using. Well, it turned out they do have a Solaris version for sparc, and a x86-64 version for Linux. No Solaris x86-64 version, and no way to get a rebuild in a timely fashion. The second case is a content management software that we will keep anonymous. It comes in the form of a java web application and a java application container. Part of integrating this software involves a proprietary plugin for Apache HTTPd that acts as a mix of mod_proxy_balancer, mod_disk_cache, and htcacheclean, as well as a cache invalidator. Originally, the java web application was supposed to be installed within a JBoss Application Server instead of the editor provided container, and Apache HTTPd would reverse proxy requests to the JBoss server. This means we already had an Apache HTTPd in place (latest version ; 2.2.11 at the time), and since we have x86-64 processors, it was built as a 64-bits binary. Contrary to the first case, this time, we had a Solaris x86 binary. Yes, you read correctly: x86 ; 32-bits only. After going through the pain of rebuilding Apache HTTPd in 32-bits (there are various reasons why we don t use sunfreeware software), it turned out the module was a 2.0 ABI module not compatible with the 2.2 ABI. It also turned out there was a 2.2 ABI version of the module for Solaris sparc. It finally worked after another build of Apache HTTPd, a 2.0.63 release, this time. The more you get used to free software, the more these kind of things get frustrating.

Kapil Paranjape: Stallman vs Torvalds (Oh! No! Not again!)

The recent issue of Frontline has an interview of Stallman. It is indeed a nice thing to see that nationally circulated news magazines in India are talking about Mukta Software. However, it is disappointing to see the FSF take one more pot-shot at Linus Torvalds.
Torvalds said he rejects GNU GPL Version 3 because he approves of tivoization, because he does not agree that users deserve the freedom to change their own copies of software.
By choosing the GPL for his own code Linus Torvalds gave users the freedom to change their own copies of Linux. This certainly seems at odds with what is quoted. There are many different points of view about how the cause of mukta software and its users can be strengthened. I think we can agree that it is certainly not strengthened by shooting each other in the foot. The GPLv3 bolsters-up one aspect of this cause: use the legal system to make if difficult (if not impossible) to take free software and put it in proprietary bottles. Linus Torvalds and all his "show me the code" buddies are addressing another aspect; to ensure that there are enough people accomplished in the task of reading, improving and writing high quality free software. The West has traditionally had more success with the law. When free programs was written in the 70's and 80's by people who did not care to preserve its freedom with a free license, this code was gobbled up and re-gurgitated as proprietary software. Once free licenses became commonplace in the 90's, the West has had a resurgence of high quality public code and coders. Unfortunately, some of the latter tend to look down on "mere users". Over here, in India, we have little faith in the legal system. (For reasons, see this report and a more recent one). We know that freedom is hard fought even when the law is technically on your side. At the same time, the brahminical approach of knowledge being safe-guarded by the self-proclaimed elite was what rang the death knell for Indian knowledge systems in the past. So it is important that the breed of skeptical people in India who tinker with their computers and software (and everything else) grows.
We have far too many cattle who will follow any flute-playing good-looking young person who is willing to fight their demons for them.

Theodore Ts'o: Aligning filesystems to an SSD s erase block size

I recently purchased a new toy, an Intel X25-M SSD, and when I was setting it up initially, I decided I wanted to make sure the file system was aligned on an erase block boundary. This is a generally considered to be a Very Good Thing to do for most SSD s available today, although there s some question about how important this really is for Intel SSD s more on that in a moment. It turns out this is much more difficult than you might first think most of Linux s storage stack is not set up well to worry about alignment of partitions and logical volumes. This is surprising, because it s useful for many things other than just SSD s. This kind of alignment is important if you are using any kind of hardware or software RAID, for example, especially RAID 5, because if writes are done on stripe boundaries, it can avoid a read-modify-write overhead. In addition, the hard drive industry is planning on moving to 4096 byte sectors instead of the way-too-small 512 byte sectors at some point in the future. Linux s default partition geometry of 255 heads and 63 sectors/track means that there are 16065 (512 byte) sectors per cylinder. The initial round of 4k sector disks will emulate 512 byte disks, but if the partitions are not 4k aligned, then the disk will end up doing a read/modify/write on two internal 4k sectors for each singleton 4k file system write, and that would be unfortunate. Vista has already started working around this problem, since it uses a default partitioning geometry of 240 heads and 63 sectors/track. This results in a cylinder boundary which is divisible by 8, and so the partitions (with the exception of the first, which is still misaligned unless you play some additional tricks) are 4k aligned. So this is one place where Vista is ahead of Linux . unfortunately the default 255 heads and 63 sectors is hard coded in many places in the kernel, in the SCSI stack, and in various partitioning programs; so fixing this will require changes in many places. However, with SSD s (remember SSD s? This is a blog post about SSD s ) you need to align partitions on at least 128k boundaries for maximum efficiency. The best way to do this that I ve found is to use 224 (32*7) heads and 56 (8*7) sectors/track. This results in 12544 (or 256*49) sectors/cylinder, so that each cylinder is 49*128k. You can do this by doing starting fdisk with the following options when first partitioning the SSD: # fdisk -H 224 -S 56 /dev/sdb The first partition will only be aligned on a 4k boundary, since in order to be compatible with MS-DOS, the first partition starts on track 1 instead of track 0, but I didn t worry too much about that since I tend to use the first partition for /boot, which tends not to get modified much. You can go into expert mode with fdisk and force the partition to begin on an 128k alignment, but many Linux partition tools will complain about potential compatibility problems (which are obsolete warnings, since the systems that would have booting systems with these issues haven t been made in about ten years), but I didn t needed to do that, so I didn t worry about it. So I created a 1 gigabyte /boot partition as /dev/sdb1, and allocated the rest of the SSD for use by LVM as /dev/sdb2. And that s where I ran into my next problem. LVM likes to allocate 192k for its header information, and 192k is not a multiple of 128k. So if you are creating file systems as logical volumes, and you want those volume to be properly aligned you have to tell LVM that it should reserve slightly more space for its meta-data, so that the physical extents that it allocates for its logical volumes are properly aligned. Unfortunately, the way this is done is slightly baroque: # pvcreate metadatasize 250k /dev/sdb2
Physical volume /dev/sdb2 successfully created Why 250k and not 256k? I can t tell you sometimes the LVM tools aren t terribly intuitive. However, you can test to make sure that physical extents start at the proper offset by using: # pvs /dev/sdb2 -o+pe_start
PV VG Fmt Attr PSize PFree 1st PE
/dev/sdb2 lvm2 73.52G 73.52G 256.00K If you use a metadata size of 256k, the first PE will be at 320k instead of 256k. There really ought to be an pe-align option to pvcreate, which would be far more user-friendly, but, we have to work with the tools that we have. Maybe in the next version of the LVM support tools . Once you do this, we re almost done. The last thing to do is to create the file system. As it turns out, if you are using ext4, there is a way to tell the file system that it should try to align files so they match up with the RAID stripe width. (These techniques can be used for RAID disks as well). If your SSD has an 128k erase block size, and you are creating the file system with the default 4k block size, you just have to specify a strip width when you create the file system, like so: # mke2fs -t ext4 -E stripe-width=32,resize=500G /dev/ssd/root (The resize=500G limits the number of blocks reserved for resizing this file system so that the guaranteed number size that the file system can be grown via online resize is 500G. The default is 1000 times the initial file system size, which is often far too big to be reasonable. Realistically, the file system I am creating is going to be used for a desktop device, and I don t foresee needing to resize it beyond 500G, so this saves about a 50 megabytes or so. Not a huge deal, but waste not, want not , as the saying goes.) With e2fsprogs 1.41.4, the journal will be 128k aligned, as will the start of the file system, and with the stripe-width specified, the ext4 allocator will try to align block writes to the stripe width where that makes sense. So this is as good as it gets without kernel changes to make the block and inode allocators more SSD aware, something which I hope to have a chance to look at.

horizontal separator All of this being said, it s time to revisit this question is all of this needed for a smart , better by design next-generation SSD such as Intel s? Aligning your file system on an erase block boundary is critical on first generation SSD s, but the Intel X25-M is supposed to have smarter algorithms that allow it to reduce the effect of write-amplification. The details are a little bit vague, but presumably there is a mapping table which maps sectors (at some internal sector size we don t know for sure whether it s 512 bytes or some larger size) to individual erase blocks. If the file system sends a series of 4k writes for file system blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, 99, followed by a barrier operation, a traditional SSD might do read/modify/write on four 128k erase blocks one covering the blocks 0-31, another for the blocks 32-63, and so on. However, the Intel SSD will simply write a single 128k block that indicates where the latest versions of blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, and 99 can be found. This technique tends to work very well. However, over time, the table will get terribly fragmented, and depending on whether the internal block sector size is 512 or 4k (or something in between), there could be a situation where all but one or two of the internal sectors on the disks have been mapped away to other erase blocks, leading to fragmentation of the erase blocks. This is not just a theoretical problem; there are reports from the field that this happens relatively easy. For example, see Allyn Malventano s Long-term performance analysis of Intel Mainstream SSDs and Marc Prieur s report from BeHardware.com which includes an official response from Intel regarding this phenomenon. Laurent Gilson posted on the Linux-Thinkpad mailing list that when he tried using the X25-M to store commit journals for a database, that after writing 170% of the capacity of an Intel SSD, the small writes caused the write performance to go through the floor. More troubling, Allyn Malventano indicated that if the drive is abused for too long with a mixture of small and large writes, it can get into a state where the performance degredation is permanent, and even a series of large writes apparently does not restore the drive s function only an ATA SECURITY ERASE command to completely reset the mapping table seems to help. So, what can be done to prevent this? Allyn s review speculates that aligning writes to erase write boundaries can help I m not 100% sure this is true, but without detailed knowledge of what is going on under the covers in Intel s SSD, we won t know for sure. It certainly can t hurt, though, and there is a distinct possibility that the internal sector size is larger than 512 bytes, which means the default partitioning scheme of 255 heads/63 sectors is probably not a good idea. (Even Vista has moved to a 240/63 scheme, which gives you 8k alignment of partitions; I prefer 224/56 partitioning, since the days when BIOS s used C/H/S I/O are long gone.) The Ext3 and Ext4 file system tend to defer meta-data writes by pinning them until a transaction commit; this definitely helps, and ext4 allows you to configure an erase block boundary, which should also be helpful. Enabling laptop mode will discourage writing to the disk except in large blocks, which probably helps significantly as well. And avoiding fsync() in applications will also be helpful, since a cache flush operation will force the SSD to write to an erase block even if it isn t completely filled. Beyond that, clearly some experimentation will be needed. My current thinking is to use a standard file system aging workload, and then performing an I/O benchmark to see if there has been any performance degradation. I can then vary various file system tuning parameters and algorithms, confirm whether or not a heavy fsync workload makes the performance worse. In the long term, hopefully Intel will release a firmware update which adds support for ATA TRIM/DISCARD commands, which will allow the file system to inform the SSD that various blocks have been deleted and no longer need to be preserved by the SSD. I suspect this will be a big help, if the SSD knows that certain sectors no longer need to be preserved, it can avoid copying them when trying to defragment the SSD. Given how expensive the X25-M SSD s are, I hope that there will be a firmware update to support this, and that Intel won t leave its early adopters high and dry by only offering this functionality in newer models of the SSD. If they were to do that, it would leave many of these early adopters, especially your humble writer (who paid for his SSD out of his own pocket), to be quite grumpy indeed. Hopefully, though, it won t come to that. Update: I ve since penned a follow-up post Should Filesystems Be Optimized for SSD s? Related posts (automatically generated):

  1. Should Filesystems Be Optimized for SSD s? In one of the comments to my last blog entry,...
  2. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...

Theodore Ts'o: Aligning filesystems to an SSD s erase block size

I recently purchased a new toy, an Intel X25-M SSD, and when I was setting it up initially, I decided I wanted to make sure the file system was aligned on an erase block boundary. This is a generally considered to be a Very Good Thing to do for most SSD s available today, although there s some question about how important this really is for Intel SSD s more on that in a moment. It turns out this is much more difficult than you might first think most of Linux s storage stack is not set up well to worry about alignment of partitions and logical volumes. This is surprising, because it s useful for many things other than just SSD s. This kind of alignment is important if you are using any kind of hardware or software RAID, for example, especially RAID 5, because if writes are done on stripe boundaries, it can avoid a read-modify-write overhead. In addition, the hard drive industry is planning on moving to 4096 byte sectors instead of the way-too-small 512 byte sectors at some point in the future. Linux s default partition geometry of 255 heads and 63 sectors/track means that there are 16065 (512 byte) sectors per cylinder. The initial round of 4k sector disks will emulate 512 byte disks, but if the partitions are not 4k aligned, then the disk will end up doing a read/modify/write on two internal 4k sectors for each singleton 4k file system write, and that would be unfortunate. Vista has already started working around this problem, since it uses a default partitioning geometry of 240 heads and 63 sectors/track. This results in a cylinder boundary which is divisible by 8, and so the partitions (with the exception of the first, which is still misaligned unless you play some additional tricks) are 4k aligned. So this is one place where Vista is ahead of Linux . unfortunately the default 255 heads and 63 sectors is hard coded in many places in the kernel, in the SCSI stack, and in various partitioning programs; so fixing this will require changes in many places. However, with SSD s (remember SSD s? This is a blog post about SSD s ) you need to align partitions on at least 128k boundaries for maximum efficiency. The best way to do this that I ve found is to use 224 (32*7) heads and 56 (8*7) sectors/track. This results in 12544 (or 256*49) sectors/cylinder, so that each cylinder is 49*128k. You can do this by doing starting fdisk with the following options when first partitioning the SSD: # fdisk -H 224 -S 56 /dev/sdb The first partition will only be aligned on a 4k boundary, since in order to be compatible with MS-DOS, the first partition starts on track 1 instead of track 0, but I didn t worry too much about that since I tend to use the first partition for /boot, which tends not to get modified much. You can go into expert mode with fdisk and force the partition to begin on an 128k alignment, but many Linux partition tools will complain about potential compatibility problems (which are obsolete warnings, since the systems that would have booting systems with these issues haven t been made in about ten years), but I didn t needed to do that, so I didn t worry about it. So I created a 1 gigabyte /boot partition as /dev/sdb1, and allocated the rest of the SSD for use by LVM as /dev/sdb2. And that s where I ran into my next problem. LVM likes to allocate 192k for its header information, and 192k is not a multiple of 128k. So if you are creating file systems as logical volumes, and you want those volume to be properly aligned you have to tell LVM that it should reserve slightly more space for its meta-data, so that the physical extents that it allocates for its logical volumes are properly aligned. Unfortunately, the way this is done is slightly baroque: # pvcreate metadatasize 250k /dev/sdb2
Physical volume /dev/sdb2 successfully created Why 250k and not 256k? I can t tell you sometimes the LVM tools aren t terribly intuitive. However, you can test to make sure that physical extents start at the proper offset by using: # pvs /dev/sdb2 -o+pe_start
PV VG Fmt Attr PSize PFree 1st PE
/dev/sdb2 lvm2 73.52G 73.52G 256.00K If you use a metadata size of 256k, the first PE will be at 320k instead of 256k. There really ought to be an pe-align option to pvcreate, which would be far more user-friendly, but, we have to work with the tools that we have. Maybe in the next version of the LVM support tools . Once you do this, we re almost done. The last thing to do is to create the file system. As it turns out, if you are using ext4, there is a way to tell the file system that it should try to align files so they match up with the RAID stripe width. (These techniques can be used for RAID disks as well). If your SSD has an 128k erase block size, and you are creating the file system with the default 4k block size, you just have to specify a strip width when you create the file system, like so: # mke2fs -t ext4 -E stripe-width=32,resize=500G /dev/ssd/root (The resize=500G limits the number of blocks reserved for resizing this file system so that the guaranteed number size that the file system can be grown via online resize is 500G. The default is 1000 times the initial file system size, which is often far too big to be reasonable. Realistically, the file system I am creating is going to be used for a desktop device, and I don t foresee needing to resize it beyond 500G, so this saves about a 50 megabytes or so. Not a huge deal, but waste not, want not , as the saying goes.) With e2fsprogs 1.41.4, the journal will be 128k aligned, as will the start of the file system, and with the stripe-width specified, the ext4 allocator will try to align block writes to the stripe width where that makes sense. So this is as good as it gets without kernel changes to make the block and inode allocators more SSD aware, something which I hope to have a chance to look at.

horizontal separator All of this being said, it s time to revisit this question is all of this needed for a smart , better by design next-generation SSD such as Intel s? Aligning your file system on an erase block boundary is critical on first generation SSD s, but the Intel X25-M is supposed to have smarter algorithms that allow it to reduce the effect of write-amplification. The details are a little bit vague, but presumably there is a mapping table which maps sectors (at some internal sector size we don t know for sure whether it s 512 bytes or some larger size) to individual erase blocks. If the file system sends a series of 4k writes for file system blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, 99, followed by a barrier operation, a traditional SSD might do read/modify/write on four 128k erase blocks one covering the blocks 0-31, another for the blocks 32-63, and so on. However, the Intel SSD will simply write a single 128k block that indicates where the latest versions of blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, and 99 can be found. This technique tends to work very well. However, over time, the table will get terribly fragmented, and depending on whether the internal block sector size is 512 or 4k (or something in between), there could be a situation where all but one or two of the internal sectors on the disks have been mapped away to other erase blocks, leading to fragmentation of the erase blocks. This is not just a theoretical problem; there are reports from the field that this happens relatively easy. For example, see Allyn Malventano s Long-term performance analysis of Intel Mainstream SSDs and Marc Prieur s report from BeHardware.com which includes an official response from Intel regarding this phenomenon. Laurent Gilson posted on the Linux-Thinkpad mailing list that when he tried using the X25-M to store commit journals for a database, that after writing 170% of the capacity of an Intel SSD, the small writes caused the write performance to go through the floor. More troubling, Allyn Malventano indicated that if the drive is abused for too long with a mixture of small and large writes, it can get into a state where the performance degredation is permanent, and even a series of large writes apparently does not restore the drive s function only an ATA SECURITY ERASE command to completely reset the mapping table seems to help. So, what can be done to prevent this? Allyn s review speculates that aligning writes to erase write boundaries can help I m not 100% sure this is true, but without detailed knowledge of what is going on under the covers in Intel s SSD, we won t know for sure. It certainly can t hurt, though, and there is a distinct possibility that the internal sector size is larger than 512 bytes, which means the default partitioning scheme of 255 heads/63 sectors is probably not a good idea. (Even Vista has moved to a 240/63 scheme, which gives you 8k alignment of partitions; I prefer 224/56 partitioning, since the days when BIOS s used C/H/S I/O are long gone.) The Ext3 and Ext4 file system tend to defer meta-data writes by pinning them until a transaction commit; this definitely helps, and ext4 allows you to configure an erase block boundary, which should also be helpful. Enabling laptop mode will discourage writing to the disk except in large blocks, which probably helps significantly as well. And avoiding fsync() in applications will also be helpful, since a cache flush operation will force the SSD to write to an erase block even if it isn t completely filled. Beyond that, clearly some experimentation will be needed. My current thinking is to use a standard file system aging workload, and then performing an I/O benchmark to see if there has been any performance degradation. I can then vary various file system tuning parameters and algorithms, confirm whether or not a heavy fsync workload makes the performance worse. In the long term, hopefully Intel will release a firmware update which adds support for ATA TRIM/DISCARD commands, which will allow the file system to inform the SSD that various blocks have been deleted and no longer need to be preserved by the SSD. I suspect this will be a big help, if the SSD knows that certain sectors no longer need to be preserved, it can avoid copying them when trying to defragment the SSD. Given how expensive the X25-M SSD s are, I hope that there will be a firmware update to support this, and that Intel won t leave its early adopters high and dry by only offering this functionality in newer models of the SSD. If they were to do that, it would leave many of these early adopters, especially your humble writer (who paid for his SSD out of his own pocket), to be quite grumpy indeed. Hopefully, though, it won t come to that. Update: I ve since penned a follow-up post Should Filesystems Be Optimized for SSD s? Related posts (automatically generated):

  1. Should Filesystems Be Optimized for SSD s? In one of the comments to my last blog entry,...
  2. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...

Alexander Reichle-Schmehl: Could someone please explkain me, what AddType x-mapp-php5 .php does?

We just had an interesting problem with a project based upon Drupal. From time to time one would get an error 500 (internal server error) for no apparent reason. Reloading the page would usually fix the issue. Attemps to search Apache logs which pages threw that error and searching for something in common turned out nothing. Via google we soon found out, that adding AddType x-mapp-php5 .php to our .htaccess file would fix the problem. Don't know why; googling in reverse that that does didn't help; most pages you get tell you that you can fix several php and/or Drupal related problems with that. So can anyone give me an hint, what happens, what the problem was and why on earth enabling php again solves the problem? PS: For testing purpouse we did something like the following: for x in $(seq 1 15000) ; do if $(wget -q $URL_TO_CHECK -O/dev/null) ; then echo "$x: klappt" ; else echo "$x: 500er" ; fi ; done >> test-log and then grep -c 500er. Before we added the line, we had nine internal server errors. With that line enabled, we didn't had a single one in over 50000 access. So that line indeed has some effect, but why?

Wouter Verhelst: New toys

Got myself a new laptop this week. Not that the other one broke down or some such, but it was getting old; and its single-core 1.3Ghz PowerPC was starting to be somewhat slow in comparison to what's available on the market today. Additionally, while it was still functional, there were some issues with wear-out; for instance, sometimes the battery would just fall out while I was walking around with it, or the network cable connection would not fit entirely, and similar minor problems The new one is quite a bargain; 2G of RAM, 2Ghz Core2Duo, one of those newfangled 'n' wireless network interfaces, plenty of diskspace, and other interesting things; and all that for only 600ish. You'd say it's a netbook, but it isn't although I guess the low price of those netbook systems is pulling down the bottom line of these machines. Anyway. After using it for slightly more than a day, I'm quite happy with it. I did have to upgrade a few packages in order to make all the hardware work properly; for instance, kernel 2.6.26 does not yet support the wireless (so I'm running 2.6.28 now), and the X.org in unstable can only get at the screen in vesa mode, which means I'm running the 7.4 X.org packages from experimental. But overall, most of it (that I care about) works flawlessly. With a single exception: I have no audio. The weird thing is, the audio chipset is recognized, and I can play around with alsamixer; but no matter, there is no sound. Which is kindof annoying. It doesn't appear that I'm the only one with this problem (witness this debugging page on the alsa wiki), so I'm sure I'll get there. Eventually. Oh wel. While fetching the laptop, I was told that the NSLU2 which I had on backorder had arrived, so I took that with me, too. Add to that the 160G disk that I'd fetched earlier, and the USB NIC that I also got, and the Thecus N2100 that's still running at my parents' place will soon be moved to my home again and its 500G RAID1 with it. Whee.

Benjamin Mako Hill: LibrePlanet 2009

If you're interested in free software --- and free network services in particular --- and should try to join me in Boston for the weekend of March 21st and 22nd. The FSF is organizing its annual members meeting again. This year the model is very different. For a start, the audience isn't limited to FSF members and the conference is not just about FSF projects and work. Instead, the meeting has been rebranded LibrePlanet and has been broken up into a two-day event that is going to talk about and then try to tackle some of the biggest problems facing the world of free and open source software. Saturday March 21st will feature a series of talks about major issues facing free software. Sunday March 22nd will be focused on an unconference attempt to tackle and explore several of the key themes or tracks: network services, high priority projects, and the nascent LibrePlanet activism network. I'll be focused on the track around free network services which I'm helping organize in part through Autonomo.us. For more information on that angle, please take a look at my blog post over at Autonomo.us. We're going to have a great group of people at the track and I'm excited by the idea that that we'll be able to make some real progress on the issues. I encourage anyone who thinks they might be able to make it to consider doing so. There are details including travel, location, hotel information and much more on the event web page and wiki (login is required to RSVP). Please spread the word!

Wouter Verhelst: On Linux viruses

Some guy from New Zealand blogged about a vulnerability in .desktop files as used by Gnome and KDE, and claims you can write a virus with them for Linux. He actually meant a Trojan rather than a virus, but still. I'm not saying the vulnerability does not exist. It does, and the trojan he describes should work in theory. However, in practice, I believe that the real reason why there aren't any Linux viruses is not the fact that Linux is somehow 'safer', but really the fact that Linux isn't a monoculture like Windows is. If you want to write a Windows virus, you have to deal with, at most, four to five different versions of windows that are currently still in active use with most users. Target that, and your virus will live like there's no tomorrow. If you want to write a Linux virus, the range of target system is much, much larger. In his very blog entry that describes the vulnerability, 'foobar' describes already two special cases he has to consider: the fact that KDE and Gnome write .desktop entries that get executed on startup to different locations, and the fact that, while most distributions ship either curl or wget, none of them has standardized on either of those, requiring the virus writer to account for that, too. While his blog entry stops at that, there's way, way more he has to deal with. If our supposed trojan writer wants his trojan to proliferate, he will have to deal with distributions that come with or without SE Linux; distributions that ship with a /tmp mounted noexec; distributions that ship with python 2.3, 2.4, 2.5, or 3.0; users who use Thunderbird, Evolution, Kmail, mutt, pine, or another mail client; and possibly much, much more (this only describes what's required as described in the referenced blog post). All these differences would make writing some malware that would exploit the described vulnerability tedious, at best, and rather impractical in most cases. Again, I'm not saying the vulnerability does not exist; it does. But I think this lack of monoculture under Linux, not the (perceived) strength of the platform, is what helps users defend against malware on Linux.

Chris Lawrence: Be still my heart

From the description of the memisc package for R:
One of the aims of this package is to make life easier for useRs who deal with survey data sets. It provides an infrastructure for the management of survey data including value labels, definable missing values, recoding of variables, production of code books, and import of (subsets of) SPSS and Stata files. Further, it provides functionality to produce tables and data frames of arbitrary descriptive statistics and (almost) publication-ready tables of regression model estimates. Also some convenience tools for graphics, programming, and simulation are provided. [emphasis added]
How did I miss this package before? It makes analyzing NES data heck, any data with value labels and missing values in R an almost sane thing to do.

19 February 2009

Ingo Juergensmann: Automatically restore files from lost+found - improved

Ok, my last version was a pure Bash solution: working, but slow. There were some comments how to improve the performance and I decided finally to reimplement the second script as Python script.
The Bash script didn't finish within a day. The Python script ends after 1-2 hours in my test scenario. So, here are the scripts again:

make-lsLR.sh - call this regularly (cron) to create the needed files that are stored in /root/. Of course you can alter the location easily and exclude other directories from being scanned. [download make-lsLR.sh]
CODE:
#!/bin/bash
#
# Usage: ./make-lsLR.sh
#
# Purpose:
# to make a file that is parseable for recovering
# a filled /lost+found directory by parsing
# filesize, md5sum, permisions and path+filename
#
# Author: Ingo Juergensman - http://blog.windfluechter.net
# License: GPL v2, see http://gnu.org for details.
#
# first: get all directories
nice -15 find / -path /sys -prune -o \
-path /proc -prune -o \
-path /var/lib/backuppc -prune -o \
-path /var/spool/squid -prune -o \
-type d -printf "%U:%G %#m " -print > /root/ls-md5sum-dirs.txt

exit 0

# next: get all relevant information
nice -15 find / -path /sys -prune -o \
-path /proc -prune -o \
-path /var/lib/backuppc -prune -o \
-path /var/spool/squid -prune -o \
-type f -printf "%s %U:%G %#m " \
-exec nice -15 md5sum \; tr -s " " > /root/ls-md5sum-files.txt


check_lost+found.py - The second script is to be run when your fsck managed to mess up with your files and stored them into lost+found directory. It takes 3 arguments: 1) the source directory where your messed up lost+found directory is, 2) the target directory to which the data will be saved and 3) a switch to actually make it happen instead of a dry-run. [download check_lostfound.py.sh]
CODE:
#!/usr/bin/python
#
# usage: check_lostfound.py <pathtolost+found> [make_it_so]
#
# Purpose: to find files in lost+found and trying to restore
# original files by comparing ls-md5sum-files.txt (generated by
# make-lsLR.sh
# Option make_it_so cause the data actually being written/moved
# whereas the script runs in dry mode per default.
#
# Author: Ingo Juergensman - http://blog.windfluechter.net
# License: GPL v2, see http://gnu.org for details.
#

from string import *
import sys, os
from os.path import *
import string

if len(sys.argv)<>4:
print "usage: "+sys.argv[0]+" <source> <target> <mode>"
sys.exit(1)
else:
mnt = sys.argv[1]
target = sys.argv[2]
mode = sys.argv[3]

dirs="/root/ls-md5sum-dirs.txt"
source="/root/ls-md5sum-files.txt"
lost="/root/lostfound-files.txt"
print "Creating list of files in %s/lost+found" % mnt
cmd = 'find %s/lost+found -type f -printf "%%s %%U:%%G %%#m " -exec nice -15 md5sum \; > /root/lostfound-files.txt' % mnt
os.system(cmd)

d = open(dirs, 'r+')
f = open(source, 'r+')
l = open(lost, 'r+')

sfiles=
lfiles=[]


# create the missing directories first
print "Creating missing directories in %s" % target
for entry in d:
ugid=string.split(entry, " ", 2)[0]
perm=string.split(entry, " ", 2)[1]
pfad="%s%s" % (target,string.split(replace(entry, "\n", ""), " ", 2)[2:][0])
res = isdir(pfad)
if (os.path.exists(pfad) and os.path.isdir(pfad)):
print "%s exists... " % pfad
else:
cmd = "%s %s - mkdir %s" % (ugid, perm, pfad)
print cmd
try:
if mode == "make_it_so":
os.makedirs(pfad)
uid, gid = string.split(ugid, ":")
os.chown(pfad, int(uid), int(gid))
os.chmod(pfad, int(perm))
except:
print "%s exists... " % pfad

# now parse /root/ls-md5sum-files.txt to get
# md5sum and /path/to/filename pairs
for line in f:
line=replace(line, "\n", "")
#size1 = split(line)[0]
#ugid1 = split(line)[1]
#perm1 = split(line)[2]
md5s1 = string.split(line)[3]
#path1 = split(line)[4]
sfiles[md5s1] = strip(str(string.split(line," ", 4)[4:][0]))

# next: do the same to the files in lost+found
for line in l:
size2 = string.split(line)[0]
ugid2 = string.split(line)[1]
perm2 = string.split(line)[2]
md5s2 = string.split(line)[3]
path2 = strip(str(string.split(replace(line, "\n", "")," ", 4)[4:][0]))
s = "%s %s %s %s %s" % (md5s2, ugid2, perm2, size2, path2)
lfiles.append(s)


# finally look at lost+found and copy the files
# to the appropriate place. Instead of copying
# the files can be moved as well, but to copy is
# safer in case of mistakes or errors
for lf in lfiles:
md5s = string.split(lf)[0]
lfile = string.split(lf, " ", 4)[4:][0]
if sfiles.has_key(md5s):
if os.path.exists(target+sfiles[md5s]):
pass
else:
targetfile = sfiles[md5s]
print "restoring %s%s " % (target, targetfile) #lfile
cmd = 'cp -p "%s" "%s%s"' % (lfile, target, targetfile)
if mode == "make_it_so":
os.system(cmd)
del sfiles[md5s]




I've chosen to copy the files to a different place instead of moving them within the same filesystem to their original place for safety reasons. Primary goal is to retrieve the files from lost+found, not to replace a full featured backup and restore application. Because of this the script doesn't handle hard- nor symlinks correctly. It just copy files.

Of course there's still room for improvements, like handling hard-/symlinks correctly or using inode number instead of md5sums to move data back to its prior location. But it works for me[tm] well enough in this way, so I'm satisfied so far. You're welcome, though, to improve this piece of ugliness if you like.

Maybe someone else finds this usefull as well. Use it on your own risk, of course. :)

Yves-Alexis Perez: Cookies!

Hey hey, look what I had in my mailbox this morning: Cookies! Cookies! Just in time for Lenny release, here are my precious cookies for BugSprint. Thanks Franklin!

Andrew Pollock: [opinion] Oh the irony

The website for Tourism Queensland's extremely popular promotion to find someone to be paid $150,000 to be an island caretaker for 6 months is hosted by a Canberra company on a server in Canberra. So much for supporting local business. (Small world, one of the Executive Directors was briefly my manager at CyberTrust, but that's the Canberra IT scene for you)

Matthew Garrett

In other news, my HP 2510p's screen was replaced last month after the hinge snapped. A chunk of plastic off the hinge cover snapped off two days ago. I'm somewhat puzzled by this, since I can't see any plausible way force could be applied to it - it's as if it came away slightly and then got crushed when I tried to close the lid. On top of the motherboard having been replaced 4 times now (twice due to faulty power connectors, one due to the fan being replaced and the motherboard being swapped at the same time, once because the machine started refusing to boot at LCA) and it still being slightly tempremental when booting, I'm not overly impressed - especially when I've only had it 18 months. Nobody else I know with one seems to have had the same level of difficulty, though dreadful thermal issues (especially when using the dock) seem to be common.

The X200s looks awfully shiny, but the 1440x900 screen option doesn't appear to be available in the UK. Oddly, despite having a SIM slot, it also doesn't seem to come with an HSDPA option.

18 February 2009

Russell Coker: Red Hat, Microsoft, and Virtualisation Support

Red Hat has just announced a deal with MS for support of RHEL virtual machines on Windows Server and Windows virtual machines on RHEL [1]. It seems that this deal won t deliver anything before calendar H2 2009 so nothing will immediately happen - but the amount of testing to get these things working correctly is significant. Red Hat has stated that the agreements contain no patent or open source licensing components and the agreements contain no financial clauses, other than industry-standard certification/validation testing fees so it seems that there is nothing controversial in this. Of course that hasn t stopped some people from getting worked up about it. I think that this deal is a good thing. I have some clients who run CentOS and RHEL servers (that I installed and manage) as well as some Windows servers. Some of these clients have made decisions about the Windows servers that concern me (such as not using ECC RAM, RAID, or backups). It seems to me that if I was to use slightly more powerful hardware for the Linux servers I could run Windows virtual machines for those clients, manage all the backups at the block device level (without bothering the Windows sysadmins). This also has the potential to save the client some costs in terms of purchasing hardware and managing it. When this deal with MS produces some results (maybe in 6 months time) I will recommend that some of my clients convert CentOS machines to RHEL to take advantage of it. If my clients take my advice in this regard then it will result in a small increase in revenue and market share for RHEL. So Red Hat s action in this regard seems to be a good business decision for them. If my clients take my advice and allow me to use virtualisation to better protect their critical data that is on Windows servers then it will be a significant benefit for the users.

Lior Kaplan: How to make an open source kindergarten


If you can read Hebrew, you ll find the original story at . Otherwise, you re welcome to read the post. Vitali Perchonok was asked to check the computer is his city of Kiryat Gat, he found a lot of old PCs (P1 and P2) with windows 98 and DOS games that don t work. He decided to find a way to use the computers with open source software. He found that he can run Gcompris, an educational software suite comprising of numerous activities for children aged 2 to 10, on the old computers. Now he just needs to update the Hebrew translation (last updated at 2005). Along side the translation he fixed bugs, recorded new sounds and replaced some of the graphics (all with the help of a few friends). To wrap it all, he decided to create a liveCD for the program, and this was the beginning of KG-Live. The changes were sent upstream and should be included in the next release. Today, the project waits for approval from the Israeli Ministry of Education, before distributing the software officially to the kindergartens. Posted in Free software applications, i18n & l10n, Israeli Community, Proud to use free software Tagged: education, free software, gcompris

Next.

Previous.