I came across the following from the ext3-users mailing list. The poor user was stuck on a never-updated RHEL 3 production server and running into kernel panic problems. He was advised to try updating to the latest kernel rpm from Red Hat, but he didn t feel he could do that. In his words:

I m caught between a rock and a hard place due to the EMC PowerPath binary only kernel crack. Which makes it painful to both me and my customers to regularly upgrade the kernel. Not to mention the EMC supportability matrix of doom.

That pretty much sums it all up right there. The good news is that I ve been told that dm-multipath is almost at the point where it has enough functionality to replace PowerPath. Of course, that version isn t yet shipping in distributions, and I m sure it needs more testing, but it ll be good when enterprise users who need this functionality can move to a 100% fully open source storage stack. About the only thing left to do is to work in a mention of the Frying Pan of Doom and the recipe for Quick After-Battle Triple Chocolate Cake into the mix. Related posts (automatically generated):

1. Tip o the hat, wag o the finger Linux power savings for laptop users It s interesting to see how far, and yet how much...
2. How active are your local Linux User s Groups? At the Linux Foundation, I recently had been brainstorming with...

Michael Anissimov writes about the theft of computers from the Los Alamos nuclear weapons lab [1]. He suggests that this incident (and others like it) pose a great risk to out civilisation. He advocates donating towards The Lifeboat Foundation [2] to try and mitigate risks to humanity. They suggest pledging $1000 per year for 25 years. It s interesting to note that people in Pakistan pay $8 per month for net access that better by most objective metrics than that which most people in first world can get [3]. It seems that we need to remove the cartel for the local loop to get good net access, either deregulate it entirely or make it owned by the local government who are more directly responsive to the residents. Bruce Schneier wrote a post about a proposed US law to force all mobile phones with cameras to make a click sound when taking a picture [4]. The law is largely irrelevant, as it s been law in Japan for a while most phones are already designed in that way. One interesting comment from MarkH was: But if congress REALLY wishes to benefit the public, I suggest that all guns in the U.S. be required, before each discharge, to make loud sounds (with appropriate time sequencing) simulating the flintlock technology that was common at the beginning of U.S. history, including cocking, use of the ramrod, etc. This would give fair warning of an impending discharge, and would limit firing rates to a few per minute. ROFL Brief review of a Google Android phone vs an iPhone [5]. The Android G1 is now on sale in Australia! [6]. LWN has an article about the panel discussion at the LCA Security Mini-conf [7]. Jonathan Corbet has quoted me quite a bit in the article, thanks Jonathan! Peter Ward gave an interesting TED talk about Hydrogen Sulphide and mass extinctions [8]. The best available evidence is that one of the worst extinctions was caused by H2S in the atmosphere which was produced by bacteria. The bacteria in question like a large amount of CO2 in the atmosphere. It s yet another reason for reducing the CO2 production. Michael Anissimov has written a good article summarising some of the dangers of space exploration [9], he suggests colonising the sea, deserts, and Antartica first (all of which are much easier and safer). Until we gain the ability to create huge (miles wide or larger) air bubbles in space enclosed by rapidly self-healing transparent membranes, it will be cramped and overwhelmingly boring. You ll spend even more time on the Internet up there than down here, and your connection will be slow . A confined space and slow net access, that s like being on a plane.

• [1] http://www.acceleratingfuture.com/michael/blog/2009/02/80-missing-computers-at-nuke-lab-watchdog/


• [2] http://lifeboat.com/


• [3] http://blogs.dialogic.com/2009/01/community-networks-the-robinhood-approach.html


• [4] http://www.schneier.com/blog/archives/2009/02/making_cameras.html


• [5] http://www.terdmonk.com/2009/02/16/yet-another-android-review


• [6] http://www.terdmonk.com/2009/02/15/android-here


• [7] http://lwn.net/Articles/315974/


• [8] http://www.ted.com/index.php/talks/peter_ward_on_mass_extinctions.html


• [9] http://www.acceleratingfuture.com/michael/blog/2009/02/space-that-boring-and-dangerous-place/

Every time something goes wrong there will be someone who tries to take advantage of the situation. The recent bushfires in Australia that have killed hundreds of people (the count is not known yet) are a good example. Pastor Nalliah of Catch the Fire Ministries [1] claims that it is due to legalising abortion. This is astoundingly wrong. In a more extreme example representatives of the Westboro Baptist Church were planning to visit Australia to launch a protest in support of the bushfires [2]. I have not yet found any news reports about whether they actually visited Australia or protested - it s most likely that they decided not to visit due to the Australian laws being very different to US laws regarding the relative importance of freedom of speech and incitement to violence. Apparently the insane Westboro Baptist Church people (who are best known for GodHatesFags.com and GodHatesAmerica.com) believe that God hates Australia and caused the fires (presumably due to Australia not persecuting homosexuals). Danny Nalliah has permanently damaged his own reputation by acting in a similar way to the Westboro Baptist Church. The reputation of Catch The Fire now depends on how quickly they get a new pastor Please note well that the vast majority of Christians have nothing in common with Westboro or Catch The Fire. I don t recall the last time I met an Australian Christian who was strongly opposed to homosexuality or abortion. Now we do have to try and investigate ways of avoiding future tragedies, and the work to do this needs to begin immediately. John Brumby (the Premier of Victoria) has announced that Victoria will get new strict building codes for fire resistant buildings [3]. There have been many anecdotes of people who claim to have been saved by attaching sprinkler systems to their homes, by building concrete bunkers to hide in while the fire passes, and using other techniques to save their home or save themselves. Some more research on the most effective ways of achieving such goals would be worthwhile, an increase in funding for the CSIRO to investigate the related issues would be a good thing. The article also has an interesting quote As the fallout from the disaster widened, the union representing the nation s 13,000 firefighters warned both the federal and state governments to take global warming seriously to prevent a repeat of last weekend s lethal firestorm . However given that traditionally Australia and the US have been the two nations most opposed to any efforts to mitigate global warming it seems unlikely that anything will change in this regard in a hurry. The attempts to link bushfires to abortion and homosexuality are offensive, but can be ignored in any remotely serious debate about politics. However there are some other groups trying to profit from the tragedy that make claims which are not as ridiculous. On the 9th of February the Australian Green party was compelled to release an official statement from Spokesperson Scott Ludlam, Sarah Hanson-Young, Rachel Siewert, Christine Milne, and Bob Brown following some political discussion about Greens policies [4]. There have been attempts to blame the Greens for the tragedy which were politically motivated, some of which came from groups that traditionally oppose the Greens for other reasons (I m not going to provide the detail - anyone who is really interested can do google searches on the people in question). On the 16th of February Bob Brown (the leader of the Green party) felt obliged to make another media release reiterating the fact that the Greens support prescribed burn-offs to limit the scope of wild fires [5], he also decried the hate mongering that has been occurring in the wake of the disaster. One of the strange memes that seems to be spread by opponents to the Greens is that the Greens are all supposedly from the city and know nothing about the country. To avoid being subject to such attack I feel obliged to note that on one of the bad fire days I visited my parents. I spent the morning with my father and some friends at a park that was not far from the fire area, my friends then returned to their home which was not far from the fire area. I then had lunch with my parents and watched the smoke through the dining room window. After that my friends didn t respond to email for a while and I was concerned that they may have lost their house or maybe suffered injury or death. I didn t know them well enough to feel it appropriate to try a dozen different ways of contacting them (I m sure that many other people were doing so), but I was rather concerned until my wife received an email from them. But I don t base my political beliefs on what I personally observe or my connections to people on the edge of the fire zone. I believe in the Green principles of Peace and Non Violence, Grassroots Democracy, Social and Economic Justice, Ecological Sustainability and the use of science and statistics to determine the best ways of achieving those goals.

• [1] http://www.livenews.com.au/articles/2009/02/11/Bushfires_caused_by_abortion_laws_Pastor_claims


• [2] http://www.news.com.au/story/0,27574,25067362-2,00.html


• [3] http://www.theage.com.au/national/never-again-brumby-20090211-84sv.html?page=-1


• [4] http://greensmps.org.au/content/time-heal-our-communities


• [5] http://greens.org.au/node/5229

Regarding the aforementioned R/Finance conference that will take place at the end of April here in Chicago, we announced earlier today that the conference website is now available. It provides information about the program, speakers and other details as well as a link to registration details. See you in Chicago in April!

Isotopp just pointed me to the blog of Ronald Bradford, a MySQL consultant, writing about a Drupal installation and some performance observations.

He observes a large table "watchdog":

Presently, volume and content is of a low volume, but expecting to ramp up. I do however find 90% of disk volume in one table called watchdog ;

Well, yes, the watchdog table logs system messages. No big surprise so far, I guess. So, turning off database logging (Administer -> Site building -> Modules -> turn off Database logging module) or reduce the maximum number of entries in the database log at Administer -> Site Configuration -> Logging and alerts -> Database Logging will reduce the size of the watchdog table.

Robert continues with some "page not found" errors. Of course these have to be fixed somewhere, either by editing the content or fixing theme related bugs. Not really a Drupal issue, I guess.

Then he mentions repeated SQL queries:

Looking at just 1 random second of SQL logging shows 1200+ SELECT statements.
355 are SELECT changed FROM node
[...]
There is plenty of information regarding monitoring the Slow Queries in MySQL, but I have also promoted that s it not the slow queries that ultimately slow a system down, but the 1000 s of repeating fast queries.

MySQL of course has the Query Cache to assist, but this is a course grade solution, and a high volume read/write environment this is meaningless.

There is a clear need for either a application level caching, or a database redesign to pull rather then poll this information, however without more in depth review of Drupal I can not make any judgment calls.

Drupal offers some kind of application level caching at Administer -> Site configuration -> Performance, so turning it on might already help, although I'm not sure whether that minimizes the SQL queries or to what degree.

Anyway, there are some more Drupal performance tips here:

1. http://drupal.org/node/97347


2. http://wimleers.com/article/improving-drupals-page-loading-performance


3. http://2bits.com/articles/drupal-performance-tuning-and-optimization-for-large-web-sites.html

Hope this helps... :-)

As if there weren't already enough reasons not to use out-of-office replies, here's a new one I haven't seen before (from a web design company, I think):

I am out of the office until Monday, 9 March. For LargeCorpA, please contact Person A at person.a@example.com. For LargeCorpB, SmallCompany1, or SmallCompany2, please contact Person B at person.b@example.com.

Well, I now know several companies who are clients of this web design company that I didn't know before. So, there's a nice corporate info gathering trick -- send e-mail to known addresses at your competitors on a regular basis (especially during holiday-prone times, like school holidays and Christmas/New Year) and see if they'll give you lists of their clients for free, like this helpful person has.

My message Intent to package a Debian control file grepper to WNPP and debian-devel is today a decade old. Apparently, the message predates the invention of the acronym ITP for Intent to Package (the first instance I can find is from May 1999). The changelog reveals that the first upload was on March 1st, 1999. There is unfortunately no record of when the package hit unstable the first time, since dinstall did not send Installed (or even the later Accepted) mails to an archived mailing list at that time. I suppose I might have it in some old private email archive, but more likely it s just gone. Similarly, the first fixed bug (#35527) predates BTS archiving and is now lost. The grep-dctrl program came out of repeated awkward grepping of the dpkg available and status files. Eventually I decided there must be a better way, and failed to find any canned solutions (I was later pointed to sgrep, which didn t look useful enough, and even later to dpkg-awk, but I had already committed to my own solution by then). I wrote a simple C program that processed these files as a sequence of records and did simple substring searches in each record. I rapidly added support for field selection, regular expressions and output field selection. By version 1.3a of March 2000 (which was released with Debian 2.2 potato ), the program was as good as it was going to get with one exception. Make disjunctive searches possible, said my TODO file those days. Conjunctive searches (that is, AND-searches) were possible even then by using more than one grep-dctrl command in a pipeline. Disjunctive (OR) was not possible. The problem was not so much that it would be hard to program (although the program s internal structure wasn t very good, to be honest, making extensive changes difficult), it was more a problem of coming up with a good command line syntax. Another thing that bothered me with the old grep-dctrl was how to implement Ben Armstrong s feature request. Again, the programming part wasn t the problem, the problem was coming up with a good, clean semantics for the feature. It was finally the appearance of ara in 2003 that got me moving again. Ara s author proudly compared eir program to grep-dctrl, claiming that my program did not do disjunctive searches while ara does. Competition being good for the soul, I took it as a challenge. In April 2003 I announced a complete rewrite of grep-dctrl, which was completed in January 2004 (the 2.0 release). The rewrite changed the way the command line was handled even though the usual Unix switch style is still used, the command line is regarded as a language with a parser (first an operator-precedence parser, then a recursive-descent one). The command line is transformed into an interpreted stack-based language which drives the actual grepping. The rewrite also generalised the internal data structures into an internal library which could be used to write other tools. The first such tool was sort-dctrl (introcuded in 2.7, June 2005), which was soon followed by tbl-dctrl (2.8, July 2005). The later appearance of join-dctrl (2.11, August 2007) finally allowed me to close Ben Armstrong s longstanding feature request mentioned above. The unpronounceable part of the names, dctrl , is an abbreviation for Debian control , which I decided to call the file format used by dpkg. Some people call it a RFC-822 format, but that really is a misnomer, since the differences between dctrl and RFC-822 outweigh the mainly superficial similarities (and the historical connection). I did consider calling my program dpkg-grep, but I didn t feel like I had the right to invade the dpkg namespace. The later rename to dctrl-tools reflects the fact that there are now several tools, grep-dctrl being just the oldest. I have several plans for the dctrl-tools suite, but my time and energy are mostly claimed by other responsibilities. The suite is currently team-maintained, but unfortunately the team is not very active. I would love it if I weren t the most active one with my busy schedule! Feel free to pop in on the dctrl-tools-devel mailing list, and to look at the Git repository and the todo list. If you decide to participate, please follow the rules.

Often I happen to come with some pretty crazy ideas about anything in life. That s me. Some of them may work, that s why I adventure to share them with the world. Working with computers, software in particular, for almost 12 years have brought interesting views regarding life. For the past 3 years I ve progressively improved the human being am I, and also the improvement process. The other day I was heading to the uni and I ve just noted that some practices of my profession can also be applied to my personal improvement. So, here I m sharing some of them. Hope you find them useful.

1. Spot your bugs and kill em. While it s important to want to improve yourself. It s more relevant to the fact of real improvement that you spot your personal bugs . This is a crucial part of it all. Once you are aware of you personal bugs you ll be able to take action on them. So, when you do, just kill them and don t let stink until you are not able to manage it at all and all your code taints and breaks.
2. Let process run on the background. Life has a limitation, which is time. Time is a scarce resource. You must be aware of it, really. While we might want to do many different things life presents to us, is not really responsible to engage on all. However, we can hack this. Here technology is our ally. There are many different ways to be able to do some things in the background while on the foreground you focus on what really matters to you at the time.
3. Outsource services that are not key for you. Delegation and distributed architecture come to mind. Professions and specialization are an evolution of humanity. Please take advantage of it. You don t need to know or do everything. Karen Sthepenson s connectivism axiom is illustrative: I store my knowledge in my friends .
4. Focus on what matters. This is somehow like the unix tradition of doing one thing and doing it well. Identify what makes you unique, special and relevant on your ecosystem. What value you can offer to others, and work on that direction. Hard work pays.
5. Release early and release often. Once you have spot bugs and managed to fix them, release a new version of yourself!. Tell the world by showing an improved version of yourself. Make this incremental and iterative, just like a software development framework. For instance I currently run a 3.1 version of myself. Just released a point version last week :)

I ve been using git this last few days and I am still working on a workflow for my projects. Unfortunately, as others have noticed, git violates POLS is so many ways, it ends up being hard to get. Creating a remote repository seems to be the first thing to bite a developer switching to git (mainly if coming from a centralized SCM). I have not decided which is the best way of doing it, but I ve been using git-daemon via inetd and a path in my remote hosting holding all my repositories for public pulling, and ssh for pushing. Here is how to create it:

local$ ssh spectra@remotehost
remotehost$ mkdir /var/git/myproject.git
remotehost$ cd /var/git/myproject.git
remotehost$ git --bare init 
remotehost$ logout
local$ cd myproject
local$ git remote add remotehost spectra@remotehost:/var/git/myproject.git
local$ git push remotehost master
...
otherlocal$ git clone git://remotelocal/myproject.git
otherlocal$ cd myproject
otherlocal$ git remote add remotehost spectra@remotehost:/var/git/myproject.git
(hack hack hack)
otherlocal$ git push remotehost master

Now everybody can pull your repository at git://remotehost/myproject.git and you can push and pull to it via ssh. Note that you have to setup git-daemon, which is pretty straight forward. I am using it as an inetd daemon, but you can use it as a standalone one. Debian has a package which does just that. Now, some people think logging in a remote server just to create an empty repository is too much. Well repositories are just .git directories. It happens that you can push for the first time by rsyncing your .git with a remote host:

local$ cd myproject
local$ rsync -a .git/ spectra@remotehost:/var/git/myproject.git
local$ git remote add remotehost spectra@remotehost:/var/git/myproject.git
(hack hack hack)
local$ git push remotehost master

(Apparently you can push using rsync every time, but it s regarded as wiser to keep your probably crappy local repository commits separated from the public repository otherwise commit messages like Please, don t use this code are likely to pop up everywhere :) ). Now, I don t know if this have side effects, but it works :-) Another thing to notice is that git is more directed at pulling than pushing. This may be because of its designer: the way Linus works is by pulling changes from others repositories and not by letting others push em into his one. And this is another violation of POLS for most of the people, who is used to commit their changes into some remote repository. Rather than that, people using git would expose their own repositories in order to have it pulled by others. I also agree with most of the git critics wrt git commands There s a lot of examples and I am not going deeper in this but I think it was a bad choice calling checkout what git does when told to checkout , for instance. Yes, I know different tools, different ways of seeing it but everyone was already used to what centralized SCMs call their operations, and I think it would only help git adopting same names for the same operations, and inventing new ones for those proper of decentralized operations. Anyway, once you get it (and I have not completely got it yet), it seems all flow as expected. If this adaptation fails, there s still Easy Git to the rescue! One last thing that I think contribute to the git sucks effect: git-svn. This is a great tool, but it was built from git s point of view Given it s intended as a glue for Subversion newcomers, it would benefit more from being built from svn s point of view. This was mentioned by a colleague developer in my company, when he just couldn t understand why one have to git svn rebase instead of a simple update. So git-svn also suffers of the bad command naming habit git do. Of course, that given that you came from this environment (I am sure git and git-svn makes perfect sense for Linus & cia :-) ). I have not tried yet, but yap seems to be targeted on providing an alternative to git-svn.

I recently purchased a new toy, an Intel X25-M SSD, and when I was setting it up initially, I decided I wanted to make sure the file system was aligned on an erase block boundary. This is a generally considered to be a Very Good Thing to do for most SSD s available today, although there s some question about how important this really is for Intel SSD s more on that in a moment. It turns out this is much more difficult than you might first think most of Linux s storage stack is not set up well to worry about alignment of partitions and logical volumes. This is surprising, because it s useful for many things other than just SSD s. This kind of alignment is important if you are using any kind of hardware or software RAID, for example, especially RAID 5, because if writes are done on stripe boundaries, it can avoid a read-modify-write overhead. In addition, the hard drive industry is planning on moving to 4096 byte sectors instead of the way-too-small 512 byte sectors at some point in the future. Linux s default partition geometry of 255 heads and 63 sectors/track means that there are 16065 (512 byte) sectors per cylinder. The initial round of 4k sector disks will emulate 512 byte disks, but if the partitions are not 4k aligned, then the disk will end up doing a read/modify/write on two internal 4k sectors for each singleton 4k file system write, and that would be unfortunate. Vista has already started working around this problem, since it uses a default partitioning geometry of 240 heads and 63 sectors/track. This results in a cylinder boundary which is divisible by 8, and so the partitions (with the exception of the first, which is still misaligned unless you play some additional tricks) are 4k aligned. So this is one place where Vista is ahead of Linux . unfortunately the default 255 heads and 63 sectors is hard coded in many places in the kernel, in the SCSI stack, and in various partitioning programs; so fixing this will require changes in many places. However, with SSD s (remember SSD s? This is a blog post about SSD s ) you need to align partitions on at least 128k boundaries for maximum efficiency. The best way to do this that I ve found is to use 224 (32*7) heads and 56 (8*7) sectors/track. This results in 12544 (or 256*49) sectors/cylinder, so that each cylinder is 49*128k. You can do this by doing starting fdisk with the following options when first partitioning the SSD: # fdisk -H 224 -S 56 /dev/sdb The first partition will only be aligned on a 4k boundary, since in order to be compatible with MS-DOS, the first partition starts on track 1 instead of track 0, but I didn t worry too much about that since I tend to use the first partition for /boot, which tends not to get modified much. You can go into expert mode with fdisk and force the partition to begin on an 128k alignment, but many Linux partition tools will complain about potential compatibility problems (which are obsolete warnings, since the systems that would have booting systems with these issues haven t been made in about ten years), but I didn t needed to do that, so I didn t worry about it. So I created a 1 gigabyte /boot partition as /dev/sdb1, and allocated the rest of the SSD for use by LVM as /dev/sdb2. And that s where I ran into my next problem. LVM likes to allocate 192k for its header information, and 192k is not a multiple of 128k. So if you are creating file systems as logical volumes, and you want those volume to be properly aligned you have to tell LVM that it should reserve slightly more space for its meta-data, so that the physical extents that it allocates for its logical volumes are properly aligned. Unfortunately, the way this is done is slightly baroque: # pvcreate metadatasize 250k /dev/sdb2
Physical volume /dev/sdb2 successfully created Why 250k and not 256k? I can t tell you sometimes the LVM tools aren t terribly intuitive. However, you can test to make sure that physical extents start at the proper offset by using: # pvs /dev/sdb2 -o+pe_start
PV VG Fmt Attr PSize PFree 1st PE
/dev/sdb2 lvm2 73.52G 73.52G 256.00K If you use a metadata size of 256k, the first PE will be at 320k instead of 256k. There really ought to be an pe-align option to pvcreate, which would be far more user-friendly, but, we have to work with the tools that we have. Maybe in the next version of the LVM support tools . Once you do this, we re almost done. The last thing to do is to create the file system. As it turns out, if you are using ext4, there is a way to tell the file system that it should try to align files so they match up with the RAID stripe width. (These techniques can be used for RAID disks as well). If your SSD has an 128k erase block size, and you are creating the file system with the default 4k block size, you just have to specify a strip width when you create the file system, like so: # mke2fs -t ext4 -E stripe-width=32,resize=500G /dev/ssd/root (The resize=500G limits the number of blocks reserved for resizing this file system so that the guaranteed number size that the file system can be grown via online resize is 500G. The default is 1000 times the initial file system size, which is often far too big to be reasonable. Realistically, the file system I am creating is going to be used for a desktop device, and I don t foresee needing to resize it beyond 500G, so this saves about a 50 megabytes or so. Not a huge deal, but waste not, want not , as the saying goes.) With e2fsprogs 1.41.4, the journal will be 128k aligned, as will the start of the file system, and with the stripe-width specified, the ext4 allocator will try to align block writes to the stripe width where that makes sense. So this is as good as it gets without kernel changes to make the block and inode allocators more SSD aware, something which I hope to have a chance to look at.

All of this being said, it s time to revisit this question is all of this needed for a smart , better by design next-generation SSD such as Intel s? Aligning your file system on an erase block boundary is critical on first generation SSD s, but the Intel X25-M is supposed to have smarter algorithms that allow it to reduce the effect of write-amplification. The details are a little bit vague, but presumably there is a mapping table which maps sectors (at some internal sector size we don t know for sure whether it s 512 bytes or some larger size) to individual erase blocks. If the file system sends a series of 4k writes for file system blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, 99, followed by a barrier operation, a traditional SSD might do read/modify/write on four 128k erase blocks one covering the blocks 0-31, another for the blocks 32-63, and so on. However, the Intel SSD will simply write a single 128k block that indicates where the latest versions of blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, and 99 can be found. This technique tends to work very well. However, over time, the table will get terribly fragmented, and depending on whether the internal block sector size is 512 or 4k (or something in between), there could be a situation where all but one or two of the internal sectors on the disks have been mapped away to other erase blocks, leading to fragmentation of the erase blocks. This is not just a theoretical problem; there are reports from the field that this happens relatively easy. For example, see Allyn Malventano s Long-term performance analysis of Intel Mainstream SSDs and Marc Prieur s report from BeHardware.com which includes an official response from Intel regarding this phenomenon. Laurent Gilson posted on the Linux-Thinkpad mailing list that when he tried using the X25-M to store commit journals for a database, that after writing 170% of the capacity of an Intel SSD, the small writes caused the write performance to go through the floor. More troubling, Allyn Malventano indicated that if the drive is abused for too long with a mixture of small and large writes, it can get into a state where the performance degredation is permanent, and even a series of large writes apparently does not restore the drive s function only an ATA SECURITY ERASE command to completely reset the mapping table seems to help. So, what can be done to prevent this? Allyn s review speculates that aligning writes to erase write boundaries can help I m not 100% sure this is true, but without detailed knowledge of what is going on under the covers in Intel s SSD, we won t know for sure. It certainly can t hurt, though, and there is a distinct possibility that the internal sector size is larger than 512 bytes, which means the default partitioning scheme of 255 heads/63 sectors is probably not a good idea. (Even Vista has moved to a 240/63 scheme, which gives you 8k alignment of partitions; I prefer 224/56 partitioning, since the days when BIOS s used C/H/S I/O are long gone.) The Ext3 and Ext4 file system tend to defer meta-data writes by pinning them until a transaction commit; this definitely helps, and ext4 allows you to configure an erase block boundary, which should also be helpful. Enabling laptop mode will discourage writing to the disk except in large blocks, which probably helps significantly as well. And avoiding fsync() in applications will also be helpful, since a cache flush operation will force the SSD to write to an erase block even if it isn t completely filled. Beyond that, clearly some experimentation will be needed. My current thinking is to use a standard file system aging workload, and then performing an I/O benchmark to see if there has been any performance degradation. I can then vary various file system tuning parameters and algorithms, confirm whether or not a heavy fsync workload makes the performance worse. In the long term, hopefully Intel will release a firmware update which adds support for ATA TRIM/DISCARD commands, which will allow the file system to inform the SSD that various blocks have been deleted and no longer need to be preserved by the SSD. I suspect this will be a big help, if the SSD knows that certain sectors no longer need to be preserved, it can avoid copying them when trying to defragment the SSD. Given how expensive the X25-M SSD s are, I hope that there will be a firmware update to support this, and that Intel won t leave its early adopters high and dry by only offering this functionality in newer models of the SSD. If they were to do that, it would leave many of these early adopters, especially your humble writer (who paid for his SSD out of his own pocket), to be quite grumpy indeed. Hopefully, though, it won t come to that. Update: I ve since penned a follow-up post Should Filesystems Be Optimized for SSD s? Related posts (automatically generated):

1. Should Filesystems Be Optimized for SSD s? In one of the comments to my last blog entry,...
2. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...

I recently purchased a new toy, an Intel X25-M SSD, and when I was setting it up initially, I decided I wanted to make sure the file system was aligned on an erase block boundary. This is a generally considered to be a Very Good Thing to do for most SSD s available today, although there s some question about how important this really is for Intel SSD s more on that in a moment. It turns out this is much more difficult than you might first think most of Linux s storage stack is not set up well to worry about alignment of partitions and logical volumes. This is surprising, because it s useful for many things other than just SSD s. This kind of alignment is important if you are using any kind of hardware or software RAID, for example, especially RAID 5, because if writes are done on stripe boundaries, it can avoid a read-modify-write overhead. In addition, the hard drive industry is planning on moving to 4096 byte sectors instead of the way-too-small 512 byte sectors at some point in the future. Linux s default partition geometry of 255 heads and 63 sectors/track means that there are 16065 (512 byte) sectors per cylinder. The initial round of 4k sector disks will emulate 512 byte disks, but if the partitions are not 4k aligned, then the disk will end up doing a read/modify/write on two internal 4k sectors for each singleton 4k file system write, and that would be unfortunate. Vista has already started working around this problem, since it uses a default partitioning geometry of 240 heads and 63 sectors/track. This results in a cylinder boundary which is divisible by 8, and so the partitions (with the exception of the first, which is still misaligned unless you play some additional tricks) are 4k aligned. So this is one place where Vista is ahead of Linux . unfortunately the default 255 heads and 63 sectors is hard coded in many places in the kernel, in the SCSI stack, and in various partitioning programs; so fixing this will require changes in many places. However, with SSD s (remember SSD s? This is a blog post about SSD s ) you need to align partitions on at least 128k boundaries for maximum efficiency. The best way to do this that I ve found is to use 224 (32*7) heads and 56 (8*7) sectors/track. This results in 12544 (or 256*49) sectors/cylinder, so that each cylinder is 49*128k. You can do this by doing starting fdisk with the following options when first partitioning the SSD: # fdisk -H 224 -S 56 /dev/sdb The first partition will only be aligned on a 4k boundary, since in order to be compatible with MS-DOS, the first partition starts on track 1 instead of track 0, but I didn t worry too much about that since I tend to use the first partition for /boot, which tends not to get modified much. You can go into expert mode with fdisk and force the partition to begin on an 128k alignment, but many Linux partition tools will complain about potential compatibility problems (which are obsolete warnings, since the systems that would have booting systems with these issues haven t been made in about ten years), but I didn t needed to do that, so I didn t worry about it. So I created a 1 gigabyte /boot partition as /dev/sdb1, and allocated the rest of the SSD for use by LVM as /dev/sdb2. And that s where I ran into my next problem. LVM likes to allocate 192k for its header information, and 192k is not a multiple of 128k. So if you are creating file systems as logical volumes, and you want those volume to be properly aligned you have to tell LVM that it should reserve slightly more space for its meta-data, so that the physical extents that it allocates for its logical volumes are properly aligned. Unfortunately, the way this is done is slightly baroque: # pvcreate metadatasize 250k /dev/sdb2
Physical volume /dev/sdb2 successfully created Why 250k and not 256k? I can t tell you sometimes the LVM tools aren t terribly intuitive. However, you can test to make sure that physical extents start at the proper offset by using: # pvs /dev/sdb2 -o+pe_start
PV VG Fmt Attr PSize PFree 1st PE
/dev/sdb2 lvm2 73.52G 73.52G 256.00K If you use a metadata size of 256k, the first PE will be at 320k instead of 256k. There really ought to be an pe-align option to pvcreate, which would be far more user-friendly, but, we have to work with the tools that we have. Maybe in the next version of the LVM support tools . Once you do this, we re almost done. The last thing to do is to create the file system. As it turns out, if you are using ext4, there is a way to tell the file system that it should try to align files so they match up with the RAID stripe width. (These techniques can be used for RAID disks as well). If your SSD has an 128k erase block size, and you are creating the file system with the default 4k block size, you just have to specify a strip width when you create the file system, like so: # mke2fs -t ext4 -E stripe-width=32,resize=500G /dev/ssd/root (The resize=500G limits the number of blocks reserved for resizing this file system so that the guaranteed number size that the file system can be grown via online resize is 500G. The default is 1000 times the initial file system size, which is often far too big to be reasonable. Realistically, the file system I am creating is going to be used for a desktop device, and I don t foresee needing to resize it beyond 500G, so this saves about a 50 megabytes or so. Not a huge deal, but waste not, want not , as the saying goes.) With e2fsprogs 1.41.4, the journal will be 128k aligned, as will the start of the file system, and with the stripe-width specified, the ext4 allocator will try to align block writes to the stripe width where that makes sense. So this is as good as it gets without kernel changes to make the block and inode allocators more SSD aware, something which I hope to have a chance to look at.

All of this being said, it s time to revisit this question is all of this needed for a smart , better by design next-generation SSD such as Intel s? Aligning your file system on an erase block boundary is critical on first generation SSD s, but the Intel X25-M is supposed to have smarter algorithms that allow it to reduce the effect of write-amplification. The details are a little bit vague, but presumably there is a mapping table which maps sectors (at some internal sector size we don t know for sure whether it s 512 bytes or some larger size) to individual erase blocks. If the file system sends a series of 4k writes for file system blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, 99, followed by a barrier operation, a traditional SSD might do read/modify/write on four 128k erase blocks one covering the blocks 0-31, another for the blocks 32-63, and so on. However, the Intel SSD will simply write a single 128k block that indicates where the latest versions of blocks 10, 12, 13, 32, 33, 34, 35, 64, 65, 66, 67, 96, 97, 98, and 99 can be found. This technique tends to work very well. However, over time, the table will get terribly fragmented, and depending on whether the internal block sector size is 512 or 4k (or something in between), there could be a situation where all but one or two of the internal sectors on the disks have been mapped away to other erase blocks, leading to fragmentation of the erase blocks. This is not just a theoretical problem; there are reports from the field that this happens relatively easy. For example, see Allyn Malventano s Long-term performance analysis of Intel Mainstream SSDs and Marc Prieur s report from BeHardware.com which includes an official response from Intel regarding this phenomenon. Laurent Gilson posted on the Linux-Thinkpad mailing list that when he tried using the X25-M to store commit journals for a database, that after writing 170% of the capacity of an Intel SSD, the small writes caused the write performance to go through the floor. More troubling, Allyn Malventano indicated that if the drive is abused for too long with a mixture of small and large writes, it can get into a state where the performance degredation is permanent, and even a series of large writes apparently does not restore the drive s function only an ATA SECURITY ERASE command to completely reset the mapping table seems to help. So, what can be done to prevent this? Allyn s review speculates that aligning writes to erase write boundaries can help I m not 100% sure this is true, but without detailed knowledge of what is going on under the covers in Intel s SSD, we won t know for sure. It certainly can t hurt, though, and there is a distinct possibility that the internal sector size is larger than 512 bytes, which means the default partitioning scheme of 255 heads/63 sectors is probably not a good idea. (Even Vista has moved to a 240/63 scheme, which gives you 8k alignment of partitions; I prefer 224/56 partitioning, since the days when BIOS s used C/H/S I/O are long gone.) The Ext3 and Ext4 file system tend to defer meta-data writes by pinning them until a transaction commit; this definitely helps, and ext4 allows you to configure an erase block boundary, which should also be helpful. Enabling laptop mode will discourage writing to the disk except in large blocks, which probably helps significantly as well. And avoiding fsync() in applications will also be helpful, since a cache flush operation will force the SSD to write to an erase block even if it isn t completely filled. Beyond that, clearly some experimentation will be needed. My current thinking is to use a standard file system aging workload, and then performing an I/O benchmark to see if there has been any performance degradation. I can then vary various file system tuning parameters and algorithms, confirm whether or not a heavy fsync workload makes the performance worse. In the long term, hopefully Intel will release a firmware update which adds support for ATA TRIM/DISCARD commands, which will allow the file system to inform the SSD that various blocks have been deleted and no longer need to be preserved by the SSD. I suspect this will be a big help, if the SSD knows that certain sectors no longer need to be preserved, it can avoid copying them when trying to defragment the SSD. Given how expensive the X25-M SSD s are, I hope that there will be a firmware update to support this, and that Intel won t leave its early adopters high and dry by only offering this functionality in newer models of the SSD. If they were to do that, it would leave many of these early adopters, especially your humble writer (who paid for his SSD out of his own pocket), to be quite grumpy indeed. Hopefully, though, it won t come to that. Update: I ve since penned a follow-up post Should Filesystems Be Optimized for SSD s? Related posts (automatically generated):

1. Should Filesystems Be Optimized for SSD s? In one of the comments to my last blog entry,...
2. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...

From the description of the memisc package for R:

One of the aims of this package is to make life easier for useRs who deal with survey data sets. It provides an infrastructure for the management of survey data including value labels, definable missing values, recoding of variables, production of code books, and import of (subsets of) SPSS and Stata files. Further, it provides functionality to produce tables and data frames of arbitrary descriptive statistics and (almost) publication-ready tables of regression model estimates. Also some convenience tools for graphics, programming, and simulation are provided. [emphasis added]

How did I miss this package before? It makes analyzing NES data heck, any data with value labels and missing values in R an almost sane thing to do.

Ok, my last version was a pure Bash solution: working, but slow. There were some comments how to improve the performance and I decided finally to reimplement the second script as Python script.
The Bash script didn't finish within a day. The Python script ends after 1-2 hours in my test scenario. So, here are the scripts again:

make-lsLR.sh - call this regularly (cron) to create the needed files that are stored in /root/. Of course you can alter the location easily and exclude other directories from being scanned. [download make-lsLR.sh]


check_lost+found.py - The second script is to be run when your fsck managed to mess up with your files and stored them into lost+found directory. It takes 3 arguments: 1) the source directory where your messed up lost+found directory is, 2) the target directory to which the data will be saved and 3) a switch to actually make it happen instead of a dry-run. [download check_lostfound.py.sh]


I've chosen to copy the files to a different place instead of moving them within the same filesystem to their original place for safety reasons. Primary goal is to retrieve the files from lost+found, not to replace a full featured backup and restore application. Because of this the script doesn't handle hard- nor symlinks correctly. It just copy files.

Of course there's still room for improvements, like handling hard-/symlinks correctly or using inode number instead of md5sums to move data back to its prior location. But it works for me[tm] well enough in this way, so I'm satisfied so far. You're welcome, though, to improve this piece of ugliness if you like.

Maybe someone else finds this usefull as well. Use it on your own risk, of course. :)

Red Hat has just announced a deal with MS for support of RHEL virtual machines on Windows Server and Windows virtual machines on RHEL [1]. It seems that this deal won t deliver anything before calendar H2 2009 so nothing will immediately happen - but the amount of testing to get these things working correctly is significant. Red Hat has stated that the agreements contain no patent or open source licensing components and the agreements contain no financial clauses, other than industry-standard certification/validation testing fees so it seems that there is nothing controversial in this. Of course that hasn t stopped some people from getting worked up about it. I think that this deal is a good thing. I have some clients who run CentOS and RHEL servers (that I installed and manage) as well as some Windows servers. Some of these clients have made decisions about the Windows servers that concern me (such as not using ECC RAM, RAID, or backups). It seems to me that if I was to use slightly more powerful hardware for the Linux servers I could run Windows virtual machines for those clients, manage all the backups at the block device level (without bothering the Windows sysadmins). This also has the potential to save the client some costs in terms of purchasing hardware and managing it. When this deal with MS produces some results (maybe in 6 months time) I will recommend that some of my clients convert CentOS machines to RHEL to take advantage of it. If my clients take my advice in this regard then it will result in a small increase in revenue and market share for RHEL. So Red Hat s action in this regard seems to be a good business decision for them. If my clients take my advice and allow me to use virtualisation to better protect their critical data that is on Windows servers then it will be a significant benefit for the users.

• [1] http://www.redhat.com/promo/svvp/?intcmp=70160000000HiHHAA0

I ve been doing some code lately, fixing things here and there, and over the last few days gave mt-daapd some attention. First, I was reported a segfault of mt-daapd when reloading web pages too fast. Surprisingly, it turned out to be a stupid omission in a simple routine and it is hard to hit without disabling the cache in the browser. Now, I just fixed something that s been bugging me for a while: mt-daapd did not handle Avahi daemon restarts, leading to mt-daapd becoming invisible to clients relying on mDNS until you restarted it. And, wow, that went a bit further than expected. As I expected, the simple piece of code needed to handle that wasn t there. But still it did not work, as it turns out the event loop for the Avahi polling wasn t being run. Replace stupid code reinventing the wheel by the Avahi-provided wheel and there, it works. But not when run as a daemon (aka foreground only). Well yes, starting the thread before daemonizing isn t going to work well, so fix that. EPIC WIN \o/ As mt-daapd s upstream is not active anymore I m accumulating fixes for mt-daapd in the Debian package, to the point I m starting to consider setting up a git repository somewhere with those fixes and possibly some more code rework if I feel like it.

If you re reading this post via the website rather than a feed/planet then you will notice that the site has gone completely black in support of the Creative Freedom Foundation s campaign against S92A of the NZ Copyright Amendment Act which is due to come into effect on 28th February 2009. I ve also joined the wave of people blacking out their avatar on Facebook/Jabber/MSN, etc. S92A introduces Guilt Upon Accusation whereby if you are accused of copyright infringement (downloading music and movies, etc) repeatedly (likely 3 or more times) you are at risk of being disconnected from the Internet by your ISP. The law does not require any proof or substantiation of the accusations and the entire process would occur outside of the courts and the established legal system. Not only does it place every user at risk, the wording is very unclear on exactly what type of organisation is considered an ISP and there is significant concern that schools, businesses, libraries and hospitals will be placed in the difficult position of determining whether their users have broken the law and require disconnection. Opposition to the law is not an attack on copyright, or a statement that we should be free to download all the movies and music that we desire. Those sorts of activities are clearly wrong, and I don t have any issue with copyright holders wanting to enforce their rights when their content is illegally copied. However, disconnecting people upon accusation, with no proof or formal legal process to prove guilt is not the right way to go about it. The fact that the law does not require proof of guilt is only the tip of the iceberg in terms of problems with it. For further background on the problems it causes for ISPs by placing them as the middle-man in copyright disputes you should refer to the following posts:

• Russel Brown of Public Address, points out that among other faults the law stipulates a penalty that no court would impose in adjudicating a copyright complaint
• David Farrar of Kiwiblog, has a guest post at the NZ Center for Political Research explaining how the law affects businesses
• Creative Freedom Foundation Background on S92

Finally, I think it is worth pointing out that S92A was removed from the proposed Amendment at the select committee stage, but was later reintroduced by Judith Tizard during the final reading of the bill. Mark Harris has an excellent post on the history of the amendment which includes facts such as the official report on the amendment also recommended removing S92A as it was unecessary given existing ISP terms and conditions which forbid illegal activity. The fact that the select committee (based on public submissions) recognised the problems with S92A and removed it, only to have it added back in again at the last stage when we no longer had any say on it really hacks me off and I cant help but feel the influence of the big money American media companies pressuring our politicians to pass a law that they don t really understand the full consequences of. So what is to be done? The Blacked Out campaign, being run by the Creative Freedom Foundation is gathering steam and international attention. Peter Dunne of United Future (who originally voted for the amendment) has declared that the amendment is wrong, and doesn t do what they thought they were voting to do, we need to convince National and the rest of the house of the same. Time is running out for this to happen before the amendment comes into effect on Feb 28th, but there is still time to write to your local MP and sign the petition against S92A Guilt Upon Accusation . The Creative Freedom Foundation site has a nice easy list of what you can do to register your protest.

Neil McGovern noticed me using drivel for my blog entries at the recent Debian UK BSP for Lenny and as he isn't using it anymore, suggested that I might want to take over maintenance. It's always better to have a package maintained by a regular user of the program.

So, this is the blog post prior to the upload of the blog client, drivel.

I have a feeling that I may be taking on another upstream project into the bargain - I'm interested but not sure how much time it should consume. Upstream is not exactly active but that isn't a huge problem for me. There is a new unreleased version that needs a few fixes but the hardest part is the external bells and whistles, not the core of the program.

The initial upload fixes all the lintian stuff in the current package but drivel is a bit different to other packages, fixing bugs in drivel means having a way to test drivel with a variety of blogs, not just my own and that means setting up admin access to a variety of blog engines. Not sure I really want to be setting aside that much time.

Once the new version is in Squeeze, I'd welcome any ideas about extra functionality, provided all such ideas come with either patches or a willingness to test the upstream code against your own blog.

After dropping all (pretense at) regular exercise after leaving high/grammar/secondary school (the german word is Gymnasium) and doing more or less nothing for more than 10 years, I've taken up weight training (after a not very successful attempt at volleyball; I like the sport but the team fell apart) two years ago. Since I actually managed to work out regularly (twice a week) in the first year, and since membership in a gym is not cheap, and since I moved and thus had to find a different solution anyway, I decided to buy a Bowflex Sport home gym. I can't compare it to other home equipment, and it apparently it has been discontinued by Nautilus, but nonetheless, a few thoughts: The conclusion first: Would I buy it again? No. Was it a waste of money? No, it's not that bad. In detail:

• The free-running seat that should allow cardio rowing is not very useful because there's no good foot-rest, and because the range of motion is too limited. This is my main grievance because being able to do cardio was one of the main reasons for chosing this engine over other offers.
• Go for the 310lbs extension at least (I did, luckily.) With leg exercises, 210lbs are just not enough (I'm 70kg/180cm and have been doing weight training two years. Certainly no muscle freak...)
• It's slightly too small. For many exercises, I'd wish for a bigger range of motion. Especially the lat tower could be 10cm higher.
• While the famous power rods haven't shown signs of wearing out after this first year, the resistance is not consistent. The 310lbs is split in rods of 5, 10, 10, 30, 50, 50lbs per side, and I've noticed that 30 + 10 + 10lbs is noticeably lighter than 50lbs. (I haven't measured at which point of the range of motion the nominal resistance applies. Resistance goes from very light at the start of the motion to hard towards the end, so the numbers don't really compare to training with real weights anyway. I haven't had any problem with this, though.)
• The documentation is lacking. Besides quality issues (wrong images, spelling), some of the stuff delivered in the box as well as some of the possible exercises are simply not used/described. (Example: it is possible to do squats, but the manual doesn't describe them.) So don't get this machine if you don't have an idea about what kinds of exercises you want to do.
• Assembling it is quite a project. But the documentation is good enough on the assembly if you have ever managed to assemble a Kinder Surprise toy, and you only do it once anyway, so it's not really an issue.

Closing remarks: I'll probably keep it for quite a while. I've thought about spending much less money and just getting a bench and a few weights, but I really like to have a lat tower. Overall I can do what I want to do with the exception of cardio workouts. The other angle is that there is quite a bit of bad press around about the founder of Nautilus, Arhur Jones. I can't comment on how trustworthy these articles are, and I can't comment on Nautilus' business practices today (Mr. Jones seems to be dead), so take this with a large pinch of salt before making a buying decision based on moral/ethical standards.

Yay! This blog post is not strictly speaking news anymore - But for those who don't know it yet, three days ago Debian 5.0 Lenny was released, after 22 months of work (plus fun, plus flamewars, plus everything that makes a Free Software lover tick). And, of course, that makes us all very happy and proud. As always, upgrading is a breeze. Hats off specially to Wolfgang and everybody who worked towards the great release notes - No, it is not a simple task, by far. And I _do_ feel ashamed I didn't even beep that way :-( So, many of us are in the middle of planning/executing our servers' migrations. So far, I'm amazed for good. Even hairy issues such as firmware removal are magically and beautifully taken care of - i.e. my firewall kindly informed me during update that I would need to install the non-free firmware for my BNX2 (Broadcom) network interfaces, and I was just a package away from absolute happiness. Lets see what happens next Friday, as I will be upgrading our storage+application server (which is _way_ more complex than the servers I've dealt with so far). Anyway... But what good is a blog post if you are not ranting? Many people recognize me as one of the most Debian-connected people in Mexico, and that's very good. And yes, besides being a Debian Developer, I am a co-sysadmin for the main Debian mirror in Mexico. Some people have already asked me for CD-ROMs and DVDs. Of course, if I had a BluRay drive, I'm sure I'd also get requests for it. People: Do you really want such media? Think again... Do you really want 31 CDs or five DVDs for your favorite architecture? Ok, maybe many will say "nah, just give me the first one" - Then, do you want to limit your Debian experience to just the software that lives on the first 1/31th (or 1/5th) of it all? Of course there are many situations where it is desirable. Low bandwidth users, or people with no regular connectivity, will be much better served by suitable media from which to install. However, most of us (computer geeks living in Mexico City - Yes, that's the people contacting me) have at least a 1Mbps connection at home. People, just get the Netinst or Businesscard (180 or 40MB) images and download whatever is left via the network. Debian is extremely network-friendly. And, believe me, even if packages are sorted by popularity (and that's why most people will be happy with the first DVD if needed), you never know if you will want precisely a package that sits towards the lonely tail of popularity. And, yes, I did have my CD images handy for the Potato, Woody and Sarge cycles - but increasingly, it became easier for me just to ask apt-get to fetch stuff from the web (which is done without me moving from my comfy chair while I do anything else) than asking apt-get to ask me to go search for the f.*ing CD which I left dont-remember-where. Anyway... I'm not saying I won't burn your CDs - If you want them, please tell me in advance and come to my office, I'll be glad to give you some Debian disks. But spare the environment. We don't need to burn more and more disks. Use the network, be a better human being!

After the release of Lenny I've upgraded all my servers to it. Earlier than I planned. The upgrade went mostly smooth, but there were some glitches:

1. somehow the roundcube webmailer was gone and uninstallable. I could fix that by building the package from Sids sources and installing it.


2. Squirrelmail was broken in that regard that you got an error message after logging in, stating something about the compatibility plugin. A Google search confimed that there are problems with the compatibility plugin. But sadly it didn't help to just deactivate the plugin in the config. Instead I needed to rename/remove the old directory from Squirrelmails plugin directory and installed the appropriate package again. Works again.


3. OpenLDAP has changed its method how replicas are being configured. But as this is just a change in configuration, it shouldn't be difficult, although I haven't found the time yet to do the changes.

I guess I'll have more "fun" in upgrade a Xen domU at work that is running Ubuntu dapper... sigh ;-)