Hey $WORLD, Philippines needs your help! We just got hit by ondoy, and there seems to be a couple more storms coming in. You can help out via donations to the Red Cross/Red Crescent. If you have friends or family in Manila, you can check the RescueHub and Sahana lists for persons missing and in distress. And lastly, pray to your $DEITY for us (or do what works best for you.) UPDATE: Google is helping out too. The local Red Cross is also accepting donations. UPDATE 2: There's a new site up with additional info on the relief efforts.

So, I'm moving back to Manila tomorrow. I hope the flood's subsided by then. UPDATE: Delaying this, after hearing some more news. I hope it gets better and not worse, as I hear there are a couple more storms coming in.

There s little that scares Kansans more than a good tornado. And, truth be told, there s little that excites us more than a good tornado, either. After all, we know it s going to be good for a story. Here on the plains, it is easy to remember how powerful nature really is. Weather sweeps in from all directions with little to hinder it, and tonight was a night few will forget. The 300-member Kansas Mennonite Men s Chorus, of which I m a member, had a concert tonight at 7, and we had a practice starting at 3. We sang in the beautiful old Memorial Hall on the campus of Bethel College. I need to describe the building a bit for you. It was built in 1938, in the day before auditorium designers decided you need to remove the outdoors. Three sides are full of windows, and it s just a few feet from the front doors into the auditorium itself. There is no air conditioning, so in a very humid day like today, every possible door and window is open. To stand on the stage and look out over the audience, you see light coming in from every direction, extremely bright light when there s lightning. It s been storming all afternoon. Our rehearsal featured thunderclaps, driving rain that was very audible inside, flickering lights. It was, in all honesty, a bit exciting. At about 6:35, word came that a tornado warning had been issued for the county. A few minutes later, about 20 minutes before start time, sirens were activated and the decision was made to evacuate to safety in the building s basement and an adjoining building. I, and a number of college students singing with the choir, wound up as a sort of impromptu ushers, directing people to the best place for safety and helping to get people into space efficiently. It was rather stuffy in those basement hallways, but it was also quite loud due to all the excited visiting going on. The KMMC concerts aren t usually like this. A few minutes later, word came that the tornado warning was canceled. I went down the steps, yelled out the news, and got a big cheer. People poured back upstairs. We got started a few minutes late, but nobody quite cared. A group of brass players was going to provide the prelude, but their first trumpet player had to leave at the last minute because his tool shed had blown over. They covered for him well in the concert though. When I got home later, Terah saw me and said, I thought they d cancel the concert because of the tornado. I just had to laugh, and pointed out, This is Kansas. After a tornado evacuation, it just got everybody more excited for the program! (Of course, had there been a strike, it would have been a different story.) And it was a great program. There was excitement in the air. And it was incredibly hot and stuffy on the stage with 300 men. I had sweat running down my face. The rain and the thunder continued, sometimes improving the program. About halfway through, we sang Prayer of the Children, written by Kurt Bestor, someone that was terribly saddened by the destruction during the civil war in the Balkans. The lyrics go, in part:

Can you hear the prayer of the children
on bended knee, in the shadow of an unknown room? Empty eyes with no more tears to cry,
turning heavenward to the light. Cryin Jesus, help me
to see the morning light of one more day. . . Can you hear the voice of the children
softly pleading for silence in their shattered world?. . . Angry guns preach a gospel full of hate
blood of the innocent on their hands. For when darkness clears, I know you re near,
brining peace again. Can you hear
the prayer of the children?

And when we get to the last phrase, there s a long rest after Can you hear . We sang it the phrase, and with perfect timing as we paused for the rest, was a loud clap of thunder. And after the last word of the song, total silence. Nobody moved for a few seconds, and then applause rang out. A couple of years ago, I was in the audience at a concert in the same building. Ominous weather lurked that day too. Before the concert started, the director said something like this, before giving information about evacuation procedures, should they be needed: We are plains people. We are privileged to see the awesome power and beauty of the weather and nature up close every day. If we should be a little too privileged today, here s what we ll do

Last night, I stayed up late reworking the avionics bay in my Giant Leap Vertical Assault kit. This is the rocket I set my personal best altitude to date of 14,141 feet above ground level with. The Missile Works RRC2 mini will move to a lesser airframe to keep it below 10k feet AGL since it's an early unit with the supposedly-buggy early firmware. In its place, I've mounted a PerfectFlite MiniAlt/WD to handle primary deployment duties... and serial number 1 of TeleMetrum! Meanwhile, Keith has completely rewritten the guts of the firmware in the last few days to eliminate FreeRTOS since it turns out to just be too heavy for our needs on this project. My board is now running a snapshot of his latest work, and in bench testing I've confirmed that it can do launch detect, log sensor data until the non-volatile memory chip is full, and dump the raw data over USB for analysis after a flight. Weather has been an issue in Colorado recently, but as soon as we get a decent launch day (maybe as early as this weekend?), I'm looking forward to flying this rocket on one or all of the Cesaroni I205, I540, and J285 currently in my box to get some initial flight data recorded. Keep your fingers crossed for me that the storm front headed towards us clears out by the weekend!

Item #1 Say you want to do something like take a quiz to tell you what Star Wars character you are, and have to result posted to your wall. It would make sense for Facebook to prompt you for authorization for the app to make a one-time post to your wall. But instead, they ask you for permission to pull your profile information, photos, your friends info, and other content that it requires to work. And we have no idea what this other content is. So yes, in order to be told that you most closely resemble Storm Trooper Extra #3, you must let the app developer have access to all your updates, your list of friends, your hometown, email address, whatever else only your close friends can see. Why can t Facebook make this a tad less invasive? Item #2 I also don t understand why people click Allow to these apps, having no idea just who is getting access to their information or what information they re getting. And Terah thinks I m being paranoid because I said that criminals probably are writing cute facebook apps ah well.

NCBA invites you to pick the Monday, April 6th edition of the New York Times to check out the front section for a very special advertorial on cooperative business. NCBA will receive a copy of the Advertorial to place on its Web Site on Monday and we will distribute a link so you can post it on yours in the next edition of the eNews. However, we hope that you ll let everyone at your cooperative know to pick up the Monday New York Times as everyone in the cooperative community should be proud to be part of this national effort to promote cooperative awareness.

These are challenging times for all businesses, but cooperatives are well-placed to weather the storm. I ll report back from our Bristol co-ops event later or tomorrow.

After reading the comments on my earlier post, Delayed allocation and the zero-length file problem as well as some of the comments on the Slashdot story as well as the Ubuntu bug, it s become very clear to me that there are a lot of myths and misplaced concerns about fsync() and how best to use it. I thought it would be appropriate to correct as many of these misunderstandings about fsync() in one comprehensive blog posting. As the Eat My Data presentation points out very clearly, the only safe way according that POSIX allows for requesting data written to a particular file descriptor be safely stored on stable storage is via the fsync() call. Linux s close(2) man page makes this point very clearly:

A successful close does not guarantee that the data has been successfully saved to disk, as the kernel defers writes. It is not common for a file system to flush the buffers when the stream is closed. If you need to be sure that the data is physically stored use fsync(2).

Why don t application programmers follow these sage words? These three reasons are most often given as excuses:

1. (Perceived) performance problems with fsync()
2. The application only needs atomicity, but not durability
3. The fsync() causing the hard drive to spin up unnecessarily in laptop_mode

Let s examine each of these excuses one at a time, to see how valid they really are. (Perceived) performance problems with fsync() Most of the bad publicity with fsync() originated with the now infamous problem with Firefox 3.0 that showed up about a year ago in May, 2008. What happened with Firefox 3.0 was that the primary user interface thread called the sqllite library each time the user clicked on a link to go to a new page. The sqllite library called fsync(), which in ext3 s data=ordered mode, caused a large, visible latency which was visible to the user if there was a large file copy happening by another process. Nearly all of the reported delays was a few seconds, which would be expected; normally there isn t that much dirty data that needs to be flushed out on a Linux system, even if it is even very busy. For example, consier the example of a laptop downloading an .iso image from a local file server; if the laptop has the exclusive link of a 100 megabit/second ethernet link, and the server has the .iso file in cache, or has a nice fast RAID array so it is not the bottleneck, then in the best case, the laptop will be able to download data at the rate of 10-12 MB/second. Assuming the default 5 second commit interval, that means that in the worst case, there will be at most 60 megabytes which must be written out before the commit can proceed. A reasonably modern 7200 rpm laptop drive can write between 60 and 70 MB/second. (The Seagate Momentus 7200.4 laptop drive is reported to be able to deliver 85-104 MB/second, but I can t find it for sale anywhere for love or money.) In this example, an fsync() will trigger a commit and might need to take a second while the download is going on; perhaps half a second if you have a really fast 7200 rpm drive, and maybe 2-3 seconds if you have a slow 5400 rpm drive. (Jump to Sidebar: What about those 30 second fsync reports?) Obviously, you can create workloads that aren t bottlenecked on the maximum ethernet download speed, or the speed of reading from a local disk drive; for example, dd if=/dev/zero of=big-zero-file will create a very large number of dirty pages that must be written to the hard drive at the next commit or fsync() call. It s important to remember though, fsync() doesn t create any extra I/O (although it may remove some optimization opportunities to avoid double writes); fsync() just pushes around when the I/O gets done, and whether it gets done synchronously or asynchronously. If you create a large number of pages that need to be flushed to disk, sooner or later it will have a significant and unfortunate effect on your system s performance. Fsync() might make things more visible, but if the fsync() is done off the main UI thread, the fact that fsync() triggers a commit won t actually disturb other processes doing normal I/O; in ext3 and ext4, we start a new transaction to take care of new file system operations while the committing transction completes. The final observation I ll make is that part of the problem is that Firefox as an application wants to make a huge number of updates to state files and was concerned about not losing that information even in the face of a crash. Every application writer should be asking themselves whether this sort of thing is really necessary. For example, doing some quick measurements using ext4, I determined that Firefox was responsible for 2.54 megabytes written to the disk for each web page visited by the user (and this doesn t include writes to the Firefox cache; I symlinked the cache directory to a tmpfs directory mounted on /tmp to reduce the write load to my SSD). So these 2.54 megabytes is just for Firefox s cookie cache and Places database to maintain its Awesome bar . Is that really worth it? If you visit 400 web pages in a day, that s 1GB of writes to your SSD, and if you write more than 20GB/day, the Intel SSD will enable its write endurance management feature which slows down the performance of the drive. In light of that, exactly how important is it to update those darned sqllite databases after every web click? What if Firefox saved a list of URL s that has been visited, and only updated every 30 or 60 minutes, instead? Is it really that every last web page that you browse be saved if the system crashes? An fsync() call every 15, 30, or 60 minutes, done by a thread which doesn t block the application s UI, would have never been noticed and would have not started the firestorm on Firefox s bugzilla #421482. Very often, after a little thinking, a small change in the application is all that s necessary for to really optimize the application s fsync() usage. (Skip over the sidebar if you ve already read it). Sidebar: What about those 30 second fsync reports? If you read through the Firefox s bugzilla entry, you ll find reports of fsync delays of 30 seconds or more. That tale has grown in the retelling, and I ve seen some hyperbolic claims of five minute delays. Where did that come from? Well, if you look that those claims, you ll find they were using a very read-heavy workload, and/or they were using the ionice command to set a real-time I/O priority. For example, something like ionice -c 1 -n 0 tar cvf /dev/null big-directory . This will cause some significant delays, first of all because ionice -c 1 causes the process to have a real-time I/O priority, such that any I/O requests issued by that process will be serviced before all others. Secondly, even without the real-time I/O priority, the I/O scheduler naturally prioritizes reads as higher priority than writes because normally processes are waiting for reads to complete, but writes are normally asynchronous. This is not at all realistic workload, and it is even more laughable that some people thought this might be an accurate representation of the I/O workload of a kernel compile. These folks had never tried the experiment, or measured how much I/O goes on during a kernel compile. If you try it, you ll find that a kernel compile sucks up a lot of CPU, and doesn t actually do that much I/O. (In fact, that s why an SSD only speeds up a kernel compile by about 20% or so, and that s in a completely cold cache case. If the commonly used include files are already in the system s page cache, the performance improvement of the SSD is much less.) Jump back to reading Performance problems with fsync. The atomicity not durability argument One argument that has commonly been made on the various comment streams is that when replacing a file by writing a new file and the renaming file.new to file , most applications don t need a guarantee that new contents of the file are committed to stable store at a certain point in time; only that either the new or the old contents of the file will be present on the disk. So the argument is essentially that the sequence:

• fd = open( foo.new , O_WRONLY);
• write(fd, buf, bufsize);
• fsync(fd);
• close(fd);
• rename( foo.new , foo );

is too expensive, since it provides atomicity and durability , when in fact all the application needed was atomicity (i.e., either the new or the old contents of foo should be present after a crash), but not durability (i.e., the application doesn t need to need the new version of foo now, but rather at some intermediate time in the future when it s convenient for the OS). This argument is flawed for two reasons. First of all, the squence above exactly provides desired atomicity without durability . It doesn t guarantee which version of the file will appear in the event of an unexpected crash; if the application needs a guarantee that the new version of the file will be present after a crash, it s necessary to fsync the containing directory. Secondly, as we discussed above, fsync() really isn t that expensive, even in the case of ext3 and data=ordered; remember, fsync() doesn t create extra I/O s, although it may introduce latency as the application waits for some of the pending I/O s to complete. If the application doesn t care about exactly when the new contents of the file will be committed to stable store, the simplest thing to do is to execute the above sequence (open-write-fsync-close-rename) in a separate, asynchronous thread. And if the complaint is that this is too complicated, it s not hard to put this in a library. For example, there is currently discussion on the gtk-devel-list on adding the fsync() call to g_file_set_contents(). Maybe if someone asks nicely, the glib developers will add an asynchronous version of this function which runs g_file_set_contents() in a separate thread. Voila! Avoiding hard drive spin-ups with laptop_mode Finally, as Nathaniel Smith said in Comment #111 of of my previous post:

The problem is that I don t, really, want to turn off fsync s, because I like my data. What I want to do is to spin up the drive as little as possible while maintaining data consistency. Really what I want is a knob that says I m willing to lose up to minutes of work, but no more . We even have that knob (laptop mode and all that), but it only works in simple cases.

This is a reasonable concern, and the way to fix this is to enhance laptop_mode in the Linux kernel. Bart Samwel, the author and maintainer of laptop_mode, actually discussed this idea with me last month at FOSDEM. Laptop_mode already adjusts /proc/sys/vm/dirty_expire_centisecs and /proc/sys/vm/dirty_writeback_centisecs based on the configuration parameter MAX_LOST_WORK_SECONDS, and it also adjusts the file system commit time (for ext3; it needs to be taught to do the same thing for ext4, which is a simple patch) to MAX_LOST_WORK_SECONDS as well. All that is necessary is a kernel patch to allow laptop_mode to disable fsync() calls, since the kernel knows that it is in laptop_mode, and it notices that the disk has spun up, it will sync out everything to disk, since once the energy has been spent to spin up the hard drive, we might as well write everything in memory that needs to be written out right away. Hence, a patch which allows fsync() calls to be disabled while in laptop_mode should do pretty much everything Nate has asked. I need to check to see if laptop_mode does this already, but if it doesn t force a file system commit when it detects that the hard drive has been spun up, it should obviously do this as well. (In addition to having a way to globally disable fsync() s, it may also be useful to have a way to selectively disable fsync() s on a per-process basis, or on the flip side, exempt some process from a global fsync-disable flag. This may be useful if there are some system daemons that really do want to wake up the hard drive and once the hard drive is spinning, naturally everything else that needs to pushed out to stable store should be immediately written.) With this relatively minor change to the kernel s support of laptop_mode, it should be possible to achieve the result that Nate desires, without needing force applications to worry about this issue; applications should be able to just simply use fsync() without fear. Summary As we ve seen, the reasons most people think fsync() should be avoided really don t hold water. The fsync() call really is your friend, and it s really not the villain that some have made it out to be. If used intelligently, it can provide your application with a portable way of assuring that your data has been safely written to stable store, without causing a user-visible latency in your application. The problem is getting people to not fear fsync(), understand fsync(), and then learning the techniques to use fsync() optimally. So just as there has been a Don t fear the penguin campaign, maybe we also need to have a Don t fear the fsync() campaign. All we need is a friendly mascot and logo for a Don t fear the fsync() campaign. Anybody want to propose an image? We can make some T-shirts, mugs, bumper stickers Related posts (automatically generated):

1. Delayed allocation and the zero-length file problem A recent Ubuntu bug has gotten slashdotted, and has started...

And there we have done it - new sections in the archive. Read more about it in my d-d-a post and the whole set of affected packages is also available - and updated slightly, as some small things slipped through. The whole process was less painful than feared. After some discussion, started a few weeks ago we finally went and coordinated the change today. There was no reliable and good way for changes that large, so we had to create our own way. (The last section changes had been done ages ago). Mike provided a little python script which was able to parse our input file with the section matches against our database, so we ended up with a list we could possibly feed back into the database. Except, not really. Thanks to various things the list generated needed some manual work (which I also managed to fuck up a little), but after that was done we were ready to go. And while Mark told the wanna-build admins about the new sections (for some reason our buildd network cares about sections) and prepared a patch for our policy, I prepared the mail for the d-d-a list. Then we all had a break, as a dinstall was still running, and we did not want to break that. The actual changes then took only a few seconds, sending the mail to d-d-a took longer, thankyou gnupg. :) I wonder how much afterwork this will generate. There are at least the policy changes, lintian, the packages control files to adjust. Also some websites, where Frank already changed packages.debian.org to know about the new names. (Before he fixed up me forgetting some PHP packages to move. This way all of the active ftpteam members did something in it :) ).

---

In other news, we are currently stress-testing our ftp-master host a little. Well, not really, it could take more, but right now we are scanning the full archive once, importing all filenames from all our Debian packages. The goal is to get projectb larger. Err, no, actually we do want to get rid of apt-ftparchive, as that is one of the worst tools in our dinstall runs. It is slow as hell in the mode it has to run for us, making up a large part of our runtime. Step 1 to get rid of it is to generate the content files on our own, and before we can do that we have to import them all Which means we have to unpack every binary package once, reading in all filenames to store them in our database. Thats a lot of data. Currently the associations table, which is nothing more than 4 integers per row, takes up 972MB, and import is still running. (The index on the binary pkg id in this table alone has 700MB). And this table does nothing more than linking the binary package to the filename and filepath. Which are two extra tables, with the filenames having only 87MB, the path entries only taking up 30MB. Yes, anyone using databases will tell me thats not much, and it really isn t that much. It s just that the largest table before this import had 120MB in it. And some of the others used more space for their indexes than for the actual data. :) Anyway, this is only step one on our way to get rid of the slowness apt-ftparchive is. When the contents all work as we want them, the next step will be to get rid of it generating Packages/Sources files. Which means we have some more data to store in projectb. And after that - we also intend to do a Contents file for our source. Each and every source file listed. Guess we really will end up with some more large tables. :) The final goal and main reason why we do this is to save us at least 60 minutes runtime per dinstall run. My own guess is that Contents alone will get us about 20 minutes down. No, before someone wants to yell at me - apt-ftparchive can t be changed as much to make this possible and keep using it. Not in the way the generate option works. It has the disadvantage that it scans the files after they are installed in their pool/ location and stores that in its own database . This approach has to negative points:

• first, it is using another database (BerkeleyDB). We do have one database already, projectb, thank you.
• second, and more important, the files only end up in pool/ on dinstall time. Which means the whole scanning is done during dinstall. Which eats time. The new way will surely use the same amount of cpu time in total - but it will spread it over the whole day. 4 runs each hour. Not 4 runs each day. And during dinstall it will boil down to one sql query per installed package moving data from a pending table into the real contents table. Which is a cheap operation.

---

The title says some more, so here is some more: We aren t only working on what I wrote above. There is also work on changing our database access. Historically dak used the pg module to do database work. While the module itself does work, the way its used isn t really the best one. Way too much groundwork everywhere. Not nearly as much of the nice things used people say python could have. So another largish work area is getting rid of pg. Migrating to psycopg2. And putting a proper OO class around it, which in the end hopefully hides those dark details from the application. And to achieve even more sanity, we (well, honestly, Mark) are looking into SQL Alchemy usage. (We also considered storm from Canonical, but that sucks when you take a look at its documentation. (Which documentation?). SQLA won there easily.) While I m ranting: WTF are people neglecting their docs that much? Ever tried looking at the documentation for psycopg2? Forget it. No, a db api compliance test suite is NOT documentation. If there would be any better python module to connect to postgres, I would veto the usage of psycopg2, but there doesn t seem to be one. Sucks.

For some time ago I ve been pondering about this question. As long as GSoC 2009 is about to start and people are looking for project ideas, I m posting here a very preliminar draft of my findings and an idea for a posible software project. It s pretty written on-the-fly while I ve managed to have some minutes between work and uni. Rigurous wording isn t P1. Debian s social contract 4th item states:

Our priorities are our users and free software We will be guided by the needs of our users and the free software community. We will place their interests first in our priorities.

When we make decisions, either technical, legal and others regarding the operating system we deliver, this is one of the most referred argument. Altough that s seen as high priority often we don t have ways to properly know what our users want. While working on translations on the Spanish team we ve faced this situation. Do our users find them useful? they are able to understand them? they find that a much familiar wording would be bette than high technical details? We didn t know. Then, the question still exists. Altough Debian has ways for users to provide feedback (BTS, mailing lists, IRC, popcon, etc) none of them is designed to offer a way for the user to provide feedback on features, development roadmap and other non-bug aspects that developers and the project can tally and use for prioritize on releases. It s rather amusing to note that one of the most valuable assets free software community has is user involvement and contribution. But before we enter in details let s start on the basics. For using this information we first need to know who our users are? do we? From my findings and interaction with the community I ve identified two kinds of Debian users who can be clearly named. 1) The derivatives or pure blends, who use the Debian base and framework to build niche distributions. 2) The lead users (using E. Von Hippel s definition), who are the developers, contributors and a group of users. Debian is, if not the only, one of the projects who is best for fostering lead users. Most of them at some point involve themselves in the development process and some, later become developers. There is a more diverse group who are end-users but it s unclear to me how we can group them. Despite that they probably represent the biggest part of the pie. So, how we do please everyone? can we? In my opinion we can, at some degree. By implementing tools for first: gather user feedback, make statistical data, tally, we later can take informed decisions. In Debian decisions are voted and the set of people who votes are only developers with their own constraints. I think this would ease the constant that release-decisions-regarding-foo-tech-legal-issue represented on each release iteration, saving time, health and bits. It could be useful for knowing more about who are the ones we develop things for. I m thinking of some sort of Dell s IdeaStorm.com for software. What do you think? I m willing to co-mentor this if someone finds that is an interesting project for GSoC. Let me know!.

Fred von Lohmman gave a lively talk tonight about public interest law (and I hope he inspired at least some NYUlawites to follow in that path) with a focus on Youtube s content ID system now being used by Warner Music to automatically delete/censor videos, such as this one from Youtube. Go Google. Go Youtube. Go Warner Music.

Fred von Lohmman gave a lively talk tonight about public interest law (and I hope he inspired at least some NYUlawites to follow in that path) with a focus on Youtube s content ID system now being used by Warner Music to automatically delete/censor videos, such as this one from Youtube. Go Google. Go Youtube. Go Warner Music.

I hate to say this, but often when somebody does lots of bug triage on a package I work on, I find it to be a net loss for me. I end up having to go through all the things that were changed, correct a bunch of them, occasionally pacify angry bug submitters, and all the rest of it, and often the benefits are minimal at best. I would very much like this not to be the case. Bug triage is supposed to help developers be more efficient, and I think most people who do bug triage are generally well-intentioned and eager to help. Accordingly, here is a series of mini-rants intended to have educational value.

• Bugs are not like fruit. Fruit goes bad if you leave it too long. By and large, bugs don't, especially if they're on software that doesn't change very much. There is no reason why a bug filed against a package in Ubuntu 4.10 where the relevant code hasn't changed much since shouldn't still be perfectly valid. Even if it isn't, it deserves proper consideration. My biggest single annoyance with bug triage is people coming around and asking if bugs are still valid when they haven't put any effort into reproducing them themselves. This annoys bug submitters too; every so often somebody replies and says "didn't you even bother to check?". This gives a very bad impression of us as a project - wouldn't it be better if we looked as if we knew what we were talking about? There is a good reason to do this kind of check, of course: random undiagnosed crash reports and the like may well go away due to related changes, and it is occasionally worth checking. But if the bug is already well-understood and/or well-described, you should just go and check whether it's still there rather than asking. As I understand it, the intended workflow is that people file bugs, then if they aren't clear enough bug triagers work with the submitter to gather information until they are, then they're passed to developers for further work. We seem to have added an extra step wherein submitters must periodically give their bug a health-check, and if they don't then it gets closed as being out of date. In a small minority of cases this is useful; in most cases, frankly, it makes us look a bit clueless. Can we please stop doing this? The more we waste people's time doing this, the less likely it is that they'll bother to respond to us, and this might help our statistics but doesn't help the project as a whole. I know that there's a problem with bug count. I think every project of non-trivial size has that problem. But, honestly, the right answer is to fix more bugs - and, personally, I would be able to spend more time doing that if I weren't often running around trying to make sure that bugs I care about aren't getting overenthusiastically closed just because somebody thinks they've been lying around too long. There is a good way to expire bugs like this, of course. It goes something like this: "I've read through your bug and tried to reproduce it with a current release, but I'm afraid I can't do so. Are you still experiencing it? If not, then I think it might have been fixed by [this change I found in the package's history that seems to be related]." You can't do this en masse, but you'll get a much better response from submitters, you'll learn more doing it, and in the process of doing the necessary investigation of each bug you'll find that there are many cases you don't have to ask about at all.
• Wishlist bugs are not intrinsically bad. There are certainly cases where something is far too broad or vague for a bug report; but there are also plenty of cases, probably far more, where the wish in question is a relatively small change to the program, or doesn't need any more sophisticated tracking, and a wishlist bug is just right. If you don't know the program very well, it may be difficult to tell whether a wishlist bug is appropriate or not; in that case, just leave the bug alone. Please, for the love of all that's holy, don't close wishlist bugs saying that people should use Brainstorm or write a specification instead! If you don't want to see wishlist bugs in your statistics, just filter them out; it's quite easy to do. Even worse, don't tell people that something probably isn't a good idea when you aren't familiar with the software; people who have gone to the effort of writing up their idea for us deserve a response from somebody who knows the software well. I've encountered cases where friends of mine submitted a bug report (sometimes even at my request) and then a triager told them it was a bad idea and closed their bug. This sort of thing puts people off Ubuntu. Specifications are software design documents. As such, they are best written by software designers. People who tell other people to go and write a specification may not realise that as a result of doing this for three years it's now essentially impossible to find anything in the specification system! The intent was never that every user of Ubuntu would need to write a specification to get anything changed; specifications are used by developers to document the results of discussions and write up plans. They are not a straightforward alternative to wishlist bugs, nor do they turn out to work very well as what many formal processes call "requirements documents"; the process of refining the latter in the context of Ubuntu might involve wishlist bugs, mailing list threads, wiki pages, private discussions with developers, or things of that nature, and probably shouldn't involve creating a specification until the requirements-gathering process is well underway.
• Closing a bug is taking an item off somebody's to-do list. You wouldn't go up to a colleague's whiteboard and take an eraser to it unless you were sure that was OK, would you? Yet people seem to do that all the time with bugs. It's OK when the bug is really just like a support request - "help, it crashed, what do I do?" - and either you're pretty sure it's user error or there's just no way to get enough information to fix it. But once the initial triage process is done, now it's on somebody's to-do list. This is closely related to ...
• If a developer has accepted it, leave it alone. Every so often I find that there's a bug that I have accepted by way of a bug comment or setting to Triaged or whatever, or even a bug that I filed on a package I work on as a reminder to myself, and somebody comes along and asks for more information or asks if we can still reproduce it or something. The hit rate on this kind of thing is extraordinarily low. There's a good chance that the developer went and verified the bug against the code, and in that case it certainly doesn't need more information (or they would have asked for it) and it probably isn't going to go away without anyone noticing. In most other free software projects, developers file bug reports themselves as a reminder about things that need to be done, and people leave them alone unless they're intending to help with the fix. In Ubuntu, developers also have to spend time making sure that those to-do items don't get expired. Nobody is helped by this. launchpad-gm-scripts includes a Greasemonkey script called lp_karma_suffix, which can help you to identify developers without having to spend lots of time clicking around.
• Check whether the package is being actively worked on. Some packages are actively worked on in Ubuntu; some aren't (e.g. we just sync packages from Debian, or they're basically orphaned, or whatever). It's worth checking which is which before doing any kind of extensive triage work. If it's being actively worked on, why not go and talk to the developer(s) in question first? It's only polite, and it will probably help you to do a better job.

I came across the following from the ext3-users mailing list. The poor user was stuck on a never-updated RHEL 3 production server and running into kernel panic problems. He was advised to try updating to the latest kernel rpm from Red Hat, but he didn t feel he could do that. In his words:

I m caught between a rock and a hard place due to the EMC PowerPath binary only kernel crack. Which makes it painful to both me and my customers to regularly upgrade the kernel. Not to mention the EMC supportability matrix of doom.

That pretty much sums it all up right there. The good news is that I ve been told that dm-multipath is almost at the point where it has enough functionality to replace PowerPath. Of course, that version isn t yet shipping in distributions, and I m sure it needs more testing, but it ll be good when enterprise users who need this functionality can move to a 100% fully open source storage stack. About the only thing left to do is to work in a mention of the Frying Pan of Doom and the recipe for Quick After-Battle Triple Chocolate Cake into the mix. Related posts (automatically generated):

1. Tip o the hat, wag o the finger Linux power savings for laptop users It s interesting to see how far, and yet how much...
2. How active are your local Linux User s Groups? At the Linux Foundation, I recently had been brainstorming with...

I came across the following from the ext3-users mailing list. The poor user was stuck on a never-updated RHEL 3 production server and running into kernel panic problems. He was advised to try updating to the latest kernel rpm from Red Hat, but he didn t feel he could do that. In his words:

I m caught between a rock and a hard place due to the EMC PowerPath binary only kernel crack. Which makes it painful to both me and my customers to regularly upgrade the kernel. Not to mention the EMC supportability matrix of doom.

That pretty much sums it all up right there. The good news is that I ve been told that dm-multipath is almost at the point where it has enough functionality to replace PowerPath. Of course, that version isn t yet shipping in distributions, and I m sure it needs more testing, but it ll be good when enterprise users who need this functionality can move to a 100% fully open source storage stack. About the only thing left to do is to work in a mention of the Frying Pan of Doom and the recipe for Quick After-Battle Triple Chocolate Cake into the mix. Related posts (automatically generated):

1. Tip o the hat, wag o the finger Linux power savings for laptop users It s interesting to see how far, and yet how much...
2. How active are your local Linux User s Groups? At the Linux Foundation, I recently had been brainstorming with...

Every time something goes wrong there will be someone who tries to take advantage of the situation. The recent bushfires in Australia that have killed hundreds of people (the count is not known yet) are a good example. Pastor Nalliah of Catch the Fire Ministries [1] claims that it is due to legalising abortion. This is astoundingly wrong. In a more extreme example representatives of the Westboro Baptist Church were planning to visit Australia to launch a protest in support of the bushfires [2]. I have not yet found any news reports about whether they actually visited Australia or protested - it s most likely that they decided not to visit due to the Australian laws being very different to US laws regarding the relative importance of freedom of speech and incitement to violence. Apparently the insane Westboro Baptist Church people (who are best known for GodHatesFags.com and GodHatesAmerica.com) believe that God hates Australia and caused the fires (presumably due to Australia not persecuting homosexuals). Danny Nalliah has permanently damaged his own reputation by acting in a similar way to the Westboro Baptist Church. The reputation of Catch The Fire now depends on how quickly they get a new pastor Please note well that the vast majority of Christians have nothing in common with Westboro or Catch The Fire. I don t recall the last time I met an Australian Christian who was strongly opposed to homosexuality or abortion. Now we do have to try and investigate ways of avoiding future tragedies, and the work to do this needs to begin immediately. John Brumby (the Premier of Victoria) has announced that Victoria will get new strict building codes for fire resistant buildings [3]. There have been many anecdotes of people who claim to have been saved by attaching sprinkler systems to their homes, by building concrete bunkers to hide in while the fire passes, and using other techniques to save their home or save themselves. Some more research on the most effective ways of achieving such goals would be worthwhile, an increase in funding for the CSIRO to investigate the related issues would be a good thing. The article also has an interesting quote As the fallout from the disaster widened, the union representing the nation s 13,000 firefighters warned both the federal and state governments to take global warming seriously to prevent a repeat of last weekend s lethal firestorm . However given that traditionally Australia and the US have been the two nations most opposed to any efforts to mitigate global warming it seems unlikely that anything will change in this regard in a hurry. The attempts to link bushfires to abortion and homosexuality are offensive, but can be ignored in any remotely serious debate about politics. However there are some other groups trying to profit from the tragedy that make claims which are not as ridiculous. On the 9th of February the Australian Green party was compelled to release an official statement from Spokesperson Scott Ludlam, Sarah Hanson-Young, Rachel Siewert, Christine Milne, and Bob Brown following some political discussion about Greens policies [4]. There have been attempts to blame the Greens for the tragedy which were politically motivated, some of which came from groups that traditionally oppose the Greens for other reasons (I m not going to provide the detail - anyone who is really interested can do google searches on the people in question). On the 16th of February Bob Brown (the leader of the Green party) felt obliged to make another media release reiterating the fact that the Greens support prescribed burn-offs to limit the scope of wild fires [5], he also decried the hate mongering that has been occurring in the wake of the disaster. One of the strange memes that seems to be spread by opponents to the Greens is that the Greens are all supposedly from the city and know nothing about the country. To avoid being subject to such attack I feel obliged to note that on one of the bad fire days I visited my parents. I spent the morning with my father and some friends at a park that was not far from the fire area, my friends then returned to their home which was not far from the fire area. I then had lunch with my parents and watched the smoke through the dining room window. After that my friends didn t respond to email for a while and I was concerned that they may have lost their house or maybe suffered injury or death. I didn t know them well enough to feel it appropriate to try a dozen different ways of contacting them (I m sure that many other people were doing so), but I was rather concerned until my wife received an email from them. But I don t base my political beliefs on what I personally observe or my connections to people on the edge of the fire zone. I believe in the Green principles of Peace and Non Violence, Grassroots Democracy, Social and Economic Justice, Ecological Sustainability and the use of science and statistics to determine the best ways of achieving those goals.

• [1] http://www.livenews.com.au/articles/2009/02/11/Bushfires_caused_by_abortion_laws_Pastor_claims


• [2] http://www.news.com.au/story/0,27574,25067362-2,00.html


• [3] http://www.theage.com.au/national/never-again-brumby-20090211-84sv.html?page=-1


• [4] http://greensmps.org.au/content/time-heal-our-communities


• [5] http://greens.org.au/node/5229

Life sometimes has extreme ways of waking people up and making them realize what matters for them. I'm going through such a phase right now in my romantic life. I won't discuss what happened here, but suffice it to say that I haven't felt so depressed in a long, long, long, long time. The good news is that life has proven to me once again how having good friends who truly care for me makes a whole world of difference. The big slap in the face that life gave me over these last few days was a harsh lesson, but I think that it was worth it. Of course, it helps that people like Gilles, Jaana, Henni, Pia, Linda, Arno and Sailesh beleived in me enough to support me through this rough spot in my life. Although their advices were vastly different, they all pointed in a very positive direction and gave me the strenght to carry on. I also must thank my beloved Anastassija; what she did was cruel beyond beleif and yet, if her few SMS from the last 24 hours are any indication, there's still hope after the storm. I'll be seeing her tonight again, after a few days of separation and I prepared her a truly romantic evening. It ain't much, but I hope that it truly succeeds in bringing the point accross that Love is stronger than everything and that even the unforgivable can be forgiven, over time, if everyone involved has enough faith in each other's future as a couple. Tonight, I truly could use the prayers of a million voices to support me and Anastassija in this difficult time. Life slapped me hard and I got the message... I think... and whatever happens after tonight, I need to know that I gave it my best shot and that, whatever her and I decide, Life can only take a turn for the better from now on.

At the Linux Foundation, I recently had been brainstorming with some my colleagues about ways in which we might be able to reach out to the various local Linux User s Group (LUGS). So I was quite surprised when I saw a blog posting from Lenovo s Connections blog asking the question: Local User Groups - gone the way of the dinosaur? and even more provocatively, Has the web now relegated the user group to history, along with dot matrix printers, cassette tapes for mass storage, and dial up modems with acoustic couplers? I checked on the web for a number of LUG s, such as the Boston Linux and Unix User Group (BLU) and the Silicon Valley LUG (SVLUG), and they appeared to be active, at least in terms of mailing list activity and event calanedars. But of course that doesn t indicate how many people are actually attending the face-to-face meetings, and how strong the LUG s are in general. So let me ask here do you attend or participate in a LUG? How active is it? What do you get out of participating in a LUG?

I have been told that not enough information about the Plone user interface sprint at Informaat is getting out. That is not due to a non-disclosure agreement for attendees; all of us have just been too busy working and enjoying ourselves to get to such mundane things as writing reports or blog posts. This sprint is somewhat different than normal sprints: the goal is not to produce as much code as possible, but to lay a foundation for the user interface for future versions of Plone. This requires a lot of thought and prototyping to figure out the right concepts, both in the user interface itself and the backend code on top of which it is built. A lot of this work is done on top of the work done by the little known deco group. There are a few guiding ideas behind the work done at this sprint:

• separate all the knobs and toggles related to content editing and page manipulation out from the normal view.
• there are three different modi of interactiving with a page: view, edit, and construction (modifying layout, managing tiles). These require different types of user interface and should be handled separately.
• make the interface as simple as possible. If possible try to use a progressive complexity approach: keep things as simple as possible, and only expose more advanced options once users are ready to use them.
• always try to use simple semantic markup. There is very rarely a need to add extra elements of pollute the markup with layout or positioning-related information; that should all be done in CSS.

We divided the activities in five areas: content editing, control panels, page assembly, backend and tests. The first day was almost entirely spent on brainstorming, which is very rare for a Plone sprint. This was extremely useful, and provides us with a solid foundation for the rest of the sprint, and ongoing improvement of the Plone user interface.

A little while back, Terah was leaving the house one morning. On that particular morning, we had an unusually intense even for Kansas wind. It caught the storm door and whipped it around with such force that it bent the metal arm that goes into the pneumatic plunger, and even bent the bracket that the arm connects to. It was not possible to un-bend either of them with household tools, even with dad s help. So I went to the hardware store and picked up a replacement. But I had been putting off actually replacing it because, well, the door almost closed automatically and there were more important things to get to. Until yesterday. I got out the tools and went to work. Jacob was around and took an interest in the activity. I was stretched out on the floor on my stomach working with the drill to try to get out some extremely tight screws without stripping them. And Jacob, at one point, got down on the floor on his stomach right beside me, and proudly announced I (am) like dad! Terah was working on something nearby, and noticed this, and went for the camera. Unfortunately, I had left its CF card in my digital audio recorder. She found the other card, and was hurrying to put it in so she could get the cute picture. Even more unfortunately, she inserted it sideways. CF slots have 50 tiny pins in the space of 1.7 inches. The slots normally force the card to align properly, but not when the card is put in sideways. It slid back and bent a pin back. The slot is thus unusable. I was going in to town anyway to pick up my bike from the bike shop, so I stopped at Moler s Camera on the way. This is my favorite camera shop it s the sort of place where you take a seat at the counter while they help you. I showed the sales guy the camera and he knew off the top of his head that the cost to fix it is $206. Which is pretty much in line with reports on the Internet. The old camera is a Canon Digital Rebel XTi (EOS 400D). It s over 2 years old, so out of warranty, but I wasn t planning to upgrade soon. On the other hand, the camera shop estimated that they could buy the XTi from me for about $150 to $200, considering its condition, and a new XSi (EOS 450D) is $570 at B&H. So about $200 to fix the old one, or about $400 cost to us to get a new one and the next model up, which uses SD instead CF, and thus will never have a bent pin again. With that kind of cost, we decided to spring for the XSi. Plus, it would take about a month to get the repair on the XTi. So there you have it. The wind can be expensive and cause camera upgrades.

Fellow R user Paul Gilbert had invited me to come to Ottawa and the Bank of Canada to give a presentation/workshop on 'high-performance computing with R' similar to the UseR 2008 tutorial and talk. I just posted the updated slides from this talk, and there is also an updated live cdrom on the Alioth server. Also, it looks like the tutorial will be held again at UseR 2009 in Rennes, see here for a brief synopsis. It was nice to get back to Canada, even if it was a 24 hour whirlwind trip. Ottaws looked quite pretty in all the snow. And it seems that I got rather lucky with the travel dates as both the days before and after my trip had a large number of flight cancellations and delays due to snow storms.