Samba.grep.be, my main server (which occasionally also hosts a domU for p2) was upgraded to lenny this monday evening. I wish I could say there were no issues, but unfortunately that's not true. At first, everything seemed to run smoothly; of course I did have to recompile a few locally written programs to run against some SONAME changes (programs that are too trivial to package, really, and too ugly too), but other than that, everything was fine. It went wrong sometime during the night. Apparently something didn't like the new kernel or the new xen, because everything started to lock up. Since I only noticed this when I was at a customer the next day, I couldn't fix it until I got home again in the evening (meaning, downtime for a full day); and unfortunately, the exact failure mode meant that not much was written to the log -- certainly not enough to figure out what went wrong:

Mar 10 04:04:41 samba kernel: [ 7033.011523] INFO: task cron:21006 blocked for more than 120 seconds.
Mar 10 06:43:46 samba kernel: [ 7033.011614] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 10 17:35:04 samba kernel: [ 7033.011701] cron          D 7fffffffffffffff     0 21006   4147

After that, nothing before the reboot, which happened at around 21:00. Take good note of the timings on those messages, and the apparent discrepancies between the syslog timestamp and the kernel timestamp. 7033 seconds after boot was around 3:30 AM... When I arrived at the physical server, this kind of message was rolling over the console several times a second, with just the name and PID of the blocking process changed. And with several more lines following the message, of course. As a result, the system was not responsive at all. I didn't file a bug since, frankly, there's so little information that I wouldn't even know what to file a bug against (the kernel? Xen? Some rogue process that runs as root and changes settings which cause a driver to block? I wouldn't know). Instead, all I could do was to reboot the server and hope for the best. Not that I like that kind of situation. Fortunately, the problem didn't reproduce itself today; here's for hoping that it never will.

As funny as these stories are for reading, any hard data is absolutely lacking. Are there similar stories which actually cite sources? OTOH the fact that even cheap main stream devices like USB sticks and WLAN routers routinely have compatible with Linux , often even giving a kernel version, printed on the package tells me that the stories are not that far off. Even so, I'd be curious who these Icelandic Microsoft Certified Partners switching to Linux / Free Software are (first link, the quote below also originates there).

Microsoft Access, the only database software on the planet that s better at printing mail-merged stickers than it is at storing data.

I'm eternally confused about the state of powermanagement in Debian (or in Linux generally? Not using any other distribution seriously, I have no idea.) There are just too many scripts who interact or merely run in parallel (see my short note in my first posting about my shiny toy. While I don't actively invest time to educate myself about the situation, I've just tried to uninstall a round of unneeded packages and got rid of apmd and hibernate, which both were installed by default (or by dependencies of other stuff I've got rid of earlier?) but seem not to be necessary. At least the laptop still suspends when I close the lid. This is without rebooting or even just logging out, though, so if it doesn't work out, I'll have to update this entry. Update: Thanks to Michael's comment, pointing to #451380. Scope for a Google Summer of Code project, perhaps? This would be 90% talk to people and get a consensus and only 10% coding, though, but I think it would be worth it so that squeeze would have a powermanagement /acpi framework where different components don't stand on other components' toes all the time.

So I am going to do something that nobody on the Internet is doing lately: post a review of the Kindle 2 after having only used it for three days. Shocking, yes, I know. I had never even seen a Kindle of either model before getting the Kindle 2. I had, though, thought about getting an eInk device for some time. The $359 Kindle 2 price tag caused me significant pause, though in the end I went for it due to the 30-day return policy. On the surface, I thought that it would be weird to have a Kindle. After all, how often am I away from the computer? And there s a small local library a few blocks from where I work. But I had a hunch it might turn out for me like my iPod did: something that didn t sound all that useful from reading about it, but turned out to be tremendously so after having actually used it. Turtleback Delivery I ordered my Kindle 2 with standard shipping, which meant that it went by FedEx Smart Post. Here is my SmartPost rant. There are two words in Smart Post that are misleading. I once had an item take literally a week to make it from St. Louis to Kansas. That is, I kid you not, slower than the Pony Express ran in 1860. This time, my Kindle made it from Kentucky to Kansas in a mere five days. Oh, and it spent more than 24 hours just sitting in St. Louis. The Device Overall, the device is physically quite nice. It is larger and thinner than I had imagined, and the screen also is a bit smaller. It is usually easier to hold than a paperback, due to not having to prevent it from closing too far at the binding edge. The buttons are easy to press, though I could occasionally wish for them to be easier, but that s a minor nit. The Screen The most important consideration for me was the screen. The eInk display as both stunningly awesome and disappointing. This is not the kind of display you get on your computer. Or, for that matter, any other device. It isn t backlit. It reacts to light as paper does. It can be viewed from any angle. And it consumes no power to sustain an image; only to change it. Consequently, it puts up a beautiful portrait of a famous author on the screen when it is put to sleep, and consumes no power to maintain it. The screen s response time isn t anywhere near as good as you d expect from a regular LCD. It flashes to black when you turn a page, and there is no scrolling. On the other hand, this is not really a problem. I found the page turning speed to be more than adequate, and probably faster than I d turn the page on a real book. The resolution of the display has the feeling of being incredible. The whole thing provides a far different, and in my eyes superior, experience to reading on an LCD or CRT screen. My nit is the level of contrast. The background is not really a pure white, but more of a light gray. This results in a contrast level that is quite clearly poorer than that of the printed page. At first I thought this would be a serious problem, though I am growing somewhat more used to it as I read more. Reading Experience Overall, I ve got to say that it is a great device. You can easily get lost in a book reading it on the Kindle. I m reading David Copperfield for the first time, and have beat a rather rapid path through the first five chapters on the Kindle already. And that, I think, is the best thing that could be said about an ebook reader. It stays out of the way and lets you immerse yourself in your reading. The Kindle s smartly-integrated Oxford-American Dictionary was useful too. One thing about a novel written 150 years ago is that there are some words I just haven t ever heard. Nosegay, for instance. You can move a cursor to a word to see a brief pop-up definition appear, or press Enter to see the entire entry. This is nice and so easy that I m looking up words I wouldn t have bothered to if I were reading the book any other way. A nosegay, by the way, is a bouquet of showy flowers. Buying Experience The Kindle has a wireless modem tied to the Sprint network on it. The data charges for this, whatever they may be, are absorbed by Amazon in the cost of the device and/or the books you buy for it. This turned out to be a very smart piece of engineering. I discovered on Amazon s Kindle Daily Post that Random House is offering five mostly highly-rated sci-fi books for free on the Kindle for a limited time. So I went over to the page for each, and made my purchase . It was only a click or two, and I saw a note saying it was being delivered. A few minutes later, I picked up the Kindle off the kitchen counter. Sure enough, my purchases were there ready to read. Impressive. This level of ease of use smells an awful lot like Apple. Actually, I think it s surpassed them. You can delete books from the Kindle and re-download them at any time. You can initiate that operation from either the PC or the Kindle. And you can also browse Amazon s Kindle store directly from the device itself. I haven t subscribed to any magazines or newspapers, but I gather that they deliver each new issue automatically the moment it s released by the publisher, in the middle of the night. I pre-ordered the (free to Kindle) Cook s Illustrated How-to-Cook Library. It makes me way happier than it should to see This item will be auto-delivered to your Kindle on March 26 in the order status. Free Books Amazon s Kindle library has a number of completely free Kindle books as well. These are mostly out-of-copyright books, probably sourced from public etext places like Project Gutenberg, and converted to the Mobipocket format that is the Kindle s native format with a minimum of human intervention. As they are free, you can see them in Amazon s library if you sort by price. And, of course, Amazon will transfer them to the Kindle wirelessly, and maintain a copy of them in your amazon.com account. Unfortunately, as with free etexts in general on the Internet, the quality of these varies. I was very annoyed to find that many free etexts look like they were done on a typewriter, rather than professionally printed. They don t use smart quotes; only the straight ones. When reading on a device that normally shows you a faithful print experience, this is jarring. And I spent an inordinate amount of time trying to find a copy of Return of Sherlock Holmes that actually had the graphic figures in Dancing Men. Ah well. Your Own Content Amazon operates a mail server, username@kindle.com. You can email stuff to it, and it will convert it to the Kindle format and wirelessly upload it to your kindle for a fee of $0.10. Alternatively, you can use username@free.kindle.com, which does the same thing at no charge, but emails you back a link to download the converted work to install via USB yourself. I tried it with a number of PDFs. It rejected about a dozen times from only my single mail message a PDF containing graphic images only. However, it does quite well with most text-heavy PDFs notably doing an excellent job with Return of Sherlock Holmes from bookstacks.org the only source I found that was both beautifully typeset and preserved the original figures. Unfortunately, the PDF converter occasionally has troubles identifying what should be a paragraph, particularly in sections of novels dealing with brief dialog. I have also sent it some HTML files to convert, which it also does a great job with. You can also create Mobipocket files yourself and upload them directly. There is a Mobipocket creator, or you can use mobiperl if you are Windows-impaired or prefer something scriptable on Linux. The device presents itself as a USB mass storage device, so you can see it under any OS. There s a documents folder to put your files in. You can back it up with your regular backup tools, too. And it charges over USB. Web Browser I haven t tried it much. It usually works, but seems to be completely down on occasion. It would get by in a pinch, but is not suitable for any serious work. The guys over at XKCD seem to love it; in fact, their blog post was what finally convinced me to try the Kindle in the first place. Final Thoughts I ve ordered a clip-on light and a leather case for the Kindle. The light, I believe, will completely resolve my contrast complaint. The leather case to protect it, of course. I can t really see myself returning the Kindle anymore. It s way too much fun, and it s making it easier to read more again. And really, if Amazon manages to reach out to a whole generation of people and make it easy and fun for them to read again and make a profit doing it, of course they may move up a notch or two from being an evil patent troll company to a positive social force company. Wow, never thought I d say that one.

One of the recent poor trends in mailing list discussions is to reply to a message with a comment such as FAIL or EPIC FAIL . The FAIL meme has been around for a while and actually does some in some situations, slate has a good article about it [1]. The first example cited in that article is that when Ben Bernanke and Henry Paulson testified before the Senate banking committee last month about Paulson s proposed bailout bill, a demonstrator in the audience held up an 8.5-by-11 piece of paper with one word scrawled on it in block letters: FAIL. . This is an effective form of political demonstration, short words generally work well on placards (if only because the letters can be larger and therefore read from a greater distance) and anyone can understand the meaning of FAIL in that context. There are some blogs dedicated to publicising supposed failures, failblog.org and ShipmentOfFail.com are two examples. I cite these as supposed failures because some of the pictures that they contain are obviously staged. It s basically an Internet equivalent of the Funniest Home Videos shows that I never watched because they were not particularly funny. So using the word FAIL on it s own can be an effective form of political protest and can be used for mildly amusing web sites. But where it falls down is when it s applied to a discussion that involves people who are from different cultures or have different levels of background knowledge - which covers most mailing list discussions. Something that might be obviously wrong to some people is often not obvious at all to others. For example being forced to reboot a computer for any reason other than a kernel upgrade seems obviously wrong to me (and to most people who use Linux or other Unix systems) but Windows users seem happy to reboot machines after applying patches or upgrades. So writing a message with FAIL as the only word in a discussion with Windows users would not be productive. It could however be reasonable to forward a link to a page on a Microsoft web site to Linux people for their amusement with FAIL as the only comment - anyone who would find the link in question amusing would require no more explanation. Sometimes when in a debate someone will write a message that only says FAIL , this is a very unconvincing argument that will not convince the opposition or any onlookers. Generally it seems that using FAIL in a discussion with other like-minded people when talking about someone outside your group for the purpose of amusement can be effective. But any other use is going to be a FAIL . As a more general rule single-word messages seem to have little value apart from certain limited situations. I have identified the following seven scenarios where a single word message is useful. Can anyone think of any others?

1. Code review - someone posts code (or design for code) and people who like it will write ACK or something similar.


2. Arranging a meeting - the question who wants to meet for lunch tomorrow has me as a valid answer.


3. Voting - yes and no are valid answers for a poll, but a mailing list or forum probably isn t the best place for it.


4. Citing an example to refute a claim - often a single word won t be a great response but may be adequate to prove a point.


5. Answering a request for a recommendation - if asked to recommend a laptop I might say Thinkpad or if asked to recommend a server I might say HP . Both those answers are poor (I recommend EeePC for netbooks and Dell for small/cheap servers), so while such an answer would be useful it would be below my usual quality standards for email (I prefer to write at least two paragraphs explaining why I recommend something).


6. Informing people that something has been done by replying to a request with the word Done .


7. Agreeing to a contract or proposal with OK or Yes .

Update: I added another two reasonable uses of single word messages,

• [1] http://www.slate.com/id/2202262/pagenum/all/

Unfortunately Wammu 0.30 was not running on some Windows versions, so here comes fixed one. There is no need to update if you did not use Windows binaries. Full list of changes:

• Fixed Windows build.
• Fixed typo in desktop file.
• Fixed spec file.

It s been quite the week. Stomach Flu Last Friday, my stomach was just starting to feel a little odd. I didn t think much off it a little food that didn t go over well or stress, I thought. Saturday I got out of bed and almost immediately felt like throwing up. Ugh. I probably caught some sort of stomach flu. I was nauseous all day and had some terrible diarrhea to boot. I spent parts of Saturday, Saturday night, Sunday, and Sunday night supervising some emergency downloads as the BOFH would say. By Sunday afternoon, I thought I was doing good enough to attend a practice of the Kansas Mennonite Men s Choir. I made it through but it wasn t quite as up to it as I thought. Monday morning I woke up and thought the worst was behind me, so I went to work. By evening, the worst clearly was not behind me. I was extremely cold, and then got very hot a few hours later. Tuesday I left work a little early because of not feeling well. Servers Wednesday a colleague called me at home before I left to say that the ERP database had a major hiccup. That s never good. The database is this creaky old dinosaur thing that has a habit of inventing novel ways to fail (favorite pastime: exceeding some arbitrary limit to the size of files that no OS has cared about for 5 years, then hanging without telling anybody why). My coworkers had been working on it since 5. I went into the office and did what I could to help out, though they had mostly taken care of it. Then we went to reboot the server. It didn t come back. I/O error on sda just after init started, and it hung. Puzzled, as it just used that disk to boot from. Try rebooting again. This time, I/O error as the fibre channel controller driver loads. Again, puzzled as it just used that controller to load grub. Power cycle this time. And now the server doesn t see the fibre channel link at all. Eep. Check our fiber optic cables, and power cycle again. And THIS time, the server doesn t power back up. Fans whir for about a second, then an ominous red light I never knew was there shows up. Eeep! So I call HP. They want me to remove one CPU. Yes, remove one CPU. I tried, and long story short, they dispatch a local guy with a replacement motherboard. Can you send along a FC controller, in case it s dead too? Nope, not until we diagnose a problem with it. Local guy comes out. He s a sharp guy and I really like him. But the motherboard wasn t in stock at the local HP warehouse, so he had to have it driven in from Oklahoma City. He gets here with it by about 4:30. At this point the single most important server to the company s business has been down almost 12 hours. He replaces the motherboard. The server now powers up yay! And it POSTs, and it . doesn t see the disks. !#$!#$ He orders the FC controller, which is so very much not in stock that they can t get it to us until 8:30AM the next morning (keep in mind this thing is on a 4-hour 24/7 contract). Next morning rolls around. Outage now more than 24 hours. He pops the FC controller in, we tweak the SAN settings appropriately, we power up the machine, and . still doesn t see any disks, and the SAN switch still doesn t see any link. EEP! Even the BIOS firmware tool built into the controller doesn t see a link, so we KNOW it s not a software issue. We try plugging and unplugging cables, trying different ports, everything. Nothing makes a difference. At this point, while he ponders what else he can replace while we start migrating the server to a different blade. We get ERP back up on its temporary home an hour later, and he basically orders us every part he can think of while we ve bought him some room. Several additional trips later, he s replaced just about everything at least once, some things 2 or 3 times, and still no FC link. Meanwhile, I ve asked my colleague to submit a new ticket to HP s SAN team so we can try checking of the switch has an issue. They take their sweet time answering until he informs them this morning that it s been *48 HOURS* since we first reported the outage. All of a sudden half a dozen people at HP take a keen interest in our case. As if they could smell this blog post coming So they advise us to upgrade the firmware in the SAN switch, but they also say we really should send this to the blade group; the problem can t be with the SAN and of course the blade people are saying the problem s GOT to be with the SAN . We try to plan the firmware upgrade. In theory, we can lose a switch and nobody ever notices due to multipathing redundancy. In practice, we haven t tested that in 2 years. None of this equipment had even been rebooted in 390 days. While investigating this, we discovered that one of the blade servers could only see one path to its disks, not two. Strange. Fortunately, THAT blade wasn t mission-critical on a Friday, so I power cycled it. And it powered back up. And it promptly lost connection to its disks entirely, causing the SAN switches to display the same mysterious error they did with the first blade the one that nobody at HP had heard of, could find in their documentation, or even on Google. Yes, that s right. Apparently power cycling a server means it loses access to its disks. Faced with the prospect of our network coming to a halt if anything else rebooted (or worse, if the problem started happening without a reboot), we decided we d power cycle one switch now and see what would happen. If it worked out, our problems would be fixed. If not, at least things would go down in our and HP s presence. And that worked? What? Yes. Power cycling the switch fixed every problem over the course of about 2 minutes, without us having to do anything. Meanwhile, HP calls back to say, Uhm, that firmware upgrade we told you to do? DON T DO IT! We power cycle the other switch, and have a normal SAN life again. I let out a WOOHOO! My colleague, however, had the opposite reaction. Now we ll never be able to reproduce this problem to get it fixed! Fair point, I suppose. Then began the fairly quick job of migrating ERP back to its rightful home it s all on Xen already, designed to be nimble for just these circumstances. Full speed restored 4:55PM today. So, to cap it all off, within the space of four hours, we had fail:

• One ERP database
• ERP server s motherboard
• Two fiber optic switches but only regarding their ability to talk to machines recently rebooted
• And possibly one FC controller

Murphy, I hate you. The one fun moment out of this was this conversation: Me to HP guy: So yeah, that machine you ve got open wasn t rebooted in 392 days until today. HP guy: WOW! That s INCRED oh wait, are you running Linux on it? Me: Yep. HP: Figures. No WAY you d get that kind of uptime from Windows. And here he was going to be all impressed.

While I have most of my albums encoded as Ogg Vorbis files, the music from my college days (and before) is still only on disc, either in a big box in storage, or in one of a couple Case Logic CD wallets I used back then to lug my tunes around the globe. I ve long been meaning to encode those and shove the boxes and cases into storage, but in more than a year, that hasn t happened. A few weeks ago, my adorable girlfriend offered the necessary encouragement, suspecting that I might enjoy going through old music again. Right she was: it s great fun. I didn t think I was ever going to listen to Drum and Bass again, and now I am quite enjoying the music I listened to in high school. Unfortunately, while encoding all those discs, there s a pattern emerging: the discs from the box are all processed without any problems; the discs from the wallets yield many read errors. Inspecting the physical media, the cause seem to be scratches in the plastic deep enough to damage the reflective layer. When CDs came out, they were touted to be rigid and sturdy. The material quality has noticeably decreased in the years, as the producers kept cutting their costs. Discs of the past 15 years aren t good enough anymore to be stored in the Case Logic way. I wish I had known 15 years ago. NP: LTJ Bukem: Logical Progression

If I open a link from within one of my terminals, it causes Firefox to load that link in a new tab, which it activates (puts on top). Unfortunately, the browser window also steals the focus. This is driving me nuts at times. I want the new tab in the foreground, but I don t want the browser to steal the window focus. Firefox would not be Firefox if it didn t have an obscure, under-documented feature that almost achieves what I want, but comes with negative side-effects. As I learnt over at UbuntuForums, I can enable browser.tabs.loadDivertedInBackground, and the browser stays unfocused. The negative side-effect is that new tabs are loaded in the background. I cannot decide whether that s more or less annoying than the focus thievery. Rock, hard place, rock, hard place could someone please offer a way out? How can I make firefox load links in foreground tabs, but to keep its greedy little fingers off my focus? The rest of this post is personal. In cooperation with my therapist, I ve stopped wondering about Firefox design choices, and my life quality took a boost from that. Every other day, however, I find myself plotting ways to get rid of the Firefox crap, only to conclude that none of the alternatives provide the functionality I want/need, which are provided by some of the Firefox plugins: AdBlock+, CustomizeGoogle, Firebug, NoScript, TreeStyleTabs, and, of course, Vimperator. And there are others too. That s like being forced to drive a GM truck because the stereo you like doesn t work in any other car. I better go and swallow some more pills. Update: I ve filed a bug with the Mozilla bug tracker: Mozilla bug #481844

When I first started running a SE Linux Play Machine [1] I used passwords such as 123456 . Then for a while I had selinux but when I created a T-shirt design (see the main Play Machine page for details) I changed the password to SELINUX because that is easier to read on a shirt. Unfortunately the last time I rebuilt the Play Machine I used a password of selinux , some people worked this out and still logged in so I didn t realise that anything was wrong until a comment was placed on my blog yesterday. So for the past three weeks or so some people have been finding themselves unable to login. The password is now SELINUX again, sorry for any inconvenience. It s a pity that I can t make sshd a little less case sensitive for passwords. A PAM module to implement a caps-lock mode where the opposite case is tried would be useful for this case and some others too.

• [1] http://www.coker.com.au/selinux/play.html

all programs that interact with e-mail are broken in one way or another. Please be careful. - Lars Wirzenius

I seem to have a cunning knack of finding problems with configuring server software, particularly involving e-mail, where a) I can t find answers in Google, b) most people I go and ask for help say they d usually ask me such things, and c) if I go onto IRC or mailing lists I end up helping other people and not getting any help with my problem. It s quite likely this is just because I m something of a perfectionist, so the ridiculous crappy hacks people come up with and seem content to entrust their mail to are unacceptable to me for one reason or another. Anyway, in my ongoing quest for the perfect mail system, I ve painted myself into a corner again. (I m currently running with postfix, postgrey, clamav-milter, dspam, dovecot using LDA, managesieve and the cmusieve & antispam plugins. If I can get the current incarnation working, I ve had enough requests to write up a full HOWTO, and seen enough around with pretty questionable content, that I ll probably do it before too long.) I ve got postfix s local transport configured to hand mail to dspam over LMTP, using mailbox_transport = lmtp:unix:/dspam/dspam.sock. dspam is configured to listen here, add X-DSPAM-Result and signature headers, and then deliver the mail with dovecot s deliver LDA (which I ve set to 4750 root:dspam). From dspam.conf: ServerParameters "--deliver=innocent,spam -d %u"
ServerDomainSocketPath "/var/spool/postfix/dspam/dspam.sock"
...
Preference "spamAction=tag"
Preference "signatureLocation=headers"
...
TrustedDeliveryAgent "/usr/lib/dovecot/deliver" My dovecot configuration is pretty standard, using PAM for both passdb and userdb, and provides the auth-master socket that deliver needs. The problem I have is that postfix s local transport is qualifying the local username with the FQDN of the machine before delivering it to dspam with LMTP (the local mail transfer protocol), even for locally-originated mail which was only addressed with a bare username! dspam doesn t mangle it or care if the user is local or not, and then cheerfully invokes deliver -d robot101@omega.example.co.uk, which returns EX_NOUSER (addressee unknown) because my username is just robot101. From mail robot101: Mar 6 02:25:40 omega postfix/pickup[13607]: DAA4942F41F: uid=1000 from=<robot101>
Mar 6 02:25:40 omega postfix/cleanup[13637]: DAA4942F41F: message-id=<20090306022540.DAA4942F41F@omega.example.co.uk>
Mar 6 02:25:40 omega postfix/qmgr[12552]: DAA4942F41F: from=<robot101@omega.example.co.uk>, size=339, nrcpt=1 (queue active)
Mar 6 02:25:40 omega dovecot: auth(default): passwd(robot101@omega.example.co.uk): unknown user
Mar 6 02:25:40 omega dspam[13527]: Delivery agent returned exit code 67: /usr/lib/dovecot/deliver -d robot101@omega.example.co.uk
Mar 6 02:25:40 omega postfix/lmtp[13640]: DAA4942F41F: to=<robot101@omega.example.co.uk>, orig_to=<robot101>, relay=omega.example.co.uk[/dspam/dspam.sock], delay=0.08, delays=0.05/0.01/0.01/0.03, dsn=4.3.0, status=deferred (host omega.example.co.uk[/dspam/dspam.sock] said: 421 4.3.0 <robot101@omega.example.co.uk> Delivered (in reply to end of DATA command)) So, no e-mail for me. Dearest lazyweb, which of the three components is behaving wrongly, and how can I fix it? (And no, I m not just going to switch to GMail. I store my data on hard drives, which are sometimes in servers, not in the cloud . Until about a month ago, most people I knew spoke about clouds which were made of particles of water in the sky, rather than as a data storage media. What if it rains? ;)

all programs that interact with e-mail are broken in one way or another. Please be careful. - Lars Wirzenius

I seem to have a cunning knack of finding problems with configuring server software, particularly involving e-mail, where a) I can t find answers in Google, b) most people I go and ask for help say they d usually ask me such things, and c) if I go onto IRC or mailing lists I end up helping other people and not getting any help with my problem. It s quite likely this is just because I m something of a perfectionist, so the ridiculous crappy hacks people come up with and seem content to entrust their mail to are unacceptable to me for one reason or another. Anyway, in my ongoing quest for the perfect mail system, I ve painted myself into a corner again. (I m currently running with postfix, postgrey, clamav-milter, dspam, dovecot using LDA, managesieve and the cmusieve & antispam plugins. If I can get the current incarnation working, I ve had enough requests to write up a full HOWTO, and seen enough around with pretty questionable content, that I ll probably do it before too long.) I ve got postfix s local transport configured to hand mail to dspam over LMTP, using mailbox_transport = lmtp:unix:/dspam/dspam.sock. dspam is configured to listen here, add X-DSPAM-Result and signature headers, and then deliver the mail with dovecot s deliver LDA (which I ve set to 4750 root:dspam). From dspam.conf: ServerParameters "--deliver=innocent,spam -d %u"
ServerDomainSocketPath "/var/spool/postfix/dspam/dspam.sock"
...
Preference "spamAction=tag"
Preference "signatureLocation=headers"
...
TrustedDeliveryAgent "/usr/lib/dovecot/deliver" My dovecot configuration is pretty standard, using PAM for both passdb and userdb, and provides the auth-master socket that deliver needs. The problem I have is that postfix s local transport is qualifying the local username with the FQDN of the machine before delivering it to dspam with LMTP (the local mail transfer protocol), even for locally-originated mail which was only addressed with a bare username! dspam doesn t mangle it or care if the user is local or not, and then cheerfully invokes deliver -d robot101@omega.example.co.uk, which returns EX_NOUSER (addressee unknown) because my username is just robot101. From mail robot101: Mar 6 02:25:40 omega postfix/pickup[13607]: DAA4942F41F: uid=1000 from=<robot101>
Mar 6 02:25:40 omega postfix/cleanup[13637]: DAA4942F41F: message-id=<20090306022540.DAA4942F41F@omega.example.co.uk>
Mar 6 02:25:40 omega postfix/qmgr[12552]: DAA4942F41F: from=<robot101@omega.example.co.uk>, size=339, nrcpt=1 (queue active)
Mar 6 02:25:40 omega dovecot: auth(default): passwd(robot101@omega.example.co.uk): unknown user
Mar 6 02:25:40 omega dspam[13527]: Delivery agent returned exit code 67: /usr/lib/dovecot/deliver -d robot101@omega.example.co.uk
Mar 6 02:25:40 omega postfix/lmtp[13640]: DAA4942F41F: to=<robot101@omega.example.co.uk>, orig_to=<robot101>, relay=omega.example.co.uk[/dspam/dspam.sock], delay=0.08, delays=0.05/0.01/0.01/0.03, dsn=4.3.0, status=deferred (host omega.example.co.uk[/dspam/dspam.sock] said: 421 4.3.0 <robot101@omega.example.co.uk> Delivered (in reply to end of DATA command)) So, no e-mail for me. Dearest lazyweb, which of the three components is behaving wrongly, and how can I fix it? (And no, I m not just going to switch to GMail. I store my data on hard drives, which are sometimes in servers, not in the cloud . Until about a month ago, most people I knew spoke about clouds which were made of particles of water in the sky, rather than as a data storage media. What if it rains? ;) Update: The problem is fixed! Even though arguably the problem is dspam s for not knowing which users are local or not, it s fixable in dovecot 1.1.x using the auth_username_format = %n option. Thanks so much to Angel Marin for helping me out. Update 2: There s also a patch for dspam floating around which adds a StripRcptDomain option, which makes the LMTP server truncate the e-mail address at the @, so essentially assumes everyone to be a local user. The problem with both of these fixes is that they re both blunt instruments which will break virtual users on the same host. I think the real fix would be something more like a LocalDomains option in dspam, to choose which domains should be considered local and truncated from the e-mail addresses for delivery purposes.

At some point in the past, when I was still thinking that Glest can be packaged in a sane way, I had a conversation with upstream developer martinho on the Glest forum (they call it board) about them integrating a patch in order to support big endian machines.

After some messages on their board, it was clear that they don't want to integrate any patches and started to throw out random fake arguments against the integration of my patch and other patches.

When he figured out that I replied and debunked all of his fake arguments, he went on and actually removed my posts from the forum. It was so weird since I have never seen that kind of behaviour before and some of his posts and other people's posts were replies to my posts and things didn't made any sense after the deletion.

At that moment I have managed to save one small bit of the conversation via Google's cache, but now I have finally decided to make it public.


Note: Back then, I kept this post private in the hope that I was maybe too involved in the conversation at that moment to make a rational decision about it, and hoped things would get better in time. They didn't. Meanwhile they migrated to another forum software and they said they lost most/all(?) of the previous content from the old forum.



Here is the conversation bit I managed to save.
Original text is in italic font. Regular font is for my current comments for clearing up the context of the discussion. The text retains the BB syntax and [quote]-d text is martinho's own words.



[quote="martinho"]thebohemian, eddyd,

I think you guys are missing the point, this is not the place for any [/quote]

Congrats, you are avoiding to answer the points I have raised.

[quote]
- The Glest project is made by a bunch of friends for fun, we ask for nothing, we answer to no one. So I beg you not co complain, because it is out of order.
[/quote]

In that case I don't see the point of the "development" forum. care to explain?
And, from my PoV, these are your users you're arrogantly dismissing.


The following was a reply to my request to add big endian support through a patch I provided at that time. The patch wasn't complete to actually make glest work on BE machines, but it didn't broke little endian support at all. He was trying to argue that there were no valid big endian machines on which the game would run. Integrating the patch would have made my life a lot easier in continuing the work on the patch.

[quote]
- Our target plarforms are PCs running Windows (I'm responsible for this) or Linux (matze is the boss here). We don't care if Glest doesn't run in a big endian dishwasher with Solaris.
[/quote]

Linux on PowerPC is a viable platform. So is Mac OS X on PowerPC.



I made the mistake of saying that some parts of the code where not that clean and could be improved. More on that below.

[quote]
- One of the keys of Glest success is how extremely clean our code is. Bringing random people in would break this, and we are not going to do it. [/quote]

So you are the only keepers of true knowledge of clean code? Hahahaha!

There were at least 3 patches that provided this through word/double-word swapping. I don't see how embedding the patch can affect the unity of the code.

BTW, you are aware you can "carve" people into coding in your style before accepting in, aren't you?



I suggested using int8, uint8, int16, uint16 in order to make sure that the parameters are of the expected fixed size since plain char, int, long, long long have different sizes depending on the architecture. The usage of int and int16 was inconsistent and I also suggested using always the numbered variants.

I said the approach that was then taken in code was not portable and that was the reason for the proposed change.

[quote]
- Glest has been coded with portability in mind, some of your comments about portability are really naive (compilers don't care if you declare function parameters with fixed byte length, they just use whatever is the fastest int in the platform). Fixed length data types are used when needed,
[/quote]

What? When did I say anything about that? I said the code uses inconsistently "int" and "int16" and should use the later always because the code assumes the sizes on i386, which are not always applicable - see long on amd64.


If you mean that I can't enforce the width of the parameter, then you should know better, those types are defined in a platform dependent way to be platform independent. Yes, there are cases where [b]on the stack[/b] the size of the passed data is bigger (they might be padded), but that is transparent to the functions and, in spite of that, the functions will see the proper values for the parameters.

Also, the packing behaviour is compiler and architecture dependent, and some compilers produce the weirdest packing (if you are allowed access to such docs, look for Green Hills compilers' for V850 and you'll be stunned).


Please, don't make a fool of yourself by trying to discredit me, just not to loose face.

Later he just removed my posts not to loose face.




As a sane programmer, I made my patch to do the swapping only on big endian platforms, and the code was enabled in through the precompiler, so there was no performance hit on the little endian arches.

Of course, any more-than-newbie programmer would agree that having two data sets for a single game is stupid and is better to do the swapping in the application itself to avoid space wastage and maintaing an alternate big endian data set.

[quote]
no more, no less. About endianness, we don't care. If we wanted to fix the problem anyway, the right way is to do the byte swap in the data, this is how it is done in commercial games, since it is more efficient.
[/quote]

You just proved how much bullshit you can deliver in one phrase just for the sake of winning an argument. All portable games and all applications that have data files prefer to do swapping on the spot just because swapping the data on the disk (or providing BE and LE data) is plain stupid and inefficient from a disk and bandwidth PoV.

For Glest, swapping the data after reading from disk is not even affecting performance in a manner that could be significant or noticed, not even for a processor running at 500MHz, since that would be done only on load, once, and just for the bits of data which are relevant; these are mostly headers, since the actual data, in most cases, is just a long byte vector. This translates in not more than 3 or 4 instructions in assembler (an init, an exchange, an increment and/or a loop instruction), and that would happen before starting a game, which is dead time anyway and wouldn't be noticed unless the data would be huge (order of hundreds of MB or GB) and the whole data set would need swapping.

Even if the data itself (not headers) represents data which needs to be swapped, that is still irrelevant in terms of speed.


BTW, the reason that in commercial games it might be done like that is because there the games are distributed in a package per platform anyway and having two data sets is not an issue since they are using the same amount of CDs/DVDs, but doing the same for a game like Glest would mean doubling the storage used and it would increase the complexity for a release, not to count the confusion of the users.

I still doubt commercial games do this, but I might be wrong.

Dear recording industry: I hate almost all of you. One of my all-time favourite albums, Poland: The Warsaw Concert by Tangerine Dream, has been castrated, probably to save cost by squeezing a double LP onto a single CD. Today, Wikipedia turned my world upside down. Screw you, money-greedy recording industry. Please fuck off and leave music to those who care. This includes smaller labels who know what they are doing. I now must have the full-length track. Unfortunately, it s seemingly impossible to come by. I searched the Web, even dreaded sites like Ebay, but after an hour I gave up without success. There s a seller on Amazon, but s/he consistently gets negative feedback, so that s a bit risky for the price asked. Thus, dear world, please save me: does anyone have the full-length third track of that album, titled Barbakane , in a high-quality digital recording and would let me have it? I will pose with the album and today s newspaper to prove legal ownership. Heck, go ahead and make a ridiculous request, such as standing on my head. I m desperate. NP: Tangerine Dream: Phaedra

Yeah, yeah, I may be living under a rock these days. Anyway, I just noticed that the GFDL v1.3 is out; it has been for four months or so. There aren't many changes; apart from some minor wording changes that have little to no effect on the meaning of the license, there are two new sections: the first adds details on what happens when you violate the license (basically, it gets temporarily terminated, which can become permanent if you do not fix it in a reasonable timeframe or under some other circumstances); the second allows one to change the license from GFDL to CC-BY-SA under certain circumstances (where "certain circumstances" in practice basically means "you're wikipedia", although it isn't written as such). And that's it. Nothing groundbreaking, nothing to follow up on the FDLv2 discussion drafts that were released about two-and-a-half years ago. Me, I find that disappointing. When the initial discussion drafts were put out on the web, I was thrilled; I spent some time reading them and providing comments. However, nothing happened on those drafts beyond initially getting them out there. It would appear that today, the FSF has even managed to lose the comments that were made. I guess they don't really care about the FDL as much as they do about the GPL.

Recently, a new version of Gnash was uploaded to Debian by Miriam Ruiz. The most important thing in the changelog from my point of view is the following:

• Using KDE4 instead of KDE3

So I kind of had to test how the KDE 4 thingies works.
Youtube: The videos being watched right now -animation works, but I can t actually watch the movies.
Random flash advertisements plays nicely. Unfortunately. I was happy to avoid those. Conclusion: Some things works, but mostly the annoying things.

Lars wonders about debhelper's approach of inserting script fragments into package maintainer scripts. It's really very interesting to survey debhelper's collection of fragments, in its autoscripts directory. Of the 55 fragments, the mean length is 4.5 lines, and the mode (and also shortest[1]) length is 3 lines. The 32 scripts that are each three lines follow a very simple pattern:

if [ "$1" = something ] && exists some-script; then
    some-script some-parameters
fi

It would be nice to get that down to Lar's more declarative one line approach:

some-script some-parameters $@

In practice though, many of these scripts are not always available, and so the postinst has to test for them before running them. And making them test the postinst's parameters to see if they should do anything makes them less general and less flexible (ie, not able to be run easily by a sysadmin), and so keeping the "$1" test in the postinst is reasonable. So, I think I've done a pretty good job at keeping debhelper script fragments short and keeping most of the possibly buggy bits that might need maintenace out of debhelper. The exceptions tend to be either older scripts, from a time when I was less careful about doing do, and fragments of code that I could not pawn off onto some other package, often because there was no package that owned the issue. (The dh_usrlocal code is the best example of that.) The most promising way to improve things seems to be using dpkg triggers, which is why I was very enthuisiastic when we finally got them, over a year ago. Triggers let many of these hooks into maintainer scripts go away completely. Rather than each package having to test for and call update-menus, menu can simply trigger on the modification of any menu file. Once menu finishes the conversion to using triggers, debhelper can drop those fragments entirely. Triggers seem likely to be able to eliminate well over 90% of the things we need to put into maintainer scripts, and I hope at least half of debhelper's script fragments can eventually go away thanks to them. The others seem more difficult. You probably cannot, for example, make a trigger that does something sane when an init script is installed. Debhelper has at least three different fragements that it can put into postinst scripts to handle init scripts, and these only cover the most common three cases (start, don't start, don't restart on upgrade). Moving even these things into a declarative form has long been a dream of mine. Maybe it could be done with a control file that specifies which of the methods for running the package's init script to use for this package. But it seems we can't do away with maintainer scripts entirely, no matter how hard we try. That's because maintainer scripts are our only way to deal with other crippling bugs in old versions of a package, and with upgrade issues, and with all the other myriad things that we put custom code in them to deal with. Pity.

---

[1] Slight oversimplification; two scripts cram the 3-line form in one line for no good reason and are counted as 3-lines above.

On occasion, you will see the advice that the ext3 file system is not suitable for Solid State Disks (SSD s) due to the extra writes caused by journaling and so Linux users using SSD s should use ext2 instead. However, is this folk wisdom actually true? This weekend, I decided to measure exactly what the write overhead of journaling actually is in actual practice. For this experiment I used ext4, since I recently added a feature to track the amount of writes to the file system over its lifetime (to better gauge the wear and tear on an SSD). Ext4 also has the advantage that (starting in 2.6.29), it can support operations with and without a journal, allowing me to do a controlled experiment where I could manipulate only that one variable. The test workload I chose was a simple one:

• Clone a git repository containing a linux source tree
• Compile the linux source tree using make -j2
• Remove the object files by running make clean

For the first test, I ran the test using no special mount options, and the only difference being the presence or absence of the has_journal feature. (That is, the first file system was created using mke2fs -t ext4 /dev/closure/testext4, while the second file system was created using mke2fs -t ext4 -O ^has_journal /dev/closure/testext4.)

Amount of data written (in megabytes) on an ext4 filesystem

Operation	with journal	w/o journal	percent change
git clone	367.7	353.0	4.00%
make	231.1	203.4	12.0%
make clean	14.6	7.7	47.3%

What the results show is that metadata-heavy workloads, such as make clean, do result in almost twice the amount data written to disk. This is to be expected, since all changes to metadata blocks are first written to the journal and the journal transaction committed before the metadata is written to their final location on disk. However, for more common workloads where we are writing data as well as modifying filesystem metadata blocks, the difference is much smaller: 4% for the git clone, and 12% for the actual kernel compile. The noatime mount option Can we do better? Yes, if we mount the file system using the noatime mount option:

Amount of data written (in megabytes) on an ext4 filesystem mounted with noatime

Operation	with journal	w/o journal	percent change
git clone	367.0	353.0	3.81%
make	207.6	199.4	3.95%
make clean	6.45	3.73	42.17%

This reduces the extra cost of the journal in the git clone and make steps to be just under 4%. What this shows is that most of the extra meta-data cost without the noatime mount option was caused by update to the last update time for kernel source files and directories. The relatime mount option There is a newer alternative to the noatime mount option, relatime. The relatime mount option updates the last access time of a file only if the last modified or last inode changed time is newer than the last accessed time. This allows programs to be able to determine whether a file has been read size it was last modified. The usual (actually, only) example that is given of such an application is the mutt mail-reader, which uses the last accessed time to determine if new mail has been delivered to Unix mail spool files. Unfortunately, relatime is not free. As you can see below, it has roughly double the overhead of noatime (but roughly half the overhead of using the standard Posix atime semantics):

Amount of data written (in megabytes) on an ext4 filesystem mounted with relatime

Operation	with journal	w/o journal	percent change
git clone	366.6	353.0	3.71%
make	216.8	203.7	6.04%
make clean	13.34	6.97	45.75%

Personally, I don t think relatime is worth it. There are other ways of working around the issue with mutt for example, you can use Maildir-style mailboxes, or you can use mutt s check_mbox_size option. If the goal is to reduce unnecessary disk writes, I would mount my file systems using noatime, and use other workarounds as necessary. Alternatively, you can use chattr +A to set the noatime flag on all files and directories where you don t want noatime semantics, and then clear the flag for the Unix mbox files where you care about the atime updates. Since the noatime flag is inherited by default, you can get this behaviour by setting running chattr +A /mntpt right after the filesystem is first created and mounted; all files and directories created in that file system will have the noatime file inherited. Comparing ext3 and ext2 filesystems

Amount of data written (in megabytes) on an ext3 and ext2 filesystem

Operation	ext3	ext2	percent change
git clone	374.6	357.2	4.64%
make	230.9	204.4	11.48%
make clean	14.56	6.54	55.08%

Finally, just to round things out, I tried the same experiment using the ext3 and ext2 file systems. The difference between these results and the ones involving ext4 are the result of the fact that ext2 does not have the directory index feature (aka htree support), and both ext2 and ext3 do not have extents support, but rather use the less efficient indirect block scheme. The ext2 and ext3 allocators are also someone different from each other, and from ext4. Still, the results are substantially similar with the first set of Posix-compliant atime update numbers (I didn t bother to do noatime and relatime benchmark runs with ext2 and ext3, but I expect the results would be similar.) Conclusion So given all of this, where did the common folk wisdom that ext3 was not suitable for SSD s come from? Some of it may have been from people worrying too much about extreme workloads such as make clean ; but while doubling the write load sounds bad, going from 4MB to 7MB worth of writes isn t that much compared to the write load of actually doing the kernel compile or populating the kernel source tree. No, the problem was that first generation SSD s had a very bad problem with what has been called the write amplification effect , where a 4k write might cause a 128k region of the SSD to be erased and rewritten. In addition in order to provide safety against system crashes, ext3 has more synchronous write operations that is where ext3 waits for the write operation to be complete before moving on, and this caused a very pronounced and noticeable stuttering effect which was fairly annoying to users. However, the next generation of SSD s, such as Intel s X25-M SSD, have worked around the write amplification affect. What else have we learned? First of all, for normal workloads that include data writes, the overhead from journaling is actually relatively small (between 4 and 12%, depending on the workload). Further, than much of this overhead can be reduced by enabling the noatime option, with relatime providing some benefit, but ultimately if the goal is to reduce your file system s write load, especially where an SSD is involved, I would strongly recommend the use of noatime over relatime. Related posts (automatically generated):

1. Should Filesystems Be Optimized for SSD s? In one of the comments to my last blog entry,...
2. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...
3. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...