When I teach my course on computer hackers, one of the first things that students learn is that many hackers are obsessed a viewpoint promulgated by academics and media types alike. I tend to agree with the assessment but turn it on its head a little by pointing out that many professionals (doctors, lawyers, investment bankers and my own profession, which, attracts and houses nerds and geeks alike) are equally obsessed. It is just that the professional work is legitimized via all sorts of norms, expectations, and practices, unlike hacking, which is often treated in quite exceptional terms.

When I teach my course on computer hackers, one of the first things that students learn is that many hackers are obsessed a viewpoint promulgated by academics and media types alike. I tend to agree with the assessment but turn it on its head a little by pointing out that many professionals (doctors, lawyers, investment bankers and my own profession, which, attracts and houses nerds and geeks alike) are equally obsessed. It is just that the professional work is legitimized via all sorts of norms, expectations, and practices, unlike hacking, which is often treated in quite exceptional terms.

Within the next days I plan to update the gimp-plugin-registry package, including the latest versions of all plugins. So if you'd like to see a new fance plugin in the package, or you have any other wish, please add a comment here, or even better, file a wishlist bug report against the package. Patches are welcome, git repository informations are listed below.

• gitweb
• git://git.recluse.de/debian/pkg-gimp-plugin-registry.git
• http://git.recluse.de/repos/debian/pkg-gimp-plugin-registry.git

Extra points if you find new/forked versions of plugins which are/were listed on registry.gimp.org but disappeared or are not updated anymore. Since the page moved to a blog-like style, and old plugin entries are not updated as before, it became harder for me keep the plugins uptodate. Help is appreciated.

I've been on Twitter for over a year and a half now. If I had to explain why I like it, and why it's so popular now, I would use an analogy:

Twitter : IRC :: Blogs : Usenet

(This applies equally to other "micro-blogging" services, but I am about to explain why I believe that's not the right metaphor. You may also substitute mailing lists for Usenet.) With the older media, you have a place -- a newsgroup, or a channel, that people went to, with a distinct culture, and that (mostly) weren't "owned" by anyone, but rather by the community. With the new ones, we are all sole proprietors of our own streams, and we "tune in" to the subset of people we find interesting, rather than topics we invest in. So, instead of bumping into the same person in a couple different groups, or never reading their words at all, you might find that your feeds overlap a bit more than they do with most people. This is how I find people to "follow" and blogs to read, in fact -- as my network expands, more people become loosely joined to it, and as I notice ones worth reading I add them. Is one model better than the other? Probably not. I could make an analogy to music. In theory, I'd rather read what my favorite critics have to say about a wide variety of new releases -- some of which I'd never know about otherwise -- than keep up with the discussion of bands and genres I really like, even if most people writing about them are terrible (remember, 90% of everything is crap). But I also lose that sense of community of being a "fan" of something; I no longer have a deep connection to what's going on in the fandom or the scene, which I also would never know about otherwise (some of it is just too obscure for my favorite writers to cover). In practice, it seems the new approach has been more popular, but maybe that's because more people are on the net now, and both kinds of communication are/were shaped by the technology available at the time (destinations make much more sense with limited/centralized computing resources, and aggregation makes much more sense with powerful clients and a wider, less specialized user base). Anyway. This post is not actually about social media theory or whatever you want to call it; it's about some software I have packaged. Because of all the above, I have always wished that I could use Twitter from something more like my IRC client. Like, say, my IRC client. One could abuse the concepts of IRC to make an "on the fly" channel of whoever happens to be in my feed. (I once read a blog comment somewhere complaining that Twitter could easily be implemented on any existing IRC server using one +m channel for everyone and some client-side direction of messages from all such channels to a single window. +1 for cleverness, but they did sort of miss the point of why normal people sign up for web sites rather than installing and configuring clients for obscure chat protocols.) So, I had grand plans to write a Twitter client which was an IRC server, with some clever mapping of IRC concepts and commands to their equivalents over there. I never got around to it, as I barely have enough time for anything I do now. But last month I noticed that someone else had implemented such a thing: tircd. I had seen Twitter/IRC services before, but like the official Twiter XMPP service, they were all implemented as bots, which I detest for this sort of application. Bitlbee, for example, translates various IM protocols to IRC, but only halfway -- for anything else you have to use a bot as a sort of poor man's command line. If I want a command line for Twitter, I already have several; IRC can do better. And tircd really does! It's great. You're not required to edit the config file, and there's no extra layer on top of IRC for things like logging in or adding people. I've finally packaged the latest release, which is waiting in NEW currently. I got a request for sneak preview packages, so if you want to install some unapproved .debs check this repository. I may still pick my own project back up, as Twitter itself, being a centralized service, feels like a stopgap solution on the way to to a more generalized 140-character equivalent of the, er, blogosphere as envisioned by open-source projects like Identi.ca. In the future, perhaps, when we use a certain micro-blogging "service" we might be randomly connected to one of any number of servers run by different individuals but all mirroring messages back and forth to each other. Which, now that I think of it, sounds vaguely like some obscure, obsolete chat and news-posting protocols I know.

A recent Ubuntu bug has gotten slashdotted, and has started raising a lot of questions about the safety of using ext4. I ve actually been meaning to blog about this for a week or so, but between a bout of the stomach flu and a huge todo list at work, I simply haven t had the time. The essential problem is that ext4 implements something called delayed allocation. Delayed allocation isn t new to Linux; xfs has had delayed allocation for years. Pretty much all modern file systems have delayed allocation, according to the Wikipedia Allocate-on-flush article, this includes HFS+, Reiser4, and ZFS; btrfs has this property as well. Delayed allocation is a major win for performance, both because it allows writes to be streamed more efficiently to disk, and because it can reduce file fragmentation so that later on they can be read more efficiently from disk. This sounds like a good thing, right? It is, except for badly written applications that don t use fsync() or fdatasync(). Application writers had gotten lazy, because ext3 by default has a commit interval of 5 seconds, and and uses a journalling mode called data=ordered. What does this mean? The journalling mode data=ordered means that before the commit takes place, any data blocks are
associated with inodes that are about to be committed in that transaction will be forced out to disk. This is primarily done for security reasons; if this is not done (which would be the case if the disk is mounted with the mount option data=writeback), then any newly allocated blocks might still contain previous data belonging to some other file or user, and after a crash, accessing that file might result in a user seeing unitialized data that had previously belonged to another user (say, their e-mail or their p0rn), which would be a Bad Thing from a security perspective. However, this had the side effect of essentially guaranteeing that anything that had been written was guaranteed to be on disk after 5 seconds. (This is somewhat modified if you are running on batteries
and have enabled laptop mode, but we ll ignore that for the purposes of this discussion.) Since ext3 became the dominant filesystem for Linux, application writers and users have started depending on this, and so they become shocked and angry when their system locks up and they lose data even though POSIX never really made any such guarantee. This become especially noticeable on Ubuntu, which uses many proprietary, binary-only drivers, which caused some Ubuntu systems to become highly unreliable, especially for Alpha releases of Ubuntu Jaunty, with the net result that some Ubuntu users have become used to their machines regularly crashing. (I use bleeding edge kernels, and I don t see the kind of unreliability that apparently at least some Ubuntu users are seeing, which came as quite a surprise to me.) So what are the solutions to this? One thing is that the applications could simply be rewritten to properly use fsync() and fdatasync(). This is what is required by POSIX, if you want to be sure that data has gotten written to stable storage. Some folks have resisted this suggestions, on two grounds; first, that it s too hard to fix all of the applications out there, and second, that fsync() is too slow. This perception that fsync() is too slow was most recently caused by a problem with Firefox 3.0. As Mike Shaver put it:

On some rather common Linux configurations, especially using the ext3 filesystem in the data=ordered mode, calling fsync doesn t just flush out the data for the file it s called on, but rather on all the buffered data for that filesystem.

Fundamentally, the problem is caused by data=ordered mode. This problem can be avoided by mounting the filesystem using data=writeback or by using a filesystem that supports delayed allocation such as ext4. This is because if you have a small sqllite database which you are fsync(), and in another process you are writing a large 2 megabyte file, the 2 megabyte file won t be be allocated right away, and so the fsync operation will not force the dirty blocks of that 2 megabyte file to disk; since the blocks haven t been allocated yet, there is no security issue to worry about with the previous contents of newly allocated blocks if the system were to crash at that point. Another solution is a set of patches to ext4 that has been queued for 2.6.30 merge window. These three patches (with git id s bf1b69c0, f32b730a, and 8411e347) will cause a file to have any delayed allocation blocks to be allocated immediately when a file is replaced. This gets done for files which were truncated using ftruncate() or opened via O_TRUNC when the file is closed, and when a file is renamed on top of an existing file. This solves the most annoying set of problems where an existing file gets rewritten, and thanks to the delayed allocation semantics, that existing file gets replaced with a zero-length file. However, it will not solve the problem for newly created files, of course, which would have delayed allocation semantics. Yet another solution would be to mount ext4 volumes with the nodelalloc mount option. This will cause a significant performance hit, but apparently some Ubuntu users are happy using proprietary Nvidia drivers, even if it means that when they are done playing World of Goo, quitting the game causes the system to hang and they must hard-reset the system. For those users, it may be that nodelalloc is the right solution for now personally, I would consider that kind of system instability to be completely unacceptable, but I guess gamers have very different priorities than I do. A final solution which might not be that hard to implement would be a new mount option, data=alloc-on-commit. This would work much like data=ordered, with the additional constraint that all blocks that had delayed allocation would be allocated and forced out to disk before a commit takes place. This would probably give slightly better performance compared to mounting with nodelalloc, but it shares many of the disadvantages of nodelalloc, including making fsync() to be potentially very slow because it would force all dirty blocks to be forced out to disk once again. What s the best path forward? For now, what I would recommend to Ubuntu gamers whose systems crash all the time and who want to use ext4, to use the nodelalloc mount option. I haven t quantified what the performance penalty will be for this mode of operation, but the performance will be better than ext3, and at least this way they won t have to worry about files getting lost as a result of delayed allocation. Long term, application writers who are worried about files getting lost on an unclena shutdown really should use fsync. Modern filesystems are all going to be using delayed allocation, because of its inherent performance benefits, and whether you think the future belongs to ZFS, or btrfs, or XFS, or ext4 all of these filesystems used delayed allocation. What do you think? Do you think all of these filesystems have gotten things wrong, and delayed allocation is evil? Should I try to implement a data=alloc-on-commit mount option for ext4? Should we try to fix applications to properly use fsync() and fdatasync()? Related posts (automatically generated):

1. Don t fear the fsync! After reading the comments on my earlier post, Delayed allocation...
2. SSD s, Journaling, and noatime/relatime On occasion, you will see the advice that the ext3...
3. Ext4 is now the primary filesystem on my laptop Over the weekend, I converted my laptop to use the...

I've just uploaded Policy 3.8.1.0 with the first batch of post-lenny changes. There isn't anything that earth-shattering in this set, but I'm hoping that we'll be able to get a lot more done during the squeeze release cycle. Policy has well over a hundred open bugs, quite a few of which we should be able to resolve. I also uploaded Lintian 2.2.8, which is synchronized against Policy 3.8.1 and also reverts some changes around install-menu due to a reverted change in debhelper 7.2.5, and fixes diagnosis of diversions. The last few versions of Lintian have all been a little bit buggy due mostly to the current pace of change. Hopefully this version will have fewer unexpected problems. Here is the summary of normative Policy changes, taken from upgrading-checklist.txt.gz in the debian-policy package.

• Care should be taken when adding functionality to essential and such additions create an obligation to support that functionality in essential forever unless significant work is done.
• Changelog files must be encoded in UTF-8.
• Tighten some format requirements for changelog files from a should to a must.
• Remove alternative changelog formats. Debian only supports one changelog format for the Debian Archive.
• New nocheck option for DEB_BUILD_OPTIONS indicating any build-time test suite provided by the package should not be run.
• All control files must be encoded in UTF-8.
• debian/control allows comment lines starting with # with no preceding whitespace.
• Init scripts ending in .sh are not handled specially. They are not sourced and are not guaranteed to be run by /bin/sh regardless of the #! line. This brings Policy in line with the long-standing behavior of the init system in Debian.
• The start action of an init script must exit successfully and not start the daemon again if it's already running.
• /var/run and /var/lock may be mounted as temporary filesystems, and init scripts must therefore create any necessary subdirectories dynamically.
• /bin/sh scripts may assume that local can take multiple variable arguments and supports assignment.
• User mailboxes may be mode 600 and owned by the user rather than mode 660, owned by user, and group mail.

I m back from this years Cebit (well, yesterday already, just didn t write about it). Had a mostly good time that week, with our booth being small but visible to the visitors. In the first picture you see Rene (not our OpenOffice.org Rene :) ) setting up our second laptop, which we used whenever we had to show something to our visitors. The upper laptop with the presentation on it is my own. As it has a pretty big display we used it to continually have a presentation about Debian run. If there is no room for a projector this is a good alternative way. Later on we also had bb running, with customized texts for Cebit. As old as bb might be, it is still the best eye catcher you can have for a fair like Cebit. Nothing (well, maybe besides sex) works that good. We had a good week there, answering all sorts of questions. We had many people that knew nothing about Debian, Linux or even OpenSource and we had many that had heard about this other system already as well as people with very specific problems with various packages in Debian. There had been many people asking about the difference between Debian and Ubuntu and what we think about it, many told us they switched to Ubuntu for their Desktop but keep Debian on the Server because Ubuntu doesn t know what stability means, lots of others told us they switched back from Ubuntu because Debian is better. We also had those people that can not understand how Debian can exists if it doesn t sell anything and sometimes it turned out to be pretty hard to explain why we do what we do in a way they understand it. So all in all it was Cebit as usual. :) As one can read in the news, Cebit was much smaller than last year. 25% less exhibitors and 20% less visitors is very visible. Yet, besides a pretty tedious Sunday with the lowest visitor rate, the Debian booth wasn t much affected by it. We did have about all the usual people that visit us every year and for the rest, well, we are generic enough to attract attention from most visitors. This years booth was setup in a different way than the past years. As you can see on the pictures, it was a huge booth which had 15 different OpenSource projects on it, all arranged in isles of 3 tables each. We had BSD and Mozilla with us and the fsfe was our direct neighbor. Past years had an own booth per Project. Both ways have their advantages and disadvantages, and I think (and hope) next year will see a mixture of it. Large projects, which are known to attract lots of attention, like Debian, KDE or Mozilla, should have their own, seperated, booth. Something like 6 square meters each, placed as far away from each other as possible. Between those one or two larger areas having such tables arranged for smaller projects, like scribus or gosa will also give them a nice demonstration point and good access for visitors. Well, lets see how it will look like next year, it depends on many factors, the biggest being the money available to the OpenSource area.

When you order good into Switzerland, they will pass the customs office, where VAT and possibly import duties are levied. The customs office does not charge any duties if the total amount to be paid is less than 5 CHF Practically, given 7.6% VAT, this means that you can import 5 CHF / 0.076 = 65.79 CHF for free . As soon as you exceed this amount, it gets complicated. All of a sudden, the carrier charges you for the customs declaration it does in your name. What really blows my mind is that the fee to pay to the carrier (more on that later) actually plays into the whole equation. Let s assume you order goods for 100 CHF, which is well above the free import limit. You also paid 10 CHF for shipping. The carrier charges you 20 CHF to do the customs presentation for you. There, they determine that import duties are 5 CHF, and you end up paying: (100 + 10 + 20 + 5) * 107.6% = 145.26. That s fun, isn t it? I continue to be amazed by all the ways in which the government sucks money from your pockets. What s similarly baffling is how some carries deal with this. The Swiss Post (and their ally TNT) makes you pay taxes, duties, and fees on delivery, and their charges are the lowest when compared to the other carriers. With this approach, you have the option to refuse the package, if you re willing to deal with the vendor to get your money back. DHL is the only other carries I know who does things like the Swiss Post. However, due to their extraordinary crap performance and service, I could never recommend them to anyone. The three letters of DHL, a German company, apparently stand for das heisst langsam this means slow . None of the other carriers give you the option to accept the duties and their services. For instance, GLS delivers your parcel, and some weeks later you get an invoice, asking for the taxes and duties, and charging a fee of 45 CHF for their service, which you ve never really asked for and which you could not refuse. UPS, FedEx, and DPD fall into the same category, but their charges are lower. The one and only time I had used GLS to have something shipped to me, I called them up to complain about their invoice, and replied with an invoice of my own, in which I charged the same amount for opening and reading your letter and giving your call-centre employee the chance to talk to me directly . They got the message and dropped the invoice. The next step should be to play them off against each other: create some pencil art, write an invoice for half a million, and pay UPS to deliver the package to GLS headquarters. Attach a miniature audio transmitter to the invoice for added fun. NP: Goldie: Timeless

A cautionary tale: Verizon's DSL salespeople are completely incompetent. Several weeks ago, I ordered residential service. I was unable to get the prequalification app on their web site to work for my address, so I called in. I was told (1) I could get a modem for free, and (2) that I didn't need to do anything for the install except plug it in -- they'd just be turning something on at the CO, perhaps even before the date my service was scheduled to begin. These were both, of course, pure fiction. On the install date, while at work, I got a call from a tech who had shown up at my house to rewire something. Luckily, I was able to rendezvous with him later in the day. The service worked briefly and then went down. Over the next few days, I called in, was told to wait at home for 8 hours, no one showed up, I called in again, was told "oh, sorry, it's going to be tomorrow", waited all day the next day, no one showed up again... a week and two actual visits later, I finally had connectivity. I can't really blame them for whatever the line problem was, but to repeatedly treat my time as worthless by making me wait at home for an entire day just in case a tech can get there, even though it's highly unlikely they will, is just offensive. Then, I received my first bill and... was charged for the modem. Calling in again, I was told that the deal I was offered was for online orders only, and even if I specifically confirmed with the sales person that he could get me the same deal since the web site wouldn't let me complete an order (and I already had an old, power-sucking modem anyway), there was no way I could get a refund. So I got an RMA and planned to go shopping for a replacement (people are selling the same model on Craigslist for $10). The RMA came in an email which also told me that if I did not return the equipment within X days I would be charged $100. This is all utter bullshit. Do not use Verizon. If your only other option is Comcast, as it is here, just steal your neighbor's wireless or tether your 3G. They could have left me reasonably happy with the whole botched situation if they had provided a honest estimate of how long it would really take to get someone out (even if it was "days"), or given me access to the same information without having to go through a call center of people only trained to read a script about how to power-cycle your router or whatever. And if they actually apologized for a saleperson convincing me to sign up by lying about what I would be charged. But why would they? What are you going to do, switch to an ISP that treats you like a human being and doesn't fuck with your traffic? Ha. Ha ha ha. Good luck with that.

Yesterday, the choir of which I've been a member for these past few years went to the provincial tournament for the first time in its existence. For those of you unaware with what this is: the "provincial tournament" isn't a tournament in the traditional sense (which would have a winner etc); instead, you get to sing in front of an expert jury which will classify you in one of six categories; the category you're in then determines government subsidy your group gets. This is done for all nonprofessional music groups, and makes for a pretty fair way of deciding who gets what, IMO. Apart from that, it's of course also fun. Anyway, as said this was the first time our choir went to such a "tournament", but of course we weren't alone; apart from us, there were six other choirs, two of which were also part of the Ekerse C-koren: Cantando (the semi-professional choir of our group) and Cantilene (for those between the ages of 18 and 25 (or 16 and 25 for boys). And we did well. Cantilene (who did very well in the most recent edition of the 'Koor van het Jaar' contest) was categorized in the 'superieure afdeling' ('superior category'), with 75% in that category. Cantando did even better than that; they achieved a score of 93%, also in the superieure afdeling. This is the highest category in the whole system, so it deserves some special praise. Caljent , our choir, didn't quite reach that level yet; considering the fact that we've only existed about three years, however, that isn't very surprising. Yet, we did quite well; we were categorized in the 'Uitmuntendheid' category, with a score of 75% in that category. Uitmuntendheid (which would translate to something like 'greatness') is the third possible category, which puts us in the top half of the choirs of the province of Antwerp. Quite an achievement for what is, after all, a fairly new choir. Of course we're all very happy with that especially so our conductor, who was rather surprised when she heard the outcome... For those interested, one of the choir members had a camera with him, and he put the four pieces we sang up on youtube: Credo, Dostoyno Yest, Wals van kwart voor middernacht, and Hodie Christus Natus Est. There're also films of the other choirse, but you'll have to look for those yourself ;-P

/me for DPL Given that it is now widespread news, I guess it's time to blog about it: I'm running for DPL in this year elections, i.e.: zack for DPL!. Am I out of my mind?
Possibly It started as a joke at last DebConf, with my (naked or dressed ...) room mates spreading that rumor. Then the idea grows into me ... and now here I am. (IOW: I'm pretending I'm not entirely guilty of inflicting another platform to be read.) Apparently we are just two candidates: me and the "ticket" Steve-Luk. For the moment I've to spare a page where I will collect material about the election, and a couple of feelings:

• Sending the nomination mail is embarrassing: we are used to do only technical stuff in Debian, declaring our will of taking care for a while of our non-technical issues ain't easy.
• Preparing a platform is an intriguing experience. Whining about our problems is a thing. Trying to distill them in an organized list, and think about long term solutions is a totally different one. I suggest to try writing your own "platform" even if you are not candidate, and even when there are no DPL elections around (as Joey and HE did a while ago, IIRC, in the form of blog posts). It can be a rewarding exercise.

Version 1.0 of my platform has been submitted to the secretary, and is now available from my homepage. If you have questions, for me or the other candidates, just post them on -vote: campaigning has just started, I'm confident it will be fun! Update the platform is now available from the vote page

I ve been working as of late with psycopg2, the Python bindings for PostgreSQL. Documentation is a bit, erm, lacking. You can get the basic stuff working by reading Python s DBAPI-2.0 specification, since psycopg2 follows it, but (as I ve only recently realized) discovering the extra features that exist needs you to poke the source. The other day I was wanting to pass to the execute method of a cursor a sequence to fill the value list in a SQL IN statement. The DBAPI spec does not mention this usage at all, and a cursory question in #python told me it wasn t supported (required?) by the spec. It is, in any case, supported in psycopg2. You just need to

  import psycopg2.extensions

and it just works: you can pass tuples, and they ll be converted to a value list. If you want to be able to pass lists as well:

  from psycopg2.extensions import SQL_IN, register_adapter
  register_adapter(list, SQL_IN)

There is also a row factory in psycopg2.extras that offers a dictionary interface, instead of tuple-based. Finally, I have no idea why, but when I started with psycopg2, I came to get the idea that only named parameter substitution was supported (INSERT ... VALUES (%(var1)s, %(var2)s)), but simple %s substitution is also supported: just pass (of course) a tuple as the second parameter to execute, instead of a dictionary.

I've been working for a while on a new reprepro-based personal Debian repository, replacing the old one that was managed by debarchiver. I was going to hold off on switching until I uploaded every package that was in the old repository, but then I realized that was kind of silly and some of those packages need more attention. So I finally did the cutover today. For details on my personal repository, see my Debian packages page. The new repository is signed with a different key, not my personal key, so that I can automatically sign package uploads. You will now want to install the eyrie-keyring package from the repository or follow the alternative instructions on that page to import the key. The packages in that repository are mostly software I developed that's not of sufficient general interest to upload or packages that aren't up to Debian quality standards for one reason or another. I love reprepro. debarchiver did very well for me for quite a while, and I'm happy to have used it, but reprepro is just very nice. It's great to be able to use package pools and do automatic propagation of new uploads into multiple repositories, too. Now I just need to find time to do more software releases.

Adam Gehr of DePaul University's Finance Department had organized a panel session about R in Finance at the Midwest Finance Association's 58th Annual Meeting which is happening this week here in Chicago. I just posted my slides on my presentations page. The slides give a brief overview of R, the CRAN network and the by now over 1600 packages, mention the Finance Task View, briefly present four different packages (or package sets) and of course beat the drum for our upcoming R/Finance conference that will take place here in Chicago at the end of next month.

file locking basics. I was looking at how to lock a file. flock(2) system call allows locking a file with LOCK_EX and LOCK_SH; if used properly, parallel processing can be serialized. It is also possible to use 'flock' from shell scripts. I found the design of 'flock fd' variant clever, and nonobvious. It's possible to assign arbitrary file to arbitrary fd number in shell, and effect of flock lasts as long as the fd is open. Interesting.

After a long hiatus for various reasons, I finally had the chance to try my hand at loading a TeleMetrum board today! Sadly, I wasn't able to completely load any boards, because I somehow ended up with the wrong Digi-Key part numbers for 4 capacitors, one of which is a critical value. I placed an order for the missing parts and a few other bits we'll need eventually, hopefully they'll be here in a couple days and I can try to load a board with all the parts. Instead, what I did today was a a partial load of a board with the goal of testing the various power supplies. Seemed like a good idea, since I've had my share of odd problems with power supplies in the past. This also gave me a chance to try out my solder paste stencil, get some experience hand-placing the tiny 0402 passive parts, and an excuse to see if I could hit the solder reflow temperature profile adequately with my electric cooking skillet and IR non-contact thermometer. The good news is that it all worked right the first time! We're successfully charging a LiPo battery from USB power, and successfully making 3.3 volts for the electronics from that. The resistor divider designed to allow the LiPo battery voltage to be sampled by the CPU's analog to digital converter is also working fine. A few observations are in order, however. Getting the right amount of solder paste down on the board requires some finesse with the spreading blade and the stencil. I think the paste was a bit heavy around the CC1111 footprint, as I ended up with some bridged pads. Since I didn't load the actual CPU part, it could just be that not having something real there to wick up some solder meant there was enough to form the bridges. Don't know. The amount of solder on the ground pad in the middle of the chip looks good, though! The 0603 sized LED in the power supply circuit was the only part that didn't self-align correctly. I may have had it a bit too far to one side. The pads on the part have notches in the end, and it looks like one side of each is more or less centered, so I think surface tension did the best it could with what I gave it to work with. The LED soldered ok, isn't shorting to anything else, and clearly works fine... but I'd be happier if it were aligned better. Found and fixed a solder bridge between two pins on the surface-mount USB connector before first application of power. Could be further data that the paste layer was a bit too thick. On the other hand, I had to wiggle the connector around a bit to get it aligned correctly, so I may have smudged the paste into a bridge while doing that. I also note that there's some visible flux left on the surface of the solder, particularly on big paste areas like the feet of the USB connector. I suspect this means I didn't dwell long enough at either or both of the preheat or ultimate reflow temperatures. Since all the parts clearly soldered adequately to function, I'm still pretty happy... but since the rockets we fly experience violent accelerations during boost, this is something I'll play with on future reflows. All in all, a great way to spend an afternoon, and a big step forward for the TeleMetrum project!

On occasion, you will see the advice that the ext3 file system is not suitable for Solid State Disks (SSD s) due to the extra writes caused by journaling and so Linux users using SSD s should use ext2 instead. However, is this folk wisdom actually true? This weekend, I decided to measure exactly what the write overhead of journaling actually is in actual practice. For this experiment I used ext4, since I recently added a feature to track the amount of writes to the file system over its lifetime (to better gauge the wear and tear on an SSD). Ext4 also has the advantage that (starting in 2.6.29), it can support operations with and without a journal, allowing me to do a controlled experiment where I could manipulate only that one variable. The test workload I chose was a simple one:

• Clone a git repository containing a linux source tree
• Compile the linux source tree using make -j2
• Remove the object files by running make clean

For the first test, I ran the test using no special mount options, and the only difference being the presence or absence of the has_journal feature. (That is, the first file system was created using mke2fs -t ext4 /dev/closure/testext4, while the second file system was created using mke2fs -t ext4 -O ^has_journal /dev/closure/testext4.)

Amount of data written (in megabytes) on an ext4 filesystem

Operation	with journal	w/o journal	percent change
git clone	367.7	353.0	4.00%
make	231.1	203.4	12.0%
make clean	14.6	7.7	47.3%

What the results show is that metadata-heavy workloads, such as make clean, do result in almost twice the amount data written to disk. This is to be expected, since all changes to metadata blocks are first written to the journal and the journal transaction committed before the metadata is written to their final location on disk. However, for more common workloads where we are writing data as well as modifying filesystem metadata blocks, the difference is much smaller: 4% for the git clone, and 12% for the actual kernel compile. The noatime mount option Can we do better? Yes, if we mount the file system using the noatime mount option:

Amount of data written (in megabytes) on an ext4 filesystem mounted with noatime

Operation	with journal	w/o journal	percent change
git clone	367.0	353.0	3.81%
make	207.6	199.4	3.95%
make clean	6.45	3.73	42.17%

This reduces the extra cost of the journal in the git clone and make steps to be just under 4%. What this shows is that most of the extra meta-data cost without the noatime mount option was caused by update to the last update time for kernel source files and directories. The relatime mount option There is a newer alternative to the noatime mount option, relatime. The relatime mount option updates the last access time of a file only if the last modified or last inode changed time is newer than the last accessed time. This allows programs to be able to determine whether a file has been read size it was last modified. The usual (actually, only) example that is given of such an application is the mutt mail-reader, which uses the last accessed time to determine if new mail has been delivered to Unix mail spool files. Unfortunately, relatime is not free. As you can see below, it has roughly double the overhead of noatime (but roughly half the overhead of using the standard Posix atime semantics):

Amount of data written (in megabytes) on an ext4 filesystem mounted with relatime

Operation	with journal	w/o journal	percent change
git clone	366.6	353.0	3.71%
make	216.8	203.7	6.04%
make clean	13.34	6.97	45.75%

Personally, I don t think relatime is worth it. There are other ways of working around the issue with mutt for example, you can use Maildir-style mailboxes, or you can use mutt s check_mbox_size option. If the goal is to reduce unnecessary disk writes, I would mount my file systems using noatime, and use other workarounds as necessary. Alternatively, you can use chattr +A to set the noatime flag on all files and directories where you don t want noatime semantics, and then clear the flag for the Unix mbox files where you care about the atime updates. Since the noatime flag is inherited by default, you can get this behaviour by setting running chattr +A /mntpt right after the filesystem is first created and mounted; all files and directories created in that file system will have the noatime file inherited. Comparing ext3 and ext2 filesystems

Amount of data written (in megabytes) on an ext3 and ext2 filesystem

Operation	ext3	ext2	percent change
git clone	374.6	357.2	4.64%
make	230.9	204.4	11.48%
make clean	14.56	6.54	55.08%

Finally, just to round things out, I tried the same experiment using the ext3 and ext2 file systems. The difference between these results and the ones involving ext4 are the result of the fact that ext2 does not have the directory index feature (aka htree support), and both ext2 and ext3 do not have extents support, but rather use the less efficient indirect block scheme. The ext2 and ext3 allocators are also someone different from each other, and from ext4. Still, the results are substantially similar with the first set of Posix-compliant atime update numbers (I didn t bother to do noatime and relatime benchmark runs with ext2 and ext3, but I expect the results would be similar.) Conclusion So given all of this, where did the common folk wisdom that ext3 was not suitable for SSD s come from? Some of it may have been from people worrying too much about extreme workloads such as make clean ; but while doubling the write load sounds bad, going from 4MB to 7MB worth of writes isn t that much compared to the write load of actually doing the kernel compile or populating the kernel source tree. No, the problem was that first generation SSD s had a very bad problem with what has been called the write amplification effect , where a 4k write might cause a 128k region of the SSD to be erased and rewritten. In addition in order to provide safety against system crashes, ext3 has more synchronous write operations that is where ext3 waits for the write operation to be complete before moving on, and this caused a very pronounced and noticeable stuttering effect which was fairly annoying to users. However, the next generation of SSD s, such as Intel s X25-M SSD, have worked around the write amplification affect. What else have we learned? First of all, for normal workloads that include data writes, the overhead from journaling is actually relatively small (between 4 and 12%, depending on the workload). Further, than much of this overhead can be reduced by enabling the noatime option, with relatime providing some benefit, but ultimately if the goal is to reduce your file system s write load, especially where an SSD is involved, I would strongly recommend the use of noatime over relatime. Related posts (automatically generated):

1. Should Filesystems Be Optimized for SSD s? In one of the comments to my last blog entry,...
2. Aligning filesystems to an SSD s erase block size I recently purchased a new toy, an Intel X25-M SSD,...
3. Fast ext4 fsck times This wasn t one of the things we were explicitly engineering...

Ingredients:

• some lame DNS server
• logcheck
• spamassassin

The last couple of days I've been plagued by some DNS errors that kept showing up in the logcheck mails for my home server which I was busy migrating from one box to another, doing an upgrade from etch/i386 to lenny/amd64 at the same time. So, plenty of stuff going on to confuse the issue. I kept getting the following messages every hour (anonymized):

named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'somedomain.org/NS/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.mmm#53
named: connection refused resolving 'ns1.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53
named: connection refused resolving 'ns2.somedomain.org/AAAA/IN': xxx.yyy.zzz.nnn#53

The times were fairly regular: once just before the hour, most 2 minutes after. I fetch mail at around that time, but also at other times, so possible but unlikely. The 2 minutes after was the first real clue: some cron job maybe? After disabling logcheck the messages no longer appeared in the log. Enable it again, and they were back. Additional confusion was caused by the fact that the domain had "debian" in its name, but it was somewhere obscure. So why was logcheck causing a lookup for that domain? This did confuse me enough to waste some time looking for some silly weird (default) configuration problem in some package. Enter spamassassin. Apparently that was parsing the message body, recognized "somedomain.org" as a host name, and proceded to do a DNS lookup as validity check. So we have the following loop, started off by something causing an initial DNS lookup for the domain, which fails and gets logged:

• logcheck reports the failure during its next check
• spamassassin processes the logcheck mail, spots the domain name and does a new set of lookups, which fail and get logged
• logcheck reports the failures during its next check
• ...

Duh. I remember struggling with probably the same problem a couple of years ago, but then it was a lot more severe: masses of repeating DNS errors for obscure domains. At that time I failed to get to the bottom of it and ended up just ignoring the errors by adding the following option in my bind9 configuration:

logging  
    category lame-servers   null;  ;
 ;

Anyway, now I just no longer pass logcheck mails through spamassassin. (Although filtering out these DNS errors in bind9 can be perfectly valid.)