I wrote last month about my failed first attempt to achieve a "level 3" high power rocketry certification. Last week, I learned that one of the commercial altimeters I flew for control of the ejection sequence has a firmware bug, which can cause premature ejection of the main parachute on flights above 10k feet! So it may in fact be the case that everything I did was perfect and I was simply the victim of a bug in software... how ironic! Yesterday, on my second certification attempt, I was successful! The rocket I flew this time was based on a Polecat Aerospace ten inch Goblin kit. I incorporated several modifications in my build, including additional fiberglass and carbon fiber reinforcement, and a payload bay in the nose cone. Once again, my wife Karen graced me with a parachute sewn from the Team Vatsaas design, this time a slightly larger one in burgundy and black rip-stop nylon. She incorporated several design tweaks based on her experiences sewing the first one which we'll try to write up at some point, that allowed her to sew this one in less time. The motor selected was again an Aerotech M1297W reload. The launch went perfectly, and the ascent was impressive. At apogee, the nose cone separated as planned and the drogue parachute deployed. Unfortunately, when the backup charge fired two seconds later, the main parachute also deployed. That wasn't intended... the main was supposed to deploy much lower, at 1300 feet. Luckily, the winds were light enough that the rocket touched down only about a mile downrange from the launch rail, well within the waiver distance, and was easily recovered without damage. There are three things I'll consider changing before flying this rocket again. First, the chute size calculator used to design the main parachute seems way off. The actual descent rate was around 32 feet per second, while our goal was 20. Other than that, the parachute performed admirably! Second, the premature deployment of the main at apogee could be cured by replacing the paper taped over that parachute bay with another piston capping the bay and held in with a pair of shear pins. Finally, the hard touch down snapped some of the nylon wire ties holding the batteries in place in the avionics bay. While no damage was done, and reducing the descent rate might prevent it happening again, improving the battery mounting would be an easy upgrade. For more information, see my project page for this rocket. James Russell took some pictures at the launch, including a great one of the rocket under boost just leaving the launch rail. And Jeff Lane captured video of the ascent and descent. It was a great flight and a great day, and represents a major milestone in our hobby rocket activities!

Today, the NYT has an interesting piece on the declining numbers of women in the field of computer science. Ultimately the article presents a bleak picture but does not give a firm sense of why this is so (I think because it is so hard to answer). I do agree that if girls are not hopping on the computer at a young age and are not using it as a tool for production (they do use it a social tool), they are always going to have trouble catching up to men. Many CS majors, not to mention most hackers, develop quite intimate relations with the computer from a young age and thus have a level of comfort and expertise they have is nothing short of astonishing. If women are not developing that expertise as children but only much later in life, there will always be two classes of citizens in computing. Men, in other words, are native speakers, while women learn computing as a second language. The comment I agree with less is the following:

Ms. Cassell identifies another explanation for the drop in interest, which is linked to the pejorative figure of the nerd or geek. She said that this school of thought was: Girls and young women don t want to be that person.

It seems to be that in fact in the 80s and prior to that, the only word in town to describe computer folks were nerds. But geek arose to take its place and in part to take away the pejorative sting. Geek is cool. Nerd is not. And geek is now associated with all sorts of computer cultures in a way that it was not before. So it seems to be that more than ever, there is a positive geeky association with computing so in fact this would open the doors to more people than before.

I played with an Eee for a bit last time I was in Boston, culminating in a patch to make the eeepc-laptop driver use standard interfaces rather than just having random files in /sys that people need to write custom scripts to use. The world became a better place.

However. Asus implemented the rfkill control on the Eee in a slightly odd way. Disabling the wifi actually causes the entire card to drop off the bus, similar to how Bluetooth is normally handled. The difference is that the Bluetooth dongles are almost exclusively USB, while the Eee's wifi is PCI. Linux supports hotplugging of PCI devices, but nothing seemed to work out of the box on the Eee. Another case of this was the SD reader in the Acer Aspire One. Unless a card was present in the slot during boot, it simply wouldn't appear on the PCI bus. It turned out that Acer have implemented things in such a way that removing the card results in the entire chip being unplugged. This was when I started looking more closely into how this functionality is implemented.

The two common cases of PCI hotplug are native PCIe hotplug and ACPI mediated hotplug. In the former case, the chipset generates an interrupt when a hotplug event occurs and the OS then rescans the bus. This is a mildly complicated operation, requiring enabling the slot, checking whether there's a card there, powering the card and all its functions up, waiting for the PCIe link to settle and then announcing the new PCI device to the rest of the OS. ACPI-mediated hotplugging puts more of the load on the firmware rather than the OS - the hotplug event generates a notify message that is caught by the ACPI interpreter in the OS, allowing the OS to check for device presence by calling another ACPI method. If the device is present it's then a simple matter of telling the PCI layer about it.

Native PCIe hotplug has the advantage that there's much less vendor code involved. ACPI is still involved to an extent - an _OSC method on the PCIe bridge is called to allow the OS to tell the firmware that it supports handling hotplug events. This allows the firmware to stop sending any ACPI notifications. ACPI hotplugging requires more support in the firmware, but can work for PCI as well as PCIe.

The general approach taken to getting the Eee's wifi hotplugging to work has been to load the pciehp driver with the pciehp_force=1 argument. This tells the driver to listen for hotplugging events even when there's no _OSC method to tell the firmware that the OS is handling things now. Since the hardware will generate the event anyway, things work. However, this is non-ideal. Some hardware exists where ACPI hotplugging will work, but due to quirks in the hardware design native PCIe hotplugging control will fail. This has been handled in their firmware by having the _OSC method fail, signalling to the pciehp driver that it shouldn't bind to the port. Using pciehp_force overrides that, leading to a situation where hardware could potentially be removed from a port that's powered up. Unfortunate.

My first approach was to add a new argument to pciehp called pciehp_passive. This would indicate to the pciehp driver that it should only listen for notifications from the hardware. User-triggered events would not be supported, avoiding the situation where anyone could remove the card by accident. This worked on my test machine (an Eee 901 somewhere in Ottawa, since I don't actually have one myself...) but was reported to work less well on a 700. Since the 700 didn't claim to have any support for power control, the code was forced to wait a second on every operation to see whether the link powered up or not. This resulted in long pauses during boot and suspend/resume operations.

The final issue that convinced me that this was the wrong approach was reading a document on Microsoft's site on how PCIe hotplugging is implemented in Windows. It turns out that XP doesn't support native PCIe hotplugging at all - that feature was added in Vista. Both the Eee and the Aspire One are available with XP, but things work there. So PCIe native hotplugging was clearly not the right answer. Time to look further.

Armed with a disassembly of the Aspire One's DSDT, I figured out why the ACPI hotplug driver didn't work on it. The first thing the driver does is walk the list of ACPI devices, looking for any that are removable. That was being implemented by looking for an _EJ0 method. _EJ0 indicates that the device can be ejected under the control of the OS. The Aspire One doesn't have an _EJ0 method on its SD readers. However, it did have an _RMV method. This can be used to indicate that a device is removable but not ejectable - that is, the device can be removed (by physically pulling it out or by the hardware taking it away itself), but there's no standard way to ask the OS to logically disconnect it. A quick patch to acpiphp later and the Aspire One now worked without any forcing or spec contravention. This also has the nice side effect of making expresscard hotplug work on a bunch of machines where it otherwise wouldn't.

But back to the Eee. acpiphp still wasn't binding, and a closer examination revealed why. There's nothing to indicate that the Eee's ports are hotpluggable, and there's no topological data in the ACPI tables that ties the wifi function to the PCIe root bridges. However, the Eee firmware was sending an ACPI notification on wifi hotplug. But it was only sending this to the PCIe root bridges, and there's no way to then tell which device had potentially appeared or vanished.

In the end, I gave up on trying to solve this generically. Instead I've got a patch that implements the hotplugging entirely in eeepc-laptop. In an ideal world nobody else will have implemented this in the same way as Asus and we can all be happy.

One month with Emacs and counting - Part 2 A while ago I've blogged about my switch from Vim to Emacs, promising a blog post series, quite a mouthful nevertheless, it's time to continue the series. The first part was about why I think that nowadays Emacs is ready to be switching to. This second part is about flawed Vim design choices which substantially contributed to my choice. (Of course all this post is under a IMO-/rant-disclaimer, it can't be otherwise, so take it cum grano salis) Flawed Vim design choices Vim started as a wonderful editor, following the path of Vi, which in turn has got right a good deal of design choices. One is the often debated modality. In spite of being a HCI PITA in general, modality is not a problem at all for software you are using every single day (and the editor is the 2nd program I use the most, with the terminal being the 1st), because you learn it as a habit. In the case of Vim, modality is good because it let you use powerful yet simple motions in command mode, which can be easily combined with operator-pending commands. This can be considered yet another incarnation of the UNIX philosophy, casted inside the editor itself. ... which brings us to the good old joke which describes Emacs as a great operating system, lacking only a decent editor. In fact, Vim is no longer the nice Unix philosophy player which it used to be, and Emacs plays much more nicely with external tools than Vim. As arguments, let's consider a few (not so) recent Vim evolutions:

• Spell checking. Starting with version 7, Vim has got support for on-the-fly spell-checking of text buffers. It is a super-nice feature, which I've advertised in the past, because spell checking is needed in many tasks we daily do with editors (mail/doc writing, code commenting), and having a single interface to do it comes handy. Unfortunately, how that is implemented in Vim is definitely not along the lines of the UNIX philosophy. We have countless different FOOspell implementations whereas Vim implemented yet another one, without relying on any pre-existing implementation. Even more so, the file format of spell files is basically ad-hoc, and not sharable with other tools. This is already a PITA for packaging (let's add to Debian another 20 binary packages vim-spellfiles-FOO), but is even worst than that: the spell files must be generated in ad-hoc ways which are seriously prone to failures (guess why in Debian we don't have yet the vim-spellfiles-FOO packages which used to exist in experimental a long time ago?). What would have been the UNIX-friendly way of achieving the same result? Well, obviously spawning FOOspell and interact with its process from the editor. Keep that in mind.
• Internal grep. A feature of Vim I've always loved is :grep (yet another instance of quick-fixing, which is also used for jumping to compile-time errors). In its original incarnation it was as simple as invoking grep and parse its output. KISS. Then, Vim started to have a problem: grep -r can take ages and in the meantime the editor was stuck. Hence, starting with Vim 7, vimgrep has born: an internal re-implementation of grep. OK, it is more portable (but how many of us do care? on how many system you use you are missing grep? damn, I used to install it also on Windoze!), and the editor retain control. Control that is exploited just to show a nice progress bar of what is being scanned ..., while the editor is still stuck. Well, to me it looks like the problem would have been solved much more nicely by adding the ability to interact with an external process (deja vu?). That process can then be grep -r. Full stop.
• Lack of debugging support (on purpose). Long standing missing feature of Vim: you can not "easily" drive a debugger from Vim, because a firm choice in Vim is to ... (OK, I confess, I'm starting to beat a dead horse) ... not have support for interaction with external processes. I found such a choice quite dumb: to me it is evident that it induces a trade-off between being feature-complete wrt programmer needs (as available external tools get available to face those needs) and not "re-implementing the wheel" inside the editor. Going back to the lack of debugging support in Vim, that has spawned projects like GdbVim, Clewn, Bram's Agide, and my own tiny teeny incarnation specific to ocamldebug called WOWcamldebug. Every single project of that list acts as an intermediary between Vim and an external process (a debugger of some sort), whereas that could have been implemented by an editor plugin, provided that the editor offer, guess what, support for interaction with external processes.
• Top-level. Similar to the debugging support issue, there is the interpreter issue, namely how to evaluate "phrases" of your interpreted language of choice directly from the editor. Nowadays a lot of languages have the so called top-levels which evaluate language phrases interactively (Python, Ruby, OCaml, many Lisps, ...) and show evaluation results. That ability can speed up testing considerably, but can become cumbersome to use for large code snippets without editor support (good luck with copy and paste). Yes, Vim has support for that, but the implementation choice is close to be ridicolous: only if you have linked Vim together the interpreted for a given language, then you will be able to interactive evaluate phrases of that language. What if you program in Python, Ruby, and Perl? Well, you need to link all of them. What if you also program in OCaml? Then you're screwed, because Vim currently does not support linking with it. (Yes, I know that there are other reasons for doing that linking, they are discussed below.) Do I need to tell you which feature will be enough to address top-level support? interaction with extern.. OK, I'll drop it, you know that already.

Well, I acknowledge that Emacs got this choice right, and that it's an important one: Emacs supports interaction with external processes, and plugin authors have been happy to exploit it to create really cool stuff (sticking to the UNIX philosophy). Please welcome: Flyspell, M-x grep (shameless smartass plug: same number of keystroke of :grep), Grand Unified Debugger, a sampling of language-specific major modes with top-level support. (But I recommend having a look at least once at the coolness of other stuff built-on top of external process support, like Flymake, mentioned by dancer not a long ago.)

• Its own scripting language. I've bored you enough with the external process topic, hence here is the second one: the saga of Vim and its scripting language, i.e., the programming language using which you can customize the editor. Note that the choice of scripting language impacts on 2 different targets: final users in need of fire and forget automations (when they start to get to complex to be implemented on top of, say, macros), and developer of editor extensions such as addons. The impact of scripting language on extension developer is crucial for the evolution of an editor aiming at being powerful. If the scripting language and the API towards the editor is flexible enough, then a lot of cool extensions and features will be developed by third party authors, otherwise the burden stays on editor upstream author.
  1. In its phase 1 (my term, Vim < 7.0) Vim pretended that it needed only a minimal scripting language (Vim script) and that everything else can be developed using external programming languages, linked in the editor. As a consequence Vim script was just a tiny teeny procedural language with editor interfacing capabilities, most notably lacking any "decent" data structures (no lists, no dictionaries, ...). In that phase, if you were a user it was fair enough. You just had to chose your external programming language (at compile time ...), and you were able to write your quick hacks with it. But if you were willing to develop an extension you were screwed, because you had either to impose your external programming language to the final user (that is what gave born to absurdities like vim-full, look at the deps!) or to enjoy a good deal of masochism to explode your 30 lines of Python extension into 200 lines of Vim script to make your Vim extension more "portable".
  2. In its phase 2 (Vim >= 7.0) it has been realized that something were wrong in the initial choices about Vim script, and data structures (lists, dictionaries, function references!) have been added to it, relieving a bit the pain of programming portable extensions. Nevertheless, and probably due to how it started, Vim script is far from being a programming language one can enjoy programming in. Consider for example the standard library of the language. It started as a random collection of unrelated functions being added on a "as-needed" basis, ... and continued along that path! Now the API reference lists together totally unrelated functions, organized inconsistently, and making really painful finding what you need (assuming it exists).

Even in this respect, I acknowledge that Emacs got this right. It has chosen a single, now full-fledged, programming language (Elisp), which is offered both to the final user (fact: the average Emacs user knows much better how to program her own editor than the average Vim user) and to extensions developer. The standard library of the language is organized with a consistent naming (even though not always intuitive) and documented in an organized manner. You might not like the language, but at least is has been taken seriously and managed as such. Finally, being a Lisp dialect, you might need what you learn with Elisp elsewhere. Vim script is too committing: it is useful only inside Vim. I believe that contributed significantly in its scarce diffusion among Vim users. The fact that Elisp was born together with Emacs and that it is also used internally for the editor implementation is not relevant here: I do care about what is offered to me as an user and as an extension developer. What I see on the Emacs side in this respect is much better that what I see on the Vim side. Moreover, from the point of view of the users, is the difference between what is inside the editor and what is outside (i.e., the extensions) that relevant? I think it is not, and that brings me to the final Vim choice which I consider flawed and which has the potential of seriously limit its future evolutions:

• Editor as a distribution. Emacs is managed as a intermediate software distribution layer, similarly to what happens in other softwares where many third party authors cooperates (e.g., TeXLive). The distribution architecture has advantages and contributes to the longevity and potentialities of a software project. What does this mean concretely in the Vim vs Emacs dispute? For example it means that Emacs get more feature-complete each passing release, still preserving uniformity in documentation and keybindings. On the contrary Vim sports many addons, which are very likely to step on each other feet when enabled simultaneously. Actually, that is one of the reason which brought me to the creation of vim-addons: you cannot enable all the content of vim-scripts together due to various kind of conflicts. As we know well in Debian, distributions have a fundamental role in blending together lightly-coupled software components, that's role is missing in Vim evolutionary path and I find that worrisome.

Where to from here in this blog post series? For sure some tips on how to migrate for hard-core Vim users, then we'll see ... If you did enjoy the read please let me know commenting in the discussion page (as you did in the last post, thanks!).

I hadn’t noticed until today when Jordi Guti rrez Hermoso pointed it out to me: We already have AGPL’ed stuff in Debian, and not only in sid but it will also be shipped in Lenny. We have discussed that license in debian-legal during August and September this year, and in November its usage in client code has also being clarified when the FSF updated their FAQ to address that question, but we never reached a definitive conclussion. I also filed a bug against FTP Master asking them to clarify the situation of AGPL licensed stuff in Debian, which still remains unanswered. In any case, debian-legal is just a mailing list for discussion, and does not have the capacity to take any final decisions (that’s FTP Master’s task) but it would have been nice if someone had pointed it out there.

I’ve been a member of April, the french association for promotion and defense of Free Software, for a bit more than a year, and I often regret not becoming a member earlier. (I was feeling so guilty and shameful about not being a member that I actually postponed becoming a member.) Stop feeling guilty and shameful, become an April member today! Why Is becoming an April member so important?

• Clearly, April doesn’t address the same problems as your local LUG. April is a country-wide organization, and it works on country-wide problems. It’s the only group able to work on such problems at this scale (I’m not sure of the situation in other countries, but I think CCC shares a similar role in Germany for example).
• Each time I talk to people really involved in April (which I’m not), I’m amazed by how powerful they have become. They are able to talk to french or european deputies or ministers’ cabinets, and are considered important. They are doing a fantastic job spreading what matters to us to legislative and executive powers in France and Europe.

Some of the things they worked on recently (from the top of my head):

• Lobbying on :
  • OOXML
  • General announcements about politics (Plan France Num rique 2012, aka Plan Besson).
  • European telecom package and HADOPI law (french graduated response) law, through Quadrature du net. (OK, it doesn’t have anything to do with April, but most of the people involved in Quadrature du Net are also involved in April :-)
  • vente li e : the fact that it’s not possible to buy a computer without a Windows license. It’s illegal in French law, but still the de facto situation almost everywhere.
• Organization of a campaign where candidates to elections in France where asked questions, or asked to sign a declaration about Free Software. In 2007, 8 out of the 12 candidates of the french presidential election answered April’s questions.

So, really, become a member today. It’s only 10 EUR, and you already know they will be well used. April is trying to reach 5000 members by the end of 2008. (Apparently, if you use that address, April will now that you came from me. No benefit for me at all.)

There have been a number of improvements to the Debian installer on Network Attached Storage (NAS) devices in the last few months and they are available to users now that RC1 has been released. We have added support for Marvell's Orion platform. Orion is a system on a chip (SoC) that integrates an ARM CPU, Ethernet, SATA, USB, and other functionality in one chip. The majority of NAS devices that have come out within the last two years are based on Orion chips, so it's exciting to have support for this significant platform in Debian. Specifically, we support the following devices based on Orion: QNAP Turbo Station (TS-109, TS-209, TS-409), HP Media Vault mv2120, and Buffalo Kurobox Pro (with more to come later). Here are some other newsworthy improvements and changes:

• When a static IP address is set in the original system that is the default from the vendor, this address will now be used. In the past, the installer would use DHCP in this case but users found this confusing. This affects the Linksys NSLU2 and Thecus N2100.
• If a static IP configuration is used in the original system but this configuration is incomplete (i.e. IP address or DNS are missing), then the installer will use DHCP (instead of hanging during start). This affects the Linksys NSLU2 and Thecus N2100.
• On the Linksys NSLU2 and Cobalt devices, all required installer modules are automatically installed now and no longer have to be selected manually from a list.
• initramfs will use MODULES=dep rather than MODULES=most. In other words, only modules that are actually needed on your system will be included in the ramdisk.
• /etc/default/rcS will be configured on supported NAS devices to contain FSCKFIX=y, so e2fsck will attempt to repair the filesystem without prompting the user for input. This is useful on NAS devices since they are usually headless.

Finally, the armel port is recommended for new installations now. Debian lenny has two different and incompatible ARM ports: the old ABI port (arm) and the new EABI port (armel). Debian lenny is the last release with support for the arm port and future releases will only support the armel port. It's therefore recommended to use armel for new installations of lenny.

It’s close! In the township where we live, Barack Obama defeated John McCain by 15 votes! I guess I should mention that the victory margin was 166 to 151. So it’s not like it was 15 votes out of millions. In all, 333 people in our township cast ballots, or about a third of the total population of our township. Just to give you a sense of scale, there are an average of 29 people per square mile out here. And the nutty jail expansion was defeated 3:1. Our county commissioners will just have to figure out some other way to house the county’s prison population (around six inmates) for awhile longer.

OCaml Batteries alpha 2: Debian packages Out of popular request (wow, people on #ocaml@freenode are very excited about the Batteries thingy, good!), I've prepared Debian packages for OCaml Batteries Included, version alpha 2 (which does include the support for transparent gzip-ed channels). Packaging shameless plug: packaging this version has been a nice showcase for TopGit as now I've a couple of inter-dependent branches. TopGit has indeed rocked my world, yay. (Well, I also have to admit to myself that this tiny bit of TopGit mention wont assolve me from the need to advertiseblog about TopGit, ... eventually!) Packages have been uploaded to experimental. As usually I build/upload for amd64, but experimental in Debian is auto-built so eventually the will show up also for other architectures, of course including i386. If you want to take them directly from experimental you have to add this line to your /etc/apt/sources.list:

    deb http://ftp.de.debian.org/debian/ experimental main non-free contrib

and then install requesting explicitly experimental (for some time now, experimental is treated specially, and it won't be used by APT unless explicitly asked for):

    apt-get install -t experimental ocaml-batteries-included

The Debian version (roughly) corresponding to alpha 2 is 0.20081112+gitBB342A7-1. Alternatively, you can grab the packages from my people.d.o space. Since people were interested I've made available both amd64 and i386 builds there.

After seeing Five to launch on Freesat over on the DTG website, I tried rescanning 28e and a channel called 6335 (or similar) appeared in the channel list but it had a FIVE logo in the corner. Maybe those DOGs do serve some purpose sometimes - but it would be better if they actually set the channel name correctly. So, if you have a free-to-air satellite set pointed at 28e (which most UK dishes are), then you now have Five. If you have Freesat, you’ve another week before it appears on your EPG. It’s better to be free-to-air. I’ve not seen much of five since leaving Norwich about a decade ago: North End King’s Lynn is “fringe” reception for even the main channels, while neither the Kewstoke nor Cardiff transmitters broadcast it yet. It looks like it’s changed quite a lot. Anyway, I’ve updated the alt.satellite.tv.europe FAQ to move five into the list of FTA channels. Anyone know about getting the Freesat EPG on MythTV yet?

I got this nice message (and some others) in my log today:
smartd: Device: /dev/sdd, 2 Currently unreadable (pending) sectors After running a smartctl -t long /dev/sdd smartctl now reports:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   089   089   000    Old_age   Always       -       8719
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   200   200   000    Old_age   Always       -       2
198 Offline_Uncorrectable   0x0010   200   200   000    Old_age   Offline      -       0
[...]
SMART Error Log Version: 1
ATA Error Count: 10 (device log contains only the most recent five errors)
[...]
Error 10 occurred at disk power-on lifetime: 8692 hours (362 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.
  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 78 e5 89 e0  Error: UNC 8 sectors at LBA = 0x0089e578 = 9037176
  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 08 78 e5 89 2f 00  45d+06:36:27.880  READ DMA EXT
  25 00 08 70 e5 89 2f 00  45d+06:36:27.798  READ DMA EXT
  25 00 08 68 e5 89 2f 00  45d+06:36:27.308  READ DMA EXT
  25 00 08 e0 77 e5 35 00  45d+06:36:27.295  READ DMA EXT
  25 00 08 60 e5 89 2f 00  45d+06:36:26.653  READ DMA EXT
[...]
Error 9 occurred at disk power-on lifetime: 8692 hours (362 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.
  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 60 e5 89 e0  Error: UNC 8 sectors at LBA = 0x0089e560 = 9037152
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%      8715         797566305
# 2  Extended offline    Completed without error       00%      8198         -

Notice that 197 Current_Pending_Sector indicates that there are 2 sectors with a problem. Note that the SMART error log shows LBA sectors 9037176 (0x89e578) and 9037152 (0x89e560) but it's limited to 0xffffff. The self test logs shows us the proper LBA error at 797566305 (0x2f89e561) which has a 0x2f in front, and also shows us that it's the second sector in the block of 8 we tried to read. The kernel log also shows:

ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata4.00: (BMDMA2 stat 0xc0009)
ata4.00: cmd 25/00:08:60:e5:89/00:00:2f:00:00/e0 tag 0 cdb 0x0 data 4096 in
         res 51/40:00:61:e5:89/00:00:2f:00:00/f0 Emask 0x9 (media error)

So I've tried to reproduce the error with:

# dd if=/dev/sdd of=797566305 skip=797566305 bs=512 count=1 iflag=direct
1+0 records in
1+0 records out
512 bytes (512 B) copied, 1.27727 s, 0.4 kB/s

Notice that it can actually read that block, it just seems to take a while. And the error log now contains:

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 01 61 e5 89 e0  Error: UNC 1 sectors at LBA = 0x0089e561 = 9037153

Because it's part of a RAID device I could just let the whole disk resync, but I was a little curious which files were in it. So I found this document that explains on how to find which files has the problem. But it only has examples covering ext2/ext3 on a partition and LVM and nothing about RAID device. So it looks like someone using RAID 0 will have a hard time finding the file that's having a problem. So following the document we start with:

# fdisk -lu /dev/sdd
Disk /dev/sdd: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x000578f9
   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1   *          63      385559      192748+  fd  Linux raid autodetect
/dev/sdd2       972864270   976768064     1951897+  fd  Linux raid autodetect
/dev/sdd3          385560   972864269   486239355   fd  Linux raid autodetect
Partition table entries are not in disk order

In /proc/mdstat we see:

md1 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1]
      1458717696 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

So sdd3 is part of md1 and I have an ext3 filesytsem on that. Tune2fs -l grep Block gives:

Block count:              364679424
Block size:               4096
Blocks per group:         32768

mdadm tells me:

# mdadm -QE /dev/sdd3
/dev/sdd3:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 747b071a:a70f723b:c3a9ed44:15845cbf
  Creation Time : Mon Nov 12 20:13:51 2007
     Raid Level : raid5
  Used Dev Size : 486239232 (463.71 GiB 497.91 GB)
     Array Size : 1458717696 (1391.14 GiB 1493.73 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 1
    Update Time : Mon Nov 10 10:57:28 2008
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 4b3147f4 - correct
         Events : 78
         Layout : left-symmetric
     Chunk Size : 64K
      Number   Major   Minor   RaidDevice State
this     3       8       51        3      active sync   /dev/sdd3
   0     0       8        3        0      active sync   /dev/sda3
   1     1       8       19        1      active sync   /dev/sdb3
   2     2       8       35        2      active sync   /dev/sdc3
   3     3       8       51        3      active sync   /dev/sdd3

I've been told that 0.90 stores the superblock at the end of the device, so that makes it a little easier. And this seems to agree:

# cat /sys/block/md1/md/dev-sdd3/offset
0

We have a left-symmetric layout, so in raid5_compute_sector() we see

	stripe = chunk_number / data_disks;
	*dd_idx = chunk_number % data_disks;
[...]
	case ALGORITHM_LEFT_SYMMETRIC:
		*pd_idx = data_disks - stripe % raid_disks;
		*dd_idx = (*pd_idx + 1 + *dd_idx) % raid_disks;

So it looks like the chunks are located on the disk like:

stripe	sda	sdb	sdc	sdd
0	D 0	D 1	D 2	P 0-2
1	D 4	D 5	P 3-5	D 3
2	D 8	P 6-8	D 6	D 7
3	P 9-11	D 9	D 10	D 11
4	D 12	D 13	D 14	P 12-14

With each of those 64K. The problem is at 797566305 and the partition started at 385560. So that's at sector 797180745 inside partition sdd3. Each sector is 512 bytes and a chunk is 65536 bytes which gets us 128 sectors per chunk. If we want to know which the stride the sector is on we do: 797180745/128=6227974.5703125 This means that the stripe is number 6227974 and that it's sector 0.5703125*128=73 on sdd3 within that chunk. The parity is on disk: pd_idx = 3 - 6227974 % 4 = 1, being sdb. To get the chunk we need to multiply with the number of data disks (3) and since it's the forth disk and the parity is on the second disk we need to add 1 and get: 6227974*3+1=18683923 So calculating that back to sectors we multiply it with the number of sectors per chunk again and add our offset in the chunk: 18683923*128+73=2391542217 To test that our math is any good we try:

# dd if=/dev/md1 of=797566305.2 skip=2391542217 bs=512 count=1 iflag=direct
1+0 records in
1+0 records out
512 bytes (512 B) copied, 5.99103 s, 0.1 kB/s

We see an error message in the kernel log, there are 2 new errors in the error log, so it seems that everything is calculated correctly. The files are also identical. The file system is in blocks of 4096 bytes, so for the file system this is block 2391542217*512/4096=298942777.125 Then we use debugfs:

# debugfs
debugfs 1.41.2 (02-Oct-2008)
debugfs:  open /dev/md1
debugfs:  icheck 298942777
Block   Inode number
298942777       <block not found>

Which isn't making any sense to me, since I get that error about every 2 hour when a certain cronjob runs. 298942777 / 32768 (Blocks per group) = 9123.00955. So I wonder if ext3 is keeping some meta data there and debugfs doesn't tell that? Anyway, at some point I was running a smartctl -t long /dev/sdd again because I couldn't reproduce the error anymore, the cronjob started, and I then found alot of errors in my log again and now it included:

raid5:md1: read error corrected (8 sectors at 797180744 on sdd3)

It's so much fun that disks always behaves the same trying to read the same sector. Smartctl now reports:

  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   200   200   000    Old_age   Offline      -       0

I assume the kernel removed the pending sector by writing to it. The other pending error was already gone for some time, I assume new data got written to that one.

I spent more time than I would like to admit massaging the process that ultimately led to the release of the the GNU Free Documentation License 1.3 (GFDL) by the Free Software Foundation (FSF). Hours counted, it was probably one of my biggest personal projects this year. The effect is to allow wikis under the GFDL to migrate to the Creative Commons BY-SA license or, as Wikimedia's Erik M ller has proposed, to some sort of dual-license arrangement. There are many reasons for this change but the most important is that the move reduces very real barriers to collaboration between wikis and free culture projects due to license compatibility. BY-SA has become the GPL of the free culture world and Wikimedia wikis were basically locked out from sharing with a larger community, and vice-versa; projects will no longer have to choose between sharing with Wikipedia and sharing with essentially everyone else. The GFDL has done a wonderful job of helping get Wikimedia projects to where they are today and M ller's proposed switch seems, in my opinion, the best option to continue that work going forward. The FSF gets a lot of credit (and a lot of flack) for what it does. Offering to "let go" of Wikipedia -- without question the crown jewel of the free culture world -- represents a real relinquishing of a type of political control and power for the FSF. Doing so was not done lightly. But giving communities the choice to increase compatibility and collaboration by switching to a fundamentally similar license was and is, in my opinion, the right thing to do. Everyone who has worked hard to make this happen deserves the free culture movement's thanks. This list includes Richard Stallman, Brett Smith and Peter Brown of the FSF; James Vasile and Eben Moglen of the SFLC; Erik M ller, Mike Godwin and Shunling Chen of the Wikimedia Foundation. The FSF in general, and RMS in particular, deserves a huge amount of credit for what it has decided to not do in this case and for giving up control in a way that was responsible and accountable to its principles and to GFDL authors and in the interest of free culture movement more generally. It has not been easy or quick. If you support or appreciate work like this, please support the FSF and express this while doing so. Doing so is an important way to support these essential and almost inherently underappreciated efforts.

“Old and New Net Wars over Free Speech, Freedom and Secrecy or How to Understand the Lulz battle against the C0$ Gabriella Coleman, Assistant Professor, New York University Thursday November 13: noon - 2:00pm
Columbia University Communications Colloquium
270B IAB (International Affairs Building update Open the public! You may need ID to get into the building but I am not sure. This is the case with NYU…

“Old and New Net Wars over Free Speech, Freedom and Secrecy or How to Understand the Lulz battle against the C0$ Gabriella Coleman, Assistant Professor, New York University Thursday November 13: noon - 2:00pm
Columbia University Communications Colloquium
270B IAB (International Affairs Building update Open the public! You may need ID to get into the building but I am not sure. This is the case with NYU…

As a brief introduction to this - I first read through the Android code when interviewing with Google for an opening with the Android kernel team. I didn't get the job, and while I don't think anything that follows here is as a result of residual bitterness you might want to take some of it with a pinch of salt.

Anyway. Android is a Linux kernel with a heavily customised user layer that bears little resemblance to any traditional Linux system. I write in C because using pointer arithmetic lets people know that you're virile so this Java-like thing is of little interest to me and I'm going to ignore it and just look at the kernel, because after all that's what I'm paid to be good at.

The short summary of the rest of this post is that I'm not planning on giving up my iphone yet.

The Android kernel source lives in a git tree here. It can pretty much be logically split into two parts - the port to Qualcomm's MSM7xxx series chips and the special Android customisations. A bunch of the MSM7xxx port code has been merged through the ARM tree and is now upstream, and Brian Swetland seems to have been fairly active in looking after that. Full marks to Google there. This code is handy outside the Android world and benefits anyone wanting to run Linux on similar devices.

The Android-specific code is more... interesting.

As I mentioned, the Android application layer isn't Unix in any real sense. The kernel reflects this. It ranges from pragmatic (if hacky) approaches like encoding security policy and capabilities in group IDs that are hardcoded into the kernel through to implementing an in-kernel IPC mechanism (apparently related to the OpenBinder implementation from Palm, but judging by the copyrights a complete rewrite). To an extent, I'm fine with this. Something like Binder is pretty clearly not going upstream, so the fact that it engages in bizarre design decisions like sticking chunks of its interface in /proc is pretty irrelevant. What's more interesting is the code that should be generalisable.

I work on power management, so I'm always interested in what kind of power management functionality and interfaces people want. Plumbers included a nice discussion with someone from an embedded company I can't remember, culminating in us deciding that the existing cpufreq interface did what they wanted and so no new interfaces needed to be defined. Google was going to be an interesting case of a large company hiring people both from the embedded world and also the existing Linux development community and then producing an embedded device that was intended to compete with the very best existing platforms. I had high hopes that this combination of factors would result in the Linux community as a whole having a better idea what the constraints and requirements for high-quality power management in the embedded world were, rather than us ending up with another pile of vendor code sitting on an FTP site somewhere in Taiwan that implements its power management by passing tokenised dead mice through a wormhole.

To a certain extent, my hopes were fulfilled. We got a git server in California.

Android contains something called the android_power driver (ignore the references to CSMI, which is a piece of OMAP-specific hardware for communicating with the baseband - I'm not sure what they're doing in there). As far as I can tell, this is an interface that handles the device being locked and unlocked, and associated powering down of certain bits of hardware. Except it's shit. Devices register device handlers and a level where they wish to be suspended. There's no direct concept of intra-device dependencies so you end up with stuff like:

android_early_suspend_t early_suspend;
android_early_suspend_t slightly_earlier_suspend;

to deal with the fact that the MSM framebuffer driver depends on the MDDI driver having brought the link bank up and so needs its suspend method called before the MDDI driver but its resume method to be called after, and the only way to handle that is to register two methods - the "slightly earlier" one which has a suspend method but no resume, and the "early" one which has a resume method but no suspend one.

Of course, this also means that all of your device runtime power management policy ends up in the kernel. Userspace indicates what state it wants to go to and the kernel decides what's going to get powered down. This kind of coarse grained approach means that as your hardware setup becomes more complex you hit combinatorial explosion. Expressing all the useful combinations of hardware state simply becomes impractical if all you're exposing is a single variable. What would be more useful is the ability for userland to interact with individual pieces of hardware.

The amusing thing is that in many cases Linux already has this. Take a look at the backlight and LCD class drivers, for instance. They provide a trivial mechanism for userspace to indicate its desires and then modify the device power state. It's true that there are other pieces of hardware that don't currently have interfaces to provide this kind of information. And that's where cooperation with the existing community comes in. We've already successfully fleshed out interfaces for runtime power management for several hardware classes, with the main thing blocking us being a lack of awareness of what the use cases for the remaining classes are. But linux-pm has seen nobody from the Android team, and so we end up with a lump of code solving a problem that shouldn't exist.

When Robert Love gave his presentation on Android at Lugradio Live in SF back in April, he talked about how one of the reasons that Google weren't releasing the source for their userland stack until they shipped phones was to prevent it being seen as throwing unfinished code over the wall and then ignoring the community. It's unfortunate that this is almost exactly what's happened with their kernel. Right now the fact that Android is based on Linux is doing almost nothing to benefit the larger Linux community. What could have been a valuable opportunity for us to gain understanding of an interesting problem has instead ended up as yet another embedded vendor kernel, despite all the assurances I got from various people at Google.

Disappointing.

I have it on very good authority^* that our favorite Indian/Nepali restaurant, Himalayan Bistro, is coming to Roslindale (from West Roxbury), into the oft-turned-over location last assigned to NuVo, and Gusto before that. Perhaps the third “-o” restaurant will be a charm. Assuming they serve up the same superlative food and atmosphere as the West Roxbury location, I’m sure this one will be a keeper. (My only concern is that having the two restaurants so close to each other may result in self-competition.) ^*Authority = Himalayan Bistro delivery guy. You don’t get much more authoritative than that. In other Roslindale Restaurant news, we heard via RVMS recently that Robyn’s Bar and Grill has changed hands to the owners of the Halfway Cafe in Dedham, which could be a change for the better. Roslindale did not need another pizza/sports-bar, and still does not need one, although maybe that’s all the Robyn’s location will ever be. Even more recently, however, we heard an unconfirmed rumor from a neighbor that the sale fell through and Robyn’s is not changing owners. In other potentially disappointing news, the Roslindale Emac and Bolio’s has changed owners again, this time to a co-owner of the Blue Star Restaurant. I want to like Blue Star, but they can’t seem to get the most basic short-order cooking tasks right. A diner must, first and foremost, deliver greasy food fast, hot, and at the same time for the entire group. At worst, serve the kids first. It also helps for the food to be consistent. Blue Star in our experience (many visits) only occasionally satisfies the basic diner requirements, much less delivers any sort of creative urban-chic reinterpretation of the classic retro diner. (By contrast, Deluxe Town Diner in Watertown is our current Greater Boston favorite — it’s always a good sign when a greasy spoon also actively caters to vegans.) Which brings me to Emac and Bolio’s, which apparently will be renamed “Select Caf .” First, why give up the goodwill associated with the Emac’s trademark? Second, the initial signs of the change-over are not promising. A handwritten sign posted in the window announces that the cafe is under new management, and (1) no credit or debit cards will be accepted and (2) no Emac and Bolio’s gift certificates will be honored. I’d love to be proved wrong about this, but I’m pessimistic that this is going to be the Change We Need in Roslindale — namely, a top rate artisanale snobby espresso shop. We can only hope that Simon’s, Flatblack, Cafe Nation, Diesel, or another of their ilk will come occupy one of the vacant storefronts on the courtyard. That we have any vacancies at all in the square, much less several, is a crime against humanity. Technorati Tags: Roslindale, Restaurants

I've had this crazy ATA over Ethernet SAN for my MythTV setup for as long as I've been running MythTV. I'm using minotaur, an old Pentium III 1RU VA Linux server as the "head". Originally, I had 4 IDE hard drives, attached to minotaur via IDE-USB adapters. This worked perfectly fine, but I missed the ability to query SMART information, and do other low-level drive tweaking with hddparm So a while ago, NewEgg had a sale on 750Gb SATA drives, and I bought a 4 port eSATA card, and took the plunge with SATA. That all went fine. The entire time, I've had the 4 hard drives themselves just sitting on top of minotaur, on cork trivets. The whole arrangement is in the bottom of our linen closet, as this is where the patch panel for the apartment is. I've graphed the temperature of linen closet for some time now, but the benefit of having hard drives connected via SATA was that I could easily graph the temperature of them as well, which I started doing recently. The catalyst for wanting to do this was that I had a drive fail, and I was concerned that they were getting too hot, just sitting in the closet with no airflow. Matt Bottrell's recent reminder of Google's research into hard drive failures also helped bring this concern to forefront. Seagate's product documentation for the Barracuda 7200.10 claims that ambient operating temperature must be between 0 C and 60 C. I'm assuming that's the environment the hard drive is in, and not the temperature of the drive itself as reported by something like hddtemp. The closet is typically in the high 30's. If you believe what SMART reports, it seems to imply that anything over 45 C is a Bad Thing^tm. So after I had a drive fail, I figured I should probably try to do something about the temperature situation, and started looking around for an external enclosure. I'd previously had problems with the IEC-to-Molex power supplies I was using to power the disks, and think that they'd been the cause of some failures of my IDE disks, so was quite eager to stop using them as well, which was something else an external enclosure would give me. I had trouble finding an enclosure that presented 4 separate eSATA ports, and I was talking to Marc about SATA port multipliers, when he mentioned he had a brand-new unused full-height SCSI enclosure, for four disks, which was surplus to his requirements. Since all I really needed was external power and cooling, I thought I could cannibalise this to my needs. Sure enough, I could just remove the SCSI cables and their Centronics connectors, and feed the eSATA cables out the gap they left, and use the molex-SATA power splitters I already had. Because the enclosure was intended for full-height 3.5" hard drives, there was plenty of air space between each disk. The enclosure included two fans. The results? Well the graph speaks for itself:

(I'm not entirely sure why minotaur's internal disk occasionally reports a very low temperature) The added bonus is I've cleaned up the rats nest of cables in the linen closet. So thanks very much, Marc!