The script isn't fully automated yet, I'm running it manually and watching for errors, but I do now have a simple script (~100 lines of perl) which runs edos-debcheck against each of the supported Emdebian architectures for unstable-grip and picks out those packages which are missing from the subset of packages which Emdebian monitors for Grip.

I was hoping to make these scripts into a normal Debian package but there are some limitations in that the scripts need to assume various pieces of Debian infrastructure. e.g. I started off using rsync to pull in the Packages files (along with Release and Sources) because apt has annoying behaviours with multiple architectures. (Emdebian Grip processes 7 architectures simultaneously on blavet.debian.org). Peter Palfrader kindly arranged for a local mirror to be available and now the entire Packages processing can be done using symlinks. Much, much better.

Actual package movement data comes directly from dak via projectb - something else which makes packaging these scripts for use on non debian.org boxes a bit hard.

Once again, I am indebted to the EDOS team (please, please keep working on edos-debcheck - it hasn't had uploads recently) because I tried to get this working with germinate but failed. (Colin - I'll be looking to borrow some of your time to work out what was going wrong.)

The dependency resolution script now means that I can meet release team requirements that when foo is updated in sid to depend on libbar2 but Emdebian Grip only has libbar1, the scripts will automatically pick up libbar2 and add it to Emdebian Grip. The current sid version gets pulled in and (the theory goes) will migrate into testing-grip as an almost automatically valid candidate. Processing of the dependency can happen within hours of the updated version of foo arriving in Emdebian Grip but as long as it happens within a few days I think everyone will be fine with it.

I've got a sync script too which can periodically run across the entire suite and check for packages which have slipped the net. I'll have to see how often that needs to run but once a week doesn't sound too painful.

Processing for Emdebian Grip is very fast. Most of the time is taken uploading:

2012-01-23 22:24:42 add sid-grip deb main armhf libfltk-images1.3 1.3.0-5em1

2012-01-23 22:24:52 add sid-grip deb main i386 libfltk-images1.3 1.3.0-5em1

2012-01-23 22:25:07 add sid-grip deb main mips libfltk-images1.3 1.3.0-5em1

2012-01-23 22:25:25 add sid-grip deb main powerpc libfltk-images1.3 1.3.0-5em1

2012-01-23 22:26:00 add sid-grip deb main amd64 libstdc++6-4.5-dev 4.5.3-12em1

2012-01-23 22:26:34 add sid-grip deb main armel libstdc++6-4.5-dev 4.5.3-12em1

(It was a lot longer, comparatively, when I was having to rsync and then dget instead of just read from the local filesystem.)

I've also been working on an init script which will gradually work through the subset of packages by order of Priority - approximately what a new architecture (like armhf) would do. Note that Emdebian Grip supports armhf already. The only slight concern is whether the init script needs to be throttled back to not flood the queues with hundreds of uploads an hour (which it could end up doing, at least for the first 24 hours).

Logs of package movements are available: http://www.emdebian.org/grip/logs.php

Packages can be searched too: http://www.emdebian.org/grip/search.php

More on Emdebian Integration on the Debian wiki: http://wiki.debian.org/EmdebianIntegration

Now all I need is a fix for #655919 FTP team?

Recently the upstream Linux kernel released a fix for a serious security vulnerability (CVE-2012-0056) without coordinating with Linux distributions, leaving a window of vulnerability open for end users. Luckily:

• it is only a serious issue in 2.6.39 and later (e.g. Ubuntu 11.10 Oneiric)
• it is only local
• it requires execute access to a setuid program that generates output

Still, it s a cross-architecture local root escalation on most common installations. Don t stop reading just because you don t have a local user base attackers can use this to elevate privileges from your user, or from the web server s user, etc. Since there is now a nearly-complete walk-through, the urgency for fixing this is higher. While you re waiting for your distribution s kernel update, you can use systemtap to change your kernel s running behavior. RedHat suggested this, and here s how to do it in Debian and Ubuntu:

• Download the am I vulnerable? tool, either from RedHat (above), or a more correct version from Brad Spengler.
• Check if you re vulnerable:

  $ make correct_proc_mem_reproducer
  ...
  $ ./correct_proc_mem_reproducer
  vulnerable
  

• Install the kernel debugging symbols (this is big over 2G installed on Ubuntu) and systemtap:
  • Debian:

    # apt-get install -y systemtap linux-image-$(uname -r)-dbg
    

  • Ubuntu:
    • Add the debug package repository and key for your Ubuntu release:

      $ sudo apt-get install -y lsb-release
      $ echo "deb http://ddebs.ubuntu.com/ $(lsb_release -cs) main restricted universe multiverse"   \
            sudo tee -a /etc/apt/sources.list.d/ddebs.list
      $ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ECDCAD72428D7C01
      $ sudo apt-get update
      

    • (This step does not work since the repository metadata isn t updating correctly at the moment see the next step for how to do this manually.) Install the debug symbols for the kernel and install systemtap:

      sudo apt-get install -y systemtap linux-image-$(uname -r)-dbgsym
      

    • (Manual version of the above, skip if the above works for you. Note that this has no integrity checking, etc.)

      $ sudo apt-get install -y systemtap dpkg-dev
      $ wget http://ddebs.ubuntu.com/pool/main/l/linux/$(dpkg -l linux-image-$(uname -r)   grep ^ii   awk ' print $2 "-dbgsym_" $3 '   tail -n1)_$(dpkg-architecture -qDEB_HOST_ARCH).ddeb
      $ sudo dpkg -i linux-image-$(uname -r)-dbgsym.ddeb
      

• Create a systemtap script to block the mem_write function, and install it:

  $ cat > proc-pid-mem.stp <<'EOM'
  probe kernel.function("mem_write@fs/proc/base.c").call  
          $count = 0
   
  EOM
  $ sudo stap -Fg proc-pid-mem.stp
  

• Check that you re no longer vulnerable (until the next reboot):

  $ ./correct_proc_mem_reproducer
  not vulnerable
  

In this case, the systemtap script is changing the argument containing the size of the write to zero bytes ($count = 0), which effectively closes this vulnerability. UPDATE: here s a systemtap script from Soren that doesn t require the full debug symbols. Sneaky, put can be rather slow since it hooks all writes in the system. :)

2012, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

So, at work we have a little single port IP KVM, it is a StarTech SV1107IPEXT - it's not a "bad" bit of kit, if you're completely configured for using it in the way they expect... When you're not, then it throws a NullPointerException when you try to connect with it's Java applet. This morning (mostly because the monitor and keyboard I had been using for building stuff on my desk has gone missing) I decided I'd try again with the device, it's not on our default office network range, mostly because it might not always be in the office, so I quickly added an ip alias using ip addr add 192.168.0.59/24 dev wlan0 on my laptop to bring up a route to it. Navigated to the page and asked it to fire up the Java applet. Straight off, a NullPointerException happens - hmm. So looked at what was being logged by the JVM - it was trying to run ifconfig (no, really, the java applet ran ifconfig... with no path...) which, on my debian laptop, is not in the default path... so I went and make a quick wrapper in ~/bin for it... right - now it can run that... still a NPE, hmm. So look a little closer, it's looking for the MAC address assigned to the IP that you're coming from... no, really! WTF! So I went and hacked my little ifconfig wrapper to rewrite the address for wlan0 to be what the KVM expected it to be, low and behold, I can now run the java client (hoorah!). Why the heck are they doing that though - that's just plain insane! On the plus side, I now have a work around, on the minus side, egad that's ugly!

I wrote about the technical details of supporting the UEFI secure boot specification with Linux. Despite me pretty clearly saying that this was ignoring issues of licensing and key distribution and the like, people are now using it to claim that Linux could support secure boot with minimal effort. In a sense, they're right. The technical implementation details are fairly straightforward. But they're not the difficult bit.
Secure boot requires that all code that can touch hardware be trustedRight now, if you can run unstrusted code before the OS then you can subvert the OS. Secure boot gives you a mechanism for making sure you only run trusted code, which protects against that. So your UEFI drivers have to be signed, your bootloader has to be signed, and your bootloader must only load a signed kernel. If you've only booted trusted code then you know that your OS is safe. But, unlike trusted boot, secure boot provides no way for you to know that only trusted code was executed. That has to be ensured by OS policy.

This doesn't sound like too much of a problem. But it is. Imagine we have a signed Linux bootloader and a signed Linux kernel, and that these signatures are made with a globally trusted key. These will boot on any hardware using secure boot. Now imagine that an attacker writes a kernel module that sets up a fake UEFI environment, stops the kernel from running code and then executes the Windows bootloader - kind of like kexec, but a little more fiddly. The Windows bootloader believes that it's performing a secure boot, but in fact everything that it believes is trustworthy is the attacker's fake UEFI code. The user's encryption passphrase is logged, the Windows kernel is modified to hide the Linux code and despite everything looking fine your credit card details are on their way to China. In this scenario, the signed Linux kernel is simply used as a malware loader. The only sign that anything is wrong is that boot will be slightly slowed down.

Signing the kernel isn't enough. Signed Linux kernels must refuse to load any unsigned kernel modules. Virtualbox on Linux? Dead. Nvidia binary driver on Linux? Dead. All out of tree kernel modules? Utterly, utterly dead. Building an updated driver locally? Not going to happen. That's going to make some people fairly unhappy.

(The same applies to Windows, of course. Windows 7 allows you to disable driver integrity checks. Windows 8 will have to forbid that when the system's using secure boot)
LicensingGPLv3 has various requirements for signing keys to be available. Microsoft's new requirement that systems support the installation of user keys would let users boot their own modified bootloaders, so that may end up being sufficient to satisfy the license. But we're then beholden on Microsoft - if they remove that requirement then users lose that freedom, and suddenly we're in an awkward licensing situation. There are ongoing conversations about exactly what we're able to do here, but it's not a solved problem.
Key distributionThe UEFI spec doesn't describe or mandate a central certifying authority. Microsoft require that everyone carry their key. We could generate our own, but we have much less sway with vendors. There's no way to guarantee that all hardware vendors will generate our key. And, obviously, if we generate a key, we can't just hand the private half out to others. That means that it becomes impossible for people to produce derivative versions of Linux distributions without getting their own key. The kind of identity verification that would be required for getting such a key is likely to be expensive, and also fairly likely to require that the distribution have a legally registered company in order to facilitate the identity verification. Think Extended Validation certificates, not Startssl Free. Hobbyist Linux distributions will be a thing of the past.
Doesn't custom mode fix this?Microsoft's certification requirements now state that all systems must support a custom mode, implying that it will be possible for a user to install their own keys. Linux vendors would then be able to ship with their own keys on the install media and impose their own policies. Everyone's happy. It's not really good enough, though. People have spent incredible amounts of time and effort making it easy to install Linux by doing little more than putting a CD in a drive. Asking them to go into the firmware and reconfigure things adds an extra barrier that restricts the ability to install Linux to more technically skilled users. And it's even worse than that. This is the full description of the requirement for custom mode:

1. It shall be possible for a physically present user to use the Custom Mode firmware setup option to modify the contents of the Secure Boot signature databases and the PK.
   
2. If the user ends up deleting the PK then, upon exiting the Custom Mode firmware setup, the system will be operating in Setup Mode with Secure Boot turned off.
   
3. The firmware setup shall indicate if Secure Boot is turned on, and if it is operated in Standard or Custom Mode. The firmware setup must provide an option to return from Custom to Standard Mode which restores the factory defaults.

There's a few things missing from this, namely:

• Any description of the UI. It's effectively impossible to document Linux installation when the first step becomes (a) complicated and (b) vendor specific. Vendors are using the UEFI transition to differentiate themselves by coming up with their own unique firmware interfaces. Custom mode is going to look different everywhere.
• Any description of the key format. A raw binary representation of the key? An EFI_SIGNATURE_DATA struct? A base64 encoding of one, further protected with ROT13? We just don't know.
• Any way to use custom mode for unattended installs. It's a firmware interface that requires a physically present user. Want to install a few thousand machines over the network? This isn't a scalable approach
• and this one's nitpicking, but there's not actually any requirement that the user be able to add keys - a vendor could conform to this language by only letting users delete keys. This is actually ok as long as the user deletes Pk, because then we'll effectively be back in setup mode and can install our own keys from the installer, but it still results in some practical problems

So no, custom mode doesn't make everything ok. Custom mode with a mandated UI and a documented key format would be much closer, but it wouldn't solve the problem of unattended automated installs.
SummaryWe can write the code required to support secure boot on Linux in a minimal amount of time - in fact, most of it's now done. But significant practical problems remain, and so far we have no workable solutions for any of them.

comments

Update: I managed to make sound work. For some weird reason, a mute switch option of some the many (and who knows how useful) switches of my sound card was enabled. Now the damn thing works. Did I mention that since I did the upgrade all my sound cards (I have a USB sound card, too) have listed as available inputs all the inputs of my internal sound card (mic, front mic, line in, CD, etc.) in Audacity? That makes for a very confusing and loooong sound input sources list! The upside is that I can finally record clips from televisions that do not provide such a feature and FastVideoDownload doesn't handle.

I also seem to have found a possible fix for the caps-ctrl issue in Xfce4 (obviously, setting "-option ctrl:swapcap" in ~/.Xkbmap, instead of that Alt modifier).

---

As I said in my previous post, I will tell you what do GNOME 3 and Windows 7 have in common.

Before everything else, I want to make it clear that when I am saying GNOME 3, I am referring to Debian Wheezy's GNOME 3, since I recently upgraded from Squeeze on my laptop. I'll probably drop a line or two about that, too.

First, I'll tell you about the (boring, probably for many) experience with Windows 7. As I said before, my new job requires me to use a Windows machine, so up until a few months ago I was using Windows XP with some additional software and tweaks to make it usable. Then came the Windows 7 upgrade . I am using quotes since the more appropriate term would be fresh installation on a new partition , not even close to what Debian users are used to call an upgrade.

So after a fresh Windows 7 installation, my first shock was the fact there was NO Quick Launch*. Some of you might be laughing, but I had never used Windows 7 up until then, just saw it on a laptop of a friend of mine (Ovidiu, one of the guys with whom I am doing this podcast, went to Denkfest with, and made these interviews). That was the first shock. Initial discussions about this with Windows users lead me to believe Quick Launch was dead and for some unexplained reason, I believed them. Later, much later, a week ago, to be precise, I found out that you can bring back the Quick Launch through some convoluted way**. Up until that point I had to have some icons pinned to the task bar, but some others on the desktop (and I hate that) because some of them, like Cygwin, if pinned, would start a cmd console, since Win 7 pins the process, not the starting script.

Among other things which broke in Win 7 and used to work fine in XP, the Virtual Dimension application which provides me with a virtual desktop, was the first one which was broken. I have been using a liniar 4 desktops-wide virtual desktop for over 5 years and I am worthless and inefficient if all my apps are on the same desktop. Mail application is always on the first desktop, work and file managers are on the second, the third is for extras and multimedia editing while the fourth is my gateway to the internet, containing the browser, instant messenger, or whatever.

The shortcuts I use to get to the various desktops are Win+1 ... Win+4 keyboard shortcuts, but the M$ Evil Empire decided that those shortcuts are going to start or bring foreward the first, second and so on applications pinned on the task bar. And you can't change those shortcuts***. Nor is disabling just those possible since they are all disabled through a huge switch which disables ALL Win+x keyboard shortcuts, among which Win+E (file expolrer) and Win+D (Show Desktop) were also. Luckly, Win+L (lock screen) was not disabled. So I disabled al those Win+ shortcuts, since I need virtual desktops.

Now, imagine if I had to start a Cygwin console and I had all sorts of apps open! Win+D was disabled, so I had to minimize the apps covering the desktop shortcut for Cygwin, click on the icon to start it, bring back the minimized windows and go on with my work. What a waste of clicks, mouse movement, energy and time, just because some dudes thought a Quick Launch-like feature was useless****.


You might wonder already what do those '*' sings mean. Well, sadly, that's what GNOME 3 / gnome-shell and Windows 7 have in common.

Gnome 3 was a shock for me. An empty desktop right after upgrade. No panels, no shortcuts*, no power indicators, no wicd indicator, no virtual desktops, no desktop icons, (I have a few dirs and docs there). Sounds like an Evil Empire decision, doesn't it?

Luckly I have been using Tilda as my always-ready console and I could fire up iceweasel from the console in order to understand where my panel disappeared.

I then realised that the upgrade brought me Network Manager, that app which wicd replaced. As a consequence, I had no working wlan since Network Manager made sure to mess up with the network manager I chose.

After looking through the documentation of Network Manager and realising I either had it set up to leave wlan0 alone or I didn't understood NM's documentation, I simply stopped the service, which let Wicd its job flawlessly.

The first thing I searched was Gnome 3 panel or something of that sort and I was confronted with the obvious option to appeal to the Forced Fallback Mode which was disabled. I figured I either had an old version, or Debian disabled this feature (hoping they provided an alternative). There was also the option to conform to this convoluted way of working** with Actions and such uselessness like that. I still wonder, what is the purpose of the Favourites bar on the left side, since it's accessible only after wasting a lot of mouse movement and time? For Joe's Pesci sake, I use focus under mouse just to avoid needless mouse and keyboard manipulation. Why? Why? WHY would I want every time I need to start or SWITCH to another application to move the mouse to the upper-left corner then take my hands off the mouse to type, move the mouse downwards or move across the whole width of the screen to get to my beloved virtual desktops and pick the app I want?

Making a long story short, after even trying XFCE4 (which for some unknown reason resets almost immediately my keyboard layout to the default layout with the Caps on Caps, instead of my preferred and set Ctrl on Caps - yes, it's global), I managed to find GNOME Shell Frippery** which made the experience better.

Later I found out that GNOME 3's file manager, Nautilus, has decided that an up on level button is useless, since the default is to use that uncopy-pastable button location bar instead of a sane text location bar. And it seems the GNOME developers decided this*** and I should conform to it.

To add insult to injury, those icons on my old panel are apparently useless**** and even in the fallback version I can't get them back. Or so the GNOME developers decided.

At some point this sunday, don't know how or why this change happened, producing sound was impossible. I know the problem is pulseaudio since when I kill the pulseaudio daemon from the console I can play audio. BTW, great timing, just when I needed sound the most, before releasing episode 32 of our podcast (yay, I reaslised that xfce just decided to reset my caps to be caps, after setting to ctrl a few minutes ago).

I know I praised pulseaudio when I first tried it, but failing to make it work out of the box or after some tinkering is a deal breaker for me, so I removed it. Now I find it that is a default in GNOME, yet all it manages to do is prevent audio from working. At least on my machine.

Other problems? Gnome Power Manager manages to hang and block my session, GNOME managed somehow to fail to start at some point. Yeah, and that sound problem which I didn't fix yet, didn't went away after removing all the pulseaudio packages which could be removed (e.g.: ryhtmbox depends on libpulse0, same do some other apps like audacity, so I couldn't remove all pulse related packages).

I got involved with Debian and GNU/Linux because it was tweakable and customisable, didn't use to force all sorts of option on me and now I find with its increasing popularity it becomes more and more like a product of a corporation which decides to change some things just to change and totally disregadring user experience and uses.

So, in the light of all of these problems I think it's time to probably consider trying KDE. Is it any good lately?

(This is not quite a Dear lazyweb post. If anything, it's Dear lazyweb, I've done my homework, now what? ) I'm looking for the ultimate distributed filesystem. Something that's simple to use, redundant, fault-tolerant, and still smart enough to avoid the more obvious performance chokepoints. Ideally, it should work for all of the following sets of files:

• Music files. They're currently stored on a plug computer at home and served over DAAP, but DAAP seems to become less and less of a priority for music players (I'll keep the music players suck rant for another post), and it has its inconveniences too. Ideally, the files should be integrated in the filesystem of my work computer, the one that serves as a media player in the living-room, the one I use for playing the backing tracks when I'm practising the drums, and the laptop I use when on the road too.
• Backups. My BackupPC currently runs on the plug computer at home, so backups of my remote servers are in a geographically distinct location; but backups of my home computers are not, and I'd like to change that without having to copy files around by hand. Backups do contain confidential data, so this adds a constraint on the filesystem.
• Bazaar repositories. I do have a script that pushes and pulls stuff across computers for the many repositories I use, but it's still awkward that I need to do that.
• Parts of my home directory, such as my browser preferences and bookmarks, or the database of the Hamster time tracking tool, or the working directories of stuff I do for clients, and so on. Again, each of those can be done with specific means (bookmarks synced to a server, CouchDB-like database, DVCS and regular commits, etc.), but wouldn't it be much simpler if there only was one file to begin with?

The requirements on the ultimate distributed filesystem (which I'll call UDFS for short, otherwise you'll get bored and go look at pictures of kittens) are as follows:

• Availability means redundancy: some of the storage nodes will be on dedicated servers in datacenters, others at home; I can imagine setting up the firewall so that the home computers are reachable from outside, but sometimes network links go down, and the home computers are far from being on 100% of the time.
• Availability/redundancy also means automated replication and rebalancing: if a node is added to the grid (or switched on), it should automatically get its share of the files so as to contribute to availability if another node goes down at a later time.
• Confidentiality: at the very least, network communications must be encrypted and authenticated; ideally, individual storage nodes wouldn't need to be able to access the stored data. If I store bits of my backups on a friend's server, I don't want to have to trust them not to look at the data; also, my friend may actually want to be unable to look at my data (to provide for deniability in case someone else wants to look at it).
• Performance: native disk performance may not be realistically reachable, but the system must be smart enough (or configurable enough) to store files on both sides of the ADSL link, for instance, so not all accesses need to go through the bottleneck.
• Integration with the system: I want a filesystem, not a storage system. All applications know how to navigate a mounted filesystem; very few will interface with an specific application designed to store and fetch chunks of data.
• Scalability would be nice, although my personal needs are still well below the terabyte range and I can't see myself using more than a dozen nodes or so.

Quite a few constraints, eh? I have a feeling they are not mutually incompatible though, so I had a look at several candidates. I started with Wikipedia, and I followed the links to Tahoe-LAFS, XtreemFS and Ceph. The following is my evaluation of these candidates based on much reading of docs and websites and wikis, some questioning on IRC, and very little testing.

• Availability/redundancy 1: all three candidates work on the net, and they all provide for data replication. XtreemFS seems to operate on a fall-back mode while the other two are more distributed (meaning there's no canonical node hosting any particular file). Tahoe uses erasure codes (files can be split across N shares, k of which are enough to reconstruct the whole file; the N/k ratio controls the amount of redundancy); it seems to require the introducer node to be always up, which introduces a single point of failure, but this node can be replaced with no loss of data if it fails (this just requires reconfiguring the storage nodes, which can probably be automated). Ceph can work with any number of meta-data servers, so redundancy is assured there (data itself can be replicated in a configurable way too).
• Availability/redundancy 2: either I'm a fool and I didn't find out about that, or none of the three candidates provides for automated rebalancing. Apparently Tahoe provides a way to repair files without enough redundancy, but that needs to be run manually on each file, rather than being systematic, and it's a kind of add-on to the system rather than being properly integrated. Ceph only does rebalancing of meta-data. XtreemFS seems to work a bit like RAID1 with spare disks, but details are scarce.
• Confidentiality: Tahoe wins, hands down. Neither network sniffers nor storage nodes can see the contents of the stored files. They can't even find out about their name or the directory structure. XtreemFS encrypts the data on the network (but it is still stored in cleartext on the nodes). Ceph doesn't even try.
• Performance: Ceph wins (or loses less, at any rate): it seems possible to configure a topology for the storage nodes, and to drive the storage location policy according to this topology; so I could define sets of nodes ( my servers in datacenters , my plug computer at home , my desktop and laptop computers , my friends' computers ) and decide that each file should be stored at least once on each set of nodes. Tahoe stores shares of files on a number of storage nodes chosed in kind of a round-robin way; read access uses the same round-robin system, which means that you will probably end up fetching a file at least partially over your slow link even when you could fetch it entirely from the local network. The XtreemFS website doesn't seem to acknowledge the possibility of the network being slow.
• Integration: Ceph and XtreemFS win: they're native filesystems. Tahoe is in the storage and retrieval system category; the FUSE layer is marked as experimental and not recommended, and the SFTP layer (which can be mounted as a filesystem with sshfs) has many documented drawbacks. Maybe a WebDAV frontend, combined with fusedav, would provide a working alternative, but it's not implemented yet.
• Scalability: all three candidates boast about scalability (in data size and in number of nodes). Ceph seems to require some configuration every time a new storage node is added. Tahoe seems much more dynamic: new nodes (storage or clients) just need to be told where the introducer node is, and then they merge into the grid seamlessly by being told about other nodes (a bit like Bittorrent, where clients learn about each other by asking the tracker).

The overall picture seems full of good things scattered across different solutions, but unfortunately none of the existing ones seems to address the whole problem; at least, not my whole problem. It would be good if each focused on one layer and did that layer well, but that seems not to be the case either, so they can't be combined to get the best of all worlds. It may be that I'm missing something, or that I failed to read some docs properly, or that I misunderstood the docs, or that the docs themselves are simply lacking; but my ideal UDFS currently doesn't seem to exist as a turn-key solution. However, the main pieces are available, and implementing the remaining parts may be doable. My humble idea of a way forward would be based on Tahoe-LAFS, with the following three changes:

• A configurable dispatch policy, so that administrators could define their own behaviour. The current Tahoe parts already turn files into a number of blobs that only need to be stored on various nodes, and the function that chooses which nodes seems to be reasonably self-contained, so it could maybe even be made pluggable, and an administrator could implement a storage policy that matches the constraints and the network topology. Ditto for the function that picks which nodes the data is read from (which, as far as I can tell, is the same; splitting could bring some benefits here).
• A working FUSE implementation, either on top of SFTP if the drawbacks can be fixed, on top of WebDAV if that gets implemented, or native to Tahoe if that can be done.
• Automatic rebalancing of data when new nodes are added or turned on, or when an intermittent network link comes up. The current script seems to be an afterthought; if it can be made more automated and more reliable, I'll be happy with that.

Also nice to have would be a way to work with multiple introducer nodes, but that seems to be in the works already. This would be pretty damn close to my UDFS; read/write performance would certainly be far from what can be obtained on native filesystems stored on local disks, but my use cases involve reasonably small files for which instant access is not compulsory, and the filesystem cache would probably absorb most of the access times. In case anyone is looking for ideas of things to do in their spare time, here are rough sketches of other possible UDFS implementations I thought of. These are wild ideas, and I'm not even sure they could be doable in practice:

• eCryptfs on top of Ceph. The file contents are still only accessible to those in possession of the adequate key, and replication/distribution is handled by Ceph. That could probably work; the main drawbacks I can see come from Ceph's administrative overhead and manual configuration.
• LUKS and a standard filesystem, on top of the part of Ceph that manages block devices (RADOS). I'm not sure this could be mounted simultaneously by several nodes, though.
• LUKS on top of Ceph's RADOS, and Lustre or GlusterFS on top of that? Am I even making sense?
• A kind of union filesystem with local caching, on top of Tahoe-LAFS. Say we take unionfs-fuse, and add local caching: if the requested file is already in cache, performance is near native; if not, we still get the advantages of Tahoe. I'm not sure how caching works with WebDAV, but once Tahoe gets a WebDAV frontend it might be a simple matter of adding a Squid cache between that and davfs.
• I even thought of building a filesystem on top of the Bittorrent protocol; redundancy would be obtained by ensuring that there are at least N distributed copies of a file among the peers. I gave up even before reaching the details phase, though, but maybe a variant of the idea could be workable.

Such is the state of my research so far. I would welcome feedback, pointers to things I neglected to read, corrections for things I misread or misunderstood, comments on the ideas and so on. I'll probably post an update if my search goes significantly forward. Update: I've already received two pieces of feedback, including a lengthy one with corrections about Tahoe-LAFS. For the sake of fairness, I'll solicit (and wait for) the same from the other candidates I looked at. I was also pointed at HekaFS, GlusterFS and git-annex, which I'll have to look at in more details. Other suggestions are still welcome, but the more I get, the more the full update will be delayed. Thanks already!

Steve McIntyre has been contributing to Debian since 1996, 2 years before I joined! But I quickly stumbled upon Steve: in 1999, he was struggling with getting his debian-cd script to produce 2 ISO images (it was the first time that Debian did no longer fit on a single CD), I helped him by rewriting debian-cd with a robust system to split packages on as many ISO images as required. I remember those times very well because Steve was very supportive of my efforts and it was a real pleasure to get this done. His friendly nature probably also explains why he got elected Debian Project Leader twice! Anyway, enough history, check out his interview to learn more about the great work he s doing nowadays. My questions are in bold, the rest is by Steve. Raphael: Who are you? Steve: I m a professional software engineer, 37, living in Cambridge (England) with my new wife Jo. I studied for the EIST degree at the University of Cambridge, then (like many people here, it seems) I just forgot to go home again afterwards and settled here. I spent more of my study time playing with Linux than working on my degree, so I guess I m lucky that it worked and I found a career in that area! Raphael: How did you start contributing to Debian? Steve: During my time in college, I started hacking on software in my free time, using Slackware as my first Linux distribution from the middle of 1994. After encountering more and more problems with Slackware, I was encouraged by a number of friends to make the jump over to Debian and in October 1996 I did. The installation process back then was much harder than anything people see today, but after a long weekend I finally had my Debian system up and running. I was already one of the main upstream developers for the Mikmod music player at that time, so that very same weekend I applied to be a DD so I could maintain it in Debian too. Back then, the NM process was much simpler: I just mailed a key to Bruce and he set me up with an account almost immediately! I then found that Joey Hess had beaten me to it and already packaged Mikmod. Grrr! Raphael: What s your biggest achievement within Debian? Steve: Without a doubt, my proudest achievement within Debian is being elected Project Leader for 2 years by the other developers. It s a great feeling to have earned the trust of your friends and peers, and also a great responsibility to go and help Debian where needed: talking to the press about Debian, assisting wherever problems crop up, etc. The DPL job is certainly a lot of hard work, and I have nothing but respect for anybody who volunteers for it.

It s a great feeling to have earned the trust of your friends and peers.

Elsewhere, I ve been leading the Debian CD team for years too, both doing most of the maintenance of the debian-cd package and producing and testing the regular installation CDs and DVDs that we ship to the world. Again, this is a time-consuming job but it needs doing and it s worthwhile. Raphael: You re currently employed by ARM. What are you working on and are they supportive of your Debian involvement? Steve: The situation within ARM is very interesting; I m employed in PDSW (Processor Division, SoftWare), a new group founded just a couple of years back to help improve the state of software on ARM. Most of the people in the group are working on Free Software at this stage (e.g. toolchains, browsers, Linux kernel), which is lovely. Some of the engineers have also been seconded into a new non-profit company Linaro, which is a collaboration between ARM and a number of other companies investing in core Linux software and tools for ARM-based CPUs. I m one of the ARM engineers in Linaro, and I m a Technical Architect in the Office of the CTO. My role includes looking at future projects for Linaro to help with (e.g. ARM servers), but for the last few months I ve been concentrating on the new armhf architecture in Debian, Ubuntu and elsewhere. armhf is a new architecture in Debian and Ubuntu terms, but it s not strictly a new type of hardware. Instead, it s a new ABI. We have two reasons for doing this work:

1. It targets the latest version of 32-bit ARM CPUs (v7) and makes better use of the hardware, for better performance. Compare targetting i686 instead of i386, for example. We ll still support the older armel port for the foreseeable future for users with older hardware that can t run armhf.
2. More importantly: we are standardising on the ABI / compiler options / hardware support for future users.

In the past, there has been a huge amount of specialisation (aka fragmentation) in the ARM Linux environment, and that worked OK for specialised devices that only ever ran the software shipped with them. ARM CPUs are now becoming more and more mainstream, so people will expect to be able to install generic software on their machines. That gives a requirement for a standard base platform, and armhf (arm-linux-gnueabihf in GNU triplet terms) is that standard that we are pushing in the community. Debian, Ubuntu, Fedora, Suse and others are all going to use this, making compatibility possible. I ve been working with a small team of people to make armhf happen, helping where needed: putting together build machines; patching Debian packages directly; discussing and fixing toolchain issues with Ubuntu folks; agreeing ABI specifications with people from Fedora; advising people from other distros bootstrapping their new ARM ports. ARM and Linaro are very supportive of this work, and it s been lovely being sponsored to work directly on Free Software like this. It s work that will directly benefit ARM and its partners (of course!), but it s also helping out more generally too: Debian QA work, cross-build support, bootstrapping efforts, multi-arch. More and more of the ARM market is driven by Free Software, and companies are acknowledging that. I should probably also mention that we re hiring ! Raphael: What are your plans for Debian Wheezy? Steve: There are three main tracks here. Obviously, I m interested in seeing armhf release with Wheezy. We ve just been added to Testing last weekend, so that s going well. We ve got over 90% of the archive built now, and we re mopping up the remaining issues. I m the primary maintainer of cdrkit at this point, but I d prefer to have it go away. Xorriso and the associated software in libisoburn is almost capable of replacing all the aging cdrtools-derived software that we have in Debian, The only missing feature that I m aware of is creating the HFS hybrid filesystems that we use for installations on Mac systems. I ve been talking with the upstream folks about this for some time already, and I m hoping we can finish this soon enough that we can get it into Wheezy. Finally, I ve got the ever-growing wishlist of things for debian-cd. We ve got the beginnings of an automated test suite that Mart n Ferrari has written, but it needs integrating and improving. I want to help get regular weekly/daily/release debian-live builds running on the main CD build machine. There s work needed if we want to make good installation media for the new multi-arch world, too. The Emdebian people are asking for help making CD images The list goes on Raphael: The ARM community seems to be very interested in multi-arch. Can you explain why? Steve: There are a number of reasons for ARM people to be interested in multi-arch; two really stand out for me:

• With the historical issues around the plethora of ARM ABIs in the wild, multi-arch will allow us to potentially support multiple ABIs cleanly on one system. That allows users to have (for example) an up-to-date system that makes the most of their current hardware, yet also run legacy programs that might use an older ABI. There s also a new 64-bit architecture coming (ARMv8) which will run older 32-bit software; again, multi-arch makes mixed installation of old and new software reasonable.
• ARM has traditionally been a common target for cross-compilation, and I d expect that to remain the case for a long time to come yet. For a lot of embedded developers, using a big fast i386/amd64 machine to compile is much faster than using a limited-power small ARM CPU. However, setting up sane cross-compilation environments has long been a bugbear for developers. Getting the toolchain and all the cross-architecture libraries to work together correctly can be like black magic. This is potentially the killer app for multi-arch: simply install the libraries for the target architecture directly on your development machine. Install a simple cross-gcc package and (maybe) qemu, and you re all set.

This is potentially the killer app for multi-arch: simply install the libraries for the target architecture [ ], install a simple cross-gcc package [ ] and you re all set.

Raphael: What s the biggest problem of Debian? Steve: For me, Debian s biggest problem has been the same for a long time: we are forever short of enough people to do the work that we re trying to do. That might sound like a weird thing to claim when Debian is one of the largest Free Software projects on the planet, but it s more a statement of just how huge our goals are. Many of the largest things in Debian are developed or controlled by very small teams working very hard, and there s always a risk of losing people due to burnout in those situations.

We are forever short of enough people to do the work that we re trying to do.

Some of the tasks that should be easy given our large membership (e.g. large-scale packaging transitions) can often instead take a very long time. We are fortunate to have more people wanting to join in Debian s work all the time, but we also need to be careful to keep on promoting what we re doing and recruiting new contributors, encouraging them to get more and more involved in core work. Debian gets ever bigger in terms of the size and the number of packages we distribute; we re not currently matching that growth rate elsewhere. Raphael: What motivates you to continue to contribute year after year? Steve: This one is much easier to answer! The thing that first attracted me to Debian was the fact that I could help to develop it, help to decide how things could and should be done within it. Instead of being forced to accept what some corporation decided I could do with my computer, I could change the software to suit my needs and preferences. Alongside that, I could get involved with a strong community of similar people all over the world, all with their own strong opinions about how software should work. I joined in and found it was great fun and very rewarding. That hasn t changed for me in the intervening years, and that s why I m still around. I work on Debian because it helps me to get the OS that I want to use. It seems that lots of people around the world find it useful too, and that s awesome. Raphael: Do you believe that Stefano Zacchiroli will be the first DPL who managed to stay 3 consecutive years on the seat? Would you like him to candidate again? Steve: To be honest, I would be very surprised if Zack stood again for DPL this year. He told me himself that he wasn t planning on it, and I can understand that decision. He s been an awesome DPL in my opinion, and I m glad that he took the job. But: it is also a very difficult and time-consuming task that would be enough to wear down anybody. If Zack does decide to stand again, I would support him 100%. But I know that we also have lots of other good people in Debian who would be ready to take up the challenge next. Raphael: Is there someone in Debian that you admire for their contributions? Steve: There are lots of people I admire in Debian, so many so that I almost don t want to list individuals here for fear of missing people out. But Bdale Garbee has been an inspiration to many of us, for many years. He s technically excellent, a great friend to many of us, an endless source of sage advice and (last but not least) he has some wonderful stories to tell about his experiences over the years. On top of that, he s just cool. Christian Perrier is another exceptional developer, in my eyes he s great at co-ordinating people in translations, working tirelessly to make this very important part of Debian work better and better with every release. He s also a really nice guy and we all love him. I also have to mention Joey Hess here, whether he likes it or not. *grin* He s been responsible for so many good things in Debian over the years, even if he did steal my first package Finally, the teams of people who make sure that Debian is always working: the security team and DSA. The rest of us can choose to take time off from Debian to go and do other things, but these people need to cover things every day. That s a major responsibility, and I salute them for taking on that challenge.

---

Thank you to Steve for the time spent answering my questions. I hope you enjoyed reading his answers as I did. Note that you can find older interviews on http://wiki.debian.org/PeopleBehindDebian.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Google+, Twitter and Facebook.

One comment Liked this article? Click here. My blog is Flattr-enabled.

I had a bit of an adventure yesterday, which would have taken some explaining if the police had gotten involved. It went a little something like this... My friend and former co-worker Sara was in the US Virgin Islands for the holidays. Her boyfriend, Karl, flew there separately for the tail end of her time there. Yesterday, I received a phone call from Sara, saying that Karl had managed to fly out to the Virgin Islands without his passport. Apparently you can get there without one, but to get back into the mainland US, you need one. She wanted to know if I could get one of my lock-picking co-workers to break into their apartment and retrieve Karl's passport and mail it them. Karl was supposed to fly out the next day. Attempts by Sara to contact her landlord had failed, so they didn't have many other options (apart from mailing me a key, which would have cost them another day). I asked one of my co-workers, Jason, who I knew was into lock picking, if he was up for it, and he offered to put me in touch with another guy who had dominated the recent lock picking night that he'd run. So now I'm talking to David, who's on board with the mission, but doesn't have his lock picking gear on him. No problem, Jason says he'll lend me his, which was at work with him. So we have a plan. Our friends Ian and Melinda are currently in Australia. They've lent us their car because it's leased, and they have some minimum mileage they're supposed to do and they're under it, so I've been driving to work in their car some days. As it happens, I drove to work in it yesterday. So now David and I set out in a car that neither of us own, with a lock picking set that belongs to another person, to break into an apartment of someone who's in the Virgin Islands. What could possibly go wrong? I'm told that it's not illegal to own a lock picking set, but if you're caught with one on your person and you're not a locksmith, you can get into all sorts of trouble. On top of that I'd have a hard time explaining the car I'm driving. We get to Sara and Karl's condo complex. It has a common gate that visitors would normal get buzzed through. Turns out it's not that hard to climb over. It's got some benign-looking spiky things on top, but I could get a leg over from the left hand side of the gate and jump over without impaling myself. Then I let David in and we proceeded upstairs to Sara and Karl's apartment door, where David set to work. Sara said that just the dead bolt was locked. David started at it with Jason's tools, trying to be as discreet as possible. It was about 3:30pm and there was no one around, but we could hear some noises from the neighbouring apartment (the two front doors were right next to each other). After what felt like about half an hour without success (the last pin of the lock was particularly tricky apparently) David was having to resort to more noisy techniques with the lock, so I decided to take the up-front approach and just inform the next door neighbour what we were doing in case he/she (I think it was a she) decided to call the cops on us. I told her through the door why we were there and what we were doing. She didn't seem to care too much. David then proceeded to start "raking" the lock, essentially brute forcing the pins with a lot of jiggling, and finally managed to pick it and we were in. I quickly found Karl's passport where it was suspected to be, and then we pondered how we were going to lock the door again. We could have just locked the door knob instead of the deadbolt and closed the door behind us, but we weren't sure if Sara and Karl had a key to the doorknob (Sara said they always just locked the deadbolt). Sara was fine with leaving the door unlocked until they got home, but weren't so keen on leaving our fingerprints all over the place and then leaving the door unlocked. David tried to re-pick the deadbolt so that he could lock it via the same means as opened it, and I scouted around for a key. I managed to find a key that locked both the deadbolt and the doorknob, so I took that with us and locked up their apartment. In David's defense, the deadbolt was a bit stiff to lock even with the key. I dropped David back at work, collected my stuff (it was now about 4:30pm) and headed to the UPS Store to ship Karl's passport to him as fast as humanly possible. I just made the 5pm pick up. Today I received an SMS from Sara informing me that they had received the passport. I was very impressed with how fast it got to them. So that was all a bit of an adventure. I'm not sure how much longer Karl is going to have to stay in the Virgin Islands as a result. I'm going to suggest that Sara and Karl leave a spare key with someone in future.

In the Squeeze version of Debian Edu / Skolelinux soon to be released, users of the system will get their default browser start page set from LDAP, allowing the system administrator to point all users to the school web page by updating one setting in LDAP. In addition to setting the default start page when a machine boots, users are shown the same page as a welcome page when they log in for the first time. The LDAP object dc=skole,dc=skolelinux,dc=no have an attribute labeledURI with "http://www/ LDAP for Debian Edu/Skolelinux" as the default content. By changing this value to another URL, all users get to see the page behind this new URL. An easy way to update it is by using the ldapvi tool. It can be called as "ldapvi -ZD '(cn=admin)'' to update LDAP with the new setting. We have written the code to adjust the default start page and show the welcome page, and I wonder if there is an easier way to do this from within Iceweasel instead.

Makers by Cory Doctorow (and Chiriaco Summit again) Tonight I'm staying at the free camping area behind the General Patton Museum at Chiriaco Summit. Not a lot has changed since the last time I stayed here. There is one other RV here as well. The picture is taken from the tank garden, with my RV in the free camping area behind. Makers is a SF novel set in the fairly near future. It is available in ebook form free from the author's web site or if you prefer you can buy the paper edition from a bookstore. I found the first portion one of the most believable near-future with a significant change I have read. (The boogie-woogie Elmo dolls that run Linux and didn't sell were a cute touch.) While the technology involved might seem the same as the typical techno-babble seen in mass-market SF to someone not in the computer industry, it is actually plausible and the details are correct. The later parts of the novel are less believable, but still internally consistent. Overall, I found this a good but not great read.

It s funny, in bad way, when certain sorts of bugs come back to bite you. People may remember the fun debian had with rpath issues around in the late 90s where some binaries wanted to set the path of where their libraries were located. After much contraversy the consensus was at least in Debian (with some specific exceptions) RPATH is bad and don t use it. I lived through that fun and remember hacking libtool or autoconf or some such file. The horror of those times came back to haunt me today while working on procps 3.3.2 packages. lintian was suddenly complaining that binary-or-shlib-defines-rpath for all my binaries. I ve not seen rpath set for many years and as I m also part of the upstream and never saw it set there it took me a while to work out what was going on. Certainly, my binaries were setting the RPATH, it wasn t lintian getting the wrong idea. You can see this with objdump myfile grep RPATH. If you get a hit, you got RPATH set, an empty result means no RPATH. libtool was definitely setting the rpath, with That s not one rpath setting but two, for double insult. Configure flags such as disable-rpath wouldn t work. I flicked back to a pure upstream archive and there was no rpath there in either the compile flags or objdump. There was obviously some sort of side-effect going on. The problem, after much testing, was in the debian/rules file: The problem is above, and all it was was a / . libtool does some checking to see if the library directory exists and if it doesn t silently turns on rpath. It s not smart enough to know that //usr/lib/x86_64-linux-gnu is the same as /usr/lib/x86_64-linux-gnu and so it enables rpath. Removing the slash before the usr fixed the problem. I d say this is bad behavour on libtools part. It should know that / is // or all the other alternative ways of specifying paths. Hopefully this post will stop someone else wasting their time like I did.

Part of our work to make it possible to use UEFI Secure Boot on Linux has been to improve our EFI variable support code. Right now this has a hardcoded assumption that variables are 1024 bytes or smaller, which was true in pre-1.0 versions of the EFI spec. Modern implementations allow the maximum variable size to be determined by the hardware, and with implementations using large key sizes and hashes 1024 bytes isn't really going to cut it. My first attempt at this was a little ugly but also fell foul of the fact that sysfs only allows writes of up to the size of a page - so 4KB on most of the platforms we're caring about. So I've now reimplemented it as a filesystem[1], which is trickier but avoids this problem nicely.

Things were almost working fine - I could read variables of arbitrary sizes, and I could write to existing variables. I was just finishing hooking up new variable creation, but in the process accidentally set the contents of the Boot0002 variable to 0xffffffff 0xffffffff 0x00000000. Boot* variables provide the UEFI firmware with the different configured boot devices on the system - they can point either at a raw device or at a bootloader on a device, and they can do so using various different namespaces. They have a defined format, as documented in chapter 9 of the UEFI spec. At boot time the boot manager reads the variables and attempts to boot from them in a configured order as found in the BootOrder variable.

Now, obviously, 0xffffffff 0x00000000 is unlikely to conform to the specification. And when I rebooted the machine, it gave me a flashing cursor and did nothing. Fair enough - I should be able to choose another boot path from the boot manager. Except the boot manager behaves identically, and I get a flashing cursor and nothing else.

I reported this to the EDK2 development list, and Andrew Fish (who invented EFI back in the 90s) pointed me at the code that's probably responsible. It's in the BDS (Boot Device Selection) library that's part of the UEFI reference implementation from Intel, and you can find it here. The relevant function is BdsLibVariableToOption, which is as follows (with irrelevant bits elided):

BdsLibVariableToOption (
  IN OUT LIST_ENTRY                   *BdsCommonOptionList,
  IN  CHAR16                          *VariableName
  )
 
  UINT16                    FilePathSize;
  UINT8                     *Variable;
  UINT8                     *TempPtr;
  UINTN                     VariableSize;
  VOID                      *LoadOptions;
  UINT32                    LoadOptionsSize;
  CHAR16                    *Description;
  //
  // Read the variable. We will never free this data.
  //
  Variable = BdsLibGetVariableAndSize (
              VariableName,
              &gEfiGlobalVariableGuid,
              &VariableSize
              );
  if (Variable == NULL)  
    return NULL;
   

So so far so good - we read the variable from flash and put it in Variable, Variable is now 0xffffffff 0xffffffff 0x00000000. If it hadn't existed we'd have skipped over and continued. VariableSize is 12.

  //
  // Get the option attribute
  //
  TempPtr   =  Variable;
  Attribute =  *(UINT32 *) Variable;
  TempPtr   += sizeof (UINT32);

Attribute is now 0xffffffff and TempPtr points to Variable + 4.

  //
  // Get the option's device path size
  //
  FilePathSize =  *(UINT16 *) TempPtr;
  TempPtr      += sizeof (UINT16);

FilePathSize is 0xffff, TempPtr points to Variable + 6.

  //
  // Get the option's description string size
  //
  TempPtr     += StrSize ((CHAR16 *) TempPtr);

TempPtr points to 0xffff 0x0000, so StrSize (which is basically strlen) will be 4. TempPtr now points to Variable + 10.

  //
  // Get the option's device path
  //
  DevicePath =  (EFI_DEVICE_PATH_PROTOCOL *) TempPtr;
  TempPtr    += FilePathSize;

TempPtr now points to Variable + 65545 (FilePathSize is 0xffff).

  LoadOptions     = TempPtr;
  LoadOptionsSize = (UINT32) (VariableSize - (UINTN) (TempPtr - Variable));

LoadOptionsSize is now 12 - (Variable + 65545 - Variable), or 12 - 65545, or -65533. But it's cast to an unsigned 32 bit integer, so it's actually 4294901763.

  Option->LoadOptions = AllocateZeroPool (LoadOptionsSize);
  ASSERT(Option->LoadOptions != NULL);

We attempt to allocate just under 4GB of RAM. This probably fails - if it does the boot manager exits. This probably means game over. But if it somehow succeeds:

CopyMem (Option->LoadOptions, LoadOptions, LoadOptionsSize);

we then proceed to read almost 4GB of content from uninitialised addresses, and since Variable was probably allocated below 4GB that almost certainly includes all of your PCI space (which is typically still below 4GB) and bits of your hardware probably generate very unhappy signals on the bus and you lose anyway.

So now I have a machine that won't boot, and the clear CMOS jumper doesn't clear the flash contents so I have no idea how to recover from it. And because this code is present in the Intel reference implementation, doing the same thing on most other UEFI machines would probably have the same outcome. Thankfully, it's not something people are likely to do by accident - using any of the standard interfaces will always generate a valid entry, so you could only trigger this when modifying variables by hand. But now I need to set up another test machine.

[1] All code in Linux will evolve until the point where it's implemented as a filesystem.

comments

By now hopefully you got a chance to see my Jeopardy! episode so it won't be much of a spoiler to mention that I didn't win. It went like this. I flew into LA on the night of October 31st. The taping was to take place on the next two days. Jeopardy! tapes two weeks of shows, two days a week for a total of 10 episodes. Contestants are randomly chosen for one of those 10 episodes but you don't find out which one you will be on until the last minute. Randomness and secrecy have become become a big part of the production process in games since the quiz show scandals of the 1950s. (Interesting fact: it is a Federal felony to tamper with a TV game show.) There were a group of just under 20 0f us altogether waiting to go on. All very nice people. Here in the New York area one tends to equate very intelligent people with "Type A" personalities but we got on well together. Interspersed between the tapings, we had short practice sessions to familiarize ourselves with the studio setup and this is where I first started getting a little apprehensive. Jeopardy! is not just about general knowledge, there is a physical element too in that you have to buzz in by pressing a button on a stick which is wired up to some producers console. You can't buzz in before Alex has finished reading the question and obviously you can't buzz in if someone is already answering the question. If you buzz at the wrong time, your buzzer locks for 0.75 seconds so just jamming your finger on the button. I didn't even get that far as on the entire first day I failed to buzz in at all despite changing hands, changing grip different ways and other strategies. I was pretty despondent by the second day but then miraculously I managed to get the hang of it. So when I was finally selected (for the 3rd game of that day.) I was back to my usual cool self. I was up against Dave Leach and Nicholas Campiz who were returning after having tied in the previous game. And the game did start well enough. Oh one thing you may notice is that Alex mispronounces my name Jaldhal. I knew he did that once or twice and even mentioned it to a producer but it was only viewing the episode right now that I see he does it consistently. I assumed such things would be fixed up in post-editting but I guess not. Oh well, after living in this country for so many years I've learned to be happy whenever two consonants are more or less in the right place. This particular mispronunciation is actually fairly common, even Indian people have been known to make it. The alternative theory is that Alex couldn't read my horrible handwriting which is also plausible. Anyway, I digress. I started off fairly well. My first mistake was in a category about alcoholic drinks. I answered vermouth when the correct response was gin. In my defense I'm a strict teetoler so none of these words mean anything to me. I also earned the disapproval of my superhero fan son by mistaking Dr. Strange for Dr. Doom. I did get the one about Spiderman villain Dr. Octopus so that mollified him somewhat. The highlight of my evening was strangely enough in a category about books of the Bible. I got a Daily Double. It was a risky move but I have always wanted to make a "true" daily double so I threw caution to the winds and wagered everything I had. To my surprise as much as anyones, got the answer right doubling my score. I was in a pretty good position for the Double Jeopardy but thats when Dave started dominating. If you haven't been watching previous games, Dave Leach is arguably the best player this season. Not only is he smart but he is almost supernaturally fast on the buzzer. You can't see because I kept my hands behind the podium but I again and again I would desperately try to ring in on a question I knew the answer to only to be beaten to the punch by Dave. This caused me to start getting a bit reckless and answering questions where I was less than certain which cost me dearly. One particularly dumb mistake was in a category where all the answers were supposed to start with the letter g. I said compass when I should said gyrocompass but actually I should have left that category alone entirely. By the time Final Jeopardy came around winning was no longer an option as Dave was too far in the lead. I was slightly ahead of Nicholas in 2nd place. This is where I again made a strategic error. My original plan to bet nothing in Final Jeopardy but the category was "1930s Novels" and I liked my chances. The clue was read out and it involved an anti-war novel whose author was blacklisted. I was elated. It had to be "All Quiet on the Western Front" right? The blacklist bit gave me pause as the author Erich Maria Remarque was German and this was obviously a reference to the anti-Communist blacklist in the 1950s USA. But somehow I remembered that he had spent the end of his life in Hollywood or maybe I just convinced myself that it was plausible. Evidently my thought process wasn't too crazy because all three of us came up with the same answer. All three of us were wrong. The correct answer was "Johnny Got His Gun" by Daltan Trumbo. In hindsight I knew this but it illustrates the strange ways the mind works. I knew because the video for Metallicas song "One" from "...And Justice for All" is based on the film adaptation of this book. However in a previous orientation session the lady who is the contestant coordinator for Jeopardy! had mentioned AQOTHWF as anexample of something and this got stuck in my brain crowding out the previously known fact. So when all was said and done I ended up in 3rd place. (I bet big whereas Nicholas was more conservative.) Oh well. Despite not winning I had a blast. I fulfilled my dream of appearing on Jeopardy! I met Alex Trebek. (And I should be receiving an autographed picture of us together soon.) My witty banter was sufficiently witty. I made enough money to cover my travel expenses and still have a nice amount left. And I got to meet lots of interesting and wonderful people. So I'm satisfied.

A discussion a couple of days ago about DPI detection (which is best summarised by this and this and I am not having this discussion again) made me remember a chain of other awful things about consumer displays and EDID and there not being enough gin in the world, and reading various bits of the internet and wikipedia seemed to indicate that almost everybody who's written about this has issues with either (a) technology or (b) English, so I might as well write something.

The first problem is unique (I hope) to 720p LCD TVs. 720p is an HD broadcast standard that's defined as having a resolution of 1280x720. A 720p TV is able to display that image without any downscaling. So, naively, you'd expect them to have 1280x720 displays. Now obviously I wouldn't bother mentioning this unless there was some kind of hilarious insanity involved, so you'll be entirely unsurprised when I tell you that most actually have 1366x768 displays. So your 720p content has to be upscaled to fill the screen anyway, but given that you'd have to do the same for displaying 720p content on a 1920x1080 device this isn't the worst thing ever in the world. No, it's more subtle than that.

EDID is a standard for a blob of data that allows a display device to express its capabilities to a video source in order to ensure that an appropriate mode is negotiated. It allows resolutions to be expressed in a bunch of ways - you can set a bunch of bits to indicate which standard modes you support (1366x768 is not one of these standard modes), you can express the standard timing resolution (the horizontal resolution divided by 8, followed by an aspect ratio) and you can express a detailed timing block (a full description of a supported resolution).

1366/8 = 170.75. Hm.

Ok, so 1366x768 can't be expressed in the standard timing resolution block. The closest you can provide for the horizontal resolution is either 1360 or 1368. You also can't supply a vertical resolution - all you can do is say that it's a 16:9 mode. For 1360, that ends up being 765. For 1368, that ends up being 769.

It's ok, though, because you can just put this in the detailed timing block, except it turns out that basically no TVs do, probably because the people making them are the ones who've taken all the gin.

So what we end up with is a bunch of hardware that people assume is 1280x720, but is actually 1366x768, except they're telling your computer that they're either 1360x765 or 1368x769. And you're probably running an OS that's doing sub-pixel anti-aliasing, which requires that the hardware be able to address the pixels directly which is obviously difficult if you think the screen is one size and actually it's another. Thankfully Linux takes care of you here, and this code makes everything ok. Phew, eh?

But ha ha, no, it's worse than that. And the rest applies to 1080p ones as well.

Back in the old days when TV signals were analogue and got turned into a picture by a bunch of magnets waving a beam of electrons about all over the place, it was impossible to guarantee that all TV sets were adjusted correctly and so you couldn't assume that the edges of a picture would actually be visible to the viewer. In order to put text on screen without risking bits of it being lost, you had to steer clear of the edges. Over time this became roughly standardised and the areas of the signal that weren't expected to be displayed were called overscan. Now, of course, we're in a mostly digital world and such things can be ignored, except that when digital TVs first appeared they were mostly used to watch analogue signals so still needed to overscan because otherwise you'd have the titles floating weirdly in the middle of the screen rather than towards the edges, and so because it's never possible to kill technology that's escaped into the wild we're stuck with it.

tl;dr - Your 1920x1080 TV takes a 1920x1080 signal, chops the edges off it and then stretches the rest to fit the screen because of decisions made in the 1930s.

So you plug your computer into a TV and even though you know what the resolution really is you still don't get to address the individual pixels. Even worse, the edges of your screen are missing.

The best thing about overscan is that it's not rigorously standardised - different broadcast bodies have different recommendations, but you're then still at the mercy of what your TV vendor decided to implement. So what usually happens is that graphics vendors have some way in their drivers to compensate for overscan, which involves you manually setting the degree of overscan that your TV provides. This works very simply - you take your 1920x1080 framebuffer and draw different sized black borders until the edge of your desktop lines up with the edge of your TV. The best bit about this is that while you're still scanning out a 1920x1080 mode, your desktop has now shrunk to something more like 1728x972 and your TV is then scaling it back up to 1920x1080. Once again, you lose.

The HDMI spec actually defines an extension block for EDID that indicates whether the display will overscan or not, but doesn't provide any way to work out how much it'll overscan. We haven't seen many of those in the wild. It's also possible to send an HDMI information frame that indicates whether or not the video source is expecting to be overscanned or not, but (a) we don't do that and (b) it'll probably be ignored even if we did, because who ever tests this stuff. The HDMI spec also says that the default behaviour for 1920x1080 (but not 1366x768) should be to assume overscan. Charming.

The best thing about all of this is that the same TV will often have different behaviour depending on whether you connect via DVI or HDMI, but some TVs will still overscan DVI. Some TVs have options in the menu to disable overscan and others don't. Some monitors will overscan if you feed them an HD resolution over HDMI, so if you have HD content and don't want to lose the edges then your hardware needs to scale it down and let the display scale it back up again. It's all awful. I recommend you drink until everything's already blurry and then none of this will matter.

comments

A few days ago, I pushed out version 2.11.13 of the Geode X driver. This is the driver used by the OLPC XO-1, by a plethora of thin clients such as the ThinCan, by low-power desktops such as the Linutop and by a few notebooks such as the eCaf EC800. While this release indeed features a few bugfixes (mainly to keep this driver compilable on the latest X server), the lion's share of the changes involve a complete overhaul of the build scripts, courtesy of Gaetan Nadon. The key motivation for this overhaul was to acknowledge the efforts made by third-party contributors to make the driver compile on BSD variants. Sure enough, this release finally compiles on FreeBSD and, low and behold, on Hurd. However, the hybrid nature of Debian's FreeBSD kernel-based GNU operating system variant posed an additional challenge, because it provides usable support for Video for Linux version 2 (V4L2), but without the full complement of Linux definitions. This resulted in one post-release commit (included in Debian and Ubuntu package 2.11.13-2), following which my assertion that we could now safely define this package as Architecture: any-i386 finally proved to be a safe one for Debian-based operating systems. At this point, I'm curious as to how many more operating systems, especialy BSD variants, can finally use this driver. Patches to further improve support of non-Linux systems are welcome.

I ve recently been reading the Sociological Images blog [1]. That site has lots of pictures and videos that are relevant to the study of Sociology (most of which have a major WTF factor) and it s run by people who have Ph.Ds in Sociology so the commentary is insightful. Since reading that I ve started photographing relevant things. I can t work out the logic behind the above advert for Energy Watch which was on a billboard near Ringwood Station in Melbourne, Australia. The only thing that it clear is that it spreads bad ideas about mental illness and psychiatric treatment. It doesn t make me want to do business with them. The above picture is a shop-front for the Antons clothing store (I m not sure if they are a tailor or if they sell ready to wear). It was taken on Lonsdale St, Melbourne where the store apparently used to be, now they are in Melbourne Central. The above pictures show more detail. Unfortunately the combination of lighting and my camera (Xperia X10 phone camera) wasn t adequate to show the apparent ethnic differences between the two men. It seems that the most likely Australian interpretation of the ethnic groups that are represented are African (maybe Afro-American), Southern-European or maybe American Hispanic, North-Western European, and Japanese. It s good to have mannequins representing the fact that not everyone in Australia is white, but different facial expressions for different races seems a strange choice (admittedly it might be a choice made by mannequin manufacturers). Also the Japanese woman with fan idea is rather outdated. I ve just started reading You May Ask Yourself: An Introduction to Thinking Like a Sociologist (Second Edition) by Dalton Conley. I ve only read the first chapter, but that was good enough that the entire book has to be good enough to recommend.

• [1] http://thesocietypages.org/socimages

I have recently switched my http server from Apache to nginx. The switch was mostly painless thanks to the good documentation available on the nginx wiki. I have been able to move all my public virtual hosts, and have already noticed the improvement on how fast the pages load. Here is a sample graph of the disk I/O rates of my linode showing before/after the switch to nginx: I only had an issue with StatusNet due to the default behavior of spawn-cgi to not spawn any PHP child except if told otherwise. I also had to fix the OpenID plugin for WordPress, as per this commit. For now, I keep apache running for the DSPAM Web UI, which requires suexec. I know I can easily run the CGI through a different user, but still have to polish my setup for FastCGI.

Symbol files are a nice little feature to reduce dependencies of packages. Before there were symbol files libraries in Debian just had shlibs files (both to be found in /var/lib/dpkg/info/. A shlibs file says for each library which packages to depend on when using this library. When a package is created, the build scripts will usually call dpkg-shlibdeps, which then looks which libraries the programs in the library use and then calculate the needed dependencies. This means the maintainers of most packages do not have to care what libraries to depend on, as it is automatically calculated. And as compiling and linking against a newer version of a library can cause the program to no longer work with an older library, it also means those dependencies are correct regardless of which version of a library is compiled against. As shlibs files only have one dependency information per soname, that also means they are quite strict: If there is any possible program that would not work with an older version of a library, then the shlibs file must pull in a dependency for the newer version, so everything needing that library ends up depending on the newer version. As most libraries added new stuff most of the time, most library packages (except some notable extremely API stable packages like for example some X libs) just chose to automatically put the latest package version in the shlibs file. This of course caused library packages to be quite strict: Almost every package depended on the latest version of all libraries, including libc, so practically no package from unstable or testing could be used in stable. To fix this problems, symbols files were introduced. A symbols file is a file (also finally installed in /var/lib/dpkg/info/ alongside the shlibs file) to give a minimum version for each symbol found in the library. The idea is that different programs use different parts of a library. Thus if new functionality is introduced, it would be nice to differentiate which functionality is used and give dependencies depending on that. As the only thing programmatically extractable from a binary file is the list of dynamic symbols used, this is the information used for that. But this only means the maintainer of the library package has now not only one question to answer ("What is the minimal version of this library a program compiled against the current version will need?"), but many questions: "What is the minimal version of this library a program compiled against the current version and referencing this symbol name will need?". Given a symbols file of the last version of a library package and the libraries in the new version of the package, there is one way to catch obvious mistakes: If a symbol was not in the old list but is not in the current library, one needs at least the current version of the library. So if dpkg-gensymbols finds a missing symbol, it will add it with the current version. While this will never create dependencies too strict, it sadly can have the opposite effect of producing dependencies that are not strict enough: Consider for example some library exporting the following header file:

enum foo_action   foo_START, foo_STOP ;
void do_foo_action(enum foo_action);

Which in the next version looks like that:

enum foo_action   foo_START, foo_STOP, foo_RESTART ;
void do_foo_action(enum foo_action);

As the new enum value was added at the end, the numbers of the old constants did not change, so the API and ABI did not change incompatibly, so a program compiled against the old version still works with the new one. (that means: upstream did their job properly). But the maintainer of the Debian package faces a challenge: There was no new symbol added, dpkg-gensymbols will not see that anything changed (as the symbols are the same). So if the maintainer forgets to manually increase the version required by the do_foo_action symbol, it will still be recorded in the symbols file as needing the old version. Thus dpkg will not complain if one tries to install the package containing the program together with the old version of the library. But if that program is called and calls do_foo_action with argument 2 (foo_RESTART), it will not behave properly. To recap:

• Symbols files are a way to minimize dependencies by looking at which symbols of a library a program uses.
• A symbol file lists for each symbol the minimal version of the library a program referencing that symbol needs
• dpkg-gensymbols can detect added functionality if it adds new symbol names
• it's the duty of the maintainer to ensure added functionality not visible by symbol references is properly recorded in the symbols file

"Static program analysis" is a technique whereby a program is verified for errors without actually running it. Finding bugs manually with a debugger and one's brain is tedious, so every shortcut that can help you avoid having to do so is great. There are some commercial tools available to do such analysis, some of which are rather expensive; but there are also some open source tools available to do similar things. One of these is built into clang, the C compiler of the LLVM project. Using it is fairly simple. Instead of compiling something with 'make', compile it with 'scan-build make'. This will set the CC (and similar) environment variable(s) so that before the compiler is ran, the clang static-analysis checker is run over the very same source code. The output of this checker is an HTML file with your source, but with comments added to explain the bugs which the tool found. What does that mean? well, let's look at an example, shall we? <iframe height="500px" src="http://barbershop.grep.be/~wouter/scan-view-example#LN1493" width="80%"></iframe> The iframe above contains one out of three reports produced by a scan-build run over the nbd source code (if there's no iframe, someone scrubbed some HTML in your RSS reader. Just follow the link just above instead). The other two are 'dead assignments', which might mean that I'm currently depending on some undefined behaviour (which would be bad), or it might mean that I'm being overly cautious (which makes my code future-proof, which would be good), or it might mean something else I still need to investigate. But this one is pretty interesting. In the example, there are eight, numbered, comments in the source code. The first seven show the code path which scan-build took through my code before getting at the eighth comment; and the eighth is where things go bad. In this case, when going through the function as shown, we have a NULL pointer dereference. When looking at the scan-build output, it's important to realize a few things. First, the code path shown may be just one of a number of possible code paths. For instance, the null pointer dereference would still happen if the phase function parameter would not contain the NEG_INIT bit with the client pointer set to NULL. However, clang does not show these other code paths; this is presumably an optimization ("if we've already shown that this kind of bug is possible at a particular location through one code path, don't bother recording future instances of that very same bug at the exact same location through another code path"). This means that sometimes, some of the branches shown may be completely irrelevant to the bug at hand. In this particular case, in fact, it's possible to show the bug with just the eighth comment; the first seven are in fact totally irrelevant. Second, the fact that the clang static analyser found a bug does not mean that it's possible to crash the application. Yet. In this particular case, the negotiate function will never be called with the NEG_INIT or NEG_MODERN bits not set, and with the client parameter set to NULL. That's an implicit assertion; there are a few ways in which this function may be called, but the client parameter may only be NULL if NEG_INIT and NEG_MODERN are both set at the same time. Since nbd-server doesn't currently call the negotiate function in that way, it is not currently possible to crash the server by exploiting this bug. But that doesn't mean it won't ever be possible, nor that there isn't a bug in the code. We may assume that the above rules are true, but we never check it. Adding an assertion to that effect should make sure that no future change to the code will accidentally introduce that error and cause a NULL pointer dereference. Is this a silly and useless precautionary measure? Not really. Usually, bugs happen in code not because someone wasn't thinking straight, but because there's so much going on inside a piece of software that it can be too much for any programmer to remember. If a function assumes that its parameters are within a given subset of all possible states, but does not check that this is in fact true, then when (not if) some future change incorrectly introduces a state that is outside of the assumed states, things will break. And that's Bad(TM).