On the testing and unstable systems that i run, i occasionally get a disturbing "there are N newly obsolete packages" message. For example

[0 root@squeak ~]# aptitude update
Hit http://cmrg.fifthhorseman.net unstable Release.gpg
 [...]
Get:42 http://ftp.us.debian.org unstable/main [2077kB]                          
Fetched 15.2MB in 57s (262kB/s)                                                 
Reading package lists... Done
Updating debtags database...... Done
Current status: 3 updates [+3], 1136 new [-21212].
There are 1804 newly obsolete packages.
[255 root@squeak ~]# 

Usually, i can fix this with another aptitude update. Does this mean i've hit a mirror at an inopportune time? Or do i have apt mis-configured somehow? Other people have documented this before, and casual conversation with friends lets me know it's not just me. If this is due to some inconsistency at the mirrors (particularly for suites with rolling updates?), is there some way to engineer the mirror transfers so that this doesn't happen? For example (i have no idea how this stuff is done, so i'm just making this up):

1. rsync the pool/ without --delete to all mirrors,
2. rsync the Packages and Release, and DiffIndex files to all mirrors,
3. rsync the pool/ with --delete

Or should apt itself deal with these circumstances differently? It seems like signed apt ought to be able to detect when something like this is amiss. I'm reluctant to consider the situation acceptable, because i'd like to be able to trust apt in general. In particular, i can imagine apt eventually becoming more assertive about suggesting removal of newly obsolete and unsupported packages. If it is wrong about what's obsolete and unsupported, this could be dangerous.

The newspapers brought good news bits in the last two days. P blico reports on Paco Rivi re's ongoing quest to get a refund for the extra money he had to pay for a Windows licence when he bought a laptop. Paco is a well known member of the Ubuntu Catalan community and has been battling for this common-sense right for 3 years. The trial took place last Monday, and hopefully he'll be able to report some good news soon. In totally unrelated news, the Valencian caveman Juan Garc a Sentandreu, leader of the right-wing Coalici n Valenciana party, was arrested yesterday, for still not too clear reasons. Being one of the biggest enemies of my language, and having a long record of violent attacks to cultural entities and other political parties in Val ncia, I can't say I pitty him at all. I hope he had fun sleeping with the yonkis in the central police station last night. :)

I maintain a bunch of SSL certificates, mostly signed by my own site authority. Too many not to automate, but not enough to warrant heavy machinery. Here's how I do it. The configuration files Each certificate needs a config to describe what's in it. I create each of these and name it with a .cnf suffix. Here's an example:

[ req ]
prompt                  = no
distinguished_name      = server_distinguished_name
[ server_distinguished_name ]
commonName              = server.usefulinc.com
stateOrProvinceName     = England
countryName             = GB
emailAddress            = edd@usefulinc.com
organizationName        = Useful Information Company
organizationalUnitName  = Hosting
[ req_extensions ]
subjectAltName=edd@usefulinc.com
issuerAltName=issuer:copy
nsCertType            = server
[ x509_extensions ]
subjectAltName=edd@usefulinc.com
issuerAltName=issuer:copy
nsCertType            = server

Let's say this config is server.cnf. I then just type make server.pem to generate the corresponding certificate and key, signed by my local certificate authority. As I don't want to attend the startup of every service, I ensure the key is password-less. The Makefile rules Here are the makefile steps I use to generate and sign keys.

.SUFFIXES: .pem .cnf
.cnf.pem:
        OPENSSL_CONF=$< openssl req -newkey rsa:1024 -keyout tempkey.pem -keyform PEM -out tempreq.pem -outform PEM
        openssl rsa <tempkey.pem >  basename $< .cnf _key.pem
        chmod 400  basename $< .cnf _key.pem
        OPENSSL_CONF=./usefulCA/openssl.cnf openssl ca -in tempreq.pem -out  basename $< .cnf _crt.pem
        rm -f tempkey.pem tempreq.pem
        cat  basename $< .cnf _key.pem  basename $< .cnf _crt.pem > $@
        chmod 400 $@
        ln -sf $@  openssl x509 -noout -hash < $@ .0

The resultant files are:

• server.pem — contains both certificate and key in one file
  
• server_crt.pem — certificate file
  
• server_key.pem — key file
  

Some notes on these steps: my site-local certificate authority is in the directory usefulCA, along with an OpenSSL config which describes my preferences. This config was created by copying and making appropriate adjustments to the default /etc/ssl/openssl.cnf which ships with Debian. For generating certificate signing requests to ship to a commercial certificate authority, it's a bit simpler. I save the config files with a .reqcnf suffix instead, and use this rule:

.SUFFIXES: .pem .cnf .reqcnf .csr
.reqcnf.csr:
        OPENSSL_CONF=$< openssl req -newkey rsa:1024 -keyout  basename $< .reqcnf .key -keyform PEM -out  basename $< .reqcnf .csr -outform PEM

And finally, a rule I use to sign incoming certificate requests from other systems:

.csr.pem:
        OPENSSL_CONF=./usefulCA/openssl.cnf openssl ca -in $< -out  basename $< .csr _crt.pem

I offer these without warranty in the hope they might be useful to somebody. They're not much more than a transcription of a how-to into a makefile, but it's just enough technology to ensure creating certificates isn't a big nuisance. Further reading

• A more complete guide to setting up a site-local certificate authority
  

Notes Why do I bother with a site-local CA, rather than just self-sign? It lets me bypass the annoyance of SSL warnings on clients once I've installed my own CA certificate, and gives me a coarse grained level of access control: for instance, only clients with certificates signed by my CA are allowed to access the site's LDAP server. My personal next step with this is to integrate the certificate production process with my emerging Puppet recipes for managing local infrastructure.Join the conversation about this post

Newegg.com just had a special on 750Gb Seagate SATA drives, so I thought it might be a good opportunity to replace my 400Gb IDE drives with ones that can be connected to the host by something less ghetto than IDE to USB 2.0 adapters. Who knows? I might even lash out on some kind of enclosure, instead of sitting them on cork trivets on top of the computer they're attached to. So once the drives turn up, all I'll need to do is replace the PCI 4 port USB 2.0 controller with a PCI 4 port eSATA controller. I'm a total SATA ignoramus, so if anyone has any suggestions on what's good, or what to avoid, shoot me an email.

… you receive more than a thousand “Mail Delivery Error” messages in a few hours. Needless to say I sent none of the original emails which I’m getting these error messages for…

Okay, I will again miss DebConf. This time it is not because of time, but because of money. The flight to Buenos Aires is far too expensive to go there just for DebConf and I didn't manage to plan some more travelling around South America... Hopefully it will get better next year, at least DebConf will be much closer :-). PS: Just after deadline for confirmation and my decision for not going there, I got an email that my sponsorship for travel was approved. Unfortunately too late to change my decision.

As the title says - the Ftpmaster dak repository is now using git instead of bzr. I never did like bzr, so I recently asked what the rest of the team thinks of switching to git. Luckily I got no objections, so here we are, with a VCS thats not dead-slow. If you want to do dak development and submit patches you want to read my post to the debian-dak list and then clone the git repo. Sorry, there is no git-daemon support, you need to use the http way if you do not have access to ries. Unfortunately debian-admin thinks git-daemon isn’t good enough to run on ftp-master and that “http is enough”. Bah. (No, I won’t work around it (git-daemon uses high-port, so could be run without admin help, but NO WAY), and I don’t want to put it on alioth). The actual conversion of the repository was done by Dato, with me commenting and asking for stuff. Like the fixup of the old revisions committer names, so all the old CVS revisions now correctly tell the full name/email of the committer, not just the unix login. Or fixing all the bzr commits done by “ajt@ries”, “joerg@ries” and other broken combinations of “login@machine”. I also use his git hook to send commit mails to get commits mailed to the debian-dak lists. [Update] Nice, with the setup we use we found a bug in git. Ha. (a stupid issue with git shared repositories, and reflogs enabled at the same time that basically means you cant commit as the logs got the wrong permissions set whenever someone else did an action, like git gc.) Comments: 2

Ten years ago, most of us wouldn't have dreamt we'd be managing terabits of storage, tens of megabits of bandwidth, arrays of network-distributed services. The height of a programmer's worry would likely be choice of UI toolkit or finding the right way to indent code, and the height of consumer concern deciding which room to put the new computer in. Now the problems associated with managing large networks are becoming real for everyone, right down to the consumer level. Stupendously large amounts of computing resource are available at an instant. Your household probably has more than a terabyte of storage already. Issues such as single sign-on are going to hit home over the next year, as networked computing and entertainment devices profilerate. Features such as Apple's Time Machine will be increasingly vital — software that makes traditionally gnarly sysadmin tasks consumer-friendly. The rebranding of .Mac into "Mobile Me" is also a step in this direction. The impact on developers As software developers, we also have to cope with the effects of this resource-richness. For small sums of money we can get access to large computing clusters, geographically redundant hosting services. Our programs have left the desktop and found their new home on the web. System administration issues loom large upon us, security concerns lurk auspiciously in the corners of our minds. Although the cost of infrastructure has dropped radically, other costs remain high and are going to stay that way. System administrators are not only grumpy, they demand high wages. Commercial software license fees spiral out of control: traditional per-CPU licensing models make little sense when you can quickly bring up tens of machines. The cost in power is already troubling large companies, and there's no reason to suspect the problems won't ripple down. Help is at hand from a variety of technologies. If they don't yet make massive resource management trivial, they at least make it possible. Some of these also inhabit the weird territory of being both the source of a problem and a solution at the same time: virtualization, for example. Distributed revision control systems Distributed revision control is a technology whose time has finally come in popular circles, thanks in part to Linus Torvald's Git system. DRCS has several important impacts on today's developer:

• Branching and, importantly, merging become much cheaper, allowing agile and flexible iterations of development.
• Loosely connected and geographically diverse development becomes much easier. Even within a single organization it is not uncommon to find teams spanning countries, time zones. Complex multi-site VPN setups aren't necessary when a few SSH keys can do the job.
• Revision control becomes packaging on the cheap. Like it or not, the mere tagging of a source tree has now become a valid option for releasing software.

All these trends lower the barrier to entry, increase collaboration and agility of development. You can the value of this as more software tools become free. Selling such tools is rapidly becoming a thing of the past, the advantages of sharing enable the developers at the sharp end to get their jobs done quicker. However, such increased agility and, well, messiness leave other problems to solve, which the next two technologies address. Virtualization Hardware-as-a-service, infrastructure-as-a-service, call it what you will. The ability to create what we used to call entire machines, pick them up and move them around the network is revolutionary, and it's something that will have a real impact on regular developers. The benefits are at several levels.

• Agile infrastructure — a ready supply of new machines makes it a lot easier and cheaper to test different scenarios, architectures, and to separate concerns. If things go wrong, throw away the image and start over. It's all about cutting the administration load.
• A packaging solution — the new macroeconomics of software distribution mean that distributing entire machine images which communicate exclusively via the network is now a feasible way to distribute your software. We must adjust to the notion of distributing appliances, not code. We may mourn the lost crafts of creating RPMs or installers, but let's face it, it's now a waste of time.
• New business models — your application can now be delivered as a black-box appliance, circumventing compatibility issues, or as a service, with virtualization part of the solution to scaling.

Configuration management Computing is a zero-sum game, and despite our increased ability to create and distribute software, problems still exist. We just pushed them to the next level. In good part, this next level is the problem of configuration management. We now have networks and clusters of (virtual) machines, software so agile we need six decimal places to describe its revision levels, and network and authentication paths that are starting to tangle. How do we manage that? One thing developers crave is repeatability. That's why we love our makefiles, autoconf, Ant, rake and so on. It's the one time even the most imperative-minded programmer writes declarative code. We like to say "let the world be like this." Our new sprawling world lacks this feature, and the best of our old toolkits — .debs, RPMs — address things only at the level of packages in a single environment. So developers must look to the world of operations, a territory we probably thought we needn't enter. In this world the new "make" is called Puppet. You write recipes to describe how things ought to be, and Puppet will make it so. I've been spending some time digging into Puppet, and feel excited by the confidence it's giving me. Now my applications exceed single source trees, and single machines, it gives me the means to tie the whole together. This article was going to be solely about Puppet, but that will have to wait now for another time. It's likely you'll have played with virtual machines and distributed revision control, but have you tried Puppet yet? Give it a spin, and let your mind wander over the benefits for your organization and development approaches. Conclusions
For developers and users alike, our world is changing. Hardware, connectivity and increasingly software is becoming cheap or free. The solidity of the old things we put value on — real things you can touch like disks — is eroding. What really matters is our data, our creations, and their communication. If they don't quite yet exist in a universal "cloud" yet, they're certainly getting frisky. As vendors provide solutions for consumers to manage their new domestic infrastructure, developers must look to network-aware toolkits and operations techniques to manage and get the best from their emergent infrastructures. Also on this topic:

• In search of agile infrastructure for web applications (June 2006)
  

Join the conversation about this post

Even though the Debian installer can set up encrypted partitions, it is optimised for systems with a single data partition, unless you want to enter multiple passphrases when the system boots. The installer configures a LUKS volume using cryptsetup, but it provides no mechanism for the use of key files, only interactive passphrases. I like partitioning my disks and use different filesystems for /tmp, /home, /var, and /usr/local for a number of reasons. I don t like entering more passphrases than necessary. If you can identify with that, the following is for you.

Installing the system The first step to setting up an encrypted Debian system is to perform a normal Debian system installation. When you are asked to partition your harddrive and create filesystems, set up all partitions as encrypted volumes (I suggest to go with the installer defaults and use dm-crypt and AES with the default settings, simply because I have no reason to doubt the installer dveeloper team s choices). Make sure to erase the disks in the process the installer has an option for that. Set up the swap partition as an encrypted volume too, but don t worry too much about the settings at this point; we will recreate the swap partition later. Unless you want to boot off an external medium, such as a USB stick, you will need to create an unencrypted partition for /boot. I will return to this topic, which has security implications, further down. The installer will ask you for passphrases for each of the volumes you create. I suggest you pick a secure passphrase for the root volume (/), but simple passphrases for all the other ones (such as a ), since we will reconfigure them to use key files instead.

Using key files A key file is like a passphrase stored in a file on disk; as opposed to what you know , it s a what you have security asset. Thus, you need to store the file somewhere. When I boot up my system, I unlock the root partition with a passphrase entered interactively, which makes the root filesystem available. I store key files for all other volumes in /etc/keys. Obviously, I need to tell cryptsetup to use those. The first step is to create a key file for each partition and to add it as a decryption key to the LUKS volume. You can do all of the following without unmounting the filesystems. See the following example for hda6, which will prompt for the simple passphrase we entered above to unlock the key ring when adding the key file:

umask 077
mkdir /etc/keys
dd if=/dev/urandom of=/etc/keys/hda6.luks bs=4k count=1
cryptsetup luksAddKey /dev/hda6 /etc/keys/hda6.luks
cryptsetup luksKillSlot /dev/hda6 0 --key-file /etc/keys/hda6.luks

The last command wipes the simple passphrase from the key ring and thus makes it unusable. Now we need to tell cryptsetup to use the key file by editing /etc/crypttab and ensuring a line such as the following exists:

hda6_crypt /dev/hda6 /etc/keys/hda6.luks luks

This tells cryptsetup to create the cryptographic volume hda6_crypt from the base device /dev/hda6, using the key file we created above, and letting it know that it s dealing with a LUKS volume. Repeat this for every partition except your root and swap partitions.

Encrypting the swap partition If you are using an encrypted Debian system, you likely have some security requirements to meet. If that s the case, you must also use an encrypted swap partition. The swap partition can be encrypted in two ways:

• it can be recreated on every boot, using a random passphrase, or
• it can be created like the other encrypted volumes with a persistent passphrase

If you want to use suspend-to-disk, you cannot use the first approach as it would overwrite your memory footprint stored in the swap partition. Furthermore, you cannot use a key file like the other partitions, since the root filesystem is not (and must not) be mounted by the time the resume process starts and needs to read the decrypted swap partition. The way I solved this is by telling cryptsetup to compute the passphrase of the swap partition from the decryption key of the volume holding the root filesystem; the cryptsetup package implements this with /lib/cryptsetup/scripts/decrypt_derived. Thus, to set up the swap partition, I do the following, assuming hda2 is the partition holding the encrypted swap and the root filesystem is in hda5_crypt:

swapoff /dev/mapper/hda2_crypt
cryptsetup luksClose hda2_crypt
dd if=/dev/urandom of=/dev/hda2
/lib/cryptsetup/scripts/decrypt_derived hda5_crypt \
    cryptsetup luksFormat /dev/hda2 --key-file -
/lib/cryptsetup/scripts/decrypt_derived hda5_crypt \
    cryptsetup luksOpen /dev/hda2 hda2_crypt --key-file -
mkswap /dev/mapper/hda2_crypt

To tell the system about this swap partition, we need to add it to /etc/crypttab and /etc/fstab; make sure, those files contain lines like the following:

/etc/crypttab:
  hda2_crypt /dev/hda2 hda5_crypt luks,keyscript=/lib/cryptsetup/scripts/decrypt_derived
/etc/fstab:
  /dev/mapper/hda2_crypt swap swap sw 0 0

With this in place, as soon as you configure the system for suspend-to-disk, the swap partition will be automatically set up alongside the root filesystem very early during the boot sequence. To figure out which swap partition to make available at that point, cryptsetup checks the following:

• a line like RESUME=/dev/mapper/hda2_crypt in /etc/initramfs-tools/conf.d/resume
• a resume device setting in /etc/uswsusp.conf (see uswsusp.conf(5))
• an entry in /etc/suspend.conf
• a resume=/dev/mapper/hda2_crypt in the kernel command line

You can inspect /usr/share/initramfs-tools/hooks/cryptroot if you want to know more about this.

Using UUIDs Even though the above is all you have to do, you might want to consider replacing the device paths in /etc/crypttab with persistent ones. One motivation might be the ability to boot off your drive via a USB adapter, which might cause it to appear as /dev/sda instead of /dev/hda. udev is installed by default on Debian systems and it makes persistent links to partitions available under /dev/disk/by-uuid, using the UUID of the content structure (e.g. the LUKS header). It uses /lib/udev/vol_id to determine those link names. If you replace the entries in /etc/crypttab, make sure to update the initramfs (update-initramfs -u -k all) and consider to use the same approach for the /boot filesystem in /etc/fstab; you ll note that all other filesystems use persistent device paths thanks to the dm-crypt layer. If you ever end up booting off the disk through a USB adapter, you might face the problem where the usb_storage subsystem takes too long to activate, so that the cryptsetup script does not find the devices in time. You can either solve this by adding the rootdelay=x parameter or break=mount to the kernel command line. The first will cause the scripts to wait x seconds before trying to configure and mount the root filesystem; the second would give you a shell that you can exit as soon as the kernel spouted its device initialisation messages at you. If you got this far, you might even want to take it further and replace hda5_crypt with cr_root or the like to abstract those silly partition numbers away even further. This is easier than it sounds, but does require several steps. Do not do this if you re not comfortable reviving your system in case it fails to come back up!

1. replace the old names with the new names in /etc/crypttab for all volumes except the root volume. If you are using a derived passphrase for the swap partition, make sure to put the new name for the root volume into the third column of the swap partition s configuration line.
2. modify /etc/fstab accordingly, leave the root filesystem s device path alone.
3. call update-initramfs -u
4. modify your bootloader to ask the kernel to boot off the new root volume.
5. replace the root volume s old name with the new one in /etc/fstab.
6. reboot, and add break=mount to the kernel command line.
7. at the busybox prompt, edit (vi) /conf/conf.d/cryptroot and change the first field of the root volume s line to the new name.
8. exit the shell and watch the boot process complete.
9. finally, replace the root volume s old name with the new one in /etc/crypttab.

If the system fails to boot up again, you can use the backup initial ramdisk, which update-initramfs left in /boot.

Security implications Apart from the usual security implications related to cryptosystems, passphrases, mathematics, user stupidity, and so on, the approach I outlined will leave you with a pretty well-secured system. Obviously, you should make sure to lock your screen whenever you leave the system unattended or the entire encryption is basically useless. There are two attack vectors on your system, both involving physical access to the machine:

• theft of the machine or its RAM chip, freezing the latter, and scanning the working memory for the passphrase
• manipulation of the kernel or initramfs stored on the (unencrypted) /boot filesystem, in such a way as to obtain the passphrase. To do this, you would have to have access to my machine and return it to me without me noticing; once I restart and enter the passphrase, you d have to steal it again

Other than that, you should be careful when travelling to totalitarian countries, like the Excited States of America, China, and probably the UK. First off, encryption arouses suspicion, and second, border agents might ask you to decrypt the partitions for them to copy or scan, and refusal to do so might get your turned away at the border. When travelling to those countries, make sure to hide your data properly.

Speed implications Obviously, having your entire system encrypted (including swap) will slow it down. I don t have any quantitative information on that, but after several years of using full-disk encryption on my laptop (an X40, which isn t very powerful), I can say that it remains usable, if you don t rely on disk-intensive operations, such as compiling kernels and the like.

Alternative approaches Several alternative approaches exist, all involving an additional device:

• if you don t like to type in the passphrase for the root filesystem, you could store it on a USB key (or the like); cryptsetup provides /lib/cryptsetup/scripts/passdev, which can be used to deal with such a situation. You can find more information in cryptsetup s README.Debian file.
• if you don t like the unencrypted /boot partition, you could boot the system off a USB key, which you can keep separate from the system except for when booting and upgrading the kernel. All you need to do for that is install the bootloader and kernel onto the device and configure it to use the proper encryption volume on the harddisk as root filesystem.
• cryptsetup also comes with support for PKCS#15 smartcards (opensc and openct).

I have chosen neither of these approaches, because the extra security does not make up for the inconvenience, and the danger of an unbootable system in case of loss of forgetting of the additional device.

So my Esc key broke today. Maybe due to high humidity last night -- the laptop was generating a lot of spurious lid open events earlier. New keyboard ordered, etc, nothing too unusual given the crummy nature of laptop keyboards (Fujitsu's in specific). The interesting thing though, is that even though I've remapped Caps Lock to generate Escape for now, this threw my typing off to an amazing extent. I don't just have trouble typing ^]:wq in vim. I have trouble typing arbitrary sentences in programs that don't use escape at all. Why? I don't touch type the way people are taught to in school, with fingers on the home keys. I touch type the way you learn to by typing a lot, ad-hoc, until your hands learn exactly how to type the common things you throw at them. This includes being able to type ^]:wq as fast as I can type "joey". My home keys are, approximatly:

Tab Q S X Space       Down . ; ' Enter

(Space and Down are covered by my thumbs.) This might seem a bit weird, but as a programmer, I hit tab a lot, and, enter lots of semicolons, periods, and quotes. As a human on the WWW, I scroll down a lot. As a command-line user, Enter is a key I hit a lot. That's my guesses why my fingers go there. This wasn't planned, it evolved. In fact, I have other home key locations, that are used for other activities too. Now that I'm typing in a lot of text here, my WPM has increased a lot, because my hands have moved closer to the center of the keyboard. Something like this.

      W E F V    J I O  

Although every time I've checked, it's been a slightly different set of keys. My hands know where they are, and seem to make pretty good guesses about where I'll want them in the next word and pick a different "home row", depending. Also, if I'm scrolling around a lot, my left hand switches to:

                      Left Up Right

Point being that this is all very dynamic, and unconcious, and very adapted to this exact keyboard. Learning a new keyboard will take me days of fumbling, that are reminiscent of someone describing switching from QWERTY to DVORAK. And, it turns out that even moving a single key like Esc is close to switching to an entire new keyboard. Needing to reach down to Caps Lock for Esc threw everything else off subtly, even when I wasn't typing that key. Clearly this is not the best way to type. It's just my way. :-/

http://www.emdebian.org/buildd/

The scripts are in emdebian-tools 1.1.6 and with a few more fixes and tweaks, that will be the main body of v1.2.0 for Debian unstable.

A few issues with ssh and sudo passphrase caching vs running buildd scripts as root - I'll need to improve the documentation of the process before any other buildds can be set up. The need to have emdebian-tools and subversion installed inside the buildd chroot does make things a little more complicated than a standard wanna-build implementation - as does the need to download, build, install and clean-up cross dependencies.

This autobuilder is the Emdebian target package buildd - Hector Oron has another to build the cross-building toolchains. The toolchain autobuilder has fewer packages but each package is larger and is built for more architectures. Adding more architectures to the target autobuilder is quite difficult - packages that use cache files will need new cache files with different values for certain cache values. See the wiki for info on how to fix cache files. Each solves a different problem within the overall design of any buildd. In each case, instead of listening to email from incoming.debian.org, each buildd picks up the incoming data from the apt cache - a useful little time delay that gives me time to head-off certain problematic updates.

The target package autobuilder runs incrementally - a package is repeatedly rebuilt until it works and is then left alone until the next Debian release. There's lots of work available on the frontend if anyone fancies helping with the PHP. (See Emdebian SVN).

The advantage now is that as more packages fix their cross building bugs, more packages will autobuild without intervention which frees up more of my time to fix the more complicated problems. Equally, I can refer to the buildd logs in bug reports, helping everyone see what is going on.

“Thou shalt not whine” was written under his desk. I just discovered him recently and was presently surprised by his directness in questioning politicians and challenging them to explain things that they have said before. Like a Howard Stern of politics. Not shy to respectfully question the existing assumptions, but (unlike Howard) still remaining neutral and not drawing conclusions himself. He will be missed. P.S. I do however feel the NBC could have included some other news in the Nightly News of the day as well. Dropping everything and just covering one man seams a bit excessive to me.

Pity poor Dr. Ari Jaaksi from Nokia. He gave a talk at the Handsets World conference in Berlin on Tuesday, where according to ZDnet, he lectured Open Source Developers that they needed to learn why DRM and other closed technologies were necessary, because of business issues such as subsidized (device) business models. I suspect he wasn’t prepared for the reaction, which took the form of a major fuss on Slashdot, as well as some declarations from a few people on the maemo-users mailing lists that they would never buy another Nokia device. Dr Jaaski then posted today on his blog an entry entitled, “Some learning to do?”, where he stated that while Nokia needs to learn how the open source world works (not just licenses and legal issues, but also the spirit), that the open source world also needed to learn as well — about WHY things are the way they are. He ended the post with “I’m not a teacher, I’m a learner” — which is fair enough. I actually think that Dr. Jaaksi is right; we all should be interested in learning, and having talked with a number of Nokia employees, I can testify that their hearts are very much in the right place, and there are some strong reasons regarding the demands of carriers, the subsidized business model of handsets in the US, the attitudes of suppliers for components in mobile devices, that very much tie their hands — as well the hands of every other handset company out there. Progress in this space will come slowly, and but I believe there is hope that in the long run things will get better. However, at the same time, I would suggest to Dr. Jaaksi that it’s also important how to communicate with the open source community — or with any community that holds views sometimes with great passion. Many people do not react well to being told that they need to learn something — even if it is true that they do. Admitting that they need to learn is for some people tantamount to admitting ignorance, or even worse admitting stupidity, and so they are loath to do this. Telling them this just makes them defensive and angry. My suggestion for how get people to learn without making them defensive is to use a modified Socratic method. It works especially well with the Open Source crowd, because many of them are engineers, and engineers are by nature problem solvers. So if you give them a problem to solve, they will go to work trying to find potential solutions, and thus start thinking and learning about the problem from a different point of view. Hence, a better approach is to give them a series of business constraints (time to market, the demands of carriers, the fact that most end users prefer to buy subsidized phones with 1-2 year lock-in contracts, the intransigence of suppliers, why a single company can’t afford to create their own chipsets from scratch and have to rely on companies like Broadcom, etc.) and then tell them — we’d love to delight our customers in any way we can, and clearly you are very passionate about DRM, open device drivers, and other OSS issues. We however have to create successful products and sell them at a price such that we don’t go out of business — can you help us work through all of these constraints? Let’s work together so we can make an open source mobile platform that meets the spirit as well as the legal requirements of the open source world! Dr. Jaaksi and his team are struggling with a hard problem, and I am very sympathetic to their issues. And, they have to be commended for investing as much time and effort on the Maemo platform, as well as other Linux and Open Source platforms as they have to date. The strength of the Open Source community is that we work together to solve our mutual problems, while celebrating and giving permission to people to “scratch their own itches”. In order to do that, we need to learn how to better communicate with each other, and that is something that goes in both directions. When I was at MIT as an undergraduate, far too many years ago than I care to admit, I took economics classes and Sloan School of Management classes (even though they were not necessary for a Computer Science degree) for that very reason. I encourage open source advocates to learn how to walk a mile or two in business people’s shoes, since if you want to influence them them to change in the ways that you desire, you need to know where they are coming from — just shouting and throwing rotten tomatoes generally won’t get you very far. On the flip side, people who are on the corporate side of the divide also need to understand that certain ways of engaging the community can be more productive than others. At the end of the day, hopefully we can all agree with Dr. Jaaksi that everyone will benefit if we all commit to learning from each other. Update: I’ve fixed Dr. Jaaksi’s name so it is spelled correctly; my apologies for the error.

There has been a lot of fear-mongering about nanotech. The idea is that little robots will eat people (or maybe eat things that we depend on such as essential food crops). It’s unfortunate that fear-mongering has replaced thought and there seems to have been little serious discussion about the issues. If (as some people believe) nanotech has the potential to be more destructive than nuclear weapons then it’s an issue that needs to be discussed in debates before elections and government actions to alleviate the threat need to be reported on the news - as suggested in the Accelerating Future blog [0]. I predict that there will be three things which could be called nanotech in the future:

1. Artifical life forms as described by Craig Venter in his talk for ted.com [1]. I believe that these should be considered along with nanotech because the boundary between creatures and machines can get fuzzy when you talk about self-replicating things devised by humans which are based on biological processes.
   I believe that artificial life forms and tweaked versions of current life forms have significant potential for harm. The BBC has an interesting article on health risks of GM food which suggests that such foods should be given the same level of testing as pharmaceuticals [2]. But that’s only the tip of the iceberg, the potential use of Terminator Gene technology [3] in biological warfare seems obvious.

But generally this form of nanotech has the same potential as bio-warfare (which currently has significantly under-performed when compared to other WMDs) and needs to be handled in the same way.

The more commonly discussed robotic nanotech, self-replicating and which can run around to do things (EG work inside a human body). I doubt that tiny robots can ever be as effective at adapting to their environment as animals, I also doubt that they can self-replicate in the wild. Currently we create CPU cores (the most intricate devices created by humans) from very pure materials in “clean rooms”. Making tiny machines in clean-rooms is not easy, making them in dirty environments is going to be almost impossible. Robots as we know them are based around environments that are artificially clean not natural environments. Robots that can self-replicate in a clean-room when provided with pure supplies of the necessary raw materials is a solvable problem. I predict that this will remain in science-fiction.

Tiny robots manufactured in factories to work as parts of larger machines. This is something that we are getting to today. It’s not going to cause any harm as long as the nano-bots can’t be manufactured on their own and can’t survive in the wild.

In summary, I think that the main area that we should be concerned about in regard to nano-bot technology is as a new development on the biological warfare theme. This seems to be a serious threat which deserves the attention of major governments.

• [0] http://www.acceleratingfuture.com/michael/blog/?p=830


• [1] http://www.ted.com/talks/view/id/227


• [2] http://news.bbc.co.uk/1/hi/health/background_briefings/food_safety/278240.stm


• [3] http://en.wikipedia.org/wiki/Terminator_gene

Share This

My project for the past three days has been to upgrade our etch-based Debian build system to add support for Ubuntu hardy. We're primarily a Debian shop, but we want to use Ubuntu for the public timeshare systems due to its faster stable update cycle. In that environment, the latest and greatest software and active security support are both important simultaneously, and server-level stability isn't as vital. Ubuntu wins over other options because it's basically Debian and therefore shares with Debian the huge package selection, which is important for university timeshare systems. (We also use the external R package repository; we have a lot of heavy R users.) First, FAI has changed a lot since the etch release and hardy has a far more recent version of fai-client, which means that just bootstrapping an Ubuntu NFS root on an etch system and trying to install it with the etch FAI doesn't work due to the mismatch between the server expectations and the fai-client and fai-nfsroot packages. FAI during the etch release boots with a traditional read-only NFS root and works around the fact that its boot device is read-only. Current FAI does essentially the same trick that a live CD does, using unionfs to allow local modifications to the NFS root with a backing tmpfs file system, which is kind of scary but allows FAI to be much simpler. The first thing I tried was debootstrapping Ubuntu (using the debootstrap from backports.org) and then downgrading its fai-client and fai-nfsroot packages to the etch versions, but changes in other part of the system (particularly mount) makes that not very viable. One of the nice things about FAI is that it's basically a collection of shell scripts, which means that it's architecture-independent and one can easily install the unstable packages on an otherwise-etch system. So I upgraded fai-server to the current Debian sid version of FAI and installed the sid versions of fai-nfsroot and fai-client into the Ubuntu NFS root after generating it with make-fai-nfsroot as normal. That works much better. However, there are a few Ubuntu-specific problems to work around (which are patched in the Ubuntu version of FAI, but I don't want to have to track down an Ubuntu system to generate the NFS roots when the rest of our infrastructure is etch):

• Ubuntu hardy ships with a unionfs that can't write to existing files, only create new files. I'm not sure whose bug this is, possibly upstream, but etch unionfs doesn't have this problem. The workaround is to delete the files that FAI wants to overwrite out of the NFS root after generating it with make-fai-nfsroot, which you can do in a hook or by wrappering it and doing the deletions afterwards (we do the latter because we have other local tweaks we make). You want to delete /etc/resolv.conf, /etc/syslog.conf, and auth.log, syslog, daemon.log, kern.log, lpr.log, mail.*, user.log, debug, and messages in /var/log.
• Ubuntu hardy defaults to using upstart, which completely fails in an FAI NFS root environment. I'm not sure why, but first upstart claims the respawn line of every file in /etc/event.d/* is a syntax error (?!) and, even if that's fixed, doesn't actually spawn a getty (at least on a serial console). The implication is that you can't Ctrl-C at the end of the build and work on the system before rebooting, which makes diagnosing problems very difficult. I tried a bunch of different possibilities for an event.d file (and wow is that ever either underdocumented or obscurely documented — I've still never found a man page or other document clearly explaining the possible configuration options) before giving up. The Ubuntu FAI package forces installation of sysvinit in the NFS root, and that works fine (with the normal inittab tweaking to spawn a getty on the serial console). upstart works fine on the newly built system, including serial console, so I'm not sure why the NFS root is broken.

Other than that, most of the work was upgrading the whole system to using the new boot method (changing to boot=live on the PXE kernel options, including live-initramfs in the NFS root, copying the kernel and initrd out of the NFS root into the tftp area instead of using fai-kernels, and adding initrd to the PXE kernel options). I'm now successfully building etch and hardy systems using our FAI setup and will probably upgrade the production service next week. There is one other fairly serious Ubuntu design misfeature that is only tangentially related but that cost me a lot of time. Ubuntu has modified update-grub to make it use ucf, which breaks the traditional update-grub semantics in extremely strange and confusing ways that took hours for me to diagnose. For those who aren't familiar with the details of how update-grub works, it uses comments in /boot/grub/menu.lst as a template to generate the kernel list at the end of menu.lst. It's not the best design, since it uses the same file as both a template and the active configuration file, but it works, and has worked the same way for many years. It's simple and reliable: you only change the defaults in the comments and then update-grub adds in the kernel-specific information and generates the active menu for you. If you want to edit the kernel list yourself for some reason, don't use update-grub; it's completely optional. To see how using ucf breaks this, remember that ucf's goal is to preserve local changes to a configuration file, but its notion of local changes is fundamentally contrary to how menu.lst works. Suppose that you're a large organization with a lot of systems with a variety of kernel revisions, for one reason or another, but with a desire to have uniform kernel parameters like console speed and timeouts. With update-grub, you can just copy your menu.lst template to all of your systems and run update-grub afterwards, and update-grub will apply your new defaults and generate a new kernel list using them. However, with Ubuntu's update-grub, it generates the initial menu.lst and kernel list, and then when you replace that with a fresh template, ucf sees that you've removed the kernel list. It thinks that's a local modification, and when update-grub generates a new kernel list, it "preserves" your modification and leaves the kernel list out of the new menu.lst. Tada, instant unbootable system. Even worse, this bug is sticky: you can re-run update-grub all you want, it succeeds without errors, and it keeps generating an empty menu.lst. This is a completely mystifying error until you dig into the details of what's going on. To recover, you have to run ucf --purge /var/run/grub/menu.lst (and that path, or for that matter any of the ucf stuff, is completely undocumented in the update-grub man page) and then re-run update-grub, at which point ucf will display a debconf prompt (which doesn't support debconf preseeding, but that's another problem). If you then select three-way merge (not the default) from that prompt, you get a correct menu.lst. I'm not sure why Ubuntu made this change, but as near as I can tell, it's fundamentally architecturally broken. The specific problem I had was that we were copying our template over and re-running update-grub in a post-install script, resulting in a system that always had an empty kernel menu and wouldn't boot. Once I finally figured out what's going on, I worked around it by being very careful to put our template in place before the initial system bootstrap and then never changing menu.lst afterwards, but this is a bad limitation for a large site that scales system administrators by automating configuration file updates. But maybe I'm missing some subtlety. I'll go file a Launchpad bug shortly after posting this and we'll see what happens.

Over the last week I have received five phone calls from Wyndham Resorts asking if I would like to be surveyed. Every time I tell them that I am not going to do their survey, on all but one call I had to repeatedly state that I would not do the survey for more than two minutes before they would go away. The advantage of phone spam over email spam is that the caller pays, I guess that they have a time limit of three minutes when calling a mobile phone to save on calling costs. There have been a number of proposals for making people pay for email to discourage spam. Even a cost of a few cents a message would make spam cease being economically viable for a mass audience (a smaller number of targeted spams would be easier to block or delete). But such plans to entirely change the way email works have of course failed totally. But for phones it could work. I’d like to have a model where anyone who calls me has to pay an extra dollar per minute which gets added on to their phone bill. When people who I want to talk to call me I could reimburse them (or maybe be able to give my phone company a list of numbers not to bill). This could also seamlessly transition to commercial use. I would be happy to accept calls from people asking for advice about Linux and networking issues for $1 per minute. With all the people who call me about such things for free already it would be good to answer some questions for money.

Share This

While free speech rights are valued and recognized in most liberal democracies, the degree and scope of these rights are by no means uniform. The United States, which has the most expansive free speech protections in the world, actually stands apart as this New York Times article on the subject makes clear. The article covers familiar ground (at least for those who follow free speech debates) but it does so well and discusses an interesting case now unfolding in Canada, concerning a pretty inflammatory magazine article denouncing Islam. The magazine is currently under trial for violating provincial hate laws.

A common myth in the computer industry seems to be that ECC (Error Correcting Code - a Hamming Code [0]) RAM is only a server feature. The difference between a server and a desktop machine (in terms of utility) is that a server performs tasks for many people while a desktop machine only performs tasks for one person. Therefore when purchasing a desktop machine you can decide how much you are willing to spend for the safety and continuity of your work. For a server it’s more difficult as everyone has a different idea of how reliable a server should be in terms of uptime and in terms of data security. When running a server for a business there is the additional issue of customer confidence. If a server goes down occasionally customers start wondering what else might be wrong and considering whether they should trust their credit card details to the online ordering system. So it is obviously apparent that servers need a different degree of reliability - and it’s easy to justify spending the money. Desktop machines also need reliability, more so than most people expect. In a business when a desktop machine crashes it wastes employee time. If a crash wastes an hour (which is not unlikely given that previously saved work may need to be re-checked) then it can easily cost the business $100 (the value of the other work that the employee might have done). Two such crashes per week could cost the business as much as $8000. The price difference between a typical desktop machine and a low-end workstation (or deskside server) is considerably less than that (when I investigated the prices almost a year ago desktop machines with server features ranged in price from $800 to $2400 [1]). Some machines in a home environment need significant reliability. For example when students are completing high-school their assignments have a lot of time invested in them. Losing an assignment due to a computer problem shortly before it’s due in could impact their ability to get a place in the university course that they most desire! Then there is also data which is irreplaceable, one example I heard of was of a woman who’s computer had a factory pre-load of Windows, during a storm the machine rebooted and reinstalled itself to the factory defaults - wiping several years of baby photos… In both cases better backups would mostly solve the problem. For business use the common scenario is to have file servers storing all data and have very little data stored on the PC (ideally have no data on the PC). In this case a disk error would not lose any data (unless the swap space was corrupted and something important was paged out when the disk failed). For home use the backup requirements are quite small. If a student is working on an important assignment then they can back it up to removable media whenever they reach a milestone. Probably the best protection against disk errors destroying assignments would be a bulk purchase of USB flash storage sticks. Disk errors are usually easy to detect. Most errors are in the form of data which can not be read back, when that happens the OS will give an error message to the user explaining what happened. Then if you have good backups you revert to them and hope that you didn’t lose too much work in the mean-time (you also hope that your backups are actually readable - but that’s another issue). The less common errors are lost-writes - where the OS writes data to disk but the disk doesn’t store it. This is a little more difficult to discover as the drive will return bad data (maybe an old version of the file data or maybe data from a different file) and claim it to be good. The general idea nowadays is that a filesystem should check the consistency of the data it returns. Two new filesystems, ZFS from Sun [2] and BTRFS from Oracle [3] implement checksums of data stored on disk. ZFS is apparently production ready while BTRFS is apparently not nearly ready. I expect that from now on whenever anyone designs a filesystem for anything but the smallest machines (EG PDAs and phones) they will include data integrity mechanisms in the design. I believe that one such features become commonly used the need for RAID on low-end systems will dramatically decrease. A combination of good backups and knowing when your live data is corrupted will often be a good substitute for preserving the integrity of the live data. Not that RAID will necessarily protect your data - with most RAID configurations if a hard disk returns bad data and claims it to be good (the case of lost writes) then the system will not read data from other disks for checksum validation and the bad data will be accepted. It’s easy to compute checksums of important files and verify them later. One simple way of doing so is to compress the files, every file compression program that I’ve seen has some degree of error detection. Now the real problem with RAM which lacks ECC is that it can lose data without the user knowing. There is no possibility of software checks because any software which checks for data integrity could itself be mislead by memory errors. I once had a machine which experienced filesystem corruption on occasion, eventually I discovered that it had a memory error (memtest86+ reported a problem). I will never know whether some data was corrupted on disk because of this. Sifting through a large amount of stored data for some files which may have been corrupted due to memory errors is almost impossible. Especially when there was a period of weeks of unreliable operation of the machine in question. Checking the integrity of file data by using the verify option of a file compression utility, fsck on a filesystem that stores checksums on data, or any of the other methods is not difficult. I have a lot of important data on machines that don’t have ECC. One reason is that machines which have ECC cost more and have other trade-offs (more expensive parts, more noise, more electricity use, and the small supply makes it difficult to get good deals). Another is that there appear to be no laptops which support ECC (I use a laptop for most of my work). On the other hand RAID is very cheap and simple to implement, just buy a second hard disk and install software RAID - I think that all modern OSs support RAID as a standard installation option. So in spite of the fact that RAID does less good than a combination of ECC RAM and good backups (which are necessary even if you have RAID), it’s going to remain more popular in high-end desktop systems for a long time. The next development that seems interesting is the large portion of the PC market which is designed not to have the space for more than one hard disk. Such compact machines (known as Small Form Factor or SFF) could easily be designed to support ECC RAM. Hopefully the PC companies will add reliability features in one area while removing them in another.

• [0] http://en.wikipedia.org/wiki/Hamming_code


• [1] http://etbe.coker.com.au/2007/08/01/ecc-ram-in-a-cheap-machine/


• [2] http://en.wikipedia.org/wiki/ZFS


• [3] http://en.wikipedia.org/wiki/Btrfs

Share This