This is the story of a weirdly unfriendly/non-compliant IMAP server, and some nice interactions that arose from a debugging session around it. Over the holidays, i got to do some computer/network debugging for friends and family. One old friend (I'll call him Fred) had a series of problems i managed to help work through, but was ultimately basically stumped based on the weird behavior of an IMAP server. Here's the details (names of the innocent and guilty have been changed), just in case it helps other folks in at least diagnosing similar situations. the diagnosis The initial symptom was that Fred's computer was "very slow". Sadly, this was a Windows machine, so my list of tricks for diagnosing sluggishness is limited. I went through a series of questions, uninstalling things, etc, until we figured it would be better to just have him do his usual work while i watched, kibitzing on what seemed acceptable and what seemed slow. Quite soon, we hit a very specific failure: Fred's Thunderbird installation (version 2, FWIW) was sometimes hanging for a very long period of time during message retrieval. This was not exhaustion of the CPU, disk, RAM, or other local resource. It was pure network delay, and it was a frequent (if unpredictable) frustrating hiccup in his workflow. One thought i had was Thunderbird's per-server max_cached_connections setting, which can sometimes cause a TB instance to hang if a remote server thinks Thunderbird is being too aggressive. After sorting out why Thunderbird was resetting the values after we'd set them to 0 (grr, thanks for the confusing UI, folks!), we set it to 1, but still had the same occasional, lengthy (about 2 minutes) hang when transfering messages between folders (including the trash folder!), or when reading new messages. Sending mail was quite fast, except for occasional (similarly lengthy) hangs writing the copy to the sent folder. So IMAP was the problem (not SMTP), and the 2-minute timeouts smelled like an issue with the networking layer to me. At this point, i busted out wireshark, the trusty packet sniffer, which fortunately works as well on Windows as it does on GNU/Linux. Since Fred was doing his IMAP traffic in the clear, i could actually see when and where in the IMAP session the hang was happening. (BTW, Fred's IMAP traffic is no longer in the clear: after all this happened, i switched him to IMAPS (IMAP wrapped in a TLS session), because although the IMAP server in question actually supports the STARTTLS directive, it fails to advertise it in response to the CAPABILITIES query, so Thunderbird refuses to try it. arrgh.) The basic sequence of Thunderbird's side of an initial IMAP conversation (using plain authentication, anyway) looks something like this:

1 capability
2 login "user" "pass"
3 lsub "" "*"
4 list "" "INBOX"
5 select "INBOX"
6 UID fetch 1:* (FLAGS)

What i found with this server was that if i issued commands 1 through 5, and then left the connection idle for over 5 minutes, then the next command (even if it was just a 6 NOOP or 6 LOGOUT) would cause the IMAP server to issue a TCP reset. No IMAP error message or anything, just a failure at the TCP level. But a nice, fast, responsive failure -- any IMAP client could recover nicely from that by just immediately opening a new connection. I don't mind busy servers killing inactive connections after a reasonable timeout. If it was just this, though, Thunderbird should have continued to be responsive. the deep weirdness But if i issued commands 1 through 6 in rapid succession (the only difference is that extra 6 UID fetch 1:* (FLAGS) command), and then let the connection idle for 5 minutes, then sent the next command: no response of any kind would come from the remote server (not even a TCP ACK or TCP RST). In this circumstance, my client OS's TCP stack would re-send the data repeatedly (staggered at appropriate intervals), until finally the client-side TCP timeout would trigger, and the OS would report the failure to the app, which could turn around and do a simple connection restart to finish up the desired operation. This was the underlying situation causing Fred's Thunderbird client to hang. In both cases above (with or without the 6th command), the magic window for the idle cutoff was a little more than 300 seconds (5 minutes) of idleness. If the client issued a NOOP at 4 minutes, 45 seconds from the last NOOP, it could keep a connection active indefinitely. Furthermore, i could replicate the exact same behavior when i used IMAPS -- the state of the IMAP session itself was somehow modifying the TCP session behavior characteristics, whether it was wrapped in a TLS tunnel or not. One interesting thing about this set of data is that it rules out most common problems in the network connectivity between the two machines. Since none of the hops between the two endpoints know anything about the IMAP state (especially under TLS), and some of the failures are reported properly (e.g. the TCP RST in the 5-command scenario), it's probably safe to say that the various routers, NAT devices, and such were not themselves responsible for the failures. So what's going on on that IMAP server? The service itself does not announce the flavor of IMAP server, though it does respond to a successful login with You are so in, and to a logout with IMAP server logging out, mate. A bit of digging on the 'net suggests that they are running a perdition IMAP proxy. (clearly written by an Aussie, mate!) But why does it not advertise its STARTTLS capability, even though it is capable? And why do some idle connections end up timing out without so much as an RST, when other idle connections give at least a clean break at the TCP level? Is there something about issuing the UID command that causes perdition to hand off the connection to some other service, which in turn doesn't do proper TCP error handling? I don't really know anything about the internals of perdition, so i'm just guessing here. the workaround I ultimately recommended to Fred to reduce the number of cached connections to 1, and to set Thunderbird's interval to check for new mail down to 4 minutes. Hopefully, this will keep his one connection active enough that nothing will timeout, and will keep the interference to his workflow to a minimum. It's an unsatisfactory solution to me, because the behavior of the remote server still seems so non-standard. However, i don't have any sort of control over the remote server, so there's not too much i can do to provide a real fix (other than point the server admins (and perdition developers?) at this writeup). I don't even know the types of backend server that their perdition proxy is balancing between, so i'm pretty lost for better diagnostics even, let alone a real resolution. some notes I couldn't have figured out the exact details listed above just using Thunderbird on Windows. Fortunately, i had a machine with a decent OS available, and was able to cobble together a fake IMAP client from a couple files (imapstart contained the lines above, and imapfinish contained 8 LOGOUT), bash, and socat. Here's the bash snippet i used as a fake IMAP client:

spoolout()   while read foo; do sleep 1 && printf "%s\r\n" "$foo" ; done  
( sleep 2 && spoolout < imapstart && sleep 4 && spoolout < imapfinish && sleep 500 )   socat STDIO TCP4:imap.fubar.example.net:143

To do the test under IMAPS, i just replaced TCP4:imap.fubar.example.net:143 with OPENSSL:imap.fubar.example.net:993. And of course, i had wireshark handy on the GNU/Linux machine as well, so i could analyze the generated packets over there. One thing to note about user empowerment: Fred isn't a tech geek, but he can be curious about the technology he relies on if the situation is right. He was with me through the whole process, didn't get antsy, and never tried to get me to "just fix it" while he did something else. I like that, and wish i got to have that kind of interaction more (though i certainly don't begrudge people the time if they do need to get other things done). I was nervous about breaking out wireshark and scaring him off with it, but it turned out it actually was a good conversation starter about what was actually happening on the network, and how IP and TCP traffic worked. Giving a crash course like that in a quarter of an hour, i can't expect him to retain any concrete specifics, of course. But i think the process was useful in de-mystifying how computers talk to each other somewhat. It's not magic, there are just a lot of finicky pieces that need to fit together a certain way. And Wireshark turned out to be a really nice window into that process, especially when it displays packets during a real-time capture. I usually prefer to do packet captures with tcpdump and analyze them as a non-privileged user afterward for security reasons. But in this case, i felt the positives of user engagement (how often do you get to show someone how their machine actually works?) far outweighed the risks. As an added bonus, it also helped Fred really understand what i meant when i said that it was a bad idea to use IMAP in the clear. He could actually see his username and password in the network traffic! This might be worth keeping in mind as an idea for a demonstration for workshops or hacklabs for folks who are curious about networking -- do a live packet capture of the local network, project it, and just start asking questions about it. Wireshark contains such a wealth of obscure packet dissectors (and today's heterogenous public/open networks are so remarkably chatty and filled with weird stuff) that you're bound to run into things that most (or all!) people in the room don't know about, so it could be a good learning activity for groups of all skill levels. Tags: debugging, imap, perdition, wireshark

This was born from a need to see how big a ZIP file I could send my accountant in Australia, and scratching the itch to write some code. The fact that most SMTP servers talk Extended SMTP makes this relatively easy, and Python has some great modules for DNS and SMTP. One gripe I do have is how long it takes the Python modules to mature. It's taken until Python 2.6 for smtplib.SMTP() to gain a timeout parameter, for example. Anyway, I was able to write something nice and generic (it works for any domain) in around 100 lines, thanks in no small part to the DNS module, which makes getting a list of MX records stupidly easy.

$ ./maxmessagesize.py andrew.net.au
daedalus.andrew.net.au: -1
$ ./maxmessagesize.py pollock.id.au
ASPMX.L.GOOGLE.COM: 35,651,584
ALT1.ASPMX.L.GOOGLE.COM: 35,651,584
ALT2.ASPMX.L.GOOGLE.COM: 35,651,584
ASPMX2.GOOGLEMAIL.COM: 35,651,584
ASPMX3.GOOGLEMAIL.COM: 35,651,584
ASPMX4.GOOGLEMAIL.COM: 35,651,584
ASPMX5.GOOGLEMAIL.COM: 35,651,584
$ ./maxmessagesize.py debian.org
master.debian.org: 104,857,600
$ ./maxmessagesize.py cameronp.com
mail1.mysmarthost.com: 30,000,000
mail2.mysmarthost.com: 30,000,000
$ ./maxmessagesize.py ubuntu.com
mx.canonical.com: 62,914,560
$ ./maxmessagesize.py clug.org.au
mx.clug.org.au: 50,000,000
$ ./maxmessagesize.py linux.org.au
morton.linux.org.au: 52,428,800

It's good to see that in most cases of domains I tried, all of the MXes had the same maximum message size. Source code is here

I recently came across the Transitive Grace Period Public License (alternate link) by Zooko Wilcox-O Hearn. I fonud it interesting because it s very similar almost identical to something I had first starting floating about ten years ago. I called it the Temporary Proprietary License (TPL). I m sure this is a case of great minds think alike . One things that I like about my write up is that I gave some of the rationale behind why this approach is a fruitful one:

A while ago, I was talking to Jim Gettys at the IETF meeting in Orlando, and the subject of software licensing issues came up, and he had a very interesting perspective to share about the X Consortium License, and what he viewed as bugs in the GPL. His basic observation was this: Many companies made various improvements to the X code, which they would keep as proprietary and give them a temporary edge in the marketplace. However, since the X code base was continually evolving, over time it became less attractive to maintain these code forks, since it would mean that they would have to be continually merging their enhancements into the evolving code base. Also, in general, the advantage in having the proprietary new feature or speed enhancement typically degraded over time. After all, most companies are quite happy if it takes 18-24 months for their competitor to match a feature in their release. So sometime later, the companies would very often donate their previously proprietary enhancement to the X consortium, which would then fold it into the public release of X. Jim Gettys complaint about the GPL was that by removing this ability for companies to recoup the investment needed to make major developmental improvements to Open Source code bases, companies might not have the incentive do this type of infrastructural improvements to GPL ed projects. Upon reflection, I think this is a very valid model. When Open Vision distributed the Kerberos Administration daemon to MIT, they wanted an 18 month sunset clause in the license which would prevent commercial competitors from turning around and bidding their work against them. My contract with Powerquest for doing the ext2 partition resizer had a similar clause which kept the resizing code proprietary until a suitable timeout period had occurred, at which point it would be released under the GPL. (Note that by releasing under the GPL, it prevents any of Partition Magic s commercial competitors from including it in their proprietary products!)

For more, read the full proposal. No related posts.

Marc Fleury, JBoss founder joins the ranks of Tonnerre, me, and other people requesting that MySQL (and MariaDB!) please finally die. Everyone, don't even fork it. Use a real database instead. Or, at least, SQLite. Really. We're going to FOSDEM 2010 (of course I've been at every FOSDEM that was not just an OSDEM, Benny and gecko2 are regular attendees as well, as are other projects of mine such as FreeWRT and Debian GNU/kFreeBSD, by proxy). There will be a recent MirBSD snapshot I've yet got to build, with the new floppy format ustarfs (idea, but no single line of their stinking ridiculously huge code, stolen from NetBSD ) and other improvements (albeit less than I wanted to get done by then). The days before, I'll attend the first FusionForge meeting to break up the French Cabal, with my work hat on. That is also my first time in France (outside of the Elsa ). People, make a good impression on me to overcome the classic prejudices ;-) This weekend I'm going to meet my Debian Application Manager zack, have some good beer (ugh... first this, then Paris, then good belgian beer...) and fix some bugs, all while learning even more. Sounds like fun, but I almost feel overwhelmed, in contrast to the years of much less travelling from my past. I've also started sort-of mentoring Simon, one of our apprentices at work, into the Debian processes. (On an unrelated side note, formorer recently said bpo will become bp.d.o soon. Great!) Please don't laugh at this excuse for a webpage, as I've yet to fill it in, but my CLI for the KDE Wallet is hereby deemed ready for public consumption, with a bug-fix release 2.01 (bugs actually found during preparation of a port to Debian sid and KDE 4, which is much much worse than KDE 3, plus it looks so absolutely disgusting I'm not even sure Windows Mistda is worse). I hope the package will end up in NEW soon (and once progressed to testing I may be able to make the KDE 3 variant official via lenny backports; my WTF *.deb repo will hold them until then.

There are more webpages I need to fill in... mksh's TaC, arc4random (which needs some major redesign as well) and BSD::arc4random, the RANDEX protocol (entropy exchange over IRC) and its plugins and patches, ... Not just Mac OSX (and, I hope, iPhoneOS) will soon come with mksh(1), but also Android (I prepared patches to make it /bin/sh, which works quite well although I need to find out how to make a hardlink so that #!/bin/mksh scripts will run) and Maemo, for which I wrote an mksh package in a garage project, which also needs some love w.r.t. testing on actual devices, menu integration, etc. (Please contact me if you can help with either of the three.) We also have lewellyn:#ksh making a package for the new OpenSolaris system (thanks again). People persuading Apple to put it on the jesusPhone are also welcome. (This does not mean I endorse any of these right now, I'd probably get the most of a WinCE PDA with built-in GPSr, WLAN and maybe GSM/GPRS.) English and French native speakers, please review, and Dutch native speakers may contribute a translation of, our flyers. (Source code for these is not available, sorry. Benny makes them in Quark on System 7 in Basilisk II, used to be Classic until Apple yanked it. But still, they use only free fonts, free imagery or such the MirOS Project is allowed to use, and beat every single other FOSS project flyer I've ever seen by far!) There's probably more I could write, I bet I forgot half of it anyway, but I'll leave it at that for now. Get yourself a nice cup of hot chocolate, pour an Espresso into it, and enjoy the mix with a piece of cake (I'd say strawberry or mousse-pear but all they had was cassis-cr me) and pity me for not knowing any French next month.

I ve been asked why I dislike surveymonkey surveys, I don t want to reply fully in that forum (because it s off-topic there) and I want to publish this for your feedback now. surveymonkey really frustrates me. In short, it s often confusing, I think it doesn t follow UK accessiblity laws and I m unsure if it complies with our privacy laws. It seems like another triumph of marketing over good sense. It s a free market failure, possibly based on lack of information on the drawbacks of this option. If a survey is on Surveymonkey, it s easiest for able-bodied users of full-fat browsers with low security and privacy settings to answer. Using NoScript or a privacy tool? You ll get an error message (which contradicts the VPAT_SurveyMonkey.pdf description). Unable to see graphics clearly? The forms will be hard to read because they ve replaced the browser form widgets (such as checkboxes and radio buttons) with javascript-driven images for no good reason. Those sort of faults will probably bias results a bit how much will depend on the subject. It s a particular case of a problem I first mentioned in the last-but-one bullet point on my debian surveys page back in 2006. Basically, I think you should keep open surveys as open as possible. Statistics literature is littered with bizarre spurious results from when the survey method accidentally favours some particular subset of the population. Finally, surveymonkey surveys can be left with answer requirements which make no sense, such as ticking Other requiring a text input, but the text box not accepting input. Is that the fault of the survey owner not being able to work the admin interface, or the interface having some subtle flaws? I m not completely sure, but I ve had that experience a few times. So, can I persuade people to use limeservice, doodle or even Google Docs rather than Surveymonkey, please? I think Limeservice is better because it s based in Germany (so EU data protection and privacy laws apply) and is cooperatively-developed FOSS software. Doodle and Google aren t, but are quicker and easier to use for straw polls. If you want something to install on your own site and the downloadable limesurvey software seems a bit heavy, then askpeople seems another good option. Our co-op can host either of those if you re willing to pay. (If someone would be willing to sponsor it, we d host some free surveys too let me know.) Lots of people use surveymonkey, but popularity doesn t always mean it s the best tool. Please consider the alternatives and pick the best tool for the job.

"We are currently working on getting the code shipped out, we have your shipping address on file and will ship the code out as soon as it is ready, this will be by the end of this month."

2/10, must try harder. Even ignoring the fact that it's taking the best part of two months to get from shipping binaries to shipping source, it'll be interesting to see which version of the firmware the source release corresponds to given that the 1.1.0 update included a new kernel.

Speaking of firmware updates - as shipped, the nook was a disaster. The firmware update improved things greatly, but I'm still stuck with cases where books render with the last couple of words in a page left out, and the order of books in the local content library is now based on last modification time rather than, say, author name. The continuing lack of any way to implement catagorisation or provide worthwhile metadata means that it's frustrating having to remember which order the books in a series go in. The fact that there's even a distinction between content you've obtained from B&N and content you've obtained from elsewhere is mindboggling. You need to explicitly switch between the two, which means that collections are broken up if you've had the audacity to buy some books from B&N and some books from elsewhere.

Having now spent a while travelling with the nook, it's clear that it's still not an especially effective example of its type. If my Sony hadn't suffered an unfortunate accident, I'd be using it instead. The nook's hardware is significantly better and as a platform it's got much more potential, but it just doesn't do a terribly good job at its core role - reading books. It'll get better with time, but if you're in the market for something that actually works today then go elsewhere.

There's other little things that are upsetting at a conceptual level. The 1.0.0 firmware will flash unsigned firmware, which is useful[1] but also an obvious wtf. The filesystem is full of scripts that are world writable and executed as root. There's a libGLES, but the coverflow implementation appears to be handled entirely in software. There's a general feeling that the platform's been put together by people who either don't know what they're doing, don't care about doing it properly or were working under sufficient time constraints that getting it right wasn't an option. I'm really not impressed.

[1] Especially given that the 1.1.0 firmware will happily flash the 1.0.0 firmware...

the ultimate text diffing helper I use Git not only for code, but also for text, my most common use case being co-authorship of scientific papers typeset in LaTeX. While reviewing other author changes, the usual line-by-line diff is often annoying since:

1. Paragraph re-formatting gets in the way (hello guys accustomed to compulsory Vim's gq or Emacs' M-q): it does not effect the final rendering, but it still hinders peer-review of changes.
2. Even in the simplest case of a single word change on a single line, it will take some time for your eyes to spot where, along the line, the actual change is.

Stuff like wdiff does help, but I've never looked at how to integrate it with Git, and I still find it a pain in the eyes due to the fancy ASCII arts used to denote additions and deletions (pain that I usually experience when looking at the output of debdiff). There are nice diffing GUIs or editor-integrated solutions out there, but what I've always dreamed of is a plain old cmdline geekism. Enter Git's --color-words, that can be passed to both show and diff commands. My favorite related Git aliases are as follows:

    [alias]
        wdiff = diff --color-words
        wshow = show --color-words

Here is a sample fancy output, yet very intuitive and console-based, of git show --color-words (text in red denotes deletions, text in green additions): Maybe not new to you, but it has been life-changing for me.

One thing everyone forgot to talk about the WebKitGTK+ hackfest was that master Dan Winship added basic Content-Encoding support to libsoup, and patched WebKitGTK+ to use it. If you are using a recent enough version of those you will finally be able to visit web sites that send gzipped content despite the browser saying it could not handle it, like the Internet Archive. This was one of those cases in which the web shows all of its potential to behave weirdly. The HTTP/1.1 RFC says that if an Accept-Encoding header is not present, the server MAY assume the client accepts any encoding, so we were having many sites send us gzip content even though we did not support it. We then started sending a header saying we support identity, and nothing else! . It turns out the web sucks, so many servers were not happy with a full header, and started giving us angry looks (slashdot, for instance, would not render correctly because it started sending encoded CSS files!). We then simplified the header we were sending, which made those servers happy again. Some sites, though, completely ignored our saying we didn t support anything except identity, and sent us gzipped content anyway. Most of these were misbehaving caches (this was the case for Wikipedia), so would work after you asked for a forced reload, which would ignore the cache, but some servers, such as the Internet Archive s didn t really want to talk about encodings - they only wanted to send gzip-encoded content. So, in the end, our only way out was implementing the damn encoding support, which finally happened during the hackfest. Take that, web!

So. I m trying to get familiar with libvirt and friends. To this end, I ve set up a Lucid virtual machine booting from PXE into an initrd environment which does a pivot_root to an AoE block device. The #virt channel on irc.oftc.net told me that in order to have libvirt provide PXE capability, I would have to install a recent version of libvirt. I built version 0.7.5-3 from sid on my karmic laptop and it seems to be working okay. I decided to set up the pxe root directoy in /var/lib/tftproot just because that s what the example code had in it. I had to manually configure a virtual network. Here is the XML config file:

$ sudo virsh net-dumpxml netboot
<network>
  <name>netboot</name>
  <uuid>81ff0d90-c91e-6742-64da-4a736edb9a9b</uuid>
  <forward mode='nat'/>
  <bridge name='virbr1' stp='off' delay='1' />
  <domain name='example.com'/>
  <ip address='192.168.123.1' netmask='255.255.255.0'>
    <tftp root='/var/lib/tftproot' />
    <dhcp>
      <range start='192.168.123.2' end='192.168.123.254' />
      <bootp file='pxelinux.0' />
    </dhcp>
  </ip>
</network>

This, of course, depends on the pxelinux.0 file. Luckily, this is packaged up in syslinux and can be installed with a simple
$ sudo apt-get install syslinux
$ sudo mkdir /var/lib/tftproot
$ sudo cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftproot
I had to create a pxelinux config file for the virtual machine (indexed by mac address). Note that I put a console=ttyS0,115200 argument on the kernel command line so that I can attach to the serial port from the host system for copy/paste debugging. Also of importance is the root=/dev/etherd/e0.1p1 argument, specifying which block device we ll be doing the pivot_root to eventually.
$ mkdir /var/lib/tftproot/pxelinux.cfg/
$ cat /var/lib/tftproot/pxelinux.cfg/01-52-54-00-44-34-67
DEFAULT linux
LABEL linux
SAY Now booting the kernel from PXELINUX...
KERNEL vmlinuz-lucid0
APPEND ro root=/dev/etherd/e0.1p1 console=ttyS0,115200 initrd=initrd.img-lucid0
I decided to use the karmic kernel for lucid initially. I ll eventually switch over to the lucid kernel ;)
$ sudo cp /boot/vmlinuz-2.6.31-17-generic /var/lib/tftproot/vmlinuz-lucid0
I copied /etc/initramfs-tools to ~/tmp/lucid so that I didn t mess up the system initrd scripts:
$ mkdir -p ~/tmp/lucid && cp -r /etc/initramfs-tools ~/tmp/lucid/
Since mkinitramfs doesn t currently have a system for AoE root, I had to do a bit of fiddling. I copied the NFS root boot script and made a couple of modifications.

$ diff -u /usr/share/initramfs-tools/scripts/nfs ~/tmp/lucid/initramfs-tools/scripts/aoe
--- /usr/share/initramfs-tools/scripts/nfs	2008-06-23 23:10:21.000000000 -0700
+++ /home/cjac/tmp/lucid/initramfs-tools/scripts/aoe	2010-01-15 14:56:28.098298027 -0800
@@ -5,59 +5,25 @@
 retry_nr=0
 # parse nfs bootargs and mount nfs
-do_nfsmount()
+do_aoemount()
  
-
 	configure_networking
-	# get nfs root from dhcp
-	if [ "x$ NFSROOT " = "xauto" ]; then
-		# check if server ip is part of dhcp root-path
-		if [ "$ ROOTPATH#*: " = "$ ROOTPATH " ]; then
-			NFSROOT=$ ROOTSERVER :$ ROOTPATH 
-		else
-			NFSROOT=$ ROOTPATH 
-		fi
-
-	# nfsroot=[<server-ip>:]<root-dir>[,<nfs-options>]
-	elif [ -n "$ NFSROOT " ]; then
-		# nfs options are an optional arg
-		if [ "$ NFSROOT#*, " != "$ NFSROOT " ]; then
-			NFSOPTS="-o $ NFSROOT#*, "
-		fi
-		NFSROOT=$ NFSROOT%%,* 
-		if [ "$ NFSROOT#*: " = "$NFSROOT" ]; then
-			NFSROOT=$ ROOTSERVER :$ NFSROOT 
-		fi
-	fi
+        ip link set up dev eth0
-	if [ -z "$ NFSOPTS " ]; then
-		NFSOPTS="-o retrans=10"
-	fi
+        ls /dev/etherd/
-	[ "$quiet" != "y" ] && log_begin_msg "Running /scripts/nfs-premount"
-	run_scripts /scripts/nfs-premount
-	[ "$quiet" != "y" ] && log_end_msg
+        echo > /dev/etherd/discover
-	if [ $ readonly  = y ]; then
-		roflag="-o ro"
-	else
-		roflag="-o rw"
-	fi
+        ls /dev/etherd/
-	nfsmount -o nolock $ roflag  $ NFSOPTS  $ NFSROOT  $ rootmnt 
+        mount $ ROOT  $ rootmnt 
  
-# NFS root mounting
+# AoE root mounting
 mountroot()
  
-	[ "$quiet" != "y" ] && log_begin_msg "Running /scripts/nfs-top"
-	run_scripts /scripts/nfs-top
-	[ "$quiet" != "y" ] && log_end_msg
-
-	modprobe nfs
-	# For DHCP
-	modprobe af_packet
+	modprobe aoe
 	# Default delay is around 180s
 	# FIXME: add usplash_write info
@@ -67,17 +33,13 @@
 		delay=$ ROOTDELAY 
 	fi
-	# loop until nfsmount succeds
+	# loop until aoemount succeds
 	while [ $ retry_nr  -lt $ delay  ] && [ ! -e $ rootmnt $ init  ]; do
 		[ $ retry_nr  -gt 0 ] && \
-		[ "$quiet" != "y" ] && log_begin_msg "Retrying nfs mount"
-		do_nfsmount
+		[ "$quiet" != "y" ] && log_begin_msg "Retrying AoE mount"
+		do_aoemount
 		retry_nr=$(( $ retry_nr  + 1 ))
 		[ ! -e $ rootmnt $ init  ] && /bin/sleep 1
 		[ $ retry_nr  -gt 0 ] && [ "$quiet" != "y" ] && log_end_msg
 	done
-
-	[ "$quiet" != "y" ] && log_begin_msg "Running /scripts/nfs-bottom"
-	run_scripts /scripts/nfs-bottom
-	[ "$quiet" != "y" ] && log_end_msg
  

(below is the full file in case udiff is less convenient)

$ cat ~/tmp/lucid/initramfs-tools/scripts/aoe
# NFS filesystem mounting			-*- shell-script -*-
# FIXME This needs error checking
retry_nr=0
# parse nfs bootargs and mount nfs
do_aoemount()
 
	configure_networking
        ip link set up dev eth0
        ls /dev/etherd/
        echo > /dev/etherd/discover
        ls /dev/etherd/
        mount $ ROOT  $ rootmnt 
 
# AoE root mounting
mountroot()
 
	modprobe aoe
	# Default delay is around 180s
	# FIXME: add usplash_write info
	if [ -z "$ ROOTDELAY " ]; then
		delay=180
	else
		delay=$ ROOTDELAY 
	fi
	# loop until aoemount succeds
	while [ $ retry_nr  -lt $ delay  ] && [ ! -e $ rootmnt $ init  ]; do
		[ $ retry_nr  -gt 0 ] && \
		[ "$quiet" != "y" ] && log_begin_msg "Retrying AoE mount"
		do_aoemount
		retry_nr=$(( $ retry_nr  + 1 ))
		[ ! -e $ rootmnt $ init  ] && /bin/sleep 1
		[ $ retry_nr  -gt 0 ] && [ "$quiet" != "y" ] && log_end_msg
	done
 

There was also a small modification to the initramfs.conf file:

$ diff -u /etc/initramfs-tools/initramfs.conf ~/tmp/lucid/initramfs-tools/initramfs.conf
--- /etc/initramfs-tools/initramfs.conf	2008-07-08 18:37:42.000000000 -0700
+++ /home/cjac/tmp/lucid/initramfs-tools/initramfs.conf	2010-01-15 14:33:38.088295207 -0800
@@ -47,14 +47,16 @@
 #
 #
-# BOOT: [ local   nfs ]
+# BOOT: [ local   nfs   aoe]
 #
 # local - Boot off of local media (harddrive, USB stick).
 #
 # nfs - Boot using an NFS drive as the root of the drive.
 #
+# aoe - Boot using an AoE drive as the root of the drive.
+#
-BOOT=local
+BOOT=aoe
 #
 # DEVICE: ...

I also needed to add aoe to the list of modules included in the initramfs:
$ echo aoe >> ~/tmp/lucid/initramfs-tools/modules
In order to generate the initrd.img file from this new config, I ran the following:
$ sudo mkinitramfs -d ~/tmp/lucid/initramfs-tools/ -o /var/lib/tftproot/initrd.img-lucid0
I created a lucid VM by installing from the desktop install disk. You can grab the ISO here: http://cdimage.ubuntu.com/daily-live/current/ I ll leave the creation of the virtual machine and installation as an exercise for the reader. I put the filesystem on an lvm volume group called vg0 in a logical volume called lucid0 (ie, /dev/vg0/lucid0). At this point, I created a new virtual machine called lucid0. Here is the xml for the domain:

$ sudo virsh dumpxml lucid0
<domain type='kvm' id='1'>
  <name>lucid0</name>
  <uuid>96fbad21-4f25-5700-ddd8-1a565c7170ee</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-0.11'>hvm</type>
    <boot dev='network'/>
  </os>
  <features>
    <pae/>
  </features>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <interface type='network'>
      <mac address='52:54:00:44:34:67'/>
      <source network='netboot'/>
      <target dev='vnet0'/>
    </interface>
    <serial type='pty'>
   <source path='/dev/pts/4'/>
      <target port='0'/>
    </serial>
  <console type='pty' tty='/dev/pts/4'>
   <source path='/dev/pts/4'/>
      <target port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5901' autoport='yes' listen='127.0.0.1' keymap='en-us'/>
    <sound model='es1370'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
    </video>
  </devices>
</domain>

Now we re ready to start the AoE target and launch the virtual machine. If you don t have vblade installed, do so now:
$ sudo apt-get install vblade
Start the target up with the following command:
$ sudo vbladed 0 1 virbr1 /dev/vg0/lucid0
Now, if all goes well, you should be able to watch the virtual machine boot up and do its thing like so:
$ sudo virsh start lucid0 && sudo screen -S lucid0 sudo virsh ttyconsole lucid0 115200
If you get errors about /dev/etherd/e0.1p1 not existing (these might look like this):

Begin: Retrying AoE mount ...
err         discover    interfaces  revalidate  flush
err         discover    interfaces  revalidate  flush
mount: mounting /dev/etherd/e0.1p1 on /root failed: No such file or directory
Done.

then you might want to try restarting vbladed like this:
$ sudo kill -9 ps auwx grep vblade grep -v grep awk ' print $2 ' && sudo vbladed 0 1 virbr1 /dev/vg0/lucid0
So. Now you should have a lucid gdm in your virt-manager console. Any questions? #virt on irc.oftc.net

Two weeks ago I filed #563195, which reads:

the current maintainer welcomed me into the usplash maintainance team, because they are busy with other stuff. Unfortunatly, I'm too busy to give usplash the attention it needs as well.

So I still plan to spend some work on usplash, but I definitly also need help, thus I'm filing this bug.

My basic idea is to upload to experimental based on the upstream version ASAP and frequently, and then upload to unstable once it works ;-)

If you are interested and able to jump in, please reply to this bug.

That's all still the case. The area where help is most welcome atm is to actually make the latest upstream version work without upstart again, as far as I could see on a quick look, upstream has removed support for other initsystems... once we have a working package again, attacking the bugs in the BTS is next.

The culture of freedom lies in the details Here is an interesting blog post by Bradley Kuhn about Ubuntu, Debian, and (warning: my interpretation ahead) the culture of freedom. While reading it, I had kinda moment of truth, because just yesterday I was musing with Mehdi and Lucas on the fact that Debian is basically the only remaining distribution among the mainstream ones (if that means something) that is free from the ground up, including its infrastructure. We "just" seek hardware via donations and then we run, thanks to the amazing work of DSA, our own free infrastructure on top of it. Let's cherish this value! Thanks to Roberto Di Cosmo for the pointer to Bradley's post. Update: there's a thread on the ubuntu-devel mailing list about Bradley's post.

My mutt said this last night:

 894   + Jan 15 Archive Adminis (0,4K) ipcfg_0.1_amd64.changes ACCEPTED

This obviously means that if you wish to use it, you no longer need to go through git; you can just add experimental to sources.list and run 'apt-get install ipcfg'. A few notes, though:

• I missed a few build-dependencies, so it's only available on amd64 for now. Yes, I should use pbuilder more (I did for the 0.3 upload, which went in today).
• The documentation is somewhat lacking. I just found out that the documentation isn't actually installed in the binary package, so you have to download the source package (or check out the git repository). This will be fixed for the next release. The documentation also currently consists of a few .txt files rather than (a) proper man page(s). This will likely take some more time.
• ipcfg Conflicts:, Replaces: and Provides: ifupdown. This is because there is no proper virtual package name for it to conflict/replace/provide. I have filed a bug, #554194, against debian policy, to request it be made, but so far this is not yet finished (and indeed was somewhat dead during the period that I had no time to work on ipcfg). This may confuse some packages that really do depend on ifupdown.
• ifdown works, but not reliably so. This will be my next main point of focus.
• If you use the insserv automatic ordering system for initscripts, then you must either purge ifupdown or remove its initscript before installing ipcfg; otherwise, the availability of two initscripts that provide network configuration will confuse the hell out of insserv.

And in case you wonder why the hell I went from 0.1 to 0.3:

ipcfg (0.2) experimental; urgency=low
  * Rebuild without .git directory. D'oh.
 -- Wouter Verhelst <wouter>  Tue, 12 Jan 2010 17:43:09 +0100

srsly

RC squashing news In the last two weeks I didn't progress as usual with RC bug squashing, even if I've some squashes to share:

• bug #545235, bug #560502 - k3d - add missing #include for gcc 4.4 (patch from Ubuntu)
• bug #552791 - acorn-fdisk - fix debian/copyright (sponsored NMU for my NM Thorsten Glaser)
• bug #527371 - qemu-kvm - remove extra stuff (fix FTBFS when built twice in a row)

The reason for my under-activity are that for a week I'd relatively scarce network connectivity, which is a badly needed good for RCBW. Then, this week, I got swamped by various tasks at the university (including co-authoring a paper with Lucas, more on this (hopefully) in the future). My residual RCBW time has been devoted to write a page about RCBW where I've tried to summarize my initial motivations and the main points of the initiative, which I consider established by the first 18 weeks of activity. Starting from next week I'll be back with daily squashes. In the week-end ... drum roll ... I'll be attending the BSP in M nchengladbach, hosted by Credativ; I'll personally be kindly hosted by bzed which I thank already for that and for the BSP organization. See you there (!?)

It seems that I m not blogging as often as I was some months ago. The main reason is microblogging, which is somehow quicker, even though briefer. It is also less informative, I guess, because 140 characters is a very limited length. Probably because of that, a friend of mine asked me to write a proper blog entry about this, and I think that he s probably right. openFrameworks is an open source toolkit, released under the MIT license, written in C++ and designed to assist the creative process by providing a simple and intuitive framework for experimentation. According to its authors, it was developed for folks using computers for creative, artistic expression, and who would like low level access to the data inside of media in order manipulate, analyze or explore. I attended a short course in openFrameworks some months ago, and decided that if I wanted to experiment with it, I should somehow structure the building process of the library a bit, remove the dependency on FMOD, which is not DFSG-free, even though that might require remove the whole audio subsystem, make it buildable without Code::Blocks, and hopefully build some more-or-less proper packages. I ve also added some pkg-config files, so that it becomes easier to build projects based in openFrameworks (as easy as g++ source.cpp -o test $(pkg-config openframeworks openframeworks-addons cflags libs) ). Even though the packages are still not as good as they should, they re absolutely usable right now. I ve uploaded them, as well as their dependencies, to my Ubuntu PPA (although I m using those packages in my Debian), in case that someone is interested. I ve also uploaded a couple of tiny examples (the second one downloaded from here) of openFrameworks.

The cheapest ADSL or Cable net access in Australia seems to be about $30 per month. I ve been using 3G net access by the Three phone company for 18 months now and it s been working well [1]. I recently bought a new 3G modem because the old one broke, so it has cost me $250 in modems plus $15 per month for the connection which compares well to $100 (or more) for an ADSL or Cable installation plus $30 per month. My Three net access gives me 1G of data per month. I have just noticed that they have pre-paid net access that gives 12G of data that must be used within one year which costs $149 per year [2] that is $12.41 per month or 83% the price of the plan access plus it means that any bandwidth quota that isn t used in one month can be used the next month (so you can save up for upgrading to a newer distribution of Linux). Dodo has pre-paid mobile net access on the Optus network for $139 which gives 15G of data that must be used within one year [3]. So that s equivalent to $11.58 per month or $9.20 per gig. A member of my local LUG mentioned that Exetel has a 3G plan which is good value if you don t use much data transfer but which has per-megabyte charges for excess data transfer. I couldn t recommend it for my parents as I never know when they will do something that may transfer a lot of data, I could just imagine them saying loading web pages was really slow for a week and then I got a big bill . Ross Barkman s GPRS/UMTS page gives some critical information on using a 3G phone with a tether [4]. Using that information I discovered that I need to use AT+CGDCONT=1,"IP","3netaccess" in my chatscript to get ppp going with my old LG U890 mobile phone (with 3netaccess being the important word). I plan to give my old mobile phone to my parents and let them use prepaid 3G net access to reduce their net access bill by more than 1/3 while also giving them more data transfer quota for times when they need to transfer a lot (EG when my sister visits them). At this stage I m not sure whether I will get them to use Three or Dodo. One advantage of Three is that I ve used them a lot and know exactly how to get it all working, the other is that my old mobile phone is locked to Three they agreed to unlock it on demand after the contract is ended (which happened over a year ago) but it will be a hassle to get it done. Saving the hassle of getting an old phone unlocked may be worth the $10 per year cost. Also I have used my 3G modem at my parents house on a few occasions and know that the reception is quite good, while the reception for Dodo (Optus) 3G is unknown One extra benefit with doing this is that my parents will have some freedom to move their PC. If they decide that the computer room is too hot in summer and want to move their PC to below their air-conditioner they will be able to do so without needing a long Ethernet cable to connect their PC to the cable modem. For my personal 3G net access (which I require for fixing servers on occasion) I am stuck with Three. When I bought a new 3G modem I decided to save about $20 by getting a device that s locked to Three. 12G per year is more than enough for sshing to servers and checking email and if I had paid extra for the unlocked modem it would it would probably have died before the savings on net access made up for the higher purchase price. Update:
Crazy John has a good deal, $129 for 7.5G which expires in a year [5]. I won t use that for my parents though, the probability of them going over 7.5G is too high to make it worth the risk for a $10 saving.

• [1] http://etbe.coker.com.au/2008/07/28/new-net-connections/
• [2] http://www.three.com.au/cs/ContentServer?c=Page&pagename=Three%2FPage%2FBusinessVideoCallingTemplate&cid=1221530707570
• [3] https://secure.dodo.com.au/prepaid-wireless/PlanSelection.aspx
• [4] http://www.taniwha.org.uk/gprs.html
• [5] http://www.crazyjohns.com.au/mobile-broadband/

Recently, I've been working on a file transfer app for the N900. It's called monorail and I uploaded it to extras-devel last night, so you can try it out right now. I implemented an OssoABookContactView with support for filtering on Contact Capabilities (or a draft of it, at least) as old-style caps aren't great. This gives a nice list of address book contacts who have at least one account that should support receiving file transfers. It would be nice if these extra caps could go into the address book which is where they belong, but it won't. If someone would find this widget useful, I'll clean up the code and maybe throw it in its own library.
Choose one of the accounts which support file transfer. I also started to implement a sharing plugin, but it turns out it's poorly documented and appears to be impossible to do anything useful for my use-case, like popping up a dialog when the user taps "Share". The example isn't very exemplary, but does contain some amusing code. This sharing plugin is only half-written and impossible to debug. If anyone has any pointers on writing these plugins, I would appreciate them. It's a shame that although Hildon widgets can look really good on the device, it can often take some fiddling to make them look consistent with the rest of the UI. Additionally, oddities in some widgets can leave you baffled, or amused. The most annoying thing I have come across regarding this app is just how little I can get it integrated. The file manager is all hard-coded so there's no way to add a Send to Contact menu item, and no buttons can be added to the libsharing dialog (which appears when the user shares a photo) which is annoying as it looks like it's missing this feature. Oh, and Telepathy is pretty sweet.

I m developing a huge backlog of things to write about, going back as far as a couple of posts inspired by the Kerberos conference in October. However, those require more effort putting together the post, so I m going to focus on something more recent. Part of running a business even a small business like Painless Security is dealing with the administrative and bookkeeping. The normal solution seems to be Intuit s Quick Books. When I set up the company, I looked into that. However, Intuit s accessibility story starts with give up, and goes down hill from there. Apparently, screen reader vendors have offered to work with Intuit to help them, but have been turned down, or at least that s the strong implication I get from reading blogs from these accessibility vendors. So, I d definitely rather not give Intuit my money. I m also looking for a bit more than the minimum in accounting software. Keeping the books in order enough to pay taxes would be easy. However I want to be able to understand where I m making money; I want to be able to figure out if the fixed price contracts I enter into end up being good ideas. I want to understand how expensive community projects Painless Security gets involved in like Debian and IETF work are both in terms of direct costs and opportunity costs. I want to understand what sorts of work ends up being the most profitable. All of these are fairly typical management questions; solutions to varying degrees are understood. however it means I m actually going to use an accounting product more than just to track my invoices and prepare taxes. I decided to see what the open source world was like in this space. I started with Ledger SMB. Ledger SMB s main claim to fame is that it tries to be better than SQL Ledger. Being better than SQl Ledger is definitely a good thing. It worked well enough to generate invoices and income statements. It nominally had facilities to track the sort of per-project information I m looking for, but the facilities are not near good enough. Also, there were some issues things like the fact that total debits didn t particularly need or tend to sum to total credits got old after a very short while. Also, facilities for correcting mistakes were unfortunate. You could either operate in a mode where you could delete a transaction, or a mode where you reverse transactions. Deleting transactions in practice meant deleting most of a transaction; balances became out of sync and half of the transaction tended to stick around using some interfaces but not other interfaces. Database cleanup was almost always required. In principle, reversals are better. However, there is no facility to indicate that you re not expecting payment on a reversed invoice. The receivables/payables account balance out because of the reversing transaction, but both the reversing and reversed transaction end up becoming over-due. Hey, can you please send me some anti-money to clean this up? The code was dreadful; the goal was to have better abstraction than SQL Ledger. Perhaps that was achieved, but man that leaves a lot to desire. Now, I m playing with Open ERP. It wants to grow up some day to be a competitor to SAP R3. In a way that sounds good, although it does mean there s a high complexity cost. There s a lot of functionality. There s reasonably good separation between view and model (and possibly even controller). The code is often clean, although the random blocks of commented out code with no explanation cause me to cringe. Many aspects of the system seem incredibly well designed; you can approach the system through a web client, graphical client, two different RPC mechanisms, XL and Open Office plugins. However, there s this mix of missing stuff and completely broken that causes me to wonder whether I m going to be sad. First, there appears to be basically no support for the US. All addresses are European format and are hard coded. Each individual report needs to be changed. Recently, I found cases where as best I can tell debits and credits are just reversed. I ve found a case where backing up the database succeeds but generates a zero length file. I almost lost data through that last one. There s a complex set of mechanisms to deal with units-of-measure for products for example, some jobs billed in hours, some are billed in days. However other parts of the code just add quantities. Playing around with this I am reminded again that I really enjoy thinking about these sorts of problems. An interesting ERP project could be fun to work on. for example, I bet handling ERP needs for some cloud-centered company would be a lot of fun. Also, as part of looking at Open ERP, I ve more or less developed the necessary scripts to migrate a simple service company from Ledger SMB to Open ERP. There are limitations of course, but if this would be useful to you, drop me a line.

There s a lot of responsibility on the shoulders of system administrators. Unless you re one, it s probably hard to grasp the full weight of it. This week has been full of reminders for me. This afternoon, shortly after lunch, I got word that people were having trouble with phones. A few minutes of testing showed that calls within our city were working fine, but it was completely impossible to place or receive long-distance calls. A little while later, a local newspaper s website indicated that numerous cities over a multi-county region, all served by the CenturyLink local telephone company, were out of service in this manner. I figure that represents hundreds of square miles, or a patch of rural Kansas roughly the size of Los Angeles. Adding to the problem was the fact that emergency 911 services are accessed via these long-distance lines that were down. For roughly three hours, 911 was completely down across this entire area. What s more, many cellphone towers and Internet access options were also taken down, since they feed from these same lines. This adds up to a situation that could very easily cost lives due to delayed response of emergency medical, fire, or police services. In the end, the problem was traced to a bad controller card on a Titan 5500 owned by AT&T. Now, here s the system administration angle: If you worked for the phone company and had to troubleshoot a problem that you knew had taken down emergency services for thousands of people, what kind of pressure would you be feeling? Would you be able to keep your cool? I m glad that I don t have that kind of job. I also wouldn t like to be the engineer (or, more likely, accountant) that decided that they didn t need any more redundancy to provide good service to the area. Especially considering this is the second time in the last year or two this has happened. But, ironically, yesterday I signed a purchase order for a new Asterisk PBX (corporate phone) server. When selecting a machine for that task, I am always completely conscious of the responsibility on my shoulders: several hundred employees rely on the machine that is ultimately my responsibility to select. Our own access to 911 would be cut if the machine were to go down. I never forget that the correct operation of the systems that our team sets up and deploys could help save someone s life, and that a malfunction could cost the company dearly in terms of revenue, productivity, image or worse. Nearly four years ago, we switched from an analog PBX, with outsourced support, to a digital VOIP system running Asterisk. Note that we use VOIP in-house, but do not use it externally. Anyhow, I can not say that the Asterisk PBX has been 100% perfect; I doubt that this could be honestly said of any PBX of any complexity. I can say, though, that it saved us well over $100,000 AND has proven far more reliable than the system it replaced. Outages are exceptionally rare and brief now. Plus we have internal expertise to fix it, rather than having to wait for a technician to be dispatched from a city 2 hours away to fix anything. I know I don t have the resources to build a perfect PBX that will never go down (if such a thing is even possible), but I take my responsibility regarding a reliable PBX extremely seriously. We used to have a frequent problem: someone would call 911, then hang up. We suspected this was often on accident maybe people hit 9 for an outside line, then misdialed their number. In any case, 911 dispatch would then call our main office, saying they got a hangup. A person or team would then go across our entire campus making sure nobody was in distress that nobody had managed to dial, then passed out, for instance. With Asterisk, I was able to help this situation. Whenever somebody calls 911 now, two emails are generated: the first contains details about the call, such as the extension number that called and the duration of the call. This goes to all people that are likely to receive a callback from 911. It may not always pinpoint the source (as with somebody using a wireless phone), but almost every time will give us a very good idea where the call came from. The second email is a recording of the call, and serves as an additional clue, but goes to fewer people. I am aware that email isn t a perfect medium, but: it let us make a dramatic (albeit imperfect) solution to a problem that very few institutions our size are able to address nearly so well. There s a lot of weight on our shoulders: keeping the accounting system up, the Internet links up, the web store or the sales phone lines, the shipping systems or the document archives up. These things going down can spell deep trouble in many ways. And sometimes the systems we maintain might save a life. Such as this morning, when someone was feeling symptoms of a heart attack, used our phones to call a colleague for help, that person called 911, and an ambulance was dispatched. I know the system worked exactly as it should, because I had two familiar emails from Asterisk in my mailbox this morning when I got to work.