One of the exciting things about having a cloud computing service is how to talk to the rest of the world. It’s all very well to have a varying number of machines in various locations, but you need constant DNS names at least (and sometimes constant IP addresses) to do most useful things. I have previously described how to start an EC2 instance and login to it - which includes discovering it’s IP address [1]. It would not be difficult (in theory at least) to use nsupdate to change DNS records after an instance is started or terminated. One problem is that there is no way of knowing when an instance is undesirably terminated (IE killed by hardware failure) apart from polling ec2-describe-instances so it seems impossible to remove a DNS name before some other EC2 customer gets a dynamic IP address. So it seems that in most cases you will want a constant IP address (which Amazon calls an Elastic IP address) if you care about this possibility. For the case of an orderly shutdown you could have a script remove the DNS record, wait for the timeout period specified by the DNS server (so that all correctly operating DNS caches have purged the record) and then terminate the instance. One thing that interests me is the possibility of running front-end mail servers on EC2. Mail servers that receive mail from the net can take significant amounts of CPU time and RAM for spam and virus filters. Instead of having the expense of running enough MX servers to sustain the highest possible load even while one of the servers has experienced a hardware failure there is a possibility of running an extra EC2 instance at peak times with the possibility of running a large instance for a peak time when one of the dedicated servers has experienced a problem. The idea of having a mail server die and have someone else’s server take the IP address and receive the mail is too horrible to contemplate, so an Elastic IP address is required. It is quite OK to have a set of mail servers of which not all servers run all the time (this is why the MX record was introduced to the DNS) so having a server run periodically at periods of high load (one of the benefits of the EC2 service) will not require changes to the DNS. I think it’s reasonably important to minimise the amount of changes to the DNS due to the possibility of accidentally breaking it (which is a real catastrophe) and the possibility of servers caching DNS data for longer than they should. The alternative is to change the MX record to not point to the hostname of the server when the instance is terminated. I will be interested to read comments on this issue. The command ec2-allocate-address will allocate a public IP address for your use. Once the address is allocated it will cost $0.01 per hour whenever it is unused. There are also commands ec2-describe-addresses (to list all addresses allocated to you), ec2-release-address (to release an allocated address), ec2-associate-address to associate an IP address with a running instance, and ec2-disassociate-address to remove such an association. The command “ec2-associate-address -i INSTANCE ADDRESS” will associate an IP address with the specified instance (replace INSTANCE with the instance ID - a code starting with “i-” that is returned from ec2-describe-instances. The command “ec2-describe-instances grep ^INSTANCE cut -f2” will give you a list of all instance IDs in use - this is handy if your use of EC2 involves only one active instance at a time (all the EC2 API commands give output in tab-separated lists and can be easily manipulated with grep and cut). Associating an IP address with an instance is documented as taking several minutes, while Amazon provides no guarantees or precise figures as to how long the various operations take it seems that assigning an IP address is one of the slower operations. I expect that is due to the requirement for reconfiguring a firewall device (which services dozens or maybe hundreds of nodes) while creating or terminating an instance is an operation that is limited in scope to a single Xen host. One result that I didn’t expect was that associating an elastic address is that the original address that was assigned to the instance is removed. I had a ssh connection open to an instance when I assigned an elastic address and my connection was broken. It makes sense to remove addresses that aren’t needed (IPv4 addresses are a precious commodity) and further reading of the documentation revealed that this is the documented behavior. One thing I have not yet investigated is whether assigning an IP address from one instance to another is atomic. Taking a few minutes to assign an IP address is usually no big deal, but having an IP address be unusable for a few minutes while in the process of transitioning between servers would be quite inconvenient. It seems that a reasonably common desire would be to have a small instance running and to then transition the IP address to a large (or high-CPU) instance if the load gets high, having this happen without the users noticing would be a good thing.

• [1] http://etbe.coker.com.au/2008/11/04/basics-of-ec2/

Share This

The aforementioned multi-seat setup allowing two kids with two screens/keyboards/mice connected to one computer required a minor update under the new Ubuntu release. The Xephyr 'x11-server inside an x11-server' that is used to display two distinct sessions on two distinct monitors didn't seem to recognise the mice and keyboards anymore. Somehow not explicitly specifying them helped. I.e. the calls to the script /usr/sbin/Xephyr-path.sh from /etc/gdm/gdm.conf now read

[server-Xephyr1]
name=Xephyr1
command=/usr/sbin/Xephyr-path.sh -display :0 -br -dpi 100 -xauthority /var/lib/gdm/:0.Xauth -screen 1280x1024
handled=true
flexible=false
[server-Xephyr2]
name=Xephyr2
command=/usr/sbin/Xephyr-path.sh -display :0 -br -dpi 100 -xauthority /var/lib/gdm/:0.Xauth -screen 1280x1024+1280+0
handled=true
flexible=false

Otherwise, the tutorial referenced in my earlier post still applies. And the kids are very impressed with new eye candy in KDE 4.1.

Today’s the last day for the Gov.UK Consultation on Forced ISP Snooping: Please Say NO. I didn’t get to LinuxExpo, partly because of Monday Machine-Mangling Madness but I saw LinuxExpo 2008 Robert Castelo from the Drupal point-of-view. I also didn’t get to Public Sector Online (report by Andrew Lewin with links to more) and I definitely didn’t travel to Toronto, so the report on the Free Software and Open Source Symposium from Randy Metcalfe is pretty welcome. I wrote about the Social Enterprise Photography Competition and Cooperatives South-West Trade Fair last Friday - the photography competition deadline has been extended to 7 November. So you’ve still time to capture a vision of a more progressive approach to business that values people and the planet as well as profit. Last Friday was Get Online Day but TTLLP didn’t do anything for it this year - I took one look and reached similar conclusions to Graham Jones: Get online…! Oh - you are…! Related to that: on Tuesday, I got really annoyed about how many branches of government (national, regional, sub-regional and local) are obsessed about the private sector and are paying even less attention to other types of businesses, includingbuying off-the-shelf software which doesn’t conform to their stated policies about accessibility and interoperability. (Sorry if you saw evidence of that outburst… yes, local councillors are pretty powerless and get as annoyed about incompetent government services as everyone else.) It seems I’m not the only one feeling that good cooperative businesses are being overlooked. The ICA have recently sent an open letter to the Governments of the G-8 reminding them that co-operative financial institutions are an alternative secure, stable and sustainable model of business owned and controlled by people. Finally, a bit of advance notice here: “The inaugural Creativity & Business International Network (c&binet) will take place at The Grove in Hertfordshire from 26 28 October 2009” - is this something that free software and free culture supporters should get involved with? (And why did they pick a name that includes an XML special character?)

Phew... that was a lot of work. I have a bunch of small sites that I migrated to slicehost, and at the same time upgraded to Rails 2.1. Here are my notes:

• x86_64 sucks up memory in a bad way. On a 32-bit x86 system davidw 20089 0.0 0.1 3320 1428 pts/1 S+ 11:41 0:00 ruby and on x86-64: davidw 24197 0.0 0.1 16756 1968 pts/0 S+ 18:33 0:00 ruby Ouch! That means that you'll need more memory, and thus to spend more money, to run code that might have fit decently in a similar amount of memory on an x86 machine. For instance, my old server had a gig, and was running low on memory, so on slicehost, I set up one 512 meg slice to handle PostgreSQL, and a gig slice to handle Apache and Rails. The database slice is performing ok, but the 'web' one is kind of iffy in terms of memory. Part of the problem here, I will admit, is that Rails is a memory hog, but still, I had hoped to come out ahead by moving from a machine with 1 gig to two machines with 1.5 gigs between them, and instead I'm more or less where I was before.
• I got caught up in a bit of a mess with Rails and the Ubuntu Intrepid update. I prefer to manage the Ruby stuff with gems, rather than traditional .deb package management, as I've had better luck with that in the past. I do use some of the system packages though, such as libpgsql-ruby1.8. However, the maintainers did something kind of ugly with the package: they swapped out the actual code from one code base to another - "Switched upstream source from ruby-postgres to ruby-pg". Which changes the way you load it, and perhaps a few other things as well. I think I would have been happier seeing the package renamed or something, as all of a sudden, programs stopped working.
• I really like the ability to resize slices. I started both my slices at the 256 size, and then made them bigger when I was ready to put them into production. The process is not instantaneous, but it's pretty handy if you need more capacity in a hurry. Adding new slices is also fast.

So, all in all, I'm happy with the move, although the memory issue is irritating.

I was just reading the latest catalogue from Dick Smith Electronics (a chain of computer stores in Australia). The first interesting thing that I noticed is that laptops are cheaper than desktops in all categories. For any combination of CPU power and RAM in a desktop system I can see a laptop advertised with similar specs at a lower price. Of course you won’t get such a big display in a laptop, but big displays don’t always work well. I just read an interesting review of LCD display technology [1] which states (among other things) that TN panels (which provide poor colors and a limited viewing angle) are used in all current 22 inch monitors! They state that the Dell 2007WFP (which I own) comes in two versions, I was fortunate to get the one that doesn’t suck. Based on that review I think I’ll refrain from all further monitor purchases until the technology gets sorted out and it becomes possible to reliably buy the better monitors at a decent price. The most expensive desktop system that Dick Smith advertised in their catalogue has a 22 inch monitor. It seems that with desktop systems being more expensive an increasing number of home users will use laptops instead, which will of course change the economics of manufacture. Maybe the desktop computer is about to die out and be replaced by laptops, PDAs, and mobile phone type devices (blackberries etc). Another interesting thing is an advert for a LASER pointer (it seems that they haven’t been banned as “terrorist weapons” yet). Being on special for a mere $27 is not the interesting thing, what is interesting is that the advert claims “projects up to 500m indoors“. I’m sure it will be handy if I ever have to give a presentation at the Airbus factory. But otherwise it seems quite unlikely that I will ever get an opportunity for a 500m indoor space. The prices on digital cameras have been dropping consistently for some time. Now they are selling a Samsung S860 (8.1MP with 3* optical zoom) for $98. This is (according to the specs at least) a very powerful camera for a price that most people won’t think twice about. I expect that an increasing number of people will buy new digital cameras every year the way white-box enthusiasts buy new motherboards! Hopefully people will use services such as Freecycle [2] to dispose of all their old cameras, to both avoid pollution and get cameras into the hands of more people. Very few monitors are being sold with resolutions greater than 2MP (1680*1050 is the highest you can get for a reasonable price). So an 8MP camera allows significant scope for cropping and resizing an image before publishing it on the web. Even the 4MP cameras that were on sale a few years ago (and which are probably being discarded now) are more than adequate for such use.

• [1] http://www.pchardwarehelp.com/guides/lcd-panel-types.php


• [2] http://www.freecycle.org/

Share This

Here’s a blog post suggesting that anti-depressant drugs such as Prozac may have helped the US mortgage crisis [1]. Apparently such drugs cause poor impulse control, so it wouldn’t be a good idea to attend a house auction while using them. Here’s an interesting idea about lecturing, give 20 minute talks with something else (practical work or group discussion) in between [2]. Michael Lee wants to “capture the power of that strict time limit, the intensity of a well-crafted 20 minutes”. While I’m not sure that a strict time limit is such a great idea. Having talks broken up into sections sounds like it has the potential to offer some benefits. A bible from the 4th century has been found and is being digitised [3]. When the digitisation is complete (next year) it will be published on the net so everyone can see how the bible has changed over the years. Interesting interview with Jim Gray (of MS Research) about storage [4]. It was conducted in 2003 so technology has moved on, but the concepts remain. His ideas for sharing two terabytes of data by using a courier to deliver an NFS or CIFS file server are interesting, the same thing could be done today with five terabytes for a lower cost. Techtarget has a white paper sponsored by Intel about the price/performance of data centers in low-density and high-density designs [5]. I don’t think I’ll ever be in a position to design a data center, but the background information in the paper is very useful. Google has an interesting set of pages describing their efforts to save power in their data centers [6]. They claim to have the most efficient server rooms ever built, and describe how it saves them a lot of money. One of the interesting things that they do is to use evaporative cooling as the primary cooling method. They also have a RE<C (Renewable Energy cheaper than Coal) project [7]. Here’s a Youtube video of an interesting presentation by Andy Thomson (a psychiatrist at the University of Virginia) about male-bonded coalitionary violence [8]. He shows the evidence of it in chimpanzees, humans, and evidence for it being in the common ancestry of chimps and humans (5-6 million years ago). He also shows a link to modern suicide bombing. It’s widely regarded that Cyrus is the fastest IMAP server. Linux-Magazin.de published an article last year comparing Cyrus, UW-IMAP, Dovecot, and Courier and the conclusion is that Courier and Dovecot are the winners [9]. I used Google Translation but the results were not particularly good so I think I missed some of the points that they were trying to make.

• [1] http://www.thedailybeast.com/blogs-and-stories/2008-10-05/did-antidepressants-cause-the-mortgage-crisis/


• [2] http://addisonrd.com/WordPress/2008/09/do-it-first-then-well-talk/


• [3] http://news.bbc.co.uk/2/hi/uk_news/magazine/7651105.stm


• [4] http://www.acmqueue.org/modules.php?name=Content&pa=showpage&pid=43


• [5] http://go.techtarget.com/r/4692606/8308947


• [6] http://www.google.com/corporate/datacenters/


• [7] http://www.google.org/rec.html


• [8] http://www.youtube.com/watch?v=uuipt15s08c


• [9] http://www.linux-magazin.de/heft_abo/ausgaben/2007/06/auf_der_teststrecke/

Share This

Preface: legal stuff Depending on where you live, use of random open wifi networks might be considered illegal. Although I know of no law in my country against doing this, where you live, things may be different, so educate yourself before following my example. Wifi roaming on the move For the past year I have used my Eee PC on the metro transit bus, as it is the perfect size for this. When I can, I take advantage of open networks to connect back home. In this article I will show how you can actually use such access, will recommend which software works best and will explain how to set it up. So, what can I really do with the net while in motion? How about irc? No kidding! This shell one-liner does the job:

while ! ssh -t 10.9.8.7 'screen -UDr' 2>/dev/null ; do echo -n "." ; sleep .1 ; done

We’re concerned about privacy and security on other people’s networks, so we use ssh. Inside that screen session, keep irssi running for a continuous irc presence. Every 10th of a second, the script tries to resume your session. Once a the net passes by, typing “[Enter]~.” drops you back out to retry again. The row of dots serves as a progress bar. If you want to get fancy, you could replace that with an ASCII ‘spinner’. I leave that as an exercise for the reader. :) Come on, is that really usable? Certainly. As the good folks on #eeepc at irc.freenode.net and #debian-eeepc at irc.oftc.net can attest, even a half dozen brief connections can be enough to provide tech support, rough out some development ideas, or just carry on some friendly chat. If your spouse or housemate is online too, it can even help you make some last-minute grocery-list changes. :) While the script is working, enjoy some offline things such as blogging, reading a cached Planet Debian feed or a PDF book. OK, so what else do I need to make it work? Well, you’ve seen so far we need on a server:

• openssh-server
• screen
• irssi

On the Eee, you should also have:

• openssh-client
• wpasupplicant
• gnome-terminal

But what do I need wpasupplicant for if I only use open networks? Well, wpasupplicant is really handy for fast control of networks in Roaming Mode (as per /usr/share/doc/wpasupplicant/README.modes.gz). A GUI is just too slow. For example, you may need to filter out ‘junk’ open wifi nets: pay-per-use hotspots or nets that hide their name. You’ll not only want a “catch-all” stanza for any open network, but also a number of the most commonly found default network names, e.g.

# Catch-all; associates with any open network:
network= 
     key_mgmt=NONE
 
# Commonly found default network names:
network= 
     ssid="default" 
     key_mgmt=NONE
 
network= 
     ssid="linksys" 
     key_mgmt=NONE
 
# etc. by watching scan_results closely you'll find several more

While the connect script is running, keep a separate tab open in gnome-terminal for wpa_cli. Use ‘disable_network 0’ to drop the catch-all and let the other stanzas kick in and ‘enable_network 0’ after filtering is no longer needed. Isn’t it really slow to get connected? Without some tweaks, yes. But here are a few things I have found by trial and error that help. One issue is that dhclient takes too long to retry, so you drop the delay between retries back to 1 second in /etc/dhcp3/dhclient.conf:

backoff-cutoff 1;
initial-interval 1;

If the signal is very weak, you’ll want the transmit power turned up to the maximum value for your hardware. For the Eee PC 4G this is 50mW, so you set this in /etc/network/interfaces for ath0, your wireless interface:

wireless-txpower 50mW

Finally, while you’re in motion you are constantly leaving no-longer-usable access points behind. The driver needs to discard them as quickly as possible and scan for new ones. So again in /etc/network/interfaces for ath0, you bring the default scanvalid 60 seconds down to 10 with:

pre-up /sbin/iwpriv ath0 scanvalid 10

Conclusion The netbook has changed how we use the net. Now that we can take it everywhere, we either pay through the nose for expensive cell network access or else use the patchy wifi coverage blanketing most urban areas. A few lucky cities have such things as municipal wifi nets, but the rest of us have to scrape what access we can wherever we go. You are now equipped to try this out on the bus. If you find any other tips that can help, I’d love to hear from you.

I've just started a new job, so I decided to take a few days holiday in between and work on something that's been on my todo list for way too long now -- making the debian-installer work on the D-Link DNS-323. After a couple of days of work, I've just finished off the install of Debian on my DNS-323, which I initiated by loading a firmware image into the web interface of the stock firmware, and then SSHing into the machine at the appropriate moment. What this means is that anyone with a DNS-323 can install Debian on it. I think that's worthy of a small, quiet "woo hoo!"... Looking back now, it seems to have been a fairly simple and straightforward operation, but I can remember that it didn't look so simple when I started a week ago. So, in the interests of perhaps short-cutting someone else's adventure into the world of d-i hacking, I'm writing everything up while it's fresh in my mind (the benefits of being on holidays -- plenty of time to write long blog posts). Alternately, if anyone has a random device they'd like to see d-i on but doesn't have the inclination to do it themselves, feel free to send me one and I'll do it for you. Before I begin, I've got a few "shout outs" (I think that's what the cool kids are calling it these days) to people who've helped me along the way. First off, Martin Michlmayr has answered innumerable dumb questions from me, and I think wrote most of the parts of d-i that I've extended -- it's fair to say that I'd have had no hope of doing this without him. Thanks also to the people in #debian-boot and #debian-arm for answering questions and putting up with my frequent bouts of self-congratulation whenever I passed the most miniscule milestone. Finally, everyone who has put work into d-i has my thanks, too -- it's not perfect, but it's a damn sight better (both to use and also to hack on) than any other installer I've ever met. So, without further ado, off we go... Work out how to get a console Unless you are a guru of unbelievable skill, you will undoubtedly not do everything quite perfectly the first time, and you'll want to see the kernel/init boot messages, and maybe in some cases try some investigation and diagnostic work. For that, you're going to have to have some sort of console access. If the device doesn't have a full screen/keyboard thing going on, the chances are you'll almost certainly end up with a serial console. A few devices have a network management thingo that will give you a console, usually over VNC or some sort of horrid Java applet, but I've only ever encountered that in rackmount servers, which have a keyboard/video interface anyway... In theory, I suppose, the kernel could provide a network console, but I've never heard of such a thing being attempted, and Google is not helping me out. I might add that to the todo list as a future hacking project... On the DNS-323, it was a serial console that was needed. Like a lot of the embedded devices I've seen, it didn't quite talk RS-232, so I needed an adapter (I got a TTL-232R-3V3 USB-to-3.3v-serial adapter from a local electronics supply place, which worked perfectly with Linux as soon as I plugged it in). I needed to solder a few wires to a connecter inside, and connect that up to the USB doohickey, but it all came good in the end. The chances are someone's done a similar job for your device of choice, so a search for things like "devicename serial console" will almost certainly pay off. Work out how to load and boot a kernel/initrd Your development work will involve booting weird kernels and initial ramdisks to make everything work; you need some way of stuffing those into the device and booting from them. Everything has a boot loader of one sort or another; working out what you've got and how to make it sing and dance is the ultimate quest here. How this is done will vary greatly from device to device; the WRT-54GL, for example, waits for a tftp connection for a couple of seconds on startup, which you can use to send a new firmware; many other devices have a boot loader that you have to explicitly control (via the console), but it will make a tftp connection out to another machine if you ask it to. Some devices you just need to load the data in via serial cable (at 115200bps it's bearable; at the 9600bps I've heard some devices work at, it would be tres painful). The DNS-323 runs uBoot as the boot loader, which will do a tftp request by default. However, D-link, or Marvell, or someone in the decision-making chain for this thing decided to knobble it and removed most of the useful commands, so sending the thing via serial port is the only option. Since writing to flash takes a pile of extra time, I found (eventually) that you can just load the data into RAM somewhere and boot from there, without needing to write it to flash (also handy to avoid wasting precious write cycles), since you can tell uBoot to boot a kernel located anywhere in the address space. Obviously you need to reload the kernel/initrd on every boot if you don't write it to flash, so if you've got a "stable" kernel or initrd that you think you'll be using for a while (when you're testing the other item) you should flash it then. One note on uBoot, that took me far too long to work out -- you can't just stuff a kernel fresh off the Makefile in and assume it'll work -- uBoot needs a special header prepended that tells it all about what you're loading (architecture, file type, compression method, etc). You "wrap" a file in this uBoot header with the "mkimage" command (in the uboot-mkimage package in Debian). Incidentally, if you need to remove the uBoot bits from a file, just strip the first 64 bytes off -- that's all the uBoot header is, so something like this will do nicely:

 dd if=file.uboot of=file bs=64 skip=1

A final uBoot warning: it's fairly important to specify a "load address" for the files; if you get them wrong, your kernel and initrd may or may not boot (I never quite got the hang of why some wrong values worked and some didn't). If your device uses uBoot, and you have the ability to extract the uBoot-enabled kernel and initrd from a vendor-provided firmware image, you can use mkimage -l to inspect what all the values need to be for your own kernels and initrds. Just stick with them. Does Debian support your hardware? Obviously, if you want to run Debian on something, Debian has to support that hardware. The easiest way to test this out is to try an install. For this, you'll need to retrieve an appropriate kernel and initrd pair for the installer. Some arches, like i386 and amd64, only have one kernel and initrd for the installer, so you just have to choose if you're going to netboot, or do a full CD install, or whatever. Other arches, though, like arm, have subtly different images for different machines. There are two reasons they vary: firstly, they often need to split into subarches, where they all share a general CPU architecture but there are enough differences in the hardware that they're not all supportable by the same kernel. At a more fine-grained level some different machines of the same sub-architecture need differently formatted firmware images to boot. On ARM, in particular, it's pretty common to need to prepend a "device ID" to the kernel image so that the kernel knows what sort of machine it's booting on, which means that every machine typically needs it's own kernel image, regardless of any other issues with firmware formats. Choosing the sub-architecture should be relatively straightforward, if you know much about the machine you're targetting. Picking a machine type can be a bit harder, but again, if you know what your boot loader needs, it shouldn't be too hard to find a kernel/initrd from the existing machines in your subarch that you can mangle to fit. Load up your kernel/initrd into the device, boot it, and see how far it gets. You might get lucky, and the kernel inits, your hard drives are detected, your NIC is initialised, and you're off and running without further ado. Or you might not be so lucky. The worst case is that the kernel doesn't even get as far as loading the initrd -- it might not print anything at all, or the kernel might hang part way through the boot. This is likely to require a fair amount of debugging fun, as initial boot failures get a bit tricky. A better situation is where the kernel boots OK, and d-i starts, but something in the install process falls over. This can either be a kernel/driver problem, or a d-i problem. Either way, since d-i has started, you're likely to be able to get a shell and do some debugging. In order to make d-i work, you need to have somewhere to install to (disks, usually) and a source of packages to install from (CD/DVD if you're on a suitably large device, or the network otherwise). These are the two key ingredients you need; most everything else is stuff that ranges between "nice to have" and "critical" for operational purposes, but which has absolutely zero impact on your ability to do the install, so we'll ignore it for now. For the DNS-323, I had a few kernel problems (which I'd been warned about by Martin before I started) -- the NIC wasn't initialised with it's MAC address, which meant it was running around my Ethernet screaming "I am 00:00:00:00:00:00! Ph33r m3!", and the SATA controller just didn't seem to exist at all. These were kinda critical to doing an install, so it was time to get hacking to make those work. There were also a couple of parts of d-i that needed tweaking -- the preseeding code didn't support the device, and I needed to package a tool to build a "complete" uploadable firmware image and teach d-i how to use it. Apart from that, though, things were looking pretty good. Make the Debian kernel support your hardware Once you've worked out what hardware you need to make work, you need to dig into the kernel and fix it. Sometimes, if you're really lucky, a kernel version newer than the one currently in Debian has better support for something, and you can backport the fix / feature improvement. Other times, there's something similar already in the kernel for a different device and you can crib off that (I did this extensively for my little fixes). Then there's the "ask an expert" school, where you find someone who knows what they're on about and pick their brain (I did a bit of that, too). Finally, there's plain old "research and development", for which there are piles of documentation already out there. My changes for the DNS-323 came to 121 added lines and 16 deleted lines, most of which was cribbed-and-tweaked from other parts of the kernel. I might have written 10-20 lines of code "from scratch". It's not always scary-hard. Cross-compiling your own kernel One thing that I think needs mentioning in the kernel hacking department is cross-compiling a modified kernel for testing purposes. Chances are your main workstation has a lot more grunt (and build tools) than whatever you're trying to get Debian onto, so cross-compilation is the way to go. You need to first grab a copy of the source you want to build (from apt-get source linux-image-blahdiblah, or a release tarball, or Linus' git repo, it doesn't really matter) and a cross-compilation toolchain. For my cross-compilation toolchain, I just added this to my sources.list:

 deb http://www.emdebian.org/debian/ unstable main

then I ran apt-get update, and installed these packages:

	binutils-arm-linux-gnueabi gcc-4.1-arm-linux-gnueabi
	gcc-4.3-arm-linux-gnueabi-base libc6-dev-armel-cross

Kudos to the emdebian team for making that such an utterly painless exercise, by the way -- the days of needing to faff round with gcc sources to get a cross-compilation toolchain are apparently over. Once you've got a source tree and a cross-compiler available, you can run any of the usual kernel make commands (like make zImage or make oldconfig), as long as you've got a couple of make variables defined. In my case, I used:

 ARCH=arm CROSS_COMPILE=arm-linux-gnueabi-

ARCH tells the kernel what architecture to build for (useful in all make commands because not all arches have the same config options in make menuconfig, for example) and CROSS_COMPILE tells the build system what to prefix the calls to gcc, ld and such with (if you installed the same cross-compiler packages as I did, you'll find it installs files like /usr/bin/arm-linux-gnueabi-gcc, so you can ls /usr/bin/*gcc to find out what you'll need for the CROSS_COMPILE setting). Hack d-i to make things work In addition to tweaking the kernel, it's quite possible that the internals of d-i might need some tuning. It's possible that d-i might need to be "taught" to recognise the hardware in your device, or you'll need to extend some aspect of the installation process to complete the installation. In order to hack d-i, you need to know a bit about how it works and how it is structured, so let's run through that first. The entireity of d-i is based around udebs, which are packages of the same format as "real" debs, but which have been built differently (against a small libc, for instance) and have had everything that isn't absolutely necessary (docs, non-essential scripts, extra functionality, etc) stripped out of them to create the smallest possible packages. There are a variety of different udebs -- most are derived from otherwise normal, law-abiding citizens of the archive, like busybox (basic unix utilities like cp and ls), the shell that's used in the installer, parted, various libraries, and so on. Others are "special", and have no big brother in the archive. These are mostly "task" udebs, that define what actually happens in the installer. When they're "installed", they have special fields in the package metadata that define where in the install sequence they go, and they end up on the menu you see if your installation goes a bit pear-shaped. The tricky thing about udebs is that, as small as they are, there's too many of them to be able to have them all available when the system first boots, so even the "setup" of the installation gets done in several stages. First, the kernel and initrd boot. The initrd contains just enough stuff to get access to the rest of the udebs, and nothing else. Even this early stage is built from udebs -- there's lists of what goes into the initrd for various architectures and flavours of installer under installer/build/pkg-list in the d-i SVN. If you need an extra udeb in your initrd, that's the place to specify it. Once the initrd has gotten access to a source of packages (off a CD, over the network, telepathy, whatever) it can drag the rest of the packages it needs from there. Space is still an issue at this point, though -- there's no diskspace yet to put anything onto, so everything needs to fit into a ramdisk. Don't go loading anything you don't need. The packages loaded at this stage are what defines most of the steps that will be done for the rest of the install, so once all those extra udebs are loaded, the remainder of the install kicks off and does it's thing. Different architectures and install methods have different requirements, too, which complicates matters. For example, when you're installing on a regular PC with screen/keyboard, you can guide the install using that. However, if you've got no screen, how can you control the install? There's serial consoles, but requiring everyone to void the warranty (more) on their little embedded doodad by installing a serial console cable to install Debian isn't so hot. So, lacking a terminal, how do you control the install? Well, on install flavours for devices that might be in this situation, you can SSH into the installer and complete it that way. That feature is, like everything else, provided in a udeb -- called network-console, as it happens. But what IP address do you connect to, and what network settings does the device use? Again, without a console you can't configure your network, and not everyone has a DHCP server (and even fewer people have a DHCP server that will hand out an address that they'll know in advance, or be able to find out). So, there's a udeb (called oldsys-preseed) that does it's best to configure your network based on the settings already in the nvram for your device. In short, there's a lot of tricky, well-thought-out shit going on inside d-i, and you might have to deal with a little bit of it. The nice thing is that by now I expect most of the really hard problems have been solved, so with a bit of luck you'll only ever have to do minor tweaking. In general, all of the packages that are "d-i specific" live in packages in the d-i SVN. If you need to modify a udeb, you need to rebuild it and put it in installer/build/localudebs/ (creating it first if need be) and every time you change it, you need to delete installer/build/apt.udeb/cache/. To give a concrete example (from the ever-present DNS-323), I had to modify oldsys-preseed so that it would get the details from the nvram that the stock firmware uses to configure itself. This, in turn, required that the minix-modules udeb be included in the initrd, because the DNS-323's nvram is actually a small minix partition stored in flash. Painfully, this required some more hacking, because the armel d-i kernel wasn't configured to generate minix-modules. That might need some explanation... Since building the kernel is a fairly massive undertaking, CPU-wise, instead of building all the udebs from scratch there's a package called kernel-wedge to split a real linux-image package into lots of little udebs. There's a separate source package for each architecture, linux-kernel-di-$ARCH-2.6, that contains the configuration data for kernel-wedge to specify what udebs need to be built, and what goes into each of them. A small tweak to the armel package and voila! I had a minix-modules udeb to include in the initrd. With the config stuff I'd done in the d-i build itself and my changes to oldsys-preseed, we were done with d-i. Tell d-i how to build your bootable artifacts If your device needs a "different" install image from other similar devices, you need to write some makefile rules to build those files. Look under the installer/build/config/$ARCH/ directory to see the various rules files available. I had to package a new tool to create uploadable firmware images for the DNS-323, along with adding a rule to use that tool to build images. None of it was particularly difficult. Polish the diamond Getting d-i working is only part of the battle; it's quite likely that there will be other hardware in the machine that needs to be supported. This could involve kernel-level stuff to interface to the hardware, or userland tools to manage the hardware. There's several bits in the DNS-323 that still need work; I can't control the LEDs on the front panel, nor the fan, and I can't use the on-board temperature sensors yet. But, I do have Debian running on the thing, which is what I was shooting for. Examples If you're interested in exactly what I did, there are a number of bugs in the BTS that contain the patches I made:

• #502821: oldsys-preseed changes to take existing network config from NVRAM;
• #502936: build a minix modules package for the armel d-i kernel;
• #503040: d-i core changes to build firmware images for the DNS-323;
• #503172: kernel patches to support the SATA controller and NIC better.

Sure, Martin, you started a meme (by the way, I roughly guessed you naming scheme which is a nice one). I wish I would have named my machines this way: patrick, bob, sandy, eugene, plankton, carlo, gary, puff, pearl, etc. (easy to guess for people who went to Debconf's day trips). Indeed, I haven't because I named the first ones too long time ago. So they're named kheops, khephren, mykerinos, snefrou, saqqarah, merenptah, amenophis, sethi, djedefre. Easy to guess. PS: 'carlo' is only for French-speaking users. For en_*, thata would be squidward.

I played with a friend’s Atom based EEEPC yesterday. The EEEPC is an amazing achievement considering it’s comparable to the Thinkpad X61, yet it’s a fifth of the price in the UK. I like the idea of carrying an EEEPC as I’m a bit nervous about losing my $$$ Thinkpad when I am out and about. If it just runs a kiosk like browser, with a HSPDA 3G modem, it would be very handy. I wouldn’t sob as much if it got stolen. In other news: I upgraded my beloved slug to lenny which I use as a NAS at home, only to get bitten by a serious NFS permission issue in Debian. Urgh. Update: Thinkpad lovers rejoice. Lenovo has released the EEEPC rival SL300, which seems like half the price of the X61.

I think I want a netbook. Although I'm still happy with my R200 (especially since I replaced the battery) something smaller would be handy for the train and general carting about. With the R200 being 12" I'm more drawn to the 8.9" variants; 10" isn't a lot smaller and at the cheaper end doesn't appear to result in any higher a resolution than 8.9"

Really I'd like something with built in 3G/HSDPA, but only the Dell Inspiron Mini 9 seems to have this, and the Linux model isn't even on sale yet in the UK, let alone the 3G version.

If I can't get 3G then I'm drawn to the EEE 901. It seems to have the best battery life, has bluetooth and isn't too expensive (though at the higher end of what I'd want to pay for a 2nd machine). Plus it has the appeal that I should be able to walk into a shop and pick one up. Main drawback is that I've read the Ralink RT2860 wifi ain't great (and the driver isn't in mainline yet, though there is one kicking around). Any truth in this? Any other gotchas?

Thoughts, people? I've seen an MSI Wind in the flesh and it was a bit big. The Acer Aspire One looks nice enough, but not great battery life and no bluetooth. Any others I should be looking at instead of the 901? Must be easily available in the UK. I will, of course, be running Debian on it.

Congratulations once again to Roger Federer, who didn t leave a doubt that he is a Great Player. Andy Murray, I tip my hat to you! You are someone to watch for: great ball control and you just hit it, as opposed to some other players slicing their way through the games. Keep it coming, I expect to see you more in the future! I was very pissed off by usopen.org and CBS sports, for their streaming was limited to US and territories . Since Eurosport and the like didn t get live coverage, I am assuming that there was shitloads of money involved (the US Open Finals, go figure ). Well, dear CBS and US Open folks, if you pocket shitloads of money, consider spending that tiny bit of a fraction in the future to make your international viewers happy. I assume you know that there s life outside of your country. Thanks to justin.tv for hooking me up with a stream that allowed me to follow, even though it was suboptimal. I wish I had been able to see this on a TV. Since we don t have one, and Irish pubs kick you out at half eleven, it was still the best option I had.

A few speakers at DebConf8 used gobby during their talks to collate ideas without breaking the flow of the session.
I've just come back from the Emdebian meeting in Extremadura (of which more later) where we also used Gobby - FAI used it too. Once the network setup was working, it was simply a case of clicking Create Session in gobby on my laptop and everyone else clicking Join Session, entering the IP of my laptop (only accessible locally) and choosing a colour.
Gobby is real-time, currently plain-text only but clearly indicates how sections have been edited and has an integrated IRC box for each session. At the end of the meeting, every gobby user has a copy of every document they opened in every session they joined. updated to the moment they quit the session.
Based on using gobby in events like this, I'd strongly recommend that someone at each worksession-type meeting starts a gobby session and maintains that session for the duration of the meeting.
Gobby even does syntax highlighting - depending on how many people join and the colours they choose, this might become difficult so it can be turned off too.
There are only a few minor issues:

1. The colour bar warning box isn't particularly intuitive - you need to click on what looks like a dimmed progress bar to select a unique colour for yourself.


2. I initially thought that Gobby had no authentication - in a local meeting, anyone who can connect to the specified IP address is able to edit the document. Since then, I've noticed it does support password authentication and there is mention of fully encrypted connections so it may be possible to use it over public networks.


3. According to Wikipedia, there is a gobby server that allows non-GNU/Linux clients to run gobby and join any gobby session - I didn't explore this and the Debian package doesn't mention anything about a server option or mixed-OS support.
   
   • Update - the server is called Sobby and is in Debian.
   
   


4. The right hand side of the window is a bit wasted - maybe some kind of sidebar would be handy, especially docking the userlist into the window so that edits can be identified more quickly.

For about eight years, I was a very happy WindowMaker user. It was very lightweight, aesthetically pleasing, and I had interiorized its behaviour and keybindings so much, I didn't feel I'd ever switch away. I periodically tried (forced myself even!) to use any of the other, more en vogue environments... Experimented using Gnome for a week, KDE for a week, XFCE for a week (so I would have enough time to learn their ways)... And always came back to my good, well-known wmaker.
In 2006, when we held DebConf6 in Mexico, I saw how other people worked with ion3. I fell immediately in love with it. Not because it is prettier, snappier or has nicer widgets than other window managers - but because it is enormously more usable. Quoting Tuomo Valkonen, the ion3 author,

Ion is not perfect and certainly not for everyone, but neither is any user interface. Usability is subjective.

Using a keyboard-oriented, tiling window manager represented -for the first time in 20 years (I had my first contact with a Macintosh in 1986)- a radical user interface change. For my way of working, I just don't need a desktop, I don't need having a background space or overlapping windows. What I need is a way to functionally organize the windows I have open at any given time, quickly switch between them (and not depending on the mouse, please!), maximizing screen space and all that. ion3 was godsent.
Now, in 2007 there was (yet another) huge flamefest. Valkonen basically does not want distributors distributing any version of ion3 that's not the latest, and introducing changes not approved by him - He basically demands ion to be non-free. So, it was moved to the non-free section of Debian where only Womble decided to keep giving it support. And, of course, by September 2007 Julien Danjou announced he had written another window manager: Awesome.
My main motivation for switching away from ion3 is that... I don't want to use non-free software. But I was very comfortable with ion3. It was only after I saw many other people using Awesome at DebConf8 I decided to bite the bullet and switch. It looked at least as comfortable as ion3.
But... Well, I cannot come up with better phrasing than what Joey said when he switched to Awesome, almost exactly a month ago. When changing between the mainstream window managers, the differences are mostly cosmetic. But with these really different window managers... I cannot but reproduce Joey's words:

I wish I had a good analogy to explain to my nontechnical readers what changing to a new window manager is about.
One way to think about it is that it's like driving a car down the road, and suddenly swapping the steering wheel and brakes out for a tiller and gear shifter. And having to downshift for braking until you learn that the brakes moved to the turn indicator lever. By trial and error.
But that's really only part of it. Another way to look at it is adopting a new philosophy. Or, in some cases a cult. (In some cases, with crazy cult leaders.) Whether they use Windows or a Mac, or Linux, most computer users are members of a big established religion, with some implicit assumptions, like "thy windows shall be overlapping, like papers on the desktop, and thou shalt move them with thy mouse".
So, changing to a new window manager is a process of being dumped into a different environment, where nothing works like you've come to expect, and trying to construct a mental model that you can use to make sense of it. But it's also a process of modifying that environment to behave the way you like.
And when done whole-heartedly, this doesn't just mean trying to make it like the environment you were used to before. It means trying to absorb the underlying philosopy of the window manager, and think up new ways of doing things, inspired by that philosopy, and modify the environment to allow doing those things.
So ideally, "I switched to a new window manager" doesn't mean "my screen has some different widgets on it now". It means "I'm looking at the screen with new eyes."

So, what's so different?
Besides learning some new keybindings (expected, of course), Awesome has several suggested layouts to help you organize your workflow, usually (although not always) consisting of a main area and a side area, tiled side by side (or one above the other, or several stranger ways). This sounds rigid, but it is incredibly comfortable - and I've only been an Awesome user for three days!
But what sets Awesome apart from basically anything else is the concept of tags. Whey you see an Awesome session, you recognize something similar to the very well known workspaces concept we have had in any Unix-like environment for many years, right? Well, but they are not workspaces. They are tags. What is the difference?
When I work with workspace, each of my windows can be in a different workspace. So in one, I'll have my mail-related stuff. In another one, I might be browsing. In another one, I have my development things, and I might be following some logfiles in yet another one.
Awesome allows you to use tags for categorization in much a more flexible way.
For example: I am mostly a web-oriented developer. I usually need four things when developing a system: A browser, Emacs, the log for my development server, and a console where I can peek and poke at my objects and interactions. Of course, cramming them all into the same screen makes no sense - It would be better to follow the good ol' desktop metaphore, and just switch the focus and raise the window, right?
In Awesome, I can have them all set to the maximum screen size - _and_ use the most common combinations as well. Each of the windows can be tagged to more than one workspace (and yes, this is immensely more flexible than the always visible hint on traditional WMs).
To begin with, I'll give each major process a tag to itself, to work full-screen. Emacs is on tag 1, my browser on tag 2. The log and the console share a terminal (i.e. via screen or terminator).
This console by itself is not too useful, so I'll set it to tag 4, and we will go back to it later.
So if I'm building a view or following online documentation, I will add tag 3 to both Emacs and the browser.
I'm also setting tag 4 to the browser - That allows me to use it next to my terminal, following the results of my website-clicking.
And, of course, tag 5 will be set to Emacs and to the console, so I can quickly check any quick API-related question that does not need the documentation or look at a newly written method.
By the way, have you noticed CapsLock is the most stupid key invented, ever? Ok, I gave a good use to it: It's called Mod4. Imagine it is just an extra Ctrl, Alt or Meta key.
So, Mod4-1 gives me Emacs. Mod4-2 gives me the browser. Mod4-3 gives Emacs+browser, Mod4-4 gives browser+console, and Mod4-5 gives Emacs+console. And, of course, the handy Mod4-0 gives me all of my open windows tiled side by side.
Even this is a pattern of being a newbie, I know - I could keep 4 and 5 free, and just tag several simultaneous tags to be active. How? Switch to tag 1 exclusively (Mod4-1), and activate tag2 as well (Ctrl-Mod4-2). There, I have Emacs and browser side by side. Want to get the console instead of the browser? Simple. Ctrl-Mod4-2 (toggle off tag 2), Ctrl-Mod4-3 (toggle on tag 3).
Anyway - As you can see, I am excited at finding a very new and nice tool to help me work better. Today I was playing a bit with the Awesome widgets, but that's something to be talked about later.
In Debian, we are at a crossing point: Awesome is just reaching ion3's popularity. And I'm adding my two main machines' votes to Awesome.
This is Awesome. Quoting (yes, one last quote) the official Awesome site, This gonna be LEGEN... wait for it... DARY!.

This is 'back to school' season aroun here. So on Saturday we went and set up two desks for our two kids. And with that I finally converted 'their' computer to a working multi-seat setup. At first, I had fiddled with two distinct (ATI Radeon) graphics cards. Somehow I never got x11 to recognise both cards properly. But two cards are not needed. As the machine is running a standard Kubuntu setup, I just followed this excellent three-part tutorial for Ubuntu multi-seat setup which describes the process using nothing but standard Ubuntu software. From setting up a 'big desktop' spanning two screens (which is easy enough using one card via the vga and dvi outputs), it is fairly straightforward to modify the gdm.conf setup to spawn two gdm greeter instances using the Xephyr nesting xserver. So far, all is well. We'll see what possible shortcomings we will find. The GL extensions are not supported, so some eye-candy will be unavailable.

I've been meaning to write up on this for quite some time now, but always have something seemingly more important things to do. Most modern Linux distributions offer ways to manage network interfaces via some kind of abstraction. Usually this abstraction allows one to dynamically rename and add network interfaces. For Debian family management is done with ifupdown, while with RedHat family it's sysconfig. In addition there is NetworkManager which is a cross platform solution for dynamic network configuration (which eliminates the need to rename interfaces). Case Xenserver Usually this comes in rather handy, but on other occasions it can be a pain. A while I was helping out with an installation of a XenServer instance. This server had, for some reason, ethernet interfaces reversed in comparison to the other XenServers on the site. Luckily the service console has the network configuration in ifcfg scripts. We were easily able to reverse the interfaces by binding the interfaces to certain hardware addresses. The only problem is that the interface is renamed only when the interface is brought up. What's worse the interfaces were enumerated before the server would bring up all of the interfaces. Only eth0 was brought up for management purposes before enumerating. This means that the original eth0 was renamed to _tmp_xxxxxx. The (oh so elegant) solution was to create a script that does ifup eth1; ifdown eth1. Problem solved. Case tap0 I also experienced similar problems when I was setting up my OpenVPN tunnel. I wanted to use a bridged connection to my network, but for that I needed to create a br0 interface with tap0 as a member. It's easy enough to create ifupdown configuration to set up tap0. Hook that up to br0 and you are all set. The same problem will bite you here. The interface tap0 is actually created only when the tap0 is brought up and br0 members are added only if they are present. The solution? In my case, create a custom script that adds the device manually after it's been created properly. I was unable to find anything that works without using a custom script =( In retrospect Looking back at these cases, I should have known better. The problem was obvious and it took me way too long to figure out the cause for my problems. Then again, the utilities should be able to create a reasonable abstraction for themselves that they use to determine the actual status of the whole system. This way the trickery needed for setting up simple interface would be obsolete. You can't have it all, but you can always hope.

More notes and reactions from DebConf video. debbugs Hadn't realised how handy bts select is! My CVE-XXX-XXXX tags are a fairly large percentage of the 3000+ unique user tags. 2461 to be exact. :-) It would be nice to be able to view a list of all users of usertags, and their tags, in the BTS web interface. Yay! Don implemented the summary idea already.
(And I implemented bts summary during his talk.) Local partial debbugs mirror: Crazy idea, but better than bts cache. :-) (And if it provides an incentive for a current and proper debbugs deb to be maintained, that's a win.) vcs-pkg I won't bother once again repeating my responses to tired arguments like "git's archive format will change!" Been there, done that, been ignored; stopped maintaining any non-native packages; your loss. Topgit looks generally handy, and it was neat to have its author on irc during the talk. video team Hundreds of remote watchers -- impressive.. Looking forward to the remote controlled telepresence robots at debconf 10! (Also to a lack of tango loop music.) bugs in large packages Bug summaries are not yet shown in the "extra information" in the bug index page.

So hot off the back of my successful upgrade from Sarge to Etch, I thought I'd try the new "Etch and a Half" 2.6.24 kernel on daedalus Let's just say that it's times like these that I'm glad I got the serial-over-LAN thing working, and have remote power. I spent many hours today rebooting, trying to get to the bottom of why it wouldn't work. At first, I thought it was hanging, but then I realised it was dropping to a shell in the initramfs, and just not making that very obvious because the serial over LAN console is a bit crappy. Once I realised what was going on, I did some poking around. It seemed like udev wasn't getting started properly, so when it went to assemble the mirrors, that failed, because the component devices couldn't be found, then it freaked out because it couldn't mount the root filesystem. If I ran the relevant bits of /scripts/init-premount/udev by hand, the SCSI devices appeared, and I could manually assemble the mirror, and mount the root filesystem, which was handy, because I also discovered that you can boot with debug on the kernel command line, and it logs the initramfs run to /tmp/initramfs.debug. So that was a convenient way of preserving the log, because there seemed to be some characters in it that made inspecting it over the serial console difficult, and there was no way to get it off the machine from the initramfs environment. As far as I can determine, it's telling udev to start, but it certainly isn't still running when it bombs out to a shell after failing to mount the root filesystem. It's not immediately clear if there's something later on that is stopping it again. I've put the log here in case anyone's interested in looking at it. This was with the addition of "x" to the options of the shebang line of /usr/share/initramfs-tools/scripts/init-premount/udev so that I could see what was happening when /scripts/init-premount/udev ran. So I don't think the kernel itself is at fault, it's some sort of weird udev/initramfs interaction. Rather than further rebooting the tripe out of daedalus, I'll have to see if I can reproduce the problem on a less important machine locally to do further debugging. In the meantime, I'll have to stick with 2.6.18.

The notorious nutsack (cc-by-2.0) The conference pack for LugRadioLive was called a “nutsack” for some reason (anyone know why?). It seems like a pretty good pack, as far as these things go. First of all, it’s a durable paper bag, so much greener than plastic. It contained:

• the event programme/schedule
• a LUGRADIO LIVE badge which I only discovered when I tipped the bag out,
• a key-shaped yahoo keyring - I don’t know what it’s meant to symbolise and I already have one far-too-big novelty keyring - seems like a total waste of metal
• UKUUG t-shirt with amusing slogan on the back and a membership application form - tempting, especially since AFFS seems to be idling since I quit and ORG is a self-perpetuator AFAICT
• flier for PyCon UK 12-14 Sep 2008 Birmingham
• subscription form for the 50/year Linux Magazine
• flier for LinuxPro Magazine’s conference streaming which says “no special software is required”. Well, users actually have a choice of Java, MPlayer, Real, WindowsMediaPlayer or anything else which can view an Ogg Theora stream, which is pretty good, but not quite “no special software”. ffplay seemed unhappy in my first tests, with errors like “[vorbis @ 0xb7dbc430]Not a Vorbis I audio packet.
  [theora @ 0xb7dbc430]Warning, unsupported keyframe coding type?!
  [theora @ 0xb7dbc430]Header packet passed to frame decoder, skipping” which I’ll follow up RSN, unless any kind soul knows how to fix that.
• a bytemark pen - always handy to advertise a decent hosting company at people in business meetings, but I don’t think it’s refillable, so that’s not going to last long,
• a fun goodbye postcard from Birmingham Perl Mongers to LugRadio - it’s actually a digital photo print, which seems a really good idea that I’ve not seen done before
• CDs of ubuntu 8.04 LTS Desktop Edition (oh well) and openSUSE 11.0 (oh dear)

All in all, smarter than your average conference pack… or am I just thinking that because it looks a bit like a picnic?