Many people still install Linux from CDs. But a growing number install from USB. In an ideal world you'd be able to download one image that would let you do either, but it turns out that that's quite difficult. Shockingly enough, it's another situation where the system firmware exists to make your life difficult.
Booting a hard drive is pretty easy. The BIOS reads the first 512 bytes off the drive, copies them to RAM and executes them. That code is then responsible for either starting your bootloader or identifying the currently active partition and jumping to its boot sector, but before too long you're in a happy place where you're executing whatever you want to. Life is good. So you'd think that CDs would work in a similar way. The ISO 9660 format even leaves a whole 32KB at the start of a filesystem, which is enough space for a pretty awesome bootloader. But no. This is not how CDs work. That would be far too easy.
Let's imagine we're back in the 90s. People want to be able to boot off CD without needing a boot floppy to do so. And you're a PC vendor with a BIOS that's been lovingly[1] forced into a tiny piece of flash and which has to execute out of an almost as tiny piece of RAM if you want your users to be able to play any games. Letting boot code read arbitrary content off the CD would mean adding a new set of interrupt hooks, and that's going to be even more complicated because CDs have a sector size of 2K while hard drives are 512 bytes[2] and who's going to pay to implement this and for the extra flash and RAM and look surely there has to be another way?
So, of course, another way was found. The El Torito specification defines a way for shoving a reference to some linear blocks into the ISO 9660 header. The BIOS reads those blocks into memory and then redirects either the floppy or hard drive access interrupts (depending on the El Torito type) to that region. The boot code can then proceed as if it had been read off a floppy without all the trouble of actually putting a floppy in the machine, and the extra code required in the system BIOS is minimal.
USB sticks, however, are treated as hard drives. The BIOS won't look for El Torito images on them. Instead, it'll try to execute a boot sector. That isn't there on a CD image. Sigh.
A few years ago a piece of software called
isohybrid popped up and solved this problem nicely. isohybrid is a companion to isolinux, which itself is a bootloader that fits into an El Torito image and can then load your kernel and installer from CD. isohybrid takes an ISO image, adds an x86 boot sector and partition table and does some more fiddling to turn a valid ISO image into one that can be copied directly onto a USB stick and booted. The world suddenly becomes a better place.
But that's BIOS. EFI makes this easier, right? Right?
No. EFI does not make this easier.
Despite EFI being a modern firmware for the modern world[3], EFI implementations are not required to be able to understand ISO 9660. In fact, I've never seen one that does. FAT is all the spec requires, and FAT is typically all you get. Nor will EFI just execute some arbitrary boot code from the start of the CD. So, how does EFI boot off CD?
El Torito. Obviously.
It's not quite as bad as it sounds, merely almost as bad as it sounds. While the typical way of using El Torito for a long time was to use floppy or hard drive emulation, it also supports a "No emulation" mode. It also supports setting a type flag for your media, which means you can distinguish between images intended for BIOS booting and EFI booting. But the fact remains that your CD has to include an embedded FAT partition that then contains a bootloader that's able to read ISO 9660 because your firmware is too inept to handle that itself[4].
How about USB sticks? Thankfully, booting these on EFI doesn't require any boot sectors at all. Instead you just have to have a partition table, a FAT partition and a bootloader in a well known location in that FAT partition. The required partition is, in fact, identical to the one you need in an El Torito image. And so this is where we start introducing some extra hacks.
Like I said earlier, isohybrid fakes up an MBR and adds some boot code that points at the actual bootloader. It needs to do a little more on EFI. The first problem is that the isohybrid MBR partition has to cover the entire ISO 9660 filesystem on the USB stick so that the operating system can access it later, but the El Torito FAT image is inside that partition. A lot of MBR-based code becomes very unhappy if you try to set up a partition that's a subset of another partition. So we can't really use MBR. On to GPT.
GPT, or the GUID Partition Table, is the EFI era's replacement for MBR partitions. It has two main advantages over MBR - firstly it can cover partitions larger than 2TB without having to increase sector size, and secondly it doesn't have the primary/logical partition horror that still makes MBR more difficult than it has any right to be. The format is pretty simple - you have a header block 1 logical block into the media (so 512 bytes on a typical USB stick), and then a pointer to a list of partitions. There's then a secondary table one block from the end of the disk, which points at another list of partitions. Both blocks have multiple CRCs that guarantee that neither the header nor the partition list have been corrupted. It turns out to be a relatively straightforward modification of isohybrid to get it to look for a secondary EFI image and construct a GPT entry pointing at it. This works surprisingly well, and media prepared this way will boot EFI machines if burned to a CD or written to a USB stick.
There's a few quirks. Macs will show two boot icons for these CDs[6], one marked "EFI Boot" and one helpfully marked "Windows"[7], with the latter booting the BIOS El Torito image. That's a little irritating, but not insurmountable. The other issue is that older Macs won't look for boot loaders in the legacy locations. This is where things start getting horrible.
Back in the old days, Apple boot media used to have a special "blessed" folder. Attempting to boot would involve the firmware looking for such a folder and then using that to start itself up. Any folder in the filesystem could be blessed. Modern hardware doesn't use boot folders, but does use boot files. For an HFS+ filesystem, the inode of the bootloader is written to a specific offset in the filesystem superblock and the firmware simply finds that inode and executes it. And this appears to be all that older Macs support.
So, having written a small tool to bless an HFS+ partition, I tried the obvious first step of burning a CD with three El Torito images (one BIOS, one FAT, one HFS+). It failed. While
Refit could see the bootloader in the HFS+ image, the firmware appeared to have no interest at all in booting off it. Yet Apple install media would boot. What was the difference?
The difference, obviously, was that these earlier Macs don't appear to support El Torito booting. The Apple install media contained an Apple partition map.
The Apple partition map (APM) is Apple's legacy partition table format. Apple mostly dropped it when they went to x86, where it's retained for two purposes. The first is for drives that need to be shared between Intel Macs and PPC ones. The second seems to be for their install DVDs. Some further playing revealed that burning a CD with an APM entry pointing at the HFS+ filesystem on the CD gave me a boot icon. Problem solved?
Not really. Remember how I earlier mentioned that ISO 9660 leaves 32KB at the start of the image, and that an isohybrid image then writes an MBR and boot sector in the first 512 bytes of that, and the GPT header starts 512 bytes into a drive? That means that it's easy to produce an ISO that has both a boot sector, MBR partition table and GPT. None of them overlap. APM, on the other hand, has a header that's located at byte 0 of the media, overlapping with the boot sector. And it has a partition listing that's located at sector 1, overlapping with the GPT. Is all lost?
No. Merely sanity.
The first thing to remember is that the boot sector is just raw assembler. It's a byte stream that's executed by the CPU. And there's a lot of things you can tell a CPU to do that result in nothing happening. Peter Jones pointed out that the only bits of the AFP header you actually need are the letters "ER", followed by the sector size as a two byte big endian integer. These disassemble to harmless instructions, so we can simply move the boot code down a little and stick these at the beginning. A PC that executes it will read straight through the bizarre (but harmless) Apple bytes and then execute the real boot code.
The second thing that's important here is that we were just given the opportunity to specify the sector size. The GPT is only relevant when the image is written to a USB stick, so assumes a sector size of 512 bytes. So when the GPT starts one sector into the drive, it's actually starting 512 bytes into the drive. APM also starts one sector into the drive, but we can simply put a different sector size into the header and suddenly we're able to choose where that's going to be. 2K seems like a good choice, and so the firmware will now look for the header at byte 2048.
That's still in them middle of the GPT partition listing, though. Except we can avoid that as well. GPT lets you specify where the partition listing starts and doesn't require it to be immediately after the header. So we can offset the partition listing to, say, byte 8192 and leave a hole for the Apple partition map.
And, shockingly, this works. Setting up a CD this way gives a boot icon on old Macs. On new Macs, it gives three - one for legacy boot, one for EFI boot via FAT and one for EFI boot via HFS. Less than ideal, but eh. The one remaining problem is that this doesn't work for USB sticks (the firmware sees the GPT and ignores the APM), so we also need to add a GPT entry for the HFS+ partition. Job done.
So, it is possible to produce install media that will work if burned to CD or written to a USB stick. It's even possible to produce a version that will work on Macs, as long as you're willing to put up with three partition tables and an x86 boot sector that doubles as an APM header. And patches to isohybrid to do all of this will be turning up as soon as I tidy the code to the point where it works without having to hack in offsets by hand.
[1] Insert some other adverb here if you feel like it
[2] Why yes, 15 years later BIOSes still tend to assume 512 bytes. Which is why your 4K sector disk is much harder to work with than you'd like it to be.
[3] Ever noticed how the modern world involves a great deal of suffering, misery and death? EFI fits into that world perfectly.
[4] Obviously if you want your media to be bootable via both BIOS and EFI you need to produce a CD with two El Torito images. BIOS systems should ignore the image that says it's for EFI, and EFI systems should ignore the BIOS one. Some especially creative BIOS authors[5] have decided that users shouldn't have their choices limited in such a way, and so pop up a screen that says:
1.
2.
Select CD-ROM boot type:
and wait for the user to press a key. The lack of labels after the numbers is not a typographical error on my part.
[5] Older (pre-2009, and some 2009 models) Apple hardware has this bug if a dual-El Torito CD is booted via the BIOS compatibility layer. This is especially unfortunate because said machines often fail to provide a working keyboard emulation at this stage, resulting in you being stuck forever at an impressively unhelpful screen. This isn't a Linux bug, since it's happening before we've run any of our code at all. It's not even limited to Linux. 64-bit install media for Vista SP1, Windows 7 and Server 2008 all have similar El Torito layout and all trigger the same bug on Apple hardware. Apple's aware of this, and has resolved the issue by
declaring that these machines don't support 64 bit Windows.
[6] Even further investigation reveals that Apple will show you as many icons as there are El Torito images, which is a rare example of Apple giving the user the freedom to brutally butcher their extremities if they so desire
[7] "Windows" is Apple code for "Booting via BIOS compatibility". The Apple boot menu will call any filesystem with a BIOS boot sector Windows.
FreedomBox activities at Debconf11
I spent the last two weeks of July 2011 in Banja Luka. The occasion was the
annual Debian developer's conference,
Debconf11 and preceding work week known as
Debcamp. This was my tenth successive year attending Debconf, and I had a
very productive and pleasant time! The facilities were good, the local team
was friendly, enthusiastic, and very helpful, and in addition to giving three
talks and hosting a couple panel discussions, I managed to put a burst
of energy into work on FreedomBox. Several
other developers working on FreedomBox were also present, and a good number of
Sheeva
and
Dream
plugs were evident in the hacklabs sporting new FreedomBox stickers.
Working together in the same place for several days, we made good progress
on several projects, and also had some great discussions about what we want
to do going forward.
image building tools
For some time, I've been working towards a light-weight tool set to build
FreedomBox software images. Shortly before Debconf started, I chose the
name 'freedom-maker' for this tool and shared a link to a readable copy of my
git repository with other developers I expected to work with in Banja Luka.
With input from Bert Agaz and Jonas Smedegaard during Debcamp, freedom-maker
went from almost useful to actually useful. It still deserves work to be
more useful to others, but I have now pushed a copy of the git repository to
git.debian.org so that we can take advantage of the tools supported there to
enable others to more easily contribute to the code.
Very soon, Bert plans to add support to freedom-maker for using Lars Wirzenius'
vmdebootstrap to build x86 images
suitable for testing in a virtualized environment. At the same time, we plan
to refactor the existing code slightly
to enable lists of desired packages for the various image flavors we expect
to produce independently of the configuration for each specific image building
tool.
Jonas continued in parallel to work on his alternate packaging toolset
boxer.
It offers some potentially interesting features for the future, and we may
eventually merge some or all of it into freedom-maker, but for now it remains
a separate utility.
uAP user space tools
Several weeks ago, we received from Marvell the source code to two user space
programs that are necessary for configuring and monitoring the binary firmware
provided for the uAP wireless chip used in the DreamPlug. Early during my
stay in Banja Luka, I packaged these for Debian as
uaputl and
uapevent, and I am pleased to note
that they were quickly accepted into the archive and are now present in Debian
mirrors.
u-boot
Another bit of code received very shortly before Debconf started was the
source for the version of u-boot shipped by
Globalscale in the DreamPlug
units we're working with. During Debcamp, Clint Adams passed a copy of this
source to Jason Cooper, who was already trying to add support for the
DreamPlug to upstream u-boot, but had stalled due to a lack of
information. Jason has now merged his own work with the sources we got
from the manufacturer, and is
making good progress towards merging DreamPlug support into upstream u-boot.
Once that happens, we should be able to flash our Sheeva and Dream plug devices
with a u-boot image built from the source in the Debian u-boot package, in the
process enabling things that matter to us like the ability to boot from an
ext2 partition, and hopefully the ability to execute command scripts from that
partition instead of having to hard-code kernel filenames in flash. This will
allow us to support the ongoing effort in Debian to move away from the need
for kernel symlinks.
DreamPlug kernels
With respect to kernels, another work stream at Debconf primarily involving
H ctor Or n and Nick Bane was to analyze the current state of the patches from
Marvell and Globalscale used to support the DreamPlug against both upstream
and current Debian kernel sources. To my surprise and our collective pleasure,
the remaining patch set required against current upstream kernels is much
smaller than we previously believed! There are still several patches critical
to us that are not merged upstream, but the work remaining to be able to build
images for our devices from mainline and Debian kernel source trees now seems
like something we might be able to complete before Debian's next stable
release.
One of our discoveries during the u-boot and kernel work during Debconf was
that Globalscale did not obtain a new machine id for the DreamPlug, but
instead re-used the one for the GuruPlug series, despite there being some
differences in the hardware that require at least one additional driver. After
much discussion, we plan to continue using the existing machine id instead
of requesting another, particularly because the ARM kernel community
has apparently stopped issuing new ids for the moment. We will add a new
kernel config option for the DreamPlug, however, and are likely to build
distinct Sheeva and Dream kernel packages that do not require initrd for use
in FreedomBox images, even if doing so is not strictly necessary. This will
allow us to optimize both the in-memory footprint and boot times for our
devices.
software configuration
Another area of investigation in Banja Luka was technology for package
configuration. Mirsal Ennaime performed various tests using debconf and
Config::Model, with some results reflected in this commit
relating to configuring the bitcoind daemon in the bitcoin package for Debian.
identity and trust management
While we did not actually do any FreedomBox specific work on the trust
management layer we know is necessary, after several rounds of conversation,
I am now more convinced than ever that the right path forward is to base our
trust relationships on OpenPGP keys using GnuPG and Monkeysphere as starting
software elements. Our thinking to date is captured on an
Identity Management
page in the wiki.
communication services
Another thing that became fairly clear to me during discussions at Debconf
is that in the near term, planning to build communication services around
XMPP is the approach most likely to give good results. Investigating the
software choices available to build an interesting XMPP infrastructure is
now a high priority for me. Jonas has done some work towards
configuring and integrating ejabberd
or Prosody, I've started studying
yate as a possible call manager and VoIP
server choice with
XMPP/jingle support, and we await with great interest a release from the
Buddycloud developers to evaluate as
a possible basis for deploying social network services.
Some of these software choices will lead us to use Apache as our web services
base technology because of the need for features that only it supports well
among daemons that are Free Software.
Jonas completed packaging GNU Sip Witch
for Debian, and it is now available in the mirror network. Tzafrir Cohen and
Jonas did some initial testing on its use.
documentation
A number of new wiki pages were written
(or at least started) in order to sum up ideas, design various aspects of
FreedomBox, and reflect discussions that happened during DebConf11. A lot
of work is needed to complete these pages though, as well as others to capture
more of the current state of the project.
press coverage
Finally, while in Banja Luka I got some great press coverage for FreedomBox!
On Sunday the 24th, I was
interviewed
by the main television network serving the Republika Srpska. This led
to a couple of minutes of coverage near the top of the national news program
that night, immediately following the lead story about the President and
several ministers appearing at Debian Day that morning to help open the
conference. This interview was later re-used in
another TV program
that summarized Debconf11. On the morning of Thursday the 28th, I was part
of a small group that spent more than an hour meeting with the Minister of
Science and Technology in his office, and the relationship between Debian
and our work on FreedomBox was one of the items of discussion in that meeting
and the associated press conference. I'm told this resulted in more press
coverage, but if true I have not seen it yet.
summary
On Friday afternoon the 29th, I gave a talk in the main Debconf program containing a FreedomBox Progress Report . In it, I talked about the structure of the FreedomBox Foundation, progress the foundation has made, and the work that was still then underway in Banja Luka. It was streamed live over the internet, and replays are available online. The reaction from Debian developers present was very positive, which was good to learn since by that time my energy level was quite low after the nearly two weeks of intense technical and social interaction that is Debconf!
All in all, we got lots of work done on FreedomBox in Banja Luka, enough that
I think at least the next few steps along the road towards an eventual "1.0"
release of a reference implementation are now much clearer than they were two
weeks ago!
Together with 
Another year, another Debconf and now it is passed. Pictures are processes and all are now 
Boy, the last three months have been very eventful, developmentally.
First off, about 2 weeks after she turned 1, Zoe started walking in earnest.
Within about 2 weeks of that there was no stopping her. It's funny to think
she's only been walking for under 3 months and yet she's very stable on her
feet.
Secondly, she developed a mild egg allergy. Sarah was giving her some
scrambled eggs (one egg's worth) for lunch one day, and she mixed in some
cream cheese, and Zoe absolutely scoffed it all down, so she scrambled
another egg and she scoffed that down too. She then 
Due to the midterm evaluation, this report will cover a span of 4 weeks. As a result, it will be quite a bit longer than my previous reports.
One of the first things I did after my last report was create a FAQ for the project. At first, the FAQ was a plain text file living only in the git repository. Then, it moved to being maintained on the Debian wiki with a cronjob pulling it into my local repository for me to commit and push. Finally, the cronjob moved to wagner. The FAQ is not actually committed to the git repository, but it is available in the 
So, the fifth Debconf I ve had the chance to attend is clearly over now. It was great, and all organizers (and sponsors) really deserve huge thank you for making this event so successful. I m already looking forward to next year s edition.
Debconf has been very productive for me. I chaired 4 sessions:
The crowd of thousands gathered here to attend PreventaConf 2011 were abuzz over the unveiling of the culmination of 3 years of aimless arsing about^ww^w concentrated development which has led to the porting of dpkg to the 
Women fainted, men wept openly, medical personnel were kept busy dealing with the many instances of heart seizure, multisensory hallucination and spontaneous combustion which accompanied this momentous event.
Much remains to be done.
Due to high demand I have prepared official 
It is half-time for me at DebCamp/
I ve actually spent a few hours searching around for a good backup solution for my mailbox until I decided to stick with 
With 