I spent most of last week working on
Ubuntu bug 693671 ("wubi install will not boot - phase 2 stops with: Try (hd0,0): NTFS5"), which was quite a challenge to debug since it involved digging into parts of the Wubi boot process I'd never really touched before. Since I don't think much of this is very well-documented, I'd like to spend a bit of time explaining what was involved, in the hope that it will help other developers in the future.
Wubi is a system for installing Ubuntu into a file in a Windows filesystem, so that it doesn't require separate partitions and can be uninstalled like any other Windows application. The purpose of this is to make it easy for Windows users to try out Ubuntu without the need to worry about repartitioning, before they commit to a full installation. Wubi started out as an external project, and initially patched the installer on the fly to do all the rather unconventional things it needed to do; we integrated it into Ubuntu 8.04 LTS, which involved turning these patches into proper installer facilities that could be accessed using preseeding, so that Wubi only needs to handle the Windows user interface and other Windows-specific tasks.
Anyone familiar with a GNU/Linux system's boot process will immediately see that this isn't as simple as it sounds. Of course,
ntfs-3g is a pretty solid piece of software so we can handle the Windows filesystem without too much trouble, and loopback mounts are well-understood so we can just have the initramfs loop-mount the root filesystem. Where are you going to get the kernel and initramfs from, though? Well, we used to copy them out to the NTFS filesystem so that GRUB could read them, but this was overly complicated and error-prone. When we switched to GRUB 2, we could instead use its built-in loopback facilities, and we were able to simplify this. So all was more or less well, except for the elephant in the room. How are you going to load GRUB?
In a Wubi installation, NTLDR (or BOOTMGR in Windows Vista and newer) still owns the boot process. Ubuntu is added as a boot menu option using BCDEdit. You might then think that you can just have the Windows boot loader chain-load GRUB. Unfortunately, NTLDR only loads 16 sectors - 8192 bytes - from disk. GRUB won't fit in that: the smallest core.img you can generate at the moment is over 18 kilobytes. Thus, you need something that is small enough to be loaded by NTLDR, but that is intelligent enough to understand NTFS to the point where it can find a particular file in the root directory of a filesystem, load boot loader code from it, and jump to that. The answer for this was
GRUB4DOS. Most of GRUB4DOS is based on GRUB Legacy, which is not of much interest to us any more, but it includes an assembly-language program called GRLDR that supports doing this very thing for FAT, NTFS, and ext2. In Wubi, we build GRLDR as
wubildr.mbr, and build a specially-configured GRUB core image as
wubildr.
Now, the messages shown in the bug report suggested a failure either within GRLDR or very early in GRUB. The first thing I did was to remember that GRLDR has been integrated into the grub-extras
ntldr-img module suitable for use with GRUB 2, so I tried building
wubildr.mbr from that; no change, but this gave me a modern baseline to work on. OK; now to try QEMU (you can use tricks like
qemu -hda /dev/sda if you're very careful not to do anything that might involve writing to the host filesystem from within the guest, such as recursively booting your host OS ... [update: Tollef Fog Heen and Zygmunt Krynicki both point out that you can use the
-snapshot option to make this safer]). No go; it hung somewhere in the middle of NTLDR. Still, I could at least insert debug statements, copy the built
wubildr.mbr over to my test machine, and reboot for each test, although it would be slow and tedious. Couldn't I?
Well, yes, I mostly could, but that 8192-byte limit came back to bite me, along with an internal 2048-byte limit that GRLDR allocates for its NTFS bootstrap code. There were only a few spare bytes. Something like this would more or less fit, to print a single mark character at various points so that I could see how far it was getting:
pushal
xorw %bx, %bx /* video page 0 */
movw $0x0e4d, %ax /* print 'M' */
int $0x10
popal
In a few places, if I removed some code I didn't need on my test machine (say, CHS compatibility), I could even fit in cheap and nasty code to print a single register in hex (as long as you didn't mind 'A' to 'F' actually being ':' to '?' in ASCII; and note that this is real-mode code, so the loop counter is
%cx not
%ecx):
/* print %edx in dumbed-down hex */
pushal
xorw %bx, %bx
movb $0xe, %ah
movw $8, %cx
1:
roll $4, %edx
movb %dl, %al
andb $0xf, %al
int $0x10
loop 1b
popal
After a considerable amount of work tracking down problems by bisection like this, I also observed that GRLDR's NTFS code bears quite a bit of resemblance in its logical flow to GRUB 2's NTFS module, and indeed the same person wrote much of both. Since I knew that the latter worked, I could use it to relieve my brain of trying to understand assembly code logic directly, and could compare the two to look for discrepancies. I did find a few of these, and corrected a simple one. Testing at this point suggested that the boot process was getting as far as GRUB but still wasn't printing anything. I removed some Ubuntu patches which quieten down GRUB's startup: still nothing - so I switched my attentions to
grub-core/kern/i386/pc/startup.S, which contains the first code executed from GRUB's core image. Code before the first call to
real_to_prot (which switches the processor into protected mode) succeeded, while code after that point failed. Even more mysteriously, code added to
real_to_prot before the actual switch to protected mode failed too. Now I was clearly getting somewhere interesting, but what was going on? What I really wanted was to be able to single-step, or at least see what was at the memory location it was supposed to be jumping to.
Around this point I was venting on IRC, and somebody asked if it was reproducible in QEMU. Although I'd tried that already, I went back and tried again. Ubuntu's
qemu is actually built from qemu-kvm, and if I used
qemu -no-kvm then it worked much better. Excellent! Now I could use GDB:
(gdb) target remote qemu -gdb stdio -no-kvm -hda /dev/sda
This let me run until the point when NTLDR was about to hand over control, then interrupt and set a breakpoint at
0x8200 (the entry point of
startup.S). This revealed that the address that should have been
real_to_prot was in fact garbage. I set a breakpoint at
0x7c00 (GRLDR's entry point) and stepped all the way through to ensure it was doing the right thing. In the process it was helpful to know that
GDB and QEMU don't handle real mode very well between them. Useful tricks here were:
- Use
set architecture i8086 before disassembling real-mode code (and set architecture i386 to switch back).
- GDB prints addresses relative to the current segment base, but if you want to enter an address then you need to calculate a linear address yourself. For example, breakpoints must be set at
(CS << 4) + IP, rather than just at IP.
Single-stepping showed that GRLDR was loading the entirety of
wubildr correctly and jumping to it. The first instruction it jumped to wasn't in
startup.S, though, and then I remembered that we prefix the core image with
grub-core/boot/i386/pc/lnxboot.S. Stepping through this required a clear head since it copies itself around and changes segment registers a few times. The interesting part was at
real_code_2, where it copies a sector of the kernel to the target load address, and then checks a known offset to find out whether the "kernel" is in fact GRUB rather than a Linux kernel. I checked that offset by hand, and there was the smoking gun. GRUB recently acquired Reed-Solomon error correction on its core image, to allow it to recover from other software writing over sectors in the boot track. This moved the magic number
lnxboot.S was checking somewhat further into the core image, after the first sector.
lnxboot.S couldn't find it because it hadn't copied it yet! A bit of
adjustment and all was well again.
The lesson for me from all of this has been to try hard to get an interactive debugger working. Really hard. It's worth quite a bit of up-front effort if it saves you from killing neurons stepping through pages of code by hand. I think the real-mode debugging tricks I picked up should be useful for working on GRUB in the future.
Since I resumed active status in Debian, I ve been thinking about how to bridge the gap between Debian and its derivatives*. I ve spoken at length with Zack, the attendees of the Derivatives BoF at DebConf 10, and the fine folks at the Derivatives Front Desk about the technical and social issues affecting derivative projects, and could probably write a very thorough series of blog posts on the subject.
Instead, Zack and I decided to try doing something about it: we have begun a project to test out a new approach to the problem.
Introducing DEX
DEX is all about action: merging patches, fixing bugs, crunching data, whatever is necessary to get changes from derivatives into Debian proper. DEX doesn t try to change the way any existing project works, but adds a fast path for getting code from one place to another.
DEX is a joint task force where developers from Debian and its derivatives work together on this common goal. As a pilot project, we ve established an Ubuntu DEX Team focused on merging code from Ubuntu into Debian. With members from both projects, we hope to be able to resolve blockage anywhere in the pipeline. Whatever needs to get done in order to merge an Ubuntu patch, someone in the Ubuntu DEX team will know what to do. If we get good results with Ubuntu, we hope that other derivatives will follow. With thanks to David Paleino, we re excited that the Utnubu project is merging into DEX as it aligns well with their goals. I m very grateful to have Colin Watson and James Westby signed up to contribute as well.
Our first project is simple: turn this list green. This is an archive of quite old patches from Ubuntu, most of which have probably been merged already or made obsolete, but they pre-date any kind of tracking system so they need to be verified. Once that s done, we ll move on to a new project with a new todo list.
If you want to see Debian benefit from technical work done in derivatives, DEX is a chance for you to act together to make it happen. If you work on a derivative and want to carry a smaller delta, come and join us. I m sure we ll learn a lot from this experience.
* There are many instances of great cooperation between Debian and derivative distributions, including joint package maintenance teams, and some derivatives are even part of the Debian project. Nonetheless, there are areas were most people I ve spoken to agree that we need to do better. This is what I ve referred to as the gap .
Rhonda suggested I document how to use the LLS (Launchpad Login
Service their implementation of an OpenID provider) as Delegate,
which basically means, you can put something up on your webpage,
which can be a simple static (X)HTML page like mine (a /index.htm
is especially nice, a /~user/index.htm works too),
and use its URI and not https://launchpad.net/~me
to login. For example, this often hides the LLS from view e.g. in
blog comments, such as those where Canonical is being criticised
but it s also yours, easier to type and to
change if you switch service providers.
The basic idea is to go to your Launchpad user page and view its
page source. Look for openid relation links in the header on 
I spent part of today looking at packaging 
Last weekend we've released 
You know it already, since Debian 6.0 non-free firmware are no longer provided by a standard Debian installation. This will cause some troubles to users who need them. I m thus going to do a small overview on the topic and teach you what you need to know to deal with the problem.
What are firmware and how are they used?
From the user s point of view, a firmware is just some data that is needed by some piece of hardware in order to function properly. The driver for that hardware typically loads the firmware on the device as part of its initialization.
In the Linux kernel, the drivers are all using a standardized interface (request_firmware) to retrieve the firmware before sending it to the device. Thanks to this standardization, it s possible to either embed the firmware in the kernel or to have it request the firmware from user-space when needed.
Debian (like most distributions) has selected the latter option. Thus when the kernel needs a firmware, it sends out a request to user-space. udev is getting the request with the name of the firmware, and in its default configuration (see 
update-initramfs will also generate a similar warning on the terminal:
I've changed jobs recently and after 5 years of not having to work with a Windows system I am having all sorts of adaptation-to-Windows problems at the new job.
For those who are interested in the world of compiz, new development
snapshots have been trickling their way into experimental for the past
week and are now available for widespread testing.
They're being sent to experimental instead of straight to unstable for a
few reasons:
Today was my first day as the new 
