I've just started a new job, so I decided to take a few days holiday in
between and work on something that's been on my todo list for way too long
now -- making the debian-installer work on the D-Link DNS-323.
After a couple of days of work, I've just finished off the install of
Debian on my DNS-323, which I initiated by loading a firmware image into
the web interface of the stock firmware, and then SSHing into the machine
at the appropriate moment. What this means is that
anyone with a
DNS-323 can install Debian on it. I think that's worthy of a small, quiet
"woo hoo!"...
Looking back now, it seems to have been a fairly simple and
straightforward operation, but I can remember that it didn't look so
simple when I started a week ago. So, in the interests of perhaps
short-cutting someone else's adventure into the world of d-i hacking, I'm
writing everything up while it's fresh in my mind (the benefits of being
on holidays -- plenty of time to write long blog posts). Alternately, if
anyone has a random device they'd like to see d-i on but doesn't have the
inclination to do it themselves, feel free to send me one and I'll do it
for you.
Before I begin, I've got a few "shout outs" (I think that's what the cool
kids are calling it these days) to people who've helped me along the way.
First off, Martin Michlmayr has answered innumerable dumb questions from
me, and I think wrote most of the parts of d-i that I've extended -- it's
fair to say that I'd have had no hope of doing this without him. Thanks
also to the people in
#debian-boot and
#debian-arm for
answering questions and putting up with my frequent bouts of
self-congratulation whenever I passed the most miniscule milestone.
Finally, everyone who has put work into d-i has my thanks, too -- it's not
perfect, but it's a damn sight better (both to use and also to hack on)
than any other installer I've ever met.
So, without further ado, off we go...
Work out how to get a console
Unless you are a guru of unbelievable skill, you will undoubtedly not do
everything quite perfectly the first time, and you'll want to see the
kernel/init boot messages, and maybe in some cases try some investigation
and diagnostic work. For that, you're going to have to have some sort of
console access.
If the device doesn't have a full screen/keyboard thing going on, the
chances are you'll almost certainly end up with a serial console. A few
devices have a network management thingo that will give you a console,
usually over VNC or some sort of horrid Java applet, but I've only ever
encountered that in rackmount servers, which have a keyboard/video
interface anyway...
In theory, I suppose, the kernel could provide a network console, but I've
never heard of such a thing being attempted, and Google is not helping me
out. I might add that to the todo list as a future hacking project...
On the DNS-323, it was a serial console that was needed. Like a lot of the
embedded devices I've seen, it didn't
quite talk RS-232, so I needed an
adapter (I got a TTL-232R-3V3 USB-to-3.3v-serial adapter from
a
local electronics supply place, which worked perfectly with Linux as
soon as I plugged it in). I needed to solder a few wires to a connecter
inside, and connect that up to the USB doohickey, but it all came good in
the end. The chances are someone's done a similar job for your device of
choice, so a search for things like "devicename serial console" will almost
certainly pay off.
Work out how to load and boot a kernel/initrd
Your development work will involve booting weird kernels and initial
ramdisks to make everything work; you need some way of stuffing those into
the device and booting from them. Everything has a boot loader of one
sort or another; working out what you've got and how to make it sing and
dance is the ultimate quest here.
How this is done will vary greatly from device to device; the WRT-54GL,
for example, waits for a tftp connection for a couple of seconds on
startup, which you can use to send a new firmware; many other devices have
a boot loader that you have to explicitly control (via the console), but
it will make a tftp connection out to another machine if you ask it to.
Some devices you just need to load the data in via serial cable (at
115200bps it's bearable; at the 9600bps I've heard some devices work at,
it would be
tres painful).
The DNS-323 runs uBoot as the boot loader, which will do a tftp request by
default. However, D-link, or Marvell, or
someone in the
decision-making chain for this thing decided to knobble it and removed most
of the useful commands, so sending the thing via serial port is the only
option. Since writing to flash takes a pile of extra time, I found
(eventually) that you can just load the data into RAM somewhere and boot
from there, without needing to write it to flash (also handy to avoid
wasting precious write cycles), since you can tell uBoot to boot a kernel
located anywhere in the address space. Obviously you need to reload the
kernel/initrd on every boot if you don't write it to flash, so if you've
got a "stable" kernel or initrd that you think you'll be using for a while
(when you're testing the other item) you should flash it then.
One note on uBoot, that took me far too long to work out -- you can't just
stuff a kernel fresh off the Makefile in and assume it'll work -- uBoot
needs a special header prepended that tells it all about what you're
loading (architecture, file type, compression method, etc). You "wrap" a
file in this uBoot header with the "
mkimage" command (in the
uboot-mkimage package in Debian).
Incidentally, if you need to remove the uBoot bits from a file, just strip
the first 64 bytes off -- that's all the uBoot header is, so something
like this will do nicely:
dd if=file.uboot of=file bs=64 skip=1
A final uBoot warning: it's fairly important to specify a "load address"
for the files; if you get them wrong, your kernel and initrd may or may
not boot (I never quite got the hang of why some wrong values worked and
some didn't). If your device uses uBoot, and you have the ability to
extract the uBoot-enabled kernel and initrd from a vendor-provided
firmware image, you can use
mkimage -l to inspect what all the
values need to be for your own kernels and initrds. Just stick with them.
Does Debian support your hardware?
Obviously, if you want to run Debian on something, Debian has to support
that hardware. The easiest way to test this out is to try an install.
For this, you'll need to
retrieve an appropriate kernel and initrd pair for the installer.
Some arches, like i386 and amd64, only have one kernel and initrd for the
installer, so you just have to choose if you're going to netboot, or do a
full CD install, or whatever. Other arches, though, like arm, have subtly
different images for different machines. There are two reasons they vary:
firstly, they often need to split into
subarches, where they all share
a general CPU architecture but there are enough differences in the
hardware that they're not all supportable by the same kernel. At a more
fine-grained level some different machines of the same sub-architecture
need differently formatted firmware images to boot. On ARM, in
particular, it's pretty common to need to prepend a "device ID" to the
kernel image so that the kernel knows what sort of machine it's booting
on, which means that every machine typically needs it's own kernel image,
regardless of any other issues with firmware formats.
Choosing the sub-architecture
should be relatively
straightforward, if you know much about the machine you're targetting.
Picking a machine type can be a bit harder, but again, if you know what
your boot loader needs, it shouldn't be too hard to find a kernel/initrd
from the existing machines in your subarch that you can mangle to fit.
Load up your kernel/initrd into the device, boot it, and see how far it
gets. You might get lucky, and the kernel inits, your hard drives are
detected, your NIC is initialised, and you're off and running without
further ado. Or you might not be so lucky.
The worst case is that the kernel doesn't even get as far as loading the
initrd -- it might not print anything at all, or the kernel might hang part
way through the boot. This is likely to require a fair amount of
debugging fun, as initial boot failures get a bit tricky.
A better situation is where the kernel boots OK, and d-i starts, but
something in the install process falls over. This can either be a
kernel/driver problem, or a d-i problem. Either way, since d-i has
started, you're likely to be able to get a shell and do some debugging.
In order to make d-i work, you need to have somewhere to install to
(disks, usually) and a source of packages to install from (CD/DVD if
you're on a suitably large device, or the network otherwise). These are
the two key ingredients you need; most everything else is stuff that
ranges between "nice to have" and "critical" for operational purposes, but
which has absolutely zero impact on your ability to do the install, so
we'll ignore it for now.
For the DNS-323, I had a few kernel problems (which I'd been warned about
by Martin before I started) -- the NIC wasn't initialised with it's MAC
address, which meant it was running around my Ethernet screaming "I am
00:00:00:00:00:00! Ph33r m3!", and the SATA controller just didn't seem
to exist at all. These were kinda critical to doing an install, so it was
time to get hacking to make those work.
There were also a couple of parts of d-i that needed tweaking -- the
preseeding code didn't support the device, and I needed to package
a tool to build a "complete" uploadable firmware image and teach d-i how
to use it. Apart from that, though, things were looking pretty good.
Make the Debian kernel support your hardware
Once you've worked out what hardware you need to make work, you need to
dig into the kernel and fix it. Sometimes, if you're really lucky, a
kernel version newer than the one currently in Debian has better support
for something, and you can backport the fix / feature improvement. Other
times, there's something similar already in the kernel for a different
device and you can crib off that (I did this extensively for my little
fixes). Then there's the "ask an expert" school, where you find someone
who knows what they're on about and pick their brain (I did a bit of that,
too). Finally, there's plain old "research and development", for which
there are piles of documentation already out there.
My changes for the DNS-323 came to 121 added lines and 16 deleted lines,
most of which was cribbed-and-tweaked from other parts of the kernel. I
might have written 10-20 lines of code "from scratch". It's not always
scary-hard.
Cross-compiling your own kernel
One thing that I think needs mentioning in the kernel hacking department
is cross-compiling a modified kernel for testing purposes. Chances are
your main workstation has a lot more grunt (and build tools) than whatever
you're trying to get Debian onto, so cross-compilation is the way to go.
You need to first grab a copy of the source you want to build (from
apt-get source linux-image-blahdiblah, or a release tarball, or
Linus' git repo, it doesn't really matter) and a cross-compilation
toolchain. For my cross-compilation toolchain, I just added this to my
sources.list:
deb http://www.emdebian.org/debian/ unstable main
then I ran
apt-get update, and installed these packages:
binutils-arm-linux-gnueabi gcc-4.1-arm-linux-gnueabi
gcc-4.3-arm-linux-gnueabi-base libc6-dev-armel-cross
Kudos to the emdebian team for making that such an utterly painless
exercise, by the way -- the days of needing to faff round with gcc sources
to get a cross-compilation toolchain are apparently over.
Once you've got a source tree and a cross-compiler available, you can run
any of the usual kernel make commands (like
make zImage or
make
oldconfig), as long as you've got a couple of make variables defined.
In my case, I used:
ARCH=arm CROSS_COMPILE=arm-linux-gnueabi-
ARCH tells the kernel what architecture to build for (useful
in all make commands because not all arches have the same config options in
make menuconfig, for example) and
CROSS_COMPILE tells the
build system what to prefix the calls to
gcc,
ld and such
with (if you installed the same cross-compiler packages as I did, you'll
find it installs files like
/usr/bin/arm-linux-gnueabi-gcc, so you
can
ls /usr/bin/*gcc to find out what you'll need for the
CROSS_COMPILE setting).
Hack d-i to make things work
In addition to tweaking the kernel, it's quite possible that the internals
of d-i might need some tuning. It's possible that d-i might need to be
"taught" to recognise the hardware in your device, or you'll need to
extend some aspect of the installation process to complete the
installation.
In order to hack d-i, you need to know a bit about how it works and how it
is structured, so let's run through that first.
The entireity of d-i is based around udebs, which are packages of the same
format as "real" debs, but which have been built differently (against a
small libc, for instance) and have had everything that isn't absolutely
necessary (docs, non-essential scripts, extra functionality, etc) stripped
out of them to create the smallest possible packages.
There are a variety of different udebs -- most are derived from otherwise
normal, law-abiding citizens of the archive, like busybox (basic unix
utilities like cp and ls), the shell that's used in the installer, parted,
various libraries, and so on. Others are "special", and have no big
brother in the archive. These are mostly "task" udebs, that define what
actually happens in the installer. When they're "installed", they have
special fields in the package metadata that define where in the install
sequence they go, and they end up on the menu you see if your installation
goes a bit pear-shaped.
The tricky thing about udebs is that, as small as they are, there's too
many of them to be able to have them all available when the system first
boots, so even the "setup" of the installation gets done in several stages.
First, the kernel and initrd boot. The initrd contains
just
enough stuff to get access to the rest of the udebs, and nothing else.
Even this early stage is built from udebs -- there's lists of what goes
into the initrd for various architectures and flavours of installer under
installer/build/pkg-list in the
d-i SVN. If you
need an extra udeb in your initrd, that's the place to specify it.
Once the initrd has gotten access to a source of packages (off a CD, over
the network, telepathy, whatever) it can drag the rest of the packages it
needs from there. Space is still an issue at this point, though --
there's no diskspace yet to put anything onto, so everything needs to fit
into a ramdisk. Don't go loading anything you don't need.
The packages loaded at this stage are what defines most of the steps that
will be done for the rest of the install, so once all those extra udebs
are loaded, the remainder of the install kicks off and does it's thing.
Different architectures and install methods have different requirements,
too, which complicates matters. For example, when you're installing on a
regular PC with screen/keyboard, you can guide the install using that.
However, if you've got no screen, how can you control the install?
There's serial consoles, but requiring everyone to void the warranty
(more) on their little embedded doodad by installing a serial console
cable to install Debian isn't so hot. So, lacking a terminal, how do you
control the install?
Well, on install flavours for devices that might be in this situation, you
can SSH into the installer and complete it that way. That feature is,
like everything else, provided in a udeb -- called
network-console,
as it happens. But what IP address do you connect to, and what network
settings does the device use? Again, without a console you can't
configure your network, and not everyone has a DHCP server (and even fewer
people have a DHCP server that will hand out an address that they'll know
in advance, or be able to find out). So, there's a udeb (called
oldsys-preseed) that does it's best to configure your network based on the
settings already in the nvram for your device.
In short, there's a lot of tricky, well-thought-out shit going on inside
d-i, and you might have to deal with a little bit of it. The nice thing
is that by now I expect most of the really hard problems have been solved,
so with a bit of luck you'll only ever have to do minor tweaking.
In general, all of the packages that are "d-i specific" live in
packages in the d-i SVN. If you need to modify a udeb, you need
to rebuild it and put it in
installer/build/localudebs/ (creating
it first if need be) and every time you change it, you need to delete
installer/build/apt.udeb/cache/.
To give a concrete example (from the ever-present DNS-323), I had to
modify oldsys-preseed so that it would get the details from the nvram that
the stock firmware uses to configure itself. This, in turn, required that
the minix-modules udeb be included in the initrd, because the DNS-323's
nvram is actually a small minix partition stored in flash. Painfully,
this required some more hacking, because the armel d-i kernel wasn't
configured to generate minix-modules. That might need some explanation...
Since building the kernel is a fairly massive undertaking, CPU-wise,
instead of building all the udebs from scratch there's a package called
kernel-wedge to split a real linux-image package into lots of little
udebs. There's a separate source package for each architecture,
linux-kernel-di-$ARCH-2.6, that contains the configuration data
for kernel-wedge to specify what udebs need to be built, and what goes
into each of them. A small tweak to the armel package and voila! I had a
minix-modules udeb to include in the initrd. With the config stuff I'd
done in the d-i build itself and my changes to oldsys-preseed, we were
done with d-i.
Tell d-i how to build your bootable artifacts
If your device needs a "different" install image from other similar
devices, you need to write some makefile rules to build those files. Look
under the
installer/build/config/$ARCH/ directory to see the various
rules files available.
I had to package a new tool to create uploadable firmware images for the
DNS-323, along with adding a rule to use that tool to build images. None
of it was particularly difficult.
Polish the diamond
Getting d-i working is only part of the battle; it's quite likely that
there will be other hardware in the machine that needs to be supported.
This could involve kernel-level stuff to interface to the hardware, or
userland tools to manage the hardware.
There's several bits in the DNS-323 that still need work; I can't control
the LEDs on the front panel, nor the fan, and I can't use the on-board
temperature sensors yet. But, I
do have Debian running on the
thing, which is what I was shooting for.
Examples
If you're interested in exactly what I did, there are a number of bugs in
the BTS that contain the patches I made:
-
#502821: oldsys-preseed
changes to take existing network config from NVRAM;
-
#502936: build a minix
modules package for the armel d-i kernel;
-
#503040: d-i core changes to
build firmware images for the DNS-323;
-
-
#503172: kernel patches to
support the SATA controller and NIC better.
Following Ganneff’s post to debian-devel-announce, several discussions have again started on the topic of Debian’s membership and several proposals have been made. Unfortunately none of these proposals try to resolve the underlying trust problem that has been growing over the years. Despite the NM process (or maybe due to it), we managed to give DD status to people who are motivated but whose technical skills are doubtful (at that point people ask for an example, and as much as I hate fingerpointing, here’s an example with 
now that ps3-utils is 
Third in a series (
Since it's apparently 
For a minute, I’d like to think about 
On the morning after the final US presidential debate that happened a
week ago, I was invited onto the excellent new 
The Linux desktop currently receives (certain) key events in two ways. Firstly, the currently active session will receive keycodes via X. Secondly, a subset of input events will be picked up by hal and sent via dbus. This information is available to all sessions. Which method you use to obtain input will depend on what you want to do:
, and partly
because as many geeks I've always had strong opinions on