tl;dr: Get vmdebootstrap replacement from
http://git.liw.fi/vmdb2 and
run it from the source tree. Tell me if something doesn't work. Send
patches.
Many years ago I wrote
vmdebootstrap, a tool for installing Debian
on a disk image for virtual machines. I had a clear personal need: I
was setting up a CI system and it needed six workers: one each for
Debian oldstable, stable, and unstable, on two architectures (i386,
amd64). Installing Debian six times in the same way is a lot of work,
so I figured how difficult can it be to automate it. Turns out that
not difficult at all, except to install a bootloader.
(Don't ask me why I didn't use any of the other tools for this. It was
long ago, and while some of the tools that now exist probably did
exist then, I like writing code and learning things while doing it.)
After a while I was happy with what the program did, but didn't want
to upload it to Debian, and didn't want to add the kinds of things
other people wanted, so I turned vmdebootstrap over to Neil Williams,
who added a ton of new features. Unfortunately, it turned out that my
initial architecture was not scaleable, and also the code I wrote
wasn't very good, and there weren't any tests.
Neil did heroic work forcing my crappy software into doing things I never
envisioned. Last year he needed a break and asked me to take
vmdebootstrap back. I did, and have been hiding from the public eye
ever since, since I was so ashamed of the code. (I created a new
identity and pretended to be an international assassin and backup
specialist, travelling the world forcing people to have at least one
tested backup of their system. If you've noticed reports in the press
about people reporting near-death experiences while holding a shiny
new USB drive, that would've been my fault.)
Pop quiz: if you have a program with ten boolean options ("do this,
except if that option is given, do the other thing"), how many black
box tests do you need to test all the functionality? If one run of the
program takes half an hour, how long will a full test suite run?
I did some hard thinking about vmdebootstrap, and came to the sad
conclusion that it had reached the end of its useful life as a living
software project. There was no reasonable way to add most of the
additional functionality people were asking for, and even maintaining
the current code was too tedious a task to consider seriously. It was
time to make a clean break of the past and start over, without caring
about backwards compatibility. After all, the old code wasn't going
anywhere so anyone who needed it could still use it. There was no need
to burden a new program with my past mistakes. All new mistakes were
called for.
At the Cambridge mini-Debconf of November, 2016, I gave a short
presentation of what I was going to do. I also posted about my
plans to the debian-cloud list. In short, I would write a new, more
flexible and cleaner replacement to be called vmdb2. For various
personal reasons, I've not been able to spend as much time on vmdb2
as I'd like to, but I've now reached the point where I'd like to
announce the first alpha version publically.
The source code is hosted here: http://git.liw.fi/vmdb2 . There are
.deb packages at my personal public APT repo (http://liw.fi/code/),
but vmdb2 is easy enough to run directly from a git checkout:
sudo ./vmdb2 foo.vmdb --output foo.img
There's no need to install it to try it.
What works:
- vmdb2 can build a disk image with Debian installed, for amd64 only
at this time
- the boot loader is GRUB, either for UEFI or BIOS
- the image boots under Qemu / KVM and also on actual hardware
What doesn't work:
- other architecures, including building for a foreign archiecture
- live CD building (no squashfs support)
I'm not opposed to adding support for those, but they're not directly
interesting to me. For example, I only have amd64 machines. The best
way to get support for additional features is to tell me how,
preferably in the form of patches. (If I have to read tons of docs, or
other people's code, and then write code and iterate while other
people tell me it doesn't work, it's probably not happening.)
Why would you be interested in vmdb2? There's a lot of other tools to
do what it does, so perhaps you shouldn't care. That's fine. I like
writing tools for myself. But if this kind of tool is of interest to
you, please do have a look.
A short tutorial: vmdb2 wants you to give it a "specification file"
(conventionally suffixed .vmdb, because someone stole the .spec
suffix, but vmdb2 doesn't care about the name). Below is an example.
vmdb2 image specification files are in YAML, since I like YAML, and
specify a sequence of steps to take to build the image. Each step is a
tiny piece of self-contained functionality provided by a plugin.
steps:
- mkimg: " output "
size: 4G
- mklabel: msdos
device: " output "
- mkpart: primary
device: " output "
start: 0%
end: 100%
part-tag: root-part
The above create an image (name is specified with the --output
option), which is four gigabytes in size, and create a partitition
table and a single partition that fills the whole disk. The "tag" is
given so that later steps can easily refer to the partition.
If you prefer another way to partition the disk, you can achieve that
by adding more "mkpart" steps. For example, for UEFI you'll want to
have an EFI partition.
- mkfs: ext4
partition: root-part
- mount: root-part
fs-tag: root-fs
The above formats the partition with the ext4 filesystem, and then
mounts it. The mount point will be a temporary directory created by vmdb2,
and a tag is again given to the mount point so it can be referred to.
- unpack-rootfs: root-fs
The above unpacks a tar archive to put content into the filesystem, if
the tar archive exists. The tar archive is specified with the
--rootfs-tarball command line option.
- debootstrap: stretch
mirror: http://http.debian.net/debian
target: root-fs
unless: rootfs_unpacked
- apt: linux-image-amd64
fs-tag: root-fs
unless: rootfs_unpacked
- cache-rootfs: root-fs
unless: rootfs_unpacked
The above will run debootstrap and install a kernel into the
filesystem, but skip doing that if the rootfs tarball was used. Also,
the tarball is created if it didn't exist. This way the tarball is
used by all but the first run, which saves a bit of time. On my laptop
and with a local mirror, debootstrap and kernel installation takes on the
order of nine minutes (500 to 600 seconds), whereas unpacking the tar
archive is a bit faster (takes around 30 seconds). When iterating over
things other than debootstrap, this speeds things up something
wonderful, and seems worth the complexity.
The "unless:" mechanism is generic. All the steps share some state,
and the unpack-rootfs step sets the "rootfs_unpacked" flag in the
shared state. The "unless:" field tells vmdb2 to check for the flag
and if it is not set, or if it is set to false ("unless it is set to
true"), vmdb2 will execute the step. vmdb2 may get more such flags in
the future, if there's need.
- chroot: root-fs
shell:
sed -i '/^root:[^:]*:/s//root::/' /etc/passwd
echo pc-vmdb2 > /etc/hostname
The above executes a couple of shell commands in a chroot of the root
filesystem we've just created. In this case they remove a login
password from root, and set the hostname. This is a replacement of the
vmdebootstrap "customize" script, but it can be inserted anywhere into
the sequence of steps. There's boot chroot and non-chroot variants of
the step.
This is a good point to mention that writing customize scripts gets
quite repetitive and tedious after a while, so vmdb2 has a plugin to
run Ansible instead. You can customize your image with that instead,
while the image is being built and not have to wait until you boot the
image and running Ansible over ssh.
- grub: bios
root-fs: root-fs
root-part: root-part
device: " output "
console: serial
Finally, install a boot loader, grub. This shows the BIOS variant,
UEFI is also supported. This also configures grub and the kernel to use a
serial console. There's a "yarn" (test suite) to build and smoke test
an image with vmdb2 to make sure at least the basic functionality
works. The smoke test boots the image under Qemu, logs in as root, and
tells the VM to power off. Very, very basic, but has already found
actual bugs in vmdb2. The smoke test needs the serial console to work.
As with vmdebootstrap originally, I don't particularly want to
maintain the package in Debian. I've added Debian packaging (so that I
can install it on my own machines), but I already have enough packages
to maintain, so I'm hoping someone else will volunteer to take on the
Debian maintainership and bug handling duties.
If you would like vmdb2 to do more things to suit you better, I'm
happy to explain how to write plugins to provide more types of steps.
If you are currently using vmdebootstrap, either directly or as part
of another tool, I encourage you to have a look at vmdb2. In the long
term, I would like to retire vmdebootstrap entirely, once vmdb2 can do
everything vmdebootstrap can do, and few people use vmdebootstrap.
This may take a while.
In any case, whether you want a new image building tool or not, happy
hacking.
As many of you may be aware, I work with Lars Wirzenius on a project
we call Subplot which is a tool for writing documentation which helps
all stakeholders involved with a proejct to understand how the project meets
its requirements.
At the start of February we had FOSDEM which was once again online, and
I decided to give a talk in the Safety and open source devroom to
introduce the concepts of safety argumentation and to bring some attention
to how I feel that Subplot could be used in that arena. You can view the
talk on the FOSDEM website at some point in the future when they
manage to finish transcoding all the amazing talks from the weekend, or if
you are more impatient, on Youtube, whichever you prefer.
If, after watching the talk, or indeed just reading about Subplot on our website,
you are interested in learning more about Subplot, or talking with us about
how it might fit into your development flow, then you can find Lars and
myself in the Subplot Matrix Room or else on any number of IRC networks
where I hang around as 
