Jonathan McDowell: Building a read-only Debian root setup: Part 2
This is the second part of how I build a read-only root setup for my router. You might want to read part 1 first, which covers the initial boot and general overview of how I tie the pieces together. This post will describe how I build the squashfs image that forms the main filesystem.
Most of the build is driven from a script, make-router, which I ll dissect below. It s highly tailored to my needs, and this is a fairly lengthy post, but hopefully the steps I describe prove useful to anyone trying to do something similar.
Breakdown of make-router
It s a bash script, and I allow building for either my RB3011 or RB5009, which means a different architecture (32 vs 64 bit). I run this script on my Pi 4 which means I don t have to mess about with QemuUserEmulation.
I build the image in a loopback ext4 file on tmpfs (my Pi4 is the 8G model), which makes things a bit faster.
Create a dpkg excludes config to drop docs, man pages and most locales before we even start the bootstrap.
Setup fstab, the hostname and SSH keys for root.
Actually do the
I haven t uploaded mqtt-arp to Debian, so I install a locally built package, and ensure it starts after mosquitto (the MQTT broker), given they re running on the same host.
watchdog timeouts were particularly an issue on the RB3011, where the default timeout didn t give enough time to reach multiuser mode before it would reset the router. Not helpful, so alter the config to start it earlier (and make sure it s configured to actually kick the device file).
Clean up any docs etc that ended up installed.
The only login method is ssh key to the root account though I suppose this allows for someone to execute a privilege escalation from a daemon user so I should probably randomise this. Does need to be known though so it s possible to login via the serial console for debugging.
Pull in any security updates, then clean out the APT lists rather than polluting the image with them.
None of these make sense on a router.
Ensure we have firewalling enabled automatically.
Create a number of users that will otherwise get created at boot, and a lock file that will otherwise get created anyway.
There are config files that are easier to replace wholesale, some of which are specific to the hardware (e.g. related to network interfaces). See below for some more details.
The kernel + its modules live outside the squashfs image, on the USB flash drive that the image lives on. That makes for easier kernel upgrades.
Always helpful to be able to check the image itself for what it was built from.
Just some niceties for when I do end up logging in.
Actually build the squashfs image.
Save off the installed package list. This was particularly useful when trying to replicate the existing router setup and making sure I had all the important packages installed. It doesn t really serve a purpose now.
In terms of the config files I copy into #!/bin/bash
# Either rb3011 (arm) or rb5009 (arm64)
#HOSTNAME="rb3011"
HOSTNAME="rb5009"
if [ "x$ HOSTNAME " == "xrb3011" ]; then
ARCH=armhf
elif [ "x$ HOSTNAME " == "xrb5009" ]; then
ARCH=arm64
else
echo "Unknown host: $ HOSTNAME "
exit 1
fi
It s a bash script, and I allow building for either my RB3011 or RB5009, which means a different architecture (32 vs 64 bit). I run this script on my Pi 4 which means I don t have to mess about with QemuUserEmulation.
BASE_DIR=$(dirname $0)
IMAGE_FILE=$(mktemp --tmpdir router.$ ARCH .XXXXXXXXXX.img)
MOUNT_POINT=$(mktemp -p /mnt -d router.$ ARCH .XXXXXXXXXX)
# Build and mount an ext4 image file to put the root file system in
dd if=/dev/zero bs=1 count=0 seek=1G of=$ IMAGE_FILE
mkfs -t ext4 $ IMAGE_FILE
mount -o loop $ IMAGE_FILE $ MOUNT_POINT
I build the image in a loopback ext4 file on tmpfs (my Pi4 is the 8G model), which makes things a bit faster.
# Add dpkg excludes
mkdir -p $ MOUNT_POINT /etc/dpkg/dpkg.cfg.d/
cat <<EOF > $ MOUNT_POINT /etc/dpkg/dpkg.cfg.d/path-excludes
# Exclude docs
path-exclude=/usr/share/doc/*
# Only locale we want is English
path-exclude=/usr/share/locale/*
path-include=/usr/share/locale/en*/*
path-include=/usr/share/locale/locale.alias
# No man pages
path-exclude=/usr/share/man/*
EOF
Create a dpkg excludes config to drop docs, man pages and most locales before we even start the bootstrap.
# Setup fstab + mtab
echo "# Empty fstab as root is pre-mounted" > $ MOUNT_POINT /etc/fstab
ln -s ../proc/self/mounts $ MOUNT_POINT /etc/mtab
# Setup hostname
echo $ HOSTNAME > $ MOUNT_POINT /etc/hostname
# Add the root SSH keys
mkdir -p $ MOUNT_POINT /root/.ssh/
cat <<EOF > $ MOUNT_POINT /root/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8NkUeVdsVdegS+JT9qwFwiHEgcC9sBwnv6RjpH6I4d3im4LOaPOatzneMTZlH8Gird+H4nzluciBr63hxmcFjZVW7dl6mxlNX2t/wKvV0loxtEmHMoI7VMCnrWD0PyvwJ8qqNu9cANoYriZRhRCsBi27qPNvI741zEpXN8QQs7D3sfe4GSft9yQplfJkSldN+2qJHvd0AHKxRdD+XTxv1Ot26+ZoF3MJ9MqtK+FS+fD9/ESLxMlOpHD7ltvCRol3u7YoaUo2HJ+u31l0uwPZTqkPNS9fkmeCYEE0oXlwvUTLIbMnLbc7NKiLgniG8XaT0RYHtOnoc2l2UnTvH5qsQ== noodles@earth.li
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQb9+qFemcwKhey3+eTh5lxp+3sgZXW2HQQEZMt9hPvVXk+MiiNMx9WUzxPJnwXqlmmVdKsq+AvjA0i505Pp8fIj5DdUBpSqpLghmzpnGuob7SSwXYj+352hjD52UC4S0KMKbIaUpklADgsCbtzhYYc4WoO8F7kK63tS5qa1XSZwwRwPbYOWBcNocfr9oXCVWD9ismO8Y0l75G6EyW8UmwYAohDaV83pvJxQerYyYXBGZGY8FNjqVoOGMRBTUcLj/QTo0CDQvMtsEoWeCd0xKLZ3gjiH3UrknkaPra557/TWymQ8Oh15aPFTr5FvKgAlmZaaM0tP71SOGmx7GpCsP4jZD1Xj/7QMTAkLXb+Ou6yUOVM9J4qebdnmF2RGbf1bwo7xSIX6gAYaYgdnppuxqZX1wyAy+A2Hie4tUjMHKJ6OoFwBsV1sl+3FobrPn6IuulRCzsq2aLqLey+PHxuNAYdSKo7nIDB3qCCPwHlDK52WooSuuMidX4ujTUw7LDTia9FxAawudblxbrvfTbg3DsiDBAOAIdBV37HOAKu3VmvYSPyqT80DEy8KFmUpCEau59DID9VERkG6PWPVMiQnqgW2Agn1miOBZeIQV8PFjenAySxjzrNfb4VY/i/kK9nIhXn92CAu4nl6D+VUlw+IpQ8PZlWlvVxAtLonpjxr9OTw== noodles@yubikey
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0I8UHj4IpfqUcGE4cTvLB0d2xmATSUzqtxW6ZhGbZxvQDKJesVW6HunrJ4NFTQuQJYgOXY/o82qBpkEKqaJMEFHTCjcaj3M6DIaxpiRfQfs0nhtzDB6zPiZn9Suxb0s5Qr4sTWd6iI9da72z3hp9QHNAu4vpa4MSNE+al3UfUisUf4l8TaBYKwQcduCE0z2n2FTi3QzmlkOgH4MgyqBBEaqx1tq7Zcln0P0TYZXFtrxVyoqBBIoIEqYxmFIQP887W50wQka95dBGqjtV+d8IbrQ4pB55qTxMd91L+F8n8A6nhQe7DckjS0Xdla52b9RXNXoobhtvx9K2prisagsHT noodles@cup
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK6iGog3WbNhrmrkglNjVO8/B6m7mN6q1tMm1sXjLxQa+F86ETTLiXNeFQVKCHYrk8f7hK0d2uxwgj6Ixy9k0Cw= noodles@sevai
EOF
Setup fstab, the hostname and SSH keys for root.
# Bootstrap our install
debootstrap \
--arch=$ ARCH \
--include=collectd-core,conntrack,dnsmasq,ethtool,iperf3,kexec-tools,mosquitto,mtd-utils,mtr-tiny,ppp,tcpdump,rng-tools5,ssh,watchdog,wget \
--exclude=dmidecode,isc-dhcp-client,isc-dhcp-common,makedev,nano \
bullseye $ MOUNT_POINT https://deb.debian.org/debian/
Actually do the
debootstrap step, including a bunch of extra packages that we want.
# Install mqtt-arp
cp $ BASE_DIR /debs/mqtt-arp_1_$ ARCH .deb $ MOUNT_POINT /tmp
chroot $ MOUNT_POINT dpkg -i /tmp/mqtt-arp_1_$ ARCH .deb
rm $ MOUNT_POINT /tmp/mqtt-arp_1_$ ARCH .deb
# Frob the mqtt-arp config so it starts after mosquitto
sed -i -e 's/After=.*/After=mosquitto.service/' $ MOUNT_POINT /lib/systemd/system/mqtt-arp.service
I haven t uploaded mqtt-arp to Debian, so I install a locally built package, and ensure it starts after mosquitto (the MQTT broker), given they re running on the same host.
# Frob watchdog so it starts earlier than multi-user
sed -i -e 's/After=.*/After=basic.target/' $ MOUNT_POINT /lib/systemd/system/watchdog.service
# Make sure the watchdog is poking the device file
sed -i -e 's/^#watchdog-device/watchdog-device/' $ MOUNT_POINT /etc/watchdog.conf
watchdog timeouts were particularly an issue on the RB3011, where the default timeout didn t give enough time to reach multiuser mode before it would reset the router. Not helpful, so alter the config to start it earlier (and make sure it s configured to actually kick the device file).
# Clean up docs + locales
rm -r $ MOUNT_POINT /usr/share/doc/*
rm -r $ MOUNT_POINT /usr/share/man/*
for dir in $ MOUNT_POINT /usr/share/locale/*/; do
if [ "$ dir " != "$ MOUNT_POINT /usr/share/locale/en/" ]; then
rm -r $ dir
fi
done
Clean up any docs etc that ended up installed.
# Set root password to root
echo "root:root" chroot $ MOUNT_POINT chpasswd
The only login method is ssh key to the root account though I suppose this allows for someone to execute a privilege escalation from a daemon user so I should probably randomise this. Does need to be known though so it s possible to login via the serial console for debugging.
# Add security to sources.list + update
echo "deb https://security.debian.org/debian-security bullseye-security main" >> $ MOUNT_POINT /etc/apt/sources.list
chroot $ MOUNT_POINT apt update
chroot $ MOUNT_POINT apt -y full-upgrade
chroot $ MOUNT_POINT apt clean
# Cleanup the APT lists
rm $ MOUNT_POINT /var/lib/apt/lists/www.*
rm $ MOUNT_POINT /var/lib/apt/lists/security.*
Pull in any security updates, then clean out the APT lists rather than polluting the image with them.
# Disable the daily APT timer
rm $ MOUNT_POINT /etc/systemd/system/timers.target.wants/apt-daily.timer
# Disable daily dpkg backup
cat <<EOF > $ MOUNT_POINT /etc/cron.daily/dpkg
#!/bin/sh
# Don't do the daily dpkg backup
exit 0
EOF
# We don't want a persistent systemd journal
rmdir $ MOUNT_POINT /var/log/journal
None of these make sense on a router.
# Enable nftables
ln -s /lib/systemd/system/nftables.service \
$ MOUNT_POINT /etc/systemd/system/sysinit.target.wants/nftables.service
Ensure we have firewalling enabled automatically.
# Add systemd-coredump + systemd-timesync user / group
echo "systemd-timesync:x:998:" >> $ MOUNT_POINT /etc/group
echo "systemd-coredump:x:999:" >> $ MOUNT_POINT /etc/group
echo "systemd-timesync:!*::" >> $ MOUNT_POINT /etc/gshadow
echo "systemd-coredump:!*::" >> $ MOUNT_POINT /etc/gshadow
echo "systemd-timesync:x:998:998:systemd Time Synchronization:/:/usr/sbin/nologin" >> $ MOUNT_POINT /etc/passwd
echo "systemd-coredump:x:999:999:systemd Core Dumper:/:/usr/sbin/nologin" >> $ MOUNT_POINT /etc/passwd
echo "systemd-timesync:!*:47358::::::" >> $ MOUNT_POINT /etc/shadow
echo "systemd-coredump:!*:47358::::::" >> $ MOUNT_POINT /etc/shadow
# Create /etc/.pwd.lock, otherwise it'll end up in the overlay
touch $ MOUNT_POINT /etc/.pwd.lock
chmod 600 $ MOUNT_POINT /etc/.pwd.lock
Create a number of users that will otherwise get created at boot, and a lock file that will otherwise get created anyway.
# Copy config files
cp --recursive --preserve=mode,timestamps $ BASE_DIR /etc/* $ MOUNT_POINT /etc/
cp --recursive --preserve=mode,timestamps $ BASE_DIR /etc-$ ARCH /* $ MOUNT_POINT /etc/
chroot $ MOUNT_POINT chown mosquitto /etc/mosquitto/mosquitto.users
chroot $ MOUNT_POINT chown mosquitto /etc/ssl/mqtt.home.key
There are config files that are easier to replace wholesale, some of which are specific to the hardware (e.g. related to network interfaces). See below for some more details.
# Build symlinks into flash for boot / modules
ln -s /mnt/flash/lib/modules $ MOUNT_POINT /lib/modules
rmdir $ MOUNT_POINT /boot
ln -s /mnt/flash/boot $ MOUNT_POINT /boot
The kernel + its modules live outside the squashfs image, on the USB flash drive that the image lives on. That makes for easier kernel upgrades.
# Put our git revision into os-release
echo -n "GIT_VERSION=" >> $ MOUNT_POINT /etc/os-release
(cd $ BASE_DIR ; git describe --tags) >> $ MOUNT_POINT /etc/os-release
Always helpful to be able to check the image itself for what it was built from.
# Add some stuff to root's .bashrc
cat << EOF >> $ MOUNT_POINT /root/.bashrc
alias ls='ls -F --color=auto'
eval "\$(dircolors)"
case "\$TERM" in
xterm* rxvt*)
PS1="\\[\\e]0;\\u@\\h: \\w\a\\]\$PS1"
;;
*)
;;
esac
EOF
Just some niceties for when I do end up logging in.
# Build the squashfs
mksquashfs $ MOUNT_POINT /tmp/router.$ ARCH .squashfs \
-comp xz
Actually build the squashfs image.
# Save the installed package list off
chroot $ MOUNT_POINT dpkg --get-selections > /tmp/wip-installed-packages
Save off the installed package list. This was particularly useful when trying to replicate the existing router setup and making sure I had all the important packages installed. It doesn t really serve a purpose now.
/etc, shared across both routers are the following:
Breakdown of shared config
The router specific config is mostly related to networking:
- apt config (disable recommends, periodic updates):
apt/apt.conf.d/10periodic,apt/apt.conf.d/local-recommends
- Adding a default, empty, locale:
default/locale
- DNS/DHCP:
dnsmasq.conf,dnsmasq.d/dhcp-ranges,dnsmasq.d/static-ipshosts,resolv.conf
- Enabling IP forwarding:
sysctl.conf
- Logs related:
logrotate.conf,rsyslog.conf
- MQTT related:
mosquitto/mosquitto.users,mosquitto/conf.d/ssl.conf,mosquitto/conf.d/users.conf,mosquitto/mosquitto.acl,mosquitto/mosquitto.confmqtt-arp.confssl/lets-encrypt-r3.crt,ssl/mqtt.home.key,ssl/mqtt.home.crt
- PPP configuration:
ppp/ip-up.d/0000usepeerdns,ppp/ipv6-up.d/defaultroute,ppp/pap-secrets,ppp/chap-secretsnetwork/interfaces.d/pppoe-wan
Breakdown of router specific config
- Firewalling:
nftables.conf
- Interfaces:
dnsmasq.d/interfacesnetwork/interfaces.d/eth0,network/interfaces.d/p1,network/interfaces.d/p2,network/interfaces.d/p7,network/interfaces.d/p8
- PPP config (network interface piece):
ppp/peers/aquiss
- SSH keys:
ssh/ssh_host_ecdsa_key,ssh/ssh_host_ed25519_key,ssh/ssh_host_rsa_key,ssh/ssh_host_ecdsa_key.pub,ssh/ssh_host_ed25519_key.pub,ssh/ssh_host_rsa_key.pub
- Monitoring:
collectd/collectd.conf,collectd/collectd.conf.d/network.conf
The idea is simply to have one coherent place with pointers to all the stuff we have and provide, without repeating nor replacing other documentation.
Hello, World!
Today s hacks come remotely, on the first day of my long trek to Cleveland. As many debian readers know, debian URLs are usually transmitted over IRC or email with some snappy abbreviations. Usually, you ll see something like:
According to Eric S. Raymond, one rule for Unix programming is the Rule of
Silence: When a program has nothing surprising to say, it should say nothing.
Since return codes are a form of information, I would like to add: when a
program has carried out its task with no surprise, it should exit(0).Take 
I was asked to write about what I do at 
Here are a few hints if you want your reports to be treated more promptly.
When testing fails, try unstable
If you are running testing, not all bug reports are interesting. More specifically, the relevant ones are about insufficient dependencies that let slip into testing a non-working combination of packages. Think of the I upgraded foo, then bar broke category.
Otherwise, the reason why reportbug looks for newer versions of a package in unstable is that, in most cases, the bug is fixed in unstable. If you can, always try the unstable version before reporting the bug.
Report bugs against the failing package
This is related to printing in a GNOME program, so I report this against libgnomeprint.
Always file reports against the failing program. If there is a library involved, reportbug will include the necessary information so that the maintainer can easily reassign. If you re not sure of the affected package, use a metapackage (for example, gnome for GNOME-related packages).
Do not second-guess the origin of the bug
After upgrading libfoo, bar starts crashing, so I m reporting it against libfoo.
Unless you can tell which function is at fault and how this is ABI breakage, you should report the bug against the program that crashes. The program s maintainer will reassign if needed, and will hit the library s maintainer with a hammer if needed.
Don t forget to explain what s wrong
The frobniz in libfrob is broken, you should change /usr/share/libfrob/blah line 13 to something else. I don t know what libfrob is about, but I have some problems that look frobniz-related.
Do not second-guess, always start explaining what bug you are seeing first. Your attempts at investigating it can be interesting, but don t forget to explain what bug lead you on investigating.
What is the use case we should cover?
I m trying to do that with your package and it looks extremely complicated/doesn t work as expected/fails miserably. Your package is clearly broken!
If your attempts to do something that should be trivial, given the purpose of the package, fail, then you are probably looking at the problem at the wrong angle. There may be a much simpler, though radically different, way to do what you are trying to achieve.
In these situations, always include in your report what you are trying to do and why you want to do it.
strace is not a general-purpose debugging tool
Whether you observe a crash, a deadlock or a livelock, strace is not the tool that will give us information. The only thing you do by adding a strace log to the bug report is making it unreadable with its thousands of lines of output.
For the specific cases that require strace for debugging, these traces are welcome but if you re not sure, don t attach them. The developers will ask them for you. Otherwise, the general-purpose debugging tool is 
I’ve got a load of boxes running Debian etch with Xen 3.0.3 with routed networking (rather than bridged, so I can do iptables and reverse path filtering etc in dom0). Since upgrading from Xen 2.x many moons ago, I’ve not known how to configure one virtual interface to have more than one IP. In the meantime, I’ve ended up doing nonsense like providing a VM with two interfaces just to give it two IPs. However, this interacts really badly with reverse path filtering unless you do a bunch of source-routing rocket science in the domU to send out through the right vif.
So, I looked at the vif-route script and it seems to support iterating through a space-separated list of IPs, but I was totally unable to find any documentation or mailing list posts explaining how to format the IPs within the formerly-Python key/value Xen domain config file syntax. After a while playing with the parser and various levels of quoting, I found that actually, the correct amount of quoting is none at all, and also uncovered a bug in another script which prevents it from working correctly. In the hope that this might help others using Google and trying to achieve the same as me, here is my recipe for configuring Xen vif devices to have multiple IPs (note that I think this might be specific to Xen 3.0.x, as I believe 3.2.x introduces config files in the S-expression format which is what xenstore uses internally):
I thought I'd throw the