In my previous articles in
the series on asynchronous communication with the modern NNCP tool, I talked about its use for asynchronous, potentially airgapped, backups. The first article,
How & Why To Use Airgapped Backups laid out the foundations for this. Now let s dig into the details.
Today s post will cover ZFS, because it has a lot of features that make it very easy to support in this setup. Non-ZFS backups will be covered later.
The setup is actually about as simple as it is for SSH, but since people are less familiar with this kind of communication, I m going to try to go into more detail here.
Assumptions
I am assuming a setup where:
- The machines being backed up run ZFS
- The disk(s) that hold the backups are also running ZFS
- zfs send / receive is desired as an efficient way to transport the backups
- The machine that holds the backups may have no network connection whatsoever
- Backups will be sent encrypted over some sort of network to a spooling machine, which temporarily holds them until they are transported to the destination backup system and ingested there. This system will be unable to decrypt the data streams it temporarily stores.
Hardware
Let s start with hardware for the machine to hold the backups. I initially considered a Raspberry Pi 4 with 8GB of RAM. That would probably have been a suitable machine, at least for smaller backup sets. However, none of the Raspberry Pi machines support hardware AES encryption acceleration, and my Pi4 benchmarks as about 60MB/s for AES encryption. I want my backups to be encrypted, and decided this would just be too slow for my purposes. Again, if you don t need encrypted backups or don t care that much about performance may people probably fall into this category you can have a fully-functional Raspberry Pi 4 system for under $100 that would make a fantastic backup server.
I wound up purchasing a
Qotom-Q355G4 micro PC with a Core i5 for about $315. It has USB 3 ports and is designed as a rugged, long-lasting system. I have been using one of their older Celeron-based models as my router/firewall for a number of years now and it s been quite reliable.
For backup storage, you can get a USB 3 external drive. My own preference is to get a USB 3 toaster (device that lets me plug in SATA drives) so that I have more control over the underlying medium and can save the expense and hassle of a bunch of power supplies. In a future post, I will discuss drive rotation so you always have an offline drive.
Then, there is the question of transport to the backup machine. A simple solution would be to have a heavily-firewalled backup system that has no incoming ports open but makes occasional outgoing connections to one specific NNCP daemon on the spooling machine. However, for airgapped operation, it would also be very simple to use
nncp-xfer to transport the data across on a USB stick or some such. You could set up automounting for a specific USB stick plug it in, all the spooled data is moved over, then plug it in to the backup system and it s processed, and any outbound email traffic or whatever is copied to the USB stick at that point too. The NNCP page has some
more commentary about this kind of setup.
Both are fairly easy to set up, and NNCP is designed to be transport-agnostic, so in this article I m going to focus on how to integrate ZFS with NNCP.
Operating System
Of course, it should be no surprise that I set this up on Debian.
As an added step, I did all the configuration in Ansible stored in a local git repo. This adds a lot of work, but it means that it is trivial to periodically wipe and reinstall if any security issue is suspected. The git repo can be copied off to another system for storage and takes the system from freshly-installed to ready-to-use state.
Security
There is, of course, nothing preventing you from running NNCP as root. The zfs commands, obviously, need to be run as root. However, from a privilege separation standpoint, I have chosen to run everything relating to NNCP as a nncp user. NNCP already does encryption, but if you prefer to have zero knowledge of the data even to NNCP, it s trivial to add gpg to the pipeline as well, and in fact I ll be demonstrating that in a future post for other reasons.
Software
Besides
NNCP, there needs to be a system that generates the zfs send streams. For this project, I looked at quite a few. Most were designed to inspect the list of snapshots on a remote end, compare it to a list on the local end, and calculate a difference from there. This, of course, won t work for this situation.
I realized my own
simplesnap project was very close to being able to do this. It already used an algorithm of using specially-named snapshots on the machine being backed up, so never needed any communication about what snapshots were present where. All it needed was a few more options to permit sending to a stream instead of zfs receive. I made those changes and they are available in simplesnap
2.0.0 or above. That version has also been uploaded to sid, and will work fine as-is on buster as well.
Preparing NNCP
I m going to assume three hosts in this setup:
- laptop is the machine being backed up. Of course, you may have quite a few of these.
- spooler holds the backup data until the backup system picks it up
- backupsvr holds the backups
The basic
NNCP workflow documentation covers the basic steps. You ll need to run
nncp-cfgnew on each machine. This generates a basic configuration, along with public and private keys for that machine. You ll copy the public key sets to the configurations of the other machines as usual. On the laptop, you ll add a via line like this:
backupsvr:
id: ....
exchpub: ...
signpub: ...
noisepub: ...
via: ["spooler"]
This tells NNCP that data destined for backupsvr should always be sent via spooler first.
You can then arrange for the nncp-daemon to run on the spooler, and nncp-caller or nncp-call on the backupsvr. Or, alternatively, airgapped between the two with nncp-xfer.
Generating Backup Data
Now, on the laptop, install simplesnap (2.0.0 or above). Although you won t be backing up to the local system, simplesnap still maintains a hostlock in ZFS. Prepate a dataset for it:
zfs create tank/simplesnap
zfs set org.complete.simplesnap:exclude=on tank/simplesnap
Then, create a script /usr/local/bin/runsimplesnap like this:
#!/bin/bash
set -e
simplesnap --store tank/simplesnap --setname backups --local --host hostname \
--receivecmd /usr/local/bin/simplesnap-queue \
--noreap
su nncp -c '/usr/local/nncp/bin/nncp-toss -noprogress -quiet'
if ip addr grep -q 192.168.65.64; then
su nncp -c '/usr/local/nncp/bin/nncp-call -noprogress -quiet -onlinedeadline 1 spooler'
fi
The call to simplesnap sets it up to send the data to
simplesnap-queue, which we ll create in a moment. The receivmd, plus noreap, sets it up to run without ZFS on the local system.
The call to nncp-toss will process any previously-received inbound NNCP packets, if there are any. Then, in this example, we do a very basic check to see if we re on the LAN (checking 192.168.65.64), and if so, will establish a connection to the spooler to transmit the data. If course, you could also do this over the Internet, with tor, or whatever, but in my case, I don t want to automatically do this in case I m tethered to mobile. I figure if I want to send backups in that case, I can fire up nncp-call myself. You can also use nncp-caller to set up automated connections on other schedules; there are a lot of options.
Now, here s what /usr/local/bin/simplesnap-queue looks like:
#!/bin/bash
set -e
set -o pipefail
DEST=" echo $1 sed 's,^tank/simplesnap/,,' "
echo "Processing $DEST" >&2
# stdin piped to this
su nncp -c "/usr/local/nncp/bin/nncp-exec -nice B -noprogress backupsvr zfsreceive '$DEST'" >&2
echo "Queued for $DEST" >&2
This is a pretty simple script. simplesnap will call it with a path based on the store, with the hostname after; so, for instance, tank/simplesnap/laptop/root or some such. This script strips off the leading tank/simplesnap (which is a local fragment), leaving the host and dataset paths. Then it just pipes it to nncp-exec. -nice B classifies it as low-priority bulk data (so if you have some more important interactive data, it would be sent first), then passes it to whatever the backupsvr defines as zfsreceive.
Receiving ZFS backups
In the NNCP configuration on the recipient s side, in the laptop section, we define what command it s allowed to run as zfsreceive:
exec:
zfsreceive: ["/usr/bin/sudo", "-H", "/usr/local/bin/nncp-zfs-receive"]
We authorize the nncp user to run this under sudo in /etc/sudoers.d/local nncp:
Defaults env_keep += "NNCP_SENDER"
nncp ALL=(root) NOPASSWD: /usr/local/bin/nncp-zfs-receive
The NNCP_SENDER is the public key ID of the sending node when nncp-toss processes the incoming data. We can use that for sanity checking later.
Now, here s a basic nncp-zfs-receive script:
#!/bin/bash
set -e
set -o pipefail
STORE=backups/simplesnap
DEST="$1"
# now process stdin
runcommand zfs receive -o readonly=on -x mountpoint "$STORE/$DEST"
And there you have it all the basics are in place.
Update 2020-12-30: An earlier version of this article had zfs receive -F instead of zfs receive -o readonly=on -x mountpoint . These changed arguments are more robust.
Update 2021-01-04: I am now recommending zfs receive -u -o readonly=on ; see my
successor article for more.
Enhancements
You could enhance the nncp-zfs-receive script to improve logging and error handling. For instance:
#!/bin/bash
set -e
set -o pipefail
STORE=backups/simplesnap
# $1 will be the host/dataset
DEST="$1"
HOST=" echo "$1" sed 's,/.*,,g' "
if [ -z "$HOST" ]; then
echo "Malformed command line"
exit 5
fi
# Log a message
logit ()
logger -p info -t " basename "$0" [$$]" "$1"
# Log an error message
logerror ()
logger -p err -t " basename "$0" [$$]" "$1"
# Log stdin with the given code. Used normally to log stderr.
logstdin ()
logger -p info -t " basename "$0" [$$/$1]"
# Run command, logging stderr and exit code
runcommand ()
logit "Running $*"
if "$@" 2> >(logstdin "$1") ; then
logit "$1 exited successfully"
return 0
else
RETVAL="$?"
logerror "$1 exited with error $RETVAL"
return "$RETVAL"
fi
exiterror ()
logerror "$1"
echo "$1" 1>&2
exit 10
# Sanity check
if [ "$HOST" = "laptop" ]; then
if [ "$NNCP_SENDER" != "12345678" ]; then
exiterror "Host $HOST doesn't match sender $NNCP_SENDER"
fi
else
exiterror "Unknown host $HOST"
fi
runcommand zfs receive -F "$STORE/$DEST"
Now you ll capture the ZFS receive output in syslog in a friendly way, so you can look back later why things failed if they did.
Further notes on NNCP
nncp-toss will examine the exit code from an invocation. If it is nonzero, it will keep the command (and associated stdin) in the queue and retry it on the next invocation. NNCP does not guarantee order of execution, so it is possible in some cases that ZFS streams may be received in the wrong order. That is fine here; zfs receive will exit with an error, and nncp-toss will just run it again after the dependent snapshots have been received. For non-ZFS backups, a simple sequence number can handle this issue.
I am running since a decade a small Debian VM, that I use for basic web and mail hosting. Since most of the VM setup is done manually and not following the Infrastructure As Code pattern, it is faster to simply copy the filesystem when switching providers instead of reconfiguring everything.
Here s a list of some Debian packaging work for December 2020.
2020-12-01: Sponsor package mangohud (0.6.1-1) for Debian unstable (mentors.debian.net request).
2020-12-01: Sponsor package spyne (2.13.16-1) for Debian unstable (Python team request).
2020-12-01: Sponsor package python-xlrd (1.2.0-1) for Debian unstable (Python team request).
2020-12-01: Sponsor package buildbot for Debian unstable (Python team request).
2020-12-08: Upload package calamares (3.2.35.1-1) to Debian unstable.
2020-12-09: Upload package btfs (2.23-1) to Debian unstable.
2020-12-09: Upload package feed2toot (0.15-1) to Debian unstable.
2020-12-09: Upload package gnome-shell-extension-harddisk-led (23-1) to Debian unstable.
2020-12-10: Upload package feed2toot (0.16-1) to Debian unstable.
2020-12-10: Upload package gnome-shell-extension-harddisk-led (24-1) to Debian unstable.
2020-12-13: Upload package xabacus (8.3.1-1) to Debian unstable.
2020-12-14: Upload package python-aniso8601 (8.1.0-1) to Debian unstable.
2020-12-19: Upload package rootskel-gtk (1.42) to Debian unstable.
2020-12-21: Sponsor package goverlay (0.4.3-1) for Debian unstable (mentors.debian.net request).
2020-12-21: Sponsor package pastel (0.2.1-1) for Debian unstable (Python team request).
2020-12-22: Sponsor package python-requests-toolbelt (0.9.1-1) for Debian unstable (Python team request).
2020-12-22: Upload kpmcore (20.12.0-1) to Debian unstable.
2020-12-26: Upload package bundlewrap (4.3.0-1) to Debian unstable.
2020-12-26: Review package python-strictyaml (1.1.1-1) (Needs some more work) (Python team request).
2020-12-26: Review package buildbot (2.9.3-1) (Needs some more work) (Python team request).
2020-12-26: Review package python-vttlib (0.9.1+dfsg-1) (Needs some more work) (Python team request).
2020-12-26: Sponsor package python-formencode (2.0.0-1) for Debian unstable (Python team request).
2020-12-26: Sponsor package pylev (1.2.0-1) for Debian unstable (Python team request).
2020-12-26: Review package python-absl (Needs some more work) (Python team request).
2020-12-26: Sponsor package python-moreorless (0.3.0-2) for Debian unstable (Python team request).
2020-12-26: Sponsor package peewee (3.14.0+dfsg-1) for Debian unstable (Python team request).
2020-12-28: Sponsor package pympler (0.9+dfsg1-1) for Debian unstable (Python team request).
2020-12-28: Sponsor package bidict (0.21.2-1) for Debian unstable (Python team request).
Reviving Very Old X Code
I've taken the week between Christmas and New Year's off this year. I
didn't really have anything serious planned, just taking a break from
the usual routine. As often happens, I got sucked into doing a project
when I received this simple bug report
On doing 540 no-source-change source-only uploads in two weeks
So I've been doing 540 no-source-change source-only uploads in the last two weeks
and am planning to do 3000 more in January 2021. We'll see how that goes 
Let me explain what I have been doing and why.
So, 
