John Goerzen: More Topics on Store-And-Forward (Possibly Airgapped) ZFS and Non-ZFS Backups with NNCP
Note: this is another article in my series on asynchronous communication in Linux with UUCP and NNCP. In my previous post, I introduced a way to use ZFS backups over NNCP. In this post, I ll expand on that and also explore non-ZFS backups. Use of nncp-file instead of nncp-exec The previous example used nncp-exec (like UUCP s uux), which lets you pipe stdin in, then queues up a request to run a given command with that input on a remote. I discussed that NNCP doesn t guarantee order of execution, but that for the ZFS use case, that was fine since zfs receive would just fail (causing NNCP to try again later). At present, nncp-exec stores the data piped to it in RAM before generating the outbound packet (the author plans to fix this shortly). That made it unusable for some of my backups, so I set it up another way: with nncp-file, the tool to transfer files to a remote machine. A cron job then picks them up and processes them. On the machine being backed up, we have to find a way to encode the dataset to be received. I chose to do that as part of the filename, so the updated simplesnap-queue could look like this:
#!/bin/bash set -e set -o pipefail DEST=" echo $1 sed 's,^tank/simplesnap/,,' " FILE="bakfsfmt2- date "+%s.%N".$$ _ echo "$DEST" sed 's,/,@,g' " echo "Processing $DEST to $FILE" >&2 # stdin piped to this zstd -8 - \ gpg --compress-algo none --cipher-algo AES256 -e -r 012345... \ su nncp -c "/usr/local/nncp/bin/nncp-file -nice B -noprogress - 'backupsvr:$FILE'" >&2 echo "Queued $DEST to $FILE" >&2I ve added compression and encryption here as well; more on that below. On the backup server, we would define a different incoming directory for each node in nncp.hjson. For instance:
host1: ... incoming: "/var/local/nncp-bakcups-incoming/host1" host2: ... incoming: "/var/local/nncp-backups-incoming/host2"I ll present the scanning script in a bit. Offsite Backup Rotation Most of the time, you don t want just a single drive to store the backups. You d like to have a set. At minimum, one wouldn t be plugged in so lightning wouldn t ruin all your backups. But maybe you d store a second drive at some other location you have access to (friend s house, bank box, etc.) There are several ways you could solve this:
- If the remote machine is at a location with network access and you trust its physical security (remember that although it will store data encrypted at rest and will transport it encrypted, it will in most cases handle un-encrypted data during processing), you could of course send NNCP packets to it over the network at the same time you send them to your local backup system.
- Alternatively, if the remote location doesn t have network access or you want to keep it airgapped, you could transport the NNCP packets by USB drive to the remote end.
- Or, if you don t want to have any kind of processing capability remotely probably a wise move you could rotate the hard drives themselves, keeping one plugged in locally and unplugging the other to take it offsite.
zstd -8 - gpg --compress-algo none --cipher-algo AES256 -e -r 07D5794CD900FAF1D30B03AC3D13151E5039C9D5 \ tee >(su nncp -c "/usr/local/nncp/bin/nncp-file -nice B+5 -noprogress - 'backupdisk1:$FILE'") \ >(su nncp -c "/usr/local/nncp/bin/nncp-file -nice B+5 -noprogress - 'backupdisk2:$FILE'") \ > /dev/nullYou could probably also more safely use pee(1) (from moreutils) to do this. This has an unfortunate result of doubling the network traffic from every machine being backed up. So an alternative option would be to queue the packets to the spooling machine, and run a distribution script from it; something like this, in part:
INCOMINGDIR="/var/local/nncp-bakfs-incoming" LOCKFILE="$INCOMINGDIR/.lock" printf -v EVAL_SAFE_LOCKFILE '%q' "$LOCKFILE" if dotlockfile -r 0 -l -p "$ LOCKFILE "; then logit "Lock obtained at $ LOCKFILE with dotlockfile" trap 'ECODE=$?; dotlockfile -u '"$ EVAL_SAFE_LOCKFILE "'; exit $ECODE' EXIT INT TERM else logit "Could not obtain lock at $LOCKFILE; $0 likely already running." exit 0 fi logit "Scanning queue directory..." cd "$INCOMINGDIR" for HOST in *; do cd "$INCOMINGDIR/$HOST" for FILE in bakfsfmt2-*; do if [ -f "$FILE" ]; then for BAKFS in backupdisk1 backupdisk2; do runcommand nncp-file -nice B+5 -noprogress "$FILE" "$BAKFS:$HOST/$FILE" done runcommand rm "$FILE" else logit "$HOST: Skipping $FILE since it doesn't exist" fi done done logit "Scan complete."Security Considerations You ll notice that in my example above, the encryption happens as the root user, but nncp is called under su. This means that even if there is a vulnerability in NNCP, the data would still be protected by GPG. I ll also note here that many sites run ssh as root unnecessarily; the same principles should apply there. (ssh has had vulnerabilities in the past as well). I could have used gpg s built-in compression, but zstd is faster and better, so we can get good performance by using fast compression and piping that to an algorithm that can use hardware acceleration for encryption. I strongly encourage considering transport, whether ssh or NNCP or UUCP, to be untrusted. Don t run it as root if you can avoid it. In my example, the nncp user, which all NNCP commands are run as, has no access to the backup data at all. So even if NNCP were compromised, my backup data wouldn t be. For even more security, I could also sign the backup stream with gpg and validate that on the receiving end. I should note, however, that this conversation assumes that a network- or USB-facing ssh or NNCP is more likely to have an exploitable vulnerability than is gpg (which here is just processing a stream). This is probably a safe assumption in general. If you believe gpg is more likely to have an exploitable vulnerability than ssh or NNCP, then obviously you wouldn t take this particular approach. On the zfs side, the use of -F with zfs receive is avoided; this could lead to a compromised backed-up machine generating a malicious rollback on the destination. Backup zpools should be imported with -R or -N to ensure that a malicious mountpoint property couldn t be used to cause an attack. I choose to use zfs receive -u -o readonly=on which is compatible with both unmounted backup datasets and zpools imported with -R (or both). To access the data in a backup dataset, you would normally clone it and access it there. The processing script So, put this all together and look at an example of a processing script that would run from cron as root and process the incoming ZFS data.
#!/bin/bash set -e set -o pipefail # Log a message logit () logger -p info -t " basename "$0" [$$]" "$1" # Log an error message logerror () logger -p err -t " basename "$0" [$$]" "$1" # Log stdin with the given code. Used normally to log stderr. logstdin () logger -p info -t " basename "$0" [$$/$1]" # Run command, logging stderr and exit code runcommand () logit "Running $*" if "$@" 2> >(logstdin "$1") ; then logit "$1 exited successfully" return 0 else RETVAL="$?" logerror "$1 exited with error $RETVAL" return "$RETVAL" fi STORE=backups/simplesnap INCOMINGDIR=/backups/nncp/incoming if ! [ -d "$INCOMINGDIR" ]; then logerror "$INCOMINGDIR doesn't exist" exit 0 fi LOCKFILE="/backups/nncp/.nncp-backups-zfs-scan.lock" printf -v EVAL_SAFE_LOCKFILE '%q' "$LOCKFILE" if dotlockfile -r 0 -l -p "$ LOCKFILE "; then logit "Lock obtained at $ LOCKFILE with dotlockfile" trap 'ECODE=$?; dotlockfile -u '"$ EVAL_SAFE_LOCKFILE "'; exit $ECODE' EXIT INT TERM else logit "Could not obtain lock at $LOCKFILE; $0 likely already running." exit 0 fi EXITCODE=0 cd "$INCOMINGDIR" logit "Scanning queue directory..." for HOST in *; do HOSTPATH="$INCOMINGDIR/$HOST" # files like backupsfmt2-134.13134_dest for FILE in "$HOSTPATH"/backupsfmt2-[0-9]*_?*; do if [ ! -f "$FILE" ]; then logit "Skipping non-existent $FILE" continue fi # Now, $DEST will be HOST/DEST. Strip off the @ also. DEST=" echo "$FILE" sed -e 's/^.*backupsfmt2[^_]*_//' -e 's,@,/,g' " if [ -z "$DEST" ]; then logerror "Malformed dest in $FILE" continue fi HOST2=" echo "$DEST" sed 's,/.*,,g' " if [ -z "$HOST2" ]; then logerror "Malformed DEST $DEST in $FILE" continue fi if [ ! "$HOST" = "$HOST2" ]; then logerror "$DIR: $HOST doesn't match $HOST2" continue fi logit "Processing $FILE to $STORE/$DEST" if runcommand gpg -q -d < "$FILE" runcommand zstdcat runcommand zfs receive -u -o readonly=on "$STORE/$DEST"; then logit "Successfully processed $FILE to $STORE/$DEST" runcommand rm "$FILE" else logerror "FAILED to process $FILE to $STORE/$DEST" EXITCODE=15 fiApplying These Ideas to Non-ZFS Backups ZFS backups made our job easier in a lot of ways:
- ZFS can calculate a diff based on an efficiently-stored previous local state (snapshot or bookmark), rather than a comparison to a remote state (rsync)
- ZFS "incremental" sends, while less efficient than rsync, are reasonably efficient, sending only changed blocks
- ZFS receive detects and enforces that the incremental source on the local machine must match the incremental source of the original stream, enforcing ordering
- Datasets using ZFS encryption can be sent in their encrypted state
- Incrementals can be done without a full scan of the filesystem