• Uses blockdev-backup QMP commands instead of the soon to be deprecated drive-backup.
• Adds --compress option: target QCOW files data can be compressed.
• Adds --speed-limit option to limit backup speed throughput.
• Adds --include-raw option so attached raw devices can be backed up too.
• Allows backing up virtual machines in paused or pre-running state.
• Adds backup option copy to allow backing up virtual machines with older qcow image formats, that are not supporting persistent bitmaps.
• Improved logging output.

• Create bootable REAR rescue image on the source system.
• Register a target system in Hetzner Cloud with at least the same disk size as the source system.
• Boot the REAR image s initrd and kernel from the running VPS system using kexec
• Make the REAR recovery console accessible via SSH (or use Hetzners console).
• Let REAR do its magic and re-partition the system.
• Restore the system data to the freshly partitioned disks
• Let REAR handle the bootloader and network re-configuration.

Example To create a rescue image on the source system:

apt install rear
echo OUTPUT=ISO > /etc/rear/local.conf
rear mkrescue -v
[..]
Wrote ISO image: /var/lib/rear/output/rear-debian12.iso (185M)

My source system had a 128 GB disk, so i registered an instance on Hetzner cloud with greater disk size to make things easier: Now copy the ISO image to the newly created instance and extract its data:

 apt install kexec-tools
 scp rear-debian12.iso root@49.13.193.226:/tmp/
 modprobe loop
 mount -o loop rear-debian12.iso /mnt/
 cp /mnt/isolinux/kernel /tmp/
 cp /mnt/isolinux/initrd.cgz /tmp/

Install kexec if not installed already:

 apt install kexec-tools

Note down the current gateway configuration, this is required later on to make the REAR recovery console reachable via SSH:

root@testme:~# ip route
default via 172.31.1.1 dev eth0
172.31.1.1 dev eth0 scope link

Reboot the running VPS instance into the REAR recovery image using somewhat the same kernel cmdline:

root@testme:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.1.0-13-amd64 root=UUID=5174a81e-5897-47ca-8fe4-9cd19dc678c4 ro consoleblank=0 systemd.show_status=true console=tty1 console=ttyS0
kexec --initrd /tmp/initrd.cgz --command-line="consoleblank=0 systemd.show_status=true console=tty1 console=ttyS0" /tmp/kernel
Connection to 49.13.193.226 closed by remote host.
Connection to 49.13.193.226 closed

Now watch the system on the Console booting into the REAR system: Login the recovery console (root without password) and fix its default route to make it reachable:

ip addr
[..]
2: enp1s0
..
$ ip route add 172.31.1.1 dev enp1s0
$ ip route add default via 172.31.1.1
ping 49.13.193.226
64 bytes from 49.13.193.226: icmp_seq=83 ttl=52 time=27.7 ms

The network configuration might differ, the source system in this example used DHCP, as the target does. If REAR detects changed static network configuration it guides you through the setup pretty nicely. Login via SSH (REAR will store your ssh public keys in the image) and start the recovery process, follow the steps as suggested by REAR:

ssh -l root 49.13.193.226
Welcome to Relax-and-Recover. Run "rear recover" to restore your system !
RESCUE debian12:~ # rear recover
Relax-and-Recover 2.7 / Git
Running rear recover (PID 673 date 2024-01-04 19:20:22)
Using log file: /var/log/rear/rear-debian12.log
Running workflow recover within the ReaR rescue/recovery system
Will do driver migration (recreating initramfs/initrd)
Comparing disks
Device vda does not exist (manual configuration needed)
Switching to manual disk layout configuration (GiB sizes rounded down to integer)
/dev/vda had size 137438953472 (128 GiB) but it does no longer exist
/dev/sda was not used on the original system and has now 163842097152 (152 GiB)
Original disk /dev/vda does not exist (with same size) in the target system
Using /dev/sda (the only available of the disks) for recreating /dev/vda
Current disk mapping table (source => target):
  /dev/vda => /dev/sda
Confirm or edit the disk mapping
1) Confirm disk mapping and continue 'rear recover'
[..]
User confirmed recreated disk layout
[..]

This step re-recreates your original disk layout and mounts it to /mnt/local/ (this example uses a pretty lame layout, but usually REAR will handle things like lvm/btrfs just nicely):

mount
/dev/sda3 on /mnt/local type ext4 (rw,relatime,errors=remount-ro)
/dev/sda1 on /mnt/local/boot type ext4 (rw,relatime)

Now clone your source systems data to /mnt/local/ with whatever utility you like to use and exit the recovery step. After confirming everything went well, REAR will setup the bootloader (and all other config details like fstab entries and adjusted network configuration) for you as required:

rear> exit
Did you restore the backup to /mnt/local ? Are you ready to continue recovery ? yes
User confirmed restored files
Updated initramfs with new drivers for this system.
Skip installing GRUB Legacy boot loader because GRUB 2 is installed (grub-probe or grub2-probe exist).
Installing GRUB2 boot loader...
Determining where to install GRUB2 (no GRUB2_INSTALL_DEVICES specified)
Found possible boot disk /dev/sda - installing GRUB2 there
Finished 'recover'. The target system is mounted at '/mnt/local'.
Exiting rear recover (PID 7103) and its descendant processes ...
Running exit tasks

Now reboot the recovery console and watch it boot into your target systems configuration: Being able to use this procedure for complete disaster recovery within Hetzner cloud VPS (using off-site backups) gives me a better feeling, too.

• Clone the virtual machine and test the upgrade.
• Create a snapshot beforehand, do an in-place upgrade and hope everything works. If not, revert the snapshot..
• Combine both methods
• Use LVM or filesystem snapshots (snapper, etc)

• Issue an backup-begin statement and freeze the virtual machines file systems during this process (using qemu-agent) so you get an consistent system state.
• Create an qcow2 image with backing image option that points to the created (readonly) NBD export via unix socket as overlay.
• Use the created overlay image to boot up the instance and test the system upgrade.
• Do so while the virtual machine is operating without any downtime.

# virsh list --all
 Id Name State 
 ------------------------------------------ 
 1 debian11_default running

# virtnbdbackup -d debian11_default -o /tmp/foo -s
INFO lib common - printVersion [MainThread]: Version: 1.9.45 Arguments: ./virtnbdbackup -d debian11_default -o /tmp/foo -s
[..] 
INFO root virtnbdbackup - main [MainThread]: Local NBD Endpoint socket: [/var/tmp/virtnbdbackup.5727] 
INFO root virtnbdbackup - startBackupJob [MainThread]: Starting backup job.
INFO fs fs - freeze [MainThread]: Freezed [2] filesystems. 
INFO fs fs - thaw [MainThread]: Thawed [2] filesystems. 
INFO root virtnbdbackup - main [MainThread]: Started backup job for debugging, exiting.

# nbdinfo "nbd+unix:///vda?socket=/var/tmp/virtnbdbackup.5727" 
    protocol: newstyle-fixed without TLS, using structured packets 
    export="vda": 
    export-size: 137438953472 (128G) 
    content: 
        DOS/MBR boot sector uri: nbd+unix:///vda?socket=/var/tmp/virtnbdbackup.5727

# qemu-img create -F raw -b nbd+unix:///vda?socket=/var/tmp/virtnbdbackup.5727 -f qcow2 upgrade.qcow2

• Create another domain config in libvirt which points to this disk image, boot.
• Extract the current qemu command line for the the running domain using virsh domxml-to-native qemu-argv --domain debian11_default and execute it manually after adjusting required parameters.
• Trust for everything to work out in the simplest way and simply boot up the image via:

# qemu-system-x86_64 -hda upgrade.qcow2 -m 2500 --enable-kvm

# virtnbdbackup -d debian11_default -o /tmp/foo -k
INFO lib common - printVersion [MainThread]: Version: 1.9.45 Arguments: ./virtnbdbackup -d debian11_default -o /tmp/foo -k 
[..]
INFO root virtnbdbackup - main [MainThread]: Stopping backup job

# rm -f upgrade.qcow2

Welcome to the April 2023 report from the Reproducible Builds project! In these reports we outline the most important things that we have been up to over the past month. And, as always, if you are interested in contributing to the project, please visit our Contribute page on our website.

General news Trisquel is a fully-free operating system building on the work of Ubuntu Linux. This month, Simon Josefsson published an article on his blog titled Trisquel is 42% Reproducible!. Simon wrote:

The absolute number may not be impressive, but what I hope is at least a useful contribution is that there actually is a number on how much of Trisquel is reproducible. Hopefully this will inspire others to help improve the actual metric.

Simon wrote another blog post this month on a new tool to ensure that updates to Linux distribution archive metadata (eg. via apt-get update) will only use files that have been recorded in a globally immutable and tamper-resistant ledger. A similar solution exists for Arch Linux (called pacman-bintrans) which was announced in August 2021 where an archive of all issued signatures is publically accessible.
Joachim Breitner wrote an in-depth blog post on a bootstrap-capable GHC, the primary compiler for the Haskell programming language. As a quick background to what this is trying to solve, in order to generate a fully trustworthy compile chain, trustworthy root binaries are needed and a popular approach to address this problem is called bootstrappable builds where the core idea is to address previously-circular build dependencies by creating a new dependency path using simpler prerequisite versions of software. Joachim takes an somewhat recursive approach to the problem for Haskell, leading to the inadvertently humourous question: Can I turn all of GHC into one module, and compile that? Elsewhere in the world of bootstrapping, Janneke Nieuwenhuizen and Ludovic Court s wrote a blog post on the GNU Guix blog announcing The Full-Source Bootstrap, specifically:

[ ] the third reduction of the Guix bootstrap binaries has now been merged in the main branch of Guix! If you run guix pull today, you get a package graph of more than 22,000 nodes rooted in a 357-byte program something that had never been achieved, to our knowledge, since the birth of Unix.

More info about this change is available on the post itself, including:

The full-source bootstrap was once deemed impossible. Yet, here we are, building the foundations of a GNU/Linux distro entirely from source, a long way towards the ideal that the Guix project has been aiming for from the start. There are still some daunting tasks ahead. For example, what about the Linux kernel? The good news is that the bootstrappable community has grown a lot, from two people six years ago there are now around 100 people in the #bootstrappable IRC channel.

Michael Ablassmeier created a script called pypidiff as they were looking for a way to track differences between packages published on PyPI. According to Micahel, pypidiff uses diffoscope to create reports on the published releases and automatically pushes them to a GitHub repository. This can be seen on the pypi-diff GitHub page (example).
Eleuther AI, a non-profit AI research group, recently unveiled Pythia, a collection of 16 Large Language Model (LLMs) trained on public data in the same order designed specifically to facilitate scientific research. According to a post on MarkTechPost:

Pythia is the only publicly available model suite that includes models that were trained on the same data in the same order [and] all the corresponding data and tools to download and replicate the exact training process are publicly released to facilitate further research.

These properties are intended to allow researchers to understand how gender bias (etc.) can affected by training data and model scale.
Back in February s report we reported on a series of changes to the Sphinx documentation generator that was initiated after attempts to get the alembic Debian package to build reproducibly. Although Chris Lamb was able to identify the source problem and provided a potential patch that might fix it, James Addison has taken the issue in hand, leading to a large amount of activity resulting in a proposed pull request that is waiting to be merged.
WireGuard is a popular Virtual Private Network (VPN) service that aims to be faster, simpler and leaner than other solutions to create secure connections between computing devices. According to a post on the WireGuard developer mailing list, the WireGuard Android app can now be built reproducibly so that its contents can be publicly verified. According to the post by Jason A. Donenfeld, the F-Droid project now does this verification by comparing their build of WireGuard to the build that the WireGuard project publishes. When they match, the new version becomes available. This is very positive news.
Author and public speaker, V. M. Brasseur published a sample chapter from her upcoming book on corporate open source strategy which is the topic of Software Bill of Materials (SBOM):

A software bill of materials (SBOM) is defined as a nested inventory for software, a list of ingredients that make up software components. When you receive a physical delivery of some sort, the bill of materials tells you what s inside the box. Similarly, when you use software created outside of your organisation, the SBOM tells you what s inside that software. The SBOM is a file that declares the software supply chain (SSC) for that specific piece of software. [ ]

Several distributions noticed recent versions of the Linux Kernel are no longer reproducible because the BPF Type Format (BTF) metadata is not generated in a deterministic way. This was discussed on the #reproducible-builds IRC channel, but no solution appears to be in sight for now.

---

Community news On our mailing list this month:

• Larry Doolittle shared an interesting puzzle with the group where three bytes in a .zip file were different between two builds.
• Alexis PM wrote a message as they had observed a difference between binaries available in the Debian archive and the ones on tests.reproducible-builds.org. The thread generated a number of replies, including interesting responses from Vagrant Cascadian [ ] and kpcyrd [ ].

Holger Levsen gave a talk at foss-north 2023 in Gothenburg, Sweden on the topic of Reproducible Builds, the first ten years. Lastly, there were a number of updates to our website, including:

• Chris Lamb attempted a number of ways to try and fix literal : .lead appearing in the page [ ][ ][ ], made all the Back to who is involved links italics [ ], and corrected the syntax of the _data/sponsors.yml file [ ].
• Holger Levsen added his recent talk [ ], added Simon Josefsson, Mike Perry and Seth Schoen to the contributors page [ ][ ][ ], reworked the People page a little [ ] [ ], as well as fixed spelling of Arch Linux [ ].

Lastly, Mattia Rizzolo moved some old sponsors to a former section [ ] and Simon Josefsson added Trisquel GNU/Linux. [ ]

---

Debian

• Vagrant Cascadian reported on the Debian s build-essential package set, which was inspired by how close we are to making the Debian build-essential set reproducible and how important that set of packages are in general . Vagrant mentioned that: I have some progress, some hope, and I daresay, some fears . [ ]
• Debian Developer Cyril Brulebois (kibi) filed a bug against snapshot.debian.org after they noticed that there are many missing dinstalls that is to say, the snapshot service is not capturing 100% of all of historical states of the Debian archive. This is relevant to reproducibility because without the availability historical versions, it is becomes impossible to repeat a build at a future date in order to correlate checksums. .
• 20 reviews of Debian packages were added, 21 were updated and 5 were removed this month adding to our knowledge about identified issues. Chris Lamb added a new build_path_in_line_annotations_added_by_ruby_ragel toolchain issue. [ ]
• Mattia Rizzolo announced that the data for the stretch archive on tests.reproducible-builds.org has been archived. This matches the archival of stretch within Debian itself. This is of some historical interest, as stretch was the first Debian release regularly tested by the Reproducible Builds project.

---

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:
  • ghc (workaround a parallelism-related issue)
• Jan Zerebecki:
  • ghc (report a parallelism-related issue)
• Chris Lamb:
  • #1034147 filed against ruby-regexp-parser.
• Vagrant Cascadian:
  • #1033954, #1033955 and #1033957 filed against pike8.0.
  • #1033958 and #1033959 filed against binutils.
  • #1034129 filed against lomiri-action-api.
  • #1034199 and #1034200 filed against lomiri.
  • #1034327 filed against nmodl.
  • #1034423 filed against php8.2.
  • #1034431 filed against qemu.
  • #1034499 filed against twisted.
  • #1034740 filed against boost1.74.
  • #1034892 filed against php8.2.
  • #1035324 filed against shaderc.
  • #1035329 and #1035331 filed against jackd2.

---

diffoscope development diffoscope version 241 was uploaded to Debian unstable by Chris Lamb. It included contributions already covered in previous months as well a change by Chris Lamb to add a missing raise statement that was accidentally dropped in a previous commit. [ ]

---

Testing framework The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In April, a number of changes were made, including:

• Holger Levsen:
  • Significant work on a new Documented Jenkins Maintenance (djm) script to support logged maintenance of nodes, etc. [ ][ ][ ][ ][ ][ ]
  • Add the new APT repo url for Jenkins itself with a new signing key. [ ][ ]
  • In the Jenkins shell monitor, allow 40 GiB of files for diffoscope for the Debian experimental distribution as Debian is frozen around the release at the moment. [ ]
  • Updated Arch Linux testing to cleanup leftover files left in /tmp/archlinux-ci/ after three days. [ ][ ][ ]
  • Mark a number of nodes hosted by Oregon State University Open Source Lab (OSUOSL) as online and offline. [ ][ ][ ]
  • Update the node health checks to detect failures to end schroot sessions. [ ]
  • Filter out another duplicate contributor from the contributor statistics. [ ]
• Mattia Rizzolo:
  • Code changes to properly archive the stretch Debian distribution. [ ][ ][ ][ ]
  • Introduce the archived suites configuration option. [ ]][ ]
  • Fix the KGB bot configuration to support pyyaml 6.0 as present in Debian bookworm. [ ]

---

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Twitter: @ReproBuilds
• Mailing list: rb-general@lists.reproducible-builds.org

virtnbdbackup -U qemu+ssh://usr@hypervisor/system -d vm1 -o /backup/vm1

• New backup mode auto which allows easy backup rotation.
• Option to freeze only specific filesystems within backed up domain.
• Remote backup via dedicated network: use --nbd-ip to bind the remote NDB service to an specific interface.
• If virtual machine requires additional files like specific UEFI/Kernel image, these are saved via SFTP from the remote host, too.
• Restore operation can now adjust domain config accordingly (and redefine it if desired).

• backing up virtual domains UEFI bios, initrd and kernel images if defined.
• virtnbdmap now uses the nbdkit COW plugin to map the backups as regular NBD device. This allows users to replay complete backup chains (full+inc/diff) to recover single files. Also makes the mapped device writable, as such one can directly boot the virtual machine from the backup images.

• Added backup mode differencial.
• Save virtual domains UEFI bios, initrd and kernel images if defined.
• virtnbdmap now uses the nbdkit COW plugin to map the backups as regular NBD device. This allows users to replay complete backup chains (full+inc/diff) to recover single files. As the resulting device is writable, one can directly boot the virtual machine from the backup images.

#!/usr/bin/python3
import sys
import logging
def warnme():
    # something bad happens
    logging.warning("warning")
    sys.exit(2)
def evil():
    # something evil happens
    logging.error("error")
    sys.exit(1)
def main():
    logging.basicConfig(
        level=logging.DEBUG,
    )   
    [..]

• exit code 0 on success
• exit code 1 on errors
• exit code 2 on warnings
• warnings or errors shall be logged in the function where they actually happen: the logging module will show the function name with a better format option: nice for debugging.
• one function that exits accordingly, preferrably main()

How to do better? As the application is using the logging module, we have a single point to collect warnings and errors that might happen accross all modules. This works by passing a custom handler to the logging module which tracks emitted messages. Heres an small example:

#!/usr/bin/python3
import sys
import logging
class logCount(logging.Handler):
    class LogType:
        def __init__(self):
            self.warnings = 0
            self.errors = 0
    def __init__(self):
        super().__init__()
        self.count = self.LogType()
    def emit(self, record):
        if record.levelname == "WARNING":
            self.count.warnings += 1
        if record.levelname == "ERROR":
            self.count.errors += 1
            
def infome():
    logging.info("hello world")
def warnme():
    logging.warning("help, an warning")
def evil():
    logging.error("yikes")
def main():
    EXIT_WARNING = 2
    EXIT_ERROR = 1
    counter = logCount()
    logging.basicConfig(
        level=logging.DEBUG,
        handlers=[counter, logging.StreamHandler(sys.stderr)],
    )
    infome()
    warnme()
    evil()
    if counter.count.errors != 0:
        raise SystemExit(EXIT_ERROR)
    if counter.count.warnings != 0:
        raise SystemExit(EXIT_WARNING)
if __name__ == "__main__":
    main()

python3 count.py ; echo $?
INFO:root:hello world
WARNING:root:help, an warning
ERROR:root:yikes
1

This also makes easy to define something like:

• hey, got 2 warnings, change exit code to error?
• got 3 warnings, but no strict passed, ingore those, exit with success!
• etc..

• Thin provisioned full and incremental backups.
• Compression (lz4).
• Freezing the virtual machine guest fs via qemu-agent (if installed)
• Multithreaded backup, if multiple disks are attached.
• Point in time recovery to a given incremental backup.
• Includes nbdkit plugin and utility that allows to map the thin provisioned backup images to a block device, for single file or instant recovery (boot the virtual machine directly from the backup image)

 <domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
 [..]
 <qemu:capabilities>
   <qemu:add capability='incremental-backup'/>
 </qemu:capabilities>
 [..]
 </domain>

# virtnbdbackup -d vm2 -l full -o /tmp/WEEKLY_BACKUP/
 [..]
[2021-11-08 15:43:46] INFO virtnbdbackup - main [MainThread]: Backup jobs finished, stopping backup task.
[2021-11-08 15:43:46] INFO virtnbdbackup - main [MainThread]: Finished
# tree /tmp/WEEKLY_BACKUP/
 /tmp/WEEKLY_BACKUP/
  backup.full.11082021154343.log
  checkpoints
    virtnbdbackup.0.xml
  sda.full.data
  vm2.cpt
  vmconfig.virtnbdbackup.0.xml

 # virtnbdbackup -d vm2 -l inc -o /tmp/WEEKLY_BACKUP/
 # tree /tmp/WEEKLY_BACKUP/
 /tmp/WEEKLY_BACKUP/
  backup.full.11082021154343.log
  backup.inc.11082021154538.log
  checkpoints
    virtnbdbackup.0.xml
    virtnbdbackup.1.xml
  sda.full.data
  sda.inc.virtnbdbackup.1.data
  vm2.cpt
  vmconfig.virtnbdbackup.0.xml
  vmconfig.virtnbdbackup.1.xml

 # virtnbdrestore -a restore -i /tmp/WEEKLY_BACKUP/ -o /tmp/VM_RESTORE

# virtnbdmap -f /tmp/WEEKLY_BACKUP/sda.full.data
 [..] INFO virtnbdmap - <module> [MainThread]: Done mapping backup image to [/dev/nbd0]
 [..] INFO virtnbdmap - <module> [MainThread]: Press CTRL+C to disconnect
# fdisk -l /dev/nbd0
Disk /dev/nbd0: 2 GiB, 2147483648 bytes, 4194304 sectors

qemu-img create -b /dev/nbd0 -f qcow2 bootme.qcow2
qemu-system-x86_64 -enable-kvm -m 2000 -hda bootme.qcow2

• Pierre Habouzit with #395522 (whitelister terminates immediately)
• Josselin Mouette with #396247 (sound-juicer: weird playback if disk is ejected)
• Christian Perrier with #396223 (gnome-applets-data: Colorado Springs listed as "Colourado Springs" in British English)
• Joey Hess with #393846 (wave at a webcam to overwrite file)
• Michael Ablassmeier with #397646 (exim4-config: reportbug mail issue)

• #393846 (wave at a webcam to overwrite file) (with 12 points)
• #396223 (gnome-applets-data: Colorado Springs listed as "Colourado Springs" in British English) (with 10 points)
• #396247 (sound-juicer: weird playback if disk is ejected) (with 9 points)
• #397646 (exim4-config: reportbug mail issue) (with 9 points)
• #395522 (whitelister terminates immediately) (with 3 points)

• Joey Hess (12 points with 1 bug)
• Christian Perrier (10 points with 1 bug)
• Michael Ablassmeier (9 points with 1 bug) and Josselin Mouette (9 points with 1 bug)
• Pierre Habouzit (3 points with 1 bug)

• Bug: #393846 (wave at a webcam to overwrite file) (with 12 points)
• Challenger: Joey Hess
• Bug reporter: Joey Hess
• Voter: David Pashley with the nearest ordered tierc : #393846, #396223 and #397646.