In case you are not yet familiar with why an initramfs (or initrd, or initial ramdisk) is typically used when starting Linux, let me quote the wikipedia definition: [ ] initrd is a scheme for loading a temporary root file system into memory, which may be used as part of the Linux startup process [ ] to make preparations before the real root file system can be mounted. Many Linux distributions do not compile all file system drivers into the kernel, but instead load them on-demand from an initramfs, which saves memory. Another common scenario, in which an initramfs is required, is full-disk encryption: the disk must be unlocked from userspace, but since userspace is encrypted, an initramfs is used.

Motivation Thus far, building a distri disk image was quite slow: This is on an AMD Ryzen 3900X 12-core processor (2019):

distri % time make cryptimage serial=1
80.29s user 13.56s system 186% cpu 50.419 total # 19s image, 31s initrd

Of these 50 seconds, dracut s initramfs generation accounts for 31 seconds (62%)! Initramfs generation time drops to 8.7 seconds once dracut no longer needs to use the single-threaded gzip(1) , but the multi-threaded replacement pigz(1) : This brings the total time to build a distri disk image down to:

distri % time make cryptimage serial=1
76.85s user 13.23s system 327% cpu 27.509 total # 19s image, 8.7s initrd

Clearly, when you use dracut on any modern computer, you should make pigz available. dracut should fail to compile unless one explicitly opts into the known-slower gzip. For more thoughts on optional dependencies, see Optional dependencies don t work . But why does it take 8.7 seconds still? Can we go faster? The answer is Yes! I recently built a distri-specific initramfs I m calling minitrd. I wrote both big parts from scratch:

1. the initramfs generator program (distri initrd)
2. a custom Go userland (cmd/minitrd), running as /init in the initramfs.

minitrd generates the initramfs image in 400ms, bringing the total time down to:

distri % time make cryptimage serial=1
50.09s user 8.80s system 314% cpu 18.739 total # 18s image, 400ms initrd

(The remaining time is spent in preparing the file system, then installing and configuring the distri system, i.e. preparing a disk image you can run on real hardware.) How can minitrd be 20 times faster than dracut? dracut is mainly written in shell, with a C helper program. It drives the generation process by spawning lots of external dependencies (e.g. ldd or the dracut-install helper program). I assume that the combination of using an interpreted language (shell) that spawns lots of processes and precludes a concurrent architecture is to blame for the poor performance. minitrd is written in Go, with speed as a goal. It leverages concurrency and uses no external dependencies; everything happens within a single process (but with enough threads to saturate modern hardware). Measuring early boot time using qemu, I measured the dracut-generated initramfs taking 588ms to display the full disk encryption passphrase prompt, whereas minitrd took only 195ms. The rest of this article dives deeper into how minitrd works.

What does an initramfs do? Ultimately, the job of an initramfs is to make the root file system available and continue booting the system from there. Depending on the system setup, this involves the following 5 steps:

1. Load kernel modules to access the block devices with the root file system Depending on the system, the block devices with the root file system might already be present when the initramfs runs, or some kernel modules might need to be loaded first. On my Dell XPS 9360 laptop, the NVMe system disk is already present when the initramfs starts, whereas in qemu, we need to load the virtio_pci module, followed by the virtio_scsi module. How will our userland program know which kernel modules to load? Linux kernel modules declare patterns for their supported hardware as an alias, e.g.:

initrd# grep virtio_pci lib/modules/5.4.6/modules.alias
alias pci:v00001AF4d*sv*sd*bc*sc*i* virtio_pci

Devices in sysfs have a modalias file whose content can be matched against these declarations to identify the module to load:

initrd# cat /sys/devices/pci0000:00/*/modalias
pci:v00001AF4d00001005sv00001AF4sd00000004bc00scFFi00
pci:v00001AF4d00001004sv00001AF4sd00000008bc01sc00i00
[ ]

Hence, for the initial round of module loading, it is sufficient to locate all modalias files within sysfs and load the responsible modules. Loading a kernel module can result in new devices appearing. When that happens, the kernel sends a uevent, which the uevent consumer in userspace receives via a netlink socket. Typically, this consumer is udev(7) , but in our case, it s minitrd. For each uevent messages that comes with a MODALIAS variable, minitrd will load the relevant kernel module(s). When loading a kernel module, its dependencies need to be loaded first. Dependency information is stored in the modules.dep file in a Makefile-like syntax:

initrd# grep virtio_pci lib/modules/5.4.6/modules.dep
kernel/drivers/virtio/virtio_pci.ko: kernel/drivers/virtio/virtio_ring.ko kernel/drivers/virtio/virtio.ko

To load a module, we can open its file and then call the Linux-specific finit_module(2) system call. Some modules are expected to return an error code, e.g. ENODEV or ENOENT when some hardware device is not actually present. Side note: next to the textual versions, there are also binary versions of the modules.alias and modules.dep files. Presumably, those can be queried more quickly, but for simplicitly, I have not (yet?) implemented support in minitrd.

2. Console settings: font, keyboard layout Setting a legible font is necessary for hi-dpi displays. On my Dell XPS 9360 (3200 x 1800 QHD+ display), the following works well:

initrd# setfont latarcyrheb-sun32

Setting the user s keyboard layout is necessary for entering the LUKS full-disk encryption passphrase in their preferred keyboard layout. I use the NEO layout:

initrd# loadkeys neo

3. Block device identification In the Linux kernel, block device enumeration order is not necessarily the same on each boot. Even if it was deterministic, device order could still be changed when users modify their computer s device topology (e.g. connect a new disk to a formerly unused port). Hence, it is good style to refer to disks and their partitions with stable identifiers. This also applies to boot loader configuration, and so most distributions will set a kernel parameter such as root=UUID=1fa04de7-30a9-4183-93e9-1b0061567121. Identifying the block device or partition with the specified UUID is the initramfs s job. Depending on what the device contains, the UUID comes from a different place. For example, ext4 file systems have a UUID field in their file system superblock, whereas LUKS volumes have a UUID in their LUKS header. Canonically, probing a device to extract the UUID is done by libblkid from the util-linux package, but the logic can easily be re-implemented in other languages and changes rarely. minitrd comes with its own implementation to avoid cgo or running the blkid(8) program.

4. LUKS full-disk encryption unlocking (only on encrypted systems) Unlocking a LUKS-encrypted volume is done in userspace. The kernel handles the crypto, but reading the metadata, obtaining the passphrase (or e.g. key material from a file) and setting up the device mapper table entries are done in user space.

initrd# modprobe algif_skcipher
initrd# cryptsetup luksOpen /dev/sda4 cryptroot1

After the user entered their passphrase, the root file system can be mounted:

initrd# mount /dev/dm-0 /mnt

5. Continuing the boot process (switch_root) Now that everything is set up, we need to pass execution to the init program on the root file system with a careful sequence of chdir(2) , mount(2) , chroot(2) , chdir(2) and execve(2) system calls that is explained in this busybox switch_root comment.

initrd# mount -t devtmpfs dev /mnt/dev
initrd# exec switch_root -c /dev/console /mnt /init

To conserve RAM, the files in the temporary file system to which the initramfs archive is extracted are typically deleted.

How is an initramfs generated? An initramfs image (more accurately: archive) is a compressed cpio archive. Typically, gzip compression is used, but the kernel supports a bunch of different algorithms and distributions such as Ubuntu are switching to lz4. Generators typically prepare a temporary directory and feed it to the cpio(1) program. In minitrd, we read the files into memory and generate the cpio archive using the go-cpio package. We use the pgzip package for parallel gzip compression. The following files need to go into the cpio archive:

minitrd Go userland The minitrd binary is copied into the cpio archive as /init and will be run by the kernel after extracting the archive. Like the rest of distri, minitrd is built statically without cgo, which means it can be copied as-is into the cpio archive.

Linux kernel modules Aside from the modules.alias and modules.dep metadata files, the kernel modules themselves reside in e.g. /lib/modules/5.4.6/kernel and need to be copied into the cpio archive. Copying all modules results in a 80 MiB archive, so it is common to only copy modules that are relevant to the initramfs s features. This reduces archive size to 24 MiB. The filtering relies on hard-coded patterns and module names. For example, disk encryption related modules are all kernel modules underneath kernel/crypto, plus kernel/drivers/md/dm-crypt.ko. When generating a host-only initramfs (works on precisely the computer that generated it), some initramfs generators look at the currently loaded modules and just copy those.

Console Fonts and Keymaps The kbd package s setfont(8) and loadkeys(1) programs load console fonts and keymaps from /usr/share/consolefonts and /usr/share/keymaps, respectively. Hence, these directories need to be copied into the cpio archive. Depending on whether the initramfs should be generic (work on many computers) or host-only (works on precisely the computer/settings that generated it), the entire directories are copied, or only the required font/keymap.

cryptsetup, setfont, loadkeys These programs are (currently) required because minitrd does not implement their functionality. As they are dynamically linked, not only the programs themselves need to be copied, but also the ELF dynamic linking loader (path stored in the .interp ELF section) and any ELF library dependencies. For example, cryptsetup in distri declares the ELF interpreter /ro/glibc-amd64-2.27-3/out/lib/ld-linux-x86-64.so.2 and declares dependencies on shared libraries libcryptsetup.so.12, libblkid.so.1 and others. Luckily, in distri, packages contain a lib subdirectory containing symbolic links to the resolved shared library paths (hermetic packaging), so it is sufficient to mirror the lib directory into the cpio archive, recursing into shared library dependencies of shared libraries. cryptsetup also requires the GCC runtime library libgcc_s.so.1 to be present at runtime, and will abort with an error message about not being able to call pthread_cancel(3) if it is unavailable.

time zone data To print log messages in the correct time zone, we copy /etc/localtime from the host into the cpio archive.

minitrd outside of distri? I currently have no desire to make minitrd available outside of distri. While the technical challenges (such as extending the generator to not rely on distri s hermetic packages) are surmountable, I don t want to support people s initramfs remotely. Also, I think that people s efforts should in general be spent on rallying behind dracut and making it work faster, thereby benefiting all Linux distributions that use dracut (increasingly more). With minitrd, I have demonstrated that significant speed-ups are achievable.

Conclusion It was interesting to dive into how an initramfs really works. I had been working with the concept for many years, from small tasks such as debug why the encrypted root file system is not unlocked to more complicated tasks such as set up a root file system on DRBD for a high-availability setup . But even with that sort of experience, I didn t know all the details, until I was forced to implement every little thing. As I suspected going into this exercise, dracut is much slower than it needs to be. Re-implementing its generation stage in a modern language instead of shell helps a lot. Of course, my minitrd does a bit less than dracut, but not drastically so. The overall architecture is the same. I hope my effort helps with two things:

1. As a teaching implementation: instead of wading through the various components that make up a modern initramfs (udev, systemd, various shell scripts, ), people can learn about how an initramfs works in a single place.
2. I hope the significant time difference motivates people to improve dracut.

Appendix: qemu development environment Before writing any Go code, I did some manual prototyping. Learning how other people prototype is often immensely useful to me, so I m sharing my notes here. First, I copied all kernel modules and a statically built busybox binary:

% mkdir -p lib/modules/5.4.6
% cp -Lr /ro/lib/modules/5.4.6/* lib/modules/5.4.6/
% cp ~/busybox-1.22.0-amd64/busybox sh

To generate an initramfs from the current directory, I used:

% find .   cpio -o -H newc   pigz > /tmp/initrd

In distri s Makefile, I append these flags to the QEMU invocation:

-kernel /tmp/kernel \
-initrd /tmp/initrd \
-append "root=/dev/mapper/cryptroot1 rdinit=/sh ro console=ttyS0,115200 rd.luks=1 rd.luks.uuid=63051f8a-54b9-4996-b94f-3cf105af2900 rd.luks.name=63051f8a-54b9-4996-b94f-3cf105af2900=cryptroot1 rd.vconsole.keymap=neo rd.vconsole.font=latarcyrheb-sun32 init=/init systemd.setenv=PATH=/bin rw vga=836"

The vga= mode parameter is required for loading font latarcyrheb-sun32. Once in the busybox shell, I manually prepared the required mount points and kernel modules:

ln -s sh mount
ln -s sh lsmod
mkdir /proc /sys /run /mnt
mount -t proc proc /proc
mount -t sysfs sys /sys
mount -t devtmpfs dev /dev
modprobe virtio_pci
modprobe virtio_scsi

As a next step, I copied cryptsetup and dependencies into the initramfs directory:

% for f in /ro/cryptsetup-amd64-2.0.4-6/lib/*; do full=$(readlink -f $f); rel=$(echo $full   sed 's,^/,,g'); mkdir -p $(dirname $rel); install $full $rel; done
% ln -s ld-2.27.so ro/glibc-amd64-2.27-3/out/lib/ld-linux-x86-64.so.2
% cp /ro/glibc-amd64-2.27-3/out/lib/ld-2.27.so ro/glibc-amd64-2.27-3/out/lib/ld-2.27.so
% cp -r /ro/cryptsetup-amd64-2.0.4-6/lib ro/cryptsetup-amd64-2.0.4-6/
% mkdir -p ro/gcc-libs-amd64-8.2.0-3/out/lib64/
% cp /ro/gcc-libs-amd64-8.2.0-3/out/lib64/libgcc_s.so.1 ro/gcc-libs-amd64-8.2.0-3/out/lib64/libgcc_s.so.1
% ln -s /ro/gcc-libs-amd64-8.2.0-3/out/lib64/libgcc_s.so.1 ro/cryptsetup-amd64-2.0.4-6/lib
% cp -r /ro/lvm2-amd64-2.03.00-6/lib ro/lvm2-amd64-2.03.00-6/

In busybox, I used the following commands to unlock the root file system:

modprobe algif_skcipher
./cryptsetup luksOpen /dev/sda4 cryptroot1
mount /dev/dm-0 /mnt

In distri, packages (e.g. emacs) are hermetic. By hermetic, I mean that the dependencies a package uses (e.g. libusb) don t change, even when newer versions are installed. For example, if package libusb-amd64-1.0.22-7 is available at build time, the package will always use that same version, even after the newer libusb-amd64-1.0.23-8 will be installed into the package store. Another way of saying the same thing is: packages in distri are always co-installable. This makes the package store more robust: additions to it will not break the system. On a technical level, the package store is implemented as a directory containing distri SquashFS images and metadata files, into which packages are installed in an atomic way.

Out of scope: plugins are not hermetic by design One exception where hermeticity is not desired are plugin mechanisms: optionally loading out-of-tree code at runtime obviously is not hermetic. As an example, consider glibc s Name Service Switch (NSS) mechanism. Page 29.4.1 Adding another Service to NSS describes how glibc searches $prefix/lib for shared libraries at runtime. Debian ships about a dozen NSS libraries for a variety of purposes, and enterprise setups might add their own into the mix. systemd (as of v245) accounts for 4 NSS libraries, e.g. nss-systemd for user/group name resolution for users allocated through systemd s DynamicUser= option. Having packages be as hermetic as possible remains a worthwhile goal despite any exceptions: I will gladly use a 99% hermetic system over a 0% hermetic system any day. Side note: Xorg s driver model (which can be characterized as a plugin mechanism) does not fall under this category because of its tight API/ABI coupling! For this case, where drivers are only guaranteed to work with precisely the Xorg version for which they were compiled, distri uses per-package exchange directories.

Implementation of hermetic packages in distri On a technical level, the requirement is: all paths used by the program must always result in the same contents. This is implemented in distri via the read-only package store mounted at /ro, e.g. files underneath /ro/emacs-amd64-26.3-15 never change. To change all paths used by a program, in practice, three strategies cover most paths:

ELF interpreter and dynamic libraries Programs on Linux use the ELF file format, which contains two kinds of references: First, the ELF interpreter (PT_INTERP segment), which is used to start the program. For dynamically linked programs on 64-bit systems, this is typically ld.so(8). Many distributions use system-global paths such as /lib64/ld-linux-x86-64.so.2, but distri compiles programs with -Wl,--dynamic-linker=/ro/glibc-amd64-2.31-4/out/lib/ld-linux-x86-64.so.2 so that the full path ends up in the binary. The ELF interpreter is shown by file(1), but you can also use readelf -a $BINARY grep 'program interpreter' to display it. And secondly, the rpath, a run-time search path for dynamic libraries. Instead of storing full references to all dynamic libraries, we set the rpath so that ld.so(8) will find the correct dynamic libraries. Originally, we used to just set a long rpath, containing one entry for each dynamic library dependency. However, we have since switched to using a single lib subdirectory per package as its rpath, and placing symlinks with full path references into that lib directory, e.g. using -Wl,-rpath=/ro/grep-amd64-3.4-4/lib. This is better for performance, as ld.so uses a per-directory cache. Note that program load times are significantly influenced by how quickly you can locate the dynamic libraries. distri uses a FUSE file system to load programs from, so getting proper -ENOENT caching into place drastically sped up program load times. Instead of compiling software with the -Wl,--dynamic-linker and -Wl,-rpath flags, one can also modify these fields after the fact using patchelf(1). For closed-source programs, this is the only possibility. The rpath can be inspected by using e.g. readelf -a $BINARY grep RPATH.

Environment variable setup wrapper programs Many programs are influenced by environment variables: to start another program, said program is often found by checking each directory in the PATH environment variable. Such search paths are prevalent in scripting languages, too, to find modules. Python has PYTHONPATH, Perl has PERL5LIB, and so on. To set up these search path environment variables at run time, distri employs an indirection. Instead of e.g. teensy-loader-cli, you run a small wrapper program that calls precisely one execve system call with the desired environment variables. Initially, I used shell scripts as wrapper programs because they are easily inspectable. This turned out to be too slow, so I switched to compiled programs. I m linking them statically for fast startup, and I m linking them against musl libc for significantly smaller file sizes than glibc (per-executable overhead adds up quickly in a distribution!). Note that the wrapper programs prepend to the PATH environment variable, they don t replace it in its entirely. This is important so that users have a way to extend the PATH (and other variables) if they so choose. This doesn t hurt hermeticity because it is only relevant for programs that were not present at build time, i.e. plugin mechanisms which, by design, cannot be hermetic.

Shebang interpreter patching The Shebang of scripts contains a path, too, and hence needs to be changed. We don t do this in distri yet (the number of packaged scripts is small), but we should.

Performance requirements The performance improvements in the previous sections are not just good to have, but practically required when many processes are involved: without them, you ll encounter second-long delays in magit which spawns many git processes under the covers, or in dracut, which spawns one cp(1) process per file.

Downside: rebuild of packages required to pick up changes Linux distributions such as Debian consider it an advantage to roll out security fixes to the entire system by updating a single shared library package (e.g. openssl). The flip side of that coin is that changes to a single critical package can break the entire system. With hermetic packages, all reverse dependencies must be rebuilt when a library s changes should be picked up by the whole system. E.g., when openssl changes, curl must be rebuilt to pick up the new version of openssl. This approach trades off using more bandwidth and more disk space (temporarily) against reducing the blast radius of any individual package update.

Downside: long env variables are cumbersome to deal with This can be partially mitigated by removing empty directories at build time, which will result in shorter variables. In general, there is no getting around this. One little trick is to use tr : '\n', e.g.:

distri0# echo $PATH
/usr/bin:/bin:/usr/sbin:/sbin:/ro/openssh-amd64-8.2p1-11/out/bin
distri0# echo $PATH   tr : '\n'
/usr/bin
/bin
/usr/sbin
/sbin
/ro/openssh-amd64-8.2p1-11/out/bin

Edge cases The implementation outlined above works well in hundreds of packages, and only a small handful exhibited problems of any kind. Here are some issues I encountered:

Issue: accidental ABI breakage in plugin mechanisms NSS libraries built against glibc 2.28 and newer cannot be loaded by glibc 2.27. In all likelihood, such changes do not happen too often, but it does illustrate that glibc s published interface spec is not sufficient for forwards and backwards compatibility. In distri, we could likely use a per-package exchange directory for glibc s NSS mechanism to prevent the above problem from happening in the future.

Issue: wrapper bypass when a program re-executes itself Some programs try to arrange for themselves to be re-executed outside of their current process tree. For example, consider building a program with the meson build system:

1. When meson first configures the build, it generates ninja files (think Makefiles) which contain command lines that run the meson --internal helper.
2. Once meson returns, ninja is called as a separate process, so it will not have the environment which the meson wrapper sets up. ninja then runs the previously persisted meson command line. Since the command line uses the full path to meson (not to its wrapper), it bypasses the wrapper.

Luckily, not many programs try to arrange for other process trees to run them. Here is a table summarizing how affected programs might try to arrange for re-execution, whether the technique results in a wrapper bypass, and what we do about it in distri:

technique to execute itself	uses wrapper	mitigation
run-time: find own basename in PATH	yes	wrapper program
compile-time: embed expected path	no; bypass!	configure or patch
run-time: argv[0] or /proc/self/exe	no; bypass!	patch

One might think that setting argv[0] to the wrapper location seems like a way to side-step this problem. We tried doing this in distri, but had to revert and go the other way.

Misc smaller issues

• Login shells are started by convention with a - character prepended to argv[0], so shells like bash or zsh cannot use wrapper programs.
• LDFLAGS leaked to pkgconfig (upstream reports)
• mozjs tries to run autoconf with the shell directly, but should use autoconf s wrapper

Appendix: Could other distributions adopt hermetic packages? At a very high level, adopting hermetic packages will require two steps:

1. Using fully qualified paths whose contents don t change (e.g. /ro/emacs-amd64-26.3-15) generally requires rebuilding programs, e.g. with --prefix set.
2. Once you use fully qualified paths you need to make the packages able to exchange data. distri solves this with exchange directories, implemented in the /ro file system which is backed by a FUSE daemon.

The first step is pretty simple, whereas the second step is where I expect controversy around any suggested mechanism.

Appendix: demo (in distri) This appendix contains commands and their outputs, run on upcoming distri version supersilverhaze, but verified to work on older versions, too. Large outputs have been collapsed and can be expanded by clicking on the output. The /bin directory contains symlinks for the union of all package s bin subdirectories:

distri0# readlink -f /bin/teensy_loader_cli

/ro/teensy-loader-cli-amd64-2.1+g20180927-7/bin/teensy_loader_cli

The wrapper program in the bin subdirectory is small:

distri0# ls -lh $(readlink -f /bin/teensy_loader_cli)

-rwxr-xr-x 1 root root 46K Apr 21 21:56 /ro/teensy-loader-cli-amd64-2.1+g20180927-7/bin/teensy_loader_cli

Wrapper programs execute quickly:

distri0# strace -fvy /bin/teensy_loader_cli & head cat -n

     1  execve("/bin/teensy_loader_cli", ["/bin/teensy_loader_cli"], ["USER=root", "LOGNAME=root", "HOME=/root", "PATH=/ro/bash-amd64-5.0-4/bin:/r"..., "SHELL=/bin/zsh", "TERM=screen.xterm-256color", "XDG_SESSION_ID=c1", "XDG_RUNTIME_DIR=/run/user/0", "DBUS_SESSION_BUS_ADDRESS=unix:pa"..., "XDG_SESSION_TYPE=tty", "XDG_SESSION_CLASS=user", "SSH_CLIENT=10.0.2.2 42556 22", "SSH_CONNECTION=10.0.2.2 42556 10"..., "SSHTTY=/dev/pts/0", "SHLVL=1", "PWD=/root", "OLDPWD=/root", "=/usr/bin/strace", "LD_LIBRARY_PATH=/ro/bash-amd64-5"..., "PERL5LIB=/ro/bash-amd64-5.0-4/ou"..., "PYTHONPATH=/ro/bash-amd64-5.b0-4/"...]) = 0
     2  arch_prctl(ARCH_SET_FS, 0x40c878)       = 0
     3  set_tid_address(0x40ca9c)               = 715
     4  brk(NULL)                               = 0x15b9000
     5  brk(0x15ba000)                          = 0x15ba000
     6  brk(0x15bb000)                          = 0x15bb000
     7  brk(0x15bd000)                          = 0x15bd000
     8  brk(0x15bf000)                          = 0x15bf000
     9  brk(0x15c1000)                          = 0x15c1000
    10  execve("/ro/teensy-loader-cli-amd64-2.1+g20180927-7/out/bin/teensy_loader_cli", ["/ro/teensy-loader-cli-amd64-2.1+"...], ["USER=root", "LOGNAME=root", "HOME=/root", "PATH=/ro/bash-amd64-5.0-4/bin:/r"..., "SHELL=/bin/zsh", "TERM=screen.xterm-256color", "XDG_SESSION_ID=c1", "XDG_RUNTIME_DIR=/run/user/0", "DBUS_SESSION_BUS_ADDRESS=unix:pa"..., "XDG_SESSION_TYPE=tty", "XDG_SESSION_CLASS=user", "SSH_CLIENT=10.0.2.2 42556 22", "SSH_CONNECTION=10.0.2.2 42556 10"..., "SSHTTY=/dev/pts/0", "SHLVL=1", "PWD=/root", "OLDPWD=/root", "=/usr/bin/strace", "LD_LIBRARY_PATH=/ro/bash-amd64-5"..., "PERL5LIB=/ro/bash-amd64-5.0-4/ou"..., "PYTHONPATH=/ro/bash-amd64-5.0-4/"...]) = 0

Confirm which ELF interpreter is set for a binary using readelf(1):

distri0# readelf -a /ro/teensy-loader-cli-amd64-2.1+g20180927-7/out/bin/teensy_loader_cli grep 'program interpreter'

[Requesting program interpreter: /ro/glibc-amd64-2.31-4/out/lib/ld-linux-x86-64.so.2]

Confirm the rpath is set to the package s lib subdirectory using readelf(1):

distri0# readelf -a /ro/teensy-loader-cli-amd64-2.1+g20180927-7/out/bin/teensy_loader_cli grep RPATH

 0x000000000000000f (RPATH)              Library rpath: [/ro/teensy-loader-cli-amd64-2.1+g20180927-7/lib]

and verify the lib subdirectory has the expected symlinks and target versions:

distri0# find /ro/teensy-loader-cli-amd64-*/lib -type f -printf '%P -> %l\n'

libc.so.6 -> /ro/glibc-amd64-2.31-4/out/lib/libc-2.31.so

libpthread.so.0 -> /ro/glibc-amd64-2.31-4/out/lib/libpthread-2.31.so
librt.so.1 -> /ro/glibc-amd64-2.31-4/out/lib/librt-2.31.so
libudev.so.1 -> /ro/libudev-amd64-245-11/out/lib/libudev.so.1.6.17
libusb-0.1.so.4 -> /ro/libusb-compat-amd64-0.1.5-7/out/lib/libusb-0.1.so.4.4.4
libusb-1.0.so.0 -> /ro/libusb-amd64-1.0.23-8/out/lib/libusb-1.0.so.0.2.0

To verify the correct libraries are actually loaded, you can set the LD_DEBUG environment variable for ld.so(8):

distri0# LD_DEBUG=libs teensy_loader_cli

[ ]
       678:     find library=libc.so.6 [0]; searching
       678:      search path=/ro/teensy-loader-cli-amd64-2.1+g20180927-7/lib            (RPATH from file /ro/teensy-loader-cli-amd64-2.1+g20180927-7/out/bin/teensy_loader_cli)
       678:       trying file=/ro/teensy-loader-cli-amd64-2.1+g20180927-7/lib/libc.so.6
       678:
[ ]

NSS libraries that distri ships:

find /lib/ -name "libnss_*.so.2" -type f -printf '%P -> %l\n'

libnss_myhostname.so.2 -> ../systemd-amd64-245-11/out/lib/libnss_myhostname.so.2

libnss_mymachines.so.2 -> ../systemd-amd64-245-11/out/lib/libnss_mymachines.so.2
libnss_resolve.so.2 -> ../systemd-amd64-245-11/out/lib/libnss_resolve.so.2
libnss_systemd.so.2 -> ../systemd-amd64-245-11/out/lib/libnss_systemd.so.2
libnss_compat.so.2 -> ../glibc-amd64-2.31-4/out/lib/libnss_compat.so.2
libnss_db.so.2 -> ../glibc-amd64-2.31-4/out/lib/libnss_db.so.2
libnss_dns.so.2 -> ../glibc-amd64-2.31-4/out/lib/libnss_dns.so.2
libnss_files.so.2 -> ../glibc-amd64-2.31-4/out/lib/libnss_files.so.2
libnss_hesiod.so.2 -> ../glibc-amd64-2.31-4/out/lib/libnss_hesiod.so.2

What is GameMode, what does it do? About two years ago, I ran into some bugs running a game on Debian, so installed Windows 10 on a spare computer and ran it on there. I learned that when you launch a game in Windows 10, it automatically disables notifications, screensaver, reduces power saving measures and gives the game maximum priority. I thought Oh, that s actually quite nice, but we probably won t see that kind of integration on Linux any time soon . The very next week, I read the initial announcement of GameMode, a tool from Feral Interactive that does a bunch of tricks to maximise performance for games running on Linux. When GameMode is invoked it:

• Sets the kernel performance governor from powersave to performance
• Provides I/O priority to the game process
• Optionally sets nice value to the game process
• Inhibits the screensaver
• Tweak the kernel scheduler to enable soft real-time capabilities (handled by the MuQSS kernel scheduler, if available in your kernel)
• Sets GPU performance mode (NVIDIA and AMD)
• Attempts GPU overclocking (on supported NVIDIA cards)
• Runs custom pre/post run scripts. You might want to run a script to disable your ethereum mining or suspend VMs when you start a game and resume it all once you quit.

How GameMode is invoked Some newer games (proprietary games like Rise of the Tomb Raider , Total War Saga: Thrones of Britannia , Total War: WARHAMMER II , DiRT 4 and Total War: Three Kingdoms ) will automatically invoke GameMode if it s installed. For games that don t, you can manually evoke it using the gamemoderun command. Lutris is a tool that makes it easy to install and run games on Linux, and it also integrates with GameMode. (Lutris is currently being packaged for Debian, hopefully it will make it in on time for Bullseye).

Screenshot of Lutris, a tool that makes it easy to install your non-Linux games, which also integrates with GameMode.

GameMode in Debian The latest GameMode is packaged in Debian (Stephan Lachnit and I maintain it in the Debian Games Team) and it s also available for Debian 10 (Buster) via buster-backports. All you need to do to get up and running with GameMode is to install the gamemode package. GameMode in Debian supports 64 bit and 32 bit mode, so running it with older games (and many proprietary games) still work. Some distributions (like Arch Linux), have dropped 32 bit support, so 32 bit games on such systems lose any kind of integration with GameMode even if you can get those games running via other wrappers on such systems. We also include a binary called gamemode-simulate-game (installed under /usr/games/). This is a minimalistic program that will invoke gamemode automatically for 10 seconds and then exit without an error if it was successful. Its source code might be useful if you d like to add GameMode support to your game, or patch a game in Debian to automatically invoke it. In Debian we install Gamemode s example config file to /etc/gamemode.ini where a user can customise their system-wide preferences, or alternatively they can place a copy of that in ~/.gamemode.ini with their personal preferences. In this config file, you can also choose to explicitly allow or deny games. GameMode might also be useful for many pieces of software that aren t games. I haven t done any benchmarks on such software yet, but it might be great for users who use CAD programs or use a combination of their CPU/GPU to crunch a large amount of data. I ve also packaged an extension for GNOME called gamemode-extension. The Debian package is called gnome-shell-extension-gamemode . You ll need to enable it using gnome-tweaks after installation, it will then display a green controller in your notification area whenever GameMode is active. It s only in testing/bullseye since it relies on a newer gnome-shell than what s available in buster.

Welcome to the July 2020 report from the Reproducible Builds project. In these monthly reports, we round-up the things that we have been up to over the past month. As a brief refresher, the motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced from the original free software source code to the pre-compiled binaries we install on our systems. (If you re interested in contributing to the project, please visit our main website.)

General news At the upcoming DebConf20 conference (now being held online), Holger Levsen will present a talk on Thursday 27th August about Reproducing Bullseye in practice , focusing on independently verifying that the binaries distributed from ftp.debian.org were made from their claimed sources. Tavis Ormandy published a blog post making the provocative claim that You don t need reproducible builds , asserting elsewhere that the many attacks that have been extensively reported in our previous reports are fantasy threat models . A number of rebuttals have been made, including one from long-time contributor Reproducible Builds contributor Bernhard Wiedemann. On our mailing list this month, Debian Developer Graham Inggs posted to our list asking for ideas why the openorienteering-mapper Debian package was failing to build on the Reproducible Builds testing framework. Chris Lamb remarked from the build logs that the package may be missing a build dependency, although Graham then used our own diffoscope tool to show that the resulting package remains unchanged with or without it. Later, Nico Tyni noticed that the build failure may be due to the relationship between the FILE C preprocessor macro and the -ffile-prefix-map GCC flag. An issue in Zephyr, a small-footprint kernel designed for use on resource-constrained systems, around .a library files not being reproducible was closed after it was noticed that a key part of their toolchain was updated that now calls --enable-deterministic-archives by default. Reproducible Builds developer kpcyrd commented on a pull request against the libsodium cryptographic library wrapper for Rust, arguing against the testing of CPU features at compile-time. He noted that:

I ve accidentally shipped broken updates to users in the past because the build system was feature-tested and the final binary assumed the instructions would be present without further runtime checks

David Kleuker also asked a question on our mailing list about using SOURCE_DATE_EPOCH with the install(1) tool from GNU coreutils. When comparing two installed packages he noticed that the filesystem birth times differed between them. Chris Lamb replied, realising that this was actually a consequence of using an outdated version of diffoscope and that a fix was in diffoscope version 146 released in May 2020. Later in July, John Scott posted asking for clarification regarding on the Javascript files on our website to add metadata for LibreJS, the browser extension that blocks non-free Javascript scripts from executing. Chris Lamb investigated the issue and realised that we could drop a number of unused Javascript files [ ][ ][ ] and added unminified versions of Bootstrap and jQuery [ ].

Development work

Website On our website this month, Chris Lamb updated the main Reproducible Builds website and documentation to drop a number of unused Javascript files [ ][ ][ ] and added unminified versions of Bootstrap and jQuery [ ]. He also fixed a number of broken URLs [ ][ ]. Gonzalo Bulnes Guilpain made a large number of grammatical improvements [ ][ ][ ][ ][ ] as well as some misspellings, case and whitespace changes too [ ][ ][ ]. Lastly, Holger Levsen updated the README file [ ], marked the Alpine Linux continuous integration tests as currently disabled [ ] and linked the Arch Linux Reproducible Status page from our projects page [ ].

diffoscope diffoscope is our in-depth and content-aware diff utility that can not only locate and diagnose reproducibility issues, it provides human-readable diffs of all kinds. In July, Chris Lamb made the following changes to diffoscope, including releasing versions 150, 151, 152, 153 & 154:

• New features:
  • Add support for flash-optimised F2FS filesystems. (#207)
  • Don t require zipnote(1) to determine differences in a .zip file as we can use libarchive. [ ]
  • Allow --profile as a synonym for --profile=-, ie. write profiling data to standard output. [ ]
  • Increase the minimum length of the output of strings(1) to eight characters to avoid unnecessary diff noise. [ ]
  • Drop some legacy argument styles: --exclude-directory-metadata and --no-exclude-directory-metadata have been replaced with --exclude-directory-metadata= yes,no . [ ]
• Bug fixes:
  • Pass the absolute path when extracting members from SquashFS images as we run the command with working directory in a temporary directory. (#189)
  • Correct adding a comment when we cannot extract a filesystem due to missing libguestfs module. [ ]
  • Don t crash when listing entries in archives if they don t have a listed size such as hardlinks in ISO images. (#188)
• Output improvements:
  • Strip off the file offset prefix from xxd(1) and show bytes in groups of 4. [ ]
  • Don t emit javap not found in path if it is available in the path but it did not result in an actual difference. [ ]
  • Fix ... not available in path messages when looking for Java decompilers that used the Python class name instead of the command. [ ]
• Logging improvements:
  • Add a bit more debugging info when launching libguestfs. [ ]
  • Reduce the --debug log noise by truncating the has_some_content messages. [ ]
  • Fix the compare_files log message when the file does not have a literal name. [ ]
• Codebase improvements:
  • Rewrite and rename exit_if_paths_do_not_exist to not check files multiple times. [ ][ ]
  • Add an add_comment helper method; don t mess with our internal list directly. [ ]
  • Replace some simple usages of str.format with Python f-strings [ ] and make it easier to navigate to the main.py entry point [ ].
  • In the RData comparator, always explicitly return None in the failure case as we return a non-None value in the success one. [ ]
  • Tidy some imports [ ][ ][ ] and don t alias a variable when we do not use it. [ ]
  • Clarify the use of a separate NullChanges quasi-file to represent missing data in the Debian package comparator [ ] and clarify use of a null diff in order to remember an exit code. [ ]
• Other changes:
  • Profile the launch of libguestfs filesystems. [ ]
  • Clarify and correct our contributing info. [ ][ ][ ][ ][ ][ ]

Jean-Romain Garnier also made the following changes:

• Allow passing a file with a list of arguments via diffoscope @args.txt. (!62)
• Improve the output of side-by-side diffs by detecting added lines better. (!64)
• Remove offsets before instructions in objdump [ ][ ] and remove raw instructions from ELF tests [ ].

Other tools strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. It is used automatically in most Debian package builds. In July, Chris Lamb ensured that we did not install the internal handler documentation generated from Perl POD documents [ ] and fixed a trivial typo [ ]. Marc Herbert added a --verbose-level warning when the Archive::Cpio Perl module is missing. (!6) reprotest is our end-user tool to build same source code twice in widely differing environments and then checks the binaries produced by each build for any differences. This month, Vagrant Cascadian made a number of changes to support diffoscope version 153 which had removed the (deprecated) --exclude-directory-metadata and --no-exclude-directory-metadata command-line arguments, and updated the testing configuration to also test under Python version 3.8 [ ].

Distributions

Debian In June 2020, Timo R hling filed a wishlist bug against the debhelper build tool impacting the reproducibility status of hundreds of packages that use the CMake build system. This month however, Niels Thykier uploaded debhelper version 13.2 that passes the -DCMAKE_SKIP_RPATH=ON and -DBUILD_RPATH_USE_ORIGIN=ON arguments to CMake when using the (currently-experimental) Debhelper compatibility level 14. According to Niels, this change:

should fix some reproducibility issues, but may cause breakage if packages run binaries directly from the build directory.

34 reviews of Debian packages were added, 14 were updated and 20 were removed this month adding to our knowledge about identified issues. Chris Lamb added and categorised the nondeterministic_order_of_debhelper_snippets_added_by_dh_fortran_mod [ ] and gem2deb_install_mkmf_log [ ] toolchain issues. Lastly, Holger Levsen filed two more wishlist bugs against the debrebuild Debian package rebuilder tool [ ][ ].

openSUSE In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update. Bernhard also published the results of performing 12,235 verification builds of packages from openSUSE Leap version 15.2 and, as a result, created three pull requests against the openSUSE Build Result Compare Script [ ][ ][ ].

Other distributions In Arch Linux, there was a mass rebuild of old packages in an attempt to make them reproducible. This was performed because building with a previous release of the pacman package manager caused file ordering and size calculation issues when using the btrfs filesystem. A system was also implemented for Arch Linux packagers to receive notifications if/when their package becomes unreproducible, and packagers now have access to a dashboard where they can all see all their unreproducible packages (more info). Paul Spooren sent two versions of a patch for the OpenWrt embedded distribution for adding a build system revision to the packages manifest so that all external feeds can be rebuilt and verified. [ ][ ]

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of these patches, including:

• Bernhard M. Wiedemann:
  • afl (fix an incorrectly built manual page varied from kernel boot options)
  • brp-check-suse (sorting issue)
  • dnscrypt-proxy (sort the output of find(1))
  • graphviz (timezone issue, forwarded from Debian)
  • guile-gcrypt (parallelism)
  • insighttoolkit (prevent CPU detection, forwarded upstream
  • ipopt (parallelism issue and use https://tracker.debian.org/pkg/strip-nondeterminism)
  • jboss-logging-tools (date, forwarded upstream)
  • kismet (date)
  • lcov (date issue, already upstream)
  • multus (date issue, already upstream)
  • multus (date)
  • paperjam (date issue, forwarded upstream)
  • pspp (scrub testsuite.log)
  • python-PyNaCl (sort Python glob/readdir)
  • python-enaml (workaround an open upstream Python issue)
  • sac (omit creation time from .zip files)
  • sql-parser (sort, already upstream)
  • ugrep (CPU-related issue, already upstream)
  • ugrep (CPU-related issue)
  • unknown-horizons (filesystem ordering issue, already upstream)
  • unknown-horizons (filesystem ordering issue)
  • xfce4-panel-profiles (POSIX.1-2001/pax headers)
  • yast2-sound (uses uname -r)
• Chris Lamb:
  • dh-fortran-mod
  • flit
  • gem2deb
  • gmap
  • jskeus
  • libtpms
  • logilab-common
  • mrbayes
  • nmap
  • numpydoc
  • pyerfa
  • python-cooler
  • python-peachpy (forwarded upstream)
  • python-pyxs
  • sratom
  • weather-util
• Guillaume Nodet:
  • apache-sshd (date)

Vagrant Cascadian also reported two issues, the first regarding a regression in u-boot boot loader reproducibility for a particular target [ ] and a non-deterministic segmentation fault in the guile-ssh test suite [ ]. Lastly, Jelle van der Waa filed a bug against the MeiliSearch search API to report that it embeds the current build date.

Testing framework We operate a large and many-featured Jenkins-based testing framework that powers tests.reproducible-builds.org. This month, Holger Levsen made the following changes:

• Debian-related changes:
  • Tweak the rescheduling of various architecture and suite combinations. [ ][ ]
  • Fix links for 404 and not for us icons. (#959363)
  • Further work on a rebuilder prototype, for example correctly processing the sbuild exit code. [ ][ ]
  • Update the sudo configuration file to allow the node health job to work correctly. [ ]
  • Add php-horde packages back to the pkg-php-pear package set for the bullseye distribution. [ ]
  • Update the version of debrebuild. [ ]
• System health check development:
  • Add checks for broken SSH [ ], logrotate [ ], pbuilder [ ], NetBSD [ ], unkillable processes [ ], unresponsive nodes [ ][ ][ ][ ], proxy connection failures [ ], too many installed kernels [ ], etc.
  • Automatically fix some failed systemd units. [ ]
  • Add notes explaining all the issues that hosts are experiencing [ ] and handle zipped job log files correctly [ ].
  • Separate nodes which have been automatically marked as down [ ] and show status icons for jobs with issues [ ].
• Misc:
  • Disable all Alpine Linux jobs until they are or Alpine is fixed. [ ]
  • Perform some general upkeep of build nodes hosted by OSUOSL. [ ][ ][ ][ ]

In addition, Mattia Rizzolo updated the init_node script to suggest using sudo instead of explicit logout and logins [ ][ ] and the usual build node maintenance was performed by Holger Levsen [ ][ ][ ][ ][ ][ ], Mattia Rizzolo [ ][ ] and Vagrant Cascadian [ ][ ][ ][ ].

---

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Twitter: @ReproBuilds
• Mastodon: @reproducible_builds@fosstodon.org
• Reddit: /r/ReproducibleBuilds
• Mailing list: rb-general@lists.reproducible-builds.org

FTP master This month I accepted 434 packages and rejected 54. The overall number of packages that got accepted was 475. Debian LTS This was my seventy-third month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. This month my all in all workload has been 25.25h. During that time I did LTS uploads of:

• [DLA 2289-1] mupdf security update for five CVEs
• [DLA 2290-1] e2fsprogs security update for one CVE
• [DLA 2294-1] salt security update for two CVEs
• [DLA 2295-1] curl security update for one CVE
• [DLA 2296-1] luajit security update for one CVE
• [DLA 2298-1] libapache2-mod-auth-openidc security update for three CVEs

I started to work on python2.7 as well but stumbled over some hurdles in the testsuite, so I did not upload a fixed version yet. This month was much influenced by the transition from Jessie LTS to Stretch LTS and one or another workflow/script needed some adjustments. Last but not least I did some days of frontdesk duties. Debian ELTS This month was the twenty fifth ELTS month. During my allocated time I uploaded:

• ELA-230-1 for luajit
• ELA-231-1 for curl

Like in LTS, I also started to work on python2.7 and encountered the same hurdles in the testsuite. So I did not upload a fixed version for ELTS as well. Last but not least I did some days of frontdesk duties. Other stuff In this section nothing much happened this month. Thanks a lot to everybody who NMUed a package to fix a bug.

The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid packages maintained in Git. This post is part of a series about the progress of the Janitor. Since the original post, merge proposals created by the janitor now include the debdiff between a build with and without the changes (showing the impact to the binary packages), in addition to the merge proposal diff (which shows the impact to the source package). New merge proposals also include a link to the diffoscope diff between a vanilla build and the build with changes. Unfortunately these can be a bit noisy for packages that are not reproducible yet, due to the difference in build environment between the two builds. This is part of the effort to keep the changes from the janitor high-quality. The rollout surfaced some bugs in lintian-brush; these have been either fixed or mitigated (e.g. by disabling specified fixers).

For more information about the Janitor's lintian-fixes efforts, see the landing page

If you are wondering what your default web browser is set to on a Debian-based system, there are several things to look at:

$ xdg-settings get default-web-browser
brave-browser.desktop
$ xdg-mime query default x-scheme-handler/http
brave-browser.desktop
$ xdg-mime query default x-scheme-handler/https
brave-browser.desktop
$ ls -l /etc/alternatives/x-www-browser
lrwxrwxrwx 1 root root 29 Jul  5  2019 /etc/alternatives/x-www-browser -> /usr/bin/brave-browser-stable*
$ ls -l /etc/alternatives/gnome-www-browser
lrwxrwxrwx 1 root root 29 Jul  5  2019 /etc/alternatives/gnome-www-browser -> /usr/bin/brave-browser-stable*

Debian-specific tools The contents of /etc/alternatives/ is system-wide defaults and must therefore be set as root:

sudo update-alternatives --config x-www-browser
sudo update-alternatives --config gnome-www-browser

The sensible-browser tool (from the sensible-utils package) will use these to automatically launch the most appropriate web browser depending on the desktop environment.

Standard MIME tools The others can be changed as a normal user. Using xdg-settings:

xdg-settings set default-web-browser brave-browser-beta.desktop

will also change what the two xdg-mime commands return:

$ xdg-mime query default x-scheme-handler/http
brave-browser-beta.desktop
$ xdg-mime query default x-scheme-handler/https
brave-browser-beta.desktop

since it puts the following in ~/.config/mimeapps.list:

[Default Applications]
text/html=brave-browser-beta.desktop
x-scheme-handler/http=brave-browser-beta.desktop
x-scheme-handler/https=brave-browser-beta.desktop
x-scheme-handler/about=brave-browser-beta.desktop
x-scheme-handler/unknown=brave-browser-beta.desktop

Note that if you delete these entries, then the system-wide defaults, defined in /etc/mailcap, will be used, as provided by the mime-support package. Changing the x-scheme-handler/http (or x-scheme-handler/https) association directly using:

xdg-mime default brave-browser-nightly.desktop x-scheme-handler/http

will only change that particular one. I suppose this means you could have one browser for insecure HTTP sites (hopefully with HTTPS Everywhere installed) and one for HTTPS sites though I'm not sure why anybody would want that.

Summary In short, if you want to set your default browser everywhere (using Brave in this example), do the following:

sudo update-alternatives --config x-www-browser
sudo update-alternatives --config gnome-www-browser
xdg-settings set default-web-browser brave-browser.desktop

I'm starting a new blog, and instead of going into the technical details on how it's made, or on how I migrated my content from the previous ones, I want focus on why I did it. It's been a while since I have written a blog post. I wanted to get back into it, but also wanted to finally self-host my blog/homepage because I have been let down before. And sadly, I was not let down by a for-profit, privacy invading corporation, but by a free software organization. The sad story of wiki.softwarelivre.org My first blog that was hosted in a blog engine written by me, which was hosted in a TWiki, and later Foswiki instance previously available at wiki.softwarelivre.org, hosted by ASL.org. I was the one who introduced the tool to the organization in the first place. I had come from a previous, very fruitful experience on the use of wikis for creation of educational material while in university, which ultimately led me to become a core TWiki, and then Foswiki developer. In 2004, I had just moved to Porto Alegre, got involved in ASL.org, and there was a demand for a tool like that. 2 years later, I left Porto Alegre, and some time after that the daily operations of ASL.org when it became clear that it was not really prepared for remote participation. I was still maintaing the wiki software and the OS for quite some years after, until I wasn't anymore. In 2016, the server that hosted it went haywire, and there were no backups. A lot of people and free software groups lost their content forever. My blog was the least important content in there. To mention just a few examples, here are some groups that lost their content in there:

• The Brazilian Foswiki user group hosted a bunch of getting started documentation in Portuguese, organized meetings, and coordinated through there.
• GNOME Brazil hosted its homepage there until the moment the server broke.
• The Inkscape Brazil user group had an amazing space there where they shared tutorials, a gallery of user-contributed drawings, and a lot more.

Some of this can still be reached via the Internet Archive Wayback Machine, but that is only useful for recovering content, not for it to be used by the public. The announced tragedy of softwarelivre.org My next blog after that was hosted at softwarelivre.org, a Noosfero instance also hosted by ASL.org. When it was introduced in 2010, this Noosfero instance became responsible for the main domain softwarelivre.org name. This was a bold move by ASL.org, and was a demonstration of trust in a local free software project, led by a local free software cooperative (Colivre). I was a lead developer in the Noosfero project for a long time, and I was also involved in maintaining that server as well. However, for several years there is little to no investment in maintaining that service. I already expect that it will probably blow up at some point as the wiki did, or that it may be just shut down on purpose. On the responsibility of organizations Today, a large part of wast mot people consider "the internet" is controlled by a handful of corporations. Most popular services on the internet might look like they are gratis (free as in beer), but running those services is definitely not without costs. So when you use services provided by for-profit companies and are not paying for them with money, you are paying with your privacy and attention. Society needs independent organizations to provide alternatives. The market can solve a part of the problem by providing ethical services and charging for them. This is legitimate, and as long as there is transparency about how peoples' data and communications are handled, there is nothing wrong with it. But that only solves part of the problem, as there will always be people who can't afford to pay, and people and communities who can afford to pay, but would rather rely on a nonprofit. That's where community-based services, provided by nonprofits, are also important. We should have more of them, not less. So it makes me worry to realize ASL.org left the community in the dark. Losing the wiki wasn't even the first event of its kind, as the listas.softwarelivre.org mailing list server, with years and years of community communications archived in it, broke with no backups in 2012. I do not intend to blame the ASL.org leadership personally, they are all well meaning and good people. But as an organization, it failed to recognize the importance of this role of service provider. I can even include myself in it: I was member of the ASL.org board some 15 years ago; I was involved in the deployment of both the wiki and Noosfero, the former as a volunteer and the later professionally. Yet, I did nothing to plan the maintenance of the infrastructure going forward. When well meaning organizations fail, people who are not willing to have their data and communications be exploited for profit are left to their own devices. I can afford a virtual private server, and have the technical knowledge to import my old content into a static website generator, so I did it. But what about all the people who can't, or don't? Of course, these organizations have to solve the challenge of being sustainable, and being able to pay professionals to maintain the services that the community relies on. We should be thankful to these organizations, and their leadership needs to recognize the importance of those services, and actively plan for them to be kept alive.

I'm starting a new blog, and instead of going into the technical details on how it's made, or on how I migrated my content from the previous ones, I want to focus on why I did it. It's been a while since I have written a blog post. I wanted to get back into it, but also wanted to finally self-host my blog/homepage because I have been let down before. And sadly, I was not let down by a for-profit, privacy invading corporation, but by a free software organization. The sad story of wiki.softwarelivre.org My first blog that was hosted in a blog engine written by me, which was hosted in a TWiki, and later Foswiki instance previously available at wiki.softwarelivre.org, hosted by ASL.org. I was the one who introduced the tool to the organization in the first place. I had come from a previous, very fruitful experience on the use of wikis for creation of educational material while in university, which ultimately led me to become a core TWiki, and then Foswiki developer. In 2004, I had just moved to Porto Alegre, got involved in ASL.org, and there was a demand for a tool like that. 2 years later, I left Porto Alegre, and some time after that the daily operations of ASL.org when it became clear that it was not really prepared for remote participation. I was still maintaing the wiki software and the OS for quite some years after, until I wasn't anymore. In 2016, the server that hosted it went haywire, and there were no backups. A lot of people and free software groups lost their content forever. My blog was the least important content in there. To mention just a few examples, here are some groups that lost their content in there:

• The Brazilian Foswiki user group hosted a bunch of getting started documentation in Portuguese, organized meetings, and coordinated through there.
• GNOME Brazil hosted its homepage there until the moment the server broke.
• The Inkscape Brazil user group had an amazing space there where they shared tutorials, a gallery of user-contributed drawings, and a lot more.

Some of this can still be reached via the Internet Archive Wayback Machine, but that is only useful for recovering content, not for it to be used by the public. The announced tragedy of softwarelivre.org My next blog after that was hosted at softwarelivre.org, a Noosfero instance also hosted by ASL.org. When it was introduced in 2010, this Noosfero instance became responsible for the main domain softwarelivre.org name. This was a bold move by ASL.org, and was a demonstration of trust in a local free software project, led by a local free software cooperative (Colivre). I was a lead developer in the Noosfero project for a long time, and I was also involved in maintaining that server as well. However, for several years there is little to no investment in maintaining that service. I already expect that it will probably blow up at some point as the wiki did, or that it may be just shut down on purpose. On the responsibility of organizations Today, a large part of wast mot people consider "the internet" is controlled by a handful of corporations. Most popular services on the internet might look like they are gratis (free as in beer), but running those services is definitely not without costs. So when you use services provided by for-profit companies and are not paying for them with money, you are paying with your privacy and attention. Society needs independent organizations to provide alternatives. The market can solve a part of the problem by providing ethical services and charging for them. This is legitimate, and as long as there is transparency about how peoples' data and communications are handled, there is nothing wrong with it. But that only solves part of the problem, as there will always be people who can't afford to pay, and people and communities who can afford to pay, but would rather rely on a nonprofit. That's where community-based services, provided by nonprofits, are also important. We should have more of them, not less. So it makes me worry to realize ASL.org left the community in the dark. Losing the wiki wasn't even the first event of its kind, as the listas.softwarelivre.org mailing list server, with years and years of community communications archived in it, broke with no backups in 2012. I do not intend to blame the ASL.org leadership personally, they are all well meaning and good people. But as an organization, it failed to recognize the importance of this role of service provider. I can even include myself in it: I was member of the ASL.org board some 15 years ago; I was involved in the deployment of both the wiki and Noosfero, the former as a volunteer and the later professionally. Yet, I did nothing to plan the maintenance of the infrastructure going forward. When well meaning organizations fail, people who are not willing to have their data and communications be exploited for profit are left to their own devices. I can afford a virtual private server, and have the technical knowledge to import my old content into a static website generator, so I did it. But what about all the people who can't, or don't? Of course, these organizations have to solve the challenge of being sustainable, and being able to pay professionals to maintain the services that the community relies on. We should be thankful to these organizations, and their leadership needs to recognize the importance of those services, and actively plan for them to be kept alive.

For most of our Desktop PC running with stock exim4 and mutt, I think sending out mail is becoming a bit rough since using random smarthost causes lots of trouble due to the measures taken to prevent spams.

As mentioned in Exim4 user FAQ , /etc/hosts should have FQDN with external DNS resolvable domain name listed instead of localdomain to get the correct EHLO/HELO line. That's the first step.

The stock configuration of exim4 only allows you to use single smarthost for all your mails. I use one address for my personal use which is checked by my smartphone too. The other account is for subscribing to the mailing list. So I needed to tweak ...

Usually, mutt is smart enough to set the From address since my .muttrc has

# Set default for From: for replyes for alternates.
set reverse_name

For my gmail accounts, each mail should be sent to the account specific SMTP connection matching your From header to get all the modern SPAM protection data in right state. DKIM, SPF, DMARC... (Besides, they overwrite From: header anyway if you use wrong connection.)

For my debian.org mails, mails should be sent from my shell account on people.debian.org so it is very unlikely to be blocked. Sometimes, I wasn't sure some of these debian.org mails sent through my ISP's smarthost are really getting to the intended person.

To these ends, I have created small patches to the /etc/exim4/conf.d files and reported it to Debian BTS: #869480 Support multiple smarthosts (gmail support). These patches are for the source package.

To use my configuration tweak idea, you have easier route no matter which exim version you are using. Please copy and read pertinent edited files from my github site to your installed /etc/exim4/conf.d files and get the benefits.

• Update router/200_exim4-config_primary by overwriting it with newer router/200_exim4-config_primary
• Add transport/30_exim4-config_remote_smtp_submission and transport/30_exim4-config_remote_smtp_submission
• Update auth/30_exim4-config_examples by overwriting with newer auth/30_exim4-config_examples
• Read passwd.client to set up multiple SMTP message submission host accounts.
• Read README.ssh for how to set up mail to be sent over to the shell account via ssh.

 .ifndef NO_EAA_REWRITE_REWRITE
*@+local_domains "$ lookup $ local_part lsearch /etc/email-addresses \
                   $value fail " f
# identical rewriting rule for /etc/mailname
*@ETC_MAILNAME "$ lookup $ local_part lsearch /etc/email-addresses \
                   $value fail " f
.endif
* "$h_from:" Frs

Here s my (tenth) monthly update about the activities I ve done in the F/L/OSS world.

Debian

This was my 17th month of contributing to Debian. I became a DM in late March last year and a DD last Christmas! \o/ Well, this month I didn t do a lot of Debian stuff, like I usually do, however, I did a lot of things related to Debian (indirectly via GSoC)! Anyway, here are the following things I did this month:

Uploads and bug fixes:

• rails (2:6.0.3.2+dfsg-1) - Fix CVE-2020-8185 (Closes #964081).
• ruby-rubocop-packaging (0.1.1-1) - Fix blank file issue.
• micro (2.0.6-2~bpo10+1) - Backport new upstream version.
• ruby2.5 (2.5.5-3+deb10u2) - Fix CVE-2020-10663 and CVE-2020-10933.
• djangorestframework-api-key (2.0.0-2) - Fix #956920.
• golang-golang-x-text (0.3.3-1~bpo10+1) - Backport CVE-2020-14040 fix to Buster.
• ruby-growl (4.1+dfsg-2) - Source-only upload.
• ruby-iso8601 (0.13.0-1) - New upstream version, drop patches as they re merged upstream.
• sup-mail (1.0-1) - Re-introduce sup-mail to the Debian archive.
• ruby-rubocop-packaging (0.2.0-1) - New version for the RelativeRequireToLib cop.
• ruby-rubocop-ast (0.1.0-2) - Source-only upload.
• ruby-rubocop-rspec (1.42.0-1) - Fixing FTBFS with RuboCop v0.88.0 via v1.42.0.
• ruby-rubocop-performance (1.7.1-1) - Fixing FTBFS with RuboCop v0.88.0 via v1.7.1.

Other $things:

• Mentoring for newcomers.
• FTP Trainee reviewing.
• Moderation of -project mailing list.
• Sponsored php-twig for William, ruby-growl, ruby-xmpp4r, and uby-uniform-notifier for Cocoa, sup-mail for Iain, and node-markdown-it for Sakshi.

---

GSoC Phase 2, Part 2! In May, I got selected as a Google Summer of Code student for Debian again! \o/
I am working on the Upstream-Downstream Cooperation in Ruby project. The first three blogs can be found here:

• GSoC Phase 1 (part 1).
• GSoC Phase 1 (part 2).
• GSoC Phase 2 (part 1).
• And this is GSoC Phase 2 (part 2).

Also, I log daily updates at gsocwithutkarsh2102.tk. Whilst the daily updates are available at the above site^, I ll breakdown the important parts of the later half of the second month here:

• Marc Andre, very kindly, helped in fixing the specs that were failing earlier this month. Well, the problem was with the specs, but I am still confused how so. Anyway..
• Finished documentation of the second cop and marked the PR as ready to be reviewed.
• David reviewed and suggested some really good changes and I fixed/tweaked that PR as per his suggestion to finally finish the last bits of the second cop, RelativeRequireToLib.
• Merged the PR upon two approvals and released it as v0.2.0!
• We had our next weekly meeting where we discussed the next steps and the things that are supposed to be done for the next set of cops.
• Introduced rubocop-packaging to the outer world and requested other upstream projects to use it! It is being used by 13 other projects already!
• Started to work on packaging-style-guide but I didn t push anything to the public repository yet.
• Worked on refactoring the cops_documentation Rake task which was broken by the new auto-corrector API. Opened PR #7 for it. It ll be merged after the next RuboCop release as it uses CopsDocumentationGenerator class from the master branch.
• Whilst working on autoprefixer-rails, I found something unusual. The second cop shouldn t really report offenses if the require_relative calls are from lib to lib itself. This is a false-positive. Opened issue #8 for the same.

---

Continuation of GSoC for other Ruby related stuff!

Whilst working on rubocop-packaging, I contributed to more Ruby projects, refactoring their library a little bit and mostly fixing RuboCop issues and fixing issues that the Packaging extension reports as offensive .
Following are the PRs that I raised:

• PR #175 for gir_ffi to drop git ls-files in gemspec.
• PR #3791 for rubygems/bundler to remove redundant bundler/setup require call from spec_helper generated by bundle gem.
• PR #2307 for puma to constrain rake-compiler to v0.9.4.
• PR #476 for cucumber-rails to drop git ls-files in gemspec.
• PR #721 for aruba to drop git ls-files in gemspec.
• PR #89 for wasabi to drop git ls-files in gemspec.
• PR #24 for ed25519 to fix RuboCop warning and offenses.
• PR #25 for ed25519 to drop git ls-files in gemspec.
• PR #59 for ISO8601 to drop git ls-files in gemspec.
• PR #60 for ISO8601 to fix other RuboCop offenses.
• PR #61 for ISO8601 to (minor) refactor the library.
• PR #580 for sup to drop git ls-files in gemspec.
• PR #166 for autoprefixer-rails to use RuboCop to enhance some bits of code.
• PR #167 for autoprefixer-rails to fix remaining RuboCop warning and offenses.
• PR #169 for autoprefixer-rails to do some minor refactoring.
• PR #170 for autoprefixer-rails to drop git ls-files in gemspec.

---

Debian (E)LTS

Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success. And Debian Extended LTS (ELTS) is its sister project, extending support to the Jessie release (+2 years after LTS support). This was my tenth month as a Debian LTS and my first as a Debian ELTS paid contributor.
I was assigned 25.25 hours for LTS and 13.25 hours for ELTS and worked on the following things:

LTS CVE Fixes and Announcements:

• Issued DLA 2269-1, fixing CVE-2020-4046, CVE-2020-4047, CVE-2020-4048, CVE-2020-4049, and CVE-2020-4050, for wordpress.
  For Debian 8 Jessie, these problems have been fixed in version 4.1.31+dfsg-0+deb8u1.
• Issued DLA 2270-1, fixing CVE-2020-14060, CVE-2020-14061, CVE-2020-14062, and CVE-2020-14195, for jackson-databind.
  For Debian 8 Jessie, these problems have been fixed in version 2.4.2-2+deb8u15.
• Issued DLA 2271-1, fixing CVE-2020-4067, for coturn.
  For Debian 8 Jessie, this problem has been fixed in version 4.2.1.2-1+deb8u2.
• Issued DLA 2275-1, fixing CVE-2020-8161 and CVE-2020-8184, for ruby-rack.
  For Debian 9 Stretch, these problems have been fixed in version 1.6.4-4+deb9u2.
• Issued DLA 2276-1, fixing CVE-2020-12108 and CVE-2020-15011, for mailman.
  For Debian 9 Stretch, these problems have been fixed in version 1:2.1.23-1+deb9u6.
• Issued DLA 2277-1, fixing CVE-2019-12973, CVE-2020-6851, CVE-2020-8112, and CVE-2020-15389, for openjpeg2.
  For Debian 9 Stretch, these problems have been fixed in version 2.1.2-1.1+deb9u5.
• Issued DLA 2288-1, fixing CVE-2017-9503, CVE-2019-12068, CVE-2019-20382, CVE-2020-1983, CVE-2020-8608, CVE-2020-10756, CVE-2020-13361, CVE-2020-13362, CVE-2020-13659, CVE-2020-13754, CVE-2020-13765, and CVE-2020-15863, for qemu. This was mostly worked upon by the maintainer, Michael.
  For Debian 9 Stretch, these problems have been fixed in version 1:2.8+dfsg-6+deb9u10.

ELTS CVE Fixes and Announcements:

• Issued ELA 240-1, fixing CVE-2020-12695, for wpa.
  For Debian 8 Jessie, these problems have been fixed in version 2.3-1+deb8u11.
• Issued ELA 241-1, fixing CVE-2020-15389, for openjpeg2.
  For Debian 8 Jessie, these problems have been fixed in version 2.1.0-2+deb8u11.
• Issued ELA 249-1, fixing CVE-2020-13659 and CVE-2020-15863, for qemu.
  For Debian 8 Jessie, these problems have been fixed in version 1:2.1+dfsg-12+deb8u16.

Other (E)LTS Work:

• Did my LTS frontdesk duty from 29th June to 5th July.
• Triaged qemu, firefox-esr, wordpress, libmediainfo, squirrelmail, xen, openjpeg2, samba, and ldb.
• Mark CVE-2020-15395/libmediainfo as no-dsa for Jessie.
• Mark CVE-2020-13754/qemu as no-dsa/intrusive for Stretch and Jessie.
• Mark CVE-2020-12829/qemu as no-dsa for Jessie.
• Mark CVE-2020-10756/qemu as not-affected for Jessie.
• Mark CVE-2020-13253/qemu as postponed for Jessie.
• Drop squirrelmail and xen for Stretch LTS.
• Add notes for tomcat8, shiro, and cacti to take care of the Stretch issues.
• Emailed team@security.d.o and debian-lts@l.d.o regarding possible clashes.
• Maintenance of LTS Survey on the self-hosted LimeSurvey instance. Received 1765 (just wow!) responses.
• Attended the fourth LTS meeting. MOM here.
• General discussion on LTS private and public mailing list.

---

Other(s)

Sometimes it gets hard to categorize work/things into a particular category.
That s why I am writing all of those things inside this category.
This includes two sub-categories and they are as follows.

Personal: This month I did the following things:

• Released v0.2.0 of rubocop-packaging on RubyGems!
  It s open-sourced and the repository is here.
  Bug reports and pull requests are welcomed!
• Released v0.1.0 of get_root on RubyGems!
  It s open-sourced and the repository is here.
• Wrote max-word-frequency, my Rails C1M2 programming assignment.
  And made it pretty neater & cleaner!
• Refactored my lts-dla and elts-ela scripts entirely and wrote them in Ruby so that there are no issues and no false-positives!
  Check lts-dla here and elts-ela here.
• And finally, built my first Rails (mini) web-application!
  The repository is here. This was also a programming assignment (C1M3).
  And furthermore, hosted it at Heroku.

Open Source: Again, this contains all the things that I couldn t categorize earlier.
Opened several issues and PRs:

• Issue #8273 against rubocop, reporting a false-positive auto-correct for Style/WhileUntilModifier.
• Issue #615 against http reporting a weird behavior of a flaky test.
• PR #3791 for rubygems/bundler to remove redundant bundler/setup require call from spec_helper generated by bundle gem.
• Issue #3831 against rubygems, reporting a traceback of undefined method, rubyforge_project=.
• Issue #238 against nheko asking for enhancement in showing the font name in the very font itself.
• PR #2307 for puma to constrain rake-compiler to v0.9.4.
• And finally, I joined the Cucumber organization! \o/

---

Thank you for sticking along for so long :) Until next time.
:wq for today.

Here is my monthly update covering what I have been doing in the free and open source software world during July 2020 (previous month):

• Opened a pull request to make the build reproducible in PyERFA, a set of Python bindings for various astronomy-related utilities (#45), as well as one for PeachPy assembler to make the output of codecode/x86_64.py reproducible (#108).

• As part of being on the board of directors of the Open Source Initiative and Software in the Public Interest I attended their respective monthly meetings and participated in various licensing and other discussions occurring on the internet, as well as the usual internal discussions regarding logistics and policy etc. This month, it was SPI's Annual General Meeting and the OSI has been running a number of remote strategy sessions for the board.
• Fixed an issue in my tickle-me-email library that implements Getting Things Done (GTD)-like behaviours in IMAP inboxes to ensure that all messages have a unique Message-Id header. [...]
• Reviewed and merged even more changes by Pavel Dolecek into my Strava Enhancement Suite, a Chrome extension to improve the user experience on the Strava athletic tracker.
• Updated travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub, to use the Travis CI continuous integration platform) to fix a compatibility issue with the latest version of mk-build-deps. [...][...]

For Lintian, the static analysis tool for Debian packages:

• Update the regular expression to search for all the released versions in a .changes file. [...]
• Avoid false-positives when matching sensible-utils utilities such as i3-sensible-pager. (#966022)
• Rename the send-patch tag to patch-not-forwarded-upstream. [...]
• Drop reminders from 26 tags that false-positives should be reported to Lintian as this is implicit in all our tags. [...]

Reproducible Builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. The project is proud to be a member project of the Software Freedom Conservancy. Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter. This month, I:

• Opened pull requests to make the build reproducible in pyerfa (#45) as well as one for PeachPy to make the output of codecode/x86_64.py reproducible (#108).
• In Debian:
  • Submitted the following patches to fix reproducibility-related toolchain issues within Debian:
    • dh-fortran-mod: Please make the output reproducible. (#965255)
    • gem2deb: should not install mkmf.log files. (#964772)
    • nmap: Please make the obfuscated nmap_service.exe file reproducible. (#964369)
  • I also submitted 13 patches to fix specific reproducibility issues in flit, gmap, jskeus, libtpms, logilab-common, mrbayes, numpydoc, pyerfa, python-cooler, python-peachpy, python-pyxs, sratom & weather-util.
  • When investigating packages, I discovered and reported that dolfinx was running its test suite even when the nocheck build profile was set. (#965946)
  • Kept isdebianreproducibleyet.com up to date. [...]
• Categorised a large number of packages and issues in the Reproducible Builds 'notes' repository.
• Drafted, published and publicised our monthly report.

• Updated the main Reproducible Builds website and documentation to drop a number of unused Javascript files [...][...][...] and added unminified versions of Bootstrap and jQuery [...] and fixed a number of broken URLs [...][...].
• strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. This month, I made sure that we did not install the internal handler documentation generated from Perl POD documents [...] and fixed a trivial typo [...].

diffoscope Elsewhere in our tooling, I made the following changes to diffoscope, including preparing and uploading versions 150, 151, 152, 153 & 154 to Debian:

• New features:
  • Add support for flash-optimised F2FS filesystems. (#207)
  • Don't require zipnote(1) to determine differences in a .zip file as we can use libarchive. [...]
  • Allow --profile as a synonym for --profile=-, ie. write profiling data to standard output. [...]
  • Increase the minimum length of the output of strings(1) to eight characters to avoid unnecessary diff noise. [...]
  • Drop some legacy argument styles: --exclude-directory-metadata and --no-exclude-directory-metadata have been replaced with --exclude-directory-metadata= yes,no . [...]
• Bug fixes:
  • Pass the absolute path when extracting members from SquashFS images as we run the command with working directory in a temporary directory. (#189)
  • Correct adding a comment when we cannot extract a filesystem due to missing libguestfs module. [...]
  • Don't crash when listing entries in archives if they don't have a listed size such as hardlinks in ISO images. (#188)
• Output improvements:
  • Strip off the file offset prefix from xxd(1) and show bytes in groups of 4. [...]
  • Don't emit javap not found in path if it is available in the path but it did not result in an actual difference. [...]
  • Fix ... not available in path messages when looking for Java decompilers that used the Python class name instead of the command. [...]
• Logging improvements:
  • Add a bit more debugging info when launching libguestfs. [...]
  • Reduce the --debug log noise by truncating the has_some_content messages. [...]
  • Fix the compare_files log message when the file does not have a literal name. [...]
• Codebase improvements:
  • Rewrite and rename exit_if_paths_do_not_exist to not check files multiple times. [...][...]
  • Add an add_comment helper method; don't mess with our internal list directly. [...]
  • Replace some simple usages of str.format with Python 'f-strings' [...] and make it easier to navigate to the main.py entry point [...].
  • In the RData comparator, always explicitly return None in the failure case as we return a non-None value in the success one. [...]
  • Tidy some imports [...][...][...] and don't alias a variable when we don't end up using. [...]
  • Clarify the use of a separate NullChanges quasi-file to represent missing data in the Debian package comparator [...] and clarify use of a 'null' diff in order to remember an exit code. [...]
• Misc:
  • Profile the launch of libguestfs filesystems. [...]
  • Clarify and correct our contributing info. [...][...][...][...][...][...]

Debian In Debian, I made the following uploads this month:

• Redis (6.0.6-1) New upstream release.
• Django:
  • 2.2.14-1 New upstream bugfix release.
  • 3.1~rc1-1 New upstream release candidate.
• lastpass-cli (1.3.3-4) Fix a build failure under GCC 10. (#957416)

Debian LTS This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 for the Extended LTS project. This included:

• Frontdesk duties, including responding to user/developer questions, reviewing packages, participating in mailing list discussions, offering comments on our survey and attended our IRC contributor meeting. I also made some changes to tools after the move from the Debian version supported [...].
• Investigated and triaged atril, ceph, cimg, ffmpeg, golang-github-seccomp-libseccomp-golang, gosa, http-parser, jruby, json-c, jupyter-notebook, keystone, ksh, libjpeg-turbo, libopenmpt, libpam-radius-auth, log4net, mailman, milkytracker, mupdf, pillow, poppler, puma, python-rtslib-fb, roundcube, ruby-zip, salt, slirp, sqlite3, transmission & wordpress.
• Issued DLA 2273-1 for shiro, first to fix a path-traversal issue where a specially-crafted request could cause an authentication bypass (CVE-2020-1957) and then to fix an encoding issue introduced in the handling of the previous CVE-2020-1957 path-traversal issue which itself could have also caused an authentication bypass (CVE-2020-11989).
• Issued DLA 2274-1 to fix a possible signature verification issue in the fwupd firmware update tool as the return value of gpgme_op_verify_result was not being checked.
• Issued DLA 2299-1 and ELA 252-1 in order to prevent a privilege escalation vulnerability discovered in Net-SNMP, a set of tools for collecting and organising information about devices on computer networks.
• Issued DLA 2300-1 as it was discovered that there was an issue where kdepim-runtime would default to using unencrypted POP3 communication despite the user interface indicating that encryption was in use.

You can find out more about the project via the following video:

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi. After having had some success with a sysroot in having a Qt5 cross-build environment that includes QtWebEngine, the next step is packaging the sysroot so it can be available both to build the cross-build environment, and to do cross-development with it. The result is this Debian source package which takes a Raspberry Pi OS disk image, provisions it in-place, extracts its contents, and packages them. Yes. You may want to reread the last paragraph. It works directly in the disk image to avoid a nasty filesystem issue on emulated 32bit Linux over a 64bit mounted filesystem. This feels like the most surreal Debian package I've ever created, and this saga looks like one of the hairiest yaks I've ever shaved. Integrating this monster codebase, full of bundled code and hacks, into a streamlined production and deployment system has been for me a full stack nightmare, and I have a renewed and growing respect for the people in the Qt/KDE team in Debian, who manage to stay on top of this mess, so that it all just works when we need it.

So a few months ago kiddo one dropped an apparently fairly large cup of coffee onto her one and only trusted computer. With a few months (then) to graduation (which by now happened), and with the apparent genuis bar verdict of it s a goner a new one was ordered. As it turns out this supposedly dead one coped well enough with the coffee so that after a few weeks of drying it booted again. But give the newer one, its apparent age and whatnot, it was deemed surplus. So I poked around a little on the interwebs and conclude that yes, this could work. Fast forward a few months and I finally got hold of it, and had some time to play with it. First, a bootable usbstick was prepared, and the machine s content was really (really, and check again: really) no longer needed, I got hold of it for good. tl;dr It works just fine. It is a little heavier than I thought (and isn t air supposed to be weightless?) The ergonomics seem quite nice. The keyboard is decent. Screen-resolution on this pre-retina simple Air is so-so at 1440 pixels. But battery live seems ok and e.g. the camera is way better than what I have in my trusted Lenovo X1 or at my desktop. So just as a zoom client it may make a lot of sense; otherwise just walking around with it as a quick portable machine seems perfect (especially as my Lenovo X1 still (ahem) suffers from one broken key I really need to fix ). Below are some lightly edited notes from the installation. Initial steps were quick: maybe an hour or less? Customizing a machine takes longer than I remembered, this took a few minutes here and there quite a few times, but always incremental.

Initial Steps

• Download of Ubuntu 20.04 LTS image: took a few moments, even on broadband, feels slower than normal (fast!) Ubuntu package updates, maybe lesser CDN or bad luck
• Startup Disk Creator using a so-far unused 8gb usb drive
• Plug into USB, recycle power, press Option on macOS keyboard: voila
• After a quick hunch no to live/test only and yes to install, whole disk
• install easy, very few questions, somehow skips wifi
• so activate wifi manually and everythings pretty much works

Customization

• First deal with fn and ctrl key swap. Install git and followed this github repo which worked just fine. Yay. First (manual) Linux kernel module build needed need in half a decade? Longer?
• Fire up firefox, go to download chrome , install chrome. Sign in. Turn on syncing. Sign into Pushbullet and Momentum.
• syncthing which is excellent. Initially via apt, later from their PPA. Spend some time remembering how to set up the mutual handshakes between devices. Now syncing desktop/server, lenovo x1 laptop, android phone and this new laptop
• keepassx via apt and set up using Sync/ folder. Now all (encrypted) passwords synced.
• Discovered synergy now longer really free, so after a quick search found and installed barrier (via apt) to have one keyboard/mouse from desktop reach laptop.
• Added emacs via apt, so far empty , so config files yet
• Added ssh via apt, need to propagate keys to github and gitlab
• Added R via add-apt-repository --yes "ppa:marutter/rrutter4.0" and add-apt-repository --yes "ppa:c2d4u.team/c2d4u4.0+". Added littler and then RStudio
• Added wajig (apt frontend) and byobu, both via apt
• Created ssh key, shipped it to server and github + gitlab
• Cloned (not-public) dotfiles repo and linked some dotfiles in
• Cloned git repo for nord-theme for gnome terminal and installed it; also added it to RStudio via this repo
• Emacs installed, activated dotfiles, then incrementally install a few elpa-* packages and a few M-x package-install including nord-theme, of course
• Installed JetBrains Mono font from my own local package; activated for Gnome Terminal and Emacs
• Install gnome-tweak-tool via apt, adjusted a few settings
• Ran gsettings set org.gnome.desktop.wm.preferences focus-mode 'sloppy'
• Set up camera following this useful GH repo
• At some point also added slack and zoom, because, well, it is 2020
• STILL TODO:
  • docker
  • bother with email setup?,
  • maybe atom/code/ ?

I'm buying rather too many books at the moment and not reading enough of them (in part because I got back into Minecraft and in part because I got a bit stuck on a few difficult books). I think I've managed to get myself unstuck again, though, and have started catching up on reviews. 2020. It's kind of a lot. And I'm not even that heavily affected. Katherine Addison The Angel of the Crows (sff)
Marie Brennan A Natural History of Dragons (sff)
Kacen Callender Queen of the Conquered (sff)
Jo Clayton Diadem from the Stars (sff)
Jo Clayton Lamarchos (sff)
Jo Clayton Irsud (sff)
Clifford D. Conner The Tragedy of American Science (nonfiction)
Kate Elliott Unconquerable Sun (sff)
Rory Fanning & Craig Hodges Long Shot (nonfiction)
Michael Harrington Socialism: Past & Future (nonfiction)
Nalo Hopkinson Brown Girl in the Ring (sff)
Kameron Hurley The Stars Are Legion (sff)
N.K. Jemisin Emergency Skin (sff)
T. Kingfisher A Wizard's Guide to Defensive Baking (sff)
T. Kingfisher Nine Goblins (sff)
Michael Lewis The Fifth Risk (nonfiction)
Paul McAuley War of the Maps (sff)
Gretchen McCulloch Because Internet (nonfiction)
Hayao Miyazaki Nausica of the Valley of the Wind (graphic novel)
Annalee Newitz The Future of Another Timeline (sff)
Nick Pettigrew Anti-Social (nonfiction)
Rivers Solomon, et al. The Deep (sff)
Jo Walton Or What You Will (sff)
Erik Olin Wright Stardust to Stardust (nonfiction) Of these, I've already read and reviewed The Fifth Risk (an excellent book).

Review: Rise of the Warrior Cop, by Radley Balko

Publisher:	PublicAffairs
Copyright:	2013
ISBN:	1-61039-212-4
Format:	Kindle
Pages:	336

As the United States tries, in fits and starts, to have a meaningful discussion about long-standing police racism, brutality, overreach, corruption, and murder, I've realized that my theoretical understanding of the history of and alternative frameworks for law enforcement is woefully lacking. Starting with a book by a conservative white guy is not the most ideal of approaches, but it's what I already had on hand, and it won't be the last book I read and review on this topic. (Most of my research so far has been in podcast form. I don't review those here, but I can recommend Ezra Klein's interviews with Ta-Nehisi Coates, Paul Butler, and, most strongly, sujatha baliga.) Rise of the Warrior Cop is from 2013 and has had several moments of fame, no doubt helped by Balko's connections to the conservative and libertarian right. One of the frustrating facts of US politics is that critiques of the justice system from the right (and from white men) get more media attention than critiques from the left. That said, it's a generally well-respected book on the factual history of the topic, and police brutality and civil rights are among the points on which I have stopped-clock agreements with US libertarians. This book is very, very libertarian. In my callow youth, I was an ardent libertarian, so I've read a lot of US libertarian literature. It's a genre with its own conventions that become obvious when you read enough of it, and Rise of the Warrior Cop goes through them like a checklist. Use the Roman Republic (never the Roman Empire) as the starting point for any political discussion, check. Analyze the topic in the context of pre-revolutionary America, check. Spend considerable effort on discerning the opinions of the US founders on the topic since their opinions are always relevant to the modern world, check. Locate some point in the past (preferably before 1960) where the political issue was as good as it has ever been, check. Frame all changes since then as an erosion of rights through government overreach, check. Present your solution as a return to a previous era of respect for civil rights, check. Once you start recognizing the genre conventions, their prevalence in libertarian writing is almost comical. The framing chapters therefore leave a bit to be desired, but the meat of the book is a useful resource. Starting with the 1970s and its use as a campaigning tool by Nixon, Balko traces a useful history of the war on drugs. And starting with the 1980s, the number of cites to primary sources and the evidence of Balko's own research increases considerably. If you want to know how US police turned into military cosplayers with body armor, heavy weapons, and armored vehicles, this book provides a lot of context and history. One of the reasons why I view libertarians as allies of convenience on this specific issue is that drug legalization and disgust with the war on drugs have been libertarian issues for decades. Ideologically honest libertarians (and Balko appears to be one) are inherently skeptical of the police, so when the police overreach in an area of libertarian interest, they notice. Balko makes a solid argument, backed up with statistics, specific programs, legislation, and court cases, that the drug war and its accompanying lies about heavily-armed drug dealers and their supposed threat to police officers was the fuel for the growth of SWAT teams, no-knock search warrants, erosion of legal protections for criminal defendants, and de facto license for the police to ignore the scope and sometimes even the existence of warrants. This book is useful support for the argument that fears for the safety of officers underlying the militarization of police forces are imaginary. One telling point that Balko makes repeatedly and backs with statistical and anecdotal evidence is that the police generally do not use raid tactics on dangerous criminals. On the contrary, aggressive raids are more likely to be used on the least dangerous criminals because they're faster, they're fun for the police (they provide an adrenaline high and let them play with toys), and they're essentially risk-free. If the police believe someone is truly dangerous, they're more likely to use careful surveillance and to conduct a quiet arrest at an unexpected moment. The middle-of-the-night armed break-ins with battering rams, tear gas, and flash-bangs are, tellingly, used against the less dangerous suspects. This is part of Balko's overall argument that police equipment and tactics have become untethered from any realistic threat and have become cultural. He traces an acceleration of that trend to 9/11 and the resulting obsession with terrorism, which further opened the spigot of military hardware and "special forces" training. This became a point of competition between police departments, with small town forces that had never seen a terrorist and had almost no chance of a terrorist incident demanding their own armored vehicles. I've encountered this bizarre terrorism justification personally; one of the reasons my local police department gave in a public hearing for not having a policy against shooting at moving vehicles was "but what if terrorism?" I don't believe there has ever been a local terrorist attack. SWAT in such places didn't involve the special training or dedicated personnel of large city forces; instead, it was a part-time duty for normal police officers, and frequently they were encouraged to practice SWAT tactics by using them at random for some otherwise normal arrest or search. Balko argues that those raids were more exciting than normal police work, leading to a flood of volunteers for that duty and a tendency to use them as much as possible. That in turn normalizes disconnecting police tactics from the underlying crime or situational risk. So far, so good. But despite the information I was able to extract from it, I have mixed feelings about Rise of the Warrior Cop as a whole. At the least, it has substantial limitations. First, I don't trust the historical survey of policing in this book. Libertarian writing makes for bad history. The constraints of the genre require overusing only a few points of reference, treating every opinion of the US founders as holy writ, and tying forward progress to a return to a previous era, all of which interfere with good analysis. Balko also didn't do the research for the historical survey, as is clear from the footnotes. The citations are all to other people's histories, not to primary sources. He's summarizing other people's histories, and you'll almost certainly get better history by finding well-respected historians who cover the same ground. (That said, if you're not familiar with Peel's policing principles, this is a good introduction.) Second, and this too is unfortunately predictable in a libertarian treatment, race rarely appears in this book. If Balko published the same book today, I'm sure he would say more about race, but even in 2013 its absence is strange. I was struck while reading by how many examples of excessive police force were raids on west coast pot farms; yes, I'm sure that was traumatic, but it's not the demographic I would name as the most vulnerable to or affected by police brutality. West coast pot growers are, however, mostly white. I have no idea why Balko made that choice. Perhaps he thought his target audience would be more persuaded by his argument if he focused on white victims. Perhaps he thought it was an easier and less complicated story to tell. Perhaps, like a lot of libertarians, he doesn't believe racism has a significant impact on society because it would be a market failure. Perhaps those were the people who more readily came to mind. But to talk about police militarization, denial of civil rights, and police brutality in the United States without putting race at the center of both the history and the societal effects leaves a gaping hole in the analysis. Given that lack of engagement, I also am dubious of Balko's policy prescriptions. His reform suggestions aren't unreasonable, but they stay firmly in the centrist and incrementalist camp and would benefit white people more than black people. Transparency, accountability, and cultural changes are all fine and good, but the cultural change Balko is focused on is less aggressive arrest tactics, more use of mediation, and better physical fitness. I would not object to those things (well, maybe the last, which seemed odd), but we need to have a discussion about police white supremacist organizations, the prevalence of spousal abuse, and the police tendency to see themselves not as public servants but as embattled warriors who are misunderstood by the naive sheep they are defending. And, of course, you won't find in Rise of the Warrior Cop any thoughtful wrestling with whether there are alternative approaches to community safety, whether punitive rather than restorative justice is effective, or whether crime is a symptom of deeper societal problems we could address but refuse to. The most radical suggestion Balko has is to legalize drugs, which is both the predictable libertarian position and, as we have seen from recent events in the United States, far from the only problem of overcriminalization. I understand why this book is so frequently mentioned on-line, and its author's political views may make it more palatable to some people than a more race-centered or radical perspective. But I don't think this is the best or most useful book on police violence that one could read today. I hope to find a better one in upcoming reviews. Rating: 6 out of 10

Review: Paladin's Grace, by T. Kingfisher

Publisher:	Red Wombat Studio
Copyright:	2020
ASIN:	B0848Q8JVW
Format:	Kindle
Pages:	399

Stephen was a paladin. Then his god died. He was a berserker, an unstoppable warrior in the service of his god. Now, well, he's still a berserker, but going berserk when you don't have a god to control the results is not a good idea. He and his brothers were taken in by the Temple of the Rat, where they serve as guards, watch out for each other, and try to get through each day with an emptiness in their souls where a god should be. Stephen had just finished escorting a healer through some of the poorer parts of town when a woman runs up to him and asks him to hide her. Their awkward simulated tryst is sufficient to fool the two Motherhood priests who were after her for picking flowers from the graveyard. Stephen then walks her home and that would have been the end of it, except that neither could get the other out of their mind. Despite first appearances, and despite being set in the same world and sharing a supporting character, this is not the promised sequel to Swordheart (which is apparently still coming). It's an entirely different paladin story. T. Kingfisher (Ursula Vernon's nom de plume when writing for adults) has a lot of things to say about paladins! And, apparently, paladin-involved romances. On the romance front, Kingfisher clearly has a type. The general shape of the story will be familiar from Swordheart and The Wonder Engine: An independent and occasionally self-confident woman with various quirks, a hunky paladin who is often maddeningly dense, and a lot of worrying on both sides about whether the other person is truly interested in them and if their personal liabilities make a relationship a horrible idea. This is not my preferred romance formula (it provokes the occasional muttered "for the love of god just talk to each other"), but I liked this iteration of it better than the previous two, mostly because of Grace. Grace is a perfumer, a trade she went into by being picked out of a lineup of orphans by a master perfumer for her sense of smell. One of Kingfisher's strengths as a writer is showing someone get lost in their routine day-to-day competence. When mixed with an inherently fascinating profession, this creates a great reading experience. Grace is also an abuse survivor, which made the communication difficulties with Stephen more interesting and subtle. Grace has created space and a life for herself, and her unwillingness to take risks on changes is a deep part of her sense of self and personal safety. As her past is slowly revealed, Kingfisher puts the reader in a position to share Stephen's anger and protectiveness, but then consistently puts Grace's own choices, coping mechanisms, and irritated refusal to be protected back into the center of the story. She has to accept some help as she gets entangled in the investigation of a highly political staged assassination attempt, but both that help and the relationship come on her own terms. It's very well-done. The plot was enjoyable enough, although it involved a bit too much of constantly rising stakes and turns for the worst for my taste, and the ending had a touch of deus ex machina. Like Kingfisher's other books, though, the delight is in the unexpected details. Stephen knitting socks. Grace's frustrated obsession with why he smells like gingerbread. The beautifully practical and respectful relationship between the Temple of the Rat and Stephen's band of former paladins. (After only two books in which they play a major role, the Temple of the Rat is already one of my favorite fantasy religions.) Everything about Bishop Beartongue. Grace's friend Marguerite. And a truly satisfying ending. The best part of this book, though, is the way Grace is shown as a complete character in a way that even most books with well-rounded characterization don't manage. Some things she does make the reader's heart ache because of the hints they provide about her past, but they're also wise and effective safety mechanisms for her. Kingfisher gives her space to be competent and prickly and absent-minded. She has a complete life: friends, work, goals, habits, and little rituals. Grace meets someone and falls in love, but one can readily imagine her not falling in love and going on with her life and that result wouldn't be tragic. In short, she feels like a grown adult who has made her own peace with where she came from and what she is doing. The book provides her an opportunity for more happiness and more closure without undermining her independence. I rarely see this in a novel, and even more rarely done this well. If you haven't read any of Kingfisher's books and are in the mood for faux-medieval city romance involving a perfumer and a bit of political skulduggery, this is a great place to start. If you liked Swordheart, you'll probably like Paladin's Grace; like me, you may even like it a bit more. Recommended, particularly if you want something light and heart-warming. Rating: 8 out of 10

At DebConf 17 in Montreal, I gave a talk about Ayatana Indicators [1] and the project's goal to continue the by then already dropped out of maintenance Ubuntu Indicators in a separate upstream project, detached from Ubuntu and its Ubuntu'isms. Stalling The whole Ayatana Indicators project received a bit of a show stopper by the fact that the IDO (Indicator Display Object) rendering was not working in vanilla GTK-3 without a certain patch [2] that only Ubuntu has in their GTK-3 package. Addressing GTK developers upstream some years back (after GTK 3.22 had already gone into long term maintenance mode) and asking for a late patch acceptance did not work out (as already assumed). Ayatana Indicators stalled at a level of 90% actually working fine, but those nice and shiny special widgets, like the calendar widget, the audio volume slider widgets, switch widgets, etc. could not be rendered appropriately in GTK based desktop environments (e.g. via MATE Indicator Applet) on other distros than Ubuntu. I never really had the guts to sit down without a defined ending and find a patch / solution to this nasty problem. Ayatana Indicators stalled as a whole. I kept it alive and defended its code base against various GLib and what-not deprecations and kept it in Debian, but the software was actually partially broken / dysfunctional. Taking the Dog for a Walk and then It Became all Light+Love Several days back, I received a mail from Robert Tari [3]. I was outside on a hike with our dog and thought, ah well, let's check emails... I couldn't believe what I read then, 15 seconds later. I could in fact, hardly breathe... I have known Robert from earlier email exchanges. Robert maintains various "little" upstream projects, like e.g. Caja Rename, Odio, Unity Mail, etc. that I have looked into earlier regarding Debian packaging. Robert is also a Manjaro contributor and he has been working on bringing Ayatana Indicators to Manjaro MATE. In the early days, without knowing Robert, I even forked one of his projects (indicator-notification) and turned it into an Ayatana Indicator. Robert and I also exchanged some emails about Ayatana Indicators already a couple of weeks ago. I got the sense of him maybe being up to something already then. Oh, yeah!!! It turned out that Robert and I share the same "love" for the Ubuntu Indicators concept [4]. From his email, it became clear that Robert had spent the last 1-2 weeks drowned in the Ayatana IDO and libayatana-indicator code and worked him self through the bowels of it in order to understand the code concept of Indicators to its very depth. When emerging back from his journey, he presented me (or rather: the world) a patch [5] against libayatana-indicator that makes it possible to render IDO objects even if a vanilla GTK-3 is installed on the system. This patch is a game changer for Indicator lovers. When Robert sent me his mail pointing me to this patch, I think, over the past five years, I have never felt more excited (except from the exact moment of getting married to my wife two-to-three years ago) than during that moment when my brain tried to process his email. "Like a kid on Christmas Eve...", Robert wrote in one of his later mails to me. Indeed, like a "kid on Christmas Eve", Robert. Try It Out As a proof of all this to the Debian people, I have just done the first release of ayatana-indicator-datetime and uploaded it to Debian's NEW queue. Robert is doing the same for Manjaro. The Ayatana Indicator Sound will follow after my vacation. For fancy widget rendering in Ayatana Indicator's system indicators, make sure you have libayatana-indicator 0.7.0 or newer installed on your system. Credits One of the biggest thanks ever I send herewith to Robert Tari! Robert is now co-maintainer of Ayatana Indicators. Welcome! Now, there is finally a team of active contributors. This is so delightful!!! References

• [1] https://ayatanaindicators.github.io/
• [2] https://salsa.debian.org/gnome-team/gtk3/-/blob/ubuntu/master/debian/pat...
• [3] https://tari.in/www/articles/ayatana/
• [4] https://wiki.ubuntu.com/StatusBar
• [5] https://github.com/AyatanaIndicators/libayatana-indicator/commit/b619354...

P.S. Expect more Ayatana Indicators to appear in your favourite distro soon...

Better late than never as we say thanks to the work of Daniel Leidert and Jorge Maldonado Ventura, we managed to complete the update of my book for Debian 10 Buster. You can get the electronic version on debian-handbook.info or the paperback version on lulu.com. Or you can just read it online. Translators are busy updating their translations, with German and Norvegian Bokmal leading the way

One comment Liked this article? Click here. My blog is Flattr-enabled.