Partners holding big jigsaw puzzle pieces flat vector illustration. Successful partnership, communication and collaboration metaphor. Teamwork and business cooperation concept.

I write this in the wake of a personal attack against my work and a project that is near and dear to me. Instead of spreading vile rumors and hearsay, talk to me. I am not known to be hard to talk to and am wide open for productive communication. I am disheartened and would like to share some thoughts of the importance of communication. Thanks for listening. Open source development thrives on collaboration, shared knowledge, and mutual respect. Yet sometimes, the very passion that drives us to contribute can lead to misunderstandings and conflicts that harm both individuals and the projects we care about. As contributors, maintainers, and community members, we have a responsibility to foster environments where constructive dialogue flourishes.

The Foundation of Healthy Open Source Communities At its core, open source is about people coming together to build something greater than what any individual could create alone. This collaborative spirit requires more than just technical skills it demands emotional intelligence, empathy, and a commitment to treating one another with dignity and respect. When disagreements arise and they inevitably will the manner in which we handle them defines the character of our community. Technical debates should focus on the merits of ideas, implementations, and approaches, not on personal attacks or character assassinations conducted behind closed doors.

The Importance of Direct Communication One of the most damaging patterns in any community is when criticism travels through indirect channels while bypassing the person who could actually address the concerns. When we have legitimate technical disagreements or concerns about someone s work, the constructive path forward is always direct, respectful communication. Consider these approaches:

• Address concerns directly: If you have technical objections to someone s work, engage with them directly through appropriate channels
• Focus on specifics: Critique implementations, documentation, or processes not the person behind them
• Assume good intentions: Most contributors are doing their best with the time and resources available to them
• Offer solutions: Instead of just pointing out problems, suggest constructive alternatives

Supporting Contributors Through Challenges Open source contributors often juggle their community involvement with work, family, and personal challenges. Many are volunteers giving their time freely, while others may be going through difficult periods in their lives job searching, dealing with health issues, or facing other personal struggles. During these times, our response as a community matters enormously. A word of encouragement can sustain someone through tough periods, while harsh criticism delivered thoughtlessly can drive away valuable contributors permanently.

Building Resilient Communities Strong open source communities are built on several key principles: Transparency in Communication: Discussions about technical decisions should happen in public forums where all stakeholders can participate and learn from the discourse. Constructive Feedback Culture: Criticism should be specific, actionable, and delivered with the intent to improve rather than to tear down. Recognition of Contribution: Every contribution, whether it s code, documentation, bug reports, or community support, has value and deserves acknowledgment. Conflict Resolution Processes: Clear, fair procedures for handling disputes help prevent minor disagreements from escalating into community-damaging conflicts.

The Long View Many successful open source projects span decades, with contributors coming and going as their life circumstances change. The relationships we build and the culture we create today will determine whether these projects continue to attract and retain the diverse talent they need to thrive. When we invest in treating each other well even during disagreements we re investing in the long-term health of our projects and communities. We re creating spaces where innovation can flourish because people feel safe to experiment, learn from mistakes, and grow together.

Moving Forward Constructively If you find yourself in conflict with another community member, consider these steps:

1. Take a breath: Strong emotions rarely lead to productive outcomes
2. Seek to understand: What are the underlying concerns or motivations?
3. Communicate directly: Reach out privately first, then publicly if necessary
4. Focus on solutions: How can the situation be improved for everyone involved?
5. Know when to step back: Sometimes the healthiest choice is to disengage from unproductive conflicts

A Call for Better Open source has given us incredible tools, technologies, and opportunities. The least we can do in return is treat each other with the respect and kindness that makes these collaborative achievements possible. Every contributor whether they re packaging software, writing documentation, fixing bugs, or supporting users is helping to build something remarkable. Let s make sure our communities are places where that work can continue to flourish, supported by constructive communication and mutual respect. The next time you encounter work you disagree with, ask yourself: How can I make this better? How can I help this contributor grow? How can I model the kind of community interaction I want to see? Our projects are only as strong as the communities that support them. Let s build communities worthy of the amazing software we create together. https://gofund.me/506c910c

Snapshot mirroring in Debian (and Ubuntu) The use of snapshots has been routine in both Debian and Ubuntu for several years now or more than 15 years for Debian, to be precise. Snapshots have become not only very reliable, but also an increasingly important part of the Debian package archive. This week, I encountered a problem at work that could be perfectly solved by correctly using the Snapshot service. However, while trying to figure it out, I ran into some shortcomings in the documentation. Until the docs are updated, I am publishing this blog post to make this information easier to find. Problem 1: Ensure fully reproducible creation of Docker containers with the exact same packages installed, even years after the original images were generated. Solution 1: Pin everything! Use a pinned source image in the FROM statement, such as debian:trixie-20250721-slim, and also pin the APT package sources to the "same" date - "20250722". Hint: The APT packages need to be newer than the Docker image base. If the APT packages are a bit newer, that's not a problem, as APT can upgrade packages without issues. However, if your Docker image has a newer package than your APT package sources, you will have a big problem. For example, if you have "libbearssl0" version 0.6-2 installed in the Docker image, but your package sources only have the older version 0.6-1, you will fail when trying to install the "libbearssl-dev" package. This is because you only have version 0.6-1 of the "-dev" package available, which hard-depends on exactly version 0.6-1 of "libbearssl0", and APT will refuse to downgrade an already installed package to satisfy that dependency. Problem 2: You are using a lot of images in a lot of executions and building tens of thousands of images per day. It would be a bad idea to put all this load on public Debian servers. Using local sources is also faster and adds extra security. Solution 2: Use local (transparently caching) mirrors for both the Docker Hub repository and the APT package source. At this point, I ran into another issue I could not easily figure out how to specify a local mirror for the snapshot part of the archive service. First of all, snapshot support in both Ubuntu and Debian accepts both syntaxes described in the Debian and Ubuntu documentation above. The documentation on both sites presents different approaches and syntax examples, but both work. The best approach nowadays is to use the "deb822" sources syntax. Remove /etc/apt/sources.list (if it still exists), delete all contents of the /etc/apt/sources.list.d directory, and instead create this file at /etc/apt/sources.list.d/debian.sources: Hint: This assumes you have a mirror service running at common.mirror-proxy.local that proxies requests (with caching) to whitelisted domains, based on the name of the first folder in the path. If you now run sudo apt update --print-uris, you will see that your configuration accesses your mirror, but does not actually use the snapshot. Next, add the following to /etc/apt/apt.conf.d/80snapshots: That should work, right? Let's try sudo apt update --print-uris again. I've got good news and bad news! The good news is that we are now actually using the snapshot we specified (twice). The bad news is that we are completely ignoring the mirror and going directly to snapshots.debian.org instead. Finding the right information was a bit of a challenge, but after a few tries, this worked: to specify a custom local mirror of the Debian (or Ubuntu) snapshot service, simply add the following line to the same file, /etc/apt/apt.conf.d/80snapshots: Now, if you check again with sudo apt update --print-uris, you will see that the requests go to your mirror and include the specified snapshot identifier. Success! Now you can install any packages you want, and everything will be completely local and fully reproducible, even years later!

Debusine aims to be an integrated solution to build, distribute and maintain a Debian-based distribution. At Debconf 25, we talked about using it to pre-test uploads to Debian unstable, and also touched on how Freexian is using it to help maintain the Debian LTS and ELTS projects. When Debian 10 (buster) moved to ELTS status in 2024, this came with a new difficulty that hadn t existed for earlier releases. Debian 10 added UEFI Secure Boot support, meaning that there are now signed variants of the boot loader and Linux kernel packages. Debian has a system where certain packages are configured as needing to be signed, and those packages include a template for a source package along with the unsigned objects themselves. The signing service generates detached signatures for all those objects, and then uses the template to build a source package that it uploads back to the archive for building in the usual way. Once buster moved to ELTS, it could no longer rely on Debian s signing service for all this. Freexian operates parallel infrastructure for the archive, and now needed to operate a parallel signing service as well. By early 2024 we were already planning to move ELTS infrastructure towards Debusine, and so it made sense to build a signing service there as well. Separately, we were able to obtain a Microsoft signature for Freexian s shim build, allowing us to chain this into the trust path for most deployed x86 machines. Freexian can help other organizations running Debian derivatives through the same process, and can provide secure signing infrastructure to the standards required for UEFI Secure Boot.

Prior art We considered both code-signing (Debian s current implementation) and lp-signing (Ubuntu s current implementation) as prior art. Neither was quite suitable for various reasons.

• code-signing relies on polling a configured URL for each archive to fetch a GPG-signed list of signing requests, which would have been awkward for us to set up, and it assumes that unsigned packages are sufficiently trusted for it to be able to run dpkg -x and dpkg-source -b on them outside any containment. dpkg -x has had the occasional security vulnerability, so this seemed unwise for a service that might need to deal with signing packages for multiple customers.
• lp-signing is a microservice accepting authenticated requests, and is careful to avoid needing to manipulate packages itself. However, this relies on a different and incompatible mechanism for indicating that packages should be signed, which wasn t something we wanted to introduce in ELTS.

Workers Debusine already had an established system of external workers that run tasks under various kinds of containment. This seems like a good fit: after all, what s a request to sign a package but a particular kind of task? But there are some problems here: workers can run essentially arbitrary code (such as build scripts in source packages), and even though that s under containment, we don t want to give such machines access to highly-sensitive data such as private keys. Fortunately, we d already introduced the idea of different kinds of workers a few months beforehand, in order to be able to run privileged server tasks that have direct access to the Debusine database. We built on that and added signing workers , which are much like external workers except that they only run signing tasks, no other types of tasks run on them, and they have access to a private database with information about the keys managed by their Debusine instance. (Django s support for multiple databases made this quite easy to arrange: we were able to keep everything in the same codebase.)

Key management It s obviously bad practice to store private key material in the clear, but at the same time the signing workers are essentially oracles that will return signatures on request while ensuring that the rest of Debusine has no access to private key material, so they need to be able to get hold of it themselves. Hardware security modules (HSMs) are designed for this kind of thing, but they can be inconvenient to manage when large numbers of keys are involved. Some keys are more valuable than others. If the signing key used for an experimental archive leaks, the harm is unlikely to be particularly serious; but if the ELTS signing key leaks, many customers will be affected. To match this, we implemented two key protection arrangements for the time being: one suitable for low-value keys encrypts the key in software with a configured key and stores the public key and ciphertext in the database, while one suitable for high-value keys stores keys as PKCS #11 URIs that can be set up manually by an instance administrator. We packaged some YubiHSM tools to make this easier for our sysadmins. The signing worker calls back to the Debusine server to check whether a given work request is authorized to use a given signing key. All operations related to private keys also produce an audit log entry in the private signing database, so we can track down any misuse.

Tasks Getting Debusine to do anything new usually requires figuring out how to model the operation as a task. In this case, that was complicated by wanting to run as little code as possible on the signing workers: in particular, we didn t want to do all the complicated package manipulations there. The approach we landed on was a chain of three tasks:

• ExtractForSigning runs on a normal external worker. It takes the result of a package build and picks out the individual files from it that need to be signed, storing them as separate artifacts.
• Sign runs on a signing worker, and (of course) makes the actual signatures, storing them as artifacts.
• AssembleSignedSource runs on a normal external worker. It takes the signed artifacts and produces a source package containing them, based on the template found in the unsigned binary package.

Workflows Of course, we don t want people to have to create all those tasks directly and figure out how to connect everything together for themselves, and that s what workflows are good at. The make_signed_source workflow does all the heavy lifting of creating the right tasks with the right input data and making them depend on each other in the right ways, including fanning out multiple copies of all this if there are multiple architectures or multiple template packages involved. Since you probably don t want to stop at just having the signed source packages, it also kicks off builds to produce signed binary packages. Even this is too low-level for most people to use directly, so we wrapped it all up in our debian_pipeline workflow, which just needs to be given a few options to enable signing support (and those options can be locked down by workspace owners).

What s next? In most cases this work has been enough to allow ELTS to carry on issuing kernel security updates without too much disruption, which was the main goal; but there are other uses for a signing system. We included OpenPGP support from early on, which allows Debusine to sign its own builds, and we ll soon be extending that to sign APT repositories hosted by Debusine. The current key protection arrangements could use some work. Supporting automatically-generated software-encrypted keys and manually-generated keys in an HSM is fine as far as it goes, but it would be good to be able to have the best of both worlds by being able to automatically generate keys protected by an HSM. This needs some care, as HSMs often have quite small limits on the number of objects they can store at any one time, and the usual workaround is to export keys from the HSM under wrap (encrypted by a key known only to the HSM) so that they can be imported only when needed. We have a general idea of how to do this, but doing it efficiently will need care. We d be very interested in hearing from organizations that need this sort of thing, especially for Debian derivatives. Debusine provides lots of other features that can help you. Please get in touch with us at sales@freexian.com if any of this sounds useful to you.

Debconf 25 photos Debconf 25 came to the end in Brest, France this year a couple weeks ago. This has been a very different and unusually interesting Debconf. For me it was for two, related reasons: for one the conference was close enough in Western Europe that I could simply drive there with a car (which reminds me that I should make a blog post about the BMW i5, before I am done with it at the end of this year) and for the other - the conference is close enough to Western Europe that many other Debian developers could join this year who have not been seen at the event for many years. Being able to arrive early, decompress and spend extra time looking around the place made the event itself even more enjoyable than usual. The French cuisine, especially in its Breton expression, has been a very welcome treat. Even if there were some rough patches with the food selection, amount, or waiting, it was still a great experience. I specifically want to say a big thank you to the organisers for everything, but very explicitly for planning all the talk/BOF rooms in the same building and almost on the same floor. It saved me a lot of footwork, but also for other participants the short walks between the talks made it possible to always have a few minutes to talk to people or grab a croissant before running to the next talk. IMHO we should come back to a tradition of organising Debconf in Europe every 2-3 years. This maximises one of the main goals of Debconf - bringing as many Debian Developers as possible together in one physical location. This works best when the location is actually close to large concentrations of existing developers. In other years, the other goal of Debconf can then take priority - recruiting new developers in new locations. However, these goals could both be achieved at the same time - there are plenty of locations in Europe and even in Western Europe that still have good potential for attracting new developers. Especially if we focus on organising the event on the campuses of some larger technically-oriented universities. This year was also very productive for me a lot of conversations with various people about all kinds of topics, especially technical packaging questions. It has been a long time since the very basic foundations of Debian packaging work have been so fundamentally refactored and modernized as in the past year. Tag2upload has become a catalyst for git-based packaging and for automated workflows via Salsa, and all of that feeds back into focusing on a few best-supported packaging workflows. There is still a bit of a documentation gap of a new contributor getting to these modern packaging workflows from the point where the New Maintainers Guide stops. In any case, next year Debconf will be happening in Santa Fe, Argentina. And the year after that it is all still open and in a close competition between Japan, Spain, Portugal, Brazil and .. El Salvador? Personally, I would love to travel to Japan (again), but Spain or Portugal would also be great locations to meet more European developers again. As for Santa Fe ... it is quite likely that I will not be able to make it there next year, for (planned) health reasons. I guess I should also write a new blog post about what it means to be a Debconf Photographer, so that someone else could do this as well, and also reduce the "bus factor" along the way. But before that - here is the main group photo from this year: You can also see it on:

• Google Photos

You can also enjoy the rest of the photos:

• on git-lfs
• on Google Photos

Additionally, check out photos from other people on GIT LFS and consider adding your own photos there as well. Other places I have updated with up-to-date information are these wiki pages:

• All Debconf Group Photos
• All photos from Debconf25

If you took part in the playing cards event, then check your photo in this folder and link to your favourite from your line in the playing card wiki

In July I attended DebCamp and DebConf in Brest, France. I very much enjoyed the opportunity to reconnect with other Debian contributors in person. I had a number of interesting and fruitful conversations there, besides the formally organised BoFs and talks. I also gave my own talk on What s new in the Linux kernel (and what s missing in Debian). Here s the usual categorisation of activity:

• Debian packages:
  • firmware-nonfree:
    • Bugs:
      • replied to #1093084: firmware-amd-graphics: Some graphical glitches and occasional stuttering with AMD graphics
      • closed #1100723: firmware-misc-nonfree: ralink firmware disapears after upgrade due backport firmware-misc-free split in multiple packages
      • replied to and closed #1108250: firmware-intel-graphics: i915 timeout errors after firmware upgrade
      • replied to and reassigned #1108599: firmware-linux-free: please add support MT7921/MT7925 WIFI Combo Cards
      • replied to #1109015: Wifi not working anymore with new atheros firmware from experimental
    • Uploads:
      • uploaded version 20250627-1 to experimental
      • uploaded version 20250708-1 to experimental
  • initramfs-tools:
    • Bugs:
      • replied to #1099456: initramfs-tools: misleading error messages
    • Merge requests:
      • reviewed !172: Use 3cpio for unmkinitramfs/lsinitramfs if available
      • reviewed and merged !179: update-initramfs: add copyright information
      • opened and merged !180: hook-functions: Add Cadence USB driver to base
      • reviewed !182: unmkinitramfs: handle both uppercase and lowercase hex numbers
  • ktls-utils:
    • Uploads:
      • uploaded version 1.2.1-1 to experimental
  • kup:
    • Bugs:
      • closed #1064112: kup-client: Recommends transitional package gnupg-agent instead of gpg-agent
    • Uploads:
      • uploaded version 0.3.6-5 to experimental
  • lintian:
    • Bugs:
      • replied to #1051140: lintian in bookworm does not know about any bookworm* suites
  • linux:
    • Bugs:
      • closed #763547: firmware-linux: please include ihex2fw tool into one of linux-* packages
      • closed #1016718: binfmt_elf: May fail to load executable with no static data next to BSS
      • replied to #1035878: rpi400: visual speckling on faulty HDMI port during mouse movement
      • closed #1056357: general: on hppa, command line of any program invocation is limited to less than 3544 bytes
      • replied to #1086638: linux-image-6.11.5: usbguard-daemon invalid opcode: 0000, usb s not usable
      • closed #1092931: Most recent kernel is causing issues with touchpad function.
      • replied to #1102175: Repeated lockups in AMDGPU DRM driver
      • replied to #1102522: linux-image-6.12.21-amd64: Fails to suspend Acer TravelMate Spin B311R-31
      • replied to #1104165: linux-image-6.12.22-amd64: Freezing with amdgpu_dm_plane_helper_prepare_fb bind failed some time after returning from suspend
      • replied to #1104670: linux-image-6.12.25-amd64: system does not shut down - GHES: Fatal hardware error
      • replied to #1107479: util-linux: blkid hangs forever after inserting a DVD-RAM
      • replied to #1107785: CLOCK_MONOTONIC increases in s2idle, triggering watchdogs, killing processes
      • replied to #1108037: linux-perf: perf report demangle only works on some(?) symbols
      • replied to #1108168: linux-image-6.12.32-amd64: black screen, then crash in nouveau when rebooting
      • replied to #1108294: kernel 6.12 breaks cgroup awareness of openjdk 21
      • closed #1108430: Kernel Regression between 6.1.0-35 and 6.1.0-37: IPv6 link-local multicast over GRE tunnel is silently dropped
      • replied to #1108496: linux-image-6.12.30+bpo-amd64: USB audio glitches: retire_capture_urb
      • replied to #1108965: linux: Enable RTL8851BE module
      • replied to #1109038: breaks with rtl8139 network interface via usb
      • replied to and closed #1109040: Subject: linux-headers-6.1.0-37-amd64 missing config files; cannot build external modules on Bookworm
      • replied to #1109666: Fails to boot on Marvell Armada 385
      • replied to #1109776: Support for non-4kB pages in Xe driver
      • replied to #1109891: hyperv-daemons no longer available for bullseye/arm64
    • Merge requests:
      • merged !1502: Add linux-misc-tools package for bootconfig, ihex2fw, thermometer, and tmon
      • reviewed !1542: 6.1 stable updates
      • reviewed !1543: Draft: [arm64] Provide an installer package for usbgadget usage
      • reviewed !1560: [arm64] Enable Qualcomm RB1 boards
      • reviewed and merged !1565: [cloud] Enable SATA_AHCI
      • opened and merged !1569: Update to 6.16-rc5
      • opened !1570: rtw89: Enable RTW89_8851BE, RTW89_8852BTE as modules
      • opened !1571: rtw88: Enable RTW88_8821AU, RTW88_8812AU, RTW88_8814AE, RTW88_8814AU as modules
      • opened !1572: Revert cgroup: Do not report unavailable v1 controllers in /proc/cgroups
      • opened !1573: hyperv-daemons: Fix and install the sample network info scripts
      • opened !1576: Clean up tools build
      • merged !1580: Update to 6.16-rc6
      • merged !1586: Update to 6.16-rc7
      • opened !1587: binder: Add module description, fixing warning from modpost
      • opened !1588: linux-misc-tools: Fix FTBFS on sparc64
      • reviewed !1591: Move all package specific build-deps
    • Uploads:
      • uploaded version 6.12.33-1~bpo12+1 to bookworm-backports
      • uploaded version 6.12.35-1~bpo12+1 to bookworm-backports
      • uploaded version 6.12.38-1~bpo12+1 to bookworm-backports
      • uploaded version 6.16~rc7-1~exp1 to experimental
    • (LTS) Updated the bullseye-security branch, but did not upload it:
      • updated to upstream version 5.10.240
      • enabled building hyperv-daemons for arm64
  • (LTS) linux-6.1:
    • Uploads:
      • uploaded version 6.1.140-1~deb11u1 to bullseye-security
  • linux-base:
    • Bugs:
      • closed #1010699: linux: Please touch /run/reboot-required in postinst
      • closed #1041189: general: Optimize kernel settings for gaming
    • Uploads:
      • uploaded version 4.13 to experimental
  • nfs-utils:
    • Bugs:
      • closed #1109999: [nfs-common] default option ($ STATDOPTS ) is not used
  • sbuild:
    • Bugs:
      • opened #1109103: Cross-build with profiles does not set DEB_BUILD_OPTIONS=nocheck
  • wireless-regdb:
    • Uploads:
      • uploaded version 2025.07.10-1 to unstable
    • (LTS) Prepared backports of the above version to bookworm and bullseye-security
• Debian non-package bugs:
  • release.debian.org:
    • opened #1109500: unblock: wireless-regdb/2025.07.10-1
  • wnpp:
    • replied to and closed #1072579: ITP: cis-tools CIS file tools from pcmcia-cs
    • replied to #1109020: ITP: sphinxcontrib-youtube Sphinx extension that allows you to embed YouTube and Vimeo
• Mailing lists:
  • cgroups:
    • replied to [PATCH 4/4] cgroup: Do not report unavailable v1 controllers in /proc/cgroups
  • debian-boot:
    • replied to Installer freeze on Gigabyte X870E Aorus Master, ath12k wifi related
  • debian-devel:
    • replied to Building multiple flavors of a package with a HUGE upstream
  • debian-kernel:
    • posted Agenda items for kernel-team meeting on 2025-07-16
    • posted Agenda items for kernel-team meeting on 2025-07-30
    • posted Installation location of BPF vmlinux.h
    • replied to Request to backport CONFIG_UCLAMP_TASK to 6.12 stable
  • linux-block:
    • replied to Bug#1107479: util-linux: blkid hangs forever after inserting a DVD-RAM
  • linux-firmware:
    • replied to Bug#1109015: Wifi not working anymore with new atheros firmware from experimental
  • linux-pci:
    • replied to Bug#1104670: linux-image-6.12.25-amd64: system does not shut down - GHES: Fatal hardware error
    • posted Failure to resume ASMedia ASM4242 on AMD systems
  • linux-pm:
    • posted User-space watchdog timers vs suspend-to-idle
  • linux-rt-users:
    • posted PREEMPT_RT vs i915
  • linux-sh:
    • posted [PATCH v2] sh: Do not use hyphen in exported variable name
  • linux-trace-kernel:
    • posted and replied to [PATCH] bootconfig: Fix unaligned access when building footer
  • stable:
    • replied to [PATCH 2/5] x86/bugs: Add a Transient Scheduler Attacks mitigation

Handling malicious requests with fail2ban I've been receiving a lot of malicious requests for a while now, so I decided to try out fail2ban as a possible solution. I see fail2ban as nice to have tool that is useful to keep down the "noise", but I wouldn't rely on it for security. If you need a tool to block unauthorized attempts or monitor log files excessively, you are probably doing something wrong. I'm currently using fail2ban 1.0.2-2 from Debian Bookworm. Unfortunatly, I quickly ran into a problem, fail2ban doesn't work out of the box with this version:

systemd[1]: Started fail2ban.service - Fail2Ban Service.
fail2ban-server[2840]: 2025-07-28 14:40:13,450 fail2ban.configreader   [2840]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
fail2ban-server[2840]: 2025-07-28 14:40:13,456 fail2ban                [2840]: ERROR   Failed during configuration: Have not found an y log file for sshd jail
fail2ban-server[2840]: 2025-07-28 14:40:13,456 fail2ban                [2840]: ERROR   Async configuration of server failed
systemd[1]: fail2ban.service: Main process exited, code=exited, status=255/EXCEPTION
systemd[1]: fail2ban.service: Failed with result 'exit-code'.

The good news is that this issue has already been addressed for Debian Trixie. Since I prefer to manage my own configuration, I removed the default file at /etc/fail2ban/jail.d/defaults-debian.conf and replaced it with a custom setup. To fix the earlier issue, I also added a systemd backend to the sshd jail so it would stop expecting a logpath. Here's the configuration I'm using:

$ cat /etc/fail2ban/jail.d/custom.conf 
[DEFAULT]
maxretry = 3
findtime = 24h
bantime  = 24h
[nginx-bad-request]
enabled  = true
port     = http,https
filter   = nginx-bad-request
logpath  = /var/log/nginx/access.log
[nginx-botsearch]
enabled  = true
port     = http,https
filter   = nginx-botsearch
logpath  = /var/log/nginx/access.log
[sshd]
enabled  = true
port     = ssh
filter   = sshd
backend  = systemd

I like to make things explicit, so I did repeat some lines from the default jail.conf file. In the end, I'm quite happy with it so far. Soon after I set it up, fail2ban was already banning a few hosts.

$ sudo fail2ban-client status nginx-bad-request
Status for the jail: nginx-bad-request
 - Filter
    - Currently failed: 42
    - Total failed: 454
 - Actions
    - Currently banned: 12
    - Total banned: 39

This method is for wayland based systems. There are better ways to do this on GNOME or KDE desktops, but the method we are going to use is independent of DE/WM that you are using. I am doing this on sway window manager, but you can try this on any other Wayland based WM or DE. I have not tried this on Xorg based systems, there are several other guides for Xorg based systems online. When we connect a physical monitor to our laptops, it creates a second display output in our display settings that we can then re-arrange in layout, set resolution, set scale etc. Since we are not connecting via a physical interface like HDMI, DP, VGA etc. We need to create a virtual display within our system and set the display properties manually.

Get a list of current display outputs. You can also just check it in display settings of your DE/WM with wdisplays

rajudev@sanganak ~> swaymsg -t get_outputs
Output LVDS-1 &aposSeiko Epson Corporation 0x3047 Unknown&apos (focused)
  Current mode: 1366x768 @ 60.002 Hz
  Power: on
  Position: 0,0
  Scale factor: 1.000000
  Scale filter: nearest
  Subpixel hinting: rgb
  Transform: normal
  Workspace: 2
  Max render time: off
  Adaptive sync: disabled
  Allow tearing: no
  Available modes:
    1366x768 @ 60.002 Hz

Single physical display of the laptop

Currently we are seeing only one display output. Our goal is to create a second virtual display that we will then share on the tablet/phone. To do this there are various tools available. We are using sway-vdctl . It is currently not available within Debian packages, so we need to install it manually.

$ git clone https://github.com/odincat/sway-vdctl.git
$ cd sway-vdctl
$ cargo build --release

This will generate the binary with the name main under target/release . We can then copy this binary to our bin folder.

$ sudo cp target/release/main /usr/local/bin/vdctl

Now we have the vdctl command available.

$ vdctl --help
Usage: vdctl [OPTIONS] <ACTION> [VALUE]
Arguments:
  <ACTION>
          Possible values:
          - create:      Create new output based on a preset
          - kill:        Terminate / unplug an active preset
          - list:        List out active presets
          - next-number: Manually set the next output number, in case something breaks
          - sync-number: Sync the next output number using &aposswaymsg -t get_outputs&apos
  [VALUE]
          Preset name to apply, alternatively a value
          
          [default: ]
Options:
      --novnc
          do not launch a vnc server, just create the output
  -h, --help
          Print help (see a summary with &apos-h&apos)

Before creating the virtual display, we need to set it&aposs properties at .config/vdctl/config.json . I am using Xiaomi Pad 6 tablet as my external display. You can adjust the properties according to the device you want to use as a second display. $ (text-editor) .config/vdctl/config.json

 
    "host": "0.0.0.0",
    "presets": [
         
            "name": "pad6",
            "scale_factor": 2,
            "port": 9901,
            "resolution":  
                "width": 2800,
                "height": 1800
             
       
    ]
 

In the JSON, you can set the display resolution according to your external device and other configurations. If you want to configure multiple displays, you can add another entry into the presets in the json file. You can refer to example json file into the git repository. Now we need to actually create the virtual monitor.

$ vdctl create pad6
Created output, presumably &aposHEADLESS-1&apos
Set resolution of &aposHEADLESS-1&apos to 2800x1800
Set scale factor of &aposHEADLESS-1&apos to 2
Preset &apospad6&apos (&aposHEADLESS-1&apos: 2800x1800) is now active on port 9901

Now if you will check the display outputs in your display settings or from command line, you will see two different displays.

$ swaymsg -t get_outputs
Output LVDS-1 &aposSeiko Epson Corporation 0x3047 Unknown&apos
  Current mode: 1366x768 @ 60.002 Hz
  Power: on
  Position: 0,0
  Scale factor: 1.000000
  Scale filter: nearest
  Subpixel hinting: rgb
  Transform: normal
  Workspace: 2
  Max render time: off
  Adaptive sync: disabled
  Allow tearing: no
  Available modes:
    1366x768 @ 60.002 Hz
Output HEADLESS-1 &aposUnknown Unknown Unknown&apos (focused)
  Current mode: 2800x1800 @ 0.000 Hz
  Power: on
  Position: 1366,0
  Scale factor: 2.000000
  Scale filter: nearest
  Subpixel hinting: unknown
  Transform: normal
  Workspace: 3
  Max render time: off
  Adaptive sync: disabled
  Allow tearing: no

Also in the display settings.

Display settings on Wayland with physical and virtual monitor output

Now we need to make this virtual display available over VNC which we will access with a VNC client on the tablet. To accomplish this I am using wayvnc but you can use any VNC server package. Install wayvnc

$ sudo apt install wayvnc

Now we will serve our virtual display HEADLESS-1 with wayvnc.

$ wayvnc -o HEADLESS-1 0.0.0.0 5900

You can adjust the port number as per your need. The process from laptop side is done. Now install any VNC software on your tablet. I am using AVNC, which is available on F-Droid. In the VNC software interface, add a new connection with the IP address of your laptop and the port started by wayvnc. Remember, both your laptop and phone need to be on the same Wi-Fi network.

AVNC interface with the connection details to connect to the virtual monitor.

Save and connect. Now you will be able to see a extended display on your tablet. Enjoy working with multiple screens in a portable setup. Till next time.. Have a great time.

I am always looking for ways of increasing the compute power I have at a reasonable price. I am very happy with my HP z840 dual CPU workstation [1] that I m using as a server and my HP z640 single CPU workstation [2]. Both of them were available second hand at quite reasonable prices and could be cheaply upgraded to faster CPUs. But if I can get something a lot faster for a reasonable price then I ll definitely get it. Socket LGA2011-v3 The home server and home workstation I currently use have socket LGA2011-v3 [3] which supports the E5-2699A v4 CPU which gives a rating of 26,939 according to Passmark [4]. That Passmark score is quite decent, you can get CPUs using DDR4 RAM that go up to almost double that but it s a reasonable speed and it works in systems that are readily available at low prices. The z640 is regularly on sale for less than $400AU and the z840 is occasionally below $600. The Dell PowerEdge T430 is an ok dual-CPU tower server using the same socket. One thing that s not well known is that is it limited to something like 135W per CPU when run with two CPUs. So it will work correctly with a single E5-2697A v4 with 145W TDP (I ve tested that) but will refuse to boot with two of them. In my test system I tried replacing the 495W PSUs with 750W PSUs and it made no difference, the motherboard has the limit. With only a single CPU you only get 8/12 DIMM sockets and not all PCIe slots work. There are many second hand T430s on sale with only a single CPU presumably because the T330 sucks. My T430 works fine with a pair of E5-2683 v4 CPUs. The Dell PowerEdge T630 also takes the same CPUs but supports higher TDP than the T430. They also support 18*3.5 disks or 32*2.5 but they are noisy. I wouldn t buy one for home use. AMD There are some nice AMD CPUs manufactured around the same time and AMD has done a better job of making multiple CPUs that fit the same socket. The reason I don t generally use AMD CPUs is that they are used in a minority of the server grade systems so as I want ECC RAM and other server features I generally can t find AMD systems at a reasonable price on ebay etc. There are people who really want second hand server grade systems with AMD CPUs and outbid me. This is probably a region dependent issue, maybe if I was buying in the US I could get some nice workstations with AMD CPUs at low prices. Socket LGA1151 Socket LGA1151 [5] is used in the Dell PowerEdge T330. It only supports 2 memory channels and 4 DIMMs compared to the 4 channels and 8 DIMMs in LGA2011, and it also has a limit of 64G total RAM for most systems and 128G for some systems. By today s standards even 128G is a real limit for server use, DDR4 RDIMMs are about $1/GB and when spending $600+ on system and CPU upgrade you wouldn t want to spend less than $130 on RAM. The CPUs with decent performance for that socket like the i9-9900K aren t supported by the T330 (possibly they don t support ECC RAM). The CPUs that Dell supports perform very poorly. I suspect that Dell deliberately nerfed the T330 to drive sales of the T430. The Lenovo P330 uses socket LGA1151-2 but has the same issues of taking slow CPUs in addition to using UDIMMs which are significantly more expensive on the second hand market. Socket LGA2066 The next Intel socket after LGA2011-v3 is LGA2066 [6]. That is in The Dell Precision 5820 and HP Z4 G4. It takes an i9-10980XE for 32,404 on Passmark or a W-2295 for 30,906. The variant of the Dell 5820 that supports the i9 CPUs doesn t seem to support ECC RAM so it s not a proper workstation. The single thread performance difference between the W-2295 and the E5-2699A v4 is 2640 to 2055, a 28% increase for the W-2295. There are High Frequency Optimized cpus for socket LGA2011-v3 but they all deliver less than 2,300 on the Passmark single-thread tests which is much less than what you can get from socket LGA2066. The W-2295 costs $1000 on ebay and the E5-2699A v4 is readily available for under $400 and a few months ago I got a matched pair for a bit over $400. Note that getting a matched pair of Intel CPUs is a major pain [7]. Comparing sockets LGA2011-v3 and LGA2066 for a single-CPU system is a $300 system (HP x640) + $400 CPU (E5-2699A v4) vs $500 system (Dell Precision 5820) + $1000 CPU (W-2295), so more than twice the price for a 30% performance benefit on some tasks. The LGA2011-v3 and USB-C both launched in 2014 so LGA2011-v3 systems don t have USB-C sockets, a $20 USB-C PCIe card doesn t change the economics. Socket LGA3647 Socket LGA3647 [8] is used in the Dell PowerEdge T440. It supports 6 channels of DDR4 RAM which is a very nice feature for bigger systems. According to one Dell web page the best CPU Dell officially supports for this is the Xeon Gold 5120 which gives performance only slightly better than the E5-2683 v4 which has a low enough TDP that a T430 can run two of them. But according to another Dell web page they support 16 core CPUs which means performance better than a T430 but less than a HP z840. The T440 doesn t seem like a great system, if I got one cheap I could find a use for it but I wouldn t pay the prices that they go for on ebay. The Dell PowerEdge T640 has the same socket and is described as supporting up to 28 core CPUs. But I anticipate that it would be as loud as the T630 and it s also expensive. This socket is also used in the HP Z6 G4 which takes a W-3265 or Xeon Gold 6258R CPU for the high end options. The HP Z6 G4 systems on ebay are all above $1500 and the Xeon Gold 6258R is also over $1000 so while the Xeon Gold 6258R in a Z6 G4 will give 50% better performance on multithreaded operations than the systems I currently have it s costing almost 3* as much. It has 6 DIMM sockets which is a nice improvement over the 4 in the z640. The Z6 G4 takes a maximum of 768G of RAM with the optional extra CPU board (which is very expensive both new and on ebay) compared to my z840 which has 512G and half it s DIMM slots empty. The HP Z8 G4 has the same socket and takes up to 3TB of RAM if used with CPUs that support it (most CPUs only support 768G and you need a M variant to support more). The higher performance CPUs supported in the Z6 G4 and Z8 G4 don t have enough entries in the Passmark database to be accurate, but going from 22 cores in the E5-2699A v4 to 28 in the Xeon Platinum 8180 when using the same RAM technology doesn t seem like a huge benefit. The Z6 and Z8 G4 systems run DDR4 RAM at up to 2666 speed while the z640 and z840 only to 2400, a 10% increase in RAM speed is nice but not a huge difference. I don t think that any socket LGA3647 systems will ever be ones I want to buy. They don t offer much over LGA2011-v3 but are in newer and fancier systems that will go for significantly higher prices. DDR5 I think that DDR5 systems will be my next step up in tower server and workstation performance after the socket LGA2011-v3 systems. I don t think anything less will offer me enough of a benefit to justify a change. I also don t think that they will be in the price range I am willing to pay until well after DDR6 is released, some people are hoping for DDR6 to be released late this year but next year seems more likely. So maybe in 2027 there will be some nice DDR5 systems going cheap. CPU Benchmark Results Here are the benchmark results of CPUs I mentioned in this post according to passmark.com [9]. I didn t reference results of CPUs that only had 1 or 2 results posted as they aren t likely to be accurate.

CPU	Single Thread	Multi Thread	TDP
E5-2683 v4	1,713	17,591	120W
Xeon Gold 5120	1,755	18,251	105W
i9-9900K	2,919	18,152	95W
E5-2697A v4	2,106	21,610	145W
E5-2699A v4	2,055	26,939	145W
W-3265	2,572	30,105	205W
W-2295	2,642	30,924	165W
i9-10980XE	2,662	32,397	165W
Xeon Gold 6258R	2,080	40,252	205W

• [1] https://etbe.coker.com.au/2025/04/05/hp-z840/
• [2] https://etbe.coker.com.au/2025/04/05/hp-ml110-gen9-z640/
• [3] https://en.wikipedia.org/wiki/LGA_2011
• [4] https://tinyurl.com/2ddumsfe
• [5] https://en.wikipedia.org/wiki/LGA_1151
• [6] https://en.wikipedia.org/wiki/LGA_2066
• [7] https://etbe.coker.com.au/2025/06/19/matching-intel-cpus/
• [8] https://en.wikipedia.org/wiki/LGA_3647
• [9] https://www.passmark.com/

(You wait ages for an archiving blog post and two come along at once!) Between 1969-2019, the Newcastle University School of Computing published a Technical Reports Series. Until 2017-ish, the full list of individually-numbered reports was available on the School's website, as well as full text PDFs for every report. At some time around 2014 I was responsible for migrating the School's website from self-managed to centrally-managed. The driver was to improve the website from the perspective of student recruitment. The TR listings (as well as full listings and texts for awarded PhD theses, MSc dissertations, Director's reports and various others) survived the initial move. After I left (as staff) in 2015, anything not specifically about student recruitment degraded and by 2017 the listings were gone. I've been trying, on and off, to convince different parts of the University to restore and take ownership of these lists ever since. For one reason or another each avenue I've pursued has gone nowhere. Recently the last remaining promising way forward failed, so I gave up and did it myself. The list is now hosted by the Historic Computing Committee, here: https://nuhc.ncl.ac.uk/computing/techreports/ It's not complete (most of the missing entries are towards the end of the run), but it's a start. The approach that finally yielded results was simply scraping the Internet Archive Wayback Machine for various pages from back when the material was represented on the School website, and then filling in the gaps from some other sources. What I envisage in the future: per-page reports with the relevant metadata (including abstracts); authors de-duplicated and cross-referenced; PDFs OCRd; providing access to the whole metadata DB (probably as as lump of JSON); a mechanism for people to report errors; a platform for students to perform data mining projects: perhaps some kind of classification/tagging by automated content analysis; cross-referencing copies of papers in other venues (lots of TRs are pre-prints).

How I deployed this Website I will describe the step-by-step process I followed to make this static website accessible on the Internet.

DNS I bought this domain on NameCheap and am using their DNS for now, where I created these records:

Record Type	Host	Value
A	sergiocipriano.com	201.54.0.17
CNAME	www	sergiocipriano.com

Virtual Machine I am using Magalu Cloud for hosting my VM, since employees have free credits. Besides creating a VM with a public IP, I only needed to set up a Security Group with the following rules:

Type	Protocol	Port	Direction	CIDR
IPv4 / IPv6	TCP	80	IN	Any IP
IPv4 / IPv6	TCP	443	IN	Any IP

Firewall The first thing I did in the VM was enabling ufw (Uncomplicated Firewall). Enabling ufw without pre-allowing SSH is a common pitfall and can lock you out of your VM. I did this once :) A safe way to enable ufw:

$ sudo ufw allow OpenSSH      # or: sudo ufw allow 22/tcp
$ sudo ufw allow 'Nginx Full' # or: sudo ufw allow 80,443/tcp
$ sudo ufw enable

To check if everything is ok, run:

$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip
To                           Action      From
--                           ------      ----
22/tcp (OpenSSH)             ALLOW IN    Anywhere                  
80,443/tcp (Nginx Full)      ALLOW IN    Anywhere                  
22/tcp (OpenSSH (v6))        ALLOW IN    Anywhere (v6)             
80,443/tcp (Nginx Full (v6)) ALLOW IN    Anywhere (v6) 

Reverse Proxy I'm using Nginx as the reverse proxy. Since I use the Debian package, I just needed to add this file:

/etc/nginx/sites-enabled/sergiocipriano.com

with this content:

server  
    listen 443 ssl;      # IPv4
    listen [::]:443 ssl; # IPv6
    server_name sergiocipriano.com www.sergiocipriano.com;
    root /path/to/website/sergiocipriano.com;
    index index.html;
    location /  
        try_files $uri /index.html;
     
 
server  
    listen 80;
    listen [::]:80;
    server_name sergiocipriano.com www.sergiocipriano.com;
    # Redirect all HTTP traffic to HTTPS
    return 301 https://$host$request_uri;
 

TLS It's really easy to setup TLS thanks to Let's Encrypt:

$ sudo apt-get install certbot python3-certbot-nginx
$ sudo certbot install --cert-name sergiocipriano.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Deploying certificate
Successfully deployed certificate for sergiocipriano.com to /etc/nginx/sites-enabled/sergiocipriano.com
Successfully deployed certificate for www.sergiocipriano.com to /etc/nginx/sites-enabled/sergiocipriano.com

Certbot will edit the nginx configuration with the path to the certificate.

HTTP Security Headers I decided to use wapiti, which is a web application vulnerability scanner, and the report found this problems:

1. CSP is not set
2. X-Frame-Options is not set
3. X-XSS-Protection is not set
4. X-Content-Type-Options is not set
5. Strict-Transport-Security is not set

I'll explain one by one:

1. The Content-Security-Policy header prevents XSS and data injection by restricting sources of scripts, images, styles, etc.
2. The X-Frame-Options header prevents a website from being embedded in iframes (clickjacking).
3. The X-XSS-Protection header is deprecated. It is recommended that CSP is used instead of XSS filtering.
4. The X-Content-Type-Options header stops MIME-type sniffing to prevent certain attacks.
5. The Strict-Transport-Security header informs browsers that the host should only be accessed using HTTPS, and that any future attempts to access it using HTTP should automatically be upgraded to HTTPS. Additionally, on future connections to the host, the browser will not allow the user to bypass secure connection errors, such as an invalid certificate. HSTS identifies a host by its domain name only.

I added this security headers inside the HTTPS and HTTP server block, outside the location block, so they apply globally to all responses. Here's how the Nginx config look like:

add_header Content-Security-Policy "default-src 'self'; style-src 'self';" always;
add_header X-Frame-Options "DENY" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

I added always to ensure that nginx sends the header regardless of the response code. To add Content-Security-Policy header I had to move the css to a separate file, because browsers block inline styles under strict CSP unless you allow them explicitly. They're considered unsafe inline unless you move to a separate file and link it like this:

<link rel="stylesheet" href="./resources/header.css">

I recently learned that, about 6 months ago, the Debian webteam deleted all news articles from the main website older than 2022. There have been several complaints from people in and outside of Debian, notably Joe Brockmeier of LWN, and this really sad one from the nephew of a deceased developer, wondering where the obituary had gone, but the team have not been swayed and are not prepared to reinstate the news. It feels very important to me, too, that historic news, and their links, are not broken. So, I hastily built a new Debian service, The Chronicles of Debian, as as permanent home for historic web content.

$ HEAD -S -H "Accept-Language: de" https://www.debian.org/News/1997/19971211
HEAD https://www.debian.org/News/1997/19971211
302 Found
HEAD https://chronicles.debian.org/www/News/1997/19971211
200 OK
 
Content-Language: de
Content-Location: 19971211.de.html
 

This was thrown up in a hurry to get something working as fast as possible, and there is plenty of room for improvement. Get in touch if there's an enhancement you would like or you would like to get involved!

Another short status update of what happened on my side last month - a lot shorter than usual due to real life events (that will also affect August) but there was some progress on stevia and it landed in Debian too now. See below for details on the above and more: phosh

• Use new, rust based) phrosh portal too (MR)
• Consistently format meson files (MR)

phoc

• Add sysprof support (MR)
• Reject input based on shell's state (MR)
• Avoid zero serial (MR)
• Allow to damage whole output on each frame (MR)
• Avoid possible crash on unlock (MR)

phosh-mobile-settings

• Use newer gmobile and make CI more flexible (MR
• Fix nightly build (MR)
• Allow to configure the OSK's automatic scaling properties (MR)

stevia (formerly phosh-osk-stub)

• Portrait keyboard scaling (MR)
• Fix translation of completer descriptions in mobile settings (MR)
• Use key-pressed events (MR)
• Fix additional completions like emojis with hunspell completer (MR)
• Document layout testing (MR)

phosh-vala-plugins

• Drop vapi files, they've made it into a phosh release now (MR)

xdg-desktop-portal-phosh

• Bump rustc dependency and simplify CI (MR)

feedbackd-device-themes

• Add key- pressed,released (MR)

livi

• Make single click play/pause video (MR)
• Update screenshot and metinfo for better display on flathub (MR)
• Release 0.3.2 (MR)
• Update on Flathub (MR)

Debian

• Upload current stevia to experimental
• phosh: Don't forget to install vapi files (MR)
• meta-phosh: Update to 0.48.0: (MR)
• Update to stevia 0.48 (MR)
• Update xkbcommon to 0.10.0 (MR)
• iio-sensor-proxy: Backport buffer mode fixes for trixie (MR), Unblock request
• livi: Update to 0.3.2 (MR)

foliate

• Don't let session go idle when in fullscreen (MR)

Cellbroadcastd

• Fix packaging build (MR)

git-buildpackage

• pull: Allow to convert local repo when remote got switched to DEP-14 (MR)

wayland-protocols

• Respin cutout protocol MR

Reviews This is not code by me but reviews on other peoples code. The list is (as usual) slightly incomplete. Thanks for the contributions!

• phosh-mobile-settings: Disable Xwayland to help e.g. distrobox (MR) - merged
• phosh-mobile-settings: Allow to search (MR) - merged
• phosh-mobile-settings: Allow to configure terminal layout shortcuts (MR) - merged
• feedbackd: Legacy led support (MR) - merged
• phosh: upcoming-events: Allow to hide days without events (MR)
• m-b-p-i: Add emergency numbers for JP (MR)
• xdg-desktop-portal-phosh: bootstrap pure rust portal (MR) - merged
• xdg-desktop-portal-phosh: portal avoidance (MR) - merged

Help Development If you want to support my work see donations. Comments? Join the Fediverse thread

In beginning of July I got my 12" framework laptop and installed Debian on it. During that setup I made some updates to my base setup scripts that I use to install Debian machines. Due to the freeze I did not do much package related work. But I was at DebConf and I uploaded a new release of labwc to experimental, mostly to test the tag2upload workflow. I started working on packaging wlr-sunclock which is a small Wayland widget that displays the sun s shadows on the earth. I also created an ITP for wayback. Wayback is an X11 compatibility layer to allow to run X11 desktop environments using Wayland. In my dayjob I did my usual work on apis-core-rdf, which is our Django application for managing prosopographic data. I implemented a password change interface and did some restructuring of the templates. We released a new version which was followed by a bugfix release a couple of days later. I also implemented a rather big refactoring in pfp-api. PFP-API is a FastAPI based REST API that uses rdfproxy to fetch data from a Triplestore, converts the data to Pydantic models and then ships the models as JSON. Most of the work is done by rdfproxy in the background, but I adapted the existing pfp-api code to make it easier to add new entity types.

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

• WikiBot: add done silent mode
• duck: add indicators
• Debian wiki pages: AutoGeneratedFiles, PrivacyIssues, SubstitutingCalibriAndCambriaFonts

Sponsors All work was done on a volunteer basis.

I ve participated in this year s Google Summer of Code (GSoC) program and have been working on the small (90h) autopkgtests for the rsync package project at Debian.

Writing my proposal Before you can start writing a proposal, you need to select an organization you want to work with. Since many organizations participate in GSoC, I ve used the following criteria to narrow things down for me:

• Programming language familiarity: For me only Python (preferably) as well as shell and Go projects would have made sense. While learning another programming language is cool, I wouldn t be as effective and helpful to the project as someone who is proficient in the language already.
• Standing of the organization: Some of the organizations participating in GSoC are well-known for the outstanding quality of the software they produce. Debian is one of them, but so is e.g. the Django Foundation or PostgreSQL. And my thinking was that the higher the quality of the organization, the more there is to learn for me as a GSoC student.
• Mentor interactions: Apart from the advantage you get from mentor feedback when writing your proposal (more on that further below), it is also helpful to gauge how responsive/helpful your potential mentor is during the application phase. This is important since you will be working together for a period of at least 2 months; if the mentor-student communication doesn t work, the GSoC project is going to be difficult.
• Free and Open-Source Software (FOSS) communication platforms: I generally believe that FOSS projects should be built on FOSS infrastructure. I personally won t run proprietary software when I want to contribute to FOSS in my spare time.
• Be a user of the project: As Eric S. Raymond has pointed out in his seminal The Cathedral and the Bazaar 25 years ago

  Every good work of software starts by scratching a developer s personal itch.

Once I had some organizations in mind whose projects I d be interested in working on, I started writing proposals for them. Turns out, I started writing my proposals way too late: In the end I only managed to hand in a single one which is risky. Competition for the GSoC projects is fierce and the more quality (!) proposals you send out, the better your chances are at getting one. However, don t write proposals for the sake of it: Reviewers get way too many AI slop proposals already and you will not do yourself a favor with a low-quality proposal. Take the time to read the instructions/ideas/problem descriptions the project mentors have provided and follow their guidelines. Don t hesitate to reach out to project mentors: In my case, I ve asked Samuel Henrique a few clarification questions whereby the following (email) discussion has helped me greatly in improving my proposal. Once I ve finalized my proposal draft, I ve sent it to Samuel for a review, which again led to some improvements to the final proposal which I ve uploaded to the GSoC program webpage.

Community bonding period Once you get the information that you ve been accepted into the GSoC program (don t take it personally if you don t make it; this was my second attempt after not making the cut in 2024), get in touch with your prospective mentor ASAP. Agree upon a communication channel and some response times. Put yourself in the loop for project news and discussions whatever that means in the context of your organization: In Debian s case this boiled down to subscribing to a bunch of mailing lists and IRC channels. Also make sure to setup a functioning development environment if you haven t done so for writing the proposal already.

Payoneer setup The by far most annoying part of GSoC for me. But since you don t have a choice if you want to get the stipend, you will need to signup for an account at Payoneer. In this iteration of GSoC all participants got a personalized link to open a Payoneer account. When I tried to open an account by following this link, I got an email after the registration and email verification that my account is being blocked because Payoneer deems the email adress I gave a temporary one. Well, the email in question is most certainly anything but temporary, so I tried to get in touch with the Payoneer support - and ended up in an LLM-infused kafkaesque support hell. Emails are answered by an LLM which for me meant utterly off-topic replies and no help whatsoever. The Payoneer website offers a real-time chat, but it is yet another instance of a bullshit-spewing LLM bot. When I at last tried to call them (the support lines are not listed on the Payoneer website but were provided by the GSoC program), I kid you not, I was being told that their platform is currently suffering from technical problems and was hung up on. Only thanks to the swift and helpful support of the GSoC administrators (who get priority support from Payoneer) I was able to setup a Payoneer account in the end. Apart from showing no respect to customers, Payoneer is also ripping them off big time with fees (unless you get paid in USD). They charge you 2% for currency conversions to EUR on top of the FX spread they take. What worked for me to avoid all of those fees, was to open a USD account at Wise and have Payoneer transfer my GSoC stipend in USD to that account. Then I exchanged the USD to my local currency at Wise for significantly less than Payoneer would have charged me. Also make sure to close your Payoneer account after the end of GSoC to avoid their annual fee.

Project work With all this prelude out of the way, I can finally get to the actual work I ve been doing over the course of my GSoC project.

Background The upstream rsync project generally sees little development. Nonetheless, they released version 3.4.0 including some CVE fixes earlier this year. Unfortunately, their changes broke the -H flag. Now, Debian package maintainers need to apply those security fixes to the package versions in the Debian repositories; and those are typically a bit older. Which usually means that the patches cannot be applied as is but will need some amendments by the Debian maintainers. For these cases it is helpful to have autopkgtests defined, which check the package s functionality in an automated way upon every build. The question then is, why should the tests not be written upstream such that regressions are caught in the development rather than the distribution process? There s a lot to say on this question and it probably depends a lot on the package at hand, but for rsync the main benefits are twofold:

1. The upstream project mocks the ssh connection over which rsync is most typically used. Mocking is better than nothing but not the real thing. In addition to being a more realisitic test scenario for the typical rsync use case, involving an ssh server in the test would automatically extend the overall resilience of Debian packages as now new versions of the openssh-server package in Debian benefit from the test cases in the rsync reverse dependency.
2. The upstream rsync test framework is somewhat idiosyncratic and difficult to port to reimplementations of rsync. Given that the original rsync upstream sees little development, an extensive test suit further downstream can serve as a threshold for drop-in replacements for rsync.

Goal(s) At the start of the project, the Debian rsync package was just running (a part of) the upstream tests as autopkgtests. The relevant snippet from the build log for the rsync_3.4.1+ds1-3 package reads:

114s ------------------------------------------------------------
114s ----- overall results:
114s 36 passed
114s 7 skipped

Samuel and I agreed that it would be a good first milestone to make the skipped tests run. Afterwards, I should write some rsync test cases for local calls, i.e. without an ssh connection, effectively using rsync as a more powerful cp. And once that was done, I should extend the tests such that they run over an active ssh connection. With these milestones, I went to work.

Upstream tests Running the seven skipped upstream tests turned out to be fairly straightforward:

• Two upstream tests concern access control lists and extended filesystem attributes. For these tests to run they rely on functionality provided by the acl and xattr Debian packages. Adding those to the Build-Depends list in the debian/control file of the rsync Debian package repo made them run.
• Four upstream tests required root privileges to run. The autopkgtest tool knows the needs-root restriction for that reason. However, Samuel and I agreed that the tests should not exclusively run with root privileges. So, instead of just adding the restiction to the existing autopkgtest test, we created a new one which has the needs-root restriction and runs the upstream-tests-as-root script - which is nothing else than a symlink to the existing upstream-tests script.

The commits to implement these changes can be found in this merge request. The careful reader will have noticed that I only made 2 + 4 = 6 upstream test cases run out of 7: The leftover upstream test is checking the functionality of the --ctimes rsync option. In the context of Debian, the problem is that the Linux kernel doesn t have a syscall to set the creation time of a file. As long as that is the case, this test will always be skipped for the Debian package.

Local tests When it came to writing Debian specific test cases I started of a completely clean slate. Which is a blessing and a curse at the same time: You have full flexibility but also full responsibility. There were a few things to consider at this point in time:

• Which language to write the tests in? The programming language I am most proficient in is Python. But testing a CLI tool in Python would have been weird: it would have meant that I d have to make repeated subprocess calls to run rsync and then read from the filesystem to get the file statistics I want to check. Samuel suggested I stick with shell scripts and make use of diffoscope - one of the main tools used and maintained by the Reproducible Builds project - to check whether the file contents and file metadata are as expected after rsync calls. Since I did not have good reasons to use bash, I ve decided to write the scripts to be POSIX compliant.
• How to avoid boilerplate? If one makes use of a testing framework, which one? Writing the tests would involve quite a bit of boilerplate, mostly related to giving informative output on and during the test run, preparing the file structure we want to run rsync on, and cleaning the files up after the test has run. It would be very repetitive and in violation of DRY to have the code for this appear in every test. Good testing frameworks should provide convenience functions for these tasks. shunit2 comes with those functions, is packaged for Debian, and given that it is already being used in the curl project, I decided to go with it.
• Do we use the same directory structure and files for every test or should every test have an individual setup? The tradeoff in this question being test isolation vs. idiosyncratic code. If every test has its own setup, it takes a) more work to write the test and b) more work to understand the differences between tests. However, one can be sure that changes to the setup in one test will have no side effects on other tests. In my opinion, this guarantee was worth the additional effort in writing/reading the tests.

Having made these decisions, I simply started writing tests and ran into issues very quickly.

rsync and subsecond mtime diffs When testing the rsync --times option, I observed a weird phenomenon: If the source and destination file have modification times which differ only in the nanoseconds, an rsync --times call will not synchronize the modification times. More details about this behavior and examples can be found in the upstream issue I raised. In the Debian tests we had to occasionally work around this by setting the timestamps explicitly with touch -d.

diffoscope regression In one test case, I was expecting a difference in the modification times but diffoscope would not report a diff. After a good amount of time spent on debugging the problem (my default, and usually correct, assumption is that something about my code is seriously broken if I run into issues like that), I was able to show that diffoscope only displayed this behavior in the version in the unstable suite, not on Debian stable (which I am running on my development machine). Since everything pointed to a regression in the diffoscope project and with diffoscope being written in Python, a language I am familiar with, I wanted to spend some time investigating (and hopefully fixing) the problem. Running git bisect on the diffoscope repo helped me in identifying the commit which introduced the regression: The commit contained an optimization via an early return for bit-by-bit identical files. Unfortunately, the early return also caused an explicitly requested metadata comparison (which could be different between the files) to be skipped. With a nicely diagnosed issue like that, I was able to go to a local hackerspace event, where people work on FOSS together for an evening every month. In a group, we were able to first, write a test which showcases the broken behavior in the latest diffoscope version, and second, make a fix to the code such that the same test passes going forward. All details can be found in this merge request.

shunit2 failures At some point I had a few autopkgtests setup and passing, but adding a new one would throw me totally inexplicable errors. After trying to isolate the problem as much as possible, it turns out that shunit2 doesn t play well together we the -e shell option. The project mentions this in the release notes for the 2.1.8 version¹, but in my opinion a constraint this severe should be featured much more prominently, e.g. in the README.

Tests over an ssh connection The centrepiece of this project; everything else has in a way only been preparation for this. Obviously, the goal was to reuse the previously written local tests in some way. Not only because lazy me would have less work to do this way, but also because of a reduced long-term maintenance burden of one rather than two test sets. As it turns out, it is actually possible to accomplish that: The remote-tests script doesn t do much apart from starting an ssh server on localhost and running the local-tests script with the REMOTE environment variable set. The REMOTE environment variable changes the behavior of the local-tests script in such a way that it prepends "$REMOTE": to the destination of the rsync invocations. And given that we set REMOTE=rsync@localhost in the remote-tests script, local-tests copies the files to the exact same locations as before, just over ssh. The implementational details for this can be found in this merge request.

proposed-updates Most of my development work on the Debian rsync package took place during the Debian freeze as the release of Debian Trixie is just around the corner. This means that uploading by Debian Developers (DD) and Debian Maintainers (DM) to the unstable suite is discouraged as it makes migrating the packages to testing more difficult for the Debian release team. If DDs/DMs want to have the package version in unstable migrated to testing during the freeze they have to file an unblock request. Samuel has done this twice (1, 2) for my work for Trixie but has asked me to file the proposed-updates request for current stable (i.e. Debian Bookworm) myself after I ve backported my tests to bookworm.

Unfinished business To run the upstream tests which check access control list and extended file system attributes functionality, I ve added the acl and xattr packages to Build-Depends in debian/control. This, however, will only make the packages available at build time: If Debian users install the rsync package, the acl and xattr packages will not be installed alongside it. For that, the dependencies would have to be added to Depends or Suggests in debian/control. Depends is probably to strong of a relation since rsync clearly works well in practice without, but adding them to Suggests might be worthwhile. A decision on this would involve checking, what happens if rsync is called with the relevant options on a host machine which has those packages installed, but where the destination machine lacks them. Apart from the issue described above, the 15 tests I managed to write are are a drop in the water in light of the infinitude of rsync options and their combinations. Most glaringly, not all options of the --archive option are covered separately (which would help indicating what code path of rsync broke in a regression). To increase the likelihood of catching regressions with the autopkgtests, the test coverage should be extended in the future.

Conclusion Generally, I am happy with my contributions to Debian over the course of my small GSoC project: I ve created an extensible, easy to understand, and working autopkgtest setup for the Debian rsync package. There are two things which bother me, however:

1. In hindsight, I probably shouldn t have gone with shunit2 as a testing framework. The fact that it behaves erratically with the -e flag is a serious drawback for a shell testing framework: You really don t want a shell command to fail silently and the test to continue running.
2. As alluded to in the previous section, I m not particularly proud of the number of tests I managed to write.

On the other hand, finding and fixing the regression in diffoscope - while derailing me from the GSoC project itself - might have a redeeming quality.

DebConf25 By sheer luck I happened to work on a GSoC project at Debian over a time period during which the annual Debian conference would take place close enough to my place of residence. Samuel pointed the opportunity to attend DebConf out to me during the community bonding period and since I could make time for the event in my schedule, I signed up. DebConf was a great experience which - aside from gaining more knowledge about Debian development - allowed me to meet the actual people usually hidden behind email adresses and IRC nicks. I can wholeheartedly recommend attending a DebConf to every interested Debian user! For those who have missed this year s iteration of the conference, I can recommend the following recorded talks:

• reproduce.debian.net: Reproducing what is distributed from ftp.debian.org
• The security of Debian: An introduction to advanced users
• Bits from the DPL
• DebConf27 in your city?

While not featuring as a keynote speaker (understandably so as the newcomer to Debian community that I am), I could still contribute a bit to the conference program.

GSoC project presentation The Debian Outreach team has scheduled a session in which all GSoC and Outreachy students over the past year had the chance to present their work in a lightning talk. The session has been recorded and is available online, just like my slides and the source for them.

Debian install workshop Additionally, with so many Debian experts gathering in one place while KDE s End of 10 campaign is ongoing, I felt it natural to organize a Debian install workhop. In hindsight I can say that I underestimated how much work it would be, especially for me who does not speak a word of French. But although the turnout of people who wanted us to install Linux on their machines was disappointingly low, it was still worth it: Not only because the material in the repo can be helpful to others planning install workshops but also because it was nice to meet a) the person behind the Debian installer images and b) the local Brest/Finist re Linux user group as well as the motivated and helpful people at Infini.

Credits I want to thank the Open Source team at Google for organizing GSoC: The highly structured program with a one-to-one mentorship is a great avenue to start contributing to well established and at times intimidating FOSS projects. And as much as I disagree with Google s surveillance capitalist business model, I have to give it to them that the company at least takes its responsibility for FOSS (somewhat) seriously - unlike many other businesses which rely on FOSS and choose to freeride of it. Big thanks to the Debian community! I ve experienced nothing but friendliness in my interactions with the community. And lastly, the biggest thanks to my GSoC mentor Samuel Henrique. He has dealt patiently and competently with all my stupid newbie questions. His support enabled me to make - albeit small - contributions to Debian. It has been a pleasure to work with him during GSoC and I m looking forward to working together with him in the future.

---

1. Obviously, I ve only read them after experiencing the problem.

The diffoscope maintainers are pleased to announce the release of diffoscope version 303. This version includes the following changes: You find out more by visiting the project homepage.

As I m writing this, we re one hour away from landing, and thus our Grand (with a capital G for sure) Japan 2025 vacation is over. Planning started about nine months ago, plane tickets bought six months in advance, most hotels booked about four months ahead, and then a wonderful, even if a bit packed, almost 3 weeks in Japan. And now we re left with lots of good memories, some mishaps that we re going to laugh about in a few months s time, and quite a few thousand pictures to process and filter, so that so they can be viewed in a single session. Oh, and I m also left with a nice bottle of plum wine, thanks to inflight shopping. Was planning to, but didn t manage to buy one in the airport, as Haneda International departures, after the security check, is a bit small. But in 15 hours of flying, there was enough time to implement 2 tiny Corydalis features, and browse the shopping catalog . I only learned on the flight that some items need to be preordered, a lesson for next time Thanks to the wonders of inflight internet, I can write and publish this, but it not being StarLink, Visual Studio Code managed to download an update for the UI, but now the remote server package is too big? slow? and can t be downloaded. Well, it started download 5 times, and aborted at about 80% each time. Well, thankful my blog is lightweight and I can write it in vi and push it . And pushing the above-mentioned features to GitHub was also possible. A proper blog post will follow, once I can select some pictures and manage to condense three weeks in an overall summary And in the meantime, back to the real world!

LWN wrote an article which opens with the assertion "Linux users who have Secure Boot enabled on their systems knowingly or unknowingly rely on a key from Microsoft that is set to expire in September". This is, depending on interpretation, either misleading or just plain wrong, but also there's not a good source of truth here, so.

First, how does secure boot signing work? Every system that supports UEFI secure boot ships with a set of trusted certificates in a database called "db". Any binary signed with a chain of certificates that chains to a root in db is trusted, unless either the binary (via hash) or an intermediate certificate is added to "dbx", a separate database of things whose trust has been revoked[1]. But, in general, the firmware doesn't care about the intermediate or the number of intermediates or whatever - as long as there's a valid chain back to a certificate that's in db, it's going to be happy.

That's the conceptual version. What about the real world one? Most x86 systems that implement UEFI secure boot have at least two root certificates in db - one called "Microsoft Windows Production PCA 2011", and one called "Microsoft Corporation UEFI CA 2011". The former is the root of a chain used to sign the Windows bootloader, and the latter is the root used to sign, well, everything else.

What is "everything else"? For people in the Linux ecosystem, the most obvious thing is the Shim bootloader that's used to bridge between the Microsoft root of trust and a given Linux distribution's root of trust[2]. But that's not the only third party code executed in the UEFI environment. Graphics cards, network cards, RAID and iSCSI cards and so on all tend to have their own unique initialisation process, and need board-specific drivers. Even if you added support for everything on the market to your system firmware, a system built last year wouldn't know how to drive a graphics card released this year. Cards need to provide their own drivers, and these drivers are stored in flash on the card so they can be updated. But since UEFI doesn't have any sandboxing environment, those drivers could do pretty much anything they wanted to. Someone could compromise the UEFI secure boot chain by just plugging in a card with a malicious driver on it, and have that hotpatch the bootloader and introduce a backdoor into your kernel.

This is avoided by enforcing secure boot for these drivers as well. Every plug-in card that carries its own driver has it signed by Microsoft, and up until now that's been a certificate chain going back to the same "Microsoft Corporation UEFI CA 2011" certificate used in signing Shim. This is important for reasons we'll get to.

The "Microsoft Windows Production PCA 2011" certificate expires in October 2026, and the "Microsoft Corporation UEFI CA 2011" one in June 2026. These dates are not that far in the future! Most of you have probably at some point tried to visit a website and got an error message telling you that the site's certificate had expired and that it's no longer trusted, and so it's natural to assume that the outcome of time's arrow marching past those expiry dates would be that systems will stop booting. Thankfully, that's not what's going to happen.

First up: if you grab a copy of the Shim currently shipped in Fedora and extract the certificates from it, you'll learn it's not directly signed with the "Microsoft Corporation UEFI CA 2011" certificate. Instead, it's signed with a "Microsoft Windows UEFI Driver Publisher" certificate that chains to the "Microsoft Corporation UEFI CA 2011" certificate. That's not unusual, intermediates are commonly used and rotated. But if we look more closely at that certificate, we learn that it was issued in 2023 and expired in 2024. Older versions of Shim were signed with older intermediates. A very large number of Linux systems are already booting certificates that have expired, and yet things keep working. Why?

Let's talk about time. In the ways we care about in this discussion, time is a social construct rather than a meaningful reality. There's no way for a computer to observe the state of the universe and know what time it is - it needs to be told. It has no idea whether that time is accurate or an elaborate fiction, and so it can't with any degree of certainty declare that a certificate is valid from an external frame of reference. The failure modes of getting this wrong are also extremely bad! If a system has a GPU that relies on an option ROM, and if you stop trusting the option ROM because either its certificate has genuinely expired or because your clock is wrong, you can't display any graphical output[3] and the user can't fix the clock and, well, crap.

The upshot is that nobody actually enforces these expiry dates - here's the reference code that disables it. In a year's time we'll have gone past the expiration date for "Microsoft Windows UEFI Driver Publisher" and everything will still be working, and a few months later "Microsoft Windows Production PCA 2011" will also expire and systems will keep booting Windows despite being signed with a now-expired certificate. This isn't a Y2K scenario where everything keeps working because people have done a huge amount of work - it's a situation where everything keeps working even if nobody does any work.

So, uh, what's the story here? Why is there any engineering effort going on at all? What's all this talk of new certificates? Why are there sensationalist pieces about how Linux is going to stop working on old computers or new computers or maybe all computers?

Microsoft will shortly start signing things with a new certificate that chains to a new root, and most systems don't trust that new root. System vendors are supplying updates[4] to their systems to add the new root to the set of trusted keys, and Microsoft has supplied a fallback that can be applied to all systems even without vendor support[5]. If something is signed purely with the new certificate then it won't boot on something that only trusts the old certificate (which shouldn't be a realistic scenario due to the above), but if something is signed purely with the old certificate then it won't boot on something that only trusts the new certificate.

How meaningful a risk is this? We don't have an explicit statement from Microsoft as yet as to what's going to happen here, but we expect that there'll be at least a period of time where Microsoft signs binaries with both the old and the new certificate, and in that case those objects should work just fine on both old and new computers. The problem arises if Microsoft stops signing things with the old certificate, at which point new releases will stop booting on systems that don't trust the new key (which, again, shouldn't happen). But even if that does turn out to be a problem, nothing is going to force Linux distributions to stop using existing Shims signed with the old certificate, and having a Shim signed with an old certificate does nothing to stop distributions signing new versions of grub and kernels. In an ideal world we have no reason to ever update Shim[6] and so we just keep on shipping one signed with two certs.

If there's a point in the future where Microsoft only signs with the new key, and if we were to somehow end up in a world where systems only trust the old key and not the new key[7], then those systems wouldn't boot with new graphics cards, wouldn't be able to run new versions of Windows, wouldn't be able to run any Linux distros that ship with a Shim signed only with the new certificate. That would be bad, but we have a mechanism to avoid it. On the other hand, systems that only trust the new certificate and not the old one would refuse to boot older Linux, wouldn't support old graphics cards, and also wouldn't boot old versions of Windows. Nobody wants that, and for the foreseeable future we're going to see new systems continue trusting the old certificate and old systems have updates that add the new certificate, and everything will just continue working exactly as it does now.

Conclusion: Outside some corner cases, the worst case is you might need to boot an old Linux to update your trusted keys to be able to install a new Linux, and no computer currently running Linux will break in any way whatsoever.

[1] (there's also a separate revocation mechanism called SBAT which I wrote about here, but it's not relevant in this scenario)

[2] Microsoft won't sign GPLed code for reasons I think are unreasonable, so having them sign grub was a non-starter, but also the point of Shim was to allow distributions to have something that doesn't change often and be able to sign their own bootloaders and kernels and so on without having to have Microsoft involved, which means grub and the kernel can be updated without having to ask Microsoft to sign anything and updates can be pushed without any additional delays

[3] It's been a long time since graphics cards booted directly into a state that provided any well-defined programming interface. Even back in 90s, cards didn't present VGA-compatible registers until card-specific code had been executed (hence DEC Alphas having an x86 emulator in their firmware to run the driver on the card). No driver? No video output.

[4] There's a UEFI-defined mechanism for updating the keys that doesn't require a full firmware update, and it'll work on all devices that use the same keys rather than being per-device

[5] Using the generic update without a vendor-specific update means it wouldn't be possible to issue further updates for the next key rollover, or any additional revocation updates, but I'm hoping to be retired by then and I hope all these computers will also be retired by then

[6] I said this in 2012 and it turned out to be wrong then so it's probably wrong now sorry, but at least SBAT means we can revoke vulnerable grubs without having to revoke Shim

[7] Which shouldn't happen! There's an update to add the new key that should work on all PCs, but there's always the chance of firmware bugs

comments

The gnulib project publish a git bundle as a stable archival copy of the gnulib git repository once in a while. Why? We don t know exactly what this may be useful for, but I m promoting for this to see if we can establish some good use-case. A git bundle may help to establish provinence in case of an attack on the Savannah hosting platform that compromise the gnulib git repository. Another use is in the Debian gnulib package: that gnulib bundle is git cloned when building some Debian packages, to get to exactly the gnulib commit used by each upstream project see my talk on gnulib at Debconf24 and this approach reduces the amount of vendored code that is part of Debian s source code, which is relevant to mitigate XZ-style attacks. The first time we published the bundle, I wanted it to be possible to re-create it bit-by-bit identically by others. At the time I discovered a well-written blog post by Paul Beacher on reproducible git bundles and thought he had solved the problem for me. Essentially it boils down to disable threading during compression when producing the bundle, and his final example show this results in a predictable bit-by-bit identical output:

$ for i in $(seq 1 100); do \
> git -c 'pack.threads=1' bundle create -q /tmp/bundle-$i --all; \
> done
$ md5sum /tmp/bundle-*   cut -f 1 -d ' '   uniq -c
    100 4898971d4d3b8ddd59022d28c467ffca

So what remains to be said about this? It seems reproducability goes deeper than that. One desirable property is that someone else should be able to reproduce the same git bundle, and not only that a single individual is able to reproduce things on one machine. It surprised me to see that when I ran the same set of commands on a different machine (started from a fresh git clone), I got a different checksum. The different checksums occured even when nothing had been committed on the server side between the two runs. I thought the reason had to do with other sources of unpredictable data, and I explored several ways to work around this but eventually gave up. I settled for the following sequence of commands:

REV=ac9dd0041307b1d3a68d26bf73567aa61222df54 # master branch commit to package
git clone https://git.savannah.gnu.org/git/gnulib.git
cd gnulib
git fsck # attempt to validate input
# inspect that the new tree matches a trusted copy
git checkout -B master $REV # put $REV at master
for b in $(git branch -r   grep origin/stable-   sort --version-sort); do git checkout $ b#origin/ ; done
git remote remove origin # drop some unrelated branches
git gc --prune=now # drop any commits after $REV
git -c 'pack.threads=1' bundle create gnulib.bundle --all
V=$(env TZ=UTC0 git show -s --date=format:%Y%m%d --pretty=%cd master)
mv gnulib.bundle gnulib-$V.bundle
build-aux/gnupload --to ftp.gnu.org:gnulib gnulib-$V.bundle

At the time it felt more important to publish something than to reach for perfection, so we did so using the above snippet. Afterwards I reached out to the git community on this and there were good discussion about my challenge. At the end of that thread you see that I was finally able to reproduce a bit-by-bit identical bundles from two different clones, by using an intermediate git -c pack.threads=1 repack -adF step. I now assume that the unpredictable data I got earlier was introduced during the git clone steps, compressing the pack differently each time due to threaded compression. The outcome could also depend on what content the server provided, so if someone ran git gc, git repack on the server side things would change for the user, even if the user forced threading to 1 during cloning more experiments on what kind of server-side alterations results in client-side differences would be good research. A couple of months passed and it is now time to publish another gnulib bundle somewhat paired to the bi-yearly stable gnulib branches so let s walk through the commands and explain what they do. First clone the repository:

REV=225973a89f50c2b494ad947399425182dd42618c   # master branch commit to package
S1REV=475dd38289d33270d0080085084bf687ad77c74d # stable-202501 branch commit
S2REV=e8cc0791e6bb0814cf4e88395c06d5e06655d8b5 # stable-202507 branch commit
git clone https://git.savannah.gnu.org/git/gnulib.git
cd gnulib
git fsck # attempt to validate input

I believe the git fsck will validate that the chain of SHA1 commits are linked together, preventing someone from smuggling in unrelated commits earlier in the history without having to do SHA1 collision. SHA1 collisions are economically feasible today, so this isn t much of a guarantee of anything though.

git checkout -B master $REV # put $REV at master
# Add all stable-* branches locally:
for b in $(git branch -r   grep origin/stable-   sort --version-sort); do git checkout $ b#origin/ ; done
git checkout -B stable-202501 $S1REV
git checkout -B stable-202507 $S2REV
git remote remove origin # drop some unrelated branches
git gc --prune=now # drop any unrelated commits, not clear this helps

This establish a set of branches pinned to particular commits. The older stable-* branches are no longer updated, so they shouldn t be moving targets. In case they are modified in the future, the particular commit we used will be found in the official git bundle.

time git -c pack.threads=1 repack -adF

That s the new magic command to repack and recompress things in a hopefully more predictable way. This leads to a 72MB git pack under .git/objects/pack/ and a 62MB git bundle. The runtime on my laptop is around 5 minutes. I experimented with -c pack.compression=1 and -c pack.compression=9 but the size was roughly the same; 76MB and 66MB for level 1 and 72MB and 62MB for level 9. Runtime still around 5 minutes. Git uses zlib by default, which isn t the most optimal compression around. I tried -c pack.compression=0 and got a 163MB git pack and a 153MB git bundle. The runtime is still around 5 minutes, indicating that compression is not the bottleneck for the git repack command. That 153MB uncompressed git bundle compresses to 48MB with gzip default settings and 46MB with gzip -9; to 39MB with zst defaults and 34MB with zst -9; and to 28MB using xz defaults with a small 26MB using xz -9. Still the inconvenience of having to uncompress a 30-40MB archive into
the much larger 153MB is probably not worth the savings compared to
shipping and using the (still relatively modest) 62MB git bundle. Now finally prepare the bundle and ship it:

git -c 'pack.threads=1' bundle create gnulib.bundle --all
V=$(env TZ=UTC0 git show -s --date=format:%Y%m%d --pretty=%cd master)
mv gnulib.bundle gnulib-$V.bundle
build-aux/gnupload --to ftp.gnu.org:gnulib gnulib-$V.bundle

Yay! Another gnulib git bundle snapshot is available from
https://ftp.gnu.org/gnu/gnulib/. The essential part of the git repack command is the -F parameter. In the thread -f was suggested, which translates into the git pack-objects --no-reuse-delta parameter:

--no-reuse-delta
When creating a packed archive in a repository that has existing packs, the command reuses existing deltas. This sometimes results in a slightly suboptimal pack. This flag tells the command not to reuse existing deltas but compute them from scratch.

When reading the man page, I though that using -F which translates into --no-reuse-object would be slightly stronger:

--no-reuse-object
This flag tells the command not to reuse existing object data at all, including non deltified object, forcing recompression of everything. This implies --no-reuse-delta. Useful only in the obscure case where wholesale enforcement of a different compression level on the packed data is desired.

On the surface, without --no-reuse-objects, some amount of earlier compression could taint the final result. Still, I was able to get bit-by-bit identical bundles by using -f so possibly reaching for -F is not necessary. All the commands were done using git 2.51.0 as packaged by Guix. I fear the result may be different with other git versions and/or zlib libraries. I was able to reproduce the same bundle on a Trisquel 12 aramo (derived from Ubuntu 22.04) machine, which uses git 2.34.1. This suggests there is some chances of this being possible to reproduce in 20 years time. Time will tell. I also fear these commands may be insufficient if something is moving on the server-side of the git repository of gnulib (even just something simple as a new commit), I tried to make some experiments with this but let s aim for incremental progress here. At least I have now been able to reproduce the same bundle on different machines, which wasn t the case last time. Happy Reproducible Git Bundle Hacking!

Louis Rossman made an informative YouTube video about right to repair and the US military [1]. This is really important as it helps promote free software and open standards. The ACM has an insightful article about hidden controls [2]. We need EU regulations about hidden controls in safety critical systems like cars. This Daily WTF article has some interesting security implications for Windows [3]. Earth.com has an interesting article about the rubber hand illusion and how it works on Octopus [4]. For a long time I have been opposed to eating Octopus because I think they are too intelligent. The Washington Post has an insightful article about the future of spies when everything is tracked by technology [5]. Micah Lee wrote an informative guide to using Signal groups for activism [6]. David Brin wrote an insightful blog post about the phases of the ongoing US civil war [7]. Christian Kastner wrote an interesting blog post about using Glibc hardware capabilities to use different builds of a shared library for a range of CPU features [8]. David Brin wrote an insightful and interesting blog post comparing President Carter with the criminals in the Republican party [9].

• [1] https://www.youtube.com/watch?v=HafelMKC9RI
• [2] https://tinyurl.com/yon77zhg
• [3] https://tinyurl.com/2d5bdpcw
• [4] https://tinyurl.com/2bv8b2hn
• [5] https://tinyurl.com/27brmavr
• [6] https://micahflee.com/using-signal-groups-for-activism/
• [7] https://tinyurl.com/2bfupoge
• [8] https://tinyurl.com/29n4jnh8
• [9] https://tinyurl.com/2b6fhozc