Almost finished the testing we're going to do at 15:29 UTC. It's all been good - we've found that at least one of the major bug reports from 12.10 is not reproducible now. All good - and many thanks to all testers: Sledge, rattusrattus, egw, smcv (and me).

I ve been looking at computer hardware on AliExpress a lot recently and I saw an advert for a motherboard which can take 256G DDR4 RDIMMs (presumably LRDIMMs). Most web pages about DDR4 state that 128G is the largest possible. The Wikipedia page for DDR4 doesn t state that 128G is the maximum but does have 128G as the largest size mentioned on the page. Recently I ve been buying 32G DDR4 RDIMMs for between $25 and $30 each. A friend can get me 64G modules for about $70 at the lowest price. If I hadn t already bought a heap of 32G modules I d buy some 64G modules right now at that price as it s worth paying 40% extra to allow better options for future expansion. Apparently the going rate for 128G modules is $300 each which is within the range for a hobbyist who has a real need for RAM. 256G modules are around $1200 each which is starting to get a big expensive. But at that price I could buy 2TB of RAM for $9600 and the computer containing it still wouldn t be the most expensive computer I ve bought the laptop that cost $5800 in 1998 takes that honour when inflation is taken into account. DDR5 RDIMMs are currently around $10/GB compared to DDR4 for $1/GB for 32G modules and DDR3 for $0.50/GB. DDR6 is supposed to be released late this year or early next year so hopefully enterprise grade systems with DDR5 RAM and DDR5 RDIMMs will be getting cheaper on ebay by the end of next year.

We're now well under way: Been joined by a Simon McVittie (smcv) and we're almost through testing most of the standard images. Live image testing is being worked through. All good so far without identifying problems other than mistyping :)

For many years now, I ve been using an old YubiKey along with the free tier of Duo Security to add a second factor to my SSH logins. This is klunky, and has a number of drawbacks (dependency on a cloud service and Internet among them). I decided it was time to upgrade, so I recently bought a couple of YubiKey 5 series security keys. These support FIDO2/U2F, which make it so much easier to integrate with ssh. But in researching how to do this, I found a lot of pages online with poor instructions. Either they didn t explain what was going on very well, or suggested what I came to learn were insecure practices, or most often both. It turns out this whole process is quite easy. But I wanted to understand how it worked. So, I figured it out, set it up myself, and then put up a new, comprehensive page on my website: https://www.complete.org/easily-using-ssh-with-fido2-u2f-hardware-security-keys/. I hope it helps!

In Cottenham with Andy and the usual suspects. The point release update files are already on the servers - anyone can do an "apt-get update ; apt-get dist-upgrade" and update any running machine. This machine has just been upgraded and "just worked".

Here to do release testing for the images that we will end up publishing later in the day.Expecting one more of us to turn up a bit later. Team will be working on IRC on #debian-cd

On the way to Trixie, polkitd (Policy Kit Daemon) has lost the functionality to evaluate its .pkla (Polkit Local Authority) files.

$ zcat /usr/share/doc/polkitd/NEWS.Debian.gz 
policykit-1 (121+compat0.1-2) experimental; urgency=medium
  This version of polkit changes the syntax used for local policy rules:
  it is now the same JavaScript-based format used by the upstream polkit
  project and by other Linux distributions.
  System administrators can override the default security policy by
  installing local policy overrides into /etc/polkit-1/rules.d/*.rules,
  which can either make the policy more restrictive or more
  permissive. Some sample policy rules can be found in the
  /usr/share/doc/polkitd/examples directory. Please see polkit(8) for
  more details.
  Some Debian packages include security policy overrides, typically to
  allow members of the sudo group to carry out limited administrative
  actions without re-authenticating. These packages should install their
  rules as /usr/share/polkit-1/rules.d/*.rules. Typical examples can be
  found in packages like flatpak, network-manager and systemd.
  Older Debian releases used the "local authority" rules format from
  upstream version 0.105 (.pkla files with an .desktop-like syntax,
  installed into subdirectories of /etc/polkit-1/localauthority
  or /var/lib/polkit-1/localauthority). The polkitd-pkla package
  provides compatibility with these files: if it is installed, they
  will be processed at a higher priority than most .rules files. If the
  polkitd-pkla package is removed, .pkla files will no longer be used.
 -- Simon McVittie   Wed, 14 Sep 2022 21:33:22 +0100

This applies now to the polkitd version 126-2 destined for Trixie. The most prominent issue is that you will get an error message: "Authentication is required to create a color profile" asking for the root(!) password every time you remotely log into a Debian Trixie system via RDP, x2go or the like. This used to be mendable with a .pkla file dropped into /etc/polkit-1/localauthority/50-local.d/ ... but these .pkla files are void now and need to be replace with a Javascript "rules" file. The background to his is quite a fascinating read ... 13 years later:
https://davidz25.blogspot.com/2012/06/authorization-rules-in-polkit.html The solution has been listed in DevAnswers as other distros (Fedora, ArchLinux, OpenSuse) have been faster to depreciate the .pkla files and require .rules files. I amended the solution given there with checking for root to be automatically authenticated, too. So, create a 50-color-manager.rules file in /etc/polkit-1/rules.d/: and run systemctl restart polkit. You should be good until polkit is rewritten in Rust.

Debian hard freeze on 2025-05-15? We bring you a new Grml release on top of that! 2025.05 codename Nudlaug. There s plenty of new stuff, check out our official release announcement for all the details. But I d like to highlight one feature that I particularly like: SSH service announcement with Avahi. The grml-full flavor ships Avahi, and when you enable SSH, it automatically announces the SSH service on your local network. So when f.e. booting Grml with boot option ssh=debian , you should be able to login on your Grml live system with ssh grml@grml.local and password debian :

% insecssh grml@grml.local
Warning: Permanently added 'grml.local' (ED25519) to the list of known hosts.
grml@grml.local's password: 
Linux grml 6.12.27-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.27-1 (2025-05-06) x86_64
Grml - Linux for geeks
grml@grml ~ %

Hint: grml-zshrc provides that useful shell alias insecssh , which is aliased to ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null . Using those options, you aren t storing the SSH host key of the (temporary) Grml live system (permanently) in your UserKnownHostsFile. BTW, you can run avahi-browse -d local _ssh._tcp resolve -t to discover the SSH services on your local network. Happy Grml-ing!

The diffoscope maintainers are pleased to announce the release of diffoscope version 296. This version includes the following changes: You find out more by visiting the project homepage.

Like each month, have a look at the work funded by Freexian s Debian LTS offering.

Debian LTS contributors In April, 22 contributors have been paid to work on Debian LTS, their reports are available:

• Adrian Bunk did 56.25h (out of 56.25h assigned).
• Andreas Henriksson did 15.0h (out of 20.0h assigned), thus carrying over 5.0h to the next month.
• Andrej Shadura did 10.0h (out of 6.0h assigned and 4.0h from previous period).
• Bastien Roucari s did 31.5h (out of 31.5h assigned).
• Ben Hutchings did 8.0h (out of 0.0h assigned and 12.0h from previous period), thus carrying over 4.0h to the next month.
• Carlos Henrique Lima Melara did 11.0h (out of 12.0h assigned), thus carrying over 1.0h to the next month.
• Chris Lamb did 18.0h (out of 18.0h assigned).
• Daniel Leidert did 26.0h (out of 26.0h assigned).
• Emilio Pozuelo Monfort did 30.0h (out of 39.25h assigned and 0.25h from previous period), thus carrying over 9.5h to the next month.
• Guilhem Moulin did 8.5h (out of 3.25h assigned and 11.75h from previous period), thus carrying over 6.5h to the next month.
• Jochen Sprickerhof did 12.5h (out of 20.75h assigned and 9.25h from previous period), thus carrying over 17.5h to the next month.
• Lee Garrett did 26.25h (out of 7.75h assigned and 31.75h from previous period), thus carrying over 13.25h to the next month.
• Lucas Kanashiro did 50.0h (out of 0.0h assigned and 52.0h from previous period), thus carrying over 2.0h to the next month.
• Markus Koschany did 39.5h (out of 39.5h assigned).
• Roberto C. S nchez did 9.0h (out of 0.0h assigned and 12.0h from previous period), thus carrying over 3.0h to the next month.
• Santiago Ruano Rinc n did 12.5h (out of 7.5h assigned and 7.5h from previous period), thus carrying over 2.5h to the next month.
• Sean Whitton did 7.0h (out of 7.0h assigned).
• Stefano Rivera did 0.5h (out of 0.0h assigned and 10.0h from previous period), thus carrying over 9.5h to the next month.
• Sylvain Beucler did 39.5h (out of 39.25h assigned and 0.25h from previous period).
• Thorsten Alteholz did 15.0h (out of 15.0h assigned).
• Tobias Frost did 12.0h (out of 7.75h assigned and 4.25h from previous period).
• Utkarsh Gupta did 2.0h (out of 2.0h assigned).

Evolution of the situation In April, we released 46 DLAs.

• Notable security updates:
  • jetty9, prepared by Markus Koschany, fixes an information disclosure and potential remote code execution vulnerability
  • zabbix, prepared by Tobias Frost, fixes several vulnerabilities, encompassing denial of service, information disclosure or remote code inclusion
  • glibc, prepared by Sean Whitton, fixes a buffer overflow vulnerability
• Notable non-security updates:
  • tzdata, prepared by Emilio Pozuelo Monfort, brings the latest timezone database release
  • php-horde-editor and php-horde-imp, prepared by Sylvain Beucler, have been updated to switch from CKEditor v3, which is EOL, to CKEditor v4; this builds upon work done last month by Sylvain and Bastien for the complete removal of ckeditor3
  • distro-info-data, prepared by Stefano Rivera, adds information concerning future Debian and Ubuntu releases

The LTS team continues to welcome the collaboration of maintainers and other interested parties from outside the regular team. In April, we had external updates contributed by: Yadd - lemonldap-ng and Moritz Schlarb - libapache2-mod-auth-openidc A point release of the current stable Debian 12 (codename bookworm ) is planned for mid-May and several LTS contributors have prepared packages for this update, many of them prepared in conjunction with related LTS updates of the same packages:

• glib2.0, haproxy, imagemagick, poppler, and python-h11, prepared by Adrian Bunk
• rubygems, prepared by Lucas Kanashiro
• ruby3.1 (in collaboration with Lucas Kanashiro), twitter-bootstrap3, twitterboot-strap4, wpa, and erlang, prepared by Bastien Roucari s (corresponding updates of twitter-bootstrap3 and twitter-bootstrap4 were also uploaded to Debian unstable)
• abseil, prepared by Tobias Frost (a corresponding update was also uploaded to Debian unstable)
• vips, prepared by Guilhem Moulin

Additional updates of ruby3.3 and rubygems were prepared for Debian unstable by Lucas Kanashiro. And finally, a highlight of our continued commitment to enhancing long term support efforts in upstream projects. Freexian, as the primary entity behind the management and execution of the LTS project, has partnered with Invisible Things Lab to extend the upstream security support of Xen 4.17, which is shipped in Debian 12 bookworm (the current stable release). This partnership will result in significantly improved lifecycle support for users of Xen on bookworm, and members of the LTS team will play a part in this endeavour. The Freexian announcement has additional details.

Thanks to our sponsors Sponsors that joined recently are in bold.

• Platinum sponsors:
  • Toshiba Corporation (for 115 months)
  • Civil Infrastructure Platform (CIP) (for 83 months)
  • VyOS Inc (for 47 months)
• Gold sponsors:
  • Roche Diagnostics International AG (for 125 months)
  • Akamai - Linode (for 120 months)
  • Babiel GmbH (for 109 months)
  • Plat Home (for 108 months)
  • University of Oxford (for 65 months)
  • Deveryware (for 53 months)
  • EDF SA (for 37 months)
  • Dataport A R (for 12 months)
  • CERN (for 10 months)
• Silver sponsors:
  • Domeneshop AS (for 130 months)
  • Nantes M tropole (for 124 months)
  • Univention GmbH (for 116 months)
  • Universit Jean Monnet de St Etienne (for 116 months)
  • Ribbon Communications, Inc. (for 110 months)
  • Exonet B.V. (for 100 months)
  • Leibniz Rechenzentrum (for 94 months)
  • Minist re de l Europe et des Affaires trang res (for 78 months)
  • Cloudways by DigitalOcean (for 67 months)
  • Dinahosting SL (for 65 months)
  • Bauer Xcel Media Deutschland KG (for 59 months)
  • Platform.sh SAS (for 59 months)
  • Moxa Inc. (for 53 months)
  • sipgate GmbH (for 51 months)
  • OVH US LLC (for 49 months)
  • Tilburg University (for 49 months)
  • GSI Helmholtzzentrum f r Schwerionenforschung GmbH (for 40 months)
  • THINline s.r.o. (for 13 months)
  • Copenhagen Airports A/S (for 7 months)
• Bronze sponsors:
  • Seznam.cz, a.s. (for 131 months)
  • Evolix (for 130 months)
  • Intevation GmbH (for 127 months)
  • Linuxhotel GmbH (for 127 months)
  • Daevel SARL (for 126 months)
  • Bitfolk LTD (for 125 months)
  • Megaspace Internet Services GmbH (for 125 months)
  • Greenbone AG (for 124 months)
  • NUMLOG (for 124 months)
  • WinGo AG (for 123 months)
  • Entr ouvert (for 115 months)
  • Adfinis AG (for 112 months)
  • Laboratoire LEGI - UMR 5519 / CNRS (for 107 months)
  • Tesorion (for 107 months)
  • Bearstech (for 98 months)
  • LiHAS (for 98 months)
  • Catalyst IT Ltd (for 93 months)
  • Supagro (for 88 months)
  • Demarcq SAS (for 87 months)
  • Universit Grenoble Alpes (for 73 months)
  • TouchWeb SAS (for 65 months)
  • SPiN AG (for 62 months)
  • CoreFiling (for 58 months)
  • Institut des sciences cognitives Marc Jeannerod (for 53 months)
  • Observatoire des Sciences de l Univers de Grenoble (for 49 months)
  • Tem Innovations GmbH (for 44 months)
  • WordFinder.pro (for 44 months)
  • CNRS DT INSU R sif (for 42 months)
  • Soliton Systems K.K. (for 38 months)
  • Alter Way (for 35 months)
  • Institut Camille Jordan (for 25 months)
  • SOBIS Software GmbH (for 10 months)
  • Tuxera Inc.

After more than ten years on my trusted X250, and with a lot of financial help for Debian (which I really thank, more on that later), I finally jumped on a new ThinkPad, an X13 Gen 5. The migration path was really easy: I'm doing daily backups with borg of the whole filesystems on an encrypted USB drive, so I just had to boot a live USB key on the new laptop, plug the USB drive, create the partitioning (encryption, LVM etc.) and then run borg extract. Since I'm using LABEL in the various fstab I didn't have much to change. I actually had a small hiccup because my daily backup scripts used ProtectKernelModules and besides preventing loading modules into the running kernel, it also prevents access to /usr/lib/modules. So when restoring I didn't have any modules for the installed kernels. No big deal, I reinstalled the kernel package from the chroot and it did work just fine. All in all it was pretty smooth. I've started a similar page as the X250 for the X13G5 but honestly I don't think I'll have to document a lot of stuff because everything basically works out of the box. It's not really a surprise because we went a long way since 2015 and Linux kernels are really tested on a lot of hardware, including laptops these days, and Intel laptops are the most standard stuff you can find. I guess it's still rocky for ARM64 laptops (and especially Apple hardware) but the point was less to do porting work for Debian and rather beeing more efficient for the current stuff I maintain (and sometimes struggle with). As said above, the laptop has been funded by Debian and I really thank the DPL and the Debian France treasurer for authorizing it and beeing really fast on the reimbursement. I had already posted a long time ago about hardware funding for Debian developers. It took me quite a while but I finally managed to ask for help because I couldn't afford the hardware at this point and it was becoming problematic. This is not something which should be done lightly (Debian wouldn't have the funds) but this is definitely something which should be done if needed. Don't hesitate to ask your fellow Debian developpers about advice on this.

To build our local voice satellite on a Debian system rather than using the ATOM Echo device we need something that can handle the wake word component; the piece that means we only send audio to the Home Assistant server for processing by whisper.cpp when we ve detected someone is trying to talk to us. openWakeWord seems to be one of the better ways to do this, and is well supported. However. It relies on TensorFlow Lite (now LiteRT) which is a complicated mess of machine learning code. tflite-runtime is available from PyPI, but that s prebuilt and we re trying to avoid that. Despite, on initial impressions, it looking quite complicated to deal with building TensorFlow - Bazel is an immediate warning - it turns out to be incredibly simple to build your own .deb: This is hiding an awful lot of complexity, however. In particular the number of 3rd party projects that are being downloaded in the background (and compiled, to be fair, rather than using binary artefacts). We can build the main C++ wrapper .so directly with cmake, allowing us to investigate a bit more: Looking at the output we can see that pthreadpool, FXdiv, FP16 + PSimd are all downloaded, and seem to have ways to point to a local copy. That seems positive. However, there are even more hidden dependencies, which we can see if we look in the _deps/ subdirectory of the build tree. These don t appear to be as easy to override, and not all of them have packages already in Debian. First, the ones that seem to be available: abseil-cpp, cpuinfo, eigen, farmhash, flatbuffers, gemmlowp, ruy + xnnpack (lots of credit to the Debian Deep Learning Team for these, and in particular Mo Zhou) Dependencies I couldn t see existing packages for are: OouraFFT, ml_dtypes & neon2sse. At this point I just used the package I built with the initial steps above. I live in hope someone will eventually package this properly for Debian, or that I ll find the time to try and help out, but that s not going to be today. I wish upstream developers made it easier to use system copies of their library dependencies. I wish library developers made it easier to build and install system copies of their work. pkgconf is not new tech these days (pkg-config appears to date back to 2000), and has decent support in CMake. I get that there can be issues with incompatibilities even in minor releases, or awkwardness in doing builds of multiple connected projects, but at least give me the option to do so.

... or how I spent my lunch break today. An increasing amount of news outlets (hello heise.de) start to embed bullshit which requires DRM playback. Since I keep that disabled I now get an infobar that tells me that I need to enable it for this page. Pretty useless and a pain in the back because it takes up screen space. Here's the quick way how to get rid of it:

1. Go to about:config and turn on toolkit.legacyUserProfileCustomizations.stylesheets.
2. Go to your Firefox profile folder (e.g. ~/.mozilla/firefox/<random-value>.default/) and mkdir chrome && touch chrome/userChrome.css.
3. Add the following to your userChrome.css file:

    .infobar[value="drmContentDisabled"]  
      display: none !important;
     
   

4. Restart Firefox and read news again with full screen space.

I'm on a bit of an Orbital kick at the moment. Last year they re-issued their 1991 debut album with 43 extra tracks. Later this month they're doing the same for their 1993 sophomore album. I thought I'd try to narrow down some tracks to recommend. I seem to have settled on roughly 5 in previous posts (for Underworld, The Cure, Coil and Gazelle Twin). This time I've done 6 (I borrowed one from Underworld) As always it's a hard choice. I've tried to select some tracks I really enjoy that don't often come up on best-of compilation albums. For a more conventional choice of best-of tracks, I recommend the recent-ish 30 something "compilation" (of sorts, previously written about)

---

1. The Naked and the Dead (1992) The Naked and the Dead by Orbital From an early EP Radiccio, which is being re-issued this month. Digital versions of the re-issue will feature a new recording "Deepest" featuring Tilda Swinton. Sadly this isn't making it onto the pressed version. She performed with them live at Glastonbury 2024. That entire performance was a real pick-me-up during my convolescence, and is recommended. Anyway I've now written more about a song I haven't recommended than the one I did
2. Remind (1993) Remind by Orbital From the Brown Album, I first heard this as the Encore from their "final show", for John Peel, when they split up in 2004. "Remind" wasn't broadcast, but an audience recording was circulated on fan site Loopz. Remarkably, 21 years on, it's still there. In writing this I discovered that it's a re-working of a remix Orbital did for Meat Beat Manifesto: MindStream (Mind The Bend The Mind)
3. You Lot (2004) From the unfairly-maligned "final" Blue album. Featuring a sample of pre-Doctor Who Christoper Eccleston, from another Russell T Davies production, Second Coming.
4. Beached (2000) Beached (Long version) by Orbital, Angelo Badalamenti Co-written by Angelo Badalamenti, it's built around a sample of Badalamenti's score for the movie "The Beach". Orbital's re-work adds some grit to the orchestral instrumentation and opens with a monologue, delivered by Leonardo Di Caprio, sampled from the movie.
5. Spare Parts Express (1999) Spare Parts Express by Orbital Critics had started to be quite unfair to Orbital by this point. The band themselves said that they'd ran out of ideas (pointing at album closer "Style", built around a Stylophone melody, as proof). Their malaise continued right up to the Blue Album, at which point the split up; ostensibly for good, before regrouping 8 years later. Spare Parts Express is a hatchet job of various bits that they didn't develop into full songs on their own. Despite this I think it works. I love long-form electronica, and this clocks in at 10:07. My favourite segment (06:37) is adjacent to a reference (05:05) to John Baker's theme for the BBC children's program Newsround (sadly they aren't using it today. Here's a rundown of Newsround themes over time)
6. Attached (1994) Attached by Orbital This originally debuted on a Peel session before appearing on the subsequent album Snivilisation a few months later. An album closer, and a good come-down song to close this list.

Everybody (who runs containers) knows this situation: you've been running happycontainer:stable for a while and it's been great but now something external changed and you need to adjust the code while there is still no release with the patch. I've encountered exactly this when our Home-Assistant stopped showing the presence of our cat correctly, but we've also been discussing this at work recently. Now the most obvious (to me?) solution would be to build a new container, based on the original one, and perform the modifications at build time. Something like this: But that's not interactive, and if you don't have a patch readily available, that's not what you want. (And I'll save you the idea of RUNing sed and friends to alter files!) You could run vim inside the container, but that requires vim to be installed there in the first place. And a reasonable configuration. And Well, turns out podman can mount the root fs of a running container. And if you're running as non-root, you'll get an error: Luckily the solution is in the error message - use podman unshare So in both cases (root and rootless) we get a path, which is the mounted root fs and we can edit things in there as we like. Once done, the container can be unmounted again, and the namespace left At this point we have modified the code inside the container, but the running process is still using the old code. If we restart the container now to restart the process, our changes will be lost. Instead, we can commit the changes as a new layer and tag the result. And now, when we restart the container, it will use the new code with our changes Is this the best workflow you can get? Probably not. Does it work? Hell yeah!

On 26th and 27th April we held a Debian bug-squashing party near Leuven, Belgium. Several longstanding and new Debian contributors gathered to work through some of the highest priority bugs affecting the upcoming release of Debian 13 trixie . We were hosted by the Familia community centre in Tildonk. As this venue currently does not have an Internet connection, we brought a mobile hotspot and a local Debian mirror. In attendance were:

• Debian Developers: Ben Hutchings, Nattie Mayer-Hutchings, Kurt Roeckx, and Geert Stappers
• New contributors: Y ce K r m, Louis Renuart, Arnout Vandecappelle

The new contributors were variously using Arch, Fedora, and Ubuntu, and the DDs spent some some time setting them up with Debian dvelopment environments. The bugs we worked on included:

• #994510: libunwind8 abuses setcontext() causing SIGSEGV on i386 with glibc >= 2.32: Kurt added a patch, though it seems that a different change is needed.
• #1016936: dwz: Unknown debugging section .debug_addr causes some builds to fail: Ben reduced the severity.
• #1063664: gcc-13-cross: file conflicts between gnat-13-<triplet> and gnat- 9,10 -<triplet>: Kurt looked at this and questioned the explanation for reopening this bug.
• #1060960: libslf4j-java: FTBFS: make: *** [debian/rules:4: build] Error 25: Ben identified that there have been 2 separate test regressions, and added a patch for the first of them.
• #1064003: src:cross-toolchain-base: unsatisfied build dependency in testing: linux-source-6.5 (>= 6.5.8): Ben closed this as already fixed.
• #1072167: grub-pc-dbg: newly-added symbol file does not provide any symbols: Ben reproduced this and added a patch.
• #1076350: Segfault with shared libuv on x86 (CVE-2025-47153): Kurt asked the maintainer for clarification of the status.
• #1078608: apt update silently leaves old index data: Ben reproduced this (again), and identified that it may be triggered by PackageKit.
• #1089192: golang-golang-x-net: FTBFS: FAIL: TestNumICANNRules: Y ce looked at this and wrote a patch. However, the maintainers took a different approach to fix this.
• #1089432: shim: Supporting rootless builds by default: Kurt asked the maintainer for a status update.
• #1091668: debian-installer: Explicitly declare requirement for root: Louis and Kurt looked at this. Kurt asked the maintainer for clarification of the status.
• #1084066: amdgcn-tools: Please upgrade build-dep to llvm/clang 18 or 19 and #1092643: llvm-toolchain-19: readd gcn targets: Kurt and Arnout looked at these. Arnout added information about the build failure, the relevant upstream changes, and an upstream bug report.
• #1095376: refpolicy: FTBFS: make[2]: *** [Rules.modular:230: validate] Segmentation fault: Arnout looked at this and was not able to reproduce it, but noted that it may actually be caused by a bug in swig.
• #1093160: rsync: failed verification update discarded: Ben reduced the severity.
• #1099013: emacs-gtk: hangs compositor when used under Wayland: Louis did a lot of work to reproduce this, but the bug was not reproducible.
• #1104169: wish: adduser _radvd on new installs and #1104229: wish: add systemd unit: Geert worked on and committed fixes for these bugs.
• #1100699: screen: hardcopy and screen-exchange are insecure by default: Y ce and Ben looked at this. Ben started a discussion of what changes would be appropriate to fix it.
• #1100981: libmlir-19 fails to coinstall: Arnout and Kurt looked at this, and pointed out an existing MR that should fix it and some other packages that are similarly affected.

Last month, I attended the KDE India conference in Gandhinagar, Gujarat from the 4th to the 6th of April. I made my mind to attend when Sahil told me about his plans to attend and giving a talk. A day after my talk submission, the organizer Bhushan contacted me on Matrix and informed me that my talk had been accepted. I was also informed that KDE will cover my travel and accommodation expenses. So, I planned to attend the conference at this point. I am a longtime KDE user, so why not ;) I arrived in Ahmedabad, the twin city of Gandhinagar, a day before the conference. The first thing that struck me as soon as I came out of the Ahmedabad airport was the heat. I felt as if I was being cooked exactly how Bhushan put it earlier in the group chat. I took a taxi to get to my hotel, which was close to the conference venue. Later that afternoon, I met Bhushan and Joseph. Joseph lived in Germany. Bhushan was taking him to get a SIM card, so I tagged along and got to roam around. Joseph was unsure about where to go after the conference, so I asked him what he wanted out of his trip and had conversations along that line. Later, Vishal convinced him to go to Lucknow. Since he was adamant about taking the train, I booked a Tatkal train ticket for him to Lucknow. He was curious about how Tatkal booking works and watched me in amusement while I was booking the ticket. The 4th of April marked the first day of the conference, with around 25 attendees. Bhushan started the day with an overview of KDE conferences in India, followed by Vishal, who discussed FOSS United s activities. After the lunch, Joseph gave an overview of his campaign to help people switch from Windows to GNU/Linux due to environmental and security reasons. He continued his session in detail the next day.

Conference hall

A key takeaway for me from Joseph s session was the idea pointed out by Adwaith: marketing GNU/Linux as a cheap alternative may not attract as much attention as marketing it as a status symbol. He gave the example of how the Tata Nano didn t do well in the Indian market due to being perceived as a poor person s car. My talk was scheduled for the evening of the first day. I hadn t prepared any slides because I wanted to make my session interactive. During my talk, I did an activity with the attendees to demonstrate the federated nature of XMPP messaging, of which Prav is a part. After the talk, I got a lot of questions, signalling engagement. The audience was cooperative (just like Prav ;)), contrary to my expectations (I thought they will be tired and sleepy). On the third day, I did a demo on editing OpenStreetMap (referred to as OSM in short) using the iD editor. It involved adding points to OSM based on the students suggestions. Since my computer didn t have an HDMI port, I used Subin s computer, and he logged into his OSM account for my session. Therefore, any mistakes I made will be under Subin s name. :) On the third day, I attended Aaruni s talk about backing up a GNU/Linux system. This was the talk that resonated with me the most. He suggested formatting the system with the btrfs file system during the installation, which helps in taking snapshots of the system and provides an easy way to roll back to a previous version if, for example, a file is accidentally deleted. I have tried many backup techniques, including this one, but I never tried backing up on the internal disk. I ll certainly give this a try. A conference is not only about the talks, that s why we had a Prav table as well ;) Just kidding. What I really mean is that a conference is more about interactions than talks. Since the conference was a three-day affair, attendees got plenty of time to bond and share ideas.

Prav stall at the conference

Conference group photo

After the conference, Bhushan took us to Adalaj Stepwell, an attraction near Gandhinagar. Upon entering the complex, we saw a park where there were many langurs. Going further, there were stairs that led down to a well. I guess this is why it is called a stepwell.

Adalaj Stepwell

Later that day, we had Gujarati Thali for dinner. It was an all-you-can-eat buffet and was reasonably priced at 300 rupees per plate. Aamras (Mango juice) was the highlight for me. This was the only time we had Gujarati food during this visit. After the dinner, Aaruni dropped Sahil and I off at the airport. The hospitality was superb - for instance, in addition to Aaruni dropping us, Bhushan also picked up some of the attendees from the airport. Finally, I would like to thank KDE for sponsoring my travel and accommodation costs. Let s wrap up this post here and meet you in the next one. Thanks to contrapunctus and Joseph for proofreading.

When I configured forgejo-actions I used a docker-compose.yaml file to execute the runner and a dind container configured to run using privileged mode to be able to build images with it; as mentioned on my post about my setup, the use of the privileged mode is not a big issue for my use case, but reduces the overall security of the installation. On a work chat the other day someone mentioned that the GitLab documentation about using kaniko says it is no longer maintained (see the kaniko issue #3348) so we should look into alternatives for kubernetes clusters. I never liked kaniko too much, but it works without privileged mode and does not need a daemon, which is a good reason to use it, but if it is deprecated it makes sense to look into alternatives, and today I looked into some of them to use with my forgejo-actions setup. I was going to try buildah and podman but it seems that they need to adjust things on the systems running them: As my runner was already configured to use dind I decided to look into sysbox as a way of removing the privileged flag to make things more secure but have the same functionality.

Installing the sysbox packageAs I use Debian and Ubuntu systems I used the .deb packages distributed from the sysbox release page to install it (in my case I used the one from the 0.6.7 version). On the machine running forgejo (a Debian 12 server) I downloaded the package, stopped the running containers (it is needed to install the package and the only ones running where the ones started by the docker-compose.yaml file) and installed the sysbox-ce_0.6.7.linux_amd64.deb package using dpkg.

Updating the docker-compose.yaml fileTo run the dind container without setting the privileged mode we set sysbox-runc as the runtime on the dind container definition and set the privileged flag to false (it is the same as removing the key, as it defaults to false):

--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -2,7 +2,9 @@ services:
   dind:
     image: docker:dind
     container_name: 'dind'
-    privileged: 'true'
+    # use sysbox-runc instead of using privileged mode
+    runtime: 'sysbox-runc'
+    privileged: 'false'
     command: ['dockerd', '-H', 'unix:///dind/docker.sock', '-G', '$RUNNER_GID']
     restart: 'unless-stopped'
     volumes:

Testing the changesAfter applying the changes to the docker-compose.yaml file we start the containers and to test things we re-run previously executed jobs to see if things work as before. In my case I re-executed the build-image-from-tag workflow #18 from the oci project and everything worked as expected.

ConclusionFor my current use case (docker + dind) seems that sysbox is a good solution but I m not sure if I ll be installing it on kubernetes anytime soon unless I find a valid reason to do it (last time we talked about it my co workers said that they are evaluating buildah and podman for kubernetes and probably we will use them to replace kaniko in our gitlab-ci pipelines and for those tools the use of sysbox seems an overkill).

Welcome to our fourth report from the Reproducible Builds project in 2025. These monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. Lastly, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. Table of contents:

1. reproduce.debian.net
2. Fifty Years of Open Source Software Supply Chain Security
3. 4th CHAINS Software Supply Chain Workshop
4. Mailing list updates
5. Canonicalization for Unreproducible Builds in Java
6. OSS Rebuild adds new TUI features
7. Distribution roundup
8. diffoscope & strip-nondeterminism
9. Website updates
10. Reproducibility testing framework
11. Upstream patches

---

reproduce.debian.net The last few months have seen the introduction, development and deployment of reproduce.debian.net. In technical terms, this is an instance of rebuilderd, our server designed monitor the official package repositories of Linux distributions and attempt to reproduce the observed results there. This month, however, we are pleased to announce that reproduce.debian.net now tests all the Debian trixie architectures except s390x and mips64el. The ppc64el architecture was added through the generous support of Oregon State University Open Source Laboratory (OSUOSL), and we can support the armel architecture thanks to CodeThink.

Fifty Years of Open Source Software Supply Chain Security Russ Cox has published a must-read article in ACM Queue on Fifty Years of Open Source Software Supply Chain Security. Subtitled, For decades, software reuse was only a lofty goal. Now it s very real. , Russ article goes on to outline the history and original goals of software supply-chain security in the US military in the early 1970s, all the way to the XZ Utils backdoor of 2024. Through that lens, Russ explores the problem and how it has changed, and hasn t changed, over time. He concludes as follows:

We are all struggling with a massive shift that has happened in the past 10 or 20 years in the software industry. For decades, software reuse was only a lofty goal. Now it s very real. Modern programming environments such as Go, Node and Rust have made it trivial to reuse work by others, but our instincts about responsible behaviors have not yet adapted to this new reality. We all have more work to do.

4th CHAINS Software Supply Chain Workshop Convened as part of the CHAINS research project at the KTH Royal Institute of Technology in Stockholm, Sweden, the 4th CHAINS Software Supply Chain Workshop occurred during April. During the workshop, there were a number of relevant workshops, including:

• Signature, Attestations and Reproducible Builds
• Does Functional Package Management Enable Reproducible Builds at Scale?
• Causes and Mitigations of Unreproducible Builds in Java [paper]
• Fixing Breaking Dependency Updates Using LLMs
• The caveats of vulnerability analysis
• maven-lockfile (Lockfiles for Java and Maven)
• observer (Generating SBOMs for C/C++)
• dirty-waters (Transparency checks for software supply chains)
• A supply chain competition. Martin Schwaighofer, the winner, created a recap video (20m43s).
• Finally, 8 posters on dependency introspection, diverse double compilation, dependency management, VEX and SBOM.

The full listing of the agenda is available on the workshop s website.

Mailing list updates On our mailing list this month:

• Luca DiMaio of Chainguard posted to the list reporting that they had successfully implemented reproducible filesystem images with both ext4 and an EFI system partition. They go on to list the various methods, and the thread generated at least fifteen replies.
• David Wheeler announced that the OpenSSF is building a glossary of sorts in order that they consistently use the same meaning for the same term and, moreover, that they have drafted a definition for reproducible build . The thread generated a significant number of replies on the definition, leading to a potential update to the Reproducible Build s own definition.
• Lastly, kpcyrd posted to the list with a timely reminder and update on their repro-env tool. As first reported in our July 2023 report, kpcyrd mentions that:

  My initial interest in reproducible builds was how do I distribute pre-compiled binaries on GitHub without people raising security concerns about them . I ve cycled back to this original problem about 5 years later and built a tool that is meant to address this. [ ]

Canonicalization for Unreproducible Builds in Java Aman Sharma, Benoit Baudry and Martin Monperrus have published a new scholarly study related to reproducible builds within Java. Titled Canonicalization for Unreproducible Builds in Java, the article s abstract is as follows:

[ ] Achieving reproducibility at scale remains difficult, especially in Java, due to a range of non-deterministic factors and caveats in the build process. In this work, we focus on reproducibility in Java-based software, archetypal of enterprise applications. We introduce a conceptual framework for reproducible builds, we analyze a large dataset from Reproducible Central and we develop a novel taxonomy of six root causes of unreproducibility. We study actionable mitigations: artifact and bytecode canonicalization using OSS-Rebuild and jNorm respectively. Finally, we present Chains-Rebuild, a tool that raises reproducibility success from 9.48% to 26.89% on 12,283 unreproducible artifacts. To sum up, our contributions are the first large-scale taxonomy of build unreproducibility causes in Java, a publicly available dataset of unreproducible builds, and Chains-Rebuild, a canonicalization tool for mitigating unreproducible builds in Java.

A full PDF of their article is available from arXiv.

OSS Rebuild adds new TUI features OSS Rebuild aims to automate rebuilding upstream language packages (e.g. from PyPI, crates.io and npm registries) and publish signed attestations and build definitions for public use. OSS Rebuild ships a text-based user interface (TUI) for viewing, launching, and debugging rebuilds. While previously requiring ownership of a full instance of OSS Rebuild s hosted infrastructure, the TUI now supports a fully local mode of build execution and artifact storage. Thanks to Giacomo Benedetti for his usage feedback and work to extend the local-only development toolkit. Another feature added to the TUI was an experimental chatbot integration that provides interactive feedback on rebuild failure root causes and suggests fixes.

Distribution roundup In Debian this month:

• Roland Clobus posted another status report on reproducible ISO images on our mailing list this month, with the summary that all live images build reproducibly from the online Debian archive .
• Debian developer Simon Josefsson published another two reproducibility-related blog posts this month, the first on the topic of Verified Reproducible Tarballs. Simon sardonically challenges the reader as follows: Do you want a supply-chain challenge for the Easter weekend? Pick some well-known software and try to re-create the official release tarballs from the corresponding Git checkout. Is anyone able to reproduce anything these days? After that, they also published a blog post on Building Debian in a GitLab Pipeline using their multi-stage rebuild approach.
• Roland also posted to our mailing list to highlight that there is now another tool in Debian that generates reproducible output, equivs . This is a tool to create trivial Debian packages that might Depend on other packages. As Roland writes, building the [equivs] package has been reproducible for a while, [but] now the output of the [tool] has become reproducible as well .
• Lastly, 9 reviews of Debian packages were added, 10 were updated and 10 were removed this month adding to our extensive knowledge about identified issues.

The IzzyOnDroid Android APK repository made more progress in April. Thanks to funding by NLnet and Mobifree, the project was also to put more time into their tooling. For instance, developers can now easily run their own verification builder in less than 5 minutes . This currently supports Debian-based systems, but support for RPM-based systems is incoming.

• The rbuilder_setup tool can now setup the entire framework within less than five minutes. The process is configurable, too, so everything from just the basics to verify builds up to a fully-fledged RB environment is also possible.
• This tool works on Debian, RedHat and Arch Linux, as well as their derivates. The project has received successful reports from Debian, Ubuntu, Fedora and some Arch Linux derivates so far.
• Documentation on how to work with reproducible builds (making apps reproducible, debugging unreproducible packages, etc) is available in the project s wiki page.
• Future work is also in the pipeline, including documentation, guidelines and helpers for debugging.

NixOS defined an Outreachy project for improving build reproducibility. In the application phase, NixOS saw some strong candidates providing contributions, both on the NixOS side and upstream: guider-le-ecit analyzed a libpinyin issue. Tessy James fixed an issue in arandr and helped analyze one in libvlc that led to a proposed upstream fix. Finally, 3pleX fixed an issue which was accepted in upstream kitty, one in upstream maturin, one in upstream python-sip and one in the Nix packaging of python-libbytesize. Sadly, the funding for this internship fell through, so NixOS were forced to abandon their search. Lastly, in openSUSE news, Bernhard M. Wiedemann posted another monthly update for their work there.

diffoscope & strip-nondeterminism diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading a number of versions to Debian:

• Use the --walk argument over the potentially dangerous alternative --scan when calling out to zipdetails(1). [ ]
• Correct a longstanding issue where many >-based version tests used in conditional fixtures were broken. This was used to ensure that specific tests were only run when the version on the system was newer than a particular number. Thanks to Colin Watson for the report (Debian bug #1102658) [ ]
• Address a long-hidden issue in the test_versions testsuite as well, where we weren t actually testing the greater-than comparisons mentioned above, as it was masked by the tests for equality. [ ]
• Update copyright years. [ ]

In strip-nondeterminism, however, Holger Levsen updated the Continuous Integration (CI) configuration in order to use the standard Debian pipelines via debian/salsa-ci.yml instead of using .gitlab-ci.yml. [ ]

Website updates Once again, there were a number of improvements made to our website this month including:

• Aman Sharma added OSS-Rebuild s stabilize tool to the Tools page. [ ][ ]
• Chris Lamb added a configure.ac (GNU Autotools) example for using SOURCE_DATE_EPOCH. [ ]. Chris also updated the SOURCE_DATE_EPOCH snippet and move the archive metadata to a more suitable location. [ ]
• Denis Carikli added GNU Boot to our ever-evolving Projects page.

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In April, a number of changes were made by Holger Levsen, including:

• reproduce.debian.net-related:
  • Add armel.reproduce.debian.net to support the armel architecture. [ ][ ]
  • Add a new ARM node, codethink05. [ ][ ]
  • Add ppc64el.reproduce.debian.net to support testing of the ppc64el architecture. [ ][ ][ ]
  • Improve the reproduce.debian.net front page. [ ][ ]
  • Make various changes to the ppc64el nodes. [ ][ ]9[ ][ ]
  • Make various changes to the arm64 and armhf nodes. [ ][ ][ ][ ]
  • Various changes related to the rebuilderd-worker entry point. [ ][ ][ ]
  • Create and deploy a pkgsync script. [ ][ ][ ][ ][ ][ ][ ][ ]
  • Fix the monitoring of the riscv64 architecture. [ ][ ]
  • Make a number of changes related to starting the rebuilderd service. [ ][ ][ ][ ]
• Backup-related:
  • Backup the rebuilder databases every week. [ ][ ][ ][ ]
  • Improve the node health checks. [ ][ ]
• Misc:
  • Re-use existing connections to the SSH proxy node on the riscv64 nodes. [ ][ ]
  • Node maintenance. [ ][ ][ ]

In addition:

• Jochen Sprickerhof fixed the risvc64 host names [ ] and requested access to all the rebuilderd nodes [ ].
• Mattia Rizzolo updated the self-serve rebuild scheduling tool, replacing the deprecated SSO -style authentication with OpenIDC which authenticates against salsa.debian.org. [ ][ ][ ]
• Roland Clobus updated the configuration for the osuosl3 node to designate 4 workers for bigger builds. [ ]

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann
  • lout
  • mpiP
  • libabigail
  • cython
  • mpiP
  • godot
  • gnome-text-editor
  • clisp
  • rpm
  • Fiona
  • qtdoc
• Chris Hofstaedtler:
  • #1104512 filed against command-not-found.
  • #1104517 filed against command-not-found.
  • #1104535 filed against cc65.
• Chris Lamb:
  • #1102659 filed against vcsh.
  • #1103797 filed against schism.
  • #1103798 filed against magic-wormhole-mailbox-server.
  • #1103800 filed against openvpn3-client.
• James Addison:
  • #1102760 filed against apg.
• Jochen Sprickerhof:
  • #1103288 filed against courier.
  • #1103563 filed against cross-toolchain-base.

Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org

After my previous posts related to Argo CD (one about argocd-autopilot and another with some usage examples) I started to look into Kluctl (I also plan to review Flux, but I m more interested on the kluctl approach right now). While reading an entry on the project blog about Cluster API somehow I ended up on the vCluster site and decided to give it a try, as it can be a valid way of providing developers with on demand clusters for debugging or run CI/CD tests before deploying things on common clusters or even to have multiple debugging virtual clusters on a local machine with only one of them running at any given time. On this post I will deploy a vcluster using the k3d_argocd kubernetes cluster (the one we created on the posts about argocd) as the host and will show how to:

Creating the virtual cluster

Installing the vcluster applicationTo create the virtual clusters we need the vcluster command, we can install it with arkade:

  arkade get vcluster

The vcluster.yaml fileTo create the cluster we are going to use the following vcluster.yaml file (you can find the documentation about all its options here):

controlPlane:
  proxy:
    # Extra hostnames to sign the vCluster proxy certificate for
    extraSANs:
    - my-vcluster-api.lo.mixinet.net
exportKubeConfig:
  context: my-vcluster_k3d-argocd
  server: https://my-vcluster-api.lo.mixinet.net:8443
  secret:
    name: my-vcluster-kubeconfig
sync:
  toHost:
    ingresses:
      enabled: true
    serviceAccounts:
      enabled: true
  fromHost:
    ingressClasses:
      enabled: true
    nodes:
      enabled: true
      clearImageStatus: true
    secrets:
      enabled: true
      mappings:
        byName:
          # sync all Secrets from the 'my-vcluster-default' namespace to the
          # virtual "default" namespace.
          "my-vcluster-default/*": "default/*"
          # We could add other namespace mappings if needed, i.e.:
          # "my-vcluster-kube-system/*": "kube-system/*"

On the controlPlane section we ve added the proxy.extraSANs entry to add an extra host name to make sure it is added to the cluster certificates if we use it from an ingress. The exportKubeConfig section creates a kubeconfig secret on the virtual cluster namespace using the provided host name; the secret can be used by GitOps tools or we can dump it to a file to connect from our machine. On the sync section we enable the synchronization of Ingress objects and ServiceAccounts from the virtual to the host cluster:

• We copy the ingress definitions to use the ingress server that runs on the host to make them work from the outside world.
• The service account synchronization is not really needed, but we enable it because if we test this configuration with EKS it would be useful if we use IAM roles for the service accounts.

On the opposite direction (from the host to the virtual cluster) we synchronize:

• The IngressClass objects, to be able to use the host ingress server(s).
• The Nodes (we are not using the info right now, but it could be interesting if we want to have the real information of the nodes running pods of the virtual cluster).
• The Secrets from the my-vcluster-default host namespace to the default of the virtual cluster; that synchronization allows us to deploy SealedSecrets on the host that generate secrets that are copied automatically to the virtual one. Initially we only copy secrets for one namespace but if the virtual cluster needs others we can add namespaces on the host and their mappings to the virtual one on the vcluster.yaml file.

Creating the virtual clusterTo create the virtual cluster we run the following command:

vcluster create my-vcluster --namespace my-vcluster --upgrade --connect=false \
  --values vcluster.yaml

It creates the virtual cluster on the my-vcluster namespace using the vcluster.yaml file shown before without connecting to the cluster from our local machine (if we don t pass that option the command adds an entry on our kubeconfig and launches a proxy to connect to the virtual cluster that we don t plan to use).

Adding an ingress TCP route to connect to the vcluster apiAs explained before, we need to create an IngressTcpRoute object to be able to connect to the vcluster API, we use the following definition:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: my-vcluster-api
  namespace: my-vcluster
spec:
  entryPoints:
    - websecure
  routes:
    - match: HostSNI( my-vcluster-api.lo.mixinet.net )
      services:
        - name: my-vcluster
          port: 443
  tls:
    passthrough: true

Once we apply those changes the cluster API will be available on the https://my-cluster-api.lo.mixinet.net:8443 URL using its own self signed certificate (we have enabled TLS passthrough) that includes the hostname we use (we adjusted it on the vcluster.yaml file, as explained before).

Getting the kubeconfig for the vclusterOnce the vcluster is running we will have its kubeconfig available on the my-vcluster-kubeconfig secret on its namespace on the host cluster. To dump it to the ~/.kube/my-vcluster-config we can do the following:

  kubectl get -n my-vcluster secret/my-vcluster-kubeconfig \
    --template=" .data.config "   base64 -d > ~/.kube/my-vcluster-config

Once available we can define the vkubectl alias to adjust the KUBECONFIG variable to access it:

alias vkubectl="KUBECONFIG=~/.kube/my-vcluster-config kubectl"

Or we can merge the configuration with the one on the KUBECONFIG variable and use kubectx or a similar tool to change the context (for our vcluster the context will be my-vcluster_k3d-argocd). If the KUBECONFIG variable is defined and only has the PATH to a single file the merge can be done running the following:

KUBECONFIG="$KUBECONFIG:~/.kube/my-vcluster-config" kubectl config view \
  --flatten >"$KUBECONFIG.new"
mv "$KUBECONFIG.new" "$KUBECONFIG"

On the rest of this post we will use the vkubectl alias when connecting to the virtual cluster, i.e. to check that it works we can run the cluster-info subcommand:

  vkubectl cluster-info
Kubernetes control plane is running at https://my-vcluster-api.lo.mixinet.net:8443
CoreDNS is running at https://my-vcluster-api.lo.mixinet.net:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Installing the dummyhttpd applicationTo test the virtual cluster we are going to install the dummyhttpd application using the following kustomization.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - https://forgejo.mixinet.net/blogops/argocd-applications.git//dummyhttp/?ref=dummyhttp-v1.0.0
# Add the config map
configMapGenerator:
  - name: dummyhttp-configmap
    literals:
      - CM_VAR="Vcluster Test Value"
    behavior: create
    options:
      disableNameSuffixHash: true
patches:
  # Change the ingress host name
  - target:
      kind: Ingress
      name: dummyhttp
    patch:  -
      - op: replace
        path: /spec/rules/0/host
        value: vcluster-dummyhttp.lo.mixinet.net
  # Add reloader annotations -- it will only work if we install reloader on the
  # virtual cluster, as the one on the host cluster doesn't see the vcluster
  # deployment objects
  - target:
      kind: Deployment
      name: dummyhttp
    patch:  -
      - op: add
        path: /metadata/annotations
        value:
          reloader.stakater.com/auto: "true"
          reloader.stakater.com/rollout-strategy: "restart"

It is quite similar to the one we used on the Argo CD examples but uses a different DNS entry; to deploy it we run kustomize and vkubectl:

  kustomize build .   vkubectl apply -f -
configmap/dummyhttp-configmap created
service/dummyhttp created
deployment.apps/dummyhttp created
ingress.networking.k8s.io/dummyhttp created

We can check that everything worked using curl:

  curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/   jq -cM .
 "c": "Vcluster Test Value","s": "" 

The objects available on the vcluster now are:

  vkubectl get all,configmap,ingress
NAME                             READY   STATUS    RESTARTS   AGE
pod/dummyhttp-55569589bc-9zl7t   1/1     Running   0          24s
NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/dummyhttp    ClusterIP   10.43.51.39    <none>        80/TCP    24s
service/kubernetes   ClusterIP   10.43.153.12   <none>        443/TCP   14m

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dummyhttp   1/1     1            1           24s
NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/dummyhttp-55569589bc   1         1         1       24s
NAME                            DATA   AGE
configmap/dummyhttp-configmap   1      24s
configmap/kube-root-ca.crt      1      14m
NAME                                CLASS   HOSTS                             ADDRESS                          PORTS AGE
ingress.networking.k8s.io/dummyhttp traefik vcluster-dummyhttp.lo.mixinet.net 172.20.0.2,172.20.0.3,172.20.0.4 80    24s

While we have the following ones on the my-vcluster namespace of the host cluster:

  kubectl get all,configmap,ingress -n my-vcluster
NAME                                                      READY   STATUS    RESTARTS   AGE
pod/coredns-bbb5b66cc-snwpn-x-kube-system-x-my-vcluster   1/1     Running   0          18m
pod/dummyhttp-55569589bc-9zl7t-x-default-x-my-vcluster    1/1     Running   0          45s
pod/my-vcluster-0                                         1/1     Running   0          19m
NAME                                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
service/dummyhttp-x-default-x-my-vcluster      ClusterIP   10.43.51.39     <none>        80/TCP                   45s
service/kube-dns-x-kube-system-x-my-vcluster   ClusterIP   10.43.91.198    <none>        53/UDP,53/TCP,9153/TCP   18m
service/my-vcluster                            ClusterIP   10.43.153.12    <none>        443/TCP,10250/TCP        19m
service/my-vcluster-headless                   ClusterIP   None            <none>        443/TCP                  19m
service/my-vcluster-node-k3d-argocd-agent-1    ClusterIP   10.43.189.188   <none>        10250/TCP                18m

NAME                           READY   AGE
statefulset.apps/my-vcluster   1/1     19m
NAME                                                     DATA   AGE
configmap/coredns-x-kube-system-x-my-vcluster            2      18m
configmap/dummyhttp-configmap-x-default-x-my-vcluster    1      45s
configmap/kube-root-ca.crt                               1      19m
configmap/kube-root-ca.crt-x-default-x-my-vcluster       1      11m
configmap/kube-root-ca.crt-x-kube-system-x-my-vcluster   1      18m
configmap/vc-coredns-my-vcluster                         1      19m
NAME                                                        CLASS   HOSTS                             ADDRESS                          PORTS AGE
ingress.networking.k8s.io/dummyhttp-x-default-x-my-vcluster traefik vcluster-dummyhttp.lo.mixinet.net 172.20.0.2,172.20.0.3,172.20.0.4 80    45s

As shown, we have copies of the Service, Pod, Configmap and Ingress objects, but there is no copy of the Deployment or ReplicaSet.

Creating a sealed secret for dummyhttpdTo use the hosts sealed secrets controller with the virtual cluster we will create the my-vcluster-default namespace and add there the sealed secrets we want to have available as secrets on the default namespace of the virtual cluster:

  kubectl create namespace my-vcluster-default
  echo -n "Vcluster Boo"   kubectl create secret generic "dummyhttp-secret" \
    --namespace "my-vcluster-default" --dry-run=client \
    --from-file=SECRET_VAR=/dev/stdin -o yaml >dummyhttp-secret.yaml
  kubeseal -f dummyhttp-secret.yaml -w dummyhttp-sealed-secret.yaml
  kubectl apply -f dummyhttp-sealed-secret.yaml
  rm -f dummyhttp-secret.yaml dummyhttp-sealed-secret.yaml

After running the previous commands we have the following objects available on the host cluster:

  kubectl get sealedsecrets.bitnami.com,secrets -n my-vcluster-default
NAME                                        STATUS   SYNCED   AGE
sealedsecret.bitnami.com/dummyhttp-secret            True     34s
NAME                      TYPE     DATA   AGE
secret/dummyhttp-secret   Opaque   1      34s

And we can see that the secret is also available on the virtual cluster with the content we expected:

  vkubectl get secrets
NAME               TYPE     DATA   AGE
dummyhttp-secret   Opaque   1      34s
  vkubectl get secret/dummyhttp-secret --template=" .data.SECRET_VAR " \
    base64 -d
Vcluster Boo

But the output of the curl command has not changed because, although we have the reloader controller deployed on the host cluster, it does not see the Deployment object of the virtual one and the pods are not touched:

  curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/   jq -cM .
 "c": "Vcluster Test Value","s": "" 

Installing the reloader applicationTo make reloader work on the virtual cluster we just need to install it as we did on the host using the following kustomization.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kube-system
resources:
- github.com/stakater/Reloader/deployments/kubernetes/?ref=v1.4.2
patches:
# Add flags to reload workloads when ConfigMaps or Secrets are created or deleted
- target:
    kind: Deployment
    name: reloader-reloader
  patch:  -
    - op: add
      path: /spec/template/spec/containers/0/args
      value:
        - '--reload-on-create=true'
        - '--reload-on-delete=true'
        - '--reload-strategy=annotations'

We deploy it with kustomize and vkubectl:

  kustomize build .   vkubectl apply -f -
serviceaccount/reloader-reloader created
clusterrole.rbac.authorization.k8s.io/reloader-reloader-role created
clusterrolebinding.rbac.authorization.k8s.io/reloader-reloader-role-binding created
deployment.apps/reloader-reloader created

As the controller was not available when the secret was created the pods linked to the Deployment are not updated, but we can force things removing the secret on the host system; after we do that the secret is re-created from the sealed version and copied to the virtual cluster where the reloader controller updates the pod and the curl command shows the new output:

  kubectl delete -n my-vcluster-default secrets dummyhttp-secret
secret "dummyhttp-secret" deleted
  sleep 2
  vkubectl get pods
NAME                         READY   STATUS        RESTARTS   AGE
dummyhttp-78bf5fb885-fmsvs   1/1     Terminating   0          6m33s
dummyhttp-c68684bbf-nx8f9    1/1     Running       0          6s
  curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/   jq -cM .
 "c":"Vcluster Test Value","s":"Vcluster Boo" 

If we change the secret on the host systems things get updated pretty quickly now:

  echo -n "New secret"   kubectl create secret generic "dummyhttp-secret" \
    --namespace "my-vcluster-default" --dry-run=client \
    --from-file=SECRET_VAR=/dev/stdin -o yaml >dummyhttp-secret.yaml
  kubeseal -f dummyhttp-secret.yaml -w dummyhttp-sealed-secret.yaml
  kubectl apply -f dummyhttp-sealed-secret.yaml
  rm -f dummyhttp-secret.yaml dummyhttp-sealed-secret.yaml
  curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/   jq -cM .
 "c":"Vcluster Test Value","s":"New secret" 

Pause and restore the vclusterThe status of pods and statefulsets while the virtual cluster is active can be seen using kubectl:

  kubectl get pods,statefulsets -n my-vcluster
NAME                                                                 READY   STATUS    RESTARTS   AGE
pod/coredns-bbb5b66cc-snwpn-x-kube-system-x-my-vcluster              1/1     Running   0          127m
pod/dummyhttp-587c7855d7-pt9b8-x-default-x-my-vcluster               1/1     Running   0          4m39s
pod/my-vcluster-0                                                    1/1     Running   0          128m
pod/reloader-reloader-7f56c54d75-544gd-x-kube-system-x-my-vcluster   1/1     Running   0          60m
NAME                           READY   AGE
statefulset.apps/my-vcluster   1/1     128m

Pausing the vclusterIf we don t need to use the virtual cluster we can pause it and after a small amount of time all Pods are gone because the statefulSet is scaled down to 0 (note that other resources like volumes are not removed, but all the objects that have to be scheduled and consume CPU cycles are not running, which can translate in a lot of savings when running on clusters from cloud platforms or, in a local cluster like the one we are using, frees resources like CPU and memory that now can be used for other things):

  vcluster pause my-vcluster
11:20:47 info Scale down statefulSet my-vcluster/my-vcluster...
11:20:48 done Successfully paused vcluster my-vcluster/my-vcluster
  kubectl get pods,statefulsets -n my-vcluster
NAME                           READY   AGE
statefulset.apps/my-vcluster   0/0     130m

Now the curl command fails:

  curl -s https://vcluster-dummyhttp.localhost.mixinet.net:8443
404 page not found

Although the ingress is still available (it returns a 404 because there is no pod behind the service):

  kubectl get ingress -n my-vcluster
NAME                                CLASS     HOSTS                               ADDRESS                            PORTS   AGE
dummyhttp-x-default-x-my-vcluster   traefik   vcluster-dummyhttp.lo.mixinet.net   172.20.0.2,172.20.0.3,172.20.0.4   80      120m

In fact, the same problem happens when we try to connect to the vcluster API; the error shown by kubectl is related to the TLS certificate because the 404 page uses the wildcard certificate instead of the self signed one:

  vkubectl get pods
Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority
  curl -s https://my-vcluster-api.lo.mixinet.net:8443/api/v1/
404 page not found
  curl -v -s https://my-vcluster-api.lo.mixinet.net:8443/api/v1/ 2>&1   grep subject
*  subject: CN=lo.mixinet.net
*  subjectAltName: host "my-vcluster-api.lo.mixinet.net" matched cert's "*.lo.mixinet.net"

Resuming the vclusterWhen we want to use the virtual cluster again we just need to use the resume command:

  vcluster resume my-vcluster
12:03:14 done Successfully resumed vcluster my-vcluster in namespace my-vcluster

Once all the pods are running the virtual cluster goes back to its previous state, although all of them were started, of course.

Cleaning upThe virtual cluster can be removed using the delete command:

  vcluster delete my-vcluster
12:09:18 info Delete vcluster my-vcluster...
12:09:18 done Successfully deleted virtual cluster my-vcluster in namespace my-vcluster
12:09:18 done Successfully deleted virtual cluster namespace my-vcluster
12:09:18 info Waiting for virtual cluster to be deleted...
12:09:50 done Virtual Cluster is deleted

That removes everything we used on this post except the sealed secrets and secrets that we put on the my-vcluster-default namespace because it was created by us. If we delete the namespace all the secrets and sealed secrets on it are also removed:

  kubectl delete namespace my-vcluster-default
namespace "my-vcluster-default" deleted

ConclusionsI believe that the use of virtual clusters can be a good option for two of the proposed use cases that I ve encountered in real projects in the past:

• need of short lived clusters for developers or teams,
• execution of integration tests from CI pipelines that require a complete cluster (the tests can be run on virtual clusters that are created on demand or paused and resumed when needed).

For both cases things can be set up using the Apache licensed product, although maybe evaluating the vCluster Platform offering could be interesting. In any case when everything is not done inside kubernetes we will also have to check how to manage the external services (i.e. if we use databases or message buses as SaaS instead of deploying them inside our clusters we need to have a way of creating, deleting or pause and resume those services).

Debian Contributions: 2025-04 Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

DebConf 25 Preparations, by Stefano Rivera and Santiago Ruano Rinc n DebConf 25 preparations continue. In April, the bursary team reviewed and ranked bursary applications. Santiago Ruano Rinc n examined the current state of the conference s finances, to see if we could allocate any more money to bursaries. Stefano Rivera supported the bursary team s work with infrastructure and advice and added some metrics to assist Santiago s budget review. Santiago was also involved in different parts of the organization, including Content team matters, as reviewing the first of proposals, preparing public information about the new Academic Track; or coordinating different aspects of the Day trip activities and the Conference Dinner.

PyPA tools updates, by Stefano Rivera Around the beginning of the freeze (in retrospect, definitely too late) Stefano looked at updating setuptools in the archive to 78.1.0. This brings support for more comprehensive license expressions (PEP-639), that people are expected to adopt soon upstream. While the reverse-autopkgtests all passed, it all came with some unexpected complications, and turned into a mini-transition. The new setuptools broke shebangs for scripts (pypa/setuptools#4952). It also required a bump of wheel to 0.46 and wheel 0.46 now has a dependency outside the standard library (it de-vendored packaging). This meant it was no longer suitable to distribute a standalone wheel.whl file to seed into new virtualenvs, as virtualenv does by default. The good news here is that setuptools doesn t need wheel any more, it included its own implementation of the bdist_wheel command, in 70.1. But the world hadn t adapted to take advantage of this, yet. Stefano scrambled to get all of these issues resolved upstream and in Debian:

• pip: Don t check for wheel when invoked with --no-use-pep517 (pypa/pip#13330), automatically do --no-use-pep517 builds without wheel (pypa/pip#13358, rejected).
• virtualenv: Don t include wheel (pypa/virtualenv#2868) except on Python 3.8 (pypa/virtualenv#2876) as pip dropped Python 3.8 support in the same release that included #13330.
• python3.13: Update bundled setuptools in test.wheeldata (python/cpython#132415).
• python-cffi: No need to install wheel any more (python-cffi/cffi#165).

We re now at the point where python3-wheel-whl is no longer needed in Debian unstable, and it should migrate to trixie.

Removing libcrypt-dev from build-essential, by Helmut Grohne The crypt function was originally part of glibc, but it got separated to libxcrypt. As a result, libc6-dev now depends on libcrypt-dev. This poses a cycle during architecture cross bootstrap. As the number of packages actually using crypt is relatively small, Helmut proposed removing the dependency. He analyzed an archive rebuild kindly performed by Santiago Vila (not affiliated with Freexian) and estimated the necessary changes. It looks like we may complete this with modifications to less than 300 source packages in the forky cycle. Half of the bugs have been filed at this time. They are tracked with libcrypt-* usertags.

Miscellaneous contributions

• Carles uploaded a new version of simplemonitor.
• Carles improved the documentation of salsa-ci-team/pipeline regarding piuparts arguments.
• Carles closed an FTBFS on gcc-15 on qnetload.
• Carles worked on Catalan translations using po-debconf-manager: reviewed 57 translations and created their merge requests in salsa, created 59 bug reports for packages that didn t merge in more than 30 days. Followed-up merge requests and comments in bug reports. Managed some translations manually for packages that are not in Salsa.
• Lucas did some work on the DebConf Content and Bursary teams.
• Lucas fixed multiple CVEs and bugs involving the upgrade from bookworm to trixie in ruby3.3.
• Lucas fixed a CVE in valkey in unstable.
• Stefano updated beautifulsoup4, python-authlib, python-html2text, python-packaging, python-pip, python-soupsieve, and unidecode.
• Stefano packaged python-dependency-groups, a new vendored library in python-pip.
• During an afternoon Bug Squashing Party in Montevideo, Santiago uploaded a couple of packages fixing RC bugs #1057226 and #1102487. The latter was a sponsored upload.
• Thorsten uploaded new upstream versions of brlaser, ptouch-driver and sane-airscan to get the latest upstream bug fixes into Trixie.
• Rapha l filed an upstream bug on zim for a graphical glitch that he has been experiencing.
• Colin Watson upgraded openssh to 10.0p1 (also known as 10.0p2), and debugged various follow-up bugs. This included adding riscv64 support to vmdb2 in passing, and enabling native wtmpdb support so that wtmpdb last now reports the correct tty for SSH connections.
• Colin fixed dput-ng s override option, which had never previously worked.
• Colin fixed a security bug in debmirror.
• Colin did his usual routine work on the Python team: 21 packages upgraded to new upstream versions, 8 CVEs fixed, and about 25 release-critical bugs fixed.
• Helmut filed patches for 21 cross build failures.
• Helmut uploaded a new version of debvm featuring a new tool debefivm-create to generate EFI-bootable disk images compatible with other tools such as libvirt or VirtualBox. Much of the work was prototyped in earlier months. This generalizes mmdebstrap-autopkgtest-build-qemu.
• Helmut continued reporting undeclared file conflicts and suggested package removals from unstable.
• Helmut proposed build profiles for libftdi1 and gnupg2. To deal with recently added dependencies in the architecture cross bootstrap package set.
• Helmut managed the /usr-move transition. He worked on ensuring that systemd would comply with Debian s policy. Dumat continues to locate problems here and there yielding discussion occasionally. He sent a patch for an upgrade problem in zutils.
• Anupa worked with the Debian publicity team to publish Micronews and Bits posts.
• Anupa worked with the DebConf 25 content team to review talk and event proposals for DebConf 25.