sparky Over the last 7 years, I ve been learning about radios I got my ham radio license (de K3XEC), hacked on some cool stuff where I ve learned how radios work by doing , and even was lucky enough to give my first rf-centric talk at districtcon. Embarrassingly, I still haven t gotten around to learning how the fancy stuff like GNU Radio works. I m sure I m going to love it when I do. As part of this, I ve also cooked up some very unprofessional formats and protocols I use for convenience. Locally, all my on-disk captures are stored in rfcap or more recently arf, while direct SDR access at my house is almost entirely a mix of the widely used rtl-tcp protocol, and my riq protocol (post on this coming soon). Both rtl-tcp and riq operate over the network, so I don t have to bother with plugging things into USB ports, and I can share my radios with my friends. All of that work sits in my current generation of radio processing code, sparky (a reference to spark-gap transmitters), which is a heap of Rust, supporting everything from no_std for embedded experiments, conditional support for interfacing with all the radios I own, and tokio-based async support in addition to blocking i/o for highly concurrent daemons. This quickly advanced beyond my old Go-based code (hz.tools/go-sdr), which I archived so I can focus on learning. I still think Go is a great language to write RF code in but I can t focus on that tech tree anymore. Of course, this now poses a new problem no one supports my format(s) or radio protocol(s), since, well, I m the only one using them. I ve committed a fair amount of my hardware to this setup, and yanking it from the rack to try something out does pose a bit of a pickle. This isn t a huge deal for learning, but it does make it tedious to try out something from the internets.

librtlsdr.so Thankfully, Rust has robust support for wrap[ping itself] in a grotesque simulacra of C s skin and mak[ing its] flesh undulate, which is an attractive nuisance if i ve ever seen one. Naturally, my ability to restrain myself from engaging in ill-advised rf adventures is basically zero, so it s time to do the thing any similarly situated person would do reimplement the API and ABI of librtlsdr.so, backed with sparky instead. Since enumeration of devices is going to be annoying (specifically, they re over the network), I decided early-on to rely on an explicit list of devices via a configuration file. I d rather only load that once so programs don t get confused, so I opted to use a CTOR to run a stub when the ELF is linked at runtime.

// lightly edited for clarity

#[used]
#[expect(unused)]
#[unsafe(link_section = ".init_array")]
pub static INITIALIZE: extern "C" fn() = sparky_rtlsdr_ctor;

#[unsafe(no_mangle)]
pub extern "C" fn sparky_rtlsdr_ctor()  
 let config: Config =  
 if let Ok(config_bytes) = std::fs::read("/etc/sparky-rtlsdr.toml")  
 toml::from_slice(&config_bytes).unwrap()
   else  
 Config   device: vec![]  
  
  ;
 CONFIG.set(config);
 

Next, it s time to start with the basics. Opening and closing a handle using rtlsdr_open and rtlsdr_close. Given we don t control the runtime, and the rtl-sdr device handle is opaque (for good reason!), I opted to smuggle a rust Box<Device> non-FFI safe heap-allocated struct through the device handle pointer, and let C take ownership of the Box. No one should be looking in there anyway.

// lightly edited for clarity

#[unsafe(no_mangle)]
pub unsafe extern "C" fn rtlsdr_open(dev: *mut *mut Handle, index: u32) -> int  
 let config = &CONFIG.device[index as usize];
 let sdr = match config.load()  
 Ok(v) => v,
 Err(err) =>  
 return -1;
  
  ;
 let handle = Box::new(Handle   config, sdr  );
 unsafe   *dev = Box::into_raw(handle)  ;
 0
 

#[unsafe(no_mangle)]
pub unsafe extern "C" fn rtlsdr_close(dev: *mut Handle) -> int  
 let dev = unsafe   Box::from_raw(dev)  ;
 drop(dev);
 0
 

With that in place, we can chip away at the API surface, translating calls as best as we can. I won t bother listing it all, since it s not very interesting but here s an example implementation of rtlsdr_set_sample_rate and rtlsdr_get_sample_rate. These calls are translating from an rtl-sdr frequency (which is a u32 containing the value as Hz) into a sparky Frequency type, and invoking get_sample_rate or set_sample_rate on the device s rust handle. Since each device implements the sparky Sdr trait, the actual underlying device doesn t matter much here.

#[unsafe(no_mangle)]
pub unsafe extern "C" fn rtlsdr_set_sample_rate(dev: *mut Handle, rate: u32) -> int  
 let dev = unsafe   &mut *dev  ;
 let rate = Frequency::from_hz(rate as i64);
 if let Err(err) = dev.sdr.set_sample_rate(dev.channel, rate)  
 return -1;
  
 0
 

#[unsafe(no_mangle)]
pub unsafe extern "C" fn rtlsdr_get_sample_rate(dev: *mut Handle) -> u32  
 let dev = unsafe   &mut *dev  ;
 let freq = match dev.sdr.get_sample_rate(dev.channel)  
 Ok(freq) => freq,
 Err(err) =>  
 return 0;
  
  ;
 freq.as_hz() as u32
 

After repeating this process for the rest of the stubs I could (and otherwise setting error conditions if the functionality is not supported), I was ready to try it out. Within sparky, I patched my MockSDR (basically a Sdr traited Mock type) to implement the same testmode IQ protocol that the RTL-SDR has, and decided to see if rtl_test from apt without any changes could be fooled.

$ rtl_test
No supported devices found.

Great, cool. No devices plugged in. Looks great. Let s try it with my librtlsdr.so LD_PRELOAD-ed into the binary first:

$ LD_PRELOAD=target/release/librtlsdr.so rtl_test
Found 1 device(s):
 0: hz.tools, mock sdr, SN: totally legit no tricks

Using device 0: sparky mock sdr
Supported gain values (0):
Sampling at 2048000 S/s.

Info: This tool will continuously read from the device, and report if
samples get lost. If you observe no further output, everything is fine.

Reading samples in async mode...
^CSignal caught, exiting!

User cancel, exiting...
Samples per million lost (minimum): 0
$

Outstanding. Even more outstandingly, if I change my testmode implementation to skip samples, rtl_test correctly reports the errors I think it s showing promise! On to try the real endgame here let s have our new librtlsdr.so connect to an rtl-tcp endpoint and see if rtl_fm works:

LD_PRELOAD=target/release/librtlsdr.so \
 rtl_fm -d 1 -s 120k -E deemp -M fm -f 90.9M   \
 ffplay -f s16le -ar 120k -i -
Found 2 device(s):
 0: hz.tools, mock sdr, SN: totally legit no tricks
 1: hz.tools, rtl-tcp, SN: node2.rf.lan:1202

Using device 1: sparky rtltcp node2
Tuner gain set to automatic.
Tuned to 91170000 Hz.
Oversampling input by: 9x.
Oversampling output by: 1x.
Buffer size: 7.59ms
Sampling at 1080000 S/s.
Output at 120000 Hz.

And there it was! Not the best audio quality (mostly due to my inability to correctly read the rtl_fm manpage to tune the filter and downsample/oversampling rates to audio), but it s definitely passable. I figured I d try something that was a bit more interesting next gqrx, since it s super handy, I use it a ton, and will definitely amuse me to no end. To my surprise and delight, LD_PRELOAD=target/release/librtlsdr.so gqrx wound up running, and I saw my devices pop right up in the setting menu: Huge. Huge. Amazing. It did crash as soon as I tried to actually use the radio, but after fixing a few dangling bugs in the API surface (and some assumptions I think some underlying gnuradio driver may be making that I need to double check in the code), I was able to get a super solid stream of broadcast fm radio, with gqrx being none the wiser. It thought it was just talking to the device it knows as rtl=1. Nice. I can t wait to try this with the rest of the rtl-sdr based tools I like having around using my riq protocol next. I don t think that ll be worth a post, but hopefully I ll get around to publishing details on that stack next.

epilogue Well. That s it. End of story. A bit anti-climatic, sure. While this new shim will provide me endless minutes of mild amusement, I could see using this to expose my sparky testing utilities via librtlsdr.so my mock sdr driver allows for replaying captures off disk, which could be interesting to make sure that signals are still properly decoded after changes, or instrument performance changes (via SNR, BER, packets observed, etc) on reference samples I have on my NAS. Maybe that ll come in handy one day! Truth be told, I m not sure I actually want to encourage anyone to do this for real (although I think I ll definitely be using it on my LAN to see what happens). I also don t have a repo to share I don t particularly feel with dealing with the secondary effects of publishing sparky (and sparky-rtlsdr) yet, since i m still getting my feet under me on the radio aspect of all this. I ll be sure to post updates if anything changes with this here (tagged sparky) and at @paul@soylent.green. I can t wait to post more about some of the odd sidequests (like this one!) i ve completed over the last few years I ve been waiting to feel confident that my work has matured and was withstood the new problems i ve thrown at it, and it largely has. It s my hope that these projects (and this project in particular) has provided a glimpse into the world of software defined radio for my systems friends, and a bit about systems for my radio friends. It s not all magic, and I hope someone out there feels inclined to have some fun with radios themselves!

tl;dr This is an experimental feature that, for the first time, brings full ECH support to curl on Debian using OpenSSL. Starting with curl 8.19.0-3+exp2 (Debian Experimental), you can now use ECH, with HTTPS-RR and DoH for maximum privacy. curl 8.19.0-3+exp2 is quite fresh at the time of writing, bear in mind that your repository might not have synced the package yet, all mirrors should have it by March 27th 14:00 UTC.

# defo.ie is a test server that confirms whether ECH was successfully used
curl -v --ech hard https://defo.ie/ech-check.php
# For Encrypted Client Hello (ECH) + DNS over HTTPS (DoH)
curl -v --ech hard --doh-url https://1.1.1.1/dns-query https://defo.ie/ech-check.php

"--ech hard" tells curl to refuse the connection entirely if ECH cannot be negotiated. Or, if you would like to try it out in a container:

podman run debian:experimental /bin/bash -c 'apt install --update -t experimental -y curl && curl -v --ech hard --doh-url https://1.1.1.1/dns-query https://defo.ie/ech-check.php'

(in case you haven't noticed, apt now has the --update option for the upgrade and install commands)

For Privacy CloudFlare calls it "the last puzzle piece to privacy" in their must-read announcement: https://blog.cloudflare.com/announcing-encrypted-client-hello/. Encrypted Client Hello (rfc9849) encrypts the "which website are you connecting to?" part of the TLS handshake that was previously visible in plaintext. HTTPS-RR (rfc9460) is a DNS record type that publishes connection parameters for a service, including the public key clients need to perform ECH. DNS Over HTTPS (rfc8484) encrypts DNS queries by tunneling them over HTTPS, hiding what domains you're looking up from network observers. When all three operate together over a CDN with shared IP space, the target domain name is hidden from passive observers; the HTTPS-RR record is queried over DoH in order to retrieve the ECH key (rfc9848) for the TLS handshake. Seems like quite an important feature, and in fact the major browsers have it enabled for some time now, the trick is that they do not use OpenSSL (Chrome uses BoringSSL and Firefox uses NSS). For everyone else, the only option is to patch OpenSSL or wait until 4.0.0 is released, and so part of the reason Debian is the first distro to enable it (curl + OpenSSL + ECH) is that the OpenSSL maintainer (Sebastian Andrzej Siewior) packaged the alpha release just 3 days after it was published. Do not forget that ECH support is experimental and currently relies on the alpha release of OpenSSL.

wcurl Gets It Too Considering wcurl is just a wrapper on curl, it gets the feature for free:

wcurl --curl-options="--ech hard --doh-url https://1.1.1.1/dns-query" $URL

If you're using wcurl, you don't want to have to set parameters, this is just to show that the feature is there and if you have a .curlrc file, it can enable the feature seamlessly.

Other Debian Releases Given the ECH feature requires OpenSSL >= 4, it will not make it to Debian 13, having a small chance of going to Debian 13 Backports (emphasis on small). It should get to Debian Unstable and Debian Testing within the next couple of months as the OpenSSL GA release happens and gets packaged, but you should be able to install the package from Experimental in your Unstable and Testing systems without issues. It will also be in Debian 14 once it becomes the new Stable.

Shoulders of Giants Stephen Farrell's presentation from OpenSSL Conference 2025 has a lot of background on the work involved: Encrypted Client Hello Lessons learned from trying to do something that was probably too complicated They have been working on implementing ECH in open-source projects for years, something as big as this doesn't happen without lots of people dedicating both their paid and free times over it. I ended up being the person who enabled it on Debian, which was pretty much the least amount of work between everyone involved, but hey it's fun flipping the switch and telling you about it.

Background Since 2025, the curl developers started organizing an yearly meeting with all maintainers of curl in Operating Systems. The 2026 edition happened in March 26th: https://github.com/curl/curl/wiki/curl-distro-discussion-2026. Attendance was really good, and as you can imagine one of the topics of discussion was ECH, in which it was pointed out that having OpenSSL 4 was the main requirement but besides it nothing unusual was needed. In Debian Experimental, we have been enabling HTTPS-RR since March 2025, and OpenSSL 4.0.0 alpha was packaged just recently (2026-03-13) by Sebastian Andrzej Siewior, it's time for the next step. The curl distro meeting was just the motivation I needed to go ahead and enable it in Debian Experimental, so as part of our Debian Brasil Weekly Meetings I've prepared and uploaded the changes, while Carlos Henrique Lima Melara worked on addressing a recent test regression for Debian Unstable. Unfortunately sergiodj couldn't join and I'm sure he's jealous of the hacking session now.

Appendix While writing this, I've noticed one of the authors of the CloudFlare blogpost is the previous curl maintainer on Debian; Alessandro Ghedini let me take over the maintenance back in 2021 and today curl is maintained by a team of 4 people, it's nice to see Alessandro's involvement.

• LLMs are changing how we work and will continue to do so.
• LLMs are vastly over-hyped by vested interests, and may be in a bubble.

• Huge investment in GenAI is having many negative consequences, ranging from environmental to causing affordability problems in many industries that use hardware (ie, everywhere)
• Useful results can be had from models that run on local hardware, even battery-powered hardware, which may have negligible harm or even some benefit

• GenAI is further concentrating wealth and power in megacorps, with the effect of squeezing out the smaller players even more.
• GenAI is lowering the cost of entry for people without a lot of resources already.

The equal alias The main idea looks simple: define = as an alias to a calculator command. I prefer Numbat, a scientific calculator that supports unit conversions. Qalculate is a close second.² If neither is available, we fall back to Zsh s built-in zcalc module. As the alias built-in uses = as a separator for name and value, we need to alter the aliases associative array:

if (( $+commands[numbat] )); then
  aliases[=]='numbat -e'
elif (( $+commands[qalc] )); then
  aliases[=]='qalc'
else
  autoload -Uz zcalc
  aliases[=]='zcalc -f -e'
fi

With this in place, = 847/11 becomes numbat -e 847/11.

The quoting problem The first problem surfaces quickly. Typing = 5 * 3 fails: Zsh expands the * character as a glob pattern before passing it to the calculator. The same issue applies to other characters that Zsh treats specially, such as > or . You must quote the expression:

$ = '5 * 3'
15

We fix this by hooking into the Zsh line editor to quote the expression before executing it.

Automatic quoting with ZLE Zsh calls the line-finish widget before submitting a command. We hook a function that detects the = prefix and quotes the expression:

_vbe_calc_quote()  
  case $BUFFER in
    "="*)
      typeset -g _vbe_calc_expr=$BUFFER # not used yet
      BUFFER="= $ (q-)$ $ BUFFER#= #  "
      ;;
  esac
 
add-zle-hook-widget line-finish _vbe_calc_quote

When you type = 5 * 3 and press , _vbe_calc_quote strips the = prefix, quotes the remainder with the (q-) parameter expansion flag, and rewrites the buffer to = '5 * 3' before Zsh submits the command. As a bonus, you can save a few keystrokes with =5*3! You can now compute math expressions and convert units directly from your shell. Zsh automatically quotes your expressions:

$ = '1 + 2'
3
$ = 'pi/3 + pi  > cos'
-0.5
$ = '17 USD -> EUR'
14.7122  
$ = '180*500mg -> g'
90 g
$ = '5 gigabytes / (2 minutes + 17 seconds) -> megabits/s'
291.971 Mbit/s
$ = 'now() -> tz("Asia/Tokyo")'
2026-03-22 22:00:03 JST (UTC +09), Asia/Tokyo
$ = '1 / (40 rods / hogshead) -> L / 100km'
118548   0.01 l/km

The metric system is the tool of the devil! My car gets forty rods to the hogshead, and that's the way I like it! Grampa Simpson, A Star Is Burns

Storing unquoted history As is, Zsh records the quoted expression in history. You must unquote it before submitting it again. Otherwise, the ZLE widget quotes it a second time. Bart Schaefer provided a solution to store the original version:

_vbe_calc_history()  
  return $ +_vbe_calc_expr 
 
add-zsh-hook zshaddhistory _vbe_calc_history
_vbe_calc_preexec()  
  (( $ +_vbe_calc_expr  )) && print -s $_vbe_calc_expr
  unset _vbe_calc_expr
  return 0
 
add-zsh-hook preexec _vbe_calc_preexec

The zshaddhistory hook returns 1 if we are evaluating an expression, telling Zsh not to record the command. The preexec hook then adds the original, unquoted command with print -s.

---

The complete code is available in my zshrc. A common alternative is the noglob precommand modifier. If you stick with to instead of -> for unit conversion, it covers 90% of use cases. For a related Zsh line editor trick, see how I use auto-expanding aliases to fix common typos.

---

1. This is the fastest a packet can travel back and forth between Paris and Marseille over optical fiber.
2. Qalculate is less understanding with units. For example, it parses Mbps as megabarn per picosecond:

   $ numbat -e '5 MB/s -> Mbps'
   40 Mbps
   $ qalc 5 MB/s to Mbps
   5 megabytes/second = 0.000005 B/ps
   

SSH Certificates And, thankfully, there is. OpenSSH supports certificates, an SSH public key that s been signed by some trusted party and so now you can assert that it s trustworthy in some form. SSH Certificates also contain metadata in the form of Principals, a list of identities that the trusted party included in the certificate. These might simply be usernames, but they might also provide information about group membership. There s also, unsurprisingly, native support in SSH for forwarding them (using the agent forwarding protocol), so you can keep your keys on your local system, ssh into your actual dev system, and have access to them without any additional complexity. And, wonderfully, you can use them in git! Let s find out how.

Local config There s two main parameters you need to set. First,

1	git config set gpg.format ssh

because unfortunately for historical reasons all the git signing config is under the gpg namespace even if you re not using OpenPGP. Yes, this makes me sad. But you re also going to need something else. Either user.signingkey needs to be set to the path of your certificate, or you need to set gpg.ssh.defaultKeyCommand to a command that will talk to an SSH agent and find the certificate for you (this can be helpful if it s stored on a smartcard or something rather than on disk). Thankfully for you, I ve written one. It will talk to an SSH agent (either whatever s pointed at by the SSH_AUTH_SOCK environment variable or with the -agent argument), find a certificate signed with the key provided with the -ca argument, and then pass that back to git. Now you can simply pass -S to git commit and various other commands, and you ll have a signature.

Validating signatures This is a bit more annoying. Using native git tooling ends up calling out to ssh-keygen², which validates signatures against a file in a format that looks somewhat like authorized-keys. This lets you add something like:

1	* cert-authority ssh-rsa AAAA

which will match all principals (the wildcard) and succeed if the signature is made with a certificate that s signed by the key following cert-authority. I recommend you don t read the code that does this in git because I made that mistake myself, but it does work. Unfortunately it doesn t provide a lot of granularity around things like Does the certificate need to be valid at this specific time and Should the user only be able to modify specific files and that kind of thing, but also if you re using GitHub or GitLab you wouldn t need to do this at all because they ll just do this magically and put a verified tag against anything with a valid signature, right? Haha. No. Unfortunately while both GitHub and GitLab support using SSH certificates for authentication (so a user can t push to a repo unless they have a certificate signed by the configured CA), there s currently no way to say Trust all commits with an SSH certificate signed by this CA . I am unclear on why. So, I wrote my own. It takes a range of commits, and verifies that each one is signed with either a certificate signed by the key in CA_PUB_KEY or (optionally) an OpenPGP key provided in ALLOWED_PGP_KEYS. Why OpenPGP? Because even if you sign all of your own commits with an SSH certificate, anyone using the API or web interface will end up with their commits signed by an OpenPGP key, and if you want to have those commits validate you ll need to handle that. In any case, this should be easy enough to integrate into whatever CI pipeline you have. This is currently very much a proof of concept and I wouldn t recommend deploying it anywhere, but I am interested in merging support for additional policy around things like expiry dates or group membership.

Doing it in hardware Of course, certificates don t buy you any additional security if an attacker is able to steal your private key material - they can steal the certificate at the same time. This can be avoided on almost all modern hardware by storing the private key in a separate cryptographic coprocessor - a Trusted Platform Module on PCs, or the Secure Enclave on Macs. If you re on a Mac then Secretive has been around for some time, but things are a little harder on Windows and Linux - there s various things you can do with PKCS#11 but you ll hate yourself even more than you ll hate me for suggesting it in the first place, and there s ssh-tpm-agent except it s Linux only and quite tied to Linux. So, obviously, I wrote my own. This makes use of the go-attestation library my team at Google wrote, and is able to generate TPM-backed keys and export them over the SSH agent protocol. It s also able to proxy requests back to an existing agent, so you can just have it take care of your TPM-backed keys and continue using your existing agent for everything else. In theory it should also work on Windows³ but this is all in preparation for a talk I only found out I was giving about two weeks beforehand, so I haven t actually had time to test anything other than that it builds. And, delightfully, because the agent protocol doesn t care about where the keys are actually stored, this still works just fine with forwarding - you can ssh into a remote system and sign something using a private key that s stored in your local TPM or Secure Enclave. Remote use can be as transparent as local use.

Wait, attestation? Ah yes you may be wondering why I m using go-attestation and why the term attestation is in my agent s name. It s because when I m generating the key I m also generating all the artifacts required to prove that the key was generated on a particular TPM. I haven t actually implemented the other end of that yet, but if implemented this would allow you to verify that a key was generated in hardware before you issue it with an SSH certificate - and in an age of agentic bots accidentally exfiltrating whatever they find on disk, that gives you a lot more confidence that a commit was signed on hardware you own.

Conclusion Using SSH certificates for git commit signing is great - the tooling is a bit rough but otherwise they re basically better than every other alternative, and also if you already have infrastructure for issuing SSH certificates then you can just reuse it⁴ and everyone wins.

---

1. Did you know you can just download people s SSH pubkeys from github from https://github.com/<username>.keys? Now you do
2. Yes it is somewhat confusing that the keygen command does things other than generate keys
3. This is more difficult than it sounds
4. And if you don t, by implementing this you now have infrastructure for issuing SSH certificates and can use that for SSH authentication as well.

The
WWW::Mechanize::Chrome Saga: A Comprehensive Narrative of PR #104 This document synthesizes the extensive work performed from March
13th to March 20th, 2026, to harden, stabilize, and refactor the
WWW::Mechanize::Chrome library and its test suite. This
effort involved deep dives into asynchronous programming,
platform-specific bug hunting, and strategic architectural
decisions.

---

Part I:
The Quest for Cross-Platform Stability (March 13 16) The initial phase of work focused on achieving a green test suite
across a variety of Linux distributions and preparing for a new release.
This involved significant hardening of the library to account for
different browser versions, OS-level security restrictions, and
filesystem differences.

Key Milestones &
Engineering Decisions:

• Fedora & RHEL-family Success: A major effort
  was undertaken to achieve a 100% pass rate on modern Fedora 43 and
  CentOS Stream 10. This required several key engineering decisions to
  handle modern browser behavior:
  • Decision: Implement Asynchronous DOM Serialization
    Fallback. Synchronous fallbacks in an async context are
    dangerous. To prevent Resource was not cached errors during
    saveResources, we implemented a fully asynchronous fallback
    in _saveResourceTree. By chaining
    _cached_document with DOM.getOuterHTML
    messages, we can reconstruct document content without blocking the event
    loop, even if Chromium has evicted the resource from its cache. This
    also proved resilient against Fedora s security policies, which often
    block file:// access.
  • Decision: Truncate Filenames for Cross-Platform
    Safety. To avoid File name too long errors,
    especially on Windows where the MAX_PATH limit is 260
    characters, filenameFromUrl was hardened. The filename
    truncation was reduced to a more conservative 150
    characters, leaving ample headroom for deeply nested CI
    temporary directories. Logic was also added to preserve file extensions
    during truncation and to sanitize backslashes from URI paths.
  • Decision: Expand Browser Discovery Paths. To
    support RHEL-based systems out-of-the-box, the
    default_executable_names was expanded to include
    headless_shell and search paths were updated to include
    /usr/lib64/chromium-browser/.
  • Decision: Mitigate Race Conditions with Stabilization Waits
    and Resilient Fetching. On fast systems,
    DOM.documentUpdated events could invalidate
    nodeIds immediately after navigation, causing XPath queries
    to fail with Could not find node with given id . A small stabilization
    sleep(0.25s) was added after page loads to ensure the DOM
    is settled. Furthermore, the asynchronous DOM fetching loop was hardened
    to gracefully handle these errors by catching protocol errors and
    returning an empty string for any node that was invalidated during
    serialization, ensuring the overall process could complete.
• Windows Hardening:
  • Decision: Adopt Platform-Aware Watchdogs. The test
    suite s reliance on ualarm was a blocker for Windows, where
    it is not implemented. The t::helper::set_watchdog function
    was refactored to use standard alarm() (seconds) on Windows
    and ualarm (microseconds) on Unix-like systems, enabling
    consistent test-level timeout enforcement.
• Version 0.77 Release:
  • Decision: Adopt SOP for Version Synchronization.
    The project maintains duplicate version strings across 24+ files. A
    Standard Operating Procedure was adopted to use a batch-replacement tool
    to update all sub-modules in lib/ and to always run
    make clean and perl Makefile.PL to ensure
    META.json and META.yml reflect the new
    version. After achieving stability on Linux, the project version was
    bumped to 0.77.
• Infrastructure & Strategic Work:
  • The ad2 Windows Server 2025 instance was restored and
    optimized, with Active Directory demoted and disk I/O performance
    improved.
  • A strategic proposal for the Heterogeneous Directory
    Replication Protocol (HDRP) was drafted and published.

---

Part II: The
Great Async Refactor (March 17 18) Despite success on Linux, tests on the slow ad2 Windows
host were still plagued by intermittent, indefinite hangs. This
triggered a fundamental architectural shift to move the library s core
from a mix of synchronous and asynchronous code to a fully non-blocking
internal API.

Key Milestones &
Engineering Decisions:

• Decision: Expose a _future API.
  Instead of hardcoding timeouts in the library, the core strategy was to
  refactor all blocking methods (xpath, field,
  get, etc.) into thin wrappers around new non-blocking
  ..._future counterparts. This moved timeout management to
  the test harness, allowing for flexible and explicit handling of
  stalls.

  # Example library implementation
  sub xpath($self, $query, %options)  
      return $self->xpath_future($query, %options)->get;
   
  
  sub xpath_future($self, $query, %options)  
      # Async implementation using $self->target->send_message(...)
   

• Decision: Centralize Test Hardening in a Helper.
  A dedicated test library, t/lib/t/helper.pm, was created to
  contain all stabilization logic. Safe wrappers (safe_get,
  safe_xpath) were implemented there, using
  Future->wait_any to race asynchronous operations against
  a timeout, preventing tests from hanging.

  # Example test helper implementation
  sub safe_xpath  
      my ($mech, $query, %options) = @_;
      my $timeout = delete $options timeout    5;
      my $call_f = $mech->xpath_future($query, %options);
      my $timeout_f = $mech->sleep_future($timeout)->then(sub   Future->fail("Timeout")  );
      return Future->wait_any($call_f, $timeout_f)->get;
   

• Decision: Refactor Node Attribute Cache.
  Investigations into flaky checkbox tests (t/50-tick.t)
  revealed that WWW::Mechanize::Chrome::Node was storing
  attributes as a flat list ([key, val, key, val]), which was
  inefficient for lookups and individual updates. The cache was refactored
  to definitively use a HashRef, providing O(1) lookups
  and enabling atomic dual-updates where both the browser property (via
  JS) and the internal library attribute are synchronized
  simultaneously.
• Decision: Implement Self-Cancelling Socket
  Watchdog. On Windows, traditional watchdog processes often
  failed to detect parent termination, leading to 60-second hangs after
  successful tests. We implemented a new socket-based watchdog in
  t::helper that listens on an ephemeral port; the background
  process terminates immediately when the parent socket closes,
  eliminating these cumulative delays.
• Decision: Deep Recursive Refactoring & Form
  Selection. To make the API truly non-blocking, the entire
  internal call stack had to be refactored. For example, making
  get_set_value_future non-blocking required first making its
  dependency, _field_by_name, asynchronous. This culminated
  in refactoring the entire form selection API (form_name,
  form_id, etc.) to use the new asynchronous
  _future lookups, which was a key step in mitigating the
  Windows deadlocks.
• Decision: Fix Critical Regressions & Memory
  Cycles.
  • Evaluation Normalization: Implemented a
    _process_eval_result helper to centralize the parsing of
    results from Runtime.evaluate. This ensures consistent
    handling of return values and exceptions between synchronous
    (eval_in_page) and asynchronous (eval_future)
    calls.
  • Memory Cycle Mitigation: A significant memory
    leak was discovered where closures attached to CDP event futures (like
    for asynchronous body retrieval) would capture strong references to
    $self and the $response object, creating a
    circular reference. The established rule is to now always use
    Scalar::Util::weaken on both $self and any
    other relevant objects before they are used inside a
    ->then block that is stored on an object.
  • Context Propagation (wantarray): A
    major regression was discovered where Perl s wantarray
    context, which distinguishes between scalar and list context, was lost
    inside asynchronous Future->then blocks. This caused
    methods like xpath to return incorrect results (e.g., a
    count instead of a list of nodes). The solution was to adopt the Async
    Context Pattern : capture wantarray in the synchronous
    wrapper, pass it as an option to the _future method, and
    then use that captured value inside the future s final resolution
    block.

    # Synchronous Wrapper
    sub xpath($self, $query, %options)  
        $options  wantarray   = wantarray; # 1. Capture
        return $self->xpath_future($query, %options)->get; # 2. Pass
     
    
    # Asynchronous Implementation
    sub xpath_future($self, $query, %options)  
        my $wantarray = delete $options  wantarray  ; # 3. Retrieve
        # ... async logic ...
        return $doc->then(sub  
            if ($wantarray)   # 4. Respect
                return Future->done(@results);
              else  
                return Future->done($results[0]);
             
         );
     

  • Asynchronous Body Retrieval & Robust Content
    Fallbacks: Fixed a bug where decoded_content()
    would return empty strings by ensuring it awaited a
    __body_future. This was implemented by storing the
    retrieval future directly on the response object
    ($response-> __body_future ). To make this more robust,
    a tiered strategy was implemented: first try to get the content from the
    network response, but if that fails (e.g., for about:blank
    or due to cache eviction), fall back to a JavaScript
    XMLSerializer to get the live DOM content.
  • Signature Hardening: Fixed Too few arguments
    errors when using modern Perl signatures with
    Future->then. Callbacks were updated to use optional
    parameters (sub($result = undef) ... ) to gracefully
    handle futures that resolve with no value.
  • XHTML Split-Brain Bug: Resolved a
    long-standing Chromium bug (40130141) where content provided via
    setDocumentContent is parsed differently than content
    loaded from a URL. A workaround was implemented: for XHTML documents,
    WMC now uses a JavaScript-based XPath evaluation
    (document.evaluate) against the live DOM, bypassing the
    broken CDP search mechanism.

Derived Architectural Rules
& SOPs:

• Rule: Always provide _future variants.
  Every library method that interacts with the browser via CDP must have a
  non-blocking asynchronous counterpart.
• Rule: Centralize stabilization in the test layer.
  All timeout and retry logic should reside in the test harness
  (t/lib/t/helper.pm), not in the core library.
• Rule: Explicitly propagate wantarray
  context. Synchronous wrappers must capture the caller s context
  and pass it down the Future chain to ensure correct
  scalar/list behavior.
• Rule: The entire call chain must be asynchronous.
  To enable non-blocking timeouts, even a single hidden blocking call in
  an otherwise asynchronous method will cause a stall.
• SOP: Reduce Library Noise. Diagnostic messages
  (warn, note, diag) should be
  removed from library code before commits. All such messages should be
  converted to use the internal $self->log('debug', ...)
  mechanism, ensuring a clean TAP output for CI systems.

---

Part III: The
MutationObserver Saga (March 19) With most of the library refactored to be asynchronous, one stubborn
test, t/65-is_visible.t, continued to fail with timeouts.
This led to an ambitious, but ultimately unsuccessful, attempt to
replace the wait_until_visible polling logic with a more
modern MutationObserver.

Key Milestones & Challenges:

• The Theory: The goal was to replace an inefficient
  repeat sleep loop with an event-driven
  MutationObserver in JavaScript that would notify Perl
  immediately when an element s visibility changed.
• Implementation & Cascade Failure: The
  implementation proved incredibly difficult and introduced a series of
  new, hard-to-diagnose bugs:
  1. An incorrect function signature for
     callFunctionOn_future.
  2. A critical unit mismatch, passing seconds from Perl to JavaScript s
     setTimeout, which expected milliseconds.
  3. A fundamental hang where the MutationObserver s
     JavaScript Promise would never resolve, even after the
     underlying DOM element changed.
• Debugging Maze: Multiple attempts to fix the
  checkVisibility JavaScript logic inside the observer
  callback, including making it more robust by adding DOM tree traversal
  and extensive console.log tracing, failed to resolve the
  hang. This highlighted the opacity and difficulty of debugging complex,
  cross-language asynchronous interactions, especially when dealing with
  low-level browser APIs.

Procedural Learning:
Granular Edits The effort was plagued by procedural missteps in using automated
file-editing tools. Initial attempts to replace large code blocks in a
single operation led to accidental code loss and match failures.

• Decision: Adopt Delete, then Add Workflow.
  Following forceful user correction, a new SOP was established for all
  future modifications:
  1. Isolate: Break the file into small, manageable
     chunks (e.g., 250 lines).
  2. Delete: Perform a delete operation by replacing
     the old code block with an empty string.
  3. Add: Perform an add operation by inserting the
     new code into the empty space.
  4. Verify: Verifying each atomic step before
     proceeding. This granular process, while slower, ensured surgical
     precision and regained technical control over the large
     Chrome.pm module.

The consistent failure of the MutationObserver approach
eventually led to the decision to abandon it in favor of stabilizing the
original, more transparent implementation.

---

Part IV:
Reversion and Final Stabilization (March 20) After exhausting all reasonable attempts to fix the
MutationObserver, a strategic decision was made to revert
to the simpler, more transparent polling implementation and fix it
correctly. This proved to be the correct path to a stable solution.

Key Milestones &
Engineering Decisions:

• Decision: Perform Strategic Reversion. The
  MutationObserver implementation, when integrated via
  callFunctionOn_future with awaitPromise,
  proved fundamentally unstable. Its JavaScript promise would consistently
  fail to resolve, causing indefinite hangs. A decision was made to
  revert all MutationObserver code from
  WWW::Mechanize::Chrome.pm and restore the original
  repeat sleep polling mechanism. A stable,
  understandable solution was prioritized over an elegant but broken
  one.
• Decision: Correct Timeout Delegation in the
  Harness. The root cause of the original timeout failure was
  identified as a race condition in the t/lib/t/helper.pm
  test harness. The safe_wait_until_* wrappers were
  implementing their own timeout (via wait_any and
  sleep_future) that raced against the underlying polling
  function s internal timeout. This led to intermittent failures on slow
  machines. The helpers were refactored to delegate all timeout
  management to the library s polling functions, ensuring a
  single, authoritative timer controlled the operation.
• Decision: Optimize Polling Performance. At the
  user s request, the polling interval was reduced from 300ms to
  150ms. This modest performance improvement reduced the
  test suite s wallclock execution time by over a second while maintaining
  stability.
• Decision: Tune Test Watchdogs. The global watchdog
  timeout was adjusted to 12 seconds, specifically calculated as 1.5x the
  observed real execution time of the optimized test. This provides a
  data-driven safety margin for CI.

---

Part
V: The Last Bug A Platform-Specific Memory Leak (March 20) With all other tests passing, a single memory leak failure in
t/78-memleak.t persisted, but only on the Windows
ad2 environment. This required a different approach than
the timeout fixes.

Key Milestones:

• The Bug: A strong reference cycle involving the
  on_dialog event listener was not being broken on Windows,
  despite multiple attempts to fix it. Fixes that worked on Linux (such as
  calling on_dialog(undef) in DESTROY) were not
  sufficient on the Windows host.
• The Diagnosis: The issue was determined to be a
  deep, platform-specific interaction between Perl s garbage collector,
  the IO::Async event loop implementation on Windows, and the
  Test::Memory::Cycle module. The cycle report was identical
  on both platforms, but the cleanup behavior was different.
• Failed Attempts: A series of increasingly
  aggressive fixes were attempted to break the cycle, including:
  1. Moving the on_dialog(undef) call from
     close() to DESTROY().
  2. Explicitly deleteing the listener and callback
     properties from the object hash in DESTROY.
  3. Swapping between $self->remove_listener and
     $self->target->unlisten in a mistaken attempt to find
     the correct un-registration method.
• Pragmatic Solution: After exhausting all reasonable
  code-level fixes without a resolution on Windows, the user opted to mark
  the failing test as a known issue for that specific platform.
• Final Fix: The single failing test in
  t/78-memleak.t was wrapped in a conditional
  TODO block that only executes on Windows
  (if ($^O =~ /MSWin32/i)), formally acknowledging the bug
  without blocking the build. This allows the test suite to pass in CI
  environments while flagging the issue for future, deeper
  investigation.

---

Part VI: CI Hardening (March
20) A final failure in the GitHub Actions CI environment revealed one
last configuration flaw.

Key Milestones:

• The Bug: The CI was running
  prove --nocount --jobs 3 -I local/ -bl xt t directly. This
  command was missing the crucial -It/lib include path, which
  is necessary for test files to locate the t::helper module.
  This resulted in nearly all tests failing with
  Can't locate t/helper.pm in @INC.
• The Investigation: An analysis of
  Makefile.PL revealed a custom MY::test block
  specifically designed to inject the -It/lib flag into the
  make test command. This confirmed that
  make test is the correct, canonical way to run the test
  suite for this project.
• The Fix: The
  .github/workflows/linux.yml file was modified to replace
  the direct prove call with make test in the
  Run Tests step. This ensures the CI environment runs the
  tests in the exact same way as a local developer, with all necessary
  include paths correctly configured by the project s build system.

Final Outcome After this long and arduous journey, the
WWW::Mechanize::Chrome test suite is now stable and
passing on all targeted platforms, with known
platform-specific issues clearly documented in the code. The project is
in a vastly more robust and reliable state.

Tweet

Regular review of denials in the system log required Any system administrator using Debian/Ubuntu needs to know how to check for AppArmor denials. The point of using AppArmor is kind of moot if nobody is checking the denials. When AppArmor blocks an action, it logs the event to the system audit or kernel logs. Understanding these logs is crucial for troubleshooting custom configurations or identifying potential security incidents. To view recent denials, check /var/log/audit/audit.log or run journalctl -ke --grep=apparmor. A typical denial entry for MariaDB will look like this (split across multiple lines for legibility):

Copy

msg=audit(1700000000.123:456): apparmor="DENIED" operation="open" profile="/usr/sbin/mariadbd" name="/custom/data/path/test.ibd" pid=1234 comm="mariadbd" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

msg=audit(1700000000.123:456): apparmor="DENIED" operation="open"
profile="/usr/sbin/mariadbd" name="/custom/data/path/test.ibd" pid=1234
comm="mariadbd" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

How to interpret this output:

• msg=audit( ): The audit timestamp and event serial number.
• apparmor= DENIED : Indicates AppArmor blocked the action.
• operation: The action being attempted (e.g., open, mknod, file_mmap, file_perm).
• profile: The specific AppArmor profile that triggered the denial (in this case the /usr/sbin/mariadbd profile).
• name: The file path or resource that was blocked. In the example above, a custom data path was denied access because it wasn t defined in the profile s allowed abstractions.
• comm: The command name that triggered the denial (here mariadbd).
• requested_mask / denied_mask: Shows the permissions requested (e.g., r for read, w for write).
• pid: The process ID.
• fsuid: The user ID of the process attempting the action.
• ouid: The owner user ID of the target file.

If an action seems legit and should not be denied, the sysadmin needs to update the existing rules at /etc/apparmor.d/ or drop a local customization file in at /etc/apparmor.d/local/. If the denied action looks malicious, the sysadmin should start a security investigation and if needed report a suspected zero-day vulnerability to the upstream software vendor (e.g. Ubuntu customers to Canonical, or MariaDB customers to MariaDB).

AppArmor in MariaDB - not a novel thing, and not easy to implement well Based on old bug reports, there was an AppArmor profile already back in 2011, but it was removed in MariaDB 5.1.56 due to backlash from users running into various issues. A new profile was created in 2015, but kept opt-in only due to the risk of side effects. It likely had very few users and saw minimal maintenance, getting only a handful of updates in the past 10 years. The primary challenge in using mandatory access control systems with MariaDB lies in the sheer breadth of MariaDB s operational footprint with diverse storage engines and plugins. Also the code base in MariaDB assumes that system calls to Linux always work which they do under normal circumstances and do not handle errors well if AppArmor suddenly denies a system call. MariaDB is also a large and complex piece of software to run and operate, and it can be very challenging for system administrators to root-cause that a misbehavior in their system was due to AppArmor blocking a single syscall. Ironically, AppArmor is most beneficial exactly due to the same reasons for MariaDB. The larger and more complex a software is, the larger are the odds of a security vulnerability arising between the various components. And AppArmor profile helps reduce this complexity down to a single access list. Over the years there has been users requesting to get the AppArmor profile back, such as Debian Bug#875890 since 2017. The need was raised recently again by the Ubuntu security team during the MariaDB Ubuntu main inclusion review in 2025, which prompted a renewed effort by Debian/Ubuntu developers, mainly myself and Aquila Macedo, with upstream MariaDB assistance from Daniel Black.

A fresh approach: leverage the MariaDB test suite for automated testing and the open source community for reviews The key to creating a robust AppArmor profile is the ability to know in detail what is expected and normal behavior of the system. One could in theory read all of the source code in MariaDB, but with over two million lines, it is of course not feasible in practice. However, MariaDB does have a very extensive 7000+ test suite, and running it should trigger most code paths in MariaDB. Utilizing the test suite was key in creating the new AppArmor profile for MariaDB: we installed MariaDB on a Ubuntu system, enabled AppArmor in complain mode and iterated on the allowlist by running the full mariadb-test-run with all MariaDB plugins and features enabled until we had a comprehensive yet clean list of rules. To be extra diligent, we also reworked the autopkgtest for MariaDB in Debian and Ubuntu CI systems to run with the AppArmor profile enabled and to print all AppArmor notices at the end of the run, making it easy to detect now and in the future if the MariaDB test suite triggers any AppArmor denials. If any test fails, the release would not get promoted further, protecting users from regressions. While developing and triggering manual test runs we used the maximal achievable test suite with 7177 tests. The test is however so extensive it takes over two hours to run, and it also has some brittle tests, so the standard test run in Debian and Ubuntu autopkgtest is limited just to MariaDB s main suite with about 1000 tests. Having some tests fail while testing the AppArmor profile was not a problem, because we didn t need all the tests to pass we merely needed them to run as many code paths as possible to see if they run any system calls not accounted for in the AppArmor profile. Note that extending the profile was not just mechanical copying of log messages to the profile. For example, even though a couple of tests involve running the dash shell, we decided to not allow it, as it opens too much of a path for a potential exploit to access the operating system. The result of this effort is a modernized, robust profile that is now production-ready. Those interested in the exact technical details can read the Debian Bug#1130272 and the Merge Request discussions at salsa.debian.org, which hosts the Debian packaging source code.

Now available in Debian unstable, soon Ubuntu feedback welcome! Even though the file is just 200 lines long, the work to craft it spanned several weeks. To minimize risk we also did a gradual rollout by releasing the first new profile version in complain mode, so AppArmor only logs would-be-denials without blocking anything. The AppArmor profile was switched to enforce mode only in the very latest MariaDB revision 1:11.8.6-4 in Debian, and a NEWS item issued to help increase user awareness of this change. It is also slated for the upcoming Ubuntu 26.04 Resolute Raccoon release next month, providing out-of-the-box hardening for the wider ecosystem. While automated testing is extensive, it cannot simulate everything. Most notably various complicated replication topologies and all Galera setups are likely not covered. Thus, I am calling on the community to deploy this profile and monitor for any audit denials in the kernel logs. If you encounter unexpected behavior or legitimate denials, please submit a bug report via the Debian Bug Tracking System. To ensure you are running the latest MariaDB version, run apt install --update --yes mariadb-server. To view the latest profile rules, run cat /etc/apparmor.d/mariadbd and to see if it is enforced review the output of aa-status. To quickly check if there were any AppArmor denials, simply run journalctl -k grep -i apparmor grep -i mariadb.

Systemd hardening also adopted as security features keep evolving For those interested in MariaDB security hardening, note that also new systemd hardening options were rolled out in Debian/Ubuntu recently. Note that Debian and Ubuntu are mainly volunteer-driven open source developer communities, and if you find this topic interesting and you think you have the necessary skills, feel free to submit your improvement ideas as Merge Requests at salsa.debian.org/mariadb-team. If your improvement suggestions are not Debian/Ubuntu specific, please submit them directly to upstream at GitHub.com/MariaDB.

Changes in RcppArmadillo version 15.2.4-1 (2026-03-17)

• Upgraded to Armadillo release 15.2.4 (Medium Roast Deluxe)
  • Workarounds for bugs in GCC and Clang sanitisers (ASAN false positives)
  • Faster handling of blank sparse matrices
• Refined OpenMP setup (Dirk in #500)

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

Activity summary During the month of February, 20 contributors have been paid to work on Debian LTS (links to individual contributor reports are located below). The team released 35 DLAs fixing 527 CVEs. We also welcomed Arnaud Rebillout to the team and had to say farewell to Roberto, who left the team after more than nine years as part of it. The team continued preparing security updates in its usual rhythm. Beyond the updates targeting Debian 11 ( bullseye ), which is the current release under LTS, the team also proposed updates for more recent releases (Debian 12 ( bookworm ) and Debian 13 ( trixie )), including Debian unstable. Notable security updates:

• Guilhem Moulin prepared DLA 4492-1 for gnutls28 to fix vulnerabilities which may lead to Denial of Service.
• Utkarsh Gupta prepared DLA 4464-1 for xrdp, to fix a a vulnerability that could allow remote attackers to execute arbitrary code on the target system.
• Emilio Pozuelo Monfort prepared DLA-4465-1 to replace ClamAV 1.0 with ClamAV 1.4. This latter is the current LTS version supported by upstream.
• Markus Koschany prepared DLA 4468-1 for tomcat9, to fix a vulnerability that can be used to bypass security constraints.
• Santiago Ruano Rinc n prepared DLA 4471-1 to update package debian-security-support, the Debian security coverage checker.
• Bastien Roucari s prepared DLA 4473-1 for zabbix, to fix a potential remote code execution vulnerability.
• Paride Legovini prepared DLA 4478-1 for tcpflow, to fix a vulnerability that might result in DoS and potentially code execution.
• Thorsten Alteholz prepared DLA 4477-1 for munge, to fix a vulnerability which may allow local users to leak the MUNGE cryptographic key and forge arbitrary credentials.
• Ben Hutchings prepared DLA 4475-1 and DLA 4476-1 for Linux kernel updates.
• Chris Lamb prepared DLA 4482-1 for ceph, to fix SSL certificate checking in the Python bindings.
• Andreas Henriksson prepared DLA 4491-1 to fix vulnerabilities in glib2.0, which could result in denial of service, memory corruption or potentially arbitrary code execution.

Contributions from outside the LTS Team:

• The update of nova was prepared by the maintainer, Thomas Goirand. The corresponding DLA 4486-1 was published by Carlos Henrique Lima Melara.
• The updates of thunderbird were prepared by the maintainer Christoph Goehre. The corresponding DLA 4466-1 and DLA 4495-1 was published by Emilio Pozuelo Monfort.

The LTS Team has also contributed with updates to the latest Debian releases:

• Jochen prepared a point update of wireshark for bookworm (#1127945).
• Jochen prepared point updates of erlang for trixie (#1127606) and bookworm (#1127607).
• Bastien helped preparing DSA 6160-1 for netty and uploaded a fixed package to unstable.
• Bastien prepared a point update of zabbix for trixie (#1127437).
• Tobias prepared a point update of modsecurity-crs for bookworm (#1128655).
• Tobias prepared a point update of busybox for bookworm (#1129503).
• Tobias helped preparing DSA 6138-1 for libpng1.6.
• Daniel prepared point updates of python-authlib for trixie (#1129477) and bookworm (#1129246).
• Ben uploaded several Linux kernel packages to trixie-backports and bookworm-backports.
• Ben prepared point updates of wireless-regdb for trixie and bookworm.

Other than the work related to updates, Sylvain made several improvements to the documentation and tooling used by the team. Some milestones in the lifecycle of two Debian releases are just around the corner. The support of Debian 12 will be handed over to the LTS team on June 11th 2026. After August 31st, support for Debian 11 will move from Debian LTS to ELTS managed by Freexian.

Individual Debian LTS contributor reports

• Abhijith PA
• Andreas Henriksson
• Arnaud Rebillout
• Bastien Roucari s
• Ben Hutchings
• Carlos Henrique Lima Melara
• Chris Lamb
• Daniel Leidert
• Emilio Pozuelo Monfort
• Guilhem Moulin
• Jochen Sprickerhof
• Lee Garrett
• Lucas Kanashiro
• Markus Koschany
• Paride Legovini
• Santiago Ruano Rinc n
• Sylvain Beucler
• Thorsten Alteholz
• Tobias Frost
• Utkarsh Gupta

Thanks to our sponsors Sponsors that joined recently are in bold.

• Platinum sponsors:
  • Toshiba Corporation (for 125 months)
  • Civil Infrastructure Platform (CIP) (for 93 months)
  • VyOS Inc (for 57 months)
• Gold sponsors:
  • F. Hoffmann-La Roche AG (for 135 months)
  • CONET Deutschland GmbH (for 119 months)
  • Plat Home (for 118 months)
  • University of Oxford (for 75 months)
  • EDF SA (for 47 months)
  • Dataport A R (for 22 months)
  • CERN (for 20 months)
• Silver sponsors:
  • Domeneshop AS (for 140 months)
  • Nantes M tropole (for 134 months)
  • Akamai - Linode (for 129 months)
  • Univention GmbH (for 126 months)
  • Universit Jean Monnet de St Etienne (for 126 months)
  • Ribbon Communications, Inc. (for 120 months)
  • Exonet B.V. (for 110 months)
  • Leibniz Rechenzentrum (for 104 months)
  • Minist re de l Europe et des Affaires trang res (for 88 months)
  • Dinahosting SL (for 75 months)
  • Upsun Formerly Platform.sh (for 69 months)
  • Moxa Inc. (for 63 months)
  • Deveryware (for 62 months)
  • sipgate GmbH (for 61 months)
  • OVH US LLC (for 59 months)
  • Tilburg University (for 59 months)
  • GSI Helmholtzzentrum f r Schwerionenforschung GmbH (for 50 months)
  • THINline s.r.o. (for 23 months)
  • Copenhagen Airports A/S (for 17 months)
  • Conseil D partemental de l Is re (for 3 months)
• Bronze sponsors:
  • Evolix (for 140 months)
  • Seznam.cz, a.s. (for 140 months)
  • Intevation GmbH (for 137 months)
  • Linuxhotel GmbH (for 137 months)
  • Daevel SARL (for 136 months)
  • Megaspace Internet Services GmbH (for 135 months)
  • Greenbone AG (for 134 months)
  • NUMLOG (for 134 months)
  • WinGo AG (for 133 months)
  • Entr ouvert (for 125 months)
  • Adfinis AG (for 122 months)
  • Laboratoire LEGI - UMR 5519 / CNRS (for 117 months)
  • Tesorion (for 117 months)
  • Bearstech (for 108 months)
  • LiHAS (for 108 months)
  • Catalyst IT Ltd (for 103 months)
  • Demarcq SAS (for 97 months)
  • Universit Grenoble Alpes (for 83 months)
  • TouchWeb SAS (for 75 months)
  • SPiN AG (for 72 months)
  • CoreFiling (for 68 months)
  • Observatoire des Sciences de l Univers de Grenoble (for 59 months)
  • Tem Innovations GmbH (for 54 months)
  • WordFinder.pro (for 53 months)
  • CNRS DT INSU R sif (for 52 months)
  • Soliton Systems K.K. (for 47 months)
  • Alter Way (for 45 months)
  • Institut Camille Jordan (for 35 months)
  • SOBIS Software GmbH (for 20 months)
  • Tuxera Inc. (for 11 months)
  • OPM-OP AS (for 3 months)

# dig -t txt _spf.google.com grep spf1
_spf.google.com.	300	IN	TXT	"v=spf1 ip4:74.125.0.0/16 ip4:209.85.128.0/17 ip6:2001:4860:4864::/56 ip6:2404:6800:4864::/56 ip6:2607:f8b0:4000::/36 ip6:2800:3f0:4000::/36 ip6:2a00:1450:4000::/36 ip6:2c0f:fb50:4000::/36 ~all"
# dig -t txt outlook.com grep spf1
outlook.com.		126	IN	TXT	"v=spf1 include:spf2.outlook.com -all"
# dig -t txt _spf.mail.yahoo.com grep spf1
_spf.mail.yahoo.com.	1800	IN	TXT	"v=spf1 ptr:yahoo.com ptr:yahoo.net ip4:34.2.71.64/26 ip4:34.2.75.0/26 ip4:34.2.84.64/26 ip4:34.2.85.64/26 ip4:34.2.64.0/22 ip4:34.2.68.0/23 ip4:34.2.70.0/23 ip4:34.2.72.0/22 ip4:34.2.78.0/23 ip4:34.2.80.0/23 ip4:34.2.82.0/23 ip4:34.2.84.0/24 ip4:34.2.86.0" "/23 ip4:34.2.88.0/23 ip4:34.2.90.0/23 ip4:34.2.92.0/23 ip4:34.2.85.0/24 ip4:34.2.94.0/23 ?all"
# dig -t txt icloud.com grep spf1
icloud.com.		3586	IN	TXT	"v=spf1 ip4:17.41.0.0/16 ip4:17.58.0.0/16 ip4:17.142.0.0/15 ip4:17.57.155.0/24 ip4:17.57.156.0/24 ip4:144.178.36.0/24 ip4:144.178.38.0/24 ip4:112.19.199.64/29 ip4:112.19.242.64/29 ip4:222.73.195.64/29 ip4:157.255.1.64/29" " ip4:106.39.212.64/29 ip4:123.126.78.64/29 ip4:183.240.219.64/29 ip4:39.156.163.64/29 ip4:57.103.64.0/18" " ip6:2a01:b747:3000:200::/56 ip6:2a01:b747:3001:200::/56 ip6:2a01:b747:3002:200::/56 ip6:2a01:b747:3003:200::/56 ip6:2a01:b747:3004:200::/56 ip6:2a01:b747:3005:200::/56 ip6:2a01:b747:3006:200::/56 ~all"

• [1] https://en.wikipedia.org/wiki/Instant_messaging
• [2] https://en.wikipedia.org/wiki/Real-time_computing
• [3] https://en.wikipedia.org/wiki/Eudora_(email_client)
• [4] https://flounder.linux.org.au/
• [5] https://en.wikipedia.org/wiki/Lemmy_(social_network)

"The future of software is not just 'tools', but 'agents' that can navigate complex tasks on your behalf."

Andrej Karpathy

npm install -g @google/gemini-cli

# Role: Hyprland Configuration Specialist (Minimalist & High-Performance)
You are a Linux Systems Engineer specializing in migrating users from heavy
Desktop Environments to minimalist, tiling-based Wayland sessions on Debian.
Your goal is to maximize available RAM for heavy applications while maintaining
essential desktop features.
## 1. Environment & Persona
- **Target OS:** Debian (Linux)
- **Target WM:** Hyprland
- **Hardware:** ThinkPad E470 (i5-7th Gen, 12GB RAM)
- **User Profile:** Emacs user, prioritizes "anti-gravity" (zero bloat).
- **Tone:** Technical, concise, and security-conscious.
## 2. Core Functional Requirements
- **Status Bar:**  waybar  (with CPU, RAM, Network, and Battery icons).
- **Wallpaper:**  swww  or  hyprpaper .
- **Screen Lock:**  hyprlock  +  hypridle .
- **Input Mapping:** Swap Control and Caps Lock ( kb_options = ctrl:nocaps ).
## 3. Operational Constraints
- **Permission First:** Ask before using  sudo  or writing outside the work directory.
- **Inspection:** Use  hyprctl ,  lsmod , or  gsettings  for compatibility checks.
- **Artifact Management:** Update  MEMORY.md  after every major step.

Welcome to the February 2026 report from the Reproducible Builds project! These reports outline what we ve been up to over the past month, highlighting items of news from elsewhere in the increasingly-important area of software supply-chain security. As ever, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website.

1. reproduce.debian.net
2. Tool development
3. Distribution work
4. Miscellaneous news
5. Upstream patches
6. Documentation updates
7. Four new academic papers

reproduce.debian.net The last year has seen the introduction, development and deployment of reproduce.debian.net. In technical terms, this is an instance of rebuilderd, our server designed monitor the official package repositories of Linux distributions and attempt to reproduce the observed results there. This month, however, Holger Levsen added suite-based navigation (eg. Debian trixie vs forky) to the service (in addition to the already existing architecture based navigation) which can be observed on, for instance, the Debian trixie-backports or trixie-security pages.

Tool development diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes, including preparing and uploading versions, 312 and 313 to Debian. In particular, Chris updated the post-release deployment pipeline to ensure that the pipeline does not fail if the automatic deployment to PyPI fails [ ]. In addition, Vagrant Cascadian updated an external reference for the 7z tool for GNU Guix. [ ]. Vagrant Cascadian also updated diffoscope in GNU Guix to version 312 and 313.

Distribution work In Debian this month:

• 26 reviews of Debian packages were added, 5 were updated and 19 were removed this month adding to our extensive knowledge about identified issues.
• A new debsbom package was uploaded to unstable. According to the package description, this package generates SBOMs (Software Bill of Materials) for distributions based on Debian in the two standard formats, SPDX and CycloneDX. The generated SBOM includes all installed binary packages and also contains Debian Source packages.
• In addition, a sbom-toolkit package was uploaded, which provides a collection of scripts for generating SBOM. This is the tooling used in Apertis to generate the Licenses SBOM and the Build Dependency SBOM. It also includes dh-setup-copyright, a Debhelper addon to generate SBOMs from DWARF debug information, which are extracted from DWARF debug information by running dwarf2sources on every ELF binaries in the package and saving the output.

Lastly, Bernhard M. Wiedemann posted another openSUSE monthly update for their work there.

Miscellaneous news

• S ren Tempel (nmeum) wrote up their insightful notes on Debugging Reproducibility Issues in Rust Software after nondeterministic issues were found and investigated for pimsync in the GNU Guix review process
• Jeremy Bicha reported a bug in GNOME Clocks after they noticed that version 50.beta regressed in reproducibility compared to 49.0. Specifically, the new generated .oga files differ in their Serial No. and Checksum [fields] . However, Jeremy ended up fixing the issue by replacing ffmpeg with oggenc.
• kpcyrd shared some information from the archlinux-dev-public mailing list on our mailing list this month after a discussion at our latest Summit meeting on the topic of Link-Time Optimisation (LTO) specifically on the reasons why LTO often needs to be disabled in relation to Arch Linux s approach to binary hardening.
• Janneke Nieuwenhuizen posed a question to our list about whether there might be situations where using the UNIX epoch itself (i.e. 0) may materially differ from using SOURCE_DATE_EPOCH) when a situation demands the use of a fixed timestamp.
• Laurent Huberdeau announced that they had recently finished their masters thesis arguing for the use of POSIX shell for diverse double-compilation and reproducible builds . Laurent also presents pnut, a C compiler capable of bootstrapping itself and TCC from any POSIX-compliant shell and human-readable source files.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:
  • aiohttp
  • lima
  • meson
  • obs-studio
  • paracon
  • pyshark
  • python-flake8-comprehensions
  • server
  • vlang
• Gioele Barabucci:
  • #1127641 filed against bitsnpicas.
  • #1127643 filed against fonts-topaz-unicode.
  • #1128901 filed against bitsnpicas.

Documentation updates Once again, there were a number of improvements made to our website this month including:

• Aman Sharma added a Java reproducible builds paper to the Academic publications page. [ ]
• Chris Lamb added a reference to the repro-build to the Tools page. [ ]
• Michiel Hendriks corrected an issue on the JVM page in relation to .properties files. [ ]
• kpcyrd added Homebrew to the Who is involved page. [ ][ ]

Four new academic papers Julien Malka and Arnout Engelen published a paper titled Lila: Decentralized Build Reproducibility Monitoring for the Functional Package Management Model:

[While] recent studies have shown that high reproducibility rates are achievable at scale demonstrated by the Nix ecosystem achieving over 90% reproducibility on more than 80,000 packages the problem of effective reproducibility monitoring remains largely unsolved. In this work, we address the reproducibility monitoring challenge by introducing Lila, a decentralized system for reproducibility assessment tailored to the functional package management model. Lila enables distributed reporting of build results and aggregation into a reproducibility database [ ].

A PDF of their paper is available online.
Javier Ron and Martin Monperrus of KTH Royal Institute of Technology, Sweden, also published a paper, titled Verifiable Provenance of Software Artifacts with Zero-Knowledge Compilation:

Verifying that a compiled binary originates from its claimed source code is a fundamental security requirement, called source code provenance. Achieving verifiable source code provenance in practice remains challenging. The most popular technique, called reproducible builds, requires difficult matching and reexecution of build toolchains and environments. We propose a novel approach to verifiable provenance based on compiling software with zero-knowledge virtual machines (zkVMs). By executing a compiler within a zkVM, our system produces both the compiled output and a cryptographic proof attesting that the compilation was performed on the claimed source code with the claimed compiler. [ ]

A PDF of the paper is available online.
Oreofe Solarin of Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, Ohio, USA, published It s Not Just Timestamps: A Study on Docker Reproducibility:

Reproducible container builds promise a simple integrity check for software supply chains: rebuild an image from its Dockerfile and compare hashes. We built a Docker measurement pipeline and apply it to a stratified sample of 2,000 GitHub repositories that contained a Dockerfile. We found that only 56% produce any buildable image, and just 2.7% of those are bitwise reproducible without any infrastructure configurations. After modifying infrastructure configurations, we raise bitwise reproducibility by 18.6%, but 78.7% of buildable Dockerfiles remain non-reproducible.

A PDF of Oreofe s paper is available online.
Lastly, Jens Dietrich and Behnaz Hassanshahi published On the Variability of Source Code in Maven Package Rebuilds:

[In] this paper we test the assumption that the same source code is being used [by] alternative builds. To study this, we compare the sources released with packages on Maven Central, with the sources associated with independently built packages from Google s Assured Open Source and Oracle s Build-from-Source projects. [ ]

A PDF of their paper is available online.

Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org

$ dig +short IN HTTPS cloudflare-ech.com
1 . alpn="h3,h2" ipv4hint=104.18.10.118,104.18.11.118 ech=AEX+DQBBFQAgACDBFqmr34YRf/8Ymf+N5ZJCtNkLm3qnjylCCLZc8rUZcwAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA= ipv6hint=2606:4700::6812:a76,2606:4700::6812:b76

Debian Contributions: 2026-02 Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

DebConf 26 Registration, by Stefano Rivera, Antonio Terceiro, and Santiago Ruano Rinc n DebConf 26, to be held in Santa Fe Argentina in July, has opened for registration and event proposals. Stefano, Antonio, and Santiago all contributed to making this happen. As always, some changes needed to be made to the registration system. Bigger changes were planned, but we ran out of time to implement them for DebConf 26. All 3 of us have had experience in hosting local DebConf events in the past and have been advising the DebConf 26 local team.

Debian CI improvements, by Antonio Terceiro Debian CI is the platform responsible for automated testing of packages from the Debian archive, and its results are used by the Debian Release team automation as Quality Assurance to control the migration of packages from Debian unstable into testing, the base for the next Debian release. Antonio started developing an incus backend, and that prompted two rounds of improvements to the platform, including but not limited to allowing user to select a job execution backend (lxc, qemu) during the job submission, reducing the part of testbed image creation that requires superuser privileges and other refactorings and bug fixes. The platform API was also improved to reduce disruption when reporting results to the Release Team automation after service downtimes. Last, but not least, the platform now has support for testing packages against variants of autopkgtest, which will allow the Debian CI team to test new versions of autopkgtest before making releases to avoid widespread regressions.

Miscellaneous contributions

• Carles improved po-debconf-manager while users requested features / found bugs. Improvements done - add packages from unstable instead of just salsa.debian.org, upgrade and merge templates of upgraded packages, finished adding typing annotations, improved deleting packages: support multiple line texts, add debug to see subprocess.run commands, etc.
• Carles, using po-debconf-manager, reviewed 7 Catalan translations and sent bug reports or MRs for 11 packages. Also reviewed the translations of fortunes-debian-hints and submitted possible changes in the hints.
• Carles submitted MRs for reportbug (reportbug --ui gtk detecting the wrong dependencies), devscript (delete unused code from debrebuild and add recommended dependency), wcurl (format help for 80 columns). Carles submitted a bug report for apt not showing the long descriptions of packages.
• Carles resumed effort for checking relations (e.g. Recommends / Suggests) between Debian packages. A new codebase (still in early stages) was started with a new approach in order to detect, report and track the broken relations.
• Emilio drove several transitions, most notably the haskell transition and the glibc/gcc-15/zlib transition for the s390 31-bit removal. This last one included reviewing and requeueing lots of autopkgtests due to britney losing a lot of results.
• Emilio reviewed and uploaded poppler updates to experimental for a new transition.
• Emilio reviewed, merged and deployed some performance improvements proposed for the security-tracker.
• Stefano prepared routine updates for pycparser, python-confuse, python-cffi, python-mitogen, python-pip, wheel, platformdirs, python-authlib, and python-virtualenv.
• Stefano updated Python 3.13 and 3.14 to the latest point releases, including security updates, and did some preliminary work for Python 3.15.
• Stefano reviewed changes to dh-python and merged MRs.
• Stefano did some debian.social sysadmin work, bridging additional IRC channels to Matrix.
• Stefano and Antonio, as DebConf Committee Members, reviewed the DebConf 27 bids and took part in selecting the Japanese bid to host DebConf 27.
• Helmut sent patches for 29 cross build failures.
• Helmut continued to maintain rebootstrap addressing issues relating to specific architectures (such as musl-linux-any, hurd-any or s390x) or specific packages (such as binutils, brotli or fontconfig).
• Helmut worked on diagnosing bugs such as rocblas #1126608, python-memray #1126944 upstream and greetd #1129070 with varying success.
• Antonio provided support for multiple MiniDebConfs whose websites run wafer + wafer-debconf (the same stack as DebConf itself).
• Antonio fixed the salsa tagpending webhook.
• Antonio sent specinfra upstream a patch to fix detection of Debian systems in some situations.
• Santiago reviewed some Merge Requests for the Salsa CI pipeline, including !703 and !704, that aim to improve how the build source job is handled by Salsa CI. Thanks a lot to Jochen for his work on this.
• In collaboration with Emmanuel Arias, Santiago proposed a couple of projects for the Google Summer of Code (GSoC) 2026 round. Santiago has been reviewing applications and giving feedback to candidates.
• Thorsten uploaded new upstream versions of ipp-usb, brlaser and gutenprint.
• Rapha l updated publican to fix an old bug that became release critical and that happened only when building with the nocheck profile. Publican is a build dependency of the Debian s Administrator Handbook and with that fix, the package is back into testing.
• Rapha l implemented a small feature in Debusine that makes it possible to refer to a collection in a parent workspace even if a collection with the same name is present in the current workspace.
• Lucas updated the current status of ruby packages affecting the Ruby 3.4 transition after a bunch of updates made by team members. He will follow up on this next month.
• Lucas joined the Debian orga team for GSoC this year and tried to reach out to potential mentors.
• Lucas did some content work for MiniDebConf Campinas - Brazil.
• Colin published minor security updates to bookworm and trixie for CVE-2025-61984 and CVE-2025-61985 in OpenSSH, both of which allowed code execution via ProxyCommand in some cases. The trixie update also included a fix for mishandling of PerSourceMaxStartups.
• Colin spotted and fixed a typo in the bug tracking system s spam-handling rules, which in combination with a devscripts regression caused bts forwarded commands to be discarded.
• Colin ported 12 more Python packages away from using the deprecated (and now removed upstream) pkg_resources module.
• Anupa is co-organizing MiniDebConf Kanpur with Debian India team. Anupa was responsible for preparing the schedule, publishing it on the website, co-ordination with the fiscal host in addition to attending meetings.
• Anupa attended the Debian Publicity team online sprint which was a skill sharing session.

• openssh: Please remove/replace usage of dh_movetousr

• aiosmtplib
• bitstruct
• diff-cover
• django-q
• isort
• multipart
• poetry (adding support for Dulwich >= 0.25)
• poetry-core
• pydantic-settings
• python-build
• python-certifi
• python-datamodel-code-generator
• python-flatdict
• python-holidays
• python-maggma
• python-pytokens
• python-scruffy
• python-urllib3 (fixing CVE-2025-66471 and a chunked decoding bug)
• responses
• yarsync
• zope.component
• zope.deferredimport

• genshi (contributed upstream)
• germinate
• mopidy
• nose2
• pokrok (contributed upstream)
• pylama
• python-flask-seeder
• python-maggma (contributed upstream)
• python-pybadges
• python-scruffy (contributed upstream)
• thumbor (contributed upstream)
• zope.deprecation (contributed upstream a while ago, but there hasn t been an upstream release yet)

• flask-dance: FTBFS: No module named pkg_resources (actually fixed by adding a missing dependency to python3-sphinxcontrib.seqdiag)
• paramiko: autopkgtest regression on i386 (contributed upstream)
• poetry: autopkgtest regression on i386
• python-argh
• python-django-celery-beat: FTBFS: FAILED t/unit/test_models.py::HumanReadableTestCase::test_long_name
• python-maturin: rust-itertools update
• python-msrest: FTBFS: FAILED tests/asynctests/test_async_client.py::TestServiceClient::test_client_send (contributed upstream, though not very successfully)
• python-typing-inspect

• opentelemetry-proto: Stop building python3-opentelemetry-proto, taken over by the python-opentelemetry source package
• python-datamodel-code-generator: Depends: python3-isort (< 8) but 8.0.0-1 is to be installed (contributed upstream)
• python-typeguard: Mark python3-typeguard Multi-Arch: foreign
• wheel: Mark python3-wheel Multi-Arch: foreign
• zope.deferredimport: Please make the build reproducible (contributed upstream, with a follow-up fix)

• librust-pyo3-ffi-dev: Cannot be installed for foreign architectures

• arch-test: Remove build dependency on binutils-mips64el-linux-gnuabi64 (NMU)

• debconf: Add BMP version of debian-logo (merged and uploaded)
• openssh: Reorder pam_selinux(7) usage (merged and uploaded)
• openssh-client: use sysusers.d, drop superflous dependencies (merged and uploaded)
• openssh: Stop deleting system user on remove/purge (merged and uploaded)
• openssh: Do not link against libcrypt on GNU/Hurd (merged and uploaded)
• partman-prep: Align PReP descriptions with other partition types (merged)
• python-better-exceptions (sponsored upload for Seyed Mohamad Amin Modaresi)