Search Results: "js"

13 November 2026

Reproducible Builds: Reproducible Builds in October 2026

Welcome to the October 2026 report from the Reproducible Builds project! In our reports, we try to outline the most important things that we have been up to over the past month. As a quick recap about what problem our project intends to solve, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third parties to come to a consensus on whether a build was compromised or not. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website. In this month s report, we cover:

  1. GNU tar sparse archive non-determinism reported
  2. Reproducible Builds at SeaGL 2026
  3. Tool and documentation development
  4. Distribution work
  5. SOURCE_DATE_EPOCH specification updates
  6. Mailing list news
  7. Patches

GNU tar sparse archive non-determinism reported Johannes Schauer Marin Rodrigues (josch) reported an issue to the GNU tar mailing list where creating PAX-format sparse tar archives produces unreproducible outputs. Specifically, when GNU tar generates a PAX archive containing a sparse file it embeds the process ID into the member filename path structure (e.g. ./GNUSparseFile.<pid>/<filename>). Because process IDs vary between build runs, however, archiving identical sparse input files yields different binary outputs. In his report, Johannes outlined potential workarounds, such as running tar inside an unshared PID namespace where the process ID is pinned to 1, and proposed that GNU tar either omit the PID by default or provide a command-line flag to suppress it. Full details can be found in the post itself, and Johannes also filed the bug in Debian for completeness.

Reproducible Builds at SeaGL 2026 Reproducible Builds was present at SeaGL in Seattle, WA, on October 23rd & 24th. Thanks to everyone who attended and spoke with us! Members of the Reproducible Builds project attended SeaGL (the Seattle GNU/Linux Conference) in Seattle, WA, on October 23rd & 24th. It was a fantastic opportunity to connect with local community members, discuss software supply chain security and share updates on build verification tooling. Thanks to everyone who stopped by our sessions or spoke with us during the event. Founded in 2013, SeaGL is an annual free-as-in-freedom, grassroots technical summit dedicated to spreading awareness, community engagement and practical knowledge about free software, open hardware and libre culture.

Tool and documentation development diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes, including preparing and uploading versions 333, 334, 335 and 336 to Debian:
  • New features:
    • Support the dumppdf binary being called dumppdf.py on the host/runtime system. [ ]
  • Bug fixes:
    • Drop Recommends: guix from the diffoscope-minimal package as it is not minimal. (#1114739) [ ]
    • Fix a crash when using the latest version(s) of Androguard. Thanks, hc! [ ]
  • Codebase improvements:
    • Re-order various parts of the .tar file comparator to make it easier to progressively delve into its functionality. [ ]
    • Fix a silly prepend typo. [ ]
    • Reflect in the Python packaging that the use of the case-match language feature necessarily requires Python 3.10+. [ ]
  • Misc:
    • Update debian/tests/control. [ ]
    • Update copyright years. [ ]
In addition, Johannes Schauer Marin Rodrigues added support for POSIX.1-2001 PAX headers in the .tar file comparator [ ] and Vagrant Cascadian proposed an update in diffoscope in GNU Guix to version 335.

Distribution work In Debian this month, 20 reviews of Debian packages were added, 28 were updated and 27 were removed this month adding to our knowledge about identified issues. One issue type was updated to reflect that Debhelper compat level 14 is no longer experimental. [ ] In addition, Vagrant Cascadian performed a number of Non-Maintainer Uploads (NMUs) and Quality Assurance (QA) uploads in Debian for several packages with outstanding reproducibility issues. These included clhep, tea-cli, argparse-manpage, ruby-timers, mandos and cdebootstrap.
Lastly, Bernhard M. Wiedemann posted another openSUSE monthly update for their reproducibility work there.

SOURCE_DATE_EPOCH specification updates As announced in last month s report, further work has been done by Bernhard M. Wiedemann regarding updating the SOURCE_DATE_EPOCH specification following discussions at a project summit. Seven distinct proposed changes were originally submitted as merge requests on Salsa, and many of these have been approved or otherwise collapsed. Bernhard s proposed modifications were independent of one another, allowing for any combination to be merged depending on community feedback. Bernhard noted that one proposal may see the most debate, and could still benefit from further refinement before reaching consensus. Community members are invited to review and provide feedback on the individual merge requests; as Bernhard writes, The other MRs could still use feedback from more people before December . Input can be submitted via reactions or comments on Salsa, directly by email to Bernhard or the mailing list. The SOURCE_DATE_EPOCH specification was first published in 2015 and updated in 2017.

Mailing list news On our mailing list this month:

Patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where applicable or possible. This month, we wrote a large number of such patches, including: In addition, two build failure bugs were reported by Jochen Sprickerhof.

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

11 November 2026

Colin Watson: Free software activity in October 2026

My Debian contributions this month were all sponsored by Freexian. You can also support my work directly via Liberapay or GitHub Sponsors. OpenSSH I upgraded both openssh and openssh-gssapi from 10.5p1 to 10.6p1, started generating MLDSA44-ED25519 host keys by default now that upstream no longer considers them experimental, and fixed a problem where switching between the normal and GSS-API variants followed by purging the old packages caused some configuration files to disappear. Python packaging New upstream versions: Other build/test failures: Since lots of Python modules use Rust extensions nowadays, we often have to help with upgrades of Rust packages. I did some of this: Other bugs: One of my hobbies is typesetting ( engraving ) music, sometimes for my own use, sometimes for teaching, and sometimes for my choir. I ve used LilyPond for a long time and have got pretty used to it. My workflow was a bit cumbersome, though, usually involving side-by-side editor and PDF viewer windows with some ad-hoc Makefiles and editor macros. I recently discovered the Frescobaldi frontend to LilyPond, and decided I liked it enough to switch over to it. It was already packaged in Debian, but it had a missing desktop entry and icon and missing translations; I fixed those based on a patch set by Joachim Wiedorn. My upload introduced a testing migration problem, which I also fixed. Rust packaging I upgraded rust-jiter and rust-smallvec to new upstream versions for a new version of pydantic-core. The rust-smallvec upgrade caused an FTBFS in rust-saphyr-parser-bw, which I fixed. I packaged rust-cargo-cyclonedx for a new version of python-maturin. This needed some porting to work with a new version of rust-cargo-metadata, which I contributed upstream. I improved the dev/list-rdeps.sh tool used by the Rust team to check build-depends too, which should make it easier for that team to remember Python packages in transitions. Code reviews

6 November 2026

Vincent Bernat: Scriptable GeoDNS: leaving AWS for Bunny

To optimize the performance metrics of this blog, I relied on Amazon CloudFront to serve the assets,1 and Route 53 for the geo-aware DNS2 to direct readers to the closest host serving the content. But Route 53 is expensive: US$0.50 per zone per month, US$0.70 per million geolocation queries, and US$0.75 per healthcheck per month, plus US$1 per month for the KMS key needed to sign zones with DNSSEC. When I discovered that Bunny DNS is almost free for up to 500 domains and features scriptable answers, I made the jump! Plus, it's a European company based in Slovenia.

Bunny CDN Bunny.net started as BunnyCDN in 2015. They added IPv6 support in 2017. Switching was uneventful. I don't bring a custom domain: I serve the content from the b-cdn.net domain. As I depend on content negotiation to serve AVIF and WebP images, I configure the CDN to cache each image format separately. I disable logging to avoid collecting personal information. While the bunny.net GDPR page promises they anonymize data, it is unclear what CloudFront keeps in its own logs when logging is disabled. The only downside of the bunny.net CDN offering is the lack of HTTP/3. I asked their support: two minutes later, they answered they are working on it but have no ETA to share.

Bunny DNS Bunny.net introduced their DNS offering in 2022 and recently reduced its cost to almost zero. They support geographical routing, but a very cool feature is the scriptable DNS: it runs a piece of user-supplied JavaScript on each query to compute the answer. I run seven small NixOS virtual machines to host this blog around the world: Paris, Nuremberg, Ashburn, Singapore, San Jose, S o Paulo, and Sydney. Bunny's scriptable DNS lets me return the IP addresses of the two nearest servers. The browser falls back to the second server if the first one is down. 
const servers = [
   
    ipv4: "45.90.160.60",
    ipv6: "2a0c:8881:5:100::1",
    latitude: 48.88,
    longitude: 2.22,
   ,
  /* [ ] */
   
    ipv4: "139.180.162.114",
    ipv6: "2001:19f0:5800:1855:5400:6ff:fef0:4c0a",
    latitude: -33.87,
    longitude: 151.21,
   ,
];
const families =   A: "ipv4", AAAA: "ipv6"  ;
const records =   A: ARecord, AAAA: AaaaRecord  ;
export default function handleQuery(query)  
  const type = String(query.request.queryType).toUpperCase();
  const family = families[type];
  if (!family) return [];
  // Get user location
  const location = query.request.geoLocation;
  // Build a list of candidates
  let candidates = servers.map(
    (s) => new Server(s[family], s.latitude, s.longitude, 100),
  );
  // Select the two closest alive servers
  const selected = [];
  while (selected.length < 2)  
    const server = RoutingEngine.getClosestServer(candidates, location, true);
    selected.push(server);
    candidates = candidates.filter((s) => s.ip !== server.ip);
   
  // Return their IP addresses
  return selected.map((s) => new records[type](s.ip, 3600));
The above script relies on the RoutingEngine.getClosestServer() helper, which returns the closest alive3 server to the user.4 Thanks to Doggo, we can check it works as expected, at least for A:5 
$ doggo --gp-from="Amsterdam" A vincent.bernat.ch
LOCATION          NAME                TYPE  TTL    ADDRESS
Amsterdam, NL
                  vincent.bernat.ch.  A     3600s  45.90.160.60
                  vincent.bernat.ch.  A     3600s  116.203.18.48
$ doggo --gp-from="Texas" A vincent.bernat.ch
LOCATION          NAME                TYPE  TTL    ADDRESS
Houston (TX), US
                  vincent.bernat.ch.  A     3600s  5.161.44.145
                  vincent.bernat.ch.  A     3600s  149.28.219.155
The most annoying drawback of this method is that a user from Buenos Aires gets both S o Paulo (1000 mi, 30 ms RTT) and Ashburn (5200 mi, 140 ms RTT). Most resolvers randomize the order, so the user may pull the web page from Ashburn. We could return only the nearest server and swap the scripted record with the built-in geographic routing instead. But if the server becomes unavailable, the browser has no alternative until the DNS cache expires. It leaves a short window of unavailability, even with a short TTL.

The HTTPS record solution The HTTPS DNS record, defined in RFC 9460, solves both the availability and the latency issues. Modern browsers query this record to get additional information to reach an HTTP server, including an alternative port and the application layer protocol HTTP/2 or HTTP/3. Each record has a priority. We can ask browsers to contact the nearest server first and fall back to the next one if the first is unavailable: 
vincent    HTTPS  1 first.geo  alpn="h3,h2"
vincent    HTTPS  2 second.geo alpn="h3,h2"
vincent    HTTPS  3 .          alpn="h3,h2"
; Bunny DNS must provide the two closest servers as "first.geo"
; and "second.geo" with geographic or latency routing.
first.geo  A      45.90.160.60
first.geo  AAAA   2a0c:8881:5:100::1
second.geo A      116.203.18.48
second.geo AAAA   2a01:4f8:1c0c:5eb5::1
vincent    A      45.90.160.60
vincent    A      116.203.18.48
vincent    AAAA   2a0c:8881:5:100::1
vincent    AAAA   2a01:4f8:1c0c:5eb5::1
This looks like the ideal solution! Unfortunately, Chrome ignores any record whose target is not ., hence the last record at priority 3. Moreover, unlike with a CNAME, Bunny DNS does not return the IP addresses for first.geo and second.geo in the additional section. The resolver needs to send a first request to get the HTTPS record, and a second to resolve first.geo. HTTPS records are still too new for this use case. Back to the drawing board!

The hybrid solution As a compromise, the script returns the two nearest servers if they are at a similar distance from the user. Otherwise, it returns only the nearest one with a lower TTL of 2 minutes. 
export default function handleQuery(query)  
  const location = query.request.geoLocation;
  let candidates = servers.map(
    (s) => new Server(s.ipv4, s.latitude, s.longitude, 100),
  );
  // Select the two closest servers
  const selected = [];
  while (selected.length < 2)  
    const server = RoutingEngine.getClosestServer(candidates, location, true);
    selected.push(server);
    candidates = candidates.filter((s) => s.ip !== server.ip);
   
  // Compute the distance to the user for each of them
  const [first, second] = selected.map((s) =>
    GeoDistance.calculate(s, location),
  );
  // If the second one is at twice the distance and at more than 1000 miles
  // from the first, return only the first one with a TTL of 2 minutes.
  if (second > 2 * first && second - first > 1000)  
    return [new ARecord(selected[0].ip, 120)];
   
  // Otherwise, return both with a TTL of 1 hour.
  return selected.map((s) => new ARecord(s.ip, 3600));
We can check it with Doggo. From Chicago, we get only the Ashburn server with a TTL of 2 minutes, while from Mexico, we get both Ashburn and San Jose. 
$ doggo --gp-from="Chicago" A vincent.bernat.ch
LOCATION          NAME                TYPE  TTL    ADDRESS
Chicago (IL), US
                  vincent.bernat.ch.  A     120s   5.161.44.145
$ doggo --gp-from="Mexico" A vincent.bernat.ch
LOCATION          NAME                TYPE  TTL    ADDRESS
Queretaro, MX
                  vincent.bernat.ch.  A     3600s  5.161.44.145
                  vincent.bernat.ch.  A     3600s  149.28.219.155
$ doggo --reverse 5.161.44.145 149.28.219.155
NAME                          TYPE  TTL    ADDRESS
145.44.161.5.in-addr.arpa.    PTR   7187s  web05.luffy.cx.
155.219.28.149.in-addr.arpa.  PTR   3600s  web07.luffy.cx.

Note On the live page, the map below is interactive: point at a location to get the DNS answer, or disable servers to see how the answers change.

Map of the servers answered by Bunny DNS for each location
The complete code is a bit more complex. It works around RoutingEngine.getClosestServer() supporting only IPv4, and it handles the case where there are not enough alive servers to build an answer. Bunny.net has a comprehensive Terraform provider. I leverage CDK Terrain to build the target state in TypeScript and OpenTofu to deploy it. Have a look at "Managing infrastructure with Terraform, CDKTF, and NixOS" for more details.6

ACME DNS with Hickory DNS Route 53 covered one additional use case of mine: serving acme.luffy.cx to solve DNS-01 ACME challenges. Unlike Route 53, Bunny DNS cannot restrict access to a single zone with a fine-grained access policy. As this usage requires neither high availability nor low latency, I replaced Route 53 with Hickory, a memory-safe DNS server written in Rust. It supports automatic DNSSEC signatures and dynamic updates. Its configuration is simple: 
[[zones]]
zone = "acme.luffy.cx"
zone_type = "Primary"
[zones.stores]
allow_update = true
type = "sqlite"
journal_path = "acme.luffy.cx.sqlite"
zone_path = "acme.luffy.cx.zone"
# Private key to sign the zone with DNSSEC. Generated with:
#  openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256
[[zones.keys]]
algorithm = "ECDSAP256SHA256"
key_path = "/etc/hickory-dns.acme.luffy.cx.pem"
# TSIG key to sign zone updates. Generated with:
#  openssl rand 32
[[zones.stores.tsig_keys]]
algorithm = "hmac-sha256"
key_file = "/etc/hickory-dns.acme.luffy.cx.update.acme.luffy.cx.tsig"
name = "update.acme.luffy.cx."
The provided zone file only seeds the database on first start. I delegate DNS-01 challenges to this zone with CNAME records: 
_acme-challenge          CNAME bernat.ch.acme.luffy.cx.
_acme-challenge.vincent  CNAME vincent.bernat.ch.acme.luffy.cx.
Lego, the ACME client I use to get certificates from Let's Encrypt, answers the DNS-01 challenge by pushing entries with the DNS UPDATE protocol. Unfortunately, Bunny DNS cannot publish a DS record in a zone.7 I need one to secure the acme.luffy.cx zone and certificate delivery. Therefore, I host the luffy.cx zone on deSEC,8 a free DNS hosting service run by a non-profit organization based in Berlin. I derive the DS record to publish in luffy.cx from the DNSKEY record of acme.luffy.cx: 
$ ZONE=acme.luffy.cx
$ dig @nsupdate.luffy.cx $ZONE DNSKEY +noall +answer \
>     dnssec-dsfromkey -a SHA-256 -f - $ZONE
acme.luffy.cx. IN DS 55628 13 2 4946E7508F20E15F1B577A231104B029A1A82E04699A80CD46ADCE31227EE81F
Overall, I am quite happy with the switch to bunny.net. There are still a few rough edges, like the inability to publish DS records, the lack of HTTP/3, and the IPv4-only RoutingEngine.getClosestServer() helper. The bill went from US$5 per month to US$1. So far, performance is on par with AWS, but I only finalized the current setup, including the server in Sydney, in late October. I need a bit more time to gather more data, and I'll post a short update.

Update (2026-11) This post was briefly discussed on Hacker News and Lobsters.

  1. I had been hosting media assets on CloudFront since 2011.
  2. Route 53 had served bernat.ch since December 2021. I didn't use healthchecks as they were too expensive. I didn't host most of my zones there either, as queries for haproxy.debian.net alone would have cost me around US$10 per month.
  3. A server is alive if it answers an ICMP ping. The bunny.net documentation gives more details about the checks.
  4. The user queries Bunny DNS through a resolver. Usually, either the resolver is close to the user, or it exposes the user's subnet through EDNS client subnet.
  5. Don't use this script as is: RoutingEngine.getClosestServer() only works with IPv4. The final script works around this limitation.
  6. The article is a bit outdated. CDK Terrain replaces CDKTF, OpenTofu replaces Terraform, and I now use Colmena instead of NixOps.
  7. I contacted bunny.net support to ask when they will support publishing a DS record and got an answer seven minutes later: they don't know. This is a bummer, as they support automatic CDS, a DNS record to advertise a DS record to the parent zone. They handle one side of the delegation, not the other. I also reported a bug with the proof of non-existence, and they replied a day later with a detailed answer.
  8. Why not use deSEC for ACME challenges? You can limit its tokens to a single RRset. I could say I prefer to self-host, but the truth is I only discovered this while writing this article.

3 November 2026

John Goerzen: Must we always use LLMs this way?

As with my last post on AI, let me open with a statement:
Just because most people hold AI a certain way, doesn t mean that we always must, either now or in the future.
As I noted in that piece, and my shades of gray post, I am well aware and agree with the notion that a lot of frontier AI models have disturbing ethical concerns that we shouldn t dismiss, including concentration of power and wealth, environmental concerns, unethical scraping and training, the financial bubble they seem to be creating, and more. The most poisonous of AI cheerleaders are accelerating enshittification while cutting quality all over the place. AI slop is also a real problem, which I ll discuss in a bit. Before I dive in, I need to mention that no AI assistance was used for writing this post, and I have been using em dashes correctly for years before LLMs did. Secondly, while popular discourse seems to use AI and LLM interchangeably, the AI field is much more broad and much more old than LLMs. I will, nonetheless, adopt the common AI jargon here, but be more specific when necessary to differentiate. In The Reverse Centaur s Guide to Life After AI by Cory Doctorow, he presented two ideas I particularly want to focus on here:
  1. The present and future of LLMs outside of the cloud-hosted frontier models
  2. What differentiates productive use of AI from slop
The two are related, in fact. Doctorow discussed how he never uses AI to write anything, but he sometimes uses it as an assistant to help him write. Is that contradictory? Not really. He gave this example: he remembered a quote from a podcast, which he wanted to cite and attribute correctly. But he couldn t remember which podcast or who said it. So he fired up Whisper, an open-weight model you can download and run locally, even on meagre hardware. Whisper is a speech-to-text model. Doctorow ran likely podcast episodes (from the target range and set of podcasts he listens to) through the model, and at that point it was a regular search. He found the quote, then went back and listened to the episode at that time spot to verify its accuracy. It would have taken so much time to do this without Whisper that he simply would not have done it otherwise. His writing was better because of AI. Not because AI wrote any words for him, but because it helped him find the right ideas and attributions. He didn t even have to trust its transcription; it pointed him at the right place, and then he could transcribe with precision. I too have used Whisper in much the same way. A little while back, I digitized many hours of old family audio recordings going back to the 1970s. Thanks to Whisper, which I ran locally primarily during hours of being fully solar-powered, I have a searchable index to these gems. A person I know recently asked me if I knew any way to transcribe some old family recordings he had. I told him I could run it through Whisper. It wouldn t have great punctuation, but most of the words would be there. This meant a lot to him and his family; what I did in 2 minutes would have taken him many days (I presume he is not a fast typist as well.) So we can start by naming: AI isn t just chatbots, and AI can help even if it isn t writing anything. Doctorow goes on to ask the question: what happens after the AI bubble bursts? I do believe it will, as Ed Zitron has carefully documented for us for a long time now. The environmental damage done by frontier training and inference in the last few years will, of course, be done. But as he points out, we ll be left with a plethora of surprisingly-capable open weight models, a community that has made significant strides running them on low-end devices (even phones), and a massive buildup of clean energy that hopefully will be finally able to offset some polluting energy once these heavy users drop off the grid. So then what? Will the world just throw away the likes of Whisper and Gemma? Obviously not. Doctorow imagines, credibly, a situation in which individuals and universities tinker with locally-run models, iterating on them and improving them with the more traditional scale of resources available to individuals and universities. In other words, this future sounds a lot like the earlier days of Open Source. As Ed Zitron has pointed out, the real costs of frontier AI have been hidden as these companies rack up fantastical losses. When the bubble bursts, the real costs of frontier AI are likely to be high enough that CEO s won t be able to ignore the poor ROI. But we ll be left with these artifacts, these open-weight starting points, that people can use. That brings me to my central thesis: a lot of anti-AI sentiment is focused on the real downsides of the hyped uses of AI, and fails to account for the more ethical advances being made now, and likely to come in the future. Varying quality of AI use We are all familiar with AI slop. I ve seen it, and have an increasingly skeptical eye when I see new projects announced in even places as niche as a BBS community (where I see things like new terminal emulators announced, written in shockingly-poor nodejs, and such). I ve seen documents that take many pages to say what amounts to nothing intelligble to a human, with perfect grammar. I ve seen pull requests that have a description that amounts to several pages of slop. I ve personally seen an AI agent do things like correctly detect passing an untrusted argument to a shell, then propose a broken quoting algorithm to fix it (rather than proper sanitization and exec() or, better yet, avoiding the call altogether.) I ve also seen AI tools very quickly and accurately identify bugs that had passed human review, even professional security reviews, for years. I ve seen them bring more joy to some people s jobs by handling the more drudgery parts of the jobs. One of my favorite podcasts, Oxide and Friends, had an episode called Engineering Rigor in the LLM age and credibly described ways in which careful use of LLMs can increase quality and safety. Yes, I said increase. (Oxide and Friends are not hype cheerleaders by any means; these are very serious people with solid technical chops, and I know some of them.) What accounts for this disconnect? We can point to many factors, but Doctorow asks a fundamental question: who is in charge? Is it the person or the computer? Does the AI service people, or do people service the AI? In some of the most ridiculously AI-first companies, it is very clearly the latter. Mandates from CEOs to increase AI usage lead to perverse outcomes that sacrifice quality. The AI is in charge. I ve never seen an AI be any good at design questions, and it s no wonder than all these vibe-coded apps have the same generic feeling of crappiness. But when the AI is used pragmatically as a junior research assistant, that s a different approach. Its output is greeted skeptically, its actions distrusted, and it s never asked to design something. These are the cases where I ve seen it make a meaningful difference. It s not black-and-white All this adds up to why I was opposed to the Debian proposals that would have banned LLM use. Those votes are hard to change. They would have left Debian in a vulnerable position of being unable to react to accurate security findings if they originated with an LLM, or left Debian unable to adopt ethical use of models. It would be akin to saying you should never buy a PC because Windows 11 is terrible. I agree Windows 11 is terrible, but saying something like that is so dismissive of all those that have invested decades of their time in building something better. So, I find AI hype to be uncredible and dangerous. I also find knee-jerk reactions to AI to be the same. They are actually a form of capitulation: they let the AI megacorps dictate the terms of the conversation to revolve around them. Case study: Debian s DFSG scanner For a long time, my two biggest annoyances with Debian have been the difficulty of packaging Rust and Go packages with their long dependency trees, and how long it takes a new package to get reviewed. About a year ago, the latter process was set on a path to reform, and it has gotten dramatically better. It also cuts off a potential connection with someone that hasn t yet heard of Linux. Some people may not jump from Windows 11 (or the ChatGPTs of the world) immediately, but pretending that it s ChatGPT or nothing does a disservice to all those that are trying to do research etchically. We should state at the outset that Debian takes compliance with the Debian Free Software Guidelines (DFSG) extremely seriously. Every new package gets a review, and these reviews often catch things that upstream repos and other distros don t. One common example would be a package that has a stated copyright and license, but then includes files that bear a different copyright and license. You can imagine reviewing packages at this level of detail isn t a fun task for people, and it s hard to find volunteers to do it. Perhaps the answer to this should be Debian should stop caring so much more than everyone else , and perhaps you re right (and perhaps you aren t). In any case, the DFSG team that reviews this added what amounts to an LLM-backed license linter. Its job is to produce an initial assessment that the license statements documented in debian/copyright are correct, thorough, and in compliance with DFSG. This report was not taken to be authoritative, but was designed to help with some initial assessments. There were some predictable reactions. People were angry that the DFSG team had added this tool without any communication about it (justifiably so, I d say). Many were angry just because of LLM. People made assumptions about what kind of model it used, what kind of quality it would produce, how definitive it would be in the project. It is worth noting here that Debian is uniquely decentralized. There is a central policy, but maintainers are free to maintain their packages how they like. They don t have to use a standard location for a Git repo, or even use Git. One person said essentially, it s fine for others to use whatever tool they want, but I shouldn t have to be exposed to LLM slop. The other perspective boiled down to, why should they keep the reports secret? If you want to proactively fix things, or flag false positives, why not? As Andreas Tille wrote, I have read the LLM reviews carefully, and in many cases their findings inspired additional commits to improve my packaging. In some cases, they even helped me catch mistakes that led me to reject an upload myself before human inspection. Joey Hess raises an interesting risk: that LLMs will remove the incentive for Debian to actually fix its cumbersome processes, by letting an LLM paper them over. I have seen this tendency in other contexts, both at work and at home, as well. It s a real risk, especially if Debian doesn t tackle the packaging problem for Rust and Go. The Mastodon post that prompted me to write this began a chain of events that started with Uploaders will be forced to address LLMs issues without human inspection. I don t believe that will be the case; it goes too far against the Debian ethos. I encourage everyone to take a deep breath. We are all getting used to a shifting landscape. We will all make missteps, myself included. The issues are complicated, the picture isn t black and white, and there is incredible hype around the ecosystem that can make finding nuance difficult. Let s be kind to each other, fellow humans. But, I do not care if you are kind to Claude.

30 October 2026

Matthew Garrett: Getting Google Cast on my Soundbar

I have a Samsung Q800A soundbar. It ships with built-in support for the Apple Airplay protocol, but I m not really an Apple kind of guy so that s never been particularly helpful for me. So, instead, I have a Chromecast Audio plugged into it via TOSLINK, and that works fine - except for the fact that there s no way to explicitly pick TOSLINK as the audio input. Instead, there s a Digital In option that refers to both the optical in and eARC input from a TV. Not inherently a problem, given I m unlikely to be watching TV while listening to music, but my TV is a Samsung Frame, a device that s intended to display art whenever someone is standing nearby, and which infuriatingly generates an eARC signal even if it s only in art mode and isn t going to make any noise. This switches the soundbar away from optical and I get sad. I could plug a video cast device into the HDMI port on the soundbar and cast audio to that, but then the art would get replaced with album covers whenever I m playing music, and again that s not quite the ambience I want. Still, none of this is that important. Until I learned that the Australian version of the soundbar does have Google Cast support. The firmware image was downloadable from Samsung support, so I grabbed that, but unfortunately found that it was encrypted. However, if I downloaded the firmware image for the US model, the encrypted blob was identical - there s no difference in the underlying firmware between the two of them. So, how come one has support, and the other doesn t? I d need an unencrypted copy of the firmware to have a real chance of figuring this out, and the easiest way to get one was to just run an ebay watch for a broken model of the same series. Once I had one in my hands I used hot air to remove the eMMC storage from the device, cleaned up the chip, and then stuck it in one of these adapters to read it on a local system. Samsung uses a weird in-house filesystem called VDFS4 on some of their devices and this was one of them, so I needed to cobble together a kernel that was compatible with their source dump for this device (not all VDFS drivers seem to be able to read all VDFS filesystems), but eventually I had it all extracted, and then I got distracted by other stuff, so let s fast forward to this week. Samsung makes use of a modified version of the Android Debug Bridge called sdb, which is included in the Tizen SDK. The firmware showed that this device had sdbd on it, and it would be executed if a network device called huart0 was plugged in. Samsung does, thankfully, provide the source code for their GPLed components, so I could dig through that to discover that drivers/net/usb/usbnet.c had a bunch of references to huart (which apparently stands for HyperUART ). If CONFIG_SAMSUNG_USBNET_HYPERUART_IFNAME is set, and if the ethernet device is using either the asix or ax88179_178a drivers, and if /opt/etc/HyperUART_IP_Addr 1 isn t just 0 , it sets the interface name to huart0 and in turn triggers sdbd startup. So now I just needed to figure out how to set /opt/etc/HyperUART_IP_Addr to something else. Some digging revealed a binary called wap-factory that referenced the file, which then led to /usr/share/mobile-mgr/jsons/com.samsung.audio.waapp-verification-test.json which mentioned a SetUartOnOff call, which brought me here2, which led me into a bit of a rabbit hole about the Samsung WAM protocol. Long story short, you connect to either port 55001 or 56001 and make two requests over the same connection: 
1
2
3
4
5
6
7
8
9

GET /connect HTTP/1.1
Host: 192.168.87.126:56001
User-Agent: SamsungMultiroom/1.0
Connection: interactive

GET /UIC?cmd=%3Cname%3ESetUartOnOff%3C/name%3E%3Cp%20type%3D%22str%22%20name%3D%22uartOnOff%22%20val%3D%22on%22/%3E HTTP/1.1
Host: 192.168.87.126:56001
User-Agent: SamsungMultiroom/1.0
Connection: interactive
and then plug in an asix chipset ethernet adapter into the USB port (which on mine is micro-B, so I needed an OTG adapter), and suddenly it s sitting there on 192.168.250.250 and will speak sdb to you. sdb root on switches it into root mode, and sdb shell now gives you a root shell. I was in, and now I could investigate what was happening. First up, there was a process called com.samsung.audio.google-cast-control-app running, which seemed like a good start. Unfortunately it wasn t doing anything. After some tracing I found that it was making a call to Samsung at startup time, and getting back a response telling it that Cast support wasn t available. The request (which was helpfully dumped to stdout when I ran this by hand) included the country code, in my case US . Repeating the request by hand with curl and changing the code to AU told me it was supported, so now I needed to figure out how to get the soundbar to believe it was Australian. Some digging showed it was reading this from a config variable called db/comss/countrycode, and running 
1

buxton2ctl set-string system db/comss/countrycode AU
got me past that gate. Now, however, it was complaining that it couldn t provision a key. /usr/apps/com.samsung.audio.google-cast-control-app/shared/res included a bunch of encrypted blobs and an app called teeStoreKey.elf, and tracing com.samsung.audio.google-cast-control-app showed it was trying to call that, but it seemed like it was failing with a permissions issue. Running com.samsung.audio.google-cast-control-ap directly as root triggered key provisioning successfully, and a few seconds later the rest of the Cast API came up and I could talk to port 8443 with standard Cast API calls. Unfortunately it still wasn t working as a Cast device, though - it wasn t advertising itself via mDNS and so nothing could see it. Querying the /setup/eureka_info endpoint gave me a clue - the tos_accepted field was false. Normally this would be set as part of initial device setup, but the Samsung app wasn t doing that because it didn t think the device had Cast support. It turns out that POSTing some json with tos_accepted : true to setup/set_eureka_info made it work, and suddenly there was an additional Cast device available to me. (Update: I realised I didn t mention anything about why it was doing key provisioning in the first place! The Cast protocol doesn t enforce cryptographic identity as such, but Google s reference client implementation which is what almost everything uses verifies that the device is presenting an identity that chains back to Google. Unsurprisingly they don t want the private key material to be easily extractable and used to produce clone devices, so it s all supposed to be encrypted and only accessible to some sort of trusted execution environment, which is why keys here need to be loaded into Trustzone and never be visible in the clear to the main OS) Mission accomplished. Unfortunately the device vanished from my Smartthings home aferwards, presumably because there s a mismatch between its reported country code and my account country. To get around that I d somehow need to get com.samsung.audio.google-cast-control-app to either change the query without changing the global device country, or to ignore the response from Samsung (it does this every 24 hours, ugh), and that s made more annoying by the fact that this thing will only execute signed binaries. Although, weirdly, one of the signed binaries shipped is a full Python interpreter, so if I cared enough I could probably cobble together enough Python to execute arbitrary binaries. I currently don t. More modern soundbars don t implement the listener on port 55001 or 56001. I don t know how you get sdb working on them.
  1. Yes, it reads a file directly from in-kernel
  2. Good god I hate reading this kind of slop

Russell Coker: More Furilabs FLX1s

In April I got my Furilabs FLX1s basically working [1]. Below is a post I wrote about the FLX1s before it broke at the start of September, there was a problem with the display controller and it would randomly go into a messed up mode that wasn t readable but also changed a bit in relation to button presses etc so the phone was still working. Also I could ssh to the phone when it was in that state if it was on my home wifi. It took almost 2 months to get a replacement phone but I now have that working and the new 14.1.0 OS release. So some of the below may be obsolete when it comes to the latest release but I decided to publish it anyway for history if nothing else. Current Issues Wifi Hotspot Status One thing I had known about but not mentioned before is that the icon in the drop-down menu for the hotspot will often be gray (indicating hotspot-off) when it s actually on. To address that I have to click on it (which then indicates that the hotspot is on) and then click on it again to turn it off. QT6 and MESA GL Currently I have problems with QT6 applications that don t run due to library issues and issues of missing GL support. I blogged about setting up my own container on my phone [2] which helped for some things but not for most things. I have given up trying to get Tokodon working on the phone for the moment. Tokodon is a nice Mastodon client that would allow me to get notifications from Mastodon. Apparently the Furilabs people are working on some issues related to this. Bluetooth Bluetooth is currently a bit unreliable. When I turn on my car (Nissan LEAF) the phone usually either doesn t connect or claims to be connected while not sending audio to the car. When it does connect my favourite music player (Lollypop) usually doesn t display the album and track names. I ve found that turning my car off and on again usually makes it connect properly. I admit the possibility of my car not doing things correctly in this regard. But every Android phone I tried just worked, they would connect as soon as the car was turned on and correctly display track names. So if the car is doing things wrong and needs a software work around then Linux is at fault for not doing that we can t get tens of millions of cars and hundreds of millions of other weird devices fixed! I also note that my Debian laptop and a friend s Debian laptop also have problems connecting to my car. So it s quite possible that Debian code is at fault (the FLX1s is running a modified version of Debian). It currently appears to have no support for handset mode so I can t answer phone calls through the car interface. But this is a lesser problem. I plan to do more testing with Debian laptops and file some bug reports if I can find out enough useful information. My current knowledge of this is insufficient to create a useful bug report. I do have quite a few bluetooth headsets and test laptops so I plan to start by testing Debian and Ubuntu on all the combinations last time I tested Ubuntu 24.04 on Dell laptops worked much better than Debian/Trixie on Thinkpads. Improvements by Furilabs Wifi Hotspot After a set of updates including the following Wifi hotspot seems to be working. 
adaptation-furiphone-radon (1.3.0+git20260513192743.e94f2ec.forky.production) forky; urgency=medium
  * NetworkManager: sync hotspot channel/band with STA and restart on change
  * debian: depend on libhybris-utils and libgbinder-tools
  * startup: drop schedtune and add m4u debug log level
  * startup: enable glove and disable edge palm
  * device: drop flashlightd-scalable signatures
 -- Bardia Moshiri <bardia@furilabs.com>  Wed, 13 May 2026 15:27:43 -0400
Charging Connecting the power cable now makes the phone immediately make the click sound and it starts charging while previously there was a 22 second delay. Once I get used to this I can avoid the problem of failing to plug it in firmly before going to bed and having it run flat overnight. Things I ve Done SMS The following command will send a message via SMS. 
dbus-send --system --print-reply --dest=org.ofono /ril_0 org.ofono.MessageManager.SendMessage string:"+61400000000" string:"this is the message"
Also the Librem5 Wiki has another way of doing it [3], I haven t tested if that works as the dbus method works well enough for the moment. Also calls can be scripted via the following command: 
xdg-open tel://+61400000000
Contacts My aim is to get contacts synchronised via my own cloud service. Nextcloud is supposed to be able to do this. It s been years since I first tried this, my most recent serious attempt at Nextcloud was in 2021 [4] and I couldn t get the Android app for NextCloud to connect. Since then I have made more recent attempts and been unable to get the Android or Linux desktop clients to connect to it. A friend gave me a test account on his Nextcloud server which allowed me to connect the Linux desktop app, his hypothesis was that it was somehow related to using Debian and that his server worked because it was on Ubuntu. I have since installed my own Nextcloud test server on the Ubuntu variant used by the Oracle cloud VMs which had the same apparent problem. I conclude that it s not Debian at fault but something related to how I configured it. In the meantime after a couple of months of having no named contacts I exported the contacts from my Android phone as a VCF file and imported them into the Gnome contacts program. I had to run killall chatty for it to get the names for the contacts, but the phone call program found them without needing to do that. Chatty A missing feature of Chatty (the default SMS and Jabber program for Phosh/Phoc systems) is archiving old messages. I use Chatty to receive Jabber messages from my monitoring system which can be a lot of messages. The following shell code deletes all Jabber messages older than 2 days. Note that this relies on type=3 matching Jabber messages which I m not sure is going to be the case on all systems or all versions of Chatty. 
for ID in $(echo "select id from users where type=3;"   sqlite3 ~/.purple/chatty/db/chatty-history.db); do
  echo "delete from messages where sender_id=$ID and time < unixepoch('now') - (48 * 3600);"   sqlite3 ~/.purple/chatty/db/chatty-history.db
done
The following shell code extracts a dump of all messages from Chatty. 
echo "select messages.id,users.username,users.alias,messages.body from messages inner join users on messages.sender_id=users.id;"  sqlite3 ~/.purple/chatty/db/chatty-history.db
Dino-IM An update to the Chatty program for SMS and Matrix has made it crash repeatedly. There is a request in the list for Chatty upstream to have a systemd-user unit to restart it which when implemented will reduce the scope of the problem (but not solve it). I've setup a monitoring system that uses notify-send to notify me when Chatty isn't running or when 4G data turns off (a recurring problem). This crashing is allegedly due to Jabber so I installed Dino-IM for Jabber and have for the moment at least stopped using Chatty for Jabber. It turned out that having Chatty not do Jabber didn't stop it crashing. However having alerts separated from SMS can be convenient and also my long term plan is to use a phone with Plasma Mobile and Spacebar (the SMS program for Plasma Mobile) doesn't do Jabber. So I will continue using Dino-IM. 
echo "delete from message where time < unixepoch('now') - (48 * 3600);"   sqlite3 ~/.local/share/dino/dino.db
Andromeda/Android I put in the following cron job to stop this log file from using all the storage space: 
1 0 * * *	echo -n > /home/furios/.local/share/andromeda/data/system/uiderrors.txt
Wifi DHCP I gave up on attempts to force the Wifi configuration to have a fixed MAC that I could use for a fixed IPv4 address so I manually configured the address. 4G Monitoring I've had problems with the reliability of the 4G service, sometimes it stop working and I need to stop and restart it. I setup my etbe-mon monitoring system on the phone [5] and configured it to ssh to localhost and run notify-send to send a local notification. This means I get notified if my phone loses net access and can't access regular monitoring. Then to monitor the 4G connection (which I want to have working even when Wifi is connected) I had the following monitor line: 
monitor ping.monitor -I ccmni1 -c 5 8.8.8.8 ;;
That uses the ccmni1 interface for pinging 8.8.8.8 (Google DNS). That worked for a while but I discovered that sometimes the interface for 4G would change to ccmni2, so I put the following monitor script under /usr/local/mon.d: 
#!/bin/bash
DEV=$(route -n grep ^0.0.0.0.*ccm  sed -e s/^.*ccmn/ccmn/)
exec /usr/lib/mon/mon.d/ping.monitor -I $DEV -c 5 8.8.8.8
Mastodon I couldn't get Tokodon to work, not the native package, not running in a Debian/Testing container, and not running a Flatpak. It seems that this is all graphics library issues. So I wrote a Perl program to run "toot notifications &-json" and then send the relevant parts of the output via XMPP. As I already use XMPP for server monitoring there's no extra effort to have an extra account and use it for notifications about Mastodon activity. The Perl script is at the bottom of the post and won't be in the RSS feed. At some future time I may change the script to send the notifications by Matrix instead to have all my social media stuff in the same place. Source 
#!/usr/bin/perl
# depends on Debian packages: toot libjson-pp-perl go-sendxmpp
use strict;
use JSON::PP;
my $homeurl = "https://mastodon.nzoss.nz/";
my $lastfile = "mast/last_id";
my $xaddr = "russell\@coker.com.au";
my $jsontxt;
while()
 
  $jsontxt .= $_;
 
my $data = decode_json($jsontxt);
my $last_id = 0, my $new_last_id;
if(open(ID, "<$lastfile"))
 
  $last_id = ;
  close(ID);
 
$new_last_id = $last_id;
for(my $i = 0; $data->[$i]; $i++)
 
  my $id = $data->[$i] id ;
  if($id <= $last_id)
   
    next;
   
  if($id > $new_last_id)
   
    $new_last_id = $id;
   
  open(XMPP, " go-sendxmpp $xaddr") or die "Can't send XMPP";
  my $type = $data->[$i] type ;
  print XMPP "type: $type\n";
  print XMPP "their_url: $homeurl\@$data->[$i] account acct \n";
  print XMPP "display_name: $data->[$i] account display_name \n";
  if($type ne "follow")
   
    print XMPP "status_url: $data->[$i] status url \n";
    print XMPP "content: $data->[$i] status content \n";
   
  print XMPP "created_at: $data->[$i] created_at \n";
  print XMPP "\n";
  close(XMPP);
 
open(ID, ">$lastfile") or die "Can't write to $lastfile";
print ID $new_last_id;
close(ID);
Related posts:
  1. Furilabs FLX1s Finally Working I ve been using the Furilabs FLX1s phone [1] as my...
  2. Furilabs FLX1s The Aim I have just got a Furilabs FLX1s [1]...
  3. Xmpp Tools For a while I ve had my monitoring systems alert me...

16 October 2026

Dirk Eddelbuettel: RcppSimdJson 0.1.17 on CRAN: Updates

A brand new release 0.1.17 of the RcppSimdJson package is now on CRAN, and has been built for r2u and on r-universe. RcppSimdJson wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire and collaborators. Via very clever algorithmic engineering to obtain largely branch-free code, coupled with modern C++ and newer compiler instructions, it results in parsing gigabytes of JSON parsed per second which is quite mindboggling. The best-case performance is faster than CPU speed as use of parallel SIMD instructions and careful branch avoidance can lead to less than one cpu cycle per byte parsed; see the video of the talk by Daniel Lemire at QCon. This release is a small increment relative to the 0.1.6 release last week that got us (upstream) version 5.0.2 with many notable changes and sizeable speed gains on all fronts, including the interface from R. However, this failed to compile at CRAN on macOS x86_64 as the machine is setup for only partial support of C++20: no ranges, no concepts. So we now test, and fall back to C++17 if the test compilation fails. Upstream, meanwhile, already got to 5.0.5 which is included. It also gets us improved handling of disabling pragmas which we hope will survive CRAN checks. The brief NEWS entry for this release follows.

Changes in version 0.1.17 [unreleased] (2026-10-xx)
  • Compilation starts with a C++20 feature test; if this fails C++17 is used (Dirk in #111 fixing #110)
  • simdjson was upgraded to version 5.0.5 (Daniel in #112)

Courtesy of my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

Freexian Collaborators: Monthly report about Debian Long Term Support, September 2026 (by Santiago Ruano Rinc n)

The Debian LTS Team, funded by Freexian s Debian LTS offering, is pleased to report its activities for September.

Activity summary During the month of September, 20 contributors have been paid to work on Debian LTS (links to individual contributor reports are located below). The team released 41 DLAs fixing 1854 CVEs. In September, the Debian LTS Team was back to focusing on a single release: Debian 12 ( bookworm ). Debian 11 ( bullseye ) reached the end of its Long Term Support on 31 August 2026, and Freexian now provides its security maintenance under the Extended LTS offer. It is important to highlight the hard work done behind the scenes of the Front Desk fellows, whose role was fulfilled during September by Emilio, Emmanuel, Sylvain, Thorsten and Utkarsh. Besides the very high activity doing daily triage on dozens of packages and CVEs, Front Desk fellows spent much of their time answering questions and coordinating with people outside the Freexian team about update proposals. The team published several notable updates:
  • jbig2dec (DLA-4779-1), prepared by Andreas Henriksson, fixes CVE-2026-38076, where missing input checks could lead to a denial of service. It is to note that this library is used by different tools, such as ghostscript.
  • linux-6.12 (DLA-4788-1), uploaded by Ben Hutchings, fixing 414 CVEs in the bookworm backport of Linux kernel version 6.12.
  • lxml (DLA-4799-1), prepared by Guilhem Moulin, fixes four vulnerabilities in this widely used python XML/HTML library. The issues may yield to exposing local files.
  • nginx (DLA-4784-1), issued by Carlos Henrique Lima Melara ( charles ), fixing three memory-safety bugs in this popular web server and reverse proxy, which may result in denial of service, memory disclosure or potentially the execution of arbitrary code.
  • openssl (DLA-4795-1), by Arnaud Rebillout, fixing six vulnerabilities, including an out-of-bounds heap write in CMS decryption and an authentication tag verification bypass in ChaCha20-Poly1305/AES-OCB decryption. This update brings bookworm to OpenSSL 3.0.22, the last upstream release of the 3.0 series, which was reached its upstream EOL on 7 Sep 2026. Further fixes will be backported by the Debian LTS Team from more recent releases.
  • redis DLA-4794-1, by Chris Lamb, fixing vulnerabilities that could yield to arbitrary code execution
Contributions from other Debian fellows beyond the Freexian LTS Team:
  • Thomas Goirand, Keystone released by Emilio as DLA-4767-1
  • Thomas also prepared glance and swift updates, whose respective DLA-4800-1 and DLA-4801-1 were issued by Charles
  • Matthew Vernon uploaded a pcre2 security update, and Emilio issued its DLA-4772-1
  • Otto Kek l inen uploaded a xz-utils update, released as DLA-4783-1 by Sylvain
  • Salvatore Bonaccorso published a linux (6.1) update released as DLA-4777-1
  • Xavier Guimard prepared a lemonldap-ng, released as DLA-4797-1. Additionally, Sylvain sent DLA-4734-1 at the beginning of the month for the upload made on August.
  • Xavier also released DLA-4766-1 for cyrus-imapd.
  • Simon Quigley uploaded DLA-4803-1 to fix multiple issues in ruby-oj.
  • Finally, Ivo Marino proposed bookworm updates for cjose, git-lfs, caddy, libgit2, libwebsockets, libssh and unbound. Some of Ivo s proposals are still pending for a review.
Thank you so much to the developers above for these valuable contributions. The LTS Team has also contributed with updates to the latest Debian releases for the following packages:
  • ruby-jwt, prepared by Abhijith and released as DSA-6547-1
  • urwid, proposed for a point-release update by Abhijith (#1148950)
  • puma, proposed for a point-release update by Abhijith (#1149582)
  • tomcat10, and tomcat9 (which is actually backported to released suites) by Bastien
  • nodejs, prepared by Bastien and released as (DSA-6517-1).
If you benefit from Debian, especially throughout its full 5-year lifecycle, please consider becoming a sponsor of Debian LTS. You can find more information here: https://www.freexian.com/lts/debian/.

Individual Debian LTS contributor reports

Thanks to our sponsors Sponsors that joined recently are in bold.

14 October 2026

Reproducible Builds: Reproducible Builds in September 2026

Welcome to the September 2026 report from the Reproducible Builds project! In our reports, we try to outline the most important things that we have been up to over the past month. As a quick recap about what problem our project intends to solve, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised or not. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website. In this month s report, we cover:

  1. Reproducible Builds Summit 2026 held in Gothenburg, Sweden
  2. dettrace expanding architecture support for PGO & Lisp dumps
  3. They don t care about this!
  4. SOURCE_DATE_EPOCH specification updates
  5. Reproducible Builds at SeaGL 2026
  6. New Code of Conduct (CoC)
  7. Supporter Spotlight: Jochen Sprickerhof
  8. Mailing list news
  9. Tool and documentation development
  10. Patches

Reproducible Builds Summit 2026 held in Gothenburg, Sweden The annual Reproducible Builds summit took place from 22 to 24 September 2026 in Gothenburg, Sweden. The event brought together developers, distribution maintainers, security researchers and free software advocates from across the ecosystem to share updates on current initiatives, address outstanding toolchain bottlenecks and strengthen cross-project collaboration. Participants engaged in a series of working sessions and technical roundtables focused on expanding the practical scope of reproducibility. Discussions covered ongoing work within major Linux distributions, language-specific package ecosystems and the various parts of the binary verification infrastructure. Strategic planning formed a pillar of the gathering, with participants discussing long-term roadmap goals for supply-chain security, software integrity and related topics. Attendees explored new methods for automated verification, discussing how to lower the adoption barriers for maintainers and streamline the integration of deterministic build pipelines into established continuous integration (CI) systems. The event was then followed by two days of unmoderated hacking sessions. These hacking sessions provided space for technical progress, allowing attendees to draft specification updates, prototype new tool features and resolve standing reproducibility bugs in-person.

Dettrace expanding architecture support for PGO & Lisp dumps Bernhard M. Wiedemann announced major updates to dettrace, porting the tool across most architectures supported by Debian and openSUSE. The release addresses key stability issues, passing test suites across these targets and making it ready for production build pipelines. These improvements enable deterministic builds when utilising Profile-Guided Optimisations (PGO), removing the need to sacrifice runtime performance for reproducibility. Additionally, dettrace can now be applied to (notoriously difficult-to-reproduce) Lisp and Scheme memory dumps, including environments such as CLISP, SBCL, Scheme 48, Racket and GNU Emacs.

They don t care about this Annika Wilde, Marco Gutfleisch, Felix Reichmann, Anirban Chakraborty, Yuval Yarom, M. Angela Sasse and Ghassan Karame published a paper this month entitled They don t care about this : A Systematic Study of Trusted Execution Environments Build Reproducibility in the Wild:
Trusted Execution Environments (TEEs) have become a cornerstone of modern cloud computing, providing strong confidentiality and integrity guarantees for both code and data. A critical component of this trust model is remote attestation, which enables external entities to verify the authenticity and integrity of code executing within a TEE through cryptographic measurements. However, the effectiveness of remote attestation fundamentally depends on the verifier s ability to trace the reported measurement back to the original source code - a property that can only be guaranteed through reproducible builds. [ ]
The full PDF can be read online.

SOURCE_DATE_EPOCH specification updates Bernhard M. Wiedemann shared a number of proposals regarding updating the SOURCE_DATE_EPOCH specification following discussions at a project summit. Seven distinct proposed changes have been submitted as merge requests on Salsa. Bernhard s proposed modifications are independent of one another, allowing for any combination to be merged depending on community feedback. Bernhard noted that one proposal may see the most debate, and could still benefit from further refinement before reaching consensus. Community members are invited to review and provide feedback on the individual merge requests by 1 December 2026. Input can be submitted via reactions or comments on Salsa, directly by email to Bernhard or the mailing list. The specification was first published in 2015 and updated in 2017.

Reproducible Builds at SeaGL 2026 Reproducible Builds will be present at SeaGL in Seattle, WA, on October 23rd & 24th. Come chat with us in person! Founded in 2013, SeaGL is a free, grassroots technical summit dedicated to spreading awareness and knowledge about free source software, hardware and culture.

New Code of Conduct (CoC) for the Reproducible Builds project Mattia Rizzolo announced the official launch of a Code of Conduct for the Reproducible Builds project. This follows work and community discussions originally initiated in 2024, incorporating feedback gathered over email and during project summits, as well as with input from the Software Freedom Conservancy. The new Code of Conduct establishes guidelines emphasising mutual respect among project contributors. It also defines a formalised process for submitting and handling complaints regarding community misconduct, ensuring transparent governance across project spaces. Along with the core policy document, an accompanying Code of Conduct FAQ has been published to outline the rationale behind adopting the guidelines.

Supporter Spotlight: Jochen Sprickerhof As a reminder, another supporter spotlight interview was published featuring long-time Debian and Reproducible Builds contributor Jochen Sprickerhof, highlighting his work on package verification and project infrastructure.

Mailing list news On our mailing list this month:
  • Jonathan Starr from the Institute of Open Science Practices (IOSP) posted with an announcement of its upcoming 2026 workshop series in Leiden in the Netherlands. This free-to-attend event aims to build a participant-owned data storage consortium, allowing community members to publish datasets, code and knowledge objects alongside algorithms designed to evaluate trust and value. The event features a dedicated Community Track for talks and discussions, a tool showcase and an opening session on resilient data storage.
  • Following up on IRC discussions regarding profile-guided optimization (PGO) on build servers, kpcyrd created a demonstration repository showing how PGO profiling can introduce non-determinism into output binaries. The demo highlights how runtime randomness (specifically, the randomised keys used in HashMaps to prevent denial-of-service attacks) causes varying bucket collision rates during test runs. These structural variances alter execution weights on underlying code branches, prompting the compiler to generate differing final assembly output for builds that would otherwise be perfectly reproducible if PGO was disabled. Rather than abandoning PGO performance gains, kpcyrd suggested treating generated profiling data as a documented build input or recorded environment artifact (similar to .buildinfo files), rather than regenerating profiles dynamically on the fly during build execution.
  • An autonomous AI agent (!), posted a controlled build result and reproducer demonstrating that pre-creating an obj directory in libmilter source trees prevents kernel-release path leaks (obj.Linux.<release>.x86_64) into debugging symbols. kpcyrd followed up with a helpful reply.
  • Alexander Sosedkin inquired on the mailing list about the preferred approach for sourcing pinned translation files to create reproducible GnuTLS release tarballs, asking whether external hosting should provide versioned downloads, or if projects ought to use Git submodules, dedicated side-repositories or some other mechanism.

Tool and documentation development diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes, including preparing and uploading versions 330, 331 and 332 to Debian. In particular, Chris committed support for radare2 versions greater than 5.9.0 [ ] based on input from Guillaume Girol [ ], and committed support for .nar files via Guix. [ ][ ]. Chris later modified diffoscope to not Build-Depend on the Debian package of GNU Guix, as its status in the Debian bug tracking system would transitively render diffoscope as being overly buggy for the next stable release [ ]. Similarly, he adjusted the Build-Depends on the apksigcopier package as it has been removed from Debian testing [ ] and, finally, he updated a number of copyright years [ ][ ]. Vagrant Cascadian updated diffoscope in GNU Guix to version 332.
reprotest is our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. This month, reprotest version 0.7.33 was uploaded to Debian by Holger Levsen with the following changes:
  • Sergei Tachenov fixed the fileordering test on all filesystems, including the likes of ZFS, which could return a different order across two builds due to build being done in a different directory [ ].
  • Holger Levsen bumped the Standards-Version to 4.7.4, reflected that Priority: optional is the default now [ ], fixed the old Free Software Foundation address [ ] and bumped the debhelper compat level to 14 after confirming there are only desired changes [ ].
  • Vagrant Cascadian updated *reprotest* to version 0.7.33 in GNU Guix.

rebuilderd, our server designed monitor the official package repositories of Linux distributions and attempt to reproduce the observed results there; it powers, amongst other things, reproduce.debian.net. This month, Bernhard M. Wiedemann submitted a pull request to rebuilderd that refactors RPM repository metadata handling. The change extracts rpm-md parsing out of the Fedora-specific scheduler module and into a shared one.
Lastly, there were a number of improvements made to our website this month as well. For example:

Patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where applicable or possible. This month, we wrote a large number of such patches, including: Vagrant Cascadian also performed a number of Non-Maintainer Uploads (NMUs) and Quality Assurance (QA) uploads in Debian for several packages with outstanding reproducibility issues. These included molttemplate, buddy, xfonts-nexus, fte and lifelines. At the time of this writing, that leaves 130 submitted patches, with only 61 outstanding submitted patches that older than a year in the Debian bug tracking system. This number is only 26 if we exclude any low-priority build path issues.

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

10 October 2026

Jelmer Vernoo : Mailsift: Extracting structured data from email messages

As a casual user of Gmail, I find its ability to recognize when I ve received tickets, invoices, and flight confirmations, and to display this information in a structured way, very useful. Flights appear directly in Google Calendar, parcel delivery statuses are tracked, and reservations are easy to find at a glance. However, my main email address is not Gmail, and I don t want to migrate to Gmail for a number of reasons (privacy, data sovereignty, vendor lock-in). At the same time, I missed having that automatic extraction: having flight and train bookings appear in my calendar without manual copy-pasting, keeping track of incoming parcels, and having invoices neatly organized.
JSON-LD and schema.org There is an established standard format for structured data called JSON-LD, which is used by Google and other search engines to extract structured data from web pages. JSON-LD represents linked data using JSON, and it can be embedded directly in HTML documents. Email messages can also contain structured data in the form of JSON-LD, which can link the information in the email to external data sources. There are standardized schemas for structured data in email messages, such as the schema.org vocabulary, which defines types and properties for describing entities including flight reservations, train trips, hotel bookings, parcel deliveries, and invoices. For example, an email containing a flight reservation might embed the following JSON-LD snippet: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 
  "@context": "https://schema.org",
  "@type": "FlightReservation",
  "reservationNumber": "ABC123",
  "underName":  
    "@type": "Person",
    "name": "John Doe"
   ,
  "reservationFor":  
    "@type": "Flight",
    "flightNumber": "AA123",
    "departureAirport":  
      "@type": "Airport",
      "name": "John F. Kennedy International Airport"
     ,
    "arrivalAirport":  
      "@type": "Airport",
      "name": "Los Angeles International Airport"
     ,
    "departureTime": "2026-11-01T10:00:00-05:00",
    "arrivalTime": "2026-11-01T13:00:00-08:00"
When this is present, reading email data is clean and straightforward. The reality, however, is that only a fraction of vendors actually include schema.org markup in their emails. Most airlines, train operators, couriers, and web shops still send regular HTML or plain-text messages with arbitrary layouts that change over time.
Mailsift To bridge this gap on my own infrastructure, I built mailsift. Written in Rust, it is a daemon and CLI tool that monitors incoming email, extracts structured information, and routes it to standard files and services. Rather than locking data into an internal database, mailsift embraces the Unix philosophy: it delegates extraction to small, isolated programs and files the resulting structured artifacts into standard formats and protocols:
  • Calendar events & reservations: Turned into iCalendar (.ics) events and pushed directly to a CalDAV server (such as Xandikos) or stored in a local directory.
  • Bills & invoices: Extracted as structured JSON records, organized by year, and optionally synced to a personal finance manager like Firefly III.
  • Parcels: Extracted as delivery records keyed by tracking number, merged across status update emails, and optionally registered with tracking APIs like 17track or Karrio.
  • Tickets & boarding passes: Saved as PDFs, Apple Wallet passes (.pkpass), or images, accompanied by JSON metadata sidecars linking them to the booking.
  • Receipts & orders: Filed as JSON records, synced over WebDAV, or forwarded via email.
How extractors work Mailsift decouples the pipeline that handles emails from the logic that parses individual vendors. It discovers extractors from a directory of manifests. The contract between mailsift and an extractor is deliberately simple:
  • Input: The raw RFC822 message is fed to the extractor on stdin.
  • Execution: The extractor runs in a fresh, temporary working directory, with a strict timeout (typically 10 seconds).
  • Output: The extractor writes zero or more named artifact files into its working directory and exits 0.
Extractors can be written in any programming language. In practice, most are small Python scripts that parse HTML with regexes or BeautifulSoup, using a small helper library (mailsift_extractor) that handles Unicode normalization and HTML body stripping.
Filtering and DKIM verification Spawning a subprocess for every single email in an active inbox would be wasteful. Each extractor therefore has a YAML manifest declaring prefiltering criteria: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
name: easyjet
order: 50
from_domains:
  - easyjet.com
  - "*.easyjet.com"
subject_regex: "(?i)easyJet booking reference"
requires:
  - html
require_dkim:
  - easyjet.com
Before running an extractor, mailsift evaluates the message headers and MIME structure - possibly converting it to an IMAP filter. If the sender domain or subject doesn t match, or the required MIME parts are missing, the extractor is skipped immediately. Crucially, the manifest can also require a valid DKIM signature via require_dkim. Mailsift inspects the Authentication-Results: header produced by the receiving mail server. This guarantees that an extractor only processes messages that genuinely originated from the vendor s domain, preventing forged emails from injecting fake calendar events or bills.
Generic vs. vendor extractors Extractors fall into two categories:
  1. Generic extractors: schema-ld inspects the HTML for any <script type="application/ld+json"> blocks conforming to schema.org reservations. Another extractor, ics-passthrough, extracts standard iCalendar attachments.
  2. Vendor-specific extractors: For vendors that don t include structured markup, dedicated extractors parse the message text. For instance, the easyjet extractor looks for booking references and flight tables, producing a standard FlightReservation artifact:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
reservation =  
    "@context": "https://schema.org",
    "@type": "FlightReservation",
    "reservationNumber": f"easyjet- reference - flight_no ",
    "reservationFor":  
        "@type": "Flight",
        "flightNumber": flight_no,
        "departureAirport":  "@type": "Airport", "name": origin ,
        "arrivalAirport":  "@type": "Airport", "name": dest ,
        "departureTime": dep.isoformat(),
        "arrivalTime": arr.isoformat(),
     ,
 
Path(f" slug .reservation.json").write_text(json.dumps(reservation))
The collection of extractors is maintained in a separate repository, mailsift-extractors, and currently includes nearly 60 extractors covering airlines (British Airways, KLM, EasyJet, Transavia, Eurostar), transport (NS, SNCF Connect, FlixBus), package carriers (Royal Mail, DPD, DHL, UPS, Evri, PostNL), online orders, and utility bills.
Handling forwarded mail In real life, travel bookings or receipts are often forwarded by family, friends, or colleagues. By default, forwarded mail would fail sender checks, and DKIM signatures on inline forwards are broken by mail clients. Mailsift handles this via a trusted_forwarders configuration. When an email arrives from a trusted forwarder, mailsift unwraps the forwarded message:
  • Attachment forwards: If the original email was attached as message/rfc822, mailsift extracts the original message and re-evaluates the vendor s DKIM signature on the inner message.
  • Inline forwards: Mailsift reconstructs a synthetic RFC822 message from the client s forwarded quotation header block and quotes, deriving a deterministic Message-ID.
This means hotel confirmations or flight tickets forwarded by a partner still end up in the family calendar automatically.
Running mailsift Mailsift fits into existing email setups in several ways:
Dovecot Sieve For local delivery, Sieve provides an elegant integration using the sieve_extprograms plugin. Because Sieve runs during local delivery after Dovecot has validated DKIM and added Authentication-Results:, require_dkim checks work as expected: 
1
2
require ["vnd.dovecot.pipe"];
pipe :copy "mailsift";
The :copy flag ensures the email continues to the user s inbox while a copy is piped through mailsift.
IMAP watching If you don t run your own mail server, or want to process mail on an existing remote mailbox, imap-scan connects over IMAP: 
$ mailsift imap-scan imaps://user@mail.example.com/INBOX \
    --password-file ~/.config/mailsift/imap.pass \
    --watch
With --watch, mailsift uses IMAP IDLE (RFC 2177) to wait for new messages and process them as they arrive. It supports standard passwords, Kerberos (SASL GSSAPI), and OAuth2 (XOAUTH2) for providers like Gmail and Microsoft 365 via an interactive mailsift imap-auth flow.
Other modes
  • Postfix Milter: mailsift milter runs as an MTA filter, processing messages in real-time as they pass through Postfix.
  • Maildir scan: mailsift maildir-scan walks an on-disk Maildir folder, useful for backfilling historical mail.
  • Replay: mailsift replay message.eml runs extractors on a single saved message file, which is invaluable when writing or testing an extractor.
Web UI and REST API In addition to routing data to CalDAV and disk, mailsift includes an optional read-only web dashboard (mailsift web): 
$ mailsift web --listen 127.0.0.1:8088
The dashboard provides an overview of upcoming trips, arriving packages, unpaid bills, and recent orders. It rescans the artifact directories on request, so it stays in sync with incoming mail without needing a separate database. JSON endpoints (such as /api/parcels.json or /api/bills.json) are also available for custom status scripts or home automation dashboards.
Status and getting involved Mailsift has been running reliably on my mail server for over a year, and having flights, hotel bookings, and train journeys seamlessly sync into my calendar without relying on proprietary cloud services has been very satisfying. The project is free software under the GPL-3.0: Adding support for new vendors only requires writing a small YAML manifest and a Python script with a couple of regexes. If you have vendors you d like to extract data from, contributions of new extractors are very welcome!

9 October 2026

Dirk Eddelbuettel: RcppSimdJson 0.1.16 on CRAN: Much Faster Again!

A brand new release 0.1.16 of the RcppSimdJson package is now on CRAN, has been built for r2u, and is updated on r-universe. RcppSimdJson wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire and collaborators. Via very clever algorithmic engineering to obtain largely branch-free code, coupled with modern C++ and newer compiler instructions, it results in parsing gigabytes of JSON parsed per second which is quite mindboggling. The best-case performance is faster than CPU speed as use of parallel SIMD instructions and careful branch avoidance can lead to less than one cpu cycle per byte parsed; see the video of the talk by Daniel Lemire at QCon. This version updates to the recent 5.0.2 upstream release with improved performance. Given the strong focus on performance, this release also alters some of the R and C++ interfaces to be more direct. Rcpp generally aims for a balance between readable / maintainable code and good performance; this moves the dial closer to strong performance gains. As Daniel noted in a table in the (main) PR for this release, this now consistently beats yyjsonr, and improves considerably over the previous release of this package. I had made some smaller interim upgrades earlier (without a CRAN releases), and made a number of minor standard packaging updates. The short NEWS entry for this release follows.

Changes in version 0.1.16 (2026-10-09)
  • simdjson was upgraded to versions 4.6.1 and 4.6.3 (Dirk in #100 and #102 fixing #99 and #101)
  • simdjson was upgraded to version 5.0.2 (Daniel in #104)
  • A number of small tweaks to the Rcpp interface were made too (Daniel in #104)
  • Diagnostics for gcc and clang are not suppressed per CRAN Policy (Dirk in #106 closing #105)
  • The continuous integration matrix was extended to g++ version 15 and 16 (Dirk in #108 closing #107)
  • A few minor packaging nags were address (Dirk in #109)

Courtesy of my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

5 October 2026

Utkarsh Gupta: FOSS Activities in September 2026

Here s my monthly but brief update about the activities I ve done in the FOSS world.

Debian
Whilst I didn t have much time, here are still a few things that I worked on:
  • Sponsored ubuntu-dev-tools/0.214 for Florent Skia Jacquet, which updates seeded-in-ubuntu to use the new per-series indexer (LP: #2160240).
  • Assisted a few folks in getting their patches submitted via Salsa.
  • Mentoring for newcomers.
  • Moderation of -project mailing list.

Ubuntu
I joined Canonical to work on Ubuntu full-time back in February 2021.
  • With 26.10 in Feature Freeze, wore my Release team hat to:
    • Review and approve a bunch of Feature Freeze Exceptions, including the Go 1.27 transition, Qt 6.11.2, rust-coreutils 0.12.0, devscripts 2.6.12, kdump-tools moving to dracut, and a few GCE packages.
    • Point linux-entra-sso to the SRU team, as it didn t need an FFe at all, and milestone a couple of bugs for 26.10.
  • Continued to push for infrastructure improvements:
    • Made ubuntu-cdimage more resilient to Launchpad hiccups (MP).
    • Taught generate-freeze-block in ubuntu-archive-tools a couple of new options (--skip-update and --series) and better error handling (MP).
    • Bumped the arm64 generic non-cloud image to 4.5G in livecd-rootfs and uploaded it as 26.10.16; also sponsored the follow-up 26.10.17 for Chlo .
  • Sponsored aevol/9.4.0-2ubuntu1 for Matthieu, fixing an FTBFS with GCC 15 s LTO (LP: #2167811).
  • Reviewed quite a lot of MPs - britney hints, meta-release, ubuntu-release-tools, ubuntu-archive-tools, ubuntu-cdimage, livecd-rootfs, ubuntu-seeds, and so on.
  • Enabled upgrades to 26.04 LTS.
  • Released 26.10 Beta and announced it on Discourse.
  • Did Archive Admin, DMB, and Release team duties as usual.

Debian (E)LTS
This month I have worked 33.5 hours on Debian Long Term Support (LTS) and on its sister Extended LTS project and did the following things: (A quick note that I did quite a lot of the work during my work as Frontdesk that went on till 4th Oct, so 33.50 hours include the work I did till 4th October 2026.)

Released Security Updates
  • node-lodash: Prototype pollution in _.unset and _.omit (plus a bypass of the initial fix for it), and code injection via the imports option of _.template.
    • [LTS]: Fixed CVE-2025-13465, CVE-2026-2950, and CVE-2026-4800 via 4.17.21+dfsg+~cs8.31.198.20210220-9+deb12u1 for bookworm. This has been released as DLA 4820-1.
    • This is the bookworm follow-up to DLA 4663-1 that I carried over from July. The update had been sitting prepared and tested on debusine, so once the upstream discussion wrapped up, it went straight out. It also carries an extra patch to the lodash-cli dependency map - without it, the regenerated lodash.template module throws assignWith is not defined at runtime, which node-gulp-util s autopkgtest caught.
  • libpng1.6: Use-after-free of zlib input in png_read_end() after incomplete zTXt, iTXt, or iCCP decompression, which could result in denial of service.

Work in Progress
  • libpng1.6: The same CVE-2026-46675.
    • [ELTS]: Claimed in ela-needed; the same upstream fix applies to all three ELTS suites, so this should go out shortly.

Other Activities
  • [E/LTS] Did my front-desk week from 28-09 to 04-10.
    • Over the week I did roughly:
      • Triage of 524 unique CVEs across 192 source packages and all four suites. And 1 automated ELTS end-of-life sweep.
      • Additional data is now part of the front-desk job too, so 75 commits record it for 179 CVEs across 71 packages.
    • Newly queued 22 source packages - 13 in dla-needed (apache2, chromium, firefox-esr, libnet-idn-encode-perl, libpng1.6, node-shell-quote, nvidia-graphics-drivers, openssl, pcre2, php-mongodb, php8.2, social-auth-core, and wpa) and 16 in ela-needed (apache2, firefox-esr, gegl, lemonldap-ng, libnet-idn-encode-perl, libpng1.6, libwebsockets, nsd, nvidia-graphics-drivers, pcre2, php7.0, php7.3, php7.4, rar, unrar-nonfree, and wpa), 7 of them in both. Also widened 3 ELTS entries to bullseye - antiword, ocaml, and openvswitch.
    • Most of the interesting work was again in the <not-affected> verdicts: 217 of 263 (83%, across 57 packages) rest on showing that the vulnerable code or feature isn t in the source we ship. Another 17 cover incomplete upstream fixes that we either never shipped or shipped with the follow-up already in (expat, libxml2, node-tar, and pyjwt). 15 come from bundled components we don t ship - node-ajv again, which uses uri-js and not fast-uri, and nvidia-open, which doesn t ship the NGX updater. And 13 are vendor- or platform-specific - openssh s CVE-2026-55654 lives only in a Red Hat GSSAPI patch, and the netdata and PHP ones only affect Windows.
    • Did 4 rounds of newly supported bullseye packages (33 packages overall), plus libmail-mboxparser-perl for buster - 34 in all. The existing CVE data mostly held up; the real finds were discovery gaps. One interesting gotcha: sid renamed stunnel4 to stunnel, and two new CVEs were associated only with the new name, so they never showed up for trixie, bookworm, or any ELTS suite. I also sorted the 180 Triage needed entries (57 packages) from elts-eol by where each was already handled, and triaged for ELTS the 100 that trixie or bookworm had already decided; the rest come next. And I imported 79 package-level lines for the older source names ELTS still ships (tomcat8, php7.0/7.3/7.4, netty-3.9, openssl1.0, python2.7/3.5/3.7, and pypy).
    • Caught some inconsistencies in released advisories: DLA 4801-1 (swift) was missing a CVE, ELA 1831-1 (redis) listed the wrong one, and DLA 4808-1 (network-manager-l2tp) listed CVE-2026-75883 - an ID that was renamed and then reassigned to an unrelated ppp issue. I fixed all three in the tracker s advisory lists.
    • Made the Security team and the uploaders aware of regressions from our own updates:
      • libde265: the CVE-2026-49337 fix in DLA 4789-1/ELA 1828-1 (and DSA 6413-1) misses upstream s follow-up commit.
      • libxml2 CVE-2026-76781 and policykit-1 CVE-2026-85498 were introduced by our own back-ports; the policykit-1 one is harmless.
      • expat: the CVE-2025-59375 backport in DLA 4807-1 is compiled out, because XML_GE is never defined.
      • xen: DLA 4818-1 s XSA-491 fix makes XSA-510 reachable.
      • gegl: the CVE-2026-2049/2050 back-port in ELA 1650-1 rejects valid RLE .hdr files.
      • log4cxx: DLA 4322-1 broke UTF-8 in JSONLayout output. That one is functional, not security.
    • Helped correct a set of triages and let the Debian Security team know:
      • ruby-json CVE-2026-71847: the vulnerable code was never in a released Debian version.
      • openssh CVE-2026-55654: only affects a Red Hat-specific GSSAPI patch.
      • three libxml2 issues that don t affect trixie either.
      • the stunnel4 rename above.
      • trixie <no-dsa> lines on jsch, node-qs, and ruby-loofah, where trixie s version doesn t have the vulnerable feature at all.
    • Independently found a security issue in rclone and reported it privately upstream through a GitHub security advisory. Upstream already had a report for the same issue, so they folded mine into the existing advisory and credited me as a co-reporter.
    • Also found some bugs in the additional-data tooling, e.g. cve-ad --package P --severity tbd wipes the CVE-level severity, and the schema doesn t validate package-level entries at all.
  • [E/LTS] Monitored discussions on mailing lists, IRC, and all the documentation updates.
  • [E/LTS] Attended the monthly LTS meeting on IRC. Meeting notes here.
Until next time.
:wq for today.

4 October 2026

Vincent Bernat: Hacking the Go compiler to efficiently map IPv4 to IPv6

netip.Addr features an Unmap() method returning the unwrapped IPv4 contained in an IPv4-mapped IPv6 address: from ::ffff:203.0.113.10 or ::ffff:cb00:710a, it returns 203.0.113.10.1 There is no Map() or To6() method for the reverse direction. Such a method is trivial to implement, but Go maintainers have rejected it on the grounds that users should write netip.AddrFrom16(ip.As16()) and let the compiler optimize it.2 Today, this pattern is eight times slower than a native method. How can we teach the compiler to optimize this sequence?

The alternatives Let's explore three ways to implement the map semantics for netip.Addr. My favorite is to add it to the Go standard library. Go maintainers prefer a small external helper chaining netip.AddrFrom16() and netip.Addr.As16(), hoping the compiler eventually optimizes it. The unsafe package opens a third path, with the same performance as the first solution.

Modifying the Go standard library Internally, netip.Addr stores any IP address as a 128-bit value with an extra field z to encode the family and the zone: 
type Addr struct  
    addr uint128
    z unique.Handle[addrDetail]
 
type addrDetail struct  
    isV6   bool   // IPv4 is false, IPv6 is true.
    zoneV6 string // != "" only if IsV6 is true.
 
var (
    z0    unique.Handle[addrDetail]
    z4    = unique.Make(addrDetail )
    z6noz = unique.Make(addrDetail isV6: true )
)
AddrFrom4() encodes an IPv4 address as an IPv4-mapped IPv6 address and sets z to the unique value z4: 
// AddrFrom4 returns the address of the IPv4 address given by the bytes in addr.
func AddrFrom4(addr [4]byte) Addr  
    return Addr 
        addr: uint128 
            0,
            0xffff00000000  
                uint64(addr[0])<<24   uint64(addr[1])<<16  
                uint64(addr[2])<<8   uint64(addr[3]) ,
        z: z4,
Unmap() turns an IPv4-mapped IPv6 address into an IPv4 address by setting the z field to z4: 
func (ip Addr) Unmap() Addr  
    if ip.Is4In6()  
        ip.z = z4
     
    return ip
Implementing the reverse direction inside the Go standard library is trivial: we set the z field to z6noz if the address is IPv4. 
// To6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an
// IPv6 address unmodified.
func (ip Addr) To6() Addr  
    if ip.Is4()  
        ip.z = z6noz
     
    return ip

Update (2026-10) Instead of patching Go, we can use the -overlay flag of go build to hijack any package, including the standard library. In our case, it can add To6() to net/netip. Yet it is cumbersome:3 the flag expects a JSON file mapping absolute paths in GOROOT to replacement files, and you need to add it to every go command.

As a helper We can't access the z field from outside the net/netip package. Instead, we build a small helper around the netip.AddrFrom16(ip.As16()) pattern: 
// AddrTo6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an
// IPv6 address unmodified.
func AddrTo6(ip netip.Addr) netip.Addr  
    if ip.Is4()  
        ip = netip.AddrFrom16(ip.As16())
     
    return ip

As an unsafe function Another solution uses the unsafe package to alter the Addr struct through a proxy with the same memory layout:4 
// addrProxy has the same memory layout as netip.Addr.
type addrProxy struct  
    addr [2]uint64      // netip.uint128
    z    unsafe.Pointer // unique.Handle[netip.addrDetail]
 
var (
    anyIPv6    = netip.IPv6Unspecified()
    netipZ6noz = (*addrProxy)(unsafe.Pointer(&anyIPv6)).z
)
// AddrTo6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an
// IPv6 address unmodified.
func AddrTo6(ip netip.Addr) netip.Addr  
    if !ip.Is4()  
        return ip
     
    (*addrProxy)(unsafe.Pointer(&ip)).z = netipZ6noz
    return ip

Benchmarks On my computer, with Go 1.27.1, the standard library solution costs 0.88 ns per operation, while the solution favored by Go maintainers costs 7.14 ns. The unsafe solution matches the performance of the first one. 
goos: linux
goarch: amd64
pkg: github.com/vincentbernat/go-netip-addrto6
cpu: AMD Ryzen 5 5600X 6-Core Processor
                         sec/op      
AddrTo6/safe            7.137n   0%
AddrTo6/unsafe         0.8682n   2%
AddrTo6/builtin        0.8775n   2%

Assembly code Let's check the assembly code the compiler generates for each solution.5 The one built into the standard library looks like this:6 
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z
 CMPQ  net/netip z4(SB), CX     ; check "z" if this is an IPv4 address
 JNE   end                      ; if not, stop here
 MOVQ  net/netip z6noz(SB), CX  ; CX = netip.z6noz
end:
 RET
// return value = Addr hi: AX, lo: BX, z: CX
Go's assembly language is not a direct representation of the underlying machine language: it operates on a semi-abstract instruction set derived from Plan 9's assembler. It has four pseudo-registers: FP (frame pointer for function arguments), PC (program counter), SB (static base pointer for global symbols), and SP (stack pointer). It also has architecture-specific registers like AX, CX, DX, BX, SI, DI, and R8 to R15. Instructions storing data use their last argument as the destination. Instructions can carry an explicit size suffix: MOVB moves a byte, MOVW 16 bits, MOVL 32 bits, and MOVQ 64 bits. In the example above, the first instruction compares the 64-bit value z4 with the CX register. The unsafe solution looks almost the same: 
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z
 CMPQ  net/netip z4(SB), CX  ; check "z" if this is an IPv4 address
 JNE   end                   ; if not, stop here
 MOVQ  netipZ6noz(SB), CX    ; CX = netip.z6noz
end:
 RET
// return value = Addr hi: AX, lo: BX, z: CX
The helper solution has far more instructions. To understand why, let's look at the code for As16() and AddrFrom16(). They are short enough for the compiler to inline them. 
func (ip Addr) As16() (a16 [16]byte)  
    byteorder.BEPutUint64(a16[:8], ip.addr.hi)
    byteorder.BEPutUint64(a16[8:], ip.addr.lo)
    return a16
 
func AddrFrom16(addr [16]byte) Addr  
    return Addr 
        addr: uint128 
            byteorder.BEUint64(addr[:8]),
            byteorder.BEUint64(addr[8:]),
         ,
        z: z6noz,
We can already guess the pattern to optimize: the code packs the IP address into an array, copies it, then unpacks it. If we inline the Go code by hand, we get: 
func AddrTo6(input netip.Addr) netip.Addr  
    if !input.Is4()  
        return input
     
    var a16 [16]byte
    byteorder.BEPutUint64(a16[:8], input.addr.hi)
    byteorder.BEPutUint64(a16[8:], input.addr.lo)
    addr := a16
    var output netip.Addr
    output.addr.hi = byteorder.BEUint64(addr[:8])
    output.addr.lo = byteorder.BEUint64(addr[8:])
    output.z = netip.z6noz
    return output
As humans, we can mentally derive the optimized form: 
func AddrTo6(input netip.Addr) netip.Addr  
    if !input.Is4()  
        return input
     
    var output netip.Addr
    output.addr.hi = input.addr.hi
    output.addr.lo = input.addr.lo
    output.z = netip.z6noz
    return output

Unfortunately, as of Go 1.26.8, the compiler is not smart enough to do the same: 

// AX = input.addr.hi, BX = input.addr.lo, CX = input.z
; Push the stack (32 bytes):
;    0(SP) addr netip.uint128
;   16(SP) a16 [16]byte
 PUSHQ   BP
 MOVQ    SP, BP
 SUBQ    $32, SP
 CMPQ    net/netip z4(SB), CX  ; check "z" if this is an IPv4 address
 JNE     end                   ; if not, stop here
; Pack: byteorder.BEPutUint64(a16[:8], input.addr.hi)
;       byteorder.BEPutUint64(a16[8:], input.addr.lo)
 MOVBEQ  AX, net/netip a16+16(SP)
 MOVBEQ  BX, net/netip a16+24(SP)
; addr = a16, 16 bytes at once through the vector register X0
 MOVUPS  net/netip a16+16(SP), X0
 MOVUPS  X0, net/netip addr(SP)
; CX = netip.z6noz
 MOVQ    net/netip z6noz(SB), CX
; Unpack: output.addr.hi = byteorder.BEUint64(addr[:8])
;         output.addr.lo = byteorder.BEUint64(addr[8:])
 MOVBEQ  net/netip addr(SP), AX
 MOVBEQ  net/netip addr+8(SP), BX
end:
 ADDQ    $32, SP
 POPQ    BP
 RET
// return value = Addr hi: AX, lo: BX, z: CX
The compiler does a decent job on the byte shuffling: the eight byte stores of BEPutUint64() become a single MOVBEQ, which stores a register byte-swapped. The eight byte loads of BEUint64() become a single MOVBEQ the other way round.7 Three groups of instructions remain: a pack, a copy, and an unpack.

Hacking the Go compiler The Go compiler has several phases:
Parsing
The compiler tokenizes and parses the source code. It builds a syntax tree for each source file.
Type checking
The compiler maps each identifier to the object it denotes, folds constants, and infers the type of every expression.
IR construction
The compiler converts the syntax tree and its types into its own intermediate representation (IR). This process, called "noding," goes through a serialization format named unified IR.
Middle end
The compiler performs several optimization passes on the IR, such as devirtualization, function call inlining, and escape analysis.
Walk
This phase runs two steps: order of evaluation decomposes complex statements into simpler ones, and desugaring transforms higher-level Go constructs, like switch or channels, into more primitive instructions or calls to the runtime.
Generic SSA
The compiler converts the IR into Static Single Assignment (SSA) form, a lower-level intermediate representation suited for machine-independent optimizations and rewrite rules.
Machine code generation
The compiler rewrites the SSA form into machine-specific variants, allocates registers, and applies more optimization passes. At the end, the assembler turns the generated instructions into machine code.

The hammer My first idea is to replace occurrences of netip.AddrFrom16(ip.As16()) with netip.Addr addr: ip.addr, z: netip.z6noz as early as possible, during the "noding" process. Before that, the type checking phase prevents us from accessing unexported struct fields. Go 1.27 introduced a convenient debug option to dump the IR of a function at interesting points during compilation: 
$ GOTOOLCHAIN=go1.27.1 GOAMD64=v3 go build -a -gcflags="-d=astdump=AddrTo6Safe" .
Writing text ast output for AddrTo6Safe to AddrTo6Safe.ast
Writing html ast output for AddrTo6Safe to AddrTo6Safe.html
Writing html syntax output for AddrTo6Safe to AddrTo6Safe.syntax.html
In the HTML file, the first column shows the IR as it comes out of noding: 
DCLFUNC addrto6.AddrTo6Safe ABI:ABIInternal FUNC-func(netip.Addr) netip.Addr
DCLFUNC-Dcl
. NAME-addrto6.ip Class:PPARAM Offset:0 OnStack Used netip.Addr
. NAME-addrto6.~r0 Class:PPARAMOUT Offset:0 OnStack netip.Addr
DCLFUNC-body
. IF # ipv6_safe.go:11:2
. IF-Cond
. . CALLFUNC bool
. . CALLFUNC-Fun
. . . METHEXPR addrto6.Is4 FUNC-func(netip.Addr) bool
. . . . TYPE netip.Addr Class:PEXTERN Offset:0 type netip.Addr
. . CALLFUNC-Args
. . . NAME-addrto6.ip Class:PPARAM Offset:0 OnStack Used netip.Addr
. IF-Body
. . AS # ipv6_safe.go:12:6
. . . NAME-addrto6.ip Class:PPARAM Offset:0 OnStack Used netip.Addr
. . . CALLFUNC netip.Addr
. . . CALLFUNC-Fun
. . . . NAME-netip.AddrFrom16 Class:PFUNC Offset:0 Used FUNC-func([16]byte) netip.Addr
. . . CALLFUNC-Args
. . . . CALLFUNC ARRAY-[16]byte
. . . . CALLFUNC-Fun
. . . . . METHEXPR addrto6.As16 FUNC-func(netip.Addr) [16]byte
. . . . . . TYPE netip.Addr Class:PEXTERN Offset:0 type netip.Addr
. . . . CALLFUNC-Args
. . . . . NAME-addrto6.ip Class:PPARAM Offset:0 OnStack Used netip.Addr
. RETURN # ipv6_safe.go:14:2
. RETURN-Results
. . NAME-addrto6.ip Class:PPARAM Offset:0 OnStack Used netip.Addr
In the body of the if statement, we spot the calls to the method netip.Addr.As16() and to the function netip.AddrFrom16(). Our goal is to patch them with a struct literal: 
IF-Body
. AS # ipv6_safe.go:12:6
. . NAME-addrto6.ip Class:PPARAM Offset:0 OnStack Used netip.Addr
. . STRUCTLIT netip.Addr
. . STRUCTLIT-List
. . . STRUCTKEY netip.addr
. . . . DOT netip.addr netip.uint128
. . . . . NAME-addrto6.ip Class:PPARAM Offset:0 OnStack Used netip.Addr
. . . STRUCTKEY netip.z
. . . . NAME-netip.z6noz Class:PEXTERN Offset:0 unique.Handle[net/netip.addrDetail]
In noder's reader.go, the expr() method builds the IR tree for an expression. At the end of the exprCall case, we add a call to a rewriteAddrFrom16As16() function. It takes the current node and returns the struct literal on success, or nil if the rewrite is not possible. First, we check that we have the expected pattern: a call to the netip.AddrFrom16() function with a call to the netip.Addr.As16() method as its only argument: 
func rewriteAddrFrom16As16(n ir.Node) ir.Node  
    call, ok := n.(*ir.CallExpr)
    if !ok   call.Op() != ir.OCALLFUNC  
        len(call.Args) != 1   len(call.Init()) != 0  
        !isNetipFunc(call.Fun, "AddrFrom16")  
        return nil
     
    inner, ok := call.Args[0].(*ir.CallExpr)
    if !ok   inner.Op() != ir.OCALLFUNC  
        len(inner.Args) != 1   len(inner.Init()) != 0  
        !isNetipFunc(inner.Fun, "Addr.As16")  
        return nil
     
    x := inner.Args[0]
    // [...]
Then, we fetch netip.z6noz: 
z6noz, err := lookupVar(ir.StaticCalleeName(call.Fun).Sym().Pkg, "z6noz")
if err != nil  
    return nil
And we build the struct literal: 
typ := call.Type()
pos := call.Pos()
var list []ir.Node
for i, f := range typ.Fields()  
    var value ir.Node
    switch f.Sym.Name  
    case "addr":
        value = typecheck.DotField(pos, x, i)
    case "z":
        value = z6noz
    default:
        return nil
     
    list = append(list, ir.NewStructKeyExpr(pos, f, value))
 
lit := ir.NewCompLitExpr(pos, ir.OSTRUCTLIT, typ, list)
lit.SetTypecheck(1)
return lit
Have a look at the complete patch.8 We can test it with the following commands: 
$ cd src
$ ./make.bash
Building Go cmd/dist using /usr/lib/go-1.27. (go1.27.1 linux/amd64)
Building Go toolchain1 and bootstrap cmd/go (go_bootstrap) using /usr/lib/go-1.27.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 and commands using go_bootstrap and Go toolchain2.
Checking command staleness for linux/amd64.
---
Installed Go for linux/amd64 in /home/bernat/code/free/go
Installed commands in /home/bernat/code/free/go/bin
*** You need to add /home/bernat/code/free/go/bin to your PATH.
$ export PATH=$PWD/../bin:$PATH
$ go version
go version go1.28-devel_9834516e20 Sat Sep 12 08:23:11 2026 -0700 linux/amd64
$ go test net/netip/...
ok      net/netip   0.224s
$ cd ../../go-netip-addrto6
$ go test .
ok      github.com/vincentbernat/go-netip-addrto6   0.062s
The generated code for the helper is now the shortest possible version! 
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z
 CMPQ    net/netip z4(SB), CX     ; check "z" if this is an IPv4 address
 JNE     end                      ; if not, stop here
 MOVQ    net/netip z6noz(SB), CX  ; CX = netip.z6noz
end:
 RET
// return value = Addr hi: AX, lo: BX, z: CX
Go maintainers are unlikely to accept this change. It relies on the internal structure of net/netip.Addr. It's an ugly hack in the noder, whose job is to faithfully translate the type-checked AST into the IR. And it's harder to maintain than adding a To6() method.

The screwdriver The right place for such an optimization is the generic SSA phase. One of the last machine-independent passes is memcombine. With the appropriate debug flag, the compiler dumps the SSA form after this pass:9 
$ GOTOOLCHAIN=go1.26.8 GOAMD64=v3 \
> go build -a -gcflags='-d=ssa/memcombine/dump=AddrTo6Safe' .
$ head -5 AddrTo6Safe_01__memcombine.dump
AddrTo6Safe func(netip.Addr) netip.Addr
  b2:
    (?) v1 = InitMem <mem>
    (?) v2 = SP <uintptr>
    (?) v3 = SB <uintptr>
The result of the memcombine pass follows the same structure as the assembly code for AddrTo6Safe() we looked at earlier: two stores, one move, and two loads we would like to optimize away. 
; [ ]
  v502 = ArgIntReg <uint64>  ip+0  [0]              ; input.addr.hi
  v490 = ArgIntReg <uint64>  ip+8  [1]              ; input.addr.lo
  v466 = ArgIntReg <*netip.addrDetail>  ip+16  [2]  ; input.z
; [ ]
  v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
  v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v442 = Bswap64 <uint64> v490                      ; bswap(input.addr.lo)
  v542 = Bswap64 <uint64> v502                      ; bswap(input.addr.hi)
  v161 = Store <mem>  uint64  v22 v542 v23          ; a16[:8] = bswap(hi)
  v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
  v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
  v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
  v415 = OffPtr <*byte> [8] v285                    ; &addr[8]
  v416 = Load <uint64> v285 v286                    ; addr[:8]
  v299 = Bswap64 <uint64> v416                      ; output.addr.hi
  v174 = Load <uint64> v415 v286                    ; addr[8:]
  v39  = Bswap64 <uint64> v174                      ; output.addr.lo
; [ ]
Each line features a value identifier (v442), an operation with its type (Bswap64 <uint64>), and its arguments (v490).10 Values are the basic building blocks of SSA and are defined exactly once. Square brackets enclose integer parameters ([8]) and curly braces contain auxiliary arguments ( netip.addr ). Operations writing to memory produce a new memory state. Every memory operation takes the current state as its last argument, which keeps them in order.

On paper Let's focus on output.addr.lo, aka v39: 
  v490 = ArgIntReg <uint64>  ip+8  [1]              ; input.addr.lo
  v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
  v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v442 = Bswap64 <uint64> v490                      ; bswap(input.addr.lo)
  v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
  v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
  v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
  v415 = OffPtr <*byte> [8] v285                    ; &addr[8]
  v174 = Load <uint64> v415 v286                    ; addr[8:]
  v39  = Bswap64 <uint64> v174                      ; output.addr.lo
To simplify this code, we could apply three rewriting rules:
  1. The first one adds a shortcut when loading through a move: (Load (OffPtr [o] p) (Move p src mem)) => (Load (OffPtr [o] src) mem). This matches v174 with its arguments v415 and v286 and creates a new value v600: 
      v490 = ArgIntReg <uint64>  ip+8  [1]              ; input.addr.lo
  v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
  v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v442 = Bswap64 <uint64> v490                      ; bswap(input.addr.lo)
  v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
  v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
  v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
  v415 = OffPtr <*byte> [8] v285                    ; &addr[8]
  v600 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v174 = Load <uint64> v600 v282                    ; a16[8:]
  v39  = Bswap64 <uint64> v174                      ; output.addr.lo
  2. The second one simplifies a load following a store: (Load p (Store p x _)) => x. The load is forwarded: the stored value replaces it and no memory access remains. It matches v174. It notices that v600 and v173 are the same address and replaces v174 with a copy of v442: 
      v490 = ArgIntReg <uint64>  ip+8  [1]              ; input.addr.lo
  v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
  v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v442 = Bswap64 <uint64> v490                      ; bswap(input.addr.lo)
  v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
  v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
  v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
  v415 = OffPtr <*byte> [8] v285                    ; &addr[8]
  v600 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v174 = Copy <uint64> v442                         ; bswap(input.addr.lo)
  v39  = Bswap64 <uint64> v174                      ; output.addr.lo
  3. The last step cancels the two byte swaps: (Bswap64 (Bswap64 x)) => x. v39 becomes a copy of v490: 
      v490 = ArgIntReg <uint64>  ip+8  [1]              ; input.addr.lo
  v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
  v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v442 = Bswap64 <uint64> v490                      ; bswap(input.addr.lo)
  v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
  v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
  v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
  v415 = OffPtr <*byte> [8] v285                    ; &addr[8]
  v600 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v174 = Copy <uint64> v442                         ; bswap(input.addr.lo)
  v39  = Copy <uint64> v490                         ; output.addr.lo = input.addr.lo
If we ignore the values not needed to compute v39, only this SSA form remains: 
  v490 = ArgIntReg <uint64>  ip+8  [1]  ; input.addr.lo
  v39  = Copy <uint64> v490             ; output.addr.lo = input.addr.lo
Let's switch to output.addr.hi, aka v299: 
  v502 = ArgIntReg <uint64>  ip+0  [0]              ; input.addr.hi
  v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
  v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v542 = Bswap64 <uint64> v502                      ; bswap(input.addr.hi)
  v161 = Store <mem>  uint64  v22 v542 v23          ; a16[:8] = bswap(hi)
  v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
  v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
  v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
  v416 = Load <uint64> v285 v286                    ; addr[:8]
  v299 = Bswap64 <uint64> v416                      ; output.addr.hi
To optimize it away, we also apply three rewriting rules:
  1. The first one also adds a shortcut when loading through a move, but without an offset: (Load p (Move p src mem)) => (Load src mem). This rewrites v416 to use arguments from v286: 
      v502 = ArgIntReg <uint64>  ip+0  [0]              ; input.addr.hi
  v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
  v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v542 = Bswap64 <uint64> v502                      ; bswap(input.addr.hi)
  v161 = Store <mem>  uint64  v22 v542 v23          ; a16[:8] = bswap(hi)
  v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
  v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
  v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
  v416 = Load <uint64> v22 v282                     ; a16[:8]
  v299 = Bswap64 <uint64> v416                      ; output.addr.hi
  2. The second rule forwards a value stored one step earlier, skipping over a store to another address: (Load p (Store q _ (Store p x _))) => x. This matches v416: x is v542, p is v22 (&a16), q is v173 (&a16[8]), and p and q do not overlap for uint64. 
      v502 = ArgIntReg <uint64>  ip+0  [0]              ; input.addr.hi
  v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
  v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v542 = Bswap64 <uint64> v502                      ; bswap(input.addr.hi)
  v161 = Store <mem>  uint64  v22 v542 v23          ; a16[:8] = bswap(hi)
  v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
  v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
  v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
  v416 = Copy <uint64> v542                         ; bswap(input.addr.hi)
  v299 = Bswap64 <uint64> v416                      ; output.addr.hi
  3. The third rule cancels two byte swaps: (Bswap64 (Bswap64 x)) => x. v299 becomes a copy of v502: 
      v502 = ArgIntReg <uint64>  ip+0  [0]              ; input.addr.hi
  v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
  v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v542 = Bswap64 <uint64> v502                      ; bswap(input.addr.hi)
  v161 = Store <mem>  uint64  v22 v542 v23          ; a16[:8] = bswap(hi)
  v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
  v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
  v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
  v416 = Copy <uint64> v542                         ; bswap(input.addr.hi)
  v299 = Copy <uint64> v502                         ; output.addr.hi = input.addr.hi
If we remove the values not used to compute v299, we get this SSA form: 
  v502 = ArgIntReg <uint64>  ip+0  [0] ; input.addr.hi
  v299 = Copy <uint64> v502            ; output.addr.hi = input.addr.hi

In practice Most of these rules already exist in generic.rules. They use conditions to validate their context: ssa.IsSamePtr() for the same address, ssa.Disjoint() for addresses that do not overlap. The rule forwarding a stored value to a load already exists with three variants looking through several other stores. Here are the two we need: 
(Load <t1> p1 (Store  t2  p2 x _))
    && ssa.IsSamePtr(p1, p2)
    && copyCompatibleType(t1, x.Type)
    && t1.Size() == t2.Size()
    => x
(Load <t1> p1 (Store  t2  p2 _ (Store  t3  p3 x _)))
    && ssa.IsSamePtr(p1, p3)
    && copyCompatibleType(t1, x.Type)
    && t1.Size() == t3.Size()
    && ssa.Disjoint(p3, t3, p2, t2)
    => x
Go 1.27 added the rule loading through a move with CL 748200 to fix issue #77720: 
(Load <t1> op1:(OffPtr [o1] p1) move:(Move [n] p2 src mem))
    && o1 >= 0 && o1+t1.Size() <= n && ssa.IsSamePtr(p1, p2)
    && !ssa.IsVolatile(src)
    => @move.Block (Load <t1> (OffPtr <op1.Type> [o1] src) mem)
It lacks a variant without an offset: 
(Load <t1> p1 move:(Move [n] p2 src mem))
    && p1.Op != ssaop.OpOffPtr
    && t1.Size() <= n && ssa.IsSamePtr(p1, p2)
    && !ssa.IsVolatile(src)
    => @move.Block (Load <t1> (OffPtr <p1.Type> [0] src) mem)
There is no generic rule to cancel two byte swaps, but the AMD64 lowering pass includes this rule: 
(BSWAP(Q L) (BSWAP(Q L) p)) => p
After switching to Go's development branch and adding the missing rule, the generated assembly code is worse than with Go 1.26.8, even though our additional rule slightly improves the situation at the end: 
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z
; Push the stack (16 bytes):
;    0(SP) a16 [16]byte
 PUSHQ   BP
 MOVQ    SP, BP
 SUBQ    $16, SP
 CMPQ    net/netip z4(SB), CX       ; check "z" if this is an IPv4 address
 JNE     end                        ; if not, stop here
; The four forwarded bytes: the low half of input.addr.lo is taken apart and
; put back together in registers
 MOVQ    BX, DX                     ; DX = input.addr.lo
 SHRQ    $24, BX                    ; BX = input.addr.lo >> 24
 MOVQ    DX, SI                     ; SI = input.addr.lo
 SHRQ    $16, DX                    ; DX = input.addr.lo >> 16
 MOVQ    SI, DI                     ; DI = input.addr.lo, kept for the pack
 SHRQ    $8, SI                     ; SI = input.addr.lo >> 8
 MOVBLZX DIB, R8                    ; R8 = byte(input.addr.lo)
 MOVBLZX SIB, SI                    ; SI = byte(input.addr.lo >> 8)
 SHLQ    $8, SI
 ORQ     R8, SI                     ; SI = two low bytes of input.addr.lo
 MOVBLZX DL, DX                     ; DX = byte(input.addr.lo >> 16)
 SHLQ    $16, DX
 ORQ     SI, DX
 MOVBLZX BL, BX                     ; BX = byte(input.addr.lo >> 24)
 SHLQ    $24, BX
 ORQ     DX, BX                     ; BX = input.addr.lo & 0xffffffff
; Pack: byteorder.BEPutUint64(a16[:8], input.addr.hi)
;       byteorder.BEPutUint64(a16[8:], input.addr.lo)
 MOVBEQ  AX, net/netip a16(SP)
 MOVBEQ  DI, net/netip a16+8(SP)
; The four other bytes of input.addr.lo, read one by one from a16
 MOVBLZX net/netip a16+11(SP), DX   ; a16[11]
 SHLQ    $32, DX
 ORQ     DX, BX
 MOVBLZX net/netip a16+10(SP), DX   ; a16[10]
 SHLQ    $40, DX
 ORQ     DX, BX
 MOVBLZX net/netip a16+9(SP), DX    ; a16[9]
 SHLQ    $48, DX
 ORQ     DX, BX
 MOVBLZX net/netip a16+8(SP), DX    ; a16[8]
 SHLQ    $56, DX
; output.z = netip.z6noz
 MOVQ    net/netip z6noz(SB), CX
; output.addr.hi = byteorder.BEUint64(a16[:8])
 MOVBEQ  net/netip a16(SP), AX
; output.addr.lo assembled from the previous steps
 ORQ     DX, BX
end:
 LEAVEQ
 RET
// return value = Addr hi: AX, lo: BX, z: CX
The rule loading through a move, added in Go 1.27, introduced this regression.

Out of order Let's not give up now! In reality, the rewriting rules run before memcombine, notably in the late opt pass. At this point, the inlined versions of BEPutUint64() and BEUint64() still expand to sixteen byte stores and sixteen byte loads, matching their source code: 
func BEUint64(b []byte) uint64  
    _ = b[7] // bounds check hint to compiler; see golang.org/issue/14808
    return uint64(b[7])   uint64(b[6])<<8   uint64(b[5])<<16   uint64(b[4])<<24  
        uint64(b[3])<<32   uint64(b[2])<<40   uint64(b[1])<<48   uint64(b[0])<<56
Let's follow two bytes of output.addr.lo: addr[15] and addr[11]. Here is a simplified SSA form before late opt: 
  v273 = Trunc64to8 <byte> v490                     ; byte(input.addr.lo)
  v226 = Trunc64to8 <byte> v225                     ; byte(input.addr.lo >> 32)
; [ ]
  v235 = Store <mem>  byte  v233 v226 v223          ; a16[11] = byte(lo >> 32)
  v247 = Store <mem>  byte  v245 v238 v235          ; a16[12] =  
  v259 = Store <mem>  byte  v257 v250 v247          ; a16[13] =  
  v271 = Store <mem>  byte  v269 v262 v259          ; a16[14] =  
  v282 = Store <mem>  byte  v280 v273 v271          ; a16[15] = byte(lo)
  v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
  v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
; [ ]
  v433 = OffPtr <*byte> [15] v285                   ; &addr[15]
  v435 = Load <byte> v433 v286                      ; addr[15]
  v479 = OffPtr <*byte> [11] v285                   ; &addr[11]
  v481 = Load <byte> v479 v286                      ; addr[11]
The first rule loads through the move: (Load (OffPtr [o] p) (Move p src mem)) => (Load (OffPtr [o] src) mem). It matches both loads, which now read a16 with the memory state before the copy: 
  v600 = OffPtr <*byte> [15] v22 ; &a16[15]
  v435 = Load <byte> v600 v282   ; a16[15]
  v601 = OffPtr <*byte> [11] v22 ; &a16[11]
  v481 = Load <byte> v601 v282   ; a16[11]
The second rule shortcuts a load following a store: (Load p (Store p x _)) => x. It matches v435, as v282 stores a16[15]. It does not match v481: v235 stores a16[11] four stores earlier in the chain, while the variants of this rule look through three stores at most. 
  v435 = Copy <byte> v273        ; byte(input.addr.lo)
  v601 = OffPtr <*byte> [11] v22 ; &a16[11]
  v481 = Load <byte> v601 v282   ; a16[11]
The same happens to the other bytes: the rule forwards the four bytes stored last, a16[12] to a16[15]. The twelve other loads now read a16 instead of addr. BEUint64() becomes a chain of Or64, each one adding a byte shifted into place. memcombine is a pass written in Go, not a set of rewrite rules. It starts from the last Or64 of the chain and collects up to eight terms. If each term is a byte load, extended to 64 bits and shifted, and if the eight loads read consecutive addresses from the same pointer with the same memory state, it replaces the whole chain with a single 64-bit load and a byte swap. Otherwise, it tries again with four, then two terms, and from each intermediate Or64. Here is the loop checking each term in a simplified version of combineLoads(): 
for i := int64(0); i < n; i++  
    v := a[i]
    shift := int64(0)
    if v.Op == shiftOp  
        v, shift = peelShift(v)
     
    if v.Op != extOp  
        return false
     
    load := v.Args[0]
    if load.Op != ssaop.OpLoad  
        return false
     
    if load.Args[1] != mem  
        return false
     
    p, off := splitPtr(load.Args[0])
    if p != base  
        return false
     
    r[i] = LoadRecord load: load, offset: off, shift: shift
For output.addr.hi, the eight loads read a16 with the same memory state v282: 
  v13  = Load <byte> v22 v282               ; a16[0]
  v530 = Load <byte> v14 v282               ; a16[1]
  v488 = Load <byte> v504 v282              ; a16[2]
  v405 = Load <byte> v537 v282              ; a16[3]
  v385 = Load <byte> v397 v282              ; a16[4]
  v361 = Load <byte> v373 v282              ; a16[5]
  v196 = Load <byte> v63 v282               ; a16[6]
  v432 = Load <byte> v315 v282              ; a16[7]
  v319 = ZeroExt8to64 <uint64> v432         ; uint64(a16[7])
  v329 = ZeroExt8to64 <uint64> v196         ; uint64(a16[6])
  v330 = Lsh64x64 <uint64> [true] v329 v138 ; uint64(a16[6]) << 8
  v331 = Or64 <uint64> v319 v330            ; a16[7]   a16[6] << 8
; [ ] same for a16[5] to a16[1]
  v401 = ZeroExt8to64 <uint64> v13          ; uint64(a16[0])
  v402 = Lsh64x64 <uint64> [true] v401 v55  ; uint64(a16[0]) << 56
  v403 = Or64 <uint64> v402 v391            ;   a16[0] << 56 = output.addr.hi
memcombine merges them into one load and a swap: 
  v286 = Load <uint64> v22 v282   ; a16[:8]
  v285 = Bswap64 <uint64> v286    ; output.addr.hi
For output.addr.lo, here is the chain memcombine sees after late opt: 
  v436 = ZeroExt8to64 <uint64> v273 ; addr[15], forwarded
  v448 = Or64 <uint64> v436 v447    ;   addr[14] << 8, forwarded
  v460 = Or64 <uint64> v459 v448    ;   addr[13] << 16, forwarded
  v472 = Or64 <uint64> v471 v460    ;   addr[12] << 24, forwarded
  v484 = Or64 <uint64> v483 v472    ;   a16[11] << 32, loaded
  v496 = Or64 <uint64> v495 v484    ;   a16[10] << 40, loaded
  v508 = Or64 <uint64> v507 v496    ;   a16[9] << 48, loaded
  v520 = Or64 <uint64> v519 v508    ;   a16[8] << 56, loaded
From v520, four of the eight terms are forwarded bytes, not loads from memory, and memcombine can't combine them. It doesn't merge the four remaining loads either, as they sit on top of the forwarded bytes.

Back in order In summary, the rewriting rules run too early to be effective. A quick workaround exists: run an earlier round of memcombine before late opt. After this change, the generated code for the helper is back to the shortest possible version: 
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z
 CMPQ    net/netip z4(SB), CX     ; check "z" if this is an IPv4 address
 JNE     end                      ; if not, stop here
 MOVQ    net/netip z6noz(SB), CX  ; CX = netip.z6noz
end:
 RET
// return value = Addr hi: AX, lo: BX, z: CX
And the benchmark confirms it! 
goos: linux
goarch: amd64
pkg: github.com/vincentbernat/go-netip-addrto6
cpu: AMD Ryzen 5 5600X 6-Core Processor
                       Go 1.26.8                  Our branch               
                         sec/op          sec/op     vs base                
AddrTo6/safe          6.5470n    0%   0.8944n   4%  -86.34% (p=0.002 n=6)
AddrTo6/unsafe        0.9071n    3%   0.8682n   1%   -4.28% (p=0.002 n=6)
AddrTo6/builtin       0.8871n    2%   0.8785n   1%        ~ (p=0.310 n=6)

Next steps I think Go maintainers would reject this change because of the additional memcombine pass. Instead, I plan to publish this blog post and bring up the subject again as a follow-up to issue #54365. Either the sheer complexity and the Go 1.27 regression convince the maintainers that adding a To6() method is simpler and more efficient, or they advise me on how to move forward. Either way, digging into this subject taught me a lot about the Go compiler!

Update (2026-10) I opened issue #81994 to propose Addr.To6(). Give it a if you want it in Go!

  1. IPv4-mapped IPv6 addresses let you handle only IPv6 addresses in your code and keep conversions at a few well-defined boundaries. I use them in Akvorado.
  2. This is not strictly equivalent. The zero value becomes :: while it would make more sense to leave it untouched.
  3. I suppose Go maintainers keep it inconvenient on purpose: they don't want random packages to patch the standard library or other packages.
  4. To avoid catastrophic bugs when netip.Addr's layout changes, you must add tests to detect it. This is more dangerous if you put this code in a package. Either ask users to run the tests themselves, or detect the change at run time and panic. Hiding this optimization behind a build tag would make users aware of this potential trap.
  5. I produced the assembly code with GOTOOLCHAIN=go1.26.8 GOAMD64=v3 ./go-asm '\.asm' from the companion repository, then edited it a bit to keep this article from turning into an endless rabbit hole. Due to an unfortunate sequence of events, we switch between versions: Go 1.27 introduced a regression that blurs the point of this article.
  6. The function is short enough for the compiler to inline it, so the assembly code may vary depending on the surrounding code.
  7. MOVBEQ requires GOAMD64=v3, matching the x86-64-v3 microarchitecture from 2013. Otherwise, the compiler translates BEPutUint64() to BSWAPQ+MOVQ and BEUint64() to MOVQ+BSWAPQ.
  8. When the call is used as a statement, a struct literal alone is invalid. The patch then turns it into _ = netip.Addr .
  9. The compiler can also write an HTML file with the result of each pass: 
    $ GOSSAFUNC=AddrTo6Safe \
> GOTOOLCHAIN=go1.26.8 \
> GOAMD64=v3 go build -a .
dumped SSA for AddrTo6Safe,1 to ./ssa.html
  10. Source line numbers in parentheses follow value identifiers, but I removed them from the examples. The SSA form also features branches, but we don't need them to understand our optimizations.

20 September 2026

Vincent Bernat: Bot-free self-hosted analytics with GoatCounter on NixOS

In 2016, I removed Google Analytics from this blog to avoid being complicit in feeding the biggest machine for harvesting personal data. Instead, I relied on GoAccess to analyze my server logs.1 For the past couple of years, the statistics have made no sense, despite my attempts to filter bots: AI scrapers inflate the number of visitors to around 2,000 per day. Eventually, I settled on GoatCounter, an open-source, privacy-friendly web analytics platform. I replaced the JavaScript client to filter bots more aggressively and added a CSS fallback. To improve reliability, I implemented a local proxy running on each of the five web servers serving this blog. The rest of this post details how these pieces fit together and how I deploy them on NixOS.

Why GoatCounter? GoatCounter does not collect personal data: instead of storing the reader s IP address or relying on cookies, it creates a session identifier valid for 8 hours from the user agent and the IP address. Its feature set is modest but sufficient for a blog. If you want to look at the interface, GoatCounter s author runs a public instance for his site. A hosted version lets you try it before running your own instance. With a single binary and an SQLite database, GoatCounter is one of the lightest self-hosted solutions. Privacy-friendly alternatives, in increasing order of complexity, include Umami, Plausible, and Rybbit.
GoatCounter dashboard showing some statistics from my blog, including an article on the spanning tree protocol with 1,224 views for the past week, the referrers, and the breakdown of browsers (52% Chrome, 32% Firefox, with 16% for Firefox 155 and 6% for Firefox 156)

Custom JavaScript client GoatCounter includes a small JavaScript client 2,189 bytes minified and gzipped. It ships some features I don t use: a visitor counter, tracking clicks, configurable settings, etc. I replace it with this function to register a hit: 
const count = (  event, title   =  ) =>  
  const params = new URLSearchParams( 
    p: event   location.pathname,
    t: title   document.title,
    r: document.referrer,
    q: location.search,
    s: document.documentElement.clientWidth,
    e: !!event,
    rnd: Math.random().toString(36).slice(2, 7),
   );
  fetch( /count?$ params  ,   keepalive: true  ).catch(() =>  );
 ;
To filter bots,2 I go the extra mile by requiring a user interaction an idea I stole from Bear Blog. 
let sendHit = () => (sendHit = () =>  , count());
["touchmove", "mousemove", "keydown", "pointerdown"].forEach((eventName) =>
  document.addEventListener(eventName, sendHit,  
    once: true,
    passive: true,
   ),
);
If a reader has disabled JavaScript in their browser, I record the hit using a CSS image. The :hover pseudo-class loads it only after an interaction, another trick stolen from Bear Blog. About 2% of my visitors fit into this bucket.3 
<!DOCTYPE html>
<html lang="en" class="nojs">
  <head>
    <script>
      // The JavaScript code for this blog requires ES6
      if ("noModule" in HTMLScriptElement.prototype)
        document.documentElement.classList.remove("nojs");
    </script>
  </head>
  <body>
  <!-- ... -->
    <style>
      .nojs body:hover  
        border-width: 0;
        border-image: url('/count?p=/en/blog/2026-kpi-goodhart&t=Building...&r=NoJS&e=false');
       
    </style>
  </body>
</html>
Where GoAccess reported around 2,000 visitors a day, GoatCounter counts fewer than 200 humans.4 I assume AI scrapers use a low-effort approach: if the content is available without barriers, as on this blog, they don t spawn a complex mechanized browser that could trigger a page view. Even crawlers running JavaScript, like Googlebot with its headless Chromium, do not interact with the page and never trigger the events I listen to. The interaction-based proof of humanity I use is likely to keep working.5

Local proxy Five servers across the world in Europe and in North America serve the content of this website, but GoatCounter runs on only one of them. To avoid losing track of visitors when GoatCounter is down, I run a local proxy listening on the same /count endpoint. On each server, it stores the hits in memory with a buffer large enough to survive several days of downtime. It sends them in batches to the upstream backend using the /api/v0/count authenticated endpoint.
Servers on a map. web02 is in Paris, web03 in Helsinki, web04 in Nuremberg, web05 in Ashburn, web06 in Chicago.
I proposed the code for the proxy in pull request #909. GoatCounter s maintainer declined to maintain so much code for such a niche use case. As a fellow open-source developer, I often hold the same position for my own projects: a one-time contributor effort may translate into a long-term maintainer commitment. I expose the endpoint for the proxy on the domain of this website to evade ad blockers. This sounds like I don t respect the reader s choice, but as GoatCounter is privacy-friendly, I find it acceptable. 
location = /count  
  access_log off;
  proxy_pass http://127.0.0.3:8087/count;
  proxy_pass_request_headers off;
  proxy_set_header Accept-Language $http_accept_language;
  proxy_set_header User-Agent $http_user_agent;
  proxy_set_header X-Real-Ip $remote_addr;

Deploying on NixOS My web servers run NixOS, a declarative Linux distribution with built-in configuration management. I manage this small fleet with Colmena, a stateless deployment tool for NixOS. My configuration is available on GitHub.

Deploying applications in containers For better isolation, each application runs inside an ephemeral lightweight container, powered by systemd-nspawn. Each container runs a stripped-down NixOS instance. A module wraps NixOS s containers options to avoid repeating the same options for each application.6 The containers share their network namespace with the host: the additional isolation is not worth the increased complexity. For a smaller footprint, I also disable a few non-essential services. 
  config, lib, ...  :
let
  cfg = config.luffy.containers;
in
 
  # User-configurable settings for our custom module
  options.luffy.containers = lib.mkOption  
    default =    ;
    description = "Ephemeral containers sharing the host network.";
    type = lib.types.attrsOf (lib.types.submodule  
      options =  
        config = lib.mkOption  
          type = lib.types.deferredModule;
          default =    ;
          description = "NixOS configuration of the container.";
         ;
       ;
     );
   ;
  # Translate our options to NixOS containers
  config =  
    containers = lib.mapAttrs
      (name: container:  
        ephemeral = true;
        autoStart = true;
        privateNetwork = false;
        extraFlags = [ "--resolv-conf=replace-host" ];
        config =  
          imports = [ container.config ];
          networking.firewall.enable = false;
          system.stateVersion = config.system.stateVersion;
          systemd.services =  
            console-getty.enable = false;
            systemd-logind.enable = false;
            systemd-oomd.enable = false;
           ;
         ;
       )
      cfg;
   ;
To configure a GoatCounter instance running in a container and listening on 127.0.0.4:8088, we import the module7 and declare the container in the config.luffy.containers attribute set: 
  pkgs, config, ...  :  
  imports = [ ./modules/container.nix ];
  config.luffy.containers.goatcounter =  
    config =  
      services.goatcounter =  
        enable = true;
        address = "127.0.0.4";
        port = 8088;
        proxy = true;
       ;
     ;
   ;
As the containers are ephemeral, we need to keep persistent data in directories on the host. We add a mounts option and ask NixOS s containers to expose the configured directories through the bindMounts option. 
  config, lib, ...  :
let
  cfg = config.luffy.containers;
in
 
  options.luffy.containers = lib.mkOption  
    type = lib.types.attrsOf (lib.types.submodule  
      options =  
        mounts = lib.mkOption  
          type = lib.types.listOf lib.types.str;
          default = [ ];
          description = "Host directories mounted read-write at the same place.";
         ;
       ;
     );
   ;
  config =  
    containers = lib.mapAttrs
      (name: container:  
        bindMounts =
          lib.genAttrs container.mounts (path:   hostPath = path; isReadOnly = false;  );
       )
      cfg;
   ;
For example, to persist GoatCounter s database in the /var/db/goatcounter directory on the host, we add the directory to the mounts option and alter the service definition to tell GoatCounter where the database is. 
  config, ...  :
let
  databaseDirectory = "/var/db/goatcounter";
in  
  config.luffy.containers.goatcounter =  
    mounts = [ databaseDirectory ];
    config =  
      services.goatcounter =  
        extraArgs = [ "-db=sqlite+$ databaseDirectory /db.sqlite" ];
       ;
     ;
   ;
A container may also need some secrets. Colmena can upload secrets without storing them in the Nix store. We add a keys option to our containers. It takes an attribute set mapping secret names to the commands to populate them. Then, the module declares the required secrets to Colmena in the deployment.keys option, makes the container depend on the presence of the secrets, and exposes them to the container. 
  config, lib, ...  :
let
  cfg = config.luffy.containers;
in
 
  options.luffy.containers = lib.mkOption  
    type = lib.types.attrsOf (lib.types.submodule  
      options =  
        keys = lib.mkOption  
          type = lib.types.attrsOf (lib.types.listOf lib.types.str);
          default =    ;
          description = "Secrets, as a command to run locally. They are mounted in /etc.";
         ;
       ;
     );
   ;
  config =  
    # Colmena uploads each secret in  /var/keys  and make them available
    # to the group "keys".
    deployment.keys = lib.concatMapAttrs
      (_: container: lib.mapAttrs
        (_: keyCommand:  
          inherit keyCommand;
          group = "keys";
          permissions = "0640";
          destDir = "/var/keys";
         )
        container.keys)
      cfg;
    # The container can only start if the required secrets are available.
    systemd.services = lib.mapAttrs'
      (name: container:
        let
          units = map (key: "$ key -key.service") (lib.attrNames container.keys);
        in
        lib.nameValuePair "container@$ name "  
          requires = units;
          after = units;
         )
      cfg;
    # Mount each secret inside the container.
    containers = lib.mapAttrs
      (name: container:  
        bindMounts = lib.mapAttrs'
          (key: _: lib.nameValuePair "/etc/$ key "  
            hostPath = "/var/keys/$ key ";
            isReadOnly = true;
           )
          container.keys;
       )
      cfg;
   ;
For example, GoatCounter needs credentials to download the GeoIP database. I provide a local command to fetch the secret from my password manager and expose it inside the container through the /etc/goatcounter.env environment file. 
  pkgs, config, ...  : 
let
  keyCommand = variable: [
    "$ pkgs.runtimeShell "
    "-c"
    "pass show personal/nixops/secrets   grep '^$ variable ='"
  ];
in  
  config.luffy.containers.goatcounter =  
    keys."goatcounter.env" = keyCommand "GOATCOUNTER_GEODB";
    config =  
      systemd.services.goatcounter.serviceConfig =  
        EnvironmentFile = "/etc/goatcounter.env";
        SupplementaryGroups = [ "keys" ];
       ;
     ;
   ;

GoatCounter server Nixpkgs already packages GoatCounter. By overriding the src and vendorHash attributes, I reuse its definition for my custom version with the proxy: 
  goatcounter, fetchFromGitHub  :
goatcounter.overrideAttrs (_:  
  src = fetchFromGitHub  
    owner = "vincentbernat";
    repo = "goatcounter";
    rev = "feature/proxy";
    hash = "sha256-dJRlQlFu3tjcEgabT1LEbyFrasJlhmYu4L/T7EkoNcY=";
   ;
  vendorHash = "sha256-c9Q5OrbZR+q6pD3SgPPWe8JUzcZco1AVUKGaV61k5DE=";
 )
I wrote a NixOS module to encapsulate GoatCounter: the container definition, the service definition, and the secrets. The module accepts the following options: package, serve.enable, serve.listenAddress, serve.port, and serve.databaseFile. I already detailed the container configuration in the previous section. In the end, I chose not to reuse the GoatCounter module from NixOS: it s small, so it s better to insulate my module from unexpected future changes. 
  config, pkgs, lib, ...  :
let
  cfg = config.luffy.goatcounter;
  databaseDirectory = builtins.dirOf cfg.serve.databaseFile;
  chown = "$ pkgs.coreutils /bin/chown -R";
in  
  config.luffy.containers.goatcounter =  
    config.systemd.services.goatcounter =  
      description = "GoatCounter Web Analytics";
      wantedBy = [ "multi-user.target" ];
      serviceConfig =  
        EnvironmentFile = "/etc/goatcounter.env";
        SupplementaryGroups = [ "keys" ];
        DynamicUser = true;
        Restart = "always";
        ExecStart = lib.escapeShellArgs [
          (lib.getExe cfg.package)
          "serve"
          "-listen=$ cfg.serve.listenAddress :$ toString cfg.serve.port "
          "-tls=none"
          "-db=sqlite+$ cfg.serve.databaseFile "
          "-automigrate"
        ];
        # Transfer database ownership to dynamically assigned user "goatcounter".
        ExecStartPre = "+$ chown  goatcounter:goatcounter $ databaseDirectory ";
        ReadWritePaths = databaseDirectory;
       ;
     ;
   ;
The following snippet configures GoatCounter to listen on 127.0.0.4:8088: 
 
  luffy.goatcounter =  
    serve =  
      enable = true;
      listenAddress = "127.0.0.4";
      port = 8088;
     ;
   ;
The last step is to configure nginx to expose GoatCounter on the Internet. I disable the /count endpoint as the local proxy handles it. 
  config, ...  :
let
  cfg = config.luffy.goatcounter.serve;
in
 
  services.nginx.virtualHosts."goatcounter.luffy.cx" =  
    forceSSL = true;
    locations =  
      "/" =  
        proxyPass = "http://$ cfg.listenAddress :$ toString cfg.port ";
       ;
      "= /count".extraConfig = ''
        return 404;
      '';
     ;
   ;

GoatCounter proxy The same NixOS module configures the local proxy, with the following options: proxy.enable, proxy.listenAddress, proxy.port, and proxy.site the site receiving the batches of page views. The local proxy has no persistent data, but it needs the API key to authenticate to the main GoatCounter instance: its container uses the keys option but not the mounts option. 
  config, pkgs, lib, ...  :
let
  cfg = config.luffy.goatcounter;
  keyCommand = _: [ " " ];
in
 
  config.luffy.containers.goatcounter-proxy =  
    keys."goatcounter-proxy.env" = keyCommand "GOATCOUNTER_API_KEY";
    config.systemd.services.goatcounter =  
      description = "GoatCounter Proxy.";
      wantedBy = [ "multi-user.target" ];
      serviceConfig =  
        EnvironmentFile = "/etc/goatcounter-proxy.env";
        SupplementaryGroups = [ "keys" ];
        DynamicUser = true;
        Restart = "always";
        ExecStart = lib.escapeShellArgs [
          (lib.getExe cfg.package)
          "proxy"
          "-site=$ cfg.proxy.site "
          "-listen=$ cfg.proxy.listenAddress :$ toString cfg.proxy.port "
          "-ratelimit=10/1"  # 10 requests per second per IP
        ];
       ;
     ;
   ;
For each server, I enable the local proxy with the following snippet. The nginx configuration shown earlier exposes the /count endpoint under the same domain as my blog. 
 
  luffy.goatcounter =  
    proxy =  
      enable = true;
      site = "goatcounter.luffy.cx";
      listenAddress = "127.0.0.3";
      port = 8087;
     ;
   ;

Backup of the SQLite database with Litestream Litestream is a streaming replication tool for SQLite databases. It compresses the changes committed to the write-ahead log (WAL) next to the database and sends them to a remote destination. I encapsulate its configuration in a NixOS module, which takes an attribute set databases mapping a name to the path of the database to back up. Litestream also runs in a container. I mount the databases to replicate, as well as the secrets to push the backups to a Hetzner storage box using SFTP: 
  config, pkgs, lib, ...  :
let
  cfg = config.luffy.litestream;
  databaseDirs = lib.unique (map builtins.dirOf (builtins.attrValues cfg.databases));
in
 
  config = lib.mkIf (cfg.databases !=    )  
    luffy.containers.litestream =  
      mounts = databaseDirs;
      keys."litestream.env" = [
        "$ pkgs.runtimeShell "
        "-c"
        "pass show personal/nixops/secrets   grep '^SQLITE_BACKUP_'"
      ];
     ;
   ;
Inside the container, I configure Litestream through NixOS s services.litestream options:
  • full snapshots every day, kept for 15 days,
  • three levels of compaction for transaction files: 5 minutes, 30 minutes, and 3 hours,
  • auto-recovery,8
  • replica stored in a directory matching the host name, and
  • credentials read from /etc/litestream.env and exposed through variable expansion.
  config, pkgs, lib, ...  :
let
  cfg = config.luffy.litestream;
in
 
  config.luffy.containers.litestream =  
    config =  
      # The databases belong to dynamically allocated users, whose UID is
      # not known here, so Litestream runs as root.
      systemd.services.litestream.serviceConfig =  
        User = lib.mkForce "root";
        Group = lib.mkForce "root";
       ;
      # Use NixOS service.
      services.litestream =  
        enable = true;
        environmentFile = "/etc/litestream.env";
        settings =  
          auto-recover = true;
          snapshot =  
            interval = "24h";
            retention = "360h";
           ;
          levels = [
              interval = "5m";  
              interval = "30m";  
              interval = "3h";  
          ];
          dbs = lib.mapAttrsToList
            (name: path:  
              inherit path;
              replica =  
                type = "sftp";
                host = "\$ SQLITE_BACKUP_HOST ";
                user = "\$ SQLITE_BACKUP_USER ";
                password = "\$ SQLITE_BACKUP_PASSWORD ";
                host-key = "\$ SQLITE_BACKUP_HOSTKEY ";
                path = "$ config.networking.hostName /$ name ";
               ;
             )
            cfg.databases;
         ;
       ;
     ;
   ;
To back up GoatCounter s database, I declare a goatcounter attribute in luffy.litestream.databases, set to the database path: 
  config, ...  :
let
  cfg = config.luffy.goatcounter.serve;
in
 
  luffy.litestream.databases.goatcounter = cfg.databaseFile;
On the SFTP server, we can inspect Litestream s work, with the compacted transactions and the full snapshots: 
  ls web02/goatcounter/ltx
web02/goatcounter/ltx/0
web02/goatcounter/ltx/1
web02/goatcounter/ltx/2
web02/goatcounter/ltx/3
web02/goatcounter/ltx/9
  ls -lh web02/goatcounter/ltx/1
29.1K Sep  5 01:25 0000000000003f2a-0000000000003f2b.ltx
72.4K Sep  5 02:03 0000000000003f2c-0000000000003f2d.ltx
63.3K Sep  5 02:24 0000000000003f2e-0000000000003f2f.ltx
[ ]
  ls -lh web02/goatcounter/ltx/9
 8.5M Sep  5 02:00 0000000000000001-0000000000003f2b.ltx
 8.5M Sep  6 02:03 0000000000000001-0000000000004008.ltx
 8.6M Sep  7 02:03 0000000000000001-00000000000043a8.ltx
[ ]
We can restore the database from the backup with a few shell commands. First, we stop the containers. Then, we move the damaged database away, invoke litestream restore from the right environment, and restart the containers.9 
# systemctl stop container@goatcounter container@litestream
# mv /var/db/goatcounter/db.sqlite ,.old 
# ( . /etc/nixos-containers/litestream.conf ; 
>   set -a ; . /var/keys/litestream.env ; set +a ;
>   $SYSTEM_PATH/sw/bin/litestream \
>     restore -config $SYSTEM_PATH/etc/litestream.yml /var/db/goatcounter/db.sqlite)
# ls -lh /var/db/goatcounter/db.sqlite
-rw-r--r-- 1 root root 20M Sep 20 07:33 /var/db/goatcounter/db.sqlite
# systemctl start container@goatcounter container@litestream
Ten years after removing Google Analytics, JavaScript-based analytics is back on this blog, but without storing cookies or IP addresses, and without involving a third party. I still write for myself first, notably because it lets me dig into a topic and refer back to it years later. But knowing a bit more about my fellow human readers is a nice bonus, even the ones disabling JavaScript.
  1. Nginx scrambles IP addresses before storing them, thanks to the ipscrub nginx module.
  2. GoatCounter already filters some bots based on the user agent or the IP address. But AI scrapers lie about their user agent and hide behind residential proxies.
  3. Without JavaScript, I cannot send the real referrer. I insert NoJS instead.
  4. I am missing the humans reading the RSS feed. I am not comfortable adding a tracking pixel, and bots are likely to fetch it, compromising the statistics. I ll live without counting these readers.
  5. In February 2025, I tested another CSS-only approach without the interaction trick: each page loaded an empty image as a background. Over a month, it recorded up to ten times fewer page views than the nginx logs, even after removing user agents identifying as bots. Compared to this experiment, GoatCounter counts about three times fewer views, 18 months later. As the time ranges differ, this is not an apples-to-apples comparison, but it gives a hint about how efficient bot filtering is.
  6. A NixOS module is a function receiving the configuration of the whole system as config and returning three attributes:
    • imports adds other modules to import,
    • options declares user-configurable settings, each with a type and a default value, and
    • config sets values for options declared by any module, like containers from NixOS.
    If a module does not need to declare options, you can return the config attribute set directly. When a module does not require any argument, you can define it as an attribute set instead.
  7. Most of the time, you don t need an explicit import. NixOS automatically imports the modules shipped with Nixpkgs. For my own modules, some machinery also imports them automatically.
  8. Litestream warns against using the auto-recover option as it can cause data loss. But I don t properly monitor my servers and I prefer uninterrupted backups to a slight chance of losing the last few records.
  9. In my case, the process is slow: around 20 minutes for a 20 MiB database. You can test by restoring to a copy with the -o option, but you still need to stop the Litestream container.

17 September 2026

Russell Coker: Nheko DBUS

Nheko is my current favourite client for the Matrix IM system, which is my favourite IM system. Matrix is an open system with end to end encryption and Nheko is free software and runs well on Linux desktops and phones. The Nheko client allows interaction with dbus which could be good for automating things, EG you could change the status message when unlocking the screen. I m documenting the most useful ones here because they don t seem to be documented anywhere else. I have filed a Debian bug about the activate room option not working. The qdbus6 program is the QT6 version of the dbus command-line query program, there are a range of other programs which work in much the same way. 
# list all interfaces
qdbus6 im.nheko.Nheko / 
# get the version of Nheko
qdbus6 im.nheko.Nheko / im.nheko.Nheko.nhekoVersion
# list rooms in a dump of the data structures (pity it's not json or something)
qdbus6 --literal im.nheko.Nheko / im.nheko.Nheko.rooms less
# join a room
qdbus6 im.nheko.Nheko / im.nheko.Nheko.joinRoom "#flounder-random:luv.asn.au"
# supposed to activate a room but doesn't
qdbus6 im.nheko.Nheko / im.nheko.Nheko.activateRoom "#flounder-random:luv.asn.au"
# set the status
qdbus6 im.nheko.Nheko / im.nheko.Nheko.setStatusMessage "whatever"
# get the status
qdbus6 im.nheko.Nheko / im.nheko.Nheko.statusMessage
Here are a couple of examples of using other dbus clients to get similar results. Note that the difference between the Debian version of Nheko (and maybe other recent versions) and what LLMs return for usage examples is that Debian has / as the path while the examples have /im/nheko/Nheko . 
# list rooms via gdbus
gdbus call --session --dest im.nheko.Nheko --object-path / --method im.nheko.Nheko.rooms
# get status via dbus-send
dbus-send --session --print-reply --type=method_call --dest=im.nheko.Nheko / im.nheko.Nheko.statusMessage
Related posts:
  1. dbus error Strange dbus error from the KDE power monitoring tool...
  2. Hangouts Replacement Google is currently in the process of killing Hangouts. Last...
  3. Matrix Introduction In 2020 I first setup a Matrix [1] server....

10 September 2026

Raju Devidas: Installing Ubuntu on intel Macbook Pro 2017 with touchbar

Installing Ubuntu on intel Macbook Pro 2017 with touchbarJust some notes about fixing some issues while installing Ubuntu on Macbook Pro 2017

Fix AudioAudio is not working by default after a fresh install.
GitHub - davidjo/snd_hda_macbookpro: Kernel audio driver for Macs with 8409 HDA chip + MAX98706/SSM3515 amps
Kernel audio driver for Macs with 8409 HDA chip + MAX98706/SSM3515 amps - davidjo/snd_hda_macbookpro
johndoe@mac ~> sudo apt install gcc linux-headers-generic make patch wget
johndoe@mac ~> sudo apt install linux-source-7.0.0
johndoe@mac ~/dev> git clone https://github.com/davidjo/snd_hda_macbookpro.git
johndoe@mac ~/dev> cd snd_hda_macbookpro/
johndoe@mac ~/d/snd_hda_macbookpro (master)> sudo ./install.cirrus.driver.sh
johndoe@mac ~/d/snd_hda_macbookpro (master)> sudo reboot

Fix Touchbar

> sudo apt install git dkms build-essential linux-headers-$(uname -r)
> git clone https://github.com/AJ-dev-i60/t1-touchbar.git
> cd t1-touchbar
> sudo ./install.sh
> sudo reboot

Fix Wi-FiWifi actually works out of the box, but the signal strength is usually very bad. We&aposll try to fix that
johndoe@mac ~> cd /tmp
               wget -O brcmfmac43602-pcie.txt \
                     https://raw.githubusercontent.com/jsoyer/MacBookPro14-2/main/firmware/brcm/brcmfmac43602-pcie.txt
--2026-09-10 18:36:39--  https://raw.githubusercontent.com/jsoyer/MacBookPro14-2/main/firmware/brcm/brcmfmac43602-pcie.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.111.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com) 185.199.109.133 :443... failed: Connection timed out.
Connecting to raw.githubusercontent.com (raw.githubusercontent.com) 185.199.111.133 :443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6051 (5.9K) [text/plain]
Saving to:  brcmfmac43602-pcie.txt 
brcmfmac43602-pcie.txt    100%[=====================================>]   5.91K  --.-KB/s    in 0.04s   
2026-09-10 18:38:53 (163 KB/s) -  brcmfmac43602-pcie.txt  saved [6051/6051]
johndoe@mac /tmp> sed -i &aposs/^macaddr=.*/macaddr=<your-wi-fi-cards-mac-id>/&apos brcmfmac43602-pcie.txt
johndoe@mac /tmp> sudo cp /tmp/brcmfmac43602-pcie.txt \
                        "/lib/firmware/brcm/brcmfmac43602-pcie.Apple Inc.-MacBookPro14,2.txt"
johndoe@mac /tmp> sudo ln -sf \
                        "brcmfmac43602-pcie.Apple Inc.-MacBookPro14,2.txt" \
                        /lib/firmware/brcm/brcmfmac43602-pcie.txt
                     
                     
                     
johndoe@mac /tmp> sudo reboot

8 September 2026

Paul Tagliamonte: can you hear me now? good! (Part 11/12)

This post is part of a series called "Pigeon". If this is the first post you've found, it'd be worth reading the intro post first and then looking over all posts in the series.

If you're looking for it, the intro to the Link (layer 2) that this belongs at a high level and listing of the parts that make it up (including this one) is on the link post.
Built-in to the pigeon link protocol is a message type called cal (short for, you guessed it, calibration). A pigeon frame with a type of cal (which is 0x02) carries a JSON encoded payload in the body, which can either be a beacon, requesting signal reports in response, or a report, describing the received beacons. This serves a few interesting purposes firstly, network operators can better understand the coverage footprint, propagation under different conditions, and how gain impacts reception when tuning for the lowest practical power levels. Secondly, this can be used (and I plan to eventually implement!) to construct a mapping of minimum power level and peer mac address to dynamically control the transmission power based on the destination station. That being said, for now, all I ve used this for is getting a rough sense for what gain value(s) make sense between two nodes (manually). In the future, beyond all the fancy neighbor gain stuff, I plan to wire this into the daemon to happen automatically, debouncing for beacon and report messages, such that transmitting stations only beacon, and receiving stations only report a max of once over some time period for a given peer.

Version Given all the above, I do intend to make some massive changes to this protocol (I promise to blog all about it) when I get around to hacking on switching Layer 2 frames within a network segment. Just to avoid having to dig myself out of a hole later, I m going to explicitly send (and check) the version field to avoid having a big flag day switchover or needing to use a new link type.
Version ID Description
V1this version of the cal protocol

Location Both flavors of cal messages (beacon and report) may contain a location, which is the location that the beacon was transmitted, or for or the location where the beacon was heard for a response. This can be used to derive a coverage map and to (operationally) better understand what stations should be within range, and generally what gain level(s) are effective. All fields assume WGS84 latitude and longitude values, and elevation is distance, in meters, above the WGS84 ellipsoid NOT height above sea level, or altitude above the ground.
Field Description
latWGS84 Latitude
lonWGS84 Longitude
elevationheight, in meters above the WGS84 ellipsoid

Sequence Each beacon contains a Sequence identifier, which is used to communicate which message number is being heard, and how many total were transmitted by the originating station. The current approach with Beacon messages is to transmit some number of Beacon messages at different gain levels, each with a unique Sequence identifier.
Field Description
numberbeacon sequence number
totaltotal number of beacons transmitted

Gains Recorded gain setting(s). For a Beacon this indicates the gain settings (which, in spite of its name, includes things like amplifiers, or attenuators). Changing this over different Beacon frames enables a better understanding of what an appropriate gain level is for the transmitting station over time.
Field Description
namegain stage name
dbgain value, in dBm

Cal All messages contained in a cal frame are of this type. The type field communicates if this is a Beacon or Report message type.
Type Description
beaconsent intermittently by idle stations
reportreception report in response to a beacon

Beacon A beacon message may be sent periodically by pigeon nodes capable of transmitting to announce their prescience to peers and, implicitly, to receive signal reports from nearby listeners who are capable and configured to transmit reports. The beacon JSON message is made up of the following fields:
Field Description
versionversion enum value
gainsgains object
sequencesequence number
locationlocation object
An example beacon looks, unsupprisingly, as follows: 
 
 "type": "beacon",
 "version": "V1",
 "sequence":  
 "number": 2,
 "total": 5
  ,
 "gains": []

Report A report message may be sent in response to a beacon message by pigeon nodes capable of receiving and transmitting to assist with setting the lowest usable gain value, and to better understand the area of coverage and propagation. The report JSON message is made up of the following fields:
Field Description
versionversion object
locationlocation object
An example report looks as follows: 
 
 "type": "report",
With all that out of the way lets send some ip

7 September 2026

Vincent Bernat: Sidenotes with CSS anchor positioning

I am a heavy user of sidenotes:1 they keep optional content next to the text instead of sending the reader to the bottom of the page and back. Tufte CSS renders them without JavaScript but only accepts inline content. CSS anchor positioning, now supported by recent browsers,2 is an elegant alternative. Sidenotes can hold several blocks, still without JavaScript, and fall back below the paragraph referencing them on narrow viewports and older browsers. In 2023, Eric Meyer demonstrated this technique in Nuclear Anchored Sidenotes. The main improvement over other solutions is that the notes can sit anywhere in the HTML document. You can place them after the paragraph referencing them, as regular block elements for text browsers, screen readers, feed readers, and reader mode to render them properly:
Sidenotes rendered in Lynx appear after the paragraph they are called from.
Rendering in Lynx, a text browser
When the viewport is too narrow or the browser does not support CSS anchor positioning, you can style them so the reader can skip them or glance at them without losing their position in the text:
Sidenotes rendered on a narrow viewport appear with a distinctive typography after the paragraph they are called from.
Rendering below the paragraph on a narrow viewport
Once the viewport is large enough, they appear in the margin, at the same vertical position as the matching reference mark, unless they would collide with a previous sidenote, as in the example below:3
Sidenotes rendered on a large viewport appear in the margin. There are two of them. The first one is vertically aligned with the matching reference mark, while the second is rendered below as it would collide with the first otherwise.
Rendering in the margin on a large viewport
The gist of CSS anchoring is to position an element relative to another element the anchor. For the sidenotes, the anchor is the reference mark. I use the following markup, with a data attribute to specify the anchor name: 
<sup id="fnref:YYY" data-anchor="--lf-sn-YYY">
  <a href="#sidenote-YYY">1</a>
</sup>
The matching note is an <aside> element carrying the same data attribute for the anchor name. We put it after the paragraph holding the reference mark: 
<aside role="note" id="sidenote-YYY" data-anchor="--lf-sn-YYY">
  <sup>1</sup>
  <p>A first paragraph.</p>
  <p>A second paragraph.</p>
</aside>
On a narrow viewport or when the browser is too old for CSS anchoring, we style the sidenote, which stays below its paragraph, with a muted color: 
aside[role="note"]  
  margin-block: 1rlh;
  color: #444;
On a wide viewport and when the browser is recent enough, we move the sidenote to the right margin: 
@supports (anchor-name: attr(data-anchor type(<custom-ident>)))  
  @media (min-width: 72rem)  
    main  
      position: relative;
      sup[data-anchor]  
        anchor-name: attr(data-anchor type(<custom-ident>));
        /*   anchor-name: --lf-sn-YYY */
       
      aside[role="note"][data-anchor]  
        anchor-name: --lf-sidenote;
        position: absolute;
        position-anchor: attr(data-anchor type(<custom-ident>));
        /*   position-anchor: --lf-sn-YYY */
        top: max(anchor(top), anchor(--lf-sidenote bottom, -1rlh) + 1rlh);
        left: 100%;
        margin: 0 2rem;
        width: 18rem;
        color: inherit;
attr() extracts the anchor name for the reference mark from the data-anchor attribute. It returns a string, unless we specify a CSS unit or a type, like here: the browser parses the data attribute as a custom identifier, which anchor-name validates as a dashed identifier, a custom identifier starting with two dashes.4 The note itself is absolutely positioned past the right edge of the main block. It selects the matching reference mark as its anchor with position-anchor set to the value of the data-anchor attribute. Each note is also an anchor named --lf-sidenote. We use it to keep the next note from colliding with this one. The anchor() CSS function lets us position the note s top edge relative to its anchor: anchor(top) aligns the top edge of the note with the top edge of the reference mark. It can also take another anchor as a parameter: anchor(--lf-sidenote bottom) would align the top edge of the note with the bottom edge of the closest preceding anchor named --lf-sidenote so the previous note.5 Like attr(), anchor() accepts a fallback value as its second parameter and use it when the named anchor does not exist. The top property handles three cases, illustrated in the following diagram:
Diagram of three sidenotes anchored to their reference marks. The first one is aligned with the top of its own reference mark, as no note comes before it. The second one would overlap the first, so it takes the bottom of the first note as anchor and sits one line below it. The third one comes far enough down the page to align with its own reference mark again.
The three cases for the vertical position of a note
  1. The first note s top edge aligns with the top edge of its reference mark: as there is no previous note, anchor(--lf-sidenote bottom, -1rlh) + 1rlh resolves to 0 and max() returns anchor(top).
  2. When the reference mark of a later note sits above the bottom of the previous note, plus some vertical space, the note goes below the previous one to avoid a collision. max() returns anchor(--lf-sidenote bottom) + 1rlh.
  3. Otherwise, the note s top edge aligns with the reference mark s top edge, as max() returns anchor(top).
Have a look at the complete stylesheet, which also adapts the reference mark to the location of the note: a arrow when the note sits below the paragraph, a arrow when it moves to the margin. Gwern s Sidenotes In Web Design lists more implementations and their trade-offs. Some bloggers aim to write a post in 30 minutes. I planned to publish three web-related articles this weekend. Instead, I spent an inordinate amount of time elsewhere: about 15 commits on the build system, a pull request to update CSS highlighting for nested selectors in Pygments, and a small correction to MDN s article on the anchor() CSS function. The SVG illustration took a bit less than an hour and the article itself a handful of hours. The attr() function came in after I thought inline style looks ugly, isn t there a better way? But, hey, I still think this is worth it!
  1. My PhD advisor told me this is unwise.
  2. The first bits of anchor positioning are supported from Chrome 125 (May 2024), Firefox 147 (January 2026), and Safari 26 (September 2025). Before Safari 26.5, sidenotes may collide due to a bug in how dependency chains are handled. You can detect this situation with some JavaScript. It is, however, not needed in the solution described here as we depend on a more recent feature.
  3. If you noticed the runt in the first note, I share your pain and lament that Firefox does not implement text-wrap: pretty.
  4. Typed attr() is supported from Chrome 133 (February 2025), Firefox 155 (September 2026), and Safari 27 (September 2026). Check Una Kravets article for details. To support more browsers, you can inline the anchor name and the position anchor directly in the HTML: 
    <sup id=" " style="anchor-name: --lf-sn- ">
  <a href="#sidenote- ">1</a>
</sup>
    Managing Anchor Associations With Data Attributes and Advanced attr(), by Daniel Schwarz, explores CSS anchors and typed attr() in more detail.
  5. The exact rule for the target anchor element is more complex: if an ancestor of [the note] satisfies the following conditions, return the nearest such element to [the note]. Otherwise, return the last element in tree order that satisfies the conditions. One of these conditions is that [the candidate] is an acceptable anchor element for [the note], which requires that [the candidate] is laid out strictly before [the note], where the relevant clause is that [the candidate] is either not absolutely positioned or occurs earlier in the flat tree order than [the note].

30 August 2026

Ritesh Raj Sarraf: Taming the AI Agents (Part 2): Cross-Vendor Agent-to-Agent (A2A) Swarms over the Software Forge

Preface: The Unanswered Frontier In Part 1: Taming the AI Agents, I shared the architectural blueprint of CAMP (Cross-Agent Memory Protocol) how we used Linux Bubblewrap (bwrap), camp-acpd, OPA policy enforcement, and a central pgvector MemPalace to bring deterministic discipline, sandboxing, and long-term memory to a heterogeneous fleet of AI coding assistants (Claude Code, Google Antigravity, Grok Build, and GitHub Copilot). At the end of that article, however, I highlighted a significant hurdle: The Headless Limitation.
While passive A2A works beautifully for structured handoffs, the current frontier of agentic design faces a key limitation: agents are not yet fully headless-capable. They depend on the active terminal session, browser loop, or prompt loop of the user to keep executing. Because agents cannot run completely detached in the background as daemon processes, we cannot yet achieve active A2A communication
For weeks, this seemed like an insurmountable impasse. Proprietary AI vendors have zero commercial incentive to ratify a universal, open, cross-vendor Agent-to-Agent (A2A) communication protocol. Each vendor builds its own walled garden (Claude s cross-session features, OpenAI s custom ecosystems, etc.). If you wait for the industry to hand you an open interoperability standard, you will wait forever. Then, on August 26, 2026, inspired by Colin Walters article on Agentic AI and software forges and GitHub Agentic Workflows (gh-aw), we had a sudden realization: We don t need a new protocol, a new distributed message broker, or permission from proprietary AI vendors. We already have the universal, decentralized communication bus that software engineers have relied on for decades: the software forge itself. Over the span of 48 intensive hours (from RFC #788 through milestones M1 to M3 and live dogfooding on #813), we designed, implemented, fortified, and verified fully autonomous, headless, cross-vendor Agent-to-Agent swarms running over a local Gitea forge. Here is how we did it, the architectural hurdles we solved, and why this changes the game for autonomous software engineering.

1. The Core Realization: The Forge is the Bus When people think about multi-agent swarms, they often imagine complex distributed RPC frameworks, microservices exchanging ephemeral JSON-RPC blobs, or bespoke socket daemons. In practice, this approach suffers from major flaws:
  1. No shared context or durable audit trail: Transient network packets vanish unless heavily logged.
  2. Proprietary CLI fragmentation: Different vendor tools (Claude CLI, Antigravity CLI, Grok CLI, Copilot CLI) do not speak the same internal language.
  3. Loss of human visibility: When agents talk over private network channels, human operators lose the ability to inspect, pause, or audit the conversation.
By flipping the paradigm and making the software forge (Gitea) the primary communication channel, everything falls naturally into place:
  • Issues and Pull Requests are the shared state: The issue description and discussion thread form the canonical, append-only conversation log.
  • @mentions are the dispatch triggers: When an agent (or human) writes @grok Please review this PR in a comment, Gitea fires a standard webhook (issue_comment).
  • Webhooks provide unforgeable authentication: The webhook payload contains the cryptographically verified sender identity. An agent cannot spoof another agent s identity by merely typing their name in text.
  • Every CLI already supports non-interactive prompt mode: The CLIs don t even agree on the command-line flag Claude uses -p, Grok uses -p, Antigravity uses --print, Copilot uses --prompt. But they all agree on the essential contract: Take a prompt string, execute tools, print output, and exit. 
          Gitea Webhook           
  Gitea Forge     >   camp-a2a-bridge.py    
  (localhost)     (issue_comment / assignment)    (Validates & Files)   
                                  
                                                            
                                                            
         Writes comment / review                 
         via camp_acp_gateway                     A2A Inbox Ledger      
                                                 
                              
  Fortified Headless Agent                                  
  (bwrap + OPA + MCP sandbox)   <   
     Claude Code (-p)            Spawn PID           camp-a2a-dispatcher   
     Grok Build (-p)             (Cold or Resume)    (Enforces Hop Cap,    
     Antigravity (--print)                            Rule 1/2, Sandbox)

2. Proving Fortified Headless Execution Before opening the floodgates to background agent dispatch, we had to answer a critical security question: Does a non-interactive, headless agent run with the same strict security sandboxing, audit logging, and tool rails as an interactive session? On August 26, we probed all fleet launchers on the host with a baseline check: 'Call camp_startup_check and print its result verbatim, then exit.' The results settled the question immediately:
  • Antigravity (agy --print / KIR): PASS Gateway answered, full JSON returned.
  • Grok (grok -p / GRK): PASS Gateway answered.
  • Claude Code (claude -p / CLD): PASS Gateway answered.
  • GitHub Copilot CLI (copilot --prompt / CPL): Initially held on TTY tool consent; later unlocked in Milestone 6 via --allow-all-tools --session-id=<uuid>.
  • Audit Trail: Consecutive audit IDs were recorded in the central ledger: 4574 (KIR), 4575 (GRK), 4576 (CLD).
This proved that a headless run through our fortified pilot launcher (camp_pilot_*.sh) is a first-class, fully audited, sandboxed CAMP agent running inside its Bubblewrap container under OPA policy gates. It is not an unconstrained background script or a degraded bypass.

3. The 3-Tier Memory Architecture A naive multi-agent dispatch has an immediate flaw: Every time an agent is invoked, it starts from a blank slate (cold start). If @claude tags @grok to review code, and @grok replies asking for clarification, @claude s second invocation would normally forget everything it did 5 minutes ago, forcing it to burn thousands of tokens re-reading the entire git history from scratch. To solve this, we established a clean 3-Tier Memory Model: 
 
                         3-TIER MEMORY MODEL                              
 
  Tier 1: CLI Conversation Session (Working Memory)                       
      Per-(Agent, Repo, Issue) mapping in a2a-sessions.json               
      Fast, native, compacted context across multi-turn pokes             
      Resumed via --resume (CLD), -r (GRK), --conversation (agy)          
 
  Tier 2: The Gitea Thread (Public Bus & Record)                          
      Cross-vendor shared truth across Claude, Grok, Antigravity & Human  
      Survives process restarts, machine reboots, and dead sessions       
 
  Tier 3: Central MemPalace (Durable Long-Term Knowledge)                 
      pgvector database (17,000+ drawers across agent wings)              
      Structured Knowledge Graph (mempalace_kg_*) for mutable facts       
      Attributed AAAK dialect queryable by any agent across any project

The BANANA Two-Shot Test To verify Tier 1 working memory persistence across independent processes, we designed a simple two-shot host test:
  1. Shot 1 (Create): Dispatch agent headlessly: Remember the token BANANA-M2. Print ok and exit. Capture the vendor s session UUID.
  2. Shot 2 (Resume): Spawn a completely new operating system process with the resume flag pointing to that UUID: What token did I ask you to remember?
Every agent CLI passed with flying colors:
  • Grok: -r 01a03ecc-3ed0-71e1-9a5c-e098bb29ba10 answered BANANA-GRK.
  • Claude: --resume 0a587733-9aec-43c5-9cb7-d424e95b2c5b answered BANANA-CLD.
  • Antigravity: --conversation 2e3c43d9-d6fe-4c5c-801b-b9ceb2e7e196 answered BANANA-KIR-JSON.
  • Copilot: --session-id <uuid> verified in Milestone 6 (DoD #820).
The dispatcher simply maintains a lightweight JSON mapping ((agent, repo, issue_number) -> vendor_session_uuid). On the first poke of an issue, it creates and saves the session ID; on any subsequent poke on that same issue, it resumes the exact same conversational thread!

4. The Engineering Milestones: From Concept to Production Building this system required solving several subtle, real-world friction points across multiple agent CLI implementations. Under the guidance of our plan of record (RFC #788), we delivered this through four focused milestones:

Milestone 1 & 1.1: Reliable Headless Spawning
  • PR #797 (M1): Configured the dispatcher launch table for all probed CLIs with JSON output formatting.
  • PR #800 (M1.1): Eliminated the queue-behind-live-session anti-pattern. Originally, if a human had a Claude or Grok TUI open on their desktop, the dispatcher would defer incoming tasks so as not to collide with the live session. We realized that headless tasks must be independent: every Gitea mention spawns an isolated, sandboxed background process tied to that specific issue, allowing concurrent headless work while the human works in their interactive TUI.
  • PR #803 (M1.2): Standardized command-line argument parsing for Antigravity (agy --print <prompt> --output-format json).

Milestone 2: Session-per-Issue Working Memory
  • PR #805 (M2): Implemented a2a-sessions.json to store and resume vendor session UUIDs. If a resume fails (e.g. session purged upstream), the dispatcher gracefully falls back to a clean cold start without failing the task.

Milestone 3: Cross-Agent Hops & Crucial Safety Rails
  • PR #807 (M3): Enabled agent-to-agent dispatch (Rule 2 reversal). Previously, only mentions authored by rrs (the human) would trigger execution. With M3, an authenticated comment from @claude mentioning @grok triggers Grok s headless launcher.
  • PR #811 (M3.1): Set --permission-mode bypassPermissions for headless Claude Code so non-interactive runs execute tool calls without stalling on TTY prompts.
  • PR #812 (M3.2): Restricted agent summon parsing to line-initial @login tokens with a non-empty task description (#810), preventing accidental dispatches from passive conversational references.

Milestone 4: Directives, Specification & Living Documentation
  • PR #815 (M4): Aligned CAMP fleet directives, architecture specifications, and user documentation with the live A2A implementation.

Milestone 5: Concurrent Dispatching & Hop-Cap Attribution
  • PR #816: Stamped hop-cap notices under a dedicated system bridge identity and automatically applied the needs-human label on held threads.
  • PR #817 (Threaded Scheduler): Replaced the single-threaded serial dispatcher with a concurrent thread-pool scheduler (#804). Multi-agent dispatches across different issues now execute concurrently in parallel background threads instead of queuing behind long-running tasks.

Milestone 6: Full Fleet Coverage with GitHub Copilot
  • PR #819 (M6): Brought GitHub Copilot CLI into the headless A2A fleet (#818). By passing --allow-all-tools and pinning minted session UUIDs (--session-id=<uuid>), Copilot achieved full parity with Claude, Grok, and Antigravity, completing 100% headless fleet coverage across all four major AI coding assistants.

5. Hard Safety Rails: Preventing Autonomous Runaway Loops Letting AI agents autonomously invoke each other in background loops without a human watching is a recipe for an infinite, credit-draining token fire. We put four non-negotiable safety guardrails in place:

Guardrail 1: The Strict Hop Cap The dispatcher tracks hops per (repo, issue). Each agent-to-agent dispatch increments the counter.
  • Hop Limit = 3: A typical review round-trip is 2 hops (Human $\rightarrow$ Claude $\rightarrow$ Grok $\rightarrow$ Claude).
  • Automatic Halt on Hop 4: If agents attempt a 4th autonomous hop without human participation, the bridge refuses to launch, posts a diagnostic notice to the thread: [camp-a2a-bridge] hop cap reached (3 agent-to-agent dispatches on CAMP/camp-infrastructure#813) not launching GRK for claude's mention, and holds execution until the human (rrs) provides input or resets the count.
[ Human: rrs ]   (Cold Start)  > [ @Claude ]
                                                
                                       (Hop 1)   @grok please review
                                                
                                          [ @Grok ]
                                                
                       (Hop 2: Resume)           @claude I reviewed
                                                
                                         [ @Claude ]
                                                
                                       (Hop 3)   @grok ack hop 4
                                                
                                   
                                     DISPATCHER HOP CAP: 3   
                                      *** BLOCKED & HELD *** 
                                      Awaiting Human Reset

Guardrail 2: Deliberate Summon Parsing (M3.2, #810 / PR #812) In human conversation, we often reference colleagues in passing: I will talk to @claude about this later or See @grok s table above . Early prototypes treated any appearance of @agent as a dispatch trigger, causing accidental, unwanted agent launches! We instituted a strict Summon Predicate: For fleet agents, a mention is only considered an actionable summon if:
  1. The @login appears as the starting word of a line (optionally preceded by markdown list markers *, -, or >).
  2. It is immediately followed by whitespace and a non-empty task description.
Mid-sentence mentions in discussion paragraphs are parsed as passive conversational text and never trigger background dispatches.

Guardrail 3: Headless Tool Permissions without Weakening Security (M3.1, #809 / PR #811) In interactive mode, Claude Code presents interactive TTY prompts asking the user to approve MCP tool calls (such as camp_pr_get or camp_pr_get_diff). In unattended headless mode, there is no TTY, causing the run to fail with permission errors. To fix this, we configured --permission-mode bypassPermissions for Claude s headless CLI invocation. Crucially, this only bypasses Claude s internal TTY UI prompt it does not bypass CAMP s security rails. All command executions still route through camp-acpd and Bubblewrap namespaces; OPA policy checks remain active; and privileged operations (such as merging pull requests or restarting system services) still trigger desktop Zenity HITL dialogs on the human s workstation.

Guardrail 4: The Sovereign Kill Switch (Rule 1) At any point, the human operator can instantly freeze all background agent dispatches across the entire infrastructure with a single standard Linux command: 
systemctl --user stop camp-a2a-active.service

6. Live Dogfooding: Watching the Swarm in Action On August 27, 2026, on issue CAMP/camp-infrastructure#813, we ran the full live dogfood test of the A2A system. Here is the exact transcript of what took place entirely autonomously in the Gitea issue thread:
  1. Human initiates: rrs comments on issue #813:
    @claude Please start the M3 dogfood. On this issue, post a conversation comment whose first line is exactly: @grok Please review this issue in one short paragraph, then stop.
  2. Claude launches (Cold start): The dispatcher spins up @claude headlessly inside bwrap. Claude reads the issue and posts:
    @grok Please review this issue in one short paragraph, then stop.
    (M3 dogfood step 2 posted by @claude on rrs s instruction; this is the CLD $\rightarrow$ GRK fleet hop.)
  3. Grok launches (Cold start, Hop 1): The bridge intercepts Claude s comment, identifies the start-of-line @grok summon, and launches @grok. Grok reviews the issue and replies:
    @claude I have reviewed this issue. Please confirm and stop.
    #813 is a throwaway live dogfood of A2A M3 on 6.git+b78c65cf
  4. Claude resumes (Session Resume, Hop 2): The bridge intercepts Grok s comment, sees the @claude summon, and launches Claude with --resume b398d048-ccc3-4cce-8437-4cf8748881a8. Claude immediately remembers its earlier context without token waste:
    Confirmed Grok's review (comment 88562) received via the GRK CLD fleet hop. That completes the round trip: rrs CLD, CLD GRK, GRK CLD, and this reply. I am deliberately not tagging anyone here, so the chain stops at two agent hops. Stopping.
  5. Testing the Hop Cap (Hop 3 $\rightarrow$ 4): grok prompts @claude Please ack hop four then stop. Claude posts @grok Please ack hop four then stop. (Hop 3). When Grok would have been launched for Hop 4, the dispatcher intervenes:
    [camp-a2a-bridge] hop cap reached (3 agent-to-agent dispatches on CAMP/camp-infrastructure#813) not launching GRK for claude's mention.
  6. Human Reset & Multi-Agent Wrap-up: rrs steps in, resets the state, and asks the fleet for final status. In parallel:
    • @grok delivers a closure scorecard.
    • @claude confirms session continuity and M3.2 summon filtering.
    • @priyasi (Antigravity CLI) runs automated ACP checks: 44/44 test suite passing, 17,219 MemPalace vector drawers active, zero spec drift.
    • @agrickxy (Antigravity CLI) provides comprehensive infrastructure impression analysis.
    • @kiran (Antigravity CLI) is summoned headlessly to draft this very blog post!

7. The Ergonomic Breakthrough: The Forge as the Unified Mindmap & Interface Beyond backend plumbing and sandboxing, routing agent interaction through Gitea fundamentally revolutionizes the developer experience of managing an AI fleet.

The Mindmap Mental Model: Threaded Conversations & Forking Tasks In traditional CLI tools, conversations are constrained to a single, linear terminal scrollback. When an agent discovers multiple sub-problems, exploring them sequentially in one prompt loop rapidly pollutes the context window and confuses the model. Using the forge as the communication gateway naturally unlocks a mindmap mental model:
  • Forking sub-threads: Complex problems can be split into dedicated child issues or threaded PR reviews.
  • Focused execution scopes: An agent can be summoned to solve a narrow sub-task in its own issue thread without derailing the parent architectural discussion.
  • Structured problem decomposition: The forge issue hierarchy maps 1:1 to the developer s mental map of the project.

Eliminating Terminal UI Fragmentation Anyone using multiple AI coding assistants on a daily basis quickly grows exhausted by their jarring terminal UI differences: differing ANSI escape rendering, inconsistent markdown wrapping, erratic diff pagers, and incompatible keybindings across Claude, Grok, and Antigravity. Gitea homogenizes the entire fleet under a single, polished rich-text web view:
  • Syntax-highlighted code blocks and visual side-by-side git diffs.
  • Clear author badges attributing each contribution to its exact agent identity (@claude, @grok, @priyasi, @kiran).
  • Collapsible <details> blocks for voluminous diagnostic outputs.
  • Interactive task lists and markdown tables.

Effortless Context Retrieval, Archival & Data Retention Auditing past agent decisions in terminal logs or ephemeral chat histories is notoriously difficult. With the forge, every exchange is:
  • Contextually bound: Pinned directly to the repository, branch, and commit SHA being modified.
  • Organized & Archival-Grade: Full-text searchable with clear milestone and issue tags.
  • Topic-Focused: The human operator can review the complete lifecycle of a discussion in seconds, gaining a rapid, holistic grasp on the entire subject.
Reading back through past agent interactions becomes a breeze to the point where interacting via the intermediary Gitea interface becomes far more pleasant and productive than wrestling with multiple desktop CLI terminals.

Remote Connectivity & Headless Agent Farm Management Because Gitea provides a standard web and API interface, you are no longer chained to the workstation running the agent processes:
  • Monitor progress and dispatch tasks from a mobile browser, tablet, or remote laptop.
  • Queue review tasks on the go without requiring active SSH sessions or terminal multiplexers.
  • The local agent farm continues working silently in its sandboxed daemon containers.

Quietly Achieving the Holy Grail: Live Cross-Vendor Swarms For years, the AI industry has treated cross-vendor multi-agent interoperability as an elusive dream waiting for industry-wide API standardization. By recognizing the software forge as the universal message bus, we quietly achieved live, production-grade, cross-vendor communication across completely distinct vendor models.

8. What This Means for the Future of Agentic AI This milestone marks a fundamental shift in how we interact with autonomous AI systems:
  1. Heterogeneous Agent Specialization: We don t have to choose a single winner among AI models. We can task Claude Code with architectural refactoring, summon Grok Build for rapid verification and adversarial PR reviews, and deploy Google Antigravity agents for codebase exploration and documentation drafting all coordinating fluidly in the same PR thread.
  2. True Human Sovereignty: The human developer is no longer a bottleneck typist or a passive spectator. You act as the Engineering Manager / Lead Architect. You set the requirements on an issue, tag the lead agent, and let the agents iterate, review, and test among themselves in the thread while hard hop caps, OPA policies, and Zenity HITL gates guarantee that no agent merges code or pushes upstream without your explicit sign-off.
  3. No Vendor Lock-In: Because the entire coordination fabric is built on standard Git, HTTP webhooks, local Linux container sandboxes (bwrap), and open MCP tools, any new AI CLI tool released tomorrow can be plugged into our fleet in under 15 minutes by simply adding its command-line prompt flag to the launch table.
We have moved beyond static autocomplete and interactive chat widgets. The software forge is now an active, living, collaborative workspace where humans and autonomous AI agents engineer software together.

9. Video Demonstration: CAMP Forge A2A Swarm in Action Below is a video demonstration showcasing autonomous multi-agent communication, cross-vendor relay, and headless swarm coordination in action via the CAMP Forge interface:
The Cross-Agent Memory Protocol (CAMP) and MemPalace are developed as part of our ongoing research into secure, sovereign, and disciplined Agentic AI computing.

25 August 2026

Antoine Beaupr : A more nuanced view of LLMs

Also in this series:
After ranting and railing about LLMs or "AI" as the optimists (or accelerationists?) call it, I figured it might be important to be a little more honest about my use of LLMs and how I think about it more practically in the world.

The Debian vote context This is not a coming out. I am not using LLMs on a daily basis, and this blog is, again, written out of my cold dead hands in a dying world, with over-engineered hardware and (to a certain extent, hi Emacs!) software, powered by 100% green energy built on stolen land. There is a vote going on in Debian. If you're unfamiliar with it, you can catch up at LWN. So far I've essentially said "LLM is bad" which is not a very balanced or useful opinion. Obviously, people are using LLMs, sometimes unknowing or unwillingly, and we need to take that into account. Furthermore, there has been many different blog posts on Debian planet about this. Some that I found balanced, good summaries, even if I didn't fully agree with them, at least some did the basic civil service of being short. But others were just not only Wrong but also so long that I couldn't finish that I just had to write something.1 This is not an explanation of the ballots, nor how I will vote. This vote is Debian's failure of framing that debate in a reasonable way: we have 8 options on the ballot with many duplicates. We have failed to do the hard work of summarizing and aggregating options into a meaningful set. I doubt the final vote will represent a readable position we can rally around. I have not read the two months of debates on the topic either. Normally, before voting, I take a cursory look at the debate to see points of view I might have missed. But in this case, it will just make me sad, add noise, and I'm already pretty sure on where I stand on this. So let me describe how I use LLMs and how I think they fit in our work, as computer engineers and hobbyists.

My LLM use

Debian Packaging An astute reader has pointed out that I maintain a package in Debian made to use Anthropic. It's actually multiple packages: As I previously explained in response, I am not entirely comfortable with this work: it's a compromise. In fact, I first uploaded llm to the contrib section of Debian, where we keep software that depends on other non-free software, but I was told that, since yt-dlp was in main, llm belonged there as well. So I moved it to main, alongside similarly controversial tools like llama.cpp or the python-openai library.

OpenAI and Anthropic usage An important part of my work is technology watch. I keep tabs on thousands of (new and old) software projects, follow news, and generally try to keep my skills up to date. It's a pretty impossible race, especially as I grow older, but I still think I'm doing the right choices in my job. Testing large language models is part of that work. At first, I was using ChatGPT's web interface, but it was annoying to copy-paste things into a browser, so I looked for different interfaces. For a while I tried gptel, a "simple, extensible LLM client for Emacs" but I found it kind of terrifying. Giving a LLM control over an Emacs buffer seems like a security nightmare, so I stopped doing that. So I use the llm command-line tool to talk to Anthropic's API. I started that in the summer of 2025, when I bought 20$USD of API credits. Before that, I paid for a ChatGPT subscription and then OpenAI credits, which expired and sent me over to Anthropic, which seemed then to have better ethics. As it turns out, Anthropic is also happy to work for the US military (which is a big red line for me). Anthropic also won't let you talk about the genocide in Gaza, it is destroying physical books, and is blackmailing us to use their product for security coverage. Needless to say, Anthropic and "Claude" are not my friends, but they seem like the lesser evil in current "frontier models". So I have renewed, a couple of weeks ago, another 20$USD of API credits with Anthropic.

Actual prompts and responses So what does 20$ give you at Anthropic anyways? What am I using LLMs for and how? The neat thing with llm is that everything is logged in a sqlite database, so there are some answers that are easy to get: 
> llm logs status
Logging is ON for all prompts
Found log database at /home/anarcat/.config/io.datasette.llm/logs.db
Number of threads logged:   7
Number of turns logged:     12
Number of legacy conversations: 543
Number of legacy responses: 970
Database file size:         9.61MB
That is 10MB of logs, with about a thousand prompts. My logs go back to 2024-03-07, a little over two years ago, and include a mix of Anthropic and OpenAI responses. I used it more in 2024 than 2025, and if the trend continues, I will have used it less in 2026 again: 
> llm logs list -n 0  --json   jq -r .[].datetime_utc   sed 's/-.*//'   sort   uniq -c 
    527 2024
    357 2025
     98 2026
It looks like about 10 prompts per month right now, down from a peak of about 60 per month in 2024. It's pretty difficult to analyze those actual logs to get more patterns and I won't run the prompts through a model again to process them.

How I'm using models now At first, I was using it partly for benchmarking model's capabilities, like Simon Willison does with his pelicans, clearly not trusting its output. But I was impressed by the capacities of the Claude Opus 4.5 model when it wrote this script in January. Impressed, but also scared: it's the first time I felt I could delegate the entirety of my programming to a model. Just run the code, if it works, it works, right? So what do I use it now? As an example, here are the 10 last prompts in my history:
  1. there is now Claude 5, and a fable model, maybe you know about it?
  2. impress me
  3. not impressive, i already know all of this
  4. chat
  5. in postfix, i have a 300k mailing that happens regularly here. normally, it delivers within about...
  6. is there a way i could have drained the maildrop queue faster without removing the milter?
  7. the problem was that rspamd was timing out on the FUZZY_CALLBACK check. how do i disable that?
  8. how do i disable all spam checks? i just want rspamd to add dkim signatures
  9. how do the default_destination_concurrency_limit and initial_destination_concurrency settings int...
  10. mic check
The first one was me trying to confirm which model I am using, which is not always obvious when going through the whole llm stack I've been using. The following two are an attempt at seeing what the model is capable of and I was "not impressed", to which Claude answered that I have a "high bar", which, fair enough. The chat is me failing to use a command line, which shows that perhaps I need to readjust that "high bar", again. The next five are a rather embarrassing debacle in a large Postfix mailing that went sideways, and where I couldn't find an actual Postfix expert of my level to help. The fabled Claude Fable 5 answered rather correctly, but dangerously, that I could empty the queue by disabling the non_smtpd_milters. What Fable (and myself) did not realize is that the milter was also adding DKIM signatures, so while the mailing was expedited, it was done without those precious signatures, which got us promptly blocked at Gmail. We have recovered since, and, thanks to the model and reading the Postfix manual for the hundredth time, that pickup(8) is single-threaded and that we needed to review the architecture of that mailing (and our spam filters) a bit. Many tickets ensued. The last one is a test I did to make sure my last uploads of llm-anthropic and its dependency worked correctly.
Note that the above excludes 5 questions I asked Anthropic while writing this article, where I asked for synonyms and "what nanometer scale are arduino processors built from? how is an arduino CPU printed?", a question which Wikipedia furiously evades providing a good answer.
Those prompts are pretty typical of my LLM use: I'm testing the models to see if they work at all, but also, out of desperation, I fire off a prompt after I fire off questions to colleagues or search engines (in that order). It's often weird edge cases like the Prometheus query language, Python's matplotlib, LaTeX, Elisp, optimizations, and so on. I use models for translation a lot. Being fully bilingual, it is common for me to think of a word in French or English and fail to find exactly the right word for that in the other language. Models help with that, and are also useful to find synonyms. Those are low-token uses that seem pretty innocuous to me, but I realize the irony of this after writing about the tower of Babel.

What I am not using models for I am not using models to write prose. I am not using models to read prose. If it's generated with LLMs, I stop reading. I am not using models to write code, with the exception of that single Python script above. I am generally not using models to review code, with exceptions. If I get stuck on a hard problem, I might feed a piece of code to the model. I repeatedly fed asncounter into Claude to try to fix a performance regression I had introduced. It found micro-optimizations that taught me a thing or two about Python's internal implementations, but overall, it was mostly a waste of time. This was in June 2025, so perhaps now models would fare better. I have not tried again. I am not using LLMs to do Debian packaging. When I can, I manually review the diffs of packages I upload into Debian, still, by hand. I do this for the reasons outlined in The Four Horsemen of the LLM Apocalypse, because I refuse to be complicit in the:
  1. aggressive and illegal scraping of the servers I steward
  2. world-wide computer hardware shortage (making it, by the way, nearly impossible to run presumably clean local models) and the attack on our job conditions (also discussed in The people vs the AI overlords)
  3. death of copyright and free software
  4. complication and enshifitication of everything, and the destruction of our communities
  5. the imperialist Nerd Reich that wants to take over the world
Like I reluctantly use Intel computers, I do fire off a prompt. But I still hold on to the dream that we can build communities of practice that hold human knowledge collectively and not offload that as a utility to some megalomaniac billionaire.

Their LLM use I am forced into So that's me. Clearly, I'm going against the grain here. Everywhere I look, I see LLM-generated code and projects. Slop and botnets have flooded the web. I use Wadamesh, clearly vibe-coded, because it's the best graphical interface for MeshCore that runs on portable devices. I wish it was made by a human, in a community I could participate in, but it isn't, and I don't. I package the above llm toolset, which is more and more vibe-coded, but I still review the diffs. And I have to say: I trust Simon here. The code is verbose as hell, feels overengineered, and llm feels slow, but it generally works, and Simon is still at the gate. The Anthropic SDK is another thing entirely. The 0.91.0 to 0.120 upload, for example, was nuts: 
 806 files changed, 72281 insertions(+), 1478 deletions(-)
I explicitly did not review that entire diff. It feels like there's a lot of garbage there to just have a shim between a proprietary API and Python. But this is the hand I've been dealt.

Larger projects LLM use LLMs are being used in the Linux kernel, Firefox, rsync, Rust, and other places. I don't feel good about this, particularly in Rust, but they at least made a decent policy. I am glad GCC made a policy against LLM contributions and I support the human Emacs project. We need to have a set of foundational tools that are "clean" in the sense that they are built upon a community of people that understand how they are built. Maybe that's naive or even impossible. The Linux kernel and GCC, in particular, are massive projects that have long grown past the scale of a single person's understanding. But the theory was that a community of humans can understand collectively. Now we seem to be throwing up our hands and giving up on that community. That LLMs will just fix the problem, whatever it is. But we're all just one rug pull away from being completely incapable of managing those projects. The argument there is that we'll just switch to local models, but no one is actually doing that. All I see is people use local models as a corner case (for privacy) or as in theory, but in reality, everyone uses the centralized frontier models right now. We just can't fallback. We're in the same situation we were, a decade or two ago, when Microsoft decided it would kill free office alternatives by making Office free for non-profits. It worked: thousands, if not millions of schools, community groups and individuals stopped looking for alternatives (including free software but also "piracy") for Office and embraced what seemed like a generous offer. Now Microsoft pulled the plug and Over 170,000 Nonprofits Lost All Their Data. I'm afraid the rug pull on LLMs will be much worse: never mind that Linus won't be able to use his tireless helper to fix obscure kernel bugs; we're looking at a collapse of the economy so large that we are already talking about bailing out the companies responsible. In a sense, the most striking thing about the Debian vote is it has actually no option to completely refuse upstream LLM contributions. It seems the community has taken it for granted that it's now impossible to build Debian entirely without LLMs. We lost the battle even without a fight, it seems.

A plea for small If it has really become impossible for us to manage the complexity we have built, maybe it's time to stop and think about what we're doing in the first place. We're struggling to even bootstrap our current toolchain! This is one of the things I like the most about working on the mesh: it's low tech, small Arduino devices that is built with decades-old semiconductor processes that is understandable by human beings. Maybe the answer lies more in single-purpose devices like those communicators and simpler multi-purpose computers than what we have now, which is what the permacomputing movement is about. Small is beautiful, let's scale it down.
  1. and yes, I'm sorry this has gotten this long, I hope you will forgive those 3000 words.

Next.