Search Results: "jan"

14 October 2026

Reproducible Builds: Reproducible Builds in September 2026

Welcome to the September 2026 report from the Reproducible Builds project! In our reports, we try to outline the most important things that we have been up to over the past month. As a quick recap about what problem our project intends to solve, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised or not. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website. In this month s report, we cover:

  1. Reproducible Builds Summit 2026 held in Gothenburg, Sweden
  2. dettrace expanding architecture support for PGO & Lisp dumps
  3. They don t care about this!
  4. SOURCE_DATE_EPOCH specification updates
  5. Reproducible Builds at SeaGL 2026
  6. New Code of Conduct (CoC)
  7. Supporter Spotlight: Jochen Sprickerhof
  8. Mailing list news
  9. Tool and documentation development
  10. Patches

Reproducible Builds Summit 2026 held in Gothenburg, Sweden The annual Reproducible Builds summit took place from 22 to 24 September 2026 in Gothenburg, Sweden. The event brought together developers, distribution maintainers, security researchers and free software advocates from across the ecosystem to share updates on current initiatives, address outstanding toolchain bottlenecks and strengthen cross-project collaboration. Participants engaged in a series of working sessions and technical roundtables focused on expanding the practical scope of reproducibility. Discussions covered ongoing work within major Linux distributions, language-specific package ecosystems and the various parts of the binary verification infrastructure. Strategic planning formed a pillar of the gathering, with participants discussing long-term roadmap goals for supply-chain security, software integrity and related topics. Attendees explored new methods for automated verification, discussing how to lower the adoption barriers for maintainers and streamline the integration of deterministic build pipelines into established continuous integration (CI) systems. The event was then followed by two days of unmoderated hacking sessions. These hacking sessions provided space for technical progress, allowing attendees to draft specification updates, prototype new tool features and resolve standing reproducibility bugs in-person.

Dettrace expanding architecture support for PGO & Lisp dumps Bernhard M. Wiedemann announced major updates to dettrace, porting the tool across most architectures supported by Debian and openSUSE. The release addresses key stability issues, passing test suites across these targets and making it ready for production build pipelines. These improvements enable deterministic builds when utilising Profile-Guided Optimisations (PGO), removing the need to sacrifice runtime performance for reproducibility. Additionally, dettrace can now be applied to (notoriously difficult-to-reproduce) Lisp and Scheme memory dumps, including environments such as CLISP, SBCL, Scheme 48, Racket and GNU Emacs.

They don t care about this Annika Wilde, Marco Gutfleisch, Felix Reichmann, Anirban Chakraborty, Yuval Yarom, M. Angela Sasse and Ghassan Karame published a paper this month entitled They don t care about this : A Systematic Study of Trusted Execution Environments Build Reproducibility in the Wild:
Trusted Execution Environments (TEEs) have become a cornerstone of modern cloud computing, providing strong confidentiality and integrity guarantees for both code and data. A critical component of this trust model is remote attestation, which enables external entities to verify the authenticity and integrity of code executing within a TEE through cryptographic measurements. However, the effectiveness of remote attestation fundamentally depends on the verifier s ability to trace the reported measurement back to the original source code - a property that can only be guaranteed through reproducible builds. [ ]
The full PDF can be read online.

SOURCE_DATE_EPOCH specification updates Bernhard M. Wiedemann shared a number of proposals regarding updating the SOURCE_DATE_EPOCH specification following discussions at a project summit. Seven distinct proposed changes have been submitted as merge requests on Salsa. Bernhard s proposed modifications are independent of one another, allowing for any combination to be merged depending on community feedback. Bernhard noted that one proposal may see the most debate, and could still benefit from further refinement before reaching consensus. Community members are invited to review and provide feedback on the individual merge requests by 1 December 2026. Input can be submitted via reactions or comments on Salsa, directly by email to Bernhard or the mailing list. The specification was first published in 2015 and updated in 2017.

Reproducible Builds at SeaGL 2026 Reproducible Builds will be present at SeaGL in Seattle, WA, on October 23rd & 24th. Come chat with us in person! Founded in 2013, SeaGL is a free, grassroots technical summit dedicated to spreading awareness and knowledge about free source software, hardware and culture.

New Code of Conduct (CoC) for the Reproducible Builds project Mattia Rizzolo announced the official launch of a Code of Conduct for the Reproducible Builds project. This follows work and community discussions originally initiated in 2024, incorporating feedback gathered over email and during project summits, as well as with input from the Software Freedom Conservancy. The new Code of Conduct establishes guidelines emphasising mutual respect among project contributors. It also defines a formalised process for submitting and handling complaints regarding community misconduct, ensuring transparent governance across project spaces. Along with the core policy document, an accompanying Code of Conduct FAQ has been published to outline the rationale behind adopting the guidelines.

Supporter Spotlight: Jochen Sprickerhof As a reminder, another supporter spotlight interview was published featuring long-time Debian and Reproducible Builds contributor Jochen Sprickerhof, highlighting his work on package verification and project infrastructure.

Mailing list news On our mailing list this month:
  • Jonathan Starr from the Institute of Open Science Practices (IOSP) posted with an announcement of its upcoming 2026 workshop series in Leiden in the Netherlands. This free-to-attend event aims to build a participant-owned data storage consortium, allowing community members to publish datasets, code and knowledge objects alongside algorithms designed to evaluate trust and value. The event features a dedicated Community Track for talks and discussions, a tool showcase and an opening session on resilient data storage.
  • Following up on IRC discussions regarding profile-guided optimization (PGO) on build servers, kpcyrd created a demonstration repository showing how PGO profiling can introduce non-determinism into output binaries. The demo highlights how runtime randomness (specifically, the randomised keys used in HashMaps to prevent denial-of-service attacks) causes varying bucket collision rates during test runs. These structural variances alter execution weights on underlying code branches, prompting the compiler to generate differing final assembly output for builds that would otherwise be perfectly reproducible if PGO was disabled. Rather than abandoning PGO performance gains, kpcyrd suggested treating generated profiling data as a documented build input or recorded environment artifact (similar to .buildinfo files), rather than regenerating profiles dynamically on the fly during build execution.
  • An autonomous AI agent (!), posted a controlled build result and reproducer demonstrating that pre-creating an obj directory in libmilter source trees prevents kernel-release path leaks (obj.Linux.<release>.x86_64) into debugging symbols. kpcyrd followed up with a helpful reply.
  • Alexander Sosedkin inquired on the mailing list about the preferred approach for sourcing pinned translation files to create reproducible GnuTLS release tarballs, asking whether external hosting should provide versioned downloads, or if projects ought to use Git submodules, dedicated side-repositories or some other mechanism.

Tool and documentation development diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes, including preparing and uploading versions 330, 331 and 332 to Debian. In particular, Chris committed support for radare2 versions greater than 5.9.0 [ ] based on input from Guillaume Girol [ ], and committed support for .nar files via Guix. [ ][ ]. Chris later modified diffoscope to not Build-Depend on the Debian package of GNU Guix, as its status in the Debian bug tracking system would transitively render diffoscope as being overly buggy for the next stable release [ ]. Similarly, he adjusted the Build-Depends on the apksigcopier package as it has been removed from Debian testing [ ] and, finally, he updated a number of copyright years [ ][ ]. Vagrant Cascadian updated diffoscope in GNU Guix to version 332.
reprotest is our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. This month, reprotest version 0.7.33 was uploaded to Debian by Holger Levsen with the following changes:
  • Sergei Tachenov fixed the fileordering test on all filesystems, including the likes of ZFS, which could return a different order across two builds due to build being done in a different directory [ ].
  • Holger Levsen bumped the Standards-Version to 4.7.4, reflected that Priority: optional is the default now [ ], fixed the old Free Software Foundation address [ ] and bumped the debhelper compat level to 14 after confirming there are only desired changes [ ].
  • Vagrant Cascadian updated *reprotest* to version 0.7.33 in GNU Guix.

rebuilderd, our server designed monitor the official package repositories of Linux distributions and attempt to reproduce the observed results there; it powers, amongst other things, reproduce.debian.net. This month, Bernhard M. Wiedemann submitted a pull request to rebuilderd that refactors RPM repository metadata handling. The change extracts rpm-md parsing out of the Fedora-specific scheduler module and into a shared one.
Lastly, there were a number of improvements made to our website this month as well. For example:

Patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where applicable or possible. This month, we wrote a large number of such patches, including: Vagrant Cascadian also performed a number of Non-Maintainer Uploads (NMUs) and Quality Assurance (QA) uploads in Debian for several packages with outstanding reproducibility issues. These included molttemplate, buddy, xfonts-nexus, fte and lifelines. At the time of this writing, that leaves 130 submitted patches, with only 61 outstanding submitted patches that older than a year in the Debian bug tracking system. This number is only 26 if we exclude any low-priority build path issues.

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

9 October 2026

Colin Watson: Free software activity in September 2026

My Debian contributions this month were all sponsored by Freexian. You can also support my work directly via Liberapay or GitHub Sponsors. OpenSSH I worked with upstream to remove libselinux linkage from /usr/sbin/sshd. I applied a fix from Mark Robillard Jr to fix reload logic when using sysvinit. groff I upgraded from 1.24.1 to 1.24.2, which fixed a few command injection security vulnerabilities. parted I upgraded from 3.7 to 3.8, which fixed CVE-2026-89085 and CVE-2026-89088. Python packaging New upstream versions: setuptools 84 is now in Debian and no longer contains pkg_resources. Removing uses of that module has been an ongoing project for a while now, but I did another batch of fixes this month: A new python-coverage version triggered several build regressions due to a known problem in pytest-cov. I worked around these by setting COVERAGE_CORE=ctrace: Other build/test failures: I investigated a build failure in python-proton-vpn-local-agent and proposed a fix upstream, but I m not comfortable applying this in Debian without review. If you re familiar with this code, please take a look. I fixed some other bugs: Rust packaging rust-derivre had an invalid Section field, which I cleaned up.

28 September 2026

Maytham Alsudany: Briefly looking at DNS and nameservers under the hood

I've always known that the browser uses the Domain Name Resolution (DNS) system to convert domain names like aezign.com.au into raw IP addresses, but have never actually understood how it works, outside of making sure my network config uses 1.1.1.1 or 8.8.8.8 instead of the crappy one the ISP provides. In this article, I set out to understand what nameservers do, and how they help anything that uses the internet find the right records.

DNS in a nutshellA well-used analogy is a phonebook; when you have someone's name and need to find their phone number, you look through the phonebook. Overall, DNS does the same thing: when you enter an address like example.com, your browser will contact a DNS server (the "phonebook") to determine its IP addresses. On Linux (or MacOS) systems, you can use dig on the command line to perform a DNS lookup and see what this looks like:
$ dig example.com
; <<>> DiG 9.20.29-1-Debian <<>> example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42782
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com.                   IN      A
;; ANSWER SECTION:
example.com.            221     IN      A       104.20.23.154
example.com.            221     IN      A       172.66.147.243
;; Query time: 20 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 21:07:50 AWST 2026
;; MSG SIZE  rcvd: 72
Here, you can see the DNS server being used is 1.1.1.1 (Cloudflare's DNS server), and that it found the IPs 104.20.23.154 and 172.66.147.243 associated with the domain name example.com. Your browser will then contact these IP addresses directly to load the website you are trying to open.

Configuring DNS serversYou can configure DNS servers through your operating system's network settings, and for some applications such as Firefox, with the application itself. If you don't manually set a DNS server, your computer will use the ones provided by your modem. If those are not configured, the default defined by the modem manufacturer or your ISP will be used.The most well-used and reliable DNS servers you can use are Cloudflare's (1.1.1.1 and 1.0.0.1) and Google's (8.8.8.8 and 8.8.4.4). Having more than one DNS server means that if the first one in the list doesn't work or doesn't find the matching IPs for a domain name, it will move to the next one in the list, a backup DNS server.DNS servers must be configured as IP addresses. If your computer tried to use a DNS server at a domain name, then where would your computer find the IP address for that domain name? It wouldn't be able to, as this is the exact gap that DNS servers fulfill.

RecordsDNS is not used only to find IPs for a domain, but a whole host of other things. Collectively they are referred to as "records" (or "DNS records"). Each record is associated to a domain (e.g. example.com) or a subdomain (e.g. bar.example.com, foo.bar.example.com) Here are some of the most common and important types:
  • A - Map to an IPv4 address (e.g. 104.20.23.154).This is the fundamental function of DNS: to map from a domain name to a server's IP address. Multiple A records leads to load balancing, where browsers will randomly select one of the returned IP addresses such that traffic is evenly split between them.
  • AAAA - Map to an IPv6 address (e.g. 2606:4700:10::6814:179a).This serves the same purpose as A records but for IPv6. Multiple AAAA records results in the same load balancing behaviour. All modern websites should have both an A record for legacy compatibility, and an AAAA record to future-proof for the gradual shift towards IPv6 addresses.
  • CNAME - Alias to another domain name (e.g. example.com).DNS servers will recursively lookup records for the aliased domain name until they reach an IP address (in an A or AAAA record).
  • MX - Specify a Mail Exchange server (e.g. mail.example.com).This record tells email SMTP servers where to direct emails to. For instance, the MX record for aezign.com.au is mail.aezign.au; this directs SMTP servers like Gmail and Outlook to send emails to the mailserver running at mail.aezign.au. This also allows specifying a priority field, so that you can define multiple MX records (such as a main and a backup) and consumers will try to connect to the listed mailservers from lowest priority to highest.
  • TXT - Store arbitrary text dataThis is most commonly used for site verification (for instance, to link your website to Google Search Console) and for email security measures like DKIM, DMARC, and SPF.
  • NS - Specify the nameservers for a domain (e.g. zeus.ns.cloudflare.com).These authoritative nameservers are responsible for providing all the other record types when needed.
dig lets you query each of these. For instance, to list the MX records against aezign.com.au:
$ dig MX aezign.com.au
; <<>> DiG 9.20.29-1-Debian <<>> MX aezign.com.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28213
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aezign.com.au.                 IN      MX
;; ANSWER SECTION:
aezign.com.au.          300     IN      MX      10 mail.aezign.au.
;; Query time: 104 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 22:04:58 AWST 2026
;; MSG SIZE  rcvd: 70
Here you can see there is only one mailserver configured for aezign.com.au, which is mail.aezign.au, and it has a priority of 10.If you compare this against something like gmail.com:
$ dig MX gmail.com
; <<>> DiG 9.20.29-1-Debian <<>> MX gmail.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16538
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;gmail.com.                     IN      MX
;; ANSWER SECTION:
gmail.com.              2497    IN      MX      40 alt4.gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      20 alt2.gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      5 gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      10 alt1.gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      30 alt3.gmail-smtp-in.l.google.com.
;; Query time: 120 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 22:06:04 AWST 2026
;; MSG SIZE  rcvd: 161
You can see that gmail.com has several mailservers configured as redundancies due to the Gmail's sheer scale. gmail-smtp-in.l.google.com has the highest priority of 5, so SMTP servers will attempt to contact that mailserver first, followed by alt1.gmail-smtp-in.l.google.com and so on.

NameserversNow there needs to be somewhere for DNS records against a domain name to be defined, right? This is where nameservers come in.Nameservers are DNS servers that answer queries about the domains they are authoritative for i.e. the domains they are in charge of. For example, the authoritative nameservers for example.com are hera.ns.cloudflare.com and elliott.ns.cloudflare.com, which means Cloudflare's nameservers are the source of truth when looking up DNS records for example.com. You can check this with dig:
$ dig NS example.com
; <<>> DiG 9.20.29-1-Debian <<>> NS example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37717
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com.                   IN      NS
;; ANSWER SECTION:
example.com.            81837   IN      NS      hera.ns.cloudflare.com.
example.com.            81837   IN      NS      elliott.ns.cloudflare.com.
;; Query time: 140 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 23:27:19 AWST 2026
;; MSG SIZE  rcvd: 95
Nameservers are configured with your domain name's registrar. The nameserver you set dictates where your remaining DNS records will live. For example.com, this means you would configure all your records on Cloudflare's dashboard, since they are responsible for the domain's DNS records.

Recursive resolutionThe DNS servers that our computers use follow a recursive resolution process behind-the-scenes to find the authoritative nameservers for a domain, followed by the records you are querying.

Finding the root nameserversDue to the recursive nature of the resolution process, there needs to be a starting point: the root (.) nameservers. The resolver uses a fixed list of root nameservers called root hints issued by IANA (named.root), to locate and contact the root nameservers. Resolvers will have this provided to them initially, and can dynamically update it themselves by querying one of the root nameservers for the root NS records to obtain a fresh list. For instance, we can query 198.41.0.4 (a.root-servers.net) to obtain a new list of root nameservers:
$ dig @198.41.0.4 NS .
; <<>> DiG 9.20.29-1-Debian <<>> @198.41.0.4 NS .
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43677
;; flags: qr aa rd; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 27
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;.				IN	NS
;; ANSWER SECTION:
.			518400	IN	NS	l.root-servers.net.
.			518400	IN	NS	j.root-servers.net.
.			518400	IN	NS	f.root-servers.net.
.			518400	IN	NS	h.root-servers.net.
.			518400	IN	NS	d.root-servers.net.
.			518400	IN	NS	b.root-servers.net.
.			518400	IN	NS	k.root-servers.net.
.			518400	IN	NS	i.root-servers.net.
.			518400	IN	NS	m.root-servers.net.
.			518400	IN	NS	e.root-servers.net.
.			518400	IN	NS	g.root-servers.net.
.			518400	IN	NS	c.root-servers.net.
.			518400	IN	NS	a.root-servers.net.
;; ADDITIONAL SECTION:
l.root-servers.net.	518400	IN	A	199.7.83.42
l.root-servers.net.	518400	IN	AAAA	2001:500:9f::42
j.root-servers.net.	518400	IN	A	192.58.128.30
j.root-servers.net.	518400	IN	AAAA	2001:503:c27::2:30
f.root-servers.net.	518400	IN	A	192.5.5.241
f.root-servers.net.	518400	IN	AAAA	2001:500:2f::f
h.root-servers.net.	518400	IN	A	198.97.190.53
h.root-servers.net.	518400	IN	AAAA	2001:500:1::53
d.root-servers.net.	518400	IN	A	199.7.91.13
d.root-servers.net.	518400	IN	AAAA	2001:500:2d::d
b.root-servers.net.	518400	IN	A	170.247.170.2
b.root-servers.net.	518400	IN	AAAA	2801:1b8:10::b
k.root-servers.net.	518400	IN	A	193.0.14.129
k.root-servers.net.	518400	IN	AAAA	2001:7fd::1
i.root-servers.net.	518400	IN	A	192.36.148.17
i.root-servers.net.	518400	IN	AAAA	2001:7fe::53
m.root-servers.net.	518400	IN	A	202.12.27.33
m.root-servers.net.	518400	IN	AAAA	2001:dc3::35
e.root-servers.net.	518400	IN	A	192.203.230.10
e.root-servers.net.	518400	IN	AAAA	2001:500:a8::e
g.root-servers.net.	518400	IN	A	192.112.36.4
g.root-servers.net.	518400	IN	AAAA	2001:500:12::d0d
c.root-servers.net.	518400	IN	A	192.33.4.12
c.root-servers.net.	518400	IN	AAAA	2001:500:2::c
a.root-servers.net.	518400	IN	A	198.41.0.4
a.root-servers.net.	518400	IN	AAAA	2001:503:ba3e::2:30
;; Query time: 476 msec
;; SERVER: 198.41.0.4#53(198.41.0.4) (UDP)
;; WHEN: Mon Sep 28 00:01:08 AWST 2026
;; MSG SIZE  rcvd: 811

Querying the root nameserversFor example.com, the first step is to query the root (.) nameservers to find the nameservers for the .com top-level domain (TLD). Using a.root-servers.net again:
$ dig @198.41.0.4 NS com
; <<>> DiG 9.20.29-1-Debian <<>> @198.41.0.4 NS com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22528
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 13, ADDITIONAL: 27
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;com.				IN	NS
;; AUTHORITY SECTION:
com.			172800	IN	NS	l.gtld-servers.net.
com.			172800	IN	NS	j.gtld-servers.net.
com.			172800	IN	NS	h.gtld-servers.net.
com.			172800	IN	NS	d.gtld-servers.net.
com.			172800	IN	NS	b.gtld-servers.net.
com.			172800	IN	NS	f.gtld-servers.net.
com.			172800	IN	NS	k.gtld-servers.net.
com.			172800	IN	NS	m.gtld-servers.net.
com.			172800	IN	NS	i.gtld-servers.net.
com.			172800	IN	NS	g.gtld-servers.net.
com.			172800	IN	NS	a.gtld-servers.net.
com.			172800	IN	NS	c.gtld-servers.net.
com.			172800	IN	NS	e.gtld-servers.net.
;; ADDITIONAL SECTION:
l.gtld-servers.net.	172800	IN	A	192.41.162.30
l.gtld-servers.net.	172800	IN	AAAA	2001:500:d937::30
j.gtld-servers.net.	172800	IN	A	192.48.79.30
j.gtld-servers.net.	172800	IN	AAAA	2001:502:7094::30
h.gtld-servers.net.	172800	IN	A	192.54.112.30
h.gtld-servers.net.	172800	IN	AAAA	2001:502:8cc::30
d.gtld-servers.net.	172800	IN	A	192.31.80.30
d.gtld-servers.net.	172800	IN	AAAA	2001:500:856e::30
b.gtld-servers.net.	172800	IN	A	192.33.14.30
b.gtld-servers.net.	172800	IN	AAAA	2001:503:231d::2:30
f.gtld-servers.net.	172800	IN	A	192.35.51.30
f.gtld-servers.net.	172800	IN	AAAA	2001:503:d414::30
k.gtld-servers.net.	172800	IN	A	192.52.178.30
k.gtld-servers.net.	172800	IN	AAAA	2001:503:d2d::30
m.gtld-servers.net.	172800	IN	A	192.55.83.30
m.gtld-servers.net.	172800	IN	AAAA	2001:501:b1f9::30
i.gtld-servers.net.	172800	IN	A	192.43.172.30
i.gtld-servers.net.	172800	IN	AAAA	2001:503:39c1::30
g.gtld-servers.net.	172800	IN	A	192.42.93.30
g.gtld-servers.net.	172800	IN	AAAA	2001:503:eea3::30
a.gtld-servers.net.	172800	IN	A	192.5.6.30
a.gtld-servers.net.	172800	IN	AAAA	2001:503:a83e::2:30
c.gtld-servers.net.	172800	IN	A	192.26.92.30
c.gtld-servers.net.	172800	IN	AAAA	2001:503:83eb::30
e.gtld-servers.net.	172800	IN	A	192.12.94.30
e.gtld-servers.net.	172800	IN	AAAA	2001:502:1ca1::30
;; Query time: 772 msec
;; SERVER: 198.41.0.4#53(198.41.0.4) (UDP)
;; WHEN: Mon Sep 28 00:09:25 AWST 2026
;; MSG SIZE  rcvd: 828

Querying the TLD nameserversNow the resolver has .com nameservers and their IPs. We can use one of these, such as 192.5.6.30 (a.gtld-servers.net) to find the nameservers for example.com:
$ dig @192.5.6.30 NS example.com
; <<>> DiG 9.20.29-1-Debian <<>> @192.5.6.30 NS example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37419
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 13
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.com.                   IN      NS
;; AUTHORITY SECTION:
example.com.            172800  IN      NS      hera.ns.cloudflare.com.
example.com.            172800  IN      NS      elliott.ns.cloudflare.com.
;; ADDITIONAL SECTION:
hera.ns.cloudflare.com. 172800  IN      A       108.162.192.162
hera.ns.cloudflare.com. 172800  IN      A       172.64.32.162
hera.ns.cloudflare.com. 172800  IN      A       173.245.58.162
hera.ns.cloudflare.com. 172800  IN      AAAA    2606:4700:50::adf5:3aa2
hera.ns.cloudflare.com. 172800  IN      AAAA    2803:f800:50::6ca2:c0a2
hera.ns.cloudflare.com. 172800  IN      AAAA    2a06:98c1:50::ac40:20a2
elliott.ns.cloudflare.com. 172800 IN    A       108.162.195.228
elliott.ns.cloudflare.com. 172800 IN    A       162.159.44.228
elliott.ns.cloudflare.com. 172800 IN    A       172.64.35.228
elliott.ns.cloudflare.com. 172800 IN    AAAA    2606:4700:58::a29f:2ce4
elliott.ns.cloudflare.com. 172800 IN    AAAA    2803:f800:50::6ca2:c3e4
elliott.ns.cloudflare.com. 172800 IN    AAAA    2a06:98c1:50::ac40:23e4
;; Query time: 80 msec
;; SERVER: 192.5.6.30#53(192.5.6.30) (UDP)
;; WHEN: Mon Sep 28 00:12:19 AWST 2026
;; MSG SIZE  rcvd: 359

Finding the domain's nameserverThe response tells the resolver that the nameservers for example.com are hera.ns.cloudflare.com and elliott.ns.cloudflare.com. Now since these Cloudflare nameservers are also .com domains and Cloudflare is so commonly used, the .com nameserver provides optional sibling glue records as defined in RFC 9471 to prevent further roundtrips and optimize the process.If we ignore the sibling glue records provided, the resolver would have to ask the .com nameservers for the nameservers for cloudflare.com:
$ dig @192.5.6.30 NS cloudflare.com
; <<>> DiG 9.20.29-1-Debian <<>> @192.5.6.30 NS cloudflare.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40307
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 5, ADDITIONAL: 21
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;cloudflare.com.                        IN      NS
;; AUTHORITY SECTION:
cloudflare.com.         172800  IN      NS      ns3.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns5.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns4.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns6.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns7.cloudflare.com.
;; ADDITIONAL SECTION:
ns3.cloudflare.com.     172800  IN      A       162.159.0.33
ns3.cloudflare.com.     172800  IN      A       162.159.7.226
ns3.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:21
ns3.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:7e2
ns5.cloudflare.com.     172800  IN      A       162.159.2.9
ns5.cloudflare.com.     172800  IN      A       162.159.9.55
ns5.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:209
ns5.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:937
ns4.cloudflare.com.     172800  IN      A       162.159.1.33
ns4.cloudflare.com.     172800  IN      A       162.159.8.55
ns4.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:121
ns4.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:837
ns6.cloudflare.com.     172800  IN      A       162.159.3.11
ns6.cloudflare.com.     172800  IN      A       162.159.5.6
ns6.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:30b
ns6.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:506
ns7.cloudflare.com.     172800  IN      A       162.159.4.8
ns7.cloudflare.com.     172800  IN      A       162.159.6.6
ns7.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:408
ns7.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:606
;; Query time: 163 msec
;; SERVER: 192.5.6.30#53(192.5.6.30) (UDP)
;; WHEN: Mon Sep 28 00:26:53 AWST 2026
;; MSG SIZE  rcvd: 573
Here, the .com nameserver provides in-domain glue records for cloudflare.com's nameservers. Without these, there would be a circular dependency, since the nameserver named is a subdomain of the domain being queried for. This is because nameservers are always defined by name, which means that in order to actually contact the nameserver, your computer needs to obtain its IP address with another DNS lookup.Next, we'd use one of these nameservers, say ns3.cloudflare.com, to obtain the IP for hera.ns.cloudflare.com:
$ dig @162.159.0.33 hera.ns.cloudflare.com
; <<>> DiG 9.20.29-1-Debian <<>> @162.159.0.33 hera.ns.cloudflare.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16801
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;hera.ns.cloudflare.com.                IN      A
;; ANSWER SECTION:
hera.ns.cloudflare.com. 86353   IN      A       108.162.192.162
hera.ns.cloudflare.com. 86353   IN      A       173.245.58.162
hera.ns.cloudflare.com. 86353   IN      A       172.64.32.162
;; Query time: 123 msec
;; SERVER: 162.159.0.33#53(162.159.0.33) (UDP)
;; WHEN: Mon Sep 28 01:11:37 AWST 2026
;; MSG SIZE  rcvd: 99

Querying the domain's authoritative nameserverNow we've reached the authoritative nameserver for example.com! The last thing to do is to fetch the A record for example.com to obtain its server's IP address:
$ dig @108.162.192.162 example.com
; <<>> DiG 9.20.29-1-Debian <<>> @108.162.192.162 example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43082
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com.                   IN      A
;; ANSWER SECTION:
example.com.            300     IN      A       104.20.23.154
example.com.            300     IN      A       172.66.147.243
;; Query time: 123 msec
;; SERVER: 108.162.192.162#53(108.162.192.162) (UDP)
;; WHEN: Mon Sep 28 01:13:47 AWST 2026
;; MSG SIZE  rcvd: 72
At last, we've determined that the IP addresses of the servers behind example.com are 104.20.23.154 and 172.66.147.243. This matches our earlier lookup directly against your computer's configured DNS server.The DNS servers your computer uses encapsulate this recursive resolution process behind the scenes, so that your computer only needs to make one round trip, and the servers can use their much faster data center connections and caching to optimize the process and make the query as fast as possible.

Checking DNS recordsTo check a website's records, the first thing to do is to ensure the domain name is pointed at the correct nameserver. For instance, aezign.com.au is supposed to use Cloudflare, since that is where all the DNS records have been set up. We can verify this using dig, (or alternatively with online tools like dnschecker.org):
$ dig NS aezign.com.au
; <<>> DiG 9.20.29-1-Debian <<>> NS aezign.com.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44172
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aezign.com.au.                 IN      NS
;; ANSWER SECTION:
aezign.com.au.          86400   IN      NS      rosalie.ns.cloudflare.com.
aezign.com.au.          86400   IN      NS      zeus.ns.cloudflare.com.
;; Query time: 92 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Mon Sep 28 01:39:22 AWST 2026
;; MSG SIZE  rcvd: 100
If this is not set correctly, the DNS records you configure will not work, as resolvers will be fetching records from somewhere else. In most cases, you can update this in your domain registrar's settings. This is especially common for new domain names, which default to the registrar's nameservers; unless you plan to use their built-in DNS management, you'll need to point them to your chosen provider.Now we can check that the DNS records configured match those your DNS server returns. For aezign.com.au, these are the MX and TXT records configured on Cloudflare.
Extract of Cloudflare's DNS record listing showing the MX record and TXT records for aezign.com.au. The MX record and some TXT records configured for aezign.com.au on Cloudflare's dashboard.
We can check these using dig, and using the +short option to show only the content of the records.
$ dig MX aezign.com.au +short
10 mail.aezign.au.
$ dig TXT aezign.com.au +short
"google-site-verification=KFqTU5xH8fFf6OtszEL4oudfxple29Tm-TvxVtCWkAM"
"v=spf1 mx a:mail.aezign.au -all"
$ dig TXT _dmarc.aezign.com.au +short
"v=DMARC1;p=quarantine;sp=quarantine;adkim=r;aspf=r"
$ dig TXT aezign.com.au-2026._domainkey.aezign.com.au +short
"v=DKIM1; h=sha256; k=rsa; s=email; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzL04yXZgB72YkoxS+tLK//aWx65TS4UA3F0qJ7el68SEuD4ehmg+61Ta9iS31H6U074dnjyoPDeaiGBa7ToTNCtiIts4/ghD/8nrENROE6hQAwRmTK18ODDHrRDW73hWe6Dg4cK5vmXi82/wWRZU9SDKbme28IE9uCVir3lCkKYLs1" "j16gR1Sjqr6a2+3o+EeVXnLJ4wXmJMm7KCIdn7zv0t9Z1eYcw672PdpYPAMpGb8uUsgaDBsNSpTcupPkz4SWj6TrOB80CPHjWseVY8EOJeRsNKQ/8LFoQEBl4Rxtb3HAmu4UIxSZ4QMVOLnXz67e85mA5XjcNIsVMzaWaowwIDAQAB"
As we can see, this matches, which means that the nameservers for aezign.com.au are correct and the DNS records are what we intend them to be.Note that if you've recently changed the nameserver or DNS records for your domain, it can take anywhere between 10 seconds and 24 hours for the changes to take effect and propagate across the different DNS servers globally. Tools like dnschecker.org can check your records on different DNS servers worldwide to see if they're reflecting your changes. The reason for this is caching: DNS servers like 1.1.1.1 will cache results so it doesn't have to do the full resolution process every time, and uses the Time-To-Live (TTL) you configure in your DNS records as a guideline for caching duration.

Maytham Alsudany: A Brief Explanation of DNS and Nameservers

The Domain Name System (DNS) is a fundamental piece of technology behind the modern internet. Thanks to its introduction in 1983, you're able to enter domain names like aezign.com.au instead of raw IP addresses in your browser's address bar and when sending emails. This article explains what it is, how it works, and how you can inspect your own website's DNS and ensure it is set up correctly.

DNS in a nutshellA well-used analogy is a phonebook; when you have someone's name and need to find their phone number, you look through the phonebook. Overall, DNS does the same thing: when you enter an address like example.com, your browser will contact a DNS server (the "phonebook") to determine its IP addresses. On Linux (or MacOS) systems, you can use dig on the command line to perform a DNS lookup and see what this looks like:
$ dig example.com
; <<>> DiG 9.20.29-1-Debian <<>> example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42782
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com.                   IN      A
;; ANSWER SECTION:
example.com.            221     IN      A       104.20.23.154
example.com.            221     IN      A       172.66.147.243
;; Query time: 20 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 21:07:50 AWST 2026
;; MSG SIZE  rcvd: 72
Here, you can see the DNS server being used is 1.1.1.1 (Cloudflare's DNS server), and that it found the IPs 104.20.23.154 and 172.66.147.243 associated with the domain name example.com. Your browser will then contact these IP addresses directly to load the website you are trying to open.

Configuring DNS serversYou can configure DNS servers through your operating system's network settings, and for some applications such as Firefox, with the application itself. If you don't manually set a DNS server, your computer will use the ones provided by your modem. If those are not configured, the default defined by the modem manufacturer or your ISP will be used.The most well-used and reliable DNS servers you can use are Cloudflare's (1.1.1.1 and 1.0.0.1) and Google's (8.8.8.8 and 8.8.4.4). Having more than one DNS server means that if the first one in the list doesn't work or doesn't find the matching IPs for a domain name, it will move to the next one in the list, a backup DNS server.DNS servers must be configured as IP addresses. If your computer tried to use a DNS server at a domain name, then where would your computer find the IP address for that domain name? It wouldn't be able to, as this is the exact gap that DNS servers fulfill.

RecordsDNS is not used only to find IPs for a domain, but a whole host of other things. Collectively they are referred to as "records" (or "DNS records"). Each record is associated to a domain (e.g. example.com) or a subdomain (e.g. bar.example.com, foo.bar.example.com) Here are some of the most common and important types:
  • A - Map to an IPv4 address (e.g. 104.20.23.154).This is the fundamental function of DNS: to map from a domain name to a server's IP address. Multiple A records leads to load balancing, where browsers will randomly select one of the returned IP addresses such that traffic is evenly split between them.
  • AAAA - Map to an IPv6 address (e.g. 2606:4700:10::6814:179a).This serves the same purpose as A records but for IPv6. Multiple AAAA records results in the same load balancing behaviour. All modern websites should have both an A record for legacy compatibility, and an AAAA record to future-proof for the gradual shift towards IPv6 addresses.
  • CNAME - Alias to another domain name (e.g. example.com).DNS servers will recursively lookup records for the aliased domain name until they reach an IP address (in an A or AAAA record).
  • MX - Specify a Mail Exchange server (e.g. mail.example.com).This record tells email SMTP servers where to direct emails to. For instance, the MX record for aezign.com.au is mail.aezign.au; this directs SMTP servers like Gmail and Outlook to send emails to the mailserver running at mail.aezign.au. This also allows specifying a priority field, so that you can define multiple MX records (such as a main and a backup) and consumers will try to connect to the listed mailservers from lowest priority to highest.
  • TXT - Store arbitrary text dataThis is most commonly used for site verification (for instance, to link your website to Google Search Console) and for email security measures like DKIM, DMARC, and SPF.
  • NS - Specify the nameservers for a domain (e.g. zeus.ns.cloudflare.com).These authoritative nameservers are responsible for providing all the other record types when needed.
dig lets you query each of these. For instance, to list the MX records against aezign.com.au:
$ dig MX aezign.com.au
; <<>> DiG 9.20.29-1-Debian <<>> MX aezign.com.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28213
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aezign.com.au.                 IN      MX
;; ANSWER SECTION:
aezign.com.au.          300     IN      MX      10 mail.aezign.au.
;; Query time: 104 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 22:04:58 AWST 2026
;; MSG SIZE  rcvd: 70
Here you can see there is only one mailserver configured for aezign.com.au, which is mail.aezign.au, and it has a priority of 10.If you compare this against something like gmail.com:
$ dig MX gmail.com
; <<>> DiG 9.20.29-1-Debian <<>> MX gmail.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16538
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;gmail.com.                     IN      MX
;; ANSWER SECTION:
gmail.com.              2497    IN      MX      40 alt4.gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      20 alt2.gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      5 gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      10 alt1.gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      30 alt3.gmail-smtp-in.l.google.com.
;; Query time: 120 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 22:06:04 AWST 2026
;; MSG SIZE  rcvd: 161
You can see that gmail.com has several mailservers configured as redundancies due to the Gmail's sheer scale. gmail-smtp-in.l.google.com has the highest priority of 5, so SMTP servers will attempt to contact that mailserver first, followed by alt1.gmail-smtp-in.l.google.com and so on.

NameserversNow there needs to be somewhere for DNS records against a domain name to be defined, right? This is where nameservers come in.Nameservers are DNS servers that answer queries about the domains they are authoritative for i.e. the domains they are in charge of. For example, the authoritative nameservers for example.com are hera.ns.cloudflare.com and elliott.ns.cloudflare.com, which means Cloudflare's nameservers are the source of truth when looking up DNS records for example.com. You can check this with dig:
$ dig NS example.com
; <<>> DiG 9.20.29-1-Debian <<>> NS example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37717
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com.                   IN      NS
;; ANSWER SECTION:
example.com.            81837   IN      NS      hera.ns.cloudflare.com.
example.com.            81837   IN      NS      elliott.ns.cloudflare.com.
;; Query time: 140 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 23:27:19 AWST 2026
;; MSG SIZE  rcvd: 95
Nameservers are configured with your domain name's registrar. The nameserver you set dictates where your remaining DNS records will live. For example.com, this means you would configure all your records on Cloudflare's dashboard, since they are responsible for the domain's DNS records.

Recursive resolutionThe DNS servers that our computers use follow a recursive resolution process behind-the-scenes to find the authoritative nameservers for a domain, followed by the records you are querying.

Finding the root nameserversDue to the recursive nature of the resolution process, there needs to be a starting point: the root (.) nameservers. The resolver uses a fixed list of root nameservers called root hints issued by IANA (named.root), to locate and contact the root nameservers. Resolvers will have this provided to them initially, and can dynamically update it themselves by querying one of the root nameservers for the root NS records to obtain a fresh list. For instance, we can query 198.41.0.4 (a.root-servers.net) to obtain a new list of root nameservers:
$ dig @198.41.0.4 NS .
; <<>> DiG 9.20.29-1-Debian <<>> @198.41.0.4 NS .
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43677
;; flags: qr aa rd; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 27
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;.				IN	NS
;; ANSWER SECTION:
.			518400	IN	NS	l.root-servers.net.
.			518400	IN	NS	j.root-servers.net.
.			518400	IN	NS	f.root-servers.net.
.			518400	IN	NS	h.root-servers.net.
.			518400	IN	NS	d.root-servers.net.
.			518400	IN	NS	b.root-servers.net.
.			518400	IN	NS	k.root-servers.net.
.			518400	IN	NS	i.root-servers.net.
.			518400	IN	NS	m.root-servers.net.
.			518400	IN	NS	e.root-servers.net.
.			518400	IN	NS	g.root-servers.net.
.			518400	IN	NS	c.root-servers.net.
.			518400	IN	NS	a.root-servers.net.
;; ADDITIONAL SECTION:
l.root-servers.net.	518400	IN	A	199.7.83.42
l.root-servers.net.	518400	IN	AAAA	2001:500:9f::42
j.root-servers.net.	518400	IN	A	192.58.128.30
j.root-servers.net.	518400	IN	AAAA	2001:503:c27::2:30
f.root-servers.net.	518400	IN	A	192.5.5.241
f.root-servers.net.	518400	IN	AAAA	2001:500:2f::f
h.root-servers.net.	518400	IN	A	198.97.190.53
h.root-servers.net.	518400	IN	AAAA	2001:500:1::53
d.root-servers.net.	518400	IN	A	199.7.91.13
d.root-servers.net.	518400	IN	AAAA	2001:500:2d::d
b.root-servers.net.	518400	IN	A	170.247.170.2
b.root-servers.net.	518400	IN	AAAA	2801:1b8:10::b
k.root-servers.net.	518400	IN	A	193.0.14.129
k.root-servers.net.	518400	IN	AAAA	2001:7fd::1
i.root-servers.net.	518400	IN	A	192.36.148.17
i.root-servers.net.	518400	IN	AAAA	2001:7fe::53
m.root-servers.net.	518400	IN	A	202.12.27.33
m.root-servers.net.	518400	IN	AAAA	2001:dc3::35
e.root-servers.net.	518400	IN	A	192.203.230.10
e.root-servers.net.	518400	IN	AAAA	2001:500:a8::e
g.root-servers.net.	518400	IN	A	192.112.36.4
g.root-servers.net.	518400	IN	AAAA	2001:500:12::d0d
c.root-servers.net.	518400	IN	A	192.33.4.12
c.root-servers.net.	518400	IN	AAAA	2001:500:2::c
a.root-servers.net.	518400	IN	A	198.41.0.4
a.root-servers.net.	518400	IN	AAAA	2001:503:ba3e::2:30
;; Query time: 476 msec
;; SERVER: 198.41.0.4#53(198.41.0.4) (UDP)
;; WHEN: Mon Sep 28 00:01:08 AWST 2026
;; MSG SIZE  rcvd: 811

Querying the root nameserversFor example.com, the first step is to query the root (.) nameservers to find the nameservers for the .com top-level domain (TLD). Using a.root-servers.net again:
$ dig @198.41.0.4 NS com
; <<>> DiG 9.20.29-1-Debian <<>> @198.41.0.4 NS com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22528
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 13, ADDITIONAL: 27
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;com.				IN	NS
;; AUTHORITY SECTION:
com.			172800	IN	NS	l.gtld-servers.net.
com.			172800	IN	NS	j.gtld-servers.net.
com.			172800	IN	NS	h.gtld-servers.net.
com.			172800	IN	NS	d.gtld-servers.net.
com.			172800	IN	NS	b.gtld-servers.net.
com.			172800	IN	NS	f.gtld-servers.net.
com.			172800	IN	NS	k.gtld-servers.net.
com.			172800	IN	NS	m.gtld-servers.net.
com.			172800	IN	NS	i.gtld-servers.net.
com.			172800	IN	NS	g.gtld-servers.net.
com.			172800	IN	NS	a.gtld-servers.net.
com.			172800	IN	NS	c.gtld-servers.net.
com.			172800	IN	NS	e.gtld-servers.net.
;; ADDITIONAL SECTION:
l.gtld-servers.net.	172800	IN	A	192.41.162.30
l.gtld-servers.net.	172800	IN	AAAA	2001:500:d937::30
j.gtld-servers.net.	172800	IN	A	192.48.79.30
j.gtld-servers.net.	172800	IN	AAAA	2001:502:7094::30
h.gtld-servers.net.	172800	IN	A	192.54.112.30
h.gtld-servers.net.	172800	IN	AAAA	2001:502:8cc::30
d.gtld-servers.net.	172800	IN	A	192.31.80.30
d.gtld-servers.net.	172800	IN	AAAA	2001:500:856e::30
b.gtld-servers.net.	172800	IN	A	192.33.14.30
b.gtld-servers.net.	172800	IN	AAAA	2001:503:231d::2:30
f.gtld-servers.net.	172800	IN	A	192.35.51.30
f.gtld-servers.net.	172800	IN	AAAA	2001:503:d414::30
k.gtld-servers.net.	172800	IN	A	192.52.178.30
k.gtld-servers.net.	172800	IN	AAAA	2001:503:d2d::30
m.gtld-servers.net.	172800	IN	A	192.55.83.30
m.gtld-servers.net.	172800	IN	AAAA	2001:501:b1f9::30
i.gtld-servers.net.	172800	IN	A	192.43.172.30
i.gtld-servers.net.	172800	IN	AAAA	2001:503:39c1::30
g.gtld-servers.net.	172800	IN	A	192.42.93.30
g.gtld-servers.net.	172800	IN	AAAA	2001:503:eea3::30
a.gtld-servers.net.	172800	IN	A	192.5.6.30
a.gtld-servers.net.	172800	IN	AAAA	2001:503:a83e::2:30
c.gtld-servers.net.	172800	IN	A	192.26.92.30
c.gtld-servers.net.	172800	IN	AAAA	2001:503:83eb::30
e.gtld-servers.net.	172800	IN	A	192.12.94.30
e.gtld-servers.net.	172800	IN	AAAA	2001:502:1ca1::30
;; Query time: 772 msec
;; SERVER: 198.41.0.4#53(198.41.0.4) (UDP)
;; WHEN: Mon Sep 28 00:09:25 AWST 2026
;; MSG SIZE  rcvd: 828

Querying the TLD nameserversNow the resolver has .com nameservers and their IPs. We can use one of these, such as 192.5.6.30 (a.gtld-servers.net) to find the nameservers for example.com:
$ dig @192.5.6.30 NS example.com
; <<>> DiG 9.20.29-1-Debian <<>> @192.5.6.30 NS example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37419
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 13
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.com.                   IN      NS
;; AUTHORITY SECTION:
example.com.            172800  IN      NS      hera.ns.cloudflare.com.
example.com.            172800  IN      NS      elliott.ns.cloudflare.com.
;; ADDITIONAL SECTION:
hera.ns.cloudflare.com. 172800  IN      A       108.162.192.162
hera.ns.cloudflare.com. 172800  IN      A       172.64.32.162
hera.ns.cloudflare.com. 172800  IN      A       173.245.58.162
hera.ns.cloudflare.com. 172800  IN      AAAA    2606:4700:50::adf5:3aa2
hera.ns.cloudflare.com. 172800  IN      AAAA    2803:f800:50::6ca2:c0a2
hera.ns.cloudflare.com. 172800  IN      AAAA    2a06:98c1:50::ac40:20a2
elliott.ns.cloudflare.com. 172800 IN    A       108.162.195.228
elliott.ns.cloudflare.com. 172800 IN    A       162.159.44.228
elliott.ns.cloudflare.com. 172800 IN    A       172.64.35.228
elliott.ns.cloudflare.com. 172800 IN    AAAA    2606:4700:58::a29f:2ce4
elliott.ns.cloudflare.com. 172800 IN    AAAA    2803:f800:50::6ca2:c3e4
elliott.ns.cloudflare.com. 172800 IN    AAAA    2a06:98c1:50::ac40:23e4
;; Query time: 80 msec
;; SERVER: 192.5.6.30#53(192.5.6.30) (UDP)
;; WHEN: Mon Sep 28 00:12:19 AWST 2026
;; MSG SIZE  rcvd: 359

Finding the domain's nameserverThe response tells the resolver that the nameservers for example.com are hera.ns.cloudflare.com and elliott.ns.cloudflare.com. Now since these Cloudflare nameservers are also .com domains and Cloudflare is so commonly used, the .com nameserver provides optional sibling glue records as defined in RFC 9471 to prevent further roundtrips and optimize the process.If we ignore the sibling glue records provided, the resolver would have to ask the .com nameservers for the nameservers for cloudflare.com:
$ dig @192.5.6.30 NS cloudflare.com
; <<>> DiG 9.20.29-1-Debian <<>> @192.5.6.30 NS cloudflare.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40307
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 5, ADDITIONAL: 21
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;cloudflare.com.                        IN      NS
;; AUTHORITY SECTION:
cloudflare.com.         172800  IN      NS      ns3.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns5.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns4.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns6.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns7.cloudflare.com.
;; ADDITIONAL SECTION:
ns3.cloudflare.com.     172800  IN      A       162.159.0.33
ns3.cloudflare.com.     172800  IN      A       162.159.7.226
ns3.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:21
ns3.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:7e2
ns5.cloudflare.com.     172800  IN      A       162.159.2.9
ns5.cloudflare.com.     172800  IN      A       162.159.9.55
ns5.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:209
ns5.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:937
ns4.cloudflare.com.     172800  IN      A       162.159.1.33
ns4.cloudflare.com.     172800  IN      A       162.159.8.55
ns4.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:121
ns4.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:837
ns6.cloudflare.com.     172800  IN      A       162.159.3.11
ns6.cloudflare.com.     172800  IN      A       162.159.5.6
ns6.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:30b
ns6.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:506
ns7.cloudflare.com.     172800  IN      A       162.159.4.8
ns7.cloudflare.com.     172800  IN      A       162.159.6.6
ns7.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:408
ns7.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:606
;; Query time: 163 msec
;; SERVER: 192.5.6.30#53(192.5.6.30) (UDP)
;; WHEN: Mon Sep 28 00:26:53 AWST 2026
;; MSG SIZE  rcvd: 573
Here, the .com nameserver provides in-domain glue records for cloudflare.com's nameservers. Without these, there would be a circular dependency, since the nameserver named is a subdomain of the domain being queried for. This is because nameservers are always defined by name, which means that in order to actually contact the nameserver, your computer needs to obtain its IP address with another DNS lookup.Next, we'd use one of these nameservers, say ns3.cloudflare.com, to obtain the IP for hera.ns.cloudflare.com:
$ dig @162.159.0.33 hera.ns.cloudflare.com
; <<>> DiG 9.20.29-1-Debian <<>> @162.159.0.33 hera.ns.cloudflare.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16801
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;hera.ns.cloudflare.com.                IN      A
;; ANSWER SECTION:
hera.ns.cloudflare.com. 86353   IN      A       108.162.192.162
hera.ns.cloudflare.com. 86353   IN      A       173.245.58.162
hera.ns.cloudflare.com. 86353   IN      A       172.64.32.162
;; Query time: 123 msec
;; SERVER: 162.159.0.33#53(162.159.0.33) (UDP)
;; WHEN: Mon Sep 28 01:11:37 AWST 2026
;; MSG SIZE  rcvd: 99

Querying the domain's authoritative nameserverNow we've reached the authoritative nameserver for example.com! The last thing to do is to fetch the A record for example.com to obtain its server's IP address:
$ dig @108.162.192.162 example.com
; <<>> DiG 9.20.29-1-Debian <<>> @108.162.192.162 example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43082
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com.                   IN      A
;; ANSWER SECTION:
example.com.            300     IN      A       104.20.23.154
example.com.            300     IN      A       172.66.147.243
;; Query time: 123 msec
;; SERVER: 108.162.192.162#53(108.162.192.162) (UDP)
;; WHEN: Mon Sep 28 01:13:47 AWST 2026
;; MSG SIZE  rcvd: 72
At last, we've determined that the IP addresses of the servers behind example.com are 104.20.23.154 and 172.66.147.243. This matches our earlier lookup directly against your computer's configured DNS server.The DNS servers your computer uses employ this recursive resolution technique behind the scenes, so that your computer only needs to make one round trip, and the servers can use their much faster data center connections and caching to optimize the process and make the query as fast as possible.

Verify your DNS recordsTo check your website's records, the first thing you'll want to do is ensure your domain name is pointed at the correct nameserver. For instance, aezign.com.au is supposed to use Cloudflare, since that is where all the DNS records have been set up. We can verify this using dig, (or alternatively with online tools like dnschecker.org):
$ dig NS aezign.com.au
; <<>> DiG 9.20.29-1-Debian <<>> NS aezign.com.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44172
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aezign.com.au.                 IN      NS
;; ANSWER SECTION:
aezign.com.au.          86400   IN      NS      rosalie.ns.cloudflare.com.
aezign.com.au.          86400   IN      NS      zeus.ns.cloudflare.com.
;; Query time: 92 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Mon Sep 28 01:39:22 AWST 2026
;; MSG SIZE  rcvd: 100
If this is not set correctly, the DNS records you configure will not work, as resolvers will be fetching records from somewhere else. In most cases, you can update this in your domain registrar's settings. This is especially common for new domain names, which default to the registrar's nameservers; unless you plan to use their built-in DNS management, you'll need to point them to your chosen provider.Now we can check that the DNS records you've configured match those your DNS server returns. For aezign.com.au, these are the MX and TXT records configured on Cloudflare.
Extract of Cloudflare's DNS record listing showing the MX record and TXT records for aezign.com.au. The MX record and some TXT records configured for aezign.com.au on Cloudflare's dashboard.
We can check these using dig, and using the +short option to show only the content of the records.
$ dig MX aezign.com.au +short
10 mail.aezign.au.
$ dig TXT aezign.com.au +short
"google-site-verification=KFqTU5xH8fFf6OtszEL4oudfxple29Tm-TvxVtCWkAM"
"v=spf1 mx a:mail.aezign.au -all"
$ dig TXT _dmarc.aezign.com.au +short
"v=DMARC1;p=quarantine;sp=quarantine;adkim=r;aspf=r"
$ dig TXT aezign.com.au-2026._domainkey.aezign.com.au +short
"v=DKIM1; h=sha256; k=rsa; s=email; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzL04yXZgB72YkoxS+tLK//aWx65TS4UA3F0qJ7el68SEuD4ehmg+61Ta9iS31H6U074dnjyoPDeaiGBa7ToTNCtiIts4/ghD/8nrENROE6hQAwRmTK18ODDHrRDW73hWe6Dg4cK5vmXi82/wWRZU9SDKbme28IE9uCVir3lCkKYLs1" "j16gR1Sjqr6a2+3o+EeVXnLJ4wXmJMm7KCIdn7zv0t9Z1eYcw672PdpYPAMpGb8uUsgaDBsNSpTcupPkz4SWj6TrOB80CPHjWseVY8EOJeRsNKQ/8LFoQEBl4Rxtb3HAmu4UIxSZ4QMVOLnXz67e85mA5XjcNIsVMzaWaowwIDAQAB"
As we can see, this matches, which means that the nameservers for aezign.com.au are correct and the DNS records are what we intend them to be.Note that if you've recently changed the nameserver or DNS records for your domain, it can take anywhere between 10 seconds and 24 hours for the changes to take effect and propagate across the different DNS servers worldwide. Tools like dnschecker.org can check your records on different DNS servers worldwide to see if they're reflecting your changes. The reason for this is caching: DNS servers like 1.1.1.1 will cache results so it doesn't have to do the full resolution process every time, and uses the Time-To-Live (TTL) you configure in your DNS records as a guideline for caching duration.

15 September 2026

Reproducible Builds: Supporter spotlight: Jochen Sprickerhof on ... Reproducible Builds!

The Reproducible Builds project relies on several projects, supporters and sponsors for financial support, but they are also valued as ambassadors who spread the word about our project and the work that we do. This is the ninth installment in a series featuring the projects, companies and individuals who support the Reproducible Builds project. We started this series by featuring the Civil Infrastructure Platform project, and followed this up with a post about the Ford Foundation as well as recent ones about ARDC, the Google Open Source Security Team (GOSST), Bootstrappable Builds, the F-Droid project, David A. Wheeler, Simon Butler and Kees Cook. Today, however, we will be talking with Jochen Sprickerhof, one of the newer members of the Reproducible Builds project core team.

Vagrant Cascadian: Could you tell me a bit about yourself? What sort of things do you work on? Jochen Sprickerhof: I am a freelance programmer working on Open Source. Mainly doing Debian, F-Droid and some smaller software projects. In general I made it a habit to look into every software I use and try to fix bugs or add features I need. In Debian, I maintain about 180 packages with topics covering home banking, build systems and robotics. Most of my time, I currently work on reproduce.debian.net, where we try to bit-for-bit reproduce the packages distributed by Debian.
Vagrant: Could you describe the path that lead you to working on reproducible builds? Jochen: I started my Debian journey as a teenager, converting my school to Debian and serving as its system administrator for 13 years. After studying Applied System Science, I joined the university s robotics labs, where I worked on the Robot Operating System (ROS) and the Point Cloud Library (PCL). In the end, I enjoyed programming more than writing papers, so I eventually left academia for a robotics startup. Some years ago, I realized that the open source work I was doing in my spare time was actually the work I cared most about. Nowadays I am really grateful that I can spend my days working on things I find important and have lots of fun with.
Vagrant: What projects did you recently make big progress on? Jochen: A recent example is metasnap.debian.net. It is a meta archive of snapshot.debian.org which is itself archive of all packages in Debian. But let me explain it the other way round: with reproduce.debian.net, we try to reproduce the packages as they are distributed by the Debian archive. For that, we need the same build environment (compilers, libraries, build tools, etc) that was used by Debian back when the original package was compiled. Luckily, snapshot.debian.org has all those packages, but they are not easily accessible via apt, Debian s package manager. So, metasnap provides a mapping from a package name and version pair to the APT repo on snapshot.debian.org needed to download it from. It was created by josch some time ago, and it s awesome work. But when we tried to reproduce more and more packages on reproduce.debian.net, we found that some were missing packages from the build environment even though they where visible on snapshot.debian.org. We found that metasnap excluded some archive areas because they where not expected to be needed. Reimporting all the data took more than two months and surfaced a couple more flaws. With this fixed, we were able to build more packages, only to find out that metasnap also needs better support for version numbers. Luckily we were able to rewrite the data in a day instead of starting the import again.
Vagrant: You have been working on infrastructure to support reproducible builds for a while. Has recent adoption of reproduce.debian.net into the Debian release tooling changed the focus of your work? Jochen: Quite a bit. When we started reproduce.debian.net in 2024, only around 33% of the packages could be reproduced successfully. Today we are above 98%. Most were not bugs in the packages themselves but in the infrastructure. Similar to the metasnap issue I reference above, packages just needed a rebuild because something else, like the toolchain, was fixed in the meantime. In May, people from the Debian release team and the Reproducible Builds project sat together and decided that the overall state is good enough, and now packages that regress on reproducibility are blocked from entering the next Debian release. But that does not mean all the work is on the shoulders of Debian package maintainers. Since then I have been constantly looking at the migration tooling to spot regressions and provide fixes. Furthermore, a couple of maintainers reached out to us for help and I hope more will do so in future.
Vagrant: What is one small thing you (or others) have not yet gotten to that you would really like to see? Jochen: The central tool to reproduce Debian packages is debrebuild, also written by josch. Currently it has two ways to retrieve the build dependencies of a package. Either it uses metasnap.debian.net (as explained above), or it can access the Debian unstable APT repository directly. This allows to test packages locally before everything is indexed on metasnap by compiling against Debian unstable. But actually there are many other APT repositories to query, like Debian stable or even derivatives. Adding support for an optional list of APT repositories in debrebuild would be great. That would also be a big step to support reproducing other Debian based distributions.
Vagrant: and one big thing? Jochen: It would be great to integrate metasnap.debian.net into snapshot.debian.org. There is some discussion on it already in Debian bug #650783.
Vagrant: What are the tools you use the most? Jochen: According to my fish shell history:
$ history   cut -d' ' -f1   sort   uniq -c   sort -nr   head -10
  36199 git
  20941 vi
  12271 rm
   8599 cd
   7917 ls
   6407 apt
   5631 grep
   4249 mv
   3655 dpkg
   2873 cp

Vagrant: So, is the fish shell reproducible? I remember it did not used to be Jochen: You can check for yourself it was last time I checked. But looking through the other commands, neovim sadly is not. I hope we can fix that in future.
Vagrant: Oh, that s a nice URL to check for reproducible package you can just pass the source package name to check the current results? Jochen: Yes. Another one is udd.debian.org/reproducibility/, where you can list all packages of a Debian maintainer. It also lists source reproducibility and has nice filters as well.
Vagrant: What tools do you use specifically working on reproducible builds? Jochen: I don t have statistics for that, but I would say sbuild to build the package, debrebuild to reproduce it, and diffoscope to analyze the differences. Obviously I also need run apt source <package> or use git-buildpackage to get the sources and all the tools I mentioned above.
Vagrant: So how many packages are left to build reproducibly, and once those are finished, what is next? Jochen: Right now, reproduce.debian.net shows over 98% reproducibility, though there are still over 650 package left and some will probably need a lot of work. But actually I think making packages reproducible is just the first step. For me, this is a project to build confidence in the system. To reproduce a package we have two parts: the source of the package and the build environment. Fixing the packages means gaining confidence in the first part but we still rely on the individual build environments for each package as we need to use the same compiler that was used when the package was build initially. Because of this, we have to keep around every historical version of all toolchain packages. I really would like to remove this extra archive, which means we would have to rebuild all of Debian around release time. I am dreaming of a Debian release where you could bit-for-bit reproduce every package just from the released versions. Due to how Debian works, however, this is not a trivial rebuild and it would need some work on the infrastructure. By the way, initially there was a third component to pay attention to: any connection to the outside world during the build. Luckily we fixed the Debian build daemons to not allow network connections during the build some time ago.
Vagrant: Thanks for all that work, and taking the time to tell us a bit about yourself! Jochen: Thanks a lot for the interview!


For more information about the Reproducible Builds project, please see our website at reproducible-builds.org. If you are interested in ensuring the ongoing security of the software that underpins our civilisation and wish to sponsor the Reproducible Builds project, please reach out to the project by emailing contact@reproducible-builds.org.

11 September 2026

Reproducible Builds: Reproducible Builds in August 2026

Welcome to the August 2026 report from the Reproducible Builds project! In our reports, we try to outline the most important things that we have been up to over the past month. As a quick recap about what problem our project intends to solve, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised or not. In this month s report, we cover:

  1. New updated SBOM specification from CISA.gov
  2. LWN on Bootstrappable builds at FOSSY 2026
  3. What s missing to have reproducible builds on PyPI?
  4. Distribution work
  5. Unreproducible builds under EROFS filesystem fixed
  6. Tool and documentation development
  7. Six new scholarly papers
  8. Patches

New updated SBOM specification from CISA.gov CISA, the Cybersecurity and Infrastructure Security Agency of the U.S. government published some the joint guidance entitled Minimum Elements for a Software Bill of Materials (SBOM), which updates and supersedes the baseline 2021 version covered in previous editions of these reports. Whilst the PDF is worth skimming, the interesting changes include that the specification now mandates standard cryptographic hashes: unlike earlier standards that allowed hash omission or manifest-only parsing, hashes must be computed from the output. This is is important for reproducible builds, as it ensures the recording of the metadata required to demonstrate the shipped software matches the build output precisely where applicable. In addition, where the top-level only dependency limitation that was present in the 2021 version has been removed in favour of complete coverage with no minimum depth. That is, SBOMs are expected to reflect all linked libraries, vendored dependencies and other build-time inclusions.

LWN on Bootstrappable builds at FOSSY 2026 In the Toolchains and Other Development Tools track at Software Freedom Conservancy s FOSSY 2026 in British Columbia, Canada, Timothy Sample gave a presentation on bootstrappable builds. This presentation was then covered in a Linux Weekly News article by Jake Edge entitled Bootstrappable builds: how and why, which serves as an excellent introduction to the concept:
The basic idea behind bootstrappable builds is to create a system that can be built without relying on pre-built artifacts. Can we go from zero to the modern day without having to just assume the existence of these already-built-for-us artifacts? The classic recipe for yogurt requires some yogurt to start the process, which is like how we normally build a C compiler today we start with an existing C compiler binary. You might think about making sourdough bread with your grandmother s starter brought over from the old country; we re basically making C compilers with Dennis Ritchie s starter carried over from Bell Labs .
The article, which goes on to cover GNU Mes and other projects that overlap with Reproducible Builds, also has a number of thought-provoking comments.

What s missing to have reproducible builds on PyPI? Core Python developer, Brett Cannon wrote an interesting blog post this month addressing What s missing to have reproducible builds on PyPI, the official public repository for third-party Python software packages:
The reason I like the idea of making reproducible builds work is that I think it can be done in such a way as to not require any work on the part of the producer of a distribution (which is a technical term for sdists or wheels, i.e., the people who upload stuff to PyPI), and thus make reproducible builds very low-friction for people to opt into supporting. [ ]
Brett goes on to outline What s missing from the specs and how reproducibility might be visible on PyPI to consumers:
Assuming all of this comes to pass and we record the where the source code is that went into a distribution and the software used to make the distribution, how do we make it useful to people? Does every person who cares about having a secure supply chain have to rebuild everything they use themselves? Is there some way for even people who don t care about this stuff to benefit? [ ]

Distribution work In Debian this month, 23 reviews of Debian packages were added, 28 were updated and 27 were removed this month adding to our knowledge about identified issues. A number of issue types have been updated as well, such as the addition of a new toolchain issue related to python-traitlets [ ], and the note for an existing issue related to texi2html was updated as well [ ]. Lastly, Bernhard M. Wiedemann posted another openSUSE monthly update for their reproducibility work there.

Unreproducible builds under EROFS filesystem fixed Martin Pitt reported on Fosstodon that they had identified an issue where the mkfs ( make filesystem ) command for the EROFS (Enhanced Read-Only File System) subsystem of the Linux kernel did not have sorted extended file attributes, leading to reproducible builds. Thankfully, Martin also reported that they had fixed this in a commit to the kernel which order[s] each inode s xattrs by name so that images stay reproducible . [ ]

Tool and documentation development diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes, including preparing and uploading versions 327, 328 and 329 to Debian. In particular, he ensured that diffoscope did not require python3-guestfs in the autopkgtests on 32-bit architectures in order to fix Debian bug (#1144372) [ ]. Colin Watson made an additional change, handling a potentially missing openssh-client package when running the autopkgtests [ ], and Jochen Sprickerhof made a similar change to cope with missing cpio and qemu-img functionality [ ] whilst also updating the XML comparator to be considered when comparing SVG images [ ].
Yet again, there were a number of improvements made to our website this month as well. For example:
  • Chris Lamb added draft for a Gothenburg summit-related news article. [ ][ ]
  • Holger Levsen then published the same article. [ ][ ]
  • Lastly, a large number of commits were pushed comprising an interview with Reproducible Builds developer Jochen Sprickerhof to be published within the next week. [ ][ ][ ][ ][ ][ ][ ]

Six new scholarly papers Jens Dietrich, Spencer Sun, Tim W. White and Behnaz Hassanshahi (the result of a collaboration between Victoria University of Wellington and Oracle Australia published a paper this month entitled No Snake Oil: Verifying Python Package Builds. Drawing on the metaphor of snake oil , that is, a fake or ineffective medicine or solution sold with exaggerated claims of curing or fixing everything, the authors write that
Two tools that are designed to automate [PyPI] rebuilds and run them at scale are macaron and oss-rebuild. We study 12,180 popular releases from PyPI and find that the byte-for-byte equivalence rate is generally low. We analyse the reasons why they produce different wheels, and find that equivalence between the original and rebuilt wheels can often still be established, preserving most of the guarantees users expect from rebuildable releases. We present and evaluate daleq4py, a tool to establish the equivalence of Python wheels through the kernel of a normalisation function that is based on provenance-preserving datalog rules. Experimental results show that daleq4py substantially expands the set of rebuilds that can be accepted as equivalent.
The full PDF of their paper can be viewed online, and Jens Dietrich to our mailing list to announce the availability of both the paper and the daleq4py tool itself.
Dimitri Kokkonis, Micha l Marcozzi and Stefano Zacchiroli published an article this month titled Not In My Git Yard: Catching Backdoors at Commit and Release Time on the topic of code-level backdoors that is, stealthy code changes that grant hidden privileges via secret triggers . These issues:
pose a persistent threat to opensource software. Known attempts to inject such backdoors into widely used projects through malicious commits, tampered release packages, or compromised third-party dependencies, were stopped only by luck and manual review. Existing Continuous Integration (CI) pipelines cannot detect these attacks, and downstream binary analysis tools require substantial manual effort. In this work, we present Lily, an automated approach that strengthens open-source development and release processes against backdoor injection. Lily integrates a backdoor detection mechanism into (1) CI pipelines to block malicious commits, and (2) release vetting workflows to prevent tampered releases or compromised dependencies from entering large ecosystems, such as Linux distributions.
The full PDF can be read online.
Ranindya Paramitha and Laurie Williams of North Carolina State University along with Christian K stner of Carnegie Mellon University published a paper this month with the title of The Software Supply Chain as a Market for Lemons: A Multivocal Review of Trust Signal Collapse. (A lemon in American English, is a vehicle that turns out to have several manufacturing defects .) Their abstract is as follows:
Practitioners evaluating open-source dependencies rely on cheap trust signals, e.g., stars, download counts, and contributor activity, as substitutes for direct code inspection, assuming those signals reflect genuine trustworthiness. Prior work has documented individual signal gaming, but the landscape of collapses across all dependency-adoption signals, as well as the ecosystem s response, remains unexplored. The goal of this study is to aid software practitioners in understanding the reliability of dependency adoption trust signals, such as download counts and contributor activity, by conducting a multivocal review of 252 Google Search sources and 870 Reddit threads.
Worryingly, after their review, the authors conclude that cheap trust signals collapse under three simultaneous forces: adversarial manipulation, gaming techniques indistinguishable from legitimate behavior, and non-adversarial AI-driven inflation. The full PDF of the paper is available online.
Julien Malka, Aman Sharma, Martin Monperrus, Stefano Zacchiroli and Th o Zimmermann published a paper this month on Trusting-Trust Attack against an Entire Linux Distribution through Binary Manipulation:
Ken Thompson s trusting-trust attack, in which a compromised compiler backdoors the programs it builds and reproduces the backdoor in subsequent rebuilds of itself, is widely regarded as a threat specific to compilers. We show that it is not. We construct a complete trusting-trust attack around GNU strip, an ordinary build utility that neither inspects nor generates source code, using only manipulations of finished ELF files.
Scarily, in the authors example, a single tampered strip in the binary seed implants a payload that propagates from one generation of strip to the next and survives into the final standard environment after the seed leaves the dependency closure [ ] without failures and backdoors . A full PDF of the paper is available for download online.
Mehdi Keshanimm, Amirhossein Rahmati, Mohammad Hossein Aref and Abbas Heydarnoori published a paper that is currently under review at Emperical Software Engineering titled AROMA+: A Study of Factors Affecting Reproducible Builds in the Maven Ecosystem. (Maven is a/the build automation tool used for Java projects.) In their paper, the authors note that
[ ] reusing external software in a project presents a security risk when the source of the component is unknown or the consistency of a component cannot be verified. The SolarWinds attack serves as a popular example in which the injection of malicious code into a library affected thousands of customers and caused a loss of billions of dollars. [ ] Our research aims to support [reproducibility] efforts in the Maven ecosystem through automation. We investigate the feasibility of automatically finding the source code of a library from its Maven release and recovering information about the original release environment. Our tool, AROMA+, can obtain this critical information from the artifact and the source repository through several heuristics and we use the results for reproduction attempts of packages on Maven Central.
The full PDF of their article can be downloaded online.
Lastly, Oreofe Solarin, Kelechi Kalu, James C. Davis and Paschal Amusuo published a paper this month titled Reproducibility is Not Enough: Artifact Verifiability in Decentralized-Build Package Ecosystems:
[A]rtifact verification requires more than deterministic builds: a verifier must also recover the source state, build environment, dependencies, and build instructions that produced the artifact. Decentralized-build ecosystems make this difficult because artifacts are produced through heterogeneous tools, maintainer-controlled workflows, and fragmented metadata. As a result, it remains unclear how often artifacts in these ecosystems can be independently verified. This paper studies artifact verifiability across four popular decentralized-build package ecosystems. We define an independent verifier model that relies only on registry-derivable metadata and an artifact comparison model with tiered equivalence levels. We implement these models in an Artifact Verification Pipeline and use it to measure artifact verifiability across the target ecosystems.
The authors conclude that beyond build determinism, verifiability is limited by missing source and build metadata, implicit release transformations, and unconventional build practices . A PDF of their paper can be reviewed online.

Patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where applicable or possible. This month, we wrote a large number of such patches, including:

If you are interested in contributing to the project, please visit our Contribute page on our website.

7 September 2026

Colin Watson: Free software activity in August 2026

My Debian contributions this month were all sponsored by Freexian. You can also support my work directly via Liberapay or GitHub Sponsors. Personal note This month, my Dad unexpectedly passed away after a short illness. As a result I obviously got less work done than usual, and I still have a lot to take care of (since I m the executor of his will, as well as helping with funeral arrangements) while grieving and generally having less focus and energy. Having routine work to do is one of the ways I cope with this sort of thing, but all the same, I hope people will bear with me and maybe remind me if I seem to be dropping the ball on something you especially need. LLM vote [Content note: strong opinions.] I voted in General Resolution: LLM usage in Debian. My vote was pretty much the opposite of what ended up winning, so I m quite disappointed. My personal opinion is that LLMs are cognitive hazards to their users that impose ecological costs far out of proportion to their utility at a time when the world absolutely cannot afford them. When the impossible economics of the large commercial models are finally allowed to catch up with reality, I expect there to be significant macroeconomic consequences, and that people who have become dependent on them will have problems; and who knows what the copyright situation on their output really is. I m not convinced that local models are better enough on these axes to be worth the costs. Debian s direct contribution to all that will be negligible on a global scale, and even the most radical proposals in the GR didn t expect that we could do much about upstreams that have gone all-in on LLMs. Even so, I d hoped that my fellow developers might be more willing to lean on our position in the free software ecosystem to make at least a moderately radical statement. Instead, we ve at best presented an undistinguished fence-sitting position to the world, and further entrenched the idea that humans can reliably do a good job of reviewing the output of tools that are designed to produce output plausible to humans. I certainly don t trust my own code review skills that far. Since I ve never voluntarily used an LLM (not counting LLMs being foisted on me by things like search results, support chatbots, or incoming pull requests, regardless of whether I asked for them), and don t intend to for the foreseeable future, I doubt this will change much for me in terms of the way I work. The winning option is a very weak one that imposes no new requirements on developers, which means that it also does nothing to stop me continuing to reject LLM-generated material from Debian bug reports and merge requests in my areas of responsibility. I know this probably won t do much to satisfy people who have decided that Debian is slop now, but it s the best I can do. OpenSSH I finally landed the GSS-API key exchange package split in our OpenSSH packaging. Here s the NEWS entry:
openssh (1:10.4p1-5) unstable; urgency=medium
  The openssh-client and openssh-server packages no longer include GSS-API
  authentication and key exchange support; this adds pre-authentication
  attack surface and generally increases complexity, and should only be used
  where specifically needed.  Users who need these features should install
  openssh-client-gssapi or openssh-server-gssapi instead.
 -- Colin Watson <cjwatson@debian.org>  Sun, 23 Aug 2026 17:39:55 +0100
I fixed a flaky autopkgtest. I upgraded from 10.4p1 to 10.5p1, which was a good test of keeping openssh and the new openssh-gssapi source package in sync. PuTTY I upgraded from 0.84 to 0.85. Python packaging New upstream versions: The version treadmill continues: we ve just finished dropping Python 3.13 as a supported version, so now we ve started working on enabling Python 3.15 as a supported version. Maximiliano Curia has been very helpfully driving this. I didn t get as much done here as I d have liked (see the top of this post), but I fixed a couple of packages: Other build/test failures: I fixed some other bugs: bugs.debian.org I deployed the fix for Invalid link rel= canonical on bugs.debian.org. In the process I found a few bugs in recent undeployed code and fixed them.

Vincent Bernat: Sidenotes with CSS anchor positioning

I am a heavy user of sidenotes:1 they keep optional content next to the text instead of sending the reader to the bottom of the page and back. Tufte CSS renders them without JavaScript but only accepts inline content. CSS anchor positioning, now supported by recent browsers,2 is an elegant alternative. Sidenotes can hold several blocks, still without JavaScript, and fall back below the paragraph referencing them on narrow viewports and older browsers. In 2023, Eric Meyer demonstrated this technique in Nuclear Anchored Sidenotes. The main improvement over other solutions is that the notes can sit anywhere in the HTML document. You can place them after the paragraph referencing them, as regular block elements for text browsers, screen readers, feed readers, and reader mode to render them properly:
Sidenotes rendered in Lynx appear after the paragraph they are called from.
Rendering in Lynx, a text browser
When the viewport is too narrow or the browser does not support CSS anchor positioning, you can style them so the reader can skip them or glance at them without losing their position in the text:
Sidenotes rendered on a narrow viewport appear with a distinctive typography after the paragraph they are called from.
Rendering below the paragraph on a narrow viewport
Once the viewport is large enough, they appear in the margin, at the same vertical position as the matching reference mark, unless they would collide with a previous sidenote, as in the example below:3
Sidenotes rendered on a large viewport appear in the margin. There are two of them. The first one is vertically aligned with the matching reference mark, while the second is rendered below as it would collide with the first otherwise.
Rendering in the margin on a large viewport
The gist of CSS anchoring is to position an element relative to another element the anchor. For the sidenotes, the anchor is the reference mark. I use the following markup, with a data attribute to specify the anchor name:
<sup id="fnref:YYY" data-anchor="--lf-sn-YYY">
  <a href="#sidenote-YYY">1</a>
</sup>
The matching note is an <aside> element carrying the same data attribute for the anchor name. We put it after the paragraph holding the reference mark:
<aside role="note" id="sidenote-YYY" data-anchor="--lf-sn-YYY">
  <sup>1</sup>
  <p>A first paragraph.</p>
  <p>A second paragraph.</p>
</aside>
On a narrow viewport or when the browser is too old for CSS anchoring, we style the sidenote, which stays below its paragraph, with a muted color:
aside[role="note"]  
  margin-block: 1rlh;
  color: #444;
 
On a wide viewport and when the browser is recent enough, we move the sidenote to the right margin:
@supports (anchor-name: attr(data-anchor type(<custom-ident>)))  
  @media (min-width: 72rem)  
    main  
      position: relative;
      sup[data-anchor]  
        anchor-name: attr(data-anchor type(<custom-ident>));
        /*   anchor-name: --lf-sn-YYY */
       
      aside[role="note"][data-anchor]  
        anchor-name: --lf-sidenote;
        position: absolute;
        position-anchor: attr(data-anchor type(<custom-ident>));
        /*   position-anchor: --lf-sn-YYY */
        top: max(anchor(top), anchor(--lf-sidenote bottom, -1rlh) + 1rlh);
        left: 100%;
        margin: 0 2rem;
        width: 18rem;
        color: inherit;
       
     
   
 
attr() extracts the anchor name for the reference mark from the data-anchor attribute. It returns a string, unless we specify a CSS unit or a type, like here: the browser parses the data attribute as a custom identifier, which anchor-name validates as a dashed identifier, a custom identifier starting with two dashes.4 The note itself is absolutely positioned past the right edge of the main block. It selects the matching reference mark as its anchor with position-anchor set to the value of the data-anchor attribute. Each note is also an anchor named --lf-sidenote. We use it to keep the next note from colliding with this one. The anchor() CSS function lets us position the note s top edge relative to its anchor: anchor(top) aligns the top edge of the note with the top edge of the reference mark. It can also take another anchor as a parameter: anchor(--lf-sidenote bottom) would align the top edge of the note with the bottom edge of the closest preceding anchor named --lf-sidenote so the previous note.5 Like attr(), anchor() accepts a fallback value as its second parameter and use it when the named anchor does not exist. The top property handles three cases, illustrated in the following diagram:
Diagram of three sidenotes anchored to their reference marks. The first one is aligned with the top of its own reference mark, as no note comes before it. The second one would overlap the first, so it takes the bottom of the first note as anchor and sits one line below it. The third one comes far enough down the page to align with its own reference mark again.
The three cases for the vertical position of a note
  1. The first note s top edge aligns with the top edge of its reference mark: as there is no previous note, anchor(--lf-sidenote bottom, -1rlh) + 1rlh resolves to 0 and max() returns anchor(top).
  2. When the reference mark of a later note sits above the bottom of the previous note, plus some vertical space, the note goes below the previous one to avoid a collision. max() returns anchor(--lf-sidenote bottom) + 1rlh.
  3. Otherwise, the note s top edge aligns with the reference mark s top edge, as max() returns anchor(top).

Have a look at the complete stylesheet, which also adapts the reference mark to the location of the note: a arrow when the note sits below the paragraph, a arrow when it moves to the margin. Gwern s Sidenotes In Web Design lists more implementations and their trade-offs. Some bloggers aim to write a post in 30 minutes. I planned to publish three web-related articles this weekend. Instead, I spent an inordinate amount of time elsewhere: about 15 commits on the build system, a pull request to update CSS highlighting for nested selectors in Pygments, and a small correction to MDN s article on the anchor() CSS function. The SVG illustration took a bit less than an hour and the article itself a handful of hours. The attr() function came in after I thought inline style looks ugly, isn t there a better way? But, hey, I still think this is worth it!

  1. My PhD advisor told me this is unwise.
  2. The first bits of anchor positioning are supported from Chrome 125 (May 2024), Firefox 147 (January 2026), and Safari 26 (September 2025). Before Safari 26.5, sidenotes may collide due to a bug in how dependency chains are handled. You can detect this situation with some JavaScript. It is, however, not needed in the solution described here as we depend on a more recent feature.
  3. If you noticed the runt in the first note, I share your pain and lament that Firefox does not implement text-wrap: pretty.
  4. Typed attr() is supported from Chrome 133 (February 2025), Firefox 155 (September 2026), and Safari 27 (September 2026). Check Una Kravets article for details. To support more browsers, you can inline the anchor name and the position anchor directly in the HTML:
    <sup id=" " style="anchor-name: --lf-sn- ">
      <a href="#sidenote- ">1</a>
    </sup>
    
    Managing Anchor Associations With Data Attributes and Advanced attr(), by Daniel Schwarz, explores CSS anchors and typed attr() in more detail.
  5. The exact rule for the target anchor element is more complex: if an ancestor of [the note] satisfies the following conditions, return the nearest such element to [the note]. Otherwise, return the last element in tree order that satisfies the conditions. One of these conditions is that [the candidate] is an acceptable anchor element for [the note], which requires that [the candidate] is laid out strictly before [the note], where the relevant clause is that [the candidate] is either not absolutely positioned or occurs earlier in the flat tree order than [the note].

31 August 2026

Russ Allbery: Review: The Hands of the Emperor

Review: The Hands of the Emperor, by Victoria Goddard
Series: Lays of the Hearth-Fire #1
Publisher: Underhill Books
Copyright: January 2019
ISBN: 1-988908-15-9
Format: Kindle
Pages: 739
The Hands of the Emperor is a self-published political fantasy novel. It's the recommended first book (although not the first published book) in a complicated set of interrelated series. I was not able to definitively confirm that Underhill Books is Goddard's self-publishing press name, but the press does not appear to have an Internet presence apart from Goddard's books and her books appear to be using the standard self-publishing channels. Cliopher Mdang is the personal secretary of the last emperor of Astandalas, the magical heart of Zunidh, a man worshiped as a god. The emperor's word is absolute, his magic supports the health of the entire world, and he cannot be physically touched without risking physical damage and severe political and religious punishment. Cliopher is one of the emperor's closest associates, but the distance between them is still vast. It therefore represents a terrifying and dangerous breach of etiquette for him to suggest the emperor may enjoy a vacation on a tropical island near Cliopher's remote home. The emperor's acceptance of the invitation is even more startling. The emperor has opinions about his life as the emperor that no one had guessed. Cliopher has not assimilated as completely into the bureaucratic machinery of the empire as it first may appear. And Cliopher's family have vastly misunderstood the nature of his role in the emperor's government. I find the marketing blurb for this book unfortunate since, at least to me, the emphasis on physical touch and intimacy implies that The Hands of the Emperor is a romance novel or at least has significant romantic elements. I've been aware of this book for years but put off reading it because I wasn't quite in the mood for that story. This is not a romance novel; there is no romance in this book whatsoever. It is a political fantasy, both in the sense that it is set in a secondary fantasy world with magic and (apparently) some form of interplanetary travel, and in the sense that it is a fantasy of governance. When I say that this book blew up in certain corners of the Internet during the pandemic, I think you will still underestimate the passion of its advocates. I heard about this book constantly, in a way that reminded me of Kushiel's Dart and the time when fans of Jacqueline Carey would bring her up in every fantasy conversation, or when we created a Usenet newsgroup for The Wheel of Time mostly to get the voluminous conversations off of the regular SFF newsgroup. I'm one of those mildly contrarian people for whom that degree of enthusiasm is a little off-putting, which is another reason why I resisted buying a copy for years and only read it in 2026. It's delightful, although also a bit embarrassing, when the book everyone was in love with turns out to be just as good as everyone said it was. I adore stories about friendship, and this is one of the best stories about friendship that I've ever read. It is a very, very slow burn, but I also thought the first three quarters of the book was exquisitely paced. There were long sections where not very much was happening, and yet I couldn't put the book down because there was so much subtle character work just beneath the surface. Almost all of the novel is told in tight third person from Cliopher's perspective, and I thought that was an excellent choice. Neither Cliopher nor the narrator comment on things that Cliopher finds obvious, which is both immersive and critical to the pacing. There are discoveries for the reader throughout the book, the sort of discoveries that make pieces fit together satisfyingly in retrospect, and the reader stays sufficiently ahead of the misunderstandings of Cliopher's friends and family that one also gets the joy of watching other people discover things that one figured out a hundred pages earlier. It helps that I truly liked nearly everyone in this book. There are no real villains, only a few supporting characters whose role is to be irritating or corrupt. If you're looking for a lot of conflict and drama, you may want to save this book for a different mood, but if you're in the mood for a varied collection of fundamentally good characters working methodically through the complexities and obstacles of politics and social systems to improve the world, there are few books I would recommend more. Goddard achieves one of the hardest tricks of slow burns: steady forward progress that does not rely on reversals, misunderstandings, or the friendship equivalent of the third-act breakup. This book spends 700 pages building towards a climax that managed to be worthy of all 700 pages without ever annoying me with artificial obstacles, and that's quite a feat. I've not said much about the details of the plot. There is one it's not just character work but I think this book benefits immensely from going in as blind as possible. I found the twists and turns and growing revelations so deeply satisfying that I don't want to rob any other reader of the experience. The fantasy world-building is intriguing but a bit unsatisfying because it is so unexplained. We get a few details of the magic system, but since Cliopher has no magic, he isn't that interested in the details. There is a catastrophic magical event in the world background, and we learn some of the details of its practical effects, but the nature of the world before the cataclysm is so obvious to the characters that it's never explained. I'm not even certain that this civilization is interplanetary; that feels like the implication of how characters talk about multiple worlds, but the method of travel is left entirely undefined. This might be frustrating to some genre readers, but I personally enjoy books where the world-building is a bit mysterious. It's a good reason to read more of Goddard's books set in the same universe. This was my favorite of the books I've read so far this year, but I do have one caution and a couple of caveats. The caution is that Cliopher comes from an island culture based heavily on (I think) Polynesian cultures. That culture is very central to the story and is treated with considerable respect, but I still get a bit nervous when a Canadian author from Nova Scotia with an academic background in European medieval studies writes a story focused this deeply on a non-European culture. Nothing about her portrayal seemed off to me (although there is a very clunky and ham-handed scene about a different native culture that worries me), and for all I know she has family background or other connections to the culture she is borrowing from, but it's possible I missed serious problems. The flip side of that caution is that I'm delighted to see a fantasy author drawing on a non-European culture, and I thought the clash of cultures was very well-handled. The first caveat is that the story is very focused on good governance, but both the process and the details of that governance are not going to satisfy someone reading primarily for the politics. The policies and reforms are very standard 21st century progressive material that felt a bit out of place in a quasi-medieval world with magic and airships. Their implementation is not the point of the story, and is therefore heavily backgrounded, but that means Goddard barely mentions the inevitable practical implementation difficulties and does not discuss how they're overcome. The world structure also means that Goddard can make use of the favorite cheat of political reformers in fiction: Absolute monarchy lets you enact a political agenda without having to do the hard and frustrating work of persuasion or political (or actual) warfare. This objection is not entirely fair because we do get some memorable scenes of persuasion, but the political portion of the plot is unrealistically devoid of setbacks or resistance that goes beyond token arguments. Whether this will bother you will depend heavily on what parts of the book you'd rather focus on. I can see why this was such a popular pandemic read: The Hands of the Emperor is focused tightly on the joy of competent people fixing things and does not focus on the arguments, division, or polarization. The heart of the book is the friendship and characterization of some deeply admirable people, and the political reform is incidental background material. I suspect this is the right choice for readers who aren't political junkies, but I kept having the niggling objection that the politics felt a bit too pat and simplistic. Goddard stressed that the characters were investing considerable effort, but even still, it is not this easy to change the direction of a political system and idealistic plans usually do not work out this neatly. The second caveat is that, as previously mentioned, I thought the pacing was excellent for about three quarters of the book. Goddard is building towards a grand climax, and I think she built a little too much and tried to make the climax a bit too grand and risked over-egging the pudding. That made the payoff feel a bit belabored to me. I still enjoyed it, and parts of it are wonderfully emotional, but I think the ending might have been stronger if Goddard had dialed Cliopher back just a little and tightened up the climax a touch. That said, this book fully commits to being a sprawling slow burn and that's part of its appeal, so it's probably better for Goddard to err in that direction than it would have been to cut short the denouement. This is one of those books that I'm not sure would exist without self-publishing. It's a little too long, a little too political in the wrong ways, a little too devoid of the typical sorts of conflicts expected in a fantasy book, and too determined to be its own peculiar thing. I think it would scare off publishers. Unlike some self-published books, though, I didn't notice any obvious editing flaws or lack of polish. It's one of those glorious novels that is so very much its own type of story that it provides an experience that would be hard to replicate with another book. I was so deeply satisfied by this book. It's a wish-fulfillment political fantasy full of diligent restraint and competence porn, so you have to be in the mood for that. This is not the book to read when you're feeling cynical, or are in the mood for action and high drama. But if you're in the mood for a long, slow, open-hearted story of friendship that offers the fantasy of giving truly good people enough power to be effective, I highly recommend this one. Followed in the direct sequel sense by At the Feet of the Sun, but there is a very complex story progression in this world that I think I'd have to read all the other books to understand. This was such a satisfying and complete experience that I'm not in a hurry to figure out which Goddard book to read next, but I'm sure I'll be returning to this world at some point. Rating: 9 out of 10

25 August 2026

Antoine Beaupr : A more nuanced view of LLMs

Also in this series:
After ranting and railing about LLMs or "AI" as the optimists (or accelerationists?) call it, I figured it might be important to be a little more honest about my use of LLMs and how I think about it more practically in the world.

The Debian vote context This is not a coming out. I am not using LLMs on a daily basis, and this blog is, again, written out of my cold dead hands in a dying world, with over-engineered hardware and (to a certain extent, hi Emacs!) software, powered by 100% green energy built on stolen land. There is a vote going on in Debian. If you're unfamiliar with it, you can catch up at LWN. So far I've essentially said "LLM is bad" which is not a very balanced or useful opinion. Obviously, people are using LLMs, sometimes unknowing or unwillingly, and we need to take that into account. Furthermore, there has been many different blog posts on Debian planet about this. Some that I found balanced, good summaries, even if I didn't fully agree with them, at least some did the basic civil service of being short. But others were just not only Wrong but also so long that I couldn't finish that I just had to write something.1 This is not an explanation of the ballots, nor how I will vote. This vote is Debian's failure of framing that debate in a reasonable way: we have 8 options on the ballot with many duplicates. We have failed to do the hard work of summarizing and aggregating options into a meaningful set. I doubt the final vote will represent a readable position we can rally around. I have not read the two months of debates on the topic either. Normally, before voting, I take a cursory look at the debate to see points of view I might have missed. But in this case, it will just make me sad, add noise, and I'm already pretty sure on where I stand on this. So let me describe how I use LLMs and how I think they fit in our work, as computer engineers and hobbyists.

My LLM use

Debian Packaging An astute reader has pointed out that I maintain a package in Debian made to use Anthropic. It's actually multiple packages: As I previously explained in response, I am not entirely comfortable with this work: it's a compromise. In fact, I first uploaded llm to the contrib section of Debian, where we keep software that depends on other non-free software, but I was told that, since yt-dlp was in main, llm belonged there as well. So I moved it to main, alongside similarly controversial tools like llama.cpp or the python-openai library.

OpenAI and Anthropic usage An important part of my work is technology watch. I keep tabs on thousands of (new and old) software projects, follow news, and generally try to keep my skills up to date. It's a pretty impossible race, especially as I grow older, but I still think I'm doing the right choices in my job. Testing large language models is part of that work. At first, I was using ChatGPT's web interface, but it was annoying to copy-paste things into a browser, so I looked for different interfaces. For a while I tried gptel, a "simple, extensible LLM client for Emacs" but I found it kind of terrifying. Giving a LLM control over an Emacs buffer seems like a security nightmare, so I stopped doing that. So I use the llm command-line tool to talk to Anthropic's API. I started that in the summer of 2025, when I bought 20$USD of API credits. Before that, I paid for a ChatGPT subscription and then OpenAI credits, which expired and sent me over to Anthropic, which seemed then to have better ethics. As it turns out, Anthropic is also happy to work for the US military (which is a big red line for me). Anthropic also won't let you talk about the genocide in Gaza, it is destroying physical books, and is blackmailing us to use their product for security coverage. Needless to say, Anthropic and "Claude" are not my friends, but they seem like the lesser evil in current "frontier models". So I have renewed, a couple of weeks ago, another 20$USD of API credits with Anthropic.

Actual prompts and responses So what does 20$ give you at Anthropic anyways? What am I using LLMs for and how? The neat thing with llm is that everything is logged in a sqlite database, so there are some answers that are easy to get:
> llm logs status
Logging is ON for all prompts
Found log database at /home/anarcat/.config/io.datasette.llm/logs.db
Number of threads logged:   7
Number of turns logged:     12
Number of legacy conversations: 543
Number of legacy responses: 970
Database file size:         9.61MB
That is 10MB of logs, with about a thousand prompts. My logs go back to 2024-03-07, a little over two years ago, and include a mix of Anthropic and OpenAI responses. I used it more in 2024 than 2025, and if the trend continues, I will have used it less in 2026 again:
> llm logs list -n 0  --json   jq -r .[].datetime_utc   sed 's/-.*//'   sort   uniq -c 
    527 2024
    357 2025
     98 2026
It looks like about 10 prompts per month right now, down from a peak of about 60 per month in 2024. It's pretty difficult to analyze those actual logs to get more patterns and I won't run the prompts through a model again to process them.

How I'm using models now At first, I was using it partly for benchmarking model's capabilities, like Simon Willison does with his pelicans, clearly not trusting its output. But I was impressed by the capacities of the Claude Opus 4.5 model when it wrote this script in January. Impressed, but also scared: it's the first time I felt I could delegate the entirety of my programming to a model. Just run the code, if it works, it works, right? So what do I use it now? As an example, here are the 10 last prompts in my history:
  1. there is now Claude 5, and a fable model, maybe you know about it?
  2. impress me
  3. not impressive, i already know all of this
  4. chat
  5. in postfix, i have a 300k mailing that happens regularly here. normally, it delivers within about...
  6. is there a way i could have drained the maildrop queue faster without removing the milter?
  7. the problem was that rspamd was timing out on the FUZZY_CALLBACK check. how do i disable that?
  8. how do i disable all spam checks? i just want rspamd to add dkim signatures
  9. how do the default_destination_concurrency_limit and initial_destination_concurrency settings int...
  10. mic check
The first one was me trying to confirm which model I am using, which is not always obvious when going through the whole llm stack I've been using. The following two are an attempt at seeing what the model is capable of and I was "not impressed", to which Claude answered that I have a "high bar", which, fair enough. The chat is me failing to use a command line, which shows that perhaps I need to readjust that "high bar", again. The next five are a rather embarrassing debacle in a large Postfix mailing that went sideways, and where I couldn't find an actual Postfix expert of my level to help. The fabled Claude Fable 5 answered rather correctly, but dangerously, that I could empty the queue by disabling the non_smtpd_milters. What Fable (and myself) did not realize is that the milter was also adding DKIM signatures, so while the mailing was expedited, it was done without those precious signatures, which got us promptly blocked at Gmail. We have recovered since, and, thanks to the model and reading the Postfix manual for the hundredth time, that pickup(8) is single-threaded and that we needed to review the architecture of that mailing (and our spam filters) a bit. Many tickets ensued. The last one is a test I did to make sure my last uploads of llm-anthropic and its dependency worked correctly.
Note that the above excludes 5 questions I asked Anthropic while writing this article, where I asked for synonyms and "what nanometer scale are arduino processors built from? how is an arduino CPU printed?", a question which Wikipedia furiously evades providing a good answer.
Those prompts are pretty typical of my LLM use: I'm testing the models to see if they work at all, but also, out of desperation, I fire off a prompt after I fire off questions to colleagues or search engines (in that order). It's often weird edge cases like the Prometheus query language, Python's matplotlib, LaTeX, Elisp, optimizations, and so on. I use models for translation a lot. Being fully bilingual, it is common for me to think of a word in French or English and fail to find exactly the right word for that in the other language. Models help with that, and are also useful to find synonyms. Those are low-token uses that seem pretty innocuous to me, but I realize the irony of this after writing about the tower of Babel.

What I am not using models for I am not using models to write prose. I am not using models to read prose. If it's generated with LLMs, I stop reading. I am not using models to write code, with the exception of that single Python script above. I am generally not using models to review code, with exceptions. If I get stuck on a hard problem, I might feed a piece of code to the model. I repeatedly fed asncounter into Claude to try to fix a performance regression I had introduced. It found micro-optimizations that taught me a thing or two about Python's internal implementations, but overall, it was mostly a waste of time. This was in June 2025, so perhaps now models would fare better. I have not tried again. I am not using LLMs to do Debian packaging. When I can, I manually review the diffs of packages I upload into Debian, still, by hand. I do this for the reasons outlined in The Four Horsemen of the LLM Apocalypse, because I refuse to be complicit in the:
  1. aggressive and illegal scraping of the servers I steward
  2. world-wide computer hardware shortage (making it, by the way, nearly impossible to run presumably clean local models) and the attack on our job conditions (also discussed in The people vs the AI overlords)
  3. death of copyright and free software
  4. complication and enshifitication of everything, and the destruction of our communities
  5. the imperialist Nerd Reich that wants to take over the world
Like I reluctantly use Intel computers, I do fire off a prompt. But I still hold on to the dream that we can build communities of practice that hold human knowledge collectively and not offload that as a utility to some megalomaniac billionaire.

Their LLM use I am forced into So that's me. Clearly, I'm going against the grain here. Everywhere I look, I see LLM-generated code and projects. Slop and botnets have flooded the web. I use Wadamesh, clearly vibe-coded, because it's the best graphical interface for MeshCore that runs on portable devices. I wish it was made by a human, in a community I could participate in, but it isn't, and I don't. I package the above llm toolset, which is more and more vibe-coded, but I still review the diffs. And I have to say: I trust Simon here. The code is verbose as hell, feels overengineered, and llm feels slow, but it generally works, and Simon is still at the gate. The Anthropic SDK is another thing entirely. The 0.91.0 to 0.120 upload, for example, was nuts:
 806 files changed, 72281 insertions(+), 1478 deletions(-)
I explicitly did not review that entire diff. It feels like there's a lot of garbage there to just have a shim between a proprietary API and Python. But this is the hand I've been dealt.

Larger projects LLM use LLMs are being used in the Linux kernel, Firefox, rsync, Rust, and other places. I don't feel good about this, particularly in Rust, but they at least made a decent policy. I am glad GCC made a policy against LLM contributions and I support the human Emacs project. We need to have a set of foundational tools that are "clean" in the sense that they are built upon a community of people that understand how they are built. Maybe that's naive or even impossible. The Linux kernel and GCC, in particular, are massive projects that have long grown past the scale of a single person's understanding. But the theory was that a community of humans can understand collectively. Now we seem to be throwing up our hands and giving up on that community. That LLMs will just fix the problem, whatever it is. But we're all just one rug pull away from being completely incapable of managing those projects. The argument there is that we'll just switch to local models, but no one is actually doing that. All I see is people use local models as a corner case (for privacy) or as in theory, but in reality, everyone uses the centralized frontier models right now. We just can't fallback. We're in the same situation we were, a decade or two ago, when Microsoft decided it would kill free office alternatives by making Office free for non-profits. It worked: thousands, if not millions of schools, community groups and individuals stopped looking for alternatives (including free software but also "piracy") for Office and embraced what seemed like a generous offer. Now Microsoft pulled the plug and Over 170,000 Nonprofits Lost All Their Data. I'm afraid the rug pull on LLMs will be much worse: never mind that Linus won't be able to use his tireless helper to fix obscure kernel bugs; we're looking at a collapse of the economy so large that we are already talking about bailing out the companies responsible. In a sense, the most striking thing about the Debian vote is it has actually no option to completely refuse upstream LLM contributions. It seems the community has taken it for granted that it's now impossible to build Debian entirely without LLMs. We lost the battle even without a fight, it seems.

A plea for small If it has really become impossible for us to manage the complexity we have built, maybe it's time to stop and think about what we're doing in the first place. We're struggling to even bootstrap our current toolchain! This is one of the things I like the most about working on the mesh: it's low tech, small Arduino devices that is built with decades-old semiconductor processes that is understandable by human beings. Maybe the answer lies more in single-purpose devices like those communicators and simpler multi-purpose computers than what we have now, which is what the permacomputing movement is about. Small is beautiful, let's scale it down.

  1. and yes, I'm sorry this has gotten this long, I hope you will forgive those 3000 words.

23 August 2026

Russ Allbery: Long delayed haul

I haven't made a new book haul post in I don't know how long, so a lot of books have piled up and many have already been reviewed. Here's the overdue catch-up in case anyone is curious what books I am finding interesting before the reviews get posted. Ilona Andrews Magic Bites (sff)
Elizabeth Bear In the House of Aryaman, a Lonely Signal Burns (sff)
Oliver Burkeman Four Thousand Weeks (non-fiction)
Miles Cameron Whalesong (sff)
Lee Child Killing Floor (thriller)
august clarke The Felicity Complex (sff)
Alison Cochrun Here We Go Again (romance)
Dan Davies The Unaccountability Machine (non-fiction)
Linzi Day Midlife in Gretna Green (sff)
Linzi Day Painting the Blues in Gretna Green (sff)
Linzi Day Ties that Bond in Gretna Green (sff)
Linzi Day Spilling the Tea in Gretna Green (sff)
Michelle Diener Dark Ambitions (sff)
Michelle Diener Dark Class (sff)
Michelle Diener Collision Course (sff)
Michelle Diener Crash Course (sff)
Henry Farrell Underground Empire (non-fiction)
Kathleen A. Flynn The Jane Austen Project (sff)
Victoria Goddard The Hands of the Emperor (sff)
James Herriot All Creatures Great and Small (mainstream)
James Herriot All Things Bright and Beautiful (mainstream)
James Herriot All Things Wise and Wonderful (mainstream)
James Herriot The Lord God Made Them All (mainstream)
James Herriot Every Living Thing (mainstream)
Lauren Hough Monster of a Land (non-fiction collection)
Bethany Jacobs This Brutal Moon (sff)
Guy Gavriel Kay Written on the Dark (sff)
Mary Robinette Kowal The Martian Contingency (sff)
Ann Leckie Radiant Star (sff)
C.B. Lee Coffeeshop in an Alternate Universe (sff)
Fonda Lee The Last Contract of Isako (sff)
Julie Leong The Teller of Small Fortunes (sff)
Julie Leong The Keeper of Magical Things (sff)
R.Z. Nicolet The Cloak and Its Wizard (sff)
Claire North Slow Gods (sff)
Rebecca Ore Writing's Writing (non-fiction collection)
Suzanne Palmer Ode to the Half-Broken (sff)
Gareth L. Powell Fleet of Knives (sff)
Cameron Reed What We Are Seeking (sff)
Beth Revis Full Speed to a Crash landing (sff)
Beth Revis How to Steal a Galaxy (sff)
Beth Revis Last Chance to Save the World (sff)
Natalie Zina Walschots Villain (sff)
Jo Walton Everybody's Perfect (sff)
Martha Wells Platform Decay (sff)
James White The Galactic Gourmet (sff)
James White Final Diagnosis (sff)
The James Herriot books were ones my parents were getting rid of. I have them marked as mainstream fiction as a short-hand since "fictionalized autobiography" seemed like too much of a mouthful.

22 August 2026

Russell Coker: Links August 2026

This YouTube video about the Cashier Girl Meme is interesting in the context of AI systems that generate images of people and can communicate with people, hotter than any real human is an achievable goal [1]. Stand Up Maths has an interesting Youtube video about LLMs solving maths problems which I highly recommend watching (it does not require any real knowledge of maths), I think this opens the door to attacks on well established cryptologic systems [2]. Adam Conover made an insightful YouTube video about how and why Hollywood is now unable to make good sitcoms and why this is bad for society [3]. Sky Croeser wrote an interesting and insightful blog post about topics covered at the Digital and sexual citizenship in an age of social media bans: Interrogating the rights of children and young people conference [4]. Zane wrote a very informative blog post about reverse engineering a trojaned Android projector with Claude Code [5]. We need much better security on home networks to break the business model for this sort of thing. Renee Stonebraker s article Puritans Wouldn t Eat Pussy, So They Invented the Western has a lot of interesting information about early days of colonising the US, and not much about eating pussy [6]. IFLScience has an interesting article about brinicles, icicles of brine that form under sea ice [7]. Nautilus has an interesting article about the Silurian Hypothesis [8]. The Conversation has an intersting article about the pros and cons of no-till farming [9]. Cold War is a 365tomorrows story about bio-warfare which raises several disturbing possibilities we need to guard against [10]. Scott Santens wrote an insightful article describing how a land value tax would reduce rent and solve the housing shortages [11]. Positive News has an interesting article about using OnlyFans to teach people about climate change [12]. The Conversation has an interesting article about cultural safety in healthcare, sounds good, and while we are at it lets deal with sexism [13]. Doctoreww has an interesting web page about ways of displaying different strings to humans and machines, this could result in you running a different command to what you thought you copied from a web site or defeating tools designed to block hostile content [14]. Cory Doctorow wrote an insightful article Commentary Hell is Other People about the way rich people want to use AI to replace all people [15]. Also psychologists who help rich people accept being greedy are worthy of a Luigi The research article Worship me at the office altar: Why narcissistic leaders resist remote work is interesting, yet another reason to get rid of narcissistic executives [16]. Renew Economy has an interesting article about clean up costs for mining (which is usually left for the government to pay) and how this could impact renewable energy production facilities [17]. Elvira Bary wrote an insightful article on the Russian financial collapse that is happening now [18]. The Guardian has an interesting article about Afro-American women who travel to South Korea for healthcare because of problems with racism and sexism in American hospitals [19]. The Conversation has an interesting article about the potential for disabled people to be more productive in space than non-disabled people [20]. Krebs has an interesting article about LG banning residential proxy code from apps after the LG store was found to have such code in 42% of it s apps [21]. Robert B Shpiner wrote an insightful article for The Guardian about the death of democracy in the US [22].

15 August 2026

Russell Coker: Hacked by Chinafans

What Happened On 2026/08/10 at 2:11 am Australian eastern standard time (2026/08/09 16:11 UTC) someone created a post titled Hacked by Chinafans on my documents blog [1]. The person in question created an account named 67965e42a3c3 on that site with the email address 67965e42a3c3@google.com associated with it (I tried emailing that address and it bounced). At 04:28:41am Australian eastern standard time (18:28 UTC) I was sent an email titled Have you been hacked by a reader of my blogs who subscribed to the RSS feed of my documents blog (a blog that I never expected anyone to read by RSS). Along the lines of the wisdom of crowds should we have the unexpected observation and problem reporting of crowds ? I appreciate the notification, I might not have noticed until the next time I watched an unusually good movie otherwise. The account in question was apparently created on 2026-07-21 at 16:43:47 (presumably UTC) even though at the time I believe creating accounts was not permitted. As an aside the timestamp of account creation is stored in the user_registered column of the wp_users table in the database, there doesn t appear to be a way to access this in a standard WordPress installation other than doing a SQL query.
2026-07-24 15:43:17 status triggers-pending wordpress:all 7.0+dfsg1-1
2026-07-24 15:43:19 upgrade wordpress:all 7.0+dfsg1-1 7.0.2+dfsg1-1
Above are the relevant sections of my dpkg log showing the WordPress versions in use. I was running version 7.0+dfsg1-1 at the time the account was apparently created. I am confident in the accuracy of the dpkg logs and believe that they did not compromise the OS, I am not sure whether they ran hostile SQL code to change fields in the MySQL database so had to consider the possibility that the account creation time could have been set to a deliberately misleading value. I checked backups of the MySQL database stored off-site and found that the account in question was not in the 2026-07-21 backup (which was done before 16:43) but in the 2026-07-22 backup. The WordPress release history [2] has version 7.0.1 released on 2026-07-09 and version 7.0.2 released on 2026-07-17. So presumably the attacker diffed the code on those releases, found an exploitable bug, and used it to create an account on my blog with admin privs. Then they waited a few weeks to see if I would notice and published a blog post when I didn t notice. WordPress Deficiencies
  1. WordPress doesn t seem to store the version it s running at the time of operations. So anyone who doesn t have a suitable external log of versions deployed (such as the dpkg.log file for a Debian managed installation) won t know for sure which version was running. It supports automatic updates but you can t be sure that they happened soon after the release.
  2. There is no log of IP addresses used for operations. There are apparently some 3rd party modules to log such things and web pages documenting how to modify the PHP to add it but nothing in the standard distribution.
  3. Software should have a standard distribution with some support for logging of security relevant data. The typical situation is that people don t plan for logging such things until after they have been attacked so the data should be recorded without users going out of their way to log it.
  4. A log of security relevant data should be stored in a database table with only insert access (no update, delete, or drop).
  5. Ideally a CMS would support different database accounts for different purposes. Someone from an internal network or VPN could talk to an instance of the web server which has a database username and password giving full access. Everyone from outside the trusted range gets an instance of the web server with database access only allowing to read the posts and appearance configuration and to enter comments. If the database didn t allow the account used for public access to create new admin users or create posts then it would be a lot harder for attackers.
  6. Ideally for everything that stores user account data there would be an easy way of getting a list of users in a plain text format to allow running diff. The design of WordPress has two tables, one for users and one for encoded metadata about users of which one will be the access level. The following SQL command will give a list of all users that aren t subscribers (everyone above the minimum level of access which is typical for new users) along with their encoded password and access level. This could be used in a monitoring system to alert about new privileged users. The TABLE_PREFIX variable is for the prefix for WordPress tables, which is wp_ by default but can be any legal value.
    select $TABLE_PREFIXusers.user_login, $TABLE_PREFIXusers.user_pass, $TABLE_PREFIXusermeta.meta_value from  $TABLE_PREFIXusers join $TABLE_PREFIXusermeta on $TABLE_PREFIXusers.id = $TABLE_PREFIXusermeta.user_id and meta_key='$TABLE_PREFIXcapabilities' and meta_value != 'a:1: s:10:"subscriber";b:1; ';
What Next? The blog post they created had a couple of links to Telegram which could presumably be used to contact them. If anyone involved in computer security wants a copy of the original post to do so then they can contact me by any of the usual methods. I am interested in communication with the attacker if they wish, Telegram is not a service I use but I presume that anyone capable of doing this sort of attack is also capable of finding other ways of contacting me. I have idly considered changing to a static site generator, here is a good list of static site generators [3]. I have also idly considered other platforms for blogging such as Lemmy. I don t know if Lemmy is better than WordPress for security and updates, but there are plenty of free instances running where it wouldn t be an issue I have to work on. 15 Years It s been 15 years since my blog server was cracked by a trojaned ssh client [4]. At least this time it was only one service that was compromised.

11 August 2026

Colin Watson: Free software activity in July 2026

About 95% of my Debian contributions this month were sponsored by Freexian. You can also support my work directly via Liberapay or GitHub Sponsors. OpenSSH Now that Ubuntu 26.04 LTS has been released, I ve been getting back to the GSS-API key exchange package split in our OpenSSH packaging. Once I started testing my draft openssh-gssapi source package, I realized that I needed to make some changes in the main openssh source package first in order to support it. The dependency from openssh-server to openssh-client was awkward, as was the (related) fact that openssh-client contained shared documentation for other OpenSSH binary packages. After some thought, I created a new openssh-common binary package, moved shared documentation and the ssh-keygen program to that, and dropped dependencies on openssh-client which were no longer necessary (fixing #699473 and #1070098 in the process). This caused a couple of regressions (#1141420 and #1141550) that I had to fix, and more subtly it also caused a number of autopkgtest regressions in other packages because openssh-client is no longer in base images as a result of a dependency from openssh-server. I believe I have fixes for all of these either pending review or merged (one of which I did in August rather than July): I upgraded from 10.3p1 to 10.4p1, and in the process contributed a GSS-API option handling fix upstream. I made openssh-ssh1 s package description more accurately describe the package, thanks to suggestions from Matthias Lang. Installer team With support from a Freexian customer, I reviewed, tested, edited, and merged a patch to add VLAN support. I described the details of what I did in a comment. This has been vaguely on my to-do list since, er, about 2014, so it was very satisfying to get it sorted out. Python packaging New upstream versions: Other build/test failures: I fixed some other bugs: I adopted transaction for the Python team. I attended the Python BoF at DebConf remotely, although a badly-timed fibre outage in the village I live in really didn t help. Code reviews Other bits and pieces Dan Poltawski pointed out in a Fediverse post that the project history didn t list Sruthi as the current DPL. I fixed that, although it doesn t look as though the fix is in the published version yet. I upgraded yubihsm-shell to 2.8.0.

5 July 2026

Dirk Eddelbuettel: Rcpp 1.1.2 on CRAN: Usual Improvements in Semi-Annual Update

rcpp logo Team Rcpp is excited to share that an brandnew new version 1.1.2 of Rcpp is now on CRAN, has also been uploaded to Debian, and has already built for r2u and r-universe; Windows etc builds at CRAN should follow in due course. Rcpp has long established itself as the most popular way of enhancing R with C or C++ code. Right now, 3236 packages on CRAN depend on Rcpp for making analytical code go faster and further. On CRAN, 13.4% of all packages depend (directly) on Rcpp, and 61.4% of all compiled packages do. From the cloud mirror of CRAN (which is but a subset of all CRAN downloads), Rcpp has been downloaded 121.6 million times. The two published papers (also included in the package as preprint vignettes) have, respectively, 2263 (JSS, 2011) and 471 (TAS, 2018) citations, while the the book (Springer useR!, 2013) has another 742. The is the second update in the 1.1.* series which had, among other changes, switched to C++11 as the minimum standard. This release continues as usual with the six-months January-July cycle started with release 1.0.5 in July 2020. Interim snapshots are always available via the r-universe page and repo. We continue to strongly encourage the use of these development released and their testing we tend to run our systems with them too. Having said that, we would like to reiterate that we strongly object to the upstream R release and change management which in this 4.6.* cycle made several abrupt changes forcing packages which consume header files to make very abrupt change. Rcpp, just like numerous other CRAN packages demonstrates that API changes can be undertaken responsibly in a managed manner which allows for transition periods followed by possible warning periods, deprecations periods and finally (but only at long last) errors. What happened here is a speed run to the final stage of forced errors. Uncool and irritating for something as widely used as R. This forced us to make an interim release 1.1.1-1.1 even though we have of course had a policy of always keeping properly tested, installable, and error-free releases candidate version in the main repository branch and hence available via R-universe tested packages for all relevant platforms, and even via binaries for most (including Ubuntu LTS). It would be nice if R Core found a way to take advantage of this. Maybe development cycles, running apart for a year as they do for R, should also include selected packages. Once again I am not attempting to summarize the different changes. The full list follows below and details all these changes, their respective PRs and, if applicable, issue tickets. Big thanks from all of us to all contributors!

Changes in Rcpp release version 1.1.2 (2026-07-01)
  • Changes in Rcpp API:
    • Use of execinfo.h is again conditional to avoid build complexity (Dirk in #1445 addressing #1442)
    • An internal state component for Datetime is now int (Dirk in #1448 and #1449 fixing #1447)
    • Three new (in R 4.6.0) attribute accessors are used conditionally (Dirk in #1450 closing #1432)
    • An UBSAN error in the Sugar-based NA comparison has been corrected (I aki in #1453 fixing #1452)
    • Treatment of Inf outside of integer range in Sugar function has been corrected (I aki in #1458 fixing #1455)
    • Integer overflow protection has been added for sugar functions (I aki in #1457 fixing #1454)
    • The parent environment is now accessed via R_ParentEnv (Dirk in #1460 fixing #1459)
    • Change to returning dataptr again for better handling of empty vectors (I aki in #1462 fixing #1461)
    • Undefined behavior errors in use of ListOf proxies have been addressed (I aki in #1464 fixing #1463)
    • Under newer R version, R_UnboundValue is no longer used (I aki in #1466 fixing #1465)
    • New R API access point R_getRegisteredNamespace() is used with current R versions (Dirk in #1469 fixing #1468)
    • The Nullable::as() exporter now uses an explicit cast to the templated type (Dirk in #1471 fixing #1470)
    • A memory leak in the variadic Rcpp::warning() template has been fixed (Kevin in #1475 fixing #1474)
    • The Nullable::operatorT() has been added as a 'opt-out' (Dirk in #1477 with coordination in #1472)
    • Add templated integer-index overload for operator[] on small systems such as WASM (Jeroen Ooms in #1482)
    • The attribute accessors in AttributeProxyPolicy no longer rely on get__() (Kevin in #1484 fixing #1483)
  • Changes in Rcpp Documentation:
    • Reference in the bibliography used by the package vignettes have been updated.
  • Changes in Rcpp Deployment:
    • Excute permissions are set consistently on scripts with shebangs (Mattias Ellert in #1467)
    • R 4.5.* has been added to the CI matrix (Dirk in #1476)
    • Three nag messages issued when obsolete build flag accessors are used now show Rcpp::: (Dirk in #1480 fixing #1456)
    • Reference GitHub Actions have been updated to their current versions (Dirk in #1481)
  • Non-release Changes:
    • A non-release hotfix 1.1.1-1 used by CRAN accommodates breaking changes to the API in R 4.6.0. It would be nice to have the same level of release management in R itself that CRAN expects from us.

Thanks to my CRANberries, you can also look at a diff to the previous interim release along with pre-releases 1.1.1-1 and 1.1.1-1.1 that were needed because R-devel once again sudden decided to move fast and break things. Not our doing. And there also should not have been a need to two such uploads but it was amateur hour all around. Questions, comments etc should go to the GitHub discussion or issue section, or the Rcpp list. Bugs reports are welcome at the GitHub issue tracker as well. GitHub offers decent search for issue, pull requests and discussions; as many topics have been covered it is worth checking as well.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

Birger Schacht: Status update, June 2026

Debian Related Work
  • Uploaded wofi 1.5.3-1 to unstable
  • Uploaded wob 0.16-1 to unstable
  • Uploaded labwc 0.20.0-1 and 0.20.1-1 to unstable; these releases come with support for wlroots-0.20, which made labwc reenter testing
  • Uploaded swaylock 1.8.5-2 to unstable to make it use the common-auth directive of pam (seeh #1140096)
  • Uploaded swayimg 5.4-1 to unstable
  • Uploaded wayback 0.3-2 to unstable, which was waiting in experimental for a reupload and I had forgotten about it; also fixed a typo in wayback upstream
  • Uploaded xdg-desktop-portal-wlr 0.8.3-1 to unstable

DH Related Work The search app I was working on last month was still a focus in June. I refactored the data model a bit and made it simpler. I stumbled over the Python Koans and Koan 15: The Invisible Ink gave me the idea of using unicode normalization when indexing the items. I released a couple of bug fix releases for the APIS framework, namely 0.64.2, 0.64.3 and 0.64.4. I also release 0.65.0 which is one step further in dropping support for the legacy apis_entities app. When the search module is merged it will give way for removing the last bits of the old cruft to be removed. During a regular dependency update session I looked at the changes in the dal dependency. After a long time with no commits, the project suddenly had a lot of commits co-authored by Claude and then released a new major version with a regression. Given the state of the project, we decided to keep using the previous release for now and look into replacing the dependency with an HTMX based solution. I implemented a POC for one of the plugins we develop and it was actually pretty easy. I also managed to combine the autocomplete approach with a multi-select form field, based on this blog post. In the PFP project I finally merged the stats endpoint which give statistics about the named graphs that are used as data sources.

Other I attended BSidesVienna 0x7EA but it was on one of the hottest days this year so far so I left after a couple of talks.

3 July 2026

Colin Watson: Free software activity in June 2026

My Debian contributions this month were all sponsored by Freexian. You can also support my work directly via Liberapay or GitHub Sponsors. Thanks to new sponsor @fernandocc17! bugs.debian.org documentation Sometimes I ask users to file bugs upstream themselves because I think they d be better placed to have the ensuing discussion with the upstream maintainers directly rather than everything having to go through me. Of course sometimes they don t want to do so, perhaps because it requires creating another account somewhere. Rarely, I ve had people refuse to do this because the letter of the bug tracking system s documentation seemed to tell them not to. Since I don t believe that was the intention, I corrected this. OpenSSH I spent two and a half hours extensively revising debian/copyright so that lrc believes it to be in sync with the output of licensecheck. I m unconvinced that this was remotely worth the mind-numbing effort - as far as I can tell, it makes no difference to the practical legal position, to policy compliance, or to any reasonable user - but the DFSG team increasingly seems to be objecting to any discrepancies here any time a package crosses their radar, so this was a pre-emptive measure to avoid problems with some upcoming trips through the NEW queue. OpenSSL 4.0 I fielded a few of the OpenSSL 4.0 build failure bugs: Python packaging New upstream versions: pytest 9.1 was uploaded to unstable this month, resulting in quite a few new build/test failure bugs. I tried to keep on top of as many of these as I could; most of them had one of a small number of similar causes. Python 3.14 became the default Python version in unstable towards the end of the month, starting a transition. These usually involve quite a bit of work, and there s much more to do, but I fixed a few things: Other build/test failures: Other bugs: Rust packaging New upstream versions: Code reviews Other bits and pieces

15 June 2026

Freexian Collaborators: Debian Contributions: Go default compatibility, Trimming build-essential, Python upstream engagement and more! (by Anupa Ann Joseph)

Debian Contributions: 2026-05 Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

Go default compatibility, by Helmut Grohne At the MiniDebConf Hamburg, Andrew Lee had prepared a talk on how Debian accidentally chooses Go compatibility. Helmut joined Tobias Quathammer and Andrew Lee in looking into the problem. Go has a compatibility system where modules declare a desired Go version to be compatible with. This influences various features such as whether RSA keys smaller than 1024 bits are accepted. Unfortunately, Debian s way of building Go packages is unique in setting GO111MODULE=off, which practically implies a very old compatibility version that enables a number of insecure settings. Most Linux distributions use the default GO111MODULE=on and therefore consult a go.mod file that often declares a sensible version. While doing so is the way for Debian longer term, getting there involves major changes so we also sought a more short term workaround. We developed a patch to the Go compiler that would enable it to pick up a compatibility version from the environment. Tobias uploaded it to unstable. The next step is communicating the declared compatibility version from go.mod to the compiler via the new variable. Then, rebuilding the archive resolves the immediate symptoms. This does not save us from having to perform the larger transition to GO111MODULE=on, but this shortcut can be backported to trixie.

Trimming build-essential, by Helmut Grohne One of the harder problems of the architecture cross bootstrap is correctly expressing the Build-Depends of glib during the toolchain bootstrap. It implicitly depends on build-essential, which happens to depend on libc6-dev. This poses a cycle. It applies even for cross building, because it is interpreted for the host architecture and that there is no way of satisfying this dependency during the toolchain bootstrap. Given discussions at MiniDebConf Hamburg with Jochen Sprickerhof and others, a seemingly stupid idea evolved: Let s delete build-essential. What looks insane on the surface might deserve a second look. Given how we moved away from C, C++ and autotools, what is in build-essential no longer is required by much of the archive. With the rise of debputy, debian/rules no longer has to be a makefile. While the task would be huge, those packages relevant to architecture bootstrap could explicitly support building without the implied dependency making their dependencies explicit. In a number of cases, this amounts to issuing a dependency on g++-for-host. This dependency requires the use of architecture-prefixed tools. Therefore, Helmut wrote a debhelper change that makes it always pass build tools to various build systems. This also enables more packages to honour environment variables such as CC and CXX.

Python upstream engagement, by Stefano Rivera Stefano attended PyCon US (at personal expense) to improve upstream relations and ensure Debian s voice is heard where it needs to be. On Friday there was a packaging summit (notes) with good discussion on the future of the wheel format, and some discussion of the new abi3t shared library format for free-threaded python. In preparation for the event, Stefano did a complete review of the current patch stack. Stefano s primary goal was to get some of Debian s patches merged during the sprints, and results were mixed. Some trivial patches (e.g. GH-150098, made progress and merged, but the most consequential patch Debian is carrying is still blocked. Stefano will continue to try to drive progress on this.

Miscellaneous contributions
  • Carles worked on po-debconf-manager: Reviewed Catalan translations for 6 packages, submitted 10 packages to maintainers, and removed 3 packages from po-debconf-manager.
  • Carles worked on check-relations: Continued improving the backend, including importing source package build dependencies to better support analysis of Debian blends. Added support for ignoring packages using regular expressions and source package names in response to user feedback. Used the tool to report 5 new bugs and followed up on previously reported issues.
  • Helmut sent a cross build patch on behalf of a customer.
  • Helmut uploaded debvm and guess_concurrency both featuring improved reproducibility and documentation.
  • Helmut continued maintaining rebootstrap and made it correctly handle binNMUs of gcc-defaults. Additionally, he poked at existing gcc patches giving answers, rebasing or closing them.
  • Helmut supported the video team in Hamburg mixing audio.
  • Helmut continued to report undeclared file conflicts of various kinds and corresponded with maintainers about them.
  • Antonio attended a debate during the Brazil Internet Forum about the impacts of the child protection regulation (ECA Digital) on free software operating systems.
  • Antonio worked on Debian CI to improve the system transparency for users. This included listing any pending jobs explicitly in the job lists for each package/architecture/suite page, as well as adding a queue status page that users can check for an estimate of test latency.
  • Antonio worked on several Debian CI maintenance tasks, including but not limited to some monitoring improvements, replacing usage of fonts-font-awesome with fonts-fork-awesome, and adding the ability in debci to configure a global notice (which is being used in Debian CI to point to the system status pages).
  • Antonio started doing some tests related to the change of default Debian CI backend from lxc to incus-lxc. This helped identify an omission in the creation of incus-lxc images. It was missing dpkg-dev, which caused a few packages that assumed its presence to fail. In the end, the incus-lxc backend will be fixed to include dpkg-dev by default in the image, but that uncovered an undeclared dependency in gem2deb (Ruby packaging helper) and in ruby-byebug, both already fixed in unstable.
  • Stefano did some minimal work on debian-reimbursements to get it working with current versions of django-allauth.
  • May included the discovery of several high-severity Linux kernel root exploits. Stefano updated kernels and rebooted debian.social infrastructure several times.
  • Stefano supported the Hamburg miniDebConf s wafer website during the event, and set up an instance for the 2027 edition too.
  • Stefano supported the bursary team issuing bursaries for DebConf 26.
  • Stefano uploaded routine updates of python-pip, pystemmer, snowball-data, snowball (making up a mini, uncoordinated snowball transition), python-authlib, python-discovery, python-installer, python-mitogen, python-pipx, python-cachecontrol, platformdirs, and python-virtualenv.
  • Stefano fixed a small number of bugs in dh-python, culminating in the 7.20260524 upload.
  • Thorsten finally managed to upload a new upstream version of hplip. He also uploaded a new upstream version of epson-inkjet-printer-escpr. Last but not least with the help of other contributors he could fix bugs in lprng.
  • Lucas and Santiago contributed significantly to the DebConf 26 Content team; helping to organize the team, review and rate talk proposals.
  • Lucas also supported a packaging sprint held in India by rebuilding and publishing the latest results of the Ruby 3.4 transition effort.
  • Santiago continued contributing to the efforts to organize DebConf 26, especially supporting the local team with different tasks.
  • In collaboration with Emmanuel Arias, Santiago is mentoring Aryan Karamtoth, a GSoC participant that is working to introduce linux live-patching support in Debian. The GSoC project started in May, with community bonding and coding. Santiago reviewed a merge request to prepare the clang-extract package for debian. clang-extract is one of the building blocks that will help to extract specific functions from large C code, so only relevant code can be patched, without recompiling the whole original basecode.
  • Anupa assisted Jean-Pierre Giraud with the point release announcements for Debian 13.5 and Debian 12.14.
  • Colin backported various security fixes from OpenSSH 10.3 to all supported releases (including LTS and ELTS).
  • Colin backported IP quality-of-service fixes to OpenSSH in trixie. The situation there had been unsatisfactory for some time, and upstream reworked their QoS support in OpenSSH 10.1 in a way that typically produces much better results.
  • Colin imported new upstream versions of 26 Python packages, and fixed around 25 RC bugs for the Python team.

10 June 2026

Colin Watson: Free software activity in May 2026

My Debian contributions this month were all sponsored by Freexian. You can also support my work directly via Liberapay or GitHub Sponsors. OpenSSH I backported various security fixes from 10.3 to trixie, bookworm, bullseye, buster, and stretch. For trixie, I also backported several IPQoS fixes to line up with upstream s traffic management settings and drop a rather hacky Debian-specific patch; this needed a quick follow-up fix. I upgraded trixie-backports to 10.3. I fixed openssh uses pidof but does not depend on procps. PuTTY I upgraded from 0.83 to 0.84. Python packaging New upstream versions: Other build/test failures: Other bugs: I updated python-treq upstream to stop vendoring multipart, now that the packaging issues with that have been sorted out. Code reviews Other bits and pieces I contributed a debian-policy patch to fix several links related to build profiles.

5 June 2026

Birger Schacht: Status update, May 2026

Debian Related Work
  • Uploaded labwc 0.9.7-1 to unstable; labwc 0.20 was released upstream since then, but it requires wlroots 0.20.1 which has not landed in Debian yet
  • Uploaded usbguard 1.1.4+ds-3 & 1.1.4+ds-4: cleaned up the packaging and fixed some long standing issues with the configuration; the legacy permission system isn t the default anymore
  • Uploaded foot 1.27.0-1 to unstable
  • Uploaded scdoc 1.11.4-2 to unstable
  • Uploaded cage 0.3.0-2 to unstable
  • Uploaded sway 1.12~rc3-2 to unstable; on the same day sway 1.12 was released and I uploaded 1.12-1 to unstable
  • Uploaded swayimg 5.2-1 to unstable
  • Uploaded git-quick-stats 2.11.0-1 to unstable
  • Uploaded grim 1.5.0+ds-1 to unstable

DH Related Work A big chunk of my DH related work went into designing & implementing a search app for the APIS framework. Our goal is to have a way of searching over various types of Django models. The app introduces a search model that indexes all registered models. We use a combination of PostgreSQLs full text search and Trigram Similarity to find the search results. Using a SearchVectorField and GinIndices for the trigram indexed fields we can reach a somewhat acceptable performance. We released versions 0.63 and 0.64 of the APIS framework. The 0.63 release introduced the new entities app, which will soon hopefully replace the legacy apis_entities & apis_metainfo modules. Version 0.64 moved some logic from the legacy modules the entities module. We made some progress in defining the endpoints for the PFP API.

Next.