Search Results: "xam"

4 October 2026

Vincent Bernat: Hacking the Go compiler to efficiently map IPv4 to IPv6

netip.Addr features an Unmap() method returning the unwrapped IPv4 contained in an IPv4-mapped IPv6 address: from ::ffff:203.0.113.10 or ::ffff:cb00:710a, it returns 203.0.113.10.1 There is no Map() or To6() method for the reverse direction. Such a method is trivial to implement, but Go maintainers have rejected it on the grounds that users should write netip.AddrFrom16(ip.As16()) and let the compiler optimize it.2 Today, this pattern is eight times slower than a native method. How can we teach the compiler to optimize this sequence?

The alternatives Let's explore three ways to implement the map semantics for netip.Addr. My favorite is to add it to the Go standard library. Go maintainers prefer a small external helper chaining netip.AddrFrom16() and netip.Addr.As16(), hoping the compiler eventually optimizes it. The unsafe package opens a third path, with the same performance as the first solution.

Modifying the Go standard library Internally, netip.Addr stores any IP address as a 128-bit value with an extra field z to encode the family and the zone:
type Addr struct  
    addr uint128
    z unique.Handle[addrDetail]
 
type addrDetail struct  
    isV6   bool   // IPv4 is false, IPv6 is true.
    zoneV6 string // != "" only if IsV6 is true.
 
var (
    z0    unique.Handle[addrDetail]
    z4    = unique.Make(addrDetail )
    z6noz = unique.Make(addrDetail isV6: true )
)
AddrFrom4() encodes an IPv4 address as an IPv4-mapped IPv6 address and sets z to the unique value z4:
// AddrFrom4 returns the address of the IPv4 address given by the bytes in addr.
func AddrFrom4(addr [4]byte) Addr  
    return Addr 
        addr: uint128 
            0,
            0xffff00000000  
                uint64(addr[0])<<24   uint64(addr[1])<<16  
                uint64(addr[2])<<8   uint64(addr[3]) ,
        z: z4,
     
 
Unmap() turns an IPv4-mapped IPv6 address into an IPv4 address by setting the z field to z4:
func (ip Addr) Unmap() Addr  
    if ip.Is4In6()  
        ip.z = z4
     
    return ip
 
Implementing the reverse direction inside the Go standard library is trivial: we set the z field to z6noz if the address is IPv4.
// To6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an
// IPv6 address unmodified.
func (ip Addr) To6() Addr  
    if ip.Is4()  
        ip.z = z6noz
     
    return ip
 

Update (2026-10) Instead of patching Go, we can use the -overlay flag of go build to hijack any package, including the standard library. In our case, it can add To6() to net/netip. Yet it is cumbersome:3 the flag expects a JSON file mapping absolute paths in GOROOT to replacement files, and you need to add it to every go command.

As a helper We can't access the z field from outside the net/netip package. Instead, we build a small helper around the netip.AddrFrom16(ip.As16()) pattern:
// AddrTo6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an
// IPv6 address unmodified.
func AddrTo6(ip netip.Addr) netip.Addr  
    if ip.Is4()  
        ip = netip.AddrFrom16(ip.As16())
     
    return ip
 

As an unsafe function Another solution uses the unsafe package to alter the Addr struct through a proxy with the same memory layout:4
// addrProxy has the same memory layout as netip.Addr.
type addrProxy struct  
    addr [2]uint64      // netip.uint128
    z    unsafe.Pointer // unique.Handle[netip.addrDetail]
 
var (
    anyIPv6    = netip.IPv6Unspecified()
    netipZ6noz = (*addrProxy)(unsafe.Pointer(&anyIPv6)).z
)
// AddrTo6 maps an IPv4 address to an IPv4-mapped IPv6 address. It returns an
// IPv6 address unmodified.
func AddrTo6(ip netip.Addr) netip.Addr  
    if !ip.Is4()  
        return ip
     
    (*addrProxy)(unsafe.Pointer(&ip)).z = netipZ6noz
    return ip
 

Benchmarks On my computer, with Go 1.27.1, the standard library solution costs 0.88 ns per operation, while the solution favored by Go maintainers costs 7.14 ns. The unsafe solution matches the performance of the first one.
goos: linux
goarch: amd64
pkg: github.com/vincentbernat/go-netip-addrto6
cpu: AMD Ryzen 5 5600X 6-Core Processor
                         sec/op      
AddrTo6/safe            7.137n   0%
AddrTo6/unsafe         0.8682n   2%
AddrTo6/builtin        0.8775n   2%

Assembly code Let's check the assembly code the compiler generates for each solution.5 The one built into the standard library looks like this:6
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z
 CMPQ  net/netip z4(SB), CX     ; check "z" if this is an IPv4 address
 JNE   end                      ; if not, stop here
 MOVQ  net/netip z6noz(SB), CX  ; CX = netip.z6noz
end:
 RET
// return value = Addr hi: AX, lo: BX, z: CX 
Go's assembly language is not a direct representation of the underlying machine language: it operates on a semi-abstract instruction set derived from Plan 9's assembler. It has four pseudo-registers: FP (frame pointer for function arguments), PC (program counter), SB (static base pointer for global symbols), and SP (stack pointer). It also has architecture-specific registers like AX, CX, DX, BX, SI, DI, and R8 to R15. Instructions storing data use their last argument as the destination. Instructions can carry an explicit size suffix: MOVB moves a byte, MOVW 16 bits, MOVL 32 bits, and MOVQ 64 bits. In the example above, the first instruction compares the 64-bit value z4 with the CX register. The unsafe solution looks almost the same:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z
 CMPQ  net/netip z4(SB), CX  ; check "z" if this is an IPv4 address
 JNE   end                   ; if not, stop here
 MOVQ  netipZ6noz(SB), CX    ; CX = netip.z6noz
end:
 RET
// return value = Addr hi: AX, lo: BX, z: CX 
The helper solution has far more instructions. To understand why, let's look at the code for As16() and AddrFrom16(). They are short enough for the compiler to inline them.
func (ip Addr) As16() (a16 [16]byte)  
    byteorder.BEPutUint64(a16[:8], ip.addr.hi)
    byteorder.BEPutUint64(a16[8:], ip.addr.lo)
    return a16
 
func AddrFrom16(addr [16]byte) Addr  
    return Addr 
        addr: uint128 
            byteorder.BEUint64(addr[:8]),
            byteorder.BEUint64(addr[8:]),
         ,
        z: z6noz,
     
 
We can already guess the pattern to optimize: the code packs the IP address into an array, copies it, then unpacks it. If we inline the Go code by hand, we get:
func AddrTo6(input netip.Addr) netip.Addr  
    if !input.Is4()  
        return input
     
    var a16 [16]byte
    byteorder.BEPutUint64(a16[:8], input.addr.hi)
    byteorder.BEPutUint64(a16[8:], input.addr.lo)
    addr := a16
    var output netip.Addr
    output.addr.hi = byteorder.BEUint64(addr[:8])
    output.addr.lo = byteorder.BEUint64(addr[8:])
    output.z = netip.z6noz
    return output
 
As humans, we can mentally derive the optimized form:
func AddrTo6(input netip.Addr) netip.Addr  
    if !input.Is4()  
        return input
     
    var output netip.Addr
    output.addr.hi = input.addr.hi
    output.addr.lo = input.addr.lo
    output.z = netip.z6noz
    return output
 

Unfortunately, as of Go 1.26.8, the compiler is not smart enough to do the same:

// AX = input.addr.hi, BX = input.addr.lo, CX = input.z
; Push the stack (32 bytes):
;    0(SP) addr netip.uint128
;   16(SP) a16 [16]byte
 PUSHQ   BP
 MOVQ    SP, BP
 SUBQ    $32, SP
 CMPQ    net/netip z4(SB), CX  ; check "z" if this is an IPv4 address
 JNE     end                   ; if not, stop here
; Pack: byteorder.BEPutUint64(a16[:8], input.addr.hi)
;       byteorder.BEPutUint64(a16[8:], input.addr.lo)
 MOVBEQ  AX, net/netip a16+16(SP)
 MOVBEQ  BX, net/netip a16+24(SP)
; addr = a16, 16 bytes at once through the vector register X0
 MOVUPS  net/netip a16+16(SP), X0
 MOVUPS  X0, net/netip addr(SP)
; CX = netip.z6noz
 MOVQ    net/netip z6noz(SB), CX
; Unpack: output.addr.hi = byteorder.BEUint64(addr[:8])
;         output.addr.lo = byteorder.BEUint64(addr[8:])
 MOVBEQ  net/netip addr(SP), AX
 MOVBEQ  net/netip addr+8(SP), BX
end:
 ADDQ    $32, SP
 POPQ    BP
 RET
// return value = Addr hi: AX, lo: BX, z: CX 
The compiler does a decent job on the byte shuffling: the eight byte stores of BEPutUint64() become a single MOVBEQ, which stores a register byte-swapped. The eight byte loads of BEUint64() become a single MOVBEQ the other way round.7 Three groups of instructions remain: a pack, a copy, and an unpack.

Hacking the Go compiler The Go compiler has several phases:
Parsing
The compiler tokenizes and parses the source code. It builds a syntax tree for each source file.
Type checking
The compiler maps each identifier to the object it denotes, folds constants, and infers the type of every expression.
IR construction
The compiler converts the syntax tree and its types into its own intermediate representation (IR). This process, called "noding," goes through a serialization format named unified IR.
Middle end
The compiler performs several optimization passes on the IR, such as devirtualization, function call inlining, and escape analysis.
Walk
This phase runs two steps: order of evaluation decomposes complex statements into simpler ones, and desugaring transforms higher-level Go constructs, like switch or channels, into more primitive instructions or calls to the runtime.
Generic SSA
The compiler converts the IR into Static Single Assignment (SSA) form, a lower-level intermediate representation suited for machine-independent optimizations and rewrite rules.
Machine code generation
The compiler rewrites the SSA form into machine-specific variants, allocates registers, and applies more optimization passes. At the end, the assembler turns the generated instructions into machine code.

The hammer My first idea is to replace occurrences of netip.AddrFrom16(ip.As16()) with netip.Addr addr: ip.addr, z: netip.z6noz as early as possible, during the "noding" process. Before that, the type checking phase prevents us from accessing unexported struct fields. Go 1.27 introduced a convenient debug option to dump the IR of a function at interesting points during compilation:
$ GOTOOLCHAIN=go1.27.1 GOAMD64=v3 go build -a -gcflags="-d=astdump=AddrTo6Safe" .
Writing text ast output for AddrTo6Safe to AddrTo6Safe.ast
Writing html ast output for AddrTo6Safe to AddrTo6Safe.html
Writing html syntax output for AddrTo6Safe to AddrTo6Safe.syntax.html
In the HTML file, the first column shows the IR as it comes out of noding:
DCLFUNC addrto6.AddrTo6Safe ABI:ABIInternal FUNC-func(netip.Addr) netip.Addr
DCLFUNC-Dcl
. NAME-addrto6.ip Class:PPARAM Offset:0 OnStack Used netip.Addr
. NAME-addrto6.~r0 Class:PPARAMOUT Offset:0 OnStack netip.Addr
DCLFUNC-body
. IF # ipv6_safe.go:11:2
. IF-Cond
. . CALLFUNC bool
. . CALLFUNC-Fun
. . . METHEXPR addrto6.Is4 FUNC-func(netip.Addr) bool
. . . . TYPE netip.Addr Class:PEXTERN Offset:0 type netip.Addr
. . CALLFUNC-Args
. . . NAME-addrto6.ip Class:PPARAM Offset:0 OnStack Used netip.Addr
. IF-Body
. . AS # ipv6_safe.go:12:6
. . . NAME-addrto6.ip Class:PPARAM Offset:0 OnStack Used netip.Addr
. . . CALLFUNC netip.Addr
. . . CALLFUNC-Fun
. . . . NAME-netip.AddrFrom16 Class:PFUNC Offset:0 Used FUNC-func([16]byte) netip.Addr
. . . CALLFUNC-Args
. . . . CALLFUNC ARRAY-[16]byte
. . . . CALLFUNC-Fun
. . . . . METHEXPR addrto6.As16 FUNC-func(netip.Addr) [16]byte
. . . . . . TYPE netip.Addr Class:PEXTERN Offset:0 type netip.Addr
. . . . CALLFUNC-Args
. . . . . NAME-addrto6.ip Class:PPARAM Offset:0 OnStack Used netip.Addr
. RETURN # ipv6_safe.go:14:2
. RETURN-Results
. . NAME-addrto6.ip Class:PPARAM Offset:0 OnStack Used netip.Addr
In the body of the if statement, we spot the calls to the method netip.Addr.As16() and to the function netip.AddrFrom16(). Our goal is to patch them with a struct literal:
IF-Body
. AS # ipv6_safe.go:12:6
. . NAME-addrto6.ip Class:PPARAM Offset:0 OnStack Used netip.Addr
. . STRUCTLIT netip.Addr
. . STRUCTLIT-List
. . . STRUCTKEY netip.addr
. . . . DOT netip.addr netip.uint128
. . . . . NAME-addrto6.ip Class:PPARAM Offset:0 OnStack Used netip.Addr
. . . STRUCTKEY netip.z
. . . . NAME-netip.z6noz Class:PEXTERN Offset:0 unique.Handle[net/netip.addrDetail]
In noder's reader.go, the expr() method builds the IR tree for an expression. At the end of the exprCall case, we add a call to a rewriteAddrFrom16As16() function. It takes the current node and returns the struct literal on success, or nil if the rewrite is not possible. First, we check that we have the expected pattern: a call to the netip.AddrFrom16() function with a call to the netip.Addr.As16() method as its only argument:
func rewriteAddrFrom16As16(n ir.Node) ir.Node  
    call, ok := n.(*ir.CallExpr)
    if !ok   call.Op() != ir.OCALLFUNC  
        len(call.Args) != 1   len(call.Init()) != 0  
        !isNetipFunc(call.Fun, "AddrFrom16")  
        return nil
     
    inner, ok := call.Args[0].(*ir.CallExpr)
    if !ok   inner.Op() != ir.OCALLFUNC  
        len(inner.Args) != 1   len(inner.Init()) != 0  
        !isNetipFunc(inner.Fun, "Addr.As16")  
        return nil
     
    x := inner.Args[0]
    // [...]
 
Then, we fetch netip.z6noz:
z6noz, err := lookupVar(ir.StaticCalleeName(call.Fun).Sym().Pkg, "z6noz")
if err != nil  
    return nil
 
And we build the struct literal:
typ := call.Type()
pos := call.Pos()
var list []ir.Node
for i, f := range typ.Fields()  
    var value ir.Node
    switch f.Sym.Name  
    case "addr":
        value = typecheck.DotField(pos, x, i)
    case "z":
        value = z6noz
    default:
        return nil
     
    list = append(list, ir.NewStructKeyExpr(pos, f, value))
 
lit := ir.NewCompLitExpr(pos, ir.OSTRUCTLIT, typ, list)
lit.SetTypecheck(1)
return lit
Have a look at the complete patch.8 We can test it with the following commands:
$ cd src
$ ./make.bash
Building Go cmd/dist using /usr/lib/go-1.27. (go1.27.1 linux/amd64)
Building Go toolchain1 and bootstrap cmd/go (go_bootstrap) using /usr/lib/go-1.27.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 and commands using go_bootstrap and Go toolchain2.
Checking command staleness for linux/amd64.
---
Installed Go for linux/amd64 in /home/bernat/code/free/go
Installed commands in /home/bernat/code/free/go/bin
*** You need to add /home/bernat/code/free/go/bin to your PATH.
$ export PATH=$PWD/../bin:$PATH
$ go version
go version go1.28-devel_9834516e20 Sat Sep 12 08:23:11 2026 -0700 linux/amd64
$ go test net/netip/...
ok      net/netip   0.224s
$ cd ../../go-netip-addrto6
$ go test .
ok      github.com/vincentbernat/go-netip-addrto6   0.062s
The generated code for the helper is now the shortest possible version!
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z
 CMPQ    net/netip z4(SB), CX     ; check "z" if this is an IPv4 address
 JNE     end                      ; if not, stop here
 MOVQ    net/netip z6noz(SB), CX  ; CX = netip.z6noz
end:
 RET
// return value = Addr hi: AX, lo: BX, z: CX 
Go maintainers are unlikely to accept this change. It relies on the internal structure of net/netip.Addr. It's an ugly hack in the noder, whose job is to faithfully translate the type-checked AST into the IR. And it's harder to maintain than adding a To6() method.

The screwdriver The right place for such an optimization is the generic SSA phase. One of the last machine-independent passes is memcombine. With the appropriate debug flag, the compiler dumps the SSA form after this pass:9
$ GOTOOLCHAIN=go1.26.8 GOAMD64=v3 \
> go build -a -gcflags='-d=ssa/memcombine/dump=AddrTo6Safe' .
$ head -5 AddrTo6Safe_01__memcombine.dump
AddrTo6Safe func(netip.Addr) netip.Addr
  b2:
    (?) v1 = InitMem <mem>
    (?) v2 = SP <uintptr>
    (?) v3 = SB <uintptr>
The result of the memcombine pass follows the same structure as the assembly code for AddrTo6Safe() we looked at earlier: two stores, one move, and two loads we would like to optimize away.
; [ ]
  v502 = ArgIntReg <uint64>  ip+0  [0]              ; input.addr.hi
  v490 = ArgIntReg <uint64>  ip+8  [1]              ; input.addr.lo
  v466 = ArgIntReg <*netip.addrDetail>  ip+16  [2]  ; input.z
; [ ]
  v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
  v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v442 = Bswap64 <uint64> v490                      ; bswap(input.addr.lo)
  v542 = Bswap64 <uint64> v502                      ; bswap(input.addr.hi)
  v161 = Store <mem>  uint64  v22 v542 v23          ; a16[:8] = bswap(hi)
  v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
  v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
  v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
  v415 = OffPtr <*byte> [8] v285                    ; &addr[8]
  v416 = Load <uint64> v285 v286                    ; addr[:8]
  v299 = Bswap64 <uint64> v416                      ; output.addr.hi
  v174 = Load <uint64> v415 v286                    ; addr[8:]
  v39  = Bswap64 <uint64> v174                      ; output.addr.lo
; [ ]
Each line features a value identifier (v442), an operation with its type (Bswap64 <uint64>), and its arguments (v490).10 Values are the basic building blocks of SSA and are defined exactly once. Square brackets enclose integer parameters ([8]) and curly braces contain auxiliary arguments ( netip.addr ). Operations writing to memory produce a new memory state. Every memory operation takes the current state as its last argument, which keeps them in order.

On paper Let's focus on output.addr.lo, aka v39:
  v490 = ArgIntReg <uint64>  ip+8  [1]              ; input.addr.lo
  v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
  v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v442 = Bswap64 <uint64> v490                      ; bswap(input.addr.lo)
  v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
  v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
  v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
  v415 = OffPtr <*byte> [8] v285                    ; &addr[8]
  v174 = Load <uint64> v415 v286                    ; addr[8:]
  v39  = Bswap64 <uint64> v174                      ; output.addr.lo
To simplify this code, we could apply three rewriting rules:
  1. The first one adds a shortcut when loading through a move: (Load (OffPtr [o] p) (Move p src mem)) => (Load (OffPtr [o] src) mem). This matches v174 with its arguments v415 and v286 and creates a new value v600:
      v490 = ArgIntReg <uint64>  ip+8  [1]              ; input.addr.lo
      v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
      v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
      v442 = Bswap64 <uint64> v490                      ; bswap(input.addr.lo)
      v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
      v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
      v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
      v415 = OffPtr <*byte> [8] v285                    ; &addr[8]
      v600 = OffPtr <*byte> [8] v22                     ; &a16[8]
      v174 = Load <uint64> v600 v282                    ; a16[8:]
      v39  = Bswap64 <uint64> v174                      ; output.addr.lo
    
  2. The second one simplifies a load following a store: (Load p (Store p x _)) => x. The load is forwarded: the stored value replaces it and no memory access remains. It matches v174. It notices that v600 and v173 are the same address and replaces v174 with a copy of v442:
      v490 = ArgIntReg <uint64>  ip+8  [1]              ; input.addr.lo
      v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
      v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
      v442 = Bswap64 <uint64> v490                      ; bswap(input.addr.lo)
      v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
      v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
      v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
      v415 = OffPtr <*byte> [8] v285                    ; &addr[8]
      v600 = OffPtr <*byte> [8] v22                     ; &a16[8]
      v174 = Copy <uint64> v442                         ; bswap(input.addr.lo)
      v39  = Bswap64 <uint64> v174                      ; output.addr.lo
    
  3. The last step cancels the two byte swaps: (Bswap64 (Bswap64 x)) => x. v39 becomes a copy of v490:
      v490 = ArgIntReg <uint64>  ip+8  [1]              ; input.addr.lo
      v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
      v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
      v442 = Bswap64 <uint64> v490                      ; bswap(input.addr.lo)
      v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
      v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
      v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
      v415 = OffPtr <*byte> [8] v285                    ; &addr[8]
      v600 = OffPtr <*byte> [8] v22                     ; &a16[8]
      v174 = Copy <uint64> v442                         ; bswap(input.addr.lo)
      v39  = Copy <uint64> v490                         ; output.addr.lo = input.addr.lo
    
If we ignore the values not needed to compute v39, only this SSA form remains:
  v490 = ArgIntReg <uint64>  ip+8  [1]  ; input.addr.lo
  v39  = Copy <uint64> v490             ; output.addr.lo = input.addr.lo
Let's switch to output.addr.hi, aka v299:
  v502 = ArgIntReg <uint64>  ip+0  [0]              ; input.addr.hi
  v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
  v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
  v542 = Bswap64 <uint64> v502                      ; bswap(input.addr.hi)
  v161 = Store <mem>  uint64  v22 v542 v23          ; a16[:8] = bswap(hi)
  v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
  v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
  v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
  v416 = Load <uint64> v285 v286                    ; addr[:8]
  v299 = Bswap64 <uint64> v416                      ; output.addr.hi
To optimize it away, we also apply three rewriting rules:
  1. The first one also adds a shortcut when loading through a move, but without an offset: (Load p (Move p src mem)) => (Load src mem). This rewrites v416 to use arguments from v286:
      v502 = ArgIntReg <uint64>  ip+0  [0]              ; input.addr.hi
      v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
      v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
      v542 = Bswap64 <uint64> v502                      ; bswap(input.addr.hi)
      v161 = Store <mem>  uint64  v22 v542 v23          ; a16[:8] = bswap(hi)
      v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
      v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
      v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
      v416 = Load <uint64> v22 v282                     ; a16[:8]
      v299 = Bswap64 <uint64> v416                      ; output.addr.hi
    
  2. The second rule forwards a value stored one step earlier, skipping over a store to another address: (Load p (Store q _ (Store p x _))) => x. This matches v416: x is v542, p is v22 (&a16), q is v173 (&a16[8]), and p and q do not overlap for uint64.
      v502 = ArgIntReg <uint64>  ip+0  [0]              ; input.addr.hi
      v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
      v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
      v542 = Bswap64 <uint64> v502                      ; bswap(input.addr.hi)
      v161 = Store <mem>  uint64  v22 v542 v23          ; a16[:8] = bswap(hi)
      v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
      v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
      v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
      v416 = Copy <uint64> v542                         ; bswap(input.addr.hi)
      v299 = Bswap64 <uint64> v416                      ; output.addr.hi
    
  3. The third rule cancels two byte swaps: (Bswap64 (Bswap64 x)) => x. v299 becomes a copy of v502:
      v502 = ArgIntReg <uint64>  ip+0  [0]              ; input.addr.hi
      v22 = LocalAddr <*[16]byte>  netip.a16  v2 v1     ; &a16
      v173 = OffPtr <*byte> [8] v22                     ; &a16[8]
      v542 = Bswap64 <uint64> v502                      ; bswap(input.addr.hi)
      v161 = Store <mem>  uint64  v22 v542 v23          ; a16[:8] = bswap(hi)
      v282 = Store <mem>  uint64  v173 v442 v161        ; a16[8:] = bswap(lo)
      v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
      v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
      v416 = Copy <uint64> v542                         ; bswap(input.addr.hi)
      v299 = Copy <uint64> v502                         ; output.addr.hi = input.addr.hi
    
If we remove the values not used to compute v299, we get this SSA form:
  v502 = ArgIntReg <uint64>  ip+0  [0] ; input.addr.hi
  v299 = Copy <uint64> v502            ; output.addr.hi = input.addr.hi

In practice Most of these rules already exist in generic.rules. They use conditions to validate their context: ssa.IsSamePtr() for the same address, ssa.Disjoint() for addresses that do not overlap. The rule forwarding a stored value to a load already exists with three variants looking through several other stores. Here are the two we need:
(Load <t1> p1 (Store  t2  p2 x _))
    && ssa.IsSamePtr(p1, p2)
    && copyCompatibleType(t1, x.Type)
    && t1.Size() == t2.Size()
    => x
(Load <t1> p1 (Store  t2  p2 _ (Store  t3  p3 x _)))
    && ssa.IsSamePtr(p1, p3)
    && copyCompatibleType(t1, x.Type)
    && t1.Size() == t3.Size()
    && ssa.Disjoint(p3, t3, p2, t2)
    => x
Go 1.27 added the rule loading through a move with CL 748200 to fix issue #77720:
(Load <t1> op1:(OffPtr [o1] p1) move:(Move [n] p2 src mem))
    && o1 >= 0 && o1+t1.Size() <= n && ssa.IsSamePtr(p1, p2)
    && !ssa.IsVolatile(src)
    => @move.Block (Load <t1> (OffPtr <op1.Type> [o1] src) mem)
It lacks a variant without an offset:
(Load <t1> p1 move:(Move [n] p2 src mem))
    && p1.Op != ssaop.OpOffPtr
    && t1.Size() <= n && ssa.IsSamePtr(p1, p2)
    && !ssa.IsVolatile(src)
    => @move.Block (Load <t1> (OffPtr <p1.Type> [0] src) mem)
There is no generic rule to cancel two byte swaps, but the AMD64 lowering pass includes this rule:
(BSWAP(Q L) (BSWAP(Q L) p)) => p
After switching to Go's development branch and adding the missing rule, the generated assembly code is worse than with Go 1.26.8, even though our additional rule slightly improves the situation at the end:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z
; Push the stack (16 bytes):
;    0(SP) a16 [16]byte
 PUSHQ   BP
 MOVQ    SP, BP
 SUBQ    $16, SP
 CMPQ    net/netip z4(SB), CX       ; check "z" if this is an IPv4 address
 JNE     end                        ; if not, stop here
; The four forwarded bytes: the low half of input.addr.lo is taken apart and
; put back together in registers
 MOVQ    BX, DX                     ; DX = input.addr.lo
 SHRQ    $24, BX                    ; BX = input.addr.lo >> 24
 MOVQ    DX, SI                     ; SI = input.addr.lo
 SHRQ    $16, DX                    ; DX = input.addr.lo >> 16
 MOVQ    SI, DI                     ; DI = input.addr.lo, kept for the pack
 SHRQ    $8, SI                     ; SI = input.addr.lo >> 8
 MOVBLZX DIB, R8                    ; R8 = byte(input.addr.lo)
 MOVBLZX SIB, SI                    ; SI = byte(input.addr.lo >> 8)
 SHLQ    $8, SI
 ORQ     R8, SI                     ; SI = two low bytes of input.addr.lo
 MOVBLZX DL, DX                     ; DX = byte(input.addr.lo >> 16)
 SHLQ    $16, DX
 ORQ     SI, DX
 MOVBLZX BL, BX                     ; BX = byte(input.addr.lo >> 24)
 SHLQ    $24, BX
 ORQ     DX, BX                     ; BX = input.addr.lo & 0xffffffff
; Pack: byteorder.BEPutUint64(a16[:8], input.addr.hi)
;       byteorder.BEPutUint64(a16[8:], input.addr.lo)
 MOVBEQ  AX, net/netip a16(SP)
 MOVBEQ  DI, net/netip a16+8(SP)
; The four other bytes of input.addr.lo, read one by one from a16
 MOVBLZX net/netip a16+11(SP), DX   ; a16[11]
 SHLQ    $32, DX
 ORQ     DX, BX
 MOVBLZX net/netip a16+10(SP), DX   ; a16[10]
 SHLQ    $40, DX
 ORQ     DX, BX
 MOVBLZX net/netip a16+9(SP), DX    ; a16[9]
 SHLQ    $48, DX
 ORQ     DX, BX
 MOVBLZX net/netip a16+8(SP), DX    ; a16[8]
 SHLQ    $56, DX
; output.z = netip.z6noz
 MOVQ    net/netip z6noz(SB), CX
; output.addr.hi = byteorder.BEUint64(a16[:8])
 MOVBEQ  net/netip a16(SP), AX
; output.addr.lo assembled from the previous steps
 ORQ     DX, BX
end:
 LEAVEQ
 RET
// return value = Addr hi: AX, lo: BX, z: CX 
The rule loading through a move, added in Go 1.27, introduced this regression.

Out of order Let's not give up now! In reality, the rewriting rules run before memcombine, notably in the late opt pass. At this point, the inlined versions of BEPutUint64() and BEUint64() still expand to sixteen byte stores and sixteen byte loads, matching their source code:
func BEUint64(b []byte) uint64  
    _ = b[7] // bounds check hint to compiler; see golang.org/issue/14808
    return uint64(b[7])   uint64(b[6])<<8   uint64(b[5])<<16   uint64(b[4])<<24  
        uint64(b[3])<<32   uint64(b[2])<<40   uint64(b[1])<<48   uint64(b[0])<<56
 
Let's follow two bytes of output.addr.lo: addr[15] and addr[11]. Here is a simplified SSA form before late opt:
  v273 = Trunc64to8 <byte> v490                     ; byte(input.addr.lo)
  v226 = Trunc64to8 <byte> v225                     ; byte(input.addr.lo >> 32)
; [ ]
  v235 = Store <mem>  byte  v233 v226 v223          ; a16[11] = byte(lo >> 32)
  v247 = Store <mem>  byte  v245 v238 v235          ; a16[12] =  
  v259 = Store <mem>  byte  v257 v250 v247          ; a16[13] =  
  v271 = Store <mem>  byte  v269 v262 v259          ; a16[14] =  
  v282 = Store <mem>  byte  v280 v273 v271          ; a16[15] = byte(lo)
  v285 = LocalAddr <*[16]byte>  netip.addr  v2 v282 ; &addr
  v286 = Move <mem>  [16]byte  [16] v285 v22 v282   ; addr = a16
; [ ]
  v433 = OffPtr <*byte> [15] v285                   ; &addr[15]
  v435 = Load <byte> v433 v286                      ; addr[15]
  v479 = OffPtr <*byte> [11] v285                   ; &addr[11]
  v481 = Load <byte> v479 v286                      ; addr[11]
The first rule loads through the move: (Load (OffPtr [o] p) (Move p src mem)) => (Load (OffPtr [o] src) mem). It matches both loads, which now read a16 with the memory state before the copy:
  v600 = OffPtr <*byte> [15] v22 ; &a16[15]
  v435 = Load <byte> v600 v282   ; a16[15]
  v601 = OffPtr <*byte> [11] v22 ; &a16[11]
  v481 = Load <byte> v601 v282   ; a16[11]
The second rule shortcuts a load following a store: (Load p (Store p x _)) => x. It matches v435, as v282 stores a16[15]. It does not match v481: v235 stores a16[11] four stores earlier in the chain, while the variants of this rule look through three stores at most.
  v435 = Copy <byte> v273        ; byte(input.addr.lo)
  v601 = OffPtr <*byte> [11] v22 ; &a16[11]
  v481 = Load <byte> v601 v282   ; a16[11]
The same happens to the other bytes: the rule forwards the four bytes stored last, a16[12] to a16[15]. The twelve other loads now read a16 instead of addr. BEUint64() becomes a chain of Or64, each one adding a byte shifted into place. memcombine is a pass written in Go, not a set of rewrite rules. It starts from the last Or64 of the chain and collects up to eight terms. If each term is a byte load, extended to 64 bits and shifted, and if the eight loads read consecutive addresses from the same pointer with the same memory state, it replaces the whole chain with a single 64-bit load and a byte swap. Otherwise, it tries again with four, then two terms, and from each intermediate Or64. Here is the loop checking each term in a simplified version of combineLoads():
for i := int64(0); i < n; i++  
    v := a[i]
    shift := int64(0)
    if v.Op == shiftOp  
        v, shift = peelShift(v)
     
    if v.Op != extOp  
        return false
     
    load := v.Args[0]
    if load.Op != ssaop.OpLoad  
        return false
     
    if load.Args[1] != mem  
        return false
     
    p, off := splitPtr(load.Args[0])
    if p != base  
        return false
     
    r[i] = LoadRecord load: load, offset: off, shift: shift 
 
For output.addr.hi, the eight loads read a16 with the same memory state v282:
  v13  = Load <byte> v22 v282               ; a16[0]
  v530 = Load <byte> v14 v282               ; a16[1]
  v488 = Load <byte> v504 v282              ; a16[2]
  v405 = Load <byte> v537 v282              ; a16[3]
  v385 = Load <byte> v397 v282              ; a16[4]
  v361 = Load <byte> v373 v282              ; a16[5]
  v196 = Load <byte> v63 v282               ; a16[6]
  v432 = Load <byte> v315 v282              ; a16[7]
  v319 = ZeroExt8to64 <uint64> v432         ; uint64(a16[7])
  v329 = ZeroExt8to64 <uint64> v196         ; uint64(a16[6])
  v330 = Lsh64x64 <uint64> [true] v329 v138 ; uint64(a16[6]) << 8
  v331 = Or64 <uint64> v319 v330            ; a16[7]   a16[6] << 8
; [ ] same for a16[5] to a16[1]
  v401 = ZeroExt8to64 <uint64> v13          ; uint64(a16[0])
  v402 = Lsh64x64 <uint64> [true] v401 v55  ; uint64(a16[0]) << 56
  v403 = Or64 <uint64> v402 v391            ;   a16[0] << 56 = output.addr.hi
memcombine merges them into one load and a swap:
  v286 = Load <uint64> v22 v282   ; a16[:8]
  v285 = Bswap64 <uint64> v286    ; output.addr.hi
For output.addr.lo, here is the chain memcombine sees after late opt:
  v436 = ZeroExt8to64 <uint64> v273 ; addr[15], forwarded
  v448 = Or64 <uint64> v436 v447    ;   addr[14] << 8, forwarded
  v460 = Or64 <uint64> v459 v448    ;   addr[13] << 16, forwarded
  v472 = Or64 <uint64> v471 v460    ;   addr[12] << 24, forwarded
  v484 = Or64 <uint64> v483 v472    ;   a16[11] << 32, loaded
  v496 = Or64 <uint64> v495 v484    ;   a16[10] << 40, loaded
  v508 = Or64 <uint64> v507 v496    ;   a16[9] << 48, loaded
  v520 = Or64 <uint64> v519 v508    ;   a16[8] << 56, loaded
From v520, four of the eight terms are forwarded bytes, not loads from memory, and memcombine can't combine them. It doesn't merge the four remaining loads either, as they sit on top of the forwarded bytes.

Back in order In summary, the rewriting rules run too early to be effective. A quick workaround exists: run an earlier round of memcombine before late opt. After this change, the generated code for the helper is back to the shortest possible version:
// AX = input.addr.hi, BX = input.addr.lo, CX = input.z
 CMPQ    net/netip z4(SB), CX     ; check "z" if this is an IPv4 address
 JNE     end                      ; if not, stop here
 MOVQ    net/netip z6noz(SB), CX  ; CX = netip.z6noz
end:
 RET
// return value = Addr hi: AX, lo: BX, z: CX 
And the benchmark confirms it!
goos: linux
goarch: amd64
pkg: github.com/vincentbernat/go-netip-addrto6
cpu: AMD Ryzen 5 5600X 6-Core Processor
                       Go 1.26.8                  Our branch               
                         sec/op          sec/op     vs base                
AddrTo6/safe          6.5470n    0%   0.8944n   4%  -86.34% (p=0.002 n=6)
AddrTo6/unsafe        0.9071n    3%   0.8682n   1%   -4.28% (p=0.002 n=6)
AddrTo6/builtin       0.8871n    2%   0.8785n   1%        ~ (p=0.310 n=6)

Next steps I think Go maintainers would reject this change because of the additional memcombine pass. Instead, I plan to publish this blog post and bring up the subject again as a follow-up to issue #54365. Either the sheer complexity and the Go 1.27 regression convince the maintainers that adding a To6() method is simpler and more efficient, or they advise me on how to move forward. Either way, digging into this subject taught me a lot about the Go compiler!

Update (2026-10) I opened issue #81994 to propose Addr.To6(). Give it a if you want it in Go!


  1. IPv4-mapped IPv6 addresses let you handle only IPv6 addresses in your code and keep conversions at a few well-defined boundaries. I use them in Akvorado.
  2. This is not strictly equivalent. The zero value becomes :: while it would make more sense to leave it untouched.
  3. I suppose Go maintainers keep it inconvenient on purpose: they don't want random packages to patch the standard library or other packages.
  4. To avoid catastrophic bugs when netip.Addr's layout changes, you must add tests to detect it. This is more dangerous if you put this code in a package. Either ask users to run the tests themselves, or detect the change at run time and panic. Hiding this optimization behind a build tag would make users aware of this potential trap.
  5. I produced the assembly code with GOTOOLCHAIN=go1.26.8 GOAMD64=v3 ./go-asm '\.asm' from the companion repository, then edited it a bit to keep this article from turning into an endless rabbit hole. Due to an unfortunate sequence of events, we switch between versions: Go 1.27 introduced a regression that blurs the point of this article.
  6. The function is short enough for the compiler to inline it, so the assembly code may vary depending on the surrounding code.
  7. MOVBEQ requires GOAMD64=v3, matching the x86-64-v3 microarchitecture from 2013. Otherwise, the compiler translates BEPutUint64() to BSWAPQ+MOVQ and BEUint64() to MOVQ+BSWAPQ.
  8. When the call is used as a statement, a struct literal alone is invalid. The patch then turns it into _ = netip.Addr .
  9. The compiler can also write an HTML file with the result of each pass:
    $ GOSSAFUNC=AddrTo6Safe \
    > GOTOOLCHAIN=go1.26.8 \
    > GOAMD64=v3 go build -a .
    dumped SSA for AddrTo6Safe,1 to ./ssa.html
    
  10. Source line numbers in parentheses follow value identifiers, but I removed them from the examples. The SSA form also features branches, but we don't need them to understand our optimizations.

30 September 2026

Dirk Eddelbuettel: RcppFastFloat 0.0.6 on CRAN: New Upstream

A new release of RcppFastFloat just arrived on CRAN, and has been built for r2u. The package wraps fast_float, another nice library by Daniel Lemire. For details, see the arXiv preprint or published paper showing that one can convert character representations of numbers into floating point at rates at or exceeding one gigabyte per second. This release updates the underlying fast_float library version to the current version 8.3.0, and also updates a number of the standard packaging details.

Changes in version 0.0.6 (2026-09-29)
  • Update to upstream release 8.3.0 (Daniel Lemire in #6)
  • Standard package maintenace adding README badge, updating URL, updating continuous integration, and adding an aspell dictionary
  • Completed help page for as_double example with @return tag

Courtesy of my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

29 September 2026

Gunnar Wolf: The Origins of GPU Computing

This post is a review for Computing Reviews for The Origins of GPU Computing , a article published in Communications of the ACM
This article appears in a Communications of the ACM series devoted to showcasing relevant examples where federal funding of academic research resulted in widely impacting, clearly identifiable technological advances that have visibly improved or at least changed life far beyond the technological arena. The road is far from trivial from the first parallel computing technologies built completely for research, to a phase that was preeminently directed at gaming, to the current graphics processing units (GPUs) that are basically massively parallel supercomputers used for large language model (LLM) training and inference and other artificial intelligence (AI) tasks and the authors do a good job outlining it in just a few pages. The article is written in a clearly divulgatory way, staying as clear of technical language as possible. The authors present three main enabling technologies for current GPU technologies: parallel computing, parallel graphics systems, and stream processing. Coming from a technical formation myself, although clearly better grounded in the software side of computer engineering than the electronics side, the most interesting nugget of information I took is that parallel computing was born due to the fact that transistor switching consumes very little energy, whereas communication along wires consumes much more. Thus, the more simultaneous operations can be carried out on-chip at once, the less power will be dissipated as heat (and the less time that will be lost due to data stores and loads). Graphics systems are a natural match for parallel systems, as rendering realistic images basically requires repeating the same blocks of instructions to a large array of items. Early GPU implementations were built on fixed-function graphics pipelines. However, by the turn of the century, when nVidia spun off the research labs at Stanford and started marketing their product, introducing along the way the GPU term, they enabled it to implement the full OpenGL graphics pipeline on a single chip. Stream processing is the arrangement of in-chip electronics so that they can better exploit producer-consumer locality between stages in order to achieve greater arithmetic intensity (this is, a greater computation-to-bandwidth ratio), reducing the total number of memory accesses. The authors explain in broad terms the described technologies and how they fit together, as well as how the technology was transferred from the universities to the companies behind today s top trending technologies. The article is engaging and easy to read. As a technologist, I did miss more details dates when specific advances were made, how some of the advances led to future inventions, and so on but I also understand the series is written for outsiders. And, as such, this work very well explains how trillion-dollar technologies are often the result of decades of funding research ideas.

28 September 2026

Maytham Alsudany: Briefly looking at DNS and nameservers under the hood

I've always known that the browser uses the Domain Name Resolution (DNS) system to convert domain names like aezign.com.au into raw IP addresses, but have never actually understood how it works, outside of making sure my network config uses 1.1.1.1 or 8.8.8.8 instead of the crappy one the ISP provides. In this article, I set out to understand what nameservers do, and how they help anything that uses the internet find the right records.

DNS in a nutshellA well-used analogy is a phonebook; when you have someone's name and need to find their phone number, you look through the phonebook. Overall, DNS does the same thing: when you enter an address like example.com, your browser will contact a DNS server (the "phonebook") to determine its IP addresses. On Linux (or MacOS) systems, you can use dig on the command line to perform a DNS lookup and see what this looks like:
$ dig example.com
; <<>> DiG 9.20.29-1-Debian <<>> example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42782
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com.                   IN      A
;; ANSWER SECTION:
example.com.            221     IN      A       104.20.23.154
example.com.            221     IN      A       172.66.147.243
;; Query time: 20 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 21:07:50 AWST 2026
;; MSG SIZE  rcvd: 72
Here, you can see the DNS server being used is 1.1.1.1 (Cloudflare's DNS server), and that it found the IPs 104.20.23.154 and 172.66.147.243 associated with the domain name example.com. Your browser will then contact these IP addresses directly to load the website you are trying to open.

Configuring DNS serversYou can configure DNS servers through your operating system's network settings, and for some applications such as Firefox, with the application itself. If you don't manually set a DNS server, your computer will use the ones provided by your modem. If those are not configured, the default defined by the modem manufacturer or your ISP will be used.The most well-used and reliable DNS servers you can use are Cloudflare's (1.1.1.1 and 1.0.0.1) and Google's (8.8.8.8 and 8.8.4.4). Having more than one DNS server means that if the first one in the list doesn't work or doesn't find the matching IPs for a domain name, it will move to the next one in the list, a backup DNS server.DNS servers must be configured as IP addresses. If your computer tried to use a DNS server at a domain name, then where would your computer find the IP address for that domain name? It wouldn't be able to, as this is the exact gap that DNS servers fulfill.

RecordsDNS is not used only to find IPs for a domain, but a whole host of other things. Collectively they are referred to as "records" (or "DNS records"). Each record is associated to a domain (e.g. example.com) or a subdomain (e.g. bar.example.com, foo.bar.example.com) Here are some of the most common and important types:
  • A - Map to an IPv4 address (e.g. 104.20.23.154).This is the fundamental function of DNS: to map from a domain name to a server's IP address. Multiple A records leads to load balancing, where browsers will randomly select one of the returned IP addresses such that traffic is evenly split between them.
  • AAAA - Map to an IPv6 address (e.g. 2606:4700:10::6814:179a).This serves the same purpose as A records but for IPv6. Multiple AAAA records results in the same load balancing behaviour. All modern websites should have both an A record for legacy compatibility, and an AAAA record to future-proof for the gradual shift towards IPv6 addresses.
  • CNAME - Alias to another domain name (e.g. example.com).DNS servers will recursively lookup records for the aliased domain name until they reach an IP address (in an A or AAAA record).
  • MX - Specify a Mail Exchange server (e.g. mail.example.com).This record tells email SMTP servers where to direct emails to. For instance, the MX record for aezign.com.au is mail.aezign.au; this directs SMTP servers like Gmail and Outlook to send emails to the mailserver running at mail.aezign.au. This also allows specifying a priority field, so that you can define multiple MX records (such as a main and a backup) and consumers will try to connect to the listed mailservers from lowest priority to highest.
  • TXT - Store arbitrary text dataThis is most commonly used for site verification (for instance, to link your website to Google Search Console) and for email security measures like DKIM, DMARC, and SPF.
  • NS - Specify the nameservers for a domain (e.g. zeus.ns.cloudflare.com).These authoritative nameservers are responsible for providing all the other record types when needed.
dig lets you query each of these. For instance, to list the MX records against aezign.com.au:
$ dig MX aezign.com.au
; <<>> DiG 9.20.29-1-Debian <<>> MX aezign.com.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28213
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aezign.com.au.                 IN      MX
;; ANSWER SECTION:
aezign.com.au.          300     IN      MX      10 mail.aezign.au.
;; Query time: 104 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 22:04:58 AWST 2026
;; MSG SIZE  rcvd: 70
Here you can see there is only one mailserver configured for aezign.com.au, which is mail.aezign.au, and it has a priority of 10.If you compare this against something like gmail.com:
$ dig MX gmail.com
; <<>> DiG 9.20.29-1-Debian <<>> MX gmail.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16538
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;gmail.com.                     IN      MX
;; ANSWER SECTION:
gmail.com.              2497    IN      MX      40 alt4.gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      20 alt2.gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      5 gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      10 alt1.gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      30 alt3.gmail-smtp-in.l.google.com.
;; Query time: 120 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 22:06:04 AWST 2026
;; MSG SIZE  rcvd: 161
You can see that gmail.com has several mailservers configured as redundancies due to the Gmail's sheer scale. gmail-smtp-in.l.google.com has the highest priority of 5, so SMTP servers will attempt to contact that mailserver first, followed by alt1.gmail-smtp-in.l.google.com and so on.

NameserversNow there needs to be somewhere for DNS records against a domain name to be defined, right? This is where nameservers come in.Nameservers are DNS servers that answer queries about the domains they are authoritative for i.e. the domains they are in charge of. For example, the authoritative nameservers for example.com are hera.ns.cloudflare.com and elliott.ns.cloudflare.com, which means Cloudflare's nameservers are the source of truth when looking up DNS records for example.com. You can check this with dig:
$ dig NS example.com
; <<>> DiG 9.20.29-1-Debian <<>> NS example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37717
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com.                   IN      NS
;; ANSWER SECTION:
example.com.            81837   IN      NS      hera.ns.cloudflare.com.
example.com.            81837   IN      NS      elliott.ns.cloudflare.com.
;; Query time: 140 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 23:27:19 AWST 2026
;; MSG SIZE  rcvd: 95
Nameservers are configured with your domain name's registrar. The nameserver you set dictates where your remaining DNS records will live. For example.com, this means you would configure all your records on Cloudflare's dashboard, since they are responsible for the domain's DNS records.

Recursive resolutionThe DNS servers that our computers use follow a recursive resolution process behind-the-scenes to find the authoritative nameservers for a domain, followed by the records you are querying.

Finding the root nameserversDue to the recursive nature of the resolution process, there needs to be a starting point: the root (.) nameservers. The resolver uses a fixed list of root nameservers called root hints issued by IANA (named.root), to locate and contact the root nameservers. Resolvers will have this provided to them initially, and can dynamically update it themselves by querying one of the root nameservers for the root NS records to obtain a fresh list. For instance, we can query 198.41.0.4 (a.root-servers.net) to obtain a new list of root nameservers:
$ dig @198.41.0.4 NS .
; <<>> DiG 9.20.29-1-Debian <<>> @198.41.0.4 NS .
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43677
;; flags: qr aa rd; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 27
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;.				IN	NS
;; ANSWER SECTION:
.			518400	IN	NS	l.root-servers.net.
.			518400	IN	NS	j.root-servers.net.
.			518400	IN	NS	f.root-servers.net.
.			518400	IN	NS	h.root-servers.net.
.			518400	IN	NS	d.root-servers.net.
.			518400	IN	NS	b.root-servers.net.
.			518400	IN	NS	k.root-servers.net.
.			518400	IN	NS	i.root-servers.net.
.			518400	IN	NS	m.root-servers.net.
.			518400	IN	NS	e.root-servers.net.
.			518400	IN	NS	g.root-servers.net.
.			518400	IN	NS	c.root-servers.net.
.			518400	IN	NS	a.root-servers.net.
;; ADDITIONAL SECTION:
l.root-servers.net.	518400	IN	A	199.7.83.42
l.root-servers.net.	518400	IN	AAAA	2001:500:9f::42
j.root-servers.net.	518400	IN	A	192.58.128.30
j.root-servers.net.	518400	IN	AAAA	2001:503:c27::2:30
f.root-servers.net.	518400	IN	A	192.5.5.241
f.root-servers.net.	518400	IN	AAAA	2001:500:2f::f
h.root-servers.net.	518400	IN	A	198.97.190.53
h.root-servers.net.	518400	IN	AAAA	2001:500:1::53
d.root-servers.net.	518400	IN	A	199.7.91.13
d.root-servers.net.	518400	IN	AAAA	2001:500:2d::d
b.root-servers.net.	518400	IN	A	170.247.170.2
b.root-servers.net.	518400	IN	AAAA	2801:1b8:10::b
k.root-servers.net.	518400	IN	A	193.0.14.129
k.root-servers.net.	518400	IN	AAAA	2001:7fd::1
i.root-servers.net.	518400	IN	A	192.36.148.17
i.root-servers.net.	518400	IN	AAAA	2001:7fe::53
m.root-servers.net.	518400	IN	A	202.12.27.33
m.root-servers.net.	518400	IN	AAAA	2001:dc3::35
e.root-servers.net.	518400	IN	A	192.203.230.10
e.root-servers.net.	518400	IN	AAAA	2001:500:a8::e
g.root-servers.net.	518400	IN	A	192.112.36.4
g.root-servers.net.	518400	IN	AAAA	2001:500:12::d0d
c.root-servers.net.	518400	IN	A	192.33.4.12
c.root-servers.net.	518400	IN	AAAA	2001:500:2::c
a.root-servers.net.	518400	IN	A	198.41.0.4
a.root-servers.net.	518400	IN	AAAA	2001:503:ba3e::2:30
;; Query time: 476 msec
;; SERVER: 198.41.0.4#53(198.41.0.4) (UDP)
;; WHEN: Mon Sep 28 00:01:08 AWST 2026
;; MSG SIZE  rcvd: 811

Querying the root nameserversFor example.com, the first step is to query the root (.) nameservers to find the nameservers for the .com top-level domain (TLD). Using a.root-servers.net again:
$ dig @198.41.0.4 NS com
; <<>> DiG 9.20.29-1-Debian <<>> @198.41.0.4 NS com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22528
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 13, ADDITIONAL: 27
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;com.				IN	NS
;; AUTHORITY SECTION:
com.			172800	IN	NS	l.gtld-servers.net.
com.			172800	IN	NS	j.gtld-servers.net.
com.			172800	IN	NS	h.gtld-servers.net.
com.			172800	IN	NS	d.gtld-servers.net.
com.			172800	IN	NS	b.gtld-servers.net.
com.			172800	IN	NS	f.gtld-servers.net.
com.			172800	IN	NS	k.gtld-servers.net.
com.			172800	IN	NS	m.gtld-servers.net.
com.			172800	IN	NS	i.gtld-servers.net.
com.			172800	IN	NS	g.gtld-servers.net.
com.			172800	IN	NS	a.gtld-servers.net.
com.			172800	IN	NS	c.gtld-servers.net.
com.			172800	IN	NS	e.gtld-servers.net.
;; ADDITIONAL SECTION:
l.gtld-servers.net.	172800	IN	A	192.41.162.30
l.gtld-servers.net.	172800	IN	AAAA	2001:500:d937::30
j.gtld-servers.net.	172800	IN	A	192.48.79.30
j.gtld-servers.net.	172800	IN	AAAA	2001:502:7094::30
h.gtld-servers.net.	172800	IN	A	192.54.112.30
h.gtld-servers.net.	172800	IN	AAAA	2001:502:8cc::30
d.gtld-servers.net.	172800	IN	A	192.31.80.30
d.gtld-servers.net.	172800	IN	AAAA	2001:500:856e::30
b.gtld-servers.net.	172800	IN	A	192.33.14.30
b.gtld-servers.net.	172800	IN	AAAA	2001:503:231d::2:30
f.gtld-servers.net.	172800	IN	A	192.35.51.30
f.gtld-servers.net.	172800	IN	AAAA	2001:503:d414::30
k.gtld-servers.net.	172800	IN	A	192.52.178.30
k.gtld-servers.net.	172800	IN	AAAA	2001:503:d2d::30
m.gtld-servers.net.	172800	IN	A	192.55.83.30
m.gtld-servers.net.	172800	IN	AAAA	2001:501:b1f9::30
i.gtld-servers.net.	172800	IN	A	192.43.172.30
i.gtld-servers.net.	172800	IN	AAAA	2001:503:39c1::30
g.gtld-servers.net.	172800	IN	A	192.42.93.30
g.gtld-servers.net.	172800	IN	AAAA	2001:503:eea3::30
a.gtld-servers.net.	172800	IN	A	192.5.6.30
a.gtld-servers.net.	172800	IN	AAAA	2001:503:a83e::2:30
c.gtld-servers.net.	172800	IN	A	192.26.92.30
c.gtld-servers.net.	172800	IN	AAAA	2001:503:83eb::30
e.gtld-servers.net.	172800	IN	A	192.12.94.30
e.gtld-servers.net.	172800	IN	AAAA	2001:502:1ca1::30
;; Query time: 772 msec
;; SERVER: 198.41.0.4#53(198.41.0.4) (UDP)
;; WHEN: Mon Sep 28 00:09:25 AWST 2026
;; MSG SIZE  rcvd: 828

Querying the TLD nameserversNow the resolver has .com nameservers and their IPs. We can use one of these, such as 192.5.6.30 (a.gtld-servers.net) to find the nameservers for example.com:
$ dig @192.5.6.30 NS example.com
; <<>> DiG 9.20.29-1-Debian <<>> @192.5.6.30 NS example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37419
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 13
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.com.                   IN      NS
;; AUTHORITY SECTION:
example.com.            172800  IN      NS      hera.ns.cloudflare.com.
example.com.            172800  IN      NS      elliott.ns.cloudflare.com.
;; ADDITIONAL SECTION:
hera.ns.cloudflare.com. 172800  IN      A       108.162.192.162
hera.ns.cloudflare.com. 172800  IN      A       172.64.32.162
hera.ns.cloudflare.com. 172800  IN      A       173.245.58.162
hera.ns.cloudflare.com. 172800  IN      AAAA    2606:4700:50::adf5:3aa2
hera.ns.cloudflare.com. 172800  IN      AAAA    2803:f800:50::6ca2:c0a2
hera.ns.cloudflare.com. 172800  IN      AAAA    2a06:98c1:50::ac40:20a2
elliott.ns.cloudflare.com. 172800 IN    A       108.162.195.228
elliott.ns.cloudflare.com. 172800 IN    A       162.159.44.228
elliott.ns.cloudflare.com. 172800 IN    A       172.64.35.228
elliott.ns.cloudflare.com. 172800 IN    AAAA    2606:4700:58::a29f:2ce4
elliott.ns.cloudflare.com. 172800 IN    AAAA    2803:f800:50::6ca2:c3e4
elliott.ns.cloudflare.com. 172800 IN    AAAA    2a06:98c1:50::ac40:23e4
;; Query time: 80 msec
;; SERVER: 192.5.6.30#53(192.5.6.30) (UDP)
;; WHEN: Mon Sep 28 00:12:19 AWST 2026
;; MSG SIZE  rcvd: 359

Finding the domain's nameserverThe response tells the resolver that the nameservers for example.com are hera.ns.cloudflare.com and elliott.ns.cloudflare.com. Now since these Cloudflare nameservers are also .com domains and Cloudflare is so commonly used, the .com nameserver provides optional sibling glue records as defined in RFC 9471 to prevent further roundtrips and optimize the process.If we ignore the sibling glue records provided, the resolver would have to ask the .com nameservers for the nameservers for cloudflare.com:
$ dig @192.5.6.30 NS cloudflare.com
; <<>> DiG 9.20.29-1-Debian <<>> @192.5.6.30 NS cloudflare.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40307
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 5, ADDITIONAL: 21
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;cloudflare.com.                        IN      NS
;; AUTHORITY SECTION:
cloudflare.com.         172800  IN      NS      ns3.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns5.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns4.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns6.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns7.cloudflare.com.
;; ADDITIONAL SECTION:
ns3.cloudflare.com.     172800  IN      A       162.159.0.33
ns3.cloudflare.com.     172800  IN      A       162.159.7.226
ns3.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:21
ns3.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:7e2
ns5.cloudflare.com.     172800  IN      A       162.159.2.9
ns5.cloudflare.com.     172800  IN      A       162.159.9.55
ns5.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:209
ns5.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:937
ns4.cloudflare.com.     172800  IN      A       162.159.1.33
ns4.cloudflare.com.     172800  IN      A       162.159.8.55
ns4.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:121
ns4.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:837
ns6.cloudflare.com.     172800  IN      A       162.159.3.11
ns6.cloudflare.com.     172800  IN      A       162.159.5.6
ns6.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:30b
ns6.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:506
ns7.cloudflare.com.     172800  IN      A       162.159.4.8
ns7.cloudflare.com.     172800  IN      A       162.159.6.6
ns7.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:408
ns7.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:606
;; Query time: 163 msec
;; SERVER: 192.5.6.30#53(192.5.6.30) (UDP)
;; WHEN: Mon Sep 28 00:26:53 AWST 2026
;; MSG SIZE  rcvd: 573
Here, the .com nameserver provides in-domain glue records for cloudflare.com's nameservers. Without these, there would be a circular dependency, since the nameserver named is a subdomain of the domain being queried for. This is because nameservers are always defined by name, which means that in order to actually contact the nameserver, your computer needs to obtain its IP address with another DNS lookup.Next, we'd use one of these nameservers, say ns3.cloudflare.com, to obtain the IP for hera.ns.cloudflare.com:
$ dig @162.159.0.33 hera.ns.cloudflare.com
; <<>> DiG 9.20.29-1-Debian <<>> @162.159.0.33 hera.ns.cloudflare.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16801
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;hera.ns.cloudflare.com.                IN      A
;; ANSWER SECTION:
hera.ns.cloudflare.com. 86353   IN      A       108.162.192.162
hera.ns.cloudflare.com. 86353   IN      A       173.245.58.162
hera.ns.cloudflare.com. 86353   IN      A       172.64.32.162
;; Query time: 123 msec
;; SERVER: 162.159.0.33#53(162.159.0.33) (UDP)
;; WHEN: Mon Sep 28 01:11:37 AWST 2026
;; MSG SIZE  rcvd: 99

Querying the domain's authoritative nameserverNow we've reached the authoritative nameserver for example.com! The last thing to do is to fetch the A record for example.com to obtain its server's IP address:
$ dig @108.162.192.162 example.com
; <<>> DiG 9.20.29-1-Debian <<>> @108.162.192.162 example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43082
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com.                   IN      A
;; ANSWER SECTION:
example.com.            300     IN      A       104.20.23.154
example.com.            300     IN      A       172.66.147.243
;; Query time: 123 msec
;; SERVER: 108.162.192.162#53(108.162.192.162) (UDP)
;; WHEN: Mon Sep 28 01:13:47 AWST 2026
;; MSG SIZE  rcvd: 72
At last, we've determined that the IP addresses of the servers behind example.com are 104.20.23.154 and 172.66.147.243. This matches our earlier lookup directly against your computer's configured DNS server.The DNS servers your computer uses encapsulate this recursive resolution process behind the scenes, so that your computer only needs to make one round trip, and the servers can use their much faster data center connections and caching to optimize the process and make the query as fast as possible.

Checking DNS recordsTo check a website's records, the first thing to do is to ensure the domain name is pointed at the correct nameserver. For instance, aezign.com.au is supposed to use Cloudflare, since that is where all the DNS records have been set up. We can verify this using dig, (or alternatively with online tools like dnschecker.org):
$ dig NS aezign.com.au
; <<>> DiG 9.20.29-1-Debian <<>> NS aezign.com.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44172
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aezign.com.au.                 IN      NS
;; ANSWER SECTION:
aezign.com.au.          86400   IN      NS      rosalie.ns.cloudflare.com.
aezign.com.au.          86400   IN      NS      zeus.ns.cloudflare.com.
;; Query time: 92 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Mon Sep 28 01:39:22 AWST 2026
;; MSG SIZE  rcvd: 100
If this is not set correctly, the DNS records you configure will not work, as resolvers will be fetching records from somewhere else. In most cases, you can update this in your domain registrar's settings. This is especially common for new domain names, which default to the registrar's nameservers; unless you plan to use their built-in DNS management, you'll need to point them to your chosen provider.Now we can check that the DNS records configured match those your DNS server returns. For aezign.com.au, these are the MX and TXT records configured on Cloudflare.
Extract of Cloudflare's DNS record listing showing the MX record and TXT records for aezign.com.au. The MX record and some TXT records configured for aezign.com.au on Cloudflare's dashboard.
We can check these using dig, and using the +short option to show only the content of the records.
$ dig MX aezign.com.au +short
10 mail.aezign.au.
$ dig TXT aezign.com.au +short
"google-site-verification=KFqTU5xH8fFf6OtszEL4oudfxple29Tm-TvxVtCWkAM"
"v=spf1 mx a:mail.aezign.au -all"
$ dig TXT _dmarc.aezign.com.au +short
"v=DMARC1;p=quarantine;sp=quarantine;adkim=r;aspf=r"
$ dig TXT aezign.com.au-2026._domainkey.aezign.com.au +short
"v=DKIM1; h=sha256; k=rsa; s=email; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzL04yXZgB72YkoxS+tLK//aWx65TS4UA3F0qJ7el68SEuD4ehmg+61Ta9iS31H6U074dnjyoPDeaiGBa7ToTNCtiIts4/ghD/8nrENROE6hQAwRmTK18ODDHrRDW73hWe6Dg4cK5vmXi82/wWRZU9SDKbme28IE9uCVir3lCkKYLs1" "j16gR1Sjqr6a2+3o+EeVXnLJ4wXmJMm7KCIdn7zv0t9Z1eYcw672PdpYPAMpGb8uUsgaDBsNSpTcupPkz4SWj6TrOB80CPHjWseVY8EOJeRsNKQ/8LFoQEBl4Rxtb3HAmu4UIxSZ4QMVOLnXz67e85mA5XjcNIsVMzaWaowwIDAQAB"
As we can see, this matches, which means that the nameservers for aezign.com.au are correct and the DNS records are what we intend them to be.Note that if you've recently changed the nameserver or DNS records for your domain, it can take anywhere between 10 seconds and 24 hours for the changes to take effect and propagate across the different DNS servers globally. Tools like dnschecker.org can check your records on different DNS servers worldwide to see if they're reflecting your changes. The reason for this is caching: DNS servers like 1.1.1.1 will cache results so it doesn't have to do the full resolution process every time, and uses the Time-To-Live (TTL) you configure in your DNS records as a guideline for caching duration.

Maytham Alsudany: A Brief Explanation of DNS and Nameservers

The Domain Name System (DNS) is a fundamental piece of technology behind the modern internet. Thanks to its introduction in 1983, you're able to enter domain names like aezign.com.au instead of raw IP addresses in your browser's address bar and when sending emails. This article explains what it is, how it works, and how you can inspect your own website's DNS and ensure it is set up correctly.

DNS in a nutshellA well-used analogy is a phonebook; when you have someone's name and need to find their phone number, you look through the phonebook. Overall, DNS does the same thing: when you enter an address like example.com, your browser will contact a DNS server (the "phonebook") to determine its IP addresses. On Linux (or MacOS) systems, you can use dig on the command line to perform a DNS lookup and see what this looks like:
$ dig example.com
; <<>> DiG 9.20.29-1-Debian <<>> example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42782
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com.                   IN      A
;; ANSWER SECTION:
example.com.            221     IN      A       104.20.23.154
example.com.            221     IN      A       172.66.147.243
;; Query time: 20 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 21:07:50 AWST 2026
;; MSG SIZE  rcvd: 72
Here, you can see the DNS server being used is 1.1.1.1 (Cloudflare's DNS server), and that it found the IPs 104.20.23.154 and 172.66.147.243 associated with the domain name example.com. Your browser will then contact these IP addresses directly to load the website you are trying to open.

Configuring DNS serversYou can configure DNS servers through your operating system's network settings, and for some applications such as Firefox, with the application itself. If you don't manually set a DNS server, your computer will use the ones provided by your modem. If those are not configured, the default defined by the modem manufacturer or your ISP will be used.The most well-used and reliable DNS servers you can use are Cloudflare's (1.1.1.1 and 1.0.0.1) and Google's (8.8.8.8 and 8.8.4.4). Having more than one DNS server means that if the first one in the list doesn't work or doesn't find the matching IPs for a domain name, it will move to the next one in the list, a backup DNS server.DNS servers must be configured as IP addresses. If your computer tried to use a DNS server at a domain name, then where would your computer find the IP address for that domain name? It wouldn't be able to, as this is the exact gap that DNS servers fulfill.

RecordsDNS is not used only to find IPs for a domain, but a whole host of other things. Collectively they are referred to as "records" (or "DNS records"). Each record is associated to a domain (e.g. example.com) or a subdomain (e.g. bar.example.com, foo.bar.example.com) Here are some of the most common and important types:
  • A - Map to an IPv4 address (e.g. 104.20.23.154).This is the fundamental function of DNS: to map from a domain name to a server's IP address. Multiple A records leads to load balancing, where browsers will randomly select one of the returned IP addresses such that traffic is evenly split between them.
  • AAAA - Map to an IPv6 address (e.g. 2606:4700:10::6814:179a).This serves the same purpose as A records but for IPv6. Multiple AAAA records results in the same load balancing behaviour. All modern websites should have both an A record for legacy compatibility, and an AAAA record to future-proof for the gradual shift towards IPv6 addresses.
  • CNAME - Alias to another domain name (e.g. example.com).DNS servers will recursively lookup records for the aliased domain name until they reach an IP address (in an A or AAAA record).
  • MX - Specify a Mail Exchange server (e.g. mail.example.com).This record tells email SMTP servers where to direct emails to. For instance, the MX record for aezign.com.au is mail.aezign.au; this directs SMTP servers like Gmail and Outlook to send emails to the mailserver running at mail.aezign.au. This also allows specifying a priority field, so that you can define multiple MX records (such as a main and a backup) and consumers will try to connect to the listed mailservers from lowest priority to highest.
  • TXT - Store arbitrary text dataThis is most commonly used for site verification (for instance, to link your website to Google Search Console) and for email security measures like DKIM, DMARC, and SPF.
  • NS - Specify the nameservers for a domain (e.g. zeus.ns.cloudflare.com).These authoritative nameservers are responsible for providing all the other record types when needed.
dig lets you query each of these. For instance, to list the MX records against aezign.com.au:
$ dig MX aezign.com.au
; <<>> DiG 9.20.29-1-Debian <<>> MX aezign.com.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28213
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aezign.com.au.                 IN      MX
;; ANSWER SECTION:
aezign.com.au.          300     IN      MX      10 mail.aezign.au.
;; Query time: 104 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 22:04:58 AWST 2026
;; MSG SIZE  rcvd: 70
Here you can see there is only one mailserver configured for aezign.com.au, which is mail.aezign.au, and it has a priority of 10.If you compare this against something like gmail.com:
$ dig MX gmail.com
; <<>> DiG 9.20.29-1-Debian <<>> MX gmail.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16538
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;gmail.com.                     IN      MX
;; ANSWER SECTION:
gmail.com.              2497    IN      MX      40 alt4.gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      20 alt2.gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      5 gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      10 alt1.gmail-smtp-in.l.google.com.
gmail.com.              2497    IN      MX      30 alt3.gmail-smtp-in.l.google.com.
;; Query time: 120 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 22:06:04 AWST 2026
;; MSG SIZE  rcvd: 161
You can see that gmail.com has several mailservers configured as redundancies due to the Gmail's sheer scale. gmail-smtp-in.l.google.com has the highest priority of 5, so SMTP servers will attempt to contact that mailserver first, followed by alt1.gmail-smtp-in.l.google.com and so on.

NameserversNow there needs to be somewhere for DNS records against a domain name to be defined, right? This is where nameservers come in.Nameservers are DNS servers that answer queries about the domains they are authoritative for i.e. the domains they are in charge of. For example, the authoritative nameservers for example.com are hera.ns.cloudflare.com and elliott.ns.cloudflare.com, which means Cloudflare's nameservers are the source of truth when looking up DNS records for example.com. You can check this with dig:
$ dig NS example.com
; <<>> DiG 9.20.29-1-Debian <<>> NS example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37717
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com.                   IN      NS
;; ANSWER SECTION:
example.com.            81837   IN      NS      hera.ns.cloudflare.com.
example.com.            81837   IN      NS      elliott.ns.cloudflare.com.
;; Query time: 140 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Sun Sep 27 23:27:19 AWST 2026
;; MSG SIZE  rcvd: 95
Nameservers are configured with your domain name's registrar. The nameserver you set dictates where your remaining DNS records will live. For example.com, this means you would configure all your records on Cloudflare's dashboard, since they are responsible for the domain's DNS records.

Recursive resolutionThe DNS servers that our computers use follow a recursive resolution process behind-the-scenes to find the authoritative nameservers for a domain, followed by the records you are querying.

Finding the root nameserversDue to the recursive nature of the resolution process, there needs to be a starting point: the root (.) nameservers. The resolver uses a fixed list of root nameservers called root hints issued by IANA (named.root), to locate and contact the root nameservers. Resolvers will have this provided to them initially, and can dynamically update it themselves by querying one of the root nameservers for the root NS records to obtain a fresh list. For instance, we can query 198.41.0.4 (a.root-servers.net) to obtain a new list of root nameservers:
$ dig @198.41.0.4 NS .
; <<>> DiG 9.20.29-1-Debian <<>> @198.41.0.4 NS .
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43677
;; flags: qr aa rd; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 27
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;.				IN	NS
;; ANSWER SECTION:
.			518400	IN	NS	l.root-servers.net.
.			518400	IN	NS	j.root-servers.net.
.			518400	IN	NS	f.root-servers.net.
.			518400	IN	NS	h.root-servers.net.
.			518400	IN	NS	d.root-servers.net.
.			518400	IN	NS	b.root-servers.net.
.			518400	IN	NS	k.root-servers.net.
.			518400	IN	NS	i.root-servers.net.
.			518400	IN	NS	m.root-servers.net.
.			518400	IN	NS	e.root-servers.net.
.			518400	IN	NS	g.root-servers.net.
.			518400	IN	NS	c.root-servers.net.
.			518400	IN	NS	a.root-servers.net.
;; ADDITIONAL SECTION:
l.root-servers.net.	518400	IN	A	199.7.83.42
l.root-servers.net.	518400	IN	AAAA	2001:500:9f::42
j.root-servers.net.	518400	IN	A	192.58.128.30
j.root-servers.net.	518400	IN	AAAA	2001:503:c27::2:30
f.root-servers.net.	518400	IN	A	192.5.5.241
f.root-servers.net.	518400	IN	AAAA	2001:500:2f::f
h.root-servers.net.	518400	IN	A	198.97.190.53
h.root-servers.net.	518400	IN	AAAA	2001:500:1::53
d.root-servers.net.	518400	IN	A	199.7.91.13
d.root-servers.net.	518400	IN	AAAA	2001:500:2d::d
b.root-servers.net.	518400	IN	A	170.247.170.2
b.root-servers.net.	518400	IN	AAAA	2801:1b8:10::b
k.root-servers.net.	518400	IN	A	193.0.14.129
k.root-servers.net.	518400	IN	AAAA	2001:7fd::1
i.root-servers.net.	518400	IN	A	192.36.148.17
i.root-servers.net.	518400	IN	AAAA	2001:7fe::53
m.root-servers.net.	518400	IN	A	202.12.27.33
m.root-servers.net.	518400	IN	AAAA	2001:dc3::35
e.root-servers.net.	518400	IN	A	192.203.230.10
e.root-servers.net.	518400	IN	AAAA	2001:500:a8::e
g.root-servers.net.	518400	IN	A	192.112.36.4
g.root-servers.net.	518400	IN	AAAA	2001:500:12::d0d
c.root-servers.net.	518400	IN	A	192.33.4.12
c.root-servers.net.	518400	IN	AAAA	2001:500:2::c
a.root-servers.net.	518400	IN	A	198.41.0.4
a.root-servers.net.	518400	IN	AAAA	2001:503:ba3e::2:30
;; Query time: 476 msec
;; SERVER: 198.41.0.4#53(198.41.0.4) (UDP)
;; WHEN: Mon Sep 28 00:01:08 AWST 2026
;; MSG SIZE  rcvd: 811

Querying the root nameserversFor example.com, the first step is to query the root (.) nameservers to find the nameservers for the .com top-level domain (TLD). Using a.root-servers.net again:
$ dig @198.41.0.4 NS com
; <<>> DiG 9.20.29-1-Debian <<>> @198.41.0.4 NS com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22528
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 13, ADDITIONAL: 27
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;com.				IN	NS
;; AUTHORITY SECTION:
com.			172800	IN	NS	l.gtld-servers.net.
com.			172800	IN	NS	j.gtld-servers.net.
com.			172800	IN	NS	h.gtld-servers.net.
com.			172800	IN	NS	d.gtld-servers.net.
com.			172800	IN	NS	b.gtld-servers.net.
com.			172800	IN	NS	f.gtld-servers.net.
com.			172800	IN	NS	k.gtld-servers.net.
com.			172800	IN	NS	m.gtld-servers.net.
com.			172800	IN	NS	i.gtld-servers.net.
com.			172800	IN	NS	g.gtld-servers.net.
com.			172800	IN	NS	a.gtld-servers.net.
com.			172800	IN	NS	c.gtld-servers.net.
com.			172800	IN	NS	e.gtld-servers.net.
;; ADDITIONAL SECTION:
l.gtld-servers.net.	172800	IN	A	192.41.162.30
l.gtld-servers.net.	172800	IN	AAAA	2001:500:d937::30
j.gtld-servers.net.	172800	IN	A	192.48.79.30
j.gtld-servers.net.	172800	IN	AAAA	2001:502:7094::30
h.gtld-servers.net.	172800	IN	A	192.54.112.30
h.gtld-servers.net.	172800	IN	AAAA	2001:502:8cc::30
d.gtld-servers.net.	172800	IN	A	192.31.80.30
d.gtld-servers.net.	172800	IN	AAAA	2001:500:856e::30
b.gtld-servers.net.	172800	IN	A	192.33.14.30
b.gtld-servers.net.	172800	IN	AAAA	2001:503:231d::2:30
f.gtld-servers.net.	172800	IN	A	192.35.51.30
f.gtld-servers.net.	172800	IN	AAAA	2001:503:d414::30
k.gtld-servers.net.	172800	IN	A	192.52.178.30
k.gtld-servers.net.	172800	IN	AAAA	2001:503:d2d::30
m.gtld-servers.net.	172800	IN	A	192.55.83.30
m.gtld-servers.net.	172800	IN	AAAA	2001:501:b1f9::30
i.gtld-servers.net.	172800	IN	A	192.43.172.30
i.gtld-servers.net.	172800	IN	AAAA	2001:503:39c1::30
g.gtld-servers.net.	172800	IN	A	192.42.93.30
g.gtld-servers.net.	172800	IN	AAAA	2001:503:eea3::30
a.gtld-servers.net.	172800	IN	A	192.5.6.30
a.gtld-servers.net.	172800	IN	AAAA	2001:503:a83e::2:30
c.gtld-servers.net.	172800	IN	A	192.26.92.30
c.gtld-servers.net.	172800	IN	AAAA	2001:503:83eb::30
e.gtld-servers.net.	172800	IN	A	192.12.94.30
e.gtld-servers.net.	172800	IN	AAAA	2001:502:1ca1::30
;; Query time: 772 msec
;; SERVER: 198.41.0.4#53(198.41.0.4) (UDP)
;; WHEN: Mon Sep 28 00:09:25 AWST 2026
;; MSG SIZE  rcvd: 828

Querying the TLD nameserversNow the resolver has .com nameservers and their IPs. We can use one of these, such as 192.5.6.30 (a.gtld-servers.net) to find the nameservers for example.com:
$ dig @192.5.6.30 NS example.com
; <<>> DiG 9.20.29-1-Debian <<>> @192.5.6.30 NS example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37419
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 13
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.com.                   IN      NS
;; AUTHORITY SECTION:
example.com.            172800  IN      NS      hera.ns.cloudflare.com.
example.com.            172800  IN      NS      elliott.ns.cloudflare.com.
;; ADDITIONAL SECTION:
hera.ns.cloudflare.com. 172800  IN      A       108.162.192.162
hera.ns.cloudflare.com. 172800  IN      A       172.64.32.162
hera.ns.cloudflare.com. 172800  IN      A       173.245.58.162
hera.ns.cloudflare.com. 172800  IN      AAAA    2606:4700:50::adf5:3aa2
hera.ns.cloudflare.com. 172800  IN      AAAA    2803:f800:50::6ca2:c0a2
hera.ns.cloudflare.com. 172800  IN      AAAA    2a06:98c1:50::ac40:20a2
elliott.ns.cloudflare.com. 172800 IN    A       108.162.195.228
elliott.ns.cloudflare.com. 172800 IN    A       162.159.44.228
elliott.ns.cloudflare.com. 172800 IN    A       172.64.35.228
elliott.ns.cloudflare.com. 172800 IN    AAAA    2606:4700:58::a29f:2ce4
elliott.ns.cloudflare.com. 172800 IN    AAAA    2803:f800:50::6ca2:c3e4
elliott.ns.cloudflare.com. 172800 IN    AAAA    2a06:98c1:50::ac40:23e4
;; Query time: 80 msec
;; SERVER: 192.5.6.30#53(192.5.6.30) (UDP)
;; WHEN: Mon Sep 28 00:12:19 AWST 2026
;; MSG SIZE  rcvd: 359

Finding the domain's nameserverThe response tells the resolver that the nameservers for example.com are hera.ns.cloudflare.com and elliott.ns.cloudflare.com. Now since these Cloudflare nameservers are also .com domains and Cloudflare is so commonly used, the .com nameserver provides optional sibling glue records as defined in RFC 9471 to prevent further roundtrips and optimize the process.If we ignore the sibling glue records provided, the resolver would have to ask the .com nameservers for the nameservers for cloudflare.com:
$ dig @192.5.6.30 NS cloudflare.com
; <<>> DiG 9.20.29-1-Debian <<>> @192.5.6.30 NS cloudflare.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40307
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 5, ADDITIONAL: 21
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;cloudflare.com.                        IN      NS
;; AUTHORITY SECTION:
cloudflare.com.         172800  IN      NS      ns3.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns5.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns4.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns6.cloudflare.com.
cloudflare.com.         172800  IN      NS      ns7.cloudflare.com.
;; ADDITIONAL SECTION:
ns3.cloudflare.com.     172800  IN      A       162.159.0.33
ns3.cloudflare.com.     172800  IN      A       162.159.7.226
ns3.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:21
ns3.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:7e2
ns5.cloudflare.com.     172800  IN      A       162.159.2.9
ns5.cloudflare.com.     172800  IN      A       162.159.9.55
ns5.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:209
ns5.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:937
ns4.cloudflare.com.     172800  IN      A       162.159.1.33
ns4.cloudflare.com.     172800  IN      A       162.159.8.55
ns4.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:121
ns4.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:837
ns6.cloudflare.com.     172800  IN      A       162.159.3.11
ns6.cloudflare.com.     172800  IN      A       162.159.5.6
ns6.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:30b
ns6.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:506
ns7.cloudflare.com.     172800  IN      A       162.159.4.8
ns7.cloudflare.com.     172800  IN      A       162.159.6.6
ns7.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:408
ns7.cloudflare.com.     172800  IN      AAAA    2400:cb00:2049:1::a29f:606
;; Query time: 163 msec
;; SERVER: 192.5.6.30#53(192.5.6.30) (UDP)
;; WHEN: Mon Sep 28 00:26:53 AWST 2026
;; MSG SIZE  rcvd: 573
Here, the .com nameserver provides in-domain glue records for cloudflare.com's nameservers. Without these, there would be a circular dependency, since the nameserver named is a subdomain of the domain being queried for. This is because nameservers are always defined by name, which means that in order to actually contact the nameserver, your computer needs to obtain its IP address with another DNS lookup.Next, we'd use one of these nameservers, say ns3.cloudflare.com, to obtain the IP for hera.ns.cloudflare.com:
$ dig @162.159.0.33 hera.ns.cloudflare.com
; <<>> DiG 9.20.29-1-Debian <<>> @162.159.0.33 hera.ns.cloudflare.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16801
;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;hera.ns.cloudflare.com.                IN      A
;; ANSWER SECTION:
hera.ns.cloudflare.com. 86353   IN      A       108.162.192.162
hera.ns.cloudflare.com. 86353   IN      A       173.245.58.162
hera.ns.cloudflare.com. 86353   IN      A       172.64.32.162
;; Query time: 123 msec
;; SERVER: 162.159.0.33#53(162.159.0.33) (UDP)
;; WHEN: Mon Sep 28 01:11:37 AWST 2026
;; MSG SIZE  rcvd: 99

Querying the domain's authoritative nameserverNow we've reached the authoritative nameserver for example.com! The last thing to do is to fetch the A record for example.com to obtain its server's IP address:
$ dig @108.162.192.162 example.com
; <<>> DiG 9.20.29-1-Debian <<>> @108.162.192.162 example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43082
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com.                   IN      A
;; ANSWER SECTION:
example.com.            300     IN      A       104.20.23.154
example.com.            300     IN      A       172.66.147.243
;; Query time: 123 msec
;; SERVER: 108.162.192.162#53(108.162.192.162) (UDP)
;; WHEN: Mon Sep 28 01:13:47 AWST 2026
;; MSG SIZE  rcvd: 72
At last, we've determined that the IP addresses of the servers behind example.com are 104.20.23.154 and 172.66.147.243. This matches our earlier lookup directly against your computer's configured DNS server.The DNS servers your computer uses employ this recursive resolution technique behind the scenes, so that your computer only needs to make one round trip, and the servers can use their much faster data center connections and caching to optimize the process and make the query as fast as possible.

Verify your DNS recordsTo check your website's records, the first thing you'll want to do is ensure your domain name is pointed at the correct nameserver. For instance, aezign.com.au is supposed to use Cloudflare, since that is where all the DNS records have been set up. We can verify this using dig, (or alternatively with online tools like dnschecker.org):
$ dig NS aezign.com.au
; <<>> DiG 9.20.29-1-Debian <<>> NS aezign.com.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44172
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;aezign.com.au.                 IN      NS
;; ANSWER SECTION:
aezign.com.au.          86400   IN      NS      rosalie.ns.cloudflare.com.
aezign.com.au.          86400   IN      NS      zeus.ns.cloudflare.com.
;; Query time: 92 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Mon Sep 28 01:39:22 AWST 2026
;; MSG SIZE  rcvd: 100
If this is not set correctly, the DNS records you configure will not work, as resolvers will be fetching records from somewhere else. In most cases, you can update this in your domain registrar's settings. This is especially common for new domain names, which default to the registrar's nameservers; unless you plan to use their built-in DNS management, you'll need to point them to your chosen provider.Now we can check that the DNS records you've configured match those your DNS server returns. For aezign.com.au, these are the MX and TXT records configured on Cloudflare.
Extract of Cloudflare's DNS record listing showing the MX record and TXT records for aezign.com.au. The MX record and some TXT records configured for aezign.com.au on Cloudflare's dashboard.
We can check these using dig, and using the +short option to show only the content of the records.
$ dig MX aezign.com.au +short
10 mail.aezign.au.
$ dig TXT aezign.com.au +short
"google-site-verification=KFqTU5xH8fFf6OtszEL4oudfxple29Tm-TvxVtCWkAM"
"v=spf1 mx a:mail.aezign.au -all"
$ dig TXT _dmarc.aezign.com.au +short
"v=DMARC1;p=quarantine;sp=quarantine;adkim=r;aspf=r"
$ dig TXT aezign.com.au-2026._domainkey.aezign.com.au +short
"v=DKIM1; h=sha256; k=rsa; s=email; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzL04yXZgB72YkoxS+tLK//aWx65TS4UA3F0qJ7el68SEuD4ehmg+61Ta9iS31H6U074dnjyoPDeaiGBa7ToTNCtiIts4/ghD/8nrENROE6hQAwRmTK18ODDHrRDW73hWe6Dg4cK5vmXi82/wWRZU9SDKbme28IE9uCVir3lCkKYLs1" "j16gR1Sjqr6a2+3o+EeVXnLJ4wXmJMm7KCIdn7zv0t9Z1eYcw672PdpYPAMpGb8uUsgaDBsNSpTcupPkz4SWj6TrOB80CPHjWseVY8EOJeRsNKQ/8LFoQEBl4Rxtb3HAmu4UIxSZ4QMVOLnXz67e85mA5XjcNIsVMzaWaowwIDAQAB"
As we can see, this matches, which means that the nameservers for aezign.com.au are correct and the DNS records are what we intend them to be.Note that if you've recently changed the nameserver or DNS records for your domain, it can take anywhere between 10 seconds and 24 hours for the changes to take effect and propagate across the different DNS servers worldwide. Tools like dnschecker.org can check your records on different DNS servers worldwide to see if they're reflecting your changes. The reason for this is caching: DNS servers like 1.1.1.1 will cache results so it doesn't have to do the full resolution process every time, and uses the Time-To-Live (TTL) you configure in your DNS records as a guideline for caching duration.

Otto Kek l inen: Using multiple git remotes for true distributed version control

Featured image of post Using multiple git remotes for true distributed version controlOne of the primary design principles for Linus Torvalds when creating Git is that it should be a distributed system, capable of working with multiple remotes. However, most people seem to use Git with just one remote, typically having GitHub or GitLab as the only origin. Version control systems before Git relied on a centralized model. For example in Subversion or CVS you could only sync with one single remote server, which was considered the primary host of the source code. Git however is not just a VCS, but a DVCS, for distributed version control system. If you currently use Git with just one single remote as if it was a centralized version control system, now is the time to change that and learn how to add multiple remotes, and configure more flexible git pull and git push rules. The first concept to grasp is that Git can have multiple remotes. For example, after running a basic clone command such as git clone https://github.com/ottok/debcraft.git you would end up with a single remote origin pointing to GitHub:
$ git remote --verbose
origin https://github.com/ottok/debcraft.git (fetch)
origin https://github.com/ottok/debcraft.git (push)
You can however configure multiple remotes with the command git remote:
$ git remote rename origin github
$ git remote add gitlab https://gitlab.com/ottok/debcraft.git
$ git remote --verbose
github https://github.com/ottok/debcraft.git (fetch)
github https://github.com/ottok/debcraft.git (push)
gitlab https://gitlab.com/ottok/debcraft.git (fetch)
gitlab https://gitlab.com/ottok/debcraft.git (push)
You can push and pull from these individually, or from all at once:
$ git pull --all
Fetching github
Fetching gitlab
Already up to date.
When you review the git history, note the labels gitlab/main and github/main which tell which commit the branch main points to on various remotes. In this case they are all in sync and point to the same 5d1815c:
$ git log --oneline
5d1815c (HEAD -> main, gitlab/main, github/main) Bump Debhelper version from 13 to 14
ca518a7 Temporarily disable automatic copyright year bumping due to frequent bugs
d6e201e Exclude .pm from codespell to decrease false positives (Closes: #1140488)
708ee35 Exclude debian/copyright from codespell due to frequent false positives
When pushing, you can choose what remote to push to. For example, if I am working on branch dev and I only want to push it to GitLab to trigger a CI run there, I can run git push gitlab dev. You can also configure a default remote by running once git push --set-upstream gitlab dev and thereafter simply run git push and it will automatically push the branch dev only to that remote. A related setting is remote.pushDefault, which tells git which remote to push to when a branch does not define one of its own, and it applies to every branch that lacks a [branch] section in .git/config. Setting it once is what lets a repository have a per-branch pull remote, like the debcraft example below, and still have a single, predictable git push target. To see which remote git will use for each of your branches, run git branch -vv, and to refresh the remote-tracking branches of all remotes at once, run git fetch --all --prune. If you want to have multiple remotes, but automatically push and pull to all of them at once, you configure one single remote to have multiple URLs with git remote set-url --add. Since git prints one line per push URL, git remote -v then shows the same remote several times, once for each URL it pushes to. As an illustration, in my actual local Debcraft development project I have all these configured:
$ git remote -v
github-pull-requests git@github.com:ottok/debcraft.git (fetch)
github-pull-requests git@github.com:ottok/debcraft.git (push)
gitlab-merge-requests git@gitlab.com:ottok/debcraft.git (fetch)
gitlab-merge-requests git@gitlab.com:ottok/debcraft.git (push)
origin git@salsa.debian.org:debian/debcraft.git (fetch)
origin git@salsa.debian.org:debian/debcraft.git (push)
origin git@gitlab.com:ottok/debcraft.git (push)
origin git@github.com:ottok/debcraft.git (push)
otto git@salsa.debian.org:otto/debcraft.git (fetch)
otto git@salsa.debian.org:otto/debcraft.git (push)
otto git@gitlab.com:ottok/debcraft.git (push)
otto git@github.com:ottok/debcraft.git (push)
otto git@git.sr.ht:~ottok/debcraft (push)
otto ssh://git@codeberg.org/ottok/Debcraft.git (push)
salsa-merge-requests git@salsa.debian.org:debian/debcraft.git (fetch)
salsa-merge-requests git@salsa.debian.org:debian/debcraft.git (push)
The settings can also be viewed directly in the config file:
$ cat .git/config
...
[remote "origin"]
url = git@salsa.debian.org:debian/debcraft.git
fetch = +refs/heads/*:refs/remotes/origin/*
pushurl = git@salsa.debian.org:debian/debcraft.git
pushurl = git@gitlab.com:ottok/debcraft.git
pushurl = git@github.com:ottok/debcraft.git
[remote "otto"]
url = git@salsa.debian.org:otto/debcraft.git
fetch = +refs/heads/*:refs/remotes/otto/*
pushurl = git@salsa.debian.org:otto/debcraft.git
pushurl = git@gitlab.com:ottok/debcraft.git
pushurl = git@github.com:ottok/debcraft.git
pushurl = git@git.sr.ht:~ottok/debcraft
pushurl = ssh://git@codeberg.org/ottok/Debcraft.git
...
[branch "main"]
remote = origin
merge = refs/heads/main
[branch "dev"]
remote = otto
merge = refs/heads/dev
...
On branch main: On branch dev: The command output will have the same message three times, once for each remote (assuming they were all already up-to-date):
$ git push
Everything up-to-date
Everything up-to-date
Everything up-to-date

Use cases for distributed version control The most obvious use case for any distributed system is improved availability. In this example, the salsa.debian.org is often overloaded/unavailable, so pushing to GitLab and GitHub as well allows collaborators to use them to fetch git commits if the primary code hosting site is unresponsive. Once it comes back online again, the next git push/pull will automatically bring it up to the same content as what backup hosts had. The second main benefit is that having the code live on multiple code forges allows collaborators to use their own favorite host. Debcraft itself is a great example of this, as collaborators who don t have a Salsa account can - and have - submitted Merge Requests on GitLab.com and Pull Requests on GitHub.com. A third use case is code hosting migrations. If you for example want to gradually phase out GitHub.com and replace it with something else, you could configure git to pull from the old location and push to both, ensuring that other collaborators can start using the new code hosting location without disruptions to those who have not yet updated their remotes or done a fresh git clone from the new location.

Automating setting up a project on multiple code forges While I don t push all projects I work on to all Salsa/GitLab/GitHub, I do set up multiple remotes for enough git projects that I have this script to automate configuring a git repository that is cloned from Salsa to also push to GitLab and GitHub. It accepts the repository whether it was cloned over SSH or HTTPS, and it adds the push URLs and then pushes the current branch, all branches, and tags for you:
#!/bin/bash
# Add GitLab and GitHub push URLs to an existing git remote, so that a plain
# "git push" writes the same commits to all three code forges.
#
# Usage: ./git-multipush-salsa.sh
#
# Environment:
# SALSA_REMOTE remote to turn into a mirror, default origin
# SALSA_USER your Salsa user, only used to detect that the source
# remote is somebody else's project
# GITLAB_USER your GitLab user, default ottok
# GITHUB_USER your GitHub user, default ottok
# PROTOCOL ssh (default) or https, for the push URLs
set -euo pipefail
SCRIPT=$(basename "$0")
SALSA_REMOTE=$ SALSA_REMOTE:-origin 
SALSA_USER=$ SALSA_USER:-otto 
GITLAB_USER=$ GITLAB_USER:-ottok 
GITHUB_USER=$ GITHUB_USER:-ottok 
PROTOCOL=$ PROTOCOL:-ssh 
die()   echo "$SCRIPT: $*" >&2; exit 1;  
# A failing command leaves the push URLs half configured behind, so say how
# to get back to where we started
fail()
 
echo "$SCRIPT: '$*' failed, the push URLs may already have been added" >&2
echo " git config --unset-all remote.$ SALSA_REMOTE .pushurl" >&2
echo " git remote set-url --add --push $ SALSA_REMOTE  $ SALSA_URL " >&2
exit 1
 
git rev-parse --git-dir > /dev/null 2>&1   die "not inside a git repository"
git remote   grep -qx "$SALSA_REMOTE"   die "no remote named '$SALSA_REMOTE'"
# Example: git@salsa.debian.org:otto/salsa-ci-pipeline.git
SALSA_URL=$(git remote get-url "$SALSA_REMOTE")
# The source remote may use any of the URL forms salsa.debian.org accepts,
# for example git@salsa.debian.org:otto/salsa-ci-pipeline.git or
# https://salsa.debian.org/otto/salsa-ci-pipeline.git
case "$SALSA_URL" in
git@salsa.debian.org:*) SALSA_PATH=$ SALSA_URL#git@salsa.debian.org:  ;;
ssh://git@salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#ssh://git@salsa.debian.org/  ;;
https://salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#https://salsa.debian.org/  ;;
http://salsa.debian.org/*) SALSA_PATH=$ SALSA_URL#http://salsa.debian.org/  ;;
*) die "unexpected source URL '$SALSA_URL', expected a salsa.debian.org URL" ;;
esac
SALSA_PATH=$ SALSA_PATH%/ 
SALSA_PATH=$ SALSA_PATH%.git 
SOURCE_USER=$ SALSA_PATH%%/* 
PROJECT=$ SALSA_PATH##*/ 
# Write the push URLs in one and the same protocol, no matter how the
# repository happens to have been cloned
if [ "$PROTOCOL" = "https" ]
then
SALSA_URL="https://salsa.debian.org/$ SALSA_PATH .git"
GITLAB_URL="https://gitlab.com/$ GITLAB_USER /$ PROJECT .git"
GITHUB_URL="https://github.com/$ GITHUB_USER /$ PROJECT .git"
else
SALSA_URL="git@salsa.debian.org:$ SALSA_PATH .git"
GITLAB_URL="git@gitlab.com:$ GITLAB_USER /$ PROJECT .git"
GITHUB_URL="git@github.com:$ GITHUB_USER /$ PROJECT .git"
fi
# Mirroring somebody else's project is fine, but it is usually not what you
# want: you most likely want to mirror your own fork instead
if [ "$SOURCE_USER" != "$SALSA_USER" ]
then
echo "Note: $SALSA_REMOTE points to $SOURCE_USER's project, not your own."
echo "To mirror your own fork, run this with SALSA_REMOTE set to its name."
echo
fi
# Bail out instead of adding the same push URLs twice
EXISTING=$(git remote get-url --push --all "$SALSA_REMOTE")
if [ "$(printf '%s\n' "$EXISTING"   wc -l)" -gt 1 ]
then
die "remote '$SALSA_REMOTE' already has multiple push URLs:
$EXISTING"
fi
echo "Mirroring $SALSA_URL to $GITLAB_URL and $GITHUB_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$SALSA_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$GITLAB_URL"
git remote set-url --add --push "$SALSA_REMOTE" "$GITHUB_URL"
# Push the branch you have checked out first, and never with --force: the
# first branch a GitLab or GitHub project receives becomes its default branch,
# the one you then protect
DEFAULT_BRANCH=$(git symbolic-ref --short HEAD)   die "not on a branch, check out a branch first"
git push -u "$SALSA_REMOTE" "$DEFAULT_BRANCH"
git push "$SALSA_REMOTE" --all
git push "$SALSA_REMOTE" --tags
# The push above creates the GitLab and GitHub projects, but GitLab projects
# are private by default, so make the new mirror public with glab if installed.
if command -v glab > /dev/null
then
echo "Make the GitLab project public and describe it as a mirror:"
glab api -X PUT "projects/$ GITLAB_USER %2F$ PROJECT " \
-f visibility=public \
-f description="Mirror of $ SALSA_URL " \
-f notification_email=disabled
else
echo "Install glab to do the same automatically, or visit"
echo "https://gitlab.com/$ GITLAB_USER /$ PROJECT /edit to make the project public"
fi
echo "To undo, drop all push URLs and add back only the original one:"
echo " git config --unset-all remote.$ SALSA_REMOTE .pushurl"
echo " git remote set-url --add --push $ SALSA_REMOTE  $ SALSA_URL "
If you prefer not to install glab, the last part simply reduces to visiting the GitLab project settings once and ticking the project public, and it is also worth setting a description like Mirror of https://salsa.debian.org/debian/debcraft and disabling notification email there, as nobody wants a mirror sending out a flood of emails every time you push. There are a few caveats worth knowing about before mirroring everything everywhere. Every push to a remote with multiple push URLs writes to all of them, so a project with continuous integration configured on three forges will run its CI three times and cost you roughly three times as much. If one of the push URLs fails, say because the host is momentarily unreachable, git prints the error and stops there, leaving the remotes listed after the failing one without the new commits. Simply re-run the push once the host is reachable again; the remotes that did get the commits will just respond with Everything up-to-date. And you should never force-push to only one of the remotes, as that leaves the mirrors permanently out of sync with your local repository.

Avoid pulling too much Related to multiple remotes, you should also be aware that git allows one to configure which branches to pull. By default a git clone will track all branches from the remote and pull everything, which in a typical open source project will bring in a lot of temporary development branches in vain. Thus it is better to run git clone with the --single-branch and --branch parameters to fetch only the branch that matters, which in most cases is the main branch. That can be accomplished with:
git clone --single-branch --branch main https://github.com/ottok/debcraft.git
If a repository was already cloned, it can be configured to track only one branch with:
git config remote.upstream.fetch "+refs/heads/main:refs/remotes/upstream/main"
git fetch upstream
git config --get-all remote.upstream.fetch
+refs/heads/main:refs/remotes/upstream/main
If you later need to track more branches, you can configure them with commands such as:
git config --add remote.upstream.fetch "+refs/heads/3.2.y:refs/remotes/upstream/3.2.y"
git fetch upstream
git checkout 3.2.y
For additional details about any of the tips in this post I recommend reading the git man pages.

Now go and spread your code around Distributing your code over multiple code forges costs you almost nothing in configuration, and in return you get availability, a choice of host for your collaborators, and freedom to migrate without a big bang. Add the push URLs to the remotes you already have, push once, and you are done. The rest of the git fundamentals are covered in my other posts. If your commits are not yet polished, start with what makes a good git commit message and git commit messages by example. If you work on Debian packaging, Git-based collaboration in Debian explains why the Salsa merge request workflow is built the way it is, and Salsa merge request best practices covers the review process itself.

27 September 2026

Dirk Eddelbuettel: #060: Using bubblewrap for R sandboxing

Welcome to post 60 in the R4 series. bubblewrap is a great tool and very suitable for using with an agent harness. It is a very compelling and lightweight alternative to using a full-blown docker container as it offers low-level unprivileged sandboxing on Linux hosts. In a nutshell, bubblewrap can turn everything off (see unshare-all below) and allow access only to selected services and directories (as shown below). This makes it a very useful tool be used on a main workstation as it can provide a lower-risk deployment quite easily while taking advantage of the already installed software stack. I continue to get a lot of value out of docker, especially as r2u makes installing R package dependencies so trivial. Its slogan easy, fast, reliable: pick all three clearly holds for r2u. But sometimes bubblewrap is compelling, for example to launch opencode (documented for example at the debian-inference site). When using R we need add more directories to the example to provide /etc/alternatives which is governing inter alia the LAPACK / BLAS resolution on Debian / Ubuntu systems; this is now on debian-inference site but wasn t when I first tried it a few days ago ;-) as well as /etc/R for config files and possibly ~/.R/Makevars for compiler settings. With that my current wrapper is
#!/bin/bash
#
# cf https://inference.debian.net/doc
#    and 'curl' command to set bearer code
bwrap \
  --ro-bind /usr /usr \
  --symlink usr/bin /bin \
  --symlink usr/lib /lib \
  --symlink usr/lib64 /lib64 \
  --ro-bind /etc/ssl /etc/ssl \
  --ro-bind /etc/resolv.conf /etc/resolv.conf \
  --ro-bind $HOME/.gitconfig $HOME/.gitconfig \
  --bind $HOME/.config/opencode $HOME/.config/opencode \
  --bind $HOME/.cache/opencode $HOME/.cache/opencode \
  --bind $HOME/.opencode $HOME/.opencode \
  --bind $HOME/.local/share/opencode $HOME/.local/share/opencode \
  --bind $HOME/.local/state/opencode $HOME/.local/state/opencode \
  --ro-bind $HOME/.R/Makevars $HOME/.R/Makevars \
  --ro-bind /etc/R/ /etc/R \
  --ro-bind /etc/alternatives/ /etc/alternatives \
  --proc /proc \
  --dev /dev \
  --tmpfs /tmp \
  --bind $(pwd) $(pwd) \
  --chdir $(pwd) \
  --unshare-all \
  --share-net \
  --die-with-parent \
  /usr/local/bin/opencode "$@"
Launched that way we can have agents use R to check packages sources, builds, and triage for bugs. this works equally well with local models served via ollama (and with that a quick shoutout to packages.lingfish.net for providing an apt service for ollama) as it does with the various cloud-based offerings. So harness away with R!

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

20 September 2026

Vincent Bernat: Bot-free self-hosted analytics with GoatCounter on NixOS

In 2016, I removed Google Analytics from this blog to avoid being complicit in feeding the biggest machine for harvesting personal data. Instead, I relied on GoAccess to analyze my server logs.1 For the past couple of years, the statistics have made no sense, despite my attempts to filter bots: AI scrapers inflate the number of visitors to around 2,000 per day. Eventually, I settled on GoatCounter, an open-source, privacy-friendly web analytics platform. I replaced the JavaScript client to filter bots more aggressively and added a CSS fallback. To improve reliability, I implemented a local proxy running on each of the five web servers serving this blog. The rest of this post details how these pieces fit together and how I deploy them on NixOS.

Why GoatCounter? GoatCounter does not collect personal data: instead of storing the reader s IP address or relying on cookies, it creates a session identifier valid for 8 hours from the user agent and the IP address. Its feature set is modest but sufficient for a blog. If you want to look at the interface, GoatCounter s author runs a public instance for his site. A hosted version lets you try it before running your own instance. With a single binary and an SQLite database, GoatCounter is one of the lightest self-hosted solutions. Privacy-friendly alternatives, in increasing order of complexity, include Umami, Plausible, and Rybbit.
GoatCounter dashboard showing some statistics from my blog, including an article on the spanning tree protocol with 1,224 views for the past week, the referrers, and the breakdown of browsers (52% Chrome, 32% Firefox, with 16% for Firefox 155 and 6% for Firefox 156)

Custom JavaScript client GoatCounter includes a small JavaScript client 2,189 bytes minified and gzipped. It ships some features I don t use: a visitor counter, tracking clicks, configurable settings, etc. I replace it with this function to register a hit:
const count = (  event, title   =  ) =>  
  const params = new URLSearchParams( 
    p: event   location.pathname,
    t: title   document.title,
    r: document.referrer,
    q: location.search,
    s: document.documentElement.clientWidth,
    e: !!event,
    rnd: Math.random().toString(36).slice(2, 7),
   );
  fetch( /count?$ params  ,   keepalive: true  ).catch(() =>  );
 ;
To filter bots,2 I go the extra mile by requiring a user interaction an idea I stole from Bear Blog.
let sendHit = () => (sendHit = () =>  , count());
["touchmove", "mousemove", "keydown", "pointerdown"].forEach((eventName) =>
  document.addEventListener(eventName, sendHit,  
    once: true,
    passive: true,
   ),
);
If a reader has disabled JavaScript in their browser, I record the hit using a CSS image. The :hover pseudo-class loads it only after an interaction, another trick stolen from Bear Blog. About 2% of my visitors fit into this bucket.3
<!DOCTYPE html>
<html lang="en" class="nojs">
  <head>
    <script>
      // The JavaScript code for this blog requires ES6
      if ("noModule" in HTMLScriptElement.prototype)
        document.documentElement.classList.remove("nojs");
    </script>
  </head>
  <body>
  <!-- ... -->
    <style>
      .nojs body:hover  
        border-width: 0;
        border-image: url('/count?p=/en/blog/2026-kpi-goodhart&t=Building...&r=NoJS&e=false');
       
    </style>
  </body>
</html>
Where GoAccess reported around 2,000 visitors a day, GoatCounter counts fewer than 200 humans.4 I assume AI scrapers use a low-effort approach: if the content is available without barriers, as on this blog, they don t spawn a complex mechanized browser that could trigger a page view. Even crawlers running JavaScript, like Googlebot with its headless Chromium, do not interact with the page and never trigger the events I listen to. The interaction-based proof of humanity I use is likely to keep working.5

Local proxy Five servers across the world in Europe and in North America serve the content of this website, but GoatCounter runs on only one of them. To avoid losing track of visitors when GoatCounter is down, I run a local proxy listening on the same /count endpoint. On each server, it stores the hits in memory with a buffer large enough to survive several days of downtime. It sends them in batches to the upstream backend using the /api/v0/count authenticated endpoint.
Servers on a map. web02 is in Paris, web03 in Helsinki, web04 in Nuremberg, web05 in Ashburn, web06 in Chicago.
I proposed the code for the proxy in pull request #909. GoatCounter s maintainer declined to maintain so much code for such a niche use case. As a fellow open-source developer, I often hold the same position for my own projects: a one-time contributor effort may translate into a long-term maintainer commitment. I expose the endpoint for the proxy on the domain of this website to evade ad blockers. This sounds like I don t respect the reader s choice, but as GoatCounter is privacy-friendly, I find it acceptable.
location = /count  
  access_log off;
  proxy_pass http://127.0.0.3:8087/count;
  proxy_pass_request_headers off;
  proxy_set_header Accept-Language $http_accept_language;
  proxy_set_header User-Agent $http_user_agent;
  proxy_set_header X-Real-Ip $remote_addr;
 

Deploying on NixOS My web servers run NixOS, a declarative Linux distribution with built-in configuration management. I manage this small fleet with Colmena, a stateless deployment tool for NixOS. My configuration is available on GitHub.

Deploying applications in containers For better isolation, each application runs inside an ephemeral lightweight container, powered by systemd-nspawn. Each container runs a stripped-down NixOS instance. A module wraps NixOS s containers options to avoid repeating the same options for each application.6 The containers share their network namespace with the host: the additional isolation is not worth the increased complexity. For a smaller footprint, I also disable a few non-essential services.
  config, lib, ...  :
let
  cfg = config.luffy.containers;
in
 
  # User-configurable settings for our custom module
  options.luffy.containers = lib.mkOption  
    default =    ;
    description = "Ephemeral containers sharing the host network.";
    type = lib.types.attrsOf (lib.types.submodule  
      options =  
        config = lib.mkOption  
          type = lib.types.deferredModule;
          default =    ;
          description = "NixOS configuration of the container.";
         ;
       ;
     );
   ;
  # Translate our options to NixOS containers
  config =  
    containers = lib.mapAttrs
      (name: container:  
        ephemeral = true;
        autoStart = true;
        privateNetwork = false;
        extraFlags = [ "--resolv-conf=replace-host" ];
        config =  
          imports = [ container.config ];
          networking.firewall.enable = false;
          system.stateVersion = config.system.stateVersion;
          systemd.services =  
            console-getty.enable = false;
            systemd-logind.enable = false;
            systemd-oomd.enable = false;
           ;
         ;
       )
      cfg;
   ;
 
To configure a GoatCounter instance running in a container and listening on 127.0.0.4:8088, we import the module7 and declare the container in the config.luffy.containers attribute set:
  pkgs, config, ...  :  
  imports = [ ./modules/container.nix ];
  config.luffy.containers.goatcounter =  
    config =  
      services.goatcounter =  
        enable = true;
        address = "127.0.0.4";
        port = 8088;
        proxy = true;
       ;
     ;
   ;
 
As the containers are ephemeral, we need to keep persistent data in directories on the host. We add a mounts option and ask NixOS s containers to expose the configured directories through the bindMounts option.
  config, lib, ...  :
let
  cfg = config.luffy.containers;
in
 
  options.luffy.containers = lib.mkOption  
    type = lib.types.attrsOf (lib.types.submodule  
      options =  
        mounts = lib.mkOption  
          type = lib.types.listOf lib.types.str;
          default = [ ];
          description = "Host directories mounted read-write at the same place.";
         ;
       ;
     );
   ;
  config =  
    containers = lib.mapAttrs
      (name: container:  
        bindMounts =
          lib.genAttrs container.mounts (path:   hostPath = path; isReadOnly = false;  );
       )
      cfg;
   ;
 
For example, to persist GoatCounter s database in the /var/db/goatcounter directory on the host, we add the directory to the mounts option and alter the service definition to tell GoatCounter where the database is.
  config, ...  :
let
  databaseDirectory = "/var/db/goatcounter";
in  
  config.luffy.containers.goatcounter =  
    mounts = [ databaseDirectory ];
    config =  
      services.goatcounter =  
        extraArgs = [ "-db=sqlite+$ databaseDirectory /db.sqlite" ];
       ;
     ;
   ;
 
A container may also need some secrets. Colmena can upload secrets without storing them in the Nix store. We add a keys option to our containers. It takes an attribute set mapping secret names to the commands to populate them. Then, the module declares the required secrets to Colmena in the deployment.keys option, makes the container depend on the presence of the secrets, and exposes them to the container.
  config, lib, ...  :
let
  cfg = config.luffy.containers;
in
 
  options.luffy.containers = lib.mkOption  
    type = lib.types.attrsOf (lib.types.submodule  
      options =  
        keys = lib.mkOption  
          type = lib.types.attrsOf (lib.types.listOf lib.types.str);
          default =    ;
          description = "Secrets, as a command to run locally. They are mounted in /etc.";
         ;
       ;
     );
   ;
  config =  
    # Colmena uploads each secret in  /var/keys  and make them available
    # to the group "keys".
    deployment.keys = lib.concatMapAttrs
      (_: container: lib.mapAttrs
        (_: keyCommand:  
          inherit keyCommand;
          group = "keys";
          permissions = "0640";
          destDir = "/var/keys";
         )
        container.keys)
      cfg;
    # The container can only start if the required secrets are available.
    systemd.services = lib.mapAttrs'
      (name: container:
        let
          units = map (key: "$ key -key.service") (lib.attrNames container.keys);
        in
        lib.nameValuePair "container@$ name "  
          requires = units;
          after = units;
         )
      cfg;
    # Mount each secret inside the container.
    containers = lib.mapAttrs
      (name: container:  
        bindMounts = lib.mapAttrs'
          (key: _: lib.nameValuePair "/etc/$ key "  
            hostPath = "/var/keys/$ key ";
            isReadOnly = true;
           )
          container.keys;
       )
      cfg;
   ;
 
For example, GoatCounter needs credentials to download the GeoIP database. I provide a local command to fetch the secret from my password manager and expose it inside the container through the /etc/goatcounter.env environment file.
  pkgs, config, ...  : 
let
  keyCommand = variable: [
    "$ pkgs.runtimeShell "
    "-c"
    "pass show personal/nixops/secrets   grep '^$ variable ='"
  ];
in  
  config.luffy.containers.goatcounter =  
    keys."goatcounter.env" = keyCommand "GOATCOUNTER_GEODB";
    config =  
      systemd.services.goatcounter.serviceConfig =  
        EnvironmentFile = "/etc/goatcounter.env";
        SupplementaryGroups = [ "keys" ];
       ;
     ;
   ;
 

GoatCounter server Nixpkgs already packages GoatCounter. By overriding the src and vendorHash attributes, I reuse its definition for my custom version with the proxy:
  goatcounter, fetchFromGitHub  :
goatcounter.overrideAttrs (_:  
  src = fetchFromGitHub  
    owner = "vincentbernat";
    repo = "goatcounter";
    rev = "feature/proxy";
    hash = "sha256-dJRlQlFu3tjcEgabT1LEbyFrasJlhmYu4L/T7EkoNcY=";
   ;
  vendorHash = "sha256-c9Q5OrbZR+q6pD3SgPPWe8JUzcZco1AVUKGaV61k5DE=";
 )
I wrote a NixOS module to encapsulate GoatCounter: the container definition, the service definition, and the secrets. The module accepts the following options: package, serve.enable, serve.listenAddress, serve.port, and serve.databaseFile. I already detailed the container configuration in the previous section. In the end, I chose not to reuse the GoatCounter module from NixOS: it s small, so it s better to insulate my module from unexpected future changes.
  config, pkgs, lib, ...  :
let
  cfg = config.luffy.goatcounter;
  databaseDirectory = builtins.dirOf cfg.serve.databaseFile;
  chown = "$ pkgs.coreutils /bin/chown -R";
in  
  config.luffy.containers.goatcounter =  
    config.systemd.services.goatcounter =  
      description = "GoatCounter Web Analytics";
      wantedBy = [ "multi-user.target" ];
      serviceConfig =  
        EnvironmentFile = "/etc/goatcounter.env";
        SupplementaryGroups = [ "keys" ];
        DynamicUser = true;
        Restart = "always";
        ExecStart = lib.escapeShellArgs [
          (lib.getExe cfg.package)
          "serve"
          "-listen=$ cfg.serve.listenAddress :$ toString cfg.serve.port "
          "-tls=none"
          "-db=sqlite+$ cfg.serve.databaseFile "
          "-automigrate"
        ];
        # Transfer database ownership to dynamically assigned user "goatcounter".
        ExecStartPre = "+$ chown  goatcounter:goatcounter $ databaseDirectory ";
        ReadWritePaths = databaseDirectory;
       ;
     ;
   ;
 
The following snippet configures GoatCounter to listen on 127.0.0.4:8088:
 
  luffy.goatcounter =  
    serve =  
      enable = true;
      listenAddress = "127.0.0.4";
      port = 8088;
     ;
   ;
 
The last step is to configure nginx to expose GoatCounter on the Internet. I disable the /count endpoint as the local proxy handles it.
  config, ...  :
let
  cfg = config.luffy.goatcounter.serve;
in
 
  services.nginx.virtualHosts."goatcounter.luffy.cx" =  
    forceSSL = true;
    locations =  
      "/" =  
        proxyPass = "http://$ cfg.listenAddress :$ toString cfg.port ";
       ;
      "= /count".extraConfig = ''
        return 404;
      '';
     ;
   ;
 

GoatCounter proxy The same NixOS module configures the local proxy, with the following options: proxy.enable, proxy.listenAddress, proxy.port, and proxy.site the site receiving the batches of page views. The local proxy has no persistent data, but it needs the API key to authenticate to the main GoatCounter instance: its container uses the keys option but not the mounts option.
  config, pkgs, lib, ...  :
let
  cfg = config.luffy.goatcounter;
  keyCommand = _: [ " " ];
in
 
  config.luffy.containers.goatcounter-proxy =  
    keys."goatcounter-proxy.env" = keyCommand "GOATCOUNTER_API_KEY";
    config.systemd.services.goatcounter =  
      description = "GoatCounter Proxy.";
      wantedBy = [ "multi-user.target" ];
      serviceConfig =  
        EnvironmentFile = "/etc/goatcounter-proxy.env";
        SupplementaryGroups = [ "keys" ];
        DynamicUser = true;
        Restart = "always";
        ExecStart = lib.escapeShellArgs [
          (lib.getExe cfg.package)
          "proxy"
          "-site=$ cfg.proxy.site "
          "-listen=$ cfg.proxy.listenAddress :$ toString cfg.proxy.port "
          "-ratelimit=10/1"  # 10 requests per second per IP
        ];
       ;
     ;
   ;
 
For each server, I enable the local proxy with the following snippet. The nginx configuration shown earlier exposes the /count endpoint under the same domain as my blog.
 
  luffy.goatcounter =  
    proxy =  
      enable = true;
      site = "goatcounter.luffy.cx";
      listenAddress = "127.0.0.3";
      port = 8087;
     ;
   ;
 

Backup of the SQLite database with Litestream Litestream is a streaming replication tool for SQLite databases. It compresses the changes committed to the write-ahead log (WAL) next to the database and sends them to a remote destination. I encapsulate its configuration in a NixOS module, which takes an attribute set databases mapping a name to the path of the database to back up. Litestream also runs in a container. I mount the databases to replicate, as well as the secrets to push the backups to a Hetzner storage box using SFTP:
  config, pkgs, lib, ...  :
let
  cfg = config.luffy.litestream;
  databaseDirs = lib.unique (map builtins.dirOf (builtins.attrValues cfg.databases));
in
 
  config = lib.mkIf (cfg.databases !=    )  
    luffy.containers.litestream =  
      mounts = databaseDirs;
      keys."litestream.env" = [
        "$ pkgs.runtimeShell "
        "-c"
        "pass show personal/nixops/secrets   grep '^SQLITE_BACKUP_'"
      ];
     ;
   ;
 
Inside the container, I configure Litestream through NixOS s services.litestream options:
  • full snapshots every day, kept for 15 days,
  • three levels of compaction for transaction files: 5 minutes, 30 minutes, and 3 hours,
  • auto-recovery,8
  • replica stored in a directory matching the host name, and
  • credentials read from /etc/litestream.env and exposed through variable expansion.
  config, pkgs, lib, ...  :
let
  cfg = config.luffy.litestream;
in
 
  config.luffy.containers.litestream =  
    config =  
      # The databases belong to dynamically allocated users, whose UID is
      # not known here, so Litestream runs as root.
      systemd.services.litestream.serviceConfig =  
        User = lib.mkForce "root";
        Group = lib.mkForce "root";
       ;
      # Use NixOS service.
      services.litestream =  
        enable = true;
        environmentFile = "/etc/litestream.env";
        settings =  
          auto-recover = true;
          snapshot =  
            interval = "24h";
            retention = "360h";
           ;
          levels = [
              interval = "5m";  
              interval = "30m";  
              interval = "3h";  
          ];
          dbs = lib.mapAttrsToList
            (name: path:  
              inherit path;
              replica =  
                type = "sftp";
                host = "\$ SQLITE_BACKUP_HOST ";
                user = "\$ SQLITE_BACKUP_USER ";
                password = "\$ SQLITE_BACKUP_PASSWORD ";
                host-key = "\$ SQLITE_BACKUP_HOSTKEY ";
                path = "$ config.networking.hostName /$ name ";
               ;
             )
            cfg.databases;
         ;
       ;
     ;
   ;
 
To back up GoatCounter s database, I declare a goatcounter attribute in luffy.litestream.databases, set to the database path:
  config, ...  :
let
  cfg = config.luffy.goatcounter.serve;
in
 
  luffy.litestream.databases.goatcounter = cfg.databaseFile;
 
On the SFTP server, we can inspect Litestream s work, with the compacted transactions and the full snapshots:
  ls web02/goatcounter/ltx
web02/goatcounter/ltx/0
web02/goatcounter/ltx/1
web02/goatcounter/ltx/2
web02/goatcounter/ltx/3
web02/goatcounter/ltx/9
  ls -lh web02/goatcounter/ltx/1
29.1K Sep  5 01:25 0000000000003f2a-0000000000003f2b.ltx
72.4K Sep  5 02:03 0000000000003f2c-0000000000003f2d.ltx
63.3K Sep  5 02:24 0000000000003f2e-0000000000003f2f.ltx
[ ]
  ls -lh web02/goatcounter/ltx/9
 8.5M Sep  5 02:00 0000000000000001-0000000000003f2b.ltx
 8.5M Sep  6 02:03 0000000000000001-0000000000004008.ltx
 8.6M Sep  7 02:03 0000000000000001-00000000000043a8.ltx
[ ]
We can restore the database from the backup with a few shell commands. First, we stop the containers. Then, we move the damaged database away, invoke litestream restore from the right environment, and restart the containers.9
# systemctl stop container@goatcounter container@litestream
# mv /var/db/goatcounter/db.sqlite ,.old 
# ( . /etc/nixos-containers/litestream.conf ; 
>   set -a ; . /var/keys/litestream.env ; set +a ;
>   $SYSTEM_PATH/sw/bin/litestream \
>     restore -config $SYSTEM_PATH/etc/litestream.yml /var/db/goatcounter/db.sqlite)
# ls -lh /var/db/goatcounter/db.sqlite
-rw-r--r-- 1 root root 20M Sep 20 07:33 /var/db/goatcounter/db.sqlite
# systemctl start container@goatcounter container@litestream

Ten years after removing Google Analytics, JavaScript-based analytics is back on this blog, but without storing cookies or IP addresses, and without involving a third party. I still write for myself first, notably because it lets me dig into a topic and refer back to it years later. But knowing a bit more about my fellow human readers is a nice bonus, even the ones disabling JavaScript.

  1. Nginx scrambles IP addresses before storing them, thanks to the ipscrub nginx module.
  2. GoatCounter already filters some bots based on the user agent or the IP address. But AI scrapers lie about their user agent and hide behind residential proxies.
  3. Without JavaScript, I cannot send the real referrer. I insert NoJS instead.
  4. I am missing the humans reading the RSS feed. I am not comfortable adding a tracking pixel, and bots are likely to fetch it, compromising the statistics. I ll live without counting these readers.
  5. In February 2025, I tested another CSS-only approach without the interaction trick: each page loaded an empty image as a background. Over a month, it recorded up to ten times fewer page views than the nginx logs, even after removing user agents identifying as bots. Compared to this experiment, GoatCounter counts about three times fewer views, 18 months later. As the time ranges differ, this is not an apples-to-apples comparison, but it gives a hint about how efficient bot filtering is.
  6. A NixOS module is a function receiving the configuration of the whole system as config and returning three attributes:
    • imports adds other modules to import,
    • options declares user-configurable settings, each with a type and a default value, and
    • config sets values for options declared by any module, like containers from NixOS.
    If a module does not need to declare options, you can return the config attribute set directly. When a module does not require any argument, you can define it as an attribute set instead.
  7. Most of the time, you don t need an explicit import. NixOS automatically imports the modules shipped with Nixpkgs. For my own modules, some machinery also imports them automatically.
  8. Litestream warns against using the auto-recover option as it can cause data loss. But I don t properly monitor my servers and I prefer uninterrupted backups to a slight chance of losing the last few records.
  9. In my case, the process is slow: around 20 minutes for a 20 MiB database. You can test by restoring to a copy with the -o option, but you still need to stop the Litestream container.

18 September 2026

Petter Reinholdtsen: Long term storage of Mattermost messages with Noark 5 XML

It is said that those who cannot remember the past are condemned to repeat it, a quote often attributed to the American philosopher George Santayana. And to remember the past, records must be maintained and kept accessible to learn from. With this in mind, it is no wonder that archivists worldwide consider it crucial to ensure the archival records are both complete and accurate, with a constant frustration caused by the knowledge that the archives are neither due to challenges in tracking down and collecting what should be archived. The last few days, I decided to simplify the collection of Messages from the Mattermost chat service, used by a few of the organisations I am involved in, to try to improve the situation slightly. To achieve this, I had written a dedicated extraction tool. Partly to see how hard it would be, and partly to ensure that if one of the services were shut down or replaced, everything in it would not be lost. Of course it is in the nature of the use of instant messages that most of them are not fit for permanent storage, at least not according to Norwegian law, where there is a threshold "worthy of the archive" (arkivverdig) that should be met, and messages like "should we go to lunch now" are below the bar and should be filtered out. The tool can not help with this filtering, and that will have to be done using other means after collection. The tools I had my bullshit generator write, under strict supervision and many iterations, will extract every message visible to the user whose credentials are used to log into Mattermost, and write it out in Noark 5 extraction XML format and visualize the result. I created the visualiser mostly to quickly be able to debug the extracted XML, but also to make life easier for anyone interested in testing out the tool set. The extractor mattermost-noark5extract create a hierarchy with arkiv/arkivdel for the Mattermost server, and then individual mappe for each channel and direct message chat, a registrering for each message thread, dokumentbeskrivelse for every message in the tread, and one or more dokumentobjekt for each message and their attachments/images. So far it is only tested on one Mattermost installation, where around 23,000 messages is extracted in 61 seconds and produce 295M with approximately 600 attachments and around 675,000 lines of XML in arkivstruktur.xml. You pass it the URL of the service, a username and password, a directory path where to store the collection and an optional channel name substring to limit the collection to only a subset of the messages available. The mattermost-noark5extract-browser viewer can load this collection and visualize it similarly to Mattermost's web interface, with the list of channels and direct messages dialogues on the left, messages chronologically in the centre and a selected thread displayed on the right. I know Mattermost provide several login options. I've only had the one used on my test server implemented so far, and know the extract tool will have to be extended a bit for it to handle servers using one of these options. If you would like to examine the new toolkit, mattermost-noark5extract is available from gitlab. I wanted to put it on codeberg, but as the recent rule change forbid code mostly written by a bullshit generator there, it was not really an option. Note that some years ago I wrote a similar tool to extract material from the request tracker system. The source for request-tracker-noark5extract is also available from codeberg. I would love to hear from you if you test the tools. As usual, if you use Bitcoin and wish to support my activities, please send donations to 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Louis-Philippe V ronneau: Fake Cash: The World Trade Game

This is a new series1 I'm starting on the tools I have been crafting and buying to play pedagogical games in my Economics classes. These games, aside from being just plain fun for the students, often aim to teach them a lesson via interactive play. Unsurprisingly, it is much easier for them to grok a particular concept this way. Moreover, many students told me these activities were the best part of the classes I teach. The World Trade Game I found out about The World Trade Game in The Handbook of Pluralist Economics Education2, which also contains many other fun pedagogical games. It was first developed by Action Aid, but has since been refined by John Sloman and then by Irene van Staveren, who wrote about her use of the game in the aforementioned book. The goal of this blog entry is not to describe the game and teach you how to play it. If that interests you, either buy the book I mentioned or read about Sloman's version here. No, our focus point here is the tools to play this game: fake cash! Fake cash can be used in many economics pedagogical games and in my opinion, is a worthwhile "investment" of your time and energy. You could indeed play with regular printed sheets of paper listing a value (I sure did!), but using Play Money (a more appropriate term, since we're not trying to defraud anyone) will up your game and create a more immersive atmosphere in your classes. At first, I tried to buy play money. Surely, some commercial offering aiming to create realistic props for movies and TV productions existed? At least in Canada, it turns out the answer was no. The only thing I could find were shady websites and weird expensive listings on AliExpress. Not being tenured yet, I did not try to convince my employer to take a 50-50 chance on supporting criminal endeavours. As such, the next best thing I found was printable play money designed by the Bank of Canada Museum. Their designs were quite nice and met my needs, but their PDF layout made it overly complicated to print large quantities of bills. As such, I played around with GIMP a little and came out with these files. A commercial printer shouldn't have issues printing them on glossy cardstock paper and cutting them up for you: Once printed and cut, you'll unfortunately have to fold each bill by hand. To secure both halves, I ended up using a strip of transparent tape on each long edge. Here is the result, with a real Canadian 20$ bill as a comparison: Stacks of play money bills, with examples of the 5$, 10$, 20$, 50$ and 100$ bills A real Canadian 20$ bill, side-to-side with the play money version

  1. Don't hold your breath though, as I do not currently have a release schedule.
  2. Reardon, J. (2009). The Handbook of Pluralist Economic Education. New York : Routledge.

17 September 2026

Russell Coker: Nheko DBUS

Nheko is my current favourite client for the Matrix IM system, which is my favourite IM system. Matrix is an open system with end to end encryption and Nheko is free software and runs well on Linux desktops and phones. The Nheko client allows interaction with dbus which could be good for automating things, EG you could change the status message when unlocking the screen. I m documenting the most useful ones here because they don t seem to be documented anywhere else. I have filed a Debian bug about the activate room option not working. The qdbus6 program is the QT6 version of the dbus command-line query program, there are a range of other programs which work in much the same way.
# list all interfaces
qdbus6 im.nheko.Nheko / 
# get the version of Nheko
qdbus6 im.nheko.Nheko / im.nheko.Nheko.nhekoVersion
# list rooms in a dump of the data structures (pity it's not json or something)
qdbus6 --literal im.nheko.Nheko / im.nheko.Nheko.rooms less
# join a room
qdbus6 im.nheko.Nheko / im.nheko.Nheko.joinRoom "#flounder-random:luv.asn.au"
# supposed to activate a room but doesn't
qdbus6 im.nheko.Nheko / im.nheko.Nheko.activateRoom "#flounder-random:luv.asn.au"
# set the status
qdbus6 im.nheko.Nheko / im.nheko.Nheko.setStatusMessage "whatever"
# get the status
qdbus6 im.nheko.Nheko / im.nheko.Nheko.statusMessage
Here are a couple of examples of using other dbus clients to get similar results. Note that the difference between the Debian version of Nheko (and maybe other recent versions) and what LLMs return for usage examples is that Debian has / as the path while the examples have /im/nheko/Nheko .
# list rooms via gdbus
gdbus call --session --dest im.nheko.Nheko --object-path / --method im.nheko.Nheko.rooms
# get status via dbus-send
dbus-send --session --print-reply --type=method_call --dest=im.nheko.Nheko / im.nheko.Nheko.statusMessage

15 September 2026

Paul Tagliamonte: DESFire EV3

I ve long been interested in hardware key material storage devices. I ve been a fan of yubikeys (I still remember when my fancy new NEO-N showed up), PIV (and its associated smattering of additional fields), SaaS HSMs, the kernel keyring, some tooling I ve fairly satisfied with the design of at prior companies, and of course, our dear friend, the TPM. All that is not even to mention the scores of exotic hardware security modules one generally comes across from time to time when you re keeping a sharp eye out that you wind up playing with. The concept of storing private key material on a disk, or even having it in RAM has always skeeved me out, so I have a natural inclination to hardware modules, and how shifting keying material around can change your risks and threat model(s) in interesting ways. I don t remember when I first came across the MIFARE DESFire EV3, but a few weeks ago I did a deep-dive into the state of the art of authentication schemes using ID cards. My complete overview of what tradeoffs exist is pretty extensive (and likely not interesting to the vast majority of the world), but the tl;dr wound up being one of use PIV or use MIFARE DESFire EV3 . I wound up picking DESFire for a recent project, and figured it s worth talking a bit about what I learned, share some thoughts, and some code. That code is published on crates.io/desox, and docs, as is our custom, may be found at docs.rs/desox DESFire supports DES (I m sure most readers saw that one coming), 3DES (I didn t bother playing with 3DES at all) or AES-128 (AFAICT always use this?) keying material. It s worth noting that the DESFire only supports symmetric keys and is not designed for public key cryptography, and operates exclusively using shared symmetric key material. The DESFire EV series use those keys and related authentication schemes to interact with files stored on the on-chip EEPROM (2k, 4k, 8k, and 16k versions exist), or applications (groups of files and authentication keys).

Talking to a DESFire EV3 Interactions with the card are done over NFC (ISO/IEC 14443 Type A), and commands to/from the card may be in the usual ISO/IEC 7816-4 APDU format, or unencapsulated bytes sent to/from the card are sent using a fixed instruction set and return code structure saving a few bytes per message. I ve opted to use their undocumented and proprietary format I found it easier to work with and with a maximum message of 60 bytes, the savings matter a lot. While powered via NFC, the card maintains a small amount of state about the connection between the reader and the card in its RAM, including if the session is authenticated or unauthenticated. I ll dig into how authentication happens later, but it s worth knowing that sessions can become authenticated using one of the symmetric keys shared by the card and the reader. The vast majority of the DESFire commands I know about tend to work while either authenticated or unauthenticated, with a few exceptions (GetUid, ChangeKey, and ChangeKeySettings for example). In general, I found working with this card particularly pleasant. There is a fair amount of backwards-compatible behavior and multiple methods of communication that confuse things a bit, but overall, it was better than average to integrate with. Kudos to the NXP team. If the docs on this chip were public, things would be orders of magnitude easier it s not entirely clear to my why they re keeping so much of the interface documentation under NDA, but it s the largest knock against the chip, by far.

Authentication I found a lot of really great resources outlining how the handshake and protocol works for a DESFire EV3, especially from Ridrix, some public datasheets ThrRealRevK and posts from AndroidCrypto. The gist here is that, because the DESFire only does symmetric key operations, the key exchange (a type of SKA Symmetric Key Agreement) uses symmetric keys to establish a unique session key which is used to sign or encrypt data exchanged between the reader and the card. I m not going to get too in-depth here, since there s a ton of other resources out there to dig into but I will do a quick high-level description to keep this post mostly self-contained. The authentication protocol serves two main functions to verify that both parties know the same shared secret, as well as to act as a SKA to construct a new session shared secret key. Here s a quick overview of how a shared session key is derived between the reader and the card using our symmetric keys (AES-128 in the case below).
  1. the reader requests to start authentication with the card (something like AA 00 to start an AES Authentication handshake with keyslot 0x00).
  2. The card will then reply with AF (a status code that indicates more data is to follow), followed by 16 bytes (in the case of AES-128) of encrypted (using CBC) data.
  3. The hosts then decrypts this block with the symmetric key from keyslot 0, returning the card s session nonce.
  4. The host generates 16 bytes (usually random) for its session nonce.
  5. The host sends an instruction of AF (indicating a continuation of the previous command), followed by 32 bytes of encrypted data. When decrypted, the first 16 bytes are our nonce generated in step #4, followed by the 16 bytes provided by the card, decrypted in step #3, except where every byte is shifted to the left by one place (the 0th byte is copied to the end).
  6. The card will reply with 00 indicating a successful operation, followed by 16 bytes, which when decrypted, is our session nonce from step #4, shifted to the left by one byte in the same way that we did in step #5 with the card s nonce.
  7. At this point, both the reader and card have confirmed the other party has the same symmetric secret key. The session is now authenticated and a session key is derived using the two nonce blocks. Two hashing keys (K1 and K2) are derived from this key, which is used to maintain an ongoing CMAC hash of the messages coming and going to/from the card.
From here on out, the session is authenticated , and responses from the card which were previously plain will now contain a 8-byte CMAC signature, which can be used to ensure that the replies in question come from the active session. In my implementation of the handshake I opted to encode the handshake state into rust types, just so I wouldn t make any mistakes. The Handshake type contains the session internals (session nonce values, keying state, to include IV, etc). This means the authentication flow (from within my code) uses the Handshake struct to generate the commands to send to the card in order:
/// Create a new  Handshake , and return the
/// start auth command (something like  AA 00 )
fn Handshake::<Initial>::begin(
 output: &mut [u8],
 key: [u8; 16],
 key_id: u8,
) -> (Self, &[u8]);
After we get a reply back from the card (the encrypted version of the card s session nonce, sometimes called Rnd_B in code I ve seen), we transition states from Initial into HalfOpen.
/// Given the card's encrypted response, generate
/// our session nonce and generate a reply
/// (something that starts with  AF  followed by
/// 32 bytes of encrypted data).
fn Handshake::<Initial>::rnd_b(
 self,
 output: &mut [u8],
 input: &[u8]
) -> (Handshake::<HalfOpen>, &[u8]);
Now that we re HalfOpen , we re waiting to hear back from the card to ensure that it, too, can byte-shift our provided nonce. Once we have the card s reply, we can check it using our complete helper, transitioning from HalfOpen to Successful.
/// Check to ensure that the card replied with
/// our nonce byte-shifted by one place, indicating
/// that they know the symmetric secret in
/// this key slot.
fn Handshake::<HalfOpen>::complete(
 self,
 input: &[u8]
) -> Handshake::<Successful>;
Once the Handshake is successful, the only thing left to do is consume the Handshake struct and turn it into the shared session key by running it through the key derivation function.
/// Consume the  Handshake  struct and return the
/// new shared session secret key.
fn Handshake::<Successful>::into_key(self) -> [u8; 16];
From here on out we can use this session key for the remainder of our interactions with the card signing messages from (and sometimes to!) the card, or encrypted messages to and from the card. This key is used in CBC block mode, where the session IV is updated with the last block of the encrypted data.

Unit Testing A nice proprietary of the SKA scheme we re using as part of DESFire is that the derived session key is actually deterministic if you control your nonce RNG (ok, actually, pretty true for most key agreements, but anyway), which means it is possible to capture traffic over the NFC interface, and replay the NFC I/O with cooked RNGs and ensure byte-identical messages and keys are generated. Within desox-rs this is called replay (I m creative), and I ve got a few replay sessions checked into VCS, which exercise a signficant amount fo the API surface. All were derived from an actual session with a real DESFire card, and can be updated with a live card and a --cfg flag. Each replay file is a set of lines (request-response transactions), each containing two space-delimited hex encoded NFC messages. For instance, here s an authentication handshake in replay format:
1a00 afc7bbd82ff8fefae8
afc6dab54df2278d2952d560821be7e4c3 007d9abe94a9b14748
The code that generated that exchange came from the test stored adjacent to that file a handshake with the default DES key (all zeros), and an RndA value hardcoded to 32c28fdafd3960de.
let mut card = card
 .authenticate_with_rnd_a(
 0x00,
 Key::Des([0; 8]),
 Key::Des(hex_literal::hex!("32 c2 8f da fd 39 60 de")),
 )
 .await
 .unwrap();
Since the card s RndB is similarly unchanging (I m replaying this file every time), this will always derive the same session key, which means messages (including encrypted ones or CMAC signed responses) will be identical, as well. If you re playing with the DESFire yourself, feel free to grab my replay files if you need a known good baseline. By default this will run using the MockBackend, replaying each file expecting a byte-identical request, and responding with the harcoded customary reply. If the code (or test!) needs to change, updating the tests is done by swapping the MockBackend out for a real one. Since I had to do this a bunch during development, running cargo test with RUSTFLAGS="--cfg desox_replay_rw" will, on run, overwrite the replay file(s) for the executed test(s), ensuring all line-protocol changes are explicitly caught and reviewed.

Observations Most commands, even ones which require authentication, are transmitted without CMAC signature(s) or encryption. CMAC signatures from the reader to the card are not really used (except for writes to a file which specifies communication must be CMAC signed), ditto for encryption (although that one is used for key change operations, in addition to file writes on files that specify encrypted communication must be used). The vast majority of commands take a plain request from the reader, and return a CMAC signed response. By my eye, this means that a malicious reader, or something otherwise capable of holding the card online after communication with an authentic reader is complete are able to execute privilaged commands (since one can simply ignore the CMAC signatures on responses), so long as the command doesn t require the reader to provide CMAC signatures (or encryption), or allow the card to power down.

Fun with DESFire I ve played around a bit with ways to use the DESFire cards in interesting configurations, given what they re capable of. Here s some half-baked thoughts I had while mucking around with the cards these are all poorly thought out sketches of some things we can do given the specific tradeoffs I see with the DESFire card. It s also worth noting that I don t have any of the actual documentation, and am not a cryptographic grown-up, so take these sketches with a massive grain of salt. The first thing that came to mind when implementing this is how the authentication scheme can shift the boundary of what is and is not trusted (assuming good secure keying, and provided the key slots and card/application permissions are configured correctly). Rather than push the key material out to the machine connected to the NFC reader ( reader machine ), I instead tried turning the NFC reader and computer into something psuedo-untrusted by merely having it pass messages from the card to a trusted remote system ( remote machine ). This means that the reader machine is exchanging NFC data with the card, but that data is being decrypted, encrypted and processed by the trusted remote machine the reader is unable to derive the session key. For each of these, I wind up needing to authenticate so there s still a few latent risks, but these can mostly be mitigated by asking for a readbacks of any changed file(s), setting key permissions carefully, and requesting the card s UID via the encrypted channel all of which would require the symmetric secrets (which undermine the whole security model if comprimised). This general construction is also subject to a hostile takeover of the untrusted reader machine , since most commands (including destructive ones!) are sent in PLAIN mode the reader machine can wait until authentication is complete and then inject commands into the card and simply ignore the CMAC signatures on responses, severing ties with the remote machine. As such, we also need to take steps to ensure that the key being used is not one that allows any access beyond what is allowed. Here were some ideas I sketched out off the back of this theory.

The second-factor Given some established (and authenticated) connection, part of the initial authentication flow may use the DESFire card to prove physical control over it as part of a handshake. This can serve as a second factor during some authentication flow, requiring physical card presence at a reader to fully initialize a connection. This does have one glaring downside, however it s phishable. To use this for real , we d need to take some steps to prevent obvious MITM flows (XOR the NFC messages with the URI as seen by the client?), but maybe there s something interesting there. This also has a second interesting attribute when used as part of a physical system authentication flow, this becomes a logical place to inject access control, being able to determine if some person is permitted to operate some device at that particular time (Is Joe current on his Laser Cutter certifications?) I think of the ideas I landed on, while conceptually interesting (using an employee id card as a 2FA token, it s very fast), this one is the least likely to turn into something real.

The takeover This time, let s say the computer attached to the NFC reader ( reader machine ) is semi-trusted. For this scheme, our trusted remote machine and the reader machine pass messages over the network to handle authentication to the card (as above), where the handshake data is being decrypted, encrypted and processed by the trusted remote machine as usual. However, once the authentication handshake is complete and a session key has been derived, the remote system return the session key to the reader machine , giving it a one-time-use key and authenticated session to the card. We need to be careful about global/application permissions and key access control to files but in this construction, we can allow the reader machine to take over privileged actions using a scope-limited DESFire key without handing over the card s true keying material (preventing cloning of the card). This can be helpful to ensure messages to/from the card are truely from the card (verifying CMAC signatures), enables the reader machine to directly read/write to/from encrypted file(s), but allows the symmetric key material to remain in as few places as possible which is critical given compromising that secret will undermine the security of the entire system.

Reproducible Builds: Supporter spotlight: Jochen Sprickerhof on ... Reproducible Builds!

The Reproducible Builds project relies on several projects, supporters and sponsors for financial support, but they are also valued as ambassadors who spread the word about our project and the work that we do. This is the ninth installment in a series featuring the projects, companies and individuals who support the Reproducible Builds project. We started this series by featuring the Civil Infrastructure Platform project, and followed this up with a post about the Ford Foundation as well as recent ones about ARDC, the Google Open Source Security Team (GOSST), Bootstrappable Builds, the F-Droid project, David A. Wheeler, Simon Butler and Kees Cook. Today, however, we will be talking with Jochen Sprickerhof, one of the newer members of the Reproducible Builds project core team.

Vagrant Cascadian: Could you tell me a bit about yourself? What sort of things do you work on? Jochen Sprickerhof: I am a freelance programmer working on Open Source. Mainly doing Debian, F-Droid and some smaller software projects. In general I made it a habit to look into every software I use and try to fix bugs or add features I need. In Debian, I maintain about 180 packages with topics covering home banking, build systems and robotics. Most of my time, I currently work on reproduce.debian.net, where we try to bit-for-bit reproduce the packages distributed by Debian.
Vagrant: Could you describe the path that lead you to working on reproducible builds? Jochen: I started my Debian journey as a teenager, converting my school to Debian and serving as its system administrator for 13 years. After studying Applied System Science, I joined the university s robotics labs, where I worked on the Robot Operating System (ROS) and the Point Cloud Library (PCL). In the end, I enjoyed programming more than writing papers, so I eventually left academia for a robotics startup. Some years ago, I realized that the open source work I was doing in my spare time was actually the work I cared most about. Nowadays I am really grateful that I can spend my days working on things I find important and have lots of fun with.
Vagrant: What projects did you recently make big progress on? Jochen: A recent example is metasnap.debian.net. It is a meta archive of snapshot.debian.org which is itself archive of all packages in Debian. But let me explain it the other way round: with reproduce.debian.net, we try to reproduce the packages as they are distributed by the Debian archive. For that, we need the same build environment (compilers, libraries, build tools, etc) that was used by Debian back when the original package was compiled. Luckily, snapshot.debian.org has all those packages, but they are not easily accessible via apt, Debian s package manager. So, metasnap provides a mapping from a package name and version pair to the APT repo on snapshot.debian.org needed to download it from. It was created by josch some time ago, and it s awesome work. But when we tried to reproduce more and more packages on reproduce.debian.net, we found that some were missing packages from the build environment even though they where visible on snapshot.debian.org. We found that metasnap excluded some archive areas because they where not expected to be needed. Reimporting all the data took more than two months and surfaced a couple more flaws. With this fixed, we were able to build more packages, only to find out that metasnap also needs better support for version numbers. Luckily we were able to rewrite the data in a day instead of starting the import again.
Vagrant: You have been working on infrastructure to support reproducible builds for a while. Has recent adoption of reproduce.debian.net into the Debian release tooling changed the focus of your work? Jochen: Quite a bit. When we started reproduce.debian.net in 2024, only around 33% of the packages could be reproduced successfully. Today we are above 98%. Most were not bugs in the packages themselves but in the infrastructure. Similar to the metasnap issue I reference above, packages just needed a rebuild because something else, like the toolchain, was fixed in the meantime. In May, people from the Debian release team and the Reproducible Builds project sat together and decided that the overall state is good enough, and now packages that regress on reproducibility are blocked from entering the next Debian release. But that does not mean all the work is on the shoulders of Debian package maintainers. Since then I have been constantly looking at the migration tooling to spot regressions and provide fixes. Furthermore, a couple of maintainers reached out to us for help and I hope more will do so in future.
Vagrant: What is one small thing you (or others) have not yet gotten to that you would really like to see? Jochen: The central tool to reproduce Debian packages is debrebuild, also written by josch. Currently it has two ways to retrieve the build dependencies of a package. Either it uses metasnap.debian.net (as explained above), or it can access the Debian unstable APT repository directly. This allows to test packages locally before everything is indexed on metasnap by compiling against Debian unstable. But actually there are many other APT repositories to query, like Debian stable or even derivatives. Adding support for an optional list of APT repositories in debrebuild would be great. That would also be a big step to support reproducing other Debian based distributions.
Vagrant: and one big thing? Jochen: It would be great to integrate metasnap.debian.net into snapshot.debian.org. There is some discussion on it already in Debian bug #650783.
Vagrant: What are the tools you use the most? Jochen: According to my fish shell history:
$ history   cut -d' ' -f1   sort   uniq -c   sort -nr   head -10
  36199 git
  20941 vi
  12271 rm
   8599 cd
   7917 ls
   6407 apt
   5631 grep
   4249 mv
   3655 dpkg
   2873 cp

Vagrant: So, is the fish shell reproducible? I remember it did not used to be Jochen: You can check for yourself it was last time I checked. But looking through the other commands, neovim sadly is not. I hope we can fix that in future.
Vagrant: Oh, that s a nice URL to check for reproducible package you can just pass the source package name to check the current results? Jochen: Yes. Another one is udd.debian.org/reproducibility/, where you can list all packages of a Debian maintainer. It also lists source reproducibility and has nice filters as well.
Vagrant: What tools do you use specifically working on reproducible builds? Jochen: I don t have statistics for that, but I would say sbuild to build the package, debrebuild to reproduce it, and diffoscope to analyze the differences. Obviously I also need run apt source <package> or use git-buildpackage to get the sources and all the tools I mentioned above.
Vagrant: So how many packages are left to build reproducibly, and once those are finished, what is next? Jochen: Right now, reproduce.debian.net shows over 98% reproducibility, though there are still over 650 package left and some will probably need a lot of work. But actually I think making packages reproducible is just the first step. For me, this is a project to build confidence in the system. To reproduce a package we have two parts: the source of the package and the build environment. Fixing the packages means gaining confidence in the first part but we still rely on the individual build environments for each package as we need to use the same compiler that was used when the package was build initially. Because of this, we have to keep around every historical version of all toolchain packages. I really would like to remove this extra archive, which means we would have to rebuild all of Debian around release time. I am dreaming of a Debian release where you could bit-for-bit reproduce every package just from the released versions. Due to how Debian works, however, this is not a trivial rebuild and it would need some work on the infrastructure. By the way, initially there was a third component to pay attention to: any connection to the outside world during the build. Luckily we fixed the Debian build daemons to not allow network connections during the build some time ago.
Vagrant: Thanks for all that work, and taking the time to tell us a bit about yourself! Jochen: Thanks a lot for the interview!


For more information about the Reproducible Builds project, please see our website at reproducible-builds.org. If you are interested in ensuring the ongoing security of the software that underpins our civilisation and wish to sponsor the Reproducible Builds project, please reach out to the project by emailing contact@reproducible-builds.org.

Freexian Collaborators: Debian Contributions: Security-tracker git performance, OpenSSH GSS-API split, Incus replacing lxc in Debian CI and more! (by Anupa Ann Joseph)

Debian Contributions: 2026-08 Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

Replacing lxc with Incus for Debian CI, by Antonio Terceiro At the beginning of this month, the default backend for tests was changed to incus-lxc, leaving only a few dozen packages explicitly configured to run as lxc. Those packages got bug reports filed and once those bugs are fixed they will be migrated over to incus-lxc. This marks an important milestone for Debian CI, as the low level of isolation between the worker host OS and the OS under test when using lxc was a significant source of instability in our platform. Common causes for test failures when comparing runs under plain lxc with runs with incus-lxc are documented in the wiki page linked from the bug reports.

security-tracker git performance, by Helmut Grohne Developers working with Debian s security-tracker have reported degrading performance for a while. The options for solving this are few with repository sizes now reaching 30GB for a repository whose working tree is a mere 60MB. While there have been a number of proposals for changing the storage layout, Helmut evaluated options not requiring such changes. Much of the problem hinges on the 50MB data/CVE/list file that is updated in most commits. Efficiency hinges on representing its content in git packs. Practically, git frequently fails to represent its content as a delta and stores a full copy that is typically compressed to 12MB. Add a few thousand 12MB full blobs and you quickly have a large repository. In particular, the copy at salsa.debian.org stores many such full blobs, so pulling from it consumes significant bandwidth. One might think that running git gc helps, but its utility is limited here. When git stores objects in packs, it stores the history in reverse. It starts with the current version and represents older objects as differences (deltas) to more recent objects. A delta effectively is a sequence of copying ranges from a base object and insertion of new content. Given that humans tend to add content over time, those additions are often represented as deletions. The crux here is finding that base object. Given that the data/CVE/list file is large, finding it involves comparing quite a few versions of it with each other. This requires both a significant amount of RAM and CPU time. How many objects git considers for this comparison is controlled with the --window option. It is beneficial if the base object does not come from a direct child commit but skips over commits. In doing so, long chains of deltas are avoided. The --depth option controls the maximum chain length. In this search, git combines all objects into a single window to facilitate file renames. As such, it tends to compare data/CVE/list with data/DSA/list, data/DLA/list and others. This is less than helpful and tends to evict all data/CVE/list versions from the comparison window such that a new full blob of it becomes required. Since the security-tracker repository rarely renames files, we can ask git to instead consider one window per filename via --path-walk. Once doing so, it becomes quite a bit better at finding suitable deltas. The technique is less applicable to older history (2025 and earlier), but given a lot of RAM and a bit of partitioning, git gc can shrink that as well. Combining these techniques, we can shrink the repository into 700MB and keep new growth somewhat under control. Temporarily, Helmut is providing a proof-of-concept mirror at git://git.subdivi.de/~helmut/security-tracker.git using these techniques. Avoid pulling from it directly as it does not provide a secure transport. While it does reduce the amount of data being downloaded, it does not resolve a number of related problems. After downloading, a git client will still expend considerable amounts of CPU on verifying received deltas and attempting to git blame data/CVE/list is not improved in any way. Still, there is work on integrating some of the improvements back into salsa.

OpenSSH GSS-API split, by Colin Watson In an option review Colin did in 2024, shortly after the xz-utils backdoor, he explained that having GSS-API authentication and key exchange support in the main OpenSSH packages is problematic. The key exchange patch is large and intrusive. Even linking to the necessary libraries isn t without risk: as the Ebury malware attack demonstrated way back in 2009, each extra library linked into security-critical daemons such as sshd (or nowadays into its privilege-separated helper programs) can modify the behaviour of the daemon even if you aren t doing anything that would involve calling into that library. Of course some of that risk remains, but as Damien Miller wrote, minimizing the number of libraries that end up in the address space of sshd and friends is still valuable. This split is now complete in testing. As of openssh 1:10.4p1-5, the OpenSSH client and server are built without GSS-API authentication and key exchange support. If you need those features, install openssh-client-gssapi or openssh-server-gssapi instead, as appropriate. Debian 13 (trixie) already has packages with those names that just depend on the regular openssh-client and openssh-server so that you can pre-emptively install them, as documented in the release notes. The new openssh-*-gssapi packages have relatively tight dependencies on openssh-common, in order for the testing migration system to ensure that we can t forget to keep them up to date. This will mean a bit more ongoing work on each new upstream version, but it should be manageable.

Catching up on DebConf video publication, by Stefano Rivera DebConf 25 videos suffered terribly from a bug in voctomix that stopped us from being able to publish the videos immediately after the conference. Ivo De Decker did some work on the videos earlier this year, fixing enough of the audio problems to make the videos at least intelligible. While at MiniDebConf Winterthur Stefano published all the videos from 2025 and 2026 to PeerTube and YouTube. DebConf 25 and DebConf 26 videos as well as all the videos from miniconfs that had got caught up in the backlog were published.

Miscellaneous contributions
  • Carles used po-debconf-manager to review and submit Catalan translations for six packages.
  • Carles prepared a prototype of building webwml and running tidy (linter) in Salsa.
  • Carles created different webwml merge requests with different fixes: invalid links to different architectures, 404s to external sites, fixed templates with invalid URL to favicon.ico, making USE_SAMPLE_FILES=1 to avoid fetching external files, typos in English, etc.
  • Carles updated the LibreOffice Wiki page.
  • Emilio managed several transitions, such as glibc 2.43, perl 5.42, kdepim, nginx, gnome-desktop 51, python3.14-only, and many others.
  • Emilio updated libxfont to 2.0.9, xorg-server to 21.1.24, xwayland to 24.1.13, and fixed a FTBFS bug in terminator.
  • Antonio deployed the website for MiniDebConf Beijing 2026, and archived the Hamburg 2026 and Kanpur 2026 MiniDebConf websites.
  • Antonio released auto-apt-proxy 18, adding new features and fixing bugs; that needed a followup 18.1 release to fix an issue identified in the testing migration tests.
  • Antonio did several Salsa maintenance tasks, including his first platform security update and investigating an issue that was causing the background tasks to be severely delayed, imparting the site usability.
  • Thorsten tried to fix some bugs in hplip and cups. Unfortunately this did not result in an upload this month, but stay tuned!
  • Helmut continued to report undeclared file conflicts and correspond about them.
  • Helmut reviewed and merged Johannes Schauer Marin Rodrigues work on adding hurd support to debvm.
  • Helmut fixed a pile of minor issues in the support libraries underpinning unschroot.
  • Helmut adapted rebootstrap to archive regressions and improved its handling of non-default gcc versions.
  • Helmut fixed the rendering of decompressed logs at crossqa.debian.net.
  • Helmut sent patches for two cross build failures.
  • Colin deployed the fix for Invalid link rel= canonical on bugs.debian.org. In the process he found a few bugs in recent undeployed code and fixed them.
  • Santiago continued helping Aryan Karamtoth with the GSoC project on introducing livepatching support for linux. Several challenges have been identified in the recent couple of months, and Santiago is trying to guide Aryan to address them and design the solution. The mentoring is done in collaboration with Emmanuel Arias.
  • Lucas onboarded a co-maintainer for Valkey.
  • Lucas has started to work on a draft for the Debian Outreach project for onboarding newcomers interested in Debian packaging via a video course.
  • Stefano worked on the after-math of DebConf 26, handling the bursary expense reimbursement process and winding down the conference finances.
  • Stefano uploaded Python 3.15 RC 1 to Debian unstable.
  • Stefano did some budgeting and travel planning for the DebConf 27 site-visit in Asahikawa, Japan. The DebConf committee sends a couple of members to meet with the local team and view the venue together, in advance of the next DebConf.

11 September 2026

Reproducible Builds: Reproducible Builds in August 2026

Welcome to the August 2026 report from the Reproducible Builds project! In our reports, we try to outline the most important things that we have been up to over the past month. As a quick recap about what problem our project intends to solve, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised or not. In this month s report, we cover:

  1. New updated SBOM specification from CISA.gov
  2. LWN on Bootstrappable builds at FOSSY 2026
  3. What s missing to have reproducible builds on PyPI?
  4. Distribution work
  5. Unreproducible builds under EROFS filesystem fixed
  6. Tool and documentation development
  7. Six new scholarly papers
  8. Patches

New updated SBOM specification from CISA.gov CISA, the Cybersecurity and Infrastructure Security Agency of the U.S. government published some the joint guidance entitled Minimum Elements for a Software Bill of Materials (SBOM), which updates and supersedes the baseline 2021 version covered in previous editions of these reports. Whilst the PDF is worth skimming, the interesting changes include that the specification now mandates standard cryptographic hashes: unlike earlier standards that allowed hash omission or manifest-only parsing, hashes must be computed from the output. This is is important for reproducible builds, as it ensures the recording of the metadata required to demonstrate the shipped software matches the build output precisely where applicable. In addition, where the top-level only dependency limitation that was present in the 2021 version has been removed in favour of complete coverage with no minimum depth. That is, SBOMs are expected to reflect all linked libraries, vendored dependencies and other build-time inclusions.

LWN on Bootstrappable builds at FOSSY 2026 In the Toolchains and Other Development Tools track at Software Freedom Conservancy s FOSSY 2026 in British Columbia, Canada, Timothy Sample gave a presentation on bootstrappable builds. This presentation was then covered in a Linux Weekly News article by Jake Edge entitled Bootstrappable builds: how and why, which serves as an excellent introduction to the concept:
The basic idea behind bootstrappable builds is to create a system that can be built without relying on pre-built artifacts. Can we go from zero to the modern day without having to just assume the existence of these already-built-for-us artifacts? The classic recipe for yogurt requires some yogurt to start the process, which is like how we normally build a C compiler today we start with an existing C compiler binary. You might think about making sourdough bread with your grandmother s starter brought over from the old country; we re basically making C compilers with Dennis Ritchie s starter carried over from Bell Labs .
The article, which goes on to cover GNU Mes and other projects that overlap with Reproducible Builds, also has a number of thought-provoking comments.

What s missing to have reproducible builds on PyPI? Core Python developer, Brett Cannon wrote an interesting blog post this month addressing What s missing to have reproducible builds on PyPI, the official public repository for third-party Python software packages:
The reason I like the idea of making reproducible builds work is that I think it can be done in such a way as to not require any work on the part of the producer of a distribution (which is a technical term for sdists or wheels, i.e., the people who upload stuff to PyPI), and thus make reproducible builds very low-friction for people to opt into supporting. [ ]
Brett goes on to outline What s missing from the specs and how reproducibility might be visible on PyPI to consumers:
Assuming all of this comes to pass and we record the where the source code is that went into a distribution and the software used to make the distribution, how do we make it useful to people? Does every person who cares about having a secure supply chain have to rebuild everything they use themselves? Is there some way for even people who don t care about this stuff to benefit? [ ]

Distribution work In Debian this month, 23 reviews of Debian packages were added, 28 were updated and 27 were removed this month adding to our knowledge about identified issues. A number of issue types have been updated as well, such as the addition of a new toolchain issue related to python-traitlets [ ], and the note for an existing issue related to texi2html was updated as well [ ]. Lastly, Bernhard M. Wiedemann posted another openSUSE monthly update for their reproducibility work there.

Unreproducible builds under EROFS filesystem fixed Martin Pitt reported on Fosstodon that they had identified an issue where the mkfs ( make filesystem ) command for the EROFS (Enhanced Read-Only File System) subsystem of the Linux kernel did not have sorted extended file attributes, leading to reproducible builds. Thankfully, Martin also reported that they had fixed this in a commit to the kernel which order[s] each inode s xattrs by name so that images stay reproducible . [ ]

Tool and documentation development diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes, including preparing and uploading versions 327, 328 and 329 to Debian. In particular, he ensured that diffoscope did not require python3-guestfs in the autopkgtests on 32-bit architectures in order to fix Debian bug (#1144372) [ ]. Colin Watson made an additional change, handling a potentially missing openssh-client package when running the autopkgtests [ ], and Jochen Sprickerhof made a similar change to cope with missing cpio and qemu-img functionality [ ] whilst also updating the XML comparator to be considered when comparing SVG images [ ].
Yet again, there were a number of improvements made to our website this month as well. For example:
  • Chris Lamb added draft for a Gothenburg summit-related news article. [ ][ ]
  • Holger Levsen then published the same article. [ ][ ]
  • Lastly, a large number of commits were pushed comprising an interview with Reproducible Builds developer Jochen Sprickerhof to be published within the next week. [ ][ ][ ][ ][ ][ ][ ]

Six new scholarly papers Jens Dietrich, Spencer Sun, Tim W. White and Behnaz Hassanshahi (the result of a collaboration between Victoria University of Wellington and Oracle Australia published a paper this month entitled No Snake Oil: Verifying Python Package Builds. Drawing on the metaphor of snake oil , that is, a fake or ineffective medicine or solution sold with exaggerated claims of curing or fixing everything, the authors write that
Two tools that are designed to automate [PyPI] rebuilds and run them at scale are macaron and oss-rebuild. We study 12,180 popular releases from PyPI and find that the byte-for-byte equivalence rate is generally low. We analyse the reasons why they produce different wheels, and find that equivalence between the original and rebuilt wheels can often still be established, preserving most of the guarantees users expect from rebuildable releases. We present and evaluate daleq4py, a tool to establish the equivalence of Python wheels through the kernel of a normalisation function that is based on provenance-preserving datalog rules. Experimental results show that daleq4py substantially expands the set of rebuilds that can be accepted as equivalent.
The full PDF of their paper can be viewed online, and Jens Dietrich to our mailing list to announce the availability of both the paper and the daleq4py tool itself.
Dimitri Kokkonis, Micha l Marcozzi and Stefano Zacchiroli published an article this month titled Not In My Git Yard: Catching Backdoors at Commit and Release Time on the topic of code-level backdoors that is, stealthy code changes that grant hidden privileges via secret triggers . These issues:
pose a persistent threat to opensource software. Known attempts to inject such backdoors into widely used projects through malicious commits, tampered release packages, or compromised third-party dependencies, were stopped only by luck and manual review. Existing Continuous Integration (CI) pipelines cannot detect these attacks, and downstream binary analysis tools require substantial manual effort. In this work, we present Lily, an automated approach that strengthens open-source development and release processes against backdoor injection. Lily integrates a backdoor detection mechanism into (1) CI pipelines to block malicious commits, and (2) release vetting workflows to prevent tampered releases or compromised dependencies from entering large ecosystems, such as Linux distributions.
The full PDF can be read online.
Ranindya Paramitha and Laurie Williams of North Carolina State University along with Christian K stner of Carnegie Mellon University published a paper this month with the title of The Software Supply Chain as a Market for Lemons: A Multivocal Review of Trust Signal Collapse. (A lemon in American English, is a vehicle that turns out to have several manufacturing defects .) Their abstract is as follows:
Practitioners evaluating open-source dependencies rely on cheap trust signals, e.g., stars, download counts, and contributor activity, as substitutes for direct code inspection, assuming those signals reflect genuine trustworthiness. Prior work has documented individual signal gaming, but the landscape of collapses across all dependency-adoption signals, as well as the ecosystem s response, remains unexplored. The goal of this study is to aid software practitioners in understanding the reliability of dependency adoption trust signals, such as download counts and contributor activity, by conducting a multivocal review of 252 Google Search sources and 870 Reddit threads.
Worryingly, after their review, the authors conclude that cheap trust signals collapse under three simultaneous forces: adversarial manipulation, gaming techniques indistinguishable from legitimate behavior, and non-adversarial AI-driven inflation. The full PDF of the paper is available online.
Julien Malka, Aman Sharma, Martin Monperrus, Stefano Zacchiroli and Th o Zimmermann published a paper this month on Trusting-Trust Attack against an Entire Linux Distribution through Binary Manipulation:
Ken Thompson s trusting-trust attack, in which a compromised compiler backdoors the programs it builds and reproduces the backdoor in subsequent rebuilds of itself, is widely regarded as a threat specific to compilers. We show that it is not. We construct a complete trusting-trust attack around GNU strip, an ordinary build utility that neither inspects nor generates source code, using only manipulations of finished ELF files.
Scarily, in the authors example, a single tampered strip in the binary seed implants a payload that propagates from one generation of strip to the next and survives into the final standard environment after the seed leaves the dependency closure [ ] without failures and backdoors . A full PDF of the paper is available for download online.
Mehdi Keshanimm, Amirhossein Rahmati, Mohammad Hossein Aref and Abbas Heydarnoori published a paper that is currently under review at Emperical Software Engineering titled AROMA+: A Study of Factors Affecting Reproducible Builds in the Maven Ecosystem. (Maven is a/the build automation tool used for Java projects.) In their paper, the authors note that
[ ] reusing external software in a project presents a security risk when the source of the component is unknown or the consistency of a component cannot be verified. The SolarWinds attack serves as a popular example in which the injection of malicious code into a library affected thousands of customers and caused a loss of billions of dollars. [ ] Our research aims to support [reproducibility] efforts in the Maven ecosystem through automation. We investigate the feasibility of automatically finding the source code of a library from its Maven release and recovering information about the original release environment. Our tool, AROMA+, can obtain this critical information from the artifact and the source repository through several heuristics and we use the results for reproduction attempts of packages on Maven Central.
The full PDF of their article can be downloaded online.
Lastly, Oreofe Solarin, Kelechi Kalu, James C. Davis and Paschal Amusuo published a paper this month titled Reproducibility is Not Enough: Artifact Verifiability in Decentralized-Build Package Ecosystems:
[A]rtifact verification requires more than deterministic builds: a verifier must also recover the source state, build environment, dependencies, and build instructions that produced the artifact. Decentralized-build ecosystems make this difficult because artifacts are produced through heterogeneous tools, maintainer-controlled workflows, and fragmented metadata. As a result, it remains unclear how often artifacts in these ecosystems can be independently verified. This paper studies artifact verifiability across four popular decentralized-build package ecosystems. We define an independent verifier model that relies only on registry-derivable metadata and an artifact comparison model with tiered equivalence levels. We implement these models in an Artifact Verification Pipeline and use it to measure artifact verifiability across the target ecosystems.
The authors conclude that beyond build determinism, verifiability is limited by missing source and build metadata, implicit release transformations, and unconventional build practices . A PDF of their paper can be reviewed online.

Patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where applicable or possible. This month, we wrote a large number of such patches, including:

If you are interested in contributing to the project, please visit our Contribute page on our website.

10 September 2026

Matthias Klumpp: JPEG-XL as default in AppStream, and better media processing

Two weeks ago, I released AppStream 1.2.0. This release contains a lot of great changes, but one of the most important ones concerns how media are being handled, and AppStream s default image export format.

AppStream is a Freedesktop metadata standard to describe software components. That can be anything from system services over fonts to console and graphical applications. AppStream metadata is supposed to give users enough information to decide whether they want to install a piece of software, to represent that piece of software, and to give the operating system enough information to decide whether a software component should be installed automatically and (to some extent) what capabilities and relations it has, to provide the user with sensible options.

Especially for the first two goals, and especially for GUI applications, AppStream supports icons and screenshots, which are used to showcase applications. Today, AppStream is used by all kinds of services, from Linux distributions over firmware updates to Flatpak and desktops directly. AppStream s original design however comes from the perspective of Linux distributions in 2011, where you may want to browse the software catalog offline, without delay, and without pinging an external server (which could be a privacy concern).

Therefore, a common way to deploy an AppStream-enabled software repository is to ship all icons of all applications in the repository to the user as part of the repository metadata download. AppStream does support remote icon downloads nowadays, and for a while I thought that this would become the default eventually. However, especially in today s world, having a bandwidth-saving, instantly responsive, privacy-protecting application browsing experience seems more important that ever.

PNG images are great!

The only format that AppStream supports for icons and screenshots (which are downloaded on-demand from your distributor s CDN) has always been exclusively PNG. PNG images are perfect for icons, because they compress well (especially for common icon shapes), are fast and simple to load, and can be loaded anywhere, by any toolkit or webbrowser. They also ensure we deliver faithful screenshot images, even though we may have scaled or re-rendered them. Still though, PNG images are less great for screenshots, as they are not very efficient, which puts strain on any CDN that has to deliver them, as well as on people s internet connections when browsing screenshots. Having smaller thumbnails alleviates that problem a little, but does not fully solve it.

But even for icons, PNG could be improved upon: In many cases, icons are re-downloaded with the repository metadata again and again, so having a large icon tarball adds up to the data transferred during metadata refreshes. AppStream also now supports large 128x128px icons, which nobody in 2012 expected we would need, adding even more data that will be re-downloaded. Saving some space here translates directly to lower bandwidth costs as well as faster downloads for users.

To improve PNG file sizes, the AppStream Compose library, which handles all image processing and metadata catalog composition, was running optipng on all generated PNG images. That does create smaller PNG images, but they were still relatively large compared to other image formats.

For a long time though, there was no alternative to PNG images for icons: There was no lossless image compression format that could give us the same quality as PNG images and that was also widely supported.

JPEG-XL vs PNG in AppStream

Since 2021 we have JPEG-XL (JXL), which offers a true lossless mode with often better compression than PNG. The issue was that JPEG-XL wasn t widely supported. Then, in 2025, the PDF Association selected JPEG-XL as the preferred image format for HDR images in PDFs, and now we are finally getting browser support and more ubiquitous availability of the format (you can try it right now in Firefox!).

For screenshots, using JXL s lossy mode, it has obvious and extreme size advantages over PNG, so supporting JXL or WebP for screenshot images was an obvious choice. If JXL would support the lossless case very well as well though, we could serve many use cases with the same exported image format, which is very attractive to me.

So, the obvious next question was whether it was worth the pain of switching the icon format, so I did some measurements on real icons. For that I used the AppStream component icon pool that Debian Unstable ships, which is almost 5000 application icons of various sizes, and converted them to PNG:

Icon sizeIconsPNG totalJXL totalPool savedPNG avgJXL avgMedian savedMean saved Worst BestLarger as JXL
48 48 1544 3.7 MiB 3.0 MiB 17.8%2.4 KiB2.0 KiB 17.9% 16.7%-118.7%60.0% 206
64 64 2018 7.0 MiB 5.8 MiB 17.8%3.6 KiB2.9 KiB 18.0% 15.8%-112.7%70.0% 279
128 128 1411 11.2 MiB 8.7 MiB 22.0%8.1 KiB6.3 KiB 20.1% 17.5% -89.7%61.0% 209
TOTAL 4973 21.9 MiB 17.5 MiB 19.9%4.5 KiB3.6 KiB 18.6% 16.6%-118.7%70.0% 694
PNG images saved with libpng at effort=4, compression=9, then optimized using optipng -o2, JXL images encoded using vips jxlsave lossless=1 effort=7 strip=1 via VIPS/libjxl.

As the table shows, using lossless JXL images over size-optimized PNG images (using optipng s default settings) provides a roughly 20% gain. This does not look like much, until you consider how often these files are downloaded: A 20% file size reduction may only save 1-2 MiB of disk space, but if they are downloaded over and over again by many clients, it will save a lot of bandwidth.

Interesting JXL encoding findings

As a sidequest, I was curious why some images were larger than their PNG counterparts when encoded with JXL, and what the ones that were significantly smaller were.

In short, the biggest size reductions for JXL existed on images that were already small as PNG, and contained large, flat color surfaces with hard edges and simple shapes. They were not very interesting, and much of JXL s wins come from accumulating smaller gains across all files, which compound the bigger icons get (especially at 128x128px, where JXL truly shines).

The events were JXL loses to PNG are more interesting: For example, it does quite poorly with pixel-art images that have a lot of repeating patterns. Those are encoded well by PNG, but less efficiently by JXL. Take for example Vonsh:

Icon of Vonsh, an SDL-based snake game, which PNG compresses better than JXL

My guess is that while PNG can exploit the repeating pixel patterns for compression, JXL s predicts surrounding pixels from its neighbours, which fails too often and makes it pay almost full entropy per pixel. In this single rare case, the PNG is at 5.4 KiB, while the JXL is almost 8 KiB in size.

Other cases I looked at were arguably buggy input data, where color channels were hidden under the alpha channel of the input image. PNG could probably again exploit repeats, while we were forcing JXL to encode pixels that were invisible in the final image. This is arguably a problem with the original input data. Currently, AppStream does not make any changes to icons at all, but in future we might add a filter that removes invisible colors from images to solve this pathological case (it was only two icons out of 5000 though, so it is not a high priority).

The third case I found where JXL loses to PNG were icons with checkerboard-like patterns:

Icon of x3270, an IBM 3270 Terminal Emulator

For those, PNG can likely again exploit the repeating patterns, while a checkerboard layout is pretty bad for left/top predictors like JXL s. However, in this case the size difference (and loss for JXL) is only 450 bytes, so even though JXL loses to PNG, it does so not by much.

JXL in AppStream

Given these findings, JPEG-XL is the default image format starting with AppStream 1.2.0. AppStream Compose will encode all images losslessly as JXL, while screenshots are encoded in lossy mode at Q=90 effort=7. Since the optipng step does not happen for JXL images, this comes at no speed penalty and is even a bit faster on modern x86_64 CPUs (where libjxl can use SIMD). PNG is still available, and Compose can be told to switch between the two formats.

Upsides of JXL in AppStream right now

If you use JXL in Compose or the recent release of appstream-generator, you will get much smaller images and, for screenshots, will benefit from other JPEG-XL features such as progressive decoding, providing a far nicer user experience. libAppStream has supported JXL icons since version 1.1.3, so your clients will need that version or a newer one, and all software centers will have to support loading JXL images (which all of them do, provided the right plugins are installed).

Downsides of switching to JXL too quickly

JXL is a very new format, so web browsers might not yet display it if you are serving webpages. Your clients may also have bugs in processing JXL images, as the format is still new . For example, switching on JXL in Debian sent KDE Discover into an infinite loop on startup while trying to load the icons (an issue which has been fixed, but clients will need that patch first before JXL is switched on).

This currently makes JXL enablement only possible when you know that your clients can support it. This is the case for me in Debian Unstable and Debian 14, which are using JXL images for a few weeks now, but not for any older releases. Platforms like Flatpak have it even harder, because they do know even less about their clients. So, even though it has big advantages, you may want to hold off on using JXL right away, and force PNG by setting the ImageFormat key to png in appstream-generator s configuration, or passing --image-format=png to appstreamcli compose.

It is also worth mentioning that JPEG-XL is much, much slower on systems that do not have SIMD instructions or for which the libjxl/jxl-rs library does not have them (such as apparently riscv64 right now). If this is a concern, you might not want to switch to JXL right away.

Media pipeline improvements

Besides the JXL default change, AppStream 1.2.0 also comes with a complete overhaul of its media processing pipeline. While libappstream, AppStream s main library, does not do any media processing and comes with very minimal dependencies to be embedded in client applications and used on servers, the same can not be said about libappstream-compose, AppStream s library to build metadata generating applications (the server-side part, usually).

The compose library has to render fonts into font specimen cards, inspect translation files, render SVG images, decode all kinds of raster images, inspect video files, etc. Especially the fonts, and the fact that fonts can appear in SVG images, has caused issues in the past, as libappstream-compose is a heavily threaded library and most font libraries can only work from a single thread. This forced the library to essentially go into single-thread mode anytime anything that could touch a font was being processed.

AppStream also originally was created for a safe world where applications were vetted by the distributors before their metadata was processed. This is increasingly not the case, so it made sense to put at least a few guardrails on the most complex part of the pipeline: The media processing. As part of the change, media processing was split out into a separate worker process. This solved two problems at once: Font handling was isolated in a single-threaded binary if we wanted to handle fonts in parallel, we could simply spawn more workers. And, being in a separate process, the media processing could now be sandboxed.

As part of the multiprocess changes, Compose also switched from using GdkPixbuf to VIPS for image processing. The latter allows for much more fine-grained control over the image output and encoding, and comes with a lot of well-maintained filters and operations, which made it possible to eliminate a fair chunk of AppStream s hand-rolled image processing operations. As part of this transition, we unfortunately lost the ability to read XPM images, which dropped about 20-30 applications from the pool at Debian. But in the name of security, this is a sensible choice, especially since most XPM icons were very small and low-resolution, and applications using them could benefit from adding a high-quality PNG icon anyway. With VIPS, we also now restrict the amount of image formats we can load to a sensible set, so extremely niche or unexpected formats will be outright rejected (this includes sane-but-unusual formats for screenshots and icons, such as TIFF images).

The Compose library, with all of these changes, will now just request high-level operations (e.g. render a font card for this font to a JXL image ) from the worker, and provide it with input data in sealed memfds and output locations as FDs as well. On Linux systems, the worker will use Landlock if available, to block all write access to the filesystem, deny device access and deny TCP and UDP as well. The sandbox can certainly be tightened a fair bit in future, but this was a good and safe start to gain some experience with it without having things break too easily, given the many places Compose is used in (also, Landlock s API is surprisingly nice to use, so it was easier than I thought to add in this early version).

With all of these changes, the libappstream-compose library is now also officially marked API-stable, so you should be able to rely on it in future to build new things (its API has barely changed in the past, and now with the new media API and defaults change in place, it was time to declare it stable).

I want to see / try this!

Currently, the easiest way to have a look at the new data is to check out Debian Unstable. If you have a JXL-enabled browser, you can also see the icons in AppStream Generator s HTML pages for Debian Sid. If you are using appstream-generator for your distribution, you will also get much more pleasant statistics and HTML pages, as well as fully deterministic media output and a whole bunch of security updates, so, update to its recent 1.0 release.

Please keep in mind that if you switch to JXL, the client tools receiving the image data have to support it. Support varies depending on the Linux distribution, so, test it first and switch the default back to PNG in case you encounter any issues.

What s next?

With so many features and changes landed, the next changes in AppStream will focus on improving what already exists and fixing any issues (there will be more blogposts about the other features 1.2.x delivers!). Testing with the entire Debian archive as data source makes me fairly confident though that there will not be many problems. In the longer term, tightening the media processing sandbox will also be something we might want to do, e.g. by hiding parts of the filesystem tree or filtering syscalls.

For JPEG-XL, one obvious question is Will you add support for it to the Freedesktop icon-theme specification as supported format alongside PNG, SVG(Z), and XPM? . For on-disk icon repositories, JXL s space-savings are less compelling, and it being HDR-capable is also not necessarily a killer feature (PNG can go a long way!). However, JPEG-XL s ability to immediately decode larger images at reduced resolution without resampling could legitimately be very powerful here, as applications could ship a single large image and quickly decode it at 1/2, 1/4 or 1/8 the size for different purposes in their UI. JPEG-XL also supports spot-color extra channels, which applications could use as masks to recolor raster icons at render time. This could be incredibly nice to color symbolic icons on-the-fly without any SVG and CSS. JXL also provides richer metadata, which might be neat for (license/author) documentation. So, the answer here is: Maybe it makes sense to allow another format, but this will have to be discussed first, as it would force JXL into every toolkit and desktop, which is a much bigger ask than supporting it only in AppStream.

As always, let me know what you think and please report any issues or bugs directly against AppStream or AppStream Generator if you encounter problems that are with the tools, and not with a project s metadata.

9 September 2026

Matthew Garrett: SystemIO conflicts are not firmware bugs

I m looking at something entirely unrelated, but tripped over some search results that made me realise that a lot of people still think getting errors like ACPI Warning: SystemIO range 0x0000000000001828-0x000000000000182F conflicts with OpRegion 0x0000000000001800-0x000000000000187F indicate a firmware bug. This is generally untrue. We need to dive a little into what ACPI is to clarify why. The Advanced Configuration and Power Interface1 specification defines a whole bunch of stuff, but what s interesting to us here is the hardware abstraction it performs. While PCs are nominally a well-defined platform that s really not true at the hardware level once you get beyond a certain level of complexity. When you suspend a system you want to power down the hardware in the correct order, for instance, and knowing what that order is requires you to know details about the specific motherboard design. The approach taken in the embedded world is to just bake that knowledge into the OS in some form, which is how we end up with Devicetree. ACPI takes an alternative approach - rather than provide that information as data that has to be consumed by OS drivers, it distributes it as code. The ACPI Source Language, or ASL, is a simple language that gets compiled into a bytecode that s then interpreted by the OS at runtime. One of the features of this language is the ability to define Operation Regions , effectively structure definitions that describe access to underlying hardware. Let s imagine a simple device with two exposed registers. The first is an index register - it describes which internal register we want to access. The second is a data register, where reading it gives us the value of the internal register whose address is currently in the index register, and writing to it modifies that register. An example operation region declaration would look something like
1
2
3
4
5
6
OperationRegion(OPR1, SystemIO, 0x400, 0x2)
Field(OPR1, ByteAcc, NoLock, Preserve)
 
  INDX, 8
  DATA, 8
 
This defines an operation region called OPR1 at IO port 0x400, 2 bytes long. Inside it are two 8-bit fields, INDX and DATA. These are to be accessed one at a time, do not need the ACPI interpreter to take a global lock when accessing them, and if a subset of the register is modified then the other values should be preserved (irrelevant in this case since the fields are only a byte wide). Now any references to INDX or DATA in this scope will trigger accesses to those registers. So, a method to read the value of register 0x03 would look something like:
1
2
3
4
Method (RD03)  
  INDX = 0x3
  Return (DATA)
 
ie, set INDX to 3, and then read the value of DATA and return it. But! What if another ACPI method is running at the same time? Let s say we have one that writes to register 0x05:
1
2
3
4
Method (WR05, 1)  
  INDX = 0x05
  DATA = Arg1
 
What happens if RD03 executes while we re part-way through WR05? INDX might get reset to 0x03, and now WR05 will modify register 0x03 instead of 0x05. Oh no! But we can avoid this - we declare a mutex (Mutex (MUTX, 0x00)), and update our methods to be something like:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
Method (RD03)  
  Acquire (MUTX, 0xFFFF)
  INDX = 0x3
  Local0 = DATA
  Release (MUTX)
  Return (Local0)
 

Method (WR05, 1)  
  Acquire (MUTX, 0xFFFF)
  INDX = 0x05
  DATA = Arg1
  Release (MUTX)
 
Each method takes a lock (waiting up to 0xffff milliseconds and then erroring out if it doesn t), and performs the access. There s now no chance of a race. Phew! Now suppose someone writes a Linux driver for this piece of hardware. It accesses the hardware directly, with no knowledge of ACPI. What stops the driver from racing against one of the ACPI access methods? Nothing at all. Oh no! Again! This isn t hypothetical, by the way - here s a relatively harmless example, but back in the day we did trip over cases where temperature monitoring chips would be accessed by the firmware and Linux simultaneously and as a result you might end up thinking you re reading a temperature when you re actually reading a status flag, resulting in an impossibly high temperature and an immediate thermal shutdown. In this case, the kernel saves you from this (potentially hardware damaging) outcome by printing a message like ACPI Warning: SystemIO range 0x0000000000000400-0x000000000000401 conflicts with OpRegion 0x0000000000000400-0x0000000000000401 (OPR1), telling you that the kernel has detected that a driver is attempting to allocate IO ports 0x400-0x401, but that there s an ACPI operation region called OPR1 that is claiming the same addresses. The kernel isn t in a position to know what type of access the firmware might perform in that region, so assumes that it might be dangerous and blocks the driver from loading. But all is not lost! The kernel also prints some helpful advice, ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver. And ACPI tables will often actually have a definition that looks like this:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Device (HDW1)
 
  Name (_HID, "VEND0001")
  OperationRegion(OPR1, SystemIO, 0x400, 0x2)
  Field(OPR1, ByteAcc, NoLock, Preserve)
   
    INDX, 8
    DATA, 8
   
  Mutex (MUTX, 0)
  Method (RD03)  
    Acquire (MUTX, 0xFFFF)
    INDX = 0x3
    Local0 = DATA
    Release (MUTX)
    Return (Local0)
   

  Method (WR05, 1)  
    Acquire (MUTX, 0xFFFF)
    INDX = 0x05
    DATA = Arg1
    Release (MUTX)
   
 
which defines an ACPI device and associated methods. The _HID field defines the device type, and a Linux driver can be written that will be automatically loaded if a device with type VEND0001 is seen. That driver can then call ACPI methods associated with the device and access the resources in a way that matches the firmware s expectations. (Interested in writing such a driver? I wrote a guide back in 2009) The firmware did absolutely nothing wrong here2, but trying to load the native driver will generate an error and the internet will tell you that PC firmware developers are incompetent3 and you should pass a kernel argument that overrides this behaviour and it never did them any harm, and it probably won t do you any harm either but it might and you might never know why your system occasionally wedges or catches fire.

  1. The ACPI spec used to live at acpi.info, but sadly that seems to have vanished some time after UEFI took over stewardship of the spec
  2. You might argue that the firmware should simply not do anything at runtime because it is not the firmware s job to do that, and I do understand that and you can certainly boot with acpi=off if you want to and no ACPI code will be executed at runtime. Let me know how that goes.
  3. I m not going to present an opinion on that here, merely say that this provides no supporting evidence for that assertion

Dirk Eddelbuettel: RcppXts 0.0.7 on CRAN: Minor Maintenance

A new maintenance release 0.0.7 of RcppXts is now on CRAN, and has been built for r2u. The RcppXts package demonstrates how to access the export C API of xts which we contributed a looong time ago. There are by now a more example packages around this C level access to another package, but this one was an early example. This release is strictly maintenance, updating continuous integration, the README.md file and other packaging conventions adopted since the last release four years ago. The NEWS entries follow.

Changes in version 0.0.7 (2026-09-09)
  • Corrected a docstring for the module
  • Updated continuous integration setup several times
  • Simplified setup by removing no-longer-needed Makevars
  • Added badges to README.md

Courtesy of my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

8 September 2026

Paul Tagliamonte: can you hear me now? good! (Part 11/12)

This post is part of a series called "Pigeon". If this is the first post you've found, it'd be worth reading the intro post first and then looking over all posts in the series.

If you're looking for it, the intro to the Link (layer 2) that this belongs at a high level and listing of the parts that make it up (including this one) is on the link post.
Built-in to the pigeon link protocol is a message type called cal (short for, you guessed it, calibration). A pigeon frame with a type of cal (which is 0x02) carries a JSON encoded payload in the body, which can either be a beacon, requesting signal reports in response, or a report, describing the received beacons. This serves a few interesting purposes firstly, network operators can better understand the coverage footprint, propagation under different conditions, and how gain impacts reception when tuning for the lowest practical power levels. Secondly, this can be used (and I plan to eventually implement!) to construct a mapping of minimum power level and peer mac address to dynamically control the transmission power based on the destination station. That being said, for now, all I ve used this for is getting a rough sense for what gain value(s) make sense between two nodes (manually). In the future, beyond all the fancy neighbor gain stuff, I plan to wire this into the daemon to happen automatically, debouncing for beacon and report messages, such that transmitting stations only beacon, and receiving stations only report a max of once over some time period for a given peer.

Version Given all the above, I do intend to make some massive changes to this protocol (I promise to blog all about it) when I get around to hacking on switching Layer 2 frames within a network segment. Just to avoid having to dig myself out of a hole later, I m going to explicitly send (and check) the version field to avoid having a big flag day switchover or needing to use a new link type.
Version ID Description
V1this version of the cal protocol

Location Both flavors of cal messages (beacon and report) may contain a location, which is the location that the beacon was transmitted, or for or the location where the beacon was heard for a response. This can be used to derive a coverage map and to (operationally) better understand what stations should be within range, and generally what gain level(s) are effective. All fields assume WGS84 latitude and longitude values, and elevation is distance, in meters, above the WGS84 ellipsoid NOT height above sea level, or altitude above the ground.
Field Description
latWGS84 Latitude
lonWGS84 Longitude
elevationheight, in meters above the WGS84 ellipsoid

Sequence Each beacon contains a Sequence identifier, which is used to communicate which message number is being heard, and how many total were transmitted by the originating station. The current approach with Beacon messages is to transmit some number of Beacon messages at different gain levels, each with a unique Sequence identifier.
Field Description
numberbeacon sequence number
totaltotal number of beacons transmitted

Gains Recorded gain setting(s). For a Beacon this indicates the gain settings (which, in spite of its name, includes things like amplifiers, or attenuators). Changing this over different Beacon frames enables a better understanding of what an appropriate gain level is for the transmitting station over time.
Field Description
namegain stage name
dbgain value, in dBm

Cal All messages contained in a cal frame are of this type. The type field communicates if this is a Beacon or Report message type.
Type Description
beaconsent intermittently by idle stations
reportreception report in response to a beacon

Beacon A beacon message may be sent periodically by pigeon nodes capable of transmitting to announce their prescience to peers and, implicitly, to receive signal reports from nearby listeners who are capable and configured to transmit reports. The beacon JSON message is made up of the following fields:
Field Description
versionversion enum value
gainsgains object
sequencesequence number
locationlocation object
An example beacon looks, unsupprisingly, as follows:
 
 "type": "beacon",
 "version": "V1",
 "sequence":  
 "number": 2,
 "total": 5
  ,
 "gains": []
 

Report A report message may be sent in response to a beacon message by pigeon nodes capable of receiving and transmitting to assist with setting the lowest usable gain value, and to better understand the area of coverage and propagation. The report JSON message is made up of the following fields:
Field Description
versionversion object
locationlocation object
An example report looks as follows:
 
 "type": "report",
 
With all that out of the way lets send some ip

7 September 2026

Vincent Bernat: Sidenotes with CSS anchor positioning

I am a heavy user of sidenotes:1 they keep optional content next to the text instead of sending the reader to the bottom of the page and back. Tufte CSS renders them without JavaScript but only accepts inline content. CSS anchor positioning, now supported by recent browsers,2 is an elegant alternative. Sidenotes can hold several blocks, still without JavaScript, and fall back below the paragraph referencing them on narrow viewports and older browsers. In 2023, Eric Meyer demonstrated this technique in Nuclear Anchored Sidenotes. The main improvement over other solutions is that the notes can sit anywhere in the HTML document. You can place them after the paragraph referencing them, as regular block elements for text browsers, screen readers, feed readers, and reader mode to render them properly:
Sidenotes rendered in Lynx appear after the paragraph they are called from.
Rendering in Lynx, a text browser
When the viewport is too narrow or the browser does not support CSS anchor positioning, you can style them so the reader can skip them or glance at them without losing their position in the text:
Sidenotes rendered on a narrow viewport appear with a distinctive typography after the paragraph they are called from.
Rendering below the paragraph on a narrow viewport
Once the viewport is large enough, they appear in the margin, at the same vertical position as the matching reference mark, unless they would collide with a previous sidenote, as in the example below:3
Sidenotes rendered on a large viewport appear in the margin. There are two of them. The first one is vertically aligned with the matching reference mark, while the second is rendered below as it would collide with the first otherwise.
Rendering in the margin on a large viewport
The gist of CSS anchoring is to position an element relative to another element the anchor. For the sidenotes, the anchor is the reference mark. I use the following markup, with a data attribute to specify the anchor name:
<sup id="fnref:YYY" data-anchor="--lf-sn-YYY">
  <a href="#sidenote-YYY">1</a>
</sup>
The matching note is an <aside> element carrying the same data attribute for the anchor name. We put it after the paragraph holding the reference mark:
<aside role="note" id="sidenote-YYY" data-anchor="--lf-sn-YYY">
  <sup>1</sup>
  <p>A first paragraph.</p>
  <p>A second paragraph.</p>
</aside>
On a narrow viewport or when the browser is too old for CSS anchoring, we style the sidenote, which stays below its paragraph, with a muted color:
aside[role="note"]  
  margin-block: 1rlh;
  color: #444;
 
On a wide viewport and when the browser is recent enough, we move the sidenote to the right margin:
@supports (anchor-name: attr(data-anchor type(<custom-ident>)))  
  @media (min-width: 72rem)  
    main  
      position: relative;
      sup[data-anchor]  
        anchor-name: attr(data-anchor type(<custom-ident>));
        /*   anchor-name: --lf-sn-YYY */
       
      aside[role="note"][data-anchor]  
        anchor-name: --lf-sidenote;
        position: absolute;
        position-anchor: attr(data-anchor type(<custom-ident>));
        /*   position-anchor: --lf-sn-YYY */
        top: max(anchor(top), anchor(--lf-sidenote bottom, -1rlh) + 1rlh);
        left: 100%;
        margin: 0 2rem;
        width: 18rem;
        color: inherit;
       
     
   
 
attr() extracts the anchor name for the reference mark from the data-anchor attribute. It returns a string, unless we specify a CSS unit or a type, like here: the browser parses the data attribute as a custom identifier, which anchor-name validates as a dashed identifier, a custom identifier starting with two dashes.4 The note itself is absolutely positioned past the right edge of the main block. It selects the matching reference mark as its anchor with position-anchor set to the value of the data-anchor attribute. Each note is also an anchor named --lf-sidenote. We use it to keep the next note from colliding with this one. The anchor() CSS function lets us position the note s top edge relative to its anchor: anchor(top) aligns the top edge of the note with the top edge of the reference mark. It can also take another anchor as a parameter: anchor(--lf-sidenote bottom) would align the top edge of the note with the bottom edge of the closest preceding anchor named --lf-sidenote so the previous note.5 Like attr(), anchor() accepts a fallback value as its second parameter and use it when the named anchor does not exist. The top property handles three cases, illustrated in the following diagram:
Diagram of three sidenotes anchored to their reference marks. The first one is aligned with the top of its own reference mark, as no note comes before it. The second one would overlap the first, so it takes the bottom of the first note as anchor and sits one line below it. The third one comes far enough down the page to align with its own reference mark again.
The three cases for the vertical position of a note
  1. The first note s top edge aligns with the top edge of its reference mark: as there is no previous note, anchor(--lf-sidenote bottom, -1rlh) + 1rlh resolves to 0 and max() returns anchor(top).
  2. When the reference mark of a later note sits above the bottom of the previous note, plus some vertical space, the note goes below the previous one to avoid a collision. max() returns anchor(--lf-sidenote bottom) + 1rlh.
  3. Otherwise, the note s top edge aligns with the reference mark s top edge, as max() returns anchor(top).

Have a look at the complete stylesheet, which also adapts the reference mark to the location of the note: a arrow when the note sits below the paragraph, a arrow when it moves to the margin. Gwern s Sidenotes In Web Design lists more implementations and their trade-offs. Some bloggers aim to write a post in 30 minutes. I planned to publish three web-related articles this weekend. Instead, I spent an inordinate amount of time elsewhere: about 15 commits on the build system, a pull request to update CSS highlighting for nested selectors in Pygments, and a small correction to MDN s article on the anchor() CSS function. The SVG illustration took a bit less than an hour and the article itself a handful of hours. The attr() function came in after I thought inline style looks ugly, isn t there a better way? But, hey, I still think this is worth it!

  1. My PhD advisor told me this is unwise.
  2. The first bits of anchor positioning are supported from Chrome 125 (May 2024), Firefox 147 (January 2026), and Safari 26 (September 2025). Before Safari 26.5, sidenotes may collide due to a bug in how dependency chains are handled. You can detect this situation with some JavaScript. It is, however, not needed in the solution described here as we depend on a more recent feature.
  3. If you noticed the runt in the first note, I share your pain and lament that Firefox does not implement text-wrap: pretty.
  4. Typed attr() is supported from Chrome 133 (February 2025), Firefox 155 (September 2026), and Safari 27 (September 2026). Check Una Kravets article for details. To support more browsers, you can inline the anchor name and the position anchor directly in the HTML:
    <sup id=" " style="anchor-name: --lf-sn- ">
      <a href="#sidenote- ">1</a>
    </sup>
    
    Managing Anchor Associations With Data Attributes and Advanced attr(), by Daniel Schwarz, explores CSS anchors and typed attr() in more detail.
  5. The exact rule for the target anchor element is more complex: if an ancestor of [the note] satisfies the following conditions, return the nearest such element to [the note]. Otherwise, return the last element in tree order that satisfies the conditions. One of these conditions is that [the candidate] is an acceptable anchor element for [the note], which requires that [the candidate] is laid out strictly before [the note], where the relevant clause is that [the candidate] is either not absolutely positioned or occurs earlier in the flat tree order than [the note].

Next.

Previous.