Search Results: "lus"

4 May 2026

Russell Coker: Tower Servers and Resizable BAR

A feature on modern PCIe implementations is Resizable BAR AKA REBAR . This basically means that instead of allocating 256MB of address space for a PCIe device to have it s memory mapped the device can ask for more, the limit can be 4G with some hardware or the combination of motherboard and expansion card can support 64bit addressing to allow the entire memory space of a GPU to be mapped in one region. Directly mapping all the memory will be faster no matter how things work, but a combination of algorithms optimised for a flat memory layout and overheads from remapping can cause 90% of performance to be lost without REBAR support. Some GPUs (or maybe the software driving them) will even refuse to work without it. I believe that almost all hardware supporting DDR4 will support REBAR at a hardware level, but in many cases the BIOS doesn t support it. There are people who have reflashed a system BIOS to add REBAR support and there are options to use a modified UEFI boot loader to replace the code that is used for mapping the GPU memory. The systems I like to use are server grade tower systems with registered ECC RAM, after a few years they become quite cheap and still give decent performance while supporting large amounts of RAM. But many such systems that could support REBAR don t, presumably because the vendor doesn t have a great interest in supporting new uses of old hardware. Comparing the Name Brand Servers The HP Z640 and Z840 systems I m running date from 2014 and give good performance with replacement CPUs that are cheap on ebay, but they don t support REBAR without a flashed BIOS. The next release of those HP servers are the HP Z6 and Z8 Gen 4 systems from 2017 that have BIOS support for enabling REBAR. The Lenovo Thinkstation Px20 (P520, P920, etc) don t support REBAR which is especially disappointing as they were on sale from 2017 to 2022 and have decently fast CPUs. The replacement for the Px20 systems are the ones that are still on sale now and they seem likely to have REBAR support but won t be affordable on ebay. The Dell PowerEdge T440 and R740 systems (and presumably all their servers from 2017) don t support REBAR. There are no google hits for T550 and R750 systems from 2021, so presumably no complaints means that Dell servers from that era support it. But the T350 servers are junk and only take slow CPUs, and the T550 systems are brutally expensive. The Precision 5520 systems don t support it and newer Precision workstations will get expensive. It seems that HP is best for this. Which HP Workstation The Z2 G4 only supports 64G of RAM so isn t worth considering. The Z4 G4 is low end and comes in two variants. The one with i5/i7/i9 CPUs doesn t support ECC RAM so isn t suitable for me, and that probably means most Z4 G4 systems on the market. The upside is that apparently 2*6pin PCIe power cables is standard so any size GPU should work and there are 8 DIMM slots supporting up to 512G of RAM. There are 3 options for PSU, 490w for 0 GPUs, 750W for 2 (small) GPUs, and 1000W for up to 4 GPUs. The Z6 G4 has an option for a second CPU that almost no-one selects, that reduces the space for RAM so there s only 6 DIMM slots. But as there is no option for a Z6 without ECC RAM every one on offer will be good. The Z8 G4 is a nice dual socket system that I would not use for a serious GPU after my experience of my Z840 having a motherboard problem from a big GPU. The Z4 G4 is going for about $500 on ebay with the 750W PSU, that is more than I want to pay but not a lot more. In 6 months they could be going for $350 or so. There are hardly any Z6 G4 systems on offer and they are all well over $1000 so I m not considering them. Conclusion I need to poll the second hand sites for Z4 G4 systems and find one going cheap. One of those could be a good ML test machine for a while and then become a workstation once the faster CPUs (which are currently around $900) become cheap.

Russell Coker: Copy Fail on Debian and SE Linux

I have just learned of the Copy Fail kernel vulnerability [1] thanks to alexanderkjall@mastodon.social (who I have just followed on Mastodon and I recommend that you follow too). The question for me (after installing the patched kernel the systems of mine that are most exposed) is whether SE Linux would have stopped that. Basic Policy Analysis For the SE Linux policy analysis the alg_socket class is the one that is related to the exploit. So the following policy analysis command (run as non-root with policy copied to /tmp from a running system) shows what domains are allowed access on my current Debian development system:
$ sesearch -A -c alg_socket /tmp/policy.35 
allow NetworkManager_t NetworkManager_t:alg_socket   accept bind create read setopt write  ;
allow bluetooth_t bluetooth_t:alg_socket   accept append bind connect create getattr getopt ioctl listen read setattr setopt shutdown write  ;
allow daemon init_t:alg_socket   getattr getopt ioctl read setopt write  ;
allow devicekit_disk_t domain:alg_socket getattr;
allow lvm_t lvm_t:alg_socket   append bind connect create getattr getopt ioctl read setattr setopt shutdown write  ;
allow sosreport_t domain:alg_socket getattr;
allow sysadm_t domain:alg_socket getattr;
allow unconfined_domain_type domain:alg_socket   accept append bind connect create getattr getopt ioctl listen lock map name_bind read recvfrom relabelfrom relabelto sendto setattr setopt shutdown write  ;
The above is the same as on the Trixie release policy as these things aren t changed often. Below is from Debian/Bookworm which is the same apart from Bookworm not allowing lvm_t:
$ sesearch -A -c alg_socket /tmp/policy.33
allow NetworkManager_t NetworkManager_t:alg_socket   accept bind create read setopt write  ;
allow bluetooth_t bluetooth_t:alg_socket   accept append bind connect create getattr getopt ioctl listen read setattr setopt shutdown write  ;
allow daemon init_t:alg_socket   getattr getopt ioctl read setopt write  ;
allow devicekit_disk_t domain:alg_socket getattr;
allow sosreport_t domain:alg_socket getattr;
allow sysadm_t domain:alg_socket getattr;
allow unconfined_domain_type domain:alg_socket   accept append bind connect create getattr getopt ioctl listen lock map name_bind read recvfrom relabelfrom relabelto sendto setattr setopt shutdown write  ;
I checked every Debian policy back to when the alg_socket class was first added and found that the older versions had fewer domains granted access. The most recently added was bluetooth_t and the one before that was NetworkManager_t. The Risky Lines Of those allow statements the following are the risks: Unconfined Domains and the unconfined_domain_type Attribute When writing policy lines like the following line aren t generally considered a problem as unconfined domains are allowed full access to the system. However it can be an issue if you have a process in an unconfined domain without root access, which means a regular user login. Unfortunately this happens to be where this exploit and the default Debian SE Linux configuration intersect.
allow unconfined_domain_type domain:alg_socket   accept append bind connect create getattr getopt ioctl listen lock map name_bind read recvfrom relabelfrom relabelto sendto setattr setopt shutdown write  ;
The following shell code gets a list of unconfined domains which can be entered from user domains.
A=""
for n in $(seinfo -x -a unconfined_domain_type grep _t$) ; do
  A="$A ($n)"
done
A=$(echo $A sed -e s/^.//)
sesearch -T -s user_application_exec_domain -c process egrep "$A;"
Below is the output on a Debian/Trixie (Stable) system. So a confined user in the user_t domain could run an X server and try and get it to run the exploit code (which seems difficult) or running a Wine or Mono program from the Window manager in a Wayland environment.
type_transition user_t xserver_exec_t:process xserver_t;
type_transition user_wm_t mono_exec_t:process mono_t;
type_transition user_wm_t wine_exec_t:process wine_t;
type_transition user_wm_t xserver_exec_t:process xserver_t;
The issue of unconfined domains in SE Linux policy needs much more work. I ll write some blog posts about it later and the next release of Debian will be significantly better in this regard. Daemons that Have Access
allow NetworkManager_t NetworkManager_t:alg_socket   accept bind create read setopt write  ;
allow bluetooth_t bluetooth_t:alg_socket   accept append bind connect create getattr getopt ioctl listen read setattr setopt shutdown write  ;
Network Manager is something that can potentially be exploited by a desktop user as it has a large attack surface for the desktop interface. But as the vast majority of desktop user accounts are unconfined that s not an issue. This might be an issue for some restricted desktop PCs, maybe kiosk systems and those PCs that were being installed in prisons. The bluetooth_t domain is used by the bluetooth daemon that runs as root. While we generally are less concerned about a root process being exploited the daemon will handle some data from hostile sources and it could be used as an escalation attack by someone with a hostile Bluetooth device. These can t be exploited without another bug. The Lines that Aren t Problems The getattr Lines
allow devicekit_disk_t domain:alg_socket getattr;
allow sosreport_t domain:alg_socket getattr;
allow sysadm_t domain:alg_socket getattr;
The above getattr access isn t an issue as it just allows seeing process information, and it s also by privileged domains. The init_t Sockets
allow daemon init_t:alg_socket   getattr getopt ioctl read setopt write  ;
The daemon access to sockets inherited from init_t probably isn t a great idea, it s from the following section in init.te which is to allow socket activation for daemons, the comment is concerning in this context. Also socket_class_set is overly broad as without even inspecting the systemd source code I m pretty sure that far fewer than 1/3 of the 55 classes allowed by that rule are actually supported in systemd.
ifdef( init_systemd', 
        # Until systemd is fixed
        allow daemon init_t:socket_class_set   getattr getopt ioctl read setopt write  ;
But that s not really a problem as systemd has to just not create a socket of that type, if a hostile party can make systemd create such sockets then you have probably already lost. SE Linux Protection Overall SE Linux systems running confined users (kiosks and other confined GUI environments) will be protected barring a bug in Network Manager or the Bluetooth daemon as long as there is no Xserver installed (or the X server won t run scripts on startup), no Wine system installed, and no Mono. SE Linux servers and VMs will be protected against daemon issues as long as the daemon isn t unconfined. To convert the default login to user_t run the following commands:
semanage login -m -s user_u -r s0 __default__
restorecon -R -v -F /home
But it is still possible to access an unconfined domain from user_t (a topic I will address in detail in a future blog post). To remove unconfined entirely (not a task for novices or something to be done on in production without testing and planning) run the following commands:
semanage login -m -s root -r s0 root
# logout and login again
semodule -X 100 -r unconfined
Then a Debian/Trixie system running SE Linux will be safe against this attack even when running a vulnerable kernel. If you still want to use root as unconfined_t but still have untrusted shell users then run the following command to remove the easiest ways for users to run a program in an unconfined domain:
semodule -X 100 -r mono wine
Success and Failure Blocked by SE Linux Below is what happens on stdout/stderr when SE Linux blocks the exploit (tested with vulnerable Debian kernel 6.12.74+deb13+1-amd64):
test@testing1:~$ python3 ./copy_fail_exp.py 
Traceback (most recent call last):
  File "/home/test/./copy_fail_exp.py", line 9, in <module>
    while i<len(e):c(f,i,e[i:i+4]);i+=4
                   ~^^^^^^^^^^^^^^
  File "/home/test/./copy_fail_exp.py", line 5, in c
    a=s.socket(38,5,0);a.bind(("aead","authencesn(hmac(sha256),cbc(aes))"));h=279;v=a.setsockopt;v(h,1,d('0800010000000010'+'0'*64));v(h,5,None,4);u,_=a.accept();o=t+4;i=d('00');u.sendmsg([b"A"*4+c],[(h,3,i*4),(h,2,b'\x10'+i*19),(h,4,b'\x08'+i*3),],32768);r,w=g.pipe();n=g.splice;n(f,w,o,offset_src=0);n(r,u.fileno(),o)
  File "/usr/lib/python3.13/socket.py", line 233, in __init__
    _socket.socket.__init__(self, family, type, proto, fileno)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^PermissionError: [Errno 13] Permission denied
test@testing1:~$ su
Password:
When the attack is blocked by SE Linux there will be no messages in the kernel message log but the SE Linux audit log (typically stored in /var/log/audit/audit.log) will have lines like the following:
type=AVC msg=audit(1777803068.070:76): avc:  denied    create   for  pid=811 comm="python3" scontext=user_u:user_r:user_t:s0 tcontext=user_u:user_r:user_t:s0 tclass=alg_socket permissive=0
type=SYSCALL msg=audit(1777803068.070:76): arch=c000003e syscall=41 success=no exit=-13 a0=26 a1=80005 a2=0 a3=0 items=0 ppid=791 pid=811 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=1 comm="python3" exe="/usr/bin/python3.13" subj=user_u:user_r:user_t:s0 key=(null)ARCH=x86_64 SYSCALL=socket AUID="test" UID="test" GID="test" EUID="test" SUID="test" FSUID="test" EGID="test" SGID="test" FSGID="test"
type=PROCTITLE msg=audit(1777803068.070:76): proctitle=707974686F6E33002E2F636F70795F6661696C5F6578702E7079
For that the :76 is the audit log entry number, the command ausearch -i -a 76 will interpret that message with the following output:
type=PROCTITLE msg=audit(05/03/26 10:11:08.070:76) : proctitle=python3 ./copy_fail_exp.py 
type=SYSCALL msg=audit(05/03/26 10:11:08.070:76) : arch=x86_64 syscall=socket success=no exit=EACCES(Permission denied) a0=alg a1=SOCK_SEQPACKET a2=ip a3=0x0 items=0 ppid=791 pid=811 auid=test uid=test gid=test euid=test suid=test fsuid=test egid=test sgid=test fsgid=test tty=pts0 ses=1 comm=python3 exe=/usr/bin/python3.13 subj=user_u:user_r:user_t:s0 key=(null) 
type=AVC msg=audit(05/03/26 10:11:08.070:76) : avc:  denied    create   for  pid=811 comm=python3 scontext=user_u:user_r:user_t:s0 tcontext=user_u:user_r:user_t:s0 tclass=alg_socket permissive=0 
When it Works Below is what happens when it works (again tested with Debian kernel 6.12.74+deb13+1-amd64):
test@testing1:~$ python3 ./copy_fail_exp.py 
# 
Here is the kernel log when the attack works:
[   30.441830] alg: No test for authencesn(hmac(sha256),cbc(aes)) (authencesn(hmac(sha256-avx2),cbc-aes-aesni))
[   30.447466] process 'su' launched '/bin/sh' with NULL argv: empty string added
When the Kernel Isn t Vulnerable If the kernel isn t vulnerable and SE Linux permits the attack (EG run from an unconfined domain) the following is seen on stdout/stderr:
$ python3 ./copy_fail_exp.py 
Password: 
su: Authentication failure
In that situation the kernel will log something like the following:
[   36.647023] alg: No test for authencesn(hmac(sha256),cbc(aes)) (authencesn(hmac-sha256-lib,cbc-aes-aesni))
This was tested on the Debian/Unstable kernel 6.19.13+deb14-amd64. Conclusion Run the following commands and then force all users to logout to make a Debian SE Linux system offering shell access reasonably safe against this bug. But also upgrade your kernel as soon as convenient because having multiple layers of protection is always good.
semanage login -m -s user_u -r s0 __default__
restorecon -R -v -F /home
semodule -X 100 -r mono wine
The GrapheneOS people are doing really good work on securing phones, I am most interested in Mobian (Debian on phones) but for people who have made different choices GrapheneOS is a good option. Here is the GrapheneOS statement on Copy Fail (they are not vulnerable to it) [3]. For people interested in running a secure Android build GrapheneOS is the best option. Their supported devices list shows Pixel 6 to Pixel 10 supported and Pixel 8 to Pixel 10a recommended [4]. In Australia Kogan sells refurbished Pixel 6 phones starting at $251 including delivery and refurbished Pixel 8 phones starting at $499 with First membership, they seem to have the cheapest Pixel phones. I want to make Debian more like Android in terms of security, but that s a topic for other blog posts. Here is the Debian page listing kernels that have been fixed against this exploit [5].

23 April 2026

Sergio Talens-Oliag: Developing a Git Worktree Helper with Copilot

Over the past few weeks I ve been developing and using a personal command-line tool called gwt (Git Worktree) to manage Git repositories using worktrees. This article explains what the tool does, how it evolved, and how I used GitHub Copilot CLI to develop it (in fact the idea of building the script was also to test the tool).

The Problem: Managing Multiple BranchesI was working on a project with multiple active branches, including orphans; the regular branches are for fixes or features, while the orphans are used to keep copies of remote documents or store processed versions of those documents. The project also uses a special orphan branch that contains the scripts and the CI/CD configuration to store and process the external documents (it is on a separate branch to avoid mixing its operation with the main project code). The plan is trigger a pipeline against the special branch from remote projects to create or update the doc branch for it in our git repository, retrieving artifacts from the remote projects to get the files and put them on an orphan branch (initially I added new commits after each update, but I changed the system to use force pushes and keep only one commit, as the history is not really needed). The original documents have to be changed, so, after ingesting them, we run a script that modifies them and adds or updates another branch with the processed version; the contents of that branch are used by the main branch build process (there we use git fetch and git archive to retrieve its contents). When working on the scripts to manage the orphan branches I discovered the worktree feature of git, a functionality that allows me to keep multiple branches checked out in parallel using a single .git folder, removing the need to use git switch and git stash when changing between branches (until now I ve been a heavy user of those commands). Reading about it I found that a lot of people use worktrees with the help of a wrapper script to simplify the management. After looking at one or two posts and the related scripts I decided to create my own using a specific directory structure to simplify things. That s how I started to work on the gwt script; as I also wanted to test copilot I decided to build it using its help (I have a pro license at work and wanted to play with the cli version instead of integrated into an editor, as I didn t want to learn a lot of new keyboard shortcuts).

The gwt Philosophy: Opinionated and Transparentgwt enforces a simple, filesystem-visible model:
  • Exactly one bare repository named bare.git (treated as an implementation detail)
  • One worktree directory per branch where the directory name matches the branch name
  • Single responsibility: gwt doesn t try to be a general git wrapper; it only handles operations that map cleanly to this layout
The repository structure looks like this:
my-repo/
+-- bare.git/           # the Git repository (internal)
+-- main/               # worktree for branch "main"
+-- feature/api/        # worktree for branch "feature/api"
+-- fix/docs/           # worktree for branch "fix/docs"
+-- orphan-history/     # worktree for the "orphan-history" branch
The tool follows five core design principles:
  1. Explicit over clever: Git commands are not hidden or reinterpreted
  2. Transparent execution: Every operation is printed before it happens
  3. Safe, preview-first operations: Destructive commands default to preview, confirmation, then apply
  4. Shell-agnostic core: The script never changes the caller s working directory (shell wrappers handle that)
  5. Opinionated but minimal: Only commands that fit the layout model are included

Core CommandsThe script provides these essential commands:
  • gwt init <url> Clone a repository and set up the gwt layout
  • gwt convert <dir> Convert an existing Git checkout to the gwt layout
  • gwt add [--orphan] <branch> [<base>] Create a new worktree (optionally orphaned)
  • gwt remove <branch> Remove a worktree and unregister it (asks the user to remove the local branch too, useful when removing already merged branches)
  • gwt rename <old> <new> Rename a branch AND its worktree directory
  • gwt list List all worktrees
  • gwt default [<branch>] Get or set the default branch
  • gwt current Print the current worktree or branch name
Except init and convert all of the commands work inside a directory structure that follows the gwt layout, which looks for the bare.git folder to find the root folder of the structure. As I don t want to hide which commands are really used by the wrapper, all git and filesystem operations pass through a single run shell function that prints each command before executing it. This gives complete visibility into what the tool is doing. Also, destructive operations (remove, rename) default to preview mode:
$ gwt remove feature-old --dry-run
+ git -C bare.git branch -d feature-old
+ git -C bare.git worktree remove feature-old/
Apply these changes? [y/N]:
The user sees exactly what will happen, can verify it s correct, and only then confirm execution.

Incremental Development with CopilotThe gwt script has grown from 597 lines in its original version (git-wt) to 1,111 lines when writing the first draft of this post. This growth happened through incremental, test-driven development, with each feature being refined based on real usage patterns. What follows is a little history of the script evolution written with the help of git log.

Initial versionFirst I wrote a design document and asked copilot to create the initial version of the git-wt script with the original core commands. I started to use the tool with a remote repostory (I made copies of the branches in some cases to avoid missing work) and fixed bugs (trivial ones with neovim, larger ones asking copilot to fix the issues for me, so I had less typing to do).

First command updateOne of the first commands I had to enhance was rename:
  • as I normally use branches with / on their name and my tool checks out the worktrees using the branch name as the path inside the gwt root folder (i.e. a fix/rename branch creates the fix directory and checks the branch inside the fix/rename folder) the rename command had to clean up the empty parent directories
  • when renaming a worktree we move the folders and fix the references using the worktree repair command to make things work locally, but the rename also affects the remote branch reference, to avoid surprises the command unsets the remote branch reference so it can be pushed again using the new name (of course, the user is responsible of managing the old remote branch, as the gwt can t guess what it should do with it).

Integration with the shellAs I use zsh with the Powerlevel10k theme I asked copilot to help me add visual elements to the prompt when working with gwt folders, something that I would have never tried without help, as it would have required a lot of digging on my part on how to do it, as I never looked into it. The initial version of the code was on an independent file that I sourced from my .zshrc file and it prints on the right part of the prompt when we are inside a gwt folder (note that if the folder is a worktree we see the existing git integration text right before it, so we have the previous behavior and we see that it is a gwt friendly repo) and if we are on the root folder or the bare.git folder we see gwt or bare (I added the text because there are no git promts on those folders). I also asked copilot to create zsh autocompletion functions (I only use zsh, so I didn t add autocompletion for other shells). The good thing here is that I wouldn t have done that manually, as it would have required some reading to get it right, but the output of copilot worked and I can update things using it or manually if I need to. One thing I was missing from the script was the possibility of changing the working directory easily, so I wrote a gwt wrapper function for zsh that intercepts commands that require shell cooperation (changing the working directory) and delegates everything else to the core script. Currently the function supports the following enhanced commands:
  • cd [<branch>]: change into a worktree or the default one if missing
  • convert <dir>: convert a checkout, then cd into the initial worktree
  • add [--orphan] <branch> [<base>]: create a worktree, then cd into it on success
  • rename <old> <new>: rename a worktree, then cd into it if we were inside it
Note that the cd command will not work on other shells or if the user does not load my wrapper, but the rest will still work without the working directory changes.

Renaming the commandAs I felt that git-wt was a long name I renamed the tool to gwt, I could have done it by hand, but using copilot I didn t have to review all files by myself and it did it right (note that I have it configured to always ask me before doing changes, as it sometimes tries to do something I don t want and I like to check its changes as I have the files in git repos, I manually add the files when I like the status and if the cli output is not clear I allow it to apply it and check the effects with git diff so I can validate or revert what was done).

The convert commandAfter playing with one repo I added the convert subcommand for migrating existing checkouts, it seemed a simple task at first, but it took multiple iterations to get it right, as I found multiple issues while testing (in fact I did copies of the existing checkouts to be able to re-test each update, as some of the iterations broke them). The version of the function when this post was first edited had the following comment explaining what it does:
# ---------------------------------------------------------------------------
# convert - convert an existing checkout into the gwt layout
# ---------------------------------------------------------------------------
#
# Must be run from the parent directory of <dir>.
#
# Steps:
#   1. Read branch from the checkout's HEAD
#   2. Rename <dir> to <dir>.wt.tmp (sibling, same filesystem)
#   3. Create <dir>/ as the new gwt root
#   4. Move <dir>.wt.tmp/.git to <dir>/bare.git; set core.bare = true
#   5. Fix fetch refspec (bare clone default maps refs directly, no remotes/)
#   6. Add a --no-checkout worktree so git wires up the metadata and
#      creates <dir>/<branch>/.git (the only file in that dir)
#   7. Move that .git file into the real working tree (<dir>.wt.tmp)
#   8. Remove the now-empty placeholder directory
#   9. Move the real working tree into place as <dir>/<branch>
#  10. Reset the index to HEAD so git status is clean
#      (--no-checkout leaves the index empty)
#  11. Create <dir>/.git -> bare.git symlink so plain git commands work
#      from the root without --git-dir
#
# The .git file ends up at the same absolute path git recorded in step 5,
# so no worktree repair is needed. Working tree files are never modified.
The .git link was added when I noticed that I could run commands that don t need the checked out files on the root of the gwt structure, which is handy sometimes (i.e. a git fetch or a git log, that shows the log of the branch marked as default). After playing with commands that used the bare.git folder I updated the init and convert commands to keep the origin refs, ensuring that the remote tracking works correctly.

Improving the add commandWhile playing with the tool on more repos I noticed that I also had to enhance the add command to better handle worktree creation, depending on my needs. Right now the tool supports the following use cases:
  • if the branch exists locally or on origin, it just checks it out.
  • if the branch does not exist, we create it using the given base branch or, if no base is given, the current worktree (if we are in the root folder or bare.git the command fails).
  • as I needed it for my project, I added a --orphan option to be able to create orphan branches directly.

Moving to a single fileEventually I decided to make the tool self contained; I removed the design document (I moved the content to comments on the top of the script and details to comments on each function definition) and added a pair of commands to print the code to source for the p10k and zsh integration (autocompletion & functions), leaving everything in a single file. Now my .zshrc file adds the following to source both things:
# After loading the p10k configuration
if type gwt >/dev/null 2>&1; then
  source <(gwt p10k)
fi
[...]
# After loading autocompletion
if type gwt >/dev/null 2>&1; then
  source <(gwt zsh)
fi

VersioningAs I modified the script I found interesting to use CalVer-based versioning (the version variable has the format YYYY.mm.dd-r#) so I added a subcommand to show its value or bump it using the current date and computing the right revision number.

About the use of copilotAlthough I ve never been a fan of AI tools I have to admit that the copilot CLI has been very useful for building the tool:
  • Rapid prototyping: Each commit represented a small feature or fix that I could implement, test immediately in my actual workflow, and iterate on based on the result
  • Edge case handling: Rather than trying to anticipate every scenario upfront, I could ask Copilot how to handle edge cases as they appeared in real usage
  • Script refinement: Questions like "how do I clean up empty directories after a rename" or "how do I detect if I m inside a specific worktree" were quickly answered with working code
  • Shell integration: The Zsh wrapper and completion system grew from simple prototypes to sophisticated features, with each iteration informed by how I actually used the tool
For example, the convert command started as a simple rename operation, but evolved to also create a .git symlink and intelligently handle various migration scenarios all because I used it repeatedly and refined the implementation each time.

Self-Contained and Opinionatedgwt is deliberately opinionated:
  • Zsh & Powerlevel10k Integration: The tool includes built-in Zsh shell integration, accessed via source <(gwt zsh) and supports adding a prompt segment when using p10k, as described earlier.
  • Directory Structure: The bare.git directory name is non-negotiable. This is how gwt discovers the repository root from any subdirectory, and how the tool knows whether a directory is a gwt repository. The simplicity of this marker means the discovery mechanism is foolproof and requires no configuration.
  • No Configuration Files: gwt deliberately has no configuration. There are no .gwtrc files or config directories. This makes it portable; the tool works the same way everywhere, and repositories can be shared across systems without synchronizing configuration.

From Script to SystemWhat started as a small helper script for managing worktrees has become a complete system:
  1. Core script (gwt): 1,111 lines of pure shell, no external dependencies
  2. Shell integration: Zsh functions and completions
  3. Prompt integration: Powerlevel10k segment
  4. Documentation: Built-in help and design philosophy documentation
The script is self-contained, everything needed for the tool to work is in a single file. This makes it trivial to update (just replace the script) or audit (no hidden dependencies).

Development with AI supportDeveloping gwt with copilot taught me some things:
  • Incremental refinement works well for small tools: Each iteration informed the next, resulting in a tool that handles real use cases elegantly
  • Transparency is a feature: Making operations visible builds confidence and is easier to debug
  • Opinionated tools can be powerful: By constraining the problem space (one bare repo, one worktree per branch), the solution becomes simpler and more robust
  • Shell integration matters: The same core commands are easier to use when they can automatically change directories and provide completions
  • Real-world testing is essential: I wouldn t have discovered the need for automatic directory cleanup or context-aware cd behavior without actually using the tool daily

What was next?The tool is stable and handles my daily workflow well, so my guess is that I would keep using it and fixing issues if or when I found them, but I do not plan to include additional features unless I find a use case that justifies it (i.e. I never added support for some of the worktree subcommands, as it is easier to use the git versions if I ever needed them).

What really happenedWhile editing this post I discovered that I needed to add another command to it and fixed a bug (see below). With those changes and the inclusion of a license and copyright notice (just in case I distribute it at some point) now the script is 1,217 lines long instead of the 1,111 it had when I started to write this entry.

Submodule SupportWhen I converted this blog repository to the gwt format and tried to preview the post using docker compose, it failed because the worktree I was on didn t have the Git submodule initialized. My blog theme is included on the repository as a submodule, and when I used gwt to check out different branches in worktrees, the submodule was not initialized in the new worktrees. This led me to add new internal function and a gwt submodule command to handle submodule initialization; the internal function is called from convert and add (when converting a repo or adding a worktree) and the public command is useful to update the submodules on existing branches.

Path Handling with Branch Names Containing SlashesThe second discovery was a bug in how the tool handled branch names containing slashes (e.g., feature/new-api, docs/user-guide), the worktree directories are created with the branch name as the path, so a branch like feature/new-api would create two nested folders (feature and new-api inside it). However, there was a mismatch in how the zsh wrapper function resolved worktree paths (initially it used shell parameter expansion, i.e. rel="$ cwd#"$REPO_ROOT"/ "), versus how the core script calculated them, causing the cd command to fail or navigate to the wrong location when branch names contained slashes. The fix involved ensuring consistent path resolution throughout the script and wrapper (now it uses a function that processes the git worktree list output), so that gwt cd feature/new-api correctly navigates to the worktree directory regardless of path depth.

Conclusiongwt is a tool that solves a real problem: managing multiple Git branches simultaneously without context-switching overhead. I m sure I m going to keep using it for my projects, as it simplifies some workflows, although I ll also use switch and stash in some cases, but I like the use of multiple worktrees in parallel. In fact I converted this blog repository checkout to the gwt format to work on a separate branch as it felt the right approach even if I m the only one using the repo now, and it helped me improve the tool, as explained before. Also, it was a good example of how to use AI tools like copilot to develop a simple tool and keep it evolving while using it. In any case, although I find the copilot useful and has saved me time, I don t trust it to work without supervision, it worked well, but got stuck some times and didn t do the things as I wanted in multiple occasions. I also have an additional problem now I ve been reading about it, but I don t really know which models to use or how the premium requests are computed (I ve only been playing with it since last month and I ran out of requests the last day of the month on purpose, just to see what happened it stops working ;). On my work machine I ve been using a specific user account with a GitHub Copilot Business subscription and I only used the Anthropic Claude Sonnet 4.6 model and with my personal account I configured the Anthropic Claude Haiku 4.5 model, but I ve only used that to create the initial draft of this post (I ended up rewriting most of it manually anyway) and to review the final version (I m not a native speaker and it was useful for finding typos and improving the style in some parts). I guess I ll try other models with copilot in the future and check other command line tools like aider or claude-code, but probably only using free accounts unless I get a payed account at work, as I have with GitHub Copilot. To be fair, what I will love to be able to do is to use local models (aider can do it), but the machines I have are not powerful enough. I tried to run a simple test and it felt really slow, but when I have the time or the need I ll try again, just in case.

22 April 2026

Vincent Bernat: CSS & vertical rhythm for text, images, and tables

Vertical rhythm aligns lines to a consistent spacing cadence down the page. It creates a predictable flow for the eye to follow. Thanks to the rlh CSS unit, vertical rhythm is now easier to implement for text.1 But illustrations and tables can disrupt the layout. The amateur typographer in me wants to follow Bringhurst s wisdom:
Headings, subheads, block quotations, footnotes, illustrations, captions and other intrusions into the text create syncopations and variations against the base rhythm of regularly leaded lines. These variations can and should add life to the page, but the main text should also return after each variation precisely on beat and in phase. Robert Bringhurst, The Elements of Typographic Style

Text Three factors govern vertical rhythm: font size, line height and margin or padding. Let s set our baseline with an 18-pixel font and a 1.5 line height:
html  
  font-size: 112.5%;
  line-height: 1.5;
 
h1, h2, h3, h4  
  font-size: 100%;
 
html, body,
h1, h2, h3, h4,
p, blockquote,
dl, dt, dd, ol, ul, li  
  margin: 0;
  padding: 0;
 
CSS Values and Units Module Level 4 defines the rlh unit, equal to the computed line height of the root element. All browsers support it since 2023.2 Use it to insert vertical spaces or to fix the line height when altering font size:3
h1, h2, h3, h4  
  margin-top: 2rlh;
  margin-bottom: 1rlh;
 
h1  
  font-size: 2.5rem;
  line-height: 2rlh;
 
h2  
  font-size: 1.5rem;
  line-height: 1rlh;
 
h3  
  font-size: 1.25rem;
  line-height: 1rlh;
 
p, blockquote, pre  
  margin-top: 1rlh;
 
aside  
  font-size: 0.875rem;
  line-height: 1rlh;
 
We can check the result by overlaying a grid4 on the content:
Screenshot of my website with a grid as an overlay and each line of text fitting on the grid
Using CSS rlh unit to set vertical space works well for text. You can display the grid using Ctrl+Shift+G.
If a child element uses a font with taller intrinsic metrics, it may stretch the line s box beyond the configured line height.5 A workaround is to reduce the line height to 1. The glyphs overflow but don t push the line taller.
code, kbd  
  line-height: 1;
 

Responsive images Responsive images are difficult to align on the grid because we don t know their height. CSS Rhythmic Sizing Module Level 1 introduces the block-step property to adjust the height of an element to a multiple of a step unit. But most browsers don t support it yet. With JavaScript, we can add padding around the image so it does not disturb the vertical rhythm:
const targets = document.querySelectorAll(".lf-media-outer");
const adjust = (el, height) =>  
  const rlh = parseFloat(getComputedStyle(document.documentElement).lineHeight);
  const padding = Math.ceil(height / rlh) * rlh - height;
  el.style.padding =  $ padding / 2 px 0 ;
 ;
targets.forEach((el) => adjust(el, el.clientHeight));
Screenshot of my website with a grid as an overlay and an image not breaking the vertical rhythm. Additional padding is visible before and after the image. The height of the image with padding is 216.
The image is snapped to the grid thanks to the additional padding computed with JavaScript. 216 is divisible by 27, our line height in this example.
As the image is responsive, its height can change. We need to wrap a resize observer around the adjust() function:
const ro = new ResizeObserver((entries) =>  
  for (const entry of entries)  
    const height = entry.contentBoxSize[0].blockSize;
    adjust(entry.target, height);
   
 );
for (const target of targets)  
  ro.observe(target);
 

Tables Table cells could set 1rlh as their height but they would feel constricted. Using 2rlh wastes too much space. Instead, we use incremental leading: we align one in every five lines.
table  
  border-spacing: 2px 0;
  border-collapse: separate;
  th  
    padding: 0.4rlh 1em;
   
  td  
    padding: 0.2rlh 0.5em;
   
 
To align the elements after the table, we need to add some padding. We can either reuse the JavaScript code from images or use a few lines of CSS that count the regular rows and compute the missing vertical padding:
table:has(tbody tr:nth-child(5n):last-child)     padding-bottom: 0.2rlh;  
table:has(tbody tr:nth-child(5n+1):last-child)   padding-bottom: 0.8rlh;  
table:has(tbody tr:nth-child(5n+2):last-child)   padding-bottom: 0.4rlh;  
table:has(tbody tr:nth-child(5n+3):last-child)   padding-bottom: 0  
table:has(tbody tr:nth-child(5n+4):last-child)   padding-bottom: 0.6rlh;  
A header cell has twice the padding of a regular cell. With two regular rows, the total padding is 2 2 0.2+2 0.4=1.6. We need to add 0.4rlh to reach 2rlh of extra vertical padding across the table.
Screenshot of my website with a grid as an overlay and a table following the vertical rhythm. Additional padding is visible after the table. The height of the table with padding is 405.
One line out of five is aligned to the grid. Additional padding is added after the table to not break the vertical rhythm. 405 is divisible by 27, our line height in this example.

None of this is necessary. But once you start looking, you can t unsee it. Until browsers implement CSS Rhythmic Sizing, a bit of CSS wizardry and a touch of JavaScript is enough to pull it off. The main text now returns after each intrusion precisely on beat and in phase.

  1. See Vertical rhythm using CSS lh and rlh units by Pawe Grzybek.
  2. For broader compatibility, you can replace 2rlh with calc(var(--line-height) * 2rem) and set the --line-height custom property in the :root pseudo-class. I wrote a simple PostCSS plugin for this purpose.
  3. It would have been nicer to compute the line height with calc(round(up, calc(2.4rem / 1rlh), 0) * 1rlh). Unfortunately, typed arithmetic is not supported by Firefox yet. Moreover, browsers support round() only since 2024. Instead, I coded a PostCSS plugin for this as well.
  4. The following CSS code defines a grid tracking the line height:
    body  
      position: relative;
     
    body::after  
      content: "";
      position: absolute;
      inset: 0;
      z-index: 9999;
      background: linear-gradient(180deg, #c8e1ff99 1px, transparent 1px);
      background-size: 20px 1rlh;
      pointer-events: none;
     
    
  5. See Deep dive CSS: font metrics, line-height and vertical-align by Vincent De Oliveira.

21 April 2026

Ravi Dwivedi: LibreOffice Conference Budapest 2025

In September 2025, I attended the LibreOffice Conference in Budapest, Hungary, on the 4th and the 5th, and a community meeting on the 3rd. Thanks to The Document Foundation (TDF) for sponsoring my travel and accommodation costs. The conference venue was Faculty of Informatics, E tv s Lor nd University (ELTE). The conference was planned to be held from the 4th to the 6th, but the program for the 6th of September had to be canceled due to the venue being unavailable because of a marathon in Budapest. So, all the talks got squeezed into just two days, making the schedule a bit hectic. The TDF had booked my room at the Corvin Hotel. It was a double bedroom with a window. The breakfast was included in the hotel booking. The hotel was walking distance from the conference venue. One could also take a tram from the hotel to reach the venue.
A double bed A shot of my room. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Tram A tram in Budapest. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

3rd of September On the 3rd of September, we had a community meeting at the above-mentioned venue. I walked with my friend Dione to the venue. Upon reaching there, I noticed that the university had no boundaries and gates. This reminded me of the previous year s conference venue in Luxembourg, which also had no boundaries or gates. In contrast, Indian universities and institutes typically have walls and gates serving as boundaries to separate them from the rest of the city. Many of these institutes also have security guards at the entrance, who may ask attendees to present proof of admission before allowing them inside. I was surprised to find that institutes in Europe, like the one where the conference was held, did not have such boundaries. The building where the conference was held was red, which happened to be the same color as the building for the previous year s conference venue. I remember joking with Dione that the criteria for the conference venue might have been the color of the building.
A red building The red building in the picture served as the conference venue. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
During the community meeting, we shared ideas on how to spread the word about LibreOffice. The meeting lasted for a couple of hours. After the community meeting, we went to the hotel for dinner sponsored by the TDF.
Cake slices These Esterh zy cake bites were really yummy. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Raspberry Currant cake slices Raspberry Currant cake slices. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

4th of September On the first day of the conference, attendees were given swag bags containing a pad, sticky notes, a pen, a conference T-shirt, and a bottle.
A blue colored T-shirt on a bed along with a pen, a bottle, a diary and a sticky note Conference swag. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
The talks started early in the morning with Eliane Domingos, Chairperson of TDF s Board of Directors, giving the inauguration talk. As always, I found Italo Vignoli s talk on the importance of document freedom interesting. During the snack break, I noticed that there were three types of milk available for coffee: cow s milk, lactose-free milk, and almond milk. Almond milk is rare in India, but I have managed to get it, but I have never seen lactose-free milk in India. Since I run fundraisers in my projects, such as Prav, I could relate to Lothar K. Becker s talk. He discussed the issue that certain implementations in LibreOffice require a budget that is too large for any single interested entity to fund independently. Furthermore, The Document Foundation (TDF) cannot legally receive funds from government entities. Therefore, there is no organization or entity to pool resources from all the interested entities to finance the implementation.
Lothar giving his presentation Lothar giving his presentation. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Another talk was by the Austrian Armed Forces on their migration to LibreOffice. I wanted to know why they migrated, and I found out that they did it for their digital sovereignty, and not for saving on the license costs. Another point presented in the talk was that LibreOffice is available on all the operating systems, while the Microsoft Office suite is not that widely available. The migration was systematic and was performed over a few years. They started working on it in 2021, and the migration was finished recently. In addition, it also required training their staff in using LibreOffice.
Presentation on migration to LibreOffice by Austrian Armed Forces Presentation on migration to LibreOffice by Austrian Armed Forces. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
The lunch was inside the university canteen. We were provided lunch coupons by the TDF. I got a vegan coupon with 4000 Ft written on it, which meant I could take lunch for up to 4000 Hungarian forints.
My lunch ticket My lunch ticket for the conference. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
The lunch I had on the first day The lunch I had on the first day. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
During the evening, it was my turn for the presentation. I was done with preparing my slides ten days before my talk. I also got my slides reviewed by friends. My talk was finished in 20 minutes, while I was given a 30-minute slot. This helped us catch up on the schedule. Furthermore, I made my talk interactive by asking questions and making sure that the audience was not asleep. During my talk, my friend Dione took my pictures with my camera. My talk was on how free software projects could give users a say in freedom to modify the software. I illustrated this using the Prav project that I am a part of. After the talks were over, we were treated to a conference dinner at Trofea Grill. It had a great selection of desserts, which helped me sample some Hungarian desserts. The sponge cake was especially good.
Desserts at Tofea Grill Desserts at Tofea Grill. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.

5th of September The next day the 5th of September I went with Dione to the venue early in the morning, as her talk was the first one of the day. Her talk was titled Managing Tasks with Nextcloud Deck. Later that day, I also attended a talk on Collabora. At lunch, I found the egg white salad quite tasty.
Dione giving her presentation Dione giving her presentation. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Egg white salad Egg white salad. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
After the lunch break, we had the conference group photo. I had a Nikon camera, which we used to take the group photo. I requested a university student to take our group photo and also taught her how to operate the camera.
People looking at the camera and smiling Group photo
By the evening, the conference ended, after which we went to a pub, which was again sponsored by TDF. I had beer, but that one really tasted bad, so I couldn t finish it. The only vegetarian option was goat cheeseburger, which my friend Manish and I opted for. The burger tasted awful. Apparently, I don t like goat cheese. The next day I went sightseeing with Dione in Budapest. Stay tuned for our adventures! Credits: Thanks to Dione and Richard for proofreading.

20 April 2026

Russ Allbery: Review: Surface Detail

Review: Surface Detail, by Iain M. Banks
Publisher: Orbit
Copyright: October 2010
Printing: May 2011
ISBN: 0-316-12341-2
Format: Trade paperback
Pages: 627
Surface Detail is the ninth novel in Banks's Culture science fiction (literary space opera?) series. As with most of the Culture novels, it can be read in any order, although this isn't the best starting point. There is an Easter egg reference to Use of Weapons that would be easier to notice if you have read that book recently, but which is not that important to the story. Lededje Y'breq is an Indented Intagliate from the Sichultian Enablement. Her body is patterned from her skin down to her bones, covered with elaborate markings similar to tattoos that extend to her internal organs. As an intagliate, she is someone's property. In her case, she is the property of Joller Veppers, the richest man in the Enablement and her father's former business partner. Intagliates are a tradition of great cultural pride in the Enablement. They are a living representation of the seriousness with which debts and honor are taken, up to and including one's not-yet-born children becoming the property of one's debtor. Such children are decorated as living works of art of the highest skill and technical sophistication; after all, the Enablement are not barbarians. As the story opens, Lededje is attempting, not for the first time, to escape. This attempt is successful in an unexpected way. Prin and Chay are Pavulean researchers and academics who, as this story opens, are in Hell. They are not dead; they have infiltrated the Hell that Pavuleans are shown to scare them into proper behavior in order to prove that it is not an illusion and their society does indeed torture people in an afterlife, in more awful ways than people dare imagine. They have reached the portal through which temporary visitors exit, hoping to escape with firm evidence of the existence and horrors of the Pavulean afterlife. They will not be entirely successful. Yime Nsokyi is a Culture agent for Quietus, the part of Contact that concerns itself with the dead. Many advanced societies throughout the galaxy have invented and reinvented the ability to digitize a mind and then run it in a virtual environment. Once a society can capture the minds of every person in that society from that point forward, it faces the question of whether to do so and, if it does, what to do with those minds. More specifically, it faces the moral question of whether to punish the minds of people who were horrible in life. It faces the question of whether to create Hell. Vatueil is a soldier in a contestation, a limited and carefully monitored virtual war. The purpose of that war game is to, once and for all, resolve the question of whether civilizations should be allowed to create Hells. Some civilizations consider them integral to their religion or self-conception. Others consider them morally abhorrent, and that conflict was in danger of spilling over into war in the Real. Hence the War in Heaven: Both sides committed to fight in a virtual space under specific and structured rules, and the winner decides the fate of the galaxy's Hells. Vatueil is fighting for the anti-Hell side. The anti-Hell side is losing. There are very few authors who were better at big-idea science fiction than Iain M. Banks. I've been reading a few books about AI ships and remembered that I had two unread Culture novels that I was saving. It felt like a good time to lose myself in something sprawling. Surface Detail does sprawl. Even by Banks's standards, there was an impressive amount of infodumping in this book. Banks always has huge and lovingly described set pieces, and this book is no exception, but there are also paragraphs and pages of background and cultural musings and galactic politics. We are introduced to not one but three new Contact divisions; as well as the already-mentioned Quietus, there is Numina, which concerns itself with the races that have sublimed (transcended), and Restoria, which deals with hegemonizing swarms (grey goo nanotech, paperclip maximizers, and their equivalents). Infodumping is both a feature and a bane of big-idea science fiction, and it helps to be in the right mood. It also helps if the info being dumped is interesting, and this is where Banks shines. This is a huge, sprawling book, but it deals with some huge, sprawling questions and it has interesting and non-reductive thoughts about them. The problems posed by the plot come with history, failed solutions, multi-sided political disputes, strategies and tactics of varying morality and efficacy, and an effort to wrestle with the irreducible complexity of trying to resolve political and ethical disagreements in a universe full of profound disagreements and moral systems that one cannot simply steamroll. It also helps that the characters are interesting, even when they're not likable. Surface Detail has one fully hissable villain (Veppers) as a viewpoint character, but even Veppers is interesting in a "let me check the publication date to see if Banks was aware of Peter Thiel" sort of way. The Culture ships, of which there are several in this story, tend towards a gently sarcastic kindness that I find utterly charming. Lededje provides the compelling motive force of someone who has no involvement in the broader philosophical questions and instead intends to resolve one specific problem through lethal violence. Vatueil and Yime were a bit bland in personality, more exposition generators than characters I warmed to, but their roles and therefore the surrounding exposition were fascinating enough that I still enjoyed their sections. I'm sure this is not an original observation, but I was struck reading this book in the first half of 2026 that the Culture functions as an implementation of what the United States likes to think it is but has never been. It has a strong sense of shared ethics and moral principles, it tries to export them to the rest of the galaxy through example, persuasion, and careful meddling, but it tries to follow some combination of pragmatic and moral rules while doing so, partly to avoid a backlash and partly to avoid becoming its own sort of hegemonizing swarm. That is a powerfully attractive vision of how to be an advanced civilization, and the fact that every hegemon that has claimed that mantle has behaved appallingly just makes it more intriguing as a fictional concept. In this book, like in many Culture books, the Culture is painfully aware of the failure modes of meddling, and the story slowly reveals the effort the Culture put into staying just on a defensible side of their own moral lines. This is, in a sense, a Prime Directive story, but with a level of hard-nosed pragmatism and political sophistication that the endless Star Trek Prime Directive episodes never reach. Surface Detail does tend to sprawl, and I'm not sure Banks pulled together all the pieces of the plot. For example, if there was a point to the subplot involving the Unfallen Bulbitian, it was lost on me. (There is always a possibility with Banks that I wasn't paying close enough attention.) But the descriptions are so elaborate and the sense of politics and history are so deep that I was never bored, even when following a plot thread that meandered off into apparent irrelevance. The main plot line comes to a satisfying conclusion that may be even more biting social commentary today than it was in 2010. A large part of the plot does involve Hell, so a warning for those who haven't read much Banks: He adores elaborate descriptions of body horror and physical torture. The sections involving Prin and Chay are rather grim and horrific, probably a bit worse than Dante's Inferno. I have a low tolerance for horror and I was able to read past and around the worst bits, but be warned that Banks indulges his love for the painfully grotesque quite a bit. This was great, and exactly what I was hoping for when I picked it up. It's not the strongest Culture novel (for me, that's either The Player of Games or Excession), but it's one of the better ones. Highly recommended, although if you're new to the Culture, I would start with one of the earlier books that provide a more gradual introduction to the Culture and Special Circumstances. Followed, in the somewhat disconnected Culture series sense, by The Hydrogen Sonata. Content warnings: Rape (largely off-screen), graphic violence, lots of Bosch-style grotesque torture, and a lot of Veppers being a thoroughly awful human being as a viewpoint character. Rating: 8 out of 10

14 April 2026

Russell Coker: Furilabs FLX1s Finally Working

I ve been using the Furilabs FLX1s phone [1] as my daily driver for 6 weeks, it s a decent phone, not as good as I hoped but good enough to use every day and rely on for phone calls about job interviews etc. I intend to keep using it as my main phone and as a platform to improve phone software in Debian as you really can t effectively find bugs unless you use the platform for important tasks. Support Problems I previously wrote about the phone after I received it without a SIM caddy on the 13th of Jan. I had a saga with support about this, on the 16th of Jan one support person said that they would ship it immediately but didn t provide a tracking number or any indication of when it would arrive. On the 5th of Feb I contacted support again and asked how long it would be, the new support person seemed to have no record of my previous communication but said that they would send it. On the 17th of Feb I made another support request including asking for a way of direct communication as the support email came from an address that wouldn t accept replies, I was asked for a photo showing where the problem is. The support person also said that they might have to send a replacement phone! The last support request I sent included my disappointment at the time taken to resolve the issue and the proposed solution of replacing the entire phone (why have two international shipments of a fragile and expensive phone when a single letter with a cheap SIM caddy would do?). I didn t receive a reply but the SIM caddy arrived on the 2nd of Mar. Here is a pic of the SIM caddy and the package it came in: One thing that should be noted is that some of the support people seemed to be very good at their jobs and they were all friendly. It was the system that failed here, turning a minor issue of a missing part into a 6 week saga. Furilabs needs to do the following to address this issue:
  1. Make it possible to reply directly to a message from a support person. Accept email with a custom subject to sort it, give a URL for a web form, anything. Collating discussions with a customer allows giving better support while taking less time for the support people.
  2. Have someone monitor every social media address that is used by the company. When someone sends a support request in a public Mastodon post it indicates that something has gone wrong and you want to move quickly to resolve it.
  3. Take care of the little things, like sending a tracking number for every parcel. If it s something too small for a parcel (the SIM caddy could have fit in a regular letter) then just tell the customer what date it was posted and where it was posted from so they have some idea of when it will arrive.
This is not just a single failure of Furilabs support, it s a systemic failure of their processes. Problems I Will Fix Unless Someone Beats Me to it Here are some issues I plan to work on. Smart Watch Support I need to port one of the smart watch programs to Debian. Also I want to make one of them support the Colmi P80 [2]. A smart watch significantly increases the utility of a phone even though IMHO they aren t doing nearly all the things that they could and should do. When we get Debian programs talking to the PineTime it will make a good platform for development of new smart phone and OS features. Nextcloud I have ongoing issues of my text Nextcloud installation on a Debian VM not allowing connection from the Linux desktop app (as packaged in Debian) and from the Android client (from f-droid). The desktop client works with a friend s Nextcloud installation on Ubuntu so I may try running it on an Ubuntu VM I run while waiting for the Debian issue to get resolved. There was a bug recently fixed in Nextcloud that appears related so maybe the next release will fix it. For the moment I ve been running without these features and I call and SMS people from knowing their number or just returning calls. Phone calls generally aren t very useful for me nowadays except when applying for jobs. If I could deal with recruiters and hiring managers via video calls then I would consider just not having a phone number. Wifi IPv6 Periodically IPv6 support just stops working, I can t ping the gateway. I turn wifi off and on again and it works. This might be an issue with my wifi network configuration. This might be an issue with the way I have configured my IPv6 networking, although that problem doesn t happen with any of my laptops. Chatty Sorting Chatty is the program for SMS that is installed by default (part of the phosh/phoc setup), it also does Jabber. Version 0.8.7 is installed which apparently has some Furios modifications and it doesn t properly support sorting SMS/Jabber conversations. Version 0.8.9 from Debian sorts in the same way as most SMS and Jabber programs with the most recent at the top. But the Debian version doesn t support Jabber (only SMS and Matrix). When I went back to the Furilabs version of Chatty it still sorted for a while but then suddenly stopped. Killing Chatty (not just closing the window and reopening it) seems to make it sort the conversations sometimes. Problems for Others to Fix Here are the current issues I have starting with the most important. Important The following issues seriously reduce the usability of the device. Hotspot The Wifi hotspot functionality wasn t working for a few weeks, this Gitlab issue seems to match it [3]. It started working correctly for a day and I was not sure if an update I applied fixed the bug or if it s some sort of race condition that worked for this boot and will return next time I reboot it. Later on I rebooted it and found that it s somewhat random whether it works or now. Also while it is mostly working it seemed to stop working about every 25 minutes or so and I had to turn it off and on again to get it going. On another day it went to a stage where it got repeated packet loss when I pinged the phone as a hotspot from my laptop. A pattern of 3 ping responses and 3 Destination Host Unreachable messages was often repeated. I don t know if this is related to the way Android software is run in a container to access the hardware. 4G Reliability Sometimes 4G connectivity has just stopped, sometimes I can stop and restart the 4G data through software to fix it and sometimes I need to use the hardware switch. I haven t noticed this for a week or two so there is a possibility that one fix addressed both Hotspot and 4G. One thing that I will do is setup monitoring to give an alert on the phone if it can t connect to the Internet. I don t want it to just quietly stop doing networking stuff and not tell me! On-screen Keyboard The compatibility issues of the GNOME and KDE on-screen keyboards are getting me. I use phosh/phoc as the login environment as I want to stick to defaults at first to not make things any more difficult than they need to be. When I use programs that use QT such as Nheko the keyboard doesn t always appear when it should and it forgets the setting for word completion (which means spelling correction). The spelling correction system doesn t suggest replacing dont with don t which is really annoying as a major advantage for spelling checkers on touch screens is inserting an apostrophy. An apostrophy takes at least 3* longer than a regular character and saving that delay makes a difference to typing speed. The spelling correction doesn t correct two words run together. Medium Priority These issues are ongoing annoyances. Delay on Power Button In the best case scenario this phone has a much slower response to pressing the power button than the Android phones I tested (Huawei Mate 10 Pro and Samsung Galaxy Note 9) and a much slower response than my recollection of the vast majority of Android phones I ve ever used. For testing pressing buttons on the phones simultaneously resulted in the Android phone screens lighting up much sooner. Something like 200ms vs 600ms I don t have a good setup to time these things but it s very obvious when I test. In a less common case scenario (the phone having been unused for some time) the response can be something like 5 seconds. The worst case scenario is something in excess of 20 seconds. For UI designers, if you get multiple press events from a button that can turn the screen on/off please make your UI leave the screen on and ignore all the stacked events. Having the screen start turning on and off repeatedly when the phone recovers and processes all the button presses isn t good, especially when each screen flash takes half a second. Notifications Touching on a notification for a program often doesn t bring it to the foreground. I haven t yet found a connection between when it does and when it doesn t. Also the lack of icons in the top bar on the screen to indicate notifications is annoying, but that seems to be an issue of design not the implementation. Charge Delay When I connect the phone to a power source there is a delay of about 22 seconds before it starts to charge. Having it miss 22 seconds of charge time is no big deal, having to wait 22 seconds to be sure it s charging before leaving it is really annoying. Also the phone makes an audible alert when it gets to 0% charge which woke me up one night when I had failed to push the USB-C connector in hard enough. This phone requires a slightly deeper connector than most phones so with some plugs it s easy to not quite insert them far enough. Torch aka Flash The light for the torch or flash for camera is not bright at all. In a quick test staring into the light from 40cm away wasn t unpleasant compared to my Huawei Mate 10 Pro which has a light bright enough that it hurts to look at it from 4 meters away. Because of this photos at night are not viable, not even when photographing something that s less than a meter away. The torch has a brightness setting which doesn t seem to change the brightness, so it seems likely that this is a software issue and the brightness is set at a low level and the software isn t changing it. Audio When I connect to my car the Lollypop player starts playing before the phone directs audio to the car, so the music starts coming from the phone for about a second. This is an annoying cosmetic error. Sometimes audio playing pauses for no apparent reason. It doesn t support the phone profile with Bluetooth so phone calls can t go through the car audio system. Also it doesn t always connect to my car when I start driving, sometimes I need to disable and enable Bluetooth to make it connect. When I initially set the phone up Lollypop would send the track name when playing music through my car (Nissan LEAF) Bluetooth connection, after an update that often doesn t happen so the car doesn t display the track name or whether the music is playing but the pause icon works to pause and resume music (sometimes it does work). About 30 seconds into a phone call it switches to hands-free mode while the icon to indicate hands-free is not highlighted, so I have to press the hands-free button twice to get it back to normal phone mode. Low Priority I could live with these things remaining as-is but it s annoying. Ticket Mode There is apparently some code written to display tickets on screen without unlocking. I want to get this working and store screen-caps of the Android barcode screens of the different loyalty cards so I can scan them without unlocking. My threat model does not include someone trying to steal my phone to get a free loaf of bread on the bakery loyalty program. Camera The camera app works with both the back and front cameras, which is nice, and sadly based on my experience with other Debian phones it s noteworthy. The problem is that it takes a long time to take a photo, something like a second after the button is pressed long enough for you to think that it just silently took a photo and then move the phone. The UI of the furios-camera app is also a little annoying, when viewing photos there is an icon at the bottom left of the screen for a video camera and an icon at the bottom right with a cross. Which every time makes me think record videos and leave this screen not return to taking photos and delete current photo . I can get used to the surprising icons, but being so slow is a real problem. GUI App Installation The program for managing software doesn t work very well. It said that there were two updates for Mesa package needed, but didn t seem to want to install them. I ran flatpak update as root to fix that. The process of selecting software defaults to including non-free, and most of the available apps are for desktop/laptop with no way to search for phone/tablet apps. Generally I think it s best to just avoid this and use apt and flatpak directly from the command-line. Being able to ssh to my phone from a desktop or laptop is good! Android Emulation The file /home/furios/.local/share/andromeda/data/system/uiderrors.txt is created by the Andromeda system which runs Android apps in a LXC container and appears to grow without end. After using the phone for a month it was 3.5G in size. The disk space usage isn t directly a problem, out of the 110G storage space only 17G is used and I don t have a need to put much else on it, even if I wanted to put backups of /home from my laptop on it when travelling that would still leave plenty of free space. But that sort of thing is a problem for backing up the phone and wasting 3.5G out of 110G total is a fairly significant step towards breaking the entire system. Also having lots of logging messages from a subsystem that isn t even being used is a bad sign. I just tried using it and it doesn t start from either the settings menu or from the f-droid icon. Android isn t that important to me as I want to get away from the proprietary app space so I won t bother trying this any more. Unfixable Problems Unlocking After getting used to fingerprint unlocking going back to a password is a pain. I think that the hardware isn t sufficient for modern quality face recognition that can t be fooled by a photo and there isn t fingerprint hardware. When I first used an Android phone using a pin to unlock didn t seem like a big deal, but after getting used to fingerprint unlock it s a real drag to go without. This is a real annoyance when doing things like checking Wikipedia while watching TV. This phone would be significantly improved with a fingerprint sensor or a camera that worked well enough for face unlock. Plasma Mobile According to Reddit Plasma Mobile (KDE for phones) doesn t support Halium and can never work on this phone because of it [4]. This is one of a number of potential issues with the phone, running on hardware that was never designed for open OSs is always going to have issues. Wifi MAC Address The MAC keeps changing on reboot so I can t assign a permanent IPv4 address to the phone. It appears from the MAC prefix of 00:08:22 that the network hardware is made in InPro Comm which is well known for using random addresses in the products it OEMs. They apparently have one allocation of 2^24 addresses and each device randomly chooses a MAC from that range on boot. In the settings for a Wifi connection the Identity tab has a field named Cloned Address which can be set to Stable for SSID that prevents it from changing and allows a static IP address allocation from DHCP. It s not ideal but it works. Network Manager can be configured to have a permanent assigned MAC address for all connections or for just some connections. In the past for such things I have copied MAC addresses from ethernet devices that were being discarded and used them for such things. For the moment the Stable for SSID setting does what I need but I will consider setting a permanent address at some future time. Docks Having the ability to connect to a dock is really handy. The PinePhonePro and Librem5 support it and on the proprietary side a lot of Samsung devices do it with a special desktop GUI named Dex and some Huawei devices also have a desktop version of the GUI. It s unfortunate that this phone can t do it. The Good Things It s good to be able to ssh in to my phone, even if the on-screen keyboard worked as well as the Android ones it would still be a major pain to use when compared to a real keyboard. The phone doesn t support connecting to a dock (unlike Samsung phones I ve used for which I found Dex to be very useful with a 4K monitor and proper keyboard) so ssh is the best way to access it. This phone has very reliable connections to my home wifi. I ve had ssh sessions from my desktop to my phone that have remained open for multiple days. I don t really need this, I ve just forgotten to logout and noticed days later that the connection is still running. None of the other phones running Debian could do that. Running the same OS on desktop and phone makes things easier to test and debug. Having support for all the things that Linux distributions support is good. For example none of the Android music players support all the encodings of audio that comes from YouTube so to play all of my music collection on Android I would need to transcode most of them which means either losing quality, wasting storage space, or both. While Lollypop plays FLAC0, mp3, m4a, mka, webm, ogg, and more. Conclusion This is a step towards where I want to go but it s far from the end goal. The PinePhonePro and Librem5 are more open hardware platforms which have some significant benefits. But the battery life issues make them unusable for me. Running Mobian on a OnePlus 6 or Droidian on a Note 9 works well for the small tablet features but without VoLTE. While the telcos have blocked phones without VoLTE data devices still work so if recruiters etc would stop requiring phone calls then I could make one of them an option. The phone works well enough that it could potentially be used by one of my older relatives. If I could ssh in to my parents phones when they mess things up that would be convenient. I ve run this phone as my daily driver since the 3rd of March and it has worked reasonably well. 6 weeks compared to my previous use of the PinePhonePro for 3 days. This is the first time in 15 years that a non-Android phone has worked for me personally. I have briefly used an iPhone 7 for work which basically did what it needed to do, it was at the bottom of the pile of unused phones at work and I didn t want to take a newer iPhone that could be used by someone who s doing more than the occasional SMS or Slack message. So this is better than it might have been, not as good as I hoped, but a decent platform to use it while developing for it.

12 April 2026

Vasudev Kamath: Hardening the Unpacakgeable: A systemd-run Sandbox for Third-Party Binaries

The Shift in Software Consumption Historically, I have been a "distribution-first" user. Sticking to tools packaged within the Debian archives provides a layer of trust; maintainers validate licenses, audit code, and ensure the entire dependency chain is verified. However, the rapid pace of development in the Generative AI space specifically with new tools like Gemini-CLI has made this traditional approach difficult to sustain. Many modern CLI tools are built within the npm or Python ecosystems. For a distribution packager, these are a nightmare; packaging a single tool often requires packaging a massive, shifting dependency chain. Consequently, I found myself forced to use third-party binaries, bypassing the safety of the Debian archive.
The Supply Chain Risk Recent supply chain attacks affecting widely used packages like axios and LiteLLM have made it clear: running unvetted binaries on a personal system is a significant risk. These scripts often have full access to your $HOME directory, SSH keys, and the system D-Bus. After discussing these concerns with a colleague, I was inspired by his approach using a Flatpak-style sandbox for even basic applications like Google Chrome. I decided to build a generalized version of this using OpenCode and Qwen 3.6 Fast (which was available for free use at the time) to create a robust, transient sandbox utility.
The Solution: safe-run-binary My script, safe-run-binary, leverages systemd-run to execute binaries within an isolated scope. It implements strict filesystem masking and resource control to ensure that even if a dependency is compromised, the "blast radius" is contained.
Key Technical Features
1. Virtualized Home Directory (tmpfs)
Instead of exposing my real home directory, the script mounts a tmpfs over $HOME. It then selectively creates and bind-mounts only the necessary subdirectories (like .cache or .config) into a virtual structure. This prevents the application from ever "seeing" sensitive files like ~/.ssh or ~/.gnupg.
2. D-Bus Isolation via xdg-dbus-proxy
For GUI applications, providing raw access to the D-Bus is a security hole. The script uses xdg-dbus-proxy to sit between the application and the system bus. By using the --filter and --talk=org.freedesktop.portal.* flags, the app can only communicate with necessary portals (like the file picker) rather than sniffing the entire bus.
3. Linux Namespace Restrictions

The sandbox utilizes several systemd execution properties to harden the process:

  • RestrictNamespaces=yes: For CLI tools, this prevents the app from creating its own nested namespaces.
  • PrivateTmp=yes: Ensures a private /tmp space that isn't shared with the host.
  • NoNewPrivileges=yes: Prevents the binary from gaining elevated permissions through SUID/SGID bits.
4. GPU and Audio Passthrough
The script intelligently detects and binds Wayland, PipeWire, and NVIDIA/DRI device nodes. This allows browsers like Firefox to run with full hardware acceleration and audio support while remaining locked out of the rest of the filesystem.
Usage To run a CLI tool like Gemini-CLI with access only to a specific directory:
safe-run-binary -b ~/.gemini-config -- npx @google/gemini-cli
For a GUI application like Firefox:
safe-run-binary --gui -b ~/.mozilla -b ~/.cache/mozilla -b ~/Downloads -- firefox
Conclusion While it is not always possible to escape the need for third-party software, it is possible to control the environment in which it operates. By leveraging native Linux primitives like systemd and namespaces, high-grade isolation is achievable. PS: If you spot any issues or have suggestions for improving the script, feel free to raise a PR on the repo.

Russ Allbery: Review: The Teller of Small Fortunes

Review: The Teller of Small Fortunes, by Julie Leong
Publisher: Ace
Copyright: November 2024
ISBN: 0-593-81590-4
Format: Kindle
Pages: 324
The Teller of Small Fortunes is a cozy found-family fantasy with a roughly medieval setting. It was Julie Leong's first novel. Tao is a traveling teller of small fortunes. In her wagon, pulled by her friendly mule Laohu, she wanders the small villages of Eshtera and reads the trivial fortunes of villagers in the tea leaves. An upcoming injury, a lost ring, a future kiss, a small business deal... she looks around the large lines of fate and finds the small threads. After a few days, she moves on, making her solitary way to another village. Tao is not originally from Eshtera. She is Shinn, which means she encounters a bit of suspicion and hostility mixed with the fascination of the exotic. (Language and culture clues lead me to think Shinara is intended to be this world's not-China, but it's not a direct mapping.) Tao uses the fascination to help her business; fortune telling is more believable from someone who seems exotic. The hostility she's learned to deflect and ignore. In the worst case, there's always another village. If you've read any cozy found-family novels, you know roughly what happens next. Tao encounters people on the road and, for various reasons, they decide to travel together. The first two are a massive mercenary (Mash) and a semi-reformed thief (Silt), who join Tao somewhat awkwardly after Tao gives Mash a fortune that is far more significant than she intended. One town later, they pick up an apprentice baker best known for her misshapen pastries. They also collect a stray cat, because of course they do. It's that sort of book. For me, this sort of novel lives or dies by the characters, so it's good news that I liked Tao and enjoyed spending time with her. She's quiet, resilient, competent, and self-contained, with a difficult past and some mysteries and emotions the others can draw over time. She's also thoughtful and introspective, which means the tight third-person narration that almost always stays on Tao offers emotional growth to mull over. I also liked Kina (the baker) and Mash; they're a bit more obvious and straightforward, but Kina adds irrepressible energy and Mash is a good example of the sometimes-gruff soldier with a soft heart. Silt was a bit more annoying and I never entirely warmed to him, but he's tolerable and does get a bit of much-needed (if superficial) character development. It takes some time for the reader to learn about the primary conflict of the story (Tao does not give up her secrets quickly), so I won't spoil it, but I thought it worked well. I was momentarily afraid the story would develop a clear villain, but Leong has some satisfying alternate surprises in store. The ending was well-done, although it is very happily-ever-after in a way that may strike some readers as too neat. The Teller of Small Fortunes aims for a quiet and relaxed mood rather than forcing character development through difficult choices; it's a fine aim for a novel, but it won't match everyone's mood. I liked the world-building, although expect small and somewhat disconnected details rather than an overarching theory of magic. Tao's ability gets the most elaboration, for obvious reasons, and I liked how Leong describes it and explores its consequences. Most of the attention in the setting is on the friction, wistfulness, and small reminders of coming from a different culture than everyone around you, but so long ago that you are not fully a part of either world. This, I thought, was very well-done and is one of the places where the story is comfortable with complex feelings and doesn't try to reach a simplifying conclusion. There is one bit of the story that felt like it was taken directly out of a Dungeons & Dragons campaign to a degree that felt jarring, but that was the only odd world-building note. This book felt like a warm cup of tea intended to comfort and relax, without large or complex thoughts about the world. It's not intended to be challenging; there are a few plot twists I didn't anticipate, but nothing that dramatic, and I doubt anyone will be surprised by the conclusions it reaches. It's a pleasant time with some nice people and just enough tension and mystery to add some motivation to find out what happens next. If that's what you're in the mood for, recommended. If you want a book that has Things To Say or will put you on the edge of your seat, maybe save this one for another mood. All the on-line sources I found for this book call it a standalone, but The Keeper of Magical Things is set in the same world, so I would call it a loose series with different protagonists. The Teller of Small Fortunes is a complete story in one book, though. Rating: 7 out of 10

1 April 2026

Matthew Garrett: Self hosting as much of my online presence as practical

Because I am bad at giving up on things, I ve been running my own email server for over 20 years. Some of that time it s been a PC at the end of a DSL line, some of that time it s been a Mac Mini in a data centre, and some of that time it s been a hosted VM. Last year I decided to bring it in house, and since then I ve been gradually consolidating as much of the rest of my online presence as possible on it. I mentioned this on Mastodon and a couple of people asked for more details, so here we are. First: my ISP doesn t guarantee a static IPv4 unless I m on a business plan and that seems like it d cost a bunch more, so I m doing what I described here: running a Wireguard link between a box that sits in a cupboard in my living room and the smallest OVH instance I can, with an additional IP address allocated to the VM and NATted over the VPN link. The practical outcome of this is that my home IP address is irrelevant and can change as much as it wants - my DNS points at the OVH IP, and traffic to that all ends up hitting my server. The server itself is pretty uninteresting. It s a refurbished HP EliteDesk which idles at 10W or so, along 2TB of NVMe and 32GB of RAM that I found under a pile of laptops in my office. We re not talking rackmount Xeon levels of performance, but it s entirely adequate for everything I m doing here. So. Let s talk about the services I m hosting.

Web This one s trivial. I m not really hosting much of a website right now, but what there is is served via Apache with a Let s Encrypt certificate. Nothing interesting at all here, other than the proxying that s going to be relevant later.

Email Inbound email is easy enough. I m running Postfix with a pretty stock configuration, and my MX records point at me. The same Let s Encrypt certificate is there for TLS delivery. I m using Dovecot as an IMAP server (again with the same cert). You can find plenty of guides on setting this up. Outbound email? That s harder. I m on a residential IP address, so if I send email directly nobody s going to deliver it. Going via my OVH address isn t going to be a lot better. I have a Google Workspace, so in the end I just made use of Google s SMTP relay service. There s various commerical alternatives available, I just chose this one because it didn t cost me anything more than I m already paying.

Blog My blog is largely static content generated by Hugo. Comments are Remark42 running in a Docker container. If you don t want to handle even that level of dynamic content you can use a third party comment provider like Disqus.

Mastodon I m deploying Mastodon pretty much along the lines of the upstream compose file. Apache is proxying /api/v1/streaming to the websocket provided by the streaming container and / to the actual Mastodon service. The only thing I tripped over for a while was the need to set the X-Forwarded-Proto header since otherwise you get stuck in a redirect loop of Mastodon receiving a request over http (because TLS termination is being done by the Apache proxy) and redirecting to https, except that s where we just came from. Mastodon is easily the heaviest part of all of this, using around 5GB of RAM and 60GB of disk for an instance with 3 users. This is more a point of principle than an especially good idea.

Bluesky I m arguably cheating here. Bluesky s federation model is quite different to Mastodon - while running a Mastodon service implies running the webview and other infrastructure associated with it, Bluesky has split that into multiple parts. User data is stored on Personal Data Servers, then aggregated from those by Relays, and then displayed on Appviews. Third parties can run any of these, but a user s actual posts are stored on a PDS. There are various reasons to run the others, for instance to implement alternative moderation policies, but if all you want is to ensure that you have control over your data, running a PDS is sufficient. I followed these instructions, other than using Apache as the frontend proxy rather than nginx, and it s all been working fine since then. In terms of ensuring that my data remains under my control, it s sufficient.

Backups I m using borgmatic, backing up to a local Synology NAS and also to my parents home (where I have another HP EliteDesk set up with an equivalent OVH IPv4 fronting setup). At some point I ll check that I m actually able to restore them.

Conclusion Most of what I post is now stored on a system that s happily living under a TV, but is available to the rest of the world just as visibly as if I used a hosted provider. Is this necessary? No. Does it improve my life? In no practical way. Does it generate additional complexity? Absolutely. Should you do it? Oh good heavens no. But you can, and once it s working it largely just keeps working, and there s a certain sense of comfort in knowing that my online presence is carefully contained in a small box making a gentle whirring noise.

31 March 2026

Russ Allbery: Review: Code Blue Emergency

Review: Code Blue Emergency, by James White
Series: Sector General #7
Publisher: Orb
Copyright: 1987
Printing: May 2003
ISBN: 0-7653-0663-8
Format: Trade paperback
Pages: 252
Code Blue Emergency (annoying em-dash in original title) is the seventh book of James White's Sector General science fiction series about a vast multi-species hospital station. While there are some references to (and spoilers for) earlier books in the series, you don't have to remember the previous books to read this one. I had no trouble despite a nine-year gap. I read this as part of the Orb General Practice omnibus, which collects this novel and The Genocidal Healer. Cha Thrat is a Sommaradvan warrior-surgeon, member of a newly-discovered species that is beginning the process of contact with the Federation. She saved a Monitor corps human after an accident on her world, performing some some highly competent surgery on a species she had never seen before. That plus her somewhat outcast status on her own world due to her very traditional attitude towards medical ethics led Sector General to extend an offer of medical internship, and led her to leap into the unknown by accepting. This may have been a mistake; there is a great deal that Sector General does not understand about Sommaradvan medical ethics. This series entry is another proper (if somewhat episodic) novel and the first book of the series that doesn't primarily focus on Conway. He makes an appearance in his new role as Diagnostician, but only as a supporting character. Code Blue Emergency is told in the tight third-person perspective of Cha Thrat, an alien who finds many things about Sector General baffling, confusing, and ethically troubling (and who therefore provides a good reader surrogate for reintroducing the basics of how the hospital works). Using an alien viewpoint is a more sophisticated narrative technique than White has used previously. I'm glad he tried it, and it mostly works, although I have some complaints. Cha Thrat comes from the middle caste of a strictly hierarchical society of three castes, but is also immensely stubborn and used to a medical system in which doctors take sole responsibility for their patients. This creates a lot of cultural conflicts, and I do enjoy science fiction where the human attitudes are portrayed as the strange ones, but the cultural analysis offered by this novel is not very deep. The pattern of this book is for Cha Thrat to stumble into a successful approach to a problem while being either oblivious to or hostile to the normal hierarchical structure expected of medical trainees. This is believable as far as it goes. She is a skilled and intelligent doctor with some good instincts and a strong commitment to patient care, but is also culturally inclined to not ask for help. It makes sense for that to be a serious problem in a hospital. Unfortunately, no one says this directly. Sector General staff get quite upset in ways that seem more territorial than oriented towards patient safety, no one directly explains to Cha Thrat why following a process is important or shows examples of what could go wrong, and plot armor means that her mistakes usually have positive outcomes. One can extrapolate the reasons why she is not a good medical student, but the reader is forced to do the extrapolation. This is the sort of book where the narration makes clear there are unresolved cultural clashes that are going to cause problems but hides the details. To Cha Thrat, her perspective is so obvious she never bothers to explain it to the reader, so the specifics come as a surprise. As with the alien perspective, I've seen this technique used with more subtlety and sophistication in other books, but White's version mostly works. Cha Thrat is a sympathetic protagonist because she is truly trying to take the most ethical and empathetic action in every situation and is clearly competent. Most of my frustration as a reader, ironically, lands on the other Sector General doctors who seem to make little to no effort to understand her perspective when she fails to conform to their expectations. This is believable in the abstract, but the whole point of Sector General is that they're supposed to be wiser about interspecies difference than this. Also, sometimes their reactions just seem petty. Cha Thrat has a very hierarchical concept of medicine that matches the social classes of her culture. For her, the highest tier of doctor are wizards who treat rulers, because the work of rulers is mostly mental and intellectual and therefore the diseases of rulers are treated with magic spells performed with words to reshape their thinking rather than surgery on their bodies. O'Mara and the other Sector General psychologists take great offense at this, muttering about being called witch doctors, which I found completely absurd. This is a comprehensible, if odd, description of psychology from a wholly alien species. Surely one's first reaction should be that words like "wizard" or "magic" are translation errors. Don't get offended; look to see if the underlying substance matches, which it clearly does. Apart from cultural and psychological clashes, Code Blue Emergency has the standard episodic Sector General structure of interesting medical mysteries that require lateral thinking. I find this sort of puzzle story satisfying, particularly given the firm belief of every character in an essentially pacifist and empathetic approach to even the most alien of creatures. This determined non-violence is one of the more interesting things about this series, and it continues here. White does tend towards both biological and gender essentialism for everyone other than the protagonist and main supporting characters, but he seemed to be walking back some of the more outrageous limitations on women that appeared in previous books. There is still some nonsense in here about how females of any species can't be Diagnosticians, but then Cha Thrat, who is female, seems to violate the justification for that rule over the course of this novel (sadly without comment). Perhaps he's setting up for proving Sector General wrong about this prejudice. I picked this up after reading Elizabeth Bear's Machine, which is essentially a (better written) Sector General novel that got me in the mood for reading more. I wouldn't give Code Blue Emergency any awards, but it delivered exactly what I was looking for. This series is not as deep or well-written as some more recent SF, but it is reliably itself and reliably entertaining. There are worse things in a series. Recommended if you're in the mood for alien ER in space. The omnibus edition that I read has an introduction to both novels by John Clute. It does add some interesting insights, but (as is somewhat typical for Clute) it also spoils parts of both books. You may want to read it after you read the novels. Followed by The Genocidal Healer. Rating: 7 out of 10

30 March 2026

Jamie McClelland: Mailman3 has 2 databases. Whoops.

At May First we have been carefully planning our migration of about 1200 lists from mailman2 to mailman3 for almost six months now. We did a lot of user communications, had several months of beta testing with a handful of lists ported over, and everything was looking good. So we kicked off the migration! But, about 15% of the way through I started seeing sqlite lock errors. Wait, what? I carefully re-configured mailman3 to use postgres, not sqlite. Well, yes, but apparently that was for the database managing the email list configuration, not the database powering the django web app, which, incidentally, also includes hundresds of gigabytes of archives. In other words, the one we really need in postgres, not sqlite.

Moving from sqlite to postgres Well that sucks. We immediately stopped the migration to deal with this. I noticed that the web is full of useful django instructions on how to migrate your database from one database to antoher. However, if you read the fine print, those convenient looking dumpdata loaddata workflows are designed to move the table definitions and a small amount of data. In our case, even after just 15% of our lists moved, our sqlite database was about 30GB. I considered some of the hacks to manage memory and try to run this via django, but eventually decided that pgloader was a more robust option. This option also allowed me to more easily test things out on a copy of our sqlite database (made while mailman was turned off). This way I could migrate and re-migrate the sqlite database over and over without impacting our live installation until I was satisfied it was all working. My first decision was to opt out of pgloader s schema creation. I used django s schema creation tool by:
  • Turning off mailman3 and mailman3-web and changing the mailman web configuration to use the new postgresql database.
  • Running mailman-web migrate
  • Changing the mailman web configuration back to sqlite and starting everything again.
Note: I tried just adding new database settings in the mailman web configuration indexed to new - django has the ability to define different databases by name, then you can run mailman-web migrate --database new. But, during the migration, I caught django querying the sqlite database for some migrations that required referencing existing fields (specifically hyperkitty s 0003_thread_starting_email). I didn t want any of these steps to touch the live database so I opted for the cleaner approach. Once I had a clean postgres schema, I dumped it so I could easily return to this spot. Next I started working on our pgloader load file. After a lot of trial and error, I ended with:
LOAD DATABASE
    FROM sqlite:///var/lib/mailman3/sqlite-postgres-migration/mailman3web.clean.backup.db
    INTO postgresql://mailmanweb:xxxxxxxxxxx@localhost:5432/mailmanweb

WITH data only,
    reset sequences,
    include no drop,
    disable triggers,
    create no tables,
    batch size = 5MB,
    batch rows = 500,
    prefetch rows = 50,
    workers = 2,
    concurrency = 1

SET work_mem to '64MB',
    maintenance_work_mem to '512MB'

CAST type datetime to timestamptz drop default drop not null,
    type date to date drop default drop not null,
    type int when (= precision 1) to boolean using tinyint-to-boolean,
    type text to varchar using remove-null-characters;
The batch, prefetch, workers and concurreny settings are all there to ensure memory doesn t blow up. I also discovered that I had to make some changes to the schema before loading data. Mostly truncating tables that the django migrate command populated to avoid duplicate key errors:
TRUNCATE TABLE django_migrations CASCADE;
TRUNCATE TABLE django_content_type CASCADE;
TRUNCATE TABLE auth_permission CASCADE;
TRUNCATE TABLE django_site CASCADE;
And also, I had to change a column type. Apparently the mailman import process allowed an attachment file name that exceeds the limit for postgres, but was allowed into sqlite:
ALTER TABLE hyperkitty_attachment ALTER COLUMN name TYPE text
When pgloader runs, we still get a lot of warnings from pgloader, which wants to cast columns differently than django does. These are harmless (I was able to import the data without a problem). And there are still a lot of warnings along the lines of:
2026-03-30T14:08:01.691990Z WARNING PostgreSQL warning: constraint hyperkitty_vote_email_id_73a50f4d_fk_hyperkitty_email_id of relation hyperkitty_vote does not exist, skipping
These are harmless as well. They appear because disable triggers disables foreign key constraints. Without it, we wouldn t be able to load tables that require values in tables that have not yet been populated. After all the tweaking, the import of our 30GB sqlite database took about 40 minutes.

Final Steps I think the reset sequences from pgloader should take care of this, but just in case:
mailman-web sqlsequencereset hyperkitty mailman_django auth   mailman-web dbshell
And, just to ensure postgres is optimized, run this in the psql shell:
ANALYZE VERBOSE;

Last thoughts I understand very well all the decisions the mailman3 devs made in designing the next version of mailman, and if I was in the same place I may have made them the same ones. For example, separating the code running the mailing list from the code managing the archives and the web interface makes perfectly good sense - many people might want to run just the mailing list part without a web interface. And building the web interface in django makes a lot of sense as well - why re-invent the wheel? I m sure a lot of time and effort was saved by simply using the built in features you get for free with django. But the unfortunate consequence of these decisions is that sys admins have a much harder time. Almost everyone wants the email lists along with the web interface and the archives. But nobody wants two different configuration files with different syntaxes and logic, not to mention two different command lines to use for maintenance and configuration with completely different APIs. Trying to understand how to change a default template or set list defaults requires a lot of research and usually you have to write a python script to do it. I have finally come to the conclusion that mailman2 is designed for sys admins, while mailman3 is designed for developers. Despite these short comings, I am impressed with the community and their quick and friendly responses to the questions of a confused sys admin. That might be more valuable than anything else.

Russ Allbery: Review: The Cloak and Its Wizard

Review: The Cloak and Its Wizard, by R.Z. Nicolet
Publisher: UpLit Press
Copyright: February 2026
ISBN: 1-917849-15-X
Format: Kindle
Pages: 423
The Cloak and Its Wizard is a standalone (at least so far) urban fantasy superhero (sort of) novel. R.Z. Nicolet is the marketing pseudonym for Rachel Reddick. This is her first novel.
I'm picky about wizards. The wizards themselves will complain about that, but of course I'm picky. When I choose a wizard, barring utter abandonment of moral scruples, it's a till-death-do-us-part situation. (Their death, not mine. I'm the next best thing to indestructible.)
The Cloak of Sunset and Starlight is a major artifact, meaning that it has its own preferences and is capable of independent action. It has been sitting in a glass case in the wizards' library for about a hundred years, waiting for someone interesting. (Well, mostly sitting. Occasionally it sneaks out to eavesdrop or move the books around.) Veronica Noble is interesting. She's older than most initiates, thoughtful, observant, and clearly had some mundane career before joining the Order. Her aura is appealing, and her mental shields and resistance to influence are intriguing. Normally, the Cloak would take its time investigating a new potential wizard, but the Sword was making thoughtful rattling sounds, and no way is the Cloak going to let the Sword claim her first. Time to choose a new wizard!
It was nice, being draped over warm shoulders, and feeling a heartbeat again. I could tell she closed her eyes without even looking. She sighed. "I just got picked by the intransigent one, didn't I?"
The last time I picked a book from the Big Idea feature in Scalzi's Whatever blog, it didn't go that well, but if you're going to write a book specifically for me, I'm going to read it. There are very few tropes of SFF that I love more than intelligent companion objects, and Nicolet's introduction to the story was compelling. So I gave this book discovery method another chance. I'm glad I did, because this was exactly what I was in the mood for and a delight from cover to cover. Veronica Noble is not a typical wizard. She's a surgeon and was quite happy to be a surgeon until an unexpected encounter with a magical creature killed her brother. The forgetting spell cast by the wizards who came to handle the Cassandra wyrm didn't work on her, so she was dragged reluctantly into the secret magical world of the Order. This long-lived society of wizards quietly defends the world against magical intrusions from other planes of existence. Now she's a wizard with a magical cloak, which she is not at all sure she wants. Veronica is not the protagonist, though. The Cloak of Sunset and Starlight is. As far as it is concerned, its job is to assist its wizard, enjoy watching interesting feats of magic, and look fabulous doing so. It's protective, dramatic, rather vain, endlessly curious, easily bored, and intensely loyal. When it becomes clear that the Order has some serious problems, the Cloak knows what side it's on. This sounds a bit like urban fantasy, so I was surprised when the first superheroes showed up, although given the explicit Doctor Strange inspiration I probably should have expected them. The Order and the superheroes do not mix, at least at the start of the novel. The wizards view the superheroes as a loud and irritating intrusion and hide magical activities from them the same as they do the rest of the world. Veronica's opening opinion on superheroes is based on being a trauma surgeon in a hospital dealing with the aftermath of their fights (which makes me wonder if the author has read Hench, although the idea is older than that book). As with the Order, the role of superheroes in this world gets more complicated as the plot develops. There is a surprising amount of plot and some very nice world-building here, including multiple twists that I was not expecting. Veronica is the sort of stubborn and deeply ethical person who will not leave a problem alone if she has the ability to fix it, which is a good recipe for getting deeper and deeper into a complex plot. She's believable as a surgeon: somewhat taciturn, calm in emergencies, detail-oriented, methodical, and not at all dramatic. This makes the Cloak a perfect foil and complement. Watching their partnership develop was very satisfying. This is a sidekick novel, and like the best sidekick novels it makes the not-protagonist more interesting and more relatable by showing them from an outside and skewed perspective. Piecing together what Veronica must be thinking is part of the fun, as is sharing the Cloak's protectiveness towards her as it becomes clear how much she's been through and how good of a person she is. The Cloak's personality was a little too much like a cat for me I would have preferred a more unique viewpoint, fewer cat-coded shenanigans, and a bit less of the running laundry machine joke. But that's a quibble. Its endless curiosity drives the plot forward and uncovers more of the world-building, and I just love reading stories from the perspective of this sort of loyal and protective magical creature. I had so much fun with this book. It's a popcorn sort of book, and I thought the ending sputtered a little, but overall it was great. Parts of it could have been designed in a lab to appeal to me specifically, so I'm not sure if other people will enjoy it as much, but its hit rate with my friends so far has been good. Highly recommended, and I will be watching for any further novels from Nicolet. The Cloak and Its Wizard reaches a satisfying conclusion and doesn't advertise itself as part of a series, but there is room for a sequel. If Nicolet ever writes one, I'd read it. Rating: 8 out of 10

29 March 2026

Russell Coker: Ebook Readers in Debian

Laptop For a while I ve been using Calibre 8.5.0+ds-1+deb13u1 in Debian/Trixie running KDE for reading ebooks on my laptop, it generally works well and has a large font size. The only downsides of it for that use are taking more RAM than I would prefer (about 780M RSS which seems a lot for a relatively simple task) and having separate windows for the list of books and reading an actual book without any options to just open the last book and not delay me. I tried Arianna 25.04.0-1 in Debian/Trixie, it has a significantly smaller font size and doesn t allow high contrast colors as the default is black on gray with the dark theme in KDE. It also only allows left and right arrows for moving through the book while Calibre uses up/down, left/right, or pgup/pgdn so whatever keys seem reasonable to you are going to work. The RSS was 762M which wasn t great but wasn t the real problem. Rumours of Arianna using less RAM than Calibre seem exaggerated. Librem5 On my Librem5 phone with Plasma Mobile Calibre 8.5.0+ds-1+deb13u1 both the initial setup screen and the main screen for selecting a book to read don t work in the width of portrait view on the phone. After putting it in landscape mode it worked, but I couldn t touch on a book title to select it I had to touch on the number of the book at the left of the list box. But once it was loaded everything was fine. On the Librem5 Arianna 25.04.0-1 just worked fine, although only using left/right swipes to change pages instead of up/down was annoying. Furilabs FLX1s On my Furilabs FLX1s with phosh Arianna 25.04.0-1 and Calibre 8.16.2+ds+~0.10.5-3 both gave the same result of not displaying text or images from the book, I m not sure if it s phosh or some other aspect of the FLX1s configuration at fault. PinePhonePro On my PinePhonePro running Debian/Testing with Plasma Mobile Arianna 25.12.3-1 worked without any issue and up/down swipes worked. Calibre 9.5.0+ds+~0.10.5-1 had the initial screen work fine in portrait mode but the main screen was too wide and needed landscape. Also the issue of having to touch the number applied. Laptop running Debian/Unstable Calibre 9.6.0+ds+~0.10.5-2 and Arianna 25.12.3-1 worked quite nicely on a Thinkpad running Debian/Unstable. One thing I discovered while testing it is that Calibre supports the CTRL-PLUS and CTRL-MINUS key combinations to change font sizes and that also works on the version in Debian/Trixie. Arianna doesn t support CTRL-PLUS/MINUS. Conclusion The problems I had were Arianna on a laptop, everything on the Furilabs FLX1s, and Calibre s UI not being well adjusted for mobile devices.

Russ Allbery: Review: The Sovereign

Review: The Sovereign, by C.L. Clark
Series: Magic of the Lost #3
Publisher: Orbit
Copyright: September 2025
ISBN: 0-316-54286-5
Format: Kindle
Pages: 575
The Sovereign is the third and concluding book of C.L. Clark's Magic of the Lost high fantasy trilogy. I recommend reading the books of this series close together, since there are a lot of characters and a lot of continuity between books that is helpful to remember, but it was not quite as difficult this time to remember where the story left off. At the end of The Faithless, the political situation in Balladaire (not-France) was more stable, but the threat of a plague lay on the horizon. That threat arrives in earnest in this book, along with new threats from both Balladaire's former colonial conscript soldiers and from neighboring Taargen (not-Germany, sort of, although the parallel isn't as close). Luca and Touraine have finally admitted that they're deeply in love, but they are still very different people with different goals and ethics. Luca is determined to do anything necessary to save her kingdom, but her definition of her kingdom is sharp and brittle. Touraine is torn between far too many loyalties, plus the lingering worry that her morals and Luca's may not be compatible. I think the hardest part of this sort of series is finding an ending the reader will find satisfying. This one, unfortunately, did not work for me, but that may be more due to personal preference than objective flaws. There have been two threads through this series: an improbable romance embedded in a network of complex personal relationships, and a political commentary on colonialism and post-colonial wars. I was enjoying the former, but it was the latter that felt fresh and interesting to me. The plot threads in The Faithless outside of Balladaire expanded that complexity, and I was hoping the final volume would continue in that direction. How could a colonial power atone for its history? How does the former colony establish its own governance? Is there a path to freedom without violence? Are attempts to chart a more moral course doomed to open lines of attack for one's other enemies? It's clear that Clark was thinking about similar themes, but The Sovereign narrows the field instead of widens it, restricts the political options, and then resolves most questions in a massive war. This is not that surprising of a conclusion, but it's one that I found unsatisfying and, honestly, a little boring. Yes, one way to resolve all the competing tensions is for everyone to try to kill each other and whoever survives wins, and historically that's one of the more likely outcomes, but that ending doesn't wrestle with the politics as much as it collapses them. Clark instead focuses this concluding volume on the romance, which becomes even more fraught, tragic, and dramatic than it was in previous books (and that's saying something). The hard questions of divided loyalties and moral conflicts are mostly framed by questions about Touraine's loyalty to Luca and Luca's trust of Touraine. This is all very Shakespearean, full of hard choices, sudden reversals, miscommunication, and a very deep conflict between Luca's realpolitik and Touraine's stubborn personal morality. If this is what you were reading the series for, if you were hoping for a maximum-drama sapphic relationship, you may thoroughly enjoy this. I thought it had its moments, but I wish they had been balanced by more moments of cool-headed practicality and creative political ingenuity. My biggest frustration with this ending is that the characters largely stop doing politics. The political complexity was the strength of both The Unbroken and The Faithless: People who intensely dislike each other negotiate because there is something larger to be gained, personal decisions made without considering the political ramifications have costs, and multiple characters are trying hard to find a way to turn a nasty, exploitative world into something better without simply killing everyone who disagrees. Many of the characters were objectively bad at politics, inexperienced and immature, but they stumbled or dragged or fought their way into political solutions anyway. I thought Clark moved too far away from that in The Sovereign. Everyone goes deep into their own emotions and desire for vengeance or conquest or revolution and stops compromising. To a depressingly large extent, the story is resolved by killing everyone who disagrees. I think the story is poorer for it. One of the other threads of the series is Balladairan magic, or rather its odd absence. Luca has one understanding of it, the rebels introduced in The Faithless have a different understanding of it, and its pursuit is set up as critical to resolving the threat of a plague. We do get an explanation of sorts, but it's not as complete or as satisfying as I was hoping, and the symbolism of Balladaire's missing magic is left frustratingly murky. For me, this has some of the same problems as the political conclusion: I wanted an intellectual catharsis alongside the emotional catharsis, but that was not the direction Clark was taking the story. I like reading about these characters. All of Luca, Touraine, and Pruett are complex, comprehensible, flawed, and often intriguing. But my favorite character in the story, the person I latched on to as an emotional path through the story, was Sabine. Her refreshingly straightforward loyalty and lack of drama was a breath of fresh air. She has some great moments in this book, but there too I got wrong-footed by the direction Clark went with her arc and found its conclusion deeply unsatisfying. I'm not sure how many of these complaints are because of missed opportunities in the novel, how many were due to a mismatch of taste, and how many were due to not being in the right mood to read this conclusion. I'm sure that it didn't help that I read this simultaneous with another novel in which the characters were always miserable, or that I read it in early 2026 with, uh, all that entails. I suspect that if you came away from the first two books invested in the messy romance and wanting MOAR DRAMA, you may get exactly what you were hoping for. That, sadly, was not what I was hoping for. I can't really recommend this. I thought it dragged in places and didn't deliver the ending I wanted. But it has some great moments, it does wrap up the threads of the trilogy as advertised, and at least the romance gets a dramatic climax worthy of the tension that has been built through the previous books. If that matches what you were enjoying in the previous books, you may well enjoy this more than I did. Rating: 5 out of 10

28 March 2026

Russell Coker: Communication and Hostile AIs

We seem to be entering an AI apocalypse of sorts, they aren t going to kill us or even take our jobs. What they are doing is destroying the Internet commons by filling it with rubbish. This isn t even real AI, just pattern matching and prediction systems, mostly LLMs. The Problem Scott Shambaugh s saga of being attacked and defamed by an OpenClaw AI bot is interesting and raises some disturbing possibilities for future online discussion [1]. Imagine what it would be like if everyone who was in any way notable for free software work had 100 such bots going after them. Dania Dumas wrote an insightful blog post about why OpenClaw is impossible to secure and why it won t go away [2]. Bruce Schneier and Nathan E. Sanders wrote an insightful article about the AI generated text arms race [3] primarily concentrating on situations in which text that was assumed to be written by humans but was actually written in bulk by bots was performing a DOS attack on people who were reviewing it. There are many situations such as book publishing and publishing letters to the editor of newspapers where getting new material from unknown people is an important part of the job but where there are also people making low quality submissions that are almost a DOS attack at the best of times. Currently the email spam problem continues to get worse and when LLM use increases it will get significantly worse. Email encryption isn t viable [4]. The PGP web of trust never really worked well as it s too difficult for most users. The amount of AI generated content that s being recommended to users on platforms like YouTube and Facebook is steadily increasing and the amount of LLM generated commentary that purports to be from real people on Twitter and Facebook is also increasing. Here s an informative blog post by Erich Schubert about this [5]. Potential Solutions Surrender? One option and possibly the default option is to surrender to this and just let everything we built on the Internet over decades get destroyed. Whether to surrender is a decision that can be made on a per-service basis. Twitter is pretty much useless anyway, I quit Twitter because Elon deliberately made it suck [6]. In my opinion this is not surrendering to what s being done there, I m just stopping wasting time on it and using better options. I used to have about 300 followers on Twitter and I don t think that many of them would ever choose to stop following me, so I presume that about 1/3 of the people following me have decided to totally quit Twitter and delete their accounts. I also presume that some of the remainder have done the same as me and just kept a mostly inactive account. If Elon suddenly stopped being a stupid asshole it probably wouldn t change anything as the value of the system was connections to others. Some people will consider my abandonment of Twitter as surrender and I accept that it s not an unreasonable opinion. I think that the possibly 100 Twitter followers of mine who deleted their accounts surrendered. Facebook has been becoming a worse service, it s business model is becoming increasingly exploitative and it s interface is designed to be addictive. It s probably best avoided unless you really need it. The only good thing about Facebook at the moment is that Facebook Marketplace doesn t take a cut on sales and there are some really good deals on computers if you know what to look for. Unfortunately Facebook has a large number of users who are from marginalised communities and have no other alternatives for communication. It would be good to get them migrated to other platforms. We could just give up on a lot of general communications services and have everyone accept that good content is drowned out by rubbish and have the Internet become divided between people who accept the rubbish and those who cease using large portions of the Internet environment to avoid it. Using Non Commercial Services Lemmy is a good FOSS federated alternative to Reddit which also covers some of the uses of Facebook. It needs more users to get critical mass but is still quite usable. A post that might get a dozen comments on Reddit may get 1 comment on Lemmy but that one comment will be a good one. Reddit doesn t appear to be attacked much by LLM generated content at least not yet. Even if the Reddit model proves to be resilient to LLM attack the Lemmy software can be used to replace some things that are done on Facebook, Mastodon is a good FOSS federated replacement for Twitter, it has a decent user-base including some VIPs. While it is aimed at the Twitter use case it can also cover a significant part of the Facebook use case. There are some other FOSS social media programs which could take over other parts of the commercial social media environment. Generally commercially run Internet services will have a financial incentive to allow the problems to get worse so we need to rely on FOSS software, non-commercial implementations, and government services. Web Search For a long time Google has had a monopoly on web search, but now they default to including an AI Overview at the start of the results which is sometimes useful but also sometimes very wrong. You can use the search URL https://www.google.com/search?q=%s&udm=web to get google results without rubbish. But I presume that they will break that if it gets too popular. Searxng is a AGPL licensed metasearch engine that aggregates results from other engines, here s the Searxng source [7] and here s a list of Searxng instances if you want to try one [8]. Even using meta search engines like Searxng won t help if the original data is overloaded with spam, but alleviating the problem is a good temporary measure. Web of Trust for the Web? I ve idly considered the possibility of having some sort of rating system for web pages that uses a web of trust so that you can securely use trust ratings of friends of friends etc. But given all the difficulties in using a web of trust for signing GPG key for software developers (the demographic that is most skilled at doing such things) it doesn t seem viable. Should we surrender the idea of having a usable public web? In the early days of the web (before Google) it was standard practice to rely on recommendations from other people or from trusted sites to find other sites, that could be considered to be an informal web of trust. We could go back to that sort of usage pattern if Google and many of the big sites get overwhelmed by LLM generated spam. Wikipedia I believe that Wikipedia will be at the front lines of this battle. It s model has always included anonymous contributions. Benjamin Mako Hill wrote an interesting blog post about research he did with Kaylea Champion into Wikipedia pages on taboo topics which have a larger portion of contributors choosing to be anonymous than non-taboo pages [9]. Wikipedia also has a long history of being abused for various reasons, one that I witnessed was someone putting false content into Wikipedia pages to immediately cite them in support of their facebook arguments. That sort of thing can be dealt with at human scale but a large scale attack by bots is a different problem to solve. Also with the recent developments in AI developing multiple web sites entirely populated for the purpose of supporting one fake entry in Wikipedia is plausible. The upside of these attacks that I predict is that they will attract the attention of all the people who have skills related to developing counter-measures. While LLM bots are filling the inboxes of publishers with rubbish and messing up the stackoverflow comments section not a lot of people are bothered, but once the attacks on Wikipedia get serious everyone will take notice. National AI Bruce Schneier and Nathan E. Sanders wrote an interesting blog post about nationalised public AI [10]. While that won t directly address this issue it will get the right technology in the hands of people who can use it in the right way. Conclusion This is going to be a difficult problem to solve, more difficult than the email spam problem we have been unable to solve after 30 years of working on it. This is also a very important problem, we are currently in an age where we have access to information that most people couldn t even dream of 30 years ago. We also have disinformation that combines some of the worst aspects of authoritarian regimes throughout history combined with the worst aspects of cult brainwashing. If we lose access to the information but the disinformation remains (or get worse) then the result will be terrible. I don t have great ideas for solving this. I have outlined some small ideas to mitigate things and I hope that others can expand on them. Please write comments with any good ideas you have, or even ideas that don t totally suck. A problem this difficult is not going to be solved in a blog comment, but a blog comment might point in the right direction.

Valhalla's Things: Ink Lightfastness Tests 2026

Posted on March 28, 2026
Tags: madeof:atoms, topic:inks
A borderless frame set on a table outdoors, with two sheets of paper a vertical half of which is covered by black paper, while the other half has lines with an ink name and a small filled rectangle, all in the ink itself.
Note
This post will be updated in the next weeks with the test results as they become available.
Note
Most of the images in this post have no real alt-text: they are all scans of the test sheet at various stages through the test, and the results visible on them are described in detail at the end of the post.
Most of the time, what people write by hand will either end up inside a notebook in a drawer or cupboard where it s well protected, or thrown in the recycling where it doesn t matter. There are times, however, when things will be exposed to light: it doesn t matter whether it s a work of artistic calligraphy that you want to frame or a passive-aggressive notice left in the atrium of a building; it is useful to know whether the work will remain legible or it will fade into nothing in a short time. A few inks are tested by the producers for lightfastness according to some established standard, a few others are declared lightfast in a generic way, but a lot come with no indication at all. Proper testing according to the standard scales requires significant equipment to precisely control the exposure, but it s significantly easier and fun to do a simple test to divide the inks into three categories:
  • suitable for framed calligraphy, i.e. it looks the same after 3 months of direct sun exposure;
  • suitable for complaining about the way your neighbours deal with the trash, i.e. still readable after 3 months of exposure;
  • not suitable for either, i.e. has faded significantly in the same time.
In the past I ve done some such tests by taping some sheets to a south-east facing window, and I ve noticed that most of the results were already apparent after a month, and there was basically no difference between two and three months of exposure, but spring equinox to summer solstice is a nice timeframe to use for such a test (and it leaves time for a second test of different materials from summer solstice to autumn equinox), so this is what I ve chosen to do this year. Rather than a window, now I have access to a south-facing covered balcony that is protected from rain but receives quite a bit of direct sun, so instead of taping sheets to the windows1 I ve prepared a sturdy cardboard panel that I can leave on a table on the balcony, hopefully safe from the rain, but well exposed to the sun. And then made a quick test, and realized that without the window glass in front, the black strip used to cover the unexposed half of the sample doesn t lay flat and lets some sun in, so I used an old cheap2 glass frame instead of the panel. The contents of an order from a fountain pen shop, spread out on a table: a couple of cheap pens, a couple bottles of ink, a converter, a small ritter sport chocolate and a bag full of 5 ml vials with 2 ml of ink each, and a thank-you note from the seller (Steffi). The next step, already in January, was mentioning in a fountain-pen enthusiasts forum that I planned such a test, and asking if people were interested in having me buy a few samples of more inks when I was buying my next pen. The word enthusiasts is probably a hint of the reason why soon afterwards I received a package with the pen I had planned to buy, its converter, and a couple dozens ink samples. And then a couple envelopes with additional samples of inks that weren t available on the shops, from said enthusiasts. Added to the inks I already had acquired since the last lightfastness test, it meant that they couldn t all fit in one single page, and thus I had some room to add some inks I had already tested: some were requests, and for others I tried to select ones that felt relevant. Since I m changing the test setup, I ve decided I should probably keep doing this until I ve tested again all of the inks I still have available. see below see below For the paper, I ve used A4 sheets of Clairefontaine Dessin Croquis 160 g/m , one of my staples that I m sure I will have available in the next years, printed with a dot pattern with a laser printer, using this pdf. And as for the pen I ve used a fresh Brause n 361 nib: loading a fountain pen with all of these inks wouldn t be a reasonable effort, and the 361 is one of the writing implements I use most anyway. I also used a glass pen to fill a couple of squares on the paper with more ink. One side of each sheet was then covered with a strip of 300 g/m black paper (also from Clairefontaine), kept in place with three dots of non-permanent two sided tape, put in the frame and set out in the sun on the morning of 2026-03-20, the day of the spring equinox.
see below While I was filling the sheet for the lightfastness tests, I decided to also prepare a second set of sheet, for a liquid resistance drop test. On each line, beside the name of the ink, I added five sets of crossing parallel lines, and let everything dry for a few days. Then I used a syringe to put a drop of a liquid on each set of lines, waited for it to be absorbed into the paper and to dry, at least overnight, but sometimes also for a day or two (life happened), and then looked at the results and did the next test. The first liquid was water, with the usual wild difference between washable and permanent inks, and all of the intermediate possibilities. The second liquid was isopropyl alcohol, and I was surprised to see that, with very few exceptions, most inks didn t change at all. I wonder whether that s related to the fact that instead of forming a drop it was absorbed almost immediately into the paper, and dried in a very short time. The third liquid was hydrogen peroxide: beside the individual results I noticed that its column yellowed visibly; I wonder whether that means that the paper I used has optical brighteners, and it will also yellow under the sun: that wouldn t be ideal, but it would also be a surprise, for paper that is acid free and sold for arts. The fourth liquid was citric acid, by mixing a bit less than a teaspoon of citric acid granules in just enough very warm water (heated to 70 C, i.e. the lowest temperature available on my kettle) to dissolve most of the acid. I forgot that I had some old PH strips until one hour after I ve put the drop on the paper, and I don t know whether something had changed, but when I did remember about them it showed a deep red between 1 and 2. I don t think I can trust those strips too much, however. This backfired badly: the drop of citric acid never dried out, but formed a sticky paste that prevented me from scanning the results, and I m not sure whether I ll do the last test, which was supposed to be household bleach. see below see below Luckily I had scanned the partial results, and they are shown here.
see below see below After one full day with plenty of sun, nothing really had changed, except possibly for a vague hint that the Herbin Bleu Myosotis may have have been a bit lighter than it started, but it may also have been a suggestion.
see below see below After three days, however, some results started to show, with the most fugitive inks starting to be visibly changed, becoming either paler or in some case duller.
see below see below And the full week showed more of that, with a few more inks starting to show visible change.
see below see below After two weeks the paper had significantly yellowed, something I did not expect from drawing paper (and which means that I will probably use a different paper when making similar tests in the future). As for the inks, there were a couple more inks with visible changes, but mostly it was more of the same as seen in the previous week.
see below see below Three weeks started to show changes in the black and most irongall inks, and of course more changes in the even less resistant inks.
see below see below Week four saw a bit more clouds and rain than the first few weeks, and there weren t big changes, but mostly more of what had already started to happen earlier.
see below see below A month didn t change much compared to the four weeks, but I did the scans for completeness, and from now on I m going to update monthly.
These are the inks I ve tested, and here I ll add notes on the results, as soon as they will be available, keeping this section updated. When nothing is mentioned, it means that there were no changes, either under the light or under the various liquids.
Lamy Sepia
Not resistant to water, the drop becomes an uniform colour spot. After one week it started to be just slightly paler, more so after three weeks.
Sheaffer Skrip Red
Not resistant to water, the drop becomes an uniform colour spot. After one week it started to be just slightly paler, more so after three weeks.
Waterman Audacious Red
Not resistant to water, the drop becomes an uniform colour spot. After three days it started to be just slightly paler, after a week visibly so. After four weeks it was very pale.
Waterman Harmonious Green
Not resistant to water, the drop becomes an uniform colour spot; the hydrogen peroxide drop looks a bit lighter than the one with just water. After one week it started to be just slightly paler, more so after three weeks. After four weeks it was very pale.
Waterman Mysterious Blue
Not resistant to water, the drop becomes an uniform colour spot; the hydrogen peroxide drop is significantly lighter and tends towards green. After two weeks it started to be just slightly paler, after three weeks it was more gray. After four weeks it was very pale.
Waterman Serenity Blue
Not resistant to water, the drop becomes an uniform colour spot; the hydrogen peroxide drop is almost completely bleached to a light yellow. After one week it started to be a bit duller. After four weeks it was paler and duller.
Visconti Blue
Not resistant to water, the drop becomes an uniform colour spot. After one week it was visibly duller, looking darker than the original. After three weeks it was duller, and lighter. After a month it was just a pale gray.
Montblanc Royal Blue
Not resistant to water, the drop becomes an uniform colour spot; the hydrogen peroxide drop is almost completely bleached to a light yellow. After one week it started to be just slightly duller, more so after two weeks. After three weeks it was also paler. After a month it was just a pale gray.
Montblanc Mystery Black
Not resistant to water, the drop becomes an uniform colour spot. After three weeks it started to be a bit paler.
Aurora Nero
Not resistant to water, the drop becomes an uniform colour spot. After three weeks it started to be a bit more brown.
Online Duft Blueberry
Not resistant to water, the drop looks very washed out, although a hint of the original shape can be guessed; the hydrogen peroxide drop is almost completely bleached to a light yellow. After one week it was visibly paler and duller. After three weeks significantly so. After a month it was a pale grey.
Diamine Forever Ink - Smoky Mauve
After a month it looked a bit more purple.
Diamine Forever Ink - Honey Pot
.
Diamine Forever Ink - Coral Blaze
.
Diamine Forever Ink - Red Ochre
.
Diamine Graphite
Not resistant to water, the drop becomes an uniform colour spot.
Diamine Rustic Brown
Not resistant to water, the drop becomes an uniform colour spot. After three weeks it started to be very slightely paler.
Diamine China Blue
Not resistant to water, the drop becomes an uniform colour spot; the hydrogen peroxide drop is almost completely bleached to a light yellow. After three weeks it started to be paler and duller.
Diamine Inkvent Purple Edition - Glacier
Not resistant to water, there is a drop of uniform colour, but it maintains a somewhat recognisable shade of the original shape. After three weeks it started to be lighter.
Fountainfeder STEVE
Not resistant to water, there is a drop of uniform colour, but it maintains a somewhat recognisable shade of the original shape. After two weeks the base colour had changed to a pink rather than purple.
Pilot Iroshizuku Syo Ro
Not resistant to water, there is a drop of uniform colour, but it maintains a somewhat recognisable shade of the original shape. After four weeks it was very slightely paler.
Pilot Iroshizuku Shin-Kai
Not resistant to water, there is a drop of uniform colour, but it maintains a somewhat recognisable shade of the original shape. After two weeks it had become lighter and more purple. After four weeks it was a purple gray.
Rohrer & Klingner IG Ebony
Not resistant to water, there is a drop of uniform colour, but it maintains a recognisable shade of the original shape; under hydrogen peroxide the shade is significantly lighter. After four weeks it was a bit lighter
KWZ IG Orange
Not resistant to water, the drop becomes an uniform colour spot; the hydrogen peroxide drop is significantly bleached to a light orange.
Kallipos.de Schwarze Eisengallus-Tinte
Water stains the paper, leaving however the original shape quite visible; is it almost completely bleached by hydrogen peroxide. After three weeks it started to be very slightely lighter.
Kallipos.de Blaue Eisengallus-Tinte
Water stains the paper, leaving however the original shape quite visible; is it almost completely bleached by hydrogen peroxide. After two weeks it had started to become lighter and more gray.
Rohrer & Klingner IG Salix
Water stains the paper, leaving however the original shape quite visible; is it almost completely bleached by hydrogen peroxide. After two weeks it had become lighter and significantly more gray. After a month it was a yellowish gray.
Rohrer & Klingner IG Scabiosa
Water stains the paper with a significant purple spot, leaving however the original shape quite visible; is is a bit bleached by hydrogen peroxide, but still quite readable.
Pelikan Edelstein Tanzanite
Not resistant to water, the drop becomes an uniform colour spot, but there is a visible trace of the original shape. After three weeks it started to be slightly paler.
Montblanc Burgundy Red
Not resistant to water, the drop becomes an uniform colour spot, with just a hint of the original shape; slightly bleached by hydrogen peroxide. After three weeks it started to be paler.
Cifra inchiostro finissimo verde alla lavanda
Not resistant to water, the drop becomes an uniform colour spot; quite bleached to a light yellowish green by hydrogen peroxide. After one week it was visibly paler. After a month it was a still readable pale trace.
Sennelier Abstract acrylic ink 917 purple
.
The Feather Pen Ink
.
Eloquentia Inchiostro nero
.
DeAtramentis Document Blue
.
DeAtramentis Document BlueGrey
.
DeAtramentis Document Brown
.
DeAtramentis Document Fuchsia
.
DeAtramentis Document Grau
.
DeAtramentis Document Green Grey
.
DeAtramentis Document Light Grey
.
DeAtramentis Document Moosgr n
.
DeAtramentis Document Orange
.
DeAtramentis Document Purpurviolett
.
DeAtramentis Document Urban Sienna
.
KWZ Sheen Machine
Not resistant to water, the drop becomes an uniform colour spot; the hydrogen peroxide bleached away the red sheen. This was one of the only two inks to react to isopropyl alcohol, which caused a pale cyan halo around the lines. After three days it was still perfectly readable, but had visibly lost some red sheen, after one week the red had completely gone and it looked very dark blue (but still shiny)
KWZ Walk over Vistula
Not resistant to water, the drop becomes an uniform colour spot. After four weeks it looked a bit darker and duller.
KWZ Warsaw Dreaming
Not resistant to water, the drop becomes an uniform colour spot. After a month it started to be a bit lighter.
Octopus Neon Violett
Water very lightly stains the paper, leaving however the original shape quite visible. The other ink that reacted to isopropyl alcohol, with a pale purple halo around the lines. After two weeks it was paler, more pink.
Octopus Write & Draw Elephant Black
.
Platinum blue black
Water stains the paper, leaving however the original shape quite visible; it is significantly bleached by hydrogen peroxide. After three weeks it started to become gray.
Pelikan 4001 Brillant-Schwarz
Not resistant to water, the drop becomes an uniform colour spot. After three weeks it was a bit more brown than black, after a month noticeably so.
Pelikan 4001 Blau-Schwarz
Water stains the paper, leaving however the original shape quite visible; it is significantly bleached by hydrogen peroxide. After three weeks it started to become gray.
Pelikan 4001 K nigsblau
Not resistant to water, the drop becomes an uniform colour spot, with just a hint of the original shape; significantly bleached by hydrogen peroxide. After three days it had started to be slightly paler. After three weeks it was significantly desaturated.
Herbin Bleu Myosotis
Not resistant to water, the drop becomes an uniform pink spot, significantly bleached by hydrogen peroxide. After three days it was already visibly paler, after one week it was a pale grey. After a month it was still somehow readable, but as a trace.
Faber Castell Royal Blue
Not resistant to water, the drop becomes an uniform colour spot, with just a hint of the original shape; significantly bleached by hydrogen peroxide. After three days it was slightly duller, after two weeks definitely so. After a month it was also quite paler.
Koh-I-Noor Fountain pen ink blue
Not resistant to water, the drop becomes an uniform colour spot, with just a hint of the original shape; significantly bleached by hydrogen peroxide. After three days it had started to be slightly paler, more so after one week when it had also turned grey. After four weeks it was very pale.
Koh-I-Noor Document Ink Blue
.
Koh-I-Noor Document Ink Black
Water leaves a very light stain, but the original shape doesn t look changed.
DeAtramentis Document Black
.
Waterman Intense Black
Not resistant to water, the drop becomes an uniform colour spot, with a trace of the original shape still visible; very lightly bleached by hydrogen peroxide. After three weeks it started to look a bit more brown, noticeably so after a month.
Herbin Perle Noir
Not resistant to water, the drop becomes an uniform colour spot, with a trace of the original shape still visible. After three weeks it started to look a bit more brown, noticeably so after a month.
Parker Quink black
Not resistant to water, the drop becomes an uniform colour spot.
Platinum Carbon black
.
Rohrer & Klingner Documentus Black
.
Sailor Pigment Kiwaguro
.
Platinum Dyestuff Red
Not resistant to water, the drop becomes an uniform colour spot; very lightly bleached by hydrogen peroxide. After three weeks it was a bit paler.
Noodler s Eternal Polar Blue
.

  1. which would be spend the day covered by mostly closed shutters anyway, because they receive quite a bit of direct sun, and we don t want that to enter the house during the summer.
  2. and thus, I hope, not especially UV-filtering.

21 March 2026

Matthew Garrett: SSH certificates and git signing

When you re looking at source code it can be helpful to have some evidence indicating who wrote it. Author tags give a surface level indication, but it turns out you can just lie and if someone isn t paying attention when merging stuff there s certainly a risk that a commit could be merged with an author field that doesn t represent reality. Account compromise can make this even worse - a PR being opened by a compromised user is going to be hard to distinguish from the authentic user. In a world where supply chain security is an increasing concern, it s easy to understand why people would want more evidence that code was actually written by the person it s attributed to. git has support for cryptographically signing commits and tags. Because git is about choice even if Linux isn t, you can do this signing with OpenPGP keys, X.509 certificates, or SSH keys. You re probably going to be unsurprised about my feelings around OpenPGP and the web of trust, and X.509 certificates are an absolute nightmare. That leaves SSH keys, but bare cryptographic keys aren t terribly helpful in isolation - you need some way to make a determination about which keys you trust. If you re using someting like GitHub you can extract that information from the set of keys associated with a user account1, but that means that a compromised GitHub account is now also a way to alter the set of trusted keys and also when was the last time you audited your keys and how certain are you that every trusted key there is still 100% under your control? Surely there s a better way.

SSH Certificates And, thankfully, there is. OpenSSH supports certificates, an SSH public key that s been signed by some trusted party and so now you can assert that it s trustworthy in some form. SSH Certificates also contain metadata in the form of Principals, a list of identities that the trusted party included in the certificate. These might simply be usernames, but they might also provide information about group membership. There s also, unsurprisingly, native support in SSH for forwarding them (using the agent forwarding protocol), so you can keep your keys on your local system, ssh into your actual dev system, and have access to them without any additional complexity. And, wonderfully, you can use them in git! Let s find out how.

Local config There s two main parameters you need to set. First,
1
git config set gpg.format ssh
because unfortunately for historical reasons all the git signing config is under the gpg namespace even if you re not using OpenPGP. Yes, this makes me sad. But you re also going to need something else. Either user.signingkey needs to be set to the path of your certificate, or you need to set gpg.ssh.defaultKeyCommand to a command that will talk to an SSH agent and find the certificate for you (this can be helpful if it s stored on a smartcard or something rather than on disk). Thankfully for you, I ve written one. It will talk to an SSH agent (either whatever s pointed at by the SSH_AUTH_SOCK environment variable or with the -agent argument), find a certificate signed with the key provided with the -ca argument, and then pass that back to git. Now you can simply pass -S to git commit and various other commands, and you ll have a signature.

Validating signatures This is a bit more annoying. Using native git tooling ends up calling out to ssh-keygen2, which validates signatures against a file in a format that looks somewhat like authorized-keys. This lets you add something like:
1
* cert-authority ssh-rsa AAAA 
which will match all principals (the wildcard) and succeed if the signature is made with a certificate that s signed by the key following cert-authority. I recommend you don t read the code that does this in git because I made that mistake myself, but it does work. Unfortunately it doesn t provide a lot of granularity around things like Does the certificate need to be valid at this specific time and Should the user only be able to modify specific files and that kind of thing, but also if you re using GitHub or GitLab you wouldn t need to do this at all because they ll just do this magically and put a verified tag against anything with a valid signature, right? Haha. No. Unfortunately while both GitHub and GitLab support using SSH certificates for authentication (so a user can t push to a repo unless they have a certificate signed by the configured CA), there s currently no way to say Trust all commits with an SSH certificate signed by this CA . I am unclear on why. So, I wrote my own. It takes a range of commits, and verifies that each one is signed with either a certificate signed by the key in CA_PUB_KEY or (optionally) an OpenPGP key provided in ALLOWED_PGP_KEYS. Why OpenPGP? Because even if you sign all of your own commits with an SSH certificate, anyone using the API or web interface will end up with their commits signed by an OpenPGP key, and if you want to have those commits validate you ll need to handle that. In any case, this should be easy enough to integrate into whatever CI pipeline you have. This is currently very much a proof of concept and I wouldn t recommend deploying it anywhere, but I am interested in merging support for additional policy around things like expiry dates or group membership.

Doing it in hardware Of course, certificates don t buy you any additional security if an attacker is able to steal your private key material - they can steal the certificate at the same time. This can be avoided on almost all modern hardware by storing the private key in a separate cryptographic coprocessor - a Trusted Platform Module on PCs, or the Secure Enclave on Macs. If you re on a Mac then Secretive has been around for some time, but things are a little harder on Windows and Linux - there s various things you can do with PKCS#11 but you ll hate yourself even more than you ll hate me for suggesting it in the first place, and there s ssh-tpm-agent except it s Linux only and quite tied to Linux. So, obviously, I wrote my own. This makes use of the go-attestation library my team at Google wrote, and is able to generate TPM-backed keys and export them over the SSH agent protocol. It s also able to proxy requests back to an existing agent, so you can just have it take care of your TPM-backed keys and continue using your existing agent for everything else. In theory it should also work on Windows3 but this is all in preparation for a talk I only found out I was giving about two weeks beforehand, so I haven t actually had time to test anything other than that it builds. And, delightfully, because the agent protocol doesn t care about where the keys are actually stored, this still works just fine with forwarding - you can ssh into a remote system and sign something using a private key that s stored in your local TPM or Secure Enclave. Remote use can be as transparent as local use.

Wait, attestation? Ah yes you may be wondering why I m using go-attestation and why the term attestation is in my agent s name. It s because when I m generating the key I m also generating all the artifacts required to prove that the key was generated on a particular TPM. I haven t actually implemented the other end of that yet, but if implemented this would allow you to verify that a key was generated in hardware before you issue it with an SSH certificate - and in an age of agentic bots accidentally exfiltrating whatever they find on disk, that gives you a lot more confidence that a commit was signed on hardware you own.

Conclusion Using SSH certificates for git commit signing is great - the tooling is a bit rough but otherwise they re basically better than every other alternative, and also if you already have infrastructure for issuing SSH certificates then you can just reuse it4 and everyone wins.

  1. Did you know you can just download people s SSH pubkeys from github from https://github.com/<username>.keys? Now you do
  2. Yes it is somewhat confusing that the keygen command does things other than generate keys
  3. This is more difficult than it sounds
  4. And if you don t, by implementing this you now have infrastructure for issuing SSH certificates and can use that for SSH authentication as well.

19 March 2026

Otto Kek l inen: Automated security validation: How 7,000+ tests shaped MariaDB's new AppArmor profile

Featured image of post Automated security validation: How 7,000+ tests shaped MariaDB's new AppArmor profileLinux kernel security modules provide a good additional layer of security around individual programs by restricting what they are allowed to do, and at best block and detect zero-day security vulnerabilities as soon as anyone tries to exploit them, long before they are widely known and reported. However, the challenge is how to create these security profiles without accidentally also blocking legitimate actions. For MariaDB in Debian and Ubuntu, a new AppArmor profile was recently created by leveraging the extensive test suite with 7000+ tests, giving good confidence that AppArmor is unlikely to yield false positive alerts with it. AppArmor is a Mandatory Access Control (MAC) system, meaning that each process controlled by AppArmor has a sort of an allowlist called profile that defines all capabilities and file paths a program can access. If a program tries to do something not covered by the rules in its AppArmor profile, the action will be denied on the Linux kernel level and a warning logged in the system journal. This additional security layer is valuable because even if a malicious user found a security vulnerability some day in the future, the AppArmor profile severely restricts the ability to exploit it and gain access to the operating system. AppArmor was originally developed by Novell for use in SUSE Linux, but nowadays the main driver is Canonical and AppArmor is extensively used in Ubuntu and Debian, and many of their derivatives (e.g. Linux Mint, Pop!_OS, Zorin OS) and in Arch. AppArmor s benefit compared to the main alternative SELinux (used mainly in the RedHat/Fedora ecosystem) is that AppArmor is easier to manage. AppArmor continues to be actively developed, with new major version 5.0 expected to arrive soon. I also have some personal history contributing some notification handler scripts in Python and I also created the website that AppArmor.net still runs.

Regular review of denials in the system log required Any system administrator using Debian/Ubuntu needs to know how to check for AppArmor denials. The point of using AppArmor is kind of moot if nobody is checking the denials. When AppArmor blocks an action, it logs the event to the system audit or kernel logs. Understanding these logs is crucial for troubleshooting custom configurations or identifying potential security incidents. To view recent denials, check /var/log/audit/audit.log or run journalctl -ke --grep=apparmor. A typical denial entry for MariaDB will look like this (split across multiple lines for legibility):
msg=audit(1700000000.123:456): apparmor="DENIED" operation="open"
profile="/usr/sbin/mariadbd" name="/custom/data/path/test.ibd" pid=1234
comm="mariadbd" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
How to interpret this output:
  • msg=audit( ): The audit timestamp and event serial number.
  • apparmor= DENIED : Indicates AppArmor blocked the action.
  • operation: The action being attempted (e.g., open, mknod, file_mmap, file_perm).
  • profile: The specific AppArmor profile that triggered the denial (in this case the /usr/sbin/mariadbd profile).
  • name: The file path or resource that was blocked. In the example above, a custom data path was denied access because it wasn t defined in the profile s allowed abstractions.
  • comm: The command name that triggered the denial (here mariadbd).
  • requested_mask / denied_mask: Shows the permissions requested (e.g., r for read, w for write).
  • pid: The process ID.
  • fsuid: The user ID of the process attempting the action.
  • ouid: The owner user ID of the target file.
If an action seems legit and should not be denied, the sysadmin needs to update the existing rules at /etc/apparmor.d/ or drop a local customization file in at /etc/apparmor.d/local/. If the denied action looks malicious, the sysadmin should start a security investigation and if needed report a suspected zero-day vulnerability to the upstream software vendor (e.g. Ubuntu customers to Canonical, or MariaDB customers to MariaDB).

AppArmor in MariaDB - not a novel thing, and not easy to implement well Based on old bug reports, there was an AppArmor profile already back in 2011, but it was removed in MariaDB 5.1.56 due to backlash from users running into various issues. A new profile was created in 2015, but kept opt-in only due to the risk of side effects. It likely had very few users and saw minimal maintenance, getting only a handful of updates in the past 10 years. The primary challenge in using mandatory access control systems with MariaDB lies in the sheer breadth of MariaDB s operational footprint with diverse storage engines and plugins. Also the code base in MariaDB assumes that system calls to Linux always work which they do under normal circumstances and do not handle errors well if AppArmor suddenly denies a system call. MariaDB is also a large and complex piece of software to run and operate, and it can be very challenging for system administrators to root-cause that a misbehavior in their system was due to AppArmor blocking a single syscall. Ironically, AppArmor is most beneficial exactly due to the same reasons for MariaDB. The larger and more complex a software is, the larger are the odds of a security vulnerability arising between the various components. And AppArmor profile helps reduce this complexity down to a single access list. Over the years there has been users requesting to get the AppArmor profile back, such as Debian Bug#875890 since 2017. The need was raised recently again by the Ubuntu security team during the MariaDB Ubuntu main inclusion review in 2025, which prompted a renewed effort by Debian/Ubuntu developers, mainly myself and Aquila Macedo, with upstream MariaDB assistance from Daniel Black.

A fresh approach: leverage the MariaDB test suite for automated testing and the open source community for reviews The key to creating a robust AppArmor profile is the ability to know in detail what is expected and normal behavior of the system. One could in theory read all of the source code in MariaDB, but with over two million lines, it is of course not feasible in practice. However, MariaDB does have a very extensive 7000+ test suite, and running it should trigger most code paths in MariaDB. Utilizing the test suite was key in creating the new AppArmor profile for MariaDB: we installed MariaDB on a Ubuntu system, enabled AppArmor in complain mode and iterated on the allowlist by running the full mariadb-test-run with all MariaDB plugins and features enabled until we had a comprehensive yet clean list of rules. To be extra diligent, we also reworked the autopkgtest for MariaDB in Debian and Ubuntu CI systems to run with the AppArmor profile enabled and to print all AppArmor notices at the end of the run, making it easy to detect now and in the future if the MariaDB test suite triggers any AppArmor denials. If any test fails, the release would not get promoted further, protecting users from regressions. While developing and triggering manual test runs we used the maximal achievable test suite with 7177 tests. The test is however so extensive it takes over two hours to run, and it also has some brittle tests, so the standard test run in Debian and Ubuntu autopkgtest is limited just to MariaDB s main suite with about 1000 tests. Having some tests fail while testing the AppArmor profile was not a problem, because we didn t need all the tests to pass we merely needed them to run as many code paths as possible to see if they run any system calls not accounted for in the AppArmor profile. Note that extending the profile was not just mechanical copying of log messages to the profile. For example, even though a couple of tests involve running the dash shell, we decided to not allow it, as it opens too much of a path for a potential exploit to access the operating system. The result of this effort is a modernized, robust profile that is now production-ready. Those interested in the exact technical details can read the Debian Bug#1130272 and the Merge Request discussions at salsa.debian.org, which hosts the Debian packaging source code.

Now available in Debian unstable, soon Ubuntu feedback welcome! Even though the file is just 200 lines long, the work to craft it spanned several weeks. To minimize risk we also did a gradual rollout by releasing the first new profile version in complain mode, so AppArmor only logs would-be-denials without blocking anything. The AppArmor profile was switched to enforce mode only in the very latest MariaDB revision 1:11.8.6-4 in Debian, and a NEWS item issued to help increase user awareness of this change. It is also slated for the upcoming Ubuntu 26.04 Resolute Raccoon release next month, providing out-of-the-box hardening for the wider ecosystem. While automated testing is extensive, it cannot simulate everything. Most notably various complicated replication topologies and all Galera setups are likely not covered. Thus, I am calling on the community to deploy this profile and monitor for any audit denials in the kernel logs. If you encounter unexpected behavior or legitimate denials, please submit a bug report via the Debian Bug Tracking System. To ensure you are running the latest MariaDB version, run apt install --update --yes mariadb-server. To view the latest profile rules, run cat /etc/apparmor.d/mariadbd and to see if it is enforced review the output of aa-status. To quickly check if there were any AppArmor denials, simply run journalctl -k grep -i apparmor grep -i mariadb.

Systemd hardening also adopted as security features keep evolving For those interested in MariaDB security hardening, note that also new systemd hardening options were rolled out in Debian/Ubuntu recently. Note that Debian and Ubuntu are mainly volunteer-driven open source developer communities, and if you find this topic interesting and you think you have the necessary skills, feel free to submit your improvement ideas as Merge Requests at salsa.debian.org/mariadb-team. If your improvement suggestions are not Debian/Ubuntu specific, please submit them directly to upstream at GitHub.com/MariaDB.

16 March 2026

Dirk Eddelbuettel: RcppClassicExamples 0.1.4 on CRAN: Maintenance

Another minor maintenance release version 0.1.4 of package RcppClassicExamples arrived earlier today on CRAN, and has been built for r2u. This package illustrates usage of the old and otherwise deprecated initial Rcpp API which no new projects should use as the normal and current Rcpp API is so much better. This release, the first in two and half years, mostly aids Rcpp in moving from Rf_error() to Rcpp::stop() for better behaviour under error conditions or excections. A few other things were updated in the interim such as standard upgrade to continuous integration, use of Authors@R, and switch to static linking and an improved build to support multiple macOS architectures. No new code or features. Full details below. And as a reminder, don t use the old RcppClassic use Rcpp instead.

Changes in version 0.1.4 (2026-03-16)
  • Continuous integration has been updated several times
  • DESCRIPTION now uses Authors@R
  • Static linking is enforced, RcppClassic (>= 0.9.14) required
  • Calls to Rf_error() have been replaced with Rcpp::stop()
  • Updated versioned dependencies

Thanks to CRANberries, you can also look at a diff to the previous release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

Next.

Previous.