I find the case of the .UA country code top level domain (ccTLD) interesting simply because of the different name server secondaries they have now. Post Russian invasion, the cyber warfare peaked, and critical infrastructure like getting one side ccTLD down would be big news in anycase. Most (g/cc)TLDs are served by two (and less likely) by three or more providers. Even in those cases, not all authoritative name servers are anycasted. Take, example of .NL ccTLD name servers:

$ dig ns nl +short
ns1.dns.nl.
ns3.dns.nl.
ns4.dns.nl.

ns1.dns.nl is SIDN which also manages their registry. ns3.dns.nl is ReCodeZero/ipcom, another anycast secondary. ns4.dns.nl is CIRA, anycast secondary. That s 3 diverse, anycast networks to serve the .NL ccTLD. .DE has a bit more at name servers at 6 but only 3 seems anycasted. Now let s take a look at .UA. Hostmaster LLC is the registry operator of the .UA ccTLD since 2001.

$ dig soa ua +short
in1.ns.ua. domain-master.cctld.ua. 2025061434 1818 909 3024000 2020

Shows in1.ns.ua as primary nameserver (which can be intentionally deceptive too). I used bgp.tools for checking anycast and dns.coffee for timeline of when secondary nameserver was added. dns.coffee only has data going back till 2011 though. Let s deep dive at who s hosting each of the name servers:

in1.ns.ua by Intuix LLC

• 74.123.224.40
• 2604:ee00:0:101:0:0:0:40
• unicast
• Serving .UA since 13/12/2018.
• Company by Dmitry Kohmanyuk and Igor Sviridov who re administrative and technical contacts for .UA zone as well as the IANA DB.

ho1.ns.ua by Hostmaster LLC

• 195.47.253.1
• 2001:67c:258:0:0:0:0:1
• bgp.tools doesn t mark the prefix as anycast but basis test from various location, this is indeed anycasted (visible in atleast DE, US, UA etc.). Total POPs unknown.
• Serving .UA atleast since 2011.
• The registry themselves.

bg.ns.ua by ClouDNS

• 185.136.96.185 and 185.136.97.185
• 2a06:fb00:1:0:0:0:4:185 and 2a06:fb00:1:0:0:0:2:185
• anycast
• Serving .UA since 01/03/2022.
• atleast 62 PoPs

cz.ns.ua by NIC.cz

• 185.43.134.15
• 2001:148f:fffd:0:0:0:0:15
• anycast
• atleast 11 PoPs.
• Serving .UA since 29/03/2024. NIC.cz adding secondary server blog.
• .CZ operator.

nn.ns.ua by Netnod

• 194.58.197.4
• 2a01:3f1:c001:0:0:0:0:53
• anycast
• atleast 80 PoPs.
• Serving .UA since 01/12/2022.
• Netnod has the distinction of being one of the 13 root server operator (i.root-servers.net) and .SE operator.

pch.ns.ua by PCH

• 204.61.216.12
• 2001:500:14:6012:ad:0:0:1
• anycast
• atleast 328 POPs.
• Serving .UA atleast since 2011.
• With more than 36 years of production anycast DNS experience, two of the root name server operators and more than 172 top-level domain registries using our infrastructure, and more than 120 million resource records in service from https://www.pch.net/services/anycast.

rcz.ns.ua by RcodeZero

• 193.46.128.10
• 2a02:850:ffe0:0:0:0:0:10
• anycast
• Atleast 56 PoPs via 2 different cloud providers.
• Serving .UA since 04/02/2022.
• sister company of nic.at (.AT operator).

Some points to note

• That s 1 unicast and 6 anycast name servers with hundreds of POPs from 7 different organizations.
• Having X number of Point of Presence (POP) doesn t always mean each location is serving the .UA nameserver prefix.
• Number of POPs keeps going up or down based on operational requirements and optimizations.
• Highest concentration of DNS queries for a ccTLD would essentially originate in the country (or larger region) itself. If one of the secondaries doesn t have POP inside UA, the query might very well be served from outside the country, which can affect resolution and may even stop during outages and fiber cuts (which have become common there it seems). - Global POPs do help in faster resolutions for others/outside users though and ofcourse availability.
• Having this much diversity does lessen the chance of the ccTLD going down. Theoretically, the adversary has to bring down 7 different networks/setups before resolution starts failing (post TTLs expiry).

Read more links

• Delegation record for .UA at IANA.
• Panel discussion: Running a top level domain in times of war at CyberChess 2023 with .UA operators post the invasion.
• History of .UA domain on Hostmaster LLCs website.
• .UA ccTLD overview talk presentation at ENOG 2011.
• RIPE NCC s DNS Monitoring Service (DNSMON) page for .UA.

A long, long time ago I have a few pictures on this blog, mostly in earlier years, because even with small pictures, the git repository became 80MiB soon this is not much in absolute terms, but the actual Markdown/Haskell/CSS/HTML total size is tiny compared to the picture, PDFs and fonts. I realised I need a better solution, probably about ten years ago, and that I should investigate git-annex. Then time passed, and I heard about git-lfs, so I thought that s the way forward. Now, I recently got interested again into doing something about this repository, and started researching.

Detour: git-lfs I was sure that git-lfs, being supported by large providers, would be the modern solution. But to my surprise, git-lfs is very server centric, which in hindsight makes sense, but for a home setup, it s not very good. Maybe I misunderstood, but git-lfs is more a protocol/method for a forge to store files, rather than an end-user solution. But then you need to backup those files separately (together with the rest of the forge), or implement another way of safeguarding them. Further details such as the fact that it keeps two copies of the files (one in the actual checked-out tree, one in internal storage) means it s not a good solution. Well, for my blog yes, but not in general. Then posts on Reddit about horror stories people being locked out of github due to quota, as an example, or this Stack Overflow post about git-lfs constraining how one uses git, convinced me that s not what I want. To each their own, but not for me I might want to push this blog s repo to github, but I definitely wouldn t want in that case to pay for github storage for my blog images (which are copies, not originals). And yes, even in 2025, those quotas are real GitHub limits and I agree with GitHub, storage and large bandwidth can t be free.

Back to the future: git-annex So back to git-annex. I thought it s going to be a simple thing, but oh boy, was I wrong. It took me half a week of continuous (well, in free time) reading and discussions with LLMs to understand a bit how it works. I think, honestly, it s a bit too complex, which is why the workflows page lists seven (!) levels of workflow complexity, from fully-managed, to fully-manual. IMHO, respect to the author for the awesome tool, but if you need a web app to help you manage git, it hints that the tool is too complex. I made the mistake of running git annex sync once, to realise it actually starts pushing to my upstream repo and creating new branches and whatnot, so after enough reading, I settled on workflow 6/7, since I don t want another tool to manage my git history. Maybe I m an outlier here, but everything automatic is a bit too much for me. Once you do managed yourself how git-annex works (on the surface, at least), it is a pretty cool thing. It uses a git-annex git branch to store metainformation, and that is relatively clean. If you do run git annex sync, it creates some extra branches, which I don t like, but meh.

Trick question: what is a remote? One of the most confusing things about git-annex was understanding its remote concept. I thought a remote is a place where you replicate your data. But not, that s a special remote. A normal remote is a git remote, but which is expected to be git/ssh/with command line access. So if you have a git+ssh remote, git-annex will not only try to push it s above-mentioned branch, but also copy the files. If such a remote is on a forge that doesn t support git-annex, then it will complain and get confused. Of course, if you read the extensive docs, you just do git config remote.<name>.annex-ignore true, and it will understand that it should not sync to it. But, aside, from this case, git-annex expects that all checkouts and clones of the repository are both metadata and data. And if you do any annex commands in them, all other clones will know about them! This can be unexpected, and you find people complaining about it, but nowadays there s a solution:

git clone   dir && cd dir
git config annex.private true
git annex init "temp copy"

This is important. Any leaf git clone must be followed by that annex.private true config, especially on CI/CD machines. Honestly, I don t understand why by default clones should be official data stores, but it is what it is. I settled on not making any of my checkouts stable , but only the actual storage places. Except those are not git repositories, but just git-annex storage things. I.e., special remotes. Is it confusing enough yet ?

Special remotes The special remotes, as said, is what I expected to be the normal git annex remotes, i.e. places where the data is stored. But well, they exist, and while I m only using a couple simple ones, there is a large number of them. Among the interesting ones: git-lfs, a remote that allows also storing the git repository itself (git-remote-annex), although I m bit confused about this one, and most of the common storage providers via the rclone remote. Plus, all of the special remotes support encryption, so this is a really neat way to store your files across a large number of things, and handle replication, number of copies, from which copy to retrieve, etc. as you with.

And many of other features git-annex has tons of other features, so to some extent, the sky s the limit. Automatic selection of what to add git it vs plain git, encryption handling, number of copies, clusters, computed files, etc. etc. etc. I still think it s cool but too complex, though!

Uses Aside from my blog post, of course. I ve seen blog posts/comments about people using git-annex to track/store their photo collection, and I could see very well how the remote encrypted repos any of the services supported by rclone could be an N+2 copy or so. For me, tracking photos would be a bit too tedious, but it could maybe work after more research. A more practical thing would probably be replicating my local movie collection (all legal, to be clear) better than just run rsync from time to time and tracking the large files in it via git-annex. That s an exercise for another day, though, once I get more mileage with it - my blog pictures are copies, so I don t care much if they get lost, but movies are primary online copies, and I don t want to re-dump the discs. Anyway, for later.

Migrating to git-annex Migrating here means ending in a state where all large files are in git-annex, and the plain git repo is small. Just moving the files to git annex at the current head doesn t remove them from history, so your git repository is still large; it won t grow in the future, but remains with old size (and contains the large files in its history). In my mind, a nice migration would be: run a custom command, and all the history is migrated to git-annex, so I can go back in time and the still use git-annex. I na vely expected this would be easy and already available, only to find comments on the git-annex site with unsure git-filter-branch calls and some web discussions. This is the discussion on the git annex website, but it didn t make me confident it would do the right thing. But that discussion is now 8 years old. Surely in 2025, with git-filter-repo, it s easier? And, maybe I m missing something, but it is not. Not from the point of view of plain git, that s easy, but because interacting with git-annex, which stores its data in git itself, so doing this properly across successive steps of a repo (when replaying the commits) is, I think, not well defined behaviour. So I was stuck here for a few days, until I got an epiphany: As I m going to rewrite the repository, of course I m keeping a copy of it from before git-annex. If so, I don t need the history, back in time, to be correct in the sense of being able to retrieve the binary files too. It just needs to be correct from the point of view of the actual Markdown and Haskell files that represent the meat of the blog. This simplified the problem a lot. At first, I wanted to just skip these files, but this could also drop commits (git-filter-repo, by default, drops the commits if they re empty), and removing the files loses information - when they were added, what were the paths, etc. So instead I came up with a rather clever idea, if I might say so: since git-annex replaces files with symlinks already, just replace the files with symlinks in the whole history, except symlinks that are dangling (to represent the fact that files are missing). One could also use empty files, but empty files are more valid in a sense than dangling symlinks, hence why I settled on those. Doing this with git-filter-repo is easy, in newer versions, with the new --file-info-callback. Here is the simple code I used:

import os
import os.path
import pathlib

SKIP_EXTENSIONS= 'jpg', 'jpeg', 'png', 'pdf', 'woff', 'woff2' 
FILE_MODES =  b"100644", b"100755" 
SYMLINK_MODE = b"120000"

fas_string = filename.decode()
path = pathlib.PurePosixPath(fas_string)
ext = path.suffix.removeprefix('.')

if ext not in SKIP_EXTENSIONS:
  return (filename, mode, blob_id)

if mode not in FILE_MODES:
  return (filename, mode, blob_id)

print(f"Replacing ' filename ' (extension '. ext ') in  os.getcwd() ")

symlink_target = '/none/binary-file-removed-from-git-history'.encode()
new_blob_id = value.insert_file_with_contents(symlink_target)
return (filename, SYMLINK_MODE, new_blob_id)

This goes and replaces files with a symlink to nowhere, but the symlink should explain why it s dangling. Then later renames or moving the files around work naturally , as the rename/mv doesn t care about file contents. Then, when the filtering is done via:

git-filter-repo --file-info-callback <(cat ~/filter-big.py ) --force

It is easy to onboard to git annex:

• remove all dangling symlinks
• copy the (binary) files from the original repository
• since they re named the same, and in the same places, git sees a type change
• then simply run git annex add on those files

For me it was easy as all such files were in a few directories, so just copying those directories back, a few git-annex add commands, and done. Of course, then adding a few rsync remotes, git annex copy --to, and the repository was ready. Well, I also found a bug in my own Hakyll setup: on a fresh clone, when the large files are just dangling symlinks, the builder doesn t complain, just ignores the images. Will have to fix.

Other resources This is a blog that I read at the beginning, and I found it very useful as an intro: https://switowski.com/blog/git-annex/. It didn t help me understand how it works under the covers, but it is well written. The author does use the sync command though, which is too magic for me, but also agrees about its complexity

The proof is in the pudding And now, for the actual first image to be added that never lived in the old plain git repository. It s not full-res/full-size, it s cropped a bit on the bottom. Earlier in the year, I went to Paris for a very brief work trip, and I walked around a bit it was more beautiful than what I remembered from way way back. So a bit random selection of a picture, but here it is:

Un bateau sur la Seine

Enjoy!

I recently wrote about How to Use SSH with FIDO2/U2F Security Keys, which I now use on almost all of my machines. The last one that needed this was my Raspberry Pi hooked up to my DEC vt510 terminal and IBM mechanical keyboard. Yes I do still use that setup! To my surprise, generating a key on it failed. I very quickly saw that /dev/hidraw0 had incorrect permissions, accessible only to root. On other machines, it looks like this:

crw-rw----+ 1 root root 243, 16 May 24 16:47 /dev/hidraw16

And, if I run getfacl on it, I see:

# file: dev/hidraw16
# owner: root
# group: root
user::rw-
user:jgoerzen:rw-
group::---
mask::rw-
other::---

Yes, something was setting an ACL on it. Thus began to saga to figure out what was doing that. Firing up inotifywatch, I saw it was systemd-udevd or its udev-worker. But cranking up logging on that to maximum only showed me that uaccess was somehow doing this. I started digging. uaccess turned out to be almost entirely undocumented. People say to use it, but there s no description of what it does or how. Its purpose appears to be to grant access to devices to those logged in to a machine by dynamically adding them to ACLs for devices. OK, that s a nice goal, but why was machine A doing this and not machine B? I dug some more. I came across a hint that uaccess may only do that for a seat . A seat? I ve not heard of that in Linux before. Turns out there s some information (older and newer) about this out there. Sure enough, on the machine with KDE, loginctl list-sessions shows me on seat0, but on the machine where I log in from ttyUSB0, it shows an empty seat. But how to make myself part of the seat? I tried various udev rules to add the seat or master-of-seat tags, but nothing made any difference. I finally gave up and did the old-fashioned rule to just make it work already:

TAG=="security-device",SUBSYSTEM=="hidraw",GROUP="mygroup"

I still don t know how to teach logind to add a seat for ttyUSB0, but oh well. At least I learned something. An annoying something, but hey. This all had a laudable goal, but when there are so many layers of indirection, poorly documented, with poor logging, it gets pretty annoying.

Like each month, have a look at the work funded by Freexian s Debian LTS offering.

Debian LTS contributors In May, 22 contributors have been paid to work on Debian LTS, their reports are available:

• Abhijith PA did 8.0h (out of 0.0h assigned and 8.0h from previous period).
• Adrian Bunk did 26.0h (out of 26.0h assigned).
• Andreas Henriksson did 1.0h (out of 15.0h assigned and 3.0h from previous period), thus carrying over 17.0h to the next month.
• Andrej Shadura did 3.0h (out of 10.0h assigned), thus carrying over 7.0h to the next month.
• Bastien Roucari s did 20.0h (out of 20.0h assigned).
• Ben Hutchings did 8.0h (out of 20.0h assigned and 4.0h from previous period), thus carrying over 16.0h to the next month.
• Carlos Henrique Lima Melara did 12.0h (out of 11.0h assigned and 1.0h from previous period).
• Chris Lamb did 15.5h (out of 0.0h assigned and 15.5h from previous period).
• Daniel Leidert did 25.0h (out of 26.0h assigned), thus carrying over 1.0h to the next month.
• Emilio Pozuelo Monfort did 21.0h (out of 16.75h assigned and 11.0h from previous period), thus carrying over 6.75h to the next month.
• Guilhem Moulin did 11.5h (out of 8.5h assigned and 6.5h from previous period), thus carrying over 3.5h to the next month.
• Jochen Sprickerhof did 3.5h (out of 8.75h assigned and 17.5h from previous period), thus carrying over 22.75h to the next month.
• Lee Garrett did 26.0h (out of 12.75h assigned and 13.25h from previous period).
• Lucas Kanashiro did 20.0h (out of 18.0h assigned and 2.0h from previous period).
• Markus Koschany did 20.0h (out of 26.25h assigned), thus carrying over 6.25h to the next month.
• Roberto C. S nchez did 20.75h (out of 24.0h assigned), thus carrying over 3.25h to the next month.
• Santiago Ruano Rinc n did 15.0h (out of 12.5h assigned and 2.5h from previous period).
• Sean Whitton did 6.25h (out of 6.0h assigned and 2.0h from previous period), thus carrying over 1.75h to the next month.
• Sylvain Beucler did 26.25h (out of 26.25h assigned).
• Thorsten Alteholz did 15.0h (out of 15.0h assigned).
• Tobias Frost did 12.0h (out of 12.0h assigned).
• Utkarsh Gupta did 1.0h (out of 15.0h assigned), thus carrying over 14.0h to the next month.

Evolution of the situation In May, we released 54 DLAs. The LTS Team was particularly active in May, publishing a higher than normal number of advisories, as well as helping with a wide range of updates to packages in stable and unstable, plus some other interesting work. We are also pleased to welcome several updates from contributors outside the regular team.

• Notable security updates:
  • containerd, prepared by Andreas Henriksson, fixes a vulnerability that could cause containers launched as non-root users to be run as root
  • libapache2-mod-auth-openidc, prepared by Moritz Schlarb, fixes a vulnerability which could allow an attacker to crash an Apache web server with libapache2-mod-auth-openidc installed
  • request-tracker4, prepared by Andrew Ruthven, fixes multiple vulnerabilities which could result in information disclosure, cross-site scripting and use of weak encryption for S/MIME emails
  • postgresql-13, prepared by Bastien Roucari s, fixes an application crash vulnerability that could affect the server or applications using libpq
  • dropbear, prepared by Guilhem Moulin, fixes a vulnerability which could potentially result in execution of arbitrary shell commands
  • openjdk-17, openjdk-11, prepared by Thorsten Glaser, fixes several vulnerabilities, which include denial of service, information disclosure or bypass of sandbox restrictions
  • glibc, prepared by Sean Whitton, fixes a privilege escalation vulnerability
• Notable non-security updates:
  • wireless-regdb, prepared by Ben Hutchings, updates information reflecting changes to radio regulations in many countries

This month s contributions from outside the regular team include the libapache2-mod-auth-openidc update mentioned above, prepared by Moritz Schlarb (the maintainer of the package); the update of request-tracker4, prepared by Andrew Ruthven (the maintainer of the package); and the updates of openjdk-17 and openjdk-11, also noted above, prepared by Thorsten Glaser. Additionally, LTS Team members contributed stable updates of the following packages:

• rubygems and yelp/yelp-xsl, prepared by Lucas Kanashiro
• simplesamlphp, prepared by Tobias Frost
• libbson-xs-perl, prepared by Roberto C. S nchez
• fossil, prepared by Sylvain Beucler
• setuptools and mydumper, prepared by Lee Garrett
• redis and webpy, prepared by Adrian Bunk
• xrdp, prepared by Abhijith PA
• tcpdf, prepared by Santiago Ruano Rinc n
• kmail-account-wizard, prepared by Thorsten Alteholz

Other contributions were also made by LTS Team members to packages in unstable:

• proftpd-dfsg DEP-8 tests (autopkgtests) were provided to the maintainer, prepared by Lucas Kanashiro
• a regular upload of libsoup2.4, prepared by Sean Whitton
• a regular upload of setuptools, prepared by Lee Garrett

Freexian, the entity behind the management of the Debian LTS project, has been working for some time now on the development of an advanced CI platform for Debian-based distributions, called Debusine. Recently, Debusine has reached a level of feature implementation that makes it very usable. Some members of the LTS Team have been using Debusine informally, and during May LTS coordinator Santiago Ruano Rinc n has made a call for the team to help with testing of Debusine, and to help evaluate its suitability for the LTS Team to eventually begin using as the primary mechanism for uploading packages into Debian. Team members who have started using Debusine are providing valuable feedback to the Debusine development team, thus helping to improve the platform for all users. Actually, a number of updates, for both bullseye and bookworm, made during the month of May were handled using Debusine, e.g. rubygems s DLA-4163-1. By the way, if you are a Debian Developer, you can easily test Debusine following the instructions found at https://wiki.debian.org/DebusineDebianNet. DebConf, the annual Debian Conference, is coming up in July and, as is customary each year, the week preceding the conference will feature an event called DebCamp. The DebCamp week provides an opportunity for teams and other interested groups/individuals to meet together in person in the same venue as the conference itself, with the purpose of doing focused work, often called sprints . LTS coordinator Roberto C. S nchez has announced that the LTS Team is planning to hold a sprint primarily focused on the Debian security tracker and the associated tooling used by the LTS Team and the Debian Security Team.

Thanks to our sponsors Sponsors that joined recently are in bold.

• Platinum sponsors:
  • Toshiba Corporation (for 116 months)
  • Civil Infrastructure Platform (CIP) (for 84 months)
  • VyOS Inc (for 48 months)
• Gold sponsors:
  • Roche Diagnostics International AG (for 126 months)
  • Akamai - Linode (for 120 months)
  • Babiel GmbH (for 110 months)
  • Plat Home (for 109 months)
  • University of Oxford (for 66 months)
  • Deveryware (for 53 months)
  • EDF SA (for 38 months)
  • Dataport A R (for 13 months)
  • CERN (for 11 months)
• Silver sponsors:
  • Domeneshop AS (for 131 months)
  • Nantes M tropole (for 125 months)
  • Univention GmbH (for 117 months)
  • Universit Jean Monnet de St Etienne (for 117 months)
  • Ribbon Communications, Inc. (for 111 months)
  • Exonet B.V. (for 100 months)
  • Leibniz Rechenzentrum (for 95 months)
  • Minist re de l Europe et des Affaires trang res (for 78 months)
  • Cloudways by DigitalOcean (for 68 months)
  • Dinahosting SL (for 66 months)
  • Bauer Xcel Media Deutschland KG (for 60 months)
  • Platform.sh SAS (for 60 months)
  • Moxa Inc. (for 54 months)
  • sipgate GmbH (for 52 months)
  • OVH US LLC (for 50 months)
  • Tilburg University (for 50 months)
  • GSI Helmholtzzentrum f r Schwerionenforschung GmbH (for 41 months)
  • THINline s.r.o. (for 14 months)
  • Copenhagen Airports A/S (for 8 months)
• Bronze sponsors:
  • Evolix (for 131 months)
  • Seznam.cz, a.s. (for 131 months)
  • Intevation GmbH (for 128 months)
  • Linuxhotel GmbH (for 128 months)
  • Daevel SARL (for 127 months)
  • Bitfolk LTD (for 126 months)
  • Megaspace Internet Services GmbH (for 126 months)
  • Greenbone AG (for 125 months)
  • NUMLOG (for 125 months)
  • WinGo AG (for 124 months)
  • Entr ouvert (for 115 months)
  • Adfinis AG (for 113 months)
  • Tesorion (for 108 months)
  • Laboratoire LEGI - UMR 5519 / CNRS (for 107 months)
  • Bearstech (for 99 months)
  • LiHAS (for 99 months)
  • Catalyst IT Ltd (for 94 months)
  • Demarcq SAS (for 88 months)
  • Universit Grenoble Alpes (for 74 months)
  • TouchWeb SAS (for 66 months)
  • SPiN AG (for 63 months)
  • CoreFiling (for 59 months)
  • Institut des sciences cognitives Marc Jeannerod (for 54 months)
  • Observatoire des Sciences de l Univers de Grenoble (for 50 months)
  • Tem Innovations GmbH (for 45 months)
  • WordFinder.pro (for 44 months)
  • CNRS DT INSU R sif (for 43 months)
  • Soliton Systems K.K. (for 38 months)
  • Alter Way (for 36 months)
  • Institut Camille Jordan (for 26 months)
  • SOBIS Software GmbH (for 11 months)
  • Tuxera Inc.

Debian LTS This was my hundred-thirty-first month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. During my allocated time I uploaded or worked on:

• [DLA 4168-1] openafs security update of three CVEs related to theft of credentials, crashes or buffer overflows.
• [DLA 4196-1] kmail-account-wizard security update to fix one CVE related to a man-in-the-middle attack when using http instead of https to get some configuration.
• [DLA 4198-1] espeak-ng security update to fix five CVEs related to buffer overflow or underflow in several functions and a floating point exception. Thanks to Samuel Thibault for having a look at my debdiff.
• [#1106867] created Bookworm pu-bug for kmail-account-wizard. Thanks to Patrick Franz for having a look at my debdiff.

I also continued my to work on libxmltok and suricata. This month I also had to do some support on seger, for example to inject packages newly needed for builds. Debian ELTS This month was the eighty-second ELTS month. During my allocated time I uploaded or worked on:

• [ELA-1444-1] kmail-account-wizard security update to fix two CVEs in Buster related to a man-in-the-middle attack when using http instead of https to get some configuration. The other issue is about a misleading UI, in which the state of encryption is shown wrong.
• [ELA-1445-1] espeak-ng security update to fix five CVEs in Stretch and Buster. The issues are related to buffer overflow or underflow in several functions and a floating point exception.

All packages I worked on have been on the list of longstanding packages. For example espeak-ng has been on this list for more than nine month. I now understood that there is a reason why packages are on this list. Some parts of the software have been almost completely reworked, so that the patches need a reverse rework. For some packages this is easy, but for others this rework needs quite some time. I also continued to work on libxmltok and suricata. Debian Printing Unfortunately I didn t found any time to work on this topic. Debian Astro This month I uploaded bugfix versions of:

• indi-eqmod
• supernovas (sponsored upload)

Debian Mobcom This month I uploaded bugfix versions of:

• smstools

misc This month I uploaded bugfix versions of:

• greylist
• setserial
• mlat-client-adsbfi
• pkcs11-proxy

Thanks a lot to the Release Team who quickly handled all my unblock bugs! FTP master It is this time of the year when just a few packages arrive in NEW: it is Hard Freeze. So I enjoy this period and basically just take care of kernels or other important packages. As people seem to be more interested in discussions than in fixing RC bugs, my period of rest seems to continue for a while. So thanks for all this valuable discussions and really thanks to the few people who still take care of Trixie. This month I accepted 146 and rejected 10 packages. The overall number of packages that got accepted was 147.

GSoC 2024: Project DebMobCom

Introduction Hi, my name is Nathan Doris and this is my report on contributing to the Debian Mobile Communication project for GSoC 2024 under the guidance of my mentor, Thorsten Alteholz.

Project MobCom Debian s Mobile Communication team takes care of updating and creating packages for the open-source mobile communication software suite known as Osmocom. The ongoing development of this software stack requires constant updating and creation of packages to support this progress. The goal of GSoC 2024 DebMobCom is to update, recreate and add these Osmocom packages.

My Work Over the few months that I contributed to Debian I successfully updated/recreated more than a dozen packages for DebMobCom. A list of my accomplishments can be found here as well as on the Debian QA page.

What I Learned Before I began this journey I had very little experience with Debian packaging. By the end of GSoC not only had I learned about building packages but I also gained more knowledge regarding Linux, Debian, shell scripting, automake, C, GNU and gitlab/Salsa. As I reflect back on the past few months I am amazed at the amount and depth of information I have learned. Listed below are examples of items that I found especially important.

• Debian Packaging Workflow
  • Every package to be built relies upon the information inside the /debian folder. This is where I learned the importance of every templated file such as the control, changelog, rules, watch, *.symbols, *.install and many others. As I worked on these files I witnessed the relationship between them and the finished .deb package. From this workflow I can appreciate the amount of imagination and work that was put into this process.
• Debian Tooling
  • I learned many tools while building packages but sbuild accompanied by lintian is what I spent the most time with. Sbuild is an autobuilder that produces the final .deb packages. It also integrates with another tool, lintian, which checks each package and informs you of any errors such as policy violations, common mistakes and best practices. Thorsten s words, keep lintian happy still resonate with me after each package build.
  • Another tool I frequently require use of is quilt. Quilt is a wonderful tool that manages patches for packages. I became quite well versed in updating patches, removing and also creating my own.
• Copyright
  • I hadn t realised how important licensing was until I started contributing to Debian. I predominantly used licensecheck to extract copyright information from source code. This was a tedious task but obviously extremely important in keeping Debian packages free.

Challenges What I found particularly challenging (and enjoyable) was the amount of troubleshooting required to complete tasks. Although I spent countless hours reading documentation such as the New Maintainers Guide, Policy Manual, Wiki and manual pages it was still extremely difficult. Complex packages are not quite covered in the docs so out-of-the-box thinking is required.

Conclusion First of all I d like to say a big thank you to my mentor Thorsten. His support was critical for my success. Contributing to Debian and the overall GSoC experience was such a positive event in my life. I enjoyed all aspects of the experience. The skills I have acquired will propel me forward into further open-source activities and communities. As I continue to contribute to this project my goal of becoming a Debian Maintainer is becoming a reality.

Welcome to our 5th report from the Reproducible Builds project in 2025! Our monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. If you are interested in contributing to the Reproducible Builds project, please do visit the Contribute page on our website. In this report:

1. Security audit of Reproducible Builds tools published
2. When good pseudorandom numbers go bad
3. Academic articles
4. Distribution work
5. diffoscope and disorderfs
6. Website updates
7. Reproducibility testing framework
8. Upstream patches

---

Security audit of Reproducible Builds tools published The Open Technology Fund s (OTF) security partner Security Research Labs recently an conducted audit of some specific parts of tools developed by Reproducible Builds. This form of security audit, sometimes called a whitebox audit, is a form testing in which auditors have complete knowledge of the item being tested. They auditors assessed the various codebases for resilience against hacking, with key areas including differential report formats in diffoscope, common client web attacks, command injection, privilege management, hidden modifications in the build process and attack vectors that might enable denials of service. The audit focused on three core Reproducible Builds tools: diffoscope, a Python application that unpacks archives of files and directories and transforms their binary formats into human-readable form in order to compare them; strip-nondeterminism, a Perl program that improves reproducibility by stripping out non-deterministic information such as timestamps or other elements introduced during packaging; and reprotest, a Python application that builds source code multiple times in various environments in order to to test reproducibility. OTF s announcement contains more of an overview of the audit, and the full 24-page report is available in PDF form as well.

When good pseudorandom numbers go bad Danielle Navarro published an interesting and amusing article on their blog on When good pseudorandom numbers go bad. Danielle sets the stage as follows:

[Colleagues] approached me to talk about a reproducibility issue they d been having with some R code. They d been running simulations that rely on generating samples from a multivariate normal distribution, and despite doing the prudent thing and using set.seed() to control the state of the random number generator (RNG), the results were not computationally reproducible. The same code, executed on different machines, would produce different random numbers. The numbers weren t just a little bit different in the way that we ve all wearily learned to expect when you try to force computers to do mathematics. They were painfully, brutally, catastrophically, irreproducible different. Somewhere, somehow, something broke.

Thanks to David Wheeler for posting about this article on our mailing list

Academic articles There were two scholarly articles published this month that related to reproducibility: Daniel Hugenroth and Alastair R. Beresford of the University of Cambridge in the United Kingdom and Mario Lins and Ren Mayrhofer of Johannes Kepler University in Linz, Austria published an article titled Attestable builds: compiling verifiable binaries on untrusted systems using trusted execution environments. In their paper, they:

present attestable builds, a new paradigm to provide strong source-to-binary correspondence in software artifacts. We tackle the challenge of opaque build pipelines that disconnect the trust between source code, which can be understood and audited, and the final binary artifact, which is difficult to inspect. Our system uses modern trusted execution environments (TEEs) and sandboxed build containers to provide strong guarantees that a given artifact was correctly built from a specific source code snapshot. As such it complements existing approaches like reproducible builds which typically require time-intensive modifications to existing build configurations and dependencies, and require independent parties to continuously build and verify artifacts.

The authors compare attestable builds with reproducible builds by noting an attestable build requires only minimal changes to an existing project, and offers nearly instantaneous verification of the correspondence between a given binary and the source code and build pipeline used to construct it , and proceed by determining that t he overhead (42 seconds start-up latency and 14% increase in build duration) is small in comparison to the overall build time.
Timo Pohl, Pavel Nov k, Marc Ohm and Michael Meier have published a paper called Towards Reproducibility for Software Packages in Scripting Language Ecosystems. The authors note that past research into Reproducible Builds has focused primarily on compiled languages and their ecosystems, with a further emphasis on Linux distribution packages:

However, the popular scripting language ecosystems potentially face unique issues given the systematic difference in distributed artifacts. This Systemization of Knowledge (SoK) [paper] provides an overview of existing research, aiming to highlight future directions, as well as chances to transfer existing knowledge from compiled language ecosystems. To that end, we work out key aspects in current research, systematize identified challenges for software reproducibility, and map them between the ecosystems.

Ultimately, the three authors find that the literature is sparse , focusing on few individual problems and ecosystems, and therefore identify space for more critical research.

Distribution work In Debian this month:

• Ian Jackson filed a bug against the debian-policy package in order to delve into an issue affecting Debian s support for cross-architecture compilation, multiple-architecture systems, reproducible builds SOURCE_DATE_EPOCH environment variable and the ability to recompile already-uploaded packages to Debian with a new/updated toolchain (binNMUs). Ian identifies a specific case, specifically in the libopts25-dev package, involving a manual page that had interesting downstream effects, potentially affecting backup systems. The bug generated a large number of replies, some of which have references to similar or overlapping issues, such as this one from 2016/2017.
• Chris Hofstaedtler filed a bug against the metasnap.debian.net service to note that some packages are not available in metasnap API.
• 22 reviews of Debian packages were added, 24 were updated and 11 were removed this month, all adding to our knowledge about identified issues.

Hans-Christoph Steiner of the F-Droid catalogue of open source applications for the Android platform published a blog post on Making reproducible builds visible. Noting that Reproducible builds are essential in order to have trustworthy software , Hans also mentions that F-Droid has been delivering reproducible builds since 2015 . However:

There is now a Reproducibility Status link for each app on f-droid.org, listed on every app s page. Our verification server shows or based on its build results, where means our rebuilder reproduced the same APK file and means it did not. The IzzyOnDroid repository has developed a more elaborate system of badges which displays a for each rebuilder. Additionally, there is a sketch of a five-level graph to represent some aspects about which processes were run.

Hans compares the approach with projects such as Arch Linux and Debian that provide developer-facing tools to give feedback about reproducible builds, but do not display information about reproducible builds in the user-facing interfaces like the package management GUIs.
Arnout Engelen of the NixOS project has been working on reproducing the minimal installation ISO image. This month, Arnout has successfully reproduced the build of the minimal image for the 25.05 release without relying on the binary cache. Work on also reproducing the graphical installer image is ongoing.
In openSUSE news, Bernhard M. Wiedemann posted another monthly update for their work there.
Lastly in Fedora news, Jelle van der Waa opened issues tracking reproducible issues in Haskell documentation, Qt6 recording the host kernel and R packages recording the current date. The R packages can be made reproducible with packaging changes in Fedora.

diffoscope & disorderfs diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 295, 296 and 297 to Debian:

• Don t rely on zipdetails --walk argument being available, and only add that argument on newer versions after we test for that. [ ]
• Review and merge support for NuGet packages from Omair Majid. [ ]
• Update copyright years. [ ]
• Merge support for an lzma comparator from Will Hollywood. [ ][ ]

Chris also merged an impressive changeset from Siva Mahadevan to make disorderfs more portable, especially on FreeBSD. disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues [ ]. This was then uploaded to Debian as version 0.6.0-1. Lastly, Vagrant Cascadian updated diffoscope in GNU Guix to version 296 [ ][ ] and 297 [ ][ ], and disorderfs to version 0.6.0 [ ][ ].

Website updates Once again, there were a number of improvements made to our website this month including:

• Chris Lamb:
  • Merged four or five suggestions from Guillem Jover for the GNU Autotools examples on the SOURCE_DATE_EPOCH example page [ ]
  • Incorporated a number of fixes for the JavaScript SOURCE_DATE_EPOCH snippet from Sebastian Davis, which did not handle non-integer values correctly. [ ]
• David A. Wheeler:
  • Fix an apostrophe in the README.md file. [ ]
• Hans-Christoph Steiner:
  • Add the F-Droid Verification Server to the Tools page. [ ]
  • Add the Creative Commons Attribution-ShareAlike 4.0 International as the website s root LICENSE file. [ ]
  • Updated the Recording the build environment page to add a section pertaining to how F-Droid handles this. [ ]
• Jochen Sprickerhof:
  • Add Chris Hofstaedtler to the Who is involved? page. [ ]
• Sebastian Davids:
  • Fix the CoffeeScript example on the SOURCE_DATE_EPOCH page. [ ]
  • Remove the JavaScript example that uses a fixed timezone on the SOURCE_DATE_EPOCH page. [ ]

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. However, Holger Levsen posted to our mailing list this month in order to bring a wider awareness to funding issues faced by the Oregon State University (OSU) Open Source Lab (OSL). As mentioned on OSL s public post, recent changes in university funding makes our current funding model no longer sustainable [and that] unless we secure $250,000 in committed funds, the OSL will shut down later this year . As Holger notes in his post to our mailing list, the Reproducible Builds project relies on hardware nodes hosted there. Nevertheless, Lance Albertson of OSL posted an update to the funding situation later in the month with broadly positive news.
Separate to this, there were various changes to the Jenkins setup this month, which is used as the backend driver of for both tests.reproducible-builds.org and reproduce.debian.net, including:

• Migrating the central jenkins.debian.net server AMD Opteron to Intel Haswell CPUs. Thanks to IONOS for hosting this server since 2012.
• After testing it for almost ten years, the i386 architecture has been dropped from tests.reproducible-builds.org. This is because that, with the upcoming release of Debian trixie, i386 is no longer supported as a regular architecture there will be no official kernel and no Debian installer for i386 systems. As a result, a large number of nodes hosted by Infomaniak have been retooled from i386 to amd64.
• Another node, ionos17-amd64.debian.net, which is used for verifying packages for all.reproduce.debian.net (hosted by IONOS) has had its memory increased from 40 to 64GB, and the number of cores doubled to 32 as well. In addition, two nodes generously hosted by OSUOSL have had their memory doubled to 16GB.
• Lastly, we have been granted access to more riscv64 architecture boards, so now we have seven such nodes, all with 16GB memory and 4 cores that are verifying packages for riscv64.reproduce.debian.net. Many thanks to PLCT Lab, ISCAS for providing those.

Outside of this, a number of smaller changes were also made by Holger Levsen:

• reproduce.debian.net-related:
  • Only use two workers for the ppc64el architecture due to RAM size. [ ]
  • Monitor nginx_request and nginx_status with the Munin monitoring system. [ ][ ]
  • Detect various variants of network and memory errors. [ ][ ][ ][ ]
  • Add a prominent link to reproducible-builds.org. [ ]
  • Add a rebuilderd-cache-cleanup.service and run it daily via timer. [ ][ ][ ][ ][ ]
  • Be more verbose what sources are being downloaded. [ ]
  • Correctly deal with packages with an epoch in their version [ ] and deal with binNMUs versions with an epoch as well [ ][ ].
  • Document how to reschedule all other errors on all archs. [ ]
  • Misc documentation improvements. [ ][ ][ ][ ]
  • Include the $HOSTNAME variable in the rebuilderd logfiles. [ ]
  • Install the equivs package on all worker nodes. [ ][ ]
• Jenkins nodes:
  • Permit the sudo tool to fix up permission issues. [ ][ ]
  • Document how to manage diskspace with OpenStack. [ ]
  • Ignore a number of spurious monitoring errors on riscv64, FreeBSD, etc.. [ ][ ][ ][ ]
  • Install ntpsec-ntpdate (instead of ntpdate) as the former is available on Debian trixie and bookworm. [ ][ ]
  • Use the same SSH ControlPath for all nodes. [ ]
  • Make sure the munin user uses the same SSH config as the jenkins user. [ ]
• tests.reproducible-builds.org-related:
  • Disable testing of the i386 architecture. [ ][ ][ ][ ][ ]
  • Document the current disk usage. [ ][ ]
  • Address some image placement now that we only test three architectures. [ ]
  • Keep track of build performance. [ ]
• Misc:
  • Fix a (harmless) typo in the multiarch_versionskew script. [ ]

In addition, Jochen Sprickerhof made a series of changes related to reproduce.debian.net:

• Add out of memory detection to the statistics page. [ ]
• Reverse the sorting order on the statistics page. [ ][ ][ ][ ]
• Improve the spacing between statistics groups. [ ]
• Update a (hard-coded) line number in error message detection pertaining to a debrebuild line number. [ ]
• Support Debian unstable in the rebuilder-debian.sh script. [ ] ]
• Rely on rebuildctl to sync only arch-specific packages. [ ][ ]

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:
  • cmake/musescore
  • netdiscover
  • autotrace, ck, cmake, crash, cvsps, gexif, gq, gtkam, ibus-table-others, krb5-appl, ktoblzcheck-data, leafnode, lib2geom, libexif-gtk, libyui, linkloop, meson, MozillaFirefox, ncurses, notify-sharp, pcsc-acr38, pcsc-asedriveiiie-serial, pcsc-asedriveiiie-usb, pcsc-asekey, pcsc-eco5000, pcsc-reflex60, perl-Crypt-RC, python-boto3, python-gevent, python-pytest-localserver, qt6-tools, seamonkey, seq24, smictrl, sobby, solfege, urfkill, uwsgi, wsmancli, xine-lib, xkeycaps, xquarto, yast-control-center, yast-ruby-bindings and yast
  • libmfx-gen, libmfx, liboqs
• Chris Hofstaedtler:
  • #1104578 filed against jabber-muc.
• Chris Lamb:
  • #1105171 filed against golang-github-lucas-clemente-quic-go.
• Jelle van der Waa:
  • gitlab-shell
• Jochen Sprickerhof:
  • #1104965 filed against bootp.
• Zhaofeng Li:
  • Add support for --mtime and --clamp-mtime to bsdtar.
• James Addison:
  • #1105119 for python3 requested enabling a LTO-adjacent option that should improve build reproducibility.
  • #1106274 upstream fix merged for freezegun for a timezone issue causing unit tests to fail during testing.
  • Opened a pull request for tutanota in an attempt to resolve a long-standing reproducibility issue.
• Zbigniew J drzejewski-Szmek:
  • 0xFFFF: Use SOURCE_DATE_EPOCH for date in manual pages.

Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org

As I wrote in my last post, Twitter's new encrypted DM infrastructure is pretty awful. But the amount of work required to make it somewhat better isn't large.

When Juicebox is used with HSMs, it supports encrypting the communication between the client and the backend. This is handled by generating a unique keypair for each HSM. The public key is provided to the client, while the private key remains within the HSM. Even if you can see the traffic sent to the HSM, it's encrypted using the Noise protocol and so the user's encrypted secret data can't be retrieved.

But this is only useful if you know that the public key corresponds to a private key in the HSM! Right now there's no way to know this, but there's worse - the client doesn't have the public key built into it, it's supplied as a response to an API request made to Twitter's servers. Even if the current keys are associated with the HSMs, Twitter could swap them out with ones that aren't, terminate the encrypted connection at their endpoint, and then fake your query to the HSM and get the encrypted data that way. Worse, this could be done for specific targeted users, without any indication to the user that this has happened, making it almost impossible to detect in general.

This is at least partially fixable. Twitter could prove to a third party that their Juicebox keys were generated in an HSM, and the key material could be moved into clients. This makes attacking individual users more difficult (the backdoor code would need to be shipped in the public client), but can't easily help with the website version[1] even if a framework exists to analyse the clients and verify that the correct public keys are in use.

It's still worse than Signal. Use Signal.

[1] Since they could still just serve backdoored Javascript to specific users. This is, unfortunately, kind of an inherent problem when it comes to web-based clients - we don't have good frameworks to detect whether the site itself is malicious.

comments

This post is an unpublished review for Humanities and big data in Ibero-America Theory, methodology and practical applications

Digital humanities is a young though established field. It deals with different expressions in which digital data manipulation techniques can be applied and used to analyze subjects that are identified as belonging to the humanities. Although most often used to analyze different aspects of literature or social network analysis, it can also be applied to other humanistic disciplines or artistic expressions. Digital humanities employs many tools, but those categorized as big data are among the most frequently employed. This book samples different takes on digital humanities, with the particularity that it focuses on Ibero-American uses. It is worth noting that this book is the second in a series of four volumes, published or set to be published between 2022 and 2026. Being the output of a field survey, I perceive this book to be targeted towards fellow Digital Humanists people interested in applying computational methods to further understand and research topics in the humanities. It is not a technical book in the sense Computer Science people would recognize as such, but several of the presented works do benefit from understanding some technical concepts. The 12 articles (plus an introduction) that make up this book are organized in three parts: (1) Theoretical Framework presents the ideas and techniques of data science (that make up the tools for handling big data), and explores how data science can contribute to literary analysis, all while noting that many such techniques are usually frowned upon in Latin America as data science smells neoliberal ; (2) Methodological Issues looks at specific issues through the lens of how they can be applied to big data, with specific attention given to works in Spanish; and (3) Practical Applications analyzes specific Spanish works and communities based on big data techniques. Several chapters treat a recurring theme: the simultaneous resistance and appropriation of big data by humanists. For example, at least three of the chapters describe the tensions between humanism ( aesthesis ) and cold, number-oriented data analysis ( mathesis ). The analyzed works of Parts 2 and 3 are interesting and relatively easy to follow. Some inescapable ideological gleans from several word uses from the book s and series name, which refers to the Spanish-speaking regions as Ibero-America , often seen as Eurocentric, in contrast with the Latin America term much more widely used throughout the region. I will end with some notes about the specific versions of the book I reviewed. I read both an EPUB version and a print copy. The EPUB did not include links for easy navigation to footnotes, that is, the typographical superindexes are not hyperlinked to the location of the notes, so it is very impractical to try to follow them. The print version (unlike the EPUB) did not have an index, that is, the six pages before the introduction are missing from the print copy I received. For a book such as this one, not having an index hampers the ease of reading and referencing.

This post is an unpublished review for Beyond data poisoning in federated learning

The current boom of artificial intelligence (AI) is based upon neural networks (NNs). In order for these to be useful, the network has to undergo a machine learning (ML) process: work over a series of inputs, and adjust the inner weights of the connections between neurons so that each of the data samples the network was trained on produces the right set of labels for each item. Federated learning (FL) appeared as a reaction given the data centralization power that traditional ML provides: instead of centrally controlling the whole training data, various different actors analyze disjoint subsets of data, and provide only the results of this analysis, thus increasing privacy while analyzing a large dataset. Finally, given multiple actors are involved in FL, how hard is it for a hostile actor to provide data that will confuse the NN, instead of helping it reach better performance? This kind of attack is termed a poisoning attack, and is the main focus of this paper. The authors set out to research how effective can a hyperdimensional data poisoning attack (HDPA) be to confuse a NN and cause it to misclassify both the items trained on and yet unseen items. Data used for NN training is usually represented as a large set of orthogonal vectors, each describing a different aspect of the item, allowing for very simple vector arithmetic operations. Thus, NN training is termed as high-dimensional or hyperdimensional. The attack method described by the authors employs cosine similarity, that is, in order to preserve similarity, a target hypervector is reflected over a given dimension, yielding a cosine-similar result that will trick ML models, even if using byzantine-robust defenses. The paper is clear, though not an easy read. It explains in detail the mathematical operations, following several related although different threat models. The authors present the results of the experimental evaluation of their proposed model, comparing it to several other well-known adversarial attacks for visual recognition tasks, over pre-labeled datasets frequently used as training data, such as MNIST, Fashion-MNIST and CIFAR-10. They show that their method is not only more effective as an attack, but falls within the same time range as other surveyed attacks. Adversarial attacks are, all in all, an important way to advance any field of knowledge; by publishing this attack, the authors will surely spark other works to detect and prevent this kind of alteration. It is important for AI implementers to understand the nature of this field and be aware of the risks that this work, as well as others cited in it, highlight: ML will train a computer system to recognize a dataset, warts and all; efficient as AI is, if noise is allowed into the training data (particularly adversarially generated noise), the trained model might present impaired performance.

This post is an unpublished review for Computational modelling of robot personhood and relationality

If humans and robots were to be able to roam around the same spaces, mutually recognizing each other for what they are, how would interaction be? How can we model such interactions in a way that we can reason about and understand the implications of a given behavior? This book aims at answering this question. The book is split into two very different parts. Chapters 1 through 3 are mostly written with a philosophical angle. It starts by framing the possibility of having sentient androids exist in the same plane as humans, without them trying to pass as us or vice versa. The first chapters look at issues related to personhood, that is, how androids can be treated as valid interaction partners in a society with humans, and how interactions with them can be seen as meaningful. In doing so, several landmarks of the past 40 years in the AI field are reviewed. The issues of the Significant Concerns that make up a society and give it coherence and of Personhood and Relationality , describing how this permeates from a society into each of the individuals that make it up, the relations between them and the social objects that bring individuals closer together (or farther apart) are introduced and explained. The second part of the book is written from a very different angle, and the change in pace took me somewhat by surprise. Each subsequent chapter presents a different angle of the Affinity system, a model that follows some aspects of human behavior over time and in a given space. Chapter 4 introduces the Affinity environment: a 3D simulated environment with simulated physical laws and characteristics, where a number of agents (30-50 is mentioned as usual) interact. Agents have a series of attributes ( value memory ), can adhere to different programs ( narratives ), and gain or lose on some vectors ( economy ). They can sense the world around them with sensors, and can modify the world or signal other agents using effectors. The last two chapters round out the book, as expected: the first presents a set of results from analyzing a given set of value systems, and the second gives readers the conclusions reached by the author. However, I was expecting more either having at least a link to download the Affinity system and continue exploring it or modifying some of the aspects it models to get it to model a set of agents with different stories and narratives, or extend it to yet unforseen behaviors, or at least have the author present a more complete comparison of results than the evaluation of patterns resulting from a given run. The author is a well-known, prolific author in the field, and I was expecting bigger insights from this book. Nevertheless, the book is an interesting and fun read, with important insights in both the first and second parts. There is a certain lack of connection between their respective rhythms, and the second part indeed builds on the concepts introduced in the first one. Overall, I enjoyed reading the book despite expecting more.

Since December 2023, I have been publishing the reviews I write for Computing Reviews as they get published. I will do a slight change now: I will start pushing the reviews to my blog as I write them, and of course, will modify them with the final wording and to link to their place as soon as they are published. I m doing this because sometimes it takes very long for reviews to be approved, and I want to share them with my blog s readers! So, please bear with this a bit: I ll send a (short!) flood of my latest four pending reviews today.

• Debian packages:
  • apt:
    • Bugs:
      • replied to #1078608: apt update silently leaves old index data
  • cis-tools:
    • Merge requests:
      • opened and merged !1: Initial Debian packaging
  • firmware-free:
    • Bugs:
      • replied to #890601: firmware-linux-free uses prebuilt blobs instead of building from source
    • Merge requests:
      • opened !10: Draft: Build CIS files and keyspan_pda firmware from source
  • firmware-nonfree:
    • Bugs:
      • closed #1106282: firmware-nonfree: Update stable-bpo to testing version
    • Uploads:
      • uploaded version 20250410-2~bpo12+1 to bookworm-backports
  • initramfs-tools:
    • Bugs:
      • closed #980021: initramfs-tools: Upgrading a LVM2 system with separate /usr to buster breaks booting
    • Merge requests:
      • merged !142: Handle simple-framebuffer drivers and framebuffer_coreboot built as modules
      • merged !167: hook-functions: Print error message on copy_file failure
      • merged !169: unmkinitramfs: Stop splitting into early and main subdirectories
      • merged !170: Fix inclusion of panel drivers
  • ktls-utils:
    • Uploads:
      • uploaded version 1.0.0-1 to unstable
  • linux:
    • Bugs:
      • replied to #763547: firmware-linux: please include ihex2fw tool into one of linux-* packages
      • replied to #1103397: Kernel Panic: copy_fpstate_to_sigframe crashes
      • replied to #1104080: linux: Please build with CONFIG_KERNEL_GZIP=y instead of CONFIG_KERNEL_XZ=y on sh4
    • Merge requests:
      • reviewed !1461: Fix some Lintian findings
      • merged !1466: Fix various lintian errors and warnings
      • merged !1471: [arm64] udeb: Add mtk-cmdq-mailbox to kernel-image
      • merged !1488: net/ipv6: Enable IPV6_RPL_LWTUNNEL
      • opened and merged !1489: Fix sh4 build and disable broken flavour
      • opened !1502: Add linux-misc-tools package for bootconfig, ihex2fw, thermometer, and tmon
      • merged !1506: [sparc64/smp] Enable NUMA
      • reviewed !1507: Drop yama: Disable by default and use upstream default to restrict ptrace
      • (LTS) opened !1524: Include target Debian release number in ABI name
    • Uploads:
      • (LTS) uploaded version 5.10.237-1 to bullseye-security
  • linux-6.1:
    • Uploads:
      • (LTS) uploaded version 6.1.137-1~deb11u1 to bullseye-security
  • linux-base:
    • Bugs:
      • closed #572712: use hardened sysctl net.* settings per default
      • closed #1076510: procps: vm.max_map_count = 65530 : Value too small for gaming
      • closed #1095744: kernel-img.conf.5: Some remarks and a patch with editorial changes for this man page
      • closed #1099472: linux-base: manpage refers to long-gone version numbers
      • closed #1100702: linux-update-symlinks.1: Some remarks and a patch with editorial changes for this man page
    • Merge requests:
      • opened !14: Add linux-run-hooks command for use by package maintainer scripts
      • opened !15: Clarify that version parameters are not package version strings
      • opened and merged !16: Manual page updates
      • opened and merged !17: linux-sysctl-defaults: Increase vm.max_map_count to 1048576
    • Uploads:
      • uploaded version 4.12 to unstable
  • procps:
    • Bugs:
      • replied to #1092550: procps: Addback/Provide example of sysctl.d/*.conf like the old sysctl.conf under /usr/share/doc/
  • reform-tools:
    • Bugs:
      • opened and replied to #1104639: Kernel hook installed in wrong directory / missing dependency
  • wireless-regdb:
    • Uploads:
      • uploaded version 2025.02.20-1~deb12u1 to bookworm
      • (LTS) uploaded version 2025.02.20-1~deb11u1 to bullseye-security
• Debian non-package bugs:
  • release.debian.org:
    • replied to #1104052: bookworm-pu: package initramfs-tools/0.142+deb12u2
    • opened #1106559: unblock: linux-base/4.12
    • opened #1106788: unblock: ktls-utils/1.0.0-1
  • wnpp:
    • replied to #1072579: ITP: cis-tools CIS file tools from pcmcia-cs
    • replied to #1105145: ITP: linux-board-support-package-rb3gen2 Firmware for RB3Gen2
    • replied to #1105839: ITP: ovpn-dkms OpenVPN data channel offload (ovpn)
• Mailing lists:
  • debian-devel:
    • posted and replied to Adding Pre-Depends from linux-image packages to linux-base
  • debian-events-eu:
    • replied to Debian BSP near Leuven, 26-27 April
  • debian-kernel:
    • posted Agenda items for kernel-team meeting on 2025-05-21
  • debian-lts:
    • replied to linux-image-6.1-* in Debian Bookworm
  • debian-lts-announce:
    • posted [SECURITY] [DLA 4171-1] wireless-regdb new upstream version
    • posted [SECURITY] [DLA 4178-1] linux security update
    • posted [SECURITY] [DLA 4193-1] linux-6.1 security update
  • debian-release:
    • posted Upstream kernel maintenance lifetime for bookworm and trixie
  • initramfs:
    • posted initramfs-tools 0.147 release
    • posted initramfs-tools 0.148 release

Another short status update of what happened on my side last month. Larger blocks besides the Phosh 0.47 release are on screen keyboard and cell broadcast improvements, work on separate volume streams, the switch of phoc to wlroots 0.19.0 and effort to make Phosh work on Debian's upcoming stable release (Trixie) out of the box. Trixie will ship with Phosh 0.46, if you want to try out 0.47 you can fetch it from Debian's experimental suite. See below for details on the above and more: phosh

• Track volume control based on media role priority (MR)
• Release 0.47~rc1, 0.47.0

phoc

• Release 0.47~rc1, 0.47.0
• More signal unlink fixes (MR)
• Further polish the wlroots 0.19 switch and undraft (MR)
• Start to track wlroots 0.20 dev branch (MR)
• Remember output configuration and restore scale,mode,transform for single output configs (MR)

phosh-mobile-settings

• Release 0.47~rc1, 0.47.0
• CI maintenance (MR, MR, MR)
• New alerts panel (MR)
• Handle p-o-s to stevia rename (MR)

phosh-osk-stub / stevia

• Smoke test completers (MR)
• Better use horizontal space in completion bar (MR)
• Add emojis to completion bar (MR)
• Drop separate emoji data (MR)
• Add some keyword completions (MR)
• Better handle lots of completions (MR)
• Release 0.47~rc1, 0.47.0
• Rename to Stevia (MR), (MR)
• Release 0.48~alpha1 to ease the rename for distros
• Some minor GTK4 preps (MR)

phosh-tour

• Release 0.47.0

phosh-osk-data

• Check packaging step (MR)

pfs

• Release 0.0.3

xdg-desktop-portal-phosh

• Release 0.47.0

phrog

• Allow systemd to fail to help bootstrapping (MR)

phosh-debs

• Skip noinsttest profiles (MR) - no need for these in nightly builds and helps nocheck triggering errors
• Work around phrog install failure until fixed upstream (MR)
• Switch from osk-stub to stevia (MR)

meta-phosh

• Support automatic restart in more projects (MR)
• Fix glob for meson checks (MR)

feedbackd

• Expand media role MR with volume tracking (MR) - this is basically the same as the Wireplumber MR from below
• vibra-pattern: Don't overwrite magnitutde when changing global level (MR)
• Update fbcli manpage (MR)
• Drop custom script (MR)
• Add key- pressed,released events (MR)
• Release 0.8.2

feedbackd-device-themes

• Release 0.8.3
• Use stronger button-pressed feedback for google,sargo (MR)

gmobile

• Install tests so they can run through ginsttest-runner (MR)
• Release 0.3.0, 0.3.1
• Small timer cleanups (MR)
• Add mcc to iso country code conversion (MR)
• Fix manpage subject (MR)

GNOME calls

• Fix crash on shutdown (MR)
• Drop meson version check (MR)
• Backport fixes for 48 (MR)

Debian

• phosh: Backport patches from 0.46 stable (MR)
• phosh: Upload 0.47~rc1, 0.47.0
• gnome-calls: Backport outging SIP call UI fix (MR)
• gmobile: Upload 0.3.0 with improved autopkg tests
• phoc: Upload 0.47~rc1, 0.47.0
• phosh-osk-stub: 0.47.0
• release-dom-compoenent: Drop tmp branch (MR)
• phosh-tour: Upload 0.47.0
• phosh-mobile-settings: Upload 0.47~rc1, 0.47.0
• xdg-desktop-portal-phosh: Upload 0.47.0
• wlroots: Upload 0.19.0
• stevia: Upload 0.48~alpha1
• meta-phosh: phosh-full: Depend on xwayland (MR)
• feedbackd-device-themes: Upload 0.8.3
• feedbackd: Upload 0.8.2
• feedbackd-device-themes: Backport key-press fix for google,sargo (MR)

ModemManager

• Simplify builds that need updated libqmi/libmbim (MR)
• CellBroadcast: Fix QMI channel reading corner cases (MR)
• Fix crash in CBM code (caused by unrelated bind-to refactor) (MR)
• Handle more data codings and expose CBM's language (MR)

osmo-cbc

• api-tool: Allow to set language of cell broadcast message (MR)

gsm-cell-testing

• Update docs to use our new MM subproject builds (MR)

mobile-broadband-provider-info

• Allow emergency number information (MR)
• Allow Cell Broadcast information (MR)
• Add .dir-locales.el and make tests easier to parse (MR)

Cellbroadcastd

• Add channel handling (MR)
• Indicate on DBus whether Cell Broadcasts are supported (MR)
• Add ASAN check and fix fallout (MR)

phosh-site

• Add a support feature matrix (MR)
• New post: OSK completion and correction

pipewire

• Simplify header use for projects that use -Wswitch-default (MR)

wireplumber

• Add lua script to track suitable volume control when role based policy linking is in use (MR)

python-dbusmock

• ModemManager: Add set-channel support (MR)

Bugs

• Standardize audio stream roles (MR). Otherwise we'll have a hard time with e.g. WirePlumbers role based policy linking as apps might use all kinds of types.

Reviews This is not code by me but reviews on other peoples code. The list is (as usual) slightly incomplete. Thanks for the contributions!

• gmobile/pp: Ignore buttons on headsets (MR)
• p-o-s: Add Arabic translation (MR)
• p-o-s: pipe completer examples (MR)
• p-o-s: US Dvorak layout (MR)
• p-m-s: Use static library for building (MR)
• p-m-s: pmos tweaks UI parts (MR)
• p-m-s: Notification categories that light up the screen (MR)
• gmobile: Border radius for oneplus,enchilada (MR)
• m-b-p-i: Telia IOT provider (MR)
• phosh-site: Highlight selected footnote (MR)
• bluez: Don't start mpris proxy for root user (Patch)
• phoc: shortcuts-inhibit: Add support for the keyboard shortcuts inhibit protocol (MR)

Help Development If you want to support my work see donations. Comments? Join the Fediverse thread

Christopher Biggs gave an informative Evrything Open lecture about voice recognition [1]. We need this on Debian phones. Guido wrote an informative blog post about booting a custom Android kernel on a Pixel 3a [2]. Good work in writing this up, but a pity that Google made the process so difficult. Interesting to read about an expert being a victim of a phishing attack [3]. It can happen to anyone, everyone has moments when they aren t concentrating. Interesting advice on how to leak to a journalist [4]. Brian Krebs wrote an informative article about the ways that Trump is deliberately reducing the cyber security of the US government [5]. Brian Krebs wrote an interesting article about the main smishng groups from China [6]. Louis Rossmann (who is known for high quality YouTube videos about computer repair) made an informative video about a scammy Australian company run by a child sex offender [7]. The Helmover was one of the wildest engineering projects of WW2, an over the horizon guided torpedo that could one-shot a battleship [8]. Interesting blog post about DDoSecrets and the utter failure of the company Telemessages which was used by the US government [9]. Jonathan McDowell wrote an interesting blog post about developing a free software competitor to Alexa etc, the listening hardware costs $13US per node [10]. Noema Magazine published an insightful article about Rewilding the Internet, it has some great ideas [11].

• [1] https://www.youtube.com/watch?v=8w8NX3AUX4c
• [2] https://tinyurl.com/25gdysxx
• [3] https://tinyurl.com/29mwthwz
• [4] https://tinyurl.com/27t2tztd
• [5] https://tinyurl.com/2am7o6qw
• [6] https://tinyurl.com/27p8z6gf
• [7] https://www.youtube.com/watch?v=qFe5LiACN9k
• [8] https://www.youtube.com/watch?v=J1W2njKMSks
• [9] https://tinyurl.com/ytq84fb3
• [10] https://www.earth.li/~noodles/blog/2025/04/voice-assistant-atom-echo.html
• [11] https://www.noemamag.com/we-need-to-rewild-the-internet/

Have you ever found yourself in the situation where you had no or anonymized logs and still wanted to figure out where your traffic was coming from? Or you have multiple upstreams and are looking to see if you can save fees by getting into peering agreements with some other party? Or your site is getting heavy load but you can't pinpoint it on a single IP and you suspect some amoral corporation is training their degenerate AI on your content with a bot army? (You might be getting onto something there.) If that rings a bell, read on.

TL;DR: ... or just skip the cruft and install asncounter:

pip install asncounter

Also available in Debian 14 or later, or possibly in Debian 13 backports (soon to be released) if people are interested:

apt install asncounter

Then count whoever is hitting your network with:

awk ' print $2 ' /var/log/apache2/*access*.log   asncounter

or:

tail -F /var/log/apache2/*access*.log   awk ' print $2 '   asncounter

or:

tcpdump -q -n   asncounter --input-format=tcpdump --repl

or:

tcpdump -q -i eth0 -n -Q in "tcp and tcp[tcpflags] & tcp-syn != 0 and (port 80 or port 443)"   asncounter --input-format=tcpdump --repl

Read on for why this matters, and why I wrote yet another weird tool (almost) from scratch.

Background and manual work This is a tool I've been dreaming of for a long, long time. Back in 2006, at Koumbit a colleague had setup TAS ("Traffic Accounting System", " " in Russian, apparently), a collection of Perl script that would do per-IP accounting. It was pretty cool: it would count bytes per IP addresses and, from that, you could do analysis. But the project died, and it was kind of bespoke. Fast forward twenty years, and I find myself fighting off bots at the Tor Project (the irony...), with our GitLab suffering pretty bad slowdowns (see issue tpo/tpa/team#41677 for the latest public issue, the juicier one is confidential, unfortunately). (We did have some issues caused by overloads in CI, as we host, after all, a fork of Firefox, which is a massive repository, but the applications team did sustained, awesome work to fix issues on that side, again and again (see tpo/applications/tor-browser#43121 for the latest, and tpo/applications/tor-browser#43121 for some pretty impressive correlation work, I work with really skilled people). But those issues, I believe were fixed.) So I had the feeling it was our turn to get hammered by the AI bots. But how do we tell? I could tell something was hammering at the costly /commit/ and (especially costly) /blame/ endpoint. So at first, I pulled out the trusted awk, sort uniq -c sort -n tail pipeline I am sure others have worked out before:

awk ' print $1 ' /var/log/nginx/*.log   sort   uniq -c   sort -n   tail -10

For people new to this, that pulls the first field out of web server log files, sort the list, counts the number of unique entries, and sorts that so that the most common entries (or IPs) show up first, then show the top 10. That, other words, answers the question of "which IP address visits this web server the most?" Based on this, I found a couple of IP addresses that looked like Alibaba. I had already addressed an abuse complaint to them (tpo/tpa/team#42152) but never got a response, so I just blocked their entire network blocks, rather violently:

for cidr in 47.240.0.0/14 47.246.0.0/16 47.244.0.0/15 47.235.0.0/16 47.236.0.0/14; do 
  iptables-legacy -I INPUT -s $cidr -j REJECT
done

That made Ali Baba and his forty thieves (specifically their AL-3 network go away, but our load was still high, and I was still seeing various IPs crawling the costly endpoints. And this time, it was hard to tell who they were: you'll notice all the Alibaba IPs are inside the same 47.0.0.0/8 prefix. Although it's not a /8 itself, it's all inside the same prefix, so it's visually easy to pick it apart, especially for a brain like mine who's stared too long at logs flowing by too fast for their own mental health. What I had then was different, and I was tired of doing the stupid thing I had been doing for decades at this point. I had recently stumbled upon pyasn recently (in January, according to my notes) and somehow found it again, and thought "I bet I could write a quick script that loops over IPs and counts IPs per ASN". (Obviously, there are lots of other tools out there for that kind of monitoring. Argos, for example, presumably does this, but it's a kind of a huge stack. You can also get into netflows, but there's serious privacy implications with those. There are also lots of per-IP counters like promacct, but that doesn't scale. Or maybe someone already had solved this problem and I just wasted a week of my life, who knows. Someone will let me know, I hope, either way.)

ASNs and networks A quick aside, for people not familiar with how the internet works. People that know about ASNs, BGP announcements and so on can skip. The internet is the network of networks. It's made of multiple networks that talk to each other. The way this works is there is a Border Gateway Protocol (BGP), a relatively simple TCP-based protocol, that the edge routers of those networks used to announce each other what network they manage. Each of those network is called an Autonomous System (AS) and has an AS number (ASN) to uniquely identify it. Just like IP addresses, ASNs are allocated by IANA and local registries, they're pretty cheap and useful if you like running your own routers, get one. When you have an ASN, you'll use it to, say, announce to your BGP neighbors "I have 198.51.100.0/24" over here and the others might say "okay, and I have 216.90.108.31/19 over here, and I know of this other ASN over there that has 192.0.2.1/24 too! And gradually, those announcements flood the entire network, and you end up with each BGP having a routing table of the global internet, with a map of which network block, or "prefix" is announced by which ASN. It's how the internet works, and it's a useful thing to know, because it's what, ultimately, makes an organisation responsible for an IP address. There are "looking glass" tools like the one provided by routeviews.org which allow you to effectively run "trace routes" (but not the same as traceroute, which actively sends probes from your location), type an IP address in that form to fiddle with it. You will end up with an "AS path", the way to get from the looking glass to the announced network. But I digress, and that's kind of out of scope. Point is, internet is made of networks, networks are autonomous systems (AS) and they have numbers (ASNs), and they announced IP prefixes (or "network blocks") that ultimately tells you who is responsible for traffic on the internet.

Introducing asncounter So my goal was to get from "lots of IP addresses" to "list of ASNs", possibly also the list of prefixes (because why not). Turns out pyasn makes that really easy. I managed to build a prototype in probably less than an hour, just look at the first version, it's 44 lines (sloccount) of Python, and it works, provided you have already downloaded the required datafiles from routeviews.org. (Obviously, the latest version is longer at close to 1000 lines, but it downloads the data files automatically, and has many more features). The way the first prototype (and later versions too, mostly) worked is that you feed it a list of IP addresses on standard input, it looks up the ASN and prefix associated with the IP, and increments a counter for those, then print the result. That showed me something like this:

root@gitlab-02:~/anarcat-scripts# tcpdump -q -i eth0 -n -Q in "(udp or tcp)"   ./asncounter.py --tcpdump                                                                                                                                                                          
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode                                                                
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes                                                             
INFO: collecting IPs from stdin, using datfile ipasn_20250523.1600.dat.gz                                                                
INFO: loading datfile /root/.cache/pyasn/ipasn_20250523.1600.dat.gz...                                                                   
INFO: loading /root/.cache/pyasn/asnames.json                       
ASN     count   AS               
136907  7811    HWCLOUDS-AS-AP HUAWEI CLOUDS, HK                                                                                         
[----]  359     [REDACTED]
[----]  313     [REDACTED]
8075    254     MICROSOFT-CORP-MSN-AS-BLOCK, US
[---]   164     [REDACTED]
[----]  136     [REDACTED]
24940   114     HETZNER-AS, DE  
[----]  98      [REDACTED]
14618   82      AMAZON-AES, US                                                                                                           
[----]  79      [REDACTED]
prefix  count                                         
166.108.192.0/20        1294                                                                                                             
188.239.32.0/20 1056                                          
166.108.224.0/20        970                    
111.119.192.0/20        951              
124.243.128.0/18        667                                         
94.74.80.0/20   651                                                 
111.119.224.0/20        622                                         
111.119.240.0/20        566           
111.119.208.0/20        538                                         
[REDACTED]  313           

Even without ratios and a total count (which will come later), it was quite clear that Huawei was doing something big on the server. At that point, it was responsible for a quarter to half of the traffic on our GitLab server or about 5-10 queries per second. But just looking at the logs, or per IP hit counts, it was really hard to tell. That traffic is really well distributed. If you look more closely at the output above, you'll notice I redacted a couple of entries except major providers, for privacy reasons. But you'll also notice almost nothing is redacted in the prefix list, why? Because all of those networks are Huawei! Their announcements are kind of bonkers: they have hundreds of such prefixes. Now, clever people in the know will say "of course they do, it's an hyperscaler; just ASN14618 (AMAZON-AES) there is way more announcements, they have 1416 prefixes!" Yes, of course, but they are not generating half of my traffic (at least, not yet). But even then: this also applies to Amazon! This way of counting traffic is way more useful for large scale operations like this, because you group by organisation instead of by server or individual endpoint. And, ultimately, this is why asncounter matters: it allows you to group your traffic by organisation, the place you can actually negotiate with. Now, of course, that assumes those are entities you can talk with. I have written to both Alibaba and Huawei, and have yet to receive a response. I assume I never will. In their defence, I wrote in English, perhaps I should have made the effort of translating my message in Chinese, but then again English is the Lingua Franca of the Internet, and I doubt that's actually the issue.

The Huawei and Facebook blocks Another aside, because this is my blog and I am not looking for a Pullitzer here. So I blocked Huawei from our GitLab server (and before you tear your shirt open: only our GitLab server, everything else is still accessible to them, including our email server to respond to my complaint). I did so 24h after emailing them, and after examining their user agent (UA) headers. Boy that was fun. In a sample of 268 requests I analyzed, they churned out 246 different UAs. At first glance, they looked legit, like:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36

Safari on a Mac, so far so good. But when you start digging, you notice some strange things, like here's Safari running on Linux:

Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.457.0 Safari/534.3

Was Safari ported to Linux? I guess that's.. possible? But here is Safari running on a 15 year old Ubuntu release (10.10):

Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Ubuntu/10.10 Chromium/12.0.702.0 Chrome/12.0.702.0 Safari/534.24

Speaking of old, here's Safari again, but this time running on Windows NT 5.1, AKA Windows XP, released 2001, EOL since 2019:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-CA) AppleWebKit/534.13 (KHTML like Gecko) Chrome/9.0.597.98 Safari/534.13

Really? Here's Firefox 3.6, released 14 years ago, there were quite a lot of those:

Mozilla/5.0 (Windows; U; Windows NT 6.1; lt; rv:1.9.2) Gecko/20100115 Firefox/3.6

I remember running those old Firefox releases, those were the days. But to me, those look like entirely fake UAs, deliberately rotated to make it look like legitimate traffic. In comparison, Facebook seemed a bit more legit, in the sense that they don't fake it. most hits are from:

meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

which, according their documentation:

crawls the web for use cases such as training AI models or improving products by indexing content directly

From what I could tell, it was even respecting our rather liberal robots.txt rules, in that it wasn't crawling the sprawling /blame/ or /commit/ endpoints, explicitly forbidden by robots.txt. So I've blocked the Facebook bot in robots.txt and, amazingly, it just went away. Good job Facebook, as much as I think you've given the empire to neo-nazis, cause depression and genocide, you know how to run a crawler, thanks. Huawei was blocked at the web server level, with a friendly 429 status code telling people to contact us (over email) if they need help. And they don't care: they're still hammering the server, from what I can tell, but then again, I didn't block the entire ASN just yet, just the blocks I found crawling the server over a couple hours.

A full asncounter run So what does a day in asncounter look like? Well, you start with a problem, say you're getting too much traffic and want to see where it's from. First you need to sample it. Typically, you'd do that with tcpdump or tailing a log file:

tail -F /var/log/apache2/*access*.log   awk ' print $2 '   asncounter

If you have lots of traffic or care about your users' privacy, you're not going to log IP addresses, so tcpdump is likely a good option instead:

tcpdump -q -n   asncounter --input-format=tcpdump --repl

If you really get a lot of traffic, you might want to get a subset of that to avoid overwhelming asncounter, it's not fast enough to do multiple gigabit/second, I bet, so here's only incoming SYN IPv4 packets:

tcpdump -q -n -Q in "tcp and tcp[tcpflags] & tcp-syn != 0 and (port 80 or port 443)"   asncounter --input-format=tcpdump --repl

In any case, at this point you're staring at a process, just sitting there. If you passed the --repl or --manhole arguments, you're lucky: you have a Python shell inside the program. Otherwise, send SIGHUP to the thing to have it dump the nice tables out:

pkill -HUP asncounter

Here's an example run:

> awk ' print $2 ' /var/log/apache2/*access*.log   asncounter
INFO: using datfile ipasn_20250527.1600.dat.gz
INFO: collecting addresses from <stdin>
INFO: loading datfile /home/anarcat/.cache/pyasn/ipasn_20250527.1600.dat.gz...
INFO: finished reading data
INFO: loading /home/anarcat/.cache/pyasn/asnames.json
count   percent ASN AS
12779   69.33   66496   SAMPLE, CA
3361    18.23   None    None
366 1.99    66497   EXAMPLE, FR
337 1.83    16276   OVH, FR
321 1.74    8075    MICROSOFT-CORP-MSN-AS-BLOCK, US
309 1.68    14061   DIGITALOCEAN-ASN, US
128 0.69    16509   AMAZON-02, US
77  0.42    48090   DMZHOST, GB
56  0.3 136907  HWCLOUDS-AS-AP HUAWEI CLOUDS, HK
53  0.29    17621   CNCGROUP-SH China Unicom Shanghai network, CN
total: 18433
count   percent prefix  ASN AS
12779   69.33   192.0.2.0/24    66496   SAMPLE, CA
3361    18.23   None        
298 1.62    178.128.208.0/20    14061   DIGITALOCEAN-ASN, US
289 1.57    51.222.0.0/16   16276   OVH, FR
272 1.48    2001:DB8::/48   66497   EXAMPLE, FR
235 1.27    172.160.0.0/11  8075    MICROSOFT-CORP-MSN-AS-BLOCK, US
94  0.51    2001:DB8:1::/48 66497   EXAMPLE, FR
72  0.39    47.128.0.0/14   16509   AMAZON-02, US
69  0.37    93.123.109.0/24 48090   DMZHOST, GB
53  0.29    27.115.124.0/24 17621   CNCGROUP-SH China Unicom Shanghai network, CN

Those numbers are actually from my home network, not GitLab. Over there, the battle still rages on, but at least the vampire bots are banging their heads against the solid Nginx wall instead of eating the fragile heart of GitLab. We had a significant improvement in latency thanks to the Facebook and Huawei blocks... Here are the "workhorse request duration stats" for various time ranges, 20h after the block:

range	mean	max	stdev
20h	449ms	958ms	39ms
7d	1.78s	5m	14.9s
30d	2.08s	3.86m	8.86s
6m	901ms	27.3s	2.43s

We went from two seconds mean to 500ms! And look at that standard deviation! 39ms! It was ten seconds before! I doubt we'll keep it that way very long but for now, it feels like I won a battle, and I didn't even have to setup anubis or go-away, although I suspect that will unfortunately come. Note that asncounter also supports exporting Prometheus metrics, but you should be careful with this, as it can lead to cardinal explosion, especially if you track by prefix (which can be disabled with --no-prefixes . Folks interested in more details should read the fine manual for more examples, usage, and discussion. It shows, among other things, how to effectively block lots of networks from Nginx, aggregate multiple prefixes, block entire ASNs, and more! So there you have it: I now have the tool I wish I had 20 years ago. Hopefully it will stay useful for another 20 years, although I'm not sure we'll have still have internet in 20 years. I welcome constructive feedback, "oh no you rewrote X", Grafana dashboards, bug reports, pull requests, and "hell yeah" comments. Hacker News, let it rip, I know you can give me another juicy quote for my blog. This work was done as part of my paid work for the Tor Project, currently in a fundraising drive, give us money if you like what you read.

Marco wrote a blog post opposing hyperscale systems which included We want to use an hyperscaler cloud because our developers do not want to operate a scalable and redundant database just means that you need to hire competent developers and/or system administrators. [1]. I previously wrote a blog post Why Clusters Usually Don t Work [2] and I believe that all the points there are valid today and possibly exacerbated by clusters getting less direct use as clustering is increasingly being done by hyperscale providers. Take a basic need, a MySQL or PostgreSQL database for example. You want it to run and basically do the job and to have good recovery options. You could set it up locally, run backups, test the backups, have a recovery plan for failures, maybe have a hot-spare server if it s really important, have tests for backups and hot-spare server, etc. Then you could have documentation for this so if the person who set it up isn t available when there s a problem they will be able to find out what to do. But the hyperscale option is to just select a database in your provider and have all this just work. If the person who set it up isn t available for recovery in the event of failure the company can just put out a job advert for person with experience on cloud company X and have them just immediately go to work on it. I don t like hyperscale providers as they are all monopolistic companies that do anti-competitive actions. Google should be broken up, Android development and the Play Store should be separated from Gmail etc which should be separated from search and adverts, and all of them should be separated from the GCP cloud service. Amazon should be broken up, running the Amazon store should be separated from selling items on the store, which should be separated from running a video on demand platform, and all of them should be separated from the AWS cloud. Microsoft should be broken up, OS development should be separated from application development all of that should be separated from cloud services (Teams and Office 365), and everything else should be separate from the Azure cloud system. But the cloud providers offer real benefits at small scale. Running a MySQL or PostgreSQL database for local services is easy, it s a simple apt command to install it and then it basically works. Doing backup and recovery isn t so easy. One could say just hire competent people but if you do hire competent people do you want them running MySQL databases etc or have them just click on the create mysql database option on a cloud control panel and then move on to more important things? The FreedomBox project is a great project for installing and managing home/personal services [3]. But it s not about running things like database servers, it s for a high level running mail servers and other things for the user not for the developer. The Debian packaging of Open Stack looks interesting [4], it s a complete setup for running your own hyper scale cloud service. For medium and large organisations running Open Stack could be a good approach. But for small organisations it s cheaper and easier to just use a cloud service to run things. The issue of when to run things in-house and when to put them in the cloud is very complex. I think that if the organisation is going to spend less money on cloud services than on the salary of one sysadmin then it s probably best to have things in the cloud. When cloud costs start to exceed the salary of one person who manages systems then having them spend the extra time and effort to run things locally starts making more sense. There is also an opportunity cost in having a good sysadmin work on the backups for all the different systems instead of letting the cloud provider just do it. Another possibility of course is to run things in-house on low end hardware and just deal with the occasional downtime to save money. Knowingly choosing less reliability to save money can be quite reasonable as long as you have considered the options and all the responsible people are involved in the discussion. The one situation that I strongly oppose is having hyper scale services setup by people who don t understand them. Running a database server on a cloud service because you don t want to spend the time managing it is a reasonable choice in many situations. Running a database server on a cloud service because you don t understand how to setup a database server is never a good choice. While the cloud services are quite resilient there are still ways of breaking the overall system if you don t understand it. Also while it is quite possible for someone to know how to develop for databases including avoiding SQL injection etc but be unable to setup a database server that s probably not going to be common, probably if someone can t set it up (a generally easy task) then they can t do the hard tasks of making it secure.

• [1] https://blog.bofh.it/debian/id_471
• [2] https://etbe.coker.com.au/2010/08/04/clusters-dont-work/
• [3] https://freedombox.org/
• [4] https://wiki.debian.org/OpenStack

Welcome to post 48 in the R⁴ series, and to video 8 in this series. Last week I had the honour of giving the opening talk at the 11eme Rencontres R at the Universit de Mons in Belgium as an invited plenary talk. Big thanks again to Philippe Grosjean and Kathy Huet for the invitation, and for organising a lovely conference. Being the opening talk, we were still sorting out projector issues when I started so I forgot to set a timer, and consequently ran out of time like a newbie. It occured to me that I could simply re-record the talk in front of my slides just as I do for my STAT 447 students. So I sat down this morning and did this, and the video is now online:

The slides are available as well. For questions or other feedback, please consider using the r2u GitHub repo issues section.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

I ve been part of the Debian Project since 2019, when I attended DebConf held in Curitiba, Brazil. That event sparked my interest in the community, packaging, and how Debian works as a distribution. In the early years of my involvement, I contributed to various teams such as the Python, Golang and Cloud teams, packaging dependencies and maintaining various tools. However, I soon felt the need to focus on packaging software I truly enjoyed, tools I was passionate about using and maintaining. That s when I turned my attention to Kubernetes within Debian.

---

A Broken Ecosystem The Kubernetes packaging situation in Debian had been problematic for some time. Given its large codebase and complex dependency tree, the initial packaging approach involved vendorizing all dependencies. While this allowed a somewhat functional package to be published, it introduced several long-term issues, especially security concerns. Vendorized packages bundle third-party dependencies directly into the source tarball. When vulnerabilities arise in those dependencies, it becomes difficult for Debian s security team to patch and rebuild affected packages system-wide. This approach broke Debian s best practices, and it eventually led to the abandonment of the Kubernetes source package, which had stalled at version 1.20.5. Due to this abandonment, critical bugs emerged and the package was removed from Debian s testing channel, as we can see in the package tracker.

---

New Debian Kubernetes Team Around this time, I became a Debian Maintainer (DM), with permissions to upload certain packages. I saw an opportunity to both contribute more deeply to Debian and to fix Kubernetes packaging. In early 2024, just before DebConf Busan in South Korea, I founded the Debian Kubernetes Team. The mission of the team was to repackage Kubernetes in a maintainable, security-conscious, and Debian-compliant way. At DebConf, I shared our progress with the broader community and received great feedback and more visibility, along with people interested in contributing to the team. Our first tasks was to migrate existing Kubernetes-related tools such as kubectx, kubernetes-split-yaml and kubetail into a dedicated namespace on Salsa, Debian s GitLab instance. Many of these tools were stored across different teams (like the Go team), and consolidating them helped us organize development and focus our efforts.

---

De-vendorizing Kubernetes Our main goal was to un-vendorize Kubernetes and bring it up-to-date with upstream releases. This meant:

• Removing the vendor directory and all embedded third-party code.
• Trimming the build scope to focus solely on building kubectl, Kubernetes CLI.
• Using Files-Excluded in debian/copyright to cleanly drop unneeded files during source imports.
• Rebuilding the dependency tree, ensuring all Go modules were separately packaged in Debian.

We used uscan, a standard Debian packaging tool that fetches upstream tarballs and prepares them accordingly. The Files-Excluded directive in our debian/copyright file instructed uscan to automatically remove unnecessary files during the repackaging process:

$ uscan
Newest version of kubernetes on remote site is 1.32.3, specified download version is 1.32.3
Successfully repacked ../v1.32.3 as ../kubernetes_1.32.3+ds.orig.tar.gz, deleting 30616 files from it.

The results were dramatic. By comparing the original upstream tarball with our repackaged version, we can see that our approach reduced the tarball size by over 75%:

$ du -h upstream-v1.32.3.tar.gz kubernetes_1.32.3+ds.orig.tar.gz
14M	upstream-v1.32.3.tar.gz
3.2M	kubernetes_1.32.3+ds.orig.tar.gz

This significant reduction wasn t just about saving space. By removing over 30,000 files, we simplified the package, making it more maintainable. Each dependency could now be properly tracked, updated, and patched independently, resolving the security concerns that had plagued the previous packaging approach.

---

Dependency Graph To give you an idea of the complexity involved in packaging Kubernetes for Debian, the image below is a dependency graph generated with debtree, visualizing all the Go modules and other dependencies required to build the kubectl binary. This web of nodes and edges represents every module and its relationship during the compilation process of kubectl. Each box is a Debian package, and the lines connecting them show how deeply intertwined the ecosystem is. What might look like a mess of blue spaghetti is actually a clear demonstration of the vast and interconnected upstream world that tools like kubectl rely on. But more importantly, this graph is a testament to the effort that went into making kubectl build entirely using Debian-packaged dependencies only, no vendoring, no downloading from the internet, no proprietary blobs.

---

Upstream Version 1.32.3 and Beyond After nearly two years of work, we successfully uploaded version 1.32.3+ds of kubectl to Debian unstable. kubernetes/-/merge_requests/1

• Closed over a dozen long-standing bugs, including:
  • Outdated version requests
  • Missing shell completions
  • VCS (version control system) metadata issues
  • (#1055411, #990793, #1009356, #1016441, #1086756, #1047881, #832706, #976428, #994438)

The new package also includes:

• Zsh, Fish, and Bash completions installed automatically
• Man pages and metadata for improved discoverability
• Full integration with kind and docker for testing purposes

---

Integration Testing with Autopkgtest To ensure the reliability of kubectl in real-world scenarios, we developed a new autopkgtest suite that runs integration tests using real Kubernetes clusters created via Kind. Autopkgtest is a Debian tool used to run automated tests on binary packages. These tests are executed after the package is built but before it s accepted into the Debian archive, helping catch regressions and integration issues early in the packaging pipeline. Our test workflow validates kubectl by performing the following steps:

• Installing Kind and Docker as test dependencies.
• Spinning up two local Kubernetes clusters.
• Switching between cluster contexts to ensure multi-cluster support.
• Deploying and scaling a sample nginx application using kubectl.
• Cleaning up the entire test environment to avoid side effects.
• debian/tests/kubectl.sh

---

Popcon: Measuring Adoption To measure real-world usage, we rely on data from Debian s popularity contest (popcon), which gives insight into how many users have each binary installed. Here s what the data tells us:

• kubectl (new binary): Already installed on 2,124 systems.
• golang-k8s-kubectl-dev: This is the Go development package (a library), useful for other packages and developers who want to interact with Kubernetes programmatically.
• kubernetes-client: The legacy package that kubectl is replacing. We expect this number to decrease in future releases as more systems transition to the new package.

Although the popcon data shows activity for kubectl before the official Debian upload date, it s important to note that those numbers represent users who had it installed from upstream source-lists, not from the Debian repositories. This distinction underscores a demand that existed even before the package was available in Debian proper, and it validates the importance of bringing it into the archive.

Also worth mentioning: this number is not the real total number of installations, since users can choose not to participate in the popularity contest. So the actual adoption is likely higher than what popcon reflects.

---

Community and Documentation The team also maintains a dedicated wiki page which documents:

• Maintained tools and packages
• Contribution guidelines
• Our roadmap for the upcoming Debian releases

https://debian-kubernetes.org

---

Looking Ahead to Debian 13 (Trixie) The next stable release of Debian will ship with kubectl version 1.32.3, built from a clean, de-vendorized source. This version includes nearly all the latest upstream features, and will be the first time in years that Debian users can rely on an up-to-date, policy-compliant kubectl directly from the archive. By comparing with upstream, our Debian package even delivers more out of the box, including shell completions, which the upstream still requires users to generate manually. In 2025, the Debian Kubernetes team will continue expanding our packaging efforts for the Kubernetes ecosystem. Our roadmap includes:

• kubelet: The primary node agent that runs on each node. This will enable Debian users to create fully functional Kubernetes nodes without relying on external packages.
• kubeadm: A tool for creating Kubernetes clusters. With kubeadm in Debian, users will then be able to bootstrap minimum viable clusters directly from the official repositories.
• helm: The package manager for Kubernetes that helps manage applications through Kubernetes YAML files defined as charts.
• kompose: A conversion tool that helps users familiar with docker-compose move to Kubernetes by translating Docker Compose files into Kubernetes resources.

---

Final Thoughts This journey was only possible thanks to the amazing support of the debian-devel-br community and the collective effort of contributors who stepped up to package missing dependencies, fix bugs, and test new versions. Special thanks to:

• Carlos Henrique Melara (@charles)
• Guilherme Puida (@puida)
• Jo o Pedro Nobrega (@jnpf)
• Lucas Kanashiro (@kanashiro)
• Matheus Polkorny (@polkorny)
• Samuel Henrique (@samueloph)
• Sergio Cipriano (@cipriano)
• Sergio Durigan Junior (@sergiodj)

I look forward to continuing this work, bringing more Kubernetes tools into Debian and improving the developer experience for everyone.