Problem statement Currently if you have an automatically installed package A (= 1) where

• A (= 1) Depends B (= 1)
• A (= 2) Depends B (= 2)

and you upgrade B from 1 to 2; then you can:

1. Remove A (= 1)
2. Upgrade A to version 2

If A was installed by a chain initiated by Recommends (say X Rec Y, Y Depends A), the solver sometimes preferred removing A (and anything depending on it until it got). I have a fix pending to introduce eager Recommends which fixes the practical case, but this is still not sound. In fact we can show that the solver produces the wrong result for small minimal test cases, as well as the right result for some others without the fix (hooray?). Ensuring sound removals is more complex, and first of all it begs the question: When is a removal sound? This, of course, is on us to define. An easy case can be found in the Debian policy, 7.6.2 Replacing whole packages, forcing their removal : If B (= 2) declares a Conflicts: A (= 1) and Replaces: A (= 1), then the removal is valid. However this is incomplete as well, consider it declares Conflicts: A (< 1) and Replaces: A (< 1); the solution to remove A rather than upgrade it would still be wrong. This indicates that we should only allow removing A if the conflicts could not be solved by upgrading it. The other case to explore is package removals. If B is removed, A should be removed as well; however it there is another package X that Provides: B (= 1) and it is marked for install, A should not be removed. That said, the solver is not allowed to install X to satisfy the depends B (= 1) - only to satisfy other dependencies [we do not want to get into endless loops where we switch between alternatives to keep reverse dependencies installed].

Proposed solution To solve this, I propose the following definition: Definition (sound removal): A removal of package P is sound if either:

1. A version v is installed that package-conflicts with B.
2. A package Q is removed and the installable versions of P package-depends on Q.

where the other definitions are: Definition (installable version): A version v is installable if either it is installed, or it is newer than an installed version of the same package (you may wish to change this to accomodate downgrades, or require strict pinning, but here be dragons). Definition (package-depends): A version v package-depends on a package B if either:

1. there exists a dependency in v that can be solved by any version of B, or
2. there exists a package C where v package-depends C and any (c in C) package-depends B (transitivity)

Definition (package-conflicts): A version v package-conflicts with an installed package B if either:

1. it declares a conflicts against an installable version of B; or
2. there exists a package C where v package-conflicts C, and b package-depends C for installable versions b.

Translating this into a (modified) SAT solver One approach may be to implement the logic in the conflict analysis that drives backtracking, i.e. we assume a package A and when we reach not A, we analyse if the implication graph for not A constitutes a sound removal, and then replace the assumption A with the assumption A or "learned reason. However, while this seems a plausible mechanism for a DPLL solver, for a modern CDCL solver, it s not immediately evident how to analyse whether not A is sound if the reason for it is a learned clause, rather than a problem clause. Instead we propose a static encoding of the rules into a slightly modified SAT solver: Given c1, , cn that transitive-conflicts A and D1, , Dn that A package-depends on, introduce the rule: A unless c1 or c2 or ... cn ... or not D1 or not D2 ... or not Dn Rules of the form A... unless B... - where A... and B... are CNF - are intuitively the same as A... or B..., however the semantic here is different: We are not allowed to select B... to satisfy this clause. This requires a SAT solver that tracks a reason for each literal being assigned, such as solver3, rather than a SAT solver like MiniSAT that only tracks reasons across propagation (solver3 may track A depends B or C as the reason for B without evaluating C, whereas MiniSAT would only track it as the reason given not C).

Is it actually sound? The proposed definition of a sound removal may still proof unsound as I either missed something in the conclusion of the proposed definition that violates my goal I set out to achieve, or I missed some of the goals. I challenge you to find cases that cause removals that look wrong :D

Context

• See hibernate-pocket, hibernate-pocket-2, hibernate-pocket-3, hibernate-pocket-4,hibernate-pocket-5, hibernate-pocket-6, hibernate-pocket-7, hibernate-pocket-8, hibernate-pocket-9 hibernate-pocket-10 hibernate-pocket-11 hibernate-pocket-12

Some progress upstream Recently Sebastian Reichel at Collabora [1] has made a few related commits, apparently inspired in part by my kvetching on this blog.

Disconnecting and reconnecting PCI busses At some point I noticed error message about the nvme device on resume. I then learned how to disconnect and reconnect PCI buses in Linux. I ended up with something like the following. At least the PCI management seems to work. I can manually disconnect all the PCI busses and rescan to connect them again on a running system. It presumably helps that I am not using the nvme device in this system.

set -x
echo platform >  /sys/power/pm_test
echo reboot > /sys/power/disk
rmmod mt76x2u
sleep 2
echo 1   tee /sys/bus/pci/devices/0003:30:00.0/remove
sleep 2
echo 1   tee /sys/bus/pci/devices/0004:41:00.0/remove
sleep 2
echo 1   tee /sys/bus/pci/devices/0004:40:00.0/remove
sleep 2
echo LSPCI:
lspci -t
sleep 2
echo disk >  /sys/power/state
sleep 2
echo 1   tee /sys/bus/pci/rescan
sleep 2
modprobe mt76x2u

Minimal changes to upstream With the ongoing work at collabora I decided to try a minimal patch stack to get the pocket reform to boot. I added the following 3 commits (available from [3]).

09868a4f2eb (HEAD -> reform-patches) copy pocket-reform dts from reform-debian-packages
152e2ae8a193 pocket/panel: sleep fix v3
18f65da9681c add-multi-display-panel-driver

It does indeed boot and seems stable.

$ uname -a
Linux anthia 6.18.0-rc1+ #19 SMP Thu Oct 16 11:32:04 ADT 2025 aarch64 GNU/Linux

Running the hibernation script above I get no output from the lspci, but seemingly issues with PCI coming back from hibernate:

[  424.645109] PM: hibernation: Allocated 361823 pages for snapshot
[  424.647216] PM: hibernation: Allocated 1447292 kbytes in 3.23 seconds (448.07 MB/s)
[  424.649321] Freezing remaining freezable tasks
[  424.654767] Freezing remaining freezable tasks completed (elapsed 0.003 seconds)
[  424.661070] rk_gmac-dwmac fe1b0000.ethernet end0: Link is Down
[  424.740716] rockchip-dw-pcie a40c00000.pcie: Failed to receive PME_TO_Ack
[  424.742041] PM: hibernation: hibernation debug: Waiting for 5 second(s).
[  430.074757] pci 0004:40:00.0: [1d87:3588] type 01 class 0x060400 PCIe Root Port
F F & Zn [  watchdog: CPU4: Watchdog detected hard LOCKUP on cpu 5
[  456.039004] Modules linked in: xt_CHECKSUM xt_tcpudp nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat x_tables bridge stp llc nf_tables aes_neon_bs aes_neon_blk ccm dwmac_rk binfmt_misc mt76x2_common mt76x02_usb mt76_usb mt76x02_lib mt76 mac80211 rk805_pwrkey snd_soc_tlv320aic31xx snd_soc_simple_card reform2_lpc(OE) libarc4 rockchip_saradc industrialio_triggered_buffer kfifo_buf industrialio cfg80211 rockchip_thermal rockchip_rng hantro_vpu cdc_acm v4l2_vp9 v4l2_jpeg rockchip_rga rfkill snd_soc_rockchip_i2s_tdm videobuf2_dma_sg v4l2_h264 panthor snd_soc_audio_graph_card drm_gpuvm snd_soc_simple_card_utils drm_exec evdev joydev dm_mod nvme_fabrics efi_pstore configfs nfnetlink autofs4 ext4 crc16 mbcache jbd2 btrfs blake2b_generic xor xor_neon raid6_pq mali_dp snd_soc_meson_axg_toddr snd_soc_meson_axg_fifo snd_soc_meson_codec_glue panfrost drm_shmem_helper gpu_sched ao_cec_g12a meson_vdec(C) videobuf2_dma_contig videobuf2_memops v4l2_mem2mem videobuf2_v4l2 videodev
[  456.039060]  videobuf2_common mc dw_hdmi_i2s_audio meson_drm meson_canvas meson_dw_mipi_dsi meson_dw_hdmi mxsfb mux_mmio panel_edp imx_dcss ti_sn65dsi86 nwl_dsi mux_core pwm_imx27 hid_generic usbhid hid xhci_plat_hcd onboard_usb_dev xhci_hcd nvme nvme_core snd_soc_hdmi_codec snd_soc_core nvme_keyring nvme_auth hkdf snd_pcm_dmaengine snd_pcm snd_timer snd soundcore fan53555 rtc_pcf8523 micrel phy_package stmmac_platform stmmac pcs_xpcs rk808_regulator phylink sdhci_of_dwcmshc mdio_devres dw_mmc_rockchip of_mdio sdhci_pltfm phy_rockchip_usbdp fixed_phy dw_mmc_pltfm fwnode_mdio typec phy_rockchip_naneng_combphy phy_rockchip_samsung_hdptx pwm_rockchip sdhci dwc3 libphy dw_wdt dw_mmc ehci_platform rockchip_dfi mdio_bus cqhci ulpi ohci_platform ehci_hcd udc_core ohci_hcd rockchipdrm phy_rockchip_inno_usb2 usbcore dw_hdmi_qp analogix_dp dw_mipi_dsi cpufreq_dt dw_mipi_dsi2 i2c_rk3x usb_common drm_dp_aux_bus [last unloaded: mt76x2u]
[  456.039111] Sending NMI from CPU 4 to CPUs 5:
[  471.942262] page_pool_release_retry() stalled pool shutdown: id 9, 2 inflight 60 sec
[  532.989611] page_pool_release_retry() stalled pool shutdown: id 9, 2 inflight 121 sec

This does look like some progress, probably thanks to Sebastien. Comparing with the logs in hibernate-pocket-12, the resume process is not bailing out complaining about PHY.

Attempt to reapply PCI reset patches Following the procedure in hibernate-pocket-12, I attempted to re-apply the pci reset patches [2]. In particular I followed the hints output by b4. Unfortunately there are too many conflicts now for me to sensibly resolve.

---

1. https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux.git#rockchip-devel
2. https://lore.kernel.org/all/20250715-pci-port-reset-v6-0-6f9cce94e7bb@oss.qualcomm.com/#r
3. https://salsa.debian.org/bremner/collabora-rockchip-3588#reform-patches

Debian Contributions: 2025-09 Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

Updating old Debian Printing software to meet C23 requirements, by Thorsten Alteholz The work of Thorsten fell under the motto gcc15 . Due to the introduction of gcc15 in Debian, the default language version was changed to C23. This means that for example, function declarations without parameters are no longer allowed. As old software, which was created with ANSI C (or C89) syntax, made use of such function declarations, it was a busy month. One could have used something like -std=c17 as compile flags, but this would have just postponed the tasks. As a result Thorsten uploaded modernized versions of ink, nm2ppa and rlpr for the Debian printing team.

Work done to decommission packages.qa.debian.org, by Rapha l Hertzog Rapha l worked to decommission the old package tracking system (packages.qa.debian.org). After figuring out that it was still receiving emails from the bug tracking system (bugs.debian.org), from multiple debian lists and from some release team tools, he reached out to the respective teams to either drop those emails or adjust them so that they are sent to the current Debian Package Tracker (tracker.debian.org).

rebootstrap uses *-for-host, by Helmut Grohne Architecture cross bootstrapping is an ongoing effort that has shaped Debian in various ways over the years. A longer effort to express toolchain dependencies now bears fruit. When cross compiling, it becomes important to express what architecture one is compiling for in Build-Depends. As these packages have become available in trixie , more and more packages add this extra information and in August, the libtool package gained a gfortran-for-host dependency. It was the first package in the essential build closure to adopt this and required putting the pieces together in rebootstrap that now has to build gcc-defaults early on. There still are hundreds of packages whose dependencies need to be updated though.

Miscellaneous contributions

• Rapha l dropped the Build Log Scan integration in tracker.debian.org since it was showing stale data for a while as the underlying service has been discontinued.
• Emilio updated pixman to 0.46.4.
• Emilio coordinated several transitions, and NMUed guestfs-tools to unblock one.
• Stefano uploaded Python 3.14rc3 to Debian unstable. It s not yet used by any packages, but it allows testing the level of support in packages to begin.
• Stefano upgraded almost all of the debian-social infrastructure to Debian trixie .
• Stefano published the sponsorship brochures for DebConf 26.
• Stefano attended the Debian Technical Committee meeting.
• Stefano uploaded routine upstream updates for a handful of Python packages (pycparser, beautifulsoup4, platformdirs, pycparser, python-authlib, python-cffi, python-mitogen, python-resolvelib, python-super-collections, twine).
• Stefano reviewed and responded to DebConf 25 feedback.
• Stefano investigated and fixed a request visibility bug in debian-reimbursements (for admin-altered requests).
• Lucas reviewed a couple of merge requests from external contributors for Go and Ruby packages.
• Lucas updated some ruby packages to its latest upstream version (thin, passenger, and puma is still WIP).
• Lucas set up the build environment to run rebuilds of reverse dependencies of ruby using ruby3.4. As an alternative, he is looking for personal repositories provided by Debusine to perform this task more easily. This is the preparation for the transition to ruby3.4 as the default in Debian.
• Lucas helped on the next round of the Outreachy internship program.
• Helmut sent patches for 30 cross build failures and responded to cross building support questions on the mailing list.
• Helmut continued to maintain rebootstrap. As gcc version 15 became the default, test jobs for version 14 had to be dropped. A fair number of patches were applied to packages and could be dropped.
• Helmut resumed removing RC-buggy packages from unstable and sponsored a termrec upload to avoid its deletion. This work was paused to give packages some time to migrate to forky .
• Santiago reviewed different merge requests created by different contributors. Those MRs include a new test to build reverse dependencies, created by Aquila Macedo as part of his GSoC internship; restore how lintian was used in experimental, thanks Otto Kek l inen; and the fix by Christian Bayle to support again extra repositories in deb822-style sources, whose support was broken with the move to sbuild+unshare last month.
• While doing some new upstream release updates, thanks to Debusine s reverse dependencies autopkgtest checks, Santiago discovered that paramiko 4.0 will introduce a regression in libcloud by the drop of support for the obsolete DSA keys. Santiago finally uploaded to unstable both paramiko 4.0, and a regression fix for libcloud.
• Santiago has taken part in different discussions and meetings for the preparation of DebConf 26. The DebConf 26 local team aims to prepare for the conference with enough time in advance.
• Carles kept working on the missing-package-relations and reporting missing Recommends. He improved the tooling to detect and report bugs creating 269 bugs and followed up comments. 37 bugs have been resolved, others acknowledged. The missing Recommends are a mixture of packages that are gone from Debian, packages that changed name, typos and also packages that were recommended but are not packaged in Debian.
• Carles improved the missing-package-relations to report broken Suggests only for packages that used to be in Debian but are removed from it now. No bugs have been created yet for this case but identified 1320 of them.
• Colin spent much of the month chasing down build/test regressions in various Python packages due to other upgrades, particularly relating to pydantic, python-pytest-asyncio, and rust-pyo3.
• Colin optimized some code in ubuntu-dev-tools (affecting e.g. pull-debian-source) that made O(n) HTTP requests when it could instead make O(1).
• Anupa published Micronews as part of Debian Publicity team work.

Welcome to the September 2025 report from the Reproducible Builds project! Welcome to the very latest report from the Reproducible Builds project. Our monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. As ever, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website. In this report:

1. Reproducible Builds Summit 2025
2. Can t we have nice things?
3. Distribution work
4. Tool development
5. Reproducibility testing framework
6. Upstream patches

Reproducible Builds Summit 2025 Please join us at the upcoming Reproducible Builds Summit, set to take place from October 28th 30th 2025 in Vienna, Austria! We are thrilled to host the eighth edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin, Hamburg and Athens. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving. If you re interesting in joining us this year, please make sure to read the event page which has more details about the event and location. Registration is open until 20th September 2025, and we are very much looking forward to seeing many readers of these reports there!

Can t we have nice things? Debian Developer Gunnar Wolf blogged that George V. Neville-Neil s Kode Vicious column in Communications of the ACM in which reproducible builds is mentioned without needing to introduce it (assuming familiarity across the computing industry and academia) . Titled, Can t we have nice things?, the article mentions:

Once the proper measurement points are known, we want to constrain the system such that what it does is simple enough to understand and easy to repeat. It is quite telling that the push for software that enables reproducible builds only really took off after an embarrassing widespread security issue ended up affecting the entire Internet. That there had already been 50 years of software development before anyone thought that introducing a few constraints might be a good idea is, well, let s just say it generates many emotions, none of them happy, fuzzy ones. [ ]

Distribution work In Debian this month, Johannes Starosta filed a bug against the debian-repro-status package, reporting that it does not work on Debian trixie. (An upstream bug report was also filed.) Furthermore, 17 reviews of Debian packages were added, 10 were updated and 14 were removed this month adding to our knowledge about identified issues. In March s report, we included the news that Fedora would aim for 99% package reproducibility. This change has now been deferred to Fedora 44 according to Phoronix. Lastly, Bernhard M. Wiedemann posted another openSUSE monthly update for their work there.

Tool development diffoscope version 306 was uploaded to Debian unstable by Chris Lamb. It included contributions already covered in previous months as well as some changes by Zbigniew J drzejewski-Szmek to address issues with the fdtump support [ ] and to move away from the deprecated codes.open method. [ ][ ] strip-nondeterminism version 1.15.0-1 was uploaded to Debian unstable by Chris Lamb. It included a contribution by Matwey Kornilov to add support for inline archive files for Erlang s escript [ ]. kpcyrd has released a new version of rebuilderd. As a quick recap, rebuilderd is an automatic build scheduler that tracks binary packages available in a Linux distribution and attempts to compile the official binary packages from their (purported) source code and dependencies. The code for in-toto attestations has been reworked, and the instances now feature a new endpoint that can be queried to fetch the list of public-keys an instance currently identifies itself by. [ ] Lastly, Holger Levsen bumped the Standards-Version field of disorderfs, with no changes needed. [ ][ ]

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In August, however, a number of changes were made by Holger Levsen, including:

• Setting up six new rebuilderd workers with 16 cores and 16 GB RAM each.
• reproduce.debian.net-related:
  • Do not expose pending jobs; they are confusing without explaination. [ ]
  • Add a link to v1 API specification. [ ]
  • Drop rebuilderd-worker.conf on a node. [ ]
  • Allow manual scheduling for any architectures. [ ]
  • Update path to trixie graphs. [ ]
  • Use the same rebuilder-debian.sh script for all hosts. [ ]
  • Add all other suites to all other archs. [ ][ ][ ][ ]
  • Update SSH host keys for new hosts. [ ]
  • Move to the pull184 branch. [ ][ ][ ][ ][ ]
  • Only allow 20 GB cache for workers. [ ]
• OpenWrt-related:
  • Grant developer aparcar full sudo control on the ionos30 node. [ ][ ]
• Jenkins nodes:
  • Add a number of new nodes. [ ][ ][ ][ ][ ]
  • Dont expect /srv/workspace to exist on OSUOSL nodes. [ ]
  • Stop hardcoding IP addresses in munin.conf. [ ]
  • Add maintenance and health check jobs for new nodes. [ ]
  • Document slight changes in IONOS resources usage. [ ]
• Misc:
  • Drop disabled Alpine Linux tests for good. [ ]
  • Move Debian live builds and some other Debian builds to the ionos10 node. [ ]
  • Cleanup some legacy support from releases before Debian trixie. [ ]

In addition, Jochen Sprickerhof made the following changes relating to reproduce.debian.net:

• Do not expose pending jobs on the main site. [ ]
• Switch the frontpage to reference Debian forky [ ], but do not attempt to build Debian forky on the armel architecture [ ].
• Use consistent and up to date rebuilder-debian.sh script. [ ]
• Fix supported worker architectures. [ ]
• Add a basic excuses page. [ ]
• Move to the pull184 branch. [ ][ ][ ][ ]
• Fix a typo in the JavaScript. [ ]
• Update front page for the new v1 API. [ ][ ]

Lastly, Roland Clobus did some maintenance relating to the reproducibility testing of the Debian Live images. [ ][ ][ ][ ]

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Aleksei Burlakov:
  • hawk2
• Bernhard M. Wiedemann:
  • ceph
  • clamav
  • cmake/libarchive
  • kf6-kirigami
  • obs-build/librcc+librcd
• Chris Lamb:
  • #1113809 filed against ms-gsl.
  • #1113813 filed against llama.cpp.
  • #1114638 filed against python-mcstasscript.
  • #1114772 filed against rocm-docs-core.
  • #1114869 filed against octave-optics.
  • #1114950 filed against g2o.
  • #1114999 filed against golang-forgejo-forgejo-levelqueue.
  • #1115999 filed against openrgb.
• Roland Clobus:
  • #1114521 filed against mdadm.

Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org

Changes in version 1.2.0 (2025-10-03)

• New maintainer
• Compiled functions are now registered, .Call() adjusted
• README.md and DESCRIPTION edited and update
• Simple unit tests and continunous integration have been added
• The package documentation site has been recreated using altdoc
• All manual pages for functions now contain \value sections

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

Changes in version 0.3.5 (2025-10-08)

• URL and BugReports fields have been added to DESCRIPTION
• The pbPost function deals more robustly with the case of multiple target emails
• The continuous integration and the README badge have been updated
• The DESCRIPTION file now use Authors@R
• The (encrypted) unit test configuration has been adjusted to reflect the current set of active devices
• The mkdocs-material documentation site is now generated via altdoc

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

• it can update and upgrade the container (currently commented-out)
• it fetches the repository key in ascii form from the llvm.org site
• it creates the sources entry for, here tagged for llvm current (22 at time of writing)
• it sets up the required ~/.R/Makevars to use that compiler
• it installs clang-22 (and clang++-22) (still using the g++ C++ library)

#!/bin/sh

## Update does not hurt but is not strictly needed
#apt update --quiet --quiet
#apt upgrade --yes

## wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key   sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc
## or as we are root in container
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key > /etc/apt/trusted.gpg.d/apt.llvm.org.asc

cat <<EOF >/etc/apt/sources.list.d/llvm-dev.sources
Types: deb
URIs: http://apt.llvm.org/unstable/
# for clang-21 
#   Suites: llvm-toolchain-21
# for current clang
Suites: llvm-toolchain
Components: main
Signed-By: /etc/apt/trusted.gpg.d/apt.llvm.org.asc
EOF

test -d ~/.R   mkdir ~/.R
cat <<EOF >~/.R/Makevars
CLANGVER=-22
# CLANGLIB=-stdlib=libc++
CXX=clang++\$(CLANGVER) \$(CLANGLIB)
CXX11=clang++\$(CLANGVER) \$(CLANGLIB)
CXX14=clang++\$(CLANGVER) \$(CLANGLIB)
CXX17=clang++\$(CLANGVER) \$(CLANGLIB)
CXX20=clang++\$(CLANGVER) \$(CLANGLIB)
CC=clang\$(CLANGVER)
SHLIB_CXXLD=clang++\$(CLANGVER) \$(CLANGLIB)
EOF

apt update
apt install --yes clang-22

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

// [[Rcpp::export]]
Rcpp::IntegerVector armadillo_version(bool single)  

    // These are declared as constexpr in Armadillo which actually does not define them
    // They are also defined as macros in arma_version.hpp so we just use that
    const unsigned int major = ARMA_VERSION_MAJOR;
    const unsigned int minor = ARMA_VERSION_MINOR;
    const unsigned int patch = ARMA_VERSION_PATCH;

    if (single)  
        return Rcpp::wrap(10000 * major + 100 * minor + patch) ;
      else  
        return Rcpp::IntegerVector::create(Rcpp::Named("major") = major,
                                           Rcpp::Named("minor") = minor,
                                           Rcpp::Named("patch") = patch);
     
 

> library(RcppArmadillo)
> armadillo_version(FALSE)
major minor patch 
   15     0     2 
> armadillo_version(TRUE)
[1] 150002
>

// [[Rcpp::export]]
Rcpp::List armadilloVersion()  
    // create a vector of major, minor, patch
    auto ver = Rcpp::IntegerVector::create(ARMA_VERSION_MAJOR, ARMA_VERSION_MINOR, ARMA_VERSION_PATCH);
    // and place it in a list (as e.g. packageVersion() in R returns)
    auto lst = Rcpp::List::create(ver);
    // and class it as 'package_version' accessing print() etc methods
    lst.attr("class") = Rcpp::CharacterVector::create("package_version", "numeric_version");
    return lst;
 

• create the integeer vector of known dimensions and compile-time known value
• embed it in a list (as that is what the R type expects)
• set the S3 class which is easy because Rcpp accesses attributes and create character vectors

> RcppDE:::armadilloVersion()
[1]  15.0.2 
> RcppDE:::armadilloVersion() >= "15.0.0"
[1] TRUE
>

> unclass(RcppDE:::armadilloVersion())
[[1]]
[1] 15  0  2

> 

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

During 2025-03-21-another-home-outage, I reflected upon what's a properly ran service and blurted out what turned out to be something important I want to outline more. So here it is, again, on its own for my own future reference.

1. backups
2. documentation
3. monitoring
4. automation
5. high availability (HA)

The five components of a proper service

Backups Duh. If data is maliciously or accidentally destroyed, you need a copy somewhere. Preferably in a way that malicious Joe can't get to. This is harder than you think.

Documentation I have an entire template for this. Essentially, it boils down to using https://diataxis.fr/ and this "audit" guide. For me, the most important parts are:

• disaster recovery (includes backups, probably)
• playbook
• install/upgrade procedures (see automation)

You probably know this is hard, and this is why you're not doing it. Do it anyways, you'll think it sucks, it will grow out of sync with reality, but you'll be really grateful for whatever scraps you wrote when you're in trouble. Any docs, in other words, is better than no docs, but are no excuse for doing the work correctly.

Monitoring If you don't have monitoring, you'll know it fails too late, and you won't know it recovers. Consider high availability, work hard to reduce noise, and don't have machine wake people up, that's literally torture and is against the Geneva convention. Consider predictive algorithm to prevent failures, like "add storage within 2 weeks before this disk fills up". This is also harder than you think.

Automation Make it easy to redeploy the service elsewhere. Yes, I know you have backups. That is not enough: that typically restores data and while it can also include configuration, you're going to need to change things when you restore, which is what automation (or call it "configuration management" if you will) will do for you anyways. This also means you can do unit tests on your configuration, otherwise you're building legacy. This is probably as hard as you think.

High availability Make it not fail when one part goes down. Eliminate single points of failures. This is easier than you think, except for storage and DNS ("naming things" not "HA DNS", that is easy), which, I guess, means it's harder than you think too.

Assessment In the above 5 items, I currently check two in my lab:

1. backups
2. documentation

And barely: I'm not happy about the offsite backups, and my documentation is much better at work than at home (and even there, I have a 15 year backlog to catchup on). I barely have monitoring: Prometheus is scraping parts of the infra, but I don't have any sort of alerting -- by which I don't mean "electrocute myself when something goes wrong", I mean "there's a set of thresholds and conditions that define an outage and I can look at it". Automation is wildly incomplete. My home server is a random collection of old experiments and technologies, ranging from Apache with Perl and CGI scripts to Docker containers running Golang applications. Most of it is not Puppetized (but the ratio is growing). Puppet itself introduces a huge attack vector with kind of catastrophic lateral movement if the Puppet server gets compromised. And, fundamentally, I am not sure I can provide high availability in the lab. I'm just this one guy running my home network, and I'm growing older. I'm thinking more about winding things down than building things now, and that's just really sad, because I feel we're losing (well that escalated quickly).

Side note about Tor The above applies to my personal home lab, not work! At work, of course, it's another (much better) story:

1. all services have backups
2. lots of services are well documented, but not all
3. most services have at least basic monitoring
4. most services are Puppetized, but not crucial parts (DNS, LDAP, Puppet itself), and there are important chunks of legacy coupling between various services that make the whole system brittle
5. most websites, DNS and large parts of email are highly available, but key services like the the Forum, GitLab and similar applications are not HA, although most services run under replicated VMs that can trivially survive a total, single-node hardware failure (through Ganeti and DRBD)

Debian

Whilst I didn t get a chance to do much, here s still a few things that I worked on:

• Helped sponsor a patch for net-tools for Helge.
  • https://salsa.debian.org/debian/net-tools/-/merge_requests/5.
• Asssited Anshul in his interest to do a Go 1.25 transition.
• Mentoring for newcomers.
• Moderation of -project mailing list.

---

Ubuntu

I joined Canonical to work on Ubuntu full-time back in February 2021. Whilst I can t give a full, detailed list of things I did, here s a quick TL;DR of what I did:

• Successfully and timely released 25.10 (Questing Quokka) Beta! \o/
• Continued to hold weekly release syncs, et al.
• Granted FFe and triaged a bunch of other bugs from both, Release team and Archive Admin POV. :)
• 360s were fab - I was a peak performer again. Yay!
• Preparing for the 25.10 Release sprints in London and then the Summit.
• Roadmap planning for the Release team.

---

Debian (E)LTS

This month I have worked 16.50 hours on Debian Long Term Support (LTS) and 05.50 hours on its sister Extended LTS project and did the following things:

• [E/LTS] Frontdesk duty from 22nd September to 28th September.
  • Triaged lemonldap-ng, ghostscript, dovecot, node-ip, webkit2gtk, wpewebkit, libscram-java, keras, openbabel, gegl, tiff, zookeeper, squid, ogre-1.12, mapserver, ruby-rack.
  • Auto-EOL d a few packages.
  • Also circled back on previously opened ticket for supported packages for ELTS.
  • Partially reviewed and added comment on Emilio s MP.
  • Re-visited an old thread (in order to fully close it) about issues being fixed in buster & bookworm but not in bullseye. And brought it up in the LTS meeting, too.
• [LTS] Partook in some internal discussions about introducing support for handling severity of CVEs, et al.
  • Santiago had asked for an input from people doing FD so spent some time reflecting on his proposal and getting back with thoughts and suggestions.
• [LTS] Helped Lee with testing gitk and git-gui aspects of his git update.
• [LTS] Attended the monthly LTS meeting on IRC. Summary here.
  • It was also followed by a 40-minute discussion of technical questions/reviews/discussions - which in my opinion was pretty helpful. :)
• [LTS] Prepared the LTS update for wordpress, bumping the package from 5.7.11 to 5.7.13.
  • Prepared an update for stable, Craig approved. Was waiting on the Security team s +1 to upload.
  • Now we ve waited enough that we have new CVEs. Oh well.
• [ELTS] Finally setup debusine for ELTS uploads.
  • Since I use Ubuntu, this required installing debusine* from bookworm-backport but that required Python 3.11.
  • So I had to upgrade from Jammy (22.04) to Noble (24.04) - which was anyway pending.. :)
  • And then followed the docs to configure it. \o/
• [E/LTS] Started working on new ruby-rack CVE.

---

Until next time.
:wq for today.

$ git diff --shortstat v1.11.5
 493 files changed, 25015 insertions(+), 21135 deletions(-)

New outlet service The major change in Akvorado 2.0 is splitting the inlet service into two parts: the inlet and the outlet. Previously, the inlet handled all flow processing: receiving, decoding, and enrichment. Flows were then sent to Kafka for storage in ClickHouse:

Akvorado flow processing before the introduction of the outlet service

Network flows reach the inlet service using UDP, an unreliable protocol. The inlet must process them fast enough to avoid losing packets. To handle a high number of flows, the inlet spawns several sets of workers to receive flows, fetch metadata, and assemble enriched flows for Kafka. Many configuration options existed for scaling, which increased complexity for users. The code needed to avoid blocking at any cost, making the processing pipeline complex and sometimes unreliable, particularly the BMP receiver.¹ Adding new features became difficult without making the problem worse.² In Akvorado 2.0, the inlet receives flows and pushes them to Kafka without decoding them. The new outlet service handles the remaining tasks:

Akvorado flow processing after the introduction of the outlet service

This change goes beyond a simple split:³ the outlet now reads flows from Kafka and pushes them to ClickHouse, two tasks that Akvorado did not handle before. Flows are heavily batched to increase efficiency and reduce the load on ClickHouse using ch-go, a low-level Go client for ClickHouse. When batches are too small, asynchronous inserts are used (e20645). The number of outlet workers scales dynamically (e5a625) based on the target batch size and latency (50,000 flows and 5 seconds by default). This new architecture also allows us to simplify and optimize the code. The outlet fetches metadata synchronously (e20645). The BMP component becomes simpler by removing cooperative multitasking (3b9486). Reusing the same RawFlow object to decode protobuf-encoded flows from Kafka reduces pressure on the garbage collector (8b580f). The effect on Akvorado s overall performance was somewhat uncertain, but a user reported 35% lower CPU usage after migrating from the previous version, plus resolution of the long-standing BMP component issue.

Other changes This new version includes many miscellaneous changes, such as completion for source and destination ports (f92d2e), and automatic restart of the orchestrator service (0f72ff) when configuration changes to avoid a common pitfall for newcomers. Let s focus on some key areas for this release: observability, documentation, CI, Docker, Go, and JavaScript.

Observability Akvorado exposes metrics to provide visibility into the processing pipeline and help troubleshoot issues. These are available through Prometheus HTTP metrics endpoints, such as /api/v0/inlet/metrics. With the introduction of the outlet, many metrics moved. Some were also renamed (4c0b15) to match Prometheus best practices. Kafka consumer lag was added as a new metric (e3a778). If you do not have your own observability stack, the Docker Compose setup shipped with Akvorado provides one. You can enable it by activating the profiles introduced for this purpose (529a8f). The prometheus profile ships Prometheus to store metrics and Alloy to collect them (2b3c46, f81299, and 8eb7cd). Redis and Kafka metrics are collected through the exporter bundled with Alloy (560113). Other metrics are exposed using Prometheus metrics endpoints and are automatically fetched by Alloy with the help of some Docker labels, similar to what is done to configure Traefik. cAdvisor was also added (83d855) to provide some container-related metrics. The loki profile ships Loki to store logs (45c684). While Alloy can collect and ship logs to Loki, its parsing abilities are limited: I could not find a way to preserve all metadata associated with structured logs produced by many applications, including Akvorado. Vector replaces Alloy (95e201) and features a domain-specific language, VRL, to transform logs. Annoyingly, Vector currently cannot retrieve Docker logs from before it was started. Finally, the grafana profile ships Grafana, but the shipped dashboards are broken. This is planned for a future version.

Documentation The Docker Compose setup provided by Akvorado makes it easy to get the web interface up and running quickly. However, Akvorado requires a few mandatory steps to be functional. It ships with comprehensive documentation, including a chapter about troubleshooting problems. I hoped this documentation would reduce the support burden. It is difficult to know if it works. Happy users rarely report their success, while some users open discussions asking for help without reading much of the documentation. In this release, the documentation was significantly improved.

$ git diff --shortstat v1.11.5 -- console/data/docs
 10 files changed, 1873 insertions(+), 1203 deletions(-)

The documentation was updated (fc1028) to match Akvorado s new architecture. The troubleshooting section was rewritten (17a272). Instructions on how to improve ClickHouse performance when upgrading from versions earlier than 1.10.0 was added (5f1e9a). An LLM proofread the entire content (06e3f3). Developer-focused documentation was also improved (548bbb, e41bae, and 871fc5). From a usability perspective, table of content sections are now collapsable (c142e5). Admonitions help draw user attention to important points (8ac894).

Example of use of admonitions in Akvorado's documentation

Continuous integration This release includes efforts to speed up continuous integration on GitHub. Coverage and race tests run in parallel (6af216 and fa9e48). The Docker image builds during the tests but gets tagged only after they succeed (8b0dce).

GitHub workflow to test and build Akvorado

End-to-end tests (883e19) ensure the shipped Docker Compose setup works as expected. Hurl runs tests on various HTTP endpoints, particularly to verify metrics (42679b and 169fa9). For example:

## Test inlet has received NetFlow flows
GET http://127.0.0.1:8080/prometheus/api/v1/query
[Query]
query: sum(akvorado_inlet_flow_input_udp_packets_total job="akvorado-inlet",listener=":2055" )
HTTP 200
[Captures]
inlet_receivedflows: jsonpath "$.data.result[0].value[1]" toInt
[Asserts]
variable "inlet_receivedflows" > 10
## Test inlet has sent them to Kafka
GET http://127.0.0.1:8080/prometheus/api/v1/query
[Query]
query: sum(akvorado_inlet_kafka_sent_messages_total job="akvorado-inlet" )
HTTP 200
[Captures]
inlet_sentflows: jsonpath "$.data.result[0].value[1]" toInt
[Asserts]
variable "inlet_sentflows" >=   inlet_receivedflows  

Docker Akvorado ships with a comprehensive Docker Compose setup to help users get started quickly. It ensures a consistent deployment, eliminating many configuration-related issues. It also serves as a living documentation of the complete architecture. This release brings some small enhancements around Docker:

• an example to configure Traefik for TLS with Let s Encrypt (1a27bb),
• HTTP compression (bee9a5), and
• support of IPv6 (a74a41).

Previously, many Docker images were pulled from the Bitnami Containers library. However, VMWare acquired Bitnami in 2019 and Broadcom acquired VMWare in 2023. As a result, Bitnami images were deprecated in less than a month. This was not really a surprise⁴. Previous versions of Akvorado had already started moving away from them. In this release, the Apache project s Kafka image replaces the Bitnami one (1eb382). Thanks to the switch to KRaft mode, Zookeeper is no longer needed (0a2ea1, 8a49ca, and f65d20). Akvorado s Docker images were previously compiled with Nix. However, building AArch64 images on x86-64 is slow because it relies on QEMU userland emulation. The updated Dockerfile uses multi-stage and multi-platform builds: one stage builds the JavaScript part on the host platform, one stage builds the Go part cross-compiled on the host platform, and the final stage assembles the image on top of a slim distroless image (268e95 and d526ca).

# This is a simplified version
FROM --platform=$BUILDPLATFORM node:20-alpine AS build-js
RUN apk add --no-cache make
WORKDIR /build
COPY console/frontend console/frontend
COPY Makefile .
RUN make console/data/frontend
FROM --platform=$BUILDPLATFORM golang:alpine AS build-go
RUN apk add --no-cache make curl zip
WORKDIR /build
COPY . .
COPY --from=build-js /build/console/data/frontend console/data/frontend
RUN go mod download
RUN make all-indep
ARG TARGETOS TARGETARCH TARGETVARIANT VERSION
RUN make
FROM gcr.io/distroless/static:latest
COPY --from=build-go /build/bin/akvorado /usr/local/bin/akvorado
ENTRYPOINT [ "/usr/local/bin/akvorado" ]

When building for multiple platforms with --platform linux/amd64,linux/arm64,linux/arm/v7, the build steps until the highlighted line execute only once for all platforms. This significantly speeds up the build. Akvorado now ships Docker images for these platforms: linux/amd64, linux/amd64/v3, linux/arm64, and linux/arm/v7. When requesting ghcr.io/akvorado/akvorado, Docker selects the best image for the current CPU. On x86-64, there are two choices. If your CPU is recent enough, Docker downloads linux/amd64/v3. This version contains additional optimizations and should run faster than the linux/amd64 version. It would be interesting to ship an image for linux/arm64/v8.2, but Docker does not support the same mechanism for AArch64 yet (792808).

Go This release includes many changes related to Go but not visible to the users.

Toolchain In the past, Akvorado supported the two latest Go versions, preventing immediate use of the latest enhancements. The goal was to allow users of stable distributions to use Go versions shipped with their distribution to compile Akvorado. However, this became frustrating when interesting features, like go tool, were released. Akvorado 2.0 requires Go 1.25 (77306d) but can be compiled with older toolchains by automatically downloading a newer one (94fb1c).⁵ Users can still override GOTOOLCHAIN to revert this decision. The recommended toolchain updates weekly through CI to ensure we get the latest minor release (5b11ec). This change also simplifies updates to newer versions: only go.mod needs updating. Thanks to this change, Akvorado now uses wg.Go() (77306d) and I have started converting some unit tests to the new test/synctest package (bd787e, 7016d8, and 159085).

Testing When testing equality, I use a helper function Diff() to display the differences when it fails:

got := input.Keys()
expected := []int 1, 2, 3 
if diff := helpers.Diff(got, expected); diff != ""  
    t.Fatalf("Keys() (-got, +want):\n%s", diff)
 

This function uses kylelemons/godebug. This package is no longer maintained and has some shortcomings: for example, by default, it does not compare struct private fields, which may cause unexpectedly successful tests. I replaced it with google/go-cmp, which is stricter and has better output (e2f1df).

Another package for Kafka Another change is the switch from Sarama to franz-go to interact with Kafka (756e4a and 2d26c5). The main motivation for this change is to get a better concurrency model. Sarama heavily relies on channels and it is difficult to understand the lifecycle of an object handed to this package. franz-go uses a more modern approach with callbacks⁶ that is both more performant and easier to understand. It also ships with a package to spawn fake Kafka broker clusters, which is more convenient than the mocking functions provided by Sarama.

Improved routing table for BMP To store its routing table, the BMP component used kentik/patricia, an implementation of a patricia tree focused on reducing garbage collection pressure. gaissmai/bart is a more recent alternative using an adaptation of [Donald Knuth s ART algorithm][] that promises better performance and delivers it: 90% faster lookups and 27% faster insertions (92ee2e and fdb65c). Unlike kentik/patricia, gaissmai/bart does not help efficiently store values attached to each prefix. I adapted the same approach as kentik/patricia to store route lists for each prefix: store a 32-bit index for each prefix, and use it to build a 64-bit index for looking up routes in a map. This leverages Go s efficient map structure. gaissmai/bart also supports a lockless routing table version, but this is not simple because we would need to extend this to the map storing the routes and to the interning mechanism. I also attempted to use Go s new unique package to replace the intern package included in Akvorado, but performance was worse.⁷

Miscellaneous Previous versions of Akvorado were using a custom Protobuf encoder for performance and flexibility. With the introduction of the outlet service, Akvorado only needs a simple static schema, so this code was removed. However, it is possible to enhance performance with planetscale/vtprotobuf (e49a74, and 8b580f). Moreover, the dependency on protoc, a C++ program, was somewhat annoying. Therefore, Akvorado now uses buf, written in Go, to convert a Protobuf schema into Go code (f4c879). Another small optimization to reduce the size of the Akvorado binary by 10 MB was to compress the static assets embedded in Akvorado in a ZIP file. It includes the ASN database, as well as the SVG images for the documentation. A small layer of code makes this change transparent (b1d638 and e69b91).

JavaScript Recently, two large supply-chain attacks hit the JavaScript ecosystem: one affecting the popular packages chalk and debug and another impacting the popular package @ctrl/tinycolor. These attacks also exist in other ecosystems, but JavaScript is a prime target due to heavy use of small third-party dependencies. The previous version of Akvorado relied on 653 dependencies. npm-run-all was removed (3424e8, 132 dependencies). patch-package was removed (625805 and e85ff0, 69 dependencies) by moving missing TypeScript definitions to env.d.ts. eslint was replaced with oxlint, a linter written in Rust (97fd8c, 125 dependencies, including the plugins). I switched from npm to Pnpm, an alternative package manager (fce383). Pnpm does not run install scripts by default⁸ and prevents installing packages that are too recent. It is also significantly faster.⁹ Node.js does not ship Pnpm but it ships Corepack, which allows us to use Pnpm without installing it. Pnpm can also list licenses used by each dependency, removing the need for license-compliance (a35ca8, 42 dependencies). For additional speed improvements, beyond switching to Pnpm and Oxlint, Vite was replaced with its faster Rolldown version (463827). After these changes, Akvorado only pulls 225 dependencies.

Next steps I would like to land three features in the next version of Akvorado:

• Add Grafana dashboards to complete the observability stack. See issue #1906 for details.
• Integrate OVH s Grafana plugin by providing a stable API for such integrations. Akvorado s web console would still be useful for browsing results, but if you want to build and share dashboards, you should switch to Grafana. See issue #1895.
• Move some work currently done in ClickHouse (custom dictionaries, GeoIP and IP enrichment) back into the outlet service. This should give more flexibility for adding features like the one requested in issue #1030. See issue #2006.

---

I started working on splitting the inlet into two parts more than one year ago. I found more motivation in recent months, partly thanks to Claude Code, which I used as a rubber duck. Almost none of the produced code was kept:¹⁰ it is like an intern who does not learn.

---

1. Many attempts were made to make the BMP component both performant and not blocking. See for example PR #254, PR #255, and PR #278. Despite these efforts, this component remained problematic for most users. See issue #1461 as an example.
2. Some features have been pushed to ClickHouse to avoid the processing cost in the inlet. See for example PR #1059.
3. This is the biggest commit:

   $ git show --shortstat ac68c5970e2c   tail -1
   231 files changed, 6474 insertions(+), 3877 deletions(-)
   

4. Broadcom is known for its user-hostile moves. Look at what happened with VMWare.
5. As a Debian developer, I dislike these mechanisms that circumvent the distribution package manager. The final straw came when Go 1.25 spent one month in the Debian NEW queue, an arbitrary mechanism I don t like at all.
6. In the early years of Go, channels were heavily promoted. Sarama was designed during this period. A few years later, a more nuanced approach emerged. See notably Go channels are bad and you should feel bad.
7. This should be investigated further, but my theory is that the intern package uses 32-bit integers, while unique uses 64-bit pointers. See commit 74e5ac.
8. This is also possible with npm. See commit dab2f7.
9. An even faster alternative is Bun, but it is less available.
10. The exceptions are part of the code for the admonition blocks, the code for collapsing the table of content, and part of the documentation.

Changes in RcppArmadillo version 15.0.2-2 (2025-09-18)

• Minor update to skeleton Makevars,Makevars.win
• Update README.md to mention ldlasb2 repository
• Minor documentation update (#487)
• Synchronized with Armadillo upstream (#488)
• Refine Armadillo version selection in coordination with CRAN maintainers to support transition towards Armadillo 15.0.*

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

What s a pixel? How would you define a pixel these days? Probably something like a uniquely-addressable square dot in a two-dimensional grid . In the world of VGA and CRTs, that was just a logical abstraction. We got an API centered around that because it was convenient. But, down the VGA cable and on the device, that s not what a pixel was. A pixel, back then, was a time interval. On a multisync monitor, which were common except in the very early days of VGA, the timings could be adjusted which produced logical pixels of different sizes. Those screens often had a maximum resolution but not necessarily a native resolution in the sense that an LCD panel does. Different timings produced different-sized pixels with equal clarity (or, on cheaper monitors, equal fuzziness). A side effect of this was that pixels need not be square. And, in fact, in the standard DOS VGA 80x25 text mode, they weren t. You might be seeing why DVI, DisplayPort, and HDMI replaced VGA for LCD monitors: with a VGA cable, you did a pixel-to-analog-timings conversion, then the display did a timings-to-pixels conversion, and this process could be a bit lossy. (Hence why you sometimes needed to fill the screen with an image and push the center button on those older LCD screens) (Note to the pedantically-inclined: yes I am aware that I have simplified several things here; for instance, a color LCD pixel is made up of approximately 3 sub-dots of varying colors, and that things like color eInk displays have two pixel grids with different sizes of pixels layered atop each other, and printers are another confusing thing altogether, and and and . MOST PEOPLE THINK OF A PIXEL AS A DOT THESE DAYS, OK?)

What was DOS text mode? We think of this as the standard display: 80 columns wide and 25 rows tall. 80x25. By the time Linux came along, the standard Linux console was VGA text mode something like the 4th incarnation of text modes on PCs (after CGA, MDA, and EGA). VGA also supported certain other sizes of characters giving certain other text dimensions, but if I cover all of those, this will explode into a ridiculously more massive page than it already is. So to display text on an 80x25 DOS VGA system, ultimately characters and attributes were written into the text buffer in memory. The VGA system then rendered it to the display as a 720x400 image (at 70Hz) with non-square pixels such that the result was approximately a 4:3 aspect ratio. The font used for this rendering was a bitmapped one using 8x16 cells. You might do some math here and point out that 8 * 80 is only 640, and you d be correct. The fonts were 8x16 but the rendered cells were 9x16. The extra pixel was normally used for spacing between characters. However, in line graphics mode, characters 0xC0 through 0xDF repeated the 8th column in the position of the 9th, allowing the continuous line-drawing characters we re used to from TUIs.

Problems rendering DOS fonts on modern systems By now, you re probably seeing some of the issues we have rendering DOS screens on more modern systems. These aren t new at all; I remember some of these from back in the days when I ran OS/2, and I think also saw them on various terminals and consoles in OS/2 and Windows. Some issues you d encounter would be:

• Incorrect aspect ratio caused by using the original font and rendering it using 1:1 square pixels (resulting in a squashed appearance)
• Incorrect aspect ratio for ANOTHER reason, caused by failing to render column 9, resulting in text that is overall too narrow
• Characters appearing to be touching each other when they shouldn t (failing to render column 9; looking at you, dosbox)
• Gaps between line drawing characters that should be continuous, caused by rendering column 9 as empty space in all cases

Character set issues DOS was around long before Unicode was. In the DOS world, there were codepages that selected the glyphs for roughly the high half of the 256 possible characters. CP437 was the standard for the USA; others existed for other locations that needed different characters. On Unix, the USA pre-Unicode standard was Latin-1. Same concept, but with different character mappings. Nowadays, just about everything is based on UTF-8. So, we need some way to map our CP437 glyphs into Unicode space. If we are displaying DOS-based content, we ll also need a way to map CP437 characters to Unicode for display later, and we need these maps to match so that everything comes out right. Whew. So, let s get on with setting this up!

Selecting the proper video mode As explained in my previous post, proper hardware support for DOS text mode is limited to x86 machines that do not use UEFI. Non-x86 machines, or x86 machines with UEFI, simply do not contain the necessary support for it. As these are now standard, most of the time, the text console you see on Linux is actually the kernel driving the video hardware in graphics mode, and doing the text rendering in software. That s all well and good, but it makes it quite difficult to actually get an 80x25 console. First, we need to be running at 720x400. This is where I ran into difficulty last time. I realized that my laptop s LCD didn t advertise any video modes other than its own native resolution. However, almost all external monitors will, and 720x400@70 is a standard VGA mode from way back, so it should be well-supported. You need to find the Linux device name for your device. You can look at the possible devices with ls -l /sys/class/drm. If you also have a GUI, xrandr may help too. But in any case, each directory under /sys/class/drm has a file named modes, and if you cat them all, you will eventually come across one with a bunch of modes defined. Drop the leading card0 or whatever from the directory name, and that s your device. (Verify that 720x400 is in modes while you re at it.) Now, you re going to edit /etc/default/grub and add something like this to GRUB_CMDLINE_LINUX_DEFAULT:

video=DP-1:720x400@70

Of course, replace DP-1 with whatever your device is. Now you can run update-grub and reboot. You should have a 720x400 display. At first, I thought I had succeeded by using Linux s built-in VGA font with that mode. But it looked too tall. After noticing that repeated 0s were touching, I got suspicious about the missing 9th column in the cells. stty -a showed that my screen was 90x25, which is exactly what it would show if I was using 8x16 instead of 9x16 cells. Sooo . I need to prepare a 9x16 font.

Preparing a font Here s where it gets complicated. I ll give you the simple version and the hard mode. The simple mode is this: Download https://www.complete.org/downloads/CP437-VGA.psf.gz and stick it in /usr/local/etc, then skip to the Activating the font section below. The font assembled here is based on the Ultimate Oldschool PC Font Pack v2.2, which is (c) 2016-2020 VileR and licensed under Creative Commons Attribution-ShareAlike 4.0 International License. My psf file is derived from this using the instructions below.

Building it yourself First, install some necessary software: apt-get install fontforge bdf2psf Start by going to the Oldschool PC Font Pack Download page. Download oldschool_pc_font_pack_v2.2_FULL.zip and unpack it. The file we re interested in is otb - Bm (linux bitmap)/Bm437_IBM_VGA_9x16.otb. Open it in fontforge by running fontforge BmPlus_IBM_VGA_9x16.otb. When it asks if you will load the bitmap fonts, hit select all, then yes. Go to File -> generate fonts. Save in a BDF, no need for outlines, and use guess for resolution. Now you have a file such as Bm437_IBM_VGA_9x16-16.bdf. Excellent. Now we need to generate a Unicode map file. We will make sure this matches the system s by enumerating every character from 0x00 to 0xFF, converting it from CP437 to Unicode, and writing the appropriate map. Here s a Python script to do that:

for i in range(0, 256):
    cp437b = b'%c' % i
    uni = ord(cp437b.decode('cp437'))
    print(f"U+ uni:04x ")

Save that file as genmap.py and run python3 genmap.py > cp437-uni. Now, we re ready to build the psf file:

bdf2psf --fb Bm437_IBM_VGA_9x16-16.bdf \
  /dev/null cp437-uni 256 CP437-VGA.psf

By convention, we normally store these files gzipped, so gzip CP437-VGA.psf. You can test it on the console with setfont CP437-VGA.psf.gz. Now copy this file into /usr/local/etc.

Activating the font Now, edit /etc/default/console-setup. It should look like this:

# CONFIGURATION FILE FOR SETUPCON

# Consult the console-setup(5) manual page.

ACTIVE_CONSOLES="/dev/tty[1-6]"

CHARMAP="UTF-8"

CODESET="Lat15"
FONTFACE="VGA"
FONTSIZE="8x16"
FONT=/usr/local/etc/CP437-VGA.psf.gz

VIDEOMODE=

# The following is an example how to use a braille font
# FONT='lat9w-08.psf.gz brl-8x8.psf'

At this point, you should be able to reboot. You should have a proper 80x25 display! Log in and run stty -a to verify it is indeed 80x25.

Using and testing CP437 Part of the point of CP437 is to be able to access BBSs, ANSI art, and similar. Now, remember, the Linux console is still in UTF-8 mode, so we have to translate CP437 to UTF-8, then let our font map translate it back to CP437. A weird trip, but it works. Let s test it using the Textfiles ANSI art collection. In the artworks section, I randomly grabbed a file near the top: borgman.ans. Download that, and display with:

clear; iconv -f CP437 -t UTF-8 < borgman.ans

You should see something similar to but actually more accurate than the textfiles PNG rendering of it, which you ll note has an incorrect aspect ratio and some rendering issues. I spot-checked with a few others and they seemed to look good. belinda.ans in particular tries quite a few characters and should give you a good sense if it is working.

Use with interactive programs That s all well and good, but you re probably going to want to actually use this with some interactive program that expects CP437. Maybe Minicom, Kermit, or even just telnet? For this, you ll want to apt-get install luit. luit maps CP437 (or any other encoding) to UTF-8 for display, and then of course the Linux console maps UTF-8 back to the CP437 font. Here s a way you can repeat the earlier experiment using luit to run the cat program:

clear; luit -encoding CP437 cat borgman.ans

You can run any command under luit. You can even run luit -encoding CP437 bash if you like. If you do this, it is probably a good idea to follow my instructions on generating locales on my post on serial terminals, and then within luit, set LANG=en_us.IBM437. But note especially that you can run programs like minicom and others for accessing BBSs under luit.

Final words This gave you a nice DOS-type console. Although it doesn t have glyphs for many codepoints, it does run in UTF-8 mode and therefore is compatible with modern software. You can achieve greater compatibility with more UTF-8 codepoints with the DOS font, at the expense of accuracy of character rendering (especially for the double-line drawing characters) by using /usr/share/bdf2psf/standard.equivalents instead of /dev/null in the bdf2psf command. Or you could go for another challenge, such as using the DEC vt-series fonts for coverage of ISO-8859-1. But just using fonts extracted from DEC ROM won t work properly, because DEC terminals had even more strangeness going on than DOS fonts.

1. You can use a traditional display and keyboard just long enough to input the kernel parameters described above
2. You can edit the bootloader configuration on the installer s filesystem prior to booting from it

serial --unit=0 --speed=57600 --word=8 --parity=no --stop=1
terminal_input console serial
terminal_output console serial

[Service]
ExecStart=
ExecStart=-/sbin/agetty --wait-cr -8 -h -L=always %I 57600 vt510

• wait-cr means to wait for the user to press Return at the terminal before attempting to display the login prompt
• -8 tells it to assume 8-bit mode on the serial line
• -h enables hardware flow control
• -L=always enables local line mode, disabling monitoring of modem control lines
• %I substitutes the name of the port from systemd
• 57600 gives the desired speed, and vt510 gives the desired setting for the TERM environment variable

GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=57600 --word=8 --parity=no --stop=1"

en_US ISO-8859-1
en_US.IBM437 IBM437
en_US.UTF-8 UTF-8 

if [  tty  = "/dev/ttyS0" -o "$TERM" = "vt510" ]; then
        stty -iutf8
        # might add ixon ixoff
        export LANG=en_US
        export MANOPT="-E ascii"
        stty rows 25
fi
if [ "$TERM" = "screen" -o "$TERM" = "vt100" ]; then
    export LANG=en_US.utf8
fi

defencoding UTF-8
startup_message off
vbell off
termcapinfo * XC=B%, -,
maptimeout 5
bindkey -k ku stuff ^[OA
bindkey -k kd stuff ^[OB
bindkey -k kr stuff ^[OC
bindkey -k kl stuff ^[OD

make install-build-deps

$ git clone https://salsa.debian.org/cloud-team/debian-vagrant-images.git

$ make install-build-deps

$ make help
To run this makefile, run:
   make <DIST>-<CLOUD>-<ARCH>
  WHERE <DIST> is bullseye, buster, stretch, sid or testing
    And <CLOUD> is azure, ec2, gce, generic, genericcloud, nocloud, vagrant, vagrantcontrib
    And <ARCH> is amd64, arm64, ppc64el
Set DESTDIR= to write images to given directory.
$ make trixie-vagrant-amd64
umask 022; \
./bin/debian-cloud-images build \
  trixie vagrant amd64 \
  --build-id vagrant-cloud-images-master \
  --build-type official
usage: debian-cloud-images build
debian-cloud-images build: error: argument RELEASE: invalid value: trixie
make: *** [Makefile:22: trixie-vagrant-amd64] Error 2

$ git clone https://salsa.debian.org/debian/debian-vagrant-images.git
Cloning into &aposdebian-vagrant-images&apos...
remote: Enumerating objects: 5310, done.
remote: Counting objects: 100% (256/256), done.
remote: Compressing objects: 100% (96/96), done.
remote: Total 5310 (delta 141), reused 241 (delta 135), pack-reused 5054 (from 1)
Receiving objects: 100% (5310/5310), 629.81 KiB   548.00 KiB/s, done.
Resolving deltas: 100% (2875/2875), done.
$ cd debian-vagrant-images/
$ git checkout 8975eb0 #the commit id of MR 

$ make help
To run this makefile, run:
   make <DIST>-<CLOUD>-<ARCH>
  WHERE <DIST> is bullseye, buster, stretch, sid or testing
    And <CLOUD> is azure, ec2, gce, generic, genericcloud, nocloud, vagrant, vagrantcontrib
    And <ARCH> is amd64, arm64, ppc64el
Set DESTDIR= to write images to given directory.
$ make trixie-vagrant-amd64
umask 022; \
./bin/debian-cloud-images build \
  trixie vagrant amd64 \
  --build-id vagrant-cloud-images-master \
  --build-type official
2025-09-17 00:36:25,919 INFO Adding class DEBIAN
2025-09-17 00:36:25,919 INFO Adding class CLOUD
2025-09-17 00:36:25,919 INFO Adding class TRIXIE
2025-09-17 00:36:25,920 INFO Adding class VAGRANT
2025-09-17 00:36:25,920 INFO Adding class AMD64
2025-09-17 00:36:25,920 INFO Adding class LINUX_IMAGE_BASE
2025-09-17 00:36:25,920 INFO Adding class GRUB_PC
2025-09-17 00:36:25,920 INFO Adding class LAST
2025-09-17 00:36:25,921 INFO Running FAI: sudo env PYTHONPATH=/home/rajudev/dev/salsa/michael/debian-vagrant-images/src/debian_cloud_images/build/../.. CLOUD_BUILD_DATA=/home/rajudev/dev/salsa/michael/debian-vagrant-images/src/debian_cloud_images/data CLOUD_BUILD_INFO= "type": "official", "release": "trixie", "release_id": "13", "release_baseid": "13", "vendor": "vagrant", "arch": "amd64", "build_id": "vagrant-cloud-images-master", "version": "20250917-1"  CLOUD_BUILD_NAME=debian-trixie-vagrant-amd64-official-20250917-1 CLOUD_BUILD_OUTPUT_DIR=/home/rajudev/dev/salsa/michael/debian-vagrant-images CLOUD_RELEASE_ID=vagrant CLOUD_RELEASE_VERSION=20250917-1 fai-diskimage --verbose --hostname debian --class DEBIAN,CLOUD,TRIXIE,VAGRANT,AMD64,LINUX_IMAGE_BASE,GRUB_PC,LAST --size 100G --cspace /home/rajudev/dev/salsa/michael/debian-vagrant-images/src/debian_cloud_images/build/fai_config debian-trixie-vagrant-amd64-official-20250917-1.raw
..... continued

Changes in version 0.1.14 (2025-09-13)

• simdjson was upgraded to version 4.0.0 (Dirk in #96
• Continuous integration now relies a token for codecov.io

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

1. Separate application-specific users with granular permissions allowing only necessary access and no more.
2. Distributing and storing passwords and credentials securely
3. Ensuring all remote connections are properly encrypted

How encrypting database connections with TLS differs from web server HTTP(S) Even though MariaDB (and other databases) use the same SSL/TLS protocol for encrypting remote connections as web servers and HTTPS, the way it is implemented is significantly different, and the different security assumptions are important for a database administrator to grasp. Firstly, most HTTP requests to a web server are unauthenticated, meaning the web server serves public web pages and does not require users to log in. Traditionally, when a user logs in over a HTTP connection, the username and password were transmitted in plaintext as a HTTP POST request. Modern TLS, which was previously called SSL, does not change how HTTP works but simply encapsulates it. When using HTTPS, a web browser and a web server will start an encrypted TLS connection as the very first thing, and only once established, do they send HTTP requests and responses inside it. There are no passwords or other shared secrets needed to form the TLS connection. Instead, the web server relies on a trusted third party, a Certificate Authority (CA), to vet that the TLS certificate offered by the web server can be trusted by the web browser. For a database server like MariaDB, the situation is quite different. All users need to first authenticate and log in to the server before getting being allowed to run any SQL and getting any data out of the server. The database server and client programs have built-in authentication methods, and passwords are not, and have never been, sent in plaintext. Over the years, MySQL and its successor, MariaDB, have had multiple password authentication methods: the original SHA-1-based hashing, later double SHA-1-based mysql_native_password, followed by sha256_password and caching_sha256_password in MySQL and ed25519 in MariaDB. The MariaDB.org blog post by Sergei Golubchik recaps the history of these well. Even though most modern MariaDB installations should be using TLS to encrypt all remote connections in 2025, having the authentication method be as secure as possible still matters, because authentication is done before the TLS connection is fully established. To further harden the authentication agains man-in-the-middle attacks, a new password the authentication method PARSEC was introduced in MariaDB 11.8, which builds upon the previous ed25519 public-key-based verification (similar to how modern SSH does), and also combines key derivation with PBKDF2 with hash functions (SHA512,SHA256) and a high iteration count. At first it may seem like a disadvantage to not wrap all connections in a TLS tunnel like HTTPS does, but actually not having the authentication done in a MitM resistant way regardless of the connection encryption status allows a clever extra capability that is now available in MariaDB: as the database server and client already have a shared secret that is being used by the server to authenticate the user, it can also be used by the client to validate the server s TLS certificate and no third parties like CAs or root certificates are needed. MariaDB 11.8 was the first LTS version to ship with this capability for zero-configuration TLS. Note that the zero-configuration TLS also works on older password authentication methods and does not require users to have PARSEC enabled. As PARSEC is not yet the default authentication method in MariaDB, it is recommended to enable it in installations that use zero-configuration TLS encryption to maximize the security of the TLS certificate validation.

Why the root user in MariaDB has no password and how it makes the database more secure Relying on passwords for security is problematic, as there is always a risk that they could leak, and a malicious user could access the system using the leaked password. It is unfortunately far too common for database passwords to be stored in plaintext in configuration files that are accidentally committed into version control and published on GitHub and similar platforms. Every application or administrative password that exists should be tracked to ensure only people who need it know it, and rotated at regular intervals to ensure old employees etc won t be able to use old passwords. This password management is complex and error-prone. Replacing passwords with other authentication methods is always advisable when possible. On a database server, whoever installed the database by running e.g. apt install mariadb-server, and configured it with e.g. nano /etc/mysql/mariadb.cnf, already has full root access to the operating system, and asking them for a password to access the MariaDB database shell is moot, since they could circumvent any checks by directly accessing the files on the system anyway. Therefore, MariaDB, since version 10.4 stopped requiring the root user to enter a password when connecting locally, and instead checks using socket authentication whether the user is the operating-system root user or equivalent (e.g. running sudo). This is an elegant way to get rid of a password that was actually unnecessary to begin with. As there is no root password anymore, the risk of an external user accessing the database as root with a leaked password is fully eliminated. Note that socket authentication only works for local connections on the same server. If you want to access a MariaDB server remotely as the root user, you would need to configure a password for it first. This is not generally recommended, as explained in the next section.

Create separate database users for normal use and keep root for administrative use only Out of the box a MariaDB installation is already secure by default, and only the local root user can connect to it. This account is intended for administrative use only, and for regular daily use you should create separate database users with access limited to the databases they need and the permissions required. The most typical commands needed to create a new database for an app and a user the app can use to connect to the database would be the following:

sql Copy

CREATE DATABASE app_db; CREATE USER 'app_user'@'%' IDENTIFIED BY 'your_secure_password'; GRANT ALL PRIVILEGES ON app_db.* TO 'app_user'@'%'; FLUSH PRIVILEGES;

CREATE DATABASE app_db;
CREATE USER 'app_user'@'%' IDENTIFIED BY 'your_secure_password';
GRANT ALL PRIVILEGES ON app_db.* TO 'app_user'@'%';
FLUSH PRIVILEGES;

Alternatively, if you want to use the parsec authentication method, run this to create the user:

sql Copy

CREATE OR REPLACE USER 'app_user'@'%' IDENTIFIED VIA parsec USING PASSWORD('your_secure_password');

CREATE OR REPLACE USER 'app_user'@'%'
 IDENTIFIED VIA parsec
 USING PASSWORD('your_secure_password');

Note that the plugin auth_parsec is not enabled by default. If you see the error message ERROR 1524 (HY000): Plugin 'parsec' is not loaded fix this by running INSTALL SONAME 'auth_parsec';. In the CREATE USER statements, the @'%' means that the user is allowed to connect from any host. This needs to be defined, as MariaDB always checks permissions based on both the username and the remote IP address or hostname of the user, combined with the authentication method. Note that it is possible to have multiple user@remote combinations, and they can have different authentication methods. A user could, for example, be allowed to log in locally using the socket authentication and over the network using a password. If you are running a custom application and you know exactly what permissions are sufficient for the database users, replace the ALL PRIVILEGES with a subset of privileges listed in the MariaDB documentation. For new permissions to take effect, restart the database or run FLUSH PRIVILEGES.

Allow MariaDB to accept remote connections and enforce TLS Using the above 'app_user'@'%' is not enough on its own to allow remote connections to MariaDB. The MariaDB server also needs to be configured to listen on a network interface to accept remote connections. As MariaDB is secure by default, it only accepts connections from localhost until the administrator updates its configuration. On a typical Debian/Ubuntu system, the recommended way is to drop a new custom config in e.g. /etc/mysql/mariadb.conf.d/99-server-customizations.cnf, with the contents:

Copy

[mariadbd] # Listen for connections from anywhere bind-address = 0.0.0.0 # Only allow TLS encrypted connections require-secure-transport = on

[mariadbd]
# Listen for connections from anywhere
bind-address = 0.0.0.0
# Only allow TLS encrypted connections
require-secure-transport = on

For settings to take effect, restart the server with systemctl restart mariadb. After this, the server will accept connections on any network interface. If the system is using a firewall, the port 3306 would additionally need to be allow-listed. To confirm that the settings took effect, run e.g. mariadb -e "SHOW VARIABLES LIKE 'bind_address';" , which should now show 0.0.0.0. When allowing remote connections, it is important to also always define require-secure-transport = on to enforce that only TLS-encrypted connections are allowed. If the server is running MariaDB 11.8 and the clients are also MariaDB 11.8 or newer, no additional configuration is needed thanks to MariaDB automatically providing TLS certificates and appropriate certificate validation in recent versions. On older long-term-supported versions of the MariaDB server one would have had to manually create the certificates and configure the ssl_key, ssl_cert and ssl_ca values on the server, and distribute the certificate to the clients as well, which was cumbersome, so good it is not required anymore. In MariaDB 11.8 the only additional related config that might still be worth setting is tls_version = TLSv1.3 to ensure only the latest TLS protocol version is used. Finally, test connections to ensure they work and to confirm that TLS is used by running e.g.:

shell Copy

mariadb --user=app_user --password=your_secure_password \ --host=192.168.1.66 -e '\s'

mariadb --user=app_user --password=your_secure_password \
 --host=192.168.1.66 -e '\s'

The result should show something along:

Copy

-------------- mariadb from 11.8.3-MariaDB, client 15.2 for debian-linux-gnu (x86_64) ... Current user: app_user@192.168.1.66 SSL: Cipher in use is TLS_AES_256_GCM_SHA384, cert is OK ...

--------------
mariadb from 11.8.3-MariaDB, client 15.2 for debian-linux-gnu (x86_64)
...
Current user: app_user@192.168.1.66
SSL: Cipher in use is TLS_AES_256_GCM_SHA384, cert is OK
...

If running a Debian/Ubuntu system, see the bundled README with zcat /usr/share/doc/mariadb-server/README.Debian.gz to read more configuration tips.

Should TLS encryption be used also on internal networks? If a database server and app are running on the same private network, the chances that the connection gets eavesdropped on or man-in-the-middle attacked by a malicious user are low. However, it is not zero, and if it happens, it can be difficult to detect or prove that it didn t happen. The benefit of using end-to-end encryption is that both the database server and the client can validate the certificates and keys used, log it, and later have logs audited to prove that connections were indeed encrypted and show how they were encrypted. If all the computers on an internal network already have centralized user account management and centralized log collection that includes all database sessions, reusing existing SSH connections, SOCKS proxies, dedicated HTTPS tunnels, point-to-point VPNs, or similar solutions might also be a practical option. Note that the zero-configuration TLS only works with password validation methods. This means that systems configured to use PAM or Kerberos/GSSAPI can t use it, but again those systems are typically part of a centrally configured network anyway and are likely to have certificate authorities and key distribution or network encryption facilities already set up. In a typical software app stac however, the simplest solution is often the best and I recommend DBAs use the end-to-end TLS encryption in MariaDB 11.8 in most cases. Hopefully with these tips you can enjoy having your MariaDB deployments both simpler and more secure than before!