A new release 0.2.19 of RInside arrived on CRAN and in Debian today. RInside provides a set of convenience classes which facilitate embedding of R inside of C++ applications and programs, using the classes and functions provided by Rcpp. This release fixes a minor bug that got tickled (after a decade and a half RInside) by environment variables (which we parse at compile time and encode in a C/C++ header file as constants) built using double quotes. CRAN currently needs that on one or two platforms, and RInside was erroring. This has been addressed. In the two years since the last release we also received two kind PRs updating the Qt examples to Qt6. And as always we also updated a few other things around the package. The list of changes since the last release:

Changes in RInside version 0.2.19 (2025-04-22)

• The qt example now supports Qt6 (Joris Goosen in #54 closing #53)
• CMake support was refined for more recent versions (Joris Goosen in #55)
• The sandboxed-server example now states more clearly that RINSIDE_CALLBACKS needs to be defined
• More routine update to package and continuous integration.
• Some now-obsolete checks for C++11 have been removed
• When parsing environment variables, use of double quotes is now supported

My CRANberries also provide a short report with changes from the previous release. More information is on the RInside page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page, or to issues tickets at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

2025 was my first year at FOSDEM, and I can say it was an incredible experience where I met many colleagues from Igalia who live around the world, and also many friends from the Linux display stack who are part of my daily work and contributions to DRM/KMS. In addition, I met new faces and recognized others with whom I had interacted on some online forums and we had good and long conversations. During FOSDEM 2025 I had the opportunity to present about kworkflow in the kernel devroom. Kworkflow is a set of tools that help kernel developers with their routine tasks and it is the tool I use for my development tasks. In short, every contribution I make to the Linux kernel is assisted by kworkflow. The goal of my presentation was to spread the word about kworkflow. I aimed to show how the suite consolidates good practices and recommendations of the kernel workflow in short commands. These commands are easily configurable and memorized for your current work setup, or for your multiple setups. For me, Kworkflow is a tool that accommodates the needs of different agents in the Linux kernel community. Active developers and maintainers are the main target audience for kworkflow, but it is also inviting for users and user-space developers who just want to report a problem and validate a solution without needing to know every detail of the kernel development workflow. Something I didn t emphasize during the presentation but would like to correct this flaw here is that the main author and developer of kworkflow is my colleague at Igalia, Rodrigo Siqueira. Being honest, my contributions are mostly on requesting and validating new features, fixing bugs, and sharing scripts to increase feature coverage. So, the video and slide deck of my FOSDEM presentation are available for download here. And, as usual, you will find in this blog post the script of this presentation and more detailed explanation of the demo presented there.

---

Kworkflow at FOSDEM 2025: Speaker Notes and Demo Hi, I m Melissa, a GPU kernel driver developer at Igalia and today I ll be giving a very inclusive talk to not let your motivation go by saving time with kworkflow. So, you re a kernel developer, or you want to be a kernel developer, or you don t want to be a kernel developer. But you re all united by a single need: you need to validate a custom kernel with just one change, and you need to verify that it fixes or improves something in the kernel. And that s a given change for a given distribution, or for a given device, or for a given subsystem Look to this diagram and try to figure out the number of subsystems and related work trees you can handle in the kernel. So, whether you are a kernel developer or not, at some point you may come across this type of situation: There is a userspace developer who wants to report a kernel issue and says:

• Oh, there is a problem in your driver that can only be reproduced by running this specific distribution. And the kernel developer asks:
• Oh, have you checked if this issue is still present in the latest kernel version of this branch?

But the userspace developer has never compiled and installed a custom kernel before. So they have to read a lot of tutorials and kernel documentation to create a kernel compilation and deployment script. Finally, the reporter managed to compile and deploy a custom kernel and reports:

• Sorry for the delay, this is the first time I have installed a custom kernel. I am not sure if I did it right, but the issue is still present in the kernel of the branch you pointed out.

And then, the kernel developer needs to reproduce this issue on their side, but they have never worked with this distribution, so they just created a new script, but the same script created by the reporter. What s the problem of this situation? The problem is that you keep creating new scripts! Every time you change distribution, change architecture, change hardware, change project - even in the same company - the development setup may change when you switch to a different project, you create another script for your new kernel development workflow! You know, you have a lot of babies, you have a collection of my precious scripts , like Sm agol (Lord of the Rings) with the precious ring. Instead of creating and accumulating scripts, save yourself time with kworkflow. Here is a typical script that many of you may have. This is a Raspberry Pi 4 script and contains everything you need to memorize to compile and deploy a kernel on your Raspberry Pi 4. With kworkflow, you only need to memorize two commands, and those commands are not specific to Raspberry Pi. They are the same commands to different architecture, kernel configuration, target device.

What is kworkflow? Kworkflow is a collection of tools and software combined to:

• Optimize Linux kernel development workflow.
• Reduce time spent on repetitive tasks, since we are spending our lives compiling kernels.
• Standardize best practices.
• Ensure reliable data exchange across kernel workflow. For example: two people describe the same setup, but they are not seeing the same thing, kworkflow can ensure both are actually with the same kernel, modules and options enabled.

I don t know if you will get this analogy, but kworkflow is for me a megazord of scripts. You are combining all of your scripts to create a very powerful tool.

What is the main feature of kworflow? There are many, but these are the most important for me:

• Build & deploy custom kernels across devices & distros.
• Handle cross-compilation seamlessly.
• Manage multiple architecture, settings and target devices in the same work tree.
• Organize kernel configuration files.
• Facilitate remote debugging & code inspection.
• Standardize Linux kernel patch submission guidelines. You don t need to double check documentantion neither Greg needs to tell you that you are not following Linux kernel guidelines.
• Upcoming: Interface to bookmark, apply and reviewed-by patches from mailing lists (lore.kernel.org).

This is the list of commands you can run with kworkflow. The first subset is to configure your tool for various situations you may face in your daily tasks.

# Manage kw and kw configurations
kw init             - Initialize kw config file
kw self-update (u)  - Update kw
kw config (g)       - Manage kernel .config files

The second subset is to build and deploy custom kernels.

# Build & Deploy custom kernels
kw kernel-config-manager (k) - Manage kernel .config files
kw build (b)        - Build kernel
kw deploy (d)       - Deploy kernel image (local/remote)
kw bd               - Build and deploy kernel

We have some tools to manage and interact with target machines.

# Manage and interact with target machines
kw ssh (s)          - SSH support
kw remote (r)       - Manage machines available via ssh
kw vm               - QEMU support

To inspect and debug a kernel.

# Inspect and debug
kw device           - Show basic hardware information
kw explore (e)      - Explore string patterns in the work tree and git logs
kw debug            - Linux kernel debug utilities
kw drm              - Set of commands to work with DRM drivers

To automatize best practices for patch submission like codestyle, maintainers and the correct list of recipients and mailing lists of this change, to ensure we are sending the patch to who is interested in it.

# Automatize best practices for patch submission
kw codestyle (c)    - Check code style
kw maintainers (m)  - Get maintainers/mailing list
kw send-patch       - Send patches via email

And the last one, the upcoming patch hub.

# Upcoming
kw patch-hub        - Interact with patches (lore.kernel.org)

How can you save time with Kworkflow? So how can you save time building and deploying a custom kernel? First, you need a .config file.

• Without kworkflow: You may be manually extracting and managing .config files from different targets and saving them with different suffixes to link the kernel to the target device or distribution, or any descriptive suffix to help identify which is which. Or even copying and pasting from somewhere.
• With kworkflow: you can use the kernel-config-manager command, or simply kw k, to store, describe and retrieve a specific .config file very easily, according to your current needs.

Then you want to build the kernel:

• Without kworkflow: You are probably now memorizing a combination of commands and options.
• With kworkflow: you just need kw b (kw build) to build the kernel with the correct settings for cross-compilation, compilation warnings, cflags, etc. It also shows some information about the kernel, like number of modules.

Finally, to deploy the kernel in a target machine.

• Without kworkflow: You might be doing things like: SSH connecting to the remote machine, copying and removing files according to distributions and architecture, and manually updating the bootloader for the target distribution.
• With kworkflow: you just need kw d which does a lot of things for you, like: deploying the kernel, preparing the target machine for the new installation, listing available kernels and uninstall them, creating a tarball, rebooting the machine after deploying the kernel, etc.

You can also save time on debugging kernels locally or remotely.

• Without kworkflow: you do: ssh, manual setup and traces enablement, copy&paste logs.
• With kworkflow: more straighforward access to debug utilities: events, trace, dmesg.

You can save time on managing multiple kernel images in the same work tree.

• Without kworkflow: now you can be cloning multiple times the same repository so you don t lose compiled files when changing kernel configuration or compilation options and manually managing build and deployment scripts.
• With kworkflow: you can use kw env to isolate multiple contexts in the same worktree as environments, so you can keep different configurations in the same worktree and switch between them easily without losing anything from the last time you worked in a specific context.

Finally, you can save time when submitting kernel patches. In kworkflow, you can find everything you need to wrap your changes in patch format and submit them to the right list of recipients, those who can review, comment on, and accept your changes. This is a demo that the lead developer of the kw patch-hub feature sent me. With this feature, you will be able to check out a series on a specific mailing list, bookmark those patches in the kernel for validation, and when you are satisfied with the proposed changes, you can automatically submit a reviewed-by for that whole series to the mailing list.

---

Demo Now a demo of how to use kw environment to deal with different devices, architectures and distributions in the same work tree without losing compiled files, build and deploy settings, .config file, remote access configuration and other settings specific for those three devices that I have.

Setup

• Three devices:

  • laptop (debian

    x86

    intel

    local)

  • SteamDeck (steamos

    x86

    amd

    remote)

  • RaspberryPi 4 (raspbian

    arm64

    broadcomm

    remote)

• Goal: To validate a change on DRM/VKMS using a single kernel tree.
• Kworkflow commands:
  • kw env
  • kw d
  • kw bd
  • kw device
  • kw debug
  • kw drm

Demo script In the same terminal and worktree.

First target device: Laptop (debian x86 intel local)

$ kw env --list # list environments available in this work tree
$ kw env --use LOCAL # select the environment of local machine (laptop) to use: loading pre-compiled files, kernel and kworkflow settings.
$ kw device # show device information
$ sudo modinfo vkms # show VKMS module information before applying kernel changes.
$ <open VKMS file and change module info>
$ kw bd # compile and install kernel with the given change
$ sudo modinfo vkms # show VKMS module information after kernel changes.
$ git checkout -- drivers

Second target device: RaspberryPi 4 (raspbian arm64 broadcomm remote)

$ kw env --use RPI_64 # move to the environment for a different target device.
$ kw device # show device information and kernel image name
$ kw drm --gui-off-after-reboot # set the system to not load graphical layer after reboot
$ kw b # build the kernel with the VKMS change
$ kw d --reboot # deploy the custom kernel in a Raspberry Pi 4 with Raspbian 64, and reboot
$ kw s # connect with the target machine via ssh and check the kernel image name
$ exit

Third target device: SteamDeck (steamos x86 amd remote)

$ kw env --use STEAMDECK # move to the environment for a different target device
$ kw device # show device information
$ kw debug --dmesg --follow --history --cmd="modprobe vkms" # run a command and show the related dmesg output
$ kw debug --dmesg --follow --history --cmd="modprobe -r vkms" # run a command and show the related dmesg output
$ <add a printk with a random msg to appear on dmesg log>
$ kw bd # deploy and install custom kernel to the target device
$ kw debug --dmesg --follow --history --cmd="modprobe vkms" # run a command and show the related dmesg output after build and deploy the kernel change

---

Q&A Most of the questions raised at the end of the presentation were actually suggestions and additions of new features to kworkflow. The first participant, that is also a kernel maintainer, asked about two features: (1) automatize getting patches from patchwork (or lore) and triggering the process of building, deploying and validating them using the existing workflow, (2) bisecting support. They are both very interesting features. The first one fits well the patch-hub subproject, that is under-development, and I ve actually made a similar request a couple of weeks before the talk. The second is an already existing request in kworkflow github project. Another request was to use kexec and avoid rebooting the kernel for testing. Reviewing my presentation I realized I wasn t very clear that kworkflow doesn t support kexec. As I replied, what it does is to install the modules and you can load/unload them for validations, but for built-in parts, you need to reboot the kernel. Another two questions: one about Android Debug Bridge (ADB) support instead of SSH and another about support to alternative ways of booting when the custom kernel ended up broken but you only have one kernel image there. Kworkflow doesn t manage it yet, but I agree this is a very useful feature for embedded devices. On Raspberry Pi 4, kworkflow mitigates this issue by preserving the distro kernel image and using config.txt file to set a custom kernel for booting. For ADB, there is no support too, and as I don t see currently users of KW working with Android, I don t think we will have this support any time soon, except if we find new volunteers and increase the pool of contributors. The last two questions were regarding the status of b4 integration, that is under development, and other debugging features that the tool doesn t support yet. Finally, when Andrea and I were changing turn on the stage, he suggested to add support for virtme-ng to kworkflow. So I opened an issue for tracking this feature request in the project github. With all these questions and requests, I could see the general need for a tool that integrates the variety of kernel developer workflows, as proposed by kworflow. Also, there are still many cases to be covered by kworkflow. Despite the high demand, this is a completely voluntary project and it is unlikely that we will be able to meet these needs given the limited resources. We will keep trying our best in the hope we can increase the pool of users and contributors too.

Eight months ago I came up my rocky driveway in an electric car, with the back full of solar panel mounting rails. I didn't know how I'd manage to keep it charged. I got the car earlier than planned, with my offgrid solar upgrade only beginning. There's no nearby EV charger, and winter was coming, less solar power every day. Still, it was the right time to take a leap to offgid EV life. My existing 1 kilowatt solar array could charge the car only 5 miles on a good day. Here's my first try at charging the car offgrid:

first feeble charging offgrid

It was not worth charging the car that way, the house battery tended to get drained while doing that, and adding cycles to that battery is not desirable. So that was only a proof of concept, I knew I'd need to upgrade. My goal with the upgrade was to charge the car directly from the sun, even when it was cloudy, using the house battery only to skate over brief darker periods (like a thunderstorm). By mid October, I had enough solar installed to do that (5 kilowatts).

me standing in front of solar fence

first charging from solar fence

Using this, in 2 days I charged the car up from 57% to 82%, and took off on a celebratory road trip to Niagra Falls, where I charged the car from hydro power from a dam my grandfather had engineered. When I got home, it was November. Days were getting ever shorter. My solar upgrade was only 1/3rd complete and could charge the car 30-some miles per day, but only on a good day, and weather was getting worse. I came back with a low state of charge (both car and me), and needed to get back to full in time for my Thanksgiving trip at the end of the month. I decided to limit my trips to town.

charging up gradually through the month of November

This kind of medium term planning about car travel was new to me. But not too unusual for offgrid living. You look at the weather forecast and make some rough plans, and get to feel connected to the natural world a bit more. December is the real test for offgrid solar, and honestly this was a bit rough, with a road trip planned for the end of the month. I did the usual holiday stuff but otherwise holed up at home a bit more than I usually would. Charging was limited and the cold made it charge less efficiently.

bleak December charging

Still, I was busy installing more solar panels, and by winter solstice, was back to charging 30 miles on a good day. Of course, from there out things improved. In January and February I was able to charge up easily enough for my usual trips despite the cold. By March the car was often getting full before I needed to go anywhere, and I was doing long round trips without bothering to fast charge along the way, coming home low, knowing even cloudy days would let it charge up enough. That brings me up to today. The car is 80% full and heading up toward 100% for a long trip on Friday. Despite the sky being milky white today with no visible sun, there's plenty of power to absorb, and the car charger turned on at 11 am with the house battery already full. My solar upgrade is only 2/3rds complete, and also I have not yet installed my inverter upgrade, so the car can only currenly charge at 9 amps despite much more solar power often being available. So I'm looking forward to how next December goes with my full planned solar array and faster charging. But first, a summer where I expect the car will mostly be charged up and ready to go at all times, and the only car expense will be fast charging on road trips!

---

By the way, the code I've written to automate offgrid charging that runs only when there's enough solar power is here. And here are the charging graphs for the other months. All told, it's charged 475 kwh offgrid, enough to drive more than 1500 miles.

January

February

March

April

Hello, hello, hello! This report for the Bug Squashing Party we held in Montreal on March 28-29th is very late ... but better late than never? We're now at our fifth BSP in a row¹, which is both nice and somewhat terrifying. Have I really been around for five Debian releases already? Geez... This year, around 13 different people showed up, including some brand new folks! All in all, we ended up working on 77 bugs, 61 of which have since been closed. This is somewhat skewed by the large number of Lintian bugs I closed by merging and releasing the very many patches submitted by Maytham Alsudany (hello Maytham!), but that was still work :D For our past few events, we have been renting a space at Ateliers de la transition socio- cologique. This building used to be nunnery (thus the huge cross on the top floor), but has since been transformed into a multi-faceted project. BSPs are great and this one was no exception. You should try to join an upcoming event or to organise one if you can. It is loads of fun and you will be helping the Debian project release its next stable version sooner! As always, thanks to Debian for granting us a budget for the food and to rent the venue. Pictures Here are a bunch of pictures of the BSP, mixed in with some other pictures I took at this venue during a previous event.

Review: Up the Down Staircase, by Bel Kaufman

Publisher:	Vintage Books
Copyright:	1964, 1991, 2019
Printing:	2019
ISBN:	0-525-56566-3
Format:	Kindle
Pages:	360

Up the Down Staircase is a novel (in an unconventional format, which I'll describe in a moment) about the experiences of a new teacher in a fictional New York City high school. It was a massive best-seller in the 1960s, including a 1967 movie, but seems to have dropped out of the public discussion. I read it from the library sometime in the late 1980s or early 1990s and have thought about it periodically ever since. It was Bel Kaufman's first novel. Sylvia Barrett is a new graduate with a master's degree in English, where she specialized in Chaucer. As Up the Down Staircase opens, it is her first day as an English teacher in Calvin Coolidge High School. As she says in a letter to a college friend:

What I really had in mind was to do a little teaching. "And gladly wolde he lerne, and gladly teche" like Chaucer's Clerke of Oxenford. I had come eager to share all I know and feel; to imbue the young with a love for their language and literature; to instruct and to inspire. What happened in real life (when I had asked why they were taking English, a boy said: "To help us in real life") was something else again, and even if I could describe it, you would think I am exaggerating.

She instead encounters chaos and bureaucracy, broken windows and mindless regulations, a librarian who is so protective of her books that she doesn't let any students touch them, a school guidance counselor who thinks she's Freud, and a principal whose sole interaction with the school is to occasionally float through on a cushion of cliches, dispensing utterly useless wisdom only to vanish again.

I want to take this opportunity to extend a warm welcome to all faculty and staff, and the sincere hope that you have returned from a healthful and fruitful summer vacation with renewed vim and vigor, ready to gird your loins and tackle the many important and vital tasks that lie ahead undaunted. Thank you for your help and cooperation in the past and future. Maxwell E. Clarke
Principal

In practice, the school is run by James J. McHare, Clarke's administrative assistant, who signs his messages JJ McH, Adm. Asst. and who Sylvia immediately starts calling Admiral Ass. McHare is a micro-managing control freak who spends the book desperately attempting to impose order over school procedures, the teachers, and the students, with very little success. The title of the book comes from one of his detention slips:

Please admit bearer to class Detained by me for going Up the Down staircase and subsequent insolence. JJ McH

The conceit of this book is that, except for the first and last chapters, it consists only of memos, letters, notes, circulars, and other paper detritus, often said to come from Sylvia's wastepaper basket. Sylvia serves as the first-person narrator through her long letters to her college friend, and through shorter but more frequent exchanges via intraschool memo with Beatrice Schachter, another English teacher at the same school, but much of the book lies outside her narration. The reader has to piece together what's happening from the discarded paper of a dysfunctional institution. Amid the bureaucratic and personal communications, there are frequent chapters with notes from the students, usually from the suggestion box that Sylvia establishes early in the book. These start as chaotic glimpses of often-misspelled wariness or open hostility, but over the course of Up the Down Staircase, some of the students become characters with fragmentary but still visible story arcs. This remains confusing throughout the novel there are too many students to keep them entirely straight, and several of them use pseudonyms for the suggestion box but it's the sort of confusion that feels like an intentional authorial choice. It mirrors the difficulty a teacher has in piecing together and remembering the stories of individual students in overstuffed classrooms, even if (like Sylvia and unlike several of her colleagues) the teacher is trying to pay attention. At the start, Up the Down Staircase reads as mostly-disconnected humor. There is a strong "kids say the darnedest things" vibe, which didn't entirely work for me, but the send-up of chaotic bureaucracy is both more sophisticated and more entertaining. It has the "laugh so that you don't cry" absurdity of a system with insufficient resources, entirely absent management, and colleagues who have let their quirks take over their personalities. Sylvia alternates between incredulity and stubbornness, and I think this book is at its best when it shows the small acts of practical defiance that one uses to carve out space and coherence from mismanaged bureaucracy. But this book is not just a collection of humorous anecdotes about teaching high school. Sylvia is sincere in her desire to teach, which crystallizes around, but is not limited to, a quixotic attempt to reach one delinquent that everyone else in the school has written off. She slowly finds her footing, she has a few breakthroughs in reaching her students, and the book slowly turns into an earnest portrayal of an attempt to make the system work despite its obvious unfitness for purpose. This part of the book is hard to review. Parts of it worked brilliantly; I could feel myself both adjusting my expectations alongside Sylvia to something less idealistic and also celebrating the rare breakthrough with her. Parts of it were weirdly uncomfortable in ways that I'm not sure I enjoyed. That includes Sylvia's climactic conversation with the boy she's been trying to reach, which was weirdly charged and ambiguous in a way that felt like the author's reach exceeding their grasp. One thing that didn't help my enjoyment is Sylvia's relationship with Paul Barringer, another of the English teachers and a frustrated novelist and poet. Everyone who works at the school has found their own way to cope with the stress and chaos, and many of the ways that seem humorous turn out to have a deeper logic and even heroism. Paul's, however, is to retreat into indifference and alcohol. He is a believable character who works with Kaufman's themes, but he's also entirely unlikable. I never understood why Sylvia tolerated that creepy asshole, let alone kept having lunch with him. It is clear from the plot of the book that Kaufman at least partially understands Paul's deficiencies, but that did not help me enjoy reading about him. This is a great example of a book that tried to do something unusual and risky and didn't entirely pull it off. I like books that take a risk, and sometimes Up the Down Staircase is very funny or suddenly insightful in a way that I'm not sure Kaufman could have reached with a more traditional novel. It takes a hard look at what it means to try to make a system work when it's clearly broken and you can't change it, and the way all of the characters arrive at different answers that are much deeper than their initial impressions was subtle and effective. It's the sort of book that sticks in your head, as shown by the fact I bought it on a whim to re-read some 35 years after I first read it. But it's not consistently great. Some parts of it drag, the characters are frustratingly hard to keep track of, and the emotional climax points are odd and unsatisfying, at least to me. I'm not sure whether to recommend it or not, but it's certainly unusual. I'm glad I read it again, but I probably won't re-read it for another 35 years, at least. If you are considering getting this book, be aware that it has a lot of drawings and several hand-written letters. The publisher of the edition I read did a reasonably good job formatting this for an ebook, but some of the pages, particularly the hand-written letters, were extremely hard to read on a Kindle. Consider paper, or at least reading on a tablet or computer screen, if you don't want to have to puzzle over low-resolution images. The 1991 trade paperback had a new introduction by the author, reproduced in the edition I read as an afterward (which is a better choice than an introduction). It is a long and fascinating essay from Kaufman about her experience with the reaction to this book, culminating in a passionate plea for supporting public schools and public school teachers. Kaufman's personal account adds a lot of depth to the story; I highly recommend it. Content note: Self-harm, plus several scenes that are closely adjacent to student-teacher relationships. Kaufman deals frankly with the problems of mostly-poor high school kids, including sexuality, so be warned that this is not the humorous romp that it might appear on first glance. A couple of the scenes made me uncomfortable; there isn't anything explicit, but the emotional overtones can be pretty disturbing. Rating: 7 out of 10

Due to my own laziness and a few functionality issues my "for work laptop" is still using a 15+ year old setup with X11 and awesome. Since trixie is now starting its freeze, it's time to update that odd machine as well and look at the fallout. Good news: It's mostly my own resistance to change which required some kick in the back to move on. Clipboard Manager Madness For the past decade or so I used parcellite which served me well. Now that is no longer available in trixie and I started to look into one of the dead end streets of X11 related tooling, searching for an alternative. Parcellite Seems upstream is doing sporadic fixes, but holds GTK2 tight. The Debian package was patched to be GTK3 compatible, but has unfixed ftbfs issues with GCC 14. clipit Next I checked for a parcellite fork named clipit, and that's when it started to get funky. It's packaged in Debian, QA maintained, and recently received at least two uploads to keep it working. Installed it and found it's greeting me with a nag screen that I should migrate to diodon. The real clipit tool is still shipped as a binary named clipit.real, so if you know it you can still use it. To achieve the nag screen it depends on zenity and to ease the migration it depends on diodon. Two things I do not really need. Also the package description prominently mentions that you should not use the package. diodon The nag screen of clipit made me look at diodon. It claims it was written for the Ubuntu Unity desktop, something where I've no idea how alive and relevant it still is. While there is still something on launchpad, it seems to receive sporadic commits on github. Not sure if it's dead or just feature complete. Interim Solution: clipit Settled with clipit for now, but decided to fork the Debian package to remove the nag screen and the dependency on diodon and zenity (package build). My hope is to convert this last X11 setup to wayland within the lifetime of trixie. I also contacted the last uploader regarding a removal of the nag screen, who then brought in the last maintainer who added the nag screen. While I first thought clipit is somewhat maintained upstream, Andrej quickly pointed out that this is not really the case. Still that leaves us in trixie with a rather odd situation. We ship now for the second stable release a package that recommends to move to a different tool while still shipping the original tool. Plus it's getting patched by some of its users who refuse to migrate to the alternative envisioned by the former maintainer. VirtualBox and moving to libvirt I always liked the GUI of VirtualBox, and it really made desktop virtualization easy. But with Linux 6.12, which enables KVM by default, it seems to get even more painful to get it up and running. In the past I just took the latest release from unstable and rebuild that one on the current stable. Currently the last release in unstable is 7.0.20, while the Linux 6.12 fixes only started to appear in VirtualBox 7.1.4 and later. The good thing is with virt-manager and the whole libvirt ecosystem there is a good enough replacement available, and it works fine with related tooling like vagrant. There are instructions available on how to set it up. I can only add that it makes sense to export VAGRANT_DEFAULT_PROVIDER=libvirt in your .bashrc to make that provider change permanent.

Remember the XZ Utils backdoor? One factor that enabled the attack was poor auditing of the release tarballs for differences compared to the Git version controlled source code. This proved to be a useful place to distribute malicious data. The differences between release tarballs and upstream Git sources is typically vendored and generated files. Lots of them. Auditing all source tarballs in a distribution for similar issues is hard and boring work for humans. Wouldn t it be better if that human auditing time could be spent auditing the actual source code stored in upstream version control instead? That s where auditing time would help the most. Are there better ways to address the concern about differences between version control sources and tarball artifacts? Let s consider some approaches:

• Stop publishing (or at least stop building from) source tarballs that differ from version control sources.
• Create recipes for how to derive the published source tarballs from version control sources. Verify that independently from upstream.

While I like the properties of the first solution, and have made effort to support that approach, I don t think normal source tarballs are going away any time soon. I am concerned that it may not even be a desirable complete solution to this problem. We may need tarballs with pre-generated content in them for various reasons that aren t entirely clear to us today. So let s consider the second approach. It could help while waiting for more experience with the first approach, to see if there are any fundamental problems with it. How do you know that the XZ release tarballs was actually derived from its version control sources? The same for Gzip? Coreutils? Tar? Sed? Bash? GCC? We don t know this! I am not aware of any automated or collaborative effort to perform this independent confirmation. Nor am I aware of anyone attempting to do this on a regular basis. We would want to be able to do this in the year 2042 too. I think the best way to reach that is to do the verification continuously in a pipeline, fixing bugs as time passes. The current state of the art seems to be that people audit the differences manually and hope to find something. I suspect many package maintainers ignore the problem and take the release source tarballs and trust upstream about this. We can do better. I have launched a project to setup a GitLab pipeline that invokes per-release scripts to rebuild that release artifact from git sources. Currently it only contain recipes for projects that I released myself. Releases which where done in a controlled way with considerable care to make reproducing the tarballs possible. The project homepage is here: https://gitlab.com/debdistutils/verify-reproducible-releases The project is able to reproduce the release tarballs for Libtasn1 v4.20.0, InetUtils v2.6, Libidn2 v2.3.8, Libidn v1.43, and GNU SASL v2.2.2. You can see this in a recent successful pipeline. All of those releases were prepared using Guix, and I m hoping the Guix time-machine will make it possible to keep re-generating these tarballs for many years to come. I spent some time trying to reproduce the current XZ release tarball for version 5.8.1. That would have been a nice example, wouldn t it? First I had to somehow mimic upstream s build environment. The XZ release tarball contains GNU Libtool files that are identified with version 2.5.4.1-baa1-dirty. I initially assumed this was due to the maintainer having installed libtool from git locally (after making some modifications) and made the XZ release using it. Later I learned that it may actually be coming from ArchLinux which ship with this particular libtool version. It seems weird for a distribution to use libtool built from a non-release tag, and furthermore applying patches to it, but things are what they are. I made some effort to setup an ArchLinux build environment, however the now-current Gettext version in ArchLinux seems to be more recent than the one that were used to prepare the XZ release. I don t know enough ArchLinux to setup an environment corresponding to an earlier version of ArchLinux, which would be required to finish this. I gave up, maybe the XZ release wasn t prepared on ArchLinux after all. Actually XZ became a good example for this writeup anyway: while you would think this should be trivial, the fact is that it isn t! (There is another aspect here: fingerprinting the versions used to prepare release tarballs allows you to infer what kind of OS maintainers are using to make releases on, which is interesting on its own.) I made some small attempts to reproduce the tarball for GNU Shepherd version 1.0.4 too, but I still haven t managed to complete it. Do you want a supply-chain challenge for the Easter weekend? Pick some well-known software and try to re-create the official release tarballs from the corresponding Git checkout. Is anyone able to reproduce anything these days? Bonus points for wrapping it up as a merge request to my project. Happy Supply-Chain Security Hacking!

Very busy releasetastic week! The versions being the same is a complete coincidence https://kde.org/announcements/gear/25.04.0 Which can be downloaded here: https://snapcraft.io/publisher/kde !

Kubuntu 25.04 Plucky Puffin released

In addition to all the regular testing I am testing our snaps in a non KDE environment, so far it is not looking good in Xubuntu. We have kernel/glibc crashes on startup for some and for file open for others. I am working on a hopeful fix. Next week I will have ( I hope ) my final surgery. If you can spare any change to help bring me over the finish line, I will be forever grateful

• GoFundMe
• Patreon
• Github
• Donorbox

Sadly, the interactive application firewall OpenSnitch have in practice been unmaintained in Debian for a while. A few days ago I decided to do something about it, and today I am happy with the result. This package monitor network traffic going in and out of a Linux machine, and show a popup dialog to the logged in desktop user, asking to approve or deny any new connections. It has proved very valuable in discovering programs calling home, giving me more control of how information leak out of my Linux machine. So far the new version is only available in Debian experimental, but I plan to upload it to unstable as soon as I know it is working on a few more machines, and make sure the new version make it into the next stable release of Debian. The package freeze is approaching, and it is not a lot of time left. If you read this blog post, I hope you can be one of the testers. The new version should be using eBPF on architectures where this is working (amd64 and arm64), and fall back to /proc/ probing where the opensnitch-ebpf-modules package is missing (so far only armhf, a unrelated bug blocks building on riscv64 and s390x). Using eBPF should provide more accurate attribution of packages responsible for network traffic for short lived processes, which some times were unavailable in /proc/ when opensnitch tried to probe for information. I have limited experience with the new version, having used it myself for a day or so. It is easily backportable to Debian 12 Bookworm without code changes, all it need is a simple 'debuild' thanks to the optional build dependencies. Due to a misfeature of llc on armhf, there is no eBPF support available there. I have not investigated the details, nor reported any bug yet, but for some reason -march=bpf is an unknown option on this architecture, causing the build in the ebpf_prog subdirectory build to fail. The package is maintained under the umbrella of Debian Go team, and you can meet the current maintainers on the #debian-golang and #opensnitch IRC channels on irc.debian.org. As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Last year, I decided to start participating in the Debian LTS and ELTS projects. It was a great opportunity to engage in something new within the Debian community. I had been following these projects for many years, observing their evolution and how they gained traction both within the ecosystem and across the industry. I was curious to explore how contributors were working internally especially how they managed security patching and remediation for older software. I ve always felt this was a particularly challenging area, and I was fortunate to experience it firsthand. As of April 2025, the Debian LTS project was primarily focused on providing security maintenance for Debian 11 Bullseye. Meanwhile, the Debian ELTS project was targeting Debian 8 Jessie, Debian 9 Stretch, and Debian 10 Buster. During my time with the projects, I worked on a variety of packages and CVEs. Some of the most notable ones include:

• bluez
• nss
• libmojolicious-perl
• uw-imap
• dnsmasq
• firmware-nonfree
• activemq
• frr
• libmodbus

There are several technical highlights I d like to share things I learned or had to apply while participating:

• CI/CD pipelines: We used CI/CD pipelines on salsa.debian.org all the times to automate tasks such as building, linting, and testing packages. For any package I worked on that lacked CI/CD integration, setting it up became my first step.
• autopkgtest: There s a strong emphasis on autopkgtest as the mechanism for running functional tests and ensuring that security patches don t introduce regressions. I contributed by both extending existing test suites and writing new ones from scratch.
• Toolchain complexity for older releases: Working with older Debian versions like Jessie brought some unique challenges. Getting a development environment up and running often meant troubleshooting issues with sbuild chroots, qemu images, and other tools that don t just work like they tend to on Debian stable.
• Community collaboration: The people involved in LTS and ELTS are extremely helpful and collaborative. Requests for help, code reviews, and general feedback were usually answered quickly.
• Shared ownership: This collaborative culture also meant that contributors would regularly pick up work left by others or hand off their own tasks when needed. That mutual support made a big difference.
• Backporting security fixes: This is probably the most intense and most rewarding activity. It involves manually adapting patches to work on older codebases when upstream patches don t apply cleanly. This requires deep code understanding and thorough testing.
• Upstream collaboration: Reaching out to upstream developers was a key part of my workflow. I often asked if they could provide patches for older versions or at least review my backports. Sometimes they were available, but most of the time, the responsibility remained on us.
• Diverse tech stack: The work exposed me to a wide range of programming languages and frameworks Python, Java, C, Perl, and more. Unsurprisingly, some modern languages (like Go) are less prevalent in older releases like Jessie.
• Security team interaction: I had frequent contact with the core Debian Security Team the folks responsible for security in Debian stable. This gave me a broader perspective on how Debian handles security holistically.

In March 2025, I decided to scale back my involvement in the projects due to some changes in my personal life. Still, this experience has been one of the highlights of my career, and I would definitely recommend it to others. I m very grateful for the warm welcome I received from the LTS/ELTS community, and I don t rule out the possibility of rejoining the LTS/ELTS efforts in the future. The Debian LTS/ELTS projects are currently coordinated by folks at Freexian. Many thanks to Freexian and sponsors for providing this opportunity!

After careful consideration, I ve decided to embark on a new chapter in my professional journey. I ve left my position at AWS to dedicate at least the next six months to developing open source software and strengthening digital ecosystems. My focus will be on contributing to Linux distributions (primarily Debian) and other critical infrastructure components that our modern society depends on, but which may not receive adequate attention or resources.

The Evolution of Open Source Open source won. Over the 25+ years I ve been involved in the open source movement, I ve witnessed its remarkable evolution. Today, Linux powers billions of devices from tiny embedded systems and Android smartphones to massive cloud datacenters and even space stations. Examine any modern large-scale digital system, and you ll discover it s built upon thousands of open source projects. I feel the priority for the open source movement should no longer be increasing adoption, but rather solving how to best maintain the vast ecosystem of software. This requires building robust institutions and processes to secure proper resourcing and ensure the collaborative development process remains efficient and leads to ever-increasing quality of software.

What is Special About Debian? Debian, established in 1993 by Ian Murdock, stands as one of these institutions that has demonstrated exceptional resilience. There is no single authority, but instead a complex web of various stakeholders, each with their own goals and sources of funding. Every idea needs to be championed at length to a wide audience and implemented through a process of organic evolution. Thanks to this approach, Debian has been consistently delivering production-quality, universally useful software for over three decades. Having been a Debian Developer for more than ten years, I m well-positioned to contribute meaningfully to this community. If your organization relies on Debian or its derivatives such as Ubuntu, and you re interested in funding cyber infrastructure maintenance by sponsoring Debian work, please don t hesitate to reach out. This could include package maintenance and version currency, improving automated upgrade testing, general quality assurance and supply chain security enhancements.

Best way to reach me is by e-mail otto at debian.org. You can also book a 15-minute chat with me for a quick introduction.

Grow or Die My four-year tenure as a Software Development Manager at Amazon Web Services was very interesting. I m grateful for my time at AWS and proud of my team s accomplishments, particularly for creating an open source contribution process that got Amazon from zero to the largest external contributor to the MariaDB open source database. During this time, I got to experience and witness a plethora of interesting things. I will surely share some of my key learnings in future blog posts. Unfortunately, the rate of progress in this mammoth 1.5 million employee organization was slowing down, and I didn t feel I learned much new in the last years. This realization, combined with the opportunity cost of not spending enough time on new cutting-edge technology, motivated me to take this leap. Being a full-time open source developer may not be financially the most lucrative idea, but I think it is an excellent way to force myself to truly assess what is important on a global scale and what areas I want to contribute to. Working fully on open source presents a fascinating duality: you re not bound by any external resource or schedule limitations, and can the progress you make is directly proportional to how much energy you decide to invest. Yet, you also depend on collaboration with people you might never meet and who are not financially incentivized to collaborate. This will undoubtedly expose me to all kinds of challenges. But what would be better in fostering holistic personal growth? I know that deep down in my DNA, I am not made to stay cozy or to do easy things. I need momentum. OK, let s get going

Another new (stable) release of the AsioHeaders package arrived at CRAN just now. Asio provides a cross-platform C++ library for network and low-level I/O programming. It is also included in Boost but requires linking when used as part of Boost. This standalone version of Asio is a header-only C++ library which can be used without linking (just like our BH package with parts of Boost). The update last week, kindly prepared by Charlie Gao, had overlooked / not covered one other nag discovered by CRAN. This new release, based on the current stable upstream release, does that. The short NEWS entry for AsioHeaders follows.

Changes in version 1.30.2-0 (2025-04-15

• Upgraded to Asio 1.30.2 (Dirk in #13 fixing #12)
• Added two new badges to README.md

Thanks to my CRANberries, there is a diffstat report for this release. Comments and suggestions about AsioHeaders are welcome via the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

It s been almost 15 months since I blogged about Storage Trends 2024 [1]. There hasn t been much change in this time (in Australia at least I m not tracking prices in other countries). The change was so small I had to check how the Australian dollar has performed against other currencies to see if changes to currencies had countered changes to storage prices, but there has been little overall change when compared to the Chinese Yuan and the Australian dollar is only about 11% worse against the US dollar when compared to a year ago. Generally there s a trend of computer parts decreasing in price by significantly more than 11% per annum. Small Storage The cheapest storage device from MSY now is a Patriot P210 128G SATA SSD for $19, cheaper than the $24 last year and the same price as the year before. So over the last 2 years there has been no change to the cheapest storage device on sale. It would almost never make sense to buy that as a 256G SATA SSD (also Patriot P210) is $25 and has twice the lifetime (120TBW vs 60TBW). There are also 256G NVMe devices for $29 and $30 which would be better options if the system has a NVMe socket built in. The cheapest 500G devices are $42.50 for a 512G SATA SSD and $45 for a 500G NVMe. Last year the prices were $33 for SATA and $36 for NVMe in that size so there s been a significant increase in price there. The difference is enough that if someone was on a tight budget they might reasonably decide to use smaller storage than they might have used last year! 2TB hard drives are still $89 the same price as last year! Last year a 2TB SATA SSD was $118 and a 2TB NVMe was $145, now a 2TB SATA SSD is $157 and a 2TB NVMe is $127. So NVMe has become cheaper than SATA in that segment but overall prices are higher than last year. Again for business use 2TB seems a sensible minimum for most systems if you are paying MSY rates (or similar rates from Amazon etc). Medium Storage Last year 4TB HDDs were $135, now they are $148. Last year the cheapest 4TB SSD was $299, now the cheapest is a $309 NVMe. While the prices have all gone up the price difference between hard drives and SSD has decreased in that size range. So for a small server (a lot of home servers and small business servers) 4TB of RAID-1 storage is all that s needed and for that SSDs are the best option. The price difference between $296 for 4TB of RAID-1 HDDs and $618 for RAID-1 NVMe is small enough to be justified by the benefits of speed and being quiet for most small server uses. In 2023 a 8TB hard drive cost $179 and a 8TB SSD cost $739. Last year a 8TB hard drive cost $239 and a 8TB SATA SSD cost, $899. Now a 8TB HDD costs $229 and MSY doesn t sell 8TB SSDs but for comparison Amazon has a Samsung 8TB SATA SSD for $919. So for storing 8TB+ there are benefits of hard drives as SSDs are difficult to get in that size range and more expensive than they were before. It seems that 8TB SSDs aren t used by enough people to have a large market in the home and small office space, so those of us who want the larger storage sizes will have to get second hand enterprise gear. It will probably be another few years before 8TB enterprise SSDs start appearing on the second hand market. Serious Storage Last year I wrote about the affordability of U.2 devices. I regret not buying some then as there are fewer on sale now and prices are higher. For hard drives they still aren t a good choice for most users because most users don t have more than 4TB of data. For large quantities of data hard drives are still a good option, a 22TB disk costs $899. For companies this is a good option for many situations. For home users there is the additional problem that determining whether a drive is Shingled Magnetic Recording which has some serious performance issues for some use and it s very difficult to determine which drives use it. Conclusion For corporate purchases the options for serious storage are probably decent. But for small companies and home users things definitely don t seem to have improved as much as we expect from the computer industry, I had expected 8TB SSDs to go for $450 by now and SSDs less than 500G to not even be sold new any more. The prices on 8TB SSDs have gone up more in the last 2 yeas than the ASX 200 (index of 200 biggest companies in the Australian stock market). I would never recommend using SSDs as an investment, but in retrospect 8TB SSDs could have been a good one. $20 seems to be about the minimum cost that SSDs approach while hard drives have a higher minimum price of a bit under $100 because they are larger, heavier, and more fragile. It seems that the market is likely to move to most SSDs being close to $20, if they can make 2TB SSDs cheaply enough to sell for about that price then that would cover the majority of the market. I ve created a table of the prices, I should have done this before but I initially didn t plan an ongoing series of posts on this topic.

	Jun 2020	Apr 2021	Apr 2023	Jan 2024	Apr 2025
128G SSD	$49		$19	$24	$19
500G SSD	$97	$73	$32	$33	$42.50
2TB HDD	$95	$72	$75	$89	$89
2TB SSD	$335	$245			$149
4TB HDD			$115	$135	$148
4TB SSD	$895		$349	$299	$309
8TB HDD			$179	$239	$229
8TB SSD		$949	$739	$899	$919
10TB HDD	$549	$395

• [1] https://etbe.coker.com.au/2024/01/22/storage-trends-2024/

Fun with -fsanitize=undefined and Picolibc Both GCC and Clang support the -fsanitize=undefined flag which instruments the generated code to detect places where the program wanders into parts of the C language specification which are either undefined or implementation defined. Many of these are also common programming errors. It would be great if there were sanitizers for other easily detected bugs, but for now, at least the undefined sanitizer does catch several useful problems. Supporting the sanitizer The sanitizer can be built to either trap on any error or call handlers. In both modes, the same problems are identified, but when trap mode is enabled, the compiler inserts a trap instruction and doesn't expect the program to continue running. When handlers are in use, each identified issue is tagged with a bunch of useful data and then a specific sanitizer handling function is called. The specific functions are not all that well documented, nor are the parameters they receive. Maybe this is because both compilers provide an implementation of all of the functions they use and don't really expect external implementations to exist? However, to make these useful in an embedded environment, picolibc needs to provide a complete set of handlers that support all versions both gcc and clang as the compiler-provided versions depend upon specific C (and C++) libraries. Of course, programs can be built in trap-on-error mode, but that makes it much more difficult to figure out what went wrong. Fixing Sanitizer Issues Once the sanitizer handlers were implemented, picolibc could be built with them enabled and all of the picolibc tests run to uncover issues within the library. As with the static analyzer adventure from last year, the vast bulk of sanitizer complaints came from invoking undefined or implementation-defined behavior in harmless ways:

• Computing pointers past &array[size+1]. I found no cases where the resulting pointers were actually used, but the mere computation is still undefined behavior. These were fixed by adjusting the code to avoid computing pointers like this. The result was clearer code, which is good.
• Signed arithmetic overflow in PRNG code. There are several linear congruential PRNGs in the library which used signed integer arithmetic. The rand48 generator carefully used unsigned short values. Of course, in C, the arithmetic performed on them is done with signed ints if int is wider than short. C specifies signed overflow as undefined, but both gcc and clang generate the expected code anyways. The fixes here were simple; just switch the computations to unsigned arithmetic, adjusting types and inserting casts as required.
• Passing pointers to the middle of a data structure. For example, free takes a pointer to the start of an allocation. The management structure appears just before that in memory; computing the address of which appears to be undefined behavior to the compiler. The only fix I could do here was to disable the sanitizer in functions doing these computations -- the sanitizer was mis-detecting correct code and it doesn't provide a way to skip checks on a per-operator basis.
• Null pointer plus or minus zero. C says that any arithmetic with the NULL pointer is undefined, even when the value being added or subtracted is zero. The fix here was to create a macro, enabled only when the sanitizer is enabled, which checks for this case and skips the arithmetic.
• Discarded computations which overflow. A couple of places computed a value, then checked if that would have overflowed and discard the result. Even though the program doesn't depend upon the computation, its mere presence is undefined behavior. These were fixed by moving the computation into an else clause in the overflow check. This inserts an extra branch instruction, which is annoying.
• Signed integer overflow in math code. There's a common pattern in various functions that want to compare against 1.0. Instead of using the floating point equality operator, they do the computation using the two 32-bit halves with ((hi - 0x3ff00000) lo) == 0. It's efficient, but because most of these functions store the 'hi' piece in a signed integer (to make checking the sign bit fast), the result is undefined when hi is a large negative value. These were fixed by inserting casts to unsigned types as the results were always tested for equality.

Signed integer shifts This is one area where the C language spec is just wrong. For left shift, before C99, it worked on signed integers as a bit-wise operator, equivalent to the operator on unsigned integers. After that, left shift of negative integers became undefined. Fortunately, it's straightforward (if tedious) to work around this issue by just casting the operand to unsigned, performing the shift and casting it back to the original type. Picolibc now has an internal macro, lsl, which does this:

    #define lsl(__x,__s) ((sizeof(__x) == sizeof(char)) ?                   \
                          (__typeof(__x)) ((unsigned char) (__x) << (__s)) :  \
                          (sizeof(__x) == sizeof(short)) ?                  \
                          (__typeof(__x)) ((unsigned short) (__x) << (__s)) : \
                          (sizeof(__x) == sizeof(int)) ?                    \
                          (__typeof(__x)) ((unsigned int) (__x) << (__s)) :   \
                          (sizeof(__x) == sizeof(long)) ?                   \
                          (__typeof(__x)) ((unsigned long) (__x) << (__s)) :  \
                          (sizeof(__x) == sizeof(long long)) ?              \
                          (__typeof(__x)) ((unsigned long long) (__x) << (__s)) : \
                          __undefined_shift_size(__x, __s))

Right shift is significantly more complicated to implement. What we want is an arithmetic shift with the sign bit being replicated as the value is shifted rightwards. C defines no such operator. Instead, right shift of negative integers is implementation defined. Fortunately, both gcc and clang define the >> operator on signed integers as arithmetic shift. Also fortunately, C hasn't made this undefined, so the program itself doesn't end up undefined. The trouble with arithmetic right shift is that it is not equivalent to right shift of unsigned values. Here's what Per Vognsen came up with using standard C operators:

    int
    __asr_int(int x, int s)  
        return x < 0 ? ~(~x >> s) : x >> s;
     

When the value is negative, we invert all of the bits (making it positive), shift right, then flip all of the bits back. Both GCC and Clang seem to compile this to a single asr instruction. This function is replicated for each of the five standard integer types and then the set of them wrapped in another sizeof-selecting macro:

    #define asr(__x,__s) ((sizeof(__x) == sizeof(char)) ?           \
                          (__typeof(__x))__asr_char(__x, __s) :       \
                          (sizeof(__x) == sizeof(short)) ?          \
                          (__typeof(__x))__asr_short(__x, __s) :      \
                          (sizeof(__x) == sizeof(int)) ?            \
                          (__typeof(__x))__asr_int(__x, __s) :        \
                          (sizeof(__x) == sizeof(long)) ?           \
                          (__typeof(__x))__asr_long(__x, __s) :       \
                          (sizeof(__x) == sizeof(long long)) ?      \
                          (__typeof(__x))__asr_long_long(__x, __s):   \
                          __undefined_shift_size(__x, __s))

The lsl and asr macros use sizeof instead of the type-generic mechanism to remain compatible with compilers that lack type-generic support. Once these macros were written, they needed to be applied where required. To preserve the benefits of detecting programming errors, they were only applied where required, not blindly across the whole codebase. There are a couple of common patterns in the math code using shift operators. One is when computing the exponent value for subnormal numbers.

for (ix = -1022, i = hx << 11; i > 0; i <<= 1)
    ix -= 1;

This code computes the exponent by shifting the significand left by 11 bits (the width of the exponent field) and then incrementally shifting it one bit at a time until the sign flips, which indicates that the most-significant bit is set. Use of the pre-C99 definition of the left shift operator is intentional here; so both shifts are replaced with our lsl operator. In the implementation of pow, the final exponent is computed as the sum of the two exponents, both of which are in the allowed range. The resulting sum is then tested to see if it is zero or negative to see if the final value is sub-normal:

hx += n << 20;
if (hx >> 20 <= 0)
    /* do sub-normal things */

In this case, the exponent adjustment, n, is a signed value and so that shift is replaced with the lsl macro. The test value needs to compute the correct the sign bit, so we replace this with the asr macro. Because the right shift operation is not undefined, we only use our fancy macro above when the undefined behavior sanitizer is enabled. On the other hand, the lsl macro should have zero cost and covers undefined behavior, so it is always used. Actual Bugs Found! The goal of this little adventure was both to make using the undefined behavior sanitizer with picolibc possible as well as to use the sanitizer to identify bugs in the library code. I fully expected that most of the effort would be spent masking harmless undefined behavior instances, but was hopeful that the effort would also uncover real bugs in the code. I was not disappointed. Through this work, I found (and fixed) eight bugs in the code:

1. setlocale/newlocale didn't check for NULL locale names
2. qsort was using uintptr_t to swap data around. On MSP430 in 'large' mode, that's a 20-bit type inside a 32-bit representation.
3. random() was returning values in int range rather than long.
4. m68k assembly for memcpy was broken for sizes > 64kB.
5. freopen returned NULL, even on success
6. The optimized version of memrchr was always performing unaligned accesses.
7. String to float conversion had a table missing four values. This caused an array access overflow which resulted in imprecise values in some cases.
8. vfwscanf mis-parsed floating point values by assuming that wchar_t was unsigned.

Sanitizer Wishes While it's great to have a way to detect places in your C code which evoke undefined and implementation defined behaviors, it seems like this tooling could easily be extended to detect other common programming mistakes, even where the code is well defined according to the language spec. An obvious example is in unsigned arithmetic. How many bugs come from this seemingly innocuous line of code?

    p = malloc(sizeof(*p) * c);

Because sizeof returns an unsigned value, the resulting computation never results in undefined behavior, even when the multiplication wraps around, so even with the undefined behavior sanitizer enabled, this bug will not be caught. Clang seems to have an unsigned integer overflow sanitizer which should do this, but I couldn't find anything like this in gcc. Summary The undefined behavior sanitizers present in clang and gcc both provide useful diagnostics which uncover some common programming errors. In most cases, replacing undefined behavior with defined behavior is straightforward, although the lack of an arithmetic right shift operator in standard C is irksome. I recommend anyone using C to give it a try.

This topic came up at a customer of mine in September 2024, when working on Debian/trixie support. Since then I wanted to blog about it to make people aware of this new OpenSSH feature and behavior. I finally found some spare minutes at Debian s BSP in Vienna, so here we are. :) Some of our Q/A jobs failed to run against Debian/trixie, in the debug logs we found:

debug1: kex_exchange_identification: banner line 0: Not allowed at this time

This Not allowed at this time pointed to a new OpenSSH feature. OpenSSH introduced options to penalize undesirable behavior with version 9.8p1, see OpenSSH Release Notes, and also sshd source code. FTR, on the SSH server side, you ll see messages like that:

Apr 13 08:57:11 grml sshd-session[2135]: error: maximum authentication attempts exceeded for root from 10.100.15.42 port 55792 ssh2 [preauth]
Apr 13 08:57:11 grml sshd-session[2135]: Disconnecting authenticating user root 10.100.15.42 port 55792: Too many authentication failures [preauth]
Apr 13 08:57:12 grml sshd-session[2137]: error: maximum authentication attempts exceeded for root from 10.100.15.42 port 55800 ssh2 [preauth]
Apr 13 08:57:12 grml sshd-session[2137]: Disconnecting authenticating user root 10.100.15.42 port 55800: Too many authentication failures [preauth]
Apr 13 08:57:13 grml sshd-session[2139]: error: maximum authentication attempts exceeded for root from 10.100.15.42 port 55804 ssh2 [preauth]
Apr 13 08:57:13 grml sshd-session[2139]: Disconnecting authenticating user root 10.100.15.42 port 55804: Too many authentication failures [preauth]
Apr 13 08:57:13 grml sshd-session[2141]: error: maximum authentication attempts exceeded for root from 10.100.15.42 port 55810 ssh2 [preauth]
Apr 13 08:57:13 grml sshd-session[2141]: Disconnecting authenticating user root 10.100.15.42 port 55810: Too many authentication failures [preauth]
Apr 13 08:57:13 grml sshd[1417]: drop connection #0 from [10.100.15.42]:55818 on [10.100.15.230]:22 penalty: failed authentication
Apr 13 08:57:14 grml sshd[1417]: drop connection #0 from [10.100.15.42]:55824 on [10.100.15.230]:22 penalty: failed authentication
Apr 13 08:57:14 grml sshd[1417]: drop connection #0 from [10.100.15.42]:55838 on [10.100.15.230]:22 penalty: failed authentication
Apr 13 08:57:14 grml sshd[1417]: drop connection #0 from [10.100.15.42]:55854 on [10.100.15.230]:22 penalty: failed authentication

This feature certainly is useful and has its use cases. But if you f.e. run automated checks to ensure that specific logins aren t working, be careful: you might hit the penalty feature, lock yourself out but also consecutive checks then don t behave as expected. Your login checks might fail, but only because the penalty behavior kicks in. The login you re verifying still might be working underneath, but you don t actually check for it exactly. Furthermore legitimate traffic from systems which accept connections from many users or behind shared IP addresses, like NAT and proxies could be denied. To disable this new behavior, you can set PerSourcePenalties no in your sshd_config, but there are also further configuration options available, see PerSourcePenalties and PerSourcePenaltyExemptList settings in sshd_config(5) for further details.

• debian-kernel mailing list:
  • replied to Agenda items for kernel-team meeting on 2025-03-05
  • posted Agenda items for kernel-team meeting on 2025-03-26 (and once more in this thread)
• debian-lts-announce mailing list:
  • posted [SECURITY] [DLA 4075-1] linux security update
  • posted [SECURITY] [DLA 4076-1] linux-6.1 security update
  • posted [SECURITY] [DLA 4102-1] linux-6.1 security update
• Debian firmware-free package:
  • Merge requests:
    • opened and merged !8: Use local copy of debian_linux Python package instead of linux-support
    • opened and merged !7: Add release-update script
• Debian firmware-nonfree package:
  • Bugs:
    • updated #1025476: firmware-linux: Unrecognized video cards and wifi
    • closed #1092342: firmware-iwlwifi: please backport firmware-iwlwifi_20241210-1_all.deb
    • updated #1079168: firmware-siano: Add Appstream metainfo announcing HW support
    • updated #1100723: firmware-misc-nonfree: ralink firmware disapears after upgrade due backport firmware-misc-free split in multiple packages
    • closed #1025476: firmware-linux: Unrecognized video cards and wifi
  • Merge requests:
    • merged !117: d/copyright: mediatek: Stop excluding *_MT7961*_1a*.bin and mt7996/* files
    • opened and merged !116: Use local copy of debian_linux Python package instead of linux-support
    • merged !115: Make firmware-intel-graphics trigger update-initramfs
    • merged !114: Update to 20250109
    • closed !113: qcom: Re-add qcm6490 firmware
    • closed !108: Add python to makedeps, add separate dedup stage
    • closed !79: Improve maintainer scripts of firmware-nonfree
• initramfs mailing list:
  • posted initramfs-tools 0.146 release
  • posted [ANNOUNCE] klibc 2.0.14
• Debian initramfs-tools package:
  • Bugs:
    • updated #1094569: initramfs-tools: Package failed to install: Error while installing package: installed initramfs-tools package post-installation script subprocess returned error exit status 1
    • updated #1088959: initramfs-tools: copy_exec copies file as symlink to the wrong binary instead
    • updated #1027458: MODULES=dep unbootable on Raspberry Pi 4 booting from USB
    • updated #1085262: Broken: ACPI DSDT loading does not work
    • updated #1065698: update-initramfs: -k all stopped working
    • updated #1027915: systemd requires /run to be mounted with a minimum size of 20MB
    • updated #1100008: initramfs-tools: causes autopkgtest regressions in cryptsetup, multipath-tools, maybe intel-microcode
    • updated #1099655: initramfs-tools 146 generates incorrect initramfs : does not boot, does not find root fs
    • closed #1082647: copy_exec: [regression] ignores trailing slash, installs file as directory name
    • closed #1088959: initramfs-tools: copy_exec copies file as symlink to the wrong binary instead
    • closed #1027458: MODULES=dep unbootable on Raspberry Pi 4 booting from USB
    • closed #1095991: initramfs-tools-core: make /dev/pts mode= consistent with systemd
    • closed #1100008: initramfs-tools: causes autopkgtest regressions in cryptsetup, multipath-tools, maybe intel-microcode
  • Merge requests:
    • merged !166: Bump Standards-Version to 4.7.2
    • merged !165: test: run quick copy-file autopkgtest first
    • closed !164: mkinitramfs: Last resort check of cryptoroot
    • merged !163: Fix spelling mistakes in d/changelog
    • opened and merged !162: unmkinitramfs: Restore split to early and main subdirectories
    • merged !161: Tighten /dev/pts permissions
    • merged !159: kernel/postinst.d: check for correct update-initramfs provider
    • opened and merged !158: hook-functions: Add reset drivers when MODULES=dep
    • merged !157: Avoid updating the initramfs twice for some cases
    • merged !154: Add qemu-net-iscsi autopkgtest
    • merged !151: hook-functions: add squashfs driver
    • merged !149: Filter block kernel modules by symbol regex from dracut
    • closed !138: fix: ACPI DSDT over-ride at boot
    • closed !137: kernel/postinst.d: Optimize execution for triggers
  • Uploads:
    • uploaded version 0.146 to unstable
    • uploaded version 0.147 to unstable
• Debian iproute2 package:
  • Bugs:
    • updated #1088739: colors make ip s output unreadable
    • closed #884060: iproute2: Upgrade from 4.9.0-2 to 4.9.0-2.1 breaks wireless.
    • closed #901660: iproute2: Using HFSC-qdisc causes WARNING-messages and stack-traces in syslog
    • updated #1054642: Failing ARP relay from external -> Linux bridge -> veth port > NS veth port
    • closed #481854: tc filters undocumented
    • closed #530564: tc htb quantum and r2q options undocumented
    • closed #1054044: iproute2: ss show wrong timer value in unit ms
• Debian klibc package:
  • Bugs:
    • updated #1063775: Plymouth, and cp: warning: behavior of -n is non-portable and may change in future; use update=none instead
    • updated #1061662: fstype support for bcachefs
    • updated #991943: klibc: please consider including machine-readable copyright file
    • closed #1075820: libklibc-dev: unusable on hppa with recent Linux: conflicting types for sigset_t
  • Uploads:
    • uploaded version 2.0.14-1 to unstable
• Debian kmod package:
  • Bugs:
    • updated #1059518: kmod: modprobe show-depends may not show all softdeps
• Debian kup package:
  • Merge requests:
    • merged !2: Recommends: gpg-agent instead of gnupg-agent (Closes: #1064112)
• Debian linux package:
  • Bugs:
    • closed #1091802: linux-image-6.1.0-27-amd64: Two times is detected touchpad on Asus Vivobook
    • updated #1100923: No config found for PCI dev 272b/40f0, rev=0x472, rfid=0x112200
    • updated #1101203: linux-image-6.12.19-amd64: Kernel 6.12.19-amd64 fails to connect display to X server
  • Merge requests:
    • closed !1431: fbdev/offb: Fix patch to update expected device name
    • opened and merged !1428: Fix a lintian error and many warnings
    • opened and closed !1427: linux-support: Remove unused files and drop ABI suffix
    • merged !1422: Drop support for HFS and HFS+ filesystems
    • merged !1355: Generate debhelper files per architecture
    • merged !1321: kconfigeditor2: document to run debian/rules orig first
  • Uploads:
    • uploaded linux-6.1 version 6.1.129-1~deb11u1 to bullseye-security
• linux-block mailing list:
  • replied to [PATCH V3] block: fix conversion of GPT partition name to 7-bit (and once more in this thread)
• linux-clk mailing list:
  • posted CVE-2024-50181: clk: imx: Remove CLK_SET_PARENT_GATE for DRAM mux for i.MX7D
• linux-kbuild mailing list:
  • replied to [PATCH] kbuild, rust: use -fremap-path-prefix to make paths relative
  • replied to [PATCH v2] kbuild: make all file references relative to source root
• linux-kselftest mailing list:
  • posted CVE-2024-56741: apparmor: test: Fix memory leak for aa_unpack_strdup()
• linux-perf-users mailing list:
  • posted [PATCH 0/2] perf docs: Fix build with asciidoctor (and several times more in this thread)
• linux-rt-users mailing list:
  • posted [PATCH 5.10] rt: Fix buid issue in be2net
• linux-spi mailing list:
  • posted CVE-2021-47469: spi: Fix deadlock when adding SPI controllers on SPI buses
• netdev mailing list:
  • posted [PATCH iproute2 1/2] color: Introduce and use default_color_opt() function (and once more in this thread)
  • posted [PATCH iproute2 0/2] Improve coloured text readability (and several times more in this thread)
• ntb mailing list:
  • posted CVE-2023-52917: ntb: intel: Fix the NULL vs IS_ERR() bug for debugfs_create_dir()
• Debian osk-sdl package:
  • Bugs:
    • updated #1089929: initramfs-tools: update-initramfs fails to generate initrd on arm64 (pinephone)
• stable mailing list:
  • replied to [PATCH 5.4,5.10 2/2] udf: Fix use of check_add_overflow() with mixed type arguments
  • replied to [PATCH 5.10 462/462] net: ipv6: fix dst refleaks in rpl, seg6 and ioam6 lwtunnels
  • replied to [PATCH 5.15 620/620] net: ipv6: fix dst refleaks in rpl, seg6 and ioam6 lwtunnels
• Debian wireless-regdb package:
  • Uploads:
    • uploaded version 2025.02.20-1 to unstable

• debian-events-eu mailing list:
  • posted Debian BSP in Leuven, March/April
• debian-kernel mailing list:
  • posted Agenda items for kernel-team meeting on 2025-02-05
  • posted Agenda items for kernel-team meeting on 2025-02-26
  • replied to Package for development of out-of-tree kernel modules written in Rust
• Debian initramfs-tools package:
  • Merge requests:
    • opened and merged !156: tests: Add e2fsprogs to dependencies
    • merged !155: test: log commands that produce output on stdout
    • merged !148: netinfo_to_resolv_conf: add net_file to local variables
    • merged !147: netinfo_to_resolv_conf: clear variables between sourcing
    • merged !146: Explicitly call set -e instead of specifying on shebang
    • merged !144: fix shellcheck complaints in tests
    • reviewed !142: Handle simple-framebuffer drivers and framebuffer_coreboot built as modules
    • merged !139: hook-functions: make copy_libgcc to not take any parameters
    • closed !134: add losetup-rootfs-img script: boot from rootfs image files
• Debian linux package:
  • Bugs:
    • updated #1087807: linux-image-6.1.0-27-amd64: Unable to boot: i40e swiotlb buffer is full
    • updated #1093371: megaraid_sas didn t work anymore with Xen
    • updated #1087807: linux-image-6.1.0-27-amd64: Unable to boot: i40e swiotlb buffer is full
    • updated #825423: supermin + sbuild + linux-image = broken chroot
    • updated #1093371: megaraid_sas didn t work anymore with Xen
    • updated #1050578: linux-image-6.1.0-11-amd64: kernel disk device cache coherency issue: stale reads on /dev/sda1
  • Updated the bullseye-security branch to upstream version 5.10.234
  • Uploads:
    • uploaded version 5.10.234-1 to bullseye-security
    • uploaded linux-6.1 version 6.1.128-1~deb11u1 to bullseye-security
• stable mailing list:
  • posted [PATCH 5.4,4.10 1/2] perf cs-etm: Add missing variable in cs_etm__process_queues() (and several more times in this thread)

• debian-kernel mailing list:
  • posted Agenda items for kernel-team meeting on 2025-01-15 (and once more in that thread)
• debian-lts-announce mailing list:
  • posted [SECURITY] [DLA 4008-1] linux-6.1 security update
• Debian initramfs-tools package:
  • Merge requests:
    • opened and merged !153: Some small fixes
    • merged !145: Drop obsolete shellcheck overrides
    • merged !143: Declare breaking miniramfs 1.0.2 and older
    • merged !141: hook-functions: Add msm to fb modules list
    • merged !140: hook-functions: Handle msm hidden depency on qrtr
    • merged !127: Filter net kernel modules by symbol regexp from dracut
    • closed !119: Fix resume device type check
    • closed !89: init: Remove tmpfs from rootfstype option
• Debian jinja-vanish package:
  • Bugs:
    • updated #1092861: jinja-vanish: please run tests at build time
• Debian kernel-team package:
  • Merge requests:
    • merged !6: Support new linux branch layout
• Debian linux package:
  • Merge requests:
    • merged !1296: d/p/debian/cdc_ncm-cdc_mbim-use-ncm-by-default.patch: drop
  • Updated the bullseye-security branch, but didn t upload
• Debian wireless-regdb package:
  • Bugs:
    • closed #1092796: Autopkgtests fail with Python 3.13: OSError: [Errno 9] Bad file descriptor
  • Merge requests:
    • merged !6: d/tests/check-signatures: Fix subprocess cleanup
  • Uploads:
    • uploaded version 2024.10.07-2 to unstable

• Debian cdrom bugs:
  • updated #1090340: cdrom: Debian 12 missing drivers for HyperV 2008 R2
• debian-kernel mailing list:
  • posted Agenda items for kernel-team meeting on 2024-12-04 (and once more in that thread)
  • replied to Debug symbols for linux-image-5.10.0-20-amd64
• Debian firmware-free package:
  • Merge requests:
    • opened and merged !6: Update to current upstream and linux-support versions
    • merged !5: Update to current upstream and linux-support versions
  • Uploads:
    • uploaded version 20241210-1 to unstable
• Debian firmware-nonfree package:
  • Merge requests:
    • opened and merged !112: Update to current upstream and linux-support versions
  • Uploads:
    • uploaded version 20241210-1 to unstable
• Debian linux package:
  • Bugs:
    • updated #1089521: linux-image-6.12.3-amd64 fails to boot due to Oops: invalid opcode: 0000 error
  • Updated the bullseye branch to upstream version 5.10.233, but didn t upload
• netdev mailing list:
  • replied to cpu_rmap maps CPUs to wrong interrupts after reprogramming affinities

• debian-boot mailing list:
  • posted partman recipes and deprecation of ext2 (and once more in that thread)
  • posted Moving kernel modules under /usr/lib in installer
• debian-devel mailing list:
  • replied to It makes no sense to link vmlinuz and initramfs to the root directory
  • replied to Is there a point to retaining src:pth?
  • replied to Moving apt (and hence bootstraps) from GnuPG to Sequioa (via gpgv-sq)
• debian-kernel mailing list:
  • posted Agenda items for kernel-team meeting on 2024-11-06
  • posted PSA: Renaming branches in kernel-team repositories (and several more times in that thread)
• Debian firmware-nonfree package:
  • Merge requests:
    • opened and merged !110: Update to linux-support-6.11.10
• Debian initramfs-tools package:
  • Bugs:
    • updated #1087697: Mkinitramfs causes machines to fail booting when used with LD_LIBRARY_PATH
  • Merge requests:
    • merged !128: Put compressed kernel modules and firmware in an uncompressed cpio
• Debian linux package:
  • Bugs:
    • updated #1086604: linux-image-6.11.4-amd64: amdgpu is unable to resume from suspend
    • updated #1086503: linux: Please get HWCAP and HWCAP2 in sync between 32bits and 64bits architecture
    • updated #1087623: linux-image-6.11.7-amd64: WLAN device disappears after resume from suspend or hibernation (RTL8852CE PCIe, rtw89_8852ce)
    • closed #1032671: linux: Enable USB_XHCI_PCI_RENESAS on arm64
    • closed #1082906: linux-image-6.11-amd64: ipu6 camera is not working
    • closed #1085600: d/patches: typo in patch file name
    • closed #1086335: linux: please enable ipu3 and cio2 kernel modules
    • closed #1086695: linux: Enable X86 userspace shadow stack
  • Merge requests:
    • merged !1276: fs/ntfs3: Enable NTFS3_FS as module
    • opened and merged !1274: linux-image: bug: Update list of related firmware packages
    • opened and merged !1265: Fix out-of-tree module build and missing test cases
    • merged !1262: libbpf: Add missing per-arch include path (fixes FTBFS on riscv64).
    • opened and merged !1260: objtool: Fix compiler flags leaking to fixdep in cross-build
    • merged !1253: [amd64] arch/x86: Enable X86_USER_SHADOW_STACK
    • merged !1251: [rt] Update to 6.12-rc4-rt6
    • merged !1250: Enable ov5670 video drivers
    • merged !1249: [amd64] Enable CIO2 and IPU3 kernel modules (Closes: #1086335)
    • merged !1246: [amd64] drivers/accel: Enable DRM_ACCEL
    • merged !1239: [x86] Enable Intel IPU supported camera sensors
    • merged !1235: Enable DRM_XE for x86 intel arc graphic card
    • merged !1233: d/config: enable IPE LSM
    • merged !1230: x86: intel_soc_pmic_bxtwc enable two sub-devices as module
    • merged !1226: [arm64] Enable CONFIG_SC_GCC/PINCTRL for 7280/7280/8180X/8280XP as builtin
    • merged !675: [arm64] drivers/usb/host: Enable USB_XHCI_PCI_RENESAS as module (Closes: #1032671)
  • Uploads:
    • uploaded version 6.12~rc6-1~exp1 to experimental
  • Updated the bullseye-security branch to upstream version 5.10.228, but didn t upload
• Debian procps package:
  • Bugs:
    • updated #1088203: linux-sysctl-defaults: apply setting after installation
• Debian systemd package:
  • Bugs:
    • updated #1085160: linux-sysctl-defaults: apply setting after installation
• Debian tech-ctte bugs:
  • updated #1065416: linux-libc-dev claims to provide linux-libc-dev-ARCH-cross, but it doesn t do that completely