About 80% of my Debian contributions this month were
sponsored by
Freexian, as well as one direct donation via GitHub
Sponsors (thanks!). If you
appreciate this sort of work and are at a company that uses Debian, have a
look to see whether you can pay for any of
Freexian s services; as well as the direct
benefits, that revenue stream helps to keep Debian development sustainable
for me and several other lovely
people.
You can also support my work directly via
Liberapay or GitHub
Sponsors.
Python packaging
New upstream versions:
Other bits and pieces
Alejandro Colomar reported that man(1) ignored the MANWIDTH environment variable in some circumstances. I investigated this and fixed it upstream.
I contributed an ubuntu-dev-tools patch to stop recommending sudo.
I added forky support to the images used in Salsa CI pipelines.
I began working on getting a release candidate of groff 1.24.0 into experimental, though haven t finished that yet.
I worked on some lower-priority security updates for OpenSSH.
Code reviews
Hello world ! I am an intern at Outreachy working with the Debian OpenQA team on images testing. We get to know what career opportunities awaits us when we work on open source projects. In open source, we are constantly learning. The community has different sets of skills and a large network of people.
So, how did I start off in this internship
I entered the community with the these skills:
MERN (Mongo DB, Express JS , React JS and Node JS) for web development
Linux and Shell Scripting for some administrative purposes
Containerization using Google Cloud
Operating Systems
A learning passion for Open Source I contributed to some open source work in the past but it was in terms of documentation and bug hunting
I was a newbie at OpenQA but, I had a month to learn and contribute. Time is a critical resource but so is understanding what you are doing. I followed the installations instructions given but whenever I got errors, I had to research why I got the errors. I took time to understand errors I was solving then continued on with the tasks I wanted to do. I communicated my logic and understanding while working on the task and awaited reviews and feedback. Within a span of two months I had learned a lot by practicing and testing.
The skills I gained
As of today, I gained these technical skills from my work with Debian OpenQA team.
Perl the tests that we run are written in this language
Ansible configuration ansible configurations and settings for the machines the team runs
Git this is needed for code versioning and diving tasks into different branches
Linux shell scripting and working with the Debian Operating system
Virtual Machines and Operating Systems I constantly view how different operating systems are booted and run on virtual machines during testing
Testing I keep watch of needles and ensure the tests work as required
Debian I use a Debian system to run my Virtual Machines
OpenQA the tool that is used to automate testing of Images
With open source comes the need of constant communication. The community is diverse and the team is usually on different time zones. These are some of the soft / social skills I gained when working with the team
Communication this is essential especially in taking tasks with confidence, talking about issues encountered and stating the progress of the tasks
Interpersonal skills this is for general communication within the community
Flexibility we have to adapt to changes because we are a community of different people with different skills
With these skills and the willingness to learn , open source is a great area to focus on . Aside from your career you will extend your network. My interests are set on open source and Linux in general. Working with a wider network has really skilled me up and I will continue learning. Working with the Debian OpenQA team has been very great. The team is great at communication and I learn every day. The knowledge I gain from the team is helping me build a great career in open source.
You might have seen Policy will reject signature within a year warnings in apt(-get) update runs like this:
root@424812bd4556:/# apt update
Get:1 http://foo.example.org/debian demo InRelease [4229 B]
Hit:2 http://deb.debian.org/debian trixie InRelease
Hit:3 http://deb.debian.org/debian trixie-updates InRelease
Hit:4 http://deb.debian.org/debian-security trixie-security InRelease
Get:5 http://foo.example.org/debian demo/main amd64 Packages [1097 B]
Fetched 5326 B in 0s (43.2 kB/s)
All packages are up to date.
Warning: http://foo.example.org/debian/dists/demo/InRelease: Policy will reject signature within a year, see --audit for details
root@424812bd4556:/# apt --audit update
Hit:1 http://foo.example.org/debian demo InRelease
Hit:2 http://deb.debian.org/debian trixie InRelease
Hit:3 http://deb.debian.org/debian trixie-updates InRelease
Hit:4 http://deb.debian.org/debian-security trixie-security InRelease
All packages are up to date.
Warning: http://foo.example.org/debian/dists/demo/InRelease: Policy will reject signature within a year, see --audit for details
Audit: http://foo.example.org/debian/dists/demo/InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is:
Signing key on 54321ABCD6789ABCD0123ABCD124567ABCD89123 is not bound:
No binding signature at time 2024-06-19T10:33:47Z
because: Policy rejected non-revocation signature (PositiveCertification) requiring second pre-image resistance
because: SHA1 is not considered secure since 2026-02-01T00:00:00Z
Audit: The sources.list(5) entry for 'http://foo.example.org/debian' should be upgraded to deb822 .sources
Audit: Missing Signed-By in the sources.list(5) entry for 'http://foo.example.org/debian'
Audit: Consider migrating all sources.list(5) entries to the deb822 .sources format
Audit: The deb822 .sources format supports both embedded as well as external OpenPGP keys
Audit: See apt-secure(8) for best practices in configuring repository signing.
Audit: Some sources can be modernized. Run 'apt modernize-sources' to do so.
If you ignored this for the last year, I would like to tell you that 2026-02-01 is not that far away (hello from the past if you re reading this because you re already affected).
Let s simulate the future:
root@424812bd4556:/# apt --update -y install faketime
[...]
root@424812bd4556:/# export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1 FAKETIME="2026-08-29 23:42:11"
root@424812bd4556:/# date
Sat Aug 29 23:42:11 UTC 2026
root@424812bd4556:/# apt update
Get:1 http://foo.example.org/debian demo InRelease [4229 B]
Hit:2 http://deb.debian.org/debian trixie InRelease
Err:1 http://foo.example.org/debian demo InRelease
Sub-process /usr/bin/sqv returned an error code (1), error message is: Signing key on 54321ABCD6789ABCD0123ABCD124567ABCD89123 is not bound: No binding signature at time 2024-06-19T10:33:47Z because: Policy rejected non-revocation signature (PositiveCertification) requiring second pre-image resistance because: SHA1 is not considered secure since 2026-02-01T00:00:00Z
[...]
Warning: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://foo.example.org/debian demo InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Signing key on 54321ABCD6789ABCD0123ABCD124567ABCD89123 is not bound: No binding signature at time 2024-06-19T10:33:47Z because: Policy rejected non-revocation signature (PositiveCertification) requiring second pre-image resistance because: SHA1 is not considered secure since 2026-02-01T00:00:00Z
[...]
root@424812bd4556:/# echo $?
100
Now, the proper solution would have been to fix the signing key underneath (via e.g. sq cert lint &dash&dashfix &dash&dashcert-file $PRIVAT_KEY_FILE > $PRIVAT_KEY_FILE-fixed).
If you don t have access to the according private key (e.g. when using an upstream repository that has been ignoring this issue), you re out of luck for a proper fix.
But there s a workaround for the apt situation (related see apt commit 0989275c2f7afb7a5f7698a096664a1035118ebf):
root@424812bd4556:/# cat /usr/share/apt/default-sequoia.config
# Default APT Sequoia configuration. To overwrite, consider copying this
# to /etc/crypto-policies/back-ends/apt-sequoia.config and modify the
# desired values.
[asymmetric_algorithms]
dsa2048 = 2024-02-01
dsa3072 = 2024-02-01
dsa4096 = 2024-02-01
brainpoolp256 = 2028-02-01
brainpoolp384 = 2028-02-01
brainpoolp512 = 2028-02-01
rsa2048 = 2030-02-01
[hash_algorithms]
sha1.second_preimage_resistance = 2026-02-01 # Extend the expiry for legacy repositories
sha224 = 2026-02-01
[packets]
signature.v3 = 2026-02-01 # Extend the expiry
Somehow this whole DevOps thing is all about generating the wildest things from some (usually equally wild) template.
And today we're gonna generate YAML from ERB, what could possibly go wrong?!
Well, actually, quitea lot,
so one wants to validate the generated result before using it to break systems at scale.
The YAML we generate is a cloud-init cloud-config,
and while checking that we generated a valid YAML document is easy (and we were already doing that),
it would be much better if we could check that cloud-init can actually use it.
Enter cloud-init schema, or so I thought.
Turns out running cloud-init schema is rather broken without root privileges,
as it tries to load a ton of information from the running system.
This seems like a bug (or multiple), as the data should not be required for the validation of the schema itself.
I've not found a way to disable that behavior.
Luckily, I know Python.
Enter evgeni-knows-better-and-can-write-python:
#!/usr/bin/env python3importsysfromcloudinit.config.schemaimportget_schema,validate_cloudconfig_file,SchemaValidationErrortry:valid=validate_cloudconfig_file(config_path=sys.argv[1],schema=get_schema())ifnotvalid:raiseRuntimeError("Schema is not valid")except(SchemaValidationError,RuntimeError)ase:print(e)sys.exit(1)
The canonical1 version if this lives in the Foreman git repo, so go there if you think this will ever receive any updates.
The hardest part was to understand thevalidate_cloudconfig_file API,
as it will sometimes raise an SchemaValidationError,
sometimes a RuntimeError and sometimes just return False.
No idea why.
But the above just turns it into a couple of printed lines and a non zero exit code,
unless of course there are no problems, then you get peaceful silence.
Hello world . I am an intern here at Outreachy working with Debian OpenQA Image testing team. The work consists of testing Images with OpenQA. The internship has reached midpoint and here are some of the highlights that I have had so far.
The mentors : Roland Clobus, Tassia Camoes and Philip Hands are very good mentors. I like the constant communication and the help I get while working on the project. I enjoy working with this team.
The community : The contributors, mentors and the greater SUSE OpenQA community are constantly in communication. I learn a lot from these meetings.
The women network : The women of Debian meet and network . The meetings are interactive and we are encouraged to interact.
The project : We are making progress one step at a time. Isoken Ibizugbe is my fellow intern working on start-stop tests. I am working on live installers tests.
Communication
I have learned a lot during my internship. I have always been on the silent path of life with little communication. I once told myself being a developer would hide me behind a computer to avoid socializing. Being in open source especially this internship has helped me out with communication and networking. The team work in the project has helped me a lot
My mentors encourage communication. Giving project updates and stating when we get stuck.
My mentors have scheduled weekly meetings to communicate about the project
We are constantly invited to the SUSE meetings by mentors or by Sam Thursfield who is part of the team.
Female contributors are encouraged to join Debian women monthly meetings for networking
Lessons so far
I have had challenges , solved problems and learned new skills all this while
I have learned Perl, OpenQA configuration, needle editing and improved my Linux and Git skills
I have known how various Images are installed , booted and run through live viewing of tests
I have solved many test errors and learned to work with applications that are needed in the OS installations. e.g. rufus
I have learned how virtual machines work and how to solve errors in regards to them
So far so good. I am grateful to be a contributor towards the project and hope to continue learning.
After experimenting at home, the very first professional-grade NAS from
Terramaster arrived at work, too, with 12 HDD bays and possibly a pair of M2s.
NVME cards. In this case, I again installed a plain Debian distribution, but HDD
monitoring required some configuration adjustments to run smartd properly.A decent approach to data safety is to run regularly scheduled short and long
SMART tests on all disks to detect potential damage. Running such tests on all
disks at once isn't ideal, so I set up a script to create a staggered
configuration and test multiple groups of disks at different times. Note that it
is mandatory to read the devices at each reboot because their names and order
can change.Of course, the same principle (short/long test at regular intervals along the
week) should be applied for a simpler configuration, as in the case of my home
NAS with a pair of RAID1 devices.What follows is a simple script to create a staggered smartd.conf at boot
time:
#!/bin/bash
#
# Save this as /usr/local/bin/create-smartd-conf.sh
#
# Dynamically generate smartd.conf with staggered SMART test scheduling
# at boot time based on discovered ATA devices
# HERE IS A LIST OF DIRECTIVES FOR THIS CONFIGURATION FILE.
# PLEASE SEE THE smartd.conf MAN PAGE FOR DETAILS
#
# -d TYPE Set the device type: ata, scsi[+TYPE], nvme[,NSID],
# sat[,auto][,N][+TYPE], usbcypress[,X], usbjmicron[,p][,x][,N],
# usbprolific, usbsunplus, sntasmedia, sntjmicron[,NSID], sntrealtek,
# ... (platform specific)
# -T TYPE Set the tolerance to one of: normal, permissive
# -o VAL Enable/disable automatic offline tests (on/off)
# -S VAL Enable/disable attribute autosave (on/off)
# -n MODE No check if: never, sleep[,N][,q], standby[,N][,q], idle[,N][,q]
# -H Monitor SMART Health Status, report if failed
# -s REG Do Self-Test at time(s) given by regular expression REG
# -l TYPE Monitor SMART log or self-test status:
# error, selftest, xerror, offlinests[,ns], selfteststs[,ns]
# -l scterc,R,W Set SCT Error Recovery Control
# -e Change device setting: aam,[N off], apm,[N off], dsn,[on off],
# lookahead,[on off], security-freeze, standby,[N off], wcache,[on off]
# -f Monitor 'Usage' Attributes, report failures
# -m ADD Send email warning to address ADD
# -M TYPE Modify email warning behavior (see man page)
# -p Report changes in 'Prefailure' Attributes
# -u Report changes in 'Usage' Attributes
# -t Equivalent to -p and -u Directives
# -r ID Also report Raw values of Attribute ID with -p, -u or -t
# -R ID Track changes in Attribute ID Raw value with -p, -u or -t
# -i ID Ignore Attribute ID for -f Directive
# -I ID Ignore Attribute ID for -p, -u or -t Directive
# -C ID[+] Monitor [increases of] Current Pending Sectors in Attribute ID
# -U ID[+] Monitor [increases of] Offline Uncorrectable Sectors in Attribute ID
# -W D,I,C Monitor Temperature D)ifference, I)nformal limit, C)ritical limit
# -v N,ST Modifies labeling of Attribute N (see man page)
# -P TYPE Drive-specific presets: use, ignore, show, showall
# -a Default: -H -f -t -l error -l selftest -l selfteststs -C 197 -U 198
# -F TYPE Use firmware bug workaround:
# none, nologdir, samsung, samsung2, samsung3, xerrorlba
# -c i=N Set interval between disk checks to N seconds
# # Comment: text after a hash sign is ignored
# \ Line continuation character
# Attribute ID is a decimal integer 1 <= ID <= 255
# except for -C and -U, where ID = 0 turns them off.
set -euo pipefail
# Test schedule configuration
BASE_SCHEDULE="L/../../6" # Long test on Saturdays
TEST_HOURS=(01 03 05 07) # 4 time slots: 1am, 3am, 5am, 7am
DEVICES_PER_GROUP=3
main()
# Get array of device names (e.g., sda, sdb, sdc)
mapfile -t devices < <(ls -l /dev/disk/by-id/ grep ata awk ' print $11 ' grep sd cut -d/ -f3 sort -u)
if [[ $ #devices[@] -eq 0 ]]; then
exit 1
fi
# Start building config file
cat << EOF
# smartd.conf - Auto-generated at boot
# Generated: $(date '+%Y-%m-%d %H:%M:%S')
#
# Staggered SMART test scheduling to avoid concurrent disk load
# Long tests run on Saturdays at different times per group
#
EOF
# Process devices into groups
local group=0
local count_in_group=0
for i in "$ !devices[@] "; do
local dev="$ devices[$i] "
local hour="$ TEST_HOURS[$group] "
# Add group header at start of each group
if [[ $count_in_group -eq 0 ]]; then
echo ""
echo "# Group $((group + 1)) - Tests at $ hour :00 on Saturdays"
fi
# Add device entry
#echo "/dev/$ dev -a -o on -S on -s ($ BASE_SCHEDULE /$ hour ) -m root"
echo "/dev/$ dev -a -o on -S on -s (L/../../6/$ hour ) -s (S/../.././$(((hour + 12) % 24))) -m root"
# Move to next group when current group is full
count_in_group=$((count_in_group + 1))
if [[ $count_in_group -ge $DEVICES_PER_GROUP ]]; then
count_in_group=0
group=$(((group + 1) % $ #TEST_HOURS[@] ))
fi
done
main "$@"
To run such a script at boot, add a unit file to the systemd configuration.
[Unit]
Description=Generate smartd.conf with staggered SMART test scheduling
# Wait for all local filesystems and udev device detection
After=local-fs.target systemd-udev-settle.service
Before=smartd.service
Wants=systemd-udev-settle.service
DefaultDependencies=no
[Service]
Type=oneshot
# Only generate the config file, don't touch smartd here
ExecStart=/bin/bash -c '/usr/local/bin/create-smartd-config.sh > /etc/smartd.conf'
StandardOutput=journal
StandardError=journal
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
mrcal 2.5 is out: the release notes. Once again, this is mostly a bug-fix
release en route to the big new features coming in 3.0.
One cool thing is that these tools have now matured enough to no longer be
considered experimental. They have been used with great success in lots of
contexts across many different projects and organizations. Some highlights:
I've calibrated extremely wide lenses
and extremely narrow lenses
and joint systems containing many different kinds of lenses
with lots of cameras at the same time. The biggest single joint calibration
I've done today had 10 cameras, but I'll almost certainly encounter bigger
systems in the future
mrcal has been used to process both visible and thermal cameras
The new triangulated-feature capability has been used in a
structure-from-motion context to compute the world geometry on-line.
mrcal has been used with weird experimental setups employing custom
calibration objects and single-view solves
Some of the above is new, and not yet fully polished and documented and tested,
but it works.
In mrcal 2.5, most of the implementation of some new big features is written
and committed, but it's still incomplete. The new stuff is there, but is lightly
tested and documented. This will be completed eventually in mrcal 3.0:
More general view of uncertainty and diffs. I want to support extrinsics-only
and/or intrinsics computations-only in lots of scenarios. Uncertainty in point
solves is already available in some conditions, for instance if the points are
fixed. New mrcal-show-stereo-pair-diff tool reports an extrinsics+intrinsics
diff between two calibrations of a stereo pair; experimental
analyses/extrinsics-stability.py tool reports an extrinsics-only diff. These
are in contrast to the intrinsics-only uncertainty and diffs in the existing
mrcal-show-projection-diff and mrcal-show-projection-uncertainty tools.
Some documentation in the uncertainty and differencing pages.
Implicit point solves, using the triangulation routines in the optimization
cost function. Should produce much more efficient structure-from-motion
solves. This is all the "triangulated-features" stuff. The cost function is
primarily built around _mrcal_triangulated_error(). This is demoed in
test/test-sfm-triangulated-points.py. And I've been using
_mrcal_triangulated_error() in structure-from-motion implementations within
other optimization routines.
mrcal is quite good already, and will be even better in the future. Try it
today!
A brand new release 0.1.15 of the RcppSimdJson
package is now on CRAN.
RcppSimdJson
wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire and collaborators. Via
very clever algorithmic engineering to obtain largely branch-free code,
coupled with modern C++ and newer compiler instructions, it results in
parsing gigabytes of JSON parsed per second which is quite
mindboggling. The best-case performance is faster than CPU speed as
use of parallel SIMD instructions and careful branch avoidance can lead
to less than one cpu cycle per byte parsed; see the video of the talk by Daniel Lemire
at QCon.
This version updates to the current 4.2.4 upstream release. It also
updates the RcppExports.cpp file with glue between C++
and R. We want move away from using Rf_error() (as
Rcpp::stop() is generally preferable). Packages (such as
this one) that are declaring an interface have an actual
Rf_error() call generated in RcppExports.cpp
which can protect which is what current Rcpp code
generation does. Long story short, a minor internal reason.
The short NEWS entry for this release follows.
Changes in version 0.1.15
(2026-01-14)
simdjson was upgraded to version 4.2.4
(Dirk in #97
RcppExports.cpp was regenerated to aid a Rcpp transition
Standard maintenance updates for continuous integration and
URLs
In the past few years, the size of the kernel images in Debian have been
steadily growing. I don't see this as a problem per se, but it has been causing
me trouble, as my /boot partition has become too small to accommodate two
kernel images at the same time.
Since I'm running Debian Unstable on my personal systems and keep them updated
with unattended-upgrade, this meant each (frequent) kernel upgrade triggered
an error like this one:
update-initramfs: failed for /boot/initrd.img-6.17.11+deb14-amd64 with 1.
dpkg: error processing package initramfs-tools (--configure):
installed initramfs-tools package post-installation script subprocess returned
error exit status 1
Errors were encountered while processing:
initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)
This would in turn break the automated upgrade process and require me to
manually delete the currently running kernel (which works, but isn't great) to
complete the upgrade.
The "obvious" solution would have been to increase the size of my /boot
partition to something larger than the default 456M. Since my systems use
full-disk encryption and LVM, this isn't trivial and would have required me to
play Tetris and swap files back and forth using another drive.
Another solution proposed by anarcat was to migrate to systemd-boot
(I'm still using grub), use Unified Kernel Images (UKI) and merge the
/boot and /boot/efi partitions. Since I already have a bunch of
configurations using grub and I am not too keen on systemd taking over all the
things on my computer, I was somewhat reluctant.
As my computers are all configured by Puppet, I could of course have done a
complete system reinstallation, but again, this was somewhat more involved than
what I wanted it to be.
After looking online for a while, I finally stumbled on this blog post by Neil
Brown detailing how to shrink the size of the initramfs images. With
MODULES=dep my images shrunk from 188M to 41M, fixing my issue. Thanks
Neil!
I was somewhat worried removing kernel modules would break something on my
systems, but so far, I only had to manually load the i2c_dev module, as I
need it to manage my home monitor's brightness using ddcutil.
dh-python development, by Stefano Rivera
In Debian we build our Python packages with the help of a debhelper-compatible
tool, dh-python. Before starting the
3.14 transition (that would rebuild many packages) we landed some updates to
dh-python to fix bugs and add features. This started
a month of attention
on dh-python, iterating through several bug fixes, and a couple of unfortunate
regressions.
dh-python is used by almost all packages containing Python (over 5000). Most
of these are very simple, but some are complex and use dh-python in unexpected
ways. It s hard to avoid almost any change (including obvious bug fixes) from
causing some unexpected knock-on behaviour. There is a fair amount of complexity
in dh-python, and some rather clever code, which can make it tricky to work on.
All of this means that good QA is important. Stefano spent some time
adding type annotations
and specialized types to make it easier to see what the code is doing and catch
mistakes. This has already made work on dh-python easier.
Now that Debusine has built-in repositories
and debdiff support, Stefano could quickly test the effects of changes on many
other packages. After each big change, he could upload dh-python to
a repository,
rebuild e.g. 50 Python packages with it, and see what differences appeared in
the output. Reviewing the diffs is still a manual process, but can be improved.
Stefano did a small test
on what it would take to replace direct setuptools setup.py calls with
PEP-517 (pyproject-style) builds. There is
more work to do here.
Python 3.14 transition, by Stefano Rivera (et al.)
In December the transition to add
Python 3.14 as a supported version started in Debian unstable. To do this, we
update the list of supported versions in python3-defaults,
and then start rebuilding modules with C extensions from the leaves inwards.
This had already been tested in a PPA and Ubuntu, so many of the biggest
blocking compatibility issues with 3.14 had already been found and fixed. But
there are always new issues to discover.
Thanks to a number of people in the Debian Python team, we got through the first
bit of the transition fairly quickly. There are still a number of
open bugs
that need attention and many failed tests
blocking migration to testing.
Python 3.14.1 released just after we started the transition, and very soon
after, a follow-up 3.14.2 release came out to address a regression. We ran into
another regression in
Python 3.14.2.
Ruby 3.4 transition, by Lucas Kanashiro (et al.)
The Debian Ruby team just started the preparation to move the default Ruby
interpreter version to 3.4. At the moment, ruby3.4 source package is already
available in experimental, also ruby-defaults added support to
Ruby 3.4. Lucas rebuilt all reverse dependencies against this new version of the
interpreter and published the results here.
Lucas also reached out to some stakeholders to coordinate the work.
Next steps are: 1) announcing the results to the whole team and asking for help
to fix packages failing to build against the new interpreter; 2) file bugs
against packages FTBFSing against Ruby 3.4 which are not fixed yet; 3) once we
have a low number of build failures against Ruby 3.4, ask the Debian Release
team to start the transition in unstable.
Surviving scraper traffic in Debian CI, by Antonio Terceiro
Like most of the open web, Debian Continuous Integration
has been struggling for a while to keep up with the insatiable hunger from data
scrapers everywhere. Solving this involved a lot of trial and error; the final
result seems to be stable, and consists of two parts.
First, all Debian CI data pages, except the direct links to test log files
(such as those provided by the Release Team s testing migration excuses), now
require users to be authenticated before being accessed. This means that the
Debian CI data is no longer publicly browseable, which is a bit sad. However,
this is where we are now.
Additionally, there is now a fail2ban powered firewall-level access limitation
for clients that display an abusive access pattern. This went through several
iterations, with some of them unfortunately blocking legitimate Debian
contributors, but the current state seems to strike a good balance between
blocking scrapers and not blocking real users. Please get in touch with the team
on the #debci OFTC channel if you are affected by this.
A hybrid dependency solver for crossqa.debian.net, by Helmut Grohne
crossqa.debian.net continuously cross builds
packages from the Debian archive. Like Debian s native build infrastructure, it
uses dose-builddebcheck to determine whether a package s dependencies can be
satisfied before attempting a build. About one third of Debian s packages fail
this check, so understanding the reasons is key to improving cross building.
Unfortunately, dose-builddebcheck stops after reporting the first problem and
does not display additional ones.
To address this, a greedy solver implemented in Python now examines each
build-dependency individually and can report multiple causes. dose-builddebcheck
is still used as a fall-back when the greedy solver does not identify any
problems. The report for bazel-bootstrap
is a lengthy example.
rebootstrap, by Helmut Grohne
Due to the changes suggested by Loongson earlier, rebootstrap now adds
debhelper to its final installability test and builds a few more packages
required for installing it. It also now uses a variant of build-essential that
has been markedMulti-Arch: same
(see foundational work
from last year).
This in turn made the use of a non-default GCC version more difficult and
required more work to make it work for gcc-16 from experimental. Ongoing
archive changes temporarily regressed building fribidi and dash.
libselinux and groff have received patches for architecture specific changes
and libverto has been NMUed to remove the glib2.0 dependency.
Miscellaneous contributions
Stefano did some administrative work on debian.social and debian.net
instances and Debian reimbursements.
Stefano did routine updates of python-authlib, python-mitogen, xdot.
Stefano spent several hours discussing Debian s Python package layout with the
PyPA upstream community. Debian has ended up with a very different on-disk
installed Python layout than other distributions, and this continues to cause
some frustration in many communities that have to have special workarounds to
handle it. This ended up impacting cross builds as Helmut discovered.
Zulip is not yet in Debian (RFP in #800052),
but Rapha l helped on the French translation as he is experimenting with that
discussion platform.
Antonio performed several routine Salsa maintenance tasks, including
fixing salsa-nm-sync,
the service that synchronizes project members data from LDAP to Salsa, which had
been broken since salsa.debian.org was upgraded to
trixie .
Antonio deployed a new amd64 worker host for Debian CI.
Carles reviewed and submitted 14 Catalan translations using
po-debconf-manager.
Carles improved po-debconf-manager: added delete-package command,
show-information now uses properly formatted output (YAML), it now attaches
the translation on the bug reports for which a merge request has been opened too
long.
Carles investigated why some packages appeared in po-debconf-manager but not
in the Debian l10n list.
Turns out that some packages had debian/po/templates.pot (appearing in
po-debconf-manager) but not the POTFILES.in file as expected.
Created a script
to find out which packages were in this or similar situation and
reported bugs.
Carles tested and documented how to set
up voices (mbrola and festival) if using Orca speech synthesizer. Commented
a few issues and possible improvements in the debian-accessibility list.
Helmut sent patches for 48 cross build failures and initiated discussions on
how to deal with two non-trivial matters. Besides Python mentioned above,
CMake introduced a cmake_pkg_config builtin which is
not aware of the host architecture. He also
forwarded a Meson patch upstream.
Thorsten uploaded a new upstream version of cups to fix a nasty bug that was
introduced by the latest security update.
Colin used Debusine to coordinate transitions for astroid and pygments,
and wrote up the astroid case
on his blog.
Emilio helped with various transitions, and provided a build fix for opencv
for the ffmpeg 8 transition.
Emilio tested the GNOME updates for trixie proposed updates (gnome-shell,
mutter, glib2.0).
Santiago helped to review the status of how to test different build profiles
in parallel on the same pipeline, using the test-build-profiles job. This means,
for example, to simultaneously test build profiles such as nocheck and nodoc
for the same git tree. Finally, Santiago provided
MR !685
to fix the documentation.
Anupa prepared a bits post for Outreachy interns
announcement
along with T ssia Cam es Ara jo and worked on publicity team tasks.
Team Rcpp is thrilled to share that an exciting new version 1.1.1 of
Rcpp is now on CRAN (and also
uploaded to Debian and already
built for r2u).
Having switchted to C++11 as the minimum standard in the
previous 1.1.0 release, this version takes full advantage of it and
removes a lot of conditional code catering to older standards
that no longer need to be supported. Consequently, the source tarball
shrinks by 39% from 3.11 mb to 1.88 mb. That is a big deal. (Size peaked
with Rcpp 1.0.12 two years ago at 3.43 mb; relative to its size we are
down 45% !!) Removing unused code also makes maintenance easier, and
quickens both compilation and installation in general.
This release continues as usual with the six-months January-July
cycle started with release
1.0.5 in July 2020. Interim snapshots are always available via the
r-universe page and
repo. We continue to strongly encourage the use of these development
released and their testing we tend to run our systems with them too.
Rcpp has long established itself
as the most popular way of enhancing R with C or C++ code. Right now,
3020 packages on CRAN depend on
Rcpp for making analytical code go
faster and further. On CRAN, 13.1% of all packages depend (directly) on
Rcpp, and 60.9% of all compiled
packages do. From the cloud mirror of CRAN (which is but a subset of all
CRAN downloads), Rcpp has been
downloaded 109.8 million times. The two published papers (also included
in the package as preprint vignettes) have, respectively, 2151 (JSS, 2011) and 405 (TAS, 2018)
citations, while the the book (Springer useR!,
2013) has another 715.
This time, I am not attempting to summarize the different changes.
The full list follows below and details all these changes, their
respective PRs and, if applicable, issue tickets. Big thanks from all of
us to all contributors!
Changes in
Rcpp release version 1.1.1 (2026-01-08)
Changes in Rcpp API:
An unused old R function for a compiler version check has been
removed after checking no known package uses it (Dirk in #1395)
A narrowing warning is avoided via a cast (Dirk in #1398)
Demangling checks have been simplified (I aki in #1401 addressing
#1400)
The treatment of signed zeros is now improved in the Sugar code
(I aki in #1404)
Preparations for phasing out use of Rf_error have
been made (I aki in #1407)
The long-deprecated function loadRcppModules() has
been removed (Dirk in #1416 closing #1415)
Some non-API includes from R were refactored to accommodate
R-devel changes (I aki in #1418 addressing
#1417)
An accessor to Rf_rnbeta has been removed (Dirk in
#1419 also
addressing #1420)
Code accessing non-API Rf_findVarInFrame now uses
R_getVarEx (Dirk in #1423 fixing #1421)
Code conditional on the R version now expects at least R 3.5.0;
older code has been removed (Dirk in #1426 fixing #1425)
The non-API ATTRIB entry point to the R API is no
longer used (Dirk in #1430 addressing
#1429)
The unwind-protect mechanism is now used unconditionally (Dirk in
#1437 closing
#1436)
Changes in Rcpp Attributes:
The OpenMP plugin has been generalized for different macOS compiler
installations (Kevin in #1414)
Changes in Rcpp Documentation:
Vignettes are now processed via a new "asis" processor adopted
from R.rsp (Dirk in #1394 fixing #1393)
R is now cited via its DOI (Dirk)
A (very) stale help page has been removed (Dirk in #1428 fixing #1427)
The main README.md was updated emphasizing r-universe in favor of
the local drat repos (Dirk in #1431)
Changes in Rcpp Deployment:
A temporary change in R-devel concerning NA part in complex
variables was accommodated, and then reverted (Dirk in #1399 fixing #1397)
The macOS CI runners now use macos-14 (Dirk in #1405)
A message is shown if R.h is included before Rcpp
headers as this can lead to errors (Dirk in #1411 closing #1410)
Old helper functions use message() to signal they
are not used, deprecation and removal to follow (Dirk in #1413 closing #1412)
Three tests were being silenced following #1413 (Dirk in #1422)
The heuristic whether to run all available tests was refined
(Dirk in #1434
addressing #1433)
Coverage has been tweaked via additional #nocov tags
(Dirk in #1435)
Non-release Changes:
Two interim non-releases 1.1.0.8.1 and .2 were made in order to
unblock CRAN due to changes in R-devel rather than Rcpp
Welcome to the December 2025 from the Reproducible Builds project!
Our monthly reports outline what we ve been up to over the past month, highlighting items of news from elsewhere in the increasingly-important area of software supply-chain security. As ever, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website.
New orig-check service to validate Debian upstream tarballs
This month, Debian Developer Lucas Nussbaum announced the orig-check service, which attempts to automatically reproduce the generation upstream tarballs (ie. the original source component of a Debian source package), comparing that to the upstream tarball actually shipped with Debian.
As of the time of writing, it is possible for a Debian developer to upload a source archive that does not actually correspond to upstream s version. Whilst this is not inherently malicious (it typically indicates some tooling/process issue), the very possibility that a maintainer s version may differ potentially permits a maintainer to make (malicious) changes that would be misattributed to upstream.
This service therefore nicely complements the whatsrc.org service, which was reported in our reports for both April and August. The orig-check is dedicated to Lunar, who sadly passed away a year ago.
disorderfs updated to FUSE 3
disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into system calls to reliably flush out reproducibility issues.
This month, however, Roland Clobus upgraded disorderfs* from FUSE 2 to FUSE 3 after its package automatically got removed from Debian testing. Some tests in Debian currently require disorderfs to make the Debian live images reproducible, although disorderfs is not a Debian-specific tool.
Luca Di Maio announced stampdalf, a filesystem timestamp preservation tool that wraps arbitrary commands and ensures filesystem timestamp reproducibility :
stampdalf allows you to run any command that modifies files in a directory tree, then automatically resets all timestamps back to their original values. Any new files created during command execution are set to [the UNIX epoch] or a custom timestamp via SOURCE_DATE_EPOCH.
The project s GitHub page helpfully reveals that the project is pronounced: stamp-dalf (stamp like time-stamp, dalf like Gandalf the wizard) as it s a wizard of time and stamps .)
Lastly, Reproducible Builds developer cen1 posted to our list announcing that early/experimental/alpha support for FreeBSD was added to rebuilderd. In their post, cen1 reports that the initial builds are in progress and look quite decent . cen1 also interestingly notes that since the upstream is currently not technically reproducible I had to relax the bit-for-bit identical requirement of rebuilderd [ ] I consider the pkg to be reproducible if the tar is content-identical (via diffoscope), ignoring timestamps and some of the manifest files. .
Most package managers also generate a lockfile, which records the exact set of resolved dependency versions. Lockfiles are used to reduce build times; to verify the integrity of resolved packages; and to support build reproducibility across environments and time. Despite these beneficial features, developers often struggle with their maintenance, usage, and interpretation. In this study, we unveil the major challenges related to lockfiles, such that future researchers and engineers can address them. [ ]
While existing studies primarily focus on software supply chain attacks prevention and detection methods, there is a need for a broad overview of attacks and comprehensive risk assessment for software supply chain security. This study conducts a systematic literature review to fill this gap. By analyzing 96 papers published between 2015-2023, we identified 19 distinct SSC attacks, including 6 novel attacks highlighted in recent studies. Additionally, we developed 25 specific security controls and established a precisely mapped taxonomy that transparently links each control to one or more specific attacks. [ ]
[Achieving] reproducibility at scale remains difficult, especially in Java, due to a range of non-deterministic factors and caveats in the build process. In this work, we focus on reproducibility in Java-based software, archetypal of enterprise applications. We introduce a conceptual framework for reproducible builds, we analyze a large dataset from Reproducible Central, and we develop a novel taxonomy of six root causes of unreproducibility. [ ]
Robert Stupp made a number of useful changes, fixing and reorganising the Groovy / Kotlin pages [][][] as well adding a note about potential non-deterministic behaviour to the JVM page [].
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
Hello world . I am an intern at Outreachy and contributing to the Debian Images Testing project since October 2025. This project is Open Source and everyone can contribute to it in any way. The project uses Open QA to automatically install Operating System Images and test them . We have a community here of contributors that is always ready to help out. The mentors and project maintainers are very open to contributions. They listen to any innovative ideas and point out what they have been doing so far.
So far contributions have been in terms of:
Documentation Adding to install guides
Pseudo Tests Suggesting an idea after finding an error or idea
Pointing out bugs / errors when trying out tests
Work on tests
Contribute towards the wider community / help out other contributors
Heck, you can even create a screen-cast video doing the set up and add it to the guide / docs
Developers with further understanding can try to work with the maintainers on packages
Contributing to this project requires some knowledge of Linux commands and Operating Systems. What we will learn later as we go on will be :
Images / Operating System Installation through dual booting
More Linux commands
Images / Operating System Installation and testing on Virtual Machine
Git commands
Writing testing documentation
Writing Pseudo Tests
Writing test modules / code using Perl
Working on configuration
Preparation
Before any contribution begins, we would want to first try out the project and run a couple of tests. Get to understand what we are doing first. Let s say you are starting out as a Windows or MacOS user and you want to start contributing. I would recommend dual partitioning your device first. Do enough research and prepare the resources. The network install image just needs at-least 4 GB USB flash drive for dual booting. You will use Debian as the second operating system. Give enough space to Debian, I recommend around 150 GB or more. Also assign at-least 1 GB space to the /boot/efi directory to prevent the low space warnings after a while.This will be a way good way to learn about image installation which is part of the work . I do not recommend Virtual Box because it will hinder full use of system resources. This process will take a day or two.
Set Up and Testing
After dual booting. We log into our Debian System . The next step of instructions will take you through how we set up and run our tests. These instructions have many Linux commands. You will be learning if you are a newbie as you go through the steps. Try to understand these commands and do not blindly copy and paste. You can start your contributions here if you have a suggestion to add to the install docs. Run some tests then log in the Web UI as per the instructions to view your tests progress. Green means they ve passed. Blue means its still running. Red means failed.
Trying Out Ideas
Kudos if you have reached this point. The community of contributors will help if you are stuck. We get to try out our own variations of tests using variables. We will also rely on documentation to understand the configurations / test commands like these
openqa-cli api -X POST isos ISO=debian-13.1.0-amd64-netinst.iso DISTRI=debian VERSION=stable FLAVOR=netinst-iso ARCH=x86_64 BUILD=1310 #This is the test you will run on the guide
openqa-cli api -X POST isos ISO=debian-13.1.0-amd64-netinst.iso DISTRI=debian VERSION=stable FLAVOR=netinst-iso ARCH=x86_64 BUILD=1310 TEST=cinnamon #I have added a TEST variable that runs only cinnamon test suite
You can check specific test suites from the WebUI :
We get some failures at times. Here are some failed tests from a build I was working on.
Here we find the cinnamon test failed at locale module. Click on any module above and it will lead us to needles and point to the where the test failed. You can check the error or try to add a needle if its needle failure.
Try editing a test module and test your changes. Try out some ideas. Read the documentation folder and write some pseudo code. Interact with the community. Try working on some tasks from the community . Create your tests and add them to the configuration. There is a lot of stuff that can you can work on in this community.
It may seem hard to grasp at first as a newbie to Open Source. The community will help you through out even if the problem seems small. We are very friendly and the code maintainers have extensive knowledge. Get to sit with us during one of our meetings and you will learn so much about the project. Learning , networking and communicating is part of contributing to the broader community.
About 95% of my Debian contributions this month were
sponsored by Freexian.
You can also support my work directly via
Liberapay or GitHub
Sponsors.
Python packaging
I upgraded these packages to new upstream versions:
Python 3.14 is now a supported version in unstable, and we re working to get
that into testing. As usual this is a pretty arduous effort because it
requires going round and fixing lots of odds and ends across the whole
ecosystem. We can deal with a fair number of problems by keeping up with
upstream (see above), but there tends to be a long tail of packages whose
upstreams are less active and where we need to chase them, or where problems
only show up in Debian for one reason or another. I spent a lot of time
working on this:
zope.proxy and zope.security had a bootstrapping loop when adding new
supported Python versions; I broke this
loop
so it shouldn t bother us again in the future.
Akvorado collects sFlow and IPFIX flows over UDP. Because UDP does
not retransmit lost packets, it needs to process them quickly. Akvorado runs
several workers listening to the same port. The kernel should load-balance
received packets fairly between these workers. However, this does not work as
expected. A couple of workers exhibit high packet loss:
Options for load-balancing
There are three methods to load-balance UDP packets across workers:
One worker receives the packets and dispatches them to the other workers.
All workers share the same socket.
Each worker has its own socket, listening to the same port, with the
SO_REUSEPORT socket option.
SO_REUSEPORT option
Tom Hebert added the SO_REUSEPORT socket option in Linux 3.9. The
cover letter for his patch series explains why this new option is better than
the two existing ones from a performance point of view:
SO_REUSEPORT allows multiple listener sockets to be bound to the same port.
[ ] Received packets are distributed to multiple sockets bound to the same
port using a 4-tuple hash.
The motivating case for SO_RESUSEPORT in TCP would be something like a web
server binding to port 80 running with multiple threads, where each thread
might have it s own listener socket. This could be done as an alternative to
other models:
have one listener thread which dispatches completed connections to workers, or
accept on a single listener socket from multiple threads.
In case #1, the listener thread can easily become the bottleneck with high
connection turn-over rate. In case #2, the proportion of connections accepted
per thread tends to be uneven under high connection load. [ ] We have seen the
disproportion to be as high as 3:1 ratio between thread accepting most
connections and the one accepting the fewest. With SO_REUSEPORT the
distribution is uniform.
The motivating case for SO_REUSEPORT in UDP would be something like a DNS
server. An alternative would be to receive on the same socket from multiple
threads. As in the case of TCP, the load across these threads tends to be
disproportionate and we also see a lot of contection on the socket lock.
Akvorado uses the SO_REUSEPORT option to dispatch the packets across the
workers. However, because the distribution uses a 4-tuple hash, a single socket
handles all the flows from one exporter.
SO_ATTACH_REUSEPORT_EBPF option
In Linux 4.5, Craig Gallek added the SO_ATTACH_REUSEPORT_EBPF
option to attach an eBPF program to select the target UDP socket. In Linux 4.6,
he extended it to support TCP. The socket(7) manual page
documents this mechanism:1
The BPF program must return an index between 0 and N-1 representing the socket
which should receive the packet (where N is the number of sockets in the
group). If the BPF program returns an invalid index, socket selection will
fall back to the plain SO_REUSEPORT mechanism.
In Linux 4.19, Martin KaFai Lau added the
BPF_PROG_TYPE_SK_REUSEPORT program type. Such an eBPF program
selects the socket from a BPF_MAP_TYPE_REUSEPORT_ARRAY map instead. This new
approach is more reliable when switching target sockets from one instance to
another for example, when upgrading, a new instance can add its sockets and
remove the old ones.
Load-balancing with eBPF and Go
Altering the load-balancing algorithm for a group of sockets requires two steps:
eBPF program in C
A simple load-balancing algorithm is to randomly choose the destination socket.
The kernel provides the bpf_get_prandom_u32() helper function to get a
pseudo-random number.
In , we declare a volatile constant for the number of sockets in the group. We
will initialize this constant before loading the eBPF program into the kernel.
In , we define the socket map. We will populate it with the socket file
descriptors. In , we randomly select the index of the target socket.3
In , we invoke the bpf_sk_select_reuseport() helper to record our decision.
Finally, in , we accept the packet.
Header files
If you compile the C source with clang, you get errors due to missing headers.
The recommended way to solve this is to generate a vmlinux.h file with
bpftool:
For my 6.17 kernel, the generated vmlinux.h is quite big: 2.7 MiB. Moreover,
bpf/bpf_helpers.h is shipped with libbpf. This adds another dependency for
users. As the eBPF program is quite small, I prefer to put the strict minimum in
vmlinux.h by cherry-picking the definitions I need.
Compilation
The eBPF Library for Go ships bpf2go, a tool to compile eBPF programs and
to generate some scaffolding code. We create a gen.go file with the following
content:
packagemain//go:generate go tool bpf2go -tags linux reuseport reuseport_kern.c
After running go generate ./..., we can inspect the resulting objects with
readelf and llvm-objdump:
$ readelf-Sreuseport_bpfeb.o
There are 14 section headers, starting at offset 0x840: [Nr] Name Type Address Offset[ ] [ 3] sk_reuseport PROGBITS 0000000000000000 00000040 [ 6] .maps PROGBITS 0000000000000000 000000c8 [ 7] license PROGBITS 0000000000000000 000000e8[ ]$ llvm-objdump-Sreuseport_bpfeb.o
reuseport_bpfeb.o: file format elf64-bpfDisassembly of section sk_reuseport:0000000000000000 <reuseport_balance_prog>:; 0: bf 61 00 00 00 00 00 00 r6 = r1; __u32 index = bpf_get_prandom_u32() % num_sockets; 1: 85 00 00 00 00 00 00 07 call 0x7[ ]
Usage from Go
Let s set up 10 workers listening to the same port.5 Each socket
enables the SO_REUSEPORT option before binding:6
The second step is to load the eBPF program, initialize the num_sockets
variable, populate the socket map, and attach the program to the first
socket.7
// Load the eBPF collection.spec,err:=loadReuseport()iferr!=nilt.Fatalf("loadVariables() error:\n%+v",err)// Set "num_sockets" global variable to the number of file descriptors we will registeriferr:=spec.Variables["num_sockets"].Set(uint32(len(fds)));err!=nilt.Fatalf("NumSockets.Set() error:\n%+v",err)// Load the map and the program into the kernel.varobjsreuseportObjectsiferr:=spec.LoadAndAssign(&objs,nil);err!=nilt.Fatalf("loadReuseportObjects() error:\n%+v",err)t.Cleanup(func()objs.Close() )// Assign the file descriptors to the socket map.forworker,fd:=rangefdsiferr:=objs.reuseportMaps.SocketMap.Put(uint32(worker),uint64(fd));err!=nilt.Fatalf("SocketMap.Put() error:\n%+v",err)// Attach the eBPF program to the first socket.socketFD:=int(fds[0])progFD:=objs.reuseportPrograms.ReuseportBalanceProg.FD()iferr:=unix.SetsockoptInt(socketFD,unix.SOL_SOCKET,unix.SO_ATTACH_REUSEPORT_EBPF,progFD);err!=nilt.Fatalf("SetsockoptInt() error:\n%+v",err)
We are now ready to process incoming packets. Each worker is a Go routine
incrementing a counter for each received packet:8
Graceful restart
You can also use SO_ATTACH_REUSEPORT_EBPF to gracefully restart an
application. A new instance of the application binds to the same address and
prepare its own version of the socket map. Once it attaches the eBPF program to
the first socket, the kernel steers incoming packets to this new instance. The
old instance needs to drain the already received packets before shutting down.
To check we are not losing any packet, we spawn a Go routine to send as many
packets as possible:
Then, while the Go routine runs, we start the second set of workers. Once they
are running, they start receiving packets. If we gracefully stop the initial set
of workers, not a single packet is lost!9
Unfortunately, gracefully shutting down a UDP socket is not trivial in Go.10
Previously, we were terminating workers by closing their sockets. However, if we
close them too soon, the application loses packets that were assigned to them
but not yet processed. Before stopping, a worker needs to call conn.Read()
until there are no more packets. A solution is to set a deadline for
conn.Read() and check if we should stop the Go routine when the deadline is
exceeded:
With TCP, this aspect is simpler: after enabling the net.ipv4.tcp_migrate_req
sysctl, the kernel automatically migrates waiting connections to a random socket
in the same group. Alternatively, eBPF can also control this
migration. Both features are available since Linux 5.14.
Addendum
After implementing this strategy in Akvorado, all workers now drop packets!
The root cause is the default limit of 32 records for Kafka batch sizes. This
limit is too low because the brokers have a large overhead when handling each
batch: they need to ensure they persist correctly before acknowledging
them. Increasing the limit to 4096 records fixes this issue.
While load-balancing incoming flows with eBPF remains useful, it did not solve
the main issue. At least the even distribution of dropped packets helped
identify the real bottleneck.
The current version of the manual page is incomplete and does not
cover the evolution introduced in Linux 4.19. There is a pending patch
about this.
Rust is another option. However, the program we use is so trivial that
it does not make sense to use Rust.
As bpf_get_prandom_u32() returns a pseudo-random 32-bit unsigned
value, this method exhibits a very slight bias towards the first indexes.
This is unlikely to be worth fixing.
Some examples include <linux/bpf.h> instead of "vmlinux.h". This
makes your eBPF program dependent on the installed kernel headers.
listenAddr is initially set to 127.0.0.1:0 to allocate a
random port. After the first iteration, it is updated with the allocated
port.
In C, we would poll() both the socket and a pipe used to signal for
shutdown. When the second condition is triggered, we drain the socket by
executing a series of non-blocking read() until we get EWOULDBLOCK.
Given we ve entered a new year it s time for my annual recap of my Free Software activities for the previous calendar year. For previous years see 2019, 2020, 2021, 2022, 2023 + 2024.
Conferences
My first conference of the year was FOSDEM. I d submitted a talk proposal about system attestation in production environments for the attestation devroom, but they had a lot of good submissions and mine was a bit more this is how we do it rather than here s some neat Free Software that does it . I m still trying to work out how to make some of the bits we do more open, but the problem is a lot of the neat stuff is about taking internal knowledge about what should be running and making sure that s the case, and what you end up with if you abstract that is a toolkit that still needs a lot of work to get something useful.
I d more luck at DebConf25 where I gave a talk (Don t fear the TPM) trying to explain how TPMs could be useful in a Debian context. Naturally the comments section descended into a discussion about UEFI Secure Boot, which is a separate, if related, thing. DebConf also featured the usual catch up with fellow team members, hanging out with folk I hadn t seen in ages, and generally feeling a bit more invigorated about Debian.
Other conferences I considered, but couldn t justify, were All Systems Go! and the Linux Plumbers Conference. I ve no doubt both would have had a bunch of interesting and relevant talks + discussions, but not enough this year.
I m going to have to miss FOSDEM this year, due to travel later in the month, and I m uncertain if I m going to make DebConf (for a variety of reasons). That means I don t have a Free Software conference planned for 2026. Ironically FOSSY moving away from Portland makes it a less appealing option (I have Portland friends it would be good to visit). Other than potential Debian MiniConfs, anything else European I should consider?
Debian
I continue to try and keep RetroArch in shape, with 1.22.2+dfsg-1 (and, shortly after, 1.22.2+dfsg-2 - git-buildpackage in trixie seems more strict about Build-Depends existing in the outside environment, and I keep forgetting I need Build-Depends-Arch and Build-Depends-Indep to be pretty much the same with a minimal Build-Depends that just has enough for the clean target) getting uploaded in December, and 1.20.0+dfsg-1, 1.20+dfsg-2 + 1.20+dfsg-3 all being uploaded earlier in the year. retroarch-assets had 1.20.0+dfsg-1 uploaded back in April. I need to find some time to get 1.22.0 packaged. libretro-snes9x got updated to 1.63+dfsg-1.
sdcc saw 4.5.0+dfsg-1, 4.5.0+dfsg-2, 4.5.0+dfsg-3 (I love major GCC upgrades) and 4.5.0-dfsg-4 uploads. There s an outstanding bug around a LaTeX error building the manual, but this turns out to be a bug in the 2.5 RC for LyX. Huge credit to Tobias Quathamer for engaging with this, and Pavel Sanda + J rgen Spitzm ller from the LyX upstream for figuring out the issue + a fix.
Pulseview saw 0.4.2-4 uploaded to fix issues with the GCC 15 + CMake upgrades. I should probably chase the sigrok upstream about new releases; I think there are a bunch of devices that have gained support in git without seeing a tagged release yet.
I did an Electronics Team upload for gputils 1.5.2-2 to fix compilation with GCC 15.
While I don t do a lot with storage devices these days if I can help it I still pay a little bit of attention to sg3-utils. That resulted in 1.48-2 and 1.48-3 uploads in 2025.
libcli got a 1.10.7-3 upload to deal with the libcrypt-dev split out.
Finally I got more up-to-date versions of libtorrent (0.15.7-1) and rtorrent (also 0.15.7-1) uploaded to experimental. There s a ppc64el build failure in libtorrent, but having asked on debian-powerpc this looks like a flaky test/code and I should probably go ahead and upload to unstable.
I sponsored some uploads for Michel Lind - the initial uploads of plymouth-theme-hot-dog, and the separated out pykdumpfile package.
Recognising the fact I wasn t contributing in a useful fashion to the Data Protection Team I set about trying to resign in an orderly fashion - see Andreas call for volunteers that went out in the last week. Shout out to Enrico for pointing out in the past that we should gracefully step down from things we re not actually managing to do, to avoid the perception it s all fine and no one else needs to step up. Took me too long to act on it.
The Debian keyring team continues to operate smoothly, maintaining our monthly release cadence with a 3 month rotation ensuring all team members stay familiar with the process, and ensure their setups are still operational (especially important after Debian releases). I handled the 2025.03.23, 2025.06.24, 2025.06.27, 2025.09.18, 2025.12.08 + 2025.12.26 pushes.
Linux
TPM related fixes were the theme of my kernel contributions in 2025, all within a work context. Somewerejust cleanups, but severalfixedrealissues that were causing us issues. I ve also tried to be more proactive about reviewing diffs in the TPM subsystem; it feels like a useful way to contribute, as well as making me more actively pay attention to what s going on there.
Personal projects
I did some work on onak, my OpenPGP keyserver. That resulted in a 0.6.4 release, mainly driven by fixes for building with more recent CMake + GCC versions in Debian. I ve got a set of changes that should add RFC9580 (v6) support, but there s not a lot of test keys out there at present for making sure I m handling things properly. Equally there s a plan to remove Berkeley DB from Debian, which I m completely down with, but that means I need a new primary backend. I ve got a draft of LMDB support to replace that, but I need to go back and confirm I ve got all the important bits implemented before publishing it and committing to a DB layout. I d also like to add sqlite support as an option, but that needs some thought about trying to take proper advantage of its features, rather than just treating it as a key-value store.
(I know everyone likes to hate on OpenPGP these days, but I continue to be interested by the whole web-of-trust piece of it, which nothing else I m aware of offers.)
That about wraps up 2025. Nothing particularly earth shaking in there, more a case of continuing to tread water on the various things I m involved. I highly doubt 2026 will be much different, but I think that s ok. I scratch my own itches, and if that helps out other folk too then that s lovely, but not the primary goal.
I just released mrcal 2.5; much more about that in a future post. Here, I'd like
to talk about some implementation details.
libpython3 and linking
mrcal is a C library and a Python library. Much of mrcal itself interfaces the C
and Python libraries. And it is common for external libraries to want to pass
Python mrcal.cameramodel objects to their C code. The obvious way to do this
is in a converter function in an O& argument to
PyArg_ParseTupleAndKeywords(). I wrote this mrcal_cameramodel_converter()
function, which opened a whole can of worms when thinking about the compiling
and linking and distribution of this thing.
mrcal_cameramodel_converter() is meant to be called by code that implements
Python-wrapping of C code. This function will be called by the
PyArg_ParseTupleAndKeywords() Python library function, and it uses the Python
C API itself. Since it uses the Python C API, it would normally link against
libpython. However:
The natural place to distribute this is in libmrcal.so, but this library
doesn't touch Python, and I'd rather not pull in all of libpython for this
utility function, even in the 99% case when that function won't even be called
In some cases linking to libpython actually breaks things, so I never do
that anymore anyway. This is fine: since this code will only ever be called by
libpython itself, we're guaranteed that libpython will already be loaded,
and we don't need to ask for it.
OK, let's not link to libpython then. But if we do that, we're going to have
unresolved references to our libpython calls, and the loader will complain
when loading libmrcal.so, even if we're not actually calling those functions.
This has an obvious solution: the references to the libpython calls should be
marked weak. That won't generate unresolved-reference errors, and everything
will be great.
OK, how do we mark things weak? There're two usual methods:
We mark the declaration (or definition?) or the relevant functions with
__attribute__((weak))
We weaken the symbols after the compile with objcopy --weaken.
Method 1 is more work: I don't want to keep track of what Python API calls I'm
actually making. This is non-trivial, because some of the Py_...() invocations
in my code are actually macros that call functions internally that I must
weaken. Furthermore, all the functions are declared in Python.h that I don't
control. I can re-declare stuff with __attribute__((weak)), but then I have to
match the prototypes. And I have to hope that re-declaring these will make
__attribute__((weak)) actually work.
So clearly I want method 2. I implemented it:
Works great on my machine! But doesn't work on other people's machines. Because
only the most recent objcopy tool actually works to weaken references.
Apparently the older tools only weaken definitions, which isn't useful to me,
and the tool only started handling references very recently.
Well that sucks. I guess I will need to mark the symbols with
__attribute__((weak)) after all. I use the nm tool to find the symbols that
should be weakened, and I apply the attribute with this macro:
The prototypes are handled by __typeof__. So are we done? With gcc, we are
done. With clang we are not done. Apparently this macro does not weaken symbols
generated by inline function calls if using clang I have no idea if this is a
bug. The Python internal machinery has some of these, so this doesn't weaken
all the symbols. I give up on the people that both have a too-old objcopy
and are using clang, and declare victory. So the logic ends up being:
Compile
objcopy --weaken
nm to find the non-weak Python references
If there aren't any, our objcopy call worked and we're done!
Otherwise, compile again, but explicitly asking to weaken those symbols
nm again to see if the compiler didn't do it
If any non-weak references still remain, complain and give up.
Whew. This logic appears here and here. There were even more things to deal
with here: calling nm and objcopy needed special attention and build-system
support in case we were cross-building. I took care of it in mrbuild.
This worked for a while. Until the converter code started to fail. Because .
Supporting old Python
. I was using PyTuple_GET_ITEM(). This is a macro to access PyTupleObject
data. So the layout of PyTupleObject ended up encoded in libmrcal.so. But
apparently this wasn't stable, and changed between Python3.13 and Python3.14. As
described above, I'm not linking to libpython, so there's no NEEDED tag to
make sure we pull in the right version. The solution was to call the
PyTuple_GetItem() function instead. This is unsatisfying, and means that in
theory other stuff here might stop working in some Python 3.future, but I'm
ready to move on for now.
There were other annoying gymnastics that had to be performed to make this work
with old-but-not-super old tooling.
The Python people deprecated PyModule_AddObject(), and added PyModule_Add()
as a replacement. I want to support Pythons before and after this happened, so I
needed some if statements. Today the old function still works, but eventually it
will stop, and I will have needed to do this typing sooner or later.
Supporting old C++ compilers
mrcal is a C project, but it is common for people to want to #include the
headers from C++. I widely use C99 designated initializers (27-years old in C!),
which causes issues with not-very-old C++ compilers. I worked around this
initialization in one spot, and disabled it a feature for a too-old compiler in
another spot. Fortunately, semi-recent tooling supports my usages, so this is
becoming a non-issue as time goes on.
Zero-Code
Instrumentation of an Envoy TCP Proxy using eBPF
I recently had to debug an Envoy
Network Load Balancer, and the options Envoy provides just weren't
enough. We were seeing a small number of HTTP 499 errors caused by
latency somewhere in our cloud, but
it wasn't clear what the bottleneck was. As a result, each team had to
set up additional instrumentation to catch latency spikes and figure out
what was going on.
My team is responsible for the LBaaS product (Load Balancer as a
Service) and, of course, we are the first suspects when this kind of
problem appear.
Before going for the current solution, I read a lot of Envoy's
documentation.
It is possible to enable access
logs for Envoy, but they don't provide the information required for
this debug. This is an example of the output:
I won't go into detail about the line above, since it's not possible
to trace the request using access logs alone.
Envoy also has OpenTelemetry
tracing, which is perfect for understanding sources of latency.
Unfortunatly, it is only available for Application Load Balancers.
Most of the HTTP 499 were happening every 10 minutes, so we managed
to get some of the requests with tcpdump, Wireshark and using http headers
to filter the requests.
This approach helped us reproduce and track down the problem, but it
wasn't a great solution. We clearly needed better tools to catch this
kind of issue the next time it happened.
Therefore, I decided to try out OpenTelemetry
eBPF Instrumentation, also referred to as OBI.
I saw the announcement of Grafana Beyla before it was renamed to OBI,
but I didn't have the time or a strong reason to try it out until now.
Even then, I really liked the idea, and the possibility of using eBPF to
solve this instrumentation problem had been in the back of my mind.
OBI promises zero-code automatic instrumentation for Linux services
using eBPF, so I put together a minimal setup to see how well it
works.
This is the simplest Envoy TCP proxy configuration: a listener on
port 8000 forwarding traffic to a backend running on port 8080.
For the backend, I used a basic Go HTTP server:
package mainimport("fmt""net/http")func main() http.Handle("/", http.FileServer(http.Dir("."))) server := http.ServerAddr:":8080" fmt.Println("Starting server on :8080")panic(server.ListenAndServe())
Finally, I wrapped everything together with Docker Compose:
services:autoinstrumenter:image: otel/ebpf-instrument:mainpid:"service:envoy"privileged:trueenvironment:OTEL_EBPF_TRACE_PRINTER: textOTEL_EBPF_OPEN_PORT:8000envoy:image: envoyproxy/envoy:v1.33-latestports:- 8000:8000volumes:- ./envoy.yaml:/etc/envoy/envoy.yaml depends_on:- target-backendtarget-backend:image: golang:1.22-alpinecommand: go run /app/backend.govolumes:- ./backend.go:/app/backend.go:roexpose:-8080
OBI should output traces to the standard output similar to this when
a HTTP request is made to Envoy:
2025-12-08 20:44:49.12884449 (305.572 s[305.572 s]) HTTPClient 200 GET /(/) [172.18.0.3 as envoy:36832]->[172.18.0.2 as localhost:8080] contentLen:78B responseLen:0B svc=[envoy generic] traceparent=[00-529458a2be271956134872668dc5ee47-6dba451ec8935e3e[06c7f817e6a5dae2]-01]
2025-12-08 20:44:49.12884449 (1.260901ms[366.65 s]) HTTP 200 GET /(/) [172.18.0.1 as 172.18.0.1:36282]->[172.18.0.3 as envoy:8000] contentLen:78B responseLen:223B svc=[envoy generic] traceparent=[00-529458a2be271956134872668dc5ee47-06c7f817e6a5dae2[0000000000000000]-01]
This is exactly what we needed, with zero-code. The above trace
shows:
2025-12-08 20:44:49.12884449: time of the trace.
(1.260901ms[366.65 s]): total response time for the
request, with the actual internal execution time of the request (not
counting the request enqueuing time).
HTTP 200 GET /: protocol, response code, HTTP method,
and URL path.
[172.18.0.1 as 172.18.0.1:36282]->[172.18.0.3 as envoy:8000]:
source and destination host:port. The initial request originates from my
machine through the gateway (172.18.0.1), hits the Envoy (172.23.0.3),
the proxy then forwards it to the backend application (172.23.0.2).
contentLen:78B: HTTP Content-Length. I used curl and
the default request size for it is 78B.
responseLen:223B: Size of the response body.
svc=[envoy generic]: traced service.
traceparent: ids to trace the parent request. We can
see that the Envoy makes a request to the target and this request has
the other one as parent.
Let's add one more Envoy to show that it's also possible to track
multiple services.
The new Envoy will listen on port 9000 and forward the request to the
other Envoy listening on port 8000. Now we just need to change OBI open
port variable to look at a range:
OTEL_EBPF_OPEN_PORT: 8000-9000
And change the pid field of the autoinstrumenter service to use the
host's PID namespace inside the container:
pid: host
This is the output I got after one curl:
2025-12-09 12:28:05.12912285 (2.202041ms[1.524713ms]) HTTP 200 GET /(/) [172.19.0.1 as 172.19.0.1:59030]->[172.19.0.5 as envoy:9000] contentLen:78B responseLen:223B svc=[envoy generic] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-856c9f700e73bf0d[0000000000000000]-01]
2025-12-09 12:28:05.12912285 (1.389336ms[1.389336ms]) HTTPClient 200 GET /(/) [172.19.0.5 as envoy:59806]->[172.19.0.4 as localhost:8000] contentLen:78B responseLen:0B svc=[envoy generic] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-caa7f1ad1c68fa77[856c9f700e73bf0d]-01]
2025-12-09 12:28:05.12912285 (1.5431ms[848.574 s]) HTTP 200 GET /(/) [172.19.0.5 as 172.19.0.5:59806]->[172.19.0.4 as envoy:8000] contentLen:78B responseLen:223B svc=[envoy generic] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-cbca9d64d3d26b40[caa7f1ad1c68fa77]-01]
2025-12-09 12:28:05.12912285 (690.217 s[690.217 s]) HTTPClient 200 GET /(/) [172.19.0.4 as envoy:34256]->[172.19.0.3 as localhost:8080] contentLen:78B responseLen:0B svc=[envoy generic] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-5502f7760ed77b5b[cbca9d64d3d26b40]-01]
2025-12-09 12:28:05.12912285 (267.9 s[238.737 s]) HTTP 200 GET /(/) [172.19.0.4 as 172.19.0.4:34256]->[172.19.0.3 as backend:8080] contentLen:0B responseLen:0B svc=[backend go] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-ac05c7ebe26f2530[5502f7760ed77b5b]-01]
Each log line represents a span belonging to the same trace
(69977bee0c2964b8fe53cdd16f8a9d19). For readability, I
ordered the spans by their traceparent relationship, showing the
request's path as it moves through the system: from the client-facing
Envoy, through the internal Envoy hop, and finally to the Go backend.
You can see both server-side (HTTP) and client-side (HTTPClient) spans
at each hop, along with per-span latency, source and destination
addresses, and response sizes, making it easy to pinpoint where time is
spent along the request chain.
The log lines are helpful, but we need better ways to visualize the
traces and the metrics generated by OBI. I'll share another setup that
more closely reflects what we actually use.
Production setup
I'll be using the following tools this time:
The goal of this setup is to mirror an environment similar to what I
used in production. This time, I've omitted the load balancer and
shifted the emphasis to observability instead.
I will run three HTTP servers on port 8080: two inside Incus
containers and one on the host machine. The OBI process will export
metrics and traces to an OpenTelemetry Collector, which will forward
traces to Jaeger and expose a metrics endpoint for Prometheus to scrape.
Grafana will also be added to visualize the collected metrics using
dashboards.
The aim of this approach is to instrument only one of the HTTP
servers while ignoring the others. This simulates an environment with
hundreds of Incus containers, where the objective is to debug a single
container without being overwhelmed by excessive and irrelevant
telemetry data from the rest of the system.
OBI can filter metrics and traces based on attribute values, but I
was not able to filter by process PID. This is where the OBI Collector
comes into play, it allows me to use a processor to filter telemetry
data by the PID of the process being instrumented.
These are the steps to reproduce this setup:
We're almost there, the OpenTelemetry Collector is just missing a
processor. To create the processor filter, we can look at the OBI logs
to find the PID of the HTTP server being instrumented:
Now we just need to add the processor to the collector
configuration:
processors: # <--- NEW BLOCKfilter/host_id:traces:span:-'resource.attributes["service.instance.id"] == "148f400ad3ea:297514"'service:pipelines:traces:receivers:[otlp]processors:[filter/host_id] # <--- NEW LINEexporters:[otlp/jaeger]metrics:receivers:[otlp]processors: # <--- NEW BLOCK- filter/host_idexporters:[prometheus]
That's it! The processor will handle the filtering for us, and we'll
only see traces and metrics from the HTTP server running in the
server01 container. Below are some screenshots from Jaeger
and Grafana:
Closing Notes
I am still amazed at how powerful OBI can be.
For those curious about the debug, we found out that a service
responsible for the network orchestration of the Envoy containers was
running netplan apply every 10 minutes because of a bug.
Netplan apply causes interfaces to go down temporarily and this made the
latency go above 500ms which caused the 499s.
## 0.23 2025-12-20
commit be15aa25dea40aea66a8534143fb81b29d2e6c08
Author: C.J. Collier
Date: Sat Dec 20 22:40:44 2025 +0000
Fixes C-level test infrastructure and adds more test cases for upb_to_sv conversions.
- **Makefile.PL:**
- Allow extra_src in c_test_config.json to be an array.
- Add ASan flags to CCFLAGS and LDDLFLAGS for better debugging.
- Corrected echo newlines in test_c target.
- **c_test_config.json:**
- Added missing type test files to deps and extra_src for convert/sv_to_upb and convert/upb_to_sv test runners.
- **t/c/convert/upb_to_sv.c:**
- Fixed a double free of test_pool .
- Added missing includes for type test headers.
- Updated test plan counts.
- **t/c/convert/sv_to_upb.c:**
- Added missing includes for type test headers.
- Updated test plan counts.
- Corrected Perl interpreter initialization.
- **t/c/convert/types/**:
- Added missing test_util.h include in new type test headers.
- Completed the set of upb_to_sv test cases for all scalar types by adding optional and repeated tests for sfixed32 , sfixed64 , sint32 , and sint64 , and adding repeated tests to the remaining scalar type files.
- **Documentation:**
- Updated 01-xs-testing.md with more debugging tips, including ASan usage and checking for double frees and typos.
- Updated xs_learnings.md with details from the recent segfault.
- Updated llm-plan-execution-instructions.md to emphasize debugging steps.
## 0.22 2025-12-19
commit 2c171d9a5027e0150eae629729c9104e7f6b9d2b
Author: C.J. Collier
Date: Fri Dec 19 23:41:02 2025 +0000
feat(perl,testing): Initialize C test framework and build system
This commit sets up the foundation for the C-level tests and the build system for the Perl Protobuf module:
1. **Makefile.PL Enhancements:**
* Integrates Devel::PPPort to generate ppport.h for better portability.
* Object files now retain their path structure (e.g., xs/convert/sv_to_upb.o ) instead of being flattened, improving build clarity.
* The MY::postamble is significantly revamped to dynamically generate build rules for all C tests located in t/c/ based on the t/c/c_test_config.json file.
* C tests are linked against libprotobuf_common.a and use ExtUtils::Embed flags.
* Added JSON::MaybeXS to PREREQ_PM .
* The test target now also depends on the test_c target.
2. **C Test Infrastructure ( t/c/ ):
* Introduced t/c/c_test_config.json to configure individual C test builds, specifying dependencies and extra source files.
* Created t/c/convert/test_util.c and .h for shared test functions like loading descriptors.
* Initial t/c/convert/upb_to_sv.c and t/c/convert/sv_to_upb.c test runners.
* Basic t/c/integration/030_protobuf_coro.c for Coro safety testing on core utils using libcoro .
* Basic t/c/integration/035_croak_test.c for testing exception handling.
* Basic t/c/integration/050_convert.c for integration testing conversions.
3. **Test Proto:** Updated t/data/test.proto with more field types for conversion testing and regenerated test_descriptor.bin .
4. **XS Test Harness ( t/c/upb-perl-test.h ):** Added like_n macro for length-aware regex matching.
5. **Documentation:** Updated architecture and plan documents to reflect the C test structure.
6. **ERRSV Testing:** Note that the C tests ( t/c/ ) will primarily check *if* a croak occurs (i.e., that the exception path is taken), but will not assert on the string content of ERRSV . Reliably testing $@ content requires the full Perl test environment with Test::More , which will be done in the .t files when testing the Perl API.
This provides a solid base for developing and testing the XS and C components of the module.
## 0.21 2025-12-18
commit a8b6b6100b2cf29c6df1358adddb291537d979bc
Author: C.J. Collier
Date: Thu Dec 18 04:20:47 2025 +0000
test(C): Add integration tests for Milestone 2 components
- Created t/c/integration/030_protobuf.c to test interactions
between obj_cache, arena, and utils.
- Added this test to t/c/c_test_config.json.
- Verified that all C tests for Milestones 2 and 3 pass,
including the libcoro-based stress test.
## 0.20 2025-12-18
commit 0fcad68680b1f700a83972a7c1c48bf3a6958695
Author: C.J. Collier
Date: Thu Dec 18 04:14:04 2025 +0000
docs(plan): Add guideline review reminders to milestones
- Added a "[ ] REFRESH: Review all documents in @perl/doc/guidelines/**"
checklist item to the start of each component implementation
milestone (C and Perl layers).
- This excludes Integration Test milestones.
## 0.19 2025-12-18
commit 987126c4b09fcdf06967a98fa3adb63d7de59a34
Author: C.J. Collier
Date: Thu Dec 18 04:05:53 2025 +0000
docs(plan): Add C-level and Perl-level Coro tests to milestones
- Added checklist items for libcoro -based C tests
(e.g., t/c/integration/050_convert_coro.c ) to all C layer
integration milestones (050 through 220).
- Updated 030_Integration_Protobuf.md to standardise checklist
items for the existing 030_protobuf_coro.c test.
- Removed the single xt/author/coro-safe.t item from
010_Build.md .
- Added checklist items for Perl-level Coro tests
(e.g., xt/coro/240_arena.t ) to each Perl layer
integration milestone (240 through 400).
- Created perl/t/c/c_test_config.json to manage C test
configurations externally.
- Updated perl/doc/architecture/testing/01-xs-testing.md to describe
both C-level libcoro and Perl-level Coro testing strategies.
## 0.18 2025-12-18
commit 6095a5a610401a6035a81429d0ccb9884d53687b
Author: C.J. Collier
Date: Thu Dec 18 02:34:31 2025 +0000
added coro testing to c layer milestones
## 0.17 2025-12-18
commit cc0aae78b1f7f675fc8a1e99aa876c0764ea1cce
Author: C.J. Collier
Date: Thu Dec 18 02:26:59 2025 +0000
docs(plan): Refine test coverage checklist items for SMARTness
- Updated the "Tests provide full coverage" checklist items in
C layer plan files (020, 040, 060, 080, 100, 120, 140, 160, 180, 200)
to explicitly mention testing all public functions in the
corresponding header files.
- Expanded placeholder checklists in 140, 160, 180, 200.
- Updated the "Tests provide full coverage" and "Add coverage checks"
checklist items in Perl layer plan files (230, 250, 270, 290, 310, 330,
350, 370, 390) to be more specific about the scope of testing
and the use of Test::TestCoverage .
- Expanded Well-Known Types milestone (350) to detail each type.
## 0.16 2025-12-18
commit e4b601f14e3817a17b0f4a38698d981dd4cb2818
Author: C.J. Collier
Date: Thu Dec 18 02:07:35 2025 +0000
docs(plan): Full refactoring of C and Perl plan files
- Split both ProtobufPlan-C.md and ProtobufPlan-Perl.md into
per-milestone files under the perl/doc/plan/ directory.
- Introduced Integration Test milestones after each component
milestone in both C and Perl plans.
- Numbered milestone files sequentially (e.g., 010_Build.md,
230_Perl_Arena.md).
- Updated main ProtobufPlan-C.md and ProtobufPlan-Perl.md to
act as Tables of Contents.
- Ensured consistent naming for integration test files
(e.g., t/c/integration/030_protobuf.c , t/integration/260_descriptor_pool.t ).
- Added architecture review steps to the end of all milestones.
- Moved Coro safety test to C layer Milestone 1.
- Updated Makefile.PL to support new test structure and added Coro.
- Moved and split t/c/convert.c into t/c/convert/*.c.
- Moved other t/c/*.c tests into t/c/protobuf/*.c.
- Deleted old t/c/convert.c.
## 0.15 2025-12-17
commit 649cbacf03abb5e7293e3038bb451c0406e9d0ce
Author: C.J. Collier
Date: Wed Dec 17 23:51:22 2025 +0000
docs(plan): Refactor and reset ProtobufPlan.md
- Split the plan into ProtobufPlan-C.md and ProtobufPlan-Perl.md.
- Reorganized milestones to clearly separate C layer and Perl layer development.
- Added more granular checkboxes for each component:
- C Layer: Create test, Test coverage, Implement, Tests pass.
- Perl Layer: Create test, Test coverage, Implement Module/XS, Tests pass, C-Layer adjustments.
- Reset all checkboxes to [ ] to prepare for a full audit.
- Updated status in architecture/api and architecture/core documents to "Not Started".
feat(obj_cache): Add unregister function and enhance tests
- Added protobuf_unregister_object to xs/protobuf/obj_cache.c .
- Updated xs/protobuf/obj_cache.h with the new function declaration.
- Expanded tests in t/c/protobuf_obj_cache.c to cover unregistering,
overwriting keys, and unregistering non-existent keys.
- Corrected the test plan count in t/c/protobuf_obj_cache.c to 17.
## 0.14 2025-12-17
commit 40b6ad14ca32cf16958d490bb575962f88d868a1
Author: C.J. Collier
Date: Wed Dec 17 23:18:27 2025 +0000
feat(arena): Complete C layer for Arena wrapper
This commit finalizes the C-level implementation for the Protobuf::Arena wrapper.
- Adds PerlUpb_Arena_Destroy for proper cleanup from Perl's DEMOLISH.
- Enhances error checking in PerlUpb_Arena_Get .
- Expands C-level tests in t/c/protobuf_arena.c to cover memory allocation
on the arena and lifecycle through PerlUpb_Arena_Destroy .
- Corrects embedded Perl initialization in the C test.
docs(plan): Refactor ProtobufPlan.md
- Restructures the development plan to clearly separate "C Layer" and
"Perl Layer" tasks within each milestone.
- This aligns the plan with the "C-First Implementation Strategy" and improves progress tracking.
## 0.13 2025-12-17
commit c1e566c25f62d0ae9f195a6df43b895682652c71
Author: C.J. Collier
Date: Wed Dec 17 22:00:40 2025 +0000
refactor(perl): Rename C tests and enhance Makefile.PL
- Renamed test files in t/c/ to better match the xs module structure:
- 01-cache.c -> protobuf_obj_cache.c
- 02-arena.c -> protobuf_arena.c
- 03-utils.c -> protobuf_utils.c
- 04-convert.c -> convert.c
- load_test.c -> upb_descriptor_load.c
- Updated perl/Makefile.PL to reflect the new test names in MY::postamble 's $c_test_config .
- Refactored the $c_test_config generation in Makefile.PL to reduce repetition by using a default flags hash and common dependencies array.
- Added a fail() macro to perl/t/c/upb-perl-test.h for consistency.
- Modified t/c/upb_descriptor_load.c to use the t/c/upb-perl-test.h macros, making its output consistent with other C tests.
- Added a skeleton for t/c/convert.c to test the conversion functions.
- Updated documentation in ProtobufPlan.md and architecture/testing/01-xs-testing.md to reflect new test names.
## 0.12 2025-12-17
commit d8cb5dd415c6c129e71cd452f78e29de398a82c9
Author: C.J. Collier
Date: Wed Dec 17 20:47:38 2025 +0000
feat(perl): Refactor XS code into subdirectories
This commit reorganizes the C code in the perl/xs/ directory into subdirectories, mirroring the structure of the Python UPB extension. This enhances modularity and maintainability.
- Created subdirectories for each major component: convert , descriptor , descriptor_containers , descriptor_pool , extension_dict , map , message , protobuf , repeated , and unknown_fields .
- Created skeleton .h and .c files within each subdirectory to house the component-specific logic.
- Updated top-level component headers (e.g., perl/xs/descriptor.h ) to include the new sub-headers.
- Updated top-level component source files (e.g., perl/xs/descriptor.c ) to include their main header and added stub initialization functions (e.g., PerlUpb_InitDescriptor ).
- Moved code from the original perl/xs/protobuf.c to new files in perl/xs/protobuf/ (arena, obj_cache, utils).
- Moved code from the original perl/xs/convert.c to new files in perl/xs/convert/ (upb_to_sv, sv_to_upb).
- Updated perl/Makefile.PL to use a glob ( xs/*/*.c ) to find the new C source files in the subdirectories.
- Added perl/doc/architecture/core/07-xs-file-organization.md to document the new structure.
- Updated perl/doc/ProtobufPlan.md and other architecture documents to reference the new organization.
- Corrected self-referential includes in the newly created .c files.
This restructuring provides a solid foundation for further development and makes it easier to port logic from the Python implementation.
## 0.11 2025-12-17
commit cdedcd13ded4511b0464f5d3bdd72ce6d34e73fc
Author: C.J. Collier
Date: Wed Dec 17 19:57:52 2025 +0000
feat(perl): Implement C-first testing and core XS infrastructure
This commit introduces a significant refactoring of the Perl XS extension, adopting a C-first development approach to ensure a robust foundation.
Key changes include:
- **C-Level Testing Framework:** Established a C-level testing system in t/c/ with a dedicated Makefile, using an embedded Perl interpreter. Initial tests cover the object cache ( 01-cache.c ), arena wrapper ( 02-arena.c ), and utility functions ( 03-utils.c ).
- **Core XS Infrastructure:**
- Implemented a global object cache ( xs/protobuf.c ) to manage Perl wrappers for UPB objects, using weak references.
- Created an upb_Arena wrapper ( xs/protobuf.c ).
- Consolidated common XS helper functions into xs/protobuf.h and xs/protobuf.c .
- **Makefile.PL Enhancements:** Updated to support building and linking C tests, incorporating flags from ExtUtils::Embed , and handling both .c and .cc source files.
- **XS File Reorganization:** Restructured XS files to mirror the Python UPB extension's layout (e.g., message.c , descriptor.c ). Removed older, monolithic .xs files.
- **Typemap Expansion:** Added extensive typemap entries in perl/typemap to handle conversions between Perl objects and various const upb_*Def* pointers.
- **Descriptor Tests:** Added a new test suite t/02-descriptor.t to validate descriptor loading and accessor methods.
- **Documentation:** Updated development plans and guidelines ( ProtobufPlan.md , xs_learnings.md , etc.) to reflect the C-first strategy, new testing methods, and lessons learned.
- **Build Cleanup:** Removed ppport.h from .gitignore as it's no longer used, due to -DPERL_NO_PPPORT being set in Makefile.PL .
This C-first approach allows for more isolated and reliable testing of the core logic interacting with the UPB library before higher-level Perl APIs are built upon it.
## 0.10 2025-12-17
commit 1ef20ade24603573905cb0376670945f1ab5d829
Author: C.J. Collier
Date: Wed Dec 17 07:08:29 2025 +0000
feat(perl): Implement C-level tests and core XS utils
This commit introduces a C-level testing framework for the XS layer and implements key components:
1. **C-Level Tests ( t/c/ )**:
* Added t/c/Makefile to build standalone C tests.
* Created t/c/upb-perl-test.h with macros for TAP-compliant C tests ( plan , ok , is , is_string , diag ).
* Implemented t/c/01-cache.c to test the object cache.
* Implemented t/c/02-arena.c to test Protobuf::Arena wrappers.
* Implemented t/c/03-utils.c to test string utility functions.
* Corrected include paths and diagnostic messages in C tests.
2. **XS Object Cache ( xs/protobuf.c )**:
* Switched to using stringified pointers ( %p ) as hash keys for stability.
* Fixed a critical double-free bug in PerlUpb_ObjCache_Delete by removing an extra SvREFCNT_dec on the lookup key.
3. **XS Arena Wrapper ( xs/protobuf.c )**:
* Corrected PerlUpb_Arena_New to use newSVrv and PTR2IV for opaque object wrapping.
* Corrected PerlUpb_Arena_Get to safely unwrap the arena pointer.
4. **Makefile.PL ( perl/Makefile.PL )**:
* Added -Ixs to INC to allow C tests to find t/c/upb-perl-test.h and xs/protobuf.h .
* Added LIBS to link libprotobuf_common.a into the main Protobuf.so .
* Added C test targets 01-cache , 02-arena , 03-utils to the test config in MY::postamble .
5. **Protobuf.pm ( perl/lib/Protobuf.pm )**:
* Added use XSLoader; to load the compiled XS code.
6. **New files xs/util.h **:
* Added initial type conversion function.
These changes establish a foundation for testing the C-level interface with UPB and fix crucial bugs in the object cache implementation.
## 0.09 2025-12-17
commit 07d61652b032b32790ca2d3848243f9d75ea98f4
Author: C.J. Collier
Date: Wed Dec 17 04:53:34 2025 +0000
feat(perl): Build system and C cache test for Perl XS
This commit introduces the foundational pieces for the Perl XS implementation, focusing on the build system and a C-level test for the object cache.
- **Makefile.PL:**
- Refactored C test compilation rules in MY::postamble to use a hash ( $c_test_config ) for better organization and test-specific flags.
- Integrated ExtUtils::Embed to provide necessary compiler and linker flags for embedding the Perl interpreter, specifically for the t/c/01-cache.c test.
- Correctly constructs the path to the versioned Perl library ( libperl.so.X.Y.Z ) using $Config archlib and $Config libperl to ensure portability.
- Removed VERSION_FROM and ABSTRACT_FROM to avoid dependency on .pm files for now.
- **C Cache Test (t/c/01-cache.c):**
- Added a C test to exercise the object cache functions implemented in xs/protobuf.c .
- Includes tests for adding, getting, deleting, and weak reference behavior.
- **XS Cache Implementation (xs/protobuf.c, xs/protobuf.h):**
- Implemented PerlUpb_ObjCache_Init , PerlUpb_ObjCache_Add , PerlUpb_ObjCache_Get , PerlUpb_ObjCache_Delete , and PerlUpb_ObjCache_Destroy .
- Uses a Perl hash ( HV* ) for the cache.
- Keys are string representations of the C pointers, created using snprintf with "%llx" .
- Values are weak references ( sv_rvweaken ) to the Perl objects ( SV* ).
- PerlUpb_ObjCache_Get now correctly returns an incremented reference to the original SV, not a copy.
- PerlUpb_ObjCache_Destroy now clears the hash before decrementing its refcount.
- **t/c/upb-perl-test.h:**
- Updated is_sv to perform direct pointer comparison ( got == expected ).
- **Minor:** Added util.h (currently empty), updated typemap .
These changes establish a working C-level test environment for the XS components.
## 0.08 2025-12-17
commit d131fd22ea3ed8158acb9b0b1fe6efd856dc380e
Author: C.J. Collier
Date: Wed Dec 17 02:57:48 2025 +0000
feat(perl): Update docs and core XS files
- Explicitly add TDD cycle to ProtobufPlan.md.
- Clarify mirroring of Python implementation in upb-interfacing.md for both C and Perl layers.
- Branch and adapt python/protobuf.h and python/protobuf.c to perl/xs/protobuf.h and perl/xs/protobuf.c, including the object cache implementation. Removed old cache.* files.
- Create initial C test for the object cache in t/c/01-cache.c.
## 0.07 2025-12-17
commit 56fd6862732c423736a2f9a9fb1a2816fc59e9b0
Author: C.J. Collier
Date: Wed Dec 17 01:09:18 2025 +0000
feat(perl): Align Perl UPB architecture docs with Python
Updates the Perl Protobuf architecture documents to more closely align with the design and implementation strategies used in the Python UPB extension.
Key changes:
- **Object Caching:** Mandates a global, per-interpreter cache using weak references for all UPB-derived objects, mirroring Python's PyUpb_ObjCache .
- **Descriptor Containers:** Introduces a new document outlining the plan to use generic XS container types (Sequence, ByNameMap, ByNumberMap) with vtables to handle collections of descriptors, similar to Python's descriptor_containers.c .
- **Testing:** Adds a note to the testing strategy to port relevant test cases from the Python implementation to ensure feature parity.
## 0.06 2025-12-17
commit 6009ce6ab64eccce5c48729128e5adf3ef98e9ae
Author: C.J. Collier
Date: Wed Dec 17 00:28:20 2025 +0000
feat(perl): Implement object caching and fix build
This commit introduces several key improvements to the Perl XS build system and core functionality:
1. **Object Caching:**
* Introduces xs/protobuf.c and xs/protobuf.h to implement a caching mechanism ( protobuf_c_to_perl_obj ) for wrapping UPB C pointers into Perl objects. This uses a hash and weak references to ensure object identity and prevent memory leaks.
* Updates the typemap to use protobuf_c_to_perl_obj for upb_MessageDef * output, ensuring descriptor objects are cached.
* Corrected sv_weaken to the correct sv_rvweaken function.
2. **Makefile.PL Enhancements:**
* Switched to using the Bazel-generated UPB descriptor sources from bazel-bin/src/google/protobuf/_virtual_imports/descriptor_proto/google/protobuf/ .
* Updated INC paths to correctly locate the generated headers.
* Refactored MY::dynamic_lib to ensure the static library libprotobuf_common.a is correctly linked into each generated .so module, resolving undefined symbol errors.
* Overrode MY::test to use prove -b -j$(nproc) t/*.t xt/*.t for running tests.
* Cleaned up LIBS and LDDLFLAGS usage.
3. **Documentation:**
* Updated ProtobufPlan.md to reflect the current status and design decisions.
* Reorganized architecture documents into subdirectories.
* Added object-caching.md and c-perl-interface.md .
* Updated llm-guidance.md with notes on upb/upb.h and sv_rvweaken .
4. **Testing:**
* Fixed xt/03-moo_immutable.t to skip tests if no Moo modules are found.
This resolves the build issues and makes the core test suite pass.
## 0.05 2025-12-16
commit 177d2f3b2608b9d9c415994e076a77d8560423b8
Author: C.J. Collier
Date: Tue Dec 16 19:51:36 2025 +0000
Refactor: Rename namespace to Protobuf, build system and doc updates
This commit refactors the primary namespace from ProtoBuf to Protobuf
to align with the style guide. This involves renaming files, directories,
and updating package names within all Perl and XS files.
**Namespace Changes:**
* Renamed perl/lib/ProtoBuf to perl/lib/Protobuf .
* Moved and updated ProtoBuf.pm to Protobuf.pm .
* Moved and updated ProtoBuf::Descriptor to Protobuf::Descriptor (.pm & .xs).
* Removed other ProtoBuf::* stubs (Arena, DescriptorPool, Message).
* Updated MODULE and PACKAGE in Descriptor.xs .
* Updated NAME , *_FROM in perl/Makefile.PL .
* Replaced ProtoBuf with Protobuf throughout perl/typemap .
* Updated namespaces in test files t/01-load-protobuf-descriptor.t and t/02-descriptor.t .
* Updated namespaces in all documentation files under perl/doc/ .
* Updated paths in perl/.gitignore .
**Build System Enhancements (Makefile.PL):**
* Included xs/*.c files in the common object files list.
* Added -I. to the INC paths.
* Switched from MYEXTLIB to LIBS => ['-L$(CURDIR) -lprotobuf_common'] for linking.
* Removed custom keys passed to WriteMakefile for postamble.
* MY::postamble now sources variables directly from the main script scope.
* Added all :: $ common_lib dependency in MY::postamble .
* Added t/c/load_test.c compilation rule in MY::postamble .
* Updated clean target to include blib .
* Added more modules to TEST_REQUIRES .
* Removed the explicit PM and XS keys from WriteMakefile , relying on XSMULTI => 1 .
**New Files:**
* perl/lib/Protobuf.pm
* perl/lib/Protobuf/Descriptor.pm
* perl/lib/Protobuf/Descriptor.xs
* perl/t/01-load-protobuf-descriptor.t
* perl/t/02-descriptor.t
* perl/t/c/load_test.c : Standalone C test for UPB.
* perl/xs/types.c & perl/xs/types.h : For Perl/C type conversions.
* perl/doc/architecture/upb-interfacing.md
* perl/xt/03-moo_immutable.t : Test for Moo immutability.
**Deletions:**
* Old test files: t/00_load.t , t/01_basic.t , t/02_serialize.t , t/03_message.t , t/04_descriptor_pool.t , t/05_arena.t , t/05_message.t .
* Removed lib/ProtoBuf.xs as it's not needed with XSMULTI .
**Other:**
* Updated test_descriptor.bin (binary change).
* Significant content updates to markdown documentation files in perl/doc/architecture and perl/doc/internal reflecting the new architecture and learnings.
## 0.04 2025-12-14
commit 92de5d482c8deb9af228f4b5ce31715d3664d6ee
Author: C.J. Collier
Date: Sun Dec 14 21:28:19 2025 +0000
feat(perl): Implement Message object creation and fix lifecycles
This commit introduces the basic structure for ProtoBuf::Message object
creation, linking it with ProtoBuf::Descriptor and ProtoBuf::DescriptorPool ,
and crucially resolves a SEGV by fixing object lifecycle management.
Key Changes:
1. ** ProtoBuf::Descriptor :** Added _pool attribute to hold a strong
reference to the parent ProtoBuf::DescriptorPool . This is essential to
prevent the pool and its C upb_DefPool from being garbage collected
while a descriptor is still in use.
2. ** ProtoBuf::DescriptorPool :**
* find_message_by_name : Now passes the $self (the pool object) to the
ProtoBuf::Descriptor constructor to establish the lifecycle link.
* XSUB pb_dp_find_message_by_name : Updated to accept the pool SV* and
store it in the descriptor's _pool attribute.
* XSUB _load_serialized_descriptor_set : Renamed to avoid clashing with the
Perl method name. The Perl wrapper now correctly calls this internal XSUB.
* DEMOLISH : Made safer by checking for attribute existence.
3. ** ProtoBuf::Message :**
* Implemented using Moo with lazy builders for _upb_arena and
_upb_message .
* _descriptor is a required argument to new() .
* XS functions added for creating the arena ( pb_msg_create_arena ) and
the upb_Message ( pb_msg_create_upb_message ).
* pb_msg_create_upb_message now extracts the upb_MessageDef* from the
descriptor and uses upb_MessageDef_MiniTable() to get the minitable
for upb_Message_New() .
* DEMOLISH : Added to free the message's arena.
4. ** Makefile.PL :**
* Added -g to CCFLAGS for debugging symbols.
* Added Perl CORE include path to MY::postamble 's base_flags .
5. **Tests:**
* t/04_descriptor_pool.t : Updated to check the structure of the
returned ProtoBuf::Descriptor .
* t/05_message.t : Now uses a descriptor obtained from a real pool to
test ProtoBuf::Message->new() .
6. **Documentation:**
* Updated ProtobufPlan.md to reflect progress.
* Updated several files in doc/architecture/ to match the current
implementation details, especially regarding arena management and object
lifecycles.
* Added doc/internal/development_cycle.md and doc/internal/xs_learnings.md .
With these changes, the SEGV is resolved, and message objects can be successfully
created from descriptors.
## 0.03 2025-12-14
commit 6537ad23e93680c2385e1b571d84ed8dbe2f68e8
Author: C.J. Collier
Date: Sun Dec 14 20:23:41 2025 +0000
Refactor(perl): Object-Oriented DescriptorPool with Moo
This commit refactors the ProtoBuf::DescriptorPool to be fully object-oriented using Moo, and resolves several issues related to XS, typemaps, and test data.
Key Changes:
1. **Moo Object:** ProtoBuf::DescriptorPool.pm now uses Moo to define the class. The upb_DefPool pointer is stored as a lazy attribute _upb_defpool .
2. **XS Lifecycle:** DescriptorPool.xs now has pb_dp_create_pool called by the Moo builder and pb_dp_free_pool called from DEMOLISH to manage the upb_DefPool lifecycle per object.
3. **Typemap:** The perl/typemap file has been significantly updated to handle the conversion between the ProtoBuf::DescriptorPool Perl object and the upb_DefPool * C pointer. This includes:
* Mapping upb_DefPool * to T_PTR .
* An INPUT section for ProtoBuf::DescriptorPool to extract the pointer from the object's hash, triggering the lazy builder if needed via call_method .
* An OUTPUT section for upb_DefPool * to convert the pointer back to a Perl integer, used by the builder.
4. **Method Renaming:** add_file_descriptor_set_binary is now load_serialized_descriptor_set .
5. **Test Data:**
* Added perl/t/data/test.proto with a sample message and enum.
* Generated perl/t/data/test_descriptor.bin using protoc .
* Removed t/data/ from .gitignore to ensure test data is versioned.
6. **Test Update:** t/04_descriptor_pool.t is updated to use the new OO interface, load the generated descriptor set, and check for message definitions.
7. **Build Fixes:**
* Corrected #include paths in DescriptorPool.xs to be relative to the upb/ directory (e.g., upb/wire/decode.h ).
* Added -I../upb to CCFLAGS in Makefile.PL .
* Reordered INC paths in Makefile.PL to prioritize local headers.
**Note:** While tests now pass in some environments, a SEGV issue persists in make test runs, indicating a potential memory or lifecycle issue within the XS layer that needs further investigation.
## 0.02 2025-12-14
commit 6c9a6f1a5f774dae176beff02219f504ea3a6e07
Author: C.J. Collier
Date: Sun Dec 14 20:13:09 2025 +0000
Fix(perl): Correct UPB build integration and generated file handling
This commit resolves several issues to achieve a successful build of the Perl extension:
1. **Use Bazel Generated Files:** Switched from compiling UPB's stage0 descriptor.upb.c to using the Bazel-generated descriptor.upb.c and descriptor.upb_minitable.c located in bazel-bin/src/google/protobuf/_virtual_imports/descriptor_proto/google/protobuf/ .
2. **Updated Include Paths:** Added the bazel-bin path to INC in WriteMakefile and to base_flags in MY::postamble to ensure the generated headers are found during both XS and static library compilation.
3. **Removed Stage0:** Removed references to UPB_STAGE0_DIR and no longer include headers or source files from upb/reflection/stage0/ .
4. **-fPIC:** Explicitly added -fPIC to CCFLAGS in WriteMakefile and ensured $(CCFLAGS) is used in the custom compilation rules in MY::postamble . This guarantees all object files in the static library are compiled with position-independent code, resolving linker errors when creating the shared objects for the XS modules.
5. **Refined UPB Sources:** Used File::Find to recursively find UPB C sources, excluding /conformance/ and /reflection/stage0/ to avoid conflicts and unnecessary compilations.
6. **Arena Constructor:** Modified ProtoBuf::Arena::pb_arena_new XSUB to accept the class name argument passed from Perl, making it a proper constructor.
7. **.gitignore:** Added patterns to perl/.gitignore to ignore generated C files from XS ( lib/*.c , lib/ProtoBuf/*.c ), the copied src_google_protobuf_descriptor.pb.cc , and the t/data directory.
8. **Build Documentation:** Updated perl/doc/architecture/upb-build-integration.md to reflect the new build process, including the Bazel prerequisite, include paths, -fPIC usage, and File::Find .
Build Steps:
1. bazel build //src/google/protobuf:descriptor_upb_proto (from repo root)
2. cd perl
3. perl Makefile.PL
4. make
5. make test (Currently has expected failures due to missing test data implementation).
## 0.01 2025-12-14
commit 3e237e8a26442558c94075766e0d4456daaeb71d
Author: C.J. Collier
Date: Sun Dec 14 19:34:28 2025 +0000
feat(perl): Initialize Perl extension scaffold and build system
This commit introduces the perl/ directory, laying the groundwork for the Perl Protocol Buffers extension. It includes the essential build files, linters, formatter configurations, and a vendored Devel::PPPort for XS portability.
Key components added:
* ** Makefile.PL **: The core ExtUtils::MakeMaker build script. It's configured to:
* Build a static library ( libprotobuf_common.a ) from UPB, UTF8_Range, and generated protobuf C/C++ sources.
* Utilize XSMULTI => 1 to create separate shared objects for ProtoBuf , ProtoBuf::Arena , and ProtoBuf::DescriptorPool .
* Link each XS module against the common static library.
* Define custom compilation rules in MY::postamble to handle C vs. C++ flags and build the static library.
* Set up include paths for the project root, UPB, and other dependencies.
* **XS Stubs ( .xs files)**:
* lib/ProtoBuf.xs : Placeholder for the main module's XS functions.
* lib/ProtoBuf/Arena.xs : XS interface for upb_Arena management.
* lib/ProtoBuf/DescriptorPool.xs : XS interface for upb_DefPool management.
* **Perl Module Stubs ( .pm files)**:
* lib/ProtoBuf.pm : Main module, loads XS.
* lib/ProtoBuf/Arena.pm : Perl class for Arenas.
* lib/ProtoBuf/DescriptorPool.pm : Perl class for Descriptor Pools.
* lib/ProtoBuf/Message.pm : Base class for messages (TBD).
* **Configuration Files**:
* .gitignore : Ignores build artifacts, editor files, etc.
* .perlcriticrc : Configures Perl::Critic for static analysis.
* .perltidyrc : Configures perltidy for code formatting.
* ** Devel::PPPort **: Vendored version 3.72 to generate ppport.h for XS compatibility across different Perl versions.
* ** typemap **: Custom typemap for XS argument/result conversion.
* **Documentation ( doc/ )**: Initial architecture and plan documents.
This provides a solid foundation for developing the UPB-based Perl extension.
About 80% of my Debian contributions this month were
sponsored by
Freexian, as well as one direct donation via GitHub
Sponsors (thanks!). If you
appreciate this sort of work and are at a company that uses Debian, have a
look to see whether you can pay for any of
Freexian s services; as well as the direct
benefits, that revenue stream helps to keep Debian development sustainable
for me and several other lovely
people.
You can also support my work directly via
Liberapay or GitHub
Sponsors.
Python packaging
New upstream versions:
! I am an intern at Outreachy working with the Debian OpenQA team on images testing. We get to know what career opportunities awaits us when we work on open source projects. In open source, we are constantly learning. The community has different sets of skills and a large network of people.
You might have seen Policy will reject signature within a year warnings in apt(-get) update runs like this:
Somehow this whole DevOps thing is all about generating the wildest things from some (usually equally wild) template.
And today we're gonna generate 
A brand new release 0.1.15 of the 
In the past few years, the size of the kernel images in Debian have been
steadily growing. I don't see this as a problem per se, but it has been causing
me trouble, as my 
Team Rcpp is thrilled to share that an exciting new version 1.1.1 of
Rcpp is now on 
. I am an intern at Outreachy and contributing to the Debian Images Testing project since October 2025. This project is Open Source and everyone can contribute to it in any way. The project uses Open QA to automatically install Operating System Images and test them . We have a community 
Given we ve entered a new year it s time for my annual recap of my Free Software activities for the previous calendar year. For previous years see 
I will run three HTTP servers on port 8080: two inside Incus
containers and one on the host machine. The OBI process will export
metrics and traces to an OpenTelemetry Collector, which will forward
traces to Jaeger and expose a metrics endpoint for Prometheus to scrape.
Grafana will also be added to visualize the collected metrics using
dashboards.
The aim of this approach is to instrument only one of the HTTP
servers while ignoring the others. This simulates an environment with
hundreds of Incus containers, where the objective is to debug a single
container without being overwhelmed by excessive and irrelevant
telemetry data from the rest of the system.
OBI can filter metrics and traces based on attribute values, but I
was not able to filter by process PID. This is where the OBI Collector
comes into play, it allows me to use a processor to filter telemetry
data by the PID of the process being instrumented.
These are the steps to reproduce this setup:
