Debian Lomiri for Debian 13 (previous project)
In our previous project around Debian and Lomiri (lasting until July
2025), we achieved to get Lomiri 0.5.0 (and with it another 130
packages) into Debian (with two minor exceptions [1]) just in time
for the Debian 13 release in August 2025.
Debian Lomiri for Debian 14
At DebConf in Brest, a follow-up project has been designed between the
project sponsor and Fre(i)e Software GmbH [2]. The new project (on paper)
started on 1st August 2025 and project duration was agreed on to be 2
years, allowing our company to work with an equivalent of ~5 FTE on
Lomiri targetting the Debian 14 release some time in the second half of
2027 (an assumed date, let's see what happens).
Ongoing work would be covered from day one of the new project and once
all contract details had been properly put on paper end of September,
Fre(i)e Software GmbH started hiring a new team of software developers
and (future) Debian maintainers. (More of that new team in our next
Q4/2025 report).
The ongoing work of Q3/2025 was basically Guido Berh rster and myself
working on Morph Browser Qt6 (mostly Guido together with Bhushan from
MiraLab [3]) and package maintenance in Debian (mostly me).
Morph Browser Qt6
The first milestone we could reach with the Qt6 porting of Morph Browser [4]
and related components (LUITK aka lomiri-ui-toolkit (big chunk! [5]),
lomiri-content-hub, lomiri-download-manager and a few other components)
was reached on 21st Sep 2025 with an upload of Morph Browser
1.2.0~git20250813.1ca2aa7+dfsg-1~exp1 to Debian experimental and the
Lomiri PPA [6]).
Preparation of Debian 13 Updates (still pending)
In background, various Lomiri updates for Debian 13 have been prepared
during Q3/2025 (with a huge patchset), but publishing those to Debian 13
are still pending as tests are still not satisfying.
[1] lomiri-push-service and nuntium
[2] https://freiesoftware.gmbh
[3] https://miralab.one/
[4] https://gitlab.com/ubports/development/core/morph-browser/-/merge_reques... et al.
[5] https://gitlab.com/ubports/development/core/lomiri-ui-toolkit/-/merge_re... et al.
[6] https://launchpad.net/~lomiri
On Wednesday I had the pleasure of attending a presentation organized
by the Norwegian Unix Users Group on
implementing
RISC-V using a small FPGA. This project is the result of a
university teacher wanting to teach students assembly programming
using a real instruction set, while still providing a simple and
transparent CPU environment. The CPU in question implements the
smallest set of opcodes needed to still call the CPU a RISC-V CPU,
the
RV32I base set. The author and presenter, Kristoffer Robin
Stokke, demonstrated how to build both the FPGA setup and a small
startup code providing a "Hello World" message over both serial port
and a small LCD display. The FPGA is programmed using VHDL,
the entire source
code is available from github, but unfortunately the target FPGA
setup is compiled using the proprietary tool Quartus. It is such
a pity that such a cool little piece of free software should be
chained down by non-free software, so my friend Jon Nordby set out to
see if we can liberate this small RISC-V CPU. After all, it would be
unforgivable sin to force students to use non-free software to study
at the University of Oslo.
The VHDL code for the CPU instructions itself is only 1138 lines,
if I am to believe wc -l lib/riscv_common/* lib/rv32i/*. On
the small FPGA used during the talk, the entire CPU, ROM, display and
serial port driver only used up half the capacity. These days, there
exists a free software toolchain for FPGA programming not only in
Verilog but also in VHDL, and we hope the support in
yosys,
ghdl, and
yosys-plugin-ghdl
(sadly and strangely enough, removed from Debian unstable) is complete
enough to at least build this small and simple project with some minor
portability fixes. Or perhaps there are other approaches that work
better? The first patches are already floating on github, to make the
VHDL code more portable and to test out the build. If you are
interested in running your own little RISC-V CPU on a FPGA chip,
please get in touch.
At the moment we sadly have hit a GHDL bug, which we do not quite
know how to work around or fix:
Thus more work is needed. For me, this simple project is the first
stepping stone for a larger dream I have of converting the MESA
machine controller system to build its firmware using a free
software toolchain. I just need to learn more FPGA programming
first. :)
As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
The FAI.me service has reached another milestone:
The 42.000th job was submitted via the web interface
since the beginning of this service in 2017.
The idea was to provide a simple web interface for end users for
creating the configs for the fully automatic installation with only
minimal questions and without knowing the syntax of the configuration files.
Thanks a lot for using this service and for all your feedback.
The next job can be yours!
P.S.: I like to get more feedback for the FAI.me service. What do you
like most? What's missing? Do you have any success story how you use the
customized ISO for your deployment? Please fill out the
FAI questionaire
or sent feedback via email to fai.me@fai-project.org
About FAI.me
FAI.me is the service for building your own customized images via a web
interface. You can create an installation or live ISO or a cloud
image.
For Debian, multiple release versions can be chosen, as well as
installations for Ubuntu Server, Ubuntu Desktop, or Linux Mint.
Multiple options are available like selecting different desktop
environments, the language and keyboard and adding a user with a
password.
Optional settings include adding your own package list,
choosing a backports kernel, adding a postinst script
and adding a ssh public key, choosing a partition
layout and some more.
Part
3: Building the Keystone Dataproc Custom Images for Secure Boot &
GPUs
In Part 1, we established a secure, proxy-only network. In Part 2, we
explored the enhanced install_gpu_driver.sh initialization
action. Now, in Part 3, we ll focus on using the LLC-Technologies-Collier/custom-images
repository (branch proxy-exercise-2025-11) to build the
actual custom Dataproc images embedded with NVIDIA drivers signed for
Secure Boot, all within our proxied environment.
Why Custom Images?
To run NVIDIA GPUs on Shielded VMs with Secure Boot enabled, the
NVIDIA kernel modules must be signed with a key trusted by the VM s EFI
firmware. Since standard Dataproc images don t include these
custom-signed modules, we need to build our own. This process also
allows us to pre-install a full stack of GPU-accelerated software.
The custom-images Toolkit
(examples/secure-boot)
The examples/secure-boot directory within the custom-images repository contains the necessary scripts and
configurations, refined through significant development to handle proxy
and Secure Boot challenges.
Key Components & Development Insights:
env.json: The central configuration
file (as used in Part 1) for project, network, proxy, and bucket
details. This became the single source of truth to avoid configuration
drift.
create-key-pair.sh: Manages the Secure
Boot signing keys (PK, KEK, DB) in Google Secret Manager, essential for
the module signing.
build-and-run-podman.sh: Orchestrates
the image build process in an isolated Podman container. This was
introduced to standardize the build environment and encapsulate
dependencies, simplifying what the user needs to install locally.
pre-init.sh: Sets up the build
environment within the container and calls generate_custom_image.py. It crucially passes metadata
derived from env.json (like proxy settings and Secure Boot
key secret names) to the temporary build VM.
generate_custom_image.py: The core
Python script that automates GCE VM creation, runs the customization
script, and creates the final GCE image.
gce-proxy-setup.sh: This script from startup_script/ is vital. It s injected into the temporary build VM and runs first to configure the OS, package
managers (apt, dnf), tools (curl, wget, GPG), Conda, and Java to use the
proxy settings passed in the metadata. This ensures the entire build
process is proxy-aware.
install_gpu_driver.sh: Used as the --customization-script within the build VM. As detailed in
Part 2, this script handles the driver/CUDA/ML stack installation and
signing, now able to function correctly due to the proxy setup by gce-proxy-setup.sh.
Layered Image Strategy:
The pre-init.sh script employs a layered approach:
secure-boot Image: Base image with
Secure Boot certificates injected.
tf Image: Based on secure-boot, this image runs the full install_gpu_driver.sh within the proxy-configured build VM
to install NVIDIA drivers, CUDA, ML libraries (TensorFlow, PyTorch,
RAPIDS), and sign the modules. This is the primary target image for our
use case.
(Note: secure-proxy and proxy-tf layers
were experiments, but the -tf image combined with runtime
metadata emerged as the most effective solution for 2.2-debian12).
Build Steps:
Clone Repos & Configure env.json: Ensure you have the custom-images and cloud-dataproc repos and a
complete env.json as described in Part 1.
Run the Build: bash # Example: Build a 2.2-debian12 based image set # Run from the custom-images repository root bash examples/secure-boot/build-and-run-podman.sh 2.2-debian12
This command will build the layered images, leveraging the proxy
settings from env.json via the metadata injected into the
build VM. Note the final image name produced (e.g., dataproc-2-2-deb12-YYYYMMDD-HHMMSS-tf).
Conclusion of Part 3
Through an iterative process, we ve developed a robust workflow
within the custom-images repository to build Secure
Boot-compatible GPU images in a proxy-only environment. The key was
isolating the build in Podman, ensuring the build VM is fully
proxy-aware using gce-proxy-setup.sh, and leveraging the
enhanced install_gpu_driver.sh from Part 2.
In Part 4, we ll bring it all together, deploying a Dataproc cluster
using this custom -tf image within the secure network, and
verifying the end-to-end functionality.
Part
1: Building a Secure Network Foundation for Dataproc with GPUs &
SWP
Welcome to the first post in our series on running GPU-accelerated
Dataproc workloads in secure, enterprise-grade environments. Many
organizations need to operate within VPCs that have no direct internet
egress, instead routing all traffic through a Secure Web Proxy (SWP).
Additionally, security mandates often require the use of Shielded VMs
with Secure Boot enabled. This series will show you how to meet these
requirements for your Dataproc GPU clusters.
In this post, we ll focus on laying the network foundation using
tools from the LLC-Technologies-Collier/cloud-dataproc
repository (branch proxy-sync-2026-01).
The Challenge: Network
Isolation & Control
Before we can even think about custom images or GPU drivers, we need
a network environment that:
Prevents direct internet access from Dataproc cluster nodes.
Forces all egress traffic through a manageable and auditable
SWP.
Provides the necessary connectivity for Dataproc to function and for
us to build images later.
Supports Secure Boot for all VMs.
The Toolkit: LLC-Technologies-Collier/cloud-dataproc
To make setting up and tearing down these complex network
environments repeatable and consistent, we ve developed a set of bash
scripts within the gcloud directory of the cloud-dataproc repository. These scripts handle the
creation of VPCs, subnets, firewall rules, service accounts, and the
Secure Web Proxy itself.
Key Script: gcloud/bin/create-dpgce-private
This script is the cornerstone for creating the private, proxied
environment. It automates:
VPC and Subnet creation (for the cluster, SWP, and management).
Setup of Certificate Authority Service and Certificate Manager for
SWP TLS interception.
Deployment of the SWP Gateway instance.
Configuration of a Gateway Security Policy to control egress.
Creation of necessary firewall rules.
Result: Cluster nodes in this VPC have NO default
internet route and MUST use the SWP.
Configuration via env.json
We use a single env.json file to drive the
configuration. This file will also be used by the custom-images scripts in Part 3. This env.json
should reside in your custom-images repository clone, and
you ll symlink it into the cloud-dataproc/gcloud
directory.
Running the Setup:
# Assuming you have cloud-dataproc and custom-images cloned side-by-side# And your env.json is in the custom-images rootcd cloud-dataproc/gcloud# Symlink to the env.json in custom-imagesln-sf ../../custom-images/env.json env.json# Run the creation script, but don't create a cluster yetbash bin/create-dpgce-private --no-create-clustercd ../../custom-images
Node
Configuration: The Metadata Startup Script for Runtime
For the Dataproc cluster nodes to function correctly in this proxied
environment, they need to be configured to use the SWP on boot. We
achieve this using a GCE metadata startup script.
The script startup_script/gce-proxy-setup.sh (from the custom-images repository) is designed to be run on each
cluster node at boot. It reads metadata like http-proxy and http-proxy-pem-uri (which our cluster creation scripts in
Part 4 will pass) to configure the OS environment, package managers, and
other tools to use the SWP.
Upload this script to your GCS bucket:
# Run from the custom-images repository rootgsutil cp startup_script/gce-proxy-setup.sh gs://$(jq-r .BUCKET env.json)/custom-image-deps/
This script is essential for the runtime behavior of the
cluster nodes.
Conclusion of Part 1
With the cloud-dataproc scripts, we ve laid the
groundwork by provisioning a secure VPC with controlled egress through
an SWP. We ve also prepared the essential node-level proxy configuration
script (gce-proxy-setup.sh) in GCS, ready to be used by our
clusters.
Stay tuned for Part 2, where we ll dive into the install_gpu_driver.sh initialization action from the LLC-Technologies-Collier/initialization-actions repository
(branch gpu-202601) and how it s been adapted to install
all GPU-related software through the proxy during the image build
process.
Hurray! I have officially reached the 6-week mark, the halfway point of my Outreachy internship. The time has flown by incredibly fast, yet it feels short because there is still so much exciting work to do.
I remember starting this journey feeling overwhelmed, trying to gain momentum. Today, I feel much more confident. I began with the apps_startstop task during the contribution period, writing manual test steps and creating preparation Perl scripts for the desktop environments. Since then, I ve transitioned into full automation and taken a liking to reading openQA upstream documentation when I have issues or for reference.
In all of this, I ve committed over 30 hours a week to the project. This dedicated time has allowed me to look in-depth into the Debian ecosystem and automated quality assurance.
The Original Roadmap vs. Reality
Reviewing my 12-week goal, which included extending automated tests for live image testing, installer testing, and documentation, I am happy to report that I am right on track. My work on desktop apps tests has directly improved the quality of both the Live Images and the netinst (network installer) ISOs.
Accomplishments
I have successfully extended the apps_startstop tests for two Desktop Environments (DEs): Cinnamon and LXQt. These tests ensure that common and DE specific apps launch and close correctly across different environments.
Merged Milestone: My Cinnamon tests have been officially merged into the upstream repository! [MR !84]
LXQt & Adaptability: I am in the final stages of the LXQt tests. Interestingly, I had to update these tests mid-way through because of a version update in the DE. This required me to update the needles (image references) to match the new UI, a great lesson in software maintenance.
Solving for Synergy
One of my favorite challenges was suggested by my mentor, Roland: synergizing the tests to reduce redundancy. I observed that some applications (like Firefox and LibreOffice) behave identically across different desktops. Instead of duplicating Perl scripts/code for every single DE, I used symbolic links. This allows the use of the same Perl script and possibly the same needles, making the test suite lighter and much easier to maintain.
The Contributor Guide
During the contribution phase, I noticed how rigid the documentation and coding style requirements are. While this ensures high standards and uniformity, it can be intimidating for newcomers and time-consuming for reviewers.
To help, I created a contributor guide [MR !97]. This guide addresses the project s writing style. My goal is to reduce the back-and-forth during reviews, making the process more efficient for everyone and helping new contributors.
Looking Forward
For the second half of the internship, I plan to:
Assist others: Help new contributors extend apps start-stop tests to even more desktop environments.
Explore new coverage: Move beyond start-stop tests into deeper functional testing.
This journey has been an amazing experience of learning and connecting with the wider open-source community, especially Debian Women and the Linux QA team.
I am deeply grateful to my mentors, Tassia Camoes Araujo, Roland Clobus, and Philip Hands, for their constant guidance and for believing in my ability to take on this project.
David Patterson does not need an introduction. Being the brain behind many
of the inventions that shaped the computing industry (repeatedly) over the
past 40 years, when he put forward an opinion article in Communications of
the ACM targeting the current day political waves in the USA, I could not
avoid choosing it to write this review.
Patterson worked for a a public university (University of California at
Berkeley) between 1976 and 2016, and in this article he argues how
government-funded academic research (GoFAR) allows for faster, more
effective and freer development than private sector-funded research would,
putting his own career milestones as an example of how public money that
went to his research has easily been amplified by a factor of 10,000:1 for
the country s economy, and 1,000:1 particularly for the government.
Patterson illustrates this quoting five of the home-run research projects
he started and pursued with government funding, eventually spinning them
off as successful startups:
RISC (Reduced Instruction Set Computing): Microprocessor
architecture that reduces the complexity and power consumption of CPUs,
yielding much smaller and more efficient processors.
RAID (Redundant Array of Inexpensive Disks): Patterson experimented
with a way to present a series of independent hard drive units as if they
were a single, larger one, leading to increases in capacity and
reliability beyond what the industry could provide in single drives, for
a fraction of the price.
NOW (Network Of Workstations): Introduced what we now know as
computer clusters (in contrast of large-scale massively multiprocessed
cache-coherent systems known as supercomputers ), which nowadays power
over 80% of the Top500 supercomputer list and are the computer platform
of choice of practically all data centers.
RAD Lab (Reliable Adaptive Distributed Systems Lab): Pursued the
technology for data centers to be self-healing and self-managing, testing
and pushing early cloud-scalability limits
ParLab (Parallel Computing Lab): Given the development of massively
parallel processing inside even simple microprocessors, this lab explores
how to improve designs of parallel software and hardware, presenting the
ground works that proved that inherently parallel GPUs were better than
CPUs at machine learning tasks. It also developed the RISC-V open
instruction set architecture.
Patterson identifies principles for the projects he has led, that are
specially compatible with the ways research works at universitary systems:
Multidisciplinary teams, demonstrative usable artifacts, seven- to ten-year
impact horizons, five-year sunset clauses (to create urgency and to lower
opportunity costs), physical proximity of collaborators, and leadership
followed on team success rather than individual recognition.
While it could be argued that it s easy to point at Patterson s work as a
success example while he is by far not the average academic, the points he
makes on how GoFAR research has been fundamental for the advance of
science and technology, but also of biology, medicine, and several other
fields are very clear.
AI and Secure Messaging Don't Mix
Over on the ACLU's Free Future blog,
I just published an article titled AI and Secure Messaging Don't Mix.
The blogpost assumes for the sake of the argument that people might actually want to have an AI involved in their personal conversations,
and explores why Meta's Private Processing
doesn't offer the level of assurance that they want it to offer.
In short, the promises of "confidential cloud computing" are built on shaky foundations, especially against adversaries as powerful as Meta themselves.
If you really want AI in your chat, the baseline step for privacy preservation is to include it in your local compute base, not to use a network service!
But these operators clearly don't value private communication as much as they value binding you to their services.
But let's imagine some secure messenger that actually does put message confidentiality first --
and imagine they had integrated some sort of AI capability into the messenger.
That at least bypasses the privacy questions around AI use.
Would you really want to talk with your friends, as augmented by their local AI, though?
Would you want an AI, even one running locally with perfect privacy, intervening in your social connections?
What if it summarized your friend's messages to you in a way that led you to misunderstand (or ignore) an important point your friend had made?
What if it encouraged you to make an edgy joke that comes across wrong?
Or to say something that seriously upsets a friend?
How would you respond?
How would you even know that it had happened?
My handle is dkg.
More times than i can count, i've had someone address me in a chat as "dog" and then cringe and apologize and blame their spellchecker/autocorrect.
I can laugh these off because the failure mode is so obvious and transparent -- and repeatable.
(also, dogs are awesome, so i don't really mind!)
But when our attention (and our responses!) are being shaped and molded by these plausibility engines, how will we even know that mistakes are being made?
What if the plausibility engine you've hooked into your messenger embeds subtle (or unsubtle!) bias?
Don't we owe it to each other to engage with actual human attention?
## 0.23 2025-12-20
commit be15aa25dea40aea66a8534143fb81b29d2e6c08
Author: C.J. Collier
Date: Sat Dec 20 22:40:44 2025 +0000
Fixes C-level test infrastructure and adds more test cases for upb_to_sv conversions.
- **Makefile.PL:**
- Allow extra_src in c_test_config.json to be an array.
- Add ASan flags to CCFLAGS and LDDLFLAGS for better debugging.
- Corrected echo newlines in test_c target.
- **c_test_config.json:**
- Added missing type test files to deps and extra_src for convert/sv_to_upb and convert/upb_to_sv test runners.
- **t/c/convert/upb_to_sv.c:**
- Fixed a double free of test_pool .
- Added missing includes for type test headers.
- Updated test plan counts.
- **t/c/convert/sv_to_upb.c:**
- Added missing includes for type test headers.
- Updated test plan counts.
- Corrected Perl interpreter initialization.
- **t/c/convert/types/**:
- Added missing test_util.h include in new type test headers.
- Completed the set of upb_to_sv test cases for all scalar types by adding optional and repeated tests for sfixed32 , sfixed64 , sint32 , and sint64 , and adding repeated tests to the remaining scalar type files.
- **Documentation:**
- Updated 01-xs-testing.md with more debugging tips, including ASan usage and checking for double frees and typos.
- Updated xs_learnings.md with details from the recent segfault.
- Updated llm-plan-execution-instructions.md to emphasize debugging steps.
## 0.22 2025-12-19
commit 2c171d9a5027e0150eae629729c9104e7f6b9d2b
Author: C.J. Collier
Date: Fri Dec 19 23:41:02 2025 +0000
feat(perl,testing): Initialize C test framework and build system
This commit sets up the foundation for the C-level tests and the build system for the Perl Protobuf module:
1. **Makefile.PL Enhancements:**
* Integrates Devel::PPPort to generate ppport.h for better portability.
* Object files now retain their path structure (e.g., xs/convert/sv_to_upb.o ) instead of being flattened, improving build clarity.
* The MY::postamble is significantly revamped to dynamically generate build rules for all C tests located in t/c/ based on the t/c/c_test_config.json file.
* C tests are linked against libprotobuf_common.a and use ExtUtils::Embed flags.
* Added JSON::MaybeXS to PREREQ_PM .
* The test target now also depends on the test_c target.
2. **C Test Infrastructure ( t/c/ ):
* Introduced t/c/c_test_config.json to configure individual C test builds, specifying dependencies and extra source files.
* Created t/c/convert/test_util.c and .h for shared test functions like loading descriptors.
* Initial t/c/convert/upb_to_sv.c and t/c/convert/sv_to_upb.c test runners.
* Basic t/c/integration/030_protobuf_coro.c for Coro safety testing on core utils using libcoro .
* Basic t/c/integration/035_croak_test.c for testing exception handling.
* Basic t/c/integration/050_convert.c for integration testing conversions.
3. **Test Proto:** Updated t/data/test.proto with more field types for conversion testing and regenerated test_descriptor.bin .
4. **XS Test Harness ( t/c/upb-perl-test.h ):** Added like_n macro for length-aware regex matching.
5. **Documentation:** Updated architecture and plan documents to reflect the C test structure.
6. **ERRSV Testing:** Note that the C tests ( t/c/ ) will primarily check *if* a croak occurs (i.e., that the exception path is taken), but will not assert on the string content of ERRSV . Reliably testing $@ content requires the full Perl test environment with Test::More , which will be done in the .t files when testing the Perl API.
This provides a solid base for developing and testing the XS and C components of the module.
## 0.21 2025-12-18
commit a8b6b6100b2cf29c6df1358adddb291537d979bc
Author: C.J. Collier
Date: Thu Dec 18 04:20:47 2025 +0000
test(C): Add integration tests for Milestone 2 components
- Created t/c/integration/030_protobuf.c to test interactions
between obj_cache, arena, and utils.
- Added this test to t/c/c_test_config.json.
- Verified that all C tests for Milestones 2 and 3 pass,
including the libcoro-based stress test.
## 0.20 2025-12-18
commit 0fcad68680b1f700a83972a7c1c48bf3a6958695
Author: C.J. Collier
Date: Thu Dec 18 04:14:04 2025 +0000
docs(plan): Add guideline review reminders to milestones
- Added a "[ ] REFRESH: Review all documents in @perl/doc/guidelines/**"
checklist item to the start of each component implementation
milestone (C and Perl layers).
- This excludes Integration Test milestones.
## 0.19 2025-12-18
commit 987126c4b09fcdf06967a98fa3adb63d7de59a34
Author: C.J. Collier
Date: Thu Dec 18 04:05:53 2025 +0000
docs(plan): Add C-level and Perl-level Coro tests to milestones
- Added checklist items for libcoro -based C tests
(e.g., t/c/integration/050_convert_coro.c ) to all C layer
integration milestones (050 through 220).
- Updated 030_Integration_Protobuf.md to standardise checklist
items for the existing 030_protobuf_coro.c test.
- Removed the single xt/author/coro-safe.t item from
010_Build.md .
- Added checklist items for Perl-level Coro tests
(e.g., xt/coro/240_arena.t ) to each Perl layer
integration milestone (240 through 400).
- Created perl/t/c/c_test_config.json to manage C test
configurations externally.
- Updated perl/doc/architecture/testing/01-xs-testing.md to describe
both C-level libcoro and Perl-level Coro testing strategies.
## 0.18 2025-12-18
commit 6095a5a610401a6035a81429d0ccb9884d53687b
Author: C.J. Collier
Date: Thu Dec 18 02:34:31 2025 +0000
added coro testing to c layer milestones
## 0.17 2025-12-18
commit cc0aae78b1f7f675fc8a1e99aa876c0764ea1cce
Author: C.J. Collier
Date: Thu Dec 18 02:26:59 2025 +0000
docs(plan): Refine test coverage checklist items for SMARTness
- Updated the "Tests provide full coverage" checklist items in
C layer plan files (020, 040, 060, 080, 100, 120, 140, 160, 180, 200)
to explicitly mention testing all public functions in the
corresponding header files.
- Expanded placeholder checklists in 140, 160, 180, 200.
- Updated the "Tests provide full coverage" and "Add coverage checks"
checklist items in Perl layer plan files (230, 250, 270, 290, 310, 330,
350, 370, 390) to be more specific about the scope of testing
and the use of Test::TestCoverage .
- Expanded Well-Known Types milestone (350) to detail each type.
## 0.16 2025-12-18
commit e4b601f14e3817a17b0f4a38698d981dd4cb2818
Author: C.J. Collier
Date: Thu Dec 18 02:07:35 2025 +0000
docs(plan): Full refactoring of C and Perl plan files
- Split both ProtobufPlan-C.md and ProtobufPlan-Perl.md into
per-milestone files under the perl/doc/plan/ directory.
- Introduced Integration Test milestones after each component
milestone in both C and Perl plans.
- Numbered milestone files sequentially (e.g., 010_Build.md,
230_Perl_Arena.md).
- Updated main ProtobufPlan-C.md and ProtobufPlan-Perl.md to
act as Tables of Contents.
- Ensured consistent naming for integration test files
(e.g., t/c/integration/030_protobuf.c , t/integration/260_descriptor_pool.t ).
- Added architecture review steps to the end of all milestones.
- Moved Coro safety test to C layer Milestone 1.
- Updated Makefile.PL to support new test structure and added Coro.
- Moved and split t/c/convert.c into t/c/convert/*.c.
- Moved other t/c/*.c tests into t/c/protobuf/*.c.
- Deleted old t/c/convert.c.
## 0.15 2025-12-17
commit 649cbacf03abb5e7293e3038bb451c0406e9d0ce
Author: C.J. Collier
Date: Wed Dec 17 23:51:22 2025 +0000
docs(plan): Refactor and reset ProtobufPlan.md
- Split the plan into ProtobufPlan-C.md and ProtobufPlan-Perl.md.
- Reorganized milestones to clearly separate C layer and Perl layer development.
- Added more granular checkboxes for each component:
- C Layer: Create test, Test coverage, Implement, Tests pass.
- Perl Layer: Create test, Test coverage, Implement Module/XS, Tests pass, C-Layer adjustments.
- Reset all checkboxes to [ ] to prepare for a full audit.
- Updated status in architecture/api and architecture/core documents to "Not Started".
feat(obj_cache): Add unregister function and enhance tests
- Added protobuf_unregister_object to xs/protobuf/obj_cache.c .
- Updated xs/protobuf/obj_cache.h with the new function declaration.
- Expanded tests in t/c/protobuf_obj_cache.c to cover unregistering,
overwriting keys, and unregistering non-existent keys.
- Corrected the test plan count in t/c/protobuf_obj_cache.c to 17.
## 0.14 2025-12-17
commit 40b6ad14ca32cf16958d490bb575962f88d868a1
Author: C.J. Collier
Date: Wed Dec 17 23:18:27 2025 +0000
feat(arena): Complete C layer for Arena wrapper
This commit finalizes the C-level implementation for the Protobuf::Arena wrapper.
- Adds PerlUpb_Arena_Destroy for proper cleanup from Perl's DEMOLISH.
- Enhances error checking in PerlUpb_Arena_Get .
- Expands C-level tests in t/c/protobuf_arena.c to cover memory allocation
on the arena and lifecycle through PerlUpb_Arena_Destroy .
- Corrects embedded Perl initialization in the C test.
docs(plan): Refactor ProtobufPlan.md
- Restructures the development plan to clearly separate "C Layer" and
"Perl Layer" tasks within each milestone.
- This aligns the plan with the "C-First Implementation Strategy" and improves progress tracking.
## 0.13 2025-12-17
commit c1e566c25f62d0ae9f195a6df43b895682652c71
Author: C.J. Collier
Date: Wed Dec 17 22:00:40 2025 +0000
refactor(perl): Rename C tests and enhance Makefile.PL
- Renamed test files in t/c/ to better match the xs module structure:
- 01-cache.c -> protobuf_obj_cache.c
- 02-arena.c -> protobuf_arena.c
- 03-utils.c -> protobuf_utils.c
- 04-convert.c -> convert.c
- load_test.c -> upb_descriptor_load.c
- Updated perl/Makefile.PL to reflect the new test names in MY::postamble 's $c_test_config .
- Refactored the $c_test_config generation in Makefile.PL to reduce repetition by using a default flags hash and common dependencies array.
- Added a fail() macro to perl/t/c/upb-perl-test.h for consistency.
- Modified t/c/upb_descriptor_load.c to use the t/c/upb-perl-test.h macros, making its output consistent with other C tests.
- Added a skeleton for t/c/convert.c to test the conversion functions.
- Updated documentation in ProtobufPlan.md and architecture/testing/01-xs-testing.md to reflect new test names.
## 0.12 2025-12-17
commit d8cb5dd415c6c129e71cd452f78e29de398a82c9
Author: C.J. Collier
Date: Wed Dec 17 20:47:38 2025 +0000
feat(perl): Refactor XS code into subdirectories
This commit reorganizes the C code in the perl/xs/ directory into subdirectories, mirroring the structure of the Python UPB extension. This enhances modularity and maintainability.
- Created subdirectories for each major component: convert , descriptor , descriptor_containers , descriptor_pool , extension_dict , map , message , protobuf , repeated , and unknown_fields .
- Created skeleton .h and .c files within each subdirectory to house the component-specific logic.
- Updated top-level component headers (e.g., perl/xs/descriptor.h ) to include the new sub-headers.
- Updated top-level component source files (e.g., perl/xs/descriptor.c ) to include their main header and added stub initialization functions (e.g., PerlUpb_InitDescriptor ).
- Moved code from the original perl/xs/protobuf.c to new files in perl/xs/protobuf/ (arena, obj_cache, utils).
- Moved code from the original perl/xs/convert.c to new files in perl/xs/convert/ (upb_to_sv, sv_to_upb).
- Updated perl/Makefile.PL to use a glob ( xs/*/*.c ) to find the new C source files in the subdirectories.
- Added perl/doc/architecture/core/07-xs-file-organization.md to document the new structure.
- Updated perl/doc/ProtobufPlan.md and other architecture documents to reference the new organization.
- Corrected self-referential includes in the newly created .c files.
This restructuring provides a solid foundation for further development and makes it easier to port logic from the Python implementation.
## 0.11 2025-12-17
commit cdedcd13ded4511b0464f5d3bdd72ce6d34e73fc
Author: C.J. Collier
Date: Wed Dec 17 19:57:52 2025 +0000
feat(perl): Implement C-first testing and core XS infrastructure
This commit introduces a significant refactoring of the Perl XS extension, adopting a C-first development approach to ensure a robust foundation.
Key changes include:
- **C-Level Testing Framework:** Established a C-level testing system in t/c/ with a dedicated Makefile, using an embedded Perl interpreter. Initial tests cover the object cache ( 01-cache.c ), arena wrapper ( 02-arena.c ), and utility functions ( 03-utils.c ).
- **Core XS Infrastructure:**
- Implemented a global object cache ( xs/protobuf.c ) to manage Perl wrappers for UPB objects, using weak references.
- Created an upb_Arena wrapper ( xs/protobuf.c ).
- Consolidated common XS helper functions into xs/protobuf.h and xs/protobuf.c .
- **Makefile.PL Enhancements:** Updated to support building and linking C tests, incorporating flags from ExtUtils::Embed , and handling both .c and .cc source files.
- **XS File Reorganization:** Restructured XS files to mirror the Python UPB extension's layout (e.g., message.c , descriptor.c ). Removed older, monolithic .xs files.
- **Typemap Expansion:** Added extensive typemap entries in perl/typemap to handle conversions between Perl objects and various const upb_*Def* pointers.
- **Descriptor Tests:** Added a new test suite t/02-descriptor.t to validate descriptor loading and accessor methods.
- **Documentation:** Updated development plans and guidelines ( ProtobufPlan.md , xs_learnings.md , etc.) to reflect the C-first strategy, new testing methods, and lessons learned.
- **Build Cleanup:** Removed ppport.h from .gitignore as it's no longer used, due to -DPERL_NO_PPPORT being set in Makefile.PL .
This C-first approach allows for more isolated and reliable testing of the core logic interacting with the UPB library before higher-level Perl APIs are built upon it.
## 0.10 2025-12-17
commit 1ef20ade24603573905cb0376670945f1ab5d829
Author: C.J. Collier
Date: Wed Dec 17 07:08:29 2025 +0000
feat(perl): Implement C-level tests and core XS utils
This commit introduces a C-level testing framework for the XS layer and implements key components:
1. **C-Level Tests ( t/c/ )**:
* Added t/c/Makefile to build standalone C tests.
* Created t/c/upb-perl-test.h with macros for TAP-compliant C tests ( plan , ok , is , is_string , diag ).
* Implemented t/c/01-cache.c to test the object cache.
* Implemented t/c/02-arena.c to test Protobuf::Arena wrappers.
* Implemented t/c/03-utils.c to test string utility functions.
* Corrected include paths and diagnostic messages in C tests.
2. **XS Object Cache ( xs/protobuf.c )**:
* Switched to using stringified pointers ( %p ) as hash keys for stability.
* Fixed a critical double-free bug in PerlUpb_ObjCache_Delete by removing an extra SvREFCNT_dec on the lookup key.
3. **XS Arena Wrapper ( xs/protobuf.c )**:
* Corrected PerlUpb_Arena_New to use newSVrv and PTR2IV for opaque object wrapping.
* Corrected PerlUpb_Arena_Get to safely unwrap the arena pointer.
4. **Makefile.PL ( perl/Makefile.PL )**:
* Added -Ixs to INC to allow C tests to find t/c/upb-perl-test.h and xs/protobuf.h .
* Added LIBS to link libprotobuf_common.a into the main Protobuf.so .
* Added C test targets 01-cache , 02-arena , 03-utils to the test config in MY::postamble .
5. **Protobuf.pm ( perl/lib/Protobuf.pm )**:
* Added use XSLoader; to load the compiled XS code.
6. **New files xs/util.h **:
* Added initial type conversion function.
These changes establish a foundation for testing the C-level interface with UPB and fix crucial bugs in the object cache implementation.
## 0.09 2025-12-17
commit 07d61652b032b32790ca2d3848243f9d75ea98f4
Author: C.J. Collier
Date: Wed Dec 17 04:53:34 2025 +0000
feat(perl): Build system and C cache test for Perl XS
This commit introduces the foundational pieces for the Perl XS implementation, focusing on the build system and a C-level test for the object cache.
- **Makefile.PL:**
- Refactored C test compilation rules in MY::postamble to use a hash ( $c_test_config ) for better organization and test-specific flags.
- Integrated ExtUtils::Embed to provide necessary compiler and linker flags for embedding the Perl interpreter, specifically for the t/c/01-cache.c test.
- Correctly constructs the path to the versioned Perl library ( libperl.so.X.Y.Z ) using $Config archlib and $Config libperl to ensure portability.
- Removed VERSION_FROM and ABSTRACT_FROM to avoid dependency on .pm files for now.
- **C Cache Test (t/c/01-cache.c):**
- Added a C test to exercise the object cache functions implemented in xs/protobuf.c .
- Includes tests for adding, getting, deleting, and weak reference behavior.
- **XS Cache Implementation (xs/protobuf.c, xs/protobuf.h):**
- Implemented PerlUpb_ObjCache_Init , PerlUpb_ObjCache_Add , PerlUpb_ObjCache_Get , PerlUpb_ObjCache_Delete , and PerlUpb_ObjCache_Destroy .
- Uses a Perl hash ( HV* ) for the cache.
- Keys are string representations of the C pointers, created using snprintf with "%llx" .
- Values are weak references ( sv_rvweaken ) to the Perl objects ( SV* ).
- PerlUpb_ObjCache_Get now correctly returns an incremented reference to the original SV, not a copy.
- PerlUpb_ObjCache_Destroy now clears the hash before decrementing its refcount.
- **t/c/upb-perl-test.h:**
- Updated is_sv to perform direct pointer comparison ( got == expected ).
- **Minor:** Added util.h (currently empty), updated typemap .
These changes establish a working C-level test environment for the XS components.
## 0.08 2025-12-17
commit d131fd22ea3ed8158acb9b0b1fe6efd856dc380e
Author: C.J. Collier
Date: Wed Dec 17 02:57:48 2025 +0000
feat(perl): Update docs and core XS files
- Explicitly add TDD cycle to ProtobufPlan.md.
- Clarify mirroring of Python implementation in upb-interfacing.md for both C and Perl layers.
- Branch and adapt python/protobuf.h and python/protobuf.c to perl/xs/protobuf.h and perl/xs/protobuf.c, including the object cache implementation. Removed old cache.* files.
- Create initial C test for the object cache in t/c/01-cache.c.
## 0.07 2025-12-17
commit 56fd6862732c423736a2f9a9fb1a2816fc59e9b0
Author: C.J. Collier
Date: Wed Dec 17 01:09:18 2025 +0000
feat(perl): Align Perl UPB architecture docs with Python
Updates the Perl Protobuf architecture documents to more closely align with the design and implementation strategies used in the Python UPB extension.
Key changes:
- **Object Caching:** Mandates a global, per-interpreter cache using weak references for all UPB-derived objects, mirroring Python's PyUpb_ObjCache .
- **Descriptor Containers:** Introduces a new document outlining the plan to use generic XS container types (Sequence, ByNameMap, ByNumberMap) with vtables to handle collections of descriptors, similar to Python's descriptor_containers.c .
- **Testing:** Adds a note to the testing strategy to port relevant test cases from the Python implementation to ensure feature parity.
## 0.06 2025-12-17
commit 6009ce6ab64eccce5c48729128e5adf3ef98e9ae
Author: C.J. Collier
Date: Wed Dec 17 00:28:20 2025 +0000
feat(perl): Implement object caching and fix build
This commit introduces several key improvements to the Perl XS build system and core functionality:
1. **Object Caching:**
* Introduces xs/protobuf.c and xs/protobuf.h to implement a caching mechanism ( protobuf_c_to_perl_obj ) for wrapping UPB C pointers into Perl objects. This uses a hash and weak references to ensure object identity and prevent memory leaks.
* Updates the typemap to use protobuf_c_to_perl_obj for upb_MessageDef * output, ensuring descriptor objects are cached.
* Corrected sv_weaken to the correct sv_rvweaken function.
2. **Makefile.PL Enhancements:**
* Switched to using the Bazel-generated UPB descriptor sources from bazel-bin/src/google/protobuf/_virtual_imports/descriptor_proto/google/protobuf/ .
* Updated INC paths to correctly locate the generated headers.
* Refactored MY::dynamic_lib to ensure the static library libprotobuf_common.a is correctly linked into each generated .so module, resolving undefined symbol errors.
* Overrode MY::test to use prove -b -j$(nproc) t/*.t xt/*.t for running tests.
* Cleaned up LIBS and LDDLFLAGS usage.
3. **Documentation:**
* Updated ProtobufPlan.md to reflect the current status and design decisions.
* Reorganized architecture documents into subdirectories.
* Added object-caching.md and c-perl-interface.md .
* Updated llm-guidance.md with notes on upb/upb.h and sv_rvweaken .
4. **Testing:**
* Fixed xt/03-moo_immutable.t to skip tests if no Moo modules are found.
This resolves the build issues and makes the core test suite pass.
## 0.05 2025-12-16
commit 177d2f3b2608b9d9c415994e076a77d8560423b8
Author: C.J. Collier
Date: Tue Dec 16 19:51:36 2025 +0000
Refactor: Rename namespace to Protobuf, build system and doc updates
This commit refactors the primary namespace from ProtoBuf to Protobuf
to align with the style guide. This involves renaming files, directories,
and updating package names within all Perl and XS files.
**Namespace Changes:**
* Renamed perl/lib/ProtoBuf to perl/lib/Protobuf .
* Moved and updated ProtoBuf.pm to Protobuf.pm .
* Moved and updated ProtoBuf::Descriptor to Protobuf::Descriptor (.pm & .xs).
* Removed other ProtoBuf::* stubs (Arena, DescriptorPool, Message).
* Updated MODULE and PACKAGE in Descriptor.xs .
* Updated NAME , *_FROM in perl/Makefile.PL .
* Replaced ProtoBuf with Protobuf throughout perl/typemap .
* Updated namespaces in test files t/01-load-protobuf-descriptor.t and t/02-descriptor.t .
* Updated namespaces in all documentation files under perl/doc/ .
* Updated paths in perl/.gitignore .
**Build System Enhancements (Makefile.PL):**
* Included xs/*.c files in the common object files list.
* Added -I. to the INC paths.
* Switched from MYEXTLIB to LIBS => ['-L$(CURDIR) -lprotobuf_common'] for linking.
* Removed custom keys passed to WriteMakefile for postamble.
* MY::postamble now sources variables directly from the main script scope.
* Added all :: $ common_lib dependency in MY::postamble .
* Added t/c/load_test.c compilation rule in MY::postamble .
* Updated clean target to include blib .
* Added more modules to TEST_REQUIRES .
* Removed the explicit PM and XS keys from WriteMakefile , relying on XSMULTI => 1 .
**New Files:**
* perl/lib/Protobuf.pm
* perl/lib/Protobuf/Descriptor.pm
* perl/lib/Protobuf/Descriptor.xs
* perl/t/01-load-protobuf-descriptor.t
* perl/t/02-descriptor.t
* perl/t/c/load_test.c : Standalone C test for UPB.
* perl/xs/types.c & perl/xs/types.h : For Perl/C type conversions.
* perl/doc/architecture/upb-interfacing.md
* perl/xt/03-moo_immutable.t : Test for Moo immutability.
**Deletions:**
* Old test files: t/00_load.t , t/01_basic.t , t/02_serialize.t , t/03_message.t , t/04_descriptor_pool.t , t/05_arena.t , t/05_message.t .
* Removed lib/ProtoBuf.xs as it's not needed with XSMULTI .
**Other:**
* Updated test_descriptor.bin (binary change).
* Significant content updates to markdown documentation files in perl/doc/architecture and perl/doc/internal reflecting the new architecture and learnings.
## 0.04 2025-12-14
commit 92de5d482c8deb9af228f4b5ce31715d3664d6ee
Author: C.J. Collier
Date: Sun Dec 14 21:28:19 2025 +0000
feat(perl): Implement Message object creation and fix lifecycles
This commit introduces the basic structure for ProtoBuf::Message object
creation, linking it with ProtoBuf::Descriptor and ProtoBuf::DescriptorPool ,
and crucially resolves a SEGV by fixing object lifecycle management.
Key Changes:
1. ** ProtoBuf::Descriptor :** Added _pool attribute to hold a strong
reference to the parent ProtoBuf::DescriptorPool . This is essential to
prevent the pool and its C upb_DefPool from being garbage collected
while a descriptor is still in use.
2. ** ProtoBuf::DescriptorPool :**
* find_message_by_name : Now passes the $self (the pool object) to the
ProtoBuf::Descriptor constructor to establish the lifecycle link.
* XSUB pb_dp_find_message_by_name : Updated to accept the pool SV* and
store it in the descriptor's _pool attribute.
* XSUB _load_serialized_descriptor_set : Renamed to avoid clashing with the
Perl method name. The Perl wrapper now correctly calls this internal XSUB.
* DEMOLISH : Made safer by checking for attribute existence.
3. ** ProtoBuf::Message :**
* Implemented using Moo with lazy builders for _upb_arena and
_upb_message .
* _descriptor is a required argument to new() .
* XS functions added for creating the arena ( pb_msg_create_arena ) and
the upb_Message ( pb_msg_create_upb_message ).
* pb_msg_create_upb_message now extracts the upb_MessageDef* from the
descriptor and uses upb_MessageDef_MiniTable() to get the minitable
for upb_Message_New() .
* DEMOLISH : Added to free the message's arena.
4. ** Makefile.PL :**
* Added -g to CCFLAGS for debugging symbols.
* Added Perl CORE include path to MY::postamble 's base_flags .
5. **Tests:**
* t/04_descriptor_pool.t : Updated to check the structure of the
returned ProtoBuf::Descriptor .
* t/05_message.t : Now uses a descriptor obtained from a real pool to
test ProtoBuf::Message->new() .
6. **Documentation:**
* Updated ProtobufPlan.md to reflect progress.
* Updated several files in doc/architecture/ to match the current
implementation details, especially regarding arena management and object
lifecycles.
* Added doc/internal/development_cycle.md and doc/internal/xs_learnings.md .
With these changes, the SEGV is resolved, and message objects can be successfully
created from descriptors.
## 0.03 2025-12-14
commit 6537ad23e93680c2385e1b571d84ed8dbe2f68e8
Author: C.J. Collier
Date: Sun Dec 14 20:23:41 2025 +0000
Refactor(perl): Object-Oriented DescriptorPool with Moo
This commit refactors the ProtoBuf::DescriptorPool to be fully object-oriented using Moo, and resolves several issues related to XS, typemaps, and test data.
Key Changes:
1. **Moo Object:** ProtoBuf::DescriptorPool.pm now uses Moo to define the class. The upb_DefPool pointer is stored as a lazy attribute _upb_defpool .
2. **XS Lifecycle:** DescriptorPool.xs now has pb_dp_create_pool called by the Moo builder and pb_dp_free_pool called from DEMOLISH to manage the upb_DefPool lifecycle per object.
3. **Typemap:** The perl/typemap file has been significantly updated to handle the conversion between the ProtoBuf::DescriptorPool Perl object and the upb_DefPool * C pointer. This includes:
* Mapping upb_DefPool * to T_PTR .
* An INPUT section for ProtoBuf::DescriptorPool to extract the pointer from the object's hash, triggering the lazy builder if needed via call_method .
* An OUTPUT section for upb_DefPool * to convert the pointer back to a Perl integer, used by the builder.
4. **Method Renaming:** add_file_descriptor_set_binary is now load_serialized_descriptor_set .
5. **Test Data:**
* Added perl/t/data/test.proto with a sample message and enum.
* Generated perl/t/data/test_descriptor.bin using protoc .
* Removed t/data/ from .gitignore to ensure test data is versioned.
6. **Test Update:** t/04_descriptor_pool.t is updated to use the new OO interface, load the generated descriptor set, and check for message definitions.
7. **Build Fixes:**
* Corrected #include paths in DescriptorPool.xs to be relative to the upb/ directory (e.g., upb/wire/decode.h ).
* Added -I../upb to CCFLAGS in Makefile.PL .
* Reordered INC paths in Makefile.PL to prioritize local headers.
**Note:** While tests now pass in some environments, a SEGV issue persists in make test runs, indicating a potential memory or lifecycle issue within the XS layer that needs further investigation.
## 0.02 2025-12-14
commit 6c9a6f1a5f774dae176beff02219f504ea3a6e07
Author: C.J. Collier
Date: Sun Dec 14 20:13:09 2025 +0000
Fix(perl): Correct UPB build integration and generated file handling
This commit resolves several issues to achieve a successful build of the Perl extension:
1. **Use Bazel Generated Files:** Switched from compiling UPB's stage0 descriptor.upb.c to using the Bazel-generated descriptor.upb.c and descriptor.upb_minitable.c located in bazel-bin/src/google/protobuf/_virtual_imports/descriptor_proto/google/protobuf/ .
2. **Updated Include Paths:** Added the bazel-bin path to INC in WriteMakefile and to base_flags in MY::postamble to ensure the generated headers are found during both XS and static library compilation.
3. **Removed Stage0:** Removed references to UPB_STAGE0_DIR and no longer include headers or source files from upb/reflection/stage0/ .
4. **-fPIC:** Explicitly added -fPIC to CCFLAGS in WriteMakefile and ensured $(CCFLAGS) is used in the custom compilation rules in MY::postamble . This guarantees all object files in the static library are compiled with position-independent code, resolving linker errors when creating the shared objects for the XS modules.
5. **Refined UPB Sources:** Used File::Find to recursively find UPB C sources, excluding /conformance/ and /reflection/stage0/ to avoid conflicts and unnecessary compilations.
6. **Arena Constructor:** Modified ProtoBuf::Arena::pb_arena_new XSUB to accept the class name argument passed from Perl, making it a proper constructor.
7. **.gitignore:** Added patterns to perl/.gitignore to ignore generated C files from XS ( lib/*.c , lib/ProtoBuf/*.c ), the copied src_google_protobuf_descriptor.pb.cc , and the t/data directory.
8. **Build Documentation:** Updated perl/doc/architecture/upb-build-integration.md to reflect the new build process, including the Bazel prerequisite, include paths, -fPIC usage, and File::Find .
Build Steps:
1. bazel build //src/google/protobuf:descriptor_upb_proto (from repo root)
2. cd perl
3. perl Makefile.PL
4. make
5. make test (Currently has expected failures due to missing test data implementation).
## 0.01 2025-12-14
commit 3e237e8a26442558c94075766e0d4456daaeb71d
Author: C.J. Collier
Date: Sun Dec 14 19:34:28 2025 +0000
feat(perl): Initialize Perl extension scaffold and build system
This commit introduces the perl/ directory, laying the groundwork for the Perl Protocol Buffers extension. It includes the essential build files, linters, formatter configurations, and a vendored Devel::PPPort for XS portability.
Key components added:
* ** Makefile.PL **: The core ExtUtils::MakeMaker build script. It's configured to:
* Build a static library ( libprotobuf_common.a ) from UPB, UTF8_Range, and generated protobuf C/C++ sources.
* Utilize XSMULTI => 1 to create separate shared objects for ProtoBuf , ProtoBuf::Arena , and ProtoBuf::DescriptorPool .
* Link each XS module against the common static library.
* Define custom compilation rules in MY::postamble to handle C vs. C++ flags and build the static library.
* Set up include paths for the project root, UPB, and other dependencies.
* **XS Stubs ( .xs files)**:
* lib/ProtoBuf.xs : Placeholder for the main module's XS functions.
* lib/ProtoBuf/Arena.xs : XS interface for upb_Arena management.
* lib/ProtoBuf/DescriptorPool.xs : XS interface for upb_DefPool management.
* **Perl Module Stubs ( .pm files)**:
* lib/ProtoBuf.pm : Main module, loads XS.
* lib/ProtoBuf/Arena.pm : Perl class for Arenas.
* lib/ProtoBuf/DescriptorPool.pm : Perl class for Descriptor Pools.
* lib/ProtoBuf/Message.pm : Base class for messages (TBD).
* **Configuration Files**:
* .gitignore : Ignores build artifacts, editor files, etc.
* .perlcriticrc : Configures Perl::Critic for static analysis.
* .perltidyrc : Configures perltidy for code formatting.
* ** Devel::PPPort **: Vendored version 3.72 to generate ppport.h for XS compatibility across different Perl versions.
* ** typemap **: Custom typemap for XS argument/result conversion.
* **Documentation ( doc/ )**: Initial architecture and plan documents.
This provides a solid foundation for developing the UPB-based Perl extension.
The Debian LTS Team, funded by [Freexian s Debian LTS offering]
(https://www.freexian.com/lts/debian/), is pleased to report its activities for
November.
Activity summary
During the month of November, 18 contributors have been
paid to work on Debian LTS (links to individual
contributor reports are located below).
The team released 33 DLAs
fixing 219 CVEs.
The LTS Team kept going with the usual cadence of preparing security updates for Debian
11 bullseye , but also for Debian 12 bookworm , Debian 13 trixie and even
Debian unstable.
As in previous months, we are pleased to say that there have been multiple
contributions of LTS uploads by Debian Fellows outside the regular LTS Team.
Notable security updates:
Guilhem Moulin prepared DLA 4365-1
for unbound, a caching DNS resolver, fixing a cache poisoning vulnerability
that could lead to domain hijacking.
Another update related to DNS software was made by Andreas Henriksson. Andreas
completed the work on bind9, released as
DLA 4364-1 to fix
cache poisoning and Denial of Service (DoS) vulnerabilities.
Chris Lamb released DLA 4374-1
to fix a potential arbitrary code execution vulnerability in pdfminer, a tool
for extracting information from PDF documents.
Ben Hutchings published a regular security update for the linux 6.1 bullseye
backport, as DLA 4379-1.
A couple of other important recurrent updates were prepared by Emilio Pozuelo,
who handled firefox-esr and thunderbird (in collaboration with Christoph
Goehre), published as DLAs
DLA 4370-1 and
DLA 4372-1,
respectively.
Contributions from fellows outside the LTS Team:
Thomas Goirand uploaded a bullseye update for
keystone
and
swift
As mentioned above, Christoph Goehre prepared the
bullseye update for thunderbird.
Mathias Behrle provided feedback about the tryton-server and tryton-sao vulnerabilities that were disclosed last month, and helped to review the bullseye patches for tryton-server.
Other than the regular LTS updates for bullseye, the LTS Team has also
contributed updates to the latest Debian releases:
Bastien Roucari s prepared a bookworm update for
squid,
the web proxy cache server.
Carlos Henrique Lima Melara filed a bookworm point update
request for gdk-pixbuf to fix
CVE-2025-7345, a heap buffer overflow vulnerability that could lead to
arbitrary code execution.
Daniel Leidert prepared bookworm and
trixie updates for r-cran-gh to fix
CVE-2025-54956, an issue that may expose user credentials in HTTP responses.
Along with the bullseye updates for unbound mentioned above, Guilhem helped
to prepare the trixie update
for unbound.
In collaboration with Lukas M rdian, Tobias Frost prepared
trixie and
bookworm
updates for log4cxx, the C++ port of the logging framework for JAVA.
Jochen Sprickerhof prepared a bookworm update for syslog-ng.
Utkarsh completed the bookworm update
for wordpress, addressing multiple security issues in the popular blogging
tool.
Beyond security updates, there has been a significant effort in revamping our
documentation, aiming to make the processes more clear and consistent for all
the members of the team. This work was mainly carried out by Sylvain, Jochen
and Roberto.
We would like to express our gratitude to the sponsors for making the Debian
LTS project possible. Also, special thanks to the fellows outside the LTS
team for their valuable help.
The Release of Debian 13 ("Trixie") last month marked another milestone on the
effort to provide automated test support for Debian packages in their installed
form. We have achieved the mark of 57% of the source packages in the archive
declaring support for autopkgtest.
Release
Packages with tests
Total number of packages
% of packages with tests
wheezy
5
17175
0%
jessie
1112
20596
5%
stretch
5110
24845
20%
buster
9966
28501
34%
bullseye
13949
30943
45%
bookworm
17868
34341
52%
trixie
21527
37670
57%
The code that generated this table is provided at the bottom.
The growth rate has been consistently decreasing at each release after stretch.
That probably means that the low hanging fruit -- adding support en masse for
large numbers of similar packages, such as team-maintained packages for a given
programming language -- has been picked, and from now on the work gets slightly
harder. Perhaps there is a significant long tail of packages that will never
get autopkgtest support.
Looking for common prefixes among the packages missing a Testsuite: field
gives me us the largest groups of packages missing autopkgtest support:
There seems to be a fair amount of Haskell and Python. If someone could figure
out a way of testing installed fonts in a meaningful way, this would a be a
good niche where we can cover 300+ packages.
There is a another analysis that can be made, which I didn't: which percentage
of new packages introduced in a given release have declared autopkgtest
support, compared with the total of new packages in that release? My data only
counts the totals, so we start with the technical debt of the almost all of the
17,000 packages with no tests in wheezy, which was the stable at the time I
started Debian CI. How many of those got tests since
then?
Note that not supporting autopkgtest does not mean that a package is not
tested at all: it can run build-time tests, which are also useful. Not
supporting autopkgtest, though, means that their binaries in the archive
can't be automatically tested in their installed, but then there is a entire
horde of volunteers running testing and unstable on a daily basis who test
Debian and report bugs.
This is the script that produced the table in the beginning of this post:
#!/bin/shset-eu
extract()local release
local url
release="$1"url="$2"if[!-f"$ release"];then
rm-f"$ release.gz"
curl --silent-o$ release.gz "$ url"gunzip"$ release.gz"fi
local with_tests
local total
with_tests="$(grep-dctrl -c-F Testsuite --regex.$release)"total="$(grep-dctrl -c-F Package --regex.$release)"echo" $ release$ with_tests$ total$((100*with_tests/total))% "echo" **Release** **Packages with tests** **Total number of packages** **% of packages with tests** "echo" ------------- ------------------------- ------------------------------ ------------------------------ "for release in wheezy jessie stretch buster;do
extract "$ release""http://archive.debian.org/debian/dists/$ release/main/source/Sources.gz"done
for release in bullseye bookworm trixie;do
extract "$ release""http://ftp.br.debian.org/debian/dists/$ release/main/source/Sources.gz"done
Posted on September 3, 2025
Tags: madeof:bits
For quite some time, I have been thinking about trying a bit of
patchwork, and English Paper Piecing looked like a technique suited to
my tastes, with the handsewing involved and the fact of having a paper
pattern of sort and everything.
The problem is, most of the scraps of fabric I get from my sewing aren t
really suitable for quilting, with a lot of them being either too black
and too thick or too white and too thin.
On the other hand, my partner wears polo shirts at work, and while I
try to mend the holes that form, after a while the edges get worn, and
they just are no longer suitable for the office, even with some creative
mending, and they get downgraded to home wear. But then more office
shirts need to be bought, and the home ones accumulate, and there is
only so much room for polo shirts in the house, and the worst ones end
up in my creative reuse pile.
Some parts are worn out and they will end up as cabbage stuffing for
things, but some are still in decent enough conditions and could be used
as fabric.
But surely, for English Paper Piecing you d need woven fabric, not knit,
even if it s the dense piqu used in polo shirts, right? Especially if
it s your first attempt at the technique, right?
Well, probably it wouldn t work with complex shapes, but what about some
5-ish cm tall Standard Compliant bestagon? So I printed out some
hexagons on thick paper, printed some bigger hexagons with sewing
allowance as a cutting aid, found two shirts in the least me colours I
could find (and one in grey because it was the best match for the other
two) and decided to sacrifice them for the experiment.
And as long as the paper was still in the pieces, the work went nicely,
so I persevered while trying to postpone the Moment of Truth.
After a while I measured things out and saw that I could squeeze a 6.5
7 hexagon pattern into something resembling a square that was a multiple
of the 2.5 cm square on the back of my Piecepack tiles, and decided to
go for another Standard for the back (because of course I wasn t going
to buy new fabric for lining the work).
I kept the paper in the pieces until both sides were ready, and used it
to sew them right sides together, leaving the usual opening in the
middle of one side.
Then I pressed, removed the paper, turned everything inside out, pressed
again and. It worked!
The hexagons look like hexagons, the squares look like squares, the
whole thing feels soft and drapey, but structurally sound. And it s a
bit lumpy, but not enough to cause issues when using it as a soft
surface to put over a noisy wooden table to throw dice on.
I considered adding some lightweight batting in the middle, but there
was really no need for it, and wondered about how to quilt the piece in
a way that worked with the patterns on the two sides, but for something
this small it wasn t really required.
However, I decided to add a buttonhole stitch border on all edges, to
close the opening I had left and to reinforce especially the small
triangles on the hexagons side, as those had a smaller sewing allowance
and could use it.
And of course, the 11 11 squares side wasn t completely an accident,
but part of A Plan.
For this project there isn t really a pattern, but I did publish the
files I used to print the paper pieces
even if they were pretty trivial.
And there are more polo shirts in that pile, and while they won t be
suitable for anything complex, maybe I could try some rhombs, or even
kites and darts?
I ve noticed that procrastination and inability to be consistently productive at work has become quite common in recent years. This is clearly visible in younger people who have grown up with an endless stream of entertainment literally at their fingertips, on their mobile phone. It is however a trap one can escape from with a little bit of help.
Procrastination is natural they say humans are lazy by nature after all. Probably all of us have had moments when we choose to postpone a task we know we should be working on, and instead spent our time doing secondary tasks (valorisation). Classic example is cleaning your apartment when you should be preparing for an exam. Some may procrastinate by not doing any work at all, and just watching YouTube videos or the like. To some people, typically those who are in their 20s and early in their career, procrastination can be a big challenge and finding the discipline to stick to planned work may need intentional extra effort, and perhaps even external help.
During my 20+ year career in software development I ve been blessed to work with engineers of various backgrounds and each with their unique set of strengths. I have also helped many grow in various areas and overcome challenges, such as lack of intrinsic motivation and managing procrastination, and some might be able to get it in check with some simple advice.
Distance yourself from the digital distractions
The key to avoiding distractions and procrastination is to make it inconvenient enough that you rarely do it. If continuing to do work is easier than switching to procrastination, work is more likely to continue.
Tips to minimize digital distractions, listed in order of importance:
Put your phone away. Just like when you go to a movie and turn off your phone for two hours, you can put the phone away completely when starting to work. Put the phone in a different room to ensure there is enough physical distance between you and the distraction, so it is impossible for you to just take a quick peek .
Turn off notifications from apps. Don t let the apps call you like sirens luring Odysseus. You don t need to have all the notifications. You will see what the apps have when you eventually open them at a time you choose to use them.
Remove or disable social media apps, games and the like from your phone and your computer. You can install them back when you have vacation. You can probably live without them for some time. If you can t remove them, explore your phone s screen time restriction features to limit your own access to apps that most often waste your time. These features are sometimes listed in the phone settings under digital health .
Have a separate work computer and work phone. Having dedicated ones just for work that are void of all unnecessary temptations helps keep distance from the devices that could derail your focus.
Listen to music. If you feel your brain needs a dose of dopamine to get you going, listening to music helps satisfy your brain s cravings while still being able to simultaneously keep working.
Doing a full digital detox is probably not practical, or not sustainable for an extended time. One needs apps to stay in touch with friends and family, and staying current in software development probably requires spending some time reading news online and such. However the tips above can help contain the distractions and minimize the spontaneous attention the distractions get.
Some of the distractions may ironically be from the work itself, for example Slack notifications or new email notifications. I recommend turning them off for a couple of hours every day to have some distraction free time. It should be enough to check work mail a couple times a day. Checking them every hour probably does not add much overall value for the company unless your work is in sales or support where the main task itself is responding to emails.
Distraction free work environment
Following the same principle of distancing yourself from distractions, try to use a dedicated physical space for working. If you don t have a spare room to dedicate to work, use a neighborhood caf or sign up for a local co-working space or start commuting to the company office to find a space to be focused on work in.
Break down tasks into smaller steps
Sometimes people postpone tasks because they feel intimidated by the size or complexity of a task. In particular in software engineering problems may be vague and appear large until one reaches the breakthrough that brings the vision of how to tackle it. Breaking down problems into smaller more manageable pieces has many advantages in software engineering. Not only can it help with task-avoidance, but it can also make the problem easier to analyze, suggest solutions and test them and build a solid foundation to expand upon to ultimately later reach a full solution on the entire larger problem.
Working on big problems as a chain of smaller tasks may also offer more opportunities to celebrate success on completing each subtask and help getting in a suitable cadence of solving a single thing, taking a break and then tackling the next issue.
Breaking down a task into concrete steps may also help with getting more realistic time estimations. Sometimes procrastination isn t real someone could just be overly ambitious and feel bad about themselves for not doing an unrealistic amount of work.
Intrinsic motivation
Of course, you should follow your passion when possible. Strive to pick a career that you enjoy, and thus maximize the intrinsic motivation you experience. However, even a dream job is still a job. Nobody is ever paid to do whatever they want. Any work will include at least some tasks that feel like a chore or otherwise like something you would not do unless paid to.
Some would say that the definition of work itself is having to do things one would otherwise not do. You can only fully do whatever you want while on vacation or when you choose to not have a job at all. But if you have a job, you simply need to find the intrinsic motivation to do it.
Simply put, some tasks are just unpleasant or boring. Our natural inclination is to avoid them in favor of more enjoyable activities. For these situations we just have to find the discipline to force ourselves to do the tasks and figuratively speaking whip ourselves into being motivated to complete the tasks.
Extrinsic motivation
As the name implies, this is something people external to you need to provide, such as your employer or manager. If you have challenges in managing yourself and delivering results on a regular basis, somebody else needs to set goals and deadlines and keep you accountable for them. At the end of the day this means that eventually you will stop receiving salary or other payments unless you did your job.
Forcing people to do something isn t nice, but eventually it needs to be done. It would not be fair for an employer to pay those who did their work the same salary as those who procrastinated and fell short on their tasks.
If you work solo, you can also simulate the extrinsic motivation by publicly announcing milestones and deadlines to build up pressure for yourself to meet them and avoid publicly humiliation. It is a well-studied and scientifically proven phenomenon that most university students procrastinate at the start of assignments, and truly start working on them only once the deadline is imminent.
External help for addictions
If procrastination is mainly due to a single distraction that is always on your mind, it may be a sign of an addiction. For example, constantly thinking about a computer game or staying up late playing a computer game, to the extent that it seriously affects your ability to work, may be a symptom of an addiction, and getting out of it may be easier with external help.
Discipline and structure
Most of the time procrastination is not due to an addiction, but simply due to lack of self-discipline and structure. The good thing is that those things can be learned. It is mostly a matter of getting into new habits, which most young software engineers pick up more or less automatically while working along the more senior ones.
Hopefully these tips can help you stay on track and ensure you do everything you are expected to do with clear focus, and on time!
Today marks both a milestone and a turning point in my journey with open source software. I m proud to announce the release of KDE Gear 25.08.0 as my final snap package release. You can find all the details about this exciting update at the official KDE announcement.
After much reflection and with a heavy heart, I ve made the difficult decision to retire from most of my open source software work, including snap packaging. This wasn t a choice I made lightly it comes after months of rejections and silence in an industry I ve loved and called home for over 20 years.
Passing the Torch
While I m stepping back, I m thrilled to share that the future of KDE snaps is in excellent hands. Carlos from the Neon team has been working tirelessly to set up snaps on the new infrastructure that KDE has made available. This means building snaps in KDE CI is now possible a significant leap forward for the ecosystem. I ll be helping Carlos get the pipelines properly configured to ensure a smooth transition.
Staying Connected (But Differently)
Though I m stepping away from most development work, I won t be disappearing entirely from the communities that have meant so much to me:
Kubuntu: I ll remain available as a backup, though Rik is doing an absolutely fabulous job getting the latest and greatest KDE packages uploaded. The distribution is in capable hands.
Ubuntu Community Council: I m continuing my involvement here because I ve found myself genuinely enjoying the community side of things. There s something deeply fulfilling about focusing on the human connections that make these projects possible.
Debian: I ll likely be submitting for emeritus status, as I haven t had the time to contribute meaningfully and want to be honest about my current capacity.
The Reality Behind the Decision
This transition isn t just about career fatigue it s about financial reality. I ve spent too many years working for free while struggling to pay my bills. The recent changes in the industry, particularly with AI transforming the web development landscape, have made things even more challenging. Getting traffic to websites now requires extensive social media work and marketing all expected to be done without compensation.
My stint at webwork was good while it lasted, but the changing landscape has made it unsustainable. I ve reached a point where I can t continue doing free work when my family and I are struggling financially. It shouldn t take breaking a limb to receive the donations needed to survive.
A Career That Meant Everything
These 20+ years in open source have been the defining chapter of my professional life. I ve watched communities grow, technologies evolve, and witnessed firsthand the incredible things that happen when passionate people work together. The relationships I ve built, the problems we ve solved together, and the software we ve created have been deeply meaningful.
But I also have to be honest about where I stand today: I cannot compete in the current job market. The industry has changed, and despite my experience and passion, the opportunities just aren t there for someone in my situation.
Looking Forward
Making a career change after two decades is terrifying, but it s also necessary. I need to find a path that can provide financial stability for my family while still allowing me to contribute meaningfully to the world.
If you ve benefited from my work over the years and are in a position to help during this transition, I would be forever grateful for any support. Every contribution, no matter the size, helps ease this difficult period: https://gofund.me/a9c55d8f
Thank You
To everyone who has collaborated with me, tested my packages, filed bug reports, offered encouragement, or simply used the software I ve helped maintain thank you. You ve made these 20+ years worthwhile, and you ve been part of something bigger than any individual contribution.
The open source world will continue to thrive because it s built on the collective passion of thousands of people like Carlos, Rik, and countless others who are carrying the torch forward. While my active development days are ending, the impact of this community will continue long into the future.
With sincere gratitude and fond farewells,
Scarlett Moore
We are pleased to announce the official release of Debian 13, codenamed trixie!
What's New in Debian 13
Official support for RISC-V (64-bit riscv64), a major architecture
milestone
Enhanced security through ROP and COP/JOP hardening on both amd64 and
arm64 (Intel CET and ARM PAC/BTI support)
HTTP Boot support in Debian Installer and Live images for UEFI/U-Boot
systems
Upgraded software stack: GNOME 48, KDE Plasma 6, Linux kernel 6.12 LTS,
GCC 14.2, Python 3.13, and more
Want to install it?
Fresh installation ISOs are now available, including the final
Debian Installer
featuring kernel 6.12.38 and mirror improvements. Choose your favourite
installation media and read the
installation manual.
You can also use an official cloud image directly on your cloud provider, or
try Debian prior to installing it using our "live" images.
Already a happy Debian user and you only want to upgrade?
Full upgrade path from Debian 12 "bookworm" is supported and documented in
the Release Notes.
Upgrade notes cover APT source preparation, handling obsoletes, and ensuring
system resilience.
Additional Information
For full details, including upgrade instructions, known issues, and
contributors, see the official
Release Notes
for Debian 13 "trixie".
Congratulations to all developers, QA testers, and volunteers who made Debian 13
"trixie" possible!
Do you want to celebrate the release?
To celebrate with us on this occassion find a
release party near to you and if
there isn't any, organize one!
Welcome to the seventh report from the Reproducible Builds project in 2025. Our monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. If you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website.
In this report:
Reproducible Builds Summit 2025
We are extremely pleased to announce the upcoming Reproducible Builds Summit, set to take place from October 28th 30th 2025 in Vienna, Austria!
We are thrilled to host the eighth edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin, Hamburg and Athens. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort.
During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving.
If you re interesting in joining us this year, please make sure to read the event page which has more details about the event and location. Registration is open until 20th September 2025, and we are very much looking forward to seeing many readers of these reports there!
[Everything] changed earlier this year when reproducible-builds for SLES-16 became an official goal for the product. More people are talking about digital sovereignty and supply-chain security now. [ ] Today, only 9 of 3319 (source) packages have significant problems left (plus 7 with pending fixes), so 99.5% of packages have reproducible builds.
There are numerous policy compliance and regulatory processes being developed that target software development but do they solve actual problems? Does it improve the quality of software? Do Software Bill of Materials (SBOMs) actually give you the information necessary to verify how a given software artifact was built? What is the goal of all these compliance checklists anyways or more importantly, what should the goals be? If a software object is signed, who should be trusted to sign it, and can they be trusted forever?
Hosted by the Software Freedom Conservancy and taking place in Portland, Oregon, USA, FOSSY aims to be a community-focused event: Whether you are a long time contributing member of a free software project, a recent graduate of a coding bootcamp or university, or just have an interest in the possibilities that free and open source software bring, FOSSY will have something for you . More information on the event is available on the FOSSY 2025 website, including the full programme schedule.
Vagrant and Chris also staffed a table, where they will be available to answer any questions about Reproducible Builds and discuss collaborations with other projects.
Automation to derive declarative build definitions for existing PyPI (Python), npm (JS/TS), and Crates.io (Rust) packages.
SLSA Provenance for thousands of packages across our supported ecosystems, meeting SLSA Build Level 3 requirements with no publisher intervention.
Build observability and verification tools that security teams can integrate into their existing vulnerability management workflows.
Infrastructure definitions to allow organizations to easily run their own instances of OSS Rebuild to rebuild, generate, sign, and distribute provenance.
One difference with most projects that aim for bit-for-bit reproducibility, OSS Rebuild aims for a kind of semantic reproducibility:
Through automation and heuristics, we determine a prospective build definition for a target package and rebuild it. We semantically compare the result with the existing upstream artifact, normalizing each one to remove instabilities that cause bit-for-bit comparisons to fail (e.g. archive compression).
The extensive post includes examples about how to access OSS Rebuild attestations using the Go-based command-line interface.
New extension of Python setuptools to support reproducible builds
Wim Jeantine-Glenn has written a PEP 517 Build backend in order to enable reproducible builds when building Python projects that use setuptools.
Called setuptools-reproducible, the project s README file contains the following:
Setuptools can create reproducible wheel archives (.whl) by setting SOURCE_DATE_EPOCH at build time, but setting the env var is insufficient for creating reproducible sdists (.tar.gz). setuptools-reproducible [therefore] wraps the hooks build_sdistbuild_wheel with some modifications to make reproducible builds by default.
diffoscopediffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 301, 302 and 303 to Debian:
Improvements:
Use Difference.from_operation in an attempt to pipeline the output of the extract-vmlinux script, potentially avoiding it all in memory. []
Memoize a number of calls to --version, saving a very large number of external subprocess calls.
Bug fixes:
Don t check for PyPDF version 3 specifically, check for versions greater than 3. []
Ensure that Java class files are named .class on the filesystem before passing them to javap(1). []
Mask stderr from extract-vmlinux script. [][]
Avoid spurious differences in h5dump output caused by exposure of absolute internal extraction paths. (#1108690)
Misc:
Use our_check_output in the ODT comparator. []
Update copyright years. []
In addition:
Siva Mahadevan made a change to use the --print-armap long option when calling nm(1) for wider compatibility. []
Lastly, Chris Lamb added a tmpfs to try.diffoscope.org so that diffoscope has a non-trivial temporary area to unpack archives, etc. []
Elsewhere in our tooling, however, reprotest is our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. This month, reprotest version 0.7.30 was uploaded to Debian unstable by Holger Levsen, chiefly including a change by Rebecca N. Palmer to not call sudo with the -h flag in order to fix Debian bug #1108550. []
New library to patch system functions for reproducibility
Nicolas Graves has written and published libfate, a simple collection of tiny libraries to patch system functions deterministically using LD_PRELOAD. According to the project s README:
libfate provides deterministic replacements for common non-deterministic system functions that can break reproducible builds. Instead of relying on complex build systems or apps or extensive patching, libfate uses the LD_PRELOAD trick to intercept system calls and return fixed, predictable values.
Describing why he wrote it, Nicolas writes:
I originally used the OpenSUSE dettrace approach to make Emacs reproducible in Guix. But when Guix switch to GCC@14, dettrace stopped working as expected. dettrace is a complex piece of software, my need was much less heavy: I don t need to systematically patch all sources of nondetermism, just the ones that make a process/binary unreproducible in a container/chroot.
One desirable property is that someone else should be able to reproduce the same git bundle, and not only that a single individual is able to reproduce things on one machine. It surprised me to see that when I ran the same set of commands on a different machine (started from a fresh git clone), I got a different checksum. The different checksums occurred even when nothing had been committed on the server side between the two runs.
Website updates
Once again, there were a number of improvements made to our website this month including:
Bernhard M. Wiedemann added an entry related to R-B-OS on the History page. []
Chris Lamb:
Replaced rbtlog run by Fay by rbtlog run by Benl on the Who is involved page. []
Debian contributors have made significant progress toward ensuring package builds produce byte-for-byte reproducible results. You can check the status for packages installed on your system using the new package debian-repro-status, or visit reproduce.debian.net for Debian s overall statistics for trixie and later. You can contribute to these efforts by joining #debian-reproducible on IRC to discuss fixes, or verify the statistics by installing the new rebuilderd package and setting up your own instance.
Reproducibility testing framework
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In July, however, a number of changes were made by Holger Levsen, including:
Make the dsa-check-packages output more useful. []
Setup the ppc64el architecture again, has it has returned this time with a 2.7 GiB database instead of 72 GiB. []
In addition, Jochen Sprickerhof improved the reproducibility statistics generation:
Enable caching of statistics. [][][]
Add some common non-reproducible patterns. []
Change output to directory. []
Add a page sorted by diffoscope size. [][]
Switch to Python s argparse module and separate output(). []
Holger also submitted a number of Debian bugs against rebuilderd and rebuilderd-worker:
Config files and scripts for a simple one machine setup. [][]
Create a rebuilderd user. []
Create rebuilderd-worker user with sbuild. []
Lastly, Mattia Rizzolo added a scheduled job to renew some SSL certificates [] and Vagrant Cascadian performed some node maintenance [][].
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
I ve participated in this year s Google Summer of Code (GSoC)
program and have been working on the small (90h) autopkgtests for the rsync
package project at Debian.
Writing my proposal
Before you can start writing a proposal, you need to select an organization
you want to work with. Since many organizations participate in GSoC, I ve
used the following criteria to narrow things down for me:
Programming language familiarity: For me only Python (preferably) as well
as shell and Go projects would have made sense. While learning another
programming language is cool, I wouldn t be as effective and helpful to
the project as someone who is proficient in the language already.
Standing of the organization: Some of the organizations participating in
GSoC are well-known for the outstanding quality of the software they
produce. Debian is one of them, but so is e.g. the Django Foundation or
PostgreSQL. And my thinking was that the higher the quality of the
organization, the more there is to learn for me as a GSoC student.
Mentor interactions: Apart from the advantage you get from mentor
feedback when writing your proposal (more on that further below), it is
also helpful to gauge how responsive/helpful your potential mentor is
during the application phase. This is important since you will be working
together for a period of at least 2 months; if the mentor-student
communication doesn t work, the GSoC project is going to be difficult.
Free and Open-Source Software (FOSS) communication platforms: I
generally believe that FOSS projects should be built on FOSS
infrastructure. I personally won t run proprietary software
when I want to contribute to FOSS in my spare time.
Be a user of the project: As Eric S. Raymond has pointed out
in his seminal The Cathedral and the Bazaar 25 years ago
Every good work of software starts by scratching a developer s personal
itch.
Once I had some organizations in mind whose projects I d be interested in
working on, I started writing proposals for them. Turns out, I started
writing my proposals way too late: In the end I only managed to hand in a
single one which is risky. Competition for the GSoC projects is fierce
and the more quality (!) proposals you send out, the better your chances are
at getting one. However, don t write proposals for the sake of it: Reviewers
get way too many AI slop proposals already and you will not do yourself a
favor with a low-quality proposal. Take the time to read the
instructions/ideas/problem descriptions the project mentors have provided
and follow their guidelines. Don t hesitate to reach out to project mentors:
In my case, I ve asked Samuel Henrique a few clarification questions whereby
the following (email) discussion has helped me greatly in improving my
proposal. Once I ve finalized my proposal draft, I ve sent it to Samuel for
a review, which again led to some improvements to the final proposal
which I ve uploaded to the GSoC program webpage.
Community bonding period
Once you get the information that you ve been accepted into the GSoC program
(don t take it personally if you don t make it; this was my second attempt
after not making the cut in 2024), get in touch with your prospective mentor
ASAP. Agree upon a communication channel and some response times. Put
yourself in the loop for project news and discussions whatever that means in
the context of your organization: In Debian s case this boiled down to
subscribing to a bunch of mailing lists and IRC channels. Also make sure to
setup a functioning development environment if you haven t done so for
writing the proposal already.
Payoneer setup
The by far most annoying part of GSoC for me. But since you don t have a
choice if you want to get the stipend, you will need to signup for an
account at Payoneer.
In this iteration of GSoC all participants got a personalized link to open a
Payoneer account. When I tried to open an account by following this link, I
got an email after the registration and email verification that my
account is being blocked because Payoneer deems the email adress I gave a
temporary one. Well, the email in question is most certainly anything but
temporary, so I tried to get in touch with the Payoneer support - and ended
up in an LLM-infused kafkaesque support hell. Emails are answered by an LLM
which for me meant utterly off-topic replies and no help whatsoever. The
Payoneer website offers a real-time chat, but it is yet another instance of
a bullshit-spewing LLM bot. When I at last tried to call them (the
support lines are not listed on the Payoneer website but were provided by
the GSoC program), I kid you not, I was being told that their platform is
currently suffering from technical problems and was hung up on. Only thanks
to the swift and helpful support of the GSoC administrators (who get
priority support from Payoneer) I was able to setup a Payoneer account in
the end.
Apart from showing no respect to customers, Payoneer is also ripping them
off big time with fees (unless you get paid in USD). They charge you 2% for
currency conversions to EUR on top of the FX spread they take. What
worked for me to avoid all of those fees, was to open a USD account at Wise
and have Payoneer transfer my GSoC stipend in USD to that account. Then I
exchanged the USD to my local currency at Wise for significantly less than
Payoneer would have charged me. Also make sure to close your Payoneer
account after the end of GSoC to avoid their annual fee.
Project work
With all this prelude out of the way, I can finally get to the actual work
I ve been doing over the course of my GSoC project.
Background
The upstream rsync project generally sees little development. Nonetheless,
they released version 3.4.0 including some CVE fixes earlier
this year. Unfortunately, their changes broke the -H
flag. Now, Debian package maintainers need to apply those security fixes to
the package versions in the Debian repositories; and those are typically a
bit older. Which usually means that the patches cannot be applied as is but
will need some amendments by the Debian maintainers. For these cases it is
helpful to have autopkgtests defined, which check the package s
functionality in an automated way upon every build.
The question then is, why should the tests not be written upstream such that
regressions are caught in the development rather than the distribution
process? There s a lot to say on this question and it probably depends a lot
on the package at hand, but for rsync the main benefits are twofold:
The upstream project mocks the ssh connection over which rsync is most
typically used. Mocking is better than nothing but not the real thing. In
addition to being a more realisitic test scenario for the typical rsync
use case, involving an ssh server in the test would automatically extend
the overall resilience of Debian packages as now new versions of the
openssh-server package in Debian benefit from the test cases in the
rsync reverse dependency.
The upstream rsync test framework is somewhat idiosyncratic and
difficult to port to reimplementations of rsync. Given that the
original rsync upstream sees little development, an extensive test suit
further downstream can serve as a threshold for drop-in replacements for
rsync.
Goal(s)
At the start of the project, the Debian rsync package was just running (a
part of) the upstream tests as autopkgtests. The relevant snippet from the
build log for the rsync_3.4.1+ds1-3 package reads:
Samuel and I agreed that it would be a good first milestone to make the
skipped tests run. Afterwards, I should write some rsync test cases for
local calls, i.e. without an ssh connection, effectively using rsync as
a more powerful cp. And once that was done, I should extend the tests such
that they run over an active ssh connection.
With these milestones, I went to work.
Upstream tests
Running the seven skipped upstream tests turned out to be fairly
straightforward:
Two upstream tests concern access control lists and extended
filesystem attributes. For these tests to run they rely on
functionality provided by the acl and xattr Debian packages. Adding
those to the Build-Depends list in the debian/control file of the
rsync Debian package repo made them run.
Four upstream tests required root privileges to run. The autopkgtest
tool knows the needs-root restriction for that reason. However, Samuel
and I agreed that the tests should not exclusively run with root
privileges. So, instead of just adding the restiction to the existing
autopkgtest test, we created a new one which has the needs-root
restriction and runs the upstream-tests-as-root script - which is
nothing else than a symlink to the existing upstream-tests script.
The commits to implement these changes can be found in this merge
request.
The careful reader will have noticed that I only made 2 + 4 = 6 upstream
test cases run out of 7: The leftover upstream test is checking the
functionality of the --ctimes rsync option. In the context of Debian, the
problem is that the Linux kernel doesn t have a syscall to set the creation
time of a file. As long as that is the case, this test will always be
skipped for the Debian package.
Local tests
When it came to writing Debian specific test cases I started of a completely
clean slate. Which is a blessing and a curse at the same time: You have full
flexibility but also full responsibility.
There were a few things to consider at this point in time:
Which language to write the tests in?
The programming language I am most proficient in is Python. But testing a
CLI tool in Python would have been weird: it would have meant that I d
have to make repeated subprocess calls to run rsync and then read from
the filesystem to get the file statistics I want to check.
Samuel suggested I stick with shell scripts and make use of diffoscope -
one of the main tools used and maintained by the Reproducible Builds
project - to check whether the file contents and
file metadata are as expected after rsync calls. Since I did not have
good reasons to use bash, I ve decided to write the scripts to be POSIX
compliant.
How to avoid boilerplate? If one makes use of a testing framework, which
one?
Writing the tests would involve quite a bit of boilerplate, mostly related
to giving informative output on and during the test run, preparing the
file structure we want to run rsync on, and cleaning the files up after
the test has run. It would be very repetitive and in violation of DRY
to have the code for this appear in every test. Good testing frameworks
should provide convenience functions for these tasks. shunit2 comes with
those functions, is packaged for Debian, and given that it is already
being used in the curl project, I decided to go with it.
Do we use the same directory structure and files for every test or should
every test have an individual setup?
The tradeoff in this question being test isolation vs. idiosyncratic
code. If every test has its own setup, it takes a) more work to write the
test and b) more work to understand the differences between
tests. However, one can be sure that changes to the setup in one test will
have no side effects on other tests. In my opinion, this guarantee was
worth the additional effort in writing/reading the tests.
Having made these decisions, I simply started writing
tests and ran into issues very quickly.
rsync and subsecond mtime diffs
When testing the rsync --times option, I observed a weird phenomenon: If
the source and destination file have modification times which differ only in
the nanoseconds, an rsync --times call will not synchronize the
modification times. More details about this behavior and examples can be
found in the upstream issue I raised. In the Debian tests we
had to occasionally work around this by setting the timestamps explicitly
with touch -d.
diffoscope regression
In one test case, I was expecting a difference in the modification times but
diffoscope would not report a diff. After a good amount of time spent
on debugging the problem (my default, and usually correct, assumption is
that something about my code is seriously broken if I run into issues like
that), I was able to show that diffoscope only displayed this behavior in
the version in the unstable suite, not on Debian stable (which I am running
on my development machine).
Since everything pointed to a regression in the diffoscope project and
with diffoscope being written in Python, a language I am familiar with, I
wanted to spend some time investigating (and hopefully fixing) the problem.
Running git bisect on the diffoscope repo helped me in identifying the
commit which introduced the regression: The commit contained an optimization
via an early return for bit-by-bit identical files. Unfortunately, the early
return also caused an explicitly requested metadata comparison (which could
be different between the files) to be skipped.
With a nicely diagnosed issue like that, I was able to
go to a local hackerspace event, where people work on FOSS together for an
evening every month. In a group, we were able to first, write a test which
showcases the broken behavior in the latest diffoscope version, and
second, make a fix to the code such that the same test passes going
forward. All details can be found in this merge request.
shunit2 failures
At some point I had a few autopkgtests setup and passing, but adding a new
one would throw me totally inexplicable errors. After trying to isolate the
problem as much as possible, it turns out that shunit2 doesn t play well
together we the -e shell option. The project mentions this in the release
notes for the 2.1.8 version1, but in my opinion a
constraint this severe should be featured much more prominently, e.g. in the
README.
Tests over an ssh connection
The centrepiece of this project; everything else has in a way only been
preparation for this.
Obviously, the goal was to reuse the previously written local tests in some
way. Not only because lazy me would have less work to do this way, but also
because of a reduced long-term maintenance burden of one rather than two
test sets.
As it turns out, it is actually possible to accomplish that: The
remote-tests script doesn t do much apart from starting an ssh server on
localhost and running the local-tests script with the REMOTE environment
variable set.
The REMOTE environment variable changes the behavior of the local-tests
script in such a way that it prepends "$REMOTE": to the destination of the
rsync invocations. And given that we set REMOTE=rsync@localhost in the
remote-tests script, local-tests copies the files to the exact same
locations as before, just over ssh.
The implementational details for this can be found in this merge
request.
proposed-updates
Most of my development work on the Debian rsync package took place during
the Debian freeze as the release of Debian Trixie is just
around the corner. This means that uploading by Debian Developers (DD) and
Debian Maintainers (DM) to the unstable suite is discouraged as it makes
migrating the packages to testing more difficult for the Debian release
team. If DDs/DMs want to have the package version in unstable migrated to
testing during the freeze they have to file an unblock request.
Samuel has done this twice (1, 2) for my work for Trixie but has
asked me to file the proposed-updates request for current
stable (i.e. Debian Bookworm) myself after I ve backported my
tests to bookworm.
Unfinished business
To run the upstream tests which check access control list and extended file
system attributes functionality, I ve added the acl and xattr packages
to Build-Depends in debian/control. This, however, will only make the
packages available at build time: If Debian users install the rsync
package, the acl and xattr packages will not be installed alongside
it. For that, the dependencies would have to be added to Depends or
Suggests in debian/control. Depends is probably to strong of a relation
since rsync clearly works well in practice without, but adding them to
Suggests might be worthwhile. A decision on this would involve checking,
what happens if rsync is called with the relevant options on a host
machine which has those packages installed, but where the destination
machine lacks them.
Apart from the issue described above, the 15 tests I managed to write are
are a drop in the water in light of the infinitude of rsync
optionsand their combinations. Most glaringly, not all
options of the --archive option are covered separately (which would help
indicating what code path of rsync broke in a regression). To increase the
likelihood of catching regressions with the autopkgtests, the test
coverage should be extended in the future.
Conclusion
Generally, I am happy with my contributions to Debian over the course of my
small GSoC project: I ve created an extensible, easy to understand, and
working autopkgtest setup for the Debian rsync package. There are two
things which bother me, however:
In hindsight, I probably shouldn t have gone with shunit2 as a testing
framework. The fact that it behaves erratically with the -e flag is a
serious drawback for a shell testing framework: You really don t want a
shell command to fail silently and the test to continue running.
As alluded to in the previous section, I m not particularly proud of the
number of tests I managed to write.
On the other hand, finding and fixing the regression in diffoscope - while
derailing me from the GSoC project itself - might have a redeeming quality.
DebConf25
By sheer luck I happened to work on a GSoC project at Debian over a time
period during which the annual Debian conference would take place
close enough to my place of residence. Samuel pointed the
opportunity to attend DebConf out to me during the community bonding period
and since I could make time for the event in my schedule, I signed up.
DebConf was a great experience which - aside from gaining more knowledge
about Debian development - allowed me to meet the actual people usually
hidden behind email adresses and IRC nicks. I can wholeheartedly recommend
attending a DebConf to every interested Debian user!
For those who have missed this year s iteration of the conference, I can
recommend the following recorded talks:
While not featuring as a keynote speaker (understandably so as the newcomer
to Debian community that I am), I could still contribute a bit to the
conference program.
Debian install workshop
Additionally, with so many Debian experts gathering in one place while KDE s
End of 10 campaign is ongoing, I felt it natural to
organize a Debian install workhop. In hindsight I can say
that I underestimated how much work it would be, especially for me who does
not speak a word of French. But although the turnout of people who wanted us
to install Linux on their machines was disappointingly low, it was still
worth it: Not only because the material in the repo can be
helpful to others planning install workshops but also because it was nice to
meet a) the person behind the Debian installer images and b) the
local Brest/Finist re Linux user group as well as the motivated and helpful
people at Infini.
Credits
I want to thank the Open Source team at Google for organizing GSoC: The
highly structured program with a one-to-one mentorship is a great avenue to
start contributing to well established and at times intimidating FOSS
projects. And as much as I disagree with Google s surveillance
capitalist business model, I have to give it to them that the company at
least takes its responsibility for FOSS (somewhat) seriously - unlike many
other businesses which rely on FOSS and choose to freeride of it.
Big thanks to the Debian community! I ve experienced nothing but
friendliness in my interactions with the community.
And lastly, the biggest thanks to my GSoC mentor Samuel Henrique. He has
dealt patiently and competently with all my stupid newbie questions. His
support enabled me to make - albeit small - contributions to Debian. It has
been a pleasure to work with him during GSoC and I m looking forward to
working together with him in the future.
Obviously, I ve only read them after experiencing the problem.
Okay, here s the deal, I pushed my first post on Reimagined Doodle - Alias Command, five years ago on July 8th, 2020. Don t think I ever mentioned that post started out as a Github Gist which I later transferred here seeking a more long-term home on an independent platform.
Writing about writings, motivations, and the blog itself has been a recurring theme here over the years. 123456789
I m unsure how I sustained expressing myself and writing here for this long. Now and then, I go months without any thought of writing, and then all of a sudden I start in bursts with sequential posts one after another. There isn t a pattern per se in topics other than whatever burning question I have at the moment.
So here s to a milestone and then some.
Dietmar Rabich,
Cape Town (ZA), Sea Point, Nachtansicht 2024 1867-70
2,
CC BY-SA 4.0This post was originally published in the Wikimedia Tech blog, authored by Arturo Borrero Gonzalez.Wikimedia Cloud VPS is a service offered by the Wikimedia
Foundation, built using OpenStack and managed by the Wikimedia Cloud Services
team. It provides cloud computing resources for projects related to the
Wikimedia movement, including virtual machines, databases, storage,
Kubernetes, and DNS.
A few weeks ago, in April 2025,
we were finally able to introduce IPv6 to
the cloud virtual network, enhancing the platform s scalability, security, and future-readiness. This is a major
milestone, many years in the making, and serves as an excellent point to take a moment to reflect on the road that got
us here.
There were definitely a number of challenges that needed to be addressed before we could get into IPv6. This post covers the journey to this
implementation.
The Wikimedia Foundation was an early adopter of the OpenStack technology, and the original OpenStack deployment in the
organization dates back to 2011. At that time, IPv6 support was still nascent and had limited implementation across
various OpenStack components.
In 2012, the Wikimedia cloud users formally requested IPv6 support.
When Cloud VPS was originally deployed, we had set up the network following some of the upstream-recommended patterns:
nova-networks as the engine in charge of the software-defined virtual network
using a flat network topology all virtual machines would share the same network
using a physical VLAN in the datacenter
using Linux bridges to make this physical datacenter VLAN available to virtual machines
using a single virtual router as the edge network gateway, also executing a global egress NAT barring some
exceptions, using what was called dmz_cidr mechanism
In order for us to be able to implement IPv6 in a way that aligned with our architectural goals and operational
requirements, pretty much all the elements in this list would need to change. First of all, we needed to migrate from
nova-networks into Neutron,
a migration effort that started in 2017.
Neutron was the more modern component to implement software-defined networks in OpenStack. To facilitate this
transition, we made the strategic decision to backport certain functionalities from nova-networks into Neutron,
specifically the dmz_cidr mechanism and some egress NAT capabilities.
Once in Neutron, we started to think about IPv6. In 2018 there was an initial attempt to decide on the network CIDR
allocations that Wikimedia Cloud Services would have. This initiative encountered unforeseen challenges
and was subsequently put on hold. We focused on removing the previously
backported nova-networks patches from Neutron.
Between 2020 and 2021, we initiated another
significant network refresh.
We were able to introduce the cloudgw project, as part of a larger effort to rework the Cloud VPS edge network. The new
edge routers allowed us to drop all the custom backported patches we had in Neutron from the nova-networks era,
unblocking further progress. Worth mentioning that the cloudgw router would use nftables as firewalling and NAT engine.
A pivotal decision in 2022 was to
expose the OpenStack APIs to the internet, which
crucially enabled infrastructure management via OpenTofu. This was key in the IPv6 rollout as will be explained later.
Before this, management was limited to Horizon the OpenStack graphical interface or the command-line interface
accessible only from internal control servers.
Later, in 2023, following the OpenStack project s announcement of the deprecation of the neutron-linuxbridge-agent, we
began to seriously consider migrating to the neutron-openvswitch-agent.
This transition would, in turn, simplify the enablement of tenant networks a feature allowing each OpenStack project
to define its own isolated network, rather than all virtual machines sharing a single flat network.
Once we replaced neutron-linuxbridge-agent with neutron-openvswitch-agent, we were ready to migrate virtual machines to
VXLAN. Demonstrating perseverance, we decided to execute the VXLAN migration in conjunction with the IPv6 rollout.
We prepared and tested several things, including the rework of the edge
routing to be based on BGP/OSPF instead of static routing. In 2024 we were ready for the initial attempt to deploy
IPv6, which failed for unknown reasons. There was a full network outage and
we immediately reverted the changes. This quick rollback was feasible due to
our adoption of OpenTofu: deploying IPv6 had
been reduced to a single code change within our repository.
We started an investigation, corrected a few issues, and
increased our network functional testing coverage before trying again. One
of the problems we discovered was that Neutron would enable the enable_snat configuration flag for our main router
when adding the new external IPv6 address.
Finally, in April 2025,
after many years in the making,
IPv6 was successfully deployed.
Compared to the network from 2011, we would have:
Neutron as the engine in charge of the software-defined virtual network
Ready to use tenant-networks
Using a VXLAN-based overlay network
Using neutron-openvswitch-agent to provide networking to virtual machines
A modern and robust edge network setup
Over time, the WMCS team has skillfully navigated numerous challenges to ensure our service offerings consistently meet
high standards of quality and operational efficiency. Often engaging in multi-year planning strategies, we have enabled
ourselves to set and achieve significant milestones.
The successful IPv6 deployment stands as further testament to the team s dedication and hard work over the years. I
believe we can confidently say that the 2025 Cloud VPS represents its most advanced and capable iteration to date.
This post was originally published in the Wikimedia Tech blog, authored by Arturo Borrero Gonzalez.
Welcome to the third report in 2025 from the Reproducible Builds project. Our monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. As usual, however, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website.
Table of contents:
Debian bookworm live images now fully reproducible from their binary packages
Roland Clobus announced on our mailing list this month that all the major desktop variants (ie. Gnome, KDE, etc.) can be reproducibly created for Debian bullseye, bookworm and trixie from their (pre-compiled) binary packages.
Building reproducible Debian live images does not require building from reproducible source code, but this is still a remarkable achievement. Some large proportion of the binary packages that comprise these live images can (and were) built reproducibly, but live image generation works at a higher level. (By contrast, full or end-to-end reproducibility of a bootable OS image will, in time, require both the compile-the-packages the build-the-bootable-image stages to be reproducible.)
Nevertheless, in response, Roland s announcement generated significant congratulations as well as some discussion regarding the finer points of the terms employed: a full outline of the replies can be found here.
The news was also picked up by Linux Weekly News (LWN) as well as to Hacker News.
LWN: Fedora change aims for 99% package reproducibilityLinux Weekly News (LWN) contributor Joe Brockmeier has published a detailed round-up on how Fedora change aims for 99% package reproducibility. The article opens by mentioning that although Debian has been working toward reproducible builds for more than a decade , the Fedora project has now:
progressed far enough that the project is now considering a change proposal for the Fedora 43 development cycle, expected to be released in October, with a goal of making 99% of Fedora s package builds reproducible. So far, reaction to the proposal seems favorable and focused primarily on how to achieve the goal with minimal pain for packagers rather than whether to attempt it.
Over the last few releases, we [Fedora] changed our build infrastructure to make package builds reproducible. This is enough to reach 90%. The remaining issues need to be fixed in individual packages. After this Change, package builds are expected to be reproducible. Bugs will be filed against packages when an irreproducibility is detected. The goal is to have no fewer than 99% of package builds reproducible.
Python adopts PEP standard for specifying package dependencies
Python developer Brett Cannonreported on Fosstodon that PEP 751 was recently accepted. This design document has the purpose of describing a file format to record Python dependencies for installation reproducibility . As the abstract of the proposal writes:
This PEP proposes a new file format for specifying dependencies to enable reproducible installation in a Python environment. The format is designed to be human-readable and machine-generated. Installers consuming the file should be able to calculate what to install without the need for dependency resolution at install-time.
The PEP, which itself supersedes PEP 665, mentions that there are at least five well-known solutions to this problem in the community .
OSS Rebuild real-time validation and tooling improvements
OSS Rebuild aims to automate rebuilding upstream language packages (e.g. from PyPI, crates.io, npm registries) and publish signed attestations and build definitions for public use.
OSS Rebuild is now attempting rebuilds as packages are published, shortening the time to validating rebuilds and publishing attestations.
Aman Sharma contributed classifiers and fixes for common sources of non-determinism in JAR packages.
Improvements were also made to some of the core tools in the project:
timewarp for simulating the registry responses from sometime in the past.
proxy for transparent interception and logging of network activity.
SimpleX Chat server components now reproducible
SimpleX Chat is a privacy-oriented decentralised messaging platform that eliminates user identifiers and metadata, offers end-to-end encryption and has a unique approach to decentralised identity. Starting from version 6.3, however, Simplex has implemented reproducible builds for its server components. This advancement allows anyone to verify that the binaries distributed by SimpleX match the source code, improving transparency and trustworthiness.
Three new scholarly papers
Aman Sharma of the KTH Royal Institute of Technology of Stockholm, Sweden published a paper on Build and Runtime Integrity for Java (PDF). The paper s abstract notes that Software Supply Chain attacks are increasingly threatening the security of software systems and goes on to compare build- and run-time integrity:
Build-time integrity ensures that the software artifact creation process, from source code to compiled binaries, remains untampered. Runtime integrity, on the other hand, guarantees that the executing application loads and runs only
trusted code, preventing dynamic injection of malicious components.
The recently mandated software bill of materials (SBOM) is intended to help mitigate software supply-chain risk. We discuss extensions that would enable an SBOM to serve as a basis for making trust assessments thus also serving as a proactive defense.
A full PDF of the paper is available.
Lastly, congratulations to Giacomo Benedetti of the University of Genoa for publishing their PhD thesis. Titled Improving Transparency, Trust, and Automation in the Software Supply Chain, Giacomo s thesis:
addresses three critical aspects of the software supply chain to enhance security: transparency, trust, and automation. First, it investigates transparency as a mechanism to empower developers with accurate and complete insights into the software components integrated into their applications. To this end, the thesis introduces SUNSET and PIP-SBOM, leveraging modeling and SBOMs (Software Bill of Materials) as foundational tools for transparency and security. Second, it examines software trust, focusing on the effectiveness of reproducible builds in major ecosystems and proposing solutions to bolster their adoption. Finally, it emphasizes the role of automation in modern software management, particularly in ensuring user safety and application reliability. This includes developing a tool for automated security testing of GitHub Actions and analyzing the permission models of prominent platforms like GitHub, GitLab, and BitBucket.
Debian developer Simon Josefsson published two reproducibility-related blog posts this month. The first was on the topic of Reproducible Software Releases which discusses some techniques and gotchas that can be encountered when generating reproducible source packages ie. ensuring that the source code archives that open-source software projects release can be reproduced by others. Simon s second post builds on his earlier experiments with reproducing parts of Trisquel/Debian. Titled On Binary Distribution Rebuilds, it discusses potential methods to bootstrap a binary distribution like Debian from some other bootstrappable environment like Guix.
Jochen Sprickerhof uploaded sbuild version 0.88.5 with a change relevant to reproducible builds: specifically, the build_as_root_when_needed functionality still supports older versions of dpkg(1). []
The IzzyOnDroid Android APK repository reached another milestone in March, crossing the 40% coverage mark specifically, more than 42% of the apps in the repository is now reproducible
Thanks to funding by NLnet/Mobifree, the project was also to put more
time into their tooling. For instance, developers can now run easily their own verification builder in less than 5 minutes . This currently supports Debian-based systems, but support for RPM-based systems is incoming. Future work in the pipeline, including documentation, guidelines and helpers for debugging.
Fedora developer Zbigniew J drzejewski-Szmek announced a work-in-progress script called fedora-repro-build which attempts to reproduce an existing package within a Koji build environment. Although the project s README file lists a number of fields will always or almost always vary (and there are a non-zero list of other known issues), this is an excellent first step towards full Fedora reproducibility (see above for more information).
Lastly, in openSUSE news, Bernhard M. Wiedemann posted another monthly update for his work there.
[What] would it take to compromise an entire Linux distribution directly through their public infrastructure? Is it possible to perform such a compromise as simple security researchers with no available resources but time?
diffoscope & strip-nondeterminismdiffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 290, 291, 292 and 293 and 293 to Debian:
Bug fixes:
file(1) version 5.46 now returns XHTML document for .xhtml files such as those found nested within our .epub tests. []
Also consider .aar files as APK files, at least for the sake of diffoscope. []
Require the new, upcoming, version of file(1) and update our quine-related testcase. []
Codebase improvements:
Ensure all calls to our_check_output in the ELF comparator have the potential CalledProcessError exception caught. [][]
Correct an import masking issue. []
Add a missing subprocess import. []
Reformat openssl.py. []
Update copyright years. [][][]
In addition, Ivan Trubach contributed a change to ignore the st_size metadata entry for directories as it is essentially arbitrary and introduces unnecessary or even spurious changes. []
Website updates
Once again, there were a number of improvements made to our website this month, including:
Herv Boutemy updated the JVM documentation to clarify that the target is rebuild attestation. []
Lastly, Holger Levsen added Julien Malka and Zbigniew J drzejewski-Szmek to our Involved people [][] as well as replaced suggestions to follow us on Twitter/X to follow us on Mastodon instead [][].
Reproducibility testing framework
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In March, a number of changes were made by Holger Levsen, including:
And finally, node maintenance was performed by Holger Levsen [][][] and Mattia Rizzolo [][].
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via: