Welcome to the August 2025 report from the Reproducible Builds project!
Welcome to the latest report from the Reproducible Builds project for August 2025. These monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. If you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website.
In this report:
Please join us at the upcoming Reproducible Builds Summit, set to take place from October 28th 30th 2025 in Vienna, Austria!**
We are thrilled to host the eighth edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin, Hamburg and Athens. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort.
During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving.
If you re interesting in joining us this year, please make sure to read the event page which has more details about the event and location. Registration is open until 20th September 2025, and we are very much looking forward to seeing many readers of these reports there!
WHY2025 (What Hackers Yearn) is a nonprofit outdoors hacker camp that takes place in Geestmerambacht in the Netherlands (approximately 40km north of Amsterdam). The event is organised for and by volunteers from the worldwide hacker community, and knowledge sharing, technological advancement, experimentation, connecting with your hacker peers, forging friendships and hacking are at the core of this event .
At this year s event, Frans Faase gave a talk on live-bootstrap, an attempt to provide a reproducible, automatic, complete end-to-end bootstrap from a minimal number of binary seeds to a supported fully functioning operating system .
Frans talk is available to watch on video and his slides are available as well.
Jens Dietrich of the Victoria University of Wellington, New Zealand and Behnaz Hassanshahi of Oracle Labs, Australia published an article this month entitled DALEQ Explainable Equivalence for Java Bytecode which explores the options and difficulties when Java binaries are not identical despite being from the same sources, and what avenues are available for proving equivalence despite the lack of bitwise correlation:
[Java] binaries are often not bitwise identical; however, in most cases, the differences can be attributed to variations in the build environment, and the binaries can still be considered equivalent. Establishing such equivalence, however, is a labor-intensive and error-prone process.Jens and Behnaz therefore propose a tool called DALEQ, which:
disassembles Java byte code into a relational database, and can normalise this database by applying Datalog rules. Those databases can then be used to infer equivalence between two classes. Notably, equivalence statements are accompanied with Datalog proofs recording the normalisation process. We demonstrate the impact of DALEQ in an industrial context through a large-scale evaluation involving 2,714 pairs of jars, comprising 265,690 class pairs. In this evaluation, DALEQ is compared to two existing bytecode transformation tools. Our findings reveal a significant reduction in the manual effort required to assess non-bitwise equivalent artifacts, which would otherwise demand intensive human inspection. Furthermore, the results show that DALEQ outperforms existing tools by identifying more artifacts rebuilt from the same code as equivalent, even when no behavioral differences are present.Jens also posted this news to our mailing list.
Tails developer intrigeri has tracked and followed a reproducibility regression in the generation of AppArmor policy caches, and has identified an issue with the 4.1.0 version of AppArmor.
Although initially tracked on the Tails issue tracker, intrigeri filed an issue on the upstream bug tracker. AppArmor developer John Johansen replied, confirming that they can reproduce the issue and went to work on a draft patch. Through this, John revealed that it was caused by an actual underlying security bug in AppArmor that is to say, it resulted in permissions not (always) matching what the policy intends and, crucially, not merely a cache reproducibility issue.
Work on the fix is ongoing at time of writing.
Rust Clippy is a linting tool for the Rust programming language. It provides a collection of lints (rules) designed to identify common mistakes, stylistic issues, potential performance problems and unidiomatic code patterns in Rust projects. This month, however, Sosth ne Gu don filed a new issue in the GitHub requesting a new check that would lint against non deterministic operations in proc-macros, such as iterating over a HashMap .
In Debian this month:
armhf architecture. From July 2015, Vagrant Cascadian has been hosting a zoo of approximately 35 armhf systems which were used for building Debian packages for that architecture.strip-nondeterminism, our program that improves reproducibility by stripping out non-deterministic information such as timestamps or other elements introduced during packaging. This new version, 1.14.2-1, adds some metadata to aid the deputy tool. ( #1111947)
duperemove package in Debian, which appears to be an issue where the order the object files are linked together is dependent on the underlying filesystem . Chris Lamb provided a detailed analysis, including a suggestion that this can be resolved by adding a locale-agnostic sort.
Lastly, Bernhard M. Wiedemann posted another openSUSE monthly update for their work there.
diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions, 303, 304 and 305 to Debian:
sed(1) backreferences when generating debian/tests/control to avoid duplicating ourselves. [ ]mono-utils dependency to versioned mono-devel mono-utils dependency, taking care to maintain the [!riscv64] architecture restriction. [ ]sed over awk to avoid mangling dependency lines containing = (equals) symbols such as version restrictions. [ ]systemd-ukify version 258~rc3. [ ].class on the filesystem before passing them to javap(1). [ ]jsondiff on files over 100KiB as the algorithm runs in O(n^2) time. [ ]>= 3. [ ]None [ ] and Zbigniew J drzejewski-Szmek fixed compatibility with RPM 6 [ ]. Lastly, John Sirois fixed a missing requests dependency in the trydiffoscope tool. [ ]
Once again, there were a number of improvements made to our website this month including:
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In August, however, a number of changes were made by Holger Levsen, including:
o4 node again in order to speed up testing. [ ][ ][ ][ ]trixie-proposed-updates and trixie-updates etc. [ ][ ]amd64-pull184 pseudo-architecture. [ ][ ]debrebuild without the --cache argument to experiment with a new version of devscripts. [ ][ ][ ]osuosl5 or ionos7 nodes are down. [ ]ppc64el architecture nodes when down. [ ]arm64 nodes to Debian trixie in anticipation of the release. [ ][ ]riscv64 architecture. [ ]armhf architecture; many thanks to Vagrant for physically hosting the nodes for ten years. [ ][ ]armhf architecture. [ ]i386 and armhfr from JSON results. [ ]openwrt_rebuilder jobs. [ ]qa-jenkins-dev@alioth-lists.debian.net as the contact for jenkins.debian.net. [ ]reproducible-builds.org/vienna25 to reproducible-builds.org/vienna2025. [ ]
megacli RAID controller requires packages from Debian bookworm. [ ]deb822 Python module in favour of debian.deb822 in the bin/reproducible_scheduler.py script [ ] and removed a note on reproduce.debian.net note after the release of Debian trixie [ ].
#reproducible-builds on irc.oftc.net.
rb-general@lists.reproducible-builds.org
In July I attended
DebCamp and DebConf in Brest,
France. I very much enjoyed the opportunity to reconnect with other
Debian contributors in person. I had a number of interesting and
fruitful conversations there, besides the formally organised BoFs and
talks.
I also gave my own talk on
What s new in the Linux kernel (and what s missing in Debian).
Here s the usual categorisation of activity:
| CPU | Single Thread | Multi Thread | TDP |
|---|---|---|---|
| E5-2683 v4 | 1,713 | 17,591 | 120W |
| Xeon Gold 5120 | 1,755 | 18,251 | 105W |
| i9-9900K | 2,919 | 18,152 | 95W |
| E5-2697A v4 | 2,106 | 21,610 | 145W |
| E5-2699A v4 | 2,055 | 26,939 | 145W |
| W-3265 | 2,572 | 30,105 | 205W |
| W-2295 | 2,642 | 30,924 | 165W |
| i9-10980XE | 2,662 | 32,397 | 165W |
| Xeon Gold 6258R | 2,080 | 40,252 | 205W |
Every good work of software starts by scratching a developer s personal itch.
rsync project generally sees little development. Nonetheless,
they released version 3.4.0 including some CVE fixes earlier
this year. Unfortunately, their changes broke the -H
flag. Now, Debian package maintainers need to apply those security fixes to
the package versions in the Debian repositories; and those are typically a
bit older. Which usually means that the patches cannot be applied as is but
will need some amendments by the Debian maintainers. For these cases it is
helpful to have autopkgtests defined, which check the package s
functionality in an automated way upon every build.
The question then is, why should the tests not be written upstream such that
regressions are caught in the development rather than the distribution
process? There s a lot to say on this question and it probably depends a lot
on the package at hand, but for rsync the main benefits are twofold:
rsync is most
typically used. Mocking is better than nothing but not the real thing. In
addition to being a more realisitic test scenario for the typical rsync
use case, involving an ssh server in the test would automatically extend
the overall resilience of Debian packages as now new versions of the
openssh-server package in Debian benefit from the test cases in the
rsync reverse dependency.rsync test framework is somewhat idiosyncratic and
difficult to port to reimplementations of rsync. Given that the
original rsync upstream sees little development, an extensive test suit
further downstream can serve as a threshold for drop-in replacements for
rsync.rsync package was just running (a
part of) the upstream tests as autopkgtests. The relevant snippet from the
build log for the rsync_3.4.1+ds1-3 package reads:
114s ------------------------------------------------------------
114s ----- overall results:
114s 36 passed
114s 7 skipped
Samuel and I agreed that it would be a good first milestone to make the
skipped tests run. Afterwards, I should write some rsync test cases for
local calls, i.e. without an ssh connection, effectively using rsync as
a more powerful cp. And once that was done, I should extend the tests such
that they run over an active ssh connection.
With these milestones, I went to work.
acl and xattr Debian packages. Adding
those to the Build-Depends list in the debian/control file of the
rsync Debian package repo made them run.autopkgtest
tool knows the needs-root restriction for that reason. However, Samuel
and I agreed that the tests should not exclusively run with root
privileges. So, instead of just adding the restiction to the existing
autopkgtest test, we created a new one which has the needs-root
restriction and runs the upstream-tests-as-root script - which is
nothing else than a symlink to the existing upstream-tests script.2 + 4 = 6 upstream
test cases run out of 7: The leftover upstream test is checking the
functionality of the --ctimes rsync option. In the context of Debian, the
problem is that the Linux kernel doesn t have a syscall to set the creation
time of a file. As long as that is the case, this test will always be
skipped for the Debian package.
subprocess calls to run rsync and then read from
the filesystem to get the file statistics I want to check.
Samuel suggested I stick with shell scripts and make use of diffoscope -
one of the main tools used and maintained by the Reproducible Builds
project - to check whether the file contents and
file metadata are as expected after rsync calls. Since I did not have
good reasons to use bash, I ve decided to write the scripts to be POSIX
compliant.
rsync on, and cleaning the files up after
the test has run. It would be very repetitive and in violation of DRY
to have the code for this appear in every test. Good testing frameworks
should provide convenience functions for these tasks. shunit2 comes with
those functions, is packaged for Debian, and given that it is already
being used in the curl project, I decided to go with it.
rsync and subsecond mtime diffs
When testing the rsync --times option, I observed a weird phenomenon: If
the source and destination file have modification times which differ only in
the nanoseconds, an rsync --times call will not synchronize the
modification times. More details about this behavior and examples can be
found in the upstream issue I raised. In the Debian tests we
had to occasionally work around this by setting the timestamps explicitly
with touch -d.
diffoscope regression
In one test case, I was expecting a difference in the modification times but
diffoscope would not report a diff. After a good amount of time spent
on debugging the problem (my default, and usually correct, assumption is
that something about my code is seriously broken if I run into issues like
that), I was able to show that diffoscope only displayed this behavior in
the version in the unstable suite, not on Debian stable (which I am running
on my development machine).
Since everything pointed to a regression in the diffoscope project and
with diffoscope being written in Python, a language I am familiar with, I
wanted to spend some time investigating (and hopefully fixing) the problem.
Running git bisect on the diffoscope repo helped me in identifying the
commit which introduced the regression: The commit contained an optimization
via an early return for bit-by-bit identical files. Unfortunately, the early
return also caused an explicitly requested metadata comparison (which could
be different between the files) to be skipped.
With a nicely diagnosed issue like that, I was able to
go to a local hackerspace event, where people work on FOSS together for an
evening every month. In a group, we were able to first, write a test which
showcases the broken behavior in the latest diffoscope version, and
second, make a fix to the code such that the same test passes going
forward. All details can be found in this merge request.
shunit2 failures
At some point I had a few autopkgtests setup and passing, but adding a new
one would throw me totally inexplicable errors. After trying to isolate the
problem as much as possible, it turns out that shunit2 doesn t play well
together we the -e shell option. The project mentions this in the release
notes for the 2.1.8 version1, but in my opinion a
constraint this severe should be featured much more prominently, e.g. in the
README.
remote-tests script doesn t do much apart from starting an ssh server on
localhost and running the local-tests script with the REMOTE environment
variable set.
The REMOTE environment variable changes the behavior of the local-tests
script in such a way that it prepends "$REMOTE": to the destination of the
rsync invocations. And given that we set REMOTE=rsync@localhost in the
remote-tests script, local-tests copies the files to the exact same
locations as before, just over ssh.
The implementational details for this can be found in this merge
request.
rsync package took place during
the Debian freeze as the release of Debian Trixie is just
around the corner. This means that uploading by Debian Developers (DD) and
Debian Maintainers (DM) to the unstable suite is discouraged as it makes
migrating the packages to testing more difficult for the Debian release
team. If DDs/DMs want to have the package version in unstable migrated to
testing during the freeze they have to file an unblock request.
Samuel has done this twice (1, 2) for my work for Trixie but has
asked me to file the proposed-updates request for current
stable (i.e. Debian Bookworm) myself after I ve backported my
tests to bookworm.
acl and xattr packages
to Build-Depends in debian/control. This, however, will only make the
packages available at build time: If Debian users install the rsync
package, the acl and xattr packages will not be installed alongside
it. For that, the dependencies would have to be added to Depends or
Suggests in debian/control. Depends is probably to strong of a relation
since rsync clearly works well in practice without, but adding them to
Suggests might be worthwhile. A decision on this would involve checking,
what happens if rsync is called with the relevant options on a host
machine which has those packages installed, but where the destination
machine lacks them.
Apart from the issue described above, the 15 tests I managed to write are
are a drop in the water in light of the infinitude of rsync
options and their combinations. Most glaringly, not all
options of the --archive option are covered separately (which would help
indicating what code path of rsync broke in a regression). To increase the
likelihood of catching regressions with the autopkgtests, the test
coverage should be extended in the future.
autopkgtest setup for the Debian rsync package. There are two
things which bother me, however:
shunit2 as a testing
framework. The fact that it behaves erratically with the -e flag is a
serious drawback for a shell testing framework: You really don t want a
shell command to fail silently and the test to continue running.diffoscope - while
derailing me from the GSoC project itself - might have a redeeming quality.
It's Sunday and I'm now sitting in the train from Brest to Paris where I will be changing to Germany, on the way back from the annual Debian conference. A full week of presentations, discussions, talks and socializing is laying behind me and my head is still spinning from the intensity.
Pollito and the gang of DebConf mascots wearing their conference badges (photo: Christoph Berg)Table of Contents
F/DF7CB in Brest (photo: Evangelos Ribeiro Tzaras)
The northern coast of Ushant (photo: Christoph Berg)
Having a nice day (photo: Christoph Berg)
free command. Over the time and many kernel versions, the summary numbers printed became better and better, but there will probably never be a version that suits all use cases alike. Later over dinner, Craig (who is also a TC member) and I shared our experiences with these numbers and customers (not) understanding them. He pointed out that for PostgreSQL and looking at used memory in the presence of large shared memory buffers, USS (unique set size) and PSS (proportional set size) should be more realistic numbers than the standard RSS (resident set size) that the top utility is showing by default.
Antonio Terceiro and Paul Gevers again joined to lead a session, now on ci.debian.net and autopkgtest, the test driver used for running tests on packages after then have been installed on a system. The PostgreSQL packages are heavily using this to make sure no regressions creep in even after builds have successfully completed and test re-runs are rescheduled periodically. The day ended with Bdale Garbee's electronics team BoF and Paul Tagliamonte and me setting up the radio station in the courtyard, again answering countless questions about ionospheric conditions and operating practice.
Saturday, July 19th
Saturday was the last conference day. In the first session, Nikos Tsipinakis and Federico Vaga from CERN announced that the LHC will be moving to Debian for the accelerator's frontend computers in their next "long shutdown" maintenance period in the next year. CentOS broke compatibility too often, and Debian trixie together with the extended LTS support will cover the time until the next long shutdown window in 2035, until when the computers should have all been replaced with newer processors covering higher x86_64 baseline versions. The audience was very delighted to hear that Debian is now also being used in this prestige project.
Ben Hutchings then presented new Linux kernel features. Particularly interesting for me was the support for atomic writes spanning more than one filesystem block. When configured correctly, this would mean PostgreSQL didn't have to record full-page images in the WAL anymore, increasing throughput and performance. After that, the Debian ftp team discussed ways to improve review of new packages in the archive, and which of their processes could be relaxed with new US laws around Open Source and cryptography algorithms export. Emmanuel Arias led a session on Salsa CI, Debian's Gitlab instance and standard CI pipeline. (I think it's too slow, but the runners are not under their control.) Julian Klode then presented new features in APT, Debian's package manager. I like the new display format (and a tiny bit of that is also from me sending in wishlist bugs).
In the last round of sessions this week, I then led the Ham radio BoF with an introduction into the hobby and how Debian can be used. Bdale mentioned that the sBitx family of SDR radios is natively running Debian, so stock packages can be used from the radio's touch display. We also briefly discussed his involvement in ARDC and the possibility to get grants from them for Ham radio projects. Finally, DebConf wrapped up with everyone gathering in the main auditorium and cheering the organizers for making the conference possible and passing Pollito, the DebConf mascot, to the next organizer team.
Pollito on stage (photo: Christoph Berg)
![]() |
| Achieving full disk encryption using FIPS, TCG OPAL and LUKS to encrypt UEFI ESP on bare-metal and in VMs |
If one has a lot of estate and a lot of encryption keys to keep track off a key management solution is likely needed. The most popular solution is likely the one from Thales Group marketed under ChiperTrust Data Security Platform (previously Vormetric), but there are many others including OEM / Vendor / Hardware / Cloud specific or agnostic solutions.
I hope this crash course guide piques your interest to learn and discover modern confidentially and integrity solutions, and to re-affirm or change your existing controls w.r.t. to data protection at rest.
Full disk encryption, including UEFI ESP /boot/efi is now widely achievable by default on both baremetal machines and in VMs including with FIPS certification. To discuss more let's connect on Linkedin.
Update on 2025-07-28: added note about Debian 13/trixie support for OpenVox (thanks, Ben Ford!)
Debian v13 with codename trixie is scheduled to be published as new stable release on 9th of August 2025.
I was the driving force at several of my customers to be well prepared for the upcoming stable release (my efforts for trixie started in August 2024). On the one hand, to make sure packages we care about are available and actually make it into the release. On the other hand, to ensure there are no severe issues that make it into the release and to get proper and working upgrades. So far everything is looking pretty well and working fine, the efforts seemed to have payed off. :)
As usual with major upgrades, there are some things to be aware of, and hereby I m starting my public notes on trixie that might be worth for other folks. My focus is primarily on server systems and looking at things from a sysadmin perspective.
Further readings
As usual start at the official Debian release notes, make sure to especially go through What s new in Debian 13 + issues to be aware of for trixie (strongly recommended read!).
Package versions
As a starting point, let s look at some selected packages and their versions in bookworm vs. trixie as of 2025-07-20 (mainly having amd64 in mind):
| Package | bookworm/v12 | trixie/v13 |
|---|---|---|
| ansible | 2.14.3 | 2.19.0 |
| apache | 2.4.62 | 2.4.64 |
| apt | 2.6.1 | 3.0.3 |
| bash | 5.2.15 | 5.2.37 |
| ceph | 16.2.11 | 18.2.7 |
| docker | 20.10.24 | 26.1.5 |
| dovecot | 2.3.19 | 2.4.1 |
| dpkg | 1.21.22 | 1.22.21 |
| emacs | 28.2 | 30.1 |
| gcc | 12.2.0 | 14.2.0 |
| git | 2.39.5 | 2.47.2 |
| golang | 1.19 | 1.24 |
| libc | 2.36 | 2.41 |
| linux kernel | 6.1 | 6.12 |
| llvm | 14.0 | 19.0 |
| lxc | 5.0.2 | 6.0.4 |
| mariadb | 10.11 | 11.8 |
| nginx | 1.22.1 | 1.26.3 |
| nodejs | 18.13 | 20.19 |
| openjdk | 17.0 | 21.0 |
| openssh | 9.2p1 | 10.0p1 |
| openssl | 3.0 | 3.5 |
| perl | 5.36.0 | 5.40.1 |
| php | 8.2+93 | 8.4+96 |
| podman | 4.3.1 | 5.4.2 |
| postfix | 3.7.11 | 3.10.3 |
| postgres | 15 | 17 |
| puppet | 7.23.0 | 8.10.0 |
| python3 | 3.11.2 | 3.13.5 |
| qemu/kvm | 7.2 | 10.0 |
| rsync | 3.2.7 | 3.4.1 |
| ruby | 3.1 | 3.3 |
| rust | 1.63.0 | 1.85.0 |
| samba | 4.17.12 | 4.22.3 |
| systemd | 252.36 | 257.7-1 |
| unattended-upgrades | 2.9.1 | 2.12 |
| util-linux | 2.38.1 | 2.41 |
| vagrant | 2.3.4 | 2.3.7 |
| vim | 9.0.1378 | 9.1.1230 |
| zsh | 5.9 | 5.9 |
DebConf25, the 26th edition of the Debian
conference is taking place in Brest Campus of
IMT Atlantique Bretagne-Pays de la Loire,
France. We appreciate the organizers for their hard work, and hope this event
will be highly beneficial for those who attend in person as well as online.
This event would not be possible without the help from our generous sponsors.
We would like to warmly welcome the sponsors of DebConf 25, and introduce them to
you.
We have five Platinum sponsors.
x86-64-v3/libfoo0.so would be loaded:
/usr/lib/glibc-hwcaps/x86-64-v4/libfoo0.so /usr/lib/glibc-hwcaps/x86-64-v3/libfoo0.so /usr/lib/glibc-hwcaps/x86-64-v2/libfoo0.so /usr/lib/libfoo0.so
GGML_CPU_ALL_VARIANTS=ON option, but this was limited to amd64.
This meant that on all the other architectures that Debian supports, I would
need to target some ancient
baseline,
thus effectively crippling the package there.
hwcaps/<level>, starting with the highest <level>
that the current CPU supports. The levels are predefined. I'm using the
amd64 levels
below.
For ggml, this meant that I simply could build the library in multiple passes,
each time targeting a different <level>, and install the result in the
corresponding subdirectory, which resulted in the following
layout (reduced to libggml.so for brevity):
/usr/lib/x86_64-linux-gnu/ggml/glibc-hwcaps/x86-64-v4/libggml.so /usr/lib/x86_64-linux-gnu/ggml/glibc-hwcaps/x86-64-v3/libggml.so /usr/lib/x86_64-linux-gnu/ggml/glibc-hwcaps/x86-64-v2/libggml.so /usr/lib/x86_64-linux-gnu/ggml/libggml.soIn practice, this means that on a CPU supporting AVX512, the linker/loader would load
x86-64-v4/libggml.so if it existed, and otherwise continue to
look for the other levels, all the way down to the lowest one. On a
CPU which supported only SSE4.2, the lookup process would be the same,
ending with picking x86-64-v2/libggml.so.
With QEMU, all of this was quickly
verified.
Note that the lowest-level library, targeting x86-64-v1, is not installed
to a subdirectory, but to the path where the library would normally have been
installed. This has the nice property that on systems not using
GLIBC, and thus not having hwcaps available, package installation will still
result in a loadable library, albeit the version with the worst performance.
And a careful observer might have noticed that in the example above, the
library is installed to a private ggml/ directory, so this mechanism also
works when using RUNPATH or LD_LIBRARY_PATH.
As mentioned above, Debian's ggml package will soon switch to
GGML_CPU_ALL_VARIANTS=ON, but this was still quite the useful feature to
discover.
AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers settingFor starters, it means I have to relearn how
MaxRequestWorkers functions in Apache:
For threaded and hybrid servers (e.g. event or worker), MaxRequestWorkers restricts the total number of threads that will be available to serve clients. For hybrid MPMs, the default value is 16 (ServerLimit) multiplied by the value of 25 (ThreadsPerChild). Therefore, to increase MaxRequestWorkers to a value that requires more than 16 processes, you must also raise ServerLimit.Ok remind me what
ServerLimit refers to?
For the prefork MPM, this directive sets the maximum configured value for MaxRequestWorkers for the lifetime of the Apache httpd process. For the worker and event MPMs, this directive in combination with ThreadLimit sets the maximum configured value for MaxRequestWorkers for the lifetime of the Apache httpd process. For the event MPM, this directive also defines how many old server processes may keep running and finish processing open connections. Any attempts to change this directive during a restart will be ignored, but MaxRequestWorkers can be modified during a restart. Special care must be taken when using this directive. If ServerLimit is set to a value much higher than necessary, extra, unused shared memory will be allocated. If both ServerLimit and MaxRequestWorkers are set to values higher than the system can handle, Apache httpd may not start or the system may become unstable. With the prefork MPM, use this directive only if you need to set MaxRequestWorkers higher than 256 (default). Do not set the value of this directive any higher than what you might want to set MaxRequestWorkers to. With worker, use this directive only if your MaxRequestWorkers and ThreadsPerChild settings require more than 16 server processes (default). Do not set the value of this directive any higher than the number of server processes required by what you may want for MaxRequestWorkers and ThreadsPerChild. With event, increase this directive if the process number defined by your MaxRequestWorkers and ThreadsPerChild settings, plus the number of gracefully shutting down processes, is more than 16 server processes (default).Got it? In other words, you can consider raising the
MaxRequestWorkers
setting all you want, but you can t just change that setting, you have to read
about several other compliated settings, do some math, and spend a lot of time
wondering if you are going to remember what you just did and how to undo it if
you blow up your server.
On the plus side, typically, nobody should increase this limit - because if the
server runs out of connections, it usually means something else is wrong.
In our case, on a shared web server running Apache2 and PHP-FPM, it s usually
because a single web site has gone out of control.
But wait! How can that happen, we are using PHP-FPM s max_children setting to prevent
a single PHP web site from taking down the server?
After years of struggling with this problem I have finally made some headway.
Our PHP pool configuration typically looks like this:
user = site342999writer
group = site342999writer
listen = /run/php/8.1-site342999.sock
listen.owner = www-data
listen.group = www-data
pm = ondemand
pm.max_children = 12
pm.max_requests = 500
php_admin_value[memory_limit] = 256M
<FilesMatch \.php$>
SetHandler "proxy:unix:/var/run/php/8.1-site342999.sock fcgi://localhost"
</FilesMatch>
max_children?
According to the docs:
By default, mod_proxy will allow and retain the maximum number of connections
that could be used simultaneously by that web server child process. Use the
max parameter to reduce the number from the default. The pool of connections
is maintained per web server child process, and max and other settings are
not coordinated among all child processes, except when only one child process
is allowed by configuration or MPM design.
The max parameter seems to default to the ThreadsPerChild, so it seems that
the default here is to allow any web site to consume ThreadsPerChild (25) x
ServerLimit (16), which is also the max number of over all connections. Not
great.
To make matter worse, there is another setting available which is mysteriously
called acquire:
If set, this will be the maximum time to wait for a free connection in the
connection pool, in milliseconds. If there are no free connections in the
pool, the Apache httpd will return SERVER_BUSY status to the client.
By default this is not set which seems to suggest Apache will just hang on to
connections forever until a free PHP process becomes available (or some other
time out happens).
So, let s try something different:
<Proxy "fcgi://localhost">
ProxySet acquire=1 max=12
</proxy>
SetHandler statement above. It s documented on the Apache mod_proxy
page.
Now we limit the maximum pool size per process to half of what is available for
the entire server and we tell Apache to immediately throw a 503 error if we
have exceeded our maximum number of connecitons.
Now, if a site is overwhelmed with traffic, instead of maxing out the available
Apache connections while leaving user with constantly spinning browsers, the
users will get 503 ed and the server will be able to server other sites.
$ dig ns nl +short
ns1.dns.nl.
ns3.dns.nl.
ns4.dns.nl.
ns1.dns.nl is SIDN which also manages their registry.
ns3.dns.nl is ReCodeZero/ipcom, another anycast secondary.
ns4.dns.nl is CIRA, anycast secondary. That s 3 diverse, anycast networks to serve the .NL ccTLD. .DE has a bit more at name servers at 6 but only 3 seems anycasted.
Now let s take a look at .UA. Hostmaster LLC is the registry operator of the .UA ccTLD since 2001.
$ dig soa ua +short
in1.ns.ua. domain-master.cctld.ua. 2025061434 1818 909 3024000 2020
Shows in1.ns.ua as primary nameserver (which can be intentionally deceptive too).
I used bgp.tools for checking anycast and dns.coffee for timeline of when secondary nameserver was added. dns.coffee only has data going back till 2011 though.
Let s deep dive at who s hosting each of the name servers:
Like each month, have a look at the work funded by Freexian s Debian LTS offering.
(Edit: Twitter could improve this significantly with very few changes - I wrote about that here. It's unclear why they'd launch without doing that, since it entirely defeats the point of using HSMs)built on Rust with (Bitcoin style) encryption, whole new architecture. Maybe this time they've got it right?
Python, Golang and Cloud teams, packaging dependencies and maintaining various tools. However, I soon felt the need to focus on packaging software I truly enjoyed, tools I was passionate about using and maintaining.
That s when I turned my attention to Kubernetes within Debian.
1.20.5.
Due to this abandonment, critical bugs emerged and the package was removed from Debian s testing channel, as we can see in the package tracker.
kubectx, kubernetes-split-yaml and kubetail into a dedicated namespace on Salsa, Debian s GitLab instance.
Many of these tools were stored across different teams (like the Go team), and consolidating them helped us organize development and focus our efforts.
kubectl, Kubernetes CLI.Files-Excluded in debian/copyright to cleanly drop unneeded files during source imports.uscan, a standard Debian packaging tool that fetches upstream tarballs and prepares them accordingly. The Files-Excluded directive in our debian/copyright file instructed uscan to automatically remove unnecessary files during the repackaging process:
$ uscan
Newest version of kubernetes on remote site is 1.32.3, specified download version is 1.32.3
Successfully repacked ../v1.32.3 as ../kubernetes_1.32.3+ds.orig.tar.gz, deleting 30616 files from it.
75%:
$ du -h upstream-v1.32.3.tar.gz kubernetes_1.32.3+ds.orig.tar.gz
14M upstream-v1.32.3.tar.gz
3.2M kubernetes_1.32.3+ds.orig.tar.gz
30,000 files, we simplified the package, making it more maintainable. Each dependency could now be properly tracked, updated, and patched independently, resolving the security concerns that had plagued the previous packaging approach.
debtree, visualizing all the Go modules and other dependencies required to build the kubectl binary.
This web of nodes and edges represents every module and its relationship during the compilation process of kubectl. Each box is a Debian package, and the lines connecting them show how deeply intertwined the ecosystem is. What might look like a mess of blue spaghetti is actually a clear demonstration of the vast and interconnected upstream world that tools like kubectl rely on.
But more importantly, this graph is a testament to the effort that went into making kubectl build entirely using Debian-packaged dependencies only, no vendoring, no downloading from the internet, no proprietary blobs.
1.32.3+ds of kubectl to Debian unstable.
kubernetes/-/merge_requests/1
Zsh, Fish, and Bash completions installed automaticallyMan pages and metadata for improved discoverabilitykind and docker for testing purposes
Here s what the data tells us:
kubectl (new binary): Already installed on 2,124 systems.golang-k8s-kubectl-dev: This is the Go development package (a library), useful for other packages and developers who want to interact with Kubernetes programmatically.kubernetes-client: The legacy package that kubectl is replacing. We expect this number to decrease in future releases as more systems transition to the new package.Also worth mentioning: this number is not the real total number of installations, since users can choose not to participate in the popularity contest. So the actual adoption is likely higher than what popcon reflects.
kubectl version 1.32.3, built from a clean, de-vendorized source. This version includes nearly all the latest upstream features, and will be the first time in years that Debian users can rely on an up-to-date, policy-compliant kubectl directly from the archive.
By comparing with upstream, our Debian package even delivers more out of the box, including shell completions, which the upstream still requires users to generate manually.
In 2025, the Debian Kubernetes team will continue expanding our packaging efforts for the Kubernetes ecosystem.
Our roadmap includes:
kubeadm in Debian, users will then be able to bootstrap minimum viable clusters directly from the official repositories.
Python, Golang and Cloud teams, packaging dependencies and maintaining various tools. However, I soon felt the need to focus on packaging software I truly enjoyed, tools I was passionate about using and maintaining.
That s when I turned my attention to Kubernetes within Debian.
1.20.5.
Due to this abandonment, critical bugs emerged and the package was removed from Debian s testing channel, as we can see in the package tracker.
kubectx, kubernetes-split-yaml and kubetail into a dedicated namespace on Salsa, Debian s GitLab instance.
Many of these tools were stored across different teams (like the Go team), and consolidating them helped us organize development and focus our efforts.
kubectl, Kubernetes CLI.Files-Excluded in debian/copyright to cleanly drop unneeded files during source imports.uscan, a standard Debian packaging tool that fetches upstream tarballs and prepares them accordingly. The Files-Excluded directive in our debian/copyright file instructed uscan to automatically remove unnecessary files during the repackaging process:
$ uscan
Newest version of kubernetes on remote site is 1.32.3, specified download version is 1.32.3
Successfully repacked ../v1.32.3 as ../kubernetes_1.32.3+ds.orig.tar.gz, deleting 30616 files from it.
75%:
$ du -h upstream-v1.32.3.tar.gz kubernetes_1.32.3+ds.orig.tar.gz
14M upstream-v1.32.3.tar.gz
3.2M kubernetes_1.32.3+ds.orig.tar.gz
30,000 files, we simplified the package, making it more maintainable. Each dependency could now be properly tracked, updated, and patched independently, resolving the security concerns that had plagued the previous packaging approach.
debtree, visualizing all the Go modules and other dependencies required to build the kubectl binary.
This web of nodes and edges represents every module and its relationship during the compilation process of kubectl. Each box is a Debian package, and the lines connecting them show how deeply intertwined the ecosystem is. What might look like a mess of blue spaghetti is actually a clear demonstration of the vast and interconnected upstream world that tools like kubectl rely on.
But more importantly, this graph is a testament to the effort that went into making kubectl build entirely using Debian-packaged dependencies only, no vendoring, no downloading from the internet, no proprietary blobs.
1.32.3+ds of kubectl to Debian unstable.
kubernetes/-/merge_requests/1
Zsh, Fish, and Bash completions installed automaticallyMan pages and metadata for improved discoverabilitykind and docker for testing purposes
Here s what the data tells us:
kubectl (new binary): Already installed on 2,124 systems.golang-k8s-kubectl-dev: This is the Go development package (a library), useful for other packages and developers who want to interact with Kubernetes programmatically.kubernetes-client: The legacy package that kubectl is replacing. We expect this number to decrease in future releases as more systems transition to the new package.Also worth mentioning: this number is not the real total number of installations, since users can choose not to participate in the popularity contest. So the actual adoption is likely higher than what popcon reflects.
kubectl version 1.32.3, built from a clean, de-vendorized source. This version includes nearly all the latest upstream features, and will be the first time in years that Debian users can rely on an up-to-date, policy-compliant kubectl directly from the archive.
By comparing with upstream, our Debian package even delivers more out of the box, including shell completions, which the upstream still requires users to generate manually.
In 2025, the Debian Kubernetes team will continue expanding our packaging efforts for the Kubernetes ecosystem.
Our roadmap includes:
kubeadm in Debian, users will then be able to bootstrap minimum viable clusters directly from the official repositories.
Next.