As DebConf22 was coming to an end, in Kosovo, talking with Eeveelweezel they
invited me to prepare a talk to give for the Chicago Python User
Group. I replied that I m not really that much of a Python
guy But would think about a topic. Two years passed. I meet Eeveelweezel
again for DebConf24 in Busan, South Korea. And the topic came up again. I had
thought of some ideas, but none really pleased me. Again, I do write some Python
when needed, and I teach using Python, as it s the language I find my students
can best cope with. But delivering a talk to ChiPy?
On the other hand, I have long used a very simplistic and limited filesystem
I ve designed as an implementation project at class:
FIUnamFS
(for Facultad de Ingenier a, Universidad Nacional Aut noma de M xico : the
Engineering Faculty for Mexico s National University, where I teach. Sorry, the
link is in Spanish but you will find several implementations of it from the
students ). It is a toy filesystem, with as many bad characteristics you can
think of, but easy to specify and implement. It is based on contiguous file
allocation, has no support for sub-directories, and is often limited to the size
of a 1.44MB floppy disk.
As I give this filesystem as a project to my students (and not as a mere
homework), I always ask them to try and provide a good, polished, professional
interface, not just the simplistic menu I often get. And I tell them the best
possible interface would be if they provide support for FIUnamFS transparently,
usable by the user without thinking too much about it. With high probability,
that would mean: Use FUSE.
But, in the six semesters I ve used this project (with 30-40 students per
semester group), only one student has bitten the bullet and presented a FUSE
implementation.
Maybe this is because it s not easy to understand how to build a FUSE-based
filesystem from a high-level language such as Python? Yes, I ve seen several
implementation examples and even nice web pages (i.e. the examples shipped with
thepython-fuse
moduleStavros
passthrough filesystem,
Dave Filesystem based upon, and further explaining,
Stavros ,
and several others) explaining how to provide basic functionality. I found a
particularly useful presentation by Matteo
Bertozzi presented
~15 years ago at PyCon4 But none of those is IMO followable enough by
itself. Also, most of them are very old (maybe the world is telling me
something that I refuse to understand?).
And of course, there isn t a single interface to work from. In Python only, we
can find
python-fuse,
Pyfuse,
Fusepy Where to start from?
So I setup to try and help.
Over the past couple of weeks, I have been slowly working on my own version, and
presenting it as a progressive set of tasks, adding filesystem calls, and
being careful to thoroughly document what I write (but maybe my documentation
ends up obfuscating the intent? I hope not and, read on, I ve provided some
remediation).
I registered a GitLab project for a hand-holding guide to writing FUSE-based
filesystems in Python. This
is a project where I present several working FUSE filesystem implementations,
some of them RAM-based, some passthrough-based, and I intend to add to this also
filesystems backed on pseudo-block-devices (for implementations such as my
FIUnamFS).
So far, I have added five stepwise pieces, starting from the barest possible
empty
filesystem,
and adding system calls (and functionality) until (so far) either a read-write
filesystem in RAM with basicstat()
support
or a read-only passthrough
filesystem.
I think providing fun or useful examples is also a good way to get students to
use what I m teaching, so I ve added some ideas I ve had: DNS
Filesystem,
on-the-fly markdown compiling
filesystem,
unzip
filesystem
and uncomment
filesystem.
They all provide something that could be seen as useful, in a way that s easy to
teach, in just some tens of lines. And, in case my comments/documentation are
too long to read, uncommentfs will happily strip all comments and whitespace
automatically!
So I will be delivering my talk tomorrow (2024.10.10, 18:30 GMT-6) at
ChiPy (virtually). I am also presenting
this talk virtually at Jornadas Regionales de Software
Libre in Santa Fe,
Argentina, next week (virtually as well). And also in November, in person, at
nerdear.la, that will be held in Mexico City for the
first time.
Of course, I will also share this project with my students in the next couple of
weeks And hope it manages to lure them into implementing FUSE in Python. At
some point, I shall report!
Update: After delivering my ChiPy talk, I have uploaded it to YouTube: A
hand-holding guide to writing FUSE-based filesystems in
Python, and after presenting at Jornadas
Regionales, I present you the video in Spanish here: Aprendiendo y ense ando a
escribir sistemas de archivo en espacio de usuario con FUSE y
Python.
Pydantic v2, by Colin Watson
Pydantic is a useful library for validating data in Python using type hints: Freexian uses it in a number of projects, including Debusine. Its Debian packaging had been stalled at 1.10.17 in testing for some time, partly due to needing to make sure everything else could cope with the breaking changes introduced in 2.x, but mostly due to needing to sort out packaging of its new Rust dependencies. Several other people (notably Alexandre Detiste, Andreas Tille, Drew Parsons, and Timo R hling) had made some good progress on this, but nobody had quite got it over the line and it seemed a bit stuck.
Colin upgraded a few Rust libraries to new upstream versions, packaged rust-jiter, and chased various failures in other packages. This eventually allowed getting current versions of both pydantic-core and pydantic into testing. It should now be much easier for us to stay up to date routinely.
Reworking of glib2.0 for cross bootstrap, by Helmut Grohne
Simon McVittie (not affiliated with Freexian) earlier restructured the libglib2.0-dev such that it would absorb more functionality and in particular provide tools for working with .gir files. Those tools practically require being run for their host architecture (practically this means running under qemu-user) which is at odds with the requirements of architecture cross bootstrap. The qemu requirement was expressed in package dependencies and also made people unhappy attempting to use libglib2.0-dev for i386 on amd64 without resorting to qemu. The use of qemu in architecture bootstrap is particularly problematic as it tends to not be ready at the time bootstrapping is needed.
As a result, Simon proposed and implemented the introduction of a libgio-2.0-dev package providing a subset of libglib2.0-dev that does not require qemu. Packages should continue to use libglib2.0-dev in their Build-Depends unless involved in architecture bootstrap. Helmut reviewed and tested the implementation and integrated the necessary changes into rebootstrap. He also prepared a patch for libverto to use the new package and proposed adding forward compatibility to glib2.0.
Helmut continued working on adding cross-exe-wrapper to architecture-properties and implemented autopkgtests later improved by Simon. The cross-exe-wrapper package now provides a generic mechanism to a program on a different architecture by using qemu when needed only. For instance, a dependency on cross-exe-wrapper:i386 provides a i686-linux-gnu-cross-exe-wrapper program that can be used to wrap an ELF executable for the i386 architecture. When installed on amd64 or i386 it will skip installing or running qemu, but for other architectures qemu will be used automatically. This facility can be used to support cross building with targeted use of qemu in cases where running host code is unavoidable as is the case for GObject introspection.
This concludes the joint work with Simon and Niels Thykier on glib2.0 and architecture-properties resolving known architecture bootstrap regressions arising from the glib2.0 refactoring earlier this year.
Analyzing binary package metadata, by Helmut Grohne
As Guillem Jover (not affiliated with Freexian) continues to work on adding metadata tracking to dpkg, the question arises how this affects existing packages. The dedup.debian.net infrastructure provides an easy playground to answer such questions, so Helmut gathered file metadata from all binary packages in unstable and performed an explorative analysis. Some results include:
Guillem also performed a cursory analysis and reported other problem categories such as mismatching directory permissions for directories installed by multiple packages and thus gained a better understanding of what consistency checks dpkg can enforce.
Python archive rebuilds, by Stefano Rivera
Last month Stefano started to write some tooling to do large-scale rebuilds in debusine, starting with finding packages that had already started to fail to build from source (FTBFS) due to the removal of setup.py test. This month, Stefano did some more rebuilds, starting with experimental versions of dh-python.
During the Python 3.12 transition, we had added a dependency on python3-setuptools to dh-python, to ease the transition. Python 3.12 removed distutils from the stdlib, but many packages were expecting it to still be available. Setuptools contains a version of distutils, and dh-python was a convenient place to depend on setuptools for most package builds. This dependency was never meant to be permanent. A rebuild without it resulted in mass-filing about 340 bugs (and around 80 more by mistake).
A new feature in Python 3.12, was to have unittest s test runner exit with a non-zero return code, if no tests were run. We added this feature, to be able to detect tests that are not being discovered, by mistake. We are ignoring this failure, as we wouldn t want to suddenly cause hundreds of packages to fail to build, if they have no tests. Stefano did a rebuild to see how many packages were affected, and found that around 1000 were. The Debian Python community has not come to a conclusion on how to move forward with this.
As soon as Python 3.13 release candidate 2 was available, Stefano did a rebuild of the Python packages in the archive against it. This was a more complex rebuild than the others, as it had to be done in stages. Many packages need other Python packages at build time, typically to run tests. So transitions like this involve some manual bootstrapping, followed by several rounds of builds. Not all packages could be tested, as not all their dependencies support 3.13 yet. The result was around 100 bugs in packages that need work to support Python 3.13. Many other packages will need additional work to properly support Python 3.13, but being able to build (and run tests) is an important first step.
Miscellaneous contributions
Carles prepared the update of python-pyaarlo package to a new upstream release.
Carles worked on updating python-ring-doorbell to a new upstream release. Unfinished, pending to package a new dependency python3-firebase-messaging RFP #1082958 and its dependency python3-http-ece RFP #1083020.
Carles improved po-debconf-manager. Main new feature is that it can open Salsa merge requests. Aiming for a lightning talk in MiniDebConf Toulouse (November) to be functional end to end and get feedback from the wider public for this proof of concept.
Carles helped one translator to use po-debconf-manager (added compatibility for bullseye, fixed other issues) and reviewed 17 package templates.
Colin upgraded the OpenSSH packaging to 9.9p1.
Colin upgraded the various YubiHSM packages to new upstream versions, enabled more tests, fixed yubihsm-shell build failures on some 32-bit architectures, made yubihsm-shell build reproducibly, and fixed yubihsm-connector to apply udev rules to existing devices when the package is installed. As usual, bookworm-backports is up to date with all these changes.
Colin fixed quite a bit of fallout from setuptools 72.0.0 removing setup.py test, backported a large upstream patch set to make buildbot work with SQLAlchemy 2.0, and upgraded 25 other Python packages to new upstream versions.
Enrico worked with Jakob Haufe to get him up to speed for managing sso.debian.org
Rapha l did remove spam entries in the list of teams on tracker.debian.org (see #1080446), and he applied a few externalcontributions, fixing a rendering issue and replacing the DDPO link with a more useful alternative. He also gave feedback on a couple of mergerequests that required more work. As part of the analysis of the underlying problem, he suggested to the ftpmasters (via #1083068) to auto-reject packages having the too-many-contacts lintian error, and he raised the severity of #1076048 to serious to actually have that 4 year old bug fixed.
Rapha l uploaded zim and hamster-time-tracker to fix issues with Python 3.12 getting rid of setuptools. He also uploaded a new gnome-shell-extension-hamster to cope with the upcoming transition to GNOME 47.
Helmut sent seven patches and sponsored one upload for cross build failures.
Helmut uploaded a Nagios/Icinga plugin check-smart-attributes for monitoring the health of physical disks.
Helmut collaborated on sbuild reviewing and improving a MR for refactoring the unshare backend.
Helmut sent a patch fixing coinstallability of gcc-defaults.
Helmut continued to monitor the evolution of the /usr-move. With more and more key packages such as libvirt or fuse3 fixed. We re moving into the boring long-tail of the transition.
Helmut continued to update patches maintained in rebootstrap. Due to the work on glib2.0 above, rebootstrap moves a lot further, but still fails for any architecture.
Santiago reviewed some Merge Request in Salsa CI, such as: !478, proposed by Otto to extend the information about how to use additional runners in the pipeline and !518, proposed by Ahmed to add support for Ubuntu images, that will help to test how some debian packages, including the complex MariaDB are built on Ubuntu.
Santiago also prepared !545, which will make the reprotest job more consistent with the result seen on reproducible-builds.
Santiago worked on different tasks related to DebConf 25. Especially he drafted the fundraising brochure (which is almost ready).
Thorsten Alteholz uploaded package libcupsfilter to fix the autopkgtest and a dependency problem of this package. After package splix was abandoned by upstream and OpenPrinting.org adopted its maintenance, Thorsten uploaded their first release.
Anupa published posts on the Debian Administrators group in LinkedIn and moderated the group, one of the tasks of the Debian Publicity Team.
Anupa helped organize DebUtsav 2024. It had over 100 attendees with hand-on sessions on making initial contributions to Linux Kernel, Debian packaging, submitting documentation to Debian wiki and assisting Debian Installations.
FTP master
This month I accepted 441 and rejected 29 packages. The overall number of packages that got accepted was 448.
I couldn t believe my eyes, but this month I really accepted the same number of packages as last month.
Debian LTS
This was my hundred-twenty-third month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. During my allocated time I uploaded or worked on:
[unstable] libcupsfilters security update to fix one CVE related to validation of IPP attributes obtained from remote printers
[unstable] cups-filters security update to fix two CVEs related to validation of IPP attributes obtained from remote printers
[unstable] cups security update to fix one CVE related to validation of IPP attributes obtained from remote printers
[DSA 5778-1] prepared package for cups-filters security update to fix two CVEs related to validation of IPP attributes obtained from remote printers
[DSA 5779-1] prepared package for cups security update to fix one CVE related to validation of IPP attributes obtained from remote printers
[DLA 3905-1] cups-filters security update to fix two CVEs related to validation of IPP attributes obtained from remote printers
[DLA 3904-1] cups security update to fix one CVE related to validation of IPP attributes obtained from remote printers
[DLA 3905-1] cups-filters security update to fix two CVEs related to validation of IPP attributes obtained from remote printers
Despite the announcement the package libppd in Debian is not affected by the CVEs related to CUPS. By pure chance there is an unrelated package with the same name in Debian. I also answered some question about the CUPS related uploads. Due to the CUPS issues, I postponed my work on other packages to October.
Last but not least I did a week of FD this month and attended the monthly LTS/ELTS meeting.
Debian ELTS
This month was the seventy-fourth ELTS month. During my allocated time I uploaded or worked on:
[ELA-1186-1]cups-filters security update for two CVEs in Stretch and Buster to fix the IPP attribute related CVEs.
[ELA-1187-1]cups-filters security update for one CVE in Jessie to fix the IPP attribute related CVEs (the version in Jessie was not affected by the other CVE).
I also started to work on updates for cups in Buster, Stretch and Jessie, but their uploads will happen only in October.
I also did a week of FD and attended the monthly LTS/ELTS meeting.
Debian Printing
This month I uploaded
libcupsfilters to also fix a dependency and autopkgtest issue besides the security fix mentioned above.
splix for a new upstream version. This package is managed now by OpenPrinting.
Last but not least I tried to prepare an update for hplip. Unfortunately this is a nerve-stretching task and I need some more time.
This work is generously funded by Freexian!
Debian Matomo
This month I even found some time to upload packages that are dependencies of Matomo
Most of the uploads were related to package migration to testing. As some of them are in non-free or contrib, one has to build all binary versions. From my point of view handling packages in non-free or contrib could be very much improved, but well, they are not part of Debian
Anyway, starting in December there is an Outreachy project that takes care of automatic updates of these packages. So hopefully it will be much easier to keep those package up to date. I will keep you informed.
Debian IoT
This month I uploaded new upstream or bugfix versions of:
Debian Mobcom
This month I did source uploads of all the packages that were prepared last month by Nathan and started the transition. It went rather smooth except for a few packages where the new version did not propagate to the tracker and they got stuck in old failing autopkgtest. Anyway, in the end all packages migrated to testing.
I also uploaded new upstream releases or fixed bugs in:
In my previous post on enabling SecureBoot, I mentioned that one
pending improvement was signing the systemd-boot EFI binary with my keys on
every upgrade. In this post, we'll explore the implementation of this process using
dpkg triggers.
For an excellent introduction to dpkg triggers, refer to this archived blog post.
The source code mentioned in that post can be downloaded from alioth archive.
From /usr/share/doc/dpkg/spec/triggers.txt, triggers are described as follows:
A dpkg trigger is a facility that allows events caused by one package
but of interest to another package to be recorded and aggregated, and
processed later by the interested package. This feature simplifies
various registration and system-update tasks and reduces duplication
of processing.
To implement this, we create a custom package with a single script that signs the
systemd-boot EFI binary using our key. The script is as simple as:
#!/bin/bashset-e
echo"Signing the new systemd-bootx64.efi"
sbsign--key/etc/secureboot/db.key--cert/etc/secureboot/db.crt\/usr/lib/systemd/boot/efi/systemd-bootx64.efi
echo"Invoking bootctl install to copy stuff"
bootctlinstall
Invoking bootctl install is optional if we have enabled
systemd-boot-update.service, which will update the signed bootloader on the next
boot.
We need to have a triggers file under the debian/ folder of the package, which
declares its interest in modifications to the path
/usr/lib/systemd/boot/efi/systemd-bootx64.efi. The trigger file looks like this:
# trigger 1 interest on systemd-bootx64.efi
interest-noawait /usr/lib/systemd/boot/efi/systemd-bootx64.efi
You can read about various directives and their meanings that can be used in
the triggers file in the deb-triggers man page.
Once we build and install the package, this request is added to
/var/lib/dpkg/triggers/File. See the screenshot below after installation of
our package:
To test the functionality, I performed a re-installation of the systemd-boot-efi
package, which provides the EFI binary for systemd-boot, using the following command:
sudoaptinstall--reinstallsystemd-boot-efi
During installation, you can see the debug message being printed in the screenshot below:
To test the systemd-boot-update.service, I commented out the bootctl install line
from the above script, performed a reinstallation, and restarted the
systemd-boot-update.service. Checking the log, I saw the following:
Sep 29 13:42:51 chamunda systemd[1]: Stopping systemd-boot-update.service - Automatic Boot Loader Update...
Sep 29 13:42:51 chamunda systemd[1]: Starting systemd-boot-update.service - Automatic Boot Loader Update...
Sep 29 13:42:51 chamunda bootctl[1801516]: Skipping "/efi/EFI/systemd/systemd-bootx64.efi", same boot loader version in place already.
Sep 29 13:42:51 chamunda bootctl[1801516]: Skipping "/efi/EFI/BOOT/BOOTX64.EFI", same boot loader version in place already.
Sep 29 13:42:51 chamunda bootctl[1801516]: Skipping "/efi/EFI/BOOT/BOOTX64.EFI", same boot loader version in place already.
Sep 29 13:42:51 chamunda systemd[1]: Finished systemd-boot-update.service - Automatic Boot Loader Update.
Sep 29 13:43:37 chamunda systemd[1]: systemd-boot-update.service: Deactivated successfully.
Sep 29 13:43:37 chamunda systemd[1]: Stopped systemd-boot-update.service - Automatic Boot Loader Update.
Sep 29 13:43:37 chamunda systemd[1]: Stopping systemd-boot-update.service - Automatic Boot Loader Update...
Indeed, the service attempted to copy the bootloader but did not do so because
there was no actual update to the binary; it was just a reinstallation trigger.
The complete code for this package can be found here.
With this post the entire series on using UKI to Secureboot with Debian comes
to an end. Happy hacking!.
A new release 0.0.4 of the RcppFastAD package
by James Yang and myself
is now on CRAN.
RcppFastAD
wraps the FastAD
header-only C++ library by James which provides a C++
implementation of both forward and reverse mode of automatic
differentiation. It offers an easy-to-use header library (which we
wrapped here) that is both lightweight and performant. With a little of
bit of Rcpp glue, it is also easy to
use from R in simple C++ applications. This release updates the quick
fix in release
0.0.3 from a good week ago. James took a good look and
properly disambiguated the statement that lead clang to complain, so we
are back to compiling as C++17 under all compilers which makes for a
slightly wider reach.
The NEWS file for this release follows.
Changes in version 0.0.4
(2024-09-24)
The package now properly addresses a clang warning on empty variadic
macros arguments and is back to C++17 (James in #10)
In this guide, we'll implement Secure Boot by taking full control of the device,
removing preinstalled keys, and installing our own. For a comprehensive overview
of the benefits and process, refer to this excellent post from rodsbooks.
Key Components
To implement Secure Boot, we need three essential keys:
Platform Key (PK): The top-level key in Secure Boot, typically provided by
the motherboard manufacturer. We'll replace the vendor-supplied PK with our
own for complete control.
Key Exchange Key (KEK): Used to sign updates for the Signatures Database and
Forbidden Signatures Database.
Database Key (DB): Used to sign or verify binaries (bootloaders, boot
managers, shells, drivers, etc.).
There's also a Forbidden Signature Key (dbx), which is the opposite of the DB
key. We won't be generating this key in this guide.
Preparing for Key Enrollment
Before enrolling our keys, we need to put the device in Secure Boot Setup
Mode. Verify the status using the bootctl status command. You should see
output similar to the following image:
Generating Keys
Follow these instructions from the Arch Wiki
to generate the keys manually. You'll need the efitools and openssl
packages. I recommend using rsa:2048 as the key size for better
compatibility with older firmware.
After generating the keys, copy all .auth files to the
/efi/loader/keys/<hostname>/ folder. For example:
Enrolling Keys in Firmware
Use systemd-boot to enroll your keys:
systemctlreboot--boot-loader-menu=0
Select the enroll option with your hostname in the systemd-boot menu.
After key enrollment, the system will reboot into the newly signed kernel.
Verify with bootctl:
Dealing with Lockdown Mode
Secure Boot enables lockdown mode on distro-shipped kernels, which restricts
certain features like kprobes/BPF and DKMS drivers. To avoid this, consider
compiling the upstream kernel directly, which doesn't enable lockdown mode by
default.
As Linus Torvalds has stated, "there is no reason to tie Secure Boot to lockdown
LSM." You can read more about Torvalds' opinion on UEFI tied with lockdown.
Next Steps
One thing that remains is automating the signing of systemd-boot on upgrade,
which is currently a manual process. I'm exploring dpkg triggers for achieving
this, and if I succeed, I will write a new post with details.
Acknowledgments
Special thanks to my anonymous colleague who provided invaluable assistance
throughout this process.
These steps may not work on your system if you are using the default Debian
installation. This guide assumes that your system is using systemd-boot as the
bootloader, which is explained in the post linked below.
A unified kernel image (UKI) is a single executable that can be booted directly
from UEFI firmware or automatically sourced by bootloaders with little or no
configuration. It combines a UEFI boot stub program like
systemd-stub(7), a Linux kernel image, an initrd, and additional resources into
a single UEFI PE file.
systemd-boot already provides a hook for kernel installation via
/etc/kernel/postinst.d/zz-systemd-boot. We just need a couple of additional
configurations to generate the UKI image.
Installation and Configuration
Install the systemd-ukify package:
sudoapt-getinstallsystemd-ukify
Create the following configuration in /etc/kernel/install.conf:
Cleanup and Reboot
Once the "Type #2" entries are generated, remove any "Type #1" entries using the bootctl unlink command. After this, reboot your system to boot from the UKI-based image.
Future Considerations
The primary use case for a UKI image is secure boot. Signing the UKI image can also be configured in the settings above, but this guide does not cover that process as it requires setting up secure boot on your system.
A new release 0.0.3 of the RcppFastAD package
by James Yang and myself
is now on CRAN.
RcppFastAD
wraps the FastAD
header-only C++ library by James which provides a C++
implementation of both forward and reverse mode of automatic
differentiation. It offers an easy-to-use header library (which we
wrapped here) that is both lightweight and performant. With a little of
bit of Rcpp glue, it is also easy to
use from R in simple C++ applications. This release turns compilation to
the C++20 standard as newer clang++ versions complained
about a particular statement (it took to be C++20) when compiled under
C++17. So we obliged.
The NEWS file for these two initial releases follows.
Changes in version 0.0.3
(2024-09-15)
The package now compiles under the C++20 standard to avoid a
warning under clang++-18 (Dirk addressing #9)
Minor updates to continuous integration and badges have been made
as well
Debian Python 3 patch review, by Stefano Rivera
Last month, at DebConf, Stefano reviewed the current patch set of Debian s cPython packages with Matthias Klose, the primary maintainer until now. As a result of that review, Stefano re-reviewed the patchset, updating descriptions, etc. A few patches were able to be dropped, and a few others were forwarded upstream.
One finds all sorts of skeletons doing reviews like this. One of the patches had been inactive (fortunately, because it was buggy) since the day it was applied, 13 years ago. One is a cleanup that probably only fixes a bug on HPUX, and is a result of copying code from xfree86 into Python 25 years ago. It was fixed in xfree86 a year later. Others support just Debian-specific functionality and probably never seemed worth forwarding. Or good cleanup that only really applies to Debian.
A trivial new patch would allow Debian to multiarch co-install Python stable ABI dynamic extensions (like we can with regular dynamic extensions). Performance concerns are stalling it in review, at the moment.
DebConf 24 Organization, by Stefano Rivera
Stefano helped organize DebConf 24, which concluded in early August. The event is run by a large entirely volunteer team. The work involved in making this happen is far too varied to describe here. While Freexian provides funding for 20% of collaborator time to spend on Debian-related work, it only covers a small fraction of contributions to time-intensive tasks like this.
Since the end of the event, Stefano has been doing some work on the conference finances, and initiated the reimbursement process for travel bursaries.
Archive rebuilds on Debusine, by Stefano Rivera
The recent setuptools 73 upload to Debian unstable removed the test subcommand, breaking many packages that were using python3 setup.py test in their Debian packaging. Stefano did a partial archive-rebuild using debusine.debian.net to find the regressions and file bugs.
Debusine will be a powerful tool to do QA work like this for Debian in the future, but it doesn t have all the features needed to coordinate rebuild-testing, yet. They are planned to be fleshed out in the next year. In the meantime, Debusine has the building blocks to work through a queue of package building tasks and store the results, it just needs to be driven from outside the system.
So, Stefano started working on a set of tools using the Debusine client API to perform archive rebuilds, found and tagged existing bugs, and filed many more.
OpenSSL vs. cryptography, by Colin Watson
Colin facilitated a discussion between Debian s OpenSSL team and the upstream maintainers of Python cryptography about a new incompatibility between Debian s OpenSSL packaging and cryptography s handling of OpenSSL s legacy provider, which was causing a number of build and test failures. While the issue remains open, the Debian OpenSSL maintainers have effectively reverted the change now, so it s no longer a pressing problem.
/usr-move, by Helmut Grohne
There are less than 40 source packages left to move files to /usr, so what we re left with is the long tail of the transition. Rather than fix all of them, Helmut started a discussion on removing packages from unstable and filed a first batch. As libvirt is being restructured in experimental, we re handling the fallout in collaboration with its maintainer Andrea Bolognani. Since base-files validates the aliasing symlinks before upgrading, it was discovered that systemd has its own ideas with no solution as of yet. Helmut also proposed that dashchecks for ineffective diversions of /bin/sh and that lintianwarns about aliased files.
rebootstrap by Helmut Grohne
Bootstrapping Debian for a new or existing CPU architecture still is a quite manual process. The rebootstrap project attempts to automate part of the early stage, but it still is very sensitive to changes in unstable. We had a number of fairly intrusive changes this year already. August included a little more fallout from the earlier gcc-for-host work where the C++ include search path would end up being wrong in the generated cross toolchain. A number of packages such as util-linux (twice), libxml2, libcap-ng or systemd had their stage profiles broken. e2fsprogs gained a cycle with libarchive-dev due to having gained support for creating an ext4 filesystem from a tar archive. The restructuring of glib2.0 remains an unsolved problem for now, but libxt and cdebconf should be buildable without glib2.0.
Salsa CI, by Santiago Ruano Rinc n
Santiago completed the initial RISC-V support (!523) in the Salsa CI s pipeline. The main work started in July, but it was required to take into account some comments in the review (thanks to Ahmed!) and some final details in [!534]. riscv64 is the most recently supported port in Debian, which will be part of trixie. As its name suggests, the new build-riscv64 job makes it possible to test that a package successfully builds in the riscv64 architecture. The RISC-V runner (salsaci riscv64 runner 01) runs in a couple of machines generously provided by lab.rvperf.org. Debian Developers interested in running this job in their projects should enable the runner (salsaci riscv64 runner 01) in Settings / CI / Runners, and follow the instructions available at https://salsa.debian.org/salsa-ci-team/pipeline/#build-job-on-risc-v.
Santiago also took part in discussions about how to optimize the build jobs and reviewed !537 to make the build-source job to only satisfy the Build-Depends and Build-Conflicts fields by Andrea Pappacoda. Thanks a lot to him!
Miscellaneous contributions
Stefano submitted patches for BeautifulSoup to support the latest soupsieve and lxml.
Stefano uploaded pypy3 7.3.17, upgrading the cPython compatibility from 3.9 to 3.10. Then ran into a GCC-14-related regression, which had to be ignored for now as it s proving hard to fix.
Colin released libpipeline 1.5.8 and man-db 2.13.0; the latter included foundations allowing adding an autopkgtest for man-db.
Colin tracked down test failures in python-asyncssh and Ruby resulting from certain odd /etc/hosts configurations.
Carles upgraded the packages python-ring-doorbell and simplemonitor to new upstream versions.
Carles started discussions and implementation of a tool (still in early days) named po-debconf-manager : a way for translators and reviewers to collaborate using git as a backend instead of mailing list; and submit the translations using salsa MR. More information next month.
Carles (dog-fooding po-debconf-manager ) reviewed debconf templates translated by a collaborator.
Carles reviewed and submitted the translation of apt .
Helmut sent 19 patches for improving cross building.
Helmut implemented the cross-exe-wrapper proposed by Simon McVittie for use with glib2.0.
Helmut detailed what it takes to make Perl s ExtUtils::PkgConfig suitable for cross building.
Helmut made the deletion of the root password work in debvm in all situations and implemented a test case using expect.
Anupa attended Debian Publicity team meeting and is moderating and posting on Debian Administrators LinkedIn group.
Thorsten uploaded package gutenprint to fix a FTBFS with gcc14 and package ipp-usb to fix a /usr-merge issue.
Santiago updated bzip2 to fix a long-standing bug that requested to include a pkg-config file. An important impact of this change is that it makes it possible to use Rust bindings for libbz2 by Sequoia, an implementation of OpenPGP.
How does Linux move from an awake machine to a hibernating one?
How does it then manage to restore all state?
These questions led me to read way too much C in trying to figure out
how this particular hardware/software boundary is navigated.
This investigation will be split into a few parts, with the first one going
from invocation of hibernation to synchronizing all filesystems to disk.
This article has been written using Linux version 6.9.9,
the source of which can be found in many places, but can be navigated
easily through the Bootlin Elixir Cross-Referencer:
https://elixir.bootlin.com/linux/v6.9.9/source
Each code snippet will begin with a link to the above giving
the file path and the line number of the beginning of the snippet.
A Starting Point for Investigation: /sys/power/state and /sys/power/disk
These two system files exist to allow debugging of hibernation,
and thus control the exact state used directly.
Writing specific values to the state file controls the exact sleep mode used
and disk controls the specific hibernation mode1.
This is extremely handy as an entry point to understand how these systems work,
since we can just follow what happens when they are written to.
Show and Store Functions
These two files are defined using the power_attr macro:
kernel/power/power.h:80
show is called on reads and store on writes.
state_show is a little boring for our purposes, as it just prints all the
available sleep states.
kernel/power/main.c:657
/*
* state - control system sleep states.
*
* show() returns available sleep state labels, which may be "mem", "standby",
* "freeze" and "disk" (hibernation).
* See Documentation/admin-guide/pm/sleep-states.rst for a description of
* what they mean.
*
* store() accepts one of those strings, translates it into the proper
* enumerated value, and initiates a suspend transition.
*/
static ssize_t state_show(struct kobject *kobj, struct kobj_attribute *attr,
char *buf)
char *s = buf;
#ifdef CONFIG_SUSPEND
suspend_state_t i;
for (i = PM_SUSPEND_MIN; i < PM_SUSPEND_MAX; i++)
if (pm_states[i])
s += sprintf(s,"%s ", pm_states[i]);
#endif
if (hibernation_available())
s += sprintf(s, "disk ");
if (s != buf)
/* convert the last space to a newline */
*(s-1) = '\n';
return (s - buf);
state_store, however, provides our entry point.
If the string disk is written to the state file, it calls hibernate().
This is our entry point.
kernel/power/main.c:715
static ssize_t state_store(struct kobject *kobj, struct kobj_attribute *attr,
const char *buf, size_t n)
suspend_state_t state;
int error;
error = pm_autosleep_lock();
if (error)
return error;
if (pm_autosleep_state() > PM_SUSPEND_ON)
error = -EBUSY;
goto out;
state = decode_state(buf, n);
if (state < PM_SUSPEND_MAX)
if (state == PM_SUSPEND_MEM)
state = mem_sleep_current;
error = pm_suspend(state);
else if (state == PM_SUSPEND_MAX)
error = hibernate();
else
error = -EINVAL;
out:
pm_autosleep_unlock();
return error ? error : n;
static suspend_state_t decode_state(const char *buf, size_t n)
#ifdef CONFIG_SUSPEND
suspend_state_t state;
#endif
char *p;
int len;
p = memchr(buf, '\n', n);
len = p ? p - buf : n;
/* Check hibernation first. */
if (len == 4 && str_has_prefix(buf, "disk"))
return PM_SUSPEND_MAX;
#ifdef CONFIG_SUSPEND
for (state = PM_SUSPEND_MIN; state < PM_SUSPEND_MAX; state++)
const char *label = pm_states[state];
if (label && len == strlen(label) && !strncmp(buf, label, len))
return state;
#endif
return PM_SUSPEND_ON;
Could we have figured this out just via function names?
Sure, but this way we know for sure that nothing else is happening before this
function is called.
Autosleep
Our first detour is into the autosleep system.
When checking the state above, you may notice that
the kernel grabs the pm_autosleep_lock before checking the current
state.
autosleep is a mechanism originally from Android
that sends the entire system to either suspend or hibernate whenever it is
not actively working on anything.
This is not enabled for most desktop configurations, since it s primarily
for mobile systems and inverts the standard suspend and hibernate interactions.
This system is implemented as a workqueue2
that checks the current number of wakeup events, processes and drivers that
need to run3, and if there aren t any, then the system is put
into the autosleep state, typically suspend. However, it could be hibernate if
configured that way via /sys/power/autosleep in a similar manner to
using /sys/power/state to manually enable hibernation.
kernel/power/main.c:841
static DEFINE_MUTEX(autosleep_lock);
static struct wakeup_source *autosleep_ws;
static void try_to_suspend(struct work_struct *work)
unsigned int initial_count, final_count;
if (!pm_get_wakeup_count(&initial_count, true))
goto out;
mutex_lock(&autosleep_lock);
if (!pm_save_wakeup_count(initial_count)
system_state != SYSTEM_RUNNING)
mutex_unlock(&autosleep_lock);
goto out;
if (autosleep_state == PM_SUSPEND_ON)
mutex_unlock(&autosleep_lock);
return;
if (autosleep_state >= PM_SUSPEND_MAX)
hibernate();
else
pm_suspend(autosleep_state);
mutex_unlock(&autosleep_lock);
if (!pm_get_wakeup_count(&final_count, false))
goto out;
/*
* If the wakeup occurred for an unknown reason, wait to prevent the
* system from trying to suspend and waking up in a tight loop.
*/
if (final_count == initial_count)
schedule_timeout_uninterruptible(HZ / 2);
out:
queue_up_suspend_work();
static DECLARE_WORK(suspend_work, try_to_suspend);
void queue_up_suspend_work(void)
if (autosleep_state > PM_SUSPEND_ON)
queue_work(autosleep_wq, &suspend_work);
The Steps of Hibernation
Hibernation Kernel Config
It s important to note that most of the hibernate-specific functions below
do nothing unless you ve defined CONFIG_HIBERNATION in your Kconfig4.
As an example, hibernate itself is defined as the following if
CONFIG_HIBERNATE is not set.
include/linux/suspend.h:407
static inline int hibernate(void) return -ENOSYS;
Check if Hibernation is Available
We begin by confirming that we actually can perform hibernation,
via the hibernation_available function.
kernel/power/hibernate.c:742
if (!hibernation_available())
pm_pr_dbg("Hibernation not available.\n");
return -EPERM;
nohibernate is controlled by the kernel command line, it s set via
either nohibernate or hibernate=no.
security_locked_down is a hook for Linux Security Modules to prevent
hibernation. This is used to prevent hibernating to an unencrypted storage
device, as specified in the manual page
kernel_lockdown(7).
Interestingly, either level of lockdown, integrity or confidentiality,
locks down hibernation because with the ability to hibernate you can extract
bascially anything from memory and even reboot into a modified kernel image.
secretmem_active checks whether there is any active use of
memfd_secret, and if so it prevents hibernation.
memfd_secret returns a file descriptor that can be mapped into a process
but is specifically unmapped from the kernel s memory space.
Hibernating with memory that not even the kernel is supposed to
access would expose that memory to whoever could access the hibernation image.
This particular feature of secret memory was apparently
controversial, though not as
controversial as performance concerns around fragmentation when unmapping
kernel memory
(which did not end up being a real problem).
cxl_mem_active just checks whether any CXL memory is active.
A full explanation is provided in the
commit introducing this check
but there s also a shortened explanation from cxl_mem_probe that
sets the relevant flag when initializing a CXL memory device.
drivers/cxl/mem.c:186
* The kernel may be operating out of CXL memory on this device,
* there is no spec defined way to determine whether this device
* preserves contents over suspend, and there is no simple way
* to arrange for the suspend image to avoid CXL memory which
* would setup a circular dependency between PCI resume and save
* state restoration.
Check Compression
The next check is for whether compression support is enabled, and if so
whether the requested algorithm is enabled.
kernel/power/hibernate.c:747
/*
* Query for the compression algorithm support if compression is enabled.
*/
if (!nocompress)
strscpy(hib_comp_algo, hibernate_compressor, sizeof(hib_comp_algo));
if (crypto_has_comp(hib_comp_algo, 0, 0) != 1)
pr_err("%s compression is not available\n", hib_comp_algo);
return -EOPNOTSUPP;
The nocompress flag is set via the hibernate command line parameter,
setting hibernate=nocompress.
If compression is enabled, then hibernate_compressor is copied to
hib_comp_algo. This synchronizes the current requested compression
setting (hibernate_compressor) with the current compression setting
(hib_comp_algo).
Both values are character arrays of size CRYPTO_MAX_ALG_NAME
(128 in this kernel).
kernel/power/hibernate.c:50
static char hibernate_compressor[CRYPTO_MAX_ALG_NAME] = CONFIG_HIBERNATION_DEF_COMP;
/*
* Compression/decompression algorithm to be used while saving/loading
* image to/from disk. This would later be used in 'kernel/power/swap.c'
* to allocate comp streams.
*/
char hib_comp_algo[CRYPTO_MAX_ALG_NAME];
hibernate_compressor defaults to lzo if that algorithm is enabled, otherwise to lz4 if
enabled5. It can be overwritten using the hibernate.compressor setting to
either lzo or lz4.
kernel/power/Kconfig:95
choice
prompt "Default compressor"
default HIBERNATION_COMP_LZO
depends on HIBERNATION
config HIBERNATION_COMP_LZO
bool "lzo"
depends on CRYPTO_LZO
config HIBERNATION_COMP_LZ4
bool "lz4"
depends on CRYPTO_LZ4
endchoice
config HIBERNATION_DEF_COMP
string
default "lzo" if HIBERNATION_COMP_LZO
default "lz4" if HIBERNATION_COMP_LZ4
help
Default compressor to be used for hibernation.
static const char * const comp_alg_enabled[] =
#if IS_ENABLED(CONFIG_CRYPTO_LZO)
COMPRESSION_ALGO_LZO,
#endif
#if IS_ENABLED(CONFIG_CRYPTO_LZ4)
COMPRESSION_ALGO_LZ4,
#endif
;
static int hibernate_compressor_param_set(const char *compressor,
const struct kernel_param *kp)
unsigned int sleep_flags;
int index, ret;
sleep_flags = lock_system_sleep();
index = sysfs_match_string(comp_alg_enabled, compressor);
if (index >= 0)
ret = param_set_copystring(comp_alg_enabled[index], kp);
if (!ret)
strscpy(hib_comp_algo, comp_alg_enabled[index],
sizeof(hib_comp_algo));
else
ret = index;
unlock_system_sleep(sleep_flags);
if (ret)
pr_debug("Cannot set specified compressor %s\n",
compressor);
return ret;
static const struct kernel_param_ops hibernate_compressor_param_ops =
.set = hibernate_compressor_param_set,
.get = param_get_string,
;
static struct kparam_string hibernate_compressor_param_string =
.maxlen = sizeof(hibernate_compressor),
.string = hibernate_compressor,
;
We then check whether the requested algorithm is supported via crypto_has_comp.
If not, we bail out of the whole operation with EOPNOTSUPP.
As part of crypto_has_comp we perform any needed initialization of the
algorithm, loading kernel modules and running initialization code as needed6.
Grab Locks
The next step is to grab the sleep and hibernation locks via
lock_system_sleep and hibernate_acquire.
kernel/power/hibernate.c:758
sleep_flags = lock_system_sleep();
/* The snapshot device should not be opened while we're running */
if (!hibernate_acquire())
error = -EBUSY;
goto Unlock;
First, lock_system_sleep marks the current thread as not freezable, which
will be important later7. It then grabs the system_transistion_mutex,
which locks taking snapshots or modifying how they are taken,
resuming from a hibernation image, entering any suspend state, or rebooting.
The GFP Mask
The kernel also issues a warning if the gfp mask is changed via either
pm_restore_gfp_mask or pm_restrict_gfp_mask
without holding the system_transistion_mutex.
GFP flags tell the kernel how it is permitted to handle a request for memory.
include/linux/gfp_types.h:12
* GFP flags are commonly used throughout Linux to indicate how memory
* should be allocated. The GFP acronym stands for get_free_pages(),
* the underlying memory allocation function. Not every GFP flag is
* supported by every function which may allocate memory.
In the case of hibernation specifically we care about the IO and FS flags,
which are reclaim operators, ways the system is permitted to attempt to free
up memory in order to satisfy a specific request for memory.
include/linux/gfp_types.h:176
* Reclaim modifiers
* -----------------
* Please note that all the following flags are only applicable to sleepable
* allocations (e.g. %GFP_NOWAIT and %GFP_ATOMIC will ignore them).
*
* %__GFP_IO can start physical IO.
*
* %__GFP_FS can call down to the low-level FS. Clearing the flag avoids the
* allocator recursing into the filesystem which might already be holding
* locks.
gfp_allowed_mask sets which flags are permitted to be set at the current time.
As the comment below outlines, preventing these flags from being set
avoids situations where the kernel needs to do I/O to allocate memory
(e.g. read/writing swap8) but the
devices it needs to read/write to/from are not currently available.
kernel/power/main.c:24
/*
* The following functions are used by the suspend/hibernate code to temporarily
* change gfp_allowed_mask in order to avoid using I/O during memory allocations
* while devices are suspended. To avoid races with the suspend/hibernate code,
* they should always be called with system_transition_mutex held
* (gfp_allowed_mask also should only be modified with system_transition_mutex
* held, unless the suspend/hibernate code is guaranteed not to run in parallel
* with that modification).
*/
static gfp_t saved_gfp_mask;
void pm_restore_gfp_mask(void)
WARN_ON(!mutex_is_locked(&system_transition_mutex));
if (saved_gfp_mask)
gfp_allowed_mask = saved_gfp_mask;
saved_gfp_mask = 0;
void pm_restrict_gfp_mask(void)
WARN_ON(!mutex_is_locked(&system_transition_mutex));
WARN_ON(saved_gfp_mask);
saved_gfp_mask = gfp_allowed_mask;
gfp_allowed_mask &= ~(__GFP_IO __GFP_FS);
Sleep Flags
After grabbing the system_transition_mutex the kernel then returns and
captures the previous state of the threads flags in sleep_flags.
This is used later to remove PF_NOFREEZE if it wasn t previously set on the
current thread.
kernel/power/main.c:52
unsigned int lock_system_sleep(void)
unsigned int flags = current->flags;
current->flags = PF_NOFREEZE;
mutex_lock(&system_transition_mutex);
return flags;
EXPORT_SYMBOL_GPL(lock_system_sleep);
#define PF_NOFREEZE 0x00008000 /* This thread should not be frozen */
Then we grab the hibernate-specific semaphore to ensure no one can open a
snapshot or resume from it while we perform hibernation.
Additionally this lock is used to prevent hibernate_quiet_exec,
which is used by the nvdimm driver to active its firmware with all
processes and devices frozen, ensuring it is the only thing running at that
time9.
kernel/power/hibernate.c:82
Prepare Console
The kernel next calls pm_prepare_console.
This function only does anything if CONFIG_VT_CONSOLE_SLEEP has been set.
This prepares the virtual terminal for a suspend state, switching away to
a console used only for the suspend state if needed.
kernel/power/console.c:130
void pm_prepare_console(void)
if (!pm_vt_switch())
return;
orig_fgconsole = vt_move_to_console(SUSPEND_CONSOLE, 1);
if (orig_fgconsole < 0)
return;
orig_kmsg = vt_kmsg_redirect(SUSPEND_CONSOLE);
return;
/*
* There are three cases when a VT switch on suspend/resume are required:
* 1) no driver has indicated a requirement one way or another, so preserve
* the old behavior
* 2) console suspend is disabled, we want to see debug messages across
* suspend/resume
* 3) any registered driver indicates it needs a VT switch
*
* If none of these conditions is present, meaning we have at least one driver
* that doesn't need the switch, and none that do, we can avoid it to make
* resume look a little prettier (and suspend too, but that's usually hidden,
* e.g. when closing the lid on a laptop).
*/
static bool pm_vt_switch(void)
struct pm_vt_switch *entry;
bool ret = true;
mutex_lock(&vt_switch_mutex);
if (list_empty(&pm_vt_switch_list))
goto out;
if (!console_suspend_enabled)
goto out;
list_for_each_entry(entry, &pm_vt_switch_list, head)
if (entry->required)
goto out;
ret = false;
out:
mutex_unlock(&vt_switch_mutex);
return ret;
There is an explanation of the conditions under which a switch is performed
in the comment above the function, but we ll also walk through the steps here.
Firstly we grab the vt_switch_mutex to ensure nothing will modify the list
while we re looking at it.
We then examine the pm_vt_switch_list.
This list is used to indicate the drivers that require a switch during suspend.
They register this requirement, or the lack thereof, via
pm_vt_switch_required.
kernel/power/console.c:31
/**
* pm_vt_switch_required - indicate VT switch at suspend requirements
* @dev: device
* @required: if true, caller needs VT switch at suspend/resume time
*
* The different console drivers may or may not require VT switches across
* suspend/resume, depending on how they handle restoring video state and
* what may be running.
*
* Drivers can indicate support for switchless suspend/resume, which can
* save time and flicker, by using this routine and passing 'false' as
* the argument. If any loaded driver needs VT switching, or the
* no_console_suspend argument has been passed on the command line, VT
* switches will occur.
*/
void pm_vt_switch_required(struct device *dev, bool required)
Next, we check console_suspend_enabled. This is set to false
by the kernel parameter no_console_suspend, but defaults to true.
Finally, if there are any entries in the pm_vt_switch_list, then we
check to see if any of them require a VT switch.
Only if none of these conditions apply, then we return false.
If a VT switch is in fact required, then we move first the currently active
virtual terminal/console10 (vt_move_to_console)
and then the current location of kernel messages (vt_kmsg_redirect)
to the SUSPEND_CONSOLE.
The SUSPEND_CONSOLE is the last entry in the list of possible
consoles, and appears to just be a black hole to throw away messages.
kernel/power/console.c:16
#define SUSPEND_CONSOLE (MAX_NR_CONSOLES-1)
Interestingly, these are separate functions because you can use
TIOCL_SETKMSGREDIRECT (an ioctl11) to send kernel messages to a specific virtual terminal,
but by default its the same as the currently active console.
The locations of the previously active console and the previous kernel
messages location are stored in orig_fgconsole and orig_kmsg, to
restore the state of the console and kernel messages after the machine wakes
up again.
Interestingly, this means orig_fgconsole also ends up storing any errors,
so has to be checked to ensure it s not less than zero before we try to do
anything with the kernel messages on both suspend and resume.
drivers/tty/vt/vt_ioctl.c:1268
/* Perform a kernel triggered VT switch for suspend/resume */
static int disable_vt_switch;
int vt_move_to_console(unsigned int vt, int alloc)
int prev;
console_lock();
/* Graphics mode - up to X */
if (disable_vt_switch)
console_unlock();
return 0;
prev = fg_console;
if (alloc && vc_allocate(vt))
/* we can't have a free VC for now. Too bad,
* we don't want to mess the screen for now. */
console_unlock();
return -ENOSPC;
if (set_console(vt))
/*
* We're unable to switch to the SUSPEND_CONSOLE.
* Let the calling function know so it can decide
* what to do.
*/
console_unlock();
return -EIO;
console_unlock();
if (vt_waitactive(vt + 1))
pr_debug("Suspend: Can't switch VCs.");
return -EINTR;
return prev;
Unlike most other locking functions we ve seen so far, console_lock
needs to be careful to ensure nothing else is panicking and needs to
dump to the console before grabbing the semaphore for the console
and setting a couple flags.
Panics
Panics are tracked via an atomic integer set to the id of the processor
currently panicking.
kernel/printk/printk.c:2649
/**
* console_lock - block the console subsystem from printing
*
* Acquires a lock which guarantees that no consoles will
* be in or enter their write() callback.
*
* Can sleep, returns nothing.
*/
void console_lock(void)
might_sleep();
/* On panic, the console_lock must be left to the panic cpu. */
while (other_cpu_in_panic())
msleep(1000);
down_console_sem();
console_locked = 1;
console_may_schedule = 1;
EXPORT_SYMBOL(console_lock);
/*
* Return true if a panic is in progress on a remote CPU.
*
* On true, the local CPU should immediately release any printing resources
* that may be needed by the panic CPU.
*/
bool other_cpu_in_panic(void)
return (panic_in_progress() && !this_cpu_in_panic());
/* Return true if a panic is in progress on the current CPU. */
bool this_cpu_in_panic(void)
/*
* We can use raw_smp_processor_id() here because it is impossible for
* the task to be migrated to the panic_cpu, or away from it. If
* panic_cpu has already been set, and we're not currently executing on
* that CPU, then we never will be.
*/
return unlikely(atomic_read(&panic_cpu) == raw_smp_processor_id());
console_locked is a debug value, used to indicate that the lock should be
held, and our first indication that this whole virtual terminal system is
more complex than might initially be expected.
kernel/printk/printk.c:373
/*
* This is used for debugging the mess that is the VT code by
* keeping track if we have the console semaphore held. It's
* definitely not the perfect debug tool (we don't know if _WE_
* hold it and are racing, but it helps tracking those weird code
* paths in the console code where we end up in places I want
* locked without the console semaphore held).
*/
static int console_locked;
console_may_schedule is used to see if we are permitted to sleep
and schedule other work while we hold this lock.
As we ll see later, the virtual terminal subsystem is not re-entrant,
so there s all sorts of hacks in here to ensure we don t leave important
code sections that can t be safely resumed.
Disable VT Switch
As the comment below lays out, when another program is handling graphical
display anyway, there s no need to do any of this, so the kernel provides
a switch to turn the whole thing off.
Interestingly, this appears to only be used by three drivers,
so the specific hardware support required must not be particularly common.
/*
* Normally during a suspend, we allocate a new console and switch to it.
* When we resume, we switch back to the original console. This switch
* can be slow, so on systems where the framebuffer can handle restoration
* of video registers anyways, there's little point in doing the console
* switch. This function allows you to disable it by passing it '0'.
*/
void pm_set_vt_switch(int do_switch)
console_lock();
disable_vt_switch = !do_switch;
console_unlock();
EXPORT_SYMBOL(pm_set_vt_switch);
The rest of the vt_switch_console function is pretty normal,
however, simply allocating space if needed to create the requested
virtual terminal
and then setting the current virtual terminal via set_console.
Virtual Terminal Set Console
With set_console, we begin (as if we haven t been already) to enter the
madness that is the virtual terminal subsystem.
As mentioned previously, modifications to its state must be made very
carefully, as other stuff happening at the same time could create complete
messes.
All this to say, calling set_console does not actually perform any
work to change the state of the current console.
Instead it indicates what changes it wants and then schedules that work.
drivers/tty/vt/vt.c:3153
int set_console(int nr)
struct vc_data *vc = vc_cons[fg_console].d;
if (!vc_cons_allocated(nr) vt_dont_switch
(vc->vt_mode.mode == VT_AUTO && vc->vc_mode == KD_GRAPHICS))
/*
* Console switch will fail in console_callback() or
* change_console() so there is no point scheduling
* the callback
*
* Existing set_console() users don't check the return
* value so this shouldn't break anything
*/
return -EINVAL;
want_console = nr;
schedule_console_callback();
return 0;
The check for vc->vc_mode == KD_GRAPHICS is where most end-user graphical
desktops will bail out of this change, as they re in graphics mode and don t
need to switch away to the suspend console.
vt_dont_switch is a flag used by the ioctls11VT_LOCKSWITCH and
VT_UNLOCKSWITCH to prevent the system from switching virtual terminal
devices when the user has explicitly locked it.
VT_AUTO is a flag indicating that automatic virtual terminal switching is enabled12,
and thus deliberate switching to a suspend terminal is not required.
However, if you do run your machine from a virtual terminal, then we
indicate to the system that we want to change to the requested virtual terminal
via the want_console variable
and schedule a callback via schedule_console_callback.
drivers/tty/vt/vt.c:315
/*
* This is the console switching callback.
*
* Doing console switching in a process context allows
* us to do the switches asynchronously (needed when we want
* to switch due to a keyboard interrupt). Synchronization
* with other console code and prevention of re-entrancy is
* ensured with console_lock.
*/
static void console_callback(struct work_struct *ignored)
console_lock();
if (want_console >= 0)
if (want_console != fg_console &&
vc_cons_allocated(want_console))
hide_cursor(vc_cons[fg_console].d);
change_console(vc_cons[want_console].d);
/* we only changed when the console had already
been allocated - a new console is not created
in an interrupt routine */
want_console = -1;
...
console_callback first looks to see if there is a console change wanted
via want_console and then changes to it if it s not the current console and
has been allocated already.
We do first remove any cursor state with hide_cursor.
drivers/tty/vt/vt.c:841
This will call a chain of power management callbacks, passing first
PM_HIBERNATION_PREPARE and then PM_POST_HIBERNATION on startup or
on error with another callback.
kernel/power/main.c:98
int pm_notifier_call_chain_robust(unsigned long val_up, unsigned long val_down)
int ret;
ret = blocking_notifier_call_chain_robust(&pm_chain_head, val_up, val_down, NULL);
return notifier_to_errno(ret);
The power management notifier is a blocking notifier chain, which means it
has the following properties.
include/linux/notifier.h:23
* Blocking notifier chains: Chain callbacks run in process context.
* Callouts are allowed to block.
The callback chain is a linked list with each entry containing a priority
and a function to call. The function technically takes in a data value,
but it is always NULL for the power management chain.
include/linux/notifier.h:49
struct notifier_block;
typedef int (*notifier_fn_t)(struct notifier_block *nb,
unsigned long action, void *data);
struct notifier_block
notifier_fn_t notifier_call;
struct notifier_block __rcu *next;
int priority;
;
Because it is prioritized, appending to the list requires walking it until
an item with lower13 priority is found to insert the current item before.
kernel/notifier.c:252
/*
* Blocking notifier chain routines. All access to the chain is
* synchronized by an rwsem.
*/
static int __blocking_notifier_chain_register(struct blocking_notifier_head *nh,
struct notifier_block *n,
bool unique_priority)
int ret;
/*
* This code gets used during boot-up, when task switching is
* not yet working and interrupts must remain disabled. At
* such times we must not call down_write().
*/
if (unlikely(system_state == SYSTEM_BOOTING))
return notifier_chain_register(&nh->head, n, unique_priority);
down_write(&nh->rwsem);
ret = notifier_chain_register(&nh->head, n, unique_priority);
up_write(&nh->rwsem);
return ret;
#define NOTIFY_DONE 0x0000 /* Don't care */
#define NOTIFY_OK 0x0001 /* Suits me */
#define NOTIFY_STOP_MASK 0x8000 /* Don't call further */
#define NOTIFY_BAD (NOTIFY_STOP_MASK 0x0002)
/* Bad/Veto action */
When notifying the chain, if a function returns STOP or BAD then
the previous parts of the chain are called again with PM_POST_HIBERNATION14
and an error is returned.
kernel/notifier.c:107
/**
* notifier_call_chain_robust - Inform the registered notifiers about an event
* and rollback on error.
* @nl: Pointer to head of the blocking notifier chain
* @val_up: Value passed unmodified to the notifier function
* @val_down: Value passed unmodified to the notifier function when recovering
* from an error on @val_up
* @v: Pointer passed unmodified to the notifier function
*
* NOTE: It is important the @nl chain doesn't change between the two
* invocations of notifier_call_chain() such that we visit the
* exact same notifier callbacks; this rules out any RCU usage.
*
* Return: the return value of the @val_up call.
*/
static int notifier_call_chain_robust(struct notifier_block **nl,
unsigned long val_up, unsigned long val_down,
void *v)
int ret, nr = 0;
ret = notifier_call_chain(nl, val_up, v, -1, &nr);
if (ret & NOTIFY_STOP_MASK)
notifier_call_chain(nl, val_down, v, nr-1, NULL);
return ret;
Each of these callbacks tends to be quite driver-specific, so we ll cease
discussion of this here.
Sync Filesystems
The next step is to ensure all filesystems have been synchronized to disk.
This is performed via a simple helper function that times how long the full
synchronize operation, ksys_sync takes.
kernel/power/main.c:69
ksys_sync wakes and instructs a set of flusher threads to write out
every filesystem, first their inodes15, then the full filesystem, and
then finally all block devices, to ensure all pages are written out
to disk.
fs/sync.c:87
/*
* Sync everything. We start by waking flusher threads so that most of
* writeback runs on all devices in parallel. Then we sync all inodes reliably
* which effectively also waits for all flusher threads to finish doing
* writeback. At this point all data is on disk so metadata should be stable
* and we tell filesystems to sync their metadata via ->sync_fs() calls.
* Finally, we writeout all block devices because some filesystems (e.g. ext2)
* just write metadata (such as inodes or bitmaps) to block device page cache
* and do not sync it on their own in ->sync_fs().
*/
void ksys_sync(void)
int nowait = 0, wait = 1;
wakeup_flusher_threads(WB_REASON_SYNC);
iterate_supers(sync_inodes_one_sb, NULL);
iterate_supers(sync_fs_one_sb, &nowait);
iterate_supers(sync_fs_one_sb, &wait);
sync_bdevs(false);
sync_bdevs(true);
if (unlikely(laptop_mode))
laptop_sync_completion();
It follows an interesting pattern of using iterate_supers to run both
sync_inodes_one_sb and then sync_fs_one_sb on each known filesystem16.
It also calls both sync_fs_one_sb and sync_bdevs twice, first without waiting
for any operations to complete and then again waiting for completion17.
When laptop_mode is enabled the system runs additional filesystem synchronization
operations after the specified delay without any writes.
mm/page-writeback.c:111
/*
* Flag that puts the machine in "laptop mode". Doubles as a timeout in jiffies:
* a full sync is triggered after this time elapses without any disk activity.
*/
int laptop_mode;
EXPORT_SYMBOL(laptop_mode);
However, when running a filesystem synchronization operation, the system will
add an additional timer to schedule more writes after the laptop_mode delay.
We don t want the state of the system to change at all while performing
hibernation, so we cancel those timers.
mm/page-writeback.c:2198
/*
* We're in laptop mode and we've just synced. The sync's writes will have
* caused another writeback to be scheduled by laptop_io_completion.
* Nothing needs to be written back anymore, so we unschedule the writeback.
*/
void laptop_sync_completion(void)
struct backing_dev_info *bdi;
rcu_read_lock();
list_for_each_entry_rcu(bdi, &bdi_list, bdi_list)
del_timer(&bdi->laptop_mode_wb_timer);
rcu_read_unlock();
As a side note, the ksys_sync function is simply called when the
system call sync is used.
fs/sync.c:111
SYSCALL_DEFINE0(sync)
ksys_sync();
return 0;
The End of Preparation
With that the system has finished preparations for hibernation.
This is a somewhat arbitrary cutoff, but next the system will begin
a full freeze of userspace to then dump memory out to an image and finally
to perform hibernation. All this will be covered in future articles!
Hibernation modes are outside of scope for this article, see the previous article for a high-level description of the different types of hibernation.
Including checking whether the algorithm is larval? Which appears to indicate that it requires additional setup, but is an interesting choice of name for such a state.
Specifically when we get to process freezing, which we ll get to in the next article in this series.
Swap space is outside the scope of this article, but in short it is a buffer on disk that the kernel uses to store memory not current in use to free up space for other things. See Swap Management for more details.
The code for this is lengthy and tangential, thus it has not been included here. If you re curious about the details of this, see kernel/power/hibernate.c:858 for the details of hibernate_quiet_exec, and drivers/nvdimm/core.c:451 for how it is used in nvdimm.
Annoyingly this code appears to use the terms console and virtual terminal interchangeably.
ioctls are special device-specific I/O operations that permit performing actions outside of the standard file interactions of read/write/seek/etc. 2
I m not entirely clear on how this flag works, this subsystem is particularly complex.
In this case a higher number is higher priority.
Or whatever the caller passes as val_down, but in this case we re specifically looking at how this is used in hibernation.
An inode refers to a particular file or directory within the filesystem. See Wikipedia for more details.
Each active filesystem is registed with the kernel through a structure known as a superblock, which contains references to all the inodes contained within the filesystem, as well as function pointers to perform the various required operations, like sync.
I m including minimal code in this section, as I m not looking to deep dive into the filesystem code at this time.
Dear Debian community,
this are my bits from DPL for August.
Happy Birthday Debian
On 16th of August Debian celebrated its 31th birthday. Since I'm
unable to write a better text than our great publicity team I'm
simply linking to their article for those who might have missed it:
https://bits.debian.org/2024/08/debian-turns-31.html
Removing more packages from unstable
Helmut Grohne argued for more aggressive package removal and
sought consensus on a way forward. He provided six examples of processes
where packages that are candidates for removal are consuming valuable
person-power. I d like to add that the Bug of the Day initiative (see
below) also frequently encounters long-unmaintained packages with popcon
votes sometimes as low as zero, and often fewer than ten.
Helmut's email included a list of packages that would meet the suggested
removal criteria. There was some discussion about whether a popcon vote
should be included in these criteria, with arguments both for and
against it. Although I support including popcon, I acknowledge that
Helmut has a valid point in suggesting it be left out.
While I ve read several emails in agreement, Scott Kitterman made
a valid point "I don't think we need more process. We just need
someone to do the work of finding the packages and filing the bugs." I
agree that this is crucial to ensure an automated process doesn t lead
to unwanted removals. However, I don t see "someone" stepping up to file
RM bugs against other maintainers' packages. As long as we have strict
ownership of packages, many people are hesitant to touch a package, even
for fixing it. Asking for its removal might be even less well-received.
Therefore, if an automated procedure were to create RM bugs based on
defined criteria, it could help reduce some of the social pressure.
In this aspect the opinion of Niels Thykier is interesting: "As
much as I want automation, I do not mind the prototype starting as a
semi-automatic process if that is what it takes to get started."
The urgency of the problem to remove packages was put by CharlesPlessy
into the words: "So as of today, it is much less work to
keep a package rotting than removing it." My observation when trying to
fix the Bug of the Day exactly fits this statement.
I would love for this discussion to lead to more aggressive removals
that we can agree upon, whether they are automated, semi-automated, or
managed by a person processing an automatically generated list
(supported by an objective procedure). To use an analogy: I ve found
that every image collection improves with aggressive pruning. Similarly,
I m convinced that Debian will improve if we remove packages that no
longer serve our users well.
DEP14 / DEP18
There are two DEPs that affect our workflow for maintaining
packages particularly for those who agree on using Git for Debian
packages. DEP-14 recommends a standardized layout for Git packaging
repositories, which benefits maintainers working across teams and makes
it easier for newcomers to learn a consistent repository structure.
DEP-14 stalled for various reasons. Sam Hartman suspected it might
be because 'it doesn't bring sufficient value.' However, the assumption
that git-buildpackage is incompatible with DEP-14 is incorrect, as
confirmed by its author, Guido G nther. As one of the two key tools
for Debian Git repositories (besides dgit) fully supports DEP-14, though
the migration from the previous default is somewhat complex.
Some investigation into mass-converting older formats to DEP-14 was
conducted by the Perl team, as Gregor Hermann pointed out..
The discussion about DEP-14 resurfaced with the suggestion of DEP-18.
Guido G nther proposed the title Encourage Continuous Integration and Merge
Request-Based Collaboration for Debian Packages ,
which more accurately reflects the DEP's technical intent.
Otto Kek l inen, who initiated DEP-18 (thank you, Otto), provided a good
summary of the current status. He also assembled a very helpful
overview of Git and GitLab usage in other Linux distros.
More Salsa CI
As a result of the DEP-18 discussion, Otto Kek l inen suggested
implementing Salsa CI for our top popcon packages.
I believe it would be a good idea to enable CI by default across Salsa
whenever a new repository is created.
Progress in Salsa migration
In my campaign, I stated that I aim to reduce the number of
packages maintained outside Salsa to below 2,000. As of March 28, 2024,
the count was 2,368. Today, it stands at 2,187 (UDD query: SELECT DISTINCT
count(*) FROM sources WHERE release = 'sid' and vcs_url not like '%salsa%' ;).
After a third of my DPL term (OMG), we've made significant progress,
reducing the amount in question (369 packages) by nearly half. I'm
pleased with the support from the DDs who moved their packages to Salsa.
Some packages were transferred as part of the Bug of the Day initiative
(see below).
Bug of the Day
As announced in my 'Bits from the DPL' talk at DebConf, I started
an initiative called Bug of the Day. The goal is to train newcomers
in bug triaging by enabling them to tackle small, self-contained QA
tasks. We have consistently identified target packages and resolved at
least one bug per day, often addressing multiple bugs in a single
package.
In several cases, we followed the Package Salvaging procedure outlined
in the Developers Reference. Most instances were either welcomed by
the maintainer or did not elicit a response. Unfortunately, there was
one exception where the recipient of the Package Salvage bug expressed
significant dissatisfaction. The takeaway is to balance formal
procedures with consideration for the recipient s perspective.
I'm pleased to confirm that the Matrix channel has seen an increase
in active contributors. This aligns with my hope that our efforts would
attract individuals interested in QA work. I m particularly pleased
that, within just one month, we have had help with both fixing bugs and
improving the code that aids in bug selection.
As I aim to introduce newcomers to various teams within Debian, I also
take the opportunity to learn about each team's specific policies
myself. I rely on team members' assistance to adapt to these policies. I
find that gaining this practical insight into team dynamics is an
effective way to understand the different teams within Debian as DPL.
Another finding from this initiative, which aligns with my goal as DPL,
is that many of the packages we addressed are already on Salsa but have
not been uploaded, meaning their VCS fields are not published. This
suggests that maintainers are generally open to managing their packages
on Salsa. For packages that were not yet on Salsa, the move was
generally welcomed.
Publicity team wants you
The publicity team has decided to resume regular meetings to coordinate
their efforts. Given my high regard for their work, I plan to attend
their meetings as frequently as possible, which I began doing with the
first IRC meeting.
During discussions with some team members, I learned that the team could
use additional help. If anyone interested in supporting Debian with
non-packaging tasks reads this, please consider introducing yourself to
debian-publicity@lists.debian.org. Note that this is a publicly archived
mailing list, so it's not the best place for sharing private
information.
Kind regards
Andreas.
All but about four hours of my Debian contributions this month were
sponsored by
Freexian. (I ended up going a bit over my 20% billing limit this month.)
You can also support my work directly via
Liberapay.
man-db and friends
I released libpipeline
1.5.8 and
man-db 2.13.0.
Since autopkgtests are great for making sure we spot regressions caused by
changes in dependencies, I added one to man-db that runs the upstream tests
against the installed package. This required some preparatory work
upstream, but otherwise was surprisingly easy to do.
OpenSSH
I fixed the various 9.8 regressions I mentioned last
month: socket activation, libssh2, and
Twisted. There were a few other regressions reported too: TCP wrappers
support,
openssh-server-udeb,
and xinetd were all broken by changes
related to the listener/per-session binary split, and I fixed all of those.
Once all that had made it through to testing, I finally uploaded the first
stage of my plan to split out GSS-API
support: there
are now openssh-client-gssapi and openssh-server-gssapi packages in
unstable, and if you use either GSS-API authentication or key exchange then
you should install the corresponding package in order for upgrades to
trixie+1 to work correctly. I ll write a release note once this has reached testing.
Multiple identical results from getaddrinfo
I expect this is really a bug in a chroot creation script somewhere, but I
haven t been able to track down what s causing it yet. My sbuild chroots,
and apparently Lucas Nussbaum s as well, have an /etc/hosts that looks
like this:
The last line clearly ought to be ::1 rather than 127.0.0.1; but things
mostly work anyway, since most code doesn t really care which protocol it
uses to talk to localhost. However, a few things try to set up test
listeners by calling getaddrinfo("localhost", ...) and binding a socket
for each result. This goes wrong if there are duplicates in the resulting
list, and the test output is typically very confusing: it looks just like
what you d see if a test isn t tearing down its resources correctly, which
is a much more common thing for a test suite to get wrong, so it took me a
while to spot the problem.
I ran into this in both python-asyncssh
(#1052788, upstream
PR) and Ruby
(ruby3.1/#1069399,
ruby3.2/#1064685,
ruby3.3/#1077462, upstream
PR). The latter took a while
since Ruby isn t one of my languages, but hey, I ve tackled much harder
side quests. I
NMUed ruby3.1 for this since it was showing up as a blocker for openssl
testing migration, but haven t done the other active versions (yet, anyway).
OpenSSL vs. cryptography
I tend to care about openssl migrating to testing promptly, since openssh
uploads have a habit of getting stuck on it otherwise.
Debian s OpenSSL packaging recently split out some legacy code (cryptography
that s no longer considered a good idea to use, but that s sometimes needed
for compatibility) to an openssl-legacy-provider package, and added a
Recommends on it. Most users install Recommends, but package build
processes don t; and the Python cryptography package requires this code
unless you set the CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1 environment variable,
which caused a bunch of packages that build-depend on it to fail to build.
After playing whack-a-mole setting that environment variable in a few
packages build process, I decided I didn t want to be caught in the middle
here and filed an upstream
issue to see if I could
get Debian s OpenSSL team and cryptography s upstream talking to each other
directly. There was some moderately spirited discussion and the issue
remains open, but for the time being the OpenSSL team has effectively
reverted the
change
so it s no longer a pressing problem.
GCC 14 regressions
Continuing from last month, I fixed build
failures in pccts (NMU) and
trn4.
Python team
I upgraded alembic, automat, gunicorn, incremental, referencing, pympler
(fixing compatibility with Python >=
3.10), python-aiohttp, python-asyncssh
(fixing CVE-2023-46445,
CVE-2023-46446, and
CVE-2023-48795), python-avro,
python-multidict (fixing a build failure with GCC
14), python-tokenize-rt, python-zipp,
pyupgrade, twisted (fixing CVE-2024-41671
and CVE-2024-41810), zope.exceptions,
zope.interface, zope.proxy, zope.security, and zope.testrunner to new
upstream versions. In the process, I added myself to Uploaders for
zope.interface; I m reasonably comfortable with the Zope Toolkit and I seem
to be gradually picking up much of its maintenance in Debian.
A few of these required their own bits of yak-shaving:
GNOME is one of the most widely used free and open-source desktop environments!
Your native language is Marathi and you are using GNOME as your
desktop environment? Then me as the coordinator for the Marathi translation
team in GNOME is excited to invite you to become part of the team who is
working on translating the GNOME Desktop into Marathi!
By this and contributing to the translation of GNOME into Marathi you would
be a member of an important project and you can help to make it more
accessible to Marathi speakers worldwide and help also to keep our language
alive in the open source world.
Why Should You Contribute?
Promote Your Language
By translating GNOME into Marathi, you help
to preserve and promote our beautiful language in the digital world.
Learn and Grow
Contributing to open-source projects like GNOME is a
great way to improve your language and technical skills, network with
like-minded individuals, and gain recognition in the global open-source
community.
Give Back to the Community
This is an opportunity to contribute to a project that has a significant
impact on users around the world. Your work will enable Marathi speakers
to use technology in their native language.
Who Can Contribute?
You don t need to be a professional translator to join us! If you are fluent in
Marathi and have a basic understanding of English, your contributions will be
invaluable. Whether you re a student, a professional, or just someone
passionate about your language, your help is needed and really appreciated!
How To Start Translating?
Once you re familiar with the tools, you can easily begin translating. We
have a list of untranslated strings waiting for your contribution!
How To Join The Team?
Follow these steps to join the Marathi translation team for GNOME and start
contributing:
Step 2: If you re a new user, click on the Create Account option to sign up.
Step 3: Once you ve created your account, log in with your credentials.
Step 4: After logging in, click the Join button to become a translator for the Marathi team.
Step 5: You ll now see a list of different modules that need translation. Choose one of the files that interests you and download it to your computer.
Step 6: Translate the content locally on your computer. Once you re done, return to the website, click Browse, and submit your translated file.
Get Familiar with the Additional Tools
Varnam
If you re not used to typing in Marathi, you can still contribute using the
Varnam website, a free and open-source tool that converts English text into
Marathi. Here s how you can get started:
Step 2: Click on the Try Now button on the website.
Step 3: In the language selection menu, choose Marathi as
your desired language.
Step 4: Now you can start typing in English, and Varnam will
automatically convert your text into Marathi. If you need more guidance,
there s a help window available on the site that you can explore for
additional support.
Need Help Or You Have Questions?
If you have any doubts or need further assistance how you get started with
translating GNOME into Marathi, don t hesitate to reach out. I m here to help
you on every step of the way!
You can connect with me directly at
kalyaniknkr@gmail.com Whether you need
technical support, guidance on using the tools, or just want to discuss the
project, feel free to get in touch.
Let s work together to make GNOME accessible to Marathi speakers around the
world. Your contributions are always invaluable, and I look forward to
welcoming you to our team!
Thank you for your interest and support!
It's clear that Twitter has been circling the drain for years, but
things have been especially bad in recent times. I haven't quit (I
have some sympathy with the viewpoint don't cede territory to
fascists) but I try to read it much less, and I certainly post
much less.
Especially at the moment, I really appreciate distractions.
Last time I wrote about Mastodon (by which
I meant the Fediverse1), I was looking for a new instance to
try. I settled on Debian's social
instance2. I'm now trying to put
any energy I might spend engaging on Twitter, into engaging in the Fediverse
instead. (You can follow me via the handle @jon@dow.land, I think, which
should repoint to my actual handle, @jmtd@pleroma.debian.social.)
There are other potential successors to Twitter: two big ones are Bluesky and
Facebook-owned Threads. They are effectively cookie-cutter copies of the
Twitter model, and so, we will repeat the same mistakes there. Sadly I see the
majority of communities and sub-cultures I follow are migrating to one or the
other of these.
The Fediverse (or the Mastodon-ish bits of it) should avoid the fate of
Twitter. JWZ puts it better and more succinctly than I
can.
The Fedi experience is, sadly, pretty clunky. So I want to try and write a bit
from time to time with tips and tricks that might improve people's experiences.
First up, something I discovered only today about Mastodon instances. As JWZ
noted, If you are worried about picking the "right" Mastodon instance,
don't. Just spin the wheel.. You can spend too much time trying to guess
a good answer to this. Better to just get started.
At the same time, individual instances are supposed to cater to specific
niches. So it could be useful to sample the public posts from an entire
instance. For example, to find people to follow, or decide to hop over to that
instance yourself. You can't (I think) follow an entire instance from within
yours, but, they usually have a public page which shows you the latest
traffic.
For example, the infosec-themed instance infosec.exchange has one here:
https://infosec.exchange/public/localThese pages don't provide RSS or Atom feeds3, sadly. I hope that's
on the software's roadmap, and hasn't been spurned for ideological reasons.
For now at least, OpenRSS provide RSS/Atom feeds for
many Mastodon instances. For example, an RSS/Atom feed of the above:
https://openrss.org/infosec.exchange/public/local
One can add these feeds to your Feed reader and over time get a flavour for
the kind of discourse that takes place on given instances.
I think the OpenRSS have to manually add Mastodon instances to their service.
I tried three instances and only one (infosec.exchange) worked. I'm not sure
but I think trying an instance that doesn't work automatically puts it on
OpenRSS's backlog.
the Fediverse-versus-Mastodon nomenclature problem is just the
the tip of the iceberg, in terms of adoption problems. Mastodon provides a
twitter-like service that participates in the Fediverse. But it isn't correct
to call the twitter-like service "Mastodon" because other softwares also
participate in/provide that service. And it's not correct to call it "Fediverse"
because that describes a bigger thing, with e.g. youtube clones also taking
part. I'm not sure what the right term should be for "the twitter-like thing".
Also, everything I wrote here is probably subtly wrong.
Debian's instance actually runs Pleroma, an alternative to
Mastodon. Why should it matter? I think it's healthy for there to be more
than one implementation in an open ecosystem. However the experience can be
janky, as the features don't perfectly align, some Mastodon features/APIs are
not documented/standardised/etc.
I have to remind myself that the concept of RSS/Atom feeds and Feedreaders
might need explaining to a modern audience too. Perhaps in another blog post.
With news like this one cited by Bruce Schneier [1] people are asking whether it s worth using Secure Boot.
Regarding the specific news article, this is always a risk with distributed public key encryption systems. Lose control of one private key and attackers can do bad things. That doesn t make it bad it just makes it less valuable. If you want to setup a system for a government agency, bank, or other high value target then it s quite reasonable to expect an adversary to purchase systems of the same make and model to verify that their attacks will work. If you want to make your home PC a little harder to attack then you can expect that the likely adversaries won t bother with such things. You don t need security to be perfect, making a particular attack slightly more difficult than other potential attacks gives a large part of the benefit.
The purpose of Secure Boot is to verify the boot loader with a public key signature and then have the boot loader verify the kernel. Microsoft signs the shim that is used by each Linux distribution to load GRUB (or another boot loader). So when I configure a Debian system with Secure Boot enabled that doesn t stop anyone from booting Ubuntu. From the signatures on the boot loader etc there is no difference from my Debian installation and a rescue image from Debian, Ubuntu, or another distribution booted by a hostile party to do things against my interests. The difference between the legitimate OS image and malware is a matter of who boots it and the reason for booting it.
It is possible to deconfigure Microsoft keys from UEFI to only boot from your own key, this document describes what is necessary to do that [2]. Basically if you boot without using any option ROMs (which among other things means the ROM from your video card) then you can disable the MS keys.
If it s impossible to disable the MS keys that doesn t make it impossible to gain a benefit from the Secure Boot process. You can use a block device decryption process that involves a signature of the kernel and the BIOS being used as part of the decryption for the device. So if a system is booted with the wrong kernel and the user doesn t recognise it then they will find that they can t unlock the device with the password. I think it s possible on some systems to run the Secure Boot functionality in a non-enforcing mode such that it will use a bootloader without a valid signature but still use the hash for TPM calculations, that appears impossible on my Thinkpad Yoga Gen3 which only has enabled and disabled as options but should work on Dell laptops which have an option to run Secure Boot in permissive mode.
I believe that the way of the future is to use something like EFIStub [3] to create unified kernel images with a signed kernel, initrd, and command-line parameters in a single bundle which can be loaded directly by the UEFI BIOS. From the perspective of a distribution developer it s good to have many people using the current standard functionality of shim and GRUB for EFI as a step towards that goal.
CloudFlare has a good blog post about Linux kernel hardening [4]. In that post they cover the benefits of a full secure boot setup (which is difficult at the current time) and the way that secure boot enables the lockdown module for kernel integrity. When Secure Boot is detected by the kernel it automatically enables lockdown=integrity functionality (see this blog post for an explanation of lockdown [5]). It is possible to enable this by putting lockdown=integrity on the kernel command line or lockdown=confidentiality if you want even more protection, but it happens by default with Secure Boot. Secure Boot is something you can set to get a selection of security features enabled and get a known minimum level of integrity even if the signatures aren t used for anything useful, restricting a system to only boot kernels from MS, Debian, Ubuntu, Red Hat, etc is not useful.
For most users I think that Secure Boot is a small increase in security but testing it on a large number of systems allows increasing the overall security of operating systems which benefits the world. Also I think that having features like EFIStub usable for a large portion of the users (possibly the majority of users) is something that can be expected to happen in the lifetime of hardware being purchased now. So ensuring that Secure Boot works with GRUB now will facilitate using EFIStub etc in future years.
The Secure Boot page on the Debian wiki is worth reading, and also worth updating for people who want to contribute [6].
Planing A Self Organized Solo Trip
You know the movie Queen?
The actor Kangana Ranaut plays in that movie the role of Rani Mehra, a
24-year-old Punjabi woman, who was a simple, homely girl that was always
reliant on her family. Similar to Rani I too rarely ventured out without my
parents and often needed my younger sibling by my side. Inspired by her
transformation, I decided it was time to take control of my own story and
discover who I truly am.
Trip Requirements
My First Passport
The journey began with a significant first step: Obtaining my first
passport
Never having had one before, I scheduled the nearest available interview
date on June 29 2022. This meant traveling to Solapur, a city 309 km from my
hometown, accompanied by my father. After successfully completing the
interview, I received my passport on July 14 2022.
Select A Country, Booking Flights And Accommodation
Excited and ready to embark on my adventure, I planed trip to Albania and
booked the flight tickets. Why? I had heard from friends that it was a
beautiful European country with beaches and other attractions, and
importantly, it didn t require a visa for Indian citizens and was more
affordable than other European destinations. Before heading to Albania, I
planned a overnight stop in Abu Dhabi with a transit visa, thanks to friend
who knew the process for obtaining it.
Some of my friends did travel also to Europe at the same time and quite
close to my plannings, but that I realized just later the trip.
Day 1, Starting The Experience
On July 20, 2022, I started my journey by traveling from Pune, Maharashtra,
to Delhi, where my brother lives. He came to see me off at the airport,
adding a touch of warmth and support to the beginning of my solo adventure.
Upon arriving in Delhi, with my next flight scheduled for July 21, I stayed
at a backpacker hostel named Zostel, Paharganj, Delhi to rest.
During my stay, I noticed that many travelers at the hostel carried
rucksacks, which sparked a desire in me to get one for my own trip to
Europe. Up until then, I had always shopped with my mom and had never bought
anything on my own. Inspired by the travelers, I set out to find a suitable
rucksack. I traveled alone by metro from Paharganj to Rohini to visit a
Decathlon store, where I purchased a 50-liter rucksack. This was a
significant step in preparing for my European adventure and marked a
milestone in my journey of self reliance.
Day 2, Flying To Abu Dhabi
The following day, July 21 2024, I had a flight to Abu Dhabi. I spent the
night at the hostel to rest before my journey. On the day of the flight, I
needed to reach the airport by 3 PM, and a friend kindly came to drop me
off. With my rucksack packed and excitement building, I was ready for the
next leg of my adventure.
When we arrived at the airport, my friend saw me off, marking the start of
my international journey. With mom made spices, chutneys, and chilly flakes
packed for comfort, I completed my immigration process in about two and a
half hours. I then settled at the gate for my flight, feeling a mix of
excitement and anxiety as thoughts raced through my mind.
To ease my nerves, I struck up a conversation with a man seated nearby who was
also traveling to Abu Dhabi for work. He provided helpful information about
safety and transportation in Abu Dhabi, which reassured me. With the boarding
process complete and my anxiety somewhat eased. I found my window seat on the
flight and settled in, excited for the journey ahead. Next to me was a young
man from Ranchi(Zarkhand, India), heading to Abu Dhabi for work at a mining
factory. We had an engaging conversation about work culture in Abu Dhabi and
recruitment from India.
Upon arriving in Abu Dhabi, I completed my transit, collected my luggage, and
began finding my way to the hotel Premier Inn AbuDhabi,
which was in the airport area. To my surprise, I ran into the same man from the
flight, now in a cab. He kindly offered to drop me at my hotel, which I gladly
accepted since navigating an unfamiliar city with a short acquaintance felt
safer.
At the hotel gate, he asked if I had local currency (Dirhams) for payment,
as sometimes online transactions can fail. That hadn t crossed my mind, and
I realized I might be left stranded if a transaction failed. Recognizing his
help as a godsend, I asked if he could lend me some Dirhams, promising to
transfer the amount later. He kindly assured me to pay him back once I
reached the hotel room. With that relief, I checked into the hotel, feeling
deeply grateful for the unexpected assistance and transferred the money to
him after getting to my room.
Day 3, Flying And Arrive In Tirana
Once in the hotel room, I found it hard to sleep, anxious about waking up on
time for my flight. I set an alarm to wake up early, but my subconscious mind
kept me alert, and I woke up before the alarm went off. I got freshened up and
went down for breakfast, where I found some vegetarian options like
Idli-Sambar and bread with butter, along with some morning tea.
After breakfast, I headed back to the airport, ready to catch my flight to my
final destination: Tirana, Albania.
I reached Tirana, Albania after a six hours flight, feeling exhausted and I was
suffering from a headache. The air pressure had blocked my ears, and jet lag
added to my fatigue. After collecting my checked luggage, I headed to the first
ATM machine at the airport. Struggling to insert my card, I asked a nearby
gentleman for help. He tried his best, but my card got stuck inside the
machine. Panic set in as I worried about how I would survive without money.
Taking a deep breath, I found an airport employee and explained the situation.
The gentleman stayed with me, offering support and repeatedly apologizing for
his mistake. However, it wasn t his fault, the ATM was out of order, which I
hadn t noticed. My focus was solely on retrieving my ATM card. The airport
employee worked diligently, using a hairpin to carefully extract my card.
Finally, the card was freed, and I felt an immense sense of relief, grateful
for the help of these kind strangers. I used another ATM, successfully withdrew
money, and then went to an airport mobile SIM shop to buy a new SIM card for
local internet and connectivity.
Day 4, Arriving In Tirana, Facing Challenges In A Foreign Country
I had booked a stay at a backpacker hostel near the city center of Tirana.
After sorting out the ATM and SIM card issues, I searched for a bus or any
transport to get there. It was quite late, around 8:30 PM, and being in a
new city, I was in a hurry. I saw a bus nearly leaving the airport, stopped
it, and asked if it went to the city center. They gave me the green flag, so
I boarded the airport service bus and reached the city center.
Feeling very tired, I discovered that the hostel was about an hour and a
half away by walking. Deciding to take a cab, I faced a challenge as the
driver couldn t understand my English or accent. Using a mobile translator
to convert my address from English to Albanian, I finally communicated my
destination to him. With that sorted out, I headed to the
Blue Door Backpacker Hostel and arrived around 9 PM,
relieved to have finally reached my destination and I checked in.
I found my top bunk bed, only to realize I had booked a mixed-gender
dormitory. This detail had completely escaped my notice during the booking
process. I felt unsure about how to handle the situation. Coincidentally,
my experience mirrored what Kangana faced in the movie Queen .
Feeling acidic due to an empty stomach and the exhaustion of heavy
traveling, I wasn t up to cooking in the hostel s kitchen.
I asked the front desk about the nearest restaurant. It was nearly 9:30 PM,
and the streets were deserted. To avoid any mishaps like in the movie
Queen, I kept my passport securely locked in my bag, ensuring it wouldn t
be a victim of theft.
Venturing out for dinner, I felt uneasy on the quiet streets. I eventually
found a restaurant recommended by the hostel, but the menu was almost
entirely non-vegetarian. I struggled to ask about vegetarian options and was
uncertain if any dishes contained eggs, as some people consider eggs to be
vegetarian. Feeling frustrated and unsure, I left the restaurant without
eating.
I noticed a nearby grocery store that was about to close and managed to get
a few extra minutes to shop. I bought some snacks, wafers, milk, and tea
bags (though I couldn t find tea powder to make Indian-style tea). Returning
to the hostel, I made do with wafers, cookies, and milk for dinner. That day
was incredibly tough for me, I filled with exhaustion and struggle in a new
country, I was on the verge of tears .
I made a video call home before sleeping on the top bunk bed. It was a new
experience for me, sharing a room with both unknown men and women. I kept my
passport safe inside my purse and under my pillow while sleeping, staying
very conscious about its security.
Day 5, Exploring Nearby Places
I woke up the next day at noon. After having some coffee, the hostel
management girl asked if I wanted breakfast. She offered curd with
cornflakes, which I refused because I don t like curd. Instead, I ordered a
pizza from a vegetarian pizza place with her help, and I started feeling
better.
I met some people in the hostel, some from Syria and others from Italy. I
struggled to understand their accents but kept pushing myself to get
involved in their discussions. Despite the challenges, I felt more at ease
and was slowly adapting to my new environment.
I went out from the hostel in the evening to buy some vegetables to cook
something. I searched for shops and found some potatoes, tomatoes, and rice. I
decided to cook Khichdi, an Indian dish made with rice, and added
some chili flakes I brought from home. After preparing my dinner, I ate and
then went to sleep again.
Day 6, Tiranas Recent History
The next day, I planned to explore the city and visited Bunkart-1,
a fascinating museum in a massive underground bunker from the communist era.
Originally built as a shelter for Albania s political and military elite, it
now offers a unique glimpse into the country s history under Enver Hoxha s
oppressive regime. The museum s exhibits include historical artifacts,
photographs, and multimedia displays that detail the lives of Albanians during
that time. Walking through the dimly lit corridors, I felt the weight of
history and gained a deeper understanding of Albania s past.
Day 7-8, Meeting Friends From India
The next day, I accidentally met with Chirag, who was returning from the
Debian Conference 2022 held in Prizren, Kosovo, and staying at the same
hostel. When I encountered him, he was talking on the phone, and I
recognized he was Indian by his accent. I introduced myself, and we
discovered we had some mutual friends.
Chirag told me that our common friend, Raju, was also coming to stay at the
hostel the next day. This news made me feel relaxed and happy to have known
people around. When Raju arrived, the three of us, Chirag, Raju, and
I planned to have dinner at an Indian restaurant and explore Tirana city. I
had a great time talking and enjoying their company.
Day 9-10, Meeting More Friends
Raju had a ticket to leave soon, so Chirag and I made a plan to visit
Shkod r and the nearby Komani Lake for kayaking. We started our journey
early in the morning by bus and reached Shkod r. There, we met new friends
from the conference, Pavit and Abraham, who were already there. We had
dinner together and enjoyed an ice cream treat from Chirag.
Day 12, Kayaking And Say Good Bye To Friends
The next day, Pavit and Abraham had a flight back to India, so Chirag and I
went to Komani Lake. We had an adventurous time kayaking, even though
neither of us knew how to swim. We took a ferry through the backwaters to
the island on Komani Lake and enjoyed a fantastic adventure together. After
our trip, Chirag returned to Tirana for his flight back to India, leaving me
to continue my journey alone.
Day 13, Climbing Rozafa Castel
By stopping at Shkod r, I visited Rozafa Castle. Despite the language
barrier, as most locals only spoke Albanian, people around me guided me
correctly on how to get there. At times, I used applications like Google
Translate to communicate. To read signs or hotel menus, I used Google
Photos' language converter. I even used the audio converter to understand
and speak some basic Albanian phrases.
I took a bus from Shkod r to the southern part of Albania, heading to
Sarand . The journey lasted about five to six hours, and I had booked a stay
at Mona s Hostel. Upon arrival, I met Eliza from America, and we went
together to Ksamil Beach, spending a wonderful day there.
Day 14, Vlora Beach: Beach Side Cycling
Next, I traveled to Vlor , where I stayed for one day. During my time there, I
enjoyed beach side cycling with a cycle provided by the hostel owner and spent
some time feeding fish. I also met a fellow traveler from Delhi who had brought
along some preserved Indian curry. He kindly shared it with me, which was a
welcome change after nearly 15 days without authentic Indian cuisine, except
for what I had cooked myself in various hostels.
Day 15-16 Visiting Durress, Travelling Back To Tirana
I then visited Durr s, exploring its beautiful beaches, before heading back
to Tirana one day before my flight home. On the day of my flight, my alarm
didn t go off, and I woke up late at the hostel. In a frantic rush, I packed
everything in just five minutes and dashed toward the city center to catch
the bus to the airport. If I had been just five minutes later, I would have
missed the bus. Thankfully, I managed to stop it just in time and began my
journey back home, reflecting on the incredible adventure I had experienced.
Fortunately, I wasn t late; I arrived at the airport just in time. After
clearing immigration, I boarded my flight, which had a layover in Warsaw,
Poland. The journey from Tirana to Warsaw took about two and a half hours,
followed by a seven to eight-hour flight from Poland back to India. Once I
arrived in Delhi, I returned to Zostel and booked a train ticket to
Aurangabad for the next three days.
Backview
This trip was an incredible adventure for me. I never imagined I could
accomplish something like this, but I did. Meeting diverse people,
experiencing different cultures, and learning so much made this journey
truly unforgettable.
Looking back, I realize how much I ve grown from this experience. Although I
may have more opportunities to travel abroad in the future, this trip will
always hold a special place in my heart. The memories I made and the
incredible people I met along the way are irreplaceable.
This experience goes beyond what I can express through this blog or words;
it was incredibly precious to me. Every moment of this journey is etched in
my memory, and I am grateful for every part of it.
Welcome to the July 2024 report from the Reproducible Builds project!
In our reports, we outline what we ve been up to over the past month and highlight news items in software supply-chain security more broadly. As always, if you are interested in contributing to the project, please visit our Contribute page on our website.
Table of contents:
Reproducible Builds Summit 2024
Last month, we were very pleased to announce the upcoming Reproducible Builds Summit, set to take place from September 17th 19th 2024 in Hamburg, Germany. We are thrilled to host the seventh edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin and Athens. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving.
If you re interesting in joining us this year, please make sure to read the event page, which has more details about the event and location. We are very much looking forward to seeing many readers of these reports there.
a bootstrappable build is one that builds existing software from scratch for example, building GCC without relying on an existing copy of GCC. In 2023, the Guix project announced that the project had reduced the size of the binary bootstrap seed needed to build its operating system to just 357-bytes not counting the Linux kernel required to run the build process.
The article goes onto to describe that now, the live-bootstrap project has gone a step further and removed the need for an existing kernel at all. and concludes:
The real benefit of bootstrappable builds comes from a few things. Like reproducible builds, they can make users more confident that the binary packages downloaded from a package mirror really do correspond to the open-source project whose source code they can inspect. Bootstrappable builds have also had positive effects on the complexity of building a Linux distribution from scratch [ ]. But most of all, bootstrappable builds are a boon to the longevity of our software ecosystem. It s easy for old software to become unbuildable. By having a well-known, self-contained chain of software that can build itself from a small seed, in a variety of environments, bootstrappable builds can help ensure that today s software is not lost, no matter where the open-source community goes from here
Towards Idempotent Rebuilds?Trisquel developer Simon Josefsson wrote an interesting blog post comparing the output of the .deb files from our tests.reproducible-builds.org testing framework and the ones in the official Debian archive. Following up from a previous post on the reproducibility of Trisquel, Simon notes that typically [the] rebuilds do not match the official packages, even when they say the package is reproducible , Simon correctly identifies that the purpose of [these] rebuilds are not to say anything about the official binary build, instead the purpose is to offer a QA service to maintainers by performing two builds of a package and declaring success if both builds match.
However, Simon s post swiftly moves on to announce a new tool called debdistrebuild that performs rebuilds of the difference between two distributions in a GitLab pipeline and displays diffoscope output for further analysis.
Reproducible Central is an initiative that curates a list of reproducible Maven libraries, but the list is limited and challenging to maintain due to manual efforts. [We] investigate the feasibility of automatically finding the source code of a library from its Maven release and recovering information about the original release environment. Our tool, AROMA, can obtain this critical information from the artifact and the source repository through several heuristics and we use the results for reproduction attempts of Maven packages. Overall, our approach achieves an accuracy of up to 99.5% when compared field-by-field to the existing manual approach [and] we reveal that automatic reproducibility is feasible for 23.4% of the Maven packages using AROMA, and 8% of these packages are fully reproducible.
Nichita Morcotilo reached out to the community, first to share their efforts to build reproducible packages cross-platform with a new build tool called rattler-build, noting that as you can imagine, building packages reproducibly on Windows is the hardest challenge (so far!) . Nichita goes onto mention that the Apple ecosystem appears to be using ZERO_AR_DATE over SOURCE_DATE_EPOCH. []
Roland Clobus announced that the Debian bookworm 12.6 live images are nearly reproducible , with more detail in the post itself and input in the thread from other contributors.
Daniel Gr ber asked for help in getting the Yosys documentation to build reproducibly, citing issues in inter alia the PDF generation causing differing CreationDate metadata values.
James Addison continued his long journey towards getting the Sphinx documentation generator to build reproducible documentation. In this thread, James concerns himself with the problem that even when SOURCE_DATE_EPOCH is configured, Sphinx projects that have configured their copyright notices using dynamic elements can produce nonsensical output under some circumstances. James query ended up generating a number of replies.
Allen gunner Gunner posted a brief update on the progress the core team is making towards introducing a Code of Conduct (CoC) such that it is in place in time for the RB Summit in Hamburg in September . In particular, gunner asks if you are interested in helping with CoC design and development in the weeks ahead, simply email rb-core@lists.reproducible-builds.org and let us know . []
[S]oftware repositories are a vital component of software development and release, with packages downloaded both for direct use and to use as dependencies for other software. Further, when software is updated due to patched vulnerabilities or new features, it is vital that users are able to see and install this patched version of the software. However, this process of updating software can also be the source of attack. To address these attacks, secure software update systems have been proposed. However, these secure software update systems have seen barriers to widespread adoption. The Update Framework (TUF) was introduced in 2010 to address several attacks on software update systems including repository compromise, rollback attacks, and arbitrary software installation. Despite this, compromises continue to occur, with millions of users impacted by such compromises. My work has addressed substantial challenges to adoption of secure software update systems grounded in an understanding of practical concerns. Work with industry and academic communities provided opportunities to discover challenges, expand adoption, and raise awareness about secure software updates. [ ]
Since build-tools >= 35.0.0-rc1, backwards-incompatible changes to apksigner break apksigcopier as it now by default forcibly replaces existing alignment padding and changed the default page alignment from 4k to 16k (same as Android Gradle Plugin >= 8.3, so the latter is only an issue when using older AGP). []
She documented multiple available workarounds and filed a bug in Google s issue tracker.
Lastly, diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb uploaded version 272 and Mattia Rizzolo uploaded version 273 to Debian, and the following changes were made as well:
Chris Lamb:
Ensure that the convert utility is from ImageMagick version 6.x. The command-line interface has seemingly changed with the 7.x series of ImageMagick. []
Factor out version detection in test_jpeg_image. []
Correct the import of the identify_version method after a refactoring change in a previous commit. []
Move away from using DSA OpenSSH keys in tests as support has been deprecated and removed in OpenSSH version 9.8p1. []
Move to assert_diff in the test_openssh_pub_key package. []
Update copyright years. []
Mattia Rizzolo:
Add support for ffmpeg version 7.x which adds some extra context to the diff. []
Rework the handling of OpenSSH testing of DSA keys if OpenSSH is strictly 9.7, and add an OpenSSH key test with a ed25519-format key [][][]
Temporarily disable a few packages that are not available in Debian testing. [][]
Stop ignoring the results of Debian testing in the continuous integration system. []
Adjust options in debian/source to make sure not to pack the Python sdist directory into the binary Debian package. []
Adjust Lintian overrides. []
Website updates
There were a number of improvements made to our website this month, including:
Bernhard M. Wiedemann updated the SOURCE_DATE_EPOCH page to include instructions on how to create reproducible .zip files from within Python using the zipfile module. []
Chris Lamb fixed a potential duplicate heading on the Projects page. []
Fay Stegerman added rbtlog to the Tools page [] and IzzyOnDroid to the Projects page [], also ensuring that the latter page was always sorted regardless of the ordering within the input data files. []
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Reproducibility testing framework
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In July, a number of changes were made by Holger Levsen, including:
Grant bremner access to the ionos7 node. [][]
Perform a dummy change to force update of all jobs. [][]
In addition, Vagrant Cascadian performed some necessary node maintenance of the underlying build hosts. []
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
The Combatants
In the red corner, weighing in at nah, I m not going to do that schtick.
The plaintiff in the case is Alegeus Technologies, LLC, a Delaware Corporation that, according to their filings, is a leading provider of a business-tobusiness, white-label funding and payment platform for healthcare carriers and third-party administrators to administer consumer-directed employee benefit programs .
Not being subject to the US bonkers health care system, I have only a passing familiarity with the sorts of things they do, but presumably it involves moving a lot of money around, which is sometimes important.
The defendant is DigiCert, a CA which, based on analysis I ve done previously, is the second-largest issuer of WebPKI certificates by volume.
The History
According to a recently opened Mozilla CA bug, DigiCert found an issue in their domain control validation workflow, that meant it may have been possible for a miscreant to have certificates issued to them that they weren t legitimately entitled to.
Given that validating domain names is basically the YOU HAD ONE JOB! of a CA, this is a big deal.
The CA/Browser Forum Baseline Requirements (BRs) (which all CAs are required to adhere to, by virtue of their being included in various browser and OS trust stores), say that revocation is required within 24 hours when [t]he CA obtains evidence that the validation of domain authorization or control for any Fully Qualified Domain Name or IP address in the Certificate should not be relied upon (section 4.9.1.1, point 5).
DigiCert appears to have at least tried to do the right thing, by opening the above Mozilla bug giving some details of the problem, and notifying their customers that their certificates were going to be revoked.
One may quibble about how fast they re doing it, but they re giving it a decent shot, at least.
A complicating factor in all this is that, only a touch over a month ago, Google Chrome announced the removal of another CA, Entrust, from its own trust store program, citing a pattern of compliance failures, unmet improvement commitments, and the absence of tangible, measurable progress in response to publicly disclosed incident reports .
Many of these compliance failures were failures to revoke certificates in a timely manner.
One imagines that DigiCert would not like to gain a reputation for tardy revocation, particularly at the moment.
The Legal Action
Now we come to Alegeus Technologies.
They ve opened a civil case whose first action is to request the issuance of a Temporary Restraining Order (TRO) that prevents DigiCert from revoking certificates issued to Alegeus (which the court has issued).
This is a big deal, because TROs are legal instruments that, if not obeyed, constitute contempt of court (or something similar) and courts do not like people who disregard their instructions.
That means that, in the short term, those certificates aren t getting revoked, despite the requirement imposed by root stores on DigiCert that the certificates must be revoked.
DigiCert is in a real rock / hard place situation here: revoke and get punished by the courts, or don t revoke and potentially (though almost certainly not, in the circumstances) face removal from trust stores (which would kill, or at least massively hurt, their business).
The reasons that Alegeus gives for requesting the restraining order is that [t]o
Reissue and Reinstall the Security Certificates, Alegeus must work with and
coordinate with its Clients, who are required to take steps to rectify the
certificates. Alegeus has hundreds of such Clients. Alegeus is generally
required by contract to give its clients much longer than 24 hours notice
before executing such a change regarding certification.
In the filing, Alegeus does acknowledge that DigiCert is a voluntary member of the Certification Authority Browser Forum (CABF), which has bylaws stating that certificates with an issue in their domain validation must be revoked within 24 hours.
This is a misstatement of the facts, though.
It is the BRs, not the CABF bylaws, that require revocation, and the BRs apply to all CAs that wish to be included in browser and OS trust stores, not just those that are members of the CABF.
In any event, given that Alegeus was aware that DigiCert is required to revoke certificates within 24 hours, one wonders why Alegeus went ahead and signed agreements with their customers that required a lengthy notice period before changing certificates.
What complicates the situation is that there is apparently a Master Services Agreement (MSA) that states that it constitutes the entire agreement between the parties and that MSA doesn t mention certificate revocation anywhere relevant.
That means that it s not quite so cut-and-dried that DigiCert does, in fact, have the right to revoke those certificates.
I d expect a lot of update to your Master Services Agreement emails to be going out from DigiCert (and other CAs) in the near future to clarify this point.
Not being a lawyer, I can t imagine which way this case might go, but there s one thing we can be sure of: some lawyers are going to able to afford that trip to a tropical paradise this year.
The Security Issues
The requirement for revocation within 24 hours is an important security control in the WebPKI ecosystem.
If a certificate is misissued to a malicious party, or is otherwise compromised, it needs to be marked as untrustworthy as soon as possible.
While revocation is far from perfect, it is the best tool we have.
In this court filing, Alegeus has claimed that they are unable to switch certificates with less than 24 hours notice (due to contractual SLAs ).
This is a pretty big problem, because there are lots of reasons why a certificate might need to be switched out Very Quickly.
As a practical example, someone with access to the private key for your SSL certificate might decide to use it in a blog post.
Letting that sort of problem linger for an extended period of time might end up being a Pretty Big Problem of its own.
An organisation that cannot respond within hours to a compromised certificate is playing chicken with their security.
The Takeaways
Contractual obligations that require you to notify anyone else of a certificate (or private key) changing are bonkers, and completely antithetical to the needs of the WebPKI.
If you have to have them, you re going to want to start transitioning to a private PKI, wherein you can do whatever you darn well please with revocation (or not).
As these sorts of problems keep happening, trust stores (and hence CAs) are going to crack down on this sort of thing, so you may as well move sooner rather than later.
If you are an organisation that uses WebPKI certificates, you ve got to be able to deal with any kind of certificate revocation event within hours, not days.
This basically boils down to automated issuance and lifecycle management, because having someone manually request and install certificates is terrible on many levels.
There isn t currently a completed standard for notifying subscribers if their certificates need premature renewal (say, due to needing to be revoked), but the ACME Renewal Information Extension is currently being developed to fill that need.
Ask your CA if they re tracking this standards development, and when they intend to have the extension available for use.
(Pro-tip: if they say we ll start doing development when the RFC is published , run for the hills; that s not how responsible organisations work on the Internet).
The Givings
If you ve found this helpful, consider shouting me a refreshing beverage.
Reading through legal filings is thirsty work!
With the work that has been done in the debian-installer/netcfg merge-proposal !9 it is possible to install a standard Debian system, using the normal Debian-Installer (d-i) mini.iso images, that will come pre-installed with Netplan and all network configuration structured in /etc/netplan/.
In this write-up, I d like to run you through a list of commands for experiencing the Netplan enabled installation process first-hand. Let s start with preparing a working directory and installing the software dependencies for our virtualized Debian system:
Next we ll prepare a VM, by copying the EFI firmware files, preparing some persistent EFIVARs file, to boot from FS0:\EFI\debian\grubx64.efi, and create a virtual disk for our machine:
Finally, let s launch the debian-installer using a preseed.cfg file, that will automatically install Netplan (netplan-generator) for us in the target system. A minimal preseed file could look like this:
For this demo, we re installing the full netplan.io package (incl. the interactive Python CLI), as well as the netplan-generator package and systemd-resolved, to show the full Netplan experience. You can choose the preseed file from a set of different variants to test the different configurations:
We re using the linux kernel and initrd.gz here to be able to pass the preseed URL as a parameter to the kernel s cmdline directly. Launching this VM should bring up the official debian-installer in its netboot/gtk form:
Now you can click through the normal Debian-Installer process, using mostly default settings. Optionally, you could play around with the networking settings, to see how those get translated to /etc/netplan/ in the target system.
After you confirmed your partitioning changes, the base system gets installed. I suggest not to select any additional components, like desktop environments, to speed up the process.
During the final step of the installation (finish-install.d/55netcfg-copy-config) d-i will detect that Netplan was installed in the target system (due to the preseed file provided) and opt to write its network configuration to /etc/netplan/ instead of /etc/network/interfaces or /etc/NetworkManager/system-connections/.
Done! After the installation finished, you can reboot into your virgin Debian Sid/Trixie system.
To do that, quit the current Qemu process, by pressing Ctrl+C and make sure to copy over the EFIVARS.fd file that was modified by grub during the installation, so Qemu can find the new system. Then reboot into the new system, not using the mini.iso image any more:
Finally, you can play around with your Netplan enabled Debian system! As you will find, /etc/network/interfaces exists but is empty, it could still be used (optionally/additionally). Netplan was configured in /etc/netplan/ according to the settings given during the d-i installation process.
In our case, we also installed the Netplan CLI, so we can play around with some of its features, like netplan status:
Thank you for following along the Netplan enabled Debian installation process and happy hacking! If you want to learn more, find us at GitHub:netplan.
Leonardo and I are happy to
announce the release of another maintenance release 0.1.3 of our dtts package
which has been on CRAN for a
good two years now.
dtts
builds upon our nanotime
package as well as the beloved data.table to bring
high-performance and high-resolution indexing at the
nanosecond level to data frames. dtts aims to
offers the time-series indexing versatility of xts (and zoo) to the immense
power of data.table while
supporting highest nanosecond resolution.
This release contains two nice and focussed contributed pull
requests. Tomas Kalibera, who
as part of R Core looks after everything concerning R on Windows, and
then some, needed an adjustment for pending / upcoming R on Windows
changes for builds with LLVM which is what Arm-on-Windows uses. We
happily obliged: neither Leonardo nor I see much of
Windows these decades. (Easy thing to say on a day like today with its
crowdstrike
hammer falling!) Similarly, Michael Chirico supplied a
PR updating one of our tests to an upcoming change at data.table which we are of
course happy to support.
The short list of changes follows.
Changes in version 0.1.3
(2024-07-18)
Windows builds use localtime_s with LLVM (Tomas
Kalibera in #16)
Tests code has been adjusted for an upstream change in data.table tests for all.equal (Michael
Chirico in #18 addressing
#17)