Search Results: "enrico"

13 March 2024

Freexian Collaborators: Debian Contributions: Upcoming Improvements to Salsa CI, /usr-move, packaging simplemonitor, and more! (by Utkarsh Gupta)

Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

/usr-move, by Helmut Grohne Much of the work was spent on handling interaction with time time64 transition and sending patches for mitigating fallout. The set of packages relevant to debootstrap is mostly converted and the patches for glibc and base-files have been refined due to feedback from the upload to Ubuntu noble. Beyond this, he sent patches for all remaining packages that cannot move their files with dh-sequence-movetousr and packages using dpkg-divert in ways that dumat would not recognize.

Upcoming improvements to Salsa CI, by Santiago Ruano Rinc n Last month, Santiago Ruano Rinc n started the work on integrating sbuild into the Salsa CI pipeline. Initially, Santiago used sbuild with the unshare chroot mode. However, after discussion with josch, jochensp and helmut (thanks to them!), it turns out that the unshare mode is not the most suitable for the pipeline, since the level of isolation it provides is not needed, and some test suites would fail (eg: krb5). Additionally, one of the requirements of the build job is the use of ccache, since it is needed by some C/C++ large projects to reduce the compilation time. In the preliminary work with unshare last month, it was not possible to make ccache to work. Finally, Santiago changed the chroot mode, and now has a couple of POC (cf: 1 and 2) that rely on the schroot and sudo, respectively. And the good news is that ccache is successfully used by sbuild with schroot! The image here comes from an example of building grep. At the end of the build, ccache -s shows the statistics of the cache that it used, and so a little more than half of the calls of that job were cacheable. The most important pieces are in place to finish the integration of sbuild into the pipeline. Other than that, Santiago also reviewed the very useful merge request !346, made by IOhannes zm lnig to autodetect the release from debian/changelog. As agreed with IOhannes, Santiago is preparing a merge request to include the release autodetection use case in the very own Salsa CI s CI.

Packaging simplemonitor, by Carles Pina i Estany Carles started using simplemonitor in 2017, opened a WNPP bug in 2022 and started packaging simplemonitor dependencies in October 2023. After packaging five direct and indirect dependencies, Carles finally uploaded simplemonitor to unstable in February. During the packaging of simplemonitor, Carles reported a few issues to upstream. Some of these were to make the simplemonitor package build and run tests reproducibly. A reproducibility issue was reprotest overriding the timezone, which broke simplemonitor s tests. There have been discussions on resolving this upstream in simplemonitor and in reprotest, too. Carles also started upgrading or improving some of simplemonitor s dependencies.

Miscellaneous contributions
  • Stefano Rivera spent some time doing admin on debian.social infrastructure. Including dealing with a spike of abuse on the Jitsi server.
  • Stefano started to prepare a new release of dh-python, including cleaning out a lot of old Python 2.x related code. Thanks to Niels Thykier (outside Freexian) for spear-heading this work.
  • DebConf 24 planning is beginning. Stefano discussed venues and finances with the local team and remotely supported a site-visit by Nattie (outside Freexian).
  • Also in the DebConf 24 context, Santiago took part in discussions and preparations related to the Content Team.
  • A JIT bug was reported against pypy3 in Debian Bookworm. Stefano bisected the upstream history to find the patch (it was already resolved upstream) and released an update to pypy3 in bookworm.
  • Enrico participated in /usr-merge discussions with Helmut.
  • Colin Watson backported a python-channels-redis fix to bookworm, rediscovered while working on debusine.
  • Colin dug into a cluster of celery build failures and tracked the hardest bit down to a Python 3.12 regression, now fixed in unstable. celery should be back in testing once the 64-bit time_t migration is out of the way.
  • Thorsten Alteholz uploaded a new upstream version of cpdb-libs. Unfortunately upstream changed the naming of their release tags, so updating the watch file was a bit demanding. Anyway this version 2.0 is a huge step towards introduction of the new Common Print Dialog Backends.
  • Helmut send patches for 48 cross build failures.
  • Helmut changed debvm to use mkfs.ext4 instead of genext2fs.
  • Helmut sent a debci MR for improving collector robustness.
  • In preparation for DebConf 25, Santiago worked on the Brest Bid.

11 February 2024

Freexian Collaborators: Debian Contributions: Upcoming Improvements to Salsa CI, /usr-move, and more! (by Utkarsh Gupta)

Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

Upcoming Improvements to Salsa CI, by Santiago Ruano Rinc n Santiago started picking up the work made by Outreachy Intern, Enock Kashada (a big thanks to him!), to solve some long-standing issues in Salsa CI. Currently, the first job in a Salsa CI pipeline is the extract-source job, used to produce a debianize source tree of the project. This job was introduced to make it possible to build the projects on different architectures, on the subsequent build jobs. However, that extract-source approach is sub-optimal: not only it increases the execution time of the pipeline by some minutes, but also projects whose source tree is too large are not able to use the pipeline. The debianize source tree is passed as an artifact to the build jobs, and for those large projects, the size of their source tree exceeds the Salsa s limits. This is specific issue is documented as issue #195, and the proposed solution is to get rid of the extract-source job, relying on sbuild in the very build job (see issue #296). Switching to sbuild would also help to improve the build source job, solving issues such as #187 and #298. The current work-in-progress is very preliminary, but it has already been possible to run the build (amd64), build-i386 and build-source job using sbuild with the unshare mode. The image on the right shows a pipeline that builds grep. All the test jobs use the artifacts of the new build job. There is a lot of remaining work, mainly making the integration with ccache work. This change could break some things, it will also be important to test how the new pipeline works with complex projects. Also, thanks to Emmanuel Arias, we are proposing a Google Summer of Code 2024 project to improve Salsa CI. As part of the ongoing work in preparation for the GSoC 2024 project, Santiago has proposed a merge request to make more efficient how contributors can test their changes on the Salsa CI pipeline.

/usr-move, by Helmut Grohne In January, we sent most of the moving patches for the set of packages involved with debootstrap. Notably missing is glibc, which turns out harder than anticipated via dumat, because it has Conflicts between different architectures, which dumat does not analyze. Patches for diversion mitigations have been updated in a way to not exhibit any loss anymore. The main change here is that packages which are being diverted now support the diverting packages in transitioning their diversions. We also supported a few packages with non-trivial changes such as netplan.io. dumat has been enhanced to better support derivatives such as Ubuntu.

Miscellaneous contributions
  • Python 3.12 migration trundles on. Stefano Rivera helped port several new packages to support 3.12.
  • Stefano updated the Sphinx configuration of DebConf Video Team s documentation, which was broken by Sphinx 7.
  • Stefano published the videos from the Cambridge MiniDebConf to YouTube and PeerTube.
  • DebConf 24 planning has begun, and Stefano & Utkarsh have started work on this.
  • Utkarsh re-sponsored the upload of golang-github-prometheus-community-pgbouncer-exporter for Lena.
  • Colin Watson added Incus support to autopkgtest.
  • Colin discovered Perl::Critic and used it to tidy up some poor practices in several of his packages, including debconf.
  • Colin did some overdue debconf maintenance, mainly around tidying up error message handling in several places (1, 2, 3).
  • Colin figured out how to update the mirror size documentation in debmirror, last updated in 2010. It should now be much easier to keep it up to date regularly.
  • Colin issued a man-db buster update to clean up some irritations due to strict sandboxing.
  • Thorsten Alteholz adopted two more packages, magicfilter and ifhp, for the debian-printing team. Those packages are the last ones of the latest round of adoptions to preserve the old printing protocol within Debian. If you know of other packages that should be retained, please don t hesitate to contact Thorsten.
  • Enrico participated in /usr-merge discussions with Helmut.
  • Helmut sent patches for 16 cross build failures.
  • Helmut supported Matthias Klose (not affiliated with Freexian) with adding -for-host support to gcc-defaults.
  • Helmut uploaded dput-ng enabling dcut migrate and merging two MRs of Ben Hutchings.
  • Santiago took part in the discussions relating to the EU Cyber Resilience Act (CRA) and the Debian public statement that was published last year. He participated in a meeting with Members of the European Parliament (MEPs), Marcel Kolaja and Karen Melchior, and their teams to clarify some points about the impact of the CRA and Debian and downstream projects, and the improvements in the last version of the proposed regulation.

13 January 2024

Freexian Collaborators: Debian Contributions: LXD/Incus backend bug, /usr-merge updates, gcc-for-host, and more! (by Utkarsh Gupta)

Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

LXD/Incus backend bug in autopkgtest by Stefano Rivera While working on the Python 3.12 transition, Stefano repeatedly ran into a bug in autopkgtest when using LXD (or in the future Incus), that caused it to hang when running cython s multi-hour autopkgtests. After some head-banging, the bug turned out to be fairly straightforward: LXD didn t shut down on receiving a SIGTERM, so when a testsuite timed out, it would hang forever. A simple fix has been applied.

/usr-merge, by Helmut Grohne Thanks to Christian Hofstaedtler and others, the effort is moving into a community effort and the work funded by Freexian becomes more difficult to separate from non-funded work. In particular, since the community fully handled all issues around lost udev rules, dh_installudev now installs rules to /usr. The story around diversions took another detour. We learned that conflicts do not reliably prevent concurrent unpack and the reiterated mitigation for molly-guard triggered this. After a bit of back and forth and consultation with the developer mailing list, we concluded that avoiding the problematic behavior when using apt or an apt-based upgrader combined with a loss mitigation would be good enough. The involved packages bfh-container, molly-guard, progress-linux-container and systemd have since been uploaded to unstable and the matter seems finally solved except that it doesn t quite work with sysvinit yet. The same approach is now being proposed for the diversions of zutils for gzip. We thank involved maintainers for their timely cooperation.

gcc-for-host, by Helmut Grohne Since forever, it has been difficult to correctly express a toolchain build dependency. This can be seen in the Build-Depends of the linux source package for instance. While this has been solved for binutils a while back, the patches for gcc have been unfinished. With lots of constructive feedback from gcc package maintainer Matthias Klose, Helmut worked on finalizing and testing these patches. Patch stacks are now available for gcc-13 and gcc-14 and Matthias already included parts of them in test builds for Ubuntu noble. Finishing this work would enable us to resolve around 1000 cross build dependency satisfiability issues in unstable.

Miscellaneous contributions
  • Stefano continued work on the Python 3.12 transition, including uploads of cython, pycxx, numpy, python-greenlet, twisted, foolscap and dh-python.
  • Stefano reviewed and selected from a new round of DebConf 24 bids, as part of the DebConf Committee. Busan, South Korea was selected.
  • For debian-printing Thorsten uploaded hplip to unstable to fix a /usr-merge bug and cups to Bookworm to fix bugs related to printing in color.
  • Utkarsh helped newcomers in mentoring and reviewing their packaging; eg: golang-github-prometheus-community-pgbouncer-exporter.
  • Helmut sent patches for 42 cross build failures unrelated to the gcc-for-host work.
  • Helmut continues to maintain rebootstrap. In December, blt started depending on libjpeg and this poses a dependency loop. Ideally, Python would stop depending on blt. Also linux-libc-dev having become Multi-Arch: foreign poses non-trivial issues that are not fully resolved yet.
  • Enrico participated in /usr-merge discussions with Helmut.

10 December 2023

Freexian Collaborators: Debian Contributions: Python 3.12 preparations, debian-printing, merged-/usr tranisition updates, and more! (by Utkarsh Gupta)

Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

Preparing for Python 3.12 by Stefano Rivera Stefano uploaded a few packages in preparation for Python 3.12, including pycxx and cython. Cython has a major new version (Cython 3), adding support for 3.12, but also bringing changes that many packages in Debian aren t ready to build with, yet. Stefano uploaded it to Debian experimental and did an archive rebuild of affected packages, and some analysis of the result. Matthias Klose has since filed bugs for all of these issues.

debian-printing, by Thorsten Alteholz This month Thorsten invested some of the previously obtained money to build his own printlab. At the moment it only consists of a dedicated computer with an USB printer attached. Due to its 64GB RAM and an SSD, building of debian-printing packages is much faster now. Over time other printers will be added and understanding bugs should be a lot easier now. Also Thorsten again adopted two packages, namely mink and ink, and moved them to the debian-printing team.

Merged-/usr transition by Helmut Grohne, et al The dumat analysis tool has been improved in quite some aspects. Beyond fixing false negative diagnostics, it now recognizes protective diversions used for mitigating Multi-Arch: same file loss. It was found that the proposed mitigation for ineffective diversions does not work as expected. Trying to fix it up resulted in more problems, some of which remain unsolved as of this writing. Initial work on moving shared libraries in the essential set has been done. Meanwhile, the wider Debian community worked on fixing all known Multi-Arch: same file loss scenarios. This work is now being driven by Christian Hofstaedler and during the Mini DebConf in Cambridge, Chris Boot, tienne Mollier, Miguel Landaeta, Samuel Henrique, and Utkarsh Gupta sent the other half of the necessary patches.

Miscellaneous contributions
  • Stefano merged patches to support loong64 and hurd-amd64 in re2.
  • For the Cambridge mini-conf, Stefano added a web player to the DebConf video streaming frontend, as the Cambridge miniconf didn t have its own website to host the player.
  • Rapha l helped the upstream developers of hamster-time-tracker to prepare a new upstream release (the first in multiple years) and packaged that new release in Debian unstable.
  • Enrico joined Hemut in brainstorming some /usr-merge solutions.
  • Thorsten took care of RM-bugs to remove no longer needed packages from the Debian archive and closed about 50 of them.
  • Helmut ported the feature of mounting a fuse connection via /dev/fd/N from fuse3 to fuse2.
  • Helmut sent a number of patches simplifying unprivileged use of piuparts.
  • Roberto worked with Helmut to prepare the Shorewall package for the ongoing /usr-move transition.
  • Utkarsh also helped with the ongoing /usr-merge work by preparing patches for gitlab, libnfc, and net-tools.
  • Utkarsh, along with Helmut, brainstormed on fixing #961138, as this affects the whole archive and all the suites and not just R packages. Utkarsh intends to follow up on the bug in December.
  • Santiago organized a MiniDebConf in Uruguay. In total, nine people attended, including most of DDs in the surrounding area. Here s a nicely written blog by Gunnar Wolf.
  • Santiago also worked on some issues on Salsa CI, fixed with some merge requests: #462, #463, and #466.

28 November 2023

Enrico Zini: Introducing Debusine

Abstract Debusine manages scheduling and distribution of Debian-related tasks (package build, lintian analysis, autopkgtest runs, etc.) to distributed worker machines. It is being developed by Freexian with the intention of giving people access to a range of pre-configured tools and workflows running on remote hardware. Freexian obtained STF funding for a substantial set of Debusine milestones, so development is happening on a clear schedule. We can present where we are and, we're going to be, and what we hope to bring to Debian with this work.

20 October 2023

Freexian Collaborators: Debian Contributions: Freexian meetup, debusine updates, lpr/lpd in Debian, and more! (by Utkarsh Gupta, Stefano Rivera)

Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

Freexian Meetup, by Stefano Rivera, Utkarsh Gupta, et al. During DebConf, Freexian organized a meetup for its collaborators and those interested in learning more about Freexian and its services. It was well received and many people interested in Freexian showed up. Some developers who were interested in contributing to LTS came to get more details about joining the project. And some prospective customers came to get to know us and ask questions. Sadly, the tragic loss of Abraham shook DebConf, both individually and structurally. The meetup got rescheduled to a small room without video coverage. With that, we still had a wholesome interaction and here s a quick picture from the meetup taken by Utkarsh (which is also why he s missing!).

Debusine, by Rapha l Hertzog, et al. Freexian has been investing into debusine for a while, but development speed is about to increase dramatically thanks to funding from SovereignTechFund.de. Rapha l laid out the 5 milestones of the funding contract, and filed the issues for the first milestone. Together with Enrico and Stefano, they established a workflow for the expanded team. Among the first steps of this milestone, Enrico started to work on a developer-friendly description of debusine that we can use when we reach out to the many Debian contributors that we will have to interact with. And Rapha l started the design work of the autopkgtest and lintian tasks, i.e. what s the interface to schedule such tasks, what behavior and what associated options do we support? At this point you might wonder what debusine is supposed to be let us try to answer this: Debusine manages scheduling and distribution of Debian-related build and QA tasks to a network of worker machines. It also manages the resulting artifacts and provides the results in an easy to consume way. We want to make it easy for Debian contributors to leverage all the great QA tools that Debian provides. We want to build the next generation of Debian s build infrastructure, one that will continue to reliably do what it already does, but that will also enable distribution-wide experiments, custom package repositories and custom workflows with advanced package reviews. If this all sounds interesting to you, don t hesitate to watch the project on salsa.debian.org and to contribute.

lpr/lpd in Debian, by Thorsten Alteholz During Debconf23, Till Kamppeter presented CPDB (Common Print Dialog Backend), a new way to handle print queues. After this talk it was discussed whether the old lpr/lpd based printing system could be abandoned in Debian or whether there is still demand for it. So Thorsten asked on the debian-devel email list whether anybody uses it. Oddly enough, these old packages are still useful:
  • Within a small network it is easier to distribute a printcap file, than to properly configure cups clients.
  • One of the biggest manufacturers of WLAN router and DSL boxes only supports raw queues when attaching an USB printer to their hardware. Admittedly the CPDB still has problems with such raw queues.
  • The Pharos printing system at MIT is still lpd-based.
As a result, the lpr/lpd stuff is not yet ready to be abandoned and Thorsten will adopt the relevant packages (or rather move them under the umbrella of the debian-printing team). Though it is not planned to develop new features, those packages should at least have a maintainer. This month Thorsten adopted rlpr, an utility for lpd printing without using /etc/printcap. The next one he is working on is lprng, a lpr/lpd printer spooling system. If you know of any other package that is also needed and still maintained by the QA team, please tell Thorsten.

/usr-merge, by Helmut Grohne Discussion about lifting the file move moratorium has been initiated with the CTTE and the release team. A formal lift is dependent on updating debootstrap in older suites though. A significant number of packages can automatically move their systemd unit files if dh_installsystemd and systemd.pc change their installation targets. Unfortunately, doing so makes some packages FTBFS and therefore patches have been filed. The analysis tool, dumat, has been enhanced to better understand which upgrade scenarios are considered supported to reduce false positive bug filings and gained a mode for local operation on a .changes file meant for inclusion in salsa-ci. The filing of bugs from dumat is still manual to improve the quality of reports. Since September, the moratorium has been lifted.

Miscellaneous contributions
  • Rapha l updated Django s backport in bullseye-backports to match the latest security release that was published in bookworm. Tracker.debian.org is still using that backport.
  • Helmut Grohne sent 13 patches for cross build failures.
  • Helmut Grohne performed a maintenance upload of debvm enabling its use in autopkgtests.
  • Helmut Grohne wrote an API-compatible reimplementation of autopkgtest-build-qemu. It is powered by mmdebstrap, therefore unprivileged, EFI-only and will soon be included in mmdebstrap.
  • Santiago continued the work regarding how to make it easier to (automatically) test reverse dependencies. An example of the ongoing work was presented during the Salsa CI BoF at DebConf 23.
    In fact, omniorb-dfsg test pipelines as the above were used for the omniorb-dfsg 4.3.0 transition, verifying how the reverse dependencies (tango, pytango and omnievents) were built and how their autopkgtest jobs run with the to-be-uploaded omniorb-dfsg new release.
  • Utkarsh and Stefano attended and helped run DebConf 23. Also continued winding up DebConf 22 accounting.
  • Anton Gladky did some science team uploads to fix RC bugs.

21 September 2023

Jonathan McDowell: DebConf23 Writeup

DebConf2023 Logo (I wrote this up for an internal work post, but I figure it s worth sharing more publicly too.) I spent last week at DebConf23, this years instance of the annual Debian conference, which was held in Kochi, India. As usual, DebConf provides a good reason to see a new part of the world; I ve been going since 2004 (Porto Alegre, Brazil), and while I ve missed a few (Mexico, Bosnia, and Switzerland) I ve still managed to make it to instances on 5 continents. This has absolutely nothing to do with work, so I went on my own time + dime, but I figured a brief write-up might prove of interest. I first installed Debian back in 1999 as a machine that was being co-located to operate as a web server / email host. I was attracted by the promise of easy online upgrades (or, at least, upgrades that could be performed without the need to be physically present at the machine, even if they naturally required a reboot at some point). It has mostly delivered on this over the years, and I ve never found a compelling reason to move away. I became a Debian Developer in 2000. As a massively distributed volunteer project DebConf provides an opportunity to find out what s happening in other areas of the project, catch up with team mates, and generally feel more involved and energised to work on Debian stuff. Also, by this point in time, a lot of Debian folk are good friends and it s always nice to catch up with them. On that point, I felt that this year the hallway track was not quite the same as usual. For a number of reasons (COVID, climate change, travel time, we re all getting older) I think fewer core teams are achieving critical mass at DebConf - I was the only member physically present from 2 teams I m involved in, and I d have appreciated the opportunity to sit down with both of them for some in-person discussions. It also means it s harder to use DebConf as a venue for advancing major changes; previously having all the decision makers in the same space for a week has meant it s possible to iron out the major discussion points, smoothing remote implementation after the conference. I m told the mini DebConfs are where it s at for these sorts of meetings now, so perhaps I ll try to attend at least one of those next year. Of course, I also went to a bunch of talks. I have differing levels of comment about each of them, but I ve written up some brief notes below about the ones I remember something about. The comment was made that we perhaps had a lower level of deep technical talks, which is perhaps true but I still think there were a number of high level technical talks that served to pique ones interest about the topic. Finally, this DebConf was the first I m aware of that was accompanied by tragedy; as part of the day trip Abraham Raji, a project member and member of the local team, was involved in a fatal accident.

Talks (videos not yet up for all, but should appear for most)
  • Opening Ceremony
    Not much to say here; welcome to DebConf!
  • Continuous Key-Signing Party introduction
    I ended up running this, as Gunnar couldn t make it. Debian makes heavy use of the OpenPGP web of trust (no mass ability to send out Yubikeys + perform appropriate levels of identity verification), so making sure we re appropriately cross-signed, and linked to local conference organisers, is a dull but important part of the conference. We use a modified keysigning approach where identity verification + fingerprint confirmation happens over the course of the conference, so this session was just to explain how that works and confirm we were all working from the same fingerprint list.
  • State of Stateless - A Talk about Immutability and Reproducibility in Debian
    Stateless OSes seem to be gaining popularity, so I went along to this to see if there was anything of note. It was interesting, but nothing earth shattering - very high level.
  • What s missing so that Debian is finally reproducible?
    Reproducible builds are something I ve been keeping an eye on for a long time, and I continue to be impressed by the work folks are putting into this - both for Debian, and other projects. From a security standpoint reproducible builds provide confidence against trojaned builds, and from a developer standpoint knowing you can build reproducibly helps with not having to keep a whole bunch of binary artefacts around.
  • Hello from keyring-maint
    In the distant past the process of getting your OpenPGP key into the Debian keyring (which is used to authenticate uploads + votes, amongst other things) was a clunky process that was often stalled. This hasn t been the case for at least the past 10 years, but there s still a residual piece of project memory that thinks keyring is a blocker. So as a team we say hi and talk about the fact we do monthly updates and generally are fairly responsive these days.
  • A declarative approach to Linux networking with Netplan
    Debian s /etc/network/interfaces is a fairly basic (if powerful) mechanism for configuring network interfaces. NetworkManager is a better bet for dynamic hosts (i.e. clients), and systemd-network seems to be a good choice for servers (I m gradually moving machines over to it). Netplan tries to provide a unified mechanism for configuring both with a single configuration language. A noble aim, but I don t see a lot of benefit for anything I use - my NetworkManager hosts are highly dynamic (so no need to push shared config) and systemd-network (or /etc/network/interfaces) works just fine on the other hosts. I m told Netplan has more use with more complicated setups, e.g. when OpenVSwitch is involved.
  • Quick peek at ZFS, A too good to be true file system and volume manager.
    People who use ZFS rave about it. I m naturally suspicious of any file system that doesn t come as part of my mainline kernel. But, as a longtime cautious mdraid+lvm+ext4 user I appreciate that there have been advances in the file system space that maybe I should look at, and I ve been trying out btrfs on more machines over the past couple of years. I can t deny ZFS has a bunch of interesting features, but nothing I need/want that I can t get from an mdraid+lvm+btrfs stack (in particular data checksumming + reflinks for dedupe were strong reasons to move to btrfs over ext4).
  • Bits from the DPL
    Exactly what it says on the tin; some bits from the DPL.
  • Adulting
    Enrico is always worth hearing talk; Adulting was no exception. Main takeaway is that we need to avoid trying to run the project on martyrs and instead make sure we build a sustainable project. I ve been trying really hard to accept I just don t have time to take on additional responsibilities, no matter how interesting or relevant they might seem, so this resonated.
  • My life in git, after subversion, after CVS.
    Putting all of your home directory in revision control. I ve never made this leap; I ve got some Ansible playbooks that push out my core pieces of configuration, which is held in git, but I don t actually check this out directly on hosts I have accounts on. Interesting, but not for me.
  • EU Legislation BoF - Cyber Resilience Act, Product Liability Directive and CSAM Regulation
    The CRA seems to be a piece of ill informed legislation that I m going to have to find time to read properly. Discussion was a bit more alarmist than I personally feel is warranted, but it was a short session, had a bunch of folk in it, and even when I removed my mask it was hard to make myself understood.
  • What s new in the Linux kernel (and what s missing in Debian)
    An update from Ben about new kernel features. I m paying less attention to such things these days, so nice to get a quick overview of it all.
  • Intro to SecureDrop, a sort-of Linux distro
    Actually based on Ubuntu, but lots of overlap with Debian as a result, and highly customised anyway. Notable, to me, for using OpenPGP as some of the backend crypto support. I managed to talk to Kunal separately about some of the pain points around that, which was an interesting discussion - they re trying to move from GnuPG to Sequoia, primarily because of the much easier integration and lack of requirement for the more complicated GnuPG features that sometimes get in the way.
  • The Docker(.io) ecosystem in Debian
    I hate Docker. I m sure it s fine if you accept it wants to take over the host machine entirely, but when I ve played around with it that s not been the case. This talk was more about the difficulty of trying to keep a fast moving upstream with lots of external dependencies properly up to date in a stable release. Vendoring the deps and trying to get a stable release exception seems like the least bad solution, but it s a problem that affects a growing number of projects.
  • Chiselled containers
    This was kinda of interesting, but I think I missed the piece about why more granular packaging wasn t an option. The premise is you can take an existing .deb and chisel it into smaller components, which then helps separate out dependencies rather than pulling in as much as the original .deb would. This was touted as being useful, in particular, for building targeted containers. Definitely appealing over custom built userspaces for containers, but in an ideal world I think we d want the information in the main packaging and it becomes a lot of work.
  • Debian Contributors shake-up
    Debian Contributors is a great site for massaging your ego around contributions to Debian; it s also a useful point of reference from a data protection viewpoint in terms of information the project holds about contributors - everything is already public, but the Contributors website provides folk with an easy way to find their own information (with various configurable options about whether that s made public or not). T ssia is working on improving the various data feeds into the site, but realistically this is the responsibility of every Debian service owner.
  • New Member BOF
    I m part of the teams that help get new folk into Debian - primarily as a member of the New Member Front Desk, but also as a mostly inactive Application Manager. It s been a while since we did one of these sessions so the Front Desk/Debian Account Managers that were present did a panel session. Nothing earth shattering came out of it; like keyring-maint this is a team that has historically had problems, but is currently running smoothly.

8 September 2023

Enrico Zini: Regular virus scan

Image of clamav running and a COVID lateral flow test Debian: when you're more likely to get a virus than your laptop

26 July 2023

Enrico Zini: Mysterious DNS issues

Uhm, salsa is not resolving:
$ git fetch
ssh: Could not resolve hostname salsa.debian.org: Name or service not known
fatal: Could not read from remote repository.
$ ping salsa.debian.org
ping: salsa.debian.org: Name or service not known
But... it is?
$ host salsa.debian.org
salsa.debian.org has address 209.87.16.44
salsa.debian.org has IPv6 address 2607:f8f0:614:1::1274:44
salsa.debian.org mail is handled by 10 mailly.debian.org.
salsa.debian.org mail is handled by 10 mitropoulos.debian.org.
salsa.debian.org mail is handled by 10 muffat.debian.org.
It really is resolving correctly at each step:
$ cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
# [...]
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
# [...]
nameserver 127.0.0.53
options edns0 trust-ad
search fritz.box
$ host salsa.debian.org 127.0.0.53
Using domain server:
Name: 127.0.0.53
Address: 127.0.0.53#53
Aliases:
salsa.debian.org has address 209.87.16.44
salsa.debian.org has IPv6 address 2607:f8f0:614:1::1274:44
salsa.debian.org mail is handled by 10 mailly.debian.org.
salsa.debian.org mail is handled by 10 muffat.debian.org.
salsa.debian.org mail is handled by 10 mitropoulos.debian.org.
# resolvectl status
Global
       Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
Link 3 (wlp108s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.178.1
       DNS Servers: 192.168.178.1 fd00::3e37:12ff:fe99:2301 2a01:b600:6fed:1:3e37:12ff:fe99:2301
        DNS Domain: fritz.box
Link 4 (virbr0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 9 (enxace2d39ce693)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.178.1
       DNS Servers: 192.168.178.1 fd00::3e37:12ff:fe99:2301 2a01:b600:6fed:1:3e37:12ff:fe99:2301
        DNS Domain: fritz.box
$ host salsa.debian.org 192.168.178.1
Using domain server:
Name: 192.168.178.1
Address: 192.168.178.1#53
Aliases:
salsa.debian.org has address 209.87.16.44
salsa.debian.org has IPv6 address 2607:f8f0:614:1::1274:44
salsa.debian.org mail is handled by 10 muffat.debian.org.
salsa.debian.org mail is handled by 10 mitropoulos.debian.org.
salsa.debian.org mail is handled by 10 mailly.debian.org.
$ host salsa.debian.org fd00::3e37:12ff:fe99:2301 2a01:b600:6fed:1:3e37:12ff:fe99:2301
Using domain server:
Name: fd00::3e37:12ff:fe99:2301
Address: fd00::3e37:12ff:fe99:2301#53
Aliases:
salsa.debian.org has address 209.87.16.44
salsa.debian.org has IPv6 address 2607:f8f0:614:1::1274:44
salsa.debian.org mail is handled by 10 muffat.debian.org.
salsa.debian.org mail is handled by 10 mitropoulos.debian.org.
salsa.debian.org mail is handled by 10 mailly.debian.org.
Could it be caching?
# systemctl restart systemd-resolved
$ dpkg -s nscd
dpkg-query: package 'nscd' is not installed and no information is available
$ git fetch
ssh: Could not resolve hostname salsa.debian.org: Name or service not known
fatal: Could not read from remote repository.
Could it be something in ssh's config?
$ grep salsa ~/.ssh/config
$ ssh git@salsa.debian.org
ssh: Could not resolve hostname salsa.debian.org: Name or service not known
Something weird with ssh's control sockets?
$ strace -fo /tmp/zz ssh git@salsa.debian.org
ssh: Could not resolve hostname salsa.debian.org: Name or service not known
enrico@ploma:~/lavori/legal/legal$ grep salsa /tmp/zz
393990 execve("/usr/bin/ssh", ["ssh", "git@salsa.debian.org"], 0x7ffffcfe42d8 /* 54 vars */) = 0
393990 connect(3,  sa_family=AF_UNIX, sun_path="/home/enrico/.ssh/sock/git@salsa.debian.org:22" , 110) = -1 ENOENT (No such file or directory)
$ strace -fo /tmp/zz1 ssh -S none git@salsa.debian.org
ssh: Could not resolve hostname salsa.debian.org: Name or service not known
$ grep salsa /tmp/zz1
394069 execve("/usr/bin/ssh", ["ssh", "-S", "none", "git@salsa.debian.org"], 0x7ffd36cbfde8 /* 54 vars */) = 0
How is ssh trying to resolve salsa.debian.org?
393990 socket(AF_UNIX, SOCK_STREAM SOCK_CLOEXEC SOCK_NONBLOCK, 0) = 3
393990 connect(3,  sa_family=AF_UNIX, sun_path="/run/systemd/resolve/io.systemd.Resolve" , 42) = 0
393990 sendto(3, " \"method\":\"io.systemd.Resolve.Re"..., 99, MSG_DONTWAIT MSG_NOSIGNAL, NULL, 0) = 99
393990 mmap(NULL, 135168, PROT_READ PROT_WRITE, MAP_PRIVATE MAP_ANONYMOUS, -1, 0) = 0x7f4fc71ca000
393990 recvfrom(3, 0x7f4fc71ca010, 135152, MSG_DONTWAIT, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
393990 ppoll([ fd=3, events=POLLIN ], 1,  tv_sec=119, tv_nsec=999917000 , NULL, 8) = 1 ([ fd=3, revents=POLLIN ], left  tv_sec=119, tv_nsec=998915689 )
393990 recvfrom(3, " \"error\":\"io.systemd.System\",\"pa"..., 135152, MSG_DONTWAIT, NULL, NULL) = 56
393990 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
393990 close(3)                         = 0
393990 munmap(0x7f4fc71ca000, 135168)   = 0
393990 getpid()                         = 393990
393990 write(2, "ssh: Could not resolve hostname "..., 77) = 77
Something weird with resolved?
$ resolvectl query salsa.debian.org
salsa.debian.org: resolve call failed: Lookup failed due to system error: Invalid argument
Let's try disrupting what ssh is trying and failing:
# mv /run/systemd/resolve/io.systemd.Resolve /run/systemd/resolve/io.systemd.Resolve.backup
$ strace -o /tmp/zz2 ssh -S none -vv git@salsa.debian.org
OpenSSH_9.2p1 Debian-2, OpenSSL 3.0.9 30 May 2023
debug1: Reading configuration data /home/enrico/.ssh/config
debug1: /home/enrico/.ssh/config line 1: Applying options for *
debug1: /home/enrico/.ssh/config line 228: Applying options for *.debian.org
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "salsa.debian.org" port 22
ssh: Could not resolve hostname salsa.debian.org: Name or service not known
$ tail /tmp/zz2
394748 prctl(PR_CAPBSET_READ, 0x29 /* CAP_??? */) = -1 EINVAL (Invalid argument)
394748 munmap(0x7f27af5ef000, 164622)   = 0
394748 rt_sigprocmask(SIG_BLOCK, [HUP USR1 USR2 PIPE ALRM CHLD TSTP URG VTALRM PROF WINCH IO], [], 8) = 0
394748 futex(0x7f27ae5feaec, FUTEX_WAKE_PRIVATE, 2147483647) = 0
394748 openat(AT_FDCWD, "/run/systemd/machines/salsa.debian.org", O_RDONLY O_CLOEXEC) = -1 ENOENT (No such file or directory)
394748 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
394748 getpid()                         = 394748
394748 write(2, "ssh: Could not resolve hostname "..., 77) = 77
394748 exit_group(255)                  = ?
394748 +++ exited with 255 +++
$ machinectl list
No machines.
# resolvectl flush-caches
$ resolvectl query salsa.debian.org
salsa.debian.org: resolve call failed: Lookup failed due to system error: Invalid argument
# resolvectl reset-statistics
$ resolvectl query salsa.debian.org
salsa.debian.org: resolve call failed: Lookup failed due to system error: Invalid argument
# resolvectl reset-server-features
$ resolvectl query salsa.debian.org
salsa.debian.org: resolve call failed: Lookup failed due to system error: Invalid argument
# resolvectl monitor
  Q: salsa.debian.org IN A
  Q: salsa.debian.org IN AAAA
  S: EINVAL
  A: debian.org IN NS sec2.rcode0.net
  A: debian.org IN NS sec1.rcode0.net
  A: debian.org IN NS nsp.dnsnode.net
  A: salsa.debian.org IN A 209.87.16.44
  A: debian.org IN NS dns4.easydns.info
I guess I won't be using salsa today, and I wish I understood why. Update: as soon as I pushed this post to my blog (via ssh) salsa started resolving again.

10 June 2023

Freexian Collaborators: Debian Contributions: /usr-merge updates, tox 4 transition, and more! (by Utkarsh Gupta, Stefano Rivera)

Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

/usr-merge, by Helmut Grohne, et al Towards the end of April, the discussion on DEP 17 on debian-devel@l.d.o initiated by Helmut Grohne took off, trying to deal with the fact that while Debian bookworm has a merged /usr, files are still being distributed to / and /usr in Debian binary packages, and moving them currently has some risk of breakage. Most participants of the discussion agreed that files should be moved, and there are several competing design ideas for doing it safely. Most of the time was spent understanding the practical implications of lifting the moratorium and moving all the files from / to /usr in a coordinated effort. With help from Emilio Pozuelo Monfort, Enrico Zini, and Raphael Hertzog, Helmut Grohne performed extensive analysis of the various aspects, including quantitative analysis of the original file move problem, analysis of effects on dpkg-divert, dpkg-statoverride, and update-alternatives, analysis of effects on filesystem bootstrapping tools. Most of the problematic cases spawned plausible workarounds, such as turning Breaks into Conflicts in selected cases or adding protective diversions for the symbolic links that enable aliasing. Towards the end of May, Andreas Beckmann reported a new failure scenario which may cause shared resources to inadvertently disappear, such as directories and even regular files in case of Multi-Arch packages, and our work on analyzing these problems and proposing mitigations is on-going. While the quantitative analysis is funded by Freexian, we wouldn t be here without the extensive feedback and ideas of many voluntary contributors from multiple areas of Debian, which are too many to name here. Thank you.

Preparing for the tox 4 transition, by Stefano Rivera While Debian was in freeze for the bookworm release, tox 4 has landed in Debian experimental, and some packages are starting to require it, upstream. It has some backwards-incompatible behavior that breaks many packages using tox through pybuild. So Stefano had to make some changes to pybuild and to many packages that run build-time tests with tox. The easy bits of this transition are now completed in git / experimental, but a few packages that integrate deeply into tox need upstream work.

Debian Printing, by Thorsten Alteholz Just before the release of Bookworm, lots of QA tools were used to inspect packages. One of these tools found a systemd service file in a wrong directory. So, Thorsten did another upload of package lprint to correct this. Thanks a lot to all the hardworking people who run such tools and file bugs. Thorsten also participated in discussions about the new Common Printing Dialog Backends (CPDB) that will be introduced in Trixie and hopefully can replace the current printing architecture in Forky.

Miscellaneous contributions
  • DebConf 23 preparations by Stefano Rivera. Some work on the website, video team planning, accounting, and team documentation.
  • Utkarsh Gupta started to prep the work on the bursary team s side for DC23.
  • Stefano spun up a website for the Hamburg mini-DebConf so that the video team could have a machine-readable schedule and a place to stream video from the event.
  • Santiago Ruano Rinc n reviewed and sponsored four python packages of a prospective Debian member.
  • Helmut Grohne supported Timo Roehling and Jochen Sprickerhof to improve cross building in 15 ROS packages.
  • Helmut Grohne supported Jochen Sprickerhof with diagnosing an e2fsprogs RC bug.
  • Helmut Grohne continued to maintain rebootstrap and located an issue with lto in gcc-13.
  • Anton Gladky fixed some RC-Bugs and uploaded a new stravalib python library.

29 April 2023

Enrico Zini: Gtk4 model-backed radio button in Python

Gtk4 has interesting ways of splitting models and views. One that I didn't find very well documented, especially for Python bindings, is a set of radio buttons backed by a common model. The idea is to define an action that takes a string as a state. Each radio button is assigned a string matching one of the possible states, and when the state of the backend action is changed, the radio buttons are automatically updated. All the examples below use a string for a value type, but anything can be used that fits into a GLib.Variant. The model This defines the action. Note that enables all the usual declarative ways of a status change:
mode = Gio.SimpleAction.new_stateful(
        name="mode-selection",
        parameter_type=GLib.VariantType("s"),
        state=GLib.Variant.new_string(""))
gtk_app.add_action(self.mode)
The view
def add_radio(model: Gio.SimpleAction, id: str, label: str):
    button = Gtk.CheckButton(label=label)
    # Tell this button to activate when the model has the given value
    button.set_action_target_value(GLib.Variant.new_string(id))
    # Build the name under which the action is registesred, plus the state
    # value controlled by this button: clicking the button will set this state
    detailed_name = Gio.Action.print_detailed_name(
            "app." + model.get_name(),
            GLib.Variant.new_string(id))
    button.set_detailed_action_name(detailed_name)
    # If the model has no current value set, this sets the first radio button
    # as selected
    if not model.get_state().get_string():
        model.set_state(GLib.Variant.new_string(id))
Accessing the model To read the currently selected value:
current = model.get_state().get_string()
To set the currently selected value:
model.set_state(GLib.Variant.new_string(id))

28 April 2023

Enrico Zini: Handling keyboard-like devices

CNC control panel and Bluetooth pedal page turner I acquired some unusual input devices to experiment with, like a CNC control panel and a bluetooth pedal page turner. These identify and behave like a keyboard, sending nice and simple keystrokes, and can be accessed with no drivers or other special software. However, their keystrokes appear together with keystrokes from normal keyboards, which is the expected default when plugging in a keyboard, but not what I want in this case. I'd also like them to be readable via evdev and accessible by my own user. Here's the udev rule I cooked up to handle this use case:
# Handle the CNC control panel
SUBSYSTEM=="input", ENV ID_VENDOR =="04d9", ENV ID_MODEL =="1203", \
   OWNER="enrico", ENV ID_INPUT =""
# Handle the Bluetooth page turner
SUBSYSTEM=="input", ENV ID_BUS =="bluetooth", ENV LIBINPUT_DEVICE_GROUP =="*/ mac ", ENV ID_INPUT_KEYBOARD ="1" \
   OWNER="enrico", ENV ID_INPUT ="", SYMLINK+="input/by-id/bluetooth- mac -kbd"
SUBSYSTEM=="input", ENV ID_BUS =="bluetooth", ENV LIBINPUT_DEVICE_GROUP =="*/ mac ", ENV ID_INPUT_TABLET ="1" \
   OWNER="enrico", ENV ID_INPUT ="", SYMLINK+="input/by-id/bluetooth- mac -tablet"
The bluetooth device didn't have standard rules to create /dev/input/by-id/ symlinks so I added them. In my own code, I watch /dev/input/by-id with inotify to handle when devices appear or disappear. I used udevadm info /dev/input/event to see what I could use to identify the device. The Static device configuration via udev page of libinput's documentation has documentation on the various elements specific to the input subsystem Grepping rule files in /usr/lib/udev/rules.d was useful to see syntax examples. udevadm test /dev/input/event was invaluable for syntax checking and testing my rule file while working on it. Finally, this is an extract of a quick prototype Python code to read keys from the CNC control panel:
import libevdev
KEY_MAP =  
    libevdev.EV_KEY.KEY_GRAVE: "EMERGENCY",
    # InputEvent(EV_KEY, KEY_LEFTALT, 1)
    libevdev.EV_KEY.KEY_R: "CYCLE START",
    libevdev.EV_KEY.KEY_F5: "SPINDLE ON/OFF",
    # InputEvent(EV_KEY, KEY_RIGHTCTRL, 1)
    libevdev.EV_KEY.KEY_W: "REDO",
    # InputEvent(EV_KEY, KEY_LEFTALT, 1)
    libevdev.EV_KEY.KEY_N: "SINGLE STEP",
    # InputEvent(EV_KEY, KEY_LEFTCTRL, 1)
    libevdev.EV_KEY.KEY_O: "ORIGIN POINT",
    libevdev.EV_KEY.KEY_ESC: "STOP",
    libevdev.EV_KEY.KEY_KPPLUS: "SPEED UP",
    libevdev.EV_KEY.KEY_KPMINUS: "SLOW DOWN",
    libevdev.EV_KEY.KEY_F11: "F+",
    libevdev.EV_KEY.KEY_F10: "F-",
    libevdev.EV_KEY.KEY_RIGHTBRACE: "J+",
    libevdev.EV_KEY.KEY_LEFTBRACE: "J-",
    libevdev.EV_KEY.KEY_UP: "+Y",
    libevdev.EV_KEY.KEY_DOWN: "-Y",
    libevdev.EV_KEY.KEY_LEFT: "-X",
    libevdev.EV_KEY.KEY_RIGHT: "+X",
    libevdev.EV_KEY.KEY_KP7: "+A",
    libevdev.EV_KEY.KEY_Q: "-A",
    libevdev.EV_KEY.KEY_PAGEDOWN: "-Z",
    libevdev.EV_KEY.KEY_PAGEUP: "+Z",
 
class KeyReader:
    def __init__(self, path: str):
        self.path = path
        self.fd: IO[bytes]   None = None
        self.device: libevdev.Device   None = None
    def __enter__(self):
        self.fd = open(self.path, "rb")
        self.device = libevdev.Device(self.fd)
        return self
    def __exit__(self, exc_type, exc, tb):
        self.device = None
        self.fd.close()
        self.fd = None
    def events(self) -> Iterator[dict[str, Any]]:
        for e in self.device.events():
            if e.type == libevdev.EV_KEY:
                if (val := KEY_MAP.get(e.code)):
                    yield  
                        "name": val,
                        "value": e.value,
                        "sec": e.sec,
                        "usec": e.usec,
                     
Edited: added rules to handle the Bluetooth page turner

31 March 2023

Enrico Zini: Things I learnt in March 2023

About JACK and Debian

30 March 2023

Jonathan McDowell: Buttering up my storage

(TL;DR: I ve been trying out btrfs in some places instead of ext4, I ve hit absolutely zero issues and there are a few features that make me plan to use it more.) Despite (or perhaps because of) working on storage products for a reasonable chunk of my career I have tended towards a conservative approach to my filesystems. By the time I came to Linux ext2 was well established, the move to ext3 was a logical one (the joys of added journalling for faster recovery after unclean shutdowns) and for a long time my default stack has been MD raid with LVM2 on top and then ext4 as the filesystem. I ve dabbled with other filesystems; I ran XFS for a while on my VDR machine, and also when I had a large tradspool with INN, but never really had a hard requirement for it. I ve ended up adminning a machine that had JFS in the past, largely for historical reasons, but don t really remember any issues (vague recollections of NFS problems but that might just have been NFS being NFS). However. ZFS has gathered itself a significant fan base and that makes me wonder about what it can offer and whether I want that. Firstly, let s be clear that I m never going to run a primary filesystem that isn t part of the mainline kernel. So ZFS itself is out, because I run Linux. So what do I want that I can t get with ext4? Firstly, I d like data checksumming. As storage gets larger there s a bigger chance of silent data corruption and while I have backups of the important stuff that doesn t help if you don t know you need to use them. Secondly, these days I have machines running containers, VMs, or with lots of source checkouts with a reasonable amount of overlap in their data. Disk space has got cheaper, but I d still like to be able to do some sort of deduplication of common blocks. So, I ve been trying out btrfs. When I installed my desktop I went with btrfs for / and /home (I kept /boot as ext4). The thought process was that this was a local machine (so easy access if it all went wrong) and I take regular backups (so if it all went wrong I could recover). That was a year and a half ago and it s been pretty dull; I mostly forget I m running btrfs instead of ext4. This is on a machine that tracks Debian testing, so currently on kernel 6.1 but originally installed with 5.10. So it seems modern btrfs is reasonably stable for a machine that isn t driven especially hard. Good start. The fact I forget what filesystem I m running points to the fact that I m not actually doing anything special here. I get the advantage of data checksumming, but not much else. 2 things spring to mind. Firstly, I don t do snapshots. Given I run testing it might be wiser if I did take a snapshot before every apt-get upgrade, and I have a friend who does just that, but even when I ve run unstable I ve never had a machine get itself into a state that I couldn t recover so I haven t spent time investigating. I note Ubuntu has apt-btrfs-snapshot but it doesn t seem to have any updates for years. The other thing I didn t do when I installed my desktop is take advantage of subvolumes. I m still trying to get my head around exactly what I want them for, but they provide a partial replacement for LVM when it comes to carving up disk space. Instead of the separate / and /home LVs I created I could have created a single LV that would have a single btrfs filesystem on it. / and /home would then be separate subvolumes, allowing me to snapshot each individually. Quotas can also be applied separately so there s still the potential to prevent one subvolume taking all available space. Encouraged by the lack of hassle with my desktop I decided to try moving my sbuild machine over to use btrfs for its build chroots. For Reasons this is a VM kindly hosted by a friend, rather than something local. To be honest these days I would probably go for local hosting, but it works and there s no strong reason to move. The point is it s remote, and so if migrating went wrong and I had to ask for assistance I d be bothering someone who s doing me a favour as it is. The build VM is, of course, running LVM, and there was luckily some free space available. I m reasonably sure the underlying storage involves spinning rust, so I did a laborious set of pvmove commands to make sure all the available space was at the start of the PV, and created a new btrfs volume there. I was advised that while btrfs-convert would do the job it was better to create a fresh filesystem where possible. This time I did create an initial root subvolume. Configuring up sbuild was then much simpler than I d expected. My setup originally started out as a set of tarballs for the chroots that would get untarred + used for the builds, which is pretty slow. Once overlayfs was mature enough I switched to that. I d had a conversation with Enrico about his nspawn/btrfs setup, but it turned out Russ Allbery had written an excellent set of instructions on sbuild with btrfs. I tweaked my existing setup based on his details, and I was in business. Each chroot is a separate subvolume - I don t actually end up having to mount them individually, but it means that only the chroot in use gets snapshotted. For example during a build the following can be observed:
# btrfs subvolume list /
ID 257 gen 111534 top level 5 path root
ID 271 gen 111525 top level 257 path srv/chroot/unstable-amd64-sbuild
ID 275 gen 27873 top level 257 path srv/chroot/bullseye-amd64-sbuild
ID 276 gen 27873 top level 257 path srv/chroot/buster-amd64-sbuild
ID 343 gen 111533 top level 257 path srv/chroot/snapshots/unstable-amd64-sbuild-328059a0-e74b-4d9f-be70-24b59ccba121
I was a little confused about whether I d got something wrong because the snapshot top level is listed as 257 rather than 271, but digging further with btrfs subvolume show on the 2 mounted directories correctly showed the snapshot had a parent equal to the chroot, not /. As a final step I ran jdupes via jdupes -1Br / to deduplicate things across the filesystem. It didn t end up providing a significant saving unfortunately - I guess there s a reasonable amount of change between Debian releases - but I think tried it on my desktop, which tends to have a large number of similar source trees checked out. There I managed to save about 5% on /home, which didn t seem too shabby. The sbuild setup has been in place for a couple of months now, and I ve run quite a few builds on it while preparing for the freeze. So I m fairly confident in the stability of the setup and my next move is to transition my local house server over to btrfs for its containers (which all run under systemd-nspawn). Those are generally running a Debian stable base so there should be a decent amount of commonality for deduping. I m not saying I m yet at the point where I ll default to btrfs on new installs, but I m definitely looking at it for situations where I think I can get benefits from deduplication, or being able to divide up disk space without hard partitioning space. (And, just to answer the worry I had when I started, I ve got nowhere near ENOSPC problems, but I believe they re handled much more gracefully these days. And my experience of ZFS when it got above 90% utilization was far from ideal too.)

5 March 2023

Enrico Zini: Heart-driven drum loop

I have Python code for reading a heart rate monitor. I have Python code to generate MIDI events. Could I resist putting them together? Clearly not. Here's Jack Of Hearts, a JACK MIDI drum loop generator that uses the heart rate for BPM, and an improvised way to compute heart rate increase/decrease to add variations in the drum pattern. It's very simple minded and silly. To me it was a fun way of putting unrelated things together, and Python worked very well for it.

Enrico Zini: Generating MIDI events with JACK and Python

I had a go at trying to figure out how to generate arbitrary MIDI events and send them out over a JACK MIDI channel. Setting up JACK and Pipewire Pipewire has a JACK interface, which in theory means one could use JACK clients out of the box without extra setup. In practice, one need to tell JACK clients which set of libraries to use to communicate to servers, and Pipewire's JACK server is not the default choice. To tell JACK clients to use Pipewire's server, you can either: Programming with JACK Python has a JACK client library that worked flawlessly for me so far. Everything with JACK is designed around minimizing latency. Everything happens around a callback that gets called form a separate thread, and which gets a buffer to fill with events. All the heavy processing needs to happen outside the callback, and the callback is only there to do the minimal amount of work needed to shovel the data your application produced into JACK channels. Generating MIDI messages The Mido library can be used to parse and create MIDI messages and it also worked flawlessly for me so far. One needs to study a bit what kind of MIDI message one needs to generate (like "note on", "note off", "program change") and what arguments they get. It also helps to read about the General MIDI standard which defines mappings between well-known instruments and channels and instrument numbers in MIDI messages. A timed message queue To keep a queue of events that happen over time, I implemented a Delta List that indexes events by their future frame number. I called the humble container for my audio experiments pyeep and here's my delta list implementation. A JACK player The simple JACK MIDI player backend is also in pyeep. It needs to protect the delta list with a mutex since we are working across thread boundaries, but it tries to do as little work under lock as possible, to minimize the risk of locking the realtime thread for too long. The play method converts delays in seconds to frame counts, and the on_process callback moves events from the queue to the jack output. Here's an example script that plays a simple drum pattern:
#!/usr/bin/python3
# Example JACK midi event generator
#
# Play a drum pattern over JACK
import time
from pyeep.jackmidi import MidiPlayer
# See:
# https://soundprogramming.net/file-formats/general-midi-instrument-list/
# https://www.pgmusic.com/tutorial_gm.htm
DRUM_CHANNEL = 9
with MidiPlayer("pyeep drums") as player:
    beat: int = 0
    while True:
        player.play("note_on", velocity=64, note=35, channel=DRUM_CHANNEL)
        player.play("note_off", note=38, channel=DRUM_CHANNEL, delay_sec=0.5)
        if beat == 0:
            player.play("note_on", velocity=100, note=38, channel=DRUM_CHANNEL)
            player.play("note_off", note=36, channel=DRUM_CHANNEL, delay_sec=0.3)
        if beat + 1 == 2:
            player.play("note_on", velocity=100, note=42, channel=DRUM_CHANNEL)
            player.play("note_off", note=42, channel=DRUM_CHANNEL, delay_sec=0.3)
        beat = (beat + 1) % 4
        time.sleep(0.3)
Running the example I ran the jack_drums script, and of course not much happened. First I needed a MIDI synthesizer. I installed fluidsynth, and ran it on the command line with no arguments. it registered with JACK, ready to do its thing. Then I connected things together. I used qjackctl, opened the graph view, and connected the MIDI output of "pyeep drums" to the "FLUID Synth input port". fluidsynth's output was already automatically connected to the audio card and I started hearing the drums playing!

20 February 2023

Jonathan McDowell: Fixing mobile viewing

It was brought to my attention recently that the mobile viewing experience of this blog was not exactly what I d hope for. In my poor defence I proof read on my desktop and the only time I see my posts on mobile is via FreshRSS. Also my UX ability sucks. Anyway. I ve updated the theme to a more recent version of minima and tried to make sure I haven t broken it all in the process (I did break tagging, but then I fixed it again). I double checked the generated feed to confirm it was the same (other than some re-tagging I did), so hopefully I haven t flooded anyone s feed. Hopefully I can go back to ignoring the underlying blog engine for another 5+ years. If not I ll have to take a closer look at Enrico s staticsite.

17 February 2023

Enrico Zini: Monitoring a heart rate monitor

I bought myself a cheap wearable Bluetooth LE heart rate monitor in order to play with it, and this is a simple Python script to monitor it and plot data. Bluetooth LE I was surprised that these things seem decently interoperable. You can use hcitool to scan for devices:
hcitool lescan
You can then use gatttool to connect to device and poke at them interactively from a command line. Bluetooth LE from Python There is a nice library called Bleak which is also packaged in Debian. It's modern Python with asyncio and works beautifully! Heart rate monitors Things I learnt: How about a proper fitness tracker? I found OpenTracks, also on F-Droid, which seems nice Why script it from a desktop computer? The question is: why not? A fitness tracker on a phone is useful, but there are lots of silly things one can do from one's computer that one can't do from a phone. A heart rate monitor is, after all, one more input device, and there are never enough input devices! There are so many extremely important use cases that seem entirely unexplored:

16 January 2023

Freexian Collaborators: Monthly report about Debian Long Term Support, December 2022 (by Anton Gladky)

Like each month, have a look at the work funded by Freexian s Debian LTS offering.

Debian LTS contributors In December, 17 contributors have been paid to work on Debian LTS, their reports are available:
  • Abhijith PA did 3.0h (out of 0h assigned and 14.0h from previous period), thus carrying over 11.0h to the next month.
  • Anton Gladky did 8.0h (out of 6.0h assigned and 9.0h from previous period), thus carrying over 7.0h to the next month.
  • Ben Hutchings did 24.0h (out of 9.0h assigned and 15.0h from previous period).
  • Chris Lamb did 18.0h (out of 18.0h assigned).
  • Dominik George did 0.0h (out of 10.0h assigned and 14.0h from previous period), thus carrying over 24.0h to the next month.
  • Emilio Pozuelo Monfort did 8.0h in December, 8.0h in November (out of 1.5h assigned and 49.5h from previous period), thus carrying over 43.0h to the next month.
  • Enrico Zini did 0.0h (out of 0h assigned and 8.0h from previous period), thus carrying over 8.0h to the next month.
  • Guilhem Moulin did 17.5h (out of 20.0h assigned), thus carrying over 2.5h to the next month.
  • Helmut Grohne did 15.0h (out of 15.0h assigned, 2.5h were taken from the extra-budget and worked on).
  • Markus Koschany did 40.0h (out of 40.0h assigned).
  • Ola Lundqvist did 10.0h (out of 7.5h assigned and 8.5h from previous period), thus carrying over 6.0h to the next month.
  • Roberto C. S nchez did 24.5h (out of 20.25h assigned and 11.75h from previous period), thus carrying over 7.5h to the next month.
  • Stefano Rivera did 2.5h (out of 20.5h assigned and 14.5h from previous period), thus carrying over 32.5h to the next month.
  • Sylvain Beucler did 20.5h (out of 37.0h assigned and 22.0h from previous period), thus carrying over 38.5h to the next month.
  • Thorsten Alteholz did 10.0h (out of 14.0h assigned), thus carrying over 4.0h to the next month.
  • Tobias Frost did 16.0h (out of 16.0h assigned).
  • Utkarsh Gupta did 51.5h (out of 42.5h assigned and 9.0h from previous period).

Evolution of the situation In December, we have released 47 DLAs, closing 232 CVEs. In the same year, in total we released 394 DLAs, closing 1450 CVEs. We are constantly growing and seeking new contributors. If you are a Debian Developer and want to join the LTS team, please contact us.

Thanks to our sponsors Sponsors that joined recently are in bold.

4 January 2023

Enrico Zini: Staticsite redesign

These are some notes about my redesign work in staticsite 2.x. Maping constraints and invariants I started keeping notes of constraints and invariants, and this helped a lot in keeping bounds on the cognitive efforts of design. I particularly liked how mapping the set of constraints added during site generation has helped breaking down processing into a series of well defined steps. Code that handles each step now has a specific task, and can rely on clear assumptions. Declarative page metadata I designed page metadata as declarative fields added to the Page class. I used typed descriptors for the fields, so that metadata fields can now have logic and validation, and are self-documenting! This is the core of the Field implementation. Lean core I tried to implement as much as possible in feature plugins, leaving to the staticsite core only what is essential to create the structure for plugins to build on. The core provides a tree structure, an abstract Page object that can render to a file and resolve references to other pages, a Site that holds settings and controls the various loading steps, and little else. The only type of content supported by the core is static asset files: Markdown, RestructuredText, images, taxonomies, feeds, directory indices, and so on, are all provided via feature plugins. Feature plugins Feature plugins work by providing functions to be called at the various loading steps, and mixins to be added to site pages. Mixins provided by feature plugins can add new declarative metadata fields, and extend Page methods: this ends up being very clean and powerful, and plays decently well with mypy's static type checking, too! See for example the code of the alias feature, that allows a page to declare aliases that redirect to it, useful for example when moving content around. It has a mixin (AliasPageMixin) that adds an aliases field that holds a list of page paths. During the "generate" step, when autogenerated pages can be created, the aliases feature iterates through all pages that defined an aliases metadata, and generates the corresponding redirection pages. Self-documenting code Staticsite can list loaded features, features can list the page subclasses that they use, and pages can list metadata fields. As a result, each feature, each type of page, and each field of each page can generate documentation about itself: the staticsite reference is autogenerated in that way, mostly from Feature, Page, and Field docstrings. Understand the language, stay close to the language Python has matured massively in the last years, and I like to stay on top of the language and standard library release notes for each release. I like how what used to be dirty hacks have now found a clean way into the language:

Next.