About 90% of my Debian contributions this month were
sponsored by Freexian.
You can also support my work directly via
Liberapay or
GitHub
Sponsors.
DebConf
I attended DebConf for the first time in 11 years (my last one was DebConf
14 in Portland). It was great! For once I had a conference where I had a
fairly light load of things I absolutely had to do, so I was able to spend
time catching up with old friends, making some new friends, and doing some
volunteering - a bit of Front Desk, and quite a lot of video team work where
I got to play with sound desks and such. Apparently one of the BoFs ( birds
of a feather , i.e. relatively open discussion sessions) where I was
talkmeister managed to break the automatic video cutting system by starting
and ending precisely on time, to the second, which I m told has never
happened before. I ll take that.
I gave a
talk about
Debusine,
along with
helping Enrico run a Debusine
BoF.
We still need to process some of the feedback from this, but are generally
pretty thrilled about the reception. My personal highlight was getting a
shout-out in a
talk from
CERN
(in the slide starting at 32:55).
Other highlights for me included a
Python team
BoF,
Ian s tag2upload
talk
and some very useful follow-up discussions, a session on
archive-wide
testing,
a somewhat brain-melting whiteboard session about the multiarch interpreter
problem ,
several
useful
discussions
about salsa.debian.org,
Matthew s talk on how Wikimedia automates their Debian package
builds,
and many others. I hope I can start attending regularly again!
OpenSSH
Towards the end of a release cycle, people tend to do more upgrade testing,
and this sometimes results in interesting problems. Manfred Stock reported
No new SSH connections possible during large part of upgrade to Debian
Trixie , and after a little testing in a
container I confirmed that this was a reproducible problem that would have
affected many people upgrading from Debian 12 (bookworm), with potentially
severe consequences for people upgrading remote systems. In fact, there
were two independent problems that each led to much the same symptom:
-
OpenSSH 9.8 split the monolithic
sshd listener process into two
pieces: a minimal network listener (still called sshd), and an
sshd-session process dealing with each individual session. (OpenSSH
10.0 further split sshd-session, adding an sshd-auth process that
deals with the user authentication phase of the protocol.) This hardens
the OpenSSH server by using different address spaces for privileged and
unprivileged code.
Before this change, when sshd received an incoming connection, it
forked and re-executed itself with some special parameters to deal with
it. After this change, it forks and executes sshd-session instead,
and sshd no longer accepts the parameters it used to accept for this.
Debian package upgrades happen in two phases: first we unpack the new
files onto disk, and then we run some package-specific configuration
steps which usually include things like restarting services. (I m
simplifying, but this is good enough for this post.) Normally this is
fine, and in fact desirable: the old service keeps on working, and this
approach often allows breaking what would otherwise be difficult cycles
by ensuring that the system is in a more coherent state before trying to
restart services. However, in this case, unpacking the new files onto
disk immediately means that new SSH connections no longer work: the old
sshd receives the connection and tries to hand it off to a
freshly-executed copy of the new sshd binary on disk, which no longer
supports this.
If you re just upgrading OpenSSH on its own or with a small number of
other packages, this isn t much of a problem as the listener will be
restarted quite soon; but if you re upgrading from bookworm to trixie,
there may be a long gap when you can t SSH to the system any more, and
if something fails in the middle of the upgrade then you could be in trouble.
So, what to do? I considered keeping a copy of the old sshd around
temporarily and patching the new sshd to re-execute it if it s being
run to handle an incoming connection, but that turned out to fail in my
first test: dependencies are normally only checked when configuring a
package, so it s possible to unpack openssh-server before unpacking a
newer libc6 that it depends on, at which point you can t execute the
new sshd at all. (That also means that the approach of restarting the
service at unpack time instead of configure time is a non-starter.) We
needed a different idea.
dpkg, the core Debian package manager, has a specialized facility
called diversions : you can tell it that when it s unpacking a
particular file it should put it somewhere else instead. This is
normally used by administrators when they want to install a
locally-modified version of a particular file at their own risk, or by
packages that knowingly override a file normally provided by some other
package. However, in this case it turns out to be useful for
openssh-server to temporarily divert one of its own files! When
upgrading from before 9.8, it now diverts /usr/sbin/sshd to
/usr/sbin/sshd.session-split before the new version is unpacked, then
removes the diversion and moves the new file into place once it s ready
to restart the service; this reduces the period when incoming
connections fail to a minimum. (We actually have to pretend that the
diversion is being performed on behalf of a slightly different package
since we re using dpkg-divert in a strange way here, but it all works.)
-
Most OpenSSH processes, including
sshd, check for a compatible version
of the OpenSSL library when they start up. This check used to be very
picky, among other things requiring both the major and minor number to
match. OpenSSL 3 has a better versioning
policy,
and so OpenSSH 9.4p1 relaxed this
check.
Unfortunately, bookworm shipped with OpenSSH 9.2p1, which means that as
soon as you unpack the new libssl3 during an upgrade (actually
libssl3t64 due to the 64-bit time_t
transition), sshd
stops working. This couldn t be fixed by a change in trixie; we needed
to change bookworm in advance of the upgrade so that it would tolerate
newer versions of OpenSSL. And time was tight if we wanted to maximize
the chance that people would apply that stable update before upgrading
to trixie; there isn t going to be another point release of Debian 12
before the release of Debian 13.
Fortunately, there s a
stable-updates
mechanism for exactly this sort of thing, and the stable release
managers kindly accepted my proposal to fix this there.
The net result is that if you apply updates to bookworm (including
stable-updates /
bookworm-updates, which is enabled by default) before
starting the upgrade to trixie, everything should be fine. Many thanks to
Manfred for reporting this with just enough time to spare that we were able
to fix it before Debian 13 is released in a few days!
debmirror
I did my twice-yearly refresh of debmirror s
mirror_size
documentation,
and applied a
patch from Christoph Goehre
to improve mirroring of installer files.
madison-lite
I
proposed renaming this project along
with the
rmadison tool in
devscripts, although I m not yet sure what a
good replacement name would be.
Python team
I upgraded python-expandvars, python-typing-extensions (in experimental),
and webtest to new upstream versions.
I backported fixes for some security vulnerabilities to unstable:
I fixed or helped to fix a number of release-critical bugs:
I fixed some other bugs, mostly
Severity: important:
I reinstated python3-mastodon s build-dependency on and recommendation of
python3-blurhash, now that the latter has been
fixed to use the correct
upstream source.
Like each month, have a look at the work funded by 
People keep asking me when I ll write the next instalment of my local voice assistant journey. I didn t mean for it to be so long since the last one, things have been busier than I d like. Anyway. Last time we d 
Debian 13 has finally 
Some variant of the following[1] worked for me.
The first line is the start of a for loop that runs on each node in my cluster a command using ssh. The argument -t is passed to attach a controlling terminal to STDIN, STDERR and STDOUT of this session, since there will not be an intervening shell to do it for us. The argument to ssh is a workflow of bash commands. They upgrade the 7.x system to the most recent packages on the repository. We then update the sources.list entries for the system to point at bookworm sources instead of bullseye. The package cache is updated and the proxmox-ve package is installed. Packages which are installed are upgraded to the versions from bookworm, and the installer concludes.
Dear reader, you might be surprised how many times I saw the word perl scroll by during the manual, serial scrolling of this install. It took hours. There were a few prompts, so stand by the keyboard!
[1]
I ve collected some more 
Some went on my laptop, of course, but some were selected for another
tool I use relatively often: more pattern weights like 
As I m currently somewhat underemployed, and could do with some extra income, I m starting 
Build your own customized Linux Mint ISO
Using the