Here s my monthly but brief update about the activities I ve done in the FOSS world.
Debian
Whilst I didn t get a chance to do much, here are still a few things that I worked on:
Prepared security update for wordpress for trixie and bookworm.
A few discussions with the new DFSG team, et al.
Assited a few folks in getting their patches submitted via Salsa.
Mentoring for newcomers.
Moderation of -project mailing list.
Ubuntu
I joined Canonical to work on Ubuntu full-time back in February 2021.
Whilst I can t give a full, detailed list of things I did, here s a quick TL;DR of what I did:
ceph: Affected by CVE-2024-47866, using the argument x-amz-copy-source to put an object and specifying an empty string as its content leads to the RGW daemon crashing, resulting in a DoS attack.
[LTS]: Whilst the patch is straightforward, backports are a bit tricky. Also trying to backport the fix for CVE-2022-0670. The update has been prepared as of December 16th and awaiting Thomas Goirand s review.
knot-resolver: Affected by CVE-2023-26249, CVE-2023-46317, and CVE-2022-40188, leading to Denial of Service.
[LTS]: The patches aren t very trivial, with
CVE-2023-46317 reverting much of the patch for CVE-2023-26249. And then it had to be re-adjusted a bit for v5.3.1. The backports are ready as of December 23rd and awaiting Jakub or Santiago s review.
adminer: Affected by CVE-2023-45195 and CVE-2023-45196, leading to SSRF and DoS, respectively.
[ELTS]: I picked it up again and the patches were already ready in Git. I ve reached out to Alexandre for a quick review and smoke test before we can release it for buster.
u-boot: Affected by CVE-2025-24857, where boot code access control flaw in U-Boot allowing arbitrary code execution via physical access.
[ELTS]: As it s only affected the version in stretch, I ve started the work to find the fixing commits and prepare a backport. Not much progress there, I ll roll it over to January.
ruby-rack: There were multiple vulnerabilities reported in Rack, leading to DoS (memory exhaustion) and proxy bypass.
[ELTS]: After completing the work for LTS myself, Bastien picked it up for ELTS and reached out about an upstream regression and we ve been doing some exchanges. Bastien has done most of the work backporting the patches but needs a review and help backporting CVE-2025-61771.
Other Activities
Frontdesk from 01-12-2025 to 07-12-2025.
Auto EOL d a bunch of packages.
Marked CVE-2025-12084/python2.7 as end-of-life for bullseye, buster, and stretch.
Marked CVE-2025-12084/jython as end-of-life for bullseye.
Marked CVE-2025-13992/chromium as end-of-life for bullseye.
Marked apache2 CVEs as postponed for bullseye, buster, and stretch.
Marked CVE-2025-13654/duc as postponed for bullseye and buster.
Marked CVE-2025-32900/kdeconnect as ignored for bullseye.
Marked CVE-2025-12084/pypy3 as postponed for bullseye.
Marked CVE-2025-14104/util-linux as postponed for bullseye, buster, and stretch.
Marked several CVEs for fastdds as postponed for bullseye.
Marked several CVEs for pytorch as postponed for bullseye.
Marked CVE-2025-2486/edk2 as postponed for bullseye.
Marked CVE-2025-6172 7,9 /golang-1.15 as postponed for bullseye.
Marked CVE-2025-65637/golang-logrus as postponed for bullseye.
Marked CVE-2025-12385/qtdeclarative-opensource-src ,gles as postponed for bullseye, buster, and stretch.
Marked TEMP-0000000-D08402/rust-maxminddb as postponed for bullseye.
Added the following packages to d,e la-needed.txt:
liblivemedia, sogo.
During my triage, I had to make the bin/elts-eol script robust to determine the lts_admin repository - did a back and forth with Emilio about this on the list.
I claimed php-horde-css-parser to work on CVE-2020-13756 for buster and did almost all the work only to realize that the patch already existed in buster and the changelog confirmed that it was intentionally fixed.
After speaking with Andreas Henriksson, we figured that the CVE ID was missed when the ELA was generated and so I fixed that via 87afaaf19ce56123bc9508d9c6cd5360b18114ef and 5621431e84818b4e650ffdce4c456daec0ee4d51 in the ELTS security tracker to reflect the situation.
Participated in a thread which I started last month around using Salsa CI for E/LTS packages and if we plan to sunset it in favor of using Debusine. The plan for now is to keep it around as it s still beneficial and Debusine is still in its early phase.
Did a lot of back and forth with Helmut about debusine uploads on #debian-elts.
While debugging a failure in dcut uploads, I ran into an SSH compatibility issue on deb-master.freexian.com that could be fixed on the server-side. I shared all my findings to Freexian s sysadmin team.
A minimal fix on the server side would be one of:
PubkeyAcceptedAlgorithms -ssh-dss
or explicitly restricting to modern algorithms, e.g.:
Jelly on #debian-lts reported that all my DLA mails had broken GMail s DKIM signature. So I set up sending replies from @debian.org and that seems to have fixed that! \o/
[LTS] Attended a rather short monthly LTS meeting on Jitsi. Summary here.
[E/LTS] Monitored discussions on mailing lists, IRC, and all the documentation updates.
Problem statement
Currently if you have an automatically installed package A (= 1) where
A (= 1) Depends B (= 1)
A (= 2) Depends B (= 2)
and you upgrade B from 1 to 2; then you can:
Remove A (= 1)
Upgrade A to version 2
If A was installed by a chain initiated by Recommends (say X Rec Y, Y Depends A), the solver sometimes preferred removing A (and anything depending on it until it got).
I have a fix pending to introduce eager Recommends which fixes the practical case, but
this is still not sound.
In fact we can show that the solver produces the wrong result for small
minimal test cases, as well as the right result for some others without the
fix (hooray?).
Ensuring sound removals is more complex, and first of all it begs the question: When
is a removal sound? This, of course, is on us to define.
An easy case can be found in the Debian policy, 7.6.2 Replacing whole packages, forcing their removal :
If B (= 2) declares a Conflicts: A (= 1) and Replaces: A (= 1), then the removal
is valid. However this is incomplete as well, consider it declares Conflicts: A (< 1)
and Replaces: A (< 1); the solution to remove A rather than upgrade it would still
be wrong.
This indicates that we should only allow removing A if the conflicts could not be solved
by upgrading it.
The other case to explore is package removals. If B is removed, A should be removed as well;
however it there is another package X that Provides: B (= 1) and it is marked for install,
A should not be removed. That said, the solver is not allowed to install X to satisfy the
depends B (= 1) - only to satisfy other dependencies [we do not want to get into endless
loops where we switch between alternatives to keep reverse dependencies installed].
Proposed solution
To solve this, I propose the following definition:
Definition (sound removal): A removal of package P is sound if either:
A version v is installed that package-conflicts with B.
A package Q is removed and the installable versions of P package-depends on Q.
where the other definitions are:
Definition (installable version): A version v is installable if either it is installed,
or it is newer than an installed version of the same package (you may wish to change this to
accomodate downgrades, or require strict pinning, but here be dragons).
Definition (package-depends): A version vpackage-depends on a package B if either:
there exists a dependency in v that can be solved by any version of B, or
there exists a package C where v package-depends C and any (c in C) package-depends B (transitivity)
Definition (package-conflicts): A version vpackage-conflicts with an installed package B if either:
it declares a conflicts against an installable version of B; or
there exists a package C where v package-conflicts C,
and b package-depends C for installable versions b.
Translating this into a (modified) SAT solver
One approach may be to implement the logic in the conflict analysis that drives backtracking, i.e.
we assume a package A and when we reach not A, we analyse if the implication graph for not A
constitutes a sound removal, and then replace the assumption A with the assumption
A or "learned reason.
However, while this seems a plausible mechanism for a DPLL solver, for a modern CDCL solver, it s
not immediately evident how to analyse whether not A is sound if the reason for it is a learned
clause, rather than a problem clause.
Instead we propose a static encoding of the rules into a slightly modified SAT solver:
Given c1, , cn that transitive-conflicts A and D1, , Dn that A package-depends on,
introduce the rule:
A unless c1 or c2 or ... cn ... or not D1 or not D2 ... or not Dn
Rules of the form A... unless B... - where A... and B... are CNF - are
intuitively the same as A... or B..., however the semantic here is different:
We are not allowed to select B... to satisfy this clause.
This requires a SAT solver that tracks a reason for each literal being assigned,
such as solver3, rather than a SAT solver like MiniSAT that only tracks reasons across
propagation (solver3 may track A depends B or C as the reason for B without evaluating
C, whereas MiniSAT would only track it as the reason given not C).
Is it actually sound?
The proposed definition of a sound removal may still proof unsound as I either missed
something in the conclusion of the proposed definition that violates my goal I set out
to achieve, or I missed some of the goals.
I challenge you to find cases that cause removals that look wrong :D
As I was shopping groceries I had a shocking realization: The active dependencies
of packages in a solver actually form a trie (a dependency A B - A or B - of
a package X is considered active if we marked X for install).
Consider the dependencies A B C, A B, B X.
In most package managers these just express alternatives, that is, the or relationship,
but in Debian packages, it also expresses a preference relationship between its operands,
so in A B C, A is preferred over B and B over C (and A transitively over C).
This means that we can convert the three dependencies into a trie as follows:
Solving the dependency here becomes a matter of trying to install the package
referenced by the first edge of the root, and seeing if that sticks. In this
case, that would be a . Let s assume that a failed to install, the next
step is to remove the empty node of a, and merging its children into the
root.
For ease of visualisation, we remove a from the dependency nodes as well,
leading us to a trie of the dependencies b , b c , and b x .
Presenting the Debian dependency problem, or the positive part of it as a
trie allows us for a great visualization of the problem but it may not proof
to be an effective implementation choice.
In the real world we may actually store this as a priority queue that we
can delete from. Since we don t actually want to delete from the queue
for real, our queue items are pairs of a pointer to dependency and an
activitity level, say A B@1.
Whenever a variable is assigned false, we look at its reverse dependencies
and bump their activity, and reinsert them (the priority of the item being
determined by the leftmost solution still possible, it has now changed).
When we iterate the queue, we remove items with a lower activity level:
Our queue is A B@1, A B C@1, B X@1
Rejecting A bump the activity for its reverse dependencies and reinset them:
Our queue is A B@1, A B C@1, (A )B@2, (A )B C@2, B X@1
We visit A B@1 but see the activity of the underlying dependency is now 2 and remove it
Our queue is A B C@1, (A )B@2, (A )B C@2, B X@1
We visit A B C@1 but see the activity of the underlying dependency is now 2 and remove it
Our queue is (A )B@2, (A )B C@2, B X@1
We visit A B@2, see the activity matches and find B is the solution.
DebConf 25, by Stefano Rivera and Santiago Ruano Rinc n
In July, DebConf 25 was held in Brest, France.
Freexian was a gold sponsor and most of the Freexian team attended the event.
Many fruitful discussions were had amongst our team and within the Debian
community.
DebConf itself was organized by a local team in Brest, that included Santiago
(who now lives in Uruguay). Stefano was also deeply involved in the
organization, as a DebConf committee member, core video team, and the lead
developer for the conference website. Running the conference took an enormous
amount of work, consuming all of Stefano and Santiago s time for most of July.
Lucas Kanashiro was active in the DebConf content team, reviewing talks and
scheduling them. There were many last-minute changes to make during the event.
Anupa Ann Joseph was part of the Debian publicity team doing live coverage of
DebConf 25 and was part of the DebConf 25 content team reviewing the talks.
She also assisted the local team to procure the lanyards.
Recorded sessions presented by Freexian collaborators, often alongside other
friends in Debian, included:
OpenSSH upgrades, by Colin Watson
Towards the end of a release cycle, people tend to do more upgrade testing, and
this sometimes results in interesting problems. Manfred Stock reported
No new SSH connections possible during large part of upgrade to Debian Trixie ,
which would have affected many people upgrading from Debian 12 (bookworm), with
potentially severe consequences for people upgrading remote systems. In fact,
there were two independent problems that each led to much the same symptom:
As part of hardening the OpenSSH server, OpenSSH 9.8 split the monolithic
sshd listener process into two pieces: a minimal network listener (still
called sshd), and an sshd-session process dealing with each individual
session. Before this change, when sshd received an incoming connection, it
forked and re-executed itself with some special parameters to deal with it;
after this change, it forks and executes sshd-session instead, and sshd no
longer accepts the parameters it used to accept for this.
Debian package upgrades happen (roughly) in two phases: first we unpack the new
files onto disk, and then we run some configuration steps which usually include
things like restarting services. Normally this is fine, because the old service
keeps on working until it s restarted. In this case, unpacking the new files
onto disk immediately stopped new SSH connections from working: the old sshd
received the connection and tried to hand it off to a freshly-executed copy of
the new sshd binary on disk, which no longer supports this. This wasn t much
of a problem when upgrading OpenSSH on its own or with a small number of other
packages, but in release upgrades it left a large gap when you can t SSH to the
system any more, and if anything fails in that interval then you could be in
trouble.
After trying a couple of other approaches, Colin landed on the idea of having
the openssh-server package divert /usr/sbin/sshd to
/usr/sbin/sshd.session-split before the unpack step of an upgrade from before
9.8, then removing the diversion and moving the new file into place once it s
ready to restart the service. This reduces the period when new connections fail
to a minimum.
Most OpenSSH processes, including sshd, check for a compatible version of
the OpenSSL library when they start up. This check used to be very picky, among
other things requiring both the major and minor part of the version number to
match. OpenSSL 3 has a better versioning policy,
and so OpenSSH 9.4p1 relaxed this check.
Unfortunately, bookworm shipped with OpenSSH 9.2p1, so as soon as you unpacked
the new OpenSSL library during an upgrade, sshd stopped working. This
couldn t be fixed by a change in trixie; we needed to change bookworm in advance
of the upgrade so that it would tolerate newer versions of OpenSSL, and time was
tight if we wanted this to be available before the release of Debian 13.
Fortunately, there s a
stable-updates
mechanism for exactly this sort of thing, and the stable release managers kindly
accepted Colin s proposal to fix this there.
The net result is that if you apply updates to bookworm (including
stable-updates / bookworm-updates, which is enabled by default) before
starting the upgrade to trixie, everything should be fine.
Cross compilation collaboration, by Helmut Grohne
Supporting cross building in Debian packages touches lots of areas of the
archive and quite some of these matters reside in shared responsibility between
different teams. Hence, DebConf was an ideal opportunity to settle long-standing
issues.
The cross building bof
sparked lively discussions as a significant
fraction of developers employ cross builds to get their work done. In the
trixie release, about two thirds of the packages can satisfy their cross
Build-Depends and about half of the packages actually can be cross built.
Miscellaneous contributions
Rapha l Hertzog updated tracker.debian.org to remove
references to Debian 10 which was moved to
archive.debian.org, and had many fruitful discussions
related to Debusine during DebConf 25.
Carles Pina prepared some data, questions and information for the DebConf 25
l10n and i18n BoF.
Carles Pina demoed and discussed possible next steps for
po-debconf-manager
with different teams in DebConf 25. He also reviewed Catalan translations and
sent them to the packages.
Carles Pina started investigating a django-compressor bug:
reproduced the bug consistently and prepared a PR for django-compressor upstream
(likely more details next month). Looked at packaging
frictionless-py.
Stefano Rivera triaged Python CVEs against pypy3.
Stefano prepared an upload of a new upstream release of pypy3 to Debian
experimental (due to the freeze).
Stefano uploaded python3.14 RC1 to Debian experimental.
Thorsten Alteholz uploaded a new upstream version of sane-airscan to
experimental. He also started to work on a new upstream version of hplip.
Colin backported fixes for CVE-2025-50181
and CVE-2025-50182 in python-urllib3, and
fixed several other release-critical or important bugs in Python team packages.
Lucas uploaded ruby3.4 to experimental as a starting point for the
ruby-defaults transition that will happen after Trixie release.
Lucas coordinated with the Release team the fix of the remaining RC bugs
involving ruby packages, and got them all fixed.
Lucas, as part of the Debian Ruby team, kicked off discussions to improve
internal process/tooling.
Lucas, as part of the Debian Outreach team, engaged in multiple discussions
around internship programs we run and also what else we could do to improve
outreach in the Debian project.
Lucas joined the Local groups BoF during DebConf 25 and shared all the good
experiences from the Brazilian community and committed to help to document
everything to try to support other groups.
Helmut reiterated the multiarch policy proposal
with a lot of help from Nattie Mayer-Hutchings, Rhonda D Vine and Stuart Prescott.
Helmut finished his work on the process based unschroot prototype
that was the main feature of his talk (see above).
Helmut analyzed a multiarch-related glibcupgrade failure
induced by a /usr-move mitigation of systemd and sent a patch and regression
fix both of which reached trixie in time. Thanks to Aurelien Jarno and the
release team for their timely cooperation.
Helmut resurrected an earlier discussion about changing the semantics of
Architecture: all packages in a multiarch context in order to improve the
long-standing interpreter problem. With help from Tollef Fog Heen better
semantics were discovered and agreement was reached with Guillem Jover and
Julian Andres Klode to consider this change. The idea is to record a concrete
architecture for every Architecture: all package in the dpkg database and
enable choosing it as non-native.
As you may recall from previous posts and elsewhere I have been busy writing a new solver for APT.
Today I want to share some of the latest changes in how to approach solving.
The idea for the solver was that manually installed packages are always protected from removals
in terms of SAT solving, they are facts. Automatically installed packages become optional unit
clauses. Optional clauses are solved after manual ones, they don t partake in normal unit propagation.
This worked fine, say you had
A # install request for A
B # manually installed, keep it
A depends on: conflicts-B C
Installing A on a system with B installed installed C, as it was not allowed to
install the conflicts-B package since B is installed.
However, I also introduced a mode to allow removing manually installed packages, and that s
where it broke down, now instead of B being a fact, our clauses looked like:
A # install request for A
A depends on: conflicts-B C
Optional: B # try to keep B installed
As a result, we installed conflicts-B and removed B; the steps the solver takes are:
A is a fact, mark it
A depends on: conflicts-B C is the strongest clause, try to install conflicts-B
We unit propagate that conflicts-B conflicts with B, so we mark not B
Optional: B is reached, but not satisfiable, ignore it because it s optional.
This isn t correct: Just because we allow removing manually installed packages doesn t mean that we should remove manually installed packages if we don t need to.
Fixing this turns out to be surprisingly easy. In addition to adding our optional (soft) clauses, let s first assume all of them!
But to explain how this works, we first need to explain some terminology:
The solver operates on a stack of decisions
enqueue means a fact is being added at the current decision level, and enqueued for propagation
assume bumps the decision level, and then enqueues the assumed variable
propagate looks at all the facts and sees if any clause becomes unit, and then enqueues it
unit is when a clause has a single literal left to assign
To illustrate this in pseudo Python code:
We introduce all our facts, and if they conflict, we are unsat:
for fact in facts:
enqueue(fact)
ifnot propagate():
returnFalse
For each optional literal, we register a soft clause and assume it. If the assumption fails,
we ignore it. If it succeeds, but propagation fails, we undo the assumption.
for optionalLiteral in optionalLiterals:
registerClause(SoftClause([optionalLiteral]))
if assume(optionalLiteral) andnot propagate():
undo()
Finally we enter the main solver loop:
whileTrue:
ifnot propagate():
ifnot backtrack():
returnFalseelif<all clauses are satisfied>:
returnTrueelif it := find("best unassigned literal satisfying a hard clause"):
assume(it)
elif it := find("best literal satisfying a soft clause"):
assume(it)
The key point to note is that the main loop will undo the assumptions in order; so
if you assume A,B,C and B is not possible, we will have also undone C. But since
C is also enqueued as a soft clause, we will then later find it again:
Solve finds a conflict, backtracks, and sets not C: State=[Assume(A),Assume(B),not(C)]
Solve finds a conflict, backtracks, and sets not B: State=[Assume(A),not(B)] C is no longer assumed either
Solve, assume C as it satisfies SoftClause([C]) as next best literal: State=[Assume(A),not(B),Assume(C)]
All clauses are satisfied, solution is A, not B, and C.
This is not (correct) MaxSAT, because we actually do not guarantee that we satisfy as many soft clauses as possible. Consider you have the following clauses:
Optional: A
Optional: B
Optional: C
B Conflicts with A
C Conflicts with A
There are two possible results here:
A If we assume A first, we are unable to satisfy B or C.
B,C If we assume either B or C first, A is unsat.
The question to ponder though is whether we actually need a global maximum or whether a local maximum is satisfactory in practice for a dependency solver
If you look at it, a naive MaxSAT solver needs to run the SAT solver 2**n times for n soft clauses, whereas our heuristic only needs n runs.
For dependency solving, it seems we do not seem have a strong need for a global maximum:
There are various other preferences between our literals, say priorities;
and empirically, from evaluating hundreds of regressions without the initial assumptions,
I can say that the assumptions do fix those cases and the result is correct.
Further improvements exist, though, and we can look into them if they are needed, such as:
Use a better heuristic:
If we assume 1 clause and solve, and we cause 2 or more clauses to become unsatisfiable,
then that clause is a local minimum and can be skipped.
This is a more common heuristical MaxSAT solver.
This gives us a better local maximum, but not a global one.
This is more or less what the Smart package manager did,
except that in Smart, all packages were optional, and the entire solution was scored.
It calculated a basic solution without optimization and then toggled each variable and saw if the score improved.
Implement an actual search for a global maximum:
This involves reading the literature.
There are various versions of this, for example:
Find unsatisfiable cores and use those to guide relaxation of clauses.
A bounds-based search, where we translate sum(satisifed clauses) > k into SAT, and then search in one of the following ways:
from 0 upward
from n downward
perform a binary search on [0, k] satisfied clauses.
Actually we do not even need to calculate sum constraints into CNF, because we can just add a specialized new type of constraint to our code.
The voting period and tally of votes for the Debian Project Leader election
has just concluded and the winner is Andreas Tille, who has been elected for
the second time. Congratulations!
Out of a total of 1,030 developers, 362 voted. As usual in Debian, the voting
method used was the
Condorcet method.
More information about the result is available in the Debian Project Leader
Elections 2025 page.
Many thanks to Andreas Tille, Gianfranco Costamagna, Julian Andres Klode, and
Sruthi Chandran for their campaigns, and to our Developers for voting.
The new term for the project leader started on April 21st and will expire on
April 20th 2026.
In my previous blog, I explored The New APT 3.0 solver.
Since then I have been at work in the test suite making tests pass and fixing some bugs.
You see for all intents and purposes, the new solver is a very stupid naive DPLL SAT solver (it just
so happens we don t actually have any pure literals in there). We can control it in a bunch of ways:
We can mark packages as install or reject
We can order actions/clauses. When backtracking the action that came later will be the first we
try to backtrack on
We can order the choices of a dependency - we try them left to right.
This is about all that we really want to do, we can t go if we reach a conflict, say oh but this
conflict was introduced by that upgrade, and it seems more important, so let s not backtrack on
the upgrade request but on this dependency instead. .
This forces us to think about lowering the dependency problem into this form, such that not only
do we get formally correct solutions, but also semantically correct ones. This is nice because
we can apply a systematic way to approach the issue rather than introducing ad-hoc rules in the
old solver which had a which of these packages should I flip the opposite way to break the conflict
kind of thinking.
Now our test suite has a whole bunch of these semantics encoded in it, and I m going to share some
problems and ideas for how to solve them. I can t wait to fix these and the error reporting and
then turn it on in Ubuntu and later Debian (the defaults change is a post-trixie change, let s
be honest).
apt upgrade is hard
The apt upgrade commands implements a safe version of dist-upgrade that essentially calculates the
dist-upgrade, and then undoes anything that would cause a package to be removed, but it (unlike its
apt-get counterpart) allows the solver to install new packages.
Now, consider the following package is installed:
X Depends: A (= 1) B
An upgrade from A=1 to A=2 is available. What should happen?
The classic solver would choose to remove X in a dist-upgrade, and then upgrade A, so it s answer
is quite clear: Keep back the upgrade of A.
The new solver however sees two possible solutions:
Install B to satisfy X Depends A (= 1) B.
Keep back the upgrade of A
Which one does it pick? This depends on the order in which it sees the upgrade action for A and the
dependency, as it will backjump chronologically. So
If it gets to the dependency first, it marks A=1 for install to satisfy A (= 1). Then it gets
to the upgrade request, which is just A Depends A (= 2) A (= 1) and sees it is satisfied
already and is content.
If it gets to the upgrade request first, it marks A=2 for install to satisfy A (= 2). Then
later it gets to X Depends: A (= 1) B, sees that A (= 1) is not satisfiable, and picks B.
We have two ways to approach this issue:
We always order upgrade requests last, so they will be kept back in case of conflicting dependencies
We require that, for apt upgrade a currently satisfied dependency must be satisfied by currently installed
packages, hence eliminating B as a choice.
Recommends are hard too
See if you have a X Recommends: A (= 1) and a new version of A, A (= 2), the solver currently
will silently break the Recommends in some cases.
But let s explore what the behavior of a X Recommends: A (= 1) in combination with an available upgrade
of A (= 2) should be. We could say the rule should be:
An upgrade should keep back A instead of breaking the Recommends
A dist-upgrade should either keep back A or remove X (if it is obsolete)
This essentially leaves us the same choices as for the previous problem, but with an interesting twist.
We can change the ordering (and we already did), but we could also introduce a new rule, promotions :
A Recommends in an installed package, or an upgrade to that installed package, where the Recommends
existed in the installed version, that is currently satisfied, must continue to be satisfied, that is,
it effectively is promoted to a Depends.
This neatly solves the problem for us. We will never break Recommends that are satisfied.
Likewise, we already have a Recommends demotion rule:
A Recommends in an installed package, or an upgrade to that installed package, where the Recommends
existed in the installed version, that is currently unsatisfied, will not be further evaluated (it
is treated like a Suggests is in the default configuration).
Whether we should be allowed to break Suggests with our decisions or not (the old autoremover did not,
for instance) is a different decision. Should we promote currently satisified Suggests to Depends as well?
Should we follow currently satisified Suggests so the solver sees them and doesn t autoremove them,
but treat them as optional?
tightening of versioned dependencies
Another case of versioned dependencies with alternatives that has complex behavior is something like
X Depends: A (>= 2) B
X Recommends: A (>= 2) B
In both cases, installing X should upgrade an A < 2 in favour of installing B. But a naive
SAT solver might not. If your request to keep A installed is encoded as A (= 1) A (= 2), then
it first picks A (= 1). When it sees the Depends/Recommends it will switch to B.
We can solve this again as in the previous example by ordering the keep A installed requests after
any dependencies. Notably, we will enqueue the common dependencies of all A versions first before
selecting a version of A, so something may select a version for us.
version narrowing instead of version choosing
A different approach to dealing with the issue of version selection is to not select a version
until the very last moment. So instead of selecting a version to satisfy A (>= 2) we instead
translate
Depends: A (>= 2)
into two rules:
The package selection rule:
Depends: A
This ensures that any version of A is installed (i.e. it adds a version choice clause, A (= 1) A (= 2)
in an example with two versions for A.
The version narrowing rule:
Conflicts: A (<< 2)
This outright would reject a choice of A (= 1).
So now we have 3 kinds of clauses:
package selection
version narrowing
version selection
If we process them in that order, we should surely be able to find the solution that best matches
the semantics of our Debian dependency model, i.e. selecting earlier choices in a dependency before
later choices in the face of version restrictions.
This still leaves one issue: What if our maintainer did not use Depends: A (>= 2) B but
e.g. Depends: A (= 3) B A (= 2). He d expect us to fall back to B if A (= 3) is not
installable, and not to B. But we d like to enqueue A and reject all choices other than 3
and 2. I think it s fair to say: Don t do that, then here.
Implementing strict pinning correctly
APT knows a single candidate version per package, this makes the solver relatively deterministic:
It will only ever pick the candidate, or an installed version. This also happens to significantly
reduce the search space which is good - less backtracking. An uptodate system will only ever have
one version per package that can be installed, so we never actually have to choose versions.
But of course, APT allows you to specify a non-candidate version of a package to install, for example:
apt install foo/oracular-proposed
The way this works is that the core component of the previous solver, which is the pkgDepCache
maintains what essentially amounts to an overlay of the policy that you could see with
apt-cache policy.
The solver currently however validates allowed version choices against the policy directly,
and hence finds these versions are not allowed and craps out. This is an interesting problem
because the solver should not be dependent on the pkgDepCache as the pkgDepCache initialization
(Building dependency tree...) accounts for about half of the runtime of APT (until the Y/n prompt)
and I d really like to get rid of it.
But currently the frontend does go via the pkgDepCache. It marks the packages in there, building
up what you could call a transaction, and then we translate it to the new solver, and once it is
done, it translates the result back into the pkgDepCache.
The current implementation of allowed version is implemented by reducing the search space, i.e.
every dependency, we outright ignore any non-allowed versions. So if you have a version 3 of A
that is ignored a Depends: A would be translated into A (= 2) A (= 1).
However this has two disadvantages. (1) It means if we show you why A could not be installed,
you don t even see A (= 3) in the list of choices and (2) you would need to keep the pkgDepCache
around for the temporary overrides.
So instead of actually enforcing the allowed version rule by filtering, a more reasonable
model is that we apply the allowed version rule by just marking every other version as not
allowed when discovering the package in the from depcache translation layer. This doesn t
really increase the search space either but it solves both our problem of making overrides
work and giving you a reasonable error message that lists all versions of A.
pulling up common dependencies to minimize backtracking cost
One of the common issues we have is that when we have a dependency group
A B C D
we try them in order, and if one fails, we undo everything it did, and move on to the next one. However,
this isn t perhaps the best choice of operation.
I explained before that one thing we do is queue the common dependencies of a package (i.e. dependencies
shared in all versions) when marking a package for install, but we don t do this here: We have already
lowered the representation of the dependency group into a list of versions, so we d need to extract the
package back out of it.
This can of course be done, but there may be a more interesting solution to the problem, in that we
simply enqueue all the common dependencies. That is, we add n backtracking levels for n possible
solutions:
We enqueue the common dependencies of all possible solutions deps(A)&deps(B)&deps(C)&deps(D)
We decide (adding a decision level) not to install D right now and enqueue deps(A)&deps(B)&deps(C)
We decide (adding a decision level) not to install C right now and enqueue deps(A)&deps(B)
We decide (adding a decision level) not to install B right now and enqueue A
Now if we need to backtrack from our choice of A we hopefully still have a lot of common dependencies
queued that we do not need to redo. While we have more backtracking levels, each backtracking level
would be significantly cheaper, especially if you have cheap backtracking (which admittedly we do not
have, yet anyway).
The caveat though is: It may be pretty expensive to find the common dependencies. We need to iterate
over all dependency groups of A and see if they are in B, C, and D, so we have a complexity of roughly
#A * (#B+#C+#D)
Each dependency group we need to check i.e. is X Y in B meanwhile has linear cost: We need to
compare the memory content of two pointer arrays containing the list of possible versions that
solve the dependency group.
This means that X Y and Y X are different dependencies of course, but that is to be expected
they are. But any dependency of the same order will have the same memory layout.
So really the cost is roughly N^4. This isn t nice.
You can apply various heuristics here on how to improve that, or you can even apply binary logic:
Enqueue common dependencies of A B C D
Move into the left half, enqueue of A B
Again divide and conquer and select A.
This has a significant advantage in long lists of choices, and also in the common case, where the
first solution should be the right one.
Or again, if you enqueue the package and a version restriction instead, you already get the common
dependencies enqueued for the chosen package at least.
APT 2.9.3 introduces the first iteration of the new solver codenamed
solver3, and now available with the solver 3.0 option. The new solver
works fundamentally different from the old one.
How does it work?
Solver3 is a fully backtracking dependency solving algorithm that defers
choices to as late as possible. It starts with an empty set of packages,
then adds the manually installed packages, and then installs packages
automatically as necessary to satisfy the dependencies.
Deferring the choices is implemented multiple ways:
First, all install requests
recursively mark dependencies with a single solution for install, and any
packages that are being rejected due to conflicts or user requests will
cause their reverse dependencies to be transitively marked as rejected,
provided their or group cannot be solved by a different package.
Second, any dependency with more than one choice is pushed to a priority
queue that is ordered by the number of possible solutions, such that we
resolve a b before a b c.
Not just by the number of solutions, though. One important point to
note is that optional dependencies, that is, Recommends, are always
sorting after mandatory dependencies. Do note on that: Recommended
packages do not nest in backtracking - dependencies of a Recommended
package themselves are not optional, so they will have to be resolved
before the next Recommended package is seen in the queue.
Another important step in deferring choices is extracting the common
dependencies of a package across its version and then installing them
before we even decide which of its versions we want to install - one
of the dependencies might cycle back to a specific version after all.
Decisions about package levels are recorded at a certain decision level,
if we reach a conflict we backtrack to the previous decision level,
mark the decision we made (install X) in the inverse (DO NOT INSTALL X),
reset all the state all decisions made at the higher level, and restore
any dependencies that are no longer resolved to the work queue.
Comparison to SAT solver design.
If you have studied SAT solver design, you ll find that essentially
this is a DPLL solver without pure literal elimination. A pure literal
eliminitation phase would not work for a package manager: First negative
pure literals (packages that everything conflicts with) do not exist,
and positive pure literals (packages nothing conflicts with) we do not want
to mark for install - we want to install as little as possible (well subject,
to policy).
As part of the solving phase, we also construct an implication graph, albeit
a partial one: The first package installing another package is marked as the
reason (A -> B), the same thing for conflicts (not A -> not B).
Once we have added the ability to have multiple parents in the implication
graph, it stands to reason that we can also implement the much more advanced
method of conflict-driven clause learning; where we do not jump back to the
previous decision level but exactly to the decision level that caused the
conflict. This would massively speed up backtracking.
What changes can you expect in behavior?
The most striking difference to the classic APT solver is that solver3 always keeps
manually installed packages around, it never offers to remove them. We will relax that
in a future iteration so that it can replace packages with new ones, that is, if your
package is no longer available in the repository (obsolete), but there is one that
Conflicts+Replaces+Provides it, solver3 will be allowed to install that and remove the
other.
Implementing that policy is rather trivial: We just need to queue obsolete replacement
as a dependency to solve, rather than mark the obsolete package for install.
Another critical difference is the change in the autoremove behavior: The new solver
currently only knows the strongest dependency chain to each package, and hence it will
not keep around any packages that are only reachable via weaker chains.
A common example is when gcc-<version> packages accumulate on your system over the
years. They all have Provides: c-compiler and the libtoolDepends: gcc c-compiler
is enough to keep them around.
New features
The new option --no-strict-pinning instructs the solver to consider all versions of
a package and not just the candidate version. For example, you could use apt install foo=2.0 --no-strict-pinning
to install version 2.0 of foo and upgrade - or downgrade - packages as needed to satisfy foo=2.0 dependencies.
This mostly comes in handy in use cases involving Debian experimental or the Ubuntu proposed pockets, where you
want to install a package from there, but try to satisfy from the normal release as much as possible.
The implication graph building allows us to implement an apt why command, that while not as nicely
detailed as aptitude, at least tells you the exact reason why a package is installed. It will only show
the strongest dependency chain at first of course, since that is what we record.
What is left to do?
At the moment, error information is not stored across backtracking in any way, but we generally
will want to show you the first conflict we reach as it is the most natural one; or all conflicts.
Currently you get the last conflict which may not be particularly useful.
Likewise, errors currently are just rendered as implication graphs of the form [not] A -> [not] B -> ...,
and we need to put in some work to present those nicely.
The test suite is not passing yet, I haven t really started working on it. A challenge is that most
packages in the test suite are manually installed as they are mocked, and the solver now doesn t remove
those.
We plan to implement the replacement logic such that foo can be replaced by foo2 Conflicts/Replaces/Provides foo
without needing to be automatically installed.
Improving the backtracking to be non-chronological conflict-driven clause learning would vastly
enhance our backtracking performance. Not that it seems to be an issue right now in my limited
testing (mostly noble 64-bit-time_t upgrades). A lot of that complexity you have normally is not
there because the manually installed packages and resulting unit propagation (single-solution
Depends/Reverse-Depends for Conflicts) already ground us fairly far in what changes we can actually make.
Once all the stuff has landed, we need to start rolling it out and gather feedback. On Ubuntu I d like
automated feedback on regressions (running solver3 in parallel, checking if result is worse and then
submitting an error to the error tracker), on Debian this could just be a role email address to send
solver dumps to.
At the same time, we can also incrementally start rolling this out. Like phased updates in Ubuntu,
we can also roll out the new solver as the default to 10%, 20%, 50% of users before going to the
full 100%. This will allow us to capture regressions early and fix them.
Ubuntu systems typically have up to 3 kernels installed, before they are auto-removed by apt on classic installs. Historically the installation was optimized for metered download size only. However, kernel size growth and usage no longer warrant such optimizations. During the 23.10 Mantic Minatour cycle, I led a coordinated effort across multiple teams to implement lots of optimizations that together achieved unprecedented install footprint improvements.
Given a typical install of 3 generic kernel ABIs in the default configuration on a regular-sized VM (2 CPU cores 8GB of RAM) the following metrics are achieved in Ubuntu 23.10 versus Ubuntu 22.04 LTS:
2x less disk space used (1,417MB vs 2,940MB, including initrd)
3x less peak RAM usage for the initrd boot (68MB vs 204MB)
0.5x increase in download size (949MB vs 600MB)
2.5x faster initrd generation (4.5s vs 11.3s)
approximately the same total time (103s vs 98s, hardware dependent)
For minimal cloud images that do not install either linux-firmware or modules extra the numbers are:
1.3x less disk space used (548MB vs 742MB)
2.2x less peak RAM usage for initrd boot (27MB vs 62MB)
0.4x increase in download size (207MB vs 146MB)
Hopefully, the compromise of download size, relative to the disk space & initrd savings is a win for the majority of platforms and use cases. For users on extremely expensive and metered connections, the likely best saving is to receive air-gapped updates or skip updates.
This was achieved by precompressing kernel modules & firmware files with the maximum level of Zstd compression at package build time; making actual .deb files uncompressed; assembling the initrd using split cpio archives - uncompressed for the pre-compressed files, whilst compressing only the userspace portions of the initrd; enabling in-kernel module decompression support with matching kmod; fixing bugs in all of the above, and landing all of these things in time for the feature freeze. Whilst leveraging the experience and some of the design choices implementations we have already been shipping on Ubuntu Core. Some of these changes are backported to Jammy, but only enough to support smooth upgrades to Mantic and later. Complete gains are only possible to experience on Mantic and later.
The discovered bugs in kernel module loading code likely affect systems that use LoadPin LSM with kernel space module uncompression as used on ChromeOS systems. Hopefully, Kees Cook or other ChromeOS developers pick up the kernel fixes from the stable trees. Or you know, just use Ubuntu kernels as they do get fixes and features like these first.
The team that designed and delivered these changes is large: Benjamin Drung, Andrea Righi, Juerg Haefliger, Julian Andres Klode, Steve Langasek, Michael Hudson-Doyle, Robert Kratky, Adrien Nader, Tim Gardner, Roxana Nicolescu - and myself Dimitri John Ledkov ensuring the most optimal solution is implemented, everything lands on time, and even implementing portions of the final solution.
APT currently knows about three types of upgrades:
upgrade without new packages (apt-get upgrade)
upgrade with new packages (apt upgrade)
upgrade with new packages and deletions (apt ,-get dist,full -upgrade)
All of these upgrade types are necessary to deal with upgrades within a
distribution release. Yes, sometimes even removals may be needed because
bug fixes require adding a Conflicts somewhere.
In Ubuntu we have a third type of upgrades, handled by a separate tool: release
upgrades. ubuntu-release-upgrader changes your sources.list, and applies various
quirks to the upgrade.
In this post, I want to look not at the quirk aspects but discuss how dependency
solving should differ between intra-release and inter-release upgrades.
Previous solver projects (such as Mancoosi) operated under the assumption that minimizing
the number of changes performed should ultimately be the main goal of a solver. This makes
sense as every change causes risks.
However it ignores a different risk, which especially applies when upgrading from one
distribution release to a newer one: Increasing divergence from the norm.
Consider a person installs foo in Debian 12. foo depends on a b, so a will
be automatically installed to satisfy the dependency. A release later, a has some
known issues and b is prefered, the dependency now reads: b a.
A classic solver would continue to keep a installed because it was installed before,
leading upgraded installs to have foo, a installed whereas new systems have foo, b
installed. As systems get upgraded over and over, they continue to diverge further and
further from new installs to the point that it adds substantial support effort.
My proposal for the new APT solver is that when we perform release upgrades, we forget
which packages where previously automatically installed.
We effectively perform a normalization: All systems with the same set of manually installed packages will end up with the same set of automatically installed packages.
Consider the solving starting with an empty set and then installing the latest version of each previously manually installed package: It will see now that foo depends
b a and install b (and a will be removed later on as its not part of the solution).
Another case of divergence is Suggests handling. Consider that foo also Suggests
s. You now install another package bar that depends s, hence s gets installed.
Upon removing bar, s is not being removed automatically because foo still suggests
it (and you may have grown used to foo s integration of s). This is because apt considers
Suggests to be important - they won t be automatically installed, but will not be automatically
removed.
In Ubuntu, we unset that policy on release upgrades to normalize the systems. The reasoning
for that is simple: While you may have grown to use s as part of foo during the release,
an upgrade to the next release already is big enough that removing s is going to have
less of an impact - breakage of workflows is expected between release upgrades.
I believe that apt release-upgrade will benefit from both of these design choices,
and in the end it boils down to a simple mantra:
On upgrades within a release, minimize changes.
On upgrades between releases, minimize divergence from fresh installs.
This is the story of the currently progressing changes to secure boot
on Ubuntu and the history of how we got to where we are.
taking a step back: how does secure boot on Ubuntu work?
Booting on Ubuntu involves three components after the firmware:
shim
grub
linux
Each of these is a PE binary signed with a key. The shim is signed by Microsoft s
3rd party key and embeds a self-signed Canonical CA certificate, and optionally a
vendor dbx (a list of revoked certificates or binaries). grub and linux (and fwupd)
are then signed by a certificate issued by that CA
In Ubuntu s case, the CA certificate is sharded: Multiple people each have a part
of the key and they need to meet to be able to combine it and sign things, such as
new code signing certificates.
BootHole
When BootHole happened in 2020, travel was suspended and we hence could not rotate
to a new signing certificate. So when it came to updating our shim for the CVEs, we
had to revoke all previously signed kernels, grubs, shims, fwupds by their hashes.
This generated a very large vendor dbx which caused lots of issues as shim exported
them to a UEFI variable, and not everyone had enough space for such large variables.
Sigh.
We decided we want to rotate our signing key next time.
This was also when upstream added SBAT metadata to shim and grub. This gives
a simple versioning scheme for security updates and easy revocation using a
simple EFI variable that shim writes to and reads from.
Spring 2022 CVEs
We still were not ready for travel in 2021, but during BootHole we developed the
SBAT mechanism, so one could revoke a grub or shim by setting a single EFI variable.
We actually missed rotating the shim this cycle as a new vulnerability was reported
immediately after it, and we decided to hold on to it.
2022 key rotation and the fall CVEs
This caused some problems when the 2nd CVE round came, as we did not have a shim
with the latest SBAT level, and neither did a lot of others, so we ended up deciding
upstream to not bump the shim SBAT requirements just yet. Sigh.
Anyway, in October we were meeting again for the first time at a Canonical sprint,
and the shardholders got together and created three new signing keys: 2022v1, 2022v2,
and 2022v3. It took us until January before they were installed into the signing service
and PPAs setup to sign with them.
We also submitted a shim 15.7 with the old keys revoked which came back at around
the same time.
Now we were in a hurry. The 22.04.2 point release was scheduled for around middle
of February, and we had nothing signed with the new keys yet, but our new shim
which we need for the point release (so the point release media remains bootable
after the next round of CVEs), required new keys.
So how do we ensure that users have kernels, grubs, and fwupd signed with the
new key before we install the new shim?
upgrade ordering
grub and fwupd are simple cases: For grub, we depend on the new version. We decided
to backport grub 2.06 to all releases (which moved focal and bionic up from 2.04), and
kept the versioning of the -signed packages the same across all releases, so we were
able to simply bump the Depends for grub to specify the new minimum version. For fwupd-efi,
we added Breaks.
(Actually, we also had a backport of the CVEs for 2.04 based grub, and we did publish that
for 20.04 signed with the old keys before backporting 2.06 to it.)
Kernels are a different story: There are about 60 kernels out there. My initial idea was
that we could just add Breaks for all of them. So our meta package linux-image-generic which
depends on linux-image-$(uname -r)-generic, we d simply add Breaks: linux-image-generic ( 5.19.0-31)
and then adjust those breaks for each series. This would have been super annoying, but
ultimately I figured this would be the safest option. This however caused concern, because
it could be that apt decides to remove the kernel metapackage.
I explored checking the kernels at runtime and aborting if we don t have a trusted
kernel in preinst. This ensures that if you try to upgrade shim without having a kernel,
it would fail to install. But this ultimately has a couple of issues:
It aborts the entire transaction at that point, so users will be unable to run
apt upgrade until they have a recent kernel.
We cannot even guarantee that a kernel would be unpacked first. So even if you got
a new kernel, apt/dpkg might attempt to unpack it first and then the preinst would fail
because no kernel is present yet.
Ultimately we believed the danger to be too large given that no kernels had yet been released
to users. If we had kernels pushed out for 1-2 months already, this would have been a viable
choice.
So in the end, I ended up modifying the shim packaging to install both the latest shim and
the previous one, and an update-alternatives alternative to select between the two:
In it s post-installation maintainer script, shim-signed checks whether all kernels with a
version greater or equal to the running one are not revoked, and if so, it will setup the
latest alternative with priority 100 and the previous with a priority of 50.
If one or more of those kernels was signed with a revoked key, it will swap the priorities
around, so that the previous version is preferred.
Now this is fairly static, and we do want you to switch to the latest shim eventually, so
I also added hooks to the kernel install to trigger the shim-signed postinst script when
a new kernel is being installed. It will then update the alternatives based on the current
set of kernels, and if it now points to the latest shim, reinstall shim and grub to the
ESP.
Ultimately this means that once you install your 2nd non-revoked kernel, or you install
a non-revoked kernel and then reconfigure shim or the kernel, you will get the latest
shim. When you install your first non-revoked kernel, your currently booted kernel is
still revoked, so it s not upgraded immediately. This has a benefit in that you will
most likely have two kernels you can boot without disabling secure boot.
regressions
Of course, the first version I uploaded had still some remaining hardcoded shimx64
in the scripts and so failed to install on arm64 where shimaa64 is used. And if that
were not enough, I also forgot to include support for gzip compressed kernels there.
Sigh, I need better testing infrastructure to be able to easily run arm64 tests as
well (I only tested the actual booting there, not the scripts).
shim-signed migrated to the release pocket in lunar fairly quickly, but this caused
images to stop working, because the new shim was installed into images, but no
kernel was available yet, so we had to demote it to proposed and block migration.
Despite all the work done for end users, we need to be careful to roll this out for
image building.
another grub update for OOM issues.
We had two grubs to release: First there was the security update for the recent set
of CVEs, then there also was an OOM issue for large initrds which was blocking critical
OEM work.
We fixed the OOM issue by cherry-picking all 2.12 memory management patches, as well
as the red hat patches to the loader we take from there. This ended up a fairly large
patch set and I was hesitant to tie the security update to that, so I ended up pushing
the security update everywhere first, and then pushed the OOM fixes this week.
With the OOM patches, you should be able to boot initrds of between 400M and 1GB, it
also depends on the memory layout of your machine and your screen resolution and background
images. So OEM team had success testing 400MB irl, and I tested up to I think it was 1.2GB
in qemu, I ran out of FAT space then and stopped going higher :D
other features in this round
Intel TDX support in grub and shim
Kernels are allocated as CODE now not DATA as per the upstream mm changes, might fix boot on X13s
am I using this yet?
The new signing keys are used in:
shim-signed 1.54 on 22.10+, 1.51.3 on 22.04, 1.40.9 on 20.04, 1.37~18.04.13 on 18.04
grub2-signed 1.187.2~ or newer (binary packages grub-efi-amd64-signed or grub-efi-arm64-signed),
1.192 on 23.04.
fwupd-signed 1.51~ or newer
various linux updates. Check apt changelog linux-image-unsigned-$(uname -r) to see if
Revoke & rotate to new signing key (LP: #2002812) is mentioned in there to see if it
signed with the new key.
If you were able to install shim-signed, your grub and fwupd-efi will have the correct
version as that is ensured by packaging. However your shim may still point to the old one.
To check which shim will be used by grub-install, you can check the status of the shimx64.efi.signed
or (on arm64) shimaa64.efi.signed alternative. The best link needs to point to the file ending in
latest:
$ update-alternatives --display shimx64.efi.signed
shimx64.efi.signed - auto mode
link best version is /usr/lib/shim/shimx64.efi.signed.latest
link currently points to /usr/lib/shim/shimx64.efi.signed.latest
link shimx64.efi.signed is /usr/lib/shim/shimx64.efi.signed
/usr/lib/shim/shimx64.efi.signed.latest - priority 100
/usr/lib/shim/shimx64.efi.signed.previous - priority 50
If it does not, but you have installed a new kernel compatible with the new shim, you can
switch immediately to the new shim after rebooting into the kernel by running dpkg-reconfigure shim-signed. You ll see in the output if the shim was updated, or you can check the output
of update-alternatives as you did above after the reconfiguration has finished.
For the out of memory issues in grub, you need grub2-signed 1.187.3~ (same binaries
as above).
how do I test this (while it s in proposed)?
upgrade your kernel to proposed and reboot into that
upgrade your grub-efi-amd64-signed, shim-signed, fwupd-signed to proposed.
If you already upgraded your shim before your kernel, don t worry:
upgrade your kernel and reboot
run dpkg-reconfigure shim-signed
And you ll be all good to go.
deep dive: uploading signed boot assets to Ubuntu
For each signed boot asset, we build one version in the latest stable release and the
development release. We then binary copy the built binaries from the latest stable release
to older stable releases. This process ensures two things: We know the next stable release
is able to build the assets and we also minimize the number of signed assets.
OK, I lied. For shim, we actually do not build in the development release but copy the
binaries upward from the latest stable, as each shim needs to go through external signing.
The entire workflow looks something like this:
Upload the unsigned package to one of the following build PPAs:
Copy the unsigned package back across all stable releases in the PPA
Upload the signed package for stable releases to the same PPA with ~<release>.1 appended to the version
Submit a request to canonical-signing-jobs to sign the uploads.
The signing job helper copies the binary -unsigned packages to the primary-2022v1 PPA where they are
signed, creating a signing tarball, then it copies the source package for the -signed package to the
same PPA which then downloads the signing tarball during build and places the signed assets into
the -signed deb.
Resulting binaries will be placed into the proposed PPA: https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/proposed
Review the binaries themselves
Unembargo and binary copy the binaries from the proposed PPA to the proposed-public PPA: https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/proposed-public.
This step is not strictly necessary, but it enables tools like sru-review to work, as they cannot access the packages from the normal private proposed PPA.
Binary copy from proposed-public to the proposed queue(s) in the primary archive
Lots of steps!
WIP
As of writing, only the grub updates have been released, other updates are still being
verified in proposed. An update for fwupd in bionic will be issued at a later point, removing
the EFI bits from the fwupd 1.2 packaging and using the separate fwupd-efi project instead
like later release series.
Z3 is a theorem prover developed at Microsoft research and available as
a dynamically linked C++ library in Debian-based distributions. While the
library is a whopping 16 MB, and the solver is a tad slow, it s permissive
licensing, and number of tactics offered give it a huge potential for use
in solving dependencies in a wide variety of applications.
Z3 does not need normalized formulas, but offers higher level abstractions
like atmost and atleast and implies, that we will make use of together
with boolean variables to translate the dependency problem to a form Z3
understands.
In this post, we ll see how we can apply Z3 to the dependency resolution
in APT. We ll only discuss the basics here, a future post will explore
optimization criteria and recommends.
Translating the universe
APT s package universe consists of 3 relevant things: packages (the tuple
of name and architecture), versions (basically a .deb), and dependencies
between versions.
While we could translate our entire universe to Z3 problems, we instead will
construct a root set from packages that were manually installed and versions
marked for installation, and then build the transitive root set from it by
translating all versions reachable from the root set.
For each package P in the transitive root set, we create a boolean literal P. We then
translate each version P1, P2, and so on. Translating a version means building
a boolean literal for it, e.g. P1, and then translating the dependencies as shown below.
We now need to create two more clauses to satisfy the basic requirements for debs:
If a version is installed, the package is installed; and vice versa. We can encode
this requirement for P above as P == atleast( P1,P2 , 1).
There can only be one version installed. We add an additional constraint of the
form atmost( P1,P2 , 1).
We also encode the requirements of the operation.
For each package P that is manually installed, add a constraint P.
For each version V that is marked for install, add a constraint V.
For each package P that is marked for removal, add a constraint !P.
Dependencies
Packages in APT have dependencies of two basic forms: Depends and Conflicts,
as well as variations like Breaks (identical to Conflicts in solving terms),
and Recommends (soft Depends) - we ll ignore those for now. We ll discuss
Conflicts in the next section.
Let s take a basic dependency list: A Depends: X Y, Z. To represent that
dependency, we expand each name to a list of versions that can satisfy
the dependency, for example X1 X2 Y1, Z1.
Translating this dependency list to our Z3 solver, we create boolean variables
X1,X2,Y1,Z1 and define two rules:
A implies atleast( X1,X2,Y1 , 1)
A implies atleast( Z1 , 1)
If there actually was nothing that satisfied the Z requirement, we d have added
a rule not A. It would be possible to simply not tell Z3 about the version at
all as an optimization, but that adds more complexity, and the not A constraint
should not cause too many problems.
Conflicts
Conflicts cannot have or in them. A dependency B Conflicts: X, Y means that only
one of B, X, and Y can be installed. We can directly encode this in Z3 by using the
constraint atmost( B,X,Y , 1). This is an optimized encoding of the constraint: We
could have encoded each conflict in the form !B or !X, !B or !X, and so on. Usually
this leads to worse performance as it introduces additional clauses.
Complete example
Let s assume we start with an empty install and want to install the package a below.
Package: a
Version: 1
Depends: c b
Package: b
Version: 1
Package: b
Version: 2
Conflicts: x
Package: d
Version: 1
Package: x
Version: 1
The translation in Z3 rules looks like this:
Package rules for a:
a == atleast( a1 , 1) - package is installed iff one version is
atmost( a1 , 1) - only one version may be installed
a a must be installed
Dependency rules for a
implies(a1, atleast( b2, b1 , 1)) the translated dependency above. note that c is gone, it s not reachable.
Package rules for b:
b == atleast( b1,b2 , 1) - package is installed iff one version is
atmost( b1, b2 , 1) - only one version may be installed
Dependencies for b (= 2):
atmost( b2, x1 , 1) - the conflicts between x and b = 2 above
Package rules for x:
x == atleast( x1 , 1) - package is installed iff one version is
atmost( x1 , 1) - only one version may be installed
The package d is not translated, as it is not reachable from the root
set a1 , the transitive root set is a1,b1,b2,x1 .
Next iteration: Optimization
We have now constructed the basic set of rules that allows us to
solve solve our dependency problems (equivalent to SAT), however
it might lead to suboptimal solutions where it removes automatically
installed packages, or installs more packages than necessary, to
name a few examples.
In our next iteration, we have to look at introducing optimization;
for example, have the minimum number of removals, the minimal
number of changed packages, or satisfy as many recommends as possible.
We will also look at the upgrade problem (upgrade as many packages as
possible), the autoremove problem (remove as many automatically installed
packages as possible).
When Julian Andres Klode and I added initial Zstandard compression support to Ubuntu s APT and dpkg in Ubuntu 18.04 LTS we planned getting the changes accepted to Debian quickly and making Ubuntu 18.10 the first release where the new compression could speed up package installations and upgrades. Well, it took slightly longer than that.
Since then many other packages have been updated to support zstd compressed packages and read-only compression has been back-ported to the 16.04 Xenial LTS release, too, on Ubuntu s side. In Debian, zstd support is available now in APT, debootstrap and reprepro (thanks Dimitri!). It is still under review for inclusion in Debian s dpkg (BTS bug 892664).
Given that there is sufficient archive-wide support for zstd, Ubuntu is switching to zstd compressed packages in Ubuntu 21.10, the current development release. Please welcome hello/2.10-2ubuntu3, the first zstd-compressed Ubuntu package that will be followed by many other built with dpkg (>= 1.20.9ubuntu2), and enjoy the speed!
This is an edited copy of an email I sent to provide guidance to users of apt-key as to how to handle things in a post apt-key world.
The manual page already provides all you need to know for replacing apt-key add usage:
Note: Instead of using this command a keyring should be placed directly in the /etc/apt/trusted.gpg.d/ directory with a descriptive name and either gpg or asc as file extension
So it s kind of surprising people need step by step instructions for how to copy/download a file into a directory.
I ll also discuss the alternative security snakeoil approach with signed-by that s become popular. Maybe we should not have added signed-by, people seem to forget that debs still run maintainer scripts as root.
Aside from this email, Debian users should look into extrepo, which manages curated external repositories for you.
Some people will tell you to download the .asc and pipe it to gpg --dearmor, but gpg might not be installed, so really, just offer a .gpg one instead that is supported on all systems.
wget might not be available everywhere so you can use apt-helper:
Pretending to be safer by using signed-by
People say it s good practice to not use trusted.gpg.d and install the file elsewhere and then refer to it from the sources.list entry
by using signed-by=<path to the file>. So this looks a lot safer, because now your key can t sign other unrelated repositories. In
practice, security increase is minimal, since package maintainer scripts run as root anyway. But I guess it s better for publicity :)
As an example, here are the instructions to install signal-desktop from signal.org. As mentioned, gpg --dearmor use in there is not a good idea, and I d personally not tell people to modify /usr as it s supposed to be managed by the package manager, but we don t have an /etc/apt/keyrings or similar at the moment; it s fine though if the keyring is installed by the package. You can also just add the file there as a starting point, and then install
a keyring package overriding it (pretend there is a signal-desktop-keyring package below that would override the .gpg we added).
# NOTE: These instructions only work for 64 bit Debian-based
# Linux distributions such as Ubuntu, Mint etc.
# 1. Install our official public software signing key
wget -O- https://updates.signal.org/desktop/apt/keys.asc gpg --dearmor > signal-desktop-keyring.gpg
cat signal-desktop-keyring.gpg sudo tee -a /usr/share/keyrings/signal-desktop-keyring.gpg > /dev/null
# 2. Add our repository to your list of repositories
echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/signal-desktop-keyring.gpg] https://updates.signal.org/desktop/apt xenial main' \
sudo tee -a /etc/apt/sources.list.d/signal-xenial.list
# 3. Update your package database and install signal
sudo apt update && sudo apt install signal-desktop
I do wonder why they do wget gpg --dearmor, pipe that into the file and then cat sudo tee it, instead of having that all in one pipeline. Maybe they want nicer progress reporting.
Scenario-specific guidance
We have three scenarios:
For system image building, shipping the key in /etc/apt/trusted.gpg.d seems reasonable to me; you are the vendor sort of, so it can be globally trusted.
Chrome-style debs and repository config debs: If you ship a deb, embedding the sources.list.d snippet (calling it $myrepo.list) and shipping a $myrepo.gpg in /usr/share/keyrings is the best approach. Whether you ship that in product debs aka vscode/chromium or provide a repository configuration deb (let s call it myrepo-repo.deb) and then tell people to run apt update followed by apt install <package inside the repo> depends on how many packages are in the repo, I guess.
Manual instructions (signal style): The third case, where you tell people to run wget themselves, I find tricky.
As we see in signal, just stuffing keyring files into /usr/share/keyrings is popular, despite /usr supposed to be managed by the package manager.
We don t have another dir inside /etc (or /usr/local), so it s hard to suggest something else.
There s no significant benefit from actually using signed-by, so it s kind of extra work for little gain, though.
Addendum: Future work
This part is new, just for this blog post. Let s look at upcoming changes and how they make things easier.
Bundled .sources files
Assuming I get my merge request merged, the next version of APT (2.4/2.3.something) will do away with all the complexity and allow you to embed the key directly into a deb822 .sources file (which have been available for some time now):
Types: deb
URIs: https://myrepo.example/ https://myotherrepo.example/
Suites: stable not-so-stable
Components: main
Signed-By:
-----BEGIN PGP PUBLIC KEY BLOCK-----
.
mDMEYCQjIxYJKwYBBAHaRw8BAQdAD/P5Nvvnvk66SxBBHDbhRml9ORg1WV5CvzKY
CuMfoIS0BmFiY2RlZoiQBBMWCgA4FiEErCIG1VhKWMWo2yfAREZd5NfO31cFAmAk
IyMCGyMFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQREZd5NfO31fbOwD6ArzS
dM0Dkd5h2Ujy1b6KcAaVW9FOa5UNfJ9FFBtjLQEBAJ7UyWD3dZzhvlaAwunsk7DG
3bHcln8DMpIJVXht78sL
=IE0r
-----END PGP PUBLIC KEY BLOCK-----
Then you can just provide a .sources files to users,
they place it into sources.list.d,
and everything magically works
Probably adding a nice apt add-source command for it I guess.
Well, python-apt s aptsources package still does not support deb822 sources, and
never will, we ll need an aptsources2 for that for backwards-compatibility reasons,
and then port software-properties and other users to it.
OpenPGP vs aptsign
We do have a better, tighter replacement for gpg in the works which uses Ed25519
keys to sign Release files. It s temporarily named aptsign, but it s a generic
signer for single-section deb822 files, similar to signify/minisign.
We believe that this solves the security nightmare that our OpenPGP integration
is while reducing complexity at the same time. Keys are much shorter, so the
bundled sources file above will look much nicer.
Debian v11 with codename bullseye is supposed to be released as new stable release soon-ish (let s hope for June, 2021! :)). Similar to what we had with #newinbuster and previous releases, now it s time for #newinbullseye!
I was the driving force at several of my customers to be well prepared for bullseye before its freeze, and since then we re on good track there overall. In my opinion, Debian s release team did (and still does) a great job I m very happy about how unblock requests (not only mine but also ones I kept an eye on) were handled so far.
As usual with major upgrades, there are some things to be aware of, and hereby I m starting my public notes on bullseye that might be worth also for other folks. My focus is primarily on server systems and looking at things from a sysadmin perspective.
Further readings
Of course start with taking a look at the official Debian release notes, make sure to especially go through What s new in Debian 11 + Issues to be aware of for bullseye.
Chris published notes on upgrading to Debian bullseye, and also anarcat published upgrade notes for bullseye.
Package versions
As a starting point, let s look at some selected packages and their versions in buster vs. bullseye as of 2021-05-27 (mainly having amd64 in mind):
Package
buster/v10
bullseye/v11
ansible
2.7.7
2.10.8
apache
2.4.38
2.4.46
apt
1.8.2.2
2.2.3
bash
5.0
5.1
ceph
12.2.11
14.2.20
docker
18.09.1
20.10.5
dovecot
2.3.4
2.3.13
dpkg
1.19.7
1.20.9
emacs
26.1
27.1
gcc
8.3.0
10.2.1
git
2.20.1
2.30.2
golang
1.11
1.15
libc
2.28
2.31
linux kernel
4.19
5.10
llvm
7.0
11.0
lxc
3.0.3
4.0.6
mariadb
10.3.27
10.5.10
nginx
1.14.2
1.18.0
nodejs
10.24.0
12.21.0
openjdk
11.0.9.1
11.0.11+9 + 17~19
openssh
7.9p1
8.4p1
openssl
1.1.1d
1.1.1k
perl
5.28.1
5.32.1
php
7.3
7.4+76
postfix
3.4.14
3.5.6
postgres
11
13
puppet
5.5.10
5.5.22
python2
2.7.16
2.7.18
python3
3.7.3
3.9.2
qemu/kvm
3.1
5.2
ruby
2.5.1
2.7+2
rust
1.41.1
1.48.0
samba
4.9.5
4.13.5
systemd
241
247.3
unattended-upgrades
1.11.2
2.8
util-linux
2.33.1
2.36.1
vagrant
2.2.3
2.2.14
vim
8.1.0875
8.2.2434
zsh
5.7.1
5.8
Linux Kernel
The bullseye release will ship a Linux kernel based on v5.10 (v5.10.28 as of 2021-05-27, with v5.10.38 pending in unstable/sid), whereas buster shipped kernel 4.19. As usual there are plenty of changes in the kernel area and this might warrant a separate blog entry, but to highlight some issues:
One surprising change might be that the scrollback buffer (Shift + PageUp) is gone from the Linux console. Make sure to always use screen/tmux or handle output through a pager of your choice if you need all of it and you re in the console.
The kernel provides BTF support (via CONFIG_DEBUG_INFO_BTF, see #973870), which means it s no longer necessary to install LLVM, Clang, etc (requiring >100MB of disk space), see Gregg s excellent blog post regarding the underlying rational. Sadly the libbpf-tools packaging didn t make it into bullseye (#978727), but if you want to use your own self-made Debian packages, my notes might be useful.
With kernel version 5.4, SUBDIRS support was removed from kbuild, so if an out-of-tree kernel module (like a *-dkms package) fails to compile on bullseye, make sure to use a recent version of it which uses M= or KBUILD_EXTMOD= instead.
Unprivileged user namespaces are enabled by default (see #898446 + #987777), so programs can create more restricted sandboxes without the need to run as root or via a setuid-root helper. If you prefer to keep this feature restricted (or tools like web browsers, WebKitGTK, Flatpak, don t work), use sysctl -w kernel.unprivileged_userns_clone=0 .
The /boot/System.map file(s) no longer provide the actual data, you need to switch to the dbg package if you rely on that information:
% cat /boot/System.map-5.10.0-6-amd64
ffffffffffffffff B The real System.map is in the linux-image-<version>-dbg package
Be aware though, that the *-dbg package requires ~5GB of additional disk space.
Systemd
systemd v247 made it into bullseye (updated from v241). Same as for the kernel this might warrant a separate blog entry, but to mention some highlights:
Systemd in bullseye activates its persistent journal functionality by default (storing its files in /var/log/journal/, see #717388).
systemd-timesyncd is no longer part of the systemd binary package itself, but available as standalone package. This allows usage of ntp, chrony, openntpd, without having systemd-timesyncd installed (which prevents race conditions like #889290, which was biting me more than once).
journalctl gained new options:
--cursor-file=FILE Show entries after cursor in FILE and update FILE
--facility=FACILITY... Show entries with the specified facilities
--image=IMAGE Operate on files in filesystem image
--namespace=NAMESPACE Show journal data from specified namespace
--relinquish-var Stop logging to disk, log to temporary file system
--smart-relinquish-var Similar, but NOP if log directory is on root mount
systemctl gained new options:
clean UNIT... Clean runtime, cache, state, logs or configuration of unit
freeze PATTERN... Freeze execution of unit processes
thaw PATTERN... Resume execution of a frozen unit
log-level [LEVEL] Get/set logging threshold for manager
log-target [TARGET] Get/set logging target for manager
service-watchdogs [BOOL] Get/set service watchdog state
--with-dependencies Show unit dependencies with 'status', 'cat', 'list-units', and 'list-unit-files'
-T --show-transaction When enqueuing a unit job, show full transaction
--what=RESOURCES Which types of resources to remove
--boot-loader-menu=TIME Boot into boot loader menu on next boot
--boot-loader-entry=NAME Boot into a specific boot loader entry on next boot
--timestamp=FORMAT Change format of printed timestamps
If you use systemctl edit to adjust overrides, then you ll now also get the existing configuration file listed as comment, which I consider very helpful.
The MACAddressPolicy behavior with systemd naming schema v241 changed for virtual devices (I plan to write about this in a separate blog post).
There are plenty of new manual pages:
Another new unit configuration is SystemCallLog= , which supports listing the system calls to be logged. This is very useful for for auditing or temporarily when constructing system call filters.
The cgroupv2 change is also documented in the release notes, but to explicitly mention it also here, quoting from /usr/share/doc/systemd/NEWS.Debian.gz:
systemd now defaults to the unified cgroup hierarchy (i.e. cgroupv2).
This change reflects the fact that cgroups2 support has matured
substantially in both systemd and in the kernel.
All major container tools nowadays should support cgroupv2.
If you run into problems with cgroupv2, you can switch back to the previous,
hybrid setup by adding systemd.unified_cgroup_hierarchy=false to the
kernel command line.
You can read more about the benefits of cgroupv2 at https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html
Note that cgroup-tools (lssubsys + lscgroup etc) don t work in cgroup2/unified hierarchy yet (see #959022 for the details).
Configuration management
puppet s upstream doesn t provide packages for bullseye yet (see PA-3624 + MODULES-11060), and sadly neither v6 nor v7 made it into bullseye, so when using the packages from Debian you re still stuck with v5.5 (also see #950182).
ansible is also available, and while it looked like that only version 2.9.16 would make it into bullseye (see #984557 + #986213), actually version 2.10.8 made it into bullseye.
chef was removed from Debian and is not available with bullseye (due to trademark issues).
Prometheus stack
Prometheus server was updated from v2.7.1 to v2.24.1, and the prometheus service by default applies some systemd hardening now. Also all the usual exporters are still there, but bullseye also gained some new ones:
prometheus-elasticsearch-exporter (v1.1.0)
prometheus-exporter-exporter (v0.4.0-1)
prometheus-hacluster-exporter (v1.2.1-1)
prometheus-homeplug-exporter (v0.3.0-2)
prometheus-ipmi-exporter (v1.2.0)
prometheus-libvirt-exporter (v0.2.0-1)
prometheus-mqtt-exporter (v0.1.4-2)
prometheus-nginx-vts-exporter (v0.10.3)
prometheus-postfix-exporter (v0.2.0-3)
prometheus-redis-exporter (v1.16.0-1)
prometheus-smokeping-prober (v0.4.1-2)
prometheus-tplink-plug-exporter (v0.2.0)
Virtualization
docker (v20.10.5), ganeti (v3.0.1), libvirt (v7.0.0), lxc (v4.0.6), openstack, qemu/kvm (v5.2), xen (v4.14.1), are all still around, though what s new and noteworthy is that podman version 3.0.1 (tool for managing OCI containers and pods) made it into bullseye.
If you re using the docker packages from upstream, be aware that they still don t seem to understand Debian package version handling. The docker* packages will not be automatically considered for upgrade, as 5:20.10.6~3-0~debian-buster is considered newer than 5:20.10.6~3-0~debian-bullseye:
Vagrant is available in version 2.2.14, the package from upstream works perfectly fine on bullseye as well. If you re relying on VirtualBox, be aware that upstream doesn t provide packages for bullseye yet, but the package from Debian/unstable (v6.1.22 as of 2021-05-27) works fine on bullseye (VirtualBox isn t shipped with stable releases since quite some time due to lack of cooperation from upstream on security support for older releases, see #794466). If you rely on the virtualbox-guest-additions-iso and its shared folders support, you might be glad to hear that v6.1.22 made it into bullseye (see #988783), properly supporting more recent kernel versions like present in bullseye.
debuginfod
There s a new service debuginfod.debian.net (see debian-devel-announce and Debian Wiki), which makes the debugging experience way smoother. You no longer need to download the debugging Debian packages (*-dbgsym/*-dbg), but instead can fetch them on demand, by exporting the following variables (before invoking gdb or alike):
BTW: if you can t rely on debuginfod (for whatever reason), I d like to point your attention towards find-dbgsym-packages from the debian-goodies package.
Vim
Sadly Vim 8.2 once again makes another change for bad defaults (hello mouse behavior!). When incsearch is set, it also applies to :substitute. This makes it veeeeeeeeeery annoying when running something like :%s/\s\+$// to get rid of trailing whitespace characters, because if there are no matches it jumps to the beginning of the file and then back, sigh. To get the old behavior back, you can use this:
au CmdLineEnter : let s:incs = &incsearch set noincsearch
au CmdLineLeave : let &incsearch = s:incs
rsync
rsync was updated from v3.1.3 to v3.2.3. It provides various checksum enhancements (see option --checksum-choice). We got new capabilities (hardlink-specials, atimes, optional protect-args, stop-at, no crtimes) and the addition of zstd and lz4 compression algorithms. And we got new options:
--atimes: preserve access (use) times
--copy-as=USER: specify user (and optionally group) for the copy
--crtimes/-N: for preserving the file s create time
--max-alloc=SIZE: change a limit relating to memory allocation
--mkpath:create the destination s path component
--open-noatime: avoid changing the atime on opened files
--stop-after=MINS: stop rsync after MINS minutes have elapsed
--write-devices: write to devices as files (implies inplace)
OpenSSH
OpenSSH was updated from v7.9p1 to 8.4p1, so if you re interested in all the changes, check out the release notes between those version (8.0, 8.1, 8.2, 8.3 + 8.4). Let s highlight some notable new features:
It now defers creation of ~/.ssh until there s something to write (e.g. the known_hosts file), so the good old admin trick to run ssh localhost and cancel immediately to create ~/.ssh with proper permissions no longer works
v8.2 brought FIDO/U2F + FIDO2 resident keys Support
The new include sshd_config keyword allows including additional configuration files via glob(3) patterns
ssh now allows %n to be expanded in ProxyCommand strings.
The scp and sftp command-lines now accept -J option as an alias to ProxyJump.
The scp and sftp command-lines allow the -A flag to explicitly enable agent forwarding.
exim: you might be interested being aware of the allow_insecure_tainted_data option, see #987133 + #987924 (Exim 4.94 includes a new tainting-feature)
glusterfs: was updated from 5.5 to v9.1, there s no direct upgrade path (only v5 towards v8 would be supported), so online migration does not work
mariadb: no longer runs as /usr/sbin/mysqld, but as /usr/sbin/mariadbd
python: Python 2.7 (and a few core packages) is still present, but there won t be any security updates for Python 2, see wiki.debian.org/Python/2Removal
APT 2.2.0 marks the freeze of the 2.1 development series and the start of the 2.2 stable series.
Let s have a look at what changed compared to 2.2. Many of you who run Debian testing or unstable,
or Ubuntu groovy or hirsute will already have seen most of those changes.
New features
Various patterns related to dependencies, such as ?depends are now available (2.1.16)
The Protected field is now supported.
It replaces the previous Important field and is like Essential,
but only for installed packages
(some minor more differences maybe in terms of ordering the installs).
The update command has gained an --error-on=any option that makes it error out on any failure,
not just what it considers persistent ons.
The rred method can now be used as a standalone program to merge pdiff files
APT now implements phased updates.
Phasing is used in Ubuntu to slow down and control the roll out of updates in the -updates pocket,
but has previously only been available to desktop users using update-manager.
Other behavioral changes
The kernel autoremoval helper code has been rewritten from shell in C++ and now runs at run-time,
rather than at kernel install time,
in order to correctly protect the kernel that is running now,
rather than the kernel that was running when we were installing the newest one.
It also now protects only up to 3 kernels, instead of up to 4, as was originally intended,
and was the case before 1.1 series.
This avoids /boot partitions from running out of space, especially on Ubuntu which has boot
partitions sized for the original spec.
Performance improvements
The cache is now hashed using XXH3 instead of Adler32 (or CRC32c on SSE4.2 platforms)
The hash table size has been increased
Bug fixes
* wildcards work normally again (since 2.1.0)
The cache file now includes all translation files in /var/lib/apt/lists,
so multi-user systems with different locales correctly show translated descriptions now.
URLs are no longer dequoted on redirects only to be requoted again,
fixing some redirects where servers did not expect different quoting.
Immediate configuration is now best-effort, and failure is no longer fatal.
various changes to solver marking leading to different/better results in some cases (since 2.1.0)
The lower level I/O bits of the HTTP method have been rewritten to hopefully improve stability
The HTTP method no longer infinitely retries downloads on some connection errors
The pkgnames command no longer accidentally includes source packages
Various fixes from fuzzing efforts by David
Security fixes
Out-of-bound reads in ar and tar implementations (CVE-2020-3810, 2.1.2)
Integer overflows in ar and tar (CVE-2020-27350, 2.1.13)
(all of which have been backported to all stable series,
back all the way to 1.0.9.8.* series in jessie eLTS)
Incompatibilities
N/A - there were no breaking changes in apt 2.2 that we are aware of.
Deprecations
apt-key(1) is scheduled to be removed for Q2/2022, and several new warnings have been added.
apt-key was made obsolete in version 0.7.25.1, released in January 2010, by
/etc/apt/trusted.gpg.d becoming a supported place to drop additional keyring files,
and was since then only intended for deleting keys in the legacy trusted.gpg keyring.
Please manage files in trusted.gpg.d yourself;
or place them in a different location such as
/etc/apt/keyrings (or make up your own, there s no standard location)
or /usr/share/keyrings,
and use signed-by in the sources.list.d files.
The legacy trusted.gpg keyring still works, but will also stop working eventually.
Please make sure you have all your keys in trusted.gpg.d.
Warnings might be added in the upcoming months when a signature could not be verified using just trusted.gpg.d.
Future versions of APT might switch away from GPG.
As a reminder, regular expressions and wildcards other than * inside package names are deprecated (since 2.0).
They are not available anymore in apt(8), and will be removed for safety reasons in apt-get in a later release.
While it s not much, this results in redundant work. For example reading/writing package database, potentially running triggers (man-page refresh, ldconfig, etc). The internal package dependency resolution stuff isn t actually different: install will also do upgrades of needed packages, etc. Combining them should be entirely possible, but I haven t found a clean way to do this yet.
The best I ve got so far is:
apt update
apt-cache dumpavail dpkg --merge-avail -
(for i in pkg1 pkg2 pkg3; do echo "$i install") dpkg --set-selections
apt-get dselect-upgrade
This gets me the effect of running install and upgrade at the same time, but not dist-upgrade (which has slightly different resolution logic that d I d prefer to use). Also, it includes the overhead of what should be an unnecessary update of dpkg s database. Anyone know a better way to do this?
Update: Julian Andres Klode pointed out that dist-upgrade actually takes package arguments too just like install. *face palm* I didn t even try it I believed the man-page and the -h output. It works perfectly!
Yesterday I got a fresh new Pixel 4a, to replace my dying OnePlus 6.
The OnePlus had developed some faults over time: It repeatedly loses connection to the AP and the network, and it got a bunch of scratches and scuffs from falling on various surfaces without any protection over the past year.
Why get a Pixel?
Camera: OnePlus focuses on stuffing as many sensors as it can into a phone, rather than a good main sensor, resulting in pictures that are mediocre blurry messes - the dreaded oil painting effect.
Pixel have some of the best camera in the smartphone world. Sure, other hardware is far more capable, but the Pixels manage consistent results, so you need to take less pictures because they don t come out blurry half the time, and the post processing is so good that the pictures you get are just great. Other phones can shoot better pictures, sure - on a tripod.
Security updates: Pixels provide 3 years of monthly updates, with security updates being published on the 5th of each month. OnePlus only provides updates every 2 months, and then the updates they do release are almost a month out of date, not counting that they are only 1st-of-month patches, meaning vendor blob updates included in the 5th-of-month updates are even a month older. Given that all my banking runs on the phone, I don t want it to be constantly behind.
Feature updates: Of course, Pixels also get Beta Android releases and the newest Android release faster than any other phone, which is advantageous for Android development and being nerdy.
Size and weight: OnePlus phones keep getting bigger and bigger. By today s standards, the OnePlus 6 at 6.18" and 177g is a small an lightweight device. Their latest phone, the Nord, has 6.44" and weighs 184g, the OnePlus 8 comes in at 180g with a 6.55" display. This is becoming unwieldy. Eschewing glass and aluminium for plastic, the Pixel 4a comes in at 144g.
First impressions
Accessories
The Pixel 4a comes in a small box with a charger, USB-C to USB-C cable, a USB-OTG adapter, sim tray ejector. No pre-installed screen protector or bumper are provided, as we ve grown accustomed to from Chinese manufacturers like OnePlus or Xiaomi. The sim tray ejector has a circular end instead of the standard oval one - I assume so it looks like the o in Google?
Google sells you fabric cases for 45 . That seems a bit excessive, although I like that a lot of it is recycled.
Haptics
Coming from a 6.18" phablet, the Pixel 4a with its 5.81" feels tiny. In fact, it s so tiny my thumb and my index finger can touch while holding it. Cute! Bezels are a bit bigger, resulting in slightly less screen to body. The bottom chin is probably impracticably small, this was already a problem on the OnePlus 6, but this one is even smaller. Oh well, form over function.
The buttons on the side are very loud and clicky. As is the vibration motor. I wonder if this Pixel thinks it s a Model M. It just feels great.
The plastic back feels really good, it s that sort of high quality smooth plastic you used to see on those high-end Nokia devices.
The finger print reader, is super fast. Setup just takes a few seconds per finger, and it works reliably. Other phones (OnePlus 6, Mi A1/A2) take like half a minute or a minute to set up.
Software
The software - stock Android 11 - is fairly similar to OnePlus' OxygenOS. It s a clean experience, without a ton of added bloatware (even OnePlus now ships Facebook out of box, eww). It s cleaner than OxygenOS in some way - there are no duplicate photos apps, for example. On the other hand, it also has quite a bunch of Google stuff I could not care less about like YT Music. To be fair, those are minor noise once all 130 apps were transferred from the old phone.
There are various things I miss coming from OnePlus such as off-screen gestures, network transfer rate indicator in quick settings, or a circular battery icon. But the Pixel has an always on display, which is kind of nice. Most of the cool Pixel features, like call screening or live transcriptions are unfortunately not available in Germany.
The display is set to display the same amount of content as my 6.18" OnePlus 6 did, so everything is a bit tinier. This usually takes me a week or two to adjust too, and then when I look at the OnePlus again I ll be like Oh the font is huge , but right now, it feels a bit small on the Pixel.
You can configure three colour profiles for the Pixel 4a: Natural, Boosted, and Adaptive. I have mine set to adaptive. I d love to see stock Android learn what OnePlus has here: the ability to adjust the colour temperature manually, as I prefer to keep my devices closer to 5500K than 6500K, as I feel it s a bit easier on the eyes. Or well, just give me the ability to load a ICM profile (though, I d need to calibrate the screen then - work!).
Migration experience
Restoring the apps from my old phone only restore settings for a few handful out of 130, which is disappointing. I had to spent an hour or two logging in to all the other apps, and I had to fiddle far too long with openScale to get it to take its data over. It s a mystery to me why people do not allow their apps to be backed up, especially something innocent like a weight tracking app. One of my banking apps restored its logins, which I did not really like. KeePass2Android settings were restored as well, but at least the key file was not restored.
I did not opt in to restoring my device settings, as I feel that restoring device settings when changing manufactures is bound to mess up some things. For example, I remember people migrating to OnePlus phones and getting their old DND schedule without any way to change it, because OnePlus had hidden the DND stuff. I assume that s the reason some accounts, like my work GSuite account were not migrated (it said it would migrate accounts during setup).
I ve setup Bitwarden as my auto-fill service, so I could login into most of my apps and websites using the stored credentials. I found that often that did not work. Like Chrome does autofill fine once, but if I then want to autofill again, I have to kill and restart it, otherwise I don t get the auto-fill menu. Other apps did not allow any auto-fill at all, and only gave me the option to copy and paste. Yikes - auto-fill on Android still needs a lot of work.
Performance
It hangs a bit sometimes, but this was likely due to me having set 2 million iterations on my Bitwarden KDF and using Bitwarden a lot, and then opening up all 130 apps to log into them which overwhelmed the phone a bit. Apart from that, it does not feel worse than the OnePlus 6 which was to be expected, given that the benchmarks only show a slight loss in performance.
Photos do take a few seconds to process after taking them, which is annoying, but understandable given how much Google relies on computation to provide decent pictures.
Audio
The Pixel has dual speakers, with the earpiece delivering a tiny sound and the bottom firing speaker doing most of the work. Still, it s better than just having the bottom firing speaker, as it does provide a more immersive experience. Bass makes this thing vibrate a lot. It does not feel like a resonance sort of thing, but you can feel the bass in your hands. I ve never had this before, and it will take some time getting used to.
Final thoughts
This is a boring phone. There s no wow factor at all. It s neither huge, nor does it have high-res 48 or 64 MP cameras, nor does it have a ton of sensors. But everything it does, it does well. It does not pretend to be a flagship like its competition, it doesn t want to wow you, it just wants to be the perfect phone for you. The build is solid, the buttons make you think of a Model M, the camera is one of the best in any smartphone, and you of course get the latest updates before anyone else. It does not feel like a only 350 phone, but yet it is. 128GB storage is plenty, 1080p resolution is plenty, 12.2MP is you guessed it, plenty.
The same applies to the other two Pixel phones - the 4a 5G and 5. Neither are particularly exciting phones, and I personally find it hard to justify spending 620 on the Pixel 5 when the Pixel 4a does job for me, but the 4a 5G might appeal to users looking for larger phones. As to 5G, I wouldn t get much use out of it, seeing as its not available anywhere I am. Because I m on Vodafone. If you have a Telekom contract or live outside of Germany, you might just have good 5G coverage already and it might make sense to get a 5G phone rather than sticking to the budget choice.
Outlook
The big question for me is whether I ll be able to adjust to the smaller display. I now have a tablet, so I m less often using the phone (which my hands thank me for), which means that a smaller phone is probably a good call.
Oh while we re talking about calls - I only have a data-only SIM in it, so I could not test calling. I m transferring to a new phone contract this month, and I ll give it a go then. This will be the first time I get VoLTE and WiFi calling, although it is Vodafone, so quality might just be worse than Telekom on 2G, who knows. A big shoutout to congstar for letting me cancel with a simple button click, and to @vodafoneservice on twitter for quickly setting up my benefits of additional 5GB per month and 10 discount for being an existing cable customer.
I m also looking forward to playing around with the camera (especially night sight), and eSIM. And I m getting a case from China, which was handed over to the Airline on Sep 17 according to Aliexpress, so I guess it should arrive in the next weeks. Oh, and screen protector is not here yet, so I can t really judge the screen quality much, as I still have the factory protection film on it, and that s just a blurry mess - but good enough for setting it up. Please Google, pre-apply a screen protector on future phones and include a simple bumper case.
I might report back in two weeks when I have spent some more time with the device.
Sporting a beautiful 10.1 1920x1200 display, the Lenovo IdeaPad Duet
Chromebook or Duet Chromebook, is one of the latest Chromebooks released,
and one of the few slate-style tablets, and it s only about 300 EUR (300 USD).
I ve had one for about 2 weeks now, and here are my thoughts.
Build & Accessories
The tablet is a fairly Pixel-style affair, in that the back has two components,
one softer blue one housing the camera and a metal feeling gray one. Build quality
is fairly good.
The volume and power buttons are located on the right side of the tablet, and
this is one of the main issues: You end up accidentally pressing the power button
when you want to turn your volume lower, despite the power button having a different
texture.
Alongside the tablet, you also find a kickstand with a textile back, and a
keyboard, both of which attach via magnets (and pogo pins for the keyboard).
The keyboard is crammed, with punctuation keys being halfed in size, and it
feels mushed compared to my usual experiences of ThinkPads and Model Ms, but
it s on par with other Chromebooks, which is surprising, given it s a tablet
attachment.
fully assembled chromebook duet
I mostly use the Duet as a tablet, and only attach the keyboard occasionally.
Typing with the keyboard on your lap is suboptimal.
My first Duet had a few bunches of dead pixels, so I returned it, as I had
a second one I could not cancel ordered as well. Oh dear. That one was fine!
Hardware & Connectivity
The Chromebook Duet is powered by a Mediatek Helio P60T SoC, 4GB of RAM,
and a choice of 64 or 128 GB of main storage.
The tablet provides one USB-C port for charging, audio output (a 3.5mm adapter
is provided in the box), USB hub, and video output; though, sadly, the latter
is restricted to a maximum of 1080p30, or 1440x900 at 60 Hz. It can be charged
using the included 10W charger, or use up to I believe 18W from a higher powered
USB-C PD charger. I ve successfully used the Chromebook with a USB-C monitor
with attached keyboard, mouse, and DAC without any issues.
On the wireless side, the tablet provides 2x2 Wifi AC and Bluetooth 4.2. WiFi
reception seemed just fine, though I have not done any speed testing, missing
a sensible connection at the moment. I used Bluetooth to connect to my smartphone
for instant tethering, and my Sony WH1000XM2 headphones, both of which worked
without any issues.
The screen is a bright 400 nit display with excellent viewing angles,
and the speakers do a decent job, meaning you can use easily use this
for watching a movie when you re alone in a room and idling around. It has
a resolution of 1920x1200.
The device supports styluses following the USI standard. As of right now, the
only such stylus I know about is an HP one, and it costs about 70 or so.
Cameras are provided on the front and the rear, but produce terrible images.
Software: The tablet experience
The Chromebook Duet runs Chrome OS, and comes with access to Android apps
using the play store (and sideloading in dev mode) and access to full Linux
environments powered by LXD inside VMs.
The screen which has 1920x1200 is scaled to a ridiculous 1080x675 by default
which is good for being able to tap buttons and stuff, but provides next to no
content. Scaling it to 1350x844 makes things more balanced.
The Linux integration is buggy. Touches register in different places than where
they happened, and the screen is cut off in full screen extremetuxracer, making
it hard to recommend for such uses.
Android apps generally work fine. There are some issues with the back gesture
not registering, but otherwise I have not found issues I can remember.
One major drawback as a portable media consumption device is that Android apps
only work in Widevine level 3, and hence do not have access to HD content, and
the web apps of Netflix and co do not support downloading. Though one of the Duets
actually said L1 in check apps at some point (reported in issue 1090330).
It s also worth
noting that Amazon Prime Video only renders in HD, unless you change your
user agent to say you are Chrome on Windows - bad Amazon!
The tablet experience also lags in some other ways, as the palm rejection is
overly extreme, causing it to reject valid clicks close to the edge of the display
(reported in issue 1090326).
The on screen keyboard is terrible. It only does one language at a time, forcing
me to switch between German and English all the time, and does not behave as you d
expect it when editing existing words - it does not know about them and thinks
you are starting a new one. It does provide a small keyboard that you can
move around, as well as a draw your letters keyboard, which could come in
handy for stylus users, I guess. In any case, it s miles away from gboard on
Android.
Stability is a mixed bag right now. As of Chrome OS 83, sites (well only Disney+
so far ) sometimes get killed with SIGILL or SIGTRAP, and the device rebooted
on its own once or twice. Android apps that use the DRM sometimes do not start,
and the Netflix Android app sometimes reports it cannot connect to the servers.
Performance
Performance is decent to sluggish, with micro stuttering in a lot of places. The
Mediatek CPU is comparable to Intel Atoms, and with only 4GB of RAM, and an entire
Android container running, it s starting to show how weak it is.
I found that Google Docs worked perfectly fine, as did websites such as
Mastodon, Twitter, Facebook. Where the device really struggled was Reddit,
where closing or opening a post, or getting a reply box could take 5 seconds
or more. If you are looking for a Reddit browsing device, this is not for you.
Performance in Netflix was fine, and Disney+ was fairly slow but still usable.
All in all, it s acceptable, and given the price point and the build quality,
probably the compromise you d expect.
Summary
tl;dr:
good: Build quality, bright screen, low price, included accessories
The Chromebook Duet or IdeaPad Duet Chromebook is a decent tablet that is built
well above its price point. It s lackluster performance and DRM woes make it
hard to give a general recommendation, though. It s not a good laptop.
I can see this as the perfect note taking device for students, and as a
cheap tablet for couch surfing, or as your on-the-go laptop replacement,
if you need it only occasionally.
I cannot see anyone using this as their main laptop, although I guess some
people only have phones these days, so: what do I know?
I can see you getting this device if you want to tinker with Linux on ARM, as
Chromebooks are quite nice to tinker with, and a tablet is super nice.
Ubuntu Focal Fossa 20.04 was released two days ago, so I took the opportunity yesterday
and this morning to upgrade my VPS from Ubuntu 18.04 to 20.04. The VPS provides:
SMTP via Postfix
Spam filtering via rspamd
HTTP(S) via nginx and letsencrypt (certbot)
Weechat relay
OpenVPN server
Shadowsocks proxy
Unbound recursive DNS resolver, for the spam filtering
I rebooted one more time than necessary, though, as my cloud provider
Hetzner
recently started offering 2nd generation EPYC instances which I upgraded to
from my Skylake Xeon based instance. I switched from the CX21 for 5.83 /mo
to the CPX11 for 4.15 /mo. This involved a RAM downgrade - from 4GB to 2GB,
but that s fine, the maximum usage I saw was about 1.3 GB when running
dose-distcheck (running hourly); and it s good for everyone that AMD is
giving Intel some good competition, I think.
Anyway, to get back to the distribution upgrade - it was fairly boring. I
started yesterday by taking a copy of the server and launching it locally
in a lxd container, and then tested the upgrade in there; to make sure I m
prepared for the real thing :)
I got a confusing prompt from postfix as to which site I m operating
(which is a normal prompt, but I don t know why I see it on an upgrade);
and a few config files I had changed locally.
As the server is managed by ansible, I just installed the distribution
config files and dropped my changes (setting DPkg::Options "--force-confnew"; ;" in apt.conf),
and then after the upgrade, ran ansible to redeploy the changes (after checking
what changes it would do and adjusting a few things).
There are two remaining flaws:
I run rspamd from the upstream repository, and that s not built for focal
yet. So I m still using the bionic binary, and have to keep bionic s icu 60
and libhyperscan4 around for it.
This is still preventing CI of the ansible config from passing for focal,
because it won t have the needed bionic packages around.
I run weechat from the upstream repository, and apt can t tell the versions
apart. Well, it can for the repositories, because they have Size fields -
but status does not. Hence, it merges the installed version with the
first repository it sees.
What happens is that it installs from weechat.org, but then it believes the installed version
is from archive.ubuntu.com and replaces it each dist-upgrade.
I worked around it by moving the weechat.org repo to the front of sources.list,
so that the it gets merged with that instead of the archive.ubuntu.com one, as
it should be, but that s a bit ugly.
I also should start the migration to EC certificates for TLS, and 0-RTT handshakes,
so that the initial visit experience is faster. I guess I ll have to move away
from certbot for that, but I have not investigated this recently.
Solving the dependency here becomes a matter of trying to install the package
referenced by the first edge of the root, and seeing if that sticks. In this
case, that would be a . Let s assume that a failed to install, the next
step is to remove the empty node of 
For ease of visualisation, we remove a from the dependency nodes as well,
leading us to a trie of the dependencies b , b c , and b x .
Presenting the Debian dependency problem, or the positive part of it as a
trie allows us for a great visualization of the problem but it may not proof
to be an effective implementation choice.
In the real world we may actually store this as a priority queue that we
can delete from. Since we don t actually want to delete from the queue
for real, our queue items are pairs of a pointer to dependency and an
activitity level, say 
The voting period and tally of votes for the Debian Project Leader election
has just concluded and the winner is Andreas Tille, who has been elected for
the second time. Congratulations!
Out of a total of 1,030 developers, 362 voted. As usual in Debian, the voting
method used was the
I frequently see a pattern in image build/refresh scripts where a set of packages is installed, and then all packages are updated:
fully assembled chromebook duet