Debian Contributions: 2024-10 Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

rebootstrap, by Helmut Grohne After significant changes earlier this year, the state of architecture cross bootstrap is normalizing again. More and more architectures manage to complete rebootstrap testing successfully again. Here are two examples of what kind of issues the bootstrap testing identifies. At some point, `libpng1.6` would fail to cross build on `musl` architectures whereas it would succeed on other ones failing to locate `zlib`. Adding `--debug-find` to the `cmake` invocation eventually revealed that it would fail to search in `/usr/lib/<triplet>`, which is the default library path. This turned out to be a bug in cmake assuming that all linux systems use glibc. `libpng1.6` also gained a baseline violation for `powerpc` and `ppc64` by enabling the use of AltiVec there. The `newt` package would fail to cross build for many 32-bit architectures whereas it would succeed for `armel` and `armhf` due to `-Wincompatible-pointer-types`. It turns out that this flag was turned into `-Werror` and it was compiling with a warning earlier. The actual problem is a difference in signedness between `wchar_t` and `FriBidChar` (aka `uint32_t`) and actually affects native building on `i386`.

Miscellaneous contributions

Helmut sent 35 patches for cross build failures.

Stefano Rivera uploaded the Python 3.13.0 final release.

Stefano continued to rebuild Python packages with C extensions using Python 3.13, to catch compatibility issues before the 3.13-add transition starts.

Stefano uploaded new versions of a handful of Python packages, including: dh-python, objgraph, python-mitogen, python-truststore, and python-virtualenv.

Stefano packaged a new release of `mkdocs-macros-plugin`, which required packaging a new Python package for Debian, python-super-collections (now in NEW review).

Stefano helped the mini-DebConf Online Brazil get video infrastructure up and running for the event. Unfortunately, Debian s online-DebConf setup has bitrotted over the last couple of years, and it eventually required new temporary Jitsi and Jibri instances.

Colin Watson fixed a number of autopkgtest failures to get ansible back into testing.

Colin fixed an ssh client failure in certain cases when using GSS-API key exchange, and added an integration test to ensure this doesn t regress in future.

Colin worked on the Python 3.13 transition, fixing problems related to it in 15 packages. This included upstream work in a number of packages (postgresfixture, python-asyncssh, python-wadllib).

Colin upgraded 41 Python packages to new upstream versions.

Carles improved po-debconf-manager: now it can create merge requests to Salsa automatically (created 17, new batch coming this month), imported almost all the packages with debconf translation templates whose VCS is Salsa (currently 449 imported), added statistics per package and language, improved command line interface options. Performed user support fixing different issues. Also prepared an abstract for the talk at MiniDebConf Toulouse.

Santiago Ruano Rinc n continued the organization work for the DebConf 25 conference, to be held in Brest, France. Part of the work relates to the initial edits of the sponsoring brochure. Thanks to Benjamin Somers who finalized the French and English versions.

Rapha l forwarded a couple of zim and hamster bugs to the upstream developers, and tried to diagnose a delayed startup of gdm on his laptop (cf #1085633).

On behalf of the Debian Publicity Team, Anupa interviewed 7 women from the Debian community, old and new contributors. The interview was published in Bits from Debian.

derive-deftly, the template-based derive-macro facility for Rust, has been a great success. It s coming up to time to declare a stable 1.x version. If you d like to try it out, and have final comments / observations, now is the time.

Introduction to derive-deftly Have you ever wished that you could that could write a new derive macro without having to mess with procedural macros? You can! derive-deftly lets you write a #[derive] macro, using a template syntax which looks a lot like macro_rules!:

use derive_deftly:: define_derive_deftly, Deftly ;
define_derive_deftly!  
    ListVariants:
    impl $ttype  
        fn list_variants() -> Vec<&'static str>  
            vec![ $( stringify!( $vname ) , ) ]
         
     
 
#[derive(Deftly)]
#[derive_deftly(ListVariants)]
enum Enum  
    UnitVariant,
    StructVariant   a: u8, b: u16  ,
    TupleVariant(u8, u16),
 
assert_eq!(
    Enum::list_variants(),
    ["UnitVariant", "StructVariant", "TupleVariant"],
);

Status derive-deftly has a wide range of features, which can be used to easily write sophisticated and reliable derive macros. We ve been using it in Arti, the Tor Project s reimplementation of Tor in Rust, and we ve found it very useful. There is comprehensive reference documentation, and more discursive User Guide for a more gentle introduction. Naturally, everything is fully tested. History derive-deftly started out as a Tor Hackweek project. It used to be called derive-adhoc. But we renamed it because we found that many of the most interesting use cases were really not very ad-hoc at all. Over the past months we ve been ticking off our 1.0 blocker tickets. We ve taken the opportunity to improve syntax, terminology, and semantics. We hope we have now made the last breaking changes. Plans - call for review/testing In the near future, we plan to declare version 1.0. After 1.x, we intend to make breaking changes very rarely. So, right now, we d like last-minute feedback. Are there any wrinkles that need to be sorted out? Please file tickets or MRs on our gitlab. Ideally, anything which might imply breaking changes would be submitted on or before the 13th of August. In the medium to long term, we have many ideas for how to make derive-deftly even more convenient, and even more powerful. But we are going to proceed cautiously, because we don t want to introduce bad syntax or bad features, which will require difficult decisions in the future about forward compatibility.

comments

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

gt: fix manual page

Debian wiki pages: CopyrightReviewTools, MassRebuilds, PortsDocs/New, Teams/DPL/New

Issues

Conffile removal in apparmor

Features in KDE, rss-blogroll-network

Warnings in python3-azure, python3-commonmark-bkrs

Communication

Respond to queries from Debian users and contributors on IRC

Sponsors All work was done on a volunteer basis.

I last reviewed email services in 2019. That review focused a lot of attention on privacy. At the time, I selected mailbox.org as my provider, and have been using them for these 5 years since. However, both their service and their support have gone significantly downhill since, so it is time for me to look at other options. Here I am focusing strongly on email. Some of the providers mentioned here provide other services (IM, video calls, groupware, etc.), and to the extent they do, I am ignoring them.

What Matters in 2024
I want to start off by acknowledging that what you need in email probably depends on your circumstances and the country in which you live. For me, I begin by naming that the largest threat most of us face isn t from state actors but from criminals: hackers, ransomware gangs, etc. It is important to take as many steps as possible to secure one s account against that. Privacy and security are both part of the mix. I still value privacy but I am acknowledging, as Migadu does, that Email as we know it and encryption are incompatible. Although some of these services strongly protect parts of the conversation, the reality is that most people will be emailing people using plain old email services which don t. For stronger security, something like Signal would be needed. (I wrote about Signal in 2021 also.) Interestingly, OpenPGP support seems to be something of a standard feature in the providers I reviewed by this point. All or almost all of them provide integration with browser-based encryption as well as server-side encryption if you prefer that. Although mailbox.org can automatically PGP-encrypt every message that arrives in plaintext, for general use, this is unwieldy; there isn t good tooling for searching mailboxes where every message is encrypted, etc. So I never enabled that feature at Mailbox. I still value security and privacy, but a pragmatic approach addresses the most pressing threats first.

My criteria
The basic requirements for an email service include:

Ability to use my own domains

Strong privacy policy

Ability for me to use my own IMAP and SMTP clients on both desktop and mobile

It must be extremely reliable

It must not be free

It must have excellent support for those rare occasions when it is needed

Support for basic aliases

Why do I say it must not be free? Because if someone is providing a service with the quality I m talking about here, and not charging for it, it implies something is fishy: either they are unscrupulous, are financially unstable, or the product is something else like ads. I am not aware of any provider that matches the other criteria with a free account anyhow. These providers range from about $30 to $90 per year, so cheaper than a Netflix subscription. Immediately, this rules out several options:

Proton doesn t let me use my own clients on mobile (their bridge is desktop-only)

Tuta also doesn t let me use my own clients

Posteo doesn t let me use my own domain

mxroute.com lacks a strong privacy policy, and its policy has numerous causes for concern (for instance, If you repeatedly send email to invalid/unroutable recipients, they may be published on our GitHub )

I will have a bit more to say about a couple of these providers below. There are some additional criteria that are strongly desired but not absolutely required:

Ability to set individual access passwords for every device/app

Support for two-factor authentication (2FA/TFA/TOTP) for web-based access

Support for basics in filtering: ability to filter on envelope recipient (so if I get BCC d, I can still filter), and ability to execute more than one action on filter match (eg, deliver to two folders, or deliver to a folder and forward to someone else)

IMAP and SMTP don t really support 2FA, so by setting individual passwords for every device, you can at least limit the blast radius and cut off a specific device if something is (or might be) compromised.

The candidates
I considered these providers: Startmail, Mailfence, Runbox, Fastmail, Kolab, Mailbox.org, and Migadu. I ll review each, and highlight the pricing of the plan I would most likely use. Each provider offers multiple plans; some may be more expensive and some may be cheaper than the one I reviewed. I included a link to each provider s full pricing information so you can compare for your needs. I set up trials with each of these (except Mailbox.org, with which I already had a paid account). It so happend that I had actual questions for support for each one, which gave me an opportunity to see how support responded. I did not fabricate questions, and would not have contacted support if I didn t have real ones. (This means that I asked different questions of each provider, because they were the REAL questions I had.) I ll jump to the spoiler right now: I eventually chose Migadu, with Fastmail and Mailfence as close seconds. I looked for providers myself, and also solicited recommendations in a Mastodon thread.

Mailbox.org
I begin with Mailbox, as it was my top choice in 2019 and the incumbent. Until this year, I had been quite happy with it. I had cause to reach their support less than once a year on average, and each time they replied the same day or next day. Now, however, they are failing on reliability and on support. Their spam filter has become overly aggressive. It has blocked quite a bit of legitimate mail. When contacting their support about a prior issue earlier this year, they initially took 4 days to reply, and then 6 days to reply after that. Ouch. They had me disable some spam settings. It didn t really help. I continue to lose mail. I don t know how much, because they block a lot of it before it even hits the spam folder. One of my friends texted to say mail was dropping. I raised a new ticket with mailbox, which took them 5 days to reply to. Their reply was unhelpful. As the Internet is not a static system, unforeseen events can always occur. Well yes, that s true, and I get it, false positives exist with email. But this was from an ISP s mail system with an address that had been established for years, and it was part of a larger pattern of rejecting quite a bit of legit mail. And every interaction with them recently hasn t resulted in them actually doing anything to resolve anything. It s just a paragraph or two of reply that does nothing and helps nothing. When I complained that it took 5 days to reply, they said We have not been able to reply sooner as we are currently experiencing a high volume of customer enquiries. Even though their SLA for my account is a not-great 48 business hour turnaround, they still missed it and their reason is we re busy. I finally asked what RBL had caught the blocked email, since when I checked, the sender wasn t on any RBL. Mailbox s reply: they only keep their logs for 7 days, so next time I should contact them within 7 days. Which, of course, I DID; it was them that kept delaying. Ugh! It s like they ve become a cable company. Even worse is how they have been blocking mail from GrapheneOS s discussion form. See their thread about it. In short, Graphene s mail server has a clean reputation and Mailbox has no problem with it. But because one of Graphene s IPv6 webservers has an IPv6 allocation of a size Mailbox doesn t like, they drop mail. It s ridiculous, and Mailbox was dismissive of this well-known and well-regarded Open Source project. So if the likes of GrapheneOS can t get good faith effort to deliver their mail, what chance does an individual like me have? I m sorry, but I m literally paying you to deliver email for me and provide good support. If you can t do either of those, you don t get to push that problem down onto me. Hire appropriate staff. On the technical side, they support aliases, my own clients, and have a reasonable privacy policy. Their 2FA support exists for the web interface (though weirdly not the support site), though it is somewhat weird. They do not support app passwords. A somewhat unique feature is the `@secure.mailbox.org` domain. If you try to receive mail at that address, mailbox.org will block it unless it uses TLS. Same for sending. This isn t E2EE, but it does at least require things not be in plaintext for the last hop to Mailbox. Verdict: not recommended due to poor reliability and support. Mailbox.Org summary:

Website: https://mailbox.org/en/

Reliability: iffy due to over-aggressive spam filtering

Support: Poor; takes 4-6 days for a reply and replies are unhelpful

Individual access passwords: No

2FA: Yes, but with a PIN instead of a password as the other factor

Filtering: Full SIEVE feature set and GUI editor

Spam settings: greylisting on/off, reject some/all spam, etc. But they re insufficient to address Mailbox s overzealousness, which support says I cannot workaround within the interface.

Server storage location: Germany

Plan as reviewed: standard [pricing link]

Cost per year: EUR 30 (about $33)

Mail storage included: 10GB

Limits on send/receive volume: none

Aliases: 50 on your domain name, 25 on mailbox.org

Additional mailboxes: Available; each one at the same fee as the primary mailbox

Startmail
I really wanted to like Startmail. Its vault is an interesting idea and should contribute to the security and privacy of an account. They clearly care about privacy. It falls down in filtering. They have no way to filter on envelope recipient (BCC or similar). Their support confirmed this to me and that s a showstopper. Startmail support was also as slow as Mailbox, taking 5 days to respond to me. Two showstoppers right there. Verdict: Not recommended due to slow support responsiveness and weak filtering. Startmail summary:

Website: https://www.startmail.com/

Reliability: Seems to be fine

Support: Mediocre; Took 5 days for a reply, but the reply was helpful

Individual app access passwords: Yes

2FA: Yes

Filtering: Poor; cannot filter on envelope recipient, and can t build filters with multiple actions

Spam settings: None

Server storage location: The Netherlands

Plan as reviewed: Custom domain (trial was Personal), [pricing link]

Cost per year: $70

Mail storage included: 20GB

Limits on send/receive volume: none

Aliases: unlimited, with lots of features: can set expiration, etc.

Additional mailboxes: not available

Kolab
Kolab Now is mainly positioned as a full groupware service, but they do have a email-only option which I investigated. There isn t much documentation about it compared to other providers, and also not much in the way of settings. You can turn greylisting on or off. And . that s it. It has a full suite of filtering options. They set an X-Envelope-To header which you can use with the arbitrary header match to do the right thing even for BCC situations. Filters can have multiple conditions and multiple actions. It is SIEVE-based and you can download your SIEVE definitions. If you enable 2FA, you disable IMAP and SMTP; not great. Verdict: Not an impressive enough email featureset to justify going with it. Kolab Now summary:

Website: https://kolabnow.com/

Reliability: Seems to be fine

Support: Fine responsiveness (next day)

Invidiaul app passwords: no

2FA: Yes, but if you enable it, they disable IMAP and SMTP

Filtering: Excellent

Spam settings: Only greylisting on/off

Server storage location: Switzerland; they have lots of details on their setup

Plan as reviewed: Just email [pricing link]

Cost per year: CHF 60, about $66

Mail storage included: 5GB

Limitations on send/receive volume: None

Aliases: Yes. Not sure if there are limits.

Additional mailboxes: Yes if you set up a group account. Flexible pricing based on user count is not documented anywhere I could find.

Mailfence
Mailfence is another option, somewhat similar to Startmail but without the unique vault. I had some questions about filters, and support was quite responsive, responding in a couple of hours. Some of their copy on their website is a bit misleading, but support clarified when I asked them. They do not offer encryption at rest (like most of the entries here). Mailfence s filtering system is the kind I d like to see. It allows multiple conditions and multiple actions for each rule, and has some unique actions as well (notify by SMS or XMPP). Support says that Recipients matches envelope recipients. However, one ommission is that I can t match on arbitrary headers; only the canned list of headers they provide. They have only two spam settings:

spam filter on/off

whitelist

Given some recent complaints about their spam filter being overly aggressive, I find this lack of control somewhat concerning. (However, I discount complaints about people begging for more features in free accounts; free won t provide the kind of service I m looking for with any provider.) There are generally just very few settings for email as well. Verdict: Response and helpful support, filtering has the right structure but lacks arbitrary header match. Could be a good option. Mailfence summary:

Website: https://mailfence.com/

Reliability: Seems to be fine

Support: Excellent responsiveness and helpful replies (after some initial confusion about my question of greylisting)

Individual app access passwords: No. You can set a per-service password (eg, an IMAP password), but those will be shared with all devices speaking that protocol.

2FA: Yes

Filtering: Good; only misses the ability to filter on arbitrary headers

Spam settings: Very few

Server storage location: Belgium

Plan as reviewed: Entry [pricing link]

Cost per year: $42

Mail storage included: 10GB, with a maximum of 50,000 messages

Limits on send/receive volume: none

Aliases: 50. Aliases can t be deleted once created (there may be an exeption to this for aliases on your own domain rather than mailfence.com)

Additional mailboxes: Their page on this is a bit confusing, and the pricing page lacks the information promised. It looks like you can pay the same $42/year for additional mailboxes, with a limit of up to 2 additional paid mailboxes and 2 additional free mailboxes tied to the account.

Runbox
This one came recommended in a Mastodon thread. I had some questions about it, and support response was fantastic I heard from two people that were co-founders of the company! Even within hours, on a weekend. Incredible! This kind of response was only surpassed by Migadu. I initially wrote to Runbox with questions about the incoming and outgoing message limits, which I hadn t seen elsewhere, as well as the bandwidth limit. They said the bandwidth limit is no longer enforced on paid accounts. The incoming and outgoing limits are enforced, and all email (even spam) counts towards the limit. Notably the outgoing limit is per recipient, so if you send 10 messages to your 50-recipient family group, that s the limit. However, they also indicated a willingness to reset the limit if something happens. Unfortunately, hitting the limit results in a hard bounce (SMTP 5xx) rather than a temporary failure (SMTP 4xx) so it can result in lost mail. This means I d be worried about some attack or other weirdness causing me to lose mail. Their filter is a pain point. Here are the challenges:

You can t directly match on a BCC recipient. Support advised to use a headers match, which will search for something anywhere in the headers. This works and is probably good enough since this data is in the Received: headers, but it is a little more imprecise.

They only have a contains , not an equals operator. So, for instance, a pattern searching for test@example.com would also match newtest@example.com . Support advised to put the email address in angle brackets to avoid this. That will work mostly. Angle brackets aren t always required in headers.

There is no way to have multiple actions on the filter (there is just no way to file an incoming message into two folders). This was the ultimate showstopper for me.

Support advised they are planning to upgrade the filter system in the future, but these are the limitations today. Verdict: A good option if you don t need much from the filtering system. Lots of privacy emphasis. Runbox summary:

Website: https://runbox.com/

Reliability: Seems to be fine, except returning 5xx codes if per-day limits are exceeded

Support: Excellent responsiveness and replies from founders

Individual app passwords: Yes

2FA: Yes

Filtering: Poor

Spam settings: Very few

Server storage location: Norway

Plan as reviewed: Mini [pricing link]

Cost per year: $35

Mail storage included: 10GB

Limited on send/receive volume: Receive 5000 messages/day, Send 500 recipients/day

Aliases: 100 on runbox.com; unlimited on your own domain

Additional mailboxes: $15/yr each, also with 10GB non-shared storage per mailbox

Fastmail
Fastmail came recommended to me by a friend I ve known for decades. Here s the thing about Fastmail, compared to all the services listed above: It all just works. Everything. Filtering, spam prevention, it is all there, all feature-complete, and all just does the right thing as you d hope. Their filtering system has a canned dropdown for To/Cc/Bcc , it supports multiple conditions and multiple actions, and just does the right thing. (Delivering to multiple folders is a little cumbersome but possible.) It has a particularly strong feature set around administering multiple accounts, including things like whether users can prevent admins from reading their mail. The not-so-great part of the picture is around privacy. Fastmail is based in Australia, where the government has extensive power around spying on data, even to the point of forcing companies to add wiretap capabilities. Fastmail s privacy policy states user data may be held in Australia, USA, India, and Netherlands. By default, they share data with unidentified spam companies , though you can disable this in settings. On the other hand, they do make a good effort towards privacy. I contacted support with some questions and got back a helpful response in three hours. However, one of the questions was about in which countries my particular data would be stored, and the support response said they would have to get back to me on that. It s been several days and no word back. Verdict: A featureful option that just works , with a lot of features for managing family accounts and the like, but lacking in the privacy area. Fastmail summary:

Website: https://www.fastmail.com/

Reliability: Seems to be fine

Support: Good response time on most questions; dropped the ball on one tha trequired research

Individual app access passwords: Yes

2FA: Yes

Filtering: Excellent

Spam settings: Can set filter aggressiveness, decide whether to share spam data with spam-fighting companies , configure how to handle backscatter spam, and evaluate the personal learning filter.

Server storage locations: Australia, USA, India, and The Netherlands. Legal jurisdiction is Australia.

Plan as reviewed: Individual [pricing link]

Cost per year: $60

Mail storage included: 50GB

Limits on send/receive volume: 300/hour

Aliases: Unlimited from what I can see

Additional mailboxes: No; requires a different plan for that

Migadu
Migadu was a service I d never heard of, but came recommended to me on Mastodon. I listed Migadu last because it is a class of its own compared to all the other options. Every other service is basically a webmail interface with a few extra settings tacked on. Migadu has a full-featured email admin console in addition. By that I mean you can:

View usage graphs (incoming, outgoing, storage) over time

Manage DNS (if you want Migadu to run your nameservers)

Manage multiple domains, and cross-domain relationships with mailboxes

View a limited set of logs

Configure accounts, reset their passwords if needed/authorized, etc.

Configure email address rewrite rules with wildcards and so forth

Basically, if you were the sort of person that ran your own mail servers back in the day, here is Migadu giving you most of that functionality. Effectively you have a web interface to do all the useful stuff, and they handle the boring and annoying bits. This is a really attractive model. Migadu support has been fantastic. They are quick to respond, and went above and beyond. I pointed out that their X-Envelope-To header, which is needed for filtering by BCC, wasn t being added on emails I sent myself. They replied 5 hours later indicating they had added the feature to add X-Envelope-To even for internal mails! Wow! I am impressed. With Migadu, you buy a pool of resources: storage space and incoming/outgoing traffic. What you do within that pool is up to you. You can set up users ( mailboxes ), aliases, domains, whatever you like. It all just shares the pool. You can restrict users further so that an individual user has access to only a subset of the pool resources. I was initially concerned about Migadu s daily send/receive message count limits, but in visiting with support and reading the documentation, what really comes out is that Migadu is a service with a personal touch. Hitting the incoming traffic limit will cause a SMTP temporary fail (4xx) response so you won t lose legit mail and support will work with you if it s a problem for legit uses. In other words, restrictions are soft and they are interpreted reasonably. One interesting thing about Migadu is that they do not offer accounts under their domain. That is, you MUST bring your own domain. That s pretty easy and cheap, of course. It also puts you in a position of power, because it is easy to migrate email from one provider to another if you own the domain. Filtering is done via SIEVE. There is a GUI editor which lets you accomplish most things, though it has an odd blind spot where you can t file a message into multiple folders. However, you can edit a SIEVE ruleset directly and you get the full SIEVE featureset, which is extensive (and does support filing a message into multiple folders). I note that the SIEVE :envelope match doesn t work, but Migadu adds an X-Envelope-To header which is just as good. I particularly love a company that tells you all the reasons you might not want to use them. Migadu s pro/con list is an honest drawbacks list (of course, their homepage highlights all the features!). Verdict: Fantastically powerful, excellent support, and good privacy. I chose this one. Migadu summary:

Website: https://migadu.com/

Reliability: Excellent

Support: Fantastic. Good response times and they added a feature (or fixed a bug?) a few hours after I requested it.

Individual access passwords: Yes. Create identities to support them.

2FA: Yes, on both the admin interface and the webmail interface

Filtering: Excellent, based on SIEVE. GUI editor doesn t support multiple actions when filing into a folder, but full SIEVE functionality is exposed.

Spam settings:

On the domain level, filter aggressiveness, Greylisting on/off, black and white lists

On the mailbox level, filter aggressiveness, black and whitelists, action to take with spam; compatible with filters.

Server storage location: France; legal jurisdiction Switzerland

Plan as reviewed: mini [pricing link]

Cost per year: $90

Mail storage included: 30GB ( soft quota)

Limits on send/receive volume: 1000 messgaes in/day, 100 messages out/day ( soft quotas)

Aliases: Unlimited on an unlimited number of domains

Additional mailboxes: Unlimited and free; uses pooled quotas, but individual quotas can be set

Others
Here are a few others that I didn t think worthy of getting a trial:

mxroute was recommended by several. Lots of concerning things in their policy, such as:

if you repeatedly send mail to unroutable recipients, they may publish the addresses on Github

they will terminate your account if they think you are rude or want to contest a charge

they reserve the right to cancel your service at any time for any (or no) reason.

Proton keeps coming up, and I will not consider it so long as I am locked into their client on mobile.

Skiff comes up sometimes, but they were acquired by Notion.

Disroot comes up; this discussion highlights a number of reasons why I avoid them. Their Terms of Service (ToS) is inconsistent with a general-purpose email account (I guess for targeting nonprofits and activists, that could make sense). Particularly laughable is that they claim to be friends of Open Source, but then would take down your account if you upload copyrighted material. News flash: in order for an Open Source license to be meaningful, the underlying work is copyrighted. It is perfectly legal to upload copyrighted material when you wrote it or have the license to do so!

Conclusions
There are a lot of good options for email hosting today, and in particular I appreciate the excellent personal support from companies like Migadu and Runbox. Support small businesses!

Like each month, have a look at the work funded by Freexian s Debian LTS offering.

Debian LTS contributors In April, 19 contributors have been paid to work on Debian LTS, their reports are available:

Abhijith PA did 0.5h (out of 0.0h assigned and 14.0h from previous period), thus carrying over 13.5h to the next month.

Adrian Bunk did 35.75h (out of 17.25h assigned and 40.5h from previous period), thus carrying over 22.0h to the next month.

Bastien Roucari s did 25.0h (out of 25.0h assigned).

Ben Hutchings did 24.0h (out of 9.0h assigned and 15.0h from previous period).

Chris Lamb did 18.0h (out of 18.0h assigned).

Daniel Leidert did 10.0h (out of 10.0h assigned).

Emilio Pozuelo Monfort did 46.0h (out of 12.0h assigned and 34.0h from previous period).

Guilhem Moulin did 14.75h (out of 20.0h assigned), thus carrying over 5.25h to the next month.

Lee Garrett did 51.25h (out of 0.0h assigned and 60.0h from previous period), thus carrying over 8.75h to the next month.

Markus Koschany did 40.0h (out of 40.0h assigned).

Ola Lundqvist did 22.5h (out of 19.5h assigned and 4.5h from previous period), thus carrying over 1.5h to the next month.

Roberto C. S nchez did 11.0h (out of 9.25h assigned and 2.75h from previous period), thus carrying over 1.0h to the next month.

Santiago Ruano Rinc n did 20.0h (out of 20.0h assigned).

Sean Whitton did 9.5h (out of 4.5h assigned and 5.5h from previous period), thus carrying over 0.5h to the next month.

Stefano Rivera did 1.5h (out of 0.0h assigned and 10.0h from previous period), thus carrying over 8.5h to the next month.

Sylvain Beucler did 12.5h (out of 22.75h assigned and 35.0h from previous period), thus carrying over 45.25h to the next month.

Thorsten Alteholz did 14.0h (out of 14.0h assigned).

Tobias Frost did 10.0h (out of 12.0h assigned), thus carrying over 2.0h to the next month.

Utkarsh Gupta did 3.25h (out of 28.5h assigned and 29.25h from previous period), thus carrying over 54.5h to the next month.

Evolution of the situation In April, we have released 28 DLAs. During the month of April, there was one particularly notable security update made in LTS. Guilhem Moulin prepared DLA-3782-1 for util-linux (part of the set of base packages and containing a number of important system utilities) in order to address a possible information disclosure vulnerability. Additionally, several contributors prepared updates for oldstable (bullseye), stable (bookworm), and unstable (sid), including:

ruby-rack: prepared for oldstable, stable, and unstable by Adrian Bunk

wpa: prepared for oldstable, stable, and unstable by Bastien Roucari s

zookeeper: prepared for stable by Bastien Roucari s

libjson-smart: prepared for unstable by Bastien Roucari s

ansible: prepared for stable and unstable, including autopkgtest fixes to increase future supportability, by Lee Garrett

wordpress: prepared for oldstable and stable by Markus Koschany

emacs and org-mode: prepared for oldstable and stable by Sean Whitton

qtbase-opensource-src: prepared for oldstable and stable by Thorsten Alteholz

libjwt: prepared for oldstable by Thorsten Alteholz

libmicrohttpd: prepared for oldstable by Thorsten Alteholz

These fixes were in addition to corresponding updates in LTS. Another item to highlight in this month s report is an update to the distro-info-data database by Stefano Rivera. This update ensures that Debian buster systems have the latest available information concerning the end-of-life dates and other related information for all releases of Debian and Ubuntu. As announced on the debian-lts-announce mailing list, it is worth to point out that we are getting close to the end of support of Debian 10 as LTS. After June 30th, no new security updates will be made available on security.debian.org. However, Freexian and its team of paid Debian contributors will continue to maintain Debian 10 going forward for the customers of the Extended LTS offer. If you still have Debian 10 servers to keep secure, it s time to subscribe!

Thanks to our sponsors Sponsors that joined recently are in bold.

Platinum sponsors:

TOSHIBA (for 104 months)

Civil Infrastructure Platform (CIP) (for 72 months)

Gold sponsors:

Roche Diagnostics International AG (for 115 months)

Linode (for 109 months)

Babiel GmbH (for 98 months)

Plat Home (for 98 months)

CINECA (for 72 months)

University of Oxford (for 54 months)

Deveryware (for 41 months)

VyOS Inc (for 36 months)

EDF SA (for 25 months)

Silver sponsors:

Domeneshop AS (for 119 months)

Nantes M tropole (for 114 months)

Univention GmbH (for 105 months)

Universit Jean Monnet de St Etienne (for 105 months)

Ribbon Communications, Inc. (for 99 months)

Exonet B.V. (for 89 months)

Leibniz Rechenzentrum (for 83 months)

Minist re de l Europe et des Affaires trang res (for 67 months)

Cloudways by DigitalOcean (for 56 months)

Dinahosting SL (for 54 months)

Bauer Xcel Media Deutschland KG (for 48 months)

Platform.sh SAS (for 48 months)

Moxa Inc. (for 42 months)

sipgate GmbH (for 39 months)

OVH US LLC (for 37 months)

Tilburg University (for 37 months)

GSI Helmholtzzentrum f r Schwerionenforschung GmbH (for 29 months)

Soliton Systems K.K. (for 26 months)

THINline s.r.o.

Bronze sponsors:

Evolix (for 120 months)

Seznam.cz, a.s. (for 120 months)

Intevation GmbH (for 117 months)

Linuxhotel GmbH (for 117 months)

Daevel SARL (for 115 months)

Bitfolk LTD (for 114 months)

Megaspace Internet Services GmbH (for 114 months)

NUMLOG (for 114 months)

Greenbone AG (for 113 months)

WinGo AG (for 113 months)

Ecole Centrale de Nantes - LHEEA (for 109 months)

Entr ouvert (for 104 months)

Adfinis AG (for 102 months)

GNI MEDIA (for 96 months)

Laboratoire LEGI - UMR 5519 / CNRS (for 96 months)

Tesorion (for 96 months)

Bearstech (for 87 months)

LiHAS (for 87 months)

Catalyst IT Ltd (for 82 months)

Supagro (for 77 months)

Demarcq SAS (for 76 months)

Universit Grenoble Alpes (for 62 months)

TouchWeb SAS (for 54 months)

SPiN AG (for 51 months)

CoreFiling (for 46 months)

Institut des sciences cognitives Marc Jeannerod (for 41 months)

Observatoire des Sciences de l Univers de Grenoble (for 38 months)

Tem Innovations GmbH (for 33 months)

WordFinder.pro (for 32 months)

CNRS DT INSU R sif (for 31 months)

Alter Way (for 24 months)

Institut Camille Jordan (for 13 months)

Welcome to the November 2023 report from the Reproducible Builds project! In these reports we outline the most important things that we have been up to over the past month. As a rather rapid recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries (more).

Reproducible Builds Summit 2023 Between October 31st and November 2nd, we held our seventh Reproducible Builds Summit in Hamburg, Germany! Amazingly, the agenda and all notes from all sessions are all online many thanks to everyone who wrote notes from the sessions. As a followup on one idea, started at the summit, Alexander Couzens and Holger Levsen started work on a cache (or tailored front-end) for the snapshot.debian.org service. The general idea is that, when rebuilding Debian, you do not actually need the whole ~140TB of data from snapshot.debian.org; rather, only a very small subset of the packages are ever used for for building. It turns out, for `amd64`, `arm64`, `armhf`, `i386`, `ppc64el`, `riscv64` and `s390` for Debian trixie, unstable and experimental, this is only around 500GB ie. less than 1%. Although the new service not yet ready for usage, it has already provided a promising outlook in this regard. More information is available on https://rebuilder-snapshot.debian.net and we hope that this service becomes usable in the coming weeks. The adjacent picture shows a sticky note authored by Jan-Benedict Glaw at the summit in Hamburg, confirming Holger Levsen s theory that rebuilding all Debian packages needs a very small subset of packages, the text states that 69,200 packages (in Debian sid) list 24,850 packages in their `.buildinfo` files, in 8,0200 variations. This little piece of paper was the beginning of rebuilder-snapshot and is a direct outcome of the summit! The Reproducible Builds team would like to thank our event sponsors who include Mullvad VPN, openSUSE, Debian, Software Freedom Conservancy, Allotropia and Aspiration Tech.

Beyond Trusting FOSS presentation at SeaGL On November 4th, Vagrant Cascadian presented Beyond Trusting FOSS at SeaGL in Seattle, WA in the United States. Founded in 2013, SeaGL is a free, grassroots technical summit dedicated to spreading awareness and knowledge about free source software, hardware and culture. The summary of Vagrant s talk mentions that it will:
[ ] introduce the concepts of Reproducible Builds, including best practices for developing and releasing software, the tools available to help diagnose issues, and touch on progress towards solving decades-old deeply pervasive fundamental security issues Learn how to verify and demonstrate trust, rather than simply hoping everything is OK!
Germane to the contents of the talk, the slides for Vagrant s talk can be built reproducibly, resulting in a PDF with a SHA1 of `cfde2f8a0b7e6ec9b85377eeac0661d728b70f34` when built on Debian bookworm and `c21fab273232c550ce822c4b0d9988e6c49aa2c3` on Debian sid at the time of writing.

Human Factors in Software Supply Chain Security Marcel Fourn , Dominik Wermke, Sascha Fahl and Yasemin Acar have published an article in a Special Issue of the IEEE s Security & Privacy magazine. Entitled A Viewpoint on Human Factors in Software Supply Chain Security: A Research Agenda, the paper justifies the need for reproducible builds to reach developers and end-users specifically, and furthermore points out some under-researched topics that we have seen mentioned in interviews. An author pre-print of the article is available in PDF form.

Community updates On our mailing list this month:

Julien Lepiller mentioned that they were interested in translating our website, and provided offered GNU Guix s translation framework (Weblate) as a potential model to emulate.

Bernhard M. Wiedemann posted a positive LibreOffice success story documenting that, after some work:
[ ] today I hold in my hands the first two bit-identical LibreOffice rpm packages. And this is the success I wanted to share with you all today [and] it makes me feel as if we can solve anything.

kpcyrd reported on their excellent results with making `esp32c3` microcontroller firmware reproducible with Rust, repro-env and Arch Linux:
I chose the `esp32c3` [board] because it has good Rust support from the `esp-rs` project, and you can get a dev board for about 6-8 . To document my build environment I used `repro-env` together with Arch Linux because its archive is very reliable and contains all the different Rust development tools I needed.

Separate to their work on LibreOffice however, Bernhard M. Wiedemann also requested assistance with a number of packages that so far refuse to build reproducibly. He writes that the common theme around them is that they use scheme or lisp to produce binaries with a `dump` command and hopes that someone may be able to help.

Finally, Fay Stegerman regrettably reports that she will no longer be able to work on Android reproducible builds nor update the Reproducible Builds community with the status of reproducibility within F-Droid.

openSUSE updates Bernhard M. Wiedemann has created a wiki page outlining an proposal to create a general-purpose Linux distribution which consists of 100% bit-reproducible packages albeit minus the embedded signature within RPM files. It would be based on openSUSE Tumbleweed or, if available, its Slowroll-variant. In addition, Bernhard posted another monthly update for his work elsewhere in openSUSE.

Reproducibility-related changes in Debian As recently reported in the most recent Debian Developer News, Paul Gevers has integrated a package s reproducibility status into the way Debian migrates packages into the next stable release. For the `amd64`, `arm64`, `i386` and `armhf` architectures, data is collected from the Reproducible Builds testing framework is collected by this migration software even though, at the time of writing, it neither causes nor migration bonuses nor blocks migration. Indeed, the information only results are visible on Britney s excuses as well as on individual packages pages on tracker.debian.org.

Ubuntu Launchpad now supports `.buildinfo` files Back in 2017, Steve Langasek filed a bug against Ubuntu s Launchpad code hosting platform to report that `.changes` files (artifacts of building Ubuntu and Debian packages) reference `.buildinfo` files that aren t actually exposed by Launchpad itself. This was causing issues when attempting to process `.changes` files with tools such as Lintian. However, it was noticed last month that, in early August of this year, Simon Quigley had resolved this issue, and `.buildinfo` files are now available from the Launchpad system.

PHP reproducibility updates There have been two updates from the PHP programming language this month. Firstly, the widely-deployed PHPUnit framework for the PHP programming language have recently released version 10.5.0, which introduces the inclusion of a `composer.lock` file, ensuring total reproducibility of the shipped binary file. Further details and the discussion that went into their particular implementation can be found on the associated GitHub pull request. In addition, the presentation Leveraging Nix in the PHP ecosystem has been given in late October at the PHP International Conference in Munich by Pol Dellaiera. While the video replay is not yet available, the (reproducible) presentation slides and speaker notes are available.

diffoscope changes diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes, including:

Improving DOS/MBR extraction by adding support for `7z`. [ ]

Adding a missing `RequiredToolNotFound` import. [ ]

As a UI/UX improvement, try and avoid printing an extended traceback if diffoscope runs out of memory. [ ]

Mark diffoscope as stable on PyPI.org. [ ]

Uploading version `252` to Debian unstable. [ ]

Website updates A huge number of notes were added to our website that were taken at our recent Reproducible Builds Summit held between October 31st and November 2nd in Hamburg, Germany. In particular, a big thanks to Arnout Engelen, Bernhard M. Wiedemann, Daan De Meyer, Evangelos Ribeiro Tzaras, Holger Levsen and Orhun Parmaks z. In addition to this, a number of other changes were made, including:

Chris Lamb migrated the website s homepage to a hero image [ ], improved the documentation related to `SOURCE_DATE_EPOCH` and CMake [ ], added iomart (ne Bytemark) and DigitalOcean to our sponsors page [ ] and dropped an unnecessary link on some horizontal navigation buttons [ ].

Holger Levsen also made a large number of notes pages from our 2022 summit in Venice [ ], migrated the website s syntax highlighter from [Pygments]https://pygments.org/() to Rouge [ ], fixed some grammar on our donate page [ ][ ][ ] and did a lot of updates to the Hamburg Summit s general information page [ ][ ].

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann:

`amber-cli` (date-related issue)

`bin86` (FTBFS-2038)

`buildah` (timestamp)

`colord` (CPU)

`google-noto-fonts` (file modification issue)

`grub2` (directory-related metadata)

`guile-fibers` (parallelism issue)

`guile-newt` (parallelism issue)

`gutenprint` (embedded date/hostname)

`hub` (random build path)

`ipxe` (nondeterministic behavoiour)

`joker` / `joker`

`kopete` (undefined behaviour)

`kraft` (embedde hostname)

`libcamera` (signature)

`libguestfs` (embeds build host file)

`llvm` (toolchain/Rust-related issue)

`nfdump` (date-related issue)

`ovmf` (unknown cause)

`quazip` (missing fonts)

`rdflib` (nondeterminstic behaviour)

`rpm` (toolchain)

`tigervnc` (embedded an RSA signature)

`whatsie` (date-related issue)

`xen` (time-related issue)

Cathy Hu:

`policycoreutils` (sort-related issue)

Chris Lamb:

#1055919 filed against `python-ansible-pygments`.

#1055920 filed against `bidict`.

#1056117 filed against `meson`.

#1056118 filed against `radsecproxy`.

#1056119 filed against `taffybar`.

#1056398 filed against `php-doc`.

#1056571 filed against `pelican`.

#1056572 filed against `maildir-utils`.

#1056573 filed against `openmrac-data`.

#1056649 filed against `vectorscan`.

Vagrant Cascadian:

#1055969 filed against `kuttypy`.

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In October, a number of changes were made by Holger Levsen:

Debian-related changes:

Track packages marked as `Priority: important` in a new package set. [ ][ ]

Stop scheduling packages that fail to build from source in bookworm [ ] and bullseye. [ ].

Add old releases dashboard link in web navigation. [ ]

Permit re-run of the `pool_buildinfos` script to be re-run for a specific year. [ ]

Grant jbglaw access to the `osuosl4` node [ ][ ] along with lynxis [ ].

Increase RAM on the `amd64` Ionos builders from 48 GiB to 64 GiB; thanks IONOS! [ ]

Move buster to archived suites. [ ][ ]

Reduce the number of `arm64` architecture workers from 24 to 16 in order to improve stability [ ], reduce the workers for `amd64` from 32 to 28 and, for `i386`, reduce from 12 down to 8 [ ].

Show the entire build history of each Debian package. [ ]

Stop scheduling already tested package/version combinations in Debian bookworm. [ ]

Snapshot service for rebuilders

Add an HTTP-based API endpoint. [ ][ ]

Add a Gunicorn instance to serve the HTTP API. [ ]

Add an NGINX config [ ][ ][ ][ ]

System-health:

Detect failures due to HTTP 503 Service Unavailable errors. [ ]

Detect failures to update package sets. [ ]

Detect unmet dependencies. (This usually occurs with builds of Debian `live-build`.) [ ]

Misc-related changes:

do install systemd-ommd on jenkins. [ ]

fix harmless typo in squid.conf for codethink04. [ ]

fixup: reproducible Debian: add gunicorn service to serve /api for rebuilder-snapshot.d.o. [ ]

Increase codethink04 s Squid `cache_dir` size setting to 16 GiB. [ ]

Don t install `systemd-oomd` as it unfortunately kills `sshd` [ ]

Use `debootstrap` from backports when commisioning nodes. [ ]

Add the `live_build_debian_stretch_gnome`, `debsums-tests_buster` and `debsums-tests_buster` jobs to the zombie list. [ ][ ]

Run `jekyll build` with the `--watch` argument when building the Reproducible Builds website. [ ]

Misc node maintenance. [ ][ ][ ]

Other changes were made as well, however, including Mattia Rizzolo fixing `rc.local` s Bash syntax so it can actually run [ ], commenting away some file cleanup code that is (potentially) deleting too much [ ] and fixing the `html_brekages` page for Debian package builds [ ]. Finally, diagnosed and submitted a patch to add a `AddEncoding gzip .gz` line to the tests.reproducible-builds.org Apache configuration so that Gzip files aren t re-compressed as Gzip which some clients can t deal with (as well as being a waste of time). [ ]

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Mailing list: `rb-general@lists.reproducible-builds.org`

Mastodon: @reproducible_builds

Twitter: @ReproBuilds

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

tracer: mention needrestart

duck: add inactive project indicator

Debian wiki pages: AutoGeneratedFiles, CopyrightReviewTools, CornerCase, CrossGrading, DeveloperNews, Hardware/Wanted, InstallingDebianOn/SiFive/HiFiveUnmatched

FOSSjobs wiki pages: Resources

vcs-home wiki: add more dotfile repos/tools

Issues

Files in wrong dir in git-cola

Conffile removal needed in dkms

New SWH listers needed for HGKeeper, Smithy, humungus, git-arr, gitile, gotweb, ABF, Bonobo Git Server, darcsden

Review

Debian packages: sponsored purple-discord x2

Debian wiki: RecentChanges for the month

Debian BTS usertags: changes for the month

Debian screenshots:

approved c-evo-dh-gtk2 fim fish foliate mpc123 nfoview qpwgraph scite viewnior

rejected hw-probe (photos), wine64 (desktop logo), phasex (artwork), qpwgraph (about dialog), fim/fish (help output), python-lunch (full desktop), ruby-full (website), ausweisapp2 (PII), pngtools (movie poster), x11vnc (web page,) mount (systemd), blastem (photo), ca-certificates (tiny, Windows)

Administration

Debian servers: extract user data from recent wiki backups

Debian wiki: fix broken user account, approve accounts

Communication

Respond to queries from Debian users and contributors on the mailing lists and IRC.

Sponsors The SWH work was sponsored. All other work was done on a volunteer basis.

One of the downsides of running Raspberry PI OS is the fact that - not having the resources of pure Debian - upgrades are not recommended, and cross-grades (migrating between armhf and arm64) is not even mentioned. Is this really true? It is, after all a Debian-based system, so it should in theory be doable. Let s try!

Upgrading The recently announced release based on Debian Bookworm here says:
We have always said that for a major version upgrade, you should re-image your SD card and start again with a clean image. In the past, we have suggested procedures for updating an existing image to the new version, but always with the caveat that we do not recommend it, and you do this at your own risk. This time, because the changes to the underlying architecture are so significant, we are not suggesting any procedure for upgrading a Bullseye image to Bookworm; any attempt to do this will almost certainly end up with a non-booting desktop and data loss. The only way to get Bookworm is either to create an SD card using Raspberry Pi Imager, or to download and flash a Bookworm image from here with your tool of choice.
Which means, it s time to actually try it turns out it s actually trivial, if you use RPIs as headless servers. I had only three issues:

if using an `initrd`, the new initrd-building scripts/hooks are looking for some binaries in `/usr/bin`, and not in `/bin`; solution: install manually the `usrmerge` package, and then re-run `dpkg --configure -a`;

also if using an initrd, the scripts are looking for the kernel config file in `/boot/config-$(uname -r)`, and the raspberry pi kernel package doesn t provide this; workaround: `modprobe configs && zcat /proc/config.gz > /boot/config-$(uname -r)`;

and finally, on normal RPI systems, that don t use manual configurations of interfaces in `/etc/network/interface`, migrating from the previous `dhcpcd` to `NetworkManager` will break network connectivity, and require you to log in locally and fix things.

I expect most people to hit only the 3rd, and almost no-one to use initrd on raspberry pi. But, overall, aside from these two issues and a couple of cosmetic ones (`login.defs` being rewritten from scratch and showing a baffling diff, for example), it was easy. Is it worth doing? Definitely. Had no data loss, and no non-booting system.

Cross-grading (32 bit to 64 bit userland) This one is actually painful. Internet searches go from it s possible, I think to it s definitely not worth trying . Examples:

rsync files over after reinstall

just install on a new sdcard

confusion about kernel vs userland

don t do it, not recommended

stackexchange post without answers

Aside from these, there are a gazillion other posts about switching the kernel to 64 bit. And that s worth doing on its own, but it s only half the way. So, armed with two different systems - a RPI4 4GB and a RPI Zero W2 - I tried to do this. And while it can be done, it takes many hours - first system was about 6 hours, second the same, and a third RPI4 probably took ~3 hours only since I knew the problematic issues. So, what are the steps? Basically:

install `devscripts`, since you will need `dget`

enable new architecture in dpkg: `dpkg --add-architecture arm64`

switch over apt sources to include the 64 bit repos, which are different than the 32 bit ones (Raspberry PI OS did a migration here; normally a single repository has all architectures, of course)

downgrade all custom rpi packages/libraries to the standard bookworm/bullseye version, since dpkg won t usually allow a single library package to have different versions (I think it s possible to override, but I didn t bother)

install libc for the arm64 arch (this takes some effort, it s actually a set of 3-4 packages)

once the above is done, install `whiptail:amd64` and rejoice at running a 64-bit binary!

then painfully go through sets of packages and migrate the set to arm64:

sometimes this work via apt, sometimes you ll need to use `dget` and `dpkg -i`

make sure you download both the `armhf` and `arm64` versions before doing `dpkg -i`, since you ll need to rollback some installs

at one point, you ll be able to switch over `dpkg` and `apt` to arm64, at which point the default architecture flips over; from here, if you ve done it at the right moment, it becomes very easy; you ll probably need an `apt install --fix-broken`, though, at first

and then, finish by replacing all packages with `arm64` versions

and then, `dpkg --remove-architecture armhf`, reboot, and profit!

But it s tears and blood to get to that point

Pain point 1: RPI custom versions of packages Since the 32bit armhf architecture is a bit weird - having many variations - it turns out that raspberry pi OS has many packages that are very slightly tweaked to disable a compilation flag or work around build/test failures, or whatnot. Since we talk here about 64-bit capable processors, almost none of these are needed, but they do make life harder since the 64 bit version doesn t have those overrides. So what is needed would be to say downgrade all armhf packages to the version in debian upstream repo , but I couldn t find the right apt pinning incantation to do that. So what I did was to remove the 32bit repos, then use `apt-show-versions` to see which packages have versions that are no longer in any repo, then downgrade them. There s a further, minor, complication that there were about 3-4 packages with same version but different hash (!), which simply needed `apt install --reinstall`, I think.

Pain point 2: architecture independent packages There is one very big issue with dpkg in all this story, and the one that makes things very problematic: while you can have a library package installed multiple times for different architectures, as the files live in different paths, a non-library package can only be installed once (usually). For binary packages (`arch:any`), that is fine. But architecture-independent packages (`arch:all`) are problematic since usually they depend on a binary package, but they always depend on the default architecture version! Hrmm, and I just realise I don t have logs from this, so I m only ~80% confident. But basically:

`vim-solarized` (`arch:all`) depends on `vim` (`arch:any`)

if you replace vim armhf with vim arm64, this will break `vim-solarized`, until the default architecture becomes `arm64`

So you need to keep track of which packages apt will de-install, for later re-installation. It is possible that `Multi-Arch: foreign` solves this, per the debian wiki which says:
Note that even though `Architecture: all` and `Multi-Arch: foreign` may look like similar concepts, they are not. The former means that the same binary package can be installed on different architectures. Yet, after installation such packages are treated as if they were native architecture (by definition the architecture of the dpkg package) packages. Thus Architecture: all packages cannot satisfy dependencies from other architectures without being marked Multi-Arch foreign.
It also has warnings about how to properly use this. But, in general, not many packages have it, so it is a problem.

Pain point 3: remove + install vs overwrite It seems that depending on how the solver computes a solution, when migrating a package from 32 to 64 bit, it can choose either to:

overwrite in place the package (akin to dpkg -i)
remove + install later

The former is OK, the later is not. Or, actually, it might be that apt never can do this, for example (edited for brevity):

# apt install systemd:arm64 --no-install-recommends
The following packages will be REMOVED:
  systemd
The following NEW packages will be installed:
  systemd:arm64
0 upgraded, 1 newly installed, 1 to remove and 35 not upgraded.
Do you want to continue? [Y/n] y
dpkg: systemd: dependency problems, but removing anyway as you requested:
 systemd-sysv depends on systemd.
Removing systemd (247.3-7+deb11u2) ...
systemd is the active init system, please switch to another before removing systemd.
dpkg: error processing package systemd (--remove):
 installed systemd package pre-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
 systemd
Processing was halted because there were too many errors.

But at the same time, overwrite in place is all good - via dpkg -i from /var/cache/apt/archives. In this case it manifested via a prerm script, in other cases is manifests via dependencies that are no longer satisfied for packages that can t be removed, etc. etc. So you will have to resort to dpkg -i a lot.

Pain point 4: lib- packages that are not lib During the whole process, it is very tempting to just go ahead and install the corresponding arm64 package for all armhf lib package, in one go, since these can coexist. Well, this simple plan is complicated by the fact that some packages are named libfoo-bar, but are actual holding (e.g.) the bar binary for the libfoo package. Examples:

libmagic-mgc contains /usr/lib/file/magic.mgc, which conflicts between the 32 and 64 bit versions; of course, it s the exact same file, so this should be an arch:all package, but
libpam-modules-bin and liblockfile-bin actually contain binaries (per the -bin suffix)

It s possible to work around all this, but it changes a 1 minute:

# apt install $(dpkg -i   grep ^ii   awk ' print $2 ' grep :amrhf sed -e 's/:armhf/:arm64')

into a 10-20 minutes fight with packages (like most other steps).

Is it worth doing? Compared to the simple bullseye bookworm upgrade, I m not sure about this. The result? Yes, definitely, the system feels - weirdly - much more responsive, logged in over SSH. I guess the arm64 base architecture has some more efficient ops than the lowest denominator armhf , so to say (e.g. there was in the 32 bit version some rpi-custom package with string ops), and thus migrating to 64 bit makes more things faster , but this is subjective so it might be actually not true. But from the point of view of the effort? Unless you like to play with dpkg and apt, and understand how these work and break, I d rather say, migrate to ansible and automate the deployment. It s doable, sure, and by the third system, I got this nailed down pretty well, but it was a lot of time spent. The good aspect is that I did 3 migrations:

rpi zero w2: bullseye 32 bit to 64 bit, then bullseye to bookworm

rpi 4: bullseye to bookworm, then bookworm 32bit to 64 bit

same, again, for a more important system

And all three worked well and no data loss. But I m really glad I have this behind me, I probably wouldn t do a fourth system, even if forced And now, waiting for the RPI 5 to be available See you!

Review: The Cassini Division, by Ken MacLeod

Series:	Fall Revolution #3
Publisher:	Tor
Copyright:	1998
Printing:	August 2000
ISBN:	0-8125-6858-3
Format:	Mass market
Pages:	305

The Cassini Division is the third book in the Fall Revolution series and a fairly direct sequel (albeit with different protagonists) to The Stone Canal. This is not a good place to start the series. It's impossible to talk about the plot of this book without discussing the future history of this series, which arguably includes some spoilers for The Star Fraction and The Stone Canal. I don't think the direction of history matters that much in enjoying the previous books, but read the first two books of the series before this review if you want to avoid all spoilers. When the Outwarders uploaded themselves and went fast, they did a lot of strange things: an interstellar probe contrary to all known laws of physics, the disassembly of Ganymede, and the Malley Mile, which plays a significant role in The Stone Canal. They also crashed the Earth. This was not entirely their fault. There were a lot of politics, religious fundamentalism, and plagues in play as well. But the storm of viruses broadcast from their transformed Jupiter shut down essentially all computing equipment on Earth, which set off much of the chaos. The results were catastrophic, and also politically transformative. Now, the Solar Union is a nearly unified anarchosocialist society, with only scattered enclaves of non-cooperators left outside that structure. Ellen May Ngewthu is a leader of the Cassini Division, the bulwark that stands between humans and the Outwarders. The Division ruthlessly destroys any remnant or probe that dares rise out of Jupiter's atmosphere, ensuring that the Outwarders, whatever they have become after untold generations of fast evolution, stay isolated to the one planet they have absorbed. The Division is very good at what they do. But there is a potential gap in that line of defense: there are fast folk in storage at the other end of the Malley Mile, on New Mars, and who knows what the deranged capitalists there will do or what forces they might unleash. The one person who knows a path through the Malley Mile isn't talking, so Ellen goes in search of the next best thing: the non-cooperator scientist Isambard Kingdom Malley. I am now thoroughly annoyed at how politics are handled in this series, and much less confused by the frequency with which MacLeod won Prometheus Awards from the Libertarian Futurist Society. Some of this is my own fault for having too high of hopes for political SF, but nothing in this series so far has convinced me that MacLeod is seriously engaging with political systems. Instead, the world-building to date makes the classic libertarian mistake of thinking societies will happily abandon stability and predictability in favor of their strange definition of freedom. The Solar Union is based on what Ellen calls the true knowledge, which is worth quoting in full so that you know what kind of politics we're talking about:

Life is a process of breaking down and using other matter, and if need be, other life. Therefore, life is aggression, and successful life is successful aggression. Life is the scum of matter, and people are the scum of life. There is nothing but matter, forces, space and time, which together make power. Nothing matters, except what matters to you. Might makes right, and power makes freedom. You are free to do whatever is in your power, and if you want to survive and thrive you had better do whatever is in your interests. If your interests conflict with those of others, let the others pit their power against yours, everyone for theirselves. If your interests coincide with those of others, let them work together with you, and against the rest. We are what we eat, and we eat everything. All that you really value, and the goodness and truth and beauty of life, have their roots in this apparently barren soil. This is the true knowledge. We had founded our idealism on the most nihilistic implications of science, our socialism on crass self-interest, our peace on our capacity for mutual destruction, and our liberty on determinism. We had replaced morality with convention, bravery with safety, frugality with plenty, philosophy with science, stoicism with anaesthetics and piety with immortality. The universal acid of the true knowledge had burned away a world of words, and exposed a universe of things. Things we could use.

This is certainly something that some people will believe, particularly cynical college students who love political theory, feeling smarter than other people, and calling their pet theories things like "the true knowledge." It is not even remotely believable as the governing philosophy of a solar confederation. The point of government for the average person in human society is to create and enforce predictable mutual rules that one can use as a basis for planning and habits, allowing you to not think about politics all the time. People who adore thinking about politics have great difficulty understanding how important it is to everyone else to have ignorable government. Constantly testing your power against other coalitions is a sport, not a governing philosophy. Given the implication that this testing is through violence or the threat of violence, it beggars belief that any large number of people would tolerate that type of instability for an extended period of time. Ellen is fully committed to the true knowledge. MacLeod likely is not; I don't think this represents the philosophy of the author. But the primary political conflict in this novel famous for being political science fiction is between the above variation of anarchy and an anarchocapitalist society, neither of which are believable as stable political systems for large numbers of people. This is a bit like seeking out a series because you were told it was about a great clash of European monarchies and discovering it was about a fight between Liberland and Sealand. It becomes hard to take the rest of the book seriously. I do realize that one point of political science fiction is to play with strange political ideas, similar to how science fiction plays with often-implausible science ideas. But those ideas need some contact with human nature. If you're going to tell me that the key to clawing society back from a world-wide catastrophic descent into chaos is to discard literally every social system used to create predictability and order, you had better be describing aliens, because that's not how humans work. The rest of the book is better. I am untangling a lot of backstory for the above synopsis, which in the book comes in dribs and drabs, but piecing that together is good fun. The plot is far more straightforward than the previous two books in the series: there is a clear enemy, a clear goal, and Ellen goes from point A to point B in a comprehensible way with enough twists to keep it interesting. The core moral conflict of the book is that Ellen is an anti-AI fanatic to the point that she considers anyone other than non-uploaded humans to be an existential threat. MacLeod gives the reader both reasons to believe Ellen is right and reasons to believe she's wrong, which maintains an interesting moral tension. One thing that MacLeod is very good at is what Bob Shaw called "wee thinky bits." I think my favorite in this book is the computer technology used by the Cassini Division, who have spent a century in close combat with inimical AI capable of infecting any digital computer system with tailored viruses. As a result, their computers are mechanical non-Von-Neumann machines, but mechanical with all the technology of a highly-advanced 24th century civilization with nanometer-scale manufacturing technology. It's a great mental image and a lot of fun to think about. This is the only science fiction novel that I can think of that has a hard-takeoff singularity that nonetheless is successfully resisted and fought to a stand-still by unmodified humanity. Most writers who were interested in the singularity idea treated it as either a near-total transformation leaving only remnants or as something that had to be stopped before it started. MacLeod realizes that there's no reason to believe a post-singularity form of life would be either uniform in intent or free from its own baffling sudden collapses and reversals, which can be exploited by humans. It makes for a much better story. The sociology of this book is difficult to swallow, but the characterization is significantly better than the previous books of the series and the plot is much tighter. I was too annoyed by the political science to fully enjoy it, but that may be partly the fault of my expectations coming in. If you like chewy, idea-filled science fiction with a lot of unexplained world-building that you have to puzzle out as you go, you may enjoy this, although unfortunately I think you need to read at least The Stone Canal first. The ending was a bit unsatisfying, but even that includes some neat science fiction ideas. Followed by The Sky Road, although I understand it is not a straightforward sequel. Rating: 6 out of 10

I recently bought an HP Chromebook from Abhas who had already flashed coreboot in it. I ran a fresh installation of Debian 12 (Bookworm) on it with KDE Plasma. Right after installation, the Wi-Fi and bluetooth were working, but I was facing two issues:

Playing a music file or any audio file does not give any audio.
Keyboard buttons like the ones for brightness and audio adjustment were not working (alphabet keys were working).

So I asked my friend Alper for help on fixing the same as he has some experience with Chromebooks. Thanks a lot Alper for the help. I am documenting our steps here for helping others who are facing this issue. Note: This works in X11. For wayland, the steps might differ. To set system-wide keyboard configuration on Debian systems:

$ sudo dpkg-reconfigure keyboard-configuration

Choose Chromebook as the Keyboard Model . Each DE should default to the system configuration, but might need its own configuration which would similarly be available in their GUI tools. But you can check and set it manually from the command line, for example as in this thread. To check the keyboard model Xorg-based DEs:

$ setxkbmap -print -query   grep model:
model:  pc104

To change it temporarily, until a reboot:

$ setxkbmap -model chromebook

If it s not there in KDE settings that would be a bug, To change it persistently for KDE:

$ cat >>.config/kxkbrc <<EOF
[Layout]
Model=chromebook
EOF

This thread was helpful.

Playing a music file or any audio file does not give any audio.
Keyboard buttons like the ones for brightness and audio adjustment were not working (alphabet keys were working).

Fixing audio I ran the script mentioned here and that fixed the audio. The instructions from that link are:

git clone https://github.com/WeirdTreeThing/chromebook-linux-audio
cd chromebook-linux-audio
./setup-audio

Fixing keyboard I asked my friend Alper for help on fixing the keyboard as he has some experience with Chromebooks. Thanks a lot Alper for the help. I am documenting our steps here for helping others who are facing this issue. Note: This works in X11. For wayland, the steps might differ. To set system-wide keyboard configuration on Debian systems:

$ sudo dpkg-reconfigure keyboard-configuration

$ setxkbmap -print -query   grep model:
model:  pc104

To change it temporarily, until a reboot:

$ setxkbmap -model chromebook

If it s not there in KDE settings that would be a bug, To change it persistently for KDE:

$ cat >>.config/kxkbrc <<EOF
[Layout]
Model=chromebook
EOF

This thread was helpful.

I m one of the few folks left who run and maintain mail servers. Recently I had major troubles receiving mails from the mail servers used by a bank, and when asking my favourite search engine, I m clearly not the only one who ran into such an issue. Actually, I should have checked off the issue and not become a customer at that bank, but the tech nerd in me couldn t resist getting to the bottom of the problem. Since I got it working and this might be useful for others, here we are. :) I was trying to get an online banking account set up, but the corresponding account creation mail didn t arrive me, at all. Looking at my mail server logs, my postfix mail server didn t accept the mail due to:

postfix/smtpd[3319640]: warning: TLS library problem: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher:../ssl/statem/statem_srvr.c:2283:
postfix/smtpd[3319640]: lost connection after STARTTLS from mx01.arz.at[193.110.182.61]

Huh, what s going on here?! Let s increase the TLS loglevel (setting smtpd_tls_loglevel = 2) and retry. But how can I retry receiving yet another mail? Luckily, on the registration website of the bank there was a URL available, that let me request a one-time password. This triggered another mail, so I did that and managed to grab this in the logs:

postfix/smtpd[3320018]: initializing the server-side TLS engine
postfix/tlsmgr[3320020]: open smtpd TLS cache btree:/var/lib/postfix/smtpd_scache
postfix/tlsmgr[3320020]: tlsmgr_cache_run_event: start TLS smtpd session cache cleanup
postfix/smtpd[3320018]: connect from mx01.arz.at[193.110.182.61]
postfix/smtpd[3320018]: setting up TLS connection from mx01.arz.at[193.110.182.61]
postfix/smtpd[3320018]: mx01.arz.at[193.110.182.61]: TLS cipher list "aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH"
postfix/smtpd[3320018]: SSL_accept:before SSL initialization
postfix/smtpd[3320018]: SSL_accept:before SSL initialization
postfix/smtpd[3320018]: SSL3 alert write:fatal:handshake failure
postfix/smtpd[3320018]: SSL_accept:error in error
postfix/smtpd[3320018]: SSL_accept error from mx01.arz.at[193.110.182.61]: -1
postfix/smtpd[3320018]: warning: TLS library problem: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher:../ssl/statem/statem_srvr.c:2283:
postfix/smtpd[3320018]: lost connection after STARTTLS from mx01.arz.at[193.110.182.61]
postfix/smtpd[3320018]: disconnect from mx01.arz.at[193.110.182.61] ehlo=1 starttls=0/1 commands=1/2
postfix/smtpd[3320018]: connect from mx01.arz.at[193.110.182.61]
postfix/smtpd[3320018]: disconnect from mx01.arz.at[193.110.182.61] ehlo=1 quit=1 commands=2

Ok, so this TLS cipher list aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH looked like the tls_medium_cipherlist setting in postfix, but which ciphers might we expect? Let s see what their SMTP server would speak to us:

% testssl --cipher-per-proto -t=smtp mx01.arz.at:25
[...]
Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
SSLv3
TLS 1
TLS 1.1
TLS 1.2
 xc030   ECDHE-RSA-AES256-GCM-SHA384       ECDH 256   AESGCM      256      TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
 xc028   ECDHE-RSA-AES256-SHA384           ECDH 256   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
 xc014   ECDHE-RSA-AES256-SHA              ECDH 256   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
 x9d     AES256-GCM-SHA384                 RSA        AESGCM      256      TLS_RSA_WITH_AES_256_GCM_SHA384
 x3d     AES256-SHA256                     RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA256
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA
 xc02f   ECDHE-RSA-AES128-GCM-SHA256       ECDH 256   AESGCM      128      TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
 xc027   ECDHE-RSA-AES128-SHA256           ECDH 256   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
 xc013   ECDHE-RSA-AES128-SHA              ECDH 256   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
 x9c     AES128-GCM-SHA256                 RSA        AESGCM      128      TLS_RSA_WITH_AES_128_GCM_SHA256
 x3c     AES128-SHA256                     RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA256
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA
TLS 1.3

Looks like a very small subset of ciphers, and they don t seem to be talking TLS v1.3 at all? Not great. :( A nice web service to verify the situation from another point of view is checktls, which also confirmed this:

[000.705] 	<-- 	220 2.0.0 Ready to start TLS
[000.705] 		STARTTLS command works on this server
[001.260] 		Connection converted to SSL
		SSLVersion in use: TLSv1_2
		Cipher in use: ECDHE-RSA-AES256-GCM-SHA384
		Perfect Forward Secrecy: yes
		Session Algorithm in use: Curve P-256 DHE(256 bits)
		Certificate #1 of 3 (sent by MX):
		Cert VALIDATED: ok
		Cert Hostname VERIFIED (mx01.arz.at = *.arz.at   DNS:*.arz.at   DNS:arz.at)
[...]
[001.517] 		TLS successfully started on this server

I got distracted by some other work, and when coming back to this problem, the one-time password procedure no longer worked, as the password reset URL was no longer valid. :( I managed to find the underlying URL, and with some web developer tools tinkering I could still use the website to let me trigger sending further one-time password mails, phew. Let s continue, so my mail server was running Debian/bullseye with postfix v3.5.18-0+deb11u1 and openssl v1.1.1n-0+deb11u5, let s see what it offers:

% testssl --cipher-per-proto -t=smtp mail.example.com:25
[...]
Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
SSLv3
TLS 1
 xc00a   ECDHE-ECDSA-AES256-SHA            ECDH 253   AES         256      TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
 xc019   AECDH-AES256-SHA                  ECDH 253   AES         256      TLS_ECDH_anon_WITH_AES_256_CBC_SHA
 x3a     ADH-AES256-SHA                    DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA
 x89     ADH-CAMELLIA256-SHA               DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA
 xc009   ECDHE-ECDSA-AES128-SHA            ECDH 253   AES         128      TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
 xc018   AECDH-AES128-SHA                  ECDH 253   AES         128      TLS_ECDH_anon_WITH_AES_128_CBC_SHA
 x34     ADH-AES128-SHA                    DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA
 x9b     ADH-SEED-SHA                      DH 2048    SEED        128      TLS_DH_anon_WITH_SEED_CBC_SHA
 x46     ADH-CAMELLIA128-SHA               DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA
TLS 1.1
 xc00a   ECDHE-ECDSA-AES256-SHA            ECDH 253   AES         256      TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
 xc019   AECDH-AES256-SHA                  ECDH 253   AES         256      TLS_ECDH_anon_WITH_AES_256_CBC_SHA
 x3a     ADH-AES256-SHA                    DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA
 x89     ADH-CAMELLIA256-SHA               DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA
 xc009   ECDHE-ECDSA-AES128-SHA            ECDH 253   AES         128      TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
 xc018   AECDH-AES128-SHA                  ECDH 253   AES         128      TLS_ECDH_anon_WITH_AES_128_CBC_SHA
 x34     ADH-AES128-SHA                    DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA
 x9b     ADH-SEED-SHA                      DH 2048    SEED        128      TLS_DH_anon_WITH_SEED_CBC_SHA
 x46     ADH-CAMELLIA128-SHA               DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA
TLS 1.2
 xc02c   ECDHE-ECDSA-AES256-GCM-SHA384     ECDH 253   AESGCM      256      TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
 xc024   ECDHE-ECDSA-AES256-SHA384         ECDH 253   AES         256      TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
 xc00a   ECDHE-ECDSA-AES256-SHA            ECDH 253   AES         256      TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
 xcca9   ECDHE-ECDSA-CHACHA20-POLY1305     ECDH 253   ChaCha20    256      TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
 xc0af   ECDHE-ECDSA-AES256-CCM8           ECDH 253   AESCCM8     256      TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8
 xc0ad   ECDHE-ECDSA-AES256-CCM            ECDH 253   AESCCM      256      TLS_ECDHE_ECDSA_WITH_AES_256_CCM
 xc073   ECDHE-ECDSA-CAMELLIA256-SHA384    ECDH 253   Camellia    256      TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_CBC_SHA384
 xc019   AECDH-AES256-SHA                  ECDH 253   AES         256      TLS_ECDH_anon_WITH_AES_256_CBC_SHA
 xa7     ADH-AES256-GCM-SHA384             DH 2048    AESGCM      256      TLS_DH_anon_WITH_AES_256_GCM_SHA384
 x6d     ADH-AES256-SHA256                 DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA256
 x3a     ADH-AES256-SHA                    DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA
 xc5     ADH-CAMELLIA256-SHA256            DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA256
 x89     ADH-CAMELLIA256-SHA               DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA
 xc05d   ECDHE-ECDSA-ARIA256-GCM-SHA384    ECDH 253   ARIAGCM     256      TLS_ECDHE_ECDSA_WITH_ARIA_256_GCM_SHA384
 xc02b   ECDHE-ECDSA-AES128-GCM-SHA256     ECDH 253   AESGCM      128      TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
 xc023   ECDHE-ECDSA-AES128-SHA256         ECDH 253   AES         128      TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
 xc009   ECDHE-ECDSA-AES128-SHA            ECDH 253   AES         128      TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
 xc0ae   ECDHE-ECDSA-AES128-CCM8           ECDH 253   AESCCM8     128      TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8
 xc0ac   ECDHE-ECDSA-AES128-CCM            ECDH 253   AESCCM      128      TLS_ECDHE_ECDSA_WITH_AES_128_CCM
 xc072   ECDHE-ECDSA-CAMELLIA128-SHA256    ECDH 253   Camellia    128      TLS_ECDHE_ECDSA_WITH_CAMELLIA_128_CBC_SHA256
 xc018   AECDH-AES128-SHA                  ECDH 253   AES         128      TLS_ECDH_anon_WITH_AES_128_CBC_SHA
 xa6     ADH-AES128-GCM-SHA256             DH 2048    AESGCM      128      TLS_DH_anon_WITH_AES_128_GCM_SHA256
 x6c     ADH-AES128-SHA256                 DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA256
 x34     ADH-AES128-SHA                    DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA
 xbf     ADH-CAMELLIA128-SHA256            DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA256
 x9b     ADH-SEED-SHA                      DH 2048    SEED        128      TLS_DH_anon_WITH_SEED_CBC_SHA
 x46     ADH-CAMELLIA128-SHA               DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA
 xc05c   ECDHE-ECDSA-ARIA128-GCM-SHA256    ECDH 253   ARIAGCM     128      TLS_ECDHE_ECDSA_WITH_ARIA_128_GCM_SHA256
TLS 1.3
 x1302   TLS_AES_256_GCM_SHA384            ECDH 253   AESGCM      256      TLS_AES_256_GCM_SHA384
 x1303   TLS_CHACHA20_POLY1305_SHA256      ECDH 253   ChaCha20    256      TLS_CHACHA20_POLY1305_SHA256
 x1301   TLS_AES_128_GCM_SHA256            ECDH 253   AESGCM      128      TLS_AES_128_GCM_SHA256

Not so bad, but sadly no overlap with any of the ciphers that mx01.arz.at offers. What about disabling STARTTLS for the mx01.arz.at (+ mx02.arz.at being another one used by the relevant domain) mail servers when talking to mine? Let s try that:

% sudo postconf -nf smtpd_discard_ehlo_keyword_address_maps
smtpd_discard_ehlo_keyword_address_maps =
    hash:/etc/postfix/smtpd_discard_ehlo_keywords
% cat /etc/postfix/smtpd_discard_ehlo_keywords
# *disable* starttls for mx01.arz.at / mx02.arz.at:
193.110.182.61 starttls
193.110.182.62 starttls

But the remote mail server doesn t seem to send mails without TLS:

postfix/smtpd[4151799]: connect from mx01.arz.at[193.110.182.61]
postfix/smtpd[4151799]: discarding EHLO keywords: STARTTLS
postfix/smtpd[4151799]: disconnect from mx01.arz.at[193.110.182.61] ehlo=1 quit=1 commands=2

Let s verify this further, but without fiddling with the main mail server too much. We can add a dedicated service to postfix (see serverfault), and run it in verbose mode, to get more detailled logging:

% sudo postconf -Mf
[...]
10025      inet  n       -       -       -       -       smtpd
    -o syslog_name=postfix/smtpd/badstarttls
    -o smtpd_tls_security_level=none
    -o smtpd_helo_required=yes
    -o smtpd_helo_restrictions=pcre:/etc/postfix/helo_badstarttls_allow,reject
    -v
[...]
% cat /etc/postfix/helo_badstarttls_allow
/mx01.arz.at/ OK
/mx02.arz.at/ OK
/193.110.182.61/ OK
/193.110.182.62/ OK

We redirect the traffic from mx01.arz.at + mx02.arz.at towards our new postfix service, listening on port 10025:

% sudo iptables -t nat -A PREROUTING -p tcp -s 193.110.182.61 --dport 25 -j REDIRECT --to-port 10025
% sudo iptables -t nat -A PREROUTING -p tcp -s 193.110.182.62 --dport 25 -j REDIRECT --to-port 10025

With this setup we get very detailed logging, and it seems to confirm our suspicion that the mail server doesn t want to talk unencrypted with us:

[...]
postfix/smtpd/badstarttls/smtpd[3491900]: connect from mx01.arz.at[193.110.182.61]
[...]
postfix/smtpd/badstarttls/smtpd[3491901]: disconnect from mx01.arz.at[193.110.182.61] ehlo=1 quit=1 commands=2
postfix/smtpd/badstarttls/smtpd[3491901]: master_notify: status 1
postfix/smtpd/badstarttls/smtpd[3491901]: connection closed
[...]

Let s step back and revert those changes, back to our original postfix setup. Might the problem be related to our Let s Encrypt certificate? Let s see what we have:

% echo QUIT   openssl s_client -connect mail.example.com:25 -starttls
[...]
issuer=C = US, O = Let's Encrypt, CN = R3
---
No client certificate CA names sent
Peer signing digest: SHA384
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4455 bytes and written 427 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 384 bit
[...]

We have an ECDSA based certificate, what about switching to RSA instead? Thanks to the wonderful dehydrated, this is as easy as:

% echo KEY_ALGO=rsa > certs/mail.example.com/config
% ./dehydrated -c --domain mail.example.com --force
% sudo systemctl reload postfix

With switching to RSA type key we get:

% echo QUIT   openssl s_client -connect mail.example.com:25 -starttls smtp
CONNECTED(00000003)
[...]
issuer=C = US, O = Let's Encrypt, CN = R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5295 bytes and written 427 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit

Which ciphers do we offer now? Let s check:

% testssl --cipher-per-proto -t=smtp mail.example.com:25
[...]
Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
SSLv3
TLS 1
 xc014   ECDHE-RSA-AES256-SHA              ECDH 253   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
 x39     DHE-RSA-AES256-SHA                DH 2048    AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA
 x88     DHE-RSA-CAMELLIA256-SHA           DH 2048    Camellia    256      TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA
 xc019   AECDH-AES256-SHA                  ECDH 253   AES         256      TLS_ECDH_anon_WITH_AES_256_CBC_SHA
 x3a     ADH-AES256-SHA                    DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA
 x89     ADH-CAMELLIA256-SHA               DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA
 x84     CAMELLIA256-SHA                   RSA        Camellia    256      TLS_RSA_WITH_CAMELLIA_256_CBC_SHA
 xc013   ECDHE-RSA-AES128-SHA              ECDH 253   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
 x33     DHE-RSA-AES128-SHA                DH 2048    AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA
 x9a     DHE-RSA-SEED-SHA                  DH 2048    SEED        128      TLS_DHE_RSA_WITH_SEED_CBC_SHA
 x45     DHE-RSA-CAMELLIA128-SHA           DH 2048    Camellia    128      TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA
 xc018   AECDH-AES128-SHA                  ECDH 253   AES         128      TLS_ECDH_anon_WITH_AES_128_CBC_SHA
 x34     ADH-AES128-SHA                    DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA
 x9b     ADH-SEED-SHA                      DH 2048    SEED        128      TLS_DH_anon_WITH_SEED_CBC_SHA
 x46     ADH-CAMELLIA128-SHA               DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA
 x96     SEED-SHA                          RSA        SEED        128      TLS_RSA_WITH_SEED_CBC_SHA
 x41     CAMELLIA128-SHA                   RSA        Camellia    128      TLS_RSA_WITH_CAMELLIA_128_CBC_SHA
TLS 1.1
 xc014   ECDHE-RSA-AES256-SHA              ECDH 253   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
 x39     DHE-RSA-AES256-SHA                DH 2048    AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA
 x88     DHE-RSA-CAMELLIA256-SHA           DH 2048    Camellia    256      TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA
 xc019   AECDH-AES256-SHA                  ECDH 253   AES         256      TLS_ECDH_anon_WITH_AES_256_CBC_SHA
 x3a     ADH-AES256-SHA                    DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA
 x89     ADH-CAMELLIA256-SHA               DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA
 x84     CAMELLIA256-SHA                   RSA        Camellia    256      TLS_RSA_WITH_CAMELLIA_256_CBC_SHA
 xc013   ECDHE-RSA-AES128-SHA              ECDH 253   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
 x33     DHE-RSA-AES128-SHA                DH 2048    AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA
 x9a     DHE-RSA-SEED-SHA                  DH 2048    SEED        128      TLS_DHE_RSA_WITH_SEED_CBC_SHA
 x45     DHE-RSA-CAMELLIA128-SHA           DH 2048    Camellia    128      TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA
 xc018   AECDH-AES128-SHA                  ECDH 253   AES         128      TLS_ECDH_anon_WITH_AES_128_CBC_SHA
 x34     ADH-AES128-SHA                    DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA
 x9b     ADH-SEED-SHA                      DH 2048    SEED        128      TLS_DH_anon_WITH_SEED_CBC_SHA
 x46     ADH-CAMELLIA128-SHA               DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA
 x96     SEED-SHA                          RSA        SEED        128      TLS_RSA_WITH_SEED_CBC_SHA
 x41     CAMELLIA128-SHA                   RSA        Camellia    128      TLS_RSA_WITH_CAMELLIA_128_CBC_SHA
TLS 1.2
 xc030   ECDHE-RSA-AES256-GCM-SHA384       ECDH 253   AESGCM      256      TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
 xc028   ECDHE-RSA-AES256-SHA384           ECDH 253   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
 xc014   ECDHE-RSA-AES256-SHA              ECDH 253   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
 x9f     DHE-RSA-AES256-GCM-SHA384         DH 2048    AESGCM      256      TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
 xcca8   ECDHE-RSA-CHACHA20-POLY1305       ECDH 253   ChaCha20    256      TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
 xccaa   DHE-RSA-CHACHA20-POLY1305         DH 2048    ChaCha20    256      TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256
 xc0a3   DHE-RSA-AES256-CCM8               DH 2048    AESCCM8     256      TLS_DHE_RSA_WITH_AES_256_CCM_8
 xc09f   DHE-RSA-AES256-CCM                DH 2048    AESCCM      256      TLS_DHE_RSA_WITH_AES_256_CCM
 x6b     DHE-RSA-AES256-SHA256             DH 2048    AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
 x39     DHE-RSA-AES256-SHA                DH 2048    AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA
 xc077   ECDHE-RSA-CAMELLIA256-SHA384      ECDH 253   Camellia    256      TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384
 xc4     DHE-RSA-CAMELLIA256-SHA256        DH 2048    Camellia    256      TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256
 x88     DHE-RSA-CAMELLIA256-SHA           DH 2048    Camellia    256      TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA
 xc019   AECDH-AES256-SHA                  ECDH 253   AES         256      TLS_ECDH_anon_WITH_AES_256_CBC_SHA
 xa7     ADH-AES256-GCM-SHA384             DH 2048    AESGCM      256      TLS_DH_anon_WITH_AES_256_GCM_SHA384
 x6d     ADH-AES256-SHA256                 DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA256
 x3a     ADH-AES256-SHA                    DH 2048    AES         256      TLS_DH_anon_WITH_AES_256_CBC_SHA
 xc5     ADH-CAMELLIA256-SHA256            DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA256
 x89     ADH-CAMELLIA256-SHA               DH 2048    Camellia    256      TLS_DH_anon_WITH_CAMELLIA_256_CBC_SHA
 x9d     AES256-GCM-SHA384                 RSA        AESGCM      256      TLS_RSA_WITH_AES_256_GCM_SHA384
 xc0a1   AES256-CCM8                       RSA        AESCCM8     256      TLS_RSA_WITH_AES_256_CCM_8
 xc09d   AES256-CCM                        RSA        AESCCM      256      TLS_RSA_WITH_AES_256_CCM
 x3d     AES256-SHA256                     RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA256
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA
 xc0     CAMELLIA256-SHA256                RSA        Camellia    256      TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256
 x84     CAMELLIA256-SHA                   RSA        Camellia    256      TLS_RSA_WITH_CAMELLIA_256_CBC_SHA
 xc051   ARIA256-GCM-SHA384                RSA        ARIAGCM     256      TLS_RSA_WITH_ARIA_256_GCM_SHA384
 xc053   DHE-RSA-ARIA256-GCM-SHA384        DH 2048    ARIAGCM     256      TLS_DHE_RSA_WITH_ARIA_256_GCM_SHA384
 xc061   ECDHE-ARIA256-GCM-SHA384          ECDH 253   ARIAGCM     256      TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384
 xc02f   ECDHE-RSA-AES128-GCM-SHA256       ECDH 253   AESGCM      128      TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
 xc027   ECDHE-RSA-AES128-SHA256           ECDH 253   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
 xc013   ECDHE-RSA-AES128-SHA              ECDH 253   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
 x9e     DHE-RSA-AES128-GCM-SHA256         DH 2048    AESGCM      128      TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
 xc0a2   DHE-RSA-AES128-CCM8               DH 2048    AESCCM8     128      TLS_DHE_RSA_WITH_AES_128_CCM_8
 xc09e   DHE-RSA-AES128-CCM                DH 2048    AESCCM      128      TLS_DHE_RSA_WITH_AES_128_CCM
 xc0a0   AES128-CCM8                       RSA        AESCCM8     128      TLS_RSA_WITH_AES_128_CCM_8
 xc09c   AES128-CCM                        RSA        AESCCM      128      TLS_RSA_WITH_AES_128_CCM
 x67     DHE-RSA-AES128-SHA256             DH 2048    AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
 x33     DHE-RSA-AES128-SHA                DH 2048    AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA
 xc076   ECDHE-RSA-CAMELLIA128-SHA256      ECDH 253   Camellia    128      TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256
 xbe     DHE-RSA-CAMELLIA128-SHA256        DH 2048    Camellia    128      TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256
 x9a     DHE-RSA-SEED-SHA                  DH 2048    SEED        128      TLS_DHE_RSA_WITH_SEED_CBC_SHA
 x45     DHE-RSA-CAMELLIA128-SHA           DH 2048    Camellia    128      TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA
 xc018   AECDH-AES128-SHA                  ECDH 253   AES         128      TLS_ECDH_anon_WITH_AES_128_CBC_SHA
 xa6     ADH-AES128-GCM-SHA256             DH 2048    AESGCM      128      TLS_DH_anon_WITH_AES_128_GCM_SHA256
 x6c     ADH-AES128-SHA256                 DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA256
 x34     ADH-AES128-SHA                    DH 2048    AES         128      TLS_DH_anon_WITH_AES_128_CBC_SHA
 xbf     ADH-CAMELLIA128-SHA256            DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA256
 x9b     ADH-SEED-SHA                      DH 2048    SEED        128      TLS_DH_anon_WITH_SEED_CBC_SHA
 x46     ADH-CAMELLIA128-SHA               DH 2048    Camellia    128      TLS_DH_anon_WITH_CAMELLIA_128_CBC_SHA
 x9c     AES128-GCM-SHA256                 RSA        AESGCM      128      TLS_RSA_WITH_AES_128_GCM_SHA256
 x3c     AES128-SHA256                     RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA256
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA
 xba     CAMELLIA128-SHA256                RSA        Camellia    128      TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256
 x96     SEED-SHA                          RSA        SEED        128      TLS_RSA_WITH_SEED_CBC_SHA
 x41     CAMELLIA128-SHA                   RSA        Camellia    128      TLS_RSA_WITH_CAMELLIA_128_CBC_SHA
 xc050   ARIA128-GCM-SHA256                RSA        ARIAGCM     128      TLS_RSA_WITH_ARIA_128_GCM_SHA256
 xc052   DHE-RSA-ARIA128-GCM-SHA256        DH 2048    ARIAGCM     128      TLS_DHE_RSA_WITH_ARIA_128_GCM_SHA256
 xc060   ECDHE-ARIA128-GCM-SHA256          ECDH 253   ARIAGCM     128      TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256
TLS 1.3
 x1302   TLS_AES_256_GCM_SHA384            ECDH 253   AESGCM      256      TLS_AES_256_GCM_SHA384
 x1303   TLS_CHACHA20_POLY1305_SHA256      ECDH 253   ChaCha20    256      TLS_CHACHA20_POLY1305_SHA256
 x1301   TLS_AES_128_GCM_SHA256            ECDH 253   AESGCM      128      TLS_AES_128_GCM_SHA256

With switching our SSL certificate to RSA, we gained around 51 new cipher options, amongst them being ones that also mx01.arz.at claimed to support. FTR, the result from above is what you get with the default settings for postfix v3.5.18, being:

smtpd_tls_ciphers = medium
smtpd_tls_mandatory_ciphers = medium
smtpd_tls_mandatory_exclude_ciphers =
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3

But the delay between triggering the password reset mail and getting a mail server connect was getting bigger and bigger. Therefore while waiting for the next mail to arrive, I decided to capture the network traffic, to be able to look further into this if it should continue to be failing:

% sudo tshark -n -i eth0 -s 65535 -w arz.pcap -f "host 193.110.182.61 or host 193.110.182.62"

A few hours later the mail server connected again, and the mail went through!

postfix/smtpd[4162835]: connect from mx01.arz.at[193.110.182.61]
postfix/smtpd[4162835]: Anonymous TLS connection established from mx01.arz.at[193.110.182.61]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
postfix/smtpd[4162835]: E50D6401E6: client=mx01.arz.at[193.110.182.61]
postfix/smtpd[4162835]: disconnect from mx01.arz.at[193.110.182.61] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 quit=1 commands=7

Now also having the captured network traffic, we can check the details there:

[...]
% tshark -o smtp.decryption:true -r arz.pcap
    1 0.000000000 193.110.182.61   203.0.113.42 TCP 74 24699   25 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=2261106119 TSecr=0 WS=128
    2 0.000042827 203.0.113.42   193.110.182.61 TCP 74 25   24699 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=3233422181 TSecr=2261106119 WS=128
    3 0.020719269 193.110.182.61   203.0.113.42 TCP 66 24699   25 [ACK] Seq=1 Ack=1 Win=29312 Len=0 TSval=2261106139 TSecr=3233422181
    4 0.022883259 203.0.113.42   193.110.182.61 SMTP 96 S: 220 mail.example.com ESMTP
    5 0.043682626 193.110.182.61   203.0.113.42 TCP 66 24699   25 [ACK] Seq=1 Ack=31 Win=29312 Len=0 TSval=2261106162 TSecr=3233422203
    6 0.043799047 193.110.182.61   203.0.113.42 SMTP 84 C: EHLO mx01.arz.at
    7 0.043811363 203.0.113.42   193.110.182.61 TCP 66 25   24699 [ACK] Seq=31 Ack=19 Win=65280 Len=0 TSval=3233422224 TSecr=2261106162
    8 0.043898412 203.0.113.42   193.110.182.61 SMTP 253 S: 250-mail.example.com   PIPELINING   SIZE 20240000   VRFY   ETRN   AUTH PLAIN   AUTH=PLAIN   ENHANCEDSTATUSCODES   8BITMIME   DSN   SMTPUTF8   CHUNKING
    9 0.064625499 193.110.182.61   203.0.113.42 SMTP 72 C: QUIT
   10 0.064750257 203.0.113.42   193.110.182.61 SMTP 81 S: 221 2.0.0 Bye
   11 0.064760200 203.0.113.42   193.110.182.61 TCP 66 25   24699 [FIN, ACK] Seq=233 Ack=25 Win=65280 Len=0 TSval=3233422245 TSecr=2261106183
   12 0.085573715 193.110.182.61   203.0.113.42 TCP 66 24699   25 [FIN, ACK] Seq=25 Ack=234 Win=30336 Len=0 TSval=2261106204 TSecr=3233422245
   13 0.085610229 203.0.113.42   193.110.182.61 TCP 66 25   24699 [ACK] Seq=234 Ack=26 Win=65280 Len=0 TSval=3233422266 TSecr=2261106204
   14 1799.888108373 193.110.182.61   203.0.113.42 TCP 74 10330   25 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=2262906007 TSecr=0 WS=128
   15 1799.888161311 203.0.113.42   193.110.182.61 TCP 74 25   10330 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM=1 TSval=3235222069 TSecr=2262906007 WS=128
   16 1799.909030335 193.110.182.61   203.0.113.42 TCP 66 10330   25 [ACK] Seq=1 Ack=1 Win=29312 Len=0 TSval=2262906028 TSecr=3235222069
   17 1799.956621011 203.0.113.42   193.110.182.61 SMTP 96 S: 220 mail.example.com ESMTP
   18 1799.977229656 193.110.182.61   203.0.113.42 TCP 66 10330   25 [ACK] Seq=1 Ack=31 Win=29312 Len=0 TSval=2262906096 TSecr=3235222137
   19 1799.977229698 193.110.182.61   203.0.113.42 SMTP 84 C: EHLO mx01.arz.at
   20 1799.977266759 203.0.113.42   193.110.182.61 TCP 66 25   10330 [ACK] Seq=31 Ack=19 Win=65280 Len=0 TSval=3235222158 TSecr=2262906096
   21 1799.977351663 203.0.113.42   193.110.182.61 SMTP 267 S: 250-mail.example.com   PIPELINING   SIZE 20240000   VRFY   ETRN   STARTTLS   AUTH PLAIN   AUTH=PLAIN   ENHANCEDSTATUSCODES   8BITMIME   DSN   SMTPUTF8   CHUNKING
   22 1800.011494861 193.110.182.61   203.0.113.42 SMTP 76 C: STARTTLS
   23 1800.011589267 203.0.113.42   193.110.182.61 SMTP 96 S: 220 2.0.0 Ready to start TLS
   24 1800.032812294 193.110.182.61   203.0.113.42 TLSv1 223 Client Hello
   25 1800.032987264 203.0.113.42   193.110.182.61 TLSv1.2 2962 Server Hello
   26 1800.032995513 203.0.113.42   193.110.182.61 TCP 1266 25   10330 [PSH, ACK] Seq=3158 Ack=186 Win=65152 Len=1200 TSval=3235222214 TSecr=2262906151 [TCP segment of a reassembled PDU]
   27 1800.053546755 193.110.182.61   203.0.113.42 TCP 66 10330   25 [ACK] Seq=186 Ack=3158 Win=36096 Len=0 TSval=2262906172 TSecr=3235222214
   28 1800.092852469 193.110.182.61   203.0.113.42 TCP 66 10330   25 [ACK] Seq=186 Ack=4358 Win=39040 Len=0 TSval=2262906212 TSecr=3235222214
   29 1800.092892905 203.0.113.42   193.110.182.61 TLSv1.2 900 Certificate, Server Key Exchange, Server Hello Done
   30 1800.113546769 193.110.182.61   203.0.113.42 TCP 66 10330   25 [ACK] Seq=186 Ack=5192 Win=41856 Len=0 TSval=2262906232 TSecr=3235222273
   31 1800.114763363 193.110.182.61   203.0.113.42 TLSv1.2 192 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
   32 1800.115000416 203.0.113.42   193.110.182.61 TLSv1.2 117 Change Cipher Spec, Encrypted Handshake Message
   33 1800.136070200 193.110.182.61   203.0.113.42 TLSv1.2 113 Application Data
   34 1800.136155526 203.0.113.42   193.110.182.61 TLSv1.2 282 Application Data
   35 1800.158854473 193.110.182.61   203.0.113.42 TLSv1.2 162 Application Data
   36 1800.159254794 203.0.113.42   193.110.182.61 TLSv1.2 109 Application Data
   37 1800.180286407 193.110.182.61   203.0.113.42 TLSv1.2 144 Application Data
   38 1800.223005960 203.0.113.42   193.110.182.61 TCP 66 25   10330 [ACK] Seq=5502 Ack=533 Win=65152 Len=0 TSval=3235222404 TSecr=2262906299
   39 1802.230300244 203.0.113.42   193.110.182.61 TLSv1.2 146 Application Data
   40 1802.251994333 193.110.182.61   203.0.113.42 TCP 2962 [TCP segment of a reassembled PDU]
   41 1802.252034015 203.0.113.42   193.110.182.61 TCP 66 25   10330 [ACK] Seq=5582 Ack=3429 Win=63616 Len=0 TSval=3235224433 TSecr=2262908371
   42 1802.252279083 193.110.182.61   203.0.113.42 TLSv1.2 1295 Application Data
   43 1802.252288316 203.0.113.42   193.110.182.61 TCP 66 25   10330 [ACK] Seq=5582 Ack=4658 Win=64128 Len=0 TSval=3235224433 TSecr=2262908371
   44 1802.272816060 193.110.182.61   203.0.113.42 TLSv1.2 833 Application Data, Application Data
   45 1802.272827542 203.0.113.42   193.110.182.61 TCP 66 25   10330 [ACK] Seq=5582 Ack=5425 Win=64128 Len=0 TSval=3235224453 TSecr=2262908392
   46 1802.338807683 203.0.113.42   193.110.182.61 TLSv1.2 131 Application Data
   47 1802.398968611 193.110.182.61   203.0.113.42 TCP 66 10330   25 [ACK] Seq=5425 Ack=5647 Win=44800 Len=0 TSval=2262908518 TSecr=3235224519
   48 1863.257457500 193.110.182.61   203.0.113.42 TLSv1.2 101 Application Data
   49 1863.257495688 203.0.113.42   193.110.182.61 TCP 66 25   10330 [ACK] Seq=5647 Ack=5460 Win=64128 Len=0 TSval=3235285438 TSecr=2262969376
   50 1863.257654942 203.0.113.42   193.110.182.61 TLSv1.2 110 Application Data
   51 1863.257721010 203.0.113.42   193.110.182.61 TLSv1.2 97 Encrypted Alert
   52 1863.278242216 193.110.182.61   203.0.113.42 TCP 66 10330   25 [ACK] Seq=5460 Ack=5691 Win=44800 Len=0 TSval=2262969397 TSecr=3235285438
   53 1863.278464176 193.110.182.61   203.0.113.42 TCP 66 10330   25 [RST, ACK] Seq=5460 Ack=5723 Win=44800 Len=0 TSval=2262969397 TSecr=3235285438

% tshark -O tls -r arz.pcap
[...]
Transport Layer Security
    TLSv1 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 152
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 148
            Version: TLS 1.2 (0x0303)
            Random: 4575d1e7c93c09a564edc00b8b56ea6f5d826f8cfe78eb980c451a70a9c5123f
                GMT Unix Time: Dec  5, 2006 21:09:11.000000000 CET
                Random Bytes: c93c09a564edc00b8b56ea6f5d826f8cfe78eb980c451a70a9c5123f
            Session ID Length: 0
            Cipher Suites Length: 26
            Cipher Suites (13 suites)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
                Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
                Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
                Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)
[...]
Transport Layer Security
    TLSv1.2 Record Layer: Handshake Protocol: Server Hello
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 89
        Handshake Protocol: Server Hello
            Handshake Type: Server Hello (2)
            Length: 85
            Version: TLS 1.2 (0x0303)
            Random: cf2ed24e3300e95e5f56023bf8b4e5904b862bb2ed8a5796444f574e47524401
                GMT Unix Time: Feb 23, 2080 23:16:46.000000000 CET
                Random Bytes: 3300e95e5f56023bf8b4e5904b862bb2ed8a5796444f574e47524401
            Session ID Length: 32
            Session ID: 63d041b126ecebf857d685abd9d4593c46a3672e1ad76228f3eacf2164f86fb9
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
[...]

In this network dump we see what cipher suites are offered, and the TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 here is the Cipher Suite Name in IANA/RFC speak. Whis corresponds to the ECDHE-RSA-AES256-GCM-SHA384 in openssl speak (see Mozilla s Mozilla s cipher suite correspondence table), which we also saw in the postfix log. Mission accomplished! :) Now, if we re interested in avoiding certain ciphers and increase security level, we can e.g. get rid of the SEED, CAMELLIA and all anonymous ciphers, and could accept only TLS v1.2 + v1.3, by further adjusting postfix s main.cf:

smtpd_tls_ciphers = high
smtpd_tls_exclude_ciphers = aNULL CAMELLIA
smtpd_tls_mandatory_ciphers = high
smtpd_tls_mandatory_protocols = TLSv1.2 TLSv1.3
smtpd_tls_protocols = TLSv1.2 TLSv1.3

Which would then gives us:

% testssl --cipher-per-proto -t=smtp mail.example.com:25
[...]
Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
SSLv3
TLS 1
TLS 1.1
TLS 1.2
 xc030   ECDHE-RSA-AES256-GCM-SHA384       ECDH 253   AESGCM      256      TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
 xc028   ECDHE-RSA-AES256-SHA384           ECDH 253   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
 xc014   ECDHE-RSA-AES256-SHA              ECDH 253   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
 x9f     DHE-RSA-AES256-GCM-SHA384         DH 2048    AESGCM      256      TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
 xcca8   ECDHE-RSA-CHACHA20-POLY1305       ECDH 253   ChaCha20    256      TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
 xccaa   DHE-RSA-CHACHA20-POLY1305         DH 2048    ChaCha20    256      TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256
 xc0a3   DHE-RSA-AES256-CCM8               DH 2048    AESCCM8     256      TLS_DHE_RSA_WITH_AES_256_CCM_8
 xc09f   DHE-RSA-AES256-CCM                DH 2048    AESCCM      256      TLS_DHE_RSA_WITH_AES_256_CCM
 x6b     DHE-RSA-AES256-SHA256             DH 2048    AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
 x39     DHE-RSA-AES256-SHA                DH 2048    AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA
 x9d     AES256-GCM-SHA384                 RSA        AESGCM      256      TLS_RSA_WITH_AES_256_GCM_SHA384
 xc0a1   AES256-CCM8                       RSA        AESCCM8     256      TLS_RSA_WITH_AES_256_CCM_8
 xc09d   AES256-CCM                        RSA        AESCCM      256      TLS_RSA_WITH_AES_256_CCM
 x3d     AES256-SHA256                     RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA256
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA
 xc051   ARIA256-GCM-SHA384                RSA        ARIAGCM     256      TLS_RSA_WITH_ARIA_256_GCM_SHA384
 xc053   DHE-RSA-ARIA256-GCM-SHA384        DH 2048    ARIAGCM     256      TLS_DHE_RSA_WITH_ARIA_256_GCM_SHA384
 xc061   ECDHE-ARIA256-GCM-SHA384          ECDH 253   ARIAGCM     256      TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384
 xc02f   ECDHE-RSA-AES128-GCM-SHA256       ECDH 253   AESGCM      128      TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
 xc027   ECDHE-RSA-AES128-SHA256           ECDH 253   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
 xc013   ECDHE-RSA-AES128-SHA              ECDH 253   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
 x9e     DHE-RSA-AES128-GCM-SHA256         DH 2048    AESGCM      128      TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
 xc0a2   DHE-RSA-AES128-CCM8               DH 2048    AESCCM8     128      TLS_DHE_RSA_WITH_AES_128_CCM_8
 xc09e   DHE-RSA-AES128-CCM                DH 2048    AESCCM      128      TLS_DHE_RSA_WITH_AES_128_CCM
 xc0a0   AES128-CCM8                       RSA        AESCCM8     128      TLS_RSA_WITH_AES_128_CCM_8
 xc09c   AES128-CCM                        RSA        AESCCM      128      TLS_RSA_WITH_AES_128_CCM
 x67     DHE-RSA-AES128-SHA256             DH 2048    AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
 x33     DHE-RSA-AES128-SHA                DH 2048    AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA
 x9c     AES128-GCM-SHA256                 RSA        AESGCM      128      TLS_RSA_WITH_AES_128_GCM_SHA256
 x3c     AES128-SHA256                     RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA256
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA
 xc050   ARIA128-GCM-SHA256                RSA        ARIAGCM     128      TLS_RSA_WITH_ARIA_128_GCM_SHA256
 xc052   DHE-RSA-ARIA128-GCM-SHA256        DH 2048    ARIAGCM     128      TLS_DHE_RSA_WITH_ARIA_128_GCM_SHA256
 xc060   ECDHE-ARIA128-GCM-SHA256          ECDH 253   ARIAGCM     128      TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256
TLS 1.3
 x1302   TLS_AES_256_GCM_SHA384            ECDH 253   AESGCM      256      TLS_AES_256_GCM_SHA384
 x1303   TLS_CHACHA20_POLY1305_SHA256      ECDH 253   ChaCha20    256      TLS_CHACHA20_POLY1305_SHA256
 x1301   TLS_AES_128_GCM_SHA256            ECDH 253   AESGCM      128      TLS_AES_128_GCM_SHA256

Don t forget to also adjust the smpt_tls_* accordingly (for your sending side). For further information see the Postfix TLS Support documentation. Also check out options like tls_ssl_options (setting it to e.g. NO_COMPRESSION) and tls_preempt_cipherlist (setting it to yes would prefer the servers order of ciphers over clients). Conclusions:

no matter what you change in your mail server settings, be aware that the type of your SSL certificate also matters for what ciphers are offered and used
there are mail servers out there that don t support SSL certificates with ECDSA, using RSA for those ensure better compatibility (nowadays postfix supports parallel usage of ECDSA and RSA keys BTW, check out the smtpd_tls_eccert_file + smtpd_tls_eckey_file options)
testssl is a very useful tool, especially with its cipher-per-proto -t=smtp option to check SMTP servers
if you re uncertain what s going on, consider capturing network data (tshark/tcpdump/ are your friends)
review your postfix configuration and logs every now and then :)

Debian Celebrates 30 years! We celebrated our birthday this year and we had a great time with new friends, new members welcomed to the community, and the world. We have collected a few comments, videos, and discussions from around the Internet, and some images from some of the DebianDay2023 events. We hope that you enjoyed the day(s) as much as we did!

"Debian 30 years of collective intelligence" -Maqsuel Maqson Brazil

Pouso Alegre, Brazil

Macei , Brazil

Curitiba, Brazil

The cake is there. :) Honorary Debian Developers: Buzz, Jessie, and Woody welcome guests to this amazing party. Sao Carlos, state of Sao Paulo, Brazil Stickers, and Fliers, and Laptops, oh my! Belo Horizonte, Brazil Bras lia, Brazil Bras lia, Brazil Mexico 30 a os! A quick Selfie We do not encourage beverages on computing hardware, but this one is okay by us. Germany

30 years of love

The German Delegation is also looking for this dog who footed the bill for the party, then left mysteriously.

We took the party outside

We brought the party back inside at CCCamp Belgium

Cake and Diversity in Belgium El Salvador

Food and Fellowship in El Salvador South Africa

Debian is also very delicious!

All smiles waiting to eat the cake Reports Debian Day 30 years in Macei - Brazil Debian Day 30 years in S o Carlos - Brazil Debian Day 30 years in Pouso Alegre - Brazil Debian Day 30 years in Belo Horizonte - Brazil Debian Day 30 years in Curitiba - Brazil Debian Day 30 years in Bras lia - Brazil Debian Day 30 years online in Brazil Articles & Blogs Happy Debian Day - going 30 years strong - Liam Dawe Debian Turns 30 Years Old, Happy Birthday! - Marius Nestor 30 Years of Stability, Security, and Freedom: Celebrating Debian s Birthday - Bobby Borisov Happy 30th Birthday, Debian! - Claudio Kuenzier Debian is 30 and Sgt Pepper Is at Least Ninetysomething - Christine Hall Debian turns 30! -Corbet Thirty years of Debian! - Lennart Hengstmengel Debian marks three decades as 'Universal Operating System' - Sam Varghese Debian Linux Celebrates 30 Years Milestone - Joshua James 30 years on, Debian is at the heart of the world's most successful Linux distros - Liam Proven Looking Back on 30 Years of Debian - Maya Posch Cheers to 30 Years of Debian: A Journey of Open Source Excellence - arindam Discussions and Social Media Debian Celebrates 30 Years - Source: News YCombinator Brand-new Linux release, which I'm calling the Debian ... Source: News YCombinator Comment: Congrats @debian !!! Happy Birthday! Thank you for becoming a cornerstone of the #opensource world. Here's to decades of collaboration, stability & #software #freedom -openSUSELinux via X (formerly Twitter) Comment: Today we #celebrate the 30th birthday of #Debian, one of the largest and most important cornerstones of the #opensourcecommunity. For this we would like to thank you very much and wish you the best for the next 30 years! Source: X (Formerly Twitter -TUXEDOComputers via X (formerly Twitter) Happy Debian Day! - Source: Reddit.com Video The History of Debian The Beginning - Source: Linux User Space Debian Celebrates 30 years -Source: Lobste.rs Video Debian At 30 and No More Distro Hopping! - LWDW388 - Source: LinuxGameCast Debian Celebrates 30 years! - Source: Debian User Forums Debian Celebrates 30 years! - Source: Linux.org

I uploaded sgt-puzzles to unstable. This brought in the new upstream version previously in experimental. I incorporated an updated German translation from Helge Kreutzmann, and made translation updates less tricky to do.
I made some changes to the nfs-utils package:
- Completed the transition from setting command-line options in /etc/default to /etc/nfs.conf.d.
- Made its shell scripts shellcheck-clean and added shellcheck to CI. (Thanks to who sent a patch for the init scripts.)
These were included in version 1:2.6.3-1.
I accepted several MRs on Salsa:
- linux/master: [armhf] drivers/staging/media/rkvdec: enable rkvdec as module
- linux/master: Update to 6.4-rcX
- linux/bookworm: udeb: add r8188eu to nic-wireless-modules (Closes: #1035824)
- nfs-utils/master: Rely on the generator units for the rpc_pipefs mount
- linux/master: mm: Enable Multi-Gen LRU implementation (by default)
- linux/master: d/rules.real: Also remove executable bit from dtbo files
- linux/master: [mips*]: Fixes for boston kernel
- linux/sid: Ignore ABI changes for xfrm_bpf_md_dst (only for use in xfrm subsystem)
I uploaded linux versions 6.4~rc6-1~exp1 and 6.4~rc7-1~exp1 to experimental.
I updated the buster-security (4.19) branch of linux to stable version 4.19.288, but didn't upload it this month.
I fixed build regressions for linux/experimental on several architectures, and sent the changes upstream where appropriate (hppa, m68k, and preemptively sparc).
I created a bookworm-backports branch for the linux package, but that suite is not yet open to uploads.
I uploaded linux version 6.1.27-1~bpo11+1 and firmware-nonfree version 20230210-5~bpo11+1 to bullseye-backports, but they still haven't been accepted.
I fixed a build regression and many other bugs in dadhi-linux.
I realised that the linux test-patches script still wasn't building all the packages needed to make the linux-headers package (or, on some architectures, the linux-image package) installable. I fixed this for unstable, backported those changes to bookworm, and backported all the test-patches changes to bullseye.
I prepared a backport of firmware-nonfree version 20220913-1 to bullseye (not -backports). This is based on the work Tobias Frost did to update it in buster-security (Debian LTS).
I updated the Debian Kernel Handbook to use the Debian stylesheet. This is now in the live version but I haven't uploaded the package.
I started backporting various kernel security fixes to the affected stable branches. These are not yet tested or submitted upstream.
I fixed a regression in cpu_rmap in the Linux kernel.
I reported a regression in rtl8192 on Linux stable branches.
I co-organised the Debian release party in Leuven. and posted a group photo of this on Mastodon.
The amdgpu driver lists some firmware files as potentially needed that aren't packaged or even publicly available, which leads to warnings from initramfs-tools on systems using this driver. I queried these upstream, which should hopefully lead to a resolution of the bug.
I wrote the arm, mips, and riscv parts of the fix for CVE-2023-3269 a.k.a. StackRot.

After an upgrade of my PC's mainboard BIOS the boot would take a minute or more to complete and sometimes the lightdm login screen would sit there but not accept keyboard input for another minute or so. Then the keyboard got enabled and I could log in normally. Everything worked fine after that bootup struggle completed. This was fully reproducible and persisted across reboots. Weird. The kernel dmesg log showed entries that looked suspicious: dmesg log excerpt showing USB error messages

dmesg log excerpt showing USB error messages

Googleing these error -110 and error -71 is a bit hard. Now why the USB driver does not give useful error messages instead of archaic errno-style numbers escapes me. This is not the 80s anymore. Citation needed (Wikipedia style)

The wisdom of the crowd says error -110 is something around "the USB port power supply was exceeded" [source]. Now lsusb -tv shows device 1-7 ... to be my USB keyboard. I somehow doubt that wants more power than the hub is willing to provide. The Archlinux BBS Forums recommend to piece together information from drivers/usb/host/ohci.h and (updated from their piece which is from 2012) /tools/include/uapi/asm-generic/errno.h. This is why some people then consider -110 to mean "Connection timed out". Nah, not likely either. Reading through the kernel source around drivers/usb/host did not enlighten me either. To the contrary. Uuugly. There seems to be no comprehensive list what these error codes mean. And the numbers are assigned to errors conditions quite arbitrarily. And - of course - there is no documentation. "It was hard to do, so it should be hard to understand as well." Luckily some of the random musings I read through contained some curious advice: power cycle the host. So I did and that did not make the error go away. Other people insisted on removing cables out of wall sockets, unplugging everything and conducting esoteric rituals. That made it dawn on me, the mainboard of course nicely powers the USB in "off" state, too. So switching the power supply off (yes, these have a separate switch, go find yours), waiting a bit for capacitors to drain and switching things back on and ... the errors were gone, the system booted within seconds again. So the takeaway message: If you get random error messages like

device descriptor read/64, error -110
device not accepting address 42, error -71

on devices that previously worked fine ... completely remove power from the host, the hubs and the USB devices. So they forget they saw each other on the bus before. And when they see each other after that blackout, they will happily go through negotiating protocol details with each other again successfully.

When I connect my Desklab USB-C monitor [1] (which has been vastly underused for the last 3 years) into a Linux system the display type is listed as DO NOT USE RTK . One of the more informative discussions of this was on Linux Mint forums [2] which revealed that it s a mapping for an code that shouldn t be used. So it s not saying don t use this monitor it s saying don t use this code . So the Desklab people when they implemented a display with an RTK chipset should have changed the ID field from RTK to something representing their use. On Debian the file /usr/share/hwdata/pnp.ids has the IDs and you can grep for RTK in that. Also for programmers, please use more descriptive strings than do not use , when I was trying to find this on Debian code search [3] it turned up hundreds of pages of results which was more than a human can read through. If the text had been something that would make sense to a user such as OEM please replace with company name it would have made it very clear to me (and all the other people searching for this) what it meant and the fact that Desklab had stuffed up. So instead of wondering about this for years before eventually finding the right Google search to find the answer I could have worked it out immediately if the text had been clearer.

First up is Minidebconf Tamilnadu 2023 that would be held on 28-29 January 2023. You can find rest of the details here. I do hope we get to see/hear some good stuff from the Minidebconf. Best of luck to all those who are applying.

Tinnitus During the lock-down of March 2020, I became aware of noise in ears and subsequently major hearing loss. It took me quite a while to know that Tinnitus happens to both those who have hearing loss as well as not. I keep running into threads like this and as shared by someone nobody knows what really causes it. I did try some of the apps (an app. called Resound on Android) that is supposed to tackle Tinnitus but it hasn t helped much. There is this but at least for me, right now pretty speculative. Also this, and again highly speculative.

Cooking After mum passed away, I haven t cooked anything. This used to give me pleasure but now just doesn t feel right. Cooking is something you enjoy when you are doing for somebody else and not just for yourself, at least that s how I feel and with that the curiosity to know more recipes. I do wanna buy a wok at sometime but when, how I just don t know.

Books Have been reading books quite a bit. And due to that had to again revisit and understand ISBN. Perhaps I might have shared it before. It really is something, the history of ISBN. And that co-relates with the book I read, Raising Steam by Terry Pratchett. Raising Steam is the 40th Book in the Discworld Series and it basically romanticizes and reminisces how the idea of an engine was born, and then a steam engine and how actually Railways started. There has been a lot of history and experiences from the early years of Steam Railway that have been taken and transplanted into the book. Also how Railways is and can be successful if only it is invested wisely and maintenance is done. This is only where imagination and reality come apart as maintenance isn t done and then you have issues. While this is and was in the UK, similar situation exists in India and many other places around the world and doesn t matter whether it is private or public. Exceptions are German, French but then that maybe due to Labor movements that happened and were successful unlike in other places. I could go on but then it will become a different article in itself. Suffice to say there is much to learn and you need serious people to look after it. Both in UK and India we lack that. And not just in Railways but Civil Aviation too, but again that is a story in itself.

Web-series Apart from books, have been seeing web-series that Willow is a good one that I enjoyed even though I hadn t seen the earlier movie. While there has been a flurry of movies and web-series both due to end of year and beginning of 2023 and yet have tried to be a bit partial on what I wanna watch or not. If it has crime, fantasy, drama then usually I like it. For e.g. I saw Blackout and pretty much was engrossed in what will happen next. It also does lead you to ask questions about centralization vs de-centralization for both power and other utilities and does make a case for communities to have their utilities apart from the grid as a fallback. How do we do over decades or centuries about it is a different question perhaps altogether. There were two books that kinda stood out for me, the first was Ian Rankin s Naming of the Dead . The book is about a cynical John Rebus, a man after my own heart. I am probably going to buy a few more of his series. In a way it also tells you why UK is the way it is right now. Another book that I liked was Shades of Grey by Jasper Fforde. This is one of the books that Mum would have clearly liked. It is pretty unusual while at the same time very close to 1984 and other such dystopian novels. The main trope of the book is what color you can see and how much you can see. The main character is somebody who can see Red, around the age of 20. One of the interesting aspects of the book is de-facting which closely resembles the Post-Truth world where alternative facts can be made out of air and they don t need any scientific evidence to back them up. In Jasper s world, they don t care about how things work and most of the technology is banned and curiosity is considered harmful and those who show that are murdered one way or the other. Interestingly, the author has just last year decided to start book 2 in the 3 book series that is supposed to be. This also tells why the U.S. is such a precarious situation in a way. A part of it is also due to the media which is in hands of chosen few, the same goes for UK and India, almost an oligopoly.

The Great Escape This is also a book but also about experiences of people, not in 19th-20th century but today that tells you slavery is alive and well and human-trafficking as well. This piece from NPR tells you about an MNC and Indian workers. What I found interesting is that there barely is an mention of the Indian Embassy that is supposed to help Indian people. I do know for a fact that the embassies of India has seen a drastic shortage of both people and materials even since the new Govt. came in place that was nine years ago. Incidentally, BBC shared about the Gujarat riots 2002 and that has been censored in India. They keep quiet about the UK Govt. who did find out that the Chief Minister was directly responsible for the killings and in facts his number 2, Amit Shah had shared that we would do 2002 again in the election cycle barely a month ago. But sadly, no hate speech FIR or any action was taken against Mr. Shah. There have been attempts by people to showcase the documentary. For e.g. JNU tried it and the rowdies from ABVP (arm of BJP) created violence. Even the questions that has been asked by the Wire, GOI will not acknowledge them. Interestingly, all India s edtechs have taken a beating in the last 6-8 months including the biggest BJYU s. Sharing a story from 2021 where things were best and today all of them are at bottom. In fact, the public has been wary as the prices of the courses has kept on increasing and most case studies have been found to be fake. Also the general outlook on jobs and growth has been pessimistic. In fact, most companies have been shedding jobs truckloads, most in the I.T. sector but other sectors as well. Hospitality and other related sectors have taken a huge beating, part of it post-pandemic, part of it Govt s refusal to either spend money or do any positive policies for either infrastructure, education, medical, you name it, they think private sector has all the answers which has been proven to be wrong again and again. I did not want to end on a discordant note but things are the way they are

I watched a 2015 video from Andreas Schiffler the other day, where he set up LinuxCNC to send status information to the MQTT broker IBM Bluemix. As I also use MQTT for graphing, it occured to me that a generic MQTT LinuxCNC component would be useful and I set out to implement it. Today I got the first draft limping along and submitted as a patch to the LinuxCNC project. The simple part was setting up the MQTT publishing code in Python. I already have set up other parts submitting data to my Mosquito MQTT broker, so I could reuse that code. Writing a LinuxCNC component in Python as new to me, but using existing examples in the code repository and the extensive documentation, this was fairly straight forward. The hardest part was creating a automated test for the component to ensure it was working. Testing it in a simulated LinuxCNC machine proved very useful, as I discovered features I needed that I had not thought of yet, and adjusted the code quite a bit to make it easier to test without a operational MQTT broker available. The draft is ready and working, but I am unsure which LinuxCNC HAL pins I should collect and publish by default (in other words, the default set of information pieces published), and how to get the machine name from the LinuxCNC INI file. The latter is a minor detail, but I expect it would be useful in a setup with several machines available. I am hoping for feedback from the experienced LinuxCNC developers and users, to make the component even better before it can go into the mainland LinuxCNC code base. Since I started on the MQTT component, I came across another video from Kent VanderVelden where he combine LinuxCNC with a set of screen glasses controlled by a Raspberry Pi, and it occured to me that it would be useful for such use cases if LinuxCNC also provided a REST API for querying its status. I hope to start on such component once the MQTT component is working well. As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Welcome to yet another report from the Reproducible Builds project, this time for November 2022. In all of these reports (which we have been publishing regularly since May 2015) we attempt to outline the most important things that we have been up to over the past month. As always, if you interested in contributing to the project, please visit our Contribute page on our website.

Reproducible Builds Summit 2022 Following-up from last month s report about our recent summit in Venice, Italy, a comprehensive report from the meeting has not been finalised yet watch this space! As a very small preview, however, we can link to several issues that were filed about the website during the summit (#38, #39, #40, #41, #42, #43, etc.) and collectively learned about Software Bill of Materials (SBOM) s and how `.buildinfo` files can be seen/used as SBOMs. And, no less importantly, the Reproducible Builds t-shirt design has been updated

Reproducible Builds at European Cyber Week 2022 During the European Cyber Week 2022, a Capture The Flag (CTF) cybersecurity challenge was created by Fr d ric Pierret on the subject of Reproducible Builds. The challenge consisted in a pedagogical sense based on how to make a software release reproducible. To progress through the challenge issues that affect the reproducibility of build (such as build path, timestamps, file ordering, etc.) were to be fixed in steps in order to get the final flag in order to win the challenge. At the end of the competition, five people succeeded in solving the challenge, all of whom were awarded with a shirt. Fr d ric Pierret intends to create similar challenge in the form of a how to in the Reproducible Builds documentation, but two of the 2022 winners are shown here:

On business adoption and use of reproducible builds Simon Butler announced on the rb-general mailing list that the Software Quality Journal published an article called On business adoption and use of reproducible builds for open and closed source software. This article is an interview-based study which focuses on the adoption and uses of Reproducible Builds in industry, with a focus on investigating the reasons why organisations might not have adopted them:
[ ] industry application of R-Bs appears limited, and we seek to understand whether awareness is low or if significant technical and business reasons prevent wider adoption.
This is achieved through interviews with software practitioners and business managers, and touches on both the business and technical reasons supporting the adoption (or not) of Reproducible Builds. The article also begins with an excellent explanation and literature review, and even introduces a new helpful analogy for reproducible builds:
[Users are] able to perform a bitwise comparison of the two binaries to verify that they are identical and that the distributed binary is indeed built from the source code in the way the provider claims. Applied in this manner, R-Bs function as a canary, a mechanism that indicates when something might be wrong, and offer an improvement in security over running unverified binaries on computer systems.
The full paper is available to download on an open access basis. Elsewhere in academia, Beatriz Michelson Reichert and Rafael R. Obelheiro have published a paper proposing a systematic threat model for a generic software development pipeline identifying possible mitigations for each threat (PDF). Under the Tampering rubric of their paper, various attacks against Continuous Integration (CI) processes:
An attacker may insert a backdoor into a CI or build tool and thus introduce vulnerabilities into the software (resulting in an improper build). To avoid this threat, it is the developer s responsibility to take due care when making use of third-party build tools. Tampered compilers can be mitigated using diversity, as in the diverse double compiling (DDC) technique. Reproducible builds, a recent research topic, can also provide mitigation for this problem. (PDF)

Misc news

A change was proposed for the Go programming language to enable reproducible builds when Link Time Optimisation (LTO) is enabled. As mentioned in the changelog, Morten Linderud s patch fixes two issues when the linker used in conjunction with the `-flto` option: the first involves solving an issue related to seeded random numbers; and the second involved the binary embedding the current working directory in compressed sections of the LTO object. Both of these issues made the build unreproducible.

In the .NET framework ecosystem, a wiki page for the Roslyn .NET C# and Visual Basic compiler was uncovered this month that details its attempts to ensure end-to-end reproducible builds by focusing on the definition on what are considered inputs to the compiler for the purpose of determinism . This is a spiritual followup to a 2016 blog post by Microsoft developer Jared Parsons on Deterministic builds in Roslyn which starts: It seems silly to celebrate features which should have been there from the start.

Ian Lance Taylor followed up an old post to report that Jakub Jelinek s patch from September 2000 is incomplete.

In F-Droid this month, Reproducible Builds contributor FC Stegerman created a set of reproducible APK tools as a workaround for issues like the order of files in APKs built on macOS being non-deterministic. In addition, the new issue documenting the overview of apps using reproducible builds shows that F-Droid added 11 new apps that use reproducible builds, and FC Stegerman released apksigcopier version 1.1.0 which adds support for APKs signed by Signflinger .

martinSusz has written up a fascinating wiki page describing how to generate quasi-reproducible firmware ROMs for System-on-a-Chip (SoC) components fabricated by Rock Chip. These chips are used in popular low-cost laptops such as the Pine64 PinebookPro and Asus C201. The link is worth viewing simply for the interesting diagram.

Our monthly IRC meeting was held on November 29th 2022. Our next meeting will be on January 31st 2023; we ll skip the meeting in December due to the proximity to Christmas, etc.

On our mailing list this month:

Adrian Diglio from Microsoft asked How to Add a New Project within Reproducible Builds which solicited a number of replies.

Vagrant Cascadian posed an interesting question regarding the difference between test builds vs rebuilds (or verification rebuilds ). As Vagrant poses in their message, they re both useful for slightly different purposes, and it might be good to clarify the distinction [ ].

Debian & other Linux distributions Over 50 reviews of Debian packages were added this month, another 48 were updated and almost 30 were removed, all of which adds to our knowledge about identified issues. Two new issue types were added as well. [ ][ ]. Vagrant Cascadian announced on our mailing list another online sprint to help clear the huge backlog of reproducible builds patches submitted by performing NMUs (Non-Maintainer Uploads). The first such sprint took place on September 22nd, but others were held on October 6th and October 20th. There were two additional sprints that occurred in November, however, which resulted in the following progress:

Chris Lamb:

`paxctl` (Fixed #1020804)

`png23d` (Fixed #1020805)

`tuxcmd-modules` (Fixed #1011500 & #941296)

`waili` (Fixed #1020751)

`zephyr` (Fixed #828867 #1021374)

Vagrant Cascadian:

`ddd` (Fixed #834016)

`libpam-ldap` (Fixed #834050)

`nsnake` (Fixed #833612)

`quvi` (Fixed #835259)

`stressapptest` (Fixed #831587 & #986653)

`tcpreen` (Fixed #831585)

`boolector` (Fixed #1023886)

`tsdecrypt` (Fixed #829713 & #1022130)

`wbxml2` (QA upload fixed build path issues)

`tercpp` (QA upload fixed build path issues)

Lastly, Roland Clobus posted his latest update of the status of reproducible Debian ISO images on our mailing list. This reports that all major desktops build reproducibly with bullseye, bookworm and sid as well as that no custom patches needed to applied to Debian unstable for this result to occur. During November, however, Roland proposed some modifications to live-setup and the rebuild script has been adjusted to fix the failing Jenkins tests for Debian bullseye [ ][ ].
In other news, Miro Hron ok proposed a change to clamp build modification times to the value of `SOURCE_DATE_EPOCH`. This was initially suggested and discussed on a `devel@` mailing list post but was later written up on the Fedora Wiki as well as being officially proposed to Fedora Engineering Steering Committee (FESCo).

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann:

`dwz` (Profile-guided optimisation issue)

`icmake` (filesystem ordering issue)

`llmnrd`

`elixir` (report a bug re. stuck build on single-core VMs)

`warzone2100` (report a bug re. parallelism-dependent output)

Chris Lamb:

#1023589 filed against `libnvme`.

#1024352 filed against `pykafka`.

Vagrant Cascadian:

#1023886 filed against `boolector`.

#1023956 filed against `fl-cow`.

#1023957 filed against `gerstensaft`.

#1023960 filed against `libcgicc`.

#1024007 filed against `haskell98-report`.

#1024125 filed against `ucspi-proxy`.

#1024126 filed against `hunt`.

#1024279 filed against `tolua++`.

#1024282 filed against `twoftpd`.

#1024283 filed against `ipsvd`.

#1024284 filed against `gentoo`.

#1024286 filed against `lcm`.

#1024288 filed against `apcupsd`.

#1024289 filed against `openfortivpn`.

#1024290 filed against `xtb`.

#1024291 filed against `gnunet`.

#1024292 filed against `swift-im`.

#1024396 filed against `brewtarget`.

#1024399 filed against `xrprof`.

#1024404 filed against `gitlint`.

#1024412 filed against `claws-mail`.

#1024413 filed against `presage`.

#1024530 filed against `jh7100-bootloader-recovery`.

Victor Westerhuis:

#1024482 & #1024638 filed against `opencv`.

John Neffenger:

`tomcat` (Fixed Apache bug #66346)

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions `226` and `227` to Debian:

Support both `python3-progressbar` and `python3-progressbar2`, two modules providing the `progressbar` Python module. [ ]

Don t run Python decompiling tests on Python bytecode that `file(1)` cannot detect yet and Python 3.11 cannot unmarshal. (#1024335)

Don t attempt to attach text-only differences notice if there are no differences to begin with. (#1024171)

Make sure we recommend `apksigcopier`. [ ]

Tidy generation of `os_list`. [ ]

Make the code clearer around generating the Debian substvars . [ ]

Use our `assert_diff` helper in `test_lzip.py`. [ ]

Drop other copyright notices from `lzip.py` and `test_lzip.py`. [ ]

In addition to this, Christopher Baines added lzip support [ ], and FC Stegerman added an optimisation whereby we don t run `apktool` if no differences are detected before the signing block [ ].
A significant number of changes were made to the Reproducible Builds website and documentation this month, including Chris Lamb ensuring the openEuler logo is correctly visible with a white background [ ], FC Stegerman de-duplicated by email address to avoid listing some contributors twice [ ], Herv Boutemy added Apache Maven to the list of affiliated projects [ ] and boyska updated our Contribute page to remark that the Reproducible Builds presence on salsa.debian.org is not just the Git repository but is also for creating issues [ ][ ]. In addition to all this, however, Holger Levsen made the following changes:

Add a number of existing publications [ ][ ] and update metadata for some existing publications as well [ ].

Hide draft posts on the website homepage. [ ]

Add the Warpforge build tool as a participating project of the summit. [ ]

Clarify in the footer that we welcome patches to the website repository. [ ]

Testing framework The Reproducible Builds project operates a comprehensive testing framework at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In October, the following changes were made by Holger Levsen:

Improve the generation of meta package sets (used in grouping packages for reporting/statistical purposes) to treat Debian bookworm as equivalent to Debian unstable in this specific case [ ] and to parse the list of packages used in the Debian cloud images [ ][ ][ ].

Temporarily allow Frederic to `ssh(1)` into our snapshot server as the `jenkins` user. [ ]

Keep some reproducible jobs Jenkins logs much longer [ ] (later reverted).

Improve the node health checks to detect failures to update the Debian cloud image package set [ ][ ] and to improve prioritisation of some kernel warnings [ ].

Always echo any IRC output to Jenkins output as well. [ ]

Deal gracefully with problems related to processing the cloud image package set. [ ]

Finally, Roland Clobus continued his work on testing Live Debian images, including adding support for specifying the origin of the Debian installer [ ] and to warn when the image has unmet dependencies in the package list (e.g. due to a transition) [ ].
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. You can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mailing list: `rb-general@lists.reproducible-builds.org`

Well this is a mouthful. I recently worked on a neat hack called puppet-package-check. It is designed to warn about manually installed packages, to make sure "everything is in Puppet". But it turns out it can (probably?) dramatically decrease the bootstrap time of Puppet bootstrap when it needs to install a large number of packages.

Detecting manual packages On a cleanly filed workstation, it looks like this:

root@emma:/home/anarcat/bin# ./puppet-package-check -v
listing puppet packages...
listing apt packages...
loading apt cache...
0 unmanaged packages found

A messy workstation will look like this:

root@curie:/home/anarcat/bin# ./puppet-package-check -v
listing puppet packages...
listing apt packages...
loading apt cache...
288 unmanaged packages found
apparmor-utils beignet-opencl-icd bridge-utils clustershell cups-pk-helper davfs2 dconf-cli dconf-editor dconf-gsettings-backend ddccontrol ddrescueview debmake debootstrap decopy dict-devil dict-freedict-eng-fra dict-freedict-eng-spa dict-freedict-fra-eng dict-freedict-spa-eng diffoscope dnsdiag dropbear-initramfs ebtables efibootmgr elpa-lua-mode entr eog evince figlet file file-roller fio flac flex font-manager fonts-cantarell fonts-inconsolata fonts-ipafont-gothic fonts-ipafont-mincho fonts-liberation fonts-monoid fonts-monoid-tight fonts-noto fonts-powerline fonts-symbola freeipmi freetype2-demos ftp fwupd-amd64-signed gallery-dl gcc-arm-linux-gnueabihf gcolor3 gcp gdisk gdm3 gdu gedit gedit-plugins gettext-base git-debrebase gnome-boxes gnote gnupg2 golang-any golang-docker-credential-helpers golang-golang-x-tools grub-efi-amd64-signed gsettings-desktop-schemas gsfonts gstreamer1.0-libav gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly gstreamer1.0-pulseaudio gtypist gvfs-backends hackrf hashcat html2text httpie httping hugo humanfriendly iamerican-huge ibus ibus-gtk3 ibus-libpinyin ibus-pinyin im-config imediff img2pdf imv initramfs-tools input-utils installation-birthday internetarchive ipmitool iptables iptraf-ng jackd2 jupyter jupyter-nbextension-jupyter-js-widgets jupyter-qtconsole k3b kbtin kdialog keditbookmarks keepassxc kexec-tools keyboard-configuration kfind konsole krb5-locales kwin-x11 leiningen lightdm lintian linux-image-amd64 linux-perf lmodern lsb-base lvm2 lynx lz4json magic-wormhole mailscripts mailutils manuskript mat2 mate-notification-daemon mate-themes mime-support mktorrent mp3splt mpdris2 msitools mtp-tools mtree-netbsd mupdf nautilus nautilus-sendto ncal nd ndisc6 neomutt net-tools nethogs nghttp2-client nocache npm2deb ntfs-3g ntpdate nvme-cli nwipe obs-studio okular-extra-backends openstack-clients openstack-pkg-tools paprefs pass-extension-audit pcmanfm pdf-presenter-console pdf2svg percol pipenv playerctl plymouth plymouth-themes popularity-contest progress prometheus-node-exporter psensor pubpaste pulseaudio python3-ldap qjackctl qpdfview qrencode r-cran-ggplot2 r-cran-reshape2 rake restic rhash rpl rpm2cpio rs ruby ruby-dev ruby-feedparser ruby-magic ruby-mocha ruby-ronn rygel-playbin rygel-tracker s-tui sanoid saytime scrcpy scrcpy-server screenfetch scrot sdate sddm seahorse shim-signed sigil smartmontools smem smplayer sng sound-juicer sound-theme-freedesktop spectre-meltdown-checker sq ssh-audit sshuttle stress-ng strongswan strongswan-swanctl syncthing system-config-printer system-config-printer-common system-config-printer-udev systemd-bootchart systemd-container tardiff task-desktop task-english task-ssh-server tasksel tellico texinfo texlive-fonts-extra texlive-lang-cyrillic texlive-lang-french texlive-lang-german texlive-lang-italian texlive-xetex tftp-hpa thunar-archive-plugin tidy tikzit tint2 tintin++ tipa tpm2-tools traceroute tree trocla ucf udisks2 unifont unrar-free upower usbguard uuid-runtime vagrant-cachier vagrant-libvirt virt-manager vmtouch vorbis-tools w3m wamerican wamerican-huge wfrench whipper whohas wireshark xapian-tools xclip xdg-user-dirs-gtk xlax xmlto xsensors xserver-xorg xsltproc xxd xz-utils yubioath-desktop zathura zathura-pdf-poppler zenity zfs-dkms zfs-initramfs zfsutils-linux zip zlib1g zlib1g-dev
157 old: apparmor-utils clustershell davfs2 dconf-cli dconf-editor ddccontrol ddrescueview decopy dnsdiag ebtables efibootmgr elpa-lua-mode entr figlet file-roller fio flac flex font-manager freetype2-demos ftp gallery-dl gcc-arm-linux-gnueabihf gcolor3 gcp gdu gedit git-debrebase gnote golang-docker-credential-helpers golang-golang-x-tools gtypist hackrf hashcat html2text httpie httping hugo humanfriendly iamerican-huge ibus ibus-pinyin imediff input-utils internetarchive ipmitool iptraf-ng jackd2 jupyter-qtconsole k3b kbtin kdialog keditbookmarks keepassxc kexec-tools kfind konsole leiningen lightdm lynx lz4json magic-wormhole manuskript mat2 mate-notification-daemon mktorrent mp3splt msitools mtp-tools mtree-netbsd nautilus nautilus-sendto nd ndisc6 neomutt net-tools nethogs nghttp2-client nocache ntpdate nwipe obs-studio openstack-pkg-tools paprefs pass-extension-audit pcmanfm pdf-presenter-console pdf2svg percol pipenv playerctl qjackctl qpdfview qrencode r-cran-ggplot2 r-cran-reshape2 rake restic rhash rpl rpm2cpio rs ruby-feedparser ruby-magic ruby-mocha ruby-ronn s-tui saytime scrcpy screenfetch scrot sdate seahorse shim-signed sigil smem smplayer sng sound-juicer spectre-meltdown-checker sq ssh-audit sshuttle stress-ng system-config-printer system-config-printer-common tardiff tasksel tellico texlive-lang-cyrillic texlive-lang-french tftp-hpa tikzit tint2 tintin++ tpm2-tools traceroute tree unrar-free vagrant-cachier vagrant-libvirt vmtouch vorbis-tools w3m wamerican wamerican-huge wfrench whipper whohas xdg-user-dirs-gtk xlax xmlto xsensors xxd yubioath-desktop zenity zip
131 new: beignet-opencl-icd bridge-utils cups-pk-helper dconf-gsettings-backend debmake debootstrap dict-devil dict-freedict-eng-fra dict-freedict-eng-spa dict-freedict-fra-eng dict-freedict-spa-eng diffoscope dropbear-initramfs eog evince file fonts-cantarell fonts-inconsolata fonts-ipafont-gothic fonts-ipafont-mincho fonts-liberation fonts-monoid fonts-monoid-tight fonts-noto fonts-powerline fonts-symbola freeipmi fwupd-amd64-signed gdisk gdm3 gedit-plugins gettext-base gnome-boxes gnupg2 golang-any grub-efi-amd64-signed gsettings-desktop-schemas gsfonts gstreamer1.0-libav gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly gstreamer1.0-pulseaudio gvfs-backends ibus-gtk3 ibus-libpinyin im-config img2pdf imv initramfs-tools installation-birthday iptables jupyter jupyter-nbextension-jupyter-js-widgets keyboard-configuration krb5-locales kwin-x11 lintian linux-image-amd64 linux-perf lmodern lsb-base lvm2 mailscripts mailutils mate-themes mime-support mpdris2 mupdf ncal npm2deb ntfs-3g nvme-cli okular-extra-backends openstack-clients plymouth plymouth-themes popularity-contest progress prometheus-node-exporter psensor pubpaste pulseaudio python3-ldap ruby ruby-dev rygel-playbin rygel-tracker sanoid scrcpy-server sddm smartmontools sound-theme-freedesktop strongswan strongswan-swanctl syncthing system-config-printer-udev systemd-bootchart systemd-container task-desktop task-english task-ssh-server texinfo texlive-fonts-extra texlive-lang-german texlive-lang-italian texlive-xetex thunar-archive-plugin tidy tipa trocla ucf udisks2 unifont upower usbguard uuid-runtime virt-manager wireshark xapian-tools xclip xserver-xorg xsltproc xz-utils zathura zathura-pdf-poppler zfs-dkms zfs-initramfs zfsutils-linux zlib1g zlib1g-dev

Yuck! That's a lot of shit to go through. Notice how the packages get sorted between "old" and "new" packages. This is because popcon is used as a tool to mark which packages are "old". If you have unmanaged packages, the "old" ones are likely things that you can uninstall, for example. If you don't have popcon installed, you'll also get this warning:

popcon stats not available: [Errno 2] No such file or directory: '/var/log/popularity-contest'

The error can otherwise be safely ignored, but you won't get "help" prioritizing the packages to add to your manifests. Note that the tool ignores packages that were "marked" (see apt-mark(8)) as automatically installed. This implies that you might have to do a little bit of cleanup the first time you run this, as Debian doesn't necessarily mark all of those packages correctly on first install. For example, here's how it looks like on a clean install, after Puppet ran:

root@angela:/home/anarcat# ./bin/puppet-package-check -v
listing puppet packages...
listing apt packages...
loading apt cache...
127 unmanaged packages found
ca-certificates console-setup cryptsetup-initramfs dbus file gcc-12-base gettext-base grub-common grub-efi-amd64 i3lock initramfs-tools iw keyboard-configuration krb5-locales laptop-detect libacl1 libapparmor1 libapt-pkg6.0 libargon2-1 libattr1 libaudit-common libaudit1 libblkid1 libbpf0 libbsd0 libbz2-1.0 libc6 libcap-ng0 libcap2 libcap2-bin libcom-err2 libcrypt1 libcryptsetup12 libdb5.3 libdebconfclient0 libdevmapper1.02.1 libedit2 libelf1 libext2fs2 libfdisk1 libffi8 libgcc-s1 libgcrypt20 libgmp10 libgnutls30 libgpg-error0 libgssapi-krb5-2 libhogweed6 libidn2-0 libip4tc2 libiw30 libjansson4 libjson-c5 libk5crypto3 libkeyutils1 libkmod2 libkrb5-3 libkrb5support0 liblocale-gettext-perl liblockfile-bin liblz4-1 liblzma5 libmd0 libmnl0 libmount1 libncurses6 libncursesw6 libnettle8 libnewt0.52 libnftables1 libnftnl11 libnl-3-200 libnl-genl-3-200 libnl-route-3-200 libnss-systemd libp11-kit0 libpam-systemd libpam0g libpcre2-8-0 libpcre3 libpcsclite1 libpopt0 libprocps8 libreadline8 libselinux1 libsemanage-common libsemanage2 libsepol2 libslang2 libsmartcols1 libss2 libssl1.1 libssl3 libstdc++6 libsystemd-shared libsystemd0 libtasn1-6 libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl libtinfo6 libtirpc-common libtirpc3 libudev1 libunistring2 libuuid1 libxtables12 libxxhash0 libzstd1 linux-image-amd64 logsave lsb-base lvm2 media-types mlocate ncurses-term pass-extension-otp puppet python3-reportbug shim-signed tasksel ucf usr-is-merged util-linux-extra wpasupplicant xorg zlib1g
popcon stats not available: [Errno 2] No such file or directory: '/var/log/popularity-contest'

Normally, there should be unmanaged packages here. But because of the way Debian is installed, a lot of libraries and some core packages are marked as manually installed, and are of course not managed through Puppet. There are two solutions to this problem:

really manage everything in Puppet (argh)
mark packages as automatically installed

I typically chose the second path and mark a ton of stuff as automatic. Then either they will be auto-removed, or will stop being listed. In the above scenario, one could mark all libraries as automatically installed with:

apt-mark auto $(./bin/puppet-package-check   grep -o 'lib[^ ]*')

... but if you trust that most of that stuff is actually garbage that you don't really want installed anyways, you could just mark it all as automatically installed:

apt-mark auto $(./bin/puppet-package-check)

In my case, that ended up keeping basically all libraries (because of course they're installed for some reason) and auto-removing this:

dh-dkms discover-data dkms libdiscover2 libjsoncpp25 libssl1.1 linux-headers-amd64 mlocate pass-extension-otp pass-otp plocate x11-apps x11-session-utils xinit xorg

You'll notice xorg in there: yep, that's bad. Not what I wanted. But for some reason, on other workstations, I did not actually have xorg installed. Turns out having xserver-xorg is enough, and that one has dependencies. So now I guess I just learned to stop worrying and live without X(org).

Optimizing large package installs But that, of course, is not all. Why make things simple when you can have an unreadable title that is trying to be both syntactically correct and click-baity enough to flatter my vain ego? Right. One of the challenges in bootstrapping Puppet with large package lists is that it's slow. Puppet lists packages as individual resources and will basically run apt install $PKG on every package in the manifest, one at a time. While the overhead of apt is generally small, when you add things like apt-listbugs, apt-listchanges, needrestart, triggers and so on, it can take forever setting up a new host. So for initial installs, it can actually makes sense to skip the queue and just install everything in one big batch. And because the above tool inspects the packages installed by Puppet, you can run it against a catalog and have a full lists of all the packages Puppet would install, even before I even had Puppet running. So when reinstalling my laptop, I basically did this:

apt install puppet-agent/experimental
puppet agent --test --noop
apt install $(./puppet-package-check --debug \
    2>&1   grep ^puppet\ packages 
      sed 's/puppet packages://;s/ /\n/g'
      grep -v -e onionshare -e golint -e git-sizer -e github-backup -e hledger -e xsane -e audacity -e chirp -e elpa-flycheck -e elpa-lsp-ui -e yubikey-manager -e git-annex -e hopenpgp-tools -e puppet
) puppet-agent/experimental

That massive grep was because there are currently a lot of packages missing from bookworm. Those are all packages that I have in my catalog but that still haven't made it to bookworm. Sad, I know. I eventually worked around that by adding bullseye sources so that the Puppet manifest actually ran. The point here is that this improves the Puppet run time a lot. All packages get installed at once, and you get a nice progress bar. Then you actually run Puppet to deploy configurations and all the other goodies:

puppet agent --test

I wish I could tell you how much faster that ran. I don't know, and I will not go through a full reinstall just to please your curiosity. The only hard number I have is that it installed 444 packages (which exploded in 10,191 packages with dependencies) in a mere 10 minutes. That might also be with the packages already downloaded. In any case, I have that gut feeling it's faster, so you'll have to just trust my gut. It is, after all, much more important than you might think.

Similar work The blueprint system is something similar to this:

It figures out what you ve done manually, stores it locally in a Git repository, generates code that s able to recreate your efforts, and helps you deploy those changes to production

That tool has unfortunately been abandoned for a decade at this point. Also note that the AutoRemove::RecommendsImportant and AutoRemove::SuggestsImportant are relevant here. If it is set to true (the default), a package will not be removed if it is (respectively) a Recommends or Suggests of another package (as opposed to the normal Depends). In other words, if you want to also auto-remove packages that are only Suggests, you would, for example, add this to apt.conf:

AutoRemove::SuggestsImportant false;

Paul Wise has tried to make the Debian installer and debootstrap properly mark packages as automatically installed in the past, but his bug reports were rejected. The other suggestions in this section are also from Paul, thanks!

Search Results: "ewt"

8 November 2024

Debian Contributions: 2024-10 Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

3 July 2024

1 July 2024

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes gt: fix manual page Debian wiki pages: CopyrightReviewTools, MassRebuilds, PortsDocs/New, Teams/DPL/New

Issues Conffile removal in apparmor Features in KDE, rss-blogroll-network Warnings in python3-azure, python3-commonmark-bkrs

Communication Respond to queries from Debian users and contributors on IRC

Sponsors All work was done on a volunteer basis.

16 May 2024

Conclusions There are a lot of good options for email hosting today, and in particular I appreciate the excellent personal support from companies like Migadu and Runbox. Support small businesses!

14 May 2024

6 December 2023

1 December 2023

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Issues Files in wrong dir in git-cola Conffile removal needed in dkms New SWH listers needed for HGKeeper, Smithy, humungus, git-arr, gitile, gotweb, ABF, Bonobo Git Server, darcsden

Administration Debian servers: extract user data from recent wiki backups Debian wiki: fix broken user account, approve accounts

Communication Respond to queries from Debian users and contributors on the mailing lists and IRC.

Sponsors The SWH work was sponsored. All other work was done on a volunteer basis.

31 October 2023

19 October 2023

26 September 2023

Fixing audio I ran the script mentioned here and that fixed the audio. The instructions from that link are: git clone https://github.com/WeirdTreeThing/chromebook-linux-audio cd chromebook-linux-audio ./setup-audio

25 September 2023

9 September 2023

17 July 2023

19 June 2023

14 June 2023

26 January 2023

8 January 2023

8 December 2022

29 September 2022

Changes

gt: fix manual page

Debian wiki pages: CopyrightReviewTools, MassRebuilds, PortsDocs/New, Teams/DPL/New

Issues

Conffile removal in apparmor

Features in KDE, rss-blogroll-network

Warnings in python3-azure, python3-commonmark-bkrs

Communication

Respond to queries from Debian users and contributors on IRC

Conclusions
There are a lot of good options for email hosting today, and in particular I appreciate the excellent personal support from companies like Migadu and Runbox. Support small businesses!

Issues

Files in wrong dir in git-cola

Conffile removal needed in dkms

New SWH listers needed for HGKeeper, Smithy, humungus, git-arr, gitile, gotweb, ABF, Bonobo Git Server, darcsden

Administration

Debian servers: extract user data from recent wiki backups

Debian wiki: fix broken user account, approve accounts

Communication

Respond to queries from Debian users and contributors on the mailing lists and IRC.

Fixing audio I ran the script mentioned here and that fixed the audio. The instructions from that link are:
`git clone https://github.com/WeirdTreeThing/chromebook-linux-audio cd chromebook-linux-audio ./setup-audio`