Combining BGP confederations and AS override can potentially
create a BGP routing loop, resulting in an indefinitely expanding AS path.
BGP confederation is a technique used to reduce the number of iBGP sessions
and improve scalability in large autonomous systems (AS). It divides an AS into
sub-ASes. Most eBGP rules apply between sub-ASes, except that next-hop, MED, and
local preferences remain unchanged. The AS path length ignores contributions
from confederation sub-ASes. BGP confederation is rarely used and BGP route
reflection is typically preferred for scaling.
AS override is a feature that allows a router to replace the ASN of a
neighbor in the AS path of outgoing BGP routes with its own. It s useful when
two distinct autonomous systems share the same ASN. However, it interferes with
BGP s loop prevention mechanism and should be used cautiously. A safer
alternative is the allowas-in directive.1
In the example below, we have four routers in a single confederation, each in
its own sub-AS. R0 originates the 2001:db8::1/128 prefix. R1, R2, and
R3 forward this prefix to the next router in the loop.
BGP routing loop using a confederation
The router configurations are available in a Git repository. They are
running Cisco IOS XR. R2 uses the following configuration for BGP:
The session with R3 uses both as-override and next-hop-self directives.
The latter is only necessary to make the announced prefix valid, as there is no
IGP in this example.2
Here s the sequence of events leading to an infinite AS path:
R1 selects it as the best path, forwarding it to R2 with AS
path (64501 64500).
R2 selects it as the best path, forwarding it to R3 with AS
path (64500 64501 64502).
R3 selects it as the best path. It would forward it to R1 with AS path
(64503 64502 64501 64500), but due to AS override, it substitutes R1 s
ASN with its own, forwarding it with AS path (64503 64502 64503 64500).
R1 accepts the prefix, as its own ASN is not in the AS path. It compares
this new prefix with the one from R0. Both (64500) and (64503 64502
64503 64500) have the same length because confederation sub-ASes don t
contribute to AS path length. The first tie-breaker is the router ID.
R0 s router ID (1.0.0.4) is higher than R3 s (1.0.0.3). The new
prefix becomes the best path and is forwarded to R2 with AS path (64501
64503 64501 64503 64500).
R2 receives the new prefix, replacing the old one. It selects it as the
best path and forwards it to R3 with AS path (64502 64501 64502 64501
64502 64500).
R3 receives the new prefix, replacing the old one. It selects it as the
best path and forwards it to R0 with AS path (64503 64502 64503 64502
64503 64502 64500).
R1 receives the new prefix, replacing the old one. Again, it competes with
the prefix from R0, and again the new prefix wins due to the lower router
ID. The prefix is forwarded to R2 with AS path (64501 64503 64501 64503
64501 64503 64501 64500).
A few iterations later, R1 views the looping prefix as follows:4
RP/0/RP0/CPU0:R1#showbgpipv6u2001:db8::1/128bestpath-compare
BGP routing table entry for 2001:db8::1/128Last Modified: Jul 28 10:23:05.560 for 00:00:00Paths: (2 available, best #2) Path #1: Received by speaker 0 Not advertised to any peer (64500) 2001:db8::1:0 from 2001:db8::1:0 (1.0.0.4), if-handle 0x00000000 Origin IGP, metric 0, localpref 100, valid, confed-external Received Path ID 0, Local Path ID 0, version 0 Higher router ID than best path (path #2) Path #2: Received by speaker 0 Advertised IPv6 Unicast paths to peers (in unique update groups): 2001:db8::2:1 (64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64500) 2001:db8::4:0 from 2001:db8::4:0 (1.0.0.3), if-handle 0x00000000 Origin IGP, metric 0, localpref 100, valid, confed-external, best, group-best Received Path ID 0, Local Path ID 1, version 37 best of AS 64503, Overall best
There s no upper bound for an AS path, but BGP messages have size limits (4096
bytes per RFC 4271 or 65535 bytes per RFC 8654). At some
point, BGP updates can t be generated. On Cisco IOS XR, the BGP process crashes
well before reaching this limit.5
The main lessons from this tale are:
never use BGP confederations under any circumstances, and
be cautious of features that weaken BGP routing loop detection.
When using BGP confederations with Cisco IOS XR, use
allowconfedas-in instead. It s available since IOS XR 7.11.
Using BGP confederations is already inadvisable. If you don t use the
same IGP for all sub-ASes, you re inviting trouble! However, the scenario
described here is also possible with an IGP.
When an AS path segment is composed of ASNs from a confederation, it
is displayed between parentheses.
By default, IOS XR paces eBGP updates. This is controlled by the
advertisement-interval directive. Its default value is 30 seconds for eBGP
peers (even in the same confederation). R1 and R2 set this value to 0,
while R3 sets it to 2 seconds. This gives some time to watch the AS path
grow.
This is CSCwk15887. It only happens when using as-override on an
AS path with a too long AS_CONFED_SEQUENCE. This should be fixed around
24.3.1.
YubiHSM packaging, by Colin Watson
Freexian is starting to use YubiHSM devices (hardware security modules)
as part of some projects, and we wanted to have the supporting software
directly in Debian rather than needing to use third-party repositories.
Since Yubico publish everything we need under free software licences,
Colin packaged yubihsm-connector, yubihsm-shell, and
python-yubihsm from
https://developers.yubico.com/, in
some cases based partly on the upstream packaging, and got them all into
Debian unstable.
Backports to bookworm will be forthcoming once they ve all reached
testing.
unschroot by Helmut Grohne
Following an in-person discussion at MiniDebConf Berlin, Helmut
attempted splitting the containment functionality of sbuild --chroot-mode=unshare into a dedicated tool interfacing with sbuild
as a variant of --chroot-mode=schroot providing a sufficiently
compatible
interface.
While this seemed technically promising initially, a discussion on
debian-devel
indicated a desire to rely on an existing container runtime such as
podman instead of using another Debian-specific tool with unclear long
term maintenance.
None of the existing container runtimes meet the specific needs of
sbuild, so further advancing this matter implies a compromise one way
or another.
Linux live-patching, by Santiago Ruano Rinc n
In collaboration with Emmanuel Arias, Santiago is
working on the development of linux live-patching for Debian.
For the moment, this is in an exploratory phase, that includes how to
handle the different patches that will need to be provided.
kpatch could help significantly
in this regard.
However, kpatch was removed from unstable because there are some RC bugs
affecting the version that was present in Debian unstable.
Santiago packaged the most recent upstream version
(0.9.9)
and filed an Intent to Salvage
bug.
Santiago is waiting for an ACK by the maintainer, and will upload to
unstable after July 10th, following the package salvaging rules.
While kpatch 0.9.9 fixes the main issues, it still needs some work to
properly support Debian and the Linux kernel versions packaged in our
distribution.
More on this in the report next month.
Salsa CI, by Santiago Ruano Rinc n
The work by Santiago in Salsa CI this month includes a merge
request
to ease testing how the production images are built from the changes
introduced by future merge requests.
By default, the pipelines triggered by a merge request build a subset of
the images built for production, to reduce the use of resources, and
because most of the time the subset of staging images is enough to test
the proposed modifications.
However, sometimes it is needed to test how the full set of production
images is built, and the above mentioned MR helps to do that.
The changes include documentation, so hopefully this will make it easier
to test future contributions.
Also, for being able to include support for RISC-V, Salsa CI needs to
replace
kaniko as
the tool used to build the images.
Santiago tested
buildah,
but there are some issues when pushing built images for non-default
platform architectures (i386, armhf, armel) to the container registry.
Santiago will continue to work on this to find a solution.
Miscellaneous contributions
Stefano Rivera prepared updates for a number of Python modules.
Stefano uploaded the latest point release of Python 3.12 and the
latest Python 3.13 beta.
Both uncovered upstream regressions that had to be addressed.
Stefano worked on preparations for DebConf 24.
Stefano helped SPI to reconcile their financial records for DebConf
23.
Colin did his usual routine work on the Python team, upgrading 36
packages to new upstream versions (including fixes for four CVEs in
python-aiohttp), fixing RC bugs in ipykernel, ipywidgets,
khard, and python-repoze.sphinx.autointerface, and packaging
zope.deferredimport which was needed for a new upstream version of
python-persistent.
Colin removed the user_readenv option from OpenSSH s PAM
configuration (#1018260), and
prepared a release note.
Thorsten Alteholz uploaded a new upstream version of cups.
Nicholas Skaggs updated xmacro to support reproducible builds
(#1014428),
DEP-3 and DEP-5 compatibility, along with utilizing hardening build
flags.
Helmut supported and uploaded package.
As a result of login having become non-essential, Helmut uploaded
debvm to unstable and stable and fixed a crossqa.debian.net worker.
Santiago worked on the Content Team activities for DebConf24.
Together with other DebConf25 team members, Santiago wrote a document
for the head of the venue to describe the project of the conference.
Armadillo is a powerful
and expressive C++ template library for linear algebra and scientific
computing. It aims towards a good balance between speed and ease of use,
has a syntax deliberately close to Matlab, and is useful for algorithm
development directly in C++, or quick conversion of research code into
production environments. RcppArmadillo
integrates this library with the R environment and language and is
widely used by (currently) 1158 other packages on CRAN, downloaded 35.1 million
times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint
/ vignette) by Conrad and myself has been cited 587 times according
to Google Scholar.
Conrad released a new
major upstream version 14.0.0 a couple of days ago. We had been testing
this new version extensively over several rounds of reverse-dependency
checks across all 1100+ packages. This revealed nine packages requiring
truly minor adjustments which eight maintainers made in a matter of
days; all this was coordinated in issue
#443. Following the upload, CRAN noticed one more issue (see
issue
#446) but this turned out to be local to the package. There are also
renewed deprecation warnings with some Armadillo changes which we will
need to address one-by-one. Last but not least with this release we also
changed the package versioning scheme to follow upstream Armadillo more closely.
The set of changes since the last CRAN release follows.
Changes in
RcppArmadillo version 14.0.0-1 (2024-07-05)
Upgraded to Armadillo release 14.0.0 (Stochastic Parrot)
C++14 is now the minimum recommended C++ standard
Faster handling of compound expressions by
as_scalar(), accu(),
dot()
Faster interactions between sparse and dense matrices
Expanded stddev() to handle sparse matrices
Expanded relational operators to handle expressions between
sparse matrices and scalars
Added .as_dense() to obtain dense vector/matrix
representation of any sparse matrix expression
Updated physical constants to NIST 2022 CODATA values
New package version numbering scheme following upstream
versions
Re-enabling ARMA_IGNORE_DEPRECATED_MARKE for silent
CRAN builds
Beyond Control is science fiction erotica (dystopian erotic
romance, per the marketing) and a direct sequel to
Beyond Shame. These books shift
protagonists with each volume and enough of the world background is
explained that you could start here, but there are significant spoilers
for the previous book. I read this book as part of the Beyond
Series Bundle (Books 1-3), which is what the sidebar information is for.
This is one of those reviews that I write because I'm stubborn about
reviewing all the books I read, not because it's likely to be useful to
anyone. There are also considerably more spoilers for the shape of the
story than I normally include, so be warned.
The Beyond series is erotica. Specifically, so far, consensual BDSM
erotica with bisexuality but otherwise typical gender stereotypes. The
authors (Kit Rocha is a pen name for Donna Herren and Bree Bridges) are
women, so it's more female gaze than male gaze, but by erotica I don't
mean romance with an above-average number of steamy scenes. I mean it
felt like half the book by page count was descriptions of sex.
This review is rather pointless because, one, I'm not going to review the
sex that's the main point of the book, and two, I skimmed all the sex and
read it for the story because I'm weird. Beyond Shame got me
interested in these absurdly horny people and their post-apocalyptic
survival struggles in the outskirts of a city run by a religious
surveillance state, and I wanted to find out what happened next. Besides,
this book promised to focus on my favorite character from the first novel,
Lex, and I wanted to read more about her.
Beyond Control uses a series pattern that I understand is common in
romance but which is not often seen in SFF (my usual genre): each book
focuses on a new couple adjacent to the previous couple, while the happily
ever after of the previous couple plays out in the background. In this
case, it also teases the protagonists of the next book. I can see why
romance uses this structure: it's an excuse to provide satisfying
interludes for the reader. In between Lex and Dallas's current
relationship problems, one gets to enjoy how well everything worked out
for Noelle and how much she's grown.
In Beyond Shame, Lex was the sort-of partner of Dallas O'Kane, the
leader of the street gang that is running Sector Four. (Picture a circle
surrounding the rich-people-only city of Eden. That circle is divided
into eight wedge-shaped sectors, which provide heavy industries,
black-market pleasures, and slums for agricultural workers.) Dallas is an
intensely possessive, personally charismatic semi-dictator who cultivates
the image of a dangerous barbarian to everyone outside and most of the
people inside Sector Four. Since he's supposed to be one of the good
guys, this is more image than reality, but it's not entirely disconnected
from reality.
This book is about Lex and Dallas forming an actual relationship, instead
of the fraught and complicated thing they had in the first book. I was
hoping that this would involve Dallas becoming less of an asshole. It
unfortunately does not, although some of what I attributed to malice may
be adequately explained by stupidity. I'm not sure that's an improvement.
Lex is great, just like she was in the first book. It's obvious by this
point in the series that she does most of the emotional labor of keeping
the gang running, and her support is central to Dallas's success. Like
most of the people in this story, she has a nasty and abusive background
that she's still dealing with in various ways. Dallas's possessiveness is
intensely appealing to her, but she wants that possessiveness on different
terms than Dallas may be willing to offer, or is even aware of. Lex was,
I thought, exceptionally clear about what she wanted out of this
relationship. Dallas thinks this is entirely about sex, and is, in
general, dumber than a sack of hammers. That means fights. Also orgies,
but, well, hopefully you knew what you were getting into if you picked up
this book.
I know, I know, it's erotica, that's the whole point, but these people
have a truly absurd amount of sex. Eden puts birth control in the water
supply, which is a neat way to simplify some of the in-story consequences
of erotica. They must be putting aphrodisiacs in the water supply as
well.
There was a lot of sector politics in this book that I found way more
interesting than it had any right to be. I really like most of these
people, even Dallas when he manages to get his three brain cells connected
for more than a few minutes. The events of the first book have a lot of
significant fallout, Lex continues being a badass, the social dynamics
between the women are very well-done (and pass the
Bechdel test yet again
even though this is mostly traditional-gender-role erotica), and if Dallas
had managed to understand what he did wrong at a deeper-than-emotional
level, I would have rather enjoyed the non-erotica story parts. Alas.
I therefore wouldn't recommend this book even if I were willing to offer
any recommendations about erotica (which I'm not). I was hoping it was
going somewhere more rewarding than it did. But I still kind of want to
read another one? I am weirdly fascinated with the lives of these people.
The next book is about Six, who has the potential to turn into the sort of
snarky, cynical character I love reading about. And it's not that hard to
skim over the orgies. Maybe Dallas will get one additional brain cell per
book?
Followed by Beyond Pain.
Rating: 5 out of 10
Debian packaging consists of a directory (debian/) containing
a number of "hard-coded" filenames such as debian/control,
debian/changelog, debian/copyright. In addition to these,
many packages will also use a number of optional files that are
named via a pattern such as debian/ PACKAGE .install.
At a high level the patterns looks deceptively simple. However,
if you start working on trying to automatically classify files
in debian/ (which could be helpful to tell the user they have
a typo in the filename), you will quickly realize these patterns
are not machine friendly at all for this purpose.
The patterns deconstructed
To appreciate the problem fully, here is a primer on the pattern
and all its issues. If you are already well-versed in these, you
might want to skip the section.
The most common patterns are debian/package.stem or debian/stem
and usually the go to example is the install stem (
a concrete example being debian/debhelper.install). However,
the full pattern consists of 4 parts where 3 of them are optional.
The package name followed by a period. Optional, but must be the
first if present.
The name segment followed by a period. Optional, but must appear
between the package name (if present) and the stem. If the package
name is not present, then the name segment must be first.
The stem. Mandatory.
An architecture restriction prefixed by a period. Optional, must
appear after the stem if present.
To visualize it with [foo] to mark optional parts, it looks like
debian/[PACKAGE.][NAME.]STEM[.ARCH]
Detecting whether a given file is in fact a packaging file now boils
down to reverse engineering its name against this pattern. Again, so
far, it might still look manageable. One major complication is that
every part (except ARCH) can contain periods. So a trivial
"split by period" is not going to cut it. As an example:
debian/g++-3.0.user.service
This example is deliberately crafted to be ambiguous and show this
problem in its full glory. This file name can be in multiple ways:
Is the stem service or user.service? (both are known
stems from dh_installsystemd and dh_installsystemduser
respectively). In fact, it can be both at the same time with
"clever" usage of --name=user passed to
dh_installsystemd.
The g++-3.0 can be a package prefix or part of the name
segment. Even if there is a g++-3.0 package in
debian/control, then debhelper (until compat 15) will
still happily match this file for the main package if you pass
--name=g++-3.0 to the helper. Side bar: Woe is you if there
is a g++-3 and a g++-3.0 package in
debian/control, then we have multiple options for the package
prefix! Though, I do not think that happens in practice.
Therefore, there are a lot of possible ways to split this filename
that all matches the pattern but with vastly different meaning
and consequences.
Making detection practical
To make this detection practical, lets look at the first problems
that we need to solve.
We need the possible stems up front to have a chance at all.
When multiple stems are an option, go for the longest match
(that is, the one with most periods) since --name is
rare and "code golfing" is even rarer.
We can make the package prefix mandatory for files with the
name segment. This way, the moment there is something before
the stem, we know the package prefix will be part of it and
can cut it. It does not solve the ambiguity if one package
name is a prefix of another package name (from the same
source), but it still a lot better. This made its way
into debhelper compat 15 and now it is "just" a slow
long way to a better future.
A simple solution to the first problem could be to have a static
list of known stems. That will get you started but the debhelper
eco-system strive on decentralization, so this feels like a
mismatch.
There is also a second problem with the static list. Namely,
a given stem is only "valid" if the command in question is
actually in use. Which means you now need to dumpster dive
into the mess that is Turning-complete debhelper
configuration file known as debian/rules to fully solve
that. Thanks to the Turning-completeness, we will never
get a perfect solution for a static analysis.
Instead, it is time to back out and instead apply some
simplifications. Here is a sample flow:
Check whether the dh sequencer is used. If so, use
some heuristics to figure out which addons are used.
Delegate to dh_assistant to figure out which
commands will be used and which debhelper config file
stems it knows about. Here we need to know which sequences
are in use from step one (if relevant). Combine this
with any other sources for stems you have.
Deconstruct all files in debian/ against the stems
and known package names from debian/control. In
theory, dumpster diving after --name options would
be helpful here, but personally I skipped that part as
I want to keep my debian/rules parsing to an
absolute minimum.
With this logic, you can now:
Provide typo detection of the stem (debian/foo.intsall -> debian/foo.install)
provided to have adequate handling of the corner cases (such as debian/*.conf
not needing correction into debian/*.config)
Detect possible invalid package prefix (debian/foo.install without foo
being a package). Note this has to be a weak warning unless the package is using
debhelper compat 15 or you dumpster dived to validate that dh_install was not
passed dh_install --name foo. Agreed, no one should do that, but they can and
false positives are the worst kind of positives for a linting tool.
With some limitations, detect files used without the relevant command being
active. As an example, the some integration modes of debputy removes
dh_install, so a debian/foo.install would not be used.
Associate a given file with a given command to assist users with the
documentation look up. Like debian/foo.user.service is related to
dh_installsystemduser, so man dh_installsystemduser is a natural
start for documentation.
I have added the logic for all these features in debputy though the documentation
association is currently not in a user facing command. All the others are now
diagnostics emitted by debputy in its editor support mode (debputy lsp server)
or via debputy lint. In the editor mode, the diagnostics are currently associated
with the package name in debian/control due to technical limitations of how the
editor integration works.
Some of these features will the latest version of debhelper (moving target
at times). Check with debputy lsp features for the Extra dh support feature,
which will be enabled if you got all you need.
Note: The detection is currently (mostly) ignoring files with architecture
restrictions. That might be lifted in the future. However, architecture restricted
config files tend to be rare, so they were not a priority at this point. Additionally,
debputy for technical reasons ignores stem typos with multiple matches. That
sadly means that typos of debian/docs will often be unreported due to its
proximity to debian/dirs and vice versa.
Diving a bit deeper on getting the stems
To get the stems, debputy has 3 primary sources:
Its own plugins can provide packager provided files. These are only relevant
if the package is using debputy.
It is als possible to provide a debputy plugin that identifies packaging
files (either static or named ones). Though in practice, we probably do not
want people to roll their own debputy plugin for this purpose, since the
detection only works if the plugin is installed. I have used this mechanism
to have debhelper provide a debhelper-documentation plugin to enrich
the auto-detected data and we can assume most people interested in this
feature would have debhelper installed.
It asks dh_assistant list-guessed-dh-config-files for config files, which
is covered below.
The dh_assistant command uses the same logic as dh to identify the active
add-ons and loads them. From there, it scans all commands mentioned in the sequence
for the PROMISE: DH NOOP WITHOUT ...-hint and a new
INTROSPECTABLE: CONFIG-FILES ...-hint. When these hints reference a packaging
file (as an example, via pkgfile(foo)) then dh_assistant records that
as a known packaging file for that helper.
Additionally, debhelper now also tracks commands that were removed from the
sequence. Several of the dh_assistant subcommand now use this to enrich
their (JSON) output with notes about these commands being known but not active.
The end result
With all of this work, you now get:
$ aptsatisfy'dh-debputy (>= 0.1.43~), debhelper (>= 13.16~), python3-lsprotocol, python3-levenshtein'# Fordemopurposes,pulltwoknownrepos(feelfreetouseyourownpackageshere)$ gitclonehttps://salsa.debian.org/debian/debhelper.git-bdebian/13.16
$ gitclonehttps://salsa.debian.org/debian/debputy.git-bdebian/0.1.43
$ cddebhelper
$ mvdebian/debhelper.installdebian/debhelper.intsall
$ debputylint
warning: File: debian/debhelper.intsall:1:0:1:0: The file "debian/debhelper.intsall" is likely a typo of "debian/debhelper.install" File-level diagnostic$ mvdebian/debhelper.intsalldebian/debhleper.install
$ debputylint
warning: File: debian/debhleper.install:1:0:1:0: Possible typo in "debian/debhleper.install". Consider renaming the file to "debian/debhelper.debhleper.install" or "debian/debhelper.install" if it is intended for debhelper File-level diagnostic$ cd../debputy
$ touchdebian/install
$ debputylint--no-warn-about-check-manifest
warning: File: debian/install:1:0:1:0: The file debian/install is related to a command that is not active in the dh sequence with the current addons File-level diagnostic
As mentioned, you also get these diagnostics in the editor via the
debputy lsp server feature. Here the diagnostics appear in
debian/control over the package name for technical reasons.
The editor side still needs a bit more work. Notably, changes to
the filename is not triggered automatically and will first be
caught on the next change to debian/control. Likewise,
changes to debian/rules to add --with to dh might
also have some limitations depending on the editor. Saving both
files and then triggering an edit of debian/control seems
to work reliable but ideally it should not be that involved.
The debhelper side could also do with some work to remove
the unnecessary support for the name segment with many file
stems that do not need them and announce that to
debputy.
Anyhow, it is still a vast improvement over the status quo
that was "Why is my file silently ignored!?".
Welcome to Boy.Net is a science fiction novel with cyberpunk vibes,
the first of a possible series.
Earth is a largely abandoned wasteland. Humanity has survived in the rest
of the solar system and spread from Earth's moon to the outer planets.
Mars is the power in the inner system, obsessed with all things Earth and
effectively run by the Earth Nations' Peacekeeping Force, the ENForcers.
An ENForcer soldier is raised in a creche from an early age, implanted
with cybernetic wetware and nanite enhancements, and extensively trained
to be an elite fighting unit. As befits a proper military, every ENForcer
is, of course, male.
The ENForcers thought Lucia Del Toro was a good, obedient soldier. They
also thought she was a man. They were wrong about those and many other
things. After her role in an atrocity that named her the Scourge of New
Shanghai, she went AWOL and stole her command ship. Now she and her
partner/girlfriend Hawk, a computer hacker from Luna, make a living with
bounty hunting jobs in the outer system. The ENForcers rarely cross the
asteroid belt; the United Miners see to that.
The appearance of an F-class ENForcer battle cruiser in Jupiter orbit is a
very unpleasant surprise. Lucia and Hawk hope it has nothing to do with
them. That hope is dashed when ENForcers turn up in the middle of their
next job: a bounty to retrieve an AI eye.
I first found Lyda Morehouse via her AngeLINK
cyberpunk series, the last of which was published in 2011. Since then,
she's been writing paranormal romance and urban fantasy as Tate Hallaway.
This return to science fiction is an adventure with trickster hackers,
throwback anime-based cowboy bars, tense confrontations with fascist
thugs, and unexpected mutual aid, but its core is a cyberpunk look at the
people who are unwilling or unable to follow the rules of social
conformity. Gender conformity, specifically.
Once you understand what this book is about, Welcome to Boy.Net is
a great title, but I'm not sure it serves its purpose as a marketing tool.
This is not the book that I would have expected from that title in
isolation, and I'm a bit worried that people who would like it might pass
it by.
Inside the story, Boy.Net is the slang term for the cybernetic network
that links all ENForcers. If this were the derogatory term used by people
outside the ENForcers, I could see it, but it's what the ENForcers
themselves call it. That left me with a few suspension of disbelief
problems, since the sort of macho assholes who are this obsessed with male
gender conformance usually consider "boys" to be derogatory and wouldn't
call their military cybernetic network something that sounds that
belittling, even as a joke. It would be named after some sort of
Orwellian reference to freedom, or something related to violence,
dominance, brutality, or some other "traditional male" virtue.
But although this term didn't work for me as world-building, it's a
beautiful touch thematically.
What Morehouse is doing here is the sort of concretized metaphor that
science fiction is so good at: an element of world-building that is both
an analogy for something the reader is familiar with and is also a
concrete piece of world background that follows believable rules and can
be manipulated by the characters. Boy.Net is trying to reconnect to Lucia
against her will. If it succeeds, it will treat the body modifications
she's made as damage and try to reverse all of them, attempting to convert
her back to the model of an ENForcer. But it is also a sharp metaphor for
how gender roles are enforced in our world: a child assigned male is
connected to a pervasive network of gender expectations and is programmed,
shaped, and monitored to match the social role of a boy. Even if they
reject those expectations, the gender role keeps trying to reconnect and
convert them back.
I really enjoyed Morehouse's handling of the gender dynamics. It's an
important part of the plot, but it's not the only thing going on or the
only thing the characters think about. Lucia is occasionally caught by
surprise by well-described gender euphoria, but mostly gender is something
other people keep trying to impose on her because they're obsessed with
forcing social conformity.
The rest of the book is a fun romp with a few memorable characters and a
couple of great moments with unexpected allies. Hawk and Lucia have an
imperfect but low drama relationship that features a great combination of
insight and the occasional misunderstanding. It's the kind of believable
human relationship that I don't see very much in science fiction, written
with the comfortable assurance of an author with over a dozen books under
her belt. Some of the supporting characters are also excellent, including
a non-binary deaf hacker that I wish had been a bit more central to the
story.
This is not the greatest science fiction novel I've read, but it was
entertaining throughout and kept me turning the pages. Recommended if you
want some solar-system cyberpunk in your life.
Welcome to Boy.Net reaches a conclusion of sorts, but there's an
obvious hook for a sequel and a lot of room left for more stories. I hope
enough people buy this book so that I can read it.
Rating: 7 out of 10
Here comes the 2nd article of the 5-episode blog post series written by Guido Berh rster, member of staff at my company Fre(i)e Software GmbH.Enjoy also this read on Guido's work on Polis,
Mike
Table of Contents of the Blog Post Series
Polis - Initial evaluation and adaptation
The Polis code base consists of a number of components, the administration
and participation interfaces, a common web backend, and a statistics processing
server. Both frontends and the backend are written in a mixture of JavaScript
and TypeScript, only the statistics processing server is written in Clojure.
In case of self hosting the preferred method of deployment is via Docker
containers using Docker Compose or any other orchestrator. The participation
frontend for conversations can either be used as a standalone web page or be
embedded via an iframe.
For our planned use case we initially defined the following goals:
custom branding and the integration into different content management systems
(CMS)
better support for mobile devices
mandatory authentication and support for a broader range of authentication
methods, including self-hosted solutions and DigiD
support for alternative email sending services
GDPR compliance
After a preliminary evaluation of our own and consulting with Policy Lab
UK who were also evaluating and testing Polis and had already made a
range of improvements related to self-hosting as well as bug fixes and
modernization changes we decided to take their work as a base for our
adaptations with the intent of submitting generally useful changes back to the
Polis project.
Subsequently, a number of changes were implemented, including the removal of
hardcoded domain names, the elimination of unnecessary cookies and third-party
requests, support for an alternative email sending service, and the option of
disabling Facebook and X integration.
For the branding our approach was to add an option allowing websites which are
embedding conversations in an iframe to load an alternative stylesheet for
overriding the native Polis branding. For this to be practical we intended to
use CSS custom properties for defining branding-related styles such as colors
and fonts. That approach turned out to be problematic because although the
Polis participation frontend stylesheet is generated via SCSS and some of the
colors are parameterized, however, they are not used consistently throughout the SCSS
stylesheets, unfortunately. In addition the frontend templates contain a large amount of
hardcoded style attributes. While we succeeded in implementing
user-defined stylesheets, it took a disproportionate amount of development
resources to parameterize all used colors and fonts via CSS custom properties
aggravated by the fact that the SCSS and template files are huge and contain
many unused rules and code.
derive-deftly, the template-based derive-macro facility for Rust, has been a great success.
It s coming up to time to declare a stable 1.x version. If you d like to try it out, and have final comments / observations, now is the time.
Introduction to derive-deftly
Have you ever wished that you could that could write a new derive macro without having to mess with procedural macros?
You can!
derive-deftly lets you write a #[derive] macro, using a template syntax which looks a lot like macro_rules!:
Status
derive-deftly has a wide range of features, which can be used to easily write sophisticated and reliable derive macros. We ve been using it in Arti, the Tor Project s reimplementation of Tor in Rust, and we ve found it very useful.
There is comprehensive reference documentation, and more discursive User Guide for a more gentle introduction. Naturally, everything is fully tested.
History
derive-deftly started out as a Tor Hackweek project. It used to be called derive-adhoc. But we renamed it because we found that many of the most interesting use cases were really not very ad-hoc at all.
Over the past months we ve been ticking off our 1.0 blocker tickets. We ve taken the opportunity to improve syntax, terminology, and semantics. We hope we have now made the last breaking changes.
Plans - call for review/testing
In the near future, we plan to declare version 1.0. After 1.x, we intend to make breaking changes very rarely.
So, right now, we d like last-minute feedback. Are there any wrinkles that need to be sorted out? Please file tickets or MRs on our gitlab. Ideally, anything which might imply breaking changes would be submitted on or before the 13th of August.
In the medium to long term, we have many ideas for how to make derive-deftly even more convenient, and even more powerful. But we are going to proceed cautiously, because we don t want to introduce bad syntax or bad features, which will require difficult decisions in the future about forward compatibility.
Bitter Angels is a stand-alone science fiction novel. It won the
Philip K. Dick award for best SF original paperback in 2010. C.L.
Anderson is a pen name for Sarah Zettel.
Terese was a Guardian, one of the agents of the Pax Solaris who find ways
to keep the peace in troubled systems and high-stress situations with the
help of an implanted Companion, an assistant AI. Forty years ago, on one
of those missions, she was captured and her Companion was forcibly
removed. She was rescued by her friend and mentor and retired afterwards,
starting a new life and a new family, trying to leave the memories behind.
Now, the woman who rescued her is dead. She was murdered on duty in the
Erasmus system, a corporate hellhole that appears to be on the verge of
exploding into a political hot spot. Bianca's last instructions asked for
Terese to replace her. Terese's family is furious at her for even
considering returning to the Guardians, but she can't say no. Duty, and
Bianca's dying request, call too strongly.
Amerand is Security on Dazzle, one of the Erasmus stations. He is one of
the refugees from Oblivion, the station that the First Bloods who rule the
system let die. He keeps his head above water and tries to protect his
father and find his mother without doing anything that the ever-present
Clerks might find concerning. Keeping an eye on newly-arriving Solaris
saints is a typical assignment, since the First Bloods don't trust the
meddling do-gooders. But something is not quite right, and a cryptic
warning from his Clerk makes him even more suspicious.
This is the second book by Sarah Zettel that
I've read, and both of them have been tense, claustrophobic thrillers set
in a world with harsh social inequality and little space for the
characters to maneuver. In this case, the structure of her future
universe reminded me a bit of Iain M. Banks's Culture, but with less
advanced technology and only humans. The Pax Solaris has eliminated war
within its borders and greatly extended lifespans. That peace is
maintained by Guardians, who play a role similar to Special Circumstances
but a bit more idealist and less lethal. They show up where there are
problems and meddle, manipulating and pushing to try to defuse the
problems before they reach the Pax Solaris.
Like a Culture novel, nearly all of the action takes place outside the Pax
Solaris in the Erasmus system. Erasmus is a corporate colony that has
turned into a cross between a hereditary dictatorship and the Corporate
Rim from Martha Wells's Murderbot series. Debt
slavery is ubiquitous, economic inequality is inconceivably vast, and the
Clerks are everywhere. Erasmus natives like Amerand have very little
leeway and even fewer options. Survival is a matter of not drawing the
attention of the wrong people. Terese and her fellow Guardians are
appalled, but also keenly aware that destabilizing the local politics may
make the situation even worse and get a lot of people killed.
Bitter Angels is structured like a mystery: who killed Bianca, and
what was her plan when she was killed? Unlike a lot of books of this
type, the villains are not idiots and their plan is both satisfyingly
complex and still depressingly relevant. I don't think I'm giving
anything away by saying that I have read recent news articles about people
with very similar plans, albeit involving less science-fiction technology.
Anderson starts with a tense situation and increases the pressure
relentlessly, leaving the heroes one step behind the villains for almost
the entire novel. It is not happy or optimistic reading at times, the
book is quite dark but it certainly was engrossing.
The one world-building quibble that I had is that the Erasmus system is
portrayed partly as a
hydraulic empire,
and while this is arguably feasible given that spaceship travel is
strictly controlled, it seemed like a weird choice given the prevalence of
water on the nearby moons. Water smuggling plays a significant role in
the plot, and I wasn't entirely convinced of the politics and logistics
behind it. If this sort of thing bugs you, there are some pieces that may
require suspension of disbelief.
Bitter Angels is the sort of tense thriller where catastrophe is
barely avoided and the cost of victory is too high, so you will want to be
in the mood for that before you dive in. But if that's what you're
looking for, I thought Anderson delivered a complex and satisfying story.
Content warning: major character suicide.
Rating: 7 out of 10
The final piece of my 5G (well, 4G) based based backup internet connection I needed to sort out was IPv6. While Three do support IPv6 in their network they only seem to enable it for certain devices, and the MC7010 is not one of those devices, even though it also supports IPv6.
I use v6 a lot - over 50% of my external traffic, last time I looked. One suggested option was that I could drop the IPv6 Router Advertisements when the main external link went down, but I have a number of internal services that are only presented on v6 addresses so I needed to ensure clients in the house continued to have access to those.
As it happens I ve used the Hurricane ElectricIPv6 Tunnel Broker in the past, so my pass was re-instating that. The 5G link has a real external IPv4 address, and it s possible to update the endpoint using a simple HTTP GET. I added the following to my /etc/dhcp/dhclient-exit-hooks.d/modem-interface-route where we are dealing with an interface IP change:
# Update IPv6 tunnel with Hurricane Electric
curl --interface $interface 'https://username:password@ipv4.tunnelbroker.net/nic/update?hostname=1234'
I needed some additional configuration to bring things up, so /etc/network/interfaces got the following, configuring the 6in4 tunnel as well as the low preference default route, and source routing via the 5g table, similar to IPv4:
pre-up ip tunnel add he-ipv6 mode sit remote 216.66.80.26
pre-up ip link set he-ipv6 up
pre-up ip addr add 2001:db8:1234::2/64 dev he-ipv6
pre-up ip -6 rule add from 2001:db8:1234::/64 lookup 5g
pre-up ip -6 route add default dev he-ipv6 table 5g
pre-up ip -6 route add default dev he-ipv6 metric 1000
post-down ip tunnel del he-ipv6
We need to deal with IPv4 changes in for the tunnel endpoint, so modem-interface-route also got:
ip tunnel change he-ipv6 local $new_ip_address
/etc/nftables.conf had to be taught to accept the 6in4 packets from the tunnel in the input chain:
# Allow HE tunnel
iifname "sfp.31" ip protocol 41 ip saddr 216.66.80.26 accept
Finally, I had to engage in something I never thought I d deal with; IPv6 NAT. HE provide a /48, and my FTTP ISP provides me with a /56, so this meant I could do a nice stateless 1:1 mapping:
This works. Mostly. The problem is that HE, not unreasonably, expect your IPv4 address to be pingable. And it turns out Three have some ranges that this works on, and some that it doesn t. Which means it s a bit hit and miss whether you can setup the tunnel.
I spent a while trying to find an alternative free IPv6 tunnel provider with a UK endpoint. There s less call for them these days, so I didn t manage to find any that actually worked (or didn t have a similar pingable requirement). I did consider whether I wanted to end up with routes via a VM, as I described in the failover post, but looking at costings for VMs with providers who could actually give me an IPv6 range I decided the cost didn t make it worthwhile; the VM cost ended up being more than the backup SIM is costing monthly.
Finally, it turns out happy eyeballs mostly means that when the 5G ends up on an IP that we can t setup the IPv6 tunnel on, things still mostly work. Browser usage fails over quickly and it s mostly my own SSH use that needs me to force IPv4. Purists will groan, but this turns out to be an acceptable trade-off for me, at present. Perhaps if I was seeing frequent failures the diverse routes approach to a VM would start to make sense, but for now I m pretty happy with the configuration in terms of having a mostly automatic backup link take over when the main link goes down.
Lyorn is the 17th Vlad Taltos book and a direct sequel to 2014's
Hawk. (Yes, actual main story
progress!) You do not want to start reading here; you would be hopelessly
confused. When this series is complete, I want to re-read the entire
thing from the beginning and pick up more of the bits I missed the first
time.
Vlad is not, in fact, free to see his friends and get entangled in
imperial politics again as I thought after Hawk. Despite the
successes of that story, there is one remaining small problem: incredibly
powerful magic users still want to kill him. His immediate solution is to
shelter in a theater, since Draegaran theaters are well-known for their
excellent magical shielding. This works well enough at first, but the
theater is rehearsing a play about Draegaran politics that is highly
offensive to the Lyorn and the theater may be shut down because of it.
Vlad's enemies are also willing to lean on his friends to find him and
kill him.
This series continues to be thoroughly enjoyable. Lyorn is "just"
more of Vlad being Vlad, meddling in everyone's business and coming up
with elaborate plans with too many moving parts that he somehow manages to
pull off, but I'll happily read lots of books like that. Vlad is both
anxious and grumpy, both of which give the plot some needed tension
without being overwhelming. There are no truly major world-building
revelations here (or, if there are, I missed them), but there's a lot of
processing of what the reader learned in Tsalmoth. It's increasingly looking like the payoff from those
revelations is going to be the series finale.
This is the first Vlad book that contains solid confirmation of where the
series as a whole is headed. Brust mentioned some time ago that the last
book is titled The Last Contract, and Lyorn comes close to
stating explicitly what that contract will be. I am sure that it will be
more complicated than it appears now and there are misdirections yet to
come, but I am excited to see where Brust takes this idea.
Vlad has been insistently apolitical for much of the series, meddling in
politics only when he has to or to help his friends and otherwise treating
it as a system that he has to navigate and survive. That was the root of
the conflict in Teckla, all the way back
at the start of the series. This may be starting to change, and when
Brust ties it together with the Jenoine, the Great Weapon Godslayer, and
the rest of the world-building he's cued up, the results are going to be
explosive.
Two books left: Chreotha and The Last Contract. I can
hardly wait.
In every Vlad book, Brust plays some sort of structural game. This time,
befitting the setting, it's a musical. The action is interspersed with
quotes from a fictional history about the play the theater is putting on,
a work called Song of the Presses about political censorship during
the reign of a Lyorn emperor in the 14th cycle, thousands of years before
the time of this book. This was, at times, nearly as interesting as the
main plot. The chapters are also numbered like the acts and scenes of a
play, although this I didn't notice as much since books often have that
structure anyway.
Since this is a musical, there are also songs. Specifically, each chapter
is introduced by a parody of songs from various musicals in our world,
rewritten so that they fit within Brust's fictional musical. Brust is
also a musician and a filker, so these songs are actually good, or at least they amused me a
great deal. I'm not much of a musical fan and I still could hear the tune
playing when I read most of them.
Lyorn is not so good that I would rave about it. It's one of those
functional connective books of a series that advances the plot, tells a
good story, and has some fun along the way. The guns on the mantelpiece
of this world have not gone off yet, and Vlad is still maneuvering into
position. But it's looking like we're going to get the conclusion, and
it's going to be spectacular. If you have read this far, you will want to
keep reading.
Followed by Chreotha, which may be a bit of a wait because
apparently Brust is going to write The Last Contract first to make
sure he ties up loose ends properly.
Rating: 7 out of 10
IPv4 is an expensive resource. However, many content providers are still
IPv4-only. The most common reason is that IPv4 is here to stay and IPv6 is an
additional complexity.1 This mindset may seem selfish, but there are
compelling reasons for a content provider to enable IPv6, even when they have
enough IPv4 addresses available for their needs.
Disclaimer
It s been a while since this article has been in my drafts. I
started it when I was working at Shadow, a content
provider, while I now work for Free, an internet service
provider.
Why ISPs need IPv6?
Providing a public IPv4 address to each customer is quite costly when each IP
address costs US$40 on the market. For fixed access, some consumer ISPs are
still providing one IPv4 address per customer.2 Other ISPs provide, by
default, an IPv4 address shared among several customers. For mobile access, most
ISPs distribute a shared IPv4 address.
There are several methods to share an IPv4 address:3
NAT44
The customer device is given a private IPv4 address, which is translated to a public
one by a service provider device. This device needs to maintain a state for each
translation.
464XLAT and DS-Lite
The customer device translates the private IPv4 address to an IPv6 address or
encapsulates IPv4 traffic in IPv6 packets. The provider device then translates
the IPv6 address to a public IPv4 address. It still needs to maintain a state
for the NAT64 translation.
Lightweight IPv4 over IPv6, MAP-E, and MAP-T
The customer device encapsulates IPv4 in IPv6 packets or performs a stateless
NAT46 translation. The provider device uses a binding table or an algorithmic
rule to map IPv6 tunnels to IPv4 addresses and ports. It does not need to
maintain a state.
Solutions to share an IPv4 address across several customers. Some of them require the ISP to keep state, some don't.
All these solutions require a translation device in the ISP s network. This
device represents a non-negligible cost in terms of money and reliability. As
half of the top 1000 websites support IPv6 and the biggest players can
deliver most of their traffic using IPv6,4 ISPs have a clear path to
reduce the cost of translation devices: provide IPv6 by default to their
customers.
Why content providers need IPv6?
Content providers should expose their services over IPv6 primarily to avoid going
through the ISP s translation devices. This doesn t help users who
don t have IPv6 or users with a non-shared IPv4 address, but it provides a
better service for all the others.
Why would the service be better delivered over IPv6 than over IPv4 when a
translation device is in the path? There are two main reasons for that:5
Translation devices introduce additional latency due to their geographical
placement inside the network: it is easier and cheaper to only install these
devices at a few points in the network instead of putting them close to the
users.
Translation devices are an additional point of failure in the path between
the user and the content. They can become overloaded or malfunction.
Moreover, as they are not used for the five most visited websites, which
serve their traffic over IPv6, the ISPs may not be incentivized to ensure
they perform as well as the native IPv6 path.
Looking at Google statistics, half of the users reach Google over IPv6.
Moreover, their latency is lower.6 In the US, all the nationwide mobile
providers have IPv6 enabled.
For France, we can refer to the annual ARCEP report: in 2022, 72% of fixed
users and 60% of mobile users had IPv6 enabled, with projections of 94% and 88%
for 2025. Starting from this projection, since all mobile users go through a
network translation device, content providers can deliver a better service for
88% of them by exposing their services over IPv6. If we exclude Orange, which
has 40% of the market share on consumer fixed access, enabling IPv6 should
positively impact more than 55% of fixed access users.
In conclusion, content providers aiming for the best user experience should
expose their services over IPv6. By avoiding translation devices, they can
ensure fast and reliable content delivery. This is crucial for latency-sensitive
applications, like live streaming, but also for websites in competitive markets,
where even slight delays can lead to user disengagement.
A way to limit this complexity is to build IPv6 services and only
provide IPv4 through reverse proxies at the edge.
In France, this includes non-profit ISPs, like FDN and
Milkywan. Additionally, Orange, the previously state-owned telecom
provider, supplies non-shared IPv4 addresses. Free also provides a dedicated
IPv4 address for customers connected to the point-to-point FTTH access.
I use the term NAT instead of the more correct term NAPT. Feel free to
do a mental substitution. If you are curious, check RFC 2663. For a
survey of the IPv6 transition technologies enumerated here, have a look at
RFC 9313.
For AS 12322, Google, Netflix, and Meta are delivering 85% of
their traffic over IPv6. Also, more than half of our traffic is delivered
over IPv6.
An additional reason is for fighting abuse: blacklisting an IPv4
address may impact unrelated users who share the same IPv4 as the culprits.
IPv6 may not be the sole reason the latency is lower: users with IPv6
generally have a better connection.
As I m gearing towards setting up a Free Software download mirror in India, it occurred to me that I haven t chronicled the work and motivation behind setting up the original mirror in the first place. Also, seems like it would be good to document stuff here for observing the progression, as the mirror is going multi-country now. Right now, my existing mirror i.e., mirrors.de.sahilister.net (was mirrors.sahilister.in), is hosted in Germany and serves traffic for Termux, NomadBSD, Blender, BlendOS and GIMP. For a while in between, it hosted OSMC project mirror as well.
To explain what is a Free Software download mirror thing is first, I ll quote myself from work blog -
As most Free Software doesn t have commercial backing and require heavy downloads, the concept of software download mirrors helps take the traffic load off of the primary server, leading to geographical redundancy, higher availability and faster download in general.
So whenever someone wants to download a particular (mirrored) software and click download, upstream redirects the download to one of the mirror server which is geographical (or in other parameters) nearby to the user, leading to faster downloads and load sharing amongst all mirrors.
Since the time I got into Linux and servers, I always wanted to help the community somehow, and mirroring seemed to be the most obvious thing. India seems to be a country which has traditionally seen less number of public download mirrors. IITB, TiFR, and some of the public institutions used to host them for popular Linux and Free Softwares, but they seem to be diminishing these days.
In the last months of 2021, I started using Termux and saw that it had only a few mirrors (back then). I tried getting a high capacity, high bandwidth node in budget but it was hard in India in 2021-22. So after much deliberation, I decided to go where it s available and chose a German hosting provider with the thought of adding India node when conditions are favorable (thankfully that happened, and India node is live too now.). Termux required only 29 GB of storage, so went ahead and started mirroring it. I raised this issue in Termux s GitHub repository in January 2022. This blog post chronicles the start of the mirror.
Termux has high request counts from a mirror point of view. Each Termux client, usually checks every mirror in selected group for availability before randomly selecting one for download (only other case is when client has explicitly selected a single mirror using termux-repo-change). The mirror started getting thousands of requests daily due to this but only a small percentage would actually get my mirror in selection, so download traffic was lower. Similar thing happened with OSMC too (which I started mirroring later).
With this start, I started exploring various project that would be benefit from additional mirrors. Public information from Academic Computer Club in Ume s mirror and Freedif s mirror stats helped to figure out storage and bandwidth requirements for potential projects. Fun fact, Academic Computer Club in Ume (which is one of the prominent Debian, Ubuntu etc.) mirror, now has 200 Gbits/s uplink to the internet through SUNET.
Later, I migrated to a different provider for better speeds and added LibreSpeed test on the mirror server. Those were fun times. Between OSMC, Termux and LibreSpeed, I was getting almost 1.2 millions hits/day on the server at its peak, crossing for the first time a TB/day traffic number.
Next came Blender, which took the longest time to set up of around 9 10 months. Blender had a push-trigger requirement for rsync from upstream that took quite some back and forth. It now contributes the most amount of traffic on the mirror. On release days, mirror does more than 3 TB/day and normal days, it hovers around 2 TB/day. Gimp project is the latest addition.
At one time, the mirror traffic touched 4.97 TB/day traffic number. That s when I decided on dropping LibreSpeed server to solely focus on mirroring for now, keeping the bandwidth allotment for serving downloads only.
The mirror projects selection grew organically. I used to reach out many projects discussing the need of for additional mirrors. Some projects outright denied mirroring request as Germany already has a good academic mirrors boosting 20-25 Gbits/s speeds from FTP era, which seems fair. Finding the niche was essential to only add softwares, which would truly benefit from additional capacity. There were months when nothing much would happen with the mirror, rsync would continue to update the mirror while nginx would keep on serving the traffic. Nowadays, the mirror pushes around 70 TB/month. I occasionally check logs, vnstat, add new security stuff here and there and pay the bills. It now saturates the Gigabit link sometimes and goes beyond that, peaking around 1.42 Gbits/s (the hosting provider seems to be upping their game). The plan is to upgrade the link to better speeds.
Yearly traffic stats (through vnstat -y )
On the way, learned quite a few things like -
IPv6 and exposing rsync module due to OSMC requirement.
Implementing user with restricted access to only trigger rsync, basically make rsync pull trigger based due to Blender.
Iterating over right client response size for LibreSpeed test project.
Mistakenly identifying torrent traffic for BlendOS as DDoS and blocking it for quite a few months. BlendOS added loads of hits for torrent traffic, making my mirror also serve as web seed. Web seeds in conjunction with normal seeds surely is a good combination for serving downloads as it combines best of both world, general availability of web seed/mirror and benefit of traditional seeds to maximize download speeds at user end.
Handling abusive traffic (a lot of it to be frank). The approach is more of a whack a mole right now, which I want to improve and automate.
Most of the traffic on non-Linux/BSD operating systems serving mirrors (like mine) is for people on Windows and Mac asking for EXEs and DMGs. Mostly because package repositories carry software distribution load for Linux/BSD OSs and partly because the number of Windows/Mac users are quite high compared to other OSs.
Load balancing through DNS and HTTP redirection (which I would implement in my India mirror now) to better maximize available resources.
In hindsight, the statistics look amazing, hundreds of TBs of traffic served from the mirror, month after month. That does show that there s still an appetite for public mirrors in time of commercially donated CDNs and GitHub. The world could have done with one less mirror, but it saved some time, lessened the burden for others, while providing redundancy and traffic localization with one additional mirror. And it s fun for someone like me who s into infrastructure that powers the Internet. Now, I ll try focusing and expanding the India mirror, which in itself started pushing almost half a TB/day. Long live Free Software and public download mirrors.
For some time now, Debian has been discussing a possible enhancement to
the way that Debian packages are uploaded to the archive. The basic idea
is to allow a package upload to be triggered by pushing a signed tag, with
some structured metadata, to Salsa, the instance of GitLab that Debian
provides for packaging repositories. This would allow Debian package
maintainers to use a more typical Git-first workflow, where releases are
triggered by Git tags and the release artifacts are built in a clean CI
environment, while still enforcing the existing Debian rules about who is
allowed to upload packages.
As part of that effort, I recently completed a
detailed security review of the
tag2upload design. I sent it to debian-vote as part of the ongoing
discussion, but have also posted it at the link above to give it a more
permanent home.
This security review may be revised based on the discussion if people
point out things that I missed.
A short status update of what happened on my side last month. A broken
gcovr in Debian triggered a bit of busy work but 0.39.0 came out
nicely nevertheless. We also reduced build time quiet a bit in phosh
and phoc.
In my previous blog, I explored The New APT 3.0 solver.
Since then I have been at work in the test suite making tests pass and fixing some bugs.
You see for all intents and purposes, the new solver is a very stupid naive DPLL SAT solver (it just
so happens we don t actually have any pure literals in there). We can control it in a bunch of ways:
We can mark packages as install or reject
We can order actions/clauses. When backtracking the action that came later will be the first we
try to backtrack on
We can order the choices of a dependency - we try them left to right.
This is about all that we really want to do, we can t go if we reach a conflict, say oh but this
conflict was introduced by that upgrade, and it seems more important, so let s not backtrack on
the upgrade request but on this dependency instead. .
This forces us to think about lowering the dependency problem into this form, such that not only
do we get formally correct solutions, but also semantically correct ones. This is nice because
we can apply a systematic way to approach the issue rather than introducing ad-hoc rules in the
old solver which had a which of these packages should I flip the opposite way to break the conflict
kind of thinking.
Now our test suite has a whole bunch of these semantics encoded in it, and I m going to share some
problems and ideas for how to solve them. I can t wait to fix these and the error reporting and
then turn it on in Ubuntu and later Debian (the defaults change is a post-trixie change, let s
be honest).
apt upgrade is hard
The apt upgrade commands implements a safe version of dist-upgrade that essentially calculates the
dist-upgrade, and then undoes anything that would cause a package to be removed, but it (unlike its
apt-get counterpart) allows the solver to install new packages.
Now, consider the following package is installed:
X Depends: A (= 1) B
An upgrade from A=1 to A=2 is available. What should happen?
The classic solver would choose to remove X in a dist-upgrade, and then upgrade A, so it s answer
is quite clear: Keep back the upgrade of A.
The new solver however sees two possible solutions:
Install B to satisfy X Depends A (= 1) B.
Keep back the upgrade of A
Which one does it pick? This depends on the order in which it sees the upgrade action for A and the
dependency, as it will backjump chronologically. So
If it gets to the dependency first, it marks A=1 for install to satisfy A (= 1). Then it gets
to the upgrade request, which is just A Depends A (= 2) A (= 1) and sees it is satisfied
already and is content.
If it gets to the upgrade request first, it marks A=2 for install to satisfy A (= 2). Then
later it gets to X Depends: A (= 1) B, sees that A (= 1) is not satisfiable, and picks B.
We have two ways to approach this issue:
We always order upgrade requests last, so they will be kept back in case of conflicting dependencies
We require that, for apt upgrade a currently satisfied dependency must be satisfied by currently installed
packages, hence eliminating B as a choice.
Recommends are hard too
See if you have a X Recommends: A (= 1) and a new version of A, A (= 2), the solver currently
will silently break the Recommends in some cases.
But let s explore what the behavior of a X Recommends: A (= 1) in combination with an available upgrade
of A (= 2) should be. We could say the rule should be:
An upgrade should keep back A instead of breaking the Recommends
A dist-upgrade should either keep back A or remove X (if it is obsolete)
This essentially leaves us the same choices as for the previous problem, but with an interesting twist.
We can change the ordering (and we already did), but we could also introduce a new rule, promotions :
A Recommends in an installed package, or an upgrade to that installed package, where the Recommends
existed in the installed version, that is currently satisfied, must continue to be satisfied, that is,
it effectively is promoted to a Depends.
This neatly solves the problem for us. We will never break Recommends that are satisfied.
Likewise, we already have a Recommends demotion rule:
A Recommends in an installed package, or an upgrade to that installed package, where the Recommends
existed in the installed version, that is currently unsatisfied, will not be further evaluated (it
is treated like a Suggests is in the default configuration).
Whether we should be allowed to break Suggests with our decisions or not (the old autoremover did not,
for instance) is a different decision. Should we promote currently satisified Suggests to Depends as well?
Should we follow currently satisified Suggests so the solver sees them and doesn t autoremove them,
but treat them as optional?
tightening of versioned dependencies
Another case of versioned dependencies with alternatives that has complex behavior is something like
X Depends: A (>= 2) B
X Recommends: A (>= 2) B
In both cases, installing X should upgrade an A < 2 in favour of installing B. But a naive
SAT solver might not. If your request to keep A installed is encoded as A (= 1) A (= 2), then
it first picks A (= 1). When it sees the Depends/Recommends it will switch to B.
We can solve this again as in the previous example by ordering the keep A installed requests after
any dependencies. Notably, we will enqueue the common dependencies of all A versions first before
selecting a version of A, so something may select a version for us.
version narrowing instead of version choosing
A different approach to dealing with the issue of version selection is to not select a version
until the very last moment. So instead of selecting a version to satisfy A (>= 2) we instead
translate
Depends: A (>= 2)
into two rules:
The package selection rule:
Depends: A
This ensures that any version of A is installed (i.e. it adds a version choice clause, A (= 1) A (= 2)
in an example with two versions for A.
The version narrowing rule:
Conflicts: A (<< 2)
This outright would reject a choice of A (= 1).
So now we have 3 kinds of clauses:
package selection
version narrowing
version selection
If we process them in that order, we should surely be able to find the solution that best matches
the semantics of our Debian dependency model, i.e. selecting earlier choices in a dependency before
later choices in the face of version restrictions.
This still leaves one issue: What if our maintainer did not use Depends: A (>= 2) B but
e.g. Depends: A (= 3) B A (= 2). He d expect us to fall back to B if A (= 3) is not
installable, and not to B. But we d like to enqueue A and reject all choices other than 3
and 2. I think it s fair to say: Don t do that, then here.
Implementing strict pinning correctly
APT knows a single candidate version per package, this makes the solver relatively deterministic:
It will only ever pick the candidate, or an installed version. This also happens to significantly
reduce the search space which is good - less backtracking. An uptodate system will only ever have
one version per package that can be installed, so we never actually have to choose versions.
But of course, APT allows you to specify a non-candidate version of a package to install, for example:
apt install foo/oracular-proposed
The way this works is that the core component of the previous solver, which is the pkgDepCache
maintains what essentially amounts to an overlay of the policy that you could see with
apt-cache policy.
The solver currently however validates allowed version choices against the policy directly,
and hence finds these versions are not allowed and craps out. This is an interesting problem
because the solver should not be dependent on the pkgDepCache as the pkgDepCache initialization
(Building dependency tree...) accounts for about half of the runtime of APT (until the Y/n prompt)
and I d really like to get rid of it.
But currently the frontend does go via the pkgDepCache. It marks the packages in there, building
up what you could call a transaction, and then we translate it to the new solver, and once it is
done, it translates the result back into the pkgDepCache.
The current implementation of allowed version is implemented by reducing the search space, i.e.
every dependency, we outright ignore any non-allowed versions. So if you have a version 3 of A
that is ignored a Depends: A would be translated into A (= 2) A (= 1).
However this has two disadvantages. (1) It means if we show you why A could not be installed,
you don t even see A (= 3) in the list of choices and (2) you would need to keep the pkgDepCache
around for the temporary overrides.
So instead of actually enforcing the allowed version rule by filtering, a more reasonable
model is that we apply the allowed version rule by just marking every other version as not
allowed when discovering the package in the from depcache translation layer. This doesn t
really increase the search space either but it solves both our problem of making overrides
work and giving you a reasonable error message that lists all versions of A.
pulling up common dependencies to minimize backtracking cost
One of the common issues we have is that when we have a dependency group
A B C D
we try them in order, and if one fails, we undo everything it did, and move on to the next one. However,
this isn t perhaps the best choice of operation.
I explained before that one thing we do is queue the common dependencies of a package (i.e. dependencies
shared in all versions) when marking a package for install, but we don t do this here: We have already
lowered the representation of the dependency group into a list of versions, so we d need to extract the
package back out of it.
This can of course be done, but there may be a more interesting solution to the problem, in that we
simply enqueue all the common dependencies. That is, we add n backtracking levels for n possible
solutions:
We enqueue the common dependencies of all possible solutions deps(A)&deps(B)&deps(C)&deps(D)
We decide (adding a decision level) not to install D right now and enqueue deps(A)&deps(B)&deps(C)
We decide (adding a decision level) not to install C right now and enqueue deps(A)&deps(B)
We decide (adding a decision level) not to install B right now and enqueue A
Now if we need to backtrack from our choice of A we hopefully still have a lot of common dependencies
queued that we do not need to redo. While we have more backtracking levels, each backtracking level
would be significantly cheaper, especially if you have cheap backtracking (which admittedly we do not
have, yet anyway).
The caveat though is: It may be pretty expensive to find the common dependencies. We need to iterate
over all dependency groups of A and see if they are in B, C, and D, so we have a complexity of roughly
#A * (#B+#C+#D)
Each dependency group we need to check i.e. is X Y in B meanwhile has linear cost: We need to
compare the memory content of two pointer arrays containing the list of possible versions that
solve the dependency group.
This means that X Y and Y X are different dependencies of course, but that is to be expected
they are. But any dependency of the same order will have the same memory layout.
So really the cost is roughly N^4. This isn t nice.
You can apply various heuristics here on how to improve that, or you can even apply binary logic:
Enqueue common dependencies of A B C D
Move into the left half, enqueue of A B
Again divide and conquer and select A.
This has a significant advantage in long lists of choices, and also in the common case, where the
first solution should be the right one.
Or again, if you enqueue the package and a version restriction instead, you already get the common
dependencies enqueued for the chosen package at least.
I last reviewed email services in 2019. That review focused a lot of attention on privacy. At the time, I selected mailbox.org as my provider, and have been using them for these 5 years since. However, both their service and their support have gone significantly downhill since, so it is time for me to look at other options.
Here I am focusing strongly on email. Some of the providers mentioned here provide other services (IM, video calls, groupware, etc.), and to the extent they do, I am ignoring them.
What Matters in 2024
I want to start off by acknowledging that what you need in email probably depends on your circumstances and the country in which you live. For me, I begin by naming that the largest threat most of us face isn t from state actors but from criminals: hackers, ransomware gangs, etc. It is important to take as many steps as possible to secure one s account against that. Privacy and security are both part of the mix. I still value privacy but I am acknowledging, as Migadu does, that Email as we know it and encryption are incompatible. Although some of these services strongly protect parts of the conversation, the reality is that most people will be emailing people using plain old email services which don t. For stronger security, something like Signal would be needed. (I wrote about Signal in 2021 also.)
Interestingly, OpenPGP support seems to be something of a standard feature in the providers I reviewed by this point. All or almost all of them provide integration with browser-based encryption as well as server-side encryption if you prefer that.
Although mailbox.org can automatically PGP-encrypt every message that arrives in plaintext, for general use, this is unwieldy; there isn t good tooling for searching mailboxes where every message is encrypted, etc. So I never enabled that feature at Mailbox. I still value security and privacy, but a pragmatic approach addresses the most pressing threats first.
My criteria
The basic requirements for an email service include:
Ability to use my own domains
Strong privacy policy
Ability for me to use my own IMAP and SMTP clients on both desktop and mobile
It must be extremely reliable
It must not be free
It must have excellent support for those rare occasions when it is needed
Support for basic aliases
Why do I say it must not be free? Because if someone is providing a service with the quality I m talking about here, and not charging for it, it implies something is fishy: either they are unscrupulous, are financially unstable, or the product is something else like ads. I am not aware of any provider that matches the other criteria with a free account anyhow. These providers range from about $30 to $90 per year, so cheaper than a Netflix subscription.
Immediately, this rules out several options:
Proton doesn t let me use my own clients on mobile (their bridge is desktop-only)
Tuta also doesn t let me use my own clients
Posteo doesn t let me use my own domain
mxroute.com lacks a strong privacy policy, and its policy has numerous causes for concern (for instance, If you repeatedly send email to invalid/unroutable recipients, they may be published on our GitHub )
I will have a bit more to say about a couple of these providers below.
There are some additional criteria that are strongly desired but not absolutely required:
Ability to set individual access passwords for every device/app
Support for two-factor authentication (2FA/TFA/TOTP) for web-based access
Support for basics in filtering: ability to filter on envelope recipient (so if I get BCC d, I can still filter), and ability to execute more than one action on filter match (eg, deliver to two folders, or deliver to a folder and forward to someone else)
IMAP and SMTP don t really support 2FA, so by setting individual passwords for every device, you can at least limit the blast radius and cut off a specific device if something is (or might be) compromised.
The candidates
I considered these providers: Startmail, Mailfence, Runbox, Fastmail, Kolab, Mailbox.org, and Migadu. I ll review each, and highlight the pricing of the plan I would most likely use. Each provider offers multiple plans; some may be more expensive and some may be cheaper than the one I reviewed. I included a link to each provider s full pricing information so you can compare for your needs.
I set up trials with each of these (except Mailbox.org, with which I already had a paid account). It so happend that I had actual questions for support for each one, which gave me an opportunity to see how support responded. I did not fabricate questions, and would not have contacted support if I didn t have real ones. (This means that I asked different questions of each provider, because they were the REAL questions I had.) I ll jump to the spoiler right now: I eventually chose Migadu, with Fastmail and Mailfence as close seconds.
I looked for providers myself, and also solicited recommendations in a Mastodon thread.
Mailbox.org
I begin with Mailbox, as it was my top choice in 2019 and the incumbent.
Until this year, I had been quite happy with it. I had cause to reach their support less than once a year on average, and each time they replied the same day or next day. Now, however, they are failing on reliability and on support.
Their spam filter has become overly aggressive. It has blocked quite a bit of legitimate mail. When contacting their support about a prior issue earlier this year, they initially took 4 days to reply, and then 6 days to reply after that. Ouch. They had me disable some spam settings.
It didn t really help. I continue to lose mail. I don t know how much, because they block a lot of it before it even hits the spam folder. One of my friends texted to say mail was dropping. I raised a new ticket with mailbox, which took them 5 days to reply to. Their reply was unhelpful. As the Internet is not a static system, unforeseen events can always occur. Well yes, that s true, and I get it, false positives exist with email. But this was from an ISP s mail system with an address that had been established for years, and it was part of a larger pattern of rejecting quite a bit of legit mail. And every interaction with them recently hasn t resulted in them actually doing anything to resolve anything. It s just a paragraph or two of reply that does nothing and helps nothing.
When I complained that it took 5 days to reply, they said We have not been able to reply sooner as we are currently experiencing a high volume of customer enquiries. Even though their SLA for my account is a not-great 48 business hour turnaround, they still missed it and their reason is we re busy. I finally asked what RBL had caught the blocked email, since when I checked, the sender wasn t on any RBL. Mailbox s reply: they only keep their logs for 7 days, so next time I should contact them within 7 days. Which, of course, I DID; it was them that kept delaying. Ugh! It s like they ve become a cable company.
Even worse is how they have been blocking mail from GrapheneOS s discussion form. See their thread about it. In short, Graphene s mail server has a clean reputation and Mailbox has no problem with it. But because one of Graphene s IPv6 webservers has an IPv6 allocation of a size Mailbox doesn t like, they drop mail. It s ridiculous, and Mailbox was dismissive of this well-known and well-regarded Open Source project. So if the likes of GrapheneOS can t get good faith effort to deliver their mail, what chance does an individual like me have?
I m sorry, but I m literally paying you to deliver email for me and provide good support. If you can t do either of those, you don t get to push that problem down onto me. Hire appropriate staff.
On the technical side, they support aliases, my own clients, and have a reasonable privacy policy. Their 2FA support exists for the web interface (though weirdly not the support site), though it is somewhat weird. They do not support app passwords.
A somewhat unique feature is the @secure.mailbox.org domain. If you try to receive mail at that address, mailbox.org will block it unless it uses TLS. Same for sending. This isn t E2EE, but it does at least require things not be in plaintext for the last hop to Mailbox.
Verdict: not recommended due to poor reliability and support.
Mailbox.Org summary:
Reliability: iffy due to over-aggressive spam filtering
Support: Poor; takes 4-6 days for a reply and replies are unhelpful
Individual access passwords: No
2FA: Yes, but with a PIN instead of a password as the other factor
Filtering: Full SIEVE feature set and GUI editor
Spam settings: greylisting on/off, reject some/all spam, etc. But they re insufficient to address Mailbox s overzealousness, which support says I cannot workaround within the interface.
Aliases: 50 on your domain name, 25 on mailbox.org
Additional mailboxes: Available; each one at the same fee as the primary mailbox
Startmail
I really wanted to like Startmail. Its vault is an interesting idea and should contribute to the security and privacy of an account. They clearly care about privacy.
It falls down in filtering. They have no way to filter on envelope recipient (BCC or similar). Their support confirmed this to me and that s a showstopper.
Startmail support was also as slow as Mailbox, taking 5 days to respond to me.
Two showstoppers right there.
Verdict: Not recommended due to slow support responsiveness and weak filtering.
Startmail summary:
Support: Mediocre; Took 5 days for a reply, but the reply was helpful
Individual app access passwords: Yes
2FA: Yes
Filtering: Poor; cannot filter on envelope recipient, and can t build filters with multiple actions
Spam settings: None
Server storage location: The Netherlands
Plan as reviewed: Custom domain (trial was Personal), [pricing link]
Cost per year: $70
Mail storage included: 20GB
Limits on send/receive volume: none
Aliases: unlimited, with lots of features: can set expiration, etc.
Additional mailboxes: not available
Kolab
Kolab Now is mainly positioned as a full groupware service, but they do have a email-only option which I investigated. There isn t much documentation about it compared to other providers, and also not much in the way of settings. You can turn greylisting on or off. And . that s it.
It has a full suite of filtering options. They set an X-Envelope-To header which you can use with the arbitrary header match to do the right thing even for BCC situations. Filters can have multiple conditions and multiple actions. It is SIEVE-based and you can download your SIEVE definitions.
If you enable 2FA, you disable IMAP and SMTP; not great.
Verdict: Not an impressive enough email featureset to justify going with it.
Kolab Now summary:
Additional mailboxes: Yes if you set up a group account. Flexible pricing based on user count is not documented anywhere I could find.
Mailfence
Mailfence is another option, somewhat similar to Startmail but without the unique vault. I had some questions about filters, and support was quite responsive, responding in a couple of hours.
Some of their copy on their website is a bit misleading, but support clarified when I asked them. They do not offer encryption at rest (like most of the entries here).
Mailfence s filtering system is the kind I d like to see. It allows multiple conditions and multiple actions for each rule, and has some unique actions as well (notify by SMS or XMPP). Support says that Recipients matches envelope recipients. However, one ommission is that I can t match on arbitrary headers; only the canned list of headers they provide.
They have only two spam settings:
spam filter on/off
whitelist
Given some recent complaints about their spam filter being overly aggressive, I find this lack of control somewhat concerning. (However, I discount complaints about people begging for more features in free accounts; free won t provide the kind of service I m looking for with any provider.) There are generally just very few settings for email as well.
Verdict: Response and helpful support, filtering has the right structure but lacks arbitrary header match. Could be a good option.
Mailfence summary:
Support: Excellent responsiveness and helpful replies (after some initial confusion about my question of greylisting)
Individual app access passwords: No. You can set a per-service password (eg, an IMAP password), but those will be shared with all devices speaking that protocol.
2FA: Yes
Filtering: Good; only misses the ability to filter on arbitrary headers
Mail storage included: 10GB, with a maximum of 50,000 messages
Limits on send/receive volume: none
Aliases: 50. Aliases can t be deleted once created (there may be an exeption to this for aliases on your own domain rather than mailfence.com)
Additional mailboxes: Their page on this is a bit confusing, and the pricing page lacks the information promised. It looks like you can pay the same $42/year for additional mailboxes, with a limit of up to 2 additional paid mailboxes and 2 additional free mailboxes tied to the account.
Runbox
This one came recommended in a Mastodon thread. I had some questions about it, and support response was fantastic I heard from two people that were co-founders of the company! Even within hours, on a weekend. Incredible! This kind of response was only surpassed by Migadu.
I initially wrote to Runbox with questions about the incoming and outgoing message limits, which I hadn t seen elsewhere, as well as the bandwidth limit. They said the bandwidth limit is no longer enforced on paid accounts. The incoming and outgoing limits are enforced, and all email (even spam) counts towards the limit. Notably the outgoing limit is per recipient, so if you send 10 messages to your 50-recipient family group, that s the limit. However, they also indicated a willingness to reset the limit if something happens. Unfortunately, hitting the limit results in a hard bounce (SMTP 5xx) rather than a temporary failure (SMTP 4xx) so it can result in lost mail. This means I d be worried about some attack or other weirdness causing me to lose mail.
Their filter is a pain point. Here are the challenges:
You can t directly match on a BCC recipient. Support advised to use a headers match, which will search for something anywhere in the headers. This works and is probably good enough since this data is in the Received: headers, but it is a little more imprecise.
They only have a contains , not an equals operator. So, for instance, a pattern searching for test@example.com would also match newtest@example.com . Support advised to put the email address in angle brackets to avoid this. That will work mostly. Angle brackets aren t always required in headers.
There is no way to have multiple actions on the filter (there is just no way to file an incoming message into two folders). This was the ultimate showstopper for me.
Support advised they are planning to upgrade the filter system in the future, but these are the limitations today.
Verdict: A good option if you don t need much from the filtering system. Lots of privacy emphasis.
Runbox summary:
Limited on send/receive volume: Receive 5000 messages/day, Send 500 recipients/day
Aliases: 100 on runbox.com; unlimited on your own domain
Additional mailboxes: $15/yr each, also with 10GB non-shared storage per mailbox
Fastmail
Fastmail came recommended to me by a friend I ve known for decades.
Here s the thing about Fastmail, compared to all the services listed above: It all just works. Everything. Filtering, spam prevention, it is all there, all feature-complete, and all just does the right thing as you d hope. Their filtering system has a canned dropdown for To/Cc/Bcc , it supports multiple conditions and multiple actions, and just does the right thing. (Delivering to multiple folders is a little cumbersome but possible.) It has a particularly strong feature set around administering multiple accounts, including things like whether users can prevent admins from reading their mail.
The not-so-great part of the picture is around privacy. Fastmail is based in Australia, where the government has extensive power around spying on data, even to the point of forcing companies to add wiretap capabilities. Fastmail s privacy policy states user data may be held in Australia, USA, India, and Netherlands. By default, they share data with unidentified spam companies , though you can disable this in settings. On the other hand, they do make a good effort towards privacy.
I contacted support with some questions and got back a helpful response in three hours. However, one of the questions was about in which countries my particular data would be stored, and the support response said they would have to get back to me on that. It s been several days and no word back.
Verdict: A featureful option that just works , with a lot of features for managing family accounts and the like, but lacking in the privacy area.
Fastmail summary:
Support: Good response time on most questions; dropped the ball on one tha trequired research
Individual app access passwords: Yes
2FA: Yes
Filtering: Excellent
Spam settings: Can set filter aggressiveness, decide whether to share spam data with spam-fighting companies , configure how to handle backscatter spam, and evaluate the personal learning filter.
Server storage locations: Australia, USA, India, and The Netherlands. Legal jurisdiction is Australia.
Additional mailboxes: No; requires a different plan for that
Migadu
Migadu was a service I d never heard of, but came recommended to me on Mastodon.
I listed Migadu last because it is a class of its own compared to all the other options. Every other service is basically a webmail interface with a few extra settings tacked on.
Migadu has a full-featured email admin console in addition. By that I mean you can:
View usage graphs (incoming, outgoing, storage) over time
Manage DNS (if you want Migadu to run your nameservers)
Manage multiple domains, and cross-domain relationships with mailboxes
View a limited set of logs
Configure accounts, reset their passwords if needed/authorized, etc.
Configure email address rewrite rules with wildcards and so forth
Basically, if you were the sort of person that ran your own mail servers back in the day, here is Migadu giving you most of that functionality. Effectively you have a web interface to do all the useful stuff, and they handle the boring and annoying bits. This is a really attractive model.
Migadu support has been fantastic. They are quick to respond, and went above and beyond. I pointed out that their X-Envelope-To header, which is needed for filtering by BCC, wasn t being added on emails I sent myself. They replied 5 hours later indicating they had added the feature to add X-Envelope-To even for internal mails! Wow! I am impressed.
With Migadu, you buy a pool of resources: storage space and incoming/outgoing traffic. What you do within that pool is up to you. You can set up users ( mailboxes ), aliases, domains, whatever you like. It all just shares the pool. You can restrict users further so that an individual user has access to only a subset of the pool resources.
I was initially concerned about Migadu s daily send/receive message count limits, but in visiting with support and reading the documentation, what really comes out is that Migadu is a service with a personal touch. Hitting the incoming traffic limit will cause a SMTP temporary fail (4xx) response so you won t lose legit mail and support will work with you if it s a problem for legit uses. In other words, restrictions are soft and they are interpreted reasonably.
One interesting thing about Migadu is that they do not offer accounts under their domain. That is, you MUST bring your own domain. That s pretty easy and cheap, of course. It also puts you in a position of power, because it is easy to migrate email from one provider to another if you own the domain.
Filtering is done via SIEVE. There is a GUI editor which lets you accomplish most things, though it has an odd blind spot where you can t file a message into multiple folders. However, you can edit a SIEVE ruleset directly and you get the full SIEVE featureset, which is extensive (and does support filing a message into multiple folders). I note that the SIEVE :envelope match doesn t work, but Migadu adds an X-Envelope-To header which is just as good.
I particularly love a company that tells you all the reasons you might not want to use them. Migadu s pro/con list is an honest drawbacks list (of course, their homepage highlights all the features!).
Verdict: Fantastically powerful, excellent support, and good privacy. I chose this one.
Migadu summary:
Aliases: Unlimited on an unlimited number of domains
Additional mailboxes: Unlimited and free; uses pooled quotas, but individual quotas can be set
Others
Here are a few others that I didn t think worthy of getting a trial:
mxroute was recommended by several. Lots of concerning things in their policy, such as:
if you repeatedly send mail to unroutable recipients, they may publish the addresses on Github
they will terminate your account if they think you are rude or want to contest a charge
they reserve the right to cancel your service at any time for any (or no) reason.
Proton keeps coming up, and I will not consider it so long as I am locked into their client on mobile.
Skiff comes up sometimes, but they were acquired by Notion.
Disroot comes up; this discussion highlights a number of reasons why I avoid them. Their Terms of Service (ToS) is inconsistent with a general-purpose email account (I guess for targeting nonprofits and activists, that could make sense). Particularly laughable is that they claim to be friends of Open Source, but then would take down your account if you upload copyrighted material. News flash: in order for an Open Source license to be meaningful, the underlying work is copyrighted. It is perfectly legal to upload copyrighted material when you wrote it or have the license to do so!
Conclusions
There are a lot of good options for email hosting today, and in particular I appreciate the excellent personal support from companies like Migadu and Runbox. Support small businesses!
BGP routing loop using a confederation
Here comes the 2nd article of the 5-episode blog post series written by Guido Berh rster, member of staff at my company Fre(i)e Software GmbH.
Enjoy also this read on Guido's work on Polis,
My Debian contributions this month were all
The final piece of my 
A short status update of what happened on my side last month. A broken
gcovr in Debian triggered a bit of busy work but 0.39.0 came out
nicely nevertheless. We also reduced build time quiet a bit in phosh
and phoc.