Legends & Lattes is a sword and sorcery fantasy novel of the
RPG-inspired, post-Dungeons-and-Dragons subtype. It was Travis
Baldree's first novel.
Viv is an orc, the heavy muscle for a roving band of adventurers who take
jobs for hire in a way familiar to any Dungeons and Dragons player.
As this book opens, she's been an adventurer for twenty-two years, and
she's done. Her band have defeated the Scalvert Queen and gained its
hoard, but all that Viv wants is the stone in its head. With that in hand
and some vague lore about how to use it, Viv leaves, rather abruptly, and
heads for the city of Thune to chase a dream she's never told anyone else
about.
Viv wants to start a coffee shop.
Legends & Lattes is an unapologetic comfort story. Viv doesn't
entirely know what she's doing, but she has a lot of experience hiring
people, negotiating, and figuring things out, and she's willing to do a
lot of hard work. She's blunt and a bit rough, but she's ethical and
kind, which lets her attract and retain her first two employees: a
taciturn expert carpenter Viv picks out by watching people work at the
docks, and a succubus she hires as a barista. From there, the story
slowly turns into a found family dynamic, full of people that you like and
are rooting for. There is one actual villain who shows up towards the end
of the book to give it some conflict, but mostly this is the story of Viv
building a small business while being a good employer and friend.
The subtitle of "a novel of high fantasy and low stakes" is therefore an
excellent description. (Pedantic aside: This is "high fantasy" in the
literary sense of not involving an otherwise-normal world, not "high" in
the RPG sense of having less realistic, more mythic characters.) You are
not going to be surprised by the outcome of the story, or even most of the
events along the way. It's a book full of basically good people trying to
do the right thing, largely succeeding, and building a community in the
process.
If that feels relaxing and fun, you have precisely the right idea.
Sometimes you want a book in which good things happen to good people, and
that's exactly what Baldree delivers.
There are also a lot of specific details about explaining coffee to
everyone and setting up a portion of the menu of a modern coffee shop in a
fantasy world, and those parts I found less interesting. Baldree uses the
handwave of gnomish machinery to import big chunks of 2020s coffee
technology wholesale, which felt oddly out of step with the vaguely
medieval-ish Dungeons and Dragons world. This is also one of those
books where the characters independently reinvent multiple ideas that
historically came from different regions and slow processes of refinement.
Here it's drinks and pastries rather than major technological advances,
which I guess is a little bit better, but I find this style of
world-building grating.
The obligatory coffee shop cat is delightfully strange and suits the
fantasy setting. I wish more of the coffee shop trappings had similar
twists.
That said, everything else about the book worked for me, and the
characters are thankfully more central to the book than the coffee shop
trappings. I liked all of them and had no trouble rooting for them. The
found family bits worked for me and the character relationships developed
slowly enough to be believable but fast enough to be satisfying. Viv is
refreshingly blunt, so I wasn't annoyed by communication failures. And
the succubus, Tandri, is a fun, complex character and a great counterpoint
to Viv.
If you're looking for something challenging or deep, this isn't the book,
but if you're in the mood for a predictable comfort read, this hit the
spot. Recommended.
Followed by Bookshops & Bonedust (not yet published), but
Legends & Lattes is a complete story.
My edition has a novella (maybe a novelette) at the end of the short
novel.
"Pages to Fill": This is a prelude to the novel, telling the
story of Viv's first encounter with a coffee shop and the point where she
made the decision to stop adventuring. That's not the main plot of the
story, though. She and her team are pursuing a shapeshifting thief, which
leads to Viv having some unexpected reactions.
This was fairly slight and predictable, but once you've read the novel,
it's fun to see how the story began. The best part is seeing more of
Gallina, the gnome who was by far my favorite of Viv's old team. (6)
Rating: 8 out of 10
From May 25th to 27th, Bras lia hosted the
MiniDebConf 2023. This gathering, composed
of various activities such as talks, workshops, sprints, BSP (Bug Squashing
Party), key signing, social events, and hacking, aimed to bring the community
together and celebrate the world's largest Free Software project: Debian.
The MiniDebConf Bras lia 2023 was a success thanks to the participation of
everyone, regardless of their level of knowledge about Debian. We valued the
presence of both beginners who are getting familiar with the system and official
project developers. The spirit of inclusion and collaboration was present
throughout the event.
MiniDebConfs are local meetings organized by members of the Debian Project,
aiming to achieve similar goals as DebConf but on a regional scale. Throughout
the year, events like this occur in different parts of the world, strengthening
the Debian community.
Activities
The MiniDebConf program was intense and diverse. On May 25th and 26th (Thursday
and Friday), we had talks, discussions, workshops, and many hands-on activities.
On the 27th (Saturday), the Hacking Day took place, which was a special moment
for Debian contributors to come together and work collaboratively on various
aspects of the project. This was the Brazilian version of Debcamp, a tradition
preceding DebConf. On this day, we prioritized practical activities such as
software packaging, translations, key signing, install fest, and the Bug
Squashing Party.
Edition numbers
The event numbers are impressive and demonstrate the community's involvement
with Debian. We had 236 registered participants, 20 submitted talks, 14
volunteers, and 125 check-ins. Furthermore, in the hands-on activities, we
achieved significant results, including 7 new installations of Debian GNU/Linux,
the update of 18 packages in the official Debian project repository by
participants, and the inclusion of 7 new contributors to the translation team.
We also highlight the remote participation of the community through live
streams. The analytics data reveals that our website received a total of 7,058
views, with 2,079 views on the homepage (which featured our sponsors' logos),
3,042 views on the program page, and 104 views on the sponsors' page. We
recorded 922 unique users during the event.
On YouTube, the live stream reached 311 views, with 56 likes and a peak of 20
concurrent views. There were an incredible 85.1 hours of watch time, and our
channel gained 30 new subscribers. All this engagement and interest from the
community further strengthen MiniDebConf.
Photos and videos
To relive the best moments of the event, we have photos and recordings
available. Photos can be accessed at: https://deb.li/pbsb2023. Video recordings
of the talks are available at the following link: https://deb.li/vbsb2023.
To stay updated and connect with the Debian Bras lia community, follow us on
our social media channels:
Thanks
We would like to express our deep gratitude to all the participants, organizers,
sponsors, and supporters who contributed to the success of MiniDebConf Bras lia
2023. In particular, we extend our thanks to our Gold sponsors:
2024.
Pencillabs,
Globo,
Policorp, and
Toradex Brasil, and our Silver sponsor,
4-Linux. We also thank Finatec and the
Instituto para Conserva o de Tecnologias Livres (ICTL)
for their support.
MiniDebConf Bras lia 2023 was a milestone for the Debian community,
demonstrating the power of collaboration and Free Software. We hope that
everyone enjoyed this enriching gathering and will continue to actively
participate in future Debian Project initiatives. Together, we can make a
difference!
After watching Democracy Now ssegment on
artificial
intelligence I
started to wonder - am I out of step on this topic?
When people claim artificial intelligence will surpass human intelligence and
thus threaten humanity with extinction, they seem to be referring specifically
to advances made with large language models.
As I understand them, large language models are probability machines that have
ingested massive amounts of text scraped from the Internet. They answer
questions based on the probability of one series of words (their answer)
following another series of words (the question).
It seems like a stretch to call this intelligence, but if we accept that
definition then it follows that this kind of intelligence is nothing remotely
like human intelligence, which makes the claim that it will surpass human
intelligence confusing. Hasn t this kind of machine learning surpassed us
decades ago?
Or when we say surpass does that simply refer to fooling people into thinking
an AI machine is a human via conversation? That is an important milestone, but
I m not ready to accept the turing
test as proof of equal
intelligence.
Furthermore, large language models hallucinate and also reflect the biases of
their training data. The word hallucinate seems like a euphemism, as if it
could be corrected with the right medication when in fact it seems hard to
avoid when your strategy is to correlate words based on probability. But even
if you could solve the here is a completely wrong answer presented with
sociopathic confidence problem, reflecting the biases of your data sources
seems fairly intractable. In what world would a system with built-in bias be
considered on the brink of surpassing human intelligence?
The danger from LLMs seems to be their ability to convince people that their
answers are correct, including their patently wrong and/or biased answers.
Why do people think they are giving correct answers? Oh right
terrifying right wing billionaires (with terrifying
agendas have been claiming AI will exceed
human intelligence and threaten
humanity and
every time they sign a hyperbolic
statement they get front
page mainstream coverage. And even progressive news outlets are spreading this
narrative with
minimal space for contrary opinions (thank you Tawana Petty from the
Algorithmic Justice League for providing the only
glimpse of reason in the segment).
The belief that artificial intelligence is or will soon become omnipotent has
real world harms today: specifically it creates the misperception that current
LLMs are accurate, which paves the way for greater adoption among police
forces, social service agencies, medical facilities and other places where
racial and economic biases have life and death consequences.
When the CEO of OpenAI calls the technology dangerous and in need of
regulation, he gets both free advertising promoting the power and supposed
accuracy of his product and the possibility of freezing further developments in
the field that might challenge OpenAI s current dominance.
The real threat to humanity is not AI, it s massive inequality and the use of
tactics ranging from mundane bureaucracy to deadly force and incarceration to
segregate the affluent from the growing number of people unable to make ends
meet. We have spent decades training bureaucrats, judges and cops to
robotically follow biased laws to maintain this order without compassion or
empathy. Replacing them with AI would be make things worse and should be
stopped. But, let s be clear, the narrative that AI is poised to surpass human
intelligence and make humanity extinct is a dangerous distraction that runs
counter to a much more important story about the very real and very present
exploitative practices of the [companies building AI], who are rapidly
centralizing power and increasing social
inequities. .
Maybe we should talk about that instead?
The Stone Canal is a sort of halfway sequel to
The Star Fraction. They both take place
in the same universe, but the characters are almost entirely disjoint.
Half of The Stone Canal happens (mostly) well before the previous
book and the other half happens well after it. This book does contain
spoilers for the ending of The Star Fraction if one connects the
events of the two books correctly (which was a bit harder than I thought
it should be), so I would not read them out of order.
At the start of The Stone Canal, Jon Wilde wakes up on New Mars
beside the titular canal, in the middle of nowhere, accompanied only by a
robot that says it made him. Wilde remembers dying on Earth; this new
life is apparently some type of resurrection. It's a long walk to Ship
City, the center of civilization of a place the robot tells him is New
Mars.
In Ship City, an android named Dee Model has escaped from her owner and is
hiding in a bar. There, she meets an AI abolitionist named Tamara, who
helps her flee out the back and down the canal on a boat when Wilde walks
into the bar and immediately recognizes her. The abolitionists provide
her protection and legal assistance to argue her case for freedom from her
owner, a man named Reid.
The third thread of the story, and about half the book, is Jon Wilde's
life on Earth, starting in 1975 and leading up to the chaotic wars,
political fracturing, and revolutions that formed the background and plot
of The Star Fraction. Eventually that story turns into a
full-fledged science fiction setting, but not until the last 60 pages of
the book.
I successfully read two books in a Ken MacLeod series! Sadly, I'm not
sure I enjoyed the experience.
I commented in my review of The Star
Fraction that the appeal for me in MacLeod's writing was his reputation
as a writer of political science fiction. Unfortunately that's been a
bust. The characters are certainly political, in the sense that they
profess to have strong political viewpoints and are usually members of
some radical (often Trotskyite) organization. There are libertarian
anarchist societies and lots of political conflict. But there is almost
no meaningful political discussion in any of these books so far.
The politics are all tactical or background, and often seem to be created
by authorial fiat.
For example, New Mars is a sort of libertarian anarchy that somehow
doesn't have corporations or a strongman ruler, even though the history
(when we finally learn it) would have naturally given rise to one or the
other (and has, in numerous other SF novels with similar plots). There's
a half-assed explanation for this towards the end of the book that I
didn't find remotely believable. Another part of the book describes the
formation of the libertarian microstate in The Star Fraction, but
never answers a "why" or "how" question I had in the previous book in a
satisfying way. Somehow people stop caring about control or
predictability or stability or traditional hierarchy without any
significant difficulties except external threats, in situations of chaos
and disorder where historically humans turn to anyone promising firm
structure.
It's common to joke about MacLeod winning multiple libertarian Prometheus
Awards for his fiction despite being a Scottish communist. I'm finding
that much less surprising now that I've read more of his books. Whether
or not he believes in it himself, he's got the cynical libertarian
smugness and hand-waving down pat.
What his characters do care deeply about is smoking, drinking, and having
casual sex. (There's more political fire here around opposition to
anti-smoking laws than there is about any of the society-changing
political structures that somehow fall into place.) I have no objections
to any of those activities from a moral standpoint, but reading about
other people doing them is a snoozefest. The flashback scenes sketch out
enough imagined history to satisfy some curiosity from the previous book,
but they're mostly about the world's least interesting love triangle,
involving two completely unlikable men and lots of tedious jealousy and
posturing.
The characters in The Stone Canal are, in general, a problem. One
of those unlikable men is Wilde, the protagonist for most of the book.
Not only did I never warm to him, I never figured out what motivates him
or what he cares about. He's a supposedly highly political person who
seems to engage in politics with all the enthusiasm of someone filling out
tax forms, and is entirely uninterested in explaining to the reader any
sort of coherent philosophical approach. The most interesting characters
in this book are the women (Annette, Dee Model, Tamara, and, very late in
the book, Meg), but other than Dee Model they rarely get much focus from
the story.
By far the best part of this book is the last 60 pages, where MacLeod
finally explains the critical bridge events between Wilde's political
history on earth and the New Mars society. I thought this was engrossing,
fast-moving, and full of interesting ideas (at least for a 1990s book;
many of them feel a bit stale now, 25 years later). It was also
frustrating, because this was the book I wanted to have been reading for
the previous 270 pages, instead of MacLeod playing coy with his invented
history or showing us interminable scenes about Wilde's insecure jealousy
over his wife. It's also the sort of book where at one point characters
(apparently uniformly male as far as one could tell from the text of the
book) get assigned sex slaves, and while MacLeod clearly doesn't
approve of this, the plot is reminiscent of a Heinlein novel: the
protagonist's sex slave becomes a very loyal permanent female companion
who seems to have the same upside for the male character in question.
This was unfortunately not the book I was hoping for. I did enjoy the
last hundred pages, and it's somewhat satisfying to have the history come
together after puzzling over what happened for 200 pages. But I found the
characters tedious and annoying and the politics weirdly devoid of
anything like sociology, philosophy, or political science. There is the
core of a decent 1990s AI and singularity novel here, but the technology
is now rather dated and a lot of other people have tackled the same idea
with fewer irritating ticks.
Not recommended, although I'll probably continue to The Cassini
Division because the ending was a pretty great hook for another book.
Followed by The Cassini Division.
Rating: 5 out of 10
DEP-17 progress, by Helmut and Emilio
We posted a proposal for modifying dpkg to better cope with directory aliasing.
After an initial period of silence, the discussion took off, but was mostly
diverted to a competing proposal by Luca Boccassi: Do not change dpkg at all,
but still move all files affected by aliasing to their canonical location and
thus removing the bad effects of aliasing. We facilitated this discussion and
performed extensive analysis of this and competing proposals highlighting
resulting problems and proposing solutions or workarounds. We performed a
detailed analysis of how aliasing affects usage of dpkg-divert,
dpkg-statoverride and update-alternatives. Details are available on the
debian-dpkg mailinglist thread.
Debian Reimbursements Web App Progress, by Stefano Rivera
In a project funded by
Freexian s Project Funding initiative,
Stefano made some more progress on the
Debian Reimbursements Web App.
The full workflow can now be exercised, completing the first milestone of the
project, the Working Prototype.
Stefano attended several DebConf planning meetings, and did some work on the
DebConf 23 website.
Stefano updated distro-info-data to include the release date of Debian
bullseye, and added the next Ubuntu release, Mantic Minotour. This required a
round of updates to all the stable releases, LTS, and ELTS.
Helmut sent patches for 13 cross build failures and filed 104 RC bugs for
missing Breaks and Replaces.
Yesterday, the UBports core developer team released Ubuntu Touch Focal OTA-1
(In fact, Raoul, Marius and I were in a conference call when Marius froze and said: the PR team already posted the release blog post; the post is out, but we haven't released yet... ahhhh... panic... Shall I?, Marius said, and we said: GO!!! This is why the release occurred in public five hours ahead of schedule. OMG.)
For all the details, please study: https://ubports.com/blog/ubports-news-1/post/ubuntu-touch-ota-1-focal-re...
Credits
Thanks to all the developers, other contributors and funding providers that helped to reach this massive milestone. I dare to drop some names here at the risk of forgetting others (I put them in alphanumerical order): Alan, Alfred, Brian, Christoffer, Daniel, Eline, Florian, Guido, Jami, Jonathan, Kugi, Lionel, Maciek, Mardy, Marius, Mike, Nigel, Nikita, Raoul, Ratchanan, Robert, Sergey.
I have been involved in the development and release process over the past four years and I feel honoured to work with so many fine and genuine people on such a unique project.
It is a pleasure to work with you guys!!!
Also a big thanks to the UBports Foundation and its BoD for being the umbrella organisation of all Ubuntu Touch related initiatives.
Consumer-Ready
Ubuntu Touch is one of the very few Open Source projects that brings fourth a 100% FLOSS phone operating system.
After using Ubuntu Touch myself for several months now, I can confirm that it is a consumer grade OS that can be used by non-tech people as a daily driver for mobile communications and connectivity.
Go for it and try it out.
I ve used hardware-backed OpenPGP keys since 2006 when I imported newly generated rsa1024 subkeys to a FSFE Fellowship card. This worked well for several years, and I recall buying more ZeitControl cards for multi-machine usage and backup purposes. As a side note, I recall being unsatisfied with the weak 1024-bit RSA subkeys at the time my primary key was a somewhat stronger 1280-bit RSA key created back in 2002 but OpenPGP cards at the time didn t support more than 1024 bit RSA, and were (and still often are) also limited to power-of-two RSA key sizes which I dislike.
I had my master key on disk with a strong password for a while, mostly to refresh expiration time of the subkeys and to sign other s OpenPGP keys. At some point I stopped carrying around encrypted copies of my master key. That was my main setup when I migrated to a new stronger RSA 3744 bit key with rsa2048 subkeys on a YubiKey NEO back in 2014. At that point, signing other s OpenPGP keys was a rare enough occurrence that I settled with bringing out my offline machine to perform this operation, transferring the public key to sign on USB sticks. In 2019 I re-evaluated my OpenPGP setup and ended up creating a offline Ed25519 key with subkeys on a FST-01G running Gnuk. My approach for signing other s OpenPGP keys were still to bring out my offline machine and sign things using the master secret using USB sticks for storage and transport. Which meant I almost never did that, because it took too much effort. So my 2019-era Ed25519 key still only has a handful of signatures on it, since I had essentially stopped signing other s keys which is the traditional way of getting signatures in return.
None of this caused any critical problem for me because I continued to use my old 2014-era RSA3744 key in parallel with my new 2019-era Ed25519 key, since too many systems didn t handle Ed25519. However, during 2022 this changed, and the only remaining environment that I still used my RSA3744 key for was in Debian and they require OpenPGP signatures on the new key to allow it to replace an older key. I was in denial about this sub-optimal solution during 2022 and endured its practical consequences, having to use the YubiKey NEO (which I had replaced with a permanently inserted YubiKey Nano at some point) for Debian-related purposes alone.
In December 2022 I bought a new laptop and setup a FST-01SZ with my Ed25519 key, and while I have taken a vacation from Debian, I continue to extend the expiration period on the old RSA3744-key in case I will ever have to use it again, so the overall OpenPGP setup was still sub-optimal. Having two valid OpenPGP keys at the same time causes people to use both for email encryption (leading me to have to use both devices), and the WKD Key Discovery protocol doesn t like two valid keys either. At FOSDEM 23 I ran into Andre Heinecke at GnuPG and I couldn t help complain about how complex and unsatisfying all OpenPGP-related matters were, and he mildly ignored my rant and asked why I didn t put the master key on another smartcard. The comment sunk in when I came home, and recently I connected all the dots and this post is a summary of what I did to move my offline OpenPGP master key to a Nitrokey Start.
First a word about device choice, I still prefer to use hardware devices that are as compatible with free software as possible, but the FST-01G or FST-01SZ are no longer easily available for purchase. I got a comment about Nitrokey start in my last post, and had two of them available to experiment with. There are things to dislike with the Nitrokey Start compared to the YubiKey (e.g., relative insecure chip architecture, the bulkier form factor and lack of FIDO/U2F/OATH support) but as far as I know there is no more widely available owner-controlled device that is manufactured for an intended purpose of implementing an OpenPGP card. Thus it hits the sweet spot for me.
Nitrokey Start
The first step is to run latest firmware on the Nitrokey Start for bug-fixes and important OpenSSH 9.0 compatibility and there are reproducible-built firmware published that you can install using pynitrokey. I run Trisquel 11 aramo on my laptop, which does not include the Python Pip package (likely because it promotes installing non-free software) so that was a slight complication. Building the firmware locally may have worked, and I would like to do that eventually to confirm the published firmware, however to save time I settled with installing the Ubuntu 22.04 packages on my machine:
$ sha256sum python3-pip*
ded6b3867a4a4cbaff0940cab366975d6aeecc76b9f2d2efa3deceb062668b1c python3-pip_22.0.2+dfsg-1ubuntu0.2_all.deb
e1561575130c41dc3309023a345de337e84b4b04c21c74db57f599e267114325 python3-pip-whl_22.0.2+dfsg-1ubuntu0.2_all.deb
$ doas dpkg -i python3-pip*
...
$ doas apt install -f
...
$
Installing pynitrokey downloaded a bunch of dependencies, and it would be nice to audit the license and security vulnerabilities for each of them. (Verbose output below slightly redacted.)
jas@kaka:~$ pip3 install --user pynitrokey
Collecting pynitrokey
Downloading pynitrokey-0.4.34-py3-none-any.whl (572 kB)
Collecting frozendict~=2.3.4
Downloading frozendict-2.3.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (113 kB)
Requirement already satisfied: click<9,>=8.0.0 in /usr/lib/python3/dist-packages (from pynitrokey) (8.0.3)
Collecting ecdsa
Downloading ecdsa-0.18.0-py2.py3-none-any.whl (142 kB)
Collecting python-dateutil~=2.7.0
Downloading python_dateutil-2.7.5-py2.py3-none-any.whl (225 kB)
Collecting fido2<2,>=1.1.0
Downloading fido2-1.1.0-py3-none-any.whl (201 kB)
Collecting tlv8
Downloading tlv8-0.10.0.tar.gz (16 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: certifi>=14.5.14 in /usr/lib/python3/dist-packages (from pynitrokey) (2020.6.20)
Requirement already satisfied: pyusb in /usr/lib/python3/dist-packages (from pynitrokey) (1.2.1.post1)
Collecting urllib3~=1.26.7
Downloading urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Collecting spsdk<1.8.0,>=1.7.0
Downloading spsdk-1.7.1-py3-none-any.whl (684 kB)
Collecting typing_extensions~=4.3.0
Downloading typing_extensions-4.3.0-py3-none-any.whl (25 kB)
Requirement already satisfied: cryptography<37,>=3.4.4 in /usr/lib/python3/dist-packages (from pynitrokey) (3.4.8)
Collecting intelhex
Downloading intelhex-2.3.0-py2.py3-none-any.whl (50 kB)
Collecting nkdfu
Downloading nkdfu-0.2-py3-none-any.whl (16 kB)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from pynitrokey) (2.25.1)
Collecting tqdm
Downloading tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting nrfutil<7,>=6.1.4
Downloading nrfutil-6.1.7.tar.gz (845 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: cffi in /usr/lib/python3/dist-packages (from pynitrokey) (1.15.0)
Collecting crcmod
Downloading crcmod-1.7.tar.gz (89 kB)
Preparing metadata (setup.py) ... done
Collecting libusb1==1.9.3
Downloading libusb1-1.9.3-py3-none-any.whl (60 kB)
Collecting pc_ble_driver_py>=0.16.4
Downloading pc_ble_driver_py-0.17.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.9 MB)
Collecting piccata
Downloading piccata-2.0.3-py3-none-any.whl (21 kB)
Collecting protobuf<4.0.0,>=3.17.3
Downloading protobuf-3.20.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
Collecting pyserial
Downloading pyserial-3.5-py2.py3-none-any.whl (90 kB)
Collecting pyspinel>=1.0.0a3
Downloading pyspinel-1.0.3.tar.gz (58 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from nrfutil<7,>=6.1.4->pynitrokey) (5.4.1)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil~=2.7.0->pynitrokey) (1.16.0)
Collecting pylink-square<0.11.9,>=0.8.2
Downloading pylink_square-0.11.1-py2.py3-none-any.whl (78 kB)
Collecting jinja2<3.1,>=2.11
Downloading Jinja2-3.0.3-py3-none-any.whl (133 kB)
Collecting bincopy<17.11,>=17.10.2
Downloading bincopy-17.10.3-py3-none-any.whl (17 kB)
Collecting fastjsonschema>=2.15.1
Downloading fastjsonschema-2.16.3-py3-none-any.whl (23 kB)
Collecting astunparse<2,>=1.6
Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting oscrypto~=1.2
Downloading oscrypto-1.3.0-py2.py3-none-any.whl (194 kB)
Collecting deepmerge==0.3.0
Downloading deepmerge-0.3.0-py2.py3-none-any.whl (7.6 kB)
Collecting pyocd<=0.31.0,>=0.28.3
Downloading pyocd-0.31.0-py3-none-any.whl (12.5 MB)
Collecting click-option-group<0.6,>=0.3.0
Downloading click_option_group-0.5.5-py3-none-any.whl (12 kB)
Collecting pycryptodome<4,>=3.9.3
Downloading pycryptodome-3.17-cp35-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
Collecting pyocd-pemicro<1.2.0,>=1.1.1
Downloading pyocd_pemicro-1.1.5-py3-none-any.whl (9.0 kB)
Requirement already satisfied: colorama<1,>=0.4.4 in /usr/lib/python3/dist-packages (from spsdk<1.8.0,>=1.7.0->pynitrokey) (0.4.4)
Collecting commentjson<1,>=0.9
Downloading commentjson-0.9.0.tar.gz (8.7 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: asn1crypto<2,>=1.2 in /usr/lib/python3/dist-packages (from spsdk<1.8.0,>=1.7.0->pynitrokey) (1.4.0)
Collecting pypemicro<0.2.0,>=0.1.9
Downloading pypemicro-0.1.11-py3-none-any.whl (5.7 MB)
Collecting libusbsio>=2.1.11
Downloading libusbsio-2.1.11-py3-none-any.whl (247 kB)
Collecting sly==0.4
Downloading sly-0.4.tar.gz (60 kB)
Preparing metadata (setup.py) ... done
Collecting ruamel.yaml<0.18.0,>=0.17
Downloading ruamel.yaml-0.17.21-py3-none-any.whl (109 kB)
Collecting cmsis-pack-manager<0.3.0
Downloading cmsis_pack_manager-0.2.10-py2.py3-none-manylinux1_x86_64.whl (25.1 MB)
Collecting click-command-tree==1.1.0
Downloading click_command_tree-1.1.0-py3-none-any.whl (3.6 kB)
Requirement already satisfied: bitstring<3.2,>=3.1 in /usr/lib/python3/dist-packages (from spsdk<1.8.0,>=1.7.0->pynitrokey) (3.1.7)
Collecting hexdump~=3.3
Downloading hexdump-3.3.zip (12 kB)
Preparing metadata (setup.py) ... done
Collecting fire
Downloading fire-0.5.0.tar.gz (88 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/lib/python3/dist-packages (from astunparse<2,>=1.6->spsdk<1.8.0,>=1.7.0->pynitrokey) (0.37.1)
Collecting humanfriendly
Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
Collecting argparse-addons>=0.4.0
Downloading argparse_addons-0.12.0-py3-none-any.whl (3.3 kB)
Collecting pyelftools
Downloading pyelftools-0.29-py2.py3-none-any.whl (174 kB)
Collecting milksnake>=0.1.2
Downloading milksnake-0.1.5-py2.py3-none-any.whl (9.6 kB)
Requirement already satisfied: appdirs>=1.4 in /usr/lib/python3/dist-packages (from cmsis-pack-manager<0.3.0->spsdk<1.8.0,>=1.7.0->pynitrokey) (1.4.4)
Collecting lark-parser<0.8.0,>=0.7.1
Downloading lark-parser-0.7.8.tar.gz (276 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: MarkupSafe>=2.0 in /usr/lib/python3/dist-packages (from jinja2<3.1,>=2.11->spsdk<1.8.0,>=1.7.0->pynitrokey) (2.0.1)
Collecting asn1crypto<2,>=1.2
Downloading asn1crypto-1.5.1-py2.py3-none-any.whl (105 kB)
Collecting wrapt
Downloading wrapt-1.15.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (78 kB)
Collecting future
Downloading future-0.18.3.tar.gz (840 kB)
Preparing metadata (setup.py) ... done
Collecting psutil>=5.2.2
Downloading psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)
Collecting capstone<5.0,>=4.0
Downloading capstone-4.0.2-py2.py3-none-manylinux1_x86_64.whl (2.1 MB)
Collecting naturalsort<2.0,>=1.5
Downloading naturalsort-1.5.1.tar.gz (7.4 kB)
Preparing metadata (setup.py) ... done
Collecting prettytable<3.0,>=2.0
Downloading prettytable-2.5.0-py3-none-any.whl (24 kB)
Collecting intervaltree<4.0,>=3.0.2
Downloading intervaltree-3.1.0.tar.gz (32 kB)
Preparing metadata (setup.py) ... done
Collecting ruamel.yaml.clib>=0.2.6
Downloading ruamel.yaml.clib-0.2.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (485 kB)
Collecting termcolor
Downloading termcolor-2.2.0-py3-none-any.whl (6.6 kB)
Collecting sortedcontainers<3.0,>=2.0
Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Requirement already satisfied: wcwidth in /usr/lib/python3/dist-packages (from prettytable<3.0,>=2.0->pyocd<=0.31.0,>=0.28.3->spsdk<1.8.0,>=1.7.0->pynitrokey) (0.2.5)
Building wheels for collected packages: nrfutil, crcmod, sly, tlv8, commentjson, hexdump, pyspinel, fire, intervaltree, lark-parser, naturalsort, future
Building wheel for nrfutil (setup.py) ... done
Created wheel for nrfutil: filename=nrfutil-6.1.7-py3-none-any.whl size=898520 sha256=de6f8803f51d6c26d24dc7df6292064a468ff3f389d73370433fde5582b84a10
Stored in directory: /home/jas/.cache/pip/wheels/39/2b/9b/98ab2dd716da746290e6728bdb557b14c1c9a54cb9ed86e13b
Building wheel for crcmod (setup.py) ... done
Created wheel for crcmod: filename=crcmod-1.7-cp310-cp310-linux_x86_64.whl size=31422 sha256=5149ac56fcbfa0606760eef5220fcedc66be560adf68cf38c604af3ad0e4a8b0
Stored in directory: /home/jas/.cache/pip/wheels/85/4c/07/72215c529bd59d67e3dac29711d7aba1b692f543c808ba9e86
Building wheel for sly (setup.py) ... done
Created wheel for sly: filename=sly-0.4-py3-none-any.whl size=27352 sha256=f614e413918de45c73d1e9a8dca61ca07dc760d9740553400efc234c891f7fde
Stored in directory: /home/jas/.cache/pip/wheels/a2/23/4a/6a84282a0d2c29f003012dc565b3126e427972e8b8157ea51f
Building wheel for tlv8 (setup.py) ... done
Created wheel for tlv8: filename=tlv8-0.10.0-py3-none-any.whl size=11266 sha256=3ec8b3c45977a3addbc66b7b99e1d81b146607c3a269502b9b5651900a0e2d08
Stored in directory: /home/jas/.cache/pip/wheels/e9/35/86/66a473cc2abb0c7f21ed39c30a3b2219b16bd2cdb4b33cfc2c
Building wheel for commentjson (setup.py) ... done
Created wheel for commentjson: filename=commentjson-0.9.0-py3-none-any.whl size=12092 sha256=28b6413132d6d7798a18cf8c76885dc69f676ea763ffcb08775a3c2c43444f4a
Stored in directory: /home/jas/.cache/pip/wheels/7d/90/23/6358a234ca5b4ec0866d447079b97fedf9883387d1d7d074e5
Building wheel for hexdump (setup.py) ... done
Created wheel for hexdump: filename=hexdump-3.3-py3-none-any.whl size=8913 sha256=79dfadd42edbc9acaeac1987464f2df4053784fff18b96408c1309b74fd09f50
Stored in directory: /home/jas/.cache/pip/wheels/26/28/f7/f47d7ecd9ae44c4457e72c8bb617ef18ab332ee2b2a1047e87
Building wheel for pyspinel (setup.py) ... done
Created wheel for pyspinel: filename=pyspinel-1.0.3-py3-none-any.whl size=65033 sha256=01dc27f81f28b4830a0cf2336dc737ef309a1287fcf33f57a8a4c5bed3b5f0a6
Stored in directory: /home/jas/.cache/pip/wheels/95/ec/4b/6e3e2ee18e7292d26a65659f75d07411a6e69158bb05507590
Building wheel for fire (setup.py) ... done
Created wheel for fire: filename=fire-0.5.0-py2.py3-none-any.whl size=116951 sha256=3d288585478c91a6914629eb739ea789828eb2d0267febc7c5390cb24ba153e8
Stored in directory: /home/jas/.cache/pip/wheels/90/d4/f7/9404e5db0116bd4d43e5666eaa3e70ab53723e1e3ea40c9a95
Building wheel for intervaltree (setup.py) ... done
Created wheel for intervaltree: filename=intervaltree-3.1.0-py2.py3-none-any.whl size=26119 sha256=5ff1def22ba883af25c90d90ef7c6518496fcd47dd2cbc53a57ec04cd60dc21d
Stored in directory: /home/jas/.cache/pip/wheels/fa/80/8c/43488a924a046b733b64de3fac99252674c892a4c3801c0a61
Building wheel for lark-parser (setup.py) ... done
Created wheel for lark-parser: filename=lark_parser-0.7.8-py2.py3-none-any.whl size=62527 sha256=3d2ec1d0f926fc2688d40777f7ef93c9986f874169132b1af590b6afc038f4be
Stored in directory: /home/jas/.cache/pip/wheels/29/30/94/33e8b58318aa05cb1842b365843036e0280af5983abb966b83
Building wheel for naturalsort (setup.py) ... done
Created wheel for naturalsort: filename=naturalsort-1.5.1-py3-none-any.whl size=7526 sha256=bdecac4a49f2416924548cae6c124c85d5333e9e61c563232678ed182969d453
Stored in directory: /home/jas/.cache/pip/wheels/a6/8e/c9/98cfa614fff2979b457fa2d9ad45ec85fa417e7e3e2e43be51
Building wheel for future (setup.py) ... done
Created wheel for future: filename=future-0.18.3-py3-none-any.whl size=492037 sha256=57a01e68feca2b5563f5f624141267f399082d2f05f55886f71b5d6e6cf2b02c
Stored in directory: /home/jas/.cache/pip/wheels/5e/a9/47/f118e66afd12240e4662752cc22cefae5d97275623aa8ef57d
Successfully built nrfutil crcmod sly tlv8 commentjson hexdump pyspinel fire intervaltree lark-parser naturalsort future
Installing collected packages: tlv8, sortedcontainers, sly, pyserial, pyelftools, piccata, naturalsort, libusb1, lark-parser, intelhex, hexdump, fastjsonschema, crcmod, asn1crypto, wrapt, urllib3, typing_extensions, tqdm, termcolor, ruamel.yaml.clib, python-dateutil, pyspinel, pypemicro, pycryptodome, psutil, protobuf, prettytable, oscrypto, milksnake, libusbsio, jinja2, intervaltree, humanfriendly, future, frozendict, fido2, ecdsa, deepmerge, commentjson, click-option-group, click-command-tree, capstone, astunparse, argparse-addons, ruamel.yaml, pyocd-pemicro, pylink-square, pc_ble_driver_py, fire, cmsis-pack-manager, bincopy, pyocd, nrfutil, nkdfu, spsdk, pynitrokey
WARNING: The script nitropy is installed in '/home/jas/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed argparse-addons-0.12.0 asn1crypto-1.5.1 astunparse-1.6.3 bincopy-17.10.3 capstone-4.0.2 click-command-tree-1.1.0 click-option-group-0.5.5 cmsis-pack-manager-0.2.10 commentjson-0.9.0 crcmod-1.7 deepmerge-0.3.0 ecdsa-0.18.0 fastjsonschema-2.16.3 fido2-1.1.0 fire-0.5.0 frozendict-2.3.5 future-0.18.3 hexdump-3.3 humanfriendly-10.0 intelhex-2.3.0 intervaltree-3.1.0 jinja2-3.0.3 lark-parser-0.7.8 libusb1-1.9.3 libusbsio-2.1.11 milksnake-0.1.5 naturalsort-1.5.1 nkdfu-0.2 nrfutil-6.1.7 oscrypto-1.3.0 pc_ble_driver_py-0.17.0 piccata-2.0.3 prettytable-2.5.0 protobuf-3.20.3 psutil-5.9.4 pycryptodome-3.17 pyelftools-0.29 pylink-square-0.11.1 pynitrokey-0.4.34 pyocd-0.31.0 pyocd-pemicro-1.1.5 pypemicro-0.1.11 pyserial-3.5 pyspinel-1.0.3 python-dateutil-2.7.5 ruamel.yaml-0.17.21 ruamel.yaml.clib-0.2.7 sly-0.4 sortedcontainers-2.4.0 spsdk-1.7.1 termcolor-2.2.0 tlv8-0.10.0 tqdm-4.65.0 typing_extensions-4.3.0 urllib3-1.26.15 wrapt-1.15.0
jas@kaka:~$
Then upgrading the device worked remarkable well, although I wish that the tool would have printed URLs and checksums for the firmware files to allow easy confirmation.
jas@kaka:~$ PATH=$PATH:/home/jas/.local/bin
jas@kaka:~$ nitropy start list
Command line tool to interact with Nitrokey devices 0.4.34
:: 'Nitrokey Start' keys:
FSIJ-1.2.15-5D271572: Nitrokey Nitrokey Start (RTM.12.1-RC2-modified)
jas@kaka:~$ nitropy start update
Command line tool to interact with Nitrokey devices 0.4.34
Nitrokey Start firmware update tool
Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.35
System: Linux, is_linux: True
Python: 3.10.6
Saving run log to: /tmp/nitropy.log.gc5753a8
Admin PIN:
Firmware data to be used:
- FirmwareType.REGNUAL: 4408, hash: ...b'72a30389' valid (from ...built/RTM.13/regnual.bin)
- FirmwareType.GNUK: 129024, hash: ...b'25a4289b' valid (from ...prebuilt/RTM.13/gnuk.bin)
Currently connected device strings:
Device:
Vendor: Nitrokey
Product: Nitrokey Start
Serial: FSIJ-1.2.15-5D271572
Revision: RTM.12.1-RC2-modified
Config: *:*:8e82
Sys: 3.0
Board: NITROKEY-START-G
initial device strings: [ 'name': '', 'Vendor': 'Nitrokey', 'Product': 'Nitrokey Start', 'Serial': 'FSIJ-1.2.15-5D271572', 'Revision': 'RTM.12.1-RC2-modified', 'Config': '*:*:8e82', 'Sys': '3.0', 'Board': 'NITROKEY-START-G' ]
Please note:
- Latest firmware available is:
RTM.13 (published: 2022-12-08T10:59:11Z)
- provided firmware: None
- all data will be removed from the device!
- do not interrupt update process - the device may not run properly!
- the process should not take more than 1 minute
Do you want to continue? [yes/no]: yes
...
Starting bootloader upload procedure
Device: Nitrokey Start FSIJ-1.2.15-5D271572
Connected to the device
Running update!
Do NOT remove the device from the USB slot, until further notice
Downloading flash upgrade program...
Executing flash upgrade...
Waiting for device to appear:
Wait 20 seconds.....
Downloading the program
Protecting device
Finish flashing
Resetting device
Update procedure finished. Device could be removed from USB slot.
Currently connected device strings (after upgrade):
Device:
Vendor: Nitrokey
Product: Nitrokey Start
Serial: FSIJ-1.2.19-5D271572
Revision: RTM.13
Config: *:*:8e82
Sys: 3.0
Board: NITROKEY-START-G
device can now be safely removed from the USB slot
final device strings: [ 'name': '', 'Vendor': 'Nitrokey', 'Product': 'Nitrokey Start', 'Serial': 'FSIJ-1.2.19-5D271572', 'Revision': 'RTM.13', 'Config': '*:*:8e82', 'Sys': '3.0', 'Board': 'NITROKEY-START-G' ]
finishing session 2023-03-16 21:49:07.371291
Log saved to: /tmp/nitropy.log.gc5753a8
jas@kaka:~$
jas@kaka:~$ nitropy start list
Command line tool to interact with Nitrokey devices 0.4.34
:: 'Nitrokey Start' keys:
FSIJ-1.2.19-5D271572: Nitrokey Nitrokey Start (RTM.13)
jas@kaka:~$
Before importing the master key to this device, it should be configured. Note the commands in the beginning to make sure scdaemon/pcscd is not running because they may have cached state from earlier cards. Change PIN code as you like after this, my experience with Gnuk was that the Admin PIN had to be changed first, then you import the key, and then you change the PIN.
jas@kaka:~$ gpg-connect-agent "SCD KILLSCD" "SCD BYE" /bye
OK
ERR 67125247 Slut p fil <GPG Agent>
jas@kaka:~$ ps auxww grep -e pcsc -e scd
jas 11651 0.0 0.0 3468 1672 pts/0 R+ 21:54 0:00 grep --color=auto -e pcsc -e scd
jas@kaka:~$ gpg --card-edit
Reader ...........: 20A0:4211:FSIJ-1.2.19-5D271572:0
Application ID ...: D276000124010200FFFE5D2715720000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 5D271572
Name of cardholder: [not set]
Language prefs ...: [not set]
Salutation .......:
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
KDF setting ......: off
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]
gpg/card> admin
Admin commands are allowed
gpg/card> kdf-setup
gpg/card> passwd
gpg: OpenPGP card no. D276000124010200FFFE5D2715720000 detected
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
Your selection? 3
PIN changed.
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
Your selection? q
gpg/card> name
Cardholder's surname: Josefsson
Cardholder's given name: Simon
gpg/card> lang
Language preferences: sv
gpg/card> sex
Salutation (M = Mr., F = Ms., or space): m
gpg/card> login
Login data (account name): jas
gpg/card> url
URL to retrieve public key: https://josefsson.org/key-20190320.txt
gpg/card> forcesig
gpg/card> key-attr
Changing card key attribute for: Signature key
Please select what kind of key you want:
(1) RSA
(2) ECC
Your selection? 2
Please select which elliptic curve you want:
(1) Curve 25519
(4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: ed25519
Note: There is no guarantee that the card supports the requested size.
If the key generation does not succeed, please check the
documentation of your card to see what sizes are allowed.
Changing card key attribute for: Encryption key
Please select what kind of key you want:
(1) RSA
(2) ECC
Your selection? 2
Please select which elliptic curve you want:
(1) Curve 25519
(4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: cv25519
Changing card key attribute for: Authentication key
Please select what kind of key you want:
(1) RSA
(2) ECC
Your selection? 2
Please select which elliptic curve you want:
(1) Curve 25519
(4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: ed25519
gpg/card>
jas@kaka:~$ gpg --card-edit
Reader ...........: 20A0:4211:FSIJ-1.2.19-5D271572:0
Application ID ...: D276000124010200FFFE5D2715720000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 5D271572
Name of cardholder: Simon Josefsson
Language prefs ...: sv
Salutation .......: Mr.
URL of public key : https://josefsson.org/key-20190320.txt
Login data .......: jas
Signature PIN ....: not forced
Key attributes ...: ed25519 cv25519 ed25519
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
KDF setting ......: on
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]
jas@kaka:~$
Once setup, bring out your offline machine and boot it and mount your USB stick with the offline key. The paths below will be different, and this is using a somewhat unorthodox approach of working with fresh GnuPG configuration paths that I chose for the USB stick.
jas@kaka:/media/jas/2c699cbd-b77e-4434-a0d6-0c4965864296$ cp -a gnupghome-backup-masterkey gnupghome-import-nitrokey-5D271572
jas@kaka:/media/jas/2c699cbd-b77e-4434-a0d6-0c4965864296$ gpg --homedir $PWD/gnupghome-import-nitrokey-5D271572 --edit-key B1D2BD1375BECB784CF4F8C4D73CF638C53C06BE
gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Secret key is available.
sec ed25519/D73CF638C53C06BE
created: 2019-03-20 expired: 2019-10-22 usage: SC
trust: ultimate validity: expired
[ expired] (1). Simon Josefsson <simon@josefsson.org>
gpg> keytocard
Really move the primary key? (y/N) y
Please select where to store the key:
(1) Signature key
(3) Authentication key
Your selection? 1
sec ed25519/D73CF638C53C06BE
created: 2019-03-20 expired: 2019-10-22 usage: SC
trust: ultimate validity: expired
[ expired] (1). Simon Josefsson <simon@josefsson.org>
gpg>
Save changes? (y/N) y
jas@kaka:/media/jas/2c699cbd-b77e-4434-a0d6-0c4965864296$
At this point it is useful to confirm that the Nitrokey has the master key available and that is possible to sign statements with it, back on your regular machine:
jas@kaka:~$ gpg --card-status
Reader ...........: 20A0:4211:FSIJ-1.2.19-5D271572:0
Application ID ...: D276000124010200FFFE5D2715720000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 5D271572
Name of cardholder: Simon Josefsson
Language prefs ...: sv
Salutation .......: Mr.
URL of public key : https://josefsson.org/key-20190320.txt
Login data .......: jas
Signature PIN ....: not forced
Key attributes ...: ed25519 cv25519 ed25519
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 1
KDF setting ......: on
Signature key ....: B1D2 BD13 75BE CB78 4CF4 F8C4 D73C F638 C53C 06BE
created ....: 2019-03-20 23:37:24
Encryption key....: [none]
Authentication key: [none]
General key info..: pub ed25519/D73CF638C53C06BE 2019-03-20 Simon Josefsson <simon@josefsson.org>
sec> ed25519/D73CF638C53C06BE created: 2019-03-20 expires: 2023-09-19
card-no: FFFE 5D271572
ssb> ed25519/80260EE8A9B92B2B created: 2019-03-20 expires: 2023-09-19
card-no: FFFE 42315277
ssb> ed25519/51722B08FE4745A2 created: 2019-03-20 expires: 2023-09-19
card-no: FFFE 42315277
ssb> cv25519/02923D7EE76EBD60 created: 2019-03-20 expires: 2023-09-19
card-no: FFFE 42315277
jas@kaka:~$ echo foo gpg -a --sign gpg --verify
gpg: Signature made Thu Mar 16 22:11:02 2023 CET
gpg: using EDDSA key B1D2BD1375BECB784CF4F8C4D73CF638C53C06BE
gpg: Good signature from "Simon Josefsson <simon@josefsson.org>" [ultimate]
jas@kaka:~$
Finally to retrieve and sign a key, for example Andre Heinecke s that I could confirm the OpenPGP key identifier from his business card.
jas@kaka:~$ gpg --locate-external-keys aheinecke@gnupg.com
gpg: key 1FDF723CF462B6B1: public key "Andre Heinecke <aheinecke@gnupg.com>" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 2 signed: 7 trust: 0-, 0q, 0n, 0m, 0f, 2u
gpg: depth: 1 valid: 7 signed: 64 trust: 7-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2023-05-26
pub rsa3072 2015-12-08 [SC] [expires: 2025-12-05]
94A5C9A03C2FE5CA3B095D8E1FDF723CF462B6B1
uid [ unknown] Andre Heinecke <aheinecke@gnupg.com>
sub ed25519 2017-02-13 [S]
sub ed25519 2017-02-13 [A]
sub rsa3072 2015-12-08 [E] [expires: 2025-12-05]
sub rsa3072 2015-12-08 [A] [expires: 2025-12-05]
jas@kaka:~$ gpg --edit-key "94A5C9A03C2FE5CA3B095D8E1FDF723CF462B6B1"
gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
pub rsa3072/1FDF723CF462B6B1
created: 2015-12-08 expires: 2025-12-05 usage: SC
trust: unknown validity: unknown
sub ed25519/2978E9D40CBABA5C
created: 2017-02-13 expires: never usage: S
sub ed25519/DC74D901C8E2DD47
created: 2017-02-13 expires: never usage: A
The following key was revoked on 2017-02-23 by RSA key 1FDF723CF462B6B1 Andre Heinecke <aheinecke@gnupg.com>
sub cv25519/1FFE3151683260AB
created: 2017-02-13 revoked: 2017-02-23 usage: E
sub rsa3072/8CC999BDAA45C71F
created: 2015-12-08 expires: 2025-12-05 usage: E
sub rsa3072/6304A4B539CE444A
created: 2015-12-08 expires: 2025-12-05 usage: A
[ unknown] (1). Andre Heinecke <aheinecke@gnupg.com>
gpg> sign
pub rsa3072/1FDF723CF462B6B1
created: 2015-12-08 expires: 2025-12-05 usage: SC
trust: unknown validity: unknown
Primary key fingerprint: 94A5 C9A0 3C2F E5CA 3B09 5D8E 1FDF 723C F462 B6B1
Andre Heinecke <aheinecke@gnupg.com>
This key is due to expire on 2025-12-05.
Are you sure that you want to sign this key with your
key "Simon Josefsson <simon@josefsson.org>" (D73CF638C53C06BE)
Really sign? (y/N) y
gpg> quit
Save changes? (y/N) y
jas@kaka:~$
This is on my day-to-day machine, using the NitroKey Start with the offline key. No need to boot the old offline machine just to sign keys or extend expiry anymore! At FOSDEM 23 I managed to get at least one DD signature on my new key, and the Debian keyring maintainers accepted my Ed25519 key. Hopefully I can now finally let my 2014-era RSA3744 key expire in 2023-09-19 and not extend it any further. This should finish my transition to a simpler OpenPGP key setup, yay!
Ken MacLeod is a Scottish science fiction writer who has become amusingly
famous for repeatedly winning the libertarian Prometheus Award despite
being a (somewhat libertarian-leaning) socialist. The Star
Fraction is the first of a loose series of four novels about future solar
system politics and was nominated for the Clarke Award (as well as winning the Prometheus). It was MacLeod's
first novel.
Moh Kohn is a mercenary, part of the Felix Dzerzhinsky Workers' Defence
collective. They're available for hire to protect research labs and
universities against raids from people such as animal liberationists and
anti-AI extremists (or, as Moh calls them, creeps and cranks). As
The Star Fraction opens, he and his smart gun are protecting a lab
against an attack.
Janis Taine is a biologist who is currently testing a memory-enhancing
drug on mice. It's her lab that is attacked, although it isn't vandalized
the way she expected. Instead, the attackers ruined her experiment by
releasing the test drug into the air, contaminating all of the controls.
This sets off a sequence of events that results in Moh, Janis, and Jordon
Brown, a stock trader for a religious theocracy, on the run from the US/UN
and Space Defense.
I had forgotten what it was like to read the uncompromising old-school
style of science fiction novel that throws you into the world and explains
nothing, leaving it to the reader to piece the world together as you go.
It's weirdly fun, but I'm either out of practice or this was a
particularly challenging example of the genre. MacLeod throws a lot of
characters at you quickly, including some that have long and complicated
personal histories, and it's not until well into the book that the pieces
start to cohere into a narrative. Even once that happens, the
relationship between the characters and the plot is unobvious until late
in the book, and comes from a surprising direction.
Science fiction as a genre is weirdly conservative about political
systems. Despite the grand, futuristic ideas and the speculation about
strange alien societies, the human governments rarely rise to the
sophistication of a modern democracy. There are a lot of empires,
oligarchies, and hand-waved libertarian semi-utopias, but not a lot of
deep engagement with the speculative variety of government systems humans
have proposed. The rare exceptions therefore get a lot of attention from
those of us who find political systems fascinating.
MacLeod has a reputation for writing political SF in that sense, and
The Star Fraction certainly delivers. Moh (despite the name of his
collective, which is explained briefly in the book) is a Trotskyist with a
family history with the Fourth International that is central to the plot.
The setting is a politically fractured Britain full of autonomous zones
with wildly different forms of government, theoretically ruled by a
restored monarchy. That monarchy is opposed by the Army of the New
Republic, which claims to be the legitimate government of the United
Kingdom and is considered by everyone else to be terrorists. Hovering in
the background is a UN entirely subsumed by the US, playing global
policeman over a chaotic world shattered by numerous small-scale wars.
This satisfyingly different political world is a major plus for me. The
main drawback is that I found the world-building and politics more
interesting than the characters. It's not that I disliked them; I found
them enjoyably quirky and odd. It's more that so much is happening and
there are so many significant characters, all set in an unfamiliar and
unexplained world and often divided into short scenes of a few pages, that
I had a hard time keeping track of them all. Part of the point of
The Star Fraction is digging into their tangled past and connecting
it up with the present, but the flashbacks added a confused timeline on
top of the other complexity and made it hard for me to get lost in the
story. The characters felt a bit too much like puzzle pieces until the
very end of the book.
The technology is an odd mix with a very 1990s feel. MacLeod is one of
the SF authors who can make computers and viruses believable, avoiding the
cyberpunk traps, but AI becomes relevant to the plot and the conception of
AI here feels oddly retro. (Not MacLeod's fault; it's been nearly 30
years and a lot has changed.) On-line discussion in the book is still
based on newsgroups, which added to the nostalgic feel. I did like the
eventual explanation for the computing part of the plot, though; I can't
say much while avoiding spoilers, but it's one of the more believable
explanations for how a technology could spread in a way required for the
plot that I've read.
I've been planning on reading this series for years but never got around
to it. I enjoyed my last try at a MacLeod
series well enough to want to keep reading, but not well enough to keep
reading immediately, and then other books happened and now it's been 19
years. I feel similarly about The Star Fraction: it's good enough
(and in a rare enough subgenre of SF) that I want to keep reading, but not
enough to keep reading immediately. We'll see if I manage to get to the
next book in a reasonable length of time.
Followed by The Stone Canal.
Rating: 6 out of 10
Posted on February 13, 2023
Some months ago I stumbled on the video where Irving Finkel teaches Tom Scott how to play the Royal Game of Ur and my takeout was:
Irving Finkel is Gandalf or something;
the game sounded quite fun!;
so I did the almost sensible thing, quickly drew a board with inkscape, printed it on 160 g/m paper and used my piecepackpieces to try a few games.
I say almost sensible, because rather than drawing the rosettes with inkscape I decided to carve a rubber stamp and use that to print them on the board (which is why the svgs on this page are missing them: if you print them you ll have to add the rosettes in some way).
And if I had been a sensible person, that s where I would have stopped, since that s perfectly enough to play games and find out that it actually is quite a fun game, and one of our staples.
As some of you probably know, I m not a sensible person.
I also have quite a few blocks and half-blocks of cernit, and one day after I ve had used some, my hands were still moving and accidentally made some pyramidal dice, and a handful of tokens.
And after baking and trying them I liked them, but they had not been planned in any way, and they were a bit too small for the board, so the next time I was using cernit I tried to make a new set.
And while I was doing that I tried a new shape for the dice, as coins marked with a dot in the middle of one of the sides, because I don t really like tetrahedral dice.
And now, I realized this wasn t going to be my last set, and urgently felt the need for some container to keep them in and avoid missing pieces.
(Yes, in the picture above one piece was already missing. While taking it I didn t realize it, and neither I did when picking up everything to put it away, getting the missing piece and storing it safely together with the rest of the set. It must have been hiding in plain sight nearby, but I will never know where.)
Anyway, back to Inkscape, and to a board printed on scrap paper that I tried to fold up until I came up with a layout that folded up in a small drawer, and then I added a case to wrap around it to keep it closed.
I played around with the case until it was big enough to actually slide around the folded board, and this is the result, ready to be printed out on A4 paper, cut, folded and glued. (This takes most of the sheet, and I m not sure that the case would still fit around the board/drawer if printed with scaling, so if you want to print it on Letter paper I d recommend to move the pieces around.)
Now, the only problem left was that green isn t really my colour, and while I did like the stone effect of this set, I wasn t exactly pleased by the colour scheme. (why did I do it this way in the first place? probably because I was trying to use up old cernit blocks before opening new ones.)
So, the only possible way out was to make yet another set, right?
I still used stone effect cernit, but this time in a red/grey scheme that knew I would have liked more, and while I was doing it I tried a few improvements on the randomization devices.
The tetrahedral dice are still the same: they work, it s what they use in the replica sets, so I keep making them even if they re not my first choice.
I ve changed the coins to make them almost square for two reasons, however: one is that the round one tended to roll away into inconvenient places when throwing them with emphasis, and the other one is to make it easier to recognise them from the tokens with no need to flip each one around before starting the game.
The lozenges were a bit of a failure, instead. They work fine when thrown, but I don t think that there is a self-evident way to decide which side should be counted, and the only intuitive way I can think of (count the ones in the player s colour) would be unbalanced.
Speaking of balance issues: of course the hand-modelled dice and coins aren t perfectly balanced but:
they don t feel obviously unbalanced;
both players use the same set, so any subtle unbalance isn t going to affect the chance of winning in an uneven way.
Maybe one day I will find a way to easily roll them a statistically significant number of times, collect data and analyze it to find out how imbalanced they are, but that s not going to happen with manual data collecting, and I m not really ready to go down the yak shaving filled road to automatize it.
To wrap up: is it going to be the last set I make for the Royal Game of Ur? lol. Is it going to be the last cernit set I make this month? definitely yes, I now have one I m happy with, I m routinely playing with it and I m currently doing other crafts rather than cernit.
I was passing through London on Friday and I had time to get to The Horror Show! Exhibit at Somerset House, over by Victoria embankment. I learned about the exhibition from Gazelle Twin s website: she wrote music for the final part of the exhibit, with Maxine Peake delivering a monologue over the top.
I loved it. It was almost like David Keenan s book England s Hidden Reverse had been made into an exhibition. It s divided into three themes: Monster, Ghost and Witch, although the themes are loose with lots of cross over and threads running throughout.
Thatcher (right)
Derek Jarman's Blue
The show is littered with artefacts from culturally significant works from a recently-bygone age. There s a strong theme of hauntology. The artefacts that stuck out to me include some of Leigh Bowery s costumes; the spitting image doll of Thatcher; the cover of a Radio Times featuring the cult BBC drama Threads; Nigel Kneale s the stone tape VHS alongside more recent artefacts from Inside Number 9 and a MR James adaptation by Mark Gatiss (a clear thread of inspiration there); various bits relating to Derek Jarman including the complete Blue screening in a separate room; Mica Levi s eerie score to under the skin playing in the Witch section, and various artefacts and references to the underground music group Coil. Too much to mention!
Having said that, the things which left the most lasting impression are the some of the stand-alone works of art: the charity box boy model staring fractured and refracted through a glass door (above); the glossy drip of blood running down a wall; the performance piece on a Betamax tape; the self portrait of a drowned man; the final piece, "The Neon Heiroglyph".
Jonathan Jones at the Guardian liked it.
The show runs until the 19th February and is worth a visit if you can.
To achieve my goal of publishing one high-quality piece of writing per week this year, I ve put together a draft writing plan and a few organisational notes.
Please let me know what you think - what s missing? what would you like to read more/less of from me?
I aim for each piece of writing to generate discussion, inspire further writing, and raise my visibility and profile with potential customers and peers. Some of the writing will be opinion, but I expect a majority of it will take a learning by teaching approach - aiming to explain and present useful information to the reader while helping me learn more!
Topic Backlog
The majority of my writing is going to fit into 4 series, allowing me to plan out a set of posts and narrative rather than having to come up with something novel to write about every week. To start with for Feb, my aim is to get an initial post in each series out the door. Long-term, it s likely that the order of posts will reflect my work focus (e.g. if I m spending a few weeks deep-diving into a particular product idea then expect more writing on that), but I will try and maintain some variety across the different series as well.
This backlog will be maintained as a living page at https://www.mattb.nz/w/queue.
Thoughts on SRE
This series of posts will be pitched primarily at potential consulting customers who want to understand how I approach the development and operations of distributed software systems. Initial topics to cover include:
What is SRE? My philosophy on how it relates to DevOps, Platform Engineering and various other hot terms.
How SRE scales up and down in size.
My approach to managing oncall responsibilities, toil and operational work.
How to grow an SRE team, including the common futility of SRE transformations .
Learning from incidents, postmortems, incident response, etc.
Business plan drafts
I have an ever-growing list of potential software opportunities and products which I think would be fun to build, but which generally don t ever leave my head due to lack of time to develop the idea, or being unable to convince myself that there s a viable business case or market for it.
I d like to start sharing some very rudimentary business plan sketches for some of these ideas as a way of getting some feedback on my assessment of their potential. Whether that s confirmation that it s not worth pursuing, an expression of interest in the product, or potential partnership/collaboration opportunities - anything is better than the idea just sitting in my head.
Initial ideas include:
Business oriented Mastodon hosting.
PDF E-signing - e.g. A Docusign competitor, but with a local twist through RealMe or drivers license validation.
A framework to enable simple, performant per-tenant at-rest encryption for SaaS products - stop the data leaks.
Product development updates
For any product ideas that show merit and develop into a project, and particularly for the existing product ideas I ve already committed to exploring, I plan to document my product investigation and market research findings as a way of structuring and driving my learning in the space.
To start with this will involve:
A series of explanatory posts diving into how NZ s electricity system works with a particular focus on how operational data that will be critical to managing a more dynamic grid flows (or doesn t flow!) today, and what opportunities or needs exist for generating, managing or distributing data that might be solvable with a software system I could build.
A series of product reviews and deep dives into existing farm management software and platforms in use by NZ farmers today, looking at the functionality they provide, how they integrate and generally testing the anecdotal feedback I have to date that they re clunky, hard to use and not well integrated.
For co2mon.nz the focus will be less on market research and more on exploring potential distribution channels (e.g. direct advertising vs partnership with air conditioning suppliers) and pricing models (e.g. buy vs rent).
Debugging walk-throughs
Being able to debug and fix a system that you re not intimately familiar with is valuable skill and something that I ve always enjoyed, but it s also a skill that I observe many engineers are uncomfortable with. There s a set of techniques and processes that I ve honed and developed over the years for doing this which I think make the task of debugging an unfamiliar system more approachable.
The idea, is that each post will take a problem or situation I ve encountered, from the initial symptom or problem report and walk through the process of how to narrow down and identify the trigger or root cause of the behaviour. Along the way, discussing techniques used, their pros and cons. In addition to learning about the process of debugging itself, the aim is to illustrate lessons that can be applied when designing and building software systems that facilitate and improve our experiences in the operational stage of a systems lifecycle where debugging takes place.
Miscellaneous topics
In addition the regular series above, stand-alone posts on the other topics may include:
The pros/cons I see of bootstrapping a business vs taking VC or other funding.
Thoughts on remote work and hiring staff.
AI - a confessional on how I didn t think it would progress in my lifetime, but maybe I was wrong.
Reflections on 15 years at Google and thoughts on subsequent events since my departure.
AWS vs GCP. Fight! Or with less click-bait, a level-headed comparison of the pros/cons I see in each platform.
Logistics
Discussion and comments
A large part of my motivation for writing regularly is to seek feedback and generate discussion on these topics. Typically this is done by including comment functionality within the website itself. I ve decided not to do this - on-site commenting creates extra infrastructure to maintain, and limits the visibility and breadth of discussion to existing readers and followers.
To provide opportunities for comment and feedback I plan to share and post notification and summarised snippets of selected posts to various social media platforms. Links to these social media posts will be added to each piece of writing to provide a path for readers to engage and discuss further while enabling the discussion and visibility of the post to grow and extend beyond my direct followers and subscribers.
My current thinking is that I ll distribute via the following platforms:
Twitter @xleem - selected posts. I m trying to reduce Twitter usage in favour of Mastodon, but there s no denying that it s still where a significant number of people and discussions are happening.
LinkedIn - probably primarily for posts in the business plan series, and notable milestones in the product development process.
In all cases, my aim will be to post a short teaser or summary paragraph that poses an question or relays an interesting fact to give some immediate value and signal to readers as to whether they want to click through rather than simply spamming links into the feed.
Feedback
In addition to social media discussion I also plan to add a direct feedback path, particularly for readers who don t have time or inclination to participate in written discussion, by providing a simple thumbs up/thumbs down feedback widget to the bottom of each post, including those delivered via RSS and email.
Organisation
To enable subscription to subsets of my writing (particularly for places like Planet Debian, etc where the more business focused content is likely to be off-topic), I plan to place each post into a set of categories:
Business
Technology
General
In addition to the categories, I ll also use more free-form tags to group writing with linked themes or that falls within one of the series described above.
So, it s this weird time of year where we make a balance and share
with the world some ideas about the future. And yes, it s time to
take care of this blog, as its activity has dropped once
again. So maybe it d be nice to start this post by checking how
much have I blogged over the years:
2004: 27
2005: 92
2006: 65
2007: 83
2008: 64
2009: 62
2010: 48
2011: 25
2012: 27
2013: 29
2014: 37
2015: 18
2016: 19
2017: 20
2018: 19
2019: 19
2020: 14
2021: 11
2022: 10
(yes, this is an old blog)
So yes, there is a clear downwards trend towards the last few
years. And it does make sense, all in all: Not only have I managed to
keep myself busier than before, but Blogging is a social
endeavor. And as people have moved over to the different flavors of
social networks, there is somewhat less fueling us to share our
thoughts and experiences in this fashion.
So this connects me to my first point: Staring at Noodles
Emptiness,
I got to a campaign to Bring Back
Blogging. I stand by all of what they
suggest: Blogs are a great invention, they allow the sharing of a
great insight into a person s mind, ideas and worldview (and even
more so if, like mine, it shows already a window of almost two
decades of life! This year my blog will be old enough to vote!), they
are completely decentralized, and can be easily grouped according to
each readers preferences via the RSS format.
Anyway I do want to write a post summing up 2022, as well as
sharing some hopes and projects I have for 2023. But I don t want to
make it too long to read. So That shall be the blog post for today!
(post caption image by Walt Stoneburner on Flickr; CC BY
2.0)
This year had a lot of things happen in it, world-wide, as is always the case.
Being more selfish here are the things I remember, in brief unless there are comments/questions:
I learned more Finnish.
Lots of things with our child.
I helped teach him to swim.
He learned to tell the time with an analog clock/watch.
I took him to a circus for the first (only) time ever.
He cut his hair for the first time in six years.
He spent his a birthday with my parents, in the UK - His languages skills were on top-form, understanding the various UK accents.
And another with family here in Finland - Where he watched me roll, naked, in the snow after sauna, and then he asked "Daddy why you did that?"
On the topic of Finnish I'm getting pretty damn good at understanding, albeit less good in speaking. Finnish is all about the suffixes, so:
A car
auto
My car
autoni
In a car
autossa
In my car
autossani
With my car
autoll ni
From a car
autosta
Car-less
autoton
Most of this is regular, so you can be childless via the "ton" suffix - lapsiton is "lapsi" (child) "ton" (less). The hard part in communication is thus twofold:
Knowing the word you want to use, be it car, cake, spoon, or smile.
Getting the appropriate suffix for the use you want.
Our child turned six recently, and most of the year was spent doing things with him, for him, and to him. He's on the verge of learning to read (English and Finnish), he's interested in maths and completes little puzzles freely and happily. He likes to help with Sodoku, for example and not just the child-versions.
In the past couple of weeks I let him play Super Mario & Super Mario Bros 3, on a NES Classic, and he dies constantly, with a smile on his face. But he does love to tell me what to do when he watches me play!
He's learned to ice-skate, and ski, and almost learned to swim. (I'll say he can swim 3m inside a pool, without aid, but then he starts to sink.) We've got a couple of regular rituals with each other - including going to sauna every week or two, and other similar things.
He's gotten more interested in helping me cook, and his mother too. (My wife and I live in separate houses..)
I guess the next big milestone will be him walking to school by himself, which will start next year. As things stand I wake up early, go over to his house, and do all the morning-things, before I take him there. I expect I'll still want to go there, give him his breakfast, his medicine, and help him get dressed. After that I guess I kick him out, and he makes his own way there.
Happily the walk to school is a few hundred meters, and doesn't involve crossing any roads. But of course it does bring other complications: if he's not collected, and walks home himself, then he needs a key to one/other house, and there's the potential need for a phone to say "I'm late", "I'm lost", or us to say "Where are you?".
Anyway .. interesting year .. good year. Despite everything else.
The Unbroken is the first book of a projected fantasy trilogy. It
is C.L. Clark's first novel.
Lieutenant Touraine is one of the Sands, the derogatory name for the
Balladairan Colonial Brigade. She, like the others of her squad, are
conscript soldiers, kidnapped by the Balladairan Empire from their
colonies as children and beaten into "civilized" behavior by Balladairan
training. They fought in the Balladairan war against the Taargens. Now,
they've been reassigned to El-Wast, capital city of Qaz l, the foremost of
the southern colonies. The place where Touraine was born, from which she
was taken at the age of five.
Balladaire is not France and Qaz l is not Algeria, but the parallels are
obvious and strongly implied by the map and the climates.
Touraine and her squad are part of the forces accompanying Princess Luca,
the crown princess of the Balladairan Empire, who has been sent to take
charge of Qaz l and quell a rebellion. Luca's parents died in the
Withering, the latest round of a recurrent plague that haunts Balladaire.
She is the rightful heir, but her uncle rules as regent and is reluctant
to give her the throne. Qaz l is where she is to prove herself. If she
can bring the colony in line, she can prove that she's ready to rule: her
birthright and her destiny.
The Qaz li are uninterested in being part of Luca's grand plan of personal
accomplishment. She steps off her ship into an assassination attempt,
foiled by Touraine's sharp eyes and quick reactions, which brings the Sand
to the princess's attention. Touraine's reward is to be assigned the
execution of the captured rebels, one of whom recognizes her and names her
mother before he dies. This sets up the core of the plot: Qaz li
rebellion against an oppressive colonial empire, Luca's attempt to use the
colony as a political stepping stone, and Touraine caught in between.
One of the reasons why I am happy to see increased diversity in SFF
authors is that the way we tell stories is shaped by our cultural
upbringing. I was taught to tell stories about colonialism and rebellion
in a specific ideological shape. It's hard to describe briefly, but the
core idea is that being under the rule of someone else is unnatural as
well as being an injustice. It's a deviation from the way the world
should work, something unexpected that is inherently unstable. Once
people unite to overthrow their oppressors, eventual success is
inevitable; it's not only right or moral, it's the natural path of
history. This is what you get when you try to peel the supremacy part
away from white supremacy but leave the unshakable self-confidence and
bedrock assumption that the universe cares what we think.
We were also taught that rebellion is primarily ideological. One may be
motivated by personal injustice, but the correct use of that injustice is
to subsume it into concepts such as freedom and democracy. Those concepts
are more "real" in some foundational sense, more central to the right
functioning of the world, than individual circumstance. When the
now-dominant group tells stories of long-ago revolution, there is no
personal experience of oppression and survival in which to ground the
story; instead, it's linked to anticipatory fear in the reader, to the
idea that one's privileges could be taken away by a foreign oppressor and
that the counter to this threat is ideological unity.
Obviously, not every white fantasy author uses this story shape, but the
tendency runs deep because we're taught it young. You can see it
everywhere in fantasy, from Lord of the Rings to
Tigana. The Unbroken uses a much
different story shape, and I don't think it's a coincidence that the
author is Black.
Touraine is not sympathetic to the Qaz li. These are not her people and
this is not her life. She went through hell in Balladairan schools, but
she won a place, however tenuous. Her personal role model is General
Cantic, the Balladairan Blood General who was also one of her instructors.
Cantic is hard as nails, unforgiving, unbending, and probably a war
criminal, but also the embodiment of a military ethic. She is tough but
fair with the conscript soldiers. She doesn't put a stop to their
harassment by the regular Balladairan troops, but neither does she let it
go too far. Cantic has power, she knows how to keep it, and there is a
place for Touraine in Cantic's world.
And, critically, that place is not just hers: it's one she shares with her
squad. Touraine's primary loyalty is not to Balladaire or to Qaz l. It's
to the Sands. Her soldiers are neither one thing nor the other, and they
disagree vehemently among themselves about what Qaz l and their other
colonial homes should be to them, but they learned together, fought
together, and died together. That theme is woven throughout The
Unbroken: personal bonds, third and fourth loyalties, and practical
ethics of survival that complicate and contradict simple dichotomies of
oppressor and oppressed.
Touraine is repeatedly offered ideological motives that the protagonist in
the typical story shape would adopt. And she repeatedly rejects them for
personal bonds: trying to keep her people safe, in a world that is not
looking out for them.
The consequence is that this book tears Touraine apart. She tries to walk
a precarious path between Luca, the Qaz li, Cantic, and the Sands, and she
falls off that path a lot. Each time I thought I knew where this book was
going, there's another reversal, often brutal. I tend to be a
happily-ever-after reader who wants the protagonist to get everything they
need, so this isn't my normal fare. The amount of hell that Touraine goes
through made for difficult reading, worse because much of it is due to her
own mistakes or betrayals. But Clark makes those decisions believable
given the impossible position Touraine is in and the lack of role models
she has for making other choices. She's set up to fail, and the price of
small victories is to have no one understand the decisions that she makes,
or to believe her motives.
Luca is the other viewpoint character of the book (and yes, this is also a
love affair, which complicates both of their loyalties). She is the
heroine of a more typical genre fantasy novel: the outsider princess with
a physical disability and a razor-sharp mind, ambitious but fair (at least
in her own mind), with a trusted bodyguard advisor who also knew her
father and a sincere desire to be kinder and more even-handed in her
governance of the colony. All of this is real; Luca is a protagonist, and
the reader is not being set up to dislike her. But compared to Touraine's
grappling with identity, loyalty, and ethics, Luca is never in any real
danger, and her concerns start to feel too calculated and superficial.
It's hard to be seriously invested in whether Luca proves herself or gets
her throne when people are being slaughtered and abused.
This, I think, is the best part of this book. Clark tells a traditional
ideological fantasy of learning to be a good ruler, but she puts it
alongside a much deeper and more complex story of multi-faceted
oppression. She has the two protagonists fall in love with each other and
challenges them to understand each other, and Luca does not come off well
in this comparison. Touraine is frustrated, impulsive, physical, and
sometimes has catastrophically poor judgment. Luca is analytical and
calculating, and in most ways understands the political dynamics far
better than Touraine. We know how this story usually goes: Luca sees
Touraine's brilliance and lifts her out of the ranks into a role of
importance and influence, which Touraine should reward with loyalty. But
Touraine's world is more real, more grounded, and more authentic, and both
Touraine and the reader know what Luca could offer is contingent and comes
with a higher price than Luca understands.
(Incidentally, the cover of The Unbroken, designed by Lauren
Panepinto with art by Tommy Arnold, is astonishingly good at capturing
both Touraine's character and the overall feeling of the book. Here's a
larger version.)
The writing is good but uneven. Clark loves reversals, and they did keep
me reading, but I think there were too many of them. By the end of the
book, the escalation of betrayals and setbacks was more exhausting than
exciting, and I'd stopped trusting anything good would last. (Admittedly,
this is an accurate reflection of how Touraine felt.) Touraine's inner
monologue also gets a bit repetitive when she's thrashing in the jaws of
an emotional trap. I think some of this is first-novel problems of
over-explaining emotional states and character reasoning, but these
problems combine to make the book feel a bit over-long.
I'm also not in love with the ending. It's perhaps the one place in the
book where I am more cynical about the politics than Clark is, although
she does lay the groundwork for it.
But this book is also full of places small and large where it goes a
different direction than most fantasy and is better for it. I think my
favorite small moment is Touraine's quiet refusal to defend herself
against certain insinuations. This is such a beautiful bit of
characterization; she knows she won't be believed anyway, and refuses to
demean herself by trying.
I'm not sure I can recommend this book unconditionally, since I think you
have to be in the mood for it, but it's one of the most thoughtful and
nuanced looks at colonialism and rebellion I can remember seeing in
fantasy. I found it frustrating in places, but I'm also still thinking
about it. If you're looking for a political fantasy with teeth, you could
do a lot worse, although expect to come out the other side a bit battered
and bruised.
Followed by The Faithless, and I have no idea where Clark is going
to go with the second book. I suppose I'll have to read and find out.
Content note: In addition to a lot of violence, gore, and death, including
significant character death, there's also a major plague. If you're not
feeling up to reading about panic caused by contageous illness, proceed
with caution.
Rating: 7 out of 10
The Fifth Elephant is the 24th Discworld and fifth Watch novel, and
largely assumes you know who the main characters are. This is not a good
place to start.
The dwarves are electing a new king. The resulting political conflict is
spilling over into the streets of Ankh-Morpork, but that's not the primary
problem. First, the replica Scone of Stone, a dwarven artifact used to
crown the Low King of the Dwarves, is stolen from the Dwarf Bread Museum.
Then, Vimes is dispatched to berwald, ostensibly to negotiate increased
fat exports with the new dwarven king. And then Angua disappears,
apparently headed towards her childhood home in berwald, which
immediately prompts Carrot to resign and head after her. The City Watch
is left in the hands of now-promoted Captain Colon.
We see lots of Lady Sybil for the first time since
Guards! Guards!, and there's a
substantial secondary plot with Angua and Carrot and a tertiary plot with
Colon making a complete mess of things back home, but this is mostly a
Vimes novel. As usual, Vetinari is pushing him outside of his comfort
zone, but he's not seriously expecting Vimes to act like an ambassador.
He's expecting Vimes to act like a policeman, even though he's way outside
his jurisdiction. This time, that means untangling a messy three-sided
political situation involving the dwarves, the werewolves, and the
vampires.
There is some Igor dialogue in this book, but
thankfully Pratchett toned it down a lot and it never started to bother
me.
I do enjoy Pratchett throwing Vimes and his suspicious morality at
political problems and watching him go at them sideways. Vimes's
definition of crimes is just broad enough to get him fully invested in a
problem, but too narrow to give him much patience with the diplomatic
maneuvering. It makes him an unpredictable diplomat in a clash of
cultures way that's fun to read about. Cheery and Detritus are great
traveling companions for this, since both of them also unsettle the
dwarves in wildly different ways.
I also have to admit that Pratchett is doing more interesting things with
the Angua and Carrot relationship than I had feared. In previous books, I
was getting tired of their lack of communication and wasn't buying the
justifications for it, but I think I finally understand why the
communication barriers are there. It's not that Angua refuses to talk to
Carrot (although there's still a bit of that going on). It's that
Carrot's attitude towards the world is very strange, and gets
stranger the closer you are to him.
Carrot has always been the character who is too earnest and
straightforward and good for Ankh-Morpork and yet somehow makes it work,
but Pratchett is doing something even more interesting with the concept of
nobility. A sufficiently overwhelming level of heroic ethics becomes
almost alien, so contrary to how people normally think that it can make
conversations baffling. It's not that Carrot is perfect (sometimes he
does very dumb things), it's that his natural behavior follows a set of
ethics that humans like to pretend they follow but actually don't and
never would entirely. His character should be a boring cliche or an
over-the-top parody, and yet he isn't at all.
But Carrot's part is mostly a side plot. Even more than
Jingo, The Fifth Elephant is
establishing Vimes as a force to be reckoned with, even if you take him
outside his familiar city. He is in so many ways the opposite of
Vetinari, and yet he's a tool that Vetinari is extremely good at using.
Colon of course is a total disaster as the head of the Watch, and that's
mostly because Colon should never be more than a sergeant, but it's also
because even when he's taking the same action as Vimes, he's not doing it
for the same reasons or with the same stubborn core of basic morality and
loyalty that's under Vimes's suspicious conservatism.
The characterization in the Watch novels doesn't seem that subtle or deep
at first, but it accumulates over the course of the series in a way that I
think is more effective than any of the other story strands. Vetinari,
Vimes, and Carrot all represent "right," or at least order, in overlapping
stories of right versus wrong, but they do so in radically different ways
and with radically different goals. Each time one of them seems
ascendant, each time one of their approaches seems more clearly correct,
Pratchett throws them at a problem where a different approach is required.
It's a great reading experience.
This was one of the better Discworld novels even though I found the
villains to be a bit tedious and stupid. Recommended.
Followed by The Truth in publication order. The next Watch novel
is Night Watch.
Rating: 8 out of 10
In my tubman setup, I started using ZFS on an old server
I had lying around. The machine is really old though (2011!) and it
"feels" pretty slow. I want to see how much of that is ZFS and how
much is the machine. Synthetic benchmarks show that ZFS may be slower
than mdadm in RAID-10 or RAID-6 configuration, so I want to confirm
that on a live workload: my workstation. Plus, I want easy, regular,
high performance backups (with send/receive snapshots) and there's no
way I'm going to use BTRFS because I find
it too confusing and unreliable.
So off we go.
Installation
Since this is a conversion (and not a new install), our procedure is
slightly different than the official documentation but otherwise
it's pretty much in the same spirit: we're going to use ZFS for
everything, including the root filesystem.
So, install the required packages, on the current system:
root@curie:/home/anarcat# sgdisk -p /dev/sdc
Disk /dev/sdc: 1953525168 sectors, 931.5 GiB
Model: ESD-S1C
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): [REDACTED]
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 16-sector boundaries
Total free space is 14 sectors (7.0 KiB)
Number Start (sector) End (sector) Size Code Name
1 48 2047 1000.0 KiB EF02
2 2048 1050623 512.0 MiB EF00
3 1050624 3147775 1024.0 MiB BF01
4 3147776 1953525134 930.0 GiB BF00
Unfortunately, we can't be sure of the sector size here, because the
USB controller is probably lying to us about it. Normally, this
smartctl command should tell us the sector size as well:
root@curie:~# smartctl -i /dev/sdb -qnoserial
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-14-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Black Mobile
Device Model: WDC WD10JPLX-00MBPT0
Firmware Version: 01.01H01
User Capacity: 1 000 204 886 016 bytes [1,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 6
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue May 17 13:33:04 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Above is the example of the builtin HDD drive. But the SSD device
enclosed in that USB controller doesn't support SMART commands,
so we can't trust that it really has 512 bytes sectors.
This matters because we need to tweak the ashift value
correctly. We're going to go ahead the SSD drive has the common 4KB
settings, which means ashift=12.
Note here that we are not creating a separate partition for
swap. Swap on ZFS volumes (AKA "swap on ZVOL") can trigger lockups and
that issue is still not fixed upstream. Ubuntu recommends using a
separate partition for swap instead. But since this is "just" a
workstation, we're betting that we will not suffer from this problem,
after hearing a report from another Debian developer running this
setup on their workstation successfully.
We do not recommend this setup though. In fact, if I were to redo this
partition scheme, I would probably use LUKS encryption and setup a
dedicated swap partition, as I had problems with ZFS encryption as
well.
Creating pools
ZFS pools are somewhat like "volume groups" if you are familiar with
LVM, except they obviously also do things like RAID-10. (Even though
LVM can technically also do RAID, people typically use mdadm
instead.)
In any case, the guide suggests creating two different pools here:
one, in cleartext, for boot, and a separate, encrypted one, for the
rest. Technically, the boot partition is required because the Grub
bootloader only supports readonly ZFS pools, from what I
understand. But I'm a little out of my depth here and just following
the guide.
Boot pool creation
This creates the boot pool in readonly mode with features that grub
supports:
-O encryption=on -O keylocation=prompt -O keyformat=passphrase:
encryption, prompt for a password, default algorithm is
aes-256-gcm, explicit in the guide, made implicit here
-O acltype=posixacl -O xattr=sa: enable ACLs, with better
performance (not enabled by default)
-O dnodesize=auto: related to extended attributes, less
compatibility with other implementations
-O compression=zstd: enable zstd compression, can be
disabled/enabled by dataset to with zfs set compression=off
rpool/example
-O relatime=on: classic atime optimisation, another that could
be used on a busy server is atime=off
-O canmount=off: do not make the pool mount automatically with
mount -a?
-O mountpoint=/ -R /mnt: mount pool on / in the future, but
/mnt for now
Those settings are all available in zfsprops(8). Other flags are
defined in zpool-create(8). The reasoning behind them is also
explained in the upstream guide and some also in [the Debian
wiki][]. Those flags were actually not used:
-O normalization=formD: normalize file names on comparisons (not
storage), implies utf8only=on, which is a bad idea (and
effectively meant my first sync failed to copy some files,
including this folder from a supysonic checkout). and this
cannot be changed after the filesystem is created. bad, bad, bad.
Side note about single-disk pools
Also note that we're living dangerously here: single-disk ZFS pools
are rumoured to be more dangerous than not running ZFS at all. The
choice quote from this article is:
[...] any error can be detected, but cannot be corrected. This
sounds like an acceptable compromise, but its actually not. The
reason its not is that ZFS' metadata cannot be allowed to be
corrupted. If it is it is likely the zpool will be impossible to
mount (and will probably crash the system once the corruption is
found). So a couple of bad sectors in the right place will mean that
all data on the zpool will be lost. Not some, all. Also there's no
ZFS recovery tools, so you cannot recover any data on the drives.
Compared with (say) ext4, where a single disk error can recovered,
this is pretty bad. But we are ready to live with this with the idea
that we'll have hourly offline snapshots that we can easily recover
from. It's trade-off. Also, we're running this on a NVMe/M.2 drive
which typically just blinks out of existence completely, and doesn't
"bit rot" the way a HDD would.
Also, the FreeBSD handbook quick start doesn't have any warnings
about their first example, which is with a single disk. So I am
reassured at least.
Creating mount points
Next we create the actual filesystems, known as "datasets" which are
the things that get mounted on mountpoint and hold the actual files.
Note that it's unclear to me why those datasets are necessary, but
they seem common practice, also used in this FreeBSD
example. The OpenZFS guide mentions the Solaris upgrades and
Ubuntu's zsys that use that container for upgrades and rollbacks.
This blog post seems to explain a bit the layout behind the
installer.
this creates the actual boot and root filesystems:
Notice here a peculiarity: we must create rpool/var/lib to
create rpool/var/lib/docker otherwise we get this error:
cannot create 'rpool/var/lib/docker': parent does not exist
... and no, just creating /mnt/var/lib doesn't fix that
problem. In fact, it makes things even more confusing because an
existing directory shadows a mountpoint, which is the opposite of
how things normally work.
Also note that you will probably need to change storage driver in
Docker, see the zfs-driver documentation for details but,
basically, I did:
Now that we have everything setup and mounted, let's copy all files
over.
Copying files
This is a list of all the mounted filesystems
for fs in /boot/ /boot/efi/ / /home/; do
echo "syncing $fs to /mnt$fs..." &&
rsync -aSHAXx --info=progress2 --delete $fs /mnt$fs
done
You can check that the list is correct with:
mount -l -t ext4,btrfs,vfat awk ' print $3 '
Note that we skip /srv as it's on a different disk.
On the first run, we had:
root@curie:~# for fs in /boot/ /boot/efi/ / /home/; do
echo "syncing $fs to /mnt$fs..." &&
rsync -aSHAXx --info=progress2 $fs /mnt$fs
done
syncing /boot/ to /mnt/boot/...
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/299)
syncing /boot/efi/ to /mnt/boot/efi/...
16,831,437 100% 184.14MB/s 0:00:00 (xfr#101, to-chk=0/110)
syncing / to /mnt/...
28,019,293,280 94% 47.63MB/s 0:09:21 (xfr#703710, ir-chk=6748/839220)rsync: [generator] delete_file: rmdir(var/lib/docker) failed: Device or resource busy (16)
could not make way for new symlink: var/lib/docker
34,081,267,990 98% 50.71MB/s 0:10:40 (xfr#736577, to-chk=0/867732)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]
syncing /home/ to /mnt/home/...
rsync: [sender] readlink_stat("/home/anarcat/.fuse") failed: Permission denied (13)
24,456,268,098 98% 68.03MB/s 0:05:42 (xfr#159867, ir-chk=6875/172377)
file has vanished: "/home/anarcat/.cache/mozilla/firefox/s2hwvqbu.quantum/cache2/entries/B3AB0CDA9C4454B3C1197E5A22669DF8EE849D90"
199,762,528,125 93% 74.82MB/s 0:42:26 (xfr#1437846, ir-chk=1018/1983979)rsync: [generator] recv_generator: mkdir "/mnt/home/anarcat/dist/supysonic/tests/assets/\#346" failed: Invalid or incomplete multibyte or wide character (84)
*** Skipping any contents from this failed directory ***
315,384,723,978 96% 76.82MB/s 1:05:15 (xfr#2256473, to-chk=0/2993950)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]
Note the failure to transfer that supysonic file? It turns out they
had a weird filename in their source tree, since then removed,
but still it showed how the utf8only feature might not be such a bad
idea. At this point, the procedure was restarted all the way back to
"Creating pools", after unmounting all ZFS filesystems (umount
/mnt/run /mnt/boot/efi && umount -t zfs -a) and destroying the pool,
which, surprisingly, doesn't require any confirmation (zpool destroy
rpool).
The second run was cleaner:
root@curie:~# for fs in /boot/ /boot/efi/ / /home/; do
echo "syncing $fs to /mnt$fs..." &&
rsync -aSHAXx --info=progress2 --delete $fs /mnt$fs
done
syncing /boot/ to /mnt/boot/...
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/299)
syncing /boot/efi/ to /mnt/boot/efi/...
0 0% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/110)
syncing / to /mnt/...
28,019,033,070 97% 42.03MB/s 0:10:35 (xfr#703671, ir-chk=1093/833515)rsync: [generator] delete_file: rmdir(var/lib/docker) failed: Device or resource busy (16)
could not make way for new symlink: var/lib/docker
34,081,807,102 98% 44.84MB/s 0:12:04 (xfr#736580, to-chk=0/867723)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]
syncing /home/ to /mnt/home/...
rsync: [sender] readlink_stat("/home/anarcat/.fuse") failed: Permission denied (13)
IO error encountered -- skipping file deletion
24,043,086,450 96% 62.03MB/s 0:06:09 (xfr#151819, ir-chk=15117/172571)
file has vanished: "/home/anarcat/.cache/mozilla/firefox/s2hwvqbu.quantum/cache2/entries/4C1FDBFEA976FF924D062FB990B24B897A77B84B"
315,423,626,507 96% 67.09MB/s 1:14:43 (xfr#2256845, to-chk=0/2994364)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1333) [sender=3.2.3]
Also note the transfer speed: we seem capped at 76MB/s, or
608Mbit/s. This is not as fast as I was expecting: the USB connection
seems to be at around 5Gbps:
anarcat@curie:~$ lsusb -tv head -4
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
ID 1d6b:0003 Linux Foundation 3.0 root hub
__ Port 1: Dev 4, If 0, Class=Mass Storage, Driver=uas, 5000M
ID 0b05:1932 ASUSTek Computer, Inc.
So it shouldn't cap at that speed. It's possible the USB adapter is
failing to give me the full speed though. It's not the M.2 SSD drive
either, as that has a ~500MB/s bandwidth, acccording to its spec.
At this point, we're about ready to do the final configuration. We
drop to single user mode and do the rest of the procedure. That used
to be shutdown now, but it seems like the systemd switch broke that,
so now you can reboot into grub and pick the "recovery"
option. Alternatively, you might try systemctl rescue, as I found
out.
I also wanted to copy the drive over to another new NVMe drive, but
that failed: it looks like the USB controller I have doesn't work with
older, non-NVME drives.
Boot configuration
Now we need to enter the new system to rebuild the boot loader and
initrd and so on.
First, we bind mounts and chroot into the ZFS disk:
mount --rbind /dev /mnt/dev &&
mount --rbind /proc /mnt/proc &&
mount --rbind /sys /mnt/sys &&
chroot /mnt /bin/bash
Next we add an extra service that imports the bpool on boot, to make
sure it survives a zpool.cache destruction:
I had to trim down /etc/fstab and /etc/crypttab to only contain
references to the legacy filesystems (/srv is still BTRFS!).
If we don't already have a tmpfs defined in /etc/fstab:
Strangely, that's not exactly what the author, Jim Salter, did in
his actual test bench used in the ZFS benchmarking
article. The first thing is there's no read test at all, which is
already pretty strange. But also it doesn't include stuff like
dropping caches or repeating results.
So here's my variation, which i called fio-ars-bench.sh for
now. It just batches a bunch of fio tests, one by one, 60 seconds
each. It should take about 12 minutes to run, as there are 3 pair of
tests, read/write, with and without async.
My bias, before building, running and analysing those results is that
ZFS should outperform the traditional stack on writes, but possibly
not on reads. It's also possible it outperforms it on both, because
it's a newer drive. A new test might be possible with a new external
USB drive as well, although I doubt I will find the time to do this.
Results
All tests were done on WD blue SN550 drives, which claims to be
able to push 2400MB/s read and 1750MB/s write. An extra drive was
bought to move the LVM setup from a WDC WDS500G1B0B-00AS40 SSD, a WD
blue M.2 2280 SSD that was at least 5 years old, spec'd at 560MB/s
read, 530MB/s write. Benchmarks were done on the M.2 SSD drive but
discarded so that the drive difference is not a factor in the test.
In practice, I'm going to assume we'll never reach those numbers
because we're not actually NVMe (this is an old workstation!) so the
bottleneck isn't the disk itself. For our purposes, it might still
give us useful results.
Rescue test, LUKS/LVM/ext4
Those tests were performed with everything shutdown, after either
entering the system in rescue mode, or by reaching that target with:
systemctl rescue
The network might have been started before or after the test as well:
systemctl start systemd-networkd
So it should be fairly reliable as basically nothing else is running.
Raw numbers, from the ?job-curie-lvm.log, converted to MiB/s and
manually merged:
test
read I/O
read IOPS
write I/O
write IOPS
rand4k4g1x
39.27
10052
212.15
54310
rand4k4g1x--fsync=1
39.29
10057
2.73
699
rand64k256m16x
1297.00
20751
1068.57
17097
rand64k256m16x--fsync=1
1290.90
20654
353.82
5661
rand1m16g1x
315.15
315
563.77
563
rand1m16g1x--fsync=1
345.88
345
157.01
157
Peaks are at about 20k IOPS and ~1.3GiB/s read, 1GiB/s write in the
64KB blocks with 16 jobs.
Slowest is the random 4k block sync write at an abysmal 3MB/s and 700
IOPS The 1MB read/write tests have lower IOPS, but that is expected.
Rescue test, ZFS
This test was also performed in rescue mode.
Raw numbers, from the ?job-curie-zfs.log, converted to MiB/s and
manually merged:
test
read I/O
read IOPS
write I/O
write IOPS
rand4k4g1x
77.20
19763
27.13
6944
rand4k4g1x--fsync=1
76.16
19495
6.53
1673
rand64k256m16x
1882.40
30118
70.58
1129
rand64k256m16x--fsync=1
1865.13
29842
71.98
1151
rand1m16g1x
921.62
921
102.21
102
rand1m16g1x--fsync=1
908.37
908
64.30
64
Peaks are at 1.8GiB/s read, also in the 64k job like above, but much
faster. The write is, as expected, much slower at 70MiB/s (compared
to 1GiB/s!), but it should be noted the sync write doesn't degrade
performance compared to async writes (although it's still below the
LVM 300MB/s).
Conclusions
Really, ZFS has trouble performing in all write conditions. The
random 4k sync write test is the only place where ZFS outperforms
LVM in writes, and barely (7MiB/s vs 3MiB/s). Everywhere else, writes
are much slower, sometimes by an order of magnitude.
And before some ZFS zealot jumps in talking about the SLOG or some
other cache that could be added to improved performance, I'll remind
you that those numbers are on a bare bones NVMe drive, pretty much as
fast storage as you can find on this machine. Adding another NVMe
drive as a cache probably will not improve write performance here.
Still, those are very different results than the tests performed by
Salter which shows ZFS beating traditional configurations in all
categories but uncached 4k reads (not writes!). That said, those tests
are very different from the tests I performed here, where I test
writes on a single disk, not a RAID array, which might explain the
discrepancy.
Also, note that neither LVM or ZFS manage to reach the 2400MB/s read
and 1750MB/s write performance specification. ZFS does manage to reach
82% of the read performance (1973MB/s) and LVM 64% of the write
performance (1120MB/s). LVM hits 57% of the read performance and ZFS
hits barely 6% of the write performance.
Overall, I'm a bit disappointed in the ZFS write performance here, I
must say. Maybe I need to tweak the record size or some other ZFS
voodoo, but I'll note that I didn't have to do any such configuration
on the other side to kick ZFS in the pants...
Real world experience
This section document not synthetic backups, but actual real world
workloads, comparing before and after I switched my workstation to
ZFS.
Docker performance
I had the feeling that running some git hook (which was firing a
Docker container) was "slower" somehow. It seems that, at runtime, ZFS
backends are significant slower than their overlayfs/ext4 equivalent:
May 16 14:42:52 curie systemd[1]: home-docker-overlay2-17e4d24228decc2d2d493efc401dbfb7ac29739da0e46775e122078d9daf3e87\x2dinit-merged.mount: Succeeded.
May 16 14:42:52 curie systemd[5161]: home-docker-overlay2-17e4d24228decc2d2d493efc401dbfb7ac29739da0e46775e122078d9daf3e87\x2dinit-merged.mount: Succeeded.
May 16 14:42:52 curie systemd[1]: home-docker-overlay2-17e4d24228decc2d2d493efc401dbfb7ac29739da0e46775e122078d9daf3e87-merged.mount: Succeeded.
May 16 14:42:53 curie dockerd[1723]: time="2022-05-16T14:42:53.087219426-04:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/af22586fba07014a4d10ab19da10cf280db7a43cad804d6c1e9f2682f12b5f10 pid=151170
May 16 14:42:53 curie systemd[1]: Started libcontainer container af22586fba07014a4d10ab19da10cf280db7a43cad804d6c1e9f2682f12b5f10.
May 16 14:42:54 curie systemd[1]: docker-af22586fba07014a4d10ab19da10cf280db7a43cad804d6c1e9f2682f12b5f10.scope: Succeeded.
May 16 14:42:54 curie dockerd[1723]: time="2022-05-16T14:42:54.047297800-04:00" level=info msg="shim disconnected" id=af22586fba07014a4d10ab19da10cf280db7a43cad804d6c1e9f2682f12b5f10
May 16 14:42:54 curie dockerd[998]: time="2022-05-16T14:42:54.051365015-04:00" level=info msg="ignoring event" container=af22586fba07014a4d10ab19da10cf280db7a43cad804d6c1e9f2682f12b5f10 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
May 16 14:42:54 curie systemd[2444]: run-docker-netns-f5453c87c879.mount: Succeeded.
May 16 14:42:54 curie systemd[5161]: run-docker-netns-f5453c87c879.mount: Succeeded.
May 16 14:42:54 curie systemd[2444]: home-docker-overlay2-17e4d24228decc2d2d493efc401dbfb7ac29739da0e46775e122078d9daf3e87-merged.mount: Succeeded.
May 16 14:42:54 curie systemd[5161]: home-docker-overlay2-17e4d24228decc2d2d493efc401dbfb7ac29739da0e46775e122078d9daf3e87-merged.mount: Succeeded.
May 16 14:42:54 curie systemd[1]: run-docker-netns-f5453c87c879.mount: Succeeded.
May 16 14:42:54 curie systemd[1]: home-docker-overlay2-17e4d24228decc2d2d493efc401dbfb7ac29739da0e46775e122078d9daf3e87-merged.mount: Succeeded.
Translating this:
container setup: ~1 second
container runtime: ~1 second
container teardown: ~1 second
total runtime: 2-3 seconds
Obviously, those timestamps are not quite accurate enough to make
precise measurements...
After I switched to ZFS:
mai 30 15:31:39 curie systemd[1]: var-lib-docker-zfs-graph-41ce08fb7a1d3a9c101694b82722f5621c0b4819bd1d9f070933fd1e00543cdf\x2dinit.mount: Succeeded.
mai 30 15:31:39 curie systemd[5287]: var-lib-docker-zfs-graph-41ce08fb7a1d3a9c101694b82722f5621c0b4819bd1d9f070933fd1e00543cdf\x2dinit.mount: Succeeded.
mai 30 15:31:40 curie systemd[1]: var-lib-docker-zfs-graph-41ce08fb7a1d3a9c101694b82722f5621c0b4819bd1d9f070933fd1e00543cdf.mount: Succeeded.
mai 30 15:31:40 curie systemd[5287]: var-lib-docker-zfs-graph-41ce08fb7a1d3a9c101694b82722f5621c0b4819bd1d9f070933fd1e00543cdf.mount: Succeeded.
mai 30 15:31:41 curie dockerd[3199]: time="2022-05-30T15:31:41.551403693-04:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/42a1a1ed5912a7227148e997f442e7ab2e5cc3558aa3471548223c5888c9b142 pid=141080
mai 30 15:31:41 curie systemd[1]: run-docker-runtime\x2drunc-moby-42a1a1ed5912a7227148e997f442e7ab2e5cc3558aa3471548223c5888c9b142-runc.ZVcjvl.mount: Succeeded.
mai 30 15:31:41 curie systemd[5287]: run-docker-runtime\x2drunc-moby-42a1a1ed5912a7227148e997f442e7ab2e5cc3558aa3471548223c5888c9b142-runc.ZVcjvl.mount: Succeeded.
mai 30 15:31:41 curie systemd[1]: Started libcontainer container 42a1a1ed5912a7227148e997f442e7ab2e5cc3558aa3471548223c5888c9b142.
mai 30 15:31:45 curie systemd[1]: docker-42a1a1ed5912a7227148e997f442e7ab2e5cc3558aa3471548223c5888c9b142.scope: Succeeded.
mai 30 15:31:45 curie dockerd[3199]: time="2022-05-30T15:31:45.883019128-04:00" level=info msg="shim disconnected" id=42a1a1ed5912a7227148e997f442e7ab2e5cc3558aa3471548223c5888c9b142
mai 30 15:31:45 curie dockerd[1726]: time="2022-05-30T15:31:45.883064491-04:00" level=info msg="ignoring event" container=42a1a1ed5912a7227148e997f442e7ab2e5cc3558aa3471548223c5888c9b142 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
mai 30 15:31:45 curie systemd[1]: run-docker-netns-e45f5cf5f465.mount: Succeeded.
mai 30 15:31:45 curie systemd[5287]: run-docker-netns-e45f5cf5f465.mount: Succeeded.
mai 30 15:31:45 curie systemd[1]: var-lib-docker-zfs-graph-41ce08fb7a1d3a9c101694b82722f5621c0b4819bd1d9f070933fd1e00543cdf.mount: Succeeded.
mai 30 15:31:45 curie systemd[5287]: var-lib-docker-zfs-graph-41ce08fb7a1d3a9c101694b82722f5621c0b4819bd1d9f070933fd1e00543cdf.mount: Succeeded.
That's double or triple the run time, from 2 seconds to 6
seconds. Most of the time is spent in run time, inside the
container. Here's the breakdown:
container setup: ~2 seconds
container run: ~4 seconds
container teardown: ~1 second
total run time: about ~6-7 seconds
That's a two- to three-fold increase! Clearly something is going on
here that I should tweak. It's possible that code path is less
optimized in Docker. I also worry about podman, but apparently it
also supports ZFS backends. Possibly it would perform better, but
at this stage I wouldn't have a good comparison: maybe it would have
performed better on non-ZFS as well...
Interactivity
While doing the offsite backups (below), the system became somewhat
"sluggish". I felt everything was slow, and I estimate it introduced
~50ms latency in any input device.
Arguably, those are all USB and the external drive was connected
through USB, but I suspect the ZFS drivers are not as well tuned with
the scheduler as the regular filesystem drivers...
Recovery procedures
For test purposes, I unmounted all systems during the procedure:
umount /mnt/boot/efi /mnt/boot/run
umount -a -t zfs
zpool export -a
And disconnected the drive, to see how I would recover this system
from another Linux system in case of a total motherboard failure.
To import an existing pool, plug the device, then import the pool with
an alternate root, so it doesn't mount over your existing filesystems,
then you mount the root filesystem and all the others:
zpool import -l -a -R /mnt &&
zfs mount rpool/ROOT/debian &&
zfs mount -a &&
mount /dev/sdc2 /mnt/boot/efi &&
mount -t tmpfs tmpfs /mnt/run &&
mkdir /mnt/run/lock
Offsite backup
Part of the goal of using ZFS is to simplify and harden backups. I
wanted to experiment with shorter recovery times specifically both
point in time recovery objective and recovery time objective
and faster incremental backups.
This is, therefore, part of my backup services.
This section documents how an external NVMe enclosure was setup in a
pool to mirror the datasets from my workstation.
The final setup should include syncoid copying datasets to the backup
server regularly, but I haven't finished that configuration yet.
Partitioning
The above partitioning procedure used sgdisk, but I couldn't figure
out how to do this with sgdisk, so this uses sfdisk to dump the
partition from the first disk to an external, identical drive:
First sync
I used syncoid to copy all pools over to the external device. syncoid
is a thing that's part of the sanoid project which is
specifically designed to sync snapshots between pool, typically over
SSH links but it can also operate locally.
The sanoid command had a --readonly argument to simulate changes,
but syncoid didn't so I tried to fix that with an upstream PR.
It seems it would be better to do this by hand, but this was much
easier. The full first sync was:
root@curie:/home/anarcat# ./bin/syncoid -r bpool bpool-tubman
CRITICAL ERROR: Target bpool-tubman exists but has no snapshots matching with bpool!
Replication to target would require destroying existing
target. Cowardly refusing to destroy your existing target.
NOTE: Target bpool-tubman dataset is < 64MB used - did you mistakenly run
zfs create bpool-tubman on the target? ZFS initial
replication must be to a NON EXISTENT DATASET, which will
then be CREATED BY the initial replication process.
INFO: Sending oldest full snapshot bpool/BOOT@test (~ 42 KB) to new target filesystem:
44.2KiB 0:00:00 [4.19MiB/s] [========================================================================================================================] 103%
INFO: Updating new target filesystem with incremental bpool/BOOT@test ... syncoid_curie_2022-05-30:12:50:39 (~ 4 KB):
2.13KiB 0:00:00 [ 114KiB/s] [===============================================================> ] 53%
INFO: Sending oldest full snapshot bpool/BOOT/debian@install (~ 126.0 MB) to new target filesystem:
126MiB 0:00:00 [ 308MiB/s] [=======================================================================================================================>] 100%
INFO: Updating new target filesystem with incremental bpool/BOOT/debian@install ... syncoid_curie_2022-05-30:12:50:39 (~ 113.4 MB):
113MiB 0:00:00 [ 315MiB/s] [=======================================================================================================================>] 100%
root@curie:/home/anarcat# ./bin/syncoid -r rpool rpool-tubman
CRITICAL ERROR: Target rpool-tubman exists but has no snapshots matching with rpool!
Replication to target would require destroying existing
target. Cowardly refusing to destroy your existing target.
NOTE: Target rpool-tubman dataset is < 64MB used - did you mistakenly run
zfs create rpool-tubman on the target? ZFS initial
replication must be to a NON EXISTENT DATASET, which will
then be CREATED BY the initial replication process.
INFO: Sending oldest full snapshot rpool/ROOT@syncoid_curie_2022-05-30:12:50:51 (~ 69 KB) to new target filesystem:
44.2KiB 0:00:00 [2.44MiB/s] [===========================================================================> ] 63%
INFO: Sending oldest full snapshot rpool/ROOT/debian@install (~ 25.9 GB) to new target filesystem:
25.9GiB 0:03:33 [ 124MiB/s] [=======================================================================================================================>] 100%
INFO: Updating new target filesystem with incremental rpool/ROOT/debian@install ... syncoid_curie_2022-05-30:12:50:52 (~ 3.9 GB):
3.92GiB 0:00:33 [ 119MiB/s] [======================================================================================================================> ] 99%
INFO: Sending oldest full snapshot rpool/home@syncoid_curie_2022-05-30:12:55:04 (~ 276.8 GB) to new target filesystem:
277GiB 0:27:13 [ 174MiB/s] [=======================================================================================================================>] 100%
INFO: Sending oldest full snapshot rpool/home/root@syncoid_curie_2022-05-30:13:22:19 (~ 2.2 GB) to new target filesystem:
2.22GiB 0:00:25 [90.2MiB/s] [=======================================================================================================================>] 100%
INFO: Sending oldest full snapshot rpool/var@syncoid_curie_2022-05-30:13:22:47 (~ 5.6 GB) to new target filesystem:
5.56GiB 0:00:32 [ 176MiB/s] [=======================================================================================================================>] 100%
INFO: Sending oldest full snapshot rpool/var/cache@syncoid_curie_2022-05-30:13:23:22 (~ 627.3 MB) to new target filesystem:
627MiB 0:00:03 [ 169MiB/s] [=======================================================================================================================>] 100%
INFO: Sending oldest full snapshot rpool/var/lib@syncoid_curie_2022-05-30:13:23:28 (~ 69 KB) to new target filesystem:
44.2KiB 0:00:00 [1.40MiB/s] [===========================================================================> ] 63%
INFO: Sending oldest full snapshot rpool/var/lib/docker@syncoid_curie_2022-05-30:13:23:28 (~ 442.6 MB) to new target filesystem:
443MiB 0:00:04 [ 103MiB/s] [=======================================================================================================================>] 100%
INFO: Sending oldest full snapshot rpool/var/lib/docker/05c0de7fabbea60500eaa495d0d82038249f6faa63b12914737c4d71520e62c5@266253254 (~ 6.3 MB) to new target filesystem:
6.49MiB 0:00:00 [12.9MiB/s] [========================================================================================================================] 102%
INFO: Updating new target filesystem with incremental rpool/var/lib/docker/05c0de7fabbea60500eaa495d0d82038249f6faa63b12914737c4d71520e62c5@266253254 ... syncoid_curie_2022-05-30:13:23:34 (~ 4 KB):
1.52KiB 0:00:00 [27.6KiB/s] [============================================> ] 38%
INFO: Sending oldest full snapshot rpool/var/lib/flatpak@syncoid_curie_2022-05-30:13:23:36 (~ 2.0 GB) to new target filesystem:
2.00GiB 0:00:17 [ 115MiB/s] [=======================================================================================================================>] 100%
INFO: Sending oldest full snapshot rpool/var/tmp@syncoid_curie_2022-05-30:13:23:55 (~ 57.0 MB) to new target filesystem:
61.8MiB 0:00:01 [45.0MiB/s] [========================================================================================================================] 108%
INFO: Clone is recreated on target rpool-tubman/var/lib/docker/ed71ddd563a779ba6fb37b3b1d0cc2c11eca9b594e77b4b234867ebcb162b205 based on rpool/var/lib/docker/05c0de7fabbea60500eaa495d0d82038249f6faa63b12914737c4d71520e62c5@266253254
INFO: Sending oldest full snapshot rpool/var/lib/docker/ed71ddd563a779ba6fb37b3b1d0cc2c11eca9b594e77b4b234867ebcb162b205@syncoid_curie_2022-05-30:13:23:58 (~ 218.6 MB) to new target filesystem:
219MiB 0:00:01 [ 151MiB/s] [=======================================================================================================================>] 100%
Funny how the CRITICAL ERROR doesn't actually stop syncoid and it
just carries on merrily doing when it's telling you it's "cowardly
refusing to destroy your existing target"... Maybe that's because my pull
request broke something though...
During the transfer, the computer was very sluggish: everything feels
like it has ~30-50ms latency extra:
anarcat@curie:sanoid$ LANG=C top -b -n 1 head -20
top - 13:07:05 up 6 days, 4:01, 1 user, load average: 16.13, 16.55, 11.83
Tasks: 606 total, 6 running, 598 sleeping, 0 stopped, 2 zombie
%Cpu(s): 18.8 us, 72.5 sy, 1.2 ni, 5.0 id, 1.2 wa, 0.0 hi, 1.2 si, 0.0 st
MiB Mem : 15898.4 total, 1387.6 free, 13170.0 used, 1340.8 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 1319.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
70 root 20 0 0 0 0 S 83.3 0.0 6:12.67 kswapd0
4024878 root 20 0 282644 96432 10288 S 44.4 0.6 0:11.43 puppet
3896136 root 20 0 35328 16528 48 S 22.2 0.1 2:08.04 mbuffer
3896135 root 20 0 10328 776 168 R 16.7 0.0 1:22.93 zfs
3896138 root 20 0 10588 788 156 R 16.7 0.0 1:49.30 zfs
350 root 0 -20 0 0 0 R 11.1 0.0 1:03.53 z_rd_int
351 root 0 -20 0 0 0 S 11.1 0.0 1:04.15 z_rd_int
3896137 root 20 0 4384 352 244 R 11.1 0.0 0:44.73 pv
4034094 anarcat 30 10 20028 13960 2428 S 11.1 0.1 0:00.70 mbsync
4036539 anarcat 20 0 9604 3464 2408 R 11.1 0.0 0:00.04 top
352 root 0 -20 0 0 0 S 5.6 0.0 1:03.64 z_rd_int
353 root 0 -20 0 0 0 S 5.6 0.0 1:03.64 z_rd_int
354 root 0 -20 0 0 0 S 5.6 0.0 1:04.01 z_rd_int
I wonder how much of that is due to syncoid, particularly because I
often saw mbuffer and pv in there which are not strictly necessary
to do those kind of operations, as far as I understand.
Once that's done, export the pools to disconnect the drive:
Monitoring
ZFS should be monitoring your pools regularly. Normally, the [[!debman
zed]] daemon monitors all ZFS events. It is the thing that will report
when a scrub failed, for example. See this configuration guide.
Scrubs should be regularly scheduled to ensure consistency of the
pool. This can be done in newer zfsutils-linux versions
(bullseye-backports or bookworm) with one of those, depending on the
desired frequency:
When the scrub runs, if it finds anything it will send an event which
will get picked up by the zed daemon which will then send a
notification, see below for an example.
TODO: deploy on curie, if possible (probably not because no RAID)
TODO: this should be in Puppet
Scrub warning example
So what happens when problems are found? Here's an example of how I
dealt with an error I received.
After setting up another server (tubman) with ZFS, I
eventually ended up getting a warning from the ZFS toolchain.
Date: Sun, 09 Oct 2022 00:58:08 -0400
From: root <root@anarc.at>
To: root@anarc.at
Subject: ZFS scrub_finish event for rpool on tubman
ZFS has finished a scrub:
eid: 39536
class: scrub_finish
host: tubman
time: 2022-10-09 00:58:07-0400
pool: rpool
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: scrub repaired 0B in 00:33:57 with 0 errors on Sun Oct 9 00:58:07 2022
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sdb4 ONLINE 0 1 0
sdc4 ONLINE 0 0 0
cache
sda3 ONLINE 0 0 0
errors: No known data errors
This, in itself, is a little worrisome. But it helpfully links to this
more detailed documentation (and props up there: the link still
works) which explains this is a "minor" problem (something that could
be included in the report).
In this case, this happened on a server setup on 2021-04-28, but the
disks and server hardware are much older. The server itself
(marcosv1) was built
around 2011, over 10 years ago now. The hard drive in question is:
root@tubman:~# smartctl -i -qnoserial /dev/sdb
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-15-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate BarraCuda 3.5
Device Model: ST4000DM004-2CV104
Firmware Version: 0001
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5425 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Tue Oct 11 11:02:32 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
That's over a year of power on, which shouldn't be so bad. It has
written about 10TB of data (21107792664 LBAs * 512 byte/LBA), which
is about two full writes. According to its specification, this
device is supposed to support 55 TB/year of writes, so we're far below
spec. Note that are still far from the "non-recoverable read error per
bits" spec (1 per 10E15), as we've basically read 13E12 bits
(3201579750 LBAs * 512 byte/LBA = 13E12 bits).
It's likely this disk was made in 2018, so it is in its fourth
year.
Interestingly, /dev/sdc is also a Seagate drive, but of a different
series:
root@tubman:~# smartctl -qnoserial -i /dev/sdb
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-15-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate BarraCuda 3.5
Device Model: ST4000DM004-2CV104
Firmware Version: 0001
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5425 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Tue Oct 11 11:21:35 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
It has seen much more reads than the other disk which is also interesting:
That's 4 years of Head_Flying_Hours, and over 4 years (4 years and
48 days) of Power_On_Hours. The copyright date on that drive's
specs goes back to 2016, so it's a much older drive.
SMART self-test succeeded.
Remaining issues
TODO: move send/receive backups to offsite host, see also
zfs for alternatives to syncoid/sanoid there
TODO: document this somewhere: bpool and rpool are both pools and
datasets. that's pretty confusing, but also very useful because it
allows for pool-wide recursive snapshots, which are used for the
backup system
fio improvements
I really want to improve my experience with fio. Right now, I'm just
cargo-culting stuff from other folks and I don't really like
it. stressant is a good example of my struggles, in the sense
that it doesn't really work that well for disk tests.
I would love to have just a single .fio job file that lists multiple
jobs to run serially. For example, this file describes the above
workload pretty well:
[global]
# cargo-culting Salter
fallocate=none
ioengine=posixaio
runtime=60
time_based=1
end_fsync=1
stonewall=1
group_reporting=1
# no need to drop caches, done by default
# invalidate=1
# Single 4KiB random read/write process
[randread-4k-4g-1x]
rw=randread
bs=4k
size=4g
numjobs=1
iodepth=1
[randwrite-4k-4g-1x]
rw=randwrite
bs=4k
size=4g
numjobs=1
iodepth=1
# 16 parallel 64KiB random read/write processes:
[randread-64k-256m-16x]
rw=randread
bs=64k
size=256m
numjobs=16
iodepth=16
[randwrite-64k-256m-16x]
rw=randwrite
bs=64k
size=256m
numjobs=16
iodepth=16
# Single 1MiB random read/write process
[randread-1m-16g-1x]
rw=randread
bs=1m
size=16g
numjobs=1
iodepth=1
[randwrite-1m-16g-1x]
rw=randwrite
bs=1m
size=16g
numjobs=1
iodepth=1
... except the jobs are actually started in parallel, even though they
are stonewall'd, as far as I can tell by the reports. I sent a
mail to the fio mailing list for clarification.
It looks like the jobs are started in parallel, but actual
(correctly) run serially. It seems like this might just be a matter of
reporting the right timestamps in the end, although it does feel like
starting all the processes (even if not doing any work yet) could
skew the results.
Hangs during procedure
During the procedure, it happened a few times where any ZFS command
would completely hang. It seems that using an external USB drive to
sync stuff didn't work so well: sometimes it would reconnect under a
different device (from sdc to sdd, for example), and this would
greatly confuse ZFS.
Here, for example, is sdd reappearing out of the blue:
May 19 11:22:53 curie kernel: [ 699.820301] scsi host4: uas
May 19 11:22:53 curie kernel: [ 699.820544] usb 2-1: authorized to connect
May 19 11:22:53 curie kernel: [ 699.922433] scsi 4:0:0:0: Direct-Access ROG ESD-S1C 0 PQ: 0 ANSI: 6
May 19 11:22:53 curie kernel: [ 699.923235] sd 4:0:0:0: Attached scsi generic sg2 type 0
May 19 11:22:53 curie kernel: [ 699.923676] sd 4:0:0:0: [sdd] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
May 19 11:22:53 curie kernel: [ 699.923788] sd 4:0:0:0: [sdd] Write Protect is off
May 19 11:22:53 curie kernel: [ 699.923949] sd 4:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 19 11:22:53 curie kernel: [ 699.924149] sd 4:0:0:0: [sdd] Optimal transfer size 33553920 bytes
May 19 11:22:53 curie kernel: [ 699.961602] sdd: sdd1 sdd2 sdd3 sdd4
May 19 11:22:53 curie kernel: [ 699.996083] sd 4:0:0:0: [sdd] Attached SCSI disk
Next time I run a ZFS command (say zpool list), the command
completely hangs (D state) and this comes up in the logs:
May 19 11:34:21 curie kernel: [ 1387.914843] zio pool=bpool vdev=/dev/sdc3 error=5 type=2 offset=71344128 size=4096 flags=184880
May 19 11:34:21 curie kernel: [ 1387.914859] zio pool=bpool vdev=/dev/sdc3 error=5 type=2 offset=205565952 size=4096 flags=184880
May 19 11:34:21 curie kernel: [ 1387.914874] zio pool=bpool vdev=/dev/sdc3 error=5 type=2 offset=272789504 size=4096 flags=184880
May 19 11:34:21 curie kernel: [ 1387.914906] zio pool=bpool vdev=/dev/sdc3 error=5 type=1 offset=270336 size=8192 flags=b08c1
May 19 11:34:21 curie kernel: [ 1387.914932] zio pool=bpool vdev=/dev/sdc3 error=5 type=1 offset=1073225728 size=8192 flags=b08c1
May 19 11:34:21 curie kernel: [ 1387.914948] zio pool=bpool vdev=/dev/sdc3 error=5 type=1 offset=1073487872 size=8192 flags=b08c1
May 19 11:34:21 curie kernel: [ 1387.915165] zio pool=bpool vdev=/dev/sdc3 error=5 type=2 offset=272793600 size=4096 flags=184880
May 19 11:34:21 curie kernel: [ 1387.915183] zio pool=bpool vdev=/dev/sdc3 error=5 type=2 offset=339853312 size=4096 flags=184880
May 19 11:34:21 curie kernel: [ 1387.915648] WARNING: Pool 'bpool' has encountered an uncorrectable I/O failure and has been suspended.
May 19 11:34:21 curie kernel: [ 1387.915648]
May 19 11:37:25 curie kernel: [ 1571.558614] task:txg_sync state:D stack: 0 pid: 997 ppid: 2 flags:0x00004000
May 19 11:37:25 curie kernel: [ 1571.558623] Call Trace:
May 19 11:37:25 curie kernel: [ 1571.558640] __schedule+0x282/0x870
May 19 11:37:25 curie kernel: [ 1571.558650] schedule+0x46/0xb0
May 19 11:37:25 curie kernel: [ 1571.558670] schedule_timeout+0x8b/0x140
May 19 11:37:25 curie kernel: [ 1571.558675] ? __next_timer_interrupt+0x110/0x110
May 19 11:37:25 curie kernel: [ 1571.558678] io_schedule_timeout+0x4c/0x80
May 19 11:37:25 curie kernel: [ 1571.558689] __cv_timedwait_common+0x12b/0x160 [spl]
May 19 11:37:25 curie kernel: [ 1571.558694] ? add_wait_queue_exclusive+0x70/0x70
May 19 11:37:25 curie kernel: [ 1571.558702] __cv_timedwait_io+0x15/0x20 [spl]
May 19 11:37:25 curie kernel: [ 1571.558816] zio_wait+0x129/0x2b0 [zfs]
May 19 11:37:25 curie kernel: [ 1571.558929] dsl_pool_sync+0x461/0x4f0 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559032] spa_sync+0x575/0xfa0 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559138] ? spa_txg_history_init_io+0x101/0x110 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559245] txg_sync_thread+0x2e0/0x4a0 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559354] ? txg_fini+0x240/0x240 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559366] thread_generic_wrapper+0x6f/0x80 [spl]
May 19 11:37:25 curie kernel: [ 1571.559376] ? __thread_exit+0x20/0x20 [spl]
May 19 11:37:25 curie kernel: [ 1571.559379] kthread+0x11b/0x140
May 19 11:37:25 curie kernel: [ 1571.559382] ? __kthread_bind_mask+0x60/0x60
May 19 11:37:25 curie kernel: [ 1571.559386] ret_from_fork+0x22/0x30
May 19 11:37:25 curie kernel: [ 1571.559401] task:zed state:D stack: 0 pid: 1564 ppid: 1 flags:0x00000000
May 19 11:37:25 curie kernel: [ 1571.559404] Call Trace:
May 19 11:37:25 curie kernel: [ 1571.559409] __schedule+0x282/0x870
May 19 11:37:25 curie kernel: [ 1571.559412] ? __kmalloc_node+0x141/0x2b0
May 19 11:37:25 curie kernel: [ 1571.559417] schedule+0x46/0xb0
May 19 11:37:25 curie kernel: [ 1571.559420] schedule_preempt_disabled+0xa/0x10
May 19 11:37:25 curie kernel: [ 1571.559424] __mutex_lock.constprop.0+0x133/0x460
May 19 11:37:25 curie kernel: [ 1571.559435] ? nvlist_xalloc.part.0+0x68/0xc0 [znvpair]
May 19 11:37:25 curie kernel: [ 1571.559537] spa_all_configs+0x41/0x120 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559644] zfs_ioc_pool_configs+0x17/0x70 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559752] zfsdev_ioctl_common+0x697/0x870 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559758] ? _copy_from_user+0x28/0x60
May 19 11:37:25 curie kernel: [ 1571.559860] zfsdev_ioctl+0x53/0xe0 [zfs]
May 19 11:37:25 curie kernel: [ 1571.559866] __x64_sys_ioctl+0x83/0xb0
May 19 11:37:25 curie kernel: [ 1571.559869] do_syscall_64+0x33/0x80
May 19 11:37:25 curie kernel: [ 1571.559873] entry_SYSCALL_64_after_hwframe+0x44/0xa9
May 19 11:37:25 curie kernel: [ 1571.559876] RIP: 0033:0x7fcf0ef32cc7
May 19 11:37:25 curie kernel: [ 1571.559878] RSP: 002b:00007fcf0e181618 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
May 19 11:37:25 curie kernel: [ 1571.559881] RAX: ffffffffffffffda RBX: 000055b212f972a0 RCX: 00007fcf0ef32cc7
May 19 11:37:25 curie kernel: [ 1571.559883] RDX: 00007fcf0e181640 RSI: 0000000000005a04 RDI: 000000000000000b
May 19 11:37:25 curie kernel: [ 1571.559885] RBP: 00007fcf0e184c30 R08: 00007fcf08016810 R09: 00007fcf08000080
May 19 11:37:25 curie kernel: [ 1571.559886] R10: 0000000000080000 R11: 0000000000000246 R12: 000055b212f972a0
May 19 11:37:25 curie kernel: [ 1571.559888] R13: 0000000000000000 R14: 00007fcf0e181640 R15: 0000000000000000
May 19 11:37:25 curie kernel: [ 1571.559980] task:zpool state:D stack: 0 pid:11815 ppid: 3816 flags:0x00004000
May 19 11:37:25 curie kernel: [ 1571.559983] Call Trace:
May 19 11:37:25 curie kernel: [ 1571.559988] __schedule+0x282/0x870
May 19 11:37:25 curie kernel: [ 1571.559992] schedule+0x46/0xb0
May 19 11:37:25 curie kernel: [ 1571.559995] io_schedule+0x42/0x70
May 19 11:37:25 curie kernel: [ 1571.560004] cv_wait_common+0xac/0x130 [spl]
May 19 11:37:25 curie kernel: [ 1571.560008] ? add_wait_queue_exclusive+0x70/0x70
May 19 11:37:25 curie kernel: [ 1571.560118] txg_wait_synced_impl+0xc9/0x110 [zfs]
May 19 11:37:25 curie kernel: [ 1571.560223] txg_wait_synced+0xc/0x40 [zfs]
May 19 11:37:25 curie kernel: [ 1571.560325] spa_export_common+0x4cd/0x590 [zfs]
May 19 11:37:25 curie kernel: [ 1571.560430] ? zfs_log_history+0x9c/0xf0 [zfs]
May 19 11:37:25 curie kernel: [ 1571.560537] zfsdev_ioctl_common+0x697/0x870 [zfs]
May 19 11:37:25 curie kernel: [ 1571.560543] ? _copy_from_user+0x28/0x60
May 19 11:37:25 curie kernel: [ 1571.560644] zfsdev_ioctl+0x53/0xe0 [zfs]
May 19 11:37:25 curie kernel: [ 1571.560649] __x64_sys_ioctl+0x83/0xb0
May 19 11:37:25 curie kernel: [ 1571.560653] do_syscall_64+0x33/0x80
May 19 11:37:25 curie kernel: [ 1571.560656] entry_SYSCALL_64_after_hwframe+0x44/0xa9
May 19 11:37:25 curie kernel: [ 1571.560659] RIP: 0033:0x7fdc23be2cc7
May 19 11:37:25 curie kernel: [ 1571.560661] RSP: 002b:00007ffc8c792478 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
May 19 11:37:25 curie kernel: [ 1571.560664] RAX: ffffffffffffffda RBX: 000055942ca49e20 RCX: 00007fdc23be2cc7
May 19 11:37:25 curie kernel: [ 1571.560666] RDX: 00007ffc8c792490 RSI: 0000000000005a03 RDI: 0000000000000003
May 19 11:37:25 curie kernel: [ 1571.560667] RBP: 00007ffc8c795e80 R08: 00000000ffffffff R09: 00007ffc8c792310
May 19 11:37:25 curie kernel: [ 1571.560669] R10: 000055942ca49e30 R11: 0000000000000246 R12: 00007ffc8c792490
May 19 11:37:25 curie kernel: [ 1571.560671] R13: 000055942ca49e30 R14: 000055942aed2c20 R15: 00007ffc8c795a40
Here's another example, where you see the USB controller bleeping out
and back into existence:
mai 19 11:38:39 curie kernel: usb 2-1: USB disconnect, device number 2
mai 19 11:38:39 curie kernel: sd 4:0:0:0: [sdd] Synchronizing SCSI cache
mai 19 11:38:39 curie kernel: sd 4:0:0:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
mai 19 11:39:25 curie kernel: INFO: task zed:1564 blocked for more than 241 seconds.
mai 19 11:39:25 curie kernel: Tainted: P IOE 5.10.0-14-amd64 #1 Debian 5.10.113-1
mai 19 11:39:25 curie kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mai 19 11:39:25 curie kernel: task:zed state:D stack: 0 pid: 1564 ppid: 1 flags:0x00000000
mai 19 11:39:25 curie kernel: Call Trace:
mai 19 11:39:25 curie kernel: __schedule+0x282/0x870
mai 19 11:39:25 curie kernel: ? __kmalloc_node+0x141/0x2b0
mai 19 11:39:25 curie kernel: schedule+0x46/0xb0
mai 19 11:39:25 curie kernel: schedule_preempt_disabled+0xa/0x10
mai 19 11:39:25 curie kernel: __mutex_lock.constprop.0+0x133/0x460
mai 19 11:39:25 curie kernel: ? nvlist_xalloc.part.0+0x68/0xc0 [znvpair]
mai 19 11:39:25 curie kernel: spa_all_configs+0x41/0x120 [zfs]
mai 19 11:39:25 curie kernel: zfs_ioc_pool_configs+0x17/0x70 [zfs]
mai 19 11:39:25 curie kernel: zfsdev_ioctl_common+0x697/0x870 [zfs]
mai 19 11:39:25 curie kernel: ? _copy_from_user+0x28/0x60
mai 19 11:39:25 curie kernel: zfsdev_ioctl+0x53/0xe0 [zfs]
mai 19 11:39:25 curie kernel: __x64_sys_ioctl+0x83/0xb0
mai 19 11:39:25 curie kernel: do_syscall_64+0x33/0x80
mai 19 11:39:25 curie kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
mai 19 11:39:25 curie kernel: RIP: 0033:0x7fcf0ef32cc7
mai 19 11:39:25 curie kernel: RSP: 002b:00007fcf0e181618 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
mai 19 11:39:25 curie kernel: RAX: ffffffffffffffda RBX: 000055b212f972a0 RCX: 00007fcf0ef32cc7
mai 19 11:39:25 curie kernel: RDX: 00007fcf0e181640 RSI: 0000000000005a04 RDI: 000000000000000b
mai 19 11:39:25 curie kernel: RBP: 00007fcf0e184c30 R08: 00007fcf08016810 R09: 00007fcf08000080
mai 19 11:39:25 curie kernel: R10: 0000000000080000 R11: 0000000000000246 R12: 000055b212f972a0
mai 19 11:39:25 curie kernel: R13: 0000000000000000 R14: 00007fcf0e181640 R15: 0000000000000000
mai 19 11:39:25 curie kernel: INFO: task zpool:11815 blocked for more than 241 seconds.
mai 19 11:39:25 curie kernel: Tainted: P IOE 5.10.0-14-amd64 #1 Debian 5.10.113-1
mai 19 11:39:25 curie kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mai 19 11:39:25 curie kernel: task:zpool state:D stack: 0 pid:11815 ppid: 2621 flags:0x00004004
mai 19 11:39:25 curie kernel: Call Trace:
mai 19 11:39:25 curie kernel: __schedule+0x282/0x870
mai 19 11:39:25 curie kernel: schedule+0x46/0xb0
mai 19 11:39:25 curie kernel: io_schedule+0x42/0x70
mai 19 11:39:25 curie kernel: cv_wait_common+0xac/0x130 [spl]
mai 19 11:39:25 curie kernel: ? add_wait_queue_exclusive+0x70/0x70
mai 19 11:39:25 curie kernel: txg_wait_synced_impl+0xc9/0x110 [zfs]
mai 19 11:39:25 curie kernel: txg_wait_synced+0xc/0x40 [zfs]
mai 19 11:39:25 curie kernel: spa_export_common+0x4cd/0x590 [zfs]
mai 19 11:39:25 curie kernel: ? zfs_log_history+0x9c/0xf0 [zfs]
mai 19 11:39:25 curie kernel: zfsdev_ioctl_common+0x697/0x870 [zfs]
mai 19 11:39:25 curie kernel: ? _copy_from_user+0x28/0x60
mai 19 11:39:25 curie kernel: zfsdev_ioctl+0x53/0xe0 [zfs]
mai 19 11:39:25 curie kernel: __x64_sys_ioctl+0x83/0xb0
mai 19 11:39:25 curie kernel: do_syscall_64+0x33/0x80
mai 19 11:39:25 curie kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
mai 19 11:39:25 curie kernel: RIP: 0033:0x7fdc23be2cc7
mai 19 11:39:25 curie kernel: RSP: 002b:00007ffc8c792478 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
mai 19 11:39:25 curie kernel: RAX: ffffffffffffffda RBX: 000055942ca49e20 RCX: 00007fdc23be2cc7
mai 19 11:39:25 curie kernel: RDX: 00007ffc8c792490 RSI: 0000000000005a03 RDI: 0000000000000003
mai 19 11:39:25 curie kernel: RBP: 00007ffc8c795e80 R08: 00000000ffffffff R09: 00007ffc8c792310
mai 19 11:39:25 curie kernel: R10: 000055942ca49e30 R11: 0000000000000246 R12: 00007ffc8c792490
mai 19 11:39:25 curie kernel: R13: 000055942ca49e30 R14: 000055942aed2c20 R15: 00007ffc8c795a40
I understand those are rather extreme conditions: I would fully expect
the pool to stop working if the underlying drives disappear. What
doesn't seem acceptable is that a command would completely hang like
this.
References
See the zfs documentation for more information about ZFS,
and tubman for another installation and migration procedure.
1.3 billion announced for new Forestry Support
Funds to be delivered through new Forestry Programme
Premiums for planting trees to be increased by between 46% and 66% and extended to 20 years for farmers
#GreensInGovernment
The Taoiseach, Miche l Martin TD, Minister of State with responsibility for Forestry, Senator Pippa Hackett, and Minister for Agriculture, Food and the Marine, Charlie McConalogue T.D today announced a proposed investment by the Government of 1.3 billion in Irish forestry. The funding will be for the next national Forestry Programme and represents the largest ever investment by an Irish Government in tree-planting. The programme will now be the subject of state-aid approval by the European Commission.
The Taoiseach said:
This commitment by the Government to such a substantial financial package reflects the seriousness with which we view the climate change and biodiversity challenges, which affect all of society. Forestry is at the heart of delivering on our sustainability goals and strong support is needed to encourage engagement from all our stakeholders in reaching our objectives.
Minister Hackett said:
I m delighted to have secured a package of 1.318 billion for forestry. This will support the biggest and best-funded Forestry Programme ever in Ireland. It comes at an appropriate time, given the urgency of taking climate mitigation measures. Planting trees is one of the most effective methods of tackling climate change as well as contributing to improved biodiversity and water quality. One of my main aims is to re-engage farmers in afforestation. I m delighted therefore to be proposing a new 20-year premium term exclusively for farmers, as well as introducing a new small-scale native woodland scheme which will allow farmers to plant up to 1 hectare of native woodland on farmland and along watercourses outside of the forestry licensing process.
Minister McConalogue said:
Today we commit to providing unprecedented incentives to encourage the planting of trees that can provide a valuable addition to farm income and help to meet national climate and biodiversity objectives. This funding guarantees continued payments to those forest owners who planted under the current scheme and who are still in receipt of premiums. It also offers new and improved financial supports to those who undertake planting and sustainable forest management under the new Programme. We intend to increase premiums for planting trees by between 46% and 66% and to extend the premium period from 15 to 20 years for farmers.
"We are approaching a new and exciting period for forestry in Ireland. The new Forestry Programme will drive a new and brighter future for forestry, for farmers and for our climate.
The proposed new Forestry Programme is currently out to public consultation as part of the Strategic Environmental Assessment and Appropriate Assessment process. The Programme is the main implementation mechanism for the new Forest Strategy (2023 -2030) which reflects the ambitions contained in the recently published Shared National Vision for Trees, Woods and Forests in Ireland until 2050. The public consultation closes on 29th November, 2022 and any changes which result from this process will be incorporated into the Programme and the Forest Strategy.
Minister Hackett commented: The draft Forestry Programme and Forest Strategy are the product of extensive stakeholder consultation and feedback, and both documents are open to public consultation for the next number of weeks. I would strongly encourage all interested parties to engage with the consultation in advance of the Strategy and Programme being finalised.
The new Programme is built around the principle of right trees in the right places for the right reasons with the right management. It aims to deliver more diverse forest which will meet multiple societal objectives, economic, social and environmental. Higher grant rates for forest establishment are also proposed with increases of approximately 20% to reflect rising living costs. The new one hectare native tree area scheme will also make it easier for landowners who wish to plant small areas of trees on their farm.
The Taoiseach concluded I welcome this milestone and I believe that this funding injection will be an important catalyst in delivering on the ambition outlined in the new Forest Strategy. Our environmental challenges are huge but so is our commitment to overcoming them and this Forestry Programme is key to delivering so many of our priorities.
The new Programme will be 100% Exchequer funded and is subject to State Aid approval from the EU Commission. The Department is in contact with the Commission in relation to this approval which is a rigorous process.
Chris: Welcome, Hans! Could you briefly tell me about yourself?
Hans: Sure. I spend most of my time trying to private communications software usable by everyone, designing interactive software with a focus on human perceptual capabilities and building networks with free software. I ve been involved in Debian since approximately 2008 and became an official Debian developer in 2011. In the little time left over from that, I sometimes compose music with computers from my home in Austria.
Chris: For those who have not heard of it before, what is the F-Droid project?
Hans: F-Droid is a community-run app store that provides free software applications for Android phones. First and foremost our goal is to represent our users. In particular, we review all of the apps that we distribute, and these reviews not only check for whether something is definitely free software or not, we additionally look for ethical problems with applications as well issues that we call Anti-Features . Since the project began in 2010, F-Droid now offers almost 4,000 free-software applications.
F-Droid is also a app store kit as well, providing all the tools that are needed to operate an free app store of your own. It also includes complete build and release tools for managing the process of turning app source code into published builds.
Chris: How exactly does F-Droid differ from the Google Play store? Why might someone use F-Droid over Google Play?
Hans: One key difference to the Google Play Store is that F-Droid does not ship proprietary software by default. All apps shipped from f-droid.org are built from source on our own builders. This is partly because F-Droid is backed by the free software community; that is, people who have engaged in the free software community long before Android was conceived, and, in particular, share many if not all of its values. Using F-Droid will therefore feel very familiar to anyone familiar with a modern Linux distribution.
Chris: How do you describe reproducibility from the F-Droid point of view?
Hans: All centralised software repositories are extremely tempting targets for exploitation by malicious third parties, and the kinds of personal or otherwise sensitive data on our phones only increase this risk. In F-Droid s case, not only could the software we distribute be theoretically replaced with nefarious copies, the build infrastructure that generates that software could be compromised as well.
F-Droid having reproducible builds is extremely important as it allows us to verify that our build systems have not been compromised and distributing malware to our users. In particular, if an independent build infrastructure can produce precisely the same results from a build, then we can be reasonably sure that they haven t been compromised. Technically-minded users can also validate their builds on their own systems too, further increasing trust in our build infrastructure. (This can be performed using fdroid verify command.)
Our signature & trust scheme means that F-Droid can verify that an app is 100% free software whilst still using the developer s original .APK file. More details about this may be found in our reproducibility documentation and on the page about our Verification Server.
Chris: How do you see F-Droid fitting into the rest of the modern security ecosystem?
Hans: Whilst F-Droid inherits all of the social benefits of free software, F-Droid takes special care to respect your privacy as well we don t attempt to track your device in any way. In particular, you don t need an account to use the F-Droid client, and F-Droid doesn t send any device-identifying information to our servers other than its own version number.
What is more, we mark all apps in our repository that track you, and users can choose to hide any apps that has been tagged with a specific Anti-Feature in the F-Droid settings. Furthermore, any personal data you decide to give us (such as your email address when registering for a forum account) goes no further than us as well, and we pledge that it will never be used for anything beyond merely maintaining your account.
Chris: What would fully reproducible mean to F-Droid? What it would look like if reproducibility was a solved problem ? Or, rather, what would be your ultimate reproducibility goals?
Hans: In an ideal world, every application submitted to F-Droid would not only build reproducibly, but would come with a cryptographically-signed signature from the developer as well. Then we would only distribute an compiled application after a build had received a number of matching signatures from multiple, independent third parties. This would mean that our users were not placing their trust solely in software developers machines, and wouldn t be trusting our centralised build servers as well.
Chris: What are the biggest blockers to reaching this state? Are there any key steps or milestones to get there?
Hans: Time is probably the main constraint to reaching this goal. Not only do we need system administrators on an ongoing basis but we also need to incorporate reproducibly checks into our Continuous Integration (CI) system. We are always looking for new developers to join our effort, as well as learning about how to better bring them up to speed.
Separate to this, we often experience blockers with reproducibility-related bugs in the Android build tooling. Luckily, upstreams do ultimately address these issues, but in some cases this has taken three or four years to reach end-users and developers. Unfortunately, upstream is not chiefly concerned with the security aspects of reproducibility builds; they care more about how it can minimise and optimise download size and speed.
Chris: Are you tracking any statistics about reproducibility in F-Droid over time? If so, can you share them? Does F-Droid track statistics about its own usage?
Hans: To underline a topic touched on above, F-Droid is dedicated to preserving the privacy of its users; we therefore do not record usage statistics. This is, of course, in contrast to other application stores.
However, we are in a position to track whether packages in F-Droid are reproducible or not. As in: what proportion of APKs in F-Droid have been independently verified over time? Unfortunately, due to time constraints, we do not yet automatically publish charts for this.
We do publish some raw data that is related, though, and we naturally welcome contributions of visualizations based on any and all of our data. The All our APIs page on our wiki is a good place to start for someone wanting to contribute, everything about reproducible F-Droid apps is available in JSON format, what s missing are apps or dashboards making use of the available raw data.
Chris: Many thanks for this interview, Hans, and for all of your work on F-Droid and elsewhere. If someone wanted to get in touch or learn more about the project, where might someone go?
Hans: The best way to find out about F-Droid is, of course, the main F-Droid homepage, but we are also on Twitter @fdroidorg. We have many avenues to participate and to learn more! We have an About page on our website and a thriving forum. We also have part of our documentation specifically dedicated to reproducible builds.
For more information about the Reproducible Builds project, please see our website at
reproducible-builds.org. If you are interested in
ensuring the ongoing security of the software that underpins our civilisation
and wish to sponsor the Reproducible Builds project, please reach out to the
project by emailing
contact@reproducible-builds.org.
I have some concerns about Matrix (the protocol, not the movie that
came out recently, although I do have concerns about that as
well). I've been watching the project for a long time, and it seems
more a promising alternative to many protocols like IRC, XMPP, and
Signal.
This review may sound a bit negative, because it focuses on those
concerns. I am the operator of an IRC network and people keep asking
me to bridge it with Matrix. I have myself considered just giving up
on IRC and converting to Matrix. This space is a living document
exploring my research of that problem space. The TL;DR: is that no,
I'm not setting up a bridge just yet, and I'm still on IRC.
This article was written over the course of the last three months, but
I have been watching the Matrix project for years (my logs seem to say
2016 at least). The article is rather long. It will likely take you
half an hour to read, so copy this over to your ebook reader,
your tablet, or dead trees, and lean back and relax as I show you
around the Matrix. Or, alternatively, just jump to a section that
interest you, most likely the conclusion.
Introduction to Matrix
Matrix is an "open standard for interoperable, decentralised,
real-time communication over IP. It can be used to power Instant
Messaging, VoIP/WebRTC signalling, Internet of Things communication -
or anywhere you need a standard HTTP API for publishing and
subscribing to data whilst tracking the conversation history".
It's also (when compared with XMPP) "an eventually consistent
global JSON database with an HTTP API and pubsub semantics - whilst
XMPP can be thought of as a message passing protocol."
According to their FAQ, the project started in 2014, has about
20,000 servers, and millions of users. Matrix works over HTTPS but
over a special port: 8448.
Security and privacy
I have some concerns about the security promises of Matrix. It's
advertised as a "secure" with "E2E [end-to-end] encryption", but how
does it actually work?
Data retention defaults
One of my main concerns with Matrix is data retention, which is a key
part of security in a threat model where (for example) an hostile
state actor wants to surveil your communications and can seize your
devices.
On IRC, servers don't actually keep messages all that long: they pass
them along to other servers and clients as fast as they can, only keep
them in memory, and move on to the next message. There are no concerns
about data retention on messages (and their metadata) other than the
network layer. (I'm ignoring the issues with user registration, which
is a separate, if valid, concern.) Obviously, an hostile server
could log everything passing through it, but IRC federations are
normally tightly controlled. So, if you trust your IRC operators, you
should be fairly safe. Obviously, clients can (and often do, even if
OTR is configured!) log all messages, but this is generally not
the default. Irssi, for example, does not log by
default. IRC bouncers are more likely to log to disk, of course,
to be able to do what they do.
Compare this to Matrix: when you send a message to a Matrix
homeserver, that server first stores it in its internal SQL
database. Then it will transmit that message to all clients connected
to that server and room, and to all other servers that have clients
connected to that room. Those remote servers, in turn, will keep a
copy of that message and all its metadata in their own database, by
default forever. On encrypted rooms those messages are encrypted, but
not their metadata.
There is a mechanism to expire entries in Synapse, but it is not
enabled by default. So one should generally assume that a message
sent on Matrix is never expired.
GDPR in the federation
But even if that setting was enabled by default, how do you control
it? This is a fundamental problem of the federation: if any user is
allowed to join a room (which is the default), those user's servers
will log all content and metadata from that room. That includes
private, one-on-one conversations, since those are essentially rooms
as well.
In the context of the GDPR, this is really tricky: who is the
responsible party (known as the "data controller") here? It's
basically any yahoo who fires up a home server and joins a room.
In a federated network, one has to wonder whether GDPR enforcement is
even possible at all. But in Matrix in particular, if you want to
enforce your right to be forgotten in a given room, you would have to:
enumerate all the users that ever joined the room while you were
there
discover all their home servers
start a GDPR procedure against all those servers
I recognize this is a hard problem to solve while still keeping an
open ecosystem. But I believe that Matrix should have much stricter
defaults towards data retention than right now. Message expiry should
be enforced by default, for example. (Note that there are also
redaction policies that could be used to implement part of the GDPR
automatically, see the privacy policy discussion below on that.)
Also keep in mind that, in the brave new peer-to-peer world that
Matrix is heading towards, the boundary between server and client is
likely to be fuzzier, which would make applying the GDPR even more difficult.
Update: this comment links to this post (in german) which
apparently studied the question and concluded that Matrix is not
GDPR-compliant.
In fact, maybe Synapse should be designed so that there's no
configurable flag to turn off data retention. A bit like how most
system loggers in UNIX (e.g. syslog) come with a log retention system
that typically rotate logs after a few weeks or month. Historically,
this was designed to keep hard drives from filling up, but it also has
the added benefit of limiting the amount of personal information kept
on disk in this modern day. (Arguably, syslog doesn't rotate logs on
its own, but, say, Debian GNU/Linux, as an installed system, does have
log retention policies well defined for installed packages, and those
can be discussed. And "no expiry" is definitely a bug.
We currently use cookies to support our use of Google Analytics on
the Website and Service. Google Analytics collects information about
how you use the Website and Service.
[...]
This helps us to provide you with a good experience when you
browse our Website and use our Service and also allows us to improve
our Website and our Service.
When I asked Matrix people about why they were using Google Analytics,
they explained this was for development purposes and they were aiming
for velocity at the time, not privacy (paraphrasing here).
They also included a "free to snitch" clause:
If we are or believe that we are under a duty to disclose or share
your personal data, we will do so in order to comply with any legal
obligation, the instructions or requests of a governmental authority
or regulator, including those outside of the UK.
Those are really broad terms, above and beyond what is typically
expected legally.
Like the current retention policies, such user tracking and
... "liberal" collaboration practices with the state set a bad
precedent for other home servers.
Thankfully, since the above policy was published (2017), the GDPR was
"implemented" (2018) and it seems like both the Element.io
privacy policy and the Matrix.org privacy policy have been
somewhat improved since.
Notable points of the new privacy policies:
2.3.1.1: the "federation" section actually outlines that
"Federated homeservers and Matrix clients which respect the Matrix
protocol are expected to honour these controls and
redaction/erasure requests, but other federated homeservers are
outside of the span of control of Element, and we cannot guarantee
how this data will be processed"
2.6: users under the age of 16 should not use the
matrix.org service
2.10: Upcloud, Mythic Beast, Amazon, and CloudFlare possibly
have access to your data (it's nice to at least mention this in the
privacy policy: many providers don't even bother admitting to this
kind of delegation)
I'm not super happy with all the trackers they have on the Element
platform, but then again you don't have to use that service. Your
favorite homeserver (assuming you are not on Matrix.org) probably has
their own Element deployment, hopefully without all that garbage.
Overall, this is all a huge improvement over the previous privacy
policy, so hats off to the Matrix people for figuring out a reasonable
policy in such a tricky context. I particularly like this bit:
We will forget your copy of your data upon your request. We will
also forward your request to be forgotten onto federated
homeservers. However - these homeservers are outside our span of
control, so we cannot guarantee they will forget your data.
It's great they implemented those mechanisms and, after all, if
there's an hostile party in there, nothing can prevent them from using
screenshots to just exfiltrate your data away from the client side
anyways, even with services typically seen as more secure, like
Signal.
As an aside, I also appreciate that Matrix.org has a fairly decent
code of conduct, based on the TODO CoC which checks all the
boxes in the geekfeminism wiki.
Metadata handling
Overall, privacy protections in Matrix mostly concern message
contents, not metadata. In other words, who's talking with who, when
and from where is not well protected. Compared to a tool like Signal,
which goes through great lengths to anonymize that data with features
like private contact discovery, disappearing messages,
sealed senders, and private groups, Matrix is definitely
behind. (Note: there is an issue open about message lifetimes in
Element since 2020, but it's not at even at the MSC stage yet.)
This is a known issue (opened in 2019) in Synapse, but this is
not just an implementation issue, it's a flaw in the protocol
itself. Home servers keep join/leave of all rooms, which gives clear
text information about who is talking to. Synapse logs may also
contain privately identifiable information that home server admins
might not be aware of in the first place. Those log rotation policies
are separate from the server-level retention policy, which may be
confusing for a novice sysadmin.
Combine this with the federation: even if you trust your home server
to do the right thing, the second you join a public room with
third-party home servers, those ideas kind of get thrown out because
those servers can do whatever they want with that information. Again,
a problem that is hard to solve in any federation.
To be fair, IRC doesn't have a great story here either: any client
knows not only who's talking to who in a room, but also typically
their client IP address. Servers can (and often do) obfuscate
this, but often that obfuscation is trivial to reverse. Some servers
do provide "cloaks" (sometimes automatically), but that's kind of a
"slap-on" solution that actually moves the problem elsewhere: now the
server knows a little more about the user.
Overall, I would worry much more about a Matrix home server seizure
than a IRC or Signal server seizure. Signal does get subpoenas,
and they can only give out a tiny bit of information about their
users: their phone number, and their registration, and last connection
date. Matrix carries a lot more information in its database.
Amplification attacks on URL previews
I (still!) run an Icecast server and sometimes share links to it
on IRC which, obviously, also ends up on (more than one!) Matrix home
servers because some people connect to IRC using Matrix. This, in
turn, means that Matrix will connect to that URL to generate a link
preview.
I feel this outlines a security issue, especially because those
sockets would be kept open seemingly forever. I tried to warn the
Matrix security team but somehow, I don't think this issue was taken
very seriously. Here's the disclosure timeline:
the bug was publicly disclosed in September 2020, and not
considered a security issue until I notified them, and even then,
I had to insist
no clear disclosure policy timeline was proposed or seems
established in the project (there is a security disclosure
policy but it doesn't include any predefined timeline)
I wasn't informed of the disclosure
the actual solution is a size limit (10MB, already implemented), a
time limit (30 seconds, implemented in PR 11784), and a
content type allow list (HTML, "media" or JSON, implemented in PR
11936), and I'm not sure it's adequate
(pure vanity:) I did not make it to their Hall of fame
I'm not sure those solutions are adequate because they all seem to
assume a single home server will pull that one URL for a little while
then stop. But in a federated network, many (possibly thousands)
home servers may be connected in a single room at once. If an attacker
drops a link into such a room, all those servers would connect to
that link all at once. This is an amplification attack: a small
amount of traffic will generate a lot more traffic to a single
target. It doesn't matter there are size or time limits: the
amplification is what matters here.
It should also be noted that clients that generate link previews
have more amplification because they are more numerous than
servers. And of course, the default Matrix client (Element) does
generate link previews as well.
That said, this is possibly not a problem specific to Matrix: any
federated service that generates link previews may suffer from this.
I'm honestly not sure what the solution is here. Maybe moderation?
Maybe link previews are just evil? All I know is there was this weird
bug in my Icecast server and I tried to ring the bell about it, and it
feels it was swept under the rug. Somehow I feel this is bound to blow
up again in the future, even with the current mitigation.
Moderation
In Matrix like elsewhere, Moderation is a hard problem. There is a
detailed moderation guide and much of this problem space is
actively worked on in Matrix right now. A fundamental problem with
moderating a federated space is that a user banned from a room can
rejoin the room from another server. This is why spam is such a
problem in Email, and why IRC networks have stopped federating ages
ago (see the IRC history for that fascinating story).
The mjolnir bot
The mjolnir moderation bot is designed to help with some of those
things. It can kick and ban users, redact all of a user's message (as
opposed to one by one), all of this across multiple rooms. It can also
subscribe to a federated block list published by matrix.org to block
known abusers (users or servers). Bans are pretty flexible and
can operate at the user, room, or server level.
Matrix people suggest making the bot admin of your channels, because
you can't take back admin from a user once given.
The command-line tool
There's also a new command line tool designed to do things like:
System notify users (all users/users from a list, specific user)
delete sessions/devices not seen for X days
purge the remote media cache
select rooms with various criteria (external/local/empty/created by/encrypted/cleartext)
purge history of theses rooms
shutdown rooms
This tool and Mjolnir are based on the admin API built into
Synapse.
Rate limiting
Synapse has pretty good built-in rate-limiting which blocks
repeated login, registration, joining, or messaging attempts. It may
also end up throttling servers on the federation based on those
settings.
Fundamental federation problems
Because users joining a room may come from another server, room
moderators are at the mercy of the registration and moderation
policies of those servers. Matrix is like IRC's +R mode ("only
registered users can join") by default, except that anyone can
register their own homeserver, which makes this limited.
Server admins can block IP addresses and home servers, but those tools
are not easily available to room admins. There is an API
(m.room.server_acl in /devtools) but it is not reliable
(thanks Austin Huang for the clarification).
Matrix has the concept of guest accounts, but it is not used very
much, and virtually no client or homeserver supports it. This contrasts with the way
IRC works: by default, anyone can join an IRC network even without
authentication. Some channels require registration, but in general you
are free to join and look around (until you get blocked, of course).
I have seen anecdotal evidence (CW: Twitter, nitter link) that "moderating bridges is hell", and
I can imagine why. Moderation is already hard enough on one
federation, when you bridge a room with another network, you inherit
all the problems from that network but without the entire abuse
control tools from the original network's API...
Room admins
Matrix, in particular, has the problem that room administrators (which
have the power to redact messages, ban users, and promote other users)
are bound to their Matrix ID which is, in turn, bound to their home
servers. This implies that a home server administrators could (1)
impersonate a given user and (2) use that to hijack the room. So in
practice, the home server is the trust anchor for rooms, not the user
themselves.
That said, if server B administrator hijack user joe on server B,
they will hijack that room on that specific server. This will not
(necessarily) affect users on the other servers, as servers could
refuse parts of the updates or ban the compromised account (or
server).
It does seem like a major flaw that room credentials are bound to
Matrix identifiers, as opposed to the E2E encryption credentials. In
an encrypted room even with fully verified members, a compromised or
hostile home server can still take over the room by impersonating an
admin. That admin (or even a newly minted user) can then send events
or listen on the conversations.
This is even more frustrating when you consider that Matrix events are
actually signed and therefore have some authentication attached
to them, acting like some sort of Merkle tree (as it contains a link
to previous events). That signature, however, is made from the
homeserver PKI keys, not the client's E2E keys, which makes E2E feel
like it has been "bolted on" later.
Availability
While Matrix has a strong advantage over Signal in that it's
decentralized (so anyone can run their own homeserver,), I couldn't
find an easy way to run a "multi-primary" setup, or even a "redundant"
setup (even if with a single primary backend), short of going full-on
"replicate PostgreSQL and Redis data", which is not typically for the
faint of heart.
How this works in IRC
On IRC, it's quite easy to setup redundant nodes. All you need is:
a new machine (with it's own public address with an open port)
a shared secret (or certificate) between that machine and an
existing one on the network
a connect block on both servers
That's it: the node will join the network and people can connect to it
as usual and share the same user/namespace as the rest of the
network. The servers take care of synchronizing state: you do not need
to worry about replicating a database server.
(Now, experienced IRC people will know there's a catch here: IRC
doesn't have authentication built in, and relies on "services" which
are basically bots that authenticate users (I'm simplifying, don't
nitpick). If that service goes down, the network still works, but
then people can't authenticate, and they can start doing nasty things
like steal people's identity if they get knocked offline. But still:
basic functionality still works: you can talk in rooms and with users
that are on the reachable network.)
User identities
Matrix is more complicated. Each "home server" has its own identity
namespace: a specific user (say @anarcat:matrix.org) is bound to
that specific home server. If that server goes down, that user is
completely disconnected. They could register a new account elsewhere
and reconnect, but then they basically lose all their configuration:
contacts, joined channels are all lost.
(Also notice how the Matrix IDs don't look like a typical user address
like an email in XMPP. They at least did their homework and got the
allocation for the scheme.)
Rooms
Users talk to each other in "rooms", even in one-to-one
communications. (Rooms are also used for other things like "spaces",
they're basically used for everything, think "everything is a file"
kind of tool.) For rooms, home servers act more like IRC nodes in that
they keep a local state of the chat room and synchronize it with other
servers. Users can keep talking inside a room if the server that
originally hosts the room goes down. Rooms can have a local,
server-specific "alias" so that, say, #room:matrix.org is also
visible as #room:example.com on the example.com home server. Both
addresses refer to the same room underlying room.
(Finding this in the Element settings is not obvious though, because
that "alias" are actually called a "local address" there. So to create
such an alias (in Element), you need to go in the room settings'
"General" section, "Show more" in "Local address", then add the alias
name (e.g. foo), and then that room will be available on your
example.com homeserver as #foo:example.com.)
So a room doesn't belong to a server, it belongs to the federation,
and anyone can join the room from any serer (if the room is public, or
if invited otherwise). You can create a room on server A and when a
user from server B joins, the room will be replicated on server B as
well. If server A fails, server B will keep relaying traffic to
connected users and servers.
A room is therefore not fundamentally addressed with the above alias,
instead ,it has a internal Matrix ID, which basically a random
string. It has a server name attached to it, but that was made just to
avoid collisions. That can get a little confusing. For example, the
#fractal:gnome.org room is an alias on the gnome.org server, but
the room ID is !hwiGbsdSTZIwSRfybq:matrix.org. That's because the
room was created on matrix.org, but the preferred branding is
gnome.org now.
As an aside, rooms, by default, live forever, even after the last user
quits. There's an admin API to delete rooms and a tombstone
event to redirect to another one, but neither have a GUI yet. The
latter is part of MSC1501 ("Room version upgrades") which allows
a room admin to close a room, with a message and a pointer to another
room.
Spaces
Discovering rooms can be tricky: there is a per-server room
directory, but Matrix.org people are trying to deprecate it in favor
of "Spaces". Room directories were ripe for abuse: anyone can create a
room, so anyone can show up in there. It's possible to restrict who
can add aliases, but anyways directories were seen as too limited.
In contrast, a "Space" is basically a room that's an index of other
rooms (including other spaces), so existing moderation and
administration mechanism that work in rooms can (somewhat) work in
spaces as well. This enables a room directory that works across
federation, regardless on which server they were originally created.
New users can be added to a space or room automatically in
Synapse. (Existing users can be told about the space with a server
notice.) This gives admins a way to pre-populate a list of rooms on a
server, which is useful to build clusters of related home servers,
providing some sort of redundancy, at the room -- not user -- level.
Home servers
So while you can workaround a home server going down at the room
level, there's no such thing at the home server level, for user
identities. So if you want those identities to be stable in the long
term, you need to think about high availability. One limitation is
that the domain name (e.g. matrix.example.com) must never change in
the future, as renaming home servers is not supported.
The documentation used to say you could "run a hot spare" but that has
been removed. Last I heard, it was not possible to run a
high-availability setup where multiple, separate locations could
replace each other automatically. You can have high performance
setups where the load gets distributed among workers, but those
are based on a shared database (Redis and PostgreSQL) backend.
So my guess is it would be possible to create a "warm" spare server of
a matrix home server with regular PostgreSQL replication, but
that is not documented in the Synapse manual. This sort of setup
would also not be useful to deal with networking issues or denial of
service attacks, as you will not be able to spread the load over
multiple network locations easily. Redis and PostgreSQL heroes are
welcome to provide their multi-primary solution in the comments. In
the meantime, I'll just point out this is a solution that's handled
somewhat more gracefully in IRC, by having the possibility of
delegating the authentication layer.
Delegations
If you do not want to run a Matrix server yourself, it's possible to
delegate the entire thing to another server. There's a server
discovery API which uses the .well-known pattern (or SRV
records, but that's "not recommended" and a bit confusing) to
delegate that service to another server. Be warned that the server
still needs to be explicitly configured for your domain. You can't
just put:
"m.server": "matrix.org:443"
... on https://example.com/.well-known/matrix/server and start using
@you:example.com as a Matrix ID. That's because Matrix doesn't
support "virtual hosting" and you'd still be connecting to rooms and
people with your matrix.org identity, not example.com as you would
normally expect. This is also why you cannot rename your home
server.
The server discovery API is what allows servers to find each
other. Clients, on the other hand, use the client-server discovery
API: this is what allows a given client to find your home server
when you type your Matrix ID on login.
Performance
The high availability discussion brushed over the performance of
Matrix itself, but let's now dig into that.
Horizontal scalability
There were serious scalability issues of the main Matrix server,
Synapse, in the past. So the Matrix team has been working hard to
improve its design. Since Synapse 1.22 the home server can
horizontally scale to multiple workers (see this blog post for details)
which can make it easier to scale large servers.
Other implementations
There are other promising home servers implementations from a
performance standpoint (dendrite, Golang, entered beta in late
2020; conduit, Rust, beta; others), but none of those
are feature-complete so there's a trade-off to be made there. Synapse
is also adding a lot of feature fast, so it's an open question whether
the others will ever catch up. (I have heard that Dendrite might
actually surpass Synapse in features within a few years, which would
put Synapse in a more "LTS" situation.)
Latency
Matrix can feel slow sometimes. For example, joining the "Matrix HQ"
room in Element (from matrix.debian.social) takes a few minutes
and then fails. That is because the home server has to sync the
entire room state when you join the room. There was promising work on
this announced in the lengthy 2021 retrospective, and some of
that work landed (partial sync) in the 1.53 release already.
Other improvements coming include sliding sync, lazy loading
over federation, and fast room joins. So that's actually
something that could be fixed in the fairly short term.
But in general, communication in Matrix doesn't feel as "snappy" as on
IRC or even Signal. It's hard to quantify this without instrumenting a
full latency test bed (for example the tools I used in the terminal
emulators latency tests), but
even just typing in a web browser feels slower than typing in a xterm
or Emacs for me.
Even in conversations, I "feel" people don't immediately respond as
fast. In fact, this could be an interesting double-blind experiment to
make: have people guess whether they are talking to a person on
Matrix, XMPP, or IRC, for example. My theory would be that people
could notice that Matrix users are slower, if only because of the TCP
round-trip time each message has to take.
Matrix: "struggled to send and receive messages", joining a room
takes forever as it has to sync all history, "took 20-30 seconds
for my messages to be sent and another 20 seconds for further
responses"
XMPP: "worked in real-time, full encryption, with nearly zero
lag"
So that was interesting. I suspect IRC would have also fared better,
but that's just a feeling.
Other improvements to the transport layer include support for
websocket and the CoAP proxy work from 2019 (targeting
100bps links), but both seem stalled at the time of writing. The
Matrix people have also announced the pinecone p2p overlay
network which aims at solving large, internet-scale routing
problems. See also this talk at FOSDEM 2022.
Usability
Onboarding and workflow
The workflow for joining a room, when you use Element web, is not
great:
As you might have guessed by now, there is a specification to
solve this, but web browsers need to adopt it as well, so that's far
from actually being solved. At least browsers generally know about the
matrix: scheme, it's just not exactly clear what they should do with
it, especially when the handler is just another web page (e.g. Element
web).
In general, when compared with tools like Signal or WhatsApp, Matrix
doesn't fare so well in terms of user discovery. I probably have some
of my normal contacts that have a Matrix account as well, but there's
really no way to know. It's kind of creepy when Signal tells you
"this person is on Signal!" but it's also pretty cool that it works,
and they actually implemented it pretty well.
Registration is also less obvious: in Signal, the app confirms your
phone number automatically. It's friction-less and quick. In Matrix,
you need to learn about home servers, pick one, register (with a
password! aargh!), and then setup encryption keys (not default),
etc. It's a lot more friction.
And look, I understand: giving away your phone number is a huge
trade-off. I don't like it either. But it solves a real problem and
makes encryption accessible to a ton more people. Matrix does have
"identity servers" that can serve that purpose, but I don't feel
confident sharing my phone number there. It doesn't help that the
identity servers don't have private contact discovery: giving them
your phone number is a more serious security compromise than with
Signal.
There's a catch-22 here too: because no one feels like giving away
their phone numbers, no one does, and everyone assumes that stuff
doesn't work anyways. Like it or not, Signal forcing people to
divulge their phone number actually gives them critical mass that
means actually a lot of my relatives are on Signal and I don't have
to install crap like WhatsApp to talk with them.
5 minute clients evaluation
Throughout all my tests I evaluated a handful of Matrix clients,
mostly from Flathub because almost none of them are packaged in
Debian.
Right now I'm using Element, the flagship client from Matrix.org, in a
web browser window, with the PopUp Window extension. This makes
it look almost like a native app, and opens links in my main browser
window (instead of a new tab in that separate window), which is
nice. But I'm tired of buying memory to feed my web browser, so this
indirection has to stop. Furthermore, I'm often getting completely
logged off from Element, which means re-logging in, recovering my
security keys, and reconfiguring my settings. That is extremely
annoying.
Coming from Irssi, Element is really "GUI-y" (pronounced
"gooey"). Lots of clickety happening. To mark conversations as read,
in particular, I need to click-click-click on all the tabs that have
some activity. There's no "jump to latest message" or "mark all as
read" functionality as far as I could tell. In Irssi the former is
built-in (alt-a) and I made a custom /READ command for
the latter:
/ALIAS READ script exec \$_->activity(0) for Irssi::windows
And yes, that's a Perl script in my IRC client. I am not aware of any
Matrix client that does stuff like that, except maybe Weechat, if we
can call it a Matrix client, or Irssi itself, now that it has a
Matrix plugin (!).
As for other clients, I have looked through the Matrix Client
Matrix (confusing right?) to try to figure out which one to try,
and, even after selecting Linux as a filter, the chart is just too
wide to figure out anything. So I tried those, kind of randomly:
Fractal
Mirage
Nheko
Quaternion
Unfortunately, I lost my notes on those, I don't actually remember
which one did what. I still have a session open with Mirage, so I
guess that means it's the one I preferred, but I remember they were
also all very GUI-y.
Maybe I need to look at weechat-matrix or gomuks. At least Weechat
is scriptable so I could continue playing the power-user. Right now my
strategy with messaging (and that includes microblogging like Twitter
or Mastodon) is that everything goes through my IRC client, so Weechat
could actually fit well in there. Going with gomuks, on the other
hand, would mean running it in parallel with Irssi or ... ditching
IRC, which is a leap I'm not quite ready to take just yet.
Oh, and basically none of those clients (except Nheko and Element)
support VoIP, which is still kind of a second-class citizen in
Matrix. It does not support large multimedia rooms, for example:
Jitsi was used for FOSDEM instead of the native videoconferencing
system.
Bots
This falls a little aside the "usability" section, but I didn't know
where to put this... There's a few Matrix bots out there, and you are
likely going to be able to replace your existing bots with Matrix
bots. It's true that IRC has a long and impressive history with lots
of various bots doing various things, but given how young Matrix is,
there's still a good variety:
maubot: generic bot with tons of usual plugins like sed, dice,
karma, xkcd, echo, rss, reminder, translate, react, exec,
gitlab/github webhook receivers, weather, etc
opsdroid: framework to implement "chat ops" in Matrix,
connects with Matrix, GitHub, GitLab, Shell commands, Slack, etc
One thing I haven't found an equivalent for is Debian's
MeetBot. There's an archive bot but it doesn't have topics
or a meeting chair, or HTML logs.
Working on Matrix
As a developer, I find Matrix kind of intimidating. The specification
is huge. The official specification itself looks somewhat
digestable: it's only 6 APIs so that looks, at first, kind of
reasonable. But whenever you start asking complicated questions about
Matrix, you quickly fall into the Matrix Spec Change
specification (which, yes, is a separate specification). And there are
literally hundreds of MSCs flying around. It's hard to tell
what's been adopted and what hasn't, and even harder to figure out if
your specific client has implemented it.
(One trendy answer to this problem is to "rewrite it in rust": Matrix
are working on implementing a lot of those specifications in a
matrix-rust-sdk that's designed to take the implementation
details away from users.)
Just taking the latest weekly Matrix report, you find that
three new MSCs proposed, just last week! There's even a graph that
shows the number of MSCs is progressing steadily, at 600+ proposals
total, with the majority (300+) "new". I would guess the "merged" ones
are at about 150.
That's a lot of text which includes stuff like 3D worlds which,
frankly, I don't think you should be working on when you have such
important security and usability problems. (The internet as a whole,
arguably, doesn't fare much better. RFC600 is a really obscure
discussion about "INTERFACING AN ILLINOIS PLASMA TERMINAL TO THE
ARPANET". Maybe that's how many MSCs will end up as well, left
forgotten in the pits of history.)
And that's the thing: maybe the Matrix people have a different
objective than I have. They want to connect everything to everything,
and make Matrix a generic transport for all sorts of applications,
including virtual reality, collaborative editors, and so on.
I just want secure, simple messaging. Possibly with good file
transfers, and video calls. That it works with existing stuff is good,
and it should be federated to remove the "Signal point of
failure". So I'm a bit worried with the direction all those MSCs are
taking, especially when you consider that clients other than Element
are still struggling to keep up with basic features like end-to-end
encryption or room discovery, never mind voice or spaces...
Conclusion
Overall, Matrix is somehow in the space XMPP was a few years ago. It
has a ton of features, pretty good clients, and a large
community. It seems to have gained some of the momentum that XMPP has
lost. It may have the most potential to replace Signal if something
bad would happen to it (like, I don't know, getting banned or
going nuts with cryptocurrency)...
But it's really not there yet, and I don't see Matrix trying to get
there either, which is a bit worrisome.
Looking back at history
I'm also worried that we are repeating the errors of the past. The
history of federated services is really fascinating:. IRC, FTP, HTTP,
and SMTP were all created in the early days of the internet, and are
all still around (except, arguably, FTP, which was removed from major
browsers recently). All of them had to face serious challenges in
growing their federation.
IRC had numerous conflicts and forks, both at the technical level
but also at the political level. The history of IRC is really
something that anyone working on a federated system should study in
detail, because they are bound to make the same mistakes if they are
not familiar with it. The "short" version is:
1988: Finnish researcher publishes first IRC source code
1989: 40 servers worldwide, mostly universities
1990: EFnet ("eris-free network") fork which blocks the "open
relay", named Eris - followers of Eris form the A-net, which
promptly dissolves itself, with only EFnet remaining
1992: Undernet fork, which offered authentication ("services"),
routing improvements and timestamp-based channel synchronisation
1994: DALnet fork, from Undernet, again on a technical disagreement
1995: Freenode founded
1996: IRCnet forks from EFnet, following a flame war of historical
proportion, splitting the network between Europe and the Americas
1997: Quakenet founded
1999: (XMPP founded)
2001: 6 million users, OFTC founded
2002: DALnet peaks at 136,000 users
2003: IRC as a whole peaks at 10 million users, EFnet peaks at
141,000 users
2004: (Facebook founded), Undernet peaks at 159,000 users
2005: Quakenet peaks at 242,000 users, IRCnet peaks at 136,000
(Youtube founded)
2006: (Twitter founded)
2009: (WhatsApp, Pinterest founded)
2010: (TextSecure AKA Signal, Instagram founded)
2011: (Snapchat founded)
~2013: Freenode peaks at ~100,000 users
2016: IRCv3 standardisation effort started (TikTok founded)
2021: Freenode self-destructs, Libera chat founded
2022: Libera peaks at 50,000 users, OFTC peaks at 30,000 users
(The numbers were taken from the Wikipedia page and
Netsplit.de. Note that I also include other networks launch in
parenthesis for context.)
Pretty dramatic, don't you think? Eventually, somehow, IRC became
irrelevant for most people: few people are even aware of it now. With
less than a million users active, it's smaller than Mastodon, XMPP, or
Matrix at this point.1 If I were to venture a guess, I'd say that
infighting, lack of a standardization body, and a somewhat annoying
protocol meant the network could not grow. It's also possible that the
decentralised yet centralised structure of IRC networks limited their
reliability and growth.
But large social media companies have also taken over the space:
observe how IRC numbers peak around the time the wave of large social
media companies emerge, especially Facebook (2.9B users!!) and Twitter
(400M users).
Where the federated services are in history
Right now, Matrix, and Mastodon (and email!) are at the
"pre-EFnet" stage: anyone can join the federation. Mastodon has
started working on a global block list of fascist servers which is
interesting, but it's still an open federation. Right now, Matrix is
totally open, but matrix.org publishes a (federated) block list
of hostile servers (#matrix-org-coc-bl:matrix.org, yes, of course
it's a room).
Interestingly, Email is also in that stage, where there are block
lists of spammers, and it's a race between those blockers and
spammers. Large email providers, obviously, are getting closer to the
EFnet stage: you could consider they only accept email from themselves
or between themselves. It's getting increasingly hard to deliver mail
to Outlook and Gmail for example, partly because of bias against small
providers, but also because they are including more and more
machine-learning tools to sort through email and those systems are,
fundamentally, unknowable. It's not quite the same as splitting the
federation the way EFnet did, but the effect is similar.
HTTP has somehow managed to live in a parallel universe, as it's
technically still completely federated: anyone can start a web server
if they have a public IP address and anyone can connect to it. The
catch, of course, is how you find the darn thing. Which is how Google
became one of the most powerful corporations on earth, and how they
became the gatekeepers of human knowledge online.
I have only briefly mentioned XMPP here, and my XMPP fans will
undoubtedly comment on that, but I think it's somewhere in the middle
of all of this. It was co-opted by Facebook and Google, and
both corporations have abandoned it to its fate. I remember fondly the
days where I could do instant messaging with my contacts who had a
Gmail account. Those days are gone, and I don't talk to anyone over
Jabber anymore, unfortunately. And this is a threat that Matrix still
has to face.
It's also the threat Email is currently facing. On the one hand
corporations like Facebook want to completely destroy it and have
mostly succeeded: many people just have an email account to
register on things and talk to their friends over Instagram or
(lately) TikTok (which, I know, is not Facebook, but they started that
fire).
On the other hand, you have corporations like Microsoft and Google who
are still using and providing email services because, frankly, you
still do need email for stuff, just like fax is still around
but they are more and more isolated in their own silo. At this point,
it's only a matter of time they reach critical mass and just decide
that the risk of allowing external mail coming in is not worth the
cost. They'll simply flip the switch and work on an allow-list
principle. Then we'll have closed the loop and email will be
dead, just like IRC is "dead" now.
I wonder which path Matrix will take. Could it liberate us from these
vicious cycles?
Update: this generated some discussions on lobste.rs.
Notable omission from that list: Youtube, with its mind-boggling
2.6 billion users...
Those are not the kind of numbers you just "need to convince a
brother or sister" to grow the network...