I am tasked with contributing to the debianhugo project which aim is to re-design the old debian pages and make the content better accessible. We've since reached a significant milestone and migrated multiple pages including the start, intro, news and now the download page.
creating "simple" and "advanced" download pages
At first we made the "simple" download page:
target audience; less experienced users
only added more common download arhitectures/options i.e amd64-64 bit pc and arm64
descriptive content for easy user experience; listing positives and negatives of each option, adding download sizes
interactive download cards with iso, torrent and debian-cd mirror selections.
"advanced" download page:
more download architectures and options including testing release streams
straight to point content
challenges/issues while developing
The mirror selection option while it might help with faster downloads depending on the region is still somewhat a manual proccess on the user end and could come with various complications like the chosen debian-cd mirror not having the latest version of debian.
We're looking into testing if the delivery of the images/ISOs is also possible to be done through the Fastly CDN, this would prevent us to provide manual mirror selection.
A Little Vice is a stand-alone self-published
magical girl novel. It
is the author's first novel.
C is a high school student and frequent near-victim of monster attacks.
Due to the nefarious work of Avaritia Wolf and her allies, his high school
is constantly attacked by Beasts, who are magical corruptions of some
internal desire taken to absurd extremes. Standing in their way are the
Angelic Saints: magical girls who transform into Saint Castitas, Saint
Diligentia, and Saint Temperantia and fight the monsters. The monsters for
some reason seem disposed to pick C as their victim for hostage-taking,
mind control, use as a human shield, and other rather traumatic
activities. He's always rescued by the Saints before any great harm is
done, but in some ways this makes the situation worse.
It is obvious to C that the Saints are his three friends Inessa, Ida, and
Temperance, even though no one else seems able to figure this out despite
the blatant clues. Inessa has been his best friend since childhood when
she was awkward and needed his support. Now, she and his other friends
have become literal heroes, beautiful and powerful and capable, constantly
protecting the school and innocent people, and C is little more than a
helpless burden to be rescued. More than anything else, he wishes he could
be an Angelic Saint like them, but of course the whole idea is impossible.
Boys don't get to be magical girls.
(I'm using he/him pronouns for C in this review because C uses them for
himself for most of the book.)
This is a difficult book to review because it is deeply focused on
portraying a specific internal emotional battle in all of its
sometimes-ugly complexity, and to some extent it prioritizes that
portrayal over conventional story-telling. You have probably already
guessed that this is a transgender coming-out story Elkin's choice of
the magical girl genre was done with deep understanding of its role in
transgender narratives but more than that, it is a transgender
coming-out story of a very specific and closely-observed type. C knows who
he wishes he was, but he is certain that this transformation is absolutely
impossible. He is very deep in a cycle of self-loathing for wanting
something so manifestly absurd and insulting to people who have the
virtues that C does not.
A Little Vice is told in the first person from C's perspective, and
most of this book is a relentless observation of C's anxiety and shame
spiral and reflexive deflection of any possibility of a way out. This is
very well-written: Elkin knows the reader is going to disagree with C's
internalized disgust and hopelessness, knows the reader desperately wants
C to break out of that mindset, and clearly signals in a myriad of adroit
ways that Elkin is on the reader's side and does not agree with C's
analysis. C's friends are sympathetic, good-hearted people, and while
sometimes oblivious, it is obvious to the reader that they're also on the
reader's side and would help C in a heartbeat if they saw an opening. But
much of the point of the book is that it's not that easy, that breaking
out of the internal anxiety spiral is nearly impossible, and that C is
very good at rejecting help, both because he cannot imagine what form it
could take but also because he is certain that he does not deserve it.
In other words, much of the reading experience of this book involves
watching C torture and insult himself. It's all the more effective because
it isn't gratuitous. C's internal monologue sounds exactly like how an
anxiety spiral feels, complete with the sort of half-effective coping
mechanisms, deflections, and emotional suppression one develops to blunt
that type of emotional turmoil.
I normally hate this kind of book. I am a happy ending and competence porn
reader by default. The world is full of enough pain that I don't turn to
fiction to read about more pain. It says a lot about how well-constructed
this book is that I stuck with it. Elkin is going somewhere with the
story, C gets moments of joy and delight along the way to keep the reader
from bogging down completely, and the best parts of the book feel like a
prolonged musical crescendo with suspended chords. There is a
climax coming, but Elkin is going to make you wait for it for far longer
than you want to.
The main element that protects A Little Vice from being too grim is
that it is a genre novel that is very playful about both magical girls and
superhero tropes in general. I've already alluded to one of those
elements: Elkin plays with the Mask Principle (the inability of people to
see through entirely obvious secret identities) in knowing and
entertaining ways. But there are also villains, and that leads me to the
absolutely delightful Avaritia Wolf, who for me was the best character in
this book.
The Angelic Saints are not the only possible approach to magical girl
powers in this universe. There are villains who can perform a similar
transformation, except they embrace a vice rather than a virtue. Avaritia
Wolf embraces the vice of greed. They (Avaritia's pronouns change over the
course of the book) also have a secret identity, which I suspect will be
blindingly obvious to most readers but which I'll avoid mentioning since
it's still arguably a spoiler.
The primary plot arc of this book is an attempt to recruit C to the side
of the villains. The Beasts are drawn to him because he has magical
potential, and the villains are less picky about gender. This initially
involves some creepy and disturbing mind control, but it also brings C
into contact with Avaritia and Avaritia's very specific understanding of
greed. As far as Avaritia is concerned, greed means wanting whatever they
want, for whatever reason they feel like wanting it, and there is
absolutely no reason why that shouldn't include being greedy for their
friends to be happy. Or doing whatever they can to make their friends
happy, whether or not that looks like villainy.
Elkin does two things with this plot that I thought were remarkably
skillful. The first is that she directly examines and then undermines the
"easy" transgender magical girl ending. In a world of transformation
magic, someone who wants to be a girl could simply turn into a girl and
thus apparently resolve the conflict in a way that makes everyone happy. I
think there is an important place for that story (I am a vigorous defender
of escapist fantasy and happy endings), but that is not the story that
Elkin is telling. I won't go into the details of why and how the story
complicates and undermines this easy ending, but it's a lot of why this
book feels both painful and honest to a specific, and very not easy,
transgender experience, even though it takes place in an utterly
unrealistic world.
But the second, which is more happy and joyful, is that Avaritia gleefully
uses a wholehearted embrace of every implication of the vice of greed to
bulldoze the binary morality of the story and question the classification
of human emotions into virtues and vices. They are not a hero, or even all
that good; they have some serious flaws and a very anarchic attitude
towards society. But Avaritia provides the compelling, infectious thrill
of the character who looks at the social construction of morality that is
constraining the story and decides that it's all bullshit and refuses to
comply. This is almost the exact opposite of C's default emotional
position at the start of the book, and watching the two characters play
off of each other in a complex friendship is an absolute delight.
The ending of this book is complicated, messy, and incomplete. It is the
sort of ending that I think could be incredibly powerful if it hits
precisely the right chords with the reader, but if you're not that reader,
it can also be a little heartbreaking because Elkin refuses to provide an
easy resolution. The ending also drops some threads that I wish Elkin
hadn't dropped; there are some characters who I thought deserved a
resolution that they don't get. But this is one of those books where the
author knows exactly what story they're trying to tell and tells it
whether or not that fits what the reader wants. Those books are often not
easy reading, but I think there's something special about them.
This is not the novel for people who want detailed world-building that
puts a solid explanation under events. I thought Elkin did a great job
playing with the conventions of an episodic anime, including starting the
book on Episode 12 to imply C's backstory with monster attacks and hinting
at a parallel light anime story by providing TV-trailer-style plot
summaries and teasers at the start and end of each chapter. There is a
fascinating interplay between the story in which the Angelic Saints are
the protagonists, which the reader can partly extrapolate, and the novel
about C that one is actually reading. But the details of the
world-building are kept at the anime plot level: There's an arch-villain,
a World Tree, and a bit of backstory, but none of it makes that much sense
or turns into a coherent set of rules. This is a psychological novel; the
background and rules exist to support C's story.
If you do want that psychological novel... well, I'm not sure whether to
recommend this book or not. I admire the construction of this book a great
deal, but I don't think appealing to the broadest possible audience was
the goal. C's anxiety spiral is very repetitive, because anxiety spirals
are very repetitive, and you have to be willing to read for the grace
notes on the doom loop if you're going to enjoy this book. The
sentence-by-sentence writing quality is fine but nothing remarkable, and
is a bit shy of the average traditionally-published novel. The main appeal
of A Little Vice is in the deep and unflinching portrayal of a
specific emotional journey. I think this book is going to work if you're
sufficiently invested in that journey that you are willing to read the
brutal and repetitive parts. If you're not, there's a chance you will
bounce off this hard.
I was invested, and I'm glad I read this, but caveat emptor. You
may want to try a sample first.
One final note: If you're deep in the book world, you may wonder, like I
did, if the title is a reference to Hanya Yanagihara's (in)famous A
Little Life. I do not know for certain I have not read that book
because I am not interested in being emotionally brutalized but if it
is, I don't think there is much similarity. Both books are to some extent
about four friends, but I couldn't find any other obvious connections from
some Wikipedia reading, and A Little Vice, despite C's emotional
turmoil, seems to be considerably more upbeat.
Content notes: Emotionally abusive parent, some thoughts of self-harm,
mind control, body dysmorphia, and a lot (a lot) of shame and
self-loathing.
Rating: 7 out of 10
Posted on February 24, 2025
Tags: madeof:atoms, craft:3dprint, craft:sewing
For quite a few years, I ve been using pattern weights instead of pins
when cutting fabric, starting with random objects and then mostly using
some big washers from the local hardware store.
However, at about 22 g per washer, I needed quite a few of them, and
dealing with them tended to get unwieldy; I don t remember how it
happened, but one day I decided to make myself some bigger weights with
a few washers each.
I suspect I had seen somebody online with some nice hexagonal pattern
weights, and hexagonal of course reminded me of the Stickers
Standard, so of course I settled on an hexagon 5 cm tall and I decided
I could 3D-print it in a way that could be filled with washers for
weight.
Rather than bothering with adding a lid (and fitting it), I decided to
close the bottom by gluing a piece of felt, with the added advantage
that it would protect whatever the weight was being used on. And of
course the top could be decorated with a nerdish sticker, because,
well, I am a nerd.
I made a few of these pattern weights, used them for a while, was happy
with them, and then a few days ago I received some new hexagonal
stickers I had had printed, and realized that while I had taken a
picture with all of the steps in assembling them, I had never published
any kind of instructions on how to make them and I had not even pushed
the source file on the craft tools git repository.
And yesterday I fixed that: the instructions are now on my craft
pattern website,
with generated STL files, the git repository has been updated with the
current sources, and now I ve even written this blog post :)
Anarcat recently wrote about Qalculate, and I think I m a convert, even though
I ve only barely scratched the surface.
The thing I almost immediately started using it for is time calculations.
When I started tracking my time, I
quickly found that Timewarrior was good at
keeping all the data I needed, but I often found myself extracting bits of
it and reprocessing it in variously clumsy ways. For example, I often don t
finish a task in one sitting; maybe I take breaks, or I switch back and
forth between a couple of different tasks. The raw output of timew
summary is a bit clumsy for this, as it shows each chunk of time spent as
a separate row:
$ timewsummary2025-02-18Debian
Wk Date Day Tags Start End Time TotalW8 2025-02-18 Tue CVE-2025-26465, Debian, 9:41:44 10:24:17 0:42:33 next, openssh Debian, FTBFS with GCC-15, 10:24:17 10:27:12 0:02:55 icoutils Debian, FTBFS with GCC-15, 11:50:05 11:57:25 0:07:20 kali Debian, Upgrade to 0.67, 11:58:21 12:12:41 0:14:20 python_holidays Debian, FTBFS with GCC-15, 12:14:15 12:33:19 0:19:04 vigor Debian, FTBFS with GCC-15, 12:39:02 12:39:38 0:00:36 python_setproctitle Debian, Upgrade to 1.3.4, 12:39:39 12:46:05 0:06:26 python_setproctitle Debian, FTBFS with GCC-15, 12:48:28 12:49:42 0:01:14 python_setproctitle Debian, Upgrade to 3.4.1, 12:52:07 13:02:27 0:10:20 1:44:48 python_charset_normalizer 1:44:48
So I wrote this Python program to help me:
#! /usr/bin/python3"""Summarize timewarrior data, grouped and sorted by time spent."""importjsonimportsubprocessfromargparseimportArgumentParser,RawDescriptionHelpFormatterfromcollectionsimportdefaultdictfromdatetimeimportdatetime,timedelta,timezonefromoperatorimportitemgetterfromrichimportbox,printfromrich.tableimportTableparser=ArgumentParser(description=__doc__,formatter_class=RawDescriptionHelpFormatter)parser.add_argument("-t","--only-total",default=False,action="store_true")parser.add_argument("range",nargs="?",default=":today",help="Time range (usually a hint, e.g. :lastweek)",)parser.add_argument("tag",nargs="*",help="Tags to filter by")args=parser.parse_args()entries:defaultdict[str,timedelta]=defaultdict(timedelta)now=datetime.now(timezone.utc)forentryinjson.loads(subprocess.run(["timew","export",args.range,*args.tag],check=True,capture_output=True,text=True,).stdout):start=datetime.fromisoformat(entry["start"])if"end"inentry:end=datetime.fromisoformat(entry["end"])else:end=nowentries[", ".join(entry["tags"])]+=end-startifnotargs.only_total:table=Table(box=box.SIMPLE,highlight=True)table.add_column("Tags")table.add_column("Time",justify="right")fortags,timeinsorted(entries.items(),key=itemgetter(1),reverse=True):table.add_row(tags,str(time))print(table)total=sum(entries.values(),start=timedelta())hours,rest=divmod(total,timedelta(hours=1))minutes,rest=divmod(rest,timedelta(minutes=1))seconds=rest.secondsprint(f"Total time: hours:02:minutes:02:seconds:02")
$ summarize-time2025-02-18Debian
Tags Time CVE-2025-26465, Debian, next, openssh 0:42:33 Debian, FTBFS with GCC-15, vigor 0:19:04 Debian, Upgrade to 0.67, python_holidays 0:14:20 Debian, Upgrade to 3.4.1, python_charset_normalizer 0:10:20 Debian, FTBFS with GCC-15, kali 0:07:20 Debian, Upgrade to 1.3.4, python_setproctitle 0:06:26 Debian, FTBFS with GCC-15, icoutils 0:02:55 Debian, FTBFS with GCC-15, python_setproctitle 0:01:50Total time: 01:44:48
Much nicer. But that only helps with some of my reporting. At the end of a
month, I have to work out how much time to bill Freexian for and fill out a
timesheet, and for various reasons those queries don t correspond to single
timew tags: they sometimes correspond to the sum of all time spent on
multiple tags, or to the time spent on one tag minus the time spent on
another tag, or similar. As a result I quite often have to do basic
arithmetic on time intervals; but that s surprisingly annoying! I didn t
previously have good tools for that, and was reduced to doing things like
str(timedelta(hours=..., minutes=..., seconds=...) + ...) in Python,
which gets old fast.
Instead:
I also often want to work out how much of my time I ve spent on Debian work
this month so far, since Freexian pays me for up to 20% of my work time on
Debian; if I m
under that then I might want to prioritize more Debian projects, and if I m
over then I should be prioritizing more Freexian projects as otherwise I m
not going to get paid for that time.
$ summarize-time-t:monthFreexian
Total time: 69:19:42$ summarize-time-t:monthDebian
Total time: 24:05:30$ qalc'24:05:30 / (24:05:30 + 69:19:42) to %'(86730 / 3600)/ ((86730 / 3600) + (249582 / 3600)) 25.78855349%
Posted on February 23, 2025
Tags: madeof:atoms, craft:sewing, FreeSoftWear
Many years ago I made myself a vest with lots of pockets1 in a few layers of cheap cotton, and wore the hell out of it,
for the added warmth, but most importantly for the convenience provided
by the pockets.
Then a few years ago the cheap cotton had started to get worn, and I
decided I needed to replace it. I found a second choice (and thus
cheaper :) ) version of a water-repellent cotton
and made another vest, lined with regular cotton, for a total of just
two layers.
This time I skipped a few pockets that I had found I didn t use that
much, and I didn t add a hood, which didn t play that well when worn
over a hoodie, but I added some detached sleeves, for additional wind
protection.
This left about 60 cm and some odd pieces of leftover fabric in my
stash, for which I had no plan.
And then February2 came, and I needed a quick simple mindless
handsewing projects for the first weekend, I saw the vest (which I m
wearing as much as the old one), the sleeves (which have been used much
less, but I d like to change this) and thought about making a matching
hood for it, using my square hood pattern.
Since the etaproof is a bit stiff and not that nice to the touch I
decide to line3 it with the same cotton as the vest and
sleeves, and in the style of the pattern I did so by finishing each
panel with its own lining (with regular cotton thread) and then
whipstitching the panels together with the corespun cotton/poly thread
recommended by the seller of the fabric. I m not sure this is the best
way to construct something that is supposed to resist the rain, but if I
notice issues I can always add some sealing tape afterwards.
I do have a waterproof cape to wear in case of real rain, so this is
only supposed to work for light rain anyway, and it may prove not to be
an issue.
As something designed to be worn in light rain, this is also something
likely to be worn in low light conditions, where 100% black may not be
the wisest look. On the vest I had added reflective piping to the
armscyes, but I was out of the same piping.
I did however have a spool of reflector thread made of glass fibre by
Rico Design, which I think was originally sold to be worked into
knitting or crochet projects (it is now discontinued) and I had never
used.
I decided to try and sew a decorative blanket stitch border, a decision
I had reasons to regret, since the thread broke and tangled like crazy,
but in the end it was done, I like how it looks, and it seems pretty
functional. I hope it won t break with time and use, and if it does I ll
either fix it or try to redo with something else.
Of course, the day I finished sewing the reflective border it stopped
raining, so I haven t worn it yet, but I hope I ll be able to, and if it
is an horrible failure I ll make sure to update this post.
and I ve just realized that I haven t migrated that pattern
to my pattern website, and I should do that. just don t hold your
breath for it to happen O:-). And for the time being it will not have
step-by-step pictures, as I currently don t need another vest.
and February of course means a weekend in front of a
screen that is showing a live-streamed conference.
and of course I updated the pattern with instructions on
how to add a lining.
For my PhD, my colleagues/collaborators and I built a distributed
stream-processing system using Haskell.
There are several other Haskell stream-processing systems. How do they
compare?
First, let's briefly discuss and define streaming in this context.
Structure and Interpretation of Computer Programs introduces Streams
as an analogue of lists, to support delayed evaluation. In brief, the
inductive list type (a list is either an empty list or a head element
pre-pended to another list) is replaced with a structure with a head
element and a promise which, when evaluated, will generate the tail
(which in turn may have a head element and a promise to generate another
tail, culminating in the equivalent of an empty list.) Later on SICP
also covers lazy evaluation.
However, the streaming we're talking about originates in the relational
community, rather than the functional one, and is subtly different. It's
about building a pipeline of processing that receives and emits data but
doesn't need to (indeed, cannot) reference the whole stream (which may
be infinite) at once.
Haskell streaming systems
Now let's go over some Haskell streaming systems.
conduit (2011-)
Conduit is the oldest of the ones I am reviewing here, but I doubt it's
the first in the Haskell ecosystem. If I've made any obvious omissions,
please let me know!
Conduit provides a new set of types to model streaming data, and a completely
new set of functions which are analogues of standard Prelude functions, e.g.
sumC in place of sum. It provides its own combinator(s) such as . (
aka fuse)
which is like composition but reads left-to-right.
The motivation for this is to enable (near?) constant memory usage for
processing large streams of data -- presumably versus using a list-based
approach and to provide some determinism: the README gives the example of
"promptly closing file handles". I think this is another way of saying
that it uses strict evaluation, or at least avoids lazy evaluation for
some things.
Conduit offers interleaved effects: which is to say, IO can be performed
mid-stream.
Conduit supports distributed operation via Data.Conduit.Network in the
conduit-extra package. Michael Snoyman, principal Conduit author, wrote
up how to use it here: https://www.yesodweb.com/blog/2014/03/network-conduit-async
To write a distributed Conduit application, the application programmer must
manually determine the boundaries between the clients/servers and write specific
code to connect them.
pipes (2012-)
The Pipes
Tutorial
contrasts itself with "Conventional Haskell stream programming": whether that
means Conduit or something else, I don't know.
Paraphrasing their pitch: Effects, Streaming Composability: pick two. That's
the situation they describe for stream programming prior to Pipes. They argue
Pipes offers all three.
Pipes offers it's own combinators (which read left-to-right)
and offers interleaved effects.
At this point I can't really see what fundamentally distinguishes Pipes from
Conduit.
Pipes has some support for distributed operation via the sister library
pipes-network. It
looks like you must send and receive ByteStrings, which means rolling
your own serialisation for other types. As with Conduit, to send or receive
over a network, the application programmer must divide their program up
into the sub-programs for each node, and add the necessary ingress/egress
code.
io-streams (2013-)
io-streams emphasises simple primitives. Reading and writing is done
under the IO Monad, thus, in an effectful (but non-pure) context. The
presence or absence of further stream data are signalled by using the
Maybe type (Just more data or Nothing: the producer has finished.)
It provides a library of functions that shadow the standard Prelude, such
as S.fromList, S.mapM, etc.
It's not clear to me what the motivation for io-streams is, beyond
providing a simple interface. There's no declaration of intent that I can find
about (e.g.) constant-memory operation.
There's no mention of or support (that I can find) for distributed
operation.
streaming (2015-)
Similar to io-streams, Streaming emphasises providing a simple
interface that gels well with traditional Haskell methods. Streaming
provides effectful streams (via a Monad -- any Monad?) and a collection
of functions for manipulating streams which are designed to closely
mimic standard Prelude (and Data.List) functions.
Streaming doesn't push its own combinators: the examples provided
use $ and read right-to-left.
The motivation for Streaming seems to be to avoid memory leaks caused by
extracting pure lists from IO with traditional functions like mapM,
which require all the list constructors to be evaluated, the list to be
completely deconstructed, and then a new list constructed.
Like io-streams, the focus of the library is providing a low-level
streaming abstraction, and there is no support for distributed operation.
streamly (2017-)
Streamly appears to have the grand goal of providing a unified programming
tool as suited for quick-and-dirty programming tasks (normally the domain of
scripting languages) and high-performance work (C, Java, Rust, etc.). Their
intended audience appears to be everyone, or at least, not just existing
Haskell programmers. See their rationale
Streamly offers an interface to permit composing concurrent (note: not
distributed) programs via combinators. It relies upon fusing a streaming
pipeline to remove intermediate list structure allocations and de-allocations
(i.e. de-forestation, similar to GHC rewrite rules)
The examples I've seen use standard combinators (e.g. Control.Function.&,
which reads left-to-right, and Applicative).
Streamly provide benchmarks
versus Haskell pure lists, Streaming, Pipes and Conduit: these generally
show Streamly several orders of magnitude faster.
I'm finding it hard to evaluate Streamly. It's big, and it's focus is wide.
It provides shadows of Prelude functions, as many of these libraries do.
wrap-up
It seems almost like it must be a rite-of-passage to write a streaming system
in Haskell. Stones and glass houses, I'm guilty of that
too.
The focus of the surveyed libraries is mostly on providing a streaming
abstraction, normally with an analogous interface to standard Haskell lists.
They differ on various philosophical points (whether to abstract away the
mechanics behind type synonyms, how much to leverage existing Haskell idioms,
etc). A few of the libraries have some rudimentary support for distributed
operation, but this is limited to connecting separate nodes together: in some
cases serialising data remains the application programmer's job, and in all
cases the application programmer must manually carve up their processing
according to a fixed idea of what nodes they are deploying to. They all
define a fixed-function pipeline.
Yesterday i released a new version of
virtnbdbackup with a nice
improvement.
The new version can now detect zeroed regions in the bitmaps by comparing the
block regions against the state within the base bitmap during incremental
backup.
This is helpful if virtual machines run fstrim, as it results in less backup
footprint. Before the incremental backups could grow the same amount of size as
fstrimmed data regions.
I also managed to enhance the tests by using the arch linux cloud images. The
automated github CI tests now actually test backup and restores against a
virtual machine running an real OS.
I can t remember exactly the joke I was making at the time in my
work s slack instance (I m sure it wasn t particularly
funny, though; and not even worth re-reading the thread to work out), but it
wound up with me writing a UEFI binary for the punchline. Not to spoil the
ending but it worked - no pesky kernel, no messing around with userland . I
guess the only part of this you really need to know for the setup here is that
it was a Severance joke,
which is some fantastic TV. If you haven t seen it, this post will seem perhaps
weirder than it actually is. I promise I haven t joined any new cults. For
those who have seen it, the payoff to my joke is that I wanted my machine to
boot directly to an image of
Kier Eagan.
As for how to do it I figured I d give the uefi
crate a shot, and see how it is to use,
since this is a low stakes way of trying it out. In general, this isn t the
sort of thing I d usually post about except this wound up being easier and
way cleaner than I thought it would be. That alone is worth sharing, in the
hopes someome comes across this in the future and feels like they, too, can
write something fun targeting the UEFI.
First thing s first gotta create a rust project (I ll leave that part to you
depending on your life choices), and to add the uefi crate to your
Cargo.toml. You can either use cargo add or add a line like this by hand:
uefi = version = "0.33", features = ["panic_handler", "alloc", "global_allocator"]
We also need to teach cargo about how to go about building for the UEFI target,
so we need to create a rust-toolchain.toml with one (or both) of the UEFI
targets we re interested in:
Unfortunately, I wasn t able to use the
image crate,
since it won t build against the uefi target. This looks like it s
because rustc had no way to compile the required floating point operations
within the image crate without hardware floating point instructions
specifically. Rust tends to punt a lot of that to libm usually, so this isnt
entirely shocking given we re nostd for a non-hardfloat target.
So-called softening requires a software floating point implementation that
the compiler can use to polyfill (feels weird to use the term polyfill here,
but I guess it s spiritually right?) the lack of hardware floating point
operations, which rust hasn t implemented for this target yet. As a result, I
changed tactics, and figured I d use ImageMagick to pre-compute the pixels
from a jpg, rather than doing it at runtime. A bit of a bummer, since I need
to do more out of band pre-processing and hardcoding, and updating the image
kinda sucks as a result but it s entirely manageable.
This will take our input file (kier.jpg), resize it to get as close to the
desired resolution as possible while maintaining aspect ration, then convert it
from a jpg to a flat array of 4 byte RGBA pixels. Critically, it s also
important to remember that the size of the kier.full.jpg file may not actually
be the requested size it will not change the aspect ratio, so be sure to
make a careful note of the resulting size of the kier.full.jpg file.
Last step with the image is to compile it into our Rust bianary, since we
don t want to struggle with trying to read this off disk, which is thankfully
real easy to do.
Remember to use the width and height from the final kier.full.jpg file as the
values for KIER_WIDTH and KIER_HEIGHT. KIER_PIXEL_SIZE is 4, since we
have 4 byte wide values for each pixel as a result of our conversion step into
RGBA. We ll only use RGB, and if we ever drop the alpha channel, we can drop
that down to 3. I don t entirely know why I kept alpha around, but I figured it
was fine. My kier.full.jpg image winds up shorter than the requested height
(which is also qemu s default resolution for me) which means we ll get a
semi-annoying black band under the image when we go to run it but it ll
work.
Anyway, now that we have our image as bytes, we can get down to work, and
write the rest of the code to handle moving bytes around from in-memory
as a flat block if pixels, and request that they be displayed using the
UEFI GOP. We ll just need to hack up a container
for the image pixels and teach it how to blit to the display.
/// RGB Image to move around. This isn't the same as an
/// image::RgbImage , but we can associate the size of
/// the image along with the flat buffer of pixels.
structRgbImage/// Size of the image as a tuple, as the
/// (width, height)
size: (usize, usize),
/// raw pixels we'll send to the display.
inner: Vec<BltPixel>,
impl RgbImage
/// Create a new RgbImage .
fnnew(width: usize, height: usize) -> Self
RgbImage
size: (width, height),
inner: vec![BltPixel::new(0, 0, 0); width * height],
/// Take our pixels and request that the UEFI GOP
/// display them for us.
fnwrite(&self, gop: &mut GraphicsOutput) -> Result
gop.blt(BltOp::BufferToVideo
buffer: &self.inner,
src: BltRegion::Full,
dest: (0, 0),
dims: self.size,
)
impl Index<(usize, usize)>for RgbImage
typeOutput= BltPixel;
fnindex(&self, idx: (usize, usize)) -> &BltPixellet (x, y) = idx;
&self.inner[y * self.size.0+ x]
impl IndexMut<(usize, usize)>for RgbImage
fnindex_mut(&mut self, idx: (usize, usize)) -> &mut BltPixel
let (x, y) = idx;
&mut self.inner[y * self.size.0+ x]
We also need to do some basic setup to get a handle to the UEFI
GOP via the UEFI crate (using
uefi::boot::get_handle_for_protocol
and
uefi::boot::open_protocol_exclusive
for the GraphicsOutput
protocol), so that we have the object we need to pass to RgbImage in order
for it to write the pixels to the display. The only trick here is that the
display on the booted system can really be any resolution so we need to do
some capping to ensure that we don t write more pixels than the display can
handle. Writing fewer than the display s maximum seems fine, though.
fnpraise() -> Result
let gop_handle = boot::get_handle_for_protocol::<GraphicsOutput>()?;
letmut gop = boot::open_protocol_exclusive::<GraphicsOutput>(gop_handle)?;
// Get the (width, height) that is the minimum of
// our image and the display we're using.
let (width, height) = gop.current_mode_info().resolution();
let (width, height) = (width.min(KIER_WIDTH), height.min(KIER_HEIGHT));
letmut buffer = RgbImage::new(width, height);
for y in0..height
for x in0..width
let idx_r = ((y * KIER_WIDTH) + x) * KIER_PIXEL_SIZE;
let pixel =&mut buffer[(x, y)];
pixel.red = KIER[idx_r];
pixel.green = KIER[idx_r +1];
pixel.blue = KIER[idx_r +2];
buffer.write(&mut gop)?;
Ok(())
Not so bad! A bit tedious we could solve some of this by turning
KIER into an RgbImage at compile-time using some clever Cow and
const tricks and implement blitting a sub-image of the image but this
will do for now. This is a joke, after all, let s not go nuts. All that s
left with our code is for us to write our main function and try and boot
the thing!
#[entry]fnmain() -> Status
uefi::helpers::init().unwrap();
praise().unwrap();
boot::stall(100_000_000);
Status::SUCCESS
If you re following along at home and so interested, the final source is over at
gist.github.com.
We can go ahead and build it using cargo (as is our tradition) by targeting
the UEFI platform.
Testing the UEFI Blob
While I can definitely get my machine to boot these blobs to test, I figured
I d save myself some time by using QEMU to test without a full boot.
If you ve not done this sort of thing before, we ll need two packages,
qemu and ovmf. It s a bit different than most invocations of qemu you
may see out there so I figured it d be worth writing this down, too.
$ doas apt install qemu-system-x86 ovmf
qemu has a nice feature where it ll create us an EFI partition as a drive and
attach it to the VM off a local directory so let s construct an EFI
partition file structure, and drop our binary into the conventional location.
If you haven t done this before, and are only interested in running this in a
VM, don t worry too much about it, a lot of it is convention and this layout
should work for you.
With all this in place, we can kick off qemu, booting it in UEFI mode using
the ovmf firmware, attaching our EFI partition directory as a drive to
our VM to boot off of.
If all goes well, soon you ll be met with the all knowing gaze of
Chosen One, Kier Eagan. The thing that really impressed me about all
this is this program worked first try it all went so boringly
normal. Truly, kudos to the uefi crate maintainers, it s incredibly
well done.
Booting a live system
Sure, we could stop here, but anyone can open up an app window and see a
picture of Kier Eagan, so I knew I needed to finish the job and boot a real
machine up with this. In order to do that, we need to format a USB stick.
BE SURE /dev/sda IS CORRECT IF YOU RE COPY AND PASTING. All my drives
are NVMe, so BE CAREFUL if you use SATA, it may very well be your
hard drive! Please do not destroy your computer over this.
$ doas fdisk /dev/sda
Welcome to fdisk (util-linux 2.40.4).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-4014079, default 2048):
Last sector, +/-sectors or +/-size K,M,G,T,P (2048-4014079, default 4014079):
Created a new partition 1 of type 'Linux' and of size 1.9 GiB.
Command (m for help): t
Selected partition 1
Hex code or alias (type L to list all): ef
Changed type of partition 'Linux' to 'EFI (FAT-12/16/32)'.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
Once that looks good (depending on your flavor of udev you may or
may not need to unplug and replug your USB stick), we can go ahead
and format our new EFI partition (BE CAREFUL THAT /dev/sda IS YOUR
USB STICK) and write our EFI directory to it.
$ doas mkfs.fat /dev/sda1
$ doas mount /dev/sda1 /mnt
$ cp -r esp/efi /mnt
$ find /mnt
/mnt
/mnt/efi
/mnt/efi/boot
/mnt/efi/boot/bootx64.efi
Of course, naturally, devotion to Kier shouldn t mean backdooring your system.
Disabling Secure Boot runs counter to the Core Principals, such as Probity, and
not doing this would surely run counter to Verve, Wit and Vision. This bit does
require that you ve taken the step to enroll a
MOK and know how
to use it, right about now is when we can use sbsign to sign our UEFI binary
we want to boot from to continue enforcing Secure Boot. The details for how
this command should be run specifically is likely something you ll need to work
out depending on how you ve decided to manage your MOK.
I figured I d leave a signed copy of boot2kier at
/boot/efi/EFI/BOOT/KIER.efi on my Dell XPS 13, with Secure Boot enabled
and enforcing, just took a matter of going into my BIOS to add the right
boot option, which was no sweat. I m sure there is a way to do it using
efibootmgr, but I wasn t smart enough to do that quickly. I let er rip,
and it booted up and worked great!
It was a bit hard to get a video of my laptop, though but lucky for me, I
have a Minisforum Z83-F sitting around (which, until a few weeks ago was running
the annual http server to control my christmas tree
) so I grabbed it out of the christmas bin, wired it up to a video capture
card I have sitting around, and figured I d grab a video of me booting a
physical device off the boot2kier USB stick.
Attentive readers will notice the image of Kier is smaller then the qemu booted
system which just means our real machine has a larger GOP display
resolution than qemu, which makes sense! We could write some fancy resize code
(sounds annoying), center the image (can t be assed but should be the easy way
out here) or resize the original image (pretty hardware specific workaround).
Additionally, you can make out the image being written to the display before us
(the Minisforum logo) behind Kier, which is really cool stuff. If we were real
fancy we could write blank pixels to the display before blitting Kier, but,
again, I don t think I care to do that much work.
But now I must away
If I wanted to keep this joke going, I d likely try and find a copy of the
original
video when Helly 100%s her file
and boot into that or maybe play a terrible midi PC speaker rendition of
Kier, Chosen One, Kier after
rendering the image. I, unfortunately, don t have any friends involved with
production (yet?), so I reckon all that s out for now. I ll likely stop playing
with this the joke was done and I m only writing this post because of how
great everything was along the way.
All in all, this reminds me so much of building a homebrew kernel to boot a
system into but like, good, though, and it s a nice reminder of both how
fun this stuff can be, and how far we ve come. UEFI protocols are light-years
better than how we did it in the dark ages, and the tooling for this is SO
much more mature. Booting a custom UEFI binary is miles ahead of trying to
boot your own kernel, and I can t believe how good the uefi crate is
specifically.
Praise Kier! Kudos, to everyone involved in making this so delightful .
TL;DR
It's difficult to find the right Debian image. We have thousands of
ISO files and cloud images and we support multiple CPU architectures
and several download methods. The directory structure of our main image server
is like a maze, and our web pages for downloading are also confusing.
Most important facts from this blog post
The Debian maze
Did you ever searched for a specific Debian image which was not the
default netinst ISO for amd64? How long did it take to find it?
Debian is very good at hiding their images for downloading by
offering a huge amount of different versions and variants of images
and multiple methods how to download them. Debian also has multiple
web pages for
This is the secret Debian maze of images. It's currently filled with 8700+ different ISO images
and another 34.000+ files (raw and qcow2) for the cloud images.
The main URL for the server hosting all Debian images is https://cdimage.debian.org/cdimage/
There, you will find installer images, live images, cloud images.
Let's try to find the right image you need
We have three different types of images:
Installer images can be booted on a computer without any OS and then
the Debian installer can be started to perform a Debian installation
Live images boot a Debian desktop without installing anything to
the local disks. You can give Debian a try and if you like it you
can use the Calamers graphical installer for installing the same
desktop onto the local disk.
Cloud images are meant for running a virtual machine with Debian
using QEMU, KVM, OpenStack or in the Amazon AWS cloud or Microsoft
Azure cloud.
Images for the stable release
Almost always, you are probably looking for the image to install the
latest stable release. The URL https://cdimage.debian.org/cdimage/release/ shows:
12.9.0
12.9.0-live
current
current-live
but you cannot see that two are symlinks:
current -> 12.9.0/
current-live -> 12.9.0-live/
Here you will find the installer images and live images for the stable
release (currently Debian 12, bookworm).
If you choose https://cdimage.debian.org/cdimage/release/12.9.0/
you will see a list of CPU architectures:
(BTW source and trace are no CPU architectures)
The typical end user will not care about most architectures, because your
computer will actually always need images from the amd64 folder.
Maybe you have heard that your computer has a 64bit CPU and even if you
have an Intel processor we call this architecture amd64.
Let's see what's in the folder amd64:
Wow. This is confusing and there's no description what all those
folders mean.
bt = BitTorrent, a peer-to-peer file sharing protocol
iso = directories containing ISO files
jigdo = a very special download option only for experts who know they really want this
list = contains lists of the names of the .deb files which are included on the images
The first three are different methods how to download an image. Use
iso when a single network connection will be fast enough for
you. Using bt can result in a faster download, because it
downloads via a peer-to-peer file sharing protocol. You need an
additional torrent program for downloading.
Then we have these variants:
bd = Blu-ray disc (size up to 8GB)
cd = CD image (size up to 700MB)
dvd = DVD images (size up to 4.7GB)
16G = for an USB stick of 16GB or larger
dlbd = dual layer Blu-ray disc
16G and dlbd images are only available via jigdo.
All iso-xx and bt-xx folders provide the same images but with a
different access method.
Here are examples of images:
Fortunately the folder explains in very detail the differences between
these images and what you also find there.
You can ignore the SHA... files if you do not know what they are needed for.
They are not important for you.
These ISO files are small and contain only the core Debian installer
code and a small set of programs. If you install a desktop
environment, the other packages will be downloaded at the end of the installation.
The folders bt-dvd and iso-dvd only contain
debian-12.9.0-amd64-DVD-1.iso or the appropriate torrent file.
In bt-bd and iso-bd you will only find debian-edu-12.9.0-amd64-BD-1.iso.
These large images contain much more Debian packages, so you will not
need a network connection during the installation.
For the other CPU architectures (other than amd64) Debian provides less variants of images but
still a lot. In total, we have 44 ISO files (or torrents) for the current
release of the Debian installer for all architectures. When using
jigdo you can choose between 268 images.
And these are only the installer images for the stable release, no
older or newer version are counted here.
Take a breath before we're diving into.....
The live images
The live images in release/12.9.0-live/amd64/iso-hybrid/ are only available for the
amd64 architecture but for newer Debian releases there will be images also
for arm64.
We have 7 different live images containing one of the most common desktop
environments and one with only a text interface (standard).
The folder name iso-hybrid is the technology that you can use those ISO files for
burning them onto a CD/DVD/BD or writing the same ISO file to a USB stick.
bt-hybrid will give you the torrent files for downloading the
same images using a torrent client program.
More recent installer and live images (aka testing)
For newer version of the images we have currently these folders:
Here you see a new variant call debian-junior, which is a Debian
blend. BitTorrent files are not available for weekly builds.
The daily-builds folder structure is different and only provide the small network
install (netinst) ISOs but several versions of the last
days. Currently we have 55 ISO files available there.
If you like to use the newest installation image fetch this one:
https://cdimage.debian.org/cdimage/daily-builds/sid_d-i/arch-latest/amd64/iso-cd/debian-testing-amd64-netinst.iso
Debian stable with a backports kernel
Unfortunately Debian does not provide any installation media using the
stable release but including a backports kernel for newer hardware. This is
because our installer environment is a very complex mix of special
tools (like anna) and special .udeb versions of packages.
But the FAIme web service of my FAI
project can build a custom installation image using the backports
kernel. Choose a desktop environment, a language and add some packages
names if you like.
Then select Debian 12 bookworm and then enable backports
repository including newer kernel. After a short time you can
download your own installation image.
Older releases
Usually you should not use older releases for a new installation.
In our archive the folder
https://cdimage.debian.org/cdimage/archive/ contains 6163 ISO
files starting from Debian 3.0 (first release was in 2002) and including every point release.
The full DVD image for the oldstable release (Debian 11.11.0 including
non-free firmware) is here
https://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-firmware/archive/latest-oldstable/amd64/iso-dvd/firmware-11.11.0-amd64-DVD-1.iso
the smaller netinst image is
https://cdimage.debian.org/cdimage/archive/11.10.0/amd64/iso-cd/debian-11.10.0-amd64-netinst.iso
The oldest ISO I could find is from 1999 using kernel 2.0.36
I still didn't managed to boot it in KVM.
UPDATE
I got a kernel panic because the VM had 4GB RAM. Reducing this to
500MB RAM (also 8MB works) started the installer of Debian 2.1
without any problems.
Anything else?
In this post, we still did not cover the ports folder (the non official
supported (older) hardware architectures) which contains around 760 ISO files
and the unofficial folder (1445 ISO files) which also provided the ISOs which included the
non-free firmware blobs in the past.
Then, there are more than 34.000 cloud images. But hey, no ISO
files are involved there. This may be part of a complete new posting.
I've just passed my 10th anniversary of starting at
Red Hat! As a personal milestone, this is the longest
I've stayed in a job: I managed 10 years at Newcastle University,
although not in one continuous role.
I haven't exactly worked in one continuous role at Red Hat either, but it
feels like what I do Today is a logical evolution from what I started doing,
whereas in Newcastle I jumped around a bit.
I've seen some changes: in my time here, we changed the logo from Shadow Man;
we transitioned to using Google Workspace for lots of stuff, instead of
in-house IT; we got bought by IBM; we changed President and CEO, twice. And
millions of smaller things.
I won't reach an 11th: my Organisation in Red Hat is moving to
IBM. I think
this is sad news for Red Hat: they're losing some great people. But I'm
optimistic for the future of my Organisation.
Browser Choice and Security Support
Google seems to be more into tracking web users and generally becoming hostile to users [1]. So using a browser other than Chrome seems like a good idea. The problem is the lack of browsers with security support. It seems that the only browser engines with the quality of security support we expect in Debian are Firefox and the Chrome engine. The Chrome engine is used in Chrome, Chromium, and Microsoft Edge. Edge of course isn t an option and Chromium still has some of the Google anti-features built in.
Firefox
So I tried to use Firefox for the things I do. One feature of Chrome based browsers that I really like is the ability to set a custom page for the new tab. This feature was removed because it was apparently being constantly attacked by malware [2]. There are addons to allow that but I prefer to have a minimal number of addons and not have any that are just to replace deliberately broken settings in the browser. Also those addons can t set a file for the URL, so I could set a web server for it but it s annoying to have to setup a web server to work around a browser limitation.
Another thing that annoyed me was YouTube videos open in new tabs not starting to play when I change to the tab. There s a Firefox setting for allowing web sites to autoplay but there doesn t seem to be a way to add sites to the list.
Firefox is getting vertical tabs which is a really nice feature for wide displays [3].
Firefox has a Mozilla service for syncing passwords etc. It is possible to run your own server for this, but the server is written in Rust which is difficult to package and run [4]. There are Docker images for it but I prefer to avoid Docker, generally I think that Docker is a sign of failure in software development. If you can t develop software that can be deployed without Docker then you aren t developing it well.
Chromium
The Ungoogled Chromium project has a lot to offer for safer web browsing [5]. But the changes are invasive and it s not included in Debian. Some of the changes like replacing many Google web domains in the source code with non-existent alternatives ending in qjz9zk are things that could be considered controversial. It definitely isn t a candidate to replace the current Chromium package in Debian but might be a possibility to have as an extra browser.
What Next?
The Falcon browser that is part of the KDE project looks good, but QtWebEngine doesn t have security support in Debian. Would it be possible to provide security support for it?
Ungoogled Chromium is available in Flatpak, so I ll test that out. But ideally it would be packaged for Debian. I ll try building a package of it and see how that goes.
The Iridium Browser is another option [6], it seems similar in design to Ungoogled-Chromium but by different people.
Python 3.13 is now the default Python 3 version in Debian, by Stefano Rivera and Colin Watson
The Python 3.13 as default transition has now completed. The next step is to
remove Python 3.12 from the archive, which should be very straightforward, it
just requires rebuilding C extension packages in no particular order. Stefano
fixed some miscellaneous bugs blocking the completion of the 3.13 as default
transition.
Fixing qtpaths6 for cross compilation, by Helmut Grohne
While Qt5 used to use qmake to query installation properties, Qt6 is moving
more and more to CMake and to ease that transition it relies on more qtpaths.
Since this tool is not naturally aware of the architecture it is called for, it
tends to produce results for the build architecture. Therefore, more than 100
packages were picking up a multiarch directory for the build architecture during
cross builds. In collaboration with the Qt/KDE team and Sandro Knau in
particular (none affiliated with Freexian), we added an architecture-specific
wrapper script in the same way qmake has one for Qt5 and Qt6 already. The
relevant CMake module has been updated to prefer the triplet-prefixed wrapper.
As a result, most of the KDE packages now cross build on unstable ready in time
for the trixie release.
/usr-move, by Helmut Grohne
In December, Emil S dergren reported that a live-build was not working for him
and in January, Colin Watson reported that the proposed mitigation for
debian-installer-utils would practically fail. Both failures were to be
attributed to a wrong understanding of implementation-defined behavior in
dpkg-divert. As a
result, all M18 mitigations had to be reviewed and many of them replaced. Many
have been uploaded already and all instances have received updated patches.
Even though dumat has been in
operation for more than a year, it gained recent changes. For one thing,
analysis of architectures other than amd64 was requested. Chris Hofstaedler
(not affiliated with Freexian) kindly provided computing resources for
repeatedly running it on the larger set. Doing so revealed various
cross-architecture undeclared file conflicts in gcc, glibc, and
binutils-z80, but it also revealed a previously unknown /usr-move issue in
rpi.rpi-common. On top of that, dumat produced false positive diagnostics
and wrongly associated Debian bugs in some cases, both of which have now been
fixed. As a result, a supposedly fixed python3-sepolicy issue had to be
reopened.
rebootstrap, by Helmut Grohne
As much as we think of our base system as stable, it is changing a lot and the
architecture cross bootstrap tooling is very sensitive to such changes requiring
permanent maintenance. A problem that recently surfaced was that building a
binutils cross toolchain would result in a binutils-for-host package that
would not be practically installable as it would depend on a binutils-common
package that was not built. This turned into an examination of binutils-common
and noticing that it actually differed across architectures even though it
should not. Johannes Schauer Marin Rodrigues (not affiliated with Freexian) and
Colin Watson kindly helped brainstorm possible solutions. Eventually, Helmut
provided a patch to move gprofng bits out of
binutils-common. Independently, Matthias Klose
(not affiliated with Freexian) split out binutils-gold into a separate source
package. As a result, binutils-common is now equal across architectures and
can be marked Multi-Arch: foreign resolving the initial problem.
Salsa CI, by Santiago Ruano Rinc n
Santiago continued the work about the sbuild support for Salsa CI, that was
mentioned in the previous month
report. The
!568
merge request that created the new build image was merged, making it easier to
test !569
with external projects. Santiago used a fork of the debusine repo to try the
draft !569,
and some issues were spotted, and part of them fixed. This is the last debusine
pipeline run with the current
!569:
https://salsa.debian.org/santiago/debusine/-/pipelines/794233.
One of the last improvements relates to how to enable projects to customize the
pipeline, in an equivalent way than they currently do in the extract-source
and build jobs. While this is work-in-progress, the results are rather
promising. Next steps include deciding on introducing schroot support for
bookworm, bookworm-security, and older releases, as they are done in the
official debian buildd.
DebConf preparations, by Stefano Rivera and Santiago Ruano Rinc n
DebConf will be happening in Brest, France, in July. Santiago continued the
DebConf 25 organization work, looking for catering providers.
Both Stefano and Santiago have been reaching out to some potential sponsors.
DebConf depends on sponsors to cover the organization cost, if your company
depends on Debian, please consider sponsoring
DebConf.
Stefano has been winding up some of the finances from previous DebConfs.
Finalizing reimbursements to team members from DebConf 23, and handling some
outstanding issues from DebConf 24. Stefano and the rest of the DebConf
committee have been reviewing bids for DebConf 26, to select the next venue.
Ruby 3.3 is now the default Ruby interpreter, by Lucas Kanashiro
Ruby 3.3 is about to become the default Ruby interpreter for Trixie. Many bugs
were fixed by Lucas and the Debian Ruby team during the sprint hold in Paris
during Jan 27-31. The
next step is to remove support of Ruby 3.1, which is the alternative Ruby
interpreter for now. Thanks to the Debian Release team for all the support,
especially Emilio Pozuelo Monfort.
Rails 7 transition, by Lucas Kanashiro
Rails 6 has been shipped by Debian since Bullseye, and as a WEB framework, many
issues (especially security related issues) have been encountered and the
maintainability of it becomes harder and harder. With that in mind, during the
Debian Ruby team sprint last
month, the transition to
Rack 3 (an important dependency of rails containing many breaking changes) was
started in Debian unstable, it is ongoing. Once it is done, the Rails 7
transition will take place, and Rails 7 should be shipped in Debian Trixie.
Miscellaneous contributions
Stefano improved a poor ImportError for users of the turtle module on Python
3, who haven t installed the python3-tk package.
Stefano updated several packages to new upstream releases.
Stefano added the Python extension to the re2 package, allowing for the use
of the Google RE2 regular expression library as a direct replacement for the
standard library re module.
Stefano started provisioning a new physical server for the
debian.social infrastructure.
Carles improved simplemonitor (documentation on systemd integration, worked
with upstream for fixing a bug).
Carles upgraded packages to new upstream versions: python-ring-doorbell and
python-asyncclick.
Carles did po-debconf translations to Catalan: reviewed 44 packages and
submitted translations to 90 packages (via salsa merge requests or bugtracker
bugs).
Carles maintained po-debconf-manager with small fixes.
Rapha l worked on some outstanding
DEP-14merge
request and
participated in the associated discussion. The discussions have been more
contentious than anticipated, somewhat exacerbated by Otto s desire to
conclude fast while the required tool support is not yet there.
Rapha l, with the help of Philipp Kern from the DSA team, upgraded
tracker.debian.org to use Django 4.2 (from bookworm-backports) which in turn
enabled him to configure authentication via salsa.debian.org. It s now
possible to login to tracker.debian.org with your salsa credentials!
Rapha l updated zim a nice desktop wiki that is very handy to organize
your day-to-day digital life to the latest upstream version (0.76).
It is a while since I posted a summary of the free software and
open culture activities and projects I have worked on. Here is a
quick summary of the major ones from last year.
I guess the biggest project of the year has been migrating orphaned
packages in Debian without a version control system to have a git
repository on salsa.debian.org. When I started in April around 450
the orphaned packages needed git. I've since migrated around 250 of
the packages to a salsa git repository, and around 40 packages were
left when I took a break. Not sure who did the around 160 conversions
I was not involved in, but I am very glad I got some help on the
project. I stopped partly because some of the remaining packages
needed more disk space to build than I have available on my
development machine, and partly because some had a strange build setup
I could not figure out. I had a time budget of 20 minutes per
package, if the package proved problematic and likely to take longer,
I moved to another package. Might continue later, if I manage to free
up some disk space.
Another rather big project was the translation to Norwegian Bokm l
and publishing of the first book ever published by a S mi woman, the
M ter
vi liv eller d d? book by Elsa Laula, with a PD0 and CC-BY
license. I released it during the summer, and to my surprise it has
already sold several copies. As I suck at marketing, I did not expect
to sell any.
A smaller, but more long term project (for more than 10 years now),
and related to orphaned packages in Debian, is my project to ensure a
simple way to install hardware related packages in Debian when the
relevant hardware is present in a machine. It made a fairly big
advance forward last year, partly because I have been poking and
begging package maintainers and upstream developers to include
AppStream metadata XML in their packages. I've also released a few
new versions of the isenkram system with some robustness improvements.
Today 127 packages in Debian provide such information, allowing
isenkram-lookup to propose them. Will keep pushing until the
around 35 package names currently hard coded in the isenkram package
are down to zero, so only information provided by individual packages
are used for this feature.
As part of the work on AppStream, I have sponsored several packages
into Debian where the maintainer wanted to fix the issue but lacked
direct upload rights. I've also sponsored a few other packages, when
approached by the maintainer.
I would also like to mention two hardware related packages in
particular where I have been involved, the megactl and mfi-util
packages. Both work with the hardware RAID systems in several Dell
PowerEdge servers, and the first one is already available in Debian
(and of course, proposed by isenkram when used on the appropriate Dell
server), the other is waiting for NEW processing since this autumn. I
manage several such Dell servers and would like the tools needed to
monitor and configure these RAID controllers to be available from
within Debian out of the box.
Vaguely related to hardware support in Debian, I have also been
trying to find ways to help out the Debian ROCm team, to improve the
support in Debian for my artificial idiocy (AI) compute node. So far
only uploaded one package, helped test the initial packaging of
llama.cpp and tried to figure out how to get good speech recognition
like Whisper into Debian.
I am still involved in the LinuxCNC project, and organised a
developer gathering in Norway last summer. A new one is planned the
summer of 2025. I've also helped evaluate patches and uploaded new
versions of LinuxCNC into Debian.
After a 10 years long break, we managed to get a new and improved
upstream version of lsdvd released just before Christmas. As
I use it regularly to maintain my DVD archive, I was very happy to
finally get out a version supporting DVDDiscID useful for uniquely
identifying DVDs. I am dreaming of a Internet service mapping DVD IDs
to IMDB movie IDs, to make life as a DVD collector easier.
My involvement in Norwegian archive standardisation and the free
software implementation of the vendor neutral Noark 5 API continued
for the entire year. I've been pushing patches into both the API and
the test code for the API, participated in several editorial meetings
regarding the Noark 5 Tjenestegrensesnitt specification, submitted
several proposals for improvements for the same. We also organised a
small seminar for Noark 5 interested people, and is organising a new
seminar in a month.
Part of the year was spent working on and coordinating a Norwegian
Bokm l translation of the marvellous children's book
Ada and
Zangemann , which focus on the right to repair and control your own
property, and the value of controlling the software on the devices you
own. The translation is mostly complete, and is now waiting for a
transformation of the project and manuscript to use Docbook XML
instead of a home made semi-text based format. Great progress is
being made and the new book build process is almost complete.
I have also been looking at how to companies in Norway can use free
software to report their accounting summaries to the Norwegian
government. Several new regulations make it very hard for companies
to do use free software for accounting, and I would like to change
this. Found a few drafts for opening up the reporting process, and
have read up on some of the specifications, but nothing much is
working yet.
These were just the top of the iceberg, but I guess this blog post
is long enough now. If you would like to help with any of these
projects, please get in touch, either directly on the project mailing
lists and forums, or with me via email, IRC or Signal. :)
As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
The Scavenger Door is a science fiction adventure and the third
book of the Finder Chronicles. While each of the books of this series
stand alone reasonably well, I would still read the series in order. Each
book has some spoilers for the previous book.
Fergus is back on Earth following the events of Driving the Deep, at loose ends and annoying his relatives. To
get him out of their hair, his cousin sends him into the Scottish hills to
find a friend's missing flock of sheep. Fergus finds things
professionally, but usually not livestock. It's an easy enough job,
though; the lead sheep was wearing a tracker and he just has to get close
enough to pick it up. The unexpected twist is also finding a metal
fragment buried in a hillside that has some strange resonance with the
unwanted gift that Fergus got in Finder.
Fergus's alien friend Ignatio is so alarmed by the metal fragment that he
turns up in person in Fergus's cousin's bar in Scotland. Before he
arrives, Fergus gets a mysteriously infuriating warning visit from alien
acquaintances he does not consider friends. He has, as usual, stepped into
something dangerous and complicated, and now somehow it's become his
problem.
So, first, we get lots of Ignatio, who is an enthusiastic large ball of
green fuzz with five limbs who mostly speaks English but does so from an
odd angle. This makes me happy because I love Ignatio and his tendency to
take things just a bit too literally.
SANTO'S, the sign read. Under it, in smaller letters, was CURIOSITIES
AND INCONVENIENCES FOR COMMENDABLE SUMS.
"Inconveniences sound just like my thing," Fergus said. "You
two want to wait in the car while I check it out?"
"Oh, no, I am not missing this," Isla said, and got out of the podcar.
"I am uncertain," Ignatio said. "I would like some curiouses, but not
any inconveniences. Please proceed while I decide, and if there is
also murdering or calamity or raisins, you will yell right away, yes?"
Also, if your story setup requires a partly-understood alien artifact that
the protagonist can get some explanations for but not have the mystery
neatly solved for them, Ignatio's explanations are perfect.
"It is a door. A doorbell. A... peephole? A key. A control light. A
signal. A stop-and-go sign. A road. A bridge. A beacon. A call. A map.
A channel. A way," Ignatio said. "It is a problem to explain. To say a
doorkey is best, and also wrong. If put together, a path may be
opened."
"And then?"
"And then the bad things on the other side, who we were trying to lock
away, will be free to travel through."
Second, the thing about Palmer's writing that continues to impress me is
her ability to take a standard science fiction plot, one whose variations
I've read probably dozens of times before, and still make it utterly
engrossing. This book is literally a fetch quest. There are a bunch of
scattered fragments, Fergus has to find them and keep them from being
assembled, various other people are after the same fragments, and Fergus
either has to get there first or get the fragments back from them. If you
haven't read this book before, you've played the video game or watched the
movie. The threat is basically a Stargate SG-1 plot. And yet, this
was so much fun.
The characters are great. This book leans less on found family than the
last one and a bit more on actual family. When I started reading this
series, Fergus felt a bit bland in the way that adventure protagonists
sometimes can, but he's fleshed out nicely as the series goes along. He's
not someone who tends to indulge in big emotions, but now the reader can
tell that's because he's the kind of person who finds things to do in
order to keep from dwelling on things he doesn't want to think about. He's
unflappable in a quietly competent way while still having a backstory and
emotional baggage and a rich inner life that the reader sees in glancing
fragments.
We get more of Fergus's backstory, particularly around Mars, but I like
that it's told in anecdotes and small pieces. The last thing Fergus wants
to do is wallow in his past trauma, so he doesn't and finds something to
do instead. There's just enough detail around the edges to deepen his
character without turning the book into a story about Fergus's emotions
and childhood. It's a tricky balancing act that Palmer handles well.
There are also more sentient ships, and I am so in favor of more sentient
ships.
"When I am adding a new skill, I import diagnostic and environmental
information specific to my platform and topology, segregate the skill
subroutines to a dedicated, protected logical space, run incremental
testing on integration under all projected scenarios and variables,
and then when I am persuaded the code is benevolent, an asset, and
provides the functionality I was seeking, I roll it into my primary
processing units," Whiro said. "You cannot do any of that,
because if I may speak in purely objective terms you may incorrectly
interpret as personal, you are made of squishy, unreliable
goo."
We get the normal pieces of a well-done fetch quest: wildly varying
locations, some great local characters (the US-based trauma surgeons on
vacation in Australia were my favorites), and believable antagonists.
There are two other groups looking for the fragments, and while one of
them is the standard villain in this sort of story, the other is an
apocalyptic cult whose members Fergus mostly feels sorry for and who add
just the right amount of surreality to the story. The more we find out
about them, the more believable they are, and the more they make this
world feel like realistic messy chaos instead of the obvious (and boring)
good versus evil patterns that a lot of adventure plots collapse into.
There are things about this book that I feel like I should be criticizing,
but I just can't. Fetch quests are usually synonymous with lazy plotting,
and yet it worked for me. The way Fergus gets dumped into the middle of
this problem starts out feeling as arbitrary and unmotivated as some video
game fetch quest stories, but by the end of the book it starts to make
sense. The story could arguably be described as episodic and cliched, and
yet I was thoroughly invested. There are a few pacing problems at the very
end, but I was too invested to care that much. This feels like a book
that's better than the sum of its parts.
Most of the story is future-Earth adventure with some heist elements. The
ending goes in a rather different direction but stays at the center of the
classic science fiction genre. The Scavenger Door reaches a
satisfying conclusion, but there are a ton of unanswered questions that
will send me on to the fourth (and reportedly final) novel in the series
shortly.
This is great stuff. It's not going to win literary awards, but if you're
in the mood for some classic science fiction with fun aliens and neat
ideas, but also benefiting from the massive improvements in
characterization the genre has seen in the past forty years, this series
is perfect. Highly recommended.
Followed by Ghostdrift.
Rating: 9 out of 10
20 years ago, I got my Debian Developer account. I was 18 at the time, it was Shrove Tuesday and - as is customary - I was drunk when I got the email. There was so much that I did not know - which is also why the process took 1.5 years from the time I applied. I mostly only maintained a package or two. I'm still amazed that Christian Perrier and Joerg Jaspert put sufficient trust in me at that time. Nevertheless now feels like a good time for a personal reflection of my involvement in Debian. During my studies I took on more things. In January 2008 I joined the Release
Team as an assistant, which taught me a lot of code review. I have been an Application Manager on the side. Going to my first Debconf was really a turning point. My first one was Mar del Plata in Argentina in August 2008, when I was 21. That was quite an excitement, traveling that far from Germany for the first time. The personal connections I made there made quite the difference. It was also a big boost for motivation. I attended 8 (Argentina), 9 (Spain), 10 (New York), 11 (Bosnia and Herzegovina), 12 (Nicaragua), 13 (Switzerland), 14 (Portland), 15 (Germany), 16 (South Africa), and hopefully I'll make it to this year's in Brest. At all of them I did not see much of the countries as I prioritized all of my time focused on Debian, even skipping some of the day trips in favor of team meetings. Yet I am very grateful to the project (and to my employer) for shipping me there.I ended up as Stable Release
Manager for a while, from August 2008 - when Martin Zobel-Helas moved
into DSA - until I got dropped in March 2020. I think my biggest achievements were pushing for the
creation of -updates in favor of a separate volatile archive and a
change of the update policy to allow for more common sense updates in
the main archive vs. the very strict "breakage or security" policy we
had previously. I definitely need to call out Adam D. Barratt for being the partner in crime, holding up the fort for even longer.In 2009 I got too annoyed at the existing wanna-build team not being responsive anymore and pushed for the system to be given to a new team. I did not build it and significant contributions were done by other people (like Andreas Barth and Joachim Breitner, and later Aurelien Jarno). I mostly reworked the way the system was triggered, investigated when it broke and was around when people wanted things merged. In the meantime I worked sys/netadmin jobs while at university, both paid and as a volunteer with the students' council. For a year or two I was the administrator of a System z mainframe IBM donated to my university. We had a mainframe course and I attended two related conferences. That's where my s390(x) interest came from, although credit for the port needs to go to Aurelien Jarno. Since completing university in 2013 I have been working for a company for almost 12 years. Debian experience was very relevant to the job and I went on maintaining a Linux distro or two at work - before venturing off into security hardening. People in megacorps - in my humble opinion - disappear from the volunteer projects because a) they might previously have been studying and thus had a lot more time on their hands and b) the job is too similar to the volunteer work and thus the same brain cells used for work are exhausted and can't be easily reused for volunteer work. I kept maintaining a couple of things (buildds, some packages) - mostly because of a sense of commitment and responsibility, but otherwise kind of scaled down my involvement. I also felt less connected as I dropped off IRC.Last year I finally made it to Debian events again: MiniDebconf in Berlin, where we discussed the aftermath of the xz incident, and the Debian BSP in Salzburg. I rejoined IRC using the Matrix bridge. That also rekindled my involvement, with me guiding a new DD through NM and ending up in DSA. To be honest, only in the last two or three years I felt like a (more) mature old-timer. I have a new gig at work lined up to start soon and next to that I have sysadmining for Debian. It is pretty motivating to me that I can just get things done - something that is much harder to achieve at work due to organizational complexities. It balances out some frustration I'd otherwise have. The work is different enough to be enjoyable and the people I work with are great.
The future I still think the work we do in Debian is important, as much as I see a lack of appreciation in a world full of containers. We are reaping most of the benefits of standing on the shoulders of giants and of great decisions made in the past (e.g. the excellent Debian policy, but also the organizational model) that made Debian what it is today.Given the increase in size and complexity of what Debian ships - and the somewhat dwindling resource of developer time, it would benefit us to have better processes for large-scale changes across all packages. I greatly respect the horizontal effects that are currently being driven and that suck up a lot of energy.A lot of our infrastructure is also aging and not super well maintained. Many take it for granted that the services we have keep existing, but most are only maintained by a person or two, if even. Software stacks are aging and it is even a struggle to have all necessary packages in the next release.Hopefully I can contribute a bit or two to these efforts in the future.
It s been quite the week of radio related nonsense for me, where I ve been
channelling my time and brainspace for radio into activity on air and system
refinements, not working on Debian.
POTA, Antennas and why do my toys not work?
Having had my interest piqued by
Ian at mastodon.radio, I
looked online and spotted a couple of parks within stumbling distance of my
house, that s good news! It looks like the list has been refactored and
expanded
since I last looked at it, so there are now more entities to activate and
explore.
My concerns about antennas noted last
week rumbled on. There was a
second strand to this concern too, my end fed 64:1 (or 49:1?!) transformer from
MM0OPX sits in my mind as not having worked very well in
Spain last year, and I want to get to the bottom of why. As with most things in
my life, it s probably a me problem.
I came up with a cunning plan - firstly, buy a new
mast to
replace the one I broke a few weeks back on Cat
Law.
Secondly, buy a couple of new connectors and some heatshrink to reterminate my
cable that I m sure is broken.
Spending more money on a problem never hurt anyone, right?
Come Wednesday, the new toys arrived and I figured combining everything into one
convenient night time walk and radio was a good plan.
So I walk out to the nearest park with my LoRa APRS
doofer going and see what
happens:
After circling a bit to find somewhere suitable (there appear to be construction
works in the park!) I set up my gear in 2C with frost on the ground, called CQ,
spotted and got nothing on either the end fed half wave or the cheap vertical.
As it was too late for 20m, I tried 40 and a bit of 80 using the inbuilt tuner,
but wasn t heard by stations I called or when calling independently.
I packed everything up and lora-doofered my way home, mildly deflated.
Try it at home
It still didn t sit with me that the end fed wasn t working, so come Friday night I
set it up in the back garden/woods behind the house to try and diagnose why it
wasn t working.
Up it went, I worked some Irish stations pretty effortlessly, and down
everything came. No complaints - the only things I did differently was have the
feedpoint a little higher and check my power, limiting it to 10W. The G90 can do
20W, I wonder if running at that was saturating the core in the 64:1.
At some point in the evening I stepped in some dog s shit too, and spent some
time cleaning my boots outside to avoid further tramping the smell through the
house.
Win some, lose some.
Take it to the Hills
On Friday, some of the other GM-ES Sota-ists had been out for an activity
day.
On account of me being busy in work, I couldn t go outside to play, but I
figured a weekend of activity was on the books.
Saturday - A day above the clouds
On Saturday I took myself up Tap O Noth, a
favourite of mine for some reason, and Lord Arthur s Hill.
Before I hit the hills, I took myself to the
hackerspace and printed myself a K6ARK
Winder and a guy
ring for the mast, cut string, tied
it together and wound the string on to the winder.
I also took time to buzz out my wonky coax and it showed great continuity. Hmm,
that can be continued later. I didn t quite get to crimping the radial network
of the Aliexpress whip with a 12mm stud crimp, that can also be put on the TODO
list.
Tap O Noth
Once finally out, the weather was a bit cloudy with passing snow showers, but in
between the showers I was above the clouds and the air was clear:
After a mild struggle on 2m, I set up the end fed the first hill and got to
work from the old hill fort:
The end fed worked flawlessly. Exactly as promised, switching between 7MHz, 14MHz, 21MHz
and 28MHz without a tuner was perfect, I chased hills on all the bands, and had a
great time. Apart from 40m, where there was absolutely no space due to a
contest. That wasn t such a fun time!
My fingers were bitterly cold, so on went the big gloves for the descent and I
felt like I was warm by the time I made it back to the car.
It worked so well, in fact, I took the 1/4 wave cheap vertical out my bag and
decided to brave it on the next activation.
Lord Arthur s Hill
GM5ALX has posted a .gpx to sotlas which is
shorter than the other ascent, but much sharper - I figured this would be a fun
new way to try up the hill!
It takes you right through the heart of the Littlewood Park estate, and I felt a
bit uncomfortable walking straight past the estate cottages, especially when
there were vehicles moving and active work happening. Presumably this is where
Lord Arthur lived, at the foot of his hill.
I cut through the woods to the west of the cottages, disturbing some deer and
many, many pheasants, but I met the path fairly quickly. From there it was a 2km
walk, 300m vertical ascent. Short and sharp!
At the top, I was treated to a view of the hill I had activated only an hour or
so before, which is a view that always makes me smile:
To get some height for the feedpoint, I wrapped the coax around my winder a
couple of turns and trapped it with the elastic while draping the coax over the
trig. This bought me some more height and I felt clever because of it. Maybe a
pole would be easier?
From here, I worked inter-G on 40m and had a wee pile up, eventually working 15
or so European stations on 20m. Pleased with that!
I had been considering a third hill, but home was the call in the failing light.
Back to the car I walked to find my key didn t have any battery, so out came the
Audi App and I used the Internet of Things to unlock my car. The modern world is
bizarre.
Sunday - Cloudy Head // Head in the Clouds
Sunday started off migraney, so I stayed within the confines of my house until I
felt safe driving! After some back and forth in my cloudy head, I opted for the
easier option of Ladylea Hill as I wasn t
feeling up for major physical exertion.
It was a long drive, after which I felt more wonky, but I hit the path
eventually - I run to Hibby Standard Time, a few hours to a few days behind the
rest of GM/ES. I was ready to bail if my head didn t improve, but it turns out,
fresh cold air, silence and bloodflow helped.
Ladylea Hill was incredibly quiet, a feature I really appreciated. It feels
incredibly remote, with a long winding drive down Glenbuchat, which still has
ice on the surface of the lochs and standing water.
A brooding summit crowned with grey cloud in fantastic scenery that only revealed itself
upon the clouds blowing through:
I set up at the cairn and picked up 30 contacts overall, split between 40m and
20m, with some inter-g on 40 and a couple of continental surprises. 20 had
longer skip today, so I saw Spain, Finland, Slovenia, Poland.
On teardown, I managed to snap the top segment of my brand new mast with my
cold, clumsy fingers, but thankfully sotabeams stock replacements. More money at
the problem, again.
Back to the car, no app needed, and homeward bound as the light faded.
At the end of the weekend, I find myself finally over 100 activator points and
over 400 chaser points. Somehow I ve collected more points this year already
than last year, the winter bonuses really do stack up!
Addendum - OSMAnd & Open Street Map
I ve been using OSMAnd on my iPhone quite extensively
recently, I think offline mapping is super important if you re going out to get
mildly lost in the hills. On more than one occasion, I have confidently set off
in the wrong direction in the mist, and maps have saved my bacon!
As you can download .gpx files, it s great to have them on the device and
available for guidance in case you get lost, coupled with an offline map. Plus,
as I drive around I love to have the dark red of a hill I ve walked appear on
the map in my car dash or in my hand:
This weekend I discovered it s possible to have height maps for nice 3d
maps and contours marked on the map - you just need to download some
additions for the maps. This is a really nice feature, it makes maps more
pretty and more useful when you re in the middle of nowhere.
Open Street Map also has designators for SOTA summits
here
and similar for POTA
here
GM5ALX has set to adding the summits around Scotland
here.
While the benefits aren t immediately obvious, it allows developers of mapping
applications access to more data at no extra cost, really. It helps add depth to
an already rich set of information, and allows us as radio amateurs to do more
interesting things with maps and not be shackled to Apple/Google.
Because it s open data, we can also fix things we find wrong as users. I like to
fix road surfaces after I ve been cycling as that will feed forward to route
planning through Komoot and data on my wahoo too, which can be modified with
osm maps.
In the future, it s possible to have an OSMAnd plugin highlighting local SOTA
summits or mimicking features of sotl.as but offline.
It s cool to be able to put open technologies to use like this in the field and
really is the convergence point of all my favourite things!
Well, 2024 will be remembered, won't it? I guess 2025 already wants to
make its mark too, but let's not worry about that right now, and
instead let's talk about me.
A little over a year ago, I was gloating
over how I had such a great blogging year in 2022, and was considering
2023 to be average, then went on to gather more stats and traffic
analysis... Then I said, and I quote:
I hope to write more next year. I've been thinking about a few posts I
could write for work, about how things work behind the scenes at Tor,
that could be informative for many people. We run a rather old setup,
but things hold up pretty well for what we throw at it, and it's worth
sharing that with the world...
What a load of bollocks.
A bad year for this blog
2024 was the second worst year ever in my blogging history, tied with
2009 at a measly 6 posts for the year:
Loads of drafts
It's not that I have nothing to say: I have no less than five drafts
in my working tree here, not counting three actual drafts recorded
in the Git repository here:
I just don't have time to wrap those things up. I think part of me is
disgusted by seeing my work stolen by large corporations to build
proprietary large language models while my idols have been pushed
to suicide for trying to share science with the world.
Another part of me wants to make those things just right. The
"tagged drafts" above are nothing more than a huge pile of chaotic
links, far from being useful for anyone else than me, and even
then.
The on-dying article, in particular, is becoming my nemesis. I've
been wanting to write that article for over 6 years now, I think. It's
just too hard.
Writing elsewhere
There's also the fact that I write for work already. A lot. Here are
the top-10 contributors to our team's wiki:
anarcat@angela:help.torproject.org$ git shortlog --numbered --summary --group="format:%al" head -10
4272 anarcat
423 jerome
117 zen
116 lelutin
104 peter
58 kez
45 irl
43 hiro
18 gaba
17 groente
... but that's a bit unfair, since I've been there half a
decade. Here's the last year:
anarcat@angela:help.torproject.org$ git shortlog --since=2024-01-01 --numbered --summary --group="format:%al" head -10
827 anarcat
117 zen
116 lelutin
91 jerome
17 groente
10 gaba
8 micah
7 kez
5 jnewsome
4 stephen.swift
So I still write the most commits! But to truly get a sense of the
amount I wrote in there, we should count actual changes. Here it is by
number of lines (from commandlinefu.com):
anarcat@angela:help.torproject.org$ git ls-files xargs -n1 git blame --line-porcelain sed -n 's/^author //p' sort -f uniq -ic sort -nr head -10
99046 Antoine Beaupr
6900 Zen Fu
4784 J r me Charaoui
1446 Gabriel Filion
1146 Jerome Charaoui
837 groente
705 kez
569 Gaba
381 Matt Traudt
237 Stephen Swift
That, of course, is the entire history of the git repo, again. We
should take only the last year into account, and probably ignore the
tails directory, as sneaky Zen Fu imported the entire docs from
another wiki there...
anarcat@angela:help.torproject.org$ find [d-s]* -type f -mtime -365 xargs -n1 git blame --line-porcelain 2>/dev/null sed -n 's/^author //p' sort -f uniq -ic sort -nr head -10
75037 Antoine Beaupr
2932 J r me Charaoui
1442 Gabriel Filion
1400 Zen Fu
929 Jerome Charaoui
837 groente
702 kez
569 Gaba
381 Matt Traudt
237 Stephen Swift
Pretty good! 75k lines. But those are the files that were modified in
the last year. If we go a little more nuts, we find that:
I wrote 126,116 words in that wiki, only in the last year. I also
deleted 37k words, so the final total is more like 89k words, but
still: that's about forty (40!) articles of the average size (~2k) I
wrote in 2022.
(And yes, I did go nuts and write a new log parser, essentially from
scratch, to figure out those word diffs. I did get the courage only
after asking GPT-4o for an example first, I must admit.)
Let's celebrate that again: I wrote 90 thousand words in that wiki
in 2024. According to Wikipedia, a "novella" is 17,500 to 40,000
words, which would mean I wrote about a novella and a novel, in the
past year.
But interestingly, if I look at the repository analytics. I
certainly didn't write that much more in the past year. So that
alone cannot explain the lull in my production here.
Arguments
Another part of me is just tired of the bickering and arguing on the
internet. I have at least two articles in there that I suspect is
going to get me a lot of push-back (NixOS and Fish). I know how to
deal with this: you need to write well, consider the controversy,
spell it out, and defuse things before they happen. But that's hard
work and, frankly, I don't really care that much about what people
think anymore.
I'm not writing here to convince people. I have stop evangelizing a
long time ago. Now, I'm more into documenting, and teaching. And,
while teaching, there's a two-way interaction: when you give out a
speech or workshop, people can ask questions, or respond, and you all
learn something. When you document, you quickly get told "where is
this? I couldn't find it" or "I don't understand this" or "I tried
that and it didn't work" or "wait, really? shouldn't we do X instead",
and you learn.
Here, it's static. It's my little soapbox where I scream in the
void. The only thing people can do is scream back.
Collaboration
So.
Let's see if we can work together here.
If you don't like something I say, disagree, or find something wrong
or to be improved, instead of screaming on social media or ignoring
me, try contributing back. This site here is backed by a git
repository and I promise to read everything you send there,
whether it is an issue or a merge request.
I will, of course, still read comments sent by email or IRC or social
media, but please, be kind.
You can also, of course, follow the latest changes on the TPA
wiki. If you want to catch up with the last year, some of the
"novellas" I wrote include:
TPA-RFC-71: Emergency email deployments, phase B: deploy a new
sender-rewriting mail forwarder, migrate mailing lists off the
legacy server to a new machine, migrate the remaining Schleuder list
to the Tails server, upgrade eugeni.
This is going to be a controversial statement because some people are
absolute nerds about this, but, I need to say it.
Qalculate is the best calculator that has ever been made.
I am not going to try to convince you of this, I just wanted to put
out my bias out there before writing down those notes. I am a total
fan.
This page will collect my notes of cool hacks I do with
Qalculate. Most examples are copy-pasted from the command-line
interface (qalc(1)), but I typically use the graphical interface as
it's slightly better at displaying complex formulas. Discoverability
is obviously also better for the cornucopia of features this fantastic
application ships.
Qalc commandline primer
On Debian, Qalculate's CLI interface can be installed with:
apt install qalc
Then you start it with the qalc command, and end up on a prompt:
anarcat@angela:~$ qalc
>
Then it's a normal calculator:
anarcat@angela:~$ qalc
> 1+1
1 + 1 = 2
> 1/7
1 / 7 0.1429
> pi
pi 3.142
>
There's a bunch of variables to control display, approximation, and so
on:
> set precision 6
> 1/7
1 / 7 0.142857
> set precision 20
> pi
pi 3.1415926535897932385
When I need more, I typically browse around the menus. One big issue I
have with Qalculate is there are a lot of menus and features. I had
to fiddle quite a bit to figure out that set precision command
above. I might add more examples here as I find them.
Bandwidth estimates
I often use the data units to estimate bandwidths. For example, here's
what 1 megabit per second is over a month ("about 300 GiB"):
> 1 megabit/s * 30 day to gibibyte
(1 megabit/second) (30 days) 301.7 GiB
Or, "how long will it take to download X", in this case, 1GiB over a
100 mbps link:
> 1GiB/(100 megabit/s)
(1 gibibyte) / (100 megabits/second) 1 min + 25.90 s
Password entropy
To calculate how much entropy (in bits) a given password structure,
you count the number of possibilities in each entry (say, [a-z] is
26 possibilities, "one word in a 8k dictionary" is 8000), extract the
base-2 logarithm, multiplied by the number of entries.
For example, an alphabetic 14-character password is:
> log2(26*2)*14
log (26 2) 14 79.81
... 80 bits of entropy. To get the equivalent in a Diceware
password with a 8000 word dictionary, you would need:
> log2(8k)*x = 80
(log (8 000) x) = 80
x 6.170
... about 6 words, which gives you:
> log2(8k)*6
log (8 1000) 6 77.79
78 bits of entropy.
Exchange rates
You can convert between currencies!
Other conversions
Here are other neat conversions extracted from my history
> teaspoon to ml
teaspoon = 5 mL
> tablespoon to ml
tablespoon = 15 mL
> 1 cup to ml
1 cup 236.6 mL
> 6 L/100km to mpg
(6 liters) / (100 kilometers) 39.20 mpg
> 100 kph to mph
100 kph 62.14 mph
> (108km - 72km) / 110km/h
((108 kilometers) (72 kilometers)) / (110 kilometers/hour)
19 min + 38.18 s
Completion time estimates
This is a more involved example I often do.
Background
Say you have started a long running copy job and you don't have the
luxury of having a pipe you can insert pv(1) into to get a nice
progress bar. For example, rsync or cp -R can have that problem
(but not tar!).
(Yes, you can use --info=progress2 in rsync, but that estimate is
incremental and therefore inaccurate unless you disable the
incremental mode with --no-inc-recursive, but then you pay a huge
up-front wait cost while the entire directory gets crawled.)
Extracting a process start time
First step is to gather data. Find the process start time. If you were
unfortunate enough to forget to run date --iso-8601=seconds before
starting, you can get a similar timestamp with stat(1) on the
process tree in /proc with:
So our start time is 2025-02-07 15:50:25, we shave off the
nanoseconds there, they're below our precision noise floor.
If you're not dealing with an actual UNIX process, you need to figure
out a start time: this can be a SQL query, a network request,
whatever, exercise for the reader.
Saving a variable
This is optional, but for the sake of demonstration, let's save this
as a variable:
Estimating data size
Next, estimate your data size. That will vary wildly with the job
you're running: this can be anything: number of files, documents being
processed, rows to be destroyed in a database, whatever. In this case,
rsync tells me how many bytes it has transferred so far:
Strip off the weird dots in there, because that will confuse
qalculate, which will count this as:
2.968252503968 bytes 2.968 B
Or, essentially, three bytes. We actually transferred almost 3TB here:
2968252503968 bytes 2.968 TB
So let's use that. If you had the misfortune of making rsync silent,
but were lucky enough to transfer entire partitions, you can use df
(without -h! we want to be more precise here), in my case:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_hdd-srv 7512681384 7258298036 179205040 98% /srv
tank/srv 7667173248 2870444032 4796729216 38% /srv-zfs
(Otherwise, of course, you use du -sh $DIRECTORY.)
Digression over bytes
Those are 1 K bytes which is actually (and rather unfortunately)
Ki, or "kibibytes" (1024 bytes), not "kilobytes" (1000 bytes). Ugh.
> 2870444032 KiB
2870444032 kibibytes 2.939 TB
> 2870444032 kB
2870444032 kilobytes 2.870 TB
At this scale, those details matter quite a bit, we're talking about a
69GB (64GiB) difference here:
Anyways. Let's take 2968252503968 bytes as our current progress.
Our entire dataset is 7258298064 KiB, as seen above.
Solving a cross-multiplication
We have 3 out of four variables for our equation here, so we can
already solve:
> (now-start)/x = (2996538438607 bytes)/(7258298064 KiB) to h
((actual start) / x) = ((2996538438607 bytes) / (7258298064
kibibytes))
x 59.24 h
The entire transfer will take about 60 hours to complete! Note that's
not the time left, that is the total time.
To break this down step by step, we could calculate how long it has
taken so far:
> now-start
now start 23 h + 53 min + 6.762 s
> now-start to s
now start 85987 s
> uptime
uptime = 5 d + 6 h + 34 min + 12.11 s
> golden
golden 1.618
> exact
golden = ( (5) + 1) / 2
Computing dates
In any case, yay! We know the transfer is going to take roughly 60
hours total, and we've already spent around 24h of that, so, we have
36h left.
But I did that all in my head, we can ask more of Qalc yet!
Let's make another variable, for that total estimated time:
> total=(now-start)/x = (2996538438607 bytes)/(7258298064 KiB)
save(((now start) / x) = ((2996538438607 bytes) / (7258298064
kibibytes)); total; Temporary; ; 1)
2 d + 11 h + 14 min + 38.22 s
And we can plug that into another formula with our start time to
figure out when we'll be done!
> start+total
start + total "2025-02-10T03:28:52"
> start+total-now
start + total now 1 d + 11 h + 34 min + 48.52 s
> start+total-now to h
start + total now 35 h + 34 min + 32.01 s
That transfer has ~1d left, or 35h24m32s, and should complete around 4
in the morning on February 10th.
But that's icing on top. I typically only do the
cross-multiplication and calculate the remaining time in my
head.
I mostly did the last bit to show Qalculate could compute dates and
time differences, as long as you use ISO timestamps. Although it can
also convert to and from UNIX timestamps, it cannot parse arbitrary
date strings (yet?).
Other functionality
Qalculate can:
Plot graphs;
Use RPN input;
Do all sorts of algebraic, calculus, matrix, statistics,
trigonometry functions (and more!);
... and so much more!
I have a hard time finding things it cannot do. When I get there, I
typically need to resort to programming code in Python, use a
spreadsheet, and others will turn to more complete engines like
Maple, Mathematica or R.
But for daily use, Qalculate is just fantastic.
And it's pink! Use it!
Hi
Cookiecutter is a tool for building coding project templates. It s often used to provide a scaffolding to build lots of similar project. I ve seen it used to create Symfony projects and several cloud infrastructures deployed with Terraform. This tool was useful to accelerate the creation of new projects.
Since these templates were bound to evolve, the teams providing these template relied on cruft to update the code provided by the template in their user s code. In other words, they wanted their users to apply a diff of the template modification to their code.
At the beginning, all was fine. But problems began to appear during the lifetime of these projects.
What went wrong ?
In both cases, we had the following scenario:
user team:
creates new project with cookiecutter template
makes modification on their code, including on code provided by template
meanwhile, provider team:
makes modifications to cookiecutter template
releases new template version
asks his users to update code brought by template using cruft
user team then:
runs cruft to update template code
discovers a lot of code conflicts (similar to git merge conflicts)
often rolls back cruft update and gives up on template update
User team giving up on updates is a major problem because these update may bring security or compliance fixes.
Note that code conflicts seen with cruft are similar to git merge conflicts, but harder to resolve because, unlike with a git merge, there s no common ancestor, so 3-way merges are not possible.
From an organisation point of view, the main problem is the ambiguous ownership of the functionalities brought by template code: who own this code ? The provider team who writes the template or the user team who owns the repository of the code generated from the template ? Conflicts are bound to happen.
Possible solutions to get out of this tar pit:
Assume that template are one shot. Template update are not practical in the long run.
Make sure that template are as thin as possible. They should contain minimal logic.
Move most if not all logic in separate libraries or scripts that are owned by provider team. This way update coming from provider team can be managed like external dependencies by upgrading the version of a dependency.
Of course your users won t be happy to be faced with a manual migration from the old big template to the new one with external dependencies. On the other hand, this may be easier to sell than updates based on cruft since the painful work will happen once. Further updates will be done by incrementing dependency versions (which can be automated with renovate).
If many projects are to be created with this template, it may be more practical to provide use a CLI that will create a skeleton project. See for instance terragrunt scaffold command.
My name is Dominique Dumont, I m a devops freelance. You can find the devops and audit services I propose on my website or reach out to me on LinkedIn.
All the best
Introduction
If you use SteamOS and you like to install third-party tools or modify the system-wide configuration some of your changes might be lost after an OS update. Read on for details on why this happens and what to do about it.
As you all know SteamOS uses an immutable root filesystem and users are not expected to modify it because all changes are lost after an OS update.
However this does not include configuration files: the /etc directory is not part of the root filesystem itself. Instead, it s a writable overlay and all modifications are actually stored under /var (together with all the usual contents that go in that filesystem such as logs, cached data, etc).
/etc contains important data that is specific to that particular machine like the configuration of known network connections, the password of the main user and the SSH keys. This configuration needs to be kept after an OS update so the system can keep working as expected. However the update process also needs to make sure that other changes to /etc don t conflict with whatever is available in the new version of the OS, and there have been issues due to some modifications unexpectedly persisting after a system update.
SteamOS 3.6 introduced a new mechanism to decide what to to keep after an OS update, and the system now keeps a list of configuration files that are allowed to be kept in the new version. The idea is that only the modifications that are known to be important for the correct operation of the system are applied, and everything else is discarded1.
However, many users want to be able to keep additional configuration files after an OS update, either because the changes are important for them or because those files are needed for some third-party tool that they have installed. Fortunately the system provides a way to do that, and users (or developers of third-party tools) can add a configuration file to /etc/atomic-update.conf.d, listing the additional files that need to be kept.
There is an example in /etc/atomic-update.conf.d/example-additional-keep-list.conf that shows what this configuration looks like.
Sample configuration file for the SteamOS updater
Developers who are targeting SteamOS can also use this same method to make sure that their configuration files survive OS updates. As an example of an actual third-party project that makes use of this mechanism you can have a look at the DeterminateSystems Nix installer:
https://github.com/DeterminateSystems/nix-installer/blob/v0.34.0/src/planner/steam_deck.rs#L273
As usual, if you encounter issues with this or any other part of the system you can check the SteamOS issue tracker. Enjoy!
A copy is actually kept under /etc/previous to give the user the chance to recover files if necessary, and up to five previous snapshots are kept under /var/lib/steamos-atomupd/etc_backup