One of the assumptions baked deeply into US society (and many others) is
that people are largely defined by the work they do, and that work is the
primary focus of life. Even in Marxist analysis, which is otherwise
critical of how work is economically organized, work itself reigns
supreme. This has been part of the feminist critique of both capitalism
and Marxism, namely that both devalue domestic labor that has
traditionally been unpaid, but even that criticism is normally framed as
expanding the definition of work to include more of human activity. A
few exceptions aside, we shy away from
fundamentally rethinking the centrality of work to human experience.
The Problem with Work begins as a critical analysis of that
centrality of work and a history of some less-well-known movements against
it. But, more valuably for me, it becomes a discussion of the types and
merits of utopian thinking, including why convincing other people is not
the only purpose for making a political demand.
The largest problem with this book will be obvious early on: the writing
style ranges from unnecessarily complex to nearly unreadable. Here's an
excerpt from the first chapter:
The lack of interest in representing the daily grind of work routines
in various forms of popular culture is perhaps understandable, as is
the tendency among cultural critics to focus on the animation and
meaningfulness of commodities rather than the eclipse of laboring
activity that Marx identifies as the source of their fetishization
(Marx 1976, 164-65). The preference for a level of abstraction that
tends not to register either the qualitative dimensions or the
hierarchical relations of work can also account for its relative
neglect in the field of mainstream economics. But the lack of
attention to the lived experiences and political textures of work
within political theory would seem to be another matter. Indeed,
political theorists tend to be more interested in our lives as
citizens and noncitizens, legal subjects and bearers of rights,
consumers and spectators, religious devotees and family members, than
in our daily lives as workers.
This is only a quarter of a paragraph, and the entire book is written like
this.
I don't mind the occasional use of longer words for their precise meanings
("qualitative," "hierarchical") and can tolerate the academic habit of
inserting mostly unnecessary citations. I have less patience with the
meandering and complex sentences, excessive hedge words ("perhaps," "seem
to be," "tend to be"), unnecessarily indirect phrasing ("can also account
for" instead of "explains"), or obscure terms that are unnecessary to the
sentence (what is "animation of commodities"?). And please have mercy and
throw a reader some paragraph breaks.
The writing style means substantial unnecessary effort for the reader,
which is why it took me six months to read this book. It stalled all of
my non-work non-fiction reading and I'm not sure it was worth the effort.
That's unfortunate, because there were several important ideas in here
that were new to me.
The first was the overview of the "wages for housework" movement, which I
had not previously heard of. It started from the common feminist position
that traditional "women's work" is undervalued and advocated taking the
next logical step of giving it equality with paid work by making it
paid work. This was not successful, obviously, although the increasing
prevalence of day care and cleaning services has made it partly true
within certain economic classes in an odd and more capitalist way. While
I, like Weeks, am dubious this was the right remedy, the observation
that household work is essential to support capitalist activity but is
unmeasured by GDP and often uncompensated both economically and socially
has only become more accurate since the 1970s.
Weeks argues that the usefulness of this movement should not be judged by
its lack of success in achieving its demands, which leads to the second
interesting point: the role of utopian demands in reframing and expanding
a discussion. I normally judge a political demand on its effectiveness at
convincing others to grant that demand, by which standard many activist
campaigns (such as wages for housework) are unsuccessful. Weeks points
out that making a utopian demand changes the way the person making the
demand perceives the world, and this can have value even if the demand
will never be granted. For example, to demand wages for housework
requires rethinking how work is defined, what activities are compensated
by the economic system, how such wages would be paid, and the implications
for domestic social structures, among other things. That, in turn, helps
in questioning assumptions and understanding more about how existing
society sustains itself.
Similarly, even if a utopian demand is never granted by society at large,
forcing it to be rebutted can produce the same movement in thinking in
others. In order to rebut a demand, one has to take it seriously and
mount a defense of the premises that would allow one to rebut it. That
can open a path to discussing and questioning those premises, which can
have long-term persuasive power apart from the specific utopian demand.
It's a similar concept as the Overton Window, but with more nuance: the
idea isn't solely to move the perceived range of accepted discussion, but
to force society to examine its assumptions and premises well enough to
defend them, or possibly discover they're harder to defend than one might
have thought.
Weeks applies this principle to universal basic income, as a utopian
demand that questions the premise that work should be central to personal
identity. I kept thinking of the Black Lives Matter movement and the
demand to abolish the police, which (at least in popular discussion) is a
more recent example than this book but follows many of the same
principles. The demand itself is unlikely to be met, but to rebut it
requires defending the existence and nature of the police. That in turn
leads to questions about the effectiveness of policing, such as clearance
rates (which are far lower than one might have assumed). Many more
examples came to mind. I've had that experience of discovering problems
with my assumptions I'd never considered when debating others, but had not
previously linked it with the merits of making demands that may be
politically infeasible.
The book closes with an interesting discussion of the types of utopias,
starting from the closed utopia in the style of Thomas More in which the
author sets up an ideal society. Weeks points out that this sort of
utopia tends to collapse with the first impossibility or inconsistency the
reader notices. The next step is utopias that acknowledge their own
limitations and problems, which are more engaging (she cites Le Guin's
The Dispossessed). More conditional
than that is the utopian manifesto, which only addresses part of society.
The least comprehensive and the most open is the utopian demand, such as
wages for housework or universal basic income, which asks for a specific
piece of utopia while intentionally leaving unspecified the rest of the
society that could achieve it. The demand leaves room to maneuver; one
can discuss possible improvements to society that would approach that
utopian goal without committing to a single approach.
I wish this book were better-written and easier to read, since as it
stands I can't recommend it. There were large sections that I read but
didn't have the mental energy to fully decipher or retain, such as the
extended discussion of Ernst Bloch and Friedrich Nietzsche in the context
of utopias. But that way of thinking about utopian demands and their
merits for both the people making them and for those rebutting them, even
if they're not politically feasible, will stick with me.
Rating: 5 out of 10
Three years ago, I organized a fun and most interesting colloquium at
Facultad de Ingenier a, UNAM about
privacy and anonymity online.
I would have loved to share this earlier with the world, but The
university s processes are quite slow (and, to be fair, I also took
quite a bit of time to push things through). But today, I m finally
happy to share the result of that work with all of you. We managed to
get 11 of the talks in the colloquium as articles. The back-cover text
reads (in Spanish):
We live in an era where human to human interactions are
more and more often mediated by technology. This, of course, means
everything leaves a digital trail, a trail that can follow and us
relentlessly.
Privacy is recognized, however, as a human right although one that
is under growing threats. Anonymity is the best tool to secure
it. Throughout history, clear steps have been taken legally,
technically and technologically to defend it. Various studies point
out this is not only a known issue for the network's users, but that a
large majority has searched for alternatives to protect their
communications' privacy.
This book stems from a colloquium held by *Laboratorio de
Investigaci n y Desarrollo de Software Libre* (LIDSOL) of Facultad de
Ingenier a, UNAM, towards the end of 2018, where we invited experts
from disciplines so far apart as law and systems development,
psychology and economics, to contribute with their experiences to a
transdisciplinary vision.
If this interests you, you can get the book at our institutional
repository.
Oh, and What about the birds?
In Spanish (Mexican only?), we have a saying, hay p jaros en el
alambre , meaning watch your words, as uninvited people might be
listening, as birds resting over the wires over which phone calls
used to be made (back in the day where wiretapping was that easy). I
found the design proposed by our editor ingenious and very fitting for
our topic!
Colson Whitehead's latest novel, Harlem Shuffle, was always going to be widely reviewed, if only because his last two books won Pulitzer prizes. Still, after enjoying both The Underground Railroad and The Nickel Boys, I was certainly going to read his next book, regardless of what the critics were saying indeed, it was actually quite agreeable to float above the manufactured energy of the book's launch.
Saying that, I was encouraged to listen to an interview with the author by Ezra Klein. Now I had heard Whitehead speak once before when he accepted the Orwell Prize in 2020, and once again he came across as a pretty down-to-earth guy. Or if I were to emulate the detached and cynical tone Whitehead embodied in The Nickel Boys, after winning so many literary prizes in the past few years, he has clearly rehearsed how to respond to the cliched questions authors must be asked in every interview. With the obligatory throat-clearing of 'so, how did you get into writing?', for instance, Whitehead replies with his part of the catechism that 'It seemed like being a writer could be a cool job. You could work from home and not talk to people.' The response is the right combination of cute and self-effacing... and with its slight tone-deafness towards enforced isolation, it was no doubt honed before Covid-19.
Harlem Shuffle tells three separate stories about Ray Carney, a furniture salesman and 'fence' for stolen goods in New York in the 1960s. Carney doesn't consider himself a genuine criminal though, and there's a certain logic to his relativistic morality. After all, everyone in New York City is on the take in some way, and if some 'lightly used items' in Carney's shop happened to have had 'previous owners', well, that's not quite his problem. 'Nothing solid in the city but the bedrock,' as one character dryly observes. Yet as Ezra pounces on in his NYT interview mentioned abov, the focus on the Harlem underworld means there are very few women in the book, and Whitehead's circular response ah well, it's a book about the criminals at that time! was a little unsatisfying. Not only did it feel uncharacteristically slippery of someone justly lauded for his unflinching power of observation (after all, it was the author who decided what to write about in the first place), it foreclosed on the opportunity to delve into why the heist and caper genres (from The Killing, The Feather Thief, Ocean's 11, etc.) have historically been a 'male' mode of storytelling.
Perhaps knowing this to be the case, the conversation quickly steered towards Ray Carney's wife, Elizabeth, the only woman in the book who could be said possesses some plausible interiority. The following off-hand remark from Whitehead caught my attention:
My wife is convinced that [Elizabeth] knows everything about Carney's criminal life, and is sort of giving him a pass. And I'm not sure if that's true. I have to have to figure out exactly what she knows and when she knows it and how she feels about it.
I was quite taken by this, although not simply due to its effect on the story it self. As in, it immediately conjured up a charming picture of Whitehead's domestic arrangements: not only does Whitehead's wife feel free to disagree with what one of Whitehead's 'own' characters knows or believes, but that Colson has no problem whatsoever sharing that disagreement with the public at large. (It feels somehow natural that Whitehead's wife believes her counterpart knows more than she lets on, whilst Whitehead himself imbues the protagonist's wife with a kind of neo-Victorian innocence.) I'm minded to agree with Whitehead's partner myself, if only due to the passages where Elizabeth is studiously ignoring Carney's otherwise unexplained freak-outs.
But all of these meta-thoughts simply underline just how emancipatory the Death of the Author can be. This product of academic literary criticism (the term was coined by Roland Barthes' 1967 essay of the same name) holds that the original author's intentions, ideas or biographical background carry no especial weight in determining how others should interpret their work. It is usually understood as meaning that a writer's own views are no more valid or 'correct' than the views held by someone else. (As an aside, I've found that most readers who encounter this concept for the first time have been reading books in this way since they were young. But the opposite is invariably true with cinephiles, who often have a bizarre obsession with researching or deciphering the 'true' interpretation of a film.) And with all that in mind, can you think of a more wry example of how freeing (and fun) nature of the Death of the Author than an author's own partner dissenting with their (Pulitzer Prize-winning) husband on the position of a lynchpin character?
The 1964 Harlem riot began after James Powell, a 15-year-old African American, was shot and killed by Thomas Gilligan, an NYPD police officer in front of 10s of witnesses. Gilligan was subsequently cleared by a grand jury.
As it turns out, the reviews for Harlem Shuffle have been almost universally positive, and after reading it in the two days after its release, I would certainly agree it is an above-average book. But it didn't quite take hold of me in the way that The Underground Railroad or The Nickel Boys did, especially the later chapters of The Nickel Boys that were set in contemporary New York and could thus make some (admittedly fairly explicit) connections from the 1960s to the present day that kind of connection is not there in Harlem Shuffle, or at least I did not pick up on it during my reading.
I can see why one might take exception to that, though. For instance, it is certainly true that the week-long Harlem Riot forms a significant part of the plot, and some events in particular are entirely contingent on the ramifications of this momentous event. But it's difficult to argue the riot's impact are truly integral to the story, so not only is this uprising against police brutality almost regarded as a background event, any contemporary allusion to the murder of George Floyd is subsequently watered down. It's nowhere near the historical rubbernecking of Forrest Gump (1994), of course, but that's not a battle you should ever be fighting.
Indeed, whilst a certain smoothness of affect is to be priced into the Whitehead reading experience, my initial overall reaction to Harlem Shuffle was fairly flat, despite all the action and intrigue on the page. The book perhaps belies its origins as a work conceived during quarantine after all, the book is essentially comprised of three loosely connected novellas, almost as if the unreality and mental turbulence of lockdown prevented the author from performing the psychological 'deep work' of producing a novel-length text with his usual depth of craft. A few other elements chimed with this being a 'lockdown novel' as well, particularly the book's preoccupation with the sheer physicality of the city compared to the usual complex interplay between its architecture and its inhabitants. This felt like it had been directly absorbed into the book from the author walking around his deserted city, and thus being able to take in details for the first time:
The doorways were entrances into different cities no, different entrances into one vast, secret city. Ever close, adjacent to all you know, just underneath. If you know where to look.
And I can't fail to mention that you can almost touch Whitehead's sublimated hunger to eat out again as well:
Stickups were chops they cook fast and hot, you re in and out. A stakeout was ribs fire down low, slow, taking your time.
[ ]
Sometimes when Carney jumped into the Hudson when he was a kid, some of that stuff got into his mouth. The Big Apple Diner served it up and called it coffee.
More seriously, however, the relatively thin personalities of minor characters then reminded me of the simulacrum of Zoom-based relationships, and the essentially unsatisfactory endings to the novellas felt reminiscent of lockdown pseudo-events that simply fizzle out without a bang. One of the stories ties up loose ends with: 'These things were usually enough to terminate a mob war, and they appeared to end the hostilities in this case as well.' They did? Well, okay, I guess.
The corner of 125th Street and Morningside Avenue in 2019, the purported location of Carney's fictional furniture store. Signage plays a prominent role in Harlem Shuffle, possibly due to the author's quarantine walks.
Still, it would be unfair to characterise myself as 'disappointed' with the novel, and none of this piece should be taken as really deep criticism. The book certainly was entertaining enough, and pretty funny in places as well:
Carney didn t have an etiquette book in front of him, but he was sure it was bad manners to sit on a man s safe.
[ ]
The manager of the laundromat was a scrawny man in a saggy undershirt painted with sweat stains. Launderer, heal thyself.
Yet I can't shake the feeling that every book you write is a book that you don't, and so we might need to hold out a little longer for Whitehead's 'George Floyd novel'. (Although it is for others to say how much of this sentiment is the expectations of a White Reader for The Black Author to ventriloquise the pain of 'their' community.)
Some room for personal critique is surely permitted. I dearly missed the junk food energy of the dry and acerbic observations that run through Whitehead's previous work. At one point he had a good line on the model tokenisation that lurks behind 'The First Negro to...' labels, but the callbacks to this idea ceased without any payoff. Similar things happened with the not-so-subtle critiques of the American Dream:
Entrepreneur? Pepper said the last part like manure. That s just a
hustler who pays taxes.
[ ]
One thing I ve learned in my job is that life is cheap, and when things start getting expensive, it gets cheaper still.
Ultimately, though, I think I just wanted more. I wanted a deeper exploration of how the real power in New York is not wielded by individual street hoodlums or even the cops but in the form of real estate, essentially serving as a synecdoche for Capital as a whole. (A recent take of this can be felt in Jed Rothstein's 2021 documentary, WeWork: Or the Making and Breaking of a $47 Billion Unicorn and it is perhaps pertinent to remember that the US President at the time this novel was written was affecting to be a real estate tycoon.). Indeed, just like the concluding scenes of J. J. Connolly's Layer Cake, although you can certainly pull off a cool heist against the Man, power ultimately resides in those who control the means of production... and a homespun furniture salesman on the corner of 125 & Morningside just ain't that. There are some nods to kind of analysis in the conclusion of the final story ('Their heist unwound as if it had never happened, and Van Wyck kept throwing up buildings.'), but, again, I would have simply liked more.
And when I attempted then file this book away into the broader media landscape, given the current cultural visibility of 1960s pop culture (e.g. One Night in Miami (2020), Judas and the Black Messiah (2021), Summer of Soul (2021), etc.), Harlem Shuffle also seemed like a missed opportunity to critically analyse our (highly-qualified) longing for the civil rights era. I can certainly understand why we might look fondly on the cultural products from a period when politics was less alienated, when society was less atomised, and when it was still possible to imagine meaningful change, but in this dimension at least, Harlem Shuffle seems to merely contribute to this nostalgic escapism.
This is the fifth and final post in a series about the interface description language Candid.
If you made it this far, you now have a good understanding of what Candid is, what it is for and how it is used. For this final post, I ll put the spotlight on specific aspects of Candid that are maybe surprising, or odd, or quirky. This section will be quite opinionated, and could maybe be called what I d do differently if I d re-do the whole thing .
Note that these quirks are not serious problems, and they don t invalidate the overall design. I am writing this up not to discourage the use of Candid, but merely help interested parties to understand it better.
References in the wire format
When the work on Candid began at DFINITY, the Internet Computer was still far away from being a thing, and many fundamental aspects about it were still in the air. I particular, there was still talk about embracing capabilities as a core feature of the application model, which would be implemented as opaque references on the system level, likely building on WebAssembly s host reference type proposal (which only landed recently), and could be used to model access permissions, custom tokens and many other things.
So Candid is designed with that in mind, and you ll find that its wire format is not just a type table and a value table, but actually
a triple (T, M, R), where T ( type ) and M ( memory ) are sequences of bytes and R ( references ) is a sequence of references.
Also the wire format for values of function service tyeps have an extra byte to distinguish between public references (represented by a principal and possible a method name in the data part), and these opaque references.
Alas, references never made it into the Internet Computer, so all Candid implementations simply ignore that part of the specification. But it s still in the spec, and if it confused you before, now you know why.
Hashed field names
Candid record and variant types look like they have textual field names:
type T = record syndactyle : nat; trustbuster: bool
But that is actually only true superficially. The wire format for Candid only stores hashes of field names. So the above is actually equivalent to
type T = record 4260381820 : nat; 3504418361 : bool
or, for that matter, to
type T = record destroys : bool; rectum : nat
(Yes, I used an english word list to find these hash collisions. There aren t that many actually.)
The benefit of such hashing is that the messages are a bit smaller in most (not all) cases, but it is a big annoyance for dynamic uses of Candid. It s the reason why tools like dfx, if they don t know the Candid interface of a service, will print the result with just the numerical hash, letting you guess which field is which.
It also complicates host languages that derive Candid types from the host language, like Motoko, as some records (e.g. record trustbuster: bool; destroys : int ) with field name hash collisions can not be represented in Candid, and either the host language s type system needs to be Candid aware now (as is the case of Motoko), or serialization/deserialization will fail at runtime, or odd bugs can happen.
(More discussion of this issue).
Tuples
Many languages have a built-in notion of a tuple type (e.g. (Int, Bool)), but Candid does not have such a type. The only first class product type is records.
This means that tuples have to encoded as records somehow. Conveniently(?) record fields are just numbers after all, so the type (Int, Bool) would be mapped to the type
record 0 : int; 1 : bool
So tuples can be expressed. But from my experience implementing the type mappings for Motoko and Haskell this is causing headaches. To get a good experience when importing from Candid, the tools have to try to reconstruct which records may have been tuples originally, and turn them into tuples.
The main argument for the status quo is that Candid types should be canonical, and there should not be more than one product type, and records are fine, and there needs to be no anonymous product type. But that has never quite resonated with me, given the practical reality of tuple types in many host languages.
Argument sequences
Did I say that Candid does not have tuple types? Turns out it does, sort of. There is no first class anonymous product, but since functions take sequences of arguments and results, there is a tuple type right there:
func foo : (bool, int) -> (int, bool)
Again, I found that ergonomic interaction with host languages becomes relatively unwieldy by requiring functions to take and return sequences of values. This is especially true for languages where functions take one argument value or return one result type (the latter being very common). Here, return sequences of length one are turned into that type directly, longer argument sequences turn into the host language s tuple type, and nullary argument sequences turn into the idiomatic unit type. But this means that the types (int, bool) -> () and (record 0: int, 1: bool ) -> () may be mapped to the same host language type, which causes problems when you hope to encode all necessary Candid type information in the host language.
Another oddity with argument and result sequences is that you can give names to the entries, e.g. write
without breaking clients, this does not have the effect you think it has and will likely break.
My suggestion is to never put names on function arguments and result values in Candid interfaces, and for anything that might be extended with new fields or where you want to name the arguments, use a single record type as the only argument:
This allows you to add and remove arguments more easily and reliably.
Type shorthands
The Candid specification defines a system of types, and then adds a number of syntactic short-hands . For example, if you write blob in a Candid type description, it ought to means the same as vec nat8.
My qualm with that is that it doesn t always mean the same. A Candid type description is interpreted by a number of, say, consumers . Two such consumers are part of the Candid specification:
The specification that defines the wire format for that type
The upgrading (subtyping) rules
But there are more! For every host language, there is some mapping from Candid types to host language types, and also generic tools like Candid UI are consumers of the type algebra. If these were to take the Candid specification as gospel, they would be forced to treat blob and vec nat8 the same, but that would be quite unergonomic and might cause performance regressions (most language try to map blob to some compact binary data type, while vec t tends to turn into some form of array structure).
So they need to be pragmatic and treat blob and vec nat8 differently. But then, for all practical purposes, blob is not just a short-hand of vec nat8. They are different types that just happens to have the same wire representations and subtyping relations.
This affects not just blob, but also tuples (record blob; int; bool ) and field names , as discussed above.
The value text format
For a while after defining Candid, the only implementation was in Motoko, and all the plumbing was automatic, so there was never a need for users to to explicitly handle Candid values, as all values were Motoko values. Still, for debugging and testing and such things, we eventually needed a way to print out Candid values, so the text format was defined ( To enable convenient debugging, the following grammar specifies a text format for values ).
But soon the dfx tool learned to talk to canisters, and now users needed to enter Candid value on the command line, possibly even when talking to canisters for which the interface was not known to dfx. And, sure enough, the textual interface defined in the Candid spec was used.
Unfortunately, since it was not designed for that use case, it is rather unwieldy:
It is quite verbose. You have to write record , not just . Vectors are written vec ; instead of some conventional syntax like [ , ]. Variants are written as variant error = " " with braces that don t any value here, and something like #error " " might have worked as well.
With a bit more care, a more concise and ergonomic syntax might have been possible.
It wasn t designed to be sufficient to create a Candid value from it. If you write 5 it s unclear whether that s a nat or an int16 or what (and all of these have different wire representations). Type annotations were later added, but are relatively unwieldy, and don t cover all cases (e.g. a service reference with a recursive type cannot be represented in the textual format at the moment).
Not really the fault of the textual format, but some useful information about the types is not reflected in the type description that s part of the wire format. In particular not the field names, and whether a value was intended to be binary data (blob) or a list of small numbers (vec nat8), so pretty-printing such values requires guesswork. The Haskell library even tries to brute-force the hash to guess the field name, if it is short or in a english word list!
In hindsight I think it was too optimistic to assume that correct static type information is always available, and instead of actively trying to discourage dynamic use, Candid might be better if we had taken these (unavoidable?) use cases into account.
Custom wire format
At the beginning of this post, I have a Candid is list. The list is relatively long, and the actual wire format is just one bullet point. Yes, defining a wire format that works isn t rocket science, and it was easiest to just make one up. But since most of the interesting meat of Candid is in other aspects (subtyping rules, host language integration), I wonder if it would have been better to use an existing, generic wire format, such as CBOR, and build Candid as a layer on top.
This would give us plenty of tools and libraries to begin with. And maybe it would have reduced barrier of entry for developers, which now led to the very strange situation that DFINITY advocates for the universal use of Candid on the Internet Computer, so that all services can smoothly interact, but two of the most important services on the Internet Computer (the registry and the ledger) use Protobuf as their primary interface format, with Candid interfaces missing or an afterthought.
Sideways Interface Evolution
This is not a quirk of Candid itself, but rather an idiom of how you can use Candid that emerged from our solution for record extensions and upgrades.
Consider our example from before, a service with interface
service add_user : (record login : text; name : text ) -> ()
where you want to add an age field, which should be a number.
The official way of doing that is to add that field with an optional type:
service add_user : (record login : text; name : text; age : opt nat ) -> ()
As explained above, this will not break old clients, as the decoder will treat a missing argument as null. So far so good.
But often when adding such a field you don t want to bother new clients with the fact that this age was, at some point in the past, not there yet. And you can do that! The trick is to distinguish between the interface you publish and the interface you implement. You can (for example in your documentation) state that the interface is
service add_user : (record login : text; name : text; age : nat ) -> ()
which is not a subtype of the old type, but it is the interface you want new clients to work with. And then your implementation uses the type with opt nat. Calls from old clients will come through as null, and calls from new clients will come through as opt 42.
We can see this idiom used in the Management Canister of the Internet Computer. The current documented interface only mentions a controllers : vec principal field in the settings, but the implementation still can handle both the old controller : principal and the new controllers field.
It s probably advisable to let your CI system check that new versions of your service continue to implement all published interfaces, including past interfaces. But as long as the actual implementation s interface is a subtype of all interfaces ever published, this works fine.
This pattern is related to when your service implements, say, http_request (so its implemented interface is a subtype of that common interface), but does not include that method in the published documentation (because clients of your service don t need to call it).
Self-describing Services
As you have noticed, Candid was very much designed assuming that all parties always have the service type of services they want to interact with. But the Candid specification does not define how one can obtain the interface of a given service, and there isn t really a an official way to do that on the Internet Computer.
That is unfortunate, because many interesting features depend on that: Such as writing import C "ic:7e6iv-biaaa-aaaaf-aaada-cai" in your Motoko program, and having it s type right there. Or tools like ic.rocks, that allow you to interact with any canister right there.
One reason why we don t really have that feature yet is because of disagreements about how dynamic that feature should be. Should you be able to just ask the canister for its interface (and allow the canister to vary the response, for example if it can change its functionality over time, even without changing the actual wasm code)? Or is the interface a static property of the code, and one should be able to query the system for that data, without the canister s active involvement. Or, completely different, should interfaces be distributed out of band, maybe together with API documentation, or in some canister registry somewhere else?
I always leaned towards the first of these options, but not convincingly enough. The second options requires system assistance, so more components to change, more teams to be involved that maybe intrinsically don t care a lot about this feature. And the third might have emerged as the community matures and builds such infrastructure, but that did not happen yet.
In the end I sneaked in an implementation of the first into Motoko, arguing that even if we don t know yet how this feature will be implemented eventually, we all want the feature to exist somehow, and we really really want to unblock all the interesting applications it enables (e.g. Candid UI). That s why every Motoko canister, and some rust canisters too, implements a method
__get_candid_interface_tmp_hack : () -> (text)
that one can use to get the Candid interface file.
The name was chosen to signal that this may not be the final interface, but like all good provisional solutions, it may last longer than intended. If that s the case, I m not sorry.
This concludes my blog post series about Candid, for now. If you want to know more, feel free to post your question on the DFINTY developer forum, and I ll probably answer.
This release is the culmination of a project that I've been wanting to do
for two years, but nearly all the work was done in the past week. That
experience made me rethink some of my skepticism, but I'll get to that
part of the story later.
In March of 1999, I got tired of writing HTML by hand and wrote a small
program called spin that implemented a macro language that translated into
HTML. This makes it one of the oldest programs for which I have a
continuous development history, predating podlators by three months. I think only News::Gateway (now very dormant) and Term::ANSIColor (still under active development but very stable) are
older, as long as I'm not counting orphaned packages like
newsyslog.
I've used spin continuously ever since. It's grown features and an
ecosystem of somewhat hackish scripts to do web publishing things I've
wanted over the years: journal entries like this one, book reviews, a
simple gallery (with some now-unfortunate decisions about maximum image
size), RSS feeds, and translation of lots of different input files into
HTML. But the core program itself, in all those years, has been one
single Perl script written mostly in my Perl coding style from the early
2000s before I read Perl Best Practices.
My web site is long overdue for an overhaul. Just to name a couple of
obvious problems, it looks like trash on mobile browsers, and I'm using
URL syntax from the early days of the web that, while it prompts some
nostalgia for tildes, means all the URLs are annoyingly long and embed
useless information such as the fact each page is written in HTML. Its
internals also use a lot of ad hoc microformats (a bit of RFC 2822 here, a
text-based format with significant indentation there, a weird
space-separated database) and are supported by programs that extract
meaning from human-written pages and perform automated updates to them
rather than having a clear separation between structure and data.
This will be a very large project, but it's the sort of quixotic personal
project that I enjoy. Maintaining my own idiosyncratic static site
generator is almost certainly not an efficient use of my time compared to,
say, converting everything to Hugo. But I have
3,428 pages (currently) written in the thread macro language, plus
numerous customizations that cater to my personal taste and interests,
and, most importantly, I like having a highly customized system that I
know exactly how to automate.
The blocker has been that I didn't want to work on spin as it existed. It
badly needed a structural overhaul and modernization, and even more badly
needed a test suite since every release involved tedious manual testing by
pouring over diffs between generations of the web site. And that was
enough work to be intimidating, so I kept putting it off.
I've separately been vaguely aware that I have been spending too much time
reading Twitter (specifically) and the news (in general). It would be one
thing if I were taking in that information to do something productive
about it, but I haven't been. It's just doomscrolling. I've been
thinking about taking a break for a while but it kept not sticking, so I
decided to make a concerted effort this week.
It took about four days to stop wanting to check Twitter and forcing
myself to go do something else productive or at least play a game instead.
Then I managed to get started on my giant refactoring project, and holy
shit, Twitter has been bad for my attention span! I haven't been able to
sustain this level of concentration for hours at a time in years.
Twitter's not the only thing to blame (there are a few other stressers
that I've fixed in the past couple of years), but it's obviously a huge
part.
Anyway, this long personal ramble is prelude to the first release of
DocKnot that includes my static site generator. This is not yet the full
tooling from my old web tools page;
specifically, it's missing faq2html, cl2xhtml, and cvs2xhtml. (faq2html
will get similar modernization treatment, cvs2xhtml will probably be
rewritten in Perl since I have some old, obsolete scripts that may live in
CVS forever, and I may retire cl2xhtml since I've stopped using the GNU
ChangeLog format entirely.) But DocKnot now contains the core of my site
generation system, including the thread macro language, POD conversion (by
way of Pod::Thread), and RSS feeds.
Will anyone else ever use this? I have no idea; realistically, probably
not. If you were starting from scratch, I'm sure you'd be better off with
one of the larger and more mature static site generators that's not the
idiosyncratic personal project of one individual. It is packaged for
Debian because it's part of the tool chain for generating files
(specifically README.md) that are included in every package I
maintain, and thus is part of the transitive closure of Debian main, but
I'm not sure anyone will install it from there for any other purpose. But
for once making something for someone else isn't the point. This is my
quirky, individual way to maintain web sites that originated in an older
era of the web and that I plan to keep up-to-date (I'm long overdue to
figure out what they did to HTML after abandoning the XHTML approach)
because it brings me joy to do things this way.
In addition to adding the static site generator, this release also has the
regular sorts of bug fixes and minor improvements: better formatting of
software pages for software that's packaged for Debian, not assuming every
package has a TODO file, and ignoring Autoconf 2.71 backup files when
generating distribution tarballs.
You can get the latest version of DocKnot from CPAN as App-DocKnot, or
from its distribution page. I know I
haven't yet updated my web tools page to reflect this move, or changed the
URL in the footer of all of my pages. This transition will be a process
over the next few months and will probably prompt several more minor
releases.
Cisco pyATS is a framework for network automation and testing. It
includes, among other things, an open-source multi-vendor set of
parsers and models, Genie Parser. It features 2700 parsers for
various commands over many network OS. On the paper, this seems a
great tool!
>>> fromgenie.conf.baseimportDevice>>> device=Device("router",os="iosxr")>>> # Hack to parse outputs without connecting to a device>>> device.custom.setdefault("abstraction", )["order"]=["os","platform"]>>> cmd="show route ipv4 unicast">>> output="""... Tue Oct 29 21:29:10.924 UTC...... O 10.13.110.0/24 [110/2] via 10.12.110.1, 5d23h, GigabitEthernet0/0/0/0.110... """>>> device.parse(cmd,output=output) 'vrf': 'default': 'address_family': 'ipv4': 'routes': '10.13.110.0/24': 'route': '10.13.110.0/24', 'active': True, 'route_preference': 110, 'metric': 2, 'source_protocol': 'ospf', 'source_protocol_codes': 'O', 'next_hop': 'next_hop_list': 1: 'index': 1, 'next_hop': '10.12.110.1', 'outgoing_interface': 'GigabitEthernet0/0/0/0.110', 'updated': '5d23h'
First deception: pyATS is closed-source with some exceptions. This
is quite annoying if you run into some issues outside Genie Parser.
For example, although pyATS is using the ssh command, it cannot
leverage my ssh_config file: pyATSresolves hostnames before
providing them to ssh. There is no plan to open source
pyATS.
Then, Genie Parser has two problems:
The data models used are dependent on the vendor and OS,
despite the documentation saying otherwise. For
example, the data model used for IPv4 interfaces is different
between NX-OS and IOS-XR.
The parsers rely on line-by-line regular expressions to extract
data and some Python code as glue. This is fragile and may break
silently.
To illustrate the second point, let s assume the output of show ipv4
vrf all interface is:
Loopback10 is Up, ipv4 protocol is Up
Vrf is default (vrfid 0x60000000)
Internet protocol processing disabled
Loopback30 is Up, ipv4 protocol is Down [VRF in FWD reference]
Vrf is ran (vrfid 0x0)
Internet address is 203.0.113.17/32
MTU is 1500 (1500 is available to IP)
Helper address is not set
Directed broadcast forwarding is disabled
Outgoing access list is not set
Inbound common access list is not set, access list is not set
Proxy ARP is disabled
ICMP redirects are never sent
ICMP unreachables are always sent
ICMP mask replies are never sent
Table Id is 0x0
Because the regular expression to parse an interface name does not
expect the extra data after the interface state, Genie Parser
ignores the line starting the definition of Loopback30 and parses
the output to this structure:1
While this bug is simple to fix, this is an uphill battle. Any
existing or future slight variation in the output of a command could
trigger another similar undetected bug, despite the extended test
coverage. I have reported and fixed several other silent parsing
errors: #516, #529, and #530. A more robust alternative
would have been to use TextFSM and to trigger a warning when some
output is not recognized, like Batfish, a configuration analysis
tool, does.
In the future, we should rely on YANG for data extraction, but it
is currently not widely supported. SNMP is still a valid possibility
but much information is not accessible through this protocol. In the
meantime, I would advise you to only use Genie Parser with caution.
As an exercise, the astute reader is asked to write the code
to extract the IPv4 from this structure.
I had this nightmare where I had a very, very important confcall.
I joined with Chrome. Chrome said Failed to access your microphone -
Cannot use microphone for an unknown reason. Could not start audio source.
I joined with Firefox. Firefox chose Monitor of Built-in Audio Analog Stereo
as a microphone, and did not let me change it. Not in the browser, not in
pavucontrol.
I joined with the browser on my phone, and the webpage said This meeting needs
to use your microphone and camera. Select *Allow* when your browser asks for
permissions. But the question never came.
I could hear people talking. I had very important things to say. I tried typing
them in the chat window, but they weren't seeing it. The meeting ended. I was
on the verge of tears.
Tell me, Mr. Anderson, what good is a phone call when you are unable to speak?
Since this nightmare happened for real, including the bit about tears in the
end, let's see that it doesn't happen again. I should now have three working
systems, which hopefully won't all break again all at the same time.
Fixing Chrome
I can reproduce this reliably, on Bullseye's standard Chromium
90.0.4430.212-1, just launched on an empty profile, no extensions.
The webpage has camera and microphone allowed. Chrome doesn't show up in the
recording tab of pulseaudio. Nothing on Chrome's stdout/stderr.
JavaScript console has:
Logger.js:154 2021-09-10Txx:xx:xx.xxxZ [features/base/tracks] Failed to create local tracks
Array(2)
DOMException: Could not start audio source
I had the similar problem once with chromium.
i could solve it by switching in preferences->microphone-> from "default" to "intern analog stereo".
Opening the little popup next to the microphone/mute button allows choosing
other microphones, which work. Only "Same as system (Default)" does not work.
Fixing Firefox
I have firefox-esr 78.13.0esr-1~deb11u1. In Jitsi, microphone selection is
disabled on the toolbar and in the settings menu. In pavucontrol, changing the
recording device for Firefox has no effect. If for some reason the wrong
microphone got chosen, those are not ways of fixing it.
What I found works is to click on the camera permission icon, remove microphone
permission, then reload the page. At that point Firefox will ask for permission
again, and that microphone selection seems to work.
Relevant bugs: on Jitsi and
on Firefox. Since this
is well known (once you find the relevant issues), I'd have appreciated Jitsi
at least showing a link to an explanation of workarounds on Firefox, instead of
just disabling microphone selection.
Fixing Jitsi on the phone side
I really don't want to preemptively give camera and microphone permissions to
my phone browser. I noticed that there's the Jitsi app on F-Droid
and much as I hate to use an app when a website would work, at least in this
case it's a way to keep the permission sets separate, so I installed that.
Fixing pavucontrol?
I tried to find out why I can't change input device for FireFox on pavucontrol.
I only managed to find an Ask Ubuntu question with no answer
and a Unix StackExchange question with no answer.
It seems that every cloud service of note has a free tier nowadays and the Oracle Cloud is the latest that I ve discovered (thanks to r/homelab which I highly recommend reading). Here s Oracle s summary of what they offer for free [1].
Oracle s always free tier (where presumable always is defined as until we change our contract ) currently offers ARM64 VMs to a total capacity of 4 CPU cores, 24G of RAM, and 200G of storage with a default VM size of 1/4 that (1 CPU core and 6G of RAM). It also includes 2 AMD64 VMs that each have 1G of RAM, but a 64bit VM with 1G of RAM isn t that useful nowadays.
Web Interface
The first thing to note is that the management interface is a massive pain to use. When a login times out for security reasons it redirects to a web page that gives a 404 error, maybe the redirection works OK if you are using it when it times out, but if you go off and spend an hour doing something else you will return to a 404 page. A web interface should never refer you to a page with a 404.
There doesn t seem to be a way of bookmarking the commonly used links (as AWS does) and the set of links on the left depend on the section you are in with no obvious way of going between sections. Sometimes I got stuck in a set of pages about authentication controls (the identity cloud ) and there seems to be no link I could click on to get me back to cloud computing, I had to go to a bookmarked link for the main cloud login page. A web interface should never force the user to type in the main URL or go to a bookmark, you should be able to navigate from every page to every other page in a logical manner. An advanced user might have their own bookmarks in their browser to suit their workflow. But a beginner should be able to go to anywhere without breaking the session.
Some parts of the interface appear to be copied from AWS, but unfortunately not the good parts. The way AWS manages IP access control is not easy to manage and it s not clear why packets are dropped, Oracle copies all this. On the upside Oracle has some good Datadog style analytics so for a new deployment you can debug IP access control by seeing records of rejected packets. Just to make it extra annoying when you create a rule with multiple ports specified the web interface will expand it out to multiple rules for one port each, having ports 80 and 443 on separate lines doesn t make things easier. Also it forces you to have IPv4 and IPv6 as separate rules, so if you want HTTP and HTTPS on both IPv4 and IPv6 (a common requirement) then you need 4 separate rules.
One final annoying thing is that the web interface doesn t make your previous settings a default. As I ve created many ARM images and haven t created a single AMD image it should know that the probability that I want to create an AMD image is very low and stop defaulting to that.
Recovery
When trying a new system you will inevitably break things and have to recover things. The way to recover from a configuration error that prevents your VM from booting and getting to a state of allowing a login is to go to stop the VM, then go to the Boot volume section under Resources and use the settings button to detach the boot volume. Then you go to another VM (which must be running), go to the Attached block volumes menu and attach it as Paravirtualised (not iSCSI and not default which will probably be iSCSI). After some time the block device will appear and you can mount it and do stuff to it. Then after umounting it you detach it from the recovery VM and attach it again to the original VM (where it will still have an entry in the Boot volume section) and boot the original VM.
As an aside it s really annoying that you can t attach a volume to a VM that isn t running.
My first attempt at image recovery started with making a snapshot of the Boot volume, this didn t work well because the image uses EFI and therefore GPT and because the snapshot was larger than the original block device (which incidentally was the default size). I admit that I might have made a mistake making the snapshot, but if so it shouldn t be so easy to do. With GPT if you have a larger block device then partitioning tools complain about the backup partition table not being found, and they complain even more if you try to go back to the smaller size later on. Generally GPT partition tables are a bad idea for VMs, when I run the host I don t use partition tables, I have a separate block device for each filesystem or swap space.
Snapshots aren t needed for recovery, they don t seem to work very well, and if it s possible to attach a snapshot to a VM in place of it s original Boot volume I haven t figured out how to do it.
Console Connection
If you boot Oracle Linux a derivative of RHEL that has SE Linux enabled in enforcing mode (yay) then you can go to the Console connection . The console is a Javascript console which allows you to login on a virtual serial console on device /dev/ttyAMA0. It tells you to type help but that isn t accepted, you have a straight Linux console login prompt.
If you boot Ubuntu then you don t get a working serial console, it tells you to type help for help but doesn t respond to that.
It seems that the Oracle Linux kernel 5.4.17-2102.204.4.4.el7uek.aarch64 is compiled with support for /dev/ttyAMA0 (the default ARM serial device) while the kernel 5.11.0-1016-oracle compiled by Oracle for their Ubuntu VMs doesn t have it.
Performance
I haven t done any detailed tests of VM performance. As a quick test I used zstd to compress a 154MB file, on my home workstation (E5-2620 v4 @ 2.10GHz) it took 11.3 seconds of CPU time to compress with zstd -9 and 7.2s to decompress. On the Oracle cloud it took 7.2s and 5.4s. So it seems that for some single core operations the ARM CPU used by the Oracle cloud is about 30% to 50% faster than a E5-2620 v4 (a slightly out of date server processor that uses DDR4 RAM).
If you ran all the free resources in a single VM that would make a respectable build server. If you want to contribute to free software development and only have a laptop with 4G of RAM then an ARM build/test server with 24G of RAM and 4 cores would be very useful.
Ubuntu Configuration
The advantage of using EFI is that you can manage the kernel from within the VM. The default Oracle kernel for Ubuntu has a lot of modules included and is compiled with a lot of security options including SE Linux.
Competitors
https://aws.amazon.com/free
AWS offers 750 hours (just over 31 days) per month of free usage of a t2.micro or t3.micro EC2 instance (which means 1GB of RAM). But that only lasts for 12 months and it s still only 1GB of RAM. AWS has some other things that could be useful like 1 million free Lambda requests per month. If you want to run your personal web site on Lambda you shouldn t hit that limit. They also apparently have some good offers for students.
https://cloud.google.com/free
The Google Cloud Project (GCP) offers $300 of credit.
https://cloud.google.com/free/docs/gcp-free-tier#free-tier-usage-limits
GCP also has ongoing free tier usage for some services. Some of them are pretty much unlimited use (50GB of storage for Cloud Source Repositories is a heap of source code). But for VMs you get the equivalent of 1*e2-micro instance running 24*7. A e2-micro has 1G of RAM. You also only get 30G of storage and 1GB of outbound data. It s clearly not as generous an offer as Oracle, but Oracle is the underdog so they have to try harder.
https://azure.microsoft.com/en-us/free/
Azure appears to be much the same as AWS, free Linux VM for a year and then other less popular services free forever (or until they change the contract).
https://www.ibm.com/cloud/free
The IBM cloud free tier is the least generous offer, a VM is only free for 30 days. But what they offer for 30 days is pretty decent. If you want to try the IBM cloud and see if it can do what your company needs then this will do well. If you want to have free hosting for your hobby stuff then it s no good.
Oracle seems like the most generous offer if you want to do stuff, but also one of the least valuable if you want to learn things that will help you at a job interview. For job interviews AWS seems the most useful and then GCP and Azure vying for second place.
The Debian Janitor is an automated
system that commits fixes for (minor) issues in Debian packages that can be
fixed by software. It gradually started proposing merges in early
December. The first set of changes sent out ran lintian-brush on sid packages maintained in
Git. This post is part of a series about the progress of the Janitor.
As covered in my post from last week, the Janitor now regularly tries to
import new upstream git snapshots or upstream releases into packages in Sid.
Moving parts
There are about 30,000 packages in sid, and it usually takes a couple of weeks
for the janitor to cycle through all of them. Generally speaking, there are up
to three moving targets for each package:
The packaging repository; vcswatch regularly scans this for changes,
and notifies the janitor when a repository has changed. For salsa
repositories it is instantly notified through a web hook
The upstream release tarballs; the QA watch service regularly polls these,
and the janitor scans for changes in the UDD tables with watch data (used for
fresh-releases)
The upstream repository; there is no service in Debian that watches this at
the moment (used for
fresh-snapshots)
When the janitor notices that one of these three targets has changed, it
prioritizes processing of a package - this means that a push to a packaging
repository on salsa usually leads to a build being kicked off within 10
minutes. New upstream releases are usually noticed by QA watch within a day or
so and then lead to a build. Now commits in upstream repositories don t get
noticed today.
Note that there are no guarantees; the scheduler tries to be clever and not
e.g. rebuild the same package over and over again if it s constantly changing
and takes a long time to build.
Packages without priority are processed with a scoring system that takes into
account perceived value (based on e.g. popcon), cost (based on wall-time
duration of previous builds) and likelihood of success (whether recent builds
were successful, and how frequently the repositories involved change).
webhooks for upstream repositories
At the moment there is no service in Debian (yet - perhaps this is something
that vcswatch or a sibling service could also do?) that scans upstream
repositories for changes.
However, if you maintain an upstream package, you can use a webhook to notify
the janitor that commits have been made to your repository, and it will create
a new package in fresh-snapshots. Webhooks from the
following hosting site software are currently supported:
You can simply use the URLhttps://janitor.debian.net/ as the target for hooks. There is no need to specify a secret, and the hook can either use a JSON or form encoding payload.
The endpoint should tell you whether it understood a webhook request, and
whether it took any action. It s fine to submit webhooks for repositories that
the janitor does not (yet) know about.
GitHub
For GitHub, you can do so in the Webhooks section of the Settings tab. Fill the form as shown below and click on Add webhook:
GitLab
On GitLab instances, you can find the Webhooks tab under the Settings menu for each repository (under the gear symbol). Fill the form in as shown below and click Add Webhook:
Launchpad
For Launchpad, go to the repository (for Git) web view and click Manage Webhooks. From there, you can add a new webhook; fill the form in as shown below and click Add Webhook:
I have been using the awesome window manager for 10 years. It is a
tiling window manager, configurable and extendable with the
Lua language. Using a general-purpose programming language to
configure every aspect is a double-edged sword. Due to laziness and
the apparent difficulty of adapting my configuration about 3000 lines to newer releases, I was stuck with the 3.4
version, whose last release is from 2013.
It was time for a rewrite. Instead, I have switched to the i3 window
manager, lured by the possibility to migrate to Wayland and
Sway later with minimal pain. Using an embedded interpreter for
configuration is not as important to me as it was in the past: it
brings both complexity and brittleness.
Dual screen desktop running i3, Emacs, some terminals, including a Quake console, Firefox, Polybar as the status bar, and Dunst as the notification daemon.
The window manager is only one part of a desktop environment. There
are several options for the other components. I am also introducing
them in this post.
i3: the window manageri3 aims to be a minimal tiling window manager. Its documentation
can be read from top to bottom in less than an hour. i3 organize
windows in a tree. Each non-leaf node contains one or several
windows and has an orientation and a layout. This information
arbitrates the window positions. i3 features three layouts: split,
stacking, and tabbed. They are demonstrated in the below screenshot:
Demonstration of the layouts available in i3. The main container is split horizontally. The first child is split vertically. The second one is tabbed. The last one is stacking.
Tree representation of the previous screenshot.
Most of the other tiling window managers, including the awesome
window manager, use predefined layouts. They usually feature a large
area for the main window and another area divided among the remaining
windows. These layouts can be tuned a bit, but you mostly stick to a
couple of them. When a new window is added, the behavior is quite
predictable. Moreover, you can cycle through the various windows
without thinking too much as they are ordered.
i3 is more flexible with its ability to build any layout on the fly,
it can feel quite overwhelming as you need to visualize the tree in
your head. At first, it is not unusual to find yourself with a complex
tree with many useless nested containers. Moreover, you have to
navigate windows using directions. It takes some time to get used to.
I set up a split layout for Emacs and a few terminals, but most of the
other workspaces are using a tabbed layout. I don t use the stacking
layout. You can find many scripts trying to emulate other tiling
window managers but I did try to get my setup pristine of these
tentatives and get a chance to familiarize myself. i3 can also save
and restore layouts, which is quite a powerful feature.
My configuration is quite similar to the default one and has
less than 200 lines.
i3 companion: the missing bitsi3 philosophy is to keep a minimal core and let the user implements
missing features using the IPC protocol:
Do not add further complexity when it can be avoided. We are
generally happy with the feature set of i3 and instead focus on
fixing bugs and maintaining it for stability. New features will
therefore only be considered if the benefit outweighs the additional
complexity, and we encourage users to implement features using the
IPC whenever possible.
Introduction to the i3 window manager
While this is not as powerful as an embedded language, it is enough
for many cases. Moreover, as high-level features may be opinionated,
delegating them to small, loosely coupled pieces of code keeps them
more maintainable. Libraries exist for this purpose in several
languages. Users have published many scripts to extend i3:
automatic layout and window promotion to mimic the behavior
of other tiling window managers, window swallowing to put a new
app on top of the terminal launching it, and cycling between
windows with Alt+Tab.
Instead of maintaining a script for each feature, I have centralized
everything into a single Python process,
i3-companion using asyncio and the
i3ipc-python library. Each feature is self-contained into a
function. It implements the following components:
make a workspace exclusive to an application
When a workspace contains Emacs or Firefox, I would like other
applications to move to another workspace, except for the terminal
which is allowed to intrude into any workspace. The
workspace_exclusive() function monitors new windows and moves them
if needed to an empty workspace or to one with the same application
already running.
implement a Quake console
The quake_console() function implements a drop-down console
available from any workspace. It can be toggled with
Mod+. This is implemented as a scratchpad
window.
back and forth workspace switching on the same output
With the workspace back_and_forth command, we can ask i3 to
switch to the previous workspace. However, this feature is not
restricted to the current output. I prefer to have one keybinding to
switch to the workspace on the next output and one keybinding to
switch to the previous workspace on the same output. This behavior
is implemented in the previous_workspace() function by keeping a
per-output history of the focused workspaces.
create a new empty workspace or move a window to an empty workspace
To create a new empty workspace or move a window to an empty
workspace, you have to locate a free slot and use workspace number
4 or move container to workspace number 4. The new_workspace()
function finds a free number and use it as the target workspace.
restart some services on output change
When adding or removing an output, some actions need to be executed:
refresh the wallpaper, restart some components unable to adapt their
configuration on their own, etc. i3 triggers an event for this
purpose. The output_update() function also takes an extra step to
coalesce multiple consecutive events and to check if there is a real
change with the low-level library xcffib.
I will detail the other features as this post goes on. On the
technical side, each function is decorated with the events it should
react to:
@on(CommandEvent("previous-workspace"),I3Event.WORKSPACE_FOCUS)asyncdefprevious_workspace(i3,event):"""Go to previous workspace on the same output."""
The CommandEvent() event class is my way to send a command to the
companion, using either i3-msg -t send_tick or binding a key to a
nop command. The latter is used to avoid spawning a shell and a
i3-msg process just to send a message. The companion listens to
binding events and checks if this is a nop command.
bindsym $mod+Tab nop "previous-workspace"
There are other decorators to avoid code duplication: @debounce() to
coalesce multiple consecutive calls, @static() to define a static
variable, and @retry() to retry a function on failure. The whole
script is a bit more than 1000 lines. I think this is
worth a read as I am quite happy with the result.
dunst: the notification daemon
Unlike the awesome window manager, i3 does not come with a built-in
notification system. Dunst is a lightweight notification daemon. I
am running a modified version with HiDPI support for X11 and
recursive icon lookup. The i3 companion has a helper function,
notify(), to send notifications using DBus. container_info() and
workspace_info() uses it to display information about the container
or the tree for a workspace.
Notification showing i3 s tree for a workspace
polybar: the status bari3 bundles i3bar, a versatile status bar, but I have opted for
Polybar. A wrapper script runs one instance for
each monitor.
The first module is the built-in support for i3 workspaces. To not
have to remember which application is running in a workspace, the i3
companion renames workspaces to include an icon for each application.
This is done in the workspace_rename() function. The icons are from
the Font Awesome project. I maintain a mapping between applications
and icons. This is a bit cumbersome but it looks great.
i3 workspaces in Polybar
For CPU, memory, brightness, battery, disk, and audio volume, I am
relying on the built-in modules. Polybar s wrapper script generates the list of filesystems to monitor and they get only
displayed when available space is low. The battery widget turns red
and blinks slowly when running out of power. Check my Polybar
configuration for more details.
Polybar displaying various information: CPU usage, memory usage, screen brightness, battery status, Bluetooth status (with a connected headset), network status (connected to a wireless network and to a VPN), notification status, and speaker volume.
For Bluetooh, network, and notification statuses, I am using Polybar s
ipc module: the next version of Polybar can receive
an arbitrary text on an IPC socket. The module is defined with a
single hook to be executed at the start to restore the latest status.
It can be updated with polybar-msg action "#network.send.XXXX". In
the i3 companion, the @polybar() decorator takes the string
returned by a function and pushes the update through the IPC socket.
The i3 companion reacts to DBus signals to update the Bluetooth and
network icons. The @on() decorator accepts a DBusSignal() object:
@on(StartEvent,DBusSignal(path="/org/bluez",interface="org.freedesktop.DBus.Properties",member="PropertiesChanged",signature="sa sv as",onlyif=lambdaargs:(args[0]=="org.bluez.Device1"and"Connected"inargs[1]orargs[0]=="org.bluez.Adapter1"and"Powered"inargs[1]),),)@retry(2)@debounce(0.2)@polybar("bluetooth")asyncdefbluetooth_status(i3,event,*args):"""Update bluetooth status for Polybar."""
The middle of the bar is occupied by the date and a weather forecast.
The latest also uses the IPC mechanism, but the source is a Python
script triggered by a timer.
Current date and weather forecast for the day in Polybar. The data is retrieved with the OpenWeather API.
I don t use the system tray integrated with Polybar. The embedded
icons usually look horrible and they all behave differently. A few
years back, Gnome has removed the system tray. Most of the
problems are fixed by the DBus-based Status Notifier Item
protocol also known as Application Indicators or Ayatana
Indicators for GNOME. However, Polybar does not support this
protocol. In the i3 companion, The implementation of Bluetooth
and network icons, including displaying notifications on change, takes
about 200 lines. I got to learn a bit about how DBus works and I get exactly
the info I want.
picom: the compositor
I like having slightly transparent backgrounds for terminals and to
reduce the opacity of unfocused windows. This requires a
compositor.1picom is a lightweight compositor. It works
well for me, but it may need some tweaking depending on your graphic
card.2 Unlike the awesome window manager, i3 does not handle
transparency, so the compositor needs to decide by itself the opacity
of each window. Check my configuration for details.
systemd: the service manager
I use systemd to start i3 and the various services around it. My
xsession script only sets some environment variables and lets
systemd handles everything else. Have a look at this article from
Micha G ral for the rationale. Notably, each component can be
easily restarted and their logs are not mangled inside the
~/.xsession-errors file.3
I am using a two-stage setup: i3.service depends on
xsession.target to start services before
i3:
rofi: the application launcherRofi is an application launcher. Its appearance can be customized
through a CSS-like language and it comes with several themes. Have a
look at my configuration for mine.
Rofi as an application launcher
It can also act as a generic menu application. I have a script
to control a media player and another one to
select the wifi network. It is quite a flexible
application.
Rofi to select a wireless network
xss-lock and i3lock: the screen lockeri3lock is a simple screen locker. xss-lock invokes it reliably
on inactivity or before a system suspend. For inactivity, it uses the
XScreenSaver events. The delay is configured using the xset s
command. The locker can be invoked immediately with xset s activate.
X11 applications know how to prevent the screen saver from running. I
have also developed a small dimmer application that is executed 20
seconds before the locker to give me a chance to move the mouse if I
am not away.4 Have a look at my configuration
script.
Demonstration of xss-lock, xss-dimmer and i3lock with a 4 speedup.
The remaining components
autorandr is a tool to detect the connected display, match them
against a set of profiles, and configure them with xrandr.
inputplug executes a script for each new mouse and keyboard
plugged. This is quite useful to load the appropriate the keyboard
map. See my configuration.
xsettingsd provides settings to X11 applications, not unlike
xrdb but it notifies applications for changes. The main use is
to configure the Gtk and DPI settings. See my article on HiDPI
support on Linux with X11.
Redshift adjusts the color temperature of the screen according
to the time of day.
maim is a utility to take screenshots. I use Prt Scn
to trigger a screenshot of a window or a specific area and
Mod+Prt Scn to capture the whole desktop to a
file. Check the helper script for details.
I have a collection of wallpapers I rotate every hour. A
script selects them using advanced machine learning
algorithms and stitches them together on multi-screen setups. The
selected wallpaper is reused by i3lock.
Apart from the eye candy, a compositor also helps to get
tear-free video playbacks.
My configuration works with both Haswell (2014) and Whiskey
Lake (2018) Intel GPUs. It also works with AMD GPU based on the
Polaris chipset (2017).
You cannot manage two different displays this way e.g.
:0 and :1. In the first implementation, I did try to
parametrize each service with the associated display, but this is
useless: there is only one DBus user session and many services
rely on it. For example, you cannot run two notification daemons.
I have only discovered later that XSecureLock ships
such a dimmer with a similar implementation. But mine has a cool
countdown!
It's major upgrade time again! The Debian project just published
the Debian 11 "bullseye" release, and it's pretty awesome! This
makes me realized that I have never written here about my peculiar
upgrade process, and figured it was worth
bringing that up to a wider audience.
My upgrade process also has a
notable changes section
which includes major version changes (e.g. Inkscape 1.0!), new
packages (e.g. podman!) and important behavior changes
(e.g. driverless scanning and printing!).
I'm particularly interested to hear about any significant change I
might have missed. If you know of a cool new package that shipped with
bullseye and that I forgot, do let me know!
But that's for the cool new stuff. We need to talk about the problems
with Debian major upgrades.
Background
I have been maintaining detailed upgrade guides,
on my wiki, starting with the jessie
release, but I have actually written such
guides for Koumbit.org as far back as Debian squeeze in 2011
(another worker wrote the older Debian lenny upgrade guide in
2009). Koumbit, since then, has kept maintaining those guides all the
way to the latest bullseye upgrade, through 7 major releases!
Over the years, those guides evolved from a quick "cheat-sheet" format
copied from the release notes into a more or less "scripted" form
that I currently use.
Each guide has a procedure made of a few steps that can be basically
copy-pasted to batch-upgrade a host (or multiple hosts in parallel) as
quickly as possible. There is also the predict-os script which
allows you to keep track of progress of the upgrades in a Puppet cluster.
Limitations of the official procedure
In comparison with my procedure, the official upgrade guide is
mostly designed to upgrade a single machine, typically a workstation,
with a rather slow and exhaustive process. The PDF version of the
upgrade guide is 14 pages long! This, obviously, does not work
when you have tens or hundreds of machines to upgrade.
Debian upgrades are notorious for being extremely reliable, but we
have a lot of packages, and there are always corner cases where
the upgrade will just fail because of a bug specific to your
environment. Those will only be fixed after some back and forth in the
community (and that's assuming users report those bugs, which is not
always the case). There's no obvious way to deploy "hot fixes" in this
context, at least not without fixing the package and publishing it
on an unofficial Debian archive while the official ones catch up. This
is slow and difficult.
Or some packages require manual labor. Examples of this are the
PostgreSQL or Ganeti packages which require you to upgrade your
clusters by hand, while the old and new packages live side by
side. Debian packages bring you far in the upgrade process, but
sometimes not all the way.
Which means every Debian install needs to be manually upgraded and
inspected when a new release comes out. That's slow and error prone
and we can do better.
How to automate major upgrades
I have a proposal to automate this. It's been mostly dormant in
the Debian wiki, for 5 years now. Fundamentally, this is a hard
problem: Debian gets installed in so many different environments, from
workstations to physical servers to virtual machines, embedded systems
and so on, that it's extremely hard to come up with a "one size fits
all" system.
The (manual) procedure I'm using is mostly targeting servers, but I'm
also using it on workstations. And I'll note that it's specific to my
home setup: I have a different procedure at work, although it has
a lot of common code.
To automate this, I would factor out that common code with hooks where
you could easily inject special code like "you need to upgrade ferm
first", "you need an extra reboot here", or "this is how you finish
the PostgreSQL upgrade".
With Debian getting closer to a 2 year release cycle, with the
previous release being supported basically only one year after the
new stable comes out, I feel more and more strongly that this needs
better automation.
So I'm thinking that I should write a prototype for this. Ubuntu has
do-release-upgrade that is too Ubuntu-specific to be reused. An
attempt at collaborating on this has been mostly met with silence
from Ubuntu's side as well.
I'm thinking that using something like Fabric, Mitogen, or
Transilience: anything that will allow me to write simple,
portable Python code that can run transparently on a local machine
(for single systems upgrades, possibly with a GUI frontend) to remote
servers (for large clusters of servers, maybe with canaries and
grouping using Cumin). I'll note that Koumbit started
experimenting with Puppet Bolt in the bullseye upgrade process,
but that feels too site-specific to be useful more broadly.
Trade-offs
I am not sure where this stands in the XKCD time trade-off
evaluation, because the table doesn't actually cover the time
frequency of Debian release (which is basically "biennial") and the
amount of time the upgrade would take across a cluster (which varies a
lot, but that I estimate to be between one to 6 hours per machine).
Assuming I have 80 machines to upgrade, that is 80 to 480 hours
(between ~3 to 20 days) of work! It's unclear how much work such an
automated system would shave off, however. Assuming things are an
order of magnitude faster (say I upgrade 10 machines at a time), I
would shave off between 3 and 18 days of work, which implies I might
allow myself to spend a minimum of 5 days working on such a
project.
The other option: never upgrade
Before people mention those: I am aware of containers, Kubernetes, and
other deployment mechanisms. Indeed, those may be a long-term
solution, we currently can't afford to migrate everything over to
containers right now: that is a huge migration and a total paradigm
shift. At that point, whatever is left might not even be Debian in the
first place. And besides, if you run Kubernetes, you still need to run
some OS underneath and upgrade that, so that problem never
completely disappears.
Still, maybe that's the final answer: never upgrade.
For some stateless machines like DNS replicas or load balancers, that
might make a lot of sense as there's no or little data to carry to the
new host. But this implies a seamless and fast provisioning process,
and we don't have that either: at my work, installing a machine takes
about as long as upgrading it, and that's after a significant amount
of work automating that process, partly writing my own Debian
installer with Fabric (!).
What is your process?
I'm curious to hear what people think of those ideas. It strikes me as
really odd that no one has really tackled that problem yet,
considering how many clusters of Debian machines are out there. Surely
people are upgrading those, and not following that slow step by step
guide, right?
I suspect everyone is doing the same thing: we all have our little
copy-paste script we batch onto multiple machines, sometimes in
parallel. That is what the Debian.org sysadmins are doing as well.
There must be a better way. What is yours?
My upgrades so far
So far, I have upgraded 2 out of my 3 home machines running buster --
others have been installed directly in bullseye -- with only my main,
old, messy server left. Upgrades have been pretty painless so far (see
another report, for example), much better than the previous
buster upgrade. Obviously, for me personal
use, automating this is pointless.
Work-side, however, is another story: we have over 80 boxes to
upgrade there and that will take a while. The last stretch to buster
cycle took about two years to complete, so we might be done by the
time the next release (12, "bookworm") is released, but that's
actually a full year after "buster" becomes EOL, so it's actually
too late...
At least I fixed the installers so that new the machines we create all
ship with bullseye, so we stopped accumulating new buster hosts...
Thanks to lelutin and pabs for reviewing a draft of this post.
Welcome to the latest report from the Reproducible Builds project. In this post, we round up the important things that happened in the world of reproducible builds in August 2021. As always, if you are interested in contributing to the project, please visit the Contribute page on our website.
There were a large number of talks related to reproducible builds at DebConf21 this year, the 21st annual conference of the Debian Linux distribution (full schedule):
Secondly, Vagrant Cascadian gave a talk titled Looking Forward to Reproducible Builds which mentions some historic blockers within Debian that have been solved, worked around or which are still in progress. It also touched on some recent developments, with an eye to what might happen as Debian embarks upon the new Bookworm development cycle after the release of Bullseye.
Lastly, Johannes Schauer Marin Rodrigues & Fr d ric Pierret gave a joint talk on Making use of snapshot.debian.org for fun and profit about various tools they have developed to interact with the snapshot.debian.org wayback machine for Debian packages. In particular, they mention how they are using the service to reproduce and validate builds as well as touch on an alternative snapshot service that has been mentioned in previous reports.
PackagingCon (@PackagingCon) is new conference for developers of package management software as well as their related communities and stakeholders. The virtual event, which is scheduled to take place on the 9th and 10th November 2021, has a mission is to bring different ecosystems together: from Python s pip to Rust s cargo to Julia s Pkg, from Debian apt over Nix to conda and mamba, and from vcpkg to Spack we hope to have many different approaches to package management at the conference . A number of people from reproducible builds community are planning on attending this new conference, and some may even present. Tickets start at $20 USD.
As reported in our May report, the president of the United States signed an executive order outlining policies aimed to improve the cybersecurity in the US. The executive order comes after a number of highly-publicised security problems such as a ransomware attack that affected an oil pipeline between Texas and New York and the SolarWinds hack that affected a large number of US federal agencies. As a followup this month, however, a detailed fact sheet was released announcing a number large-scale initiatives and that will undoubtedly be related to software supply chain security and, as a result, reproducible builds.
Lastly, We ran another productive meeting on IRC in August (original announcement) which ran for just short of two hours. A full set of notes from the meeting is available.
Software development
kpcyrd announced an interesting new project this month called I probably didn t backdoor this which is an attempt to be:
a practical attempt at shipping a program and having reasonably solid evidence there s probably no backdoor. All source code is annotated and there are instructions explaining how to use reproducible builds to rebuild the artifacts distributed in this repository from source.
The idea is shifting the burden of proof from you need to prove there s a backdoor to we need to prove there s probably no backdoor . This repository is less about code (we re going to try to keep code at a minimum actually) and instead contains technical writing that explains why these controls are effective and how to verify them. You are very welcome to adopt the techniques used here in your projects. ()
As the project s README goes on the mention: the techniques used to rebuild the binary artifacts are only possible because the builds for this project are reproducible . This was also announced on our mailing list this month in a thread titled i-probably-didnt-backdoor-this: Reproducible Builds for upstreams.
kpcyrd also wrote a detailed blog post about the problems surrounding Linux distributions (such as Alpine and Arch Linux) that distribute compiled Python bytecode in the form of .pyc files generated during the build process.
diffoscopediffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb made a number of changes, including releasing version 180), version 181) and version 182) as well as the following changes:
New features:
Add support for extracting the signing block from Android APKs. []
If we specify a suffix for a temporary file or directory within the code, ensure it starts with an underscore (ie. _ ) to make the generated filenames more human-readable. []
Don t include short GCC lines that differ on a single prefix byte either. These are distracting, not very useful and are simply the strings(1) command s idea of the build ID, which is displayed elsewhere in the diff. [][]
Don t include specific .debug-like lines in the ELF-related output, as it is invariably a duplicate of the debug ID that exists better in the readelf(1) differences for this file. []
Bug fixes:
Add a special case to SquashFS image extraction to not fail if we aren t the superuser. []
Only use java -jar /path/to/apksigner.jar if we have an apksigner.jar as newer versions of apksigner in Debian use a shell wrapper script which will be rejected if passed directly to the JVM. []
Reduce the maximum line length for calculating Wagner-Fischer, improving the speed of output generation a lot. []
Don t require apksigner in order to compare .apk files using apktool. []
Update calls (and tests) for the new version of odt2txt. []
Output improvements:
Mention in the output if the apksigner tool is missing. []
Profile diffoscope.diff.linediff and specialize. [][]
Logging improvements:
Format debug-level messages related to ELF sections using the diffoscope.utils.format_class. []
Print the size of generated reports in the logs (if possible). []
Include profiling information in --debug output if --profile is not set. []
Codebase improvements:
Clarify a comment about the HUGE_TOOLS Python dictionary. []
We can pass -f to apktool to avoid creating a strangely-named subdirectory. []
Drop an unused File import. []
Update the supported & minimum version of Black. []
We don t use the logging variable in a specific place, so alias it to an underscore (ie. _ ) instead. []
Update some various copyright years. []
Clarify a comment. []
Test improvements:
Update a test to check specific contents of SquashFS listings, otherwise it fails depending on the test systems user ID to username passwd(5) mapping. []
Assign seen and expected values to local variables to improve contextual information in failed tests. []
Don t print an orphan newline when the source code formatting test passes. []
In addition Santiago Torres Arias added support for Squashfs version 4.5 [] and Felix C. Stegerman suggested a number of small improvements to the output of the new APK signing block []. Lastly, Chris Lamb uploaded python-libarchive-c version 3.1-1 to Debian experimental for the new 3.x branch python-libarchive-c is used by diffoscope.
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Update the website self-build process to not use the buster-backports suite now that Debian Bullseye is the stable release. []
Holger Levsen:
Add a new page documenting various package rebuilder solutions. []
Add some historical talks and slides from DebConf20. [][]
Various improvements to the history page. [][][]
Rename the Comparison protocol documentation category to Verification . []
Update links to F-Droid documentation. []
Ian Muchina:
Increase the font size of titles and de-emphasize event details on the talk page. []
Rename the README file to README.md to improve the user experience when browsing the Git repository in a web browser. []
Mattia Rizzolo:
Drop a position:fixed CSS statement that is negatively affecting with some width settings. []
Fix the sizing of the elements inside the side navigation bar. []
Show gold level sponsors and above in the sidebar. []
Updated the documentation within reprotest to mention how ldconfig conflicts with the kernel variation. []
Roland Clobus:
Added a ticket number for the issue with the live Cinnamon image and diffoscope. []
Testing framework
The Reproducible Builds project runs a testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:
Make a large number of changes to support the new Debian bookworm release, including adding it to the dashboard [], start scheduling tests [], adding suitable Apache redirects [] etc. [][][][][]
Make the first build use LANG=C.UTF-8 to match the official Debian build servers. []
Only test Debian Live images once a week. []
Upgrade all nodes to use Debian Bullseye [] []
Update README documentation for the Debian Bullseye release. []
Other changes:
Only include rsync output if the $DEBUG variable is enabled. []
Don t try to install mock, a tool used to build Fedora packages some time ago. []
Drop an unused function. []
Various documentation improvements. [][]
Improve the node health check to detect zombie jobs. []
Update the location and branch name for the main FreeBSD Git repository. []
Correctly ignore the source tarball when comparing build results. []
Drop an outdated version number from the documentation. []
Mattia Rizzolo:
Block F-Droid jobs from running whilst the setup is running. []
Enable debugging for the rsync job related to Debian Live images. []
Pass BUILD_TAG and BUILD_URL environment for the Debian Live jobs. []
Refactor the master_wrapper script to use a Bash array for the parameters. []
Prefer YAML s safe_load() function over the unsafe variant. []
Use the correct variable in the Apache config to match possible existing files on disk. []
Stop issuing HTTP 301 redirects for things that not actually permanent. []
Roland Clobus (Debian live image generation):
Increase the diffoscope timeout from 120 to 240 minutes; the Cinnamon image should now be able to finish. []
Use the new snapshot service. []
Make a number of improvements to artifact handling, such as moving the artifacts to the Jenkins host [] and correctly cleaning them up at the right time. [][][]
Where possible, link to the Jenkins build URL that created the artifacts. [][]
Only allow only one job to run at the same time. []
Vagrant Cascadian:
Temporarily disable armhf nodes for DebConf21. [][]
Lastly, if you are interested in contributing to the Reproducible Builds project, please visit the Contribute page on our website. You can get in touch with us via:
I have been Debian Stretch running on my HP Pavilion 14-ce0000nq laptop since buying it back in April 2019, just before my presence at Oxidizeconf where I presented "How to Rust When Standards Are Defined in C".Debian Buster (aka Debian 10) was released about 4 months later and I've been postponing the upgrade as my free time isn't what it used to be. I also tend to wait for the first or even second update of the release to avoid any sharp edges.As this laptop has a Realtek 8821CE wireless card that wasn't officially supported in the Linux kernel, I had to use an out-of-tree hacked driver to have the wireless work on Stretch kernels such as 4.19, it didn't even got along with DKMS, so all compilations and installations of it, I did them manually. More reason to wait for a newer release that would contain a driver inside the official kernel.I was waiting for the inevitable and dreading the wireless issues, but since mid-august Bullseye became stable, turning Stretch into oldoldstable, I decided that I had to do the upgrade, at least to buster.The Grub error and the fix Everything went quite smooth, except that after the reboot, the laptop failed to boot with this Grub error:
error: symbol grub_is_lockdown not found
I looked for a solution and it seemed everyone was stuckor the solution was unclear.There is even a bug report in Debian about this error, bug #984760. Adding to the pile of confusion my own confused solution: I tried supergrubdisk2/rescatux, it didn't work for me, it might have been a combination of me using LVM and grub-efi-amd64. I also tried to boot in rescue mode the Buster first DVD (to avoid the need for network), I was able to enter the partition, mount the EFI partition, too, but since I didn't want to mess the setup even more or depend on an external USB stick, I didn't know where should I try to write the Grub EFI config - the root partition is on an NVME storage.When buying the laptop it had FreeDOS installed on it and some HP rescue app, which I did not wipe when installing Debian. I even forgot where or how was the EFI installed on the disk and EFI, even if it should be more reliable and simpler, I never got the hang of it. In the end, I realized that I could via BIOS actually select manually which EFI executable should be booted into, so I was able to boot with some manual intervention during boot into the regular system.I tried regenerating the grub configuration, installing it and also tried restoring the default proper boot sequence (and I even installed refind in the system during my fumbling), but I think somewhere between grub-efi-amd64 reconfiguration and its reinstallation I managed to do the right thing, as the default boot screen is the Grub one now.Hints for anyone reading this in the hope to fix the same issue, hopefully it will make things better, not worse (see the text below):1) regenerate the grub config:
update-grub2
2) reinstall grub-efi-amd64 and make Debian the default
dpkg-reconfigure -plow grub-efi-amd64
When reinstalling grub-efi-amd64 onto the disk, I think the scariest questions were to these:
Force extra installation to the EFI removable media path?
Some EFI-based systems are buggy and do not handle new bootloaders correctly. If you force an extra installation of GRUB to the EFI removable media path, this should ensure that this system will boot Debian correctly despite such a problem. However, it may remove the ability to boot any other operating systems that also depend on this path. If so, you will need to make sure that GRUB is configured successfully to be able to boot any other OS installations correctly.
and
Update NVRAM variables to automatically boot into Debian?
GRUB can configure your platform's NVRAM variables so that it boots into Debian automatically when powered on. However, you may prefer to disable this behavior and avoid changes to your boot configuration. For example, if your NVRAM variables have been set up such that your system contacts a PXE server on every boot, this would preserve that behavior.
I think the first can be safely answered "No" if you don't plan on booting via a removable USB stick, while the second is the one that does the restoring.The second question is probably safe if you don't use PXE boot or other boot method, at least that's what I understand. But if you do, I suspect by installing refind, by playing with the multiple efi* named packages and tools, you can restore that, or it might be that your BIOS allows that directly. I just did a walk through of these 2 steps again on my laptop and answered "No" to the removable media question as it leads to errors when the media was not inserted (in my case the internal SD card reader), and "Yes" to making Debian the default.It seems that for me this broke the FreeDOS and HP utilities boot entries from Grub, but I still can boot via the BIOS options and my goal was to have Debian boot correctly by default. Fixing the missing RTL811/8168/8411 Ethernet card issue As a side note for people with computers having Realtek RTL8111/8168/8411 Gigabit Ethernet Controller and upgrading to Buster or switching to a newer kernel, please note that you might end up having the unpleasant surprise even your Ethernet card to disappear because the r8169 driver is not loader by default.I had to add it to /etc/modules so is loaded by default:
eddy@aptonia:/ $ cat /etc/modules # /etc/modules: kernel modules to load at boot time. # # This file contains the names of kernel modules that should be loaded # at boot time, one per line. Lines beginning with "#" are ignored. r8169
The 5.10 compatible driver for RTL8821CE wireless adapterAfter the upgrade to Buster, the oldstable version of the kernel, 4.19, the hacked version of the driver I've been using on Stretch on 4.9 kernels was no longer compatible - failed to compile due to missing symbols.The fix for me was to switch to the DKMS compatible driver from https://github.com/tomaspinho/rtl8821ce, as this seems to work for both 4.19 and 5.10 kernels (installed from backports). I installed it via a modification of the manual install method only for the 4.19 and 5.10 kernels, leaving the legacy 4.9 kernels working with the hacked driver. You can do the same if instead of running the provided script, you do its steps manually and you install only for the kernel versions you want, instead of the default to install for all:I looked inside the dkms-install.sh script to do the required steps:Copy the driver, add it to the dkms set of known drivers:
dkms status should confirm everything is in place and I think you need to update grub2 again after this. Please note this driver is no longer maintained and the 5.10 tree should support the RTL8821CE wireless card with the rtw88 driver from the kernel, but for me it did not. I'll probably try this at a later time, or after I upgrade to the current Debian stable, Bullseye.
I'm guessing the last glasses will be through the dishwasher (again) and Pepper the dog can settle down without having to cope with so many people.For those who don't know - Steve and his wife Jo (Sledge and Randombird) hold a barbeque in their garden every August Bank Holiday weekend [UK Bank Holiday on the last Monday in August]. The barbeque is not small - it's the dominating feature in the suburban garden, brick built, with a dedication stone, lights, electricity. The garden is small, generally made smaller by forty or so Debian friends and allies standing and sitting around. People are talking, arguing, hugging people they've not seen for (literal) years and putting the world to rights. This is Debian central point - with large quantities of meat and salads, an amount of beer/alcohol and "Cambridge gin" and general goodwill. This year was more than usually atmospheric because for some of us it was the first time with a large group of people in a while. Side conversations abound: for me it was learning something about the high energy particle physics community, how to precision build helicopters, fly quadcopters and precision 3D print anything, the maths of Isy counting crochet stitches to sew together randomly sized squares ... and, of course, obligatory things like how random is random and what's good enough entropy. And a few sessions of the game of our leader. This is also a place for stuff to get done: I was unashamedly using this to upgrade the storage in my laptop while there were sensible engineers around. A corner of the table, a RattusRattus and it was quickly sorted - then a discussion around the internals of Thinkpads as he took his apart. Then getting a full install - Gb Ethernet to the Debian mirror in the cupboard six feet away is faster bandwidth than a jumbo jet full of tapes. Then getting mail to work again - it's handy when the mailserver owner is next to you, having come in from the garden to help, and finally IRC. And not just me: "You need a GPG key signed - there's three DPLs here, there's a release manager - but you've just missed one of the DAMs." plus an in-depth GPG how-to session on the other side of the table.I was the luckiest one with the most comfortable bed in the house overnight but I couldn't stay for last night. Thanks once again to all involved but especially Steve and Jo who do this for the love of it, and the fun, and the community and the family. Oh, and thanks to Lenovo - not just for being a platinum sponsor of Debconf but also for providing the official laptop of this and most Debian occasions
Aug 28, 2021, I gave a short talk about "Latest topics about fabre.debian.net" at DebConf21.
You can see the presentation slide here:
slide.rabbit-shocker.org
Short talk explanation is here:
debconf21.debconf.org
In this talk, I've explained mainly 3 topics
Domain migration to debian.net subdomain - fabre.debian.net
I've sent a pre-recorded video in advance, it is ok, but there is trouble with the mic during the session. :-(
The recorded video will be available on the talk detail page soon.
CVE-2020-22036: A heap-based Buffer Overflow vulnerability in filter_intra at libavfilter/vf_bwdif.c, which might lead
to memory corruption and other potential consequences.
CVE-2020-22032: A heap-based Buffer Overflow vulnerability in
gaussian_blur, which might lead to memory corruption and other
potential consequences.
CVE-2020-22031: A Heap-based Buffer Overflow vulnerability in
filter16_complex_low, which might lead to memory corruption and
other potential consequences.
CVE-2020-22028: Buffer Overflow vulnerability
in filter_vertically_8 at libavfilter/vf_avgblur.c, which could cause a remote
Denial of Service.
CVE-2020-22026: Buffer Overflow vulnerability exists in the config_input function at libavfilter/af_tremolo.c, which could let a remote malicious
user cause a Denial of Service.
CVE-2020-22025: A heap-based Buffer Overflow vulnerability
exists in gaussian_blur at libavfilter/vf_edgedetect.c, which might lead to memory
corruption and other potential consequences.
CVE-2020-22023: A heap-based Buffer Overflow vulnerabililty
exists in filter_frame at libavfilter/vf_bitplanenoise.c, which might lead to
memory corruption and other potential consequences.
CVE-2020-22022: A heap-based Buffer Overflow vulnerability exists in
filter_frame at libavfilter/vf_fieldorder.c, which might lead to memory
corruption and other potential consequences.
CVE-2020-22021: Buffer Overflow vulnerability at filter_edges function in libavfilter/vf_yadif.c, which could let a remote malicious user cause a
Denial of Service.
CVE-2020-22020: Buffer Overflow vulnerability in the build_diff_map function
in libavfilter/vf_fieldmatch.c, which could let a remote malicious user
cause a Denial of Service.
CVE-2020-22016: A heap-based Buffer Overflow vulnerability
at libavcodec/get_bits.h when writing .mov files, which might lead to memory corruption and other potential consequences.
CVE-2020-22015: Buffer Overflow vulnerability in mov_write_video_tag due to the out of bounds in libavformat/movenc.c, which could let a remote
malicious user obtain sensitive information, cause a Denial of Service, or
execute arbitrary code.
CVE-2020-21041: Buffer Overflow vulnerability exists via
apng_do_inverse_blend in libavcodec/pngenc.c, which could let a remote
malicious user cause a Denial of Service
CVE-2021-3566: The tty demuxer did not have a read_probe function
assigned to it. By crafting a legitimate ffconcat file that references an
image, followed by a file the triggers the tty demuxer, the contents of the
second file will be copied into the output file verbatim (as long as the
-vcodec copy option is passed to ffmpeg).
CVE-2021-38114: libavcodec/dnxhddec.c does not check the return value of the init_vlc function. Crafted DNxHD data can cause unspecified impact.
DLA 2742-2 ffmpeg_7:3.2.15-0+deb9u4
During the backporting of one of patches in CVE-2020-22021 one line was wrongly
interpreted and it caused the regression during the deinterlacing process.
Thanks to Jari Ruusu for the reporting the issue and for the testing of
prepared update.
Other LTS-related work
Analyzed CVE-2020-22027 and
marked as ignored for stretch. Original patch is not appliable. The issue is reproducible though.
DebConf21 is taking place online, from 24 August to 28 August 2021.
It is the 22nd Debian conference, and organizers and participants are working hard
together at creating interesting and fruitful events.
We would like to warmly welcome the 19 sponsors of DebConf21, and introduce them to you.
We have five Platinum sponsors.
Our first Platinum sponsor is Lenovo.
As a global technology leader manufacturing a wide portfolio of connected products,
including smartphones, tablets, PCs and workstations as well as AR/VR devices,
smart home/office and data center solutions, Lenovo
understands how critical open systems and platforms are to a connected world.
Our next Platinum sponsor is Infomaniak.
Infomaniak is Switzerland's largest web-hosting company,
also offering backup and storage services, solutions for event organizers,
live-streaming and video on demand services.
It wholly owns its datacenters and all elements critical
to the functioning of the services and products provided by the company
(both software and hardware).
Roche is our third Platinum sponsor.
Roche is a major international pharmaceutical provider and research company
dedicated to personalized healthcare. More than 100,000 employees worldwide
work towards solving some of the greatest challenges for humanity using
science and technology. Roche is strongly involved in publicly funded
collaborative research projects with other industrial and academic partners
and has supported DebConf since 2017.
Amazon Web Services (AWS) is our fourth Platinum
sponsor.
Amazon Web Services is one of the world's
most comprehensive and broadly adopted cloud platform,
offering over 175 fully featured services from data centers globally
(in 77 Availability Zones within 24 geographic regions).
AWS customers include the fastest-growing startups, largest enterprises
and leading government agencies.
Google is our fifth Platinum sponsor.
Google is one of the largest technology companies in the
world, providing a wide range of Internet-related services and products such
as online advertising technologies, search, cloud computing, software, and hardware.
Google has been supporting Debian by sponsoring DebConf for more than
ten years, and is also a Debian partner sponsoring parts
of Salsa's continuous integration infrastructure
within Google Cloud Platform.
Our Gold sponsor is the Matanel Foundation.
The Matanel Foundation operates in Israel,
as its first concern is to preserve the cohesion of a society and a nation
plagued by divisions. The Matanel Foundation also works in Europe, in Africa
and in South America.
Our Silver sponsors are:
arm: the World s Best SoC Design Portfolio,
Arm powered solutions have been supporting innovation for more than 30 years
and are deployed in over 160 billion chips to date,
Hudson-Trading,
a company researching and developing automated trading algorithms
using advanced mathematical techniques,
Ubuntu
the Operating System delivered by Canonical,
Globo,
the largest media conglomerate in Brazil, founded in Rio de Janeiro in 1925
and distributing high-quality content across multiple platforms,
Two Sigma,
rigorous inquiry, data analysis, and invention to help solve the toughest
challenges across financial services
and
GitLab,
an open source end-to-end software development platform with built-in version
control, issue tracking, code review, CI/CD, and more.
Bronze sponsors:
Univention,
Gandi.net,
daskeyboard,
InterFace AG and
credativ.
And finally, our Supporter level sponsor, ISG.EE.
Thanks to all our sponsors for their support!
Their contributions make it possible for a large number
of Debian contributors from all over the globe to work together,
help and learn from each other in DebConf21.
Participating in DebConf21 online
The 22nd Debian Conference is being held online, due to COVID-19,
from August 24 to 28, 2021. Talks, discussions, panels and other activities run from 12:00 to 02:00 UTC.
Visit the DebConf21 website at https://debconf21.debconf.org
to learn about the complete schedule, watch the live streaming and join the different
communication channels for participating in the conference.
Like each month, have a look at the work funded by Freexian s Debian LTS offering.
Debian project funding
In July, we put aside 2400 EUR to fund Debian projects. We haven t received proposals of projects to fund in the last months, so we have scheduled a discussion during Debconf to try to to figure out why that is and how we can fix that. Join us on August 26th at 16:00 UTC on this link.
We are pleased to announce that Jeremiah Foster will help out to make this initiative a success : he can help Debian members to come up with solid proposals, he can look for people willing to do the work once the project has been formalized and approved, and he will make sure that the project implementation keeps on track when the actual work has begun.
We re looking forward to receive more projects from various Debian teams! Learn more about the rationale behind this initiative in this article.
Debian LTS contributors
In July, 12 contributors have been paid to work on Debian LTS, their reports are available:
Abhijith PA did 5.0h (out of 7h assigned and 3h remaining), thus carrying over 5h to August.
Emilio Pozuelo Monfort did not report back about their work so we assume they did nothing (out of 39.75h assigned plus 11h from June), thus is carrying over 50.75h for August.
Holger Levsen s work was coordinating/managing the LTS team, he did 3.5h (out of 12h assigned) and gave back 8.5h to the pool.
Markus Koschany did 30h (out of 30h assigned and 30h from June), thus carrying over 30h to August.
Ola Lundqvist did nothing (out of 12h assigned plus 20h from June), thus is carrying over 32h for August.
Roberto C. S nchez did 13.5h (out of 32h assigned and 20h from June), and gave back 38.5h to the pool.
Evolution of the situation
In July we released 30 DLAs. Also we were glad to welcome Neil Williams and Lee Garrett who became active contributors.
The security tracker currently lists 63 packages with a known CVE and the dla-needed.txt file has 17 packages needing an update.
We would like to thank Holger Levsen for the years of work where he managed/coordinated the paid LTS contributors. Jeremiah Foster will take over his duties.
Thanks to our sponsors
Sponsors that joined recently are in bold.
Three years ago, I organized a fun and most interesting colloquium at
I would have loved to share this earlier with the world, but The
university s processes are quite slow (and, to be fair, I also took
quite a bit of time to push things through). But today, I m finally
happy to share the result of that work with all of you. We managed to
get 11 of the talks in the colloquium as articles. The back-cover text
reads (in Spanish):
Colson Whitehead's latest novel, 
This is the fifth and final post in a 
I have been Debian Stretch running on my 
Aug 28, 2021, I gave a short 
DebConf21 is taking place online, from 24 August to 28 August 2021.
It is the 22nd Debian conference, and organizers and participants are working hard
together at creating interesting and fruitful events.
We would like to warmly welcome the 19 sponsors of DebConf21, and introduce them to you.
We have five Platinum sponsors.
Our first Platinum sponsor is 
