Shadow Scale, despite confusing publisher marketing that calls it a
"companion" to Seraphina, is a direct
sequel. It picks up shortly after Seraphina and resolves most of
the loose ends of the previous book.
This is a book for which my completionist tendencies did me no favors.
The book I was intending to read, when I started on Hartman's work, is
Tess of the Road, but I hate starting series in the middle and it
was clear that Tess was set after Seraphina. (I have been
repeatedly assured that this doesn't matter and that one can start with
Tess. Such reassurances rarely work on me; do as I say, not as I
do.) For Seraphina itself, this turned out fine; I'm mildly
surprised by the book's Andre Norton award nomination, but it was
enjoyable enough and I liked the first-person protagonist.
Shadow Scale I approached with a bit more trepidation. I hadn't
heard much about it and the few reviews I saw were lukewarm.
Unfortunately, there's a reason for that.
Seraphina left obvious room for a sequel, including a brewing war,
significant unresolved interpersonal relationships, and Seraphina's own
newfound understanding of the nature of her internal menagerie. Alas, the
start of the book uses the war primarily as plot device (and introduces a
brand-new bit of magic that I never found interesting), largely ignores
the relationship, and focuses on that third plot element. And by focuses,
I mean Seraphina is sent out of the country of Goredd on a journey of map
exploration to collect plot coupons.
The best description I have for the middle of this book is tedious and
depressing. Like a lot of novels, it has a U-shaped plot: things get
worse and worse until a crisis, and then start getting better. This plot
can work, but the reader has to have a good reason to stick through the
depressing bits. One of the better reasons is if the plot allows the main
character some small triumphs, maintaining their agency throughout even if
larger events are spiraling out of control. This is not one of those
books. After some early successes tracking down some objects of her
search, Seraphina encounters an antagonist from her own past (barely
hinted at in the first book) who can systematically corrupt everything she
is trying to do. She spends most of the book feeling like what she's
doing is futile, or hoping for things the reader knows aren't going to
happen. Given that this is happening during plodding map exploration
fantasy through largely indistinguishable faux-medieval countries, or
(later) somewhat more interesting but obviously irrelevant local politics
in a remote trading city, it's hard to avoid sharing that sense of
futility.
The other structural problem with Shadow Scale is that the plot
coupons are people, which means this book has an excessive cast size
problem. Seraphina collects too many people for me to even keep straight,
let alone care about. Critical developments (usually for the worst) in
the lives of one of these characters were frequently met with reader
mutterings like, "Now which one was Brasidas again, was he the plague
doctor?" This tends to undermine the emotional impact. It didn't help
that the plot was enough of a slog that I kept putting the book aside for
a few days.
This does get better, but not enough better to redeem the middle of the
book, and one has to put up with a lot of helpless despair to get there.
Shadow Scale is one of those stories where the protagonist has the
innate power to resolve the plot, is told cryptically by various people
that this is the case, but has absolutely no idea how to use it and her
supposed mentors are essentially useless. The result is that she feels
both hopeless and guilty, which was not the reading experience I wanted.
I did enjoy the moment when she finally figures it out, and I thought
Hartman's idea was reasonably clever, but it would have been better if
that had happened faster. Like, 200 pages faster. At least.
The major world-building in Seraphina was the dragons. The dragons
also show up in this book (and feel less like autism spectrum archetypes,
which I appreciated), and in theory are central to the plot, but I'm not
entirely sure why? It was an odd reading experience. I think Hartman was
attempting to set up dual villains posing different threats, but the
dragon one is off-screen for nearly the entire book and never developed,
so it feels perfunctory. Near the end of the book, Hartman abruptly picks
up the dragons again, but that whole section felt oddly disconnected from
the rest of the plot and is only barely relevant to the resolution. At
least for me, the plot structure didn't cohere.
Shadow Scale does go up a whole point in rating for me because of
the romance plot and how Hartman resolves it, which I will not spoil but
which I loved. The process of getting there is immensely frustrating
because it feels like Hartman is forcing the characters into a corner
where only stupid resolutions are possible, but in this case the U-shaped
emotional structure worked on me. The ending is completely true to the
characters in a way that I thought Hartman had made impossible (and which
does a lovely bit of undermining of traditional roles), so full credit
there. It helped that the relationship is put on ice for most of the book
and only appears at the end (which is also the best part of the book), so
it didn't drag on like the other parts of the plot.
Overall, though, I tentatively agree with the general advice to skip this
one, and suspect that advice will become less tentative once I read
Tess of the Road. It's a largely unpleasant slog. There are some
mildly interesting world-building revelations that fill in the background
of Seraphina, the ending was reasonably good, and the relationships
were much better than I was expecting through most of the book, but the
amount of time and patience required to get there was not a good trade-off
for me.
Followed (in the sense that it's set in the same universe but is not a
sequel and I suspect does not depend heavily on this plot) by Tess
of the Road.
Rating: 4 out of 10
A bug-fix release of the dang package arrived at CRAN a little while ago. The dang package regroups a few functions of mine that had no other home as for example lsos() from a StackOverflow question from 2009 (!!), the overbought/oversold price band plotter from an older blog post, the market monitor from the last release as well the checkCRANStatus() function tweeted about by Tim Taylor.
This release corrects a small mistake wrapping extern "C" ... around code that is not actually C but C++ which g++ kept silent about yet clang++ complains about, correctly. So CRAN asked me to correct this, which this version does.
The NEWS entry follows.
Changes in version 0.0.15 (2021-10-26)
Corrected scope of 'extern "C"' declaration to the actually C-callable function (noticed by clang++, change requested by CRAN)
I grew up with the Internet and its ethics and politics have always
been important in my life. But I have also been involved at other
levels, against police brutality, for Food, Not Bombs,
worker autonomy, software freedom, etc. For a long time,
that all seemed coherent.
But the more I look at the modern Internet -- and the
mega-corporations that control it -- and the less confidence I have in
my original political analysis of the liberating potential of
technology. I have come to believe that most of our technological
development is harmful to the large majority of the population of the
planet, and of course the rest of the biosphere. And now I feel this
is not a new problem.
This is because the Internet is a neo-colonial device, and has been
from the start. Let me explain.
What is Neo-Colonialism?
The term "neo-colonialism" was coined by Kwame Nkrumah,
first president of Ghana. In Neo-Colonialism, the Last Stage of
Imperialism (1965), he wrote:
In place of colonialism, as the main instrument of imperialism, we
have today neo-colonialism ... [which] like colonialism, is an
attempt to export the social conflicts of the capitalist
countries. ...
The result of neo-colonialism is that foreign capital is used for
the exploitation rather than for the development of the less
developed parts of the world. Investment, under neo-colonialism,
increases, rather than decreases, the gap between the rich and the
poor countries of the world.
So basically, if colonialism is Europeans bringing genocide, war,
and its religion to the Africa, Asia, and the Americas,
neo-colonialism is the Americans (note the "n") bringing capitalism to
the world.
Before we see how this applies to the Internet, we must therefore make
a detour into US history. This matters, because anyone would be
hard-pressed to decouple neo-colonialism from the empire under which
it evolves, and here we can only name the United States of America.
US Declaration of Independence
Let's start with the United States declaration of independence
(1776). Many Americans may roll their eyes at this, possibly because
that declaration is not actually part of the US constitution and
therefore may have questionable legal standing. Still, it was
obviously a driving philosophical force in the founding of the
nation. As its author, Thomas Jefferson, stated:
it was intended to be an expression of the American mind, and to
give to that expression the proper tone and spirit called for by the
occasion
In that aging document, we find the following pearl:
We hold these truths to be self-evident, that all men are created
equal, that they are endowed by their Creator with certain
unalienable Rights, that among these are Life, Liberty and the
pursuit of Happiness.
As a founding document, the Declaration still has an impact in the
sense that the above quote has been called an:
"immortal declaration", and "perhaps [the] single phrase" of the
American Revolutionary period with the greatest "continuing
importance." (Wikipedia)
Let's read that "immortal declaration" again: "all men are created
equal". "Men", in that context, is limited to a certain number of
people, namely "property-owning or tax-paying white males, or about
6% of the population". Back when this was written, women didn't
have the right to vote, and slavery was legal. Jefferson himself owned
hundreds of slaves.
The declaration was aimed at the King and was a list of
grievances. A concern of the colonists was that the King:
has excited domestic insurrections amongst us, and has endeavoured
to bring on the inhabitants of our frontiers, the merciless Indian
Savages whose known rule of warfare, is an undistinguished
destruction of all ages, sexes and conditions.
This is a clear mark of the frontier myth which paved the way for
the US to exterminate and colonize the territory some now call the
United States of America.
The declaration of independence is obviously a colonial document,
having being written by colonists. None of this is particularly
surprising, historically, but I figured it serves as a good reminder
of where the Internet is coming from, since it was born in the US.
A Declaration of the Independence of Cyberspace
Two hundred and twenty years later, in 1996, John Perry Barlow
wrote a declaration of independence of cyberspace. At this
point, (almost) everyone has a right to vote (including women),
slavery was abolished (although some argue it still exists in the
form of the prison system); the US has made tremendous
progress. Surely this text will have aged better than the previous
declaration it is obviously derived from. Let's see how it reads today
and how it maps to how the Internet is actually built now.
Borders of Independence
One of the key ideas that Barlow brings up is that "cyberspace does
not lie within your borders". In that sense, cyberspace is the final
frontier: having failed to colonize the moon, Americans turn
inwards, deeper into technology, but still in the frontier
ideology. And indeed, Barlow is one of the co-founder of the
Electronic Frontier Foundation (the beloved EFF), founded six
years prior.
But there are other problems with this idea. As Wikipedia quotes:
The declaration has been criticized for internal
inconsistencies.[9] The declaration's assertion that
'cyberspace' is a place removed from the physical world has also
been challenged by people who point to the fact that the Internet is
always linked to its underlying geography.[10]
And indeed, the Internet is definitely a physical object. First
controlled and severely restricted by "telcos" like AT&T, it was
somewhat "liberated" from that monopoly in 1982 when an anti-trust
lawsuitbroke up the monopoly, a key historical event that,
one could argue, made the Internet possible.
(From there on, "backbone" providers could start competing and emerge,
and eventually coalesce into new monopolies: Google has a monopoly on
search and advertisement, Facebook on communications for a few
generations, Amazon on storage and computing, Microsoft on hardware,
etc. Even AT&T is now pretty much as consolidated as it was
before.)
The point is: all those companies have gigantic data centers and
intercontinental cables. And those are definitely prioritizing the
western world, the heart of the empire. Take for example Google's
latest 3,900 mile undersea cable: it does not connect Argentina to
South Africa or New Zealand, it connects the US to UK and
Spain. Hardly a revolutionary prospect.
Private Internet
But back to the Declaration:
Do not think that you can build it, as though it were a public
construction project. You cannot. It is an act of nature and it
grows itself through our collective actions.
In Barlow's mind, the "public" is bad, and private is good,
natural. Or, in other words, a "public construction project" is
unnatural. And indeed, the modern "nature" of development is private:
most of the Internet is now privately owned and operated.
I must admit that, as an anarchist, I loved that sentence when I read
it. I was rooting for "us", the underdogs, the revolutionaries. And,
in a way, I still do: I am on the board of Koumbit and work for a
non-profit that has pivoted towards censorship and surveillance
evasion. Yet I cannot help but think that, as a whole, we have failed
to establish that independence and put too much trust in private
companies. It is obvious in retrospect, but it was not, 30 years
ago.
Now, the infrastructure of the Internet has zero accountability to
traditional political entities supposedly representing the people, or
even its users. The situation is actually worse than when the US was
founded (e.g. "6% of the population can vote"), because the owners of the
tech giants are only a handful of people who can override any
decision. There's only one Amazon CEO, he's called Jeff Bezos, and he
has total control. (Update: Bezos actually ceded the CEO role to Andy
Jassy, AWS and Amazon music founder, while remaining executive
chairman. I would argue that, as the founder and the richest man
on earth, he still has strong control over Amazon.)
Social Contract
Here's another claim of the Declaration:
We are forming our own Social Contract.
I remember the early days, back when "netiquette" was a word, it
did feel we had some sort of a contract. Not written in standards of
course -- or barely (see RFC1855) -- but as a tacit
agreement. How wrong we were. One just needs to look at Facebook to
see how problematic that idea is on a global network.
Facebook is the quintessential "hacker" ideology put in practice. Mark
Zuckerberg explicitly refused to be "arbiter of truth" which
implicitly means he will let lies take over its platforms.
He also sees Facebook as place where everyone is equal, something
that echoes the Declaration:
We are creating a world that all may enter without privilege or
prejudice accorded by race, economic power, military force, or
station of birth.
(We note, in passing, the omission of gender in that list, also
mirroring the infamous "All men are created equal" claim of the US
declaration.)
As the Wall Street Journal's (WSJ) Facebook files later shown,
both of those "contracts" have serious limitations inside Facebook. There are
VIPs who systematically bypass moderation systems including
fascists and rapists. Drug cartels and human traffickers
thrive on the platform. Even when Zuckerberg himself tried to
tame the platform -- to get people vaccinated or to make it
healthier -- he failed: "vaxxer" conspiracies multiplied and
Facebook got angrier.
This is because the "social contract" behind Facebook and those large
companies is a lie: their concern is profit and that means
advertising, "engagement" with the platform, which causes increased
anxiety and depression in teens, for example.
Facebook's response to this is that they are working really hard on
moderation. But the truth is that even that system is severely
skewed. The WSJ showed that Facebook has translators for only 50
languages. It's a surprisingly hard to count human languages but
estimates range the number of distinct languages between 2500
and 7000. So while 50 languages seems big at first, it's actually a
tiny fraction of the human population using Facebook. Taking the first
50 of the Wikipedia list of languages by native speakers we omit
languages like Dutch (52), Greek (74), and Hungarian (78), and that's
just a few random nations picks from Europe.
As an example, Facebook has trouble moderating even a major language
like Arabic. It censored content from legitimate Arab news sources
when they mentioned the word al-Aqsa because Facebook associates
it with the al-Aqsa Martyrs' Brigades when they were talking
about the Al-Aqsa Mosque... This bias against Arabs also shows
how Facebook reproduces the American colonizer politics.
The WSJ also pointed out that Facebook spends only 13% of its
moderation efforts outside of the US, even if that represents 90% of
its users. Facebook spends three more times moderating on "brand
safety", which shows its priority is not the safety of its users, but
of the advertisers.
Military Internet
Sergey Brin and Larry Page are the Lewis and Clark of
our generation. Just like the latter were sent by Jefferson (the same)
to declare sovereignty over the entire US west coast, Google declared
sovereignty over all human knowledge, with its mission statement "to
organize the world's information and make it universally accessible
and useful". (It should be noted that Page somewhat questioned that
mission but only because it was not ambitious enough, Google
having "outgrown" it.)
The Lewis and Clark expedition, just like Google, had a scientific
pretext, because that is what you do to colonize a world,
presumably. Yet both men were military and had to receive scientific
training before they left. The Corps of Discovery was made up of
a few dozen enlisted men and a dozen civilians, including York an
African American slave owned by Clark and sold after the
expedition, with his final fate lost in history.
And just like Lewis and Clark, Google has a strong military
component. For example, Google Earth was not originally built at
Google but is the acquisition of a company called Keyhole which had
ties with the CIA. Those ties were brought inside Google during
the acquisition. Google's increasing investment inside the
military-industrial complex eventually led Google to workers
organizing a revolt although it is currently unclear to me how
much Google is involved in the military apparatus. Other companies,
obviously, do not have such reserve, with Microsoft, Amazon, and
plenty of others happily bidding on military contracts all the time.
Spreading the Internet
I am obviously not the first to identify colonial structures in the
Internet. In an article titled The Internet as an Extension of
Colonialism, Heather McDonald correctly identifies fundamental
problems with the "development" of new "markets" of Internet
"consumers", primarily arguing that it creates a digital divide
which creates a "lack of agency and individual freedom":
Many African people have gained access to these technologies but not
the freedom to develop content such as web pages or social media
platforms in their own way. Digital natives have much more power and
therefore use this to create their own space with their own norms,
shaping their online world according to their own outlook.
But the digital divide is certainly not the worst problem we have to
deal with on the Internet today. Going back to the Declaration, we
originally believed we were creating an entirely new world:
This governance will arise according to the conditions of our
world, not yours. Our world is different.
How I dearly wished that was true. Unfortunately, the Internet is
not that different from the offline world. Or, to be more accurate,
the values we have embedded in the Internet, particularly of free
speech absolutism, sexism, corporatism, and exploitation, are now
exploding outside of the Internet, into the "real" world.
The Internet was built with free software which, fundamentally, was
based on quasi-volunteer labour of an elite force of white men with
obviously too much time on their hands (and also: no children). The
mythical writing of GCC and Emacs by Richard Stallman is a good
example of this, but the entirety of the Internet now seems to be
running on random bits and pieces built by hit-and-run programmers
working on their copious free time. Whenever any of those fails,
it can compromise or bring down entire systems. (Heck, I wrote
this article on my day off...)
This model of what is fundamentally "cheap labour" is spreading out
from the Internet. Delivery workers are being exploited to the bone by
apps like Uber -- although it should be noted that workers organise
and fight back. Amazon workers are similarly exploited beyond
belief, forbidden to take breaks until they pee in bottles, with
ambulances nearby to carry out the bodies. During peak of the
pandemic, workers were being dangerously exposed to the virus in
warehouses. All this while Amazon is basically taking over the entire
economy.
The Declaration culminates with this prophecy:
We will spread ourselves across the Planet so that no one can arrest
our thoughts.
This prediction, which first felt revolutionary, is now chilling.
Colonial Internet
The Internet is, if not neo-colonial, plain colonial. The US colonies
had cotton fields and slaves, we have disposable cell phones and
Foxconn workers. Canada has its cultural genocide, Facebook
has his own genocides in Ethiopia, Myanmar, and mob violence
in India. Apple is at least implicitly accepting the Uyghur
genocide. And just like the slaves of the colony, those atrocities
are what makes the empire run.
The Declaration actually ends like this, a quote which I have in my
fortune cookies file:
We will create a civilization of the Mind in Cyberspace. May it be
more humane and fair than the world your governments have made
before.
That is still inspiring to me. But if we want to make "cyberspace"
more humane, we need to decolonize it. Work on cyberpeace instead of
cyberwar. Establish clear code of conduct, discuss ethics, and
question your own privileges, biases, and culture. For me the first
step in decolonizing my own mind is writing this article. Breaking
uptech monopolies might be an important step, but it won't be
enough: we have to do a culture shift as well, and that's the hard
part.
Appendix: an apology to Barlow
I kind of feel bad going through Barlow's declaration like this, point
by point. It is somewhat unfair, especially since Barlow passed away a
few years ago and cannot mount a response (even humbly assuming that
he might read this). But then again, he himself recognized he was
a bit too "optimistic" in 2009, saying: "we all get older and
smarter":
I'm an optimist. In order to be libertarian, you have to be an
optimist. You have to have a benign view of human nature, to believe
that human beings left to their own devices are basically good. But
I'm not so sure about human institutions, and I think the real point
of argument here is whether or not large corporations are human
institutions or some other entity we need to be thinking about
curtailing. Most libertarians are worried about government but not
worried about business. I think we need to be worrying about
business in exactly the same way we are worrying about government.
And, in a sense, it was a little naive to expect Barlow to not be a
colonist. Barlow is, among many things, a cattle rancher who grew up
on a colonial ranch in Wyoming. The ranch was founded in 1907 by his
great uncle, 17 years after the state joined the Union, and only a
generation or two after the Powder River War (1866-1868) and
Black Hills War (1876-1877) during which the US took over lands
occupied by Lakota, Cheyenne, Arapaho, and other native American
nations, in some of the last major First Nations Wars.
Appendix: further reading
There is another article that almost has the same title as this one:
Facebook and the New Colonialism. (Interestingly, the <title>
tag on the article is actually "Facebook the Colonial Empire" which I
also find appropriate.) The article is worth reading in full, but I
loved this quote so much that I couldn't resist reproducing it here:
Representations of colonialism have long been present in digital
landscapes. ( Even Super Mario Brothers, the video game designer
Steven Fox told me last year. You run through the landscape, stomp
on everything, and raise your flag at the end. ) But web-based
colonialism is not an abstraction. The online forces that shape a
new kind of imperialism go beyond Facebook.
It goes on:
Consider, for example, digitization projects that focus primarily on
English-language literature. If the web is meant to be humanity s new
Library of Alexandria, a living repository for all of humanity s
knowledge, this is a problem. So is the fact that the vast majority of
Wikipedia pages are about a relatively tiny square of the planet. For
instance, 14 percent of the world s population lives in Africa, but
less than 3 percent of the world s geotagged Wikipedia articles
originate there, according to a 2014 Oxford Internet Institute
report.
And they introduce another definition of Neo-colonialism, while
warning about abusing the word like I am sort of doing here:
I m loath to toss around words like colonialism but it s hard to
ignore the family resemblances and recognizable DNA, to wit, said
Deepika Bahri, an English professor at Emory University who focuses
on postcolonial studies. In an email, Bahri summed up those
similarities in list form:
ride in like the savior
bandy about words like equality, democracy, basic rights
mask the long-term profit motive (see 2 above)
justify the logic of partial dissemination as better than nothing
partner with local elites and vested interests
accuse the critics of ingratitude
In the end, she told me, if it isn t a duck, it shouldn t quack
like a duck.
Another good read is the classic Code and other laws of
cyberspace (1999, free PDF) which is also critical of
Barlow's Declaration. In "Code is law", Lawrence Lessig argues that:
computer code (or "West Coast Code", referring to Silicon Valley)
regulates conduct in much the same way that legal code (or "East
Coast Code", referring to Washington, D.C.) does (Wikipedia)
And now it feels like the west coast has won over the east coast, or
maybe it recolonized it. In any case, Internet now christens
emperors.
The diffoscope maintainers are pleased to announce the release of diffoscope
version 188. This version includes the following changes:
* Add support for Python Sphinx inventory files, usually named objects.inv.
* Fix Python bytecode decompilation tests with Python 3.10+.
(Closes: reproducible-builds/diffoscope#278)
In the last week, KDE released version 5.23 25th Anniversary Edition of the Plasma desktop with the usual long list of updates and improvements. This release celebrates 25 years of KDE, and Plasma 5.23.0 was released right on the day 25 years ago Matthias Ettrich sent an email to the de.comp.os.linux.misc newsgroup explaining a project he was working on. And Plasma 5.23 (with the bug fix 5.23.1) is now available for all Debian releases. (And don t forget KDE Gears/Apps 21.08!)
As usual, I am providing packages via my OBS builds. If you have used my packages till now, then you only need to change the plasma522 line to read plasma523. To give full details, I repeat (and update) instructions for all here: First of all, you need to add my OBS key say in /etc/apt/trusted.gpg.d/obs-npreining.asc and add a file /etc/apt/sources.lists.d/obs-npreining-kde.list, containing the following lines, replacing the DISTRIBUTION part with one of Debian_11 (for Bullseye), Debian_Testing, or Debian_Unstable:
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/other-deps/DISTRIBUTION/ ./
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/frameworks/DISTRIBUTION/ ./
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/plasma523/DISTRIBUTION/ ./
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/apps2108/DISTRIBUTION/ ./
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/other/DISTRIBUTION/ ./
The sharp eye might have detected also the apps2108 line, yes the KDE Gear suite of packages hgas been updated to 21.08 some time ago and is also available in my OBS builds (and in Debian/experimental).
Uploads to Debian
Plasma 5.23.0 has been uploaded to Debian, and is currently in transition to testing. Due to incorrect/insufficient Break/Depends, currently Debian/testing with the official packages for Plasma are broken. And as it looks this situation will continue for considerable time, considering that kwin is blocked by mesa, which in turn is blocked by llvm-toolchain-12, which has quite some RC bugs preventing it from transitioning. What a bad coincidence.
KDE Gears 21.08 are all in Debian Unstable and Testing, so the repositories here are mostly for users of Debian Bullseye (stable).
Krita beta
Krita has released the second beta of Krita 5.0, and this is available from the krita-beta repository, but only for amd64 architectures. Just add
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/krita-beta/DISTRIBUTION/ ./
This post was originally published in the Wikimedia Tech blog, authored by
Arturo Borrero Gonzalez.
NFS is a central piece of infrastructure that is essential to services like Toolforge.
Recently, the Cloud Services team at Wikimedia had been reviewing how we do NFS.
The current situation
NFS is a central piece of technology for some of the services that the
Wikimedia Cloud Services team offers to the community. We have several shares that power
different use cases: Toolforge user home directories live on NFS, and
Cloud VPS users can also access dumps using this protocol. The current setup involves
several physical hardware servers, with about 20TB of storage, offering shares over 10G links to
the cloud. For the system to be more fault-tolerant, we duplicate each share for redundancy using
DRBD.
Running NFS on dedicated hardware servers has traditionally offered us advantages: mostly on the
performance and the capacity fields.
As time has passed, we have been enumerating more and more reasons to review how we do NFS. For
one, the current setup is in violation of some of our internal rules regarding realm separation.
Additionally, we had been longing for additional flexibility managing our servers: we wanted to use
virtual machines managed by Openstack Nova. The DRBD-based high-availability system
required mostly a hand-crafted procedure for failover/failback. There s also some scalability
concerns as NFS is easy to grow up, but not to grow horizontally, and of course, we have to be able
to keep the tenancy setup while doing so, something that NFS does by using LDAP/Unix users and may
get complicated too when growing. In general, the servers have become too big to fail , clearly
technical debt, and it has taken us years to decide on taking on the task to rethink the
architecture. It s worth mentioning that in an ideal world, we wouldn t depend on NFS, but the
truth is that it will still be a central piece of infrastructure for years to come in services like
Toolforge.
Over a series of brainstorming meetings, the WMCS team evaluated the situation and sorted out the
many moving parts. The team managed to boil down the potential service future to two competing
options:
Adopt and introduce a new Openstack component into our cloud: Manila this was the right choice
if we were interested in a general NFS as a service offering for our Cloud VPS users.
Put the data on Cinder volumes and serve NFS from a couple of virtual machines created by hand
this was the right choice if we wanted something that required low effort to engineer and adopt.
Then we decided to research both options in parallel. For a number of reasons, the evaluation was
timeboxed to three weeks. Both ideas had a couple of points in common: the NFS data would be stored
on our Ceph farm via Cinder volumes, and we would rely on Ceph reliability to avoid using DRBD.
Another open topic was how to back up data from Ceph, to store our important bits in more than one
basket. We will get to the back up topic later.
The manila experiment
The Wikimedia Foundation was an early adopter of some Openstack components (Nova,
Glance, Designate, Horizon), but Manila was never
evaluated for usage until now. Our approach for this experiment was to closely follow the upstream
guidelines. We read the documentation and tried to understand the different setups you can build
with Manila. As we often feel with other Openstack components, the documentation doesn t perfectly
describe how to introduce a given component in your particular local setup. Here we use an
admin-controller flat-topology Neutron network. This network is shared by all tenants (or projects)
in our Openstack deployment. Also, Manila can use many different driver backends, for
things like NetApps or CephFS that we don t use , yet. After some research, the
generic driver was the one that seemed to better fit our use case. The generic driver
leverages Nova virtual machines instances plus Cinder volume to create and manage the shares. In
general, Manila supports two operational modes, whether it should create/destroy the share servers
(i.e, the virtual machine instances) or not. This option is called driver_handles_share_server (or
DHSS) and takes a boolean value.
We were interested in trying with DHSS=true, to really benefit from the potential of the setup.
NFS idea 6, original image in Wikitech
So, after sorting all these variables, we moved on with our initial testing. We built a PoC setup
as depicted in the diagram above, with the manila-share component running in a virtual machine
inside the cloud. The PoC led to us reporting several bugs upstream:
It s worth mentioning that the upstream community was extra-welcoming to us, and we re thankful for
that. However, at the end of our three-week period, our Manila setup still wasn t working as
expected. Your experience may change with other drivers perhaps the ZFSonLinux or the CephFS ones.
In general, we were having trouble making the setup work as expected, so we decided to abandon this
approach in favor of the other option we were considering at the beginning.
Simple virtual machine serving NFS
The alternative was to create a Nova virtual machine instance by hand and to configure it using
puppet. We have been investing in an automation framework lately, so the idea is to
not actually create the server by hand. Anyway, the data would be decoupled from the instance into
Cinder volumes, which led us to the question we left for later: How should we back up those
terabytes of important information? Just to be clear, the backup problem was independent of the above
options; with Manila we would still have had to solve the same challenge. We would like to see our
data be backed up somewhere else other than in Ceph. And that s exactly where we are at right now.
We ve been exploring different backup strategies and will finally use the Cinder backup
API.
Conclusion
The iteration will end with the dedicated NFS hardware servers being stopped, and the shares being
served from within the cloud. The migration will take some time to happen because we will check and
double-check that everything works as expected (including from the performance point of view)
before making definitive changes. We already have some plans to make sure our users experience as
little service impact as possible. The most troublesome shares will be those related to Toolforge.
At some point we will need to disallow writes to the NFS share, rsync the data out of the hardware
servers into the Cinder volumes, point the NFS clients to the new virtual machines, and then enable
writes again. The main Toolforge share has about 8TB of data, so this will take a while.
We will have more updates in the future. Who knows, perhaps our next-next iteration, in a couple of
years, will see us adopting Openstack Manila for good.
Featured image credit: File:(from break water) Manila Skyline panoramio.jpg, ewol, CC BY-SA 3.0This post was originally published in the Wikimedia Tech blog, authored
by Arturo Borrero Gonzalez.
Like each month, have a look at the work funded by Freexian s Debian LTS offering.
Debian project funding
Folks from the LTS team, along with members of the Debian Android Tools team and Phil Morrel, have proposed work on the Java build tool, gradle, which is currently blocked due to the need to build with a plugin not available in Debian. The LTS team reviewed the project submission and it has been approved. After approval we ve created a Request for Bids which is active now.
You ll hear more about this through official Debian channels, but in the meantime, if you feel you can help with this project, please submit a bid. Thanks!
This September, Freexian set aside 2550 EUR to fund Debian projects.
We re looking forward to receive more projects from various Debian teams! Learn more about the rationale behind this initiative in this article.
Debian LTS contributors
In September, 15 contributors have been paid to work on Debian LTS, their reports are available:
Abhijith PA has returned hours and marked themselves inactive, at least for the time being. He did 0h out of 14h, carried over 14h and returned 28h.
Adrian Bunk did 19.5h (out of 24.75h assigned and 12.75 from August), carrying over 18h to October.
Emilio Pozuelo Monfort did not report back about their work so we assume they did nothing (out of 5.5h assigned plus 74.5h from August), thus is carrying over 80h for October.
Holger Levsen did 3h (out of 12h assigned) and gave back 9h and carried over 3h.
Jeremiah Foster worked 10 hours (out of 20h assigned) on LTS work, carrying over 10h.
Lee Garrett did not report back about their work so we assume they did nothing (out of 24.75h assigned and 23.75 from August), thus is carrying over 48.50h for October.
Markus Koschany did 43.5h (out of 24.75h assigned and 18.75h from August)
Utkarsh Gupta did 24.75h (out of 24.75h assigned) but did not publish his report yet.
Ola Lundqvist did 2 hours (out of 21h carried over from previous months), and is thus carrying 19h for October.
Evolution of the situation
In September we released 30 DLAs. September was also the second month of Jeremiah coordinating LTS contributors.
Also, we would like say that we are always looking for new contributors to LTS. Please contact Jeremiah if you are interested!
The security tracker currently lists 33 packages with a known CVE and the dla-needed.txt file has 26 packages needing an update.
Thanks to our sponsors
Sponsors that joined recently are in bold.
FTP master
This month I accepted 224 and rejected 47 packages. This is almost thrice the rejects of last month. Please, be more careful and check your package twice before uploading. The overall number of packages that got accepted was 233.
Debian LTS
This was my eighty-seventh month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.
This month my all in all workload has been 24.75h. During that time I did LTS and normal security uploads of:
[DLA 2755-1] btrbk security update for one CVE
[DLA 2762-1] grilo security update for one CVE
[DLA 2766-1] openssl security update for one CVE
[DLA 2774-1] openssl1.0 security update for one CVE
[DLA 2773-1] curl security update for two CVEs
I also started to work on exiv2 and faad2.
Last but not least I did some days of frontdesk duties.
Debian ELTS
This month was the thirty-ninth ELTS month.
Unfortunately during my allocated time I could not process any upload. I worked on openssl, curl and squashfs-tools but for one reason or another the prepared packages didn t pass all tests. In order to avoid regressions, I postponed the uploads (meanwhile an ELA for curl was published ).
Last but not least I did some days of frontdesk duties.
Other stuff
On my neverending golang challenge I again uploaded some packages either for NEW or as source upload.
As Odyx took a break from all Debian activities, I volunteered to take care of the printing packages. Please be merciful when somethings breaks after I did an upload. My first printing upload was hplip
A new release 0.4.14 of RQuantLib was uploaded to CRAN earlier today, and has by now been uploaded to Debian as well.
QuantLib is a very comprehensice free/open-source library for quantitative finance; RQuantLib connects it to the R environment and language.
The release of RQuantLib comes just one months after the previous release, and brings three changes. First, we added both two more US-based calendars (including FederalReserve ) along with a bunch of not-yet-included other calendars which should complete the coverage in the R package relative to the upstream library. Should we have forgotten any, feel free to open an issue. Second, CRAN currently aims to have older autoconf conventions updated and notified maintainers of affected packages. I received a handful of these, and just like yesterday s update to littler refreshed this here. Third, we set up automated container builds on GitHub. No other changes were made, details follow.
Changes in RQuantLib version 0.4.14 (2021-10-06)
Changes in RQuantLib code:
Several new calendars were added (Dirk in #159 closing #155)
Changes in RQuantLib package and setup:
Docker containers are now updated on a monthly schedule via GitHub Actions
The configure files were updated to the standard of version 2.69 following a CRAN request
The goal behind reproducible builds is to ensure that no deliberate flaws have been introduced during compilation processes via promising or mandating that identical results are always generated from a given source. This allowing multiple third-parties to come to an agreement on whether a build was compromised or not by a system of distributed consensus.
In these reports we outline the most important things that have been happening in the world of reproducible builds in the past month:
First mentioned in our March 2021 report, Martin Heinz published two blog posts on sigstore, a project that endeavours to offer software signing as a public good, [the] software-signing equivalent to Let s Encrypt . The two posts, the first entitled Sigstore: A Solution to Software Supply Chain Security outlines more about the project and justifies its existence:
Software signing is not a new problem, so there must be some solution already, right? Yes, but signing software and maintaining keys is very difficult especially for non-security folks and UX of existing tools such as PGP leave much to be desired. That s why we need something like sigstore - an easy to use software/toolset for signing software artifacts.
Some time ago I checked Signal s reproducibility and it failed. I asked others to test in case I did something wrong, but nobody made any reports. Since then I tried to test the Google Play Store version of the apk against one I compiled myself, and that doesn t match either.
BitcoinBinary.org was announced this month, which aims to be a repository of Reproducible Build Proofs for Bitcoin Projects :
Most users are not capable of building from source code themselves, but we can at least get them able enough to check signatures and shasums. When reputable people who can tell everyone they were able to reproduce the project s build, others at least have a secondary source of validation.
Related to this, there was continuing discussion on how to embed/encode the build metadata for the Debian live images which were being worked on by Roland Clobus.
Ariadne Conill published another detailed blog post related to various security initiatives within the Alpine Linux distribution. After summarising some conventional security work being done (eg. with sudo and the release of OpenSSH version 3.0), Ariadne included another section on reproducible builds: The main blocker [was] determining what to do about storing the build metadata so that a build environment can be recreated precisely .
Finally, Bernhard M. Wiedemann posted his monthly reproducible builds status report.
Community news
On our website this month, Bernhard M. Wiedemann fixed some broken links [] and Holger Levsen made a number of changes to the Who is Involved? page [][][]. On our mailing list, Magnus Ihse Bursie started a thread with the subject Reproducible builds on Java, which begins as follows:
I m working for Oracle in the Build Group for OpenJDK which is primary responsible for creating a built artifact of the OpenJDK source code. [ ] For the last few years, we have worked on a low-effort, background-style project to make the build of OpenJDK itself building reproducible. We ve come far, but there are still issues I d like to address. []
diffoscopediffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions 183, 184 and 185 as well as performed significant triaging of merge requests and other issues in addition to making the following changes:
New features:
Support a newer format version of the R language s .rds files. []
Don t call close_archive when garbage collecting Archive instances, unless open_archive definitely returned successfully. This prevents, for example, an AttributeError where PGPContainer s cleanup routines were rightfully assuming that its temporary directory had actually been created. []
Fix (and test) the comparison of R language s .rdb files after refactoring temporary directory handling. []
Ensure that RPM archives exists in the Debian package description, regardless of whether python3-rpm is installed or not at build time. []
Codebase improvements:
Use our assert_diff routine in tests/comparators/test_rdata.py. []
Move diffoscope.versions to diffoscope.tests.utils.versions. []
Reformat a number of modules with Black. [][]
However, the following changes were also made:
Mattia Rizzolo:
Fix an autopkgtest caused by the androguard module not being in the (expected) python3-androguard Debian package. []
Appease a shellcheck warning in debian/tests/control.sh. []
Ignore a warning from h5py in our tests that doesn t concern us. []
Drop a trailing .1 from the Standards-Version field as it s required. []
Zbigniew J drzejewski-Szmek:
Stop using the deprecated distutils.spawn.find_executable utility. [][][][][]
Adjust an LLVM-related test for LLVM version 13. []
Update invocations of llvm-objdump. []
Adjust a test with a one-byte text file for file version 5.40. []
And, finally, Benjamin Peterson added a --diff-context option to control unified diff context size [] and Jean-Romain Garnier fixed the Macho comparator for architectures other than x86-64 [].
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
#990246 filed against vlc (forwarded upstream [][])
Testing framework
The Reproducible Builds project runs a testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:
Holger Levsen:
Drop my package rebuilder prototype as it s not useful anymore. []
Schedule old packages in Debian bookworm. []
Stop scheduling packages for Debian buster. [][]
Don t include PostgreSQL debug output in package lists. []
Detect Python library mismatches during build in the node health check. []
Update a note on updating the FreeBSD system. []
Mattia Rizzolo:
Silence a warning from Git. []
Update a setting to reflect that Debian bookworm is the new testing. []
Upgrade the PostgreSQL database to version 13. []
Roland Clobus (Debian live image generation):
Workaround non-reproducible config files in the libxml-sax-perl package. []
Use the new DNS for the snapshot service. []
Vagrant Cascadian:
Also note that the armhf architecture also systematically varies by the kernel. []
Contributing
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
Like each month, have a look at the work funded by Freexian s Debian LTS offering.
Debian project funding
In August, we put aside 2460 EUR to fund Debian projects. We received a new project proposal that got approved and there s an associated bid request if you feel like proposing yourself to implement this project.
We re looking forward to receive more projects from various Debian teams! Learn more about the rationale behind this initiative in this article.
Debian LTS contributors
In August, 14 contributors have been paid to work on Debian LTS, their reports are available:
Abhijith PA did 4.0h (out of 14h assigned and 5h from August), thus carrying over 15h to September.
Adrian Bunk did 11h (out of 23.75h assigned), thus carrying over 12.75h to September.
Emilio Pozuelo Monfort did not report back about their work so we assume they did nothing (out of 23.75h assigned plus 50.75h from August), thus is carrying over 74.5h for September.
Holger Levsen did 3h (out of 12h assigned) to help coordinate the team, and gave back the remaining hours.
Lee Garrett did nothing (out of 23.75h assigned), thus is carrying over 23.75h for September.
Markus Koschany did 35h (out of 23.75h assigned and 30h from August), thus carrying over 18.75h to September.
Neil Williams did 24h (out of 23.75h assigned), thus anticipating 0.25h of October.
Roberto C. S nchez did 22.25h (out of 23.75h assigned), thus carrying over 1.5h to September.
Sylvain Beucler did 21.5h (out of 23.75h assigned), thus carrying over 2.25h to September.
Evolution of the situation
In August we released 30 DLAs.
This is the first month of Jeremiah coordinating LTS contributors. We would like to thank Holger Levsen for his work on this role up to now.
Also, we would like to remark once again that we are constantly looking for new contributors. Please contact Jeremiah if you are interested!
The security tracker currently lists 73 packages with a known CVE and the dla-needed.txt file has 29 packages needing an update.
Thanks to our sponsors
Sponsors that joined recently are in bold.
Using podman for most of my local development environment.
For my personal/upstream development I started using podman
instead of lxc and pbuilder and other toolings. Most
projects provide reasonable docker images (such as rust) and
I am happier keeping my environment as a whole stable while
I can iterate. I have a Dockerfile for the development
environment like this:
Or perhaps this, if you also wanted to send ssh to the background:
ssh -NT -L 3306:db.example.com:3306 example.com &
Both of these use at least one option that is entirely redundant,
and the second can cause ssh to fail to connect if you happen to be
using password authentication.
However, they seem to still persist in various articles about ssh port forwarding.
I myself was using the first variation until just recently, and I figured
I would write this up to inform others who might be still using these
solutions.
The correct option for this situation is not -nNT but simply
-N, as in:
If you want to also send ssh to the background,
then you ll want to add -f instead of using your shell s built-in &
feature, because you can then input passwords into ssh if necessary1
Honestly, that s the point of this article, so you can stop here
if you want. If you re looking for a detailed explaination of what
each of these options actually does, or if you have no idea what
I m talking about, read on!
What is SSH Port Forwarding?
ssh is a powerful tool for remote access to servers,
allowing you to execute commands on a remote machine.
It can also forward ports
through a secure tunnel with the -L and -R options.
Basically, you can forward a connection to a local port to a remote server
like so:
ssh -L 8080:other.example.com:80 ssh.example.com
In this example, you connect to ssh.example.com and then ssh forwards
any traffic on your local machine port 80802
to other.example.com port 80 via ssh.example.com.
This is a really powerful feature, allowing you to jump3 inside your firewall
with just an ssh server exposed to the world.
It can work in reverse as well with the -R option, allowing connections
on a remote host in to a server running on your local machine.
For example, say you were running a website on your local machine on port 8080
but wanted it accessible on example.com port 804.
You could use something like:
ssh -R 8080:example.com:80 example.com
The trouble with ssh port forwarding is that, absent any additional options,
you also open a shell on the remote machine.
If you re planning to both work on a remote machine and use it to forward
some connection, this is fine, but if you just need to forward a port quickly
and don t care about a shell at that moment, it can be annoying, especially since
if the shell closes ssh will close the forwarding port as well.
This is where the -N option comes in.
SSH just forwarding ports
In the ssh manual page5, -N is explained like so:
Do not execute a remote command. This is useful for just forwarding ports.
This is all we need. It instructs ssh to run no commands on the remote server,
just forward the ports specified in the -L or -R options.
But people seem to think that there are a bunch of other necessary options,
so what do those do?
SSH and stdin
-n controls how ssh interacts with standard input,
specifically telling it not to:
Redirects stdin from /dev/null (actually, prevents reading from stdin). This must be used when ssh is run in the
background. A common trick is to use this to run X11 programs on a remote machine. For example, ssh -n
shadows.cs.hut.fi emacs & will start an emacs on shadows.cs.hut.fi, and the X11 connection will be automatically for
warded over an encrypted channel. The ssh program will be put in the background. (This does not work if ssh needs to
ask for a password or passphrase; see also the -f option.)
SSH passwords and backgrounding
-f sends ssh to background, freeing up the terminal in which you ran ssh
to do other things.
Requests ssh to go to background just before command execution. This is useful if ssh is going to ask for passwords
or passphrases, but the user wants it in the background. This implies -n. The recommended way to start X11 programs
at a remote site is with something like ssh -f host xterm.
As indicated in the description of -n, this does the same thing as using
the shell s & feature
with -n, but allows you to put in any necessary passwords first.
SSH and pseudo-terminals
-T is a little more complicated than the others and has a very short
explanation:
Disable pseudo-terminal allocation.
It has a counterpart in -t, which is explained a little better:
Force pseudo-terminal allocation. This can be used to execute arbitrary screen-based programs on a remote machine,
which can be very useful, e.g. when implementing menu services. Multiple -t options force tty allocation, even if ssh
has no local tty.
As the description of -t indicates, ssh is allocating a pseudo-terminal on
the remote machine, not the local one.
However, I have confirmed6 that -N doesn t allocate a pseudo-terminal either,
since it doesn t run any commands.
Thus this option is entirely unnecessary.
What s a pseudo-terminal?
This is a bit complicated, but basically it s an interface used in UNIX-like
systems, like Linux or BSD, that pretends to be a terminal
(thus pseudo-terminal).
Programs like your shell, or any text-based menu system made in libraries
like ncurses, expect to be connected to one (when used interactively at least).
Basically it fakes as if the input it is given
(over the network, in the case of ssh) was typed on a physical terminal
device and do things like raise an interrupt (SIGINT) if Ctrl+C is pressed.
Why?
I don t know why these incorrect uses of ssh got passed around as correct,
but I suspect it s a form of cargo cult,
where we use example commands others provide and don t question what they do.
One stack overflow answer I read that provided these options seemed to think
-T was disabling the local pseudo-terminal, which might go some way towards
explaining why they thought it was necessary.
I guess the moral of this story is to question everything and actually read
the manual, instead of just googling it.
Not that you SHOULD be using ssh with password authentication anyway, but people do.
Only on your loopback address by default, so that you re not allowing random people on your network to use your tunnel.
In fact, ssh even supports Jump Hosts, allowing you to automatically forward an ssh connection through another machine.
I can t say I recommend a setup like this for anything serious, as you d need to ssh as root to forward ports less than 1024. SSH forwarding is not for permanent solutions, just short-lived connections to machines that would be otherwise inaccessible.
Specifically, my source is the ssh(1) manual page in OpenSSH 8.4, shipped as 1:8.4p1-5 in Debian bullseye.
I just forwarded ports with -N and then logged in to that same machine and looked at psuedo-terminal allocations via ps ux. No terminal is associated with ssh connections using just the -N option.
Kali is a Debian based distribution aimed at penetration testing. I haven t felt a need to use it in the past because Debian has packages for all the scanning tools I regularly use, and all the rest are free software that can be obtained separately. But I recently decided to try it.
Here s the URL to get Kali [1]. For a VM you can get VMWare or VirtualBox images, I chose VMWare as it s the most popular image format and also a much smaller download (2.7G vs 4G). For unknown reasons the torrent for it didn t work (might be a problem with my torrent client). The download link for it was extremely slow in Australia, so I downloaded it to a system in Germany and then copied it from there.
I don t want to use either VMWare or VirtualBox because I find KVM/Qemu sufficient to do everything I want and they are in the Main section of Debian, so I needed to convert the image files. Some of the documentation on converting image formats to use with QEMU/KVM says to use a program called kvm-img which doesn t seem to exist, I used qemu-img from the qemu-utils package in Debian/Bullseye. The man page qemu-img(1) doesn t list the types of output format supported by the -O option and the examples returned by a web search show using -O qcow2 . It turns out that the following command will convert the image to raw format which is the format I prefer. I use BTRFS for storing all my VM images and that does all the copy-on-write I need.
After converting it the file was 500M smaller than the VMWare files (10.2 vs 10.7G). Probably the Kali distribution file could be reduced in size by converting it to raw and then back to VMWare format. The Kali VMWare image is compressed with 7zip which has a good compression ratio, I waited almost 90 minutes for zstd to compress it with -19 and the result was 12% larger than the 7zip file.
VMWare apparently likes to use an emulated SCSI controller, I spent some time trying to get that going in KVM. Apparently recent versions of QEMU changed the way this works and therefore older web pages aren t helpful. Also allegedly the SCSI emulation is buggy and unreliable (but I didn t manage to get it going so can t be sure). It turns out that the VM is configured to work with the virtio interface, the initramfs.conf has the configuration option MODULES=most which makes it boot on all common configurations (good work by the initramfs-tools maintainers). The image works well with the Spice display interface, so it doesn t capture my mouse, the window for the VM works the same way as other windows on my desktop and doesn t capture the mouse cursor. I don t know if this level of Spice integration is in Debian now, last time I tested it didn t work that way.
I also downloaded Metasploitable [2] which is a VM image designed to be full of security flaws for testing the tools that are in Kali. Again it worked nicely after converting from VMWare to raw format. One thing to note about Metasploitable is that you must not make it available on the public Internet. My home network has NAT for IPv4 but all systems get public IPv6 addresses. It s usually nice that those things just work on VMs but not for this. So I added an iptables command to block IPv6 to /etc/rc.local.
Conclusion
Installing VMs for both these distributions was quite easy. Most of my time was spent downloading from a slow server, trying to get SCSI emulation working, working out how to convert image files, and testing different compression options. The time spent doing stuff once I knew what to do was very small.
Kali has zsh as the default shell, it s quite nice. I ve been happy with bash for decades, but I might end up trying zsh out on other machines.
Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 912 other packages on CRAN.
This new release brings us Armadillo 10.7.0 released this morning by Conrad. Leading up to this were three runs of reverse dependencies the first of which uncovered the need for a small PR for subview_cols support which Conrad kindly supplied.
The full set of changes follows. We include the last interim release (sent as usual to the drat repo) as well.
Changes in RcppArmadillo version 0.10.7.0.0 (2021-09-30)
Upgraded to Armadillo release 10.7.0 (Entropy Maximizer)
faster handling of submatrix views accessed by X.cols(first_col,last_col)
faster handling of element-wise min() and max() in compound expressions
expanded solve() with solve_opts::force_approx option to force use of the approximate solver
Changes in RcppArmadillo version 0.10.6.2.0 (2021-08-05)
Upgraded to Armadillo release 10.6.2 (Keep Calm)
fix incorrect use of constexpr for handling fixed-size matrices and vectors
My home automation setup has been fairly static recently; it does what we need and generally works fine. One area I think could be better is controlling it; we have access Home Assistant on our phones, and the Alexa downstairs can control things, but there are no smart assistants upstairs and sometimes it would be nice to just push a button to turn on the light rather than having to get my phone out. Thanks to the fact the UK generally doesn t have neutral wire in wall switches that means looking at something battery powered. Which means wifi based devices are a poor choice, and it s necessary to look at something lower power like Zigbee or Z-Wave.
Zigbee seems like the better choice; it s a more open standard and there are generally more devices easily available from what I ve seen (e.g. Philips Hue and IKEA TR DFRI). So I bought a couple of Xiaomi Mi Smart Home Wireless Switches, and a CC2530 module and then ignored it for the best part of a year. Finally I got around to flashing the Z-Stack firmware that Koen Kanters kindly provides. (Insert rant about hardware manufacturers that require pay-for tool chains. The CC2530 is even worse because it s 8051 based, so SDCC should be able to compile for it, but the TI Zigbee libraries are only available in a format suitable for IAR s embedded workbench.)
Flashing the CC2530 is a bit of faff. I ended up using the CCLib fork by Stephan Hadinger which supports the ESP8266. The nice thing about the CC2530 module is it has 2.54mm pitch pins so nice and easy to jumper up. It then needs a USB/serial dongle to connect it up to a suitable machine, where I ran Zigbee2MQTT. This scares me a bit, because it s a bunch of node.js pulling in a chunk of stuff off npm. On the flip side, it Just Works and I was able to pair the Xiaomi button with the device and see MQTT messages that I could then use with Home Assistant. So of course I tore down that setup and went and ordered a CC2531 (the variant with USB as part of the chip). The idea here was my test setup was upstairs with my laptop, and I wanted something hooked up in a more permanent fashion.
Once the CC2531 arrived I got distracted writing support for the Desk Viking to support CCLib (and modified it a bit for Python3 and some speed ups). I flashed the dongle up with the Z-Stack Home 1.2 (default) firmware, and plugged it into the house server. At this point I more closely investigated what Home Assistant had to offer in terms of Zigbee integration. It turns out the ZHA integration has support for the ZNP protocol that the TI devices speak (I m reasonably sure it didn t when I first looked some time ago), so that seemed like a better option than adding the MQTT layer in the middle.
I hit some complexity passing the dongle (which turns up as /dev/ttyACM0) through to the Home Assistant container. First I needed an override file in /etc/systemd/nspawn/hass.nspawn:
(I m not clear why the VirtualEthernet needed to exist; without it networking broke entirely but I couldn t see why it worked with no override file.)
A udev rule on the host to change the ownership of the device file so the root user and dialout group in the container could see it was also necessary, so into /etc/udev/rules.d/70-persistent-serial.rules went:
In the container itself I had to switch PrivateDevices=true to PrivateDevices=false in the home-assistant.service file (which took me a while to figure out; yay for locking things down and then needing to use those locked down things).
Finally I added the hass user to the dialout group. At that point I was able to go and add the integration with Home Assistant, and add the button as a new device. Excellent. I did find I needed a newer version of Home Assistant to get support for the button, however. I was still on 2021.1.5 due to upstream dropping support for Python 3.7 and not being prepared to upgrade to Debian 11 until it was actually released, so the version of zha-quirks didn t have the correct info. Upgrading to Home Assistant 2021.8.7 sorted that out.
There was another slight problem. Range. Really I want to use the button upstairs. The server is downstairs, and most of my internal walls are brick. The solution turned out to be a TR DFRI socket, which replaced the existing ESP8266 wifi socket controlling the stair lights. That was close enough to the server to have a decent signal, and it acts as a Zigbee router so provides a strong enough signal for devices upstairs. The normal approach seems to be to have a lot of Zigbee light bulbs, but I have mostly kept overhead lights as uncontrolled - we don t use them day to day and it provides a nice fallback if the home automation has issues.
Of course installing Zigbee for a single button would seem to be a bit pointless. So I ordered up a Sonoff door sensor to put on the front door (much smaller than expected - those white boxes on the door are it in the picture above). And I have a 4 gang wireless switch ordered to go on the landing wall upstairs.
Now I ve got a Zigbee setup there are a few more things I m thinking of adding, where wifi isn t an option due to the need for battery operation (monitoring the external gas meter springs to mind). The CC2530 probably isn t suitable for my needs, as I ll need to write some custom code to handle the bits I want, but there do seem to be some ARM based devices which might well prove suitable
One of the assumptions baked deeply into US society (and many others) is
that people are largely defined by the work they do, and that work is the
primary focus of life. Even in Marxist analysis, which is otherwise
critical of how work is economically organized, work itself reigns
supreme. This has been part of the feminist critique of both capitalism
and Marxism, namely that both devalue domestic labor that has
traditionally been unpaid, but even that criticism is normally framed as
expanding the definition of work to include more of human activity. A
few exceptions aside, we shy away from
fundamentally rethinking the centrality of work to human experience.
The Problem with Work begins as a critical analysis of that
centrality of work and a history of some less-well-known movements against
it. But, more valuably for me, it becomes a discussion of the types and
merits of utopian thinking, including why convincing other people is not
the only purpose for making a political demand.
The largest problem with this book will be obvious early on: the writing
style ranges from unnecessarily complex to nearly unreadable. Here's an
excerpt from the first chapter:
The lack of interest in representing the daily grind of work routines
in various forms of popular culture is perhaps understandable, as is
the tendency among cultural critics to focus on the animation and
meaningfulness of commodities rather than the eclipse of laboring
activity that Marx identifies as the source of their fetishization
(Marx 1976, 164-65). The preference for a level of abstraction that
tends not to register either the qualitative dimensions or the
hierarchical relations of work can also account for its relative
neglect in the field of mainstream economics. But the lack of
attention to the lived experiences and political textures of work
within political theory would seem to be another matter. Indeed,
political theorists tend to be more interested in our lives as
citizens and noncitizens, legal subjects and bearers of rights,
consumers and spectators, religious devotees and family members, than
in our daily lives as workers.
This is only a quarter of a paragraph, and the entire book is written like
this.
I don't mind the occasional use of longer words for their precise meanings
("qualitative," "hierarchical") and can tolerate the academic habit of
inserting mostly unnecessary citations. I have less patience with the
meandering and complex sentences, excessive hedge words ("perhaps," "seem
to be," "tend to be"), unnecessarily indirect phrasing ("can also account
for" instead of "explains"), or obscure terms that are unnecessary to the
sentence (what is "animation of commodities"?). And please have mercy and
throw a reader some paragraph breaks.
The writing style means substantial unnecessary effort for the reader,
which is why it took me six months to read this book. It stalled all of
my non-work non-fiction reading and I'm not sure it was worth the effort.
That's unfortunate, because there were several important ideas in here
that were new to me.
The first was the overview of the "wages for housework" movement, which I
had not previously heard of. It started from the common feminist position
that traditional "women's work" is undervalued and advocated taking the
next logical step of giving it equality with paid work by making it
paid work. This was not successful, obviously, although the increasing
prevalence of day care and cleaning services has made it partly true
within certain economic classes in an odd and more capitalist way. While
I, like Weeks, am dubious this was the right remedy, the observation
that household work is essential to support capitalist activity but is
unmeasured by GDP and often uncompensated both economically and socially
has only become more accurate since the 1970s.
Weeks argues that the usefulness of this movement should not be judged by
its lack of success in achieving its demands, which leads to the second
interesting point: the role of utopian demands in reframing and expanding
a discussion. I normally judge a political demand on its effectiveness at
convincing others to grant that demand, by which standard many activist
campaigns (such as wages for housework) are unsuccessful. Weeks points
out that making a utopian demand changes the way the person making the
demand perceives the world, and this can have value even if the demand
will never be granted. For example, to demand wages for housework
requires rethinking how work is defined, what activities are compensated
by the economic system, how such wages would be paid, and the implications
for domestic social structures, among other things. That, in turn, helps
in questioning assumptions and understanding more about how existing
society sustains itself.
Similarly, even if a utopian demand is never granted by society at large,
forcing it to be rebutted can produce the same movement in thinking in
others. In order to rebut a demand, one has to take it seriously and
mount a defense of the premises that would allow one to rebut it. That
can open a path to discussing and questioning those premises, which can
have long-term persuasive power apart from the specific utopian demand.
It's a similar concept as the Overton Window, but with more nuance: the
idea isn't solely to move the perceived range of accepted discussion, but
to force society to examine its assumptions and premises well enough to
defend them, or possibly discover they're harder to defend than one might
have thought.
Weeks applies this principle to universal basic income, as a utopian
demand that questions the premise that work should be central to personal
identity. I kept thinking of the Black Lives Matter movement and the
demand to abolish the police, which (at least in popular discussion) is a
more recent example than this book but follows many of the same
principles. The demand itself is unlikely to be met, but to rebut it
requires defending the existence and nature of the police. That in turn
leads to questions about the effectiveness of policing, such as clearance
rates (which are far lower than one might have assumed). Many more
examples came to mind. I've had that experience of discovering problems
with my assumptions I'd never considered when debating others, but had not
previously linked it with the merits of making demands that may be
politically infeasible.
The book closes with an interesting discussion of the types of utopias,
starting from the closed utopia in the style of Thomas More in which the
author sets up an ideal society. Weeks points out that this sort of
utopia tends to collapse with the first impossibility or inconsistency the
reader notices. The next step is utopias that acknowledge their own
limitations and problems, which are more engaging (she cites Le Guin's
The Dispossessed). More conditional
than that is the utopian manifesto, which only addresses part of society.
The least comprehensive and the most open is the utopian demand, such as
wages for housework or universal basic income, which asks for a specific
piece of utopia while intentionally leaving unspecified the rest of the
society that could achieve it. The demand leaves room to maneuver; one
can discuss possible improvements to society that would approach that
utopian goal without committing to a single approach.
I wish this book were better-written and easier to read, since as it
stands I can't recommend it. There were large sections that I read but
didn't have the mental energy to fully decipher or retain, such as the
extended discussion of Ernst Bloch and Friedrich Nietzsche in the context
of utopias. But that way of thinking about utopian demands and their
merits for both the people making them and for those rebutting them, even
if they're not politically feasible, will stick with me.
Rating: 5 out of 10
Release 0.6.28 of the digest package arrived at CRAN earlier today, and has already been uploaded Debian as well.
digest creates hash digests of arbitrary R objects (using the md5, sha-1, sha-256, sha-512, crc32, xxhash32, xxhash64, murmur32, spookyhash, and blake3 algorithms) permitting easy comparison of R language objects. It is a mature and widely-used as many tasks may involve caching of objects for which it provides convenient general-purpose hash key generation.
This release comes eleven months after the previous releases and rounds out a number of corners. Continuous Integration was updated using r-ci. Several contribututors help with a small fix applied to avoid unaligned reads, a rewording for a help page as well as windows path encoding for in the vectorised use case.
My CRANberries provides the usual summary of changes to the previous version. For questions or comments use the issue tracker off the GitHub repo.
If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.
Colson Whitehead's latest novel, Harlem Shuffle, was always going to be widely reviewed, if only because his last two books won Pulitzer prizes. Still, after enjoying both The Underground Railroad and The Nickel Boys, I was certainly going to read his next book, regardless of what the critics were saying indeed, it was actually quite agreeable to float above the manufactured energy of the book's launch.
Saying that, I was encouraged to listen to an interview with the author by Ezra Klein. Now I had heard Whitehead speak once before when he accepted the Orwell Prize in 2020, and once again he came across as a pretty down-to-earth guy. Or if I were to emulate the detached and cynical tone Whitehead embodied in The Nickel Boys, after winning so many literary prizes in the past few years, he has clearly rehearsed how to respond to the cliched questions authors must be asked in every interview. With the obligatory throat-clearing of 'so, how did you get into writing?', for instance, Whitehead replies with his part of the catechism that 'It seemed like being a writer could be a cool job. You could work from home and not talk to people.' The response is the right combination of cute and self-effacing... and with its slight tone-deafness towards enforced isolation, it was no doubt honed before Covid-19.
Harlem Shuffle tells three separate stories about Ray Carney, a furniture salesman and 'fence' for stolen goods in New York in the 1960s. Carney doesn't consider himself a genuine criminal though, and there's a certain logic to his relativistic morality. After all, everyone in New York City is on the take in some way, and if some 'lightly used items' in Carney's shop happened to have had 'previous owners', well, that's not quite his problem. 'Nothing solid in the city but the bedrock,' as one character dryly observes. Yet as Ezra pounces on in his NYT interview mentioned abov, the focus on the Harlem underworld means there are very few women in the book, and Whitehead's circular response ah well, it's a book about the criminals at that time! was a little unsatisfying. Not only did it feel uncharacteristically slippery of someone justly lauded for his unflinching power of observation (after all, it was the author who decided what to write about in the first place), it foreclosed on the opportunity to delve into why the heist and caper genres (from The Killing, The Feather Thief, Ocean's 11, etc.) have historically been a 'male' mode of storytelling.
Perhaps knowing this to be the case, the conversation quickly steered towards Ray Carney's wife, Elizabeth, the only woman in the book who could be said possesses some plausible interiority. The following off-hand remark from Whitehead caught my attention:
My wife is convinced that [Elizabeth] knows everything about Carney's criminal life, and is sort of giving him a pass. And I'm not sure if that's true. I have to have to figure out exactly what she knows and when she knows it and how she feels about it.
I was quite taken by this, although not simply due to its effect on the story it self. As in, it immediately conjured up a charming picture of Whitehead's domestic arrangements: not only does Whitehead's wife feel free to disagree with what one of Whitehead's 'own' characters knows or believes, but that Colson has no problem whatsoever sharing that disagreement with the public at large. (It feels somehow natural that Whitehead's wife believes her counterpart knows more than she lets on, whilst Whitehead himself imbues the protagonist's wife with a kind of neo-Victorian innocence.) I'm minded to agree with Whitehead's partner myself, if only due to the passages where Elizabeth is studiously ignoring Carney's otherwise unexplained freak-outs.
But all of these meta-thoughts simply underline just how emancipatory the Death of the Author can be. This product of academic literary criticism (the term was coined by Roland Barthes' 1967 essay of the same name) holds that the original author's intentions, ideas or biographical background carry no especial weight in determining how others should interpret their work. It is usually understood as meaning that a writer's own views are no more valid or 'correct' than the views held by someone else. (As an aside, I've found that most readers who encounter this concept for the first time have been reading books in this way since they were young. But the opposite is invariably true with cinephiles, who often have a bizarre obsession with researching or deciphering the 'true' interpretation of a film.) And with all that in mind, can you think of a more wry example of how freeing (and fun) nature of the Death of the Author than an author's own partner dissenting with their (Pulitzer Prize-winning) husband on the position of a lynchpin character?
The 1964 Harlem riot began after James Powell, a 15-year-old African American, was shot and killed by Thomas Gilligan, an NYPD police officer in front of 10s of witnesses. Gilligan was subsequently cleared by a grand jury.
As it turns out, the reviews for Harlem Shuffle have been almost universally positive, and after reading it in the two days after its release, I would certainly agree it is an above-average book. But it didn't quite take hold of me in the way that The Underground Railroad or The Nickel Boys did, especially the later chapters of The Nickel Boys that were set in contemporary New York and could thus make some (admittedly fairly explicit) connections from the 1960s to the present day that kind of connection is not there in Harlem Shuffle, or at least I did not pick up on it during my reading.
I can see why one might take exception to that, though. For instance, it is certainly true that the week-long Harlem Riot forms a significant part of the plot, and some events in particular are entirely contingent on the ramifications of this momentous event. But it's difficult to argue the riot's impact are truly integral to the story, so not only is this uprising against police brutality almost regarded as a background event, any contemporary allusion to the murder of George Floyd is subsequently watered down. It's nowhere near the historical rubbernecking of Forrest Gump (1994), of course, but that's not a battle you should ever be fighting.
Indeed, whilst a certain smoothness of affect is to be priced into the Whitehead reading experience, my initial overall reaction to Harlem Shuffle was fairly flat, despite all the action and intrigue on the page. The book perhaps belies its origins as a work conceived during quarantine after all, the book is essentially comprised of three loosely connected novellas, almost as if the unreality and mental turbulence of lockdown prevented the author from performing the psychological 'deep work' of producing a novel-length text with his usual depth of craft. A few other elements chimed with this being a 'lockdown novel' as well, particularly the book's preoccupation with the sheer physicality of the city compared to the usual complex interplay between its architecture and its inhabitants. This felt like it had been directly absorbed into the book from the author walking around his deserted city, and thus being able to take in details for the first time:
The doorways were entrances into different cities no, different entrances into one vast, secret city. Ever close, adjacent to all you know, just underneath. If you know where to look.
And I can't fail to mention that you can almost touch Whitehead's sublimated hunger to eat out again as well:
Stickups were chops they cook fast and hot, you re in and out. A stakeout was ribs fire down low, slow, taking your time.
[ ]
Sometimes when Carney jumped into the Hudson when he was a kid, some of that stuff got into his mouth. The Big Apple Diner served it up and called it coffee.
More seriously, however, the relatively thin personalities of minor characters then reminded me of the simulacrum of Zoom-based relationships, and the essentially unsatisfactory endings to the novellas felt reminiscent of lockdown pseudo-events that simply fizzle out without a bang. One of the stories ties up loose ends with: 'These things were usually enough to terminate a mob war, and they appeared to end the hostilities in this case as well.' They did? Well, okay, I guess.
The corner of 125th Street and Morningside Avenue in 2019, the purported location of Carney's fictional furniture store. Signage plays a prominent role in Harlem Shuffle, possibly due to the author's quarantine walks.
Still, it would be unfair to characterise myself as 'disappointed' with the novel, and none of this piece should be taken as really deep criticism. The book certainly was entertaining enough, and pretty funny in places as well:
Carney didn t have an etiquette book in front of him, but he was sure it was bad manners to sit on a man s safe.
[ ]
The manager of the laundromat was a scrawny man in a saggy undershirt painted with sweat stains. Launderer, heal thyself.
Yet I can't shake the feeling that every book you write is a book that you don't, and so we might need to hold out a little longer for Whitehead's 'George Floyd novel'. (Although it is for others to say how much of this sentiment is the expectations of a White Reader for The Black Author to ventriloquise the pain of 'their' community.)
Some room for personal critique is surely permitted. I dearly missed the junk food energy of the dry and acerbic observations that run through Whitehead's previous work. At one point he had a good line on the model tokenisation that lurks behind 'The First Negro to...' labels, but the callbacks to this idea ceased without any payoff. Similar things happened with the not-so-subtle critiques of the American Dream:
Entrepreneur? Pepper said the last part like manure. That s just a
hustler who pays taxes.
[ ]
One thing I ve learned in my job is that life is cheap, and when things start getting expensive, it gets cheaper still.
Ultimately, though, I think I just wanted more. I wanted a deeper exploration of how the real power in New York is not wielded by individual street hoodlums or even the cops but in the form of real estate, essentially serving as a synecdoche for Capital as a whole. (A recent take of this can be felt in Jed Rothstein's 2021 documentary, WeWork: Or the Making and Breaking of a $47 Billion Unicorn and it is perhaps pertinent to remember that the US President at the time this novel was written was affecting to be a real estate tycoon.). Indeed, just like the concluding scenes of J. J. Connolly's Layer Cake, although you can certainly pull off a cool heist against the Man, power ultimately resides in those who control the means of production... and a homespun furniture salesman on the corner of 125 & Morningside just ain't that. There are some nods to kind of analysis in the conclusion of the final story ('Their heist unwound as if it had never happened, and Van Wyck kept throwing up buildings.'), but, again, I would have simply liked more.
And when I attempted then file this book away into the broader media landscape, given the current cultural visibility of 1960s pop culture (e.g. One Night in Miami (2020), Judas and the Black Messiah (2021), Summer of Soul (2021), etc.), Harlem Shuffle also seemed like a missed opportunity to critically analyse our (highly-qualified) longing for the civil rights era. I can certainly understand why we might look fondly on the cultural products from a period when politics was less alienated, when society was less atomised, and when it was still possible to imagine meaningful change, but in this dimension at least, Harlem Shuffle seems to merely contribute to this nostalgic escapism.
A bug-fix release of the 
In the last week, KDE 
As usual, I am providing packages via my OBS builds. If you have used my packages till now, then you only need to change the 
This post was originally published in the 
Using podman for most of my local development environment.
For my personal/upstream development I started using podman
instead of lxc and pbuilder and other toolings. Most
projects provide reasonable docker images (such as rust) and
I am happier keeping my environment as a whole stable while
I can iterate. I have a Dockerfile for the development
environment like this:
My 
Colson Whitehead's latest novel, 
.png){kind=link}
_Manila_Skyline_-_panoramio.jpg){kind=link}