The Motoko programming language has a built-in data type for optional values, named ?t with values null and ?v (for v : t); this is the equivalent of Haskell s Maybe, Ocaml s option or Rust s Option. In this post, I explain how Motoko represents such optional values (almost) without allocation. I neither claim nor expect that any of this is novel; I just hope it s interesting.

Uniform representation The Motoko programming language, designed by Andreas Rossberg and implemented by a pretty cool team at DFINITY is a high-level language with strict semantics and a strong, structural, equi-recursive type system that compiles down to WebAssembly. Because the type system supports polymorphism, it represents all values uniformly. Simplified for the purpose of this blog post, they are always pointers into a heap object where the first word of the heap object, the heap tag, contains information about the value:

 
  tag    
 

The tag is something like array, int64, blob, variant, record, , and it has two purposes:

• The garbage collector uses it to understand what kind of object it is looking at, so that it can move it around and follow pointers therein. Variable size objects such as arrays include the object size in a subsequent word of the heap object.
• Some types have values that may have different shapes on the heap. For example, the ropes used in our text representation can either be a plain blob, or a concatenation node of two blobs. For these types, the tag of the heap object is inspected.

The optional type, naively The optional type (?t) is another example of such a type: Its values can either be null, or ?v for some value v of type t, and the primitive operations on this type are the two introduction forms, an analysis function, and a projection for non-null values:

null : () -> ?t
some : t -> ?t
is_some : ?t -> bool
project : ?t -> t     // must only be used if is_some holds

It is natural to use the heap tag to distinguish the two kind of values:

• The null value is a simple one-word heap object with just a tag that says that this is null:

   
    null  
   

• The other values are represented by a two-word object: The tag some, indicating that it is a ?v, and then the payload, which is the pointer that represents the value v:

   
    some   payload  
   

With this, the four operations can be implemented as follows:

def null():
  ptr <- alloc(1)
  ptr[0] = NULL_TAG
  return ptr
def some(v):
  ptr <- alloc(2)
  ptr[0] = SOME_TAG
  ptr[1] = v
  return ptr
def is_some(p):
  return p[0] == SOME_TAG
def project(p):
  return p[1]

The problem with this implementation is that null() and some(v) both allocate memory. This is bad if they are used very often, and very liberally, and this is the case in Motoko: For example the iterators used for the for (x in e) construct have type

type Iter<T> =   next : () -> ?T  

and would unavoidably allocate a few words on each iteration. Can we avoid this?

Static values It is quite simple to avoid this for for null: Just statically create a single null value and use it every time:

static_null = [NULL_TAG]
def null():
  return static_null

This way, at least null() doesn t allocate. But we gain more: Now every null value is represented by the same pointer, and since the pointer points to static memory, it does not change even with a moving garbage collector, so we can speed up is_some:

def is_some(p):
  return p != static_null

This is not a very surprising change so far, and most compilers out there can and will do at least the static allocation of such singleton constructors. For example, in Haskell, there is only a single empty list ([]) and a single Nothing value in your program, as you can see in my videos exploring the Haskell heap. But can we get rid of the allocation in some(v) too?

Unwrapped optional values If we don t want to allocate in some(v), what can we do? How about simply

def some(v):
  return v

That does not allocate! But it is also broken. At type ??Int, the values null, ?null and ??null are distinct values, but here this breaks. Or, more formally, the following laws should hold for our four primitive operations:

1. is_some(null()) = false
2. v. is_some(some(v)) = true
3. p. project(some(p)) = p

But with the new definition of some, we d get is_some(some(null())) = false. Not good! But note that we only have a problem if we are wrapping a value that is null or some(v). So maybe take the shortcut only then, and write the following:

def some(v):
  if v == static_null   v[0] == SOME_TAG:
    ptr <- alloc(2)
    ptr[0] = SOME_TAG
    ptr[1] = v
    return ptr
  else:
    return v

The definition of is_some can stay as it is: It is still the case that all null values are represented by static_null. But the some values are now no longer all of the same shape, so we have to change project():

def project(p):
  if p[0] == SOME_TAG:
    return p[1]
  else:
    return p

Now things fall into place: A value ?v can, in many cases, be represented the same way as v, and no allocation is needed. Only when v is null or ?null or ??null or ???null etc. we need to use the some heap object, and thus have to allocate. In fact, it doesn t cost much to avoid allocation even for ?null:

static_some_null = [SOME_TAG, static_null]
def some(v):
  if v == static_null:
    return static_some_null
  else if v[0] == SOME_TAG:
    ptr <- alloc(2)
    ptr[0] = SOME_TAG
    ptr[1] = v
    return ptr
  else:
    return v

So unless one nests the ? type two levels deep, there is no allocation whatsoever, and the only cost is a bit more branching in some and project. That wasn t hard, but quite rewarding, as one can now use idioms like the iterator shown above with greater ease.

Examples The following tables shows the representation of various values before and after. Here [ ] is a pointed-to dynamically allocated heap object, a statically allocated heap object, N = NULL_TAG and S = SOME_TAG.

type	value	before	after
Null	null	N	N
?Int	null	N	N
?Int	?23	[S,23]	23
??Int	null	N	N
??Int	?null	[S, N ]	S, N
??Int	??23	[S,[S,23]]	23
???Int	null	N	N
???Int	?null	[S, N ]	S, N
???Int	??null	[S,[S, N ]]	[S, S, N ]
???Int	???23	[S,[S,[S,23]]]	23

Concluding remarks

• If you know what parametric polymorphism is, and wonder how this encoding can work in a language that has that, note that this representation is on the low-level of the untyped run-time value representation: We don t need to know the type of v in some(v), merely its heap representation.
• The uniform representation in Motoko is a bit more involved: The pointers are tagged (by subtracting 1) and some scalar values are represented directly in place (shifted left by 1 bit). But this is luckily orthogonal to what I showed here.
• Could Haskell do this for its Maybe type? Not so easily:
  • The Maybe type is not built-in, but rather a standard library-defined algebraic data type. But the compiler could feasible detect that this is option-like?
  • Haskell is lazy, so at runtime, the type Maybe could be Nothing, or Just v, or, and this is crucial, a yet to be evaluated expression, also called a thunk. And one definitely needs to distinguish between a thunk t :: Maybe a that may evaluate to Nothing, and a value Just t :: Maybe a that definitely is Just, but contains a value, which may be a thunk.
  Something like this may work for a strict Maybe type or unlifted sums like (# (# #) a #), but may clash with other ticks in GHC, such as pointer tagging.
• As I said above, I don t expect this to be novel, and I am happy to add references to prior art here.
• Given that a heap object with tag SOME_TAG now always encodes a tower ? null for n>0, one could try to optimize that even more by just storing the n:

   
    some    n   
   

  But that seems unadvisable: It is only a win if you have deep towers, which is probably rare. Worse, now the project function would have to return such a heap object with n decremented, so now projection might have to allocate, which goes against the cost model expected by the programmer.
• If you rather want to see code than blog posts, feel free to check out Motoko PR #2115.
• Does this kind of stuff excite you? Motoko is open source, so your contributions may be welcome!

For some months now I ve been working on some patches to Consfigurator to add support for Linux containers. My goal is to make Consfigurator capable of both performing the initial setup of a container and of entering the running container to apply configuration. For the case of unprivileged LXCs running as non-root, my work-in-progress branch can now do both of these things. As Consfigurator enters the container directly using system calls, it should be decently fast at configuring multiple containers on a host, and it will also be possible to have it do this in parallel. The initial setup for the container uses Consfigurator s existing support for building root filesystems, and it should be easy to extend that to support arbitrary GNU/Linux distributions by teaching Consfigurator how to invoke bootstrapping tools other than debootstrap(8). Here s an example:

(defhost lxc1.silentflame.com ()
  (os:debian-stable "bullseye" :amd64)
  (basic-props)
  (apt:installed "systemd" "netcat")
  (apache:https-vhost ...))
(defhost lxctest.laptop.silentflame.com ()
  (os:debian-stable "bullseye" :amd64)
  (apt:proxy "http://192.168.122.1:3142")
  (basic-props)
  (apt:installed "linux-image-amd64" "lxc")
  (lxc:usernet-usable-by "spwhitton" "lxcbr0")
  (lxc:user-containers-autostart "spwhitton")
  (lxc:user-container-for '(:additional-lines
                            ("lxc.net.0.type = veth"
                             "lxc.net.0.flags = up"
                             "lxc.net.0.link = lxcbr0"
                             ...))
                          "spwhitton"
                          lxc1.silentflame.com))
(defhost laptop.silentflame.com ()
  ...
  (libvirt:kvm-boots-chroot-for '(:always-deploys t)
                                lxctest.laptop.silentflame.com))

This code is a simplified definition of my testing setup for this work. It defines three hosts: a container lxc1, a container host lxctest, and my laptop. When Consfigurator is asked to deploy the laptop, it will set up the root filesystem for lxctest and then boot it as a KVM virtual machine. Preparing that root filesystem will include setting up the root filesystem for lxc1, too, including shifting the ownership and ACLs to match the user namespace LXC will use when booting the container. Thus, once the deployment of the laptop is finished, it will be possible to boot the lxctest VM, connect to it as the user spwhitton, and start lxc1. Consfigurator includes only minimal support for setting up container networking, as there are so many different ways in which you might want to do it. In my own consfig I ve been developing properties to connect containers directly to my tinc VPN. A single tinc daemon runs on the container host, and other tinc daemons route a whole subnet, containing the addresses for each of the containers, to the container host s tinc daemon. As the LXCs Consfigurator sets up run as non-root, some sort of setuid facility is required to configure this networking. Consfigurator s ability to dump executable Lisp images is helping here. I define a function which runs as root to set up the networking:

(defun route-athenet-container-veth (host)
  (let ((user (getenv "USERV_USER"))
        (peer (getenv "USERV_U_PEER"))
        (ip (car (uiop:command-line-arguments))))
    (unless (string-prefix-p (format nil "veth~D_" (getenv "USERV_UID")) peer)
      (error "~A does not belong to requester." peer))
    (unless (member (cons user ip) (get-hostattrs 'veth-ips host) :test #'equal)
      (error "~A does not have permission to route ~A." user ip))
    (flet ((r (&rest args)
             ;; Explicitly passing nil means UIOP will not invoke a shell.
             (run-program args :force-shell nil)))
      (eswitch ((getenv "USERV_U_HOOK_TYPE") :test #'string=)
        ("up"
         (apply #'r
                "sysctl" "-w"
                "net.ipv6.conf.all.forwarding=1"
                ...)
         (r "ip" "addr" "flush" "dev" peer "scope" "link")
         (r "ip" "-6" "addr" "add" "fe80::1/64" "dev" peer)
         (r "ip" "-6" "route" "add" (strcat ip "/128") "dev" peer)
         ...)
        ("down"
         ...
         (r "ip" "-6" "route" "del" (strcat ip "/128") "dev" peer))))))

and then apply the following property to lxctest to dump an image which will call this function and then exit:

(image-dumped
 "/usr/lib/userv/route-athenet-container-veth"
  (route-athenet-container-veth ,(intern (string-upcase (get-hostname)))))

I m using GNU userv to enable ordinary users to run this image as root, so there there s a small script which converts LXC s LXC_HOOK_* environment variables into appropriate command line arguments to userv(1) such that the function above is able to access that information from its environment (the USERV_U_* variables above). You could just as easily do this with sudo, by giving permission for the relevant LXC_HOOK_* environment variables to survive the switch to root. What s particularly nice about this is that there s no need to write any code to keep a config file updated, specifying which users are allowed to route which IPs to their containers. ROUTE-ATHENET-CONTAINER-VETH receives a HOST value for the container host and can just look at the metadata established by properties for particular containers. Each time this metadata is updated and lxctest is deployed, a fresh image is dumped containing the updated metadata. This work has provided opportunities to make various other improvements to Consfigurator, especially with regard to dumping and reinvoking images. Making SBCL capable of entering user namespaces required a change upstream, which made it into the recent SBCL 2.1.8 release. I m very grateful to the SBCL developers for their engagement with my project. I ve been able to add a workaround so that Consfigurator can still enter user namespaces when run on the version of SBCL included in Debian stable. I also discovered that deploying all of my laptop, lxctest and lxc1 at once generates enough output to fill up a pipe, thus revealing a deadlock in Consfigurator s IPC, which it was good to become aware of and fix. That involved writing my first multi-threaded Lisp, as there are two pipes that need to be kept from filling up, and to my surprise it worked first time. Take that Haskell

I grew up with the Internet and its ethics and politics have always been important in my life. But I have also been involved at other levels, against police brutality, for Food, Not Bombs, worker autonomy, software freedom, etc. For a long time, that all seemed coherent. But the more I look at the modern Internet -- and the mega-corporations that control it -- and the less confidence I have in my original political analysis of the liberating potential of technology. I have come to believe that most of our technological development is harmful to the large majority of the population of the planet, and of course the rest of the biosphere. And now I feel this is not a new problem. This is because the Internet is a neo-colonial device, and has been from the start. Let me explain.

What is Neo-Colonialism? The term "neo-colonialism" was coined by Kwame Nkrumah, first president of Ghana. In Neo-Colonialism, the Last Stage of Imperialism (1965), he wrote:

In place of colonialism, as the main instrument of imperialism, we have today neo-colonialism ... [which] like colonialism, is an attempt to export the social conflicts of the capitalist countries. ... The result of neo-colonialism is that foreign capital is used for the exploitation rather than for the development of the less developed parts of the world. Investment, under neo-colonialism, increases, rather than decreases, the gap between the rich and the poor countries of the world.

So basically, if colonialism is Europeans bringing genocide, war, and its religion to the Africa, Asia, and the Americas, neo-colonialism is the Americans (note the "n") bringing capitalism to the world. Before we see how this applies to the Internet, we must therefore make a detour into US history. This matters, because anyone would be hard-pressed to decouple neo-colonialism from the empire under which it evolves, and here we can only name the United States of America.

US Declaration of Independence Let's start with the United States declaration of independence (1776). Many Americans may roll their eyes at this, possibly because that declaration is not actually part of the US constitution and therefore may have questionable legal standing. Still, it was obviously a driving philosophical force in the founding of the nation. As its author, Thomas Jefferson, stated:

it was intended to be an expression of the American mind, and to give to that expression the proper tone and spirit called for by the occasion

In that aging document, we find the following pearl:

We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness.

As a founding document, the Declaration still has an impact in the sense that the above quote has been called an:

"immortal declaration", and "perhaps [the] single phrase" of the American Revolutionary period with the greatest "continuing importance." (Wikipedia)

Let's read that "immortal declaration" again: "all men are created equal". "Men", in that context, is limited to a certain number of people, namely "property-owning or tax-paying white males, or about 6% of the population". Back when this was written, women didn't have the right to vote, and slavery was legal. Jefferson himself owned hundreds of slaves. The declaration was aimed at the King and was a list of grievances. A concern of the colonists was that the King:

has excited domestic insurrections amongst us, and has endeavoured to bring on the inhabitants of our frontiers, the merciless Indian Savages whose known rule of warfare, is an undistinguished destruction of all ages, sexes and conditions.

This is a clear mark of the frontier myth which paved the way for the US to exterminate and colonize the territory some now call the United States of America. The declaration of independence is obviously a colonial document, having being written by colonists. None of this is particularly surprising, historically, but I figured it serves as a good reminder of where the Internet is coming from, since it was born in the US.

A Declaration of the Independence of Cyberspace Two hundred and twenty years later, in 1996, John Perry Barlow wrote a declaration of independence of cyberspace. At this point, (almost) everyone has a right to vote (including women), slavery was abolished (although some argue it still exists in the form of the prison system); the US has made tremendous progress. Surely this text will have aged better than the previous declaration it is obviously derived from. Let's see how it reads today and how it maps to how the Internet is actually built now.

Borders of Independence One of the key ideas that Barlow brings up is that "cyberspace does not lie within your borders". In that sense, cyberspace is the final frontier: having failed to colonize the moon, Americans turn inwards, deeper into technology, but still in the frontier ideology. And indeed, Barlow is one of the co-founder of the Electronic Frontier Foundation (the beloved EFF), founded six years prior. But there are other problems with this idea. As Wikipedia quotes:

The declaration has been criticized for internal inconsistencies.[9] The declaration's assertion that 'cyberspace' is a place removed from the physical world has also been challenged by people who point to the fact that the Internet is always linked to its underlying geography.[10]

And indeed, the Internet is definitely a physical object. First controlled and severely restricted by "telcos" like AT&T, it was somewhat "liberated" from that monopoly in 1982 when an anti-trust lawsuit broke up the monopoly, a key historical event that, one could argue, made the Internet possible. (From there on, "backbone" providers could start competing and emerge, and eventually coalesce into new monopolies: Google has a monopoly on search and advertisement, Facebook on communications for a few generations, Amazon on storage and computing, Microsoft on hardware, etc. Even AT&T is now pretty much as consolidated as it was before.) The point is: all those companies have gigantic data centers and intercontinental cables. And those are definitely prioritizing the western world, the heart of the empire. Take for example Google's latest 3,900 mile undersea cable: it does not connect Argentina to South Africa or New Zealand, it connects the US to UK and Spain. Hardly a revolutionary prospect.

Private Internet But back to the Declaration:

Do not think that you can build it, as though it were a public construction project. You cannot. It is an act of nature and it grows itself through our collective actions.

In Barlow's mind, the "public" is bad, and private is good, natural. Or, in other words, a "public construction project" is unnatural. And indeed, the modern "nature" of development is private: most of the Internet is now privately owned and operated. I must admit that, as an anarchist, I loved that sentence when I read it. I was rooting for "us", the underdogs, the revolutionaries. And, in a way, I still do: I am on the board of Koumbit and work for a non-profit that has pivoted towards censorship and surveillance evasion. Yet I cannot help but think that, as a whole, we have failed to establish that independence and put too much trust in private companies. It is obvious in retrospect, but it was not, 30 years ago. Now, the infrastructure of the Internet has zero accountability to traditional political entities supposedly representing the people, or even its users. The situation is actually worse than when the US was founded (e.g. "6% of the population can vote"), because the owners of the tech giants are only a handful of people who can override any decision. There's only one Amazon CEO, he's called Jeff Bezos, and he has total control. (Update: Bezos actually ceded the CEO role to Andy Jassy, AWS and Amazon music founder, while remaining executive chairman. I would argue that, as the founder and the richest man on earth, he still has strong control over Amazon.)

Social Contract Here's another claim of the Declaration:

We are forming our own Social Contract.

I remember the early days, back when "netiquette" was a word, it did feel we had some sort of a contract. Not written in standards of course -- or barely (see RFC1855) -- but as a tacit agreement. How wrong we were. One just needs to look at Facebook to see how problematic that idea is on a global network. Facebook is the quintessential "hacker" ideology put in practice. Mark Zuckerberg explicitly refused to be "arbiter of truth" which implicitly means he will let lies take over its platforms. He also sees Facebook as place where everyone is equal, something that echoes the Declaration:

We are creating a world that all may enter without privilege or prejudice accorded by race, economic power, military force, or station of birth.

(We note, in passing, the omission of gender in that list, also mirroring the infamous "All men are created equal" claim of the US declaration.) As the Wall Street Journal's (WSJ) Facebook files later shown, both of those "contracts" have serious limitations inside Facebook. There are VIPs who systematically bypass moderation systems including fascists and rapists. Drug cartels and human traffickers thrive on the platform. Even when Zuckerberg himself tried to tame the platform -- to get people vaccinated or to make it healthier -- he failed: "vaxxer" conspiracies multiplied and Facebook got angrier. This is because the "social contract" behind Facebook and those large companies is a lie: their concern is profit and that means advertising, "engagement" with the platform, which causes increased anxiety and depression in teens, for example. Facebook's response to this is that they are working really hard on moderation. But the truth is that even that system is severely skewed. The WSJ showed that Facebook has translators for only 50 languages. It's a surprisingly hard to count human languages but estimates range the number of distinct languages between 2500 and 7000. So while 50 languages seems big at first, it's actually a tiny fraction of the human population using Facebook. Taking the first 50 of the Wikipedia list of languages by native speakers we omit languages like Dutch (52), Greek (74), and Hungarian (78), and that's just a few random nations picks from Europe. As an example, Facebook has trouble moderating even a major language like Arabic. It censored content from legitimate Arab news sources when they mentioned the word al-Aqsa because Facebook associates it with the al-Aqsa Martyrs' Brigades when they were talking about the Al-Aqsa Mosque... This bias against Arabs also shows how Facebook reproduces the American colonizer politics. The WSJ also pointed out that Facebook spends only 13% of its moderation efforts outside of the US, even if that represents 90% of its users. Facebook spends three more times moderating on "brand safety", which shows its priority is not the safety of its users, but of the advertisers.

Military Internet Sergey Brin and Larry Page are the Lewis and Clark of our generation. Just like the latter were sent by Jefferson (the same) to declare sovereignty over the entire US west coast, Google declared sovereignty over all human knowledge, with its mission statement "to organize the world's information and make it universally accessible and useful". (It should be noted that Page somewhat questioned that mission but only because it was not ambitious enough, Google having "outgrown" it.) The Lewis and Clark expedition, just like Google, had a scientific pretext, because that is what you do to colonize a world, presumably. Yet both men were military and had to receive scientific training before they left. The Corps of Discovery was made up of a few dozen enlisted men and a dozen civilians, including York an African American slave owned by Clark and sold after the expedition, with his final fate lost in history. And just like Lewis and Clark, Google has a strong military component. For example, Google Earth was not originally built at Google but is the acquisition of a company called Keyhole which had ties with the CIA. Those ties were brought inside Google during the acquisition. Google's increasing investment inside the military-industrial complex eventually led Google to workers organizing a revolt although it is currently unclear to me how much Google is involved in the military apparatus. Other companies, obviously, do not have such reserve, with Microsoft, Amazon, and plenty of others happily bidding on military contracts all the time.

Spreading the Internet I am obviously not the first to identify colonial structures in the Internet. In an article titled The Internet as an Extension of Colonialism, Heather McDonald correctly identifies fundamental problems with the "development" of new "markets" of Internet "consumers", primarily arguing that it creates a digital divide which creates a "lack of agency and individual freedom":

Many African people have gained access to these technologies but not the freedom to develop content such as web pages or social media platforms in their own way. Digital natives have much more power and therefore use this to create their own space with their own norms, shaping their online world according to their own outlook.

But the digital divide is certainly not the worst problem we have to deal with on the Internet today. Going back to the Declaration, we originally believed we were creating an entirely new world:

This governance will arise according to the conditions of our world, not yours. Our world is different.

How I dearly wished that was true. Unfortunately, the Internet is not that different from the offline world. Or, to be more accurate, the values we have embedded in the Internet, particularly of free speech absolutism, sexism, corporatism, and exploitation, are now exploding outside of the Internet, into the "real" world. The Internet was built with free software which, fundamentally, was based on quasi-volunteer labour of an elite force of white men with obviously too much time on their hands (and also: no children). The mythical writing of GCC and Emacs by Richard Stallman is a good example of this, but the entirety of the Internet now seems to be running on random bits and pieces built by hit-and-run programmers working on their copious free time. Whenever any of those fails, it can compromise or bring down entire systems. (Heck, I wrote this article on my day off...) This model of what is fundamentally "cheap labour" is spreading out from the Internet. Delivery workers are being exploited to the bone by apps like Uber -- although it should be noted that workers organise and fight back. Amazon workers are similarly exploited beyond belief, forbidden to take breaks until they pee in bottles, with ambulances nearby to carry out the bodies. During peak of the pandemic, workers were being dangerously exposed to the virus in warehouses. All this while Amazon is basically taking over the entire economy. The Declaration culminates with this prophecy:

We will spread ourselves across the Planet so that no one can arrest our thoughts.

This prediction, which first felt revolutionary, is now chilling.

Colonial Internet The Internet is, if not neo-colonial, plain colonial. The US colonies had cotton fields and slaves, we have disposable cell phones and Foxconn workers. Canada has its cultural genocide, Facebook has his own genocides in Ethiopia, Myanmar, and mob violence in India. Apple is at least implicitly accepting the Uyghur genocide. And just like the slaves of the colony, those atrocities are what makes the empire run. The Declaration actually ends like this, a quote which I have in my fortune cookies file:

We will create a civilization of the Mind in Cyberspace. May it be more humane and fair than the world your governments have made before.

That is still inspiring to me. But if we want to make "cyberspace" more humane, we need to decolonize it. Work on cyberpeace instead of cyberwar. Establish clear code of conduct, discuss ethics, and question your own privileges, biases, and culture. For me the first step in decolonizing my own mind is writing this article. Breaking up tech monopolies might be an important step, but it won't be enough: we have to do a culture shift as well, and that's the hard part.

Appendix: an apology to Barlow I kind of feel bad going through Barlow's declaration like this, point by point. It is somewhat unfair, especially since Barlow passed away a few years ago and cannot mount a response (even humbly assuming that he might read this). But then again, he himself recognized he was a bit too "optimistic" in 2009, saying: "we all get older and smarter":

I'm an optimist. In order to be libertarian, you have to be an optimist. You have to have a benign view of human nature, to believe that human beings left to their own devices are basically good. But I'm not so sure about human institutions, and I think the real point of argument here is whether or not large corporations are human institutions or some other entity we need to be thinking about curtailing. Most libertarians are worried about government but not worried about business. I think we need to be worrying about business in exactly the same way we are worrying about government.

And, in a sense, it was a little naive to expect Barlow to not be a colonist. Barlow is, among many things, a cattle rancher who grew up on a colonial ranch in Wyoming. The ranch was founded in 1907 by his great uncle, 17 years after the state joined the Union, and only a generation or two after the Powder River War (1866-1868) and Black Hills War (1876-1877) during which the US took over lands occupied by Lakota, Cheyenne, Arapaho, and other native American nations, in some of the last major First Nations Wars.

Appendix: further reading There is another article that almost has the same title as this one: Facebook and the New Colonialism. (Interestingly, the <title> tag on the article is actually "Facebook the Colonial Empire" which I also find appropriate.) The article is worth reading in full, but I loved this quote so much that I couldn't resist reproducing it here:

Representations of colonialism have long been present in digital landscapes. ( Even Super Mario Brothers, the video game designer Steven Fox told me last year. You run through the landscape, stomp on everything, and raise your flag at the end. ) But web-based colonialism is not an abstraction. The online forces that shape a new kind of imperialism go beyond Facebook.

It goes on:

Consider, for example, digitization projects that focus primarily on English-language literature. If the web is meant to be humanity s new Library of Alexandria, a living repository for all of humanity s knowledge, this is a problem. So is the fact that the vast majority of Wikipedia pages are about a relatively tiny square of the planet. For instance, 14 percent of the world s population lives in Africa, but less than 3 percent of the world s geotagged Wikipedia articles originate there, according to a 2014 Oxford Internet Institute report.

And they introduce another definition of Neo-colonialism, while warning about abusing the word like I am sort of doing here:

I m loath to toss around words like colonialism but it s hard to ignore the family resemblances and recognizable DNA, to wit, said Deepika Bahri, an English professor at Emory University who focuses on postcolonial studies. In an email, Bahri summed up those similarities in list form:

1. ride in like the savior
2. bandy about words like equality, democracy, basic rights
3. mask the long-term profit motive (see 2 above)
4. justify the logic of partial dissemination as better than nothing
5. partner with local elites and vested interests
6. accuse the critics of ingratitude

In the end, she told me, if it isn t a duck, it shouldn t quack like a duck.

Another good read is the classic Code and other laws of cyberspace (1999, free PDF) which is also critical of Barlow's Declaration. In "Code is law", Lawrence Lessig argues that:

computer code (or "West Coast Code", referring to Silicon Valley) regulates conduct in much the same way that legal code (or "East Coast Code", referring to Washington, D.C.) does (Wikipedia)

And now it feels like the west coast has won over the east coast, or maybe it recolonized it. In any case, Internet now christens emperors.

Dropdown for GitHub workflows input parameters Sometimes when we look at CI/CD tools embedded within git-based software repository manager like GitHub, GitLab or Bitbucket, we ran into a lack of some features. This time me and my DevOps/SRE team were facing a pain of not being able to have the option to create drop-downs within GitHub workflows using input parameters. Although this functionality is already available on other platforms such as Bitbucket, the specific client we were working on stored the code inside GitHub. At first I thought that someone has already solved this problem somehow, but doing an extensive search on the internet I found several angry GitHub users opening requests within the Support Community and even in the stack overflow. So I decided to create a solution for this, always thinking about simplicity and in a way that makes it easy to get this missing functionality. I started by creating an input array pattern using commas and using a tag (the selector) e.g brackets as the default value marker. Here is an example of what an input string would look like:

name: gh-action-dropdown-list-input
on:
  workflow_dispatch:
    inputs:
      environment:
        description: 'Environment'
        required: true
        default: 'dev,staging,[uat],prod'

Now the final question that would turn out to be the most complicated to deal with. How can I change the GitHub Actions interface to replace the input pattern we created earlier to a dropdown? The simplest answer I thought was to create a chrome and firefox extension that would do all this logic behind the scenes and replace the HTML input element with the selected tag containing the array values and leaving the tag value (selector) always as the default. All code was developed in pure JavaScript, open-source licensed under Apache 2.0 and available at https://github.com/arthurbdiniz/gh-action-dropdown-list-input.

Install extension

• Chrome
• Firefox

Once installed, the extension is ready to use and the final result we see is the Actions interface with drop-downs. :)

Configuring selectors Go to the top right corner of the browser you are using and click on the extension logo. A screen will popup with tag options. Choose the right tags for you and save it.

This action might require reloading the GitHub workflow tab.

---

Have fun using drop-downs inside GitHub. If you liked this project please share this post and if possible star within the repository. Also feel free to connect with me on LinkedIn: https://www.linkedin.com/in/arthurbdiniz

References

• https://github.community/t/can-workflow-dispatch-input-be-option-list/127338
• https://stackoverflow.com/questions/69296314/dropdown-for-github-workflows-input-parameters

Almost a year ago I added a script made by Helmut Grohne that is able to create a CMake toolchain file pre-filled with Debian-specifics ross compilers. The tool is installed by the cmake package and located at /usr/share/cmake/debtoolchainfilegen. It's usage is simple: Where $arch can be any of the Debian supported architectures, like arm64 (aka aarch64): Note: I kept the warnings, which can be ignored and won't end up on the final file. As you might have noticed the file itself has instructions on how to use it. Of course we will need the requires cross toolchain for the selected arch. For example using arm64: That's it, we can now start cross building our cmake-based software.

This post was originally published in the Wikimedia Tech blog, authored by Arturo Borrero Gonzalez. NFS is a central piece of infrastructure that is essential to services like Toolforge. Recently, the Cloud Services team at Wikimedia had been reviewing how we do NFS. The current situation NFS is a central piece of technology for some of the services that the Wikimedia Cloud Services team offers to the community. We have several shares that power different use cases: Toolforge user home directories live on NFS, and Cloud VPS users can also access dumps using this protocol. The current setup involves several physical hardware servers, with about 20TB of storage, offering shares over 10G links to the cloud. For the system to be more fault-tolerant, we duplicate each share for redundancy using DRBD. Running NFS on dedicated hardware servers has traditionally offered us advantages: mostly on the performance and the capacity fields. As time has passed, we have been enumerating more and more reasons to review how we do NFS. For one, the current setup is in violation of some of our internal rules regarding realm separation. Additionally, we had been longing for additional flexibility managing our servers: we wanted to use virtual machines managed by Openstack Nova. The DRBD-based high-availability system required mostly a hand-crafted procedure for failover/failback. There s also some scalability concerns as NFS is easy to grow up, but not to grow horizontally, and of course, we have to be able to keep the tenancy setup while doing so, something that NFS does by using LDAP/Unix users and may get complicated too when growing. In general, the servers have become too big to fail , clearly technical debt, and it has taken us years to decide on taking on the task to rethink the architecture. It s worth mentioning that in an ideal world, we wouldn t depend on NFS, but the truth is that it will still be a central piece of infrastructure for years to come in services like Toolforge. Over a series of brainstorming meetings, the WMCS team evaluated the situation and sorted out the many moving parts. The team managed to boil down the potential service future to two competing options:

• Adopt and introduce a new Openstack component into our cloud: Manila this was the right choice if we were interested in a general NFS as a service offering for our Cloud VPS users.
• Put the data on Cinder volumes and serve NFS from a couple of virtual machines created by hand this was the right choice if we wanted something that required low effort to engineer and adopt.

Then we decided to research both options in parallel. For a number of reasons, the evaluation was timeboxed to three weeks. Both ideas had a couple of points in common: the NFS data would be stored on our Ceph farm via Cinder volumes, and we would rely on Ceph reliability to avoid using DRBD. Another open topic was how to back up data from Ceph, to store our important bits in more than one basket. We will get to the back up topic later. The manila experiment The Wikimedia Foundation was an early adopter of some Openstack components (Nova, Glance, Designate, Horizon), but Manila was never evaluated for usage until now. Our approach for this experiment was to closely follow the upstream guidelines. We read the documentation and tried to understand the different setups you can build with Manila. As we often feel with other Openstack components, the documentation doesn t perfectly describe how to introduce a given component in your particular local setup. Here we use an admin-controller flat-topology Neutron network. This network is shared by all tenants (or projects) in our Openstack deployment. Also, Manila can use many different driver backends, for things like NetApps or CephFS that we don t use , yet. After some research, the generic driver was the one that seemed to better fit our use case. The generic driver leverages Nova virtual machines instances plus Cinder volume to create and manage the shares. In general, Manila supports two operational modes, whether it should create/destroy the share servers (i.e, the virtual machine instances) or not. This option is called driver_handles_share_server (or DHSS) and takes a boolean value. We were interested in trying with DHSS=true, to really benefit from the potential of the setup. NFS idea 6, original image in Wikitech So, after sorting all these variables, we moved on with our initial testing. We built a PoC setup as depicted in the diagram above, with the manila-share component running in a virtual machine inside the cloud. The PoC led to us reporting several bugs upstream:

• #1946002 generic driver: service instance ends up with 2 ports in the admin network
• #1945463 generic driver: service instance name template is ignored
• #1944980 generic driver: support additional SSH options for service instance
• #1944696 [DOC] document additional generic driver configuration options

In some cases we tried to address these bugs ourselves:

• https://review.opendev.org/c/openstack/manila/+/810702
• https://review.opendev.org/c/openstack/manila/+/810719
• https://review.opendev.org/c/openstack/manila/+/811736

It s worth mentioning that the upstream community was extra-welcoming to us, and we re thankful for that. However, at the end of our three-week period, our Manila setup still wasn t working as expected. Your experience may change with other drivers perhaps the ZFSonLinux or the CephFS ones. In general, we were having trouble making the setup work as expected, so we decided to abandon this approach in favor of the other option we were considering at the beginning. Simple virtual machine serving NFS The alternative was to create a Nova virtual machine instance by hand and to configure it using puppet. We have been investing in an automation framework lately, so the idea is to not actually create the server by hand. Anyway, the data would be decoupled from the instance into Cinder volumes, which led us to the question we left for later: How should we back up those terabytes of important information? Just to be clear, the backup problem was independent of the above options; with Manila we would still have had to solve the same challenge. We would like to see our data be backed up somewhere else other than in Ceph. And that s exactly where we are at right now. We ve been exploring different backup strategies and will finally use the Cinder backup API. Conclusion The iteration will end with the dedicated NFS hardware servers being stopped, and the shares being served from within the cloud. The migration will take some time to happen because we will check and double-check that everything works as expected (including from the performance point of view) before making definitive changes. We already have some plans to make sure our users experience as little service impact as possible. The most troublesome shares will be those related to Toolforge. At some point we will need to disallow writes to the NFS share, rsync the data out of the hardware servers into the Cinder volumes, point the NFS clients to the new virtual machines, and then enable writes again. The main Toolforge share has about 8TB of data, so this will take a while. We will have more updates in the future. Who knows, perhaps our next-next iteration, in a couple of years, will see us adopting Openstack Manila for good. Featured image credit: File:(from break water) Manila Skyline panoramio.jpg, ewol, CC BY-SA 3.0 This post was originally published in the Wikimedia Tech blog, authored by Arturo Borrero Gonzalez.

Like each month, have a look at the work funded by Freexian s Debian LTS offering. Debian project funding Folks from the LTS team, along with members of the Debian Android Tools team and Phil Morrel, have proposed work on the Java build tool, gradle, which is currently blocked due to the need to build with a plugin not available in Debian. The LTS team reviewed the project submission and it has been approved. After approval we ve created a Request for Bids which is active now. You ll hear more about this through official Debian channels, but in the meantime, if you feel you can help with this project, please submit a bid. Thanks! This September, Freexian set aside 2550 EUR to fund Debian projects. We re looking forward to receive more projects from various Debian teams! Learn more about the rationale behind this initiative in this article. Debian LTS contributors In September, 15 contributors have been paid to work on Debian LTS, their reports are available:

• Abhijith PA has returned hours and marked themselves inactive, at least for the time being. He did 0h out of 14h, carried over 14h and returned 28h.
• Adrian Bunk did 19.5h (out of 24.75h assigned and 12.75 from August), carrying over 18h to October.
• Anton Gladky did 12h (out of 12h assigned).
• Ben Hutchings did 2h (out of 12.75h assigned and 19.25h from August), thus carrying over 30h to October.
• Chris Lamb did 18h (out of 18h assigned).
• Emilio Pozuelo Monfort did not report back about their work so we assume they did nothing (out of 5.5h assigned plus 74.5h from August), thus is carrying over 80h for October.
• Holger Levsen did 3h (out of 12h assigned) and gave back 9h and carried over 3h.
• Jeremiah Foster worked 10 hours (out of 20h assigned) on LTS work, carrying over 10h.
• Lee Garrett did not report back about their work so we assume they did nothing (out of 24.75h assigned and 23.75 from August), thus is carrying over 48.50h for October.
• Markus Koschany did 43.5h (out of 24.75h assigned and 18.75h from August)
• Neil Williams did 24.5h (out of 24.75h assigned)
• Roberto C. S nchez did 6h (out of 24.75h assigned and gave back 18.75h)
• Sylvain Beucler did 27h (out of 24.75h assigned).
• Thorsten Alteholz did 24.75h (out of 24.75h assigned).
• Utkarsh Gupta did 24.75h (out of 24.75h assigned) but did not publish his report yet.
• Ola Lundqvist did 2 hours (out of 21h carried over from previous months), and is thus carrying 19h for October.

Evolution of the situation In September we released 30 DLAs. September was also the second month of Jeremiah coordinating LTS contributors. Also, we would like say that we are always looking for new contributors to LTS. Please contact Jeremiah if you are interested! The security tracker currently lists 33 packages with a known CVE and the dla-needed.txt file has 26 packages needing an update. Thanks to our sponsors Sponsors that joined recently are in bold.

• Platinum sponsors:
  • TOSHIBA (for 73 months)
  • GitHub (for 64 months)
  • Civil Infrastructure Platform (CIP) (for 41 months)
• Gold sponsors:
  • Roche Diagnostics International AG (for 84 months)
  • Linode (for 78 months)
  • Babiel GmbH (for 67 months)
  • Plat Home (for 67 months)
  • University of Oxford (for 23 months)
  • Deveryware (for 10 months)
  • VyOS
• Silver sponsors:
  • The Positive Internet Company (for 89 months)
  • Domeneshop AS (for 88 months)
  • Nantes M tropole (for 82 months)
  • Univention GmbH (for 74 months)
  • Universit Jean Monnet de St Etienne (for 74 months)
  • Ribbon Communications, Inc. (for 68 months)
  • Exonet B.V. (for 58 months)
  • Leibniz Rechenzentrum (for 52 months)
  • CINECA (for 41 months)
  • Minist re de l Europe et des Affaires trang res (for 35 months)
  • Cloudways Ltd (for 24 months)
  • Dinahosting SL (for 22 months)
  • Bauer Xcel Media Deutschland KG (for 17 months)
  • Platform.sh (for 17 months)
  • Moxa Intelligence Co., Ltd. (for 11 months)
  • sipgate GmbH (for 8 months)
  • OVH US LLC (for 6 months)
  • Tilburg University (for 6 months)
• Bronze sponsors:
  • Evolix (for 89 months)
  • Seznam.cz, a.s. (for 89 months)
  • Linuxhotel GmbH (for 86 months)
  • Intevation GmbH (for 85 months)
  • Daevel SARL (for 84 months)
  • Bitfolk LTD (for 83 months)
  • Megaspace Internet Services GmbH (for 83 months)
  • Greenbone Networks GmbH (for 82 months)
  • NUMLOG (for 82 months)
  • WinGo AG (for 81 months)
  • Ecole Centrale de Nantes LHEEA (for 78 months)
  • Entr ouvert (for 73 months)
  • Adfinis AG (for 70 months)
  • Laboratoire LEGI UMR 5519 / CNRS (for 65 months)
  • Tesorion (for 65 months)
  • GNI MEDIA (for 64 months)
  • Bearstech (for 56 months)
  • LiHAS (for 56 months)
  • People Doc (for 52 months)
  • Catalyst IT Ltd (for 51 months)
  • Demarcq SAS (for 45 months)
  • Universit Grenoble Alpes (for 31 months)
  • TouchWeb SAS (for 23 months)
  • SPiN AG (for 19 months)
  • CoreFiling (for 15 months)
  • Institut des sciences cognitives Marc Jeannerod (for 10 months)
  • Observatoire des Sciences de l Univers de Grenoble (for 7 months)
  • Tem Innovations GmbH
  • WordFinder.pro

FTP master This month I accepted 224 and rejected 47 packages. This is almost thrice the rejects of last month. Please, be more careful and check your package twice before uploading. The overall number of packages that got accepted was 233. Debian LTS This was my eighty-seventh month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. This month my all in all workload has been 24.75h. During that time I did LTS and normal security uploads of:

• [DLA 2755-1] btrbk security update for one CVE
• [DLA 2762-1] grilo security update for one CVE
• [DLA 2766-1] openssl security update for one CVE
• [DLA 2774-1] openssl1.0 security update for one CVE
• [DLA 2773-1] curl security update for two CVEs

I also started to work on exiv2 and faad2. Last but not least I did some days of frontdesk duties. Debian ELTS This month was the thirty-ninth ELTS month. Unfortunately during my allocated time I could not process any upload. I worked on openssl, curl and squashfs-tools but for one reason or another the prepared packages didn t pass all tests. In order to avoid regressions, I postponed the uploads (meanwhile an ELA for curl was published ). Last but not least I did some days of frontdesk duties. Other stuff On my neverending golang challenge I again uploaded some packages either for NEW or as source upload. As Odyx took a break from all Debian activities, I volunteered to take care of the printing packages. Please be merciful when somethings breaks after I did an upload. My first printing upload was hplip

I recently did mentoring a new Debian contributor. This is carried out in a framework with OSS Gate on-boarding. oss-gate.github.io In "OSS Gate on-boarding", recruit a new contributor who want to work on continuously. Then, corporation sponsor its employee as a mentor. Thus, employees can do it as a one of their job. During Aug - Oct period, I worked with a new debian contributor every 2h in a week. This experience is lots of fun, and learned a new things for me. The most important point is: a new Debian contributor aimed to do their work continuously even though mentoring period has finished. So, some of the work has been finished, but not for all. I tried to transfer knowledge for it. I'm looking forward that he makes things forward in consultation with other person's help. Here is the report about my activity as a mentor. First OSS Gate onboarding (The article is written by Japanese) The original blog entry is written by Japanese, I don't afford to translate it, so just paste link to google translate for your hints I hope someone can do a similar attempt too! For the record, I worked with a new Debian contributor about:

• ITP: slack-term WIP https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994283
• ITP: golang-github-lithammer-fuzzysearch (already in Debian) https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=992838
• ITP: golang-github-0xax-notificator (waiting ftp-master's review) https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993842
• ITP: golang-github-erroneousboat-termui WIP https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994282
• ITP: golang-github-slack-go-slack WIP https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993615

A new release 0.4.14 of RQuantLib was uploaded to CRAN earlier today, and has by now been uploaded to Debian as well. QuantLib is a very comprehensice free/open-source library for quantitative finance; RQuantLib connects it to the R environment and language. The release of RQuantLib comes just one months after the previous release, and brings three changes. First, we added both two more US-based calendars (including FederalReserve ) along with a bunch of not-yet-included other calendars which should complete the coverage in the R package relative to the upstream library. Should we have forgotten any, feel free to open an issue. Second, CRAN currently aims to have older autoconf conventions updated and notified maintainers of affected packages. I received a handful of these, and just like yesterday s update to littler refreshed this here. Third, we set up automated container builds on GitHub. No other changes were made, details follow.

Changes in RQuantLib version 0.4.14 (2021-10-06)

• Changes in RQuantLib code:
  • Several new calendars were added (Dirk in #159 closing #155)
• Changes in RQuantLib package and setup:
  • Docker containers are now updated on a monthly schedule via GitHub Actions
  • The configure files were updated to the standard of version 2.69 following a CRAN request

Courtesy of my CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the RQuantLib page. Questions, comments etc should go to the new rquantlib-devel mailing list. Issue tickets can be filed at the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

The goal behind reproducible builds is to ensure that no deliberate flaws have been introduced during compilation processes via promising or mandating that identical results are always generated from a given source. This allowing multiple third-parties to come to an agreement on whether a build was compromised or not by a system of distributed consensus. In these reports we outline the most important things that have been happening in the world of reproducible builds in the past month:
First mentioned in our March 2021 report, Martin Heinz published two blog posts on sigstore, a project that endeavours to offer software signing as a public good, [the] software-signing equivalent to Let s Encrypt . The two posts, the first entitled Sigstore: A Solution to Software Supply Chain Security outlines more about the project and justifies its existence:

Software signing is not a new problem, so there must be some solution already, right? Yes, but signing software and maintaining keys is very difficult especially for non-security folks and UX of existing tools such as PGP leave much to be desired. That s why we need something like sigstore - an easy to use software/toolset for signing software artifacts.

The second post (titled Signing Software The Easy Way with Sigstore and Cosign) goes into some technical details of getting started.
There was an interesting thread in the /r/Signal subreddit that started from the observation that Signal s apk doesn t match with the source code:

Some time ago I checked Signal s reproducibility and it failed. I asked others to test in case I did something wrong, but nobody made any reports. Since then I tried to test the Google Play Store version of the apk against one I compiled myself, and that doesn t match either.

BitcoinBinary.org was announced this month, which aims to be a repository of Reproducible Build Proofs for Bitcoin Projects :

Most users are not capable of building from source code themselves, but we can at least get them able enough to check signatures and shasums. When reputable people who can tell everyone they were able to reproduce the project s build, others at least have a secondary source of validation.

Distribution work Fr d ric Pierret announced a new testing service at beta.tests.reproducible-builds.org, showing actual rebuilds of binaries distributed by both the Debian and Qubes distributions. In Debian specifically, however, 51 reviews of Debian packages were added, 31 were updated and 31 were removed this month to our database of classified issues. As part of this, Chris Lamb refreshed a number of notes, including the build_path_in_record_file_generated_by_pybuild_flit_plugin issue. Elsewhere in Debian, Roland Clobus posted his Fourth status update about reproducible live-build ISO images in Jenkins to our mailing list, which mentions (amongst other things) that:

• All major configurations are still built regularly using live-build and bullseye.
• All major configurations are reproducible now; Jenkins is green.
  • I ve worked around the issue for the Cinnamon image.
  • The patch was accepted and released within a few hours.
• My main focus for the last month was on the live-build tool itself.
  • It will properly use the proxy for all HTTP traffic.

Related to this, there was continuing discussion on how to embed/encode the build metadata for the Debian live images which were being worked on by Roland Clobus.
Ariadne Conill published another detailed blog post related to various security initiatives within the Alpine Linux distribution. After summarising some conventional security work being done (eg. with sudo and the release of OpenSSH version 3.0), Ariadne included another section on reproducible builds: The main blocker [was] determining what to do about storing the build metadata so that a build environment can be recreated precisely . Finally, Bernhard M. Wiedemann posted his monthly reproducible builds status report.

Community news On our website this month, Bernhard M. Wiedemann fixed some broken links [ ] and Holger Levsen made a number of changes to the Who is Involved? page [ ][ ][ ]. On our mailing list, Magnus Ihse Bursie started a thread with the subject Reproducible builds on Java, which begins as follows:

I m working for Oracle in the Build Group for OpenJDK which is primary responsible for creating a built artifact of the OpenJDK source code. [ ] For the last few years, we have worked on a low-effort, background-style project to make the build of OpenJDK itself building reproducible. We ve come far, but there are still issues I d like to address. [ ]

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions 183, 184 and 185 as well as performed significant triaging of merge requests and other issues in addition to making the following changes:

• New features:
  • Support a newer format version of the R language s .rds files. [ ]
  • Update tests for OCaml 4.12. [ ]
  • Add a missing format_class import. [ ]
• Bug fixes:
  • Don t call close_archive when garbage collecting Archive instances, unless open_archive definitely returned successfully. This prevents, for example, an AttributeError where PGPContainer s cleanup routines were rightfully assuming that its temporary directory had actually been created. [ ]
  • Fix (and test) the comparison of R language s .rdb files after refactoring temporary directory handling. [ ]
  • Ensure that RPM archives exists in the Debian package description, regardless of whether python3-rpm is installed or not at build time. [ ]
• Codebase improvements:
  • Use our assert_diff routine in tests/comparators/test_rdata.py. [ ]
  • Move diffoscope.versions to diffoscope.tests.utils.versions. [ ]
  • Reformat a number of modules with Black. [ ][ ]

However, the following changes were also made:

• Mattia Rizzolo:
  • Fix an autopkgtest caused by the androguard module not being in the (expected) python3-androguard Debian package. [ ]
  • Appease a shellcheck warning in debian/tests/control.sh. [ ]
  • Ignore a warning from h5py in our tests that doesn t concern us. [ ]
  • Drop a trailing .1 from the Standards-Version field as it s required. [ ]
• Zbigniew J drzejewski-Szmek:
  • Stop using the deprecated distutils.spawn.find_executable utility. [ ][ ][ ][ ][ ]
  • Adjust an LLVM-related test for LLVM version 13. [ ]
  • Update invocations of llvm-objdump. [ ]
  • Adjust a test with a one-byte text file for file version 5.40. [ ]

And, finally, Benjamin Peterson added a --diff-context option to control unified diff context size [ ] and Jean-Romain Garnier fixed the Macho comparator for architectures other than x86-64 [ ].

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:
  • gtk4 (date-related issue)
  • build-compare (random tempfile problem)
  • itinerary (time-related build failure)
• Chris Lamb:
  • #993950 filed against lcalc.
  • #993952 filed against htscodecs.
  • #994123 filed against osdlyrics.
  • #994976 filed against xtermcontrol.
  • #994978 filed against rust-insta.
  • #994979 filed against python-tomli.
  • #995258 filed against python-pairix.
  • #995259 filed against python-pybedtools (forwarded upstream).
• Simon McVittie:
  • #993673 filed against clazy.
  • #993674 filed against ga.
  • #993675 filed against opensmtpd.
  • #993681 filed against sphinxcontrib-programoutput.
• Vagrant Cascadian:
  • #995092 filed against guile-3.0.
  • #995131 filed against nut.
  • #995143 filed against xdmf.
  • #995145 filed against cyrus-sasl2.
  • #995211 filed against libbrahe.
  • #995217 filed against maildirsync.
  • #995401 filed against canna.
  • #990246 filed against vlc (forwarded upstream [ ][ ])

Testing framework The Reproducible Builds project runs a testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:

• Holger Levsen:
  • Drop my package rebuilder prototype as it s not useful anymore. [ ]
  • Schedule old packages in Debian bookworm. [ ]
  • Stop scheduling packages for Debian buster. [ ][ ]
  • Don t include PostgreSQL debug output in package lists. [ ]
  • Detect Python library mismatches during build in the node health check. [ ]
  • Update a note on updating the FreeBSD system. [ ]
• Mattia Rizzolo:
  • Silence a warning from Git. [ ]
  • Update a setting to reflect that Debian bookworm is the new testing. [ ]
  • Upgrade the PostgreSQL database to version 13. [ ]
• Roland Clobus (Debian live image generation):
  • Workaround non-reproducible config files in the libxml-sax-perl package. [ ]
  • Use the new DNS for the snapshot service. [ ]
• Vagrant Cascadian:
  • Also note that the armhf architecture also systematically varies by the kernel. [ ]

Contributing If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Twitter: @ReproBuilds
• Mailing list: rb-general@lists.reproducible-builds.org

Like each month, have a look at the work funded by Freexian s Debian LTS offering. Debian project funding In August, we put aside 2460 EUR to fund Debian projects. We received a new project proposal that got approved and there s an associated bid request if you feel like proposing yourself to implement this project. We re looking forward to receive more projects from various Debian teams! Learn more about the rationale behind this initiative in this article. Debian LTS contributors In August, 14 contributors have been paid to work on Debian LTS, their reports are available:

• Abhijith PA did 4.0h (out of 14h assigned and 5h from August), thus carrying over 15h to September.
• Adrian Bunk did 11h (out of 23.75h assigned), thus carrying over 12.75h to September.
• Anton Gladky did 12h (out of 12h assigned).
• Ben Hutchings did 1.25h (out of 13.25h assigned and 6h from August), thus carrying over 18h to September.
• Chris Lamb did 18h (out of 18h assigned).
• Emilio Pozuelo Monfort did not report back about their work so we assume they did nothing (out of 23.75h assigned plus 50.75h from August), thus is carrying over 74.5h for September.
• Holger Levsen did 3h (out of 12h assigned) to help coordinate the team, and gave back the remaining hours.
• Lee Garrett did nothing (out of 23.75h assigned), thus is carrying over 23.75h for September.
• Markus Koschany did 35h (out of 23.75h assigned and 30h from August), thus carrying over 18.75h to September.
• Neil Williams did 24h (out of 23.75h assigned), thus anticipating 0.25h of October.
• Roberto C. S nchez did 22.25h (out of 23.75h assigned), thus carrying over 1.5h to September.
• Sylvain Beucler did 21.5h (out of 23.75h assigned), thus carrying over 2.25h to September.
• Thorsten Alteholz did 23.75h (out of 23.75h assigned).
• Utkarsh Gupta did 23.75h (out of 23.75h assigned).

Evolution of the situation In August we released 30 DLAs.

This is the first month of Jeremiah coordinating LTS contributors. We would like to thank Holger Levsen for his work on this role up to now.

Also, we would like to remark once again that we are constantly looking for new contributors. Please contact Jeremiah if you are interested! The security tracker currently lists 73 packages with a known CVE and the dla-needed.txt file has 29 packages needing an update. Thanks to our sponsors Sponsors that joined recently are in bold.

• Platinum sponsors:
  • TOSHIBA (for 72 months)
  • GitHub (for 63 months)
  • Civil Infrastructure Platform (CIP) (for 40 months)
• Gold sponsors:
  • Roche Diagnostics International AG (for 83 months)
  • Linode (for 77 months)
  • Babiel GmbH (for 67 months)
  • Plat Home (for 66 months)
  • University of Oxford (for 22 months)
  • Deveryware (for 9 months)
  • VyOS
• Silver sponsors:
  • The Positive Internet Company (for 89 months)
  • Domeneshop AS (for 88 months)
  • Nantes M tropole (for 82 months)
  • Universit Jean Monnet de St Etienne (for 74 months)
  • Univention GmbH (for 73 months)
  • Ribbon Communications, Inc. (for 67 months)
  • Exonet B.V. (for 57 months)
  • Leibniz Rechenzentrum (for 51 months)
  • CINECA (for 41 months)
  • Minist re de l Europe et des Affaires trang res (for 35 months)
  • Cloudways Ltd (for 24 months)
  • Dinahosting SL (for 22 months)
  • Bauer Xcel Media Deutschland KG (for 16 months)
  • Platform.sh (for 16 months)
  • Moxa Intelligence Co., Ltd. (for 10 months)
  • sipgate GmbH (for 7 months)
  • Tilburg University (for 6 months)
  • OVH US LLC (for 5 months)
• Bronze sponsors:
  • Evolix (for 88 months)
  • Seznam.cz, a.s. (for 88 months)
  • Intevation GmbH (for 85 months)
  • Linuxhotel GmbH (for 85 months)
  • Daevel SARL (for 84 months)
  • Bitfolk LTD (for 83 months)
  • Greenbone Networks GmbH (for 82 months)
  • Megaspace Internet Services GmbH (for 82 months)
  • NUMLOG (for 82 months)
  • WinGo AG (for 81 months)
  • Ecole Centrale de Nantes LHEEA (for 77 months)
  • Entr ouvert (for 72 months)
  • Adfinis AG (for 70 months)
  • GNI MEDIA (for 64 months)
  • Laboratoire LEGI UMR 5519 / CNRS (for 64 months)
  • Tesorion (for 64 months)
  • Bearstech (for 56 months)
  • LiHAS (for 56 months)
  • People Doc (for 52 months)
  • Catalyst IT Ltd (for 50 months)
  • Demarcq SAS (for 44 months)
  • Universit Grenoble Alpes (for 30 months)
  • TouchWeb SAS (for 22 months)
  • SPiN AG (for 19 months)
  • CoreFiling (for 15 months)
  • Institut des sciences cognitives Marc Jeannerod (for 9 months)
  • Observatoire des Sciences de l Univers de Grenoble (for 6 months)
  • Tem Innovations GmbH
  • WordFinder.pro

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

• gadget-tool: build cleanup
• bootstrappable.org: https (1 2)
• iotop-py: fix crash, handle Linux kernel API changes (1 2)
• jq: document security issue
• esprima-python: cleanup typos, links, license mistakes
• feed2exec: save page resources in wayback machine plugin
• purple-discord: add missing linking, fix compile warnings
• check-all-the-things: remove duplicate TODO, add pigz-test check
• devscripts: sync BTS tags
• Debian puppet: block BTS spammer
• Debian package uploads: pytest-rerunfailures, purple-discord (1 2), harmony, quesoglc
• Debian wiki pages: AutomatedUpgrade, Debhelper, DebianInstaller/Build, DebianInstaller/Loader/SuspectedVirus, Derivatives/Census (Pengwin, Quirinux), DeveloperNews, Hardware/Database, InstallingDebianOn/Turris/Omnia, Printing, StaticLinking, Statistics, SuitesAndReposExtension, Teams/ReleaseTeam/ReleaseCheckList
• FOSSjobs wiki pages: Resources

Issues

• Crashes in lintian-brush, gnome-shell-extension-shortcuts
• Broken symlinks in speech-dispatcher, ucx
• Dependency bloat in librecaptcha
• Test hangs in pyemd
• Features in apt-listchanges, evolution (1 2 3), baobab, pipewire, reportbug (1 2), systemd-coredump (1 2), find-dbgsym-packages (1 2), gdb
• Syntax highlighting issues in gtksourceview (1 2)
• Outdated links in duktape
• License change needed in cloudflared
• Rebuild needed for ricochet-im
• New versions of pidgin
• Usability issues in GNOME Keyring certificate viewer (also mistakenly filed against Firefox)
• Underlinking in ucx
• False positives in lintian
• Security issues in systemd-coredump (sent privately, rejected)

Review

• Spam: reported 1 Sourceware Bugzilla comment, 1 KDE Bugzilla comment, 580 Debian bug reports and 178 Debian mailing list posts
• Debian packages: sponsored iotop-c backports
• Debian wiki: RecentChanges for the month
• Debian BTS usertags: changes for the month
• Debian screenshots:
  • approved ark audacious bpfmon chameleon-cursor-theme chromium color-picker convertall convert-pgn cpufrequtils cpuid cpuidtool cpulimit cpustat dclock desktop-base docker.io elementary-xfce-icon-theme engrampa ephoto eterm exadrums exifprobe exiftags file-roller firefox fspanel imgp ipython3 linphone-desktop linux-cpupower live-image-xfce-desktop lxqt-archiver mate-desktop mwm network-manager nm-tray pamix pulsemixer python3-pip supertuxkart transmission-remote-gtk xarchiver xfce4 xfce4-datetime-plugin xscreensaver
  • rejected fonts-recommended (just debian/control), obfsproxy (private data), gpart/p7zip/p7zip-full/unzip/unrar (usage), cruft (doesn't show use of tool), weboob-qt/acetoneiso/klettres (screenshot of screenshots website), default-jre-headless/minetest-mod-basic-materials (illustration), gamine/weboob-qt/libstdc++6/med-imaging (photograph), drumstick-tools (wrong package), totem (newspaper article)

Administration

• Debian BTS: reopened bugs closed by a spammer
• Debian wiki: unblock IP addresses, approve accounts

Communication

• Respond to queries from Debian users and contributors on the mailing lists and IRC

Sponsors The purple-discord/harmony/pyemd/librecaptcha/esprima-python work was sponsored by my employer. All other work was done on a volunteer basis.

Using podman for most of my local development environment. For my personal/upstream development I started using podman instead of lxc and pbuilder and other toolings. Most projects provide reasonable docker images (such as rust) and I am happier keeping my environment as a whole stable while I can iterate. I have a Dockerfile for the development environment like this:

Active noise cancellation (ANC) has been all the rage lately in the headphones and in-ear monitors market. It seems after Apple got heavily praised for their AirPods Pro, every somewhat serious electronics manufacturer released their own design incorporating this technology. The first headphones with ANC I remember trying on (in the early 2010s) were the Bose QuietComfort 15. Although the concept did work (they indeed cancelled some sounds), they weren't amazing and did a great job of convincing me ANC was some weird fad for people who flew often. As the years passed, chip size decreased, battery capacity improved and machine learning blossomed truly a perfect storm for the wireless ANC headphones market. I had mostly stayed a sceptic of this tech until recently a kind friend offered to let me try a pair of Sony WH-1000X M3. Having tested them thoroughly, I have to say I'm really tempted to buy them from him, as they truly are fantastic headphones¹. They are very light, comfortable, work without a proprietary app and sound very good with the ANC on² if a little bass-heavy for my taste³. The ANC itself is truly astounding and is leaps and bounds beyond what was available five years ago. It still isn't perfect and doesn't cancel ALL sounds, but transforms the low hum of the subway I find myself sitting in too often these days into a light *swoosh*. When you turn the ANC on, HVAC simply disappears. Most impressive to me is the way they completely cancel the dreaded sound of your footsteps resonating in your headphones when you walk with them. I won't be keeping them though. Whilst I really like what Sony has achieved here, I've grown to understand ANC simply isn't for me. Some of the drawbacks of ANC somewhat bother me: the ear pressure it creates is tolerable, but is an additional energy drain over long periods of time and eventually gives me headaches. I've also found ANC accentuates the motion sickness I suffer from, probably because it messes up with some part of the inner ear balance system. Most of all, I found that it didn't provide noticeable improvements over good passive noise cancellation solutions, at least in terms of how high I have to turn the volume up to hear music or podcasts clearly. The human brain works in mysterious ways and it seems ANC cancelling a class of noises (low hums, constant noises, etc.) makes other noises so much more noticeable. People talking or bursty high pitched noises bothered me much more with ANC on than without. So for now, I'll keep using my trusty Senheiser HD 280 Pro⁴ at work and good in-ear monitors with Comply foam tips on the go.

JMP offers VoIP calling via XMPP, but it's also possibly to use the VoIP using SIP. The underlying VoIP calling functionality in JMP is provided by Bandwidth, but their old Asterisk instructions didn't quite work for me. Here's how I set it up in my Asterisk server.

Get your SIP credentials After signing up for JMP and setting it up in your favourite XMPP client, send the following message to the cheogram.com gateway contact:

reset sip account

In response, you will receive a message containing:

• a numerical username
• a password (e.g. three lowercase words separated by spaces)

Add SIP account to your Asterisk config First of all, I added the following near the top of my /etc/asterisk/sip.conf:

[general]
register => username:three secret words@jmp.cbcbc7.auth.bandwidth.com:5008

The other non-default options I have set in [general] are:

context=public
allowoverlap=no
udpbindaddr=0.0.0.0
tcpenable=yes
tcpbindaddr=0.0.0.0
tlsenable=yes
transport=udp
srvlookup=no
vmexten=voicemail
relaxdtmf=yes
useragent=Asterisk PBX
tlscertfile=/etc/asterisk/asterisk.cert
tlsprivatekey=/etc/asterisk/asterisk.key
tlscapath=/etc/ssl/certs/
externhost=machinename.dyndns.org
localnet=192.168.0.0/255.255.0.0

Note that you can have more than one register line in your config if you use more than one SIP provider, but you must register with the server whether you want to receive incoming calls or not. Then I added a new blurb to the bottom of the same file:

[jmp]
type=peer
host=mp.cbcbc7.auth.bandwidth.com
port=5008
secret=three secret words
defaultuser=username
context=from-jmp
disallow=all
allow=ulaw
allow=g729
insecure=port,invite
canreinvite=no
dtmfmode=rfc2833

and for reference, here's the blurb for my Snom 300 SIP phone:

[1001]
; Snom 300
type=friend
qualify=yes
secret=password
encryption=no
context=full
host=dynamic
nat=no
directmedia=no
mailbox=1000@internal
vmexten=707
dtmfmode=rfc2833
call-limit=2
disallow=all
allow=g722
allow=ulaw

I checked that the registration was successful by running asterisk -r and then typing:

sip set debug on

before reloading the configuration using:

reload

Create Asterisk extensions to send and receive calls Once I got registration to work, I hooked this up with my other extensions so that I could send and receive calls using my JMP number. In /etc/asterisk/extensions.conf, I added the following:

[from-jmp]
include => home
exten => s,1,Goto(1000,1)

where home is the context which includes my local SIP devices and 1000 is the extension I want to ring. Then I added the following to enable calls to any destination within the North American Numbering Plan:

[pstn-jmp]
exten => _1NXXNXXXXXX,1,Set(CALLERID(all)=Francois Marier <5551231434>)
exten => _1NXXNXXXXXX,n,Dial(SIP/jmp/$ EXTEN )
exten => _1NXXNXXXXXX,n,Hangup()
exten => _NXXNXXXXXX,1,Set(CALLERID(all)=Francois Marier <5551231234>)
exten => _NXXNXXXXXX,n,Dial(SIP/jmp/1$ EXTEN )
exten => _NXXNXXXXXX,n,Hangup()

Here 5551231234 is my JMP phone number, not my bwsip numerical username. For reference, here's the rest of my dialplan in /etc/asterisk/extensions.conf:

[general]
static=yes
writeprotect=no
clearglobalvars=no
[public]
exten => _X.,1,Hangup(3)
[sipdefault]
exten => _X.,1,Hangup(3)
[default]
exten => _X.,1,Hangup(3)
[internal]
include => home
[full]
include => internal
include => pstn-jmp
exten => 707,1,VoiceMailMain(1000@internal)
[home]
; Internal extensions
exten => 1000,1,Dial(SIP/1001,20)
exten => 1000,n,Goto(in1000-$ DIALSTATUS ,1)
exten => 1000,n,Hangup
exten => in1000-BUSY,1,Hangup(17)
exten => in1000-CONGESTION,1,Hangup(3)
exten => in1000-CHANUNAVAIL,1,VoiceMail(1000@internal,su)
exten => in1000-CHANUNAVAIL,n,Hangup(3)
exten => in1000-NOANSWER,1,VoiceMail(1000@internal,su)
exten => in1000-NOANSWER,n,Hangup(16)
exten => _in1000-.,1,Hangup(16)

Firewall Finally, I opened a few ports in my firewall by putting the following in /etc/network/iptables.up.rules:

# SIP and RTP on UDP (jmp.cbcbc7.auth.bandwidth.com)
-A INPUT -s 67.231.2.13/32 -p udp --dport 5008 -j ACCEPT
-A INPUT -s 216.82.238.135/32 -p udp --dport 5008 -j ACCEPT
-A INPUT -s 67.231.2.13/32 -p udp --sport 5004:5005 --dport 10001:20000 -j ACCEPT
-A INPUT -s 216.82.238.135/32 -p udp --sport 5004:5005 --dport 10001:20000 -j ACCEPT

Outbound calls not working While the above setup works for me for inbound calls, it doesn't currently work for outbound calls. The hostname currently resolves to one of two IP addresses:

$ dig +short jmp.cbcbc7.auth.bandwidth.com
67.231.2.13
216.82.238.135

If I pin it to the first one by putting the following in my /etc/hosts file:

67.231.2.13 jmp.cbcbc7.auth.bandwidth.com

then I get a 486 error back from the server when I dial 1-555-456-4567:

<--- SIP read from UDP:67.231.2.13:5008 --->
SIP/2.0 486 Busy Here
Via: SIP/2.0/UDP 127.0.0.1:5060;branch=z9hG4bK03210a30
From: "Francois Marier" <sip:5551231234@127.0.0.1>
To: <sip:15554564567@jmp.cbcbc7.auth.bandwidth.com:5008>
Call-ID: 575f21f36f57951638c1a8062f3a5201@127.0.0.1:5060
CSeq: 103 INVITE
Content-Length: 0

On the other hand, if I pin it to 216.82.238.135, then I get a 600 error:

<--- SIP read from UDP:216.82.238.135:5008 --->
SIP/2.0 600 Busy Everywhere
Via: SIP/2.0/UDP 127.0.0.1:5060;branch=z9hG4bK7b7f7ed9
From: "Francois Marier" <sip:5551231234@127.0.0.1>
To: <sip:15554564567@jmp.cbcbc7.auth.bandwidth.com:5008>
Call-ID: 5bebb8d05902c1732c6b9f4776844c66@127.0.0.1:5060
CSeq: 103 INVITE
Content-Length: 0

If you have any idea what might be wrong here, or if you got outbound calls to work on Bandwidth.com, please leave a comment!

Someone is Wrong on the Internet If you look up how to only forward ports with ssh, you may come across solutions like this:

ssh -nNT -L 8000:example.com:80 user@bastion.example.com

Or perhaps this, if you also wanted to send ssh to the background:

ssh -NT -L 3306:db.example.com:3306 example.com &

Both of these use at least one option that is entirely redundant, and the second can cause ssh to fail to connect if you happen to be using password authentication. However, they seem to still persist in various articles about ssh port forwarding. I myself was using the first variation until just recently, and I figured I would write this up to inform others who might be still using these solutions. The correct option for this situation is not -nNT but simply -N, as in:

ssh -N -L 8000:example.com:80 user@bastion.example.com

If you want to also send ssh to the background, then you ll want to add -f instead of using your shell s built-in & feature, because you can then input passwords into ssh if necessary¹ Honestly, that s the point of this article, so you can stop here if you want. If you re looking for a detailed explaination of what each of these options actually does, or if you have no idea what I m talking about, read on!

What is SSH Port Forwarding? ssh is a powerful tool for remote access to servers, allowing you to execute commands on a remote machine. It can also forward ports through a secure tunnel with the -L and -R options. Basically, you can forward a connection to a local port to a remote server like so:

ssh -L 8080:other.example.com:80 ssh.example.com

In this example, you connect to ssh.example.com and then ssh forwards any traffic on your local machine port 8080² to other.example.com port 80 via ssh.example.com. This is a really powerful feature, allowing you to jump³ inside your firewall with just an ssh server exposed to the world. It can work in reverse as well with the -R option, allowing connections on a remote host in to a server running on your local machine. For example, say you were running a website on your local machine on port 8080 but wanted it accessible on example.com port 80⁴. You could use something like:

ssh -R 8080:example.com:80 example.com

The trouble with ssh port forwarding is that, absent any additional options, you also open a shell on the remote machine. If you re planning to both work on a remote machine and use it to forward some connection, this is fine, but if you just need to forward a port quickly and don t care about a shell at that moment, it can be annoying, especially since if the shell closes ssh will close the forwarding port as well. This is where the -N option comes in.

SSH just forwarding ports In the ssh manual page⁵, -N is explained like so:

Do not execute a remote command. This is useful for just forwarding ports.

This is all we need. It instructs ssh to run no commands on the remote server, just forward the ports specified in the -L or -R options. But people seem to think that there are a bunch of other necessary options, so what do those do?

SSH and stdin -n controls how ssh interacts with standard input, specifically telling it not to:

Redirects stdin from /dev/null (actually, prevents reading from stdin). This must be used when ssh is run in the background. A common trick is to use this to run X11 programs on a remote machine. For example, ssh -n shadows.cs.hut.fi emacs & will start an emacs on shadows.cs.hut.fi, and the X11 connection will be automatically for warded over an encrypted channel. The ssh program will be put in the background. (This does not work if ssh needs to ask for a password or passphrase; see also the -f option.)

SSH passwords and backgrounding -f sends ssh to background, freeing up the terminal in which you ran ssh to do other things.

Requests ssh to go to background just before command execution. This is useful if ssh is going to ask for passwords or passphrases, but the user wants it in the background. This implies -n. The recommended way to start X11 programs at a remote site is with something like ssh -f host xterm.

As indicated in the description of -n, this does the same thing as using the shell s & feature with -n, but allows you to put in any necessary passwords first.

SSH and pseudo-terminals -T is a little more complicated than the others and has a very short explanation:

Disable pseudo-terminal allocation.

It has a counterpart in -t, which is explained a little better:

Force pseudo-terminal allocation. This can be used to execute arbitrary screen-based programs on a remote machine, which can be very useful, e.g. when implementing menu services. Multiple -t options force tty allocation, even if ssh has no local tty.

As the description of -t indicates, ssh is allocating a pseudo-terminal on the remote machine, not the local one. However, I have confirmed⁶ that -N doesn t allocate a pseudo-terminal either, since it doesn t run any commands. Thus this option is entirely unnecessary.

What s a pseudo-terminal? This is a bit complicated, but basically it s an interface used in UNIX-like systems, like Linux or BSD, that pretends to be a terminal (thus pseudo-terminal). Programs like your shell, or any text-based menu system made in libraries like ncurses, expect to be connected to one (when used interactively at least). Basically it fakes as if the input it is given (over the network, in the case of ssh) was typed on a physical terminal device and do things like raise an interrupt (SIGINT) if Ctrl+C is pressed.

Why? I don t know why these incorrect uses of ssh got passed around as correct, but I suspect it s a form of cargo cult, where we use example commands others provide and don t question what they do. One stack overflow answer I read that provided these options seemed to think -T was disabling the local pseudo-terminal, which might go some way towards explaining why they thought it was necessary. I guess the moral of this story is to question everything and actually read the manual, instead of just googling it.

1. Not that you SHOULD be using ssh with password authentication anyway, but people do.
2. Only on your loopback address by default, so that you re not allowing random people on your network to use your tunnel.
3. In fact, ssh even supports Jump Hosts, allowing you to automatically forward an ssh connection through another machine.
4. I can t say I recommend a setup like this for anything serious, as you d need to ssh as root to forward ports less than 1024. SSH forwarding is not for permanent solutions, just short-lived connections to machines that would be otherwise inaccessible.
5. Specifically, my source is the ssh(1) manual page in OpenSSH 8.4, shipped as 1:8.4p1-5 in Debian bullseye.
6. I just forwarded ports with -N and then logged in to that same machine and looked at psuedo-terminal allocations via ps ux. No terminal is associated with ssh connections using just the -N option.

Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 912 other packages on CRAN. This new release brings us Armadillo 10.7.0 released this morning by Conrad. Leading up to this were three runs of reverse dependencies the first of which uncovered the need for a small PR for subview_cols support which Conrad kindly supplied. The full set of changes follows. We include the last interim release (sent as usual to the drat repo) as well.

Changes in RcppArmadillo version 0.10.7.0.0 (2021-09-30)

• Upgraded to Armadillo release 10.7.0 (Entropy Maximizer)
  • faster handling of submatrix views accessed by X.cols(first_col,last_col)
  • faster handling of element-wise min() and max() in compound expressions
  • expanded solve() with solve_opts::force_approx option to force use of the approximate solver

Changes in RcppArmadillo version 0.10.6.2.0 (2021-08-05)

• Upgraded to Armadillo release 10.6.2 (Keep Calm)
  • fix incorrect use of constexpr for handling fixed-size matrices and vectors
  • improved documentation
• GitHub- and drat-only release

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

My home automation setup has been fairly static recently; it does what we need and generally works fine. One area I think could be better is controlling it; we have access Home Assistant on our phones, and the Alexa downstairs can control things, but there are no smart assistants upstairs and sometimes it would be nice to just push a button to turn on the light rather than having to get my phone out. Thanks to the fact the UK generally doesn t have neutral wire in wall switches that means looking at something battery powered. Which means wifi based devices are a poor choice, and it s necessary to look at something lower power like Zigbee or Z-Wave. Zigbee seems like the better choice; it s a more open standard and there are generally more devices easily available from what I ve seen (e.g. Philips Hue and IKEA TR DFRI). So I bought a couple of Xiaomi Mi Smart Home Wireless Switches, and a CC2530 module and then ignored it for the best part of a year. Finally I got around to flashing the Z-Stack firmware that Koen Kanters kindly provides. (Insert rant about hardware manufacturers that require pay-for tool chains. The CC2530 is even worse because it s 8051 based, so SDCC should be able to compile for it, but the TI Zigbee libraries are only available in a format suitable for IAR s embedded workbench.) Flashing the CC2530 is a bit of faff. I ended up using the CCLib fork by Stephan Hadinger which supports the ESP8266. The nice thing about the CC2530 module is it has 2.54mm pitch pins so nice and easy to jumper up. It then needs a USB/serial dongle to connect it up to a suitable machine, where I ran Zigbee2MQTT. This scares me a bit, because it s a bunch of node.js pulling in a chunk of stuff off npm. On the flip side, it Just Works and I was able to pair the Xiaomi button with the device and see MQTT messages that I could then use with Home Assistant. So of course I tore down that setup and went and ordered a CC2531 (the variant with USB as part of the chip). The idea here was my test setup was upstairs with my laptop, and I wanted something hooked up in a more permanent fashion. Once the CC2531 arrived I got distracted writing support for the Desk Viking to support CCLib (and modified it a bit for Python3 and some speed ups). I flashed the dongle up with the Z-Stack Home 1.2 (default) firmware, and plugged it into the house server. At this point I more closely investigated what Home Assistant had to offer in terms of Zigbee integration. It turns out the ZHA integration has support for the ZNP protocol that the TI devices speak (I m reasonably sure it didn t when I first looked some time ago), so that seemed like a better option than adding the MQTT layer in the middle. I hit some complexity passing the dongle (which turns up as /dev/ttyACM0) through to the Home Assistant container. First I needed an override file in /etc/systemd/nspawn/hass.nspawn: (I m not clear why the VirtualEthernet needed to exist; without it networking broke entirely but I couldn t see why it worked with no override file.) A udev rule on the host to change the ownership of the device file so the root user and dialout group in the container could see it was also necessary, so into /etc/udev/rules.d/70-persistent-serial.rules went: In the container itself I had to switch PrivateDevices=true to PrivateDevices=false in the home-assistant.service file (which took me a while to figure out; yay for locking things down and then needing to use those locked down things). Finally I added the hass user to the dialout group. At that point I was able to go and add the integration with Home Assistant, and add the button as a new device. Excellent. I did find I needed a newer version of Home Assistant to get support for the button, however. I was still on 2021.1.5 due to upstream dropping support for Python 3.7 and not being prepared to upgrade to Debian 11 until it was actually released, so the version of zha-quirks didn t have the correct info. Upgrading to Home Assistant 2021.8.7 sorted that out. There was another slight problem. Range. Really I want to use the button upstairs. The server is downstairs, and most of my internal walls are brick. The solution turned out to be a TR DFRI socket, which replaced the existing ESP8266 wifi socket controlling the stair lights. That was close enough to the server to have a decent signal, and it acts as a Zigbee router so provides a strong enough signal for devices upstairs. The normal approach seems to be to have a lot of Zigbee light bulbs, but I have mostly kept overhead lights as uncontrolled - we don t use them day to day and it provides a nice fallback if the home automation has issues. Of course installing Zigbee for a single button would seem to be a bit pointless. So I ordered up a Sonoff door sensor to put on the front door (much smaller than expected - those white boxes on the door are it in the picture above). And I have a 4 gang wireless switch ordered to go on the landing wall upstairs. Now I ve got a Zigbee setup there are a few more things I m thinking of adding, where wifi isn t an option due to the need for battery operation (monitoring the external gas meter springs to mind). The CC2530 probably isn t suitable for my needs, as I ll need to write some custom code to handle the bits I want, but there do seem to be some ARM based devices which might well prove suitable

This post is about some changes recently made to Rust's ErrorKind, which aims to categorise OS errors in a portable way. Audiences for this post

• The educated general reader interested in a case study involving error handling, stability, API design, and/or Rust.
• Rust users who have tripped over these changes. If this is you, you can cut to the chase and skip to How to fix.

Background and context Error handling principles Handling different errors differently is often important (although, sadly, often neglected). For example, if a program tries to read its default configuration file, and gets a "file not found" error, it can proceed with its default configuration, knowing that the user hasn't provided a specific config. If it gets some other error, it should probably complain and quit, printing the message from the error (and the filename). Otherwise, if the network fileserver is down (say), the program might erroneously run with the default configuration and do something entirely wrong. Rust's portability aims The Rust programming language tries to make it straightforward to write portable code. Portable error handling is always a bit tricky. One of Rust's facilities in this area is std::io::ErrorKind which is an enum which tries to categorise (and, sometimes, enumerate) OS errors. The idea is that a program can check the error kind, and handle the error accordingly. That these ErrorKinds are part of the Rust standard library means that to get this right, you don't need to delve down and get the actual underlying operating system error number, and write separate code for each platform you want to support. You can check whether the error is ErrorKind::NotFound (or whatever). Because ErrorKind is so important in many Rust APIs, some code which isn't really doing an OS call can still have to provide an ErrorKind. For this purpose, Rust provides a special category ErrorKind::Other, which doesn't correspond to any particular OS error. Rust's stability aims and approach Another thing Rust tries to do is keep existing code working. More specifically, Rust tries to:

1. Avoid making changes which would contradict the previously-published documentation of Rust's language and features.
2. Tell you if you accidentally rely on properties which are not part of the published documentation.

By and large, this has been very successful. It means that if you write code now, and it compiles and runs cleanly, it is quite likely that it will continue work properly in the future, even as the language and ecosystem evolves. This blog post is about a case where Rust failed to do (2), above, and, sadly, it turned out that several people had accidentally relied on something the Rust project definitely intended to change. Furthermore, it was something which needed to change. And the new (corrected) way of using the API is not so obvious. Rust enums, as relevant to io::ErrorKind (Very briefly:) When you have a value which is an io::ErrorKind, you can compare it with specific values:

    if error.kind() == ErrorKind::NotFound   ...
  

But in Rust it's more usual to write something like this (which you can read like a switch statement):

    match error.kind()  
      ErrorKind::NotFound => use_default_configuration(),
      _ => panic!("could not read config file  :  ", &file, &error),
     
  

Here _ means "anything else". Rust insists that match statements are exhaustive, meaning that each one covers all the possibilities. So if you left out the line with the _, it wouldn't compile. Rust enums can also be marked non_exhaustive, which is a declaration by the API designer that they plan to add more kinds. This has been done for ErrorKind, so the _ is mandatory, even if you write out all the possibilities that exist right now: this ensures that if new ErrorKinds appear, they won't stop your code compiling. Improving the error categorisation The set of error categories stabilised in Rust 1.0 was too small. It missed many important kinds of error. This makes writing error-handling code awkward. In any case, we expect to add new error categories occasionally. I set about trying to improve this by proposing new ErrorKinds. This obviously needed considerable community review, which is why it took about 9 months. The trouble with Other and tests Rust has to assign an ErrorKind to every OS error, even ones it doesn't really know about. Until recently, it mapped all errors it didn't understand to ErrorKind::Other - reusing the category for "not an OS error at all". Serious people who write serious code like to have serious tests. In particular, testing error conditions is really important. For example, you might want to test your program's handling of disk full, to make sure it didn't crash, or corrupt files. You would set up some contraption that would simulate a full disk. And then, in your tests, you might check that the error was correct. But until very recently (still now, in Stable Rust), there was no ErrorKind::StorageFull. You would get ErrorKind::Other. If you were diligent you would dig out the OS error code (and check for ENOSPC on Unix, corresponding Windows errors, etc.). But that's tiresome. The more obvious thing to do is to check that the kind is Other. Obvious but wrong. ErrorKind is non_exhaustive, implying that more error kinds will appears, and, naturally, these would more finely categorise previously-Other OS errors. Unfortunately, the documentation note

Errors that are Other now may move to a different or a new ErrorKind variant in the future.

was only added in May 2020. So the wrongness of the "obvious" approach was, itself, not very obvious. And even with that docs note, there was no compiler warning or anything. The unfortunate result is that there is a body of code out there in the world which might break any time an error that was previously Other becomes properly categorised. Furthermore, there was nothing stopping new people writing new obvious-but-wrong code. Chosen solution: Uncategorized The Rust developers wanted an engineered safeguard against the bug of assuming that a particular error shows up as Other. They chose the following solution: There is now a new ErrorKind::Uncategorized which is now used for all OS errors for which there isn't a more specific categorisation. The fallback translation of unknown errors was changed from Other to Uncategorised. This is de jure justified by the fact that this enum has always been marked non_exhaustive. But in practice because this bug wasn't previously detected, there is such code in the wild. That code now breaks (usually, in the form of failing test cases). Usually when Rust starts to detect a particular programming error, it is reported as a new warning, which doesn't break anything. But that's not possible here, because this is a behavioural change. The new ErrorKind::Uncategorized is marked unstable. This makes it impossible to write code on Stable Rust which insists that an error comes out as Uncategorized. So, one cannot now write code that will break when new ErrorKinds are added. That's the intended effect. The downside is that this does break old code, and, worse, it is not as clear as it should be what the fixed code looks like. Alternatives considered and rejected by the Rust developers Not adding more ErrorKinds This was not tenable. The existing set is already too small, and error categorisation is in any case expected to improve over time. Just adding ErrorKinds as had been done before This would mean occasionally breaking test cases (or, possibly, production code) when an error that was previously Other becomes categorised. The broken code would have been "obvious", but de jure wrong, just as it is now, So this option amounts to expecting this broken code to continue to be written and continuing to break it occasionally. Somehow using Rust's Edition system The Rust language has a system to allow language evolution, where code declares its Edition (2015, 2018, 2021). Code from multiple editions can be combined, so that the ecosystem can upgrade gradually. It's not clear how this could be used for ErrorKind, though. Errors have to be passed between code with different editions. If those different editions had different categorisations, the resulting programs would have incoherent and broken error handling. Also some of the schemes for making this change would mean that new ErrorKinds could only be stabilised about once every 3 years, which is far too slow. How to fix code broken by this change Most main-line error handling code already has a fallback case for unknown errors. Simply replacing any occurrence of Other with _ is right. How to fix thorough tests The tricky problem is tests. Typically, a thorough test case wants to check that the error is "precisely as expected" (as far as the test can tell). Now that unknown errors come out as an unstable Uncategorized variant that's not so easy. If the test is expecting an error that is currently not categorised, you want to write code that says "if the error is any of the recognised kinds, call it a test failure". What does "any of the recognised kinds" mean here ? It doesn't meany any of the kinds recognised by the version of the Rust stdlib that is actually in use. That set might get bigger. When the test is compiled and run later, perhaps years later, the error in this test case might indeed be categorised. What you actually mean is "the error must not be any of the kinds which existed when the test was written". IMO therefore the right solution for such a test case is to cut and paste the current list of stable ErrorKinds into your code. This will seem wrong at first glance, because the list in your code and in Rust can get out of step. But when they do get out of step you want your version, not the stdlib's. So freezing the list at a point in time is precisely right. You probably only want to maintain one copy of this list, so put it somewhere central in your codebase's test support machinery. Periodically, you can update the list deliberately - and fix any resulting test failures. Unfortunately this approach is not suggested by the documentation. In theory you could work all this out yourself from first principles, given even the situation prior to May 2020, but it seems unlikely that many people have done so. In particular, cutting and pasting the list of recognised errors would seem very unnatural. Conclusions This was not an easy problem to solve well. I think Rust has done a plausible job given the various constraints, and the result is technically good. It is a shame that this change to make the error handling stability more correct caused the most trouble for the most careful people who write the most thorough tests. I also think the docs could be improved.

edited shortly after posting, and again 2021-09-22 16:11 UTC, to fix HTML slips

comments