How our for-profit company became a nonprofit, to better tackle the digital divide.
Originally posted on the Endless OS Foundation blog.
An 8-year journey to a nonprofit
On the 1st of April 2020, our for-profit Endless Mobile officially became a nonprofit as the Endless OS Foundation. Our launch as a nonprofit just as the global pandemic took hold was, predictably, hardly noticed, but for us the timing was incredible: as the world collectively asked What can we do to help others in need? , we framed our mission statement and launched our .org with the same very important question in mind. Endless always had a social impact mission at its heart, and the challenges related to students, families, and communities falling further into the digital divide during COVID-19 brought new urgency and purpose to our team s decision to officially step in the social welfare space.
On April 1st 2020, our for-profit Endless Mobile officially became a nonprofit as the Endless OS Foundation, focused on the #DigitalDivide.
Our updated status was a long time coming: we began our transformation to a nonprofit organization in late 2019 with the realization that the true charter and passions of our team would be greatly accelerated without the constraints of for-profit goals, investors and sales strategies standing in the way of our mission of digital access and equity for all.
But for 8 years we made a go of it commercially, headquartered in Silicon Valley and framing ourselves as a tech startup with access to the venture capital and partnerships on our doorstep. We believed that a successful commercial channel would be the most efficient way to scale the impact of bringing computer devices and access to communities in need. We still believe this we ve just learned through our experience that we don t have the funding to enter the computer and OS marketplace head-on. With the social impact goal first, and the hope of any revenue a secondary goal, we have had many successes in those 8 years bridging the digital divide throughout the world, from Brazil, to Kenya, and the USA. We ve learned a huge amount which will go on to inform our strategy as a nonprofit.
Endless always had a social impact mission at its heart. COVID-19 brought new urgency and purpose to our team s decision to officially step in the social welfare space.
Our unique perspective
One thing we learned as a for-profit is that the OS and technology we ve built has some unique properties which are hugely impactful as a working solution to digital equity barriers. And our experience deploying in the field around the world for 8 years has left us uniquely informed via many iterations and incremental improvements.
With this knowledge in-hand, we ve been refining our strategy throughout 2020 and now starting to focus on what it really means to become an effective nonprofit and make that impact. In many ways it is liberating to abandon the goals and constraints of being a for-profit entity, and in other ways it s been a challenging journey for me and the team to adjust our way of thinking and let these for-profit notions and models go. Previously we exclusively built and sold a product that defined our success; and any impact we achieved was a secondary consequence of that success and seen through that lens. Now our success is defined purely in terms of social impact, and through our actions, those positive impacts can be made with or without our product . That means that we may develop and introduce technology to solve a problem, but it is equally as valid to find another organization s existing offering and design a way to increase that positive impact and scale.
We develop technology to solve access equity issues, but it s equally as valid to find another organization s offering and partner in a way that increases their positive impact.
The analogy to Free and Open Source Software is very strong while Endless has always used and contributed to a wide variety of FOSS projects, we ve also had a tension where we ve been trying to hold some pieces back and capture value such as our own application or content ecosystem, our own hardware platform necessarily making us competitors to other organisations even though they were hoping to achieve the same things as us. As a nonprofit we can let these ideas go and just pick the best partners and technologies to help the people we re trying to reach.
Digital equity 4 barriers we need to overcome
In future, our decisions around which projects to build or engage with will revolve around 4 barriers to digital equity, and how our Endless OS, Endless projects, or our partners offerings can help to solve them. We define these 4 equity barriers as: barriers to devices, barriers to connectivity, barriers to literacy in terms of your ability to use the technology, and barriers to engagement in terms of whether using the system is rewarding and worthwhile.
We define the 4 digital equity barriers we exist to impact as: 1. barriers to devices 2. barriers to connectivity 3. barriers to literacy 4. barriers to engagement
It doesn t matter who makes the solutions that break these barriers; what matters is how we assist in enabling people to use technology to gain access to the education and opportunities these barriers block. Our goal therefore is to simply ensure that solutions exist building them ourselves and with partners such as the FOSS community and other nonprofits proving them with real-world deployments, and sharing our results as widely as possible to allow for better adoption globally.
If we define our goal purely in terms of whether people are using Endless OS, we are effectively restricting the reach and scale of our solutions to the audience we can reach directly with Endless OS downloads, installs and propagation. Conversely, partnerships that scale impact are a win-win-win for us, our partners, and the communities we all serve.
Engineering impact
Our Endless engineering roots and capabilities feed our unique ability to build and deploy all of our solutions, and the practical experience of deploying them gives us evidence and credibility as we advocate for their use. Either activity would be weaker without the other.
Our engineering roots and capabilities feed our unique ability to build and deploy digital divide solutions.
Our partners in various engineering communities will have already seen our change in approach. Particularly, with GNOME we are working hard to invest in upstream and reconcile the long-standing differences between our experience and GNOME. If successful, many more people can benefit from our work than just users of Endless OS. We re working with Learning Equality on Kolibri to build a better app experience for Linux desktop users and bring content publishers into our ecosystem for the first time, and we ve also taken our very own Hack, the immersive and fun destination for kids learning to code, released it for non-Endless systems on Flathub, and made it fully open-source.
What s next for our OS?
What then is in store for the future of Endless OS, the place where we have invested so much time and planning through years of iterations? For the immediate future, we need the capacity to deploy everything we ve built all at once, to our partners. We built an OS that we feel is very unique and valuable, containing a number of world-firsts: first production OS shipped with OSTree, first Flatpak-only desktop, built-in support for updating OS and apps from USBs, while still providing a great deal of reliability and convenience for deployments in offline and educational-safe environments with great apps and content loaded on every system.
However, we need to find a way to deliver this Linux-based experience in a more efficient way, and we d love to talk if you have ideas about how we can do this, perhaps as partners. Can the idea of Endless OS evolve to become a spec that is provided by different platforms in the future, maybe remixes of Debian, Fedora, openSUSE or Ubuntu?
Build, Validate, Advocate
Beyond the OS, the Endless OS Foundation has identified multiple programs to help underserved communities, and in each case we are adopting our build, validate, advocate strategy. This approach underpins all of our projects: can we build the technology (or assist in the making), will a community in-need validate it by adoption, and can we inspire others by telling the story and advocating for its wider use?
We are adopting a build, validate, advocate strategy. 1. build the technology (or assist in the making) 2. validate by community adoption 3. advocate for its wider use
As examples, we have just launched the Endless Key (link) as an offline solution for students during the COVID-19 at-home distance learning challenges. This project is also establishing a first-ever partnership of well-known online educational brands to reach an underserved offline audience with valuable learning resources. We are developing a pay-as-you-go platform and new partnerships that will allow families to own laptops via micro-payments that are built directly into the operating system, even if they cannot qualify for standard retail financing. And during the pandemic, we ve partnered with Teach For America to focus on very practical digital equity needs in the USA s urban and rural communities.
One part of the world-wide digital divide solution
We are one solution provider for the complex matrix of issues known collectively as the #DigitalDivide, and these issues will not disappear after the pandemic. Digital equity was an issue long before COVID-19, and we are not so naive to think it can be solved by any single institution, or by the time the pandemic recedes. It will take time and a coalition of partnerships to win. We are in for the long-haul and we are always looking for partners, especially now as we are finding our feet in the nonprofit world. We d love to hear from you, so please feel free to reach out to me I m ramcq on IRC, RocketChat, Twitter, LinkedIn or rob@endlessos.org.
This post is part of a series about trying to setup a gitlab runner based on
systemd-nspawn. I published the polished
result as nspawn-runner on GitHub.
The plan
Back to custom runners, here's my plan:
config can be a noop
prepare starts the nspawn machine
run runs scripts with machinectl shell
cleanup runs machinectl stop
The scripts
Here are the scripts based on Federico's work:
base.sh with definitions sourced by all scripts:
This post is part of a series about trying to setup a gitlab runner based on
systemd-nspawn. I published the polished
result as nspawn-runner on GitHub.
Here I try to figure out possible ways of invoking nspawn for the prepare,
run, and cleanup steps of gitlab custom runners. The results might be
useful invocations beyond Gitlab's scope of application.
I begin with a chroot which will be the base for our build environments:
Fully ephemeral nspawn
This would be fantastic: set up a reusable chroot, mount readonly, run the CI
in a working directory mounted on tmpfs. It sets up quickly, it cleans up after
itself, and it would make prepare and cleanup noops:
However, run gets run multiple times, so I need the side effects of run to
persist inside the chroot between runs.
Also, if the CI uses a large amount of disk space, tmpfs may get into trouble.
nspawn with overlay
Federico used --overlay
to keep the base chroot readonly while allowing persistent writes on a
temporary directory on the filesystem.
Note that using --overlay requires systemd and systemd-container from
buster-backports because of systemd bug #3847.
Example:
I can run this twice, and changes in the file system will persist between
systemd-nspawn executions. Great! However, any process will be killed at the
end of each execution.
machinectl
I can give a name to systemd-nspawn invocations using --machine, and it
allows me to run multiple commands during the machine lifespan using
machinectl and systemd-run.
In theory machinectl can also fully manage chroots and disk images in
/var/lib/machines, but I haven't found a way with machinectl to start
multiple machines sharing the same underlying chroot.
It's ok, though: I managed to do that with systemd-nspawn invocations.
I can use the --machine=name argument to systemd-nspawn to make it visible
to machinectl. I can use the --boot argument to systemd-nspawn to start
enough infrastructure inside the container to allow machinectl to interact
with it.
This gives me any number of persistent and named running systems, that share
the same underlying chroot, and can cleanup after themselves. I can run
commands in any of those systems as I like, and their side effects persist
until a system is stopped.
The chroot needs systemd and dbus for machinectl to be able to interact with it:
# machinectl list
MACHINE CLASS SERVICE OS VERSION ADDRESSES
test container systemd-nspawn debian 10 -
1 machines listed.
# machinectl shell --quiet test /bin/ls -la /
total 60
[ ]
To run commands, rather than machinectl shell, I need to use systemd-run
--wait --pipe --machine=name, otherwise machined won't forward the exit
code. The result however is
pretty good, with working stdin/stdout/stderr redirection and forwarded exit
code.
Good, I'm getting somewhere.
The terminal where I ran systemd-nspawn is currently showing a nice getty for
the booted system, which is cute, and not what I want for the setup process of
a CI.
Spawning machines without needing a terminal
machinectl uses /lib/systemd/system/systemd-nspawn@.service to start
machines. I suppose there's limited magic in there: start systemd-nspawn as a
service, use --machine to give it a name, and machinectl manages it as if
it started it itself.
What if, instead of installing a unit file for each CI run, I try to do the
same thing with systemd-run?
It works! I can interact with it using machinectl, and fine tune DevicePolicy
as needed to lock CI machines down.
This setup has a race condition where if I try to run a command inside the
machine in the short time window before the machine has finished booting, it
fails:
# systemd-run [ ] systemd-nspawn [ ] ; machinectl --quiet shell test /bin/ls -la /
Failed to get shell PTY: Protocol error
# machinectl shell test /bin/ls -la /
Connected to machine test. Press ^] three times within 1s to exit session.
total 60
[ ]
systemd-nspawn has the option --notify-ready=yes that solves exactly this
problem:
# systemd-run [ ] systemd-nspawn [ ] --notify-ready=yes ; machinectl --quiet shell test /bin/ls -la /
Running as unit: run-r5a405754f3b740158b3d9dd5e14ff611.service
total 60
[ ]
On nspawn's side, I should now have all I need.
Next steps
My next step will be wrapping it all together in
a gitlab runner.
This is a first post in a series about trying to setup a gitlab runner based on
systemd-nspawn. I published the polished
result as nspawn-runner on GitHub.
The goal
I need to setup gitlab runners, and I try to not involve docker in my
professional infrastructure if I can avoid it.
Let's try systemd-nspawn. It's widely
available and reasonably reliable.
I'm not the first to have this idea:
Federico Ceratto made a setup based on custom runners
and Josef Kufner one based on ssh runners.
I'd like to skip the complication of ssh, and to expand Federico's version to
persist not just filesystem changes but also any other side effect of CI
commands. For example, one CI command may bring up a server and the next CI
command may want to test interfacing with it.
Understanding gitlab-runner
First step: figuring out gitlab-runner.
Test runs of gitlab-runner
I found that I can run gitlab-runner manually without needing to go through a
push to Gitlab. It needs a local git repository with a .gitlab-ci.yml file:
mkdir test
cd test
git init
cat > .gitlab-ci.yml << EOF
tests:
script:
- env sort
- pwd
- ls -la
EOF
git add .gitlab-ci.yml
git commit -am "Created a test repo for gitlab-runner"
Then I can go in the repo and test gitlab-runner:
gitlab-runner exec shell tests
It doesn't seem to use /etc/gitlab-runner/config.toml and it needs all the
arguments passed to its command line: I used the shell runner for a simple
initial test.
Later I'll try to brew a gitlab-runner exec custom invocation that uses
nspawn.
Basics of custom runners
A custom runner runs a
few scripts to manage the run:
config, to allow to override the run configuration outputting JSON data
prepare, to prepare the environment
run, to run scripts in the environment (might be ran multiple times)
cleanup to clean up the environment
run gets at least one argument which is a path to the script to run. The other
scripts get no arguments by default.
The runner configuration
controls the paths of the scripts to run, and optionally extra arguments to
pass to them
Next steps
My next step will be to figure out possible
ways of invoking nspawn for the prepare, run, and cleanup scripts.
How can I put four years into a pie? I m thinking of Inauguration Day 2017 through to today, Inauguration Day 2021. In truth things started back in 2015, when Donald Trump announced his run for the United States presidency, and I don t know how long things will continue past the moment when President-Elect Joe Biden becomes President Joe Biden.
For the United States, it s been a hell of a time. For the world, it d been even worse. Every generation thinks that they lived through more than anyone else, that they had it worse. I had a Boomer tell me that the existential stress of COVID is nothing compared to the Vietnam War. I m sure when we are living through a global water crisis, I ll tell the kids that we had it bad too. Everyday I listen to the radio and read Twitter, aware that the current state of endless wars wars against terrorism and drugs, organized crime and famine, climate change and racism is global, and not limited to just what s happening to and around me. That makes it feel worse and bigger and I wonder if earlier generations can really grasp how big that is.
The last four years brought me in closer working relationships with people in India and Nigeria. I would call these people my friends in that if they were in town I would want to see them and show them around. Most of them I would offer a space in my small apartment, in case they needed somewhere to sleep and wanted to save the money. We chat, though we only have the internet as opposed to elevator rides in tall office buildings and slow walks down to the shops during lunch breaks.
From these relationships I have learned very little about life in India or Nigeria, and I only visited India separately from any of my colleagues there. (I went for a wedding. My visa to Nigeria was denied on account of a medical issue.) But, I follow these people on social media and see what they share, the same political and social utterances that could be the same here or virtually any other place, as long as we replace the right keywords. Exchange the name of one leader, conservative party, or government unit for another. When I first saw the #EndSARS hashtag show up, I thought the images were from Black Lives Matter protests. Stop police brutality.
And that was only in the last few months.
I ve had three jobs since Trump first announced his candidacy in three very different places. In the first I felt like I wasn t able to talk about the sexism and discrimination I was dealing with in the office, and how much more so my views on an organizational partnership with a government whose policies I strongly disagreed with. In the second I was able to talk about these things, but there was nothing to do about them.
I ve been in love and had my heart broken three times in three very different ways that all came down to someone valuing someone else more than they valued me. Can I bake heartbreak into a pie? Is it even fair to distract from the political world with my own loss?
What about COVID-19? There s bitterness and anger and tears and pain emotional and physical. There is desperation and desolation and loneliness. Covid has colored everything during the past year. It is a burden our new political leaders will take on. Biden and Harris, all of the new people in Congress, and everyone else who has taken on an elected position now must content with Covid with new levels of responsibility. Not only do their decisions affect the people they come into contact with, they now affect everyone their policies touch and perhaps even more than that.
The government hired people to build walls. Our government approved it and people willingly took on the job of building those walls. Families were separated. Children were placed in inhumane conditions; children were tortured. Remember when the guards at border detention facilities were raping children? Remember when children had guards? Women were forcefully operated upon and had their bodies permanently changed without their permission, against their desires. People were executed by the state.
There were so many things I ve lost track of them all. I remember bits and pieces as I write this, coming back to me like singing a song I haven t thought about in years. With each line, I remember another one. Being worried about coming home from Cuba, when the visitation rules were changed in the middle of my trip. Climate change, again and again. Pollution and microplastics and watching the country being broken into pieces and sold off in the name of economy and progress. People losing their access to healthcare, through clinics closing down and loss of insurance.
What do you bake into a pie that tastes like sedition? What are the flavors of loss and racism and hate? How to you balance the sourness with subtle hints of hope, which feels to tender and fragile? Do we pair equal parts of the palatable with the unpalatable, in the name of our neatly divided senate?
I have hope, of course I have hope, and I have always had hope, but now it feels thinner than ever, like a ganache or a caramel after your hand slips and you pour too much cream in. A custard or compote or curd that that refuses to thicken no matter how long you cook it. I see that things could be better, but better does not mean good and better does not mean enough.
So I will put my hope into this pie. I put my pain and anger into the dough. I will put my tears and helplessness and bitterness into the filling. I will cover it sweetness and the delicate hope I ve spun out of sugar. Soon I will bake it and share it with the three other people I see because the most important thing about surviving these past years, these past months and weeks and days, is that we did it together. We will commiserate on what we ve overcome, and we will share our hope and the sweetness of the moment, as the spun sugar dissolves on our tongues. There is so much we have left to do, so much we must do. We will be angry in the future, we may be angry later today, but until then, we have pie.
Like each month, have a look at the work funded by Freexian s Debian LTS offering.
Debian project funding
In December, we put aside 2100 EUR to fund Debian projects. The first project proposal (a tracker.debian.org improvement for the security team) was received and quickly approved by the paid contributors, then we opened a request for bids and the bid winner was announced today (it was easy, we had only one candidate). Hopefully this first project will be completed until our next report.
We re looking forward to receive more projects from various Debian teams! Learn more about the rationale behind this initiative in this article.
Debian LTS contributors
In December, 12 contributors have been paid to work on Debian LTS, their reports are available:
Abhijith PA did 7.0h (out of 14h assigned), thus carrying over 7h to January.
Ben Hutchings did 16.5h (out of 16h assigned and 9h from November), thus carrying over 8.5h to January.
Evolution of the situation
December was a quiet month as we didn t have a team meeting nor any other unusual activity and we released 43 DLAs.
The security tracker currently lists 30 packages with a known CVE and the dla-needed.txt file has 25 packages needing an update.
This month we are pleased to welcome Deveryware as new sponsor!
Thanks to our sponsors
Sponsors that joined recently are in bold.
Amid all the conversation about Signal, and the debate over decentralization, one thing has often not been raised: all of these things require an Internet connection.
Of course, you might say. Internet is everywhere these days. Well, not so much, and it turns out there are some very good reasons that people might want messengers that work offline. Here are some examples:
Internet-using messengers leak certain metadata (eg, that a person is using it, or perhaps a sophisticated adversary could use timing analysis to determine that two people are talking using it)
Cell signal outages due to natural disaster, large influx of people (protests, unusual sporting events, festivals, etc), or other factors
Locations where cell signals are not available (rural areas, camping locations, wilderness areas, etc.)
Devices that don t have cell data capability (many tablets, phones that have had service expire, etc.)
How do they work?
These all use some form of local radio signal. Some, such as Briar, may use short-range Bluetooth and Wifi, while others use radios such as LoRa that can reach several miles with low power. I ve written quite a bit about LoRa before, and its unique low-speed but extreme-distance radio capabilities even on low power.
One common thread through these is that most of them are Android-only, though many are compatible with F-Droid and privacy-enhanced Android distributions.
Every item on this list uses full end-to-end encryption (E2EE).
Let s dive on in.
Briar
Of all the options mentioned here, Briar is the one that bridges the traditional Internet-based approach with alternative options the best. It offers three ways for distributing data:
Over the Internet, via Tor onion services
Via Bluetooth to nearby devices
Via Wifi, to other devices connected to the same access point, even if Internet isn t wokring on that AP
As far as I can tell, there is no centralized server in Briar at all. Your account , such as it is, lives entirely within your device; if you wipe your device, you will have to make a new account and re-establish contacts. The use of Tor is also neat to see; it ensures that an adversary can t tell, just from that, that you re using Briar at all, though of course timing analysis may still be possible (and Bluetooth and Wifi uses may reval some of who is communicating).
Briar features several types of messages (detailed in the manual), which really are just different spins on communication, which they liken to metaphors people are familiar with:
Basic 1-to-1 private messaging
Private groups , in which one particular person invites people to the chat group, and can dissolve it at any time
Forums , similar to private groups, but any existing member can invite more people to them, and they continue to exist until the last member leaves (founder isn t special)
Blogs , messages that are automatically shared with all your contacts
By default, Briar raises an audible notification for incoming messages of all types. This is configurable for each type.
Blogs have a way to reblog (even a built-in RSS reader to facilitate that), but framed a different way, they are broadcast messages. They could, for instance, be useful for a send help message to everyone (assuming that people haven t all shut off notifications of blogs due to others using them different ways).
Briar s how it works page has an illustration specifically of how blogs are distributed. I m unclear on some of the details, and to what extent this applies to other kinds of messages, but one thing that you can notice from this is that a person A could write a broadcast message without Internet access, person B could receive it via Bluetooth or whatever, and then when person B gets Internet access again, the post could be distributed more widely. However, it doesn t appear that Briar is really a full mesh, since only known contacts in the distribution path for the message would repeat it.
There are some downsides to Briar. One is that, since an account is fully localized to a device, one must have a separate account for each device. That can lead to contacts having to pick a specific device to send a message to. There is an online indicator, which may help, but it s definitely not the kind of seamless experience you get from Internet-only messengers. Also, it doesn t support migrating to a new phone, live voice/video calls, or attachments, but attachments are in the works.
All in all, a solid communicator, and is the only one on this list that works 100% with the hardware everyone already has. While Bluetooth and Wifi have far more limited range than the other entries, there is undeniably convenience in not needing any additional hardware, and it may be particularly helpful when extra bags/pockets aren t available. Also, Briar is fully Open Source.
Meshtastic
Meshtastic is a radio-first LoRa mesh project. What do I mean by radio-first? Well, basically cell phones are how you interact with Meshtastic, but they are optional. The hardware costs about $30 and the batteries last about 8 days. Range between nodes is a few miles in typical conditions (up to 11km / 7mi in ideal conditions), but nodes act as repeaters, so it is quite conceivable to just drop a node in the middle if you and contacts will be far apart. The project estimates that around 2000 nodes are in operation, and the network is stronger the more nodes are around.
The getting started site describes how to build one.
Most Meshtastic device builds have a screen and some buttons. They can be used independently from the Android app to display received messages, distance and bearing to other devices (assuming both have a GPS enabled), etc. This video is an introduction showing it off, this one goes over the hardware buttons. So even if your phone is dead, you can at least know where your friends are. Incidentally, the phone links up to the radio board using Bluetooth, and can provide a location source if you didn t include one in your build. There are ideas about solar power for Meshtastic devices, too.
Meshtastic doesn t, as far as I know, have an option for routing communication over the Internet, but the devices appear to be very thoughtfully-engineered and easy enough to put together. This one is definitely on my list to try.
Ripple-based devices
This is based on the LoRa Mesh Radio Instructables project, and is similar in concept to Meshtastic. It uses similar hardware, a similar app, but also has an option with a QWERTY hardware keyboard available, for those that want completely phone-free operation while still being able to send messages.
There are a number of related projects posted at Instructables: a GPS tracker, some sensors, etc. These are variations on the same basic concept.
These use the Ripple firmware, which is not open source, so I haven t pursued it further.
GoTenna
For people that want less of a DIY model, and don t mind proprietary solutions, there are two I ll mention. The first is GoTenna Mesh, which is LoRa-based and sells units for $90 each. However, there are significant community concerns about the longevity of the project, as GoTenna has re-focused on government and corporate work. The Android app hasn t been updated in 6 monnths despite a number of reviews citing issues, and the iOS app is also crusty.
Beartooth
Even more expensive at $125 each is the Beartooth. Also a proprietary option, I haven t looked into it more, but they are specifically targetting backwoods types of markets.
Do not use: Bridgefy
Bridgefy was briefly prominent since it was used during the Hong Kong protests. However, numerous vulnerabilities have been demonstrated, and the developers have said they are re-working the app to address them. I wouldn t recommend it for now.
Alternatives: GMRS handhelds
In the USA, GMRS voice handhelds are widely available. Although a license is required, it is simple (no exam) and cheap ($35) and extends to a whole family. GMRS radios also interoperate with FRS radios, which require no license and share some frequencies, but are limited to lower power (though are often sufficient).
Handheld GMRS radios that use up to 5W of power are readily available. A voice signal is a lot harder to carry for a long distance than a very low-bandwidth digital one, so even with much more power you will probably not get the same kind of range you will with something like Meshtastic, and they don t come with any kind of security or encryption at all. However, for basic communication, they are often a useful tool.
Review: The Secret Barrister, by The Secret Barrister
Publisher:
Picador
Copyright:
2018
Printing:
2019
ISBN:
1-5098-4115-6
Format:
Kindle
Pages:
344
The Secret Barrister is a survey and critique of the criminal legal
system of England and Wales. The author is an anonymous barrister who
writes a legal blog of the same name (which I have not read).
A brief and simplified primer for those who, like me, are familiar with
the US legal system but not the English one: A barrister is a lawyer who
argues cases in court, as distinct from a solicitor who does all the other
legal work (and may make limited court appearances). If you need criminal
legal help in England and Wales, you hire a solicitor, and they are your
primary source of legal advise. If your case goes to court, your
solicitor will generally (not always) refer the work of arguing your case
before a judge and jury to a barrister and "instruct" them in the details
of your argument. The job of the barrister is then to handle the
courtroom trial, offer trial-specific legal advice, and translate your
defense (or the crown's prosecution) into persuasive courtroom arguments.
Unlike the United States, with its extremely sharp distinction between
prosecutors and criminal defense attorneys, criminal barristers in England
and Wales argue both prosecutions and defenses depending on who hires
them. (That said, the impression I got from this book is that the
creation of the Crown Prosecution Service is moving England closer to the
US model and more prosecutions are now handled by barristers employed
directly by the CPS, whom I assume do not take defense cases.) Barristers
follow the cab-rank rule, which means that, like a taxicab, they are
professionally obligated to represent people on a first-come, first-serve
basis and are not allowed to pick and choose clients.
(Throughout, I'm referencing the legal system of England and Wales because
the author restricts his comments to it. Presumably this is because the
Scottish and Northern Irish? legal systems are different yet again in
ways I do not know.)
If details like this sound surprising, you can see the appeal of this book
to me. It's easy, in the US, to have a vast ignorance about the legal
systems of other countries or even the possibility of different systems,
which makes it hard to see how our system could be improved. I had a
superficial assumption that since US law started as English common law,
the US and English legal systems would be substantially similar. And they
are to an extent; they're both adversarial rather than inquisitorial, for
example (more on that in a moment). But the current system of criminal
prosecution evolved long after US independence and thus evolved
differently despite similar legal foundations. Those differences are
helpful for this American to ponder the road not taken and the impact of
our respective choices.
That said, explaining the criminal legal system to Americans isn't the
author's purpose. The first fifty pages are that beginner's overview,
since apparently even folks who live in England are confused by the
ubiquity of US legal dramas (not that those are very accurate
representations of the US legal system either). The rest of the book, and
its primary purpose, is an examination of the system's failings, starting
with the magistrates' courts (which often use lay judges and try what in
the US would be called misdemeanors, although as discussed in this book
their scope is expanding). Other topics include problems with bail, how
prosecution is structured, how victims and witnesses are handled, legal
aid, sentencing, and the absurd inadequacy of compensation for erroneous
convictions.
The most useful part of this book for me, apart from the legal system
introduction, was the two chapters the author spends arguing first for and
then against replacing an adversarial system with an inquisitorial system
(the French criminal justice system, for example). When one is as
depressed about the state of one's justice system as both I and the author
are, something radically different sounds appealing. The author first
makes a solid case for the inquisitorial system and then tries to demolish
it, still favoring the adversarial system, and I liked that argument
construction.
The argument in favor of an adversarial system is solid and convincing,
but it's also depressing. It's the argument of someone who has seen the
corruption, sloppiness, and political motivations in an adversarial system
and fears what would happen if they were able to run rampant under a fig
leaf of disinterested objectivity. I can't disagree, particularly when
starting from an adversarial system, but this argument feels profoundly
cynical. It reminds me of the libertarian argument for capitalism: humans
are irredeemably awful, greed and self-interest are the only reliable or
universal human motives, and therefore the only economic system that can
work is one based on and built to harness greed, because expecting any
positive characteristics from humans collectively is hopelessly naive.
The author of this book is not quite that negative in their argument for
an adversarial system, but it's essentially the same reasoning: the only
way a system can be vaguely honest is if it's constantly questioned and
attacked. It can never be trusted to be objective on its own terms. I
wish the author had spent more time on the obvious counter-argument: when
the system is designed for adversarial combat, it normalizes and even
valorizes every dirty tactic that might result in a victory. The system
reinforces our worst impulses, not to mention grinding up and destroying
people who cannot afford their own dirty tricks.
The author proposes several explanations for the problems they see in the
criminal legal system, including "tough on crime" nonsense from politicians
that sounds familiar to this American reader. Most problems, though, they
trace back to lack of funding: of the police, of the courts, of the
prosecutors, and of legal aid. I don't know enough about English politics
to have an independent opinion on this argument, but the stories of
outsourcing to the lowest bidder, overworked civil servants, ridiculously
low compensation rates, flawed metrics like conviction rates, and
headline-driven political posturing that doesn't extend to investing in
necessary infrastructure like better case-tracking systems sounds
depressingly familiar.
This is one of those books where I appreciated the content but not the
writing. It's not horrible, but the sentences are ponderous and strained
and the author is a bit too fond of two-dollar words. They also have a
dramatic and self-deprecating way of describing their own work that I
suspect they thought was funny but that I found grating. By the end of
this book, I was irritated enough that I can't recommend it. But the
content was interesting, even the critique of a political system that
isn't mine, and it prompted some new thoughts on the difficulties of
creating a fair justice system. If you can deal with the author's writing
style, you may also enjoy it.
Rating: 6 out of 10
Prompted by tweets from Lesley and Dave, I thought about KVM switches again and came up with a rather cheap solution to my individual situation (YMMY, as usual).
As I've written last year, my desk has one monitor, keyboard and mouse and two computers. Since writing that post I got a new (bigger) monitor, but also an USB switch again (a DIGITUS USB 3.0 Sharing Switch) - this time one that doesn't freak out my dock \o/
However, having to switch the used computer in two places (USB and monitor) is rather inconvenient, but also getting an KVM switch that can do 4K@60Hz was out of question.
Luckily, hackers gonna hack, everything, and not only receipt printers ( ). There is a tool called ddcutil that can talk to your monitor and change various settings. And udev can execute commands when (USB) devices connect You see where this is going?
After installing the package (available both in Debian and Fedora), we can inspect our system with ddcutil detect. You might have to load the i2c_dev module (thanks Philip!) before this works -- it seems to be loaded automatically on my Fedora, but you never know .
$ sudo ddcutil detect
Invalid display I2C bus: /dev/i2c-4 EDID synopsis: Mfg id: BOE Model: Serial number: Manufacture year: 2017 EDID version: 1.4 DDC communication failed This is an eDP laptop display. Laptop displays do not support DDC/CI.Invalid display I2C bus: /dev/i2c-5 EDID synopsis: Mfg id: AOC Model: U2790B Serial number: Manufacture year: 2020 EDID version: 1.4 DDC communication failedDisplay 1 I2C bus: /dev/i2c-7 EDID synopsis: Mfg id: AOC Model: U2790B Serial number: Manufacture year: 2020 EDID version: 1.4 VCP version: 2.2
The first detected display is the built-in one in my laptop, and those don't support DDC anyways. The second one is a ghost (see ddcutil#160) which we can ignore. But the third one is the one we can (and will control). As this is the only valid display ddcutil found, we don't need to specify which display to talk to in the following commands. Otherwise we'd have to add something like --display 1 to them.
A ddcutil capabilities will show us what the monitor is capable of (or what it thinks, I've heard some give rather buggy output here) -- we're mostly interested in the "Input Source" feature (Virtual Control Panel (VCP) code 0x60):
Seems mine supports it, and I should be able to switch the inputs by jumping between 0x0f, 0x11 and 0x12. You can see other values defined by the spec in ddcutil vcpinfo 60 --verbose, some monitors are using wrong values for their inputs . Let's see if ddcutil getvcp agrees that I'm using DisplayPort now:
Cool, cool. So now we just need a way to trigger input source switching based on some event
There are three devices connected to my USB switch: my keyboard, my mouse and my Yubikey. I do use the mouse and the Yubikey while the laptop is not docked too, so these are not good indicators that the switch has been turned to the laptop. But the keyboard is!
Let's see what vendor and product IDs it has, so we can write an udev rule for it:
$ lsusb
Bus 005 Device 006: ID 17ef:6047 Lenovo ThinkPad Compact Keyboard with TrackPoint
Okay, so let's call ddcutil setvcp 0x60 0x0f when the USB device 0x17ef:0x6047 is added to the system:
$ sudo vim /etc/udev/rules.d/99-ddcutil.rules
$ sudo udevadm control --reload
And done! Whenever I connect my keyboard now, it will force my screen to use DisplayPort-1.
On my workstation, I deployed the same rule, but with ddcutil setvcp 0x60 0x11 to switch to HDMI-1 and my cheap not-really-KVM-but-in-the-end-KVM-USB-switch is done, for the price of one USB switch (~30 ).
Note: if you want to use ddcutil with a Lenovo Thunderbolt 3 Dock (or any other dock using Displayport Multi-Stream Transport (MST)), you'll need kernel 5.10 or newer, which fixes a bug that prevents ddcutil from talking to the monitor using I C.
The ps program gives a snapshot of the processes running on your Unix-like system. On most Linux installations, this will be the ps program from the procps project.
While you can get a lot of information from the tool, a lot of the fields need further explanation or can give wrong or confusing information; or putting it another way, they provide the right information that looks wrong.
One of these confusing fields is the %CPU or pcpu field. You can see this as the third field with the ps aux command. You only really need the u option to see it, but ps aux is a pretty common invokation.
More than 100%?
This post was inspired by procps issue 186 where the submitter said that the sum of the processes cannot be more than the number of CPUs times 100%. If you have 1 CPU then the sum of %CPU for all processes should be 100% or less, have 16 CPUs then 1600% is your maximum number.
Some people reason for the oddity of over 100% CPU as some rounding thing gone wrong and at first I did think that; except I know we get a lot of reports about comparing the top header CPU load vs process load not lining up and its because they re different .
The trick here, is ps is reporting a percentage of what? Or, perhaps to give a better clue, a percentage of when?
PCPU Calculations
So to get to the bottom of this, let s look at the relevant code. In ps/output.c we have a function pr_pcpu that prints the percent CPU. The relevant lines are:
OK, ignoring the include _dead_time line (you get this from the S option and means you include the time this process waited for its children processes) and the scaling (process times are in Jiffies, we have the CPU as 0 to 999 for reasons) you can reduce this down to.
%CPU = ( Tutime + Tstime ) / Tetime
So we find the amount of time the CPU(s) have been busy either in userland or the system, add them together, then divide the sum by the total time. The utime and stime increment like a car s odometer. So if a process uses one Jiffy of CPU time in userland, that counter goes to 1. If it does it again a few seconds later, then that counter goes to 2.
To give an example, if a process has run for ten seconds and within those ten seconds the CPU has been busy in userland for that process, then we get 10/10 = 100% which makes sense.
Not all Start times are the same
Let s take another example, a process still consumes ten seconds CPU time but been running for twenty seconds, the answer is 10/20 or 50%. With our single CPU example system both of these cannot be running at the same time otherwise we have 150% CPU utilisation which is not possible.
However, let s adjust this slightly. We have assumed uniform utilisation. But take the following scenario:
At time T: Process P1 starts and uses 100% CPU
At time T+10 seconds: Process P1 stops using CPU but still runs, perhaps waiting for I/O or sleeping.
Also at time T+10 seconds: Process P2 starts and uses 100% CPU
At time T+20 we run the ps command and look at the %CPU column
The output for ps -o times,etimes,pcpu,comm would look something like:
What we will see is P1 has 10/20 or 50% CPU and P2 has 10/10 or 100% CPU. Add those up, and you have 150% CPU, magic!
The key here is the ELAPSED column. P1 has given you the CPU utilisation across 20 seconds of system time and P2 the CPU utilisation across only 10 seconds. You directly add them together you get the wrong answer.
What s the point of %CPU?
Probably the %CPU column gives results that a lot of people are not expecting, so what s the point of it? Don t use it to see why the CPU is running hot; you can see above those two processes were working the CPU hard at different times. What it is useful for is to see how busy a process is, but be warned its an average. It s helpful for something that starts busy but if the process idles or hardly uses CPU for a week then goes bananas you won t see it.
The top program, because a lot of its statistics are deltas from the last refresh, is a much better program for this sort of information about what is happening right now.
... isn't ready yet, but it's getting there.
I had planned to release a new version of SReview, my online video review and
transcoding system that I wrote originally for FOSDEM but is being used for
DebConf, too, after it was set up and running properly for FOSDEM 2020.
However, things got a bit busy (both in my personal life and in the world at
large), so it fell a bit by the wayside.
I've now also been working on things a bit more, in preparation for an
improved administrator's interface, and have started implementing a REST
API to deal with talks etc through HTTP calls. This seems to be coming
along nicely, thanks to
OpenAPI and the Mojolicious
plugin for parsing
that. I can now
design the API nicely, and autogenerate client side libraries to call
them.
While at it, because libmojolicious-plugin-openapi-perl isn't available
in Debian 10 "buster", I moved the docker
containers
over from stable to testing. This revealed that both
bs1770gain and
inkscape changed their command
line incompatibly, resulting in me having to work around those
incompatibilities. The good news is that I managed to do so in a way
that keeps running SReview on Debian 10 viable, provided one installs
Mojolicious::Plugin::OpenAPI from CPAN rather than from a Debian
package. Or installs a backport of that package, of course. Or, heck,
uses the Docker containers in a kubernetes environment or some such --
I'd love to see someone use that in production.
Anyway, I'm still finishing the API, and the implementation of that API
and the test suite that ensures the API works correctly, but progress is
happening; and as soon as things seem to be working properly, I'll do a
release of SReview 0.6, and will upload that to Debian.
Hopefully that'll be soon.
... Why do you have to be so effing difficult about a YouTube API
project that is used for a single event per year?
FOSDEM creates 600+ videos on a yearly basis. There is no way I am
going to manually upload 600+ videos through your webinterface, so we
use the API you provide, using a
script written by
Stefano Rivera. This script grabs video filenames and metadata from a
YAML file, and then uses your APIs to upload said videos with said
metadata. It works quite well. I run it from cron, and it uploads files
until the quota is exhausted, then waits until the next time the cron
job runs. It runs so well, that the first time we used it, we could
upload 50+ videos on a daily basis, and so the uploads were done as soon
as all the videos were created, which was a few months after the event.
Cool!
The second time we used the script, it did not work at all. We asked one
of our key note speakers who happened to be some hotshot at your
company, to help us out. He contacted the YouTube people, and whatever
had been broken was quickly fixed, so yay, uploads worked again.
I found out later that this is actually a normal thing if you don't use
your API quota for 90 days or more. Because it's happened to us every
bloody year.
For the 2020 event, rather than going through back channels (which
happened to be unavailable this edition), I tried to use your normal
ways of unblocking the API project. This involves creating a screencast
of a bloody command line script and describing various things that don't
apply to FOSDEM and ghaah shoot me now so meh, I created a new API
project instead, and had the uploads go through that. Doing so gives me
a limited quota that only allows about 5 or 6 videos per day, but that's
fine, it gives people subscribed to our channel the time to actually
watch all the videos while they're being uploaded, rather than being
presented with a boatload of videos that they can never watch in a day.
Also it doesn't overload subscribers, so yay.
About three months ago, I started uploading videos. Since then, every
day, the "fosdemtalks" channel on YouTube has published five or six
videos.
Given that, imagine my surprise when I found this in my mailbox this
morning...
This is an outright lie, Google.
The project has been created 90 days ago, yes, that's correct. It has
been used every day since then to upload videos.
I guess that means I'll have to deal with your broken automatic content
filters to try and get stuff unblocked...
... or I could just give up and not do this anymore. After all, all the
FOSDEM content is available on our public video
host, too.
The other day, we went to a designer's fashion shop whose owner was
rather adamant that he was never ever going to wear a face mask, and
that he didn't believe the COVID-19 thing was real. When I argued for
the opposing position, he pretty much dismissed what I said out of hand,
claiming that "the hospitals are empty dude" and "it's all a lie". When
I told him that this really isn't true, he went like "well, that's just
your opinion". Well, no -- certain things are facts, not opinions. Even
if you don't believe that this disease kills people, the idea that this
is a matter of opinion is missing the ball by so much that I was pretty
much stunned by the level of ignorance.
His whole demeanor pissed me off rather quickly. While I disagree with
the position that it should be your decision whether or not to wear a
mask, it's certainly possible to have that opinion. However, whether or
not people need to go to hospitals is not an opinion -- it's something
else entirely.
After calming down, the encounter got me thinking, and made me focus on
something I'd been thinking about before but hadn't fully forumlated:
the fact that some people in this world seem to misunderstand the nature
of what it is to do science, and end up, under the claim of being
"sceptical", with various nonsense things -- see scientology, flat earth
societies, conspiracy theories, and whathaveyou.
So, here's something that might (but probably won't) help some people
figuring out stuff. Even if it doesn't, it's been bothering me and I
want to write it down so it won't bother me again. If you know all this
stuff, it might be boring and you might want to skip this post.
Otherwise, take a deep breath and read on...
Statements are things people say. They can be true or false; "the
sun is blue" is an example of a statement that is trivially false. "The
sun produces light" is another one that is trivially true. "The sun
produces light through a process that includes hydrogen fusion" is
another statement, one that is a bit more difficult to prove true or
false. Another example is "Wouter Verhelst does not have a favourite
color". That happens to be a true statement, but it's fairly difficult
for anyone that isn't me (or any one of the other Wouters Verhelst out
there) to validate as true.
While statements can be true or false, combining statements without more
context is not always possible. As an example, the statement "Wouter
Verhelst is a Debian Developer" is a true statement, as is the statement
"Wouter Verhelst is a professional Volleybal player"; but the statement
"Wouter Verhelst is a professional Volleybal player and a Debian
Developer" is not, because while I am a Debian Developer, I am not a
professional Volleybal player -- I just happen to share a name with
someone who is.
A statement is never a fact, but it can describe a fact. When a
statement is a true statement, either because we trivially know what it
states to be true or because we have performed an experiment that
proved beyond any possible doubt that the statement is true, then what
the statement describes is a fact. For example, "Red is a color" is a
statement that describes a fact (because, yes, red is definitely a
color, that is a fact). Such statements are called statements of
fact. There are other possible statements. "Grass is purple" is a
statement, but it is not a statement of fact; because as everyone knows,
grass is (usually) green.
A statement can also describe an opinion. "The Porsche 911 is a nice
car" is a statement of opinion. It is one I happen to agree with, but it
is certainly valid for someone else to make a statement that conflicts
with this position, and there is nothing wrong with that. As the saying
goes, "opinions are like assholes: everyone has one". Statements
describing opinions are known as statements of opinion.
The differentiating factor between facts and opinions is that facts are
universally true, whereas opinions only hold for the people who state
the opinion and anyone who agrees with them. Sometimes it's difficult or
even impossible to determine whether a statement is true or not. The
statement "The numbers that win the South African Powerball lottery on
the 31st of July 2020 are 2, 3, 5, 19, 35, and powerball 14" is not a
statement of fact, because at the time of writing, the 31st of July 2020
is in the future, which at this point gives it a 1 in 24,435,180 chance
to be true). However, that does not make it a statement of opinion; it
is not my opinion that the above numbers will win the South African
powerball; instead, it is my guess that those numbers will be correct.
Another word for "guess" is hypothesis: a hypothesis is a statement
that may be universally true or universally false, but for which the
truth -- or its lack thereof -- cannot currently be proven beyond doubt.
On Saturday, August 1st, 2020 the above statement about the South
African Powerball may become a statement of fact; most likely however,
it will instead become a false statement.
An unproven hypothesis may be expressed as a matter of belief. The
statement "There is a God who rules the heavens and the Earth" cannot
currently (or ever) be proven beyond doubt to be either true or false,
which by definition makes it a hypothesis; however, for matters of
religion this is entirely unimportant, as for believers the belief that
the statement is correct is all that matters, whereas for nonbelievers
the truth of that statement is not at all relevant. A belief is not an
opinion; an opinion is not a belief.
Scientists do not deal with unproven hypotheses, except insofar that
they attempt to prove, through direct observation of nature (either out
in the field or in a controlled laboratory setting) that the hypothesis
is, in fact, a statement of fact. This makes unprovable hypotheses
unscientific -- but that does not mean that they are false, or even
that they are uninteresting statements. Unscientific statements are
merely statements that science cannot either prove or disprove, and that
therefore lie outside of the realm of what science deals with.
Given that background, I have always found the so-called "conflict"
between science and religion to be a non-sequitur. Religion deals in one
type of statements; science deals in another. The do not overlap, since
a statement can either be proven or it cannot, and religious statements
by their very nature focus on unprovable belief rather than universal
truth. Sure, the range of things that science has figured out the facts
about has grown over time, which implies that religious statements have
sometimes been proven false; but is it heresy to say that "animals exist
that can run 120 kph" if that is the truth, even if such animals don't
exist in, say, Rome?
Something very similar can be said about conspiracy theories. Yes, it is
possible to hypothesize that NASA did not send men to the moon, and that
all the proof contrary to that statement was somehow fabricated.
However, by its very nature such a hypothesis cannot be proven or
disproven (because the statement states that all proof was
fabricated), which therefore implies that it is an unscientific
statement.
It is good to be sceptical about what is being said to you. People can
have various ideas about how the world works, but only one of those
ideas -- one of the possible hypotheses -- can be true. As long as a
hypothesis remains unproven, scientists love to be sceptical
themselves. In fact, if you can somehow prove beyond doubt that a
scientific hypothesis is false, scientists will love you -- it means
they now know something more about the world and that they'll have to
come up with something else, which is a lot of fun.
When a scientific experiment or observation proves that a certain
hypothesis is true, then this probably turns the hypothesis into a
statement of fact. That is, it is of course possible that there's a flaw
in the proof, or that the experiment failed (but that the failure was
somehow missed), or that no observance of a particular event happened
when a scientist tried to observe something, but that this was only
because the scientist missed it. If you can show that any of those
possibilities hold for a scientific proof, then you'll have turned a
statement of fact back into a hypothesis, or even (depending on the
exact nature of the flaw) into a false statement.
There's more. It's human nature to want to be rich and famous, sometimes
no matter what the cost. As such, there have been scientists who have
falsified experimental results, or who have claimed to have observed
something when this was not the case. For that reason, a scientific
paper that gets written after an experiment turned a hypothesis into
fact describes not only the results of the experiment and the observed
behavior, but also the methodology: the way in which the experiment
was run, with enough details so that anyone can retry the experiment.
Sometimes that may mean spending a large amount of money just to be able
to run the experiment (most people don't have an LHC in their backyard,
say), and in some cases some of the required materials won't be
available (the latter is expecially true for, e.g., certain chemical
experiments that involve highly explosive things); but the information
is always there, and if you spend enough time and money reading
through the available papers, you will be able to independently prove
the hypothesis yourself. Scientists tend to do just that; when the
results of a new experiment are published, they will try to rerun the
experiment, partially because they want to see things with their own
eyes; but partially also because if they can find fault in the
experiment or the observed behavior, they'll have reason to write a
paper of their own, which will make them a bit more rich and famous.
I guess you could say that there's three types of people who deal with
statements: scientists, who deal with provable hypotheses and
statements of fact (but who have no use for unprovable hypotheses and
statements of opinion); religious people and conspiracy theorists,
who deal with unprovable hypotheses (where the religious people deal
with these to serve a large cause, while conspiracy theorists only
care about the unprovable hypotheses); and politicians, who should
care about proven statements of fact and produce statements of
opinion, but who usually attempt the reverse of those two these days
Anyway...
[[!img Error: Image::Magick is not installed]]
I require moderately powerful boxes for my work. Moderately powerful boxes
generate a non-negligible amount of heat, and therefore require appropriate
cooling. Most cooling solutions involve a certain degree of noise, which I
dislike.
Water-cooling is supposed to be the most effective solution for noise
reduction, but it always felt like too much of a hassle to try to me.
Considering the various solutions I've tried in the past years, two observations
I've made really stick out:
Noctua's reputation for CPU cooling is well-earned.
A "silent" (meaning sound-proofed) case, isn't. At least, not anymore.
I've tried numerous coolers in the past, some of monstrous proportions (always
thinking that more mass must be better, and reputable brands are equally good),
but I was never really satisfied; hence, I was doubtful that trying yet another
cooler would make a difference. I'm glad I tried the
Noctua NH-D15 anyway. With some
tweaking to the fan profile in the BIOS, it's totally inaudible at normal to
medium workloads, and just a very gentle hum at full load subtle enough to
disappear in the background.
For the past decade, I've also regularly purchased sound-proofed cases, but
this habit appears anachronistic now. Years ago, sound-proofed cases helped
contain the noise of a few HDDs. However, all of my boxes now contain NVMe
drives (which, to me, are the biggest improvement to computing since CPUs
going multi-core).
On the other hand, some of my boxes now contain powerful GPUs used for GPGPU
computing, and with the recent higher-end Nvidia and AMD cards all pulling
in over 300W, there is a lot of heat to manage. The best way to quickly dump
heat is with good airflow. Sound-proofing works against that. Its insulation
restricts airflow, which ultimately causes even more noise, as the GPU's fans
need to spin at very high RPMs. This is, of course, totally obvious in
hindsight.
I removed the front cover from a new GPU box, and the noise went down from
almost-hair-dryer to acceptable-whoosh levels. I'm wondering whether a
completely open box might not be even more effective.
The Rcpp team is proud to announce release 1.0.6 of Rcpp which arrived at CRAN earlier today, and has been uploaded to Debian too. Windows and macOS builds should appear at CRAN in the next few days. This marks the first release on the new six-months cycle announced with release 1.0.5 in July. As reminder, interim dev or rc releases will often be available in the Rcpp drat repo; this cycle there were four.
Rcpp has become the most popular way of enhancing R with C or C++ code. As of today, 2174 packages on CRAN depend on Rcpp for making analytical code go faster and further (which is an 8.5% increase just since the last release), along with 207 in BioConductor.
This release features six different pull requests from five different contributors, mostly fixing fairly small corner cases, plus some minor polish on documentation and continuous integration. Before releasing we once again made numerous reverse dependency checks none of which revealed any issues. So the passage at CRAN was pretty quick despite the large dependency footprint, and we are once again grateful for all the work the CRAN maintainers do.
Changes in Rcpp patch release version 1.0.6 (2021-01-14)
Changes in Rcpp API:
Replace remaining few uses of EXTPTR_PTR with R_ExternalPtrAddr (Kevin in #1098 fixing #1097).
Add push_back and push_front for DataFrame (Walter Somerville in #1099 fixing #1094).
Remove a misleading-to-wrong comment (Mattias Ellert in #1109 cleaning up after #1049).
Address a sanitizer report by initializing two private bool variables (Benjamin Christoffersen in #1113).
External pointer finalizer toggle default values were corrected to true (Dirk in #1115).
Changes in Rcpp Documentation:
Several URLs were updated to https and/or new addresses (Dirk).
Changes in Rcpp Deployment:
Added GitHub Actions CI using the same container-based setup used previously, and also carried code coverage over (Dirk in #1128).
Changes in Rcpp support functions:
Rcpp.package.skeleton() avoids warning from R. (Dirk)
Mainly to recall what happened last year and to give thoughts and plan for the upcoming year(s) I m once again revisiting my previous year (previous editions: 2019, 2018, 2017, 2016, 2015, 2014, 2013 + 2012).
Due to the Coronavirus disease (COVID-19) pandemic, 2020 was special for several reasons, but overall I consider myself and my family privileged and am very grateful for that.
In terms of IT events, I planned to attend Grazer Linuxdays and DebConf in Haifa/Israel. Sadly Grazer Linuxdays didn t take place at all, and DebConf took place online instead (which I didn t really participate in for several reasons). I took part in the well organized DENOG12 + ATNOG 2020/1 online meetings. I still organize our monthly Security Treff Graz (STG) meetups, and for half of the year, those meetings took place online (which worked OK-ish overall IMO).
Only at the beginning of 2020, I managed to play Badminton (still playing in the highest available training class (in german: Kader ) at the University of Graz / Universit ts-Sportinstitut, USI). For the rest of the year except for ~2 weeks in October or so the sessions couldn t occur.
Plenty of concerts I planned to attend were cancelled for obvious reasons, including the ones I would have played myself. But I managed to attend Jazz Redoute 2020 Dom im Berg, Martin Grubinger in Musikverein Graz and Emiliano Sampaio s Mega Mereneu Project at WIST Moserhofgasse (all before the corona situation kicked in). The concert from Ton Feinig & RTV Slovenia Big Band occurred under strict regulations in Summer, as well as Elektra Opera by Richard Strau in a very special setting (only one piano player instead of the orchestra because of a Corona case in the orchestra) in Autumn. At the beginning of 2020, I also visited Literaturshow Roboter mit Senf at Literaturhaus Graz.
The lack of concerts and rehearsals also severely impacted my playing the drums (including at HTU BigBand Graz), which pretty much didn t take place. :(
Grml-wise we managed to publish release 2020.06, codename Ausgehfuahangl. Regarding jenkins-debian-glue I tried to clarify its state and received some really lovely feedback.
I consider 2020 as the year where I dropped regular usage of Jabber (so far my accounts still exist, but I m no longer regularly online and am not sure for how much longer I ll keep my accounts alive as such).
Business-wise it was our seventh year of business with SynPro Solutions GmbH. No big news but steady and ongoing work with my other business duties Grml Solutions and Grml-Forensic.
As usual, I shared childcare with my wife. Due to the corona situation, my wife got a new working schedule, which shuffled around our schedule a bit on Mondays + Tuesdays. Still, we managed to handle the homeschooling/distance learning quite well. Currently we re sitting in the third lockdown, and yet another round of homeschooling/distance learning is going on those days (let s see how long ). I counted 112 actual school days in all of 2020 for our older daughter with only 68 school days since our first lockdown on 16th of March, whereas we had 213(!) press conferences by our Austrian government in 2020. (Further rants about the situation in Austria snipped.)
Book reading-wise I managed to complete 60 books (see Mein Lesejahr 2020 ). Once again, I noticed that what felt like good days for me always included reading books, so I ll try to keep my reading pace for 2021. I ll also continue with my hobbies Buying Books and Reading Books , to get worse at Tsundoku.
Hoping for vaccination and a more normal 2021, Schwuppdiwupp!
Before and during FOSDEM 2020, I agreed with the people (developers, supporters, managers) of the UBports Foundation to package the Unity8 Operating Environment for Debian. Since 27th Feb 2020, Unity8 has now become Lomiri.
Things got delayed a little recently as my main developer contact on the upstream side was on sick leave for a while. Fortunately, he has now fully recovered and work is getting back on track.
Recent Uploads to Debian related to Lomiri
Over the past 3 months I worked on the following bits and pieces regarding Lomiri in Debian:
Work on DEB package qtmir.
Revisit upstream status of lomiri-ui-toolkit (and remind upstream on open issues to be resolved).
Debug googletest / cmake-extras build failures in qtmir DEB package.
Prepare MR!2 f r cmake-extras upstream, upload cmake-extras 1.5-3 to Debian unstable.
Upload qtmir to unstable/NEW, submit patches upstream.
Upload to unstable as NEW: ayatana-indicator-bluetooth.
Upload to unstable: qtmir 0.6.1-2.
Upload to unstable: cmake-extras 1.5-5.
More debugging on googletest / cmake-extras.
Qt5.15 transition in Debian, various uploads.
Upload unstable: cmake-extras 1.5-6.
Debug and fix libqtdbustest / libqtdbusmock failures in Debian.
Get in touch with RAOF (on IRC, from the Mir Team) on possible fixes of Mir FTBFS in Debian (after a Mesa upload).
Upload to unstable: mir 1.8.0+dfsg1-11 (fix FTBFS).
Fix dbus-cpp autopkgtests.
Fix Mir FTBFS after Boost 1.74 transition.
Fix FTBFS in unit tests in lomiri-url-dispatcher.
Drop autopilot dependency from DEB package qtmir.
Heads-Up Meeting with Marius Gripsgard (discussion next steps).
The next projects / packages ahead are lomiri-ui-toolkit, integrating changes required for Lomiri into Ayatana Indicators and then moving on with several other, smaller packages.
Credits
Many big thanks go to Marius and Dalton for their work on the UBports project and being always available for questions, feedback, etc.
Thanks to Florian Leeber for being my point of contact for topcis regarding my cooperation with the UBports Foundation.
Note: this is another article in my series on asynchronous communication in Linux with UUCP and NNCP.
In the previous installment on store-and-forward backups, I mentioned how easy it is to do with ZFS, and some of the tools that can be used to do it without ZFS. A lot of those tools are a bit less robust, so we need some sort of store-and-forward mechanism to verify backups. To be sure, verifying backups is good with ANY scheme, and this could be used with ZFS backups also.
So let s say you have a shiny new backup scheme in place, and you d like to verify that it s working correctly. To do that, you need to compare the source directory tree on machine A with the backed-up directory tree on machine B.
Assuming a conventional setup, here are some ways you might consider to do that:
Just copy everything from machine A to machine B and compare locally
Or copy everything from machine A to a USB drive, plug that into machine B, and compare locally
Use rsync in dry-run mode and see if it complains about anything
The first two options are not particularly practical for large datasets, though I note that the second is compatible with airgapping. Using rsync requires both systems to be online at the same time to perform the comparison.
What would be really nice here is a tool that would write out lots of information about the files on a system: their names, sizes, last modified dates, maybe even sha256sum and other data. This file would be far smaller than the directory tree itself, would compress nicely, and could be easily shipped to an airgapped system via NNCP, UUCP, a USB drive, or something similar.
Tool choices
It turns out there are already quite a few tools in Debian (and other Free operating systems) to do this, and half of them are named mtree (though, of course, not all mtrees are compatible with each other.) We ll look at some of the options here.
I ve made a simple test directory for illustration purposes with these commands:
mkdir test
cd test
echo hi > hi
ln -s hi there
ln hi foo
touch empty
mkdir emptydir
mkdir somethingdir
cd somethingdir
ln -s ../there
I then also used touch to set all files to a consistent timestamp for illustration purposes.
Tool option: getfacl (Debian package: acl)
This comes with the acl package, but can be used with other than ACL purposes. Unfortunately, it doesn t come with a tool to directly compare its output with a filesystem (setfacl, for instance, can apply the permissions listed but won t compare.) It ignores symlinks and doesn t show sizes or dates, so is ineffective for our purposes.
Example output:
Tool option: fmtree, the FreeBSD mtree (Debian package: freebsd-buildutils)
fmtree can prepare a specification based on a directory tree, and compare a directory tree to that specification. The comparison also is aware of files that exist in a directory tree but not in the specification. The specification format is a bit on the odd side, but works well enough with fmtree. Here s a sample output with defaults:
You might be wondering here what it does about special characters, and the answer is that it has octal escapes, so it is 8-bit clean.
To compare, you can save the output of fmtree to a file, then run like this:
cd test
fmtree < ../test.fmtree
If there is no output, then the trees are identical. Change something and you get a line of of output explaining each difference. You can also use fmtree -U to change things like modification dates to match the specification.
fmtree also supports quite a few optional keywords you can add with -K. They include things like file flags, user/group names, various tipes of hashes, and so forth. I'll note that none of the options can let you determine which files are hardlinked together.
Here's an excerpt with -K sha256digest added:
If you include a sha256digest in the spec, then when you verify it with fmtree, the verification will also include the sha256digest. Obviously fmtree -U can't correct a mismatch there, but of course it will detect and report it.
Tool option: mtree, the NetBSD mtree (Debian package: mtree-netbsd)
mtree produces (by default) output very similar to fmtree. With minor differences (such as the name of the sha256digest in the output), the discussion above about fmtree also applies to mtree.
There are some differences, and the most notable is that mtree adds a -C option which reads a spec and converts it to a "format that's easier to parse with various tools." Here's an example:
Most definitely an improvement in both space and convenience, while still retaining the relevant information. Note that if you want the sha256digest in the formatted output, you need to pass the -K to both mtree invocations. I could have done that here, but it is easier to read without it.
mtree can verify a specification in either format. Given what I'm about to show you about bsdtar, this should illustrate why I bothered to package mtree-netbsd for Debian.
Unlike fmtree, the mtree -U command will not adjust modification times based on the spec, but it will report on differences.
Tool option: bsdtar (Debian package: libarchive-tools)
bsdtar is a fascinating program that can work with many formats other than just tar files. Among the formats it supports is is the NetBSD mtree "pleasant" format (mtree -C compatible).
bsdtar can also convert between the formats it supports. So, put this together: bsdtar can convert a tar file to an mtree specification without extracting the tar file. bsdtar can also use an mtree specification to override the permissions on files going into tar -c, so it is a way to prepare a tar file with things owned by root without resorting to tools like fakeroot.
Let's look at how this can work:
You can use mtree -U to verify that as before. With the --options mtree: set, you can also add hashes and similar to the bsdtar output. Since bsdtar can use input from tar, pax, cpio, zip, iso9660, 7z, etc., this capability can be used to create verification of the files inside quite a few different formats. You can convert with bsdtar -cf output.mtree --format=mtree @input.tar. There are some foibles with directly using these converted files with mtree -U, but usually minor changes will get it there.
Side mention: stat(1) (Debian package: coreutils)
This tool isn't included because it won't operate recursively, but is a tool in the similar toolbox.
Putting It Together
I will still be developing a complete non-ZFS backup system for NNCP (or UUCP) in a future post. But in the meantime, here are some ideas you can reflect on:
Let's say your backup scheme involves sending a full backup every night. On the source system, you could pipe the generated tar file through something like tee >(bsdtar -cf bcakup.mtree @-) to generate an mtree file in-band while generating the tar file. This mtree file could be shipped over for verification.
Perhaps your backup scheme involves sending incremental backup data via rdup or even ZFS, but you would like to periodically verify that everything is good -- that an incremental didn't miss something. Something like mtree -K sha256 -c -x -p / mtree -C -K sha256 would let you accomplish that.
I will further develop at least one of these ideas in a future post.
Bonus: cross-tool comparisons
In my mtree-netbsd packaging, I added tests like this to compare between tools:
It is undeniable that banning Donald Trump from Facebook, Twitter, and similar sites is a benefit for the moment. It may well save lives, perhaps lots of lives. But it raises quite a few troubling issues.
First, as EFF points out, these platforms have privileged speakers with power, especially politicians, over regular users. For years now, it has been obvious to everyone that Donald Trump has been violating policies on both platforms, and yet they did little or nothing about it. The result we saw last week was entirely forseeable and indeed, WAS forseen, including by elements in those companies themselves. (ACLU also raises some good points)
Contrast that with how others get treated. Facebook, two days after the coup attempt, banned Benjamin Wittes, apparently because he mentioned an Atlantic article opposed to nutcase conspiracy theories. The EFF has also documented many more egregious examples: taking down documentation of war crimes, childbirth images, black activists showing the racist messages they received, women discussing online harassment, etc. The list goes on; YouTube, for instance, has often been promoting far-right violent videos while removing peaceful LGBTQ ones.
In short, have we simply achieved legal censorship by outsourcing it to dominant corporations?
It is worth pausing at this point to recognize two important princples:
First, that we do not see it as right to compel speech.
Secondly, that there exist communications channels and other services that nobody is calling on to suspend Donald Trump.
Let s dive into those a little bit.
There have been no prominent calls for AT&T, Verizon, Gmail, or whomever provides Trump and his campaign with cell phones or email to suspend their service to him. Moreover, the gas stations that fuel his vehicles and the airports that service his plane continue to provide those services, and nobody has seriously questioned that, either. Even his Apple phone that he uses to post to Twitter remains, as far as I know, fully active.
Secondly, imagine you were starting up a small web forum focused on raising tomato plants. It is, and should be, well within your rights to keep tomato-haters out, as well as people that have no interest in tomatoes but would rather talk about rutabagas, politics, or Mars. If you are going to host a forum about tomatoes, you have the right to keep it a forum about tomatoes; you cannot be forced to distribute someone else s speech. Likewise in traditional media, a newspaper cannot be forced to print every letter to the editor in full.
In law, there is a notion of a common carrier, that provides services to the general public without discrimination. Phone companies and ISPs fall under this.
Facebook, Twitter, and tomato sites don t. But consider what happens if Facebook bans you. You might be using Facebook-owned Whatsapp to communicate with family and friends, and suddenly find yourself unable to ask someone to pick you up. Or your treasured family photos might be in Facebook-owned Instagram, lost forever. It s not just Facebook; similar things happen with Google, locking people out of their phones and laptops, their emails, even their photos.
Is it right that Facebook and Google aren t regulated as common carriers? Perhaps, or perhaps we need some line of demarcation between their speech-to-the-public services (Facebook timeline posts, YouTube) and private communication (Whatsapp, Gmail). It s a thorny issue; should government be regulating speech instead? That s also fraught. So is corporate control.
Decentralization Helps Dramatically
With email, you get to pick your email provider (yes, there are two or three big ones, but still plenty of others). Each email provider will have its own set of things it considers acceptable, and its own set of other servers and accounts it s willing to exchange mail with. (It is extremely common for mail providers to choose not to accept mail from various other mail servers based on ISP, IP address, reputation, and so forth.)
What if we could do something like that for Twitter and Facebook?
Let you join whatever instance you like. Maybe one instance is all about art and they don t talk about politics. Or another is all about Free Software and they don t have advertising. And then there are plenty of open instances that accept anything that s respectful. And, like email, people of one server can interact with those using another just as easily as if they were using the same one.
Well, this isn t hypothetical; it already exists in the Fediverse. The most common option is Mastodon, and it so happens that a month ago I wrote about its benefits for other reasons, and included some links on getting started.
There is no reason that we must all let our online speech be controlled by companies with a profit motive to keep hate speech on their platforms. There is no reason that we must all have a single set of rules, or accept strong corporate or government control, either. The quality of conversation on Mastodon is far higher than either Twitter or Facebook; decentralization works and it s here today.
Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 802 other packages on CRAN.
This release was needed because we use the Matrix package for some (optional) tests related to sparse matrices, and a small and subtle change and refinement in the recent 1.3.0 release of Matrix required us to make an update for the testing. Nothing has changed in how we set up, or operate on, sparse matrices. My thanks to Binxiang and Martin Maechler for feedback and suggestion on the initial fix both Binxiang and I set up independently. At the same time we upgrade some package internals related to continuous integration (for that, also see my blog post and video from earlier this week). Lastly Conrad sent in a one-line upstream fix for dealing with NaN in sign().
The full set of changes follows.
Changes in RcppArmadillo version 0.10.1.2.2 (2021-01-08)
Correct one unit test for Matrix 1.3.0-caused changed (Binxiang in #319 and Dirk in #322).
Like 
... isn't ready yet, but it's getting there.
I had planned to release a new version of SReview, my online video review and
transcoding system that I wrote originally for FOSDEM but is being used for
DebConf, too, after it was set up and running properly for FOSDEM 2020.
However, things got a bit busy (both in my personal life and in the world at
large), so it fell a bit by the wayside.
I've now also been working on things a bit more, in preparation for an
improved administrator's interface, and have started implementing a REST
API to deal with talks etc through HTTP calls. This seems to be coming
along nicely, thanks to
Anyway...
[[!img 
The Rcpp team is proud to announce release 1.0.6 of 
Mainly to recall what happened last year and to give thoughts and plan for the upcoming year(s) I m once again revisiting my previous year (previous editions: 
Before and during FOSDEM 2020, I agreed with the people (developers, supporters, managers) of the UBports Foundation to package the Unity8 Operating Environment for Debian. Since 27th Feb 2020, Unity8 has now become Lomiri.
Things got delayed a little recently as my main developer contact on the upstream side was on sick leave for a while. Fortunately, he has now fully recovered and work is getting back on track.
Recent Uploads to Debian related to Lomiri
Over the past 3 months I worked on the following bits and pieces regarding Lomiri in Debian:
