Search Results: "xam"

13 March 2021

Enrico Zini: nspawn-runner and ansible

This post is part of a series about trying to setup a gitlab runner based on systemd-nspawn. I published the polished result as nspawn-runner on GitHub. New goal: make it easier to configure and maintain chroots. For example, it should be possible to maintain a rolling testing or sid chroot without the need to manually log into it to run apt upgrade. It should be also be easy to have multiple runners reasonably in sync by carrying around a few simple configuration files, representing the set of images available to the CI. Ideally, those configuration files could simply be one ansible playbook per chroot. nspawn-runner could have a 'chroot-maintenance' command that runs all the playbooks on their corresponding chroots, and that would be all I need. ansible and systemd-nspawn ansible being inadequate as usual, it still does not have a nspawn or machinectl connector, even though the need is there, and one can find requests, pull requests, and implementation attempts by all sorts of people, including me. However, I don't want to have nspawn-runner depend on random extra plugins. There's a machinectl become plugin available in ansible from buster-backports, but no matter how I read its scant documentation, looked around the internet, and tried all sorts of things, I couldn't manage to figure out what it is for. This said, simply using systemd-nspawn instead of chroot is quite trivial: use ansible_connection: chroot, set ansible_chroot_exe to this shellscript, and it just works, with things properly mounted, internet access, correct hostnames, and everything:
#!/bin/sh
chroot="$1"
shift
exec systemd-nspawn --console=pipe -qD "$chroot" -- "$@"
I guess that's a, uhm, plan, I guess? Running playbooks As an initial prototype, I made nspawn check the list of chroots in /var/lib/nspawn-runner, and for each directory found there, check if there's an equivalent .yaml or .yml file next to nspawn-runner. For each chroot+playbook combination, it creates an inventory with the right setup, including the custom chroot command, and runs ansible-playbook. As a prototype it works. I assume once it will see usage there's going to be feedback and fine tuning; meanwhile, I have the beginning of some automated maintenance for the CI chroots. Next step It would be nice to also have nspawn-runner create the chroots from configuration files if they are missing, so that a new runner can be deployed with a minimal effort, and it will proceed to generate all the images required in a single command. For this, I'd like to find a clean way to store the chroot creation command inside the playbooks, to keep just one configuration file per chroot. I'd also like to have it flexible enough to run debootstrap, as well as commands for different distributions. Time will tell. This is probably enough for study/design posts on my blog. Further updates will be in the issue tracker.

Enrico Zini: nspawn-runner and btrfs

This post is part of a series about trying to setup a gitlab runner based on systemd-nspawn. I published the polished result as nspawn-runner on GitHub. systemd-nspawn has an interesting --ephemeral option that sets up temporary copy-on-write filesystem snapshots on filesystems that support it, like btrfs. Using copy on write means that one could perform maintenance on the source chroots, without disrupting existing CI sessions. btrfs and copy on write btrfs snapshots work on subvolumes. As I understand it, if one uses btrfs subvolume create instead of mkdir, what is inside the resulting directory is managed as a subvolume that can be snapshotted and managed in all sorts of interesting ways. I managed to delete a subvolume equally well with btrfs subvolume delete and with rm -r. btrfs subvolume snapshot src dst is similar to cp -a, but it makes a copy-on-write snapshot of a btrfs subvolume. If I change nspawn-runner to manage each chroot in its own subvolume, I should be able to build on all these features, and systemd-nspawn should be able to do that, too. There's a cute shortcut to migrate a subdirectory to a subvolume: create the subvolume, then use cp -r --reflink to populate the subvolume with the directory contents. systemd-nspawn and btrfs Passing -x/--ephemeral to systemd-nspawn makes it do all the transient copy-on-write work automatically:
# systemd-nspawn -xD buster
Spawning container buster-7fd47ac79296c5d3 on /var/lib/nspawn-runner/t/.#machine.buster0939fbc61fcbca28.
Press ^] three times within 1s to kill container.
root@buster-7fd47ac79296c5d3:~# mkdir foo
root@buster-7fd47ac79296c5d3:~# ls -la
total 12
drwx------ 1 root root  62 Mar 13 16:30 .
drwxr-xr-x 1 root root 154 Mar 13 16:26 ..
-rw------- 1 root root 102 Mar 13 16:26 .bash_history
-rw-r--r-- 1 root root 570 Mar 13 16:26 .bashrc
-rw-r--r-- 1 root root 148 Mar 13 16:26 .profile
drwxr-xr-x 1 root root   0 Mar 13 16:30 foo
root@buster-7fd47ac79296c5d3:~# logout
Container buster-7fd47ac79296c5d3 exited successfully.
root@runner2:/var/lib/nspawn-runner/t# ls -la buster/root/
totale 12
drwx------ 1 root root  56 mar 13 16:26 .
drwxr-xr-x 1 root root 154 mar 13 16:26 ..
-rw------- 1 root root 102 mar 13 16:26 .bash_history
-rw-r--r-- 1 root root 570 mar 13 16:26 .bashrc
-rw-r--r-- 1 root root 148 mar 13 16:26 .profile
It also works on a directory that is not a subvolume, making reflinks of its contents instead of a subvolume snapshot, although this has a performance penalty on setup: Snapshotting a subvolume:
# time systemd-nspawn -xD buster ls
Spawning container buster-7ab8f4123420b5d5 on /var/lib/nspawn-runner/t/.#machine.bustercd54ef4971229ff5.
Press ^] three times within 1s to kill container.
bin  boot  dev  etc  home  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
Container buster-7ab8f4123420b5d5 exited successfully.
real    0m0,164s
user    0m0,032s
sys 0m0,014s
Reflink-ing a subdirectory:
# time systemd-nspawn -xD buster ls
Spawning container buster-ebc9dc77db0c972d on /var/lib/nspawn-runner/.#machine.buster2ecbcbd1a1a058b8.
Press ^] three times within 1s to kill container.
bin  boot  dev  etc  home  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
Container buster-ebc9dc77db0c972d exited successfully.
real    0m3,022s
user    0m0,326s
sys 0m2,406s
Detecting filesystem type I can change nspawn-runner to use btrfs-specific features only when /var/lib/nspawn-runner is on btrfs. Here's a command to detect the filesystem type:
# stat -f -c %T /var/lib/nspawn-runner/
btrfs
nspawn-runner updated I've refactored nspawn-runner splitting backend and frontend code, and implementing multiple backends based on what's the filesystem type of /var/lib/nspawn-runner/. It works really nicely, and with no special configuration required: if /var/lib/nspawn-runner is on btrfs, things run faster, with less kludges, and one can do maintenance on the base chroots without interfering with running CI jobs. Next step The next step is making it easier to configure and maintain chroots. For example, it should be possible to maintain a rolling testing or sid chroot without the need to manually log into it to run apt upgrade.

Louis-Philippe V ronneau: Preventing an OpenPGP Smartcard from caching the PIN eternally

While I'm overall very happy about my migration to an OpenPGP hardware token, the process wasn't entirely seamless and I had to hack around some issues, for example the PIN caching behavior in GnuPG. As described in this bug the cache-ttl parameter in GnuPG is not implemented and thus does nothing. This means once you type in your PIN, it is cached for as long as the token is plugged. Security-wise, this is not great. Instead of manually disconnecting the token frequently, I've come up with a script that restarts scdameon if the token hasn't been used during the last X minutes. It seems to work well and I call it using this cron entry: */5 * * * * my_user /usr/local/bin/restart-scdaemon To get a log from scdaemon, you'll need a ~/.gnupg/scdaemon.conf file that looks like this:
debug-level basic
log-file /var/log/scdaemon.log
Hopefully it can be useful to others!
 #!/usr/bin/python3
# Copyright 2021, Louis-Philippe V ronneau <pollo@debian.org>
#
# This script is free software: you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free Software
# Foundation, either version 3 of the License, or (at your option) any later
# version.
# 
# This script is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
# details.
# 
# You should have received a copy of the GNU General Public License along with
# this script. If not, see <http://www.gnu.org/licenses/>.
"""
This script restarts scdaemon after X minutes of inactivity to reset the PIN
cache. It is meant to be ran by cron each X/2 minutes.
This is needed because there is currently no way to set a cache time for
smartcards. See https://dev.gnupg.org/T3362#137811 for more details.
"""
import os
import sys
import subprocess
from datetime import datetime, timedelta
from argparse import ArgumentParser
p = ArgumentParser(description=__doc__)
p.add_argument('-l', '--log', default="/var/log/scdaemon.log",
               help='Path to the scdaemon log file.')
p.add_argument('-t', '--timeout', type=int, default="10",
               help=("Desired cache time in minutes."))
args = p.parse_args()
def get_last_line(scdaemon_log):
    """Returns the last line of the scdameon log file."""
    with open(scdaemon_log, 'rb') as f:
        f.seek(-2, os.SEEK_END)
        while f.read(1) != b'\n':
            f.seek(-2, os.SEEK_CUR)
        last_line = f.readline().decode()
    return last_line
def check_time(last_line, timeout):
    """Returns True if scdaemon hasn't been called since the defined timeout."""
    # We don't need to restart scdaemon if no gpg command has been run since
    # the last time it was restarted.
    should_restart = True
    if "OK closing connection" in last_line:
        should_restart = False
    else:
        last_time = datetime.strptime(last_line[:19], '%Y-%m-%d %H:%M:%S')
        now = datetime.now()
        delta = now - last_time
        if delta <= timedelta(minutes = timeout):
            should_restart = False
    return should_restart
def restart_scdaemon(scdaemon_log):
    """Restart scdaemon and verify the restart process was successful."""
    subprocess.run(['gpgconf', '--reload', 'scdaemon'], check=True)
    last_line = get_last_line(scdaemon_log)
    if "OK closing connection" not in last_line:
        sys.exit("Restarting scdameon has failed.")
def main():
    """Main function."""
    last_line = get_last_line(args.log)
    should_restart = check_time(last_line, args.timeout)
    if should_restart:
        restart_scdaemon(args.log)
if __name__ == "__main__":
    main()

12 March 2021

Ryan Kavanagh: Static Comments in Hugo

I switched from Jekyll to Hugo last week for a variety of reasons. One thing that was missing was a port of the jekyll-static-comments plugin that I used to use. I liked it because it saved readers from being tracked by Disqus or other comments solutions, and it required no javascript. To comment, users would email me their comment following a template attached to the bottom of each post. I then piped their email through a script to add it to the right post. As an added benefit, I could delegate comment spam detection to my mail server. I ve managed to reimplement this setup using Hugo. For those who are interested in a similar setup, here is what you need to do.

Pages with comments Instead of being single files, pages need to be leaf bundles. For example, this means that your blog post must be located at /content/blog/2021-03-12-static-comments-in-hugo/index.md instead of /content/blog/2021-03-12-static-comments-in-hugo.md. This lets you store the comments as page resources in the subdirectory /content/blog/2021-03-12-static-comments-in-hugo/comments/.

Partials You should create a comments.html partial and include it in the layout for the pages which should get comments:
<div class="post-comments">
  <p class="comment-notice"><b>Comments</b>: To comment on this post,
	send me an email following the template below. Your email address
	will not be posted, unless you choose to include it in
	the <span style="font-family: monospace;">link:</span> field.</p>
  <pre class="comment-notice">
To: Your Name &lt;your.email<span>@</span>example.org&gt;
Subject: [blog-comment]   .Page.RelPermalink  
post_id:   .Page.RelPermalink  
author: [How should you be identified? Usually your name or "Anonymous"]
link: [optional link to your website]
Your comments here. Markdown syntax accepted.</pre>
    $scratch := newScratch  
    $scratch.Set "comments" (.Resources.Match "comments/*yml")  
    if eq 1 (len ($scratch.Get "comments"))  
  <h2>1 Comment</h2>
    else  
  <h2>  len ($scratch.Get "comments")   Comments</h2>
    end  
    range ($scratch.Get "comments")  
  <div class="post-comment  % cycle 'odd', 'even' % ">
	  $comment := (.Content   transform.Unmarshal)  
	<span class="post-meta">
		 - $comment.date   dateFormat "Jan 2, 2006 at 15:04" - 
	</span>
	<h3 class="comment-header">
	    if $comment.link  
	  <a href="  $comment.link  ">  $comment.author  </a>
	    else  
	    $comment.author  
	    end  
	  <br />
	</h3>
	  $comment.comment   markdownify  
  </div>
    end  
</div>

Comments To associate comments received by email to posts, I pipe them from mutt (using the keybinding) to the following (admittedly janky) shell script. It takes the comment, reformats it appropriately, and puts it in the post s comments subdirectory. Note that it determines which filename to use based on the email s contents, so make sure to check that the email doesn t contain anything nefarious before you pipe it into the script!
#!/bin/sh
# Copyright (C) 2016-2021 Ryan Kavanagh <rak@rak.ac>
# Distributed under the ISC license
BLOG_BASE="/media/t/work/blog"
MESSAGE=$(cat)
EMAIL=$(echo "$ MESSAGE "   grep "From:"   sed -e 's/From[^<]*<\?\([^>]*\)>\?.*/\1/g;s/@/-at-/g')
DATE=$(echo "$ MESSAGE "   grep "Date:"   sed -e 's/Date:\s*//g'   xargs -0 date -Iseconds -u -d)
POST_ID=$(echo "$ MESSAGE "   grep "post_id:"   sed -e 's/post_id: //g')
COMMENTS_DIR="$ BLOG_BASE /content/$ POST_ID /comments/"
COMMENT_FILE="$ COMMENTS_DIR /$ DATE _$ EMAIL .yml"
# Strip out the email headers and whitespace until the start of the comment
COMMENT_WHOLE=$(echo "$ MESSAGE "   sed -e '/^\s*$/,$!d;/^[^\s]/,$!d')
# Indent everything after the comment header
COMMENT_INDENTED=$(echo "$ COMMENT_WHOLE "   sed -e '/^\s*$/,$ s/.*/  &/g ')
# And add the comment header
COMMENT_PREFIXED=$(echo "$ COMMENT_INDENTED "   sed -e '0,/^\s*$/ s/^\s*$/comment:  / ')
[ -d "$ COMMENTS_DIR " ]   mkdir -p "$ COMMENTS_DIR "
echo "Saving the comment to $ COMMENT_FILE "
echo "date: $ DATE "   tee "$ COMMENT_FILE "
echo "$ COMMENT_PREFIXED "   tee -a "$ COMMENT_FILE "
For example, the following comment in an email body:
post_id: /blog/2021-03-12-static-comments-in-hugo/
author: Ryan Kavanagh
link: https://rak.ac/
Dear self,
Here is a test comment for your blog post.
It supports *markdown* **syntax** and  stuff .
Best,
Yourself
results in a file content/blog/2021-03-12-static-comments-in-hugo/comments/2021-03-12T18:47:25+00:00_rak-at-example.org.yml containing:
date: 2021-03-12T18:47:25+00:00
post_id: /blog/2021-03-12-static-comments-in-hugo/
author: Ryan Kavanagh
link: https://rak.ac/
comment:  
  Dear self,

  Here is a test comment for your blog post.
  It supports *markdown* **syntax** and  stuff .

  Best,
  Yourself  
You can see the rendered output at the bottom of this page.

10 March 2021

Sean Whitton: consfigurator 0.3.1

I d like to briefly introduce my new project, Consfigurator:
Consfigurator is a system for declarative configuration management using Common Lisp. You can use it to configure hosts as root, deploy services as unprivileged users, build and deploy containers, and produce disc images. [not all of these are implemented yet, but the design permits them to be] Consfigurator s design gives you a great deal of flexibility about how to control the hosts you want to configure. If there is a command you can run which will obtain input and output streams attached to an interactive POSIX sh running on the target host/container, then with a little glue code, you can use much of Consfigurator s functionality to configure that host/container. But if it is possible to get an implementation of Common Lisp started up on the host, then Configurator can transparently execute your deployment code over on the remote side, rather than exchanging information via POSIX sh. This lets you use the full power of Common Lisp to deploy your configuration. Configurator has convenient abstractions for combining these different ways to execute your configuration on hosts with different ways of connecting to them. Connections can be arbitrarily nested. For example, to combine SSHing to a Debian machine as an unprivileged user, using sudo to become root, and then starting up a Lisp image to execute your deployment code, you would evaluate
    (deploy ((:ssh (:sudo :as "spwhitton@athena.example.com") :debian-sbcl)) athena.example.com)
Declarative configuration management systems like Consfigurator and Propellor share a number of goals with projects like the GNU Guix System and NixOS. However, tools like Consfigurator and Propellor try to layer the power of declarative and reproducible configuration on top of traditional, battle-tested unix system administration infrastructure like apt, dpkg, yum, and distro package archives, rather than seeking to replace any of those. Let s get as much as we can out of all that existing distro policy-compliant work!
Please check out the user s manual, which includes a tutorial/quick start guide, and come join us in #consfigurator on irc.oftc.net. It s early days but you can already do a fair few things with Consfigurator. It s a good time to come help get all the basic properties defined!

9 March 2021

Matthew Garrett: Unauthenticated MQTT endpoints on Linksys Velop routers enable local DoS

(Edit: this is CVE-2021-1000002)

Linksys produces a series of wifi mesh routers under the Velop line. These routers use MQTT to send messages to each other for coordination purposes. In the version I tested against, there was zero authentication on this - anyone on the local network is able to connect to the MQTT interface on a router and send commands. As an example:
mosquitto_pub -h 192.168.1.1 -t "network/master/cmd/nodes_temporary_blacklist" -m ' "data": "client": "f8:16:54:43:e2:0c", "duration": "3600", "action": "start" '
will ask the router to block the client with MAC address f8:16:54:43:e2:0c from the network for an hour. Various other MQTT topics pass parameters to shell scripts without quoting them or escaping metacharacters, so more serious outcomes may be possible.

The vendor has released two firmware updates since report - I have not verified whether either fixes this, but the changelog does not indicate any security issues were addressed.

Timeline:

2020-07-30: Submitted through the vendor's security vulnerability report form, indicating that I plan to disclose in either 90 days or after a fix is released. The form turns out to file a Bugcrowd submission.
2020-07-30: I claim the Bugcrowd submission.
2020-08-19: Vendor acknowledges the issue, is able to reproduce and assigns it a P3 priority.
2020-12-15: I ask if there's an update.
2021-02-02: I ask if there's an update.
2021-02-03: Bugcrowd raise a blocker on the issue, asking the vendor to respond.
2021-02-17: I ask for permission to disclose.
2021-03-09: In the absence of any response from the vendor since 2020-08-19, I violate Bugcrowd disclosure policies and unilaterally disclose.

comment count unavailable comments

Emmanuel Kasper: Displaying CSV files in a readable way on the terminal

Until this week I did not know about the column command.

$ head -5 zillow.csv
"Index", "Living Space (sq ft)", "Beds", "Baths", "Zip", "Year", "List Price ($)"
1, 2222, 3, 3.5, 32312, 1981, 250000
2, 1628, 3, 2, 32308, 2009, 185000
3, 3824, 5, 4, 32312, 1954, 399000
4, 1137, 3, 2, 32309, 1993, 150000

Turned out this file is much more readable with a good pipe (and a large screen)
$ head -5 zillow.csv column --table --separator ,
"Index" "Living Space (sq ft)" "Beds" "Baths" "Zip" "Year" "List Price ($)"
1 2222 3 3.5 32312 1981 250000
2 1628 3 2 32308 2009 185000
3 3824 5 4 32312 1954 399000
4 1137 3 2 32309 1993 150000

column is part of util-linux and is thus available in all distributions.
Example file taken from this example list.

Sylvestre Ledru: Debian running on Rust coreutils

tldr: Rust/coreutils ( https://github.com/uutils/coreutils/ ) is now available in Debian, good enough to boot a Debian with GNOME, install the top 1000 packages, build Firefox, the Linux Kernel and LLVM/Clang. Even if I wrote more than 100 patches to achieve that, it will probably be a bumpy ride for many other use cases.
It is also a terrific project to learn Rust. See the list of good first bugs. Even if I see Rust code every day at Mozilla, I was looking for an actual personal project (i.e. this isn't a Mozilla project) to learn Rust during the various COVID lockdowns. I started contributing to the alternative Coreutils developed in Rust. The project aims at proposing a drop-in replacement of the C-based GNU Coreutils, and I wanted to evaluate if this could be used to run a regular Debian. Similar to what I have done with clang.debian.net a few years ago (rebuilding the Debian archive using clang instead of gcc). I expect that most of the readers know what is the Coreutils. It is a set of programs performing simple operations (copy/move file, change permissions/ownership, etc). Even if some commands are from the 70s, they are at the base of Linux, Unix and macOS. While different implementations can be found, they are trying to remain compatible in terms of arguments, options, etc. This implementation of Coreutils isn t different! If you want to learn more about the history of Unix, I recommend this great Corecursive podcast with Brian Kernighan. While a lot of people contributed to this project, much was left to be done: To start easy, I defined 4 goals for this work:
  1. Package Coreutils in Debian/Ubuntu
  2. Boot a Debian system with a Rust-based coreutils
  3. Install the top 1000 packages in Debian - including GNOME
  4. Build Firefox, the Linux Kernel and LLV/Clang

Packaging of Coreutils in Debian Packaging in Debian isn't a trivial or even simple task. It requires uploading independently all the dependencies in the archive. Rust, with its new ecosystem and small crates, is making this task significantly harder. The package is called rust-coreutils - https://tracker.debian.org/pkg/rust-coreutils For Debian/Ubuntu users, to have an idea of the complexity of packaging such applications, just run
debtree --build-dep rust-coreutils dot -Tsvg > coreutils.svg (should be around 1M). Since it isn't production ready, the rust-coreutils is installable in parallel with coreutils. This package does NOT replace the GNU/coreutils files (yet?), the new files are installed in /usr/lib/cargo/bin/. They can be used with: export PATH=/usr/lib/cargo/bin/:$PATH Or, uglier, overriding the files with the new ones.

Booting Debian with rust-coreutils To achieve this, because I knew I would likely break the image a few times, I created a new project to quickly install a full Debian with PXE and preseed. The project is available here: https://github.com/opencollab/qemu-debian-install-pxe-preseed/ A script to create the full qemu image: build_qemu_debian_image.sh A second script to boot on the newly created image: boot.sh Then, building and installing coreutils on the system (yeah, it is ugly - don t do that at home): apt install rust-coreutils
cd /usr/lib/cargo/bin/
for f in *; do
cp -f $f /usr/bin/
done
First surprise, unlike the old init.d init system, as systemd is not relying on a series of scripts (it is mostly written in C), replacing the coreutils did not have an impact. Therefore, I didn't experience any issue during the boot process

Implementing missing options A significant number of problems could be easily identified as a lack of support for some options. Here is a list of most of the fixes I had to implement to make this plan work:

Different behavior Most of the programs behaved as expected. Here is a list of differences:
  • install doesn't support using /dev/null as source file
    Setting up libreoffice-common (1:6.1.5-3+deb10u6) ...
    install: error: install: cannot install /dev/null to /etc/apparmor.d/local/usr.lib.libreoffice.program.oosplash : the source path is not an existing regular file
    A limitation of rust itself https://github.com/rust-lang/rust/issues/79390

Compile Firefox, Clang and the Linux Kernel Build systems can vary significantly one from the other. To verify their usage of coreutils, I built these three major projects

Firefox As Firefox relies mostly on Python as a build system, it went smoothly. I didn t encounter any issue. The only unrelated issue that I noticed working on it was apt-key was broken because the script relied on a buggy option of mktemp.

Linux Kernel I identified only two issues compared to GNU Coreutils:
  • The chown command on a non-existing symlink target doesn t fail on the GNU version, the Rust one was triggering an error.
    https://github.com/uutils/coreutils/pull/1694
  • Linux kernel
    ln -fsn ../../x86/boot/bzImage ./arch/x86_64/boot/bzImage
    ln: error: Unrecognized option: 'n'

LLVM/Clang The llvm toolchain relies on Cmake. Just like for Firefox, I didn t face any issue.

Comparing with GNU coreutils using its testsuite Recently, James Robson added a new test to run the GNU testsuite on the Rust/coreutils.

# TOTAL: 611
# PASS: 144
# SKIP: 86
# XFAIL: 0
# FAIL: 342
# XPASS: 0
# ERROR: 39
compared to 546 test passing with the GNU version. Even if a bunch of errors are just different outputs, it demonstrates that there is still a long road ahead.

Next steps & contribute First, we will need more motivated contributors to work on this project. Many features remain to be implemented, optimizations to be done (e.g. decreasing the memory usage), etc.
I started to create a list of good first bugs for newcomers: https://github.com/uutils/coreutils/issues?q=is%3Aissue+is%3Aopen+label%3A%22Good+first+bug%22
I will update this list of there is some interest for this project.
Helping improve the support of the GNU coreutils testsuite would be a huge step while being a great way to learn Rust! Then, once it is in a better state, we will be able to make it a reliable alternative in Debian/Ubuntu to the GNU/Coreutils. This might be also interesting for other folks who prefer a BSD license over a GPL.

6 March 2021

Shirish Agarwal: Making life difficult

Freedom house puts India in partly free Just couple of days ago, freedom house published its 2020 rankings for all countries including India. While freedom house shared how democracy in the world has weakened, India chose to take offense about it being called partly free .
India, leader in Internet shutdowns Access Now (copyright)
The above illustration is shared by accessnow . The next big ones who have Internet shutdowns are Yemen 6 and Ethiopia 4. Such internet shutdowns have and will have sad repercussions as would share in another story as well.

Color-coding journalists A story was broken by caravan magazine yesterday and which was followed by newslaundry which shows how the Govt. is looking to just drive some narrative, does not matter whether it s true or false, it should just show that the Govt. is right and others are all wrong. As can be seen, almost all reporters barring a few have kept silent rather than refuting statements attributed to them or happenings which didn t happen. And this goes to a much larger narrative and disinformation route taken by the Govt. which doesn t have any semblance to the truth or reality as people know it. I would illustrate couple of examples below which shares that. In all my young and even adult-life I hadn t seen a Govt. this much against its own people.

Omega Seiki puts a manufacturing plant in Bangladesh Now Omega Seiki is an Indian vendor who chose or had to go to manufacture their electric vehicles in Bangladesh. Now while this is a slightly old story this was broken on social media recently. Everybody starting blaming both the vendor and saying we should break FTA (Free Trade Agreement) with Bangladesh, not knowing that despite the FTA, India has put tariff barriers between India and Bangladesh. I had to share research from Brookings to show where India has been losing. Of course, those who don t want to see, wouldn t see anything wrong in the picture.

Teen raped, asked to marry the accused when she turns 18 Now you may see the above headlines and feel it is ridiculous, but the fact is that these orders were put or given by Madras High Court couple of months back. This was then reported by both Livelaw and BarandBench respectively. Now to be truthful, this news didn t make much noise as it should have, probably as I had shared previously that the Govt. wants to lower the marriageable age to 15 or even less. And this is despite all the medical evidence on the contrary, because it assuages this Govt s masculinity. There is also the very recent case where the SC CJI asked the rapist if he is willing to marry a girl who was underage when she reaches maturity. Another one in which it seems martial rape is not a crime according to the CJI. So it seems these are the state of things in which India finds itself today. There are judges like Vrinda Grover who do question CJI but they are few and there are costs to them who ask questions. Although, as shared this news was overtaken by other news and would have remained so, if not one of the leaders of the present Govt. , a Ramesh Jarkiholi, who hails from Belagavi region of north Karnataka was caught in a sex CD scandal basically asking sexual favors for a permanent Govt. job. He had made statements after the Madras High Court case applauding the judgement given by the judge. While, due to public pressure he had to resign, but not before stating that he had everybody blue films including the Chief Minister of the State. And sad to report that six Karnataka Ministers rushed today or rather yesterday to put a petition in the civil court to restrain media from airing/printing/publishing any defamatory content against them. The court has granted a media gag against 68 media houses for the same. Sadly, the recent happening only reinforce what has been happening in Karnataka since a decade. Update 07/03/2021 Seems yesterday another 10 odd ministers rushed to get the same order. Seems different laws apply to politicians vis-a-vis others. A recent example of Rhea Chakravarthy, an actress and girlfriend of Sushant Singh who was hounded in his suicide case and many accusations made on TV but no evidence till date. From what we know as facts, Sushant committed suicide as he was not getting work due to cronyism in Bollywood. In fact, those who were behind it have white-washed themselves, deleted their tweets etc. and while the public knows, no accountability on them. In fact, there is and was so much that I wanted to share as to what has been happening to women, sadly and thankfully arre did the needful for me. They wrote an entire article which tells what the situation for women in India today is. And if you are wondering why I said, that is because when a site which was made exclusively for people to laugh and have a good time and get relief, when they start writing serious articles, you can be sure that things have gone horribly wrong

Asking Tesla to come to India and at the same time ambivalent on battery Recently, Mr. Nitin Gadkari, a prominent minister of the present Govt. invited Tesla and gave all sorts of incentives to start a manufacturing plant here in India. And while it seems that Tesla has accepted, looking at the Vodafone case, hopefully Tesla does make such contracts where if something goes wrong and they need to sue the Govt. they can do it in States or elsewhere. The way the Govt. acted in the Vodafone case had been a dampener to any MNC investments so far. Although to be fair to both Tesla and GOI, the basic models even if they are manufactured in India will go to less than 1% of the population. The cheapest Tesla Model Y which retails in the U.S. for USD 40k would be around INR 30 lakh. And this is their cheapest car to date. I do know there are rumors of the 25k but that is probably 2-3 years away as shared by Tesla China President Tom Zhu in an interview shared on YT.

https://www.youtube.com/watch?v=aH5leMWFBxI There are couple of interesting comments being made. The fact that China is going to fully open its automobile market to western companies shows how confident China feels about their own vendors. And I m not much impressed about Tesla as I am about the tiny car revolution happening in China. India, if it wanted to, could learn many lessons from China. Even the electric buses they had started in 2010 itself where people in our auto industry thought it was all a fad. Sadly, we are missing most of the technology and if and by the time Tesla starts a production line, dunno where we could get our lithium. India hasn t been as aggressive as other countries when it comes to securing raw natural resources in other countries, as some other countries have.  Even besides that, it has been tough when you have so many people who still believe that ICE vehicles (Internal Combustion Engines) are better than EV s and even if they know they choose to believe the propaganda. Couple of months ago a young UK girl who had died due to asthma, an inquest found that air pollution was a factor. The girl s name was Ella Adoo-Kissi-Debrah. This is the first case where the doctor ruled air pollution as a chief factor in a person s death. Probably first of its kind of ruling anywhere in the world. I also shared TCO studies between BEV and ICE vehicles done by people who are considering an electric vehicle but those studies seem to fall on deaf ears.

Starlink This is another of Elon Musk s ventures and would be a money spinner for sure in around the globe. While Starlink has asked TRAI for permission, I don t think they will get it. There is also Bharti Global s Oneweb which probably has a better chance of getting permissions. The reason is censorship. As shared above, India is now a leader in Internet shutdowns and do see this trend only accelerate rather than go the other way around. For people who don t remember, remember how satellite phones were made illegal even though only businessmen could afford it. And this was just 5 years ago. As shared Oneweb would have better shot as they would accept all Government directives without a second thought. Unless Starlink gives a binding to the Govt. to be a willing partner when it wants to have internet shutdowns, it will not work. Now how Elon approaches that is to be seen and known. FWIW, you can t access Starlink webpage on BSNL broadband. My broadband provider gives at the most 300 kbps and sometimes, at late nights or early mornings, around 500 kbps.

Farmer Protests
Lastly, farmer protests have entered 100 days. In the interim, Vivek Kaul, an economist took stock of the Bihar APMC to see if things have really worked as the Govt. supporters were telling. The investigation and the results didn t inspire the confidence as the Govt. said. The sad part is though, that nowadays nobody, at least those in power as well as those who are supporters are keen to read, understand and even argue otherwise. They are all happy with whatsapp knowledge. Till date 200+ people have died in the farmer protests. All mainstream media houses have stopped talking about farmers in the hope that they will disappear. At the end of the day the Govt. wants that the corporates should win at whatever the cost.

Michael Stapelberg: Debian Code Search: OpenAPI now available

Debian Code Search now offers an OpenAPI-based API! Various developers have created ad-hoc client libraries based on how the web interface works. The goal of offering an OpenAPI-based API is to provide developers with automatically generated client libraries for a large number of programming languages, that target a stable interface independent of the web interface s implementation details.

Getting started
  1. Visit https://codesearch.debian.net/apikeys/ to download your personal API key. Login via Debian s GitLab instance salsa.debian.org; register there if you have no account yet.
  2. Find the Debian Code Search client library for your programming language. If none exists yet, auto-generate a client library on editor.swagger.io: click Generate Client .
  3. Search all code in Debian from your own analysis tool, migration tracking dashboard, etc.

curl example
curl \
  -H "x-dcs-apikey: $(cat dcs-apikey-stapelberg.txt)" \
  -X GET \
  "https://codesearch.debian.net/api/v1/search?query=i3Font&match_mode=regexp" 

Web browser example You can try out the API in your web browser in the OpenAPI documentation.

Code example (Go) Here s an example program that demonstrates how to set up an auto-generated Go client for the Debian Code Search OpenAPI, run a query, and aggregate the results:
func burndown() error  
	cfg := openapiclient.NewConfiguration()
	cfg.AddDefaultHeader("x-dcs-apikey", apiKey)
	client := openapiclient.NewAPIClient(cfg)
	ctx := context.Background()
	// Search through the full Debian Code Search corpus, blocking until all
	// results are available:
	results, _, err := client.SearchApi.Search(ctx, "fmt.Sprint(err)", &openapiclient.SearchApiSearchOpts 
		// Literal searches are faster and do not require escaping special
		// characters, regular expression searches are more powerful.
		MatchMode: optional.NewString("literal"),
	 )
	if err != nil  
		return err
	 
	// Print to stdout a CSV file with the path and number of occurrences:
	wr := csv.NewWriter(os.Stdout)
	header := []string "path", "number of occurrences" 
	if err := wr.Write(header); err != nil  
		return err
	 
	occurrences := make(map[string]int)
	for _, result := range results  
		occurrences[result.Path]++
	 
	for _, result := range results  
		o, ok := occurrences[result.Path]
		if !ok  
			continue
		 
		// Print one CSV record per path:
		delete(occurrences, result.Path)
		record := []string result.Path, strconv.Itoa(o) 
		if err := wr.Write(record); err != nil  
			return err
		 
	 
	wr.Flush()
	return wr.Error()
 
The full example can be found under burndown.go.

Feedback? File a GitHub issue on github.com/Debian/dcs please!

Migration status I m aware of the following third-party projects using Debian Code Search:
Tool Migration status
Debian Code Search CLI tool Updated to OpenAPI
identify-incomplete-xs-go-import-path Update pending
gnome-codesearch makes no API queries
If you find any others, please point them to this post in case they are not using Debian Code Search s OpenAPI yet.

5 March 2021

Sergio Durigan Junior: A debuginfod service for Debian

Hi there. Long time no write! On Tuesday, February 23, 2021, I made an announcement at debian-devel-announce about a new service that I configured for Debian: a debuginfod server. This post serves two purposes: pay the promise I made to Jonathan Carter that I would write a blog post about the service, and go into a bit more detail about it. What's debuginfod? From the announcement above:
debuginfod is a new-ish project whose purpose is to serve
ELF/DWARF/source-code information over HTTP.  It is developed under the
elfutils umbrella.  You can find more information about it here:
  https://sourceware.org/elfutils/Debuginfod.html
In a nutshell, by using a debuginfod service you will not need to
install debuginfo (a.k.a. dbgsym) files anymore; the symbols will be
served to GDB (or any other debuginfo consumer that supports debuginfod)
over the network.  Ultimately, this makes the debugging experience much
smoother (I myself never remember the full URL of our debuginfo
repository when I need it).
Perhaps not everybody knows this, but until last year I was a Debugger Engineer (a.k.a. GDB hacker) at Red Hat. I was not involved with the creation of debuginfod directly, but I witnessed discussions about "having way to serve debug symbols over the internet" multiple times during my tenure at the company. So this is not a new idea, and it's not even the first implementation, but it's the first time that some engineers actually got their hands dirty enough to have something concrete to show. The idea to set up a debuginfod server for Debian started to brew after 2019's GNU Tools Cauldron, but as usual several things happened in $LIFE (including a global pandemic and leaving Red Hat and starting a completely different job at Canonical) which had the effect of shuffling my TODO list "a little". Benefits for Debian Debian unfortunately is lagging behind when it comes to offer its users a good debugging experience. Before the advent of our debuginfod server, if you wanted to debug a package in Debian you would need to:
  1. Add the debian-debug apt repository to your /etc/apt/sources.list.
  2. Install the dbgsym package that contains the debug symbols for the package you are debugging. Note that the version of the dbgsym package needs to be exactly the same as the version of the package you want to debug.
  3. Figure out which shared libraries your package uses and install the dbgsym packages for all of them. Arguably, this step is optional but recommended if you would like to perform a more in-depth debugging.
  4. Download the package source, possibly using apt source or some equivalent command.
  5. Open GDB, and make sure you adjust the source paths properly (more below). This can be non-trivial.
  6. Finally, debug the program.
Now, with the new service, you will be able to start from step 4, without having to mess with sources.list, dbgsym packages and version mismatches. The package source It is important to mention an existing (but perhaps not well-known) limitation of our debugging experience in Debian: the need to manually download the source packages and adjust GDB to properly find them (see step 4 above). debuginfod is able to serve source code as well, but our Debian instance is not doing that at the moment. Debian does not provide a patched source tree that is ready to be consumed by GDB or debuginfod (for a good example of a distribution that does this, see Fedora's debugsource packages). Let me show you an example of debugging GDB itself (using debuginfod) on Debian:
$ HOME=/tmp DEBUGINFOD_URLS=https://debuginfod.debian.net gdb -q gdb
Reading symbols from gdb...
Downloading separate debug info for /tmp/gdb...
Reading symbols from /tmp/.cache/debuginfod_client/02046bac4352940d19d9164bab73b2f5cefc8c73/debuginfo...
(gdb) start
Temporary breakpoint 1 at 0xd18e0: file /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c, line 28.
Starting program: /usr/bin/gdb 
Downloading separate debug info for /lib/x86_64-linux-gnu/libreadline.so.8...
Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1...
Downloading separate debug info for /lib/x86_64-linux-gnu/libncursesw.so.6...
Downloading separate debug info for /lib/x86_64-linux-gnu/libtinfo.so.6...
Downloading separate debug info for /tmp/.cache/debuginfod_client/d6920dbdd057f44edaf4c1fbce191b5854dfd9e6/debuginfo...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Downloading separate debug info for /lib/x86_64-linux-gnu/libexpat.so.1...
Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbabeltrace.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbabeltrace-ctf.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libipt.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libmpfr.so.6...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libsource-highlight.so.4...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libxxhash.so.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libdebuginfod.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libstdc++.so.6...
Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0...
Downloading separate debug info for /tmp/.cache/debuginfod_client/dbfea245d26065975b4084f4e9cd2d83c65973ee/debuginfo...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libdw.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libelf.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libuuid.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgmp.so.10...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.74.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4...
Downloading separate debug info for /lib/x86_64-linux-gnu/libbz2.so.1.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libicui18n.so.67...
Downloading separate debug info for /tmp/.cache/debuginfod_client/acaa831dbbc8aa70bb2131134e0c83206a0701f9/debuginfo...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libicuuc.so.67...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libnghttp2.so.14...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libidn2.so.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/librtmp.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libssh2.so.1...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libpsl.so.5...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libnettle.so.8...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgnutls.so.30...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbrotlidec.so.1...
Downloading separate debug info for /tmp/.cache/debuginfod_client/39739740c2f8a033de95c1c0b1eb8be445610b31/debuginfo...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libunistring.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libhogweed.so.6...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libgcrypt.so.20...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libp11-kit.so.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libtasn1.so.6...
Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libsasl2.so.2...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libbrotlicommon.so.1...
Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0...
Downloading separate debug info for /usr/lib/x86_64-linux-gnu/libffi.so.7...
Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1...
Temporary breakpoint 1, main (argc=1, argv=0x7fffffffebf8) at /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c:28
28      /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c: Directory not empty.
(gdb) list
23      in /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c
(gdb) 
(See all those Downloading separate debug info for... lines? Nice!) As you can see, when we try to list the contents of the file we're in, nothing shows up. This happens because GDB doesn't know where the file is. So you have to tell it. In this case, it's relatively easy: you see that the GDB package's build directory is /build/gdb-Nav6Es/gdb-10.1/. When you apt source gdb, you will end up with a directory called $PWD/gdb-10.1/ containing the full source of the package. Notice that the last directory's name in both paths is the same, so in this case we can use GDB's set substitute-path command do the job for us (in this example $PWD is /tmp/):
$ HOME=/tmp DEBUGINFOD_URLS=https://debuginfod.debian.net gdb -q gdb
Reading symbols from gdb...
Reading symbols from /tmp/.cache/debuginfod_client/02046bac4352940d19d9164bab73b2f5cefc8c73/debuginfo...
(gdb) set substitute-path /build/gdb-Nav6Es/ /tmp/
(gdb) start
Temporary breakpoint 1 at 0xd18e0: file /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c, line 28.
Starting program: /usr/bin/gdb 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Temporary breakpoint 1, main (argc=1, argv=0x7fffffffebf8) at /build/gdb-Nav6Es/gdb-10.1/gdb/gdb.c:28
warning: Source file is more recent than executable.
28        memset (&args, 0, sizeof args);
(gdb) list
23      int
24      main (int argc, char **argv)
25       
26        struct captured_main_args args;
27
28        memset (&args, 0, sizeof args);
29        args.argc = argc;
30        args.argv = argv;
31        args.interpreter_p = INTERP_CONSOLE;
32        return gdb_main (&args);
(gdb)
Much better, huh? The problem is that this process is manual, and changes depending on how the package you're debugging was built. What can we do to improve this? What I personally would like to see is something similar to what the Fedora project already does: create a new debug package which will contain the full, patched source package. This would mean changing our building infrastructure and possibly other somewhat complex things. Using the service (by default) At the time of this writing, I am working on an elfutils Merge Request whose purpose is to implement a debconf question to ask the user whether she wants to use our service by default. If you would like to start using the service right now, all you have to do is set the following environment variable in your shell:
DEBUGINFOD_URLS="https://debuginfod.debian.net"
More information You can find more information about our debuginfod service here. Try to keep an eye on the page as it's being constantly updated. If you'd like to get in touch with me, my email is my domain at debian dot org. I sincerely believe that this service is a step in the right direction, and hope that it can be useful to you :-).

2 March 2021

Russ Allbery: Review: The Lion, the Witch and the Wardrobe

Review: The Lion, the Witch and the Wardrobe, by C.S. Lewis
Illustrator: Pauline Baynes
Series: Chronicles of Narnia #1
Publisher: Collier Books
Copyright: 1950
Printing: 1978
ISBN: 0-02-044220-3
Format: Mass market
Pages: 186
Although it's been more than 20 years since I last read it, I believe I have read The Lion, the Witch and the Wardrobe more times than any other book. The count is certainly in double digits. As you might guess, I also have strong opinions about it, some of which are unorthodox, and I've been threatening to write this review for years. It seemed a fitting choice for my 1000th review. There is quite a lot that can and has been said about this book and this series, and this review is already going to be much too long, so I'm only going to say a fraction of it. I'm going to focus on my personal reactions as someone raised a white evangelical Christian but no longer part of that faith, and the role this book played in my religion. I'm not going to talk much about some of its flaws, particularly Lewis's treatment of race and gender. This is not because I don't agree they're there, but only that I don't have much to say that isn't covered far better in other places. Unlike my other reviews, this one will contain major spoilers. If you have managed to remain unspoiled for a 70-year-old novel that spawned multiple movies and became part of the shared culture of evangelical Christianity, and want to stay that way, I'll warn you in ALL CAPS when it's time to go. But first, a few non-spoiler notes. First, reading order. Most modern publications of The Chronicles of Narnia will list The Magician's Nephew as the first book. This follows internal chronological order and is at C.S. Lewis's request. However, I think Lewis was wrong. You should read this series in original publication order, starting with The Lion, the Witch and the Wardrobe (which I'm going to abbreviate as TLtWatW like everyone else who writes about it). I will caveat this by saying that I have a bias towards reading books in the order an author wrote them because I like seeing the development of the author's view of their work, and I love books that jump back in time and fill in background, so your experience may vary. But the problem I see with the revised publication order is that The Magician's Nephew explains the origins of Narnia and, thus, many of the odd mysteries of TLtWatW that Lewis intended to be mysterious. Reading it first damages both books, like watching a slow-motion how-to video for a magic trick before ever seeing it performed. The reader is not primed to care about the things The Magician's Nephew is explaining, which makes it less interesting. And the bits of unexpected magic and mystery in TLtWatW that give it so much charm (and which it needs, given the thinness of the plot) are already explained away and lose appeal because of it. I have read this series repeatedly in both internal chronological order and in original publication order. I have even read it in strict chronological order, wherein one pauses halfway through the last chapter of TLtWatW to read The Horse and His Boy before returning. I think original publication order is the best. (The Horse and His Boy is a side story and it doesn't matter that much where you read it as long as you read it after TLtWatW. For this re-read, I will follow original publication order and read it fifth.) Second, allegory. The common understanding of TLtWatW is that it's a Christian allegory for children, often provoking irritated reactions from readers who enjoyed the story on its own terms and later discovered all of the religion beneath it. I think this view partly misunderstands how Lewis thought about the world and there is a more interesting way of looking at the book. I'm not as dogmatic about this as I used to be; if you want to read it as an allegory, there are plenty of carefully crafted parallels to the gospels to support that reading. But here's my pitch for a different reading. To C.S. Lewis, the redemption of the world through the death of Jesus Christ is as foundational a part of reality as gravity. He spent much of his life writing about religion and Christianity in both fiction and non-fiction, and this was the sort of thing he constantly thought about. If somewhere there is another group of sentient creatures, Lewis's theology says that they must fit into that narrative in some way. Either they would have to be unfallen and thus not need redemption (roughly the position taken by The Space Trilogy), or they would need their own version of redemption. So yes, there are close parallels in Narnia to events of the Christian Bible, but I think they can be read as speculating how Christian salvation would play out in a separate creation with talking animals, rather than an attempt to disguise Christianity in an allegory for children. It's a subtle difference, but I think Narnia more an answer to "how would Christ appear in this fantasy world?" than to "how do I get children interested in the themes of Christianity?", although certainly both are in play. Put more bluntly, where other people see allegory, I see the further adventures of Jesus Christ as an anthropomorphic lion, which in my opinion is an altogether more delightful way to read the books. So much for the preamble; on to the book. The Pevensie kids, Peter, Susan, Edmund, and Lucy, have been evacuated to a huge old house in the country due to the air raids (setting this book during World War II, something that is passed over with barely a mention and not a hint of trauma in a way that a modern book would never do). While exploring this house, which despite the scant description is still stuck in my mind as the canonical huge country home, Lucy steps into a wardrobe because she wants to feel the fur of the coats. Much to her surprise, the wardrobe appears not to have a back, and she finds herself eventually stepping into a snow-covered pine forest where she meets a Fawn named Mr. Tumnus by an unlikely lamp-post. MAJOR SPOILERS BELOW, so if you don't want to see those, here's your cue to stop reading. Two things surprised me when re-reading TLtWatW. The first, which I remember surprising me every time I read it, is how far into this (very short) book one has to go before the plot kicks into gear. It's not until "What Happened After Dinner" more than a third in that we learn much of substance about Narnia, and not until "The Spell Begins to Break" halfway through the book that things start to happen. The early chapters are concerned primarily with the unreliability of the wardrobe portal, with a couple of early and brief excursions by Edmund and Lucy, and with Edmund being absolutely awful to Lucy. The second thing that surprised me is how little of what happens is driven by the kids. The second half of TLtWatW is about the fight between Aslan and the White Witch, but this fight was not set off by the children and their decisions don't shape it in any significant way. They're primarily bystanders; the few times they take action, it's either off-camera or they're told explicitly what to do. The arguable exception is Edmund, who provides the justification for the final conflict, but he functions more as plot device than as a character with much agency. When that is combined with how much of the story is also on rails via its need to recapitulate part of the gospels (more on that in a moment), it makes the plot feel astonishingly thin and simple. Edmund is the one protagonist who gets to make some decisions, all of them bad. As a kid, I hated reading these parts because Edmund is an ass, the White Witch is obviously evil, and everyone knows not to eat the food. Re-reading now, I have more appreciation for how Lewis shows Edmund's slide into treachery. He starts teasing Lucy because he thinks it's funny (even though it's not), has a moment when he realizes he was wrong and almost apologizes, but then decides to blame his discomfort on the victim. From that point, he is caught, with some help from the White Witch's magic, in a spiral of doubling down on his previous cruelty and then feeling unfairly attacked. Breaking the cycle is beyond him because it would require admitting just how badly he behaved and, worse, that he was wrong and his little sister was right. He instead tries to justify himself by spreading poisonous bits of doubt, and looks for reasons to believe the friends of the other children are untrustworthy. It's simplistic, to be sure, but it's such a good model of how people slide into believing conspiracy theories and joining hate groups. The Republican Party is currently drowning in Edmunds. That said, Lewis does one disturbing thing with Edmund that leaped out on re-reading. Everyone in this book has a reaction when Aslan's name is mentioned. For the other three kids, that reaction is awe or delight. For Edmund, it's mysterious horror. I know where Lewis is getting this from, but this is a nasty theological trap. One of the problems that religion should confront directly is criticism that questions the moral foundations of that religion. If one postulates that those who have thrown in with some version of the Devil have an instinctual revulsion for God, it's a free intellectual dodge. Valid moral criticism can be hand-waved away as Edmund's horrified reaction to Aslan: a sign of Edmund's guilt, rather than a possible flaw to consider seriously. It's also, needless to say, not the effect you would expect from a god who wants universal salvation! But this is only an odd side note, and once Edmund is rescued it's never mentioned again. This brings us to Aslan himself, the Great Lion, and to the heart of why I think this book and series are so popular. In reinterpreting Christianity for the world of Narnia, Lewis created a far more satisfying and relatable god than Jesus Christ, particularly for kids. I'm not sure I can describe, for someone who didn't grow up in that faith, how central the idea of a personal relationship with Jesus is to evangelical Christianity. It's more than a theological principle; it's the standard by which one's faith is judged. And it is very difficult for a kid to mentally bootstrap themselves into a feeling of a personal relationship with a radical preacher from 2000 years ago who spoke in gnomic parables about subtle points of adult theology. It's hard enough for adults with theological training to understand what that phrase is intended to mean. For kids, you may as well tell them they have a personal relationship with Aristotle. But a giant, awe-inspiring lion with understanding eyes, a roar like thunder, and a warm mane that you can bury your fingers into? A lion who sacrifices himself for your brother, who can be comforted and who comforts you in turn, and who makes a glorious surprise return? That's the kind of god with which one can imagine having a personal relationship. Aslan felt physical and embodied and present in the imagination in a way that Jesus never did. I am certain I was not the only Christian kid for whom Aslan was much more viscerally real than Jesus, and who had a tendency to mentally substitute Aslan for Jesus in most thoughts about religion. I am getting ahead of myself a bit because this is a review of TLtWatW and not of the whole series, and Aslan in this book is still a partly unformed idea. He's much more mundanely present here than he is later, more of a field general than a god, and there are some bits that are just wrong (like him clapping his paws together). But the scenes with Susan and Lucy, the night at the Stone Table and the rescue of the statues afterwards, remain my absolute favorite parts of this book and some of the best bits of the whole series. They strike just the right balance of sadness, awe, despair, and delight. The image of a lion also lets Lewis show joy in a relatable way. Aslan plays, he runs, he wrestles with the kids, he thrills in the victory over evil just as much as Susan and Lucy do, and he is clearly having the time of his life turning people back to flesh from stone. The combination of translation, different conventions, and historical distance means the Bible has none of this for the modern reader, and while people have tried to layer it on with Bible stories for kids, none of them (and I read a lot of them) capture anything close to the sheer joy of this story. The trade-off Lewis makes for that immediacy is that Aslan is a wonderful god, but TLtWatW has very little religion. Lewis can have his characters interact with Aslan directly, which reduces the need for abstract theology and difficult questions of how to know God's will. But even when theology is unavoidable, this book doesn't ask for the type of belief that Christianity demands. For example, there is a crucifixion parallel, because in Lewis's world view there would have to be. That means Lewis has to deal with substitutionary atonement (the belief that Christ died for the sins of the world), which is one of the hardest parts of Christianity to justify. How he does this is fascinating. The Narnian equivalent is the Deep Magic, which says that the lives of all traitors belong to the White Witch. If she is ever denied a life, Narnia will be destroyed by fire and water. The Witch demands Edmund's life, which sets up Aslan to volunteer to be sacrificed in Edmund's place. This triggers the Deeper Magic that she did not know about, freeing Narnia from her power. You may have noticed the card that Lewis is palming, and to give him credit, so do the kids, leading to this exchange when the White Witch is still demanding Edmund:
"Oh, Aslan!" whispered Susan in the Lion's ear, "can't we I mean, you won't, will you? Can't we do something about the Deep Magic? Isn't there something you can work against it?" "Work against the Emperor's magic?" said Aslan, turning to her with something like a frown on his face. And nobody ever made that suggestion to him again.
The problem with substitutionary atonement is why would a supposedly benevolent god create such a morally abhorrent rule in the first place? And Lewis totally punts. Susan is simply not allowed to ask the question. Lewis does try to tackle this problem elsewhere in his apologetics for adults (without, in my opinion, much success). But here it's just a part of the laws of this universe, which all of the characters, including Aslan, have to work within. That leads to another interesting point of theology, which is that if you didn't already know about the Christian doctrine of the trinity, you would never guess it from this book. The Emperor-Beyond-the-Sea and Aslan are clearly separate characters, with Aslan below the Emperor in the pantheon. This makes rules like the above work out more smoothly than they do in Christianity because Aslan is bound by the Emperor's rules and the Emperor is inscrutable and not present in the story. (The Holy Spirit is Deity Not-appearing-in-this-book, but to be fair to Lewis, that's largely true of the Bible as well.) What all this means is that Aslan's death is presented straightforwardly as a magic spell. It works because Aslan has the deepest understanding of the fixed laws of the Emperor's magic, and it looks nothing like what we normally think of as religion. Faith is not that important in this book because Aslan is physically present, so it doesn't require any faith for the children to believe he exists. (The Beavers, who believed in him from prophecy without having seen him, are another matter, but this book never talks about that.) The structure of religion is therefore remarkably absent despite the story's Christian parallels. All that's expected of the kids is the normal moral virtues of loyalty and courage and opposition to cruelty. I have read this book so many times that I've scrutinized every word, so I have to resist the temptation to dig into every nook and cranny: the beautiful description of spring, the weird insertion of Lilith as Adam's first wife, how the controversial appearance of Santa Claus in this book reveals Lewis's love of Platonic ideals... the list is endless, and the review is already much longer than normal. But I never get to talk about book endings in reviews, so one more indulgence. The best thing that can be said about the ending of TLtWatW is that it is partly redeemed by the start of Prince Caspian. Other than that, the last chapter of this book has always been one of my least favorite parts of The Chronicles of Narnia. For those who haven't read it (and who by this point clearly don't mind spoilers), the four kids are immediately and improbably crowned Kings and Queens of Narnia. Apparently, to answer the Professor from earlier in the book, ruling magical kingdoms is what they were teaching in those schools? They then spend years in Narnia, never apparently giving a second thought to their parents (you know, the ones who are caught up in World War II, which prompted the evacuation of the kids to the country in the first place). This, for some reason, leaves them talking like medieval literature, which may be moderately funny if you read their dialogue in silly voices to a five-year-old and is otherwise kind of tedious. Finally, in a hunt for the white stag, they stumble across the wardrobe and tumble back into their own world, where they are children again and not a moment has passed. I will give Lewis credit for not doing a full reset and having the kids not remember anything, which is possibly my least favorite trope in fiction. But this is almost as bad. If the kids returned immediately, that would make sense. If they stayed in Narnia until they died, that arguably would also make sense (their poor parents!). But growing up in Narnia and then returning as if nothing happened doesn't work. Do they remember all of their skills? How do you readjust to going to school after you've lived a life as a medieval Queen? Do they remember any of their friends after fifteen years in Narnia? Argh. It's a very "adventures are over, now time for bed" sort of ending, although the next book does try to patch some of this up. As a single book taken on its own terms, TLtWatW is weirdly slight, disjointed, and hits almost none of the beats that one would expect from a children's novel. What saves it is a sense of delight and joy that suffuses the descriptions of Narnia, even when locked in endless winter, and Aslan. The plot is full of holes, the role of the children in that plot makes no sense, and Santa Claus literally shows up in the middle of the story to hand out plot devices and make an incredibly sexist statement about war. And yet, I memorized every gift the children received as a kid, I can still feel the coziness of the Beaver's home while Mr. Beaver is explaining prophecy, and the night at the Stone Table remains ten times more emotionally effective for me than the description of the analogous event in the Bible. And, of course, there's Aslan.
"Safe?" said Mr. Beaver. "Don't you hear what Mrs. Beaver tells you? Who said anything about safe? 'Course he isn't safe. But he's good. He's the king, I tell you."
Aslan is not a tame lion, to use the phrase that echos through this series. That, I think, is the key to the god that I find the most memorable in all of fantasy literature, even in this awkward, flawed, and decidedly strange introduction. Followed by Prince Caspian, in which the children return to a much-changed Narnia. Lewis has gotten most of the obligatory cosmological beats out of the way in this book, so subsequent books can tell more conventional stories. Rating: 7 out of 10

1 March 2021

Alo s Micard: Open sourcing my blog

I have received a lot of positives feedback for my blog lately, and I do really appreciate it and try to integrate the suggestions to update my posts and make things better. With the aim of continous improvement of this blog, I have decided (a bit late?) to open source it. The source code is now available on Github! Feel free to open a PR if you want to fix a typo or if you see a mistake in one examples while reading an article.

28 February 2021

Dima Kogan: mrcal: principled camera calibrations

This is a big deal. In my day job I work with images captured by cameras, using those images to infer something about the geometry of the scene being observed. Naturally, to get good results you need to have a good estimate of the behavior of the lens (the "intrinsics"), and of the relative geometry of the cameras (the "extrinsics"; if there's more than one camera). The usual way to do this is to perform a "calibration" procedure to compute the intrinsics and extrinsics, and then to use the resulting "camera model" to process the subsequent images. Wikipedia has an article. And from experience, the most common current toolkit to do this appears to be OpenCV. People have been doing this for a while, but for whatever reason the existing tools all suck. They make basic questions like "how much data should I gather for a calibration?" and "how good is this calibration I just computed?" and "how different are these two models?" unanswerable. This is clearly seen from the links above. The wikipedia article talks about fitting a pinhole model to lenses, even though no real lenses follow this model (telephoto lenses do somewhat; wider lenses don't at all). And the OpenCV tutorial cheerfully says that
Re-projection error gives a good estimation of just how exact the found
parameters are. The closer the re-projection error is to zero, the more accurate
the parameters we found are.
This statement is trivially proven false: throw away most of your calibration data, and your reprojection error becomes very low. But we can all agree that a calibration computed from less data is actually worse. Right? All the various assumptions and hacks in the existing tooling are fine as long as you don't need a whole lot of accuracy out of your results. I need a lot of accuracy, however, so all the existing tools don't work for my applications. So I built a new set of tools, and have been using them with great delight. I just got the blessing to do a public release, so I'm announcing it here. The tools are mrcal does a whole lot to produce calibrations that are as good as possible, and it will tell you just how good they are, and it includes visualization capabilities for extensive user feedback. An overview of the capabilities of the toolkit (with lots of pretty pictures!) is at the tour of mrcal. There's a lot of documentation and examples, but up to now I have been the primary user of the tools. So I expect this to be somewhat rough when others look at it. Bug reports and patches are welcome. mrcal is an excellent base, but it's nowhere near "done". The documentation has some notes about the planned features and improvements, and I'm always reachable by email. Let me know if you try it out!

27 February 2021

Dima Kogan: horizonator: terrain renderer based on SRTM DEMs

Check this out:
example-interactive.png
I just resurrected and cleaned up an old tool I had lying around. It's now nice and usable by others. This tool loads terrain data, and renders it from the ground, simulating what a human or a camera would see. This is useful for armchair exploring or for identifying peaks. This was relatively novel when I wrote it >10 years ago, but there are a number of similar tools in existence now. This implementation is still useful in that it's freely licensed and contains APIs, so fancier processing can be performed on its output. Sources and (barely-complete-enough) documentation live here: https://github.com/dkogan/horizonator

26 February 2021

Ritesh Raj Sarraf: Wayland KDE X11

KDE Impressions These days, I often hear a lot about Wayland. And how much of effort is being put into it; not just by the Embedded world but also the usual Desktop systems, namely KDE and GNOME. In recent past, I switched back to KDE and have been (very) happy about the switch. Even though the KDE 4 (and initial KDE 5) debacle had burnt many, coming back to a usable KDE desktop is always a delight. It makes me feel home with the elegance, while at the same time the flexibility, it provides. It feels so nice to draft this blog article from Kwrite + VI Input Mode Thanks to the great work of the Debian KDE Team, but Norbert Preining in particular, who has helped bring very up-to-date KDE packages into Debian. Right now, I m on a Plamsa 5.21.1 desktop, which is recent by all standards.

Wayland Almost all the places in the Linux world these days are busy with integrating Wayland as the primary display service. Not sure what the current status on the GNOME side is but I definitely keep trying KDE + Wayland with every release. I keep trying with every release because it still is not prime for daily use. And it makes me get back to X11, no matter how dated some may call. Fact is, X11 still shines to me as an end-user. Glitches with Wayland still are (Based on this week s test on Plasma 5.21.1):
  • Horrible performance compared to X11
  • Very crashy, especially when hotplugging secondary display. Plasma would just crash. X11 is very resilient to such things, part of the reason I can think is the age of the codebase.
  • Many many applications still need to be fixed for Wayland. Or Wayland needs to accomodate them in some way. XWayland does not really feel like the answer.
And while KDE keeps insisting users to switch to Wayland, as that s where all the new enhancements and fixes are put in, someone like me still needs to stick to X11 for the time being. So to get my shiny new LG 27" 4K Monitor (3840x2160 60.00*+) to work without too much glitch, I had to live with an alias:
$ alias   grep xrandr
alias rrs_xrandr_lg='xrandr --output DP-1 --mode 3840x2160 --scale .75x.75'
18:31                    

Plasma 5.21 On the brighter side, the Plasma 5.21.1 release brings some nice enhancements in other areas.
  • I m now able to make use of tighter integration with systemd/cgroups, with better organization and management of processes overall.
  • The new Plasma theme, Breeze Twilight, is a good blend of Light + Dark.
I also appreciate the work put in by Michail Vourlakos. The KDE project is lucky to have a developer/designer like him. His vision and work into the KDE desktop is well beyond a writing by me.
$ usystemctl status plasma-plasmashell.service 
  plasma-plasmashell.service - KDE Plasma Workspace
     Loaded: loaded (/usr/lib/systemd/user/plasma-plasmashell.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-02-26 18:34:23 IST; 13s ago
   Main PID: 501806 (plasmashell)
      Tasks: 21 (limit: 18821)
     Memory: 759.8M
        CPU: 13.706s
     CGroup: /user.slice/user-1000.slice/user@1000.service/session.slice/plasma-plasmashell.service
              501806 /usr/bin/plasmashell --no-respawn
             
Feb 26 18:35:00 priyasi plasmashell[501806]: qml: recreating buttons
Feb 26 18:35:21 priyasi plasmashell[501806]: qml: recreating buttons
Feb 26 18:35:49 priyasi plasmashell[501806]: qml: recreating buttons
Feb 26 18:35:57 priyasi plasmashell[501806]: qml: recreating buttons
18:36                  

OBS - Open Build Service I should also thank the OpenSUSE folks for the OBS work. It has enabled the close equivalent (or better, in my experience) of PPAs for Debian. And that is what has enabled developers like Norbert to easily and quickly be able to deliver the entire KDE suite.

OBS - Some detail Christian asked for some more details on the OBS side of things, of my view. I m updating this article with it because the comment system may not always be reliable and I hate losing content. Having been using OBS myself, and also others in the Debian community who are making use of it, I surely think we as project should consider making use of OBS. Given that OBS is Free Software, it is a perfect fit for Debian. Gitlab is another example of what we ve made available in Debian. OBS is divided into multiple parts
  • OBS Server
  • OBS DoD service
  • OBS Publisher
  • OBS Workers
  • OBS Warden
  • OBS Rep Server
For every Debian release I care about, I add an OBS project per release. So I have OBS projects for: Sid, Bullseye, Buster, Jessie. Now, say you have a package, foo . You prep your package and enable all the releases that you want to build the package for. So the same package gets built, in separate clean environments, for every release I mentioned as an example above. You don t have to manually trigger the build for every release/every architcture. You should add the release (as projects) in OBS, set their supported architectures, and then add those enabled release projets as bits to your package. Every build involves:
  • Creating a new chroot for each build
  • Building the package
Builds can be scattered across multiple hosts, known as workers in OBS terminology. Your workers are independent machine entities, supporting different architectures. The machines can be Bare-Metal ones, VMs, even containers. So this allows for very nice scale-in and scale-out. There may be auto-scaling too but that is something worth investigating. Think of things like cross architecture builds. Let s assume the cloud vendors decide to donate resources to the Debian project. We could enable OBS worker instances on the respective clouds (different architectures) and plug them into the master OBS instance that Debian hosts. Fully distributed. Similarly, big hardware vendors willing to donate compute resources could house them in their premises and Debian could just easily establish a connection to them. All of this just a TCP connection away. So when I look at the features of OBS, from the point of view of Debian, I like it more. Extensibility won t be an issue. Supporting a new Debian release would just be a matter of bootstrapping the Debian release as a project in OBS, and then all done. The single effort of setting of the target release project is a one time job, and then all can leverage it. The PPA was a long craved feature missing in Debian, in my opinion. OBS allows to not just fulfil that gap but also extend it in a very easy way. Andrew Lee had put in a nice video presentation about the same @ Debconf 20

24 February 2021

Dirk Eddelbuettel: RcppSimdJson 0.1.4 on CRAN: Small Bugfix

A quick note to say that we finalized a bugfix release 0.1.5 of RcppSimdJson yesterday which got onto CRAN earlier today. RcppSimdJson wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire and collaborators. Via very clever algorithmic engineering to obtain largely branch-free code, coupled with modern C++ and newer compiler instructions, it results in parsing gigabytes of JSON parsed per second which is quite mindboggling. The best-case performance is faster than CPU speed as use of parallel SIMD instructions and careful branch avoidance can lead to less than one cpu cycle per byte parsed; see the video of the talk by Daniel Lemire at QCon (also voted best talk). This version brings a small fix to Windows where temporary filenames constructed from URLs (as for example constructed by our td package) need an extra sanization for a possible ? character. We will be addressing that properly in due course but for now a simple gsub() will do. The NEWS entry follows.

Changes in version 0.1.5 (2021-02-23)
  • Temporary filenames from request URLs need an extra path sanitization on Windows as seen with package td.
  • A few #nocov tags were added to the code.

Courtesy of my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

22 February 2021

Dima Kogan: feedgnuplot: labelled bar charts and a guide

I just released feedgnuplot 1.57, which includes two new pieces that I've long thought about adding:

Labelled bar charts
I've thought about adding these for a while, but had no specific need for them. Finally, somebody asked for it, and I wrote the code. Now that I can, I will probably use these all the time. The new capability can override the usual numerical tic labels on the x axis, and instead use text from a column in the data stream. The most obvious use case is labelled bar graphs:
echo "# label value
      aaa     2
      bbb     3
      ccc     5
      ddd     2"   \
feedgnuplot --vnl \
            --xticlabels \
            --with 'boxes fill solid border lt -1' \
            --ymin 0 --unset grid
xticlabels-basic.svg
But the usage is completely generic. All --xticlabels does, is to accept a data column as labels for the x-axis tics. Everything else that's supported by feedgnuplot and gnuplot works as before. For instance, I can give a domain, and use a style that takes y values and a color:
echo "# x label y color
        5 aaa   2 1
        6 bbb   3 2
       10 ccc   5 4
       11 ddd   2 1"   \
feedgnuplot --vnl --domain \
            --xticlabels \
            --tuplesizeall 3 \
            --with 'points pt 7 ps 2 palette' \
            --xmin 4 --xmax 12 \
            --ymin 0 --ymax 6 \
            --unset grid
xticlabels-points-palette.svg
And we can use gnuplot's support for clustered histograms:
echo "# x label a b
        5 aaa   2 1
        6 bbb   3 2
       10 ccc   5 4
       11 ddd   2 1"   \
vnl-filter -p label,a,b   \
feedgnuplot --vnl \
            --xticlabels \
            --set 'style data histogram' \
            --set 'style histogram cluster gap 2' \
            --set 'style fill solid border lt -1' \
            --autolegend \
            --ymin 0 --unset grid
xticlabels-clustered.svg
Or we can stack the bars on top of one another:
echo "# x label a b
        5 aaa   2 1
        6 bbb   3 2
       10 ccc   5 4
       11 ddd   2 1"   \
vnl-filter -p label,a,b   \
feedgnuplot --vnl \
            --xticlabels \
            --set 'style data histogram' \
            --set 'style histogram rowstacked' \
            --set 'boxwidth 0.8' \
            --set 'style fill solid border lt -1' \
            --autolegend \
            --ymin 0 --unset grid
xticlabels-stacked.svg
This is gnuplot's "row stacking". It also supports "column stacking", which effectively transposes the data, and it's not obvious to me that makes sense in the context of feedgnuplot. Similarly, it can label y and/or z axes; I can't think of a specific use case, so I don't have a realistic usage in mind, and I don't support that yet. If anybody can think of a use case, email me. Notes and limitations:
  • Since with --domain you can pass in both an x value and a tic label, it is possible to give it conflicting tic labels for the same x value. gnuplot itself has this problem too, and it just takes the last label it has for a given x. This is probably good-enough.
  • feedgnuplot uses whitespace-separated columns with no escape mechanism, so the field labels cannot have whitespace in it. Fixing this is probably not worth the effort.
  • These tic labels do not count towards the tuplesize
  • I really need to add a similar feature to gnuplotlib. This will happen when I need it or when somebody asks for it, whichever comes first.

A feedgnuplot guide
This fills in a sorely needed missing part of the documentation: the main feedgnuplot website now has a page containing examples and corresponding graphical output. This serves as a tutorial and a gallery demonstrating some usages. It's somewhat incomplete, since it can't show streaming plots, or real-world interfacing with stuff that produces data: some of those usages remain the the README. It's a million times better than what I had before though, which was nothing. Internally this is done just like the gnuplotlib guide: the thing is an org-mode document with org-babel snippets that are evaluated by emacs to make the images. There's some fancy emacs lisp to tie it all together. Works great!

John Goerzen: Recovering Our Lost Free Will Online: Tools and Techniques That Are Available Now

As I ve been thinking and writing about privacy and decentralization lately, I had a conversation with a colleague this week, and he commented about how loss of privacy is related to loss of agency: that is, loss of our ability to make our own choices, pursue our own interests, and be master of our own attention. In terms of telecommunications, we have never really been free, though in terms of Internet and its predecessors, there have been times where we had a lot more choice. Many are too young to remember this, and for others, that era is a distant memory. The irony is that our present moment is one of enormous consolidation of power, and yet also one of a proliferation of technologies that let us wrest back some of that power. In this post, I hope to enlighten or remind us of some of the choices we have lost and also talk about the ways in which we can choose to regain them, already, right now. I will talk about the possibilities, the big dreams that are possible now, and then go into more detail about the solutions. The Problems & Possibilities The limitations of online We make the assumption that we must be online to exchange data. This is reinforced by many modern protocols; Twitter clients, for instance, don t tend to let you make posts by relaying them through disconnected devices. What would it be like if you could fully participate in global communities without a constant Internet connection? If you could share photos with your friends, read the news, read your email, etc. even if you don t have a connection at present? Even if the device you use to do that never has a connection, but can route messages via other devices that do? Would it surprise you to learn that this was once the case? Back in the days of UUCP, much email and Usenet news a global discussion forum that didn t require an Internet connection was relayed via occasional calls over phone lines. This technology remains with us, and has even improved. Sadly, many modern protocols make no effort in this regard. Some email clients will let you compose messages offline to send when you get online later, but the assumption always is that you will be connected to an IP network again soon. NNCP, on the other hand, lets you relay messages over TCP, a radio, a satellite, or a USB stick. Email and Usenet, since they were designed in an era where store-and-forward was valued, can actually still be used in an entirely offline fashion (without ever touching an IP-based network). All it takes is for someone to care to make it happen. You can even still do it over UUCP if you like. The physical and data link layers Many of us just accept that we communicate in a few ways: Wifi for short distances, and then cable modems or DSL for our local Internet connection, and then many people are fuzzy about what happens after that. Or, alternatively, we have 4G phones that are the local Internet connection, and the same fuzzy things happen after. Think about this for a moment. Which of these do you control in any way? Sometimes just wifi, sometimes maybe you have choices of local Internet providers. After that, your traffic is handled by enormous infrastructure companies. There is choice here. People in ham radio have been communicating digitally over long distances without the support of the traditional Internet for decades, but the technology to do this is now more accessible to anyone. Long-distance radio has had tremendous innovation in the last decade; cheap radios can now communicate over several miles/km without any other infrastructure at all. We all carry around radios (Wifi and Bluetooth) in our pockets that don t have to be used as mere access points to the Internet or as drivers of headphones, but can also form their own networks directly (Briar). Meshtastic is an example; it s an instant messenger that can form a mesh over many miles/km and requires no IP infrastructure at all. Briar is similar. XBee radios form a mesh in hardware, allowing peers to reach each other (also over many miles/km) with a serial or framed protocol. Loss of peer-to-peer Back in the late 90s, I worked at a university. I had a 386 on my desk for a workstation not a powerful computer even then. But I put the boa webserver on it and could just serve pages on the Internet. I didn t have to get permission. Didn t have to pay a hosting provider. I could just DO it. And of course that is because the university had no firewall and no NAT. Every PC at the university was a full participant on the Internet as much as the servers at Microsoft or DEC. All I needed was a DNS entry. I could run my own SMTP server if I wanted, run a web or Gopher server, and that was that. There are many reasons why this changed. Nowadays most residential ISPs will block SMTP for their customers, and if they didn t, others would; large email providers have decided not to federate with IPs in residential address spaces. Most people have difficulty even getting a static IP address in the first place. Many are behind firewalls, NATs, or both, meaning that incoming connections of any kind are problematic. Do you see what that means? It has weakened the whole point of the Internet being a network of peers. While IP still acts that way, as a practical matter, there are clients that are prevented from being servers by administrative policy they have no control over. Imagine if you, a person with an Internet connection to your laptop or phone, could just decide to host a website, or a forum on it. For moderate levels of load, they are certainly capable of this. The only thing in the way is the network management policies you can t control. Elaborate technologies exist to try to bridge this divide, and some, like Tor or cjdns, can work quite well. More on this below. Expense of running something popular Related to the loss of peer-to-peer infrastructure is the very high cost of hosting something popular. Do you want to share videos with lots of people? That almost certainly is going to require expensive equipment and bandwidth. There is a reason that there are only a small handful of popular video streaming sites online. It requires a ton of money to host videos at scale. What if it didn t? What if you could achieve economies of scale so much that you, an individual, could compete with the likes of YouTube? You wouldn t necessarily have to run ads to support the service. You wouldn t have to have billions of dollars or billions of viewers just to make it work. This technology exists right now. Of course many of you are aware of how Bittorrent leverages the swarm for files. But projects like IPFS, Dat, and Peertube have taken this many steps further to integrate it into a global ecosystem. And, at least in the case of Peertube, this is a thing that works right now in any browser already! Application-level walled gardens I was recently startled at how much excitement there was when Github introduced dark mode . Yes, Github now offers two colors on its interface. Already back in the 80s and 90s, many DOS programs had more options than that. Git is a decentralized protocol, but Github has managed to make it centralized. Email is a decentralized protocol pick your own provider, and they all communicate but Facebook and Twitter aren t. You can t just pick your provider for Facebook. It s Facebook or nothing. There is a profit motive in locking others out; these networks want to keep you using their platforms because their real customers are advertisers, and they want to keep showing you ads. Is it possible to have a world where you get to pick your own app for sharing photos, and it works even if your parents use a different one? Yes, yes it is. Mastodon and the Fediverse are fantastic examples for social media. Pixelfed is specifically designed for photos, Mastodon for short-form communication, there s Pleroma for more long-form communication, and they all work together. You can use Mastodon to read Pleroma content or look at Pixelfed photos, and there are many (free) providers of each. Freedom from manipulation I recently wrote about the dangers of the attention economy, so I won t go into a lot of detail here. Fundamentally, you are not the customer of Facebook or Google; advertisers are. They optimize their site to keep you on it as much as possible so that they can show you as many ads as possible which makes them as much money as possible. Ads, of course, are fundamentally seeking to manipulate your behavior ( buy this product ). By lowering the cost of running services, we can give a huge boost to hobbyists and nonprofits that want to do so without an ultimate profit motive. For-profit companies benefit also, with a dramatically reduced cost structure that frees them to pursue their mission instead of so many ads. Freedom from snooping (privacy and anonymity) These days, it s not just government snooping that people think about. It s data stolen by malware, spies at corporations (whether human or algorithmic), and even things like basic privacy of one s own security footage. Here the picture is improving; encryption in transit, at least at a basic level, has become much more common with TLS being a standard these days. Sadly, end-to-end encryption (E2EE) is not nearly as much, perhaps because corporations have a profit motive to have access to your plaintext and metadata. Closely related to privacy is anonymity: that is, being able to do things in an anonymous fashion. The two are not necessarily equal: you could send an encrypted message but reveal who the correspondents are, as with email; or, you could send a plaintext message over a Tor exit node that hides who the correspondents are. It is sometimes difficult to achieve both. Nevertheless, numerous answers exist here that tackle one or both problems, from the Signal messenger to Tor. Solutions That Exist Today Let s dive in to some of the things that exist today. One concept you ll see in many of these is integrated encryption with public keys used for addressing. In other words, your public key is akin to an IP address (and in some cases, is literally your IP address.) Data link and networking technologies (some including P2P) P2P Infrastructure While some of the technologies above, such as cjdns, explicitly facitilitate peer-to-peer communication, there are some other application-level technologies to look at. Instant Messengers and Chat I won t go into a lot of detail here since I recently wrote a roundup of secure mesh messengers and also a followup article about Signal and some hidden drawbacks of P2P. Please refer to those articles for some interesting things that are happening in this space. Matrix is a distributed IM platform similar in concept to Slack or IRC, but globally distributed in a mesh. It supports optional E2EE. Social Media I wrote recently about how to join the Fediverse, which covered joining Mastodon, a federeated, decentralized social network. Mastodon is the largest of these, with several million users, and is something of a much nicer version of Twitter. Mastodon is also part of what is known as the Fediverse , which are applications that are loosely joined together by their support of the ActivityPub protocol. Other popular Fediverse applications include Pixelfed (similar to Instagram) and Peertube for sharing video. Peertube is particularly interesting in that it supports Webtorrent for efficiently distributing popular videos. Webtorrent is akin to Bittorrent running efficiently inside your browser. Concluding Remarks Part of my goal with this is encouraging people to dream big, to ask questions like: What could you do if offline were easy? What is possible if you have freedom in the physical and data link layers? Dream big. We re so used to thinking that it s quite difficult for two devices on the Internet to talk to each other. What would be possible if this were actually quite easy? The assumption that costs rise dramatically as popularity increases is also baked into our thought processes. What if that weren t the case could you take on Youtube from your garage? Would lowering barriers to entry lower the ad economy and let nonprofits have more equal footing with large corporations? We have so many walled gardens, from Github to Facebook, that we almost forget it doesn t have to be that way. So having asked these questions, my secondary point is to suggest that these aren t pie-in-the-sky notions. These possibilites are with us right now. You ll notice from this list that virtually every one of these technologies is ad-free at its heart (though some would be capable of serving ads). They give you back your attention. Many preserve privacy, anonymity, or both. Many dramatically improve your freedom of association and communication. Technologies like IPFS and Bittorrent ease the burden of running something popular. Some are quite easy to use (Mastodon or Peertube) while others are much more complex (libp2p or the lower-level mesh network systems). Clearly there is still room for improvement in many areas. But my fundamental point is this: good technology is here, right now. Technical people can vote with their feet and wallets and start using it. Early adopters will help guide the way for the next set of improvements. Join us!

21 February 2021

Matthew Garrett: Making hibernation work under Linux Lockdown

Linux draws a distinction between code running in kernel (kernel space) and applications running in userland (user space). This is enforced at the hardware level - in x86-speak[1], kernel space code runs in ring 0 and user space code runs in ring 3[2]. If you're running in ring 3 and you attempt to touch memory that's only accessible in ring 0, the hardware will raise a fault. No matter how privileged your ring 3 code, you don't get to touch ring 0.

Kind of. In theory. Traditionally this wasn't well enforced. At the most basic level, since root can load kernel modules, you could just build a kernel module that performed any kernel modifications you wanted and then have root load it. Technically user space code wasn't modifying kernel space code, but the difference was pretty semantic rather than useful. But it got worse - root could also map memory ranges belonging to PCI devices[3], and if the device could perform DMA you could just ask the device to overwrite bits of the kernel[4]. Or root could modify special CPU registers ("Model Specific Registers", or MSRs) that alter CPU behaviour via the /dev/msr interface, and compromise the kernel boundary that way.

It turns out that there were a number of ways root was effectively equivalent to ring 0, and the boundary was more about reliability (ie, a process running as root that ends up misbehaving should still only be able to crash itself rather than taking down the kernel with it) than security. After all, if you were root you could just replace the on-disk kernel with a backdoored one and reboot. Going deeper, you could replace the bootloader with one that automatically injected backdoors into a legitimate kernel image. We didn't have any way to prevent this sort of thing, so attempting to harden the root/kernel boundary wasn't especially interesting.

In 2012 Microsoft started requiring vendors ship systems with UEFI Secure Boot, a firmware feature that allowed[5] systems to refuse to boot anything without an appropriate signature. This not only enabled the creation of a system that drew a strong boundary between root and kernel, it arguably required one - what's the point of restricting what the firmware will stick in ring 0 if root can just throw more code in there afterwards? What ended up as the Lockdown Linux Security Module provides the tooling for this, blocking userspace interfaces that can be used to modify the kernel and enforcing that any modules have a trusted signature.

But that comes at something of a cost. Most of the features that Lockdown blocks are fairly niche, so the direct impact of having it enabled is small. Except that it also blocks hibernation[6], and it turns out some people were using that. The obvious question is "what does hibernation have to do with keeping root out of kernel space", and the answer is a little convoluted and is tied into how Linux implements hibernation. Basically, Linux saves system state into the swap partition and modifies the header to indicate that there's a hibernation image there instead of swap. On the next boot, the kernel sees the header indicating that it's a hibernation image, copies the contents of the swap partition back into RAM, and then jumps back into the old kernel code. What ensures that the hibernation image was actually written out by the kernel? Absolutely nothing, which means a motivated attacker with root access could turn off swap, write a hibernation image to the swap partition themselves, and then reboot. The kernel would happily resume into the attacker's image, giving the attacker control over what gets copied back into kernel space.

This is annoying, because normally when we think about attacks on swap we mitigate it by requiring an encrypted swap partition. But in this case, our attacker is root, and so already has access to the plaintext version of the swap partition. Disk encryption doesn't save us here. We need some way to verify that the hibernation image was written out by the kernel, not by root. And thankfully we have some tools for that.

Trusted Platform Modules (TPMs) are cryptographic coprocessors[7] capable of doing things like generating encryption keys and then encrypting things with them. You can ask a TPM to encrypt something with a key that's tied to that specific TPM - the OS has no access to the decryption key, and nor does any other TPM. So we can have the kernel generate an encryption key, encrypt part of the hibernation image with it, and then have the TPM encrypt it. We store the encrypted copy of the key in the hibernation image as well. On resume, the kernel reads the encrypted copy of the key, passes it to the TPM, gets the decrypted copy back and is able to verify the hibernation image.

That's great! Except root can do exactly the same thing. This tells us the hibernation image was generated on this machine, but doesn't tell us that it was done by the kernel. We need some way to be able to differentiate between keys that were generated in kernel and ones that were generated in userland. TPMs have the concept of "localities" (effectively privilege levels) that would be perfect for this. Userland is only able to access locality 0, so the kernel could simply use locality 1 to encrypt the key. Unfortunately, despite trying pretty hard, I've been unable to get localities to work. The motherboard chipset on my test machines simply doesn't forward any accesses to the TPM unless they're for locality 0. I needed another approach.

TPMs have a set of Platform Configuration Registers (PCRs), intended for keeping a record of system state. The OS isn't able to modify the PCRs directly. Instead, the OS provides a cryptographic hash of some material to the TPM. The TPM takes the existing PCR value, appends the new hash to that, and then stores the hash of the combination in the PCR - a process called "extension". This means that the new value of the TPM depends not only on the value of the new data, it depends on the previous value of the PCR - and, in turn, that previous value depended on its previous value, and so on. The only way to get to a specific PCR value is to either (a) break the hash algorithm, or (b) perform exactly the same sequence of writes. On system reset the PCRs go back to a known value, and the entire process starts again.

Some PCRs are different. PCR 23, for example, can be reset back to its original value without resetting the system. We can make use of that. The first thing we need to do is to prevent userland from being able to reset or extend PCR 23 itself. All TPM accesses go through the kernel, so this is a simple matter of parsing the write before it's sent to the TPM and returning an error if it's a sensitive command that would touch PCR 23. We now know that any change in PCR 23's state will be restricted to the kernel.

When we encrypt material with the TPM, we can ask it to record the PCR state. This is given back to us as metadata accompanying the encrypted secret. Along with the metadata is an additional signature created by the TPM, which can be used to prove that the metadata is both legitimate and associated with this specific encrypted data. In our case, that means we know what the value of PCR 23 was when we encrypted the key. That means that if we simply extend PCR 23 with a known value in-kernel before encrypting our key, we can look at the value of PCR 23 in the metadata. If it matches, the key was encrypted by the kernel - userland can create its own key, but it has no way to extend PCR 23 to the appropriate value first. We now know that the key was generated by the kernel.

But what if the attacker is able to gain access to the encrypted key? Let's say a kernel bug is hit that prevents hibernation from resuming, and you boot back up without wiping the hibernation image. Root can then read the key from the partition, ask the TPM to decrypt it, and then use that to create a new hibernation image. We probably want to prevent that as well. Fortunately, when you ask the TPM to encrypt something, you can ask that the TPM only decrypt it if the PCRs have specific values. "Sealing" material to the TPM in this way allows you to block decryption if the system isn't in the desired state. So, we define a policy that says that PCR 23 must have the same value at resume as it did on hibernation. On resume, the kernel resets PCR 23, extends it to the same value it did during hibernation, and then attempts to decrypt the key. Afterwards, it resets PCR 23 back to the initial value. Even if an attacker gains access to the encrypted copy of the key, the TPM will refuse to decrypt it.

And that's what this patchset implements. There's one fairly significant flaw at the moment, which is simply that an attacker can just reboot into an older kernel that doesn't implement the PCR 23 blocking and set up state by hand. Fortunately, this can be avoided using another aspect of the boot process. When you boot something via UEFI Secure Boot, the signing key used to verify the booted code is measured into PCR 7 by the system firmware. In the Linux world, the Shim bootloader then measures any additional keys that are used. By either using a new key to tag kernels that have support for the PCR 23 restrictions, or by embedding some additional metadata in the kernel that indicates the presence of this feature and measuring that, we can have a PCR 7 value that verifies that the PCR 23 restrictions are present. We then seal the key to PCR 7 as well as PCR 23, and if an attacker boots into a kernel that doesn't have this feature the PCR 7 value will be different and the TPM will refuse to decrypt the secret.

While there's a whole bunch of complexity here, the process should be entirely transparent to the user. The current implementation requires a TPM 2, and I'm not certain whether TPM 1.2 provides all the features necessary to do this properly - if so, extending it shouldn't be hard, but also all systems shipped in the past few years should have a TPM 2, so that's going to depend on whether there's sufficient interest to justify the work. And we're also at the early days of review, so there's always the risk that I've missed something obvious and there are terrible holes in this. And, well, given that it took almost 8 years to get the Lockdown patchset into mainline, let's not assume that I'm good at landing security code.

[1] Other architectures use different terminology here, such as "supervisor" and "user" mode, but it's broadly equivalent
[2] In theory rings 1 and 2 would allow you to run drivers with privileges somewhere between full kernel access and userland applications, but in reality we just don't talk about them in polite company
[3] This is how graphics worked in Linux before kernel modesetting turned up. XFree86 would just map your GPU's registers into userland and poke them directly. This was not a huge win for stability
[4] IOMMUs can help you here, by restricting the memory PCI devices can DMA to or from. The kernel then gets to allocate ranges for device buffers and configure the IOMMU such that the device can't DMA to anything else. Except that region of memory may still contain sensitive material such as function pointers, and attacks like this can still cause you problems as a result.
[5] This describes why I'm using "allowed" rather than "required" here
[6] Saving the system state to disk and powering down the platform entirely - significantly slower than suspending the system while keeping state in RAM, but also resilient against the system losing power.
[7] With some handwaving around "coprocessor". TPMs can't be part of the OS or the system firmware, but they don't technically need to be an independent component. Intel have a TPM implementation that runs on the Management Engine, a separate processor built into the motherboard chipset. AMD have one that runs on the Platform Security Processor, a small ARM core built into their CPU. Various ARM implementations run a TPM in Trustzone, a special CPU mode that (in theory) is able to access resources that are entirely blocked off from anything running in the OS, kernel or otherwise.

comment count unavailable comments

Next.

Previous.