In the last week, KDE released version 5.23 25th Anniversary Edition of the Plasma desktop with the usual long list of updates and improvements. This release celebrates 25 years of KDE, and Plasma 5.23.0 was released right on the day 25 years ago Matthias Ettrich sent an email to the de.comp.os.linux.misc newsgroup explaining a project he was working on. And Plasma 5.23 (with the bug fix 5.23.1) is now available for all Debian releases. (And don t forget KDE Gears/Apps 21.08!)
As usual, I am providing packages via my OBS builds. If you have used my packages till now, then you only need to change the plasma522 line to read plasma523. To give full details, I repeat (and update) instructions for all here: First of all, you need to add my OBS key say in /etc/apt/trusted.gpg.d/obs-npreining.asc and add a file /etc/apt/sources.lists.d/obs-npreining-kde.list, containing the following lines, replacing the DISTRIBUTION part with one of Debian_11 (for Bullseye), Debian_Testing, or Debian_Unstable:
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/other-deps/DISTRIBUTION/ ./
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/frameworks/DISTRIBUTION/ ./
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/plasma523/DISTRIBUTION/ ./
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/apps2108/DISTRIBUTION/ ./
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/other/DISTRIBUTION/ ./
The sharp eye might have detected also the apps2108 line, yes the KDE Gear suite of packages hgas been updated to 21.08 some time ago and is also available in my OBS builds (and in Debian/experimental).
Uploads to Debian
Plasma 5.23.0 has been uploaded to Debian, and is currently in transition to testing. Due to incorrect/insufficient Break/Depends, currently Debian/testing with the official packages for Plasma are broken. And as it looks this situation will continue for considerable time, considering that kwin is blocked by mesa, which in turn is blocked by llvm-toolchain-12, which has quite some RC bugs preventing it from transitioning. What a bad coincidence.
KDE Gears 21.08 are all in Debian Unstable and Testing, so the repositories here are mostly for users of Debian Bullseye (stable).
Krita has released the second beta of Krita 5.0, and this is available from the krita-beta repository, but only for amd64 architectures. Just add
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/krita-beta/DISTRIBUTION/ ./
This post was originally published in the Wikimedia Tech blog, authored by
Arturo Borrero Gonzalez.
NFS is a central piece of infrastructure that is essential to services like Toolforge.
Recently, the Cloud Services team at Wikimedia had been reviewing how we do NFS.
The current situation
NFS is a central piece of technology for some of the services that the
Wikimedia Cloud Services team offers to the community. We have several shares that power
different use cases: Toolforge user home directories live on NFS, and
Cloud VPS users can also access dumps using this protocol. The current setup involves
several physical hardware servers, with about 20TB of storage, offering shares over 10G links to
the cloud. For the system to be more fault-tolerant, we duplicate each share for redundancy using
Running NFS on dedicated hardware servers has traditionally offered us advantages: mostly on the
performance and the capacity fields.
As time has passed, we have been enumerating more and more reasons to review how we do NFS. For
one, the current setup is in violation of some of our internal rules regarding realm separation.
Additionally, we had been longing for additional flexibility managing our servers: we wanted to use
virtual machines managed by Openstack Nova. The DRBD-based high-availability system
required mostly a hand-crafted procedure for failover/failback. There s also some scalability
concerns as NFS is easy to grow up, but not to grow horizontally, and of course, we have to be able
to keep the tenancy setup while doing so, something that NFS does by using LDAP/Unix users and may
get complicated too when growing. In general, the servers have become too big to fail , clearly
technical debt, and it has taken us years to decide on taking on the task to rethink the
architecture. It s worth mentioning that in an ideal world, we wouldn t depend on NFS, but the
truth is that it will still be a central piece of infrastructure for years to come in services like
Over a series of brainstorming meetings, the WMCS team evaluated the situation and sorted out the
many moving parts. The team managed to boil down the potential service future to two competing
Adopt and introduce a new Openstack component into our cloud: Manila this was the right choice
if we were interested in a general NFS as a service offering for our Cloud VPS users.
Put the data on Cinder volumes and serve NFS from a couple of virtual machines created by hand
this was the right choice if we wanted something that required low effort to engineer and adopt.
Then we decided to research both options in parallel. For a number of reasons, the evaluation was
timeboxed to three weeks. Both ideas had a couple of points in common: the NFS data would be stored
on our Ceph farm via Cinder volumes, and we would rely on Ceph reliability to avoid using DRBD.
Another open topic was how to back up data from Ceph, to store our important bits in more than one
basket. We will get to the back up topic later.
The manila experiment
The Wikimedia Foundation was an early adopter of some Openstack components (Nova,
Glance, Designate, Horizon), but Manila was never
evaluated for usage until now. Our approach for this experiment was to closely follow the upstream
guidelines. We read the documentation and tried to understand the different setups you can build
with Manila. As we often feel with other Openstack components, the documentation doesn t perfectly
describe how to introduce a given component in your particular local setup. Here we use an
admin-controller flat-topology Neutron network. This network is shared by all tenants (or projects)
in our Openstack deployment. Also, Manila can use many different driver backends, for
things like NetApps or CephFS that we don t use , yet. After some research, the
generic driver was the one that seemed to better fit our use case. The generic driver
leverages Nova virtual machines instances plus Cinder volume to create and manage the shares. In
general, Manila supports two operational modes, whether it should create/destroy the share servers
(i.e, the virtual machine instances) or not. This option is called driver_handles_share_server (or
DHSS) and takes a boolean value.
We were interested in trying with DHSS=true, to really benefit from the potential of the setup.
NFS idea 6, original image in Wikitech
So, after sorting all these variables, we moved on with our initial testing. We built a PoC setup
as depicted in the diagram above, with the manila-share component running in a virtual machine
inside the cloud. The PoC led to us reporting several bugs upstream:
It s worth mentioning that the upstream community was extra-welcoming to us, and we re thankful for
that. However, at the end of our three-week period, our Manila setup still wasn t working as
expected. Your experience may change with other drivers perhaps the ZFSonLinux or the CephFS ones.
In general, we were having trouble making the setup work as expected, so we decided to abandon this
approach in favor of the other option we were considering at the beginning.
Simple virtual machine serving NFS
The alternative was to create a Nova virtual machine instance by hand and to configure it using
puppet. We have been investing in an automation framework lately, so the idea is to
not actually create the server by hand. Anyway, the data would be decoupled from the instance into
Cinder volumes, which led us to the question we left for later: How should we back up those
terabytes of important information? Just to be clear, the backup problem was independent of the above
options; with Manila we would still have had to solve the same challenge. We would like to see our
data be backed up somewhere else other than in Ceph. And that s exactly where we are at right now.
We ve been exploring different backup strategies and will finally use the Cinder backup
The iteration will end with the dedicated NFS hardware servers being stopped, and the shares being
served from within the cloud. The migration will take some time to happen because we will check and
double-check that everything works as expected (including from the performance point of view)
before making definitive changes. We already have some plans to make sure our users experience as
little service impact as possible. The most troublesome shares will be those related to Toolforge.
At some point we will need to disallow writes to the NFS share, rsync the data out of the hardware
servers into the Cinder volumes, point the NFS clients to the new virtual machines, and then enable
writes again. The main Toolforge share has about 8TB of data, so this will take a while.
We will have more updates in the future. Who knows, perhaps our next-next iteration, in a couple of
years, will see us adopting Openstack Manila for good.
Featured image credit: File:(from break water) Manila Skyline panoramio.jpg, ewol, CC BY-SA 3.0This post was originally published in the Wikimedia Tech blog, authored
by Arturo Borrero Gonzalez.
Like each month, have a look at the work funded by Freexian s Debian LTS offering.
Debian project funding
Folks from the LTS team, along with members of the Debian Android Tools team and Phil Morrel, have proposed work on the Java build tool, gradle, which is currently blocked due to the need to build with a plugin not available in Debian. The LTS team reviewed the project submission and it has been approved. After approval we ve created a Request for Bids which is active now.
You ll hear more about this through official Debian channels, but in the meantime, if you feel you can help with this project, please submit a bid. Thanks!
This September, Freexian set aside 2550 EUR to fund Debian projects.
We re looking forward to receive more projects from various Debian teams! Learn more about the rationale behind this initiative in this article.
Debian LTS contributors
In September, 15 contributors have been paid to work on Debian LTS, their reports are available:
Abhijith PA has returned hours and marked themselves inactive, at least for the time being. He did 0h out of 14h, carried over 14h and returned 28h.
Adrian Bunk did 19.5h (out of 24.75h assigned and 12.75 from August), carrying over 18h to October.
Utkarsh Gupta did 24.75h (out of 24.75h assigned) but did not publish his report yet.
Ola Lundqvist did 2 hours (out of 21h carried over from previous months), and is thus carrying 19h for October.
Evolution of the situation
In September we released 30 DLAs. September was also the second month of Jeremiah coordinating LTS contributors.
Also, we would like say that we are always looking for new contributors to LTS. Please contact Jeremiah if you are interested!
The security tracker currently lists 33 packages with a known CVE and the dla-needed.txt file has 26 packages needing an update.
Thanks to our sponsors
Sponsors that joined recently are in bold.
This month I accepted 224 and rejected 47 packages. This is almost thrice the rejects of last month. Please, be more careful and check your package twice before uploading. The overall number of packages that got accepted was 233.
This was my eighty-seventh month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.
This month my all in all workload has been 24.75h. During that time I did LTS and normal security uploads of:
[DLA 2755-1] btrbk security update for one CVE
[DLA 2762-1] grilo security update for one CVE
[DLA 2766-1] openssl security update for one CVE
[DLA 2774-1] openssl1.0 security update for one CVE
[DLA 2773-1] curl security update for two CVEs
I also started to work on exiv2 and faad2.
Last but not least I did some days of frontdesk duties.
This month was the thirty-ninth ELTS month.
Unfortunately during my allocated time I could not process any upload. I worked on openssl, curl and squashfs-tools but for one reason or another the prepared packages didn t pass all tests. In order to avoid regressions, I postponed the uploads (meanwhile an ELA for curl was published ).
Last but not least I did some days of frontdesk duties.
On my neverending golang challenge I again uploaded some packages either for NEW or as source upload.
As Odyx took a break from all Debian activities, I volunteered to take care of the printing packages. Please be merciful when somethings breaks after I did an upload. My first printing upload was hplip
A new release 0.4.14 of RQuantLib was uploaded to CRAN earlier today, and has by now been uploaded to Debian as well.
QuantLib is a very comprehensice free/open-source library for quantitative finance; RQuantLib connects it to the R environment and language.
The release of RQuantLib comes just one months after the previous release, and brings three changes. First, we added both two more US-based calendars (including FederalReserve ) along with a bunch of not-yet-included other calendars which should complete the coverage in the R package relative to the upstream library. Should we have forgotten any, feel free to open an issue. Second, CRAN currently aims to have older autoconf conventions updated and notified maintainers of affected packages. I received a handful of these, and just like yesterday s update to littler refreshed this here. Third, we set up automated container builds on GitHub. No other changes were made, details follow.
Changes in RQuantLib version 0.4.14 (2021-10-06)
Changes in RQuantLib code:
Several new calendars were added (Dirk in #159 closing #155)
Changes in RQuantLib package and setup:
Docker containers are now updated on a monthly schedule via GitHub Actions
The configure files were updated to the standard of version 2.69 following a CRAN request
The goal behind reproducible builds is to ensure that no deliberate flaws have been introduced during compilation processes via promising or mandating that identical results are always generated from a given source. This allowing multiple third-parties to come to an agreement on whether a build was compromised or not by a system of distributed consensus.
In these reports we outline the most important things that have been happening in the world of reproducible builds in the past month:
First mentioned in our March 2021 report, Martin Heinz published two blog posts on sigstore, a project that endeavours to offer software signing as a public good, [the] software-signing equivalent to Let s Encrypt . The two posts, the first entitled Sigstore: A Solution to Software Supply Chain Security outlines more about the project and justifies its existence:
Software signing is not a new problem, so there must be some solution already, right? Yes, but signing software and maintaining keys is very difficult especially for non-security folks and UX of existing tools such as PGP leave much to be desired. That s why we need something like sigstore - an easy to use software/toolset for signing software artifacts.
Some time ago I checked Signal s reproducibility and it failed. I asked others to test in case I did something wrong, but nobody made any reports. Since then I tried to test the Google Play Store version of the apk against one I compiled myself, and that doesn t match either.
BitcoinBinary.org was announced this month, which aims to be a repository of Reproducible Build Proofs for Bitcoin Projects :
Most users are not capable of building from source code themselves, but we can at least get them able enough to check signatures and shasums. When reputable people who can tell everyone they were able to reproduce the project s build, others at least have a secondary source of validation.
On our website this month, Bernhard M. Wiedemann fixed some broken links  and Holger Levsen made a number of changes to the Who is Involved? page . On our mailing list, Magnus Ihse Bursie started a thread with the subject Reproducible builds on Java, which begins as follows:
I m working for Oracle in the Build Group for OpenJDK which is primary responsible for creating a built artifact of the OpenJDK source code. [ ] For the last few years, we have worked on a low-effort, background-style project to make the build of OpenJDK itself building reproducible. We ve come far, but there are still issues I d like to address. 
diffoscopediffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions 183, 184 and 185 as well as performed significant triaging of merge requests and other issues in addition to making the following changes:
Support a newer format version of the R language s .rds files. 
Don t call close_archive when garbage collecting Archive instances, unless open_archive definitely returned successfully. This prevents, for example, an AttributeError where PGPContainer s cleanup routines were rightfully assuming that its temporary directory had actually been created. 
Fix (and test) the comparison of R language s .rdb files after refactoring temporary directory handling. 
Ensure that RPM archives exists in the Debian package description, regardless of whether python3-rpm is installed or not at build time. 
Use our assert_diff routine in tests/comparators/test_rdata.py. 
Move diffoscope.versions to diffoscope.tests.utils.versions. 
Reformat a number of modules with Black. 
However, the following changes were also made:
Fix an autopkgtest caused by the androguard module not being in the (expected) python3-androguard Debian package. 
Appease a shellcheck warning in debian/tests/control.sh. 
Ignore a warning from h5py in our tests that doesn t concern us. 
Drop a trailing .1 from the Standards-Version field as it s required. 
Zbigniew J drzejewski-Szmek:
Stop using the deprecated distutils.spawn.find_executable utility. 
Adjust an LLVM-related test for LLVM version 13. 
Update invocations of llvm-objdump. 
Adjust a test with a one-byte text file for file version 5.40. 
And, finally, Benjamin Peterson added a --diff-context option to control unified diff context size  and Jean-Romain Garnier fixed the Macho comparator for architectures other than x86-64 .
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
#990246 filed against vlc (forwarded upstream )
The Reproducible Builds project runs a testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:
Drop my package rebuilder prototype as it s not useful anymore. 
Schedule old packages in Debian bookworm. 
Stop scheduling packages for Debian buster. 
Don t include PostgreSQL debug output in package lists. 
Detect Python library mismatches during build in the node health check. 
Update a note on updating the FreeBSD system. 
Silence a warning from Git. 
Update a setting to reflect that Debian bookworm is the new testing. 
Upgrade the PostgreSQL database to version 13. 
Roland Clobus (Debian live image generation):
Workaround non-reproducible config files in the libxml-sax-perl package. 
Use the new DNS for the snapshot service. 
Also note that the armhf architecture also systematically varies by the kernel. 
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
Evolution of the situation
In August we released 30 DLAs.
This is the first month of Jeremiah coordinating LTS contributors. We would like to thank Holger Levsen for his work on this role up to now.
Also, we would like to remark once again that we are constantly looking for new contributors. Please contact Jeremiah if you are interested!
The security tracker currently lists 73 packages with a known CVE and the dla-needed.txt file has 29 packages needing an update.
Thanks to our sponsors
Sponsors that joined recently are in bold.
Using podman for most of my local development environment.
For my personal/upstream development I started using podman
instead of lxc and pbuilder and other toolings. Most
projects provide reasonable docker images (such as rust) and
I am happier keeping my environment as a whole stable while
I can iterate. I have a Dockerfile for the development
environment like this:
Or perhaps this, if you also wanted to send ssh to the background:
ssh -NT -L 3306:db.example.com:3306 example.com &
Both of these use at least one option that is entirely redundant,
and the second can cause ssh to fail to connect if you happen to be
using password authentication.
However, they seem to still persist in various articles about ssh port forwarding.
I myself was using the first variation until just recently, and I figured
I would write this up to inform others who might be still using these
The correct option for this situation is not -nNT but simply
-N, as in:
If you want to also send ssh to the background,
then you ll want to add -f instead of using your shell s built-in &
feature, because you can then input passwords into ssh if necessary1
Honestly, that s the point of this article, so you can stop here
if you want. If you re looking for a detailed explaination of what
each of these options actually does, or if you have no idea what
I m talking about, read on!
What is SSH Port Forwarding?
ssh is a powerful tool for remote access to servers,
allowing you to execute commands on a remote machine.
It can also forward ports
through a secure tunnel with the -L and -R options.
Basically, you can forward a connection to a local port to a remote server
ssh -L 8080:other.example.com:80 ssh.example.com
In this example, you connect to ssh.example.com and then ssh forwards
any traffic on your local machine port 80802
to other.example.com port 80 via ssh.example.com.
This is a really powerful feature, allowing you to jump3 inside your firewall
with just an ssh server exposed to the world.
It can work in reverse as well with the -R option, allowing connections
on a remote host in to a server running on your local machine.
For example, say you were running a website on your local machine on port 8080
but wanted it accessible on example.com port 804.
You could use something like:
ssh -R 8080:example.com:80 example.com
The trouble with ssh port forwarding is that, absent any additional options,
you also open a shell on the remote machine.
If you re planning to both work on a remote machine and use it to forward
some connection, this is fine, but if you just need to forward a port quickly
and don t care about a shell at that moment, it can be annoying, especially since
if the shell closes ssh will close the forwarding port as well.
This is where the -N option comes in.
SSH just forwarding ports
In the ssh manual page5, -N is explained like so:
Do not execute a remote command. This is useful for just forwarding ports.
This is all we need. It instructs ssh to run no commands on the remote server,
just forward the ports specified in the -L or -R options.
But people seem to think that there are a bunch of other necessary options,
so what do those do?
SSH and stdin
-n controls how ssh interacts with standard input,
specifically telling it not to:
Redirects stdin from /dev/null (actually, prevents reading from stdin). This must be used when ssh is run in the
background. A common trick is to use this to run X11 programs on a remote machine. For example, ssh -n
shadows.cs.hut.fi emacs & will start an emacs on shadows.cs.hut.fi, and the X11 connection will be automatically for
warded over an encrypted channel. The ssh program will be put in the background. (This does not work if ssh needs to
ask for a password or passphrase; see also the -f option.)
SSH passwords and backgrounding
-f sends ssh to background, freeing up the terminal in which you ran ssh
to do other things.
Requests ssh to go to background just before command execution. This is useful if ssh is going to ask for passwords
or passphrases, but the user wants it in the background. This implies -n. The recommended way to start X11 programs
at a remote site is with something like ssh -f host xterm.
As indicated in the description of -n, this does the same thing as using
the shell s & feature
with -n, but allows you to put in any necessary passwords first.
SSH and pseudo-terminals
-T is a little more complicated than the others and has a very short
Disable pseudo-terminal allocation.
It has a counterpart in -t, which is explained a little better:
Force pseudo-terminal allocation. This can be used to execute arbitrary screen-based programs on a remote machine,
which can be very useful, e.g. when implementing menu services. Multiple -t options force tty allocation, even if ssh
has no local tty.
As the description of -t indicates, ssh is allocating a pseudo-terminal on
the remote machine, not the local one.
However, I have confirmed6 that -N doesn t allocate a pseudo-terminal either,
since it doesn t run any commands.
Thus this option is entirely unnecessary.
What s a pseudo-terminal?
This is a bit complicated, but basically it s an interface used in UNIX-like
systems, like Linux or BSD, that pretends to be a terminal
Programs like your shell, or any text-based menu system made in libraries
like ncurses, expect to be connected to one (when used interactively at least).
Basically it fakes as if the input it is given
(over the network, in the case of ssh) was typed on a physical terminal
device and do things like raise an interrupt (SIGINT) if Ctrl+C is pressed.
I don t know why these incorrect uses of ssh got passed around as correct,
but I suspect it s a form of cargo cult,
where we use example commands others provide and don t question what they do.
One stack overflow answer I read that provided these options seemed to think
-T was disabling the local pseudo-terminal, which might go some way towards
explaining why they thought it was necessary.
I guess the moral of this story is to question everything and actually read
the manual, instead of just googling it.
Not that you SHOULD be using ssh with password authentication anyway, but people do.
Only on your loopback address by default, so that you re not allowing random people on your network to use your tunnel.
In fact, ssh even supports Jump Hosts, allowing you to automatically forward an ssh connection through another machine.
I can t say I recommend a setup like this for anything serious, as you d need to ssh as root to forward ports less than 1024. SSH forwarding is not for permanent solutions, just short-lived connections to machines that would be otherwise inaccessible.
Specifically, my source is the ssh(1) manual page in OpenSSH 8.4, shipped as 1:8.4p1-5 in Debian bullseye.
I just forwarded ports with -N and then logged in to that same machine and looked at psuedo-terminal allocations via ps ux. No terminal is associated with ssh connections using just the -N option.
Kali is a Debian based distribution aimed at penetration testing. I haven t felt a need to use it in the past because Debian has packages for all the scanning tools I regularly use, and all the rest are free software that can be obtained separately. But I recently decided to try it.
Here s the URL to get Kali . For a VM you can get VMWare or VirtualBox images, I chose VMWare as it s the most popular image format and also a much smaller download (2.7G vs 4G). For unknown reasons the torrent for it didn t work (might be a problem with my torrent client). The download link for it was extremely slow in Australia, so I downloaded it to a system in Germany and then copied it from there.
I don t want to use either VMWare or VirtualBox because I find KVM/Qemu sufficient to do everything I want and they are in the Main section of Debian, so I needed to convert the image files. Some of the documentation on converting image formats to use with QEMU/KVM says to use a program called kvm-img which doesn t seem to exist, I used qemu-img from the qemu-utils package in Debian/Bullseye. The man page qemu-img(1) doesn t list the types of output format supported by the -O option and the examples returned by a web search show using -O qcow2 . It turns out that the following command will convert the image to raw format which is the format I prefer. I use BTRFS for storing all my VM images and that does all the copy-on-write I need.
After converting it the file was 500M smaller than the VMWare files (10.2 vs 10.7G). Probably the Kali distribution file could be reduced in size by converting it to raw and then back to VMWare format. The Kali VMWare image is compressed with 7zip which has a good compression ratio, I waited almost 90 minutes for zstd to compress it with -19 and the result was 12% larger than the 7zip file.
VMWare apparently likes to use an emulated SCSI controller, I spent some time trying to get that going in KVM. Apparently recent versions of QEMU changed the way this works and therefore older web pages aren t helpful. Also allegedly the SCSI emulation is buggy and unreliable (but I didn t manage to get it going so can t be sure). It turns out that the VM is configured to work with the virtio interface, the initramfs.conf has the configuration option MODULES=most which makes it boot on all common configurations (good work by the initramfs-tools maintainers). The image works well with the Spice display interface, so it doesn t capture my mouse, the window for the VM works the same way as other windows on my desktop and doesn t capture the mouse cursor. I don t know if this level of Spice integration is in Debian now, last time I tested it didn t work that way.
I also downloaded Metasploitable  which is a VM image designed to be full of security flaws for testing the tools that are in Kali. Again it worked nicely after converting from VMWare to raw format. One thing to note about Metasploitable is that you must not make it available on the public Internet. My home network has NAT for IPv4 but all systems get public IPv6 addresses. It s usually nice that those things just work on VMs but not for this. So I added an iptables command to block IPv6 to /etc/rc.local.
Installing VMs for both these distributions was quite easy. Most of my time was spent downloading from a slow server, trying to get SCSI emulation working, working out how to convert image files, and testing different compression options. The time spent doing stuff once I knew what to do was very small.
Kali has zsh as the default shell, it s quite nice. I ve been happy with bash for decades, but I might end up trying zsh out on other machines.
Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 912 other packages on CRAN.
This new release brings us Armadillo 10.7.0 released this morning by Conrad. Leading up to this were three runs of reverse dependencies the first of which uncovered the need for a small PR for subview_cols support which Conrad kindly supplied.
The full set of changes follows. We include the last interim release (sent as usual to the drat repo) as well.
Changes in RcppArmadillo version 0.10.7.0.0 (2021-09-30)
Upgraded to Armadillo release 10.7.0 (Entropy Maximizer)
faster handling of submatrix views accessed by X.cols(first_col,last_col)
faster handling of element-wise min() and max() in compound expressions
expanded solve() with solve_opts::force_approx option to force use of the approximate solver
Changes in RcppArmadillo version 0.10.6.2.0 (2021-08-05)
Upgraded to Armadillo release 10.6.2 (Keep Calm)
fix incorrect use of constexpr for handling fixed-size matrices and vectors
My home automation setup has been fairly static recently; it does what we need and generally works fine. One area I think could be better is controlling it; we have access Home Assistant on our phones, and the Alexa downstairs can control things, but there are no smart assistants upstairs and sometimes it would be nice to just push a button to turn on the light rather than having to get my phone out. Thanks to the fact the UK generally doesn t have neutral wire in wall switches that means looking at something battery powered. Which means wifi based devices are a poor choice, and it s necessary to look at something lower power like Zigbee or Z-Wave.
Zigbee seems like the better choice; it s a more open standard and there are generally more devices easily available from what I ve seen (e.g. Philips Hue and IKEA TR DFRI). So I bought a couple of Xiaomi Mi Smart Home Wireless Switches, and a CC2530 module and then ignored it for the best part of a year. Finally I got around to flashing the Z-Stack firmware that Koen Kanters kindly provides. (Insert rant about hardware manufacturers that require pay-for tool chains. The CC2530 is even worse because it s 8051 based, so SDCC should be able to compile for it, but the TI Zigbee libraries are only available in a format suitable for IAR s embedded workbench.)
Flashing the CC2530 is a bit of faff. I ended up using the CCLib fork by Stephan Hadinger which supports the ESP8266. The nice thing about the CC2530 module is it has 2.54mm pitch pins so nice and easy to jumper up. It then needs a USB/serial dongle to connect it up to a suitable machine, where I ran Zigbee2MQTT. This scares me a bit, because it s a bunch of node.js pulling in a chunk of stuff off npm. On the flip side, it Just Works and I was able to pair the Xiaomi button with the device and see MQTT messages that I could then use with Home Assistant. So of course I tore down that setup and went and ordered a CC2531 (the variant with USB as part of the chip). The idea here was my test setup was upstairs with my laptop, and I wanted something hooked up in a more permanent fashion.
Once the CC2531 arrived I got distracted writing support for the Desk Viking to support CCLib (and modified it a bit for Python3 and some speed ups). I flashed the dongle up with the Z-Stack Home 1.2 (default) firmware, and plugged it into the house server. At this point I more closely investigated what Home Assistant had to offer in terms of Zigbee integration. It turns out the ZHA integration has support for the ZNP protocol that the TI devices speak (I m reasonably sure it didn t when I first looked some time ago), so that seemed like a better option than adding the MQTT layer in the middle.
I hit some complexity passing the dongle (which turns up as /dev/ttyACM0) through to the Home Assistant container. First I needed an override file in /etc/systemd/nspawn/hass.nspawn:
(I m not clear why the VirtualEthernet needed to exist; without it networking broke entirely but I couldn t see why it worked with no override file.)
A udev rule on the host to change the ownership of the device file so the root user and dialout group in the container could see it was also necessary, so into /etc/udev/rules.d/70-persistent-serial.rules went:
In the container itself I had to switch PrivateDevices=true to PrivateDevices=false in the home-assistant.service file (which took me a while to figure out; yay for locking things down and then needing to use those locked down things).
Finally I added the hass user to the dialout group. At that point I was able to go and add the integration with Home Assistant, and add the button as a new device. Excellent. I did find I needed a newer version of Home Assistant to get support for the button, however. I was still on 2021.1.5 due to upstream dropping support for Python 3.7 and not being prepared to upgrade to Debian 11 until it was actually released, so the version of zha-quirks didn t have the correct info. Upgrading to Home Assistant 2021.8.7 sorted that out.
There was another slight problem. Range. Really I want to use the button upstairs. The server is downstairs, and most of my internal walls are brick. The solution turned out to be a TR DFRI socket, which replaced the existing ESP8266 wifi socket controlling the stair lights. That was close enough to the server to have a decent signal, and it acts as a Zigbee router so provides a strong enough signal for devices upstairs. The normal approach seems to be to have a lot of Zigbee light bulbs, but I have mostly kept overhead lights as uncontrolled - we don t use them day to day and it provides a nice fallback if the home automation has issues.
Of course installing Zigbee for a single button would seem to be a bit pointless. So I ordered up a Sonoff door sensor to put on the front door (much smaller than expected - those white boxes on the door are it in the picture above). And I have a 4 gang wireless switch ordered to go on the landing wall upstairs.
Now I ve got a Zigbee setup there are a few more things I m thinking of adding, where wifi isn t an option due to the need for battery operation (monitoring the external gas meter springs to mind). The CC2530 probably isn t suitable for my needs, as I ll need to write some custom code to handle the bits I want, but there do seem to be some ARM based devices which might well prove suitable
One of the assumptions baked deeply into US society (and many others) is
that people are largely defined by the work they do, and that work is the
primary focus of life. Even in Marxist analysis, which is otherwise
critical of how work is economically organized, work itself reigns
supreme. This has been part of the feminist critique of both capitalism
and Marxism, namely that both devalue domestic labor that has
traditionally been unpaid, but even that criticism is normally framed as
expanding the definition of work to include more of human activity. A
few exceptions aside, we shy away from
fundamentally rethinking the centrality of work to human experience.
The Problem with Work begins as a critical analysis of that
centrality of work and a history of some less-well-known movements against
it. But, more valuably for me, it becomes a discussion of the types and
merits of utopian thinking, including why convincing other people is not
the only purpose for making a political demand.
The largest problem with this book will be obvious early on: the writing
style ranges from unnecessarily complex to nearly unreadable. Here's an
excerpt from the first chapter:
The lack of interest in representing the daily grind of work routines
in various forms of popular culture is perhaps understandable, as is
the tendency among cultural critics to focus on the animation and
meaningfulness of commodities rather than the eclipse of laboring
activity that Marx identifies as the source of their fetishization
(Marx 1976, 164-65). The preference for a level of abstraction that
tends not to register either the qualitative dimensions or the
hierarchical relations of work can also account for its relative
neglect in the field of mainstream economics. But the lack of
attention to the lived experiences and political textures of work
within political theory would seem to be another matter. Indeed,
political theorists tend to be more interested in our lives as
citizens and noncitizens, legal subjects and bearers of rights,
consumers and spectators, religious devotees and family members, than
in our daily lives as workers.
This is only a quarter of a paragraph, and the entire book is written like
I don't mind the occasional use of longer words for their precise meanings
("qualitative," "hierarchical") and can tolerate the academic habit of
inserting mostly unnecessary citations. I have less patience with the
meandering and complex sentences, excessive hedge words ("perhaps," "seem
to be," "tend to be"), unnecessarily indirect phrasing ("can also account
for" instead of "explains"), or obscure terms that are unnecessary to the
sentence (what is "animation of commodities"?). And please have mercy and
throw a reader some paragraph breaks.
The writing style means substantial unnecessary effort for the reader,
which is why it took me six months to read this book. It stalled all of
my non-work non-fiction reading and I'm not sure it was worth the effort.
That's unfortunate, because there were several important ideas in here
that were new to me.
The first was the overview of the "wages for housework" movement, which I
had not previously heard of. It started from the common feminist position
that traditional "women's work" is undervalued and advocated taking the
next logical step of giving it equality with paid work by making it
paid work. This was not successful, obviously, although the increasing
prevalence of day care and cleaning services has made it partly true
within certain economic classes in an odd and more capitalist way. While
I, like Weeks, am dubious this was the right remedy, the observation
that household work is essential to support capitalist activity but is
unmeasured by GDP and often uncompensated both economically and socially
has only become more accurate since the 1970s.
Weeks argues that the usefulness of this movement should not be judged by
its lack of success in achieving its demands, which leads to the second
interesting point: the role of utopian demands in reframing and expanding
a discussion. I normally judge a political demand on its effectiveness at
convincing others to grant that demand, by which standard many activist
campaigns (such as wages for housework) are unsuccessful. Weeks points
out that making a utopian demand changes the way the person making the
demand perceives the world, and this can have value even if the demand
will never be granted. For example, to demand wages for housework
requires rethinking how work is defined, what activities are compensated
by the economic system, how such wages would be paid, and the implications
for domestic social structures, among other things. That, in turn, helps
in questioning assumptions and understanding more about how existing
society sustains itself.
Similarly, even if a utopian demand is never granted by society at large,
forcing it to be rebutted can produce the same movement in thinking in
others. In order to rebut a demand, one has to take it seriously and
mount a defense of the premises that would allow one to rebut it. That
can open a path to discussing and questioning those premises, which can
have long-term persuasive power apart from the specific utopian demand.
It's a similar concept as the Overton Window, but with more nuance: the
idea isn't solely to move the perceived range of accepted discussion, but
to force society to examine its assumptions and premises well enough to
defend them, or possibly discover they're harder to defend than one might
Weeks applies this principle to universal basic income, as a utopian
demand that questions the premise that work should be central to personal
identity. I kept thinking of the Black Lives Matter movement and the
demand to abolish the police, which (at least in popular discussion) is a
more recent example than this book but follows many of the same
principles. The demand itself is unlikely to be met, but to rebut it
requires defending the existence and nature of the police. That in turn
leads to questions about the effectiveness of policing, such as clearance
rates (which are far lower than one might have assumed). Many more
examples came to mind. I've had that experience of discovering problems
with my assumptions I'd never considered when debating others, but had not
previously linked it with the merits of making demands that may be
The book closes with an interesting discussion of the types of utopias,
starting from the closed utopia in the style of Thomas More in which the
author sets up an ideal society. Weeks points out that this sort of
utopia tends to collapse with the first impossibility or inconsistency the
reader notices. The next step is utopias that acknowledge their own
limitations and problems, which are more engaging (she cites Le Guin's
The Dispossessed). More conditional
than that is the utopian manifesto, which only addresses part of society.
The least comprehensive and the most open is the utopian demand, such as
wages for housework or universal basic income, which asks for a specific
piece of utopia while intentionally leaving unspecified the rest of the
society that could achieve it. The demand leaves room to maneuver; one
can discuss possible improvements to society that would approach that
utopian goal without committing to a single approach.
I wish this book were better-written and easier to read, since as it
stands I can't recommend it. There were large sections that I read but
didn't have the mental energy to fully decipher or retain, such as the
extended discussion of Ernst Bloch and Friedrich Nietzsche in the context
of utopias. But that way of thinking about utopian demands and their
merits for both the people making them and for those rebutting them, even
if they're not politically feasible, will stick with me.
Rating: 5 out of 10
Release 0.6.28 of the digest package arrived at CRAN earlier today, and has already been uploaded Debian as well.
digest creates hash digests of arbitrary R objects (using the md5, sha-1, sha-256, sha-512, crc32, xxhash32, xxhash64, murmur32, spookyhash, and blake3 algorithms) permitting easy comparison of R language objects. It is a mature and widely-used as many tasks may involve caching of objects for which it provides convenient general-purpose hash key generation.
This release comes eleven months after the previous releases and rounds out a number of corners. Continuous Integration was updated using r-ci. Several contribututors help with a small fix applied to avoid unaligned reads, a rewording for a help page as well as windows path encoding for in the vectorised use case.
My CRANberries provides the usual summary of changes to the previous version. For questions or comments use the issue tracker off the GitHub repo.
If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.
Colson Whitehead's latest novel, Harlem Shuffle, was always going to be widely reviewed, if only because his last two books won Pulitzer prizes. Still, after enjoying both The Underground Railroad and The Nickel Boys, I was certainly going to read his next book, regardless of what the critics were saying indeed, it was actually quite agreeable to float above the manufactured energy of the book's launch.
Saying that, I was encouraged to listen to an interview with the author by Ezra Klein. Now I had heard Whitehead speak once before when he accepted the Orwell Prize in 2020, and once again he came across as a pretty down-to-earth guy. Or if I were to emulate the detached and cynical tone Whitehead embodied in The Nickel Boys, after winning so many literary prizes in the past few years, he has clearly rehearsed how to respond to the cliched questions authors must be asked in every interview. With the obligatory throat-clearing of 'so, how did you get into writing?', for instance, Whitehead replies with his part of the catechism that 'It seemed like being a writer could be a cool job. You could work from home and not talk to people.' The response is the right combination of cute and self-effacing... and with its slight tone-deafness towards enforced isolation, it was no doubt honed before Covid-19.
Harlem Shuffle tells three separate stories about Ray Carney, a furniture salesman and 'fence' for stolen goods in New York in the 1960s. Carney doesn't consider himself a genuine criminal though, and there's a certain logic to his relativistic morality. After all, everyone in New York City is on the take in some way, and if some 'lightly used items' in Carney's shop happened to have had 'previous owners', well, that's not quite his problem. 'Nothing solid in the city but the bedrock,' as one character dryly observes. Yet as Ezra pounces on in his NYT interview mentioned abov, the focus on the Harlem underworld means there are very few women in the book, and Whitehead's circular response ah well, it's a book about the criminals at that time! was a little unsatisfying. Not only did it feel uncharacteristically slippery of someone justly lauded for his unflinching power of observation (after all, it was the author who decided what to write about in the first place), it foreclosed on the opportunity to delve into why the heist and caper genres (from The Killing, The Feather Thief, Ocean's 11, etc.) have historically been a 'male' mode of storytelling.
Perhaps knowing this to be the case, the conversation quickly steered towards Ray Carney's wife, Elizabeth, the only woman in the book who could be said possesses some plausible interiority. The following off-hand remark from Whitehead caught my attention:
My wife is convinced that [Elizabeth] knows everything about Carney's criminal life, and is sort of giving him a pass. And I'm not sure if that's true. I have to have to figure out exactly what she knows and when she knows it and how she feels about it.
I was quite taken by this, although not simply due to its effect on the story it self. As in, it immediately conjured up a charming picture of Whitehead's domestic arrangements: not only does Whitehead's wife feel free to disagree with what one of Whitehead's 'own' characters knows or believes, but that Colson has no problem whatsoever sharing that disagreement with the public at large. (It feels somehow natural that Whitehead's wife believes her counterpart knows more than she lets on, whilst Whitehead himself imbues the protagonist's wife with a kind of neo-Victorian innocence.) I'm minded to agree with Whitehead's partner myself, if only due to the passages where Elizabeth is studiously ignoring Carney's otherwise unexplained freak-outs.
But all of these meta-thoughts simply underline just how emancipatory the Death of the Author can be. This product of academic literary criticism (the term was coined by Roland Barthes' 1967 essay of the same name) holds that the original author's intentions, ideas or biographical background carry no especial weight in determining how others should interpret their work. It is usually understood as meaning that a writer's own views are no more valid or 'correct' than the views held by someone else. (As an aside, I've found that most readers who encounter this concept for the first time have been reading books in this way since they were young. But the opposite is invariably true with cinephiles, who often have a bizarre obsession with researching or deciphering the 'true' interpretation of a film.) And with all that in mind, can you think of a more wry example of how freeing (and fun) nature of the Death of the Author than an author's own partner dissenting with their (Pulitzer Prize-winning) husband on the position of a lynchpin character?
As it turns out, the reviews for Harlem Shuffle have been almost universally positive, and after reading it in the two days after its release, I would certainly agree it is an above-average book. But it didn't quite take hold of me in the way that The Underground Railroad or The Nickel Boys did, especially the later chapters of The Nickel Boys that were set in contemporary New York and could thus make some (admittedly fairly explicit) connections from the 1960s to the present day that kind of connection is not there in Harlem Shuffle, or at least I did not pick up on it during my reading.
I can see why one might take exception to that, though. For instance, it is certainly true that the week-long Harlem Riot forms a significant part of the plot, and some events in particular are entirely contingent on the ramifications of this momentous event. But it's difficult to argue the riot's impact are truly integral to the story, so not only is this uprising against police brutality almost regarded as a background event, any contemporary allusion to the murder of George Floyd is subsequently watered down. It's nowhere near the historical rubbernecking of Forrest Gump (1994), of course, but that's not a battle you should ever be fighting.
Indeed, whilst a certain smoothness of affect is to be priced into the Whitehead reading experience, my initial overall reaction to Harlem Shuffle was fairly flat, despite all the action and intrigue on the page. The book perhaps belies its origins as a work conceived during quarantine after all, the book is essentially comprised of three loosely connected novellas, almost as if the unreality and mental turbulence of lockdown prevented the author from performing the psychological 'deep work' of producing a novel-length text with his usual depth of craft. A few other elements chimed with this being a 'lockdown novel' as well, particularly the book's preoccupation with the sheer physicality of the city compared to the usual complex interplay between its architecture and its inhabitants. This felt like it had been directly absorbed into the book from the author walking around his deserted city, and thus being able to take in details for the first time:
The doorways were entrances into different cities no, different entrances into one vast, secret city. Ever close, adjacent to all you know, just underneath. If you know where to look.
And I can't fail to mention that you can almost touch Whitehead's sublimated hunger to eat out again as well:
Stickups were chops they cook fast and hot, you re in and out. A stakeout was ribs fire down low, slow, taking your time.
Sometimes when Carney jumped into the Hudson when he was a kid, some of that stuff got into his mouth. The Big Apple Diner served it up and called it coffee.
More seriously, however, the relatively thin personalities of minor characters then reminded me of the simulacrum of Zoom-based relationships, and the essentially unsatisfactory endings to the novellas felt reminiscent of lockdown pseudo-events that simply fizzle out without a bang. One of the stories ties up loose ends with: 'These things were usually enough to terminate a mob war, and they appeared to end the hostilities in this case as well.' They did? Well, okay, I guess.
Still, it would be unfair to characterise myself as 'disappointed' with the novel, and none of this piece should be taken as really deep criticism. The book certainly was entertaining enough, and pretty funny in places as well:
Carney didn t have an etiquette book in front of him, but he was sure it was bad manners to sit on a man s safe.
The manager of the laundromat was a scrawny man in a saggy undershirt painted with sweat stains. Launderer, heal thyself.
Yet I can't shake the feeling that every book you write is a book that you don't, and so we might need to hold out a little longer for Whitehead's 'George Floyd novel'. (Although it is for others to say how much of this sentiment is the expectations of a White Reader for The Black Author to ventriloquise the pain of 'their' community.)
Some room for personal critique is surely permitted. I dearly missed the junk food energy of the dry and acerbic observations that run through Whitehead's previous work. At one point he had a good line on the model tokenisation that lurks behind 'The First Negro to...' labels, but the callbacks to this idea ceased without any payoff. Similar things happened with the not-so-subtle critiques of the American Dream:
Entrepreneur? Pepper said the last part like manure. That s just a
hustler who pays taxes.
One thing I ve learned in my job is that life is cheap, and when things start getting expensive, it gets cheaper still.
Ultimately, though, I think I just wanted more. I wanted a deeper exploration of how the real power in New York is not wielded by individual street hoodlums or even the cops but in the form of real estate, essentially serving as a synecdoche for Capital as a whole. (A recent take of this can be felt in Jed Rothstein's 2021 documentary, WeWork: Or the Making and Breaking of a $47 Billion Unicorn and it is perhaps pertinent to remember that the US President at the time this novel was written was affecting to be a real estate tycoon.). Indeed, just like the concluding scenes of J. J. Connolly's Layer Cake, although you can certainly pull off a cool heist against the Man, power ultimately resides in those who control the means of production... and a homespun furniture salesman on the corner of 125 & Morningside just ain't that. There are some nods to kind of analysis in the conclusion of the final story ('Their heist unwound as if it had never happened, and Van Wyck kept throwing up buildings.'), but, again, I would have simply liked more.
And when I attempted then file this book away into the broader media landscape, given the current cultural visibility of 1960s pop culture (e.g. One Night in Miami (2020), Judas and the Black Messiah (2021), Summer of Soul (2021), etc.), Harlem Shuffle also seemed like a missed opportunity to critically analyse our (highly-qualified) longing for the civil rights era. I can certainly understand why we might look fondly on the cultural products from a period when politics was less alienated, when society was less atomised, and when it was still possible to imagine meaningful change, but in this dimension at least, Harlem Shuffle seems to merely contribute to this nostalgic escapism.
Some time ago I looked briefly at an Envertech data logger
for small scale photovoltaic setups. Turned out that PV inverter are kinda
unreliable, and you really have to monitor them to notice downtimes and defects.
Since my pal shot for a quick win I've cobbled together another Python script
to query the portal at
www.envertecportal.com, and report
back if the generated power is down to 0. The script is currently run on a vserver
via cron and reports back via the system MTA. So yeah, you need to have something
like that already at hand.
Script and Configuration
You've to provide your PV systems location with latitude and longitude so the
script can calculate
the sunrise and sunset times. At the location we deal with we expect to generate
some power at least from sunrise + 1h to sunet - 1h. That is tunable via the
configuration option toleranceSeconds.
Retrieving the stationId is a bit ugly because it's not provided via any API,
instead it's rendered serverside into the website. So I just logged in on the
portal and picked it up by looking into the page source.
I guess this is some classic in the IoT land, but neither the documentation
provided on the portal frontpage as docx, nor the API docs at port 8090 are complete and correct. The few bits I gathered via the
Firefox Web Developer Tools are:
Login https://www.envertecportal.com/apiaccount/login - POST, sent userName and
pwd containing your login name and password. The response JSON is very explicit if
your login was not successful and why.
Store the session cookie called ASP.NET_SessionId for use on all subsequent
Retrieve station info https://www.envertecportal.com/ApiStations/getStationInfo -
POST, sent ASP.NET_SessionId and stationId with the ID of the station. Returns
a JSON with an object named Data. The field Power contains the currently
generated power as a float with two digits (e.g. 0.01).
Logout https://www.envertecportal.com/apiAccount/Logout - POST, sent
There were a few surprises, maybe they help others dealing with an Envertech setup.
The portal truncates passwords at 16 chars.
The "Forget Password?" function mails you back the password in plain text
(that's how I learned about 1.).
The login API endpoint reporting the exact reason why the login failed is somewhat
out of fashion. Though this one is probably not a credential stuffing target because
there is no money to make, so don't care.
The data logger reports the data to www.envertecportal.com at port 10013.
There is some checksuming done on the reported data, but the system is not replay
safe. So you can sent it any valid data string at a later time and get wrong data
People at forum.fhem.de
decoded some values but could not figure out the checksuming so far.
This is the fifth and final post in a series about the interface description language Candid.
If you made it this far, you now have a good understanding of what Candid is, what it is for and how it is used. For this final post, I ll put the spotlight on specific aspects of Candid that are maybe surprising, or odd, or quirky. This section will be quite opinionated, and could maybe be called what I d do differently if I d re-do the whole thing .
Note that these quirks are not serious problems, and they don t invalidate the overall design. I am writing this up not to discourage the use of Candid, but merely help interested parties to understand it better.
References in the wire format
When the work on Candid began at DFINITY, the Internet Computer was still far away from being a thing, and many fundamental aspects about it were still in the air. I particular, there was still talk about embracing capabilities as a core feature of the application model, which would be implemented as opaque references on the system level, likely building on WebAssembly s host reference type proposal (which only landed recently), and could be used to model access permissions, custom tokens and many other things.
So Candid is designed with that in mind, and you ll find that its wire format is not just a type table and a value table, but actually
a triple (T, M, R), where T ( type ) and M ( memory ) are sequences of bytes and R ( references ) is a sequence of references.
Also the wire format for values of function service tyeps have an extra byte to distinguish between public references (represented by a principal and possible a method name in the data part), and these opaque references.
Alas, references never made it into the Internet Computer, so all Candid implementations simply ignore that part of the specification. But it s still in the spec, and if it confused you before, now you know why.
Hashed field names
Candid record and variant types look like they have textual field names:
type T = record syndactyle : nat; trustbuster: bool
But that is actually only true superficially. The wire format for Candid only stores hashes of field names. So the above is actually equivalent to
type T = record 4260381820 : nat; 3504418361 : bool
or, for that matter, to
type T = record destroys : bool; rectum : nat
(Yes, I used an english word list to find these hash collisions. There aren t that many actually.)
The benefit of such hashing is that the messages are a bit smaller in most (not all) cases, but it is a big annoyance for dynamic uses of Candid. It s the reason why tools like dfx, if they don t know the Candid interface of a service, will print the result with just the numerical hash, letting you guess which field is which.
It also complicates host languages that derive Candid types from the host language, like Motoko, as some records (e.g. record trustbuster: bool; destroys : int ) with field name hash collisions can not be represented in Candid, and either the host language s type system needs to be Candid aware now (as is the case of Motoko), or serialization/deserialization will fail at runtime, or odd bugs can happen.
(More discussion of this issue).
Many languages have a built-in notion of a tuple type (e.g. (Int, Bool)), but Candid does not have such a type. The only first class product type is records.
This means that tuples have to encoded as records somehow. Conveniently(?) record fields are just numbers after all, so the type (Int, Bool) would be mapped to the type
record 0 : int; 1 : bool
So tuples can be expressed. But from my experience implementing the type mappings for Motoko and Haskell this is causing headaches. To get a good experience when importing from Candid, the tools have to try to reconstruct which records may have been tuples originally, and turn them into tuples.
The main argument for the status quo is that Candid types should be canonical, and there should not be more than one product type, and records are fine, and there needs to be no anonymous product type. But that has never quite resonated with me, given the practical reality of tuple types in many host languages.
Did I say that Candid does not have tuple types? Turns out it does, sort of. There is no first class anonymous product, but since functions take sequences of arguments and results, there is a tuple type right there:
func foo : (bool, int) -> (int, bool)
Again, I found that ergonomic interaction with host languages becomes relatively unwieldy by requiring functions to take and return sequences of values. This is especially true for languages where functions take one argument value or return one result type (the latter being very common). Here, return sequences of length one are turned into that type directly, longer argument sequences turn into the host language s tuple type, and nullary argument sequences turn into the idiomatic unit type. But this means that the types (int, bool) -> () and (record 0: int, 1: bool ) -> () may be mapped to the same host language type, which causes problems when you hope to encode all necessary Candid type information in the host language.
Another oddity with argument and result sequences is that you can give names to the entries, e.g. write
without breaking clients, this does not have the effect you think it has and will likely break.
My suggestion is to never put names on function arguments and result values in Candid interfaces, and for anything that might be extended with new fields or where you want to name the arguments, use a single record type as the only argument:
This allows you to add and remove arguments more easily and reliably.
The Candid specification defines a system of types, and then adds a number of syntactic short-hands . For example, if you write blob in a Candid type description, it ought to means the same as vec nat8.
My qualm with that is that it doesn t always mean the same. A Candid type description is interpreted by a number of, say, consumers . Two such consumers are part of the Candid specification:
The specification that defines the wire format for that type
The upgrading (subtyping) rules
But there are more! For every host language, there is some mapping from Candid types to host language types, and also generic tools like Candid UI are consumers of the type algebra. If these were to take the Candid specification as gospel, they would be forced to treat blob and vec nat8 the same, but that would be quite unergonomic and might cause performance regressions (most language try to map blob to some compact binary data type, while vec t tends to turn into some form of array structure).
So they need to be pragmatic and treat blob and vec nat8 differently. But then, for all practical purposes, blob is not just a short-hand of vec nat8. They are different types that just happens to have the same wire representations and subtyping relations.
This affects not just blob, but also tuples (record blob; int; bool ) and field names , as discussed above.
The value text format
For a while after defining Candid, the only implementation was in Motoko, and all the plumbing was automatic, so there was never a need for users to to explicitly handle Candid values, as all values were Motoko values. Still, for debugging and testing and such things, we eventually needed a way to print out Candid values, so the text format was defined ( To enable convenient debugging, the following grammar specifies a text format for values ).
But soon the dfx tool learned to talk to canisters, and now users needed to enter Candid value on the command line, possibly even when talking to canisters for which the interface was not known to dfx. And, sure enough, the textual interface defined in the Candid spec was used.
Unfortunately, since it was not designed for that use case, it is rather unwieldy:
It is quite verbose. You have to write record , not just . Vectors are written vec ; instead of some conventional syntax like [ , ]. Variants are written as variant error = " " with braces that don t any value here, and something like #error " " might have worked as well.
With a bit more care, a more concise and ergonomic syntax might have been possible.
It wasn t designed to be sufficient to create a Candid value from it. If you write 5 it s unclear whether that s a nat or an int16 or what (and all of these have different wire representations). Type annotations were later added, but are relatively unwieldy, and don t cover all cases (e.g. a service reference with a recursive type cannot be represented in the textual format at the moment).
Not really the fault of the textual format, but some useful information about the types is not reflected in the type description that s part of the wire format. In particular not the field names, and whether a value was intended to be binary data (blob) or a list of small numbers (vec nat8), so pretty-printing such values requires guesswork. The Haskell library even tries to brute-force the hash to guess the field name, if it is short or in a english word list!
In hindsight I think it was too optimistic to assume that correct static type information is always available, and instead of actively trying to discourage dynamic use, Candid might be better if we had taken these (unavoidable?) use cases into account.
Custom wire format
At the beginning of this post, I have a Candid is list. The list is relatively long, and the actual wire format is just one bullet point. Yes, defining a wire format that works isn t rocket science, and it was easiest to just make one up. But since most of the interesting meat of Candid is in other aspects (subtyping rules, host language integration), I wonder if it would have been better to use an existing, generic wire format, such as CBOR, and build Candid as a layer on top.
This would give us plenty of tools and libraries to begin with. And maybe it would have reduced barrier of entry for developers, which now led to the very strange situation that DFINITY advocates for the universal use of Candid on the Internet Computer, so that all services can smoothly interact, but two of the most important services on the Internet Computer (the registry and the ledger) use Protobuf as their primary interface format, with Candid interfaces missing or an afterthought.
Sideways Interface Evolution
This is not a quirk of Candid itself, but rather an idiom of how you can use Candid that emerged from our solution for record extensions and upgrades.
Consider our example from before, a service with interface
service add_user : (record login : text; name : text ) -> ()
where you want to add an age field, which should be a number.
The official way of doing that is to add that field with an optional type:
service add_user : (record login : text; name : text; age : opt nat ) -> ()
As explained above, this will not break old clients, as the decoder will treat a missing argument as null. So far so good.
But often when adding such a field you don t want to bother new clients with the fact that this age was, at some point in the past, not there yet. And you can do that! The trick is to distinguish between the interface you publish and the interface you implement. You can (for example in your documentation) state that the interface is
service add_user : (record login : text; name : text; age : nat ) -> ()
which is not a subtype of the old type, but it is the interface you want new clients to work with. And then your implementation uses the type with opt nat. Calls from old clients will come through as null, and calls from new clients will come through as opt 42.
We can see this idiom used in the Management Canister of the Internet Computer. The current documented interface only mentions a controllers : vec principal field in the settings, but the implementation still can handle both the old controller : principal and the new controllers field.
It s probably advisable to let your CI system check that new versions of your service continue to implement all published interfaces, including past interfaces. But as long as the actual implementation s interface is a subtype of all interfaces ever published, this works fine.
This pattern is related to when your service implements, say, http_request (so its implemented interface is a subtype of that common interface), but does not include that method in the published documentation (because clients of your service don t need to call it).
As you have noticed, Candid was very much designed assuming that all parties always have the service type of services they want to interact with. But the Candid specification does not define how one can obtain the interface of a given service, and there isn t really a an official way to do that on the Internet Computer.
That is unfortunate, because many interesting features depend on that: Such as writing import C "ic:7e6iv-biaaa-aaaaf-aaada-cai" in your Motoko program, and having it s type right there. Or tools like ic.rocks, that allow you to interact with any canister right there.
One reason why we don t really have that feature yet is because of disagreements about how dynamic that feature should be. Should you be able to just ask the canister for its interface (and allow the canister to vary the response, for example if it can change its functionality over time, even without changing the actual wasm code)? Or is the interface a static property of the code, and one should be able to query the system for that data, without the canister s active involvement. Or, completely different, should interfaces be distributed out of band, maybe together with API documentation, or in some canister registry somewhere else?
I always leaned towards the first of these options, but not convincingly enough. The second options requires system assistance, so more components to change, more teams to be involved that maybe intrinsically don t care a lot about this feature. And the third might have emerged as the community matures and builds such infrastructure, but that did not happen yet.
In the end I sneaked in an implementation of the first into Motoko, arguing that even if we don t know yet how this feature will be implemented eventually, we all want the feature to exist somehow, and we really really want to unblock all the interesting applications it enables (e.g. Candid UI). That s why every Motoko canister, and some rust canisters too, implements a method
__get_candid_interface_tmp_hack : () -> (text)
that one can use to get the Candid interface file.
The name was chosen to signal that this may not be the final interface, but like all good provisional solutions, it may last longer than intended. If that s the case, I m not sorry.
This concludes my blog post series about Candid, for now. If you want to know more, feel free to post your question on the DFINTY developer forum, and I ll probably answer.
The Glasgow Haskell Compiler (GHC) has support for user-supplied rewrite rules,
which applied during one of the compiler optimisation stages. An example rule is
"streamFilter fuse" forall f g xs.
streamFilter f (streamFilter g xs) = streamFilter (f.g) xs
I spent some time today looking at these more closely.
In order for rewrite rules to be applied, optimisation needs to be enabled. This
conflicts with interactive use of GHC, so you can't explore these things in GHCi.
I think rewrite rules are enabled by default (with optimisation), but you can ask
for them explicitly. When investigating these it's also useful to ask ghc to
always recompile, otherwise you have to change the source or manually remove .hi
or .o files (etc.) between invocations.
A starting set of command-line options to use then, are
-O -fenable-rewrite-rules -fforce-recomp
GHC runs several compilation stages, and the source program is transformed into
several different languages or language dialects as it goes. Before the phase
where rewrite rules are applied, some other optimisations take place, and the
source gets desugared. You can see the results of the desugaring by passing the
argument -ddump-ds. Here's an example program
main = print (unStream (streamFilter (>3) (streamFilter (<10)
And here's what it looks like after the first pass optimisation and desugaring:
(Note: I used -dsuppress-all and -dsuppress-uniques to improve the clarity
of the above output. See Suppressing unwanted
information for further details).
Those short-hand sections ((<3)) in the input program are expanded to
something quite a lot longer. Out of curiosity I tried it again with plain
lambdas, not sections, and the result was smaller
(\ x -> > $fOrdInteger x 3)
(\ x -> < $fOrdInteger x 10)
(mkStream (enumFromTo $fEnumInteger 0 20)))))
Rewrite rules happen after this. Once they're done (and several other passes),
the program is translated into an intermediate representation called Tiny
Core. This language
faintly resembles the input Haskell. GHC will output the Tiny Core program
if you supply the argument -ddump-simpl. Here's (most) of the program in Tiny
Core (I've substituted some constants for clarity):
main = hPutStr' stdout main1 True
main1 = $fShowInteger_$cshowList (catMaybes1 (main_go 0)) 
= \ x ->
case gtInteger# x 20 of
case ltInteger# x 10 of
DEFAULT -> main_go (plusInteger x 1);
case gtInteger# x 3 of
DEFAULT -> main_go (plusInteger x 1);
1# -> : (Just x) (main_go (plusInteger x 1))
1# -> 
After Tiny Core, GHC continues to translate the program into other forms
(including STG, CMM, ASM) and you can ask GHC to dump those representations too,
but this is all downstream from the rewrites so not relevant to them.
The rewrite rule at the top of this blog post is never applied: It doesn't get
a chance, because the function it operates on (streamFilter) is inlined by
GHC. GHC can detect this in some circumstances (-Winline-rule-shadowing). You
instruct GHC to report on which rules fired with -ddump-rule-firings and can
see before-and-after snippets of Tiny Core for each rule applied with
I played around with adding -# NOINLINE functionName #- pragmas to disable
inlining various functions to try and provoke a situation where the above rule
could match, but it was a losing battle: GHC's built-in optimisations are just
too good. But, it's also moot: the outcome I want (the filters to be fused) is
happening, it's just the built-in rewrite rules are achieving it,
once striot's functions
have been inlined away.
Cisco pyATS is a framework for network automation and testing. It
includes, among other things, an open-source multi-vendor set of
parsers and models, Genie Parser. It features 2700 parsers for
various commands over many network OS. On the paper, this seems a
>>> fromgenie.conf.baseimportDevice>>> device=Device("router",os="iosxr")>>> # Hack to parse outputs without connecting to a device>>> device.custom.setdefault("abstraction", )["order"]=["os","platform"]>>> cmd="show route ipv4 unicast">>> output="""... Tue Oct 29 21:29:10.924 UTC...... O 10.13.110.0/24 [110/2] via 10.12.110.1, 5d23h, GigabitEthernet0/0/0/0.110... """>>> device.parse(cmd,output=output) 'vrf': 'default': 'address_family': 'ipv4': 'routes': '10.13.110.0/24': 'route': '10.13.110.0/24', 'active': True, 'route_preference': 110, 'metric': 2, 'source_protocol': 'ospf', 'source_protocol_codes': 'O', 'next_hop': 'next_hop_list': 1: 'index': 1, 'next_hop': '10.12.110.1', 'outgoing_interface': 'GigabitEthernet0/0/0/0.110', 'updated': '5d23h'
The data models used are dependent on the vendor and OS,
despite the documentation saying otherwise. For
example, the data model used for IPv4 interfaces is different
between NX-OS and IOS-XR.
The parsers rely on line-by-line regular expressions to extract
data and some Python code as glue. This is fragile and may break
To illustrate the second point, let s assume the output of show ipv4
vrf all interface is:
Loopback10 is Up, ipv4 protocol is Up
Vrf is default (vrfid 0x60000000)
Internet protocol processing disabled
Loopback30 is Up, ipv4 protocol is Down [VRF in FWD reference]
Vrf is ran (vrfid 0x0)
Internet address is 203.0.113.17/32
MTU is 1500 (1500 is available to IP)
Helper address is not set
Directed broadcast forwarding is disabled
Outgoing access list is not set
Inbound common access list is not set, access list is not set
Proxy ARP is disabled
ICMP redirects are never sent
ICMP unreachables are always sent
ICMP mask replies are never sent
Table Id is 0x0
While this bug is simple to fix, this is an uphill battle. Any
existing or future slight variation in the output of a command could
trigger another similar undetected bug, despite the extended test
coverage. I have reported and fixed several other silent parsing
errors: #516, #529, and #530. A more robust alternative
would have been to use TextFSM and to trigger a warning when some
output is not recognized, like Batfish, a configuration analysis
In the future, we should rely on YANG for data extraction, but it
is currently not widely supported. SNMP is still a valid possibility
but much information is not accessible through this protocol. In the
meantime, I would advise you to only use Genie Parser with caution.
As an exercise, the astute reader is asked to write the code
to extract the IPv4 from this structure.
On Saturday 28 August 2021, the annual Debian Developers
and Contributors Conference came to a close.
DebConf21 has been held online for the second time, due to the coronavirus
(COVID-19) disease pandemic.
All of the sessions have been streamed, with a variety of ways of participating:
via IRC messaging, online collaborative text documents,
and video conferencing meeting rooms.
With 740 registered attendees from more than 15 different countries and a
total of over 70 event talks, discussion sessions,
Birds of a Feather (BoF) gatherings and other activities,
DebConf21 was a large success.
The setup made for former online events involving Jitsi, OBS,
Voctomix, SReview, nginx, Etherpad, a web-based
frontend for voctomix has been improved and used for DebConf21 successfully.
All components of the video infrastructure are free software, and configured through the
Video Team's public ansible repository.
The DebConf21 schedule included
a wide variety of events, grouped in several tracks:
Introduction to Free Software and Debian,
Packaging, policy, and Debian infrastructure,
Systems administration, automation and orchestration,
Cloud and containers,
Community, diversity, local outreach and social context,
Internationalization, Localization and Accessibility,
Embedded and Kernel,
Debian Blends and Debian derived distributions,
Debian in Arts and Science
The talks have been streamed using two rooms, and several of these activities
have been held in different languages: Telugu, Portuguese, Malayalam, Kannada,
Hindi, Marathi and English, allowing a more diverse audience to enjoy and participate.
Between talks, the video stream has been showing the usual sponsors on the loop, but also
some additional clips including photos from previous DebConfs, fun facts about Debian
and short shout-out videos sent by attendees to communicate with their Debian friends.
The Debian publicity team did the usual live coverage to encourage participation
with micronews announcing the different events. The DebConf team also provided several
mobile options to follow the schedule.
For those who were not able to participate, most of the talks and sessions are already
available through the
Debian meetings archive website,
and the remaining ones will appear in the following days.
The DebConf21 website
will remain active for archival purposes and will continue to offer
links to the presentations and videos of talks and events.
Next year, DebConf22 is planned to be held
in Prizren, Kosovo, in July 2022.
DebConf is committed to a safe and welcome environment for all participants.
During the conference, several teams (Front Desk, Welcome team and Community team)
have been available to help so participants get their best experience
in the conference, and find solutions to any issue that may arise.
See the web page about the Code of Conduct in DebConf21 website
for more details on this.
Debian thanks the commitment of numerous sponsors
to support DebConf21, particularly our Platinum Sponsors:
Amazon Web Services (AWS)
The Debian Project was founded in 1993 by Ian Murdock to be a truly
free community project. Since then the project has grown to be one of
the largest and most influential open source projects. Thousands of
volunteers from all over the world work together to create and
maintain Debian software. Available in 70 languages, and
supporting a huge range of computer types, Debian calls itself the
universal operating system.
DebConf is the Debian Project's developer conference. In addition to a
full schedule of technical, social and policy talks, DebConf provides an
opportunity for developers, contributors and other interested people to
meet in person and work together more closely. It has taken place
annually since 2000 in locations as varied as Scotland, Argentina, and
Bosnia and Herzegovina. More information about DebConf is available from
As a global technology leader manufacturing a wide portfolio of connected products,
including smartphones, tablets, PCs and workstations as well as AR/VR devices,
smart home/office and data center solutions, Lenovo
understands how critical open systems and platforms are to a connected world.
Infomaniak is Switzerland's largest web-hosting company,
also offering backup and storage services, solutions for event organizers,
live-streaming and video on demand services.
It wholly owns its datacenters and all elements critical
to the functioning of the services and products provided by the company
(both software and hardware).
Roche is a major international
pharmaceutical provider and research company dedicated to personalized
healthcare. More than 100.000 employees worldwide work towards solving some
of the greatest challenges for humanity using science and technology. Roche
is strongly involved in publicly funded collaborative research projects with
other industrial and academic partners and have supported DebConf since 2017.
About Amazon Web Services (AWS)
Amazon Web Services (AWS) is one of the world's
most comprehensive and broadly adopted cloud platform,
offering over 175 fully featured services from data centers globally
(in 77 Availability Zones within 24 geographic regions).
AWS customers include the fastest-growing startups, largest enterprises
and leading government agencies.
Google is one of the largest technology companies in the
world, providing a wide range of Internet-related services and products such
as online advertising technologies, search, cloud computing, software, and hardware.
Google has been supporting Debian by sponsoring DebConf for more than
ten years, and is also a Debian partner sponsoring parts
of Salsa's continuous integration infrastructure
within Google Cloud Platform.
For further information, please visit the DebConf21 web page at
or send mail to email@example.com.