Nextcloud is an open-source software suite that
enables you to set up and manage your own cloud storage and collaboration
platform. It offers a range of features similar to popular cloud services
like Google Drive or Dropbox but with the added benefit of complete control
over your data and the server where it s hosted.
I wanted to have a look at Nextcloud and the steps to setup a own instance
with a PostgreSQL based database together with NGinx as the webserver to
serve the WebUI. Before doing a full productive setup I wanted to play
around locally with all the needed steps and worked out all the steps within
KVM machine.
While doing this I wrote down some notes to mostly document for myself what
I need to do to get a Nextcloud installation running and usable. So this
manual describes how to setup a Nextcloud installation on Debian 12 Bookworm
based on NGinx and PostgreSQL.
Nextcloud Installation
Install PHP and PHP extensions for Nextcloud
Nextcloud is basically a PHP application so we need to install PHP packages
to get it working in the end. The following steps are based on the upstream
documentation about how to install a own
Nextcloud instance.
Installing the virtual package package php on a Debian Bookworm system
would pull in the depending meta package php8.2. This package itself would
then pull also the package libapache2-mod-php8.2 as an dependency which
then would pull in also the apache2 webserver as a depending package. This
is something I don t wanted to have as I want to use NGinx that is already
installed on the system instead.
To get this we need to explicitly exclude the package libapache2-mod-php8.2
from the list of packages which we want to install, to achieve this we have
to append a hyphen - at the end of the package name, so we need to use
libapache2-mod-php8.2- within the package list that is telling apt to
ignore this package as an dependency. I ended up with this call to get all
needed dependencies installed.
To make these settings effective, restart the php-fpm service
$ sudo systemctl restart php8.2-fpm
Install PostgreSQL, Create a database and user
This manual assumes we will use a PostgreSQL server on localhost, if you
have a server instance on some remote site you can skip the installation
step here.
$ sudo apt install postgresql postgresql-contrib postgresql-client
Check version after installation (optinal step):
$ sudo -i -u postgres$ psql -version
This output will be seen:
psql (15.12 (Debian 15.12-0+deb12u2))
Exit the PSQL shell by using the command \q.
postgres=# \q
Exit the CLI of the postgres user:
postgres@host:~$ exit
Create a PostgreSQL Database and User:
Create a new PostgreSQL user (Use a strong password!):
$ sudo -u postgres psql -c "CREATE USER nextcloud_user PASSWORD '1234';"
Create new database and grant access:
$ sudo -u postgres psql -c "CREATE DATABASE nextcloud_db WITH OWNER nextcloud_user ENCODING=UTF8;"
(Optional) Check if we now can connect to the database server and the
database in detail (you will get a question about the password for the database
user!). If this is not working it makes no sense to proceed further! We need to
fix first the access then!
$ psql -h localhost -U nextcloud_user -d nextcloud_db
or
$ psql -h 127.0.0.1 -U nextcloud_user -d nextcloud_db
Log out from postgres shell using the command \q.
Download and install Nextcloud
Use the following command to download the latest version of Nextcloud:
$ wget https://download.nextcloud.com/server/releases/latest.zip
Extract file into the folder /var/www/html with the following command:
$ sudo unzip latest.zip -d /var/www/html
Change ownership of the /var/www/html/nextcloud directory to www-data.
$ sudo chown -R www-data:www-data /var/www/html/nextcloud
Configure NGinx for Nextcloud to use a certificate
In case you want to use self signed certificate, e.g. if you play around to
setup Nextcloud locally for testing purposes you can do the following steps.
If you want or need to use the service of Let s Encrypt (or similar) drop
the step above and create your required key data by using this command:
$ sudo certbot --nginx -d nextcloud.your-domain.com
You will need to adjust the path to the key and certificate in the next
step!
Change the NGinx configuration:
$ sudo vi /etc/nginx/sites-available/nextcloud.conf
Add the following snippet into the file and save it.
# /etc/nginx/sites-available/nextcloud.conf
upstreamphp-handler#server 127.0.0.1:9000;
serverunix:/run/php/php8.2-fpm.sock;
# Set the immutable cache control options only for assets with a cache
# busting v argument
map $arg_v $asset_immutable
"""";
default",immutable";
serverlisten80;
listen[::]:80;
# Adjust this to the correct server name!
server_namenextcloud.local;
# Prevent NGinx HTTP Server Detection
server_tokensoff;
# Enforce HTTPS
return301https://$server_name$request_uri;
serverlisten443sslhttp2;
listen[::]:443sslhttp2;
# Adjust this to the correct server name!
server_namenextcloud.local;
# Path to the root of your installation
root/var/www/html/nextcloud;
# Use Mozilla's guidelines for SSL/TLS settings
# https://mozilla.github.io/server-side-tls/ssl-config-generator/
# Adjust the usage and paths of the correct key data! E.g. it you want to use Let's Encrypt key material!
ssl_certificate/etc/ssl/certs/nextcloud.crt;
ssl_certificate_key/etc/ssl/private/nextcloud.key;
# ssl_certificate /etc/letsencrypt/live/nextcloud.your-domain.com/fullchain.pem;
# ssl_certificate_key /etc/letsencrypt/live/nextcloud.your-domain.com/privkey.pem;
# Prevent NGinx HTTP Server Detection
server_tokensoff;
# HSTS settings
# WARNING: Only add the preload option once you read about
# the consequences in https://hstspreload.org/. This option
# will add the domain to a hardcoded list that is shipped
# in all major browsers and getting removed from this list
# could take several months.
#add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload" always;
# set max upload size and increase upload timeout:
client_max_body_size512M;
client_body_timeout300s;
fastcgi_buffers644K;
# Enable gzip but do not remove ETag headers
gzipon;
gzip_varyon;
gzip_comp_level4;
gzip_min_length256;
gzip_proxiedexpiredno-cacheno-storeprivateno_last_modifiedno_etagauth;
gzip_typesapplication/atom+xmltext/javascriptapplication/javascriptapplication/jsonapplication/ld+jsonapplication/manifest+jsonapplication/rss+xmlapplication/vnd.geo+jsonapplication/vnd.ms-fontobjectapplication/wasmapplication/x-font-ttfapplication/x-web-app-manifest+jsonapplication/xhtml+xmlapplication/xmlfont/opentypeimage/bmpimage/svg+xmlimage/x-icontext/cache-manifesttext/csstext/plaintext/vcardtext/vnd.rim.location.xloctext/vtttext/x-componenttext/x-cross-domain-policy;
# Pagespeed is not supported by Nextcloud, so if your server is built
# with the ngx_pagespeed module, uncomment this line to disable it.
#pagespeed off;
# The settings allows you to optimize the HTTP2 bandwidth.
# See https://blog.cloudflare.com/delivering-http-2-upload-speed-improvements/
# for tuning hints
client_body_buffer_size512k;
# HTTP response headers borrowed from Nextcloud .htaccess
add_headerReferrer-Policy"no-referrer"always;
add_headerX-Content-Type-Options"nosniff"always;
add_headerX-Frame-Options"SAMEORIGIN"always;
add_headerX-Permitted-Cross-Domain-Policies"none"always;
add_headerX-Robots-Tag"noindex,nofollow"always;
add_headerX-XSS-Protection"1; mode=block"always;
# Remove X-Powered-By, which is an information leak
fastcgi_hide_headerX-Powered-By;
# Set .mjs and .wasm MIME types
# Either include it in the default mime.types list
# and include that list explicitly or add the file extension
# only for Nextcloud like below:
includemime.types;
typestext/javascriptjsmjs;
application/wasmwasm;
# Specify how to handle directories -- specifying /index.php$request_uri
# here as the fallback means that NGinx always exhibits the desired behaviour
# when a client requests a path that corresponds to a directory that exists
# on the server. In particular, if that directory contains an index.php file,
# that file is correctly served; if it doesn't, then the request is passed to
# the front-end controller. This consistent behaviour means that we don't need
# to specify custom rules for certain paths (e.g. images and other assets,
# /updater , /ocs-provider ), and thus
# try_files $uri $uri/ /index.php$request_uri
# always provides the desired behaviour.
indexindex.phpindex.html/index.php$request_uri;
# Rule borrowed from .htaccess to handle Microsoft DAV clients
location = /if( $http_user_agent ~ ^DavClnt)return302/remote.php/webdav/$is_args$args;
location = /robots.txtallowall;
log_not_foundoff;
access_logoff;
# Make a regex exception for /.well-known so that clients can still
# access it despite the existence of the regex rule
# location ~ /(\. autotest ...) which would otherwise handle requests
# for /.well-known .
location^~/.well-known# The rules in this block are an adaptation of the rules
# in .htaccess that concern /.well-known .
location = /.well-known/carddavreturn301/remote.php/dav/;
location = /.well-known/caldavreturn301/remote.php/dav/;
location/.well-known/acme-challengetry_files $uri $uri/ =404;
location/.well-known/pki-validationtry_files $uri $uri/ =404;
# Let Nextcloud's API for /.well-known URIs handle all other
# requests by passing them to the front-end controller.
return301/index.php$request_uri;
# Rules borrowed from .htaccess to hide certain paths from clients
location ~ ^/(?:build tests config lib 3rdparty templates data)(?:$ /)return404;
location ~ ^/(?:\. autotest occ issue indie db_ console)return404;
# Ensure this block, which passes PHP files to the PHP process, is above the blocks
# which handle static assets (as seen below). If this block is not declared first,
# then NGinx will encounter an infinite rewriting loop when it prepend /index.php
# to the URI, resulting in a HTTP 500 error response.
location ~ \.php(?:$ /)# Required for legacy support
rewrite^/(?!index remote public cron core\/ajax\/update status ocs\/v[12] updater\/.+ ocs-provider\/.+ .+\/richdocumentscode(_arm64)?\/proxy)/index.php$request_uri;
fastcgi_split_path_info^(.+?\.php)(/.*)$;
set $path_info $fastcgi_path_info;
try_files $fastcgi_script_name =404;
includefastcgi_params;
fastcgi_paramSCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_paramPATH_INFO $path_info;
fastcgi_paramHTTPSon;
fastcgi_parammodHeadersAvailabletrue; # Avoid sending the security headers twice
fastcgi_paramfront_controller_activetrue; # Enable pretty urls
fastcgi_passphp-handler;
fastcgi_intercept_errorson;
fastcgi_request_bufferingoff;
fastcgi_max_temp_file_size0;
# Serve static files
location ~ \.(?:css js mjs svg gif png jpg ico wasm tflite map ogg flac)$try_files $uri /index.php$request_uri;
# HTTP response headers borrowed from Nextcloud .htaccess
add_headerCache-Control"public,max-age=15778463$asset_immutable";
add_headerReferrer-Policy"no-referrer"always;
add_headerX-Content-Type-Options"nosniff"always;
add_headerX-Frame-Options"SAMEORIGIN"always;
add_headerX-Permitted-Cross-Domain-Policies"none"always;
add_headerX-Robots-Tag"noindex,nofollow"always;
add_headerX-XSS-Protection"1; mode=block"always;
access_logoff; # Optional: Don't log access to assets
location ~ \.woff2?$try_files $uri /index.php$request_uri;
expires7d; # Cache-Control policy borrowed from .htaccess
access_logoff; # Optional: Don't log access to assets
# Rule borrowed from .htaccess
location/remotereturn301/remote.php$request_uri;
location/try_files $uri $uri/ /index.php$request_uri;
Symlink configuration site available to site enabled.
$ ln -s /etc/nginx/sites-available/nextcloud.conf /etc/nginx/sites-enabled/
Restart NGinx and access the URI in the browser.
Go through the installation of Nextcloud.
The user data on the installation dialog should point e.g to
administrator or similar, that user will become administrative access
rights in Nextcloud!
To adjust the database connection detail you have to edit the file
$install_folder/config/config.php.
Means here in the example within this post you would need to modify
/var/www/html/nextcloud/config/config.php to control or change the
database connection.
---%<---'dbname'=>'nextcloud_db',
'dbhost'=>'localhost', #(Or your remote PostgreSQL server address if you have.)
'dbport'=>'',
'dbtableprefix'=>'oc_',
'dbuser'=>'nextcloud_user',
'dbpassword'=>'1234', #(The password you set for database user.)
--->%---
After the installation and setup of the Nextcloud PHP application there are
more steps to be done. Have a look into the WebUI what you will need to do
as additional steps like create a cronjob or tuning of some more PHP
configurations.
If you ve done all things correct you should see a login page similar to
this:
Optional other steps for more enhanced configuration modifications
Move the data folder to somewhere else
The data folder is the root folder for all user content. By default it is
located in $install_folder/data, so in our case here it is in
/var/www/html/nextcloud/data.
Move the data directory outside the web server document root.
$ sudo mv /var/www/html/nextcloud/data /var/nextcloud_data
Ensure access permissions, mostly not needed if you move the folder.
$ sudo chown -R www-data:www-data /var/nextcloud_data$ sudo chown -R www-data:www-data /var/www/html/nextcloud/
Update the Nextcloud configuration:
Open the config/config.php file of your Nextcloud installation.
$ sudo vi /var/www/html/nextcloud/config/config.php
Update the datadirectory parameter to point to the new location of your data directory.
Make the installation available for multiple FQDNs on the same server
Adjust the Nextcloud configuration to listen and accept requests for
different domain names. Configure and adjust the key trusted_domains
accordingly.
$ sudo vi /var/www/html/nextcloud/config/config.php
Create and adjust the needed site configurations for the webserver.
Restart the NGinx unit.
An error message about .ocdata might occur
.ocdata is not found inside the data directory
Create file using touch and set necessary permissions.
$ sudo touch /var/nextcloud_data/.ocdata$ sudo chown -R www-data:www-data /var/nextcloud_data/
The password for the administrator user is unknown
Log in to your server:
SSH into the server where your PostgreSQL database is hosted.
Switch to the PostgreSQL user:
$ sudo -i -u postgres
Access the PostgreSQL command line
psql
List the databases: (If you re unsure which database is being used by Nextcloud, you can list all the databases by the list command.)
\l
Switch to the Nextcloud database:
Switch to the specific database that Nextcloud is using.
\c nextclouddb
Reset the password for the Nextcloud database user:
ALTER USER nextcloud_user WITH PASSWORD 'new_password';
Exit the PostgreSQL command line:
\q
Verify Database Configuration:
Check the database connection details in the config.php file to ensure they are correct.
sudo vi /var/www/html/nextcloud/config/config.php
Replace nextcloud_db, nextcloud_user, and your_password with your actual database name, user, and password.
---%<---'dbname'=>'nextcloud_db',
'dbhost'=>'localhost', #(or your PostgreSQL server address)
'dbport'=>'',
'dbtableprefix'=>'oc_',
'dbuser'=>'nextcloud_user',
'dbpassword'=>'1234', #(The password you set for nextcloud_user.)
--->%---
Restart NGinx and access the UI through the browser.
Please note: This review is not meant to be part of my
usual contributions to ACM's Computing Reviews . I do want, though, to
share it with people that follow my general interests and such stuff.
This article was published almost a year ago, and I read it just after
relocating from Argentina back to Mexico. I came from a country starting to
realize the shock it meant to be ruled by an autocratic, extreme right-wing
president willing to overrun its Legislative and bent on destroying the
State itself not too different from what we are now witnessing on a
global level.
I have been a strong proponent and defender of Free Software and of Free
Culture throughout my adult life. And I have been a Socialist since my
early teenage years. I cannot say there is a strict correlation between
them, but there is a big intersection of people and organizations who
aligns to both sides And rtica (and Mariana Fossatti) are clearly among
them.
Freedom is a word that has brought us many misunderstanding throughout
the past many decades. We will say that Freedom can only be brought
hand-by-hand with Equality, Fairness and Tolerance. But the extreme-right
wing (is it still bordering Fascism, or has it finally embraced it as its
true self?) that has grown so much in many countries over the last years
also seems to have appropriated the term, even taking it as their
definition. In English (particularly, in USA English), liberty is a more
patriotic term, and freedom is more personal (although the term used for
the market is free market); in Spanish, we conflate them both under
libre.
Mariana refers to a third blog, by Rolando Astarita, where the author
introduces the concepts positive and negative
freedom/liberties. Astarita
characterizes negative freedom as an individual s possibility to act
without interferences or coertion, and is limited by other people s
freedom, while positive freedom is the real capacity to exercise one s
autonomy and achieve self-realization; this does not depend on a person on
its own, but on different social conditions; Astarita understands the
Marxist tradition to emphasize on the positive freedom.
Mariana brings this definition to our usual discussion on licensing: If we
follow negative freedom, we will understand free licenses as the idea of
access without interference to cultural or information goods, as long as
it s legal (in order not to infringe other s property rights). Licensing is
seen as a private content, and each individual can grant access and use to
their works at will.
The previous definition might be enough for many, but she says, is missing
something important. The practical effect of many individuals renouncing
a bit of control over their property rights produce, collectively, the
common goods. They constitute a pool of knowledge or culture that are no
longer an individual, contractual issue, but grow and become social,
collective. Negative freedom does not go further, but positive liberty
allows broadening the horizon, and takes us to a notion of free culture
that, by strengthening the commons, widens social rights.
She closes the article by stating (and I ll happily sign as if they were my
own words) that we are Free Culture militants not only because it affirms
the individual sovereignty to deliver and receive cultural resources, in an
intellectual property framework guaranteed by the state. Our militancy is
of widening the cultural enjoying and participation to the collective
through the defense of common cultural goods ( ) We want to build
Free Culture for a Free Society. But a Free Society is not a society of
free owners, but a society emancipated from the structures of economic
power and social privilege that block this potential collective .
Inspired by some friends,1 I too wanted to make a tiny website telling which
event I am at this exact moment. Thankfully I already had an another toy project
with that information easily available, so generating the web page was a matter
of just querying that project's API and feeding that data to a HTML template.
Now the obvious way to host that would be to hook up the HTML-generating code
to a web server, maybe add some caching for the API calls, and then route
external HTTPS traffic to it. However, that'd a) require that server to be
constantly available to serve traffic, and b) be boring.
For context: I have an existing setup called staticweb, which is effectively
a fancy name for a couple of Puppet-managed servers that run Apache httpd to
serve static web pages and have a bunch of systemd timers running rsync to
ensure they're serving the same content. It works really well and I use it for
things ranging from my website or whyisbetabroken.com to things like my
internal apt repository.
Now, there are two ways to get new content into that mechanism: it can be
manually pushed in from e.g. a CI job, or the system can be configured to
periodically pull it from a separate server. The latter mechanism was initially
created so that I could pull the Debian packages from my separate reprepro
server into the staticweb setup. It turns out that the latter makes a really
neat method for handling other dynamically-generated static sites as well.
So, for my "where is Taavi at" site, I ended up writing the server part in
Go, and included an rsync server using the gokrazy/rsync package. Initially I
just implemented a static temporary directory with a timer to regularly update
the HTML file in it, but then I got an even more cursed idea: what if the HTML
was dynamically generated when an rsync client connected to the server? So I
did just that.
For deployment, I slapped the entire server part in a container and deployed it
to my Kubernetes cluster. The rsync server is exposed directly as a service
to my internal network with no authentication or encryption - I think that's
fine since that's a read-only service in a private LAN and the resulting HTML
is going to be publicly exposed anyway. (Thanks to some DNS magic, just
creating a LoadBalancer Service object with a special annotation is enough to
have a DNS name provisioned for the assigned IP address, which is neat.)
Overall the setup works nice, at least for now. I need to add some sort of a
cache to not fetch unchanged information from the API since for every update.
And I guess I could write some cursed rsyncd reverse proxy with per-module
rules if I end up creating more sites like this to avoid creating new
LoadBalancer services for each of them.
Another short status update of what happened on my side last
month. Some more ModemManager bits landed, Phosh
0.46 is out, haptic feedback
is now better tunable plus some more. See below for details (no April 1st
joke in there, I promise):
phosh
Rust, and resistance to it in some parts of the Linux community, has been in my feed recently. One undercurrent seems to be the notion that Rust is woke (and should therefore be rejected as part of culture wars).
I m going to argue that Rust, the language, is woke. So the opponents are right, in that sense. Of course, as ever, dissing something for being woke is nasty and fascist-adjacent.
Community
The obvious way that Rust may seem woke is that it has the trappings, and many of the attitudes and outcomes, of a modern, nice, FLOSS community. Rust certainly does better than toxic environments like the Linux kernel, or Debian. This is reflected in a higher proportion of contributors from various kinds of minoritised groups. But Rust is not outstanding in this respect. It certainly has its problems. Many other projects do as well or better.
And this is well-trodden ground. I have something more interesting to say:
Technological values - particularly, compared to C/C++
Rust is woke technology that embodies a woke understanding of what it means to be a programming language.
Ostensible values
Let s start with Rust s strapline:
A language empowering everyone to build reliable and efficient software.
Surprisingly, this motto is not mere marketing puff. For Rustaceans, it is a key goal which strongly influences day-to-day decisions (big and small).
Empowering everyone is a key aspect of this, which aligns with my own personal values. In the Rust community, we care about empowerment. We are trying to help liberate our users. And we want to empower everyone because everyone is entitled to technological autonomy. (For a programming language, empowering individuals means empowering their communities, of course.)
This is all very airy-fairy, but it has concrete consequences:
Attitude to the programmer s mistakes
In Rust we consider it a key part of our job to help the programmer avoid mistakes; to limit the consequences of mistakes; and to guide programmers in useful directions.
If you write a bug in your Rust program, Rust doesn t blame you. Rust asks how could the compiler have spotted that bug .
This is in sharp contrast to C (and C++). C nowadays is an insanely hostile programming environment. A C compiler relentlessly scours your program for any place where you may have violated C s almost incomprehensible rules, so that it can compile your apparently-correct program into a buggy executable. And then the bug is considered your fault.
These aren t just attitudes implicitly embodied in the software. They are concrete opinions expressed by compiler authors, and also by language proponents. In other words:
Rust sees programmers writing bugs as a systemic problem, which must be addressed by improvements to the environment and the system. The toxic parts of the C and C++ community see bugs as moral failings by individual programmers.
Sound familiar?
The ideology of the hardcore programmer
Programming has long suffered from the myth of the rockstar . Silicon Valley techbro culture loves this notion.
In reality, though, modern information systems are far too complicated for a single person. Developing systems is a team sport. Nontechnical, and technical-adjacent, skills are vital: clear but friendly communication; obtaining and incorporating the insights of every member of your team; willingness to be challenged. Community building. Collaboration. Governance.
The hardcore C community embraces the rockstar myth: they imagine that a few super-programmers (or super-reviewers) are able to spot bugs, just by being so brilliant. Of course this doesn t actually work at all, as we can see from the atrocious bugfest that is the Linux kernel.
These rockstars want us to believe that there is a steep hierarchy in programmming; that they are at the top of this hierarchy; and that being nice isn t important.
Sound familiar?
Memory safety as a power struggle
Much of the modern crisis of software reliability arises from memory-unsafe programming languages, mostly C and C++.
Addressing this is a big job, requiring many changes. This threatens powerful interests; notably, corporations who want to keep shipping junk. (See also, conniptions over the EU Product Liability Directive.)
The harms of this serious problem mostly fall on society at large, but the convenience of carrying on as before benefits existing powerful interests.
Sound familiar?
Memory safety via Rust as a power struggle
Addressing this problem via Rust is a direct threat to the power of established C programmers such as gatekeepers in the Linux kernel. Supplanting C means they will have to learn new things, and jostle for status against better Rustaceans, or be replaced. More broadly, Rust shows that it is practical to write fast, reliable, software, and that this does not need (mythical) rockstars .
So established C programmer experts are existing vested interests, whose power is undermined by (this approach to) tackling this serious problem.
Sound familiar?
NotesThis is not a RIIR manifesto
I m not saying we should rewrite all the world s C in Rust. We should not try to do that.
Rust is often a good choice for new code, or when a rewrite or substantial overhaul is needed anyway. But we re going to need other techniques to deal with all of our existing C. CHERI is a very promising approach. Sandboxing, emulation and automatic translation are other possibilities. The problem is a big one and we need a toolkit, not a magic bullet.
But as for Linux: it is a scandal that substantial new drivers and subsystems are still being written in C. We could have been using Rust for new code throughout Linux years ago, and avoided very many bugs. Those bugs are doing real harm. This is not OK.
Disclosure
I first learned C from K&R I in 1989. I spent the first three decades of my life as a working programmer writing lots and lots of C. I ve written C++ too. I used to consider myself an expert C programmer, but nowadays my C is a bit rusty and out of date. Why is my C rusty? Because I found Rust, and immediately liked and adopted it (despite its many faults).
I like Rust because I care that the software I write actually works: I care that my code doesn t do harm in the world.
On the meaning of woke
The original meaning of woke is something much more specific, to do with racism. For the avoidance of doubt, I don t think Rust is particularly antiracist.
I m using woke (like Rust s opponents are) in the much broader, and now much more prevalent, culture wars sense.
Pithy conclusion
If you re a senior developer who knows only C/C++, doesn t want their authority challenged, and doesn t want to have to learn how to write better software, you should hate Rust.
Also you should be fired.
Edited 2025-03-28 17:10 UTC to fix minor problems and add a new note about the meaning of the word "woke".
Around a year ago I discussed two concerns with software release archives (tarball artifacts) that could be improved to increase confidence in the supply-chain security of software releases. Repeating the goals for simplicity:
Release artifacts should be built in a way that can be reproduced by others
It should be possible to build a project from source tarball that doesn t contain any generated or vendor files (e.g., in the style of git-archive).
While implementing these ideas for a small project was accomplished within weeks see my announcement of Libntlm version 1.8 adressing this in complex projects uncovered concerns with tools that had to be addressed, and things stalled for many months pending that work.
I had the notion that these two goals were easy and shouldn t be hard to accomplish. I still believe that, but have had to realize that improving tooling to support these goals takes time. It seems clear that these concepts are not universally agreed on and implemented generally.
I m now happy to recap some of the work that led to releases of libtasn1 v4.20.0, inetutils v2.6, libidn2 v2.3.8, libidn v1.43. These releases all achieve these goals. I am working on a bunch of more projects to support these ideas too.
What have the obstacles so far been to make this happen? It may help others who are in the same process of addressing these concerns to have a high-level introduction to the issues I encountered. Source code for projects above are available and anyone can look at the solutions to learn how the problems are addressed.
First let s look at the problems we need to solve to make git-archive style tarballs usable:
Version Handling
To build usable binaries from a minimal tarballs, it need to know which version number it is. Traditionally this information was stored inside configure.ac in git. However I use gnulib s git-version-gen to infer the version number from the git tag or git commit instead. The git tag information is not available in a git-archive tarball. My solution to this was to make use of the export-subst feature of the .gitattributes file. I store the file .tarball-version-git in git containing the magic cookie like this:
$Format:%(describe)$
With this, git-archive will replace with a useful version identifier on export, see the libtasn1 patch to achieve this. To make use of this information, the git-version-gen script was enhanced to read this information, see the gnulib patch. This is invoked by ./configure to figure out which version number the package is for.
Translations
We want translations to be included in the minimal source tarball for it to be buildable. Traditionally these files are retrieved by the maintainer from the Translation project when running ./bootstrap, however there are two problems with this. The first one is that there is no strong authentication or versioning information on this data, the tools just download and place whatever wget downloaded into your source tree (printf-style injection attack anyone?). We could improve this (e.g., publish GnuPG signed translations messages with clear versioning), however I did not work on that further. The reason is that I want to support offline builds of packages. Downloading random things from the Internet during builds does not work when building a Debian package, for example. The translation project could solve this by making a monthly tarball with their translations available, for distributors to pick up and provide as a separate package that could be used as a build dependency. However that is not how these tools and projects are designed. Instead I reverted back to storing translations in git, something that I did for most projects back when I was using CVS 20 years ago. Hooking this into ./bootstrap and gettext workflow can be tricky (ideas for improvement most welcome!), but I used a simple approach to store all directly downloaded po/*.po files directly as po/*.po.in and make the ./bootstrap tool move them in place, see the libidn2 commit followed by the actual make update-po commit with all the translations where one essential step is:
# Prime po/*.po from fall-back copy stored in git.
for poin in po/*.po.in; do
po=$(echo $poin sed 's/.in//')
test -f $po cp -v $poin $po
done
ls po/*.po sed 's .*/ ; s \.po$ ' > po/LINGUAS
Fetching vendor files like gnulib
Most build dependencies are in the shape of You need a C compiler . However some come in the shape of source-code files intended to be vendored , and gnulib is a huge repository of such files. The latter is a problem when building from a minimal git archive. It is possible to consider translation files as a class of vendor files, since they need to be copied verbatim into the project build directory for things to work. The same goes for *.m4 macros from the GNU Autoconf Archive. However I m not confident that the solution for all vendor files must be the same. For translation files and for Autoconf Archive macros, I have decided to put these files into git and merge them manually occasionally. For gnulib files, in some projects like OATH Toolkit I also store all gnulib files in git which effectively resolve this concern. (Incidentally, the reason for doing so was originally that running ./bootstrap took forever since there is five gnulib instances used, which is no longer the case since gnulib-tool was rewritten in Python.) For most projects, however, I rely on ./bootstrap to fetch a gnulib git clone when building. I like this model, however it doesn t work offline. One way to resolve this is to make the gnulib git repository available for offline use, and I ve made some effort to make this happen via a Gnulib Git Bundle and have explained how to implement this approach for Debian packaging. I don t think that is sufficient as a generic solution though, it is mostly applicable to building old releases that uses old gnulib files. It won t work when building from CI/CD pipelines, for example, where I have settled to use a crude way of fetching and unpacking a particular gnulib snapshot, see this Libntlm patch. This is much faster than working with git submodules and cloning gnulib during ./bootstrap. Essentially this is doing:
Test the git-archive tarball
This goes without saying, but if you don t test that building from a git-archive style tarball works, you are likely to regress at some point. Use CI/CD techniques to continuously test that a minimal git-archive tarball leads to a usable build.
Mission Accomplished
So that wasn t hard, was it? You should now be able to publish a minimal git-archive tarball and users should be able to build your project from it.
I recommend naming these archives as PROJECT-vX.Y.Z-src.tar.gz replacing PROJECT with your project name and X.Y.Z with your version number. The archive should have only one sub-directory named PROJECT-vX.Y.Z/ containing all the source-code files. This differentiate it against traditional PROJECT-X.Y.Z.tar.gz tarballs in that it embeds the git tag (which typically starts with v) and contains a wildcard-friendly -src substring. Alas there is no consistency around this naming pattern, and GitLab, GitHub, Codeberg etc all seem to use their own slightly incompatible variant.
Let s go on to see what is needed to achieve reproducible make dist source tarballs. This is the release artifact that most users use, and they often contain lots of generated files and vendor files. These files are included to make it easy to build for the user. What are the challenges to make these reproducible?
Build dependencies causing different generated content
The first part is to realize that if you use tool X with version A to generate a file that goes into the tarball, version B of that tool may produce different outputs. This is a generic concern and it cannot be solved. We want our build tools to evolve and produce better outputs over time. What can be addressed is to avoid needless differences. For example, many tools store timestamps and versioning information in the generated files. This causes needless differences, which makes audits harder. I have worked on some of these, like Autoconf Archive timestamps but solving all of these examples will take a long time, and some upstream are reluctant to incorporate these changes. My approach meanwhile is to build things using similar environments, and compare the outputs for differences. I ve found that the various closely related forks of GNU/Linux distributions are useful for this. Trisquel 11 is based on Ubuntu 22.04, and building my projects using both and comparing the differences only give me the relevant differences to improve. This can be extended to compare AlmaLinux with RockyLinux (for both versions 8 and 9), Devuan 5 against Debian 12, PureOS 10 with Debian 11, and so on.
Timestamps
Sometimes tools store timestamps in files in a way that is harder to fix. Two notable examples of this are *.po translation files and Texinfo manuals. For translation files, I have resolved this by making sure the files use a predictable POT-Creation-Date timestamp, and I set it to the modification timestamps of the NEWS file in the repository (which I set to the git commit of the latest commit elsewhere) like this:
dist-hook: po-CreationDate-to-mtime-NEWS
.PHONY: po-CreationDate-to-mtime-NEWS
po-CreationDate-to-mtime-NEWS: mtime-NEWS-to-git-HEAD
$(AM_V_GEN)for p in $(distdir)/po/*.po $(distdir)/po/$(PACKAGE).pot; do \
if test -f "$$p"; then \
$(SED) -e 's,POT-Creation-Date: .*\\n",POT-Creation-Date: '"$$(env LC_ALL=C TZ=UTC0 stat --format=%y $(srcdir)/NEWS cut -c1-16,31-)"'\\n",' < $$p > $$p.tmp && \
if cmp $$p $$p.tmp > /dev/null; then \
rm -f $$p.tmp; \
else \
mv $$p.tmp $$p; \
fi \
fi \
done
Similarily, I set a predictable modification time of the texinfo source file like this:
dist-hook: mtime-NEWS-to-git-HEAD
.PHONY: mtime-NEWS-to-git-HEAD
mtime-NEWS-to-git-HEAD:
$(AM_V_GEN)if test -e $(srcdir)/.git \
&& command -v git > /dev/null; then \
touch -m -t "$$(git log -1 --format=%cd \
--date=format-local:%Y%m%d%H%M.%S)" $(srcdir)/NEWS; \
fi
However I ve realized that this needs to happen earlier and probably has to be run during ./configure time, because the doc/version.texi file is generated on first build before running make dist and for some reason the file is not rebuilt at release time. The Automake texinfo integration is a bit inflexible about providing hooks to extend the dependency tracking.
The method to address these differences isn t really important, and they change over time depending on preferences. What is important is that the differences are eliminated.
ChangeLog
Traditionally ChangeLog files were manually prepared, and still is for some projects. I maintain git2cl but recently I ve settled with gnulib s gitlog-to-changelog because doing so avoids another build dependency (although the output formatting is different and arguable worse for my git commit style). So the ChangeLog files are generated from git history. This means a shallow clone will not produce the same ChangeLog file depending on how deep it was cloned. For Libntlm I simply disabled use of generated ChangeLog because I wanted to support an even more extreme form of reproducibility: I wanted to be able to reproduce the full make dist source archives from a minimal git-archive source archive. However for other projects I ve settled with a middle ground. I realized that for git describe to produce reproducible outputs, the shallow clone needs to include the last release tag. So it felt acceptable to assume that the clone is not minimal, but instead has some but not all of the history. I settled with the following recipe to produce ChangeLog's covering all changes since the last release.
dist-hook: gen-ChangeLog
.PHONY: gen-ChangeLog
gen-ChangeLog:
$(AM_V_GEN)if test -e $(srcdir)/.git; then \
LC_ALL=en_US.UTF-8 TZ=UTC0 \
$(top_srcdir)/build-aux/gitlog-to-changelog \
--srcdir=$(srcdir) -- \
v$(PREV_VERSION)~.. > $(distdir)/cl-t && \
printf '\n\nSee the source repo for older entries\n' \
>> $(distdir)/cl-t && \
rm -f $(distdir)/ChangeLog && \
mv $(distdir)/cl-t $(distdir)/ChangeLog; \
fi
I m undecided about the usefulness of generated ChangeLog files within make dist archives. Before we have stable and secure archival of git repositories widely implemented, I can see some utility of this in case we lose all copies of the upstream git repositories. I can sympathize with the concept of ChangeLog files died when we started to generate them from git logs: the files no longer serve any purpose, and we can ask people to go look at the git log instead of reading these generated non-source files.
Long-term reproducible trusted build environment
Distributions comes and goes, and old releases of them goes out of support and often stops working. Which build environment should I chose to build the official release archives? To my knowledge only Guix offers a reliable way to re-create an older build environment (guix gime-machine) that have bootstrappable properties for additional confidence. However I had two difficult problems here. The first one was that I needed Guix container images that were usable in GitLab CI/CD Pipelines, and this side-tracked me for a while. The second one delayed my effort for many months, and I was inclined to give up. Libidn distribute a C# implementation. Some of the C# source code files included in the release tarball are generated. By what? You guess it, by a C# program, with the source code included in the distribution. This means nobody could reproduce the source tarball of Libidn without trusting someone elses C# compiler binaries, which were built from binaries of earlier releases, chaining back into something that nobody ever attempts to build any more and likely fail to build due to bit-rot. I had two basic choices, either remove the C# implementation from Libidn (which may be a good idea for other reasons, since the C and C# are unrelated implementations) or build the source tarball on some binary-only distribution like Trisquel. Neither felt appealing to me, but a late christmas gift of a reproducible Mono came to Guix that resolve this.
Embedded images in Texinfo manual
For Libidn one section of the manual has an image illustrating some concepts. The PNG, PDF and EPS outputs were generated via fig2dev from a *.fig file (hello 1985!) that I had stored in git. Over time, I had also started to store the generated outputs because of build issues. At some point, it was possible to post-process the PDF outputs with grep to remove some timestamps, however with compression this is no longer possible and actually the grep command I used resulted in a 0-byte output file. So my embedded binaries in git was no longer reproducible. I first set out to fix this by post-processing things properly, however I then realized that the *.fig file is not really easy to work with in a modern world. I wanted to create an image from some text-file description of the image. Eventually, via the Guix manual on guix graph, I came to re-discover the graphviz language and tool called dot (hello 1993!). All well then? Oh no, the PDF output embeds timestamps. Binary editing of PDF s no longer work through simple grep, remember? I was back where I started, and after some (soul- and web-) searching I discovered that Ghostscript (hello 1988!) pdfmarks could be used to modify things here. Cooperating with automake s texinfo rules related to make dist proved once again a worthy challenge, and eventually I ended up with a Makefile.am snippet to build images that could be condensed into:
info_TEXINFOS = libidn.texi
libidn_TEXINFOS += libidn-components.png
imagesdir = $(infodir)
images_DATA = libidn-components.png
EXTRA_DIST += components.dot
DISTCLEANFILES = \
libidn-components.eps libidn-components.png libidn-components.pdf
libidn-components.eps: $(srcdir)/components.dot
$(AM_V_GEN)$(DOT) -Nfontsize=9 -Teps < $< > $@.tmp
$(AM_V_at)! grep %%CreationDate $@.tmp
$(AM_V_at)mv $@.tmp $@
libidn-components.pdf: $(srcdir)/components.dot
$(AM_V_GEN)$(DOT) -Nfontsize=9 -Tpdf < $< > $@.tmp
# A simple sed on CreationDate is no longer possible due to compression.
# 'exiftool -CreateDate' is alternative to 'gs', but adds ~4kb to file.
# Ghostscript add <1kb. Why can't 'dot' avoid setting CreationDate?
$(AM_V_at)printf '[ /ModDate ()\n /CreationDate ()\n /DOCINFO pdfmark\n' > pdfmarks
$(AM_V_at)$(GS) -q -dBATCH -dNOPAUSE -sDEVICE=pdfwrite -sOutputFile=$@.tmp2 $@.tmp pdfmarks
$(AM_V_at)rm -f $@.tmp pdfmarks
$(AM_V_at)mv $@.tmp2 $@
libidn-components.png: $(srcdir)/components.dot
$(AM_V_GEN)$(DOT) -Nfontsize=9 -Tpng < $< > $@.tmp
$(AM_V_at)mv $@.tmp $@
pdf-recursive: libidn-components.pdf
dvi-recursive: libidn-components.eps
ps-recursive: libidn-components.eps
info-recursive: $(top_srcdir)/.version libidn-components.png
Surely this can be improved, but I m not yet certain in what way is the best one forward. I like having a text representation as the source of the image. I m sad that the new image size is ~48kb compared to the old image size of ~1kb. I tried using exiftool -CreateDate as an alternative to GhostScript, but using it to remove the timestamp added ~4kb to the file size and naturally I was appalled by this ignorance of impending doom.
Test reproducibility of tarball
Again, you need to continuously test the properties you desire. This means building your project twice using different environments and comparing the results. I ve settled with a small GitLab CI/CD pipeline job that perform bit-by-bit comparison of generated make dist archives. It also perform bit-by-bit comparison of generated git-archive artifacts. See the Libidn2 .gitlab-ci.yml 0-compare job which essentially is:
Notice that I discovered that git archive outputs differ over time too, which is natural but a bit of a nuisance. The output of the job is illuminating in the way that all SHA256 checksums of generated tarballs are included, for example the libidn2 v2.3.8 job log:
I m sure I have forgotten or suppressed some challenges (sprinkling LANG=C TZ=UTC0 helps) related to these goals, but my hope is that this discussion of solutions will inspire you to implement these concepts for your software project too. Please share your thoughts and additional insights in a comment below. Enjoy Happy Hacking in the course of practicing this!
After some investigation, it's caused by the recent upload of netavark
1.14.0-2. In this version, netavark
switched from iptables to nftables as the default firewall driver. That doesn't
really fly on GitLab Saas shared runners.
For the complete background, refer to
https://discussion.fedoraproject.org/t/125528. Note that the issue with
GitLab was reported back in November, but at this point the conversation had
died out.
Fortunately, it's easy to workaround, we can tell netavark to keep using
iptables via the environment variables NETAVARK_FW. The .gitlab-ci.yml file
above becomes:
Two years after OpenAI launched ChatGPT 3.5, humanity is not on the cusp of
extinction and Elon Musk seems more responsible for job loss than any AI agent.
However, ask any web administrator and you will learn that large language
models are having a significant impact on the world wide
web
(or, for a less technical account, see Forbes articles on
bots).
At May First, a membership organization that has been
supporting thousands of web site for over 20 years, we have never seen anything
like this before.
Wow, my site is getting really popular
It started in 2023. Web sites that performed quite well with a steady
viewership started having traffic spikes. These were relatively easy to
diagnose, since most of the spikes came from visitors that properly identified
themselves as bots, allowing us to see that the big players - OpenAI, Bing,
Google, Facebook - were increasing their efforts to scrape as much content from
web sites as possible.
Small brochure sites were mostly unaffected because they could be scraped in a
matter of minutes. But large sites with an archive of high quality human
written content were getting hammered. Any web site with a search feature or a
calendar or any interface that generated exponential hits that could be
followed were particularly vulnerable.
But hey, that s what robots.txt is for, right? To
tell robots to back off if you don t want them scraping your site?
Eventually, the cracks began to show. Bots were ignoring robots.txt (did they
ever pay that much attention to it in the first place?). Furthermore, rate
limiting requests by user agent also began to fail. When you post a link on
Facebook, a bot identifying itself as facebooketernalhit is invoked to
preview the page so it can show a picture and other meta data. We don t want to
rate limit that bot, right? Except, Facebook is also using this bot to scrape
your site, often bringing your site to its knees. And don t get me started on
TwitterBot.
Eventually, it became clear that the majority of the armies of bots scraping
our sites have completely given up on identifying themselves as bots and are
instead using user agents indistinguishable from regular browsers. By using
thousands of different IP addresses, it has become really hard to separate the
real humans from the bots.
Now what?
So, no, unfortunately, your web site is not suddenly getting really popular.
And, you are blessed with a whole new set of strategic decisions.
Fortunately, May First has undergone a major infrastructure transition,
resulting in centralized logging of all web sites and a fleet of web proxy
servers that intercept all web traffic. Centralized logging means we can
analyze traffic and identify bots more easily, and a web proxy fleet allows us
to more easily implement rules across all web sites.
However, even with all of our latest changes and hours upon hours of work to
keep out the bots, our members are facing some hard decisions about maintaining
an open web.
One member of May First provides Google translations of their web site to every
language available. But wow, that is now a disaster because instead of having
every bot under the sun scrapping all 843 (a made up number) pieces of unique
content on their site, the same bots are scraping 843 * (number of available
languages) pieces of content on their site. Should they stop providing this
translation service in order to ensure people can access their site in the
site s primary language?
Should web sites turn off their search features that include drop down options
of categories to prevent bots from systematically refreshing the search page
with every possible combination of search terms?
Do we need to alter our calendar software to avoid providing endless links into
the future (ok, that is an easy one)?
What s next?
Something has to change.
Lock down web 2.0. Web 2.0 brought us wonderful dynamic web sites, which
Drupal and WordPress and many other pieces of amazing software have
supported for over a decade. This is the software that is getting bogged
down by bots. Maybe we need to figure out a way to lock down the dynamic
aspects of this software to logged in users and provide static content for
everyone else?
Paywalls and accounts everywhere. There s always been an amazing
non-financial reward to providing a web site with high quality movement
oriented content for free. It populates the search engines, provides links
to inspiring and useful content in moments of crises, and can galvanize
movements. But these moments of triumph happen between long periods of hard
labor that now seems to mostly feed capitalist AI scumbags. If we add a new
set of expenses and labor to keep the sites running for this purpose, how
sustainable is that? Will our treasure of free movement content have to move
behind paywalls or logins? If we provide logins, will that keep the bots out
or just create a small hurdle for them to automate the account creation
process? What happens when we can t search for this kind of content via
search engines?
Cutting deals. What if our movement content providers are forced to cut
deals with the AI entrepreneurs to allow the paying scumbags to fund the content
creation. Eww. Enough said.
Bot detection. Maybe we just need to get better at bot detection? This
will surely be an arms race, but would have some good benefits. Bots have
also been filling out our forms and populating our databases with spam,
testing credit cards against our donation pages, conducting denial of
service attacks and all kinds of other irritating acts of vandalism. If we
were better at stopping bots automatically it would have a lot of benefits.
But what impact would it have on our web sites and the experience of using
them? What about good bots (RSS feed readers, payment processors,
web hooks, uptime detectors)? Will we cut the legs off any developer trying
to automate something?
I m not really sure where this is going, but it seems that the world wide web is
about to head in a new direction.
Armadillo is a powerful
and expressive C++ template library for linear algebra and scientific
computing. It aims towards a good balance between speed and ease of use,
has a syntax deliberately close to Matlab, and is useful for algorithm
development directly in C++, or quick conversion of research code into
production environments. RcppArmadillo
integrates this library with the R environment and language and is
widely used by (currently) 1234 other packages on CRAN, downloaded 38.8 million
times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint
/ vignette) by Conrad and myself has been cited 617 times according
to Google Scholar.
Conrad released a new
minor version 14.4.0 last month. That was preceding by several extensive
rounds of reverse-dependency checks covering the 1200+ packages at CRAN.
We eventually narrowed the impact down to just eight packages, and I
opened issue
#462 to manage the transition along with GitHub-only release
14.4.0-0 of RcppArmadillo.
Several maintainers responded very promptly and updated within days
this is truly appreciated. Yesterday the last package updated at CRAN coinciding nicely with our
planned / intended upload to CRAN one month after the release.
So this new release, at version -1, is now on CRAN. It brings the usual number
of small improvements to Armadillo itself as well as
updates to packaging.
The changes since the last CRAN release are summarised
below.
Changes in
RcppArmadillo version 14.4.0-1 (2025-03-17)
CRAN release having given a few packages time to catch-up to
small upstream change as discussed and managed in #462
Updated bibliography, and small edits to sparse matrix
vignette
Switched continuous integration action to r-ci with implicit
bootstrap
Changes
in RcppArmadillo version 14.4.0-0 (2025-02-17) (GitHub Only)
Upgraded to Armadillo release 14.4.0 (Filtered Espresso)
Faster handling of pow() and square()
within accu() and sum() expressions
Faster sort() and sort_index() for
complex matrices
Expanded the field class with .reshape() and
.resize() member functions
More efficient handling of compound expressions by
sum(), reshape(),
trans()
Better detection of vector expressions by pow(),
imag(), conj()
The package generator helper function now supports additional
DESCRIPTIONs
This release revealed a need for very minor changes for a handful
reverse-dependency packages which will be organized via GitHub issue
tracking
As promised on my previous post, on this entry I ll explain how I ve set up forgejo
actions on the source repository of this site to build it using a runner instead of doing it on the public server using
a webhook to trigger the operation.
Setting up the systemThe first thing I ve done is to disable the forgejo webhook call that was used to publish the site, as I don t want to
run it anymore.
After that I added a new workflow to the repository that does the following things:
push the result to a branch that contains the generated site (we do this because the server is already configured
to work with the git repository and we can use force pushes to keep only the last version of the site, removing the
need of extra code to manage package uploads and removals).
uses curl to send a notification to an instance of the webhook server installed on the
remote server that triggers a script that updates the site using the git branch.
Setting up the webhook serviceOn the server machine we have installed and configured the webhook service to run a script that updates the site.
To install the application and setup the configuration we have used the following script:
#!/bin/shset-e# ---------# VARIABLES# ---------ARCH="$(dpkg --print-architecture)"WEBHOOK_VERSION="2.8.2"DOWNLOAD_URL="https://github.com/adnanh/webhook/releases/download"WEBHOOK_TGZ_URL="$DOWNLOAD_URL/$WEBHOOK_VERSION/webhook-linux-$ARCH.tar.gz"WEBHOOK_SERVICE_NAME="webhook"# FilesWEBHOOK_SERVICE_FILE="/etc/systemd/system/$WEBHOOK_SERVICE_NAME.service"WEBHOOK_SOCKET_FILE="/etc/systemd/system/$WEBHOOK_SERVICE_NAME.socket"WEBHOOK_TML_TEMPLATE="/srv/blogops/action/webhook.yml.envsubst"WEBHOOK_YML="/etc/webhook.yml"# Config file valuesWEBHOOK_USER="$(id-u)"WEBHOOK_GROUP="$(id-g)"WEBHOOK_LISTEN_STREAM="172.31.31.1:4444"# ----# MAIN# ----# Install binary from releases (on Debian only version 2.8.0 is available, but# I need the 2.8.2 version to support the systemd activation mode).
curl -fsSL-o"/tmp/webhook.tgz""$WEBHOOK_TGZ_URL"tar-C /tmp -xzf /tmp/webhook.tgz
sudo install-m 755 "/tmp/webhook-linux-$ARCH/webhook" /usr/local/bin/webhook
rm-rf"/tmp/webhook-linux-$ARCH" /tmp/webhook.tgz
# Service filesudo sh -c"cat >'$WEBHOOK_SERVICE_FILE'"<<EOF
[Unit]
Description=Webhook server
[Service]
Type=exec
ExecStart=webhook -nopanic -hooks $WEBHOOK_YML
User=$WEBHOOK_USER
Group=$WEBHOOK_GROUPEOF
# Socket configsudo sh -c"cat >'$WEBHOOK_SOCKET_FILE'"<<EOF
[Unit]
Description=Webhook server socket
[Socket]
# Set FreeBind to listen on missing addresses (the VPN can be down sometimes)
FreeBind=true
# Set ListenStream to the IP and port you want to listen on
ListenStream=$WEBHOOK_LISTEN_STREAM
[Install]
WantedBy=multi-user.target
EOF
# Config fileBLOGOPS_TOKEN="$(uuid)"\
envsubst <"$WEBHOOK_TML_TEMPLATE"sudo sh -c"cat >$WEBHOOK_YML"chmod 0640 "$WEBHOOK_YML"
chwon "$WEBHOOK_USER:$WEBHOOK_GROUP""$WEBHOOK_YML"# Restart and enable servicesudo systemctl daemon-reload
sudo systemctl stop "$WEBHOOK_SERVICE_NAME.socket"sudo systemctl start "$WEBHOOK_SERVICE_NAME.socket"sudo systemctl enable"$WEBHOOK_SERVICE_NAME.socket"# ----# vim: ts=2:sw=2:et:ai:sts=2
As seen on the code, we ve installed the application using a binary from the project repository instead of a package
because we needed the latest version of the application to use systemd with socket activation.
The configuration file template is the following one:
The version on /etc/webhook.yml has the BLOGOPS_TOKEN adjusted to a random value that has to exported as a secret on
the forgejo project (see later).
Once the service is started each time the action is executed the webhook daemon will get a notification and will run
the following update-blogops.sh script to publish the updated version of the site:
#!/bin/shset-e# ---------# VARIABLES# ---------# ValuesREPO_URL="ssh://git@forgejo.mixinet.net/mixinet/blogops.git"REPO_BRANCH="html"REPO_DIR="public"MAIL_PREFIX="[BLOGOPS-UPDATE-ACTION] "# Address that gets all messages, leave it empty if not wantedMAIL_TO_ADDR="blogops@mixinet.net"# DirectoriesBASE_DIR="/srv/blogops"PUBLIC_DIR="$BASE_DIR/$REPO_DIR"NGINX_BASE_DIR="$BASE_DIR/nginx"PUBLIC_HTML_DIR="$NGINX_BASE_DIR/public_html"ACTION_BASE_DIR="$BASE_DIR/action"ACTION_LOG_DIR="$ACTION_BASE_DIR/log"# FilesOUTPUT_BASENAME="$(date +%Y%m%d-%H%M%S.%N)"ACTION_LOGFILE_PATH="$ACTION_LOG_DIR/$OUTPUT_BASENAME.log"# ---------# Functions# ---------
action_log()echo"$(date-R)$*">>"$ACTION_LOGFILE_PATH"
action_check_directories()for _d in"$ACTION_BASE_DIR""$ACTION_LOG_DIR";do[-d"$_d"]mkdir"$_d"done
action_clean_directories()# Try to remove empty dirsfor _d in"$ACTION_LOG_DIR""$ACTION_BASE_DIR";do
if[-d"$_d"];then
rmdir"$_d" 2>/dev/null true
fi
done
mail_success()to_addr="$MAIL_TO_ADDR"if["$to_addr"];then
subject="OK - updated blogops site"
mail -s"$ MAIL_PREFIX $ subject""$to_addr" <"$ACTION_LOGFILE_PATH"fi
mail_failure()to_addr="$MAIL_TO_ADDR"if["$to_addr"];then
subject="KO - failed to update blogops site"
mail -s"$ MAIL_PREFIX $ subject""$to_addr" <"$ACTION_LOGFILE_PATH"fi
exit 1
# ----# MAIN# ----ret="0"# Check directories
action_check_directories
# Go to the base directorycd"$BASE_DIR"# Remove the old build dir if presentif[-d"$PUBLIC_DIR"];then
rm-rf"$PUBLIC_DIR"fi# Update the repository checkout
action_log "Updating the repository checkout"
git fetch --all>>"$ACTION_LOGFILE_PATH" 2>&1 ret="$?"if["$ret"-ne"0"];then
action_log "Failed to update the repository checkout"
mail_failure
fi# Get it from the repo branch & extract it
action_log "Downloading and extracting last site version using 'git archive'"
git archive --remote="$REPO_URL""$REPO_BRANCH""$REPO_DIR"\tar xf - >>"$ACTION_LOGFILE_PATH" 2>&1 ret="$?"# Fail if public dir was missingif["$ret"-ne"0"][!-d"$PUBLIC_DIR"];then
action_log "Failed to download or extract site"
mail_failure
fi# Remove old public_html copies
action_log 'Removing old site versions, if present'
find $NGINX_BASE_DIR-mindepth 1 -maxdepth 1 -name'public_html-*'-type d \-execrm-rf\;>>"$ACTION_LOGFILE_PATH" 2>&1 ret="$?"if["$ret"-ne"0"];then
action_log "Removal of old site versions failed"
mail_failure
fi# Switch site directoryTS="$(date +%Y%m%d-%H%M%S)"if[-d"$PUBLIC_HTML_DIR"];then
action_log "Moving '$PUBLIC_HTML_DIR' to '$PUBLIC_HTML_DIR-$TS'"mv"$PUBLIC_HTML_DIR""$PUBLIC_HTML_DIR-$TS">>"$ACTION_LOGFILE_PATH" 2>&1 ret="$?"fi
if["$ret"-eq"0"];then
action_log "Moving '$PUBLIC_DIR' to '$PUBLIC_HTML_DIR'"mv"$PUBLIC_DIR""$PUBLIC_HTML_DIR">>"$ACTION_LOGFILE_PATH" 2>&1 ret="$?"fi
if["$ret"-ne"0"];then
action_log "Site switch failed"
mail_failure
else
action_log "Site updated successfully"
mail_success
fi# ----# vim: ts=2:sw=2:et:ai:sts=2
The hugo-adoc workflowThe workflow is defined in the .forgejo/workflows/hugo-adoc.yml file and looks like this:
name:hugo-adoc# Run this job on push events to the main branchon:push:branches:-'main'jobs:build-and-push:if:$ vars.BLOGOPS_WEBHOOK_URL != '' && secrets.BLOGOPS_TOKEN != '' runs-on:dockercontainer:image:forgejo.mixinet.net/oci/hugo-adoc:latest# Allow the job to write to the repository (not really needed on forgejo)permissions:contents:writesteps:-name:Checkout the repouses:actions/checkout@v4with:submodules:'true'-name:Build the siteshell:shrun:rm -rf publichugo-name:Push compiled site to html branchshell:shrun:# Set the git usergit config --global user.email "blogops@mixinet.net"git config --global user.name "BlogOps"# Create a new orphan branch called html (it was not pulled by the# checkout step)git switch --orphan html# Add the public directory to the branchgit add public# Commit the changesgit commit --quiet -m "Updated site @ $(date -R)" public# Push the changes to the html branchgit push origin html --force# Switch back to the main branchgit switch main-name:Call the blogops update webhook endpointshell:shrun:HEADER="X-Blogops-Token: $ secrets.BLOGOPS_TOKEN "curl --fail -k -H "$HEADER" $ vars.BLOGOPS_WEBHOOK_URL
The only relevant thing is that we have to add the BLOGOPS_TOKEN variable to the project secrets (its value is the one
included on the /etc/webhook.yml file created when installing the webhook service) and the BLOGOPS_WEBHOOK_URL
project variable (its value is the URL of the webhook server, in my case
http://172.31.31.1:4444/hooks/update-blogops); note that the job includes the -k flag on the curl command just in
case I end up using TLS on the webhook server in the future, as discussed previously.
ConclusionNow that I have forgejo actions on my server I no longer need to build the site on the public server as I did initially,
a good thing when the server is a small OVH VPS that only runs a couple of containers and a web server directly on the
host.
I m still using a notification system to make the server run a script to update the site because that way the forgejo
server does not need access to the remote machine shell, only the webhook server which, IMHO, is a more secure setup.
Last week I decided I wanted to try out forgejo actions to build this blog instead of using
webhooks, so I looked the documentation and started playing with it until I had it working as I wanted.
This post is to describe how I ve installed and configured a forgejo runner, how I ve added an
oci organization to my instance to build, publish and mirror container images and added a couple of
additional organizations (actions and docker for now) to mirror interesting
actions.
The changes made to build the site using actions will be documented on a separate post, as I ll be using this entry to
test the new setup on the blog project.
Installing the runnerThe first thing I ve done is to install a runner on my server, I decided to use the
OCI image installation method, as it seemed to be the easiest and fastest
one.
The commands I ve used to setup the runner are the following:
$ cd /srv
$ git clone https://forgejo.mixinet.net/blogops/forgejo-runner.git
$ cd forgejo-runner
$ sh ./bin/setup-runner.sh
The setup-runner.sh script does multiple things:
create a forgejo-runner user and group
create the necessary directories for the runner
create a .runner file with a predefined secret and the docker label
The setup-runner.sh code is available here.
After running the script the runner has to be registered with the forgejo server, it can be done using the following
command:
The RUNNER_NAME variable is defined on the setup-runner.sh script and the FORGEJO_SECRET must match the value used
on the .runner file.
Starting it with docker-composeTo launch the runner I m going to use a docker-compose.yml file that starts two containers, a docker in docker
service to run the containers used by the workflow jobs and another one that runs the forgejo-runner itself.
The initial version used a TCP port to communicate with the dockerd server from the runner, but when I tried to build
images from a workflow I noticed that the containers launched by the runner were not going to be able to execute
another dockerd inside the dind one and, even if they were, it was going to be expensive computationally.
To avoid the issue I modified the dind service to use a unix socket on a shared volume that can be used by the
runner service to communicate with the daemon and also re-shared with the job containers so the dockerd server can
be used from them to build images.
Warning: The use of the same docker server that runs the jobs from them has security implications, but this instance is
for a home server where I am the only user, so I am not worried about it and this way I can save some resources (in
fact, I could use the host docker server directly instead of using a dind service, but just in case I want to run
other containers on the host I prefer to keep the one used for the runner isolated from it).
For those concerned about sharing the same server an alternative would be to launch a second dockerd only for the jobs
(i.e. actions-dind) using the same approach (the volume with its socket will have to be shared with the runner
service so it can be re-shared, but the runner does not need to use it).
There are multiple things to comment about this file:
The dockerd server is started with the -H unix:///dind/docker.sock flag to use the unix socket to communicate
with the daemon instead of using a TCP port (as said, it is faster and allows us to share the socket with the
containers started by the runner).
We are running the dockerd daemon with the RUNNER_GID group so the runner can communicate with it (the socket
gets that group which is the same used by the runner).
The runner container mounts three volumes: the data directory, the dind folder where docker creates the unix
socket and a config.yaml file used by us to change the default runner configuration.
The config.yaml file was originally created using the forgejo-runner:
$ docker run --rm data.forgejo.org/forgejo/runner:6.2.2 \
forgejo-runner generate-config > config.yaml
The changes to it are minimal, the runner capacity has been increased to 2 (that allows it to run two jobs at the
same time) and the /dind/docker.sock value has been added to the valid_volumes key to allow the containers launched
by the runner to mount it when needed; the diff against the default version is as follows:
@@ -13,7 +13,8 @@
# Where to store the registration result.
file: .runner
# Execute how many tasks concurrently at the same time.
- capacity: 1
+ # STO: Allow 2 concurrent tasks
+ capacity: 2
# Extra environment variables to run jobs.
envs:
A_TEST_ENV_NAME_1: a_test_env_value_1
@@ -87,7 +88,9 @@
# If you want to allow any volume, please use the following configuration:
# valid_volumes:
# - '**'
- valid_volumes: []
+ # STO: Allow to mount the /dind/docker.sock on the containers
+ valid_volumes:
+ - /dind/docker.sock
# overrides the docker client host with the specified one.
# If "-" or "", an available docker host will automatically be found.
# If "automount", an available docker host will automatically be found and ...
To start the runner we export the RUNNER_UID and RUNNER_GID variables and call docker-compose up to start the
containers on the background:
$ RUNNER_UID="$(id-u forgejo-runner)"RUNNER_GID="$(id-g forgejo-runner)"\
docker compose up -d
If the server was configured right we are now able to start using actions with this runner.
Preparing the system to run things locallyTo avoid unnecessary network traffic we are going to create a multiple organizations in our forgejo instance to maintain
our own actions and container images and mirror remote ones.
The rationale behind the mirror use is that we reduce a lot the need to connect to remote servers to download the
actions and images, which is good for performance and security reasons.
In fact, we are going to build our own images for some things to install the tools we want without needing to do it over
and over again on the workflow jobs.
Mirrored actionsThe actions we are mirroring are on the actions and docker organizations, we have
created the following ones for now (the mirrors were created using the forgejo web interface and we have disabled
manually all the forgejo modules except the code one for them):
To use our actions by default (i.e., without needing to add the server URL on the uses keyword) we have added the
following section to the app.ini file of our forgejo server:
Setting up credentials to push imagesTo be able to push images to the oci organization I ve created a token with package:write permission for my own
user because I m a member of the organization and I m authorized to publish packages on it (a different user could be
created, but as I said this is for personal use, so there is no need to complicate things for now).
To allow the use of those credentials on the actions I have added a secret (REGISTRY_PASS) and a variable
(REGISTRY_USER) to the oci organization to allow the actions to use them.
I ve also logged myself on my local docker client to be able to push images to the oci group by hand, as I it is
needed for bootstrapping the system (as I m using local images on the worflows I need to push them to the server before
running the ones that are used to build the images).
Local and mirrored imagesOur images will be stored on the packages section of a new organization called oci, inside it we have
created two projects that use forgejo actions to keep things in shape:
images: contains the source files used to generate our own images and the actions to build, tag and
push them to the oci organization group.
mirrors: contains a configuration file for the regsync tool to mirror containers and an
action to run it.
On the next sections we are going to describe the actions and images we have created and mirrored from those projects.
The oci/images projectThe images project is a monorepo that contains the source files for the images we are going to build and a couple of
actions.
The image sources are on sub directories of the repository, to be considered an image the folder has to contain a
Dockerfile that will be used to build the image.
The repository has two workflows:
build-image-from-tag: Workflow to build, tag and push an image to the oci organization
multi-semantic-release: Workflow to create tags for the images using the multi-semantic-release tool.
As the workflows are already configured to use some of our images we pushed some of them from a checkout of the
repository using the following commands:
On the next sub sections we will describe what the workflows do and will show their source code.
build-image-from-tag workflowThis workflow uses a docker client to build an image from a tag on the repository with the format
image-name-v[0-9].[0-9].[0-9]+.
As the runner is executed on a container (instead of using lxc) it seemed unreasonable to run another dind
container from that one, that is why, after some tests, I decided to share the dind service server socket with the
runner container and enabled the option to mount it also on the containers launched by the runner when needed (I only
do it on the build-image-from-tag action for now).
The action was configured to run using a trigger or when new tags with the right format were created, but when the tag
is created by multi-semantic-release the trigger does not work for some reason, so now it only runs the job on
triggers and checks if it is launched for a tag with the right format on the job itself.
The source code of the action is as follows:
name:build-image-from-tagon:workflow_dispatch:jobs:build:# Don't build the image if the registry credentials are not set, the ref is not a tag or it doesn't contain '-v'if:$ vars.REGISTRY_USER != '' && secrets.REGISTRY_PASS != '' && startsWith(github.ref, 'refs/tags/') && contains(github.ref, '-v') runs-on:dockercontainer:image:forgejo.mixinet.net/oci/node-mixinet:latest# Mount the dind socket on the container at the default locationoptions:-v /dind/docker.sock:/var/run/docker.socksteps:-name:Extract image name and tag from git and get registry name from envid:job_datarun:echo "::set-output name=img_name::$ GITHUB_REF_NAME%%-v* "echo "::set-output name=img_tag::$ GITHUB_REF_NAME##*-v "echo "::set-output name=registry::$(echo "$ github.server_url " sed -e 's%https://%%')"echo "::set-output name=oci_registry_prefix::$(echo "$ github.server_url /oci" sed -e 's%https://%%')"-name:Checkout the repouses:actions/checkout@v4-name:Export build dir and Dockerfileid:build_datarun:img="$ steps.job_data.outputs.img_name "build_dir="$(pwd)/$ img "dockerfile="$ build_dir /Dockerfile"if [ -f "$dockerfile" ]; thenecho "::set-output name=build_dir::$build_dir"echo "::set-output name=dockerfile::$dockerfile"elseecho "Couldn't find the Dockerfile for the '$img' image"exit 1fi-name:Login to the Container Registryuses:docker/login-action@v3with:registry:$ steps.job_data.outputs.registry username:$ vars.REGISTRY_USER password:$ secrets.REGISTRY_PASS -name:Set up Docker Buildxuses:docker/setup-buildx-action@v3-name:Build and Pushuses:docker/build-push-action@v6with:push:truetags:$ steps.job_data.outputs.oci_registry_prefix /$ steps.job_data.outputs.img_name :$ steps.job_data.outputs.img_tag $ steps.job_data.outputs.oci_registry_prefix /$ steps.job_data.outputs.img_name :latestcontext:$ steps.build_data.outputs.build_dir file:$ steps.build_data.outputs.dockerfile build-args:OCI_REGISTRY_PREFIX=$ steps.job_data.outputs.oci_registry_prefix /
Some notes about this code:
The if condition of the build job is not perfect, but it is good enough to avoid wrong uses as long as nobody
uses manual tags with the wrong format and expects things to work (it checks if the REGISTRY_USER and
REGISTRY_PASS variables are set, if the ref is a tag and if it contains the -v string).
To be able to access the dind socket we mount it on the container using the options key on the container section
of the job (this only works if supported by the runner configuration as explained before).
We use the job_data step to get information about the image from the tag and the registry URL from the environment
variables, it is executed first because all the information is available without checking out the repository.
We use the job_data step to get the build dir and Dockerfile paths from the repository (right now we are
assuming fixed paths and checking if the Dockerfile exists, but in the future we could use a configuration file to
get them, if needed).
As we are using a docker daemon that is already running there is no need to use the
docker/setup-docker-action to install it.
On the build and push step we pass the OCI_REGISTRY_PREFIX build argument to the Dockerfile to be able to use it
on the FROM instruction (we are using it in our images).
multi-semantic-release workflowThis workflow is used to run the multi-semantic-release tool on pushes to the main branch.
It is configured to create the configuration files on the fly (it prepares things to tag the folders that contain a
Dockerfile using a couple of template files available on the repository s .forgejo directory) and run the
multi-semantic-release tool to create tags and push them to the repository if new versions are to be built.
Initially we assumed that the tag creation pushed by multi-semantic-release would be enough to run the
build-tagged-image-task action, but as it didn t work we removed the rule to run the action on tag creation and added
code to trigger the action using an api call for the newly created tags (we get them from the output of the
multi-semantic-release execution).
The source code of the action is as follows:
name:multi-semantic-releaseon:push:branches:-'main'jobs:multi-semantic-release:runs-on:dockercontainer:image:forgejo.mixinet.net/oci/multi-semantic-release:lateststeps:-name:Checkout the repouses:actions/checkout@v4-name:Generate multi-semantic-release configurationshell:shrun:# Get the list of images to work with (the folders that have a Dockerfile)images="$(for img in */Dockerfile; do dirname "$img"; done)"# Generate a values.yaml file for the main packages.json filepackage_json_values_yaml=".package.json-values.yaml"echo "images:" >"$package_json_values_yaml"for img in $images; doecho " - $img" >>"$package_json_values_yaml"doneecho "::group::Generated values.yaml for the project"cat "$package_json_values_yaml"echo "::endgroup::"# Generate the package.json file validating that is a good json file with jqtmpl -f "$package_json_values_yaml" ".forgejo/package.json.tmpl" jq . > "package.json"echo "::group::Generated package.json for the project"cat "package.json"echo "::endgroup::"# Remove the temporary values filerm -f "$package_json_values_yaml"# Generate the package.json file for each imagefor img in $images; dotmpl -v "img_name=$img" -v "img_path=$img" ".forgejo/ws-package.json.tmpl" jq . > "$img/package.json"echo "::group::Generated package.json for the '$img' image"cat "$img/package.json"echo "::endgroup::"done-name:Run multi-semantic-releaseshell:shrun:multi-semantic-release tee .multi-semantic-release.log-name:Trigger buildsshell:shrun:# Get the list of tags published on the previous stepstags="$(sed -n -e 's/^\[.*\] \[\(.*\)\] .* Published release \([0-9]\+\.[0-9]\+\.[0-9]\+\) on .*$/\1-v\2/p' \.multi-semantic-release.log)"rm -f .multi-semantic-release.logif [ "$tags" ]; then# Prepare the url for building the imagesworkflow="build-image-from-tag.yaml"dispatch_url="$ github.api_url /repos/$ github.repository /actions/workflows/$workflow/dispatches"echo "$tags" while read -r tag; doecho "Triggering build for tag '$tag'"curl \-H "Content-Type:application/json" \-H "Authorization: token $ secrets.GITHUB_TOKEN " \-d " \"ref\":\"$tag\" " "$dispatch_url"donefi
Notes about this code:
The use of the tmpl tool to process the multi-semantic-release configuration templates comes from previous uses,
but on this case we could use a different approach (i.e. envsubst could be used) but we left it because it keeps
things simple and can be useful in the future if we want to do more complex things with the template files.
We use tee to show and dump to a file the output of the multi-semantic-release execution.
We get the list of pushed tags using sed against the output of the multi-semantic-release execution and for
each one found we use curl to call the forgejo API to trigger the build job; as the call is against the same
project we can use the GITHUB_TOKEN generated for the workflow to do it, without creating a user token that has to
be shared as a secret.
The .forgejo/package.json.tmpl file is the following one:
"name":"multi-semantic-release","version":"0.0.0-semantically-released","private":true,"multi-release":"tagFormat":"$ name -v$ version " ,"workspaces":.imagestoJson
As can be seen it only needs a list of paths to the images as argument (the file we generate contains the names and
paths, but it could be simplified).
And the .forgejo/ws-package.json.tmpl file is the following one:
The oci/mirrors projectThe repository contains a template for the configuration file we are going to use with regsync
(regsync.envsubst.yml) to mirror images from remote registries using a workflow that generates a configuration file
from the template and runs the tool.
The initial version of the regsync.envsubst.yml file is prepared to mirror alpine containers from version 3.21 to
3.29 (we explicitly remove version 3.20) and needs the forgejo.mixinet.net/oci/node-mixinet:latest image to run
(as explained before it was pushed manually to the server):
mirror workflowThe mirror workflow creates a configuration file replacing the value of the REGISTRY environment variable (computed
by removing the protocol from the server_url), the REGISTRY_USER organization value and the REGISTRY_PASS secret
using the envsubst command and running the regsync tool to mirror the images using the configuration file.
The action is configured to run daily, on push events when the regsync.envsubst.yml file is modified on the main
branch and can also be triggered manually.
The source code of the action is as follows:
.forgejo/workflows/mirror.yaml
ConclusionWe have installed a forgejo-runner and configured it to run actions for our own server and things are working fine.
This approach allows us to have a powerful CI/CD system on a modest home server, something very useful for maintaining
personal projects and playing with things without needing SaaS platforms like github or
gitlab.
A few months ago I explained that one reason why this blog has become more quiet is that all my work on Lean is covered elsewhere.
This post is an exception, because it is an observation that is (arguably) interesting, but does not lead anywhere, so where else to put it than my own blog
Want to share your thoughts about this? Please join the discussion on the Lean community zulip!
Background
When defining a function recursively in Lean that has nested recursion, e.g. a recusive call that is in the argument to a higher-order function like List.map, then extra attention used to be necessary so that Lean can see that xs.map applies its argument only elements of the list xs. The usual idiom is to write xs.attach.map instead, where List.attach attaches to the list elements a proof that they are in that list. You can read more about this my Lean blog post on recursive definitions and our new shiny reference manual, look for Example Nested Recursion in Higher-order Functions .
To make this step less tedious I taught Lean to automatically rewrite xs.map to xs.attach.map (where suitable) within the construction of well-founded recursion, so that nested recursion just works (issue #5471). We already do such a rewriting to change if c then else to the dependent if h : c then else , but the attach-introduction is much more ambitious (the rewrites are not definitionally equal, there are higher-order arguments etc.) Rewriting the terms in a way that we can still prove the connection later when creating the equational lemmas is hairy at best. Also, we want the whole machinery to be extensible by the user, setting up their own higher order functions to add more facts to the context of the termination proof.
I implemented it like this (PR #6744) and it ships with 4.18.0, but in the course of this work I thought about a quite different and maybe better way to do this, and well-founded recursion in general:
WellFounded.fix : (hwf : WellFounded r) (F : (x : ) ((y : ) r y x C y) C x) (x : ) : C x
we have to rewrite the functorial of the recursive function, which naturally has type
F : ((y : ) C y) ((x : ) C x)
to the one above, where all recursive calls take the termination proof r y x. This is a fairly hairy operation, mangling the type of matcher s motives and whatnot.
Things are simpler for recursive definitions using the new partial_fixpoint machinery, where we use Lean.Order.fix
so the functorial s type is unmodified (here will be ((x : ) C x)), and everything else is in the propositional side-condition montone F. For this predicate we have a syntax-guided compositional tactic, and it s easily extensible, e.g. by
theorem monotone_mapM (f : m ) (xs : List ) (hmono : monotone f) :
monotone (fun x => xs.mapM (f x))
Once given, we don t care about the content of that proof. In particular proving the unfolding theorem only deals with the unmodified F that closely matches the function definition as written by the user. Much simpler!
Isabelle has it easier
Isabelle also supports well-founded recursion, and has great support for nested recursion. And it s much simpler!
There, all you have to do to make nested recursion work is to define a congruence lemma of the form, for List.map something like our List.map_congr_left
List.map_congr_left : (h : a l, f a = g a) :
List.map f l = List.map g l
This is because in Isabelle, too, the termination proofs is a side-condition that essentially states the functorial F calls its argument f only on smaller arguments .
Can we have it easy, too?
I had wished we could do the same in Lean for a while, but that form of congruence lemma just isn t strong enough for us.
But maybe there is a way to do it, using an existential to give a witness that F can alternatively implemented using the more restrictive argument. The following callsOn P F predicate can express that F calls its higher-order argument only on arguments that satisfy the predicate P:
section setup
variable : Sort u
variable : Sort v
variable : Sort w
def callsOn (P : Prop) (F : ( y, y) ) :=
(F': ( y, P y y) ), f, F' (fun y _ => f y) = F f
variable (R : Prop)
variable (F : ( y, y) ( x, x))
local infix:50 " " => R
def recursesVia : Prop := x, callsOn ( x) (fun f => F f x)
noncomputable def fix (wf : WellFounded R) (h : recursesVia R F) : ( x, x) :=
wf.fix (fun x => (h x).choose)
def fix_eq (wf : WellFounded R) h x :
fix R F wf h x = F (fix R F wf h) x := by
unfold fix
rw [wf.fix_eq]
apply (h x).choose_spec
This allows nice compositional lemmas to discharge callsOn predicates:
theorem callsOn_base (y : ) (hy : P y) :
callsOn P (fun (f : x, x) => f y) := by
exists fun f => f y hy
intros; rfl
@[simp]
theorem callsOn_const (x : ) :
callsOn P (fun (_ : x, x) => x) :=
fun _ => x, fun _ => rfl
theorem callsOn_app
: Sort uu : Sort ww
(F : ( y, y) ) -- can this also support dependent types?
(F : ( y, y) )
(h : callsOn P F )
(h : callsOn P F ) :
callsOn P (fun f => F f (F f)) := by
obtain F ', h := h
obtain F ', h := h
exists (fun f => F ' f (F ' f))
intros; simp_all
theorem callsOn_lam
: Sort uu
(F : ( y, y) ) -- can this also support dependent types?
(h : x, callsOn P (F x)) :
callsOn P (fun f x => F x f) := by
exists (fun f x => (h x).choose f)
intro f
ext x
apply (h x).choose_spec
theorem callsOn_app2
: Sort uu : Sort ww
(g : )
(F : ( y, y) ) -- can this also support dependent types?
(F : ( y, y) )
(h : callsOn P F )
(h : callsOn P F ) :
callsOn P (fun f => g (F f) (F f)) := by
apply_rules [callsOn_app, callsOn_const]
With this setup, we can have the following, possibly user-defined, lemma expressing that List.map calls its arguments only on elements of the list:
theorem callsOn_map ( : Type uu) ( : Type ww)
(P : Prop) (F : ( y, y) ) (xs : List )
(h : x, x xs callsOn P (fun f => F f x)) :
callsOn P (fun f => xs.map (fun x => F f x)) := by
suffices callsOn P (fun f => xs.attach.map (fun x, h => F f x)) by
simpa
apply callsOn_app
apply callsOn_app
apply callsOn_const
apply callsOn_lam
intro x', hx'
dsimp
exact (h x' hx')
apply callsOn_const
end setup
So here is the (manual) construction of a nested map for trees:
section examples
structure Tree ( : Type u) where
val :
cs : List (Tree )
-- essentially
-- def Tree.map (f : ) : Tree Tree :=
-- fun t => f t.val, t.cs.map Tree.map )
noncomputable def Tree.map (f : ) : Tree Tree :=
fix (sizeOf < sizeOf ) (fun map t => f t.val, t.cs.map map )
(InvImage.wf (sizeOf ) WellFoundedRelation.wf) < by
intro v, cs
dsimp only
apply callsOn_app2
apply callsOn_const
apply callsOn_map
intro t' ht'
apply callsOn_base
-- ht' : t' cs -- !
-- sizeOf t' < sizeOf val := v, cs := cs
decreasing_trivial
end examples
This makes me happy!
All details of the construction are now contained in a proof that can proceed by a syntax-driven tactic and that s easily and (likely robustly) extensible by the user. It also means that we can share a lot of code paths (e.g. everything related to equational theorems) between well-founded recursion and partial_fixpoint.
I wonder if this construction is really as powerful as our current one, or if there are certain (likely dependently typed) functions where this doesn t fit, but the above is dependent, so it looks good.
With this construction, functions defined by well-founded recursion will reduce even worse in the kernel, I assume. This may be a good thing.
The cake is a lie
What unfortunately kills this idea, though, is the generation of the functional induction principles, which I believe is not (easily) possible with this construction: The functional induction principle is proved by massaging F to return a proof, but since the extra assumptions (e.g. for ite or List.map) only exist in the termination proof, they are not available in F.
Oh wey, how anticlimactic.
PS: Path dependencies
Curiously, if we didn t have functional induction at this point yet, then very likely I d change Lean to use this construction, and then we d either not get functional induction, or it would be implemented very differently, maybe a more syntactic approach that would re-prove termination. I guess that s called path dependence.
The other day, I noted that the emacs integration with debputy stopped working.
After debugging for a while, I realized that emacs no longer sent the didOpen
notification that is expected of it, which confused debputy. At this point, I was
already several hours into the debugging and I noted there was some discussions on
debian-devel about emacs and byte compilation not working. So I figured I would
shelve the emacs problem for now.
But I needed an LSP capable editor and with my vi skills leaving much to be desired,
I skipped out on vim-youcompleteme. Instead, I pulled out kate, which I had not
been using for years. It had LSP support, so it would fine, right?
Well, no. Turns out that debputy LSP support had some assumptions that worked for
emacs but not kate. Plus once you start down the rabbit hole, you stumble on
things you missed previously.
Getting started
First order of business was to tell kate about debputy. Conveniently, kate has
a configuration tab for adding language servers in a JSON format right next to the tab where
you can see its configuration for built-in LSP (also in JSON format9. So a quick bit of
copy-paste magic and that was done.
Yesterday, I opened an MR against upstream to have the configuration added
(https://invent.kde.org/utilities/kate/-/merge_requests/1748) and they already merged it.
Today, I then filed a wishlist against kate in Debian to have the Debian maintainers
cherry-pick it, so it works out of the box for Trixie (https://bugs.debian.org/1099876).
So far so good.
Inlay hint woes
Since July (2024), debputy has support for Inlay hints. They are basically small
bits of text that the LSP server can ask the editor to inject into the text to provide
hints to the reader.
Typically, you see them used to provide typing hints, where the editor or the underlying
LSP server has figured out the type of a variable or expression that you did not
explicitly type. Another common use case is to inject the parameter name for positional
arguments when calling a function, so the user do not have to count the position to
figure out which value is passed as which parameter.
In debputy, I have been using the Inlay hints to show inherited fields in
debian/control. As an example, if you have a definition like:
Source: foo-src
Section: devel
Priority: optional
Package: foo-bin
Architecture: any
Then foo-bin inherits the Section and Priority field since it does not supply
its own. Previously, debputy would that by injecting the fields themselves and their
value just below the Package field as if you had typed them out directly. The editor
always renders Inlay hints distinctly from regular text, so there was no risk of
confusion and it made the text look like a valid debian/control file end to end. The
result looked something like:
With the second instances of Section and Priority being rendered differently than
its surrendering (usually faded or colorlessly).
Unfortunately, kate did not like injecting Inlay hints with a newline in them,
which was needed for this trick. Reading into the LSP specs, it says nothing about
multi-line Inlay hints being a thing and I figured I would see this problem again
with other editors if I left it be.
I ended up changing the Inlay hints to be placed at the end of the Package field
and then included surrounding () for better visuals. So now, it looks like:
Unfortunately, it is no longer 1:1 with the underlying syntax which I liked about the
previous one. But it works in more editors and is still explicit. I also removed the
Inlay hint for the Homepage field. It takes too much space and I have yet to
meet someone missing it in the binary stanza.
If you have any better ideas for how to render it, feel free to reach out to me.
Spurious completion and hover
As I was debugging the Inlay hints, I wanted to do a quick restart of debputy after
each fix. Then I would trigger a small change to the document to ensure kate would
request an update from debputy to render the Inlay hints with the new code.
The full outgoing payloads are sent via the logs to the client, so it was really about
minimizing which LSP requests are sent to debputy. Notably, two cases would flood the
log:
Completion requests. These are triggered by typing anything at all and since I wanted
to a change, I could not avoid this. So here it was about making sure there would be
nothing to complete, so the result was a small as possible.
Hover doc requests. These are triggered by mouse hovering over field, so this was
mostly about ensuring my mouse movement did not linger over any field on the way
between restarting the LSP server and scrolling the log in kate.
In my infinite wisdom, I chose to make a comment line where I would do the change. I figured
it would neuter the completion requests completely and it should not matter if my cursor
landed on the comment as there would be no hover docs for comments either.
Unfortunately for me, debputy would ignore the fact that it was on a comment line.
Instead, it would find the next field after the comment line and try to complete based on
that. Normally you do not see this, because the editor correctly identifies that none of
the completion suggestions start with a \#, so they are all discarded.
But it was pretty annoying for the debugging, so now debputy has been told to explicitly
stop these requests early on comment lines.
Hover docs for packages
I added a feature in debputy where you can hover over package names in your relationship
fields (such as Depends) and debputy will render a small snippet about it based on
data from your local APT cache.
This doc is then handed to the editor and tagged as markdown provided the editor supports
markdown rendering. Both emacs and kate support markdown. However, not all
markdown renderings are equal. Notably, emacs's rendering does not reformat the text
into paragraphs. In a sense, emacs rendering works a bit like <pre>...</pre> except
it does a bit of fancy rendering inside the <pre>...</pre>.
On the other hand, kate seems to convert the markdown to HTML and then throw the result
into an HTML render engine. Here it is important to remember that not all newlines are equal
in markdown. A Foo<newline>Bar is treated as one "paragraph" (<p>...</p>) and the HTML
render happily renders this as single line Foo Bar provided there is sufficient width to
do so.
A couple of extra newlines made wonders for the kate rendering, but I have a feeling this
is not going to be the last time the hover docs will need some tweaking for prettification.
Feel free to reach out if you spot a weirdly rendered hover doc somewhere.
Making quickfixes available in kate
Quickfixes are treated as generic code actions in the LSP specs. Each code action has a "type"
(kind in the LSP lingo), which enables the editor to group the actions accordingly or
filter by certain types of code actions.
The design in the specs leads to the following flow:
The LSP server provides the editor with diagnostics (there are multiple ways to trigger
this, so we will keep this part simple).
The editor renders them to the user and the user chooses to interact with one of them.
The interaction makes the editor asks the LSP server, which code actions are available
at that location (optionally with filter to only see quickfixes).
The LSP server looks at the provided range and is expected to return the relevant
quickfixes here.
This flow is really annoying from a LSP server writer point of view. When you do the diagnostics
(in step 1), you tend to already know what the possible quickfixes would be. The LSP spec
authors realized this at some point, so there are two features the editor provides to simplify
this.
In the editor request for code actions, the editor is expected to provide the diagnostics
that they received from the server. Side note: I cannot quite tell if this is optional or
required from the spec.
The editor can provide support for remembering a data member in each diagnostic. The
server can then store arbitrary information in that member, which they will see again in
the code actions request. Again, provided that the editor supports this optional feature.
All the quickfix logic in debputy so far has hinged on both of these two features.
As life would have it, kate provides neither of them.
Which meant I had to teach debputy to keep track of its diagnostics on its own. The plus side
is that makes it easier to support "pull diagnostics" down the line, since it requires a similar
feature. Additionally, it also means that quickfixes are now available in more editors. For
consistency, debputy logic is now always used rather than relying on the editor support
when present.
The downside is that I had to spend hours coming up with and debugging a way to find the
diagnostics that overlap with the range provided by the editor. The most difficult part was keeping
the logic straight and getting the runes correct for it.
Making the quickfixes actually work
With all of that, kate would show the quickfixes for diagnostics from debputy and you could
use them too. However, they would always apply twice with suboptimal outcome as a result.
The LSP spec has multiple ways of defining what need to be changed in response to activating a
code action. In debputy, all edits are currently done via the WorkspaceEdit type. It
has two ways of defining the changes. Either via changes or documentChanges with
documentChanges being the preferred one if both parties support this.
I originally read that as I was allowed to provide both and the editor would pick the one it
preferred. However, after seeing kate blindly use both when they are present, I reviewed
the spec and it does say "The edit should either provide changes or documentChanges",
so I think that one is on me.
None of the changes in debputy currently require documentChanges, so I went with just
using changes for now despite it not being preferred. I cannot figure
out the logic of whether an editor supports documentChanges. As I read the notes for this
part of the spec, my understanding is that kate does not announce its support for
documentChanges but it clearly uses them when present. Therefore, I decided to keep it
simple for now until I have time to dig deeper.
Remaining limitations with kate
There is one remaining limitation with kate that I have not yet solved. The kate
program uses KSyntaxHighlighting for its language detection, which in turn is the
basis for which LSP server is assigned to a given document.
This engine does not seem to support as complex detection logic as I hoped from it. Concretely,
it either works on matching on an extension / a basename (same field for both cases) or
mime type. This combined with our habit in Debian to use extension less files like
debian/control vs. debian/tests/control or debian/rules or
debian/upstream/metadata makes things awkward a best.
Concretely, the syntax engine cannot tell debian/control from debian/tests/control as
they use the same basename. Fortunately, the syntax is close enough to work for both and
debputy is set to use filename based lookups, so this case works well enough.
However, for debian/rules and debian/upstream/metadata, my understanding is that if
I assign these in the syntax engine as Debian files, these rules will also trigger for any
file named foo.rules or bar.metadata. That seems a bit too broad for me, so I have
opted out of that for now. The down side is that these files will not work out of the box
with kate for now.
The current LSP configuration in kate does not recognize makefiles or YAML either. Ideally,
we would assign custom languages for the affected Debian files, so we do not steal the ID
from other language servers. Notably, kate has a built-in language server for YAML and
debputy does nothing for a generic YAML document. However, adding YAML as a supported
language for debputy would cause conflict and regressions for users that are already
happy with their generic YAML language server from kate.
So there are certainly still work to be done. If you are good with KSyntaxHighlighting
and know how to solve some of this, I hope you will help me out.
Changes unrelated to kate
While I was working on debputy, I also added some other features that I want to mention.
The debputy lint command will now show related context to diagnostic in its terminal
report when such information is available and is from the same file as the diagnostic
itself (cross file cases are rendered without related information).
The related information is typically used to highlight a source of a conflict. As an
example, if you use the same field twice in a stanza of debian/control, then
debputy will add a diagnostic to the second occurrence. The related information
for that diagnostic would provide the position of the first occurrence.
This should make it easier to find the source of the conflict in the cases where
debputy provides it. Let me know if you are missing it for certain diagnostics.
The diagnostics analysis of debian/control will now identify and flag simple
duplicated relations (complex ones like OR relations are ignored for now). Thanks
to Matthias Geiger for suggesting the feature and Otto Kek l inen for reporting
a false positive that is now fixed.
Closing
I am glad I tested with kate to weed out most of these issues in time before
the freeze. The Debian freeze will start within a week from now. Since debputy
is a part of the toolchain packages it will be frozen from there except for
important bug fixes.
From April 27th to 30th, 2024,
MiniDebConf Belo Horizonte 2024 was held at
the Pampulha Campus of
UFMG - Federal University of Minas Gerais, in Belo
Horizonte city.
This was the fifth time that a MiniDebConf (as an exclusive in-person event
about Debian) took place in Brazil. Previous editions were in Curitiba
(2016,
2017, and
2018), and in
Bras lia 2023. We had other MiniDebConfs
editions held within Free Software events such as
FISL and Latinoware, and other
online events. See our
event history.
Parallel to MiniDebConf, on 27th (Saturday)
FLISOL - Latin American Free Software Installation Festival took place. It's the largest event in Latin America to promote Free Software,
and It has been held since 2005 simultaneously in several cities.
MiniDebConf Belo Horizonte 2024 was a success (as were previous editions) thanks to the participation of everyone, regardless of their level of knowledge about
Debian. We value the presence of both beginner users who are familiarizing
themselves with the system and the official project developers. The spirit of
welcome and collaboration was present during all the event.
2024 edition numbers
During the four days of the event, several activities took place for all
levels of users and collaborators of the Debian project. The official schedule
was composed of:
06 rooms in parallel on Saturday;
02 auditoriums in parallel on Monday and Tuesday;
30 talks/BoFs of all levels;
05 workshops for hands-on activities;
09 lightning talks on general topics;
01 Live Electronics performance with Free Software;
Install fest to install Debian on attendees' laptops;
BSP (Bug Squashing Party);
Uploads of new or updated packages.
The final numbers for MiniDebConf Belo Horizonte 2024 show that we had a
record number of participants.
Total people registered: 399
Total attendees in the event: 224
Of the 224 participants, 15 were official Brazilian contributors,
10 being DDs (Debian Developers) and 05 (Debian Maintainers), in addition to
several unofficial contributors.
The organization was carried out by 14 people who started working at the end of
2023, including Prof. Lo c Cerf from the Computing Department who made the event possible at UFMG, and 37 volunteers who helped during the event.
As MiniDebConf was held at UFMG facilities, we had the help of more than
10 University employees.
See the list with the
names of people who helped in some way in organizing MiniDebConf Belo Horizonte
2024.
The difference between the number of people registered and the number of
attendees in the event is probably explained by the fact that there is no
registration fee, so if the person decides not to go to the event, they will
not suffer financial losses.
The 2024 edition of MiniDebconf Belo Horizonte was truly grand and shows the
result of the constant efforts made over the last few years to attract more
contributors to the Debian community in Brazil. With each edition the numbers
only increase, with more attendees, more activities, more rooms, and more
sponsors/supporters.
Activities
The MiniDebConf schedule was intense and diverse. On the 27th, 29th and 30th
(Saturday, Monday and Tuesday) we had talks, discussions, workshops and many
practical activities.
On the 28th (Sunday), the Day Trip took place, a day dedicated to sightseeing
around the city. In the morning we left the hotel and went, on a chartered bus,
to the
Belo Horizonte Central Market. People took
the opportunity to buy various things such as cheeses, sweets, cacha as and
souvenirs, as well as tasting some local foods.
After a 2-hour tour of the Market, we got back on the bus and hit the road for
lunch at a typical Minas Gerais food restaurant.
With everyone well fed, we returned to Belo Horizonte to visit the city's
main tourist attraction: Lagoa da Pampulha and Capela S o Francisco de Assis,
better known as
Igrejinha da Pampulha.
We went back to the hotel and the day ended in the hacker space that we set up
in the events room for people to chat, packaging, and eat pizzas.
Crowdfunding
For the third time we ran a crowdfunding campaign and it was incredible how
people contributed! The initial goal was to raise the amount equivalent to a
gold tier of R$ 3,000.00. When we reached this goal, we defined a new one,
equivalent to one gold tier + one silver tier (R$ 5,000.00). And again we
achieved this goal. So we proposed as a final goal the value of a gold + silver
+ bronze tiers, which would be equivalent to R$ 6,000.00. The result was that
we raised R$7,239.65 (~ USD 1,400) with the help of more than 100 people!
Thank you very much to the people who contributed any amount. As a thank you, we list the names of the people who donated.
Food, accommodation and/or travel grants for participants
Each edition of MiniDebConf brought some innovation, or some different benefit
for the attendees. In this year's edition in Belo Horizonte, as with DebConfs, we offered bursaries for food, accommodation and/or travel to help those people who would like to come to the event but who would need
some kind of help.
In the registration form, we included the option for the person to request a
food, accommodation and/or travel bursary, but to do so, they would have to
identify themselves as a contributor (official or unofficial) to Debian and
write a justification for the request.
Number of people benefited:
Food: 69
Accommodation: 20
Travel: 18
The food bursary provided lunch and dinner every day. The lunches included
attendees who live in Belo Horizonte and the region. Dinners were paid for
attendees who also received accommodation and/or travel. The accommodation was
held at the BH Jaragu Hotel. And the
travels included airplane or bus tickets, or fuel (for those who came by car or
motorbike).
Much of the money to fund the bursaries came from the Debian Project, mainly
for travels. We sent a budget request to the former Debian leader Jonathan
Carter, and He promptly approved our request.
In addition to this event budget, the leader also approved individual requests
sent by some DDs who preferred to request directly from him.
The experience of offering the bursaries was really good because it allowed
several people to come from other cities.
Photos and videos
You can watch recordings of the talks at the links below:
Thanks
We would like to thank all the attendees, organizers, volunteers, sponsors and
supporters who contributed to the success of MiniDebConf Belo Horizonte 2024.
Sponsors
Gold:
Welcome to the second report in 2025 from the Reproducible Builds project. Our monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. As usual, however, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website.
Table of contents:
Reproducible Builds at FOSDEM 2025
Similar to last year s event, there was considerable activity regarding Reproducible Builds at FOSDEM 2025, held on on 1st and 2nd February this year in Brussels, Belgium. We count at least four talks related to reproducible builds. (You can also read our news report from last year s event in which Holger Levsen presented in the main track.)
Jelle van der Waa, Holger Levsen and kpcyrd presented in the Distributions track on A Tale of several distros joining forces for a common goal. In this talk, three developers from two different Linux distributions (Arch Linux and Debian), discuss this goal which is, of course, reproducible builds. The presenters discuss both what is shared and different between the two efforts, touching on the history and future challenges alike. The slides of this talk are available to view, as is the full video (30m02s). The talk was also discussed on Hacker News.
Zbigniew J drzejewski-Szmek presented in the ever-popular Python track a on Rewriting .pyc files for fun and reproducibility, i.e. the bytecode files generated by Python in order to speed up module imports: It s been known for a while that those are not reproducible: on different architectures, the bytecode for exactly the same sources ends up slightly different. The slides of this talk are available, as is the full video (28m32s).
In the Nix and NixOS track, Julien Malka presented on the Saturday asking How reproducible is NixOS: We know that the NixOS ISO image is very close to be perfectly reproducible thanks to reproducible.nixos.org, but there doesn t exist any monitoring of Nixpkgs as a whole. In this talk I ll present the findings of a project that evaluated the reproducibility of Nixpkgs as a whole by mass rebuilding packages from revisions between 2017 and 2023 and comparing the results with the NixOS cache. Unfortunately, no video of the talk is available, but there is a blog and article on the results.
Lastly, Simon Tournier presented in the Open Research track on the confluence of GNU Guix and Software Heritage: Source Code Archiving to the Rescue of Reproducible Deployment. Simon s talk describes design and implementation we came up and reports on the archival coverage for package source code with data collected over five years. It opens to some remaining challenges toward a better open and reproducible research. The slides for the talk are available, as is the full video (23m17s).
Reproducible Builds at PyCascades 2025
Vagrant Cascadian presented at this year s PyCascades conference which was held on February 8th and 9th February in Portland, OR, USA. PyCascades is a regional instance of PyCon held in the Pacific Northwest. Vagrant s talk, entitled Re-Py-Ducible Builds caught the audience s attention with the following abstract:
Crank your Python best practices up to 11 with Reproducible Builds! This talk will explore Reproducible Builds by highlighting issues identified in Python projects, from the simple to the seemingly inscrutable. Reproducible Builds is basically the crazy idea that when you build something, and you build it again, you get the exact same thing or even more important, if someone else builds it, they get the exact same thing too.
reproduce.debian.net updates
The last few months have seen the introduction of reproduce.debian.net. Announced first at the recent Debian MiniDebConf in Toulouse, reproduce.debian.net is an instance of rebuilderd operated by the Reproducible Builds project.
Powering this work is rebuilderd, our server which monitors the official package repositories of Linux distributions and attempt to reproduce the observed results there. This month, however, Holger Levsen:
Split packages that are not specific to any architecture away from amd64.reproducible.debian.net service into a new all.reproducible.debian.net page.
Increased the number of riscv64 nodes to a total of 4, and added a new amd64 node added thanks to our (now 10-year sponsor), IONOS.
Uploaded the devscripts package, incorporating changes from Jochen Sprickerhof to the debrebuild script specifically to fix the handling the Rules-Requires-Root header in Debian source packages.
Uploaded a number of Rust dependencies of rebuilderd (rust-libbz2-rs-sys, rust-actix-web, rust-actix-server, rust-actix-http, rust-actix-server, rust-actix-http, rust-actix-web-codegen and rust-time-tz) after they were prepared by kpcyrd :
Jochen Sprickerhof also updated the sbuild package to:
Obey requests from the user/developer for a different temporary directory.
Use the root/superuser for some values of Rules-Requires-Root.
Don t pass --root-owner-group to old versions of dpkg.
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
go (clear GOROOT for func ldShared when -trimpath is used)
Distribution work
There as been the usual work in various distributions this month, such as:
In Debian, 17 reviews of Debian packages were added, 6 were updated and 8 were removed this month adding to our knowledge about identified issues.
Fedora developers Davide Cavalca and Zbigniew J drzejewski-Szmek gave a talk on Reproducible Builds in Fedora (PDF), touching on SRPM-specific issues as well as the current status and future plans.
Thanks to an investment from the Sovereign Tech Agency, the FreeBSD project s work on unprivileged and reproducible builds continued this month. Notable fixes include:
The Yocto Project has been struggling to upgrade to the latest Go and Rust releases due to reproducibility problems in the newer versions. Hongxu Jia tracked down the issue with Go which meant that the project could upgrade from the 1.22 series to 1.24, with the fix being submitted upstream for review (see above). For Rust, however, the project was significantly behind, but has made recent progress after finally identifying the blocking reproducibility issues. At time of writing, the project is at Rust version 1.82, with patches under review for 1.83 and 1.84 and fixes being discussed with the Rust developers. The project hopes to improve the tests for reproducibility in the Rust project itself in order to try and avoid future regressions.
Yocto continues to maintain its ability to binary reproduce all of the recipes in OpenEmbedded-Core, regardless of the build host distribution or the current build path.
Finally, Douglas DeMaio published an article on the openSUSE blog on announcing that the Reproducible-openSUSE (RBOS) Project Hits [Significant] Milestone. In particular:
The Reproducible-openSUSE (RBOS) project, which is a proof-of-concept fork of openSUSE, has reached a significant milestone after demonstrating a usable Linux distribution can be built with 100% bit-identical packages.
diffoscope & strip-nondeterminismdiffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 288 and 289 to Debian:
Add asar to DIFFOSCOPE_FAIL_TESTS_ON_MISSING_TOOLS in order to address Debian bug #1095057) []
Catch a CalledProcessError when calling html2text. []
Additionally, Vagrant Cascadian updated diffoscope in GNU Guix to version 287 [][] and 288 [][] as well as submitted a patch to update to 289 []. Vagrant also fixed an issue that was breaking reprotest on Guix [][].
strip-nondeterminism is our sister tool to remove specific non-deterministic results from a completed build. This month version 1.14.1-2 was uploaded to Debian unstable by Holger Levsen.
Website updates
There were a large number of improvements made to our website this month, including:
Holger Levsen clarified the name of a link to our old Wiki pages on the History page [] and added a number of new links to the Talks & Resources page [][].
James Addison update the website s own README file to document a couple of additional dependencies [][], as well as did more work on a future Getting Started guide page [][].
Reproducibility testing framework
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In January, a number of changes were made by Holger Levsen, including:
Fix /etc/cron.d and /etc/logrotate.d permissions for Jenkins nodes. []
Add support for riscv64 architecture nodes. [][]
Grant Jochen Sprickerhof access to the o4 node. []
Disable the janitor-setup-worker. [][]
In addition:
kpcyrd fixed the /all/api/ API endpoints on reproduce.debian.net by altering the nginx configuration. []
James Addison updated reproduce.debian.net to display the so-called bad reasons hyperlink inline [] and merged the Categorized issues links into the Reproduced builds column [].
Jochen Sprickerhof also made some reproduce.debian.net-related changes, adding support for detecting a bug in the mmdebstrap package [] as well as updating some documentation [].
Roland Clobus continued their work on reproducible live images for Debian, making changes related to new clustering of jobs in openQA. []
And finally, both Holger Levsen [][][] and Vagrant Cascadian performed significant node maintenance. [][][][][]
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
Some of you may remember that I recently felt a bit underwhelmed
by the last pager I reverse engineered the Retekess TD-158,
mostly due to how intuitive their design decions were. It was pretty easy
to jump to conclusions because they had made some pretty good decisions on
how to do things.
I figured I d spin the wheel again and try a new pager system this time I
went for a SU-68G-10 pager, since I recognized the form factor as another
fairly common unit I ve seen around town. Off to Amazon I went, bought a set,
and got to work trying to track down the FCC filings on this model. I
eventually found what seemed to be the right make/model, and it, once again,
indicated that this system should be operating in the 433 MHz ISM band likely
using OOK modulation. So, figured I d start with the center of the band (again)
at 433.92 MHz, take a capture, test my luck, and was greeted with a now very
familiar sight.
Same as the last goarounds, except the premable here is a 0 symbol followed
by 6-ish symbol durations of no data, followed by 25 bits of a packet. Careful
readers will observe 26 symbols above after the preamble I did too! The last
0 in the screenshot above is not actually a part of the packet rather,
it s part of the next packet s preamble. Each packet is packed in pretty tight.
By Hand Demodulation
Going off the same premise as last time, I figured i d give it a manual demod
and see what shakes out (again). This is now the third time i ve run this play,
so check out either of my prior twoposts for a
better written description of what s going on here I ll skip all the details
since i d just be copy-pasting from those posts into here. Long story short, I
demodulated a call for pager 1, call for pager 10, and a power off command.
What
Bits
Call 1
1101111111100100100000000
Call 10
1101111111100100010100000
Off
1101111111100111101101110
A few things jump out at me here the first 14 bits are fixed (in my case,
11011111111001), which means some mix of preamble, system id, or other
system-wide constant. Additionally, The last 9 bits also look like they are our
pager the 1 and 10 pager numbers (LSB bit order) jump right out
(100000000 and 010100000, respectively). That just leaves the two remaining
bits which look to be the action 00 for a Call , and 11 for a Power
off . I don t super love this since command has two bits rather than one, the
base station ID seems really long, and a 9-bit Pager ID is just weird. Also,
what is up with that power-off pager id? Weird. So, let s go and see what we
can do to narrow down and confirm things by hand.
Testing bit flips
Rather than call it a day at that, I figure it s worth a bit of diligence to
make sure it s all correct so I figured we should try sending packets to
my pagers and see how they react to different messages after flipping bits
in parts of the packet.
I implemented a simple base station for the pagers using my Ettus B210mini, and
threw together a simple OOK modulator and transmitter program which allows me
to send specifically crafted test packets on frequency. Implementing the base
station is pretty straightforward, because of the modulation of the signal
(OOK), it s mostly a matter of setting a buffer to 1 and 0 for where the
carrier signal is on or off timed to the sample rate, and sending that off to
the radio. If you re interested in a more detailed writeup on the steps
involved, there s a bit more in my christmas tree post.
First off, I d like to check the base id. I want to know if all the bits in
what I m calling the base id are truly part of the base station ID, or
perhaps they have some other purpose (version, preamble?). I wound up following
a three-step process for every base station id:
Starting with an unmodified call packet for the pager under test:
Flip the Nth bit, and transmit the call. See if the pager reacts.
Hold SET , and pair the pager with the new packet.
Transmit the call. See if the pager reacts.
After re-setting the ID, transmit the call with the physical base station,
see if the pager reacts.
Starting with an unmodified off packet for the pager system
Flip the Nth bit, transmit the off, see if the pager reacts.
What wound up happening is that changing any bit in the first 14 bits meant
that the packet no longer worked with any pager until it was re-paired, at
which point it begun to work again. This likely means the first 14 bits are
part of the base station ID and not static between base stations, or some
constant like a version or something. All bits appear to be used.
I repeated the same process with the command bits, and found that only 11
and 00 caused the pagers to react for the pager ids i ve tried.
I repeated this process one last time with the pager id bits this time, and
found the last bit in the packet isn t part of the pager ID, and can be either
a 1 or a 0 and still cause the pager to react as if it were a 0. This means
that the last bit is unknown but it has no impact on either a power off or
call, and all messages sent by my base station always have a 0 set. It s not
clear if this is used by anything likely not since setting a bit there
doesn t result in any change of behavior I can see yet.
Final Packet Structure
After playing around with flipping bits and testing, the final structure
I was able to come up with based on behavior I was able to observe from
transmitting hand-crafted packets and watching pagers buzz:
base id
command
pager id
???
Commands
The command section bit comes in two flavors either a call or an off
command.
Type
Id (2 bits)
Description
Call
00
Call the pager identified by the id in pager id
Off
11
Request pagers power off, pager id is always 10110111
As for the actual RF PHY characteristics, here s my best guesses at what s
going on with them:
What
Description
Center Frequency
433.92 MHz
Modulation
OOK
Symbol Duration
1300us
Bits
25
Preamble
325us of carrier, followed by 8800us of no carrier
I m not 100% on the timings, but they appear to be close enough to work
reliabily. Same with the center frequency, it s roughly right but there
may be a slight difference i m missing.
Lingering Questions
This was all generally pretty understandable another system that had some
good decisions, and wasn t too bad to reverse engineer. This was a bit more fun
to do, since there was a bit more ambiguity here, but still not crazy. At least
this one was a bit more ambiguous that needed a bit of followup to confirm
things, which made it a bit more fun.
I am left with a few questions, though which I m kinda interested in
understanding, but I ll likely need a lot more data and/or original source:
Why is the command two bits here? This was a bit tough to understand because
of the number of bits they have at their disposal given the one last bit at
the end of the packet that doesn t seem to do anything, there s no reason this
couldn t have been a 16 bit base station id, and an 8 bit pager id along with a
single bit command (call or off).
When sending an off why is power off that bit pattern? Other pager IDs
don t seem to work with off , so it has some meaning, but I m not sure what
that is. You press and hold 9 on the physical base station, but the code winds
up coming out to 0xED, 237 or maybe -19 if it s signed. I can t quite
figure out why it s this value. Are there other codes?
Finally what s up with the last bit? Why is it 25 bits and not 24? It must
take more work to process something that isn t 8 bit aligned and all for
something that s not being used!
A Little Vice is a stand-alone self-published
magical girl novel. It
is the author's first novel.
C is a high school student and frequent near-victim of monster attacks.
Due to the nefarious work of Avaritia Wolf and her allies, his high school
is constantly attacked by Beasts, who are magical corruptions of some
internal desire taken to absurd extremes. Standing in their way are the
Angelic Saints: magical girls who transform into Saint Castitas, Saint
Diligentia, and Saint Temperantia and fight the monsters. The monsters for
some reason seem disposed to pick C as their victim for hostage-taking,
mind control, use as a human shield, and other rather traumatic
activities. He's always rescued by the Saints before any great harm is
done, but in some ways this makes the situation worse.
It is obvious to C that the Saints are his three friends Inessa, Ida, and
Temperance, even though no one else seems able to figure this out despite
the blatant clues. Inessa has been his best friend since childhood when
she was awkward and needed his support. Now, she and his other friends
have become literal heroes, beautiful and powerful and capable, constantly
protecting the school and innocent people, and C is little more than a
helpless burden to be rescued. More than anything else, he wishes he could
be an Angelic Saint like them, but of course the whole idea is impossible.
Boys don't get to be magical girls.
(I'm using he/him pronouns for C in this review because C uses them for
himself for most of the book.)
This is a difficult book to review because it is deeply focused on
portraying a specific internal emotional battle in all of its
sometimes-ugly complexity, and to some extent it prioritizes that
portrayal over conventional story-telling. You have probably already
guessed that this is a transgender coming-out story Elkin's choice of
the magical girl genre was done with deep understanding of its role in
transgender narratives but more than that, it is a transgender
coming-out story of a very specific and closely-observed type. C knows who
he wishes he was, but he is certain that this transformation is absolutely
impossible. He is very deep in a cycle of self-loathing for wanting
something so manifestly absurd and insulting to people who have the
virtues that C does not.
A Little Vice is told in the first person from C's perspective, and
most of this book is a relentless observation of C's anxiety and shame
spiral and reflexive deflection of any possibility of a way out. This is
very well-written: Elkin knows the reader is going to disagree with C's
internalized disgust and hopelessness, knows the reader desperately wants
C to break out of that mindset, and clearly signals in a myriad of adroit
ways that Elkin is on the reader's side and does not agree with C's
analysis. C's friends are sympathetic, good-hearted people, and while
sometimes oblivious, it is obvious to the reader that they're also on the
reader's side and would help C in a heartbeat if they saw an opening. But
much of the point of the book is that it's not that easy, that breaking
out of the internal anxiety spiral is nearly impossible, and that C is
very good at rejecting help, both because he cannot imagine what form it
could take but also because he is certain that he does not deserve it.
In other words, much of the reading experience of this book involves
watching C torture and insult himself. It's all the more effective because
it isn't gratuitous. C's internal monologue sounds exactly like how an
anxiety spiral feels, complete with the sort of half-effective coping
mechanisms, deflections, and emotional suppression one develops to blunt
that type of emotional turmoil.
I normally hate this kind of book. I am a happy ending and competence porn
reader by default. The world is full of enough pain that I don't turn to
fiction to read about more pain. It says a lot about how well-constructed
this book is that I stuck with it. Elkin is going somewhere with the
story, C gets moments of joy and delight along the way to keep the reader
from bogging down completely, and the best parts of the book feel like a
prolonged musical crescendo with suspended chords. There is a
climax coming, but Elkin is going to make you wait for it for far longer
than you want to.
The main element that protects A Little Vice from being too grim is
that it is a genre novel that is very playful about both magical girls and
superhero tropes in general. I've already alluded to one of those
elements: Elkin plays with the Mask Principle (the inability of people to
see through entirely obvious secret identities) in knowing and
entertaining ways. But there are also villains, and that leads me to the
absolutely delightful Avaritia Wolf, who for me was the best character in
this book.
The Angelic Saints are not the only possible approach to magical girl
powers in this universe. There are villains who can perform a similar
transformation, except they embrace a vice rather than a virtue. Avaritia
Wolf embraces the vice of greed. They (Avaritia's pronouns change over the
course of the book) also have a secret identity, which I suspect will be
blindingly obvious to most readers but which I'll avoid mentioning since
it's still arguably a spoiler.
The primary plot arc of this book is an attempt to recruit C to the side
of the villains. The Beasts are drawn to him because he has magical
potential, and the villains are less picky about gender. This initially
involves some creepy and disturbing mind control, but it also brings C
into contact with Avaritia and Avaritia's very specific understanding of
greed. As far as Avaritia is concerned, greed means wanting whatever they
want, for whatever reason they feel like wanting it, and there is
absolutely no reason why that shouldn't include being greedy for their
friends to be happy. Or doing whatever they can to make their friends
happy, whether or not that looks like villainy.
Elkin does two things with this plot that I thought were remarkably
skillful. The first is that she directly examines and then undermines the
"easy" transgender magical girl ending. In a world of transformation
magic, someone who wants to be a girl could simply turn into a girl and
thus apparently resolve the conflict in a way that makes everyone happy. I
think there is an important place for that story (I am a vigorous defender
of escapist fantasy and happy endings), but that is not the story that
Elkin is telling. I won't go into the details of why and how the story
complicates and undermines this easy ending, but it's a lot of why this
book feels both painful and honest to a specific, and very not easy,
transgender experience, even though it takes place in an utterly
unrealistic world.
But the second, which is more happy and joyful, is that Avaritia gleefully
uses a wholehearted embrace of every implication of the vice of greed to
bulldoze the binary morality of the story and question the classification
of human emotions into virtues and vices. They are not a hero, or even all
that good; they have some serious flaws and a very anarchic attitude
towards society. But Avaritia provides the compelling, infectious thrill
of the character who looks at the social construction of morality that is
constraining the story and decides that it's all bullshit and refuses to
comply. This is almost the exact opposite of C's default emotional
position at the start of the book, and watching the two characters play
off of each other in a complex friendship is an absolute delight.
The ending of this book is complicated, messy, and incomplete. It is the
sort of ending that I think could be incredibly powerful if it hits
precisely the right chords with the reader, but if you're not that reader,
it can also be a little heartbreaking because Elkin refuses to provide an
easy resolution. The ending also drops some threads that I wish Elkin
hadn't dropped; there are some characters who I thought deserved a
resolution that they don't get. But this is one of those books where the
author knows exactly what story they're trying to tell and tells it
whether or not that fits what the reader wants. Those books are often not
easy reading, but I think there's something special about them.
This is not the novel for people who want detailed world-building that
puts a solid explanation under events. I thought Elkin did a great job
playing with the conventions of an episodic anime, including starting the
book on Episode 12 to imply C's backstory with monster attacks and hinting
at a parallel light anime story by providing TV-trailer-style plot
summaries and teasers at the start and end of each chapter. There is a
fascinating interplay between the story in which the Angelic Saints are
the protagonists, which the reader can partly extrapolate, and the novel
about C that one is actually reading. But the details of the
world-building are kept at the anime plot level: There's an arch-villain,
a World Tree, and a bit of backstory, but none of it makes that much sense
or turns into a coherent set of rules. This is a psychological novel; the
background and rules exist to support C's story.
If you do want that psychological novel... well, I'm not sure whether to
recommend this book or not. I admire the construction of this book a great
deal, but I don't think appealing to the broadest possible audience was
the goal. C's anxiety spiral is very repetitive, because anxiety spirals
are very repetitive, and you have to be willing to read for the grace
notes on the doom loop if you're going to enjoy this book. The
sentence-by-sentence writing quality is fine but nothing remarkable, and
is a bit shy of the average traditionally-published novel. The main appeal
of A Little Vice is in the deep and unflinching portrayal of a
specific emotional journey. I think this book is going to work if you're
sufficiently invested in that journey that you are willing to read the
brutal and repetitive parts. If you're not, there's a chance you will
bounce off this hard.
I was invested, and I'm glad I read this, but caveat emptor. You
may want to try a sample first.
One final note: If you're deep in the book world, you may wonder, like I
did, if the title is a reference to Hanya Yanagihara's (in)famous A
Little Life. I do not know for certain I have not read that book
because I am not interested in being emotionally brutalized but if it
is, I don't think there is much similarity. Both books are to some extent
about four friends, but I couldn't find any other obvious connections from
some Wikipedia reading, and A Little Vice, despite C's emotional
turmoil, seems to be considerably more upbeat.
Content notes: Emotionally abusive parent, some thoughts of self-harm,
mind control, body dysmorphia, and a lot (a lot) of shame and
self-loathing.
Rating: 7 out of 10
For my PhD, my colleagues/collaborators and I built a distributed
stream-processing system using Haskell.
There are several other Haskell stream-processing systems. How do they
compare?
First, let's briefly discuss and define streaming in this context.
Structure and Interpretation of Computer Programs introduces Streams
as an analogue of lists, to support delayed evaluation. In brief, the
inductive list type (a list is either an empty list or a head element
pre-pended to another list) is replaced with a structure with a head
element and a promise which, when evaluated, will generate the tail
(which in turn may have a head element and a promise to generate another
tail, culminating in the equivalent of an empty list.) Later on SICP
also covers lazy evaluation.
However, the streaming we're talking about originates in the relational
community, rather than the functional one, and is subtly different. It's
about building a pipeline of processing that receives and emits data but
doesn't need to (indeed, cannot) reference the whole stream (which may
be infinite) at once.
Haskell streaming systems
Now let's go over some Haskell streaming systems.
conduit (2011-)
Conduit is the oldest of the ones I am reviewing here, but I doubt it's
the first in the Haskell ecosystem. If I've made any obvious omissions,
please let me know!
Conduit provides a new set of types to model streaming data, and a completely
new set of functions which are analogues of standard Prelude functions, e.g.
sumC in place of sum. It provides its own combinator(s) such as . (
aka fuse)
which is like composition but reads left-to-right.
The motivation for this is to enable (near?) constant memory usage for
processing large streams of data -- presumably versus using a list-based
approach and to provide some determinism: the README gives the example of
"promptly closing file handles". I think this is another way of saying
that it uses strict evaluation, or at least avoids lazy evaluation for
some things.
Conduit offers interleaved effects: which is to say, IO can be performed
mid-stream.
Conduit supports distributed operation via Data.Conduit.Network in the
conduit-extra package. Michael Snoyman, principal Conduit author, wrote
up how to use it here: https://www.yesodweb.com/blog/2014/03/network-conduit-async
To write a distributed Conduit application, the application programmer must
manually determine the boundaries between the clients/servers and write specific
code to connect them.
pipes (2012-)
The Pipes
Tutorial
contrasts itself with "Conventional Haskell stream programming": whether that
means Conduit or something else, I don't know.
Paraphrasing their pitch: Effects, Streaming Composability: pick two. That's
the situation they describe for stream programming prior to Pipes. They argue
Pipes offers all three.
Pipes offers it's own combinators (which read left-to-right)
and offers interleaved effects.
At this point I can't really see what fundamentally distinguishes Pipes from
Conduit.
Pipes has some support for distributed operation via the sister library
pipes-network. It
looks like you must send and receive ByteStrings, which means rolling
your own serialisation for other types. As with Conduit, to send or receive
over a network, the application programmer must divide their program up
into the sub-programs for each node, and add the necessary ingress/egress
code.
io-streams (2013-)
io-streams emphasises simple primitives. Reading and writing is done
under the IO Monad, thus, in an effectful (but non-pure) context. The
presence or absence of further stream data are signalled by using the
Maybe type (Just more data or Nothing: the producer has finished.)
It provides a library of functions that shadow the standard Prelude, such
as S.fromList, S.mapM, etc.
It's not clear to me what the motivation for io-streams is, beyond
providing a simple interface. There's no declaration of intent that I can find
about (e.g.) constant-memory operation.
There's no mention of or support (that I can find) for distributed
operation.
streaming (2015-)
Similar to io-streams, Streaming emphasises providing a simple
interface that gels well with traditional Haskell methods. Streaming
provides effectful streams (via a Monad -- any Monad?) and a collection
of functions for manipulating streams which are designed to closely
mimic standard Prelude (and Data.List) functions.
Streaming doesn't push its own combinators: the examples provided
use $ and read right-to-left.
The motivation for Streaming seems to be to avoid memory leaks caused by
extracting pure lists from IO with traditional functions like mapM,
which require all the list constructors to be evaluated, the list to be
completely deconstructed, and then a new list constructed.
Like io-streams, the focus of the library is providing a low-level
streaming abstraction, and there is no support for distributed operation.
streamly (2017-)
Streamly appears to have the grand goal of providing a unified programming
tool as suited for quick-and-dirty programming tasks (normally the domain of
scripting languages) and high-performance work (C, Java, Rust, etc.). Their
intended audience appears to be everyone, or at least, not just existing
Haskell programmers. See their rationale
Streamly offers an interface to permit composing concurrent (note: not
distributed) programs via combinators. It relies upon fusing a streaming
pipeline to remove intermediate list structure allocations and de-allocations
(i.e. de-forestation, similar to GHC rewrite rules)
The examples I've seen use standard combinators (e.g. Control.Function.&,
which reads left-to-right, and Applicative).
Streamly provide benchmarks
versus Haskell pure lists, Streaming, Pipes and Conduit: these generally
show Streamly several orders of magnitude faster.
I'm finding it hard to evaluate Streamly. It's big, and it's focus is wide.
It provides shadows of Prelude functions, as many of these libraries do.
wrap-up
It seems almost like it must be a rite-of-passage to write a streaming system
in Haskell. Stones and glass houses, I'm guilty of that
too.
The focus of the surveyed libraries is mostly on providing a streaming
abstraction, normally with an analogous interface to standard Haskell lists.
They differ on various philosophical points (whether to abstract away the
mechanics behind type synonyms, how much to leverage existing Haskell idioms,
etc). A few of the libraries have some rudimentary support for distributed
operation, but this is limited to connecting separate nodes together: in some
cases serialising data remains the application programmer's job, and in all
cases the application programmer must manually carve up their processing
according to a fixed idea of what nodes they are deploying to. They all
define a fixed-function pipeline.
I can t remember exactly the joke I was making at the time in my
work s slack instance (I m sure it wasn t particularly
funny, though; and not even worth re-reading the thread to work out), but it
wound up with me writing a UEFI binary for the punchline. Not to spoil the
ending but it worked - no pesky kernel, no messing around with userland . I
guess the only part of this you really need to know for the setup here is that
it was a Severance joke,
which is some fantastic TV. If you haven t seen it, this post will seem perhaps
weirder than it actually is. I promise I haven t joined any new cults. For
those who have seen it, the payoff to my joke is that I wanted my machine to
boot directly to an image of
Kier Eagan.
As for how to do it I figured I d give the uefi
crate a shot, and see how it is to use,
since this is a low stakes way of trying it out. In general, this isn t the
sort of thing I d usually post about except this wound up being easier and
way cleaner than I thought it would be. That alone is worth sharing, in the
hopes someome comes across this in the future and feels like they, too, can
write something fun targeting the UEFI.
First thing s first gotta create a rust project (I ll leave that part to you
depending on your life choices), and to add the uefi crate to your
Cargo.toml. You can either use cargo add or add a line like this by hand:
uefi = version = "0.33", features = ["panic_handler", "alloc", "global_allocator"]
We also need to teach cargo about how to go about building for the UEFI target,
so we need to create a rust-toolchain.toml with one (or both) of the UEFI
targets we re interested in:
Unfortunately, I wasn t able to use the
image crate,
since it won t build against the uefi target. This looks like it s
because rustc had no way to compile the required floating point operations
within the image crate without hardware floating point instructions
specifically. Rust tends to punt a lot of that to libm usually, so this isnt
entirely shocking given we re nostd for a non-hardfloat target.
So-called softening requires a software floating point implementation that
the compiler can use to polyfill (feels weird to use the term polyfill here,
but I guess it s spiritually right?) the lack of hardware floating point
operations, which rust hasn t implemented for this target yet. As a result, I
changed tactics, and figured I d use ImageMagick to pre-compute the pixels
from a jpg, rather than doing it at runtime. A bit of a bummer, since I need
to do more out of band pre-processing and hardcoding, and updating the image
kinda sucks as a result but it s entirely manageable.
This will take our input file (kier.jpg), resize it to get as close to the
desired resolution as possible while maintaining aspect ration, then convert it
from a jpg to a flat array of 4 byte RGBA pixels. Critically, it s also
important to remember that the size of the kier.full.jpg file may not actually
be the requested size it will not change the aspect ratio, so be sure to
make a careful note of the resulting size of the kier.full.jpg file.
Last step with the image is to compile it into our Rust bianary, since we
don t want to struggle with trying to read this off disk, which is thankfully
real easy to do.
Remember to use the width and height from the final kier.full.jpg file as the
values for KIER_WIDTH and KIER_HEIGHT. KIER_PIXEL_SIZE is 4, since we
have 4 byte wide values for each pixel as a result of our conversion step into
RGBA. We ll only use RGB, and if we ever drop the alpha channel, we can drop
that down to 3. I don t entirely know why I kept alpha around, but I figured it
was fine. My kier.full.jpg image winds up shorter than the requested height
(which is also qemu s default resolution for me) which means we ll get a
semi-annoying black band under the image when we go to run it but it ll
work.
Anyway, now that we have our image as bytes, we can get down to work, and
write the rest of the code to handle moving bytes around from in-memory
as a flat block if pixels, and request that they be displayed using the
UEFI GOP. We ll just need to hack up a container
for the image pixels and teach it how to blit to the display.
/// RGB Image to move around. This isn't the same as an
/// image::RgbImage , but we can associate the size of
/// the image along with the flat buffer of pixels.
structRgbImage/// Size of the image as a tuple, as the
/// (width, height)
size: (usize, usize),
/// raw pixels we'll send to the display.
inner: Vec<BltPixel>,
impl RgbImage
/// Create a new RgbImage .
fnnew(width: usize, height: usize) -> Self
RgbImage
size: (width, height),
inner: vec![BltPixel::new(0, 0, 0); width * height],
/// Take our pixels and request that the UEFI GOP
/// display them for us.
fnwrite(&self, gop: &mut GraphicsOutput) -> Result
gop.blt(BltOp::BufferToVideo
buffer: &self.inner,
src: BltRegion::Full,
dest: (0, 0),
dims: self.size,
)
impl Index<(usize, usize)>for RgbImage
typeOutput= BltPixel;
fnindex(&self, idx: (usize, usize)) -> &BltPixellet (x, y) = idx;
&self.inner[y * self.size.0+ x]
impl IndexMut<(usize, usize)>for RgbImage
fnindex_mut(&mut self, idx: (usize, usize)) -> &mut BltPixel
let (x, y) = idx;
&mut self.inner[y * self.size.0+ x]
We also need to do some basic setup to get a handle to the UEFI
GOP via the UEFI crate (using
uefi::boot::get_handle_for_protocol
and
uefi::boot::open_protocol_exclusive
for the GraphicsOutput
protocol), so that we have the object we need to pass to RgbImage in order
for it to write the pixels to the display. The only trick here is that the
display on the booted system can really be any resolution so we need to do
some capping to ensure that we don t write more pixels than the display can
handle. Writing fewer than the display s maximum seems fine, though.
fnpraise() -> Result
let gop_handle = boot::get_handle_for_protocol::<GraphicsOutput>()?;
letmut gop = boot::open_protocol_exclusive::<GraphicsOutput>(gop_handle)?;
// Get the (width, height) that is the minimum of
// our image and the display we're using.
let (width, height) = gop.current_mode_info().resolution();
let (width, height) = (width.min(KIER_WIDTH), height.min(KIER_HEIGHT));
letmut buffer = RgbImage::new(width, height);
for y in0..height
for x in0..width
let idx_r = ((y * KIER_WIDTH) + x) * KIER_PIXEL_SIZE;
let pixel =&mut buffer[(x, y)];
pixel.red = KIER[idx_r];
pixel.green = KIER[idx_r +1];
pixel.blue = KIER[idx_r +2];
buffer.write(&mut gop)?;
Ok(())
Not so bad! A bit tedious we could solve some of this by turning
KIER into an RgbImage at compile-time using some clever Cow and
const tricks and implement blitting a sub-image of the image but this
will do for now. This is a joke, after all, let s not go nuts. All that s
left with our code is for us to write our main function and try and boot
the thing!
#[entry]fnmain() -> Status
uefi::helpers::init().unwrap();
praise().unwrap();
boot::stall(100_000_000);
Status::SUCCESS
If you re following along at home and so interested, the final source is over at
gist.github.com.
We can go ahead and build it using cargo (as is our tradition) by targeting
the UEFI platform.
Testing the UEFI Blob
While I can definitely get my machine to boot these blobs to test, I figured
I d save myself some time by using QEMU to test without a full boot.
If you ve not done this sort of thing before, we ll need two packages,
qemu and ovmf. It s a bit different than most invocations of qemu you
may see out there so I figured it d be worth writing this down, too.
$ doas apt install qemu-system-x86 ovmf
qemu has a nice feature where it ll create us an EFI partition as a drive and
attach it to the VM off a local directory so let s construct an EFI
partition file structure, and drop our binary into the conventional location.
If you haven t done this before, and are only interested in running this in a
VM, don t worry too much about it, a lot of it is convention and this layout
should work for you.
With all this in place, we can kick off qemu, booting it in UEFI mode using
the ovmf firmware, attaching our EFI partition directory as a drive to
our VM to boot off of.
If all goes well, soon you ll be met with the all knowing gaze of
Chosen One, Kier Eagan. The thing that really impressed me about all
this is this program worked first try it all went so boringly
normal. Truly, kudos to the uefi crate maintainers, it s incredibly
well done.
Booting a live system
Sure, we could stop here, but anyone can open up an app window and see a
picture of Kier Eagan, so I knew I needed to finish the job and boot a real
machine up with this. In order to do that, we need to format a USB stick.
BE SURE /dev/sda IS CORRECT IF YOU RE COPY AND PASTING. All my drives
are NVMe, so BE CAREFUL if you use SATA, it may very well be your
hard drive! Please do not destroy your computer over this.
$ doas fdisk /dev/sda
Welcome to fdisk (util-linux 2.40.4).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-4014079, default 2048):
Last sector, +/-sectors or +/-size K,M,G,T,P (2048-4014079, default 4014079):
Created a new partition 1 of type 'Linux' and of size 1.9 GiB.
Command (m for help): t
Selected partition 1
Hex code or alias (type L to list all): ef
Changed type of partition 'Linux' to 'EFI (FAT-12/16/32)'.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
Once that looks good (depending on your flavor of udev you may or
may not need to unplug and replug your USB stick), we can go ahead
and format our new EFI partition (BE CAREFUL THAT /dev/sda IS YOUR
USB STICK) and write our EFI directory to it.
$ doas mkfs.fat /dev/sda1
$ doas mount /dev/sda1 /mnt
$ cp -r esp/efi /mnt
$ find /mnt
/mnt
/mnt/efi
/mnt/efi/boot
/mnt/efi/boot/bootx64.efi
Of course, naturally, devotion to Kier shouldn t mean backdooring your system.
Disabling Secure Boot runs counter to the Core Principals, such as Probity, and
not doing this would surely run counter to Verve, Wit and Vision. This bit does
require that you ve taken the step to enroll a
MOK and know how
to use it, right about now is when we can use sbsign to sign our UEFI binary
we want to boot from to continue enforcing Secure Boot. The details for how
this command should be run specifically is likely something you ll need to work
out depending on how you ve decided to manage your MOK.
I figured I d leave a signed copy of boot2kier at
/boot/efi/EFI/BOOT/KIER.efi on my Dell XPS 13, with Secure Boot enabled
and enforcing, just took a matter of going into my BIOS to add the right
boot option, which was no sweat. I m sure there is a way to do it using
efibootmgr, but I wasn t smart enough to do that quickly. I let er rip,
and it booted up and worked great!
It was a bit hard to get a video of my laptop, though but lucky for me, I
have a Minisforum Z83-F sitting around (which, until a few weeks ago was running
the annual http server to control my christmas tree
) so I grabbed it out of the christmas bin, wired it up to a video capture
card I have sitting around, and figured I d grab a video of me booting a
physical device off the boot2kier USB stick.
Attentive readers will notice the image of Kier is smaller then the qemu booted
system which just means our real machine has a larger GOP display
resolution than qemu, which makes sense! We could write some fancy resize code
(sounds annoying), center the image (can t be assed but should be the easy way
out here) or resize the original image (pretty hardware specific workaround).
Additionally, you can make out the image being written to the display before us
(the Minisforum logo) behind Kier, which is really cool stuff. If we were real
fancy we could write blank pixels to the display before blitting Kier, but,
again, I don t think I care to do that much work.
But now I must away
If I wanted to keep this joke going, I d likely try and find a copy of the
original
video when Helly 100%s her file
and boot into that or maybe play a terrible midi PC speaker rendition of
Kier, Chosen One, Kier after
rendering the image. I, unfortunately, don t have any friends involved with
production (yet?), so I reckon all that s out for now. I ll likely stop playing
with this the joke was done and I m only writing this post because of how
great everything was along the way.
All in all, this reminds me so much of building a homebrew kernel to boot a
system into but like, good, though, and it s a nice reminder of both how
fun this stuff can be, and how far we ve come. UEFI protocols are light-years
better than how we did it in the dark ages, and the tooling for this is SO
much more mature. Booting a custom UEFI binary is miles ahead of trying to
boot your own kernel, and I can t believe how good the uefi crate is
specifically.
Praise Kier! Kudos, to everyone involved in making this so delightful .
Another short status update of what happened on my side last
month. Some more ModemManager bits landed, 
As promised on my previous 
A few months ago 
From April 27th to 30th, 2024,
This was the fifth time that a MiniDebConf (as an exclusive in-person event
about Debian) took place in Brazil. Previous editions were in Curitiba
(
2024 edition numbers
During the four days of the event, several activities took place for all
levels of users and collaborators of the Debian project. The official schedule
was composed of:
The final numbers for MiniDebConf Belo Horizonte 2024 show that we had a
record number of participants.
Activities
The MiniDebConf schedule was intense and diverse. On the 27th, 29th and 30th
(Saturday, Monday and Tuesday) we had talks, discussions, workshops and many
practical activities.
On the 28th (Sunday), the Day Trip took place, a day dedicated to sightseeing
around the city. In the morning we left the hotel and went, on a chartered bus,
to the
After a 2-hour tour of the Market, we got back on the bus and hit the road for
lunch at a typical Minas Gerais food restaurant.
With everyone well fed, we returned to Belo Horizonte to visit the city's
main tourist attraction: Lagoa da Pampulha and Capela S o Francisco de Assis,
better known as
We went back to the hotel and the day ended in the hacker space that we set up
in the events room for people to chat, packaging, and eat pizzas.
Crowdfunding
For the third time we ran a crowdfunding campaign and it was incredible how
people contributed! The initial goal was to raise the amount equivalent to a
gold tier of R$ 3,000.00. When we reached this goal, we defined a new one,
equivalent to one gold tier + one silver tier (R$ 5,000.00). And again we
achieved this goal. So we proposed as a final goal the value of a gold + silver
+ bronze tiers, which would be equivalent to R$ 6,000.00. The result was that
we raised R$7,239.65 (~ USD 1,400) with the help of more than 100 people!
Thank you very much to the people who contributed any amount. As a thank you, we 
Food, accommodation and/or travel grants for participants
Each edition of MiniDebConf brought some innovation, or some different benefit
for the attendees. In this year's edition in Belo Horizonte, as with DebConfs, we 
Photos and videos
You can watch recordings of the talks at the links below:
Sponsors
Gold:
Some of you may remember that I recently felt a bit underwhelmed
by the last 
Same as the last goarounds, except the premable here is a 
For my 
