I'm on a bit of an Orbital kick at the moment. Last year they re-issued their
1991
debut album with 43 extra tracks. Later this month they're doing the same for
their 1993 sophomore album.
I thought I'd try to narrow down some tracks to recommend. I seem to have
settled on roughly 5 in previous posts (for
Underworld, The Cure, Coil and
Gazelle Twin). This time I've done 6 (I borrowed one from Underworld)
As always it's a hard choice. I've tried to select some tracks I really
enjoy that don't often come up on best-of compilation albums.
For a more conventional choice of best-of tracks, I recommend the
recent-ish 30 something
"compilation" (of sorts, previously written about)
The Naked and the Dead (1992)
The Naked and the Dead by Orbital
From an early EP Radiccio, which is being
re-issued this month. Digital versions of the re-issue will feature
a new recording "Deepest" featuring Tilda Swinton. Sadly this isn't
making it onto the pressed version. She performed with them live at
Glastonbury 2024. That entire performance was a real pick-me-up
during my convolescence, and is recommended.
Anyway I've now written more about a song I haven't recommended than
the one I did
Remind (1993)
Remind by Orbital
From the Brown Album, I first heard this as the Encore
from their "final show", for John Peel, when they split up
in 2004. "Remind" wasn't broadcast, but an audience recording was
circulated on fan site Loopz. Remarkably, 21 years on, it's still
there.
In writing this I discovered that it's a re-working of a remix
Orbital did for Meat Beat Manifesto: MindStream (Mind The Bend The Mind)
You Lot (2004)
From the unfairly-maligned "final" Blue album.
Featuring a sample of pre-Doctor Who Christoper Eccleston, from another
Russell T Davies production, Second Coming.
Beached (2000)
Beached (Long version) by Orbital, Angelo Badalamenti
Co-written by Angelo Badalamenti, it's built around a sample of
Badalamenti's score for the movie "The Beach". Orbital's re-work
adds some grit to the orchestral instrumentation and opens with
a monologue, delivered by Leonardo Di Caprio, sampled from the
movie.
Spare Parts Express (1999)
Spare Parts Express by Orbital
Critics had started to be quite unfair to Orbital by this point.
The band themselves said that they'd ran out of ideas (pointing
at album closer "Style", built around a Stylophone melody, as proof).
Their malaise continued right up to the Blue Album, at which point
the split up; ostensibly for good, before regrouping 8 years later.
Spare Parts Express is a hatchet job of various bits that they didn't
develop into full songs on their own. Despite this I think it works.
I love long-form electronica, and this clocks in at 10:07. My favourite
segment (06:37) is adjacent to a reference (05:05) to John Baker's theme for
the BBC children's program Newsround (sadly they aren't using it
today. Here's a rundown of Newsround themes over time)
Attached (1994)
Attached by Orbital
This originally debuted on a Peel session before appearing on
the subsequent album Snivilisation a few months later.
An album closer, and a good come-down song to close this list.
On 26th and 27th April we held a Debian bug-squashing party near
Leuven, Belgium. Several longstanding and new Debian contributors
gathered to work through some of the highest priority bugs affecting
the upcoming release of Debian 13 trixie .
We were hosted by the Familia community
centre in Tildonk. As this venue currently does not have an Internet
connection, we brought a mobile hotspot and a local Debian mirror.
In attendance were:
Debian Developers: Ben Hutchings, Nattie Mayer-Hutchings, Kurt
Roeckx, and Geert Stappers
New contributors: Y ce K r m, Louis Renuart, Arnout Vandecappelle
The new contributors were variously using Arch, Fedora, and Ubuntu,
and the DDs spent some some time setting them up with Debian
dvelopment environments.
The bugs we worked on included:
#1100981: libmlir-19 fails to coinstall:
Arnout and Kurt looked at this, and pointed out an existing MR that
should fix it and some other packages that are similarly affected.
Last month, I attended the KDE India conference in Gandhinagar, Gujarat from the 4th to the 6th of April. I made my mind to attend when Sahil told me about his plans to attend and giving a talk.
A day after my talk submission, the organizer Bhushan contacted me on Matrix and informed me that my talk had been accepted. I was also informed that KDE will cover my travel and accommodation expenses. So, I planned to attend the conference at this point. I am a longtime KDE user, so why not ;)
I arrived in Ahmedabad, the twin city of Gandhinagar, a day before the conference. The first thing that struck me as soon as I came out of the Ahmedabad airport was the heat. I felt as if I was being cooked exactly how Bhushan put it earlier in the group chat. I took a taxi to get to my hotel, which was close to the conference venue.
Later that afternoon, I met Bhushan and Joseph. Joseph lived in Germany. Bhushan was taking him to get a SIM card, so I tagged along and got to roam around. Joseph was unsure about where to go after the conference, so I asked him what he wanted out of his trip and had conversations along that line.
Later, Vishal convinced him to go to Lucknow. Since he was adamant about taking the train, I booked a Tatkal train ticket for him to Lucknow. He was curious about how Tatkal booking works and watched me in amusement while I was booking the ticket.
The 4th of April marked the first day of the conference, with around 25 attendees. Bhushan started the day with an overview of KDE conferences in India, followed by Vishal, who discussed FOSS United s activities. After the lunch, Joseph gave an overview of his campaign to help people switch from Windows to GNU/Linux due to environmental and security reasons. He continued his session in detail the next day.
Conference hall
A key takeaway for me from Joseph s session was the idea pointed out by Adwaith: marketing GNU/Linux as a cheap alternative may not attract as much attention as marketing it as a status symbol. He gave the example of how the Tata Nano didn t do well in the Indian market due to being perceived as a poor person s car.
My talk was scheduled for the evening of the first day. I hadn t prepared any slides because I wanted to make my session interactive. During my talk, I did an activity with the attendees to demonstrate the federated nature of XMPP messaging, of which Prav is a part. After the talk, I got a lot of questions, signalling engagement. The audience was cooperative (just like Prav ;)), contrary to my expectations (I thought they will be tired and sleepy).
On the third day, I did a demo on editing OpenStreetMap (referred to as OSM in short) using the iD editor. It involved adding points to OSM based on the students suggestions. Since my computer didn t have an HDMI port, I used Subin s computer, and he logged into his OSM account for my session. Therefore, any mistakes I made will be under Subin s name. :)
On the third day, I attended Aaruni s talk about backing up a GNU/Linux system. This was the talk that resonated with me the most. He suggested formatting the system with the btrfs file system during the installation, which helps in taking snapshots of the system and provides an easy way to roll back to a previous version if, for example, a file is accidentally deleted. I have tried many backup techniques, including this one, but I never tried backing up on the internal disk. I ll certainly give this a try.
A conference is not only about the talks, that s why we had a Prav table as well ;) Just kidding. What I really mean is that a conference is more about interactions than talks. Since the conference was a three-day affair, attendees got plenty of time to bond and share ideas.
Prav stall at the conference
Conference group photo
After the conference, Bhushan took us to Adalaj Stepwell, an attraction near Gandhinagar. Upon entering the complex, we saw a park where there were many langurs. Going further, there were stairs that led down to a well. I guess this is why it is called a stepwell.
Adalaj Stepwell
Later that day, we had Gujarati Thali for dinner. It was an all-you-can-eat buffet and was reasonably priced at 300 rupees per plate. Aamras (Mango juice) was the highlight for me. This was the only time we had Gujarati food during this visit. After the dinner, Aaruni dropped Sahil and I off at the airport. The hospitality was superb - for instance, in addition to Aaruni dropping us, Bhushan also picked up some of the attendees from the airport.
Finally, I would like to thank KDE for sponsoring my travel and accommodation costs.
Let s wrap up this post here and meet you in the next one.
Thanks to contrapunctus and Joseph for proofreading.
Welcome to our fourth report from the Reproducible Builds project in 2025. These monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. Lastly, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website.
Table of contents:
reproduce.debian.net
The last few months have seen the introduction, development and deployment of reproduce.debian.net. In technical terms, this is an instance of rebuilderd, our server designed monitor the official package repositories of Linux distributions and attempt to reproduce the observed results there.
This month, however, we are pleased to announce that reproduce.debian.net now tests all the Debian trixie architectures except s390x and mips64el.
The ppc64el architecture was added through the generous support of Oregon State University Open Source Laboratory (OSUOSL), and we can support the armel architecture thanks to CodeThink.
Fifty Years of Open Source Software Supply Chain Security
Russ Cox has published a must-read article in ACM Queue on Fifty Years of Open Source Software Supply Chain Security. Subtitled, For decades, software reuse was only a lofty goal. Now it s very real. , Russ article goes on to outline the history and original goals of software supply-chain security in the US military in the early 1970s, all the way to the XZ Utils backdoor of 2024. Through that lens, Russ explores the problem and how it has changed, and hasn t changed, over time.
He concludes as follows:
We are all struggling with a massive shift that has happened in the past 10 or 20 years in the software industry. For decades, software reuse was only a lofty goal. Now it s very real. Modern programming environments such as Go, Node and Rust have made it trivial to reuse work by others, but our instincts about responsible behaviors have not yet adapted to this new reality.
We all have more work to do.
Luca DiMaio of Chainguard posted to the list reporting that they had successfully implemented reproducible filesystem images with both ext4and an EFI system partition. They go on to list the various methods, and the thread generated at least fifteen replies.
David Wheeler announced that the OpenSSF is building a glossary of sorts in order that they consistently use the same meaning for the same term and, moreover, that they have drafted a definition for reproducible build . The thread generated a significant number of replies on the definition, leading to a potential update to the Reproducible Build s own definition.
My initial interest in reproducible builds was how do I distribute pre-compiled binaries on GitHub without people raising security concerns about them . I ve cycled back to this original problem about 5 years later and built a tool that is meant to address this. []
[ ] Achieving reproducibility at scale remains difficult, especially in Java, due to a range of non-deterministic factors and caveats in the build process. In this work, we focus on reproducibility in Java-based software, archetypal of enterprise applications. We introduce a conceptual framework for reproducible builds, we analyze a large dataset from Reproducible Central and we develop a novel taxonomy of six root causes of unreproducibility. We study actionable mitigations: artifact and bytecode canonicalization using OSS-Rebuild and jNorm respectively. Finally, we present Chains-Rebuild, a tool that raises reproducibility success from 9.48% to 26.89% on 12,283 unreproducible artifacts. To sum up, our contributions are the first large-scale taxonomy of build unreproducibility causes in Java, a publicly available dataset of unreproducible builds, and Chains-Rebuild, a canonicalization tool for mitigating unreproducible builds in Java.
OSS Rebuild adds new TUI features
OSS Rebuild aims to automate rebuilding upstream language packages (e.g. from PyPI, crates.io and npm registries) and publish signed attestations and build definitions for public use.
OSS Rebuild ships a text-based user interface (TUI) for viewing, launching, and debugging rebuilds. While previously requiring ownership of a full instance of OSS Rebuild s hosted infrastructure, the TUI now supports a fully local mode of build execution and artifact storage. Thanks to Giacomo Benedetti for his usage feedback and work to extend the local-only development toolkit.
Another feature added to the TUI was an experimental chatbot integration that provides interactive feedback on rebuild failure root causes and suggests fixes.
Debian developer Simon Josefsson published another two reproducibility-related blog posts this month, the first on the topic of Verified Reproducible Tarballs. Simon sardonically challenges the reader as follows: Do you want a supply-chain challenge for the Easter weekend? Pick some well-known software and try to re-create the official release tarballs from the corresponding Git checkout. Is anyone able to reproduce anything these days? After that, they also published a blog post on Building Debian in a GitLab Pipeline using their multi-stage rebuild approach.
Roland also posted to our mailing list to highlight that there is now another tool in Debian that generates reproducible output, equivs . This is a tool to create trivial Debian packages that might Depend on other packages. As Roland writes, building the [equivs] package has been reproducible for a while, [but] now the output of the [tool] has become reproducible as well .
The IzzyOnDroid Android APK repository made more progress in April. Thanks to funding by NLnet and Mobifree, the project was also to put more time into their tooling. For instance, developers can now easily run their own verification builder in less than 5 minutes . This currently supports Debian-based systems, but support for RPM-based systems is incoming.
The rbuilder_setup tool can now setup the entire framework within less than five minutes. The process is configurable, too, so everything from just the basics to verify builds up to a fully-fledged RB environment is also possible.
This tool works on Debian, RedHat and Arch Linux, as well as their derivates. The project has received successful reports from Debian, Ubuntu, Fedora and some Arch Linux derivates so far.
Documentation on how to work with reproducible builds (making apps reproducible, debugging unreproducible packages, etc) is available in the project s wiki page.
Future work is also in the pipeline, including documentation, guidelines and helpers for debugging.
NixOS defined an Outreachy project for improving build reproducibility. In the application phase, NixOS saw some strong candidates providing contributions, both on the NixOS side and upstream: guider-le-ecit analyzed a libpinyin issue. Tessy James fixed an issue in arandr and helped analyze one in libvlc that led to a proposed upstream fix. Finally, 3pleX fixed an issue which was accepted in upstream kitty, one in upstream maturin, one in upstream python-sip and one in the Nix packaging of python-libbytesize. Sadly, the funding for this internship fell through, so NixOS were forced to abandon their search.
Lastly, in openSUSE news, Bernhard M. Wiedemann posted another monthly update for their work there.
diffoscope & strip-nondeterminismdiffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading a number of versions to Debian:
Use the --walk argument over the potentially dangerous alternative --scan when calling out to zipdetails(1). []
Correct a longstanding issue where many >-based version tests used in conditional fixtures were broken. This was used to ensure that specific tests were only run when the version on the system was newer than a particular number. Thanks to Colin Watson for the report (Debian bug #1102658) []
Address a long-hidden issue in the test_versions testsuite as well, where we weren t actually testing the greater-than comparisons mentioned above, as it was masked by the tests for equality. []
Update copyright years. []
In strip-nondeterminism, however, Holger Levsen updated the Continuous Integration (CI) configuration in order to use the standard Debian pipelines via debian/salsa-ci.yml instead of using .gitlab-ci.yml. []
Website updates
Once again, there were a number of improvements made to our website this month including:
Aman Sharma added OSS-Rebuild s stabilize tool to the Tools page. [][]
Chris Lamb added a configure.ac (GNU Autotools) example for using SOURCE_DATE_EPOCH. []. Chris also updated the SOURCE_DATE_EPOCH snippet and move the archive metadata to a more suitable location. []
Denis Carikli added GNU Boot to our ever-evolving Projects page.
Reproducibility testing framework
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In April, a number of changes were made by Holger Levsen, including:
Make various changes to the ppc64el nodes. [][]9[][]
Make various changes to the arm64 and armhf nodes. [][][][]
Various changes related to the rebuilderd-worker entry point. [][][]
Create and deploy a pkgsync script. [][][][][][][][]
Fix the monitoring of the riscv64 architecture. [][]
Make a number of changes related to starting the rebuilderd service. [][][][]
Backup-related:
Backup the rebuilder databases every week. [][][][]
Improve the node health checks. [][]
Misc:
Re-use existing connections to the SSH proxy node on the riscv64 nodes. [][]
Node maintenance. [][][]
In addition:
Jochen Sprickerhof fixed the risvc64 host names [] and requested access to all the rebuilderd nodes [].
Mattia Rizzolo updated the self-serve rebuild scheduling tool, replacing the deprecated SSO -style authentication with OpenIDC which authenticates against salsa.debian.org. [][][]
Roland Clobus updated the configuration for the osuosl3 node to designate 4 workers for bigger builds. []
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
After my previous posts related to Argo CD (one about
argocd-autopilot and another with some
usage examples) I started to look into
Kluctl (I also plan to review Flux, but I m more interested on the kluctl
approach right now).
While reading an entry on the project blog about Cluster API
somehow I ended up on the vCluster site and decided to give it a try, as it can be a valid
way of providing developers with on demand clusters for debugging or run CI/CD tests before deploying things on common
clusters or even to have multiple debugging virtual clusters on a local machine with only one of them running at any
given time.
On this post I will deploy a vcluster using the k3d_argocd kubernetes cluster (the one we created on the posts about
argocd) as the host and will show how to:
use its ingress (in our case traefik) to access the API of the virtual one (removes the need of having to use the
vcluster connect command to access it with kubectl),
publish the ingress objects deployed on the virtual cluster on the host ingress, and
use the sealed-secrets of the host cluster to manage the virtual cluster secrets.
Creating the virtual cluster
Installing the vcluster applicationTo create the virtual clusters we need the vcluster command, we can install it with arkade:
arkade get vcluster
The vcluster.yaml fileTo create the cluster we are going to use the following vcluster.yaml file (you can find the documentation about all
its options here):
controlPlane:proxy:# Extra hostnames to sign the vCluster proxy certificate forextraSANs:-my-vcluster-api.lo.mixinet.netexportKubeConfig:context:my-vcluster_k3d-argocdserver:https://my-vcluster-api.lo.mixinet.net:8443secret:name:my-vcluster-kubeconfigsync:toHost:ingresses:enabled:trueserviceAccounts:enabled:truefromHost:ingressClasses:enabled:truenodes:enabled:trueclearImageStatus:truesecrets:enabled:truemappings:byName:# sync all Secrets from the 'my-vcluster-default' namespace to the# virtual "default" namespace."my-vcluster-default/*":"default/*"# We could add other namespace mappings if needed, i.e.:# "my-vcluster-kube-system/*": "kube-system/*"
On the controlPlane section we ve added the proxy.extraSANs entry to add an extra host name to make sure it is added
to the cluster certificates if we use it from an ingress.
The exportKubeConfig section creates a kubeconfig secret on the virtual cluster namespace using the provided
host name; the secret can be used by GitOps tools or we can dump it to a file to connect from our machine.
On the sync section we enable the synchronization of Ingress objects and ServiceAccounts from the virtual to the
host cluster:
We copy the ingress definitions to use the ingress server that runs on the host to make them work from the outside
world.
The service account synchronization is not really needed, but we enable it because if we test this configuration with
EKS it would be useful if we use IAM roles for the service accounts.
On the opposite direction (from the host to the virtual cluster) we synchronize:
The IngressClass objects, to be able to use the host ingress server(s).
The Nodes (we are not using the info right now, but it could be interesting if we want to have the real information
of the nodes running pods of the virtual cluster).
The Secrets from the my-vcluster-default host namespace to the default of the virtual cluster; that
synchronization allows us to deploy SealedSecrets on the host that generate secrets that are copied
automatically to the virtual one. Initially we only copy secrets for one namespace but if the virtual cluster needs
others we can add namespaces on the host and their mappings to the virtual one on the vcluster.yaml file.
Creating the virtual clusterTo create the virtual cluster we run the following command:
It creates the virtual cluster on the my-vcluster namespace using the vcluster.yaml file shown before without
connecting to the cluster from our local machine (if we don t pass that option the command adds an entry on our
kubeconfig and launches a proxy to connect to the virtual cluster that we don t plan to use).
Adding an ingress TCP route to connect to the vcluster apiAs explained before, we need to create an IngressTcpRoute object to be able to connect to the vcluster API, we use the
following definition:
Once we apply those changes the cluster API will be available on the https://my-cluster-api.lo.mixinet.net:8443 URL
using its own self signed certificate (we have enabled TLS passthrough) that includes the hostname we use (we adjusted
it on the vcluster.yaml file, as explained before).
Getting the kubeconfig for the vclusterOnce the vcluster is running we will have its kubeconfig available on the my-vcluster-kubeconfig secret on its
namespace on the host cluster.
To dump it to the ~/.kube/my-vcluster-config we can do the following:
Once available we can define the vkubectl alias to adjust the KUBECONFIG variable to access it:
alias vkubectl="KUBECONFIG=~/.kube/my-vcluster-config kubectl"
Or we can merge the configuration with the one on the KUBECONFIG variable and use kubectx or a similar tool to change
the context (for our vcluster the context will be my-vcluster_k3d-argocd). If the KUBECONFIG variable is defined and
only has the PATH to a single file the merge can be done running the following:
On the rest of this post we will use the vkubectl alias when connecting to the virtual cluster, i.e. to check that it
works we can run the cluster-info subcommand:
vkubectl cluster-info
Kubernetes control plane is running at https://my-vcluster-api.lo.mixinet.net:8443
CoreDNS is running at https://my-vcluster-api.lo.mixinet.net:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Installing the dummyhttpd applicationTo test the virtual cluster we are going to install the dummyhttpd application using the following
kustomization.yaml file:
apiVersion:kustomize.config.k8s.io/v1beta1kind:Kustomizationresources:-https://forgejo.mixinet.net/blogops/argocd-applications.git//dummyhttp/?ref=dummyhttp-v1.0.0# Add the config mapconfigMapGenerator:-name:dummyhttp-configmapliterals:-CM_VAR="Vcluster Test Value"behavior:createoptions:disableNameSuffixHash:truepatches:# Change the ingress host name-target:kind:Ingressname:dummyhttppatch: -- op: replacepath: /spec/rules/0/hostvalue: vcluster-dummyhttp.lo.mixinet.net# Add reloader annotations -- it will only work if we install reloader on the# virtual cluster, as the one on the host cluster doesn't see the vcluster# deployment objects-target:kind:Deploymentname:dummyhttppatch: -- op: addpath: /metadata/annotationsvalue:reloader.stakater.com/auto: "true"reloader.stakater.com/rollout-strategy: "restart"
It is quite similar to the one we used on the Argo CD examples but uses a different DNS entry; to deploy it we run
kustomize and vkubectl:
kustomize build . vkubectl apply -f -
configmap/dummyhttp-configmap created
service/dummyhttp created
deployment.apps/dummyhttp created
ingress.networking.k8s.io/dummyhttp created
We can check that everything worked using curl:
curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/ jq -cM."c": "Vcluster Test Value","s": ""
The objects available on the vcluster now are:
vkubectl get all,configmap,ingress
NAME READY STATUS RESTARTS AGE
pod/dummyhttp-55569589bc-9zl7t 1/1 Running 0 24s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dummyhttp ClusterIP 10.43.51.39 <none>80/TCP 24s
service/kubernetes ClusterIP 10.43.153.12 <none>443/TCP 14m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/dummyhttp 1/1 1 1 24s
NAME DESIRED CURRENT READY AGE
replicaset.apps/dummyhttp-55569589bc 1 1 1 24s
NAME DATA AGE
configmap/dummyhttp-configmap 1 24s
configmap/kube-root-ca.crt 1 14m
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/dummyhttp traefik vcluster-dummyhttp.lo.mixinet.net 172.20.0.2,172.20.0.3,172.20.0.4 80 24s
While we have the following ones on the my-vcluster namespace of the host cluster:
kubectl get all,configmap,ingress -n my-vcluster
NAME READY STATUS RESTARTS AGE
pod/coredns-bbb5b66cc-snwpn-x-kube-system-x-my-vcluster 1/1 Running 0 18m
pod/dummyhttp-55569589bc-9zl7t-x-default-x-my-vcluster 1/1 Running 0 45s
pod/my-vcluster-0 1/1 Running 0 19m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dummyhttp-x-default-x-my-vcluster ClusterIP 10.43.51.39 <none>80/TCP 45s
service/kube-dns-x-kube-system-x-my-vcluster ClusterIP 10.43.91.198 <none>53/UDP,53/TCP,9153/TCP 18m
service/my-vcluster ClusterIP 10.43.153.12 <none>443/TCP,10250/TCP 19m
service/my-vcluster-headless ClusterIP None <none>443/TCP 19m
service/my-vcluster-node-k3d-argocd-agent-1 ClusterIP 10.43.189.188 <none>10250/TCP 18m
NAME READY AGE
statefulset.apps/my-vcluster 1/1 19m
NAME DATA AGE
configmap/coredns-x-kube-system-x-my-vcluster 2 18m
configmap/dummyhttp-configmap-x-default-x-my-vcluster 1 45s
configmap/kube-root-ca.crt 1 19m
configmap/kube-root-ca.crt-x-default-x-my-vcluster 1 11m
configmap/kube-root-ca.crt-x-kube-system-x-my-vcluster 1 18m
configmap/vc-coredns-my-vcluster 1 19m
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/dummyhttp-x-default-x-my-vcluster traefik vcluster-dummyhttp.lo.mixinet.net 172.20.0.2,172.20.0.3,172.20.0.4 80 45s
As shown, we have copies of the Service, Pod, Configmap and Ingress objects, but there is no copy of the
Deployment or ReplicaSet.
Creating a sealed secret for dummyhttpdTo use the hosts sealed secrets controller with the virtual cluster we will create the my-vcluster-default namespace
and add there the sealed secrets we want to have available as secrets on the default namespace of the virtual cluster:
After running the previous commands we have the following objects available on the host cluster:
kubectl get sealedsecrets.bitnami.com,secrets -n my-vcluster-default
NAME STATUS SYNCED AGE
sealedsecret.bitnami.com/dummyhttp-secret True 34s
NAME TYPE DATA AGE
secret/dummyhttp-secret Opaque 1 34s
And we can see that the secret is also available on the virtual cluster with the content we expected:
vkubectl get secrets
NAME TYPE DATA AGE
dummyhttp-secret Opaque 1 34s
vkubectl get secret/dummyhttp-secret --template=" .data.SECRET_VAR " \
base64 -d
Vcluster Boo
But the output of the curl command has not changed because, although we have the reloader controller deployed on the
host cluster, it does not see the Deployment object of the virtual one and the pods are not touched:
curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/ jq -cM."c": "Vcluster Test Value","s": ""
Installing the reloader applicationTo make reloader work on the virtual cluster we just need to install it as we did on the host using the following
kustomization.yaml file:
apiVersion:kustomize.config.k8s.io/v1beta1kind:Kustomizationnamespace:kube-systemresources:-github.com/stakater/Reloader/deployments/kubernetes/?ref=v1.4.2patches:# Add flags to reload workloads when ConfigMaps or Secrets are created or deleted-target:kind:Deploymentname:reloader-reloaderpatch: -- op: addpath: /spec/template/spec/containers/0/argsvalue:- '--reload-on-create=true'- '--reload-on-delete=true'- '--reload-strategy=annotations'
We deploy it with kustomize and vkubectl:
kustomize build . vkubectl apply -f -
serviceaccount/reloader-reloader created
clusterrole.rbac.authorization.k8s.io/reloader-reloader-role created
clusterrolebinding.rbac.authorization.k8s.io/reloader-reloader-role-binding created
deployment.apps/reloader-reloader created
As the controller was not available when the secret was created the pods linked to the Deployment are not updated, but
we can force things removing the secret on the host system; after we do that the secret is re-created from the sealed
version and copied to the virtual cluster where the reloader controller updates the pod and the curl command shows the
new output:
kubectl delete -n my-vcluster-default secrets dummyhttp-secret
secret "dummyhttp-secret" deleted
sleep 2
vkubectl get pods
NAME READY STATUS RESTARTS AGE
dummyhttp-78bf5fb885-fmsvs 1/1 Terminating 0 6m33s
dummyhttp-c68684bbf-nx8f9 1/1 Running 0 6s
curl -s https://vcluster-dummyhttp.lo.mixinet.net:8443/ jq -cM .
"c":"Vcluster Test Value","s":"Vcluster Boo"
If we change the secret on the host systems things get updated pretty quickly now:
Pause and restore the vclusterThe status of pods and statefulsets while the virtual cluster is active can be seen using kubectl:
kubectl get pods,statefulsets -n my-vcluster
NAME READY STATUS RESTARTS AGE
pod/coredns-bbb5b66cc-snwpn-x-kube-system-x-my-vcluster 1/1 Running 0 127m
pod/dummyhttp-587c7855d7-pt9b8-x-default-x-my-vcluster 1/1 Running 0 4m39s
pod/my-vcluster-0 1/1 Running 0 128m
pod/reloader-reloader-7f56c54d75-544gd-x-kube-system-x-my-vcluster 1/1 Running 0 60m
NAME READY AGE
statefulset.apps/my-vcluster 1/1 128m
Pausing the vclusterIf we don t need to use the virtual cluster we can pause it and after a small amount of time all Pods are gone because
the statefulSet is scaled down to 0 (note that other resources like volumes are not removed, but all the objects that
have to be scheduled and consume CPU cycles are not running, which can translate in a lot of savings when running on
clusters from cloud platforms or, in a local cluster like the one we are using, frees resources like CPU and memory that
now can be used for other things):
vcluster pause my-vcluster
11:20:47 info Scale down statefulSet my-vcluster/my-vcluster...
11:20:48 done Successfully paused vcluster my-vcluster/my-vcluster
kubectl get pods,statefulsets -n my-vcluster
NAME READY AGE
statefulset.apps/my-vcluster 0/0 130m
Now the curl command fails:
curl -s https://vcluster-dummyhttp.localhost.mixinet.net:8443
404 page not found
Although the ingress is still available (it returns a 404 because there is no pod behind the service):
kubectl get ingress -n my-vcluster
NAME CLASS HOSTS ADDRESS PORTS AGE
dummyhttp-x-default-x-my-vcluster traefik vcluster-dummyhttp.lo.mixinet.net 172.20.0.2,172.20.0.3,172.20.0.4 80 120m
In fact, the same problem happens when we try to connect to the vcluster API; the error shown by kubectl is related
to the TLS certificate because the 404 page uses the wildcard certificate instead of the self signed one:
vkubectl get pods
Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority
curl -s https://my-vcluster-api.lo.mixinet.net:8443/api/v1/
404 page not found
curl -v -s https://my-vcluster-api.lo.mixinet.net:8443/api/v1/ 2>&1 grep subject
* subject: CN=lo.mixinet.net
* subjectAltName: host "my-vcluster-api.lo.mixinet.net" matched cert's "*.lo.mixinet.net"
Resuming the vclusterWhen we want to use the virtual cluster again we just need to use the resume command:
Once all the pods are running the virtual cluster goes back to its previous state, although all of them were started, of
course.
Cleaning upThe virtual cluster can be removed using the delete command:
vcluster delete my-vcluster
12:09:18 info Delete vcluster my-vcluster...
12:09:18 done Successfully deleted virtual cluster my-vcluster in namespace my-vcluster
12:09:18 done Successfully deleted virtual cluster namespace my-vcluster
12:09:18 info Waiting for virtual cluster to be deleted...
12:09:50 done Virtual Cluster is deleted
That removes everything we used on this post except the sealed secrets and secrets that we put on the
my-vcluster-default namespace because it was created by us.
If we delete the namespace all the secrets and sealed secrets on it are also removed:
ConclusionsI believe that the use of virtual clusters can be a good option for two of the proposed use cases that I ve encountered
in real projects in the past:
need of short lived clusters for developers or teams,
execution of integration tests from CI pipelines that require a complete cluster (the tests can be run on virtual
clusters that are created on demand or paused and resumed when needed).
For both cases things can be set up using the Apache licensed product, although maybe evaluating the vCluster Platform
offering could be interesting.
In any case when everything is not done inside kubernetes we will also have to check how to manage the external services
(i.e. if we use databases or message buses as SaaS instead of deploying them inside our clusters we need to have a way
of creating, deleting or pause and resume those services).
DebConf 25 Preparations, by Stefano Rivera and Santiago Ruano Rinc n
DebConf 25 preparations continue. In April,
the bursary team reviewed and ranked bursary applications. Santiago Ruano Rinc n
examined the current state of the conference s finances, to see if we could
allocate any more money to bursaries. Stefano Rivera supported the bursary
team s work with infrastructure and advice and added some metrics to assist
Santiago s budget review. Santiago was also involved in different parts of the
organization, including Content team matters, as reviewing the first of
proposals, preparing public
information
about the new Academic Track; or coordinating different
aspects of the Day trip activities and the Conference Dinner.
PyPA tools updates, by Stefano Rivera
Around the beginning of the freeze (in retrospect, definitely too late) Stefano
looked at updating setuptools in the archive to 78.1.0. This brings support
for more comprehensive license expressions
(PEP-639), that people are expected to
adopt soon upstream. While the reverse-autopkgtests all passed, it all came with
some unexpected complications, and turned into a mini-transition. The new
setuptools broke shebangs for scripts
(pypa/setuptools#4952).
It also required a bump of wheel to 0.46 and wheel 0.46 now has a dependency
outside the standard library (it de-vendored packaging). This meant it was no
longer suitable to distribute a standalone wheel.whl file to seed into new
virtualenvs, as virtualenv does by default. The good news here is that
setuptools doesn t need wheel any more, it included its own
implementation of the bdist_wheel command, in 70.1. But the world hadn t
adapted to take advantage of this, yet. Stefano scrambled to get all of these
issues resolved upstream and in Debian:
pip: Don t check for wheel when invoked with --no-use-pep517
(pypa/pip#13330), automatically do
--no-use-pep517 builds without wheel
(pypa/pip#13358, rejected).
virtualenv: Don t include wheel
(pypa/virtualenv#2868) except on
Python 3.8
(pypa/virtualenv#2876) as pip
dropped Python 3.8 support in the same release that included #13330.
We re now at the point where python3-wheel-whl is no longer needed in Debian
unstable, and it should migrate to trixie.
Removing libcrypt-dev from build-essential, by Helmut Grohne
The crypt function was originally part of glibc, but it got separated to
libxcrypt. As a result, libc6-dev now depends on libcrypt-dev. This poses
a cycle during architecture cross bootstrap. As the number of packages actually
using crypt is relatively small, Helmut
proposed removing
the dependency. He analyzed an archive rebuild kindly performed by Santiago Vila
(not affiliated with Freexian) and estimated the necessary changes. It looks
like we may complete this with modifications to less than 300 source packages in
the forky cycle. Half of the bugs have been filed at this time. They are
tracked with libcrypt-*usertags.
Miscellaneous contributions
Carles uploaded a new version of simplemonitor.
Carles improved the documentation of salsa-ci-team/pipeline regarding piuparts
arguments.
Carles closed an FTBFS on gcc-15 on qnetload.
Carles worked on Catalan translations using po-debconf-manager: reviewed 57
translations and created their merge requests in salsa, created 59 bug reports
for packages that didn t merge in more than 30 days. Followed-up merge requests
and comments in bug reports. Managed some translations manually for packages
that are not in Salsa.
Lucas did some work on the DebConf Content and Bursary teams.
Lucas fixed multiple CVEs and bugs involving the upgrade from bookworm to
trixie in ruby3.3.
Lucas fixed a CVE in valkey in unstable.
Stefano updated beautifulsoup4, python-authlib, python-html2text,
python-packaging, python-pip, python-soupsieve, and unidecode.
Stefano packaged python-dependency-groups, a new vendored library in python-pip.
During an afternoon Bug Squashing Party in Montevideo, Santiago uploaded a
couple of packages fixing RC bugs #1057226
and #1102487. The
latter was a sponsored upload.
Thorsten uploaded new upstream versions of brlaser, ptouch-driver and
sane-airscan to get the latest upstream bug fixes into Trixie.
Rapha l filed an upstream bug on
zim for a
graphical glitch that he has been experiencing.
Colin did his usual routine work on the Python team: 21 packages upgraded to
new upstream versions, 8 CVEs fixed, and about 25 release-critical bugs fixed.
Helmut filed patches for 21 cross build failures.
Helmut uploaded a new version of
debvm
featuring a new tool
debefivm-create
to generate EFI-bootable disk images compatible with other tools such as libvirt
or VirtualBox. Much of the work was prototyped in earlier months. This
generalizes mmdebstrap-autopkgtest-build-qemu.
Helmut continued reporting undeclared file conflicts and suggested package
removals from unstable.
Helmut proposed build profiles for libftdi1
and gnupg2. To deal with recently added
dependencies in the architecture cross bootstrap package set.
Helmut managed the /usr-move transition. He worked on ensuring that systemd
would comply with Debian s policy.
Dumat continues to locate problems
here and there yielding discussion occasionally. He sent a patch for an upgrade
problem in zutils.
Anupa worked with the Debian publicity team to publish Micronews and Bits
posts.
Anupa worked with the DebConf 25 content team to review talk and event
proposals for DebConf 25.
With the trixie release approaching, I had the idea back in April to
organize a bug squashing party with the Debian Brasil community. I
believe the outcome was very positive, and we were able to tackle and
fix quite a number of release-critical bugs. This is a brief report
of what we did.
A remote BSP
It s not the first time I organize a BSP: back in 2019, I helped throw
another similar party in Toronto. The difference this time is that,
because Brazil is a big country and (perhaps most importantly) because
I m not currently living there, the BSP had to be done online.
I m a fan of social interactions (especially with the Brazilian
community), and in my experience we usually can achieve much more when
we get together in a physical place, but hey, you gotta do what you
gotta do
Most (if not all) of the folks interested in participating had busy
weekdays, so it was decided that we would meet during the weekends and
try to work on a few bugs over Jitsi. Nothing stopped people from
working on bugs during the week as well, of course.
A tag to rule them all
We used the bsp-2025-04-brazil usertag to mark those bugs that were
touched by us. You can see the full list of bugs here, although the
current list (as of 2025-05-11) is smaller than the one we had by
the end of April. I don t know what happened; maybe it s some glitch
with the BTS, or maybe someone removed the usertag by mistake.
Stats
In total, we had:
7 participants
37 bugs handled. Of those,
35 bugs fixed
The BSP officially started on 04 April 2025, and ended on 30
April 2025. I was able to attend meetings during two weekends; other
people participated more sporadically.
Outcome
As I said above, the Debian Brasil community is great and very engaged
in the project. Speaking more specifically about the Debian Brasil
Devel group, I can say that We have contributors with strong technical
skills, and I really love that we have this inclusive, extremely
technical culture where debugging and understanding things is really
core to pretty much all our discussions.
We already meet weekly on Thursdays to talk shop and help newcomers,
so having a remote BSP with this group seemed like a logical thing to
do. I m really glad to see our results and even happier to hear
positive feedback from the community during the last MiniDebConf in
Macei .
There s some interest in organizing another BSP, this time
face-to-face and during the next DebConf. I m all for it, as I love
fixing bugs and having a great time with friends. If you re
interested in attending, let me know.
Thanks, and until next time.
Dear Debian community,
This is bits from the DPL for April.
End of 10
I am sure I was speaking in the interest of the whole project when
joining the "End of 10" campaign. Here is what I wrote to the
initiators:
Hi Joseph and all drivers of the "End of 10" campaign,
On behalf of the entire Debian project, I would like to say that we
proudly join your great campaign. We stand with you in promoting Free
Software, defending users' freedoms, and protecting our planet by
avoiding unnecessary hardware waste.
Thank you for leading this important initiative.
Andreas Tille
Debian Project Leader
I have some goals I would like to share with you for my second term.
Ftpmaster delegation
This splits up into tasks that can be done before and after Trixie release.
Before Trixie:
1. Reducing Barriers to DFSG Compliance Checks
Back in 2002, Debian established a way to distribute cryptographic
software in the main archive, whereas such software had previously been
restricted to the non-US archive. One result of this arrangement which
influences our workflow is that all packages uploaded to the NEW queue
must remain on the server that hosts it. This requirement means that
members of the ftpmaster team must log in to that specific machine,
where they are limited to a restricted set of tools for reviewing
uploaded code.
This setup may act as a barrier to participation--particularly for
contributors who might otherwise assist with reviewing packages for DFSG
compliance. I believe it is time to reassess this limitation and work
toward removing such hurdles.
In October last year, we had some initial contact with SPI's legal
counsel, who noted that US regulations around cryptography have been
relaxed somewhat in recent years (as of 2021). This suggests it may now
be possible to revisit and potentially revise the conditions under which
we manage cryptographic software in the NEW queue.
I plan to investigate this further. If you have expertise in software or
export control law and are interested in helping with this topic, please
get in touch with me.
The ultimate goal is to make it easier for more people to contribute to
ensuring that code in the NEW queue complies with the DFSG.
2. Discussing Alternatives
My chances to reach out to other distributions remained limited.
However, regarding the processing of new software, I learned that
OpenSUSE uses a Git-based workflow that requires five "LGTM" approvals
from a group of trusted developers. As far as I know, Fedora follows a
similar approach.
Inspired by this, a recent community initiative--the Gateway to NEW
project--enables peer review of new packages for DFSG compliance before
they enter the NEW queue. This effort allows anyone to contribute by
reviewing packages and flagging potential issues in advance viaGit.
I particularly appreciate that the DFSG review is coupled with CI,
allowing for both license and technical evaluation.
While this process currently results in some duplication of work--since
final reviews are still performed by the ftpmaster team--it offers a
valuable opportunity to catch issues early and improve the overall
quality of uploads. If the community sees long-term value in this
approach, it could serve as a basis for evolving our workflows.
Integrating it more closely into DAK could streamline the process, and
we've recently seen that merge requests reflecting community suggestions
can be accepted promptly.
For now, I would like to gather opinions about how such initiatives
could best complement the current NEW processing, and whether greater
consensus on trusted peer review could help reduce the burden on the
team doing DFSG compliance checks. Submitting packages for review and
automated testing before uploading can improve quality and encourage
broader participation in safeguarding Debian's Free Software principles.
My explicit thanks go out to the Gateway to NEW team for their valuable
and forward-looking contribution to Debian.
3. Documenting Critical Workflows
Past ftpmaster trainees have told me that understanding the full set of
ftpmaster workflows can be quite difficult. While there is some useful
documentation thanks in particular to Sean Whitton for his work on
documenting NEW processing rules many other important tasks carried
out by the ftpmaster team remain undocumented or only partially so.
Comprehensive and accessible documentation would greatly benefit current
and future team members, especially those onboarding or assisting in
specific workflows. It would also help ensure continuity and
transparency in how critical parts of the archive are managed.
If such documentation already exists and I have simply overlooked it, I
would be happy to be corrected. Otherwise, I believe this is an area
where we need to improve significantly. Volunteers with a talent for
writing technical documentation are warmly invited to contact me--I'd be
happy to help establish connections with ftpmaster team members who are
willing to share their knowledge so that it can be written down and
preserved.
Once Trixie is released (hopefully before DebConf):
4. Split of the Ftpmaster Team into DFSG and Archive Teams
As discussed during the "Meet the ftpteam" BoF at DebConf24, I would
like to propose a structural refinement of the current Ftpmaster team
by introducing two different delegated teams:
DFSG Team
Archive Team (responsible for DAK maintenance and process tooling,
including releases)
(Alternative name suggestions are, of course, welcome.) The primary task
of the DFSG team would be the processing of the NEW queue and ensuring
that packages comply with the DFSG. The Archive team would focus on
maintaining DAK and handling the technical aspects of archive
management.
I am aware that, in the recent past, the ftpmaster team has decided not
to actively seek new members. While I respect the autonomy of each team,
the resulting lack of a recruitment pipeline has led to some friction
and concern within the wider community, including myself. As Debian
Project Leader, it is my responsibility to ensure the long-term
sustainability and resilience of our project, which includes fostering
an environment where new contributors can join and existing teams remain
effective and well-supported. Therefore, even if the current team does
not prioritize recruitment, I will actively seek and encourage new
contributors for both teams, with the aim of supporting openness and
collaboration.
This proposal is not intended as criticism of the current team's
dedication or achievements--on the contrary, I am grateful for the hard
work and commitment shown, often under challenging circumstances. My
intention is to help address the structural issues that have made
onboarding and specialization difficult and to ensure that both teams
are well-supported for the future.
I also believe that both teams should regularly inform the Debian
community about the policies and procedures they apply. I welcome any
suggestions for a more detailed description of the tasks involved, as
well as feedback on how best to implement this change in a way that
supports collaboration and transparency.
My intention with this proposal is to foster a more open and effective
working environment, and I am committed to working with all involved to
ensure that any changes are made collaboratively and with respect for
the important work already being done.
I'm aware that the ideas outlined above touch on core parts of how
Debian operates and involve responsibilities across multiple teams.
These are not small changes, and implementing them will require
thoughtful discussion and collaboration.
To move this forward, I've registered a dedicated BoF for DebConf. To
make the most of that opportunity, I'm looking for volunteers who feel
committed to improving our workflows and processes. With your help, we
can prepare concrete and sensible proposals in advance--so the limited
time of the BoF can be used effectively for decision-making and
consensus-building.
In short: I need your help to bring these changes to life. From my
experience in my last term, I know that when it truly matters, the
Debian community comes together--and I trust that spirit will guide us
again.
Please also note: we had a "Call for volunteers" five years ago, and
much of what was written there still holds true today. I've been told
that the response back then was overwhelming--but that training such a
large number of volunteers didn't scale well. This time, I hope we can
find a more sustainable approach: training a few dedicated people first,
and then enabling them to pass on their knowledge. This will also be a
topic at the DebCamp sprint.
Dealing with Dormant Packages
Debian was founded on the principle that each piece of software should
be maintained by someone with expertise in it--typically a single,
responsible maintainer. This model formed the historical foundation of
Debian's packaging system and helped establish high standards of quality
and accountability. However, as the project has grown and the number of
packages has expanded, this model no longer scales well in all areas.
Team maintenance has since emerged as a practical complement, allowing
multiple contributors to share responsibility and reduce
bottlenecks--depending on each team's internal policy.
While working on the Bug of the Day initiative, I observed a
significant number of packages that have not been updated in a long
time. In the case of team-maintained packages, addressing this is often
straightforward: team uploads can be made, or the team can be asked
whether the package should be removed. We've also identified many
packages that would fit well under the umbrella of active teams, such as
language teams like Debian Perl and Debian Python, or blends like Debian
Games and Debian Multimedia. Often, no one has taken action--not because
of disagreement, but simply due to inattention or a lack of initiative.
In addition, we've found several packages that probably should be
removed entirely. In those cases, we've filed bugs with pre-removal
warnings, which can later be escalated to removal requests.
When a package is still formally maintained by an individual, but shows
signs of neglect (e.g., no uploads for years, unfixed RC bugs, failing
autopkgtests), we currently have three main tools:
The MIA process, which handles inactive or unreachable
maintainers.
Package Salvaging, which allows contributors to take over
maintenance if conditions are met.
Non-Maintainer Uploads (NMUs), which are limited to specific,
well-defined fixes (which do not include things like migration
to Salsa).
These mechanisms are important and valuable, but they don't always allow
us to react swiftly or comprehensively enough. Our tools for identifying
packages that are effectively unmaintained are relatively weak, and the
thresholds for taking action are often high.
The Package Salvage team is currently trialing a process we've
provisionally called "Intend to NMU" (ITN). The name is admittedly
questionable--some have suggested alternatives like "Intent to
Orphan"--and discussion about this is ongoing on debian-devel. The
mechanism is intended for situations where packages appear inactive but
aren't yet formally orphaned, introducing a clear 21-day notice period
before NMUs, similar in spirit to the existing ITS process. The
discussion has sparked suggestions for expanding NMU rules.
While it is crucial not to undermine the autonomy of maintainers who
remain actively involved, we also must not allow a strict interpretation
of this autonomy to block needed improvements to obviously neglected
packages.
To be clear: I do not propose to change the rights of maintainers who
are clearly active and invested in their packages. That model has served
us well. However, we must also be honest that, in some cases,
maintainers stop contributing--quietly and without transition plans. In
those situations, we need more agile and scalable procedures to uphold
Debian's high standards.
To that end, I've registered a BoF session for DebConf25 to discuss
potential improvements in how we handle dormant packages. These
discussions will be prepared during a sprint at DebCamp, where I hope to
work with others on concrete ideas.
Among the topics I want to revisit is my proposal from last November on
debian-devel, titled "Barriers between packages and other people".
While the thread prompted substantial discussion, it understandably
didn't lead to consensus. I intend to ensure the various viewpoints are
fairly summarised--ideally by someone with a more neutral stance than
myself--and, if possible, work toward a formal proposal during the
DebCamp sprint to present at the DebConf BoF.
My hope is that we can agree on mechanisms that allow us to act more
effectively in situations where formerly very active volunteers have,
for whatever reason, moved on. That way, we can protect both Debian's
quality and its collaborative spirit.
Building Sustainable Funding for Debian
Debian incurs ongoing expenses to support its
infrastructure--particularly hardware maintenance and upgrades--as well as
to fund in-person meetings like sprints and mini-DebConfs. These
investments are essential to our continued success: they enable
productive collaboration and ensure the robustness of the operating
system we provide to users and derivative distributions around the
world.
While DebConf benefits from generous sponsorship, and we regularly
receive donated hardware, there is still considerable room to grow our
financial base--especially to support less visible but equally critical
activities. One key goal is to establish a more constant and predictable
stream of income, helping Debian plan ahead and respond more flexibly to
emerging needs.
This presents an excellent opportunity for contributors who may not be
involved in packaging or technical development. Many of us in Debian are
engineers first--and fundraising is not something we've been trained to
do. But just like technical work, building sustainable funding requires
expertise and long-term engagement.
If you're someone who's passionate about Free Software and has
experience with fundraising, donor outreach, sponsorship acquisition, or
nonprofit development strategy, we would deeply value your help.
Supporting Debian doesn't have to mean writing code. Helping us build a
steady and reliable financial foundation is just as important--and could
make a lasting impact.
Kind regards
Andreas.
PS: In April I also planted my 5000th tree and while this is off-topic here
I'm proud to share this information with my fellow Debian friends.
The Linux kernel's development process makes use of PGP. The most relevant part
here is that subsystem maintainers are supposed to use signed tags in their
pull requests to Linus Torvalds. As the concept of keyservers is considered
broken,
Konstantin Ryabitsev maintains a collection of relevant keys in a git
repository.
As of today (at commit
a0bc65fb27f5033beddf9d1ad97d67c353849be2)
there are 602 valid keys tracked in that repository. The requirement for a key
to be added there is that there must be at least one trust path from Linus
Torvalds' key to this key of length at most 5 within that keyring.
Occasionally it happens that a key loses its trust paths because someone in
these paths replaced their key, or keys expired. Currently this affects 2 keys.
However there is a problem on the horizon: GnuPG 2.4.x started to reject
third-party key signatures using the SHA-1 hash algorithm. In general that's
good, SHA-1 isn't considered secure any more for more than 20
years.
This doesn't directly affect the kernel-pgpkeys repo, because the trust path
checking doesn't rely on GnuPG trusting the signatures; there is a dedicated
tool that parses the keyring contents and currently accepts signatures using
SHA-1. Also signatures are not thrown away usually, but there are exceptions:
Recently Theodore Ts'o asked to update his
certificate.
When Konstantin imported the updated certificate GnuPG's "cleaning" was applied
which dropped all SHA-1 signatures. So Theodore Ts'o's key lost 168 signatures,
among them one by Linus Torvalds on his primary UID.
That made me wonder what would be the effect on the web of trust if all SHA-1
signatures were dropped. Here are the facts:
There are 7976 signatures tracked in the korg-pgpkeys repo that are
considered valid, 6045 of them use SHA-1.
Only considering the primary UID Linus Torvalds directly signed 40 public
keys, 38 of these using SHA-1. One of the two keys that is still "properly"
signed, doesn't sign any other key. So nearly all trust paths go through a
single key.
When not considering SHA-1 signatures there are 485 public keys without a
trust path from Linus Torvalds of length 5 or less. So today these 485
public keys would not qualify to be added to the pgpkeys git repository.
Among the people being dropped are Andrew Morton, Greg Kroah-Hartman, H.
Peter Anvin, Ingo Molnar, Junio C Hamano, Konstantin Ryabitsev, Peter
Zijlstra, Stephen Rothwell and Thomas Gleixner.
The size of the kernel strong set is reduced from 358 to 94.
If you attend Embedded Recipes 2025 next
week, there is an opportunity to improve the situation: Together with Ahmad
Fatoum I'm organizing a keysigning session. If
you want to participate, send your public key to
er2025-keysigning@baylibre.com before
2025-05-12 08:00 UTC.
After attending the 2024 LibreOffice conference in Luxembourg, I visited Paris in October 2024.
If you are wondering whether I needed another visa to cross the border into France I didn t! Further, they are both also EU members, which means you don t need to go through customs either. Thus, crossing the Luxembourg-France border is no different from crossing Indian state borders - like going from Rajasthan to Uttar Pradesh.
I took a TGV train from Luxembourg Central Station, which was within walking distance from my hostel. The train took only 2 hours and 20 minutes to cover the 300 km distance to Paris. It departed from Luxembourg at 10:00 AM and reached Paris at 12:20 PM. The ride was smooth and comfortable, arriving on time. It gave me an opportunity to see the countryside of France. I booked the train ticket online a couple of days prior through the Omio website.
TGV train I rode from Luxembourg to Paris
I planned the first day with my friend Joenio, whom I met upon arriving in Paris Gare de l Est station, along with his wife Mari. We went to my hostel (which was within walking distance from the station) to store my luggage, but we were informed that we needed to wait for a couple of hours before I could check in. Consequently, we went to an Italian restaurant nearby for lunch, where I ordered pasta. My hostel was unbelievably cheap by French standards (25 euros per night) that Joenio was shocked when he learned about it.
Pasta I had in Paris
Walking in the city, I noticed it had separate cycling tracks and wide footpaths, just like Luxembourg. The traffic was also organized. For instance, there were traffic lights even for pedestrian crossings, unlike India, where crossing roads can be a nightmare. Car drivers stopping for pedestrians is a big improvement over what I am used to in India. The weather was also pleasant. It was a bit on the cooler side - around 15 degrees Celsius - and I had to wear a jacket.
A cycling track in Paris
After lunch, we returned to my hostel for my check-in at around 3 o clock. Then, we went to the Luxembourg Museum (Mus e du Luxembourg in French) as Joenio had booked tickets for an exhibition of paintings by the Brazilian painter Tarsila do Amaral. To reach there, we took a subway train from Gare du Nord station. The Paris subway charges 2.15 euros irrespective of the distance (or number of stations) traveled, as opposed to other metro systems I have used.
We reached the museum at around 4 o clock. I found the paintings beautiful, but I would have appreciated them much more if the descriptions were in English.
Luxembourg Museum
Afterward, we went to a beautiful garden just behind the museum. It served as a great spot to relax and take pictures. Following this, we walked to the Pantheon - a well-known attraction in the city. It is a church built a couple of centuries ago. It has a dome-shaped structure at the top, recognizable from far away.
A shot of the park near to the Luxembourg Museum
Pantheon, one of the attractions of Paris.
Then we went to Notre Dame after having evening snacks and coffee at a nearby bakery. The Notre Dame was just over a kilometer from the Pantheon, so we took a walk. We also crossed the beautiful Seine river. On the way, I sampled a cr pe, a signature dish of France. The shop was named Cr perie and had many varieties of Cr pe. I took the one with eggs and Emmental cheese. It was savory and delicious.
Photo with Joenio and Mari
Notre Dame, another tourist attraction of Paris.
By the time we reached Notre Dame, it was 07:30 PM. I learned from Joenio that Notre Dame was closed and being renovated due to a fire a couple of years ago, so we just sat around and clicked photos. It is a catholic cathedral built in French Gothic architecture (I read that on Wikipedia ;)). I read on Wikipedia that it is located on an island named le de la Cit and I didn t even realize we are on an island.
At night, we visited the most well-known attraction of Paris, The Eiffel Tower. We again took the subway, alighting at the Bir-Hakeim station, followed by a short walk. We reached the Eiffel Tower at 9 o clock. It was lit bright yellow. There was not much to do there, so we just clicked photos and hung out. After that, I came back to my hostel.
My photo with Eiffel Tower in the background
Next day, I roamed around the city by walking mostly. France is known for its bakeries, so I checked out a couple of local bakeries. I had espresso a couple of times and sampled croissant, pain au chocolat and lemon meringue tartlet.
Items at a bakery in Paris. Items from left to right are: Chocolate Twist, Sugar briochette, Pain au Chocolat, Croissant with almonds, Croissant, Croissant with chocolate hazlenut filling.
Here are some random shots:
The Paris subway
Inside a Paris subway
A random building and road in Paris
A shot near Seine river
A view of Seine river
On the third day, I had my flight for India. Thus, I checked out of the hostel early in the morning, took an RR train from Gare du Nord station to reach the airport. It costs 11.8 euros.
I heard some of my friends had bad experiences in France. Thus, I had the impression that I would not feel welcomed. Furthermore, I have encountered language problems in my previous Europe trip to Albania and Kosovo. Likewise, I learned a couple of French words, like how to say thank you and good morning, which went a long way.
However, I didn t have bad experiences in Paris, except for one instance in which I asked my hostel s reception about my misplaced watch and the person at the reception asked me to be polite by being rude. She said, Excuse me! You don t know how to say Good Morning?
Overall, I enjoyed my time in Paris and would like to thank Joenio and Mari for joining me. I would also like to thank Sophie, who gave me a map of Paris.
Let s end this post here. I ll meet you in the next one!
Credits: Thanks to contrapunctus for reviewing this post before publishing
As a followup of my post about the use of argocd-autopilot
I m going to deploy various applications to the cluster using Argo CD from the same
repository we used on the previous post.
For our examples we are going to test a solution to the problem we had when we updated a ConfigMap used by the
argocd-server (the resource was updated but the application Pod was not because there was no change on the
argocd-server deployment); our original fix was to kill the pod manually, but the manual operation is something we
want to avoid.
The proposed solution to this kind of issues on the
helm documentation is to add
annotations to the Deployments with values that are a hash of the ConfigMaps or Secrets used by them, this way if
a file is updated the annotation is also updated and when the Deployment changes are applied a roll out of the pods is
triggered.
On this post we will install a couple of controllers and an application to show how we can handle Secrets with
argocd and solve the issue with updates on ConfigMaps and Secrets, to do it we will execute the following tasks:
Deploy the Reloader controller to our cluster. It is a tool that watches
changes in ConfigMaps and Secrets and does rolling upgrades on the Pods that use them from Deployment,
StatefulSet, DaemonSet or DeploymentConfig objects when they are updated (by default we have to add some
annotations to the objects to make things work).
Deploy a simple application that can use ConfigMaps and Secrets and test that the Reloader controller does its
job when we add or update a ConfigMap.
Install the Sealed Secrets controller to manage secrets inside our
cluster, use it to add a secret to our sample application and see that the application is reloaded automatically.
Creating the test project for argocd-autopilotAs we did our installation using argocd-autopilot we will use its structure to manage the applications.
The first thing to do is to create a project (we will name it test) as follows:
argocd-autopilot project create test
INFO cloning git repository: https://forgejo.mixinet.net/blogops/argocd.git
Enumerating objects: 18, done.
Counting objects: 100% (18/18), done.
Compressing objects: 100% (16/16), done.
Total 18 (delta 1), reused 0 (delta 0), pack-reused 0
INFO using revision: "", installation path: "/"
INFO pushing new project manifest to repo
INFO project created: 'test'
Now that the test project is available we will use it on our argocd-autopilot invocations when creating
applications.
Installing the reloader controllerTo add the reloader application to the test project as a kustomize application and deploy it on the tools
namespace with argocd-autopilot we do the following:
argocd-autopilot app create reloader \
--app 'github.com/stakater/Reloader/deployments/kubernetes/?ref=v1.4.2' \
--project test --type kustomize --dest-namespace tools
INFO cloning git repository: https://forgejo.mixinet.net/blogops/argocd.git
Enumerating objects: 19, done.
Counting objects: 100% (19/19), done.
Compressing objects: 100% (18/18), done.
Total 19 (delta 2), reused 0 (delta 0), pack-reused 0
INFO using revision: "", installation path: "/"
INFO created 'application namespace' file at '/bootstrap/cluster-resources/in-cluster/tools-ns.yaml'
INFO committing changes to gitops repo...
INFO installed application: reloader
That command creates four files on the argocd repository:
One to create the tools namespace:bootstrap/cluster-resources/in-cluster/tools-ns.yaml
The kustomization.yaml file for the test project (by default it includes the same configuration used on the
base definition, but we could make other changes if needed):apps/reloader/overlays/test/kustomization.yaml
The config.json file used to define the application on argocd for the test project (it points to the folder
that includes the previous kustomization.yaml file):apps/reloader/overlays/test/config.json
We can check that the application is working using the argocd command line application:
argocd app get argocd/test-reloader -o tree
Name: argocd/test-reloader
Project: test
Server: https://kubernetes.default.svc
Namespace: tools
URL: https://argocd.lo.mixinet.net:8443/applications/test-reloader
Source:
- Repo: https://forgejo.mixinet.net/blogops/argocd.git
Target:
Path: apps/reloader/overlays/test
SyncWindow: Sync Allowed
Sync Policy: Automated (Prune)
Sync Status: Synced to (2893b56)
Health Status: Healthy
KIND/NAME STATUS HEALTH MESSAGE
ClusterRole/reloader-reloader-role Synced
ClusterRoleBinding/reloader-reloader-role-binding Synced
ServiceAccount/reloader-reloader Synced serviceaccount/reloader-reloader created
Deployment/reloader-reloader Synced Healthy deployment.apps/reloader-reloader created
ReplicaSet/reloader-reloader-5b6dcc7b6f Healthy
Pod/reloader-reloader-5b6dcc7b6f-vwjcx Healthy
Adding flags to the reloader serverThe runtime configuration flags for the reloader server are described on the project
README.md
file, in our case we want to adjust three values:
We want to enable the option to reload a workload when a ConfigMap or Secret is created,
We want to enable the option to reload a workload when a ConfigMap or Secret is deleted,
We want to use the annotations strategy for reloads, as it is the recommended mode of operation when using argocd.
To pass them we edit the apps/reloader/overlays/test/kustomization.yaml file to patch the pod container template, the
text added is the following:
patches:# Add flags to reload workloads when ConfigMaps or Secrets are created or deleted-target:kind:Deploymentname:reloader-reloaderpatch: -- op: addpath: /spec/template/spec/containers/0/argsvalue:- '--reload-on-create=true'- '--reload-on-delete=true'- '--reload-strategy=annotations'
After committing and pushing the updated file the system launches the application with the new options.
The dummyhttp applicationTo do a quick test we are going to deploy the dummyhttp web server using an
image generated using the following Dockerfile:
# Image to run the dummyhttp application <https://github.com/svenstaro/dummyhttp># This arg could be passed by the container build command (used with mirrors)ARG OCI_REGISTRY_PREFIX# Latest tested version of alpineFROM $ OCI_REGISTRY_PREFIX alpine:3.21.3# Tool versionsARG DUMMYHTTP_VERS=1.1.1# Download binaryRUN ARCH="$(apk --print-arch)"&&\
VERS="$DUMMYHTTP_VERS"&&\
URL="https://github.com/svenstaro/dummyhttp/releases/download/v$VERS/dummyhttp-$VERS-$ARCH-unknown-linux-musl"&&\
wget "$URL"-O"/tmp/dummyhttp"&&\
install /tmp/dummyhttp /usr/local/bin &&\
rm-f /tmp/dummyhttp
# Set the entrypoint to /usr/local/bin/dummyhttpENTRYPOINT [ "/usr/local/bin/dummyhttp" ]
The kustomize base application is available on a monorepo that contains the following files:
A Deployment definition that uses the previous image but uses /bin/sh -c as its entrypoint (command in the
k8s Pod terminology) and passes as its argument a string that runs the eval command to be able to expand
environment variables passed to the pod (the definition includes two optional variables, one taken from a
ConfigMap and another one from a Secret):
Deploying the dummyhttp application from argocdWe could create the dummyhttp application using the argocd-autopilot command as we ve done on the reloader case,
but we are going to do it manually to show how simple it is.
First we ve created the apps/dummyhttp/base/kustomization.yaml file to include the application from the previous
repository:
And finally we add the apps/dummyhttp/overlays/test/config.json file to configure the application as the
ApplicationSet defined by argocd-autopilot expects:
Patching the applicationNow we will add patches to the apps/dummyhttp/overlays/test/kustomization.yaml file:
One to add annotations for reloader (one to enable it and another one to set the roll out strategy to restart to
avoid touching the deployments, as that can generate issues with argocd).
Another to change the ingress hostname (not really needed, but something quite reasonable for a specific project).
After committing and pushing the changes we can use the argocd cli to check the status of the application:
argocd app get argocd/test-dummyhttp -o tree
Name: argocd/test-dummyhttp
Project: test
Server: https://kubernetes.default.svc
Namespace: default
URL: https://argocd.lo.mixinet.net:8443/applications/test-dummyhttp
Source:
- Repo: https://forgejo.mixinet.net/blogops/argocd.git
Target:
Path: apps/dummyhttp/overlays/test
SyncWindow: Sync Allowed
Sync Policy: Automated (Prune)
Sync Status: Synced to (fbc6031)
Health Status: Healthy
KIND/NAME STATUS HEALTH MESSAGE
Deployment/dummyhttp Synced Healthy deployment.apps/dummyhttp configured
ReplicaSet/dummyhttp-55569589bc Healthy
Pod/dummyhttp-55569589bc-qhnfk Healthy
Ingress/dummyhttp Synced Healthy ingress.networking.k8s.io/dummyhttp configured
Service/dummyhttp Synced Healthy service/dummyhttp unchanged
Endpoints/dummyhttp
EndpointSlice/dummyhttp-x57bl
As we can see, the Deployment and Ingress where updated, but the Service is unchanged.
To validate that the ingress is using the new hostname we can use curl:
curl -s https://dummyhttp.lo.mixinet.net:8443/
404 page not found
curl -s https://test-dummyhttp.lo.mixinet.net:8443/
"c": "", "s": ""
Adding a ConfigMapNow that the system is adjusted to reload the application when the ConfigMap or Secret is created, deleted or
updated we are ready to add one file and see how the system reacts.
We modify the apps/dummyhttp/overlays/test/kustomization.yaml file to create the ConfigMap using the
configMapGenerator as follows:
After committing and pushing the changes we can see that the ConfigMap is available, the pod has been deleted and
started again and the curl output includes the new value:
kubectl get configmaps,pods
NAME READY STATUS RESTARTS AGE
configmap/dummyhttp-configmap 1 11s
configmap/kube-root-ca.crt 1 4d7h
NAME DATA AGE
pod/dummyhttp-779c96c44b-pjq4d 1/1 Running 0 11s
pod/dummyhttp-fc964557f-jvpkx 1/1 Terminating 0 2m42s
curl -s https://test-dummyhttp.lo.mixinet.net:8443 jq -M .
"c": "Default Test Value",
"s": ""
Using helm with argocd-autopilotRight now there is no direct support in argocd-autopilot to manage applications using helm (see the issue
#38 on the project), but we want to use a chart in our
next example.
There are multiple ways to add the support, but the simplest one that allows us to keep using argocd-autopilot is to
use kustomize applications that call helm as described
here.
The only thing needed before being able to use the approach is to add the kustomize.buildOptions flag to the
argocd-cm on the bootstrap/argo-cd/kustomization.yaml file, its contents now are follows:
bootstrap/argo-cd/kustomization.yaml
apiVersion:kustomize.config.k8s.io/v1beta1configMapGenerator:-behavior:mergeliterals:# Enable helm usage from kustomize (see https://github.com/argoproj/argo-cd/issues/2789#issuecomment-960271294)-kustomize.buildOptions="--enable-helm"-repository.credentials=- passwordSecret:key: git_tokenname: autopilot-secreturl: https://forgejo.mixinet.net/usernameSecret:key: git_usernamename: autopilot-secretname:argocd-cm# Disable TLS for the Argo Server (see https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#traefik-v30)-behavior:mergeliterals:-"server.insecure=true"name:argocd-cmd-params-cmkind:Kustomizationnamespace:argocdresources:-github.com/argoproj-labs/argocd-autopilot/manifests/base?ref=v0.4.19-ingress_route.yaml
On the following section we will explain how the application is defined to make things work.
Installing the sealed-secrets controllerTo manage secrets in our cluster we are going to use the
sealed-secrets controller and to install it we are going to use its
chart.
As we mentioned on the previous section, the idea is to create a kustomize application and use that to deploy the
chart, but we are going to create the files manually, as we are not going import the base kustomization files from a
remote repository.
As there is no clear way to override helm Chart values using
overlays we are going to use a generator to create the helm configuration from an external resource and include it
from our overlays (the idea has been taken from this repository,
which was referenced from a comment
on the kustomize issue #38 mentioned earlier).
The sealed-secrets applicationWe have created the following files and folders manually:
apps/sealed-secrets/
helm
chart.yaml
kustomization.yaml
overlays
test
config.json
kustomization.yaml
values.yaml
The helm folder contains the generator template that will be included from our overlays.
The kustomization.yaml includes the chart.yaml as a resource:
apps/sealed-secrets/helm/kustomization.yaml
And the chart.yaml file defines the HelmChartInflationGenerator:
apps/sealed-secrets/helm/chart.yaml
apiVersion:builtinkind:HelmChartInflationGeneratormetadata:name:sealed-secretsreleaseName:sealed-secretsname:sealed-secretsnamespace:kube-systemrepo:https://bitnami-labs.github.io/sealed-secretsversion:2.17.2includeCRDs:true# Add common values to all argo-cd projects inlinevaluesInline:fullnameOverride:sealed-secrets-controller# Load a values.yaml file from the same directory that uses this generatorvaluesFile:values.yaml
For this chart the template adjusts the namespace to kube-system and adds the fullnameOverride on the
valuesInline key because we want to use those settings on all the projects (they are the values expected by the
kubeseal command line application, so we adjust them to avoid the need to add additional parameters to it).
We adjust global values as inline to be able to use a the valuesFile from our overlays; as we are using a generator
the path is relative to the folder that contains the kustomization.yaml file that calls it, in our case we will need
to have a values.yaml file on each overlay folder (if we don t want to overwrite any values for a project we can
create an empty file, but it has to exist).
Finally, our overlay folder contains three files, a kustomization.yaml file that includes the generator from the
helm folder, the values.yaml file needed by the chart and the config.json file used by argocd-autopilot to
install the application.
The kustomization.yaml file contents are:
apps/sealed-secrets/overlays/test/kustomization.yaml
apiVersion:kustomize.config.k8s.io/v1beta1kind:Kustomization# Uncomment if you want to add additional resources using kustomize#resources:#- ../../basegenerators:-../../helm
The values.yaml file enables the ingress for the application and adjusts its hostname:
apps/sealed-secrets/overlays/test/values.yaml
Once we commit and push the files the sealed-secrets application is installed in our cluster, we can check it using
curl to get the public certificate used by it:
That invocation needs to have access to the cluster to do its job and in our case it works because we modified the chart
to use the kube-system namespace and set the controller name to sealed-secrets-controller as the tool expects.
If we need to create the secrets without credentials we can connect to the ingress address we added to retrieve the
public key:
Or, if we don t have access to the ingress address, we can save the certificate on a file and use it instead of the URL.
The sealed version of the secret looks like this:
This file can be deployed to the cluster to create the secret (in our case we will add it to the argocd application),
but before doing that we are going to check the output of our dummyhttp service and get the list of Secrets and
SealedSecrets in the default namespace:
curl -s https://test-dummyhttp.lo.mixinet.net:8443 jq -M .
"c": "Default Test Value",
"s": ""
kubectl get sealedsecrets,secrets
No resources found in default namespace.
Now we add the SealedSecret to the dummyapp copying the file and adding it to the kustomization.yaml file:
Once we commit and push the files Argo CD creates the SealedSecret and the controller generates the Secret:
kubectl apply -f /tmp/dummyhttp-sealed-secret.yaml
sealedsecret.bitnami.com/dummyhttp-secret created
kubectl get sealedsecrets,secrets
NAME STATUS SYNCED AGE
sealedsecret.bitnami.com/dummyhttp-secret True 3s
NAME TYPE DATA AGE
secret/dummyhttp-secret Opaque 1 3s
If we check the command output we can see the new value of the secret:
Using sealed-secrets in production clustersIf you plan to use sealed-secrets look into its
documentation to understand how it manages the
private keys, how to backup things and keep in mind that, as the documentation
explains, you can rotate
your sealed version of the secrets, but that doesn t change the actual secrets.
If you want to rotate your secrets you have to update them and commit the sealed version of the updates (as the
controller also rotates the encryption keys your new sealed version will also be using a newer key, so you will be doing
both things at the same time).
Final remarksOn this post we have seen how to deploy applications using the argocd-autopilot model, including the use of helm
charts inside kustomize applications and how to install and use the sealed-secrets controller.
It has been interesting and I ve learnt a lot about argocd in the process, but I believe that if I ever want to use it
in production I will also review the native helm support in argocd using a separate repository to manage the
applications, at least to be able to compare it to the model explained here.
The subtitle of Paper Soldiers is "How the Weaponization of the
Dollar Changed the World Order," which may give you the impression that
this book is about US use of the dollar system for political purposes such
as sanctions. Do not be fooled like I was; this subtitle is, at best,
deceptive. Coverage of the weaponization of the dollar is superficial and
limited to a few chapters. This book is, instead, a history of the strong
dollar policy told via a collection of hagiographies of US Treasury
Secretaries and written with all of the skeptical cynicism of a poleaxed
fawn.
There is going to be some grumbling about the state of journalism in this
review.
Per the author's note, Saleha Mohsin is the Bloomberg News beat reporter
for the US Department of the Treasury. That is, sadly, exactly what this
book reads like: routine beat reporting. Mohsin asked current and former
Treasury officials what they were thinking at various points in history
and then wrote down their answers without, so far as I can tell,
considering any contradictory evidence or wondering whether they were
telling the truth. Paper Soldiers does contain extensive notes
(those plus the index fill about forty pages), so I guess you could do the
cross-checking yourself, although apparently most of the interviews for
this book were "on background" and are therefore unattributed. (Is this
weird? I feel like this is weird.) Mohsin adds a bit of utterly
conventional and uncritical economic framing and casts the whole project
in the sort of slightly breathless and dramatized prose style that infests
routine news stories in the US.
I find this style of book unbelievably frustrating because it represents
such a wasted opportunity. To me, the point of book-length journalism is
precisely to not write in this style. When you're trying to crank
out two or three articles a week covering current events, I understand why
there isn't always space or time to go deep into background, skepticism,
and contrary opinions. But when you expand that material into a book,
surely the whole point is to take the time to do some real reporting. Dig
into what people told you, see if they're lying, talk to the people who
disagree with them, question the conventional assumptions, and show your
work on the page so that the reader is smarter after finishing your book
than they were before they started. International political economics is
not a sequence of objective facts. It's a set of decisions made in pursuit
of economic and political theories that are disputed and arguable, and I
think you owe the reader some sense of the argument and, ideally, some
defensible position on the merits that is more than a transcription of
your interviews.
This is... not that.
It's a power loop that the United States still enjoys today: trust in
America's dollar (and its democratic government) allows for cheap debt
financing, which buys health care built on the most advanced research
and development and inventions like airplanes and the iPhone. All of
this is propelled by free market innovation and the superpowered
strength to keep the nation safe from foreign threats. That investment
boosts the nation's economic, military, and technological prowess,
making its economy (and the dollar) even more attractive.
Let me be precise about my criticism. I am not saying that every
contention in the above excerpt is wrong. Some of them are probably
correct; more of them are at least arguable. This book is strictly about
the era after Bretton Woods, so using airplanes as an example invention is
a bizarre choice, but sure, whatever, I get the point. My criticism is
that paragraphs like this, as written in this book, are not introductions
to deeper discussions that question or defend that model of economic and
political power. They are simple assertions that stand entirely
unsupported. Mohsin routinely writes paragraphs like the above as if they
are self-evident, and then immediately moves on to the next anecdote about
Treasury dollar policy.
Take, for example, the role of the US dollar as the world's reserve
currency, which roughly means that most international transactions are
conducted in dollars and numerous countries and organizations around the
world hold large deposits in dollars instead of in their native currency.
The conventional wisdom holds that this is a great boon to the US economy,
but there are also substantive critiques and questions about that
conventional wisdom. You would never know that from this book; Mohsin
asserts the conventional wisdom about reserve currencies without so much
as a hint that anyone might disagree.
For example, one common argument, repeated several times by Mohsin, is
that the US can only get away with the amount of deficit spending and
cheap borrowing that it does because the dollar is the world's reserve
currency. Consider two other countries whose currencies are clearly not
the international reserve currency: Japan and the United Kingdom. The
current US debt to GDP ratio is about 125% and the current interest rate
on US 10-year bonds is about 4.2%. The current Japanese debt to GDP ratio
is about 260% and the current interest rate on Japanese 10-year bonds is
about 1.2%. The current UK debt to GDP ratio is 160% and the current
interest rate on UK 10-year bonds is 4.5%. Are you seeing the dramatic
effects of the role of the dollar as reserve currency? Me either.
Again, I am not saying that this is a decisive counter-argument. I am not
an economist; I'm just some random guy on the Internet who finds
macroeconomics interesting and reads a few newsletters. I know the
Japanese bond market is unusual in ways I'm not accounting for. There may
well be compelling arguments for why reserve currency status matters
immensely for US borrowing capacity. My point is not that Mohsin is wrong;
my point is that you have to convince me and she doesn't even try.
Nowhere in this book is a serious effort to view conventional wisdom with
skepticism or confront it with opposing arguments. Instead, this book is
full of blithe assertions that happen to support the narrative the author
was fed by a bunch of former Treasury officials and does not appear to
question in any way. I want books like this to increase my understanding
of the world. To do that, they need to show me multiple sides of debates
and teach me how to evaluate evidence, not simply reinforce a superficial
conventional wisdom.
It doesn't help that whatever fact-checking process this book went through
left some glaring errors. For example, on the
Plaza Accord:
With their central banks working in concert, enough dollars were
purchased on the open market to weaken the currency, making American
goods more affordable for foreign buyers.
I don't know what happened after the Plaza Accord (I read books like this
to find out!), but clearly it wasn't that. This is utter nonsense. Buying
dollars on the open market would increase the value of the dollar,
not weaken it; this is basic supply and demand that you learn in the first
week of a college economics class. This is the type of error that makes me
question all the other claims in the book that I can't easily check.
Mohsin does offer a more credible explanation of the importance of a
reserve currency late in the book, although it's not clear to me that she
realizes it: The widespread use of the US dollar gives US government
sanctions vast international reach, allowing the US to punish and coerce
its enemies through the threat of denying them access to the international
financial system. Now we're getting somewhere! This is a more believable
argument than a small and possibly imaginary effect on government
borrowing costs. It is clear why a bellicose US government, particularly
one led by advocates of a unitary executive theory that elevates the US
president to a status of near-emperor, want to turn the dollar into a
weapon of international control. It's much less obvious how comfortable
the rest of the world should be with that concentration of power.
This would be a fascinating topic for a journalistic non-fiction book.
Some reporter should dive deep into the mechanics of sanctions and ask
serious questions about the moral, practical, and diplomatic consequences
of this aggressive wielding of US power. One could give it a title like
Paper Soldiers that reflected the use of banks and paper currency
as foot soldiers enforcing imperious dictates on the rest of the world.
Alas, apart from a brief section in which the US scared other countries
away from questioning the dollar, Mohsin does not tug at this thread.
Maybe someone should write that book someday.
As you will have gathered by now, I think this is a bad book and I do not
recommend that you read it. Its worst flaw is one that it shares with far
too much mainstream US print and TV journalism: the utter credulity of the
author. I have the old-fashioned belief that a journalist should be more
than a transcriptionist for powerful people. They should be skeptical,
they should assume public figures may be lying, they should look for
ulterior motives, and they should try to bring the reader closer to some
objective truths about the world, wherever they may lie.
I have no solution for this degradation of journalism. I'm not even sure
that it's a change. There were always reporters eager to transcribe the
voice of power into the newspaper, and if we remember the history of
journalism differently, that may be because we have elevated the rare
exceptions and forgotten the average. But after watching too many
journalists I once respected start parroting every piece of nonsense
someone tells them, from NFTs to UFOs to the existential threat of AI,
I've concluded that the least I can do as a reader is to stop rewarding
reporters who cannot approach powerful subjects with skepticism,
suspicion, and critical research.
I failed in this case, but perhaps I can serve as a warning to others.
Rating: 3 out of 10
I wanted an all-in-one unit which meant a built-in keyboard. I was keen on analogue oscillators,
partly for the sound, but mostly to ensure that most of the controls were immediately accessible.
The Minilogue-XD has two analogue oscillators and an analogue filter. It also has some useful,
pure digital stuff: post-effects (chorus, flanger, echo, etc.); and a third, digital oscillator.
The digital oscillator is programmable. There's an SDK,
shared between the Minilogue-XD and some other Korg synths (at least the Prologue and NTS-1).
There's a cottage industry of independent musicians writing and selling digital patches, e.g.
STRING User Oscillator.
Here's an example of a drone programmed using the SDK for the NTS-1:
Eventually I expect to have fun exploring the SDK, but for now I'm keeping it firmly away from
computers (hence the Zoom R8 multitrack recorder in the above image: more on that in a future
blog post). The Korg has been gathering dust whilst I was writing up, but now
I hope to find some time to play.
The first install of a Gentoo kernel needs to be somewhat manual if you want to optimize the kernel for the (virtual) system it boots on.
In part 1 I laid out how to improve the subsequent emerges of sys-kernel/gentoo-sources with a small drop in script to build the kernel as part of the ebuild.
Since end of last year Gentoo also supports a less manual way of emerging a kernel:
The following kernel blends are available:
sys-kernel/gentoo-kernel (the Gentoo kernel you can configure and compile locally - typically this is what you want if you run Gentoo)
sys-kernel/gentoo-kernel-bin (a pre-compiled Gentoo kernel similar to what genkernel would get you)
sys-kernel/vanilla-kernel (the upstream Linux kernel, again configurable and locally compiled)
So a quick walk-through for the gentoo-kernel variant:
1. Set up the correct package USE flags
We do not want an initrd and we want our own config to be re-used so:
4. Update grub and reboot
Unfortunately this ebuild does not update grub, so we have to run grub-mkconfig manually. This can again be automated via a post_pkg_postinst() script. See the step 7 below.
But for now, let's do it manually:
grub-mkconfig -o/boot/grub/grub.cfg # All fine? Time to reboot the machine: reboot
5. (Optional) Prepare for the next kernel build
Run etc-update and merge the new kernel config entries into your savedconfig.
The kernel should auto-build once new versions become available via portage.
Again the etc-update can be automated if you feel that is sufficiently safe to do in your environment. See step 7 below for details.
6. (Optional) Remove the old kernel sources
If you want to switch from the method based on gentoo-sources to the gentoo-kernel one, you can remove the kernel sources:
emerge -C"=sys-kernel/gentoo-sources-5*"
Be sure to update the /usr/src/linux symlink to the new kernel sources directory from gentoo-kernel, e.g.:
This may be a good time for a bit more house-keeping: Clean up a bit in /usr/src/ to remove old build artefacts, /boot/ to remove old kernels and /lib/modules/ to get rid of old kernel modules.
7. (Optional) Further automate the ebuild
In part 1 we automated the kernel compile, install and a bit more via a helper function for post_pkg_postinst().
We can do the similarly for what is (currently) missing from the gentoo-kernel ebuilds:
Create /etc/portage/env/sys-kernel/gentoo-kernel with the following:
The upside of gentoo-kernel over gentoo-sources is that you can put "config override files" in /etc/kernel/config.d/. That way you theoretically profit from config improvements made by the upstream developers. See the Gentoo distribution kernel documentation for a sample snippet. I am fine with savedconfig for now but it is nice that Gentoo provides the flexibility to support both approaches.
Team Cancel: 3019 signers from 1415 individual commit authors
Team Support: 6853 signers from 5418 individual commit authors
Git shortlog (Top 10):
rms_cancel.git (Last update: 2021-08-16 00:11:15 (UTC))
1230 Neil McGovern
251 Joan Touzet
99 Elana Hashman
73 Molly de Blanc
36 Shauna
19 Juke
18 Stefano Zacchiroli
17 Alexey Mirages
16 Devin Halladay
14 Nader Jafari
rms_support.git (Last update: 2021-09-29 07:14:39 (UTC))
1821 shenlebantongying
1585 nukeop
1560 Ivanq
1057 Victor
880 Job Bautista
123 nekonee
101 Victor Gridnevsky
41 Patrick Spek
25 Borys Kabakov
17 KIM Taeyeob
(data as of 2021-10-01)
Technical info:
Signers are counted from their "Signed / Individuals" sections. Commits are counted with git shortlog -s.
Team Cancel also has organizational signatures with Mozilla, Suse and X.Org being among the notable signatories.
The 16 original signers of the Cancel petition are added in their count.
Neil McGovern, Juke and shenlebantongying need .mailmap support as they have committed with different names.
Further reading:
18.04.2021 Debian General Resolution
The Debian General Resolution (GR) vote of the developers has concluded to not issue a public statement at all, see
https://www.debian.org/vote/2021/vote_002#outcome for the results.
It is better to keep quiet and seem ignorant than to speak up and remove all doubt.
See Quote Investigator for the many people that rephrased these words over the centuries.
They still need to be recalled more often as too many people in the FLOSS community have forgotten about that wisdom...
01.10.2021 Final stats
It seems enough dust has settled on this unfortunate episode of mob activity now. Hence I stopped the cronjob that updated the stats above regularly. Team Support has kept adding signature all the time while Team Cancel gave up very soon after the FSF decided to stand with Mr. Stallman. So this battle was decided within two months. The stamina of the accused and determined support from some dissenting web devs trumped the orchestrated outrage of well known community figures and their publicity power this time. But history teaches us that does not mean the war is over. There will a the next opportunity to call for arms. And people will call. Unfortunately.
01.11.2024 Team Cancel is opening a new round; Team Support responds with exposing the author of "The Stallman report"
I hate to be right. Three years later than the above:
An anonymous member of team Cancel has published https://stallman-report.org/[local pdf mirror, 504kB] to "justify our unqualified condemnation of Richard Stallman". It contains a detailed collection of quotes that are used to allege supporting (sexual) misconduct. The demand is again that Mr. Stallman "step[s] down from all positions at the FSF and the GNU project". Addressing him: "the scope and extent of your misconduct disqualifies you from formal positions of power within our community indefinitely".
Team Support has not issues a rebuttal (yet?) but has instead identified the anonymous author as Drew "sircmpwn" DeVault, a gifted software developer, but also a vocal and controversial figure in the Open Source / Free Software space. Ironically quite similar to Richard "rms" Stallman. Their piece is published at https://dmpwn.info/[local pdf mirror, 929kB]. They also allege a proximity of Mr. DeVault to questionable "Lolita" anime preferences and societal positions to disqualify him.
I've long said that the main tools in the Open Source security space, OpenSSL and GnuPG (gpg), are broken and only a complete re-write will solve this. And that is still pending as nobody came forward with the funding. It's not a sexy topic, so it has to get really bad before it'll get better.
Gpg has a UI that is close to useless.
That won't substantially change with more bolted-on improvements.
Now Robert J. Hansen and Daniel Kahn Gillmor had somebody add ~50k signatures (read 1, 2, 3, 4 for the g l ory details) to their keys and - oops - they say that breaks gpg.
But does it?
I downloaded Robert J. Hansen's key off the SKS-Keyserver network.
It's a nice 45MB file when de-ascii-armored (gpg --dearmor broken_key.asc ; mv broken_key.asc.gpg broken_key.gpg).
Now a friendly:
$ /usr/bin/time-v gpg --no-default-keyring--keyring ./broken_key.gpg --batch--quiet--edit-key 0x1DCBDC01B44427C7 clean save quit
pub rsa3072/0x1DCBDC01B44427C7 erzeugt: 2015-07-16 verf llt: niemals Nutzung: SC Vertrauen: unbekannt G ltigkeit: unbekannt sub ed25519/0xA83CAE94D3DC3873 erzeugt: 2017-04-05 verf llt: niemals Nutzung: S sub cv25519/0xAA24CC81B8AED08B erzeugt: 2017-04-05 verf llt: niemals Nutzung: E sub rsa3072/0xDC0F82625FA6AADE erzeugt: 2015-07-16 verf llt: niemals Nutzung: E [ unbekannt ](1). Robert J. Hansen <rjh@sixdemonbag.org> [ unbekannt ](2) Robert J. Hansen <rob@enigmail.net> [ unbekannt ](3) Robert J. Hansen <rob@hansen.engineering>
User-ID "Robert J. Hansen <rjh@sixdemonbag.org>": 49705 Signaturen entfernt User-ID "Robert J. Hansen <rob@enigmail.net>": 49704 Signaturen entfernt User-ID "Robert J. Hansen <rob@hansen.engineering>": 49701 Signaturen entfernt
pub rsa3072/0x1DCBDC01B44427C7 erzeugt: 2015-07-16 verf llt: niemals Nutzung: SC Vertrauen: unbekannt G ltigkeit: unbekannt sub ed25519/0xA83CAE94D3DC3873 erzeugt: 2017-04-05 verf llt: niemals Nutzung: S sub cv25519/0xAA24CC81B8AED08B erzeugt: 2017-04-05 verf llt: niemals Nutzung: E sub rsa3072/0xDC0F82625FA6AADE erzeugt: 2015-07-16 verf llt: niemals Nutzung: E [ unbekannt ](1). Robert J. Hansen <rjh@sixdemonbag.org> [ unbekannt ](2) Robert J. Hansen <rob@enigmail.net> [ unbekannt ](3) Robert J. Hansen <rob@hansen.engineering>
Command being timed: "gpg --no-default-keyring --keyring ./broken_key.gpg --batch --quiet --edit-key 0x1DCBDC01B44427C7 clean save quit" User time(seconds): 3911.14 System time(seconds): 2442.87 Percent of CPU this job got: 99% Elapsed (wall clock)time(h:mm:ss or m:ss): 1:45:56 Average shared text size(kbytes): 0 Average unshared data size(kbytes): 0 Average stack size(kbytes): 0 Average total size(kbytes): 0 Maximum resident setsize(kbytes): 107660 Average resident setsize(kbytes): 0 Major (requiring I/O) page faults: 1 Minor (reclaiming a frame) page faults: 26630 Voluntary context switches: 43 Involuntary context switches: 59439 Swaps: 0 File system inputs: 112 File system outputs: 48 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size(bytes): 4096 Exit status: 0
And the result is a nicely useable 3835 byte file of the clean public key.
If you supply a keyring instead of --no-default-keyring it will also keep the non-self signatures that are useful for you (as you apparently know the signing party).
So it does not break gpg. It does break things that call gpg at runtime and not asynchronously. I heard Enigmail is affected, quelle surprise.
Now the main problem here is the runtime. 1h45min is just ridiculous. As Filippo Valsorda puts it:
Someone added a few thousand entries to a list that lets anyone append to it.
GnuPG, software supposed to defeat state actors, suddenly takes minutes to process entries.
How big is that list you ask? 17 MiB. Not GiB, 17 MiB. Like a large picture.
https://dev.gnupg.org/T4592
If I were a gpg / SKS keyserver developer, I'd
speed this up so the edit-key run above completes in less than 10 s (just getting rid of the lseek/read dance and deferring all time-based decisions should get close)
(ideally) make the drop-sig import-filter syntax useful (date-ranges, non-reciprocal signatures, ...)
clean affected keys on the SKS keyservers (needs coordination of sysops, drop servers from unreachable people)
only accept new keys and new signatures on keys extending the strong set (rather small change to the existing codebase)
That way another key can only be added to the keyserver network if it contains at least one signature from a previously known strong-set key.
Attacking the keyserver network would become at least non-trivial. And the web-of-trust thing may make sense again.
Updates
09.07.2019
GnuPG 2.2.17 has been released with another set of quickly bolted together fixes:
gpg: Ignore all key-signatures received from keyservers. This
change is required to mitigate a DoS due to keys flooded with
faked key-signatures. The old behaviour can be achieved by adding
keyserver-options no-self-sigs-only,no-import-clean
to your gpg.conf. [#4607]
gpg: If an imported keyblocks is too large to be stored in the
keybox (pubring.kbx) do not error out but fallback to an import
using the options "self-sigs-only,import-clean". [#4591]
gpg: New command --locate-external-key which can be used to
refresh keys from the Web Key Directory or via other methods
configured with --auto-key-locate.
gpg: New import option "self-sigs-only".
gpg: In --auto-key-retrieve prefer WKD over keyservers. [#4595]
dirmngr: Support the "openpgpkey" subdomain feature from
draft-koch-openpgp-webkey-service-07. [#4590].
dirmngr: Add an exception for the "openpgpkey" subdomain to the
CSRF protection. [#4603]
dirmngr: Fix endless loop due to http errors 503 and 504. [#4600]
dirmngr: Fix TLS bug during redirection of HKP requests. [#4566]
gpgconf: Fix a race condition when killing components. [#4577]
Bug T4607 shows that these changes are all but well thought-out.
They introduce artificial limits, like 64kB for WKD-distributed keys or 5MB for local signature imports (Bug T4591) which weaken the web-of-trust further.
I recommend to not run gpg 2.2.17 in production environments without extensive testing as these limits and the unverified network traffic may bite you. Do validate your upgrade with valid and broken keys that have segments (packet groups) surpassing the above mentioned limits. You may be surprised what gpg does. On the upside: you can now refresh keys (sans signatures) via WKD. So if your buddies still believe in limiting their subkey validities, you can more easily update them bypassing the SKS keyserver network. NB: I have not tested that functionality. So test before deploying.
10.08.2019
Christopher Wellons (skeeto) has released his pgp-poisoner tool. It is a go program that can add thousands of malicious signatures to a GNUpg key per second. He comments "[pgp-poisoner is] proof that such attacks are very easy to pull off. It doesn't take a nation-state actor to break the PGP ecosystem, just one person and couple evenings studying RFC 4880. This system is not robust." He also hints at the next likely attack vector, public subkeys can be bound to a primary key of choice.
After thinking about multi-stage Debian rebuilds I wanted to implement the idea. Recall my illustration:
Earlier I rebuilt all packages that make up the difference between Ubuntu and Trisquel. It turned out to be a 42% bit-by-bit identical similarity. To check the generality of my approach, I rebuilt the difference between Debian and Devuan too. That was the debdistreproduce project. It only had to orchestrate building up to around 500 packages for each distribution and per architecture.
Differential reproducible rebuilds doesn t give you the full picture: it ignore the shared package between the distribution, which make up over 90% of the packages. So I felt a desire to do full archive rebuilds. The motivation is that in order to trust Trisquel binary packages, I need to trust Ubuntu binary packages (because that make up 90% of the Trisquel packages), and many of those Ubuntu binaries are derived from Debian source packages. How to approach all of this? Last year I created the debdistrebuild project, and did top-50 popcon package rebuilds of Debian bullseye, bookworm, trixie, and Ubuntu noble and jammy, on a mix of amd64 and arm64. The amount of reproducibility was lower. Primarily the differences were caused by using different build inputs.
Last year I spent (too much) time creating a mirror of snapshot.debian.org, to be able to have older packages available for use as build inputs. I have two copies hosted at different datacentres for reliability and archival safety. At the time, snapshot.d.o had serious rate-limiting making it pretty unusable for massive rebuild usage or even basic downloads. Watching the multi-month download complete last year had a meditating effect. The completion of my snapshot download co-incided with me realizing something about the nature of rebuilding packages. Let me below give a recap of the idempotent rebuilds idea, because it motivate my work to build all of Debian from a GitLab pipeline.
One purpose for my effort is to be able to trust the binaries that I use on my laptop. I believe that without building binaries from source code, there is no practically feasible way to trust binaries. To trust any binary you receive, you can de-assemble the bits and audit the assembler instructions for the CPU you will execute it on. Doing that on a OS-wide level this is unpractical. A more practical approach is to audit the source code, and then confirm that the binary is 100% bit-by-bit identical to one that you can build yourself (from the same source) on your own trusted toolchain. This is similar to a reproducible build.
My initial goal with debdistrebuild was to get to 100% bit-by-bit identical rebuilds, and then I would have trustworthy binaries. Or so I thought. This also appears to be the goal of reproduce.debian.net. They want to reproduce the official Debian binaries. That is a worthy and important goal. They achieve this by building packages using the build inputs that were used to build the binaries. The build inputs are earlier versions of Debian packages (not necessarily from any public Debian release), archived at snapshot.debian.org.
I realized that these rebuilds would be not be sufficient for me: it doesn t solve the problem of how to trust the toolchain. Let s assume the reproduce.debian.net effort succeeds and is able to 100% bit-by-bit identically reproduce the official Debian binaries. Which appears to be within reach. To have trusted binaries we would only have to audit the source code for the latest version of the packages AND audit the tool chain used. There is no escaping from auditing all the source code that s what I think we all would prefer to focus on, to be able to improve upstream source code.
The trouble is about auditing the tool chain. With the Reproduce.debian.net approach, that is a recursive problem back to really ancient Debian packages, some of them which may no longer build or work, or even be legally distributable. Auditing all those old packages is a LARGER effort than auditing all current packages! Doing auditing of old packages is of less use to making contributions: those releases are old, and chances are any improvements have already been implemented and released. Or that improvements are no longer applicable because the projects evolved since the earlier version.
See where this is going now? I reached the conclusion that reproducing official binaries using the same build inputs is not what I m interested in. I want to be able to build the binaries that I use from source using a toolchain that I can also build from source. And preferably that all of this is using latest version of all packages, so that I can contribute and send patches for them, to improve matters.
The toolchain that Reproduce.Debian.Net is using is not trustworthy unless all those ancient packages are audited or rebuilt bit-by-bit identically, and I don t see any practical way forward to achieve that goal. Nor have I seen anyone working on that problem. It is possible to do, though, but I think there are simpler ways to achieve the same goal.
My approach to reach trusted binaries on my laptop appears to be a three-step effort:
Encourage an idempotently rebuildable Debian archive, i.e., a Debian archive that can be 100% bit-by-bit identically rebuilt using Debian itself.
Construct a smaller number of binary *.deb packages based on Guix binaries that when used as build inputs (potentially iteratively) leads to 100% bit-by-bit identical packages as in step 1.
How to go about achieving this? Today s Debian build architecture is something that lack transparency and end-user control. The build environment and signing keys are managed by, or influenced by, unidentified people following undocumented (or at least not public) security procedures, under unknown legal jurisdictions. I always wondered why none of the Debian-derivates have adopted a modern GitDevOps-style approach as a method to improve binary build transparency, maybe I missed some project?
If you want to contribute to some GitHub or GitLab project, you click the Fork button and get a CI/CD pipeline running which rebuild artifacts for the project. This makes it easy for people to contribute, and you get good QA control because the entire chain up until its artifact release are produced and tested. At least in theory. Many projects are behind on this, but it seems like this is a useful goal for all projects. This is also liberating: all users are able to reproduce artifacts. There is no longer any magic involved in preparing release artifacts. As we ve seen with many software supply-chain security incidents for the past years, where the magic is involved is a good place to introduce malicious code.
To allow me to continue with my experiment, I thought the simplest way forward was to setup a GitDevOps-centric and user-controllable way to build the entire Debian archive. Let me introduce the debdistbuild project.
Debdistbuild is a re-usable GitLab CI/CD pipeline, similar to the Salsa CI pipeline. It provide one build job definition and one deploy job definition. The pipeline can run on GitLab.org Shared Runners or you can set up your own runners, like my GitLab riscv64 runner setup. I have concerns about relying on GitLab (both as software and as a service), but my ideas are easy to transfer to some other GitDevSecOps setup such as Codeberg.org. Self-hosting GitLab, including self-hosted runners, is common today, and Debian rely increasingly on Salsa for this. All of the build infrastructure could be hosted on Salsa eventually.
The build job is simple. From within an official Debian container image build packages using dpkg-buildpackage essentially by invoking the following commands.
sed -i 's/ deb$/ deb deb-src/' /etc/apt/sources.list.d/*.sources
apt-get -o Acquire::Check-Valid-Until=false update
apt-get dist-upgrade -q -y
apt-get install -q -y --no-install-recommends build-essential fakeroot
env DEBIAN_FRONTEND=noninteractive \
apt-get build-dep -y --only-source $PACKAGE=$VERSION
useradd -m build
DDB_BUILDDIR=/build/reproducible-path
chgrp build $DDB_BUILDDIR
chmod g+w $DDB_BUILDDIR
su build -c "apt-get source --only-source $PACKAGE=$VERSION" > ../$PACKAGE_$VERSION.build
cd $DDB_BUILDDIR
su build -c "dpkg-buildpackage"
cd ..
mkdir out
mv -v $(find $DDB_BUILDDIR -maxdepth 1 -type f) out/
The deploy job is also simple. It commit artifacts to a Git project using Git-LFS to handle large objects, essentially something like this:
if ! grep -q '^pool/**' .gitattributes; then
git lfs track 'pool/**'
git add .gitattributes
git commit -m"Track pool/* with Git-LFS." .gitattributes
fi
POOLDIR=$(if test "$(echo "$PACKAGE" cut -c1-3)" = "lib"; then C=4; else C=1; fi; echo "$DDB_PACKAGE" cut -c1-$C)
mkdir -pv pool/main/$POOLDIR/
rm -rfv pool/main/$POOLDIR/$PACKAGE
mv -v out pool/main/$POOLDIR/$PACKAGE
git add pool
git commit -m"Add $PACKAGE." -m "$CI_JOB_URL" -m "$VERSION" -a
if test "$ DDB_GIT_TOKEN:- " = ""; then
echo "SKIP: Skipping git push due to missing DDB_GIT_TOKEN (see README)."
else
git push -o ci.skip
fi
That s it! The actual implementation is a bit longer, but the major difference is for log and error handling.
You may review the source code of the base Debdistbuild pipeline definition, the base Debdistbuild script and the rc.d/-style scripts implementing the build.d/ process and the deploy.d/ commands.
There was one complication related to artifact size. GitLab.org job artifacts are limited to 1GB. Several packages in Debian produce artifacts larger than this. What to do? GitLab supports up to 5GB for files stored in its package registry, but this limit is too close for my comfort, having seen some multi-GB artifacts already. I made the build job optionally upload artifacts to a S3 bucket using SHA256 hashed file hierarchy. I m using Hetzner Object Storage but there are many S3 providers around, including self-hosting options. This hierarchy is compatible with the Git-LFS .git/lfs/object/ hierarchy, and it is easy to setup a separate Git-LFS object URL to allow Git-LFS object downloads from the S3 bucket. In this mode, only Git-LFS stubs are pushed to the git repository. It should have no trouble handling the large number of files, since I have earlier experience with Apt mirrors in Git-LFS.
To speed up job execution, and to guarantee a stable build environment, instead of installing build-essential packages on every build job execution, I prepare some build container images. The project responsible for this is tentatively called stage-N-containers. Right now it create containers suitable for rolling builds of trixie on amd64, arm64, and riscv64, and a container intended for as use the stage-0 based on the 20250407 docker images of bookworm on amd64 and arm64 using the snapshot.d.o 20250407 archive. Or actually, I m using snapshot-cloudflare.d.o because of download speed and reliability. I would have prefered to use my own snapshot mirror with Hetzner bandwidth, alas the Debian snapshot team have concerns about me publishing the list of (SHA1 hash) filenames publicly and I haven t been bothered to set up non-public access.
Debdistbuild has built around 2.500 packages for bookworm on amd64 and bookworm on arm64. To confirm the generality of my approach, it also build trixie on amd64, trixie on arm64 and trixie on riscv64. The riscv64 builds are all on my own hosted runners. For amd64 and arm64 my own runners are only used for large packages where the GitLab.com shared runners run into the 3 hour time limit.
What s next in this venture? Some ideas include:
Optimize the stage-N build process by identifying the transitive closure of build dependencies from some initial set of packages.
Create a build orchestrator that launches pipelines based on the previous list of packages, as necessary to fill the archive with necessary packages. Currently I m using a basic /bin/sh for loop around curl to trigger GitLab CI/CD pipelines with names derived from https://popcon.debian.org/.
Create and publish a dists/ sub-directory, so that it is possible to use the newly built packages in the stage-1 build phase.
Produce diffoscope-style differences of built packages, both stage0 against official binaries and between stage0 and stage1.
Create the stage-1 build containers and stage-1 archive.
Review build failures. On amd64 and arm64 the list is small (below 10 out of ~5000 builds), but on riscv64 there is some icache-related problem that affects Java JVM that triggers build failures.
Provide GitLab pipeline based builds of the Debian docker container images, cloud-images, debian-live CD and debian-installer ISO s.
Provide integration with Sigstore and Sigsum for signing of Debian binaries with transparency-safe properties.
Implement a simple replacement for dpkg and apt using /bin/sh for use during bootstrapping when neither packaging tools are available.
November 2024 will be known as the time when killing peanut, a pet squirrel, by the New York State DEC swung the US presidential elections and shaped history forever.
The hundreds of millions of dollars spent on each side, the tireless campaigning by the candidates, the celebrity endorsements ... all made for an open race for months. Investments evened each other out.
But an OnlyFans producer showing people an overreaching, bureaucracy driven State raiding his home to confiscate a pet squirrel and kill it ... swung enough voters to decide the elections.
That is what we need to understand in times of instant worldwide publication and a mostly attention driven economy: Human fates, elections, economic cycles and wars can be decided by people killing squirrels.
RIP, peanut.
P.S.: Trump Media & Technology Group Corp. (DJT) stock is up 30% pre-market.
*[DEC]: Department of Environmental Conservation
For a long time I ve been wanting to try GitOps tools, but I haven t had the chance to try them for real on the projects
I was working on.
As now I have some spare time I ve decided I m going to play a little with Argo CD,
Flux and Kluctl to test them and be able to use one of them in a real project
in the future if it looks appropriate.
On this post I will use Argo-CD Autopilot to install argocd on a
k3d local cluster installed using OpenTofu to test the autopilot approach of
managing argocd and test the tool (as it manages argocd using a git repository it can be used to test argocd as
well).
Installing tools locally with arkadeRecently I ve been using the arkade tool to install kubernetes related
applications on Linux servers and containers, I usually get the applications with it and install them on the
/usr/local/bin folder.
For this post I ve created a simple script that checks if the tools I ll be using are available and installs them on the
$HOME/.arkade/bin folder if missing (I m assuming that docker is already available, as it is not installable with
arkade):
#!/bin/sh# TOOLS LISTARKADE_APPS="argocd argocd-autopilot k3d kubectl sops tofu"# Add the arkade binary directory to the path if missingcase":$ PATH:"in*:"$ HOME/.arkade/bin":*);;*)export PATH="$ PATH:$ HOME/.arkade/bin";;esac# Install or update arkadeif command-v arkade >/dev/null;then
echo"Trying to update the arkade application"sudo arkade update
else
echo"Installing the arkade application"
curl -sLS https://get.arkade.dev sudo sh
fi
echo""echo"Installing tools with arkade"echo""for app in$ARKADE_APPS;do
app_path="$(command-v$app)"true
if["$app_path"];then
echo"The application '$app' already available on '$app_path'"else
arkade get "$app"fi
done
cat<<EOF
Add the ~/.arkade/bin directory to your PATH if tools have been installed there
EOF
The rest of scripts will add the binary directory to the PATH if missing to make sure things work if something was
installed there.
Creating a k3d cluster with opentofuAlthough using k3d directly will be a good choice for the creation of the cluster, I m using tofu to do it because
that will probably be the tool used to do it if we were working with Cloud Platforms like AWS or Google.
The main.tf file is as follows:
The k3d configuration is quite simple, as I plan to use the default traefik ingress controller with TLS I publish
the 443 port on the hosts 8443 port, I ll explain how I add a valid certificate on the next step.
I ve prepared the following script to initialize and apply the changes:
#!/bin/shset-e# VARIABLES# Default token for the argocd clusterK3D_CLUSTER_TOKEN="argocdToken"# Relative PATH to install the k3d cluster using terr-iaformK3D_TF_RELPATH="k3d-tf"# Secrets yaml fileSECRETS_YAML="secrets.yaml"# Relative PATH to the workdir from the script directoryWORK_DIR_RELPATH=".."# Compute WORKDIRSCRIPT="$(readlink-f"$0")"SCRIPT_DIR="$(dirname"$SCRIPT")"WORK_DIR="$(readlink-f"$SCRIPT_DIR/$WORK_DIR_RELPATH")"# Update the PATH to add the arkade bin directory# Add the arkade binary directory to the path if missingcase":$ PATH:"in*:"$ HOME/.arkade/bin":*);;*)export PATH="$ PATH:$ HOME/.arkade/bin";;esac# Go to the k3d-tf dircd"$WORK_DIR/$K3D_TF_RELPATH"exit 1
# Create secrets.yaml file and encode it with sops if missingif[!-f"$SECRETS_YAML"];then
echo"token: $K3D_CLUSTER_TOKEN">"$SECRETS_YAML"
sops encrypt -i"$SECRETS_YAML"fi# Initialize terraform
tofu init
# Apply the configuration
tofu apply
Adding a wildcard certificate to the k3d ingressAs an optional step, after creating the k3d cluster I m going to add a default wildcard certificate for the traefik
ingress server to be able to use everything with HTTPS without certificate issues.
As I manage my own DNS domain I ve created the lo.mixinet.net and *.lo.mixinet.net DNS entries on my public and
private DNS servers (both return 127.0.0.1 and ::1) and I ve created a TLS certificate for both entries using
Let s Encrypt with Certbot.
The certificate is updated automatically on one of my servers and when I need it I copy the contents of the
fullchain.pem and privkey.pem files from the /etc/letsencrypt/live/lo.mixinet.net server directory to the local
files lo.mixinet.net.crt and lo.mixinet.net.key.
After copying the files I run the following file to install or update the certificate and configure it as the default
for traefik:
#!/bin/sh# Script to update thesecret="lo-mixinet-net-ingress-cert"cert="$ 1:-lo.mixinet.net.crt"key="$ 2:-lo.mixinet.net.key"if[-f"$cert"]&&[-f"$key"];then
kubectl -n kube-system create secret tls $secret\--key=$key\--cert=$cert\--dry-run=client --save-config-o yaml kubectl apply -f -
kubectl apply -f - <<EOF
apiVersion: traefik.containo.us/v1alpha1
kind: TLSStore
metadata:
name: default
namespace: kube-system
spec:
defaultCertificate:
secretName: $secretEOF
else
cat<<EOF
To add or update the traefik TLS certificate the following files are needed:
- cert: '$cert'
- key: '$key'
Note: you can pass the paths as arguments to this script.
EOF
fi
Creating a repository and a token for autopilotI ll be using a project on my forgejo instance to manage argocd, the repository I ve created is on the URL
https://forgejo.mixinet.net/blogops/argocd and I ve created a private user named argocd that only has write access to
that repository.
Logging as the argocd user on forgejo I ve created a token with permission to read and write repositories that I ve
saved on my pass password store on the mixinet.net/argocd@forgejo/repository-write
entry.
Bootstrapping the installationTo bootstrap the installation I ve used the following script (it uses the previous GIT_REPO and GIT_TOKEN values):
#!/bin/shset-e# VARIABLES# Relative PATH to the workdir from the script directoryWORK_DIR_RELPATH=".."# Compute WORKDIRSCRIPT="$(readlink-f"$0")"SCRIPT_DIR="$(dirname"$SCRIPT")"WORK_DIR="$(readlink-f"$SCRIPT_DIR/$WORK_DIR_RELPATH")"# Update the PATH to add the arkade bin directory# Add the arkade binary directory to the path if missingcase":$ PATH:"in*:"$ HOME/.arkade/bin":*);;*)export PATH="$ PATH:$ HOME/.arkade/bin";;esac# Go to the working directorycd"$WORK_DIR"exit 1
# Set GIT variablesif[-z"$GIT_REPO"];then
export GIT_REPO="https://forgejo.mixinet.net/blogops/argocd.git"fi
if[-z"$GIT_TOKEN"];then
GIT_TOKEN="$(pass mixinet.net/argocd@forgejo/repository-write)"export GIT_TOKEN
fi
argocd-autopilot repo bootstrap --provider gitea
The output of the execution is as follows:
bin/argocd-bootstrap.sh
INFO cloning repo: https://forgejo.mixinet.net/blogops/argocd.git
INFO empty repository, initializing a new one with specified remote
INFO using revision: "", installation path: ""
INFO using context: "k3d-argocd", namespace: "argocd"
INFO applying bootstrap manifests to cluster...
namespace/argocd created
customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io created
serviceaccount/argocd-application-controller created
serviceaccount/argocd-applicationset-controller created
serviceaccount/argocd-dex-server created
serviceaccount/argocd-notifications-controller created
serviceaccount/argocd-redis created
serviceaccount/argocd-repo-server created
serviceaccount/argocd-server created
role.rbac.authorization.k8s.io/argocd-application-controller created
role.rbac.authorization.k8s.io/argocd-applicationset-controller created
role.rbac.authorization.k8s.io/argocd-dex-server created
role.rbac.authorization.k8s.io/argocd-notifications-controller created
role.rbac.authorization.k8s.io/argocd-redis created
role.rbac.authorization.k8s.io/argocd-server created
clusterrole.rbac.authorization.k8s.io/argocd-application-controller created
clusterrole.rbac.authorization.k8s.io/argocd-applicationset-controller created
clusterrole.rbac.authorization.k8s.io/argocd-server created
rolebinding.rbac.authorization.k8s.io/argocd-application-controller created
rolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created
rolebinding.rbac.authorization.k8s.io/argocd-dex-server created
rolebinding.rbac.authorization.k8s.io/argocd-notifications-controller created
rolebinding.rbac.authorization.k8s.io/argocd-redis created
rolebinding.rbac.authorization.k8s.io/argocd-server created
clusterrolebinding.rbac.authorization.k8s.io/argocd-application-controller created
clusterrolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created
clusterrolebinding.rbac.authorization.k8s.io/argocd-server created
configmap/argocd-cm created
configmap/argocd-cmd-params-cm created
configmap/argocd-gpg-keys-cm created
configmap/argocd-notifications-cm created
configmap/argocd-rbac-cm created
configmap/argocd-ssh-known-hosts-cm created
configmap/argocd-tls-certs-cm created
secret/argocd-notifications-secret created
secret/argocd-secret created
service/argocd-applicationset-controller created
service/argocd-dex-server created
service/argocd-metrics created
service/argocd-notifications-controller-metrics created
service/argocd-redis created
service/argocd-repo-server created
service/argocd-server created
service/argocd-server-metrics created
deployment.apps/argocd-applicationset-controller created
deployment.apps/argocd-dex-server created
deployment.apps/argocd-notifications-controller created
deployment.apps/argocd-redis created
deployment.apps/argocd-repo-server created
deployment.apps/argocd-server created
statefulset.apps/argocd-application-controller created
networkpolicy.networking.k8s.io/argocd-application-controller-network-policy created
networkpolicy.networking.k8s.io/argocd-applicationset-controller-network-policy created
networkpolicy.networking.k8s.io/argocd-dex-server-network-policy created
networkpolicy.networking.k8s.io/argocd-notifications-controller-network-policy created
networkpolicy.networking.k8s.io/argocd-redis-network-policy created
networkpolicy.networking.k8s.io/argocd-repo-server-network-policy created
networkpolicy.networking.k8s.io/argocd-server-network-policy created
secret/autopilot-secret created
INFO pushing bootstrap manifests to repo
INFO applying argo-cd bootstrap application
INFO pushing bootstrap manifests to repo
INFO applying argo-cd bootstrap application
application.argoproj.io/autopilot-bootstrap created
INFO running argocd login to initialize argocd config
Context 'autopilot' updated
INFO argocd initialized. password: XXXXXXX-XXXXXXXX
INFO run:
kubectl port-forward -n argocd svc/argocd-server 8080:80
Now we have the argocd installed and running, it can be checked using the port-forward and connecting to
https://localhost:8080/ (the certificate will be wrong, we are going to fix that in the next step).
Updating the argocd installation in gitNow that we have the application deployed we can clone the argocd repository and edit the deployment to disable TLS
for the argocd server (we are going to use TLS termination with traefik and that needs the server running as insecure,
see the Argo CD documentation)
ssh clone ssh://git@forgejo.mixinet.net/blogops/argocd.git
cd argocd
edit bootstrap/argo-cd/kustomization.yaml
git commit -m 'Disable TLS for the argocd-server'
The changes made to the kustomization.yaml file are the following:
As this simply changes the ConfigMap we have to restart the argocd-server to read it again, to do it we delete the
server pods so they are re-created using the updated resource:
After doing this the port-forward command is killed automatically, if we run it again the connection to get to the
argocd-server has to be done using HTTP instead of HTTPS.
Instead of testing that we are going to add an ingress definition to be able to connect to the server using HTTPS and
GRPC against the address argocd.lo.mixinet.net using the wildcard TLS certificate we installed earlier.
To do it we to edit the bootstrap/argo-cd/kustomization.yaml file to add the ingress_route.yaml file to the
deployment:
After pushing the changes and waiting a little bit the change is applied and we can access the server using HTTPS and
GRPC, the first way can be tested from a browser and the GRPC using the command line interface:
argocd --grpc-web login argocd.lo.mixinet.net:8443
Username: admin
Password:
'admin:login' logged in successfully
Context 'argocd.lo.mixinet.net:8443' updated
argocd app list -o name
argocd/argo-cd
argocd/autopilot-bootstrap
argocd/cluster-resources-in-cluster
argocd/root
So things are working fine and that is all on this post, folks!
Orbital at NX, Newcastle in 2023
On 26th and 27th April we held a Debian bug-squashing party near
Leuven, Belgium. Several longstanding and new Debian contributors
gathered to work through some of the highest priority bugs affecting
the upcoming release of Debian 13 trixie .
We were hosted by the 
Conference hall
Prav stall at the conference
Conference group photo
Adalaj Stepwell
After my previous posts related to 
Dear Debian community,
This is bits from the DPL for April.
End of 10
I am sure I was speaking in the interest of the whole project when
joining the 
TGV train I rode from Luxembourg to Paris
Pasta I had in Paris
A cycling track in Paris
Luxembourg Museum
A shot of the park near to the Luxembourg Museum
Pantheon, one of the attractions of Paris.
Photo with Joenio and Mari
Notre Dame, another tourist attraction of Paris.
My photo with Eiffel Tower in the background
Items at a bakery in Paris. Items from left to right are: Chocolate Twist, Sugar briochette, Pain au Chocolat, Croissant with almonds, Croissant, Croissant with chocolate hazlenut filling.
The Paris subway
Inside a Paris subway
A random building and road in Paris
A shot near Seine river
A view of Seine river
The first install of a Gentoo kernel needs to be somewhat manual if you want to optimize the kernel for the (virtual) system it boots on.
In 
The kernel should auto-build once new versions become available via portage.
Again the 
RIP, 