Freexian Collaborators: Discover release 0.3.0 of the debusine software factory (by Colin Watson)
Debusine is a Free Software project developed by Freexian to manage
scheduling and distribution of Debian-related tasks to a network of worker
machines. It was started some time back, but its development pace has
recently increased significantly thanks to funding from the Sovereign Tech
Fund. You can read more about it in its
documentation.
For more background, Enrico Zini and Carles Pina i Estany gave a talk on
Debusine
in November 2023 at the mini-DebConf in Cambridge.
We described the work from our first funded milestone in a post to
debian-devel-announce
in March.
We ve recently finished work on our second funded milestone, culminating in
releasing version 0.3.0 to
unstable.
Our focus on this milestone was on new building blocks to allow us to
automatically orchestrate QA tasks in bulk. Full details are in our
release
history
document. As usual, debusine.debian.net is
up to date with our latest work.
Collections
In the previous milestone, debusine could store artifacts and run tasks
against those artifacts. However, on its own this required the user to do a
lot of manual work, because the only way to refer to an artifact was by its
ID.
We now have the concept of a
collection,
which can store references to other artifacts (or indeed to other
collections) with some attached metadata. These are structured by category,
so for example a
Collections
In the previous milestone, debusine could store artifacts and run tasks
against those artifacts. However, on its own this required the user to do a
lot of manual work, because the only way to refer to an artifact was by its
ID.
We now have the concept of a
collection,
which can store references to other artifacts (or indeed to other
collections) with some attached metadata. These are structured by category,
so for example a debian:suite
collection contains references to source and
binary package artifacts with their names, versions, and architectures as
metadata. This allows us to look up artifacts using a simple query
language
instead of just by ID.
At the moment, the main visible effect of this is that our Getting started
with
debusine
tutorial no longer needs users of debusine.debian.net to create their own
build environments before being able to submit other work requests: they can
refer to existing environments using something like
debian/match:codename=trixie:variant=sbuild
instead.
We also have a basic user interface allowing you to browse existing
collections, accessible via the relevant workspace (such as the default
System workspace).
Workflows
We ve always known that individual tasks were just a starting point:
real-world requirements often involve chaining many tasks together, as many
Debian developers already do using the Salsa CI
pipeline. debusine
intends to approach a similar problem from a different angle, defining
common workflows that can be applied at the scale of a whole distribution
without being tightly coupled to where each package s code is hosted.
In time we intend to define a way for users to specify their own workflows,
but rather than getting too bogged down in this we started by building a
couple of predefined
workflows
into debusine. The update_environments
workflow is used to create
multiple build environments in bulk, while the sbuild
workflow builds a
source package for all the architectures that it supports and for which
debusine has workers. (debusine.debian.net currently has amd64 and arm64
workers, supporting the amd64, arm64, armel, armhf, and i386 architectures
between them.)
Upcoming work will build on this by adding more workflows that chain tasks
together in various ways, such as workflows that build a package and run QA
tasks on the results, or a workflow that builds a package and uploads the
result to an upload queue.
Next steps
Our next planned milestone involves expanding debusine s capability as a
build daemon. For that, we already know that there are a number of specific
extra workflow steps we need to add, and we ve reached out to some members
of Debian s buildd team to ask for feedback on what they consider necessary.
We hope to be able to replace some of Freexian s own build infrastructure
with debusine in the near future.
update_environments
workflow is used to create
multiple build environments in bulk, while the sbuild
workflow builds a
source package for all the architectures that it supports and for which
debusine has workers. (debusine.debian.net currently has amd64 and arm64
workers, supporting the amd64, arm64, armel, armhf, and i386 architectures
between them.)
Upcoming work will build on this by adding more workflows that chain tasks
together in various ways, such as workflows that build a package and run QA
tasks on the results, or a workflow that builds a package and uploads the
result to an upload queue.