Search Results: "Reinhard Tartler"

25 April 2020

Reinhard Tartler: Building Packages with Buildah in Debian

1 Building Debian Packages with buildah
Building packages in Debian seems to be a solved problem. But is it? At the bottom, installing the dpkg-dev package provides all the basic tools needed. Assuming that you already succeeded with creating the necessary packaging metadata (i.e., debian/changelog, debian/control, debian/copyright, etc., and there are great helper tools for this such ash dh-make, dh-make-golang, etc.,) it should be as simple as invoking the dpkg-buildpackage tool. So what's the big deal here?

The issue is that dpkg-buildpackage expects to be called with an appropriately setup build context, that is, it needs to be called in an environment that satisfies all build dependencies on the system. Let's say you are building a package for Debian unstable on your Debian stable system (this is the common scenario for the official Debian build machines), you would need your build to link against libraries in unstable, not stable. So how to tell the package build process where to find its dependencies?

The answer (in Debian and many other Linux distributions) is you do not at all. This is actually a somewhat surprising answer for software developers without a Linux distribution development background1. Instead, chroots "simulate" an environment that has all dependencies that we want to build against at the system locations, that is. /usr/lib, etc.

Chroots are basically full system installations in a subdirectory that includes system and application libraries. In order to use them, a package build needs to use the chroot(2) system call, which is a privileged operation. Also creating these system installations is a somewhat finicky process. In Debian, we have tools that make this process easier, the most popular ones are probably pbuilder(1)2 and sbuild(1)3. Still, they are somewhat clumsy to use, add significant levels of abstraction in the sense that they do quite a bit of magic behind the scenes to work hide the fact that privileged operations (need root to run chroot(2), etc.) are required. They are also somewhat brittle, for instance, if a build process is aborted (SIGKILL, or system crash), you may end up with temporary directories and files under userids other than your own that again may require root-privileges to cleanup.

What if there was an easy way to do all of the above without any process running as root? Enter rootless buildah.

Modern Linux container technologies allow unprivileged users to "untie" a process from the system (cf. the unshare(2) system call). This means that a regular user may run a process (and its child processes) in an "environment" where system calls behave differently and provide a configurable amount of isolation levels. This article demonstrates a novel build tool buildah, which is:
  • easy to setup build environment from the command line
  • secure, as no process needs to run as root
  • simple in architecture, requires no running daemon like for example docker
  • convenient to use: you can debug your debian/rules interactively
Architecturally, buildah is written in golang and compiled as a (mostly) statically linked executable. It builds on top a number of libraries written in golang, including and The overlay functionality is provided by the fuse-overlayfs(1) utility.

1.1 Preparation:
The Kernel in Debian bullseye (and in buster, and recent Ubuntu kernels to the best of my knowledge) do support usernamespaces, but leave them disabled by default. Here is how to enable them:
echo kernel.unprivileged_userns_clone = 1   sudo tee -a /etc/sysctl.d/containers.conf
systemctl -p /etc/sysctl.d/containers.conf
I have to admit that I'm not entirely sure why the Debian kernels don't enable usernamespaces by default. I've found a reference on stackexchange4 that claims this disables some "hardening" features in the Debian kernel. I also understand this step is not necessary if you chose to compile and run a vanilla upstream kernel. I'd appreciate a better reference and am happy to update this text.
$ c=$(buildah from
Getting image source signatures
Copying blob 2bbc6b8c460d done
Copying config 9b90abe801 done
Writing manifest to image destination
Storing signatures
This command downloads the image from and stores it locally in your home directory. Let's install essential utilities for building Debian packages:
1: buildah run $c apt-get update -qq
2: buildah run $c apt-get install dpkg-dev -y
3: buildah config --workingdir /src $c
4: buildah commit $c dpkg-dev
The command on line 1 and 2 execute the installation of compilers and dpkg development tools such as dpkg-buildpackage, etc. The buildah config command in line 4 arranges that whenever you start a shell in the container, the current working directory in the container is in /src. Don't worry about this location not existing yet, we will make sources from your host system available there. The last command creates an OCI image with the name dpkg-dev. BTW, the name you use for the image in the commit command can be used in podman (but not the "containers"). See 5 and 6 for a comparison between podman and buildah.
buildah images -a
This output might look like this:
  REPOSITORY                 TAG      IMAGE          ID   CREATED   SIZE             
localhost/dpkg-dev latest b85c34f95d3e 16 seconds ago 406 MB sid 9b90abe801db 11 hours ago 124 MB

1.2 Running a package build
Now we have a working container with the reference in the variable $c. To use it conveniently with source packages that I have stored in /srv/scratch/packages/containers, let's introduce a shell alias r like this:
alias r='buildah run --tty -v /srv/scratch/packages/containers:/src $c '
This allows you to easily execute commands in that container:
r pwd
r bash
The last command will give you an interactive shell that we'll be using for building packages!
siretart@x1:~/scratch/packages/containers/libpod$ r bash

root@x1:/src# cd golang-github-openshift-imagebuilder

root@x1:/src/golang-github-openshift-imagebuilder# dpkg-checkbuilddeps
dpkg-checkbuilddeps: error: Unmet build dependencies: debhelper (>= 11) dh-golang golang-any golang-github-containers-storage-dev (>= 1.11) golang-github-docker-docker-dev (>= 18.09.3+dfsg1) golang-github-fsouza-go-dockerclient-dev golang-glog-dev

root@x1:/src/golang-github-openshift-imagebuilder# apt-get build-dep -y .
Note, using directory '.' to get the build dependencies
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:

root@x1:/src/golang-github-openshift-imagebuilder# dpkg-checkbuilddeps

Now we have a environment that has all build-dependencies available and we are ready to build the package:
root@x1:/src/golang-github-openshift-imagebuilder# dpkg-buildpackage -us -uc -b
Assuming the package builds, the package build results are placed in /src inside the container, and are visible at ~/scratch/packages/containers on the host. There you can inspect, extract or even install them. The latter part allows you to interactively rebuild packages against updated dependencies, without the need of setting up an apt archive or similar.

2 Availability
The buildah(1) tool is available in debian/bullseye since 2019-12-17. Since is a golang binary, it only links dynamically against system libraries such as libselinux and libseccomp, which are all available in buster-backports. I'd expect buildah to just work on a debian/buster system as well provided you install those system libraries and possibly the backported Linux kernel.

Keeping the package at the latest upstream version is challenging because of its fast development pace that picks up new dependencies on golang libraries and new versions of existing libraries with every new upstream release. In general, this requires updating several source packages, and in many cases also uploading new source packages to the Debian archive that need to be processed by the Debian ftp-masters.

As an exercise, I suggest to install the buildah package from bullseye, 'git clone' the packaging repository from and build the latest version yourself. Note, I would expect the above to even work on a Fedora laptop.

The Debian packages have not swept into the Ubuntu distributions yet, but I expect them to be included in the Ubuntu 20.10 release. In the mean time, ubuntu users can install the package that are provided by the upstream maintainers in the "Project Atomic" PPA at

3 Related Tools
The buildah tool is accompanied with two "sister" tools:

The Skopeo package provides tooling to work with remote images registries. This allows you download, upload images to remote registries, convert container images between different formats. It has been available in Debian since 2020-04-20.

The podman tool is a 'docker' replacement. It provides a command-line interface that mimics the original docker command to an extend that a user familiar with docker might want to place this in their ~/.bashrc file:
alias docker='podman'
Unfortunately, at the time of writing podman is still being processed by the ftp-masters since 01-03-2020. At this point, I recommend building the package from our salsa repository at

4 Conclusion
Building packages in the right build context is a fairly technical issue for which many tools have been written for. They come with different trade-off when it comes to usability. Containers promise a secure alternative to the tried and proven chroot-based approaches, and the buildah makes using this technology very easy to use.

I'd love to get into a conversation with you on how these tools work for you, and would like to encourage participation and assistance with keeping the complicated software stack up-to-date in Debian (and by extension, in derived distribution such as Ubuntu, etc.).

At my day-job, we build millions of lines of C++ code on Solaris10 and AIX6, where concepts such as "chroots" are restricted to the super user 'root' and is therefore not available to developers, not even through wrappers. Instead, libraries and headers are installed into "refroots", that is, subdirectories that mimic the structure of the "sysroot" directories that are used in the embedded Linux community for cross-compiling packages, and we use Makefiles that set include flags (-I rules and -L flags) to tell the compiler and linker where to look.

6 July 2016

Mike Gabriel: [Arctica Project] Release of nx-libs (version

Introduction NX is a software suite which implements very efficient compression of the X11 protocol. This increases performance when using X applications over a network, especially a slow one. NX (v3) has been originally developed by NoMachine and has been Free Software ever since. Since NoMachine obsoleted NX (v3) some time back in 2013/2014, the maintenance has been continued by a versatile group of developers. The work on NX (v3) is being continued under the project name "nx-libs". History Until January 2015, nx-libs was more mainly a redistribution approach of the original NX (v3) software. We (we as in mainly a group of X2Go developers) kept changes applied to the original NoMachine sources as minimal as possible. We also kept the original files and folders structure. Patches had been maintained via the quilt utility on top of a Git repository, the patches had always been kept separate. That was the 3.5.0.x series of nx-libs. In January 2015, the characteristics of nx-libs as maintained by the X2Go project between 2011 and 2015 changed. A decision was reached: This effort is now about to be released as "nx-libs". Contributors Since 2011, the nx-libs code base has to a great extent been maintained in the context of the X2Go Project [1]. Qindel Group joining in... In 2014, developers from the Qindel Group [2] joined the nx-libs maintenance. They found X2Go's work on nx-libs and suggested joining forces as best as possible on hacking nx-libs. The Arctica Project comming up... Starting in January 2015, the development on the 3.6.x branch of the project was moved into a new project called the Arctica Project [3]. Development Funding by Qindel In September 2015, a funding project was set up at Qindel. Since then, the Qindel group greatly supports the development of nx-libs 3.6.x monetarily and with provided man power. The funding project officially is a cooperation between Qindel and DAS-NETZWERKTEAM (business run by Mike Gabriel, long term maintainer of nx-libs). The funding is split into two subprojects and lasts until August 2017: The current nx-libs development effort is coordinated in the context of the Arctica Project (by Mike Gabriel), with use cases in Arctica, X2Go and TheQVD (VDI product worked on at Qindel) in mind. People involved There are various people involved in nx-libs 3.6.x maintenance and development, some of them shall be explicitly named here (in alphabetical order of surnames): Some other people have contributed, but have left the project already. Thanks for your work on nx-libs: A big thanks to everyone involved!!! Special thanks go to Stefan Baur for employing Mihai Moldovan and handling all the bureaucracy, so that Mihai can work on this project and get funded for his work. Achievements of nx-libs We are very close to our self-defined release goal 3.6.0. The below tasks have already been (+/-) completed for version In a previous blog post [4], the code reduction in nx-libs has already been discussed. With this announcement, we want to give a status update about our effort of shrinking the nx-libs code base:
    [mike@minobo nx-libs (3.6.x)]$ cloc --match-f '.*\.(c cpp h)' .
        1896 text files.
        1896 unique files.                                          
        7430 files ignored. v 1.60  T=5.88 s (307.3 files/s, 143310.5 lines/s)
    Language                     files          blank        comment           code
    C                              958          68574          74891         419638
    C/C++ Header                   730          25683          33957         130418
    C++                            120          17007          11721          61292
    SUM:                          1808         111264         120569         611348
The previous statistics had these sums in the last line. First the nx-libs 3.5.0.x code tree (where we came from):
    SUM:                          5614         329279         382337        1757743
Then the nx-libs 3.6.x status as it was on May 9th 2016:
    SUM:                          1922         118581         126783         662635
Another 50.000 lines of code have been removed over the past two months. Work pending for the final 3.6.0 release goal Known Issues There are several open issues on the nx-libs Github project space, see Testing the nx-libs We are currently working on provisioning release builds and nightly builds of nx-libs for various recent Linux distributions. Please stay tuned and watch Mike Gabriel's blog [5]. We already have nightly builds of nx-libs for Debian and Ubuntu [6], but there are more to come soon. Until now, please use the build recipes provided in the file of the nx-libs source tree [7]. References

29 October 2015

B lint R czey: FFmpeg and Kodi arrived to jessie-backports!

FFmpeg Kodi Debian JessieDebian has switched to FFmpeg in testing in July but the work on the package did not stop at that point. After careful testing we can now provide official packages for Jessie users through jessie-backports. See installation instructions here. FFmpeg becoming available in jessie-backports also enabled us to provide Kodi from Debian in the same official repository. Thanks to everyone in the Debian Multimedia Maintainers team, especially Andreas Cadhalpun who is also upstream developer at the FFmpeg project, Reinhard Tartler who maintained FFmpeg then Libav then FFmpeg again in Debian for long years and everyone else I could not name here but helped making this possible!

17 May 2011

Reinhard Tartler: Mini Server PCs

Dear Lazyweb,

Hot or not, which of the following devices is the best? And which of them are actually available for purchase in Europe, preferably from Germany?

Please share your opinions in the comments.

3 August 2010

Rog rio Brito: First post from DebConf 10

Even though I am late with this post, it is nice to say that I am writing here from this year s DebConf10, in NYC. Today (well, yesterday) was the day of the Cheese and Wine party and I think that it was cool, at least for the moments that I were there. This post, though, isn t technical in any sense. I only talks shortly about my impressions of the community, as this is my first DebConf ever (despite the fact that I have been using Debian since the late nineties). I was very pleased to have met Bdale Garbee. I saw him the other day arriving with Keith Packard, but I just didn t want to disturb them at that point. We only talked for, say, 2 minutes, and his was one of the nicest receptions that I had here. And there were some other people that were equally easy to approach, nice to talk with and, to my surprise, knew my name after some brief moments (yes, this does make a difference, especially when you are in a strange country, when you don t know anybody with whom you have worked for some years). Being involved in the organization stuff, one would think that Jimmy Kaplowitz would be so busy, but he was so kind. I had longer conversations with T ssia Cam es, Tiago Vaz (as always) and some other people that I had not yet had the pleasure of meeting in person. In particular, Daniel Baumann (who apparently is now crazy about our FISL and wants to drink all Guaran that he can get :-) ), Chris Lamb and Ot vio Salvador and his mom. Those people are so cool and it is nice to discuss some legal issues among different continents in the lounge of their building at late night. :-) Too bad that I am allocated to the other building. :-) I am forgetting many other people (hey, it is 2 am here in NYC), but I would feel guilty if I had not mentioned at least three people more: Martin Michlmayr, Phil Hands, and Reinhard Tartler (who is uploading lame to the debian repository, as the patents regarding it are expiring or expired already). Thanks! P.S.: I just created an account on flickr that I hope to populate with some photos that I took here. And even with a nice squirrel. :-)

3 July 2008

Reinhard Tartler: ffmpeg-uploaded

Finally ffmpeg version 0.svn20080206-9 was uploaded to unstable and is currently waiting to get out of NEW . Special thanks go out to Fabian Greffrath, for testing, reviewing and pushing me, to L oc Minier, for his thoughts on the package renaming and handling of potential unstripped replacement packages and Darren Salt for his work on the debian/rules file. This upload is targeted lenny. Highlights of that upload include:
  • introduction of ffmpeg-doc 438369
  • build flavors (read more below)
  • renaming of the source package to ffmpeg-debian
  • DEB_BUILD_OPTIONS=noopt now compiles the package with -O0 (and works!)
  • ffmpeg-config has been removed. please port your packages to use pkg-config instead, cf. 487917 and 487922.
As for the build flavors: The debian/rules file was refactored so that flavors can easily be added. The flavors are architecture specific. All architectures support a static and a dynamic flavor. The following architectures support the following flavors:
  • i386: cmov (an i686 optimized flavor using the cmov instruction)
  • powerpc: altivec (requires G4 or better)
  • sparc: vis (uses v9 instructions)
If you are interested how we manage these build flavors, have a look at our debian/rules file. I remember that this was recently discussed on debian-devel at some point. Anyway, if you need/want additional flavors, please file wishlist bugs.

30 July 2007

Reinhard Tartler: revu

Scott Kitterman writes:
Why do REVU and mentors need to be separate?
I’ve been thinking about this as well. Here my thoughts: I never actively used mentors. To me, mentors is mainly an apt-get’able repository, where interested contributors can submit packages. It acts mainly as a hosting service for packages. There are quite some differences to REVU, but I think the most important one is that REVU lets you collect opinions on a contribution in the web application. In mentors.debian,net, this happens on the mailing list. Do we need this disctinction? I don’t think so. See e.g. Bundle Buggy (, which is used on the bazaar development mailing list to collect and review contributions. It has got a new feature which allows Developers to vote on a submission via email commands on the mailing list, to which Bundle Buggy is subscribed. I think that ideally, we could join all three mediums: Mail, Web-app and IRC. REVU could look for comments and commands for a package on a mailing list. On the other hand, reviews and submissions could be announced on an IRC Channel. We could even think about giving comments/commands via irc. It mainly needs one: Someone to do it. I’m sorry to say that REVU isn’t really under development since some time. I’d be more than happy if some people would approach me and said: “I want to implement some of the ideas floating around REVU2”. I think this would perhaps force me to start again coding on it. ATM, REVU is in maintenance only mode. :(

15 June 2007

Reinhard Tartler: create-chroot

I’ve crafted a small perl script for (re)creating chroots on lvm snapshots. You can find it on create-chroot. I’m not very used to program in perl, and I don’t enjoy it that much. This was a small test to get used to it, so if you have suggestions and/or improvments, feel free to edit it straight away. Perhaps it is a useful addition to the devscripts package?