Firebuild s Rust + Cargo support Firebuild treats programs as black boxes intercepting C standard library calls and system calls. It shortcuts the program invocations that predictably generate the same outputs because the program itself is known to be deterministic and all inputs are known in advance. Rust s compiler, rustc is deterministic in itself and simple rustc invocations were already accelerated, but parallel builds driven by Cargo needed a few enhancements in Firebuild.

Cargo s jobserver Cargo uses the Rust variant of the GNU Make s jobserver to control the parallelism in a build. The jobserver creates a file descriptor from which descendant processes can read tokens and are allowed to run one extra thread or parallel process per token received. After the extra threads or processes are finished the tokens must be returned by writing to the other file descriptor the jobserver created. The jobserver s file descriptors are shared with the descendant processes via environment variables:

# rustc's environment variables
...
CARGO_MAKEFLAGS="-j --jobserver-fds=4,5 --jobserver-auth=4,5"
...

Since getting tokens from the jobserver involves reading them as nondeterministic bytes from an inherited file descriptor this is clearly an operation that would depend on input not known in advance. Firebuild needs to make an exception and ignore jobserver usage related reads and writes since they are not meant to change the build results. However, there are programs not caring at all about jobservers and their file descriptors. They happily close the inherited file descriptors and open new ones with the same id, to use them for entirely different purposes. One such program is the widely used ./configure script, thus the case is far from being theoretical. To stay on the safe side firebuild ignores jobserver fd usage only in programs which are known to use the jobserver properly. The list of the programs is now configurable in /etc/firebuild.conf and since rustc is on the list by default parallel Rust builds are accelerated out of the box!

Writable dependency dir The other issue that prevented highly accelerated Rust builds was rustc s -L dependency=<dir> parameter. This directory is populated in a not fully deterministic order in parallel builds. Firebuild on the other hand hashes directory listings of open()-ed directories treating them as inputs assuming that the directory content will influence the intercepted programs outputs. As rustc programs started in parallel scanned the dependency directory in different states depending on what other Rust compilations finished already Firebuild had to store the full directory content as an input for each rustc cache entry resulting low hit rate when rustc was started again with otherwise identical inputs. The solution here is ignoring rustc scanning the dependency directory, because the dependencies actually used are still treated as input and are checked when shortcutting rustc. With that implemented in firebuild, too, librsvg s build that uses Rust and Cargo can be accelerated by more than 90%, even on a system having 12 cores/24 threads!:

Firebuild accelerating librsvg s Rust + Cargo build from 38s to 2.8s on a Ryzen 5900X (12C/24T) system

On the way to accelerate anything Firebuild s latest release incorporated more than 100 changes just from the last two months. They unlocked acceleration of Rust builds with Cargo, fixed Firebuild to work with the latest Java update that slightly changed its behavior, started accelerating Intel s Fortran compiler in addition to accelerating gfortran that was already supported and included many smaller changes improving the acceleration of other compilers and tools. If your favorite toolchain is not mentioned, there is still a good chance that it is already supported. Give Firebuild a try and tell us about your experience! Update 1: Comparison to sccache came up in the reddit topic about Firebuild s Rust acceleration , thus by popular demand this is how sccache performs on the same project:

Firebuild 0.8.0 vs. sccache 0.4.2 accelerating librsvg s Rust + Cargo build

All builds took place on the same Ryzen 5900X system with 12 cores / 24 threads in LXC containers limited to using 1-12 virtual CPUs. A warm-up build took place before the vanilla (without any instrumentation) build to download and compile the dependency crates to measure only the project s build time. A git clean command cleared all the build artifacts from the project directory before each build and ./autogen.sh was run to measure only clean rebuilds (without autotools). See test configuration in the Firebuild performance test repository for more details and easy reproduction. Firebuild had lower overhead than sccache (2.83% vs. 6.10% on 1 CPU and 7.71% vs. 22.05% on 12 CPUs) and made the accelerated build finish much faster (2.26% vs. 19.41% of vanilla build s time on 1 CPU and 7.5% vs. 27.4% of vanilla build s time on 12 CPUs).

Build performance with 18 cores The Linux kernel s build time is a widely used benchmark for compilers, making it a prime candidate to test a build accelerator as well. In the first run on Russell s 18 core test system the observed user+sys CPU time was cut by 44% with an actual increase in wall clock time which was quite unusual. Firebuild performed much better than that in prior tests. To replicate the results I ve set up a clean Debian Bookworm VM on my machine:

lxc launch images:debian/bookworm  vm -c limits.cpu=18 -c limits.memory=16GB bookworm-vm

Compiling Linux 6.1.10 in this clean Debian VM showed build times closer to what I expected to see, ~72% less wall clock time and ~97% less user+sys CPU time:

$ make defconfig && time make bzImage -j18
real	1m31.157s
user	20m54.256s
sys	2m25.986s
$ make defconfig && time firebuild make bzImage -j18
# first run:
real	2m3.948s
user	21m28.845s
sys	4m16.526s
# second run
real	0m25.783s
user	0m56.618s
sys	0m21.622s

There are multiple differences between Russell s and my test system including having different CPUs (E5-2696v3 vs. virtualized Ryzen 5900X) and different file systems (BTRFS RAID-1 vs ext4), but I don t think those could explain the observed mismatch in performance. The difference may be worth further analysis, but let s get back to squeezing out more performance from Firebuild. Firebuild was developed on Ubuntu. I was wondering if Firebuild was faster there, but I got only slightly better build times in an identical VM running Ubuntu 22.10 (Kinetic Kudu):

$ make defconfig && time make bzImage -j18
real	1m31.130s
user	20m52.930s
sys	2m12.294s
$ make defconfig && time firebuild make bzImage -j18
# first run:
real	2m3.274s
user	21m18.810s
sys	3m45.351s
# second run
real	0m25.532s
user	0m53.087s
sys	0m18.578s

The KVM virtualization certainly introduces an overhead, thus builds must be faster in LXC containers. Indeed, all builds are faster by a few percents:

$ lxc launch ubuntu:kinetic kinetic-container
...
$ make defconfig && time make bzImage -j18
real	1m27.462s
user	20m25.190s
sys	2m13.014s
$ make defconfig && time firebuild make bzImage -j18
# first run:
real	1m53.253s
user	21m42.730s
sys	3m41.067s
# second run
real	0m24.702s
user	0m49.120s
sys	0m16.840s
# Cache size:    1.85 GB

Apparently this ~72% reduction in wall clock time is what one should expect by simply prefixing the build command with firebuild on a similar configuration, but we should not stop here. Firebuild does not accelerate quicker commands by default to save cache space. This howto suggests letting firebuild accelerate all commands including even "sh by passing "-o 'processes.skip_cache = []' to firebuild. Accelerating all commands in this build s case increases cache size by only 9%, and increases the wall clock time saving to 91%, not only making the build more than 10X faster, but finishing it in less than 8 seconds, which may be a new world record!:

$ make defconfig && time firebuild -o 'processes.skip_cache = []' make bzImage -j18
# first run:
real	1m54.937s
user	21m35.988s
sys	3m42.335s
# second run
real	0m7.861s
user	0m15.332s
sys	0m7.707s
# Cache size:    2.02 GB

There are even faster CPUs on the market than this 5900X. If you happen to have access to one please leave a comment if you could go below 5 seconds! Scaling to higher core counts and comparison with ccache Russell raised the very valid point about Firebuild s single threaded supervisor being a bottleneck on high core systems and comparison to ccache also came up in comments. Since ccache does not have a central supervisor it could scale better with more cores, but let s see if ccache could go below 10 seconds with the build times

firebuild -o processes.skip_cache = [] and ccache scaling to 24 cores

Well, no. The best time time for ccache is 18.81s, with -j24. Both firebuild and ccache keep gaining from extra cores up to 8 cores, but beyond that the wall clock time improvements diminish. The more interesting difference is that firebuild s user and sys time is basically constant from -j1 to -j24 similarly to ccache s user time, but ccache s sys time increases linearly or exponentially with the number of used cores. I suspect this is due to the many parallel ccache processes performing file operations to check if cache entries could be reused, while in firebuild s case the supervisor performs most of that work not requiring in-kernel synchronization across multiple cores. It is true, that the single threaded firebuild supervisor is a bottleneck, but the supervisor also implements a central filesystem cache, thus checking if a command s cache entry can be reused can be implemented with much fewer system calls and much less user space hashing making the architecture more efficient overall than ccache s. The beauty of Firebuild is not being faster than ccache, but being faster than ccache with basically no hard-coded information about how C compilers work. It can accelerate any other compiler or program that generates deterministic output from its input, just by observing what they did in their prior runs. It is like having ccache for every compiler including in-house developed ones, and also for random slow scripts.

OK, but how does it work? Firebuild intercepts all processes started by the command to cache their outputs. Next time when the command or any of its descendant commands is executed with the same parameters, inputs and environment, the outputs are replayed (the command is shortcut) from the cache instead of running the command again. This is similar to how ccache and other compiler-specific caches work, but firebuild can shortcut any deterministic command, not only a specific list of compilers. Since the inputs of each command is determined at run time firebuild does not need a maintained complete dependency graph in the source like Bazel. It can work with any build system that does not implement its own caching mechanism. Determinism of commands is detected at run-time by preloading libfirebuild.so and interposing standard library calls and syscalls. If the command and all its descendants inputs are available when the command starts and all outputs can be calculated from the inputs then the command can be shortcut, otherwise it will be executed again. The interception comes with a 5-10% overhead, but rebuilds can be 5-20 times, or even faster depending on the changes between the builds. Can I try it? It is already available in Debian Unstable and Testing, Ubuntu s development release and the latest stable version is back-ported to supported Ubuntu releases via a PPA. How can I analyze my builds with firebuild? Firebuild can generate an HTML report showing each command s contribution to the build time. Below are the before and after reports of json4s, a Scala project. The command call graphs (lower ones) show that java (scalac) took 99% of the original build. Since the scalac invocations are shortcut (cutting the second build s time to less than 2% of the first one) they don t even show up in the accelerated second build s call graph. What s left to be executed again in the second run are env, perl, make and a few simple commands.

First run (102% of vanilla build)

Second run (1.4 % of vanilla build)

Command details

The upper graphs are the process trees, with expandable nodes (in blue) also showing which command invocations were shortcut (green). Clicking on a node shows details of the command and the reason if it was not shortcut. Could I accelerate my project more? Firebuild works best for builds with CPU-intensive processes and comes with defaults to not cache very quick commands, such as sh, grep, sed, etc., because caching those would take cache space and shortcutting them may not speed up the build that much. They can still be shortcut with their parent command. Firebuild s strength is that it can find shortcutting points in the process tree automatically, e.g. from sh -c 'bash -c "sh -c echo Hello World!"' bash would be shortcut, but none of the sh commands would be cached. In typical builds there are many such commands from the skip_cache list. Caching those commands with firebuild -o 'processes.skip_cache = []' can improve acceleration and make the reports smaller. Firebuild also supports several debug flags and -d proc helps finding reasons for not shortcutting some commands:

...
FIREBUILD: Command "/usr/bin/make" can't be short-cut due to: Executable set to be not shortcut,  ExecedProcess 1329.2, running, "make -f debian/rules build", fds=[ FileFD fd=0  FileOFD ...
FIREBUILD: Command "/usr/bin/sort" can't be short-cut due to: Process read from inherited fd ,  ExecedProcess 4161.1, running, "sort", fds=[ FileFD fd=0  FileOFD ...
FIREBUILD: Command "/usr/bin/find" can't be short-cut due to: fstatfs() family operating on fds is not supported,  ExecedProcess 1360.1, running, "find -mindepth 1 ...
...

make, ninja and other incremental build tool binaries are not shortcut because they compare the timestamp of files, but they are fast at least and every build step they perform can still be shortcut. Ideally the slower build steps that could not be shortcut can be re-implemented in ways that can be shortcut by avoiding tools performing unsupported operations. I hope those tools help speeding up your build with very little effort, but if not and you find something to fix or improve in firebuild itself, please report it or just leave a feedback! Happy speeding, but not on public roads!

It is now widely known that Ubuntu 22.04 LTS (Jammy Jellyfish) ships Firefox as a snap, but some people (like me) may prefer installing it from .deb packages to retain control over upgrades or to keep extensions working. Luckily there is still a PPA serving firefox (and thunderbird) debs at https://launchpad.net/~mozillateam/+archive/ubuntu/ppa maintained by the Mozilla Team. (Thank you!) You can block the Ubuntu archive s version that just pulls in the snap by pinning it:

$ cat /etc/apt/preferences.d/firefox-no-snap 
Package: firefox*
Pin: release o=Ubuntu*
Pin-Priority: -1

Now you can remove the transitional package and the Firefox snap itself:

sudo apt purge firefox
sudo snap remove firefox
sudo add-apt-repository ppa:mozillateam/ppa
sudo apt update
sudo apt install firefox

Since the package comes from a PPA unattended-upgrades will not upgrade it automatically, unless you enable this origin:

echo 'Unattended-Upgrade::Allowed-Origins:: "LP-PPA-mozillateam:$ distro_codename ";'   sudo tee /etc/apt/apt.conf.d/51unattended-upgrades-firefox

Happy browsing! Update: I have found a few other, similar guides at https://fostips.com/ubuntu-21-10-two-firefox-remove-snap and https://ubuntuhandbook.org/index.php/2022/04/install-firefox-deb-ubuntu-22-04 and I ve updated the pinning configuration based on them.

When Julian Andres Klode and I added initial Zstandard compression support to Ubuntu s APT and dpkg in Ubuntu 18.04 LTS we planned getting the changes accepted to Debian quickly and making Ubuntu 18.10 the first release where the new compression could speed up package installations and upgrades. Well, it took slightly longer than that. Since then many other packages have been updated to support zstd compressed packages and read-only compression has been back-ported to the 16.04 Xenial LTS release, too, on Ubuntu s side. In Debian, zstd support is available now in APT, debootstrap and reprepro (thanks Dimitri!). It is still under review for inclusion in Debian s dpkg (BTS bug 892664). Given that there is sufficient archive-wide support for zstd, Ubuntu is switching to zstd compressed packages in Ubuntu 21.10, the current development release. Please welcome hello/2.10-2ubuntu3, the first zstd-compressed Ubuntu package that will be followed by many other built with dpkg (>= 1.20.9ubuntu2), and enjoy the speed!

These are the last hours when we can update packages and they migrate to testing after 10 days right before the full freeze on 5 February. The latest Wireshark upstream version, 2.2.4 has been released on Monday and it is waiting patiently to be part of next Debian stable. I have just tested the fix for Kodi s bug preventing playing DVD-s and today it will be fixed in unstable as well. If you have packages which could be updated to make Stretch even better you can still do it today, but don t wait too long! Thanks to everyone working on Debian! Stretch will be awesome!

I pledge that before sending each email to the debian-devel mailing list I move forward at least one actionable bug in my packages.

I have just finished level one, fixing all RC bugs in packages under my name, even in team-maintained ones. Next level is no unclassified bug reports, which gonna be harder since I have just adopted shadow with 70+ open bugs. :-\ Luckily I can still go on bonus tracks which is fixing (RC) bugs in others packages, but one should not spend all the time on those track before finishing level 1! PS: Last time I tried playing a conventional game I ended up fixing it in a few minutes instead.

Hardening all executables by making them position independent by default is basically ready with a few packages to fix (bugs). On the other hand bindnow is not enabled globally (#835146) and it seems it will not be for the next stable release despite my plan :-(. If you are a maintainer you can still have your packages hardened in Stretch by enabling bindnow per package before 25 January, 2017. It could be a nice present for your users! update: It is nice to see how enabling PIE in GCC increased PIE coverage while bindnow coverage is improving slowly with maintainers enabling it package by package: update 2: Changed the deadline of enabling bindnow per package to align with the start of the full freeze, not the soft freeze.

Shipping Position Independent Executables and using read-only Global Offset Table was already possible for packages but needed package maintainers to opt-in for each package (see Hardening wiki) using the pie and bindnow Dpkg hardening flags. Many critical packages enabled the extra flags but there are still way more left out according to Lintian hardening-no-bindnow and hardening-no-pie warnings. Now we can change that. We can make those hardening flags the default for every package.
We already have the needed patches for GCC (#835148) and dpkg (#835146, #835149). We already have all packages rebuilt once to test which breaks (Thanks to Lucas Nussbaum!). The Release Team already asked porters if they feel their ports ready for enabling PIE and most ports tentatively opted-in (Thanks to Niels Thykier for pushing this!). What is left is fixing the ~75 open bugs found during the test rebuilds and this is where You can help, too! Please check if your packages are affected or give a helping hand to other maintainers who need it. (See PIEByDefaultTransition wiki for hints on fixing the bugs.) Many thanks to those who already fixed their packages! If we can get past those last bugs we can enable those badly needed security features and make Stretch the most secure release ever!

It was more that one and a half years ago when I proposed creating a new security &QA focused port for Debian and now I m happy to share the first bits of it. Last year I started the bootstrapping during the holidays and I now have the prototype in the form of cross built packages which can be installed next to amd64 packages using multiarch. The aim of creating the port is still the same, letting people mix fast (amd64) and reasonably hardened (hardened1-linux-amd64) packages on the same system. You can already try the cross-built packages in an amd64 unstable chroot, but be warned that the packages are not stable yet. In the following session I tested curl which seems to be working OK, and groff, which seems to be too buggy even for debugging:

debootstrap --arch=amd64 unstable test-hardened1
# mount /proc for ASAN
mount --bind /proc test-hardened1/proc
chroot test-hardened1/
apt-get install debian-keyring
# this is my key, I'll create one dedicated release key later
gpg --keyring /usr/share/keyrings/debian-keyring.gpg -a --export 0x21E764DF   apt-key add -
echo "deb http://hardened1-debian.s3.amazonaws.com/debian-cross-built hardened1-unstable main" >> \
/etc/apt/sources.list
apt-get update
# update apt and dpkg to versions handling the new port
apt-get upgrade
apt-get update
dpkg --add-architecture hardened1-linux-amd64
apt-get update
apt-get install curl:hardened1-linux-amd64
curl -s https://www.debian.org   tail -n2
</body>
</html>
apt-get install -t hardened1-unstable groff:hardened1-linux-amd64
groff
ASAN:SIGSEGV
=================================================================
==20642==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f79cd84698a bp 0x619000006980 sp 0x7ffe89b3a930 T0)
ASAN:SIGSEGV
==20642==AddressSanitizer: while reporting a bug found another one. Ignoring.

The next steps are finalizing the changes to apt, dpkg, GCC, glibc and other packages, rebuilding all packages in hardened1-linux-amd64 sbuild chroots and building the rest of the archive.
Some of the patches are not submitted yet but they are available in a temporary fork of rebootstrap
I hope I ll be back soon with the recompiled and finalized packages, but until then feel free to try the cross-compiled ones! Patches fixing crashes are always welcome! update 1: Some packages like dpkg-dev are not installable, I m working on them.
update 2: There is one similar project I know of which aims creating an address sanitized Gentoo variant and Hanno B ck will give a presentation about that at FOSDEM.

With the latest release the Wireshark Project decided to make the Qt GUI the default interface. In line with Debian s Policy the packages shipped by Debian also switched the default GUI to minimize the difference from upstream. The GTK+ interface which was the previous default is still available from the wireshark-gtk package. You can read more about the new 2.0.0 release in the release notes or on the Wireshark Blog featuring some of the improvements. Happy sniffing! update: Wireshark 2.0.0 will be available from testing and jessie-backports in a week. Ubuntu users can already download binary packages from the Wireshark stable releases PPA maintained by the Wireshark Project (including me:-)).

Debian has switched to FFmpeg in testing in July but the work on the package did not stop at that point. After careful testing we can now provide official packages for Jessie users through jessie-backports. See installation instructions here. FFmpeg becoming available in jessie-backports also enabled us to provide Kodi from Debian in the same official repository. Thanks to everyone in the Debian Multimedia Maintainers team, especially Andreas Cadhalpun who is also upstream developer at the FFmpeg project, Reinhard Tartler who maintained FFmpeg then Libav then FFmpeg again in Debian for long years and everyone else I could not name here but helped making this possible!

The Linux Kernel Oops website collects kernel errors from all over the World helping kernel developers finding issues occurring in the wild but they cannot help if no one sends reports to them. The Kerneloops client used to be part of Debian releases but it has been removed from the archive due to not working with the new collector site. When I started observing oopses on my machine I first thought of submitting a bug against the linux package in BTS, but looking at the numerous bugs opened already I looked for a more automated solution which would also help others. Reviving the kerneloops package involved switching it to the new submission URL, fixing a few memory allocation bugs in C (this is the first package I found using Valgrind by default for running tests) and ensuring that upstream was still active. The last step took the most of the time but finally Anton Arapov kindly accepted my patches and everything was set for the new upload. The package is now available from unstable and if you feel so (especially if you experience oopses) please give it a try and report any problems you find. I m also happy to receive success stories about oopses fixed after discovering and collecting them with the client.

Erlang lets you write applications supporting zero downtime by switching one live system to another running a different application version converting the application s state on the fly to the new representation. Debian packages however can have only one installed version on a system which prevents using Erlang s hot-upgrade feature easily. Engineers at Yakaz (Jean-S bastien P dron) came up with a nice solution by creating separate directories for each application release and creating .deb packages for managing the transitions. I had to solve the same problem recently and found that the erlsvc Perl application they created needed a few patches to be usable with latest Erlang and other packages and with the changes it worked perfectly. Yakaz was not interested in accepting the patches and developing it further, but let me continue the maintenance. Please find the updated erlsvc application under my GitHub account and feel free to submit patches if you find something to fix in it. I have also packaged erlsvc as an official Debian package and it is waiting in the NEW queue for being accepted. When it enters unstable you will have to make very little effort to make your applications hot-upgradeable on Debian!

The well known XBMC Media Center has been renamed to Kodi with the 14.0 Helix release and following upstream s decision the xbmc packages are renamed to kodi as well. Debian ships a slightly changed version of XBMC using the XBMC from Debian name and following that tradition ladies and gentlemen let me introduce you Kodi from Debian : As of today Kodi from Debian uses the FFmpeg packages instead of the Libav ones which have been used by XBMC from Debian. The reason for the switch was upstream s decision of dropping the Libav compatibility code and FFmpeg becoming available again packaged in Debian (thanks to Andreas Cadhalpun). It is worth noting that while upstream Kodi 14.0 downloads and builds FFmpeg 2.4.4 by default, Debian ships FFmpeg 2.5.1 already and FFmpeg under Kodi will be updated independently from Kodi thanks to the packaging mechanism. The new kodi packages are uploaded to the NEW queue and are waiting for being accepted by the FTP Masters who are busy with preparing Jessie for the release (Many thanks to them for their hard work!), but in the meantime you can install kodi from https://people.debian.org/~rbalint/ppa/xbmc-ffmpeg/. Happy recovery from the holidays!

Thanks to the great work of the XBMC Team XBMC 13.0 Gotham has been released last Sunday and now XBMC from Debian can be downloaded from experimental to Jessie and Sid systems. It will take some time to enter unstable since it is blocked by the Libav 10 transition, but that will happen, too, eventually. I have also set up a separate repository at http://people.debian.org/~rbalint/ppa/xbmc-ffmpeg/ based on the Debian packages in main but using XBMC s internal copy of FFmpeg because I received several request asking for this variant. The packages there can be used on Wheezy (stable), Jessie (testing) and Sid (unstable) but are not part of Debian. Update 1: For the interested parties the XBMC 13 Libav compatibility patches are available from a git branch in the packaging repository.