The following contributors got their Debian Developer accounts in the last two months:
load-one.cmds:
load $ 1 apply-formula y2=y**2 /extra-columns=1 flag /flags=processedWhen this script is run with the name of a spectrum file as argument, it loads it (replaces
$ 1 by the first argument, the file name), adds a column y2 containing the square of the y column, and flag it with the processed flag. This is not absolutely necessary, but it makes it much easier to refer to all the spectra when they are processed. Then to process all the spectra, one just has to run the following commands:
run-for-each load-one.cmds Spectrum-1.dat Spectrum-2.dat Spectrum-3.dat average flagged:processed apply-formula y2=(y2-y**2)**0.5 dataset-options /yerrors=y2The
run-for-each command runs the load-one.cmds script for all the spectra (one could also have used Spectra-*.dat to not have to give all the file names). Then, the average averages the values of the columns over all the datasets. To be clear, it finds all the values that have the same X (or very close X values) and average them, column by column. The result of this command is therefore a dataset with the average of the original \(y\) data as y column and the average of the original \(y^2\) data as y2 column. So now, the only thing left to do is to use the above equation, which is done by the apply-formula code. The last command, dataset-options, is not absolutely necessary but it signals to QSoas that the standard error of the y column should be found in the y2 column. This is now available as script method-one.cmds in the git repository.
apply-formula, the value $stats.y_stddev corresponds to the standard deviation of the whole y column... Loading the spectra yields just a series of x,y datasets. We can contract them into a single dataset with one x column and several y columns:
load Spectrum-*.dat /flags=spectra contract flagged:spectraAfter these commands, the current dataset contains data in the form of:
lambda1 a1_1 a1_2 a1_3 lambda2 a2_1 a2_2 a2_3 ...in which the
ai_1 come from the first file, ai_2 the second and so on. We need to use transpose to transform that dataset into:
0 a1_1 a2_1 ... 1 a1_2 a2_2 ... 2 a1_3 a2_3 ...In this dataset, values of the absorbance for the same wavelength for each dataset is now stored in columns. The next step is just to use
expand to obtain a series of datasets with the same x column and a single y column (each corresponding to a different wavelength in the original data). The game is now to replace these datasets with something that looks like:
0 a_average 1 a_stddevFor that, one takes advantage of the
$stats.y_average and $stats.y_stddev values in apply-formula, together with the i special variable that represents the index of the point:
apply-formula "if i == 0; then y=$stats.y_average; end; if i == 1; then y=$stats.y_stddev; end" strip-if i>1Then, all that is left is to apply this to all the datasets created by
expand, which can be just made using run-for-datasets, and then, we reverse the splitting by using contract and transpose ! In summary, this looks like this. We need two files. The first, process-one.cmds contains the following code:
apply-formula "if i == 0; then y=$stats.y_average; end; if i == 1; then y=$stats.y_stddev; end" strip-if i>1 flag /flags=processedThe main file,
method-two.cmds looks like this:
load Spectrum-*.dat /flags=spectra contract flagged:spectra transpose expand /flags=tmp run-for-datasets process-one.cmds flagged:tmp contract flagged:processed transpose dataset-options /yerrors=y2Note some of the code above can be greatly simplified using new features present in the upcoming 3.0 version, but that is the topic for another post. About QSoasQSoas is a powerful open source data analysis program that focuses on flexibility and powerful fitting capacities. It is released under the GNU General Public License. It is described in Fourmond, Anal. Chem., 2016, 88 (10), pp 5050 5052. Current version is 2.2. You can download its source code and compile it yourself or buy precompiled versions for MacOS and Windows there.
Welcome to the October 2020 report from the Reproducible Builds project. In our monthly reports, we outline the major things that we have been up to over the past month. As a brief reminder, the motivation behind the Reproducible Builds effort is to ensure flaws have not been introduced in the binaries we install on our systems. If you are interested in contributing to the project, please visit our main website.
On Saturday 10th October, Morten Linderud gave a talk at Arch Conf Online 2020 on The State of Reproducible Builds in Arch. The video should be available later this month, but as a teaser:
The previous year has seen great progress in Arch Linux to get reproducible builds in the hands of the users and developers. In this talk we will explore the current tooling that allows users to reproduce packages, the rebuilder software that has been written to check packages and the current issues in this space.During the Reproducible Builds summit in Marrakesh in 2019, developers from the GNU Guix, NixOS and Debian distributions were able to produce a bit-for-bit identical GNU Mes binary despite using three different versions of GCC. Since this summit, additional work resulted in a bit-for-bit identical Mes binary using
tcc, and last month a fuller update was posted to this effect by the individuals involved. This month, however, David Wheeler updated his extensive page on Fully Countering Trusting Trust through Diverse Double-Compiling, remarking that:
GNU Mes rebuild is definitely an application of [Diverse Double-Compiling]. [..] This is an awesome application of DDC, and I believe it s the first publicly acknowledged use of DDC on a binaryThere was a small, followup discussion on our mailing list. In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update.
This month, the Reproducible Builds project restarted our IRC meetings, managing to convene twice: the first time on October 12th (summary & logs), and later on the 26th (logs). As mentioned in previous reports, due to the unprecedented events throughout 2020, there will be no in-person summit event this year.
On our mailing list this month El as Alejandro posted a request for help with a local configuration
In August, Lucas Nussbaum performed an archive-wide rebuild of packages to test enabling the reproducible=+fixfilepath Debian build flag by default. Enabling this fixfilepath feature will likely fix reproducibility issues in an estimated 500-700 packages. However, this month Vagrant Cascadian posted to the debian-devel mailing list:
It would be great to see theDebian Developer Stuart Prescott has been improvingreproducible=+fixfilepathfeature enabled by default indpkg-buildflags, and we would like to proceed forward with this soon unless we hear any major concerns or other outstanding issues. [ ] We would like to move forward with this change soon, so please raise any concerns or issues not covered already.
python-debian, a Python library that is used to parse Debian-specific files such as changelogs, .dscs, etc. In particular, Stuart is working on adding support for .buildinfo files used for recording reproducibility-related build metadata:
This can mostly be a very thin layer around the existingA total of 159 Debian packages were categorised, 69 had their categorisation updated, and 33 had their classification removed this month, adding to our knowledge about identified issues. As part of this, Chris Lamb identified and classified two new issues:Deb822types, using the existingChangescode for the file listings, the existingPkgRelationscode for the package listing andgpg_*functions for signature handling.
build_path_captured_in_emacs_el_file and rollup_embeds_build_path.
go (version 1.15.3 has improved reproducibility over 1.14)goxel (sort SCons-related filesystem ordering issue)lal (rework an old date-related patch)lalmetaio (date)libsemigroups (build failure in single-CPU mode)memcached (build failure in 2025 due to expired SSL certificate)octant (SUSE-specific date issue)openmpi4 (date-related problem, revive old patch)sbcl (datetime and hostname issue)selinux-policy/policycoreutils (date-related issue in timezone)evince (forwarded upstream).libsass-python (forwarded upstream).pitivi.sound-juicer.pcbasic.ora2pg (forwarded upstream).fckit.gita.libgrokj2k.softether-vpn.perl.ruby-appraiser.gmerlin-avdecoder.node-proxy.yard.emacs.netcdf-parallel.dh-fortran-mod.bison, ibus and postgresql12.
diffoscope is our in-depth and content-aware diff utility. Not only could you locate and diagnose reproducibility issues, it provides human-readable diffs of all kinds too. This month, Chris Lamb uploaded version 161 to Debian (later backported by Mattia Rizzolo), as well as made the following changes:
test_ocaml to the assert_diff helper. [ ]20.8b1. (#972518)radare2 to ensure our test pipelines continue to work [ ], and for the GNU Guix distribution Vagrant Cascadian diffoscope to version 161 [ ].
In related development, trydiffoscope is the web-based version of diffoscope. This month, Chris Lamb made the following changes:
--help-only test as being a superficial test. (#971506)try.diffoscope.org service. [ ]debhelper compatibility level to 13 [ ] and bump Standards-Version to 4.5.0 [ ].0.5.10-2 was uploaded to Debian unstable by Holger Levsen, which enabled security hardening via DEB_BUILD_MAINT_OPTIONS [ ] and dropped debian/disorderfs.lintian-overrides [ ].
This month, a number of updates to the main Reproducible Builds website and related documentation were made by Chris Lamb:
dettrace [ ], and added yet another supply-chain security attack publication [ ].relative_url to fix missing translation icon on various pages. [ ]
The Reproducible Builds project operates a Jenkins-based testing framework that powers tests.reproducible-builds.org. This month, Holger Levsen made the following changes:
dpkg-buildpackage. [ ]
ath79 from ath97. [ ]
Build node maintenance was performed by both Holger Levsen [ ][ ] and Vagrant Cascadian [ ][ ][ ], Vagrant Cascadian also updated the page listing the variations made when testing to reflect changes for in build paths [ ] and Hans-Christoph Steiner made a number of changes for F-Droid, the free software app repository for Android devices, including:
sudo command if we are not actually running libvirt. [ ]#reproducible-builds on irc.oftc.net.
rb-general@lists.reproducible-builds.org
My maybe most impactful piece of code is tttool and the surrounding project, which allows you to create your own content for the Ravensburger Tiptoi platform. The program itself is a command line tool, and in this blog post I want to show how I go about building that program for Linux (both normal and static builds), Windows (cross-compiled from Linux), OSX (only on CI), all combined into and released as a single zip file.
Maybe some of it is useful or inspiring to my readers, or can even serve as a template. This being a blob post, though, note that it may become obsolete or outdated.
ghc. For many years I resisted learning about nix, but when I eventually had to, I didn t want to go back.
This project provides an alternative Haskell build infrastructure for nix. While this is not crucial for tttool, it helps that they tend to have some cross-compilation-related patches more than the official nixpkgs. I also like that it more closely follows the cabal build work-flow, where cabal calculates a build plan based on your projects dependencies. It even has decent documentation (which is a new thing compared to two years ago).
Niv is a neat little tool to keep track of your dependencies. You can quickly update them with, say niv update nixpkgs. But what s really great is to temporarily replace one of your dependencies with a local checkout, e.g. via NIV_OVERRIDE_haskellNix=$HOME/build/haskell/haskell.nix nix-instantiate -A osx-exe-bundle There is a Github action that will keep your niv-managed dependencies up-to-date.
This service (proprietary, but free to public stuff up to 10GB) gives your project its own nix cache. This means that build artifacts can be cached between CI builds or even build steps, and your contributors. A cache like this is a must if you want to use nix in more interesting ways where you may end up using, say, a changed GHC compiler. Comes with GitHub actions integration.
.cabal file (but multi-package projects should work just fine). To make things more interesting, I also have a cabal.project which configures one dependency to be fetched via git from a specific fork.
To start building the nix infrastructure, we can initialize niv and configure it to use the haskell.nix repo:
niv init
niv add input-output-hk/haskell.nix -n haskellNixnix/sources.json (which you can also edit by hand) and nix/sources.nix (which you can treat like a black box).
Now we can start writing the all-important default.nix file, which defines almost everything of interest here. I will just go through it line by line, and explain what I am doing here.
checkMaterialization ? false :nix-build, by passing --arg checkMaterialization true, and which is off by default. I ll get to that flag later.
let
sources = import nix/sources.nix;
haskellNix = import sources.haskellNix ;niv/sources.json, and loads the pinned revision of the haskell.nix repository.
# windows crossbuilding with ghc-8.10 needs at least 20.09.
# A peek at https://github.com/input-output-hk/haskell.nix/blob/master/ci.nix can help
nixpkgsSrc = haskellNix.sources.nixpkgs-2009;
nixpkgsArgs = haskellNix.nixpkgsArgs;
pkgs = import nixpkgsSrc nixpkgsArgs;pkgs, which is our version of the nixpkgs package set, extended with the haskell.nix machinery. We rely on haskell.nix to pin of a suitable revision of the nixpkgs set (see how we are using their niv setup).
Here we could our own configuration, overlays, etc to nixpkgsArgs. In fact, we do in
pkgs-osx = import nixpkgsSrc (nixpkgsArgs // system = "x86_64-darwin"; ); # a nicer filterSource
sourceByRegex =
src: regexes: builtins.filterSource (path: type:
let relPath = pkgs.lib.removePrefix (toString src + "/") (toString path); in
let match = builtins.match (pkgs.lib.strings.concatStringsSep " " regexes); in
( type == "directory" && match (relPath + "/") != null
match relPath != null)) src; tttool-exe = pkgs: sha256:
(pkgs.haskell-nix.cabalProject cabalProject function takes a cabal project and turns it into a nix project, running cabal v2-configure under the hood to let cabal figure out a suitable build plan. Since we want to have multiple variants of the tttool, this is so far just a function of two arguments pkgs and sha256, which will be explained in a bit.
src = sourceByRegex ./. [
"cabal.project"
"src/"
"src/.*/"
"src/.*.hs"
".*.cabal"
"LICENSE"
];cabalProject function wants to know the source of the Haskell projects. There are different ways of specifying this; in this case I went for a simple whitelist approach. Note that cabal.project.freze (which exists in the directory) is not included.
# Pinning the input to the constraint solver
compiler-nix-name = "ghc8102";ghc to use, that is still my choice. I am using GHC-8.10.2 here. It may require a bit of experimentation to see which version works for your project, especially when cross-compiling to odd targets.
index-state = "2020-11-08T00:00:00Z";cabal suddenly pick different package versions just because something got uploaded. Therefore I specify which snapshot of the Hackage package index it should consider.
plan-sha256 = sha256;
inherit checkMaterialization; modules = [
# smaller files
packages.tttool.dontStrip = false;
] ++tttool binary to be stripped (saves a few megabyte).
pkgs.lib.optional pkgs.hostPlatform.isMusl
packages.tttool.configureFlags = [ "--ghc-option=-static" ];-static to GHC. This seems to be enough in terms of flags to produce static binaries. It helps that my project is using mostly pure Haskell libraries; if you link against C libraries you might have to jump through additional hoops to get static linking going. The haskell.nix documentation has a section on static building with some flags to cargo-cult.
# terminfo is disabled on musl by haskell.nix, but still the flag
# is set in the package plan, so override this
packages.haskeline.flags.terminfo = false;
;musl) seems to be necessary to workaround what might be a big in haskell.nix.
).tttool.components.exes.tttool;cabalProject function returns a data structure with all Haskell packages of the project, and for each package the different components (libraries, tests, benchmarks and of course executables). We only care about the tttool executable, so let s project that out.
osx-bundler = pkgs: tttool:
pkgs.stdenv.mkDerivation
name = "tttool-bundle";
buildInputs = [ pkgs.macdylibbundler ];
builder = pkgs.writeScript "tttool-osx-bundler.sh" ''
source $ pkgs.stdenv /setup
mkdir -p $out/bin/osx
cp $ tttool /bin/tttool $out/bin/osx
chmod u+w $out/bin/osx/tttool
dylibbundler \
-b \
-x $out/bin/osx/tttool \
-d $out/bin/osx \
-p '@executable_path' \
-i /usr/lib/system \
-i $ pkgs.darwin.Libsystem /lib
'';
;tttool, finds all the system libraries it is linking against, and copies them next to the executable, using the nice macdylibbundler. This way we can get a self-contained executable.
A nix expert will notice that this probably should be written with pkgs.runCommandNoCC, but then dylibbundler fails because it lacks otool. This should work eventually, though.
in rec
linux-exe = tttool-exe pkgs
"0rnn4q0gx670nzb5zp7xpj7kmgqjmxcj2zjl9jqqz8czzlbgzmkh";
windows-exe = tttool-exe pkgs.pkgsCross.mingwW64
"01js5rp6y29m7aif6bsb0qplkh2az0l15nkrrb6m3rz7jrrbcckh";
static-exe = tttool-exe pkgs.pkgsCross.musl64
"0gbkyg8max4mhzzsm9yihsp8n73zw86m3pwvlw8170c75p3vbadv";
osx-exe = tttool-exe pkgs-osx
"0rnn4q0gx670nzb5zp7xpj7kmgqjmxcj2zjl9jqqz8czzlbgzmkh";tttool. In each case we use the tttool-exe function from above, passing the package set (pkgs, ) and a SHA256 hash.
The package set is either the normal one, or it is one of those configured for cross compilation, building either for Windows or for Linux using musl, or it is the OSX package set that we instantiated earlier.
The SHA256 hash describes the result of the cabal plan calculation that happens as part of cabalProject. By noting down the expected result, nix can skip that calculation, or fetch it from the nix cache etc.
How do we know what number to put there, and when to change it? That s when the --arg checkMaterialization true flag comes into play: When that is set, cabalProject will not blindly trust these hashes, but rather re-calculate them, and tell you when they need to be updated. We ll make sure that CI checks them.
osx-exe-bundle = osx-bundler pkgs-osx osx-exe;osx-bundler defined above, to make it independent of any library paths in /nix/store.
This is already good enough to build the tool for the various systems! The rest of the the file is related to packaging up the binaries, to tests, and various other things, but nothing too essentially. So if you got bored, you can more or less stop now.
static-files = sourceByRegex ./. [
"README.md"
"Changelog.md"
"oid-decoder.html"
"example/.*"
"Debug.yaml"
"templates/"
"templates/.*\.md"
"templates/.*\.yaml"
"Audio/"
"Audio/digits/"
"Audio/digits/.*\.ogg"
];
contrib = ./contrib; book = ; os-switch = pkgs.writeScript "tttool-os-switch.sh" ''
#!/usr/bin/env bash
case "$OSTYPE" in
linux*) exec "$(dirname "''$ BASH_SOURCE[0] ")/linux/tttool" "$@" ;;
darwin*) exec "$(dirname "''$ BASH_SOURCE[0] ")/osx/tttool" "$@" ;;
msys*) exec "$(dirname "''$ BASH_SOURCE[0] ")/tttool.exe" "$@" ;;
cygwin*) exec "$(dirname "''$ BASH_SOURCE[0] ")/tttool.exe" "$@" ;;
*) echo "unsupported operating system $OSTYPE" ;;
esac
'';tttool command that works on all systems. To that end, I implement a simple platform switch using bash. I use pks.writeScript so that I can include that file directly in default.nix, but it would have been equally reasonable to just save it into nix/tttool-os-switch.sh and include it from there.
release = pkgs.runCommandNoCC "tttool-release"
buildInputs = [ pkgs.perl ];
''
# check version
version=$($ static-exe /bin/tttool --help perl -ne 'print $1 if /tttool-(.*) -- The swiss army knife/')
doc_version=$(perl -ne "print \$1 if /VERSION: '(.*)'/" $ book /book.html/_static/documentation_options.js)
if [ "$version" != "$doc_version" ]
then
echo "Mismatch between tttool version \"$version\" and book version \"$doc_version\""
exit 1
fi$ static-exe refers to a path with the built static Linux build, and $ book the output of the book building process.
mkdir -p $out/
cp -vsr $ static-files /* $out
mkdir $out/linux
cp -vs $ static-exe /bin/tttool $out/linux
cp -vs $ windows-exe /bin/* $out/
mkdir $out/osx
cp -vsr $ osx-exe-bundle /bin/osx/* $out/osx
cp -vs $ os-switch $out/tttool
mkdir $out/contrib
cp -vsr $ contrib /* $out/contrib/
cp -vsr $ book /* $out
'';static-exe (built on Linux) and osx-exe-bundle (built on Mac)! This means you can only build the release if you either have setup a remote osx builder (a pretty nifty feature of nix, which I unfortunately can t use, since I don't have access to a Mac), or the build product must be available in a nix cache (which it is in my case, as I will explain later).
The output of this derivation is a directory with all the files I want to put in the release.
release-zip = pkgs.runCommandNoCC "tttool-release.zip"
buildInputs = with pkgs; [ perl zip ];
''
version=$(bash $ release /tttool --help perl -ne 'print $1 if /tttool-(.*) -- The swiss army knife/')
base="tttool-$version"
echo "Zipping tttool version $version"
mkdir -p $out/$base
cd $out
cp -r $ release /* $base/
chmod u+w -R $base
zip -r $base.zip $base
rm -rf $base
''; gme-downloads = ;
tests = ;default.nix are related to some simple testing, and again not relevant for this post.
cabal-freeze = pkgs.stdenv.mkDerivation
name = "cabal-freeze";
src = linux-exe.src;
buildInputs = [ pkgs.cabal-install linux-exe.env ];
buildPhase = ''
mkdir .cabal
touch .cabal/config
rm cabal.project # so that cabal new-freeze does not try to use HPDF via git
HOME=$PWD cabal new-freeze --offline --enable-tests true
'';
installPhase = ''
mkdir -p $out
echo "-- Run nix-shell -A check-cabal-freeze to update this file" > $out/cabal.project.freeze
cat cabal.project.freeze >> $out/cabal.project.freeze
'';
;cabal, and ideally it should take the same library versions that the nix-based build does. But pinning the version of ghc in cabal.project is not sufficient, I also need to pin the precise versions of the dependencies. This is best done with a cabal.project.freeze file.
The above derivation runs cabal new-freeze in the environment set up by haskell.nix and grabs the resulting cabal.project.freeze. With this I can run nix-build -A cabal-freeze and fetch the file from result/cabal.project.freeze and add it to the repository.
check-cabal-freeze = pkgs.runCommandNoCC "check-cabal-freeze"
nativeBuildInputs = [ pkgs.diffutils ];
expected = cabal-freeze + /cabal.project.freeze;
actual = ./cabal.project.freeze;
cmd = "nix-shell -A check-cabal-freeze";
shellHook = ''
dest=$ toString ./cabal.project.freeze
rm -f $dest
cp -v $expected $dest
chmod u-w $dest
exit 0
'';
''
diff -r -U 3 $actual $expected
echo "To update, please run"; echo "nix-shell -A check-cabal-freeze"; exit 1;
touch $out
'';nix-shell -A check-cabal-freeze it will update the file in the repository automatically, which is much more convenient than manually copying.
Lately, I have been using this pattern regularly when adding generated files to a repository: * Create one nix derivation that creates the files * Create a second derivation that compares the output of that derivation against the file in the repo * Create a derivation that, when run in nix-shell, updates that file. Sometimes that derivation is its own file (so that I can just run nix-shell nix/generate.nix), or it is merged into one of the other two.
This concludes the tour of default.nix.
.github/workflows/build.yml, which tells Github Actions what to do:
name: "Build and package"
on:
pull_request:
push:jobs:
build:
strategy:
fail-fast: false
matrix:
include:
- target: linux-exe
os: ubuntu-latest
- target: windows-exe
os: ubuntu-latest
- target: static-exe
os: ubuntu-latest
- target: osx-exe-bundle
os: macos-latest
runs-on: $ matrix.os build job is a matrix job, i.e. there are four variants, one for each of the different tttool builds, together with an indication of what kind of machine to run this on.
- uses: actions/checkout@v2
- uses: cachix/install-nix-action@v12nix via the install-nix-action.
- name: "Cachix: tttool"
uses: cachix/cachix-action@v7
with:
name: tttool
signingKey: '$ secrets.CACHIX_SIGNING_KEY ' - run: nix-build --arg checkMaterialization true -A $ matrix.target checkMaterialization to true so that CI will tell us if we need to update these hashes.
# work around https://github.com/actions/upload-artifact/issues/92
- run: cp -RvL result upload
- uses: actions/upload-artifact@v2
with:
name: tttool ($ matrix.target )
path: upload/ test:
runs-on: ubuntu-latest
needs: build
steps:
- uses: actions/checkout@v2
- uses: cachix/install-nix-action@v12
- name: "Cachix: tttool"
uses: cachix/cachix-action@v7
with:
name: tttool
signingKey: '$ secrets.CACHIX_SIGNING_KEY '
- run: nix-build -A teststests. Because of needs: build it will not start before the builds job has completed. This also means that it should get the actual tttool executable to test from our nix cache.
check-cabal-freeze:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: cachix/install-nix-action@v12
- name: "Cachix: tttool"
uses: cachix/cachix-action@v7
with:
name: tttool
signingKey: '$ secrets.CACHIX_SIGNING_KEY '
- run: nix-build -A check-cabal-freezecheck-cabal-freeze test mentioned above. Quite annoying to repeat the setup instructions for each job
package:
runs-on: ubuntu-latest
needs: build
steps:
- uses: actions/checkout@v2
- uses: cachix/install-nix-action@v12
- name: "Cachix: tttool"
uses: cachix/cachix-action@v7
with:
name: tttool
signingKey: '$ secrets.CACHIX_SIGNING_KEY '
- run: nix-build -A release-zip
- run: unzip -d upload ./result/*.zip
- uses: actions/upload-artifact@v2
with:
name: Release zip file
path: uploadbuild to finish so that the built programs are in the nix cache. This is especially important since this runs on linux, so it cannot build the OSX binary and has to rely on the cache.
Note that we don t need to checkMaterialization again.
Annoyingly, the upload-artifact action insists on zipping the files you hand to it. A zip file that contains just a zipfile is kinda annoying, so I unpack the zipfile here before uploading the contents.
tttool, I just bump the version numbers, wait for CI to finish building, run nix-build -A release-zip and upload result/tttool-n.m.zip. A single file that works on all target platforms. I have not yet automated making the actual release, but with one release per year this is fine.
Also, when trying out a new feature, I can easily create a branch or PR for that and grab the build products from Github s CI, or ask people to try them out (e.g. to see if they fixed their bugs). Note, though, that you have to sign into Github before being able to download these artifacts.
One might think that this is a fairly hairy setup finding the right combinations of various repertories so that cross-compilation works as intended. But thanks to nix s value propositions, this does work! The setup presented here was a remake of a setup I did two years ago, with a much less mature haskell.nix. Back then, I committed a fair number of generated files to git, and juggled more complex files but once it worked, it kept working for two years. I was indeed insulated from upstream changes. I expect that this setup will also continue to work reliably, until I choose to upgrade it again. Hopefully, then things are even more simple, and require less work-around or manual intervention.
I've sent a merge request to show "Already in Debian" column in packages list on mentors.debian.net.
salsa.debian.org
At first, I've used Emoji, but for consistency, it has been modified to use "Yes or No".
This feature is not fully merged yet, but it may be useful to distinguish "This package is already in Debian or not" for sponsor.
Aready in Debian
Last week I received the official notice: There is now a German company named "Fre(i)e Software GmbH" registered with the German Trade Register.
Founding a New Company
Over the past months I have put my energy into founding a new company. As a freelancing IT consultant I started facing the limitation of other companies having strict policies that forbid the cooperation with one person businesses (Personengesellschaften).
Thus, the requirement for setting up a GmbH business came onto my agenda. I will move some of my business activities into this new company, starting next year.
Policy Ideas
The "Fre(i)e Software GmbH" will be a platform to facilitate the growth and spreading of Free Software on this planet.
Here are some first ideas for company policies:
A couple of years ago, I moved into a new flat that comes with RJ45 sockets wired for 10 Gigabit (but currently offering 1 Gigabit) Ethernet.This also meant changing the settings on my router box for my new ISP.I took this opportunity to review my router's other settings too. I'll be blogging about these over the next few posts.
Migrating to Predictable Network Interface Names
Ever since Linus decided to flip the network interface enumeration order in the Linux kernel, I had been relying on udev's persistent network interface rules to maintain some semblance of consistency in the NIC naming scheme of my hosts. It has never been a totally satisfactory method, since it required manually editing the file to list the MAC addresses of all Ethernet cards and WiFi dongles likely to appear on that host to consistently use an easy-to-remember name that I could adopt for ifupdown configuration files.
Enter predictable interface names. What started as a Linux kernel module project at Dell was eventually re-implemented in systemd. However, clear documentation on the naming scheme had been difficult to find and udev's persistent network interface rules gave me what I needed, so I postponed the transition for years. Relocating to a new flat and rethinking my home network to match gave me an opportunity to revisit the topic.
The naming scheme is surprisingly simple and logical, once proper explanations have been found. The short version:
#!/bin/sh lspci grep -i -e ethernet -e network sudo dmesg grep -i renamed for n in $(ls -X /sys/class/net/ grep -v lo); do echo $n: && udevadm test-builtin net_id /sys/class/net/$n 2>/dev/null grep NAME; sudo rgrep $n /etc sudo find /etc -name '*$n*' doneThis combined ideas found on the Debian wiki with a few of my own. Running the script before and after the migration ensured that I hadn't missed any configuration file. Once I was satisfied with that, I commented out the old udev persistent network interface rules, ran
dpkg-reconfigure on all my Linux kernel images to purge the rules from the initrd images, and called it a day.
... well, not quite. It turns out that with bridge-utils, bridge_ports all no longer works. One must manually list all interfaces to be bridged. Debian bug report filed.
PS: Luca Capello pointed out that Debian 10/Buster's Release Notes include migration instructions.
I've started struggling to see my mouse pointer on my desktop. I probably need
to take an eye test and possibly buy some new glasses. In the meantime, I've
changed my mouse pointer to something larger and more colourful: The Amiga
Workbench 1.3 mouse pointer that I grew up with, but scaled up to 128x128
pixels (from 16x16, I think)
See if you can spot it
sbuild, autopkgtest-build-qemu and vmdb2 to
provide a trivial interface for creating and using QEMU-based environments for
package building:
qemu-sbuild-create creates VM imagesqemu-sbuild-update is to VM images what sbuild-update is to chrootsqemu-sbuild wraps sbuild, adding all necessary (and some useful)
options for autopkgtest mode.$ sudo qemu-sbuild-create -o simple.img unstable http://deb.debian.org/debian $ qemu-sbuild --image simple.img -d unstable [sbuild-options] FOO.dscThat's it. Both
qemu-sbuild-create and qemu-sbuild automate certain things, but
also accept a number of options. For example, qemu-sbuild-create can
install additional packages either from one of the APT sources, or .deb
packages from the local filesystem.
qemu-sbuild will pass on every option it does not consume itself to
sbuild, so it should mostly work as a drop-in replacement for it (see the
Limitations section below for where it doesn't).
The created images can also be used for running autopkgtest itself, of
course.
qemu-sbuild-utils is not quite there yet.
amd64 architecture, for
building packages in amd64 and i386 VMs. There are plans to support
arm64 in the near future.
qemu-sbuild-create needs root, for the debootstrap stage. I'm looking into
ways around this (by extending vmdb2). In any case, image updating and
package building do not need privileges.
autopkgtest mode does not yet support interactivity, so one cannot drop
into a shell with --build-failed-commands, for example. The easy workaround
is to connect to the VM with SSH. For this, the image must contain the
openssh-server package.
qemubuilder, but had troubles getting it to work. In any case,
the autopkgtest chroot mode of sbuild seemed far more powerful and
useful to me.
vectis looks incredibly promising,
but I had already written qemu-sbuild-utils by the time I stumbled over it,
and as my current setup works well for me for now and is simple enough to
maintain, I decided to polish and publish it.
I'm also looking into Docker- and other namespace-based isolation solutions (of
which there are
many), which
I think are the way forward for the majority of packages (those that aren't
too close to the kernel and/or hardware).
Rather than relying on the kernel for isolation, KVM-based solutions like
Amazon's Firecracker
and QEMU's
microvm machine
type provide minimal VMs with almost no boot overhead. Firecracker, for
example, claims less than 125ms from launch to /sbin/init. Investigating
these is a medium-term project for me.
sbuild, autopkgtest, and vmdb2.
About a month worth of updates on KDE/Plasma in Debian has accumulated, so here we go. The highlights are: Plasma 5.19.5 based on Qt 5.15 is in Debian/experimental and hopefully soon in Debian/unstable, and my own builds at OBS have been updated to Plasma 5.20.2, Frameworks 5.75, Apps 20.08.2.
Thanks to the dedicated work of the Qt maintainers, Qt 5.15 has finally entered Debian/unstable and we can finally target Plasma 5.20.
OBS packages
The OBS packages as usual follow the latest release, and currently ship KDE Frameworks 5.75, KDE Apps 20.08.2, and new, Plasma 5.20.2. The package sources are as usual (note the different path for the Plasma packages and the App packages, containing the release version!), for Debian/unstable:
deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/frameworks/Debian_Unstable/ ./ deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/plasma520/Debian_Unstable/ ./ deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/apps2008/Debian_Unstable/ ./ deb https://download.opensuse.org/repositories/home:/npreining:/debian-kde:/other/Debian_Unstable/ ./
Testing instead of Unstable for Debian/testing.
The update to Plasma 5.20 took a bit of time, not only because of the wait for Qt 5.15, but also because I couldn t get it running on my desktop, only in the VM. It turned out that the Plasmoid Event Calendar needed an update, and the old version crashed Plasma ( v68 and below crash in Arch after the Qt 5.15.1 update. ). After I realized that, it was only a question of updating to get Plasma 5.20 running.
There are two points I have to mention (and I will fix sooner or later):
A couple of year ago, I moved into a new flat that comes with RJ45 sockets wired for 10 Gigabit (but currently offering 1 Gigabit) Ethernet.This also meant changing the settings on my router box for my new ISP.I took this opportunity to review my router's other settings too. I'll be blogging about these over the next few posts.
Adding IPv6 support to my home LAN
I have been following the evolution of IPv6 ever since the KAME project produced the first IPv6 implementation. I have also been keeping track of the IPv4 address depletion.
Around the time the IPv6 Day was organized in 2011, I started investigating the situation of IPv6 support at local ISPs.
Well, never mind all those rumors about Finland being some high-tech mecca. Back then, no ISP went beyond testing their routers for IPv6 compatibility and producing white papers on what their limited test deployments accomplished.
Not that it matters much, in practice. Most IPv6 documentation out there, including Debian's own, still focuses on configuring transitional mechanisms, especially how to connect to a public IPv6 tunnel broker.
Relocating to a new flat and rethinking my home network to match gave me an opportunity to revisit the topic. Much to my delight, my current ISP offers native IPv6.
This prompted me to go back and read up on IPv6 one more time. One important detail:
IPv6 hosts are globally reachable.
The implications of this don't immediately spring to mind for someone used to IPv4 network address translation (NAT):
Any network service running on an IPv6 host can be reached by anyone anywhere.
Contrary to IPv4, there is no division between private and public IP addresses. Whereas a device behind an IPv4 NAT essentially is shielded from the outside world, IPv6 breaks this assumption in more than one way. Not only is the host reachable from anywhere, its default IPv6 address is a mathematical conversion (EUI-64) of the network interface's MAC address, which makes every connection forensically traceable to a unique device.
Basically, if you hadn't given much thought to firewalls until now, IPv6 should give you enough goose bumps to get around it. Tightening the configuration of every network service is also an absolute must. For instance, I configured sshd to only listen to private IPv4 addresses.
What /etc/network/interfaces might look like on an dual-stack (IPv4 + IPv6) host:
allow-hotplug enp9s0 iface enp9s0 inet dhcp iface enp9s0 inet6 auto privext 2 dhcp 1The
auto method means that IPv6 will be auto-configured using SLAAC; privext 2 enables IPv6 privacy options and specifies that we prefer connecting via the randomly-generated IPv6 address, rather than the EUI-64 calculated MAC specific address; dhcp 1 enables passive DHCPv6 to fetch additional routing information.
The above works for most desktop and laptop configurations.
Where things got more complicated is on the router. I decided early on to keep NAT to provide an IPv4 route to the outside world. Now how exactly is IPv6 routing done? Every node along the line must have its own IPv6 address... including the router's LAN interface. This is accomplished using the sample script found in Debian's IPv6 prefix delegation wiki page. I modified mine as follows (the rest of the script is omitted for clarity):
#Both LAN interfaces on my private network are bridged via br0 IA_PD_IFACE="br0" IA_PD_SERVICES="dnsmasq" IA_PD_IPV6CALC="/usr/bin/ipv6calc"Just put the script at the suggested location. We'll need to request a prefix on the router's outside interface to utilize it. This gives us the following
interfaces file:
allow-hotplug enp2s4 enp2s8 enp2s9 auto br0 iface enp2s4 inet dhcp iface enp2s4 inet6 auto request_prefix 1 privext 2 dhcp 1 iface enp2s8 inet manual iface enp2s8 inet6 manual iface enp2s9 inet manual iface enp2s9 inet6 manual iface br0 inet static bridge_ports enp2s8 enp2s9 address 10.10.10.254 iface br0 inet6 manual bridge_ports enp2s8 enp2s9 # IPv6 from /etc/dhcp/dhclient-exit-hooks.d/prefix_delegationThe IPv4 NAT and IPv6 Bridge script on my router looks as follows:
#!/bin/sh PATH="/usr/sbin:/sbin:/usr/bin:/bin" wan=enp2s4 lan=br0 ######################################################################## # IPv4 NAT iptables -F; iptables -t nat -F; iptables -t mangle -F iptables -X; iptables -t nat -X; iptables -t mangle -X iptables -Z; iptables -t nat -Z; iptables -t mangle -Z iptables -t nat -A POSTROUTING -o $wan -j MASQUERADE echo 1 > /proc/sys/net/ipv4/ip_forward ######################################################################## # IPv6 bridge ip6tables -F; ip6tables -X; ip6tables -Z # Default policy DROP ip6tables -P FORWARD DROP # Allow ICMPv6 forwarding ip6tables -A FORWARD -p ipv6-icmp -j ACCEPT # Allow established connections ip6tables -I FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT # Accept packets FROM LAN to everywhere ip6tables -I FORWARD -i $lan -j ACCEPT echo 1 > /proc/sys/net/ipv6/conf/all/forwarding echo 1 > /proc/sys/net/ipv6/conf/default/forwarding # IPv6 propagation via /etc/dhcp/dhclient-exit-hooks.d/prefix_delegationThe above already provided enough IPv6 connectivity to pass the IPv6 test on my desktop inside the LAN. To make things more fun, I enabled DHCPv6 support for my LAN on the router's dnsmasq by adding the last 3 lines to the configuration:
dhcp-hostsfile=/etc/dnsmasq-ethersfile bind-interfaces interface=br0 except-interface=enp2s4 no-dhcp-interface=enp2s4 dhcp-range=tag:br0,10.10.10.0,static,infinite dhcp-range=tag:br0,::1,constructor:br0,ra-names,ra-stateless,infinite enable-ra dhcp-option=option6:dns-server,[::],[2606:4700:4700::1111],[2001:4860:4860::8888]The 5 first lines (included here for emphasis) are extremely important: they ensure that dnsmasq won't provide any IPv4 or IPv6 service to the outside interface (enp2s4) and that DHCP will only be provided for LAN hosts whose MAC address is known. Line 6 shows how dnsmasq's DHCP service syntax differs between IPv4 and IPv6. The rest of my configuration was omitted on purpose. Enabling native IPv6 on my LAN has been an interesting experiment. I'm sure that someone could come up with even better ip6tables rules for the router or for my desktop hosts. Feel free to mention them in the blog's comment.
A couple of years ago, I moved into a new flat that comes with RJ45 sockets wired for 10 Gigabit (but currently offering 1 Gigabit) Ethernet.This also meant changing the settings on my router box for my new ISP.I took this opportunity to review my router's other settings too. I'll be blogging about these over the next few posts.
GRUB fine-tuning
One thing that had been annoying me ever since Debian migrated to systemd as /sbin/init is that boot message verbosity hasn't been the same. Previously, the cmdline option quiet merely suppressed the kernel's output to the bootscreen, but left the daemon startup messages alone. Not anymore. Nowadays, quiet produces a blank screen.
After some googling, I found the solution to that:
GRUB_CMDLINE_LINUX_DEFAULT="noquiet loglevel=5"
The former restores daemon startup messages, while the later makes the kernel output only significant notices or more serious messages. On most of my hosts, it mostly reports inconsistencies in the ACPI configuration of the BIOS.
Another setting I find useful is a reboot delay in case a kernel panic happens:
GRUB_CMDLINE_LINUX="panic=15"
This gives me enough time to snap a picture of the screen output to attach to the bug report that will follow.
issoPackage = with pkgs.python3Packages; buildPythonPackage rec pname = "isso"; version = "custom"; src = pkgs.fetchFromGitHub # Use my fork owner = "vincentbernat"; repo = pname; rev = "vbe/master"; sha256 = "0vkkvjcvcjcdzdj73qig32hqgjly8n3ln2djzmhshc04i6g9z07j"; ; propagatedBuildInputs = [ itsdangerous jinja2 misaka html5lib werkzeug bleach flask-caching ]; buildInputs = [ cffi ]; checkInputs = [ nose ]; checkPhase = '' $ python.interpreter setup.py nosetests ''; ;
"$ issoEnv /bin/gunicorn", like with a virtual
environment.
issoEnv = pkgs.python3.buildEnv.override extraLibs = [ issoPackage pkgs.python3Packages.gunicorn pkgs.python3Packages.gevent ]; ;
issoConfig = pkgs.writeText "isso.conf" '' [general] dbpath = /db/comments.db host = https://vincent.bernat.ch http://localhost:8080 notify = smtp [ ] '';
Dockerfile:
issoDockerImage = pkgs.dockerTools.buildImage name = "isso"; tag = "latest"; extraCommands = '' mkdir -p db ''; config = Cmd = [ "$ issoEnv /bin/gunicorn" "--name" "isso" "--bind" "0.0.0.0:$ port " "--worker-class" "gevent" "--workers" "2" "--worker-tmp-dir" "/dev/shm" "--preload" "isso.run" ]; Env = [ "ISSO_SETTINGS=$ issoConfig " "SSL_CERT_FILE=$ pkgs.cacert /etc/ssl/certs/ca-bundle.crt" ]; ; ;
issoEnv derivation in config.Cmd, the
whole derivation, including Isso and Gunicorn, is copied inside the
Docker image. The same applies for issoConfig, the configuration
file we created earlier, and pkgs.cacert, the derivation containing
trusted root certificates. The resulting image is 171 MB once
installed, which is comparable to the Debian Buster image generated by
the official Dockerfile.
NixOS features an abstraction to run Docker containers. It is not
currently documented in NixOS manual but you can look at the source
code of the module for the available options. I choose to use
Podman instead of Docker as the backend because it does not
require running an additional daemon.
virtualisation.oci-containers = backend = "podman"; containers = isso = image = "isso"; imageFile = issoDockerImage; ports = ["127.0.0.1:$ port :$ port "]; volumes = [ "/var/db/isso:/db" ]; ; ; ;
$ systemctl status podman-isso.service podman-isso.service Loaded: loaded (/nix/store/a66gzqqwcdzbh99sz8zz5l5xl8r8ag7w-unit-> Active: active (running) since Sun 2020-11-01 16:04:16 UTC; 4min 44s ago Process: 14564 ExecStartPre=/nix/store/95zfn4vg4867gzxz1gw7nxayqcl> Main PID: 14697 (podman) IP: 0B in, 0B out Tasks: 10 (limit: 2313) Memory: 221.3M CPU: 10.058s CGroup: /system.slice/podman-isso.service 14697 /nix/store/pn52xgn1wb2vr2kirq3xj8ij0rys35mf-podma> 14802 /nix/store/7vsba54k6ag4cfsfp95rvjzqf6rab865-conmo> nov. 01 16:04:17 web03 podman[14697]: container init (image=localhost/isso:latest) nov. 01 16:04:17 web03 podman[14697]: container start (image=localhost/isso:latest) nov. 01 16:04:17 web03 podman[14697]: container attach (image=localhost/isso:latest) nov. 01 16:04:19 web03 conmon[14802]: INFO: connected to SMTP server nov. 01 16:04:19 web03 conmon[14802]: INFO: connected to https://vincent.bernat.ch nov. 01 16:04:19 web03 conmon[14802]: [INFO] Starting gunicorn 20.0.4 nov. 01 16:04:19 web03 conmon[14802]: [INFO] Listening at: http://0.0.0.0:8080 (1) nov. 01 16:04:19 web03 conmon[14802]: [INFO] Using worker: gevent nov. 01 16:04:19 web03 conmon[14802]: [INFO] Booting worker with pid: 8 nov. 01 16:04:19 web03 conmon[14802]: [INFO] Booting worker with pid: 9
comments.luffy.cx to the container. NixOS provides a simple
integration to grab a Let s Encrypt certificate.
services.nginx.virtualHosts."comments.luffy.cx" = root = "/data/webserver/comments.luffy.cx"; enableACME = true; forceSSL = true; extraConfig = '' access_log /var/log/nginx/comments.luffy.cx.log anonymous; ''; locations."/" = proxyPass = "http://127.0.0.1:$ port "; extraConfig = '' proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_hide_header Set-Cookie; proxy_hide_header X-Set-Cookie; proxy_ignore_headers Set-Cookie; ''; ; ; security.acme.certs."comments.luffy.cx" = email = lib.concatStringsSep "@" [ "letsencrypt" "vincent.bernat.ch" ]; ;
phpmyadmin, php-bacon-baconqrcode, twig, php-dasprid-enum, sql-parser, and mariadb-mysql-kbs for William.
:wq for today.
Here is my monthly update covering what I have been doing in the free software world during October 2020 (previous month):
ora2pg is a tool used to migrate an Oracle database to PostgreSQL. This month, I submitted a patch to make it build reproducibly. [...]
declares-possibly-conflicting-debhelper-compat-versions tag as we may specify the Debhelper compatibility level in debian/rules or debian/control (#972464) and dropped a reference to missing manual page [...].
perl: Please make the build mostly reproducible. (#972559)
fckit: Please make the build (mostly) reproducible. (#972378)
libgrokj2k: Please make the documentation reproducible. (#972494)
netcdf-parallel: Please make the settings file reproducible. (#972930)
dh-fortran-mod: Please make the output reproducible
version graph. (#965255)
emacs package to make the generated .el files reproducible, a regression that is causing many packages to become unreproducible. (#972861)
dpkg-buildflags to enable reproducible=+fixfilepath by default.
dettrace [...], and added yet another supply-chain security attack publication [...].relative_url to fix missing translation icon on various pages. [...]161 to Debian:
test_ocaml to the assert_diff helper. [...]20.8b1. (#972518)--help-only test as being a 'superficial' test. (#971506)try.diffoscope.org service. [...]debhelper compatibility level to 13 [...] and bump Standards-Version to 4.5.0 [...].jackson-databind, a Java library for processing JSON, to address an external entity expansion vulnerability.
tomcat8, the Java application server. This was to fix an issue where an excessive number of concurrent streams could have resulted in users seeing responses for unexpected resources.
3.1.2-1) New upstream bugfix release.
6.0.8-2 Apply a patch from Yossi Gottlieb to fix a crash when reporting RDB/AOF file errors. (#972683)6.0.9-1 New upstream release.1.6.8+dfsg-1) New upstream release)
4.0.25-1) New upstream release, where parsing configuration file now works correctly with Turkish locale. (#972387)
2.0-1) New upstream release.
20.8b1-2) Non-maintainer upload to correct version handling to avoid a ModuleNotFoundError error which was affecting a number of related packages. (#970901)
We all know these situations when we receive an email asking Can you
check the design of X, I need a reply by tonight. Or an instant
message: My website went down, can you check? Another email: I
canceled a plan at the hosting company, can you restore my website as
fast as possible? A phone call: The TLS certificate didn t get
updated, and now we can t access service Y. Yet another email: Our
super important medical advice website is suddenly being censored in
country Z, can you help?
Everyone knows those messages that have URGENT in capital letters in
the email subject. It might be that some of them really are urgent.
Others are the written signs of someone having a hard time properly
planning their own work and passing their delays on to someone who comes
later in the creation or production chain. And others again come from
people who are overworked and try to delegate some of their tasks to a
friendly soul who is likely to help.
when an Asker meets a Guesser, unpleasantness results. An Asker won t think it s rude to request two weeks in your spare room, but a Guess culture person will hear it as presumptuous and resent the agony involved in saying no. Your boss, asking for a project to be finished early, may be an overdemanding boor or just an Asker, who s assuming you might decline. If you re a Guesser, you ll hear it as an expectation.Askers should also be aware that there might be Guessers in their team. It can help to define clear guidelines about making requests (when do I expect an answer, under which budget/contract/responsibility does the request fall, what other task can be put aside to handle the urgent task?) Last, but not least, Making work visible has a lot of other proposals on how to visibilize and then deal with unplanned work.
Next.