and to use it you evaluate this at the REPL:
(defhost foo.silentflame.com (:deploy ((:ssh :user "root") :sbcl)) (os:debian-stable "buster" :amd64) ;; Hetzner's Debian 10 image comes with a three-partition layout and boots ;; with traditional BIOS. (disk:has-volumes (physical-disk :device-file "/dev/sda" :boots-with '(grub:grub :target "i386-pc"))) (on-change (installer:cleanly-installed-once nil ;; This is a specification of the OS Hetzner's image has, so ;; Consfigurator knows how to install SBCL and debootstrap(8). ;; In this case it's the same Debian release as the replacement. '(os:debian-stable "buster" :amd64)) ;; Clear out the old OS's EFI system partition contents, in case we can ;; switch to booting with EFI at some point (if we wanted we could specify ;; an additional x86_64-efi target above, and grub-install would get run ;; to repopulate /boot/efi, but I don't think Hetzner can boot from it yet). (file:directory-does-not-exist "/boot/efi/EFI") (apt:installed "linux-image-amd64") (installer:bootloaders-installed) (fstab:entries-for-volumes (disk:volumes (mounted-ext4-filesystem :mount-point "/") (partition (mounted-fat32-filesystem :mount-options '("umask=0077") :mount-point "/boot/efi")))) (file:lacks-lines "/etc/fstab" "# UNCONFIGURED FSTAB FOR BASE SYSTEM") (file:is-copy-of "/etc/resolv.conf" "/old-os/etc/resolv.conf") (mount:unmounted-below-and-removed "/old-os")) (apt:mirror "http://ftp.de.debian.org/debian") (apt:no-pdiffs) (apt:standard-sources.list) (sshd:installed) (as "root" (ssh:authorized-keys +spwsshkey+)) (sshd:no-passwords) (timezone:configured "Etc/UTC") (swap:has-swap-file "2G") (network:clean-/etc/network/interfaces) (network:static "enp1s0" "xxx.xxx.xxx.xxx" "xxx.xxx.1.1" "255.255.255.255"))
CONSFIG> (deploy ((:ssh :user "root" :hop "xxx.xxx.xxx.xxx") :sbcl) foo.silentflame.com)
:HOPparameter specifies the IP address of the new machine, as DNS hasn t been updated yet. Consfigurator installs SBCL and debootstrap(8), prepares a minimal system, replaces the contents of
/, gets to work applying the other properties, and then reboots. This gets us a properly populated fstab:
(slightly doctored for more readable alignment) There s ordering logic so that the swapfile will end up after whatever filesystem contains it; a UUID is used for ext4 filesystems, but for fat32 filesystems, to be safe, a PARTUUID is used. The application of
UUID=... / ext4 relatime 0 1 PARTUUID=... /boot/efi vfat umask=0077 0 2 /var/lib/swapfile swap swap defaults 0 0
(INSTALLER:BOOTLOADERS-INSTALLED)handles calling both
grub-install(8), relying on the metadata specified about
/dev/sda. Next time we execute Consfigurator against the machine, it ll ignore all the property applications attached to the application of
ON-CHANGE, and just apply everything following that block. There are a few things I don t have good solutions for. When you boot Hetzner s image the primary network interface is
eth0, but then for a freshly debootstrapped Debian you get
enp1s0, and I haven t got a good way of knowing what it ll be (if you know it ll have the same name, you can use
(NETWORK:PRESERVE-STATIC-ONCE)to create a file in /etc/network/interfaces.d based on the current default route and corresponding interface). Another tricky thing is SSH host keys. It s easy to use Consfigurator to add host keys to your laptop s
~/.ssh/known_hosts, but in this case the host key changes back and forth from whatever the Hetzner image has and the newly generated key you get afterwards. One option might be to copy the old host keys out of
/old-osbefore it gets deleted, like how
/etc/resolv.confis copied. This work is based on Propellor s equivalent functionality. I think my approach to handling /etc/fstab and bootloader installation is an improvement on what Joey does.
BLOG_RSS_LIMITin my Django Mezzanine blog to a thousand (to export all posts via RSS/ATOM feed), fixing some bugs in the import_feed plugin, waiting a few minutes for the full feed to generate and to be imported, adjusting the config of the resulting site, posting that to git and writing a simple shell script to pull that repo periodically and call
nikola buildon it, as well as config to serve ther result via ngnix. Done. After that creating a new blog post is just
nikola new_postand editing it in vim and pushing to git. I prefer Markdown, but it supports all kinds of formats. And the old posts are just stored as HTML. Really simple. I think I will spend more time fighting with Google to allow me to forward email from my domain to my GMail postbox without it refusing all of it as spam.
$_vbe_prompt_compactset to 1 when we want a compact prompt. We use the following function to define the prompt appearance:
_vbe_prompt () local retval=$? # When compact, just time + prompt sign if (( $_vbe_prompt_compact )); then # Current time (with timezone for remote hosts) _vbe_prompt_segment cyan default "%D %H:%M$ SSH_TTY+ %Z " # Hostname for remote hosts [[ $SSH_TTY ]] && \ _vbe_prompt_segment black magenta "%B%M%b" # Status of the last command if (( $retval )); then _vbe_prompt_segment red default $ PRCH[reta] else _vbe_prompt_segment green cyan $ PRCH[ok] fi # End of prompt _vbe_prompt_end return fi # Regular prompt with many information # [ ] setopt prompt_subst PS1='$(_vbe_prompt) '
Update (2021.05) The following part has been rewritten to be more robust. The code is stolen from Powerlevel10k s issue #888. See the comments for more details.
_vbe-zle-line-init() [[ $CONTEXT == start ]] return 0 # Start regular line editor (( $+zle_bracketed_paste )) && print -r -n - $zle_bracketed_paste zle .recursive-edit local -i ret=$? (( $+zle_bracketed_paste )) && print -r -n - $zle_bracketed_paste # If we received EOT, we exit the shell if [[ $ret == 0 && $KEYS == $'\4' ]]; then _vbe_prompt_compact=1 zle .reset-prompt exit fi # Line edition is over. Shorten the current prompt. _vbe_prompt_compact=1 zle .reset-prompt unset _vbe_prompt_compact if (( ret )); then # Ctrl-C zle .send-break else # Enter zle .accept-line fi return ret zle -N zle-line-init _vbe-zle-line-init
bind-key -T copy-mode M-w \ send -X copy-pipe-and-cancel "sed 's/ .* /%/g' xclip -i -selection clipboard" \;\ display-message "Selection saved to clipboard!"
14:21 % ssh eizo.luffy.cx Linux eizo 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 Last login: Fri Apr 23 14:20:39 2021 from 2a01:cb00:3f:b02:9db6:efa4:d85:7f9f 14:21 CEST % uname -a Linux eizo 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux 14:21 CEST % Connection to eizo.luffy.cx closed. 14:22 % git status On branch article/zsh-transient Untracked files: (use "git add <file>..." to include in what will be committed) ../../firstname.lastname@example.org nothing added to commit but untracked files present (use "git add" to track)
dratrelease (which can always be had easily too via
install.packages("RcppArmadillo", repos="https://RcppCore.github.io/drat").) The full set of changes follows. We include the aforementioned interim release as well.
Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can sponsor me at GitHub.
Changes in RcppArmadillo version 0.10.5.0 (2021-05-21)
- Upgraded to Armadillo release 10.5 (Antipodean Fortress)
- expanded the standalone
clamp()function to handle complex values
- more efficient use of OpenMP
- vector, matrix and cube constructors now initialise elements to zero by default; use the
mat X(4,5,fill::none), to disable element initialisation
codecov.ymlto exclude Armadillo from coverage analysis
Changes in RcppArmadillo version 0.10.4.1.0 (2021-04-23)
- Upgraded to Armadillo release 10.4.1 (Pressure Cooker)
- GitHub-only release
Wow, what a mess! Let's see if I can make sense of this:
remotes/private/attic/novena 822ca2bb add letter i sent to novena, never published remotes/private/attic/secureboot de09d82b quick review, add note and graph remotes/private/attic/wireguard 5c5340d1 wireguard review, tutorial and comparison with alternatives remotes/private/backlog/dat 914c5edf Merge branch 'master' into backlog/dat remotes/private/backlog/packet 9b2c6d1a ham radio packet innovations and primer remotes/private/backlog/performance-tweaks dcf02676 config notes for http2 remotes/private/backlog/serverless 9fce6484 postponed until kubecon europe remotes/private/fin/cost-of-hosting 00d8e499 cost-of-hosting article online remotes/private/fin/kubecon f4fd7df2 remove published or spun off articles remotes/private/fin/kubecon-overview 21fae984 publish kubecon overview article remotes/private/fin/kubecon2018 1edc5ec8 add series remotes/private/fin/netconf 3f4b7ece publish the netconf articles remotes/private/fin/netdev 6ee66559 publish articles from netdev 2.2 remotes/private/fin/pgp-offline f841deed pgp offline branch ready for publication remotes/private/fin/primes c7e5b912 publish the ROCA paper remotes/private/fin/runtimes 4bee1d70 prepare publication of runtimes articles remotes/private/fin/token-benchmarks 5a363992 regenerate timestamp automatically remotes/private/ideas/astropy 95d53152 astropy or python in astronomy remotes/private/ideas/avaneya 20a6d149 crowdfunded blade-runner-themed GPLv3 simcity-like simulator remotes/private/ideas/backups-benchmarks fe2f1f13 review of backup software through performance and features remotes/private/ideas/cumin 7bed3945 review of the cumin automation tool from WM foundation remotes/private/ideas/future-of-distros d086ca0d modern packaging problems and complex apps remotes/private/ideas/on-dying a92ad23f another dying thing remotes/private/ideas/openpgp-discovery 8f2782f0 openpgp discovery mechanisms (WKD, etc), thanks to jonas meurer remotes/private/ideas/password-bench 451602c0 bruteforce estimates for various password patterns compared with RSA key sizes remotes/private/ideas/prometheus-openmetrics 2568dbd6 openmetrics standardizing prom metrics enpoints remotes/private/ideas/telling-time f3c24a53 another way of telling time remotes/private/ideas/wallabako 4f44c5da talk about wallabako, read-it-later + kobo hacking remotes/private/stalled/bench-bench-bench 8cef0504 benchmarking http benchmarking tools remotes/private/stalled/debian-survey-democracy 909bdc98 free software surveys and debian democracy, volunteer vs paid work
novena: the project is ooold now, didn't seem to fit a LWN article. it was basically "how can i build my novena now" and "you guys rock!" it seems like the MNT Reform is the brain child of the Novena now, and I dare say it's even cooler!
secureboot: my LWN editors were critical of my approach, and probably rightly so - it's a really complex subject and I was probably out of my depth... it's also out of date now, we did manage secureboot in Debian
wireguard: LWN ended up writing extensive coverage, and I was biased against Donenfeld because of conflicts in a previous project
dat: I already had written Sharing and archiving data sets with Dat, but it seems I had more to say... mostly performance issues, beaker, no streaming, limited adoption... to be investigated, I guess?
packet: a primer on data communications over ham radio, and the cool new tech that has emerged in the free software world. those are mainly notes about Pat, Direwolf, APRS and so on... just never got around to making sense of it or really using the tech...
performance-tweaks: "optimizing websites at the age of http2", the unwritten story of the optimization of this website with HTTP/2 and friends
serverless: god. one of the leftover topics at Kubecon, my notes on this were thin, and the actual subject, possibly even thinner... the only lie worse than the cloud is that there's no server at all! concretely, that's a pile of notes about Kubecon which I wanted to sort through. Probably belongs in the attic now.
astropy: "Python in astronomy" - had a chat with saimn while writing about sigal, and it turns out he actually works on free software in astronomy, in Python... I actually expect LWN to cover this sooner than later, after Lee Phillips's introduction to SciPy
avaneya: crowdfunded blade-runner-themed GPLv3 simcity-like simulator, i just have that link so far
backups-benchmarks: review of backup software through performance and features, possibly based on those benchmarks, maybe based on this list from restic although they refused casync. benchmark articles are hard though, especially when you want to "cover them all"... I did write a silly Attic vs Bup back when those programs existed (2014), in a related note...
ideas/cumin: review of the Cumin automation tool from WikiMedia Foundation... I ended up using the tool at work and writing service documentation for it
ideas/future-of-distros: modern packaging problems and complex apps, starting from this discussion about the removal of Dolibarr from Debian, a summary of the thread from liw, and ideas from joeyh (now from the outside of Debian), then debates over the power of FTP masters - ugh, glad I didn't step in that rat's nest
ideas/on-dying: "what happens when a hacker dies?" rather grim subject, but a more and more important one... joeyh has ideas again, phk as well, then there's a protocol for dying (really grim)... then there are site policies like GitHub, Facebook, etc... more in the branch, but that one I can't help but think about now that family has taken a bigger place in my life...
ideas/openpgp-discovery: OpenPGP discovery mechanisms (WKD, etc), suggested by Jonas Meurer (somewhere?), only links to Mailveloppe, LEAP, WKD (or is it WKS?), another standard, probably would need to talk about OpenPGP CA now and how Debian and Tor manage their keyrings... pain in the back.
ideas/password-bench: bruteforce estimates for various password patterns compared with RSA key sizes, spinoff of my smartcard article, in the crypto-bench, look at this shiny graph, surely that must mean an article, right?
ideas/prometheus-openmetrics: "Evolving the Prometheus exposition format into a standard", seems like this happened
ideas/telling-time: telling time to users is hard. xclock vs ttyclock, etc. maybe gameclock and undertime as well? syncing time is hard, but it turns out showing it is non trivial as well... basically turning this bug report into an article. for some reason I linked to this meme, derived from this meme, presumably a premonition of my stupid idea of writing undertime TIMEZONES!
ideas/wallabako: "talk about wallabako, read-it-later + kobo hacking", that's it, not even a link to the project!
stalled/bench-bench-benchbenchmarking http benchmarking tools, a horrible mess of links, copy-paste from terminals, and ideas about benchmarking... some of this trickled out into this benchmarking guide at Tor, but not much more than the list of tools
stalled/debian-survey-democracy: "free software surveys and Debian democracy, volunteer vs paid work"... A long standing concern of mine is that all Debian work is supposed to be volunteer, and paying explicitly for work inside Debian has traditionally been frowned upon, even leading to serious drama and dissent (remember Dunc-Tank)? back when I was writing for LWN, I was also doing paid work for Debian LTS. I also learned that a lot (most?) Debian Developers were actually being paid by their job to work on Debian. So I was confused by this apparent contradiction, especially given how the LTS project has been mostly accepted, while Dunc-Tank was not... See also this talk at Debconf 16. I had hopes that this study would show the "hunch" people have offered (that most DDs are paid to work on Debian) but it seems to show the reverse (only 36% of DDs, and 18% of all respondents paid). So I am still confused and worried about the sustainability of Debian.
sudo raspi-config, I changed the following:
apt install unattended-upgrades anacron echo 'Unattended-Upgrade::Origins-Pattern "origin=Debian,codename=$ distro_codename ,label=Debian"; "origin=Debian,codename=$ distro_codename ,label=Debian-Security"; "origin=Raspbian,codename=$ distro_codename ,label=Raspbian"; "origin=Raspberry Pi Foundation,codename=$ distro_codename ,label=Raspberry Pi Foundation"; ;' sudo tee /etc/apt/apt.conf.d/51unattended-upgrades-raspbian
sshin the boot partition. Plug it into your router and boot it up. Check the IP that it received by looking at the active DHCP leases in your router's admin panel. Then login:
using the default password of
ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no email@example.com
and changed the
adduser francois addgroup sshuser adduser francois sshuser adduser francois sudo
piuser password to a random one:
before removing its admin permissions:
pwgen -sy 32 sudo passwd pi
Finally, I enabled the Uncomplicated Firewall by installing its package:
deluser pi adm deluser pi sudo deluser pi dialout deluser pi cdrom deluser pi lpadmin
and only allowing ssh connections. After starting ufw using
apt install ufw
systemctl start ufw.service, you can check that it's configured as expected using
ufw status. It should display the following:
Status: active To Action From -- ------ ---- 22/tcp ALLOW Anywhere 22/tcp (v6) ALLOW Anywhere (v6)
To make it start at boot/login, while still being able to exit and use other apps if needed:
apt install kodi
cp /etc/xdg/lxsession/LXDE-pi/autostart ~/.config/lxsession/LXDE-pi/ echo "@kodi" >> ~/.config/lxsession/LXDE-pi/autostart
/etc/hostsfile on your NFS server:
Install the NFS server package:
Setup the directories to share in
apt instal nfs-kernel-server
Open the right ports on your firewall by putting this in
/pub/movies pi(ro,insecure,all_squash,subtree_check) /pub/tv_shows pi(ro,insecure,all_squash,subtree_check)
Finally, apply all of these changes:
-A INPUT -s 192.168.1.3 -p udp -j ACCEPT -A INPUT -s 192.168.1.0/24 -p tcp --dport 111 -j ACCEPT -A INPUT -s 192.168.1.0/24 -p udp --dport 111 -j ACCEPT -A INPUT -s 192.168.1.0/24 -p udp --dport 123 -j ACCEPT -A INPUT -s 192.168.1.0/24 -p tcp --dport 600:1124 -j ACCEPT -A INPUT -s 192.168.1.0/24 -p udp --dport 600:1124 -j ACCEPT -A INPUT -s 192.168.1.0/24 -p tcp --dport 2049 -j ACCEPT -A INPUT -s 192.168.1.0/24 -p udp --dport 2049 -j ACCEPT
On the Pi, put the server's static IP in
iptables-apply systemctl restart nfs-kernel-server.service
and this in
Then create the mount points and mount everything:
fileserver:/data/movies /kodi/movies nfs ro,bg,hard,noatime,async,nolock 0 0 fileserver:/data/tv /kodi/tv nfs ro,bg,hard,noatime,async,nolock 0 0
mkdir -p /kodi/movies mkdir /kodi/tv mount /kodi/movies mount /kodi/tv
gitfrom their gemspec.
gitfrom gemspec, fixing the above issue, heh.
Welcome to the October 2020 report from the Reproducible Builds project. In our monthly reports, we outline the major things that we have been up to over the past month. As a brief reminder, the motivation behind the Reproducible Builds effort is to ensure flaws have not been introduced in the binaries we install on our systems. If you are interested in contributing to the project, please visit our main website.
The previous year has seen great progress in Arch Linux to get reproducible builds in the hands of the users and developers. In this talk we will explore the current tooling that allows users to reproduce packages, the rebuilder software that has been written to check packages and the current issues in this space.During the Reproducible Builds summit in Marrakesh in 2019, developers from the GNU Guix, NixOS and Debian distributions were able to produce a bit-for-bit identical GNU Mes binary despite using three different versions of GCC. Since this summit, additional work resulted in a bit-for-bit identical Mes binary using
tcc, and last month a fuller update was posted to this effect by the individuals involved. This month, however, David Wheeler updated his extensive page on Fully Countering Trusting Trust through Diverse Double-Compiling, remarking that:
GNU Mes rebuild is definitely an application of [Diverse Double-Compiling]. [..] This is an awesome application of DDC, and I believe it s the first publicly acknowledged use of DDC on a binaryThere was a small, followup discussion on our mailing list. In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update. This month, the Reproducible Builds project restarted our IRC meetings, managing to convene twice: the first time on October 12th (summary & logs), and later on the 26th (logs). As mentioned in previous reports, due to the unprecedented events throughout 2020, there will be no in-person summit event this year. On our mailing list this month El as Alejandro posted a request for help with a local configuration
reproducible=+fixfilepathDebian build flag by default. Enabling this
fixfilepathfeature will likely fix reproducibility issues in an estimated 500-700 packages. However, this month Vagrant Cascadian posted to the debian-devel mailing list:
It would be great to see theDebian Developer Stuart Prescott has been improving
reproducible=+fixfilepathfeature enabled by default in
dpkg-buildflags, and we would like to proceed forward with this soon unless we hear any major concerns or other outstanding issues. [ ] We would like to move forward with this change soon, so please raise any concerns or issues not covered already.
python-debian, a Python library that is used to parse Debian-specific files such as changelogs,
.dscs, etc. In particular, Stuart is working on adding support for
.buildinfofiles used for recording reproducibility-related build metadata:
This can mostly be a very thin layer around the existingA total of 159 Debian packages were categorised, 69 had their categorisation updated, and 33 had their classification removed this month, adding to our knowledge about identified issues. As part of this, Chris Lamb identified and classified two new issues:
Deb822types, using the existing
Changescode for the file listings, the existing
PkgRelationscode for the package listing and
gpg_*functions for signature handling.
go(version 1.15.3 has improved reproducibility over 1.14)
goxel(sort SCons-related filesystem ordering issue)
lal(rework an old date-related patch)
libsemigroups(build failure in single-CPU mode)
memcached(build failure in 2025 due to expired SSL certificate)
octant(SUSE-specific date issue)
openmpi4(date-related problem, revive old patch)
sbcl(datetime and hostname issue)
selinux-policy/policycoreutils(date-related issue in timezone)
161to Debian (later backported by Mattia Rizzolo), as well as made the following changes:
assert_diffhelper. [ ]
radare2to ensure our test pipelines continue to work [ ], and for the GNU Guix distribution Vagrant Cascadian diffoscope to version 161 [ ]. In related development, trydiffoscope is the web-based version of diffoscope. This month, Chris Lamb made the following changes:
--help-only test as being a superficial test. (#971506)
try.diffoscope.orgservice. [ ]
debhelpercompatibility level to 13 [ ] and bump
Standards-Versionto 4.5.0 [ ].
0.5.10-2was uploaded to Debian unstable by Holger Levsen, which enabled security hardening via
DEB_BUILD_MAINT_OPTIONS[ ] and dropped
dettrace[ ], and added yet another supply-chain security attack publication [ ].
relative_urlto fix missing translation icon on various pages. [ ]
tests.reproducible-builds.org. This month, Holger Levsen made the following changes:
ath97. [ ]
sudocommand if we are not actually running
libvirt. [ ]
One hiccup we ve encountered in SecureDrop development is that not all Python wheels can be built reproducibly. We ship multiple (Python) projects in Debian packages, with Python dependencies included in those packages as wheels. In order for our Debian packages to be reproducible, we need that wheel build process to also be reproducibleParallel to this, transparencylog.com was also launched, a service that verifies the contents of URLs against a publicly recorded cryptographic log. It keeps an append-only log of the cryptographic digests of all URLs it has seen. (GitHub repo) On 18th September, Bernhard M. Wiedemann will give a presentation in German, titled Wie reproducible builds Software sicherer machen ( How reproducible builds make software more secure ) at the Internet Security Digital Days 2020 conference.
ftp.debian.orgare made from their claimed sources. It also served as a general update on the status of reproducible builds within Debian. The video (145 MB) and slides are available. There were also a number of other talks that involved Reproducible Builds too. For example, the Malayalam language mini-conference had a talk titled , ? ( I want to join Debian, what should I do? ) presented by Praveen Arimbrathodiyil, the Clojure Packaging Team BoF session led by Elana Hashman, as well as Where is Salsa CI right now? that was on the topic of Salsa, the collaborative development server that Debian uses to provide the necessary tools for package maintainers, packaging teams and so on. Jonathan Bustillos (Jathan) also gave a talk in Spanish titled Un camino verificable desde el origen hasta el binario ( A verifiable path from source to binary ). (Video, 88MB)
openwrt-develmailing list asking for clarification on when to raise the
PKG_RELEASEidentifier of a package. This is needed in order to successfully perform rebuilds in a reproducible builds context. In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update. Chris Lamb provided some comments and pointers on an upstream issue regarding the reproducibility of a Snap / SquashFS archive file. [ ]
.buildinfobuild certificates have been tainted on the official Debian build servers, as these environments have files underneath the
/usr/local/sbindirectory [ ]. He also filed against bug for
debrebuildafter spotting that it can fail to download packages from
snapshot.debian.org[ ]. This month, several issues were uncovered (or assisted) due to the efforts of reproducible builds. For instance, Debian bug #968710 was filed by Simon McVittie, which describes a problem with detached debug symbol files (required to generate a traceback) that is unlikely to have been discovered without reproducible builds. In addition, Jelmer Vernooij called attention that the new Debian Janitor tool is using the property of reproducibility (as well as diffoscope when applying archive-wide changes to Debian:
New merge proposals also include a link to the diffoscope diff between a vanilla build and the build with changes. Unfortunately these can be a bit noisy for packages that are not reproducible yet, due to the difference in build environment between the two builds. [ ]56 reviews of Debian packages were added, 38 were updated and 24 were removed this month adding to our knowledge about identified issues. Specifically, Chris Lamb added and categorised the
lessc_nondeterministic_keystoolchain issues. [ ][ ] Holger Levsen sponsored Lukas Puehringer s upload of the python-securesystemslib pacage, which is a dependency of in-toto, a framework to secure the integrity of software supply chains. [ ] Lastly, Chris Lamb further refined his merge request against the
debian-installercomponent to allow all arguments from
sources.listfiles (such as
[check-valid-until=no]) in order that we can test the reproducibility of the installer images on the Reproducible Builds own testing infrastructure and sent a ping to the team that maintains that code.
getfem(embeds datetime and user, submitted via email)
getdp(hostname and user)
httpcomponents-client(Java documentation generator
lal(date and time issue, submitted via email)
OBS(discuss how to track old build
prjconfmetadata in buildinfo)
openblas(disable CPU detection)
python-eventlet(fails to build far in the future)
rna-star(date and hostname)
xz/b4(workaround CPU count influencing output, reported upstream)
<!ENTITY>declarations inside the Document Type Definition (DTD), or when a DTD or entity references an external resource. (#212)
pgpdump(1)can successfully parse some binary files, so check that the parsed output contains something sensible before accepting it. [ ]
gnumericfrom the Debian build-dependencies as it has been removed from the testing distribution. (#968742)
fallback_recognisesto prevent matching
.xsbbinary XML files.
ppudumpversion does not match our file header. [ ]
repr(object)output in Calling external command messages. [ ]
ppudumpversion 3.2.0 or higher. [ ]
setup.pythat diffoscope works with Python version 3.8 [ ] and Frazer Clews applied some Pylint suggestions [ ] and removed some deprecated methods [ ].
SOURCE_DATE_EPOCHage. [ ]
tests.reproducible-builds.org. This month, Holger Levsen made the following changes:
arm64architecture. [ ]
armhf. [ ][ ][ ]
buildinfos.debian.net, etc.). [ ][ ][ ][ ][ ]
arm64architecture anymore. [ ]
If you think you know how to spread the word about reproducibility in the context of Bitcoin wallets through WalletScrutiny, your contributions are highly welcome on this PR [ ]Julien Lepiller posted to the list linking to a blog post by Tavis Ormandy titled You don t need reproducible builds. Morten Linderud (foxboron) responded with a clear rebuttal that Tavis was only considering the narrow use-case of proprietary vendors and closed-source software. He additionally noted that the criticism that reproducible builds cannot prevent against backdoors being deliberately introduced into the upstream source ( bugdoors ) are decidedly (and deliberately) outside the scope of reproducible builds to begin with. Chris Lamb included the Reproducible Builds mailing list in a wider discussion regarding a tentative proposal to include
.debpackages, adding his remarks regarding requiring a custom tool in order to determine whether generated build artifacts are identical in a reproducible context. [ ] Jonathan Bustillos (Jathan) posted a quick email to the list requesting whether there was a list of To do tasks in Reproducible Builds. Lastly, Chris Lamb responded at length to a query regarding the status of reproducible builds for Debian ISO or installation images. He noted that most of the technical work has been performed but there are at least four issues until they can be generally advertised as such . He pointed that the privacy-oriented Tails operation system, which is based directly on Debian, has had reproducible builds for a number of years now. [ ]
Dealing with the void during MiniDebConf Online #1 Between 28 and 31 May this year, we set out to create our first ever online MiniDebConf for Debian. Many people have been meaning to do something similar for a long time, but it just didn t work out yet. With many of us being in lock down due to COVID-19, and with the strong possibility looming that DebConf20 might have had to become an online event, we rushed towards organising the first ever Online MiniDebConf and put together some form of usable video stack for it. I could go into all kinds of details on the above, but this post is about a bug that lead to a pretty nifty feature for DebConf20. The tool that we use to capture Jitsi calls is called Jibri (Jitsi Broadcasting Infrustructure). It had a bug (well, bug for us, but it s an upstream feature) where Jibri would hang up after 30s of complete silence, because it would assume that the call has ended and that the worker can be freed up again. This would result in the stream being ended at the end of every talk, so before the next talk, someone would have to remember to press play again in their media player or on the video player on the stream page. Hrmph. Easy solution on the morning that the conference starts? I was testing a Debian Live image the night before in a KVM and thought that I might as well just start a Jitsi call from there and keep a steady stream of silence so that Jibri doesn t hang up. It worked! But the black screen and silence on stream was a bit eery. Because this event was so experimental in nature, and because we were on such an incredibly tight timeline, we opted not to seek sponsors for this event, so there was no sponsors loop that we d usually stream during a DebConf event. Then I thought Ah! I could just show the schedule! . The stream looked bright and colourful (and was even useful!) and Jitsi/Jibri didn t die. I thought my work was done. As usual, little did I know how untrue that was. The silence was slightly disturbing after the talks, and people asked for some music. Playing music on my VM and capturing the desktop audio in to Jitsi was just a few pulseaudio settings away, so I spent two minutes finding some freely licensed tracks that sounded ok enough to just start playing on the stream. I came across mini-albums by Captive Portal and Cinema Noir, During the course of the MiniDebConf Online I even started enjoying those. Someone also pointed out that it would be really nice to have a UTC clock on the stream. I couldn t find a nice clock in a hurry so I just added a tmux clock in the meantime while we deal with the real-time torrent of issues that usually happens when organising events like this. Speaking of issues, during our very first talk of the last day, our speaker had a power cut during the talk and abruptly dropped off. Oops! So, since I had a screenshare open from the VM to the stream, I thought I d just pop in a quick message in a text editor to let people know that we re aware of it and trying to figure out what s going on. In the end, MiniDebConf Online worked out all right. Besides the power cut for our one speaker, and another who had a laptop that was way too under-powered to deal with video, everything worked out very well. Even the issues we had weren t show-stoppers and we managed to work around them.
DebConf20 Moves Online For DebConf, we usually show a sponsors loop in between sessions. It s great that we give our sponsors visibility here, but in reality people see the sponsors loop and think Talk over! and then they look away. It s also completely silent and doesn t provide any additional useful information. I was wondering how I could take our lessons from MDCO#1 and integrate our new tricks with the sponsors loop. That is, add the schedule, time, some space to type announcements on the screen and also add some loopable music to it. I used OBS before in making my videos, and like the flexibility it provides when working with scenes and sources. A scene is what you would think of as a screen or a document with its own collection of sources or elements. For example, a scene might contain sources such as a logo, clock, video, image, etc. A scene can also contain another scene. This is useful if you want to contain a banner or play some background music that is shared between scenes.
The Loopy Loop Music The two mini albums that mostly played during the first few days were just a copy and paste from the MDCO#1 music, which was:freepd.com:
clangon macOS needed extra curlies to be happy, another manifestation of Solaris having no idea what a timezone setting America/New_York is, plus some extra pickyness from the SAN tests and whatnot. So Leonardo and I gave it some extra care over the weekend, uploaded it late yesterday and here we are with 0.3.1. Thanks again to CRAN for prompt processing even though they are clearly deluged shortly before their (brief) summer break. nanotime relies on the RcppCCTZ package for (efficient) high(er) resolution time parsing and formatting up to nanosecond resolution, and the bit64 package for the actual
integer64arithmetic. Initially implemented using the S3 system, it has benefitted greatly from work by Leonardo Silvestri who rejigged internals in S4 and now added new types for periods, intervals and durations. The NEWS snippet adds full details.
Thanks to CRANberries there is also a diff to the previous version. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.
Changes in version 0.3.1 (2020-08-09)
Welcome to the July 2020 report from the Reproducible Builds project. In these monthly reports, we round-up the things that we have been up to over the past month. As a brief refresher, the motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced from the original free software source code to the pre-compiled binaries we install on our systems. (If you re interested in contributing to the project, please visit our main website.)
ftp.debian.orgwere made from their claimed sources. Tavis Ormandy published a blog post making the provocative claim that You don t need reproducible builds , asserting elsewhere that the many attacks that have been extensively reported in our previous reports are fantasy threat models . A number of rebuttals have been made, including one from long-time contributor Reproducible Builds contributor Bernhard Wiedemann. On our mailing list this month, Debian Developer Graham Inggs posted to our list asking for ideas why the
openorienteering-mapperDebian package was failing to build on the Reproducible Builds testing framework. Chris Lamb remarked from the build logs that the package may be missing a build dependency, although Graham then used our own diffoscope tool to show that the resulting package remains unchanged with or without it. Later, Nico Tyni noticed that the build failure may be due to the relationship between the
FILEC preprocessor macro and the
-ffile-prefix-mapGCC flag. An issue in Zephyr, a small-footprint kernel designed for use on resource-constrained systems, around
.alibrary files not being reproducible was closed after it was noticed that a key part of their toolchain was updated that now calls
--enable-deterministic-archivesby default. Reproducible Builds developer kpcyrd commented on a pull request against the libsodium cryptographic library wrapper for Rust, arguing against the testing of CPU features at compile-time. He noted that:
I ve accidentally shipped broken updates to users in the past because the build system was feature-tested and the final binary assumed the instructions would be present without further runtime checksDavid Kleuker also asked a question on our mailing list about using
READMEfile [ ], marked the Alpine Linux continuous integration tests as currently disabled [ ] and linked the Arch Linux Reproducible Status page from our projects page [ ].
zipnote(1)to determine differences in a
.zipfile as we can use
libarchive. [ ]
--profileas a synonym for
--profile=-, ie. write profiling data to standard output. [ ]
strings(1)to eight characters to avoid unnecessary diff noise. [ ]
--no-exclude-directory-metadatahave been replaced with
--exclude-directory-metadata= yes,no. [ ]
xxd(1)and show bytes in groups of 4. [ ]
javap not found in pathif it is available in the path but it did not result in an actual difference. [ ]
... not available in pathmessages when looking for Java decompilers that used the Python class name instead of the command. [ ]
--debuglog noise by truncating the
has_some_contentmessages. [ ]
compare_fileslog message when the file does not have a literal name. [ ]
exit_if_paths_do_not_existto not check files multiple times. [ ][ ]
add_commenthelper method; don t mess with our internal list directly. [ ]
str.formatwith Python f-strings [ ] and make it easier to navigate to the
main.pyentry point [ ].
Nonein the failure case as we return a non-
Nonevalue in the success one. [ ]
NullChangesquasi-file to represent missing data in the Debian package comparator [ ] and clarify use of a null diff in order to remember an exit code. [ ]
diffoscope @args.txt. (!62)
objdump[ ][ ] and remove raw instructions from ELF tests [ ].
--verbose-level warning when the Archive::Cpio Perl module is missing. (!6) reprotest is our end-user tool to build same source code twice in widely differing environments and then checks the binaries produced by each build for any differences. This month, Vagrant Cascadian made a number of changes to support diffoscope version 153 which had removed the (deprecated)
--no-exclude-directory-metadatacommand-line arguments, and updated the testing configuration to also test under Python version 3.8 [ ].
debhelperbuild tool impacting the reproducibility status of hundreds of packages that use the CMake build system. This month however, Niels Thykier uploaded
debhelperversion 13.2 that passes the
-DBUILD_RPATH_USE_ORIGIN=ONarguments to CMake when using the (currently-experimental) Debhelper compatibility level 14. According to Niels, this change:
should fix some reproducibility issues, but may cause breakage if packages run binaries directly from the build directory.34 reviews of Debian packages were added, 14 were updated and 20 were removed this month adding to our knowledge about identified issues. Chris Lamb added and categorised the
nondeterministic_order_of_debhelper_snippets_added_by_dh_fortran_mod[ ] and
gem2deb_install_mkmf_log[ ] toolchain issues. Lastly, Holger Levsen filed two more wishlist bugs against the
debrebuildDebian package rebuilder tool [ ][ ].
afl(fix an incorrectly built manual page varied from kernel boot options)
dnscrypt-proxy(sort the output of
graphviz(timezone issue, forwarded from Debian)
insighttoolkit(prevent CPU detection, forwarded upstream
ipopt(parallelism issue and use https://tracker.debian.org/pkg/strip-nondeterminism)
jboss-logging-tools(date, forwarded upstream)
lcov(date issue, already upstream)
multus(date issue, already upstream)
paperjam(date issue, forwarded upstream)
python-PyNaCl(sort Python glob/readdir)
python-enaml(workaround an open upstream Python issue)
sac(omit creation time from
sql-parser(sort, already upstream)
ugrep(CPU-related issue, already upstream)
unknown-horizons(filesystem ordering issue, already upstream)
unknown-horizons(filesystem ordering issue)
tests.reproducible-builds.org. This month, Holger Levsen made the following changes:
sbuildexit code. [ ][ ]
php-hordepackages back to the
pkg-php-pearpackage set for the bullseye distribution. [ ]
debrebuild. [ ]
pbuilder[ ], NetBSD [ ], unkillable processes [ ], unresponsive nodes [ ][ ][ ][ ], proxy connection failures [ ], too many installed kernels [ ], etc.
systemdunits. [ ]
init_nodescript to suggest using sudo instead of explicit logout and logins [ ][ ] and the usual build node maintenance was performed by Holger Levsen [ ][ ][ ][ ][ ][ ], Mattia Rizzolo [ ][ ] and Vagrant Cascadian [ ][ ][ ][ ].
^W^Wpublic IRC team meeting