Photo by Pixabay |
Given a typical install of 3 generic kernel ABIs in the default configuration on a regular-sized VM (2 CPU cores 8GB of RAM) the following metrics are achieved in Ubuntu 23.10 versus Ubuntu 22.04 LTS:
2x less disk space used (1,417MB vs 2,940MB, including initrd)
3x less peak RAM usage for the initrd boot (68MB vs 204MB)
0.5x increase in download size (949MB vs 600MB)
2.5x faster initrd generation (4.5s vs 11.3s)
approximately the same total time (103s vs 98s, hardware dependent)
For minimal cloud images that do not install either linux-firmware or modules extra the numbers are:
1.3x less disk space used (548MB vs 742MB)
2.2x less peak RAM usage for initrd boot (27MB vs 62MB)
0.4x increase in download size (207MB vs 146MB)
Hopefully, the compromise of download size, relative to the disk space & initrd savings is a win for the majority of platforms and use cases. For users on extremely expensive and metered connections, the likely best saving is to receive air-gapped updates or skip updates.
This was achieved by precompressing kernel modules & firmware files with the maximum level of Zstd compression at package build time; making actual .deb files uncompressed; assembling the initrd using split cpio archives - uncompressed for the pre-compressed files, whilst compressing only the userspace portions of the initrd; enabling in-kernel module decompression support with matching kmod; fixing bugs in all of the above, and landing all of these things in time for the feature freeze. Whilst leveraging the experience and some of the design choices implementations we have already been shipping on Ubuntu Core. Some of these changes are backported to Jammy, but only enough to support smooth upgrades to Mantic and later. Complete gains are only possible to experience on Mantic and later.
The discovered bugs in kernel module loading code likely affect systems that use LoadPin LSM with kernel space module uncompression as used on ChromeOS systems. Hopefully, Kees Cook or other ChromeOS developers pick up the kernel fixes from the stable trees. Or you know, just use Ubuntu kernels as they do get fixes and features like these first.
The team that designed and delivered these changes is large: Benjamin Drung, Andrea Righi, Juerg Haefliger, Julian Andres Klode, Steve Langasek, Michael Hudson-Doyle, Robert Kratky, Adrien Nader, Tim Gardner, Roxana Nicolescu - and myself Dimitri John Ledkov ensuring the most optimal solution is implemented, everything lands on time, and even implementing portions of the final solution.
Hi, It's me, I am a Staff Engineer at Canonical and we are hiring https://canonical.com/careers.
Lots of additional technical details and benchmarks on a huge range of diverse hardware and architectures, and bikeshedding all the things below:
[ ] In March 2023, Ken gave the closing keynote [and] during the Q&A session, someone jokingly asked about the Turing award lecture, specifically can you tell us right now whether you have a backdoor into every copy of gcc and Linux still today?Although Ken reveals (or at least claims!) that he has no such backdoor, he does admit that he has the actual code which Russ requests and subsequently dissects in great but accessible detail.
Arch Linux packages become reproducible a median of 30 days quicker when compared to Debian packages, while Debian packages remain reproducible for a median of 68 days longer once fixed.A full PDF of their paper is available online, as are many other interesting papers on MCIS publication page.
nixos-minimal
image that is used to install NixOS. In their post, Arnout details what exactly can be reproduced, and even includes some of the history of this endeavour:
You may remember a 2021 announcement that the minimal ISO was 100% reproducible. While back then we successfully tested that all packages that were needed to build the ISO were individually reproducible, actually rebuilding the ISO still introduced differences. This was due to some remaining problems in the hydra cache and the way the ISO was created. By the time we fixed those, regressions had popped up (notably an upstream problem in Python 3.10), and it isn t until this week that we were back to having everything reproducible and being able to validate the complete chain.Congratulations to NixOS team for reaching this important milestone! Discussion about this announcement can be found underneath the post itself, as well as on Hacker News.
arm64
hardware from Codethink
Long-time sponsor of the project, Codethink, have generously replaced our old Moonshot-Slides , which they have generously hosted since 2016 with new KVM-based arm64
hardware. Holger Levsen integrated these new nodes to the Reproducible Builds continuous integration framework.
ext4
filesystem images. [ ]
SOURCE_DATE_EPOCH
environment variable in order to close bug #1034422. In addition, 8 reviews of packages were added, 74 were updated and 56 were removed this month, all adding to our knowledge about identified issues.
Bernhard M. Wiedemann published another monthly report about reproducibility within openSUSE.
edje_cc
(race condition)elasticsearch
(build failure)erlang-retest
(embedded .zip
timestamp)fdo-client
(embeds private keys)fftw3
(random ordering)gsoap
(date issue)gutenprint
(date)hub/golang
(embeds random build path)Hyprland
(filesystem issue)kitty
(sort-related issue, .tar
file embeds modification time)libpinyin
(ASLR)maildir-utils
(date embedded in copyright)mame
(order-related issue)mingw32-binutils
& mingw64-binutils
(date)MooseX
(date from perl-MooseX-App)occt
(sorting issue)openblas
(embeds CPU count)OpenRGB
(corruption-related issue)python-numpy
(random file names)python-pandas
(FTBFS)python-quantities
(date)python3-pyside2
(order)qemu
(date and Sphinx issue)qpid
(sorting problem)rakudo
(filesystem ordering issue)SLOF
(date-related issue)spack
(CPU counting issue)xemacs-packages
(date-related issue)file -i
returns text/plain
, fallback to comparing as a text file. This was originally filed as Debian bug #1053668) by Niels Thykier. [ ] This was then uploaded to Debian (and elsewhere) as version 251
.
#debian-reproducible-changes
IRC channel. [ ][ ][ ]systemd-oomd
on all Debian bookworm nodes (re. Debian bug #1052257). [ ]schroots
. [ ]arm64
machines from Codethink. [ ][ ][ ][ ][ ][ ]#reproducible-builds
on irc.oftc.net
.
rb-general@lists.reproducible-builds.org
Series: | Machine Mandate #1 |
Publisher: | Prime Books |
Copyright: | 2019 |
ISBN: | 1-60701-533-1 |
Format: | Kindle |
Pages: | 86 |
pandoc couronne.md --standalone --css styling.css \
--to html5 --table-of-contents > couronne.html
I feel I will never need a word processor again. It produces this nice looking document without pain.
Software signing is not a new problem, so there must be some solution already, right? Yes, but signing software and maintaining keys is very difficult especially for non-security folks and UX of existing tools such as PGP leave much to be desired. That s why we need something like sigstore - an easy to use software/toolset for signing software artifacts.The second post (titled Signing Software The Easy Way with Sigstore and Cosign) goes into some technical details of getting started.
Some time ago I checked Signal s reproducibility and it failed. I asked others to test in case I did something wrong, but nobody made any reports. Since then I tried to test the Google Play Store version of the apk against one I compiled myself, and that doesn t match either.
Most users are not capable of building from source code themselves, but we can at least get them able enough to check signatures and shasums. When reputable people who can tell everyone they were able to reproduce the project s build, others at least have a secondary source of validation.
Related to this, there was continuing discussion on how to embed/encode the build metadata for the Debian live images which were being worked on by Roland Clobus.
- All major configurations are still built regularly using live-build and bullseye.
- All major configurations are reproducible now; Jenkins is green.
- I ve worked around the issue for the Cinnamon image.
- The patch was accepted and released within a few hours.
- My main focus for the last month was on the live-build tool itself.
- It will properly use the proxy for all HTTP traffic.
I m working for Oracle in the Build Group for OpenJDK which is primary responsible for creating a built artifact of the OpenJDK source code. [ ] For the last few years, we have worked on a low-effort, background-style project to make the build of OpenJDK itself building reproducible. We ve come far, but there are still issues I d like to address. [ ]
183
, 184
and 185
as well as performed significant triaging of merge requests and other issues in addition to making the following changes:
.rds
files. [ ]format_class
import. [ ]close_archive
when garbage collecting Archive
instances, unless open_archive
definitely returned successfully. This prevents, for example, an AttributeError
where PGPContainer
s cleanup routines were rightfully assuming that its temporary directory had actually been created. [ ].rdb
files after refactoring temporary directory handling. [ ]python3-rpm
is installed or not at build time. [ ]androguard
module not being in the (expected) python3-androguard
Debian package. [ ]shellcheck
warning in debian/tests/control.sh
. [ ]h5py
in our tests that doesn t concern us. [ ].1
from the Standards-Version
field as it s required. [ ]--diff-context
option to control unified diff context size [ ] and Jean-Romain Garnier fixed the Macho comparator for architectures other than x86-64
[ ].
gtk4
(date-related issue)build-compare
(random tempfile problem)itinerary
(time-related build failure)lcalc
.htscodecs
.osdlyrics
.xtermcontrol
.rust-insta
.python-tomli
.python-pairix
.python-pybedtools
(forwarded upstream).#reproducible-builds
on irc.oftc.net
.
rb-general@lists.reproducible-builds.org
184
. This version includes the following changes:
[ Chris Lamb ]
* Fix the semantic comparison of R's .rdb files after a refactoring of
temporary directory handling in a previous version.
* Support a newer format version of R's .rds files.
* Update tests for OCaml 4.12. (Closes: reproducible-builds/diffoscope#274)
* Move diffoscope.versions to diffoscope.tests.utils.versions.
* Use assert_diff in tests/comparators/test_rdata.py.
* Reformat various modules with Black.
[ Zbigniew J drzejewski-Szmek ]
* Stop using the deprecated distutils module by adding a version
comparison class based on the RPM version rules.
* Update invocations of llvm-objdump for the latest version of LLVM.
* Adjust a test with one-byte text file for file(1) version 5.40.
* Improve the parsing of the version of OpenSSH.
[ Benjamin Peterson ]
* Add a --diff-context option to control the unified diff context size.
(reproducible-builds/diffoscope!88)
I wrote this blog post with Kaylea Champion and a version of this post was originally posted on the Community Data Science Collective blog. Critical software we all rely on can silently crumble away beneath us. Unfortunately, we often don t find out software infrastructure is in poor condition until it is too late. Over the last year or so, I have been supporting Kaylea Champion on a project my group announced earlier to measure software underproduction a term we use to describe software that is low in quality but high in importance. Underproduction reflects an important type of risk in widely used free/libre open source software (FLOSS) because participants often choose their own projects and tasks. Because FLOSS contributors work as volunteers and choose what they work on, important projects aren t always the ones to which FLOSS developers devote the most attention. Even when developers want to work on important projects, relative neglect among important projects is often difficult for FLOSS contributors to see. Given all this, what can we do to detect problems in FLOSS infrastructure before major failures occur? Kaylea Champion and I recently published a paper laying out our new method for measuring underproduction at the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) 2021 that we believe provides one important answer to this question.
For more details, check out the full paper which is available as a freely accessible preprint.
This project was supported by the Ford/Sloan Digital Infrastructure Initiative. Wm Salt Hale of the Community Data Science Collective and Debian Developers Paul Wise and Don Armstrong provided valuable assistance in accessing and interpreting Debian bug data. Ren Just generously provided insight and feedback on the manuscript.
Paper Citation: Kaylea Champion and Benjamin Mako Hill. 2021. Underproduction: An Approach for Measuring Risk in Open Source Software. In Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER 2021). IEEE.
Contact Kaylea Champion (kaylea@uw.edu) with any questions or if you are interested in following up.
Thanks to my CRANberries, you can also look at a diff to the previous release. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. Bugs reports are welcome at the GitHub issue tracker as well (where one can also search among open or closed issues); questions are also welcome underChanges in Rcpp patch release version 1.0.6 (2021-01-14)
- Changes in Rcpp API:
- Replace remaining few uses of
EXTPTR_PTR
withR_ExternalPtrAddr
(Kevin in #1098 fixing #1097).- Add
push_back
andpush_front
forDataFrame
(Walter Somerville in #1099 fixing #1094).- Remove a misleading-to-wrong comment (Mattias Ellert in #1109 cleaning up after #1049).
- Address a sanitizer report by initializing two private
bool
variables (Benjamin Christoffersen in #1113).- External pointer finalizer toggle default values were corrected to true (Dirk in #1115).
- Changes in Rcpp Documentation:
- Several URLs were updated to https and/or new addresses (Dirk).
- Changes in Rcpp Deployment:
- Added GitHub Actions CI using the same container-based setup used previously, and also carried code coverage over (Dirk in #1128).
- Changes in Rcpp support functions:
Rcpp.package.skeleton()
avoids warning from R. (Dirk)
rcpp
tag at StackOverflow which also allows searching among the (currently) 2616 previous questions.
If you like this or other open-source work I do, you can sponsor me at GitHub. My sincere thanks to my current sponsors for me keeping me caffeinated.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Next.