In 2004, I was living in London, and decided it was time I had my own little virtual private server somewhere online. As a Debian developer since the start of 2000, it had to be Debian, and it still is This was before cloud as we know it today. Virtual Private Servers (VPS) was a Continue reading "My own little server"
Welcome to the October 2024 report from the Reproducible Builds project.
Our reports attempt to outline what we ve been up to over the past month, highlighting news items from elsewhere in tech where they are related. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.
Table of contents:
Jens Dietrich, Tim White, of Victoria University of Wellington, New Zealand along with Behnaz Hassanshahi and Paddy Krishnan of Oracle Labs Australia published a paper entitled Levels of Binary Equivalence for the Comparison of Binaries from Alternative Builds :
The availability of multiple binaries built from the same sources creates new challenges and opportunities, and raises questions such as: Does build A confirm the integrity of build B? or Can build A reveal a compromised build B? . To answer such questions requires a notion of equivalence between binaries. We demonstrate that the obvious approach based on bitwise equality has significant shortcomings in practice, and that there is value in opting for alternative notions. We conceptualise this by introducing levels of equivalence, inspired by clone detection types.A PDF of the paper is freely available.
On Friday 8th November, Vagrant Cascadian will present a talk entitled Two Ways to Trustworthy at SeaGL in Seattle, WA.
Founded in 2013, SeaGL is a free, grassroots technical summit dedicated to spreading awareness and knowledge about free source software, hardware and culture. Vagrant s talk:
[ ] delves into how two project[s] approaches fundamental security features through Reproducible Builds, Bootstrappable Builds, code auditability, etc. to improve trustworthiness, allowing independent verification; trustworthy projects require little to no trust. Exploring the challenges that each project faces due to very different technical architectures, but also contextually relevant social structure, adoption patterns, and organizational history should provide a good backdrop to understand how different approaches to security might evolve, with real-world merits and downsides.
the D8 Java to DEX compiler (part of the Android toolchain) eliminated a redundant field load if running the class s static initialiser was known to be free of side effects, which ended up accidentally depending on the sharding of the input, which is dependent on the number of CPU cores used during the build.To make it easier to understand the bug and the patch, Fay also made a small example to illustrate when and why the optimisation involved is valid.
CONFIG_MODULE_SIG and the unreproducible Linux Kernel to add: I wonder whether it would be possible to use the Linux kernel s Integrity Policy Enforcement to deploy a policy that would prevent loading of anything except a set of expected kernel modules. [ ]
diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 279, 280, 281 and 282 to Debian:
.ar archives (#1085257). [ ]systemd-ukify in the Debian stable distribution. [ ]Depends on the deprecated python3-pkg-resources (#1083362). [ ]
The IzzyOnDroid project has reached a good milestone by reaching over 25% of the ~1,200 Android apps provided by their repository (of official APKs built by the original application developers) having been confirmed to be reproducible by a rebuilder.
In Debian this month:
devscripts version 2.24.2, including many changes to the debootsnap, debrebuild and reproducible-check scripts. This is the first time that debrebuild actually works (using sbuild s unshare backend). As part of this, Holger also fixed an issue in the reproducible-check script where a typo in the code led to incorrect results [ ]
The new server has no problems keeping up with importing the full archives on every update, as each run finishes comfortably in time before it s time to run again. [While] the new server is the one doing all the importing of updated archives, the HTTP interface is being served by both the new server and one of the VM s at LeaseWeb.The entry list a number of specific updates surrounding the API endpoints and rate limiting.
Elsewhere in distribution news, Zbigniew J drzejewski-Szmek performed another rebuild of Fedora 42 packages, with the headline result being that 91% of the packages are reproducible. Zbigniew also reported a reproducibility problem with QImage.
Finally, in openSUSE, Bernhard M. Wiedemann published another report for that distribution.
There were an enormous number of improvements made to our website this month, including:
hulkoba
README page for building the website under NixOS. [ ][ ][ ][ ][ ]
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In October, a number of changes were made by Holger Levsen, including:
index.html for rebuilderd. [ ]nginx.conf configuration file for rebuilderd. [ ]riscv64 architecture. [ ]rebuilderd-related TODO. [ ]inos5 node [ ] and Vagrant Cascadian brought 4 virt nodes back online [ ].
The Open Source Summit EU took place recently, and covered plenty of topics related to supply-chain security, including:
apache-ivy (.zip modification time)ccache (build failure)colord (CPU)efivar (CPU/march=native)gsl (no check)libcamera (date/copyright year)libreoffice (possible rpm/build toolchain corruption bug)moto (.gz modification time)openssl-1_1 (date-related issue)python-pygraphviz (benchmark)sphinx/python-pygraphviz (benchmark)python-panel (package.lock has random port)python-propcache (random temporary path)python314 (.gz-related modification time)rusty_v8 (random .o files)scapy (date)wine (parallelism)ibmtss (FTBFS-2026)pymol (date)pandas (ASLR)linutil (drop date)lsof (also filed in openSUSE: uname -r in LSOF_VSTR)schily (also filed in openSUSE: uname -r)superlu (nocheck)util (random test failure)ceph (year-2038 variation from embedded boost)distro-info.calibre (two sort issues) [ ][ ]#reproducible-builds on irc.oftc.net.
rb-general@lists.reproducible-builds.org
libpng1.6 would fail to cross build on musl architectures whereas it would succeed on other ones failing to locate zlib. Adding --debug-find to the cmake invocation eventually revealed that it would fail to search in /usr/lib/<triplet>, which is the default library path. This turned out to be a bug in cmake assuming that all linux systems use glibc. libpng1.6 also gained a baseline violation for powerpc and ppc64 by enabling the use of AltiVec there.
The newt package would fail to cross build for many 32-bit architectures whereas it would succeed for armel and armhf due to -Wincompatible-pointer-types. It turns out that this flag was turned into -Werror and it was compiling with a warning earlier. The actual problem is a difference in signedness between wchar_t and FriBidChar (aka uint32_t) and actually affects native building on i386.
mkdocs-macros-plugin, which required packaging a new Python package for Debian, python-super-collections (now in NEW review).
The coffee shop Ibtehal and me used to visit during the midnight
Grating at a chemist shop in Mombasa, Kenya
Me with players at the Nairobi Gymkhana Club
Cricket pitch at the Nairobi Gymkhana Club
A view of the cricket ground inside the Nairobi Gymkhana Club
Scoreboard at the Nairobi Gymkhana cricket ground
A matatu with a Notorious BIG portrait.
Inside the Giraffe Center
SGR train at Nairobi Terminus.
Interior of the SGR train
Mombasa Terminus station.
The balcony at Tulia Backpackers Hostel had a nice view of the ocean.
A room inside the hostel with fans and mosquito nets on the beds
Nyali beach is a white sand beach
This is the ferry service for crossing the river.
Diani beach is a popular beach in Kenya. It is a white sand beach.
Selfie with truck drivers who gave me the free ride
Newspapers at a stand in Kenya covering news on the airport closure
Spanish Omelette served in breakfast at Nairobi Transit Hotel
Ugali with eggs.
I have no idea either.
Happy Maladay1 to those who celebrate it, I guess.
If you care about the how, it started as china ink on tracing paper,
with the help of a template (and a correction sheet for one page where I
used the wrong line on the template).
A rubber stamp was carved with the author s signature and stamped on
white paper because the ink from the pad wasn t working well on tracing
paper.
Then everything was scanned (with the correction on top of the wrong
page) asemic_zine_scans.tar.
Imported in Inkscape and traced asemic_zine_svg.tar.
Printed, cut in half, folded and stapled. The magenta lines weren t by
design, but are there because my printer is currently2
cursed.
And finally, asemic_zine.pdf was created,
joining the pages together with pdfjam, for convenience in case somebody
wants to download the full thing.
All the .tar and .pdf downloads from this page are released
under the WTFPL, or All Rites Reversed..
Welcome to the September 2024 report from the Reproducible Builds project!
Our reports attempt to outline what we ve been up to over the past month, highlighting news items from elsewhere in tech where they are related. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.
Table of contents:
binsider tool to analyse ELF binaries
Reproducible Builds developer Orhun Parmaks z has announced a fantastic new tool to analyse the contents of ELF binaries. According to the project s README page:
Binsider can perform static and dynamic analysis, inspect strings, examine linked libraries, and perform hexdumps, all within a user-friendly terminal user interface!More information about Binsider s features and how it works can be found within Binsider s documentation pages.
A seven-year-old bug about the nondeterminism of object code generated by the Glasgow Haskell Compiler (GHC) received a recent update, consisting of Rodrigo Mesquita noting that the issue is:
95% fixed by [merge request] !12680 when -fobject-determinism is enabled. [ ]
The linked merge request has since been merged, and Rodrigo goes on to say that:
After that patch is merged, there are some rarer bugs in both interface file determinism (eg.#25170) and in object determinism (eg.#25269) that need to be taken care of, but the great majority of the work needed to get there should have been merged already. When merged, I think we should close this one in favour of the more specific determinism issues like the two linked above.
zlib/deflate compression in .zip and .apk files and later followed up with the results of her subsequent investigation.
CONFIG_MODULE_SIG flag. [ ]
zlib to zlib-ng as reproducibility requires identical compressed data streams. [ ]
maven-lockfile that is designed aid building Maven projects with integrity . [ ]
This is a report of Part 1 of my journey: building 100% bit-reproducible packages for every package that makes up [openSUSE s] minimalVM image. This target was chosen as the smallest useful result/artifact. The larger package-sets get, the more disk-space and build-power is required to build/verify all of them.
This work was sponsored by NLnet s NGI Zero fund.
Marvin Strangfeld published his bachelor thesis, Reproducibility of Computational Environments for Software Development from RWTH Aachen University. The author offers a more precise theoretical definition of computational environments compared to previous definitions, which can be applied to describe real-world computational environments. Additionally, Marvin provide a definition of reproducibility in computational environments, enabling discussions about the extent to which an environment can be made reproducible. The thesis is available to browse or download in PDF format.
In addition, Shenyu Zheng, Bram Adams and Ahmed E. Hassan of Queen s University, ON, Canada have published an article on hermeticity in Bazel-based build systems:
A hermetic build system manages its own build dependencies, isolated from the host file system, thereby securing the build process. Although, in recent years, new artifact-based build technologies like Bazel offer build hermeticity as a core functionality, no empirical study has evaluated how effectively these new build technologies achieve build hermeticity. This paper studies 2,439 non-hermetic build dependency packages of 70 Bazel-using open-source projects by analyzing 150 million Linux system file calls collected in their build processes. We found that none of the studied projects has a completely hermetic build process, largely due to the use of non-hermetic top-level toolchains. [ ]
In Debian this month, 14 reviews of Debian packages were added, 12 were updated and 20 were removed, all adding to our knowledge about identified issues. A number of issue types were updated as well. [ ][ ]
In addition, Holger opened 4 bugs against the debrebuild component of the devscripts suite of tools. In particular:
#1081047: Fails to download .dsc file.#1081048: Does not work with a proxy.#1081050: Fails to create a debrebuild.tar.#1081839: Fails with E: mmdebstrap failed to run error.build_path variation. Holger Levsen provided a rationale for this change in the issue, which has already been made to the tests being performed by tests.reproducible-builds.org. This month, this issue was closed by Santiago R. R., nicely explaining that build path variation is no longer the default, and, if desired, how developers may enable it again.
In openSUSE news, Bernhard M. Wiedemann published another report for that distribution.
diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading version 278 to Debian:
python3-setuptools dependency. (#1080825)Standards-Version to 4.7.0. [ ]0.5.11-4 was uploaded to Debian unstable by Holger Levsen making the following changes:
pkg-config package with one on pkgconf, following a Lintian check. [ ]Standards-Version field to 4.7.0, with no related changes needed. [ ]0.7.28 was uploaded to Debian unstable by Holger Levsen including a change by Jelle van der Waa to move away from the pipes Python module to shlex, as the former will be removed in Python version 3.13 [ ].
classes.dex file (and thus a different .apk) depending on the number of cores available during the build, thereby breaking Reproducible Builds:
We ve rebuilt [tagv3.6.1] multiple times (each time in a fresh container): with 2, 4, 6, 8, and 16 cores available, respectively:
- With 2 and 4 cores we always get an unsigned APK with SHA-256
14763d682c9286ef.- With 6, 8, and 16 cores we get an unsigned APK with SHA-256
35324ba4c492760instead.
A new plugin for the Gradle build tool for Java has been released. This easily-enabled plugin results in:
reproducibility settings [being] applied to some of Gradle s built-in tasks that should really be the default. Compatible with Java 8 and Gradle 8.3 or later.
There were a rather substantial number of improvements made to our website this month, including:
ext4, erofs and FAT filesystems can now be made reproducible . [ ]
agama-integration-tests (random)contrast (FTBFS-nocheck)cpython (FTBFS-2038)crash (parallelism, race)ghostscript (toolchain date)glycin-loaders (FTBFS -j1)gstreamer-plugins-rs (date, other)kernel-doc/Sphinx (toolchain bug, parallelism/race)kernel (parallelism in BTF)libcamera (random key)libgtop (uname -r)libsamplerate (random temporary directory)lua-luarepl (FTBFS)meson (toolchain)netty (modification time in .a)nvidia-persistenced (date)nvidia-xconfig (date-related issue)obs-build (build-tooling corruption)perl (Perl records kernel version)pinentry (make efl droppable)python-PyGithub (FTBFS 2024-11-25)python-Sphinx (parallelism/race)python-chroma-hnswlib (CPU)python-libcstpython-pygraphviz (random timing)python312 (.pyc embeds modification time)python312 (drop .pyc from documentation time)scap-security-guide (date)seahorse (parallelism)subversion (minor Java .jar modification times)xen/acpica (date-related issue in toolchain)xmvn (random)magic-wormhole-transit-relay.python-sphobjinv.lomiri-content-hub.python-mt-940.tree-puzzle.muon-meson.
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In September, a number of changes were made by Holger Levsen, including:
osuosl4 node to Debian trixie in anticipation of running debrebuild and rebuilderd there. [ ][ ][ ]osuosl4 node as offline due to ongoing xfs_repair filesystem maintenance. [ ][ ]risc64 architecture to the multiarch version skew tests for Debian trixie and sid. [ ][ ][ ]virt 32,64 b nodes as down. [ ]virt32b and virt64b nodes [ ], performed some maintenance of the cbxi4a node [ ][ ] and marked most armhf architecture systems as being back online.
#reproducible-builds on irc.oftc.net.
rb-general@lists.reproducible-builds.org
A new release 0.0.4 of the RcppFastAD package
by James Yang and myself
is now on CRAN.
RcppFastAD
wraps the FastAD
header-only C++ library by James which provides a C++
implementation of both forward and reverse mode of automatic
differentiation. It offers an easy-to-use header library (which we
wrapped here) that is both lightweight and performant. With a little of
bit of Rcpp glue, it is also easy to
use from R in simple C++ applications. This release updates the quick
fix in release
0.0.3 from a good week ago. James took a good look and
properly disambiguated the statement that lead clang to complain, so we
are back to compiling as C++17 under all compilers which makes for a
slightly wider reach.
The NEWS file for this release follows.
Courtesy of my CRANberries, there is also a diffstat report for the most recent release. More information is available at the repository or the package page. If you like this or other open-source work I do, you can sponsor me at GitHub.Changes in version 0.0.4 (2024-09-24)
- The package now properly addresses a clang warning on empty variadic macros arguments and is back to C++17 (James in #10)
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
default_destination_recipient_limit
and
default_destination_rate_delay
If default_destination_recipient_limit is over 1, then
default_destination_rate_delay is equal to the minimum delay between sending
email to the same domain.
So, I m staring our IP addresses out at 30m - which prevents any single domain
from receiving more than 2 messages per hour. Sadly, there are a lot of
different domain names that deliver to the same set of popular corporate MX
servers, so I am not sure I can accurately control how many messages a given
provider sees coming from a given IP address. But it s a start.
A bigger problem is that messages that exceed the limit hang out in the
active queue until they can be sent without violating the rate limit. Since I
can t fully control the number of messages a given queue receives (due to my
inability to control the DNS round robin weights), a lot of messages are going
to be severely delayed, especially ones with an @gmail.com domain name.
I know I can temporarily set relayhost to a different queue and flush
deferred messages, however, as far as I can tell, it doesn t work with
active messages.
To help mitigate the problem I m only using our bulk mail queue to warm up IPs,
but really, this is not ideal.
Suggestions welcome!
# Put the message on hold in the warming up instance
postsuper -c /etc/postfix-warmingup -h $queueid
# Copy to a warmed up instance
cp --preserve=mode,ownership,timestamp /var/spool/postfix-warmingup/hold/$queueid /var/spool/postfix-warmedup/incoming/
# Queue the message
postqueue -c /etc/postfix-warmedup -i $queueid
# Delete from the original queue.
postsuper -c /etc/postfix-warmingup -d $queueid
421 4.7.0 [TSS04] Messages from 204.19.241.9 temporarily deferred due to unexpected volume or user complaints - 4.16.55.1; see https://postmaster.yahooinc.com/error-codes (in replySo I will keep it as 30m for another 24 hours or so and then move to 15m. Now that I can flush the backlog of active messages I am in less of a hurry.
systemctl reload. I
wonder if there is a setting to control this retry? Or better yet, why can t
these messages that exceed the rate delayed be deferred instead?
[redacted]
X-Spam-Flag: YES
X-Spam-Level: ******
X-Spam-Status: Yes, score=6.3 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
[redacted]
* 1.0 RCVD_IN_XBL RBL: Received via a relay in Spamhaus XBL
* [185.220.101.64 listed in xxxxxxxxxxxxx.zen.dq.spamhaus.net]
* 3.0 RCVD_IN_SBL_CSS Received via a relay in Spamhaus SBL-CSS
* 2.5 RCVD_IN_AUTHBL Received via a relay in Spamhaus AuthBL
* 0.0 RCVD_IN_PBL Received via a relay in Spamhaus PBL
[redacted]
[very first received line follows...]
Received: from [10.137.0.13] ([185.220.101.64])
by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-378956d2ee6sm12487760f8f.83.2024.09.11.15.05.52
for <xxxxx@mayfirst.org>
(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
Wed, 11 Sep 2024 15:05:53 -0700 (PDT)
185.220.101.64 which appears
to be a Tor exit node.
The sender is using their own client to relay email directly to Gmail. Like any
sane person, they don t trust Gmail to protect their privacy, so they are
sending via Tor. But WTF, Gmail is not stripping the sending IP address from
the header.
I m a big fan of harm reduction and have always considered using your own
client to relay email with Gmail as a nice way to avoid some of the
surveillance tax Google imposes.
However, it seems that if you pursue this option you have two unpleasant
choices:
I realize that because I have several chairs, the phrase my chair is ambiguous. To reduce confusion, I will refer to the head of my academic department as my office chair going forward.
A new release 0.0.3 of the RcppFastAD package
by James Yang and myself
is now on CRAN.
RcppFastAD
wraps the FastAD
header-only C++ library by James which provides a C++
implementation of both forward and reverse mode of automatic
differentiation. It offers an easy-to-use header library (which we
wrapped here) that is both lightweight and performant. With a little of
bit of Rcpp glue, it is also easy to
use from R in simple C++ applications. This release turns compilation to
the C++20 standard as newer clang++ versions complained
about a particular statement (it took to be C++20) when compiled under
C++17. So we obliged.
The NEWS file for these two initial releases follows.
Courtesy of my CRANberries, there is also a diffstat report for the most recent release. More information is available at the repository or the package page. If you like this or other open-source work I do, you can sponsor me at GitHub.Changes in version 0.0.3 (2024-09-15)
- The package now compiles under the C++20 standard to avoid a warning under
clang++-18(Dirk addressing #9)- Minor updates to continuous integration and badges have been made as well
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Sep 11 05:08:03 Warning: mysqldump: Error 2013: Lost connection to server during query when dumping table 1C4Uonkwhe_options at row: 1402
Sep 11 05:08:03 Warning: Failed to dump mysql databases ic_wp
Sep 11 13:50:11 mysql007 mariadbd[580]: 2024-09-11 13:50:11 69577 [Warning] Aborted connection 69577 to db: 'ic_wp' user: 'root' host: 'localhost' (Got an error writing communication packets)
root@mysql007:~# mysql --protocol=socket -e 'select * from 1C4Uonkwhe_options' ic_wp > /dev/null
ERROR 2013 (HY000) at line 1: Lost connection to server during query
root@mysql007:~#
root@mysql007:~# mysql --protocol=socket -e 'select * from 1C4Uonkwhe_options limit 1 offset 1402' ic_wp
ERROR 2013 (HY000) at line 1: Lost connection to server during query
root@mysql007:~#
root@mysql007:~# mysql --protocol=socket -e 'select option_id,option_name,option_value,autoload from 1C4Uonkwhe_options limit 1 offset 1402' ic_wp
ERROR 2013 (HY000) at line 1: Lost connection to server during query
root@mysql007:~#
root@mysql007:~# mysql --protocol=socket -e 'select option_id from 1C4Uonkwhe_options limit 1 offset 1402' ic_wp
+-----------+
option_id
+-----------+
16296351
+-----------+
root@mysql007:~#
root@mysql007:~# mysql --protocol=socket -e 'select option_value from 1C4Uonkwhe_options limit 1 offset 1402' ic_wp
ERROR 2013 (HY000) at line 1: Lost connection to server during query
root@mysql007:~#
root@mysql007:~# mysql --protocol=socket -e 'select * from 1C4Uonkwhe_options where option_id = 16296351' ic_wp
+-----------+----------------------+--------------+----------+
option_id option_name option_value autoload
+-----------+----------------------+--------------+----------+
16296351 z_taxonomy_image8905 yes
+-----------+----------------------+--------------+----------+
root@mysql007:~#
option_value right?
root@mysql007:~# mysql --protocol=socket -e 'select CHAR_LENGTH(option_value) from 1C4Uonkwhe_options where option_id = 16296351' ic_wp
+---------------------------+
CHAR_LENGTH(option_value)
+---------------------------+
0
+---------------------------+
root@mysql007:~# mysql --protocol=socket -e 'select HEX(option_value) from 1C4Uonkwhe_options where option_id = 16296351' ic_wp
+-------------------+
HEX(option_value)
+-------------------+
+-------------------+
root@mysql007:~#
root@mysql007:~# mysql --protocol=socket -e 'update 1C4Uonkwhe_options set option_value = "" where option_id = 16296351' ic_wp
root@mysql007:~# mysql --protocol=socket -e 'select * from 1C4Uonkwhe_options' ic_wp > /dev/null
ERROR 2013 (HY000) at line 1: Lost connection to server during query
root@mysql007:~#
root@mysql007:~# mysql --protocol=socket -e 'delete from 1C4Uonkwhe_options where option_id = 16296351' ic_wp
root@mysql007:~# mysql --protocol=socket -e 'select * from 1C4Uonkwhe_options' ic_wp > /dev/null
ERROR 2013 (HY000) at line 1: Lost connection to server during query
root@mysql007:~# mysqldump ic_wp > /dev/null
mysqldump: Error 2013: Lost connection to server during query when dumping table 1C4Uonkwhe_options at row: 1401
root@mysql007:~#
root@mysql007:~# mysql --protocol=socket -e 'insert into 1C4Uonkwhe_options VALUES(16296351,"z_taxonomy_image8905","","yes");' ic_wp
root@mysql007:~# mysqldump ic_wp > /dev/null
mysqldump: Error 2013: Lost connection to server during query when dumping table 1C4Uonkwhe_options at row: 1402
root@mysql007:~#
oot@mysql007:~# mysql --protocol=socket -e 'create table 1C4Uonkwhe_new_options like 1C4Uonkwhe_options;' ic_wp
root@mysql007:~# mysql --protocol=socket -e 'insert into 1C4Uonkwhe_new_options select * from 1C4Uonkwhe_options limit 1402 offset 0;' ic_wp
--- There is only 33 more records, not sure how to specify unlimited limit but 100 does the trick.
root@mysql007:~# mysql --protocol=socket -e 'insert into 1C4Uonkwhe_new_options select * from 1C4Uonkwhe_options limit 100 offset 1403;' ic_wp
root@mysql007:~# mysql --protocol=socket -e 'select * from 1C4Uonkwhe_new_options' ic_wp >/dev/null;
root@mysql007:~# mysql --protocol=socket -e 'select option_id from 1C4Uonkwhe_options where option_id not in (select option_id from 1C4Uonkwhe_new_options) ;' ic_wp
+-----------+
option_id
+-----------+
18405297
+-----------+
root@mysql007:~#
ORDER BY or you won t get consistent results.
root@mysql007:~# mysql --protocol=socket -e 'select option_id from 1C4Uonkwhe_options order by option_id limit 1 offset 1402' ic_wp ;
+-----------+
option_id
+-----------+
18405297
+-----------+
root@mysql007:~#
root@mysql007:~# mysql --protocol=socket -e 'select * from 1C4Uonkwhe_options where option_id = 18405297' ic_wp ;
ERROR 2013 (HY000) at line 1: Lost connection to server during query
root@mysql007:~#
root@mysql007:~# mysql --protocol=socket -e 'select CHAR_LENGTH(option_value) from 1C4Uonkwhe_options where option_id = 18405297' ic_wp ;
+---------------------------+
CHAR_LENGTH(option_value)
+---------------------------+
50814767
+---------------------------+
root@mysql007:~#
LONGTEXT
field so it should be able to handle it. But now I have a better idea of what
could be going wrong. The name of the option is rewrite_rules so it seems
like something is going wrong with the generation of that option.
I imagine there is some tweak I can make to allow MariaDB to cough up the value
(read_buffer_size? tmp_table_size?). But I ll start with checking in with
the database owner because I don t think 35,000 pages of rewrite rules is
appropriate for any site.
Welcome to the August 2024 report from the Reproducible Builds project!
Our reports attempt to outline what we ve been up to over the past month, highlighting news items from elsewhere in tech where they are related. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.
Table of contents:
The free software newspaper of record, Linux Weekly News, published an in-depth article based on Holger Levsen s talk, Reproducible Builds: The First Eleven Years which was presented at the recent DebConf24 conference in Busan, South Korea.
Titled The history, status, and plans for reproducible builds and written by Jake Edge, LWN s article not only summarises Holger s talk and clarifies its message but it links to external information as well. Holger s original talk can also be watched on the DebConf24 webpage (direct .webm link and his HTML slides are available also). There are also a significant number of comments on LWN s page as well.
Holger Levsen also headed a scheduled discussion session at DebConf24 on Preserving *other* build artifacts addressing a topic where a number of Debian packages are (or would like to) produce results that are neither the .deb files, the build logs nor the logs of CI tests. This is an issue for reproducible builds as this 4th type of build artifact are typically shipped within the binary .deb packages, and are invariably non-deterministic; thus making the .deb files unreproducible. (A direct .webm link and HTML slides are available).
Peter Eisentraut wrote a detailed blog post on the subject of The new PostgreSQL 17 make dist . Like many projects, the PostgreSQL database has previously pre-built parts of its GNU Autotools build system: the reason for this is a mix of convenience and traditional practice . Peter astutely notes that this arrangement in the build system is quite tricky as:
You need to carefully maintain the different states of clean source code , partially built source code , and fully built source code , and the commands to transition between them.However, Peter goes on to mention that:
a lot more attention is nowadays paid to the software supply chain. There are security and legal reasons for this. When users install software, they want to know where it came from, and they want to be sure that they got the right thing, not some fake version or some version of dubious legal provenance.And cites the XZ Utils backdoor as a reason to care about transparent and reproducible ways of distributing and communicating a source tarball and provenance. Because of this, intermediate build artifacts are now henceforth essentially disallowed from PostgreSQL distribution tarballs.
In Debian this month, 30 reviews of Debian packages were added, 17 were updated and 10 were removed this month adding to our knowledge about identified issues. One issue type was added by Chris Lamb, too. [ ]
In addition, an issue was filed to update the Salsa CI pipeline (used by 1,000s of Debian packages) to no longer test for reproducibility with reprotest s build_path variation. Holger Levsen provided a rationale for this change in the issue, which has already been made to the tests being performed by tests.reproducible-builds.org.
In Arch Linux this month, Jelle van der Waa published a short blog post on the topic of Investigating creating reproducible images with mkosi, motivated by the desire to make it possible for anyone to re-recreate the official Arch cloud image bit-by-bit identical on their own machine as per [the] reproducible builds definition. In addition, Jelle filed a patch for pacman, the Arch Linux package manager, to respect the SOURCE_DATE_EPOCH environment variable when installing a package.
In openSUSE news, Bernhard M. Wiedemann published another report for that distribution.
In Android news, the IzzyOnDroid project added 49 new rebuilder recipes and now features 256 total reproducible applications representing 21% of the total offerings in the repository. IzzyOnDroid is an F-Droid style repository for Android apps[:] applications in this repository are official binaries built by the original application developers, taken from their resp. repositories (mostly GitHub).
codegen-units = 16 default and resulting in a bug being filed in the Rust issue tracker. [ ]
armhf architecture, enquiring how they might gain temporary access to such a machine in order to perform some deeper testing. [ ]
diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb released versions 274, 275, 276 and 277, uploaded these to Debian, and made the following changes as well:
dumppdf(1) in more cases, avoiding situations where we fallback to a binary diff. [ ]RuntimeError exceptions when importing the PyPDF library so that it, or, crucially, its transitive dependencies, cannot not cause diffoscope to traceback at runtime and build time. [ ]marshal.load( ) of precompiled Python bytecode as it, alas, inherently unsafe. Replace for now with a brief summary of the code section of .pyc. [ ][ ]dumppdf(1). [ ]test_python.py: the fixture for this test is fixed. [ ][ ]test_zip text fixtures and definitions to support new changes to the Perl IO::Compress library. [ ]
There were a rather substantial number of improvements made to our website this month, including:
SOURCE_DATE_EPOCH [ ], fixing the test statistics link [ ], adjusting the Google Summer of Code application link [ ], a link to a Debian bug [ ], and removed a dead link to the debhelper utility [ ].jekyll-sitemap plugin to create a sitemap for the website. [ ] .lead directive appearing in the page. [ ]man2html patch on the SOURCE_DATE_EPOCH documentation page. [ ]agama-integration-tests (uses a random TCP-port number in .lock file)ca-certificates-mozilla:ca-certificates-mozilla-prebuiltcosmic (hash order issue)openSUSE (meta-issue to test reproducibility in the openSUSE Build Service)pop-launcher (parallelism-related issue)post (toolchain-issue, avoiding Rust parallelism)rpm-config-SUSE (date-related issue)rust (Rust toolchain issue)weblate (build gets stuck)bind9-docgnome-maps
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In August, a number of changes were made by Holger Levsen, including:
openssl-provider-legacy package for the Debian unstable environments for running diffoscope due to Debian bug #1078944. [ ][ ][ ][ ]armhf architecture nodes as being down due to proxy down. [ ][ ]index-buildinfo for the builtin-pho script with the -q switch. [ ]ruby-jekyll-sitemap package as it is now used in the website [ ], Roland Clobus updated the script to build Debian live images to treat openQA issues as warnings [ ], and Vagrant Cascadian marked the cbxi4b node as down [ ].
#reproducible-builds on irc.oftc.net.
rb-general@lists.reproducible-builds.org
Welcome to the July 2024 report from the Reproducible Builds project!
In our reports, we outline what we ve been up to over the past month and highlight news items in software supply-chain security more broadly. As always, if you are interested in contributing to the project, please visit our Contribute page on our website.
Table of contents:
rbtlog
Last month, we were very pleased to announce the upcoming Reproducible Builds Summit, set to take place from September 17th 19th 2024 in Hamburg, Germany. We are thrilled to host the seventh edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin and Athens. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving.
If you re interesting in joining us this year, please make sure to read the event page, which has more details about the event and location. We are very much looking forward to seeing many readers of these reports there.
a bootstrappable build is one that builds existing software from scratch for example, building GCC without relying on an existing copy of GCC. In 2023, the Guix project announced that the project had reduced the size of the binary bootstrap seed needed to build its operating system to just 357-bytes not counting the Linux kernel required to run the build process.The article goes onto to describe that now, the live-bootstrap project has gone a step further and removed the need for an existing kernel at all. and concludes:
The real benefit of bootstrappable builds comes from a few things. Like reproducible builds, they can make users more confident that the binary packages downloaded from a package mirror really do correspond to the open-source project whose source code they can inspect. Bootstrappable builds have also had positive effects on the complexity of building a Linux distribution from scratch [ ]. But most of all, bootstrappable builds are a boon to the longevity of our software ecosystem. It s easy for old software to become unbuildable. By having a well-known, self-contained chain of software that can build itself from a small seed, in a variety of environments, bootstrappable builds can help ensure that today s software is not lost, no matter where the open-source community goes from here
.deb files from our tests.reproducible-builds.org testing framework and the ones in the official Debian archive. Following up from a previous post on the reproducibility of Trisquel, Simon notes that typically [the] rebuilds do not match the official packages, even when they say the package is reproducible , Simon correctly identifies that the purpose of [these] rebuilds are not to say anything about the official binary build, instead the purpose is to offer a QA service to maintainers by performing two builds of a package and declaring success if both builds match.
However, Simon s post swiftly moves on to announce a new tool called debdistrebuild that performs rebuilds of the difference between two distributions in a GitLab pipeline and displays diffoscope output for further analysis.
Mehdi Keshani, Tudor-Gabriel Velican, Gideon Bot and Sebastian Proksch of the Delft University of Technology, Netherlands, have published a new paper in the ACM Software Engineering on a new tool to automatically reproduce Apache Maven artifacts:
Reproducible Central is an initiative that curates a list of reproducible Maven libraries, but the list is limited and challenging to maintain due to manual efforts. [We] investigate the feasibility of automatically finding the source code of a library from its Maven release and recovering information about the original release environment. Our tool, AROMA, can obtain this critical information from the artifact and the source repository through several heuristics and we use the results for reproduction attempts of Maven packages. Overall, our approach achieves an accuracy of up to 99.5% when compared field-by-field to the existing manual approach [and] we reveal that automatic reproducibility is feasible for 23.4% of the Maven packages using AROMA, and 8% of these packages are fully reproducible.
rattler-build, noting that as you can imagine, building packages reproducibly on Windows is the hardest challenge (so far!) . Nichita goes onto mention that the Apple ecosystem appears to be using ZERO_AR_DATE over SOURCE_DATE_EPOCH. [ ]
CreationDate metadata values.
SOURCE_DATE_EPOCH is configured, Sphinx projects that have configured their copyright notices using dynamic elements can produce nonsensical output under some circumstances. James query ended up generating a number of replies.
rb-core@lists.reproducible-builds.org and let us know . [ ]
rbtlog
On our mailing list, Fay Stegerman announced a new Reproducible Builds collaboration in the Android ecosystem:
We are pleased to announce Reproducible Builds, special client support and more in our repo : a collaboration between various independent interoperable projects: the IzzyOnDroid team, 3rd-party clients Droid-ify & Neo Store, and rbtlog (part of my collection of tools for Android Reproducible Builds) to bring Reproducible Builds to IzzyOnDroid and the wider Android ecosystem.
Congratulations to Marina Moore of the New York Tandon School of Engineering who has submitted her PhD thesis on Extending the Scalability, Flexibility and Responsiveness of Secure Software Update Systems. The introduction outlines its contributions to the field:
[S]oftware repositories are a vital component of software development and release, with packages downloaded both for direct use and to use as dependencies for other software. Further, when software is updated due to patched vulnerabilities or new features, it is vital that users are able to see and install this patched version of the software. However, this process of updating software can also be the source of attack. To address these attacks, secure software update systems have been proposed. However, these secure software update systems have seen barriers to widespread adoption. The Update Framework (TUF) was introduced in 2010 to address several attacks on software update systems including repository compromise, rollback attacks, and arbitrary software installation. Despite this, compromises continue to occur, with millions of users impacted by such compromises. My work has addressed substantial challenges to adoption of secure software update systems grounded in an understanding of practical concerns. Work with industry and academic communities provided opportunities to discover challenges, expand adoption, and raise awareness about secure software updates. [ ]
In Debian this month, 12 reviews of Debian packages were added, 13 were updated and 6 were removed this month adding to our knowledge about identified issues. A new toolchain issue type was identified as well, specifically ordering_differences_in_pkg_info.
i386 binaries on the i386 architecture is different when building i386 binaries under amd64. The fix was narrowed down to x87 excess precision, which can result in slightly different register choices when the compiler is hosted on x86_64 or i386 and a fix committed. [ ]
.apk files signed with the latest apksigner could no longer be verified as reproducible. Fay identified the issue as follows:
SinceShe documented multiple available workarounds and filed a bug in Google s issue tracker.build-tools>= 35.0.0-rc1, backwards-incompatible changes toapksignerbreakapksigcopieras it now by default forcibly replaces existing alignment padding and changed the default page alignment from 4k to 16k (same as Android Gradle Plugin >= 8.3, so the latter is only an issue when using older AGP). [ ]
Lastly, diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb uploaded version 272 and Mattia Rizzolo uploaded version 273 to Debian, and the following changes were made as well:
convert utility is from ImageMagick version 6.x. The command-line interface has seemingly changed with the 7.x series of ImageMagick. [ ]test_jpeg_image. [ ]identify_version method after a refactoring change in a previous commit. [ ]assert_diff in the test_openssh_pub_key package. [ ]ffmpeg version 7.x which adds some extra context to the diff. [ ]ed25519-format key [ ][ ][ ]debian/source to make sure not to pack the Python sdist directory into the binary Debian package. [ ]
There were a number of improvements made to our website this month, including:
SOURCE_DATE_EPOCH page to include instructions on how to create reproducible .zip files from within Python using the zipfile module. [ ]
rbtlog to the Tools page [ ] and IzzyOnDroid to the Projects page [ ], also ensuring that the latter page was always sorted regardless of the ordering within the input data files. [ ]
rattler-build to the Projects page. [ ][ ][ ]
armagetron (date)blaspp (hostname)cligen (GnuTLSs date)cloudflared (date)dpdk (Sphinx doctrees)fonttosfnt/xorg-x11-fonts (toolchain, date)gegl (build machine details)gettext-runtime (jar mtime)kf6-kirigami+kf6-qqc2-desktop-style (race-condition)kubernetes1.26 (backport upstream fix for random path)lapackpp (hostname)latex2html (nochecks)libdb-4_8 (.jar modification time)librcc (already merged upstream)libreoffice (strip .jar mtimes + clucene-core toolchain)maliit-keyboard (nocheck)nautilus (date)openblas (CPU type, fixed)openssl-3 (random-related issue)python-ruff (ASLR)python3 (date, parallelism/race)reproducible-faketools (0.5.2)sphinx (GZip modification time)sphinxcontrib (gzip mtime)
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In July, a number of changes were made by Holger Levsen, including:
bremner access to the ionos7 node. [ ][ ]#reproducible-builds on irc.oftc.net.
rb-general@lists.reproducible-builds.org
free, only half of the RAM was in use (125GB). About 4GB was
free, with the remainer used by the file cache.
I m used to seeing unexpected free RAM numbers like this and have been
assured that the kernel is simply not wasting RAM. If it s not needed, use it
to cache files to save on disk I/O. That make sense.
However why is the oomkiller being called instead of flushing the file
cache?
I came up with all kinds of amazing and wrong theories: maybe the RAM is
fragmented (is that even a thing?!?), maybe there is a spike in RAM and the
kernel can t flush the cache quickly enough (I really don t think that s a
thing). Maybe our kvm-manager has a weird bug (nope, but that didn t stop me
from opening a spurious bug
report).
I learned lots of cool things, like the oomkiller report includes a
table of the memory in use by each process (via the rss column) - and you
have to muliply that number by 4096 because it s in 4K pages.
That s how I discovered that the oomkiller was killing off processes with only
half the memory in use.
I also learned that lsof sometimes lists the same open file multiple times,
which made me think a bunch of files were being opened repeatedly causing a
memory problem, but really it amounted to nothing.
That last thing I learned, courtesy of an askubuntu
post is that
the /dev filesystem is allocated by default exactly half the RAM on the
system. What a coincidence! That is exactly how much RAM is useable on the
server.
And, on the server in question, that filesystem is full. What?!? Normally, that
filesystem should be using 0 bytes because it s not a real filesystem. But in
our case a process created a 127GB file there - it was only stopped because the
file system filled up.
Welcome to the June 2024 report from the Reproducible Builds project!
In our reports, we outline what we ve been up to over the past month and highlight news items in software supply-chain security more broadly. As always, if you are interested in contributing to the project, please visit our Contribute page on our website.
Table of contents:
We are very pleased to announce the upcoming Reproducible Builds Summit, set to take place from September 17th 19th 2024 in Hamburg, Germany.
We are thrilled to host the seventh edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin and Athens. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving.
If you re interesting in joining us this year, please make sure to read the event page which has more details about the event and location. We are very much looking forward to seeing many readers of these reports there.
Vagrant Cascadian will be holding a Reproducible Builds session as part of the monthly Guix patch review series on July 11th at 17:00 UTC.
These online events are intended to encourage everyone everyone becoming a patch reviewer and the goal of reviewing patches is to help Guix project accept contributions while maintaining our quality standards and learning how to do patch reviews together in a friendly hacking session.
An Industry Interview Study of Software Signing for Supply Chain Security was published by Kelechi G. Kalu, Tanmay Singla, Chinenye Okafor, Santiago Torres-Arias and James C. Davis of Electrical and Computer Engineering department of Purdue University, Indiana, USA, and is concerned with:
To understand software signing in practice, we interviewed 18 high-ranking industry practitioners across 13 organizations. We provide possible impacts of experienced software supply chain failures, security standards, and regulations on software signing adoption. We also study the challenges that affect an effective software signing implementation.
DiVerify: Diversifying Identity Verification in Next-Generation Software Signing was written by Chinenye L. Okafor, James C. Davis and Santiago Torres-Arias also of Purdue University and is interested in:
Code signing enables software developers to digitally sign their code using cryptographic keys, thereby associating the code to their identity. This allows users to verify the authenticity and integrity of the software, ensuring it has not been tampered with. Next-generation software signing such as Sigstore and OpenPubKey simplify code signing by providing streamlined mechanisms to verify and link signer identities to the public key. However, their designs have vulnerabilities: reliance on an identity provider introduces a single point of failure, and the failure to follow the principle of least privilege on the client side increases security risks. We introduce Diverse Identity Verification (DiVerify) scheme, which strengthens the security guarantees of next-generation software signing by leveraging threshold identity validations and scope mechanisms.
Felix Lagn hed published their thesis on the Integration of Reproducibility Verification with Diffoscope in GNU Make. This work, amongst some other results:
[ ] resulted in an extension of GNU make which is called rmake, where diffoscope a tool for detecting differences between a large number of file types was integrated into the workflow of make. rmake was later used to answer the posed research questions for this thesis. We found that different build paths and offsets are a big problem as three out of three tested Free and Open Source Software projects all contained these variations. The results also showed that gcc s optimisation levels did not affect reproducibility, but link-time optimisation embeds a lot of unreproducible information in build artefacts. Lastly, the results showed that build paths, build ID s and randomness are the three most common groups of variations encountered in the wild and potential solutions for some variations were proposed.
Lastly, Pol Dellaiera completed his master thesis on Reproducibility in Software Engineering at the University of Mons, Belgium, under the supervision of Dr. Tom Mens, professor and director of the Software Engineering Lab.
The thesis serves as an introduction to the concept of reproducibility in software engineering, offering a comprehensive overview of formalizations using mathematical notations for key concepts and an empirical evaluation of several key tools. By exploring various case studies, methodologies and tools, the research aims to provide actionable insights for practitioners and researchers alike.
In Debian this month, 4 reviews of Debian packages were added, 11 were updated and 14 were removed this month adding to our knowledge about identified issues. Only one issue types was updated, though, explaining that we don t vary the build path anymore.
In openSUSE news, Bernhard M. Wiedemann published another report for that distribution.
In NixOS, with the 24.05 release out, it was again validated that our minimal ISO is reproducible by building it on a virtual machine with no access to the binary cache.
qutebrowser, samba and systemd), Chris Lamb filing Debian bug #1074214 against the fastfetch package and Arnout Engelen proposing fixes to refind and for the Scala compiler [ .
Lastly, diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb uploaded two versions (270 and 271) to Debian, and made the following changes as well:
Build-Depends on liblz4-tool in order to fix Debian bug #1072575. [ ]zipdetails version 4.004 that is shipped with Perl 5.40. [ ]
There were a number of improvements made to our website this month, including Akihiro Suda very helpfully making the <h4> elements more distinguishable from the <h3> level [ ][ ] as well as adding a guide for Dockerfile reproducibility [ ]. In addition Fay Stegerman added two tools, apksigcopier and reproducible-apk-tools, to our Tools page.
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In June, a number of changes were made by Holger Levsen, including:
virt(32 64)c-armhf nodes as down. [ ]osuosl4 node in order to debug a regression on the ppc64el architecture. [ ]osuosl4 node. [ ][ ]/etc/default/jenkins file with changes performed upstream [ ] and changed how configuration files are handled on the rb-mail1 host. [ ], whilst Vagrant Cascadian documented the failure of the virt32c and virt64c nodes after initial investigation [ ].
#reproducible-builds on irc.oftc.net.
rb-general@lists.reproducible-builds.org
Next.