Russ Allbery: Review: Moose Madness
Publisher: | Kalikoi |
Copyright: | May 2021 |
ASIN: | B094HGT1ZB |
Format: | Kindle |
Pages: | 68 |
Publisher: | Kalikoi |
Copyright: | May 2021 |
ASIN: | B094HGT1ZB |
Format: | Kindle |
Pages: | 68 |
Books
Elif Batuman: Either/Or (2022)
Stella Gibbons: Cold Comfort Farm (1932)
Michel Faber: Under The Skin (2000)
Wallace Stegner: Crossing to Safety (1987)
Gustave Flaubert: Madame Bovary (1857)
Rachel Cusk: Outline (2014)
Sara Gran: The Book of the Most Precious Substance (2022)
Anonymous: The Railway Traveller s Handy Book (1862)
Natalie Hodges: Uncommon Measure: A Journey Through Music, Performance, and the Science of Time (2022)
Gary K. Wolf: Who Censored Roger Rabbit? (1981)
Films Recent releases
openssh-client-gssapi
and openssh-server-gssapi
packages in
unstable, and if you use either GSS-API authentication or key exchange then
you should install the corresponding package in order for upgrades to
trixie+1 to work correctly. I ll write a release note once this has reached testing.
Multiple identical results from getaddrinfo
I expect this is really a bug in a chroot creation script somewhere, but I
haven t been able to track down what s causing it yet. My sbuild chroots,
and apparently Lucas Nussbaum s as well, have an /etc/hosts
that looks
like this:
$ cat /var/lib/schroot/chroots/sid-amd64/etc/hosts
127.0.0.1 localhost
127.0.1.1 [...]
127.0.0.1 localhost ip6-localhost ip6-loopback
::1
rather than 127.0.0.1
; but things
mostly work anyway, since most code doesn t really care which protocol it
uses to talk to localhost. However, a few things try to set up test
listeners by calling getaddrinfo("localhost", ...)
and binding a socket
for each result. This goes wrong if there are duplicates in the resulting
list, and the test output is typically very confusing: it looks just like
what you d see if a test isn t tearing down its resources correctly, which
is a much more common thing for a test suite to get wrong, so it took me a
while to spot the problem.
I ran into this in both python-asyncssh
(#1052788, upstream
PR) and Ruby
(ruby3.1/#1069399,
ruby3.2/#1064685,
ruby3.3/#1077462, upstream
PR). The latter took a while
since Ruby isn t one of my languages, but hey, I ve tackled much harder
side quests. I
NMUed ruby3.1 for this since it was showing up as a blocker for openssl
testing migration, but haven t done the other active versions (yet, anyway).
OpenSSL vs. cryptography
I tend to care about openssl migrating to testing promptly, since openssh
uploads have a habit of getting stuck on it otherwise.
Debian s OpenSSL packaging recently split out some legacy code (cryptography
that s no longer considered a good idea to use, but that s sometimes needed
for compatibility) to an openssl-legacy-provider
package, and added a
Recommends on it. Most users install Recommends, but package build
processes don t; and the Python cryptography
package requires this code
unless you set the CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1
environment variable,
which caused a bunch of packages that build-depend on it to fail to build.
After playing whack-a-mole setting that environment variable in a few
packages build process, I decided I didn t want to be caught in the middle
here and filed an upstream
issue to see if I could
get Debian s OpenSSL team and cryptography s upstream talking to each other
directly. There was some moderately spirited discussion and the issue
remains open, but for the time being the OpenSSL team has effectively
reverted the
change
so it s no longer a pressing problem.
GCC 14 regressions
Continuing from last month, I fixed build
failures in pccts (NMU) and
trn4.
Python team
I upgraded alembic, automat, gunicorn, incremental, referencing, pympler
(fixing compatibility with Python >=
3.10), python-aiohttp, python-asyncssh
(fixing CVE-2023-46445,
CVE-2023-46446, and
CVE-2023-48795), python-avro,
python-multidict (fixing a build failure with GCC
14), python-tokenize-rt, python-zipp,
pyupgrade, twisted (fixing CVE-2024-41671
and CVE-2024-41810), zope.exceptions,
zope.interface, zope.proxy, zope.security, and zope.testrunner to new
upstream versions. In the process, I added myself to Uploaders
for
zope.interface; I m reasonably comfortable with the Zope Toolkit and I seem
to be gradually picking up much of its maintenance in Debian.
A few of these required their own bits of yak-shaving:
Multi-Arch: foreign
tagging
(python-importlib-metadata,
python-typing-extensions,
python-zipp).
I fixed build failures in pipenv,
python-stdlib-list,
psycopg3, and
sen, and fixed autopkgtest failures in
autoimport
(upstream PR),
python-semantic-release
and rstcheck.
Upstream for zope.file (not in Debian) filed an issue about a test failure
with Python 3.12,
which I tracked down to a Python 3.12 compatibility
PR in zope.security.
I made python-nacl build reproducibly (upstream
PR).
I moved aliased files from /
to /usr
in timekpr-next
(#1073722).
Installer team
I applied a patch from Ubuntu to make os-prober support building with the
noudeb
profile (#983325).
Series: | The Meaning Wars #1 |
Publisher: | Michelle Browne |
Copyright: | 2012, 2021 |
Printing: | 2021 |
ASIN: | B0075G7GEA |
Format: | Kindle |
Pages: | 85 |
pyproject.toml
files, I wanted to investigate how the popularity of build
backends used in pyproject.toml
files evolved over the years since the
introduction of PEP-0517 in 2015.
Getting the data
Tom Forbes provides a huge
dataset that contains information
about every file within every release uploaded to PyPI. To
get the current dataset, we can use:
curl -L --remote-name-all $(curl -L "https://github.com/pypi-data/data/raw/main/links/dataset.txt")
describe select * from '*.parquet';
column_name column_type null
varchar varchar varchar
project_name VARCHAR YES
project_version VARCHAR YES
project_release VARCHAR YES
uploaded_on TIMESTAMP YES
path VARCHAR YES
archive_path VARCHAR YES
size UBIGINT YES
hash BLOB YES
skip_reason VARCHAR YES
lines UBIGINT YES
repository UINTEGER YES
11 rows 6 columns
pyproject.toml
files that are in the project s root directory. Since we ll still have to
download the actual files, we need to get the path
and the repository
to
construct the corresponding URL to the mirror that contains all files in a
bunch of huge git repositories. Some files are not available on the mirrors; to
skip these, we only take files where the skip_reason
is empty. We also care
about the timestamp of the upload (uploaded_on
) and the hash
to avoid
processing identical files twice:
select
path,
hash,
uploaded_on,
repository
from '*.parquet'
where
skip_reason == '' and
lower(string_split(path, '/')[-1]) == 'pyproject.toml' and
len(string_split(path, '/')) == 5
order by uploaded_on desc
repository
and path
, we can now construct an URL from which we
can fetch the actual file for further processing:
url = f"https://raw.githubusercontent.com/pypi-data/pypi-mirror- repository /code/ path "
pyproject.toml
files and parse them to read
the build-backend
into a dictionary mapping the file-hash
to the build
backend. Downloads on GitHub are rate-limited, so downloading 1.2M files
will take a couple of days. By skipping files with a hash we ve already
processed, we can avoid downloading the same file more than once, cutting the
required downloads by circa 50%.
Results
Assuming the data is complete and my analysis is sound, these are the findings:
There is a surprising amount of build backends in use, but the overall amount
of uploads per build backend decreases quickly, with a long tail of single
uploads:
>>> results.backend.value_counts()
backend
setuptools 701550
poetry 380830
hatchling 56917
flit 36223
pdm 11437
maturin 9796
jupyter 1707
mesonpy 625
scikit 556
...
postry 1
tree 1
setuptoos 1
neuron 1
avalon 1
maturimaturinn 1
jsonpath 1
ha 1
pyo3 1
Name: count, Length: 73, dtype: int64
pyproject.toml
files. During that early
period, Flit started as the most popular build backend, but was eventually
displaced by Setuptools and Poetry.
Between 2020 and 2020, the overall usage of pyproject.toml
files increased
significantly. By the end of 2022, the share of Setuptools peaked at 70%.
After 2020, other build backends experienced a gradual rise in popularity.
Amongh these, Hatch emerged as a notable contender, steadily gaining
traction and ultimately stabilizing at 10%.
We can also look into the absolute distribution of build backends over time:
pyproject.toml
files yourself.
/etc/network/interfaces
is a fairly basic (if powerful) mechanism for configuring network interfaces. NetworkManager is a better bet for dynamic hosts (i.e. clients), and systemd-network
seems to be a good choice for servers (I m gradually moving machines over to it). Netplan tries to provide a unified mechanism for configuring both with a single configuration language. A noble aim, but I don t see a lot of benefit for anything I use - my NetworkManager hosts are highly dynamic (so no need to push shared config) and systemd-network
(or /etc/network/interfaces
) works just fine on the other hosts. I m told Netplan has more use with more complicated setups, e.g. when OpenVSwitch is involved.
.deb
and chisel it into smaller components, which then helps separate out dependencies rather than pulling in as much as the original .deb
would. This was touted as being useful, in particular, for building targeted containers. Definitely appealing over custom built userspaces for containers, but in an ideal world I think we d want the information in the main packaging and it becomes a lot of work.
Welcome to the June 2023 report from the Reproducible Builds project
In our reports, we outline the most important things that we have been up to over the past month. As always, if you are interested in contributing to the project, please visit our Contribute page on our website.
Corrupted build environments can deliver compromised cryptographically signed binaries. Several exploits in critical supply chains have been demonstrated in recent years, proving that this is not just theoretical. The most well secured build environments are still single points of failure when they fail. [ ] This talk will focus on the state of the art from several angles in related Free and Open Source Software projects, what works, current challenges and future plans for building trustworthy toolchains you do not need to trust.Hosted by the Software Freedom Conservancy and taking place in Portland, Oregon, FOSSY aims to be a community-focused event: Whether you are a long time contributing member of a free software project, a recent graduate of a coding bootcamp or university, or just have an interest in the possibilities that free and open source software bring, FOSSY will have something for you . More information on the event is available on the FOSSY 2023 website, including the full programme schedule.
The 2020 Solarwinds attack was a tipping point that caused a heightened awareness about the security of the software supply chain and in particular the large amount of trust placed in build systems. Reproducible Builds (R-Bs) provide a strong foundation to build defenses for arbitrary attacks against build systems by ensuring that given the same source code, build environment, and build instructions, bitwise-identical artifacts are created.However, in contrast to other papers that touch on some theoretical aspect of reproducible builds, the authors paper takes a different approach. Starting with the observation that much of the software industry believes R-Bs are too far out of reach for most projects and conjoining that with a goal of to help identify a path for R-Bs to become a commonplace property , the paper has a different methodology:
We conducted a series of 24 semi-structured expert interviews with participants from the Reproducible-Builds.org project, and iterated on our questions with the reproducible builds community. We identified a range of motivations that can encourage open source developers to strive for R-Bs, including indicators of quality, security benefits, and more efficient caching of artifacts. We identify experiences that help and hinder adoption, which heavily include communication with upstream projects. We conclude with recommendations on how to better integrate R-Bs with the efforts of the open source and free software community.A PDF of the paper is now available, as is an entry on the CISPA Helmholtz Center for Information Security website and an entry under the TeamUSEC Human-Centered Security research group.
comp.unix.programming
. Larry notes that it starts with Jayan asking about comparing binaries that might have difference in their embedded timestamps (that is, perhaps, Foreshadowing diffoscope, amiright? ) and goes on to observe that:
The antagonist is David Schwartz, who correctly says There are dozens of complex reasons why what seems to be the same sequence of operations might produce different end results, but goes on to say I totally disagree with your general viewpoint that compilers must provide for reproducability [sic]. Dwight Tovey and I (Larry Doolittle) argue for reproducible builds. I assert Any program especially a mission-critical program like a compiler that cannot reproduce a result at will is broken. Also it s commonplace to take a binary from the net, and check to see if it was trojaned by attempting to recreate it from source.
SOURCE_DATE_EPOCH
environment variable [ ], Chris Lamb made it easier to parse our summit announcement at a glance [ ], Mattia Rizzolo added the summit announcement at a glance [ ] itself [ ][ ][ ] and Rahul Bajaj added a taxonomy of variations in build environments [ ].
randomness_in_documentation_generated_by_mkdocs
toolchain issue was added by Chris Lamb [ ], and the deterministic
flag on the paths_vary_due_to_usrmerge
issue as we are not currently testing usrmerge
issues [ ] issues.
bullseye
, bookworm
, trixie
and sid
, but he also mentioned amongst many changes that not only are the non-free
images being built (and are reproducible) but that the live images are generated officially by Debian itself. [ ]
CFLAGS
environment variable. [ ]
bcachefs
(sort find / filesys)build-compare
(reports files as identical)build-time
(toolchain date)cockpit
(merged, gzip mtime)gcc13
(gcc13 toolchain LTO parallelism)ghc-rpm-macros
(toolchain parallelism)golangcli-lint
(date)gutenprint
(date+time)mage
(date (golang))mumble
(filesys)pcr
(date)python-nss
(drop sphinx .doctrees)python310
(merged, bisected+backported)warpinator
(merged, date)xroachng
(date)elinks
.multipath-tools
.mkdocstrings-python-handlers
.fribidi
.jtreg7
.python-bitstring
(forwarded upstream).gradle-kotlin-dsl
.libsdl-console
.kawari8
.freetds
.gbrowse
.bglibs
.advi
.afterstep
.simstring
.manderlbot
.erlang-proper
.comedilib
.libint
.newlib
.binutils-msp430
.c-munipack
.python-marshmallow-sqlalchemy
.mplayer
.menu
.mini-buildd
.pnetcdf
.liblopsub
.wcc
.shotcut
.icu
.libapache-poi-java
.atf
.valgrind
.amd64
, armhf
, and i386
architectures to Debian bookworm, with the exception of the Jenkins host itself which will be upgraded after the release of Debian 12.1. In addition, Mattia Rizzolo updated the email configuration for the @reproducible-builds.org
domain to correctly accept incoming mails from jenkins.debian.net
[ ] as well as to set up DomainKeys Identified Mail (DKIM) signing [ ]. And working together with Holger, Mattia also updated the Jenkins configuration to start testing Debian trixie which resulted in stopped testing Debian buster. And, finally, Jan-Benedict Glaw contributed patches for improved NetBSD testing.
#reproducible-builds
on irc.oftc.net
.
rb-general@lists.reproducible-builds.org
Next.