Combining BGP confederations and AS override can potentially
create a BGP routing loop, resulting in an indefinitely expanding AS path.
BGP confederation is a technique used to reduce the number of iBGP sessions
and improve scalability in large autonomous systems (AS). It divides an AS into
sub-ASes. Most eBGP rules apply between sub-ASes, except that next-hop, MED, and
local preferences remain unchanged. The AS path length ignores contributions
from confederation sub-ASes. BGP confederation is rarely used and BGP route
reflection is typically preferred for scaling.
AS override is a feature that allows a router to replace the ASN of a
neighbor in the AS path of outgoing BGP routes with its own. It s useful when
two distinct autonomous systems share the same ASN. However, it interferes with
BGP s loop prevention mechanism and should be used cautiously. A safer
alternative is the allowas-in directive.1
In the example below, we have four routers in a single confederation, each in
its own sub-AS. R0 originates the 2001:db8::1/128 prefix. R1, R2, and
R3 forward this prefix to the next router in the loop.
BGP routing loop using a confederation
The router configurations are available in a Git repository. They are
running Cisco IOS XR. R2 uses the following configuration for BGP:
The session with R3 uses both as-override and next-hop-self directives.
The latter is only necessary to make the announced prefix valid, as there is no
IGP in this example.2
Here s the sequence of events leading to an infinite AS path:
R1 selects it as the best path, forwarding it to R2 with AS
path (64501 64500).
R2 selects it as the best path, forwarding it to R3 with AS
path (64500 64501 64502).
R3 selects it as the best path. It would forward it to R1 with AS path
(64503 64502 64501 64500), but due to AS override, it substitutes R1 s
ASN with its own, forwarding it with AS path (64503 64502 64503 64500).
R1 accepts the prefix, as its own ASN is not in the AS path. It compares
this new prefix with the one from R0. Both (64500) and (64503 64502
64503 64500) have the same length because confederation sub-ASes don t
contribute to AS path length. The first tie-breaker is the router ID.
R0 s router ID (1.0.0.4) is higher than R3 s (1.0.0.3). The new
prefix becomes the best path and is forwarded to R2 with AS path (64501
64503 64501 64503 64500).
R2 receives the new prefix, replacing the old one. It selects it as the
best path and forwards it to R3 with AS path (64502 64501 64502 64501
64502 64500).
R3 receives the new prefix, replacing the old one. It selects it as the
best path and forwards it to R0 with AS path (64503 64502 64503 64502
64503 64502 64500).
R1 receives the new prefix, replacing the old one. Again, it competes with
the prefix from R0, and again the new prefix wins due to the lower router
ID. The prefix is forwarded to R2 with AS path (64501 64503 64501 64503
64501 64503 64501 64500).
A few iterations later, R1 views the looping prefix as follows:4
RP/0/RP0/CPU0:R1#showbgpipv6u2001:db8::1/128bestpath-compare
BGP routing table entry for 2001:db8::1/128Last Modified: Jul 28 10:23:05.560 for 00:00:00Paths: (2 available, best #2) Path #1: Received by speaker 0 Not advertised to any peer (64500) 2001:db8::1:0 from 2001:db8::1:0 (1.0.0.4), if-handle 0x00000000 Origin IGP, metric 0, localpref 100, valid, confed-external Received Path ID 0, Local Path ID 0, version 0 Higher router ID than best path (path #2) Path #2: Received by speaker 0 Advertised IPv6 Unicast paths to peers (in unique update groups): 2001:db8::2:1 (64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64503 64502 64500) 2001:db8::4:0 from 2001:db8::4:0 (1.0.0.3), if-handle 0x00000000 Origin IGP, metric 0, localpref 100, valid, confed-external, best, group-best Received Path ID 0, Local Path ID 1, version 37 best of AS 64503, Overall best
There s no upper bound for an AS path, but BGP messages have size limits (4096
bytes per RFC 4271 or 65535 bytes per RFC 8654). At some
point, BGP updates can t be generated. On Cisco IOS XR, the BGP process crashes
well before reaching this limit.5
The main lessons from this tale are:
never use BGP confederations under any circumstances, and
be cautious of features that weaken BGP routing loop detection.
When using BGP confederations with Cisco IOS XR, use
allowconfedas-in instead. It s available since IOS XR 7.11.
Using BGP confederations is already inadvisable. If you don t use the
same IGP for all sub-ASes, you re inviting trouble! However, the scenario
described here is also possible with an IGP.
When an AS path segment is composed of ASNs from a confederation, it
is displayed between parentheses.
By default, IOS XR paces eBGP updates. This is controlled by the
advertisement-interval directive. Its default value is 30 seconds for eBGP
peers (even in the same confederation). R1 and R2 set this value to 0,
while R3 sets it to 2 seconds. This gives some time to watch the AS path
grow.
This is CSCwk15887. It only happens when using as-override on an
AS path with a too long AS_CONFED_SEQUENCE. This should be fixed around
24.3.1.
updating old chroot in sbuild that predates usrmerge.
I have been lazy and haven't been updating the chroot; but I no longer could, so had to resolve this issue about usrmerge.
There was a file /etc/unsupported-skip-usrmerge-conversion that usrmerge package errored out on; it seemed like it means
a mark of not doing usrmerge conversion, because sbuild is a system for autobuilding and staying in split /usr for the duration of the release might be better.
I don't fully understand the rationale of the file and effects, but I just deleted it and jumped into merged usr.
DebConf24, the 25th edition of the Debian
conference is taking place in Pukyong National University at Busan, Republic of
Korea.
Thanks to the hard work of its organizers, it again will be an interesting and
fruitful event for attendees.
We would like to warmly welcome the sponsors of DebConf24, and introduce them to
you.
We have three Platinum sponsors.
Proxmox is the first Platinum sponsor.
Proxmox provides powerful and user-friendly Open Source server software.
Enterprises of all sizes and industries use Proxmox solutions to deploy
efficient and simplified IT infrastructures, minimize total cost
of ownership, and avoid vendor lock-in.
Proxmox also offers commercial support, training services, and an extensive
partner ecosystem to ensure business continuity for its customers.
Proxmox Server Solutions GmbH was established in 2005
and is headquartered in Vienna, Austria.
Proxmox builds its product offerings on top of the Debian operating system.
Our second Platinum sponsor is Infomaniak.
Infomaniak is an independent cloud service provider recognised throughout
Europe for its commitment to privacy, the local economy and the environment.
Recording growth of 18% in 2023, the company is developing a suite of online
collaborative tools and cloud hosting, streaming, marketing and events
solutions.
Infomaniak uses exclusively renewable energy, builds its own data centers and
develops its solutions in Switzerland, without relocating.
The company powers the website of the Belgian radio and TV
service (RTBF) and provides streaming for more than 3,000
TV and radio stations in Europe.
Wind River is our third Platinum sponsor.
For nearly 20 years, Wind River has led in commercial Open Source Linux
solutions for mission-critical enterprise edge computing.
With expertise across aerospace, automotive, industrial, telecom, and more,
the company is committed to Open Source through initiatives like eLxr, Yocto,
Zephyr, and StarlingX.
Our Gold sponsors are:
Ubuntu,
the Operating System delivered by Canonical.
Freexian,
a services company specialized in Free Software and in particular Debian
GNU/Linux, covering consulting, custom developments, support and training.
Freexian has a recognized Debian expertise thanks to the participation of
Debian developers.
Lenovo,
a global technology leader manufacturing a wide portfolio of connected
products including smartphones, tablets, PCs and workstations as well as
AR/VR devices, smart home/office and data center solutions.
Korea Tourism Organization,
which purpose is to advance tourism as a key driver for national economic
growth and enhancement of national welfare and intends to be a public
organization that makes the Korean people happier; it promotes national wealth
through tourism.
Busan IT Industry Promotion Agency,
an industry promotion organization that contributes to the innovation of the
digital economy with the power of IT and CT and
supports the ecosystem for innovative local startups and companies to grow.
Microsoft,
who enables digital transformation for the era of an intelligent cloud and an
intelligent edge.
Its mission is to empower every person and every organization on the planet to
achieve more.
doubleO,
a company that specializes in consulting and developing empirical services
using big data analysis and artificial intelligence.
doubleO provides a variety of data-centered services together with small and
medium-sized businesses in Busan/Gyeongnam.
Our Silver sponsors are:
Roche, a major international pharmaceutical
provider and research company dedicated to personalized healthcare.
Two Sigma, rigorous inquiry, technology
data science, and invention to bring science to finance and help solve the
toughest challenges across financial services.
Arm: leading technology provider of processor
IP, Arm powered solutions have been supporting innovation for
more than 30 years and are deployed in over 280 billion chips to date.
Google, one of the largest technology companies in
the world, providing a wide range of Internet-related services and products
such as online advertising technologies, search, cloud computing, software,
and hardware.
FSIJ, the Free Software Initiative
of Japan, a non-profit organization dedicated to supporting Free Software
growth and development.
Busan Tourism Organisation:
leading public corporation that generates social and economic values in Busan
tourism industry, developing tourism resources in accordance with government
policies and invigorate tourism industry.
Civil Infrastructure Platform,
a collaborative project hosted by the Linux Foundation,
establishing an open source base layer of industrial grade software.
Collabora, a global consultancy delivering
Open Source software solutions to the commercial world.
Matanel Foundation, which operates in Israel,
as its first concern is to preserve the cohesion of a society and a nation
plagued by divisions.
A special thanks to the
Pukyong National University, our Venue
Partner and our Network Partners KOREN
and KREONET!
Thanks to all our sponsors for their support!
Their contributions make it possible for a large number of Debian contributors
from all over the globe to work together, help and learn from each other in
DebConf24.
A new minor release 0.4.23 of RQuantLib
just arrived at CRAN earlier
today, and will be uploaded to Debian in due course.
QuantLib is a rather
comprehensice free/open-source library for quantitative
finance. RQuantLib
connects (some parts of) it to the R environment and language, and has
been part of CRAN for more than
twenty-two years (!!) as it was one of the first packages I
uploaded.
This release of RQuantLib
updates to QuantLib version 1.35
released this morning. It accommodates some removals following earlier
deprecations, and also updates most of the code in the function for a
more readable and compact form of creating shared pointers via
make_shared() along with auto.
Changes in RQuantLib version 0.4.23 (2024-07-23)
Adjustments for QuantLib 1.35 and removal of deprecated code (in
utility functions and dividend case of vanilla options)
Adjustments for new changes in QuantLib 1.35
Refactoring most C++ files making more use of both
auto and make_shared to simplify and shorten
expressions
My work on overhauling dhcpcd as the prime replacement for ISC's discontinued DHCP client is done. The package has achieved stability, both upstream and at Debian. The only remaining points are bug #1038882 to swap the Priorities of isc-dhcp-client and dhcpcd-base in the repository's override, and swaping ifupdown's search order to put dhcpcd first.
Meanwhile, ifupdown's de-facto maintainer prompted me to sollicit opinions on which of the 4 ifupdown implementations should ship with a minimal installation for Trixie. This, in turn, re-opened the debate of what should be Debian's default network configuation framework (see the thread starting with this post).
networkd
Given how most Debian ports (except for Hurd) ship with systemd, which includes a disabled networkd by standard, many people in the thread feel that this should become the default network configuration tool for minimal installations. As it happens, most of my hosts fit that scenario, so I figured that I would give another go at testing networkd on one host.
I used the following minimalistic /etc/systemd/network/dhcp.network:
[Match]
Name=en* wl*
[Network]
DHCP=yes
This correctly configured IPv4 via DHCP, with the small caveat that it doesn't update /etc/resolv.conf without installing resolvconf or systemd-resolved.
However, networkd's default IPv6 settings really are not suitable for public consumption. The key issues (see Bug #1076432):
Temporary addresses are not enabled by default. Worse, the setting is ignored if it was enabled by sysctl during bootup. This is a major privacy issue. Adding IPv6PrivacyExtensions=yes to the above exposed another issue: instead of using the fe80 address generated by the kernel, networkd adds a new one.
Networkd uses EUI64 addresses by default. This is another major privacy issue, since EUI64 addresses are forensically traceable to the interface's MAC address. Worse, the setting is ignored if stable-privacy was enabled by sysctl during bootup. To top it all, networkd does stable-privacy using systemd's time-proven brain-dead approach of reinventing the wheel: instead of merely setting the kernel's address generation mode to 3 and letting it configure the secret address, it expects the secret address to be spelled out in the systemd unit.
Conclusion: networkd works well enough for someone configuring an IPv4-only network from 20 years ago, but is utterly inadequate for IPv6 or dual-stack installations, doubly so on a software distribution that claims to care about privacy and network security.
Here comes the 3rd article of the 5-episode blog post series on Polis, written by Guido Berh rster, member of staff at my company Fre(i)e Software GmbH.Enjoy also this read on Guido's work on Polis,
Mike
Table of Contents of the Blog Post Series
Issues extending Polis and adjusting our goals (this article)
Creating (a) new frontend(s) for Polis
Current status and roadmap
Polis - Issues extending Polis and adjusting our Goals
After the initial implementation of limited branding support, user
feedback and the involvement of an UX designer lead to the conclusion that we
needed more far-reaching changes to the user interface in order to reduce
visual clutter, rearrange and improve UI elements, and provide better
integration with the websites in which conversations are embedded.
Challenges when visualizing Data in Polis
Polis visualizes groups using a spatial projection of users based on
similarities in voting behavior and places them in two to five groups
using a clustering algorithm.
During our testing and evaluation users were rarely able to interpret the
visualization and often intuitively made incorrect assumptions e.g. by
associating the filled area of a group with its significance or size. After
consultation with a member of the Multi-Agent Systems (MAS) Group at the
University of Groningen we chose to temporarily replace the visualization
offered by Polis with simple bar charts representing agreement or disagreement
with statements of a group or the majority. We intend to revisit this and
explore different forms of visualization at a later point in time.
The different factors playing into the weight attached to statements which
determine the pseuodo-random order in which they are presented for voting
( comment routing ) proved difficult to explain to stakeholders and users and
the admission of the ad-hoc and heuristic nature of the used algorithm1 by
Polis authors lead to the decision to temporarily remove this feature.
Instead, statements should be placed into three groups, namely
metadata questions,
seed statements,
and participant statements
Statements should then be sorted by group but in a fully randomized order
within the group so that metadata questions would be presented before seed
statements which would be presented before participant s statements. This
simpler method was deemed sufficient for the scale of our pilot projects,
however we intend to revisit this decision and explore different methods of
comment routing in cooperation with our scientific partners at a later point
in time.
An evaluation of the requirements for implementing mandatory authentication and
adding support for additional authentication methods to Polis showed that
significant changes to both the administration and participation frontend were
needed due to a lack of an abstraction layer or extension mechanism and the
current authentication providers being hardcoded in many parts of the code
base.
A New Frontend is born: Particiapp
Based on the implementation details of the participation frontend, the invasive
nature of the changes required, and the overhead of keeping up with active
upstream development it became clear that a different, more flexible approach
to development was needed. This ultimately lead to the creation of
Particiapp, a new Open Source project providing the building blocks and
necessary abstraction layers for rapid protoyping and experimentation with
different fontends which are compatible with but independent from Polis.
Small, Christopher T., Bjorkegren, Michael, Erkkil , Timo, Shaw, Lynette and Megill, Colin (2021). Polis: Scaling deliberation by mapping high dimensional opinion spaces. Recerca. Revista de Pensament i An lisi, 26(2), pp. 1-26.
The Rcpp Core Team is once again pleased to announce a new release
(now at 1.0.13) of the Rcpp package.
It arrived on CRAN earlier
today, and has since been uploaded to Debian. Windows and macOS builds
should appear at CRAN in the next few days, as will builds in different
Linux distribution and of course r2u should catch up
tomorrow too. The release was uploaded last week, but not only does Rcpp always gets flagged because of the
grandfathered .Call(symbol) but CRAN also found two packages
regressing which then required them to take five days to get back to
us. One issue was known; another
did not reproduce under our tests against over 2800 reverse dependencies
leading to the eventual release today. Yay. Checks are good and
appreciated, and it does take time by humans to review them.
This release continues with the six-months January-July cycle started
with release
1.0.5 in July 2020. As a reminder, we do of course make interim
snapshot dev or rc releases available via the Rcpp drat repo as well as
the r-universe page and
repo and strongly encourage their use and testing I run my systems
with these versions which tend to work just as well, and are also fully
tested against all reverse-dependencies.
Rcpp has long established itself
as the most popular way of enhancing R with C or C++ code. Right now,
2867 packages on CRAN depend on
Rcpp for making analytical code go
faster and further, along with 256 in BioConductor. On CRAN, 13.6% of
all packages depend (directly) on Rcpp, and 59.9% of all compiled packages
do. From the cloud mirror of CRAN (which is but a subset of all CRAN
downloads), Rcpp has been downloaded
86.3 million times. The two published papers (also included in the
package as preprint vignettes) have, respectively, 1848 (JSS, 2011) and 324 (TAS, 2018)
citations, while the the book (Springer useR!,
2013) has another 641.
This release is incremental as usual, generally preserving existing
capabilities faithfully while smoothing our corners and / or extending
slightly, sometimes in response to changing and tightened demands from
CRAN or R standards. The move towards a
more standardized approach for the C API of R leads to a few changes;
Kevin did most of the PRs for this. Andrew Johnsom also provided a very
nice PR to update internals taking advantage of variadic templates.
The full list below details all changes, their respective PRs and, if
applicable, issue tickets. Big thanks from all of us to all
contributors!
Changes in
Rcpp release version 1.0.13 (2024-07-11)
Changes in Rcpp API:
Set R_NO_REMAP if not already defined (Dirk in #1296)
Add variadic templates to be used instead of generated code
(Andrew Johnson in #1303)
Count variables were switches to size_t to avoid
warnings about conversion-narrowing (Dirk in #1307)
Rcpp now avoids the usage of the (non-API) DATAPTR function when
accessing the contents of Rcpp Vector objects where possible. (Kevin in
#1310)
Rcpp now emits an R warning on out-of-bounds Vector accesses.
This may become an error in a future Rcpp release. (Kevin in #1310)
Switch VECTOR_PTR and STRING_PTR to new
API-compliant RO variants (Kevin in #1317 fixing #1316)
Changes in Rcpp Deployment:
Small updates to the CI test containers have been made (#1304)
Lomiri Weather App goes Open Meteo
In Ubuntu Touch / Lomiri, Maciej Sopy o has updated Lomiri's Weather App to operate against a different weather forecast provider (Open Meteo). Additionally, the new implementation is generic and pluggable, so other weather data providers can be added-in later.
Big thanks to Maciej for working on this just in time (the previous implementation's API has recently been EOL'ed and is not available anymore to Ubuntu Touch / Lomiri users).
Lomiri Weather App - new Meteorological Terms part of the App now
While the old weather data provider implementation obtained all the meteorological information as already localized strings from the provider, the new implementation requires all sorts of weather conditions being translated within the Lomiri Weather App itself.
The meteorological terms are probably not easy to translate for the usual software translator, so special help might be required here.
Call for Translations: Lomiri Weather App
So, if you feel entitled to help here, please join the Hosted Weblate service [1] and start working on Lomiri Weather App.
Thanks a lot!
light+love
Mike Gabriel (aka sunweaver)
[1] https://hosted.weblate.org/
[2] https://hosted.weblate.org/projects/lomiri/lomiri-weather-app/
We are pleased to announce that
Wind River has committed to sponsor
DebConf24 as a Platinum Sponsor.
For nearly 20 years, Wind River has led in commercial open source Linux
solutions for mission-critical enterprise edge computing. With expertise
across aerospace, automotive, industrial, telecom, more, the company is
committed to open source through initiatives like eLxr, Yocto, Zephyr,
and StarlingX.
With this commitment as Platinum Sponsor, Wind River is contributing to
make possible our annual conference, and directly supporting the
progress of Debian and Free Software, helping to strengthen the
community that continues to collaborate on Debian projects throughout
the rest of the year.
Wind River plans to announce an exiting new project based on Debian at
this year's DebConf!
Thank you very much, Wind River, for your support of DebConf24!
Become a sponsor too!
DebConf24 will take place from 28th
July to 4th August 2024 in Busan, South Korea, and will be preceded by
DebCamp, from 21st to 27th July 2024.
DebConf24 is accepting sponsors! Interested companies and organizations
should contact the DebConf team through sponsors@debconf.org, or visit the
DebConf24 website at https://debconf24.debconf.org/sponsors/become-a-sponsor/.
Twenty years ago, it was easy to dislike Microsoft. It was the quintessential evil MegaCorp that was quick to squash competition, often ruthlessly, but in some cases slowly through a more insidious process of embracing, extending, and exterminating anything that got in the way. This was the signature personality of Ballmer-era Microsoft that also inspired and united the software freedom fighting forces that came together to safeguard things that mattered to them and were at risk.I remember the era when the Novell, SCO, and Microsoft saga cast fear, uncertainty, and doubt on the future of open Unix and Linux and on what would happen to the operating systems that we loved if the suits of Redmond prevailed. Looking back, I&aposm glad that the arc of this story has bent towards justice, and I shudder at the possibilities had it worked out differently.Looking at today&aposs Microsoft, I&aposm amazed at how much change a leader with the right vision can make to the trajectory of a company that even makes an old-school software freedom advocate as me admire and even applaud the strides it has taken in the last 10 or so years that has dramatically shifted the perception of Microsoft. The personality of the Satya-era Microsoft is one to behold. While it will take more time to win back the trust, we see the tides changing and the positivity is important for the entire industry.For Microsoft, it was TypeScript and VS Code that helped change the narrative internally which led to its internal resurgence and acceptance of open source. Its acquisition of GitHub propelled it forward within the community overnight. Its contributions to the Linux kernel and other major software projects have also been consequential in changing its public perceptions.It takes a while to claw back trust and is very easy to breach. This time, however, Microsoft seems to understand this dynamic more than it did 20 years ago. All it took was the right leadership.
Welcome to the June 2024 report from the Reproducible Builds project!
In our reports, we outline what we ve been up to over the past month and highlight news items in software supply-chain security more broadly. As always, if you are interested in contributing to the project, please visit our Contribute page on our website.
Table of contents:
Next Reproducible Builds Summit dates announced
We are very pleased to announce the upcoming Reproducible Builds Summit, set to take place from September 17th 19th 2024 in Hamburg, Germany.
We are thrilled to host the seventh edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin and Athens. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving.
If you re interesting in joining us this year, please make sure to read the event page which has more details about the event and location. We are very much looking forward to seeing many readers of these reports there.
GNU Guix patch review session for reproducibility
Vagrant Cascadian will be holding a Reproducible Builds session as part of the monthly Guix patch review series on July 11th at 17:00 UTC.
These online events are intended to encourage everyone everyone becoming a patch reviewer and the goal of reviewing patches is to help Guix project accept contributions while maintaining our quality standards and learning how to do patch reviews together in a friendly hacking session.
New reproducibility-related academic papers
Multiple scholarly papers related to Reproducible Builds were published this month:
An Industry Interview Study of Software Signing for Supply Chain Security was published by Kelechi G. Kalu, Tanmay Singla, Chinenye Okafor, Santiago Torres-Arias and James C. Davis of Electrical and Computer Engineering department of Purdue University, Indiana, USA, and is concerned with:
To understand software signing in practice, we interviewed 18 high-ranking industry practitioners across 13 organizations. We provide possible impacts of experienced software supply chain failures, security standards, and regulations on software signing adoption. We also study the challenges that affect an effective software signing implementation.
Code signing enables software developers to digitally sign their code using cryptographic keys, thereby associating the code to their identity. This allows users to verify the authenticity and integrity of the software, ensuring it has not been tampered with. Next-generation software signing such as Sigstore and OpenPubKey simplify code signing by providing streamlined mechanisms to verify and link signer identities to the public key. However, their designs have vulnerabilities: reliance on an identity provider introduces a single point of failure, and the failure to follow the principle of least privilege on the client side increases security risks. We introduce Diverse Identity Verification (DiVerify) scheme, which strengthens the security guarantees of next-generation software signing by leveraging threshold identity validations and scope mechanisms.
[ ] resulted in an extension of GNU make which is called rmake, where diffoscope a tool for detecting differences between a large number of file types was integrated into the workflow of make. rmake was later used to answer the posed research questions for this thesis. We found that different build paths and offsets are a big problem as three out of three tested Free and Open Source Software projects all contained these variations. The results also showed that gcc s optimisation levels did not affect reproducibility, but link-time optimisation embeds a lot of unreproducible information in build artefacts. Lastly, the results showed that build paths, build ID s and randomness are the three most common groups of variations encountered in the wild and potential solutions for some variations were proposed.
The thesis serves as an introduction to the concept of reproducibility in software engineering, offering a comprehensive overview of formalizations using mathematical notations for key concepts and an empirical evaluation of several key tools. By exploring various case studies, methodologies and tools, the research aims to provide actionable insights for practitioners and researchers alike.
Development news
In Debian this month, 4 reviews of Debian packages were added, 11 were updated and 14 were removed this month adding to our knowledge about identified issues. Only one issue types was updated, though, explaining that we don t vary the build path anymore.
On our mailing list this month, Bernhard M. Wiedemann wrote that whilst he had previously collected issues that introduce non-determinism he has now moved on to discuss about mitigations , in the sense of how can we avoid whole categories of problem without patching an infinite number of individual packages . In addition, Janneke Nieuwenhuizen announced the release of two versions of GNU Mes. [][]
In openSUSE news, Bernhard M. Wiedemann published another report for that distribution.
In NixOS, with the 24.05 release out, it was again validated that our minimal ISO is reproducible by building it on a virtual machine with no access to the binary cache.
What s more, we continued to write patches in order to fix specific reproducibility issues, including Bernhard M. Wiedemann writing three patches (for qutebrowser, samba and systemd), Chris Lamb filing Debian bug #1074214 against the fastfetch package and Arnout Engelen proposing fixes to refind and for the Scala compiler [.
Lastly, diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb uploaded two versions (270 and 271) to Debian, and made the following changes as well:
Drop Build-Depends on liblz4-tool in order to fix Debian bug #1072575. []
Update tests to support zipdetails version 4.004 that is shipped with Perl 5.40. []
Website updates
There were a number of improvements made to our website this month, including Akihiro Suda very helpfully making the <h4> elements more distinguishable from the <h3> level [][] as well as adding a guide for Dockerfile reproducibility []. In addition Fay Stegerman added two tools, apksigcopier and reproducible-apk-tools, to our Tools page.
Reproducibility testing framework
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In June, a number of changes were made by Holger Levsen, including:
Marking the virt(32 64)c-armhf nodes as down. []
Granting a developer access to the osuosl4 node in order to debug a regression on the ppc64el architecture. []
Granting a developer access to the osuosl4 node. [][]
In addition, Mattia Rizzolo re-aligned the /etc/default/jenkins file with changes performed upstream [] and changed how configuration files are handled on the rb-mail1 host. [], whilst Vagrant Cascadian documented the failure of the virt32c and virt64c nodes after initial investigation [].
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
I wrote a comment on a social media post where someone claimed that there s no computer advances in the last 12 years which got long so it s worth a blog post.
In the last decade or so new laptops have become cheaper than new desktop PCs. USB-C has taken over for phones and for laptop charging so all recent laptops support USB-C docks and monitors with USB-C docks built in have become common. 4K monitors have become cheap and common and higher than 4K is cheap for some use cases such as ultra wide. 4K TVs are cheap and TVs with built-in Android computers for playing internet content are now standard. For most use cases spinning media hard drives are obsolete, SSDs large enough for all the content most people need to store are cheap. We have gone from gigabit Ethernet being expensive to 2.5 gigabit being cheap.
12 years ago smart phones were very limited and every couple of years there would be significant improvements. Since about 2018 phones have been capable of doing most things most people want. 5yo Android phones can run the latest apps and take high quality pics. Any phone that supports VoLTE will be good for another 5+ years if it has security support. Phones without security support still work and are quite usable apart from being insecure. Google and Samsung have significantly increased their minimum security support for their phones and the GKI project from Google makes it easier for smaller vendors to give longer security support. There are a variety of open Android projects like LineageOS which give longer security support on a variety of phones. If you deliberately choose a phone that is likely to be well supported by projects like LineageOS (which pretty much means just Pixel phones) then you can expect to be able to actually use it when it is 10 years old. Compare this to the Samsung Galaxy S3 released in 2012 which was a massive improvement over the original Galaxy S (the S2 felt closer to the S than the S3). The Samsung Galaxy S4 released in 2013 was one of the first phones to have FullHD resolution which is high enough that most people can t easily recognise the benefits of higher resolution. It wasn t until 2015 that phones with 4G of RAM became common which is enough that for most phone use it s adequate today.
Now that 16G of RAM is affordable in laptops running more secure OSs like Qubes is viable for more people. Even without Qubes, OS security has been improving a lot with better compiler features, new languages like Rust, and changes to software design and testing. Containers are being used more but we still aren t getting all the benefits of that. TPM has become usable in the last few years and we are only starting to take advantage of what it can offer.
In 2012 BTRFS was still at an early stage of development and not many people wanted to use it in production, I was using it in production then and while I didn t lose any data from bugs I did have some downtime because of BTRFS issues. Now BTRFS is quite solid for server use.
DDR4 was released in 2014 and gave significant improvements over DDR3 for performance and capacity. My home workstation now has 256G of DDR4 which wasn t particularly expensive while the previous biggest system I owned had 96G of DDR3 RAM. Now DDR5 is available to again increase performance and size while also making DDR4 cheap on the second hand market.
This isn t a comprehensive list of all advances in the computer industry over the last 12 years or so, it s just some things that seem particularly noteworthy to me.
Please comment about what you think are the most noteworthy advances I didn t mention.
YubiHSM packaging, by Colin Watson
Freexian is starting to use YubiHSM devices (hardware security modules)
as part of some projects, and we wanted to have the supporting software
directly in Debian rather than needing to use third-party repositories.
Since Yubico publish everything we need under free software licences,
Colin packaged yubihsm-connector, yubihsm-shell, and
python-yubihsm from
https://developers.yubico.com/, in
some cases based partly on the upstream packaging, and got them all into
Debian unstable.
Backports to bookworm will be forthcoming once they ve all reached
testing.
unschroot by Helmut Grohne
Following an in-person discussion at MiniDebConf Berlin, Helmut
attempted splitting the containment functionality of sbuild --chroot-mode=unshare into a dedicated tool interfacing with sbuild
as a variant of --chroot-mode=schroot providing a sufficiently
compatible
interface.
While this seemed technically promising initially, a discussion on
debian-devel
indicated a desire to rely on an existing container runtime such as
podman instead of using another Debian-specific tool with unclear long
term maintenance.
None of the existing container runtimes meet the specific needs of
sbuild, so further advancing this matter implies a compromise one way
or another.
Linux live-patching, by Santiago Ruano Rinc n
In collaboration with Emmanuel Arias, Santiago is
working on the development of linux live-patching for Debian.
For the moment, this is in an exploratory phase, that includes how to
handle the different patches that will need to be provided.
kpatch could help significantly
in this regard.
However, kpatch was removed from unstable because there are some RC bugs
affecting the version that was present in Debian unstable.
Santiago packaged the most recent upstream version
(0.9.9)
and filed an Intent to Salvage
bug.
Santiago is waiting for an ACK by the maintainer, and will upload to
unstable after July 10th, following the package salvaging rules.
While kpatch 0.9.9 fixes the main issues, it still needs some work to
properly support Debian and the Linux kernel versions packaged in our
distribution.
More on this in the report next month.
Salsa CI, by Santiago Ruano Rinc n
The work by Santiago in Salsa CI this month includes a merge
request
to ease testing how the production images are built from the changes
introduced by future merge requests.
By default, the pipelines triggered by a merge request build a subset of
the images built for production, to reduce the use of resources, and
because most of the time the subset of staging images is enough to test
the proposed modifications.
However, sometimes it is needed to test how the full set of production
images is built, and the above mentioned MR helps to do that.
The changes include documentation, so hopefully this will make it easier
to test future contributions.
Also, for being able to include support for RISC-V, Salsa CI needs to
replace
kaniko as
the tool used to build the images.
Santiago tested
buildah,
but there are some issues when pushing built images for non-default
platform architectures (i386, armhf, armel) to the container registry.
Santiago will continue to work on this to find a solution.
Miscellaneous contributions
Stefano Rivera prepared updates for a number of Python modules.
Stefano uploaded the latest point release of Python 3.12 and the
latest Python 3.13 beta.
Both uncovered upstream regressions that had to be addressed.
Stefano worked on preparations for DebConf 24.
Stefano helped SPI to reconcile their financial records for DebConf
23.
Colin did his usual routine work on the Python team, upgrading 36
packages to new upstream versions (including fixes for four CVEs in
python-aiohttp), fixing RC bugs in ipykernel, ipywidgets,
khard, and python-repoze.sphinx.autointerface, and packaging
zope.deferredimport which was needed for a new upstream version of
python-persistent.
Colin removed the user_readenv option from OpenSSH s PAM
configuration (#1018260), and
prepared a release note.
Thorsten Alteholz uploaded a new upstream version of cups.
Nicholas Skaggs updated xmacro to support reproducible builds
(#1014428),
DEP-3 and DEP-5 compatibility, along with utilizing hardening build
flags.
Helmut supported and uploaded package.
As a result of login having become non-essential, Helmut uploaded
debvm to unstable and stable and fixed a crossqa.debian.net worker.
Santiago worked on the Content Team activities for DebConf24.
Together with other DebConf25 team members, Santiago wrote a document
for the head of the venue to describe the project of the conference.
After rebuilding all added/modified packages in Trisquel, I have been circling around the elephant in the room: 99% of the binary packages in Trisquel comes from Ubuntu, which to a large extent are built from Debian source packages. Is it possible to rebuild the official binary packages identically? Does anyone make an effort to do so? Does anyone care about going through the differences between the official package and a rebuilt version? Reproducible-build.org s effort to track reproducibility bugs in Debian (and other systems) is amazing. However as far as I know, they do not confirm or deny that their rebuilds match the official packages. In fact, typically their rebuilds do not match the official packages, even when they say the package is reproducible, which had me surprised at first. To understand why that happens, compare the buildinfo file for the official coreutils 9.1-1 from Debian bookworm with the buildinfo file for reproducible-build.org s build and you will see that the SHA256 checksum does not match, but still they declare it as a reproducible package. As far as I can tell of the situation, the purpose of their rebuilds are not to say anything about the official binary build, instead the purpose is to offer a QA service to maintainers by performing two builds of a package and declaring success if both builds match.
I have felt that something is lacking, and months have passed and I haven t found any project that address the problem I am interested in. During my earlier work I created a project called debdistreproduce which performs rebuilds of the difference between two distributions in a GitLab pipeline, and display diffoscope output for further analysis. A couple of days ago I had the idea of rewriting it to perform rebuilds of a single distribution. A new project debdistrebuild was born and today I m happy to bless it as version 1.0 and to announces the project! Debdistrebuild has rebuilt the top-50 popcon packages from Debian bullseye, bookworm and trixie, on amd64 and arm64, as well as Ubuntu jammy and noble on amd64, see the summary status page for links. This is intended as a proof of concept, to allow people experiment with the concept of doing GitLab-based package rebuilds and analysis. Compare how Guix has the guix challenge command.
Or I should say debdistrebuild has attempted to rebuild those distributions. The number of identically built packages are fairly low, so I didn t want to waste resources building the rest of the archive until I understand if the differences are due to consequences of my build environment (plain apt-get build-dep followed by dpkg-buildpackage in a fresh container), or due to some real difference. Summarizing the results, debdistrebuild is able to rebuild 34% of Debian bullseye on amd64, 36% of bookworm on amd64, 32% of bookworm on arm64. The results for trixie and Ubuntu are disappointing, below 10%.
So what causes my rebuilds to be different from the official rebuilds? Some are trivial like the classical problem of varying build paths, resulting in a different NT_GNU_BUILD_ID causing a mismatch. Some are a bit strange, like a subtle difference in one of perl s headers file. Some are due to embedded version numbers from a build dependency. Several of the build logs and diffoscope outputs doesn t make sense, likely due to bugs in my build scripts, especially for Ubuntu which appears to strip translations and do other build variations that I don t do. In general, the classes of reproducibility problems are the expected. Some are assembler differences for GnuPG s gpgv-static, likely triggered by upload of a new version of gcc after the original package was built. There are at least two ways to resolve that problem: either use the same version of build dependencies that were used to produce the original build, or demand that all packages that are affected by a change in another package are rebuilt centrally until there are no more differences.
The current design of debdistrebuild uses the latest version of a build dependency that is available in the distribution. We call this a idempotent rebuild . This is usually not how the binary packages were built originally, they are often built against earlier versions of their build dependency. That is the situation for most binary distributions.
Instead of using the latest build dependency version, higher reproducability may be achieved by rebuilding using the same version of the build dependencies that were used during the original build. This requires parsing buildinfo files to find the right version of the build dependency to install. We believe doing so will lead to a higher number of reproducibly built packages. However it begs the question: can we rebuild that earlier version of the build dependency? This circles back to really old versions and bootstrappable builds eventually.
While rebuilding old versions would be interesting on its own, we believe that is less helpful for trusting the latest version and improving a binary distribution: it is challenging to publish a new version of some old package that would fix a reproducibility bug in another package when used as a build dependency, and then rebuild the later packages with the modified earlier version. Those earlier packages were already published, and are part of history. It may be that ultimately it will no longer be possible to rebuild some package, because proper source code is missing (for packages using build dependencies that were never part of a release); hardware to build a package could be missing; or that the source code is no longer publicly distributable.
I argue that getting to 100% idempotent rebuilds is an interesting goal on its own, and to reach it we need to start measure idempotent rebuild status.
One could conceivable imagine a way to rebuild modified versions of earlier packages, and then rebuild later packages using the modified earlier packages as build dependencies, for the purpose of achieving higher level of reproducible rebuilds of the last version, and to reach for bootstrappability. However, it may be still be that this is insufficient to achieve idempotent rebuilds of the last versions. Idempotent rebuilds are different from a reproducible build (where we try to reproduce the build using the same inputs), and also to bootstrappable builds (in which all binaries are ultimately built from source code). Consider a cycle where package X influence the content of package Y, which in turn influence the content of package X. These cycles may involve several packages, and it is conceivable that a cycle could be circular and infinite. It may be difficult to identify these chains, and even more difficult to break them up, but this effort help identify where to start looking for them. Rebuilding packages using the same build dependency versions as were used during the original build, or rebuilding packages using a bootsrappable build process, both seem orthogonal to the idempotent rebuild problem.
Our notion of rebuildability appears thus to be complementary to reproducible-builds.org s definition and bootstrappable.org s definition. Each to their own devices, and Happy Hacking!
Addendum about terminology: With idempotent rebuild I am talking about a rebuild of the entire operating system, applied to itself. Compare how you build the latest version of the GNU C Compiler: it first builds itself using whatever system compiler is available (often an earlier version of gcc) which we call step 1. Then step 2 is to build a copy of itself using the compiler built in step 1. The final step 3 is to build another copy of itself using the compiler from step 2. Debian, Ubuntu etc are at step 1 in this process right now. The output of step 2 and step 3 ought to be bit-by-bit identical, or something is wrong. The comparison between step 2 and 3 is what I refer to with an idempotent rebuild. Of course, most packages aren t a compiler that can compile itself. However entire operating systems such as Trisquel, PureOS, Ubuntu or Debian are (hopefully) a self-contained system that ought to be able to rebuild itself to an identical copy. Or something is amiss. The reproducible build and bootstrappable build projects are about improve the quality of step 1. The property I am interested is the identical rebuild and comparison in step 2 and 3. I feel the word idempotent describes the property I m interested in well, but I realize there may be better ways to describe this. Ideas welcome!
Raising Steam is the 40th Discworld novel and the third Moist von
Lipwig novel, following Making Money.
This is not a good place to start reading the series.
Dick Simnel is a tinkerer from a line of tinkerers. He has been obsessed
with mastering the power of steam since the age of ten, when his father
died in a steam accident. That pursuit took him deeper into mathematics
and precision, calculations and experiments, until he built Iron Girder:
Discworld's first steam-powered locomotive. His early funding came from
some convenient family pirate treasure, but turning his prototype into
something more will require significantly more resources. That is how he
ends up in the office of Harry King, Ankh-Morpork's sanitation magnate.
Simnel's steam locomotive has the potential to solve some obvious
logistical problems, such as getting fish from the docks of Quirm to the
streets of Ankh-Morpork before it stops being vaguely edible. That's not
what makes railways catch fire, however. As soon as Iron Girder is
huffing and puffing its way around King's compound, it becomes the most
popular attraction in the city. People stand in line for hours to ride it
over and over again for reasons that they cannot entirely explain. There
is something wild and uncontrollable going on.
Vetinari is not sure he likes wild and uncontrollable, but he knows the
lap into which such problems can be dumped: Moist von Lipwig, who is
already getting bored with being a figurehead for the city's banking
system.
The setup for Raising Steam reminds me more of
Moving Pictures than the other Moist von
Lipwig novels. Simnel himself is a relentlessly practical engineer, but
the trains themselves have tapped some sort of primal magic. Unlike
Moving Pictures, Pratchett doesn't provide an explicit fantasy
explanation involving intruding powers from another world. It might have
been a more interesting book if he had. Instead, this book expects the
reader to believe there is something inherently appealing and fascinating
about trains, without providing much logic or underlying justification. I
think some readers will be willing to go along with this, and others
(myself included) will be left wishing the story had more world-building
and fewer exclamation points.
That's not the real problem with this book, though. Sadly, its true
downfall is that Pratchett's writing ability had almost completely
collapsed by the time he wrote it.
As mentioned in my review of Snuff,
we're now well into the period where Pratchett was suffering the effects
of early-onset Alzheimer's. In that book, his health issues mostly
affected the dialogue near the end of the novel. In this book, published
two years later, it's pervasive and much worse. Here's a typical passage
from early in the book:
It is said that a soft answer turneth away wrath, but this assertion
has a lot to do with hope and was now turning out to be patently
inaccurate, since even a well-spoken and thoughtful soft answer could
actually drive the wrong kind of person into a state of fury if wrath
was what they had in mind, and that was the state the elderly dwarf
was now enjoying.
One of the best things about Discworld is Pratchett's ability to drop
unexpected bits of wisdom in a sentence or two, or twist a verbal knife in
an unexpected and surprising direction. Raising Steam still shows
flashes of that ability, but it's buried in run-on sentences, drowned in
cliches and repetition, and often left behind as the containing sentence
meanders off into the weeds and sputters to a confused halt. The idea is
still there; the delivery, sadly, is not.
This is the first Discworld novel that I found mentally taxing to read.
Sentences are often so overpacked that they require real effort to
untangle, and the untangled meaning rarely feels worth the effort. The
individual voice of the characters is almost gone. Vetinari's monologues,
rather than being a rare event with dangerous layers, are frequent,
rambling, and indecisive, often sounding like an entirely different
character than the Vetinari we know. The constant repetition of the name
any given character is speaking to was impossible for me to ignore. And
the momentum of the story feels wrong; rather than constructing the events
of the story in a way that sweeps the reader along, it felt like Pratchett
was constantly pushing, trying to convince the reader that trains were the
most exciting thing to ever happen to Discworld.
The bones of a good story are here, including further development of dwarf
politics from The Fifth Elephant and
Thud! and the further fallout of the
events of Snuff. There are also
glimmers of Pratchett's typically sharp observations and turns of phrase
that could have been unearthed and polished. But at the very least this
book needed way more editing and a lot of rewriting. I suspect it
could have dropped thirty pages just by tightening the dialogue and
removing some of the repetition.
I'm afraid I did not enjoy this. I am a bit of a hard sell for the magic
fascination of trains I love trains, but my model railroad days are
behind me and I'm now more interested in them as part of urban
transportation policy. Previous Discworld books on technology and social
systems did more of the work of drawing the reader in, providing character
hooks and additional complexity, and building a firmer foundation than
"trains are awesome." The main problem, though, was the quality of the
writing, particularly when compared to the previous novels with the same
characters.
I dragged myself through this book out of a sense of completionism and
obligation, and was relieved when I finished it.
This is the first Discworld novel that I don't recommend. I think the
only reason to read it is if you want to have read all of Discworld.
Otherwise, consider stopping with Snuff and letting it be the
send-off for the Ankh-Morpork characters.
Followed by The Shepherd's Crown, a Tiffany Aching story and the
last Discworld novel.
Rating: 3 out of 10
Armadillo is a powerful
and expressive C++ template library for linear algebra and scientific
computing. It aims towards a good balance between speed and ease of use,
has a syntax deliberately close to Matlab, and is useful for algorithm
development directly in C++, or quick conversion of research code into
production environments. RcppArmadillo
integrates this library with the R environment and language and is
widely used by (currently) 1158 other packages on CRAN, downloaded 35.1 million
times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint
/ vignette) by Conrad and myself has been cited 587 times according
to Google Scholar.
Conrad released a new
major upstream version 14.0.0 a couple of days ago. We had been testing
this new version extensively over several rounds of reverse-dependency
checks across all 1100+ packages. This revealed nine packages requiring
truly minor adjustments which eight maintainers made in a matter of
days; all this was coordinated in issue
#443. Following the upload, CRAN noticed one more issue (see
issue
#446) but this turned out to be local to the package. There are also
renewed deprecation warnings with some Armadillo changes which we will
need to address one-by-one. Last but not least with this release we also
changed the package versioning scheme to follow upstream Armadillo more closely.
The set of changes since the last CRAN release follows.
Changes in
RcppArmadillo version 14.0.0-1 (2024-07-05)
Upgraded to Armadillo release 14.0.0 (Stochastic Parrot)
C++14 is now the minimum recommended C++ standard
Faster handling of compound expressions by
as_scalar(), accu(),
dot()
Faster interactions between sparse and dense matrices
Expanded stddev() to handle sparse matrices
Expanded relational operators to handle expressions between
sparse matrices and scalars
Added .as_dense() to obtain dense vector/matrix
representation of any sparse matrix expression
Updated physical constants to NIST 2022 CODATA values
New package version numbering scheme following upstream
versions
Re-enabling ARMA_IGNORE_DEPRECATED_MARKE for silent
CRAN builds
Christian
David,
CC BY-SA 4.0, via Wikimedia CommonsThis post was originally published in the Wikimedia Tech blog, authored by Arturo Borrero Gonzalez.
Summary: this article shares the experience and learnings of migrating away from Kubernetes PodSecurityPolicy into
Kyverno in the Wikimedia Toolforge platform.
Wikimedia Toolforge is a Platform-as-a-Service, built with
Kubernetes, and maintained by the Wikimedia Cloud Services team (WMCS). It is completely free and open, and we welcome
anyone to use it to build and host tools (bots, webservices, scheduled jobs, etc) in support of Wikimedia projects.
We provide a set of platform-specific services, command line interfaces, and shortcuts to help in the task of setting up
webservices, jobs, and stuff like building container images, or using databases. Using these interfaces makes the
underlying Kubernetes system pretty much invisible to users. We also allow direct access to the Kubernetes API, and some
advanced users do directly interact with it.
Each account has a Kubernetes namespace where they can freely deploy their workloads. We have a number of controls in
place to ensure performance, stability, and fairness of the system, including quotas, RBAC permissions, and up until
recently PodSecurityPolicies (PSP). At the time of this writing, we had around 3.500 Toolforge tool accounts in the
system. We early adopted PSP in 2019 as a way to make sure Pods had the correct runtime configuration. We needed Pods to
stay within the safe boundaries of a set of pre-defined parameters. Back when we adopted PSP there was already the
option to use 3rd party agents, like OpenPolicyAgent Gatekeeper,
but we decided not to invest in them, and went with a native, built-in mechanism instead.
In 2021 it was announced
that the PSP mechanism would be deprecated, and removed in Kubernetes 1.25. Even though we had been warned years in
advance, we did not prioritize the migration of PSP until we were in Kubernetes 1.24, and blocked, unable to upgrade
forward without taking actions.
The WMCS team explored different alternatives for this migration, but eventually we decided to go with
Kyverno as a replacement for PSP. And so with that decision it began the
journey described in this blog post.
First, we needed a source code refactor for one of the key components of our Toolforge Kubernetes:
maintain-kubeusers. This custom piece of
software that we built in-house, contains the logic to fetch accounts from LDAP and do the necessary instrumentation on
Kubernetes to accommodate each one: create namespace, RBAC, quota, a kubeconfig file, etc. With the refactor, we
introduced a proper reconciliation loop, in a way that the software would have a notion of what needs to be done for
each account, what would be missing, what to delete, upgrade, and so on. This would allow us to easily deploy new
resources for each account, or iterate on their definitions.
The initial version of the refactor had a number of problems, though. For one, the new version of maintain-kubeusers was
doing more filesystem interaction than the previous version, resulting in a slow reconciliation loop over all the
accounts. We used NFS as the underlying storage system for Toolforge, and it could be very slow because of reasons
beyond this blog post. This was corrected in the next few days after the initial refactor rollout. A side note with an
implementation detail: we stored a configmap on each account namespace with the state of each resource. Storing more
state on this configmap was our solution to avoid additional NFS latency.
I initially estimated this refactor would take me a week to complete, but unfortunately it took me around three weeks
instead. Previous to the refactor, there were several manual steps and cleanups required to be done when updating the
definition of a resource. The process is now automated, more robust, performant, efficient and clean. So in my opinion
it was worth it, even if it took more time than expected.
Then, we worked on the Kyverno policies themselves. Because we had a very particular PSP setting, in order to ease the
transition, we tried to replicate their semantics on a 1:1 basis as much as possible. This involved things like
transparent mutation of Pod resources, then validation. Additionally, we had one different PSP definition for each
account, so we decided to create one different Kyverno namespaced policy resource for each account namespace remember,
we had 3.5k accounts.
We created a Kyverno policy
template
that we would then render and inject for each account.
For developing and testing all this, maintain-kubeusers and the Kyverno bits, we had a project called
lima-kilo, which was a local Kubernetes setup
replicating production Toolforge. This was used by each engineer in their laptop as a common development environment.
We had planned the migration from PSP to Kyverno policies in stages, like this:
update our internal template generators to make Pod security settings explicit
introduce Kyverno policies in Audit mode
see how the cluster would behave with them, and if we had any offending resources reported by the new policies, and
correct them
modify Kyverno policies and set them in Enforce mode
drop PSP
In stage 1, we updated things like the toolforge-jobs-framework and tools-webservice.
In stage 2, when we deployed the 3.5k Kyverno policy resources, our production cluster died almost immediately.
Surprise. All the monitoring went red, the Kubernetes apiserver became irresponsibe, and we were unable to perform any
administrative actions in the Kubernetes control plane, or even the underlying virtual machines. All Toolforge users
were impacted. This was a full scale
outage that required the
energy of the whole WMCS team to recover from. We temporarily disabled Kyverno until we could learn what had occurred.
This incident happened despite having tested before in lima-kilo and in another pre-production cluster we had, called
Toolsbeta. But we had not tested that many
policy resources. Clearly, this was something scale-related. After the incident, I went on and created 3.5k Kyverno
policy resources on lima-kilo, and indeed I was able to reproduce the outage. We took a number of measures, corrected a
few errors in our infrastructure, reached out to the Kyverno upstream
developers, asking for advice, and at the end we did the following to
accommodate the setup to our needs:
corrected the external HAproxy kubernetes apiserver health checks, from checking just for open TCP ports, to actually
checking the /healthz HTTP endpoint, which more accurately reflected the health of each k8s apiserver.
having a more realistic development environment. In lima-kilo, we created a couple of helper scripts to create/delete
4000 policy resources, each on a different namespace.
greatly over-provisioned memory in the Kubernetes control plane servers. This is, bigger memory in the base virtual
machine hosting the control plane. Scaling the memory headroom of the apiserver would prevent it from running out of
memory, and therefore crashing the whole system. We went from 8GB RAM per virtual machine to 32GB. In our cluster, a
single apiserver pod could eat 7GB of memory on a normal day, so having 8GB on the base virtual machine was clearly
not enough. I also sent a patch proposal to Kyverno upstream documentation suggesting they clarify the additional
memory pressure on the apiserver.
increased the number of replicas of the Kyverno admission controller to 7, so admission requests could be handled more
timely by Kyverno.
I have to admit, I was briefly tempted to drop Kyverno, and even stop pursuing using an external policy agent entirely,
and write our own custom admission controller out of concerns over performance of this architecture. However, after
applying all the measures listed above, the system became very stable, so we decided to move forward. The second attempt
at deploying it all went through just fine. No outage this time
When we were in stage 4 we detected another bug. We had been following the Kubernetes upstream documentation for setting
securityContext to the right values. In particular, we were enforcing the procMount to be set to the default value,
which per the docs it was
DefaultProcMount .
However, that string is the name of the internal variable in the source code, whereas the actual default value is the
string Default . This caused pods to be
rightfully rejected by Kyverno while we figured the problem. I sent a patch
upstream to fix this problem.
We finally had everything in place, reached stage 5, and we were able to disable PSP. We unloaded the PSP controller
from the kubernetes apiserver, and deleted every individual PSP definition. Everything was very smooth in this last step
of the migration.
This whole PSP project, including the maintain-kubeusers refactor, the outage, and all the different migration stages
took roughly three months to complete.
For me there are a number of valuable reasons to learn from this project. For one, the scale is something to consider,
and test, when evaluating a new architecture or software component. Not doing so can lead to service outages, or
unexpectedly poor performances. This is in the first chapter of the SRE handbook, but we got a reminder the hard way
This post was originally published in the Wikimedia Tech blog, authored by Arturo Borrero Gonzalez.
I became interested in Network Time Protocol (NTP) last year after learning how fundamental this protocol is to the functioning of the global Internet. NTP helps synchronize clocks on devices over the Internet, which is essential for secure browsing, timestamping, keeping everyone in sync or just checking what time it is. Computers usually have a hardware real-time clock (RTC) but that deviates over time, so an occasional sync over NTP is required to keep the time accurate. Many network and IoT devices don t have hardware RTC so have even more reliance on NTP.
Accurate time keeping starts with reference clocks like atomic clocks, GPS etc. Multiple government standard agencies host these reference clocks, which are regarded as Stratum 0. Stratum 1 servers are known as primary servers, and directly connect to Stratum 0 clocks for time. Stratum 1 servers then distribute time to Stratum 2 and further down the hierarchy. Computers typically connects to one or more Stratum 1/2/3 servers to get their time.
Someone has to host these public Stratum 1,2,3 NTP servers. That s what NTP pool, a global effort by volunteers does. They provide NTP servers for the public to use. As of today, there are 4700+ servers in the pool which are free to use for anyone.
Now let s come to the reason for writing this post. Indian Computer Emergency Response Team (CERT-In) in April 2022 released a set of cybersecurity directions which set the alarm bells ringing. Internet Society (and almost everyone else) wrote about it.
And then there was this specific section about NTP:
All service providers, intermediaries, data centres, body corporate and Government organisations shall connect to the Network Time Protocol (NTP) Server of National Informatics Centre (NIC) or National Physical Laboratory (NPL) or with NTP servers traceable to these NTP servers, for synchronisation of all their ICT systems clocks. Entities having ICT infrastructure spanning multiple geographies may also use accurate and standard time source other than NPL and NIC, however it is to be ensured that their time source shall not deviate from NPL and NIC.
CSIR-National Physical Laboratory (NPL) is the official timekeeper for India and hosts the only public Stratum 1 clock in India, according to NTP pool website. So I was naturally curious to know what kind of infrastructure they re running for NTP. India has a Right to Information (RTI) Act which, like the Freedom of Information Act (FOIA) in the United States, gives citizens rights to request information from governmental entities, to which they have to respond in under 30 days. So last year, I filed two sets of RTI (one after the first reply came) inquiring about NPL s public NTP server setup.
The first RTI had some generic questions:
Feel free to make your conclusions from it now. Bear in mind these were filled last year so things might have changed. Do let me know if you have more information about it.
Update (07/07/2024): Found an article from Medianama about Indian government time servers with information sourced through RTI.
Availability (comes installed with the curl package):
Debian unstable - Since 2024-07-02
Debian testing - Since 2024-07-18
Debian 12/bookworm backports - Expected by the end of August 2024.
Debian 12/bookworm - Depends on whether Debian's release team will approve
it, it could be available in the next point release.
Debian derivatives - Rolling releases will get it by the time it's on Debian
testing (e.g.: Kali Linux). Stable derivatives only in their next major release.
Smoother CLI experience
Starting with curl version 8.8.0-2, the Debian's curl package now ships a wcurl
executable.
wcurl is the solution for those who just need to download files without having
to remember curl's parameters for things like automatically naming the files.
Some people, myself included, would fall back to using wget whenever there was a
need to download a file. Sometimes even installing wget just for that usecase.
After all, it's easier to remember "apt install wget" rather than "curl -L -O -C - ...".
wcurl consists of a simple shell script that provides sane defaults for the curl
invocation, for when the use case is to just download files.
By default, wcurl will:
Encode whitespaces in URLs;
Download multiple URLs in parallel if the installed curl's version is >= 7.66.0;
Follow redirects;
Automatically choose a filename as output;
Avoid overwriting files if the installed curl's version is >= 7.83.0 (--no-clobber);
Perform retries;
Set the downloaded file timestamp to the value provided by the server, if available;
Default to the protocol used as https if the URL doesn't contain any;
Disable curl's URL globbing parser so and [] characters in URLs are not treated specially.
Example to download a single file:
wcurl example.com/filename.txt
If you ever need to set a custom flag, you can make use of the --curl-options
wcurl option, anything set there will be passed to the curl invocation.
Just beware that if you need to set any custom flags, it's likely you will be
better served by calling curl directly. The --curl-options option is there to
allow for some flexibility in unforeseen circumstances.
The need for wcurl
I've always felt a bit ashamed of not remembering curl's parameters for
downloading a file and automatically naming it, having resorted to wget most of
the times this was needed (even installing wget when it wasn't there, just for
this). I've spoken to a few other experienced people I know and confirmed what
could be obvious to others: a lot of people struggle with this.
Recently, the curl project released the results of 2024's curl
survey, which
also showed this is as a much needed feature, just look at some of the answers:
Q: Which curl command line option do you think needs improvement and how?
-O, I really want wget like functionality where I don't have to specify the name
Downloading a file (like wget) could be improved - with automatic naming of the file
downloading files - wget is much cleaner
I wish the default behaviour when GETting a binary was to drop it on disk. That's the only
reason 'wget foo.tgz" is still ingrained in my muscle memory .
Maybe have a way to download without specifying something in -o (the only reason i used wget
still)
--remote-time should be default
--remote-name-all could really use a short flag
Q: If you miss support for something, tell us what!
"Write the data to the file named in the URL (or in redirects if I'm feeling daring), and
timestamp the file to the last-modified-date". This is the main reason I'm still using wget.
I can finally feel less bad about falling back to wget due to not remembering the
parameters I want.
Idealization vs. reality
I don't believe curl will ever change its default behavior in such a way that
would accommodate this need, as that would have a side-effect of breaking things
which expect the current behavior (the blast radius is literally the
solar system).
This means a new executable needs to be shipped side-by-side with curl, an
opportunity to start fresh and work with a more focused use case (to download
files).
Ideally, this new executable would be maintained by the curl project, make use
of libcurl under-the-hood, and be available everywhere. Nobody wants to worry
if their systems have the tool or not, it should always be there.
Given I'm just a Debian Developer, with not as much free time as I wish, I've
decided to write a simple shell script wrapper calling the curl CLI
under-the-hood.
wcurl will come installed with the curl package from now on, and I will check
with the release team about shipping it on the current Debian stable as well.
Shipping wcurl in other distros will be up to them (Debian-derivatives should
pick it up automatically, though).
We've tried to make it easy for anyone to ship this by using the curl license,
keeping the script POSIX-compliant, and shipping a manpage.
Maybe if there's enough interest across distributions, someone might sign up
for implementing this in upstream curl and increase its reach. I would be happy
with the curl project reusing the wcurl name when that happens. It's unlikely
that wcurl would be shipped by curl upstream as it is, assuming they would
prefer a solution that uses libcurl direclty (more similar to curl the CLI, to
maintain).
In the worst case, wcurl becomes a Debian-specific tool that only a few people
are aware of, in the best case, it becomes the new go-to CLI tool for simply
downloading files. I would be happy if at least someone other than me finds
it useful.
Naming is hard
When I started working on it, I was calling the new executable "curld"
(stands for "curl download"), but then when discussing this in one of our
weekly calls in the Debian Bras lia community, it was mentioned that this could
be confused for a daemon.
We then settled for the name "wcurl", suggested by Carlos Henrique Lima
Melara <charles>. It doesn't really stand for anything,
but it's very easy to remember.
You know... "it's that wget alternative for when you want to use curl instead"
:)
Acknowledgments
The idea for wcurl came a few days before the curl-up conference
2024.
I've been thinking a lot about developer productivity in the terminal lately,
different tools and better defaults. Before curl-up, I was also thinking about
packaging improvements for the curl package. I don't remember what exactly
happened, but I likely had to download something and felt a bit ashamed of
maintaining curl and not remembering the parameters to download files the way I
wanted.
I first discussed this idea in the conference, where I asked the
participants about it and there were no concerns raised, and some people said I should give it a go.
Participating in curl-up was a really great experience and I'm thankful for the
interactions I've had there.
On the Debian side, I've got reviews of the code and manpage by Sergio Durigan
Junior <sergiodj>, Guilherme Puida Moreira <puida> and Carlos Henrique Lima
Melara <charles>. Sergio ended up rewriting the tool to be POSIX-compliant (my
version was written in bash), so he takes all the credit for the portability.
Changes since publication
2024-07-18
Update date of availability for Debian testing and expected date for bookworm backports.
Mention charles as the person who suggested "wcurl" as a name.
Update wcurl's -o/--opts options, it's now just --curl-options.
Remove mention of language spoken in the Matrix room, we are using English now.
Dear Debian community,
Statement on Daniel Pocock
The Debian project has successfully taken action to secure its
trademarks and interests worldwide, as detailed in our press
statement. I would like
to personally thank everyone in the community who was involved in this
process. I would have loved for you all to have spent your volunteer
time on more fruitful things.
Debian Boot team might need help
I think I've identified the issue that finally motivated me to contact
our teams: for a long time, I have had the impression that Debian is
driven by several "one-person teams" (to varying extents of individual
influence and susceptibility to burnout). As DPL, I see it as my task to
find ways to address this issue and provide support.
I received private responses from Debian Boot team members, which
motivated me to kindly invite volunteers to some prominent and highly
visible fields of work that you might find personally challenging. I
recommend subscribing to the
Debian Boot mailing list to see
where you might be able to provide assistance.
/usrmerge
Helmut Grohne confirmed
that the last remaining packages shipping aliased files inside the package set
relevant to debootstrap were uploaded. Thanks a lot for Helmut and all
contributors that helped to implement
DEP17.
Contacting more teams
I'd like to repeat that I've registered a BoF for DebConf24 in Busan
with the following description:
This BoF is an attempt to gather as much as possible teams inside
Debian to exchange experiences, discuss workflows inside teams, share
their ways to attract newcomers etc.
Each participant team should prepare a short description of their work
and what team roles ( openings ) they have for new contributors. Even
for delegated teams (membership is less fluid), it would be good to
present the team, explain what it takes to be a team member, and what
steps people usually go to end up being invited to participate. Some
other teams can easily absorb contributions from salsa MRs, and at some
point people get commit access. Anyway, the point is that we work on the
idea that the pathway to become a team member becomes more clear from an
outsider point-of-view.
I'm lagging a bit behind my team contacting schedule and will not manage
to contact every team before DebConf. As a (short) summary, I can draw
some positive conclusions about my efforts to reach out to teams. I was
able to identify some issues that were new to me and which I am now
working on. Examples include limitations in Salsa and Salsa CI. I
consider both essential parts of our infrastructure and will support
both teams in enhancing their services.
Some teams confirmed that they are basically using some common
infrastructure (Salsa team space, mailing lists, IRC channels) but that
the individual members of the team work on their own problems without
sharing any common work. I have also not read about convincing
strategies to attract newcomers to the team, as we have established, for
instance, in the Debian Med team.
DebConf attendance
The amount of money needed to fly people to South Korea was higher than
usual, so the DebConf bursary team had to make some difficult decisions
about who could be reimbursed for travel expenses. I extended the budget
for diversity and newcomers, which enabled us to invite some additional
contributors. We hope that those who were not able to come this year can
make it next year to Brest or to MiniDebConf
Cambridge or
Toulouse
tag2upload
On June 12, Sean Whitton requested comments on the debian-vote list
regarding a General Resolution (GR) about
tag2upload.
The discussion began with technical details but unfortunately, as often
happens in long threads, it drifted into abrasive language, prompting
the community team to address the behavior of an opponent of the GR
supporters. After 560 emails covering technical details, including a
detailed security review
by Russ Allbery, Sean finally proposed the
GR
on June 27, 2024 (two weeks after requesting comments).
Firstly, I would like to thank the drivers of this GR and acknowledge
the technical work behind it, including the security review. I am
positively convinced that Debian can benefit from modernizing its
infrastructure, particularly through stronger integration of Git into
packaging workflows.
Sam Hartman provided some historical context
[1],
[2],
[3],
[4],
noting that this discussion originally took place five years ago with no
results from several similarly lengthy threads. My favorite
summary of the
entire thread was given by Gregor Herrmann, which reflects the same gut
feeling I have and highlights a structural problem within Debian that
hinders technical changes. Addressing this issue is definitely a matter
for the Debian Project Leader, and I will try to address it during my
term.
At the time of writing these bits, a
proposal
from ftpmaster, which is being continuously discussed, might lead to a
solution. I was also asked to extend the
GR discussion periods
which I will do in separate mail.
Talk: Debian GNU/Linux for Scientific Research
I was invited to have a
talk
in the Systems-Facing Track of University of British Columbia (who is
sponsoring rack space for several Debian servers). I admit it felt a bit
strange to me after working more than 20 years for establishing Debian in
scientific environments to be invited to such a talk "because I'm DPL".
Kind regards
Andreas.
BGP routing loop using a confederation
updating old chroot in sbuild that predates usrmerge.
I have been lazy and haven't been updating the chroot; but I no longer could, so had to resolve this issue about usrmerge.
There was a file /etc/unsupported-skip-usrmerge-conversion that usrmerge package errored out on; it seemed like it means
a mark of not doing usrmerge conversion, because sbuild is a system for autobuilding and staying in split /usr for the duration of the release might be better.
I don't fully understand the rationale of the file and effects, but I just deleted it and jumped into merged usr.
A new minor release 0.4.23 of 
My work on overhauling dhcpcd as the prime replacement for ISC's discontinued DHCP client is done. The package has achieved stability, both upstream and at Debian. The only remaining points are bug #1038882 to swap the Priorities of isc-dhcp-client and dhcpcd-base in the repository's override, and swaping ifupdown's search order to put dhcpcd first.
Meanwhile, ifupdown's de-facto maintainer prompted me to sollicit opinions on which of the 4 ifupdown implementations should ship with a minimal installation for Trixie. This, in turn, re-opened the debate of what should be Debian's default network configuation framework (see the thread starting with 
Here comes the 3rd article of the 5-episode blog post series on Polis, written by Guido Berh rster, member of staff at my company Fre(i)e Software GmbH.
Enjoy also this read on Guido's work on Polis,
The Rcpp Core Team is once again pleased to announce a new release
(now at 1.0.13) of the 
Twenty years ago, it was easy to dislike Microsoft. It was the quintessential evil MegaCorp that was quick to squash competition, often ruthlessly, but in some cases slowly through a more insidious process of embracing, extending, and exterminating anything that got in the way. This was the signature personality of Ballmer-era Microsoft that also inspired and united the software freedom fighting forces that came together to safeguard things that mattered to them and were at risk.I remember the era when the Novell, SCO, and Microsoft saga cast fear, uncertainty, and doubt on the future of open Unix and Linux and on what would happen to the operating systems that we loved if the suits of Redmond prevailed. Looking back, I&aposm glad that the arc of this story has bent towards justice, and I shudder at the possibilities had it worked out differently.Looking at today&aposs Microsoft, I&aposm amazed at how much change a leader with the right vision can make to the trajectory of a company that even makes an old-school software freedom advocate as me admire and even applaud the strides it has taken in the last 10 or so years that has dramatically shifted the perception of Microsoft. The personality of the Satya-era Microsoft is one to behold. While it will take more time to win back the trust, we see the tides changing and the positivity is important for the entire industry.For Microsoft, it was TypeScript and VS Code that helped change the narrative internally which led to its internal resurgence and acceptance of open source. Its acquisition of GitHub propelled it forward within the community overnight. Its contributions to the Linux kernel and other major software projects have also been consequential in changing its public perceptions.It takes a while to claw back trust and is very easy to breach. This time, however, Microsoft seems to understand this dynamic more than it did 20 years ago. All it took was the right leadership.
