The Reproducible Builds project relies on several projects, supporters and sponsors for financial support, but they are also valued as ambassadors who spread the word about our project and the work that we do.
This is the
sixth instalment in a series featuring the projects, companies and individuals who support the Reproducible Builds project. We started this series by
featuring the Civil Infrastructure Platform project and followed this up with a
post about the Ford Foundation as well as a recent ones about
ARDC, the
Google Open Source Security Team (GOSST),
Jan Nieuwenhuizen on Bootstrappable Builds, GNU Mes and GNU Guix and
Hans-Christoph Steiner of the F-Droid project.
Today, however, we will be talking with
David A. Wheeler, the Director of Open Source Supply Chain Security at the
Linux Foundation.
Holger Levsen: Welcome, David, thanks for taking the time to talk with us today. First, could you briefly tell me about yourself?
David: Sure! I m David A. Wheeler and
I work for the
Linux Foundation as the Director of Open Source Supply Chain Security.
That just means that my job is to help open source software projects
improve their security, including its development, build, distribution,
and incorporation in larger works, all the way out to its eventual use by end-users.
In my copious free time I also teach at
George Mason University (GMU); in particular,
I teach a graduate course on how to design and implement secure software.
My background is technical. I have a Bachelor s in Electronics Engineering,
a Master s in Computer Science and a PhD in Information Technology.
My PhD dissertation is connected to reproducible builds.
My PhD dissertation was on countering the Trusting Trust attack, an attack
that subverts fundamental build system tools such as compilers.
The attack was discovered by Karger & Schell in the 1970s, and later
demonstrated & popularized by Ken Thompson.
In
my dissertation on trusting trust I showed that a process
called Diverse Double-Compiling (DDC) could detect trusting trust attacks.
That process is a specialized kind of reproducible build specifically designed
to detect trusting trust style attacks. In addition, countering the trusting trust
attack primarily becomes more important only when reproducible builds become
more common. Reproducible builds enable detection of
build-time subversions.
Most attackers wouldn t bother with a trusting trust attack if they could just
directly use a build-time subversion of the software they actually want to subvert.
Holger: Thanks for taking the time to introduce yourself to us. What do you think are the biggest challenges today in computing?
There are many big challenges in computing today. For example:
- Lack of resilience & capacity in chip fabrication. Fabs are extraordinarily expensive,
and at the high end continue to have technological advancement.
As a result, supply is failing to meet demand, and geopolitical issues raise further concerns.
We ve seen cars, gaming consoles and many other devices
unable to be delivered due to chip shortages. More fabs are
being built, and some politicians are raising concerns, but it s unclear
that current efforts will be enough.
- Lack of enough developers able to develop the software that people & organizations need.
Computers are far faster, and open source software has made software reuse
incredibly easy. However, organizations still struggle to automate
many tasks. The bottleneck is the lack of enough talented developers able to convert
ideas into working software. Low-code and no-code approaches help in specialized areas,
just like all previous automate the programmer efforts of the last 60 years, but
there s no reason to believe they will help enough.
- Large scale of software. Small systems are easier to develop & maintain, but today s
systems increasingly get bigger to meet users needs & are much harder to manage.
Even small embedded systems are often supported by huge back-end systems.
- Ending tail of Moore s law & rise of smartphones. Historically people would just wait a few years for their
software to speed up, but Moore s law is petering out, and smartphones are necessarily
limited by power & size limits. As a result, software developers
can t wait for the hardware to save their slow systems; they must redesign.
Switching to faster languages, or using multiple processors, is much more difficult than
waiting for performance problems to disappear.
- Continuous change in interfaces. Developers continuously find reasons to change
component interfaces: perhaps they re too inflexible, too hard to use, and so on.
But now that developers are reusing hundreds, thousands, or tens of thousands of components,
managing the continuous change of the reused components is challenging.
Package managers make updating easy but don t automatically handle interface changes.
I think this is mostly a self-inflicted problem most components could support old interfaces
(like the Linux kernel does) but because it s often not acknowledged as a problem, it s often not addressed.
- Security & privacy. Decades ago there were fewer computers and most computers weren t connected to a network.
Today things are different. Criminals have found many ways to attack computer systems to
make money, and nation-states have found many ways to attack computer systems for their own reasons.
Attackers now have very strong motivations to perform attacks.
Yet many developers aren t told how to develop software that resists attacks, nor
how to protect their supply chains. Operations try to monitor and recover from
attacks, but their job is difficult due to inadequately secure software that doesn t
support those monitoring & recovery efforts well either. The results are terrible security.
Holger: Do you think reproducible builds are an important part in secure computing today already?
David: Yes, but first let s put things in context.
Today, when attackers exploit software vulnerabilities, they re primarily
exploiting unintentional vulnerabilities that were created by the software
developers. There are a lot of efforts to counter this:
- Train & education developers in how to develop secure software.
The OpenSSF provides a free course on how to do that (full disclosure: I m the author).
Take that course or something like it!
- Add tools to your CI pipeline to detect potential vulnerabilities. Yes, they have false
positives and false negatives, so you have to also use your brain but that just means you
need to be smart about using tools, instead of not using them.
- Get projects & organizations to update the components they use,
since often the vulnerabilities are well-known publicly
(e.g., Equifax in 2017). Add some tools to your development process to warn you about
components with known vulnerabilities! GitHub & GitLab both provide tools to do this,
and there are many other tools.
- When starting new projects, try to use memory-safe languages. On average 70% of the
vulnerabilities in Chrome and in Microsoft are from memory safety problems; using a memory-safe
language eliminates most of them.
We re just starting to get better at this, which is good. However, attackers always
try to attack the easiest target. As our deployed software has started to be hardened
against attack, attackers have dramatically increased their attacks
on the software supply chain (
Sonatype found in 2022 that there s been a 742% increase year-over-year).
The software supply chain hasn t historically gotten much attention, making it the easy target.
There are simple supply chain attacks with simple solutions:
- In almost every year the top attack has been typosquatting. In typo squatting,
an attacker creates packages with almost the right name. This is an easy attack to
counter developers just need to double-check the name of a package before adding it.
But we aren t warning developers enough about it!
For more information, see papers such as the Backstabber s Knife Collection.
- Last year the top software supply chain attack was dependency confusion convincing
projects to use the wrong repo for a given package. There are simple solutions to this, such as
specifying the package source and/or requiring a cryptographic hash to match.
- Some attacks involve takeovers of developer accounts. In almost all cases, these are
caused by stolen passwords. Using a multi-factor authentication (MFA) token eliminates
stolen password attacks, which is why several
repositories are starting to require MFA tokens in some cases.
Unfortunately, attackers know there are other lines of attack.
One of the most dangerous is subverted build systems, as demonstrated by
the subversion of SolarWinds Orion system. In a subverted build system,
developers can review the software source code all day and see no problem,
because there
is no problem there. Instead, the process to convert source code
into the code people run, called the build system , is subverted by an attacker.
One solution for countering subverted build systems is to make the build systems harder
to attack. That s a good thing to do, but you can never be confident that it was good enough .
How can you be sure it s not subverted, if there s no way to know?
A stronger defense against subverted build systems is the idea of verified reproducible builds.
A build is reproducible if given the same source code, build environment and build instructions,
any party can recreate bit-by-bit identical copies of all specified artifacts.
A build is
verified if multiple different parties verify that they get the same result for that situation.
When you have a verified reproducible build, either all the parties colluded
(and you could always double-check it yourself), or the build process isn t subverted.
There is one last turtle: What if the build system tools or machines are subverted themselves?
This is not a common attack today, but it s important to know if we
can address them
when the time comes. The good news is that we
can address this.
For some situations reproducible builds can also counter such attacks.
If there s a loop (that is, a compiler is used to generate itself), that s called the trusting trust attack,
and that is more challenging. Thankfully, the trusting trust attack has been known about for
decades and there are known solutions. The diverse double-compiling (DDC) process that
I explained in my PhD dissertation, as well as the bootstrappable builds process, can
both counter trusting trust attacks in the software space. So there is no reason to lose hope:
there is a bottom turtle , as it were.
Holger: Thankfully, this has all slowly started to change and supply chain issues are now widely discussed, as evident by efforts like
Securing the Software Supply Chain: Recommended Practices Guide for Developers
which you shared
on our mailing list. In there, Reproducible Builds are mentioned as recommended advanced practice, which is both pretty cool (we ve come a long way!), but to me it also sounds like this will take another decade until it s become standard normal procedure. Do you agree on that timeline?
David: I don t think there will be any particular timeframe. Different projects and
ecosystems will move at different speeds. I wouldn t be surprised if it
took a decade or so for them to become relatively common there are
good reasons for that.
Today the most common kinds of attacks based on software
vulnerabilities still involve unintentional vulnerabilities in operational systems.
Attackers are starting to apply supply chain attacks, but the top such attacks
today are typosquatting (creating packages with similar names) and
dependency confusion) (convincing projects to download packages from the wrong
repositories).
Reproducible builds don t counter those kinds of attacks, they
counter subverted builds. It s important to eventually have verified
reproducible builds, but understandably other issues are currently getting
prioritized first.
That said, reproducible builds are important long term.
Many people are working on countering unintentional vulnerabilities
and the most common kinds of supply chain attacks.
As these other threats are countered, attackers will increasingly target
build systems. Attackers always go for the weakest link.
We will eventually need verified reproducible builds in many situations, and
it ll take a while to get build systems able to widely perform reproducible builds,
so we need to start that work now. That s true for anything where you know
you ll need it but it will take a long time to get ready you need to start now.
Holger: What are your suggestions to accelerate adoption?
David: Reproducible builds need to be:
- Easy (ideally automatic). Tools need to be modified so that reproducible builds
are the default or at least easier to do.
- Transparent to projects & potential users. Many projects have no idea that their results aren t
reproducible, and many potential users of the project don t know either.
That information needs to be obvious. I ve proposed that the OpenSSF
Dashboard SIG try to reproduce builds, for at least some packages, to make it
more obvious to everyone when a project isn t reproducible. I don t know if that
will happen in that particular case, but the point is to help people learn that information
as soon as possible.
- Deployed.
Experiments are great, but experiments showing that a project could be reproducible
are inadequate. We need the projects that people use to be reproducible.
I think there s a snowball effect. Once many projects packages are reproducible,
it will be easier to convince other projects to make their packages reproducible.
I also think there should be some prioritization. If a package is in wide use
(e.g., part of minimum set of packages for a widely-used Linux distribution or
framework), its reproducibility should be a special focus. If a package is vital for
supporting some societally important critical infrastructure (e.g., running dams),
it should also be considered important. You can then work on the
ones that are less important over time.
Holger: How is the
Best Practices Badge going? How many projects are participating and how many are missing?
David: It s going very well. You can
see some automatically-generated statistics, showing we have over 5,000 projects, adding more than 1/day on average.
We have more than 900 projects that have earned at least the passing badge level.
Holger: How many of the projects participating in the Best Practices badge engaging with reproducible builds?
David: As of this writing there are 168 projects that report meeting the reproducible builds criterion.
That s a relatively small percentage of projects. However, note that this criterion (labelled
build_reproducible)
is only required for the gold badge. It s not required for the passing or silver level badge.
Currently we ve been strategically focused on getting projects to at least earn a passing badge,
and less on earning silver or gold badges.
We would
love for all projects to get earn a silver or gold badge, of course, but
our theory is that projects that can t even earn a passing badge present the most risk to their users.
That said, there are some projects we especially want to see implementing higher badge levels.
Those include projects that are very widely used, so that
vulnerabilities in them can impact many systems.
Examples of such projects include the Linux kernel and curl.
In addition, some projects are used within
systems where it s important to society that they not have serious security vulnerabilities.
Examples include projects used by
chemical manufacturers, financial systems and weapons.
We definitely encourage any of those kinds of projects to earn higher badge levels.
Holger: Many thanks for this interview, David, and for all of your work at the Linux Foundation and elsewhere!
For more information about the Reproducible Builds project, please see our website at
reproducible-builds.org. If you are interested in
ensuring the ongoing security of the software that underpins our civilisation
and wish to sponsor the Reproducible Builds project, please reach out to the
project by emailing
contact@reproducible-builds.org.
Long-term Linux users may remember that 
In my two most recent posts, I listed the 
Two weeks ago, I had the chance to go see Cory Doctorow at my local independent
bookstore, in Montr al. He was there to present his latest essay, co-written
with Rebecca Giblin
Although Doctorow is known for his strong political stances, I have to say I'm
quite surprised by the quality of the research Giblin and he did for this book.
They both show a pretty advanced understanding of the market dynamics they look
at, and even though most of the solutions they propose aren't new or
groundbreaking, they manage to be convincing and clear.
That is to say, you certainly don't need to be an economist to understand or
enjoy this book :)
As I have mentioned before, the book heavily criticises monopolies, but also
monopsonies a market structure that has only one buyer (instead of one
seller). I find this quite interesting, as whereas people are often familiar
with the concept of monopolies, monopsonies are frequently overlooked.
The classic example of a monopsony is a labor market with a single employer:
there is a multitude of workers trying to sell their labor power, but in the
end, working conditions are dictated by the sole employer, who gets to decide
who has a job and who hasn't. Mining towns are good real-world examples of
monopsonies.
In the book, the authors argue most of the contemporary work produced by
creative workers (especially musicians and writers) is sold to monopsonies and
. What is the basis of that or why is there such kind of confidence I have no clue or idea, it is what it is. Approximately 24 hrs. ago, apparently we had a cloudburst. What I came to know later is that we got 100 mm of rain. Sharing from local 
. For those who have read this blog, do know that India is and has been in casteism from ever. They even took the fair comment and applied it to all Brahmins. According to them, all Brahmins are fair and hence have god-given right to lord over others. What is called the Eton boy s network serves the same in this casteism. The only solution is those idea under limelight and investigate. To take the above, how does one prove that all fair people are wise and peaceful while all people black and brown are violent. If that is so, how does one count for Mahatma Gandhi, Martin Luther King Junior, Nelson Mandela, Michael Jackson the list is probably endless. And not to forget that when Mahatma Gandhiji did his nonviolent movements either in India or in South Africa, both black and brown people in millions took part. Similar examples of Martin Luther King Jr. I know and read of so many non-violent civl movements that took place in the U.S. For e.g. 
It's time for another update on Debian's firmware GR. I wrote about
the
problem