Back to the Future Time to admit something. I really love Plan 9. It s just so good. So many ideas from Plan 9 are just so prescient, and everything just feels right. Not just right like, feels good like, correct. The bit that I ve always liked the most is 9p, the network protocol for serving a filesystem over a network. This leads to all sorts of fun programs, like the Plan 9 ftp client being a 9p server you mount the ftp server and access files like any other files. It s kinda like if fuse were more fully a part of how the operating system worked, but fuse is all running client-side. With 9p there s a single client, and different servers that you can connect to, which may be backed by a hard drive, remote resources over something like SFTP, FTP, HTTP or even purely synthetic. The interesting (maybe sad?) part here is that 9p wound up outliving Plan 9 in terms of adoption 9p is in all sorts of places folks don t usually expect. For instance, the Windows Subsystem for Linux uses the 9p protocol to share files between Windows and Linux. ChromeOS uses it to share files with Crostini, and qemu uses 9p (virtio-p9) to share files between guest and host. If you re noticing a pattern here, you d be right; for some reason 9p is the go-to protocol to exchange files between hypervisor and guest. Why? I have no idea, except maybe due to being designed well, simple to implement, and it s a lot easier to validate the data being shared and validate security boundaries. Simplicity has its value. As a result, there s a lot of lingering 9p support kicking around. Turns out Linux can even handle mounting 9p filesystems out of the box. This means that I can deploy a filesystem to my LAN or my localhost by running a process on top of a computer that needs nothing special, and mount it over the network on an unmodified machine unlike fuse, where you d need client-specific software to run in order to mount the directory. For instance, let s mount a 9p filesystem running on my localhost machine, serving requests on 127.0.0.1:564 (tcp) that goes by the name mountpointname to /mnt.

$ mount -t 9p \
-o trans=tcp,port=564,version=9p2000.u,aname=mountpointname \
127.0.0.1 \
/mnt

Linux will mount away, and attach to the filesystem as the root user, and by default, attach to that mountpoint again for each local user that attempts to use it. Nifty, right? I think so. The server is able to keep track of per-user access and authorization along with the host OS.

WHEREIN I STYX WITH IT Since I wanted to push myself a bit more with rust and tokio specifically, I opted to implement the whole stack myself, without third party libraries on the critical path where I could avoid it. The 9p protocol (sometimes called Styx, the original name for it) is incredibly simple. It s a series of client to server requests, which receive a server to client response. These are, respectively, T messages, which transmit a request to the server, which trigger an R message in response (Reply messages). These messages are TLV payload with a very straight forward structure so straight forward, in fact, that I was able to implement a working server off nothing more than a handful of man pages. Later on after the basics worked, I found a more complete spec page that contains more information about the unix specific variant that I opted to use (9P2000.u rather than 9P2000) due to the level of Linux specific support for the 9P2000.u variant over the 9P2000 protocol.

MR ROBOTO The backend stack over at zoo is rust and tokio running i/o for an HTTP and WebRTC server. I figured I d pick something fairly similar to write my filesystem with, since 9P can be implemented on basically anything with I/O. That means tokio tcp server bits, which construct and use a 9p server, which has an idiomatic Rusty API that partially abstracts the raw R and T messages, but not so much as to cause issues with hiding implementation possibilities. At each abstraction level, there s an escape hatch allowing someone to implement any of the layers if required. I called this framework arigato which can be found over on docs.rs and crates.io.

/// Simplified version of the arigato File trait; this isn't actually
/// the same trait; there's some small cosmetic differences. The
/// actual trait can be found at:
///
/// https://docs.rs/arigato/latest/arigato/server/trait.File.html
trait File  
/// OpenFile is the type returned by this File via an Open call.
 type OpenFile: OpenFile;
/// Return the 9p Qid for this file. A file is the same if the Qid is
 /// the same. A Qid contains information about the mode of the file,
 /// version of the file, and a unique 64 bit identifier.
 fn qid(&self) -> Qid;
/// Construct the 9p Stat struct with metadata about a file.
 async fn stat(&self) -> FileResult<Stat>;
/// Attempt to update the file metadata.
 async fn wstat(&mut self, s: &Stat) -> FileResult<()>;
/// Traverse the filesystem tree.
 async fn walk(&self, path: &[&str]) -> FileResult<(Option<Self>, Vec<Self>)>;
/// Request that a file's reference be removed from the file tree.
 async fn unlink(&mut self) -> FileResult<()>;
/// Create a file at a specific location in the file tree.
 async fn create(
&mut self,
name: &str,
perm: u16,
ty: FileType,
mode: OpenMode,
extension: &str,
) -> FileResult<Self>;
/// Open the File, returning a handle to the open file, which handles
 /// file i/o. This is split into a second type since it is genuinely
 /// unrelated -- and the fact that a file is Open or Closed can be
 /// handled by the  arigato  server for us.
 async fn open(&mut self, mode: OpenMode) -> FileResult<Self::OpenFile>;
 
/// Simplified version of the arigato OpenFile trait; this isn't actually
/// the same trait; there's some small cosmetic differences. The
/// actual trait can be found at:
///
/// https://docs.rs/arigato/latest/arigato/server/trait.OpenFile.html
trait OpenFile  
/// iounit to report for this file. The iounit reported is used for Read
 /// or Write operations to signal, if non-zero, the maximum size that is
 /// guaranteed to be transferred atomically.
 fn iounit(&self) -> u32;
/// Read some number of bytes up to  buf.len()  from the provided
 ///  offset  of the underlying file. The number of bytes read is
 /// returned.
 async fn read_at(
&mut self,
buf: &mut [u8],
offset: u64,
) -> FileResult<u32>;
/// Write some number of bytes up to  buf.len()  from the provided
 ///  offset  of the underlying file. The number of bytes written
 /// is returned.
 fn write_at(
&mut self,
buf: &mut [u8],
offset: u64,
) -> FileResult<u32>;
 

Thanks, decade ago paultag! Let s do it! Let s use arigato to implement a 9p filesystem we ll call debugfs that will serve all the debug files shipped according to the Packages metadata from the apt archive. We ll fetch the Packages file and construct a filesystem based on the reported Build-Id entries. For those who don t know much about how an apt repo works, here s the 2-second crash course on what we re doing. The first is to fetch the Packages file, which is specific to a binary architecture (such as amd64, arm64 or riscv64). That architecture is specific to a component (such as main, contrib or non-free). That component is specific to a suite, such as stable, unstable or any of its aliases (bullseye, bookworm, etc). Let s take a look at the Packages.xz file for the unstable-debug suite, main component, for all amd64 binaries.

$ curl \
https://deb.debian.org/debian-debug/dists/unstable-debug/main/binary-amd64/Packages.xz \
  unxz

This will return the Debian-style rfc2822-like headers, which is an export of the metadata contained inside each .deb file which apt (or other tools that can use the apt repo format) use to fetch information about debs. Let s take a look at the debug headers for the netlabel-tools package in unstable which is a package named netlabel-tools-dbgsym in unstable-debug.

Package: netlabel-tools-dbgsym
Source: netlabel-tools (0.30.0-1)
Version: 0.30.0-1+b1
Installed-Size: 79
Maintainer: Paul Tagliamonte <paultag@debian.org>
Architecture: amd64
Depends: netlabel-tools (= 0.30.0-1+b1)
Description: debug symbols for netlabel-tools
Auto-Built-Package: debug-symbols
Build-Ids: e59f81f6573dadd5d95a6e4474d9388ab2777e2a
Description-md5: a0e587a0cf730c88a4010f78562e6db7
Section: debug
Priority: optional
Filename: pool/main/n/netlabel-tools/netlabel-tools-dbgsym_0.30.0-1+b1_amd64.deb
Size: 62776
SHA256: 0e9bdb087617f0350995a84fb9aa84541bc4df45c6cd717f2157aa83711d0c60

So here, we can parse the package headers in the Packages.xz file, and store, for each Build-Id, the Filename where we can fetch the .deb at. Each .deb contains a number of files but we re only really interested in the files inside the .deb located at or under /usr/lib/debug/.build-id/, which you can find in debugfs under rfc822.rs. It s crude, and very single-purpose, but I m feeling a bit lazy.

Who needs dpkg?! For folks who haven t seen it yet, a .deb file is a special type of .ar file, that contains (usually) three files inside debian-binary, control.tar.xz and data.tar.xz. The core of an .ar file is a fixed size (60 byte) entry header, followed by the specified size number of bytes.

[8 byte .ar file magic]
[60 byte entry header]
[N bytes of data]
[60 byte entry header]
[N bytes of data]
[60 byte entry header]
[N bytes of data]
...

First up was to implement a basic ar parser in ar.rs. Before we get into using it to parse a deb, as a quick diversion, let s break apart a .deb file by hand something that is a bit of a rite of passage (or at least it used to be? I m getting old) during the Debian nm (new member) process, to take a look at where exactly the .debug file lives inside the .deb file.

$ ar x netlabel-tools-dbgsym_0.30.0-1+b1_amd64.deb
$ ls
control.tar.xz debian-binary
data.tar.xz netlabel-tools-dbgsym_0.30.0-1+b1_amd64.deb
$ tar --list -f data.tar.xz   grep '.debug$'
./usr/lib/debug/.build-id/e5/9f81f6573dadd5d95a6e4474d9388ab2777e2a.debug

Since we know quite a bit about the structure of a .deb file, and I had to implement support from scratch anyway, I opted to implement a (very!) basic debfile parser using HTTP Range requests. HTTP Range requests, if supported by the server (denoted by a accept-ranges: bytes HTTP header in response to an HTTP HEAD request to that file) means that we can add a header such as range: bytes=8-68 to specifically request that the returned GET body be the byte range provided (in the above case, the bytes starting from byte offset 8 until byte offset 68). This means we can fetch just the ar file entry from the .deb file until we get to the file inside the .deb we are interested in (in our case, the data.tar.xz file) at which point we can request the body of that file with a final range request. I wound up writing a struct to handle a read_at-style API surface in hrange.rs, which we can pair with ar.rs above and start to find our data in the .deb remotely without downloading and unpacking the .deb at all. After we have the body of the data.tar.xz coming back through the HTTP response, we get to pipe it through an xz decompressor (this kinda sucked in Rust, since a tokio AsyncRead is not the same as an http Body response is not the same as std::io::Read, is not the same as an async (or sync) Iterator is not the same as what the xz2 crate expects; leading me to read blocks of data to a buffer and stuff them through the decoder by looping over the buffer for each lzma2 packet in a loop), and tarfile parser (similarly troublesome). From there we get to iterate over all entries in the tarfile, stopping when we reach our file of interest. Since we can t seek, but gdb needs to, we ll pull it out of the stream into a Cursor<Vec<u8>> in-memory and pass a handle to it back to the user. From here on out its a matter of gluing together a File traited struct in debugfs, and serving the filesystem over TCP using arigato. Done deal!

A quick diversion about compression I was originally hoping to avoid transferring the whole tar file over the network (and therefore also reading the whole debug file into ram, which objectively sucks), but quickly hit issues with figuring out a way around seeking around an xz file. What s interesting is xz has a great primitive to solve this specific problem (specifically, use a block size that allows you to seek to the block as close to your desired seek position just before it, only discarding at most block size - 1 bytes), but data.tar.xz files generated by dpkg appear to have a single mega-huge block for the whole file. I don t know why I would have expected any different, in retrospect. That means that this now devolves into the base case of How do I seek around an lzma2 compressed data stream ; which is a lot more complex of a question. Thankfully, notoriously brilliant tianon was nice enough to introduce me to Jon Johnson who did something super similar adapted a technique to seek inside a compressed gzip file, which lets his service oci.dag.dev seek through Docker container images super fast based on some prior work such as soci-snapshotter, gztool, and zran.c. He also pulled this party trick off for apk based distros over at apk.dag.dev, which seems apropos. Jon was nice enough to publish a lot of his work on this specifically in a central place under the name targz on his GitHub, which has been a ton of fun to read through. The gist is that, by dumping the decompressor s state (window of previous bytes, in-memory data derived from the last N-1 bytes) at specific checkpoints along with the compressed data stream offset in bytes and decompressed offset in bytes, one can seek to that checkpoint in the compressed stream and pick up where you left off creating a similar block mechanism against the wishes of gzip. It means you d need to do an O(n) run over the file, but every request after that will be sped up according to the number of checkpoints you ve taken. Given the complexity of xz and lzma2, I don t think this is possible for me at the moment especially given most of the files I ll be requesting will not be loaded from again especially when I can just cache the debug header by Build-Id. I want to implement this (because I m generally curious and Jon has a way of getting someone excited about compression schemes, which is not a sentence I thought I d ever say out loud), but for now I m going to move on without this optimization. Such a shame, since it kills a lot of the work that went into seeking around the .deb file in the first place, given the debian-binary and control.tar.gz members are so small.

The Good First, the good news right? It works! That s pretty cool. I m positive my younger self would be amused and happy to see this working; as is current day paultag. Let s take debugfs out for a spin! First, we need to mount the filesystem. It even works on an entirely unmodified, stock Debian box on my LAN, which is huge. Let s take it for a spin:

$ mount \
-t 9p \
-o trans=tcp,version=9p2000.u,aname=unstable-debug \
192.168.0.2 \
/usr/lib/debug/.build-id/

And, let s prove to ourselves that this actually mounted before we go trying to use it:

$ mount   grep build-id
192.168.0.2 on /usr/lib/debug/.build-id type 9p (rw,relatime,aname=unstable-debug,access=user,trans=tcp,version=9p2000.u,port=564)

Slick. We ve got an open connection to the server, where our host will keep a connection alive as root, attached to the filesystem provided in aname. Let s take a look at it.

$ ls /usr/lib/debug/.build-id/
00 0d 1a 27 34 41 4e 5b 68 75 82 8E 9b a8 b5 c2 CE db e7 f3
01 0e 1b 28 35 42 4f 5c 69 76 83 8f 9c a9 b6 c3 cf dc E7 f4
02 0f 1c 29 36 43 50 5d 6a 77 84 90 9d aa b7 c4 d0 dd e8 f5
03 10 1d 2a 37 44 51 5e 6b 78 85 91 9e ab b8 c5 d1 de e9 f6
04 11 1e 2b 38 45 52 5f 6c 79 86 92 9f ac b9 c6 d2 df ea f7
05 12 1f 2c 39 46 53 60 6d 7a 87 93 a0 ad ba c7 d3 e0 eb f8
06 13 20 2d 3a 47 54 61 6e 7b 88 94 a1 ae bb c8 d4 e1 ec f9
07 14 21 2e 3b 48 55 62 6f 7c 89 95 a2 af bc c9 d5 e2 ed fa
08 15 22 2f 3c 49 56 63 70 7d 8a 96 a3 b0 bd ca d6 e3 ee fb
09 16 23 30 3d 4a 57 64 71 7e 8b 97 a4 b1 be cb d7 e4 ef fc
0a 17 24 31 3e 4b 58 65 72 7f 8c 98 a5 b2 bf cc d8 E4 f0 fd
0b 18 25 32 3f 4c 59 66 73 80 8d 99 a6 b3 c0 cd d9 e5 f1 fe
0c 19 26 33 40 4d 5a 67 74 81 8e 9a a7 b4 c1 ce da e6 f2 ff

Outstanding. Let s try using gdb to debug a binary that was provided by the Debian archive, and see if it ll load the ELF by build-id from the right .deb in the unstable-debug suite:

$ gdb -q /usr/sbin/netlabelctl
Reading symbols from /usr/sbin/netlabelctl...
Reading symbols from /usr/lib/debug/.build-id/e5/9f81f6573dadd5d95a6e4474d9388ab2777e2a.debug...
(gdb)

Yes! Yes it will!

$ file /usr/lib/debug/.build-id/e5/9f81f6573dadd5d95a6e4474d9388ab2777e2a.debug
/usr/lib/debug/.build-id/e5/9f81f6573dadd5d95a6e4474d9388ab2777e2a.debug: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter *empty*, BuildID[sha1]=e59f81f6573dadd5d95a6e4474d9388ab2777e2a, for GNU/Linux 3.2.0, with debug_info, not stripped

The Bad Linux s support for 9p is mainline, which is great, but it s not robust. Network issues or server restarts will wedge the mountpoint (Linux can t reconnect when the tcp connection breaks), and things that work fine on local filesystems get translated in a way that causes a lot of network chatter for instance, just due to the way the syscalls are translated, doing an ls, will result in a stat call for each file in the directory, even though linux had just got a stat entry for every file while it was resolving directory names. On top of that, Linux will serialize all I/O with the server, so there s no concurrent requests for file information, writes, or reads pending at the same time to the server; and read and write throughput will degrade as latency increases due to increasing round-trip time, even though there are offsets included in the read and write calls. It works well enough, but is frustrating to run up against, since there s not a lot you can do server-side to help with this beyond implementing the 9P2000.L variant (which, maybe is worth it).

The Ugly Unfortunately, we don t know the file size(s) until we ve actually opened the underlying tar file and found the correct member, so for most files, we don t know the real size to report when getting a stat. We can t parse the tarfiles for every stat call, since that d make ls even slower (bummer). Only hiccup is that when I report a filesize of zero, gdb throws a bit of a fit; let s try with a size of 0 to start:

$ ls -lah /usr/lib/debug/.build-id/e5/9f81f6573dadd5d95a6e4474d9388ab2777e2a.debug
-r--r--r-- 1 root root 0 Dec 31 1969 /usr/lib/debug/.build-id/e5/9f81f6573dadd5d95a6e4474d9388ab2777e2a.debug
$ gdb -q /usr/sbin/netlabelctl
Reading symbols from /usr/sbin/netlabelctl...
Reading symbols from /usr/lib/debug/.build-id/e5/9f81f6573dadd5d95a6e4474d9388ab2777e2a.debug...
warning: Discarding section .note.gnu.build-id which has a section size (24) larger than the file size [in module /usr/lib/debug/.build-id/e5/9f81f6573dadd5d95a6e4474d9388ab2777e2a.debug]
[...]

This obviously won t work since gdb will throw away all our hard work because of stat s output, and neither will loading the real size of the underlying file. That only leaves us with hardcoding a file size and hope nothing else breaks significantly as a result. Let s try it again:

$ ls -lah /usr/lib/debug/.build-id/e5/9f81f6573dadd5d95a6e4474d9388ab2777e2a.debug
-r--r--r-- 1 root root 954M Dec 31 1969 /usr/lib/debug/.build-id/e5/9f81f6573dadd5d95a6e4474d9388ab2777e2a.debug
$ gdb -q /usr/sbin/netlabelctl
Reading symbols from /usr/sbin/netlabelctl...
Reading symbols from /usr/lib/debug/.build-id/e5/9f81f6573dadd5d95a6e4474d9388ab2777e2a.debug...
(gdb)

Much better. I mean, terrible but better. Better for now, anyway.

Kilroy was here Do I think this is a particularly good idea? I mean; kinda. I m probably going to make some fun 9p arigato-based filesystems for use around my LAN, but I don t think I ll be moving to use debugfs until I can figure out how to ensure the connection is more resilient to changing networks, server restarts and fixes on i/o performance. I think it was a useful exercise and is a pretty great hack, but I don t think this ll be shipping anywhere anytime soon. Along with me publishing this post, I ve pushed up all my repos; so you should be able to play along at home! There s a lot more work to be done on arigato; but it does handshake and successfully export a working 9P2000.u filesystem. Check it out on on my github at arigato, debugfs and also on crates.io and docs.rs. At least I can say I was here and I got it working after all these years.

grab Docker s APT repo GPG key

GNUPGHOME="$(mktemp -d)"; export GNUPGHOME
gpg --keyserver ha.pool.sks-keyservers.net --recv-keys 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
sudo mkdir -p /etc/apt/tianon.gpg.d
gpg --export --armor 9DC858229FC7DD38854AE2D88D81803C0EBFCD88   sudo tee /etc/apt/tianon.gpg.d/docker.gpg.asc
rm -rf "$GNUPGHOME"

add Docker s APT source

source /etc/os-release
echo "deb [ arch=amd64 signed-by=/etc/apt/tianon.gpg.d/docker.gpg.asc ] https://download.docker.com/linux/debian $VERSION_CODENAME stable"   sudo tee /etc/apt/sources.list.d/docker.list

$ sudo apt update
...
Get:6 https://download.docker.com/linux/debian buster/stable amd64 Packages [17.8 kB]
...
Reading package lists... Done

exclude (unwated) CLI plugins

echo 'path-exclude /usr/libexec/docker/cli-plugins/*'   sudo tee /etc/dpkg/dpkg.cfg.d/unwanted-docker-cli-plugins

pin Docker versions

sudo vim /etc/apt/preferences.d/docker.pref

Package: *aufs* *rootless* cgroupfs-mount
Pin: version *
Pin-Priority: -10
Package: docker*
Pin: version 5:20.10*
Pin-Priority: 999
Package: containerd*
Pin: version 1.4*
Pin-Priority: 999

pre-configure Docker

sudo mkdir -p /etc/docker
sudo vim /etc/docker/daemon.json

 
	"storage-driver": "overlay2"
 

configure boot parameters

I usually set a few boot parameters as well (in /etc/default/grub s GRUB_CMDLINE_LINUX_DEFAULT option run sudo update-grub after adding these, space-separated).

• cgroup_enable=memory enable memory accounting for containers (allows docker run --memory for setting hard memory limits on containers)
• swapaccount=1 enable swap accounting for containers (allows docker run --memory-swap for setting hard swap memory limits on containers)
• vsyscall=emulate allow older binaries to run (debian:wheezy, etc.; see docker/docker#28705)
• systemd.legacy_systemd_cgroup_controller=yes newer versions of systemd may disable the legacy cgroup interfaces Docker currently uses; this instructs systemd to keep those enabled (for more details, see systemd/systemd#4628, opencontainers/runc#1175, docker/docker#28109)
  • NOTE: this one gets more complicated in Debian 11+ ( Bullseye ); possibly worth switching to cgroupv2 and systemd.unified_cgroup_hierarchy=1

All together:

...
GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1 vsyscall=emulate systemd.legacy_systemd_cgroup_controller=yes"
...

(Don t forget to sudo update-grub and potentially reboot check /proc/cmdline to verify.)

install Docker!

$ sudo apt-get install -V docker-ce
...
Unpacking containerd.io (1.4.4-1) ...
...
Unpacking docker-ce-cli (5:20.10.5~3-0~debian-buster) ...
...
Unpacking docker-ce (5:20.10.5~3-0~debian-buster) ...
...

$ sudo usermod -aG docker "$(id -un)"

Welcome to the February 2020 report from the Reproducible Builds project. One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the reproducible builds effort is to provide the ability to demonstrate these binaries originated from a particular, trusted, source release: if identical results are generated from a given source in all circumstances, reproducible builds provides the means for multiple third-parties to reach a consensus on whether a build was compromised via distributed checksum validation or some other scheme. In this month s report, we cover:

• Media coverage & upstream news A new paper on reproducible containers, Ruby updates, etc.
• Distribution work More work in Debian, openSUSE & friends.
• Software development Updates and improvements to our tooling.
• Getting in touch How to contribute, more venues for discussion.

If you are interested in contributing to the project, please visit our Contribute page on our website.

Media coverage & upstream news Omar Navarro Leija, a PhD student at the University Of Pennsylvania, published a paper entitled Reproducible Containers that describes in detail the workings of a new user-space container tool called DetTrace:

All computation that occurs inside a DetTrace container is a pure function of the initial filesystem state of the container. Reproducible containers can be used for a variety of purposes, including replication for fault-tolerance, reproducible software builds and reproducible data analytics. We use DetTrace to achieve, in an automatic fashion, reproducibility for 12,130 Debian package builds, containing over 800 million lines of code, as well as bioinformatics and machine learning workflows.

There was also considerable discussion on our mailing list regarding this research and a presentation based on the paper will occur at the ASPLOS 2020 conference between March 16th 20th in Lausanne, Switzerland. The many virtues of Reproducible Builds were touted as benefits for software compliance in a talk at FOSDEM 2020, debating whether the Careful Inventory of Licensing Bill of Materials Have Impact of FOSS License Compliance which pitted Jeff McAffer and Carol Smith against Bradley Kuhn and Max Sills. (~47 minutes in). Nobuyoshi Nakada updated the canonical implementation of the Ruby programming language a change such that filesystem globs (ie. calls to list the contents of filesystem directories) will henceforth be sorted in ascending order. Without this change, the underlying nondeterministic ordering of the filesystem is exposed to the language which often results in an unreproducible build. Vagrant Cascadian reported on our mailing list regarding a quick reproducible test for the GNU Guix distribution, which resulted in 81.9% of packages registering as reproducible in his installation:

$ guix challenge --verbose --diff=diffoscope ...
2,463 store items were analyzed:
  - 2,016 (81.9%) were identical
  - 37 (1.5%) differed
  - 410 (16.6%) were inconclusive

Jeremiah Orians announced on our mailing list the release of a number of tools related to cross-compilation such as M2-Planet and mescc-tools-seed. This project attemps a full bootstrap of a cross-platform compiler for the C programming language (written in C itself) from hex, the ultimate goal being able to demonstrate fully-bootstrapped compiler from hex to the GCC GNU Compiler Collection. This has many implications in and around Ken Thompson s Trusting Trust attack outlined in Thompson s 1983 Turing Award Lecture. Twitter user @TheYoctoJester posted an executive summary of reproducible builds in the Yocto Project: Finally, Reddit user tofflos posted to the /r/Java subreddit asking about how to achieve reproducible builds with Maven and Chris Lamb noticed that the Linux kernel documentation about reproducible builds of it is available on the kernel.org homepages in an attractive HTML format.

Distribution work

Debian Chris Lamb created a merge request for the core debian-installer package to allow all arguments and options from sources.list files (such as [check-valid-until=no] , etc.) in order that we can test the reproducibility of the installer images on the Reproducible Builds own testing infrastructure. (#13) Thorsten Glaser followed-up to a bug filed against the dpkg-source component that was originally filed in late 2015 that claims that the build tool does not respect permissions when unpacking tarballs if the umask is set to 0002. Matthew Garrett posted to the debian-devel mailing list on the topic of Producing verifiable initramfs images as part of a wider conversation on being able to trust the entire software stack on our computers. 59 reviews of Debian packages were added, 30 were updated and 42 were removed this month adding to our knowledge about identified issues. Many issue types were noticed and categorised by Chris Lamb, including:

• openstack_pkg_tools_python_shebang_and_dependencies
• build_path_in_code_generated_by_dgbus_codegen
• random_argument_handling_in_javatools_jh_build
• fortran_captures_build_path
• etc.

openSUSE In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update as well as provided the following patches:

• python-rpm-macros (do not save time-based .pyc files for tests)
• solfege (filesystem ordering issue sent upstream via email; package is orphaned upstream)
• DVDStyler (zip timestamps, submitted upstream)
• python-pikepdf (recreate unreproducible .pyc files)

Software development

diffoscope diffoscope is our in-depth and content-aware diff-like utility that can locate and diagnose reproducibility issues. It is run countless times a day on our testing infrastructure and is essential for identifying fixes and causes of nondeterministic behaviour. Chris Lamb made the following changes this month, including uploading version 137 to Debian:

• The sng image utility appears to return with an exit code of 1 if there are even minor errors in the file. (#950806)
• Also extract classes2.dex, classes3.dex from .apk files extracted by apktool. (#88)
• No need to use str.format if we are just returning the string. [ ]
• Add generalised support for ignoring returncodes [ ] and move special-casing of returncodes in zip to use Command.VALID_RETURNCODES. [ ]

Other tools disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues. This month, Vagrant Cascadian updated the Vcs-Git to specify the debian packaging branch. [ ] reprotest is our end-user tool to build same source code twice in widely differing environments and then checks the binaries produced by each build for any differences. This month, versions 0.7.13 and 0.7.14 were uploaded to Debian unstable by Holger Levsen after Vagrant Cascadian added support for GNU Guix [ ].

Project documentation & website There was more work performed on our documentation and website this month. Bernhard M. Wiedemann added a Java Gradle Build Tool snippet to the SOURCE_DATE_EPOCH documentation [ ] and normalised various terms to unreproducible [ ]. Chris Lamb added a Meson.build example [ ] and improved the documentation for the CMake [ ] to the SOURCE_DATE_EPOCH documentation, replaced anyone can with anyone may as, well, not everyone has the resources, skills, time or funding to actually do what it refers to [ ] and improved the pre-processing for our report generation [ ][ ][ ][ ] etc. In addition, Holger Levsen updated our news page to improve the list of reports [ ], added an explicit mention of the weekly news time span [ ] and reverted sorting of news entries to have latest on top [ ] and Mattia Rizzolo added Codethink as a non-fiscal sponsor [ ] and lastly Tianon Gravi added a Docker Images link underneath the Debian project on our Projects page [ ].

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann (for the openSUSE distribution):
  • rdiff-backup fixed, build date in Python manpage
  • click-man fixed, toolchain, build date in Python manpage
  • rpm report filesystem-order related nondeterminism
  • python-enaml report nondeterministic .enamlc compiler output
  • k9s date
  • python-xcffib fix a test with a race condition
• Chris Lamb:
  • #943956 re-opened against snakemake.
  • #950630 filed against designate.
  • #950936 filed against pynwb.
  • #950942 filed against python-oslo.reports.
  • #951357 filed against mate-desktop (forwarded upstream).
  • #951573 filed against javatools.
  • #952493 filed against xavs2.
  • #952694 filed against snapd-glib (forwarded upstream).
  • #952762 filed against openstack-pkg-tools.
  • ccextractor issues with MacOSX BSD date not understanding GNU date options
  • python-django (Always build the documentation in English)
  • python_example (sort glob / readdir)

Vagrant Cascadian submitted patches via the Debian bug tracking system targeting the packages the Civil Infrastructure Platform has identified via the CIP and CIP build depends package sets:

• #950410 filed against autogen.
• #950417 & 950416 & 950415 filed against autoconf.
• #950419 filed against m4.
• #950444 filed against intltool.
• #950585 filed against binutils-dev.
• #950603 filed against yodl-doc.
• #950606 filed against gdb-source.
• #950704 filed against automake-1.15.
• #951031 filed against libtool.

Testing framework We operate a fully-featured and comprehensive Jenkins-based testing framework that powers tests.reproducible-builds.org. This month, the following changes were made by Holger Levsen:

• Debian:
  • Configured an instance of David Bremner s builtin-pho database of .buildinfo files. This has resulted in so that we now know that Debian bullseye contains 4,557 source packages for the amd64 architecture without corresponding .buildinfo files and 25,668 source packages with .buildinfo files. [ ][ ][ ][ ][ ]
  • Forward mails addressed to buildinfo@ to the root user. [ ]
  • Temporarily revert using backports [ ][ ] and use devscripts from buster-backports [ ].
  • Update URL for PureOS package set. [ ]
  • Deprioritise the scheduling of older old packages in bullseye and unstable. [ ][ ]
  • Treat the bullseye distribution like unstable when scheduling the i386 architecture, to allow the latter to catch up a bit. [ ]
• Misc/other:
  • Disable the last active Alpine builder too for now. [ ]
  • Improve generated HTML output. [ ][ ]
  • Ignore Fedora jobs. [ ]
  • Include links to failed and unstable jobs in HTML output. [ ]
  • Start weighting job importance and five a lot more weight to nodes acting as a proxy for others. [ ][ ][ ]
  • Add new job to calculate the overall system health for tests.reproducible-builds.org for usage with Jelle van der Waa s Reproducible Builds status display. [ ]

In addition, Mattia Rizzolo added an Apache web server redirect for buildinfos.debian.net [ ] and reverted the reshuffling of arm64 architecture builders [ ]. The usual build node maintenance was performed by Holger Levsen, Mattia Rizzolo [ ][ ] and Vagrant Cascadian.

Getting in touch If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Twitter: @ReproBuilds
• Reddit: /r/ReproducibleBuilds
• Mailing list: rb-general@lists.reproducible-builds.org

---

This month s report was written by Bernhard M. Wiedemann, Chris Lamb and Holger Levsen. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.

These steps should be fairly similar to what s found in upstream s Install Docker on Debian document, but do differ slightly in a few minor ways.

grab Docker s APT repo GPG key

# "Docker Release (CE deb)"
export GNUPGHOME="$(mktemp -d)"
gpg --keyserver ha.pool.sks-keyservers.net --recv-keys 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
# stretch+
gpg --export --armor 9DC858229FC7DD38854AE2D88D81803C0EBFCD88   sudo tee /etc/apt/trusted.gpg.d/docker.gpg.asc
# jessie
# gpg --export 9DC858229FC7DD38854AE2D88D81803C0EBFCD88   sudo tee /etc/apt/trusted.gpg.d/docker.gpg > /dev/null
rm -rf "$GNUPGHOME"

Verify:

$ apt-key list
...
/etc/apt/trusted.gpg.d/docker.gpg.asc
-------------------------------------
pub   rsa4096 2017-02-22 [SCEA]
      9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid           [ unknown] Docker Release (CE deb) <docker@docker.com>
sub   rsa4096 2017-02-22 [S]
...

add Docker s APT source With the switch to download.docker.com, HTTPS is now mandated:

$ apt-get update && apt-get install apt-transport-https

Setup sources.list:

echo "deb [arch=amd64] https://download.docker.com/linux/debian stretch stable"   sudo tee /etc/apt/sources.list.d/docker.list

Add edge component for every-month releases and test for release candidates (ie, ... stretch stable edge). Replace stretch with jessie for Jessie installs. At this point, you should be safe to run apt-get update to verify the changes:

$ sudo apt-get update
...
Get:5 https://download.docker.com/linux/debian stretch/stable amd64 Packages [1227 B]
...
Reading package lists... Done

(There shouldn t be any warnings or errors about missing keys, etc.)

configure Docker

This step could be done after Docker s installed (and indeed, that s usually when I do it because I forget that I should until I ve got Docker installed and realize that my configuration is suboptimal), but doing it before ensures that Docker doesn t have to be restarted later.

sudo mkdir -p /etc/docker
sudo sensible-editor /etc/docker/daemon.json

(sensible-editor can be replaced by whatever editor you prefer, but that command should choose or prompt for a reasonable default)

I then fill daemon.json with at least a default storage-driver. Whether I use aufs or overlay2 depends on my kernel version and available modules if I m on Ubuntu, AUFS is still a no-brainer (since it s included in the default kernel if the linux-image-extra-XXX/linux-image-extra-virtual package is installed), but on Debian AUFS is only available in either 3.x kernels (jessie s default non-backports kernel) or recently in the aufs-dkms package (as of this writing, still only available on stretch and sid no jessie-backports option).

If my kernel is 4.x+, I m likely going to choose overlay2 (or if that errors out, the older overlay driver).

Choosing an appropriate storage driver is a fairly complex topic, and I d recommend that for serious production deployments, more research on pros and cons is performed than I m including here (especially since AUFS and OverlayFS are not the only options they re just the two I personally use most often).

 
	"storage-driver": "overlay2"
 

configure boot parameters

I usually set a few boot parameters as well (in /etc/default/grub s GRUB_CMDLINE_LINUX_DEFAULT option run sudo update-grub after adding these, space-separated).

• cgroup_enable=memory enable memory accounting for containers (allows docker run --memory for setting hard memory limits on containers)
• swapaccount=1 enable swap accounting for containers (allows docker run --memory-swap for setting hard swap memory limits on containers)
• systemd.legacy_systemd_cgroup_controller=yes newer versions of systemd may disable the legacy cgroup interfaces Docker currently uses; this instructs systemd to keep those enabled (for more details, see systemd/systemd#4628, opencontainers/runc#1175, docker/docker#28109)
• vsyscall=emulate allow older binaries to run (debian:wheezy, etc.; see docker/docker#28705)

All together:

...
GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1 systemd.legacy_systemd_cgroup_controller=yes vsyscall=emulate"
...

install Docker! Finally, the time has come.

$ sudo apt-get install -V docker-ce
...
   docker-ce (17.03.1~ce-0~debian-stretch)
...
$ sudo docker version
Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:07:28 2017
 OS/Arch:      linux/amd64
Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:07:28 2017
 OS/Arch:      linux/amd64
 Experimental: false
$ sudo usermod -aG docker "$(id -un)"

grab Docker s APT repo GPG key The way I do this is probably a bit unconventional, but the basic gist is something like this:

export GNUPGHOME="$(mktemp -d)"
gpg --keyserver ha.pool.sks-keyservers.net --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
gpg --export --armor 58118E89F3A912897C070ADBF76221572C52609D   sudo tee /etc/apt/trusted.gpg.d/docker.gpg.asc
rm -rf "$GNUPGHOME"

(On jessie or another release whose APT doesn t support .asc files in /etc/apt/trusted.gpg.d, I d drop --armor and the .asc and go with simply /.../docker.gpg.) This creates me a new GnuPG directory to work with (so my personal ~/.gnupg doesn t get cluttered with this new key), downloads Docker s signing key from the keyserver gossip network (verifying the fetched key via the full fingerprint I ve provided), exports the key into APT s keystore, then cleans up the leftovers. For completeness, other popular ways to fetch this include:

sudo apt-key adv --keyserver ha.pool.sks-keyservers.net --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

(worth noting that man apt-key discourages the use of apt-key adv)

wget -qO- 'https://apt.dockerproject.org/gpg'   sudo apt-key add -

(no verification of the downloaded key) Here s the relevant output of apt-key list on a machine where I ve got this key added in the way I outlined above:

$ apt-key list
...
/etc/apt/trusted.gpg.d/docker.gpg.asc
-------------------------------------
pub   rsa4096 2015-07-14 [SCEA]
      5811 8E89 F3A9 1289 7C07  0ADB F762 2157 2C52 609D
uid           [ unknown] Docker Release Tool (releasedocker) <docker@docker.com>
...

add Docker s APT source If you prefer to fetch sources via HTTPS, install apt-transport-https, but I m personally fine with simply doing GPG verification of fetched packages, so I forgo that in favor of less packages installed. YMMV.

echo 'deb http://apt.dockerproject.org/repo debian-stretch main'   sudo tee /etc/apt/sources.list.d/docker.list

Hopefully it s obvious, but debian-stretch in that line should be replaced by debian-jessie, ubuntu-xenial, etc. as desired. It s also worth pointing out that this will not include Docker s release candidates. If you want those as well, add testing after main, ie ... debian-stretch main testing' .... At this point, you should be safe to run apt-get update to verify the changes:

$ sudo apt-get update
...
Hit:1 http://apt.dockerproject.org/repo debian-stretch InRelease
...
Reading package lists... Done

(There shouldn t be any warnings or errors about missing keys, etc.)

configure Docker This step could be done after Docker s installed (and indeed, that s usually when I do it because I forget that I should until I ve got Docker installed and realize that my configuration is suboptimal), but doing it before ensures that Docker doesn t have to be restarted later.

sudo mkdir -p /etc/docker
sudo sensible-editor /etc/docker/daemon.json

(sensible-editor can be replaced by whatever editor you prefer, but that command should choose or prompt for a reasonable default) I then fill daemon.json with at least a default storage-driver. Whether I use aufs or overlay2 depends on my kernel version and available modules if I m on Ubuntu, AUFS is still a no-brainer (since it s included in the default kernel if the linux-image-extra-XXX/linux-image-extra-virtual package is installed), but on Debian AUFS is only available in either 3.x kernels (jessie s default non-backports kernel) or recently in the aufs-dkms package (as of this writing, still only available on stretch and sid no jessie-backports option). If my kernel is 4.x+, I m likely going to choose overlay2 (or if that errors out, the older overlay driver). Choosing an appropriate storage driver is a fairly complex topic, and I d recommend that for serious production deployments, more research on pros and cons is performed than I m including here (especially since AUFS and OverlayFS are not the only options they re just the two I personally use most often).

 
	"storage-driver": "overlay2"
 

configure boot parameters I usually set a few boot parameters as well (in /etc/default/grub s GRUB_CMDLINE_LINUX_DEFAULT option run sudo update-grub after adding these, space-separated).

• cgroup_enable=memory enable memory accounting for containers (allows docker run --memory for setting hard memory limits on containers)
• swapaccount=1 enable swap accounting for containers (allows docker run --memory-swap for setting hard swap memory limits on containers)
• systemd.legacy_systemd_cgroup_controller=yes newer versions of systemd may disable the legacy cgroup interfaces Docker currently uses; this instructs systemd to keep those enabled (for more details, see systemd/systemd#4628, opencontainers/runc#1175, docker/docker#28109)
• vsyscall=emulate allow older binaries to run (debian:wheezy, etc.; see docker/docker#28705)

All together:

...
GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1 systemd.legacy_systemd_cgroup_controller=yes vsyscall=emulate"
...

install Docker! Finally, the time has come.

$ sudo apt-get install -V docker-engine
...
$ sudo docker version
Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 21:45:16 2016
 OS/Arch:      linux/amd64
Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 21:45:16 2016
 OS/Arch:      linux/amd64
$ sudo usermod -aG docker "$(id -un)"

(Reboot or logout/login to update your session to include docker group membership and thus no longer require sudo for using docker commands.) Hope this is useful to someone! If nothing else, it ll serve as a concise single-page reference for future-tianon.

• Default IP: 2001:db8::5400:00ff:fe20:2295
• Network: 2001:db8::
• CIDR: 64

# /etc/systemd/system/docker.service.d/ipv6.conf
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --ipv6 --fixed-cidr-v6 2001:db8::/80

# /etc/ndppd.conf
proxy ens3  
	rule 2001:db8::/80  
	 
 

• Gianfranco Costamagna (locutusofborg)
• Graham Inggs (ginggs)
• Ximin Luo (infinity0)
• Christian Kastner (ckk)
• Tianon Gravi (tianon)
• Iain R. Learmonth (irl)
• Laura Arjona Reina (larjona)

• Senthil Kumaran
• Riley Baird
• Robie Basak
• Alex Muntada
• Johan Van de Wauw
• Benjamin Barenblat
• Paul Novotny
• Jose Luis Rivero
• Chris Knadle
• Lennart Weller

• gregor herrmann uploaded libextutils-depends-perl/0.404-1 which makes its output deterministic.
• Christian Hofstaedtler uploaded yard/0.8.7.4-2 which will not write timestamps in the generated documentation. Original patch by Chris Lamb, does not write timestamps in the generated documentation anymore.
• Emmanuel Bourg uploaded maven-plugin-tools/3.3-2 which removes the date from the plugin descriptor. Patch by Reiner Herrmann.
• Emmanuel Bourg uploaded maven-archiver/2.6-1 which now uses the date set in the DEB_CHANGELOG_DATETIME environment variable for the timestamp in the pom.properties file embedded in the jar files. Original patch by Chris West.
• Nicolas Boulenguez uploaded dh-ada-library/6.4 which will warn against non deterministic ALI for sources newer than changelog.

• cwirc/2.0.0-8 uploaded by Colin Tuckley, original patch by Reiner Herrmann.
• darkplaces/0~20140513+svn12208-1 by Simon McVittie.
• exactimage/0.9.1-4 by Sven Eckelmann.
• gnupg/1.4.19-1 by Daniel Kahn Gillmor.
• httpunit/1.7+dfsg-11 by Emmanuel Bourg.
• hy/0.10.1-2 uploaded by Tianon Gravi, original patch by Reiner Herrmann.
• ioquake3/1.36+u20150412+dfsg1-2 by Simon McVittie, original patch by Reiner Herrmann.
• kiwi/1.9.22-3 by Jelmer Vernooij.
• lava-server/2015.05-1 uploaded by Neil Williams, original patch by Reiner Herrmann.
• libelixirfm-perl/1.1.976-4 uploaded by gregor herrmann, original patch by Chris Lamb.
• littler/0.2.3-2 by Dirk Eddelbuettel.
• mednafen/0.9.38.1-1 by Stephen Kitt.
• nftables/0.4-4 by Arturo Borrero Gonzalez.
• ntdb/1.0-7 by Jelmer Vernooij.
• onioncat/0.2.2+svn566-1 by intrigeri.
• openarena/0.8.8-13 by Simon McVittie.
• openarena-085-data/0.8.5split-6 by Simon McVittie.
• openarena-088-data/0.8.8-3 by Simon McVittie.
• openarena-data/0.8.5split-6 by Simon McVittie.
• openarena-maps/0.8.5split-6 by Simon McVittie.
• openarena-players/0.8.5split-6 by Simon McVittie.
• openarena-players-mature/0.8.5split-6 by Simon McVittie.
• openarena-textures/0.8.5split-6 by Simon McVittie.
• pybik/2.0-1 by B. Clausius.
• python-xmp-toolkit/2.0.1+git20140309.5437b0a-1 by Daniel Stender.
• quakespasm/0.90.0-3 by Stephen Kitt.
• traceroute/1:2.0.21-1 uploaded by Laszlo Boszormenyi, original patch by Lunar.
• unar/1.8.1-4 uploaded by Matt Kraai, original patch by Lunar.
• websvn/2.3.3-1.3 uploaded by Thijs Kinkhorst, original patch by Chris Lamb.
• xd/3.23.01-2 uploaded by Frank B. Brokken, original patch by Chris Lamb.

• ada-reference-manual/1:2012.2-5 by Nicolas Boulenguez.
• apparmor/2.9.2-2 by intrigeri.
• argyll/1.7.0+repack-1 by J rg Frings-F rst.
• lava-dispatcher/2015.05-1 by Neil Williams.
• libaunit/3.7.1-2 by Nicolas Boulenguez.
• libflorist/2014-2 by Nicolas Boulenguez.
• mailcrypt/3.5.9-8 uploaded by Barak A. Pearlmutter, original patch by Chris Lamb.
• openchange/1:2.2-7 by Jelmer Vernooij.
• sane-backends/1.0.24-11 by J rg Frings-F rst timestamps in .dvi and .ps
• tomcat6/6.0.41-4 by Emmanuel Bourg.
• tomcat7/7.0.61-1 by Emmanuel Bourg; currently FTBFS.
• tomcat8/8.0.22-2 by Emmanuel Bourg.

• #784541 on yasm by Lunar: remove build date from version strings.
• #784694 on smcroute by Micha Lenk: remove build date from version string.
• #784672 on gnumeric by Daniel Kahn Gillmor: remove timestamps in embedded gzip'ed data in shared library.
• #774347 on sed by Lunar: fix permissions before creating the package.
• #784352 on icebreaker by Reiner Herrmann: use UTC timezone when calculating version date.
• #784325 on kde-workspace by Lunar: make the output of kdm confproc.pl stable.
• #784602 on monkeysign by Daniel Kahn Gillmor: use time of debian/changelog entry when generating documentation.
• #784723 on alot by Juan Picca: pass time of debian/changelog entry to Sphinx.
• #784538 on file-rc by Lunar: use sed instead of grep+mv to keep correct file permissions.
• #784335 on libapache2-mod-perl2 by Lunar: set PERL_HASH_SEED=0 during configure to make the generated .c and .h files stable.
• #784267 on mpv by Lunar: pass --disable-build-date to ./configure.
• #784793 on bugs-everywhere by Daniel Kahn Gillmor: use time of debian/changelog entry as build date.
• #784318 on gnome-desktop3 by Lunar: use time of debian/chanelog entry as build date.
• #774504 on debianutils by Lunar: fix file permissions.

FROM scratch
ADD rootfs.tar.xz /
CMD ["/bin/bash"]

mkimage.sh -t tianon/debian:wheezy -d . debootstrap --variant=minbase --components=main --include=inetutils-ping,iproute wheezy http://http.debian.net/debian

If the expectation is that an experienced Unix person who found it missing would say "What on earth is going on, where is 'foo'?", it must be an 'important' package. -- Debian Policy

DPkg::Post-Invoke   "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin   true";  ;
APT::Update::Post-Invoke   "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin   true";  ;
Dir::Cache::pkgcache "";
Dir::Cache::srcpkgcache "";
Acquire::Languages "none";

Linux viper 3.12.20-gentoo #1 SMP Sun May 18 12:36:24 MDT 2014 x86_64

Binary files pure-debootstrap/etc/apt/trustdb.gpg and from-docker/etc/apt/trustdb.gpg differ

<joeyh> I'll bet I could publish an image that just did a killall5 as root on startup and get plenty of people to nuke their container hosts

• Make a package that can build docker images of Debian, in a fully reproducible fashion. Ie, same versions of debs in, same byte stream out.
• If it makes sense for the docker image to not contain all the packages in a standard debootstrap (maybe leaving out init systems), or to contain other packages, write down the rationalle for this, and make a --variant=docker.
• Make a package that provides appropriate tweaks for Debian in a container. This might include a policy-rc.d that allows restarting daemons on upgrade if they're already running in the container, and otherwise prevents running daemons.
• Make a low-disk-space package that eg, prevents apt from caching debs.
• Provide some way to verify, through gpg signatures, that docker has pulled an actual trusted image and not some https-MITMed thing. (See also #746394