Breathalyzers In the United States, Latinx folks are less likely to drink than white people and, overall, less likely to be arrested for DUIs3,4. However, they are more likely to be stopped by police while driving5,6. Who is being stopped by police is up to the police and they pull over a disproportionate number of Latinx drivers. After someone is pulled over for suspected drunk driving, they are given a breathalyzer test. Breathalyzers are so easy to (un)intentionally mis-calibrate that they have been banned as valid evidence in multiple states. The biases of the police are not canceled out by the technology that should, in theory, let us know whether someone is actually drunk.

Facial Recognition I could talk about for quite some time and, in fact, have. So have others. Google s image recognition software recognized black people as gorillas and to fix the issue it removed gorillas from it s image-labeling technology. Facial recognition software does a bad job at recognizing black people. In fact, it s also terrible at identifying indigenous people and other people of color. (Incidentally, it s also not great at recognizing women, but let s not talk about that right now.) As we use facial recognition technology for more things, from automated store checkouts (even more relevant in the socially distanced age of Covid-19), airport ticketing, phone unlocking, police identification, and a number of other things, it becomes a bigger problem that this software cannot tell the difference between two Asian people.

Targeted Advertising Black kids see 70% more online ads for food than white kids, and twice as many ads for junk food. In general BIPOC youth are more likely to see junk food advertisements online. This is intentional, and happens after they are identified as BIPOC youth.

Technology Reinforces Racism; Racism Builds Technology The technology we have developed reinforces racism on a society wide scale because it makes it harder for BIPOC people to interact with this world that is run by computers and software. It s harder to not be racist when the technology around us is being used to perpetuate racist paradigms. For example, if a store implements facial recognition software for checkout, black women are less likely to be identified. They are then more likely to be targeted as trying to steal from the store. We are more likely to take this to mean that black women are more likely to steal. This is how technology builds racism, People are being excluded largely because they are not building these technologies, because they are not welcome in our spaces. There simply are not enough Black and Hispanic technologists and that is a problem. We need to care about this because when software doesn t work for everyone, it doesn t work. We cannot build on the promise of free and open source software when we are excluding the majority of people.

int socket = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);

int socket = socket(PF_INET, SOCK_STREAM, IPPROTO_TLS);

How does Smokeping work Leaving those two aside for now, I'm left with Smokeping, which I used in my previous job to diagnose routing issues, using Smokeping as a decentralized looking glass, which was handy to debug long term issues. Smokeping is a strange animal: it's fundamentally similar to Munin, except it's harder to write plugins for it, so most people just use it for Ping, something for which it excels at. Its trick is this: instead of doing a single ping and returning this metrics, it does multiple ones and returns multiple metrics. Specifically, smokeping will send multiple ICMP packets (20 by default), with a low interval (500ms by default) and a single retry. It also pings multiple hosts at once which means it can quickly scan multiple hosts simultaneously. You therefore see network conditions affecting one host reflected in further hosts down (or up) the chain. The multiple metrics also mean you can draw graphs with "error bars" which Smokeping shows as "smoke" (hence the name). You also get per-metric packet loss. Basically, smokeping runs this command and collects the output in a RRD database:

fping -c $count -q -b $backoff -r $retry -4 -b $packetsize -t $timeout -i $mininterval -p $hostinterval $host [ $host ...]

... where those parameters are, by default:

• $count is 20 (packets)
• $backoff is 1 (avoid exponential backoff)
• $timeout is 1.5s
• $mininterval is 0.01s (minimum wait interval between any target)
• $hostinterval is 1.5s (minimum wait between probes on a single target)

It can also override stuff like the source address and TOS fields. This probe will complete between 30 and 60 seconds, if my math is right (0% and 100% packet loss).

How do draw Smokeping graphs in Grafana A naive implementation of Smokeping in Prometheus/Grafana would be to use the blackbox exporter and create a dashboard displaying those metrics. I've done this at home, and then I realized that I was missing something. Here's what I did.

1. install the blackbox exporter:

   apt install prometheus-blackbox-exporter
   

2. make sure to allow capabilities so it can ping:

   dpkg-reconfigure prometheus-blackbox-exporter
   

3. hook monitoring targets into prometheus.yml (the default blackbox exporter configuration is fine):

   scrape_configs:
     - job_name: blackbox
         metrics_path: /probe
         params:
           module: [icmp]
         scrape_interval: 5s
         static_configs:
           - targets:
             - octavia.anarc.at
             # hardcoded in DNS
             - nexthop.anarc.at
             - koumbit.net
             - dns.google
         relabel_configs:
           - source_labels: [__address__]
             target_label: __param_target
           - source_labels: [__param_target]
             target_label: instance
           - target_label: __address__
             replacement: 127.0.0.1:9115  # The blackbox exporter's real hostname:port.
   

   Notice how we lower the scrape_interval to 5 seconds to get more samples. nexthop.anarc.at was added into DNS to avoid hardcoding my upstream ISP's IP in my configuration.
4. use this Grafana panel to graph the results. It was created with this query:

   sum(probe_icmp_duration_seconds phase="rtt" ) by (instance)
   

   • Set the Legend field to instance RTT
   • Set Draw modes to lines and Mode options to staircase
   • Set the Left Y axis Unit to duration(s)
   • Show the Legend As table, with Min, Avg, Max and Current enabled
   Then this query, for packet loss:

   1-avg_over_time(probe_success[$__interval])!=0 or null
   

   • Set the Legend field to instance packet loss
   • Set a Add series override to Lines: false, Null point mode: null, Points: true, Points Radius: 1, Color: deep red, and, most importantly, Y-axis: 2
   • Set the Right Y axis Unit to percent (0.0-1.0) and set Y-max to 1
   Then set the entire thing to Repeat, on target, vertically. And you need to add a target variable like label_values(probe_success, instance).

The result looks something like this:

Not bad, but not Smokeping

This actually looks pretty good! The resulting dashboard is available in the Grafana dashboard repository.

What is missing? Now, that doesn't exactly look like Smokeping, does it. It's pretty good, but it's not quite what we want. What is missing is variance, the "smoke" in Smokeping. There's a good article about replacing Smokeping with Grafana. They wrote a custom script to write samples into InfluxDB so unfortunately we can't use it in this case, since we don't have InfluxDB's query language. I couldn't quite figure out how to do the same in PromQL. I tried:

stddev(probe_icmp_duration_seconds phase="rtt",instance=~"$instance" )
stddev_over_time(probe_icmp_duration_seconds phase="rtt",instance=~"$instance" [$__interval])
stddev_over_time(probe_icmp_duration_seconds phase="rtt",instance=~"$instance" [1m])

The first two give zero for all samples. The latter works, but doesn't look as good as Smokeping. So there might be something I'm missing. SuperQ wrote a special exporter for this called smokeping_prober that came out of this discussion in the blackbox exporter. Instead of delegating scheduling and target definition to Prometheus, the targets are set in the exporter. They also take a different approach than Smokeping: instead of recording the individual variations, they delegate that to Prometheus, through the use of "buckets". Then they use a query like this:

histogram_quantile(0.9 rate(smokeping_response_duration_seconds_bucket[$__interval]))

This is the rationale to SuperQ's implementation:

Yes, I know about smokeping's bursts of pings. IMO, smokeping's data model is flawed that way. This is where I intentionally deviated from the smokeping exact way of doing things. This prober sends a smooth, regular series of packets in order to be measuring at regular controlled intervals. Instead of 20 packets, over 10 seconds, every minute. You send one packet per second and scrape every 15. This has the same overall effect, but the measurement is, IMO, more accurate, as it's a continuous stream. There's no 50 second gap of no metrics about the ICMP stream. Also, you don't get back one metric for those 20 packets, you get several. Min, Max, Avg, StdDev. With the histogram data, you can calculate much more than just that using the raw data. For example, IMO, avg and max are not all that useful for continuous stream monitoring. What I really want to know is the 90th percentile or 99th percentile. This smokeping prober is not intended to be a one-to-one replacement for exactly smokeping's real implementation. But simply provide similar functionality, using the power of Prometheus and PromQL to make it better. [...] one of the reason I prefer the histogram datatype, is you can use the heatmap panel type in Grafana, which is superior to the individual min/max/avg/stddev metrics that come from smokeping. Say you had two routes, one slow and one fast. And some pings are sent over one and not the other. Rather than see a wide min/max equaling a wide stddev, the heatmap would show a "line" for both routes.

That's an interesting point. I have also ended up adding a heatmap graph to my dashboard, independently. And it is true it shows those "lines" much better... So maybe that, if we ignore legacy, we're actually happy with what we get, even with the plain blackbox exporter. So yes, we're missing pretty "fuzz" lines around the main lines, but maybe that's alright. It would be possible to do the equivalent to the InfluxDB hack, with queries like:

min_over_time(probe_icmp_duration_seconds phase="rtt",instance=~"$instance" [30s])
avg_over_time(probe_icmp_duration_seconds phase="rtt",instance=~"$instance" [5m])
max_over_time(probe_icmp_duration_seconds phase="rtt",instance=~"$instance" [30s])

The output looks something like this:

Looks more like Smokeping!

But there's a problem there: see how the middle graph "dips" sometimes below 20ms? That's the min_over_time function (incorrectly, IMHO) returning zero. I haven't quite figured out how to fix that, and I'm not sure it is better. But it does look more like Smokeping than the previous graph. Update: I forgot to mention one big thing that this setup is missing. Smokeping has this nice feature that you can order and group probe targets in a "folder"-like hierarchy. It is often used to group probes by location, which makes it easier to scan a lot of targets. This is harder to do in this setup. It might be possible to setup location-specific "jobs" and select based on that, but it's not exactly the same.

Credits Credits to Chris Siebenmann for his article about Prometheus and pings which gave me the avg_over_time query idea.

Welcome to the May 2020 report from the Reproducible Builds project. One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. Nonetheless, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into seemingly secure software during the various compilation and distribution processes. In these reports we outline the most important things that we and the rest of the community have been up to over the past month.

News The Corona-Warn app that helps trace infection chains of SARS-CoV-2/COVID-19 in Germany had a feature request filed against it that it build reproducibly. A number of academics from Cornell University have published a paper titled Backstabber s Knife Collection which reviews various open source software supply chain attacks:

Recent years saw a number of supply chain attacks that leverage the increasing use of open source during software development, which is facilitated by dependency managers that automatically resolve, download and install hundreds of open source packages throughout the software life cycle.

In related news, the LineageOS Android distribution announced that a hacker had access to the infrastructure of their servers after exploiting an unpatched vulnerability. Marcin Jachymiak of the Sia decentralised cloud storage platform posted on their blog that their siac and siad utilities can now be built reproducibly:

This means that anyone can recreate the same binaries produced from our official release process. Now anyone can verify that the release binaries were created using the source code we say they were created from. No single person or computer needs to be trusted when producing the binaries now, which greatly reduces the attack surface for Sia users.

Synchronicity is a distributed build system for Rust build artifacts which have been published to crates.io. The goal of Synchronicity is to provide a distributed binary transparency system which is independent of any central operator. The Comparison of Linux distributions article on Wikipedia now features a Reproducible Builds column indicating whether distributions approach and progress towards achieving reproducible builds.

Distribution work In Debian this month:

• Paul Wise continued a discussion that was started in February regarding the storing and distribution of build logs and other related artifacts and their relationship to reproducible builds. For example, the binutils package ships its own, unreproducible, log files in its binary packages. It was followed-up by replies from Chris Lamb and Matthias Klose.
• 34 reviews of Debian packages were added, 20 were updated and 122 were removed this month adding to our knowledge about identified issues. Chris Lamb added and categorised a new ocaml_cmti_files toolchain issue.

In Alpine Linux, an issue was filed and closed regarding the reproducibility of .apk packages. Allan McRae of the ArchLinux project posted their third Reproducible builds progress report to the arch-dev-public mailing list which includes the following call for help:

We also need help to investigate and fix the packages that fail to reproduce that we have not investigated as of yet.

In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update.

Software development

diffoscope Chris Lamb made the changes listed below to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. He also prepared and uploaded versions 142, 143, 144, 145 and 146 to Debian, PyPI, etc.

• Comparison improvements:
  • Improve fuzzy matching of JSON files as file now supports recognising JSON data. (#106)
  • Refactor .changes and .buildinfo handling to show all details (including the GnuPG header and footer components) even when referenced files are not present. (#122)
  • Use our BuildinfoFile comparator (etc.) regardless of whether the associated files (such as the orig.tar.gz and the .deb) are present. [ ]
  • Include GnuPG signature data when comparing .buildinfo, .changes, etc. [ ]
  • Add support for printing Android APK signatures via apksigner(1). (#121)
  • Identify iOS App Zip archive data as .zip files. (#116)
  • Add support for Apple Xcode .mobilepovision files. (#113)
• Bug fixes:
  • Don t print a traceback if we pass a single, missing argument to diffoscope (eg. a JSON diff to re-load). [ ]
  • Correct differences typo in the ApkFile handler. (#127)
• Output improvements:
  • Never emit the same id="foo" anchor reference twice in the HTML output, otherwise identically-named parts will not be able to linked to via a #foo anchor. (#120)
  • Never emit an empty id anchor either; it is not possible to link to #. [ ]
  • Don t pretty-print the output when using the --json presenter; it will usually be too complicated to be readable by the human anyway. [ ]
  • Use the SHA256 over MD5 hash when generating page names for the HTML directory-style presenter. (#124)
• Reporting improvements:
  • Clarify the message when we truncate the number of lines to standard error [ ] and reduce the number of maximum lines printed to 25 as usually the error is obvious by then [ ].
  • Print the amount of free space that we have available in our temporary directory as a debugging message. [ ]
  • Clarify Command [ ] failed with exit code messages to remove duplicate exited with exit but also to note that diffoscope is interpreting this as an error. [ ]
  • Don t leak the full path of the temporary directory in Command [ ] exited with 1 messages. (#126)
  • Clarify the warning message when we cannot import the debian Python module. [ ]
  • Don t repeat stderr from if both commands emit the same output. [ ]
  • Clarify that an external command emits for both files, otherwise it can look like we are repeating itself when, in reality, it is being run twice. [ ]
• Testsuite improvements:
  • Prevent apksigner test failures due to lack of binfmt_misc, eg. on Salsa CI and elsewhere. [ ]
  • Drop .travis.yml as we use Salsa instead. [ ]
• Dockerfile improvements:
  • Add a .dockerignore file to whitelist files we actually need in our container. (#105)
  • Use ARG instead of ENV when setting up the DEBIAN_FRONTEND environment variable at runtime. (#103)
  • Run as a non-root user in container. (#102)
  • Install/remove the build-essential during build so we can install the recommended packages from Git. [ ]
• Codebase improvements:
  • Bump the officially required version of Python from 3.5 to 3.6. (#117)
  • Drop the (default) shell=False keyword argument to subprocess.Popen so that the potentially-unsafe shell=True is more obvious. [ ]
  • Perform string normalisation in Black [ ] and include the Black output in the assertion failure too [ ].
  • Inline MissingFile s special handling of deb822 to prevent leaking through abstract layers. [ ][ ]
  • Allow a bare try/except block when cleaning up temporary files with respect to the flake8 quality assurance tool. [ ]
  • Rename in_dsc_path to dsc_in_same_dir to clarify the use of this variable. [ ]
  • Abstract out the duplicated parts of the debian_fallback class [ ] and add descriptions for the file types. [ ]
  • Various commenting and internal documentation improvements. [ ][ ]
  • Rename the Openssl command class to OpenSSLPKCS7 to accommodate other command names with this prefix. [ ]
• Misc:
  • Rename the --debugger command-line argument to --pdb. [ ]
  • Normalise filesystem stat(2) birth times (ie. st_birthtime) in the same way we do with the stat(1) command s Access: and Change: times to fix a nondeterministic build failure in GNU Guix. (#74)
  • Ignore case when ordering our file format descriptions. [ ]
  • Drop, add and tidy various module imports. [ ][ ][ ][ ]

In addition:

• Jean-Romain Garnier fixed a general issue where, for example, LibarchiveMember s has_same_content method was called regardless of the underlying type of file. [ ]
• Daniel Fullmer fixed an issue where some filesystems could only be mounted read-only. (!49)
• Emanuel Bronshtein provided a patch to prevent a build of the Docker image containing parts of the build s. (#123)
• Mattia Rizzolo added an entry to debian/py3dist-overrides to ensure the rpm-python module is used in package dependencies (#89) and moved to using the new execute_after_* and execute_before_* Debhelper rules [ ].

Chris Lamb also performed a huge overhaul of diffoscope s website:

• Add a completely new design. [ ][ ]
• Dynamically generate our contributor list [ ] and supported file formats [ ] from the main Git repository.
• Add a separate, canonical page for every new release. [ ][ ][ ]
• Generate a latest release section and display that with the corresponding date on the homepage. [ ]
• Add an RSS feed of our releases [ ][ ][ ][ ][ ] and add to Planet Debian [ ].
• Use Jekyll s absolute_url and relative_url where possible [ ][ ] and move a number of configuration variables to _config.yml [ ][ ].

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:
  • golang-packaging (toolchain issue, affecting times in minikube)
  • jboss-logging-tools (toolchain issue, affecting date for resteasy)
  • linux_logo (sort find output to avoid inheriting filesystem order)
  • moonjit (generate reproducible output by default if SOURCE_DATE_EPOCH is set)
  • vala (report ASLR nondeterminism)
• Jelle van der Waa:
  • earlyoom (timestamps in Gzip files)
  • fmt (Don t install sphinx-build cached files as they are unneeded & unreproducible)
  • nvidia-settings (timestamp in Gzip files)
• Chris Lamb:
  • #959714 filed against ataqv.
  • #960313 filed against elinks.
  • #960386 filed against briquolo.
  • #960388 filed against cryptominisat.
  • #960590 filed against wolfssl.
  • #960591 filed against mistral.
  • #960607 filed against python-watcherclient.
  • #960669 filed against tree-puzzle.
  • #961009 filed against nulib2.
  • #961202 filed against process-cpp.
  • #961494 filed against bowtie2.
  • #961495 filed against properties-cpp.
  • #961582 filed against wand (forwarded upstream)
  • #961657 filed against vows.
• Vagrant Cascadian:
  • #961747 filed against libstatgrab.
  • #961764 filed against texi2html.
  • #961766 filed against grub.
  • #961830 filed against systemtap.
  • #961942 filed against mono.
  • mescc-tools: Inherit CFLAGS in a Makefile, allowing -ffile-prefix-map/-fdebug-prefix-map to sanitise build paths (merged upstream).

Other tools Elsewhere in our tooling: strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. In May, Chris Lamb uploaded version 1.8.1-1 to Debian unstable and Bernhard M. Wiedemann fixed an off-by-one error when parsing PNG image modification times. (#16) In disorderfs, our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues, Chris Lamb replaced the term dirents in place of directory entries in human-readable output/log messages [ ] and used the astyle source code formatter with the default settings to the main disorderfs.cpp source file [ ]. Holger Levsen bumped the debhelper-compat level to 13 in disorderfs [ ] and reprotest [ ], and for the GNU Guix distribution Vagrant Cascadian updated the versions of disorderfs to version 0.5.10 [ ] and diffoscope to version 145 [ ].

Project documentation & website

• Carl Dong:
  • Clarify some potential confusion around GCC libtool. [ ]
• Chris Lamb:
  • Rename the Who page to Projects . [ ]
  • Ensure that Jekyll enters the _docs subdirectory to find the _docs/index.md file after an internal move. (#27)
  • Wrap ltmain.sh etc. in preformatted quotes. [ ]
  • Wrap the SOURCE_DATE_EPOCH Python examples onto more lines to prevent visual overflow on the page. [ ]
  • Correct a preferred spelling error. [ ]
• Holger Levsen:
  • Sort our Academic publications page by publication year [ ] and add Trusting Trust and Fully Countering Trusting Trust through Diverse Double-Compiling [ ].
• Juri Dispan:
  • Update the URL for faketime to the project s Github page. (!57)

Testing framework We operate a large and many-featured Jenkins-based testing framework that powers tests.reproducible-builds.org that, amongst many other tasks, tracks the status of our reproducibility efforts as well as identifies any regressions that have been introduced. Holger Levsen made the following changes:

• System health status:
  • Improve page description. [ ]
  • Add more weight to proxy failures. [ ]
  • More verbose debug/failure messages. [ ][ ][ ]
  • Work around strangeness in the Bash shell let VARIABLE=0 exits with an error. [ ]
• Debian:
  • Fail loudly if there are more than three .buildinfo files with the same name. [ ]
  • Fix a typo which prevented /usr merge variation on Debian unstable. [ ]
  • Temporarily ignore PHP s horde](https://www.horde.org/) packages in Debian bullseye. [ ]
  • Document how to reboot all nodes in parallel, working around molly-guard. [ ]
• Further work on a Debian package rebuilder:
  • Workaround and document various issues in the debrebuild script. [ ][ ][ ][ ]
  • Improve output in the case of errors. [ ][ ][ ][ ]
  • Improve documentation and future goals [ ][ ][ ][ ], in particular documentiing two real world tests case for an impossible to recreate build environment [ ].
  • Find the right source package to rebuild. [ ]
  • Increase the frequency we run the script. [ ][ ][ ][ ]
  • Improve downloading and selection of the sources to build. [ ][ ][ ]
  • Improve version string handling.. [ ]
  • Handle build failures better. [ ]. [ ]. [ ]
  • Also consider architecture all .buildinfo files. [ ][ ]

In addition:

• kpcyrd, for Alpine Linux, updated the alpine_schroot.sh script now that a patch for abuild had been released upstream. [ ]
• Alexander Couzens of the OpenWrt project renamed the brcm47xx target to bcm47xx. [ ]
• Mattia Rizzolo fixed the printing of the build environment during the second build [ ][ ][ ] and made a number of improvements to the script that deploys Jenkins across our infrastructure [ ][ ][ ].

Lastly, Vagrant Cascadian clarified in the documentation that you need to be user jenkins to run the blacklist command [ ] and the usual build node maintenance was performed was performed by Holger Levsen [ ][ ][ ], Mattia Rizzolo [ ][ ] and Vagrant Cascadian [ ][ ][ ].

Mailing list: There were a number of discussions on our mailing list this month: Paul Spooren started a thread titled Reproducible Builds Verification Format which reopens the discussion around a schema for sharing the results from distributed rebuilders:

To make the results accessible, storable and create tools around them, they should all follow the same schema, a reproducible builds verification format. The format tries to be as generic as possible to cover all open source projects offering precompiled source code. It stores the rebuilder results of what is reproducible and what not.

Hans-Christoph Steiner of the Guardian Project also continued his previous discussion regarding making our website translatable. Lastly, Leo Wandersleb posted a detailed request for feedback on a question of supply chain security and other issues of software review; Leo is the founder of the Wallet Scrutiny project which aims to prove the security of Android Bitcoin Wallets:

Do you own your Bitcoins or do you trust that your app allows you to use your coins while they are actually controlled by them ? Do you have a backup? Do they have a copy they didn t tell you about? Did anybody check the wallet for deliberate backdoors or vulnerabilities? Could anybody check the wallet for those?

Elsewhere, Leo had posted instructions on his attempts to reproduce the binaries for the BlueWallet Bitcoin wallet for iOS and Android platforms.

---

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Twitter: @ReproBuilds
• Reddit: /r/ReproducibleBuilds
• Mailing list: rb-general@lists.reproducible-builds.org

This month s report was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen, Jelle van der Waa and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.

P.step :: Int -> Int -> Context -> Context

S.step :: Int -> Int -> State Context ()
S.step x y = modify (P.step x y)

mkWrap fn = do
     
    let name = mkName (nameBase fn)
    return $ FunD name [ Clause (map VarP args) (NormalB rhs) [] ]

info    <- reify fn
let ty   = (\(VarI _ t _ ) -> t) info
let n    = arity ty - 1
args    <- replicateM n (newName "arg")

-- mkFnApp f [a,b,c] => ((f a) b) c => f a b c
mkFnApp = foldl (\e -> appE e . varE)
rhs     <- [  modify $(mkFnApp (varE fn) args) :: State Context ()  ]

import Liquorice.Pure
wrapPureFunctions = mapM mkWrap pureFns

P.place :: Int -> Int -> (Context -> Context) -> Context -> Context

Changes in littler version 0.3.10 (2020-06-02)

• Changes in examples
  • The update.r script only considers writeable directories.
  • The rcc.r script tries to report full logs by setting _R_CHECK_TESTS_NLINES_=0.
  • The tt.r script has an improved ncpu fallback.
  • Several installation and updating scripts set _R_SHLIB_STRIP_ to TRUE.
  • A new script installBioc.r was added.
  • The --error option to install2.r was generalized (Sergio Oller in #78).
  • The roxy.r script was extended a little.
• Changes in package
  • Travis CI now uses R 4.0.0 and the bionic distro

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

TLDR Instead of patching clang itself, I used a different approach this time: patching Debian tools or implementing some workaround to mitigate an issue.
The percentage of packages failing drop from 4.5% to 3.6% (1400 packages to 1110 - on a total of 31014). I focused on two classes of issues:

• The qmake bug - 250 issues - https://clang.debian.net/status.php?version=9.0.1&key=FAILED_PARSE_DEFAULT
  Basically, Qmake refuses to build a package when gcc's binary is replaced by clang

• Different symbols generated in the libraries - 137 issues - https://clang.debian.net/status.php?version=9.0.1&key=CHANGE_SYM_LIB
  Clang and gcc differ in term of symbols generated in the libraries. While there isn't any impact on the execution or linking, the Debian tooling doesn't have the capability to manage the two cases.
  For more details, see https://bugs.llvm.org/show_bug.cgi?id=30441 & https://bugs.llvm.org/show_bug.cgi?id=45322.

Qmake As I have no intention to merge the patch upstream, I used a very dirty workaround. I overwrote the g++ qmake file by clang's:
https://salsa.debian.org/lucas/collab-qa-tools/-/blob/master/modes/clang10#L44-47 I dropped the number of this failure to 0, making some packages build flawlessly (example: qtcreator, chessx, fwbuilder, etc). However, some packages are still failing later and therefore increasing the number of failures in some other categories like link error. For example, qtads fails because of ordered comparison between pointer and zero or oscar fails on a -Werror,-Wdeprecated-copy error. Breaking the build later also highlighted some new classes of issues which didn't occur with clang < 10.
For example, warnings related to C++ range loop or implicit int float conversion (I fixed a bunch of them in Firefox) .

Symbol differences Historically, symbol management for C++ in Debian has been a pain. Russ Allbery wrote a blog post in 2012 explaining the situation. AFAIK, it hasn't changed much.
Once more, I took the dirty approach: if there new or missing symbols, don't fail the build.
The rational is the following: Packages in the Debian archive are supposed to build without any issue. If there is new or missing symbols, it is probably clang generating a different library but this library is very likely working as expected (and usable by a program compiled with g++ or clang). It is purely a different approach taken by the compiler developer. In order to mitigate this issue, before the build starts, I am modifying dpkg-gensymbols to transform the error into a warning.
So, the typical Debian error some new symbols appeared in the symbols file or some symbols or patterns disappeared in the symbols file will NOT fail the build. Unsurprisingly, all but one package (libktorrent) build. Even if I am pessimistic, I reported a bug on dpkg-dev to evaluate if we could improve dpkg-gensymbol not to fail on these cases.

Next steps The next offender is Imake.tmpl:2243:10: fatal error: ' X11 .rules' file not found with more than an hundred occurrences, reported upstream quite sometime ago. Then, the big issues are going to be much harder to fix as they are real issues/warnings (with -Werror) in the code of the packages. Example: -Wc++11-narrowing & Wreserved-user-defined-literal... The list is long.
I will probably work on that when llvm/clang 11 are in RC phase.

For maintainers & upstream Maintainer of Debian/Ubuntu packages? I am providing a list of failing packages per maintainer: https://clang.debian.net/maintainers.php
For upstream, it is also easy to test with clang. Usually, apt install clang && CC=clang CXX=clang++ <build step> is good enough.

Conclusion With these two changes, I have been able to fix about 290 packages. I think I will be able to get that down a bit more but we will soon reach a plateau as many warnings/issues will have to fix in the C/C++ code itself.

• Opened a pull request against the kitty shell to set a default socket timeout when retrieving remote items via the icat command-line tool. (#659)
• Opened a pull request to make the documentation for the Wand Python/ImageMagick graphics library to build in reproducible manner. [...]
• Fixed an issue in my tickle-me-email library that implements Gettings Things Done (GTD)-like behaviours in IMAP inboxes to prevent a traceback when adding text attachments that were not valid UTF-8. ...]

• The Open Source Initiative held their twice-annual multi-day 'face-to-face' board meeting this time held virtually and participated in the accompanying conversations on strategy, tactical and governance issues, as well as the usual discussions regarding licensing and policy (minutes pending). I also attended the regular monthly meeting for Software in the Public Interest (minutes).
• For my django-slack library (which provides a convenient library between projects using the Django and the Slack chat platform) I reviewed and merged a pull request from Cleiton de Lima to fix an issue due to the force_text method was deprecated favour of force_str. [...]

• Attended more of OpenUK's Teabreak Tuesday and Future Leaders' Training events as well did more work as part of being on the judging panel of the OpenUK Awards. Nominations are open through 15th June in five different categories.
• Opened a pull request for the Redis key-value database to correct a "conflicts" spelling error. [...]

• New features:
  • Check for packages that use $ misc:Pre-Depends) in the Depends field. (#961290)
  • Check for packages installing icon cache files directly under /usr/share/icons/hicolor as they will invariably clash with other packages. (#959855)
  • Check for Homepage fields in debian/control that point to known directory listing pages. (#960366)
  • Update data/fields/perl-provides. [...]
• Bug fixes:
  • Mark that dh-sequence-gnome provides the services of dh_gnome and dh_gnome_clean. (#958500)
  • Accommodate commented-out lines when checking for no-dh-sequencer. (#960485)
• Reporting/output:
  • Clarify error message when the file to check does not exist. (#960970)
  • Mark the debian-rules-uses-as-needed-linker-flag tag as being pedantic & experimental until at least the release of Debian bullseye. (#959696)
• Code improvements:
  • Replace Copyright (C) with the Unicode copyright symbol for consistency [...] and update my copyright years [...].
  • Factor out matching Homepage fields to data/fields/bad-homepages. [...]
  • Various alterations for the continuous integration pipeline. [...][...]

• Opened a pull request to make the documentation for the Wand Python/ImageMagick graphics library to build in reproducible manner. [...]
• In Debian:
  • Submitted 16 patches to fix specific reproducibility issues in ataqv, bowtie2, briquolo, cryptominisat, elinks, mistral, nulib2, process-cpp, properties-cpp, python-watcherclient, tree-puzzle, tree-puzzle, tree-puzzle, vows, wand & wolfssl.
  • Kept isdebianreproducibleyet.com up to date. [...]
• Categorised a large number of packages and issues in the Reproducible Builds "notes" repository.
• In disorderfs, our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues, I replaced the term "dirents" in place of "directory entries" in human-readable output/log messages [...] and used the astyle source code formatter with the default settings to the main disorderfs.cpp file [...].
• Drafted, published and publicised our monthly report.

• Comparison improvements:
  • Improve fuzzy matching of JSON files as file now supports recognising JSON data. (#106)
  • Refactor .changes and .buildinfo handling to show all details (including the GnuPG header and footer components) even when referenced files are not present. (#122)
  • Use our BuildinfoFile comparator (etc.) regardless of whether the associated files (such as the orig.tar.gz and the .deb) are present. [...]
  • Include GnuPG signature data when comparing .buildinfo, .changes, etc. [...]
  • Add support for printing Android APK signatures via apksigner(1). (#121)
  • Identify "iOS App Zip archive data" as .zip files. (#116)
  • Add support for Apple Xcode .mobilepovision files. (#113)
• Bug fixes:
  • Don't print a traceback if we pass a single, missing argument to diffoscope (eg. a JSON diff to re-load). [...]
  • Correct differences typo in the ApkFile handler. (#127)
• Output improvements:
  • Never emit the same id="foo" anchor reference twice in the HTML output, otherwise identically-named parts will not be able to linked to via a #foo anchor. (#120)
  • Never emit an empty "id" anchor either; it is not possible to link to #. [...]
  • Don't pretty-print the output when using the --json presenter; it will usually be too complicated to be readable by the human anyway. [...]
  • Use the SHA256 over MD5 hash when generating page names for the HTML directory-style presenter. (#124)
• Reporting improvements:
  • Clarify the message when we truncate the number of lines to standard error [...] and reduce the number of maximum lines printed to 25 as usually the error is obvious by then [...].
  • Print the amount of free space that we have available in our temporary directory as a debugging message. [...]
  • Clarify Command [ ] failed with exit code messages to remove duplicate exited with exit but also to note that diffoscope is interpreting this as an error. [...]
  • Don't leak the full path of the temporary directory in Command [ ] exited with 1 messages. (#126)
  • Clarify the warning message when we cannot import the debian Python module. [...]
  • Don't repeat stderr from if both commands emit the same output. [...]
  • Clarify that an external command emits for both files, otherwise it can look like we are repeating itself when, in reality, it is being run twice. [...]
• Testsuite improvements:
  • Prevent apksigner test failures due to lack of binfmt_misc, eg. on Salsa CI and elsewhere. [...]
  • Drop .travis.yml as we use Salsa instead. [...]
• Dockerfile improvements:
  • Add a .dockerignore file to whitelist files we actually need in our container. (#105)
  • Use ARG instead of ENV when setting up the DEBIAN_FRONTEND environment variable at runtime. (#103)
  • Run as a non-root user in container. (#102)
  • Install/remove the build-essential during build so we can install the recommended packages from Git. [...]
• Codebase improvements:
  • Bump the officially required version of Python from 3.5 to 3.6. (#117)
  • Drop the (default) shell=False keyword argument to subprocess.Popen so that the potentially-unsafe shell=True is more obvious. [...]
  • Perform string normalisation in Black [...] and include the Black output in the assertion failure too [...].
  • Inline MissingFile's special handling of deb822 to prevent leaking through abstract layers. [...][...]
  • Allow a bare try/except block when cleaning up temporary files with respect to the flake8 quality assurance tool. [...]
  • Rename in_dsc_path to dsc_in_same_dir to clarify the use of this variable. [...]
  • Abstract out the duplicated parts of the debian_fallback class [...] and add descriptions for the file types. [...]
  • Various commenting and internal documentation improvements. [...][...]
  • Rename the Openssl command class to OpenSSLPKCS7 to accommodate other command names with this prefix. [...]
• Misc:
  • Rename the --debugger command-line argument to --pdb. [...]
  • Normalise filesystem stat(2) "birth times" (ie. st_birthtime) in the same way we do with the stat(1) command's Access: and Change: times to fix a nondeterministic build failure in GNU Guix. (#74)
  • Ignore case when ordering our file format descriptions. [...]
  • Drop, add and tidy various module imports. [...][...][...][...]

• Add a completely new design. [...][...]
• Add a separate, canonical page for every new release. [...][...][...]
• Generate a 'latest release' section and display that with the corresponding date on the homepage. [...]
• Add an RSS feed of our releases [...][...][...][...][...][...] and add to Planet Debian [...].
• Dynamically generate our contributor list [...] and supported file formats [...] from the main Git repository.
• Use Jekyll's absolute_url and relative_url where possible [...][...] and move a number of configuration variables to _config.yml [...][...].

• Rename the "Who" page to "Projects". [...]
• Ensure that Jekyll enters the _docs subdirectory to find the _docs/index.md file after an internal move. (#27)
• Wrap ltmain.sh etc. in preformatted quotes. [...]
• Wrap the SOURCE_DATE_EPOCH Python examples onto more lines to prevent visual overflow on the page. [...]
• Correct a "preferred" spelling error. [...]

• Investigated and triaged freerdp, keystone, nginx, tcpreplay & thunderbird, as well as tended to the general upkeep of the dla-needed.txt and ela-needed.txt files, adding various notes, references, attributions and citations.
• Frontdesk duties including responding to user/developer questions, reviewing others' packages, participating in mailing list discussions as well as attending our second regular IRC contributor meeting.
• Issued DLA 2201-1 to prevent a Denial of Service (DoS) vulnerability the ntp network time protocol server/client. ntp allowed an "off-path" attacker to block unauthenticated synchronisation via a server mode packet with a spoofed source IP address because transmissions were rescheduled even if a packet lacked a valid "origin timestamp".
• Issued DLA 2203-1 for the SQLite database to prevent a denial of service attack. In the event of a semantic error in an aggregate query, SQLite did not return early from the resetAccumulator() function which would lead to a crash via a segmentation fault.
• Issued DLA 2204-1 for the Mailman mailing list manager to prevent an arbitrary content injection attack.
• Issued DLA 2211-1 in order to prevent an XML external entity vulnerability in log4net, a logging API for the ECMA Common Language Infrastructure (CLI), sometimes referred to as "Mono". This type of attack occurs when XML input containing a reference to an internet-faced entity is processed by a weakly configured XML parser.
• Prepared and issued ELA-229-1 and DLA 2217-1 for the Apache Tomcat Java server to prevent a remote code execution exploit.

• apksigner: Uses en-dashes (U+2013) in manpage over two hyphens. (#960778)
• devscripts: dd-list -nou results in "unknown option: [ ]". (#960891)
• node-redis: autopkgtest regressions against Redis 6.x. (#960105)

• python-django (3.0.6-1) New upstream bugfix release.
• redis:
  • 6.0.0-1 New upstream "General Availability" release.
  • 6.0.0-2 Mark an autopkgtest as being flaky for now.
  • 6.0.1-1, 6.0.1-2 & 6.0.1-2~bpo10+1 New upstream "General Availability" releases.
  • 6.0.3-1 & 6.0.3-1~bpo10+1 New upstream "General Availability" releases.
• memcached (1.6.6-1) New upstream release, fixing a build failure on s390x and similar architectures. (#960459)
• bfs (1.7-1) New upstream release & move to debhelper-compat level 13.

1. setting a mark just so you can jump back to it (i) is a distinct operation you have to think of separately; and (ii) requires two keypresses, C-SPC C-SPC, rather than just one keypress
2. using exchange-point-and-mark activates the region, so to use it for navigation you need to use either C-u C-x C-x or C-x C-x C-g, neither of which are convenient to type, or else it will be difficult to set regions at the place you ve just jumped to because you ll already have one active.

(defun spw/exchange-point-and-mark (arg)
  "Exchange point and mark, but reactivate mark a bit less often.

Specifically, invert the meaning of ARG in the case where
Transient Mark mode is on but the region is inactive."
  (interactive "P")
  (exchange-point-and-mark
   (if (and transient-mark-mode (not mark-active))
       (not arg)
     arg)))
(global-set-key [remap exchange-point-and-mark] &aposspw/exchange-point-and-mark)

(setq transient-mark-mode nil)
(defun spw/remap-mark-command (command &optional map)
  "Remap a mark-* command to temporarily activate Transient Mark mode."
  (let* ((cmd (symbol-name command))
         (fun (intern (concat "spw/" cmd)))
         (doc (concat "Call  "
                      cmd
                      "&apos and temporarily activate Transient Mark mode.")))
    (fset fun  (lambda ()
                 ,doc
                 (interactive)
                 (call-interactively #&apos,command)
                 (activate-mark)))
    (if map
        (define-key map (vector &aposremap command) fun)
      (global-set-key (vector &aposremap command) fun))))
(dolist (command &apos(mark-word
                   mark-sexp
                   mark-paragraph
                   mark-defun
                   mark-page
                   mark-whole-buffer))
  (spw/remap-mark-command command))
(with-eval-after-load &aposorg
  (spw/remap-mark-command &aposorg-mark-element org-mode-map)
  (spw/remap-mark-command &aposorg-mark-subtree org-mode-map))
;; optional
(global-set-key "\M-=" (lambda () (interactive) (activate-mark)))
;; resettle the previous occupant
(global-set-key "\C-cw" &aposcount-words-region)

1. under the effects of mitigation #1, how much of a barrier to using marks for navigational purposes is it to have to press C-SPC C-SPC instead of having a single binding, C-SPC, for all manual mark setting²
2. under the effects of mitigation #2, how much of a barrier to taking advantage of commands which act differently when the region is active is it to have to temporarily enable Transient Mark mode with C-SPC C-SPC, M-= or one of the mark-* commands?

• KASan support for vmap memory expands KASan s features to include analysis of memory allocated with vmalloc(). More specifically, this means KASan can examine the stack again, since it can be in that region since CONFIG_VMAP_STACK was introduced.
• MIPS can build with GCC plugins and KCOV now, providing those systems with the associated features like stack wiping, coverage analysis, etc.
• userfaultfd requires CAP_SYS_PTRACE for UFFD_FEATURE_EVENT_FORK. This was done to block unprivileged users from abusing the userfaultfd feature, as it is implemented in a way that provides the ability to inject file descriptors into unsuspecting processes.

2020, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

The thought has barely formed when a different shape looms over him, grinning widely enough to show every tooth in its head. They are even, white, and perfect, and yet he somehow can't stop himself from thinking there's something wrong with them, that they're mismatched, that this assortment of teeth was never meant to share a single jaw, a single terrible smile.

• [1] https://www.gq.com/story/inside-diamond-princess-cruise-ship-nightmare-voyage
• [2] http://tinyurl.com/y7wfhgw9

• Debian funded bursaries are available to active Debian contributors.
• Debian diversity bursaries are available to newcomers to Debian/DebConf. Especially from under-represented communities.

Converting a PDF to DjVu The best solution I found makes use of a different file format: .djvu Such files are not PDFs, but can still be opened in Evince and Okular, as well as in the dedicated DjVuLibre application. As an example, I was unable to print page 11 of this paper. Using pdfinfo, I found that it is in PDF 1.5 format and so the transparency effects could be the cause of the out-of-memory printer error. Here's how I converted it to a high-quality DjVu file I could print without problems using Evince:

pdf2djvu -d 1200 2002.04049.pdf > 2002.04049-1200dpi.djvu

Converting a PDF to PDF 1.3 I also tried the DjVu trick on a different unprintable PDF, but it failed to print, even after lowering the resolution to 600dpi:

pdf2djvu -d 600 dow-faq_v1.1.pdf > dow-faq_v1.1-600dpi.djvu

In this case, I used a different technique and simply converted the PDF to version 1.3 (from version 1.6 according to pdfinfo):

ps2pdf13 -r1200x1200 dow-faq_v1.1.pdf dow-faq_v1.1-1200dpi.pdf

This eliminates the problematic transparency and rasterizes the elements that version 1.3 doesn't support.

Changes in version 0.0.5 (2020-05-23)

• Add parseExample from earlier upstream announcement (Dirk).
• Synced with upstream (Brendan in #12) closing #11).
• Updated example parseExample to API changes (Brendan).

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Apache configuration Start by turning on the include filter by adding the following in /etc/apache2/conf-available/ssi.conf:

AddType text/html .shtml
AddOutputFilter INCLUDES .shtml

and making that configuration file active:

a2enconf ssi

Then, find the vhost file where you want to enable SSI and add the following options to a Location or Directory section:

<Location /ssi_files>
    Options +IncludesNOEXEC
    SSLRequireSSL
    Header set Content-Security-Policy: "default-src 'none'"
    Header set X-Content-Type-Options: "nosniff"
</Location>

before adding the necessary modules:

a2enmod headers
a2enmod include

and restarting Apache:

apache2ctl configtest && systemctl restart apache2.service

Create an shtml page With the web server ready to process SSI instructions, the following HTML blurb can be used to display the client IP address:

<!--#echo var="REMOTE_ADDR" -->

or any other built-in variable. Note that you don't need to write a valid HTML for the variable to be substituted and so the above one-liner is all I use on my server.

Security concerns The first thing to note is that the configuration section uses the IncludesNOEXEC option in order to disable arbitrary command execution via SSI. In addition, you can also make sure that the cgi module is disabled since that's a dependency of the more dangerous side of SSI:

a2dismod cgi

Of course, if you rely on this IP address to be accurate, for example because you'll be putting it in your DNS, then you should make sure that you only serve this page over HTTPS, which can be enforced via the SSLRequireSSL directive. I included two other headers in the above vhost config (Content-Security-Policy and X-Content-Type-Options) in order to limit the damage that could be done in case a malicious file was accidentally dropped in that directory. Finally, I suggest making sure that only the root user has writable access to the directory which has server-side includes enabled:

$ ls -la /var/www/ssi_includes/
total 12
drwxr-xr-x  2 root     root     4096 May 18 15:58 .
drwxr-xr-x 16 root     root     4096 May 18 15:40 ..
-rw-r--r--  1 root     root        0 May 18 15:46 index.html
-rw-r--r--  1 root     root       32 May 18 15:58 whatsmyip.shtml