Chris Lamb: Free software activities in March 2020
Here is my monthly update covering what I have been doing in the free software world during March 2020 (previous month):
- Patched the browserify-lite Javascript compiler to ensure that module dependency maps are rendered in a deterministic order. (#14)
- Pushed a change to the "pmemkv" key-value data store to make their documentation build reproducibly. (#615)
-
As part of my duties of being on the board of directors of the Open Source Initiative and Software in the Public Interest I attended their respective monthly meetings and participated in various licensing and other discussions occurring on the internet as well as the usual internal discussions regarding logistics, licensing, policy and liaising with the ClearlyDefined and so
Congratulations to Megan Sanicki and Josh Simmons who were elected and re-elected respectively to the OSI Board's two individual member seats and to Italo Vignoli who was elected to the affiliate member seat. These newly-elected Directors will took their seats on March 20, 2020.
- Opened a pull request against the onionbalance load balancer for hidden Tor services to fix test failures under Pytest 3.x. (#75)
- Filed an issue against IMAP Spam Begone (isbg), a script by Louis-Philippe V ronneau (pollo) that makes it easy to process an email inbox using SpamAssassin, to report that a duplicate documentation entry includes a nondeterministic value taken from the value of the XDG cache directory. (#151)
- As part of being on being on the judging panel of the OpenUK Awards I am pleased to announce that after some discussion nominations are now open until 15th June in five different categories.
-
Merged a number of contributions to my django-cache-toolbox "non-magical" caching library for Django web applications, including caching negative relation lookups locally (#14) and to include the
READMEfile in the package long description (#17). -
Made some small changes to my tickle-me-email library which implements Gettings Things Done (GTD)-like behaviours in IMAP inboxes to support to optionally limiting the number of messages in the
send-laterfunctionality. [...]
-
New features:
- Check for
py3versions -iin autopkgtests anddebian/rulesfiles. (#954763) - Warn when
py3versions -sis used without apython3-alldependency. (#954763) - Expand
possible-missing-colon-in-closesto also check for semicolons used in place of colons. (#954484) - Check for new packages that use a date-based versioning scheme (eg.
YYYYMMDD-1) without a0~suffix. (#953036)
- Check for
-
Improvements:
- Don't emit
duplicate-short-descriptionorduplicate-long-descriptionif they contain substitution variables. (#947168) - Don't report about using a newer
Standards-Versionwhen uploading to the backports repository. [...] - Don't check the length of
Checksums-Sha256fields. (#954798) - Don't check for
DEBIAN/md5sumscontrol files in udebs. (#954803) - Apply a patch from gregoa to remove some
<!nocheck>annotations; some packages are required to build successfully. (#954338)
- Don't emit
-
Misc:
- Correct reference to build dependencies in the long description of the
debian-rules-uses-installed-python-versionstag. [...] - Make some cosmetic improvements to the
CONTRIBUTING.mdfile. [...] - Correct reference to a bug in a previous
debian/changelogentry. [...] - Avoid indenting approximately 150 lines by returning early from a subroutine and other code improvements. [...]
- Correct reference to build dependencies in the long description of the
Reproducible builds
One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as
pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes.
The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.
The initiative is proud to be a member project of the Software Freedom Conservancy, a not-for-profit 501(c)(3) charity focused on ethical technology and user freedom.
Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter.
This month, I:
- Filed an issue against IMAP Spam Begone a script by Louis-Philippe V ronneau (pollo) that makes it easy to process an email inbox using SpamAssassin to report that a (duplicate) documentation entry includes nondeterministic value taken from the value of the XDG cache directory (#151) and filed an upstream pull requests against the pmemkv key-value data store to make their documentation build reproducibly (#615).
-
Further refined my merge request against the
debian-installercomponent to allow all arguments fromsources.listfiles (such as[check-valid-until=no]) in order that we can test the reproducibility of the installer images on the Reproducible Builds own testing infrastructure. (#13) - Submitted two following patches to fix reproducibility-related toolchain issues within Debian:
-
Submitted eight patches to fix specific reproducibility issues in beep (caused by a variation between
/bin/dashand/bin/bash), cloudkitty (due to a default value being taken from the number of CPUs on the build machine), font-manager (embedding the value of@abs_top_srcdir@into the resulting binary), gucharmap (due to embedding the absolute build path when generating a comment in a header file), infernal (timestamps are injected into a Python example, which should not be shipped anyway), ndisc6 (embeds the value ofCFLAGSinto the binary without sanitising any absolute build paths), node-nodedbi (embedded timestamp in binary) & pmemkv (does not respectSOURCE_DATE_EPOCHwhen populating aYEARvariable). - Kept isdebianreproducibleyet.com up to date. [...]
- Continued collaborative work on an academic paper to be published within the next few months.
- Categorised a large number of packages and issues in the Reproducible Builds "notes" repository.
- Drafted, published and publicised our monthly report.
-
Improved our website, including correcting the syntax of some CSS class formatting [...], improved some "filed against" copy a little better [...] and corrected a reference to
calendar.monthrangePython method. [...]
In our tooling, I also made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues, including preparing and uploading version 138 to Debian:
-
Improvements:
- Don't allow errors with R script deserialisation cause the entire operation to fail, for example if an external library cannot be loaded. (#91)
- Experiment with memoising output from expensive external commands, eg.
readelf. (#93) - Use
dumppdffrom thepython3-pdfminerif we do not see any other differences frompdftext, etc. (#92) - Prevent a traceback when comparing two R
.rdxfiles directly as theget_membermethod will return a file even if the file is missing. [...]
-
Reporting:
- Display the supported file formats into the package long description. (#90)
- Print a potentially-helpful message if the PyPDF2 module is not installed. [...]
- Remove any duplicate comparator descriptions when formatting in the
--helpoutput or in the package long description. [...] - Weaken "Install the X package to get a better output." message to "... may produce a better output." as the former is not guaranteed. [...]
-
Misc:
- Ensure we only parse the recommended packages from
--list-debian-substvarswhen we want them fordebian/tests/controlgeneration. [...] - Add upstream metadata file [...] and add a Lintian override for
upstream-metadata-in-native-sourceas "we" are upstream. [...] - Inline the
RequiredToolNotFound.get_packagemethod's functionality as it is only used once. [...] - Drop the deprecated "
py36 = [..]" argument in thepyproject.tomlfile. [...]
- Ensure we only parse the recommended packages from
-
Require a HTTP
POSTmethod in the web-based scheduler as not only should HTTP GET requests be idempotent but this will allow many future improvements in the user interface. [...][...][...] - Improve the authentication error message in the rescheduler to suggest that the developer's SSL certificate may have expired. [...]
Debian LTS This month I have worked 18 hours on Debian Long Term Support (LTS) and 8 hours on its sister Extended LTS project.
-
Investigated and triaged
glibc(CVE-2020-1751),jackson-databind,libbsd(CVE-2019-20367),libvirt(CVE-2019-20485),netkit-telnet&netkit-telnet-ssl(CVE-2020-10188),pdfresurrect(CVE-2020-9549) &shiro(CVE-2020-1957), etc. - In the script that reserves a unique advisory number don't warn about potential duplicate work when issuing a regression in order to avoid this message being missed when it does apply. [...]
- Frontdesk duties, responding to user/developer questions, reviewing others' packages, participating in mailing list discussions, etc.
-
xtrlockversions2.8+deb9u1(#949112) and2.8+deb10u1(#949113) was accepted to the Debian jessie and buster distributions. - Issued DLA 2115-2 to correct a regression in a previous fix (a use-after-free vulnerability) in the ProFTPD FTP server.
- Issued DLA 2132-1 to fix an issue where incorrect default permissions on a HTTP cookie store could have allowed local attackers to read private credentials in libzypp, the library underpinning package management tools such as YaST, zypper and the openSUSE/SLE implementation of PackageKit.
- Issued DLA 2134-1 to patch an out-of-bounds write vulnerability in pdfresurrect, a tool for extracting or scrubbing versioning data from PDF documents.
- Issued DLA 2136-1, addressing an out-of-bounds buffer read vulnerability in libvpx, a library implementing the VP8 & VP9 video codecs.
-
Issued DLA 2142-1. It was discovered that there was a buffer overflow vulnerability in slirp, a SLIP/PPP emulator for using a dial up shell account. This was caused by the incorrect usage of return values from
snprintf(3). -
Issued DLA 2145-1 and DLA 2145-2 for
twistedto prevent a large number of HTTP request splitting vulnerabilities in Twisted, a Python event-based framework for building various types of internet applications. -
Issued ELA-219-1 to address an out-of-bounds read vulnerability during string comparisons in
libbsd, a library of functions commonly available on BSD systems but not on others such as GNU.
Debian Uploads
-
python-django (
2.2.11-1&3.0.4-1) New upstream security releases. (#953102) -
redis:
6.0~rc1-3Install OpenSSL when running the testsuite as it is required for generating test certificates.6.0~rc2-1New upstream beta release.
-
xtrlock (
2.13) Add a warning that X11 does not support grabbing events under the Wayland display server protocol. (#953319) -
gunicorn (
20.0.4-4) Ensure that the Python 3.xgunicornbinary package replaces the now-legacygunicorn3that was removed in19.9.0-2upon installation. I also backported this version to buster-backports. (#953883) -
hiredis (
0.14.1-1) New upstream release. -
libfiu (
1.00-7) Fix build failures under Python 3.8. (#954287) -
memcached:
1.6.0-1New upstream release.1.6.1-1New upstream release.1.6.2-1New upstream security release. (#954808)
-
bfs (
1.6-2,1.6-3&1.6-4) Fix various build failures on kFreeBSD architectures. -
mtools (
4.0.24-1) New upstream release. -
xcb (
2.4-6) Remove a vague "pigeon holes" metaphor from package description.
pyptlib package be removed from the archive (#953429) as well as uploading onionbalance (0.1.8-6) to fix test failures under Pytest 3.x (#953535) and a new upstream release of nautilus-wipe.
Finally, I sponsored an upload of bilibop (0.6.1) on behalf of Yann Amar.
TL;DR: The zoom meeting link you have probably look like this:
No more nagging me about the placeholder image from Wikipedia I used in
Here's what the team did today:
tumbleweed
Stefano started the day by hacking away on our video archive. We eventually want
to upload all our videos to YouTube to give them exposure, but sadly our archive
metadata is in a pretty poor shape.
With the script tumbleweed wrote, we can scrape the archive for matches against
the old DebConf's pentabarf XML we have.
tumbleweed also helped Ivo with the ansible PXE setup he's working on. Some
recent contributions from a collaborator implemented new features (like a nice
menu to choose from) but also came with a few annoying bugs.
ivodd
Ivo continued working on the PXE setup today. He also tried to break our ansible
setup by using fresh installs with different user cases (locales, interfaces,
etc.), with some success.
The reason he and Stefano are working so hard on the PXE boot is that we had a
discussion about the future of our USB install method. The general consensus on
this was although we would not remove it, we would not actively maintain it
anymore.
PXE is less trouble for multiple machines. For single machines or if you don't
control the DHCP server, using ansible manually on a fresh Debian install will
be the recommended way.
olasd
After a very long drive, olasd arrived late in the evening with all our gear.
Hurray! We were thus able to set up some test boxes and start wiring the airbnb
properly. Tomorrow will certainly be more productive with all this stuff at our
disposition.
pollo
Today I mainly worked on setting up our documentation website. After some
debate, we decided that

Of course, reality has trouble matching all the post-processing filters.
Plan for the week
Now on a more serious note; apart from enjoying the beautiful city of Cambridge,
here's what the team plans to do this week:
tumbleweed
Stefano wants to continue refactoring our ansible setup. A lot of things have
been added in the last year, but some of it are hacks we should remove and
implement correctly.
highvoltage
Jonathan won't be able to come to Cambridge, but plans to work remotely, mainly
on our desktop/xfce session implementation. Another pile of hacks waiting to be
cleaned!
ivodd
Ivo has been working a lot of the pre-ansible part of our installation and plans
to continue working on that. At the moment, creating an installation USB key is
pretty complicated and he wants to make that simpler.
olasd
Nicolas completely reimplemented our streaming setup for DC17 and wants to
continue working on that.
More specifically, he wants to write scripts to automatically setup and teardown
- via API calls - the distributed streaming network we now use.
Finding a way to push TLS certificates to those mirrors, adding a live stream
viewer on
The twelveth update in the 0.12.* series of
It's been a while. And currently I shouldn't even post but rather pack my stuff because I'll get the keys to my flat in 6 days. Yay!
But, for packing I need a good sound track. And today it is
Thats So Raven
If I could sum up the past year in one word, that word would be distraction. There have been so many strange, confusing or simply unforseen things going on I have had trouble focusing like never before.
For instance, on the opposite side of the street from me is one of Jersey City's old resorvoirs. It's not used for drinking water anymore and the city eventually plans on merging it into the park on the other side. In the meantime it has become something of a wildlife refuge. Which is nice except one of the newly settled critters was a bird of prey -- the consensus is possibly some kind of hawk or raven. Starting your morning commute under the eyes of a harbinger of death is very goth and I even learned to deal with the occasional piece of deconstructed rodent on my doorstep but nighttime was a big problem. For contrary to popular belief, ravens do not quoth "nevermore" but "KRRAAAA". Very loudly. Just as soon as you have drifted of to sleep. Eventually my sleep-deprived neighbors and I appealed to the NJ division of enviromental protection to get it removed but by the time they were ready to swing into action the bird had left for somewhere more congenial like Transylvania or Newark.
Or here are some more complete wastes of time: I go the doctor for my annual physical. The insurance company codes it as Adult Onset Diabetes by accident. One day I opened the lid of my laptop and there's a "ping" sound and a piece of the hinge flies off. Apparently that also severed the connection to the screen and naturally the warranty had just expired so I had to spend the next month tethered to an external monitor until I could afford to buy a new one. Mix in all the usual social, political, family and work drama and you can see that it has been a very trying time for me.
Dovecot
I have managed to get some Debian work done. On Dovecot, my principal package, I have gotten tremendous support from Apollon Oikonomopolous who I belatedly welcome as a member of the Dovecot maintainer team. He has been particularly helpful in fixing our systemd support and cleaning out a lot of the old and invalid bugs. We're in pretty good shape for the freeze. Upstream has released an RC of 2.2.26 and hopefully the final version will be out in the next couple of days so we can include it in Stretch. We can always use more help with the package so let me know if you're interested.
Debian-IN
Most of the action has been going on without me but I've been lending support and sponsoring whenever I can. We have several new DDs and DMs but still no one north of the Vindhyas I'm afraid.
Debian Perl Group
gregoa did a ping of inactive maintainers and I regretfully had to admit to myself that I wasn't going to be of use anytime soon so I resigned. Perl remains my favorite language and I've actually been more involved in the meetings of
What happened in the
I got an email last year pointing out a cosmetic issue with
changelogs.debian.net. I think at the time of the email, the only problem was
some bitrot in PHP's built-in server variables making some text appear
incorrectly.
I duly added something to my TODO list to fix it, and it subsequently sat there
for like 13 months. In the ensuing time, Debian changed some stuff, and my code
started incorrectly handling a 302 as well, which actually broke it good and
proper.
I finally got around to fixing it.
I also fixed a problem where sometimes there can be multiple entries in the
Sources file for a package (switching to using
api.ftp-master.debian.org would also address this), which caused
sometimes caused an incorrect version of the changelog to be returned.
In the resulting tinkering, I learned about