FTP master This month I accepted 159 packages and rejected 16. The overall number of packages that got accepted was 172. Debian LTS This was my seventy-fourth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. This month my all in all workload has been 21.75h. During that time I did LTS uploads of:

• [DLA 2336-1] firejail security update for two CVEs
• [DLA 2337-1] python2.7 security update for nine CVEs
• [DLA 2353-1] bacula security update for one CVE
• [DLA 2354-1] ndpi security update for one CVE
• [DLA 2355-1] bind9 security update for two CVEs
• [DLA 2359-1] xorg-server security update for five CVEs

I also started to work on curl but did not upload a fixed version yet. As usual, testing the package takes up some time. Last but not least I did some days of frontdesk duties. Debian ELTS This month was the twenty sixth ELTS month. During my allocated time I uploaded:

• ELA-265-1 for python2.7
• ELA-270-1 for bind9
• ELA-272-1 for xorg-server

Like in LTS, I also started to work on curl and encountered the same problems as in LTS above. Last but not least I did some days of frontdesk duties. Other stuff This month I found again some time for other Debian work and uploaded packages to fix bugs, mainly around gcc10:

• arpalert
• libosmo-sccp
• libosmocore
• npd6
• openbsc
• osmo-bsc
• osmo-bts
• osmo-iuh
• osmo-msc
• osmo-sgsn
• siggen
• scheme48
• smstools

I also uploaded new upstream versions of:

• cc-tools
• libosmo-abis
• libosmo-netif
• libosmo-sccp
• libosmocore
• openbsc
• osmo-bsc
• osmo-bts
• osmo-fl2k
• osmo-ggsn
• osmo-hlr
• osmo-iuh
• osmo-mgw
• osmo-msc
• osmo-pcu
• osmo-sgsn
• osmo-trx
• pyicloud

All package called *osmo* are developed by the Osmocom project, that is about Open Source MObile COMmunication. They are really doing a great job and I apologize that my uploads of new versions are mostly far behind their development. Some of the uploads are related to new packages:

• puppet-module-alteholz-tdc
• puppet-module-cirrax-gitolite

To ensure Dronefly always remains free, the Dronefly project has been relicensed under two copyleft licenses. Read the license change and learn more about copyleft at these links. I was prompted to make this change after a recent incident in the Red DiscordBot development community that made me reconsider my prior position that the liberal MIT license was best for our project. While on the face of it, making your license as liberal as possible might seem like the most generous and hassle-free way to license any project, I was shocked into the realization that its liberality was also its fatal flaw: all is well and good so long as everyone is being cooperative, but it does not afford any protection to developers or users should things suddenly go sideways in how a project is run. A copyleft license is the best way to avoid such issues. In this incident a sad story of conflict between developers I respect on both sides of the rift, and owe a debt to for what they ve taught me three cogs we had come to depend on suddenly stopped being viable for us to use due to changes to the license & the code. Effectively, those cogs became unsupported and unsupportable. To avoid any such future disaster with the Dronefly project, I started shopping for a new license that would protect developers and users alike from similarly losing support, or losing control of their contributions. I owe thanks to Dale Floer, a team member who early on advised me the AGPL might be a better fit, and later was helpful in selecting the doc license and encouraging me to follow through. We ran the new licenses by each contributor and arrived at this consensus: the AGPL is best suited for our server-based code, and CC-BY-SA is best suited for our documentation. The relicensing was made official this morning. On Discord platform alternatives You might well question what I, a Debian developer steeped in free software culture and otherwise in agreement with its principles, am doing encouraging a community to grow on the proprietary Discord platform! I have no satisfying answer to that. I explained when I introduced my project here some of the backstory, but that s more of an account of its beginnings than justification for it to continue on this platform. Honestly, all I can offer is a rather dissatisfying it seemed like the right thing to do at the time. Time will tell whether we could successfully move off of it to a freedom-respecting and privacy-respecting alternative chat platform that is both socially and technically viable to migrate to. That platform would ideally:

• not be under the control of a single, central commercial entity running proprietary code, so their privacy is safeguarded, and they are protected from disaster, should it become unattractive to remain on the platform;
• have a vibrant and supportive open source third party extension development community;
• support our image-rich content and effortless sharing of URLs with previews automatically provided from the page s content (e.g. via OpenGraph tags);
• be effortless to join regardless of what platform/device each user uses;
• keep a history of messages so that future members joining the community can benefit from past conversations, and existing members can catch up on conversations they missed;
• but above all else: be acceptable and compelling to the existing community to move over onto it.

I m intrigued by Matrix and wonder if it provides some or all of the above in its current form. Are you a developer writing bots for this platform? If so, I especially want to hear from you in the comments about your experience. Or in any case, if you ve been there before if you ve faced the same issue with your community and have a success story to share, I would love to hear from you.

Ansible ships a lot of modules you can combine for your configuration management needs. However, the quality of these modules may vary widely. Sometimes, it may be quicker and more robust to write your own module instead of shopping and assembling existing ones.¹ In my opinion, a robust module exhibits the following characteristics:

• idempotency,
• diff support,
• check mode compatibility,
• correct change signaling, and
• lifecycle management.

In a nutshell, it means the module can run with --diff --check and shows the changes it would apply. When run twice in a row, the second run won t apply or signal changes. The last bullet point suggests the module should be able to delete outdated objects configured during previous runs.² The module code should be minimal and tailored to your needs. Making the module generic for use by other users is a non-goal. Less code usually means less bugs and easier to understand. I do not cover testing here. It is undeniably a good practice, but it requires a significant effort. In my opinion, it is preferable to have a well written module matching the above characteristics rather than a module that is well tested but without them or a module requiring further (untested) assembly to meet your needs.

Module skeleton Ansible documentation contains instructions to build a module, along with some best practices. As one of our non-goal is to distribute it, we choose to take some shortcuts and skip some of the boilerplate. Let s assume we build a module with the following signature:

custom_module:
  user: someone
  password: something
  data: "some random string"

There are various locations you can put a module in Ansible. A common possibility is to include it into a role. In a library/ subdirectory, create an empty __init__.py file and a custom_module.py file with the following code:³

#!/usr/bin/python
import yaml
from ansible.module_utils.basic import AnsibleModule
def main():
    # Define options accepted by the module.  
    module_args = dict(
        user=dict(type='str', required=True),
        password=dict(type='str', required=True, no_log=True),
        data=dict(type='str', required=True),
    )
    module = AnsibleModule(
        argument_spec=module_args,
        supports_check_mode=True
    )
    result = dict(
        changed=False
    )
    got =  
    wanted =  
    # Populate both  got  and  wanted .  
    # [...]
    if got != wanted:
        result['changed'] = True
        result['diff'] = dict(
            before=yaml.safe_dump(got),
            after=yaml.safe_dump(wanted)
        )
    if module.check_mode or not result['changed']:
        module.exit_json(**result)
    # Apply changes.  
    # [...]
    module.exit_json(**result)
if __name__ == '__main__':
    main()

The first part, in , defines the module, with the accepted options. Refer to the documentation on argument_spec for more details. The second part, in , builds the got and wanted variables. got is the current state while wanted is the target state. For example, if you need to modify records in a database server, got would be the current rows while wanted would be the modified rows. Then, we compare got and wanted. If there is a difference, changed is switched to True and we prepare the diff object. Ansible uses it to display the differences between the states. If we are running in check mode or if no change is detected, we stop here. The last part, in , applies the changes. Usually, it means iterating over the two structures to detect the differences and create the missing items, delete the unwanted ones and update the existing ones.

Documentation Ansible provides a fairly complete page on how to document a module. I advise you to take a more minimal approach by only documenting each option sparingly,⁴ skipping the examples and only documenting return values if it needs to. I usually limit myself to something like this:

DOCUMENTATION = """
---
module: custom_module.py
short_description: Pass provided data to remote service
description:
  - Mention anything useful for your workmate.
  - Also mention anything you want to remember in 6 months.
options:
  user:
    description:
      - user to identify to remote service
  password:
    description:
      - password for authentication to remote service
  data:
    description:
      - data to send to remote service
"""

Error handling If you run into an error, you can stop the execution with module.fail_json():

module.fail_json(
    msg=f"remote service answered with  code :  message ",
    **result
)

There is no requirement to intercept all errors. Sometimes, not swallowing an exception provides better information than replacing it with a generic message.

Returning additional values A module may return additional information that can be captured to be used in another task through the register directive. For this purpose, you can add arbitrary fields to the result dictionary. Have a look at the documentation for common return values. You should try to add these fields before exiting the module when in check mode. The returned values can be documented.

Examples Here are several examples of custom modules following the previous skeleton. Each example highlight why a custom module was written instead of assembling existing modules.

• Syncing SSH keys on Cisco IOS-XR
• Syncing MySQL tables
• Syncing NetBox
• Syncing RIPE, ARIN and APNIC objects (not released yet)
• Syncing GCP IPsec VPNs (not done yet)

---

1. Also, when using modules from Ansible Galaxy, you introduce a dependency to a third-party. This is not something that should be decided lightly: it may break later, it may only meet 80% of the needs, it may add bugs.
2. Some declarative systems, like Terraform, exhibits all these behaviors.
3. Do not worry about the shebang. It is hardcoded to /usr/bin/python. Ansible will modify it to match the chosen interpreter on the remote host. You can write Python 3 code if ansible_python_interpreter evaluates to a Python 3 interpreter.
4. The main issue I have with this non-programmatic approach to documentation is that it partly repeats the information contained in argument_spec. I think an auto-documenting structure would avoid this.

Here s my (eleventh) monthly update about the activities I ve done in the F/L/OSS world.

Debian

This was my 20th month of contributing to Debian. I became a DM in late March last year and a DD last Christmas! \o/ Well, this month we had DebConf! \o/
(more about this later this week!) Anyway, here are the following things I did in Debian this month:

Uploads and bug fixes:

• rubocop (0.89.1+dfsg-1) - New upstream version for RuboCop::Packaging.
• ruby-rubocop-ast (0.3.0+dfsg-1) - New upstream version for RuboCop's latest version.
• ruby-rubocop-packaging (0.3.0-1) - Shouldn t check lib/ and gemspec file.
• bidi-clojure (2.1.3-1) - New upstream version for Puppet6.
• Source-only uploads for ruby-anima, ruby-uniform-notifier, ruby-unparser, ruby-morpher, and ruby-path-expander.

Other $things:

• Mentoring for newcomers.
• FTP Trainee reviewing.
• Moderation of -project mailing list.
• Sponsored php-dasprid-enum and php-bacon-baconqrcode for William and ruby-unparser, ruby-morpher, and ruby-path-exapander for Cocoa.

---

Goodbye GSoC! \o/ In May, I got selected as a Google Summer of Code student for Debian again! \o/
I am working on the Upstream-Downstream Cooperation in Ruby project. The other 5 blogs can be found here:

• GSoC Phase 1 (part 1).
• GSoC Phase 1 (part 2).
• GSoC Phase 2 (part 1).
• GSoC Phase 2 (part 2).
• GSoC Phase 3 (part 1).
• And this is GSoC Phase 3 (part 2).

Also, I log daily updates at gsocwithutkarsh2102.tk. Since this is a wrap and whilst the daily updates are already available at the above site^, I ll quickly mention the important points and links here.

• The git repository is hosted on GitHub: https://github.com/utkarsh2102/rubocop-packaging.
• It is a linter, an extension of RuboCop, focused on enforcing upstream best practices and coding conventions.
• There have been 5 releases in all, including 4 cops and other bug fixes (including false-positives and false-negatives).
• The entire source code is documented with a separate docs/ directory, which is hosted at https://docs.rubocop.org/rubocop-packaging.
• The packaging style guide is hosted at https://packaging.rubystyle.guide.
• At the time of writing this, it is being used by around 30 other projects ^.^
• Not only could you install this via gem install rubocop-packaging, but also via apt install ruby-rubocop-packaging.
• The total work consists of around 85 commits with 15 PRs, contributed by 4 amazing people (including me :P) in the last 3 months (June 20 to August 20).
• And finally, many thanks to two amazing people (the mentors for this project), Antonio Terceiro and David Rodr guez!

---

Continuation of GSoC for other Ruby related stuff!

Whilst working on Rubocop::Packaging, I contributed to more Ruby projects, refactoring their library a little bit and mostly fixing RuboCop issues and fixing issues that the Packaging extension reports as offensive .
Following are the PRs that I raised:

• PR #170 for autoprefixer-rails to drop git ls-files in gemspec.
• PR #479 for cucumber-rails to update RuboCop and RuboCop::Packaging.
• PR #1465 for cucumber-ruby for updating RuboCop to v0.89.
• PR #208 for cucumber-ruby-core to fix .rubocop.yml and add RuboCop::Packaging as a development dependency.
• PR #178 for webdrivers to update the RuboCop and RuboCop::RSpec version to the latest.
• PR #179 for webdrivers to drop git ls-files in gemspec.

---

Debian (E)LTS

Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success. And Debian Extended LTS (ELTS) is its sister project, extending support to the Jessie release (+2 years after LTS support). This was my eleventh month as a Debian LTS and my second as a Debian ELTS paid contributor.
I was assigned 21.75 hours for LTS and 14.25 hours for ELTS and worked on the following things:

LTS CVE Fixes and Announcements:

• Issued DLA 2304-1, fixing CVE-2015-9542, for libpam-radius-auth.
  For Debian 9 Stretch, these problems have been fixed in version 1.3.16-5+deb9u1.
• Issued DLA 2305-1, fixing CVE-2018-10756, for transmission.
  For Debian 9 Stretch, these problems have been fixed in version 2.92-2+deb9u2.
• Issued DLA 2307-1, fixing CVE-2018-1000544, for ruby-zip.
  For Debian 9 Stretch, these problems have been fixed in version 1.2.0-1.1+deb9u1.
• Issued DLA 2308-1, fixing CVE-2019-17113, for libopenmpt.
  For Debian 9 Stretch, these problems have been fixed in version 0.2.7386~beta20.3-3+deb9u4.
• Issued DLA 2317-1, fixing CVE-2020-10177, for pillow.
  For Debian 9 Stretch, these problems have been fixed in version 4.0.0-4+deb9u2.
• Issued DLA 2318-1, fixing CVE-2019-10064 and CVE-2020-12695, for wpa.
  For Debian 9 Stretch, these problems have been fixed in version 2:2.4-1+deb9u7.
• Started working on uwsgi update for CVE-2020-11984. It seems that src:apache2 wasn t affected by that, but src:uwsgi was.

ELTS CVE Fixes and Announcements:

• Issued ELA 255-1, fixing CVE-2020-14344, for libx11.
  For Debian 8 Jessie, these problems have been fixed in version 2:1.6.2-3+deb8u3.
• Issued ELA 259-1, fixing CVE-2020-10177, for pillow.
  For Debian 8 Jessie, these problems have been fixed in version 2.6.1-2+deb8u5.
• Issued ELA 269-1, fixing CVE-2020-11985, for apache2.
  For Debian 8 Jessie, these problems have been fixed in version 2.4.10-10+deb8u17.
• Started working on clamAV update, it s a major bump from v0.101.5 to v0.102.4. There were lots of movings parts. Contacted upstream maintainers to help reduce the risk of regression. Came up with a patch to loosen the libcurl version requirement. Hopefully, the update could be rolled out soon!

Other (E)LTS Work:

• I spent an additional 11.15 hours working on compiling the responses of the LTS survey and preparing a gist of it for its presentation during the Debian LTS BoF at DebConf20.
• Triaged qemu, pillow, gupnp, clamav, apache2, and uwsgi.
• Marked CVE-2020-11538/pillow as not-affected for Stretch.
• Marked CVE-2020-11984/apache2 as not-affected for Stretch.
• Marked CVE-2020-10378/pillow as not-affected for Jessie.
• Marked CVE-2020-11538/pillow as not-affected for Jessie.
• Marked CVE-2020-3481/clamav as not-affected for Jessie.
• Marked CVE-2020-11984/apache2 as not-affected for Jessie.
• Marked CVE-2020- 9490,11993 /apache2 as not-affected for Jessie.
• Hosted Debian LTS BoF at DebConf20. Recording here.
• General discussion on LTS private and public mailing list.

---

Until next time.
:wq for today.

Here is another monthly update covering what I have been doing in the free software world during August 2020 (previous month):

• Filed a pull request against django-enumfield, a library that provides an enumeration-like model field for the Django web development framework. The classproperty helper has been moved to django.utils.functional in newer versions of Django. [...]
• Transferred the maintainership of my Strava Enhancement Suite Chrome extension to improve the user experience on the Strava athletic tracker to Pavel Dolecek.

• As part of my role of being the assistant Secretary of the Open Source Initiative and a board director of Software in the Public Interest, I attended their respective monthly meetings and participated in various licensing and other discussions occurring on the internet, as well as the usual internal discussions, etc.
• Filed a pull request for JSON-C, a reference counting library to allow you to easily manipulate JSON objects from C in order to make the documentation build reproducibly. [...]
• Reviewed and merged some changes to my django-auto-one-to-one library for Django from Dan Palmer (which automatically creates and destroys associated model instances) to not configure signals for models that aren't installed and to honour INSTALLED_APPS during model setup. [...]
• Merged a pull request from Michael K. to cleanup the codebase after dropping support for Python 2 and Django 1.x [...] in my django-slack library which provides a convenient wrapper between projects using the Django and the Slack chat platform.
• Updated my django-staticfiles-dotd utility that concatenates Debian .d-style directories containing Javascript and CSS to drop unquote usage from the six compatibility library. [...]

I uploaded Lintian versions 2.86.0, 2.87.0, 2.88.0, 2.89.0, 2.90.0, 2.91.0 and 2.92.0, as well as made the following changes:

• New features:
  • Check for StandardOutput= and StandardError= fields that use the deprecated syslog or syslog-console systemd facilities. (#966617)
  • Add support for clzip as an alternative for lzip. (#967083)
  • Check for User=nobody and Group=nogroup in systemd .service files. (#966623)
• Bug fixes:
  • Don't emit the patch-not-forwarded-upstream tag for README files under debian/patches. (#968845)
  • Fix a false positive with the no-dh-sequencer tag due to indirect target dependences. (#968108)
• Reporting/interface:
  • Clarify the grammar of the package-uses-old-debhelper-compat-version tag. [...]
  • Update link to the detagtive web interface code as it has been moved to the lintian salsa namespace. [...]
  • Drop the misleading "Beta testing" from the webpage header. [...]
• Misc:
  • Add justification for the use of the lzip dependency in a previous debian/changelog entry. (#966817)
  • Update the generate-tag-summary release script to reflect change of tag definition filename extension change from .desc .tag. [...]
  • Revert a change to the spelling-error-in-rules-requires-root tag's severity; this is not a "spelling" check in the sense that it does not use our dictionary. [...]
  • Drop an unused $skip_tag argument in the extract_service_file_values routine. [...]

Reproducible Builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. The project is proud to be a member project of the Software Freedom Conservancy. Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter.

• Further refined my merge request against the debian-installer component to allow all arguments from sources.list files (such as [check-valid-until=no]) in order that we can test the reproducibility of the installer images on the Reproducible Builds own testing infrastructure and sent a gentle ping to the team that maintains that code.
• Responded at length to a discussion about the status of the reproducibility of ISO images [...], wrote back to the maintainer of WalletScrutiny asking why Android developers appear to be "allergic" to their application not being reproducible [...] and passed on an issue about the diffoscope website being down [...]. I also added a few short remarks on a proposal to bundle .buildinfo files within .deb archives [...].

This month, I:

• Filed a pull request for JSON-C, a reference counting library to allow you to easily construct JSON objects from C in order to make the documentation build reproducibly. [...]
• In Debian, I:
  • Kept isdebianreproducibleyet.com up to date. [...][...][...][...]
  • Submitted the following patches to fix reproducibility-related toolchain issues within Debian:
    • debhelper: Please make the examples from dh_missing reproducible. (#968187)
    • evolution: Please make the shipped .h files reproducible. (#968700)
  • I also submitted 9 patches to fix specific reproducibility issues in aflplusplus, chirp, golang-gonum-v1-plot, json-c, nmh, pencil2d, pixelmed-codec, serd & tpot.
• Categorised a large number of packages and issues in the Reproducible Builds "notes" repository.
• Filed a build-failure bug against the muroar package that was discovered while doing reproducibility testing. (#968189)
• We operate a large and many-featured Jenkins-based testing framework that powers tests.reproducible-builds.org. This month, I updated the self-serve package rescheduler to use HTML <pre> tags when dumping any debugging data. [...]

• Updated the main Reproducible Builds website and documentation to:
  • Clarify & fix a few entries on the "who" page [...][...] and ensure that images do not get to large on some viewports [...].
  • Clarify use of a pronoun re. Conservancy. [...]
  • Use "View all our monthly reports" over "View all monthly reports". [...]
  • Move a "is a" suffix out of the link target on the SOURCE_DATE_EPOCH age. [...]
• Drafted, published and publicised our monthly report.

diffoscope I made the following changes to diffoscope, including preparing and uploading versions 155, 156, 157 and 158 to Debian:

• New features:
  • Support extracting data of PGP signed data. (#214)
  • Try files named .pgp against pgpdump(1) to determine whether they are Pretty Good Privacy (PGP) files. (#211)
  • Support multiple options for all file extension matching. [...]
• Bug fixes:
  • Don't raise an exception when we encounter XML files with <!ENTITY> declarations inside the Document Type Definition (DTD), or when a DTD or entity references an external resource. (#212)
  • pgpdump(1) can successfully parse some binary files, so check that the parsed output contains something sensible before accepting it. [...]
  • Temporarily drop gnumeric from the Debian build-dependies as it has been removed from the testing distribution. (#968742)
  • Correctly use fallback_recognises to prevent matching .xsb binary XML files.
  • Correct identify signed PGP files as file(1) returns "data". (#211)
• Logging improvements:
  • Emit a message when ppudump version does not match our file header. [...]
  • Don't use Python's repr(object) output in "Calling external command" messages. [...]
  • Include the filename in the "... not identified by any comparator" message. [...]
• Codebase improvements:
  • Bump Python requirement from 3.6 to 3.7. Most distributions are either shipping with Python 3.5 or 3.7, so supporting 3.6 is not only somewhat unnecessary but also cumbersome to test locally. [...]
  • Drop some unused imports [...], drop an unnecessary dictionary comprehensions [...] and some unnecessary control flow [...].
  • Correct typo of "output" in a comment. [...]
• Release process:
  • Move generation of debian/tests/control to an external script. [...]
  • Add some URLs for the site that will appear on PyPI.org. [...]
  • Update "author" and "author email" in setup.py for PyPI.org and similar. [...]
• Testsuite improvements:
  • Update PPU tests for compatibility with Free Pascal versions 3.2.0 or greater. (#968124) [...][...][...]
  • Mark that our identification test for .ppu files requires ppudump version 3.2.0 or higher. [...]
  • Add an assert_diff helper that loads and compares a fixture output. [...][...][...][...]
• Misc:
  • Duplicate docker instructions in the "Get diffoscope" section of the diffoscope website. [...]

Debian Debian LTS This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.

• Investigated and triaged chrony [...], golang-1.8 [...], golang-go.crypto [...], golang-golang-x-net-dev [...], icingaweb2 [...], lua5.3 [...], mongodb [...], net-snmp, php7.0 [...], qt4-x11, qtbase-opensource-src, ruby-actionpack-page-caching [...], ruby-doorkeeper [...], ruby-json-jwt [...], ruby-kaminari [...], ruby-kaminari [...], ruby-rack-cors [...], shiro [...] & squirrelmail.
• Frontdesk duties, responding to user/developer questions, reviewing others' packages, participating in mailing list discussions, attending the Debian LTS BoF at DebConf20 etc.
• Issued DLA 2313-1 and ELA-257-1 to fix a privilege escalation vulnerability in Net-SNMP.
• Issued ELA-263-1 for qtbase-opensource-src and ELA-261-1 for qt4-x11, two components of cross-platform C++ application framework. A specially-crafted XBM image file could have caused a buffer overread.
• Issued ELA-268-1 to address unsafe serialisation vulnerabilities that were discovered in the PHP-based squirrelmail webmail client.
• Issued DLA 2311-1 for zabbix, the PHP-based monitoring system to fix a potential cross-site scripting vulnerability via <iframe> HTML elements.
• Issued DLA 2334-1 to fix a denial of service vulnerability in ruby-websocket-extensions, a library for managing long-lived HTTP 'WebSocket' connections.
• Issued DLA 2345-1 for PHP 7.0 as it was discovered that there was a use-after-free vulnerability when parsing PHAR files, a method of putting entire PHP applications into a single file.
• I also updated the Extended LTS website to pluralise the "Related CVEs" text in announcement emails [...] and dropped some trailing whitespace [...].

You can find out more about the project via the following video:

Uploads to Debian

• memcached:
  • 1.6.6-2 Enable TLS capabilities by default. (#968603)
  • 1.6.6-3 Add libio-socket-ssl-perl to test TLS support and perform a general package refresh.
• python-django:
  • 2.2.15-1 (unstable) New upstream bugfix release
  • 3.1-1 (experimental) New upstream release.
  • 2.2.15-2 (unstable) & 3.1-2 (experimental) Set the same PYTHONPATH when executing the runtime tests as we do in the package build. (#968577)
• docbook-to-man:
  • 2.0.0-43 Refresh packaging, and upload some changes from the Debian Janitor.
  • 2.0.0-44 Fix compatibility with GCC 10, restoring the missing /usr/bin/instant binary. (#968900)
• hiredis (1.0.0-1) New upstream release to experimental.

On Saturday 29 August 2020, the annual Debian Developers and Contributors Conference came to a close. DebConf20 has been held online for the first time, due to the coronavirus (COVID-19) disease pandemic. All of the sessions have been streamed, with a variety of ways of participating: via IRC messaging, online collaborative text documents, and video conferencing meeting rooms. With more than 850 attendees from 80 different countries and a total of over 100 event talks, discussion sessions, Birds of a Feather (BoF) gatherings and other activities, DebConf20 was a large success. When it became clear that DebConf20 was going to be an online-only event, the DebConf video team spent much time over the next months to adapt, improve, and in some cases write from scratch, technology that would be required to make an online DebConf possible. After lessons learned from the MiniDebConfOnline in late May, some adjustments were made, and then eventually we came up with a setup involving Jitsi, OBS, Voctomix, SReview, nginx, Etherpad, and a newly written web-based frontend for voctomix as the various elements of the stack. All components of the video infrastructure are free software, and the whole setup is configured through their public ansible repository. The DebConf20 schedule included two tracks in other languages than English: the Spanish language MiniConf, with eight talks in two days, and the Malayalam language MiniConf, with nine talks in three days. Ad-hoc activities, introduced by attendees over the course of the entire conference, have been possible too, streamed and recorded. There have also been several team gatherings to sprint on certain Debian development areas. Between talks, the video stream has been showing the usual sponsors on the loop, but also some additional clips including photos from previous DebConfs, fun facts about Debian and short shout-out videos sent by attendees to communicate with their Debian friends. For those who were not able to participate, most of the talks and sessions are already available through the Debian meetings archive website, and the remaining ones will appear in the following days. The DebConf20 website will remain active for archival purposes and will continue to offer links to the presentations and videos of talks and events. Next year, DebConf21 is planned to be held in Haifa, Israel, in August or September. DebConf is committed to a safe and welcome environment for all participants. During the conference, several teams (Front Desk, Welcome team and Community team) have been available to help so participants get their best experience in the conference, and find solutions to any issue that may arise. See the web page about the Code of Conduct in DebConf20 website for more details on this. Debian thanks the commitment of numerous sponsors to support DebConf20, particularly our Platinum Sponsors: Lenovo, Infomaniak, Google and Amazon Web Services (AWS). About Debian The Debian Project was founded in 1993 by Ian Murdock to be a truly free community project. Since then the project has grown to be one of the largest and most influential open source projects. Thousands of volunteers from all over the world work together to create and maintain Debian software. Available in 70 languages, and supporting a huge range of computer types, Debian calls itself the universal operating system. About DebConf DebConf is the Debian Project's developer conference. In addition to a full schedule of technical, social and policy talks, DebConf provides an opportunity for developers, contributors and other interested people to meet in person and work together more closely. It has taken place annually since 2000 in locations as varied as Scotland, Argentina, and Bosnia and Herzegovina. More information about DebConf is available from https://debconf.org/. About Lenovo As a global technology leader manufacturing a wide portfolio of connected products, including smartphones, tablets, PCs and workstations as well as AR/VR devices, smart home/office and data center solutions, Lenovo understands how critical open systems and platforms are to a connected world. About Infomaniak Infomaniak is Switzerland's largest web-hosting company, also offering backup and storage services, solutions for event organizers, live-streaming and video on demand services. It wholly owns its datacenters and all elements critical to the functioning of the services and products provided by the company (both software and hardware). About Google Google is one of the largest technology companies in the world, providing a wide range of Internet-related services and products such as online advertising technologies, search, cloud computing, software, and hardware. Google has been supporting Debian by sponsoring DebConf for more than ten years, and is also a Debian partner sponsoring parts of Salsa's continuous integration infrastructure within Google Cloud Platform. About Amazon Web Services (AWS) Amazon Web Services (AWS) is one of the world's most comprehensive and broadly adopted cloud platforms, offering over 175 fully featured services from data centers globally (in 77 Availability Zones within 24 geographic regions). AWS customers include the fastest-growing startups, largest enterprises and leading government agencies. Contact Information For further information, please visit the DebConf20 web page at https://debconf20.debconf.org/ or send mail to press@debian.org.

DebConf20 is taking place online, from 23 August to 29 August 2020. It is the 21st Debian conference, and organizers and participants are working hard together at creating interesting and fruitful events. We would like to warmly welcome the 17 sponsors of DebConf20, and introduce them to you. We have four Platinum sponsors. Our first Platinum sponsor is Lenovo. As a global technology leader manufacturing a wide portfolio of connected products, including smartphones, tablets, PCs and workstations as well as AR/VR devices, smart home/office and data center solutions, Lenovo understands how critical open systems and platforms are to a connected world. Our next Platinum sponsor is Infomaniak. Infomaniak is Switzerland's largest web-hosting company, also offering backup and storage services, solutions for event organizers, live-streaming and video on demand services. It wholly owns its datacenters and all elements critical to the functioning of the services and products provided by the company (both software and hardware). Google is our third Platinum sponsor. Google is one of the largest technology companies in the world, providing a wide range of Internet-related services and products such as online advertising technologies, search, cloud computing, software, and hardware. Google has been supporting Debian by sponsoring DebConf for more than ten years, and is also a Debian partner. Amazon Web Services (AWS) is our fourth Platinum sponsor. Amazon Web Services is one of the world's most comprehensive and broadly adopted cloud platforms, offering over 175 fully featured services from data centers globally (in 77 Availability Zones within 24 geographic regions). AWS customers include the fastest-growing startups, largest enterprises and leading government agencies. Our Gold sponsors are Deepin, the Matanel Foundation, Collabora, and HRT. Deepin is a Chinese commercial company focusing on the development and service of Linux-based operating systems. They also lead research and development of the Deepin Debian derivative. The Matanel Foundation operates in Israel, as its first concern is to preserve the cohesion of a society and a nation plagued by divisions. The Matanel Foundation also works in Europe, in Africa and in South America. Collabora is a global consultancy delivering Open Source software solutions to the commercial world. In addition to offering solutions to clients, Collabora's engineers and developers actively contribute to many Open Source projects. Hudson-Trading is a company led by mathematicians, computer scientists, statisticians, physicists and engineers. They research and develop automated trading algorithms using advanced mathematical techniques. Our Silver sponsors are: Linux Professional Institute, the global certification standard and career support organization for open source professionals, Civil Infrastructure Platform, a collaborative project hosted by the Linux Foundation, establishing an open source base layer of industrial grade software, Ubuntu, the Operating System delivered by Canonical, and Roche, a major international pharmaceutical provider and research company dedicated to personalized healthcare. Bronze sponsors: IBM, MySQL, Univention. And finally, our Supporter level sponsors, ISG.EE and Pengwin. Thanks to all our sponsors for their support! Their contributions make it possible for a large number of Debian contributors from all over the globe to work together, help and learn from each other in DebConf20. Participating in DebConf20 online The 21st Debian Conference is being held online, due to COVID-19, from August 23 to 29, 2020. Talks, discussions, panels and other activities run from 10:00 to 01:00 UTC. Visit the DebConf20 website at https://debconf20.debconf.org to learn about the complete schedule, watch the live streaming and join the different communication channels for participating in the conference.

Like each month, albeit a bit later due to vacation, here comes a report about the work of paid contributors to Debian LTS. Individual reports In July, 249.25 work hours have been dispatched among 14 paid contributors. Their reports are available:

• Abhijith PA did 18.0h (out of 14h assigned and 6h from June), and gave back 2h to the pool.
• Adrian Bunk did 16.0h (out of 25.25h assigned), thus carrying over 9.25h to August.
• Ben Hutchings did 5h (out of 20h assigned), and gave back the remaining 15h.
• Brian May did 10h (out of 10h assigned).
• Chris Lamb did 18h (out of 18h assigned).
• Emilio Pozuelo Monfort did 60h (out of 5.75h assigned and 54.25h from June).
• Holger Levsen spent 10h (out of 10h assigned) for managing LTS and ELTS contributors.
• Markus Koschany did 15h (out of 25.25h assigned), thus carrying over 10.25h to August.
• Mike Gabriel did nothing (out of 8h assigned), thus is carrying over 8h for August.
• Ola Lundqvist did 3h (out of 12h assigned and 7h from June), thus carrying over 16h to August.
• Roberto C. S nchez did 26.5h (out of 25.25h assigned and 1.25h from June).
• Sylvain Beucler did 25.25h (out of 25.25h assigned).
• Thorsten Alteholz did 25.25h (out of 25.25h assigned).
• Utkarsh Gupta did 25.25h (out of 25.25h assigned).

Evolution of the situation July was our first month of Stretch LTS! Given this is our fourth LTS release we anticipated a smooth transition and it seems everything indeed went very well. Many thanks to the members of the Debian ftpmaster-, security, release- and publicity- teams who helped us make this happen!
Stretch LTS begun on July 18th 2020 after the 13th and final Stretch point release. and is currently scheduled to end on June 30th 2022. Last month, we asked you to participate in a survey and we got 1764 submissions, which is pretty awesome. Thank you very much for participating!. Right now we are still busy crunching the results, but we already shared some early analysis during the Debconf LTS bof this week. The security tracker currently lists 54 packages with a known CVE and the dla-needed.txt file has 52 packages needing an update. Thanks to our sponsors New sponsors are in bold.

• Platinum sponsors:
• TOSHIBA (for 59 months)
• GitHub (for 50 months)
• Civil Infrastructure Platform (CIP) (for 27 months)
• Gold sponsors:
• Blablacar (for 74 months)
• Roche Diagnostics International AG (for 70 months)
• Linode (for 64 months)
• Babiel GmbH (for 53 months)
• Plat Home (for 53 months)
• University of Oxford (for 9 months)
• Silver sponsors:
• The Positive Internet Company (for 75 months)
• Domeneshop AS (for 74 months)
• Nantes M tropole (for 68 months)
• Univention GmbH (for 60 months)
• Universit Jean Monnet de St Etienne (for 60 months)
• Ribbon Communications, Inc. (for 54 months)
• Exonet B.V. (for 44 months)
• Leibniz Rechenzentrum (for 38 months)
• CINECA (for 27 months)
• Minist re de l Europe et des Affaires trang res (for 21 months)
• Cloudways Ltd (for 11 months)
• Dinahosting SL (for 9 months)
• Bauer Xcel Media Deutschland KG (for 3 months)
• Platform.sh (for 3 months)
• Bronze sponsors:
• Evolix (for 75 months)
• Seznam.cz, a.s. (for 75 months)
• Linuxhotel GmbH (for 72 months)
• Intevation GmbH (for 71 months)
• Daevel SARL (for 70 months)
• Bitfolk LTD (for 69 months)
• Megaspace Internet Services GmbH (for 69 months)
• Greenbone Networks GmbH (for 68 months)
• NUMLOG (for 68 months)
• WinGo AG (for 68 months)
• Ecole Centrale de Nantes LHEEA (for 64 months)
• Entr ouvert (for 59 months)
• Adfinis AG (for 56 months)
• GNI MEDIA (for 51 months)
• Laboratoire LEGI UMR 5519 / CNRS (for 51 months)
• Tesorion (for 51 months)
• Bearstech (for 42 months)
• LiHAS (for 42 months)
• People Doc (for 39 months)
• Catalyst IT Ltd (for 37 months)
• Supagro (for 32 months)
• Demarcq SAS (for 31 months)
• Universit Grenoble Alpes (for 17 months)
• TouchWeb SAS (for 9 months)
• SPiN AG (for 6 months)
• CoreFiling

No comment Liked this article? Click here. My blog is Flattr-enabled.

Welcome to the 29th post in the randomly repeating R recommendations series or R⁴ for short. Our last post #28 introduced RSPM, and just before that we also talked in #27 about binary installations on Ubuntu (which was also a T⁴ video). This post is joined with I aki Ucar and mainly about work we have done with his bspm package.

Background CRAN has been a cornerstone of the success of R in recent years. As a well-maintained repository with stringent quality control, it ensures users have access to highest-quality statistical / analytical software that just works . Users on Windows and macOS also benefit from faster installation via pre-compiled binary packages. Linux users generally install from source, which can be more tedious and, often, much slower. Those who know where to look have had access to (at least some) binaries for years as well (and one of us blogged and vlogged about this at length). Debian users get close to 1000 CRAN and BioConductor packages (and, true to Debian form, for well over a dozen hardware platforms). Michael Rutter maintains a PPA with 4600 binaries for three different Ubuntu flavors (see c2d4u4.0+). More recently, Fedora joined the party with 16000 (!!) binaries, essentially all of CRAN, via a Copr repository (see iucar/cran). The buzz currently is however with RSPM, a new package manager by RStudio. An audacious project, it provides binaries for several Linux distributions and releases. It has already been tested in many RStudio Cloud sessions (including with some of our students) as well as some CI integrations. RSPM cuts across and takes the breadth of CRAN across several Linux distributions, bringing installation of pre-built CRAN packages a binaries under their normal CRAN package names. Another nice touch is the integration with install.packages(): these binaries are installed in a way that is natural for R users but as binaries. It is however entirely disconnected from the system package management. This means that the installation of a package requiring an external library may succeed and still fail, as a required library simply cannot be pulled in directly by RSPM. So what is needed is a combination. We want binaries that are aware of their system dependencies but accessible directly from R just like RSPM offers it. Enter BSPM the Bridge to System Package Manager package (also on CRAN). The first illustration (using Ubuntu 18.04) shows RSPM on the left, and BSPM on the right, both installing the graphics package Cairo (and both using custom Rocker containers). This fails for RSPM as no binary is present and a source build fails for the familiar lack of a -dev package. It proceeds just fine on the right under BSPM. A second illustration shows once again RSPM on the left, and BSPM on the right (this time on Fedora), both installing the units package without a required system dependency. The installation of units works for BSPM as the dependency libudunits is brought in, but fails under RSPM. The binary installation succeeds in both cases, but the missing dependency (the UDUNITS2 library) is brought in only by BSPM. Consequently, the package fails to load under RSPM.

Summary To conclude, highlights of BSPM are:

• direct installation of binary packages from R via R commands under their normal CRAN names (just like RSPM);
• full integration with the system package manager, delivering system installations (improving upon RSPM);
• full dependency resolution for R packages, including system requirements (improving upon RSPM).

This offers easy, reliable, fast installation of R packages, and we invite you to pick all three. We recommend usage with either Ubuntu with the 4.6k packages via the Rutter PPA, or Fedora via the even more complete Copr repository (which already includes a specially-tailored version of BSPM called CoprManager). We hope this short note wets your appetite to learn more about bspm (which is itself on CRAN) and the two sets of Rocker containers shown. The rocker/r-rspm container comes in two two flavours for Ubuntu 18.04 and 20.04. Similarly, the rocker/r-bspm container comes in the same two two flavours for Ubuntu 18.04 and 20.04, as well as in a Debian testing variant. Feedback is appreciated at the bspm or rocker issue trackers. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Shortly after yesterday s start of the Debian Conference 2020, I had the honor to participate in a BoF on social equality in free software, led by the OSI vice president and head of the FOSSASIA community, Hong Phuc Dang. The group of discussants consisted of OSS representatives from a wide variety of countries (India, Indonesia, China, Hong Kong, Germany, Vietnam, Singapore, Japan). After a short introduction by Hong Phuc we turned to a self-introduction and what is equality for me round. This brought up already a wide variety of issues that need to be addressed if we want to counter inequality in free software (culture differences, language barriers, internet connection, access to services, onboarding difficulties, political restrictions, ). Unfortunately, on-air time was rather restricted, but even after the DebConf related streaming time slot was finished, we continued discussing problems and possible approaches for another two hours. We have agreed to continue our collaboration and meetings in the hope that we, in particular the FOSSASIA community, can support those in need to counter inequality. Concluding, I have to say I am very happy to be part of the FOSSASIA community where real diversity is lived and everyone strives for and tries to increase social equality. In the DebConf IRC chat I was asked why at FOSSASIA we have about a 50:50 quote between women and men, in contrast to the usual 10:90 predominant in most software communities including Debian. For me this boils down to many reasons, one being competent female leadership, Hong Phuc is inspiring and competent to a degree I haven t seen in anyone else. Another reason is of course that software development is, especially in developing countries, one of the few escape pods for any gender, and thus fully embraced by normally underrepresented groups. Finally, but this is a typical chicken-egg problem, the FOSSASIA community is not doing any specific gender politics, but simply remains open and friendly to everyone. I think Debian, and in particular the diversity movement in Debian can learn a lot from the FOSSASIA community. At the end we are all striving for more equality in our projects and in the realm of free software as a whole! Thanks again for all the participants for the very inspiring discussion, and I am looking forward to our next meetings!

As client certificates are on the way out and Debian's SSO solution is effectively not maintained any longer, I switched self-service buildd givebacks over to Salsa authentication. It lives again at https://buildd.debian.org/auth/giveback.cgi. For authorization you still need to be in the "debian" group for now, i.e. be a regular Debian member.For convenience the package status web interface now features an additional column "Actions" with generated "giveback" links.Please remember to file bugs if you give builds back because of flakiness of the package rather than the infrastructure and resist the temptation to use this excessively to let your package migrate. We do not want to end up with packages that require multiple givebacks to actually build in stable, as that would hold up both security and stable updates needlessly and complicate development.

Such an obvious problem, convert a piece of html/js/css, often with animations, to a video (mp4 or similar). We were just put before this problem for the TUG 2020 online conference. Searching the internet it turned up mostly web services, some of them even with lots of money to pay. At the end (below I will give a short history) it turned out to be rather simple.

The key is to use timesnap, a tool to take screenshots from web pages. It is actively maintained, and internally uses puppeteer, which in turn uses Google Chrome browser headless. This also means that rendering quality is very high. So having an html file available, with all the necessary assets, either online or local, one simply creates enough single screenshots per second so that they can be assembled later on into a video with ffmpeg. In our case, we wanted our leaders to last 10secs before the actual presentation video starts. I decided to render at 30fps, which left me with the simple invocation: followed by conversion of the various png images to an mp4: The -r is the fps, so needs to agree with the --fps above. Also the --viewport and -s values should better agree. -crf is the video quality, and -pix_fmt the pixel format. With that very simple and quick invocation a nice leader video was ready! History It was actually more complicated than normal. For similar problems, it usually takes me about 5min of googling and a bit of scripting, but this time, it was actually a long way. Simply searching for convert html to mp4 doesn t give a lot but web services, often paid for. At some point I came up with the idea to use Electron and led to Electron Recorder, which looked promising, but didn t work. A bit more searching led me to PhantomJS, which is not developed anymore, but there was some explanation how to dump frames using phantomjs and merge them using ffmpeg, very similar to the above. Unfortunately, the rendering of the html page by phantomjs was broken, and thus not usable. Thus I ventured off into searching for alternatives of PhantomJS, which brought me to puppeteer, and from there it wasn t too long a way that pointed me at timesnap. Till now it is surprising to me that such a basic task is neither well documented, so hopefully this page helps some users.

After quite some time maintaining Qt in Debian both Dmitry Shachnev and I decided to not maintain Qt 6 when it's published (expected in December 2020, see https://wiki.qt.io/Qt_6.0_Release). We will do our best to keep the Qt 5 codebase up and running.

We **love** Qt, but it's a huge codebase and requires time and build power, both things that we are currently lacking, so we decided it's time for us to step down and pass the torch. And a new major version seems the right point to do that.

We will be happy to review and/or sponsor other people's work or even occasionally do uploads, but we can't promise to do it regularly.

Some things we think potential Qt 6 maintainers should be familiar with are, of course, C++ packaging (specially symbols files) and CMake, as Qt 6 will be built with it.

We also encourage prospective maintainers to remove the source's -everywhere-src suffixes and just keep the base names as source package names: qtbase6, qtdeclarative6, etc.

It has been an interesting ride all these years, we really hope you enjoyed using Qt.

Thanks for everything,

Dmitry and Lisandro.Note 20200818 12:12 ARST: I was asked if the move has anything to do with code quality or licensing. The answer is a huge no, Qt is a **great** project which we love. As stated before it's mostly about lack of free time to properly maintain it.

Hooks are an extension feature provided by all package managers that are used in larger Linux distributions. For example, Debian uses apt, which has various maintainer scripts. Fedora uses rpm, which has scriptlets. Different package managers use different names for the concept, but all of them offer package maintainers the ability to run arbitrary code during package installation and upgrades. Example hook use cases include adding daemon user accounts to your system (e.g. postgres), or generating/updating cache files. Triggers are a kind of hook which run when other packages are installed. For example, on Debian, the man(1) package comes with a trigger which regenerates the search database index whenever any package installs a manpage. When, for example, the nginx(8) package is installed, a trigger provided by the man(1) package runs. Over the past few decades, Open Source software has become more and more uniform: instead of each piece of software defining its own rules, a small number of build systems are now widely adopted. Hence, I think it makes sense to revisit whether offering extension via hooks and triggers is a net win or net loss.

Hooks preclude concurrent package installation Package managers commonly can make very little assumptions about what hooks do, what preconditions they require, and which conflicts might be caused by running multiple package s hooks concurrently. Hence, package managers cannot concurrently install packages. At least the hook/trigger part of the installation needs to happen in sequence. While it seems technically feasible to retrofit package manager hooks with concurrency primitives such as locks for mutual exclusion between different hook processes, the required overhaul of all hooks seems like such a daunting task that it might be better to just get rid of the hooks instead. Only deleting code frees you from the burden of maintenance, automated testing and debugging. In Debian, there are 8620 non-generated maintainer scripts, as reported by find shard*/src/*/debian -regex ".*\(pre\ post\)\(inst\ rm\)$" on a Debian Code Search instance.

Triggers slow down installing/updating other packages Personally, I never use the apropos(1) command, so I don t appreciate the man(1) package s trigger which updates the database used by apropos(1). The process takes a long time and, because hooks and triggers must be executed serially (see previous section), blocks my installation or update. When I tell people this, they are often surprised to learn about the existance of the apropos(1) command. I suggest adopting an opt-in model.

Unnecessary work if programs are not used between updates Hooks run when packages are installed. If a package s contents are not used between two updates, running the hook in the first update could have been skipped. Running the hook lazily when the package contents are used reduces unnecessary work. As a welcome side-effect, lazy hook evaluation automatically makes the hook work in operating system images, such as live USB thumb drives or SD card images for the Raspberry Pi. Such images must not ship the same crypto keys (e.g. OpenSSH host keys) to all machines, but instead generate a different key on each machine. Why do users keep packages installed they don t use? It s extra work to remember and clean up those packages after use. Plus, users might not realize or value that having fewer packages installed has benefits such as faster updates. I can also imagine that there are people for whom the cost of re-installing packages incentivizes them to just keep packages installed you never know when you might need the program again

Implemented in an interpreted language While working on hermetic packages (more on that in another blog post), where the contained programs are started with modified environment variables (e.g. PATH) via a wrapper bash script, I noticed that the overhead of those wrapper bash scripts quickly becomes significant. For example, when using the excellent magit interface for Git in Emacs, I encountered second-long delays when using hermetic packages compared to standard packages. Re-implementing wrappers in a compiled language provided a significant speed-up. Similarly, getting rid of an extension point which mandates using shell scripts allows us to build an efficient and fast implementation of a predefined set of primitives, where you can reason about their effects and interactions. magit needs to run git a few times for displaying the full status, so small overhead quickly adds up.

Incentivizing more upstream standardization Hooks are an escape hatch for distribution maintainers to express anything which their packaging system cannot express. Distributions should only rely on well-established interfaces such as autoconf s classic ./configure && make && make install (including commonly used flags) to build a distribution package. Integrating upstream software into a distribution should not require custom hooks. For example, instead of requiring a hook which updates a cache of schema files, the library used to interact with those files should transparently (re-)generate the cache or fall back to a slower code path. Distribution maintainers are hard to come by, so we should value their time. In particular, there is a 1:n relationship of packages to distribution package maintainers (software is typically available in multiple Linux distributions), so it makes sense to spend the work in the 1 and have the n benefit.

Can we do without them? If we want to get rid of hooks, we need another mechanism to achieve what we currently achieve with hooks. If the hook is not specific to the package, it can be moved to the package manager. The desired system state should either be derived from the package contents (e.g. required system users can be discovered from systemd service files) or declaratively specified in the package build instructions more on that in another blog post. This turns hooks (arbitrary code) into configuration, which allows the package manager to collapse and sequence the required state changes. E.g., when 5 packages are installed which each need a new system user, the package manager could update /etc/passwd just once. If the hook is specific to the package, it should be moved into the package contents. This typically means moving the functionality into the program start (or the systemd service file if we are talking about a daemon). If (while?) upstream is not convinced, you can either wrap the program or patch it. Note that this case is relatively rare: I have worked with hundreds of packages and the only package-specific functionality I came across was automatically generating host keys before starting OpenSSH s sshd(8) . There is one exception where moving the hook doesn t work: packages which modify state outside of the system, such as bootloaders or kernel images. Even that can be moved out of a package-specific hook, as Fedora demonstrates.

Conclusion Global state modifications performed as part of package installation today use hooks, an overly expressive extension mechanism. Instead, all modifications should be driven by configuration. This is feasible because there are only a few different kinds of desired state modifications. This makes it possible for package managers to optimize package installation.

In case you are not yet familiar with why an initramfs (or initrd, or initial ramdisk) is typically used when starting Linux, let me quote the wikipedia definition: [ ] initrd is a scheme for loading a temporary root file system into memory, which may be used as part of the Linux startup process [ ] to make preparations before the real root file system can be mounted. Many Linux distributions do not compile all file system drivers into the kernel, but instead load them on-demand from an initramfs, which saves memory. Another common scenario, in which an initramfs is required, is full-disk encryption: the disk must be unlocked from userspace, but since userspace is encrypted, an initramfs is used.

Motivation Thus far, building a distri disk image was quite slow: This is on an AMD Ryzen 3900X 12-core processor (2019):

distri % time make cryptimage serial=1
80.29s user 13.56s system 186% cpu 50.419 total # 19s image, 31s initrd

Of these 50 seconds, dracut s initramfs generation accounts for 31 seconds (62%)! Initramfs generation time drops to 8.7 seconds once dracut no longer needs to use the single-threaded gzip(1) , but the multi-threaded replacement pigz(1) : This brings the total time to build a distri disk image down to:

distri % time make cryptimage serial=1
76.85s user 13.23s system 327% cpu 27.509 total # 19s image, 8.7s initrd

Clearly, when you use dracut on any modern computer, you should make pigz available. dracut should fail to compile unless one explicitly opts into the known-slower gzip. For more thoughts on optional dependencies, see Optional dependencies don t work . But why does it take 8.7 seconds still? Can we go faster? The answer is Yes! I recently built a distri-specific initramfs I m calling minitrd. I wrote both big parts from scratch:

1. the initramfs generator program (distri initrd)
2. a custom Go userland (cmd/minitrd), running as /init in the initramfs.

minitrd generates the initramfs image in 400ms, bringing the total time down to:

distri % time make cryptimage serial=1
50.09s user 8.80s system 314% cpu 18.739 total # 18s image, 400ms initrd

(The remaining time is spent in preparing the file system, then installing and configuring the distri system, i.e. preparing a disk image you can run on real hardware.) How can minitrd be 20 times faster than dracut? dracut is mainly written in shell, with a C helper program. It drives the generation process by spawning lots of external dependencies (e.g. ldd or the dracut-install helper program). I assume that the combination of using an interpreted language (shell) that spawns lots of processes and precludes a concurrent architecture is to blame for the poor performance. minitrd is written in Go, with speed as a goal. It leverages concurrency and uses no external dependencies; everything happens within a single process (but with enough threads to saturate modern hardware). Measuring early boot time using qemu, I measured the dracut-generated initramfs taking 588ms to display the full disk encryption passphrase prompt, whereas minitrd took only 195ms. The rest of this article dives deeper into how minitrd works.

What does an initramfs do? Ultimately, the job of an initramfs is to make the root file system available and continue booting the system from there. Depending on the system setup, this involves the following 5 steps:

1. Load kernel modules to access the block devices with the root file system Depending on the system, the block devices with the root file system might already be present when the initramfs runs, or some kernel modules might need to be loaded first. On my Dell XPS 9360 laptop, the NVMe system disk is already present when the initramfs starts, whereas in qemu, we need to load the virtio_pci module, followed by the virtio_scsi module. How will our userland program know which kernel modules to load? Linux kernel modules declare patterns for their supported hardware as an alias, e.g.:

initrd# grep virtio_pci lib/modules/5.4.6/modules.alias
alias pci:v00001AF4d*sv*sd*bc*sc*i* virtio_pci

Devices in sysfs have a modalias file whose content can be matched against these declarations to identify the module to load:

initrd# cat /sys/devices/pci0000:00/*/modalias
pci:v00001AF4d00001005sv00001AF4sd00000004bc00scFFi00
pci:v00001AF4d00001004sv00001AF4sd00000008bc01sc00i00
[ ]

Hence, for the initial round of module loading, it is sufficient to locate all modalias files within sysfs and load the responsible modules. Loading a kernel module can result in new devices appearing. When that happens, the kernel sends a uevent, which the uevent consumer in userspace receives via a netlink socket. Typically, this consumer is udev(7) , but in our case, it s minitrd. For each uevent messages that comes with a MODALIAS variable, minitrd will load the relevant kernel module(s). When loading a kernel module, its dependencies need to be loaded first. Dependency information is stored in the modules.dep file in a Makefile-like syntax:

initrd# grep virtio_pci lib/modules/5.4.6/modules.dep
kernel/drivers/virtio/virtio_pci.ko: kernel/drivers/virtio/virtio_ring.ko kernel/drivers/virtio/virtio.ko

To load a module, we can open its file and then call the Linux-specific finit_module(2) system call. Some modules are expected to return an error code, e.g. ENODEV or ENOENT when some hardware device is not actually present. Side note: next to the textual versions, there are also binary versions of the modules.alias and modules.dep files. Presumably, those can be queried more quickly, but for simplicitly, I have not (yet?) implemented support in minitrd.

2. Console settings: font, keyboard layout Setting a legible font is necessary for hi-dpi displays. On my Dell XPS 9360 (3200 x 1800 QHD+ display), the following works well:

initrd# setfont latarcyrheb-sun32

Setting the user s keyboard layout is necessary for entering the LUKS full-disk encryption passphrase in their preferred keyboard layout. I use the NEO layout:

initrd# loadkeys neo

3. Block device identification In the Linux kernel, block device enumeration order is not necessarily the same on each boot. Even if it was deterministic, device order could still be changed when users modify their computer s device topology (e.g. connect a new disk to a formerly unused port). Hence, it is good style to refer to disks and their partitions with stable identifiers. This also applies to boot loader configuration, and so most distributions will set a kernel parameter such as root=UUID=1fa04de7-30a9-4183-93e9-1b0061567121. Identifying the block device or partition with the specified UUID is the initramfs s job. Depending on what the device contains, the UUID comes from a different place. For example, ext4 file systems have a UUID field in their file system superblock, whereas LUKS volumes have a UUID in their LUKS header. Canonically, probing a device to extract the UUID is done by libblkid from the util-linux package, but the logic can easily be re-implemented in other languages and changes rarely. minitrd comes with its own implementation to avoid cgo or running the blkid(8) program.

4. LUKS full-disk encryption unlocking (only on encrypted systems) Unlocking a LUKS-encrypted volume is done in userspace. The kernel handles the crypto, but reading the metadata, obtaining the passphrase (or e.g. key material from a file) and setting up the device mapper table entries are done in user space.

initrd# modprobe algif_skcipher
initrd# cryptsetup luksOpen /dev/sda4 cryptroot1

After the user entered their passphrase, the root file system can be mounted:

initrd# mount /dev/dm-0 /mnt

5. Continuing the boot process (switch_root) Now that everything is set up, we need to pass execution to the init program on the root file system with a careful sequence of chdir(2) , mount(2) , chroot(2) , chdir(2) and execve(2) system calls that is explained in this busybox switch_root comment.

initrd# mount -t devtmpfs dev /mnt/dev
initrd# exec switch_root -c /dev/console /mnt /init

To conserve RAM, the files in the temporary file system to which the initramfs archive is extracted are typically deleted.

How is an initramfs generated? An initramfs image (more accurately: archive) is a compressed cpio archive. Typically, gzip compression is used, but the kernel supports a bunch of different algorithms and distributions such as Ubuntu are switching to lz4. Generators typically prepare a temporary directory and feed it to the cpio(1) program. In minitrd, we read the files into memory and generate the cpio archive using the go-cpio package. We use the pgzip package for parallel gzip compression. The following files need to go into the cpio archive:

minitrd Go userland The minitrd binary is copied into the cpio archive as /init and will be run by the kernel after extracting the archive. Like the rest of distri, minitrd is built statically without cgo, which means it can be copied as-is into the cpio archive.

Linux kernel modules Aside from the modules.alias and modules.dep metadata files, the kernel modules themselves reside in e.g. /lib/modules/5.4.6/kernel and need to be copied into the cpio archive. Copying all modules results in a 80 MiB archive, so it is common to only copy modules that are relevant to the initramfs s features. This reduces archive size to 24 MiB. The filtering relies on hard-coded patterns and module names. For example, disk encryption related modules are all kernel modules underneath kernel/crypto, plus kernel/drivers/md/dm-crypt.ko. When generating a host-only initramfs (works on precisely the computer that generated it), some initramfs generators look at the currently loaded modules and just copy those.

Console Fonts and Keymaps The kbd package s setfont(8) and loadkeys(1) programs load console fonts and keymaps from /usr/share/consolefonts and /usr/share/keymaps, respectively. Hence, these directories need to be copied into the cpio archive. Depending on whether the initramfs should be generic (work on many computers) or host-only (works on precisely the computer/settings that generated it), the entire directories are copied, or only the required font/keymap.

cryptsetup, setfont, loadkeys These programs are (currently) required because minitrd does not implement their functionality. As they are dynamically linked, not only the programs themselves need to be copied, but also the ELF dynamic linking loader (path stored in the .interp ELF section) and any ELF library dependencies. For example, cryptsetup in distri declares the ELF interpreter /ro/glibc-amd64-2.27-3/out/lib/ld-linux-x86-64.so.2 and declares dependencies on shared libraries libcryptsetup.so.12, libblkid.so.1 and others. Luckily, in distri, packages contain a lib subdirectory containing symbolic links to the resolved shared library paths (hermetic packaging), so it is sufficient to mirror the lib directory into the cpio archive, recursing into shared library dependencies of shared libraries. cryptsetup also requires the GCC runtime library libgcc_s.so.1 to be present at runtime, and will abort with an error message about not being able to call pthread_cancel(3) if it is unavailable.

time zone data To print log messages in the correct time zone, we copy /etc/localtime from the host into the cpio archive.

minitrd outside of distri? I currently have no desire to make minitrd available outside of distri. While the technical challenges (such as extending the generator to not rely on distri s hermetic packages) are surmountable, I don t want to support people s initramfs remotely. Also, I think that people s efforts should in general be spent on rallying behind dracut and making it work faster, thereby benefiting all Linux distributions that use dracut (increasingly more). With minitrd, I have demonstrated that significant speed-ups are achievable.

Conclusion It was interesting to dive into how an initramfs really works. I had been working with the concept for many years, from small tasks such as debug why the encrypted root file system is not unlocked to more complicated tasks such as set up a root file system on DRBD for a high-availability setup . But even with that sort of experience, I didn t know all the details, until I was forced to implement every little thing. As I suspected going into this exercise, dracut is much slower than it needs to be. Re-implementing its generation stage in a modern language instead of shell helps a lot. Of course, my minitrd does a bit less than dracut, but not drastically so. The overall architecture is the same. I hope my effort helps with two things:

1. As a teaching implementation: instead of wading through the various components that make up a modern initramfs (udev, systemd, various shell scripts, ), people can learn about how an initramfs works in a single place.
2. I hope the significant time difference motivates people to improve dracut.

Appendix: qemu development environment Before writing any Go code, I did some manual prototyping. Learning how other people prototype is often immensely useful to me, so I m sharing my notes here. First, I copied all kernel modules and a statically built busybox binary:

% mkdir -p lib/modules/5.4.6
% cp -Lr /ro/lib/modules/5.4.6/* lib/modules/5.4.6/
% cp ~/busybox-1.22.0-amd64/busybox sh

To generate an initramfs from the current directory, I used:

% find .   cpio -o -H newc   pigz > /tmp/initrd

In distri s Makefile, I append these flags to the QEMU invocation:

-kernel /tmp/kernel \
-initrd /tmp/initrd \
-append "root=/dev/mapper/cryptroot1 rdinit=/sh ro console=ttyS0,115200 rd.luks=1 rd.luks.uuid=63051f8a-54b9-4996-b94f-3cf105af2900 rd.luks.name=63051f8a-54b9-4996-b94f-3cf105af2900=cryptroot1 rd.vconsole.keymap=neo rd.vconsole.font=latarcyrheb-sun32 init=/init systemd.setenv=PATH=/bin rw vga=836"

The vga= mode parameter is required for loading font latarcyrheb-sun32. Once in the busybox shell, I manually prepared the required mount points and kernel modules:

ln -s sh mount
ln -s sh lsmod
mkdir /proc /sys /run /mnt
mount -t proc proc /proc
mount -t sysfs sys /sys
mount -t devtmpfs dev /dev
modprobe virtio_pci
modprobe virtio_scsi

As a next step, I copied cryptsetup and dependencies into the initramfs directory:

% for f in /ro/cryptsetup-amd64-2.0.4-6/lib/*; do full=$(readlink -f $f); rel=$(echo $full   sed 's,^/,,g'); mkdir -p $(dirname $rel); install $full $rel; done
% ln -s ld-2.27.so ro/glibc-amd64-2.27-3/out/lib/ld-linux-x86-64.so.2
% cp /ro/glibc-amd64-2.27-3/out/lib/ld-2.27.so ro/glibc-amd64-2.27-3/out/lib/ld-2.27.so
% cp -r /ro/cryptsetup-amd64-2.0.4-6/lib ro/cryptsetup-amd64-2.0.4-6/
% mkdir -p ro/gcc-libs-amd64-8.2.0-3/out/lib64/
% cp /ro/gcc-libs-amd64-8.2.0-3/out/lib64/libgcc_s.so.1 ro/gcc-libs-amd64-8.2.0-3/out/lib64/libgcc_s.so.1
% ln -s /ro/gcc-libs-amd64-8.2.0-3/out/lib64/libgcc_s.so.1 ro/cryptsetup-amd64-2.0.4-6/lib
% cp -r /ro/lvm2-amd64-2.03.00-6/lib ro/lvm2-amd64-2.03.00-6/

In busybox, I used the following commands to unlock the root file system:

modprobe algif_skcipher
./cryptsetup luksOpen /dev/sda4 cryptroot1
mount /dev/dm-0 /mnt

Welcome to the July 2020 report from the Reproducible Builds project. In these monthly reports, we round-up the things that we have been up to over the past month. As a brief refresher, the motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced from the original free software source code to the pre-compiled binaries we install on our systems. (If you re interested in contributing to the project, please visit our main website.)

General news At the upcoming DebConf20 conference (now being held online), Holger Levsen will present a talk on Thursday 27th August about Reproducing Bullseye in practice , focusing on independently verifying that the binaries distributed from ftp.debian.org were made from their claimed sources. Tavis Ormandy published a blog post making the provocative claim that You don t need reproducible builds , asserting elsewhere that the many attacks that have been extensively reported in our previous reports are fantasy threat models . A number of rebuttals have been made, including one from long-time contributor Reproducible Builds contributor Bernhard Wiedemann. On our mailing list this month, Debian Developer Graham Inggs posted to our list asking for ideas why the openorienteering-mapper Debian package was failing to build on the Reproducible Builds testing framework. Chris Lamb remarked from the build logs that the package may be missing a build dependency, although Graham then used our own diffoscope tool to show that the resulting package remains unchanged with or without it. Later, Nico Tyni noticed that the build failure may be due to the relationship between the FILE C preprocessor macro and the -ffile-prefix-map GCC flag. An issue in Zephyr, a small-footprint kernel designed for use on resource-constrained systems, around .a library files not being reproducible was closed after it was noticed that a key part of their toolchain was updated that now calls --enable-deterministic-archives by default. Reproducible Builds developer kpcyrd commented on a pull request against the libsodium cryptographic library wrapper for Rust, arguing against the testing of CPU features at compile-time. He noted that:

I ve accidentally shipped broken updates to users in the past because the build system was feature-tested and the final binary assumed the instructions would be present without further runtime checks

David Kleuker also asked a question on our mailing list about using SOURCE_DATE_EPOCH with the install(1) tool from GNU coreutils. When comparing two installed packages he noticed that the filesystem birth times differed between them. Chris Lamb replied, realising that this was actually a consequence of using an outdated version of diffoscope and that a fix was in diffoscope version 146 released in May 2020. Later in July, John Scott posted asking for clarification regarding on the Javascript files on our website to add metadata for LibreJS, the browser extension that blocks non-free Javascript scripts from executing. Chris Lamb investigated the issue and realised that we could drop a number of unused Javascript files [ ][ ][ ] and added unminified versions of Bootstrap and jQuery [ ].

Development work

Website On our website this month, Chris Lamb updated the main Reproducible Builds website and documentation to drop a number of unused Javascript files [ ][ ][ ] and added unminified versions of Bootstrap and jQuery [ ]. He also fixed a number of broken URLs [ ][ ]. Gonzalo Bulnes Guilpain made a large number of grammatical improvements [ ][ ][ ][ ][ ] as well as some misspellings, case and whitespace changes too [ ][ ][ ]. Lastly, Holger Levsen updated the README file [ ], marked the Alpine Linux continuous integration tests as currently disabled [ ] and linked the Arch Linux Reproducible Status page from our projects page [ ].

diffoscope diffoscope is our in-depth and content-aware diff utility that can not only locate and diagnose reproducibility issues, it provides human-readable diffs of all kinds. In July, Chris Lamb made the following changes to diffoscope, including releasing versions 150, 151, 152, 153 & 154:

• New features:
  • Add support for flash-optimised F2FS filesystems. (#207)
  • Don t require zipnote(1) to determine differences in a .zip file as we can use libarchive. [ ]
  • Allow --profile as a synonym for --profile=-, ie. write profiling data to standard output. [ ]
  • Increase the minimum length of the output of strings(1) to eight characters to avoid unnecessary diff noise. [ ]
  • Drop some legacy argument styles: --exclude-directory-metadata and --no-exclude-directory-metadata have been replaced with --exclude-directory-metadata= yes,no . [ ]
• Bug fixes:
  • Pass the absolute path when extracting members from SquashFS images as we run the command with working directory in a temporary directory. (#189)
  • Correct adding a comment when we cannot extract a filesystem due to missing libguestfs module. [ ]
  • Don t crash when listing entries in archives if they don t have a listed size such as hardlinks in ISO images. (#188)
• Output improvements:
  • Strip off the file offset prefix from xxd(1) and show bytes in groups of 4. [ ]
  • Don t emit javap not found in path if it is available in the path but it did not result in an actual difference. [ ]
  • Fix ... not available in path messages when looking for Java decompilers that used the Python class name instead of the command. [ ]
• Logging improvements:
  • Add a bit more debugging info when launching libguestfs. [ ]
  • Reduce the --debug log noise by truncating the has_some_content messages. [ ]
  • Fix the compare_files log message when the file does not have a literal name. [ ]
• Codebase improvements:
  • Rewrite and rename exit_if_paths_do_not_exist to not check files multiple times. [ ][ ]
  • Add an add_comment helper method; don t mess with our internal list directly. [ ]
  • Replace some simple usages of str.format with Python f-strings [ ] and make it easier to navigate to the main.py entry point [ ].
  • In the RData comparator, always explicitly return None in the failure case as we return a non-None value in the success one. [ ]
  • Tidy some imports [ ][ ][ ] and don t alias a variable when we do not use it. [ ]
  • Clarify the use of a separate NullChanges quasi-file to represent missing data in the Debian package comparator [ ] and clarify use of a null diff in order to remember an exit code. [ ]
• Other changes:
  • Profile the launch of libguestfs filesystems. [ ]
  • Clarify and correct our contributing info. [ ][ ][ ][ ][ ][ ]

Jean-Romain Garnier also made the following changes:

• Allow passing a file with a list of arguments via diffoscope @args.txt. (!62)
• Improve the output of side-by-side diffs by detecting added lines better. (!64)
• Remove offsets before instructions in objdump [ ][ ] and remove raw instructions from ELF tests [ ].

Other tools strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. It is used automatically in most Debian package builds. In July, Chris Lamb ensured that we did not install the internal handler documentation generated from Perl POD documents [ ] and fixed a trivial typo [ ]. Marc Herbert added a --verbose-level warning when the Archive::Cpio Perl module is missing. (!6) reprotest is our end-user tool to build same source code twice in widely differing environments and then checks the binaries produced by each build for any differences. This month, Vagrant Cascadian made a number of changes to support diffoscope version 153 which had removed the (deprecated) --exclude-directory-metadata and --no-exclude-directory-metadata command-line arguments, and updated the testing configuration to also test under Python version 3.8 [ ].

Distributions

Debian In June 2020, Timo R hling filed a wishlist bug against the debhelper build tool impacting the reproducibility status of hundreds of packages that use the CMake build system. This month however, Niels Thykier uploaded debhelper version 13.2 that passes the -DCMAKE_SKIP_RPATH=ON and -DBUILD_RPATH_USE_ORIGIN=ON arguments to CMake when using the (currently-experimental) Debhelper compatibility level 14. According to Niels, this change:

should fix some reproducibility issues, but may cause breakage if packages run binaries directly from the build directory.

34 reviews of Debian packages were added, 14 were updated and 20 were removed this month adding to our knowledge about identified issues. Chris Lamb added and categorised the nondeterministic_order_of_debhelper_snippets_added_by_dh_fortran_mod [ ] and gem2deb_install_mkmf_log [ ] toolchain issues. Lastly, Holger Levsen filed two more wishlist bugs against the debrebuild Debian package rebuilder tool [ ][ ].

openSUSE In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update. Bernhard also published the results of performing 12,235 verification builds of packages from openSUSE Leap version 15.2 and, as a result, created three pull requests against the openSUSE Build Result Compare Script [ ][ ][ ].

Other distributions In Arch Linux, there was a mass rebuild of old packages in an attempt to make them reproducible. This was performed because building with a previous release of the pacman package manager caused file ordering and size calculation issues when using the btrfs filesystem. A system was also implemented for Arch Linux packagers to receive notifications if/when their package becomes unreproducible, and packagers now have access to a dashboard where they can all see all their unreproducible packages (more info). Paul Spooren sent two versions of a patch for the OpenWrt embedded distribution for adding a build system revision to the packages manifest so that all external feeds can be rebuilt and verified. [ ][ ]

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of these patches, including:

• Bernhard M. Wiedemann:
  • afl (fix an incorrectly built manual page varied from kernel boot options)
  • brp-check-suse (sorting issue)
  • dnscrypt-proxy (sort the output of find(1))
  • graphviz (timezone issue, forwarded from Debian)
  • guile-gcrypt (parallelism)
  • insighttoolkit (prevent CPU detection, forwarded upstream
  • ipopt (parallelism issue and use https://tracker.debian.org/pkg/strip-nondeterminism)
  • jboss-logging-tools (date, forwarded upstream)
  • kismet (date)
  • lcov (date issue, already upstream)
  • multus (date issue, already upstream)
  • multus (date)
  • paperjam (date issue, forwarded upstream)
  • pspp (scrub testsuite.log)
  • python-PyNaCl (sort Python glob/readdir)
  • python-enaml (workaround an open upstream Python issue)
  • sac (omit creation time from .zip files)
  • sql-parser (sort, already upstream)
  • ugrep (CPU-related issue, already upstream)
  • ugrep (CPU-related issue)
  • unknown-horizons (filesystem ordering issue, already upstream)
  • unknown-horizons (filesystem ordering issue)
  • xfce4-panel-profiles (POSIX.1-2001/pax headers)
  • yast2-sound (uses uname -r)
• Chris Lamb:
  • dh-fortran-mod
  • flit
  • gem2deb
  • gmap
  • jskeus
  • libtpms
  • logilab-common
  • mrbayes
  • nmap
  • numpydoc
  • pyerfa
  • python-cooler
  • python-peachpy (forwarded upstream)
  • python-pyxs
  • sratom
  • weather-util
• Guillaume Nodet:
  • apache-sshd (date)

Vagrant Cascadian also reported two issues, the first regarding a regression in u-boot boot loader reproducibility for a particular target [ ] and a non-deterministic segmentation fault in the guile-ssh test suite [ ]. Lastly, Jelle van der Waa filed a bug against the MeiliSearch search API to report that it embeds the current build date.

Testing framework We operate a large and many-featured Jenkins-based testing framework that powers tests.reproducible-builds.org. This month, Holger Levsen made the following changes:

• Debian-related changes:
  • Tweak the rescheduling of various architecture and suite combinations. [ ][ ]
  • Fix links for 404 and not for us icons. (#959363)
  • Further work on a rebuilder prototype, for example correctly processing the sbuild exit code. [ ][ ]
  • Update the sudo configuration file to allow the node health job to work correctly. [ ]
  • Add php-horde packages back to the pkg-php-pear package set for the bullseye distribution. [ ]
  • Update the version of debrebuild. [ ]
• System health check development:
  • Add checks for broken SSH [ ], logrotate [ ], pbuilder [ ], NetBSD [ ], unkillable processes [ ], unresponsive nodes [ ][ ][ ][ ], proxy connection failures [ ], too many installed kernels [ ], etc.
  • Automatically fix some failed systemd units. [ ]
  • Add notes explaining all the issues that hosts are experiencing [ ] and handle zipped job log files correctly [ ].
  • Separate nodes which have been automatically marked as down [ ] and show status icons for jobs with issues [ ].
• Misc:
  • Disable all Alpine Linux jobs until they are or Alpine is fixed. [ ]
  • Perform some general upkeep of build nodes hosted by OSUOSL. [ ][ ][ ][ ]

In addition, Mattia Rizzolo updated the init_node script to suggest using sudo instead of explicit logout and logins [ ][ ] and the usual build node maintenance was performed by Holger Levsen [ ][ ][ ][ ][ ][ ], Mattia Rizzolo [ ][ ] and Vagrant Cascadian [ ][ ][ ][ ].

---

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Twitter: @ReproBuilds
• Mastodon: @reproducible_builds@fosstodon.org
• Reddit: /r/ReproducibleBuilds
• Mailing list: rb-general@lists.reproducible-builds.org

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor. In July, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 25.25h for LTS (out of 30 max; all done) and 13.25h for ELTS (out of 20 max; all done). We shifted suites: welcome Stretch LTS and Jessie ELTS. The LTS->ELTS switch happened at the start of the month, but the oldstable->LTS switch happened later (after finalizing and flushing proposed-updates to a last point release), causing some confusion but nothing major. ELTS - Jessie

• New local build setup
• ELTS buildds: request timezone harmonization
• Reclassify in-progress updates from jessie-LTS to jessie-ELTS
• python3.4: finish preparing update, security upload ELA 239-1
• net-snmp: global triage: bisect CVE-2019-20892 to identify affected version, jessie/stretch not-affected
• nginx: global triage: clarify CVE-2013-0337 status; locate CVE-2020-11724 original patch and regression tests, update MITRE
• nginx: security upload ELA-247-1 with 2 CVEs

LTS - Stretch

• Reclassify in-progress/needed updates from stretch/oldstable to stretch-LTS
• rails: upstream security: follow-up on CVE-2020-8163 (RCE) on upstream bug tracker and create pull request for 4.x (merged), hence getting some upstream review
• rails: global security: continue coordinating upload in multiple Debian versions, prepare fixes for common stretch/buster vulnerabilities in buster
• rails: security upload DLA-2282 fixing 3 CVEs
• python3.5: security upload DLA-2280-1 fixing 13 pending non-critical vulnerabilities, and its test suite
• nginx: security upload DLA-2283 (cf. common ELTS work)
• net-snmp: global triage (cf. common ELTS work)
• public IRC monthly team meeting
• reach out to clarify the intro from last month's report, following unsettled feedback during meeting

Documentation/Scripts

• ELTS/README.how-to-release-an-update: fix typo
• ELTS buildd: attempt to diagnose slow perfs, provide comparison with Debian and local builds
• LTS/Meetings: improve presentation
• SourceOnlyUpload: clarify/de-dup pbuilder doc
• LTS/Development: reference build logs URL, reference proposed-updates issue during dists switch, reference new-upstream-versioning discussion, multiple jessie->stretch fixes and clean-ups
• LTS/Development/Asan: drop wheezy documentation
• Warn about jruby mis-triage
• Provide feedback for ksh/CVE-2019-14868
• Provide feedback for condor update
• LTS/TestsSuites/nginx: test with new request smuggling test cases

Here is my monthly update covering what I have been doing in the free and open source software world during July 2020 (previous month):

• Opened a pull request to make the build reproducible in PyERFA, a set of Python bindings for various astronomy-related utilities (#45), as well as one for PeachPy assembler to make the output of codecode/x86_64.py reproducible (#108).

• As part of being on the board of directors of the Open Source Initiative and Software in the Public Interest I attended their respective monthly meetings and participated in various licensing and other discussions occurring on the internet, as well as the usual internal discussions regarding logistics and policy etc. This month, it was SPI's Annual General Meeting and the OSI has been running a number of remote strategy sessions for the board.
• Fixed an issue in my tickle-me-email library that implements Getting Things Done (GTD)-like behaviours in IMAP inboxes to ensure that all messages have a unique Message-Id header. [...]
• Reviewed and merged even more changes by Pavel Dolecek into my Strava Enhancement Suite, a Chrome extension to improve the user experience on the Strava athletic tracker.
• Updated travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub, to use the Travis CI continuous integration platform) to fix a compatibility issue with the latest version of mk-build-deps. [...][...]

For Lintian, the static analysis tool for Debian packages:

• Update the regular expression to search for all the released versions in a .changes file. [...]
• Avoid false-positives when matching sensible-utils utilities such as i3-sensible-pager. (#966022)
• Rename the send-patch tag to patch-not-forwarded-upstream. [...]
• Drop reminders from 26 tags that false-positives should be reported to Lintian as this is implicit in all our tags. [...]

Reproducible Builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. The project is proud to be a member project of the Software Freedom Conservancy. Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter. This month, I:

• Opened pull requests to make the build reproducible in pyerfa (#45) as well as one for PeachPy to make the output of codecode/x86_64.py reproducible (#108).
• In Debian:
  • Submitted the following patches to fix reproducibility-related toolchain issues within Debian:
    • dh-fortran-mod: Please make the output reproducible. (#965255)
    • gem2deb: should not install mkmf.log files. (#964772)
    • nmap: Please make the obfuscated nmap_service.exe file reproducible. (#964369)
  • I also submitted 13 patches to fix specific reproducibility issues in flit, gmap, jskeus, libtpms, logilab-common, mrbayes, numpydoc, pyerfa, python-cooler, python-peachpy, python-pyxs, sratom & weather-util.
  • When investigating packages, I discovered and reported that dolfinx was running its test suite even when the nocheck build profile was set. (#965946)
  • Kept isdebianreproducibleyet.com up to date. [...]
• Categorised a large number of packages and issues in the Reproducible Builds 'notes' repository.
• Drafted, published and publicised our monthly report.

• Updated the main Reproducible Builds website and documentation to drop a number of unused Javascript files [...][...][...] and added unminified versions of Bootstrap and jQuery [...] and fixed a number of broken URLs [...][...].
• strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. This month, I made sure that we did not install the internal handler documentation generated from Perl POD documents [...] and fixed a trivial typo [...].

diffoscope Elsewhere in our tooling, I made the following changes to diffoscope, including preparing and uploading versions 150, 151, 152, 153 & 154 to Debian:

• New features:
  • Add support for flash-optimised F2FS filesystems. (#207)
  • Don't require zipnote(1) to determine differences in a .zip file as we can use libarchive. [...]
  • Allow --profile as a synonym for --profile=-, ie. write profiling data to standard output. [...]
  • Increase the minimum length of the output of strings(1) to eight characters to avoid unnecessary diff noise. [...]
  • Drop some legacy argument styles: --exclude-directory-metadata and --no-exclude-directory-metadata have been replaced with --exclude-directory-metadata= yes,no . [...]
• Bug fixes:
  • Pass the absolute path when extracting members from SquashFS images as we run the command with working directory in a temporary directory. (#189)
  • Correct adding a comment when we cannot extract a filesystem due to missing libguestfs module. [...]
  • Don't crash when listing entries in archives if they don't have a listed size such as hardlinks in ISO images. (#188)
• Output improvements:
  • Strip off the file offset prefix from xxd(1) and show bytes in groups of 4. [...]
  • Don't emit javap not found in path if it is available in the path but it did not result in an actual difference. [...]
  • Fix ... not available in path messages when looking for Java decompilers that used the Python class name instead of the command. [...]
• Logging improvements:
  • Add a bit more debugging info when launching libguestfs. [...]
  • Reduce the --debug log noise by truncating the has_some_content messages. [...]
  • Fix the compare_files log message when the file does not have a literal name. [...]
• Codebase improvements:
  • Rewrite and rename exit_if_paths_do_not_exist to not check files multiple times. [...][...]
  • Add an add_comment helper method; don't mess with our internal list directly. [...]
  • Replace some simple usages of str.format with Python 'f-strings' [...] and make it easier to navigate to the main.py entry point [...].
  • In the RData comparator, always explicitly return None in the failure case as we return a non-None value in the success one. [...]
  • Tidy some imports [...][...][...] and don't alias a variable when we don't end up using. [...]
  • Clarify the use of a separate NullChanges quasi-file to represent missing data in the Debian package comparator [...] and clarify use of a 'null' diff in order to remember an exit code. [...]
• Misc:
  • Profile the launch of libguestfs filesystems. [...]
  • Clarify and correct our contributing info. [...][...][...][...][...][...]

Debian In Debian, I made the following uploads this month:

• Redis (6.0.6-1) New upstream release.
• Django:
  • 2.2.14-1 New upstream bugfix release.
  • 3.1~rc1-1 New upstream release candidate.
• lastpass-cli (1.3.3-4) Fix a build failure under GCC 10. (#957416)

Debian LTS This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 for the Extended LTS project. This included:

• Frontdesk duties, including responding to user/developer questions, reviewing packages, participating in mailing list discussions, offering comments on our survey and attended our IRC contributor meeting. I also made some changes to tools after the move from the Debian version supported [...].
• Investigated and triaged atril, ceph, cimg, ffmpeg, golang-github-seccomp-libseccomp-golang, gosa, http-parser, jruby, json-c, jupyter-notebook, keystone, ksh, libjpeg-turbo, libopenmpt, libpam-radius-auth, log4net, mailman, milkytracker, mupdf, pillow, poppler, puma, python-rtslib-fb, roundcube, ruby-zip, salt, slirp, sqlite3, transmission & wordpress.
• Issued DLA 2273-1 for shiro, first to fix a path-traversal issue where a specially-crafted request could cause an authentication bypass (CVE-2020-1957) and then to fix an encoding issue introduced in the handling of the previous CVE-2020-1957 path-traversal issue which itself could have also caused an authentication bypass (CVE-2020-11989).
• Issued DLA 2274-1 to fix a possible signature verification issue in the fwupd firmware update tool as the return value of gpgme_op_verify_result was not being checked.
• Issued DLA 2299-1 and ELA 252-1 in order to prevent a privilege escalation vulnerability discovered in Net-SNMP, a set of tools for collecting and organising information about devices on computer networks.
• Issued DLA 2300-1 as it was discovered that there was an issue where kdepim-runtime would default to using unencrypted POP3 communication despite the user interface indicating that encryption was in use.

You can find out more about the project via the following video:

This is part of a series of posts on compiling a custom version of Qt5 in order to develop for both amd64 and a Raspberry Pi. After having had some success with a sysroot in having a Qt5 cross-build environment that includes QtWebEngine, the next step is packaging the sysroot so it can be available both to build the cross-build environment, and to do cross-development with it. The result is this Debian source package which takes a Raspberry Pi OS disk image, provisions it in-place, extracts its contents, and packages them. Yes. You may want to reread the last paragraph. It works directly in the disk image to avoid a nasty filesystem issue on emulated 32bit Linux over a 64bit mounted filesystem. This feels like the most surreal Debian package I've ever created, and this saga looks like one of the hairiest yaks I've ever shaved. Integrating this monster codebase, full of bundled code and hacks, into a streamlined production and deployment system has been for me a full stack nightmare, and I have a renewed and growing respect for the people in the Qt/KDE team in Debian, who manage to stay on top of this mess, so that it all just works when we need it.

Review: Rise of the Warrior Cop, by Radley Balko

Publisher:	PublicAffairs
Copyright:	2013
ISBN:	1-61039-212-4
Format:	Kindle
Pages:	336

As the United States tries, in fits and starts, to have a meaningful discussion about long-standing police racism, brutality, overreach, corruption, and murder, I've realized that my theoretical understanding of the history of and alternative frameworks for law enforcement is woefully lacking. Starting with a book by a conservative white guy is not the most ideal of approaches, but it's what I already had on hand, and it won't be the last book I read and review on this topic. (Most of my research so far has been in podcast form. I don't review those here, but I can recommend Ezra Klein's interviews with Ta-Nehisi Coates, Paul Butler, and, most strongly, sujatha baliga.) Rise of the Warrior Cop is from 2013 and has had several moments of fame, no doubt helped by Balko's connections to the conservative and libertarian right. One of the frustrating facts of US politics is that critiques of the justice system from the right (and from white men) get more media attention than critiques from the left. That said, it's a generally well-respected book on the factual history of the topic, and police brutality and civil rights are among the points on which I have stopped-clock agreements with US libertarians. This book is very, very libertarian. In my callow youth, I was an ardent libertarian, so I've read a lot of US libertarian literature. It's a genre with its own conventions that become obvious when you read enough of it, and Rise of the Warrior Cop goes through them like a checklist. Use the Roman Republic (never the Roman Empire) as the starting point for any political discussion, check. Analyze the topic in the context of pre-revolutionary America, check. Spend considerable effort on discerning the opinions of the US founders on the topic since their opinions are always relevant to the modern world, check. Locate some point in the past (preferably before 1960) where the political issue was as good as it has ever been, check. Frame all changes since then as an erosion of rights through government overreach, check. Present your solution as a return to a previous era of respect for civil rights, check. Once you start recognizing the genre conventions, their prevalence in libertarian writing is almost comical. The framing chapters therefore leave a bit to be desired, but the meat of the book is a useful resource. Starting with the 1970s and its use as a campaigning tool by Nixon, Balko traces a useful history of the war on drugs. And starting with the 1980s, the number of cites to primary sources and the evidence of Balko's own research increases considerably. If you want to know how US police turned into military cosplayers with body armor, heavy weapons, and armored vehicles, this book provides a lot of context and history. One of the reasons why I view libertarians as allies of convenience on this specific issue is that drug legalization and disgust with the war on drugs have been libertarian issues for decades. Ideologically honest libertarians (and Balko appears to be one) are inherently skeptical of the police, so when the police overreach in an area of libertarian interest, they notice. Balko makes a solid argument, backed up with statistics, specific programs, legislation, and court cases, that the drug war and its accompanying lies about heavily-armed drug dealers and their supposed threat to police officers was the fuel for the growth of SWAT teams, no-knock search warrants, erosion of legal protections for criminal defendants, and de facto license for the police to ignore the scope and sometimes even the existence of warrants. This book is useful support for the argument that fears for the safety of officers underlying the militarization of police forces are imaginary. One telling point that Balko makes repeatedly and backs with statistical and anecdotal evidence is that the police generally do not use raid tactics on dangerous criminals. On the contrary, aggressive raids are more likely to be used on the least dangerous criminals because they're faster, they're fun for the police (they provide an adrenaline high and let them play with toys), and they're essentially risk-free. If the police believe someone is truly dangerous, they're more likely to use careful surveillance and to conduct a quiet arrest at an unexpected moment. The middle-of-the-night armed break-ins with battering rams, tear gas, and flash-bangs are, tellingly, used against the less dangerous suspects. This is part of Balko's overall argument that police equipment and tactics have become untethered from any realistic threat and have become cultural. He traces an acceleration of that trend to 9/11 and the resulting obsession with terrorism, which further opened the spigot of military hardware and "special forces" training. This became a point of competition between police departments, with small town forces that had never seen a terrorist and had almost no chance of a terrorist incident demanding their own armored vehicles. I've encountered this bizarre terrorism justification personally; one of the reasons my local police department gave in a public hearing for not having a policy against shooting at moving vehicles was "but what if terrorism?" I don't believe there has ever been a local terrorist attack. SWAT in such places didn't involve the special training or dedicated personnel of large city forces; instead, it was a part-time duty for normal police officers, and frequently they were encouraged to practice SWAT tactics by using them at random for some otherwise normal arrest or search. Balko argues that those raids were more exciting than normal police work, leading to a flood of volunteers for that duty and a tendency to use them as much as possible. That in turn normalizes disconnecting police tactics from the underlying crime or situational risk. So far, so good. But despite the information I was able to extract from it, I have mixed feelings about Rise of the Warrior Cop as a whole. At the least, it has substantial limitations. First, I don't trust the historical survey of policing in this book. Libertarian writing makes for bad history. The constraints of the genre require overusing only a few points of reference, treating every opinion of the US founders as holy writ, and tying forward progress to a return to a previous era, all of which interfere with good analysis. Balko also didn't do the research for the historical survey, as is clear from the footnotes. The citations are all to other people's histories, not to primary sources. He's summarizing other people's histories, and you'll almost certainly get better history by finding well-respected historians who cover the same ground. (That said, if you're not familiar with Peel's policing principles, this is a good introduction.) Second, and this too is unfortunately predictable in a libertarian treatment, race rarely appears in this book. If Balko published the same book today, I'm sure he would say more about race, but even in 2013 its absence is strange. I was struck while reading by how many examples of excessive police force were raids on west coast pot farms; yes, I'm sure that was traumatic, but it's not the demographic I would name as the most vulnerable to or affected by police brutality. West coast pot growers are, however, mostly white. I have no idea why Balko made that choice. Perhaps he thought his target audience would be more persuaded by his argument if he focused on white victims. Perhaps he thought it was an easier and less complicated story to tell. Perhaps, like a lot of libertarians, he doesn't believe racism has a significant impact on society because it would be a market failure. Perhaps those were the people who more readily came to mind. But to talk about police militarization, denial of civil rights, and police brutality in the United States without putting race at the center of both the history and the societal effects leaves a gaping hole in the analysis. Given that lack of engagement, I also am dubious of Balko's policy prescriptions. His reform suggestions aren't unreasonable, but they stay firmly in the centrist and incrementalist camp and would benefit white people more than black people. Transparency, accountability, and cultural changes are all fine and good, but the cultural change Balko is focused on is less aggressive arrest tactics, more use of mediation, and better physical fitness. I would not object to those things (well, maybe the last, which seemed odd), but we need to have a discussion about police white supremacist organizations, the prevalence of spousal abuse, and the police tendency to see themselves not as public servants but as embattled warriors who are misunderstood by the naive sheep they are defending. And, of course, you won't find in Rise of the Warrior Cop any thoughtful wrestling with whether there are alternative approaches to community safety, whether punitive rather than restorative justice is effective, or whether crime is a symptom of deeper societal problems we could address but refuse to. The most radical suggestion Balko has is to legalize drugs, which is both the predictable libertarian position and, as we have seen from recent events in the United States, far from the only problem of overcriminalization. I understand why this book is so frequently mentioned on-line, and its author's political views may make it more palatable to some people than a more race-centered or radical perspective. But I don't think this is the best or most useful book on police violence that one could read today. I hope to find a better one in upcoming reviews. Rating: 6 out of 10