Dear Debianites This morning I decided to just start writing Bits from DPL and send whatever I have by 18:00 local time. Here it is, barely proof read, along with all it's warts and grammar mistakes! It's slightly long and doesn't contain any critical information, so if you're not in the mood, don't feel compelled to read it! Get ready for a new DPL! Soon, the voting period will start to elect our next DPL, and my time as DPL will come to an end. Reading the questions posted to the new candidates on debian-vote, it takes quite a bit of restraint to not answer all of them myself, I think I can see how that aspect contributed to me being reeled in to running for DPL! In total I've done so 5 times (the first time I ran, Sam was elected!). Good luck to both Andreas and Sruthi, our current DPL candidates! I've already started working on preparing handover, and there's multiple request from teams that have came in recently that will have to wait for the new term, so I hope they're both ready to hit the ground running! Things that I wish could have gone better Communication Recently, I saw a t-shirt that read:

Adulthood is saying, 'But after this week things will slow down a bit' over and over until you die.

I can relate! With every task, crisis or deadline that appears, I think that once this is over, I'll have some more breathing space to get back to non-urgent, but important tasks. "Bits from the DPL" was something I really wanted to get right this last term, and clearly failed spectacularly. I have two long Bits from the DPL drafts that I never finished, I tend to have prioritised problems of the day over communication. With all the hindsight I have, I'm not sure which is better to prioritise, I do rate communication and transparency very highly and this is really the top thing that I wish I could've done better over the last four years. On that note, thanks to people who provided me with some kind words when I've mentioned this to them before. They pointed out that there are many other ways to communicate and be in touch with the community, and they mentioned that they thought that I did a good job with that. Since I'm still on communication, I think we can all learn to be more effective at it, since it's really so important for the project. Every time I publicly spoke about us spending more money, we got more donations. People out there really like to see how we invest funds in to Debian, instead of just making it heap up. DSA just spent a nice chunk on money on hardware, but we don't have very good visibility on it. It's one thing having it on a public line item in SPI's reporting, but it would be much more exciting if DSA could provide a write-up on all the cool hardware they're buying and what impact it would have on developers, and post it somewhere prominent like debian-devel-announce, Planet Debian or Bits from Debian (from the publicity team). I don't want to single out DSA there, it's difficult and affects many other teams. The Salsa CI team also spent a lot of resources (time and money wise) to extend testing on AMD GPUs and other AMD hardware. It's fantastic and interesting work, and really more people within the project and in the outside world should know about it! I'm not going to push my agendas to the next DPL, but I hope that they continue to encourage people to write about their work, and hopefully at some point we'll build enough excitement in doing so that it becomes a more normal part of our daily work. Founding Debian as a standalone entity This was my number one goal for the project this last term, which was a carried over item from my previous terms. I'm tempted to write everything out here, including the problem statement and our current predicaments, what kind of ground work needs to happen, likely constitutional changes that need to happen, and the nature of the GR that would be needed to make such a thing happen, but if I start with that, I might not finish this mail. In short, I 100% believe that this is still a very high ranking issue for Debian, and perhaps after my term I'd be in a better position to spend more time on this (hmm, is this an instance of "The grass is always better on the other side", or "Next week will go better until I die?"). Anyway, I'm willing to work with any future DPL on this, and perhaps it can in itself be a delegation tasked to properly explore all the options, and write up a report for the project that can lead to a GR. Overall, I'd rather have us take another few years and do this properly, rather than rush into something that is again difficult to change afterwards. So while I very much wish this could've been achieved in the last term, I can't say that I have any regrets here either. My terms in a nutshell COVID-19 and Debian 11 era My first term in 2020 started just as the COVID-19 pandemic became known to spread globally. It was a tough year for everyone, and Debian wasn't immune against its effects either. Many of our contributors got sick, some have lost loved ones (my father passed away in March 2020 just after I became DPL), some have lost their jobs (or other earners in their household have) and the effects of social distancing took a mental and even physical health toll on many. In Debian, we tend to do really well when we get together in person to solve problems, and when DebConf20 got cancelled in person, we understood that that was necessary, but it was still more bad news in a year we had too much of it already. I can't remember if there was ever any kind of formal choice or discussion about this at any time, but the DebConf video team just kind of organically and spontaneously became the orga team for an online DebConf, and that lead to our first ever completely online DebConf. This was great on so many levels. We got to see each other's faces again, even though it was on screen. We had some teams talk to each other face to face for the first time in years, even though it was just on a Jitsi call. It had a lasting cultural change in Debian, some teams still have video meetings now, where they didn't do that before, and I think it's a good supplement to our other methods of communication. We also had a few online Mini-DebConfs that was fun, but DebConf21 was also online, and by then we all developed an online conference fatigue, and while it was another good online event overall, it did start to feel a bit like a zombieconf and after that, we had some really nice events from the Brazillians, but no big global online community events again. In my opinion online MiniDebConfs can be a great way to develop our community and we should spend some further energy into this, but hey! This isn't a platform so let me back out of talking about the future as I see it... Despite all the adversity that we faced together, the Debian 11 release ended up being quite good. It happened about a month or so later than what we ideally would've liked, but it was a solid release nonetheless. It turns out that for quite a few people, staying inside for a few months to focus on Debian bugs was quite productive, and Debian 11 ended up being a very polished release. During this time period we also had to deal with a previous Debian Developer that was expelled for his poor behaviour in Debian, who continued to harass members of the Debian project and in other free software communities after his expulsion. This ended up being quite a lot of work since we had to take legal action to protect our community, and eventually also get the police involved. I'm not going to give him the satisfaction by spending too much time talking about him, but you can read our official statement regarding Daniel Pocock here: https://www.debian.org/News/2021/20211117 In late 2021 and early 2022 we also discussed our general resolution process, and had two consequent votes to address some issues that have affected past votes:

• https://www.debian.org/vote/2021/vote_003
• https://www.debian.org/vote/2022/vote_001

In my first term I addressed our delegations that were a bit behind, by the end of my last term all delegation requests are up to date. There's still some work to do, but I'm feeling good that I get to hand this over to the next DPL in a very decent state. Delegation updates can be very deceiving, sometimes a delegation is completely re-written and it was just 1 or 2 hours of work. Other times, a delegation updated can contain one line that has changed or a change in one team member that was the result of days worth of discussion and hashing out differences. I also received quite a few requests either to host a service, or to pay a third-party directly for hosting. This was quite an admin nightmare, it either meant we had to manually do monthly reimbursements to someone, or have our TOs create accounts/agreements at the multiple providers that people use. So, after talking to a few people about this, we founded the DebianNet team (we could've admittedly chosen a better name, but that can happen later on) for providing hosting at two different hosting providers that we have agreement with so that people who host things under debian.net have an easy way to host it, and then at the same time Debian also has more control if a site maintainer goes MIA. More info: https://wiki.debian.org/Teams/DebianNet You might notice some Openstack mentioned there, we had some intention to set up a Debian cloud for hosting these things, that could also be used for other additional Debiany things like archive rebuilds, but these have so far fallen through. We still consider it a good idea and hopefully it will work out some other time (if you're a large company who can sponsor few racks and servers, please get in touch!) DebConf22 and Debian 12 era DebConf22 was the first time we returned to an in-person DebConf. It was a bit smaller than our usual DebConf - understandably so, considering that there were still COVID risks and people who were at high risk or who had family with high risk factors did the sensible thing and stayed home. After watching many MiniDebConfs online, I also attended my first ever MiniDebConf in Hamburg. It still feels odd typing that, it feels like I should've been at one before, but my location makes attending them difficult (on a side-note, a few of us are working on bootstrapping a South African Debian community and hopefully we can pull off MiniDebConf in South Africa later this year). While I was at the MiniDebConf, I gave a talk where I covered the evolution of firmware, from the simple e-proms that you'd find in old printers to the complicated firmware in modern GPUs that basically contain complete operating systems- complete with drivers for the device their running on. I also showed my shiny new laptop, and explained that it's impossible to install that laptop without non-free firmware (you'd get a black display on d-i or Debian live). Also that you couldn't even use an accessibility mode with audio since even that depends on non-free firmware these days. Steve, from the image building team, has said for a while that we need to do a GR to vote for this, and after more discussion at DebConf, I kept nudging him to propose the GR, and we ended up voting in favour of it. I do believe that someone out there should be campaigning for more free firmware (unfortunately in Debian we just don't have the resources for this), but, I'm glad that we have the firmware included. In the end, the choice comes down to whether we still want Debian to be installable on mainstream bare-metal hardware. At this point, I'd like to give a special thanks to the ftpmasters, image building team and the installer team who worked really hard to get the changes done that were needed in order to make this happen for Debian 12, and for being really proactive for remaining niggles that was solved by the time Debian 12.1 was released. The included firmware contributed to Debian 12 being a huge success, but it wasn't the only factor. I had a list of personal peeves, and as the hard freeze hit, I lost hope that these would be fixed and made peace with the fact that Debian 12 would release with those bugs. I'm glad that lots of people proved me wrong and also proved that it's never to late to fix bugs, everything on my list got eliminated by the time final freeze hit, which was great! We usually aim to have a release ready about 2 years after the previous release, sometimes there are complications during a freeze and it can take a bit longer. But due to the excellent co-ordination of the release team and heavy lifting from many DDs, the Debian 12 release happened 21 months and 3 weeks after the Debian 11 release. I hope the work from the release team continues to pay off so that we can achieve their goals of having shorter and less painful freezes in the future! Even though many things were going well, the ongoing usr-merge effort highlighted some social problems within our processes. I started typing out the whole history of usrmerge here, but it's going to be too long for the purpose of this mail. Important questions that did come out of this is, should core Debian packages be team maintained? And also about how far the CTTE should really be able to override a maintainer. We had lots of discussion about this at DebConf22, but didn't make much concrete progress. I think that at some point we'll probably have a GR about package maintenance. Also, thank you to Guillem who very patiently explained a few things to me (after probably having have to done so many times to others before already) and to Helmut who have done the same during the MiniDebConf in Hamburg. I think all the technical and social issues here are fixable, it will just take some time and patience and I have lots of confidence in everyone involved. UsrMerge wiki page: https://wiki.debian.org/UsrMerge DebConf 23 and Debian 13 era DebConf23 took place in Kochi, India. At the end of my Bits from the DPL talk there, someone asked me what the most difficult thing I had to do was during my terms as DPL. I answered that nothing particular stood out, and even the most difficult tasks ended up being rewarding to work on. Little did I know that my most difficult period of being DPL was just about to follow. During the day trip, one of our contributors, Abraham Raji, passed away in a tragic accident. There's really not anything anyone could've done to predict or stop it, but it was devastating to many of us, especially the people closest to him. Quite a number of DebConf attendees went to his funeral, wearing the DebConf t-shirts he designed as a tribute. It still haunts me when I saw his mother scream "He was my everything! He was my everything!", this was by a large margin the hardest day I've ever had in Debian, and I really wasn't ok for even a few weeks after that and I think the hurt will be with many of us for some time to come. So, a plea again to everyone, please take care of yourself! There's probably more people that love you than you realise. A special thanks to the DebConf23 team, who did a really good job despite all the uphills they faced (and there were many!). As DPL, I think that planning for a DebConf is near to impossible, all you can do is show up and just jump into things. I planned to work with Enrico to finish up something that will hopefully save future DPLs some time, and that is a web-based DD certificate creator instead of having the DPL do so manually using LaTeX. It already mostly works, you can see the work so far by visiting https://nm.debian.org/person/ACCOUNTNAME/certificate/ and replacing ACCOUNTNAME with your Debian account name, and if you're a DD, you should see your certificate. It still needs a few minor changes and a DPL signature, but at this point I think that will be finished up when the new DPL start. Thanks to Enrico for working on this! Since my first term, I've been trying to find ways to improve all our accounting/finance issues. Tracking what we spend on things, and getting an annual overview is hard, especially over 3 trusted organisations. The reimbursement process can also be really tedious, especially when you have to provide files in a certain order and combine them into a PDF. So, at DebConf22 we had a meeting along with the treasurer team and Stefano Rivera who said that it might be possible for him to work on a new system as part of his Freexian work. It worked out, and Freexian funded the development of the system since then, and after DebConf23 we handled the reimbursements for the conference via the new reimbursements site: https://reimbursements.debian.net/ It's still early days, but over time it should be linked to all our TOs and we'll use the same category codes across the board. So, overall, our reimbursement process becomes a lot simpler, and also we'll be able to get information like how much money we've spent on any category in any period. It will also help us to track how much money we have available or how much we spend on recurring costs. Right now that needs manual polling from our TOs. So I'm really glad that this is a big long-standing problem in the project that is being fixed. For Debian 13, we're waving goodbye to the KFreeBSD and mipsel ports. But we're also gaining riscv64 and loongarch64 as release architectures! I have 3 different RISC-V based machines on my desk here that I haven't had much time to work with yet, you can expect some blog posts about them soon after my DPL term ends! As Debian is a unix-like system, we're affected by the Year 2038 problem, where systems that uses 32 bit time in seconds since 1970 run out of available time and will wrap back to 1970 or have other undefined behaviour. A detailed wiki page explains how this works in Debian, and currently we're going through a rather large transition to make this possible. I believe this is the right time for Debian to be addressing this, we're still a bit more than a year away for the Debian 13 release, and this provides enough time to test the implementation before 2038 rolls along. Of course, big complicated transitions with dependency loops that causes chaos for everyone would still be too easy, so this past weekend (which is a holiday period in most of the west due to Easter weekend) has been filled with dealing with an upstream bug in xz-utils, where a backdoor was placed in this key piece of software. An Ars Technica covers it quite well, so I won't go into all the details here. I mention it because I want to give yet another special thanks to everyone involved in dealing with this on the Debian side. Everyone involved, from the ftpmasters to security team and others involved were super calm and professional and made quick, high quality decisions. This also lead to the archive being frozen on Saturday, this is the first time I've seen this happen since I've been a DD, but I'm sure next week will go better! Looking forward It's really been an honour for me to serve as DPL. It might well be my biggest achievement in my life. Previous DPLs range from prominent software engineers to game developers, or people who have done things like complete Iron Man, run other huge open source projects and are part of big consortiums. Ian Jackson even authored dpkg and is now working on the very interesting tag2upload service! I'm a relative nobody, just someone who grew up as a poor kid in South Africa, who just really cares about Debian a lot. And, above all, I'm really thankful that I didn't do anything major to screw up Debian for good. Not unlike learning how to use Debian, and also becoming a Debian Developer, I've learned a lot from this and it's been a really valuable growth experience for me. I know I can't possible give all the thanks to everyone who deserves it, so here's a big big thanks to everyone who have worked so hard and who have put in many, many hours to making Debian better, I consider you all heroes! -Jonathan

This is my third update on writing a language server for Debian packaging files, which aims at providing a better developer experience for Debian packagers. Lets go over what have done since the last report.

Welcome to the February 2022 report from the Reproducible Builds project. In these reports, we try to round-up the important things we and others have been up to over the past month. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

---

Jiawen Xiong, Yong Shi, Boyuan Chen, Filipe R. Cogo and Zhen Ming Jiang have published a new paper titled Towards Build Verifiability for Java-based Systems (PDF). The abstract of the paper contains the following:

Various efforts towards build verifiability have been made to C/C++-based systems, yet the techniques for Java-based systems are not systematic and are often specific to a particular build tool (eg. Maven). In this study, we present a systematic approach towards build verifiability on Java-based systems.

---

GitBOM is a flexible scheme to track the source code used to generate build artifacts via Git-like unique identifiers. Although the project has been active for a while, the community around GitBOM has now started running weekly community meetings.

---

The paper Chris Lamb and Stefano Zacchiroli is now available in the March/April 2022 issue of IEEE Software. Titled Reproducible Builds: Increasing the Integrity of Software Supply Chains (PDF), the abstract of the paper contains the following:

We first define the problem, and then provide insight into the challenges of making real-world software build in a reproducible manner-this is, when every build generates bit-for-bit identical results. Through the experience of the Reproducible Builds project making the Debian Linux distribution reproducible, we also describe the affinity between reproducibility and quality assurance (QA).

---

In openSUSE, Bernhard M. Wiedemann posted his monthly reproducible builds status report.

---

On our mailing list this month, Thomas Schmitt started a thread around the SOURCE_DATE_EPOCH specification related to formats that cannot help embedding potentially timezone-specific timestamp. (Full thread index.)

---

The Yocto Project is pleased to report that it s core metadata (OpenEmbedded-Core) is now reproducible for all recipes (100% coverage) after issues with newer languages such as Golang were resolved. This was announced in their recent Year in Review publication. It is of particular interest for security updates so that systems can have specific components updated but reducing the risk of other unintended changes and making the sections of the system changing very clear for audit. The project is now also making heavy use of equivalence of build output to determine whether further items in builds need to be rebuilt or whether cached previously built items can be used. As mentioned in the article above, there are now public servers sharing this equivalence information. Reproducibility is key in making this possible and effective to reduce build times/costs/resource usage.

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions 203, 204, 205 and 206 to Debian unstable, as well as made the following changes to the code itself:

• Bug fixes:
  • Fix a file(1)-related regression where Debian .changes files that contained non-ASCII text were not identified as such, therefore resulting in seemingly arbitrary packages not actually comparing the nested files themselves. The non-ASCII parts were typically in the Maintainer or in the changelog text. [ ][ ]
  • Fix a regression when comparing directories against non-directories. [ ][ ]
  • If we fail to scan using binwalk, return False from BinwalkFile.recognizes. [ ]
  • If we fail to import binwalk, don t report that we are missing the Python rpm module! [ ]
• Testsuite improvements:
  • Add a test for recent file(1) issue regarding .changes files. [ ]
  • Use our assert_diff utility where we can within the test_directory.py set of tests. [ ]
  • Don t run our binwalk-related tests as root or fakeroot. The latest version of binwalk has some new security protection against this. [ ]
• Codebase improvements:
  • Drop the _PATH suffix from module-level globals that are not paths. [ ]
  • Tidy some control flow in Difference._reverse_self. [ ]
  • Don t print a warning to the console regarding NT_GNU_BUILD_ID changes. [ ]

In addition, Mattia Rizzolo updated the Debian packaging to ensure that diffoscope and diffoscope-minimal packages have the same version. [ ]

Debian-related updates Vagrant Cascadian wrote to the debian-devel mailing list after noticing that the binutils source package contained unreproducible logs in one of its binary packages. Vagrant expanded the discussion to one about all kinds of build metadata in packages and outlines a number of potential solutions that support reproducible builds and arbitrary metadata. Vagrant also started a discussion on debian-devel after identifying a large number of packages that embed build paths via RPATH when building with CMake, including a list of packages (grouped by Debian maintainer) affected by this issue. Maintainers were requested to check whether their package still builds correctly when passing the -DCMAKE_BUILD_RPATH_USE_ORIGIN=ON directive. On our mailing list this month, kpcyrd announced the release of rebuilderd-debian-buildinfo-crawler a tool to parse the Packages.xz Debian package index file, attempts to discover the right .buildinfo file from buildinfos.debian.net and outputs it in a format that can be understood by rebuilderd. The tool, which is available on GitHub, solves a problem regarding correlating Debian version numbers with their builds. bauen1 provided two patches for debian-cd, the software used to make Debian installer images. This involved passing --invariant and -i deb00001 to mkfs.msdos(8) and avoided embedding timestamps into the gzipped Packages and Translations files. After some discussion, the patches in question were merged and will be included in debian-cd version 3.1.36. Roland Clobus wrote another in-depth status update about status of live Debian images, summarising the current situation that all major desktops build reproducibly with bullseye, bookworm and sid . The python3.10 package was uploaded to Debian by doko, fixing an issue where [.pyc files were not reproducible because the elements in frozenset data structures were not ordered reproducibly. This meant that to creating a bit-for-bit reproducible Debian chroot which included .pyc files was not reproducible. As of writing, the only remaining unreproducible parts of a standard chroot is man-db, but Guillem Jover has a patch for update-alternatives which will likely be part of the next release of dpkg. Elsewhere in Debian, 139 reviews of Debian packages were added, 29 were updated and 17 were removed this month adding to our knowledge about identified issues. A large number of issue types have been updated too, including the addition of captures_kernel_variant, erlang_escript_file, captures_build_path_in_r_rdb_rds_databases, captures_build_path_in_vo_files_generated_by_coq and build_path_in_vo_files_generated_by_coq.

Website updates There were quite a few changes to the Reproducible Builds website and documentation this month as well, including:

• Chris Lamb:
  • Considerably rework the Who is involved? page. [ ][ ]
  • Move the contributors.sh Bash/shell script into a Python script. [ ][ ][ ]
• Daniel Shahaf:
  • Try a different Markdown footnote content syntax to work around a rendering issue. [ ][ ][ ]
• Holger Levsen:
  • Make a huge number of changes to the Who is involved? page, including pre-populating a large number of contributors who cannot be identified from the metadata of the website itself. [ ][ ][ ][ ][ ]
  • Improve linking to sponsors in sidebar navigation. [ ]
  • drop sponsors paragraph as the navigation is clearer now. [ ]
  • Add Mullvad VPN as a bronze-level sponsor . [ ][ ]
• Vagrant Cascadian:
  • Remove a stray parenthesis from the Who is involved? page. [ ]

Upstream patches The Reproducible Builds project attempts to fix as many currently-unreproducible packages as possible. February s patches included the following:

• Bernhard M. Wiedemann:
  • btop (sort-related issue)
  • complexity (date)
  • giac (update the version with upstreamed date patch)
  • htcondor (use CMake timestamp)
  • libint (readdir system call related)
  • libnet (date-related issue)
  • librime-lua (sort filesystem ordering)
  • linux_logo (sort-related issue)
  • micro-editor (date-related issue)
  • openvas-smb (date-related issue)
  • ovmf (sort-related issue)
  • paperjam (date-related issue)
  • python-PyQRCode (date-related issue)
  • quimb (single-CPU build failure)
  • radare2 (Meson date/time-related issue)
  • radare2 (Rework SOURCE_DATE_EPOCH usage to be portable)
  • siproxd (date, with Sebastian Kemper + follow-up
  • xonsh (Address Space Layout Randomisation-related issue)
  • xsnow (date & tar(1)-related issue)
  • zip (toolchain issue related to filesystem ordering)
• Chris Lamb:
  • #1005029 filed against ltsp (forwarded upstream).
  • #1005197 filed against pcmemtest.
  • #1005825 filed against hatchling.
  • #1005826 filed against mpl-sphinx-theme (forwarded upstream)
  • #1005827 filed against gap-hapcryst.
  • #1005901 filed against tree-puzzle.
  • #1005954 filed against jcabi-aspects.
  • #1005955 filed against paper-icon-theme.
• Roland Clobus:
  • #1006358 filed against libxmlb.
• Vagrant Cascadian:
  • #1005408 filed against wcwidth.
  • #1005420 filed against xir.
  • #1005421 filed against xir.
  • #1005726 filed against ruby-github-markup.
  • #1005727 filed against ruby-tioga.
  • #1005792 filed against btop.
  • #1005793 filed against libadwaita-1.
  • #1005794 filed against snibbetracker.
  • #1006252 filed against cctbx.
  • #1006254 filed against mdnsd.
  • #1006256 filed against gmerlin.
  • #1006302 filed against beav.
  • #1006385 filed against krita.
  • #1006407 filed against qt6-base.
  • #1006455 filed against onevpl-intel-gpu.
  • #1006471 filed against ruby3.0.
  • #1006473 filed against nix.
  • #1006474 filed against foma.
  • #1006476 filed against ruby3.0.

Testing framework The Reproducible Builds project runs a significant testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:

• Daniel Golle:
  • Update the OpenWrt configuration to not depend on the host LLVM, adding lines to the .config seed to build LLVM for eBPF from source. [ ]
  • Preserve more OpenWrt-related build artifacts. [ ]
• Holger Levsen:
• Temporary use a different Git tree when building OpenWrt as our tests had been broken since September 2020. This was reverted after the patch in question was accepted by Paul Spooren into the canonical openwrt.git repository the next day.
  • Various improvements to debugging OpenWrt reproducibility. [ ][ ][ ][ ][ ]
  • Ignore useradd warnings when building packages. [ ]
  • Update the script to powercycle armhf architecture nodes to add a hint to where nodes named virt-*. [ ]
  • Update the node health check to also fix failed logrotate and man-db services. [ ]
• Mattia Rizzolo:
  • Update the website job after contributors.sh script was rewritten in Python. [ ]
  • Make sure to set the DIFFOSCOPE environment variable when available. [ ]
• Vagrant Cascadian:
  • Various updates to the diffoscope timeouts. [ ][ ][ ]

Node maintenance was also performed by Holger Levsen [ ] and Vagrant Cascadian [ ].

Finally If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Twitter: @ReproBuilds
• Mailing list: rb-general@lists.reproducible-builds.org

(Blogging this, since this is a recurring anti-pattern I noticed at several customers and often comes up during deployments of 3rd party repositories.) Update on 2021-02-19: clarified, that Signed-By requires apt >= 1.1, thanks Vincent Bernat Many upstream projects provide Debian repository instructions like this:

curl -fsSL https://example.com/stable/debian.gpg   sudo apt-key add -

Do not follow this, for different reasons, including:

1. You do not see what you get before adding the GPG key to your global apt trust store
2. You can t easily script this via your preferred configuration management (the apt-key manpage clearly discourages programmatic usage)
3. The signing key is considered valid for all your enabled Debian repositories (instead of only a specific one)
4. You need GnuPG (either gnupg2 or gnupg1) on your system for usage with apt-key

There s a much better approach to this: download the GPG key, make sure it s in the appropriate format, then use it via deb [signed-by=/usr/share/keyrings/ ] in your apt s sources list configuration. Note and FTR: the Signed-By feature is available starting with apt 1.1 (so apt in Debian jessie/8 and older does not support it). TL;DR:

• Install GPG keys in ascii-armored / old public key block format as /usr/share/keyrings/example.asc and use deb [signed-by=/usr/share/keyrings/example.asc] https://example.com/ in apt s sources.list configuration
• Install GPG keys in binary OpenPGP format as /usr/share/keyrings/example.gpg and use deb [signed-by=/usr/share/keyrings/example.gpg] https://example.com/ in apt s sources.list configuration

As an example, let s demonstrate this with the Tailscale Debian repository for buster.
Downloading the GPG file will give you an ascii-armored GPG file:

% curl -fsSL -o buster.gpg https://pkgs.tailscale.com/stable/debian/buster.gpg
% gpg --keyid-format long buster.gpg 
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
pub   rsa4096/458CA832957F5868 2020-02-25 [SC]
      2596A99EAAB33821893C0A79458CA832957F5868
uid                           Tailscale Inc. (Package repository signing key) <info@tailscale.com>
sub   rsa4096/B1547A3DDAAF03C6 2020-02-25 [E]
% file buster.gpg
buster.gpg: PGP public key block Public-Key (old)

If you have apt version >= 1.4 available (Debian >=stretch/9 and Ubuntu >=bionic/18.04), you can use this file directly as follows:

% sudo mv buster.gpg /usr/share/keyrings/tailscale.asc
% cat /etc/apt/sources.list.d/tailscale.list
deb [signed-by=/usr/share/keyrings/tailscale.asc] https://pkgs.tailscale.com/stable/debian buster main
% sudo apt update
[...]

And you re done! Iff your apt version really is older than 1.4, you need to convert the ascii-armored GPG file into a GPG key public ring file (AKA binary OpenPGP format), either by just dearmor-ing it (if you don t care about checking ID + fingerprint):

% gpg --dearmor < buster.gpg > tailscale.gpg

or if you prefer to go via GPG, you can also use a temporary GPG home directory (if you don t care about going through your personal GPG setup):

% mkdir --mode=700 /tmp/gpg-tmpdir
% gpg --homedir /tmp/gpg-tmpdir --import ./buster.gpg
gpg: keybox '/tmp/gpg-tmpdir/pubring.kbx' created
gpg: /tmp/gpg-tmpdir/trustdb.gpg: trustdb created
gpg: key 458CA832957F5868: public key "Tailscale Inc. (Package repository signing key) <info@tailscale.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1
% gpg --homedir /tmp/gpg-tmpdir --output tailscale.gpg  --export-options=export-minimal --export 0x458CA832957F5868
% rm -rf /tmp/gpg-tmpdir

The resulting GPG key public ring file should look like that:

% file tailscale.gpg 
tailscale.gpg: PGP/GPG key public ring (v4) created Tue Feb 25 04:51:20 2020 RSA (Encrypt or Sign) 4096 bits MPI=0xc00399b10bc12858...
% gpg tailscale.gpg 
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
pub   rsa4096/458CA832957F5868 2020-02-25 [SC]
      2596A99EAAB33821893C0A79458CA832957F5868
uid                           Tailscale Inc. (Package repository signing key) <info@tailscale.com>
sub   rsa4096/B1547A3DDAAF03C6 2020-02-25 [E]

Then you can use this GPG file on your system as follows:

% sudo mv tailscale.gpg /usr/share/keyrings/tailscale.gpg
% cat /etc/apt/sources.list.d/tailscale.list
deb [signed-by=/usr/share/keyrings/tailscale.gpg] https://pkgs.tailscale.com/stable/debian buster main
% sudo apt update
[...]

Such a setup ensures:

1. You can verify the GPG key file (ID + fingerprint)
2. You can easily ship files via /usr/share/keyrings/ and refer to it in your deployment scripts, configuration management, (and can also easily update or get rid of them again!)
3. The GPG key is valid only for the repositories with the corresponding [signed-by=/usr/share/keyrings/ ] entry
4. You don t need to install GnuPG (neither gnupg2 nor gnupg1) on the system which is using the 3rd party Debian repository

Thanks: Guillem Jover for reviewing an early draft of this blog article.

Here is my monthly update covering what I have been doing in the free software world during January 2021 (previous month):

• As part of my duties of being on the board of directors of the Open Source Initiative, I attended its monthly meeting and participated in various licensing and other related discussions occurring on the internet. Unfortunately, I could not attend the parallel meeting for Software in the Public Interest this month.

• After a rather turbulent 2020, I was very grateful to have been chosen in OpenUK's 2021 honours as one of the 100 top influencers in the UK's open technology community, which recognises contributions to open source software, open data and open hardware. Congratulations to all of the other open source heroes and heroines who were also listed am looking forward to an exciting year together.

Reproducible Builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. This month, I:

• Categorised a huge number of packages and issues in the Reproducible Builds "notes" repository.
• Drafted, published and publicised our monthly report and attended a number of IRC meetings.
• In Debian:
  • Kept isdebianreproducibleyet.com up to date. [...]
  • Submitted the following patches to fix reproducibility-related toolchain issues within Debian:
  • I also submitted a number of patches to fix specific reproducibility issues including apertium-anaphora & davs2, as well as a number of issues found during reproducibility testing. For example, I discovered that node-tap-parser's manual pages contained errors instead of documentation (#980293).
• disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into system calls to reliably flush out reproducibility issues. This month, I updated our benchmarking tools to call use a tool that will call stat(2) repeatedly. [...]

• Updated the main Reproducible Builds website and documentation with many changes, including adding a missing image [...] and to ignore commits that start with, for example, '2020 12' when generating commit listings [...].
• strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. This month, I ensured that the tool did not process unwritable files and printing a warning in this case (#980356) as well as a number of codebase improvements such as reflowing logic to make future changes easier. [...]

I also made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues, releasing version 164, version 165 and version 166 as well as triaging and merging many contributions from others:

• New features:
  • Save sys.argv in our top-level temporary directory, in case it helps debug why temporary directories might not get cleaned up. [...]
  • Collapse the --acl and --xattr arguments into --extended-filesystem-attributes to cover all of these extended attributes, defaulting the new option to false (ie. to not check these expensive external calls). [...][...]
• Bug fixes:
  • Explicitly remove our top-level temporary directory. (#981123)
  • Adjust the fuzzy matching threshold to ensure that we show more differences. [...][...]
  • Use magic.Magic() over the now-deprecated magic.open() compatibility interface. [...]
• Output improvements:
  • Show the 'fuzziness' amount in percentage terms, not out of the rather-arbitrary '400'. [...]
  • Improve help text for the --exclude-directory-metadata argument. [...]
  • Wrap our external call to cmp(1) with a missing profiling point. [...]
  • Truncate jsondiff differences at 512 bytes, in case they consume the entire page. [...]
  • Improve the logging around fuzzy matching. [...]
• Codebase improvements:
  • Clarify in a comment that __del__ is not always called in Python, so temporary directories are not necessarily removed the moment they go out of scope. [...]
  • Print the free space in our temporary directory when we create it, not from within diffoscope.main. [...]
  • Tidy the diffoscope.comparators.utils.fuzzy module. [...]
  • Add a note regarding the special ordering of test_all_tools_are_listed within that module. [...]

Debian Uploads

• python-django:
  • 3.1.5-1 New upstream bugfix release.
  • 3.2~alpha1-1 New upstream 3.2 alpha release & misc packaging updates.
• redis:
  • 6.0.9-2 (to unstable) Enable systemd Type=notify support. (#977852lllllllllll)
  • 6.0.9-3 (to unstable) Allow /etc/redis to be rewritten and also removed upon purge. (#981000)
  • 6.0.9-4 (to unstable) Send systemd readiness notification when we are ready to accept connections in order to fix replicaof support. Thanks to Guillem Jover for the report and patch. (#981226)
  • 6.2~rc2-1 (to experimental) New upstream release.
  • 6.2~rc2-2 (to experimental) Allow /etc/redis to be rewritten and also removed upon purge. (#981000)

Debian LTS This month I worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.

• Investigated and triaged: cairo (CVE-2020-35492), firefox-esr (CVE-2020-16044), jackson-databind (CVE-2020-36179 CVE-2020-36189), nvidia-graphics-drivers (CVE-2021-1052, CVE-2021-1053 & CVE-2021-1056)), opensmtpd (CVE-2020-35680 & CVE-2020-35679), php7.0 (CVE-2020-7071)), python-autobahn (CVE-2020-35678) & tlslite-ng (CVE-2020-26263.
• Frontdesk duties, responding to user/developer questions, reviewing others' packages, participating in mailing list discussions, attending a monthly meeting, etc.
• Issued DLA 2515-1 for csync2. The cluster synchronization tool did not correctly check for the return value from GnuTLS security routines; it neglected to repeatedly call this function as required by the design of the API.
• Issued DLA 2516-1 for gssproxy to fix a privilege separation issue that arose from failing to unlock a shared resource.
• Issued DLA 2517-1 and ELA 342-1 for the Dovecot to fix two issues. The first was an issue where an attacker could cause the IMAP server to discover filesystem directory structures and even access other users' emails using a specially crafted command (CVE-2020-24386). The second was an issue where a malicious sender could crash Dovecot by sending messages with more than 10,000 (!) attachments (CVE-2020-25275).
• It was discovered that there was an in issue in the command-line tool for the Pacemaker High Availability stack. Local attackers were able to execute commands via shell code injection to the crm history command-line tool, potentially allowing escalation of privileges. DLA 2533-1 was issued to address this.

You can find out more about the project via the following video:

Background I usually don t mind ads, as not as they not very intrusive. I get that the current media model is basically ad-funded, and that unless I want to pay $1/month or so to 50 web sites, I have to accept ads, so I don t run an ad-blocker. Sure, sometimes are annoying (hey YT, mid-roll ads are borderline), but I ve also seen many good ads, as in interesting or funny even. Well, I don t think I ever bought anything as direct result from ads, so I don t know how useful ads are for the companies, but hey, what do I care. Except there a few ad networks that run what I would say are basically revolting ads. Things I don t want to ever accidentally see while eating, or things that are really make you go WTF? Maybe you know them, maybe you don t, but I guess there are people who don t know how to clean their ears, or people for whom a fast 7 day weight loss routine actually works. Thankfully, most of the time I don t browse sites which use this networks, but randomly they do leak to even sites I do browse. If I m not very stressed already, I can ignore them, otherwise they really, really annoy me. Case in point, I was on Slashdot, and because I was logged on and recently had mod points, the right side column had a check-box disable ads . That sidebar had some relatively meaningful ads, like a VPN subscription (not that I would use it, but it is a tech thing), or even a book about Kali Linux, etc. etc. So I click the disable ads , and the right column goes away. I scroll down happily, only to be met, at the bottom, by the best way to clean your ear , the most 50 useless planes ever built (which had a drawing of something that was for sure never ever built outside of in movies), you won t believe how this child actor looks today , etc.

Solving the problem The above really, really pissed me off, so I went to search how to block ad network . To my surprise, the fix was not that simple, for standard users at least.

Method 1: hosts file The hosts file is reasonable as it is relatively cross-platform (Linux and Windows and Mac, I think), but how the heck do you edit hosts on your phone? And furthermore, it has some significant downsides. First, /etc/hosts lists individual hosts, so for an entire ad network, the example I had had two screens of host names. This is really unmaintainable, since rotating host names, or having a gazillion of them is trivial. Second, it cannot return negative answers. I.e. you have to give each of those hosts a valid IPv4/IPv6, and have something either reply with 404 or another 4xx response, or not listen on port 80/443. Too annoying. And finally, it s a client-side solution, so one would have to replicate it across all clients in a home, and keep it in sync.

Method 2: ad-blockers I dislike ad-blockers on principle, since they need wide permissions on all pages, but it is a recommended solution. However, to my surprise, one finds threads saying ad-blocker foo has whitelisted ad network bar, at which point you re WTF? Why do I use an ad-blocker if they get paid by the lowest of the ad networks to show the ads? And again, it s a client-side solution, and one would have to deploy it across N clients, and keep them in sync, etc.

Method 3: HTTP proxy blocking To my surprise, I didn t find this mentioned in a quick internet search. Well, HTTP proxies have long gone the way of the dodo due to HTTPs everywhere , and while one can still use them even with HTTPS, it s not that convenient:

• you need to tunnel all traffic through them, which might result in bottlenecks (especially for media playing/maybe video-conference/etc.).
• or even worse, there might be protocol issues/incompatibilities due to 100% tunneling.
• running a proxy opens up some potential security issues on the internal network, so you need to harden the proxy as well, and maintain it.
• you need to configure all clients to know about the proxy (via DHCP or manually), which might or might not work well, since it s client-dependent.
• you can only block at CONNECT level (host name), and you have to build up regexes for the host name.

On the good side, the actual blocking configuration is centralised, and the only distributed configuration is pointing the clients through the proxy. While I used to run a proxy back in HTTP times, the gains were significant back them (media elements caching, downloads caching, all with a slow pipe, etc.), but today is not worth it, so I ve stopped and won t bring a proxy back just for this.

Method 4: DNS resolver filtering After thinking through all the options, I thought - hey, a caching/recursive DNS resolver is what most people with a local network run, right? How difficult is to block at resolver level? and oh my, it is so trivial, for some resolvers at least. And yes, I didn t know about this a week ago

Response Policy Zones Now, of course, there is a standard for this, called Response Policy Zone, and which is supported across multiple resolvers. There are many tutorials on how to use RPZs to configure things, some of them quite detailed - e.g. this one, or a simple/more straightforward one here. The upstream BIND documentation also explains things quite well here, so you can go that route as well. It looks a bit hairy for me thought, but it works, and since it is a standard, it can be more easily deployed. There are many discussions on the internet about how to configure RPZs, how to not even resolve the names (if you re going to return something explicitly/statically), etc. so there are docs, but again it seems a bit overdone.

Resolver hooks There s another way too, if your resolver allows scripting. For example, the PowerDNS resolver allow Lua scripting, and has a relatively simple API at least, to me it looks way, way simpler than the RPZ equivalent. After 20 minutes of reading the docs, I ended up with this, IMO trivial, solution (in a file named e.g. rules.lua):

ads = newDS()
ads:add( 'evilads.com', 'evilads.well-known-cdn.com', 'moreads.net' )

function preresolve(dq)
  if ads:check(dq.qname) then
    dq.rcode = pdns.NXDOMAIN
    return true;
  end
  return false;
end

and that s it. Well, enable it/load the file in the configuration, but nothing else. Syntax is pretty straightforward, matching by suffix here, and if you need more complex stuff, you can of course do it; it s just Lua and a simple API. I don t see any immediate equivalent in Bind, so there s that, but if you can use PowerDNS, then the above solution seems simple for simple cases, and could be extended if needed (not sure in which cases). The only other thing one needs to do is to serve the local/custom resolver to all clients, whether desktop or mobile, and that s it. DNS server is bread-and-butter in DHCP, so better support than proxy, and once the host name has been (mis)resolved, nothing is involved anymore in the communication path. True, your name server might get higher CPU usage, but for home network, this should not be a problem. Can this filtering method (either RPZ or hooks) be worked around by ad networks? Sure, like anything. But changing the base domain is not fun. DNSSEC might break it (note Bind RPZ can be configure to ignore DNSSEC), but I m more worried about DNS-over-HTTPS, which I thought initially it s done for the user, but now I m not so sure anymore. Not being in control even of your own DNS resolver seems evil , but what do I know.

Combined authoritative + recursive solution This solution was provided by Guillem Jover, who uses unbound, which is a combined authoritative name server and recursive resolver in one, and dnsmasq (which is even more things, I think):

For my LANs I use unbound, and then block this kind of thing in /etc/unbound/unbound.conf.d/block.conf, with stuff like:

server:
 local-zone: adsite.example.com refuse

But then for things that are mobile, and might get out of the LAN, such as laptops, I also block with dnsmasq in /etc/dnsmasq.d/block.conf, with stuff like:

 address=/adsite.example.com/

I still use ublock-origin to block stuff at the browser level, though, for yet an extra layer of noise suppression. :)

Thanks for the info!

Happy browsing! 10 lines of Lua, and now for sure I m going to get even fatter without the this natural method will melt your belly fat in 7 days information. Or I will just throw away banana peels without knowing what I could do with hem. After a few days, I asked myself but ads are not so bad, why did I and then realised that yes, ads are not so bad anymore. And Slashdot actually loads faster So, happy browsing!

Turns out that it is surprisingly easy to build most packages without (fake)root. You just need to basic changes:

1. A way to set ownership to root:root of paths when dpkg-deb build constructs the binary.
2. A way to have debhelper not do a bunch of (now) pointless chowns to root:root .

The above is sufficient for dpkg, debhelper, lintian, apt-file, mscgen, pbuilder and a long list of other packages that only provide paths owned by root:root . Obviously, packages differ and yours might need more tweaks than this (e.g. dh_usrlocal had to change behaviour to support this). But for me, the best part is that the above is not just some random prototype stuck in two git repos on alioth:

• dpkg and debhelper in unstable have all the bits required to do this and only requires you to opt-in via the Rules-Requires-Root -field (R ) after doing all the necessary tweaks (if any). Most of the above listed packages have actually already opted in by just adding the R -field.
• Guillem and I have posted an RFC for specification of Rules-Requires-Root -field that describes the field and what it is expected from you as a Debian package maintainer , a packaging-tool maintainer and a package builder . Review of this specification is very welcome.
• The implementations in dpkg and debhelper are generally safe for backporters. While YMMV, we expect that the vast majority will find that backporting an R =no package just works.
• Before you upload any R -related changes to unstable, please note that the specification and implementations are still subject to changes (assuming there are any concerns with either). That said, I believe that it is unlikely that there will be fundamental changes to the basic R =no variant. Accordingly, I have started to opt-in in my packages.

Unfortunately, if you are working with games or core packages like shadow with need for static ownership different from root:root (usually with a setuid or setgid bit), then our first implementation does not support your needs at the moment[1]. We are working on a separate way to solve static ownership in a declarative way. [1] Note regarding /usr/local : If your package needs to provide directories there owned by root:staff with mode 02775, then dh_usrlocal can handle that. The non- root:root ownership here works because the directories are created in a maintainer script run as root during installation. Unfortunately, it cannot provide different ownership or modes with R != binary-targets at the moment.
Filed under: Debhelper, Debian

Here is my monthly update covering what I have been doing in the free software world in September 2017 (previous month):

• Submitted a pull request to Quadrapassel (the Gnome version of Tetris) to start a new game when the pause button is pressed outside of a game. This means you would no longer have to use the mouse to start a new game. [...]
• Made a large number of improvements to AptFS my FUSE-based filesystem that provides a view on unpacked Debian source packages as regular folders including moving away from manual parsing of package lists [...] and numerous code tidying/refactoring changes.
• Sent a small patch to django-sitetree, a Django library for menu and breadcrumb navigation elements to not mask test exit codes from the surrounding shell. [...]
• Updated travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds:
  • Add support for "sloppy" backports. Thanks to Bernd Zeimetz for the idea and ongoing testing. [...]
  • Merged a pull request from James McCoy to pass DEB_BUILD_PROFILES through to the build. [...]
  • Workaround Travis CI's HTTP proxy which does not appear to support SRV records. [...]
  • Run debc from devscripts if the build was successful [...] and output the .buildinfo file if it exists [...].
• Fixed a few issues in local-debian-mirror, my package to easily maintain and customise a local Debian mirror via the DebConf configuration tool:
  • Fix an issue where file permissions from the remote could result in a local archive that was impossible to access. [...]
  • Clear out empty directories on the local repository. [...]
• Updated django-staticfiles-dotd, my Django staticfiles adaptor to concatentate static media in .d-style directories to support Python 3.x by using bytes objects (commit) and move away from monkeypatch as it does not have a Python 3.x port yet (commit).
• I also posted a short essay to my blog entitled "Ask the Dumb Questions" as well as provided an update on the latest Lintian release.

---

Lintian is a static analysis tool for Debian packages, reporting on various errors, omissions and general quality-assurance issues to maintainers. I've previously written about my exploits with Lintian as well as authoring a short tutorial on how to write your own Lintian check. Anyway, I recently uploaded version 2.5.53 about two months since previous release. The biggest changes you may notice are supporting the latest version of the Debian Policy as well the addition of checks to encourage the migration to Python 3. Thanks to all who contributed patches, code review and bug reports to this release. The full changelog is as follows:

lintian (2.5.53) unstable; urgency=medium
  The "we are all Perl developers now" release.
  * Summary of tag changes:
    + Added:
      - alternatively-build-depends-on-python-sphinx-and-python3-sphinx
      - build-depends-on-python-sphinx-only
      - dependency-on-python-version-marked-for-end-of-life
      - maintainer-script-interpreter
      - missing-call-to-dpkg-maintscript-helper
      - node-package-install-in-nodejs-rootdir
      - override-file-in-wrong-package
      - package-installs-java-bytecode
      - python-foo-but-no-python3-foo
      - script-needs-depends-on-sensible-utils
      - script-uses-deprecated-nodejs-location
      - transitional-package-should-be-oldlibs-optional
      - unnecessary-testsuite-autopkgtest-header
      - vcs-browser-links-to-empty-view
    + Removed:
      - debug-package-should-be-priority-extra
      - missing-classpath
      - transitional-package-should-be-oldlibs-extra
  * checks/apache2.pm:
    + [CL] Fix an apache2-unparsable-dependency false positive by allowing
      periods (".") in dependency names.  (Closes: #873701)
  * checks/binaries.pm:
    + [CL] Apply patches from Guillem Jover & Boud Roukema to improve the
      description of the binary-file-built-without-LFS-support tag.
      (Closes: #874078)
  * checks/changes. pm,desc :
    + [CL] Ignore DFSG-repacked packages when checking for upstream
      source tarball signatures as they will never match by definition.
      (Closes: #871957)
    + [CL] Downgrade severity of orig-tarball-missing-upstream-signature
      from "E:" to "W:" as many common tools do not make including the
      signatures easy enough right now.  (Closes: #870722, #870069)
    + [CL] Expand the explanation of the
      orig-tarball-missing-upstream-signature tag to include the location
      of where dpkg-source will look. Thanks to Theodore Ts'o for the
      suggestion.
  * checks/copyright-file.pm:
    + [CL] Address a number of issues in copyright-year-in-future:
      - Prevent false positives in port numbers, email addresses, ISO
        standard numbers and matching specific and general street
        addresses.  (Closes: #869788)
      - Match all violating years in a line, not just the first (eg.
        "2000-2107").
      - Ignore meta copyright statements such as "Original Author". Thanks
        to Thorsten Alteholz for the bug report.  (Closes: #873323)
      - Expand testsuite.
  * checks/cruft. pm,desc :
    + [CL] Downgrade severity of file-contains-fixme-placeholder
      tag from "important" (ie. "E:") to "wishlist" (ie. "I:").
      Thanks to Gregor Herrmann for the suggestion.
    + [CL] Apply patch from Alex Muntada (alexm) to use "substr" instead
      of "substring" in mentions-deprecated-usr-lib-perl5-directory's
      description.  (Closes: #871767)
    + [CL] Don't check copyright_hints file for FIXME placeholders.
      (Closes: #872843)
    + [CL] Don't match quoted "FIXME" variants as they are almost always
      deliberate. Thanks to Adrian Bunk for the report.  (Closes: #870199)
    + [CL] Avoid false positives in missing source checks for "CSS Browser
      Selector".  (Closes: #874381)
  * checks/debhelper.pm:
    + [CL] Prevent a false positive of
      missing-build-dependency-for-dh_-command that can be exposed by
      following the advice for the recently added
      useless-autoreconf-build-depends tag.  (Closes: #869541)
  * checks/debian-readme. pm,desc :
    + [CL] Ensure readme-debian-contains-debmake-template also checks
      for templates "Automatically generated by debmake".
  * checks/description. desc,pm :
    + [CL] Clarify explanation of description-starts-with-leading-spaces
      tag. Thanks to Taylor Kline  for the report
      and patch.  (Closes: #849622)
    + [NT] Skip capitalization-error-in-description-synopsis for
      auto-generated packages (such as dbgsym packages).
  * checks/fields. desc,pm :
    + [CL] Ensure that python3-foo packages have "Section: python", not
      just python2-foo.  (Closes: #870272)
    + [RG] Do no longer require debug packages to be priority extra.
    + [BR] Use Lintian::Data for name/section mapping
    + [CL] Check for packages including "?rev=0&sc=0" in Vcs-Browser.
      (Closes: #681713)
    + [NT] Transitional packages should now be "oldlibs/optional" rather
      than "oldlibs/extra".  The related tag has been renamed accordingly.
  * checks/filename-length.pm:
    + [NT] Skip the check on auto-generated binary packages (such as
      dbgsym packages).
  * checks/files. pm,desc :
    + [BR] Avoid privacy-breach-generic false positives for legal.xml.
    + [BR] Detect install of node package under /usr/lib/nodejs/[^/]*$
    + [CL] Check for packages shipping compiled Java class files. Thanks
      Carn  Draug .  (Closes: #873211)
    + [BR] Privacy breach is no longer experimental.
  * checks/init.d.desc:
    + [RG] Do not recommend a versioned dependency on lsb-base in
      init.d-script-needs-depends-on-lsb-base.  (Closes: #847144)
  * checks/java.pm:
    + [CL] Additionally consider .cljc files as code to avoid false-
      positive codeless-jar warnings.  (Closes: #870649)
    + [CL] Drop problematic missing-classpath check.  (Closes: #857123)
  * checks/menu-format.desc:
    + [CL] Prevent false positives in desktop-entry-lacks-keywords-entry
      for "Link" and "Directory" .desktop files.  (Closes: #873702)
  * checks/python. pm,desc :
    + [CL] Split out Python checks from "scripts" check to a new, source
      check of type "source".
    + [CL] Check for python-foo without corresponding python3-foo packages
      to assist in Python 2.x deprecation.  (Closes: #870681)
    + [CL] Check for packages that Build-Depend on python-sphinx only.
      (Closes: #870730)
    + [CL] Check for packages that alternatively Build-Depend on the
      Python 2 and Python 3 versions of Sphinx.  (Closes: #870758)
    + [CL] Check for binary packages that depend on Python 2.x.
      (Closes: #870822)
  * checks/scripts.pm:
    + [CL] Correct false positives in
      unconditional-use-of-dpkg-statoverride by detecting "if !" as a
      valid shell prefix.  (Closes: #869587)
    + [CL] Check for missing calls to dpkg-maintscript-helper(1) in
      maintainer scripts.  (Closes: #872042)
    + [CL] Check for packages using sensible-utils without declaring a
      dependency after its split from debianutils.  (Closes: #872611)
    + [CL] Warn about scripts using "nodejs" as an interpreter now that
      nodejs provides /usr/bin/node.  (Closes: #873096)
    + [BR] Add a statistic tag giving interpreter.
  * checks/testsuite. desc,pm :
    + [CL] Remove recommendations to add a "Testsuite: autopkgtest" field
      to debian/control as it is added when needed by dpkg-source(1)
      since dpkg 1.17.1.  (Closes: #865531)
    + [CL] Warn if we see an unnecessary "Testsuite: autopkgtest" header
      in debian/control.
    + [NT] Recognise "autopkgtest-pkg-go" as a valid test suite.
    + [CL] Recognise "autopkgtest-pkg-elpa" as a valid test suite.
      (Closes: #873458)
    + [CL] Recognise "autopkgtest-pkg-octave" as a valid test suite.
      (Closes: #875985)
    + [CL] Update the description of unknown-testsuite to reflect that
      "autopkgtest" is not the only valid value; the referenced URL
      is out-of-date (filed as #876008).  (Closes: #876003)
  * data/binaries/embedded-libs:
    + [RG] Detect embedded copies of heimdal, libgxps, libquicktime,
      libsass, libytnef, and taglib.
    + [RG] Use an additional string to detect embedded copies of
      openjpeg2.  (Closes: #762956)
  * data/fields/name_section_mappings:
    + [BR] node- package section is javascript.
    + [CL] Apply patch from Guillem Jover to add more section mappings.
      (Closes: #874121)
  * data/fields/obsolete-packages:
    + [MR] Add dh-systemd.  (Closes: #872076)
  * data/fields/perl-provides:
    + [CL] Refresh perl provides.
  * data/fields/virtual-packages:
    + [CL] Update data file from archive. This fixes a false positive for
      "bacula-director".  (Closes: #835120)
  * data/files/obsolete-paths:
    + [CL] Add note to /etc/bash_completion.d entry regarding stricter
      filename requirements.  (Closes: #814599)
  * data/files/privacy-breaker-websites:
    + [BR] Detect custom donation logos like apache.
    + [BR] Detect generic counter website.
  * data/standards-version/release-dates:
    + [CL] Add 4.0.1 and 4.1.0 as known standards versions.
      (Closes: #875509)
  * debian/control:
    + [CL] Mention Debian Policy v4.1.0 in the description.
    + [CL] Add myself to Uploaders.
    + [CL] Drop unnecessary "Testsuite: autopkgtest"; this is implied from
      debian/tests/control existing.
  * commands/info.pm:
    + [CL] Add a --list-tags option to print all tags Lintian knows about.
      Thanks to Rajendra Gokhale for the suggestion.  (Closes: #779675)
  * commands/lintian.pm:
    + [CL] Apply patch from Maia Everett to avoid British spelling when
      using en_US locale.  (Closes: #868897)
  * lib/Lintian/Check.pm:
    + [CL] Stop emitting  maintainer,uploader -address-causes-mail-loops
      for @packages.debian.org addresses.  (Closes: #871575)
  * lib/Lintian/Collect/Binary.pm:
    + [NT] Introduce an "auto-generated" argument for "is_pkg_class".
  * lib/Lintian/Data.pm:
    + [CL] Modify Lintian::Data's "all" to always return keys in insertion
      order, dropping dependency on libtie-ixhash-perl.
  * helpers/coll/objdump-info-helper:
    + [CL] Apply patch from Steve Langasek to accommodate binutils 2.29
      outputting symbols in a different format on ppc64el.
      (Closes: #869750)
  * t/tests/fields-perl-provides/tags:
    + [CL] Update expected output to match new Perl provides.
  * t/tests/files-privacybreach/*:
    + [CL] Add explicit test for packages including external fonts via
      the Google Font API. Thanks to Ian Jackson for the report.
      (Closes: #873434)
    + [CL] Add explicit test for packages including external fonts via
      the Typekit API via <script/> HTML tags.
  * t/tests/*/desc:
    + [CL] Add missing entries in "Test-For" fields to make
      development/testing workflow less error-prone.
  * private/generate-tag-summary:
    + [CL] git-describe(1) will usually emit 7 hexadecimal digits as the
      abbreviated object name,  However, as this can be user-dependent,
      pass --abbrev=0 to ensure it does not vary between systems.  This
      also means we do not need to strip it ourselves.
  * private/refresh-*:
    + [CL] Use deb.debian.org as the default mirror.
    + [CL] Update locations of Contents-<arch> files; they are now
      namespaced by distribution (eg. "main").
 -- Chris Lamb <lamby@debian.org>  Wed, 20 Sep 2017 09:25:06 +0100

I've wanted to start making one of these posts for a few months but have struggled to find the time. But it seems like a good idea, particularly since I get more done when I write down what I do, so you all get a rather belated one. This covers July and August; hopefully the September one will come closer to the end of September. Debian August was DebConf, which included a ton of Policy work thanks to Sean Whitton's energy and encouragement. During DebConf, we incorporated work from Hideki Yamane to convert Policy to reStructuredText, which has already made it far easier to maintain. (Thanks also to David Bremner for a lot of proofreading of the result.) We also did a massive bug triage and closed a ton of older bugs on which there had been no forward progress for many years. After DebConf, as expected, we flushed out various bugs in the reStructuredText conversion and build infrastructure. I fixed a variety of build and packaging issues and started doing some more formatting cleanup, including moving some footnotes to make the resulting document more readable. During July and August, partly at DebConf and partly not, I also merged wording fixes for seven bugs and proposed wording (not yet finished) for three more, as well as participated in various Policy discussions. Policy was nearly all of my Debian work over these two months, but I did upload a new version of the webauth package to build with OpenSSL 1.1 and drop transitional packages. Kerberos I still haven't decided my long-term strategy with the Kerberos packages I maintain. My personal use of Kerberos is now fairly marginal, but I still care a lot about the software and can't convince myself to give it up. This month, I started dusting off pam-krb5 in preparation for a new release. There's been an open issue for a while around defer_pwchange support in Heimdal, and I spent some time on that and tracked it down to an upstream bug in Heimdal as well as a few issues in pam-krb5. The pam-krb5 issues are now fixed in Git, but I haven't gotten any response upstream from the Heimdal bug report. I also dusted off three old Heimdal patches and submitted them as upstream merge requests and reported some more deficiencies I found in FAST support. On the pam-krb5 front, I updated the test suite for the current version of Heimdal (which changed some of the prompting) and updated the portability support code, but haven't yet pulled the trigger on a new release. Other Software I merged a couple of pull requests in podlators, one to fix various typos (thanks, Jakub Wilk) and one to change the formatting of man page references and function names to match the current Linux manual page standard (thanks, Guillem Jover). I also documented a bad interaction with line-buffered output in the Term::ANSIColor man page. Neither of these have seen a new release yet.

Here's what happened in the Reproducible Builds effort between Sunday August 20 and Saturday August 26 2017: Debian development

• "Packages should build reproducibly" was released in Debian Policy 4.1.0.0. For more background please see last week's post.
• A patch by Chris Lamb to make Dpkg::Substvars warnings output deterministic was merged by Guillem Jover. This helps the Reproducible Builds effort as it removes unnecessary differences in logs of two package builds. (#870221)

Packages reviewed and fixed, and bugs filed Forwarded upstream:

• Debian bug #872728 filed against desktop-file-utils. Filed upstream. (lamby, via a reproducibility issue in Tails).
• Debian bug #872729 filed against gtk+2.0. Filed upstream. (lamby, via a reproducibility issue in Tails).
• sawfish:
  • SOURCE_DATE_EPOCH
  • .tar.gz (merged)
  • Sorting (merged)
• librep:
  • SOURCE_DATE_EPOCH (merged)
  • Sorting (merged)
• zstd (merged) sort
• lirc (solved)
• autogen (tar.gz)

Accepted repoducibility NMUs in Debian:

• Chris Lamb:
  • jsmath-fonts (for bug #792319).
  • xvier (for bug #777330).
• Mattia Rizzolo:
  • aewan (for bug #842864).

Other issues:

• Adrian Bunk:
  • #872675 filed against pdp.
  • #872678 filed against gem.
  • #872860 filed against csound.
  • #872918 filed against sketch.
• Dmitry Shachnev:
  • #872992 filed against pytest-qt.

Reviews of unreproducible packages 16 package reviews have been added, 38 have been updated and 48 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated:

• Add new captures_build_dir_in_qmake_prl_files toolchain issue (Chris Lamb)
• Remove ftbfs_uninvestigated_big_packages (Mattia Rizzolo)

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (37)
• Dmitry Shachnev (1)
• James Cowgill (1)

diffoscope development

• Chris Lamb:
  • Tidy diffoscope.progress a little.
• Santiago Torres-Arias:
  • diffoscope.progress: Support the new fork of python-progressbar. (#873157)

disorderfs development Version 0.5.2-1 was uploaded to unstable by Ximin Luo. It included contributions from:

• Ximin Luo:
  • Add -q,--quiet CLI option
  • Update to latest Standards-Version; no changes required
  • Update debian/changelog
  • Update NEWS
  • Adopt as team maintenance
  • Add my signing key only
  • Add instructions for Debian release
• Holger Levsen:
  • Add myself to uploaders
  • Move package to 'optional' section

reprotest development

• Ximin Luo:
  • Don't vary user_group in the tests - it's not usually possible in automated builders
  • Default verbosity to 0 and silence disorderfs if set
  • Add the ability to vary the user. (#872412)
  • Add TODO

Misc. This week's edition was written in alphabetical order by Bernhard M. Wiedemann, Chris Lamb, Mattia Rizzolo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Today, about a month later than I had intended due to having three consecutive work weeks that mostly drained me of energy, I finally uploaded Debian Policy 4.0.0.0 to Debian experimental. This went to experimental rather than unstable for two reasons:

• Policy 4.0.0.0 is not targeted for Debian stretch. We're getting ready for the start of the buster development cycle. Please don't prioritize updating your packages until stretch is released. Please don't include Standards-Version changes in your uploads targeted for stretch.
• This release finally converts Policy from DebianDoc-SGML to DocBook. Many thanks to Osamu Aoki and Guillem Jover for their hard work on this conversion. This is a significant change with downstream impact on all the rendered versions. There will be bugs and formatting problems, and there will be unexpected impact on downstream consumers of Policy. It will take a bit to sort those out.

I expect there to be a few more point-release changes to packaging and formatting uploaded to experimental before uploading to unstable for the start of the buster development cycle. (I've indeed already noticed about six minor bugs, including the missing release date in the upgrading checklist....) Due to the DocBook conversion, and the resources rightly devoted to the stretch release instead, it may be a bit before the new Policy version shows up properly in all the places it's published. As you might expect from it having been more than a year since the previous release, there were a lot of accumulated changes. I posted the full upgrading-checklist entries to debian-devel-announce, or of course you can install the debian-policy package from experimental and review them in /usr/share/doc/debian-policy/upgrading-checklist.txt.gz.

Recently we had a WTF moment at a customer of mine which is worth sharing. In an automated deployment procedure we re installing Debian systems and setting up MySQL HA/Scalability. Installation of the first node works fine, but during installation of the second node something weird is going on. Even though the deployment procedure reported that everything went fine: it wasn t fine at all. After bisecting to the relevant command lines where it s going wrong we identified that the failure is happening between two ssh/scp commands, which are invoked inside a chroot through a shell wrapper. The ssh command caused a wrong exit code showing up: instead of bailing out with an error (we re running under set -e ) it returned with exit code 0 and the deployment procedure continued, even though there was a fatal error. Initially we triggered the bug when two ssh/scp command lines close to each other were executed, but I managed to find a minimal example for demonstration purposes:

# cat ssh_wrapper 
chroot << "EOF" / /bin/bash
ssh root@localhost hostname >/dev/null
exit 1
EOF
echo "return code = $?"

What we d expect is the following behavior, receive exit code 1 from the last command line in the chroot wrapper:

# ./ssh_wrapper 
return code = 1

But what we actually get is exit code 0:

# ./ssh_wrapper 
return code = 0

Uhm?! So what s going wrong and what s the fix? Let s find out what s causing the problem:

# cat ssh_wrapper 
chroot << "EOF" / /bin/bash
ssh root@localhost command_does_not_exist >/dev/null 2>&1
exit "$?"
EOF
echo "return code = $?"
# ./ssh_wrapper 
return code = 127

Ok, so if we invoke it with a binary that does not exist we properly get exit code 127, as expected.
What about switching /bin/bash to /bin/sh (which corresponds to dash here) to make sure it s not a bash bug:

# cat ssh_wrapper 
chroot << "EOF" / /bin/sh
ssh root@localhost hostname >/dev/null
exit 1
EOF
echo "return code = $?"
# ./ssh_wrapper 
return code = 1

Oh, but that works as expected!? When looking at this behavior I had the feeling that something is going wrong with file descriptors. So what about wrapping the ssh command line within different tools? No luck with stdbuf -i0 -o0 -e0 ssh root@localhost hostname , nor with script -c ssh root@localhost hostname /dev/null and also not with socat EXEC: ssh root@localhost hostname STDIO . But it works under unbuffer(1) from the expect package:

# cat ssh_wrapper 
chroot << "EOF" / /bin/bash
unbuffer ssh root@localhost hostname >/dev/null
exit 1
EOF
echo "return code = $?"
# ./ssh_wrapper 
return code = 1

So my bet on something with the file descriptor handling was right. Going through the ssh manpage, what about using ssh s -n option to prevent reading from standard input (stdin)?

# cat ssh_wrapper
chroot << "EOF" / /bin/bash
ssh -n root@localhost hostname >/dev/null
exit 1
EOF
echo "return code = $?"
# ./ssh_wrapper 
return code = 1

Bingo! Quoting ssh(1):

     -n      Redirects stdin from /dev/null (actually, prevents reading from stdin).
             This must be used when ssh is run in the background.  A common trick is
             to use this to run X11 programs on a remote machine.  For example,
             ssh -n shadows.cs.hut.fi emacs & will start an emacs on shadows.cs.hut.fi,
             and the X11 connection will be automatically forwarded over an encrypted
             channel.  The ssh program will be put in the background.  (This does not work
             if ssh needs to ask for a password or passphrase; see also the -f option.)

Let s execute the scripts through strace -ff -s500 ./ssh_wrapper to see what s going in more detail.
In the strace run without ssh s -n option we see that it s cloning stdin (file descriptor 0), getting assigned to file descriptor 4:

dup(0)            = 4
[...]
read(4, "exit 1\n", 16384) = 7

while in the strace run with ssh s -n option being present there s no file descriptor duplication but only:

open("/dev/null", O_RDONLY) = 4

This matches ssh.c s ssh_session2_open function (where stdin_null_flag corresponds to ssh s -n option):

        if (stdin_null_flag)                                              
                in = open(_PATH_DEVNULL, O_RDONLY);
          else  
                in = dup(STDIN_FILENO);
         

This behavior can also be simulated if we explicitly read from /dev/null, and this indeed works as well:

# cat ssh_wrapper
chroot << "EOF" / /bin/bash
ssh root@localhost hostname >/dev/null </dev/null
exit 1
EOF
echo "return code = $?"
# ./ssh_wrapper 
return code = 1

The underlying problem is that both bash and ssh are consuming from stdin. This can be verified via:

# cat ssh_wrapper
chroot << "EOF" / /bin/bash
echo "Inner: pre"
while read line; do echo "Eat: $line" ; done
echo "Inner: post"
exit 3
EOF
echo "Outer: exit code = $?"
# ./ssh_wrapper
Inner: pre
Eat: echo "Inner: post"
Eat: exit 3
Outer: exit code = 0

This behavior applies to bash, ksh, mksh, posh and zsh. Only dash doesn t show this behavior.
To understand the difference between bash and dash executions we can use the following test scripts:

# cat stdin-test-cmp
#!/bin/sh
TEST_SH=bash strace -v -s500 -ff ./stdin-test 2>&1   tee stdin-test-bash.out
TEST_SH=dash strace -v -s500 -ff ./stdin-test 2>&1   tee stdin-test-dash.out
# cat stdin-test
#!/bin/sh
: $ TEST_SH:=dash 
$TEST_SH <<"EOF"
echo "Inner: pre"
while read line; do echo "Eat: $line"; done
echo "Inner: post"
exit 3
EOF
echo "Outer: exit code = $?"

When executing ./stdin-test-cmp and comparing the generated files stdin-test-bash.out and stdin-test-dash.out you ll notice that dash consumes all stdin in one single go (a single read(0, ) ), instead of character-by-character as specified by POSIX and implemented by bash, ksh, mksh, posh and zsh. See stdin-test-bash.out on the left side and stdin-test-dash.out on the right side in this screenshot: So when ssh tries to read from stdin there s nothing there anymore. Quoting POSIX s sh section:

When the shell is using standard input and it invokes a command that also uses standard input, the shell shall ensure that the standard input file pointer points directly after the command it has read when the command begins execution. It shall not read ahead in such a manner that any characters intended to be read by the invoked command are consumed by the shell (whether interpreted by the shell or not) or that characters that are not read by the invoked command are not seen by the shell. When the command expecting to read standard input is started asynchronously by an interactive shell, it is unspecified whether characters are read by the command or interpreted by the shell. If the standard input to sh is a FIFO or terminal device and is set to non-blocking reads, then sh shall enable blocking reads on standard input. This shall remain in effect when the command completes.

So while we learned that both bash and ssh are consuming from stdin and this needs to prevented by either using ssh s -n or explicitly specifying stdin, we also noticed that dash s behavior is different from all the other main shells and could be considered a bug (which we reported as #862907). Lessons learned:

• Be aware of ssh s -n option when using ssh/scp inside scripts.
• Feeding shell scripts via stdin is not only error-prone but also very inefficient, as for a standards compliant implementation it requires a read(2) system call per byte of input. Instead create a temporary script you safely execute then.
• When debugging problems make sure to explore different approaches and tools to ensure you re not relying on a buggy behavior in any involved tool.

Thanks to Guillem Jover for review and feedback regarding this blog post.

Here's what happened in the Reproducible Builds effort between Sunday March 5 and Saturday March 11 2017: Upcoming events

• On March 23rd Holger Levsen will give a talk at the German Unix User Group's "Fr hjahrsfachgespr ch" called Reproducible Builds everywhere.
• Verifying Software Freedom with Reproducible Builds will be presented by Vagrant Cascadian at Libreplanet2017 in Boston, March 25th.
• You, too, can write reproducible software! workshop by Ximin Luo, Vagrant Cascadian and Valerie Young at Libreplanet2017 in Boston, March 25th.

Reproducible Builds Hackathon Hamburg The Reproducible Builds Hamburg Hackathon 2017, or RB-HH-2017 for short, is a 3 day hacking event taking place in the CCC Hamburg Hackerspace located inside the Frappant, which is collective art space located in a historical monument in Hamburg, Germany. The aim of the hackathon is to spent some days working on Reproducible Builds in every distribution and project. The event is open to anybody interested on working on Reproducible Builds issues in any distro or project, with or without prio experience! Packages filed Chris Lamb:

• #856834 filed against tendermint-go-rpc.
• #856860 filed against archvsync.
• #857122 filed against python-gdata (sent upstream)
• #857313 filed against cylc.
• #857454 filed against qtltools.

Toolchain development

• Guillem Jover uploaded dpkg 1.18.23 to unstable, declaring .buildinfo format 1.0 as "stable".
• Jams McCoy uploaded devscripts 2.17.2 to unstable addingd support for .buildinfo files to the debsign utility via patches from Ximin Luo and Guillem Jover.
• Hans-Christoph Steiner noted that the first reproducibility-related patch in the Android SDK was marked as confirmed.

Reviews of unreproducible packages 39 package reviews have been added, 7 have been updated and 9 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been added:

• randomness_in_binaries_generated_by_ruby_mkmf
• build_dir_in_documentation_generated_by_doxygen

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Chris Lamb (2)

buildinfo.debian.net development

• Chris Lamb:
  • Show reproducible status of package.
  • Inject operator.getitem as a template filter.
  • Include version in BuildInfo's __unicode__

reproducible-website development

• Holger Levsen:
  • Add first batch of post-it notes as pics
  • Add FAQ feedback pad
  • Update report to latest version from Beatrice
  • Polish diffoscope pad
  • Add (unverified) Berlin summit pads from riseup, without interlinks yet

tests.reproducible-builds.org

• Hans-Christoph Steiner gave a progress report on testing F-Droid: we now have a complete vagrant workflow working in nested KVM! So we can provision a new KVM guest, then package it using vagrant box all inside of a KVM guest (which is a profitbricks build node). So we finally have a working setup on jenkins.debian.net. Next up is fixing bugs in our libvirt snapshoting support.
• Then Hans-Christoph was also able to enable building of all F-Droid apps in our setup, though this is still work in progress
• Daniel Shahaf spotted a subtile error in our FreeBSD sudoers configuration and as a result the FreeBSD reproducibility results are back.
• Holger once again adjusted the Debian armhf scheduling frequency, to cope with the ever increasing amount of armhf builds.
• Mattia spotted a refactoring error which resulted in no maintenance mails for a week.
• Holger also spent some time on improving IRC notifications further, though there is still some improvements to be made.

Misc. This week's edition was written by Chris Lamb, Holger Levsen, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday February 12 and Saturday February 18 2017: Upcoming Events The Reproducible Build Zoo will be presented by Vagrant Cascadian at the Embedded Linux Conference in Portland, Oregon, February 22nd. Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th. Toolchain development and fixes Ximin Luo posted a preliminary spec for BUILD_PATH_PREFIX_MAP, bringing together work and research from previous weeks. Ximin refactored and consolidated much of our existing documentation on both SOURCE_DATE_EPOCH and BUILD_PATH_PREFIX_MAP into one unified page, Standard Environment Variables, with extended discussion on related solutions and how these all fit into people's ideas of what reproducible builds should look like in the long term. The specific pages for each variable still remain, at Timestamps Proposal and Build Path Proposal, only without content that was previously duplicated on both pages. Ximin filed #855282 against devscripts for debsign(1) to support buildinfo files, and wrote an initial series of patches for it with some further additions from Guillem Jover. Packages reviewed and fixed, and bugs filed Chris Lamb:

• #854999 filed against moin, and forwarded upstream.
• #855002 filed against samplv1, and forwarded upstream.
• #855426 filed against fritzing-parts.
• #855480 filed against examl.

Reviews of unreproducible packages 35 package reviews have been added, 1 have been updated and 17 have been removed in this week, adding to our knowledge about identified issues. 1 issue type has been added:

• nondeterminism_in_java_classes_generated_by_jxc

Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (2)

diffoscope development diffoscope 77 was uploaded to unstable by Mattia Rizzolo. It included contributions from:

• Chris Lamb:
  • Some fixes to tests and testing config
  • Don't track archive directory locations, a better fix for CVE-2017-0359.
  • Add --exclude option. Closes: #854783
• Mattia Rizzolo:
  • Add my key to debian/upstream/signing-key.asc
  • Add CVE-2017-0359 to the changelog of v76
• Ximin Luo:
  • When extracting archives, try to keep directory sizes small

strip-nondeterminism development strip-nondeterminism 0.031-1 was uploaded to unstable by Chris Lamb. It included contributions from:

• Chris Lamb:
  • Make the tests less brittle, by not testing for stat(2) blksize and blocks. #854937

strip-nondeterminism 0.031-1~bpo8+1 was uploaded to jessie-backports by Mattia. tests.reproducible-builds.org

• Vagrant Cascadian and Holger Levsen set up two new armhf nodes, p64b and p64c running on pine64 boards with an arm64 kernel and armhf userland. This introduces kernel variations to armhf. New setup & maintenance jobs were set up too, plus 6 new builder jobs for armhf.

Misc. This week's edition was written by Ximin Luo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday January 29 and Saturday February 4 2017: Media coverage Dennis Gilmore and Holger Levsen presented "Reproducible Builds and Fedora" (Video, Slides) at Devconf.cz on February 27th 2017. On February 1st, stretch/armhf reached 90% reproducible packages in our test framework, so that now all four tested architectures are 90% reproducible in stretch. Yay! For armhf this means 22472 reproducible source packages (in main); for amd64, arm64 and i386 these figures are 23363, 23062 and 22607 respectively. Chris Lamb appeared on the Changelog podcast to talk about reproducible builds: Holger Levsen pitched Reproducible Builds and our need for a logo in the "Open Source Design" room at FOSDEM 2017 (Video, 09:36 - 12:00). Upcoming Events

• The Reproducible Build Zoo will be presented by Vagrant Cascadian at the Embedded Linux Conference in Portland, Oregon, February 22nd.
• Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th.
• Verifying Software Freedom with Reproducible Builds will be presented by Vagrant Cascadian at Libreplanet2017 in Boston, March 25th-26th.

Reproducible work in other projects We learned that the "slightly more secure" Heads firmware (a Coreboot payload) is now reproducibly built regardless of host system or build directory. A picture says more than a thousand words: Docker started preliminary work on making image builds reproducible. Toolchain development and fixes Ximin Luo continued to write code and test cases for the BUILD_PATH_PREFIX_MAP environment variable. He also did extensive research on cross-platform and cross-language issues with enviroment variables, filesystem paths, and character encodings, and started preparing a draft specification document to describe all of this. Chris Lamb asked CPython to implement an environment variable PYTHONREVERSEDICTKEYORDER to add an an option to reverse iteration order of items in a dict. However this was rejected because they are planning to formally fix this order in the next language version. Bernhard Wiedemann and Florian Festi added support for our SOURCE_DATE_EPOCH environment variable, to the RPM Package Manager. James McCoy uploaded devscripts 2.17.1 with a change from Guillem Jover for dscverify(1), adding support for .buildinfo files. (Closes: #852801) Piotr O arowski uploaded dh-python 2.20170125 with a change from Chris Lamb for a patch to fix #835805. Chris Lamb added documentation to diffoscope, strip-nondeterminism, disorderfs, reprotest and trydiffoscope about uploading signed tarballs when releasing. He also added a link to these on our website's tools page. Packages reviewed and bugs filed Bugs filed:

• "Z. Ren":
  • #854293 filed against manpages-tr.
  • #854294 filed against regina-rexx.
• Chris Lamb:
  • #853039 filed against fontypython.
  • #853912 filed against python-testfixtures, merged upstream as PR #56.
  • #854111 filed against aprx.
  • #854112 filed against pnmixer.
• Reiner Herrmann:
  • #854145 filed against daemontools.
  • #854146 filed against diploma.

Reviews of unreproducible packages 83 package reviews have been added, 86 have been updated and 276 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated:

• captures_build_path_via_assert
• randomness_in_documentation_generated_by_epydoc

Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (6)

diffoscope development Work on the next version (71) continued in git this week:

• Mattia Rizzolo:
  • Override a lintian warning.
• Chris Lamb:
  • Update and consolidate documentation
  • Many test additions and improvements
  • Various code quality and software architecture improvements
• anthraxx:
  • Update arch package, cdrkit -> cdrtools.

reproducible-website development Daniel Shahaf added more notes on our "How to chair a meeting" document. tests.reproducible-builds.org Holger unblacklisted pspp and tiledarray. If you think further packages should also be unblacklisted (possibly only on some architectures), please tell us. Misc. This week's edition was written by Ximin Luo, Holger Levsen and Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donors (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS I was allocated 10 hours to work on security updates for Debian 7 Wheezy. During this time I did the following:

• I reviewed multiple CVE affecting ntp and opted to mark them no-dsa (just like what has been done for jessie).
• I pinged upstream authors of jbig2dec (here) and XML::Twig (by private email) where the upstream report had not gotten any upstream reply yet.
• I asked on oss-security for more details about CVE-2016-9584 because it was not clear whether it had already been reported upstream. Turns out that it was. I then updated the security tracker accordingly.
• Once I got a reply on jbig2dec, I started to backport the patch pointed out by upstream, it was hard work. When I was done, I had also received by private email the fuzzed file at the origin of the report unfortunately that file did not trigger the same problem with the old jbig2dec version in wheezy. That said valgrind still identified read outside of allocated memory. At this point I had a closer look at the git history only to discover that the last 3 years of work consisted mainly of security fixes for similar cases that were never reported to CVE. I thus opened a discussion about how to handle this situation.
• Matthias Geerdsen reported in #852610 a regression in libtiff4. I confirmed the problem and spent multiple hours to come up with a fix. The patch that introduced the regression was Debian-specific as upstream did not fix those issues yet. I released a fixed package in DLA-610-2.

Debian packaging With the deep freeze approaching, I made some last-minute updates:

• schroot 1.6.10-3 fixing some long-standing issues with the way bind mounts are shared (#761435) and other important fixes.
• live-boot 1:20170112 to fix a failure when booting on a FAT filesystem and other small fixes.
• live-config 5.20170112 merging useful patches from the BTS.
• I finished the update of hashcat 3.30 with its new private library and fixed RC bug #851497 at the same time. The work was initiated by fellow members of the pkg-security team.

Misc work Sponsorship. I sponsored a new asciidoc upload demoting a dependency into a recommends (#850301). I sponsored a new upstream version of dolibarr. Discussions. I seconded quite a few changes prepared by Russ Allbery on debian-policy. I helped Scott Kitterman with #849584 about a misunderstanding of how the postfix service files are supposed to work. I discussed in #849913 about a regression in building of cross-compilers, and made a patch to avoid the problem. In the end, Guillem developed a better fix. Bugs. I investigated #850236 where a django test failed during the first week after each leap year. I filed #853224 on desktop-base about multiple small problems in the maintainer scripts. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Here's what happened in the Reproducible Builds effort between Sunday January 22 and Saturday January 28 2017: Media coverage

• The Reproducible Build Zoo was presented by Vagrant Cascadian at the Embedded Linux Conference in Portland, Oregon on February 22nd.
• Dennis Gilmore and Holger Levsen presented "Reproducible Builds and Fedora" at Devconf.cz on February 27th.

Upcoming Events

• Introduction to Reproducible Builds will be presented by Vagrant Cascadian at Scale15x in Pasadena, California, March 5th.
• "Verifying Software Freedom with Reproducible Builds" will be presented by Vagrant Cascadian at Libreplanet2017 in Boston, March 25th-26th.

Reproducible work in other projects John Gilmore wrote an interesting mail about how Cygnus.com worked on reproducible builds in the early 1990s. It's eye opening to see how the dealt with basically the very same problems we're dealing with today, how they solved them and then to realize that most of this has been forgotten and bit-rotted in the last 20 years. How will we prevent history repeating itself here? Toolchain development and fixes Christoph Biedl wrote a mail describing an interesting problem in to the way binNMUs are done in Debian. Guillem Jover made a number of changes to dpkg that affect the Reproducible Builds effort within Debian:

• Always set SOURCE_DATE_EPOCH in dpkg-buildpackage and dpkg-source. Use the current date if the changelog does not have one. Closes: #849081
• Add initial support for DEB_BUILD_OPTIONS to dpkg-genbuildinfo. This will make it possible to enable or disable specific features that should be recorded in the .buildinfo file. For now only all and path are supported. Closes: #848705
• Include .buildinfo files also for source-only uploads in dpkg-genchanges. Closes: #846164
• Add support for signed .buildinfo files to dpkg-buildpackage. Add new -ui and --unsigned-buildinfo options. Closes: #843925
• Make dpkg-buildpackage --unsigned-changes not sign .buildinfo either. This breaks the expectations of users and tools, because there was no way previously to request no signing at all. Closes: #852822

Packages reviewed and fixed, and bugs filed Chris Lamb:

• #852482 filed against flask-limiter.
• #853039 filed against fontypython.

Dhole:

• #852289 filed against python-passlib.

Reviews of unreproducible packages 17 package reviews have been added, 4 have been updated and 6 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been added:

• python_versioneer_uses_parentdir
• captures_kernel_version_via_CMAKE_SYSTEM

1 issue type has been removed:

• ftbfs_due_to_jenkins_semaphore_setup

Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by:

• Chris Lamb (6)
• Holger Levsen (1)

diffoscope development

• diffoscope 70 was uploaded to unstable by Mattia Rizzolo. It included contributions from:
• Chris Lamb:
  • tests.presenters: Prevent FTBFS by loading fixtures as UTF-8 in case surrounding terminal is not Unicode-aware. (Closes: #852926)
  • comparators: Tidy re_tests with list comprehensions and implicit "x, y" unpacking over indexing; lambda/filter is not idiomatic Python 3.
  • tests: Increase coverage by adding "# noqa" in relevant parts.
  • tests: Test that no arguments (beyond the filenames) prints the text output.
  • tests: Test --text-color output format.
  • tests: Add a test comparing two empty directories.
  • tests: Don't warn about coverage lines that raise NotImplementedError.
• anthraxx:
  • tools: switch Arch Linux dependency for pedump to mono
• Ximin Luo:
  • Fix lazy expression, filter is lazy in Python 3
  • Fix bug introduced in commit 36d1c964 that only worked "accidentally"

reprotest development

• reprotest 0.6 was uploaded to unstable by Holger Levsen. It included contributions:
• Ximin Luo:
  • Test the extra variations we added recently and ensure they don't get missed in the future
  • Better logging messages that actually get controlled by the verbosity flag
  • Add a man page using rst2man and help2man
  • Add a --config-file option and fix the loading of configs
  • Fix the reading of config options
  • Fix a bug where the sha256sum of a reproduction won't be displayed if --store-dir is not given

buildinfo.debian.net development

• Chris Lamb:
  • Show SHA256 checksums on larger displays
  • Move default storage to S3.
  • Store files directly onto S3.
  • Add tool to move data to S3.
  • Load data from S3 if we don't have it locally.

tests.reproducible-builds.org

• h01ger experimented with reusing SSH control connections but stopped that experiment when we ran into more network issues than before. To be continued, as we're having 10k SSH connections per day and saving 2 seconds each time would sum up, especially on the Jenkins host itself.
• h01ger made the scheduler run 3 times a day, 2.5h after dinstall runs, instead of every 3h as before.
• h01ger restructured the https://tests.reproducible-builds.org/debian/index_breakages.html and improved the corresponding Jenins job.
• h01ger also unblacklisted xmds2, sofia-sip and ck - if you think other packages should be unblacklisted (maybe only on some architectures), please do tell us.

Misc. This week's edition was written by Chris Lamb and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

This is my general collection of utility functions, standard tests, and portability code, mostly for C but also including a fair bit of Perl these days. This release improves probing for GCC and Clang warning flags (thanks to Guillem Jover for a good hint there), adds a new is_file_contents test function to the Perl Test::RRA module for comparing whole files, and adds a new test that scans for non-https eyrie.org URLs in my documentation. It also fixes some issues with the Perl strictness test caused by not requiring a new enough version of Test::Strict. You can get the latest release from the rra-c-util distribution page.