TL;DR: firmware support in Debian sucks, and we need to change this. See the "My preference, and rationale" Section below. In my opinion, the way we deal with (non-free) firmware in Debian is a mess, and this is hurting many of our users daily. For a long time we've been pretending that supporting and including (non-free) firmware on Debian systems is not necessary. We don't want to have to provide (non-free) firmware to our users, and in an ideal world we wouldn't need to. However, it's very clearly no longer a sensible path when trying to support lots of common current hardware. Background - why has (non-free) firmware become an issue? Firmware is the low-level software that's designed to make hardware devices work. Firmware is tightly coupled to the hardware, exposing its features, providing higher-level functionality and interfaces for other software to use. For a variety of reasons, it's typically not Free Software. For Debian's purposes, we typically separate firmware from software by considering where the code executes (does it run on a separate processor? Is it visible to the host OS?) but it can be difficult to define a single reliable dividing line here. Consider the Intel/AMD CPU microcode packages, or the U-Boot firmware packages as examples. In times past, all necessary firmware would normally be included directly in devices / expansion cards by their vendors. Over time, however, it has become more and more attractive (and therefore more common) for device manufacturers to not include complete firmware on all devices. Instead, some devices just embed a very simple set of firmware that allows for upload of a more complete firmware "blob" into memory. Device drivers are then expected to provide that blob during device initialisation. There are a couple of key drivers for this change:

• Cost: it's typically cheaper to fit smaller flash memory (or no flash at all) onto a device. The cost difference may seem small in many cases, but reducing the bill of materials (BOM) even by a few cents can make a substantial difference to the economics of a product. For most vendors, they will have to implement device drivers anyway and it's not difficult to include firmware in that driver.
• Flexibility: it's much easier to change the behaviour of a device by simply changing to a different blob. This can potentially cover lots of different use cases:
  • separating deadlines for hardware and software in manufacturing (drivers and firmware can be written and shipped later);
  • bug fixes and security updates (e.g. CPU microcode changes);
  • changing configuration of a device for different users or products (e.g. potentially different firmware to enable different frequencies on a radio product);
  • changing fundamental device operation (e.g. switching between RAID and JBOD functionality on a disk controller).

Due to these reasons, more and more devices in a typical computer now need firmware to be uploaded at runtime for them to function correctly. This has grown:

• Going back 10 years or so, most computers only needed firmware uploads to make WiFi hardware work.
• A growing number of wired network adapters now demand firmware uploads. Some will work in a limited way but depend on extra firmware to allow advanced features like TCP segmentation offload (TSO). Others will refuse to work at all without a firmware upload.
• More and more graphics adapters now also want firmware uploads to provide any non-basic functions. A simple basic (S)VGA-compatible framebuffer is not enough for most users these days; modern desktops expect 3D acceleration, and a lot of current hardware will not provide that without extra firmware.
• Current generations of common Intel-based laptops also need firmware uploads to make audio work (see the firmware-sof-signed package).

At the beginning of this timeline, a typical Debian user would be able to use almost all of their computer's hardware without needing any firmware blobs. It might have been inconvenient to not be able to use the WiFi, but most laptops had wired ethernet anyway. The WiFi could always be enabled and configured after installation. Today, a user with a new laptop from most vendors will struggle to use it at all with our firmware-free Debian installation media. Modern laptops normally don't come with wired ethernet now. There won't be any usable graphics on the laptop's screen. A visually-impaired user won't get any audio prompts. These experiences are not acceptable, by any measure. There are new computers still available for purchase today which don't need firmware to be uploaded, but they are growing less and less common. Current state of firmware in Debian For clarity: obviously not all devices need extra firmware uploading like this. There are many devices that depend on firmware for operation, but we never have to think about them in normal circumstances. The code is not likely to be Free Software, but it's not something that we in Debian must spend our time on as we're not distributing that code ourselves. Our problems come when our user needs extra firmware to make their computer work, and they need/expect us to provide it. We have a small set of Free firmware binaries included in Debian main, and these are included on our installation and live media. This is great - we all love Free Software and this works. However, there are many more firmware binaries that are not Free. If we are legally able to redistribute those binaries, we package them up and include them in the non-free section of the archive. As Free Software developers, we don't like providing or supporting non-free software for our users, but we acknowledge that it's sometimes a necessary thing for them. This tension is acknowledged in the Debian Free Software Guidelines. This tension extends to our installation and live media. As non-free is officially not considered part of Debian, our official media cannot include anything from non-free. This has been a deliberate policy for many years. Instead, we have for some time been building a limited parallel set of "unofficial non-free" images which include non-free firmware. These non-free images are produced by the same software that we use for the official images, and by the same team. There are a number of issues here that make developers and users unhappy:

1. Building, testing and publishing two sets of images takes more effort.
2. We don't really want to be providing non-free images at all, from a philosophy point of view. So we mainly promote and advertise the preferred official free images. That can be a cause of confusion for users. We do link to the non-free images in various places, but they're not so easy to find.
3. Using non-free installation media will cause more installations to use non-free software by default. That's not a great story for us, and we may end up with more of our users using non-free software and believing that it's all part of Debian.
4. A number of users and developers complain that we're wasting their time by publishing official images that are just not useful for a lot (a majority?) of users.

We should do better than this. Options The status quo is a mess, and I believe we can and should do things differently. I see several possible options that the images team can choose from here. However, several of these options could undermine the principles of Debian. We don't want to make fundamental changes like that without the clear backing of the wider project. That's why I'm writing this...

1. Keep the existing setup. It's horrible, but maybe it's the best we can do? (I hope not!)
2. We could just stop providing the non-free unofficial images altogether. That's not really a promising route to follow - we'd be making it even harder for users to install our software. While ideologically pure, it's not going to advance the cause of Free Software.
3. We could stop pretending that the non-free images are unofficial, and maybe move them alongside the normal free images so they're published together. This would make them easier to find for people that need them, but is likely to cause users to question why we still make any images without firmware if they're otherwise identical.
4. The images team technically could simply include non-free into the official images, and add firmware packages to the input lists for those images. However, that would still leave us with problem 3 from above (non-free generally enabled on most installations).
5. We could split out the non-free firmware packages into a new non-free-firmware component in the archive, and allow a specific exception only to allow inclusion of those packages on our official media. We would then generate only one set of official media, including those non-free firmware packages. (We've already seen various suggestions in recent years to split up the non-free component of the archive like this, for example into non-free-firmware, non-free-doc, non-free-drivers, etc. Disagreement (bike-shedding?) about the split caused us to not make any progress on this. I believe this project should be picked up and completed. We don't have to make a perfect solution here immediately, just something that works well enough for our needs today. We can always tweak and improve the setup incrementally if that's needed.)

These are the most likely possible options, in my opinion. If you have a better suggestion, please let us know! I'd like to take this set of options to a GR, and do it soon. I want to get a clear decision from the wider Debian project as to how to organise firmware and installation images. If we do end up changing how we do things, I want a clear mandate from the project to do that. My preference, and rationale Mainly, I want to see how the project as a whole feels here - this is a big issue that we're overdue solving. What would I choose to do? My personal preference would be to go with option 5: split the non-free firmware into a special new component and include that on official media. Does that make me a sellout? I don't think so. I've been passionately supporting and developing Free Software for more than half my life. My philosophy here has not changed. However, this is a complex and nuanced situation. I firmly believe that sharing software freedom with our users comes with a responsibility to also make our software useful. If users can't easily install and use Debian, that helps nobody. By splitting things out here, we would enable users to install and use Debian on their hardware, without promoting/pushing higher-level non-free software in general. I think that's a reasonable compromise. This is simply a change to recognise that hardware requirements have moved on over the years. Further work If we do go with the changes in option 5, there are other things we could do here for better control of and information about non-free firmware:

1. Along with adding non-free firmware onto media, when the installer (or live image) runs, we should make it clear exactly which firmware packages have been used/installed to support detected hardware. We could link to docs about each, and maybe also to projects working on Free re-implementations.
2. Add an option at boot to explicitly disable the use of the non-free firmware packages, so that users can choose to avoid them.

Acknowledgements Thanks to people who reviewed earlier versions of this document and/or made suggestions for improvement, in particular:

• Cyril Brulebois
• Matthew Garrett
• David Leggett
• Martin Michlmayr
• Andy Simpkins
• Neil Williams

I first started using Debian sometime in the mid 90s and started contributing as a developer and package maintainer more than two decades years ago. My first very first scholarly publication, collaborative work led by Martin Michlmayr that I did when I was still an undergrad at Hampshire College, was about quality and the reliance on individuals in Debian. To this day, many of my closest friends are people I first met through Debian. I met many of them at Debian s annual conference DebConf. Given my strong connections to Debian, I find it somewhat surprising that although all of my academic research has focused on peer production, free culture, and free software, I haven t actually published any Debian related research since that first paper with Martin in 2003! So it felt like coming full circle when, several days ago, I was able to sit in the virtual DebConf audience and watch two of my graduate student advisees Kaylea Champion and Wm Salt Hale present their research about Debian at DebConf21. Salt presented his masters thesis work which tried to understand the social dynamics behind organizational resilience among free software projects. Kaylea presented her work on a new technique she developed to identifying at risk software packages that are lower quality than we might hope given their popularity (you can read more about Kaylea s project in our blog post from earlier this year). If you missed either presentation, check out the blog post my research collective put up or watch the videos below. If you want to hear about new work we re doing including work on Debian you should follow our research group blog, and/or follow or engage with us in the Fediverse (@communitydata@social.coop), or on Twitter (@comdatasci). And if you re interested in joining us perhaps to do more research on FLOSS and/or Debian and/or a graduate degree of your own? please be in touch with me directly!

Wm Salt Hale s presentation plus Q&A. (WebM available)

Kaylea Champion s presentation plus Q&A. (WebM available)

I worked on research on FOSS foundations and published two reports: Growing Open Source Projects with a Stable Foundation This primer covers non-technical aspects that the majority of projects will have to consider at some point. It also explains how FOSS foundations can help projects grow and succeed. This primer explains:

• What issues and areas to consider
• How other projects and foundations have approached these topics
• What FOSS foundations bring to the table
• How to choose a FOSS foundation

You can download Growing Open Source Projects with a Stable Foundation. Research report The research report describes the findings of the research and aims to help understand the operations and challenges FOSS foundations face. This report covers topics such as:

• Role and activities of foundations
• Challenges faced and gaps in the service offerings
• Operational aspects, including reasons for starting an org and choice of jurisdiction
• Trends, such as the "foundation in a foundation" model
• Recommendations for different stakeholders

You can download the research report. Acknowledgments This research was sponsored by Ford Foundation and Alfred P. Sloan Foundation. The research was part of their Critical Digital Infrastructure Research initiative, which investigates the role of open source in digital infrastructure.

I released version 2.6 of ledger2beancount, a ledger to beancount converter. Here are the changes in 2.6:

• Round calculated total if needed for price==cost comparison
• Add narration_tag config variable to set narration from metadata
• Retain unconsummated payee/payer metadata
• Ensure UTF-8 output and assume UTF-8 input
• Document UTF-8 issue on Windows systems
• Add option to move posting-level tags to the transaction itself
• Add support for the alias sub-directive of account declarations
• Add support for the payee sub-directive of account declarations
• Support configuration file called .ledger2beancount.yaml
• Fix uninitialised value warning in hledger mode
• Print warning if account in assertion has sub-accounts
• Set commodity for commodity-less balance assertion
• Expand path name of beancount_header config variable
• Document handling of buckets
• Document pre- and post-processing examples
• Add Dockerfile to create Docker image

Thanks to Alexander Baier, Daniele Nicolodi, and GitHub users bratekarate, faaafo and mefromthepast for various bug reports and other input. Thanks to Dennis Lee for adding a Dockerfile and to Vinod Kurup for fixing a bug. Thanks to Stefano Zacchiroli for testing. You can get ledger2beancount from GitHub.

As a reader of Planet Debian I see a bunch of updates at the start of each month about what people are up to in terms of their Free Software activities. I m not generally active enough in the Free Software world to justify a monthly report, but I did a report of my Free Software Activities for 2019 and thought I d do another for 2020. I ended up not doing as much as last year; I put a lot of that down to fatigue about the state of the world and generally not wanting to spend time on the computer at the end of the working day.

Conferences 2020 was unsurprisingly not a great year for conference attendance. I was fortunate enough to make it to FOSDEM and CopyleftConf 2020 - I didn t speak at either, but had plenty of interesting hallway track conversations as well as seeing some good talks. I hadn t been planning to attend DebConf20 due to time constraints, but its move to an entirely online conference meant I was able to attend a few talks at least. I have to say I don t like virtual conferences as much as the real thing; it s not as easy to have the casual chats at them, and it s also harder to carve out the exclusive time when you re at home. That said I spoke at NIDevConf this year, which was also fully virtual. It s not a Free Software focussed conference, but there s a lot of crossover in terms of technologies and I spoke on my experiences with Go, some of which are influenced by my packaging experiences within Debian.

Debian Most of my contributions to Free software happen within Debian. As part of the Data Protection Team I responded to various inbound queries to that team. Some of this involved chasing up other project teams who had been slow to respond - folks, if you re running a service that stores personal data about people then you need to be responsive to requests about it. The Debian Keyring was possibly my largest single point of contribution. We re in a roughly 3 month rotation of who handles the keyring updates, and I handled 2020.02.02, 2020.03.24, 2020.06.24, 2020.09.24 + 2020.12.24 For Debian New Members I m mostly inactive as an application manager - we generally seem to have enough available recently. If that changes I ll look at stepping in to help, but I don t see that happening. I continue to be involved in Front Desk, having various conversations throughout the year with the rest of the team, but there s no doubt Mattia and Pierre-Elliott are the real doers at present. In terms of package uploads I continued to work on gcc-xtensa-lx106, largely doing uploads to deal with updates to the GCC version or packaging (5, 6 + 7). sigrok had a few minor updates, libsigkrok 0.5.2-2, libsigrokdecode 0.5.3-2 as well as a new upstream release of Pulseview 0.4.2-1 and a fix to cope with change to QT 0.4.2-2. Due to the sigrok-firmware requirement on sdcc I also continued to help out there, updating to 4.0.0+dfsg-1 and doing some fixups in 4.0.0+dfsg-2. Despite still not writing an VHDL these days I continue to try and make sure ghdl is ok, because I found it a useful tool in the past. In 2020 that meant a new upstream release, 0.37+dfsg-1 along with a couple of more minor updates (0.37+dfsg-2 + 0.37+dfsg-3. libcli had a new upstream release, 1.10.4-1, and I did a long overdue update to sendip to the latest upstream release, 2.6-1 having been poked about an outstanding bug by the Reproducible Builds folk. OpenOCD is coming up to 4 years since its last stable release, but I did a snapshot upload to Debian experimental (0.10.0+g20200530-1) and a subsequent one to unstable (0.10.0+g20200819-1). There are also moves to produce a 0.11.0 release and I uploaded 0.11.0~rc1-1 as a result. libjaylink got a bump as a result (0.2.0-1) after some discussion with upstream.

OpenOCD On the subject of OpenOCD I ve tried to be a bit more involved upstream. I m not familiar enough with the intricacies of JTAG/SWD/the various architectures supported to contribute to the core, but I pushed the config for my HIE JTAG adapter upstream and try and review patches that don t require in depth hardware knowledge.

Linux I ve been contributing to the Linux kernel for a number of years now, mostly just minor bits here and there for issues I hit. This year I spent a lot of time getting support for the MikoTik RB3011 router upstreamed. That included the basic DTS addition, fixing up QCA8K to support SGMII CPU connections, adding proper 802.1q VLAN support to QCA8K and cleaning up an existing QCOM ADM driver that s required for the NAND. There were a number of associated bugfixes/minor changes found along the way too. It can be a little frustrating at times going round the review loop with submitting things upstream, but I do find it quite satisfying when it all comes together and I have no interest in weird vendor trees that just bitrot over time.

Software in the Public Interest I haven t sat on the board of SPI since 2015 but I was still acting as the primary maintainer of the membership website (with Martin Michlmayr as the other active contributor) and hosting it on my own machine. I managed to finally extricate myself from this role in August. I remain a contributing member.

Personal projects 2020 finally saw another release (0.6.0, followed swiftly by 0.6.1 to allow the upload of 0.6.1-1 to Debian) of onak. This release finally adds various improvements to deal with the hostility shown to the OpenPGP keyserver network in recent years, including full signature verification as an option. I fixed an oversight in my Digoo/1-wire temperature decoder and a bug that turned up on ARM but not MIPS in my mqtt-arp code. I should probably package it for Debian (even if I don t upload it), as I m running it on my RB3011 now.

I released version 1.3 of beancount2ledger, the beancount to ledger converter that was moved from bean-report ledger into a standalone tool. You can get beancount2ledger from GitHub or via pip install. Here are the changes in 1.3:

• Add rounding postings only when required (issue #9)
• Avoid printing too much precision for a currency (issue #21)
• Avoid creating two or more postings with null amount (issue #23)
• Add price to cost when needed by ledger (issue #22)
• Preserve posting order (issue #18)
• Add config option indent
• Show metadata with hledger output
• Support setting auxiliary dates and posting dates from metadata (issue #14)
• Support setting the code of transactions from metadata
• Support mapping of account and currency names (issue #24)
• Improve documentation:
  • Add user guide
  • Document limitations (issue #12)

I released version 2.5 of ledger2beancount, a ledger to beancount converter. Here are the changes in 2.5:

• Don't create negative cost for lot without cost
• Support complex implicit conversions
• Handle typed metadata with value 0 correctly
• Set per-unit instead of total cost when cost is missing from lot
• Support commodity-less amounts
• Convert transactions with no amounts or only 0 amounts to notes
• Fix parsing of transaction notes
• Keep tags in transaction notes on same line as transaction header
• Add beancount config options for non-standard root names automatically
• Fix conversion of fixated prices to costs
• Fix removal of price when price==cost but when they use different number formats
• Fix removal of price when price==cost but per-unit and total notation mixed
• Fix detection of tags and metadata after posting/aux date
• Use D directive to set default commodity for hledger
• Improve support for postings with commodity-less amounts
• Allow empty comments
• Preserve leading whitespace in comments in postings and transaction headers
• Preserve indentation for tags and metadata
• Preserve whitespace between amount and comment
• Refactor code to use more data structures
• Remove dependency on Config::Onion module

Thanks to input from Remco R nders, Yuri Khan, and Thierry. Thanks to Stefano Zacchiroli and Kirill Goncharov for testing my changes. You can get ledger2beancount from GitHub

I released version 2.4 of ledger2beancount, a ledger to beancount converter. There are two notable changes in this release:

1. I fixed two regressions introduced in the last release. Sorry about the breakage!
2. I improved support for hledger. I believe all syntax differences in hledger are supported now.

Here are the changes in 2.4:

• Fix regressions introduced in version 2.3
  • Handle price directives with comments
  • Don't assume implicit conversion when price is on second posting
• Improve support for hledger
  • Fix parsing of hledger tags
  • Support commas as decimal markers
  • Support digit group marks through commodity and D directives
  • Support end aliases directive
  • Support regex aliases
  • Recognise total balance assertions
  • Recognise sub-account balance assertions
• Add support for define directive
• Convert all uppercase metadata tags to all lowercase
• Improve handling of ledger lots without cost
• Allow transactions without postings
• Fix parsing issue in commodity declarations
• Support commodities that contain quotation marks
• Add --version option to show version
• Document problem of mixing apply and include

Thanks to Kirill Goncharov for pointing out one regressions, to Taylor R Campbell for for a patch, to Stefano Zacchiroli for some input, and finally to Simon Michael for input on hledger! You can get ledger2beancount from GitHub

Martin Blais recently announced that he'd like to re-organize the beancount code and split out some functionality into separate projects, including the beancount to ledger/hledger conversion code previously provided by bean-report. I agreed to take on the maintenance of this code and I've now released beancount2ledger, a beancount to ledger/hledger converter. You can install beancount2ledger with pip:

pip3 install beancount2ledger

Please report issues to the GitHub tracker. There are a number of outstanding issues I'll fix soon, but please report any other issues you encounter. Note that I'm not very familiar with hledger. I intend to sync up with hledger author Simon Michael soon, but please file an issue if you notice any problems with the hledger conversion. Version 1.1 contains a number of fixes compared to the latest code in bean-report: 1.1 (2020-07-24)

• Preserve metadata information (issue #3)
• Preserve cost information (lot dates and lot labels/notes) (issue #5)
• Avoid adding two prices in hledger (issue #2)
• Avoid trailing whitespace in account open declarations (issue #6)
• Fix indentation issue in postings (issue #8)
• Fix indentation issue in price entries
• Drop time information from price (P) entries
• Add documentation
• Relicense under GPL-2.0-or-later (issue #1)

1.0 (2020-07-22)

• Split ledger and hledger conversion from bean-report into a standalone tool
• Add man page for beancount2ledger(1)

I released version 2.3 of ledger2beancount, a ledger to beancount converter. There are three notable changes with this release:

1. Performance has significantly improved. One large, real-world test case has gone from around 160 seconds to 33 seconds. A smaller test case has gone from 11 seconds to ~3.5 seconds.
2. The documentation is available online now (via Read the Docs).
3. The repository has moved to the beancount GitHub organization.

Here are the changes in 2.3:

• Improve speed of ledger2beancount significantly
• Improve parsing of postings for accuracy and speed
• Improve support for inline math
• Handle lots without cost
• Fix parsing of lot notes followed by a virtual price
• Add support for lot value expressions
• Make parsing of numbers more strict
• Fix behaviour of dates without year
• Accept default ledger date formats without configuration
• Fix implicit conversions with negative prices
• Convert implicit conversions in a more idiomatic way
• Avoid introducing trailing whitespace with hledger input
• Fix loading of config file
• Skip ledger directive import
• Convert documentation to mkdocs

Thanks to Colin Dean for some feedback. Thanks to Stefano Zacchiroli for prompting me into investigating performance issues (and thanks to the developers of the Devel::NYTProf profiler). You can get ledger2beancount from GitHub

I released version 2.2 of ledger2beancount, a ledger to beancount converter. Here are the changes in 2.2:

• Show warning for unknown apply directive
• Recognize apply rate directive (an alias of apply fixed)
• Don't convert meta-data on ignored virtual postings but keep as comments
• Update location of beancount repository

You can get ledger2beancount from GitHub. Thanks to GitHub user MarinBernard for reporting a bug with virtual postings!

I released version 2.1 of ledger2beancount, a ledger to beancount converter. Here are the changes in 2.1:

• Handle postings with posting dates and comments but no amount
• Show transactions with only one posting (without bucket)
• Adding spacing between automatic declarations
• Preserve preliminary info at the top

You can get ledger2beancount from GitHub. Thanks to Thierry (thdox) for reporting a bug and for fixing some typos in the documentation. Thanks to Stefano Zacchiroli for some good feedback.

Here's what happened in the Reproducible Builds effort between Sunday June 11 and Saturday June 17 2017: Upcoming events

• On June 19th, Chris Lamb presented at LinuxCon China 2017 on Reproducible Builds.
• h01ger created a poll for a date for the next Reproducible Builds summit, please vote if you are interested in attending.
• Our next IRC meeting will be on the 6th of July at 17:00 UTC with the following agenda.

Upstream patches and bugs filed

• Bernhard M. Wiedemann:
  • gnuradio + volk
  • pymol
  • distorm
  • qtscriptgenerator
  • cpython
  • x3270 x3270
  • sphinx
  • obs-service-tar_scm
  • osc
  • matplotlib merged
  • pyparted merged
  • bjoern merged

Reviews of unreproducible packages 1 package review has been added, 19 have been updated and 2 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (1)
• Edmund Grimley Evans (1)

diffoscope development

• Chris Lamb:
  • Document the 'h' variable in our raw_feeder
  • Split diffoscope.difference into feeders
  • Tidy diffoscope/difference.py
• Ximin Luo:
  • Add a PartialString class
  • More ydiff/linediff from diffoscope. difference => diff to group unified_diff related things together
  • difference: has_children -> has_visible_children, and take into account comments
  • Add various traverse_ methods to Difference
  • Move side-by-side and linediff algorithms to difference.py
  • Refactor html-dir presenter to be a class instance, avoiding global state
  • html-dir: show/hide diff comments, which can be very large

tests.reproducible-builds.org As you might have noticed, Debian stretch was released last week. Since then, Mattia and Holger renamed our testing suite to stretch and added a buster suite so that we keep our historic results for stretch visible and can continue our development work as usual. In this sense, happy hacking on buster; may it become the best Debian release ever and hopefully the first reproducible one!

• Vagrant Cascadian:
  • Proposed reducing the number of concurrent builds each armhf machine runs, to reduce out-of-memory errors, crashes and false positives with FTBFS on armhf, reducing the number of armhf build workers from 70 to 51.
• Valerie Young: Add highlighting in navigation for the new nodes health pages.
• Mattia Rizzolo:
  • Do not dump database ACL in the backups.
  • Deduplicate SSLCertificateFile directive into the common-directives-ssl macro
  • Apache: t.r-b.o: redirect /testing/ to /stretch/
  • db: s/testing/stretch/g
  • Start adding code to test buster...
• Holger Levsen:
  • Update README.infrastructure to explain who has root access where.
  • reproducible_nodes_info.sh: correctly recognize zero builds per day.
  • Add build nodes health overview page, then split it in three: health overview, daily munin graphs and weekly munin graphs.
  • reproducible_worker.sh: improve handling of systemctl timeouts.
  • reproducible_build_service: sleep less and thus restart failed workers sooner.
  • Replace ftp.(de uk us).debian.org with deb.debian.org everywhere.
  • Performance page: also show local problems with _build_service.sh (which are autofixed after a maximum of 133.7 minutes).
  • Rename nodes_info job to html_nodes_info.
  • Add new node health check jobs, split off from maintenance jobs, run every 15 minutes.
    • Add two new checks: 1. for correct future (2019 is incorrect atm, and we sometimes got that). 2.) for writeable /tmp (sometimes happens on borked armhf nodes).
  • Add jobs for testing buster.
  • s/testing/stretch/g in all the code.
  • Finish the code to deal with buster.
  • Teach jessie and Ubuntu 16.04 how to debootstrap buster.

Axel Beckert is currently in the process of setting up eight LeMaker HiKey960 boards. These boards were sponsored by Hewlett Packard Enterprise and will be hosted by the SOSETH students association at ETH Zurich. Thanks to everyone involved here and also thanks to Martin Michlmayr and Steve Geary who initiated getting these boards to us. Misc. This week's edition was written by Chris Lamb, Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Debian is now working on the NVIDIA Jetson TX1 developer kit, a development board based on the Tegra X1 chip (Tegra 210), a 64-bit ARM chip. We have a pre-built u-boot image in Debian as well as kernel and installer support. There are some minor kernel glitches but NVIDIA is very active upstream and I hope they'll get resolved soon. The Jetson TX1 developer kit makes a pretty good 64-bit ARM development platform. The board is supported in mainline u-boot and the mainline kernel and NVIDIA are pretty responsive to bug reports. Unfortunately, a proprietary blob is required for USB (and Ethernet is connected via USB). If you're interested in a good 64-bit ARM development platform, give Debian on the Jetson TX1 development kit a try.

As I have said here a couple of times already, I am teaching a diploma course on embedded Linux at UNAM, and one of the modules I'm teaching (with Sandino Araico) is the boot process. We focus on ARM for obvious reasons, and while I have done my reading on the topic, I am very far from considering myself an expert. So, after attending Martin Michlmayr's Debian on ARM devices talk, I decided to do its subtitles as part of my teaching job. This talk gives a great panorama on what actually has to happen in order to get an ARM machine to boot, and how support for new ARM devices comes around to Linux in general and to Debian in particular Perfect for our topic! But my students are not always very fluent in English, so giving a hand is always most welcome. In case any of you dear readers didn't know, we have a DebConf subtitling team. Yes, our work takes much longer to reach the public, and we have no hopes whatsoever in getting it completed, but every person lending a hand and subtitling a talk that they thought was interesting helps a lot to improve our talks' usability. Even if you don't have enough time to do the whole talk (we are talking about some 6hr per 45 minute session), adding a bit of work is very very very welcome. So...

• Do you want to follow Martin's talk with subtitles?
  • Get the video file
  • Get the subtitles
  • Use your favorite program In my case:
    $ vlc Debian_on_ARM_devices_2.webm --sub-file Debian_on_ARM_devices.srt
  • [update] Or watch online at Amara!
• Do you want to help the subtitling effort?
  • Read the Wiki page on the Subtitling team work (feel free to fill in bits of missing information you will surely find!)
  • Choose a video from the list to work on
  • You can either work online on the DebConf team of the Amara webapp or join the Alioth team and contribute directly to the Git tree

Enjoy And thanks in advance for your work!

I became interested in running Debian on NVIDIA's Tegra platform recently. NVIDIA is doing a great job getting support for Tegra upstream (u-boot, kernel, X.org and other projects). As part of ensuring good Debian support for Tegra, I wanted to install Debian on a Jetson TK1, a development board from NVIDIA based on the Tegra K1 chip (Tegra 124), a 32-bit ARM chip. Ian Campbell enabled u-boot and Linux kernel support and added support in the installer for this device about a year ago. I updated some kernel options since there has been a lot of progress upstream in the meantime, performed a lot of tests and documented the installation process on the Debian wiki. Wookey made substantial improvements to the wiki as well. If you're interested in a good 32-bit ARM development platform, give Debian on the Jetson TK1 a try. There's also a 64-bit board. More on that later...

The majority of NAS devices supported in Debian are based on Marvell's Kirkwood platform. This platform is quite dated now and can only run Debian's armel port. Debian now supports the Seagate Personal Cloud and Seagate NAS devices. They are based on Marvell's Armada 370, a platform which can run Debian's armhf port. Unfortunately, even the Armada 370 is a bit dated now, so I would not recommend these devices for new purchases. If you have one already, however, you now have the option to run native Debian. There are some features I like about the Seagate NAS devices:

• Network console: you can connect to the boot loader via the network. This is useful to load Debian or to run recovery commands if needed.
• Mainline support: the devices are supported in the mainline kernel.
• Good contacts: Seagate engineer Simon Guinot is interested in Debian support and is a joy to work with. There's also a community for LaCie NAS devices (Seagate acquired LaCie).

If you have a Seagate Personal Cloud and Seagate NAS, you can follow the instructions on the Debian wiki. If Seagate releases more NAS devices on Marvell's Armada platform, I intend to add Debian support.

What happened in the Reproducible Builds effort between April 24th and 30th 2016. Media coverage Reproducible builds were mentioned explicitly in two talks at the Mini-DebConf in Vienna:

• Martin Michlmayr had a talk in which he presented an overview about innovations and changes in Debian in the last years. Martin expressed his disappointment that there was no talk from us in Vienna (we'll fix this at DebConf16 in Cape Town) and described the reproducible builds work as "a real innovation". His talk is very much worth seeing, whatever your current perspective, it might change your view on Debian.
• Ben Hutchings explains how Secure Boot will use signed kernels via separate signature packages and how this was designed with reproducible builds in mind.

Aspiration together with the OTF CommunityLab released their report about the Reproducible Builds summit in December 2015 in Athens. Toolchain fixes Now that the GCC development window has been opened again, the SOURCE_DATE_EPOCH patch by Dhole and Matthias Klose to address the issue timestamps_from_cpp_macros (__DATE__ / __TIME__) has been applied upstream and will be released with GCC 7. Following that Matthias Klose also has uploaded gcc-5/5.3.1-17 and gcc-6/6.1.1-1 to unstable with a backport of that SOURCE_DATE_EPOCH patch. Emmanuel Bourg uploaded maven/3.3.9-4, which uses SOURCE_DATE_EPOCH for the maven.build.timestamp. (SOURCE_DATE_EPOCH specification) Other upstream changes Alexis Bienven e submitted a patch to Sphinx which extends SOURCE_DATE_EPOCH support for copyright years in generated documentation. Packages fixed The following 12 packages have become reproducible due to changes in their build dependencies: hhvm jcsp libfann libflexdock-java libjcommon-java libswingx1-java mobile-atlas-creator not-yet-commons-ssl plexus-utils squareness svnclientadapter The following packages have became reproducible after being fixed:

• anope/2.0.3-2 by Dominic Hargreaves, original patch by Alexis Bienven e.
• apparmor-profiles-extra/1.7 by intrigeri, patch by Felix Geyer.
• basket/2.10~beta+git20160425.b77687f-1 by Luigi Toscano, original patch by Alexis Bienven e.
• bind-dyndb-ldap/8.0-2 by Timo Aaltonen.
• bliss/0.73-1 by Jerome Benoit.
• gap-alnuth/3.0.0-3 by Bill Allombert.
• gap-radiroot/2.7-2 by Bill Allombert.
• genometools/1.5.8+ds-3 by Sascha Steinbiss.
• gprbuild/2015-3 by Nicolas Boulenguez.
• gtkhash/0.7.0-3 by M nica Ram rez Arceda.
• josm/0.0.svn10161+dfsg-1 by Bas Couwenberg.
• libjgoodies-animation-java/1.4.3-1 by Markus Koschany.
• ltrsift/1.0.2-7 by Sascha Steinbiss.
• mutt/1.6.0-1 by Matteo F. Vescovi, original patch by Daniel Shahaf.
• netsniff-ng/0.6.1-1 by Kartik Mistry, original patch by Reiner Herrmann.
• netrw/1.3.2-3 by Giovanni Mascellani.
• openthesaurus/20160424-3 by Rene Engelhard, original patch by Dhole.
• php-pear/1:1.10.1+submodules+notgz-8 by Mathieu Parent.
• samba/2:4.4.2+dfsg-2 by Jelmer Vernoo .
• swift/2.7.0-3 by Ond ej Nov .
• tp-smapi/0.42-1 by Evgeni Golov, original patch by Chris Lamb.
• vifm/0.8.1a-0.1 by Ond ej Nov .
• xfonts-mplus/1:2.2.4-2 by Hideki Yamane, original patch by Chris Lamb.
• xpa/2.1.17-2 by Ole Streicher, original patch by Alexis Bienven e.

Some uploads have fixed some reproducibility issues, but not all of them:

• camitk/4.0.0~beta-1 by Emmanuel Promayon, original patch by Maria Valentina Marin.
• elastix/4.8-8 by Gert Wollny.
• freefem++/3.46+dfsg1-1 by Dimitrios Eftaxiopoulos, original patch by Alexis Bienven e.
• grib-api/1.15.0-1 by Alastair McKinstry, original patch by Santiago Vila.
• isorelax/20041111-9 by Emmanuel Bourg.
• libical/2.0.0-0.1 by Andreas Henriksson, original patch by Chris Lamb.
• sawfish/1:1.11.90-1 by Jose M Calhariz, original patch by Alexis Bienven e.
• singular/4.0.3-p1+ds-1 by Jerome Benoit.
• syslinux/3:6.03+dfsg-12 by Mattia Rizzolo, original patch by Reiner Herrmann.
• transdecoder/2.0.1+dfsg-3 by Michael R. Crusoe, original patch by Dhole.

Patches submitted that have not made their way to the archive yet:

• #822566 against stk by Alexis Bienven e: sort lists of object files for reproducible linking order.
• #822948 against shotwell by Alexis Bienven e: normalize tarball permissions and use locale/timezone-independent modification time.
• #822963 against htop by Alexis Bienven e: use SOURCE_DATE_EPOCH for embedded copyright year, which has before already been applied in git and upstream.

Package reviews 95 reviews have been added, 15 have been updated and 129 have been removed in this week. 22 FTBFS bugs have been reported by Chris Lamb and Martin Michlmayr. diffoscope development

• diffoscope 52~bpo8+1 has been uploaded to jessie-backports by Mattia Rizzolo, where it is currently waiting for NEW-approval.
• Support for the deb(5) format (uncompressed data.tar/control.tar, control.tar.xz) (Closes: #818414) has been completed by Reiner Herrmann in git.

strip-nondeterminism development

• Support for EPUB documents has been added (to the development version in git) by Holger Levsen, to address the timestamps_in_epub issue.

tests.reproducible-builds.org

• To monitor the uptodateness of diffoscope everywhere, tests checking this on FreeBSD, NetBSD & MacPorts were added. Tests on other distributions will be added once the relevant bugs in whohas are fixed in jessie. (h01ger)

Misc. Amongst the 29 interns who will work on Debian through GSoC and Outreachy there are four who will be contributing to Reproducible Builds for Debian and Free Software. We are very glad to welcome ceridwen, Satyam Zode, Scarlett Clark and Valerie Young and look forward to working together with them the coming months (and maybe beyond)! This week's edition was written by Reiner Herrmann and Holger Levsen and reviewed by a bunch of Reproducible builds folks on IRC.

What happened in the reproducible builds effort between February 28th and March 5th:

Toolchain fixes

• Antonio Terceiro uploaded gem2deb/0.27 that forces generated gemspecs to use the date from debian/changelog.
• Antonio Terceiro uploaded gem2deb/0.28 that forces generated gemspecs to have their contains file lists sorted.
• Robert Luberda uploaded ispell/3.4.00-5 which make builds of hashes reproducible.
• C dric Boutillier uploaded ruby-ronn/0.7.3-4 which will make the output locale agnostic. Original patch by Chris Lamb.
• Markus Koschany uploaded spring/101.0+dfsg-1. Fixed by Alexandre Detiste.

Ximin Luo resubmitted the patch adding the --clamp-mtime option to Tar on Savannah's bug tracker. Lunar rebased our experimental dpkg on top of the current master branch. Changes in the test infrastructure are required before uploading a new version to our experimental repository. Reiner Herrmann rebased our custom texlive-bin against the latest uploaded version.

Packages fixed The following 77 packages have become reproducible due to changes in their build dependencies: asciidoctor, atig, fuel-astute, jekyll, libphone-ui-shr, linkchecker, maven-plugin-testing, node-iscroll, origami-pdf, plexus-digest, pry, python-avro, python-odf, rails, ruby-actionpack-xml-parser, ruby-active-model-serializers, ruby-activerecord-session-store, ruby-api-pagination, ruby-babosa, ruby-carrierwave, ruby-classifier-reborn, ruby-compass, ruby-concurrent, ruby-configurate, ruby-crack, ruby-css-parser, ruby-cucumber-rails, ruby-delorean, ruby-encryptor, ruby-fakeweb, ruby-flexmock, ruby-fog-vsphere, ruby-gemojione, ruby-git, ruby-grack, ruby-htmlentities, ruby-jekyll-feed, ruby-json-schema, ruby-listen, ruby-markerb, ruby-mathml, ruby-mini-magick, ruby-net-telnet, ruby-omniauth-azure-oauth2, ruby-omniauth-saml, ruby-org, ruby-origin, ruby-prawn, ruby-pygments.rb, ruby-raemon, ruby-rails-deprecated-sanitizer, ruby-raindrops, ruby-rbpdf, ruby-rbvmomi, ruby-recaptcha, ruby-ref, ruby-responders, ruby-rjb, ruby-rspec-rails, ruby-rspec, ruby-rufus-scheduler, ruby-sass-rails, ruby-sass, ruby-sentry-raven, ruby-sequel-pg, ruby-sequel, ruby-settingslogic, ruby-shoulda-matchers, ruby-slack-notifier, ruby-symboltable, ruby-timers, ruby-zip, ticgit, tmuxinator, vagrant, wagon, yard. The following packages became reproducible after getting fixed:

• air-quality-sensor/0.1.4-1 uploaded by Benedikt Wildenhain, fixed upstream, original patch by Chris Lamb.
• device3dfx/2013.08.08-4 by Guillem Jover.
• fldigi/3.23.08-1 by Kamal Mostafa.
• fltk1.1/1.1.10-22 by Aaron M. Ucko.
• freeimage/3.17.0+ds1-2 by Ghislain Antony Vaillant.
• gimagereader/3.1.2+git368fa8f-2 by Philip Rinn.
• ginkgocadx/3.7.5-1 by Gert Wollny, fixed upstream.
• jadetex/3.13-17 by Norbert Preining.
• opensips/2.1.2-1 by Razvan Crainea.
• ruby-sqlite3/1.3.11-2 uploaded by C dric Boutillier, original patch by Lunar.
• runawk/1.6.0-2 uploaded by Andrew Shadura, patch by Reiner Herrmann.
• systraq/20160303-1 by Joost van Baal-Ili .

Some uploads fixed some reproducibility issues, but not all of them:

• auto-multiple-choice/1.2.1-4 by Georges Khaznadar.
• avfs/1.0.3-1 uploaded by Michael Meskes, original patch by Chris Lamb.
• console-setup/1.138 uploaded by Anton Zinoviev, original patch by Reiner Herrmann.
• gromacs/5.1.2-1 by Nicholas Breen.
• mrrescue/1.02c-2 by Alexandre Detiste.
• usb-modeswitch-data/20160112-2 by Didier Raboud.

Patches submitted which have not made their way to the archive yet:

• #816209 on elog by Reiner Herrmann: use printf instead of echo which is shell-independent.
• #816214 on python-pip by Reiner Herrmann: removes timestamp from generated Python scripts.
• #816230 on rows by Reiner Herrmann: tell grep to always treat the input as text.
• #816232 on eficas by Reiner Herrmann: use printf instead of echo which is shell-independent.

Florent Daigniere and bancfc reported that linux-grsec was currently built with GRKERNSEC_RANDSTRUCT which will prevent reproducible builds with the current packaging.

tests.reproducible-builds.org pbuilder has been updated to the last version to be able to support Build-Depends-Arch and Build-Conflicts-Arch. (Mattia Rizzolo, h01ger) New package sets have been added for Subgraph OS, which is based on Debian Stretch: packages and build dependencies. (h01ger) Two new armhf build nodes have been added (thanks Vagrant Cascadian) and integrated in our Jenkins setup with 8 new armhf builder jobs. (h01ger)

strip-nondeterminism development strip-nondeterminism version 0.016-1 was released on Sunday 28th. It will now normalize the POT-Creation-Date field in GNU Gettext .mo files. (Reiner Herrmann) Several improvements to the packages metadata have also been made. (h01ger, Ben Finney)

Package reviews 185 reviews have been removed, 91 added and 33 updated in the previous week. New issue: fileorder_in_gemspec_files_list. 43 FTBFS bugs were reported by Chris Lamb, Martin Michlmayr, and gregor herrmann.

Misc. After merging the patch from Dhiru Kholia adding support for SOURCE_DATE_EPOCH in rpm, Florian Festi opened a discussion on the rpm-ecosystem mailing list about reproducible builds. On March 4th, Lunar gave an overview of the general reproducible builds effort at the Internet Freedom Festival in Valencia.

What happened in the reproducible builds effort between January 17th and January 23rd:

Toolchain fixes James McCoy uploaded subversion/1.9.3-2 which removes -Wdate-time from CPPFLAGS passed to swig enabling several packages to build again. The switch made in binutils/2.25-6 to use deterministic archives by default had the unfortunate effect of breaking a seldom used feature of make. Manoj Srivastava asked on debian-devel the best way to communicate the changes to Debian users. Lunar quickly came up with a patch that displays a warning when Make encounters deterministic archives. Manoj made it available in make/4.1-2 together with a NEWS file advertising the change. Following Guillem Jover's comment on the latest patch to make mtimes of packaged files deterministic, Daniel Kahn Gillmor updated and extended the patch adding the --clamp-mtime option to GNU Tar. Mattia Rizzolo updated texlive-bin in the reproducible experimental repository.

Packages fixed The following packages became reproducible after getting fixed:

• apt-cacher-ng/0.8.9-1 by Eduard Bloch.
• beep/1.3-4 by Rhonda D'Vine, obsolete patch by Chris Lamb.
• clblas/2.10-1~exp1 by Ghislain Antony Vaillant.
• cortado/0.6.0-3 uploaded by Markus Koschany, original patch by Dhole.
• magic/8.0.210-2 uploaded by Roland Stigge, original patch by Chris Lamb.
• mailagent/1:3.1-81-4 by Manoj Srivastava.
• maven-shade-plugin/2.4.3-1 by Emmanuel Bourg.
• nekohtml/1.9.22-1 by Emmanuel Bourg.
• pd-iemguts/0.2-1 by IOhannes m zm lnig.
• pd-mediasettings/0.1.1-1 by IOhannes m zm lnig.
• polymake/3.0-1 uploaded by David Bremner, fixed upstream, original patch by Chris Lamb.
• wyrd/1.4.6-4 by Rhonda D'Vine.

Some uploads fixed some reproducibility issues, but not all of them:

• desktop-profiles/1.4.21 uploaded by Petter Reinholdtsen, original patch by Chris Lamb.
• gradle/2.10-1 by Kai-Chung Yan.

Patches submitted which have not made their way to the archive yet:

• #811285 on strace by Reiner Herrmann: sort symbol list using the C locale.
• #812428 on libgcrypt20 by Lunar: add support for SOURCE_DATE_EPOCH.

reproducible.debian.net Transition from reproducible.debian.net to the more general tests.reproducible-builds.org has started. More visual changes are coming. (h01ger) A plan on how to run tests for F-Droid has been worked out. (hc, mvdan, h01ger) A first step has been made by adding a Jenkins job to setup an F-Droid build environment. (h01ger)

diffoscope development diffoscope 46 has been released on January 19th, followed-up by version 47 made available on January 23rd. Try it online at try.diffoscope.org! The biggest visible change is the improvement to ELF file handling. Comparisons are now done section by section, using the most appropriate tool and options to get meaningful results, thanks to Dhole's work and Mike Hommey's suggestions. Also suggested by Mike, symbols for IP-relative ops are now filtered out to remove clutter. Understanding differences in ELF files belonging to Debian packages should also be much easier as diffoscope will now try to extract debug information from the matching dbgsym package. This means objdump disassembler should output line numbers for packages built with recent debhelper as long as the associated debug package is in the same directory. As diff tends to consume huge amount of memory on large inputs, diffoscope has a limit in place to prevent crashes. diffoscope used to display a difference every time the limit was hit. Because this was confusing in case there were actually no differences, a hash is now internally computed to only report a difference when one exists. Files in archives and other container members are now compared in the original order. This should not matter in most case but overall give more predictable results. Debian .buildinfo files are now supported. Amongst other minor fixes and improvements, diffoscope will now properly compare symlinks in directories. Thanks Tuomas Tynkkynen for reporting the problem.

Package reviews 70 reviews have been removed, 125 added and 33 updated in the previous week, gcc-5 amongst others. 25 FTBFS issues have been filled by Chris Lamb, Daniel Stender, Martin Michlmayr.

Misc. The 16th FOSDEM will happen in Brussels, Belgium on January 30-31st. Several talks will be about reproducible builds: h01ger about the general ecosystem, Fabian Keil about the security oriented ElectroBSD, Baptiste Daroussin about FreeBSD packages, Ludovic Court s about Guix.