Shockingly enough, my focus started out on Gitano once more. We managed a 1.1 release of Gitano during the Debian conference's "camp" which occurs in the week before the conference. This was a joint effort of myself, Richard Maw, and Richard Ipsum. I have to take my hat off to Richard Maw, because without his dedication to features, 1.1 would lack some stuff which Richard Ipsum proposed around ruleset support for basic readers/writers and frankly 1.1 would be a weaker release without it. Because of the debconf situation, we didn't have a Gitano developer day which, while sad, didn't slow us down much...

• Once again, we reviewed our current task state
• I submitted a series which fixed our test suite for Git 2.13 which was an FTBFS bug submitted against the Debian package for Gitano. Richard Maw reviewed and merged it.
• Richard Maw supplied a series to add testing for dangling HEAD syndrome. I reviewed and merged that.
• Richard Maw submitted a patch to improve the auditability of the 'as' command and I reviewed and merged that.
• Richard Ipsum submitted a patch to add reader/writer configs to ease simple project management in Gitano. I'm not proud to say that I was too busy to look at this and ended up saying it was unlikely it'd get in. Richard Maw, quite rightly, took umbrage at that and worked on the patch, eventually submitting a new series with tests which I then felt obliged to review and I merged the series eventually.

  This is an excellent example of where just because one person is too busy doesn't mean that a good idea should be dropped, and I am grateful to Richard Maw for getting this work mergeable and effectively guilt-tripping me into reviewing/merging. This is a learnable moment for me and I hope to do better into the future.

• During all that time, I was working on a plugin to support git-multimail.py in Gitano. This work ranged across hooks and caused me to spend a long time thinking about the semantics of configuration overriding etc. Fortunately I got there in the end, and with a massive review effort from Richard Maw, we got it merged into Gitano.
• Finally I submitted a patch which caused the tests we run in Gitano to run from an 'install' directory which ensures that we catch bugs such as those which happened in earlier series where we missed out rules files for installation etc. Richard Maw reviewed and merged that.
• And then we released the new version of Gitano and subsidiary libraries. There was Luxio version 13 which switched us to readdir() from readdir_r() thanks to Richard Ipsum; Gall 1.3 which contained a bunch of build cleanups, and also a revparse_single() implementation in the C code to speed things up thanks to Richard Maw; Supple 1.0.8 which improved wrapper environment cleanups thanks to Richard Ipsum, allowed baking of paths in which means Nix is easier to support (again thanks to Richard Ipsum), fixed setuid handling so that Nix is easier to support (guess what? Richard Ipsum again); Lace 1.4 which now verifies definition names in allow/deny/anyof/allof and also produces better error messages from nested includes. And, of course, Gitano 1.1 whose changes were somewhat numerous and so you are invited to read them in the Gitano NEWS file for the release.

Not Gitano Of course, not everything I did in August was Gitano related. In fact once I had completed the 1.1 release and uploaded everything to Debian I decided that I was going to take a break from Gitano until the next developer day. (In fact there's even some patch series still unread on the mailing list which I will get to when I start the developer day.) I have long been interested in STM32 microcontrollers, using them in a variety of projects including the Entropy Key which some of you may remember. Jorge Aparicio was working on Cortex-M3 support (among other microcontrollers) in Rust and he then extended that to include a realtime framework called RTFM and from there I got interested in what I might be able to do with Rust on STM32. I noticed that there weren't any pure Rust implementations of the USB device stack which would be necessary in order to make a device, programmed in Rust, appear on a USB port for a computer to control/use. This tweaked my interest. As many of my readers are aware, I am very bad at doing things without some external motivation. As such, I then immediately offered to give a talk at a conference which should be happening in November, just so that I'd be forced to get on with learning and implementing the stack. I have been chronicling my work in this blog, and you're encouraged to go back and read them if you have similar interests. I'm sure that as my work progresses, I'll be doing more and more of that and less of Gitano, for at least the next two months. To bring that into context as F/LOSS work, I did end up submitting some patches to Jorge's STM32F103xx repository to support a couple more clock configuration entries so that USB and ADCs can be set up cleanly. So at least there's that.

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donors (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I was allocated 12h and during this time I did 4 days of front desk handling CVE triage (28 commits to the security tracker). I had a bit of time left and I opted to work on a package that had been lingering for a while: exiv2. It turns out the security researchers who requested the CVE did not even contact the upstream author so I opened 12 tickets on GitHub. The upstream author was unaware of those issues and is relatively unfamiliar with the general process of handling security updates. I started the work of reproducing each issue and so far they only affect the version 0.26 in experimental. Misc Debian/Kali work live-build and live-config. I pushed a few updates: dropping the useless xorriso hardlinks option (as discussed in https://bugs.kali.org/view.php?id=4109), adding a .disk/mkisofs file on request of Thomas Schmitt, fixing a severe issue with the handling of locales configuration that broke wayland sessions entirely. open-vm-tools and vmwgfx. The switch of GNOME to Wayland by default resulted in multiple regressions reported by Kali users, in particular for VMWare users where desktop resizing was no longer working. There was a patch available but it did not work for me, so I worked with Thomas Hellstrom (of VMWare) to identify the problems and he provided me an updated patch. I submitted this patch to Debian too (bug report, pull request). Linux 4.12 also showed another regression for VMWare users where the screen would not be refreshed/updated when you are using Wayland/KMS. I did multiple tests for Thomas and provided the requested data so that they could create a fix (which I incorporated into Kali and should come to Debian through the upstream stable tree). Packaging. I uploaded zim 0.67 to unstable. I fixed an RC bug on shiboken to get pyside and ubertooth back into testing. I had to hack the package to use gcc-6 on mips64el because that architecture is suffering from a severe gcc bug which probably broke a large part of the code compiled since the switch to gcc-7 (and which triggered a test failure in shiboken, fortunately) I wonder if anybody will make sure to recompile all packages that might have been misbuilt. Infrastructure. In a discussion on debian-devel, the topic of using tracker.debian.org to store who is maintaining what came up again. I responded to let know that this is something that I d like to see done and that I have already taken measures to go into this direction. I wanted to make an experiment with my zim package but quickly came on a problem with ftpmaster s lintian auto-rejects (which I submitted in #871575). The BTS is now linking to tracker.debian.org on its web interface. To continue and give a push to this move, I scanned all the files in the qa SVN repository and updated many occurrences of packages.qa.debian.org with tracker.debian.org. I also spotted a small problem in the way we handle autoremovals mails in tracker.debian.org, we often get them twice: I filed #871683 to get this fixed on release.debian.org. Bug reports. vmdebootstrap creates unbootable qemu image (#872999). bugs in udebs are not shown on view by source package (#872784). New upstream release of ethtool (#873692). Upstream bugreport on systemd: support a systemd.swap=no boot command-line option. I also shared some of my ideas/dreams in #859867 speaking of a helper tool to setup and maintain up-to-date build chroots and autopkgtest qemu images. More bug fixes and pull requests. I created a patch to fix a build failure of systemd when /tmp is an overlayfs (#854400, the pull request has been discarded). I fixed the RC bug #853570 on ncrack and forwarded my changes upstream (here and here). Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Review: Regeneration, by Julie E. Czerneda

Series:	Species Imperative #3
Publisher:	DAW
Copyright:	2006
ISBN:	0-7564-0345-6
Format:	Hardcover
Pages:	543

This is the third book of the Species Imperative trilogy, and this is the type of trilogy that's telling a single story in three books. You don't want to read this out of order, and I'll have to be cautious about aspects of the plot to not spoil the earlier books. Mac is still recovering from the effects of the first two books of the series, but she's primarily worried about a deeply injured friend. Worse, that friend is struggling to explain or process what's happened, and the gaps in her memory and her very ability to explain may point at frightening, lingering risks to humanity. As much as she wants to, Mac can't give her friend all of her focus, since she's also integral to the team trying to understand the broader implications of the events of Migration. Worse, some of the non-human species have their own contrary interpretations that, if acted on, Mac believes would be desperately risky for humanity and all the other species reachable through the transects. That set of competing priorities and motivations eventually sort themselves out into a tense and rewarding multi-species story, but they get off to an awkward start. The first 150 pages of Regeneration are long on worry, uncertainty, dread, and cryptic conversations, and short on enjoyable reading. Czerneda's recaps of the previous books are appreciated, but they weren't very smoothly integrated into the story. (I renew my occasional request for series authors to include a simple plot summary of the previous books as a prefix, without trying to weave it into the fiction.) I was looking forward to this book after the excellent previous volumes, but struggled to get into the story. That does change. It takes a bit too long, with a bit too much nameless dread, a bit too much of an irritating subplot between Fourteen and Oversight that I didn't think added anything to the book, and not enough of Mac barreling forward doing sensible things. But once Mac gets back into space, with a destination and a job and a collection of suspicious (or arrogant) humans and almost-incomprehensible aliens to juggle, Czerneda hits her stride. Czerneda doesn't entirely avoid Planet of the Hats problems with her aliens, but I think she does better than most of science fiction. Alien species in this series do tend to be a bit all of a type, and Mac does figure them out by drawing conclusions from biology, but those conclusions are unobvious and based on Mac's mix of biological and human social intuition. They refreshingly aren't as simple as biology completely shaping culture. (Czerneda's touch is more subtle than James White's Sector General, for example.) And Mac has a practical, determined, and selfless approach that's deeply likable and admirable. It's fun as a reader to watch her win people over by just being competent, thoughtful, observant, and unrelentingly ethical. But the best part of this book, by far, are the Sinzi. They first appeared in the second book, Migration, and seemed to follow the common SF trope of a wise elder alien race that can bring some order to the universe and that humanity can learn from. They, or more precisely the one Sinzi who appeared in Migration, was very good at that role. But Czerneda had something far more interesting planned, and in Regeneration they become truly alien in their own right, with their own nearly incomprehensible way of viewing the universe. There are so many ways that this twist can go wrong, and Czerneda avoids all of them. She doesn't undermine their gravitas, nor does she elevate them to the level of Arisians or other semi-angelic wise mentors of other series. Czerneda makes them different in profound ways that are both advantage and disadvantage, pulls that difference into the plot as a complicating element, and has Mac stumble yet again into a role that is accidentally far more influential than she intends. Mac is the perfect character to do that to: she has just the right mix of embarrassment, ethics, seat-of-the-pants blunt negotiation skills, and a strong moral compass. Given a lever and a place to stand, one can believe that Mac can move the world, and the Sinzi are an absolutely fascinating lever. There are also three separate, highly differentiated Sinzi in this story, with different goals, life experience, personalities, and levels of gravitas. Czerneda's aliens are good in general, but her focus is usually more on biology than individual differentiation. The Sinzi here combine the best of both types of character building. I think the ending of Regeneration didn't entirely work. After all the intense effort the characters put into understanding the complexity of the universe over the course of the series, the denouement has a mopping-up feel and a moral clarity that felt a bit too easy. But the climax has everything I was hoping for, there's a lot more of Mac being Mac, and I loved every moment of the Sinzi twist. Now I want a whole new series exploring the implications of the Sinzi's view of the universe on the whole history of galactic politics that sat underneath this story. But I'll settle for moments of revelation that sent shivers down my spine. This is a bit of an uneven book that falls short of its potential, but I'll remember it for a long time. Add it on to a deeply rewarding series, and I will recommend the whole package unreservedly. The Species Imperative is excellent science fiction that should be better-known than it is. I still think the romance subplot was unfortunate, and occasionally the aliens get too cartoony (Fourteen, in particular, goes a bit too far in that direction), but Czerneda never lingers too long on those elements. And the whole work is some of the best writing about working scientific research and small-group politics that I've read. Highly recommended, but read the whole series in order. Rating: 9 out of 10

Here is my monthly update covering what I have been doing in the free software world in August 2017 (previous month):

• Created ZeroCoolOS, a live operating system that plays the film Hackers (1995) on a continuous loop.
• Sent a patch for pristine-tar to allow storage of detached upstream signatures. (#871809)
• Worked more on Lintian, a static analysis tool for Debian packages, reporting on various errors, omissions and quality-assurance issues to the maintainer (previous changes):
  • Fix an apache2-unparsable-dependency false positive by allowing periods in dependency names. (#873701)
  • Ignore "repacked" packages when checking for upstream source tarball signatures as they will never match.
  • Downgrade the severity of orig-tarball-missing-upstream-signature. (#870722)
  • From a suggestion by Theodore Ts'o, expand the explanation of orig-tarball-missing-upstream-signature to include the location of where dpkg-source looks.
  • Address a number of issues in the copyright-year-in-future tag including preventing false positives in port numbers, email addresses, ISO standard numbers and street addresses (#869788), as well as "meta" or testing statements (#873323). In addition, report all violating years in a line and expand the testsuite.
  • Don't match quoted "FIXME" variants of file-contains-fixme-placeholder (#870199), avoid checking copyright_hints files (#872843) and downgrade the tag's severity.
  • Apply a patch from Alex Muntada to recommend "substr" over of "substring" in mentions-deprecated-usr-lib-perl5-directory. (#871767)
  • Prevent missing-build-dependency-for-dh_-command false positives exposed by following the advice in useless-autoreconf-build-depends. (#869541)
  • Ensure readme-debian-contains-debmake-template also checks for files containing "Automatically generated by debmake".
  • Check python3-foo packages have a Section: python, not just python2-foo. (#870272)
  • Check for packages shipping compiled Java class files. (#873211)
  • Additionally consider .cljc files to avoid codeless-jar warnings. (#870649)
  • Prevent desktop-entry-lacks-keywords-entry false positives for Link and Directory-style .desktop files. (#873702)
  • Split out Python checks from checks/scripts.pm check to a new, source check of type source.
  • Check for python-foo without a corresponding python3-foo package. (#870681)
  • Complain about packages that Build-Depend on python-sphinx only. (#870730)
  • Warn about packages that alternatively Build-Depend on the Python 2 and Python 3 versions of Sphinx. (#870758)
  • Check for packages that depend on Python 2.x. (#870822)
  • Correct false positives in unconditional-use-of-dpkg-statoverride by detecting "if !" as a shell prefix. (#869587)
  • Alert on for missing calls to dpkg-maintscript-helper(1) in maintainer scripts. (#872042)
  • Check for packages using sensible-utils without declaring a dependency after splitting from debianutils. (#872611)
  • Warn about scripts using nodejs as an interpreter now that the nodejs script provides /usr/bin/node. (#873096)
  • Remove recommendations to add a Testsuite: autopkgtest field to debian/control and emit a new tag the package if it does so. (#865531)
  • Recognise autopkgtest-pkg-elpa as a valid test suite. (#873458)
  • Add note to /etc/bash_completion.d's obsolete path warning output regarding stricter filename requirements. (#814599)
  • Add 4.0.1 and 4.1.0 as known Policy standards versions.
  • Apply a patch from Maia Everett to avoid British spellings under the en_US locale. (#868897)
  • Stop emitting maintainer,uploader -address-causes-mail-loops for @packages.debian.org addresses. (#871575)
  • Modify Lintian::Data's all subroutine to always return keys in insertion order.
  • Apply a patch from Steve Langasek to accomodate binutils outputting symbols in a different format on the ppc64el architecture. (#869750)
  • Add an explicit test for packages including external fonts via the Google Font and TypeKit APIs. (#873434)
  • Add missing entries in internal Test-For fields to make development/testing workflow less error-prone.
• Sent three pull requests to git-buildpackage, a tool to assist in Debian packaging from Git repositories:
  • Make pq --abbrev= configurable. (#872351)
  • Use build profiles to avoid installation of test dependencies. (#31)
  • Correct "allow to" grammar. (#30)
• Updated travis.debian.net (my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform for testing):
  • Move away from deb.debian.org; Travis appears to be using a HTTP proxy that strips SRV records. (commit)
  • Highlight double quotes are required for TRAVIS_DEBIAN_EXTRA_REPOSITORY. (commit)
  • Use force-unsafe-io. (commit)
  • Clarify docs when upstream already has a travis.yml file. (#46)
  • Make documentation easier to copy-paste. (commit)
• Merged a pull request in django-slack, my library to easily post messages to the Slack group-messaging utility, where instantiation of a SlackException was failing. (#71)
• Assigned two pull requests to the Redis key-value database store to correct "did not received" and "faield" typos. (#4216 & #4215).

---

Ohai, I have recently started to serve as a Feature Editor for the ACM XRDS magazine. As such, I was also invited to post some general blog posts on XRDS blog And I just started yesterday by posting about DebConf. I'm not going to pull (or mention) each of my posts in my main blog, nor will I syndicate it to Planet Debian (where most of my readership comes from), although I did add it to my dlvr.it account (that relays my posts to Twitter and Facebook, for those of you that care about said services). This mention is a one-off thing. So, if you want to see yet another post explaining what is DebConf and what is Debian to the wider public, well... That's what I came up with :)

Ohai, I have recently started to serve as a Feature Editor for the ACM XRDS magazine. As such, I was also invited to post some general blog posts on XRDS blog And I just started yesterday by posting about DebConf. I'm not going to pull (or mention) each of my posts in my main blog, nor will I syndicate it to Planet Debian (where most of my readership comes from), although I did add it to my dlvr.it account (that relays my posts to Twitter and Facebook, for those of you that care about said services). This mention is a one-off thing. So, if you want to see yet another post explaining what is DebConf and what is Debian to the wider public, well... Thate's what I came up with :) [Update]: Of course, I wanted to thank Aigars Mahinovs for the photos I used on that post. Have you looked at them all? I spent a moste enjoyable time going through them :-]

In this, our next exciting installment of STM32 and Rust for USB device drivers, we're going to look at what the STM32 calls the 'packet memory area'. If you've been reading along with the course, including reading up on the datasheet content then you'll be aware that as well as the STM32's normal SRAM, there's a 512 byte SRAM dedicated to the USB peripheral. This SRAM is called the 'packet memory area' and is shared between the main bus and the USB peripheral core. Its purpose is, simply, to store packets in transit. Both those IN to the host (so stored queued for transmission) or OUT from the host (so stored, queued for the application to extract and consume). It's time to actually put hand to keyboard on some Rust code, and the PMA is the perfect starting point, since it involves two basic structures. Packets are the obvious first structure, and they are contiguous sets of bytes which for the purpose of our work we shall assume are one to sixty-four bytes long. The second is what the STM32 datasheet refers to as the BTABLE or Buffer Descriptor Table. Let's consider the BTABLE first.

The Buffer Descriptor Table The BTABLE is arranged in quads of 16bit words. For "normal" endpoints this is a pair of descriptors, each consisting of two words, one for transmission, and one for reception. The STM32 also has a concept of double buffered endpoints, but we're not going to consider those in our proof-of-concept work. The STM32 allows for up to eight endpoints (EP0 through EP7) in internal register naming, though they support endpoints numbered from zero to fifteen in the sense of the endpoint address numbering. As such there're eight descriptors each four 16bit words long (eight bytes) making for a buffer descriptor table which is 64 bytes in size at most.

Buffer Descriptor Table

Byte offset in PMA	Field name	Description
(EPn * 8) + 0	USB_ADDRn_TX	The address (inside the PMA) of the TX buffer for EPn
(EPn * 8) + 2	USB_COUNTn_TX	The number of bytes present in the TX buffer for EPn
(EPn * 8) + 4	USB_ADDRn_RX	The address (inside the PMA) of the RX buffer for EPn
(EPn * 8) + 6	USB_COUNTn_RX	The number of bytes of space available for the RX buffer for EPn (and once received, the number of bytes received)

The TX entries are trivial to comprehend. To transmit a packet, part of the process involves writing the packet into the PMA, putting the address into the appropriate USB_ADDRn_TX entry, and the length into the corresponding USB_COUNTn_TX entry, before marking the endpoint as ready to transmit. To receive a packet though is slightly more complex. The application must allocate some space in the PMA, setting the address into the USB_ADDRn_RX entry of the BTABLE before filling out the top half of the USB_COUNTn_RX entry. For ease of bit sizing, the STM32 only supports space allocations of two to sixty-two bytes in steps of two bytes at a time, or thirty-two to five-hundred-twelve bytes in steps of thirty-two bytes at a time. Once the packet is received, the USB peripheral will fill out the lower bits of the USB_COUNTn_RX entry with the actual number of bytes filled out in the buffer.

Packets themselves Since packets are, typically, a maximum of 64 bytes long (for USB 2.0) and are simply sequences of bytes with no useful structure to them (as far as the USB peripheral itself is concerned) the PMA simply requires that they be present and contiguous in PMA memory space. Addresses of packets are relative to the base of the PMA and are byte-addressed, however they cannot start on an odd byte, so essentially they are 16bit addressed. Since the BTABLE can be anywhere within the PMA, as can the packets, the application will have to do some memory management (either statically, or dynamically) to manage the packets in the PMA.

Accessing the PMA The PMA is accessed in 16bit word sections. It's not possible to access single bytes of the PMA, nor is it conveniently structured as far as the CPU is concerned. Instead the PMA's 16bit words are spread on 32bit word boundaries as far as the CPU knows. This is done for convenience and simplicity of hardware, but it means that we need to ensure our library code knows how to deal with this. First up, to convert an address in the PMA into something which the CPU can use we need to know where in the CPU's address space the PMA is. Fortunately this is fixed at 0x4000_6000. Secondly we need to know what address in the PMA we wish to access, so we can determine which 16bit word that is, and thus what the address is as far as the CPU is concerned. If we assume we only ever want to access 16bit entries, we can just multiply the PMA offset by two before adding it to the PMA base address. So, to access the 16bit word at byte-offset 8 in the PMA, we'd look for the 16bit word at 0x4000_6000 + (0x08 * 2) => 0x4000_6010.

Bundling the PMA into something we can use I said we'd do some Rust, and so we shall

    // Thanks to the work by Jorge Aparicio, we have a convenient wrapper
    // for peripherals which means we can declare a PMA peripheral:
    pub const PMA: Peripheral<PMA> = unsafe   Peripheral::new(0x4000_6000)  ;
    // The PMA struct type which the peripheral will return a ref to
    pub struct PMA  
        pma_area: PMA_Area,
     
    // And the way we turn that ref into something we can put a useful impl on
    impl Deref for PMA  
        type Target = PMA_Area;
        fn deref(&self) -> &PMA_Area  
            &self.pma_area
         
     
    // This is the actual representation of the peripheral, we use the C repr
    // in order to ensure it ends up packed nicely together
    #[repr(C)]
    pub struct PMA_Area  
        // The PMA consists of 256 u16 words separated by u16 gaps, so lets
        // represent that as 512 u16 words which we'll only use every other of.
        words: [VolatileCell<u16>; 512],
     

That block of code gives us three important things. Firstly a peripheral object which we will be able to (later) manage nicely as part of the set of peripherals which RTFM will look after for us. Secondly we get a convenient packed array of u16s which will be considered volatile (the compiler won't optimise around the ordering of writes etc). Finally we get a struct on which we can hang an implementation to give our PMA more complex functionality. A useful first pair of functions would be to simply let us get and put u16s in and out of that word array, since we're only using every other word

    impl PMA_Area  
        pub fn get_u16(&self, offset: usize) -> u16  
            assert!((offset & 0x01) == 0);
            self.words[offset].get()
         
        pub fn set_u16(&self, offset: usize, val: u16)  
            assert!((offset & 0x01) == 0);
            self.words[offset].set(val);
         
     

These two functions take an offset in the PMA and return the u16 word at that offset. They only work on u16 boundaries and as such they assert that the bottom bit of the offset is unset. In a release build, that will go away, but during debugging this might be essential. Since we're only using 16bit boundaries, this means that the first word in the PMA will be at offset zero, and the second at offset two, then four, then six, etc. Since we allocated our words array to expect to use every other entry, this automatically converts into the addresses we desire. If we pop (and please don't worry about the unsafe stuff for now):

    unsafe   (&*usb::pma::PMA.get()).set_u16(4, 64);  

into our main function somewhere, and then build and objdump our test binary we can see the following set of instructions added:

 80001e4:   f246 0008   movw    r0, #24584  ; 0x6008
 80001e8:   2140        movs    r1, #64 ; 0x40
 80001ea:   f2c4 0000   movt    r0, #16384  ; 0x4000
 80001ee:   8001        strh    r1, [r0, #0]

This boils down to a u16 write of 0x0040 (64) to the address 0x4006008 which is the third 32 bit word in the CPU's view of the PMA memory space (where offset 4 is the third 16bit word) which is exactly what we'd expect to see. We can, from here, build up some functions for manipulating a BTABLE, though the most useful ones for us to take a look at are the RX counter functions:

    pub fn get_rxcount(&self, ep: usize) -> u16  
        self.get_u16(BTABLE + (ep * 8) + 6) & 0x3ff
     
    pub fn set_rxcount(&self, ep: usize, val: u16)  
        assert!(val <= 1024);
        let rval: u16 =  
            if val > 62  
                assert!((val & 0x1f) == 0);
                (((val >> 5) - 1) << 10)   0x8000
              else  
                assert!((val & 1) == 0);
                (val >> 1) << 10
             
         ;
        self.set_u16(BTABLE + (ep * 8) + 6, rval)
     

The getter is fairly clean and clear, we need the BTABLE base in the PMA, add the address of the USB_COUNTn_RX entry to that, retrieve the u16 and then mask off the bottom ten bits since that's the size of the relevant field. The setter is a little more complex, since it has to deal with the two possible cases, this isn't pretty and we might be able to write some better peripheral structs in the future, but for now, if the length we're setting is 62 or less, and is divisible by two, then we put a zero in the top bit, and the number of 2-byte lumps in at bits 14:10, and if it's 64 or more, we mask off the bottom to check it's divisible by 32, and then put the count (minus one) of those blocks in, instead, and set the top bit to mark it as such. Fortunately, when we set constants, Rust's compiler manages to optimise all this very quickly. For a BTABLE at the bottom of the PMA, and an initialisation statement of:

    unsafe   (&*usb::pma::PMA.get()).set_rxcount(1, 64);  

then we end up with the simple instruction sequence:

80001e4:    f246 001c   movw    r0, #24604  ; 0x601c
80001e8:    f44f 4104   mov.w   r1, #33792  ; 0x8400
80001ec:    f2c4 0000   movt    r0, #16384  ; 0x4000
80001f0:    8001        strh    r1, [r0, #0]

We can decompose that into a C like *((u16*)0x4000601c) = 0x8400 and from there we can see that it's writing to the u16 at 0x1c bytes into the CPU's view of the PMA, which is 14 bytes into the PMA itself. Since we know we set the BTABLE at the start of the PMA, it's 14 bytes into the BTABLE which is firmly in the EP1 entries. Specifically it's USB_COUNT1_RX which is what we were hoping for. To confirm this, check out page 651 of the datasheet. The value set was 0x8400 which we can decompose into 0x8000 and 0x0400. The first is the top bit and tells us that BL_SIZE is one, and thus the blocks are 32 bytes long. Next the 0x4000 if we shift it right ten places, we get the value 2 for the field NUM_BLOCK and multiplying 2 by 32 we get the 64 bytes we asked it to set as the size of the RX buffer. It has done exactly what we hoped it would, but the compiler managed to optimise it into a single 16 bit store of a constant value to a constant location. Nice and efficient. Finally, let's look at what happens if we want to write a packet into the PMA. For now, let's assume packets come as slices of u16s because that'll make our life a little simpler:

    pub fn write_buffer(&self, base: usize, buf: &[u16])  
        for (ofs, v) in buf.iter().enumerate()  
            self.set_u16(base + (ofs * 2), *v);
         
     

Yes, even though we're deep in no_std territory, we can still get an iterator over the slice, and enumerate it, getting a nice iterator of (index, value) though in this case, the value is a ref to the content of the slice, so we end up with *v to deref it. I am sure I could get that automatically happening but for now it's there. Amazingly, despite using iterators, enumerators, high level for loops, function calls, etc, if we pop:

    unsafe   (&*usb::pma::PMA.get()).write_buffer(0, &[0x1000, 0x2000, 0x3000]);  

into our main function and compile it, we end up with the instruction sequence:

80001e4:    f246 0000   movw    r0, #24576  ; 0x6000
80001e8:    f44f 5180   mov.w   r1, #4096   ; 0x1000
80001ec:    f2c4 0000   movt    r0, #16384  ; 0x4000
80001f0:    8001        strh    r1, [r0, #0]
80001f2:    f44f 5100   mov.w   r1, #8192   ; 0x2000
80001f6:    8081        strh    r1, [r0, #4]
80001f8:    f44f 5140   mov.w   r1, #12288  ; 0x3000
80001fc:    8101        strh    r1, [r0, #8]

which, as you can see, ends up being three sequential halfword stores directly to the right locations in the CPU's view of the PMA. You have to love seriously aggressive compile-time optimisation Hopefully, by next time, we'll have layered some more pleasant routines on our PMA code, and begun a foray into the setup necessary before we can begin handling interrupts and start turning up on a USB port.

LTSP Manager has just been announced in the ltsp-discuss mailing list! After long hours of testing and fixing issues that were related to the internationalization process, it s now time to make it available to a broader audience. It s currently available for Ubuntu and Debian via a PPA, and it will shortly get uploaded to Debian experimental. We ve documented all the necessary steps to install and maintain LTSP using LTSP Manager in the LTSP wiki: http://wiki.ltsp.org/wiki/Ltsp-manager. The initial documentation is deliberately concise so that new users can read all of it. We ve also included best practices about user account management in schools etc. This concludes all the tasks outlined by my Outreachy project, but of course not my involvement to LTSP Manager. I ll keep using it in my schools and be an active part of its ecosystem. Many thanks to Debian Outreachy and to my mentors for the opportunity to work on this excellent project!

I was recently asked to sort things out so that snap builds on Launchpad could themselves install snaps as build-dependencies. To make this work we need to start doing builds in LXD containers rather than in chroots. As a result I ve been doing some quite extensive refactoring of launchpad-buildd: it previously had the assumption that it was going to use a chroot for everything baked into lots of untested helper shell scripts, and I ve been rewriting those in Python with unit tests and with a single Backend abstraction that isolates the high-level logic from the details of where each build is being performed. This is all interesting work in its own right, but it s not what I want to talk about here. While I was doing all this refactoring, I ran across a couple of methods I wrote a while back which looked something like this: (I ve already replaced the chroot method with a call to Backend.run, but it s easier to see what I m talking about in the original form.) One thing to notice about this code is that it uses several adverbial commands: that is, commands that run another command in a different way. For example, sudo runs another command as another user, while chroot runs another command with a different root directory, and env runs another command with different environment variables set. These commands chain neatly, and they also have the useful property that they take the subsidiary command and its arguments as a list of arguments. coreutils has several other commands that behave this way, and adverbio is another useful example. By contrast, su -c is something you might call a quasi-adverbial command: it does modify the behaviour of another command, but it takes it as a single argument which it then passes to sh -c. Every time you have something that s passed to a shell like this, you need a corresponding layer of shell quoting to escape any shell metacharacters that should be interpreted literally. This is often cumbersome and is easy to get wrong. My Python implementation is as follows, and I wouldn t be totally surprised to discover that it contained a bug: Python >= 3.3 has shlex.quote, which is an improvement and we should probably use that instead, but it s still another thing to forget to call. This is why process-spawning libraries such as Python s subprocess, Perl s system and open, and my own libpipeline for C encourage programmers to use a list syntax and to avoid involving the shell entirely wherever possible. One thing that the standard Unix tools don t let you do in an adverbial way is to change your working directory, and I ve run into this annoying limitation several times. This means that it s difficult to chain that operation together with other adverbs, for example to run a command in a particular working directory inside a chroot. The workaround I used above was to invoke a shell that runs cd /build && ..., but that s another command that s only quasi-adverbial, since the extra shell means an extra layer of shell quoting. (Ian Jackson rightly observes that you can in fact write the necessary adverb as something like sh -ec 'cd "$1"; shift; exec "$@"' chdir. I think that s a bit uglier than I ideally want to use in production code, but you might reasonably think that it s worth it to avoid the extra layer of shell quoting.) I therefore decided that this was a feature that belonged in coreutils, and after a bit of mailing list discussion we felt it was best implemented as a new option to env(1). I sent a patch for this which has been accepted. This means that we have a new composable adverb, env --chdir=NEWDIR, which will allow the run_build_command method above to be rewritten as something like this: The env --chdir option will be in coreutils 8.28. We won t be able to use it in launchpad-buildd until that s available in all Ubuntu series we might want to build for, so in this particular application that s going to take a few years; but other applications may well be able to make use of it sooner.

"Qubes OS from the POV of a Debian developer" and "Qubes OS user meetup at Bornhack" I wrote the following while on my way home from Bornhack which was an awesome hacking camp on the Danish island of Bornholm, where about 200 people gathered for a week, with a nice beach in walking distance (and a not too cold Baltic Sea) and vegan and not so vegan grills, to give some hints why it was awesome. (Actually it was mostly awesome due to the people there, not the things, but anyway ) And there were of course also talks and workshops, and one was a Qubes OS users meetup, which amazingly was attended by 20 Qubes OS users (and some Qubes OS users present were even missing), so in other words: Qubes OS had a >10% user base at this event! And I even found one heads user! At DebConf17 I gave a talk titled "Qubes OS from the POV of a Debian developer" and while the video was immediatly available (thanks to the DebConf videoteam!) I've also now put my slides for it online. Since then, I learned a few things:

• I should have mentioned Standalone-VM (=non-template based VMs) in the talk, as those are also possible, and sometimes handy.
• IPv6 works with manual fiddling (but this is undocumented)
• I've given Salt (for configuation management) a short try (with the help of an experienced Salt users, thanks nodens!) and I must say I'm not impressed yet. But Qubes 4.0 will bring huge changes to this area (including introducing an AdminVM), so while I will not use Salt for the time I'll still be using Qubes 3.2, I will come back and look at this later.
• after adding a single line containing "iwldvm iwlwifi" to /rw/config/suspend-module-blacklist in the NetVM the wireless comes back nicely (on my X230) after Suspend/Resume.

I'm looking forward to more Qubes OS user meetups in future!

Do you need to send messages to both Twitter and Mastodon? Use the Remindr bot! Remindr is written in Python, released under the GPLv3 license. 1. How to install Remindr Install Remindr from PyPI:

# pip3 install remindr

2. How to configure Remindr First, start by writing a messages.txt file with the following content:

o en Send scheduled messages to both Twitter and Mastodon with the Remindr bot https://carlchenet.com/send-scheduled-messages-to-both-twitter-and-mastodon-with-the-remindr-bot #remindr #twitter #mastodon
x en Follow Carl Chenet for great news about Free Software! https://carlchenet.com #freesoftware

The first field only indicates if the line is the next one to be considered by Remindr, the o indicates the next line to be sent, x means it won t. The second field is the 2-letters code language your content is using, in my example en or fr. Next content on the line will compose the body of your messages to Mastodon and Twitter. You need to configure the Mastodon and the Twitter credentials in order to allow Remindr to send the messages. First you need to generate the credentials. For Twitter, you need to manually create an app on apps.twitter.com. For Mastodon, just launch the following command:

$ register_remindr_app

Some information will be asked by the command. At the end, two files are created, remindr_usercred.txt and remindr_clientcred.txt. You re going to need them for the Remindr configuration above. For the Remindr configuration, here is a complete configuration using the

[mastodon]
instance_url=https://mastodon.social
user_credentials=remindr_usercred.txt
client_credentials=remindr_clientcred.txt
[twitter]
consumer_key=a6lv2gZxkvk6UbQ30N4vFmlwP
consumer_secret=j4VxM2slv0Ud4rbgZeGbBzPG1zoauBGLiUMOX0MGF6nsjcyn4a
access_token=1234567897-Npq5fYybhacYxnTqb42Kbb3A0bKgmB3wm2hGczB
access_token_secret=HU1sjUif010DkcQ3SmUAdObAST14dZvZpuuWxGAV0xFnC
[image]
path_to_image=/home/chaica/blog-carl-chenet.png
[entrylist]
path_to_list=/etc/remindr/messages.txt

Your configuration is complete! Now we have to check if everything is fine. 3. How to use Remindr Now let s try your configuration by launching Remindr the first time by-hand:

$ remindr -c /etc/remindr/remindr.ini

The messages should appear on both Twitter and Mastodon. 4. How to schedule the Remindr execution The easiest way is to use you user crontab. Just add the following line in your crontab file, editing it with crontab -e

00 10 * * * remindr -c /etc/remindr/remindr.ini

From now on, your message will be sent every day at 10:00AM. Going further with Remindr

• Remindr on Gitlab.com (would be awesome starring it if you like/use it)
• The official documentation on Readthedocs.io

and finally You can help me developing tools for Mastodon and other social networks by donating anything through Liberaypay (also possible with cryptocurrencies). Any contribution will be appreciated. That s a big factor motivation
Donate You also may follow my account @carlchenet on Mastodon

The GNOME 3.26 release cycle is in its final bugfix stage before release. Here s a look at what s new in GNOME Tweaks since my last post. I ve heard people say that GNOME likes to remove stuff. If that were true, how would there be anything left in GNOME? But maybe it s partially true. And maybe it s possible for removals to be a good thing? Removal #1: Power Button Settings The Power page in Tweaks 3.25.91 looks a bit empty. In previous releases, the Tweaks app had a When the Power button is pressed setting that nearly duplicated the similar setting in the Settings app (gnome-control-center). I worked to restore support for Power Off as one of its options. Since this is now in Settings 3.25.91, there s no need for it to be in Tweaks any more. Removal #2: Hi-DPI Settings GNOME Tweaks offered a basic control to scale windows 2x for Hi-DPI displays. More advanced support is now in the Settings app. I suspect that fractional scaling won t be supported in GNOME 3.26 but it s something to look forward to in GNOME 3.28! Removal #3 Global Dark Theme I am announcing today that one of the oldest and popular tweaks will be removed from Tweaks 3.28 (to be released next March). Global Dark Theme is being removed because:

• Changing the Global Dark Theme option required closing any currently running apps and reopening them to get the correct theme.
• It didn t work for sandboxed apps (Flatpak and Snap)
• It only worked for gtk3 apps (it can t work on gtk2 apps)
• Some themes never supported a Dark variant. The switch wouldn t do anything at all with a theme like that.

Adwaita now has a separate Adwaita Dark theme. Arc has 2 different dark variations. Therefore, if you are a theme developer, you have about 6-7 months to offer a dark version of your theme. The dark version can be distributed the same way as your regular version. Removal #4 Some letters from our name In case you haven t noticed, GNOME Tweak Tool is now GNOME Tweaks. This better matches the GNOME app naming style. Thanks Alberto Fanjul for this improvement! For other details of what s changed including a helpful scrollbar fix from Ant nio Fernandes, see the NEWS file.

This semester I am taking JPN 530, Haruki Murakami and the Literature of Modern Japan . My department are letting me count it for the Philosophy Ph.D., and in fact my supervisor is joining me for the class. I have no idea what the actual class sessions will be like first one this afternoon and I m anxious about writing a literature term paper. But I already know that my weekends this semester are going to be great because I ll be reading Murakami s novels. What s particularly wonderful about this, and what I wanted to write about, is how nourishing I find reading literary fiction to be. For example, this weekend I read

This was something sure to be crammed full of warm secrets, like an antique clock built when peace filled the world. Potentiality knocks at the door of my heart.

and I was fed for the day. All my perceived needs dropped away; that s all it takes. This stands in stark contrast to reading philosophy, which is almost always draining rather than nourishing even philosophy I really want to read. Especially having to read philosophy at the weekend. (quotation is from On Seeing the 100% Perfect Girl on a Beautiful April Morning)

This story involves boys pretending to be pizza deliverymen using a working automated Strowger telephone exchange demonstrator on display in a museum, which is very old and is, to my knowledge, the only such working exhibit in the world. (Yes, I have video.) But first, a thought on exploration. There are those that would say that there is nothing left to explore anymore that the whole earth is mapped, photographed by satellites, and, well, known. I prefer to look at it a different way: the earth is full of places that billions of people will never see, and probably don t even know about. Those places may be quiet country creeks, peaceful neighborhoods one block away from major tourist attractions, an MTA museum in Brooklyn, a state park in Arkansas, or a beautiful church in Germany. Martha is not yet two months old, and last week she and I spent a surprisingly long amount of time just gazing at tree branches she was mesmerized, and why not, because to her, everything is new. As I was exploring in Portland two weeks ago, I happened to pick up a nearly-forgotten book by a nearly-forgotten person, Beryl Markham, a woman who was a pilot in Africa about 80 years ago. The passage that I happened to randomly flip to in the bookstore, which really grabbed my attention, was this:

The available aviation maps of Africa in use at that time all bore the cartographer s scale mark, 1/2,000,000 one over two million. An inch on the map was about thitry-two miles in the air, as compared to the flying maps of Europe on which one inch represented no more than four air miles. Moreover, it seemed that the printers of the African maps had a slightly malicious habit of including, in large letters, the names of towns, junctions, and villages which, while most of them did exist in fact, as a group of thatched huts may exist or a water hold, they were usually so inconsequential as completely to escape discovery from the cockpit. Beyond this, it was even more disconcerting to examine your charts before a proposed flight only to find that in many cases the bulk of the terrain over which you had to fly was bluntly marked: UNSURVEYED . It was as if the mapmakers had said, We are aware that between this spot and that one, there are several hundred thousands of acres, but until you make a forced landing there, we won t know whether it is mud, desert, or jungle and the chances are we won t know then! Beryl Markham, West With the Night

My aviation maps today have no such markings. The continent is covered with radio beacons, the world with GPS, the maps with precise elevations of the ground and everything from skyscrapers to antenna towers. And yet, despite all we know, the world is still a breathtaking adventure. Yesterday, the boys and I were going to fly to Abilene, KS, to see a museum (Seelye Mansion). Circumstances were such that we neither flew, nor saw that museum. But we still went to Abilene, and wound up at the Museum of Independent Telephony, a wondrous place for anyone interested in the history of technology. As it is one of those off-the-beaten-path sorts of places, the boys got 2.5 hours to use the hands-on exhibits of real old phones, switchboards, and also the schoolhouse out back. They decided why not? to use this historic equipment to pretend to order pizzas. Jacob and Oliver proceeded to invent all sorts of things to use the phones for: ordering pizza, calling the cops to chase the pizza delivery guys, etc. They were so interested that by 2PM we still hadn t had lunch and they claimed we re not hungry despite the fact that we were going to get pizza for lunch. And I certainly enjoyed the exhibits on the evolution of telephones, switching (from manual plugboards to automated switchboards), and such. This place was known it even has a website, I had been there before, and in fact so had the boys (my parents took them there a couple of years ago). But yesterday, we discovered the Strowger switch had been repaired since the last visit, and that it, in fact, is great for conversations about pizza. Whether it s seeing an eclipse, discovering a fascination with tree branches, or historic telephones, a spirit of curiosity and exploration lets a person find fun adventures almost anywhere.

Remember, Mastodon is a new decentralized social network, based on a free software which is rapidly gaining users (already there is more than 1.5 million accounts). As I ve created my account in June, I was a fast addict and I ve already created several tools for this network, Feed2toot, Remindr and Boost (mostly written in Python).

Now, with all this experience I have to stress out the importance of choosing the correct Mastodon instance.

Some technical reminders on how Mastodon works

First, let s quickly clarify something about the decentralized part. In Mastodon, decentralization is made through a federation of dedicated servers, called instances , each one with a complete independent administration. Your user account is created on one specific instance. You have two choices:

• Create your own instance. Which requires advanced technical knowledge.
• Create your user account on a public instance. Which is the easiest and fastest way to start using Mastodon.

You can move your user account from one instance to another, but you have to follow a special procedure which can be quite long, considering your own interest for technical manipulation and the total amount of your followers you ll have to warn about your change. As such, you ll have to create another account on a new instance and import three lists: the one with your followers, the one with the accounts you have blocked, and the one with the account you have muted.

From this working process, several technical and human factors will interest us.

A good technical administration for instance

As a social network, Mastodon is truly decentralized, with more than 1.5 million users on more than 2350 existing instances. As such, the most common usage is to create an account on an open instance. To create its own instance is way too difficult for the average user. Yet, using an open instance creates a strong dependence on the technical administrator of the chosen instance.

The technical administrator will have to deal with several obligations to ensure its service continuity, with high-quality hardware and regular back-ups. All of these have a price, either in money and in time. Regarding the time factor, it would be better to choose an administration team over an individual, as life events can change quite fast everyone s interests. As such, Framasoft, a French association dedicated to promoting the Free software use, offers its own Mastodon instance named: Framapiaf. The creator of the mastodon project, also offers a quite solid instance, Mastodon.social (see below).

Regarding the money factor, many instance administrators with a large number of users are currently asking for donation via Patreon, as hosting an instance server or renting one cost money.

Mastodon.social, the first instance of the Mastodon network

The Ideological Trend Of Your Instance

If anybody could have guessed the previous technical points since the recent registration explosion on the Mastodon social network, the following point took almost everyone by surprise. Little by little, different instances show their culture , their protest action, and their propaganda on this social network.

As the instance administrator has all the powers over its instance, he or she can block the instance of interacting with some other instances, or ban its instance s users from any interaction with other instances users.

With everyone having in mind the main advantages to have federalized instance from, this partial independence of some instances from the federation was a huge surprise. One of the most recent example was when the Unixcorn.xyz instance administrator banned its users from reading Aeris account, which was on its own instance. It was a cataclysm with several consequences, which I ve named the #AerisGate as it shows the different views on moderation and on its reception by various Mastodon users.

If you don t manage your own instance, when you ll have to choose the one where to create your account, make sure that the content you plan to toot is within the rules and compatible with the ideology of said instance s administrator. Yes, I know, it may seem surprising but, as stated above, by entering a public instance you become dependent on someone else s infrastructure, who may have an ideological way to conceive its Mastodon hosting service. As such, if you re a nazi, for example, don t open your Mastodon account on a far-left LGBT instance. Your account wouldn t stay open for long.

The moderation rules are described in the about/more page of the instance, and may contain ideological elements.

To ease the process for newcomers, it is now possible to use a great tool to select what instance should be the best to host your account.

Remember that, as stated above, Mastodon is decentralized, and as such there is no central authority which can be reached in case you have a conflict with your instance administrator. And nobody can force said administrator to follow its own rules, or not to change them on the fly. Think Twice Before Creating Your Account

If you want to create an account on an instance you don t control, you need to check two elements: the availability of the instance hosting service in the long run, often linked to the administrator or the administration group of said instance, and the ideological orientation of your instance. With these two elements checked, you ll be able to let your Mastodon account growth peacefully, without fearing an outage of your instance, or simple your account blocked one morning because it doesn t align with your instance s ideological line.

in Conclusion

To help me get involved in free software and writing articles for this blog, please consider a donation through my Liberapay page, even if it s only a few cents per week. My contact Bitcoin and Monero are also available on this page. Follow me on Mastodon Translated from French to English by St phanie Chaptal.

One set of gazebos put up: kilos of mushrooms eaten, bacon, mushrooms and all the trimmings barbequed and consumed by the hordes. Now laptops are sprouting in the garden under the gazebos as the temperature is soaring,

Some folk are quiet in the house under fans typing and cooling off.

Masses of washing up is being done - as ever, it's how many people you can fit into a kitchen.

Now it will all go quiet for a bit as everyone lets the breakfast go down :)

Superb hospitality - we're _SO_ lucky to have Steve and Jo do this so readily.

For those of you who might not be aware, freenode is an IRC network that caters to free and open source projects. We have had a goal for a number of years to hold freenode #live, an in-person conference where our staff, projects, FOSS supporters, and IRC enthusiasts could all get together in one place. Thanks to some very generous sponsorship, primarily by Private Internet Access, this conference is finally happening! The first ever freenode #live conference will take place October 28 - 29, 2017 in Bristol, United Kingdom. This conference will only be a success with support from the community. Luckily, there are many ways to help out.

1. Register to attend. We are working on putting together a great lineup of speakers that you will not want to miss. Registering allows us to finalize many logistical details surrounding the event to make sure everything runs smoothly.
2. Do you represent a FOSS group? We would love to have your group present at the event. There will be an exhibitor hall where you can advertise your organization and attract new users and contributors.
3. Submit a talk. Our call for proposals is still open. New and experienced speakers are welcome. Feel free to contact 2017-team@freenode.live and we would be happy to work with you to come up with a talk idea or provide feedback.
4. If you or your company are interested in helping to sponsor this event, please reach out to exhibit@freenode.live. We have opportunities for budgets of all sizes.

I look forward to hopefully seeing you at freenode #live!

Here's what happened in the Reproducible Builds effort between Sunday August 13 and Saturday August 19 2017: Reproducible Builds finally mandated by Debian Policy "Packages should build reproducibly" was merged into Debian policy! The added text is as follows and has been included into debian-policy 4.1.0.0:

Reproducibility
---------------
Packages should build reproducibly, which for the purposes of this
document [#]_ means that given
- a version of a source package unpacked at a given path;
- a set of versions of installed build dependencies;
- a set of environment variable values;
- a build architecture; and
- a host architecture,
repeatedly building the source package for the build architecture on
any machine of the host architecture with those versions of the build
dependencies installed and exactly those environment variable values
set will produce bit-for-bit identical binary packages.
It is recommended that packages produce bit-for-bit identical binaries
even if most environment variables and build paths are varied.  It is
intended for this stricter standard to replace the above when it is
easier for packages to meet it.
.. [#]
   This is Debian's precisification of the  reproducible-builds.org
   definition  _.

• Holger Levsen wrote a blog post briefly describing the background and implications of this. To quote him: "we are not 94% done yet, rather more like half done or so. We still need tools and processes to enable anyone to indepently verify that a given binary comes from the sources it is said to be coming, this will involve distributing .buildinfo files and providing user interfaces in APT and elsewhere and probably also systematic rebuilds by us and other parties. And 6% or 7% of the archive is still a lot of packages, eg. in Buster we currently still have 273 unreproducible key packages and for a large part we don't have patches yet so there is still a lot of work ahead."
• There were discussion threads on Hacker News and Reddit.
• Our long-term goal is that Policy mandates that packages "must" be reproducible, but for that we need to show further progress and also reach a consensus on .buildinfo files and much more.

Reproducible work in other projects Bernhard M. Wiedemann's reproducibleopensuse scripts now work on Debian buster on the openSUSE Build Service with the latest versions of osc and obs-build. Toolchain development and fixes #872514 was opened on devscripts by Chris Lamb to add a reproducible-check program to report on the reproducibility status of installed packages. Packages reviewed and fixed, and bugs filed Upstream reports:

• Bernhard M. Wiedemann:
  • qt5/base, SOURCE_DATE_EPOCH support.

Debian reports:

• Adrian Bunk:
  • #872262 filed against jellyfish.
  • #872675 filed against pdp.
  • #872678 filed against gem.
• Chris Lamb:
  • #872453 filed against isa-support.
  • #872459 filed against python-numpy.
  • #872460 filed against gcab, forwarded upstream.
  • #872514 filed against devscripts.
  • #872728 filed against desktop-file-utils.
  • #872729 filed against gtk+2.0, forwarded upstream, found via a reproducibility issue in Tails.
• Federico Brega:
  • #872285 filed against pyqt5.
• Jeremy Bicha:
  • #871963 filed against birdfont.
• Philip Rinn:
  • #872184 filed against quilt.

Debian non-maintainer uploads:

• Mattia Rizzolo:
  • console-data (for bug #799871).

Reviews of unreproducible packages 47 package reviews have been added, 58 have been updated and 39 have been removed in this week, adding to our knowledge about identified issues. 4 issue types have been updated:

• Added nondeterministic_output_generated_by_gcab.
• Added build_path_captured_by_python_numpy_misc_util.
• Added captures_build_path_in_sphinx_attr_links.
• Re-added nondeterminstic_ordering_in_gsettings_glib_enums_xml.

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (59)
• Bastien Roucari s (1)
• James Clarke (1)
• Jeremy Bicha (1)

diffoscope development Development continued in git, including the following contributions:

• Ximin Luo:
  • presenters: html: Don't traverse children whose parents were already limited (Closes: #871413)
  • On a non-GNU system, prefer tools that start with "g" for certain whitelisted commands. (Closes: #871029)
  • Add a --tool-prefix-binutils CLI flag. (Closes: #869868)
• Chris Lamb:
  • Temporarily revert "Bump Standards-Version to 4.0.1" to avoid spurious CI test failures.
  • comparators.xml: Use name attribute over path to avoid leaking comparison full path in output.
  • Code style fixes.

disorderfs development Development continued in git, including the following contributions:

• Chris Lamb:
  • Add simple autopkgtest.

reprotest development Development continued in git, including the following contributions:

• Ximin Luo:
  • Choose an existent HOME for the "control" build. (Closes: #860428)
  • Update debian/changelog with Santiago's changes.
• Santiago Torres:
  • Abstract parts of autopkgtest to support running on non-Debian systems.
  • Add a --host-distro flag to support that too.

tests.reproducible-builds.org Mattia fixed the script which creates the HTML representation of our database scheme to not append .html twice to the filename. Misc. This week's edition was written by Ximin Luo, Chris Lamb and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

tldr: The percentage of failure is decreasing, Clang support is improving but there is a long way to go. The goal of this initiative is to rebuild Debian using Clang as a compiler instead of gcc. I have been doing this analysis for the last 6 years. Recently, we rebuilt the archive of the Debian archive with Clang 3.9.1 (July 6th), 4.0.1 (July 6th) and 5.0 rc2 (August 20th). For various reasons, we didn't perform a rebuild since June 2016 with version 3.8. Therefor, we took the opportunity to do three over the last month. Now, the 3.9 & 4.0 results are impacted by a build failure when building all haskell packages (the -no-pie option in Clang doesn't exist - I introduced it in clang 5.0). Fixing this issue with 5.0 removed more than 860 failures. Also, for the same versions, a Qt compiler detection is considering that Clang is not a C++11 compiler because clang++, by default, defines __cplusplus as 199711L (-std=c++11 has to be added to define a correct __cplusplus). See https://bugreports.qt.io/browse/QTBUG-62535 for more information. Some discussions happened on the upstream mailing list about changing the default C++ dialect.
For example, with 4.0, this is causing 132 errors. With 5.0, probably thanks to a new Qt version, roughly the same number of packages are failing but because gcc just triggers a warning with the "nodiscard" attribute being incorrectly used when clang triggers an error. In parallel, ignoring the haskell build failures, the numbers sightly increased since last year even if the overall percentage decreased (new packages being uploaded in the archive).

Version	Build failures	Ignoring haskell pkgs
3.8		1367 / 5.6%
3.9	2274 / 8.1%	1618 / 5.8%
4.0	2311 / 8.3%	1655 / 5.9%
5.0		1445 / 5.1%

In parallel, new warnings and errors showed up in Clang.
This is causing a new set of build failures (especially with the usage of -Werror). As few examples:
* Starting with 4.0, clang triggers an error ordered comparison between pointer and zero ('char *' and 'int').
* Similarly, with this version, -Wmain introduces a new warning which will trigger a warning when a bool literal is returned from main.
* clang also introduced a new warning called -Waddress-of-packed-member causing 5 new errors.
* With the same version, clang can trigger a new error when auto is used in function return type. Now, as a conclusion, having Debian being built with clang by default is still a long shot.
First, when Clang became usable for a general audience, gcc was lagging in term of warning and error detections. Now, gcc is in a much better position than it was, decreasing the interest to have clang replacing gcc. In parallel, most of the efforts in term of warnings
and mistake detections are currently done under the clang tidy umbrella, making them less intrusive as part of this initiative (but harder to use and to deploy).
As an example, the gcc warning -Wmisleading-indentation has been implemented under a clang-tidy checker.
Second, the very permissive license of clang has been a key factor for some operating systems to switch like the PS4, Mac OS X or FreeBSD. With Debian, the community is generally happy with the GPL.
Third, the performances are similar enough that it is not worth the work, except for some projects with very special needs. Last, despite that it is much easier to contribute to llvm/clang than gcc (not copyright assignment or actual review system for example), this isn't a big differentiator for most of the projects. Of course, I will continue to run and analysis these rebuilds as this is a great source of information for clang upstream developers to improve the compatibility with gcc and understand some impacts. However, until there is a big game changer, I will stop pursuing the goal of having Debian switching to clang instead of gcc. I will stop effort on the debile project (which was aiming to rebuild in the background packages).

At DebConf17, John Sullivan, the executive director of the FSF, gave a talk on the supposed decline of the use of copyleft licenses use free-software projects. In his presentation, Sullivan questioned the notion that permissive licenses, like the BSD or MIT licenses, are gaining ground at the expense of the traditionally dominant copyleft licenses from the FSF. While there does seem to be a rise in the use of permissive licenses, in general, there are several possible explanations for the phenomenon.

When the rumor mill starts Sullivan gave a recent example of the claim of the decline of copyleft in an article on Opensource.com by Jono Bacon from February 2017 that showed a histogram of license usage between 2010 and 2017 (seen below).

From that, Bacon elaborates possible reasons for the apparent decline of the GPL. The graphic used in the article was actually generated by Stephen O'Grady in a January article, The State Of Open Source Licensing, which said:

In Black Duck's sample, the most popular variant of the GPL version 2 is less than half as popular as it was (46% to 19%). Over the same span, the permissive MIT has gone from 8% share to 29%, while its permissive cousin the Apache License 2.0 jumped from 5% to 15%.

Sullivan, however, argued that the methodology used to create both articles was problematic. Neither contains original research: the graphs actually come from the Black Duck Software "KnowledgeBase" data, which was partly created from the old Ohloh web site now known as Open Hub. To show one problem with the data, Sullivan mentioned two free-software projects, GNU Bash and GNU Emacs, that had been showcased on the front page of Ohloh.net in 2012. On the site, Bash was (and still is) listed as GPLv2+, whereas it changed to GPLv3 in 2011. He also claimed that "Emacs was listed as licensed under GPLv3-only, which is a license Emacs has never had in its history", although I wasn't able to verify that information from the Internet archive. Basically, according to Sullivan, "the two projects featured on the front page of a site that was using [the Black Duck] data set were wrong". This, in turn, seriously brings into question the quality of the data:

I reported this problem and we'll continue to do that but when someone is not sharing the data set that they're using for other people to evaluate it and we see glimpses of it which are incorrect, that should give us a lot of hesitation about accepting any conclusion that comes out of it.

Reproducible observations are necessary to the establishment of solid theories in science. Sullivan didn't try to contact Black Duck to get access to the database, because he assumed (rightly, as it turned out) that he would need to "pay for the data under terms that forbid you to share that information with anybody else". So I wrote Black Duck myself to confirm this information. In an email interview, Patrick Carey from Black Duck confirmed its data set is proprietary. He believes, however, that through a "combination of human and automated techniques", Black Duck is "highly confident at the accuracy and completeness of the data in the KnowledgeBase". He did point out, however, that "the way we track the data may not necessarily be optimal for answering the question on license use trend" as "that would entail examination of new open source projects coming into existence each year and the licenses used by them". In other words, even according to Black Duck, its database may not be useful to establish the conclusions drawn by those articles. Carey did agree with those conclusions intuitively, however, saying that "there seems to be a shift toward Apache and MIT licenses in new projects, though I don't have data to back that up". He suggested that "an effective way to answer the trend question would be to analyze the new projects on GitHub over the last 5-10 years." Carey also suggested that "GitHub has become so dominant over the recent years that just looking at projects on GitHub would give you a reasonable sampling from which to draw conclusions".

Indeed, GitHub published a report in 2015 that also seems to confirm MIT's popularity (45%), surpassing copyleft licenses (24%). The data is, however, not without its own limitations. For example, in the above graph going back to the inception of GitHub in 2008, we see a rather abnormal spike in 2013, which seems to correlate with the launch of the choosealicense.com site, described by GitHub as "our first pass at making open source licensing on GitHub easier". In his talk, Sullivan was critical of the initial version of the site which he described as biased toward permissive licenses. Because the GitHub project creation page links to the site, Sullivan explained that the site's bias could have actually influenced GitHub users' license choices. Following a talk from Sullivan at FOSDEM 2016, GitHub addressed the problem later that year by rewording parts of the front page to be more accurate, but that any change in license choice obviously doesn't show in the report produced in 2015 and won't affect choices users have already made. Therefore, there can be reasonable doubts that GitHub's subset of software projects may not actually be that representative of the larger free-software community.

In search of solid evidence So it seems we are missing good, reproducible results to confirm or dispel these claims. Sullivan explained that it is a difficult problem, if only in the way you select which projects to analyze: the impact of a MIT-licensed personal wiki will obviously be vastly different from, say, a GPL-licensed C compiler or kernel. We may want to distinguish between active and inactive projects. Then there is the problem of code duplication, both across publication platforms (a project may be published on GitHub and SourceForge for example) but also across projects (code may be copy-pasted between projects). We should think about how to evaluate the license of a given project: different files in the same code base regularly have different licenses often none at all. This is why having a clear, documented and publicly available data set and methodology is critical. Without this, the assumptions made are not clear and it is unreasonable to draw certain conclusions from the results. It turns out that some researchers did that kind of open research in 2016 in a paper called "The Debsources Dataset: Two Decades of Free and Open Source Software" [PDF] by Matthieu Caneill, Daniel M. Germ n, and Stefano Zacchiroli. The Debsources data set is the complete Debian source code that covers a large history of the Debian project and therefore includes thousands of free-software projects of different origins. According to the paper:

The long history of Debian creates a perfect subject to evaluate how FOSS licenses use has evolved over time, and the popularity of licenses currently in use.

Sullivan argued that the Debsources data set is interesting because of its quality: every package in Debian has been reviewed by multiple humans, including the original packager, but also by the FTP masters to ensure that the distribution can legally redistribute the software. The existence of a package in Debian provides a minimal "proof of use": unmaintained packages get removed from Debian on a regular basis and the mere fact that a piece of software gets packaged in Debian means at least some users found it important enough to work on packaging it. Debian packagers make specific efforts to avoid code duplication between packages in order to ease security maintenance. The data set covers a period longer than Black Duck's or GitHub's, as it goes all the way back to the Hamm 2.0 release in 1998. The data and how to reproduce it are freely available under a CC BY-SA 4.0 license.

Sullivan presented the above graph from the research paper that showed the evolution of software license use in the Debian archive. Whereas previous graphs showed statistics in percentages, this one showed actual absolute numbers, where we can't actually distinguish a decline in copyleft licenses. To quote the paper again:

The top license is, once again, GPL-2.0+, followed by: Artistic-1.0/GPL dual-licensing (the licensing choice of Perl and most Perl libraries), GPL-3.0+, and Apache-2.0.

Indeed, looking at the graph, at most do we see a rise of the Apache and MIT licenses and no decline of the GPL per se, although its adoption does seem to slow down in recent years. We should also mention the possibility that Debian's data set has the opposite bias: toward GPL software. The Debian project is culturally quite different from the GitHub community and even the larger free-software ecosystem, naturally, which could explain the disparity in the results. We can only hope a similar analysis can be performed on the much larger Software Heritage data set eventually, which may give more representative results. The paper acknowledges this problem:

Debian is likely representative of enterprise use of FOSS as a base operating system, where stable, long-term and seldomly updated software products are desirable. Conversely Debian is unlikely representative of more dynamic FOSS environments (e.g., modern Web-development with micro libraries) where users, who are usually developers themselves, expect to receive library updates on a daily basis.

The Debsources research also shares methodology limitations with Black Duck: while Debian packages are reviewed before uploading and we can rely on the copyright information provided by Debian maintainers, the research also relies on automated tools (specifically FOSSology) to retrieve license information. Sullivan also warned against "ascribing reason to numbers": people may have different reasons for choosing a particular license. Developers may choose the MIT license because it has fewer words, for compatibility reasons, or simply because "their lawyers told them to". It may not imply an actual deliberate philosophical or ideological choice. Finally, he brought up the theory that the rise of non-copyleft licenses isn't necessarily at the detriment of the GPL. He explained that, even if there is an actual decline, it may not be much of a problem if there is an overall growth of free software to the detriment of proprietary software. He reminded the audience that non-copyleft licenses are still free software, according to the FSF and the Debian Free Software Guidelines, so their rise is still a positive outcome. Even if the GPL is a better tool to accomplish the goal of a free-software world, we can all acknowledge that the conversion of proprietary software to more permissive and certainly simpler licenses is definitely heading in the right direction.

[I would like to thank the DebConf organizers for providing meals for me during the conference.] Note: this article first appeared in the Linux Weekly News.