Thanks to all the sponsors of this BBQ who have made this so awesome.

This is also post 100 in this blog - looking back, 90 or so of the 100 have been from Cambridge which just goes to show how much of the world revolves around a radius of about five miles from here

Likewise, there are folk in the room whom I've known for 20 years even if I'm dreadful with remembering stuff. There's also scope for remembering absent friends who have got us this far and are no longer with us, for whatever reason.

I've just handed over some CDs and DVDs which, if readable, have a collective memory back to Debian 0.93 in about 1994 - even if not readable, they're a document of how far we've come from boot floppies to VMs, Bu-Ray size images and architectures undreamt of all those years ago.

We were all up until about 0100 :) House full of folk talking about all sorts, a game of Mao. Garden full of people clustered round the barbeque or sitting chatting - I had a long chat about Debian, what it means and how it's often an easier world to deal with and move in than the world of work, office politics or whatever - being here is being at home.

Arguments in the kitchen over how far coffee "just happens" with the magic bean to cup machine, some folk are in the garden preparing for breakfast at noon.

I missed the significance of this week's date - the 26th anniversary of Linus' original announcement of Linux in 1991 fell on Friday. Probably the first user of Linux who installed it from scratch was Lars Wirzenius - who was here yesterday.

Debian's 24th birthday was just about ten days ago on 16th August, making it the second oldest distribution and I reckon I've been using it for twenty one of those years - I wouldn't change it for the world.

Funny this - I only blog when I'm in Cambridge :) I'm sure there's a blog back in the day from a BBQ a good few years ago. This is almost deja vu - a room full of Debian types - the crazy family - Thinkpads on a lot of laps and lots of chat around the room.

Colin showing round an amateur radio project in a tobacco tin, coding happening at the back of the room. Pepper the dog sitting on my foot - everything pretty well normal.

Here's what happened in the Reproducible Builds effort between Sunday August 13 and Saturday August 19 2017: Reproducible Builds finally mandated by Debian Policy "Packages should build reproducibly" was merged into Debian policy! The added text is as follows and has been included into debian-policy 4.1.0.0:

Reproducibility
---------------
Packages should build reproducibly, which for the purposes of this
document [#]_ means that given
- a version of a source package unpacked at a given path;
- a set of versions of installed build dependencies;
- a set of environment variable values;
- a build architecture; and
- a host architecture,
repeatedly building the source package for the build architecture on
any machine of the host architecture with those versions of the build
dependencies installed and exactly those environment variable values
set will produce bit-for-bit identical binary packages.
It is recommended that packages produce bit-for-bit identical binaries
even if most environment variables and build paths are varied.  It is
intended for this stricter standard to replace the above when it is
easier for packages to meet it.
.. [#]
   This is Debian's precisification of the  reproducible-builds.org
   definition  _.

• Holger Levsen wrote a blog post briefly describing the background and implications of this. To quote him: "we are not 94% done yet, rather more like half done or so. We still need tools and processes to enable anyone to indepently verify that a given binary comes from the sources it is said to be coming, this will involve distributing .buildinfo files and providing user interfaces in APT and elsewhere and probably also systematic rebuilds by us and other parties. And 6% or 7% of the archive is still a lot of packages, eg. in Buster we currently still have 273 unreproducible key packages and for a large part we don't have patches yet so there is still a lot of work ahead."
• There were discussion threads on Hacker News and Reddit.
• Our long-term goal is that Policy mandates that packages "must" be reproducible, but for that we need to show further progress and also reach a consensus on .buildinfo files and much more.

Reproducible work in other projects Bernhard M. Wiedemann's reproducibleopensuse scripts now work on Debian buster on the openSUSE Build Service with the latest versions of osc and obs-build. Toolchain development and fixes #872514 was opened on devscripts by Chris Lamb to add a reproducible-check program to report on the reproducibility status of installed packages. Packages reviewed and fixed, and bugs filed Upstream reports:

• Bernhard M. Wiedemann:
  • qt5/base, SOURCE_DATE_EPOCH support.

Debian reports:

• Adrian Bunk:
  • #872262 filed against jellyfish.
  • #872675 filed against pdp.
  • #872678 filed against gem.
• Chris Lamb:
  • #872453 filed against isa-support.
  • #872459 filed against python-numpy.
  • #872460 filed against gcab, forwarded upstream.
  • #872514 filed against devscripts.
  • #872728 filed against desktop-file-utils.
  • #872729 filed against gtk+2.0, forwarded upstream, found via a reproducibility issue in Tails.
• Federico Brega:
  • #872285 filed against pyqt5.
• Jeremy Bicha:
  • #871963 filed against birdfont.
• Philip Rinn:
  • #872184 filed against quilt.

Debian non-maintainer uploads:

• Mattia Rizzolo:
  • console-data (for bug #799871).

Reviews of unreproducible packages 47 package reviews have been added, 58 have been updated and 39 have been removed in this week, adding to our knowledge about identified issues. 4 issue types have been updated:

• Added nondeterministic_output_generated_by_gcab.
• Added build_path_captured_by_python_numpy_misc_util.
• Added captures_build_path_in_sphinx_attr_links.
• Re-added nondeterminstic_ordering_in_gsettings_glib_enums_xml.

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (59)
• Bastien Roucari s (1)
• James Clarke (1)
• Jeremy Bicha (1)

diffoscope development Development continued in git, including the following contributions:

• Ximin Luo:
  • presenters: html: Don't traverse children whose parents were already limited (Closes: #871413)
  • On a non-GNU system, prefer tools that start with "g" for certain whitelisted commands. (Closes: #871029)
  • Add a --tool-prefix-binutils CLI flag. (Closes: #869868)
• Chris Lamb:
  • Temporarily revert "Bump Standards-Version to 4.0.1" to avoid spurious CI test failures.
  • comparators.xml: Use name attribute over path to avoid leaking comparison full path in output.
  • Code style fixes.

disorderfs development Development continued in git, including the following contributions:

• Chris Lamb:
  • Add simple autopkgtest.

reprotest development Development continued in git, including the following contributions:

• Ximin Luo:
  • Choose an existent HOME for the "control" build. (Closes: #860428)
  • Update debian/changelog with Santiago's changes.
• Santiago Torres:
  • Abstract parts of autopkgtest to support running on non-Debian systems.
  • Add a --host-distro flag to support that too.

tests.reproducible-builds.org Mattia fixed the script which creates the HTML representation of our database scheme to not append .html twice to the filename. Misc. This week's edition was written by Ximin Luo, Chris Lamb and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Hosted Weblate provides also free hosting for free software projects. The hosting requests queue has grown too long, so it's time to process it and include new project. This time, the newly hosted projects include:

• Gadgetbridge - cloudless replacement for your gadget vendors' closed source Android applications.
• SUSI AI Chat - artificial intelligence for personal assistants, robots, help desks and chatbots
• SUSI AI Android app - Susi Android App
• TeamWard Client - personal Android League Of Legends assistant
• User-Agent Switcher - easily override the browser's User-Agent string
• Safe Eyes - protect your eyes from asthenopia
• XCSoar - the open-source glide computer

If you want to support this effort, please donate to Weblate, especially recurring donations are welcome to make this service alive. You can do them on Liberapay or Bountysource.

Filed under: Debian English SUSE Weblate

It's that time again! Another year, another OMGWTFBBQ! We're expecting 50 or so Debian folks at our place in Cambridge this weekend, ready to natter, geek, socialise and generally have a good time. Let's hope the weather stays nice, but if not we have gazebo technology... :-) Many thanks to a number of awesome companies and people near and far who are sponsoring the important refreshments for the weekend:

• Mythic Beasts
• Collabora
• Codethink
• QvarnLabs
• Chris Boot

I've even been working on the garden this week to improve it ready for the event. If you'd like to come and haven't already told us, please add yourself to the wiki page!

The BH package on CRAN was updated today to version 1.65.0. BH provides a sizeable portion of the Boost C++ libraries as a set of template headers for use by R, possibly with Rcpp as well as other packages. This release upgrades the version of Boost to the rather new upstream version Boost 1.65.0 released earlier this week, and adds two new libraries: align and sort. I had started the upgrade process a few days ago under release 1.64.0. Rigorous checking of reverse dependencies showed that mvnfast needed a small change (which was trivial: just seeding the RNG prior to running tests), which Matteo did in no time with a fresh CRAN upload. rstan is needing a bit more work but should be ready real soon now and we are awaiting a new version. And once I switched to the just release Boost 1.65.0 it became apparent that Cyclops no longer needs its embedded copy of Boost iterator---and Marc already made that change with yet another fresh CRAN upload. It is a true pleasure to work in such a responsive and collaborative community.

Changes in version 1.65.0-1 (2017-08-24)

• Upgraded to Boost 1.64 and then 1.65 installed directly from upstream source with several minor tweaks (as before)
• Fourth tweak corrects a misplaced curly brace (see the Boost ublas GitHub repo and its issue #40)
• Added Boost align (as requested in #32)
• Added Boost sort (as requested in #35)
• Added Boost multiprecision by fixing a script typo (as requested in #42)
• Updated Travis CI support via newer run.sh

Via CRANberries, there is a diffstat report relative to the previous release. Comments and suggestions are welcome via the mailing list or the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Since some time we're using siphash algorithm to speed up looking up strings in Weblate. Even though it is used by Python internally, it's not exposed in the standard library so several third party modules appeared in the PyPI. Out of all these siphashc or rather it's Python 3 fork siphashc3 seemed to perform best, so I've started to use that. However it turned out that none of them is in active maintenance anymore. The original version lacks Python 3 support, while the siphashc3 uses odd versioning which causes problems to some pip versions. After trying to get fix into siphashc3 without much of success, I've spoken to original author of siphashc and he has agreed to hand over maintainership to me. So it's new home is at https://github.com/WeblateOrg/siphashc and new release is already available on PyPI. Note: Originally we were using MD5 in Weblate, but siphash has shown to be faster and fits into 64-bits, what makes it easier to store and index in SQL databases as LONGINT.

Filed under: Debian English SUSE Weblate

On 5th August I got a chance to attend, speak and experience DebConf 2017 at Montreal, Canada. The conference was stretch ed from 6 August to 12 August .

Seasons of Debian Summer of Code & Winter of Outreachy

Pretty late for me to document my DebConf fun-learning-experiences, thanks to my delaying tactics.. I need to overcome.
But better late than ever, I had amazing time at DebConf. I got to meet and learn from my Outreachy Mentor, Daniel Pocock!

Picture of Daniel and me captured by : Dorina Mosku

One thing about DebConf I loved was the amount of Diversity in Debian family! As a beginner, I got to get a big picture of what all projects are there. Daniel helped me a lot in getting started with packaging in Debian. I really appreciate the time he took out to guide me @DebConf and Pranav, remotely. One specific line I liked about Daniel s talk on Open Day, 5th August Free Communications with Free Software and Debian while talking about free RTC (Real Time Communication) is that, ..Instead of communication controlling the user, the user can control the communcation.. I talked about free RTC, my Project Lumicall and about my journey being an Outreachy Intern with Debian. I also covered my co-speaker s project work on Lumicall being a GSoC 2016 student.

Picture captured by Aigars Mahinovs

Managing Debian s RTC services Daniel Pocock

Meeting the Outreachy family feels amaazzing! Karen Sandler, executive director of the Software Freedom Conservancy gave a talk on the Significance and Impact of Outreachy and Debian s support for the programme.

with Karen Sandler and Outreachy alumini

DebConf 2017 has been a wonderful conference with the community being very welcoming and helpful

tldr: The percentage of failure is decreasing, Clang support is improving but there is a long way to go. The goal of this initiative is to rebuild Debian using Clang as a compiler instead of gcc. I have been doing this analysis for the last 6 years. Recently, we rebuilt the archive of the Debian archive with Clang 3.9.1 (July 6th), 4.0.1 (July 6th) and 5.0 rc2 (August 20th). For various reasons, we didn't perform a rebuild since June 2016 with version 3.8. Therefor, we took the opportunity to do three over the last month. Now, the 3.9 & 4.0 results are impacted by a build failure when building all haskell packages (the -no-pie option in Clang doesn't exist - I introduced it in clang 5.0). Fixing this issue with 5.0 removed more than 860 failures. Also, for the same versions, a Qt compiler detection is considering that Clang is not a C++11 compiler because clang++, by default, defines __cplusplus as 199711L (-std=c++11 has to be added to define a correct __cplusplus). See https://bugreports.qt.io/browse/QTBUG-62535 for more information. Some discussions happened on the upstream mailing list about changing the default C++ dialect.
For example, with 4.0, this is causing 132 errors. With 5.0, probably thanks to a new Qt version, roughly the same number of packages are failing but because gcc just triggers a warning with the "nodiscard" attribute being incorrectly used when clang triggers an error. In parallel, ignoring the haskell build failures, the numbers sightly increased since last year even if the overall percentage decreased (new packages being uploaded in the archive).

Version	Build failures	Ignoring haskell pkgs
3.8		1367 / 5.6%
3.9	2274 / 8.1%	1618 / 5.8%
4.0	2311 / 8.3%	1655 / 5.9%
5.0		1445 / 5.1%

In parallel, new warnings and errors showed up in Clang.
This is causing a new set of build failures (especially with the usage of -Werror). As few examples:
* Starting with 4.0, clang triggers an error ordered comparison between pointer and zero ('char *' and 'int').
* Similarly, with this version, -Wmain introduces a new warning which will trigger a warning when a bool literal is returned from main.
* clang also introduced a new warning called -Waddress-of-packed-member causing 5 new errors.
* With the same version, clang can trigger a new error when auto is used in function return type. Now, as a conclusion, having Debian being built with clang by default is still a long shot.
First, when Clang became usable for a general audience, gcc was lagging in term of warning and error detections. Now, gcc is in a much better position than it was, decreasing the interest to have clang replacing gcc. In parallel, most of the efforts in term of warnings
and mistake detections are currently done under the clang tidy umbrella, making them less intrusive as part of this initiative (but harder to use and to deploy).
As an example, the gcc warning -Wmisleading-indentation has been implemented under a clang-tidy checker.
Second, the very permissive license of clang has been a key factor for some operating systems to switch like the PS4, Mac OS X or FreeBSD. With Debian, the community is generally happy with the GPL.
Third, the performances are similar enough that it is not worth the work, except for some projects with very special needs. Last, despite that it is much easier to contribute to llvm/clang than gcc (not copyright assignment or actual review system for example), this isn't a big differentiator for most of the projects. Of course, I will continue to run and analysis these rebuilds as this is a great source of information for clang upstream developers to improve the compatibility with gcc and understand some impacts. However, until there is a big game changer, I will stop pursuing the goal of having Debian switching to clang instead of gcc. I will stop effort on the debile project (which was aiming to rebuild in the background packages).

At DebConf17, John Sullivan, the executive director of the FSF, gave a talk on the supposed decline of the use of copyleft licenses use free-software projects. In his presentation, Sullivan questioned the notion that permissive licenses, like the BSD or MIT licenses, are gaining ground at the expense of the traditionally dominant copyleft licenses from the FSF. While there does seem to be a rise in the use of permissive licenses, in general, there are several possible explanations for the phenomenon.

When the rumor mill starts Sullivan gave a recent example of the claim of the decline of copyleft in an article on Opensource.com by Jono Bacon from February 2017 that showed a histogram of license usage between 2010 and 2017 (seen below).

From that, Bacon elaborates possible reasons for the apparent decline of the GPL. The graphic used in the article was actually generated by Stephen O'Grady in a January article, The State Of Open Source Licensing, which said:

In Black Duck's sample, the most popular variant of the GPL version 2 is less than half as popular as it was (46% to 19%). Over the same span, the permissive MIT has gone from 8% share to 29%, while its permissive cousin the Apache License 2.0 jumped from 5% to 15%.

Sullivan, however, argued that the methodology used to create both articles was problematic. Neither contains original research: the graphs actually come from the Black Duck Software "KnowledgeBase" data, which was partly created from the old Ohloh web site now known as Open Hub. To show one problem with the data, Sullivan mentioned two free-software projects, GNU Bash and GNU Emacs, that had been showcased on the front page of Ohloh.net in 2012. On the site, Bash was (and still is) listed as GPLv2+, whereas it changed to GPLv3 in 2011. He also claimed that "Emacs was listed as licensed under GPLv3-only, which is a license Emacs has never had in its history", although I wasn't able to verify that information from the Internet archive. Basically, according to Sullivan, "the two projects featured on the front page of a site that was using [the Black Duck] data set were wrong". This, in turn, seriously brings into question the quality of the data:

I reported this problem and we'll continue to do that but when someone is not sharing the data set that they're using for other people to evaluate it and we see glimpses of it which are incorrect, that should give us a lot of hesitation about accepting any conclusion that comes out of it.

Reproducible observations are necessary to the establishment of solid theories in science. Sullivan didn't try to contact Black Duck to get access to the database, because he assumed (rightly, as it turned out) that he would need to "pay for the data under terms that forbid you to share that information with anybody else". So I wrote Black Duck myself to confirm this information. In an email interview, Patrick Carey from Black Duck confirmed its data set is proprietary. He believes, however, that through a "combination of human and automated techniques", Black Duck is "highly confident at the accuracy and completeness of the data in the KnowledgeBase". He did point out, however, that "the way we track the data may not necessarily be optimal for answering the question on license use trend" as "that would entail examination of new open source projects coming into existence each year and the licenses used by them". In other words, even according to Black Duck, its database may not be useful to establish the conclusions drawn by those articles. Carey did agree with those conclusions intuitively, however, saying that "there seems to be a shift toward Apache and MIT licenses in new projects, though I don't have data to back that up". He suggested that "an effective way to answer the trend question would be to analyze the new projects on GitHub over the last 5-10 years." Carey also suggested that "GitHub has become so dominant over the recent years that just looking at projects on GitHub would give you a reasonable sampling from which to draw conclusions".

Indeed, GitHub published a report in 2015 that also seems to confirm MIT's popularity (45%), surpassing copyleft licenses (24%). The data is, however, not without its own limitations. For example, in the above graph going back to the inception of GitHub in 2008, we see a rather abnormal spike in 2013, which seems to correlate with the launch of the choosealicense.com site, described by GitHub as "our first pass at making open source licensing on GitHub easier". In his talk, Sullivan was critical of the initial version of the site which he described as biased toward permissive licenses. Because the GitHub project creation page links to the site, Sullivan explained that the site's bias could have actually influenced GitHub users' license choices. Following a talk from Sullivan at FOSDEM 2016, GitHub addressed the problem later that year by rewording parts of the front page to be more accurate, but that any change in license choice obviously doesn't show in the report produced in 2015 and won't affect choices users have already made. Therefore, there can be reasonable doubts that GitHub's subset of software projects may not actually be that representative of the larger free-software community.

In search of solid evidence So it seems we are missing good, reproducible results to confirm or dispel these claims. Sullivan explained that it is a difficult problem, if only in the way you select which projects to analyze: the impact of a MIT-licensed personal wiki will obviously be vastly different from, say, a GPL-licensed C compiler or kernel. We may want to distinguish between active and inactive projects. Then there is the problem of code duplication, both across publication platforms (a project may be published on GitHub and SourceForge for example) but also across projects (code may be copy-pasted between projects). We should think about how to evaluate the license of a given project: different files in the same code base regularly have different licenses often none at all. This is why having a clear, documented and publicly available data set and methodology is critical. Without this, the assumptions made are not clear and it is unreasonable to draw certain conclusions from the results. It turns out that some researchers did that kind of open research in 2016 in a paper called "The Debsources Dataset: Two Decades of Free and Open Source Software" [PDF] by Matthieu Caneill, Daniel M. Germ n, and Stefano Zacchiroli. The Debsources data set is the complete Debian source code that covers a large history of the Debian project and therefore includes thousands of free-software projects of different origins. According to the paper:

The long history of Debian creates a perfect subject to evaluate how FOSS licenses use has evolved over time, and the popularity of licenses currently in use.

Sullivan argued that the Debsources data set is interesting because of its quality: every package in Debian has been reviewed by multiple humans, including the original packager, but also by the FTP masters to ensure that the distribution can legally redistribute the software. The existence of a package in Debian provides a minimal "proof of use": unmaintained packages get removed from Debian on a regular basis and the mere fact that a piece of software gets packaged in Debian means at least some users found it important enough to work on packaging it. Debian packagers make specific efforts to avoid code duplication between packages in order to ease security maintenance. The data set covers a period longer than Black Duck's or GitHub's, as it goes all the way back to the Hamm 2.0 release in 1998. The data and how to reproduce it are freely available under a CC BY-SA 4.0 license.

Sullivan presented the above graph from the research paper that showed the evolution of software license use in the Debian archive. Whereas previous graphs showed statistics in percentages, this one showed actual absolute numbers, where we can't actually distinguish a decline in copyleft licenses. To quote the paper again:

The top license is, once again, GPL-2.0+, followed by: Artistic-1.0/GPL dual-licensing (the licensing choice of Perl and most Perl libraries), GPL-3.0+, and Apache-2.0.

Indeed, looking at the graph, at most do we see a rise of the Apache and MIT licenses and no decline of the GPL per se, although its adoption does seem to slow down in recent years. We should also mention the possibility that Debian's data set has the opposite bias: toward GPL software. The Debian project is culturally quite different from the GitHub community and even the larger free-software ecosystem, naturally, which could explain the disparity in the results. We can only hope a similar analysis can be performed on the much larger Software Heritage data set eventually, which may give more representative results. The paper acknowledges this problem:

Debian is likely representative of enterprise use of FOSS as a base operating system, where stable, long-term and seldomly updated software products are desirable. Conversely Debian is unlikely representative of more dynamic FOSS environments (e.g., modern Web-development with micro libraries) where users, who are usually developers themselves, expect to receive library updates on a daily basis.

The Debsources research also shares methodology limitations with Black Duck: while Debian packages are reviewed before uploading and we can rely on the copyright information provided by Debian maintainers, the research also relies on automated tools (specifically FOSSology) to retrieve license information. Sullivan also warned against "ascribing reason to numbers": people may have different reasons for choosing a particular license. Developers may choose the MIT license because it has fewer words, for compatibility reasons, or simply because "their lawyers told them to". It may not imply an actual deliberate philosophical or ideological choice. Finally, he brought up the theory that the rise of non-copyleft licenses isn't necessarily at the detriment of the GPL. He explained that, even if there is an actual decline, it may not be much of a problem if there is an overall growth of free software to the detriment of proprietary software. He reminded the audience that non-copyleft licenses are still free software, according to the FSF and the Debian Free Software Guidelines, so their rise is still a positive outcome. Even if the GPL is a better tool to accomplish the goal of a free-software world, we can all acknowledge that the conversion of proprietary software to more permissive and certainly simpler licenses is definitely heading in the right direction.

[I would like to thank the DebConf organizers for providing meals for me during the conference.] Note: this article first appeared in the Linux Weekly News.

Over the last few days, Rcpp passed another noteworthy hurdle. It is now used by over 10 percent of packages on CRAN (as measured by Depends, Imports and LinkingTo, but excluding Suggests). As of this morning 1130 packages use Rcpp out of a total of 11275 packages. The graph on the left shows the growth of both outright usage numbers (in darker blue, left axis) and relative usage (in lighter blue, right axis). Older posts on this blog took note when Rcpp passed round hundreds of packages, most recently in April for 1000 packages. The growth rates for both Rcpp, and of course CRAN, are still staggering. A big thank you to everybody who makes this happen, from R Core and CRAN to all package developers, contributors, and of course all users driving this. We have built ourselves a rather impressive ecosystem. So with that a heartfelt Thank You! to all users and contributors of R, CRAN, and of course Rcpp, for help, suggestions, bug reports, documentation, encouragement, and, of course, code.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I upgraded my last major machine from Jessie to Stretch last week. That machine was the one running the most services, but I d made notes while updating various others to ensure it went smoothly. Below are the things I noted along the way, both for my own reference and in case they are of use to anyone else.

• Roundcube with the sqlite3 backend stopped working after the upgrade; fix was to edit /etc/roundcube/debian-db-roundcube.php and change sqlite3:// to sqlite:// in the $config['db_dsnw'] line.
• Dovecot no longer supports SSLv2 so had to remove !SSLv2 from the ssl_protocols list in /etc/dovecot/conf.d/10-ssl.conf
• Duplicity now tries to do a mkdir so I had to change from the scp:// backend to the sftp:// backend in my backup scripts.
• Needed to add needs_root_rights=yes to /etc/X11/Xwrapper.config so Kodi systemd unit could still start it on a new VT. Need to figure out how to get this working without the need for root.
• Upgrading fail2ban would have been easier if I d dropped my additions in /etc/fail2ban/jail.d/ rather than the master config. Fixed for next time.
• ejabberd continues to be a pain; I do wonder if it s worth running an XMPP server these days. I certainly don t end up using it to talk to people myself.
• Upgrading 1200+ packages takes a long time, even when the majority of them don t have any questions to ask during the process.
• PostgreSQL upgrades have got so much easier. pg_upgradecluster 9.4 main chugged away but did exactly what I needed.

Other than those points things were pretty smooth. Nice work by all those involved!

Highway US-81 in northern Kansas and southern Nebraska is normally a pleasant, sleepy sort of drive. It was upgraded to a 4-lane road not too long ago, but as far as 4-lane roads go, its traffic is typically light. For drives from Kansas to South Dakota, it makes a pleasant route. Yesterday was eclipse day. I strongly suspect that highway 81 had more traffic that day than it ever has before, or ever will again. For nearly the entire 3-hour drive to Geneva, NE, it was packed though mostly still moving at a good speed. And for our entire drive back, highway 81 and every other southbound road we used was so full it felt like rush hour in Dallas. (Well, not quite. Traffic was still moving.) I believe scenes like this were played out across the continent. I ve been taking a lot of photos, and writing about our new baby Martha lately. Now it s time to write a bit about some more adventures with Jacob and Oliver they re now in third and fifth grades in school. We had been planning to fly, and airports I called were either full, or were planning to park planes in the grass, or even shut down some runways to use for parking. The airport in the little town of Beatrice, NE (which I had visited twice before) was even going to have a temporary FAA control tower. At the last minute, due to some storm activity near home at departure time, we unloaded the plane and drove instead. The atmosphere at the fairgrounds in Geneva was festive. One family had brought bubbles for their kids and extras to share. I had bought the boys a book about the eclipse, which they were reading before and during the event. They were both great, safe users of their eclipse glasses. Jacob caught a toad, and played with it for awhile. He wanted to bring it home with us, but I convinced him to let me take a picture of him with his toad friend instead. While we were waiting for totality, a number of buses from the local school district arrived. So by the time the big moment arrived, we could hear the distant roar of delight and applause from the school children gathered at the far end of the field, plus all the excitement nearby. Both boys were absolutely ecstatic to be witnessing it (and so was I!) Wow! Awesome! And simple cackles of delight were heard. On the drive home, they both kept talking about how amazing it was, and it was once in a lifetime. We enjoyed our eclipse neighbors the woman from San Antonio next to us, the surprise discovery of another family from just a few miles from us parked two cars down, even running into relatives at a restaurant on the way home. The applause from all around when it started and when it ended. And the feeling, which is hard to describe, of awe and amazement at the wonders of our world and our universe. There are many problems with the world right now, but somehow there s something right about people coming together from all over to enjoy it.

Lately I experienced some new kind of spam, at least new to myself. Seems that spammer abuse registration input fields that do not implement strong enough validation, and echo back several values from the registration process in some kind of welcome mail. Basically filling the spam message in name and surname fields. So far I found a bunch of those originating from the following AS: AS49453, AS50896, AS200557 and AS61440. The first three belong to something identifying itself as "QUALITYNETWORK". The last one, AS61440, seems to be involved only partially with some networks being delegated to "Atomohost". To block them it's helpful to query the public radb service whois.radb.net for all networks belonging to the specific AS like this:

whois -h whois.radb.net -- '-i origin AS50896'

Another bunch of batch capable whois services are provided by Team Cymru. They've some examples at the end of https://www.team-cymru.org/IP-ASN-mapping.html. In this specific case the spam was for "www.robotas.ru" which is currently terminated at CloudFlare and redirects via JS document.location to "http://link31.net/b494d/ooo/", which in turn redirects via JS window.location to "http://revizor-online.ga/" which is again hosted at CloudFlare. The page at the end plays some strange youtube video - currently at around 1900 plays, so not that widely spread. In the end an interesting indicator about spam campaign success.

I have been playing around with Neo4j as graph database, and searching for a big dataset I decided to look at Debian packages (source and binary) from stable, testing, sid, and experimental, and represent all of that in a big graph database. While this is far from ready, the following entities and relations a represented:

• source packages, unversioned and versioned
• binary packages, unversioned and versioned
• maintainers
• all dependencies, including alternatives and versioned dependencies
• relations like maintains, builds, etc
• suites (stable, testing, sid, experimental)

The graph currently has 220618 nodes and 782323 edges, and my first trial to import this into the database was by generating a long cypher statement, and then throwing that at cypher-shell. Well, that was not the best idea. After 24h I stopped the process and rewrote the generation script to generate csv files. Using neo4j-import the same amount of data was imported in 5secs (!!!). What I would like to get in the future is the whole package history as well, and maybe also include all the bugs into the database if I only would have easily accessible and parseable information about these items (Debian Q&A maybe?). If you have any suggestions, please let me know. More to come, stay tuned.

In order to proceed further on our USB/STM32 oddessy, we need to start to build a USB descriptor set for our first prototype piece of code. For this piece, we're going to put together a USB device which is vendor-specific class-wise and has a single configuration with a interface with a single endpoint which we're not going to actually implement anything of. What we're after is just to get the information presented to the computer so that lsusb can see it. To get these built, let's refer to information we discovered and recorded in a previous post about how descriptors go together.

Device descriptor Remembering that values which are > 1 byte in length are always stored little-endian, we can construct our device descriptor as:

Our device descriptor

Field Name	Value	Bytes
bLength	18	0x12
bDescriptorType	DEVICE	0x01
bcdUSB	USB 2.0	0x00 0x02
bDeviceClass	0	0x00
bDeviceSubClass	0	0x00
bDeviceProtocol	0	0x00
bMaxPacketSize	64	0x40
idVendor	TEST	0xff 0xff
idProduct	TEST	0xff 0xff
bcdDevice	0.0.1	0x01 0x00
iManufacturer	1	0x01
iProduct	2	0x02
iSerialNumber	3	0x03
bNumConfigurations	1	0x01

We're using the vendor ID and product id 0xffff because at this point we don't have any useful values for this (it costs $5,000 to register a vendor ID). This gives us a final byte array of:

0x12 0x01 0x00 0x02 0x00 0x00 0x00 0x40 (Early descriptor) 0xff 0xff 0xff 0xff 0x01 0x00 0x01 0x02 0x03 0x01 (and the rest)

We're reserving string ids 1, 2, and 3, for the manufacturer string, product name string, and serial number string respectively. I'm deliberately including them all so that we can see it all come out later in lsusb. If you feed the above hex sequence into a USB descriptor decoder then you can check my working.

Endpoint Descriptor We want a single configuration, which covers our one interface, with one endpoint in it. Let's start with the endpoint...

Our bulk IN endpoint

Field Name	Value	Bytes
bLength	7	0x07
bDescriptorType	ENDPOINT	0x05
bEndpointAddress	EP2IN	0x82
bmAttributes	BULK	0x02
wMaxPacketSize	64	0x40 0x00
bInterval	IGNORED	0x00

We're giving a single bulk IN endpoint, since that's the simplest thing to describe at this time. This endpoint will never be ready and so nothing will ever be read into the host. All that gives us:

0x07 0x05 0x82 0x02 0x40 0x00 0x00

Interface Descriptor The interface descriptor prefaces the endpoint set, and thanks to our simple single endpoint, and no plans for alternate interfaces, we can construct the interface simply as:

Our single simple interface

Field Name	Value	Bytes
bLength	9	0x09
bDescriptorType	INTERFACE	0x04
bInterfaceNumber	1	0x01
bAlternateSetting	1	0x01
bNumEndpoints	1	0x01
bInterfaceClass	0	0x00
bInterfaceSubClass	0	0x00
bInterfaceProtocol	0	0x00
iInterface	5	0x05

All that gives us:

0x09 0x04 0x01 0x01 0x01 0x00 0x00 0x00 0x05

Configuration descriptor Finally we can put it all together and get the configuration descriptor...

Our sole configuration, encapsulating the interface and endpoint above

Field Name	Value	Bytes
bLength	9	0x09
bDescriptorType	CONFIG	0x02
wTotalLength	9+9+7	0x19 0x00
bNumInterfaces	1	0x01
bConfigurationValue	1	0x01
iConfiguration	4	0x04
bmAttributes	Bus powered, no wake	0x80
bMaxPower	500mA	0xfa

The wTotalLength field is interesting. It contains the configuration length, the interface length, and the endpoint length, hence 9 plus 9 plus 7 is 25. This gives:

0x09 0x02 0x19 0x00 0x01 0x01 0x04 0x80 0xfa

String descriptors We allowed ourselves a total of five strings, they were iManufacturer, iProduct, iSerial (from the device descriptor), iConfiguration (from the configuration descriptor), and iInterface (from the interface descriptor) respectively. Our string descriptors will therefore be:

String descriptor zero, en_GB only

Field Name	Value	Bytes
bLength	4	0x04
bDescriptorType	STRING	0x03
wLangID[0]	en_GB	0x09 0x08

0x04 0x03 0x09 0x08

...and...

String descriptor one, iManufacturer

Field Name	Value	Bytes
bLength	38	0x26
bDescriptorType	STRING	0x03
bString	"Rusty Manufacturer"	...

0x26 0x03 0x52 0x00 0x75 0x00 0x73 0x00 0x74 0x00 0x79 0x00 0x20 0x00 0x4d 0x00 0x61 0x00 0x6e 0x00 0x75 0x00 0x66 0x00 0x61 0x00 0x63 0x00 0x74 0x00 0x75 0x00 0x72 0x00 0x65 0x00 0x72 0x00

(You get the idea, there's no point me breaking down the rest of the string descriptors here, suffice it to say that the other strings are appropriate for the values they represent - namely product, serial, configuration, and interface.)

Putting it all together Given all the above, we have a device descriptor which is standalone, then a configuration descriptor which encompasses the interface and endpoint descriptors too. Finally we have a string descriptor table with six entries, the first is the language sets available, and the rest are our strings. In total we have:

    // Device descriptor
    const DEV_DESC: [u8; 18] =  
        0x12, 0x01, 0x00, 0x02, 0x00, 0x00, 0x00, 0x40,
        0xff, 0xff, 0xff, 0xff, 0x01, 0x00, 0x01, 0x02,
        0x03, 0x01
     ;
    // Configuration descriptor
    const CONF_DESC: [u8; 25] =  
        0x09, 0x02, 0x19, 0x00, 0x01, 0x01, 0x04, 0x80, 0xfa,
        0x09, 0x04, 0x01, 0x01, 0x01, 0x00, 0x00, 0x00, 0x05,
        0x07, 0x05, 0x82, 0x02, 0x40, 0x00, 0x00
     ;
    // String Descriptor zero
    const STR_DESC_0: [u8; 4] =  0x04, 0x03, 0x09, 0x08 ;
    // String Descriptor 1, "Rusty Manufacturer"
    const STR_DESC_1: [u8; 38] =  
        0x26, 0x03, 0x52, 0x00, 0x75, 0x00, 0x73, 0x00,
        0x74, 0x00, 0x79, 0x00, 0x20, 0x00, 0x4d, 0x00,
        0x61, 0x00, 0x6e, 0x00, 0x75, 0x00, 0x66, 0x00,
        0x61, 0x00, 0x63, 0x00, 0x74, 0x00, 0x75, 0x00,
        0x72, 0x00, 0x65, 0x00, 0x72, 0x00
     ;
    // String Descriptor 2, "Rusty Product"
    const STR_DESC_2: [u8; 28] =  
        0x1c, 0x03, 0x52, 0x00, 0x75, 0x00, 0x73, 0x00,
        0x74, 0x00, 0x79, 0x00, 0x20, 0x00, 0x50, 0x00,
        0x72, 0x00, 0x6f, 0x00, 0x64, 0x00, 0x75, 0x00,
        0x63, 0x00, 0x74, 0x00
     ;
    // String Descriptor 3, "123ABC"
    const STR_DESC_3: [u8; 14] =  
        0x0e, 0x03, 0x31, 0x00, 0x32, 0x00, 0x33, 0x00,
        0x41, 0x00, 0x42, 0x00, 0x43, 0x00
     ;
    // String Descriptor 4, "Rusty Configuration"
    const STR_DESC_4: [u8; 40] =  
        0x28, 0x03, 0x52, 0x00, 0x75, 0x00, 0x73, 0x00,
        0x74, 0x00, 0x79, 0x00, 0x20, 0x00, 0x43, 0x00,
        0x6f, 0x00, 0x6e, 0x00, 0x66, 0x00, 0x69, 0x00,
        0x67, 0x00, 0x75, 0x00, 0x72, 0x00, 0x61, 0x00,
        0x74, 0x00, 0x69, 0x00, 0x6f, 0x00, 0x6e, 0x00
     ;
    // String Descriptor 5, "Rusty Interface"
    const STR_DESC_5: [u8; 32] =  
        0x20, 0x03, 0x52, 0x00, 0x75, 0x00, 0x73, 0x00,
        0x74, 0x00, 0x79, 0x00, 0x20, 0x00, 0x49, 0x00,
        0x6e, 0x00, 0x74, 0x00, 0x65, 0x00, 0x72, 0x00,
        0x66, 0x00, 0x61, 0x00, 0x63, 0x00, 0x65, 0x00
     ;

With the above, we're a step closer to our first prototype which will hopefully be enumerable. Next time we'll look at beginning our prototype low level USB device stack mock-up.

Each time I remind people about one of my best blog posts, I do have positive reviews and a peak of traffic on my blog. But as an IT guy, I hate (so much) manually (gosh!) posting reminders of my articles on both my Twitter account and my Mastodon account. So I wrote Remindr. Each time you launch it, it posts a content through both Mastodon and Twitter. You can attach an image for each message and using different languages is managed for your different content is managed. Under the hood, it s a self hosted (my instance just runs on my workstation) Python 3 application released under the GPLv3 license. Going further with Remindr

• Remindr on Gitlab.com (would be awesome starring it if you like/use it)
• The official documentation on Readthedocs.io

How does exactly Remindr work? Remindr iterates through a list of messages in a file you write and extract one line for each execution of Remindr, adding a user-defined prefix and send them to both the Mastodon and Twitter social networks. Here is the format of the file:

o en Automatically Send Toots To The Mastodon Social Network https://carlchenet.com/automatically-send-toots-to-the-mastodon-social-network/ #Mastodon
x fr Sur Mastodon, cr er son compte de secours  ou tout perdre https://carlchenet.com/sur-mastodon-creer-son-compte-de-secours-ou-tout-perdre/ #Mastodon
x en Automatically boost cool toots on Mastodon with the Boost bot https://carlchenet.com/automatically-boost-cool-toots-on-mastodon-with-the-boost-bot/ #Mastodon
x en Cryptocurrencies On the New Social Network Mastodon #Mastodon #bitcoin #ethereum #monero

The first field only indicates if the line is the next one to be considered by Remindr, the o indicates the next line to be sent. The second field is the 2-letters code language your content is using, in my example en or fr. Next content on the line will compose the body of your messages to Mastodon and Twitter. So each time you launch Remindr, one of the lines of your file will be sent to Mastodon and Twitter. How easy to use is that? and finally You can help me developing tools for Mastodon and other social networks by donating anything through Liberaypay (also possible with cryptocurrencies). Any contribution will be appreciated. That s a big factor motivation
Donate You also may follow my account @carlchenet on Mastodon