Search Results: "reg"

12 May 2020

Daniel Silverstone: The Lars, Mark, and Daniel Club

Last night, Lars, Mark, and I discussed Jeremy Kun's The communicative value of using Git well post. While a lot of our discussion was spawned by the article, we did go off-piste a little, and I hope that my notes below will enlighten you all as to a bit of how we see revision control these days. It was remarkably pleasant to read an article where the comments section wasn't a cesspool of horror, so if this posting encourages you to go and read the article, don't stop when you reach the bottom -- the comments are good and useful too.
This was a fairly non-contentious article for us though each of us had points we wished to bring up and chat about it turned into a very convivial chat. We saw the main thrust of the article as being about using the metadata of revision control to communicate intent, process, and decision making. We agreed that it must be possible to do so effectively with Mercurial (thus deciding that the mention of it was simply a bit of clickbait / red herring) and indeed Mark figured that he was doing at least some of this kind of thing way back with CVS. We all discussed how knowing the fundamentals of Git's data model improved our ability to work wih the tool. Lars and I mentioned how jarring it has originally been to come to Git from revision control systems such as Bazaar (bzr) but how over time we came to appreciate Git for what it is. For Mark this was less painful because he came to Git early enough that there was little more than the fundamental data model, without much of the porcelain which now exists. One point which we all, though Mark in particular, felt was worth considering was that of distinguishing between published and unpublished changes. The article touches on it a little, but one of the benefits of the workflow which Jeremy espouses is that of using the revision control system as an integral part of the review pipeline. This is perhaps done very well with Git based workflows, but can be done with other VCSs. With respect to the points Jeremy makes regarding making commits which are good for reviewing, we had a general agreement that things were good and sensible, to a point, but that some things were missed out on. For example, I raised that commit messages often need to be more thorough than one-liners, but Jeremy's examples (perhaps through expedience for the article?) were all pretty trite one-liners which perhaps would have been entirely obvious from the commit content. Jeremy makes the point that large changes are hard to review, and Lars poined out that Greg Wilson did research in this area, and at least one article mentions 200 lines as a maximum size of a reviewable segment. I had a brief warble at this point about how reviewing needs to be able to consider the whole of the intended change (i.e. a diff from base to tip) not just individual commits, which is also missing from Jeremy's article, but that such a review does not need to necessarily be thorough and detailed since the commit-by-commit review remains necessary. I use that high level diff as a way to get a feel for the full shape of the intended change, a look at the end-game if you will, before I read the story of how someone got to it. As an aside at this point, I talked about how Jeremy included a 'style fixes' commit in his example, but I loathe seeing such things and would much rather it was either not in the series because it's unrelated to it; or else the style fixes were folded into the commits they were related to. We discussed how style rules, as well as commit-bisectability, and other rules which may exist for a codebase, the adherence to which would form part of the checks that a code reviewer may perform, are there to be held to when they help the project, and to be broken when they are in the way of good communication between humans. In this, Lars talked about how revision control histories provide high level valuable communication between developers. Communication between humans is fraught with error and the rules are not always clear on what will work and what won't, since this depends on the communicators, the context, etc. However whatever communication rules are in place should be followed. We often say that it takes two people to communicate information, but when you're writing commit messages or arranging your commit history, the second party is often nebulous "other" and so the code reviewer fulfils that role to concretise it for the purpose of communication. At this point, I wondered a little about what value there might be (if any) in maintaining the metachanges (pull request info, mailing list discussions, etc) for historical context of decision making. Mark suggested that this is useful for design decisions etc but not for the style/correctness discussions which often form a large section of review feedback. Maybe some of the metachange tracking is done automatically by the review causing the augmentation of the changes (e.g. by comments, or inclusion of design documentation changes) to explain why changes are made. We discussed how the "rebase always vs. rebase never" feeling flip-flopped in us for many years until, as an example, what finally won Lars over was that he wants the history of the project to tell the story, in the git commits, of how the software has changed and evolved in an intentional manner. Lars said that he doesn't care about the meanderings, but rather about a clear story which can be followed and understood. I described this as the switch from "the revision history is about what I did to achieve the goal" to being more "the revision history is how I would hope someone else would have done this". Mark further refined that to "The revision history of a project tells the story of how the project, as a whole, chose to perform its sequence of evolution." We discussed how project history must necessarily then contain issue tracking, mailing list discussions, wikis, etc. There are exist free software projects where part of their history is forever lost because, for example, the project moved from Sourceforge to Github, but made no effort (or was unable) to migrate issues or somesuch. Linkages between changes and the issues they relate to can easily be broken, though at least with mailing lists you can often rewrite URLs if you have something consistent like a Message-Id. We talked about how cover notes, pull request messages, etc. can thus also be lost to some extent. Is this an argument to always use merges whose message bodies contain those details, rather than always fast-forwarding? Or is it a reason to encapsulate all those discussions into git objects which can be forever associated with the changes in the DAG? We then diverted into discussion of CI, testing every commit, and the benefits and limitations of automated testing vs. manual testing; though I think that's a little too off-piste for even this summary. We also talked about how commit message audiences include software perhaps, with the recent movement toward conventional commits and how, with respect to commit messages for machine readability, it can become very complex/tricky to craft good commit messages once there are multiple disparate audiences. For projects the size of the Linux kernel this kind of thing would be nearly impossible, but for smaller projects, perhaps there's value. Finally, we all agreed that we liked the quote at the end of the article, and so I'd like to close out by repeating it for you all... Hal Abelson famously said:
Programs must be written for people to read, and only incidentally for machines to execute.
Jeremy agrees, as do we, and extends that to the metacommit information as well.

Petter Reinholdtsen: Debian Edu interview: Yvan Masson

It has been way too long since my last interview, but as the Debian Edu / Skolelinux community is still active, and new people keep showing up on the IRC channel #debian-edu and the debian-edu mailing list, I decided to give it another go. I was hoping someone else might pick up the idea and run with it, but this has not happened as far as I can tell, so here we are This time the announcement of a new free software tool to create a school year book triggered my interest, and I decided to learn more about its author. Who are you, and how do you spend your days? My name is Yvan MASSON, I live in France. I have my own one person business in computer services. The work consist of visiting my customers (person's home, local authority, small business) to give advise, install computers and software, fix issues, and provide computing usage training. I spend the rest of my time enjoying my family and promoting free software. What is your approach for promoting free software? When I think that free software could be suitable for someone, I explain what it is, with simple words, give a few known examples, and explain that while there is no fee it is a viable alternative in many situations. Most people are receptive when you explain how it is better (I simplify arguments here, I know that it is not so simple): Linux works on older hardware, there are no viruses, and the software can be audited to ensure user is not spied upon. I think the most important is to keep a clear but moderated speech: when you try to convince too much, people feel attacked and stop listening. How did you get in contact with the Skolelinux / Debian Edu project? I can not remember how I first heard of Skolelinux / Debian Edu, but probably on planet.debian.org. As I have been working for a school, I have interest in this type of project. The school I am involved in is a school for "children" between 14 and 18 years old. The French government has recommended free software since 2012, but they do not always use free software themselves. The school computers are still using the Windows operating system, but all of them have the classic set of free software: Firefox ESR, LibreOffice (with the excellent extension Grammalecte that indicates French grammatical errors), SumatraPDF, Audacity, 7zip, KeePass2, VLC, GIMP, Inkscape What do you see as the advantages of Skolelinux / Debian Edu? It is free software! Built on Debian, I am sure that users are not spied upon, and that it can run on low end hardware. This last point is very important, because we really need to improve "green IT". I do not know enough about Skolelinux / Debian Edu to tell how it is better than another free software solution, but what I like is the "all in one" solution: everything has been thought of and prepared to ease installation and usage. I like Free Software because I hate using something that I can not understand. I do not say that I can understand everything nor that I want to understand everything, but knowing that someone / some company intentionally prevents me from understanding how things work is really unacceptable to me. Secondly, and more importantly, free software is a requirement to prevent abuses regarding human rights and environmental care. Humanity can not rely on tools that are in the hands of small group of people. What do you see as the disadvantages of Skolelinux / Debian Edu? Again, I don't know this project enough. Maybe a dedicated website? Debian wiki works well for documentation, but is not very appealing to someone discovering the project. Also, as Skolelinux / Debian Edu uses OpenLDAP, it probably means that Windows workstations cannot use centralized authentication. Maybe the project could use Samba as an Active Directory domain controller instead, allowing Windows desktop usage when necessary. (Editors note: In fact Windows workstations can use the centralized authentication in a Debian Edu setup, at least for some versions of Windows, but the fact that this is not well known can be seen as an indication of the need for better documentation and marketing. :) Which free software do you use daily? Nothing original: Debian testing/sid with Gnome desktop, Firefox, Thunderbird, LibreOffice Which strategy do you believe is the right one to use to get schools to use free software? Every effort to spread free software into schools is important, whatever it is. But I think, at least where I live, that IT professionals maintaining schools networks are still very "Microsoft centric". Schools will use any working solution, but they need people to install and maintain it. How to make these professionals sensitive about free software and train them with solutions like Debian Edu / Skolelinux is a really good question :-)

Jacob Adams: Roman Finger Counting

I recently wrote a final paper on the history of written numerals. In the process, I discovered this fascinating tidbit that didn t really fit in my paper, but I wanted to put it somewhere. So I m writing about it here. If I were to ask you to count as high as you could on your fingers you d probably get up to 10 before running out of fingers. You can t count any higher than the number of fingers you have, right? The Romans could! They used a place-value system, combined with various gestures to count all the way up to 9,999 on two hands.

The System Finger Counting (Note that in this diagram 60 is, in fact, wrong, and this picture swaps the hundreds and the thousands.) We ll start with the units. The last three fingers of the left hand, middle, ring, and pinkie, are used to form them. Zero is formed with an open hand, the opposite of the finger counting we re used to. One is formed by bending the middle joint of the pinkie, two by including the ring finger and three by including the middle finger, all at the middle joint. You ll want to keep all these bends fairly loose, as otherwise these numbers can get quite uncomfortable. For four, you extend your pinkie again, for five, also raise your ring finger, and for six, you raise your middle finger as well, but then lower your ring finger. For seven you bend your pinkie at the bottom joint, for eight adding your ring finger, and for nine, including your middle finger. This mirrors what you did for one, two and three, but bending the finger at the bottom joint now instead. This leaves your thumb and index finger for the tens. For ten, touch the nail of your index finger to the inside of your top thumb joint. For twenty, put your thumb between your index and middle fingers. For thirty, touch the nails of your thumb and index fingers. For forty, bend your index finger slightly towards your palm and place your thumb between the middle and top knuckle of your index finger. For fifty, place your thumb against your palm. For sixty, leave your thumb where it is and wrap your index finger around it (the diagram above is wrong). For seventy, move your thumb so that the nail touches between the middle and top knuckle of your index finger. For eighty, flip your thumb so that the bottom of it now touches the spot between the middle and top knuckle of your index finger. For ninety, touch the nail of your index finger to your bottom thumb joint. The hundreds and thousands use the same positions on the right hand, with the units being the thousands and the tens being the hundreds. One account, from which the picture above comes, swaps these two, but the first account we have uses this ordering. Combining all these symbols, you can count all the way to 9,999 yourself on just two hands. Try it!

History

The Venerable Bede The first written record of this system comes from the Venerable Bede, an English Benedictine monk who died in 735. He wrote De computo vel loquela digitorum, On Calculating and Speaking with the Fingers, as the introduction to a larger work on chronology, De temporum ratione. (The primary calculation done by monks at the time was calculating the date of Easter, the first Sunday after the first full moon of spring). He also includes numbers from 10,000 to 1,000,000, but its unknown if these were inventions of the author and were likely rarely used regardless. They require moving your hands to various positions on your body, as illustrated below, from Jacob Leupold s Theatrum Arilhmetico-Geometricum, published in 1727: Finger Counting with Large Numbers

The Romans If Bede was the first to write it, how do we know that it came from Roman times? It s referenced in many Roman writings, including this bit from the Roman satirist Juvenal who died in 130:
Felix nimirum qui tot per saecula mortem distulit atque suos iam dextera computat annos. Happy is he who so many times over the years has cheated death And now reckons his age on the right hand.
Because of course the right hand is where one counts hundreds! There s also this Roman riddle:
Nunc mihi iam credas fieri quod posse negatur: octo tenes manibus, si me monstrante magistro sublatis septem reliqui tibi sex remanebunt. Now you shall believe what you would deny could be done: In your hands you hold eight, as my teacher once taught; Take away seven, and six still remain.
If you form eight with this system and then remove the symbol for seven, you get the symbol for six!

Sources My source for this blog post is Paul Broneer s 1969 English translation of Karl Menninger s Zahlwort und Ziffer (Number Words and Number Symbols).

10 May 2020

Dirk Eddelbuettel: #1 T^4: Adding Some Color to the Shell

The first proper video (following last week s announcement) is up for new T^4 series of video lightning talks with tips, tricks, tools, and toys. Today we just to a little enhancement for the shell enabled color output (if not already on by default).

The slides are available here. Next week we continue on shell customization by looking at the prompt. Also of note, a new repo at GitHub to support the series: use it to open issues for comments, criticism, suggestions, or feedback. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Russ Allbery: Review: Golden Gates

Review: Golden Gates, by Conor Dougherty
Publisher: Penguin
Copyright: 2020
ISBN: 0-525-56022-X
Format: Kindle
Pages: 249
This review, for reasons that will hopefully become clear later, starts with a personal digression. I have been interested in political theory my entire life. That sounds like something admirable, or at least neutral. It's not. "Interested" means that I have opinions that are generally stronger than my depth of knowledge warrants. "Interested" means that I like thinking about and casting judgment on how politics should be done without doing the work of politics myself. And "political theory" is different than politics in important ways, not the least of which is that political actions have rarely been a direct danger to me or my family. I have the luxury of arguing about politics as a theory. In short, I'm at high risk of being one of those people who has an opinion about everything and shares it on Twitter. I'm still in the process (to be honest, near the beginning of the process) of making something useful out of that interest. I've had some success when I become enough a part of a community that I can do some of the political work, understand the arguments at a level deeper than theory, and have to deal with the consequences of my own opinions. But those communities have been on-line and relatively low stakes. For the big political problems, the ones that involve governments and taxes and laws, those that decide who gets medical treatment and income support and who doesn't, to ever improve, more people like me need to learn enough about the practical details that we can do the real work of fixing them, rather than only making our native (and generally privileged) communities better for ourselves. I haven't found my path helping with that work yet. But I do have a concrete, challenging, local political question that makes me coldly furious: housing policy. Hence this book. Golden Gates is about housing policy in the notoriously underbuilt and therefore incredibly expensive San Francisco Bay Area, where I live. I wanted to deepen that emotional reaction to the failures of housing policy with facts and analysis. Golden Gates does provide some of that. But this also turns out to be a book about the translation of political theory into practice, about the messiness and conflict that results, and about the difficult process of measuring success. It's also a book about how substantial agreement on the basics of necessary political change can still founder on the shoals of prioritization, tribalism, and people who are interested in political theory. In short, it's a book about the difficulty of changing the world instead of arguing about how to change it. This is not a direct analysis of housing policy, although Dougherty provides the basics as background. Rather, it's the story of the political fight over housing told primarily through two lenses: Sonja Trauss, founder of BARF (the Bay Area Renters' Federation); and a Redwood City apartment complex, the people who fought its rent increases, and the nun who eventually purchased it. Around that framework, Dougherty writes about the Howard Jarvis Taxpayers Association and the history of California's Proposition 13, a fight over a development in Lafayette, the logistics challenge of constructing sufficient housing even when approved, and the political career of Scott Wiener, the hated opponent of every city fighting for the continued ability to arbitrarily veto any new housing. One of the things Golden Gates helped clarify for me is that there are three core interest groups that have to be part of any discussion of Bay Area housing: homeowners who want to limit or eliminate local change, renters who are vulnerable to gentrification and redevelopment, and the people who want to live in that area and can't (which includes people who want to move there, but more sympathetically includes all the people who work there but can't afford to live locally, such as teachers, day care workers, food service workers, and, well, just about anyone who doesn't work in tech). (As with any political classification, statements about collectives may not apply to individuals; there are numerous people who appear to fall into one group but who vote in alignment with another.) Dougherty makes it clear that housing policy is intractable in part because the policies that most clearly help one of those three groups hurt the other two. As advertised by the subtitle, Dougherty's focus is on the fight for more housing. Those who already own homes whose values have been inflated by artificial scarcity, or who want to preserve such stratified living conditions as low-density, large-lot single-family dwellings within short mass-transit commute of one of the densest cities in the United States, don't get a lot of sympathy or focus here except as opponents. I understand this choice; I also don't have much sympathy. But I do wish that Dougherty had spent more time discussing the unsustainable promise that California has implicitly made to homeowners: housing may be impossibly expensive, but if you can manage to reach that pinnacle of financial success, the ongoing value of your home is guaranteed. He does mention this in passing, but I don't think he puts enough emphasis on the impact that a single huge, illiquid investment that is heavily encouraged by government policy has on people's attitude towards anything that jeopardizes that investment. The bulk of this book focuses on the two factions trying to make housing cheaper: Sonja Trauss and others who are pushing for construction of more housing, and tenant groups trying to manage the price of existing housing for those who have to rent. The tragedy of Bay Area housing is that even the faintest connection of housing to the economic principle of supply and demand implies that the long-term goals of those two groups align. Building more housing will decrease the cost of housing, at least if you build enough of it over a long enough period of time. But in the short term, particularly given the amount of Bay Area land pre-emptively excluded from housing by environmental protection and the actions of the existing homeowners, building more housing usually means tearing down cheap lower-density housing and replacing it with expensive higher-density housing. And that destroys people's lives. I'll admit my natural sympathy is with Trauss on pure economic grounds. There simply aren't enough places to live in the Bay Area, and the number of people in the area will not decrease. To the marginal extent that growth even slows, that's another tale of misery involving "super commutes" of over 90 minutes each way. But the most affecting part of this book was the detailed look at what redevelopment looks like for the people who thought they had housing, and how it disrupts and destroys existing communities. It's impossible to read those stories and not be moved. But it's equally impossible to not be moved by the stories of people who live in their cars during the week, going home only on weekends because they have to live too far away from their jobs to commute. This is exactly the kind of politics that I lose when I take a superficial interest in political theory. Even when I feel confident in a guiding principle, the hard part of real-world politics is bringing real people with you in the implementation and mitigating the damage that any choice of implementation will cause. There are a lot of details, and those details matter. Without the right balance between addressing a long-term deficit and providing short-term protection and relief, an attempt to alleviate unsustainable long-term misery creates more short-term misery for those least able to afford it. And while I personally may have less sympathy for the relatively well-off who have clawed their way into their own mortgage, being cavalier with their goals and their financial needs is both poor ethics and poor politics. Mobilizing political opponents who have resources and vote locally isn't a winning strategy. Dougherty is a reporter, not a housing or public policy expert, so Golden Gates poses problems and tells stories rather than describes solutions. This book didn't lead me to a brilliant plan for fixing the Bay Area housing crunch, or hand me a roadmap for how to get effectively involved in local politics. What it did do is tell stories about what political approaches have worked, how they've worked, what change they've created, and the limitations of that change. Solving political problems is work. That work requires understanding people and balancing concerns, which in turn requires a lot of empathy, a lot of communication, and sometimes finding a way to make unlikely allies. I'm not sure how broad the appeal of this book will be outside of those who live in the region. Some aspects of the fight for housing generalize, but the Bay Area (and I suspect every region) has properties specific to it or to the state of California. It has also reached an extreme of housing shortage that is rivaled in the United States only by New York City, which changes the nature of the solutions. But if you want to seriously engage with Bay Area housing policy, knowing the background explained here is nearly mandatory. There are some flaws I wish Dougherty would have talked more about traffic and transit policy, although I realize that could be another book but this is an important story told well. If this somewhat narrow topic is within your interests, highly recommended. Rating: 8 out of 10

9 May 2020

Andrew Cater: CD image testing for Buster release 4 - 202005091950 - Most install images checking out well

Lots of hard work going on. schweer has just validated all of the Debian Edu images. Most of the normal install images have gone through tests with only a few minor hitches. Now moving on to the Live images. These take longer to download and test but we're working through them gradually.

As ever: a point release doesn't mean that the Debian you have is now obsolete - an apt-get / aptitude update will bring you up to the latest release very quickly. If you are updating regularly, you will have most of these files anyway. One small thing: the tools may report that the release version has changed. This is quite normal - base files have changed to reflect the new point release and this causes the notification. The notification is a small warning so that you are not taken by complete surprise but it is quite normal in the circumstances of a Debian point release.

Thanks to the other folk doing the hard work: 10+ hours and continuing.

Dirk Eddelbuettel: ttdo 0.0.5: Reflect tinytest update

A maintenance release of our (still small) ttdo package just arrived on CRAN. As introduced last fall, the ttdo package extends the most excellent (and very minimal / zero depends) unit testing package tinytest by Mark van der Loo with the very clever and well-done diffobj package by Brodie Gaslam to give us test results with visual diffs: ttdo screenshot tinytest has an extension mechanism we use, and as tinytest was just upgraded to version 1.2.0 changing, among other nice extensions, one interface by allowing for a new error class argument, we had to rebuild as well in order to document the new argument. The release was actually prepared three days ago when tinytest itself was updated, but we waited for the binaries at CRAN to be updated and rebuilt to take advantage of the fully automated submission and test process at CRAN. The NEWS entry follow.

Changes in ttdo version 0.0.5 (2020-05-06)
  • Rebuilt under tinytest 1.2.0 to add support for class argument in error-code test predicates

CRANberries provides the usual summary of changes to the previous version. Please use the GitHub repo and its issues for any questions. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Andrew Cater: And again - CD release for Buster release 4 is going to happen over the weekend ...

And for our regular readers - it's happening again. I'm sitting here with two laptops, a desktop, a connection to IRC - and friends in Cambridge and elsewhere involved in this too.

All good fun, as ever :)

8 May 2020

Dirk Eddelbuettel: Rcpp Virtual Talk on June 5

We had to cancel R/Finance 2020 due to what is happening all around us. But I plan to present the one-hour workshop I often give in the tutorial session preceding the first day but this time online! To keep it simple, we will stick with the same day, and possibly the same time: Friday morning at 8:00am! So that makes Friday, June 5, at 08:00h Central time. This YouTube! link should then provide the stream, I reckon there may also be a recording afterwards. The talk / demo / presentation will be about an hour long, and material should be similar to the previous ones (of the same length) still available at the talks page (which also has longer talks all the way to the two-day workshops). If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

6 May 2020

Reproducible Builds: Reproducible Builds in April 2020

Welcome to the April 2020 report from the Reproducible Builds project. In our regular reports we outline the most important things that we and the rest of the community have been up to over the past month. What are reproducible builds? One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. But whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into seemingly secure software during the various compilation and distribution processes.

News It was discovered that more than 725 malicious packages were downloaded thousands of times from RubyGems, the official channel for distributing code for the Ruby programming language. Attackers used a variation of typosquatting and replaced hyphens and underscores (for example, uploading a malevolent atlas-client in place of atlas_client) that executed a script that intercepted Bitcoin payments. (Ars Technica report) Bernhard M. Wiedemann launched ismypackagereproducibleyet.org, a service that takes a package name as input and displays whether the package is reproducible in a number of distributions. For example, it can quickly show the status of Perl as being reproducible on openSUSE but not in Debian. Bernhard also improved the documentation of his unreproducible package to add some example patches for hash issues. [ ]. There was a post on Chaos Computer Club s website listing Ten requirements for the evaluation of Contact Tracing apps in relation to the SARS-CoV-2 epidemic. In particular:
4. Transparency and verifiability: The complete source code for the app and infrastructure must be freely available without access restrictions to allow audits by all interested parties. Reproducible build techniques must be used to ensure that users can verify that the app they download has been built from the audited source code.
Elsewhere, Nicolas Boulenguez wrote a patch for the Ada programming language component of the GCC compiler to skip -f.*-prefix-map options when writing Ada Library Information files. Amongst other properties, these .ali files embed the compiler flags used at the time of the build which results in the absolute build path being recorded via -ffile-prefix-map, -fdebug-prefix-map, etc. In the Arch Linux project, kpcyrd reported that they held their first rebuilder workshop . The session was held on IRC and participants were provided a document with instructions on how to install and use Arch s repro tool. The meeting resulted in multiple people with no prior experience of Reproducible Builds validate their first package. Later in the month he also announced that it was now possible to run independent rebuilders under Arch in a hands-off, everything just works solution to distributed package verification. Mathias Lang submitted a pull request against dmd, the canonical compiler for the D programming languageto add support for our SOURCE_DATE_EPOCH environment variable as well the other C preprocessor tokens such __DATE__, __TIME__ and __TIMESTAMP__ which was subsequently merged. SOURCE_DATE_EPOCH defines a distribution-agnostic standard for build toolchains to consume and emit timestamps in situations where they are deemed to be necessary. [ ] The Telegram instant-messaging platform announced that they had updated to version 5.1.1 continuing their claim that they are reproducible according to their full instructions and therefore verifying that its original source code is exactly the same code that is used to build the versions available on the Apple App Store and Google Play distribution platforms respectfully. Lastly, Herv Boutemy reported that 97% of the current development versions of various Maven packages appear to have a reproducible build. [ ]

Distribution work In Debian this month, 89 reviews of Debian packages were added, 21 were updated and 33 were removed this month adding to our knowledge about identified issues. Many issue types were noticed, categorised and updated by Chris Lamb, including: In addition, Holger Levsen filed a feature request against debrebuild, a tool for rebuilding a Debian package given a .buildinfo file, proposing to add --standalone or --one-shot-mode functionality.
In openSUSE, Bernhard M. Wiedemann made the following changes: In Arch Linux, a rebuilder instance has been setup at reproducible.archlinux.org that is rebuilding Arch s [core] repository directly. The first rebuild has led to approximately 90% packages reproducible contrasting with 94% on the Reproducible Build s project own ArchLinux status page on tests.reproducible-builds.org that continiously builds packages and does not verify Arch Linux packages. More information may be found on the corresponding wiki page and the underlying decisions were explained on our mailing list.

Software development

diffoscope Chris Lamb made the following changes to diffoscope, the Reproducible Builds project s in-depth and content-aware diff utility that can locate and diagnose reproducibility issues (including preparing and uploading versions 139, 140, 141, 142 and 143 to Debian which were subsequently uploaded to the backports repository):
  • Comparison improvements:
    • Dalvik .dex files can also serve as APK containers so restrict the narrower identification of .dex files to files ending with this extension and widen the identification of APK files to when file(1) discovers a Dalvik file. (#28)
    • Add support for Hierarchical Data Format (HD5) files. (#95)
    • Add support for .p7c and .p7b certificates. (#94)
    • Strip paths from the output of zipinfo(1) warnings. (#97)
    • Don t uselessly include the JSON similarity percentage if it is 0.0% . [ ]
    • Render multi-line difference comments in a way to show indentation. (#101)
  • Testsuite improvements:
    • Add pdftotext as a requirement to run the PDF test_metadata text. (#99)
    • apktool 2.5.0 changed the handling of output of XML schemas so update and restrict the corresponding test to match. (#96)
    • Explicitly list python3-h5py in debian/tests/control.in to ensure that we have this module installed during a test run to generate the fixtures in these tests. [ ]
    • Correct parsing of ./setup.py test --pytest-args arguments. [ ]
  • Misc:
    • Capitalise Ordering differences only in text comparison comments. [ ]
    • Improve documentation of FILE_TYPE_HEADER_PREFIX and FALLBACK_FILE_TYPE_HEADER_PREFIX to highlight that only the first 16 bytes are used. [ ]
Michael Osipov created a well-researched merge request to return diffoscope to using zipinfo directly instead of piping input via /dev/stdin in order to ensure portability to the BSD operating system [ ]. In addition, Ben Hutchings documented how --exclude arguments are matched against filenames [ ] and Jelle van der Waa updated the LLVM test fixture difference for LLVM version 10 [ ] as well as adding a reference to the name of the h5dump tool in Arch Linux [ ]. Lastly, Mattia Rizzolo also fixed in incorrect build dependency [ ] and Vagrant Cascadian enabled diffoscope to locate the openssl and h5dump packages on GNU Guix [ ][ ], and updated diffoscope in GNU Guix to version 141 [ ] and 143 [ ].

strip-nondeterminism strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. In April, Chris Lamb made the following changes:
  • Add deprecation plans to all handlers documenting how or if they could be disabled and eventually removed, etc. (#3)
  • Normalise *.sym files as Java archives. (#15)
  • Add support for custom .zip filename filtering and exclude two patterns of files generated by Maven projects in fork mode. (#13)

disorderfs disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues. This month, Chris Lamb fixed a long-standing issue by not drop UNIX groups in FUSE multi-user mode when we are not root (#1) and uploaded version 0.5.9-1 to Debian unstable. Vagrant Cascadian subsequently refreshed disorderfs in GNU Guix to version 0.5.9 [ ].

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including: In addition, Bernhard informed the following projects that their packages are not reproducible:
  • acoular (report unknown non-determinism)
  • cri-o (report a date issue)
  • gnutls (report certtool being unable to extend certificates beyond 2049)
  • gnutls (report copyright year variation)
  • libxslt (report a bug about non-deterministic output from data corruption)
  • python-astropy (report a future build failure in 2021)

Project documentation This month, Chris Lamb made a large number of changes to our website and documentation in the following categories:
  • Community engagement improvements:
    • Update instructions to register for Salsa on our Contribute page now that the signup process has been overhauled. [ ]
    • Make it clearer that joining the rb-general mailing list is probably a first step for contributors to take. [ ]
    • Make our full contact information easier to find in the footer (#19) and improve text layout using bullets to separate sections [ ].
  • Accessibility:
    • To improve accessibility, make all links underlined. (#12)
    • Use an enhanced foreground/background contrast ratio of 7.04:1. (#11)
  • General improvements:
  • Internals:
    • Move to using jekyll-redirect-from over manual redirect pages [ ][ ] and add a redirect from /docs/buildinfo/ to /docs/recording/. (#23)
    • Limit the website self-check to not scan generated files [ ] and remove the old layout checker now that I have migrated all them [ ].
    • Move the news archive under the /news/ namespace [ ] and improve formatting of archived news links [ ].
    • Various improvements to the draft template generation. [ ][ ][ ][ ]
In addition, Holger Levsen clarified exactly which month we ceased to do weekly reports [ ] and Mattia Rizzolo adjusted the title style of an event page [ ]. Marcus Hoffman also started a discussion on our website s issue tracker asking for clarification on embedded signatures and Chris Lamb subsequently replied and asked Marcus to go ahead and propose a concrete change.

Testing framework We operate a large and many-featured Jenkins-based testing framework that powers tests.reproducible-builds.org that, amongst many other tasks, tracks the status of our reproducibility efforts as well as identifies any regressions that have been introduced.
  • Chris Lamb:
    • Print the build environment prior to executing a build. [ ]
    • Drop a misleading disorderfs-debug prefix in log output when we change non-disorderfs things in the file and, as it happens, do not run disorderfs at all. [ ]
    • The CSS for the package report pages added a margin to all <a> HTML elements under <li> ones, which was causing a comma/bullet spacing issue. [ ]
    • Tidy the copy in the project links sidebar. [ ]
  • Holger Levsen:
    • General:
    • Debian:
      • Reduce scheduling frequency of the buster distribution on the arm64 architecture, etc.. [ ][ ]
      • Show builds per day on a per-architecture basis for the last year on the Debian dashboard. [ ]
      • Drop the Subgraph OS package set as development halted in 2017 or 2018. [ ]
      • Update debrebuild to version from the latest version of devscripts. [ ][ ]
      • Add or improve various parts of the documentation. [ ][ ][ ]
    • Work on a Debian rebuilder:
      • Integrate sbuild. [ ][ ][ ][ ][ ]
      • Select a random .buildinfo file and attempt to build and compare the result. [ ][ ][ ][ ]
      • Improve output and related output formatting. [ ][ ][ ][ ][ ]
      • Outline next steps for the development of the tool. [ ][ ][ ]
      • Various refactoring and code improvements. [ ][ ][ ]
Lastly, Mattia Rizzolo fixed some log parsing code regarding potentially-harmless warnings from package installation [ ][ ] and the usual build node maintenance was performed by Holger Levsen [ ][ ][ ] and Mattia Rizzolo [ ][ ][ ].

Misc news On our mailing list this month, Santiago Torres asked whether we were still publishing releases of our tools to our website and Chris Lamb replied that this was not the case and fixed the issue. Later in the month Santiago also reported that the signature for the disorderfs package did not pass its GPG verification which was also fixed by Chris Lamb. Hans-Christoph Steiner of the Guardian Project asked whether there would be interest in making our website translatable which resulted in a WIP merge request being filed against the website and a discussion on how to track translation updates.
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

This month s report was written by Bernhard M. Wiedemann, Chris Lamb, Daniel Shahaf, Holger Levsen, Jelle van der Waa, kpcyrd, Mattia Rizzolo and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.

Russell Coker: About Reopening Businesses

Currently there is political debate about when businesses should be reopened after the Covid19 quarantine. Small Businesses One argument for reopening things is for the benefit of small businesses. The first thing to note is that the protests in the US say I need a haircut not I need to cut people s hair . Small businesses won t benefit from reopening sooner. For every business there is a certain minimum number of customers needed to be profitable. There are many comments from small business owners that want it to remain shutdown. When the government has declared a shutdown and paused rent payments and provided social security to employees who aren t working the small business can avoid bankruptcy. If they suddenly have to pay salaries or make redundancy payouts and have to pay rent while they can t make a profit due to customers staying home they will go bankrupt. Many restaurants and cafes make little or no profit at most times of the week (I used to be 1/3 owner of an Internet cafe and know this well). For such a company to be viable you have to be open most of the time so customers can expect you to be open. Generally you don t keep a cafe open at 3PM to make money at 3PM, you keep it open so people can rely on there being a cafe open there, someone who buys a can of soda at 3PM one day might come back for lunch at 1:30PM the next day because they know you are open. A large portion of the opening hours of a most retail companies can be considered as either advertising for trade at the profitable hours or as loss making times that you can t close because you can t send an employee home for an hour. If you have seating for 28 people (as my cafe did) then for about half the opening hours you will probably have 2 or fewer customers in there at any time, for about a quarter the opening hours you probably won t cover the salary of the one person on duty. The weekend is when you make the real money, especially Friday and Saturday nights when you sometimes get all the seats full and people coming in for takeaway coffee and snacks. On Friday and Saturday nights the 60 seat restaurant next door to my cafe used to tell customers that my cafe made better coffee. It wasn t economical for them to have a table full for an hour while they sell a few cups of coffee, they wanted customers to leave after dessert and free the table for someone who wants a meal with wine (alcohol is the real profit for many restaurants). The plans of reopening with social distancing means that a 28 seat cafe can only have 14 chairs or less (some plans have 25% capacity which would mean 7 people maximum). That means decreasing the revenue of the most profitable times by 50% to 75% while also not decreasing the operating costs much. A small cafe has 2-3 staff when it s crowded so there s no possibility of reducing staff by 75% when reducing the revenue by 75%. My Internet cafe would have closed immediately if forced to operate in the proposed social distancing model. It would have been 1/4 of the trade and about 1/8 of the profit at the most profitable times, even if enough customers are prepared to visit and social distancing would kill the atmosphere. Most small businesses are barely profitable anyway, most small businesses don t last 4 years in normal economic circumstances. This reopen movement is about cutting unemployment benefits not about helping small business owners. Destroying small businesses is also good for big corporations, kill the small cafes and restaurants and McDonald s and Starbucks will win. I think this is part of the motivation behind the astroturf campaign for reopening businesses. Forbes has an article about this [1]. Psychological Issues Some people claim that we should reopen businesses to help people who have psychological problems from isolation, to help victims of domestic violence who are trapped at home, to stop older people being unemployed for the rest of their lives, etc. Here is one article with advice for policy makers from domestic violence experts [2]. One thing it mentions is that the primary US federal government program to deal with family violence had a budget of $130M in 2013. The main thing that should be done about family violence is to make it a priority at all times (not just when it can be a reason for avoiding other issues) and allocate some serious budget to it. An agency that deals with problems that affect families and only has a budget of $1 per family per year isn t going to be able to do much. There are ongoing issues of people stuck at home for various reasons. We could work on better public transport to help people who can t drive. We could work on better healthcare to help some of the people who can t leave home due to health problems. We could have more budget for carers to help people who can t leave home without assistance. Wanting to reopen restaurants because some people feel isolated is ignoring the fact that social isolation is a long term ongoing issue for many people, and that many of the people who are affected can t even afford to eat at a restaurant! Employment discrimination against people in the 50+ age range is an ongoing thing, many people in that age range know that if they lose their job and can t immediately find another they will be unemployed for the rest of their lives. Reopening small businesses won t help that, businesses running at low capacity will have to lay people off and it will probably be the older people. Also the unemployment system doesn t deal well with part time work. The Australian system (which I think is similar to most systems in this regard) reduces the unemployment benefits by $0.50 for every dollar that is earned in part time work, that effectively puts people who are doing part time work because they can t get a full-time job in the highest tax bracket! If someone is going to pay for transport to get to work, work a few hours, then get half the money they earned deducted from unemployment benefits it hardly makes it worthwhile to work. While the exact health impacts of Covid19 aren t well known at this stage it seems very clear that older people are disproportionately affected, so forcing older people to go back to work before there is a vaccine isn t going to help them. When it comes to these discussions I think we should be very suspicious of people who raise issues they haven t previously shown interest in. If the discussion of reopening businesses seems to be someone s first interest in the issues of mental health, social security, etc then they probably aren t that concerned about such issues. I believe that we should have a Universal Basic Income [3]. I believe that we need to provide better mental health care and challenge the gender ideas that hurt men and cause men to hurt women [4]. I believe that we have significant ongoing problems with inequality not small short term issues [5]. I don t think that any of these issues require specific changes to our approach to preventing the transmission of disease. I also think that we can address multiple issues at the same time, so it is possible for the government to devote more resources to addressing unemployment, family violence, etc while also dealing with a pandemic.
Related posts:
  1. McDonalds Wifi Without Power I am writing this post at a small cafe while...
  2. Nando s Voucher Interpretation Every year my parents buy a book of vouchers for...
  3. economics of a computer store (why they don t stock what you want) In some mailing list discussions recently some people demonstrated a...

5 May 2020

Jonathan Dowland: Introducing Red Hat UBI OpenJDK runtime images

At Red Hat we've just shipped Universal Base Image (UBI) build images for OpenJDK. Download them with your favourite container manager, e.g.: 
podman pull registry.access.redhat.com/ubi8/openjdk-8
podman pull registry.access.redhat.com/ubi8/openjdk-11
UBI, announced a year ago, is an initiative where you can obtain, share and build upon official Red Hat container images without needing a Red Hat subscription. Unlike something like CentOS, they aren't modified in any way (e.g. to remove branding), they're exactly the same base images that Red Hat products are built upon, composed entirely of Open Source software. Your precise rights are covered in the EULA. I work on the Red Hat OpenJDK container images, which are designed primarily for use with OpenShift. We've been based upon the RHEL base images since inception. Although our containers are open source (of course), we haven't been able to distribute the binary images more widely than to Red Hat customers, until now. These are based upon the "minimal" flavour base image and add an OpenJDK runtime and development packages (either JDK8 or JDK11, as per the image name) as well as Maven and OpenShift integration scripts that allow the image to be used with Source To Image. If you give these a try, please let us know what you think! Comment here, on twitter, or file Issues or Pull Requests in the GitHub project.

3 May 2020

Dirk Eddelbuettel: #0: Introducing T^4: Tips, Tricks, Tools, and Toys

For way too long now something I had meant to start was a little series about tips, tricks, tools, and toys. I had mentioned the idea a few times to a friend or two, and generally received a thumbs up or a go for it . But it takes a little to get over the humb and get going. And it turns out that last week s r^4 talk on upgrading to R 4.0.0 hit some latent demand as we are now at 1400 views on YouTube. Wowser. So hence without further ado, let s kick off T^4. Similar in spirit to R^4, but broader in scope and going beyond R. The opening slides explaining what we plan to do are here, and the video link follows below:

With some luck we should have the first actual talk next week. See you then! If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Iain R. Learmonth: Hetzner Dedicated Server Reverse DNS + Ansible

Continuing on the path towards all my stuff being managed by Ansible, I ve figured out a method of managing the reverse DNS entries for subnets on the Hetzner Dedicated Server. There s a bunch of Ansible modules for handling Hetzner Cloud, but these servers are managed in Robot which the Cloud API doesn t cover. Instead, you need to use the Robot Webservice. Ansible does have a module for doing pretty arbitrary things with web APIs though, so using that I ve got the following playbook figured out to keep the reverse DNS entries in sync: 
---
- hosts:
  - vmhost_vm1
  gather_facts: False
  tasks:
  - name: import hetzner webservice credentials
    include_vars:
      file: "hetzner_ws.yml"
  - name: set rdns for hetzner hosts
    block:
    - name: get current rdns entry
      uri:
        user: "  hetzner_ws_user  "
        password: "  hetzner_ws_password  "
        url: "https://robot-ws.your-server.de/rdns/  vmip4  "
        status_code: [200, 404]
      register: rdns_result
    - name: update incorrect rdns entry
      uri:
        user: "  hetzner_ws_user  "
        password: "  hetzner_ws_password  "
        url: "https://robot-ws.your-server.de/rdns/  vmip4  "
        method: "POST"
        body_format: form-urlencoded
        body:
          ptr: "  inventory_hostname  "
        status_code: [200, 201]
      when: '"rdns" not in rdns_result.json or inventory_hostname != rdns_result.json.rdns.ptr'
      changed_when: True
    delegate_to: localhost
The host groups this runs on are currently hardcoded as the VMs that live in the Hetzner Dedicated Server. A future iteration of this might use some inventory plugin to look up the subnets that are managed on Hetzner and create a group for all of those. Right now it won t be setting the reverse DNS for the router interface on that subnet, and won t automatically include new server s subnets. Gathering facts is disabled because all of these run locally. There is at least one VM running on this server that I can t log in to because I host it for someone else, so running locally is a necessity. The webservice credentials are stored in an Ansible Vault encrypted YAML file and loaded explicitly. An important note: the webservice username and password is not the same as your regular Robot username and password. You need to create a webservice user in Robot under Settings; Webservice and app settings . If you attempt to authenticate with an incorrect username and password 3 times in a row, your IP address will be blocked for 10 minutes. There are 6 hosts in this group, so I did this a few times before I realised there was a different user account I needed to create. I d suggest limiting to a single host while you re testing to get the authentication figured out. The actual calls to the webservice take place in a block just to avoid having to specify the delegate_to: localhost twice. The first step looks up the current PTR record, and accepts success if it gives either a 200 or 404 status code (it will be 404 if there is no existing pointer record). The result of this is used to conditionally create/update the PTR record in the next task. If nothing needs to be done, nothing will be changed and the second task will be skipped. If a change is needed, the second step is successful if either the PTR record is updated (status 200) or created (status 201). This was actually a lot easier than I thought it would be, and the uri module looks to be really flexible, so I bet there are other things that I could easily integrate into my Ansible playbooks.

1 May 2020

Paul Wise: FLOSS Activities April 2020

Changes

Issues

Review

Administration
  • myrepos: fix the forum
  • Debian: restart non-responsive tor daemon, restart processes due to OOM, apply debian.net changes for DD with expired key
  • Debian wiki: approve accounts
  • Debian QA services: deploy changes, auto-disable oldoldstable pockets

Communication

Sponsors The purple-discord work was sponsored by my employer. All other work was done on a volunteer basis.

Utkarsh Gupta: FOSS Activites in April 2020

Here s my (seventh) monthly update about the activities I ve done in the F/L/OSS world.

Debian
It s been 14 months since I ve started contributing to Debian. And 4 months since I ve been a Debian Developer. And in this beautiful time, I had this opprotunity to do and learn lots of new and interesting things. And most importantly, meet and interact with lots of lovely people!
Debian is $home.

Uploads:

Other $things:
  • Attended Ruby team meeting. Logs here.
  • Attended Perl team LHF. Report here.
  • Sponsored a lot of uploads for William Desportes and Adam Cecile.
  • Mentoring for newcomers.
  • FTP Trainee reviewing.
  • Moderation of -project mailing list.
  • Applied for DUCI project for Google Summer of Code 2020.

Ruby2.7 Migration:
Ruby2.7 was recently released on 25th December, 2019. Santa s gift. Believe it or not. We, the Debian Ruby team, have been trying hard to make it migrate to testing. And it finally happened. The default version in testing is ruby2.7. Here s the news! \o/
Here s what I worked on this month for this transition.

Upstream: Opened several issues and proposed patches (in the form of PRs):
  • Issue #35 against encryptor for Ruby2.7 test failures.
  • Issue #28 against image_science for removing relative paths.
  • Issue #106 against ffi-yajl for Ruby2.7 test failures.
  • PR #5 against aggregate for simply using require.
  • PR #6 against aggregate for modernizing CI and adding Ruby 2.5 and 2.7 support.
  • Issue #13 against espeak-ruby for Ruby2.7 test failures.
  • Issue #4 against tty-which for test failures in general.
  • Issue #11 against packable for Ruby2.7 test failures. PR #12 has been proposed.
  • Issue #10 against growl for test failures and proposed an initial patch.

Downstream: I fixed and uploaded the following packages in Debian:

Debian LTS
Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success.
This was my seventh month as a Debian LTS paid contributor. I was assigned 24.00 hours and worked on the following things:

CVE Fixes and Announcements:

Other LTS Work:

Other(s)
Sometimes it gets hard to categorize work/things into a particular category.
That s why I am writing all of those things inside this category.
This includes two sub-categories and they are as follows.

Personal: This month I could get the following things done:
  • Most importantly, I finally migrated to a new website. Huge UI imporvement! \o/
    From Jekyll to Hugo, it was not easy. But it was worth it! Many thanks to Luiz for writing hugo-coder, Clement, and Samyak!
    If you find any flaws, issues and pull requests are welcomed at utkarsh2102/utkarsh2102.com
  • Wrote battery-alert, a mini-project of my own to show battery alerts at <10% and >90%.
    Written in shell, it brings me all the satisfaction as it has saved my life on many occasions.
    And guess what? It has more users than just myself!
    Reviews and patches are welcomed \o/
  • Mentored in HackOn Hackathon. Thanks to Manvi for reaching out!
    It was fun to see people developing some really nice projects.
  • Thanks to Ray and John, I became a GitLab Hero!
    (I am yet to figure out my role and responibility though)
  • Atteneded Intro Sec Con and had the most fun!
    Heard Ian s keynote and attended other talks and learned how to use WireShark!

Open Source: Again, this contains all the things that I couldn t categorize earlier.
Opened several issues and pull requests:
  • Issue #297 against hugo-coder, asking to enable RSS feed for blogs.
  • PR #316 for hugo-coder for fixing the above issue myself.
  • Issue #173 against arbre for requesting a release.
  • Issue #104 against combustion, asking to relax dependency on rubocop. Fixed in this commit.
  • Issue #16 against ffi-compiler for requesting to fix homepage and license.
  • Issue #57 against gographviz for requesting a release.
  • Issue #14 against crb-blast, suggesting compatability with bio 2.0.x.
  • Issue #58 against uniform_notifier for asking to drop the use of ruby-growl.
  • PR #2072 for polybar, adding installation instructions on Debian systems.
Until next time.
:wq for today.

30 April 2020

Chris Lamb: Free software activities in April 2020

Here is my monthly update covering what I have been doing in the free software world during April 2020 (previous month's report). Looking it over prior to publishing, I am surprised how much I got done this month I felt that I was not only failing to do all the extra things I had planned, but I was doing far less than normal. But let us go easy on ourselves; nobody is nailing this. In addition, I did more hacking on the Lintian static analysis tool for Debian packages:
Reproducible builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. The initiative is proud to be a member project of the Software Freedom Conservancy, a not-for-profit 501(c)(3) charity focused on ethical technology and user freedom. Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter. Elsewhere in our tooling, I made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues, including preparing and uploading versions 139, 140, 141 and 142 to Debian: Lastly, I made a large number of changes to our website and documentation in the following categories:
Debian LTS This month I have contributed 18 hours to Debian Long Term Support (LTS) and 7 hours on its sister Extended LTS project. You can find out more about the project via the following video:
Debian I only filed three bugs in April, including one against snapshot.debian.org to report that a Content-Type HTTP header is missing when downloading .deb files (#956471) and to report build failures in the macs & ruby-enumerable-statistics packages:

29 April 2020

Ian Jackson: subdirmk 1.0 - ergonomic preprocessing assistant for non-recursive make

I have made the 1.0 release of subdirmk. subdirmk is a tool to help with writing build systems in make, without use of recursive make. Why Peter Miller's 1997 essay Recursive Make Considered Harmful persuasively argues that it is better to arrange to have a single make invocation with the project's complete dependency tree, rather than the conventional $(MAKE) -C subdirectory approach. This has become much more relevant with modern projects which tend to be large and have deep directory trees. Invoking make separately for each of these subdirectories can be very slow. Nowadays everyone needs to run a parallel build, but with the recursive make approach great discipline is needed to avoid introducing races which cause the build to sometimes fail. There are various new systems which aim to replace make. My general impression of these is that they mostly threw away the good parts of make (often, they discard the flexibility, and the use of the shell command as the basic unit of execution, making them hard to extend), or make other unfortunate assumptions. And there are a lot of programming-language-specific systems - a very unsatisfactory development. Having said all that, I admit I haven't properly evaluated every make competitor. Other reasons for staying with make including that it is widely available, relatively widely understood, and has a model relatively free of high-level abstract concepts. (I like my languages with high-level concepts, but not my build systems.) But, with make, I found that actually writing a project's build system in non-recursive make was not very ergonomic. So with some help and prompting from Mark Wooding, I have made a tool to help. What subdirmk is a makefile preprocessor and aggregator, typically run from autoconf. subdirmk provides convenience syntaxes for references to per-directory variables and pathnames. It also helps by providing a little syntactic sugar for GNU make's macro facilities, which are awkward to use in raw make. subdirmk's features are triggered by the sigil &. The syntax is carefully designed to avoid getting in the way of makefile programming (and programming of shell commands in make rules). subdirmk is fully documented in the README. There is a demo in the example directory (which also serves as part of the test suite). What's new The version number. I have not felt the need to make any changes since releasing 0.4 in mid-February. The last non-docs change was a (backwards-compatible) extension, in late January, to pass through unaltered GNU make's new grouped multiple targets syntax. Advantages and disadvantages of subdirmk Compared to recursive make, subdirmk is easier and simpler, although you do have to decorate a lot of your variables and filenames with & to indicate that they are directory-local. It is much easier to avoid writing parallel make bugs. You naturally get properly working per-subdirectory targets. subdirmk-based nonrecursive make is much, much faster than recursive make. Compared to many other recent build system tools, subdirmk retains all the flexibility and extensibility of make, and operates at a fairly low level of abstraction. subdirmk-based makefiles can easily invoke other build systems. make knows it's not the only thing in the universe. You can adopt subdirmk incrementally or partially, gradually bringing your recursive submakefiles into the unified build. The build system code in subdirmk's Dir.sd.mk files will be readily navigable by most readers; much will be familiar. Because subdirmk is a small collection of (fairly simple) scripting and makefile code, there is no need to build it; you can simply ship it with your project using git-subtree. For an autoconf-based project, there need be no change to how your users and downstreams invoke your build. On the other hand the price you (continue to) pay is make's punctation soup, which subdirmk adds a new sigil to. subdirmk-based makefiles are terse and help you use make's facilities to abstract away repetition, but that can make them dense. The new & sigil will faze some readers. Currently, the provided mechanism for incorporating subdirmk into your project assumes you are using autoconf but not automake. It would be possible to use subdirmk with autoconf-less projects, or with automake-based ones, but I haven't done the glue work to make that easy. subdirmk does require GNU make and it assumes you have perl installed. But GNU make is very portable, and perl is very widely available. (The perl used is very conservative.) The make competitors are, themselves, even less standard build tools. I don't think a build-dependency on GNU make, or perl, is a significant barrier nowadays, for most projects. Note about comment moderation I have deliberately been vague about other build systems and avoided specific criticisms or references. I don't want the comments to become a build system advocacy debate. Comments may be screened and moderated accordingly. Pointers to other obscure build system tools are very welcome. If you want to write a survey of build tools, or a critique of subdirmk, please do so on your own blog; I would be happy to consider linking to it.

comment count unavailable comments

28 April 2020

Antoine Beaupr : Drowned my camera: dealing with liquid spills in electronics

Folks who acutely dig into this website might know that I have been taking more pictures recently, as I got a new camera since January 2018: a beautiful Fujifilm X-T2 that I really like. Recently, I went out on a photo shoot in the rain. It was intermittent, light rain when I left so I figured the "weather proofing" (dpreview.com calls this "environmentally sealed") would keep the camera secure. After an hour of walking outside, however, rain intensified and I was just quickly becoming more and more soaked. Still trusting the camera would function, I carried on. But after about 90 minutes of dutiful work, the camera just turned off and wouldn't power back on. It had drowned. I couldn't believe it; "but this is supposed to be waterproof! This can't be happening!", I thought. I tried swapping out the battery for a fresh one, which was probably a bad idea (even if I was smart enough to do this under cover): still no luck, yet I could still not believe it was dead, so I figured I would look at it later when I was home. I still eventually removed the battery after a while, remembering that it mattered. Turns out the camera was really dead. Even at home, it wouldn't power up, even with fresh batteries. After closer inspection, the camera was as soaked as I was...
Two Sandisk memory cards with water droplets on them ...even the SD cards were wet!
I was filled with despair! My precious camera! I had been waiting for litterally decades to find the right digital camera that was as close to the good old film cameras I was used to. I was even working on black and white "film" to get back to basics, which turned into a project to witness the impact of the coronavirus on city life! All that was lost, or at least stopped: amazingly, the SD cards were just absolutely fine and survived the flooding without problem.
A one-way sign broken, fallen on the side in a gray cityscape The last photo my camera took before it died
A good photographer friend told me that this was actually fairly common: "if you shoot outside, get used to this, it will happen". So I tried "the rice trick": plunge your camera in a pile of rice and let it rest there for a long time. It didn't work so well: I didn't have a big enough container to hold the camera and the rice. I was also worried about rice particles inserting themselves into the camera holes, as I had opened all the ports to let it dry. I could also not keep myself from inserting a battery and trying it out again: amazingly, it powered up, only once, and died again. After shopping in desperation for dessicators (who would have thought you should keep those little bags from the stuff you order online!), I ended up buying silica gel dehumidifier from Lee Valley (13$, the small one!) which comes in a neat little metal box. But that never arrived in time so I had to find another solution. My partner threw the idea out in jest, but the actual solution worked, and it was surprisingly simple!
My camera and lens drying in a food dehydrator, at 30 C with 22 hours left Tada! Turns out you can dehydrate hardware too!
We have a food dehydrator at home (a Sedna Express if you really want to know) since we do a lot of backpacking and canot-camping, but I never thought I would put electronics in there. Turns out a food dehydrator is perfect: it has a per degree temperature control that can go very low and a timer. I set it to 30 C for 24 hours. (I originally set it to 40 C but it smelled like plastic after a while so my partner turned it off thinking it was melting the camera.) And now the camera is back! I was so happy! There is probably some permanent damage to the delicate circuitry in the camera. And I will probably not go back out in heavy rain again with the camera, or at least not without a rainjacket (35$USD at B&H) on the camera. And I am now in a position to tell other people what to do if they suffer the same fate...

Tips for dealing with electronic liquid damage So, lessons learned...
  1. when you have a liquid spill over your electronics: IMMEDIATELY REMOVE ALL ELECTRIC POWER, including the battery! (this is another reason why all batteries should be removable)
  2. if the spill is "sticky" (e.g. coffee, beer, maple syrup, etc) or "salty", do try to wash it with water, yet without flooding it any further (delicate balance, I know) some devices are especially well adapted to this: I have washed a keyboard with a shower head and drowned the thing completely, it worked fine after drying.
  3. do NOT power it back on until you are certain the equipment is dry
  4. let the electronics device dry for 24 to 48 hours with all ports open in a humidity-absorbing environment: a bag of rice works, but a food dehydrator is best. make sure the rice doesn't get stuck inside the machine: use a small mesh bag if necessary
  5. once you are confident the device has dried, fiddle with the controls and see if water comes out: it might not have dried because it was stuck inside a button or dial. if dry, try powering it back on and watch the symptoms. if it's still weird, try drying it for another day.
  6. if you get tired of waiting and the machine doesn't come back up, you will have to send it to the repair shop or open it up yourself to see if there is soldering damage you can fix.
I hope it might help careless people who dropped their coffee or ran out in the rain, believing the hype of waterproof cameras. Amateur tip: waterproof cameras are not waterproof...

Dirk Eddelbuettel: RcppArmadillo 0.9.870.2.0

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 705 other packages on CRAN. A new upstream release 9.870.2 of Armadillo was released a few days ago. We had tested two release candidates, and this caught one bug. The release was held up at CRAN for a few days as one package had an overly sensitive test depending on random input data; eventually we all convinced ourselves that there was no (Rcpp)Armadillo issue here. So morale: friends don t let friends have tests depend on random behavior. Changes in the new release are noted below.

Changes in RcppArmadillo version 0.9.870.2.0 (2020-04-24)
  • Upgraded to Armadillo release 9.870.2 (Roasted Mocha Retox)
    • faster handling of matrix multiplication expressions by diagvec() and diagmat()
    • added trimatu_ind() and trimatl_ind()
    • more consistent detection of sparse vector expressions

Courtesy of CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.