Search Results: "cts"

1 April 2020

Russ Allbery: Review: A Grand and Bold Thing

Review: A Grand and Bold Thing, by Ann Finkbeiner
Publisher: Free Press
Copyright: August 2010
ISBN: 1-4391-9647-8
Format: Kindle
Pages: 200
With the (somewhat excessively long) subtitle of An Extraordinary New Map of the Universe Ushering In a New Era of Discovery, this is a history of the Sloan Digital Sky Survey. It's structured as a mostly chronological history of the project with background profiles on key project members, particularly James Gunn. Those who follow my blog will know that I recently started a new job at Vera C. Rubin Observatory (formerly the Large Synoptic Survey Telescope). Our goal is to take a complete survey of the night sky several times a week for ten years. That project is the direct successor of the Sloan Digital Sky Survey, and its project team includes many people who formerly worked on Sloan. This book (and another one, Giant Telescopes) was recommended to me as a way to come up to speed on the history of this branch of astronomy. Before reading this book, I hadn't understood how deeply the ready availability of the Sloan sky survey data had changed astronomy. Prior to the availability of that survey data, astronomers would develop theories and then try to book telescope time to make observations to test those theories. That telescope time was precious and in high demand, so was not readily available, and was vulnerable to poor weather conditions (like overcast skies) once the allocated time finally arrived. The Sloan project changed all of that. Its output was a comprehensive sky survey available digitally whenever and wherever an astronomer needed it. One could develop a theory and then search the Sloan Digital Sky Survey for relevant data and, for at least some types of theories, test that theory against the data without needing precious telescope time or new observations. It was a transformational change in astronomy, made possible by the radical decision, early in the project, to release all of the data instead of keeping it private to a specific research project. The shape of that change is one takeaway from this book. The other is how many problems the project ran into trying to achieve that goal. About a third of the way into this book, I started wondering if the project was cursed. So many things went wrong, from institutional politics through equipment failures to software bugs and manufacturing problems with the telescope mirror. That makes it all the more impressive how much impact the project eventually had. It's also remarkable just how many bad things can happen to a telescope mirror without making the telescope unusable. Finkbeiner provides the most relevant astronomical background as she tells the story so that the unfamiliar reader can get an idea of what questions the Sloan survey originally set out to answer (particularly about quasars), but this is more of a project history than a popular astronomy book. There's enough astronomy here for context, but not enough to satisfy curiosity. If you're like me, expect to have your curiosity piqued, possibly resulting in buying popular surveys of current astronomy research. (At least one review is coming soon.) Obviously this book is of special interest to me because of my new field of work, my background at a research university, and because it features some of my co-workers. I'm not sure how interesting it will be to someone without that background and personal connection. But if you've ever been adjacent to or curious about how large-scale science projects are done, this is a fascinating story. Both the failures and problems and the way they were eventually solved is different than how the more common stories of successful or failed companies are told. (It helps, at least for me, that the shared goal was to do science, rather than to make money for a corporation whose fortunes are loosely connected to those of the people doing the work.) Recommended if this topic sounds at all interesting. Rating: 7 out of 10

Paul Wise: FLOSS Activities March 2020

Changes

Issues

Review

Administration
  • Debian wiki: approve accounts

Communication

Sponsors The dh-make-perl feature requests, file bug report, File::Libmagic changes, autoconf-archive change, libpst work and the purple-discord upload were sponsored by my employer. All other work was done on a volunteer basis.

31 March 2020

Pau Garcia i Quiles: Uyuni 2020.03 released with enhanced Debian support!

Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. Uyuni is a fork of Spacewalk that leverages Salt, Cobbler and containers to modernize it. Uyuni is the upstream for SUSE Manager (the main difference is support: with SUSE Manager you get it from SUSE; with Uyuni you get it from the community) and our development and feature discussion is done in the open. Last week we released Uyuni 2020.03, with much improved Debian support, coming from the community: we have got client tools (both the Salt stack and the traditional stack) for Debian 9 and 10, and bootstrapping support! In addition to that, Uyuni 2020.03 brings many other new features: While this version of Uyuni provides a much better experience for Debian sysadmins, we still have a lot of room for improvement: Do you want to help us with development, or just with feedback? Join our community on IRC, Gitter or the mailing lists. And check our user documentation, developer documentation and presentations. We are also participating in Google Summer of Code 2020. Hurry up and submit a proposal to provide Uyuni for Debian, and/or enhance Debian support!

Chris Lamb: Free software activities in March 2020

Here is my monthly update covering what I have been doing in the free software world during March 2020 (previous month): In addition, I did even more hacking on the Lintian static analysis tool for Debian packages, including:
Reproducible builds One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. The initiative is proud to be a member project of the Software Freedom Conservancy, a not-for-profit 501(c)(3) charity focused on ethical technology and user freedom. Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter. This month, I: In our tooling, I also made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues, including preparing and uploading version 138 to Debian: The Reproducible Builds project also operates a fully-featured and comprehensive Jenkins-based testing framework that powers tests.reproducible-builds.org. This month, I reworked the web-based package rescheduling tool to:
Debian LTS This month I have worked 18 hours on Debian Long Term Support (LTS) and 8 hours on its sister Extended LTS project. You can find out more about the Debian LTS project via the following video:
Debian Uploads For the Debian Privacy Maintainers team I requested that the pyptlib package be removed from the archive (#953429) as well as uploading onionbalance (0.1.8-6) to fix test failures under Pytest 3.x (#953535) and a new upstream release of nautilus-wipe. Finally, I sponsored an upload of bilibop (0.6.1) on behalf of Yann Amar.

Russ Allbery: pam-krb5 4.9

This is a security release fixing a one-byte buffer overflow when relaying prompts from the underlying Kerberos library. All users of my pam-krb5 module should upgrade as soon as possible. See the security advisory for more information. There are also a couple more minor security improvements in this release: The module now rejects passwords as long or longer than PAM_MAX_RESP_SIZE (normally 512 octets) since they can be a denial of service attack via the Kerberos string-to-key function, and uses explicit_bzero where available to clear passwords before releasing memory. Also in this release, use_pkinit is now supported with MIT Kerberos, the Kerberos prompter function returns more accurate error messages, I fixed an edge-case memory leak in pam_chauthtok, and the module/basic test will run properly with a system krb5.conf file that doesn't specify a realm. You can get the latest release from the pam-krb5 distribution page. I've also uploaded the new version to Debian unstable and patched security releases with only the security fix to Debian stable and oldstable.

30 March 2020

Mike Gabriel: UBports: Packaging of Lomiri Operating Environment for Debian (part 02)

Before and during FOSDEM 2020, I agreed with the people (developers, supporters, managers) of the UBports Foundation to package the Unity8 Operating Environment for Debian. Since 27th Feb 2020, Unity8 has now become Lomiri. Recent Uploads to Debian related to Lomiri Over the past 7-8 weeks the packaging progress has been slowed down due to other projects I am working on in parallel. However, quite a few things have been achieved: The packages qtsystems, qtfeedback, and qtpim are no official Qt5 components, and so I had to package Git snapshots of them; with all implicit consequences regarding ABI and API compatibilities, possibly Debian-internal library transitions, etc. Esp. packaging qtsystems was pretty tricky due to a number of failing unit tests when the package had been built in a clean chroot (like it is the case on Debian's buildd infrastructure). I learned a lot about DBus and DBus mocking while working on all those unit tests to finally pass in chrooted builds. Unfortunately, the Lomiri App Launch component still needs more work due to (finally only) one unit test (jobs-systemd) not always passing. Sometimes, the test gets stucks and then fails after having reached a time out. I'll add it to my list of those unreproducible build failures I have recently seen in several GTest related unit test scenarios. Sigh... Credits A great thanks goes to Lisandro Perez Meyer from the Debian KDE/Qt Team for providing an intro and help on Qt Debian packaging and an intro on symbols handling with C++ projects. Another big thanks goes to Dmitry Shachnev from the Debian KDE/Qt Team for doing a sponsored upload [1] of qtpim (and also a nice package review). Also a big thanks goes to Marius Gripsgard for his work on forking the first Lomiri components on the UBports upstream side. Previous Posts about my Debian UBports Team Efforts References

Mike Gabriel: Mailman3 - Call for Translations (@Weblate)

TL;DR; please help localizing Mailman3 [1]. You can find it on hosted Weblate [2].The next component releases are planned in 1-2 weeks from now. Thanks for your contribution! If you can't make it now, please consider working on Mailman3 translations at some later point of time. Thanks! Time has come for Mailman3 Over the last months I have found an interest in Mailman3. Given the EOL of Python2 in January 2020 and also being a heavy Mailman2 provider for various of my projects and also for customers, I felt it was time to look at Mailman2's successor: Mailman3 [1]. One great novelty in Mailman3 is the strict split up between backend (Mailman Core), and the frontend components (django-mailman3, Postorius, Hyperkitty). All three are Django applications. Postorius is the list management web frontend whereas Hyperkitty is an archive viewer. Other than in Mailman2, you can also drop list posts into Hyperkitty directly (instead of sending a mail to the list). This makes Hyperkitty also some sort of forum software with a mailing list core in the back. The django-mailman3 module knits the previous two together (and handles account management, login dialog, profile settings, etc.). Looking into Mailman3 Upstream Code Some time back in midst 2019 I decided to deploy Mailman3 at a customer's site and also for my own business (which still is the test installation). Living and working in Germany, my customers' demand often is a fully localized WebUI. And at that time, Mailman3 could not provide this. Many exposed parts of the Mailman3 components were still not localized (or not localizable). Together with my employee I put some hours of effort into providing merge requests, filing bug reports, request better Weblate integration (meaning: hosted Weblate), improving the membership scripting support, etc. It felt a bit like setting the whole i18n thing in motion. Call for Translations Over the past months I had to focus on other work and two days ago I was delighted that Abhilash Raj (one of the Mailman3 upstream maintainers) informed me (via closing one of the related bugs [3]) that Mailman3 is now fully integrated with the hosted Weblate service and a continous translation workflow is set to go. The current translation stati of the Mailman3 components are at ~ 10%. We can do better than this, I sense. So, if you are a non-native English speaker and feel like contributing to Mailman3, please visit the hosted Weblate site [2], sign up for an account (if you don't have one already), and chime in into the translation of one of the future mailing list software suites run by many FLOSS projects all around the globe. Thanks a lot for your help. As a side note, if you plan working on translating Mailman Core into your language (and can't find it in the list of supported languages), please request this new language via the Weblate UI. All other components have all available languages enabled by default. References:

Axel Beckert: How do you type on a keyboard with only 46 or even 28 keys?

Some of you might have noticed that I m into keyboards since a few years ago into mechanical keyboards to be precise. Preface It basically started with the Swiss Mechanical Keyboard Meetup (whose website I started later on) was held in the hackerspace of the CCCZH. I mostly used TKL keyboards (i.e. keyboards with just the for me useless number block missing) and tried to get my hands on more keyboards with Trackpoints (but failed so far). At some point a year or two ago, I looking into smaller keyboards for having a mechanical keyboard with me when travelling. I first bought a Vortex Core at Candykeys. The size was nice and especially having all layers labelled on the keys was helpful, but nevertheless I soon noticed that the smaller the keyboards get, the more important is, that they re properly programmable. The Vortex Core is programmable, but not the keys in the bottom right corner which are exactly the keys I wanted to change to get a cursor block down there. (Later I found out that there are possibilities to get this done, either with an alternative firmware and a hack of it or desoldering all switches and mounting an alternative PCB called Atom47.) 40% Keyboards So at some point I ordered a MiniVan keyboard from The Van Keyboards (MiniVan keyboards will soon be available again at The Key Dot Company), here shown with GMK Paperwork (also bought from and designed by The Van Keyboards):
The MiniVan PCBs are fully programmable with the free and open source firmware QMK and started to use that more and more instead of bigger keyboards. Layers With the MiniVan I learned the concepts of layers. Layers are similar to what many laptop keyboards do with the Fn key and to some extent also what the German standard layout does with the AltGr key: Layers are basically alternative key maps you can switch with a special key (often called Fn , Fn1 , Fn2 , etc., or especially if there are two additional layers Raise and Lower ). There are several concepts how these layers can be reached with these keys: My MiniVan Layout For the MiniVan, two additional layers suffice easily, but since I have a few characters on multiple layers and also have mouse control and media keys crammed in there, I have three additional layers on my MiniVan keyboards:

TRNS means transparent, i.e. use the settings from lower layers.
I also use a feature that allows me to mind different actions to a key depending if I just tap the key or if I hold it. Some also call this tap dance . This is especially very popular on the usually rather huge spacebar. There, the term SpaceFn has been coined, probably after this discussion on Geekhack. I use this for all my layer switching keys: With this layout I can type English texts as fast as I can type them on a standard or TKL layout. German umlauts are a bit more difficult because it requires 4 to 6 key presses per umlaut as I use the Compose key functionality (mapped to the Menu key between the spacebars and the cursor block. So to type an on my MiniVan, I have to:
  1. press and release Menu (i.e. Compose); then
  2. press and hold either Shift-Spacebar (i.e. Shift-Fn1) or Slash (i.e. Fn2), then
  3. press N for a double quote (i.e. Shift-Fn1-N or Fn2-N) and then release all keys, and finally
  4. press and release the base character for the umlaut, in this case Shift-A.
And now just use these concepts and reduce the amount of keys to 28: 30% and Sub-30% Keyboards In late 2019 I stumbled upon a nice little keyboard kit shop on Etsy which I (and probably most other people in the mechanical keyboard scene) didn t take into account for looking for keyboards called WorldspawnsKeebs. They offer mostly kits for keyboards of 40% size and below, most of them rather simple and not expensive. For about 30 you get a complete sub-30% keyboard kit (without switches and keycaps though, but that very common for keyboard kits as it leaves the choice of switches and key caps to you) named Alpha28 consisting of a minimal Acrylic case and a PCB and electronics set. This Alpha28 keyboard is btw. fully open source as the source code, (i.e. design files) for the hardware are published under a free license (MIT license) on GitHub. And here s how my Alpha28 looks like with GMK Mitolet (part of the GMK Pulse group-buy) key caps:
So we only have character keys, Enter (labelled Data as there was no 1u Enter key with that row profile in that key cap set; I ll also call it Data for the rest of this posting) and a small spacebar, not even modifier keys. The Default Alpha28 Layout The original key layout by the developer of the Alpha28 used the spacbar as Shift on hold and as space if just tapped, and the Data key switches always to the next layer, i.e. it switches the layer permanently on tap and not just on hold. This way that key rotates through all layers. In all other layers, V switches back to the default layer. I assume that the modifiers on the second layer are also on tap and apply to the next other normal key. This has the advantage that you don t have to bend your fingers for some key combos, but you have to remember on which layer you are at the moment. (IIRC QMK allows you to show that via LEDs or similar.) Kinda just like vi. My Alpha28 Layout But maybe because I m more an Emacs person, I dislike remembering states myself and don t bind bending my fingers. So I decided to develop my own layout using tap-or-hold and only doing layer switches by holding down keys:

A triangle means that the settings from lower layers are used, N/A means the key does nothing.
It might not be very obvious, but on the default layer, all keys in the bottom row and most keys on the row ends have tap-or-hold configurations. Basic ideasBottom row if holdOther rows if holdHow the keys are divided into layersUsing the Alpha28 This layout works surprisingly well for me. Only for Minus, Equal, Single Quote and Semicolon I still often have to think or try if they re on Layer 1 or 2 as on my 40%s (MiniVan, Zlant, etc.) I have them all on layer 1 (and in general one layer less over all). And for really seldom used keys like Insert, PrintScreen, ScrollLock or Pause, I might have to consult my own documentation. They re somewhere in the middle of the keyboard, either on layer 1, 2, or 3. ;-) And of course, typing umlauts takes even two keys more per umlaut as on the MiniVan since on the one hand Menu is not on the default layer and on the other hand, I don t have this nice shifted number row and actually have to also press Shift to get a double quote. So to type an on my Alpha, I have to:
  1. press and release Space-F (i.e. Fn1-F) for Menu (i.e. Compose); then
  2. press and hold A-Spacebar-L (i.e. Shift-Fn1-L) for getting a double quote, then
  3. press and release the base character for the umlaut, in this case L-A for Shift-A (because we can t use A for Shift as I can t hold a key and then press it again :-).
Conclusion If the characters on upper layers are not labelled like on the Vortex Core, i.e. especially on all self-made layouts, typing is a bit like playing that old children s game Memory: as soon as you remember (or your muscle memory knows) where some special characters are, typing gets faster. Otherwise, you start with trial and error or look the documentation. Or give up. ;-) Nevertheless, typing on a sub-30% keyboard like the Alpha28 is much more difficult and slower than on a 40% keyboard like the MiniVan. So the Alpha28 very likely won t become my daily driver while the MiniVan defacto is my already my daily driver. But I like these kind of challenges as others like the game Memory . So I ordered three more 30% and sub-30% keyboard kits and WorldspawnsKeebs for soldering on the upcoming weekend during the COVID19 lockdown: And if I at some point want to try to type with even fewer keys, I ll try a Butterstick keyboard with just 20 keys. It s a chorded keyboard where you have to press multiple keys at the same time to get one charcter: So to get an A from the missing middle row, you have to press Q and Z simultaneously, to get Escape, press Q and W simultaneously, to get Control, press Q, W, Z and X simultaneously, etc. And if that s not even enough, I already bought a keyboard kit named Ginny (or Ginni, the developer can t seem to decide) with just 10 keys from an acquaintance. Couldn t resist when offered his surplus kits. :-) It uses the ASETNIOP layout which was initially developed for on-screen keyboards on tablets.

29 March 2020

Enrico Zini: Politics links

How tech's richest plan to save themselves after the apocalypse
politics privilege archive.org
Silicon Valley s elite are hatching plans to escape disaster and when it comes, they ll leave the rest of us behind
Heteronomy refers to action that is influenced by a force outside the individual, in other words the state or condition of being ruled, governed, or under the sway of another, as in a military occupation.
Poster P590CW $9.00 Early Warning Signs Of Fascism Laurence W. Britt wrote about the common signs of fascism in April, 2003, after researching seven fascist regimes: Hitler's Nazi Germany; Mussolini's Italy; Franco's Spain; Salazar's Portugal; Papadopoulos' Greece; Pinochet's Chile; Suharto's Indonesia. Get involved! Text: Early Warning Signs of Fascism Powerful and Continuing Nationalism Disdain For Human Rights Identification of Enemies As a unifying cause Supremacy of the military Rampant Sexism Controlled Mass Media Obsession With National Security
Political and social scientist Stefania Milan writes about social movements, mobilization and organized collective action. On the one hand, interactions and networks achieve more visibility and become a proxy for a collective we . On the other hand: Law enforcement can exercise preemptive monitorin
How new technologies and techniques pioneered by dictators will shape the 2020 election
A regional election offers lessons on combatting the rise of the far right, both across the Continent and in the United States.
The Italian diaspora is the large-scale emigration of Italians from Italy. There are two major Italian diasporas in Italian history. The first diaspora began more or less around 1880, a decade or so after the Unification of Italy (with most leaving after 1880), and ended in the 1920s to early-1940s with the rise of Fascism in Italy. The second diaspora started after the end of World War II and roughly concluded in the 1970s. These together constituted the largest voluntary emigration period in documented history. Between 1880-1980, about 15,000,000 Italians left the country permanently. By 1980, it was estimated that about 25,000,000 Italians were residing outside Italy. A third wave is being reported in present times, due to the socio-economic problems caused by the financial crisis of the early twenty-first century, especially amongst the youth. According to the Public Register of Italian Residents Abroad (AIRE), figures of Italians abroad rose from 3,106,251 in 2006 to 4,636,647 in 2015, growing by 49.3% in just ten years.

Sven Hoexter: Looking into Envertech Enverbridge EVB 202 SetID tool

Disclaimer: I'm neither an experienced programmer nor proficient in reverse engineering, but I like at least to try to figure out how things work. Sometimes the solution is so easy, that even I manage to find it, still take this with a grain of salt. I lately witnessed the setup of an Envertech EnverBridge ENB-202 which is kind of a classic Chinese IoT device. Buy it, plug it in, use some strange setup software, and it will report your PV statistics to a web portal. The setup involved downloading a PE32 Windows executable, with an UI that basically has two input boxes and a sent button. You've to input the serial number(s) of your inverter boxes and the ID of your EnverBridge. That made me interested in what this setup process really looks like. The EnverBridge device itself has on one end a power plug, which is also used to communicate with the inverter via some Powerline protocol, and a network plug with a classic RJ45 end you plug into your network. If you power it up it will request an IPv4 address via DHCP. That brings us to the first oddity, the MAC address is in the BC:20:90 prefix which I could not find in the IEEE lists. Setting Up the SetID Software You can download the Windows software in a Zipfile, once you unpack it you end up with a Nullsoft installer .exe. Since this is a PE32 executable we've to add i386 as foreign architecture to install the wine32 package.
dpkg --add-architecture i386
apt update
apt install wine32:i386
wget http://www.envertec.com/uploads/bigfiles/Set%20ID.zip
unzip Set\ ID.zip
wine Set\ ID.exe
The end result is an installation in ~/.wine/drive_c/Program Files/SetID which reveals that this software is build with Qt5 according to the shipped dll's. The tool itself is in udpCilentNow.exe, and looks like this: Envertech SetID software The Network Communication To my own surprise, the communication is straight forward. A single UDP paket is sent to the broadcast address (255.255.255.255) on port 8765. Envertech SetID udp paket in wireshark I expected some strange binary protocol, but the payload is just simple numbers. They're a combination of the serial numbers of the inverter and the ID of the Enverbridge device. One thing I'm not 100% sure about are the inverter serial numbers, there are two of them, but on the inverter I've seen the serial numbers are always the same. So the payload is assembled like this: If you've more inverter the serials are just appended in the same way. Another strange thing is that the software does close to no input validation, they only check that the inverter serials start with CN and then just extract char 9 to 16. The response from the EnverBridge is also a single UDP paket to the broadcast address on port 8764, with exactly the same content we've sent. Writing a Replacement My result is probably an insult for all proficient Python coders, but based on a bit of research and some cut&paste programming I could assemble a small script to replicate the function of the Windows binary. I guess the usefulness of this exercise was mostly my personal entertainment, though it might help some none Windows users to setup this device. Usage is also very simple:
./enverbridge.py -h
Usage: enverbridge.py [options] MIIDs
Options:
  -h, --help         show this help message and exit
  -b BID, --bid=BID  Serial Number of your EnverBridge
./enverbridge.py -b 90087654 CN19100912345678 CN19100912345679
This is basically 1:1 the behaviour of the Windows binary, though I tried to add a bit more validation then the original binary and some more error messages. I also assume the serial numbers are also always the same so I take only one as the input, and duplicate it for the paket data.

Paulo Henrique de Lima Santana: My free software activities in February 2020

My free software activities in february 2020 March is ending but I finally wrote my monthly report about activities in Debian and Free Software in general for February. As I already wrote here, I attended to FOSDEM 2020 on February 1st and 2nd in Brussels. It was a amazing experience. After my return to Curitiba, I felt my energies renewed to start new challenges.

MiniDebConf Macei 2020 I continued helping to organize MiniDebConf and I got positive answers from 4Linux and Globo.com and they are sponsorsing the event.

FLISOL 2020 I started to talk with Maristela from IEP - Instituto de Engenharia do Paran and after some messages and I joined a meeting with her and other members of C mara T cnica de Eletr nica, Computa o e Ci ncias de Dados. I explained about FLISOL in Curitiba to them and they agreed to host the event at IEP. I asked to use three spaces: Auditorium for FLISOL talks, Sal o Nobre for meetups from WordPress and PostgreSQL Communities, and the hall for Install Fest. Besides FLISOL, they would like to host other events and meetups from Communities in Curitiba as Python, PHP, and so on. At least one per month. I helped to schedule a PHP Paran Community meetup on March.

New job Since 17th I started to work at Rentcars as Infrastructure Analyst. I m very happy to work there because we use a lot of FLOSS and with nice people. Ubuntu LTS is the approved OS for desktops but I could install Debian on my laptop :-)

Misc I signed pgp keys from friends I met in Brussels and I had my pgp key signed by them. Finally my MR to the DebConf20 website fixing some texts was accepted. I have watched v deos from FOSDEM
  1. Until now, I saw these great talks:
  • Growing Sustainable Contributions Through Ambassador Networks
  • Building Ethical Software Under Capitalism
  • Cognitive biases, blindspots and inclusion
  • Building a thriving community in company-led open source projects
  • Building Community for your Company s OSS Projects
  • The Ethics of Open Source
  • Be The Leader You Need in Open Source
  • The next generation of contributors is not on IRC
  • Open Source Won, but Software Freedom Hasn t Yet
  • Open Source Under Attack
  • Lessons Learned from Cultivating Open Source Projects and Communities
That s all folks!

28 March 2020

Jonathan Dowland: Nintendo Switch - Virtua Racing

Screenshot of Virtua Racing
I don't play many video games any more, and I certainly don't consider myself a gamer. If I have time and brain power, I like to write code or work on my PhD; if I don't have brain power I read, watch movies or TV. However last Summer, lured by the launch titles Super Mario Odyssey and The Legend of Zelda: Breath of the Wild, I bought a Nintendo Switch. I've actually played some games on it now, although I surprised myself by playing lesser-known and indie titles rather than the AAA names. My favourite games are ones I can pick up and play for 5-10 minutes at a time without losing time to tutorials, introductory videos, and other such nonsense. I thought I'd recommend some games. First up, Virtua Racing. I actually bought this for the office Switch before I had my own. It's re-release of the 1992 arcade racer. I have fond memories of Arcades in the 90s (although I don't think I played this game). It's conceptually simple: three tracks, few options (you can choose manual or automatic transmission). But there's some hidden depth in there: it models slipstream, tyre wear and other effects. The fun of it for me is trying to beat my personal best and that of my colleagues. But I haven't actually placed above 3rd on the intermediate course or even finished the advanced one, so despite its simplicity, there's plenty of mileage in it for me. It also looks great, rending at a smooth 60fps. The large untextured polygons were once a design decision due to the limitations of the hardware in 1992 and are now an in-fashion aesthetic.

Dirk Eddelbuettel: RProtoBuf 0.4.17: Robustified

A new release 0.4.17 of RProtoBuf is now on CRAN. RProtoBuf provides R with bindings for the Google Protocol Buffers ( ProtoBuf ) data encoding and serialization library used and released by Google, and deployed very widely in numerous projects as a language and operating-system agnostic protocol. This release contains small polishes related to the release 0.4.16 which added JSON support for messages, and switched to ByteSizeLong. This release now makes sure JSON functionality is only tested where available (on version 3 of the Protocol Buffers library), and that ByteSizeLong is only called where available (version 3.6.0 or later). Of course, older versions build as before and remain fully supported.

Changes in RProtoBuf version 0.4.17 (2020-03-xx)
  • Condition use of ByteSizeLong() on building with ProtoBuf 3.6.0 or later (Dirk in #71 fixing #70).
  • The JSON unit tests are skipped if ProtoBuf 2.* is used (Dirk, also #71).
  • The configure script now extracts the version from the DESCRIPTION file ( (Dirk, also #71).

CRANberries provides the usual diff to the previous release. The RProtoBuf page has copies of the (older) package vignette, the quick overview vignette, and the pre-print of our JSS paper. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

24 March 2020

Dirk Eddelbuettel: New package RcppDate 0.0.1 now on CRAN!

A new small package with a new C++ header library is now on CRAN. It brings the date library by Howard Hinnant to R. This library has been in pretty wide-spread use for a while now, and adds to C++11/C++14/C++17 what will be (with minor modifications) the date library in C++20. I had been aware of it for a while, but not needed thanks to CCTZ library out of Google and our RcppCCTZ package. And like CCTZ, it builds upon std::chron adding a whole lot of functionality and useability enhancement. But a some upcoming (and quite exciting!) changes in nanotime required it, I had a reason to set about packaging it as RcppDate. And after a few days of gestation and review it is now available via CRAN. Two simple example files are included and can be accessed by Rcpp::sourceCpp(). Some brief excerpts follow. The first example shows three date constructors. Note how the month (and the leading digits) are literals. No quotes for strings anywhere. And no format (just like our anytime package for R).
  constexpr auto x1 = 2015_y/March/22;
  constexpr auto x2 = March/22/2015;
  constexpr auto x3 = 22_d/March/2015;
Note that these are constexpr that resolve at compile-time, and that the resulting year_month_day type is inferred via auto. A second example constructs the last day of the months similarly:
  constexpr auto x1 = 2015_y/February/last;
  constexpr auto x2 = February/last/2015;
  constexpr auto x3 = last/February/2015;
For more, see the copious date.h documentation. The (very bland first) NEWS entry (from a since-added NEWS file) for the initial upload follows.

Changes in version 0.0.1 (2020-01-17)
  • Initial CRAN upload of first version

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Russ Allbery: Review: Lost in Math

Review: Lost in Math, by Sabine Hossenfelder
Publisher: Basic
Copyright: June 2018
ISBN: 0-465-09426-0
Format: Kindle
Pages: 248
Listening to experts argue can be one of the better ways to learn about a new field. It does require some basic orientation and grounding or can be confusing or, worse, wildly misleading, so some advance research or Internet searches are warranted. But it provides some interesting advantages over reading multiple popular introductions to a field. First, experts arguing with each other are more precise about their points of agreement and disagreement because they're trying to persuade someone who is well-informed. The points of agreement are often more informative than the points of disagreement, since they can provide a feel for what is uncontroversial among experts in the field. Second, internal arguments tend to be less starry-eyed. One of the purposes of popularizations of a field is to get the reader excited about it, and that can be fun to read. But to generate that excitement, the author has a tendency to smooth over disagreements and play up exciting but unproven ideas. Expert disagreements pull the cover off of the uncertainty and highlight the boundaries of what we know and how we know it. Lost in Math (subtitled How Beauty Leads Physics Astray) is not quite an argument between experts. That's hard to find in book form; most of the arguments in the scientific world happen in academic papers, and I rarely have the energy or attention span to read those. But it comes close. Hossenfelder is questioning the foundations of modern particle physics for the general public, but also for her fellow scientists. High-energy particle physics is facing a tricky challenge. We have a solid theory (the standard model) which explains nearly everything that we have currently observed. The remaining gaps are primarily at very large scales (dark matter and dark energy) or near phenomena that are extremely difficult to study (black holes). For everything else, the standard model predicts our subatomic world to an exceptionally high degree of accuracy. But physicists don't like the theory. The details of why are much of the topic of this book, but the short version is that the theory does not seem either elegant or beautiful. It relies on a large number of measured constants that seem to have no underlying explanation, which is contrary to a core aesthetic principle that physicists use to judge new theories. Accompanying this problem is another: New experiments in particle physics that may be able to confirm or disprove alternate theories that go beyond the standard model are exceptionally expensive. All of the easy experiments have been done. Building equipment that can probe beyond the standard model is incredibly expensive, and thus only a few of those experiments have been done. This leads to two issues: Particle physics has an overgrowth of theories (such as string theory) that are largely untethered from experiments and are not being tested and validated or disproved, and spending on new experiments is guided primarily by a sense of scientific aesthetics that may simply be incorrect. Enter Lost in Math. Hossenfelder's book picks up a thread of skepticism about string theory (and, in Hossenfelder's case, supersymmetry as well) that I previously read in Lee Smolin's The Trouble with Physics. But while Smolin's critique was primarily within the standard aesthetic and epistemological framework of particle physics, Hossenfelder is questioning that framework directly. Why should nature be beautiful? Why should constants be small? What if the universe does have a large number of free constants? And is the dislike of an extremely reliable theory on aesthetic grounds a good basis for guiding which experiments we fund?
Do you recall the temple of science, in which the foundations of physics are the bottommost level, and we try to break through to deeper understanding? As I've come to the end of my travels, I worry that the cracks we're seeing in the floor aren't really cracks at all but merely intricate patterns. We're digging in the wrong places.
Lost in Math will teach you a bit less about physics than Smolin's book, although there is some of that here. Smolin's book was about two-thirds physics and one-third sociology of science. Lost in Math is about two-thirds sociology and one-third physics. But that sociology is engrossing. It's obvious in retrospect, but I hadn't thought before about the practical effects of running out of unexplained data on a theoretical field, or about the transition from more data than we can explain to having to spend billions of dollars to acquire new data. And Hossenfelder takes direct aim at the human tendency to find aesthetically appealing patterns and unified explanations, and scores some palpable hits.
I went into physics because I don't understand human behavior. I went into physics because math tells it how it is. I liked the cleanliness, the unambiguous machinery, the command math has over nature. Two decades later, what prevents me from understanding physics is that I still don't understand human behavior. "We cannot give exact mathematical rules that define if a theory is attractive or not," says Gian Francesco Giudice. "However, it is surprising how the beauty and elegance of a theory are universally recognized by people from different cultures. When I tell you, 'Look, I have a new paper and my theory is beautiful,' I don't have to tell you the details of my theory; you will get why I'm excited. Right?" I don't get it. That's why I am talking to him. Why should the laws of nature care what I find beautiful? Such a connection between me and the universe seems very mystical, very romantic, very not me. But then Gian doesn't think that nature cares what I find beautiful, but what he finds beautiful.
The structure of this book is half tour of how physics judges which theories are worthy of investigation and half personal quest to decide whether physics has lost contact with reality. Hossenfelder approaches this second thread with multiple interviews of famous scientists in the field. She probes at their bases for preferring one theory over another, at how objective those preferences can or should be, and what it means for physics if they're wrong (as increasingly appears to be the case for supersymmetry). In so doing, she humanizes theory development in a way that I found fascinating. The drawback to reading about ongoing arguments is the lack of a conclusion. Lost in Math, unsurprisingly, does not provide an epiphany about the future direction of high-energy particle physics. Its conclusion, to the extent that it has one, is a plea to find a way to put particle physics back on firmer experimental footing and to avoid cognitive biases in theory development. Given the cost of experiments and the nature of humans, this is challenging. But I enjoyed reading this questioning, contrarian take, and I think it's valuable for understanding the limits, biases, and distortions at the edge of new theory development. Rating: 7 out of 10

23 March 2020

Dima Kogan: org-babel for documentation

So I just gave a talk at SCaLE 18x about numpysane and gnuplotlib, two libraries I wrote to make using numpy bearable. With these two, it's actually quite nice! Prior to the talk I overhauled the documentation for both these projects. The gnuplotlib docs now have a tutorial/gallery page, which is interesting-enough to write about. Check it out! Mostly it is a sequence of Clearly you want the plots in the documentation to correspond to the code, so you want something to actually run each code snippet to produce each plot. Automatically. I don't want to maintain these manually, and periodically discover that the code doesn't make the plot I claim it does or worse: that the code barfs. This is vaguely what Jupyter notebooks do, but they're ridiculous, so I'm doing something better: That's it. The git repo is hosted by github, which has a rudimentary renderer for .org documents. I'm committing the .svg files, so that's enough to get rendered documentation that looks nice. Note that the usual workflow is to use org to export to html, but here I'm outsourcing that job to github; I just make the .svg files, and that's enough. Look at the link again: gnuplotlib tutorial/gallery. This is just a .org file committed to the git repo. github is doing its normal org->html thing to display this file. This has drawbacks too: github is ignoring the :noexport: tag on the init section at the end of the file, so it's actually showing all the emacs lisp goop that makes this work (described below!). It's at the end, so I guess this is good-enough. Those of us that use org-mode would be completely unsurprised to hear that the talk is also written as .org document. And the slides that show gnuplotlib plots use the same org-babel system to render the plots. It's all oh-so-nice. As with anything as flexible as org-babel, it's easy to get into a situation where you're bending it to serve a not-quite-intended purpose. But since this all lives in emacs, you can make it do whatever you want with a bit of emacs lisp. I ended up advising a few things (mailing list post here). And I stumbled on an (arguable) bug in emacs that needed working around (mailing list post here). I'll summarize both here.

Handling large Local Variables blocks
The advises I ended up with ended up longer than emacs expected, which made emacs not evaluate them when loading the buffer. As I discovered (see the mailing list post) the loading code looks for the string Local Variables in the last 3000 bytes of the buffer only, and I exceeded that. Stefan Monnier suggested a workaround in this post. Instead of the normal Local Variables block at the end:
Local Variables:
eval: (progn ... ...
             ... ...
             LONG chunk of emacs-lisp
      )
End:
I do this:
(progn ;;local-config
   lisp lisp lisp
   as long as I want
)
Local Variables:
eval: (progn (re-search-backward "^(progn ;;local-config") (eval (read (current-buffer))))
End:
So emacs sees a small chunk of code that searches backwards through the buffer (as far back as needed) for the real lisp to evaluate. As an aside, this blog is also an .org document, and the lisp snippets above are org-babel blocks that I'm not evaluating. The exporter knows to respect the emacs-lisp syntax highlighting, however.

Advises
OK, so what was all the stuff I needed to tell org-babel to do specially here? First off, org needed to be able to communicate to the Python session the name of the file to write the plot to. I do this by making the whole plist for this org-babel snippet available to python:
;; THIS advice makes all the org-babel parameters available to python in the
;; _org_babel_params dict. I care about _org_babel_params['_file'] specifically,
;; but everything is available
(defun dima-org-babel-python-var-to-python (var)
  "Convert an elisp value to a python variable.
  Like the original, but supports (a . b) cells and symbols
"
  (if (listp var)
      (if (listp (cdr var))
          (concat "[" (mapconcat #'org-babel-python-var-to-python var ", ") "]")
        (format "\"\"\"%s\"\"\"" var))
    (if (symbolp var)
        (format "\"\"\"%s\"\"\"" var)
      (if (eq var 'hline)
          org-babel-python-hline-to
        (format
         (if (and (stringp var) (string-match "[\n\r]" var)) "\"\"%S\"\"" "%S")
         (if (stringp var) (substring-no-properties var) var))))))
(defun dima-alist-to-python-dict (alist)
  "Generates a string defining a python dict from the given alist"
  (let ((keyvalue-list
         (mapcar (lambda (x)
                   (format "%s = %s, "
                           (replace-regexp-in-string
                            "[^a-zA-Z0-9_]" "_"
                            (symbol-name (car x)))
                           (dima-org-babel-python-var-to-python (cdr x))))
                 alist)))
    (concat
     "dict( "
     (apply 'concat keyvalue-list)
     ")")))
(defun dima-org-babel-python-pass-all-params (f params)
  (cons
   (concat
    "_org_babel_params = "
    (dima-alist-to-python-dict params))
   (funcall f params)))
(unless
    (advice-member-p
     #'dima-org-babel-python-pass-all-params
     #'org-babel-variable-assignments:python)
  (advice-add
   #'org-babel-variable-assignments:python
   :around #'dima-org-babel-python-pass-all-params))
So if there's a :file plist key, the python code can grab that, and write the plot to that filename. But I don't really want to specify an output file for every single org-babel snippet. All I really care about is that each plot gets a unique filename. So I omit the :file key entirely, and use this advice to generate one for me:
;; This sets a default :file tag, set to a unique filename. I want each demo to
;; produce an image, but I don't care what it is called. I omit the :file tag
;; completely, and this advice takes care of it
(defun dima-org-babel-python-unique-plot-filename
    (f &optional arg info params)
  (funcall f arg info
           (cons (cons ':file
                       (format "guide-%d.svg"
                               (condition-case nil
                                   (setq dima-unique-plot-number (1+ dima-unique-plot-number))
                                 (error (setq dima-unique-plot-number 0)))))
                 params)))
(unless
    (advice-member-p
     #'dima-org-babel-python-unique-plot-filename
     #'org-babel-execute-src-block)
  (advice-add
   #'org-babel-execute-src-block
   :around #'dima-org-babel-python-unique-plot-filename))
This uses the dima-unique-plot-number integer to keep track of each plot. I increment this with each plot. Getting closer. It isn't strictly required, but it'd be nice if each plot had the same output filename each time I generated it. So I want to reset the plot number to 0 each time:
;; If I'm regenerating ALL the plots, I start counting the plots from 0
(defun dima-reset-unique-plot-number
    (&rest args)
    (setq dima-unique-plot-number 0))
(unless
    (advice-member-p
     #'dima-reset-unique-plot-number
     #'org-babel-execute-buffer)
  (advice-add
   #'org-babel-execute-buffer
   :after #'dima-reset-unique-plot-number))
Finally, I want to lie to the user a little bit. The code I'm actually executing writes each plot to an .svg. But the code I'd like the user to see should use the default output: an interactive, graphical window. I do that by tweaking the python session to tell the gnuplotlib object to write to .svg files from org by default, instead of using the graphical terminal:
;; I'm using github to display guide.org, so I'm not using the "normal" org
;; exporter. I want the demo text to not contain the hardcopy= tags, but clearly
;; I need the hardcopy tag when generating the plots. I add some python to
;; override gnuplotlib.plot() to add the hardcopy tag somewhere where the reader
;; won't see it. But where to put this python override code? If I put it into an
;; org-babel block, it will be rendered, and the :export tags will be ignored,
;; since github doesn't respect those (probably). So I put the extra stuff into
;; an advice. Whew.
(defun dima-org-babel-python-set-demo-output (f body params)
  (with-temp-buffer
    (insert body)
    (beginning-of-buffer)
    (when (search-forward "import gnuplotlib as gp" nil t)
      (end-of-line)
      (insert
       "\n"
       "if not hasattr(gp.gnuplotlib, 'orig_init'):\n"
       "    gp.gnuplotlib.orig_init = gp.gnuplotlib.__init__\n"
       "gp.gnuplotlib.__init__ = lambda self, *args, **kwargs: gp.gnuplotlib.orig_init(self, *args, hardcopy=_org_babel_params['_file'] if 'file' in _org_babel_params['_result_params'] else None, **kwargs)\n"))
    (setq body (buffer-substring-no-properties (point-min) (point-max))))
  (funcall f body params))
(unless
    (advice-member-p
     #'dima-org-babel-python-set-demo-output
     #'org-babel-execute:python)
  (advice-add
   #'org-babel-execute:python
   :around #'dima-org-babel-python-set-demo-output))
)
And that's it. The advises in the talk are slightly different, in uninteresting ways. Some of this should be upstreamed to org-babel somehow. Now entirely clear which part, but I'll cross that bridge when I get to it.

19 March 2020

Dirk Eddelbuettel: RProtoBuf 0.4.16: Now with JSON

A new release 0.4.16 of RProtoBuf is now on CRAN. RProtoBuf provides R with bindings for the Google Protocol Buffers ( ProtoBuf ) data encoding and serialization library used and released by Google, and deployed very widely in numerous projects as a language and operating-system agnostic protocol. This release contains a PR contributed by Siddhartha Bagaria which adds JSON support for messages, which had been an open wishlist item. I also appeased a clang deprecation warning that had come up on one of the CRAN test machines.

Changes in RProtoBuf version 0.4.16 (2020-03-19)
  • Added support for parsing and printing JSON (Siddhartha Bagaria in #68 closing wishlist #61).
  • Switched ByteSize() to ByteSizeLong() to appease clang (Dirk).

CRANberries provides the usual diff to the previous release. The RProtoBuf page has copies of the (older) package vignette, the quick overview vignette, and the pre-print of our JSS paper. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

John Goerzen: COVID-19 is serious for all ages. Treat it like WWII

Today I d like to post a few updates about COVID-19 which I have gathered from credible sources, as well as some advice also gathered from credible sources. Summary
  1. Coronavirus causes health impacts requiring hospitalization in a significant percentage of all adult age groups.
  2. Coronavirus also can cause no symptoms at all in many, especially children.
  3. Be serious about social distancing.
COVID-19 is serious for young adults too According to this report based on a CDC analysis, between 14% and 20% of people aged 20 to 44 require hospitalization due to COVID-19. That s enough to be taken seriously. See also this CNN story. Act as if you are a carrier because you may be infected and not even know it, even children Information on this is somewhat preliminary, but it is certainly known that a certain set of cases is asymptomatic. This article discusses manifestations in children, while this summary of a summary (note: not original research) suggests that 17.9% of people may not even know they are infected. How serious is this? Serious. This excellent article by Daniel W. Johnson, MD, is a very good read. Among the points it makes: Advice I m going to just copy Dr. Johnson s advice here:
  1. You and your kids should stay home. This includes not going to church, not going to the gym, not going anywhere.
  2. Do not travel for enjoyment until this is done. Do not travel for work unless your work truly requires it.
  3. Avoid groups of people. Not just crowds, groups. Just be around your immediate family. I think kids should just play with siblings at this point no play dates, etc.
  4. When you must leave your home (to get groceries, to go to work), maintain a distance of six feet from people. REALLY stay away from people with a cough or who look sick.
  5. When you do get groceries, etc., buy twice as much as you normally do so that you can go to the store half as often. Use hand sanitizer immediately after your transaction, and immediately after you unload the groceries.
I m not saying people should not go to work. Just don t leave the house for anything unnecessary, and if you can work from home, do it. Everyone on this email, besides Mom and Dad, are at low risk for severe disease if/when they contract COVID-19. While this is great, that is not the main point. When young, well people fail to do social distancing and hygiene, they pick up the virus and transmit it to older people who are at higher risk for critical illness or death. So everyone needs to stay home. Even young people. Tell every person over 60, and every person with significant medical conditions, to avoid being around people. Please do not have your kids visit their grandparents if you can avoid it. FaceTime them. Our nation is the strongest one in the world. We have been through other extreme challenges and succeeded many times before. We WILL return to normal life. Please take these measures now to flatten the curve, so that we can avoid catastrophe.
I d also add that many supermarkets offer delivery or pickup options that allow you to get your groceries without entering the store. Some are also offering to let older people shop an hour before the store opens to the general public. These could help you minimize your exposure. Other helpful links Here is a Reddit megathread with state-specific unemployment resources. Scammers are already trying to prey on people. Here are some important tips to avoid being a victim. Although there are varying opinions, some are recommending avoiding ibuprofen when treating COVID-19. Bill Gates had some useful advice. Here s a summary emphasizing the need for good testing.

18 March 2020

Antoine Beaupr : How can I trust this git repository?

Join me in the rabbit hole of git repository verification, and how we could improve it.

Problem statement As part of my work on automating install procedures at Tor, I ended up doing things like:
git clone REPO
./REPO/bootstrap.sh
... something eerily similar to the infamous curl pipe bash method which I often decry. As a short-term workaround, I relied on the SHA-1 checksum of the repository to make sure I have the right code, by running this both on a "trusted" (ie. "local") repository and the remote, then visually comparing the output:
$ git show-ref master
9f9a9d70dd1f1e84dec69a12ebc536c1f05aed1c refs/heads/master
One problem with this approach is that SHA-1 is now considered as flawed as MD5 so it can't be used as an authentication mechanism anymore. It's also fundamentally difficult to compare hashes for humans. The other flaw with comparing local and remote checksums is that we assume we trust the local repository. But how can I trust that repository? I can either:
  1. audit all the code present and all the changes done to it after
  2. or trust someone else to do so
The first option here is not practical in most cases. In this specific use case, I have audited the source code -- I'm the author, even -- what I need is to transfer that code over to another server. (Note that I am replacing those procedures with Fabric, which makes this use case moot for now as the trust path narrows to "trust the SSH server" which I already had anyways. But it's still important for my fellow Tor developers who worry about trusting the git server, especially now that we're moving to GitLab.) But anyways, in most cases, I do need to trust some other fellow developer I collaborate with. To do this, I would need to trust the entire chain between me and them:
  1. the git client
  2. the operating system
  3. the hardware
  4. the network (HTTPS and the CA cartel, specifically)
  5. then the hosting provider (and that hardware/software stack)
  6. and then backwards all the way back to that other person's computer
I want to shorten that chain as much as possible, make it "peer to peer", so to speak. Concretely, it would eliminate the hosting provider and the network, as attackers.

OpenPGP verification My first reaction is (perhaps perversely) to "use OpenPGP" for this. I figured that if I sign every commit, then I can just check the latest commit and see if the signature is good. The first problem here is that this is surprisingly hard. Let's pick some arbitrary commit I did recently:
commit b3c538898b0ed4e31da27fc9ca22cb55e1de0000
Author: Antoine Beaupr  <anarcat@debian.org>
Date:   Mon Mar 16 14:37:28 2020 -0400
    fix test autoloading
    pytest only looks for file names matching  test  by default. We inline
    tests inside the source code directly, so hijack that.
diff --git a/fabric_tpa/pytest.ini b/fabric_tpa/pytest.ini
new file mode 100644
index 0000000..71004ea
--- /dev/null
+++ b/fabric_tpa/pytest.ini
@@ -0,0 +1,3 @@
+[pytest]
+# we inline tests directly in the source code
+python_files = *.py
That's the output of git log -p in my local repository. I signed that commit, yet git log is not telling me anything special. To check the signature, I need something special: --show-signature, which looks like this:
commit b3c538898b0ed4e31da27fc9ca22cb55e1de0000
gpg: Signature faite le lun 16 mar 2020 14:37:53 EDT
gpg:                avec la clef RSA 7B164204D096723B019635AB3EA1DDDDB261D97B
gpg: Bonne signature de  Antoine Beaupr  <anarcat@orangeseeds.org>  [ultime]
gpg:                 alias  Antoine Beaupr  <anarcat@torproject.org>  [ultime]
gpg:                 alias  Antoine Beaupr  <anarcat@anarc.at>  [ultime]
gpg:                 alias  Antoine Beaupr  <anarcat@koumbit.org>  [ultime]
gpg:                 alias  Antoine Beaupr  <anarcat@debian.org>  [ultime]
Author: Antoine Beaupr  <anarcat@debian.org>
Date:   Mon Mar 16 14:37:28 2020 -0400
    fix test autoloading
    pytest only looks for file names matching  test  by default. We inline
    tests inside the source code directly, so hijack that.
Can you tell if this is a valid signature? If you speak a little french, maybe you can! But even if you would, you are unlikely to see that output on your own computer. What you would see instead is:
commit b3c538898b0ed4e31da27fc9ca22cb55e1de0000
gpg: Signature made Mon Mar 16 14:37:53 2020 EDT
gpg:                using RSA key 7B164204D096723B019635AB3EA1DDDDB261D97B
gpg: Can't check signature: No public key
Author: Antoine Beaupr  <anarcat@debian.org>
Date:   Mon Mar 16 14:37:28 2020 -0400
    fix test autoloading
    pytest only looks for file names matching  test  by default. We inline
    tests inside the source code directly, so hijack that.
Important part: Can't check signature: No public key. No public key. Because of course you would see that. Why would you have my key lying around, unless you're me. Or, to put it another way, why would that server I'm installing from scratch have a copy of my OpenPGP certificate? Because I'm a Debian developer, my key is actually part of the 800 keys in the debian-keyring package, signed by the APT repositories. So I have a trust path. But that won't work for someone who is not a Debian developer. It will also stop working when my key expires in that repository, as it already has on Debian buster (current stable). So I can't assume I have a trust path there either. One could work with a trusted keyring like we do in the Tor and Debian project, and only work inside that project, that said. But I still feel uncomfortable with those commands. Both git log and git show will happily succeed (return code 0 in the shell) even though the signature verification failed on the commits. Same with git pull and git merge, which will happily push your branch ahead even if the remote has unsigned or badly signed commits. To actually verify commits (or tags), you need the git verify-commit (or git verify-tag) command, which seems to do the right thing:
$ LANG=C.UTF-8 git verify-commit b3c538898b0ed4e31da27fc9ca22cb55e1de0000
gpg: Signature made Mon Mar 16 14:37:53 2020 EDT
gpg:                using RSA key 7B164204D096723B019635AB3EA1DDDDB261D97B
gpg: Can't check signature: No public key
[1]$
At least it fails with some error code (1, above). But it's not flexible: I can't use it to verify that a "trusted" developer (say one that is in a trusted keyring) signed a given commit. Also, it is not clear what a failure means. Is a signature by an expired certificate okay? What if the key is signed by some random key in my personal keyring? Why should that be trusted?

Worrying about git and GnuPG In general, I'm worried about git's implementation of OpenPGP signatures. There has been numerous cases of interoperability problems with GnuPG specifically that led to security, like EFAIL or SigSpoof. It would be surprising if such a vulnerability did not exist in git. Even if git did everything "just right" (which I have myself found impossible to do when writing code that talks with GnuPG), what does it actually verify? The commit's SHA-1 checksum? The tree's checksum? The entire archive as a zip file? I would bet it signs the commit's SHA-1 sum, but I just don't know, on the top of my head, and neither do git-commit or git-verify-commit say exactly what is happening. I had an interesting conversation with a fellow Debian developer (dkg) about this and we had to admit those limitations:
<anarcat> i'd like to integrate pgp signing into tor's coding practices more, but so far, my approach has been "sign commits" and the verify step was "TBD" <dkg> that's the main reason i've been reluctant to sign git commits. i haven't heard anyone offer a better subsequent step. if torproject could outline something useful, then i'd be less averse to the practice. i'm also pretty sad that git remains stuck on sha1, esp. given the recent demonstrations. all the fancy strong signatures you can make in git won't matter if the underlying git repo gets changed out from under the signature due to sha1's weakness
In other words, even if git implements the arcane GnuPG dialect just so, and would allow us to setup the trust chain just right, and would give us meaningful and workable error messages, it still would fail because it's still stuck in SHA-1. There is work underway to fix that, but in February 2020, Jonathan Corbet described that work as being in a "relatively unstable state", which is hardly something I would like to trust to verify code. Also, when you clone a fresh new repository, you might get an entirely different repository, with a different root and set of commits. The concept of "validity" of a commit, in itself, is hard to establish in this case, because an hostile server could put you backwards in time, on a different branch, or even on an entirely different repository. Git will warn you about a different repository root with warning: no common commits but that's easy to miss. And complete branch switches, rebases and resets from upstream are hardly more noticeable: only a tiny plus sign (+) instead of a star (*) will tell you that a reset happened, along with a warning (forced update) on the same line. Miss those and your git history can be compromised.

Possible ways forward I don't consider the current implementation of OpenPGP signatures in git to be sufficient. Maybe, eventually, it will mature away from SHA-1 and the interface will be more reasonable, but I don't see that happening in the short term. So what do we do?

git evtag The git-evtag extension is a replacement for git tag -s. It's not designed to sign commits (it only verifies tags) but at least it uses a stronger algorithm (SHA-512) to checksum the tree, and will include everything in that tree, including blobs. If that sounds expensive to you, don't worry too much: it takes about 5 seconds to tag the Linux kernel, according to the author. Unfortunately, that checksum is then signed with GnuPG, in a manner similar to git itself, in that it exposes GnuPG output (which can be confusing) and is likely similarly vulnerable to mis-implementation of the GnuPG dialect as git itself. It also does not allow you to specify a keyring to verify against, so you need to trust GnuPG to make sense of the garbage that lives in your personal keyring (and, trust me, it doesn't). And besides, git-evtag is fundamentally the same as signed git tags: checksum everything and sign with GnuPG. The difference is it uses SHA-512 instead of SHA-1, but that's something git will eventually fix itself anyways.

kernel patch attestations The kernel also faces this problem. Linus Torvalds signs the releases with GnuPG, but patches fly all over mailing list without any form of verification apart from clear-text email. So Konstantin Ryabitsev has proposed a new protocol to sign git patches which uses SHA256 to checksum the patch metadata, commit message and the patch itself, and then sign that with GnuPG. It's unclear to me what this solves, if anything, at all. As dkg argues, it would seem better to add OpenPGP support to git-send-email and teach git tools to recognize that (e.g. git-am) at least if you're going to keep using OpenPGP anyways. And furthermore, it doesn't resolve the problems associated with verifying a full archive either, as it only attests "patches".

jcat Unhappy with the current state of affairs, the author of fwupd (Richard Hughes) wrote his own protocol as well, called jcat, which provides signed "catalog files" similar to the ones provided in Microsoft windows. It consists of a "gzip-compressed JSON catalog files, which can be used to store GPG, PKCS-7 and SHA-256 checksums for each file". So yes, it is yet again another wrapper to GnuPG, probably with all the flaws detailed above, on top of being a niche implementation, disconnected from git.

The Update Framework One more thing dkg correctly identified is:
<dkg> anarcat: even if you could do exactly what you describe, there are still some interesting wrinkles that i think would be problems for you. the big one: "git repo's latest commits" is a loophole big enough to drive a truck through. if your adversary controls that repo, then they get to decide which commits to include in the repo. (since every git repo is a view into the same git repo, just some have more commits than others)
In other words, unless you have a repository that has frequent commits (either because of activity or by a bot generating fake commits), you have to rely on the central server to decide what "the latest version" is. This is the kind of problems that binary package distribution systems like APT and TUF solve correctly. Unfortunately, those don't apply to source code distribution, at least not in git form: TUF only deals with "repositories" and binary packages, and APT only deals with binary packages and source tarballs. That said, there's actually no reason why git could not support the TUF specification. Maybe TUF could be the solution to ensure end-to-end cryptographic integrity of the source code itself. OpenPGP-signed tarballs are nice, and signed git tags can be useful, but from my experience, a lot of OpenPGP (or, more accurately, GnuPG) derived tools are brittle and do not offer clear guarantees, and definitely not to the level that TUF tries to address. This would require changes on the git servers and clients, but I think it would be worth it.

Other Projects

OpenBSD There are other tools trying to do parts of what GnuPG is doing, for example minisign and OpenBSD's signify. But they do not integrate with git at all right now. Although I did find a hack] to use signify with git, it's kind of gross...

Golang Unsurprisingly, this is a problem everyone is trying to solve. Golang is planning on hosting a notary which would leverage a "certificate-transparency-style tamper-proof log" which would be ran by Google (see the spec for details). But that doesn't resolve the "evil server" attack, if we treat Google as an adversary (and we should).

Python Python had OpenPGP going for a while on PyPI, but it's unclear if it ever did anything at all. Now the plan seems to be to use TUF but my hunch is that the complexity of the specification is keeping that from moving ahead.

Docker Docker and the container ecosystem has, in theory, moved to TUF in the form of Notary, "a project that allows anyone to have trust over arbitrary collections of data". In practice however, in my somewhat limited experience, setting up TUF and image verification in Docker is far from trivial.

Android and iOS Even in what is possibly one of the strongest models (at least in terms of user friendliness), mobile phones are surprisingly unclear about those kind of questions. I had to ask if Android had end-to-end authentication and I am still not clear on the answer. I have no idea of what iOS does.

Conclusion One of the core problems with everything here is the common usability aspect of cryptography, and specifically the usability of verification procedures. We have become pretty good at encryption. The harder part (and a requirement for proper encryption) is verification. It seems that problem still remains unsolved, in terms of usability. Even Signal, widely considered to be a success in terms of adoption and usability, doesn't properly solve that problem, as users regularly ignore "The security number has changed" warnings... So, even though they deserve a lot of credit in other areas, it seems unlikely that hardcore C hackers (e.g. git and kernel developers) will be able to resolve that problem without at least a little bit of help. And TUF seems like the state of the art specification around here, it would seem wise to start adopting it in the git community as well. Update: git 2.26 introduced a new gpg.minTrustLevel to "tell various signature verification codepaths the required minimum trust level", presumably to control how Git will treat keys in your keyrings, assuming the "trust database" is valid and up to date. For an interesting narrative of how "normal" (without PGP) git verification can fail, see also A Git Horror Story: Repository Integrity With Signed Commits.

17 March 2020

Dirk Eddelbuettel: Rcpp 1.0.4: Lots of goodies

rcpp logo The fourth maintenance release 1.0.4 of Rcpp, following up on the 10th anniversary and the 1.0.0. release sixteen months ago, arrived on CRAN this morning. This follows a few days of gestation at CRAN. To help during the wait we provided this release via drat last Friday. And it followed a pre-release via drat a week earlier. But now that the release is official, Windows and macOS binaries will be built by CRAN over the next few days. The corresponding Debian package will be uploaded as a source package shortly after which binaries can be built. As with the previous releases Rcpp 1.0.1, Rcpp 1.0.2 and Rcpp 1.0.3, we have the predictable and expected four month gap between releases which seems appropriate given both the changes still being made (see below) and the relative stability of Rcpp. It still takes work to release this as we run multiple extensive sets of reverse dependency checks so maybe one day we will switch to six month cycle. For now, four months still seem like a good pace. Rcpp has become the most popular way of enhancing R with C or C++ code. As of today, 1873 packages on CRAN depend on Rcpp for making analytical code go faster and further, along with 191 in BioConductor. And per the (partial) logs of CRAN downloads, we are running steasy at one millions downloads per month. This release features quite a number of different pull requests by seven different contributors as detailed below. One (personal) highlight is the switch to tinytest.

Changes in Rcpp version 1.0.4 (2020-03-13)
  • Changes in Rcpp API:
    • Safer Rcpp_list*, Rcpp_lang* and Function.operator() (Romain in #1014, #1015).
    • A number of #nocov markers were added (Dirk in #1036, #1042 and #1044).
    • Finalizer calls clear external pointer first (Kirill M ller and Dirk in #1038).
    • Scalar operations with a rhs matrix no longer change the matrix value (Qiang in #1040 fixing (again) #365).
    • Rcpp::exception and Rcpp::stop are now more thread-safe (Joshua Pritikin in #1043).
  • Changes in Rcpp Attributes:
    • The cppFunction helper now deals correctly with mulitple depends arguments (TJ McKinley in #1016 fixing #1017).
    • Invisible return objects are now supported via new option (Kun Ren in #1025 fixing #1024).
    • Unavailable packages referred to in LinkingTo are now reported (Dirk in #1027 fixing #1026).
    • The sourceCpp function can now create a debug DLL on Windows (Dirk in #1037 fixing #1035).
  • Changes in Rcpp Documentation:
    • The .github/ directory now has more explicit guidance on contributing, issues, and pull requests (Dirk).
    • The Rcpp Attributes vignette describe the new invisible return object option (Kun Ren in #1025).
    • Vignettes are now included as pre-made pdf files (Dirk in #1029)
    • The Rcpp FAQ has a new entry on the recommended importFrom directive (Dirk in #1031 fixing #1030).
    • The bib file for the vignette was once again updated to current package versions (Dirk).
  • Changes in Rcpp Deployment:
    • Added unit test to check if C++ version remains remains aligned with the package number (Dirk in #1022 fixing #1021).
    • The unit test system was switched to tinytest (Dirk in #1028, #1032, #1033).

Please note that the change to execptions and Rcpp::stop() in pr #1043 has been seen to have a minor side effect on macOS issue #1046 which has already been fixed by Kevin in pr #1047 for which I may prepare a 1.0.4.1 release for the Rcpp drat repo in a day or two. Thanks to CRANberries, you can also look at a diff to the previous release. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. Bugs reports are welcome at the GitHub issue tracker as well (where one can also search among open or closed issues); questions are also welcome under rcpp tag at StackOverflow which also allows searching among the (currently) 2356 previous questions. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.