Search Results: "reg"

1 January 2021

Keith Packard: kgames

Reviving Very Old X Code I've taken the week between Christmas and New Year's off this year. I didn't really have anything serious planned, just taking a break from the usual routine. As often happens, I got sucked into doing a project when I received this simple bug report Debian Bug #974011
I have been researching old terminal and X games recently, and realized
that much of the code from 'xmille' originated from the terminal game
'mille', which is part of bsdgames.
...
[The copyright and license information] has been stripped out of all
code in the xmille distribution.  Also, none of the included materials
give credit to the original author, Ken Arnold.
The reason the 'xmille' source is missing copyright and license information from the 'mille' files is that they were copied in before that information was added upstream. Xmille forked from Mille around 1987 or so. I wrote the UI parts for the system I had at the time, which was running X10R4. A very basic port to X11 was done at some point, and that's what Debian has in the archive today. At some point in the 90s, I ported Xmille to the Athena widget set, including several custom widgets in an Xaw extension library, Xkw. It's a lot better than the version in Debian, including displaying the cards correctly (the Debian version has some pretty bad color issues). Here's what the current Debian version looks like: Fixing The Bug To fix the missing copyright and license information, I imported the mille source code into the "latest" Xaw-based version. The updated mille code had a number of bug fixes and improvements, along with the copyright information. That should have been sufficient to resolve the issue and I could have constructed a suitable source package from whatever bits were needed and and uploaded that as a replacement 'xmille' package. However, at some later point, I had actually merged xmille into a larger package, 'kgames', which also included a number of other games, including Reversi, Dominoes, Cribbage and ten Solitaire/Patience variants. (as an aside, those last ten games formed the basis for my Patience Palm Pilot application, which seems to have inspired an Android App of the same name...) So began my yak shaving holiday. Building Kgames in 2020 Ok, so getting this old source code running should be easy, right? It's just a bunch of C code designed in the 80s and 90s to work on VAXen and their kin. How hard could it be?
  1. Everything was a 32-bit computer back then; pointers and ints were both 32 bits, so you could cast them with wild abandon and cause no problems. Today, testing revealed segfaults in some corners of the code.
  2. It's K&R C code. Remember that the first version of ANSI-C didn't come out until 1989, and it was years later that we could reliably expect to find an ANSI compiler with a random Unix box.
  3. It's X11 code. Fortunately (?), X11 hasn't changed since these applications were written, so at least that part still works just fine. Imagine trying to build Windows or Mac OS code from the early 90's on a modern OS...
I decided to dig in and add prototypes everywhere; that found a lot of pointer/int casting issues, as well as several lurking bugs where the code was just plain broken. After a day or so, I had things building and running and was no longer hitting crashes. Kgames 1.0 uploaded to Debian New Queue With that done, I decided I could at least upload the working bits to the Debian archive and close the bug reported above. kgames 1.0-2 may eventually get into unstable, presumably once the Debian FTP team realizes just how important fixing this bug is. Or something. Here's what xmille looks like in this version: And here's my favorite solitaire variant too: But They Look So Old Yeah, Xaw applications have a rustic appearance which may appeal to some, but for people with higher resolution monitors and well seasoned eyesight, squinting at the tiny images and text makes it difficult to enjoy these games today. How hard could it be to update them to use larger cards and scalable fonts? Xkw version 2.0 I decided to dig in and start hacking the code, starting by adding new widgets to the Xkw library that used cairo for drawing instead of core X calls. Fortunately, the needs of the games were pretty limited, so I only needed to implement a handful of widgets: The other Xkw widgets all got their rendering switched to using cairo, plus using double buffering to make updates look better. SVG Playing Cards Looking on wikimedia, I found a page referencing a large number of playing cards in SVG form That led me to Adrian Kennard's playing card web site that let me customize and download a deck of cards, licensed using the CC0 Public Domain license. With these cards, I set about rewriting the Xkw playing card widget, stripping out three different versions of bitmap playing cards and replacing them with just these new SVG versions. SVG Xmille Cards Ok, so getting regular playing cards was good, but the original goal was to update Xmille, and that has cards hand drawn by me. I could just use those images, import them into cairo and let it scale them to suit on the screen. I decided to experiment with inkscape's bitmap tracing code to see what it could do with them. First, I had to get them into a format that inkscape could parse. That turned out to be a bit tricky; the original format is as a set of X bitmap layers; each layer painting a single color. I ended up hacking the Xmille source code to generate the images using X, then fetching them with XGetImage and walking them to construct XPM format files which could then be fed into the portable bitmap tools to create PNG files that inkscape could handle. The resulting images have a certain charm: I did replace the text in the images to make it readable, otherwise these are untouched from what inkscape generated. The Results Remember that all of these are applications built using the venerable X toolkit; there are still some non-antialiased graphics visible as the shaped buttons use the X Shape extension. But, all rendering is now done with cairo, so it's all anti-aliased and all scalable. Here's what Xmille looks like after the upgrades: And here's spider: Once kgames 1.0 reaches Debian unstable, I'll upload these new versions.

31 December 2020

Daniel Kahn Gillmor: New OpenPGP certificate for dkg, 2021

dkg's 2021 OpenPGP transition As 2021 begins, I'm changing to a new OpenPGP certificate. I did a similar transition two years ago, and a fair amount has changed since then. You might know my old OpenPGP certificate as:
pub   ed25519 2019-01-19 [C] [expires: 2021-01-18]
      C4BC2DDB38CCE96485EBE9C2F20691179038E5C6
uid          Daniel Kahn Gillmor <dkg@fifthhorseman.net>
uid          Daniel Kahn Gillmor <dkg@debian.org>
My new OpenPGP certificate is:
pub   ed25519 2020-12-27 [C] [expires: 2023-12-24]
      C29F8A0C01F35E34D816AA5CE092EB3A5CA10DBA
uid           [ unknown] Daniel Kahn Gillmor
uid           [ unknown] <dkg@debian.org>
uid           [ unknown] <dkg@fifthhorseman.net>
You can find a signed transition statement if you're into that sort of thing. If you're interested in the rationale for why I'm making this transition, read on.

Dangers of Offline Primary Secret Keys There are several reasons for transitioning, but one i simply couldn't argue with was my own technical failure. I put the primary secret key into offline storage some time ago for "safety", and used ext4's filesystem-level encryption layered on top of dm-crypt for additional security. But either the tools changed out from under me, or there were failures on the storage medium, or I've failed to remember my passphrase correctly, because I am unable to regain access to the cleartext of the secret key. In particular, I find myself unable to use e4crypt add_key with the passphrase I know to get a usable working directory. I confess I still find e4crypt pretty difficult to use and I don't use it often, so the problem may entirely be user error (either now, or two years ago when I did the initial setup). Anyway, lesson learned: don't use cryptosystems that you're not comfortable with to encrypt data that you care about recovering. This is a lesson I'm pretty sure I've learned before, sigh, but it's a good reminder.

Split User IDs I'm trying to split out my User IDs again -- this way if you know me by e-mail address, you don't have to think/worry about certifying my name, and if you know me by name, you don't have to think/worry about certifying my e-mail address. I think that's simpler and more sensible. It's also nice because e-mail address-only User IDs can be used effectively in contexts like Autocrypt, which I think are increasingly important if we want to have usable encrypted e-mail. Last time around I initially tried split User IDs but rolled them back and I think most of the bugs I discovered then have been fixed.

Certificate Flooding Another reason for making a transition to a new certificate is that my older certificate is one of the ones that was "flooded" on the SKS keyserver network last year, which was one of the final straws for that teetering project. Transitioning to a new certificate lets that old flooded cert expire and people can just simply move on from it, ideally deleting it from their local keyrings. Hopefully as a community we can move on from SKS to key distribution mechanisms like WKD, Autocrypt, DANE, and keys.openpgp.org, all of which address some of the known problems with keyserver abuse.

Trying New Tools Finally, I'm also interested in thinking about how key and certificate management might be handled in different ways. While I'm reasonably competent in handling GnuPG, the larger OpenPGP community (which I'm a part of) has done a lot of thinking and a lot of work about how people can use OpenPGP. I'm particularly happy with the collaborative work that has gone into the Stateless OpenPGP CLI (aka sop), which helps to generate a powerful interoperability test suite. While sop doesn't offer the level of certificate management I'd need to use it to manage this new certificate in full, I wish something like it would! Starting from a fresh certificate and actually using it helps me to think through what I might actually need from a tool that is roughly as straightforward and opinionated as sop is. If you're a software developer who might use or implement OpenPGP, or a protocol designer, and you haven't played around with any of the various implementations of sop yet, I recommend taking a look. And feedback on the specification is always welcome, too, including ideas for new functionality (maybe even like certificate management).

Next Steps If you're the kind of person who's into making OpenPGP certifications, feel free to check in with me via whatever channels you're used to using to verify that this transition is legit. If you think it is, and you're comfortable, please send me (e-mail is probably best) your certifications over the new certficate. I'll keep on working to make OpenPGP more usable and acceptable. Hopefully, 2021 will be a better year ahead for all of us.

30 December 2020

Russ Allbery: Review: The House in the Cerulean Sea

Review: The House in the Cerulean Sea, by TJ Klune
Publisher: Tor
Copyright: March 2020
ISBN: 1-250-21732-6
Format: Kindle
Pages: 398
Linus Baker is a case worker for the Department in Charge of Magical Youth. His job is to evaluate the orphanages to which children with magical powers are sent and ensure those children are treated properly. This must be done properly, with professional care and detachment and careful documentation so that his superiors can make the correct choices. Linus has no ambition or desire to be anything other than a case worker. He does his job meticulously by the book, namely the DICOMY Rules and Regulations, which he studies religiously. Linus is therefore caught entirely by surprise when he is summoned without explanation to the offices of Extremely Upper Management on the intimidating 5th floor. That summons turns out to be a new assignment for which Extremely Upper Management believes Linus to be ideal: an orphanage on an island far from the city, home to some highly unusual (and, in one case, disturbing) children and a headmaster who is as odd as his charges. Management is correct about Linus up to a point. He is a bureaucrat and a stickler for rules. However, Extremely Upper Management did not account for the possibility that Linus does the work that he does because he cares deeply about children, or that, if the two come into conflict, the children might matter more than the rules. The House in the Cerulean Sea is the sort of book that tells you exactly what it's doing, assures you that you'll enjoy it anyway, and then you actually do. The plot developments are obvious well in advance, the tugs on the heart-strings are telegraphed and obvious, and there are very few surprises. But the story is told with such charm and feeling that I at least didn't mind at all, and indeed would have been furious at the author if he hadn't delivered the expected ending. This is not the book to read if you want to delve deep into the implications of the world-building. It follows children's book logic in several important places: intentions matter more than institutional structure, sincerity is persuasive, and courage is rewarded. A few reviews compare it to Nineteen Eighty-Four, but this is wildly misleading beyond a few trappings of the setting. The internal logic of the book is closer to a cross between Good Omens and Dr. Seuss. It's a "good people are rewarded" sort of book and wears that on its sleeve. One of the things that Klune does extremely well is subtlety of characterization despite the relative lack of subtlety in the plot. Linus starts the book as a caricature whose life is nearly a blank canvas, and by the end of the book he feels like someone you know. Klune pulls this off less by filling in the canvas than by making careful use of all that negative space to put emotional weight and depth behind a few telling details like Linus's pyjamas, records, and cat. He also does an exceptional job with the mannerisms and habits of speech of both Linus and the headmaster of the orphanage. To say much more would be a spoiler; suffice it to say that Klune captures an interaction style that I have rarely seen in novels and certainly not in a fantasy novel of this type, and does it so well that I was strongly reminded of people I've known. Anyone who knows me is unlikely to be surprised that my favorite character in the book is Zoe, the sarcastic and fiercely protective sprite who owns the island on which the orphanage is located, but almost all of the characters are wonderful. One or two of the children are a bit one-note, but the group dynamics make up for that, and the suspicious, thoughtful, and respectful interplay between the three adults is beautifully done. My only real complaints on the characterization front are that some of the confrontations in the nearby village felt a bit strained, and I think Helen got short shrift and could have been fleshed out some more. I will say that the motives of the villains never made much sense, but they didn't have to; they're essentially monsters in Linus's story, and they do enough to serve that purpose. I now want to read the story of how Zoe and the headmaster met, although I'm not sure Klune is the right person to write it, and I'm not sure he left enough emotional space for another novel. Sometimes I want to read a positive, hopeful book in which people learn and grow and good things happen to good people, one where the author is unapologetically manipulating my emotions and I don't care because I want these people to be happy. If you feel the same way, I cannot recommend this book highly enough. It's lovely. Rating: 9 out of 10

29 December 2020

Shirish Agarwal: Inequality in Indian Education

Farmer on-going protests Before I start with the education system in India which I have talked about many times in the past, first let me share couple of pieces about the farmer movement which is still at Sanghu Delhi border.
<Manjeet Kaur,62 at farmer protest with her friends.
The above picture became somewhat viral as it showed Manjeet Kaur, who drove down from Patiala, Punjab to Sanghu border along with her friends to take part in the on-going protests. The picture not only shares how the women are part and parcel of this protest but also they are independently taking part in the protest. The other were two articles I read today, first was an article in tribune which questions that if the policy worked, why it didn t work in the state of Bihar. The other by a young law student who had to go from Chandigarh to Delhi with family for some work and her experience with the ongoing protest. In fact, an interesting observation was made by the CJI in the many suits against farmer protests in the SC. This makes for much more interesting read when you see an RTI query filed by Saket Gokhale to NHAI , a Central Govt. agency which is supposed to be independent and asks if they had filed an FIR and asked compensation from Haryana State and Haryana State Police which had dug up National Highway 44 and if any permission was asked for the same from NHAI. And NHAI unable to take any action for the same. If this isn t shameful, I dunno what is
Saket Gokhale s RTI query on digging up NH44
NHAI response to Saket Gokhale s query.
Sadly, the way the response has been worded makes it impossible for NHAI to discharge its own responsibilities and this becomes a precedent for other states now that know that NHAI is vulnerable. A pretty sad turn on events. Indian Education can t go online There was a recent article on scroll which shared how Indian education can t go online as only a few have computers with decent netlink speeds as well as other factors which are needed for online education. But there are also many things that the article doesn t take into account which actually make the task more difficult and raise the boundary more. Now in most schools and colleges, the number of students to teacher ratio could be anywhere between 70-150 or even more. In the last few years, a lot of schools have been closed down by various Governments, including and not limited to the ruling Govt. They have in fact intensified closures of public schools wherever their Govt. has been in power. Closing to 5000+ schools in one state in a year is a dramatic shift and such has been happening time and again. In fact, the rising costs of Indian education has made many to leave Indian shores and do studies abroad. And once they do their masters or whatever, the chances of them coming back to India become more and more remote. In India the costs have been becoming so bad that NBFC s have started products targeting the same. How NBFC and Banks have (both public and private) have fared with respect to Indian consumers needs its own blog post but one word to describe it is bad . But as shared above, needs its own blog post. Coming to the Indian context though, what has not been captured in that article is that the responsibility of making new content also raises huge barriers for teachers. My own experience in teacher s trainings for ICT usage has shown that most teachers do not know and use internet effectively both to sustain their own curiosity as well as their students. Part of which is whether you are private employee or a public school teacher, the teacher is not paid enough. I have had multiple conversations with friends over the years who are teachers who shared that they get 50% salary in-hand while they sign for 100%. This is more in the case or private schools though. In Govt. schools, the teachers apart from their regular administrative duties apart from teaching duties are also unpaid labor for Govt. policy. Take the recent covid crisis, it was the teachers who for months together went from door-to-door asking if they had a covid patient. This was all over India. Even for voter registration, census, polio and various other immunization efforts, the teachers are roped in. So apart from that, they somehow have to figure out how to make ends meet and also boost student morale. Hence the attention is only limited to the first couple of benches rather than the whole as a 45-minute to an hr. session is just not enough to go through a class of 70-150 school students giving individual attention. And this is when for most teachers, teaching is a means to an end and not the end itself. I am going to take one example of science and sort of break-it-down in multiple steps and how I would have approached that topic for say the 5th-6th standard students in say a public school in Pune and especially if Covid would not have been an issue so you have face-to-face meetup. There was recent news about a mysterious radio signal which came from one of our closest galactic neighbors Proxima Centauri. Now let s say there was a class I was teaching/sharing which I had shared before and there already is trust formed. So before coming to the news, I would tell the students about frequency and more generally the notation of why we like to measure things and how we measure things. There is so much beautiful history which could be acted and enacted which can show and remains in mind why measurement is needed. Once that is understood, discussed and an underpinning is established, we could move to human perception or the lack of it. We know that humans have lots of limits in almost everything, whether it is talking, touching, hearing, all of our five senses are pretty limited with what we know of spectrum available in the immediate family kingdom as well as in the Universe. I would start with how far can a person throw his voice and be heard without using any other means. There does come a point where they need to use anything from a megaphone to a loudspeaker and what it actually does. The other thing I would then talk is about the radio and ask the students to find more about the internals of a radio. If possible to bring an old radio to school where it could be disassembled. After they are familiar with some names of the electronic components and what they do, take them to the electronics market where they try to source all the things needed to make a radio and whatever they encounter. This would allow the students to try and do bargain shopping as well as learn from where to source things. Some might even get a copy or two of electronic projects where the shop themselves sell blueprints to hobbyists so that they can tinker. If there is a place in the school where soldering can be done, then all can try and sooner or later we come to know if something works or not. There is also possibility of talking about noise cancellation and then the topic of ITU can also be bought up and how they do frequency allocation. Last but not the least then the topic can be approached about an alien civilization and an unknown radio signal and what it means and what it can mean. Now if just one topic can give such a wide range of things to do and develop an understanding about not the subject itself but surrounding subjects as well I see no reason why teachers can t do this except they are handcuffed to lot of policy as well as real-life constraints. For e.g. I remember in my school days, we used to go out once or twice a year and that used to be either a school picnic or something similar. The only other I know is going to Mumbai for Nehru planetarium and Nehru science Centre. Unfortunately, I didn t go at that time because the school was taking students via air and the tickets were super costly at the time and that too for a 10 minute journey between the two cities. Those were different days, today you can t have a direct flight between the two cities as it doesn t make an economic sense. It makes more sense to go to Mumbai via train or bus as you will reach Mumbai in about couple of hrs. Of course, Pune does have its own planetarium at New English school and there are a few amateur astronomy clubs in Pune but nothing on the scale that what Mumbai has, but then this is getting off-topic. Now, again in an online world could this be done? Not without both the teacher and the student both spending lot of resources online and even then will be a lower understanding as both the hands-on experience as well as interacting with other students and learning from other students (aping) would be hugely limited. Even the social skills that students develop in a school setting will be rusted. My own social skills probably have weakened and rusted as I have very limited interaction with people over the past few months due to Covid fears and would be at least for the next few years till a large enough population is not vaccinated.

28 December 2020

Guido G nther: phosh overview

phosh is graphical shell for mobile, touch based devices like smart phones. It's the default graphical shell on Purism's Librem 5 (and that's where it came to life) but projects like postmarketOS, Mobian and Debian have picked it up putting it into use on other devices as well and contributing patches. This post is meant as a short overview how things are tied together so further posts can provide more details. A PHone SHell As mobile shell phosh provides the interface components commonly found on mobile devices to phosh's overview 2 phosh's lockscreen phosh's overview 1 It uses GObject object system and GTK to build up the user interface components. Mobile specific patterns are brought in via libhandy. Since phosh is meant to blend into GNOME as seamlessly as possible it uses the common interfaces present there via D-Bus like org.gnome.Screensaver or org.gnome.keyring.SystemPrompter and retrieves user configuration like keybindings via GSettings from preexisting schema. The components of a running graphical session roughly look like this: phosh session The blue boxes are the very same found on GNOME desktop sessions while the white ones are currently only found on phones. feedbackd is explained quickly: It's used for providing haptic or visual user feedback and makes your phone rumble and blink when applications (or the shell) want to notify the user about certain events like incoming phone calls or new messages. What about phoc and squeekboard? phoc and squeekboard Although some stacks combine the graphical shell with the display server (the component responsible for drawing applications and handling user input) this isn't the case for phosh. phosh relies on a Wayland compositor to be present for that. Keeping shell and compositor apart has some advantages like being able to restart the shell without affecting other applications but also adds the need for some additional communication between compositor and shell. This additional communication is implemented via Wayland protocols. The Wayland compositor used with phosh is called phoc for PHone Compositor. One of these additional protocols is wlr-layer-shell. It allows the shell to reserve space on the screen that is not used by other applications and allows it to draw things like the top and bottom bar or lock screen. Other protocols used by phosh (and hence implemented by phoc) are wlr-output-management to get information on and control properties of monitors or wlr-foreign-toplevel-management to get information about other windows on the display. The later is used to allow to switch between running applications. However these (and other) Wayland protocols are not implemented in phoc from scratch. phoc leverages the wlroots library for that. The library also handles many other compositor parts like interacting with the video and input hardware. The details on how phoc actually puts things up on the screen deserves a separate post. For the moment it's sufficient to note that phosh requires a Wayland compositor like phoc. We've not talked about entering text without a physical keyboard yet - phosh itself does not handle that either. squeekboard is the on screen keyboard for text (and emoji) input. It again uses Wayland protocols to talk to the Wayland compositor and it's (like phosh) a component that wants exclusive access to some areas of the screen (where the keyboard is drawn) and hence leverages the layer-shell protocol. Very roughly speaking it turns touch input in that area into text and sends that back to the compositor that then passes it back to the application that currently gets the text input. squeekboard's main author dcz has some more details here. The session So how does the graphical session in the picture above come into existence? As this is meant to be close to a regular GNOME session it's done via gnome-session that is invoked somewhat like:
phoc -E 'gnome-session --session=phosh'
So the compositor phoc is started up, launches gnome-session which then looks at phosh.session for the session's components. These are phosh, squeekboard and gnome-settings-daemon. These then either connect to already running services via D-Bus (e.g. NetworkManager, ModemManager, ...) or spawn them via D-Bus activation when required (e.g. feedbackd). Calling conventions So when talking about phosh it's good to keep several things apart: On top of that people sometimes refer to 'Phosh' as the software collection consisting of the above plus more components from GNOME (Settings, Contacs, Clocks, Weather, Evince, ...) and components that currently aren't part of GNOME but adapt to small screen sizes, use the same technologies and are needed to make a phone fun to use e.g. Geary for email, Calls for making phone calls and Chats for SMS handling. Since just overloading the term Phosh is confusing GNOME/Phosh Mobile Environment or Phosh Mobile Environment have been used to describe the above collection of software and I've contacted GNOME on how to name this properly, to not infringe on the GNOME trademark but also give proper credit and hopefully being able to move things upstream that can live upstream. That's it for a start. phosh's development documentation can be browsed here but is also available in the source code. Besides the projects mentioned above credits go to Purism for allowing me and others to work on the above and other parts related to moving Free Software on mobile Linux forward.

27 December 2020

Russ Allbery: Review: King of the Murgos

Review: King of the Murgos, by David Eddings
Series: The Malloreon #2
Publisher: Del Rey
Copyright: April 1988
Printing: November 1991
ISBN: 0-345-35880-5
Format: Mass market
Pages: 403
This is the second book of the Malloreon, which in turn is a sequel trilogy to The Belgariad. You could start here, since Eddings does a good job at summarizing previous volumes, but I'm not sure why you'd want to. This is the sort of book that gives epic fantasy series a bad name. After a lot of action in Guardians of the West, our band of heroes has been mostly assembled. They set out after the villain through the novel strategy of exploring the parts of the world map we have not yet seen. By the end of 400 pages of traveling they... are on the trail of the villain while exploring parts of the world map we have not yet seen. What I'm saying is that you'd better be really enjoying Silk's nose twitching and Belgarath being grumpy at Polgara, because that's about all you get. The protagonists collect a few important plot coupons and set up some emotional conflicts whose resolutions are already telegraphed, and that's about it. After wasting a bunch of time getting to and wandering around Nyissa, all of which felt pointless except for one moment of Errand doing something nifty (often the best parts of this series), the band of heroes ventures into enemy territory as foreshadowed by the title. The structure of Eddings's series is basically Suikoden (collect all the protagonists), so as one might expect the point of this excursion is to pick up another protagonist. In the process, we get to see a bit more of the Murgos than we have before. I've commented before that Eddings takes racial essentialism to such an absurd degree that these books start feeling like animal fairy tales, which is why I'm willing to tolerate all of the stereotyping. It's Planet of Hats except with fantasy countries. That tolerance is stretched rather thin when it comes to the bad guys, since the congruence with real-life racism and dehumanization of enemies is a bit too strong. Eddings dodges this mostly by not putting many of the bad guys in the page, but in King of the Murgos he has a golden opportunity to undercut his racial structure and surprise the reader. What if one of the Murgos is a protagonist and is vital to the series plot? I won't spoil what Eddings actually does, but if you imagine the stupidest and most obvious way to avoid having to acknowledge his fantasy races are not uniform, you probably just guessed the big reveal in this book. It's sadly not surprising, since Eddings is all in on his racial construction of this fantasy world, but it's still disgusting. The main reason why I decided to re-read this series, which is notorious for being a re-run of the Belgariad, is because I think what Eddings does here with prophecy, the voice in Garion's head, and the meddling of the seers is hilarious. The plot is openly on rails to the point that the characters actively grumble about it. I find that oddly entertaining, a bit like watching a Twitch streamer play an RPG. In this installment, Errand does something random in the middle of a Murgo temple, Garion gets a level-up chime, the prophecy gets very smug about how well things are going, and no one knows why. It's an absolute delight, and I don't know of any other series that isn't pure parody that would be willing to use that as a plot element. Sadly, King of the Murgos has only one or two enjoyable moments like that, a whole lot of frankly boring map exploration, some truly egregious racist claptrap, and no plot development worth speaking of. I probably should have skipped this one when re-reading this series. Followed by Demon Lord of Karanda. Rating: 4 out of 10

26 December 2020

Sandro Tosi: QNAP firmware 4.5.1.1465: disable ssh management menu

as a good boy i just upgraded my QNAP NAS to the latest available firmware, 4.5.1.1465, but after the reboot there's an ugly surprise awaiting for meonce i ssh'd into the box to do my stuff, instead of a familiar bash prompt i'm greeted by a management menu that allows me to perform some basic management tasks or quit it and go back to the shell. i dont really need this menu (in particular because i have automations that regularly ssh into the box and they are not meant to be interactive).to disable it: edit /etc/profile and comment the line "[[ "admin" = "$USER" ]] && /sbin/qts-console-mgmt -f" (you can judge me later for sshing as root)UPDATE (2020-12-26): as pointed out in the comments, and discovered now after an upgrade, the menu comes back up after because /etc/profile is a system file for QNAP so gets replaced/updated by them often

Paul Tagliamonte: Reverse Engineering my Christmas Tree

Over the course of the last year and a half, I ve been doing some self-directed learning on how radios work. I ve gone from a very basic understanding of wireless communications (there s usually some sort of antenna, I guess?) all the way through the process of learning about and implementing a set of libraries to modulate and demodulate data using my now formidable stash of SDRs. I ve been implementing all of the RF processing code from first principals and purely based on other primitives I ve written myself to prove to myself that I understand each concept before moving on. I figured that there was a fun capstone to be done here - the blind reverse engineering and implementation of the protocol my cheep Amazon power switch uses to turn on and off my Christmas Tree. All the work described in this post was done over the course of a few hours thanks to help during the demodulation from Tom Bereknyei and hlieberman.

Going in blind When I first got my switch, I checked it for any FCC markings in order to look up the FCC filings to determine the operational frequency of the device, and maybe some other information such as declared modulation or maybe even part numbers and/or diagrams. However, beyond a few regulatory stickers, there were no FCC ids or other distinguishing IDs on the device. Worse yet, it appeared to be a whitelabeled version of another product, so searching Google for the product name was very unhelpful. Since operation of this device is unlicensed, I figured I d start looking in the ISM band. The most common band used that I ve seen is the band starting at 433.05MHz up to 434.79MHz. I fired up my trusty waterfall tuned to a center frequency of 433.92MHz (since it s right in the middle of the band, and it let me see far enough up and down the band to spot the remote) and pressed a few buttons. Imagine my surprise when I realize the operational frequency of this device is 433.920MHz, exactly dead center. Weird, but lucky! After taking a capture, I started to look at understanding what the modulation type of the signal was, and how I may go about demodulating it. Using inspectrum, I was able to clearly see the signal in the capture, and it immediately stuck out to my eye to be encoded using OOK / ASK. Next, I started to measure the smallest pulse, and see if I could infer the symbols per second, and try to decode it by hand. These types of signals are generally pretty easy to decode by eye. This wound up giving me symbol rate of 2.2 Ksym/s, which is a lot faster than I expected. While I was working by hand, Tom demodulated a few messages in Python, and noticed that if you grouped the bits into groups of 4, you either had a 1000 or a 1110 which caused me to realize this was encoded using something I saw documented elsewhere, where the 0 is a short pulse, and a 1 is a long pulse, not unlike morse code, but where each symbol takes up a fixed length of time (monospace morse code?). Working on that assumption, I changed my inspectrum symbol width, and demodulated a few more by hand. This wound up demodulating nicely (and the preamble / clock sync could be represented as repeating 0s, which is handy!) and gave us a symbol rate of 612(ish) symbols per second a lot closer to what I was expecting. If we take the code for on in the inspectrum capture above and demodulate it by hand, we get 0000000000110101100100010 (treat a short pulse as a 0, and a long pulse as a 1). If you re interested in following along at home, click on the inspectrum image, and write down the bits you see, and compare it to what I have! Right, so it looks like from what we can tell so far that the packet looks something like this:
preamble / sync
stuff
Next, I took a capture of all the button presses and demodulated them by hand, and put them into a table to try and understand the format of the messages:
Button Demod'd Bits
On 0000000000110101100100010
Off 00000000001101011001010000
Dim Up 0000000000110101100110100
Dim Down 0000000000110101100100100
Timer 1h 0000000000110101100110010
Timer 2h 0000000000110101100100110
Timer 4h 0000000000110101100100000
Dim 100% 0000000000110101000101010
Dim 75% 00000000001101010001001100
Dim 50% 00000000001101010001001000
Dim 25% 0000000000110101000100000
Great! So, this is enough to attempt to control the tree with, I think so I wrote a simple modulator. My approach was to use the fact that I can break down a single symbol into 4 sub-symbol components which is to say, go back to representing a 1 as 1110, and a 0 as 1000. This let me allocate IQ space for the symbol, break the bit into 4 symbols, and if that symbol is 1, write out values from a carrier wave (cos in the real values, and sin in the imaginary values) to the buffer. Now that I can go from bits to IQ data, I can transmit that IQ data using my PlutoSDR or HackRF and try and control my tree. I gave it a try, and the tree blinked off! Success! But wait that s not enough for me I know I can t just demodulate bits and try and replay the bits forever there s stuff like addresses and keys and stuff, and I want to get a second one of these working. Let s take a look at the bits to see if we spot anything fun & interesting. At first glance, a few things jumped out at me as being weird? First is that the preamble is 10 bits long (fine, let s move along - maybe it just needs 8 in a row and there s two to ensure clocks sync?). Next is that the messages are not all the same length. I double (and triple!) checked the messages, and it s true, the messages are not all the same length. Adding an extra bit at the end didn t break anything, but I wonder if that s just due to the implementation rather than the protocol. But, good news, it looks like we have a stable prefix to the messages from the remote must be my device s address! The stable 6 bits that jump out right away are 110101. Something seems weird, though, 6 bits is a bit awkward, even for a bit limited embedded device. Why 6? But hey, wait, we had 10 bits in the preamble, what if we have an 8 bit address meaning my device is 00110101, and the preamble is 8 0 symbols! Those are numbers that someone working on an 8 bit aligned platform would pick! To test this, I added a 0 to the preamble to see if the message starts at the first 1, or if it requires all the bits to be fully decoded, and lo and behold, the tree did not turn on or off. This would seem to me to confirm that the 0s are part of the address, and I can assume we have two 8 bit aligned bytes in the prefix of the message.
preamble / sync
address
stuff
Now, when we go through the 9-10 bits of stuff , we see all sorts of weird bits floating all over the place. The first 4 bits look like it s either 1001 or 0001, but other than that, there s a lot of chaos. This is where things get really squishy. I needed more information to try and figure this out, but no matter how many times I sent a command it was always the same bits (so, no counters), and things feel very opaque still. The only way I was going to make any progress is to get another switch and see how the messages from the remote change. Off to Amazon I went, and ordered another switch from the same page, and eagerly waited its arrival.

Switch #2 The second switch showed up, and I hurriedly unboxed the kit, put batteries into the remote, and fired up my SDR to take a capture. After I captured the first button ( Off ), my heart sunk as I saw my lights connected to Switch #1 flicker off. Apparently the new switch and the old switch have the same exact address. To be sure, I demodulated the messages as before, and came out with the exact same bit pattern. This is a setback and letdown I was hoping to independently control my switches, but it also means I got no additional information about the address or button format. The upside to all of this, though, is that because the switches are controlled by either remote, I only needed one remote, so why not pull it apart and see if I can figure out what components it s using to transmit, and find any datasheets I can. The PCB was super simple, and I wound up finding a WL116SC IC on the PCB. After some googling, I found a single lone datasheet, entirely in Chinese. Thankfully, Google Translate seems to have worked well enough on technical words, and I was able to put together at least a little bit of understanding based on the documentation that was made available. I took a few screenshots below - I put the google translated text above the hanzi. From that sheet, we can see we got the basics of the 1 and 0 symbol encoding right (I was halfway expecting the bits to be flipped), and a huge find by way of a description of the bits in the message! It s a bummer that we missed the clock sync / preamble pulse before the data message, but that s OK somehow. It also turns out that 8 or 10 bit series of of 0"s wasn t clock sync at all - it was part of the address! Since it also turns out that all devices made by this manufacturer have the hardcoded address of []byte 0x00, 0x35 , that means that the vast majority of bits sent are always going to be the same for any button press on any remote made by this vendor. Seems like a waste of bits to me, but hey, what do I know. Additionally, this also tells us the trailing zeros are not part of the data encoding scheme, which is progress!
address
keycode
Now, working on the assumptions validated by the datasheet, here s the updated list of scancodes we ve found:
Button Scancode Bits Integer
On 10010001 145 / 0x91
Off 10010100 148 / 0x94
Dim Up 10011010 154 / 0x9A
Dim Down 10010010 146 / 0x92
Timer 1h 10011001 154 / 0x99
Timer 2h 10010011 147 / 0x93
Timer 4h 10010000 144 / 0x90
Dim 100% 00010101 21 / 0x15
Dim 75% 00010011 19 / 0x13
Dim 50% 00010010 18 / 0x12
Dim 25% 00010000 16 / 0x10
Interestingly, I think the Dim keys may have a confirmation that we have a good demod the codes on the bottom are missing the most significant bit, and when I look back at the scancode table in the datasheet, they make an interesting pattern the bottom two rows, right and left side values match up! If you take a look, Dim 100% is S1 , Dim 75% is S19 , Dim 50% is S8 , and Dim 25% is S20 . Cool! Since none of the other codes line up, I am willing to bet the most significant bit is a Combo indicator, and not part of the button (leaving 7 bits for the keycode). And even more interestingly, one of our scancodes ( Off , which is 0x94) shows up just below this table, in the examples. Over all, I think this tells us we have the right bits to look at for determining the scan code! Great news there!

Back to the modulation! So, armed with this knowledge, I was able to refactor my code to match the timings and understanding outlined by the datasheet and ensure things still work. The switch itself has a high degree of tolerance, so being wildly off frequency or a wildly wrong symbol rate may actually still work. It s hard to know if this is more or less correct, but matching documentation seems like a more stable foundation if nothing else. This code has been really reliable, and tends to work just as well as the remote from what I ve been able to determine. I ve been using incredibly low power to avoid any interference, and it s been very robust - a testament to the engineering that went into the outlet hardware, even though it cost less than of a lot of other switches! I have a lot of respect for the folks who built this device - it s incredibly simple, reliable and my guess is this thing will keep working even in some fairly harsh RF environments. The only downside is the fact the manufacturer used the same address for all their devices, rather than programming a unique address for each outlet and remote when the underlying WL116SC chip supports it. I m sure this was done to avoid complexity in assembly (e.g. pairing the remote and outlet, and having to keep those two items together during assembly), but it s still a bummer. I took apart the switch to see if I could dump an EEPROM and change the address in ROM, but the entire thing was potted in waterproof epoxy, which is a very nice feature if this was ever used outdoors. Not good news for tinkering, though!

Unsolved Mysteries At this point, even though I understand the protocol enough to control the device, it still feels like I hit a dead end in my understanding. I m not able to figure out how exactly the scancodes are implemented, and break them down into more specific parts. They are stable and based on the physical wiring of the remote, so I think I m going to leave it a magic number. I have what I was looking for, and these magic constants appear to be the right one to use, even if I did understand how to create the codes itself. This does leave us with a few bits we never resolved, which I ll memorialize below just to be sure I don t forget about them. Question #1: According to the datasheet there should be a preamble. Why do I not see one leading the first message? My hunch is that the trailing 0 at the end of the payload is actually just the preamble for the next message (always rendering the first message invalid?). This would let us claim there s an engineering reason why we are ignoring the weird bit, and also explain away something from the documentation. It s just weird that it wouldn t be present on the first message. This theory is mostly confirmed by measuring the timing and comparing it to the datasheet, but it s not exactly in line with the datasheet timings either (specifically, it s off by 200 s, which is kinda a lot for a system using 400 s timings). I think I could go either way on the last 0 being the preamble for the next message. It could be that the first message is technically invalid, or it could also be that this was not implemented or actively disabled by the vendor for this specific application / device. It s really hard to know without getting the source code for the WL116SC chip in this specific remote or the source in the outlet itself. Question #2: Why are some keycodes 8 bits and others 9 bits? I still have no idea why there sometimes 8 bits (for instance, On ) and other times there are 9 bits (for instance, Off ) in the 8 bit keycode field. I spent some time playing with the trailing zeros, when I try and send an Off with the most significant 8 bits (without the least significant / last 9th bit, which is a 0 ), it does not turn the tree off. If I send an On with 9 bits (an additional 0 after the least significant bit), it does work, but both On and Off work when I send 10, 11 or 12 bits padded with trailing zeros. I suspect my outlet will ignore data after the switch is done reading bits regardless of trailing zeros. The docs tell me there should only be 8 bits, but it won t work unless I send 9 bits for some commands. There s something fishy going on here, and the datasheet isn t exactly right either way. Question #3: How in the heck do those scancodes work? This one drove me nuts. I ve spent countless hours on trying to figure this out, including emailing the company that makes the WL116SC (they re really nice!), and even though they were super kind and generous with documentation and example source, I m still having a hard time lining up their documentation and examples with what I see from my remote. I think the manufacturer of my remote and switch has modified the protocol enough to where there s actually something different going on here. Bummer. I wound up in my place of last resort asking friends over Signal to try and see if they could find a pattern, as well as making multiple pleas to the twittersphere, to no avail (but thank you to Ben Hilburn, devnulling, Andreas Bombe and Larme for your repiles, help and advice!) I still don t understand how they assemble the scan code for instance, if you merely add, you won t know if a key press of 0x05 is 0x03 + 0x02 or if it s 0x01 + 0x04. On the other hand, treating it as two 4-bit integers won t work for 0x10 to 0x15 (since they need 5 bits to represent). It s also likely the most significant bit is a combo indicator, which only leaves 7 bits for the actual keypress data. Stuffing 10 bits of data into 7 bits is likely resulting in some really intricate bit work. On a last ditch whim, I tried to XOR the math into working, but some initial brute forcing to make the math work given the provided examples did not result in anything. It could be a bitpacked field that I don t understand, but I don t think I can make progress on that without inside knowledge and much more work. Here s the table containing the numbers I was working off of:
Keys Key Codes Scancode
S3 + S9 0x01 + 0x03 0x96
S6 + S12 0x07 + 0x09 0x94
S22 + S10 0x0D + 0x0F 0x3F
If anyone has thoughts on how these codes work, I d love to hear about it! Send me an email or a tweet or something - I m a bit stumped. There s some trick here that is being used to encode the combo key in a way that is decodeable. If it s actually not decodeable (which is a real possibility!), this may act as a unique button combo hash which allows the receiver to not actually determine which keys are pressed, but have a unique button that gets sent when a combo is used. I m not sure I know enough to have a theory as to which it may be.

Russ Allbery: Review: The Biggest Bluff

Review: The Biggest Bluff, by Maria Konnikova
Publisher: Penguin Press
Copyright: 2020
ISBN: 0-525-52263-8
Format: Kindle
Pages: 335
After a particularly unlucky year for her family, Maria Konnikova was reading about the balance between luck and control in life and discovered, to her surprise, that John von Neumann, one of the foundational thinkers of computer science, was fascinated by poker. He found most card games boring because they relied on luck. Poker, however, he thought was the perfect balance between luck and skill: enough skill to make its effect undeniable, but enough luck that one could not control the game fully regardless of skill. Konnikova decided on a research project: spend one year learning No Limit Texas Hold'em from one of the best poker players in the world, with a goal of competing in the World Series of Poker. She had studied the description-experience gap during her doctoral research in psychology and wanted to see if the experience of randomness in poker would teach her something the description of the randomness of life could not. Before starting this project, she didn't know the basic rules and had never watched a game. Erik Seidel agreed to mentor her, and The Biggest Bluff is her account of that experience. This book is simultaneously frustrating and fascinating in ways that I don't think can be untangled without making it a far different book. Fitting, I suppose, for a book about how our brains entangle luck and skill. First, if you're looking for a book about poker play, this is not one. Konnikova rarely talks about specific hands or tournaments in more detail than her overall trajectory. That was a disappointment. In the few places she does describe some of her betting decisions, analysis of the other players, and tournament strategy, her accounts are engrossing and suspenseful. I would have happily read a book chronicling her poker tournaments and the decisions she made, but this is not that book. What The Biggest Bluff is instead is a psychological and philosophical examination of the process of learning poker. Konnikova uses her experiences as launching points into philosophical digressions. Even the lessons that have limited surface utility outside of poker, such as learning to suppress body language to avoid giving away information, turn into digressions about interpersonal dynamics and personality types. There are interesting tidbits here, but I've read a lot of popular psychology and was more interested in the poker. My frequent reading experience was impatiently waiting for Konnikova to finish lecturing and get back to her narrative. What Konnikova does do though, at a level that I haven't seen before before in a book of this type, is be brutally honest about her mistakes and her learning process. And I do mean brutally: The book opens with her throwing up in a casino bathroom. (This is not a fun book to read if you don't like reading about stress reactions and medical problems. There aren't many of them, but they're... memorable.) Konnikova knows and can explain the psychological state she's trying to reach, but still finds it hard to do in the moment. Correctly reacting to probabilities, cutting losses, and neither being too over-confident or too scared is very hard. Most books of this type elide over the repeated failures in a sort of training montage, which makes the process look easier than it feels. Konnikova tries to realistically show the setbacks and failures, and I think succeeds. That relentless introspection and critical honesty is the best part of this book, but I think it's also behind the stream of consciousness digressions about psychology and philosophy. It's a true portrayal of how Konnikova makes sense of the world. A more polished and streamlined book about the poker would have been more dramatically engrossing, but it might have lost the deep examination of how she combines poker with her knowledge of psychology to change her thinking. The one place where I think that self-reflection may fall a bit short, which I want to mention because I thought it was a missed opportunity, is around knowledge of hand probabilities. Konnikova makes a point, early in the book, of not approaching poker through the memorization and mathematics route and instead trying to find a play style that focuses on analyzing the other players and controlling her own emotions. This is a good hook, but by the end of the book it's not entirely true. The point that I think she was trying to make is that her edge against other poker players at her same level comes more from psychology than from calculation of precise odds in rare situations. This is true. But by the time she reaches high levels of play, she is using statistical simulators, practice tools, and intensive study just like any professional poker player. There is a minimum level of pure knowledge and memorization required that cannot be avoided. It's clear from the few things she says about this that those tools became more interesting to her as she became better at poker, and I wish she would have dug more into why and how that happened. How much of her newfound ability to make decisions and stick to a plan comes from emotional changes, and how much from that background store of confident knowledge? Or maybe those are different ways of looking at the same change? I did appreciate Konnikova's explicit acknowledgment at the end of the book that poker did not, in the end, provide some deep insight into the balance of luck and skill in real life. Learning poker instead gave Konnikova more personal ability to make a plan for the things that she can control and let go of the things she can't. I'm glad that worked for her, but since reading this book I have noticed former poker players who think life is more knowable than it is. Poker combines random chance with psychological play against other people, but it does so in a way and to an extent that is quantifiable. You can make the correct play and still lose a hand, but you can also know when this has happened. When you're used to analyzing the world through that frame and real life fails to provide that certainty, it's tempting to impose it anyway and insist your simplified models are more accurate because they're more comprehensible. But poker is a game, not a model; being more predictable and more constrained than real life is part of what makes it fun. The skill that Konnikova learned from it has a potential downside that she doesn't talk about. I'm not sure how to sum up this book. Konnikova's internal analysis and honesty is truly admirable and illuminating, but it left me wanting to read a different book that was more focused on poker narration. I know there are lots of those books out there, but I doubt they would be written with Konnikova's self-awareness and lack of ego. However, they would probably also lack the moments that made me cringe or that were deeply uncomfortable to read. My feelings are mixed. But if you want popular psychology wrapped around a deeply honest account of the process of learning poker, I suspect this book is one of a kind. Rating: 7 out of 10

22 December 2020

Andrej Shadura: Vendoring Rust dependencies for a Debian derivative

Recently, I needed to package a Rust crate libslirp for a Apertis, a Debian derivative. libslirp is used by the newly release UML backend of debos, our Debian image build tool. Unfortunately, this crate hasn t yet been properly packaged for Debian proper, so I could not simply pull the packaging from Debian. Even worse, its build dependencies haven t all been packaged yet. Most importantly, I have only uploaded zbus to Debian today, and at that time none of its dependencies were in Debian either. Another issue with this were that each crate is packaged for Debian as a separate package, making the process a bit more tricky since I d need to import all of the crates into Apertis separately. Doing that takes time and is further complicated by the CI loop we re using which requires the full build process to complete for a package before a pipeline a dependent package can run. Having all that considered, I took a shortcut: I vendored all the build dependencies with the package itself, and here s how. I started with debcargo-conf, where the not-yet-complete packaging lives. With debcargo, I generated the source package as if I were to upload to Debian. Since I ll be vendoring the dependencies, I have removed most of the Rust dependencies and replaced them with those extra dependencies needed by those crates. So, I replaced this:
Build-Depends: debhelper (>= 12),
 dh-cargo (>= 24),
 cargo:native,
 rustc:native,
 libstd-rust-dev,
 librust-enumflags2-0.6+default-dev (>= 0.6.4-~~),
 librust-ipnetwork-0.17+default-dev,
 librust-lazy-static-1+default-dev (>= 1.4-~~),
 librust-libc-0.2+default-dev,
 librust-libslirp-sys-4+default-dev (>= 4.2.0-~~),
 librust-libsystemd-0.2+default-dev,
 librust-mio-0.6+default-dev,
 librust-mio-extras-2+default-dev (>= 2.0.5-~~),
 librust-nix-0.17+default-dev,
 librust-slab-0.4+default-dev,
 librust-structopt-0.3+default-dev,
 librust-url-2+default-dev (>= 2.1-~~),
 librust-zbus-1+default-dev,
 librust-zvariant-2+default-dev
By this:
Build-Depends: debhelper (>= 12),
 dh-cargo (>= 17),
 cargo:native,
 rustc:native,
 libstd-rust-dev,
 pkg-config,
 libglib2.0-dev,
 libslirp-dev
I wanted to keep the delta to the future Debian package at a bare minimum, so I kept the original dh-cargo build process, only making sure the vendored dependencies are found in the right place. The cleanest approach in my view was to use the fact that the 3.0 (quilt) source format allows multiple original tarballs: the main tarball would be the same orig tarball as in Debian, and an extra tarball would contain the vendored code. When we sync with Debian, the extra tarball would be dropped, just as the adjustments to the debian/rules. To generate the vendor tarball, I added this to the rules file:
include /usr/share/dpkg/pkg-info.mk
vendor:
        -[ -d vendor ] && rm -rf vendor
        rm -rf Cargo.lock
        cargo vendor
        tar Jcf ../$(DEB_SOURCE)_$(DEB_VERSION_UPSTREAM).orig-vendor.tar.xz vendor/
.PHONY: vendor
This target, being run manually and not as part of the build process, download the latest dependencies satisfying the requirements in Cargo.toml and rolls them into a vendor tarball. Now, to make dh-cargo use the vendored code, we need to link it into the right place. dh-cargo creates a Cargo registry tree in a subdirectly under debian/ where it symlinks crates from the Rust development packages. This extra code makes sure the vendored dependencies are available there too:
override_dh_auto_configure:
        dh_auto_configure $@
        for d in vendor/* ; \
        do \
                [ -L debian/cargo_registry/$$(basename $$d) ] && rm debian/cargo_registry/$$(basename $$d) ; \
                ln -rs $$d debian/cargo_registry ; \
        done
An important thing here is that if the build happens on a non-clean system, dh-cargo may have symlinked a system crate already installed so we need to remove that symlink and replace it by ours. We build packages in clean chroots, so this is not that important, otherwise for reproducibility reasons it would be better to create the registry directory from scratch. The final package can be found at the Apertis GitLab.

21 December 2020

Chris Lamb: Interview: Reproducibility and the future of free software

I recently took part in an interview with Vladimir Bejdo, an intern at the Software Freedom Conservancy, in order to talk about the Reproducible Builds project, my own participation in software freedom, the importance of reproducibility in software development and to have a brief discussion on the issues facing free software as a whole:
VB: To start off with, it might be useful to first ask you this how would you relate the importance of reproducibility to a user who is non-technical? CL: I sometimes use the analogy of the food supply chain to quickly relate our work to non-technical audiences. The multiple stages of how our food reaches our plates today (such as seeding, harvesting, picking, transportation, packaging, etc.) can loosely translate to how software actually ends up on our computers, particularly in the way that if any of the steps in the multi-stage food supply chain has an issue then it quickly becomes a serious problem. For example, even if we could guarantee that only the most wholesome apples were picked in our orchards, if they became tainted on the way to the supermarket it will be a real problem for us at the end of the day. We may not even be able to even tell by simply inspecting our Pink Ladies or Honeycrisps, and washing them thoroughly under the tap may not be enough either. In an ideal world, we would be able to personally inspect the provenance of our food at all of the stages of manufacturing and transportation. But at some point, we must place our trust in the process and in brands, as well as various regulatory bodies to ensure that potential problems in our food are minimized, possibly even paying a time/effort premium by growing our own or buying direct from local markets in order to minimize the number of steps, etc. However, when we use free software we can do better: Reproducible builds are a set of software development practices, ideas and tools that create an independently-verifiable path all the way from the original source code to what actually runs on our machines. Reproducible builds can reveal the injection of back-doors introduced by the hacking of developers own computers, build servers and package repositories, and also expose where volunteers or companies have been coerced into making changes via blackmail, court order, and so on. With reproducible builds, there is no longer any need to trust any particular source of authority. In the same way that, say, a Mr Smith might check that his calculator is giving him the right answer to 2+2=4 by asking enough of his friends to check theirs too, users and developers of a reproducible build can verify the software they are using by creating a collective consensus instead.
The full interview can be found on the Conservancy webpages.

Sven Hoexter: docker buildx sugar - dumping results to disk

The latest docker 20.10.x release unlocks the buildx subcommands which allow for some sugar, like building something in a container and dumping the result to your local directory in one command. Dockerfile
FROM docker-registry.mycorp.com/debian-node:lts as builder
USER service
COPY . /opt/service
RUN cd /opt/service; npm install; npm run build
FROM scratch as dist
COPY --from=builder /opt/service/dist /
build with
docker buildx build --target=dist --output type=local,dest=$(pwd)/pages/ .
Here we build a page, copy the result with all assets from the /opt/service/dist directory to an empty image and dump it into the local pages directory.

20 December 2020

Russ Allbery: Review: Can't Even

Review: Can't Even, by Anne Helen Petersen
Publisher: Houghton Mifflin
Copyright: 2020
ISBN: 0-358-31659-6
Format: Kindle
Pages: 230
Like many other people, I first became aware of Anne Helen Petersen's journalism when her Buzzfeed article "How Millennials Became the Burnout Generation" went viral. Can't Even is the much-awaited (at least by me) book-length expansion of that thesis: The United States is, as a society, burning out, and that burnout is falling on millennials the hardest. We're not recognizing the symptoms because we think burnout looks like something dramatic and flashy. But for most people burnout looks less like a nervous breakdown and more like constant background anxiety and lack of energy.
Laura, who lives in Chicago and works as a special ed teacher, never wants to see her friends, or date, or cook she's so tired, she just wants to melt into the couch. "But then I can't focus on what I'm watching, and end up unfocused again, and not completely relaxing," she explained. "Here I am telling you I don't even relax right! I feel bad about feeling bad! But by the time I have leisure time, I just want to be alone!"
Petersen explores this idea across childhood, education, work, family, and parenting, but the core of her thesis is the precise opposite of the pervasive myth that millennials are entitled and lazy (a persistent generational critique that Petersen points out was also leveled at their Baby Boomer parents in the 1960s and 1970s). Millennials aren't slackers; they're workaholics from childhood, for whom everything has become a hustle and a second (or third or fourth) job. The struggle with "adulting" is a symptom of the burnout on the other side of exhaustion, the mental failures that happen when you've forced yourself to keep going on empty so many times that it's left lingering damage. Petersen is a synthesizing writer who draws together the threads of other books rather than going deep on a novel concept, so if you've been reading about work, psychology, stress, and productivity, many of the ideas here will be familiar. But she's been reading the same authors that I've been reading (Tressie McMillan Cottom, Emily Guendelsberger, Brigid Schulte, and even Cal Newport), and this was the book that helped me pull those analyses together into a coherent picture. That picture starts with the shift of risk in the 1970s and 1980s from previously stable corporations with long-lasting jobs and retirement pensions onto individual employees. The corresponding rise in precarity and therefore fear led to a concerted effort to re-establish a feeling of control. Baby Boomers doubled down on personal responsibility and personal capability, replacing unstructured childhood for their kids with planned activities and academic achievement. That generation, in turn, internalized the need for constant improvement, constant grading, and constant achievement, accepting an implied bargain that if they worked very hard, got good grades, got into good schools, and got a good degree, it would pay off in a good life and financial security. They were betrayed. The payoff never happened; many millennials graduated into the Great Recession and the worst economy since World War II. In response, millennials doubled down on the only path to success they were taught. They took on more debt, got more education, moved back in with their parents to cut expenses, and tried even harder.
Even after watching our parents get shut out, fall from, or simply struggle anxiously to maintain the American Dream, we didn't reject it. We tried to work harder, and better, more efficiently, with more credentials, to achieve it.
Once one has this framework in mind, it's startling how pervasive the "just try harder" message is and how deeply we've internalized it. It is at the center of the time management literature: Getting Things Done focuses almost entirely on individual efficiency. Later time management work has become more aware of the importance of pruning the to-do list and doing fewer things, but addresses that through techniques for individual prioritization. Cal Newport is more aware than most that constant busyness and multitasking interacts poorly with the human brain, and has taken a few tentative steps towards treating the problem as systemic rather than individual, but his focus is still primarily on individual choices. Even when tackling a problem that is clearly societal, such as the monetization of fear and outrage on social media, the solutions are all individual: recognize that those platforms are bad for you, make an individual determination that your attention is being exploited, and quit social media through your personal force of will. And this isn't just productivity systems. Most of public discussion of environmentalism in the United States is about personal energy consumption, your individual carbon footprint, household recycling, and whether you personally should eat meat. Discussions of monopoly and monopsony become debates over whether you personally should buy from Amazon. Concerns about personal privacy turn into advocacy for using an ad blocker or shaming people for using Google products. Articles about the growth of right-wing extremism become exhortations to take responsibility for the right-wing extremist in your life and argue them out of their beliefs over the dinner table. Every major systemic issue facing society becomes yet another personal obligation, another place we are failing as individuals, something else that requires trying harder, learning more, caring more, doing more. This advice is well-meaning (mostly; sometimes it is an intentional and cynical diversion), and can even be effective with specific problems. But it's also a trap. If you're feeling miserable, you just haven't found the right combination of time-block scheduling, kanban, and bullet journaling yet. If you're upset at corporate greed and the destruction of the environment, the change starts with you and your household. The solution is in your personal hands; you just have try a little harder, work a little harder, make better decisions, and spend money more ethically (generally by buying more expensive products). And therefore, when we're already burned out, every topic becomes another failure, increasing our already excessive guilt and anxiety. Believing that we're in control, even when we're not, does have psychological value. That's part of what makes it such a beguiling trap. While drafting this review, I listened to Ezra Klein's interview with Robert Sapolsky on poverty and stress, and one of the points he made is that, when mildly or moderately bad things happen, believing you have control is empowering. It lets you recast the setback as a larger disaster that you were able to prevent and avoid a sense of futility. But when something major goes wrong, believing you have control is actively harmful to your mental health. The tragedy is now also a personal failure, leading to guilt and internal recrimination on top of the effects of the tragedy itself. This is why often the most comforting thing we can say to someone else after a personal disaster is "there's nothing you could have done." Believing we can improve our lives if we just try a little harder does work, until it doesn't. And because it does work for smaller things, it's hard to abandon; in the short term, believing we're at the mercy of forces outside our control feels even worse. So we double down on self-improvement, giving ourselves even more things to attempt to do and thus burning out even more. Petersen is having none of this, and her anger is both satisfying and clarifying.
In writing that article, and this book, I haven't cured anyone's burnout, including my own. But one thing did become incredibly clear. This isn't a personal problem. It's a societal one and it will not be cured by productivity apps, or a bullet journal, or face mask skin treatments, or overnight fucking oats. We gravitate toward those personal cures because they seem tenable, and promise that our lives can be recentered, and regrounded, with just a bit more discipline, a new app, a better email organization strategy, or a new approach to meal planning. But these are all merely Band-Aids on an open wound. They might temporarily stop the bleeding, but when they fall off, and we fail at our new-found discipline, we just feel worse.
Structurally, Can't Even is half summaries of other books and essays put into this overall structure and half short profiles and quotes from millennials that illustrate her point. This is Petersen's typical journalistic style if you're familiar with her other work. It gains a lot from the voices of individuals, but it can also feel like argument from anecdote. If there's a epistemic flaw in this book, it's that Petersen defends her arguments more with examples than with scientific study. I've read enough of the other books she cites, many of which do go into the underlying studies and statistics, to know that her argument is well-grounded, but I think Can't Even works better as a roadmap and synthesis than as a primary source of convincing data. The other flaw that I'll mention is that although Petersen tries very hard to incorporate poorer and non-white millennials, I don't think the effort was successful, and I'm not sure it was possible within the structure of this book. She frequently makes a statement that's accurate and insightful for millennials from white, middle-class families, acknowledges that it doesn't entirely apply to, for example, racial minorities, and then moves on without truly reconciling those two perspectives. I think this is a deep structural problem: One's experience of American life is very different depending on race and class, and the phenomenon that Petersen is speaking to is to an extent specific to those social classes who had a more comfortable and relaxing life and are losing it. One way to see the story of the modern economy is that white people are becoming as precarious as everyone else already was, and are reacting by making the lives of non-white people yet more miserable. Petersen is accurately pointing to significant changes in relationships with employers, productivity, family, and the ideology of individualism, but experiencing that as a change is more applicable to white people than non-white people. That means there are, in a way, two books here: one about the slow collapse of the white middle class into constant burnout, and a different book about the much longer-standing burnout of being non-white in the United States and our systemic failure to address the causes of it. Petersen tries to gesture at the second book, but she's not the person to write it and those two books cannot comfortably live between the same covers. The gestures therefore feel awkward and forced, and while the discomfort itself serves some purpose, it lacks the insight that Petersen brings to the rest of the book. Those critiques aside, I found Can't Even immensely clarifying. It's the first book that explained to me in a way I understood what's so demoralizing and harmful about Instagram and its allure of cosplaying as a successful person. It helped me understand how productivity and individual political choices fit into a system that emphasizes individual action as an excuse to not address collective problems. And it also gave me a strange form of hope, because if something can't go on forever, it will, at some point, stop.
Millennials have been denigrated and mischaracterized, blamed for struggling in situations that set us up to fail. But if we have the endurance and aptitude and wherewithal to work ourselves this deeply into the ground, we also have the strength to fight. We have little savings and less stability. Our anger is barely contained. We're a pile of ashes smoldering, a bad memory of our best selves. Underestimate us at your peril: We have so little left to lose.
Nothing will change without individual people making different decisions and taking different actions than they are today. But we have gone much too far down the path of individual, atomized actions that may produce feelings of personal virtue but that are a path to ineffectiveness and burnout when faced with systemic problems. We need to make different choices, yes, but choices towards solidarity and movement politics rather than personal optimization. There is a backlash coming. If we let it ground itself in personal grievance, it could turn ugly and take a racist and nationalist direction. But that's not, by in large, what millennials have done, and that makes me optimistic. If we embrace the energy of that backlash and help shape it to be more inclusive, just, and fair, we can rediscover the effectiveness of collective solutions for collective problems. Rating: 8 out of 10

18 December 2020

Russell Coker: Thinkpad Storage Problem

For a while I ve had a problem with my Thinkpad X1 Carbon Gen 1 [1] where storage pauses for 30 seconds, it s become more common recently and unfortunately everything seems to depend on storage (ideally a web browser playing a video from the Internet could keep doing so with no disk access, but that s not what happens).
echo 3 > /sys/block/sda/device/timeout
I ve put the above in /etc/rc.local which should make it only take 3 seconds to recover from storage problems instead of 30 seconds, the problem hasn t recurred since that change so I don t know if it works as desired. The Thinkpad is running BTRFS and no data is reported as being lost, so it s not a problem to use it at this time. The bottom of this post has an example of the errors I m getting. A friend said that his searches for such errors shows people claiming that it s a SATA controller or cable problem, which for a SATA device bolted directly to the motherboard means it s likely to be a motherboard problem (IE more expensive to fix than the $289 I paid for the laptop almost 3 years ago). A Lenovo forum discussion says that the X1 Carbon Gen1 uses an unusual connector that s not M.2 SATA and not mSATA [2]. This means that a replacement is going to be expensive, $100US for a replacement from eBay when a M.2 SATA or NVMe device would cost about $50AU from my local store. It also means that I can t use a replacement for anything else. If my laptop took a regular NVMe I d buy one and test it out, if it didn t solve the problem a spare NVMe device is always handy to have. But I m not interested in spending $100US for a part that may turn out to be useless. I bought the laptop expecting that there was nothing I could fix inside it. While it is theoretically possible to upgrade the CPU etc in a laptop most people find that the effort and expense makes it better to just replace the entire laptop. With the ultra light category of laptops the RAM is soldered on the motherboard and there are even less options for replacing things. So while it s annoying that Lenovo didn t use a standard interface for this device (they did so for other models in the range) it s not a situation that I hadn t expected. Now the question is what to do next. The Thinkpad has a SD socket and micro SD cards are getting large capacities nowadays. If it won t boot from a SD card I could boot from USB and then run from SD. So even if the internal SATA device fails entirely it should still be useful. I wonder if I could find a Thinkpad with a similar problem going cheap as I m pretty sure that a Windows user couldn t make any storage device usable the way I can with Linux. I ve looked at prices on auction sites but haven t seen a Thinkpad X1 Carbon going for anywhere near the price I got this one. The nearest I ve seen is around $350 for a laptop with significant damage. While I am a little unhappy at Lenovo s choice of interface, I ve definitely got a lot more than $289 of value out of this laptop. So I m not really complaining.
[315041.837612] ata1.00: status:   DRDY  
[315041.837613] ata1.00: failed command: WRITE FPDMA QUEUED
[315041.837616] ata1.00: cmd 61/20:48:28:1e:3e/00:00:00:00:00/40 tag 9 ncq dma 
16384 out
                         res 40/00:01:00:00:00/00:00:00:00:00/e0 Emask 0x4 
(timeout)                
[315041.837617] ata1.00: status:   DRDY  
[315041.837618] ata1.00: failed command: READ FPDMA QUEUED
[315041.837621] ata1.00: cmd 60/08:50:e0:26:84/00:00:00:00:00/40 tag 10 ncq 
dma 4096 in
                         res 40/00:01:00:00:00/00:00:00:00:00/e0 Emask 0x4 
(timeout)                
[315041.837622] ata1.00: status:   DRDY  
[315041.837625] ata1: hard resetting link
[315042.151781] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[315042.163368] ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) 
succeeded
[315042.163370] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) 
filtered out
[315042.163372] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered 
out
[315042.183332] ata1.00: ACPI cmd ef/02:00:00:00:00:a0 (SET FEATURES) 
succeeded
[315042.183334] ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) 
filtered out
[315042.183336] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered 
out
[315042.193332] ata1.00: configured for UDMA/133
[315042.193789] sd 0:0:0:0: [sda] tag#10 FAILED Result: hostbyte=DID_OK 
driverbyte=DRIVER_SENSE
[315042.193791] sd 0:0:0:0: [sda] tag#10 Sense Key : Illegal Request [current] 
[315042.193793] sd 0:0:0:0: [sda] tag#10 Add. Sense: Unaligned write command
[315042.193795] sd 0:0:0:0: [sda] tag#10 CDB: Read(10) 28 00 00 84 26 e0 00 00 
08 00
[315042.193797] print_req_error: I/O error, dev sda, sector 8660704
[315042.193810] ata1: EH complete

17 December 2020

Dirk Eddelbuettel: RcppEigen 0.3.3.9.1: New upstream

A new release 0.3.3.9.1 of RcppEigen arrived on CRAN today (and just went to Debian too) bringing support for Eigen 3.3.9 to R. Eigen is a C++ template library for linear algebra: matrices, vectors, numerical solvers, and related algorithms. The release started with a gentle nudge from the Stan team to package Eigen 3.3.8. And once I got around to it and sat down, Eigen 3.3.9 had just been released the previous day. The freshly prepared release 0.3.3.9.0 was then announced on Twitter (and the rcpp-devel mailing list) along with a request for testing alongside the usual extended reverse dependency checking we do (just like CRAN asks us to). And it turns out that Eigen 3.3.9 required an update to StanHeaders, a CRAN package that is itself quite widely used. This took a couple of days and its round of testing, but Ben Goodrich shipped it to CRAN last night, so we were able to ship RcppEigen in turn today. So this release once again bring few required changes to Eigen (see this directory for details). As we said for the previous three releases:
One additional and recent change was the accomodation of a recent CRAN Policy change to not allow gcc or clang to mess with diagnostic messages. A word of caution: this may make your compilation of packages using RcppEigen very noisy so consider adding -Wno-ignored-attributes to the compiler flags added in your ~/.R/Makevars.
I actually find this requirement rather annoying. Eigen is then only usable if you set, say,
-Wno-deprecated-declarations -Wno-parentheses -Wno-ignored-attributes -Wno-unused-function
as g++ options in ~/.R/Makevars. But CRAN makes the rules. Maybe if a few of us gently and politely nudge them they may relent one day. One can only hope. The complete NEWS file entry follows.

Changes in RcppEigen version 0.3.3.9.1 (2020-12-17)
  • Upgraded to Eigen 3.3.9 (Dirk in #92 fixing #91).
  • Added GitHub Actions CI using run.sh from r-ci (Dirk)

Courtesy of CRANberries, there is also a diffstat report for the most recent release. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

16 December 2020

Jonathan McDowell: DeskPi Pro + 8GB Pi 4

DeskPi Pro Raspberry Pi case Despite having worked on a number of ARM platforms I ve never actually had an ARM based development box at home. I have a Raspberry Pi B Classic (the original 256MB rev 0002 variant) a coworker gave me some years ago, but it s not what you d choose for a build machine and generally gets used as a self contained TFTP/console server for hooking up to devices under test. Mostly I ve been able to do kernel development with the cross compilers already built as part of Debian, and either use pre-built images or Debian directly when I need userland pieces. At a previous job I had a Marvell MACCHIATObin available to me, which works out as a nice platform - quad core A72 @ 2GHz with 16GB RAM, proper SATA and a PCIe slot. However they re still a bit pricey for a casual home machine. I really like the look of the HoneyComb LX2 - 16 A72 cores, up to 64GB RAM - but it s even more expensive. So when I saw the existence of the 8GB Raspberry Pi 4 I was interested. Firstly, the Pi 4 is a proper 64 bit device (my existing Pi B is ARMv6 which means it needs to run Raspbian instead of native Debian armhf), capable of running an upstream kernel and unmodified Debian userspace. Secondly the Pi 4 has a USB 3 controller sitting on a PCIe bus rather than just the limited SoC USB 2 controller. It s not SATA, but it s still a fairly decent method of attaching some storage that s faster/more reliable than an SD card. Finally 8GB RAM is starting to get to a decent amount - for a headless build box 4GB is probably generally enough, but I wanted some headroom. The Pi comes as a bare board, so I needed a case. Ideally I wanted something self contained that could take the Pi, provide a USB/SATA adaptor and take the drive too. I came across the pre-order for the DeskPi Pro, decided it was the sort of thing I was after, and ordered one towards the end of September. It finally arrived at the start of December, at which point I got round to ordering a Pi 4 from CPC. Total cost ~ 120 for the case + Pi.

The Bad First, let s get the bad parts out of the way. Broken USB port (right) I managed to break a USB port on the Desk Pi. It has a pair of forward facing ports, I plugged my wireless keyboard dongle into it and when trying to remove it the solid spacer bit in the socket broke off. I ve never had this happen to me before and I ve been using USB devices for 20 years, so I m putting the blame on a shoddy socket. The first drive I tried was an old Crucial M500 mSATA device. I have an adaptor that makes it look like a normal 2.5 drive so I used that. Unfortunately it resulted in a boot loop; the Pi would boot its initial firmware, try to talk to the drive and then reboot before even loading Linux. The DeskPi Pro comes with an m2 adaptor and I had a spare m2 drive, so I tried that and it all worked fine. This might just be power issues, but it was an unfortunate experience especially after the USB port had broken off. (Given I ended up using an M.2 drive another case option would have been the Argon ONE M.2, which is a bit more compact.)

The Annoying DeskPi Pro without rear bezel The case is a little snug; I was worried I was going to damage things as I slid it in. Additionally the construction process is a little involved. There s a good set of instructions, but there are a lot of pieces and screws involved. This includes a couple of FFC cables to join things up. I think this is because they ve attempted to make a compact case rather than allowing a little extra room, and it does have the advantage that once assembled it feels robust without anything loose in it. DeskPi Pro with rear bezel and USB3 dongle I hate the need for an external USB3 dongle to bridge from the Pi to the USB/SATA adaptor. All the cases I ve seen with an internal drive bay have to do this, because the USB3 isn t brought out internally by the Pi, but it just looks ugly to me. It s hidden at the back, but meh. Fan control is via a USB/serial device, which is fine, but it attaches to the USB C power port which defaults to being a USB peripheral. Raspbian based kernels support device tree overlays which allows easy reconfiguration to host mode, but for a Debian based system I ended up rolling my own dtb file. I changed
#include "bcm283x-rpi-usb-peripheral.dtsi"
to
#include "bcm283x-rpi-usb-host.dtsi"
in arch/arm/boot/dts/bcm2711-rpi-4-b.dts and then I did:
cpp -nostdinc -I include -I arch -undef -x assembler-with-cpp \
    arch/arm/boot/dts/bcm2711-rpi-4-b.dts > rpi4.preprocessed
dtc -I dts -O dtb rpi4.preprocessed -o bcm2711-rpi-4-b.dtb
and the resulting bcm2711-rpi-4-b.dtb file replaced the one in /boot/firmware. This isn t a necessary step if you don t want to use the cooling fan in the case, or the front USB ports, and it s not really anyone s fault, but it was an annoying extra step to have to figure out. The DeskPi came with a microSD card that was supposed to have RaspiOS already on it. It didn t, it was blank. In my case that was fine, because I wanted to use Debian, but it was a minor niggle.

The Good I used Gunnar s pre-built Pi Debian image and it Just Worked; I dd d it to the microSD as instructed and the Pi 4 came up with working wifi, video and USB enabling me to get it configured for my network. I did an apt upgrade and got updated to the Buster 10.7 release, as well as the latest 5.9 backport kernel, and everything came back without effort after a reboot. It s lovely to be able to run Debian on this device without having to futz around with self-compiled kernels. The DeskPi makes a lot of effort to route things externally. The SD slot is brought out to the front, making it easy to fiddle with the card contents without having to open the case to replace it. All the important ports are brought out to the back either through orientation of the Pi, or extenders in the case. That means the built in Pi USB ports, the HDMI sockets (conveniently converted to full size internally), an audio jack and a USB-C power port. The aforementioned USB3 dongle for the bridge to the drive is the only external thing that s annoying. Thermally things seem good too. I haven t done a full torture test yet, but with the fan off the system is sitting at about 40 C while fairly idle. Some loops in bash that push load up to above 2 get the temperature up to 46 C or so, and turning the fan on brings it down to 40 C again. It s audible, but quieter than my laptop and not annoying. I liked the way the case came with everything I needed other than the Pi 4 and a suitable disk drive. There was an included PSU (a proper USB-C PD device, UK plug), the heatsink/fan is there, the USB/SATA converter is there and even an SD card is provided (though that s just because I had a pre-order). Speaking of the SD, I only needed it for initial setup. Recent Pi 4 bootloaders are capable of booting directly from USB mass storage devices. So I upgraded using the RPi EEPROM Recovery image (which just needs extracted to the SD FAT partition, no need for anything complicated - boot with it and the screen goes all green and you know it s ok), then created a FAT partition at the start of the drive for the kernel / bootloader config and a regular EXT4 partition for root. Copies everything over, updated paths, took out the SD and it all just works happily.

Summary My main complaint is the broken USB port, which feels like the result of a cheap connector. For a front facing port expected to see more use than the rear ports I think there s a reasonable expectation of robustness. However I m an early adopter and maybe future runs will be better. Other than that I m pretty happy. The case is exactly the sort of thing I wanted; I was looking for something that would turn the Pi into a box that can sit on my desk on the network and that I don t have to worry about knocking wires out of or lots of cables hooking bits up. Everything being included made it very convenient to get up and running. I still haven t poked the Pi that hard, but first impressions are looking good for it being a trouble free ARM64 dev box in the corner, until I can justify a HoneyComb.

Dirk Eddelbuettel: RcppAnnoy 0.0.18: Tweaks

annoy image A new maintenance release, now at version 0.0.18, of RcppAnnoy is now on CRAN. RcppAnnoy is the Rcpp-based R integration of the nifty Annoy library by Erik Bernhardsson. Annoy is a small and lightweight C++ template header library for very fast approximate nearest neighbours originally developed to drive the famous Spotify music discovery algorithm. This release is follow-up to release 0.0.17 which was made four weeks ago, and which brought the new upstream release 1.17 of Annoy. We now have an updated upstream with a PR by Aaron aiming for improved control of RNG seeding to ensure better reproducibility, along with extended tests and new helpers for RcppAnnoy and Annoy version numbers. The release was once again coordinated with Annoy and James whose BiocNeighbors and uwot packages both consume the Annoy header library shipped here. Detailed changes follow below.

Changes in version 0.0.18 (2020-12-15)
  • Small tweaks to threading policy header defines (Dirk closing #65)
  • Vignette code is again compiled during testing (Aaron Lum and Dirk in #66 addressing #64)
  • Upstream code (with Aaron's PR) was synchronized once more (Dirk in #67)
  • A new helper function was added to report the Annoy version (Aaron in #68)

Courtesy of my CRANberries, there is also a diffstat report for this release. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

13 December 2020

Russ Allbery: Review: Because Internet

Review: Because Internet, by Gretchen McCulloch
Publisher: Riverhead Books
Copyright: 2019
ISBN: 0-7352-1095-0
Format: Kindle
Pages: 276
If you're familiar with linguistics as a field of scientific study (as opposed to the tool-based fields of grammar or writing advice), you'll be familiar with the dichotomy between written and spoken language. We may spend more time thinking about written language since it is central to most types of education and carries much of the intellectual and social weight of society. Linguists, however, see spoken language as more fundamental, since speech is wired into our brains and universal in human societies. Written language is a recent and somewhat artificial invention. One also learns from linguistics that spoken language does not follow many of the rules of written language that we painstakingly memorized in school. In casual speech, people split infinitives, speak in partial and run-on sentences, ignore nit-picking pronoun case rules, and rarely notice or care about the difference between less and fewer. Spoken language does have rules, but they're more subtle and nuanced than the grammar rules we learn in school. (I think the real fun of linguistics is separating the rules that native speakers follow effortlessly from the artificial rules used as education markers.) This is, in part, because nearly all spoken language is informal, whereas nearly all written language is formal. Enter the Internet, and enter this book. For the first time in human history we have both an explosion of informal writing and easy availability of that writing to linguists for study. Informal writing is not entirely new, of course. We've had personal letters for nearly as long as we've had writing, not to mention private notes, diaries, and other writing intended for tiny audiences. But consider who wrote private letters and, on top of that historical filter, whose private letters were preserved for linguistic research. Until relatively recently, only the upper classes were literate and had access to the infrastructure to write and send letters. Someone's letters or private notes were unlikely to be preserved unless they were someone famous and important, and thus often well-educated and more likely to take a more formal tone in writing. If you compare this to the Internet-driven blizzard of work and personal email, SMS conversations, chatrooms, and social media posts, the difference is obvious in both volume and level of informality. We're all on the Internet, we all read and write with a frequency that would be staggering to the average person from even fifty years ago, and while one may take a bit of additional care with a tricky email to one's manager, the SMS message to one's friend is as informal of a use of language as a conversation over coffee. Gretchen McCulloch is a professional linguist and Because Internet is about exactly this phenomenon: the new conventions of informal writing, how it has changed and evolved, and the new subtleties and shortcuts we've invented to make written communication easier. That goes beyond words and grammar to encompass punctuation, emoji and emoticons, memes and reaction gifs, and even the subtleties of timing, whitespace, and the construction of virtual places via our choices in how and where we write. This topic is my catnip, so it's not surprising I love this book. I've been heavily involved with online communities that communicate in writing since 1993 (making me, in McCulloch's classification, an Old Internet Person; each wave of introduction to the Internet has its own conventions that can be in conflict with later waves). I've now spent more than half my life carrying out most of my social activity and most of my closest friendships primarily in writing, so I found a lot of satisfaction in a linguistic study that takes that seriously rather than treating it as a curiosity. But, even better, I was amazed at how much I didn't know, in part because I am from a specific wave. I have a deep intuition for the Usenet conventions, but not as good of an understanding of the ones from AIM and LiveJournal one wave later (the Full Internet People). And I had a lot to learn about the conventions of the Instagram and Snapchat cluster (the Post Internet People, who have never known life without the Internet). One of the things that struck me while reading this book is how most of the language innovations that McCulloch describes are addressing the old complaint that written communication is inferior to face-to-face conversation because it lacks emotional nuance. My knee-jerk reply is that, no, written communication is full of emotional nuance and the complainer is just bad at reading it, but that's somewhat unfair. A better statement of the problem is that there is not a standardized language for emotional nuance in written communication, in part because it's so new in human history. Most humans are extremely good at reading facial expressions and body language for emotional cues, and those physical expressions are largely subconscious, reliable, and similar among different people (particularly within a culture; one can get in trouble with body language variations across cultures). This is not true of writing. With friends I've talked to over chat for twenty-five years, I can read volumes about their emotional state in a couple of short lines of text. But with strangers, despite decades of Internet communications, I will still misread cues and misinterpret simple intentions. The other standard response to this complaint is that it is possible to put extensive emotional nuance into formal writing. Just get better at writing! This is true, but unhelpful. There's a reason why we give book contracts to people who are very good at investing formal writing with emotional nuance. It's difficult, time-consuming, and requires a great deal of practice. That may be appropriate for formal, paid writing, but it won't do for informal writing, which by definition needs to be as effortless as possible. It's therefore unsurprising that once millions of people were using the Internet regularly for informal writing, they started adding new mechanisms, shortcuts, and conventions for emotional nuance. The standardization is growing, but conventions still vary widely between waves of Internet users. One of the most fascinating parts of this book for me was McCulloch's explanation of why periods (and, to a lesser extent, capital letters) in short chat messages are perceived by younger users as harsh or passive-aggressive. I still have the formal writing mindset of treating proper capitalization and punctuation as a point of pride, but McCulloch makes an excellent argument for letting go of my biases and understanding how and why language is changing. The realization I had while reading this is that many of the changes that look like sloppiness or laziness to someone trained in formal writing have the effect of giving language greater dynamic range. If one always uses periods uniformly, the period becomes meaningless except as a sentence boundary (which is redundant with newlines in most short informal chat messages). If one normally doesn't use it, and then suddenly starts using it, the period can carry semantic weight. It can convey a snippy tone of voice, a note of annoyance, or other subtle shades of meaning. I still use periods in most of my Slack messages because habits are hard to break, but I'm remembering to leave them off some of the time and paying more attention to what emotional weight they're carrying when present. Because Internet is therefore the rare book that meets the bar of changing my day-to-day behavior. "lol" is another excellent example that McCulloch spends some time on. It started life as LOL, an abbreviation for "laughing out loud," and that's still how it's stuck in my head. But, as McCulloch explains, it no longer means that to newer waves of Internet users. It now carries a far more complicated and nuanced meaning that has very little to do with physical laughter and that doesn't easily translate to a single word or sentence. I went from being mildly irritated by and mildly superior towards the ubiquitous "lol" to realizing that it's a fascinating new word that carries primarily emotional nuance and that I don't understand well enough to read or use properly (yet). One more example of the type of analysis McCulloch brings to this book: emoji. The tendency when talking about emoji is to treat them as rebuses (pictures that stand in for a word, or at least a specific concept). They are sometimes used that way, but McCulloch argues that they more often function in the same role that gestures play in informal speech, including the gestures that have no simple name and no independent meaning outside of the context of the words being said at the same time. This seems obvious in retrospect, but before reading Because Internet I had never thought about what a gesture is, what function it plays in speech, and how that could be translated into informal written communication. If you're as interested in this area as I am, this is great stuff. I'd seen several mentions of this book go past on Twitter and kept holding off because I had lots of things to read and was worried it would only cover the superficial things I already knew as a long-time Internet user who has listened to a few lectures on linguistics. That was not the case at all. I learned so much from this book and had a delightful time reading it. If you're also interested in these topics, recommended. Rating: 9 out of 10

12 December 2020

Russell Coker: Electromagnetic Space Launch

The G-Force Wikipedia page says that humans can survive 20G horizontally eyes in for up to 10 seconds and 10G for 1 minute. An accelerator of 14G for 10 seconds (well below the level that s unsafe) gives a speed of mach 4 and an acceleration distance of just under 7km. Launching a 100 metric ton spacecraft in that way would require 14MW at the end of the launch path plus some extra for the weight of the part that contains magnets which would be retrieved by parachute. 14MW is a small fraction of the power used by a train or tram network and brown-outs of the transit network is something that they deal with so such a launch could be powered by diverting power from a transit network. The Rocky Mountains in the US peak at 4.4KM above sea level, so a magnetic launch that starts 2.6KM below sea level and extends the height of the Rocky Mountains would do. A speed of mach 4 vertically would get a height of 96Km if we disregard drag, that s almost 1/4 of the orbital altitude of the ISS. This seems like a more practical way to launch humans into space than a space elevator. The Mass Driver page on Wikipedia documents some of the past research on launching satellites that way, with shorter launch hardware and significantly higher G forces.

11 December 2020

Dirk Eddelbuettel: RcppSpdlog 0.0.4: New upstream

Not quite two months after the last release, we are happy to announce version 0.0.4 of RcppSpdlog. It contains release 1.8.2 of spdlog made today, along with version 7.1.3 of fmt. RcppSpdlog bundles spdlog, a wonderful header-only C++ logging library with all the bells and whistles you would want that was written by Gabi Melman, and also includes fmt by Victor Zverovich. The NEWS entry for this release follows.

Changes in RcppSpdlog version 0.0.4 (2020-12-11)
  • Upgraded to upstream release spdlog 1.8.2
  • Added GitHub Actions CI using run.sh from r-ci

Courtesy of my CRANberries, there is also a diffstat report. More detailed information is on the RcppSpdlog page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.