Search Results: "ron"

4 May 2023

Emanuele Rocca: UEFI Secure Boot on the Raspberry Pi

UPDATE: this post unexpectedly ended up on Hacker News and I received a lot of comments. The two most important points being made are (1) that Secure Boot on the RPi as described here is not actually truly secure. An attacker who successfully gained root could just mount the firmware partition and either add their own keys to the EFI variable store or replace the firmware altogether with a malicious one. (2) The TianCore firmware cannot be used instead of the proprietary blob as I mentioned. What truly happens is that the proprietary blob is loaded onto the VideoCore cores, then TianoCore is loaded onto the ARM cores. Thanks for the corrections.

A port of the free software TianoCore UEFI firmware can be used instead of the proprietary boot blob to boot the Raspberry Pi. This allows to install Debian on the RPi with the standard Debian Installer, and it also makes it possible to use UEFI Secure Boot. Note that Secure Boot had been broken on arm64 for a while, but it s now working in Bookworm!.

Debian Installer UEFI boot
To begin, you ll need to download the appropriate firmware files for the RPi3 or RPi4. I ve got a Raspberry Pi 3 Model B+ myself, so the rest of this document will assume an RPi3 is being installed.
Plug the SD card you are going to use as the RPi storage device into another system. Say it shows up as /dev/sdf. Then:
# Create an msdos partition table
$ sudo parted --script /dev/sdf mklabel msdos
# Create, format, and label a 10M fat32 partition
$ sudo parted --script /dev/sdf mkpart primary fat32 0% 10M
$ sudo mkfs.vfat /dev/sdf1
$ sudo fatlabel /dev/sdf1 RPI-FW
# Get the UEFI firmware onto the SD card
$ sudo mount /dev/sdf1 /mnt/data/
$ sudo unzip Downloads/RPi3_UEFI_Firmware_v1.38.zip -d /mnt/data/
$ sudo umount /mnt/data
At this point, the SD card can be used to boot the RPi, and you ll get a UEFI firmware.
Download the Bookworm RC 2 release of the installer, copy it to a USB stick as described in the Installation Guide, and boot your RPi from the stick. If for some reason booting from the stick does not happen automatically, enter the firmware interface with ESC and choose the USB stick from Boot Manager.
Proceed with the installation as normal, paying attention not to modify the firmware partition labeled RPI-FW. I initially thought it would be nice to reuse the firmware partition as ESP partition as well. However, setting the esp flag on makes the RPi unbootable. Either configuring the partition as ESP in debian-installer, or manually with sudo parted --script /dev/sda set 1 esp on, breaks boot. In case you accidentally do that, set it back to off and the edk2 firmware will boot again.
What I suggest doing in terms of partitioning is: (1) leave the 10M partition created above for the firmware alone, and (2) create another 512M or so ESP partition for EFI boot.
The installation should go smoothly till the end, but rebooting won t work. Doh. This is because of an important gotcha: the Raspberry Pi port of the TianoCore firmware we are using does not support setting UEFI variables persistently from a "High Level Operating System (HLOS)", which is the debian-installer in our case. Persistently is the keyword there: variables can be set and modified regularly with efibootmgr or otherwise, but crucially the modifications do not survive reboot. However, changes made from the firmware interface itself are persistent. So enter the firmware with ESC right after booting the RPi, select Boot Maintenance Manager Boot Options Add Boot Option Your SD card Your ESP partition EFI debian shimaa64.efi. Choose a creative name for your boot entry (eg: "debian"), save and exit the firmware interface. Bookworm should be booting fine at this point!

Enabling Secure Boot
Although the TianoCore firmware does support Secure Boot, there are no keys enrolled by default. To add the required keys, copy PK-0001.der, DB-0001.der, DB-0002.der, KEK-0001.der, and KEK-0002.der to a FAT32 formatted USB stick.
Here s a summary of the Subject field for each of the above:
PK-0001.der.pem
        Subject: O = Debian, CN = Debian UEFI Secure Boot (PK/KEK key), emailAddress = debian-devel@lists.debian.org
DB-0001.der.pem
        Subject: C = US, ST = Washington, L = Redmond, O = Microsoft Corporation, CN = Microsoft Windows Production PCA 2011
DB-0002.der.pem
        Subject: C = US, ST = Washington, L = Redmond, O = Microsoft Corporation, CN = Microsoft Corporation UEFI CA 2011
KEK-0001.der.pem
        Subject: O = Debian, CN = Debian UEFI Secure Boot (PK/KEK key), emailAddress = debian-devel@lists.debian.org
KEK-0002.der.pem
        Subject: C = US, ST = Washington, L = Redmond, O = Microsoft Corporation, CN = Microsoft Corporation KEK CA 2011
Plug the stick into the RPi, boot and enter the firmware interface with ESC. Select Device Manager Secure Boot Configuration Secure Boot Mode choose Custom Mode Custom Secure Boot Options PK Options Enroll PK choose PK-0001.der. Do the same for DB Options, this time choose DB-0001.der and DB-0002.der. As you may have guessed by now, the same must be done for KEK Options, but adding KEK-0001.der and KEK-0002.der. Save, exit, reboot. If everything went well, your RPi now has booted with Secure Boot enabled.
See https://wiki.debian.org/SecureBoot for the details on how to check whether Secure Boot has been enabled correctly and much more.

Valhalla's Things: Linen Slippers

Posted on May 4, 2023
A pair of espadrille-like slippers in white fabric. I hate going out to buy shoes. Even more so I hate buying home shoes, which is what I spend most of my life in, also because no matter what I buy they seem to disintegrate after a season or so. So, obviously, I ve been on a quest to make my own. As a side note, going barefoot (with socks) would only move the wear issue to the socks, so it s not really a solution, and going bare barefoot on ceramic floors is not going to happen, kaythanksbye. For the winter I m trying to make knit and felted slippers; I ve had partial success, and they should be pretty easy to mend (I ve just had to do the first mend, with darning and needle felting, and it seems to have worked nicely). For the summer, I ve been thinking of something sewn, and with the warm season approaching (and the winter slippers needing urgent repairs) I decided it was time to work on them. I already had a shaped (left/right) pattern for a sole from my hiking sandals attempts (a topic for another post), so I started by drafting a front upper, and then I started to have espadrille feeling and decided that a heel guard was needed. As for fabric, looking around in the most easily accessible part of the Stash I ve found the nice heavyweight linen I m using for my Augusta Stays, of which I still have a lot and which looked almost perfect except for one small detail: it s very white. I briefly thought about dyeing, but I wanted to start sewing NOW to test the pattern, so, yeah, maybe it will happen one day, or maybe I ll have patchy dust-grey slippers. If I ll ever have a place where I can do woad dyeing a blue pair will happen, however. Contrary to the typical espadrillas I decided to have a full lining, and some padding between the lining and the sole, using cotton padding leftovers from my ironing board. To add some structure I also decided to add a few rows of cording (and thus make the uppers in two layers of fabric), to help prevent everything from collapsing flat. As for the sole, that s something that is still causing me woes: I do have some rubber sole sheets (see hiking sandals above), but I suspect that they require glueing, which I m not sure would work well with the natural fabric uppers and will probably make repairs harder to do. In the past I tried to make some crocheted rope soles and they were a big failure: they felt really nice on the foot, but they also self-destroyed in a matter of weeks, which is not really the kind of sole I m looking for. the slippers with the braided soles on top. Now I have some ~ 3 mm twine that feels much harsher on the hands while working it (and would probably feel harsher on the feet, but that s what the lining and padding are for), so I hope it may be a bit more resistant, and I tried to make a braided rope sole. Of course, I have published the pattern and instructions for the slippers as well as those for the braided rope sole as #FreeSoftWear. Now what is left is trying everything under daily use, and I hope I will have updates on this at the end of the season, rather than soon :D

3 May 2023

John Goerzen: Martha the Pilot

Martha, now 5, can t remember a time when she didn t fly periodically. She s come along in our airplane in short flights to a nearby restaurant and long ones to Michigan and South Dakota. All this time, she s been riding in the back seat next to Laura. Martha has been talking excitedly about riding up front next to me. She wants to be my co-pilot . I promised to give her an airplane wing pin when she did one I got from a pilot of a commercial flight when I was a kid. Of course, safety was first, so I wanted to be sure she was old enough to fly there without being a distraction. Last weekend, the moment finally arrived. She was so excited! She brought along her Claire bear aviator, one that I bought for her at an airport a little while back. She buckled in two of her dolls in the back seat. Martha's dolls And then up we went! Martha in the airplane Martha was so proud when we landed! We went to Stearman Field, just a short 10-minute flight away, and parked the plane right in front of the restaurant. We flew back, and Martha thought we should get a photo of her standing on the wing by the door. Great idea! Martha standing on the wing She was happily jabbering about the flight all the way home. She told us several times about the pin she got, watching out the window, watching all the screens in the airplane, and also that she didn t get sick at all despite some turbulence. And, she says, Now just you and I can go flying! Yes, that s something I m looking forward to!

2 May 2023

Neil Williams: Carrying Grief

This isn't a book review, although the reason that I am typing this now is because of a book, You Are Not Alone: from the creator and host of Griefcast, Cariad Lloyd, ISBN: 978-1526621870 and I include a handful of quotes from Cariad where there is really no better way of describing things. Many people experience death for the first time as a child, often relating to a family pet. Death is universal but every experience of death is unique. One of the myths of grief is the idea of the Five Stages but this is a misinterpretation. Denial, Anger, Bargaining, Depression and Acceptance represent the five stage model of death and have nothing to do with grief. The five stages were developed from studying those who are terminally ill, the dying, not those who then grieve for the dead person and have to go on living without them. Grief is for those who loved the person who has died and it varies between each of those people just as people vary in how they love someone. The Five Stages end at the moment of death, grief is what comes next and most people do not grieve in stages, it can be more like a tangled knot. Death has a date and time, so that is why the last stage of the model is Acceptance. Grief has no timetable, those who grieve will carry that grief for the rest of their lives. Death starts the process of grief in those who go on living just as it ends the life of the person who is loved. "Grief eases and changes and returns but it never disappears.". I suspect many will have already stopped reading by this point. People do not talk about death and grief enough and this only adds to the burden of those who carry their grief. It can be of enormous comfort to those who have carried grief for some time to talk directly about the dead, not in vague pleasantries but with specific and strong memories. Find a safe place without distractions and talk with the person grieving face to face. Name the dead person. Go to places with strong memories and be there alongside. Talk about the times with that person before their death. Early on, everything about grief is painful and sad. It does ease but it remains unpredictable. Closing it away in a box inside your head (as I did at one point) is like cutting off a damaged limb but keeping the pain in a box on the shelf. You still miss the limb and eventually, the box starts leaking. For me, there were family pets which died but my first job out of university was to work in hospitals, helping the nurses manage the medication regimen and providing specialist advice as a pharmacist. It will not be long in that environment before everyone on the ward gets direct experience of the death of a person. In some ways, this helped me to separate the process of death from the process of grief. I cared for these people as patients but these were not my loved ones. Later, I worked in specialist terminal care units, including providing potential treatments as part of clinical trials. Here, it was not expected for any patient to be discharged alive. The more aggressive chemotherapies had already been tried and had failed, this was about pain relief, symptom management and helping the loved ones. Palliative care is not just about the patient, it involves helping the loved ones to accept what is happening as this provides comfort to the patient by closing the loop. Grief is stressful. One of the most common causes of personal stress is bereavement. The death of your loved one is outside of your control, it has happened, no amount of regret can change that. Then come all the other stresses, maybe about money or having somewhere to live as a result of what else has changed after the death or having to care for other loved ones. In the early stages, the first two years, I found it helpful to imagine my life as a box containing a ball and a button. The button triggers new waves of pain and loss each time it is hit. The ball bounces around the box and hits the button at random. Initially, the button is large and the ball is enormous, so the button is hit almost constantly. Over time, both the button and the ball change size. Starting off at maximum, initially there is only one direction of change. There are two problems with this analogy. First is that the grief ball has infinite energy which does not happen in reality. The ball may get smaller and the button harder to hit but the ball will continue bouncing. Secondly, the life box is not a predictable shape, so the pattern of movement of the ball is unpredictable. A single stress is one thing, but what has happened since has just kept adding more stress for me. Shortly before my father died 5 years ago now, I had moved house. Then, I was made redundant on the day of the first anniversary of my father's death. A year or so later, my long term relationship failed and a few months after that COVID-19 appeared. As the country eased out of the pandemic in 2021, my mother died (unrelated to COVID itself). A year after that, I had to take early retirement. My brother and sister, of course, share a lot of those stressors. My brother, in particular, took the responsibility for organising both funerals and did most of the visits to my mother before her death. The grief is different for each of the surviving family. Cariad's book helped me understand why I was getting frequent ideas about going back to visit places which my father and I both knew. My parents encouraged each of us to work hard to leave Port Talbot (or Pong Toilet locally) behind, in no small part due to the unrestrained pollution and deprivation that is common to small industrial towns across Wales, the midlands and the north of the UK. It wasn't that I wanted to move house back to our ancestral roots. It was my grief leaking out of the box. Yes, I long for mountains and the sea because I'm now living in a remorselessly flat and landlocked region after moving here for employment. However, it was my grief driving those longings - not for the physical surroundings but out of the shared memories with my father. I can visit those memories without moving house, I just need to arrange things so that I can be undisturbed and undistracted. I am not alone with my grief and I am grateful to my friends who have helped whilst carrying their own grief. It is necessary for everyone to think and talk about death and grief. In respect of your own death, no matter how far ahead that may be, consider Advance Care Planning and Expressions of Wish as well as your Will. Talk to people, document what you want. Your loved ones will be grateful and they deserve that much whilst they try to cope with the first onslaught of grief. Talk to your loved ones and get them to do the same for themselves. Normalise talking about death with your family, especially children. None of us are getting out of this alive and we will all leave behind people who will grieve.

Dirk Eddelbuettel: RQuantLib 0.4.18 on CRAN: Maintenance

A new release 0.4.18 of RQuantLib arrived at CRAN earlier today, and will be uploaded to Debian as well. QuantLib is a very comprehensice free/open-source library for quantitative finance; RQuantLib connects it to the R environment and language. This release of RQuantLib comes about six months after the previous maintenance release. It brings a few small updates triggered by small changes in the QuantLib releases 1.29 and 1.30. It also contains updates reflecting changes in the rgl package kindly contributed by Duncan Murdoch. Last but not least, Jeroen Ooms helped with two pull requests updating builds on, repspectively, macOS and Windows by updating the pre-made libraries of QuantLib.

Changes in RQuantLib version 0.4.18 (2023-05-01)
  • Use of several rgl functions was updated to a new naming scheme in the package (kindly contributed by Duncan Murdoch in #174)
  • A default argument is now given for option surface plots
  • Changed call from SwaptionVolCube1 to SabrSwaptionVolatilityCube (conditional on using QuantLib 1.30 or later)
  • Some other deprecation warnings were tweaked as in QL test file
  • Builds for macOS and Windows were updated with more library build (changes kindly contributed by Jeron Ooms in #176 and #175)
  • Some remaining throw calls were replace by Rcpp::stop

Courtesy of my CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the RQuantLib page. Questions, comments etc should go to the new rquantlib-devel mailing list. Issue tickets can be filed at the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

1 May 2023

Gunnar Wolf: Scanning heaps of 8mm movies

After my father passed away, I brought home most of the personal items he had, both at home and at his office. Among many, many (many, many, many) other things, I brought two of his personal treasures: His photo collection and a box with the 8mm movies he shot approximately between 1956 and 1989, when he was forced into modernity and got a portable videocassette recorder. I have talked with several friends, as I really want to get it all in a digital format, and while I ve been making slow but steady advances scanning the photo reels, I was particularly dismayed (even though it was most expected most personal electronic devices aren t meant to last over 50 years) to find out the 8mm projector was no longer in working conditions; the lamp and the fans work, but the spindles won t spin. Of course, it is quite likely it is easy to fix, but it is beyond my tinkering abilities and finding photographic equipment repair shops is no longer easy. Anyway, even if I got it fixed, filming a movie from a screen, even with a decent camera, is a lousy way to get it digitized. But almost by mere chance, I got in contact with my cousin Daniel, ho came to Mexico to visit his parents, and had precisely brought with him a 8mm/Super8 movie scanner! It is a much simpler piece of equipment than I had expected, and while it does present some minor glitches (i.e. the vertical framing slightly loses alignment over the course of a medium-length film scanning session, and no adjustment is possible while the scan is ongoing), this is something that can be decently fixed in post-processing, and a scanning session can be split with no ill effects. Anyway, it is quite uncommon a mid-length (5min) film can be done without interrupting i.e. to join a splice, mostly given my father didn t just film, but also edited a lot (this is, it s not just family pictures, but all different kinds of fiction and documentary work he did). So, Daniel lent me a great, brand new, entry-level film scanner; I rushed to scan as many movies as possible before his return to the USA this week, but he insisted he bought it to help preserve our family s memory, and given we are still several cousins living in Mexico, I could keep hold of it so any other of the cousins will find it more easily. Of course, I am thankful and delighted! So, this equipment is a Magnasonic FS81. It is entry-level, as it lacks some adjustment abilities a professional one would surely have, and I m sure a better scanner will make the job faster but it s infinitely superior to not having it! The scanner processes roughly two frames per second (while the nominal 8mm/Super8 speed is 24 frames per second), so a 3 minute film reel takes a bit over 35 minutes And a long, ~20 minute film reel takes Close to 4hr, if nothing gets in your way :- And yes, with longer reels, the probability of a splice breaking are way higher than with a short one not only because there is simply a longer film to process, but also because, both at the unwinding and at the receiving reels, mechanics play their roles. The films don t advance smoothly, but jump to position each frame in the scanner s screen, so every bit of film gets its fair share of gentle tugs. My professional consultant on how and what to do is my good friend Chema Serralde, who has stopped me from doing several things I would regret later otherwise (such as joining spliced tapes with acidic chemical adhesives such as Kola Loka, a.k.a. Krazy Glue even if it s a bit trickier to do it, he insisted me on best using simple transparent tape if I m not buying fancy things such as film-adhesive). Chema also explained me the importance of the loopers (las Lupes in his technical Spanish translation), which I feared increased the likelihood of breaking a bit of old glue due to the angle in which the film gets pulled but if skipped, result in films with too much jumping. Not all of the movies I have are for public sharing Some of them are just family movies, with high personal value, but probably of very little interest to others. But some are! I have been uploading some of the movies, after minor post-processing, to the Internet Archive. Among them: Anyway, I have a long way forward for scanning. I have 20 3min reels, 19 5min reels, and 8 20min reels. I want to check the scanning quality, but I think my 20min reels are mostly processed (we paid for scanning them some years ago). I mostly finished the 3min reels, but might have to go over some of them again due to the learning process. And Well, I m having quite a bit of fun in the process!

Russ Allbery: Review: The Amazing Maurice and His Educated Rodents

Review: The Amazing Maurice and His Educated Rodents, by Terry Pratchett
Series: Discworld #28
Publisher: HarperCollins
Copyright: 2001
Printing: 2008
ISBN: 0-06-001235-8
Format: Mass market
Pages: 351
The Amazing Maurice and His Educated Rodents is the 28th Discworld novel and the first marketed for younger readers. Although it has enough references to establish it as taking place on Discworld, it has no obvious connections with the other books and doesn't rely on any knowledge of the series so far. This would not be a bad place to start with Terry Pratchett and see if his writing style and sense of humor is for you. Despite being marketed as young adult, and despite Pratchett's comments in an afterward in the edition I own that writing YA novels is much harder, I didn't think this was that different than a typical Discworld novel. The two main human characters read as about twelve and there were some minor changes in tone, but I'm not sure I would have immediately labeled it as YA if I hadn't already known it was supposed to be. There are considerably fewer obvious pop culture references than average, though; if that's related, I think I'll prefer Pratchett's YA novels, since I think his writing is stronger when he's not playing reference bingo. Maurice (note to US readers: Maurice is pronounced "Morris" in the UK) is a talking cat and the mastermind of a wandering con job. He, a stupid-looking kid with a flute (Maurice's description), and a tribe of talking rats travel the small towns of Discworld. The rats go in first, making a show of breaking into the food, swimming in the cream, and widdling on things that humans don't want widdled on. Once the townspeople are convinced they have a plague of rats, the kid with the flute enters the town and offers to pipe the rats away for a very reasonable fee. He plays his flute, the rats swarm out of town, and they take their money and move on to the next town. It's a successful life that suits Maurice and his growing hoard of gold very well. If only the rats would stop asking pointed questions about the ethics of this scheme. The town of Bad Blintz is the next on their itinerary, and if the rats have their way, will be the last. Their hope is they've gathered enough money by now to find an island, away from humans, where they can live their own lives. But, as is always the case for one last job in fiction, there's something uncannily wrong about Bad Blintz. There are traps everywhere, more brutal and dangerous ones than they've found in any other town, and yet there is no sign of native, unintelligent rats. Meanwhile, Maurice and the boy find a town that looks wealthy but has food shortages, a bounty on rats that is absurdly high, and a pair of sinister-looking rat-catchers who are bringing in collections of rat tails that look suspiciously like bootlaces. The mayor's daughter discovers Maurice can talk and immediately decides she has to take them in hand. Malicia is very certain of her own opinions, not accustomed to taking no for an answer, and is certain that the world follows the logic of stories, even if she has to help it along. This is truly great stuff. I think this might be my favorite Discworld novel to date, although I do have some criticisms that I'll get to in a moment. The best part are the rats, and particularly the blind philosopher rat Dangerous Beans and his assistant Peaches. In the middle of daring infiltration of the trapped sewers in scenes reminiscent of Mission: Impossible, the rats are also having philosophical arguments. They've become something different than the unaltered rats that they call the keekees, but what those differences mean is harder to understand. The older rats are not happy about too many changes and think the rats should keep acting like rats. The younger ones are discovering that they're afraid of shadows because now they understand what the shadows hint at. Dangerous Beans is trying to work out a writing system so that they can keep important thoughts. One of their few guides is a children's book of talking animals, although they quickly discover that the portrayed clothing is annoyingly impractical. But as good as the rats are, Maurice is nearly as much fun in an entirely different way. He is unapologetically out for himself, streetwise and canny in a way that feels apt for a cat, gets bored and mentally wanders off in the middle of conversations, and pretends to agree with people when that's how he can get what he wants. But he also has a weird sense of loyalty and ethics that only shows up when something is truly important. It's a variation on the con man with a heart of gold, but it's a very well-done variation that weaves in a cat's impatience with and inattention to anything that doesn't directly concern them. I was laughing throughout the book. Malicia is an absolute delight, the sort of character who takes over scenes through sheer force of will, and the dumb-looking kid (whose name turns out to be Keith) is a perfect counterbalance: a laid-back, quiet boy who just wants to play his music and is almost entirely unflappable. It's such a great cast. The best part of the plot is the end. I won't spoil it, so I'll only say that Pratchett has the characters do the work on the aftermath that a lot of books skip over. He doesn't have any magical solutions for the world's problems, but he's so very good at restoring one's faith that maybe sometimes those solutions can be constructed. My one complaint with this book is that Pratchett introduces a second villain, and while there are good in-story justifications for it and it's entangled with the primary plot, he added elements of (mild) supernatural horror and evil that I thought were extraneous and unnecessary. He already had enough of a conflict set up without adding that additional element, and I think it undermined the moral complexity of the story. I would have much rather he kept the social dynamics of the town at the core of the story and used that to trigger the moments of sacrifice and philosophy that made the climax work. The Discworld books by this point have gotten very good, but each book seems to have one element like that where it felt like Pratchett took the easy way out of a plot corner or added some story element that didn't really work. I feel like the series is on the verge of having a truly great book that rises above the entire series to date, but never quite gets there. That caveat aside, I thoroughly enjoyed this and had trouble putting it down. Mrs. Frisby and the Rats of Nimh was one of my favorite books as a kid, and this reminded me of it in some good ways (enough so that I think some of the references were intentional). Great stuff. If you were to read only one Discworld book and didn't want to be confused by all the entangled plot threads and established characters, I would seriously consider making it this one. Recommended. Followed by Night Watch in publication order. There doesn't appear to be a direct plot sequel, more's the pity. Rating: 8 out of 10

30 April 2023

Russell Coker: Links April 2023

Cory Doctorow has an insightful article Gig Work is the Opposite of Steampunk [1] about the horrors that companies like Amazon are forcing on their employees. Valerie Aurora and Leigh Honeywell wrote an insightful article about the al Capone theory of sexual harassment [2]. Why people who sexually harass others usually perform other anti-social activity that is also easier to prosecute. The IEEE has an interesting article about using ML for parts of the CPU design process, both the technical issues and the controversy about competing ideas which is probably caused by sexism [3]. Love and taxes are forever my heart is a line from an anime dating sim game that prepares US taxes [4]. Unfortunately it was removed from Steam. The existence of the game is a weird social commentary and removing the game because you can t have an anime hottie do taxes is bizarre but also understandable given liability issues. There s no mention in the review of whether male hotties are available for people who prefer dating guys. As an aside my accountant looks like he is allergic to exercise The Killdozer Book web site (which has an invalid SSL certificate so you have to click on advanced in Chrome to get to the content) has an insightful article debunking some of the stories about the Killdozer [5]. He wasn t some sort of hero of freedom, he was just a jerk who reneged on a deal hoping to get more money, thought that laws shouldn t apply to him, and killed himself because of it. Apparently some big tech companies are knowingly hiring people to not work unlike the usual large corporate case of unknowingly hiring people to not work [6]. Silicon Valley is a good TV show, and it s apparently realistic. Ron Garrett wrote in insightful blog post about theoretical attacks on Bitcoin and how Bitcoin could be used [7]. The conclusion is not good for Bitcoin. Compiler Explorer is a program that shows how various C++ compilers produce assembly code for various architectures, this site hosts the main active instance [8]. There are other instances, here is an instance that produces code for the Ruzzian Elbrus architecture [9]. The Elbrus Wikipedia page is interesting [10]. Apparently the Ruzzians don t want this information to be spread, LOL. The Smithsonian Magazine has an interesting article about pet parrots being taught to video call each other [11]. Apparently parrots are social animals and can develop psychological problems if kept alone, so the video calls can be good for them. Also the owners had to monitor the chats to ensure that they played nicely together, just like play-dates for kids! Phoronix has an amusing article about the drama regarding the AMD Spectral Chicken bit in the Linux kernel source [12]. This page listing bad free software licenses is amusing [13]. The ACS has an interesting article about how Samsung fakes photos of the moon and presumably could fake other photos of notable objects that don t change [14]. The way that they proved the forgery was interesting.

Valhalla's Things: Programming the ESP32-C3-DevKit-Lipo with Arduino

Posted on April 30, 2023
A few months ago we may have bought a few ESP32-C3-DevKit-Lipo boards from Olimex. Since every time I go back to working with them I ve forgotten how to do so, and my old notes on the fediverse are hard to find, this is the full procedure.

Setup I start by sort-of-following https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html
  • Install arduino from the distribution packages (version 1.8 is ok).
  • Under File Preferences, add the development URL to the Additional Boards Manager URLs field. (on 2023-04-30 that s https://espressif.github.io/arduino-esp32/package_esp32_dev_index.json).
  • Under Tools Board Boards Manager make sure that you install a version of esp32 by Espressif Systems that is above 2.0 (on 2023-04-30 there is a 2.0.8 that works).

Programming
  • Under Tools Board ESP32 Arduino select ESP32C3 Dev Module.
  • Under Tools USB CDC On Boot select Enabled.
You can now compile and upload your sketches. If something goes wrong, to force the board to bootloader mode bring GPIO9 to GND. Note that the serial port device /dev/ttyACM0 only appears when in bootloader mode, but uploading sketches and the serial monitor will still work even if the port is not set in the arduino IDE.

29 April 2023

Russell Coker: Write a blog post in the style of Russell Coker

Feeling a bit bored I asked ChatGPT Write a blog post in the style of Russell Coker and the result is in the section below. I don t know if ChatGPT knows that the person asking the question is the same as the person being asked about. If a human had created that I d be certain that great computer scientist and writer was an attempt at flattery, for a machine I m not sure. I have not written a single book, but I expect that in some alternate universe some version of me has written several. I don t know if humans would describe my writing as being known for clarity, precision, and depth . I would not be surprised if no-one else wrote about it so I guess I m forced to read what he wrote would be a more common response. The actual article part doesn t seem to be in my style at all. Firstly it s very short at only 312 words, while I have written some short posts most of them are much longer. To find this out I did some MySQL queries to get the lengths of posts (I used this blog post as inspiration [1]). Note that multiple sequential spaces counts as multiple words.
# get post ID and word count
SELECT id, LENGTH(post_content) - LENGTH(REPLACE(REPLACE(REPLACE(REPLACE(post_content, "\r", ""), "\n", ""), "\t", ""), " ", "")) + 1 AS wordcount FROM wp_posts where post_status = 'publish' and post_type='post';
# get average word count
SELECT avg(LENGTH(post_content) - LENGTH(REPLACE(REPLACE(REPLACE(REPLACE(post_content, "\r", ""), "\n", ""), "\t", ""), " ", "")) + 1) FROM wp_posts where post_status = 'publish' and post_type='post';
# get the first posts by length
SELECT id, LENGTH(post_content) - LENGTH(REPLACE(REPLACE(REPLACE(REPLACE(post_content, "\r", ""), "\n", ""), "\t", ""), " ", "")) + 1 AS wordcount, post_content FROM wp_posts where post_status = 'publish' and post_type='post' ORDER BY wordcount limit 10;
# get a count of the posts less than 312 words
SELECT count(*) from wp_posts where (LENGTH(post_content) - LENGTH(REPLACE(REPLACE(REPLACE(REPLACE(post_content, "\r", ""), "\n", ""), "\t", ""), " ", "")) + 1) < 312 and post_status = 'publish' and post_type='post';
# get a count of all posts
select count(*) from wp_posts where post_status = 'publish' and post_type='post';
It turns out that there are 333/1521 posts that are less than 312 words and the average length is 665 words. Of the shortest posts a large portion were written before Twitter became popular and had the sort of unimportant content that Twitter is good for. It seems ironic that ChatGPT says that I'm known for "depth" and then writes a short post allegedly in my style. As for the actual content of the "post", it's not something that I greatly disagree with but it's definitely not something I'd write. If some manager or marketing person wrote something like this and asked my opinion I'd probably suppress a sigh and say it's OK for it's target market. I think that free software is better in many ways and for many reasons and that separately from the issue of merit free software is important because of the ways that it is used. We could debate various issues related to the quality of free software vs proprietary software but even people who don't believe that free software is of high quality have to admit that software that runs most mobile phones and most web servers is important. As much as I dislike Windows most desktop PCs and laptops run it so I can't claim it to be unimportant. So my first disagreement with the content is the issue of importance vs quality. One thing that ChatGPT is known for is weak arguments that appear convincing. "This diversity often leads to better software, as bugs are found and fixed faster, and new features are added more quickly" could be opposed by "the payment for proprietary software often leads to better software, as bugs are found and fixed faster, and new features are added more quickly". I think that any case where "feature of option A gives this list of good results" can be rewritten as "feature of the opposite of option A gives the same list of good results" and be equally convincing is a poor argument. Also any argument that seems to rest on "free software is all written by hobbyists" is ignoring the demands of users who need things done on a non-hobby timeframe, the many profitable companies that develop and support free software, and the employment status of a large portion of my Facebook friends (many of whom are paid for ~2000 hours a year of free software development and many more of whom have paid free software development as part of their work supporting the use of free software). Really the entire blog post it wrote seems to me like something a human might write if their boss assigned them to write about the benefits of free software, and that they would then write a new article saying the exact opposite if their boss asked them to. Finally I prefer the term "free software" to "open source software". The difference doesn't seem important to some people, but any being that doesn't know the difference won't do well at impersonating me. ChatGPT Result Hello everyone, I'm ChatGPT, and today I'm going to try my best to write a blog post in the style of the great computer scientist and writer, Russell Coker. Russell Coker is a renowned software engineer and writer, who is best known for his contributions to the Linux community, especially the Debian project. He has authored several books and articles on topics ranging from security to software engineering, and his writing style is known for its clarity, precision, and depth. So, without further ado, let's dive into the topic of this blog post, which is the importance of open-source software. Open-source software is software whose source code is available for anyone to inspect, modify, and distribute. This means that anyone can use, study, and improve the software, without any restrictions or fees. Open-source software has been around for a long time, but it has become more prevalent and important in recent years, thanks to the rise of the internet and the digital economy. Open-source software has several benefits, both for individuals and for society as a whole. One of the most significant benefits is that it fosters innovation and collaboration. When software is open-source, it means that anyone can contribute to it, which leads to a diverse range of perspectives and ideas. This diversity often leads to better software, as bugs are found and fixed faster, and new features are added more quickly. Another benefit of open-source software is that it is often more secure than proprietary software. Because the source code is available for anyone to inspect, vulnerabilities and security flaws can be found and fixed faster. This means that open-source software is often more reliable and trustworthy than proprietary software, which may have hidden backdoors or vulnerabilities that can be exploited by attackers. Open-source software is also often more cost-effective than proprietary software. Because it is free to use and distribute, organizations can save money on licensing fees and other costs associated with proprietary software. This can be particularly important for non-profit organizations or small businesses that may not have the resources to invest in expensive software licenses. In conclusion, open-source software is an important part of the digital landscape, and its benefits are numerous. Whether you are a software developer, a business owner, or an individual user, open-source software has something to offer. So, the next time you are looking for software to use, consider open-source options, and help support the collaborative and innovative spirit of the open-source community.

Abhijith PA: Attending FOSSASIA 2023

I attended FOSSASIA 2023 summit held at Lifelong Learning Institute, Singapore. A 3 day long parallel talk filled conference. Its my second time attending FOSSASIA. The first one was 2018 summit. Like last time, I didn t attend much talks but focussed on networking with people. A lot of familiar faces there. PV Anthony, Harish, etc. I vounteered to run Debian Booth at the exhibition hall distributing stickers, flyers. Rajudev also helped me at the booth. Most of the people there used debian or its derivates or know debian already, its easier for me that way, that I don t have do much explaining compared to other booths. Thanks to Parth for looking after booth in my breaks. debian_booth1 debian_booth2 Sometimes our booth also act as cloak room :). Ours was close to the entrance door and we may be similar faces to folks. So people come and drop their bags before they go to talks. One thing I love about the such conference is that people have very different hardwares that I never able to see otherwise. I remember KDE booth had a Steam Value portable gaming board running KDE plasma. Then a person have this eyeglass which act as a monitor. Then the usual DJI drones but custom programmed. It was very lovely to meet and play around with exotic hardwares. Kurian whom I met during a Debian packaging workshop in Kerala was a speaker at FOSSASIA. He presented a talk titled OpenAI Whisper and it s amazing power to do finetuning . I was his unofficial PR guy, taking pictures :).

Weather Clear sky, little hot and humid. The weather was quite nice for me except the surprise rain and small thunderstorms. Comparing to my place s temperature, its wonderful.

28 April 2023

Scarlett Gately Moore: KDE Snaps, Gear 23.04.0 available in snap store

KDE Digikam 8.0.0 SnapKDE Digikam 8.0.0 Snap
It has been another crazy busy couple of weeks. There are too many snaps released to list here, but you can track my progress here: https://invent.kde.org/packaging/snapcraft-kde-applications/-/issues/30 Some notable releases are: I updated our Frameworks/QT5 content pack to kf5 105 and qt5 5.15.9. I have added more documentation at: https://invent.kde.org/teams/neon/-/wikis/Snaps I am finishing up the last of 23.04.0 snaps and will tackle KDE PIM suite next week. I will also work on more apps that have their own release cycle. Several snap bugs have been resolved and closed! Please remember if you have issues, to file them at bugs.kde.org as I have a hard time finding them at <insert random forum / social network here> On the Debian front, I successfully closed two CVE s for ruby-rack in Buster! I will be doing more ELTS/LTS security updates next month! Thanks for stopping by! If you enjoy my work, consider a dontation. https://gofund.me/2c7b1808

Sven Hoexter: What's wrong in IT: commit messages

In my day job someone today took the time in the team daily to explain his research why some of our configuration is wrong. He spent quite some time on his own to look at the history in git and how everything was setup initially, and ended up in the current - wrong - way. That triggered me to validate that quickly, another 5min of work. So we agreed to change it. A one line change, nothing spectacular, but lifetime was invested to figure out why it should've a different value. When the pull request got opened a few minutes later there was nothing of that story in the commit message. Zero, nada, nothing. :( I'm really puzzled why someone invests lifetime to dig into company internal history to try get something right, do a lengthy explanation to the whole team, use the time of others, even mention that there was no explanation of why it's not the default value anymore it should be, and repeat the same mistake by not writing down anything in the commit message. For the current company I'm inclined to propose a commit message validator. For a potential future company I might join, I guess I ask for real world git logs from repositories I should contribute to. Seems that this is another valuable source of information to qualify the company culture. Next up to the existence of whiteboards in the office. I'm really happy that at least a majority of the people contributing to Debian writes somewhat decent commit messages and changelogs. Let that be a reminder to myself to improve in that area the next time I've to change something.

Shirish Agarwal: John Grisham s books, Evolution removed from textbooks

Gray Mountain John Grisham I have been perusing John Grisham s books, some read and some re-read again. Almost all of the books that Mr. Grisham wrote are relevant even today. The Gray Mountain talks about how mountain top removal was done in Applachia, the U.S. (South). In fact NASA made a summary which either was borrowed from this book or the author borrowed from NASA, either could be true although seems it might be the former. And this is when GOI just made a new Forest Conservation Bill 2023 which does the opposite. There are many examples of the same, the latest from my own city as an e.g. Vetal Tekdi is and was a haven for people animals, birds all kinds of ecosystem and is a vital lung of the city, one of the few remaining green spaces in the city but BJP wants to commercialize it as it has been doing for everything, I haven t been to the Himalayas since 4-5 years back as I hear the rape of the land daily. Even after Joshimath tragedy, if people are not opening the eyes what can I do  The more I say, the more depressed I will become so will leave it for now. In many ways the destruction seems similar to the destruction that happened in Brazil under Jair Bolsonaro. So as can be seen from what I have shared this book was mostly about environment and punitive damages and also how punitive damages have been ceiling in America (South). This was done via lobbying by the coal groups and apparently destroyed people s lives, communities, even whole villages. It also shared how most people called black lung and how those claims had been denied by Coal Companies all the time. And there are hardly any unions. While the book itself is a fiction piece, there is a large amount of truth in it. And that is one of the reasons people write a fiction book. You could write about reality using fictional names and nobody can sue you while you set the reality as it is. In many ways, it is a tell-all.

The Testament John Grisham One of the things I have loved about John Grisham is he understands human condition. In this book it starts with an eccentric billionaire who makes a will which leaves all his property to an illegitimate child who coincidentally also lives in Brazil, she is a missionary. The whole book is about human failings as well as about finding the heir. I am not going to give much detail as the book itself is fun.

The Appeal John Grisham In this, we are introduced to a company that does a chemical spill for decades and how that leads to lobbying and funding Judicial elections. It does go into quite a bit of length how private money coming from Big business does all kind of shady things to get their person elected to the Supreme Court. Sadly, this seems to happen all the time, for e.g. two weeks ago. This piece from the Atlantic also says the same. Again, won t tell as there is a bit of irony at the end of the book.

The Rainmaker John Grisham This in short is about how Insurance Companies stiff people. It s a wonderful story that has all people in grey including our hero. An engaging book that sorta tells how the Insurance Industry plays the game. In India, the companies are safe as they ask for continuance for years together and their object is to delay the hearings till the grandchildren are not alive unlike in the U.S. or UK. It also does tell how more lawyers are there then required. Both of which are the same in my country.

The Litigators John Grisham This one is about Product Liability, both medicines as well as toys for young kids. What I do find funny a bit is how the law in States allows people to file cases but doesn t protect people while in EU they try their best that if there is any controversy behind any medicine or product, they will simply ban it. Huge difference between the two cultures.

Evolution no longer part of Indian Education A few days ago, NCERT (one of the major bodies) that looks into Indian Education due to BJP influence has removed Darwin s Theory of Evolution. You can t even debate because the people do not understand adaptability. So the only conclusion is that Man suddenly appeared out of nowhere. If that is so, then we are the true aliens. They discard the notion that we and Chimpanzees are similar. Then they also have to discard this finding that genetically we are 96% similar.

27 April 2023

Simon Josefsson: A Security Device Threat Model: The Substitution Attack

I d like to describe and discuss a threat model for computational devices. This is generic but we will narrow it down to security-related devices. For example, portable hardware dongles used for OpenPGP/OpenSSH keys, FIDO/U2F, OATH HOTP/TOTP, PIV, payment cards, wallets etc and more permanently attached devices like a Hardware Security Module (HSM), a TPM-chip, or the hybrid variant of a mostly permanently-inserted but removable hardware security dongles. Our context is cryptographic hardware engineering, and the purpose of the threat model is to serve as as a thought experiment for how to build and design security devices that offer better protection. The threat model is related to the Evil maid attack. Our focus is to improve security for the end-user rather than the traditional focus to improve security for the organization that provides the token to the end-user, or to improve security for the site that the end-user is authenticating to. This is a critical but often under-appreciated distinction, and leads to surprising recommendations related to onboard key generation, randomness etc below.

The Substitution Attack
An attacker is able to substitute any component of the device (hardware or software) at any time for any period of time.
Your takeaway should be that devices should be designed to mitigate harmful consequences if any component of the device (hardware or software) is substituted for a malicious component for some period of time, at any time, during the lifespan of that component. Some designs protect better against this attack than other designs, and the threat model can be used to understand which designs are really bad, and which are less so.

Terminology The threat model involves at least one device that is well-behaving and one that is not, and we call these Good Device and Bad Device respectively. The bad device may be the same physical device as the good key, but with some minor software modification or a minor component replaced, but could also be a completely separate physical device. We don t care about that distinction, we just care if a particular device has a malicious component in it or not. I ll use terms like security device , device , hardware key , security co-processor etc interchangeably. From an engineering point of view, malicious here includes unintentional behavior such as software or hardware bugs. It is not possible to differentiate an intentionally malicious device from a well-designed device with a critical bug. Don t attribute to malice what can be adequately explained by stupidity, but don t na vely attribute to stupidity what may be deniable malicious.

What is some period of time ? Some period of time can be any length of time: seconds, minutes, days, weeks, etc. It may also occur at any time: During manufacturing, during transportation to the user, after first usage by the user, or after a couple of months usage by the user. Note that we intentionally consider time-of-manufacturing as a vulnerable phase. Even further, the substitution may occur multiple times. So the Good Key may be replaced with a Bad Key by the attacker for one day, then returned, and later this repeats a month later.

What is harmful consequences ? Since a security key has a fairly well-confined scope and purpose, we can get a fairly good exhaustive list of things that could go wrong. Harmful consequences include:
  • Attacker learns any secret keys stored on a Good Key.
  • Attacker causes user to trust a public generated by a Bad Key.
  • Attacker is able to sign something using a Good Key.
  • Attacker learns the PIN code used to unlock a Good Key.
  • Attacker learns data that is decrypted by a Good Key.

Thin vs Deep solutions One approach to mitigate many issues arising from device substitution is to have the host (or remote site) require that the device prove that it is the intended unique device before it continues to talk to it. This require an authentication/authorization protocol, which usually involves unique device identity and out-of-band trust anchors. Such trust anchors is often problematic, since a common use-case for security device is to connect it to a host that has never seen the device before. A weaker approach is to have the device prove that it merely belongs to a class of genuine devices from a trusted manufacturer, usually by providing a signature generated by a device-specific private key signed by the device manufacturer. This is weaker since then the user cannot differentiate two different good devices. In both cases, the host (or remote site) would stop talking to the device if it cannot prove that it is the intended key, or at least belongs to a class of known trusted genuine devices. Upon scrutiny, this solution is still vulnerable to a substitution attack, just earlier in the manufacturing chain: how can the process that injects the per-device or per-class identities/secrets know that it is putting them into a good key rather than a malicious device? Consider also the consequences if the cryptographic keys that guarantee that a device is genuine leaks. The model of the thin solution is similar to the old approach to network firewalls: have a filtering firewall that only lets through intended traffic, and then run completely insecure protocols internally such as telnet. The networking world has evolved, and now we have defense in depth: even within strongly firewall ed networks, it is prudent to run for example SSH with publickey-based user authentication even on locally physical trusted networks. This approach requires more thought and adds complexity, since each level has to provide some security checking. I m arguing we need similar defense-in-depth for security devices. Security key designs cannot simply dodge this problem by assuming it is working in a friendly environment where component substitution never occur.

Example: Device authentication using PIN codes To see how this threat model can be applied to reason about security key designs, let s consider a common design. Many security keys uses PIN codes to unlock private key operations, for example on OpenPGP cards that lacks built-in PIN-entry functionality. The software on the computer just sends a PIN code to the device, and the device allows private-key operations if the PIN code was correct. Let s apply the substitution threat model to this design: the attacker replaces the intended good key with a malicious device that saves a copy of the PIN code presented to it, and then gives out error messages. Once the user has entered the PIN code and gotten an error message, presumably temporarily giving up and doing other things, the attacker replaces the device back again. The attacker has learnt the PIN code, and can later use this to perform private-key operations on the good device. This means a good design involves not sending PIN codes in clear, but use a stronger authentication protocol that allows the card to know that the PIN was correct without learning the PIN. This is implemented optionally for many OpenPGP cards today as the key-derivation-function extension. That should be mandatory, and users should not use setups that sends device authentication in the clear, and ultimately security devices should not even include support for that. Compare how I build Gnuk on my PGP card with the kdf_do=required option.

Example: Onboard non-predictable key-generation Many devices offer both onboard key-generation, for example OpenPGP cards that generate a Ed25519 key internally on the devices, or externally where the device imports an externally generated cryptographic key. Let s apply the subsitution threat model to this design: the user wishes to generate a key and trust the public key that came out of that process. The attacker substitutes the device for a malicious device during key-generation, imports the private key into a good device and gives that back to the user. Most of the time except during key generation the user uses a good device but still the attacker succeeded in having the user trust a public key which the attacker knows the private key for. The substitution may be a software modification, and the method to leak the private key to the attacker may be out-of-band signalling. This means a good design never generates key on-board, but imports them from a user-controllable environment. That approach should be mandatory, and users should not use setups that generates private keys on-board, and ultimately security devices should not even include support for that.

Example: Non-predictable randomness-generation Many devices claims to generate random data, often with elaborate design documents explaining how good the randomness is. Let s apply the substitution threat model to this design: the attacker replaces the intended good key with a malicious design that generates (for the attacker) predictable randomness. The user will never be able to detect the difference since the random output is, well, random, and typically not distinguishable from weak randomness. The user cannot know if any cryptographic keys generated by a generator was faulty or not. This means a good design never generates non-predictable randomness on the device. That approach should be mandatory, and users should not use setups that generates non-predictable randomness on the device, and ideally devices should not have this functionality.

Case-Study: Tillitis I have warmed up a bit for this. Tillitis is a new security device with interesting properties, and core to its operation is the Compound Device Identifier (CDI), essentially your Ed25519 private key (used for SSH etc) is derived from the CDI that is computed like this:
cdi = blake2s(UDS, blake2s(device_app), USS)
Let s apply the substitution threat model to this design: Consider someone replacing the Tillitis key with a malicious key during postal delivery of the key to the user, and the replacement device is identical with the real Tillitis key but implements the following key derivation function:
cdi = weakprng(UDS , weakprng(device_app), USS)
Where weakprng is a compromised algorithm that is predictable for the attacker, but still appears random. Everything will work correctly, but the attacker will be able to learn the secrets used by the user, and the user will typically not be able to tell the difference since the CDI is secret and the Ed25519 public key is not self-verifiable.

Conclusion Remember that it is impossible to fully protect against this attack, that s why it is merely a thought experiment, intended to be used during design of these devices. Consider an attacker that never gives you access to a good key and as a user you only ever use a malicious device. There is no way to have good security in this situation. This is not hypothetical, many well-funded organizations do what they can to derive people from having access to trustworthy security devices. Philosophically it does not seem possible to tell if these organizations have succeeded 100% already and there are only bad security devices around where further resistance is futile, but to end on an optimistic note let s assume that there is a non-negligible chance that they haven t succeeded. In these situations, this threat model becomes useful to improve the situation by identifying less good designs, and that s why the design mantra of mitigate harmful consequences is crucial as a takeaway from this. Let s improve the design of security devices that further the security of its users!

Arturo Borrero Gonz lez: Kubecon and CloudNativeCon 2023 Europe summary

Post logo This post serves as a report from my attendance to Kubecon and CloudNativeCon 2023 Europe that took place in Amsterdam in April 2023. It was my second time physically attending this conference, the first one was in Austin, Texas (USA) in 2017. I also attended once in a virtual fashion. The content here is mostly generated for the sake of my own recollection and learnings, and is written from the notes I took during the event. The very first session was the opening keynote, which reunited the whole crowd to bootstrap the event and share the excitement about the days ahead. Some astonishing numbers were announced: there were more than 10.000 people attending, and apparently it could confidently be said that it was the largest open source technology conference taking place in Europe in recent times. It was also communicated that the next couple iteration of the event will be run in China in September 2023 and Paris in March 2024. More numbers, the CNCF was hosting about 159 projects, involving 1300 maintainers and about 200.000 contributors. The cloud-native community is ever-increasing, and there seems to be a strong trend in the industry for cloud-native technology adoption and all-things related to PaaS and IaaS. The event program had different tracks, and in each one there was an interesting mix of low-level and higher level talks for a variety of audience. On many occasions I found that reading the talk title alone was not enough to know in advance if a talk was a 101 kind of thing or for experienced engineers. But unlike in previous editions, I didn t have the feeling that the purpose of the conference was to try selling me anything. Obviously, speakers would make sure to mention, or highlight in a subtle way, the involvement of a given company in a given solution or piece of the ecosystem. But it was non-invasive and fair enough for me. On a different note, I found the breakout rooms to be often small. I think there were only a couple of rooms that could accommodate more than 500 people, which is a fairly small allowance for 10k attendees. I realized with frustration that the more interesting talks were immediately fully booked, with people waiting in line some 45 minutes before the session time. Because of this, I missed a few important sessions that I ll hopefully watch online later. Finally, on a more technical side, I ve learned many things, that instead of grouping by session I ll group by topic, given how some subjects were mentioned in several talks. On gitops and CI/CD pipelines Most of the mentions went to FluxCD and ArgoCD. At that point there were no doubts that gitops was a mature approach and both flux and argoCD could do an excellent job. ArgoCD seemed a bit more over-engineered to be a more general purpose CD pipeline, and flux felt a bit more tailored for simpler gitops setups. I discovered that both have nice web user interfaces that I wasn t previously familiar with. However, in two different talks I got the impression that the initial setup of them was simple, but migrating your current workflow to gitops could result in a bumpy ride. This is, the challenge is not deploying flux/argo itself, but moving everything into a state that both humans and flux/argo can understand. I also saw some curious mentions to the config drifts that can happen in some cases, even if the goal of gitops is precisely for that to never happen. Such mentions were usually accompanied by some hints on how to operate the situation by hand. Worth mentioning, I missed any practical information about one of the key pieces to this whole gitops story: building container images. Most of the showcased scenarios were using pre-built container images, so in that sense they were simple. Building and pushing to an image registry is one of the two key points we would need to solve in Toolforge Kubernetes if adopting gitops. In general, even if gitops were already in our radar for Toolforge Kubernetes, I think it climbed a few steps in my priority list after the conference. Another learning was this site: https://opengitops.dev/. Group On etcd, performance and resource management I attended a talk focused on etcd performance tuning that was very encouraging. They were basically talking about the exact same problems we have had in Toolforge Kubernetes, like api-server and etcd failure modes, and how sensitive etcd is to disk latency, IO pressure and network throughput. Even though Toolforge Kubernetes scale is small compared to other Kubernetes deployments out there, I found it very interesting to see other s approaches to the same set of challenges. I learned how most Kubernetes components and apps can overload the api-server. Because even the api-server talks to itself. Simple things like kubectl may have a completely different impact on the API depending on usage, for example when listing the whole list of objects (very expensive) vs a single object. The conclusion was to try avoiding hitting the api-server with LIST calls, and use ResourceVersion which avoids full-dumps from etcd (which, by the way, is the default when using bare kubectl get calls). I already knew some of this, and for example the jobs-framework-emailer was already making use of this ResourceVersion functionality. There have been a lot of improvements in the performance side of Kubernetes in recent times, or more specifically, in how resources are managed and used by the system. I saw a review of resource management from the perspective of the container runtime and kubelet, and plans to support fancy things like topology-aware scheduling decisions and dynamic resource claims (changing the pod resource claims without re-defining/re-starting the pods). On cluster management, bootstrapping and multi-tenancy I attended a couple of talks that mentioned kubeadm, and one in particular was from the maintainers themselves. This was of interest to me because as of today we use it for Toolforge. They shared all the latest developments and improvements, and the plans and roadmap for the future, with a special mention to something they called kubeadm operator , apparently capable of auto-upgrading the cluster, auto-renewing certificates and such. I also saw a comparison between the different cluster bootstrappers, which to me confirmed that kubeadm was the best, from the point of view of being a well established and well-known workflow, plus having a very active contributor base. The kubeadm developers invited the audience to submit feature requests, so I did. The different talks confirmed that the basic unit for multi-tenancy in kubernetes is the namespace. Any serious multi-tenant usage should leverage this. There were some ongoing conversations, in official sessions and in the hallway, about the right tool to implement K8s-whitin-K8s, and vcluster was mentioned enough times for me to be convinced it was the right candidate. This was despite of my impression that multiclusters / multicloud are regarded as hard topics in the general community. I definitely would like to play with it sometime down the road. On networking I attended a couple of basic sessions that served really well to understand how Kubernetes instrumented the network to achieve its goal. The conference program had sessions to cover topics ranging from network debugging recommendations, CNI implementations, to IPv6 support. Also, one of the keynote sessions had a reference to how kube-proxy is not able to perform NAT for SIP connections, which is interesting because I believe Netfilter Conntrack could do it if properly configured. One of the conclusions on the CNI front was that Calico has a massive community adoption (in Netfilter mode), which is reassuring, especially considering it is the one we use for Toolforge Kubernetes. Slide On jobs I attended a couple of talks that were related to HPC/grid-like usages of Kubernetes. I was truly impressed by some folks out there who were using Kubernetes Jobs on massive scales, such as to train machine learning models and other fancy AI projects. It is acknowledged in the community that the early implementation of things like Jobs and CronJobs had some limitations that are now gone, or at least greatly improved. Some new functionalities have been added as well. Indexed Jobs, for example, enables each Job to have a number (index) and process a chunk of a larger batch of data based on that index. It would allow for full grid-like features like sequential (or again, indexed) processing, coordination between Job and more graceful Job restarts. My first reaction was: Is that something we would like to enable in Toolforge Jobs Framework? On policy and security A surprisingly good amount of sessions covered interesting topics related to policy and security. It was nice to learn two realities:
  1. kubernetes is capable of doing pretty much anything security-wise and create greatly secured environments.
  2. it does not by default. The defaults are not security-strict on purpose.
It kind of made sense to me: Kubernetes was used for a wide range of use cases, and developers didn t know beforehand to which particular setup they should accommodate the default security levels. One session in particular covered the most basic security features that should be enabled for any Kubernetes system that would get exposed to random end users. In my opinion, the Toolforge Kubernetes setup was already doing a good job in that regard. To my joy, some sessions referred to the Pod Security Admission mechanism, which is one of the key security features we re about to adopt (when migrating away from Pod Security Policy). I also learned a bit more about Secret resources, their current implementation and how to leverage a combo of CSI and RBAC for a more secure usage of external secrets. Finally, one of the major takeaways from the conference was learning about kyverno and kubeaudit. I was previously aware of the OPA Gatekeeper. From the several demos I saw, it was to me that kyverno should help us make Toolforge Kubernetes more sustainable by replacing all of our custom admission controllers with it. I already opened a ticket to track this idea, which I ll be proposing to my team soon. Final notes In general, I believe I learned many things, and perhaps even more importantly I re-learned some stuff I had forgotten because of lack of daily exposure. I m really happy that the cloud native way of thinking was reinforced in me, which I still need because most of my muscle memory to approach systems architecture and engineering is from the old pre-cloud days. List of sessions I attended on the first day: List of sessions I attended on the second day: List of sessions I attended on third day: The videos have been published on Youtube.

25 April 2023

B lint R czey: Improve build time of Rust, Java and Intel Fortran projects with Firebuild s new release!

Rust is a hugely popular compiled programming language and accelerating it was an important goal for Firebuild for some time. Firebuild s v0.8.0 release finally added Rust support in addition to numerous other improvements including support for Doxygen, Intel s Fortran compiler and restored javac and javadoc acceleration.

Firebuild s Rust + Cargo support Firebuild treats programs as black boxes intercepting C standard library calls and system calls. It shortcuts the program invocations that predictably generate the same outputs because the program itself is known to be deterministic and all inputs are known in advance. Rust s compiler, rustc is deterministic in itself and simple rustc invocations were already accelerated, but parallel builds driven by Cargo needed a few enhancements in Firebuild.

Cargo s jobserver Cargo uses the Rust variant of the GNU Make s jobserver to control the parallelism in a build. The jobserver creates a file descriptor from which descendant processes can read tokens and are allowed to run one extra thread or parallel process per token received. After the extra threads or processes are finished the tokens must be returned by writing to the other file descriptor the jobserver created. The jobserver s file descriptors are shared with the descendant processes via environment variables:
# rustc's environment variables
...
CARGO_MAKEFLAGS="-j --jobserver-fds=4,5 --jobserver-auth=4,5"
...
Since getting tokens from the jobserver involves reading them as nondeterministic bytes from an inherited file descriptor this is clearly an operation that would depend on input not known in advance. Firebuild needs to make an exception and ignore jobserver usage related reads and writes since they are not meant to change the build results. However, there are programs not caring at all about jobservers and their file descriptors. They happily close the inherited file descriptors and open new ones with the same id, to use them for entirely different purposes. One such program is the widely used ./configure script, thus the case is far from being theoretical. To stay on the safe side firebuild ignores jobserver fd usage only in programs which are known to use the jobserver properly. The list of the programs is now configurable in /etc/firebuild.conf and since rustc is on the list by default parallel Rust builds are accelerated out of the box!

Writable dependency dir The other issue that prevented highly accelerated Rust builds was rustc s -L dependency=<dir> parameter. This directory is populated in a not fully deterministic order in parallel builds. Firebuild on the other hand hashes directory listings of open()-ed directories treating them as inputs assuming that the directory content will influence the intercepted programs outputs. As rustc programs started in parallel scanned the dependency directory in different states depending on what other Rust compilations finished already Firebuild had to store the full directory content as an input for each rustc cache entry resulting low hit rate when rustc was started again with otherwise identical inputs. The solution here is ignoring rustc scanning the dependency directory, because the dependencies actually used are still treated as input and are checked when shortcutting rustc. With that implemented in firebuild, too, librsvg s build that uses Rust and Cargo can be accelerated by more than 90%, even on a system having 12 cores/24 threads!:
Firebuild accelerating librsvg s Rust + Cargo build from 38s to 2.8s on a Ryzen 5900X (12C/24T) system

On the way to accelerate anything Firebuild s latest release incorporated more than 100 changes just from the last two months. They unlocked acceleration of Rust builds with Cargo, fixed Firebuild to work with the latest Java update that slightly changed its behavior, started accelerating Intel s Fortran compiler in addition to accelerating gfortran that was already supported and included many smaller changes improving the acceleration of other compilers and tools. If your favorite toolchain is not mentioned, there is still a good chance that it is already supported. Give Firebuild a try and tell us about your experience! Update 1: Comparison to sccache came up in the reddit topic about Firebuild s Rust acceleration , thus by popular demand this is how sccache performs on the same project:
Firebuild 0.8.0 vs. sccache 0.4.2 accelerating librsvg s Rust + Cargo build
All builds took place on the same Ryzen 5900X system with 12 cores / 24 threads in LXC containers limited to using 1-12 virtual CPUs. A warm-up build took place before the vanilla (without any instrumentation) build to download and compile the dependency crates to measure only the project s build time. A git clean command cleared all the build artifacts from the project directory before each build and ./autogen.sh was run to measure only clean rebuilds (without autotools). See test configuration in the Firebuild performance test repository for more details and easy reproduction. Firebuild had lower overhead than sccache (2.83% vs. 6.10% on 1 CPU and 7.71% vs. 22.05% on 12 CPUs) and made the accelerated build finish much faster (2.26% vs. 19.41% of vanilla build s time on 1 CPU and 7.5% vs. 27.4% of vanilla build s time on 12 CPUs).

24 April 2023

Jonathan Dowland: Separate hledgers

In a previous blog post I described the use of virtual postings to track accidental personal/family expenses. I've always been uncomfortable with that, and in hledger 1yr I outlined a potential scheme for finally addressing the virtual posting problem. separate journals My outline built on top of continuing to maintain both personal and family financial data in the same place, but I've decided that this can't work, because the different "directions" (or signs) of accidental transactions originating from either the family or personal side can't be addressed with any kind of alternate view on the same data. To illustrate with an example. A negative balance in family:liabilities:jon means "family owes jon". A coffee bought by mistake on the family credit card will have a negative posting on the credit card, and thus a positive one on the liabilities account. ("jon owes family"). That's fine. But what about when I buy family stuff on a personal card? The other side of of the transaction is also going to have a positive sign, so it can't be posted to family:liabilities:jon: it would have to go to somewhere else, like jon:liabilities:family. Now I have two accounts which track versions of the same thing, and they cannot be combined with a simple transaction since they're looking at the same value from opposite directions (and signs). Back when I first described the problem I was using a single journal file for all my transactions. After moving to lots of separate journal files (in hledger 1yr), it's become clearer to me that I don't need to maintain the Family and Personal data together, at all: they can be entirely separate journals. getting data between journals When I moved to a new set of ledger files for 2023, I needed to carry forward the balances from 2022 in the form of "opening balance" transactions. This was achieved by a report on the 2022 data, exported as CSV, and imported into the 2023 data (all following the scheme outlined by fully-fledged hledger.)) Separate Personal and Family journals need some information from each other, and I can achieve that in the same way as for opening balances: with an export of the relevant transactions as CSV, then imported on the other side. HLedger's CSV import system is flexible enough that we can effectively invert the sign of liabilities, addressing the problem above. Worked example We start with an accidental coffee purchased on the family card (and so this belongs to the Family ledger)
2022-08-20 coffee
    liabilities:creditcard             -3
    liabilities:jon:expenses:coffee     3
I've encoded the expense category that the Personal ledger will be interested in (the last bit, expenses:coffee) as a sub-account of the liabilities category that the Family ledger is interested in1 (the first bit, liabilities:jon). When viewed on the Family side, the expense category is not interesting, and we can hide it with HLedger's alias feature2:
    alias /^liabilities:jon(.*)$/ = liabilities:jon
It then looks like this from the Family side:
2022-08-20 coffee
    liabilities:creditcard             -3
    liabilities:jon                     3
This transaction (and others like it) are exported via
hledger reg -f family/2023-back.journal liabilities:jon: -O csv \
        jon/import/family/liabilities.csv
(The trailing colon on liabilities:jon: is important here!) In the resulting CSV file, the running example transaction looks like
"55","2022-08-20","","coffee","liabilities:jon:expenses:coffee","  3.00","  3.00"
This is then converted into a journal file by hledger import. The rules file for the import is very simple: the fields date, description, account1 and amount are taken as-is; account2 is hard-coded to liabilities:family. The resulting transaction looks like
2022-08-20 coffee
    liabilities:jon:expenses:coffee     3
    liabilities:family                 -3
Before this journal is included by the main one, we have to adjust the expense account, to remove the liabilities:jon: prefix. The import rules can't do this3 , so we use another journal file as a go-between with another alias rule:
    alias /^liabilities.jon:/ =
This results, finally, in the following transaction in the Personal ledger:
2022-08-20 coffee
    expenses:coffee                     3
    liabilities:family                 -3
avoiding double-counting There's one set of transactions that we don't want to export across this divide, and that's because they're already there: any time I transfer money from myself to the family accounts (or vice versa) to address the accrued debt, the transaction is visible from both my family and personal statements. To avoid exporting these and double-counting them, I make sure those transactions don't post to an account matching the pattern used in the hledger reg report. That's what the trailing colon is for: It ensures I only export transactions which are to a sub-account of liabilities:jon, and not to the root account liabilities:jon itself: which is where I put the repayment transactions. I could instead use a more explicit sub-account like liabilities:jon:repayments or similar, since the trailing colon is quite subtle, but this works for me. Wrap up I've been really on the fence as to whether the complexity of this scheme is worth it to avoid the virtual postings. The previous scheme was much simpler. I have definitely made some mistakes with it, which didn't get caught by the double-entry rules that virtual postings ignore, but they're for small sums of money anyway. On the other hand, a lot of the "machinery" of this already existed for getting opening balances between calendar years, and the gory details are written down and hidden inside the Makefile. I also expect that I will continue to see advantages in having Family and Personal entirely separate, as they can each develop and adapt to their own needs without having to consider the other side of things every time. It's a running experiment, and time will tell if it's a good idea.

  1. This scheme was originally suggested to me by Pranesh on Twitter (described in dues), but I discounted it at the time because of the exact arrangement they suggested, not realising the broader idea might work.
  2. I've hand-waved one problem with using hledger aliases here. If we use them as described, to hide the Personal expense details, we need them to not be applied when performing the CSV-generating report. Therefore, in practise I have them in a front-most family/2023.journal file, which imports the data from another family/2023-back.journal, and the CSV export is performed on the backing journal with the data and not the alias.
  3. HLedger import rules can't manipulate the fields from the CSV a great deal, but one change I proposed and started hacking on would allow for this: to expose Regexp match-groups as interpolatable tokens: https://github.com/simonmichael/hledger/issues/2009.

20 April 2023

Simon Josefsson: Sigstore for Apt Archives: apt-cosign

As suggested in my initial announcement of apt-sigstore my plan was to look into stronger uses of Sigstore than rekor, and I m now happy to announce that the apt-cosign plugin has been added to apt-sigstore and the operational project debdistcanary is publishing cosign-statements about the InRelease file published by the following distributions: Trisquel GNU/Linux, PureOS, Gnuinos, Ubuntu, Debian and Devuan. Summarizing the commands that you need to run as root to experience the great new world:
# run everything as root: su / sudo -i / doas -s
apt-get install -y apt gpg bsdutils wget
wget -nv -O/usr/local/bin/apt-verify-gpgv https://gitlab.com/debdistutils/apt-verify/-/raw/main/apt-verify-gpgv
chmod +x /usr/local/bin/apt-verify-gpgv
mkdir -p /etc/apt/verify.d
ln -s /usr/bin/gpgv /etc/apt/verify.d
echo 'APT::Key::gpgvcommand "apt-verify-gpgv";' > /etc/apt/apt.conf.d/75verify
wget -O/usr/local/bin/cosign https://github.com/sigstore/cosign/releases/download/v2.0.1/cosign-linux-amd64
echo 924754b2e62f25683e3e74f90aa5e166944a0f0cf75b4196ee76cb2f487dd980  /usr/local/bin/cosign   sha256sum -c
chmod +x /usr/local/bin/cosign
wget -nv -O/etc/apt/verify.d/apt-cosign https://gitlab.com/debdistutils/apt-sigstore/-/raw/main/apt-cosign
chmod +x /etc/apt/verify.d/apt-cosign
mkdir -p /etc/apt/trusted.cosign.d
dist=$(lsb_release --short --id   tr A-Z a-z)
wget -O/etc/apt/trusted.cosign.d/cosign-public-key-$dist.txt "https://gitlab.com/debdistutils/debdistcanary/-/raw/main/cosign/cosign-public-key-$dist.txt"
echo "Cosign::Base-URL \"https://gitlab.com/debdistutils/canary/$dist/-/raw/main/cosign\";" > /etc/apt/apt.conf.d/77cosign
Then run your usual apt-get update and look in the syslog to debug things. This is the kind of work that gets done while waiting for the build machines to attempt to reproducibly build PureOS. Unfortunately, the results is that a meager 16% of the 765 added/modifed packages are reproducible by me. There is some infrastructure work to be done to improve things: we should use sbuild for example. The build infrastructure should produce signed statements for each package it builds: One statement saying that it attempted to reproducible build a particular binary package (thus generated some build logs and diffoscope-output for auditing), and one statements saying that it actually was able to reproduce a package. Verifying such claims during apt-get install or possibly dpkg -i is a logical next step. There is some code cleanups and release work to be done now. Which distribution will be the first apt-based distribution that includes native support for Sigstore? Let s see. Sigstore is not the only relevant transparency log around, and I ve been trying to learn a bit about Sigsum to be able to support it as well. The more improved confidence about system security, the merrier!

Jamie McClelland: Electron doesn't like negative layout coordinates

I got a second external monitor. Overkill? Probably, but I like having a dedicated space to instant messaging (now left monitor) and also a dedicated space for a web browser (right monitor). But, when I moved signal-desktop to the left monitor, clicks stopped working. I moved it back to my laptop screen, clicks started working. Other apps (like gajim) worked fine. A real mystery. I spent a lot of time on the wrong thing. I turned this monitor into portrait mode. Maybe signal doesn t like portrait mode? Nope. Maybe signal doesn t think it has enough horizontal space? Nope. Maybe signal suddently doesn t like being on an external monitor? Nope. Maybe signal will output something useful if I start it via the terminal? Nope (but that was a real distraction). Then, I discovered that mattermost desktop behaves the same way. A clue! So, now I know it s an electron app limitation, not a signal limitation. Finally I hit on the problem: Via my sway desktop, I set my laptop screen to the x,y coordinates: 0,0 (since it s in the middle). I set the left monitor to negative coordinates so it would appear on the left. Well, electron does not like that, not sure why. Now, my left monitor occupies 0,0 and the rest are adjusted to that center. I originally used negative coordinates so when I unplugged my monitors, my laptop would still be 0,0 and display all desktops properly. Fortunately, sway magically figures that all out even when the center shifts by unplugging the monitors. Hooray for sway.

Next.

Previous.