As regular readers would know, since I never stop banging on about it, I run Pwnedkeys, a service which finds and collates private keys which have been disclosed or are otherwise compromised. Until now, the only way to check if a key is compromised has been to use the Pwnedkeys API, which is not necessarily trivial for everyone. Starting today, that s changing. The next phase of Pwnedkeys is to start offering more user-friendly tools for checking whether keys being used are compromised. These will typically be web-based or command-line tools intended to answer the question is the key in this (certificate, CSR, authorized_keys file, TLS connection, email, etc) known to Pwnedkeys to have been compromised? .

Opening the Toolbox Available right now are the first web-based key checking tools in this arsenal. These tools allow you to:

1. Check the key in a PEM-format X509 data structure (such as a CSR or certificate);
2. Check the keys in an authorized_keys file you upload; and
3. Check the SSH keys used by a user at any one of a number of widely-used code-hosting sites.

Further planned tools include live checking of the certificates presented in TLS connections (for HTTPS, etc), SSH host keys, command-line utilities for checking local authorized_keys files, and many other goodies.

If You Are Intrigued By My Ideas and wish to subscribe to my newsletter, now you can! I m not going to be blogging every little update to Pwnedkeys, because that would probably get a bit tedious for readers who aren t as intrigued by compromised keys as I am. Instead, I ll be posting every little update in the Pwnedkeys newsletter. So, if you want to keep up-to-date with the latest and greatest news and information, subscribe to the newsletter.

Supporting Pwnedkeys All this work I m doing on my own time, and I m paying for the infrastructure from my own pocket. If you ve got a few dollars to spare, I d really appreciate it if you bought me a refreshing beverage. It helps keep the lights on here at Pwnedkeys Global HQ.

The final piece of my 5G (well, 4G) based based backup internet connection I needed to sort out was IPv6. While Three do support IPv6 in their network they only seem to enable it for certain devices, and the MC7010 is not one of those devices, even though it also supports IPv6. I use v6 a lot - over 50% of my external traffic, last time I looked. One suggested option was that I could drop the IPv6 Router Advertisements when the main external link went down, but I have a number of internal services that are only presented on v6 addresses so I needed to ensure clients in the house continued to have access to those. As it happens I ve used the Hurricane Electric IPv6 Tunnel Broker in the past, so my pass was re-instating that. The 5G link has a real external IPv4 address, and it s possible to update the endpoint using a simple HTTP GET. I added the following to my /etc/dhcp/dhclient-exit-hooks.d/modem-interface-route where we are dealing with an interface IP change: I needed some additional configuration to bring things up, so /etc/network/interfaces got the following, configuring the 6in4 tunnel as well as the low preference default route, and source routing via the 5g table, similar to IPv4: We need to deal with IPv4 changes in for the tunnel endpoint, so modem-interface-route also got: /etc/nftables.conf had to be taught to accept the 6in4 packets from the tunnel in the input chain: Finally, I had to engage in something I never thought I d deal with; IPv6 NAT. HE provide a /48, and my FTTP ISP provides me with a /56, so this meant I could do a nice stateless 1:1 mapping: This works. Mostly. The problem is that HE, not unreasonably, expect your IPv4 address to be pingable. And it turns out Three have some ranges that this works on, and some that it doesn t. Which means it s a bit hit and miss whether you can setup the tunnel. I spent a while trying to find an alternative free IPv6 tunnel provider with a UK endpoint. There s less call for them these days, so I didn t manage to find any that actually worked (or didn t have a similar pingable requirement). I did consider whether I wanted to end up with routes via a VM, as I described in the failover post, but looking at costings for VMs with providers who could actually give me an IPv6 range I decided the cost didn t make it worthwhile; the VM cost ended up being more than the backup SIM is costing monthly. Finally, it turns out happy eyeballs mostly means that when the 5G ends up on an IP that we can t setup the IPv6 tunnel on, things still mostly work. Browser usage fails over quickly and it s mostly my own SSH use that needs me to force IPv4. Purists will groan, but this turns out to be an acceptable trade-off for me, at present. Perhaps if I was seeing frequent failures the diverse routes approach to a VM would start to make sense, but for now I m pretty happy with the configuration in terms of having a mostly automatic backup link take over when the main link goes down.

Review: Lyorn, by Steven Brust

Series:	Vlad Taltos #17
Publisher:	Tor
Copyright:	2024
ISBN:	1-4668-8971-3
Format:	Kindle
Pages:	274

Lyorn is the 17th Vlad Taltos book and a direct sequel to 2014's Hawk. (Yes, actual main story progress!) You do not want to start reading here; you would be hopelessly confused. When this series is complete, I want to re-read the entire thing from the beginning and pick up more of the bits I missed the first time. Vlad is not, in fact, free to see his friends and get entangled in imperial politics again as I thought after Hawk. Despite the successes of that story, there is one remaining small problem: incredibly powerful magic users still want to kill him. His immediate solution is to shelter in a theater, since Draegaran theaters are well-known for their excellent magical shielding. This works well enough at first, but the theater is rehearsing a play about Draegaran politics that is highly offensive to the Lyorn and the theater may be shut down because of it. Vlad's enemies are also willing to lean on his friends to find him and kill him. This series continues to be thoroughly enjoyable. Lyorn is "just" more of Vlad being Vlad, meddling in everyone's business and coming up with elaborate plans with too many moving parts that he somehow manages to pull off, but I'll happily read lots of books like that. Vlad is both anxious and grumpy, both of which give the plot some needed tension without being overwhelming. There are no truly major world-building revelations here (or, if there are, I missed them), but there's a lot of processing of what the reader learned in Tsalmoth. It's increasingly looking like the payoff from those revelations is going to be the series finale. This is the first Vlad book that contains solid confirmation of where the series as a whole is headed. Brust mentioned some time ago that the last book is titled The Last Contract, and Lyorn comes close to stating explicitly what that contract will be. I am sure that it will be more complicated than it appears now and there are misdirections yet to come, but I am excited to see where Brust takes this idea. Vlad has been insistently apolitical for much of the series, meddling in politics only when he has to or to help his friends and otherwise treating it as a system that he has to navigate and survive. That was the root of the conflict in Teckla, all the way back at the start of the series. This may be starting to change, and when Brust ties it together with the Jenoine, the Great Weapon Godslayer, and the rest of the world-building he's cued up, the results are going to be explosive. Two books left: Chreotha and The Last Contract. I can hardly wait. In every Vlad book, Brust plays some sort of structural game. This time, befitting the setting, it's a musical. The action is interspersed with quotes from a fictional history about the play the theater is putting on, a work called Song of the Presses about political censorship during the reign of a Lyorn emperor in the 14th cycle, thousands of years before the time of this book. This was, at times, nearly as interesting as the main plot. The chapters are also numbered like the acts and scenes of a play, although this I didn't notice as much since books often have that structure anyway. Since this is a musical, there are also songs. Specifically, each chapter is introduced by a parody of songs from various musicals in our world, rewritten so that they fit within Brust's fictional musical. Brust is also a musician and a filker, so these songs are actually good, or at least they amused me a great deal. I'm not much of a musical fan and I still could hear the tune playing when I read most of them. Lyorn is not so good that I would rave about it. It's one of those functional connective books of a series that advances the plot, tells a good story, and has some fun along the way. The guns on the mantelpiece of this world have not gone off yet, and Vlad is still maneuvering into position. But it's looking like we're going to get the conclusion, and it's going to be spectacular. If you have read this far, you will want to keep reading. Followed by Chreotha, which may be a bit of a wait because apparently Brust is going to write The Last Contract first to make sure he ties up loose ends properly. Rating: 7 out of 10

I received a mail today from Google (noreply-findmydevice@google.com) notifying me that they would unconditionally enable the Find my device functionality I have been repeatedly marking as unwanted in my Android phone. The mail goes on to explain this functionality works even when the device is disconnected, by Bluetooth signals (aha, so turn off Bluetooth will no longer turn off Bluetooth? Hmmm ) Of course, the mail hand-waves that only I can know the location of my device. Google cannot see or use it for other ends . First, should we trust this blanket statement? Second, the fact they don t do it now means they won t ever? Not even if law enforcement requires them to? The devices will be generating this information whether we want it or not, so it s just a matter of opening the required window. Of course, it is a feature many people will appreciate and find useful. And it s not only for finding lost (or stolen) phones, but the mail also mentions tags can be made available to store in your wallet, bike, keys or whatever. But it should be opt-in. As it is, it seems it s not even to opt out of it.

IPv4 is an expensive resource. However, many content providers are still IPv4-only. The most common reason is that IPv4 is here to stay and IPv6 is an additional complexity.¹ This mindset may seem selfish, but there are compelling reasons for a content provider to enable IPv6, even when they have enough IPv4 addresses available for their needs.

Why ISPs need IPv6? Providing a public IPv4 address to each customer is quite costly when each IP address costs US$40 on the market. For fixed access, some consumer ISPs are still providing one IPv4 address per customer.² Other ISPs provide, by default, an IPv4 address shared among several customers. For mobile access, most ISPs distribute a shared IPv4 address. There are several methods to share an IPv4 address:³

NAT44

The customer device is given a private IPv4 address, which is translated to a public one by a service provider device. This device needs to maintain a state for each translation.

464XLAT and DS-Lite

The customer device translates the private IPv4 address to an IPv6 address or encapsulates IPv4 traffic in IPv6 packets. The provider device then translates the IPv6 address to a public IPv4 address. It still needs to maintain a state for the NAT64 translation.

Lightweight IPv4 over IPv6, MAP-E, and MAP-T

The customer device encapsulates IPv4 in IPv6 packets or performs a stateless NAT46 translation. The provider device uses a binding table or an algorithmic rule to map IPv6 tunnels to IPv4 addresses and ports. It does not need to maintain a state.

Solutions to share an IPv4 address across several customers. Some of them require the ISP to keep state, some don't.

All these solutions require a translation device in the ISP s network. This device represents a non-negligible cost in terms of money and reliability. As half of the top 1000 websites support IPv6 and the biggest players can deliver most of their traffic using IPv6,⁴ ISPs have a clear path to reduce the cost of translation devices: provide IPv6 by default to their customers.

Why content providers need IPv6? Content providers should expose their services over IPv6 primarily to avoid going through the ISP s translation devices. This doesn t help users who don t have IPv6 or users with a non-shared IPv4 address, but it provides a better service for all the others. Why would the service be better delivered over IPv6 than over IPv4 when a translation device is in the path? There are two main reasons for that:⁵

1. Translation devices introduce additional latency due to their geographical placement inside the network: it is easier and cheaper to only install these devices at a few points in the network instead of putting them close to the users.
2. Translation devices are an additional point of failure in the path between the user and the content. They can become overloaded or malfunction. Moreover, as they are not used for the five most visited websites, which serve their traffic over IPv6, the ISPs may not be incentivized to ensure they perform as well as the native IPv6 path.

Looking at Google statistics, half of the users reach Google over IPv6. Moreover, their latency is lower.⁶ In the US, all the nationwide mobile providers have IPv6 enabled. For France, we can refer to the annual ARCEP report: in 2022, 72% of fixed users and 60% of mobile users had IPv6 enabled, with projections of 94% and 88% for 2025. Starting from this projection, since all mobile users go through a network translation device, content providers can deliver a better service for 88% of them by exposing their services over IPv6. If we exclude Orange, which has 40% of the market share on consumer fixed access, enabling IPv6 should positively impact more than 55% of fixed access users.

---

In conclusion, content providers aiming for the best user experience should expose their services over IPv6. By avoiding translation devices, they can ensure fast and reliable content delivery. This is crucial for latency-sensitive applications, like live streaming, but also for websites in competitive markets, where even slight delays can lead to user disengagement.

---

1. A way to limit this complexity is to build IPv6 services and only provide IPv4 through reverse proxies at the edge.
2. In France, this includes non-profit ISPs, like FDN and Milkywan. Additionally, Orange, the previously state-owned telecom provider, supplies non-shared IPv4 addresses. Free also provides a dedicated IPv4 address for customers connected to the point-to-point FTTH access.
3. I use the term NAT instead of the more correct term NAPT. Feel free to do a mental substitution. If you are curious, check RFC 2663. For a survey of the IPv6 transition technologies enumerated here, have a look at RFC 9313.
4. For AS 12322, Google, Netflix, and Meta are delivering 85% of their traffic over IPv6. Also, more than half of our traffic is delivered over IPv6.
5. An additional reason is for fighting abuse: blacklisting an IPv4 address may impact unrelated users who share the same IPv4 as the culprits.
6. IPv6 may not be the sole reason the latency is lower: users with IPv6 generally have a better connection.

I wrote about my blog writing methodology back April 2021. My writing method has undergone a significant shift now, so thought about writing this update. New blog topics are added to my note-taking app quite frequently now. Occasionally going through the list, I merge topics, change order to prioritize certain topics or purely drop ideas which seems not worth a write-up. Due to this, I have the liberty to work on blogs according to mood. Writing the last one was tiring, so I chose to work on an easy one, i.e. this blog now. Topic decided, everything starts on etherpad now. Etherpad has this nice font and sync feature, which helps me write from any device of choice. Actual writing usually happens in the morning, right after I wake up. For most topics, I quickly jot down pointers and keep on expanding them over the course of multiple days at a leisurely pace. Though, sometime it adds too many pieces in the puzzle and takes additional time to put everything in flow. New pointer addition keeps on happening along with writing. Nowadays, pictures too dot my blog, which I rarely use to do earlier. I have come to believe on less usage of external links. These breaks the flow of readers. If someone is sufficiently motivated to learn more about something, finding useful sources isn t. As the first draft comes into being, I run it through LanguageTool for spelling corrections (which typically are many) and fixing grammatical issues. Post that, for the first time I read the complete write-up in one go for formation of coherent storyline, moving paragraphs around for form a structure , adding explainers wherever something new or unexplained is introduced, removing elaborate sentences, making amends wherever required and moving paragraphs around for forming structure. Another round of LanguageTool follows. All set now, I try to space out my final read before publishing, which helps find additional mistakes or loopholes. When everything is set, I do hugo to generate the site and rsync everything to the web server. A final git sync closes the publication part. After a day or two, I come back to read the blog on the website. This entails another round finding and fixing trivial mistakes. After this, it s set for good. Nowadays, in addition to being on my blog, everything is syndicated on Planet FSCI and Planet Debian, which has given it more visibility. As someone who s into infrastructure and Internet as a lot, I do pay attention to logs on my server, but as a disconnected exercise to if the blog is being read or not. More hits on the blog doesn t translate to any gratification for me, at least for writers point of view. Occasionally, people do mention my blog, which does flatter me. Four years and nearly a hundred posts later, I still wonder how I kept on writing for this long.

Review: And the Stars Will Sing, by Michelle Browne

Series:	The Meaning Wars #1
Publisher:	Michelle Browne
Copyright:	2012, 2021
Printing:	2021
ASIN:	B0075G7GEA
Format:	Kindle
Pages:	85

And the Stars Will Sing is a self-published science fiction novella, the first of a (currently) five book series. I believe it may be Browne's first publication, although I don't have a good data source to confirm. Crystal Weiss is a new graduate from Mars, about to leave the solar system to her first job assignment: installation of a permanent wormhole in the vicinity of Messier 14. Her expertise is the placement calculations. The heavy mathematical lifting is of course done by computers, but humans have to do the mapping and some of the guidance. And the Stars Will Sing is an epistolary novel, told in the form of her letters to her friend Sarah. I feel bad when I stumble across a book like this. I want to stick with my habit of writing a review of each book I read, but it's one thing to pan a bad book by a famous author and another thing to pick on a self-published novella that I read due to some recommendation or mention whose details I've forgotten. Worse, I think this wasn't even the recommended book; I looked up the author, saw that the first of a series was on sale, and thought "oh, hey, I like epistolary novels and I'm in the mood for some queer space opera." This book didn't seem that queer (there is a secondary lesbian relationship but the main relationship seemed rather conventional), but I'll get to the romance in a moment. I was not the reader for this book. There's a reason why most of the books I read are from traditional publishers; I'm too critical of a reader for a lot of early self-published work. It's not that I dislike self-publishing as a concept many self-published books are excellent and the large publishers have numerous problems but publishers enforce a quality bar. Inconsistently, unfairly, and by rejecting a lot of good work, but still, they do. I'm fairly sure traditional publishers would have passed on this book; the quality of the writing isn't there yet. (It's certainly a better book than I could have written! But that's why I'm writing my reviews over in my quiet corner of the Internet and not selling fiction to other people.) The early chapters aren't too bad, although they have a choppy, cliched style that more writing experience usually smoothes out. The later chapters have more dialogue, enough that I started wondering how Crystal could remember that much dialogue verbatim to put into a letter, and it's not good. All of the characters talk roughly the same (even the aliens), the dialogue felt even more cliched than the rest of the writing, and I started getting distracted by the speech tags. Crystal comes across as very young, impulsive, and a drama magnet who likes being all up in her coworkers' business. None of these are objective flaws in the book, but I could tell early on that I was going to find her annoying. She has a heavily-foreshadowed enemies-to-lovers thing with one of her male coworkers. Her constant complaining about him at the start of the story was bad enough, but the real problem is that in the very few places where he has more personality than plastic lawn furniture, he's being obnoxious to Crystal. I'm used to being puzzled by a protagonist's choice in love interests, but this one felt less like an odd personality choice and more a lack of writing skill. Even if the relationship is being set up for failure (not true by the end of this book), you've got to help me understand what the protagonist saw in him or was getting out of the relationship. The plot was so predictable that it ironically surprised me. I was sure that some sort of twist or complication was coming, but no. I will give Browne some credit for writing a slightly more realistic character reaction to violence than most SF authors, but there was nothing in the plot to hold my interest. The world-building was generic science fiction with aliens. It had a few glimmers of promise, but there was some sort of psychic hand-waving involved in siting wormholes that didn't work for me and the plot climax made no sense to me whatsoever. This is the kind of bad book that I don't want to hold against the writer. Twelve years later and with numerous other novels and novellas under her belt, her writing is probably much better. I do think this book would have benefited from an editor telling her it wasn't good enough for publication yet, but that's not how the Kindle self-publishing world works. Mostly, this is my fault: I half-followed a recommendation into an area of publishing that I know from past experience I should avoid without a solid review from an equally critical reader. Followed by The Stolen, a two-story collection. Rating: 3 out of 10

Shortly before coming back from Argentina, I got news of a very interesting set of little machines, the MilkV Duo. The specs looked really interesting and fun to play with, particularly those of the bigger model, Milk-V DUO S Some of the highlights:

• The SG2000 SoC is a Dual-architecture beast. A hardware switch controls whether the CPU is an ARM or a RISC-V.
  • Not only that: It has a second (albeit lesser) RISC-V core that can run independently. They mention this computer can run simultaneously Linux and FreeRTOS!
• 512MB RAM
• Sweet form factor (4.2 4.2cm)
• Peeking around their Web site, it is one of the most open and well documented computers in their hardware range.
  • Schematics at different levels of detail
  • Datasheet. It s preliminary, and it s written in Mandarin, but the information it contains is quite descriptive and clear.
  • Full specifications (all 710 pages of it!) Again, it has a lot of Mandarin in it, but it conveys a lot of useful information.

Naturally, for close to only US$12 (plus shipping) for the configuration I wanted I bought one, and got it delivered in early May. The little box sat on my desk for close to six weeks until I had time to start tinkering with it I must say I am surprised. Not only the little bugger delivers what it promises, but it is way more mature than what I expected: It can be used right away without much tinkering with! I mean, I have played with it for less than an hour by now, and I ve even managed to get (almost) regular Debian working. Milk-V distributes a simple, 58.9MB compressed Linux image, based on Buildroot, a simple Linux image generator mostly used for embedded applications, as well as its source tree. I thought that would be a good starting point to work on setting up a minimal Debian filesystem, as I did with the CuBox-i4Pro ten years ago, and maybe even to grow towards a more official solution, akin to what we currently have for the Raspberry Pi family Until I discovered what looks like a friendly and very active online community of Milk-V users! I haven t yet engaged in it, but I stumbled across a thread announcing the availability of Debian images for the Milk-V family. And yes, it feels like a very normal Debian system. /etc/apt/sources.list does point to a third-party repository, but it s for only four packages, all related to pinmux controlfor CVITEK chips. It does feel like a completely normal Debian system! It is not as snappy and fast to load as Buildroot, but given Debian s generality, that s completely as expected. Even the wireless network, one of the usual pain points, works just out of the box! The Debian images can be built or downloaded from this Git repository. In case you wonder how is this system booting or what hardware it detects, I captured two boot logs:

• Milk- Duo S booting with Buildroot (24K)
• Milk- Duo S booting with Debian (49K)

Hello folks, I ve been working this year on better integrating NVIDIA hardware with the Google Cloud Dataproc product (Hadoop on Google Cloud) running the default cluster node image. We have an open bug[1] in the initialization-actions repo regarding creation failures upon enabling secure boot. This is because with secure boot, kernel driver code has its signature verified before insmod places the symbols into kernel memory. The verification process involves reading trust root certificates from EFI variables, and validating that the signatures on the kernel driver either a) were made directly by one of the certificates in the boot sector or b) were made by certificates which chain up to one of them. This means that Dataproc disk images must have a certificate installed into them. My work on the internals will likely start producing images which have certificates from Google in them. In the meantime, however, our users are left without a mechanism to have both secure boot enabled and install out-of-tree kernel modules such as the NVIDIA GPU drivers. To that end, I ve got PR #83[2] open with the GoogleCloudDataproc/custom-images github repository. This PR introduces a new argument to the custom image creation script, trusted-cert , the argument of which is the path to a DER-encoded certificate to be included in the certificate database in the EFI variables of the disk s boot sector. I ve written up the instructions on creating a custom image with a trusted certificate here: https://github.com/cjac/custom-images/blob/secure-boot-custom-image/examples/secure-boot/README.md Here is a set of commands that can be used to create a Dataproc custom image with certificate installed to the EFI s db variable. You can run these commands from the root directory of a checkout such as this:

git clone https://github.com/cjac/custom-images.git --branch secure-boot-custom-image --single-branch
pushd custom-images
PROJECT_ID=your-project-here
PROJECT_NUMBER=your-project-nnnn-here
my_bucket=your-bucket-here
custom_image_zone=your-zone-here
gcloud projects add-iam-policy-binding $ PROJECT_ID  \
        --member=serviceAccount:$ PROJECT_NUMBER -compute@developer.gserviceaccount.com \
        --role=roles/secretmanager.secretAccessor
gcloud config set project $ PROJECT_ID 
gcloud auth login
eval $(bash examples/secure-boot/create-key-pair.sh)
metadata="public_secret_name=$ public_secret_name "
metadata="$ metadata ,private_secret_name=$ private_secret_name "
metadata="$ metadata ,secret_project=$ secret_project "
metadata="$ metadata ,secret_version=$ secret_version "
#dataproc_version=2.1-debian11
dataproc_version=2.2-debian12
#customization_script=examples/secure-boot/install-nvidia-driver-debian11.sh
customization_script=examples/secure-boot/install-nvidia-driver-debian12.sh
#image_name="nvidia-open-kernel-bullseye-$(date +%F)"
image_name="nvidia-open-kernel-bookworm-$(date +%F)"
disk_size_gb="50"
python generate_custom_image.py \
    --image-name $ image_name  \
    --dataproc-version $ dataproc_version  \
    --trusted-cert "tls/db.der" \
    --customization-script $ customization_script  \
    --metadata "$ metadata " \
    --zone "$ custom_image_zone " \
    --disk-size "$ disk_size_gb " \
    --no-smoke-test \
    --gcs-bucket "$ my_bucket "
popd

I d love to hear your feedback! [1] https://github.com/GoogleCloudDataproc/initialization-actions/issues/1058
[2] https://github.com/GoogleCloudDataproc/custom-images/pull/83

Leonardo and I are happy to annunce that a new version 0.3.8 of our nanotime package arrived on CRAN today. It is the first release in over 1 1/2 years. nanotime relies on the RcppCCTZ package (as well as the RcppDate package for additional C++ operations) and offers efficient high(er) resolution time parsing and formatting up to nanosecond resolution, using the bit64 package for the actual integer64 arithmetic. Initially implemented using the S3 system, it has benefitted greatly from a rigorous refactoring by Leonardo who not only rejigged nanotime internals in S4 but also added new S4 types for periods, intervals and durations. This release responds to a number of enhancements including a new paramter accurate for POSIXct to nanotime conversions, a vector date converter, a switch to double return value when durations objects are dividded as well as a small battery of CRAN requests for changes and updates. This started with a move away from the now non-API function SET_S4_OBJECT which has been replaced by use of Rf_asS4. We also no longer need a custom compiler flag on Windows (where for some reasons nobody understands or remembers, bitfields are not packed) to small enhancements of manual page formatting and last-but-not-least avoidance of some new UBSAN warnings. The NEWS snippet has the full details.

Changes in version 0.3.8 (2024-06-19)

• Time format documentation now has a reference to RcppCCTZ
• The package no longer sets a default C++ compilation standard of C++11 (Dirk initially in #114, and later switched to C++17)
• New accurate parameter for conversion from POSIXct to nanotime (Davor Josipovic and Leonardo in #116 closing #115)
• The as.Date() function is now vectorized and can take a TZ argument (Leonardo and Dirk in #119 closing #118)
• Use of internal function SET_S4_OBJECT has been replaced by API function Rf_asS4 (Leonardo in #121 closing #120)
• An nanoduration / nanoduration expression now returns a double (Leonardo in #122 closing #117)
• Bitfield calculations no longer require an Windows-only compiler switch (Leonardo in #124)
• A simple manual page format nag involving has been addressed (Dirk in #126 fixing #125)
• An set of tests tickling an UBSAN issue via Rcpp code no longer run unless CI is set (Dirk in #127 fixing #123)

Thanks to my CRANberries, there is a diffstat report for this release. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository and all documentation is provided at the nanotime documentation site. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

As I m gearing towards setting up a Free Software download mirror in India, it occurred to me that I haven t chronicled the work and motivation behind setting up the original mirror in the first place. Also, seems like it would be good to document stuff here for observing the progression, as the mirror is going multi-country now. Right now, my existing mirror i.e., mirrors.de.sahilister.net (was mirrors.sahilister.in), is hosted in Germany and serves traffic for Termux, NomadBSD, Blender, BlendOS and GIMP. For a while in between, it hosted OSMC project mirror as well. To explain what is a Free Software download mirror thing is first, I ll quote myself from work blog -

As most Free Software doesn t have commercial backing and require heavy downloads, the concept of software download mirrors helps take the traffic load off of the primary server, leading to geographical redundancy, higher availability and faster download in general.

So whenever someone wants to download a particular (mirrored) software and click download, upstream redirects the download to one of the mirror server which is geographical (or in other parameters) nearby to the user, leading to faster downloads and load sharing amongst all mirrors. Since the time I got into Linux and servers, I always wanted to help the community somehow, and mirroring seemed to be the most obvious thing. India seems to be a country which has traditionally seen less number of public download mirrors. IITB, TiFR, and some of the public institutions used to host them for popular Linux and Free Softwares, but they seem to be diminishing these days. In the last months of 2021, I started using Termux and saw that it had only a few mirrors (back then). I tried getting a high capacity, high bandwidth node in budget but it was hard in India in 2021-22. So after much deliberation, I decided to go where it s available and chose a German hosting provider with the thought of adding India node when conditions are favorable (thankfully that happened, and India node is live too now.). Termux required only 29 GB of storage, so went ahead and started mirroring it. I raised this issue in Termux s GitHub repository in January 2022. This blog post chronicles the start of the mirror. Termux has high request counts from a mirror point of view. Each Termux client, usually checks every mirror in selected group for availability before randomly selecting one for download (only other case is when client has explicitly selected a single mirror using termux-repo-change). The mirror started getting thousands of requests daily due to this but only a small percentage would actually get my mirror in selection, so download traffic was lower. Similar thing happened with OSMC too (which I started mirroring later). With this start, I started exploring various project that would be benefit from additional mirrors. Public information from Academic Computer Club in Ume s mirror and Freedif s mirror stats helped to figure out storage and bandwidth requirements for potential projects. Fun fact, Academic Computer Club in Ume (which is one of the prominent Debian, Ubuntu etc.) mirror, now has 200 Gbits/s uplink to the internet through SUNET. Later, I migrated to a different provider for better speeds and added LibreSpeed test on the mirror server. Those were fun times. Between OSMC, Termux and LibreSpeed, I was getting almost 1.2 millions hits/day on the server at its peak, crossing for the first time a TB/day traffic number. Next came Blender, which took the longest time to set up of around 9 10 months. Blender had a push-trigger requirement for rsync from upstream that took quite some back and forth. It now contributes the most amount of traffic on the mirror. On release days, mirror does more than 3 TB/day and normal days, it hovers around 2 TB/day. Gimp project is the latest addition. At one time, the mirror traffic touched 4.97 TB/day traffic number. That s when I decided on dropping LibreSpeed server to solely focus on mirroring for now, keeping the bandwidth allotment for serving downloads only. The mirror projects selection grew organically. I used to reach out many projects discussing the need of for additional mirrors. Some projects outright denied mirroring request as Germany already has a good academic mirrors boosting 20-25 Gbits/s speeds from FTP era, which seems fair. Finding the niche was essential to only add softwares, which would truly benefit from additional capacity. There were months when nothing much would happen with the mirror, rsync would continue to update the mirror while nginx would keep on serving the traffic. Nowadays, the mirror pushes around 70 TB/month. I occasionally check logs, vnstat, add new security stuff here and there and pay the bills. It now saturates the Gigabit link sometimes and goes beyond that, peaking around 1.42 Gbits/s (the hosting provider seems to be upping their game). The plan is to upgrade the link to better speeds. On the way, learned quite a few things like -

• IPv6 and exposing rsync module due to OSMC requirement.
• Implementing user with restricted access to only trigger rsync, basically make rsync pull trigger based due to Blender.
• Iterating over right client response size for LibreSpeed test project.
• Mistakenly identifying torrent traffic for BlendOS as DDoS and blocking it for quite a few months. BlendOS added loads of hits for torrent traffic, making my mirror also serve as web seed. Web seeds in conjunction with normal seeds surely is a good combination for serving downloads as it combines best of both world, general availability of web seed/mirror and benefit of traditional seeds to maximize download speeds at user end.
• Handling abusive traffic (a lot of it to be frank). The approach is more of a whack a mole right now, which I want to improve and automate.
• Most of the traffic on non-Linux/BSD operating systems serving mirrors (like mine) is for people on Windows and Mac asking for EXEs and DMGs. Mostly because package repositories carry software distribution load for Linux/BSD OSs and partly because the number of Windows/Mac users are quite high compared to other OSs.
• Load balancing through DNS and HTTP redirection (which I would implement in my India mirror now) to better maximize available resources.

In hindsight, the statistics look amazing, hundreds of TBs of traffic served from the mirror, month after month. That does show that there s still an appetite for public mirrors in time of commercially donated CDNs and GitHub. The world could have done with one less mirror, but it saved some time, lessened the burden for others, while providing redundancy and traffic localization with one additional mirror. And it s fun for someone like me who s into infrastructure that powers the Internet. Now, I ll try focusing and expanding the India mirror, which in itself started pushing almost half a TB/day. Long live Free Software and public download mirrors.

It s been a while since I played with something new been stuck in a bit of a rut with radios recently - working on refining and debugging stuff I mostly understand for the time being. The other day, I was out getting some food and I idly wondered how the restaurant pager system worked. Idle curiosity gave way to the realization that I, in fact, likely had the means and ability to answer this question, so I bought the first set of the most popular looking restaurant pagers I could find on eBay, figuring it d be a fun multi-week adventure.

Order up! I wound up buying a Retekess brand TD-158 Restaurant Pager System (they looked like ones I d seen before and seemed to be low-cost and popular), and quickly after, had a pack of 10 pagers and a base station in-hand. The manual stated that the radios operated at 433 MHz (cool! can do! Love a good ISM band device), and after taking an inital read through the manual for tips on the PHY, I picked out a few interesting things. First is that the base station ID was limited to 0-999, which is weird because it means the limiting factor is likely the base-10 display on the base station, not the protocol we need enough bits to store 999 at least 10 bits. Nothing else seemed to catch my eye, so I figured may as well jump right to it. Not being the type to mess with success, I did exactly the same thing as I did in my christmas tree post, and took a capture at 433.92MHz since it was in the middle of the band, and immediately got deja-vu. Not only was the signal at 433.92MHz, but throwing the packet into inspectrum gave me the identical plot of the OOK encoding scheme. Not just similar identical. The only major difference was the baud rate and bit structure of the packets, and the only minor difference was the existence of what I think is a wakeup preamble packet (of all zeros), rather than a preamble symbol that lasted longer than usual PHY symbol (which makes this pager system a bit easier to work with than my tree, IMHO). Getting down to work, I took some measurements to determine what the symbol duration was over the course of a few packets, I was able to determine the symbol rate was somewhere around 858 microseconds (0.000858 seconds per symbol), which is a weird number, but maybe I m slightly off or there s some larger math I m missing that makes this number satisfyingly round (internal low cost crystal clock or something? I assume this is some hardware constrait with the pager?) Anyway, good enough. Moving along, let s try our hand at a demod let s just assume it s all the same as the chrismas tree post and demod ones and zeros the same way here. That gives us 26 bits:

00001101110000001010001000

Now, I know we need at least 10 bits for the base station ID, some number of bits for the pager ID, and some bits for the command. This was a capture of me hitting call from a base station ID of 55 to a pager with the ID of 10, so let s blindly look for 10 bit chunks with the numbers we re looking for:

0000110111 0000001010 001000

Jeez. First try. 10 bits for the base station ID (55 in binary is 0000110111), 10 bits for the pager ID (10 in binary is 0000001010), which leaves us with 6 bits for a command (and maybe something else too?) which is 8 here. Great, cool, let s work off that being the case and revisit it if we hit bugs. Besides our data packet, there s also a preamble packet that I ll add in, in case it s used for signal detection or wakeup or something which is fairly easy to do since it s the same packet structure as the above, just all zeros. Very kind of them to leave it with the same number of bits and encoding scheme it s nice that it can live outside the PHY. Once I got here, I wrote a quick and dirty modulator, and was able to ring up pagers! Unmitigated success and good news only downside was that it took me a single night, and not the multi-week adventure I was looking for. Well, let s finish the job and document what we ve found for the sake of completeness.

Boxing everything up My best guess on the packet structure is as follows:

base id

argument

command

For a call or F2 operation, the argument is the Pager s ID code, but for other commands it s a value or an enum, depending. Here s a table of my by-hand demodulation of all the packet types the base station produces:

Type	Cmd Id	Description
Call	8	Call the pager identified by the id in argument
Off	60	Request any pagers on the charger power off when power is removed, argument is all zero
F2	40	Program a pager to the specified Pager ID (in argument) and base station
F3	44	Set the reminder duration in seconds specified in argument
F4	48	Set the pager's beep mode to the one in argument (0 is disabled, 1 is slow, 2 is medium, 3 is fast)
F5	52	Set the pager's vibration mode to the one in argument (0 is disabled, 1 is enabled)

Kitchen s closed for the night I m not going to be publishing this code since I can t think of a good use anyone would have for this besides folks using a low cost SDR and annoying local resturants; but there s enough here for folks who find this interesting to try modulating this protocol on their own hardware if they want to buy their own pack of pagers and give it a shot, which I do encourage! It s fun! Radios are great, and this is a good protocol to hack with it s really nice. All in all, this wasn t the multi-week adventure I was looking for, this was still a great exercise and a fun reminder that I ve come a far way from when I ve started. It felt a lot like cheating since I was able to infer a lot about the PHY because I d seen it before, but it was still a great time. I may grab a few more restaurant pagers and see if I can find one with a more exotic PHY to emulate next. I mean why not, I ve already got the thermal printer libraries working

Whenever a large corporation disgorges the personal information of millions of people onto the Internet, there is a standard playbook that is followed. Security is our top priority . Passwords were hashed . No credit card numbers were disclosed . record scratch Let s talk about that last one a bit.

A Case Study This post could have been written any time in the past well, decade or so, really. But the trigger for my sitting down and writing this post is the recent breach of wallet-finding and criminal-harassment-enablement platform Tile. As reported by Engadget, a statement attributed to Life360 CEO Chris Hulls says

The potentially impacted data consists of information such as names, addresses, email addresses, phone numbers, and Tile device identification numbers.

But don t worry though; even though your home address is now public information

It does not include more sensitive information, such as credit card numbers

Aaaaaand here is where I get salty.

Why Credit Card Numbers Don t Matter Describing credit card numbers as more sensitive information is somewhere between disingenuous and a flat-out lie. It was probably included in the statement because it s part of the standard playbook. Why is it part of the playbook, though? Not being a disaster comms specialist, I can t say for sure, but my hunch is that the post-breach playbook includes this line because (a) credit cards are less commonly breached these days (more on that later), and (b) it s a way to insinuate that all your financial data is safe, no need to worry without having to say that (because that statement would absolutely be a lie). The thing that not nearly enough people realise about credit card numbers is:

1. The credit card holder is not usually liable for most fraud done via credit card numbers; and
2. In terms of actual, long-term damage to individuals, credit card fraud barely rates a mention. Identity fraud, Business Email Compromise, extortion, and all manner of other unpleasantness is far more damaging to individuals.

Why Credit Card Numbers Do Matter Losing credit card numbers in a data breach is a huge deal but not for the users of the breached platform. Instead, it s a problem for the company that got breached. See, going back some years now, there was a wave of huge credit card data breaches. If you ve been around a while, names like Target and Heartland will bring back some memories. Because these breaches cost issuing banks and card brands a lot of money, the Payment Card Industry Security Standards Council (PCI-SSC) and the rest of the ecosystem went full goblin mode. Now, if you lose credit card numbers in bulk, it will cost you big. Massive fines for breaches (typically levied by the card brands via the acquiring bank), increased transaction fees, and even the Credit Card Death Penalty (being banned from charging credit cards), are all very big sticks.

Now Comes the Finding Out In news that should not be surprising, when there are actual consequences for failing to do something, companies take the problem seriously. Which is why no credit card numbers were disclosed is such an interesting statement. Consider why no credit card numbers were disclosed. It s not that credit card numbers aren t valuable to criminals because they are. Instead, it s because the company took steps to properly secure the credit card data. Next, you ll start to consider why, if the credit card numbers were secured, why wasn t the personal information that did get disclosed similarly secured? Information that is far more damaging to the individuals to whom that information relates than credit card numbers. The only logical answer is that it wasn t deemed financially beneficial to the company to secure that data. The consequences of disclosure for that information isn t felt by the company which was breached. Instead, it s felt by the individuals who have to spend weeks of their life cleaning up from identity fraud committed against them. It s felt by the victim of intimate partner violence whose new address is found in a data dump, letting their ex find them again. Until there are real, actual consequences for the companies which hemorrhage our personal data (preferably ones that have percentage of global revenue at the end), data breaches will continue to happen. Not because they re inevitable because as credit card numbers show, data can be secured but because there s no incentive for companies to prevent our personal data from being handed over to whoever comes along.

Support my Salt My salty takes are powered by refreshing beverages. If you d like to see more of the same, buy me one.

#
# Debian / libvirt
#
# By default, qemu guests network interfaces are created with the
# trustGuestRxFilters attribute set to no.  This setting blocks
# neighbor solicitation and other ipv6 multicast traffic.  To enable
# neighbor solicitations, modify your interface definition in your
# guest definition XML.
# For example if you have a guest named bird0 with an interface
# definition that looks like this:
$ sudo virsh dumpxml bird0   grep -A7 "type='direct'"
    <interface type='direct' trustGuestRxFilters='no'>
      <mac address='52:54:00:AA:BB:CC'/>
      <source dev='ens2f0np0' mode='bridge'/>
      <target dev='macvtap3'/>
      <model type='virtio'/>
      <alias name='net4'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </interface>
# Then use 'virsh edit bird0' change the attribute in the <interface>
# element to have the value 'yes' instead:
    <interface type='direct' trustGuestRxFilters='yes'>
      <mac address='52:54:00:AA:BB:CC'/>
      <source dev='ens2f0np0' mode='bridge'/>
      <target dev='macvtap3'/>
      <model type='virtio'/>
      <alias name='net4'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </interface>
# If the trustGuestRxFilters attribute is missing, it defaults to no,
# so be sure that the attribute exists and that the attribute has a
# value of 'yes'

A while back, I wrote about using the SSH agent protocol to satisfy WebAuthn requests. The main problem with this approach is that it required starting the SSH agent with a special argument and also involved being a little too friendly with the implementation - things worked because I could provide an arbitrary public key and the implementation never validated that, but it would be legitimate for it to start doing so and then break everything. And it also only worked for keys stored on tokens that ssh supports - there was no way to extend this to other keystores on the client (such as the Secure Enclave on Macs, or TPM-backed keys on PCs). I wanted a better solution.

It turns out that it was far easier than I expected. The ssh agent protocol is documented here, and the interesting part is the extension support extension mechanism. Basically, you can declare an extension and then just tunnel whatever you want over it. As before, my goto was the go ssh agent package which conveniently implements both the client and server side of this. Implementing the local agent is trivial - look up SSH_AUTH_SOCK, connect to it, create a new agent client that can communicate with that by calling NewClient, and then implement the ExtendedAgent interface, create a new socket, and call ServeAgent against that. Most of the ExtendedAgent functions should simply call through to the original agent, with the exception of Extension(). Just add a case statement against extensionType, define some reasonably namespaced extension, and you're done.

Now you need to use this agent. You probably don't want to use this for arbitrary hosts (agent forwarding should only be enabled for remote systems you trust, not arbitrary machines you connect to - if you enabled agent forwarding for github and github got compromised, github would be able to use any private keys loaded into your agent, and you probably don't want that). So the right approach is to add a Host entry to the ssh config with a ForwardAgent stanza pointing at the socket you created in your new agent. This way the configured subset of remote hosts will automatically talk to this new custom agent, while forwarding for anything else will still be at the user's discretion.

For the remote end things are even easier. Look up SSH_AUTH_SOCK and call NewClient as before, and then simply call client.Extension(). Whatever you stick in the contents argument will simply end up being received at the client end. You now have a communication channel between a the remote system and the local client, and what you do with that is up to you. I'm using it to allow a remote system to obtain auth tokens from Okta and forward WebAuthn challenges that can either be satisfied via a local WebAuthn token or by passing the query off to Mac TouchID, but there's fundamentally no constraints whatsoever on what can be done here.

(If you want to do this on Windows and still have everything work with existing clients you'll need to take this into account - Windows didn't really do Unix sockets until recently so everything there is awful)

comments

No dia 16 agosto comemorado o anivers rio do Projeto Debian, e todos os anos comunidades ao redor do mundo organizam encontros para celebrar esta data. Chamado de Debian Day (Dia do Debian), o evento sempre conta com uma quantidade expressiva de comunidadades brasileiras organizando atividades nas suas cidades no dia 16 (ou no s bado mais pr ximo). Em 2024 o Debian Day celebrar os 31 anos do Projeto Debian e o dia 16 de agosto ser numa sexta-feira, por isso provavelmente a maioria das comunidades organizar o suas atividades no s bado, dia 17. Estamos fazendo uma chamada de organizadores(as) para o Debian Day em 2024. A ideia reunir, em um grupo no telegram, as pessoas interessadas em coordenar as atividades das suas comunidades locais para trocar experi ncias, ajudar os(as) novatos(as), e discutir a possibilidade do Projeto Debian ajudar financeiramente as comunidades. O Debian Day na sua cidade pode ser desde um encontro em uma pizzaria/bar/restaurante para promover a reuni o das pessoas, at um evento mais amplo com palestras/oficinas. Ent o n o existe obrigatoriedade sobre como deve ser o encontro, tudo depende do que voc e a sua comunidade querem e podem fazer. Existe a possibilidade de solicitarmos ao l der do projeto Debian para reembolsar algumas despesas. Por exemplo, para produzir adesivos, pagar as pizzas, encomendar um bolo, etc. Venha fazer parte do grupo Debian Day BR no telegram e discutir as ideias: https://t.me/debian_day_br Se voc topa esse desafio e vai organizar um Debina Day na sua cidade, n o deixe de adicionar a sua cidade com as informa es necess rias aqui.

No dia 16 agosto comemorado o anivers rio do Projeto Debian, e todos os anos comunidades ao redor do mundo organizam encontros para celebrar esta data. Chamado de Debian Day (Dia do Debian), o evento sempre conta com uma quantidade expressiva de comunidadades brasileiras organizando atividades nas suas cidades no dia 16 (ou no s bado mais pr ximo). Em 2024 o Debian Day celebrar os 31 anos do Projeto Debian e o dia 16 de agosto ser numa sexta-feira, por isso provavelmente a maioria das comunidades organizar o suas atividades no s bado, dia 17. Estamos fazendo uma chamada de organizadores(as) para o Debian Day em 2024. A ideia reunir, em um grupo no telegram, as pessoas interessadas em coordenar as atividades das suas comunidades locais para trocar experi ncias, ajudar os(as) novatos(as), e discutir a possibilidade do Projeto Debian ajudar financeiramente as comunidades. O Debian Day na sua cidade pode ser desde um encontro em uma pizzaria/bar/restaurante para promover a reuni o das pessoas, at um evento mais amplo com palestras/oficinas. Ent o n o existe obrigatoriedade sobre como deve ser o encontro, tudo depende do que voc e a sua comunidade querem e podem fazer. Existe a possibilidade de solicitarmos ao l der do projeto Debian para reembolsar algumas despesas. Por exemplo, para produzir adesivos, pagar as pizzas, encomendar um bolo, etc. Venha fazer parte do grupo Debian Day BR no telegram e discutir as ideias: https://t.me/debian_day_br Se voc topa esse desafio e vai organizar um Debina Day na sua cidade, n o deixe de adicionar a sua cidade com as informa es necess rias aqui. Veja na wiki do Debian quais grupos est o ativos e como entrar em contato com eles. Algumas cidades ou estados possuem grupos de telegram pr prios. Se a sua cidade n o estiver l , voc pode liderar a cria o do seu grupo local :-)

FTP master This month I accepted 347 and rejected 49 packages. The overall number of packages that got accepted was 348.

Debian LTS This was my hundred-nineteenth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. During my allocated time I uploaded or worked on:

• [#1070154] bullseye-pu: qtbase-opensource-src/5.15.2+dfsg-9+deb11u1 package upload
• [#1064550] bullseye-pu: libjwt 1.10.2-1+deb11u1 has been marked for accept
• [#1067544] bullseye-pu: libmicrohttpd 0.9.72-2+deb11u1 has been marked for accept

I also continued to work on tiff and last but not least did a week of FD and attended the monthly LTS/ELTS meeting. Unfortunately I used lots of time to debug an issue with nghttp2. Please see my odyssey below. Debian ELTS This month was the seventieth ELTS month. During my allocated time I uploaded:

• [ELA-1104-1-1]nghttp2 security update for one CVEs to fix an DoS resulting from bad handling of CONTINUATION frames in Stretch

For some tests I installed the new nghttp2 package on my Stretch VM and started the daemon. Unfortunately I got an unexpected error from getaddrinfo() about ai_socktype not supported. The daemon was configured to listen on lo, the device was available, but the error remained. I was pretty sure that my patch was not the reason for this and indeed the unpatched version showed this error as well. I didn t want to release an untested package, so nghttp2 had to start at least! Therefore I built a minimal example to reproduce the issue. getaddrinfo() failed for hints.ai_socktype=SOCK_STREAM and a numerical IP address. Having no hints at all or localhost instead of 127.0.0.1 made the error disappear (as a remark: localhost resolves to 127.0.0.1, the ipv6 variant is ip6-localhost ). I could see that in nghttp2 as well. Configuring it with localhost let the error vanish but the daemon still exited due to other reasons. After some time of debugging, I added another network interface to my VM and configured it with a dummy IPv4 address. Voila, everything worked as expected. According to Wikipedia, IPv6 was ratified as standard in 2017 and Stretch was also released in 2017. No wonder that a IPv6-only-VM had problems back then and these problems survived to the present. I also continued to work on an update for tiff in Jessie and Stretch, did a week of FD and attended the LTS/ELTS meeting. Debian Printing This month I uploaded new upstream or bugfix versions of:

• cups-bjnp

This work is generously funded by Freexian! Debian Astro This month I uploaded a new upstream or bugfix version of:

• astap
• astap-cli
• c-munipack
• libahp-xc
• planetary-system-stacker
• fxload
• virtualgps

Debian IoT This month I uploaded new upstream or bugfix versions of:

• pywws
• openzwave
• openoverlayrouter

Debian Mobcom Due to more and more problems with time_t, I removed osmo-iuh and all dependencies from armel, armhf and i386, sorry. If there is really anybody using this software on 32-bit architectures don t hesitate to get in touch. It is official now, the GSoC student working on the Mobcom packages is Nathan Doris. He already finished the hardest part of the job and I could upload the latest version of libosmocore. I really enjoy working with him and look forward to a pleasant SoC :-). misc This month I uploaded new upstream or bugfix versions of:

• dateutils
• chktex
• liburjtag
• pollinate
• usb-modeswitch
• greylistd
• entropybroker

Did I already mention that I love lists with topics I can work on. I print out such lists and enjoy checking off one after the other. End of May Helmut told me that I am a bit lazy and gave me such a list with all my packages that have one or the other issue with /usr-move. Most of the uploads above are packages on that list and I could check off a lot :-).

Welcome to the May 2024 report from the Reproducible Builds project! In these reports, we try to outline what we have been up to over the past month and highlight news items in software supply-chain security more broadly. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website. Table of contents:

1. A peek into build provenance for Homebrew
2. Distribution news
3. Mailing list news
4. Miscellaneous news
5. Two new academic papers
6. diffoscope
7. Website updates
8. Upstream patches
9. Reproducibility testing framework

---

A peek into build provenance for Homebrew Joe Sweeney and William Woodruff on the Trail of Bits blog wrote an extensive post about build provenance for Homebrew, the third-party package manager for MacOS. Their post details how each bottle (i.e. each release):

[ ] built by Homebrew will come with a cryptographically verifiable statement binding the bottle s content to the specific workflow and other build-time metadata that produced it. [ ] In effect, this injects greater transparency into the Homebrew build process, and diminishes the threat posed by a compromised or malicious insider by making it impossible to trick ordinary users into installing non-CI-built bottles.

The post also briefly touches on future work, including work on source provenance:

Homebrew s formulae already hash-pin their source artifacts, but we can go a step further and additionally assert that source artifacts are produced by the repository (or other signing identity) that s latent in their URL or otherwise embedded into the formula specification.

Distribution news In Debian this month, Johannes Schauer Marin Rodrigues (aka josch) noticed that the Debian binary package bash version 5.2.15-2+b3 was uploaded to the archive twice. Once to bookworm and once to sid but with differing content. This is problem for reproducible builds in Debian due its assumption that the package name, version and architecture triplet is unique. However, josch highlighted that

This example with bash is especially problematic since bash is Essential:yes, so there will now be a large portion of .buildinfo files where it is not possible to figure out with which of the two differing bash packages the sources were compiled.

In response to this, Holger Levsen performed an analysis of all .buildinfo files and found that this needs almost 1,500 binNMUs to fix the fallout from this bug. Elsewhere in Debian, Vagrant Cascadian posted about a Non-Maintainer Upload (NMU) sprint to take place during early June, and it was announced that there is now a #debian-snapshot IRC channel on OFTC to discuss the creation of a new source code archiving service to, perhaps, replace snapshot.debian.org. Lastly, 11 reviews of Debian packages were added, 15 were updated and 48 were removed this month adding to our extensive knowledge about identified issues. A number of issue types have been updated by Chris Lamb as well. [ ][ ]
Elsewhere in the world of distributions, deep within a larger announcement from Colin Percival about the release of version 14.1-BETA2, it was mentioned that the FreeBSD kernels are now built reproducibly.
In Fedora, however, the change proposal mentioned in our report for April 2024 was approved, so, per the ReproduciblePackageBuilds wiki page, the add-determinism tool is now running in new builds for Fedora 41 ( rawhide ). The add-determinism tool is a Rust program which, as its name suggests, adds determinism to files that are given as input by attempting to standardize metadata contained in binary or source files to ensure consistency and clamping to $SOURCE_DATE_EPOCH in all instances . This is essentially the Fedora version of Debian s strip-nondeterminism. However, strip-nondeterminism is written in Perl, and Fedora did not want to pull Perl in the buildroot for every package. The add-determinism tool eliminates many causes of non-determinism and work is ongoing to continue the scope of packages it can operate on.

Mailing list news On our mailing list this month, regular contributor kpcyrd wrote to the list with an update on their source code indexing project, whatsrc.org. The whatsrc.org project, which was launched last month in response to the XZ Utils backdoor, now contains and indexes almost 250,000 unique source code archives. In their post, kpcyrd gives an example of its intended purpose, noting that it shown that whilst there seems to be consensus about [the] source code for zsh 5.9 in various Linux distributions, it does not align with the contents of the zsh Git repository . Holger Levsen also posted to the list with a pre-announcement of sorts for the 2024 Reproducible Builds summit. In particular:

[Whilst] the dates and location are not fixed yet, however if you don help us with finding a suitable location soon, it is very likely that we ll meet again in Hamburg in the 2nd half of September 2024 [ ].

Lastly, Frederic-Emmanuel Picca wrote to the list asking for help understanding the non-reproducible status of the Debian silx package and received replies from both Vagrant Cascadian and Chris Lamb.

Miscellaneous news strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. This month strip-nondeterminism version 1.14.0-1 was uploaded to Debian unstable by Chris Lamb chiefly to incorporate a change from Alex Muntada to avoid a dependency on Sub::Override to perform monkey-patching and break circular dependencies related to debhelper [ ]. Elsewhere in our tooling, Jelle van der Waa modified reprotest because the pipes module will be removed in Python version 3.13 [ ].
It was also noticed that a new blog post by Daniel Stenberg detailing How to verify a Curl release mentions the SOURCE_DATE_EPOCH environment variable. This is because:

The [curl] release tools document also contains another key component: the exact time stamp at which the release was done using integer second resolution. In order to generate a correct tarball clone, you need to also generate the new version using the old version s timestamp. Because the modification date of all files in the produced tarball will be set to this timestamp.

Furthermore, Fay Stegerman filed a bug against the Signal messenger app for Android to report that their reproducible builds cannot, in fact, be reproduced. However, Fay is quick to note that she has:

found zero evidence of any kind of compromise. Some differences are yet unexplained but everything I found seems to be benign. I am disappointed that Reproducible Builds have been broken for months but I have zero reason to doubt Signal s security in any way.

Lastly, it was observed that there was a concise and diagrammatic overview of supply chain threats on the SLSA website.

Two new academic papers Two new scholarly papers were published this month. Firstly, Mathieu Acher, Beno t Combemale, Georges Aaron Randrianaina and Jean-Marc J z quel of University of Rennes on Embracing Deep Variability For Reproducibility & Replicability. The authors describe their approach as follows:

In this short [vision] paper we delve into the application of software engineering techniques, specifically variability management, to systematically identify and explicit points of variability that may give rise to reproducibility issues (e.g., language, libraries, compiler, virtual machine, OS, environment variables, etc.). The primary objectives are: i) gaining insights into the variability layers and their possible interactions, ii) capturing and documenting configurations for the sake of reproducibility, and iii) exploring diverse configurations to replicate, and hence validate and ensure the robustness of results. By adopting these methodologies, we aim to address the complexities associated with reproducibility and replicability in modern software systems and environments, facilitating a more comprehensive and nuanced perspective on these critical aspects.

(A PDF of this article is available.)
Secondly, Ludovic Court s, Timothy Sample, Simon Tournier and Stefano Zacchiroli have collaborated to publish a paper on Source Code Archiving to the Rescue of Reproducible Deployment. Their paper was motivated because:

The ability to verify research results and to experiment with methodologies are core tenets of science. As research results are increasingly the outcome of computational processes, software plays a central role. GNU Guix is a software deployment tool that supports reproducible software deployment, making it a foundation for computational research workflows. To achieve reproducibility, we must first ensure the source code of software packages Guix deploys remains available.

(A PDF of this article is also available.)

diffoscope diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes such as uploading versions 266, 267, 268 and 269 to Debian, making the following changes:

• New features:
  • Use xz --list to supplement output when comparing .xz archives; essential when metadata differs. (#1069329)
  • Include xz --verbose --verbose (ie. double) output. (#1069329)
  • Strip the first line from the xz --list output. [ ]
  • Only include xz --list --verbose output if the xz has no other differences. [ ]
  • Actually append the xz --list after the container differences, as it simplifies a lot. [ ]
• Testing improvements:
  • Allow Debian testing to fail right now. [ ]
  • Drop apktool from Build-Depends; we can still test APK functionality via autopkgtests. (#1071410)
  • Add a versioned dependency for at least version 5.4.5 for the xz tests as they fail under (at least) version 5.2.8. (#374)
  • Fix tests for 7zip 24.05. [ ][ ]
  • Fix all tests after additon of xz --list. [ ][ ]
• Misc:
  • Update copyright years. [ ]

In addition, James Addison fixed an issue where the HTML output showed only the first difference in a file, while the text output shows all differences [ ][ ][ ], Sergei Trofimovich amended the 7zip version test for older 7z versions that include the string [64] [ ][ ] and Vagrant Cascadian relaxed the versioned dependency to allow version 5.4.1 for the xz tests [ ] and proposed updates to guix for versions 267, 268 and pushed version 269 to Guix. Furthermore, Eli Schwartz updated the diffoscope.org website in order to explain how to install diffoscope on Gentoo [ ].

Website updates There were a number of improvements made to our website this month, including Chris Lamb making the print CSS stylesheet nicer [ ]. Fay Stegerman made a number of updates to the page about the SOURCE_DATE_EPOCH environment variable [ ][ ][ ] and Holger Levsen added some of their presentations to the Resources page. Furthermore, IOhannes zm lnig stipulated support for SOURCE_DATE_EPOCH in clang version 16.0.0+ [ ], Jan Zerebecki expanded the Formal definition page and fixed a number of typos on the Buy-in page [ ] and Simon Josefsson fixed the link to Trisquel GNU/Linux on the Projects page [ ].

Upstream patches This month, we wrote a number of patches to fix specific reproducibility issues, including:

• Bernhard M. Wiedemann:
  • nauty (CPU-detection issue)
  • emacs (ASLR)
• Chris Lamb:
  • #1070754 filed against gensio.
  • #1071064 filed against tkgate.
  • #1072094 filed against ruby-pgplot.

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In May, a number of changes were made by Holger Levsen:

• Debian-related changes:
  • Enable the rebuilder-snapshot API on osuosl4. [ ]
  • Schedule the i386 architecture a bit more often. [ ]
  • Adapt cleanup_nodes.sh to the new way of running our build services. [ ]
  • Add 8 more workers for the i386 architecture. [ ]
  • Update configuration now that the infom07 and infom08 nodes have been reinstalled as real i386 systems. [ ]
  • Make diffoscope timeouts more visible on the #debian-reproducible-changes IRC channel. [ ]
  • Mark the cbxi4a-armhf node as down. [ ][ ]
  • Only install the hdmi2usb-mode-switch package only on Debian bookworm and earlier [ ] and only install the haskell-platform package on Debian bullseye [ ].
• Misc:
  • Install the ntpdate utility as we need it later. [ ]
  • Document the progress on the i386 architecture nodes at Infomaniak. [ ]
  • Drop an outdated and unnoticed notice. [ ]
  • Add live_setup_schroot to the list of so-called zombie jobs. [ ]

In addition, Mattia Rizzolo reinstalled the infom07 and infom08 nodes [ ] and Vagrant Cascadian marked the cbxi4a node as online [ ].

---

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Twitter: @ReproBuilds
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org

Contributing to Debian is part of Freexian s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

DebConf Bursary updates, by Utkarsh Gupta Utkarsh is the bursaries team lead for DebConf 24. Bursary requests are dispatched to a team of volunteers to review. The results are collated, adjusted and merged to produce priority lists of requests to fund. Utkarsh raised the team, coordinated the review, and issued bursaries to attendees.

/usr-move, by Helmut Grohne More and more, the /usr-move transition is being carried out by multiple contributors and many performed around a hundred of the requested uploads. Of these, Helmut contributed five patches and two uploads. As a result, there are less than 350 packages left to be converted, and all of the non-trivial cases have patches. We started with three times that number. Thanks to everyone involved for supporting this effort. For people interested in background information of this transition, Helmut gave a presentation at MiniDebConf Berlin 2024 (slides).

sbuild, by Helmut Grohne While unshare mode of sbuild has existed for quite a while, it is now getting significant use in Debian, and new problems are popping up. Helmut looked into an apparmor-related failure and provided a diagnosis. While relevant code would detect the chroot nature of a schroot backend and skip apparmor tests, the unshare environment would be just good enough to run and fail the test. As sbuild exposes fewer special kernel filesystems, the tests will be skipped again. Another problem popped up when gobject-introspection added a dependency on the host architecture Python interpreter in a cross build environment. sbuild would prefer installing (and failing) a host architecture Python to installing the qemu alternative. Attempts to fix this would result in systemd killing sbuild. ischroot as used by libc6.postinst would not classify the unshare environment as a chroot. Therefore libc6.postinst would run telinit which would kill the build process. This is a complex interaction problem that shall eventually be solved by providing triggers from libc6 to be implemented by affected init systems.

Salsa CI updates, by Santiago Ruano Rinc n Several issues arose about Salsa CI last month, and it is probably worth mentioning part of the challenges of defining its framework in YAML. With the upcoming end-of-support of Debian 10 buster as LTS, armel was removed from deb.debian.org, making the jobs that build images for buster/armel to fail. While the removal of buster/armel from the repositories is a natural change, it put some light on the flaws in the Salsa CI design regarding the support of the different Debian releases. Currently, the images are defined like these (from .images-debian.yml):

.all-supported-releases: &all-supported-releases
  - stretch
  - stretch-backports
  - buster
  - bullseye
  - bullseye-backports
  - bookworm
  - bookworm-backports
  - trixie
  - sid
  - experimental

And from them, different images are built according to the different jobs and how they are supported, for example:

images-prod-arm:
  stage: build
  extends: .build_template
  tags:
    - $SALSA_CI_ARM_RUNNER_TAG
  parallel:
    matrix:
      # Base image, all releases, all arches
      - IMAGE_NAME: base
        ARCH:
          - arm32v5
          - arm32v7
          - arm64v8
        RELEASE: *all-supported-releases

The removal of buster/armel could be easily reflected as:

images-prod-arm:
  stage: build
  extends: .build_template
  tags:
    - $SALSA_CI_ARM_RUNNER_TAG
  parallel:
    matrix:
      # Base image, fully supported releases, all arches
      - IMAGE_NAME: base
        ARCH:
          - arm32v5
          - arm32v7
          - arm64v8
        RELEASE:
          - stretch
          - buster
          - bullseye
          - bullseye-backports
          - bookworm
          - bookworm-backports
          - trixie
          - sid
          - experimental
      # buster only supports armhf and arm64
      - IMAGE_NAME: base
        ARCH:
          - arm32v7
          - arm64v8
        RELEASE: buster

Evidently, this increases duplication of the release support data, which is of course not optimal and it is error prone when changing the data about supported releases. A better approach would be to have two different YAML lists, such as:

# releases that have partial support. E.g.: buster is transitioning to
# Debian LTS, and buster armel is no longer found in deb.debian.org
.old-releases: &old-releases
  - stretch
  - buster

.currently-supported-releases: &currently-supported-releases
  - bullseye
  - bullseye-backports
  - bookworm
  - bookworm-backports
  - trixie
  - sid
  - experimental

and then a unified list:

.all-supported-releases: &all-supported-releases
  - *old-releases
  - *currently-supported-releases

that could be used in the matrix of the jobs that build all the images available in the pipeline container registry. However, due to limitations in GitLab, it is not possible to expand the variables or mapping values in a parallel:matrix context. At least not in an elegant fashion. This is the kind of issue that recently arose and that Santiago is currently working to solve, in the simplest possible way. Astute readers would notice that stretch is listed in the fully supported releases. And there is no problem with stretch, because it is built from archive.debian.org. Otto actually has tried to fix the broken image build job doing the same, but it is still incorrect, because the security repository is not (yet) available in archive.debian.org. Additionally, Santiago has also worked on other merge requests, such as:

1. support branch/tags as target head in the test projects,
2. build autopkgtest image on top of stable
3. Add .yamllint and make it happy in the autopkgtest-lxc project
4. enable FF_SCRIPT_SECTIONS to log multiline commands, among others.

Archiving DebConf Websites, by Stefano Rivera DebConf, the annual Debian conference, has its own new website every year. These are typically complex dynamic web applications (featuring registration, call for papers, scheduling, etc.) Once the conference is over, there is no need to keep maintaining these applications, so we archive the sites off as static HTML, and serve them from Debian s static CDN. Stefano archived the websites for the last two DebConfs. The schedule system behind DebConf 14 and 15 s websites was a derivative of Canonical s summit system. This was only used for a couple of years before migrating to wafer, the current system. Archiving summit content has been on the nice to have list for years, but nobody has ever tackled it. The machine that served the sites went away a couple of years ago. After much digging, a backup of the database was found, and Stefano got this code running on an ancient Python 2.7. Recently Stefano put this all together and hooked in an archive export to finally get this content preserved.

Python 3.x and pypy3 security bug triage, by Stefano Rivera Stefano Rivera triaged all the open security bugs against the Python 3.x and PyPy3 packages for Debian s stable and LTS releases. Several had been fixed but this wasn t recorded in the security tracker.

Linux livepatching support for Debian, by Santiago Ruano Rinc n In collaboration with Emmanuel Arias, Santiago filed ITP bug #1070494. As stated in the bug, more than an Intent to Package, it is an Intent to Design and Implement live patching support for the Linux kernel in Debian. For now, Emmanuel and Santiago have done exploratory work and they are working to understand the different possibilities to implement livepatching. One possible direction is to rely on kpatch, and the other is to package the modules using regular packaging tools. Also, it is needed to evaluate if it is possible to rely on distributing the modules via packages, or instead as a service, as it is done by some commercial distributions.

Miscellaneous contributions

• Thorsten Alteholz uploaded cups-bjnp to improve packaging.
• Colin Watson tracked down a baffling CI issue in openssh to unblock several merge requests, removed the user_readenv=1 option from its PAM configuration, and started on the first stage of his plan to split out GSS-API key exchange support to separate packages.
• Colin did his usual routine work on the Python team, upgrading 26 packages to new upstream versions, and cherry-picking an upstream PR to fix a pytest 8 incompatibility in ipywidgets.
• Colin NMUed a couple of packages to reduce the need for explicit overrides in Packages-arch-specific, and removed some other obsolete entries from there.
• Emilio managed various library transitions, and helped finish a few of the remaining t64 transitions.
• Helmut sent a patch for enabling piuparts to work as a regular user building on earlier work.
• Helmut sent patches for 7 cross build failures, 6 other debian bugs and fixed an infrastructure problem in crossqa.debian.net.
• Nicholas worked on a sponsored package upload, and discovered the blhc tool for diagnosing build hardening.
• Stefano Rivera started and completed the re2 transition. The release team suggested moving to a virtual package scheme that includes the absl ABI (as re2 now depends on it). Adopted this.
• Stefano continued to work on DebConf 24 planning.
• Santiago continued to work on DebConf24 Content tasks as well as Debconf25 organisation.