For over ten years, the only way I have had of knowing what our solar panels are doing has been an impulse monitor on the generation meter, counting the flashes of its LED. It has always been a bit flaky due to battery changes or loose connections, and there has never been a way to see solar generation alongside the energy used by the rest of the house. This week I have fixed that, by building a small ESPHome sensor which talks directly to the inverter. I am still reeling slightly: with AI assistance, it took under a week from idea to working implementation, and it is a project I would certainly never have attempted otherwise. The result is zeversolar-esphome. The inverter is a Zeversolar Zeverlution 3680, which has an RS485 port but no Wi-Fi or Ethernet module. It doesn t speak Modbus; it uses Eversolar s own master/slave protocol, which nobody has officially documented but which several people have reverse-engineered over the years (credit to eversolar-monitor, Heliograph and others). Manufacturer support for Eversolar/Zeversolar inverters apparently ended in December 2019 when the company went out of business.

The rating plate on my inverter.

A while ago I had looked into what ports were available on this inverter with a view to connecting some sort of monitoring, but I had abandoned it as too hard . While working on some Home Assistant configuration, Claude suggested the idea again, but similarly wrote it off initially. I encouraged it to explore a bit further, and supplied the model number of the inverter, at which point the picture got a lot more promising. After a bit of back-and-forth about hardware, we settled on an M5Stack Atom Lite on an Atomic RS485 Base. The main attraction was that it already comes in a case, so I didn t need to design (or, more likely, bodge) an enclosure. While I waited for the parts to arrive, Claude wrote the firmware: an ESPHome external component built around a small, dependency-free C++ driver, with host-side unit tests against byte captures that other people had published from other models. I found it slightly surreal to have a tested protocol driver before owning any hardware to run it on.

Cables The parts came before the tools I needed to get into the inverter safely (a DC clamp meter and a T25 screwdriver bit), so in the meantime I made up an RS485 cable - this uses an RJ45 connection at one end, and some custom wiring at the other. My first attempt was with some CAT 6, but (as Claude had pointed out and I had ignored) its solid-core conductors are much harder to twist together. So I gave up and sacrificed an old CAT 5e patch lead instead, which went much better.

The custom end of the cable, before being taped up.

The risky bit On a Zeverlution, the RJ45 sockets are inside the wiring compartment, which means taking the cover off to install the monitor. The manual addresses this part to qualified electricians. So the step I was most nervous about was not the software at all, but safely isolating the inverter.

Inside the inverter, with the cover off.

I did this with appropriate caution, following the instructions from the manual carefully, and also waiting until after dark just in case. Now that I understand how the DC clips from the panels work, I can see how to do this safely during daytime too but hopefully this won t be necessary.

Plugged in, and threaded through the cable gland.

Warning: if you re not confident, I would always advise to get an electrician to do this bit. Do what I say, not what I do.

Debugging over Wi-Fi Once the device was installed, the nice thing about ESPHome is that I never need to go back in the loft again to update it it reflashes over Wi-Fi. I ve taken full advantage of this for debugging, and for reading the wire protocol directly from the logs.

The finished device. Green means generating.

Separately, I ve used Claude to integrate the new sensors into Home Assistant. Solar generation finally sits in the Energy dashboard next to the rest of the house s usage, which was the whole point.

Reflections This was a fun, quick project, and it solved a problem that has eluded me for a decade. I want to be honest about how much of that is down to the AI: I would not have attempted any of this on my own. I could not have told you the protocol existed, never mind implemented it. My contribution was everything involving a screwdriver, and ordering some hardware.

My hobby: following GOV.UK to look for interesting announcements. Today was an update on the MOD s Rapid AI Delivery Taskforce which was previously announced in June during London Tech Week. I like this line: Success is measured in operational advantage delivered, not technology demonstrated. To me it recalls Working software is the primary measure of progress from Principles behind the Agile Manifesto if you understand working to mean working in production . Which I do. For anyone interested in suggesting ideas to the taskforce, the four operational challenge areas include:

• Understanding and decision advantage
• Electromagnetic and information advantage
• Planning and automation
• Autonomous systems

Yesterday s announcement of UK access to Ukraine s Avengers AI Labs database seems incredibly relevant to that last point. Machine assistance for handling and interpreting huge volumes of data would probably benefit decision advantage and interpretation of a crowded EM spectrum, but this is hopefully(?) more than just LLMs. Of course, there s more to AI than large language models right? I worry that planning and automation might amount to generating large amounts of text faster . Nothing could possibly go wrong with this.

Bruce Schneier and Nathan E. Sanders wrote a disturbing and informative article about the use of AI by the US government [1]. Bruce Schneier wrote an interesting blog post about corporate liability for AI decisions and the German court ruling about Google s AI summaries [2]. Cybersecurity News has an interesting article about how Pliny the Liberator succeeded in jailbreaking Anthropic s latest LLM to give instructions on writing exploits, writing exploitable code (backdoors?), and making meth [3]. Andrew Pam wrote about the number of cars with internal combustion engines in NSW decreasing for the first time since 1910, EVs are taking over [4]. Cory Doctorow wrote an informative blog post about Facebook s attempts to silence whistleblowers and what a pathetic little loser Zuckerberg is [5]. Scott Santens wrote an insightful article about how to effectively levy taxes in the future when AI significantly reduces the number of workers [6]. Ron Garrett wrote an insightful post about birthright citizenship in the US and his status as a US citizen who was not born there [7]. The BBC has an interesting article about the Scottish Violence Reduction Unit and how treating violence as a disease can significantly address the problem [8]. The LA Times has an interesting article about Covid19 causing cancer that had been in remission to return, sparking some new research into the effects of viruses on mammals [9]. CMU has an interesting video about ways to physically modify QR codes and how they could be used in real life [10]. Tim Retout wrote an informative post about the pervasive forms of advertising on the Internet, even on the BBC s site and how some of it can be blocked [11]. The Intercept Fund is a project to address respiratory illnesses and the long term mostly unnoticed costs they cause to society, we need governments and corporations to get on board with this [12]. The German news site DW has an interesting article about activists registering neo-nazi slang as trademarks to prevent the sale of nazi merchanise which funds racism [13]. The Conversation has an insightful article about how in South Sudan and other war ravaged countries the peace process usually just allocates the spoils of war and therefore encourages more war [14].

• [1] https://tinyurl.com/26cnn3og
• [2] https://tinyurl.com/22fewyml
• [3] https://tinyurl.com/2a5t693d
• [4] https://glasswings.com.au/blog/2026-07/05-185304.html
• [5] https://tinyurl.com/26qzcadm
• [6] https://tinyurl.com/2csg4huw
• [7] https://tinyurl.com/2blnzzxg
• [8] https://tinyurl.com/25v5j7v9
• [9] https://archive.md/Gq2jB
• [10] https://www.youtube.com/watch?v=3ddTIa-AhXE
• [11] https://tinyurl.com/2xt9dgfp
• [12] https://tinyurl.com/2y5f26rj
• [13] https://tinyurl.com/29ok2qsu
• [14] https://tinyurl.com/2yxzsp7s

When you pass an AWS certification exam, sometimes it can extend the life of related lesser AWS certifications. But I could not find an illustration of exactly which ones, so here s an up-to-date diagram: Figure: Renewal relationships between AWS certifications. Each arrow is a renews relation there s no obligation to pass lesser exams before the harder ones, but you could also follow the arrows backwards if you want to learn easier material before sitting the more difficult exams. I have made two important simplifications to the graph:

1. The Advanced Networking Specialty certification is being retired soon, so I ve omitted it entirely. The last date to take that exam is 25th August 2026. But Specialty certs don t renew anything else anyway.
2. I have left out transitive relationships in order to simplify the graph so e.g. if you pass a Solutions Architect Professional exam, it also renews any Cloud Practitioner certificate you might hold, even if you do not currently hold the relevant Associate certificate.

You also have the option of sitting each exam again to renew of course, and there s a new scheme to maintain various certs via AWS Skill Builder which can extend them by one year rather than three.

You are what you eat but perhaps this is even more true of our information diet. It is hard to strike a balance between remaining a well-informed citizen versus spending hours ingesting unnecessary news about issues and events we can t affect. But I m increasingly convinced that my hours lost to doomscrolling are down to design choices by web publishers rather than a failure of individual willpower.

We have created an obesogenic information environment I don t think it is just me I think our information environment has been progressively altered over time as news sites look to maximize engagement. Even outside of social media, the invisible hand of the market for eyeballs forces sites to optimize for browse time or risk irrelevance. Even as newspapers find it increasingly difficult to fund good journalism through advertising in an online world, especially local journalism, they need to keep readers on their sites, clicking through as many articles as possible. Clickbait headlines, urgent flashing live icons to draw the attention, and many opportunities to leap from one article to another, and another. But this design approach even extends to news organisations with a different funding model, like BBC News, which is a public service (state-owned but arms-length) organisation funded through a mandatory television licence a matter of controversy in some quarters. And it extends even to sites where I pay a subscription fee; I might get adverts removed, but I am still bombarded with the same design philosophy; too many opportunities to be pulled away from what I m reading towards some other unrelated article. Even if I try and limit my exposure to algorithmic discovery of new news, via RSS feeds or similar, if I m reading the full article in a browser then I am prompted to read more stuff that I didn t intend. This defeats the benefit of curating a set of feeds, because you still get dragged away to random articles.

Only 44% of BBC News is news To show you what I mean, I m going to pick on the BBC, although I love them dearly and the same issue very much applies elsewhere. I ve taken a screenshot of a random BBC News article in mobile view (my preferred doomscrolling user access device), and measured approximately what proportion of the full length of the page is taken up by each section. This is a fairly in-depth news article, so I reckon if anything the figures would be worse than this on shorter articles. (These numbers will not sum to 100% for reasons which are obvious if you look at the crossbars. Also they re approximations.) Less than half of the page (44% if you exclude the inline related links) is actual news text/images; the rest are links trying to help you find the next thing to read/watch. I do not want this. I m sure this A/B tests well in terms of reader figures, but it sometimes leaves me exhausted it must take subconscious mental energy to ignore, or I spend too much time trying to keep on top of things. And remember, this is a publicly-funded site that does not rely on advertising!

Blocking out the noise If you are technically-minded, you can use an ad-blocker such as uBlock Origin to take back some control. Applying the following lines as a custom filter (Settings > My filters) brutally cuts out almost all of these links:

bbc.co.uk##aside
bbc.co.uk##footer>div:has(h2)
bbc.co.uk##[data-block="uploaderEmbed"]
bbc.co.uk##[data-block="links"]

Caveat emptor: I have not road-tested this for more than half an hour, so who knows what consequences this could have on your web browsing. In particular, international readers outside the UK will likely be redirected to bbc.com, the commercial arm of the BBC, where these rules will need adapting. Is it unethical to use an ad-blocker to remove these links? I would argue not. I am not depriving the BBC of any revenue, because I pay my licence fee. I might reduce the amount of time I spend on their website, but if anything the subjectively better experience might encourage me to consume more news from them, not less. In other circumstances (outside the UK for instance, where the BBC relies on advertising), the balance might be different.

Product managers, please find better metrics I lament the state of the internet in 2026. I now can t unsee these innocuous related stories links as a mechanism to grab my attention, and it s gone too far. If you are a normal person just browsing the news and looking to discover the latest important stories relatively quickly, I can see that these types of links might actually be useful for discovery; but I m actually reasonably sure that I m not going to miss out on anything major. You still have the option of the news home page if you want to be presented with more news for example, and it feels natural to go back to there when you ve run out of stories to consume. But it shouldn t be down to individual responsibility to ignore or geekily block these types of link; news sites with alternative funding models should find better metrics for engagement than hours spent on site how about optimizing for customer mental wellbeing, or minimizing time required to catch up with the news? There s no need to maximize clicks and eyeballs. This is a societal level issue, because we are all going mad with news over-engagement. Product managers, over to you.

I am somewhat jet-lagged, having returned from Washington DC just before the 250th anniversary celebrations which will be happening today. I was part of a delegation sent by my employer to the AWS Summit there this week, partly to kindle interactions between PA Consulting and Jacobs who have recently taken a 100% share in PA. Much of our conference time was spent in meetings with AWS executives impressing the facts of the Jacobs/PA partnership upon them, and discussing plans to broaden our collaboration in different sectors. So I spent even less time than usual at conference keynotes, talks etc. This was my first time to DC, and I did find some time to see some sights unfortunately the White House is rather fenced off at the moment following the UFC match, but I did make it to the Capitol and the Washington Monument in the heat. Last Sunday a select few of us attended the baseball in Baltimore rather than the game, the thing that stood out for me was the military jets flying in formation over the stadium every few minutes, and the block-booked seats for the Navy in uniform, who were having a great time! This is obviously a hearts-and-minds thing, but it provides a stark contrast with the UK I can t think of a time I ve seen uniformed military at the football (soccer) or cricket for example. Or Union Jacks flying at shopping centres. Speaking of soccer, England just about beat DR Congo while I was out there, but it was a close-run thing as we were 1-0 down at half time. I can t claim to be following the World Cup too closely, but I overheard comments (from US passers-by) that made clear it would have had a significant reputational impact on our standing in the world had we lost. Another highlight for me was the Church of the Ascension and St. Agnes, where I was able to get my fix of Anglican plainchant and four-part harmony for the week. At morning prayer, I noted they use God save this land rather than God save the King during the responses I ve since found other sources online that choose God save the State . It s strange to think that the words of the BCP dating back to 1549/1662 are a point of continuity since well before the 1776 declaration of independence, and yet are still adapted and used in worship today.

The seL4 organisation on GitHub uses git-repo to manage multiple source repositories, and so there are a large number of projects to get your head around when figuring out the ecosystem. As an experiment, I have taken the various manifest files across the org, and constructed a graph based on how frequently each pair of repositories is mentioned in a manifest together. See below: [This may render badly when syndicated outside of my blog; and also on small screens. And probably large screens. I ve attempted to make sure there s a non-JS fallback on my site with JS enabled, if you hover over a node, it should highlight connected nodes.] The colouring of the nodes is mostly manual; I experimented with graph clustering algorithms but have not found a satisfactory result so far. Still, some clusters are obvious:

• Kernel the seL4 microkernel proper. This often but not always co-exists with the main cluster of core libraries, but it is pulled away slightly by the verification and microkit manifests.
• Verification the verification repositories (l4v, HOL, graph-refine, polyml, isabelle) form a very distinct group. These are connected only to the seL4 microkernel itself, which is the only component formally verified.
• Microkit microkit is a newer operating system framework that does not use CAmkES, so stands apart from the rest of the pack. I chose to scope this work to the seL4 org, so the LionsOS ecosystem and sDDF which are maintained by Trustworthy Systems are not shown. Also not linked is rust-sel4, because this modern world isn t using git-repo in the main to manage its repositories.
• RefOS I d not come across refos before, but it appears to be an example OS from 2021 built on the seL4 kernel.

It s quite hard to pull apart the CAmkES framework and the core libraries; there are definitely some which are more associated with VM management, but the overall shape of this co-occurence data is a messy ball in the middle with some outliers in orbit. One observation is that camkes is correctly identified as more peripheral than camkes-tool, which contains the actual core CAmkES code. Reflecting on this approach, in hindsight I m surprised that using co-occurences worked as well as it did there was no attempt to actually inspect the code and find direct mentions of other code e.g. library header dependencies. As the newer microkit effort largely eschews git-repo, better results might be found by actually taking that more detailed approach, so that graph edges could represent real dependencies between two packages. Additionally, this could allow diving into the various libraries held in the different libs repos, to get a more granular graph of relationships between them. However, I think I spent more time on making it possible to render graphviz graphs easily on my blog than actually gaining any insight into the codebase!

I have proposed the deletion of an obsolete script, but it makes me feel complicated feelings so I m going to try and express those. This particular script was written in 2014, but the concept goes back much further before git was invented. When I started university in 2003, I seem to remember the computing society used to run tutorials for first-year students on how to use Apache Subversion for your group project a vast upgrade on CVS (or worse, no version control at all). Back then, the idea of viewing your changesets in a web browser was relatively new while it was possible to look at an SVN repository through a web UI, features were limited unless you installed something compicated like Trac. Figure 1: Data flow when distributing commits via a mailing list Perhaps because reading email on your desktop computer (I don t think I could afford an IBM ThinkPad?) was the only vaguely real-time notification system available at the time (except I guess SMS, which cost 10p per text), a common pattern seemed to be to use a post-commit hook to send every single commit to a mailing list, named something like foo-commits . Indeed, for a long time Fedora had an scm-commits list which appears to be a topic of recent discussion. I can t really explain why people wanted to have every commit sent to a mailing list except as a way of getting notified of activity I can t believe people would import raw patches from those lists, ala LKML, rather than run actual version control commands to fetch the new source directly. Maybe you d have to go back to NNTP for this. I do like the vendor-neutrality of the everything-as-text approach, building on the open ecosystem of SMTP. But I doubt we d see a widespread resurgence of commit lists now most code hosting must allow anyone to subscribe to email notifications, I assume, and I don t see a huge benefit in a mailing list archive of commit messages. In the case of seL4, I m even more confused about why this script was committed in 2014, shortly after the kernel was put on GitHub. I can only assume it was imported from previous infrastructure. I do know that the implementation is quite Python 2 heavy, with the conversion between unicode and bytes featuring heavily. So rather than risk breaking its logic with patching, I think it s time to thank it for its service and let go.

I have been looking at seL4 some more recently, and had a small patch merged today to remove a legacy Python module from a helper script. (I was trying to run the script on a system without that module installed, and it was almost easier to patch it out.) However, the more I think about this code and how it s used, the more it seems wrong on at least five other levels. The patch itself is quite uninteresting; this script was importing the past module (part of future?) to use the xrange function. Python 2 used to have separate xrange and range functions, where range returned a list in memory while xrange generated an iterator. Because this seL4 script is iterating over a large range of values, it s important the list is not generated in-memory. But Python 3 removed the xrange function and just has range return an object, so it s trivial to avoid the module import. Having thought carefully some more about the specific line, there s surely an off-by-one error in it - range iterates over 0 to n-1, so this line shouldn t be subtracting one if it s looking to test all 32-bit values: But then again, this is being used for a sanity check of a magic bit shift algorithm that speeds up division operations to convert CPU ticks to microseconds on 32-bit arm platforms. Surely if the algorithm s good, it shouldn t be necessary to validate it exhaustively against every possible 32-bit value? Also, 32 bits isn t enough, because this is 64-bit division. include/api/types.h shows that ticks_t is always a uint64_t, so if this were a proof by exhaustion it should run to 2**64 (though that would take infeasibly long). As discussed in issue #1352, lots of people have been running this code with the wrong divisor anyway. But because the bit shift path is only used on 32-bit platforms, it s not clear to me that there s even any point in specifying CLK_SHIFT/MAGIC on platforms which are 64-bit only (e.g. the tx2 port). And to follow this rabbit hole to the very end, in comments on PR #1435 and issue #1509 it s clear that the future of this code is to remove it, as it s 1. unnecessarily clever (on 64-bit platforms the equivalent code just uses a division, so performance can t be that important), and 2. the entire concept of converting to microseconds breaks the seL4 principle of not abstracting away details of the hardware. So this has left me unclear on whether my small patch was a good thing or not, but I certainly learnt something about this corner of seL4 timer handling. And I ve ordered a copy of Hacker s Delight on the recommendation of a code comment.

While watching the Vienna New Year s Concert today, reading about its perhaps somewhat problematic origins, I was struck by the observation that the Strauss family s polkas were seen as pop music during their lifetime, not as serious as proper classical composers, and so it took some time before the Vienna Philharmonic would actually play their work. (Perhaps the space-themed interval today and the ballet dancers pretending to be a steam train were a continuation of the true spirit of this? It felt very Eurovision.) I can t decide if it s remarkable that this year was the first time a female composer (Constanze Geiger) was represented at this concert, or if that is what you get when you set up a tradition of playing mainly Strauss?

Recently I revisited my previous interest in seL4 - a fast, highly assured operating system microkernel for building secure systems. The seL4 Microkit tutorial uses a simple Wordle game example to teach the basics of seL4 Microkit (formerly known as the seL4 Core Platform), which is a framework for creating static embedded systems on top of the seL4 microkernel. Microkit is also at the core of LionsOS, a new project to make seL4 accessible to a wider audience. The tutorial is easy to follow, needing few prerequisites beyond a QEMU emulator and an AArch64 cross-compiler toolchain (Microkit being limited to 64-bit ARM systems currently). Use of an emulator makes for a quick test-debug cycle with a couple of Makefile targets, so time is spent focusing on walking through the Microkit concepts rather than on tooling issues. This is an unusually good learning experience, probably because of the academic origins of the project itself. The Di taxis documentation framework would class this as truly a tutorial rather than a how-to guide - you do learn a lot by implementing the exercises.

It can be incredibly easy for a frontend developer to accidentally write a client-side cross-site-scripting (DOM-XSS) security issue, and yet these are hard for security teams to detect. Vulnerability scanners are slow, and suffer from false positives. Can smarter collaboration between development, operations and security teams provide a way to eliminate these problems altogether? Google claims that Trusted Types has all but eliminated DOM-XSS exploits on those of their sites which have implemented it. Let s find out how this can work!

DOM-XSS vulnerabilities are easy to write, but hard for security teams to catch It is very easy to accidentally introduce a client-side XSS problem. As an example of what not to do, suppose you are setting an element s text to the current URL, on the client side:

// Don't do this
para.innerHTML = location.href;

Unfortunately, an attacker can now manipulate the URL (and e.g. send this link in a phishing email), and any HTML tags they add will be interpreted by the user s browser. This could potentially be used by the attacker to send private data to a different server. Detecting DOM-XSS using vulnerability scanning tools is challenging - typically this requires crawling each page of the website and attempting to detect problems such as the one above, but there is a significant risk of false positives, especially as the complexity of the logic increases. There are already ways to avoid these exploits developers should validate untrusted input before making use of it. There are libraries such as DOMPurify which can help with sanitization.¹ However, if you are part of a security team with responsibility for preventing these issues, it can be complex to understand whether you are at risk. Different developer teams may be using different techniques and tools. It may be impossible for you to work closely with every developer so how can you know that the frontend team have used these libraries correctly?

Trusted Types closes the DevSecOps feedback loop for DOM-XSS, by allowing Ops and Security to verify good Developer practices Trusted Types enforces sanitization in the browser², by requiring the web developer to assign a particular kind of JavaScript object rather than a native string to .innerHTML and other dangerous properties. Provided these special types are created in an appropriate way, then they can be trusted not to expose XSS problems. This approach will work with whichever tools the frontend developers have chosen to use, and detection of issues can be rolled out by infrastructure engineers without requiring frontend code changes.

Content Security Policy allows enforcement of security policies in the browser itself Because enforcing this safer approach in the browser for all websites would break backwards-compatibility, each website must opt-in through Content Security Policy headers. Content Security Policy (CSP) is a mechanism that allows web pages to restrict what actions a browser should execute on their page, and a way for the site to receive reports if the policy is violated. Figure 1: Content-Security-Policy browser communication This is revolutionary, because it allows servers to receive feedback in real time on errors that may be appearing in the browser s console.

Trusted Types can be rolled out incrementally, with continuous feedback Web.dev s article on Trusted Types explains how to safely roll out the feature using the features of CSP itself:

• Deploy a CSP collector if you haven t already
• Switch on CSP reports without enforcement (via Content-Security-Policy-Report-Only headers)
• Iteratively review and fix the violations
• Switch to enforcing mode when there are a low enough rate of reports

Static analysis in a continuous integration pipeline is also sensible you want to prevent regressions shipping in new releases before they trigger a flood of CSP reports. This will also give you a chance of finding any low-traffic vulnerable pages.

Smart security teams will use techniques like Trusted Types to eliminate entire classes of bugs at a time Rather than playing whack-a-mole with unreliable vulnerability scanning or bug bounties, techniques such as Trusted Types are truly in the spirit of Secure by Design build high quality in from the start of the engineering process, and do this in a way which closes the DevSecOps feedback loop between your Developer, Operations and Security teams.

---

1. Sanitization libraries are especially needed when the examples become more complex, e.g. if the application must manipulate the input. DOMPurify version 1.0.9 also added Trusted Types support, so can still be used to help developers adopt this feature.
2. Trusted Types has existed in Chrome and Edge since 2020, and should soon be coming to Firefox as well. However, it s not necessary to wait for Firefox or Safari to add support, because the large market share of Chrome and Edge will let you identify and fix your site s DOM-XSS issues, even if you do not set enforcing mode, and users of all browsers will benefit. Even so, it is great that Mozilla is now on board.

This post was possibly inspired by my daughter s homework assignment to interview an old person about technology change. Guess who s old now? Sometimes I look back at how life used to be, and remember what it was like. There are two key nostalgia points for me: before internet, and before smartphones.

Before the internet Before the internet, there were computers. There were always computers in my life, they were just less fancy, with mainly text and fewer graphics at first. These days we adapt our user interfaces to look more like DOS and call it retro , although claiming it s more efficient that way. Sometimes it is. The written word is a fundamental mode of human communication it resonates. Some of my earliest programming memories were typing lines of BASIC code into some sort of BBC Micro, usually not successfully. Most of my computer learning was largely theoretical, through Usborne books (now available online! Thanks internet for reducing the marginal cost of publishing to approximately zero) but without the benefit of a computer of my own. I went for a walk this morning, with a slight bite to the air, and bought a newspaper, like the old person I am (I m leaning in). Imagine that - a physical paper full of news. When I was an inquisitive sixth-former, TheGuardian cost 50 pence today it was 2.80. This increase far outstrips the rate of inflation, which would bring it to around 90 pence. No, this must surely reflect the drop in circulation of the physical edition, and the rise in advertisement-filled web content.

Before smartphones As the internet became mainstream, I remember dial-up, and Freeserve, and even the tail end of USENET, and IRC. This was still a time when email was a thing you had to be at your desk to check. Feature phones were very useful, but it was still possible to get lost. My university halls had a shared land-line phone on each floor. And I wrote on the internet. I remember commenting on blogs, and I think my writing was better back then; more free, less inhibited. I do find myself giving less attention to news squeezed into the small form-factor of a mobile phone. Chat messages and notification pings. I m not convinced this is great progress for humanity.

At ArgoCon today, Thomas Fricke gave a nice talk on Cloud Native Deployments in Air Gapped Environments describing container vulnerability scanning in the German energy sector and since he didn t mention data diodes, and since some of my colleagues at Oakdoor/PA Consulting make data diodes for a living, I thought this might be interesting to write about! It s one thing to have an air-gapped system, but eventually in order to be useful you re going to have to move data into it, and this is going to need something better than just plugging a USB stick into your critical system. Just ask Iran how well this goes. Eight years after Stuxnet, the UK National Cyber Security Centre published the NCSC Safely Importing Data Pattern - but I found this a bit cryptic on first reading, because it s not clear what type of systems the pattern applies to, and deliberately uses technology-neutral language. Also, this was published around the same time GDPR was being implemented, mentions sensitive or personal data , and claims to be aimed at small to medium organisations - but I don t know how many small businesses implement a MILS security architecture. So without picking up on the mention of data diode , you can be left scratching your head about how to actually implement the pattern. One answer using Oakdoor components:

• PySISL, a Python library which you use to transform the data into a very simple format called SISL
• an Oakdoor Import Diode, which can verify the syntax of SISL in hardware, and prevent any data moving back the other way
• then some more PySISL code to validate the semantics of SISL on the high side and reconstruct the original format

The Oakdoor diodes themselves are quite interesting - they re electrical rather than optical like most data diodes. The other thing I d always wondered is how on earth you could even establish a TCP handshake across one - the answer is, you can t, so you use a UDP-based protocol like TFTP for file transfer. In this way, you build the transform/verify and protocol break that the NCSC pattern requires. Congratulations, you can now import your documents to your otherwise air-gapped system without also importing malicious code, and without risking data exfiltration. Note carefully that the Safely Importing Data pattern makes no guarantees about the integrity of your documents - they could be severely modified going through this process. For the same reason, I anticipate challenges applying this pattern to software binaries.

Welcome to the February 2023 report from the Reproducible Builds project. As ever, if you are interested in contributing to our project, please visit the Contribute page on our website.

---

FOSDEM 2023 was held in Brussels on the 4th & 5th of February and featured a number of talks related to reproducibility. In particular, Akihiro Suda gave a talk titled Bit-for-bit reproducible builds with Dockerfile discussing deterministic timestamps and deterministic apt-get (original announcement). There was also an entire track of talks on Software Bill of Materials (SBOMs). SBOMs are an inventory for software with the intention of increasing the transparency of software components (the US National Telecommunications and Information Administration (NTIA) published a useful Myths vs. Facts document in 2021).
On our mailing list this month, Larry Doolittle was puzzled why the Debian verilator package was not reproducible [ ], but Chris Lamb pointed out that this was due to the use of Python s datetime.fromtimestamp over datetime.utcfromtimestamp [ ].
James Addison also was having issues with a Debian package: in this case, the alembic package. Chris Lamb was also able to identify the Sphinx documentation generator as the cause of the problem, and provided a potential patch that might fix it. This was later filed upstream [ ].
Anthony Harrison wrote to our list twice, first by introducing himself and their background and later to mention the increasing relevance of Software Bill of Materials (SBOMs):

As I am sure everyone is aware, there is a growing interest in [SBOMs] as a way of improving software security and resilience. In the last two years, the US through the Exec Order, the EU through the proposed Cyber Resilience Act (CRA) and this month the UK has issued a consultation paper looking at software security and SBOMs appear very prominently in each publication. [ ]

Tim Retout wrote a blog post discussing AlmaLinux in the context of CentOS, RHEL and supply-chain security in general [ ]:

Alma are generating and publishing Software Bill of Material (SBOM) files for every package; these are becoming a requirement for all software sold to the US federal government. What s more, they are sending these SBOMs to a third party (CodeNotary) who store them in some sort of Merkle tree system to make it difficult for people to tamper with later. This should theoretically allow end users of the distribution to verify the supply chain of the packages they have installed?

---

Debian

• Vagrant Cascadian noted that the Debian bookworm distribution has finally surpassed bullseye for reproducibility: 96.1% vs. 96.0%, despite having over 3500 more packages in the distribution.
• Roland Clobus posted his latest update of the status of reproducible Debian ISO images noting that all major desktops build reproducibly with bullseye, bookworm and sid, with the caveat that when non-free firmware is activated, some non-reproducible files are generated .
• FC Stegerman submitted a new Intent to Package (ITP) bug report representing an intention to package repro-apk, a set of scripts to make Android .apk files reproducible.
• 23 reviews of Debian packages were added, 24 were updated and 20 were removed this month adding to our knowledge about identified issues. A new issue was added and identified by Chris Lamb [ ], and the timestamps_embedded_in_manpages_by_node_marked_man issue has been marked as resolved [ ].

---

F-Droid & Android

• This month, F-Droid added 21 apps published with reproducible builds (out of 33 new apps in total), the overview of F-Droid apps published with Reproducible Builds now includes graphs, and there are now also some graphs of F-Droid apps verified by the Verification Server.
• FC Stegerman noticed that signatures made by older versions of Android Gradle plugin cannot be copied because the signing method differs too much from that used by apksigner (and signflinger).
• FC Stegerman also created a helpful HOWTO page on the F-Droid Wiki detailing how to compare and subsequently make APKs reproducible.
• A long-running thread on Hiding data/code in Android APK embedded signatures continued on our mailing list this month; apksigcopier v1.1.1 and reproducible-apk-tools v0.2.2 + v0.2.3 were also announced on the same list.
• Lastly, FC Stegerman reported two issues on Google s own issue tracker: one related to a non-deterministic Dependency Info Block [ ] and another about a virtual entry added by the signflinger tool causing unexpected differences between signed and unsigned APKs [ ].

---

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb released versions 235 and 236; Mattia Rizzolo later released version 237. Contributions include:

• Chris Lamb:
  • Fix compatibility with PyPDF2 (re. issue #331) [ ][ ][ ].
  • Fix compatibility with ImageMagick version 7.1 [ ].
  • Require at least version 23.1.0 to run the Black source code tests [ ].
  • Update debian/tests/control after merging changes from others [ ].
  • Don t write test data during a test [ ].
  • Update copyright years [ ].
  • Merged a large number of changes from others.
• Akihiro Suda edited the .gitlab-ci.yml configuration file to ensure that versioned tags are pushed to the container registry [ ].
• Daniel Kahn Gillmor provided a way to migrate from PyPDF2 to pypdf (#1029741).
• Efraim Flashner updated the tool metadata for isoinfo on GNU Guix [ ].
• FC Stegerman added support for Android resources.arsc files [ ], improved a number of file-matching regular expressions [ ][ ] and added support for Android dexdump [ ]; they also fixed a test failure (#1031433) caused by Debian s black package having been updated to a newer version.
• Mattia Rizzolo:
  • updated the release documentation [ ],
  • fixed a number of Flake8 errors [ ][ ],
  • updated the autopkgtest configuration to only install aapt and dexdump on architectures where they are available [ ], making sure that the latest diffoscope release is in a good fit for the upcoming Debian bookworm freeze.

---

reprotest Reprotest version 0.7.23 was uploaded to both PyPI and Debian unstable, including the following changes:

• Holger Levsen improved a lot of documentation [ ][ ][ ], tidied the documentation as well [ ][ ], and experimented with a new --random-locale flag [ ].
• Vagrant Cascadian adjusted reprotest to no longer randomise the build locale and use a UTF-8 supported locale instead [ ] (re. #925879, #1004950), and to also support passing --vary=locales.locale=LOCALE to specify the locale to vary [ ].

Separate to this, Vagrant Cascadian started a thread on our mailing list questioning the future development and direction of reprotest.

---

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:
  • aiohttp (build fails in the future)
  • diff-pdf
  • dpdk
  • ebumeter (CPU-related issue)
  • firecracker (hashmap ordering issue)
  • jhead/gcc (used random temporary directory name)
  • libhugetlbfs (drop unused unreproducible file)
  • prosody (generates nondeterministic example SSL certificates)
  • python-sqlalchemy-migrate (clean files leftover by Sphinx)
  • tigervnc (random RSA key)
• Chris Lamb:
  • #1030708 filed against gap-browse.
  • #1030714 filed against cwltool.
  • #1030715 filed against adacgi.
  • #1030724 filed against node-marked-man (forwarded upstream).
  • #1030727 filed against multipath-tools.
  • #1031030 filed against ruby-pgplot.
  • #1031412 filed against pysdl2.
  • #1031829 filed against gawk.
  • #1032057 filed against pyproject-api.
• Gioele Barabucci:
  • #1032056 filed against systemtap.
• Larry Doolittle:
  • #1031711 filed against verilator.
• Vagrant Cascadian:
  • #1030270 filed against libreoffice.

---

Testing framework The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In February, the following changes were made by Holger Levsen:

• Add three new OSUOSL nodes [ ][ ][ ] and decommission the osuosl174 node [ ].
• Change the order of listed Debian architectures to show the 64-bit ones first [ ].
• Reduce the frequency that the Debian package sets and dd-list HTML pages update [ ].
• Sort Tested suite consistently (and Debian unstable first) [ ].
• Update the Jenkins shell monitor script to only query disk statistics every 230min [ ] and improve the documentation [ ][ ].

---

Other development work disorderfs version 0.5.11-3 was uploaded by Holger Levsen, fixing a number of issues with the manual page [ ][ ][ ].
Bernhard M. Wiedemann published another monthly report about reproducibility within openSUSE.

---

If you are interested in contributing to the Reproducible Builds project, please visit the Contribute page on our website. You can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Twitter: @ReproBuilds
• Mastodon: @reproducible_builds@fosstodon.org
• Mailing list: rb-general@lists.reproducible-builds.org

At CentOS Connect yesterday, Jack Aboutboul and Javier Hernandez presented a talk about AlmaLinux and SBOMs [video], where they are exploring a novel supply-chain security effort in the RHEL ecosystem. Now, I have unfortunately ignored the Red Hat ecosystem for a long time, so if you are in a similar position to me: CentOS used to produce debranded rebuilds of RHEL; but Red Hat changed the project round so that CentOS Stream now sits in between Fedora Rawhide and RHEL releases, allowing the wider community to try out/contribute to RHEL builds before their release. This is credited with making early RHEL point releases more stable, but left a gap in the market for debranded rebuilds of RHEL; AlmaLinux and Rocky Linux are two distributions that aim to fill that gap. Alma are generating and publishing Software Bill of Material (SBOM) files for every package; these are becoming a requirement for all software sold to the US federal government. What s more, they are sending these SBOMs to a third party (CodeNotary) who store them in some sort of Merkle tree system to make it difficult for people to tamper with later. This should theoretically allow end users of the distribution to verify the supply chain of the packages they have installed? I am currently unclear on the differences between CodeNotary/ImmuDB vs. Sigstore/Rekor, but there s an SBOM devroom at FOSDEM tomorrow so maybe I ll soon be learning that. This also makes me wonder if a Sigstore-based approach would be more likely to be adopted by Fedora/CentOS/RHEL, and whether someone should start a CentOS Software Supply Chain Security SIG to figure this out, or whether such an effort would need to live with the build system team to be properly integrated. It would be nice to understand the supply-chain story for CentOS and RHEL. As I write this, I m also reflecting that perhaps it would be helpful to explain what happens next in the SBOM consumption process; i.e. can this effort demonstrate tangible end user value, like enabling AlmaLinux to integrate with a vendor-neutral approach to vulnerability management? Aside from the value of being able to sell it to the US government!

LWN reminds us that Git still uses SHA-1 by default. Commit or tag signing is not a mitigation, and to understand why you need to know a little about Git s internal structure. Git internally looks rather like a content-addressable filesystem, with four object types: tags, commits, trees and blobs. Content-addressable means changing the content of an object changes the way you address or reference it, and this is achieved using a cryptographic hash function. Here is an illustration of the internal structure of an example repository I created, containing two files (./foo.txt and ./bar/bar.txt) committed separately, and then tagged: You can see how trees represent directories, blobs represent files, and so on. Git can avoid internal duplication of files or directories which remain identical. The hash function allows very efficient lookup of each object within git s on-disk storage. Tag and commit signatures do not directly sign the files in the repository; that is, the input to the signature function is the content of the tag/commit object, rather than the files themselves. This is analogous to the way that GPG signatures actually sign a cryptographic hash of your email, and there was a time when this too defaulted to SHA-1. An attacker who can break that hash function can bypass the guarantees of the signature function. A motivated attacker might be able to replace a blob, commit or tree in a git repository using a SHA-1 collision. Replacing a blob seems easier to me than a commit or tree, because there is no requirement that the content of the files must conform to any particular format. There is one key technical mitigation to this in Git, which is the SHA-1DC algorithm; this aims to detect and prevent known collision attacks. However, I will have to leave the cryptanalysis of this to the cryptographers! So, is this in your threat model? Do we need to lobby GitHub for SHA-256 support? Either way, I look forward to the future operational challenge of migrating the entire world s git repositories across to SHA-256.

At the end of March, the source code to StackRox was released, following the 2021 acquisition by Red Hat. StackRox is a Kubernetes security tool which is now badged as Red Hat Advanced Cluster Security (RHACS), offering features such as vulnerability management, validating cluster configurations against CIS benchmarks, and some runtime behaviour analysis. In fact, it s such a diverse range of features that I have trouble getting my head round it from the product page or even the documentation. Source code is available via the StackRox organisation on GitHub, and the most obviously interesting repositories seem to be:

• stackrox/stackrox, containing the main application, written in Go
• stackrox/scanner, the vulnerability scanner, also in Go. From a first glance at the go.mod file, it does not seem to share much code with Clair, which is interesting.
• stackrox/collector, the runtime analysis component, in C++ but also with hooks into the kernel.

My initial curiosity has been around the collector , to better understand what runtime behaviour the tool can actually pick up. I was intrigued to find that the actual kernel component is a patched version of Falco s kernel module/eBPF probes; a few features are disabled compared to Falco, e.g. page faults and signal events. There s a list of supported syscalls in driver/syscall_table.c, which seems to have drifted slightly or be slightly behind the upstream Falco version? In particular I note the absence of io_uring, but given RHACS is mainly deployed on Linux 4.18 at the moment (RHEL 8) this is probably a non-issue. (But relevant if anyone were to run it on newer kernels.) That s as far as I ve got for now. Red Hat are making great efforts to reach out to the community; there s a Slack channel, and office hours recordings, and a community hub to explore further. It s great to see new free software projects created through acquisition in this way - I m not sure I remember seeing a comparable example.

Public Service Announcement for anyone relying on Jenkins for continuous deployment - the milestone step plugin as of version 1.3.1 will not function correctly if you could have more than two builds running at once - older builds could get deployed after newer builds. See JENKINS-46097. A possible workaround is to add an initial milestone at the start of the pipeline, which will then allow builds to be killed early. (Builds are only killed early once they have passed their first milestone.) Going by the source history, I reckon this bug has been present since the milestone-step plugin was created.