Welcome to post 59 in the
R4 series.
How did we get here: A initial words about GitHub.
GitHub Actions
provides (essentially unlimited) compute time. This further boosts a
service already in a market-dominating position:
GitHub as a code repository.
Those of us old enough to remember the start of
git (the
program and protocol) may remember the
extremely bare-bones
initial hosting site
repo.or.cz (launched in 2006). GitHub
came two years later, and put an enormous amount of focus into design
and user interfaces. To cut a long story short, GitHub won the services
war. And with it
git won the platform war. To a first
approximation, everybody and everything is on GitHub. So
the repository is already dominant. And then free compute
was added.
So given its scale and positioning, and its essentially free
provisioning of free multi-core compute setups with generally decent
connectivity, widespread adoption happened. And as is goes, some
mischief is bound to happen. And it did. More on that below.
A few words about r2u:
r2u makes all packages on
CRAN, i.e. the code repository
network for R, install
fast,
reliably and
easy on Ubuntu by making them available to
apt,
the native package manager. It is to our knowledge also the first and
only time an entire open source programming repository is available in
binary form with all dependencies resolved. It is going strongly: the
last monthly use
topped
five million packages. See the
r2u website for more.
r2u and GitHub: For the first few years,
builds for
r2u
were done locally on my machine, and then uploaded to the primary
repositry r2u.stat.illinois.edu. I do not recall systemic outages or
connection issues though occassional network timeouts were seen. Once we
started to support arm64 (in addition to the default amd64) binaries,
building those switched to GitHub Actions simply because they had
runners for arm64 while I had no arm64 hardware. The experience of
building packages (in bulk) was rather positive. So we investigated
builds for amd64 too. If memory serves we first did this for either one
of the semi-annual BioConductor updates. Before long, builds for amd64
followed meaning
all of
r2u was being built in
GitHub Actions.
During these builds, I would regularly encounter builds failures:
cannot connect to r2u.stat.illinois.edu . I misdiagnosed this as a
resource issue on the GitHub side, and consequently made (several)
attempts at robustifying the builds via for example longer (download)
timeout limits as well as checks for build failures and conditional
rebuilds. Needless to say, and given what we know now (more on that
below), this did not work. But it went on for a few months this spring
and summer. What did work was to simply relaunch under re-run failed
jobs . Given the distributed nature of GitHub Action this generally
allocates to a different machine and address and succeeds. In the grand
scheme of things a nuisance as we a need second run, but given the
fourty (!!) concurrent jobs this tends to be quick. So a minor
nuisance.
This discribed the
production side. On the
consumption side, one prominent user of
r2u, especially at GitHub,
is our
r-ci setup for
continuous integration. It too could fail at times, and a simple re-run
would fix it. Annoying, if addressable manually. Usage by others I
cannot monitor so I can only assume that the random failure nature must
have frustrated them too. Potentially a much bigger nuisance.
As users were getting annoyed, some took action.
Jeffrey Girard opened
discussion
topic #159 which contained a thorough investigation of his
confirming that only amd64 nodes were affected. This had not been
noticed before.
Troy
Hernandez set up a full harness with tests in an
ad-hoc repo
designed for repeated remote triggering. This also logged the IP
addresses for success or failure. Through both these approaches it
became (eventually) clear that the failures were limited to either
certain (individual) IP addresses, or IP subnets.
When taking the conversation back to network service at U of
Illinois, we realized that the issue was in fact caused by a network
policy at the university. And specific to GitHub.
In fact, what happened initially were waves of port scanning attacks
originating from GitHub IP addresses. As (essentially) anybody can run
code there, bad actors can too. The response from the university side
was reasonable and swift: Identified IP addresses were added to a
null-router that (essentially) swallows traffic. And that was the
cause of the perceived-as-random outages: Jobs that ended up failing at
GitHub Actions were the ones assigned to addresses that have previously
been seen as port scanning.
Shifting production: Once this was confirmed, I
investiaged alternatives. On the
production side using
different machines would help. So I tried
blacksmith.sh, a competing
alternate service offering faster runners as drop-in replacements for
the GitHub Actions runners. This worked
great, until I ran up
against my free cpu minutes quota . In a mere two days (that were
arguably overly busy as it was shortly after
CRAN reopened after the summer
break). Given that the service would not sponsor us a supported open
source software project with sufficient quota, we moved off
blacksmith.sh after two days.
A first programmatic response: consumption-side: For
the
r-ci client side,
it was straightforward to setup a check and subsequent workaround. When
curl fails with a silent HEAD attempt at the primary
repository failed, we take this to be caused by presence of a
null-router entry for the IP we are on, and switch the
apt
setup to the secondary repository. Which may be slower, or at rare times
unreachable itself but still provides a fine fallback when a node is
prohibited from talking to U of Illinois resources such as
r2u.stat.illinois.edu. Having used this for a few days in
r-ci it seems to work.
A second programmatic response: production-side: For
the
r2u builds, and
given that
blacksmith.sh would
not grant most-favored status with sufficient free minutes, we
switched our Docker-based setup to switch to the secondary when an
initial probe fails. That was added last weekend, and appears to work
just swimmingly. Another application to the
fundamental
theorem of software engineering: another layer of indirection can
solve just about any problem.
For completeness, the corresponding code is
webstatus=$(curl --head --silent --no-fail --output /dev/null \
--write-out "% http_code " https://r2u.stat.illinois.edu true)
if test "$ webstatus " = "200"; then
echo "The r2u repository is reachable."
else
extip=$(curl --silent https://ipinfo.io/ip)
echo "::notice::The primary r2u repository is **not reachable** from $ extip ."
fi
We run an initial
curl test (without failing) and have
it report the HTTP return code. 200 means no issue, all others are
suspect here so we run a second
curl query to obtain our
external IP and log it. We use the same logic in another spot from
inside the build container and use the else branch to switch
apt to the secondary repository via
sed call
on the
.sources file.
Logging of bad IPs: On both our sides,
i.e. production as well as consumption, we now also log the IP addresses
of the failing nodes and will ask network security to remove these from
the null router. If our jobs can be assigned to them it clearly shows
the machines are part of the normal compute pool and are
not
doing anything nefarious at the moment. So they should be removed from
the null-router list. We will see how that fares.
Putting it all together: Providing a free resources
can, sadly, lead to an a decline the service experience just as the
tragedy of
the commons analysis would predict. Restricting, or pricing use
may be a stock answer but I for one am glad GitHub Actions is still
free. But we need to do our bit of upkeep. Just as network security logs
bad actors (taking advantage of the free resource) we should make an
effort to unlist nodes no longer part of any portscan (or alike)
swarm.
For r-ci users, there is hopefully little to do (if
you rely on the
standard
action). We do now catch a node that was assigned a continuous
integration job cannot connect to
r2u as we can test this
easily (and cheaply). Pivoting to the secondary repository is a valid,
and working, answer. Hopefully over time we can also work towards
restricting the null-router list down to recent entries and fewer
overall, thereby lowering the chance of gitting a bad IP. Eventually, we
could also overly a CDN proxy to avoid the bad IP problem. It is
something to consider.
Summing up: We are still chuffed at how successful
r2u has become, and how
much can be done with
GitHub Actions. Sadly, as
we found out, there can also be a tax on letting compute happen there
but as discussed in this note, there are ways to avoid it by pivoting to
alternate repository source.
This post by Dirk
Eddelbuettel originated on his Thinking inside the box
blog. If you like this or other open-source work I do, you can now sponsor me at
GitHub.
This is a new series1 I'm starting on the tools I have been crafting and
buying to play pedagogical games in my Economics classes.
These games, aside from being just plain fun for the students, often aim to
teach them a lesson via interactive play. Unsurprisingly, it is much easier for
them to grok a particular concept this way. Moreover, many students told me
these activities were the best part of the classes I teach.
The World Trade Game
I found out about The World Trade Game in The Handbook of Pluralist Economics
Education2, which also contains many other fun pedagogical games. It
was first developed by Action Aid, but has since been refined by John
Sloman and then by Irene van Staveren, who wrote about her use of the game in
the aforementioned book.
The goal of this blog entry is not to describe the game and teach you how to
play it. If that interests you, either buy the book I mentioned or read about
Sloman's version here.
No, our focus point here is the tools to play this game: fake cash! Fake cash
can be used in many economics pedagogical games and in my opinion, is a
worthwhile "investment" of your time and energy. You could indeed play with
regular printed sheets of paper listing a value (I sure did!), but using Play
Money (a more appropriate term, since we're not trying to defraud anyone) will
up your game and create a more immersive atmosphere in your classes.
At first, I tried to buy play money. Surely, some commercial offering aiming to
create realistic props for movies and TV productions existed? At least in
Canada, it turns out the answer was no. The only thing I could find were shady
websites and weird expensive listings on AliExpress. Not being tenured yet, I
did not try to convince my employer to take a 50-50 chance on supporting
criminal endeavours.
As such, the next best thing I found was printable play money designed by the
Bank of Canada Museum. Their designs were quite nice and met my needs,
but their PDF layout made it overly complicated to print large quantities of
bills. As such, I played around with GIMP a little and came out with these
files. A commercial printer shouldn't have issues printing them on glossy
cardstock paper and cutting them up for you:
The Reproducible Builds project 
Given i m going to stub most of that out, let s take a look at what some
similar Layer 2 protocols use things like Ethernet or WiFi frames. Both
contain structured information regarding the transmitter, desired recipient,
type of data, and the higher-level data itself (such as IP packets). As a
result of attempting to learn from others, the Pigeon Layer 2 (called, simply,
While the process here can look like a lot, it s really not that bad.
We begin by taking the incoming bytes, converting the bytes into bits, and chunk
those bits into parts which are sized to fit completely within an LDPC message. We will then
Here, we start with an incoming stream of IQ, where we will process
Not shockingly, the code looks like this:
As an aside, I likely should have called this a
The following contributor got their Debian Developer account in the last two
months:
The 
SIMD bitpacking works like regular bitpacking, but processes 8 
Sharing source code from host to guest VM
Finally we need to make sure we have access in the dev-vm to our source code repositories.
For this I will share the directory 
Finally we mount the shared directory, and enable the mount on each boot.
Summer Vacation for my kids is over. And Peace is back to my life.
AI is transforming how I operate and view things. It was very different a few months back.
AI (as a product) is useful in generating code, useful in analysing things. It seems to be able to retrieve and show me information relatively quickly, doesn't need me to scan the search results to find which one is more useful.
I feel I am less reliable than an AI, even when AI is prone to failure.
The text generated by AI is better worded than me myself, albeit they have their own tone.
Is it still fun if all my hobby programming is overtaken by AI? I am not sure, did I enjoy writing the fixtures and build environment for the
open source programming stuff? Do I enjoy reviewing other people's code?
Reviewing other people's contributions is usually not great, because by definition the code you own you have better knowledge about,
and the code you generate yourself is the best code, others will not fit naturally, they don't have the historical context, and the undocumented future plans.
I just released a new version of 
I spent 6 hours working with an AI to write a bash script that would show me both active & inactive reserved DHCP addresses, as well as active leases from pool. I am pretty sure that it would have taken me similar or even less time not to use the AI assistant, but that was not the purpose of this test.