hz.tools will be tagged
#hztools.Frequency Changes and
Discontinuities. Since then, ARF has grown a few more.
After getting all that down, I opted to start at what I thought the simplest
container would look like,
TLV
(tag-length-value) encoded packets. This is a fairly well trodden path,
and used by a bunch of existing protocols
we
all
know
and
love.
Each ARF file (or stream) was a set of
encoded packets (sometimes called data units in other specs). This means that
unknown packet types may be skipped (since the length is included) and
additional data can be added after the existing fields without breaking
existing decoders.
| Flag | Description |
| 0x01 | Critical (tag must be understood) |
tag field. This tag indicates how the contents of the
value field should be interpreted.
| Tag ID | Description |
| 0x01 | Header |
| 0x02 | Stream Header |
| 0x03 | Samples |
| 0x04 | Frequency Change |
| 0x05 | Timing |
| 0x06 | Discontinuity |
| 0x07 | Location |
| 0xFE | Vendor Extension |
00, // tag (0; no subpacket is 0 yet)
00, // flags (0; no flags)
00, 00 // length (0; no data)
// data would go here, but there is none
sparky)
stores Frequency data internally, which makes conversion between the two
natural.
| ID | Name | Description |
| 0x01 | f32 | interleaved 32 bit floating point scalar values |
| 0x02 | i8 | interleaved 8 bit signed integer scalar values |
| 0x03 | i16 | interleaved 16 bit signed integer scalar values |
| 0x04 | u8 | interleaved 8 bit unsigned integer scalar values |
| 0x05 | f64 | interleaved 64 bit floating point scalar values |
| 0x06 | f16 | interleaved 16 bit floating point scalar values |
00, 00, 00, fa, de, dc, ab, 1e, // magic
00, 00, 00, 00, 00, 00, 00, 00, // flags
18, 27, a6, c0, b5, 3b, 06, 07, // start time (1740543127)
// guid (fb47f2f0-957f-4545-94b3-75bc4018dd4b)
fb, 47, f2, f0, 95, 7f, 45, 45,
94, b3, 75, bc, 40, 18, dd, 4b,
// site_id (ba07c5ce-352b-4b20-a8ac-782628e805ca)
ba, 07, c5, ce, 35, 2b, 4b, 20,
a8, ac, 78, 26, 28, e8, 05, ca
num streams field of the Header. This has the nice effect of
enabling clients to read all the stream headers without requiring buffering of
unread packets from the stream.
00, 01, // id (1)
00, 00, 00, 00, 00, 00, 00, 00, // flags
01, // format (float32)
01, // byte order (Little Endian)
00, 00, 01, d1, a9, 4a, 20, 00, // rate (2 MHz)
00, 00, 5a, f3, 10, 7a, 40, 00, // frequency (100 MHz)
// guid (7b98019d-694e-417a-8f18-167e2052be4d)
7b, 98, 01, 9d, 69, 4e, 41, 7a,
8f, 18, 16, 7e, 20, 52, be, 4d,
// site_id (98c98dc7-c3c6-47fe-bc05-05fb37b2e0db)
98, c9, 8d, c7, c3, c6, 47, fe,
bc, 05, 05, fb, 37, b2, e0, db,
format and
byte_order field sent in the related Stream Header.
01, // id
ab, cd, ab, cd, // iq samples
01, // id
00, 00, b5, e6, 20, f4, 80, 00 // frequency (200 MHz)
01, // id
sys field indicates the Geodetic system to be used for the provided
latitude, longitude and elevation fields. The full list of supported
geodetic systems is currently just WGS84, but in case something meaningfully
changes in the future, it d be nice to migrate forward.
Unfortunately, being a bit of a coward here, the accuracy field is a bit of a
cop-out. I d really rather it be what we see out of kinematic state estimation
tools like a kalman filter, or at minimum, some sort of ellipsoid. This is
neither of those - it s a perfect sphere of error where we pick the largest
error in any direction and use that. Truthfully, I can t be bothered to model
this accurately, and I don t want to contort myself into half-assing something
I know I will half-ass just because I know better.
| System | Description |
| 0x01 | WGS84 - World Geodetic System 1984 |
00, 00, 00, 00, 00, 00, 00, 00, // flags
01, // system (wgs84)
3f, f3, be, 76, c8, b4, 39, 58, // latitude (1.234)
40, 02, c2, 8f, 5c, 28, f5, c3, // longitude (2.345)
40, 59, 00, 00, 00, 00, 00, 00, // elevation (100)
40, 24, 00, 00, 00, 00, 00, 00 // accuracy (10)
// extension id (b24305f6-ff73-4b7a-ae99-7a6b37a5d5cd)
b2, 43, 05, f6, ff, 73, 4b, 7a,
ae, 99, 7a, 6b, 37, a5, d5, cd,
// data (0x01, 0x02, 0x03, 0x04, 0x05)
01, 02, 03, 04, 05
u16 65535 bytes. Given the u8 sample header, this limits us
to 8191 32 bit sample pairs at a time. I wound up believing that the overhead in
terms of additional packet framing is worth it because always encoding 4
byte lengths felt like overkill, and a dynamic length scheme ballooned
codepaths in the decoder that I was trying to keep as easy to change as
possible as I worked with the format.
| Publisher: | Ace |
| Copyright: | November 2024 |
| ISBN: | 0-593-81590-4 |
| Format: | Kindle |
| Pages: | 324 |
On June 19 and 20, I will cycle a little over 100 miles from downtown
Chicago and its wonderful Millenium Park to New Buffalo, Michigan, as
part of the Tour de Shore
2026. The ride passes through northwest Indiana and the extended Indiana Dunes National
Park ending the next morning in the southwestern Michigan town of
New Buffalo. I rode Tour de Shore once before in 2024 and had a
generally wonderful time (even considering some soreness after a century
of miles over 1 1/2 days).
Tour de Shore is riding
in support of Maywood Fine Arts
Center, a local arts and sports center in Maywood,
Illinois, a suburb one over from where I live and hence just a few
good miles west of downtown. Maywood,
Illinois is home to legends such as the late John Prine as well
as several NBA players such as player and coach Doc Rivers.
But Maywood,
Illinois is also little less well off than other western suburbs.
The Maywood Fine Arts Center
is simply legendary is what they do for this community (and surrounding
communities), and especially the youth support. They can use a dollar a
two. Their
story about Tour de Shore is worth a read too for background and
motivation.
I have bootstrapped my
donation page page with a dollar for each mile to be cycled. It
would be simply terrific if you could join me. A nickel, a dime, or a
quarter per mile cycled would help. Multiples of that help too: More is
of course still always better.
Anything you can afford will go a long way towards a worthy goal in a
community that could use the help.
Of and if you are local to the area, I believe you can still register for Tour de
Shore 2026. So see you out there in June? And if not, maybe help
with a dollar or two?
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog.
Because I am bad at giving up on things, I ve been running my own email
server for over 20 years. Some of that time it s been a PC at the end of a
DSL line, some of that time it s been a Mac Mini in a data centre, and some
of that time it s been a hosted VM. Last year I decided to bring it in
house, and since then I ve been gradually consolidating as much of the rest
of my online presence as possible on it. I mentioned this on
Mastodon and a
couple of people asked for more details, so here we are.
First: my ISP doesn t guarantee a static
IPv4 unless I m on a business plan and that seems like it d cost a bunch
more, so I m doing what I described
here: running a Wireguard link
between a box that sits in a cupboard in my living room and the smallest
OVH instance I can, with an additional IP
address allocated to the VM and NATted over the VPN link. The practical
outcome of this is that my home IP address is irrelevant and can change as
much as it wants - my DNS points at the OVH IP, and traffic to that all ends
up hitting my server.
The server itself is pretty uninteresting. It s a refurbished HP EliteDesk
which idles at 10W or so, along 2TB of NVMe and 32GB of RAM that I found
under a pile of laptops in my office. We re not talking rackmount Xeon
levels of performance, but it s entirely adequate for everything I m doing
here.
So. Let s talk about the services I m hosting.
pmbr_boot flag on the GPT s protective MBR. Strict UEFIefidisk0 is empty and lacks both the trust certificatesEF00). Clear thepmbr_boot flag from the MBR. 2. Binaryshim and grub-efi packages. Usegrub2-mkconfig to populate the new ESP. 3. Trustvirt-fw-vars utility on theefidisk. 4. Boot Pinning: Explicitly setBootOrder to point to the shimx64.efivirt-fw-vars --append-boot-filepath.
root):
# Map and Clean MBR
DEV=$(rbd map pool/disk)
parted -s $DEV disk_set pmbr_boot off
# Inject Trust and Boot Path (VM must be stopped)
virt-fw-vars --inplace /dev/rbd/mapped_efidisk \
--enroll-redhat \
--add-db <GUID> /path/to/ipa-ca.crt \
--append-boot-filepath '\EFI\centos\shimx64.efi' \
--sb
/etc/apt/sources.list.d/r-samueloph-nvidia-ai.sources, run sudo apt update and install the packages, you might need to disable Secure Boot.
.sources file you need, create the
file /etc/apt/sources.list.d/r-samueloph-nvidia-ai.sources with the sources for your release.
Run sudo apt update and install the packages you need, if you already have a
previous version installed, sudo apt upgrade --update would update them.
If there are no upgrades, meaning you don't have a previous version installed,
then you need to explicitly install them.
sudo apt install nvidia-open-kernel-dkms nvidia-driver
If you run into issues in Debian Stable, consider using the Linux kernel package
from the backports repository, if you need an up-to-date NVIDIA driver, you
likely should also be running the backports kernel package (if you can't
upgrade to Debian Testing).
Note: I have not published blog posts about my academic papers over the past few years. To ensure that my blog contains a more comprehensive record of my published papers and to surface them for folks who missed them, I will periodically (re) publish blog posts about some older published projects. This post draws material from a previously published post by Kaylea Champion on the Community Data Science Blog.
Taboo subjects such as sexuality and mental health are as important to discuss as they are difficult to raise in conversation. Although many people turn to online resources for information on taboo subjects, censorship and low-quality information are common in search results. In two papers I recently published at CSCW both led by Kaylea Champion we presented a series of analyses showing how taboo shapes the process of collaborative knowledge building on English Wikipedia.
The first study is a quantitative analysis showing that articles on taboo subjects are much more popular and are the subject of more vandalism than articles on non-taboo topics. In surprising news, we also found that they were edited more often and were of higher quality!
The first challenge we faced in conducting this work was identifying taboo articles. Kaylea had a brilliant idea for a new computational approach to doing so without relying on our individual intuitions about what qualifies as taboo (something we understood would be highly specific to our own culture, class, etc). Her approach was to make use of an insight from linguistics: people develop euphemisms as ways to talk about taboos (i.e., think about all the euphemisms we ve devised for death, or sex, or menstruation, or mental health).
We used this insight to build a new machine-learning classifier based on English Wiktionary definitions. If a sense of a word was tagged as euphemistic, we treated the words in the definition as indicators of taboo. The end result was a series of words and phrases that most powerfully differentiate taboo from non-taboo. We then did a simple match between those words and phrases and the titles of Wikipedia articles. The topics were taboo enough that we were a little uncomfortable discussing them in our meetings! We built a comparison sample of articles whose titles are words that, like our taboo articles, appear in Wiktionary definitions. In the first paper, we used this new dataset to test a series of hypotheses about how taboo shapes collaborative production in Wikipedia. Our initial hypotheses were based on the idea that taboo information is often in high demand but that Wikipedians might be reluctant to associate their names (or usernames) with taboo topics. The result, we argued, would be articles that were in high demand but of low quality. We found that taboo articles are thriving on Wikipedia! In summary, we found that in comparison to non-taboo articles:
Image of the estimated qualiy of articles of the four articles in the second mixed-methods paper. Extreme dips reflect periods of frequent vandalism.The full citation for the papers are: (1) Champion, Kaylea, and Benjamin Mako Hill. 2023. Taboo and Collaborative Knowledge Production: Evidence from Wikipedia. Proceedings of the ACM on Human-Computer Interaction 7 (CSCW2): 299:1-299:25. https://doi.org/10.1145/3610090. (2) Champion, Kaylea, and Benjamin Mako Hill. 2024. Life Histories of Taboo Knowledge Artifacts. Proceedings of the ACM: Human-Computer Interaction 8 (CSCW2): 505:1-505:32. https://doi.org/10.1145/3687044.
We have also released replication materials for the paper, including all the data and code used to conduct the analyses.
This blog post and the paper it describes are collaborative work by Kaylea Champion and Benjamin Mako Hill.
| Series: | Class 5 #5 |
| Publisher: | Eclipse |
| Copyright: | 2022 |
| ISBN: | 0-6454658-2-8 |
| Format: | Kindle |
| Pages: | 349 |
A map of Vietnam with points of places we went to labeled. CARTO MAPTILER OPENSTREETMAP
A picture of the park we roamed around in Ho Chi Minh City. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
This is the restaurant Badri s friend took us to. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
One of the dishes we had in the restaurant. This one was especially tasty. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
One of the dishes we had in the restaurant. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
These noodles are called Pho and are very popular in Vietnam. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Interior of our train coach. Trains in Vietnam don t have side berths, unlike India. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
A picture of the berths from our coach. It had three tiers, similar to a 3 AC coach in Indian trains. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
The train had a cabin to put the bags in. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Nha Trang train station. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
I got a complimentary cup of green tea along with coffee in Nha Trang. In this trip, Badri and I found out that this is customary at local places in Vietnam. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Eggs being cooked on a pan for my order. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
The beach we went to in Nha Trang. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
This one was shot in the city center. In this trip, Badri and I found out that this is customary at local places in Vietnam. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Inside a Highlands Coffee cafe in Nha Trang. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
A coffee I got from Highlands Coffee in Nha Trang. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
The restaurant we went to in Nha Trang. The word Chay in the name means it was a vegan restaurant. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Soupy noodles we got at that restaurant. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Dry noodles we got at that restaurant. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
This was the place we had our coffee in Tra Kieu. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Photo of Badri with taxi driver. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
A picture of Tra Kieu area near the train station we got down at. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Our breakfast in Hoi An. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
A photo of the hostel we stayed in Hoi An. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Typical yellow house with gabled roof in Hoi An old town. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Egg coffee I had in Hoi An. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Bahn Bao in Hue. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Hot Pot. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Added soup to the noodles. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Steamed rolls made of rice flour. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Menu from a restaurant in Ho Chi Minh City with detailed description of the food. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Our berths in the train. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Egg coffee in Hanoi. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Hanoi train street is a tourist attraction in Hanoi. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
A picture of pho we had in Hanoi. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Bahn it is served packed in banana leaves. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
Bahn it. Photo by Ravi Dwivedi, released under CC-BY-SA 4.0.
$ dig +short IN HTTPS cloudflare-ech.com
1 . alpn="h3,h2" ipv4hint=104.18.10.118,104.18.11.118 ech=AEX+DQBBFQAgACDBFqmr34YRf/8Ymf+N5ZJCtNkLm3qnjylCCLZc8rUZcwAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA= ipv6hint=2606:4700::6812:a76,2606:4700::6812:b76
systemd-networkd, ifupdown, NetworkManager,
nothing)ifupdown (/etc/network/interfaces): traditional static
configuration system, mostly for workstations and servers that has
been there forever in Debian (since at least 2000), documented
in the Debian wikisystemd-network: used more for servers, see Debian reference Doc
Chapter 5 (introduced some time around Debian 8 "jessie", in
2015)ifupdown is on its way out, possibly replaced
by systemd-networkd. NetworkManager already manages most desktop
configurations.
ip= on the Linux kernel command line: for servers with a
single IPv4 or IPv6 address, no software required other than the
kernel and a boot loader (since 2002 or older)So by "new" I mean "new to me". This option is really old. The
nfsroot.txt where it is documented predates the git import of the
Linux kernel: it's part of the 2005 git import of 2.6.12-rc2. That's
already 20+ years old already.
The oldest trace I found is in this 2002 commit, which imports
the whole file at once, but the option might goes back as far as
1996-1997, if the copyright on the file is correct and the option
was present back then.
ip= parameter to the kernel's
command-line. The syntax, as mentioned above, is in nfsroot.txt
and looks like this:
ip=<client-ip>:<server-ip>:<gw-ip>:<netmask>:<hostname>:<device>:<autoconf>:<dns0-ip>:<dns1-ip>:<ntp0-ip>
Most settings are pretty self-explanatory, if you ignore the useless
ones:
<client-ip>: IP address of the server<gw-ip>: address of the gateway<netmask>: netmask, in quad notation<device>: interface name, if multiple available<autoconf>: how to configure the interface, namely:
off or none: no autoconfiguration (static)on or any: use any protocol (default)dhcp, essentially like on for all intents and purposes<dns0-ip>, <dns1-ip>: IP address of primary and secondary name
servers, exported to /proc/net/pnp, can by symlinked to
/etc/resolv.conf<server-ip>: IP address of the NFS server, exported to /proc/net/pnp<hostnname>: Name of the client, typically sent over the DHCP
requests, which may lead to a DNS record to be created in some
networks<ntp0-ip>: exported to /proc/net/ipconfig/ntp_servers, unused by
the kernelip=[<server-id>]:<gateway-IP-number>:<netmask>:<client-hostname>:inteface:[dhcp dhcp6 auto6 on any none off]
It's essentially the same (although server-id is weird), and the
autoconf variable has other settings, so that's a bit odd.
ip=192.0.2.42::192.0.2.1:255.255.255.0:::off
... will set the IP address to 192.0.2.42/24 and the gateway to
192.0.2.1. This will properly guess the network interface if there's a
single one.
A DHCP only configuration will look like this:
ip=::::::dhcp
Of course, you don't want to type this by hand every time you boot the
machine. That wouldn't work. You need to configure the kernel
commandline, and that depends on your boot loader.
/etc/default/grub
(ugh) and find a line like:
GRUB_CMDLINE_LINUX=
and change it to:
GRUB_CMDLINE_LINUX=ip=::::::dhcp
systemd-boot UKI setups, it's simpler: just add the setting to
the /etc/kernel/cmdline file. Don't forget to include anything
that's non-default from /proc/cmdline.
This assumes that is the Cmdline=@ setting in
/etc/kernel/uki.conf. See 2025-08-20-luks-ukify-conversion for
my minimal documentation on this.
/etc/default/grub,
/boot/loader/entries/arch.conf for systemd-boot or
/etc/kernel/cmdline for UKI)/etc/default/grub, may be more RHEL mentions
grubby, possibly some systemd-boot things here as well)/etc/default/grub,
/efi/loader/entries/gentoo-sources-kernel.conf for systemd-boot,
or /etc/kernel/install.d/95-uki-with-custom-opts.install)/etc/default/grub is consistent across all
distributions above, while the systemd-boot setups are all over the
place (except for the UKI case), while I would have expected those be
more standard than GRUB.
dropbear-initramfs is setup, it already requires you to have
such a configuration, and it might not work out of the box.
This is because, by default, it disables the interfaces configured
in the kernel after completing its tasks (typically unlocking the
encrypted disks).
To fix this, you need to disable that "feature":
IFDOWN="none"
This will keep dropbear-initramfs from disabling the configured
interface.
ifupdown on servers
and NetworkManager on laptops, because that's essentially the
default. But on some machines, I've started using systemd-networkd
because ifupdown has ... issues, particularly with reloading network
configurations. ifupdown is a old hack, feels like legacy, and is
Debian-specific.
Not excited about configuring another service, I figured I would try
something else: just configure the network at boot, through the kernel
command-line.
I was already doing such configurations for dropbear-initramfs
(see this documentation), which requires the network the be up
for unlocking the full-disk encryption keys.
So in a sense, this is a "Don't Repeat Yourself" solution.
ifupdown does not either./etc/resolv.conf for you but the dns0-ip
and dns1-ip do end up in /proc/net/pnp which has a compatible
syntax, so a common configuration is:
ln -s /proc/net/pnp /etc/resolv.conf
apt purge systemd-networkd ifupdown network-manager netplan.io
Note that ifupdown (and probably others) leave stray files in (e.g.)
/etc/network which you might want to cleanup, or keep in case all
this fails and I have put you in utter misery. Configuration files for
other packages might also be left behind, I haven't tested this, no
warranty.
The original FOAAS site provided
a rather wide variety of REST access points, but it sadky is no more
(while the old repo
is still there). A newer replacement site FOASS is up and running, but with
a somewhat reduced offering. (For example, the two accessors shown in
the screenshot are no more. C est la vie.)
Recognising that perfect may once again be the enemy of (somewhat)
good (enough), we have rejigged the rfoaas package
in a new release 2.3.3. (The precding version number 2.3.2 corresponded
to the upstream version, indicating which API release we matched. Now we
just went + 0.0.1 but there is no longer a correspondence to the
service version at FOASS.)
Accessor functions for each of the now available access points are
provided, ans the random sampling accessor getRandomFO()
now picks from that set.
My CRANberries
service provides a comparison to the
previous release. Questions, comments etc should go to the GitHub issue
tracker. More background information is on the project page
as well as on the github repo
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.
Edit -> Preferences -> Protocols -> TLS -> (Pre)-Master-Secret log filename which is
uncomfortable to reach. Looking up the config setting in the
Advanced settings
one can learn that it's called internally tls.keylog_file. Thus we can set it up with:
sudo wireshark -o "tls.keylog_file:/home/sven/curl.keylog"
SSLKEYLOGFILE=/home/sven/curl.keylog curl -v https://www.cloudflare.com/cdn-cgi/trace
Depending on the setup root might be unable to access the wayland session, that can
be worked around by letting sudo keep the relevant env variables:
$ cat /etc/sudoers.d/wayland
Defaults env_keep += "XDG_RUNTIME_DIR"
Defaults env_keep += "WAYLAND_DISPLAY"
Or setup wireshark properly and use the wireshark group to be able to dump
traffic. Might require a sudo dpkg-reconfigure wireshark-common.
Regarding curl: In some situations it could be desirable to force
a specific older TLS version for testing, which requires a
minimal
and
maximal
version. E.g. to force TLS 1.2 only:
curl -v --tlsv1.2 --tls-max 1.2 https://www.cloudflare.com/cdn-cgi/trace
/backup/serverA.domain/.sync/foo
/backup/serverA.domain/daily.0/foo
/backup/serverA.domain/daily.1/foo
/backup/serverA.domain/daily.2/foo
...
/backup/serverA.domain/daily.6/foo
/backup/serverA.domain/weekly.0/foo
/backup/serverA.domain/weekly.1/foo
...
/backup/serverA.domain/monthly.0/foo
/backup/serverA.domain/monthly.1/foo
...
/backup/serverA.domain/yearly.0/foo
I can browse and rescue files easily, going back in time when needed.
The rsnapshot project README explains more, there is a long rsnapshot HOWTO although I usually find the rsnapshot man page the easiest to digest.
I have stored multi-TB Git-LFS data on GitLab.com for some time. The yearly renewal is coming up, and the price for Git-LFS storage on GitLab.com is now excessive (~$10.000/year). I have reworked my work-flow and finally migrated debdistget to only store Git-LFS stubs on GitLab.com and push the real files to S3 object storage. The cost for this is barely measurable, I have yet to run into the 25/month warning threshold.
But how do you backup stuff stored in S3?
For some time, my S3 backup solution has been to run the minio-client mirror command to download all S3 objects to my laptop, and rely on rsnapshot to keep backups of this. While 4TB NVME s are relatively cheap, I ve felt that this disk and network churn on my laptop is unsatisfactory for quite some time.
What is a better approach?
I find S3 hosting sites fairly unreliable by design. Only a couple of clicks in your web browser and you have dropped 100TB of data. Or by someone else who steal your plaintext-equivalent cookie. Thus, I haven t really felt comfortable using any S3-based backup option. I prefer to self-host, although continously running a mirror job is not sufficient: if I accidentally drop the entire S3 object store, my mirror run will remove all files locally too.
The rsnapshot approach that allows going back in time and having data on self-managed servers feels superior to me.
What if we could use rsnapshot with a S3 client instead of rsync?
Someone else asked about this several years ago, and the suggestion was to use the fuse-based s3fs which sounded unreliable to me. After some experimentation, working around some hard-coded assumption in the rsnapshot implementation, I came up with a small configuration pattern and a wrapper tool to implement what I desired.
Here is my configuration snippet:
cmd_rsync /backup/s3/s3rsync
rsync_short_args -Q
rsync_long_args --json --remove
lockfile /backup/s3/rsnapshot.pid
snapshot_root /backup/s3
backup s3:://hetzner/debdistget-gnuinos ./debdistget-gnuinos
backup s3:://hetzner/debdistget-tacos ./debdistget-tacos
backup s3:://hetzner/debdistget-diffos ./debdistget-diffos
backup s3:://hetzner/debdistget-pureos ./debdistget-pureos
backup s3:://hetzner/debdistget-kali ./debdistget-kali
backup s3:://hetzner/debdistget-devuan ./debdistget-devuan
backup s3:://hetzner/debdistget-trisquel ./debdistget-trisquel
backup s3:://hetzner/debdistget-debian ./debdistget-debian
The idea is to save a backup of a couple of S3 buckets under /backup/s3/.
I have some scripts that take a complete rsnapshot.conf file and append my per-directory configuration so that this becomes a complete configuration. If you are curious how I roll this, backup-all invokes backup-one appending my rsnapshot.conf template with the snippet above.
The s3rsync wrapper script is the essential hack to convert rsnapshot s rsync parameters into something that talks S3 and the script is as follows:
#!/bin/sh
set -eu
S3ARG=
for ARG in "$@"; do
case $ARG in
s3:://*) S3ARG="$S3ARG "$(echo $ARG sed -e 's,s3:://,,');;
-Q*) ;;
*) S3ARG="$S3ARG $ARG";;
esac
done
echo /backup/s3/mc mirror $S3ARG
exec /backup/s3/mc mirror $S3ARG
It uses the minio-client tool. I first tried s3cmd but its sync command read all files to compute MD5 checksums every time you invoke it, which is very slow. The mc mirror command is blazingly fast since it only compare mtime s, just like rsync or git.
First you need to store credentials for your S3 bucket. These are stored in plaintext in ~/.mc/config.json which I find to be sloppy security practices, but I don t know of any better way to do this. Replace AKEY and SKEY with your access token and secret token from your S3 provider:
/backup/s3/mc alias set hetzner AKEY SKEY
If I invoke a sync job for a fully synced up directory the output looks like this:
root@hamster /backup# /run/current-system/profile/bin/rsnapshot -c /backup/s3/rsnapshot.conf -V sync
Setting locale to POSIX "C"
echo 1443 > /backup/s3/rsnapshot.pid
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-gnuinos \
/backup/s3/.sync//debdistget-gnuinos
/backup/s3/mc mirror --json --remove hetzner/debdistget-gnuinos /backup/s3/.sync//debdistget-gnuinos
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-tacos \
/backup/s3/.sync//debdistget-tacos
/backup/s3/mc mirror --json --remove hetzner/debdistget-tacos /backup/s3/.sync//debdistget-tacos
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-diffos \
/backup/s3/.sync//debdistget-diffos
/backup/s3/mc mirror --json --remove hetzner/debdistget-diffos /backup/s3/.sync//debdistget-diffos
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-pureos \
/backup/s3/.sync//debdistget-pureos
/backup/s3/mc mirror --json --remove hetzner/debdistget-pureos /backup/s3/.sync//debdistget-pureos
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-kali \
/backup/s3/.sync//debdistget-kali
/backup/s3/mc mirror --json --remove hetzner/debdistget-kali /backup/s3/.sync//debdistget-kali
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-devuan \
/backup/s3/.sync//debdistget-devuan
/backup/s3/mc mirror --json --remove hetzner/debdistget-devuan /backup/s3/.sync//debdistget-devuan
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-trisquel \
/backup/s3/.sync//debdistget-trisquel
/backup/s3/mc mirror --json --remove hetzner/debdistget-trisquel /backup/s3/.sync//debdistget-trisquel
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-debian \
/backup/s3/.sync//debdistget-debian
/backup/s3/mc mirror --json --remove hetzner/debdistget-debian /backup/s3/.sync//debdistget-debian
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
touch /backup/s3/.sync/
rm -f /backup/s3/rsnapshot.pid
/run/current-system/profile/bin/logger -p user.info -t rsnapshot[1443] \
/run/current-system/profile/bin/rsnapshot -c /backup/s3/rsnapshot.conf \
-V sync: completed successfully
root@hamster /backup#
You can tell from the paths that this machine runs Guix. This was the first production use of the Guix System for me, and the machine has been running since 2015 (with the occasional new hard drive). Before, I used rsnapshot on Debian, but some stable release of Debian dropped the rsnapshot package, paving the way for me to test Guix in production on a non-Internet exposed machine. Unfortunately, mc is not packaged in Guix, so you will have to install it from the MinIO Client GitHub page manually.
Running the daily rotation looks like this:
root@hamster /backup# /run/current-system/profile/bin/rsnapshot -c /backup/s3/rsnapshot.conf -V daily
Setting locale to POSIX "C"
echo 1549 > /backup/s3/rsnapshot.pid
mv /backup/s3/daily.5/ /backup/s3/daily.6/
mv /backup/s3/daily.4/ /backup/s3/daily.5/
mv /backup/s3/daily.3/ /backup/s3/daily.4/
mv /backup/s3/daily.2/ /backup/s3/daily.3/
mv /backup/s3/daily.1/ /backup/s3/daily.2/
mv /backup/s3/daily.0/ /backup/s3/daily.1/
/run/current-system/profile/bin/cp -al /backup/s3/.sync /backup/s3/daily.0
rm -f /backup/s3/rsnapshot.pid
/run/current-system/profile/bin/logger -p user.info -t rsnapshot[1549] \
/run/current-system/profile/bin/rsnapshot -c /backup/s3/rsnapshot.conf \
-V daily: completed successfully
root@hamster /backup#
Hopefully you will feel inspired to take backups of your S3 buckets now!
Next.