smartd properly.A decent approach to data safety is to run regularly scheduled short and long
SMART tests on all disks to detect potential damage. Running such tests on all
disks at once isn't ideal, so I set up a script to create a staggered
configuration and test multiple groups of disks at different times. Note that it
is mandatory to read the devices at each reboot because their names and order
can change.Of course, the same principle (short/long test at regular intervals along the
week) should be applied for a simpler configuration, as in the case of my home
NAS with a pair of RAID1 devices.What follows is a simple script to create a staggered smartd.conf at boot
time:#!/bin/bash
#
# Save this as /usr/local/bin/create-smartd-conf.sh
#
# Dynamically generate smartd.conf with staggered SMART test scheduling
# at boot time based on discovered ATA devices
# HERE IS A LIST OF DIRECTIVES FOR THIS CONFIGURATION FILE.
# PLEASE SEE THE smartd.conf MAN PAGE FOR DETAILS
#
# -d TYPE Set the device type: ata, scsi[+TYPE], nvme[,NSID],
# sat[,auto][,N][+TYPE], usbcypress[,X], usbjmicron[,p][,x][,N],
# usbprolific, usbsunplus, sntasmedia, sntjmicron[,NSID], sntrealtek,
# ... (platform specific)
# -T TYPE Set the tolerance to one of: normal, permissive
# -o VAL Enable/disable automatic offline tests (on/off)
# -S VAL Enable/disable attribute autosave (on/off)
# -n MODE No check if: never, sleep[,N][,q], standby[,N][,q], idle[,N][,q]
# -H Monitor SMART Health Status, report if failed
# -s REG Do Self-Test at time(s) given by regular expression REG
# -l TYPE Monitor SMART log or self-test status:
# error, selftest, xerror, offlinests[,ns], selfteststs[,ns]
# -l scterc,R,W Set SCT Error Recovery Control
# -e Change device setting: aam,[N off], apm,[N off], dsn,[on off],
# lookahead,[on off], security-freeze, standby,[N off], wcache,[on off]
# -f Monitor 'Usage' Attributes, report failures
# -m ADD Send email warning to address ADD
# -M TYPE Modify email warning behavior (see man page)
# -p Report changes in 'Prefailure' Attributes
# -u Report changes in 'Usage' Attributes
# -t Equivalent to -p and -u Directives
# -r ID Also report Raw values of Attribute ID with -p, -u or -t
# -R ID Track changes in Attribute ID Raw value with -p, -u or -t
# -i ID Ignore Attribute ID for -f Directive
# -I ID Ignore Attribute ID for -p, -u or -t Directive
# -C ID[+] Monitor [increases of] Current Pending Sectors in Attribute ID
# -U ID[+] Monitor [increases of] Offline Uncorrectable Sectors in Attribute ID
# -W D,I,C Monitor Temperature D)ifference, I)nformal limit, C)ritical limit
# -v N,ST Modifies labeling of Attribute N (see man page)
# -P TYPE Drive-specific presets: use, ignore, show, showall
# -a Default: -H -f -t -l error -l selftest -l selfteststs -C 197 -U 198
# -F TYPE Use firmware bug workaround:
# none, nologdir, samsung, samsung2, samsung3, xerrorlba
# -c i=N Set interval between disk checks to N seconds
# # Comment: text after a hash sign is ignored
# \ Line continuation character
# Attribute ID is a decimal integer 1 <= ID <= 255
# except for -C and -U, where ID = 0 turns them off.
set -euo pipefail
# Test schedule configuration
BASE_SCHEDULE="L/../../6" # Long test on Saturdays
TEST_HOURS=(01 03 05 07) # 4 time slots: 1am, 3am, 5am, 7am
DEVICES_PER_GROUP=3
main()
# Get array of device names (e.g., sda, sdb, sdc)
mapfile -t devices < <(ls -l /dev/disk/by-id/ grep ata awk ' print $11 ' grep sd cut -d/ -f3 sort -u)
if [[ $ #devices[@] -eq 0 ]]; then
exit 1
fi
# Start building config file
cat << EOF
# smartd.conf - Auto-generated at boot
# Generated: $(date '+%Y-%m-%d %H:%M:%S')
#
# Staggered SMART test scheduling to avoid concurrent disk load
# Long tests run on Saturdays at different times per group
#
EOF
# Process devices into groups
local group=0
local count_in_group=0
for i in "$ !devices[@] "; do
local dev="$ devices[$i] "
local hour="$ TEST_HOURS[$group] "
# Add group header at start of each group
if [[ $count_in_group -eq 0 ]]; then
echo ""
echo "# Group $((group + 1)) - Tests at $ hour :00 on Saturdays"
fi
# Add device entry
#echo "/dev/$ dev -a -o on -S on -s ($ BASE_SCHEDULE /$ hour ) -m root"
echo "/dev/$ dev -a -o on -S on -s (L/../../6/$ hour ) -s (S/../.././$(((hour + 12) % 24))) -m root"
# Move to next group when current group is full
count_in_group=$((count_in_group + 1))
if [[ $count_in_group -ge $DEVICES_PER_GROUP ]]; then
count_in_group=0
group=$(((group + 1) % $ #TEST_HOURS[@] ))
fi
done
main "$@"To run such a script at boot, add a unit file to the systemd configuration.sudo systemctl edit --full /etc/systemd/system/regenerate-smartd-conf.service
sudo systemctl enable regenerate-smartd-conf.serviceWhere the unit service is the following:[Unit]
Description=Generate smartd.conf with staggered SMART test scheduling
# Wait for all local filesystems and udev device detection
After=local-fs.target systemd-udev-settle.service
Before=smartd.service
Wants=systemd-udev-settle.service
DefaultDependencies=no
[Service]
Type=oneshot
# Only generate the config file, don't touch smartd here
ExecStart=/bin/bash -c '/usr/local/bin/create-smartd-config.sh > /etc/smartd.conf'
StandardOutput=journal
StandardError=journal
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target

Note 9

Power Use
In 93 minutes having the PinePhone Pro, Librem 5, and FLX1s online with open ssh sessions from my workstation the PinePhone Pro went from 100% battery to 26%, the Librem 5 went from 95% to 69%, and the FLX1s went from 100% to 99%. The battery discharge rate of them was reported as 3.0W, 2.6W, and 0.39W respectively. Based on having a 16.7Wh battery 93 minutes of use should have been close to 4% battery use, but in any case all measurements make it clear that the FLX1s will have a much longer battery life. Including the measurement of just putting my fingers on the phones and feeling the temperature (FLX1s felt cool and the others felt hot).
The PinePhone Pro and the Librem 5 have an optional Caffeine mode which I enabled for this test, without that enabled the phone goes into a sleep state and disconnects from Wifi. So those phones would use much less power with caffeine mode enabled, but they also couldn t get fast response to notifications etc. I found the option to enable a Caffeine mode switch on the FLX1s but the power use was reported as being the same both with and without it.
Charging
One problem I found with my phone is that in every case it takes 22 seconds to negotiate power. Even when using straight USB charging (no BC or PD) it doesn t draw any current for 22 seconds. When I connect it it will stay at 5V and varying between 0W and 0.1W (current rounded off to zero) for 22 seconds or so and then start charging. After the 22 second display the phone will make the tick sound indicating that it s charging and the power meter will measure that it s drawing some current.
I added the table from my previous post about phone charging speed [6] with an extra row for the FLX1s. For charging from my PC USB ports the results were the worst ever, the port that does BC did not work at all it was looping trying to negotiate after a 22 second negotiation delay the port would turn off. The non-BC port gave only 2.4W which matches the 2.5W given by the spec for a High-power device which is what that port is designed to give. In a discussion on the Purism forum about the Librem5 charging speed one of their engineers told me that the reason why their phone would draw 2A from that port was because the cable was identifying itself as a USB-C port not a High-power device port. But for some reason out of the 7 phones I tested the FLX1s and the One Plus 6 are the only ones to limit themselves to what the port is apparently supposed to do. Also the One Plus 6 charges slowly on every power supply so I don t know if it is obeying the spec or just sucking.
On a cheap AliExpress charger the FLX1s gets 5.9V and on a USB battery it gets 5.8V. Out of all 42 combinations of device and charger I tested these were the only ones to involve more than 5.1V but less than 9V. I welcome comments suggesting an explanation.
The case that I received has a hole for the USB-C connector that isn t wide enough for the plastic surrounds on most of my USB-C cables (including the Dell dock). Also to make a connection requires a fairly deep insertion (deeper than the One Plus 6 or the Note 9). So without adjustment I have to take the case off to charge it. It s no big deal to adjust the hole (I have done it with other cases) but it s an annoyance.
| Phone | Top z640 | Bottom Z640 | Monitor | Ali Charger | Dell Dock | Battery | Best | Worst |
|---|---|---|---|---|---|---|---|---|
| FLX1s | FAIL | 5.0V 0.49A 2.4W | 4.8V 1.9A 9.0W | 5.9V 1.8A 11W | 4.8V 2.1A 10W | 5.8V 2.1A 12W | 5.8V 2.1A 12W | 5.0V 0.49A 2.4W |
| Note9 | 4.8V 1.0A 5.2W | 4.8V 1.6A 7.5W | 4.9V 2.0A 9.5W | 5.1V 1.9A 9.7W | 4.8V 2.1A 10W | 5.1V 2.1A 10W | 5.1V 2.1A 10W | 4.8V 1.0A 5.2W |
| Pixel 7 pro | 4.9V 0.80A 4.2W | 4.8V 1.2A 5.9W | 9.1V 1.3A 12W | 9.1V 1.2A 11W | 4.9V 1.8A 8.7W | 9.0V 1.3A 12W | 9.1V 1.3A 12W | 4.9V 0.80A 4.2W |
| Pixel 8 | 4.7V 1.2A 5.4W | 4.7V 1.5A 7.2W | 8.9V 2.1A 19W | 9.1V 2.7A 24W | 4.8V 2.3A 11.0W | 9.1V 2.6A 24W | 9.1V 2.7A 24W | 4.7V 1.2A 5.4W |
| PPP | 4.7V 1.2A 6.0W | 4.8V 1.3A 6.8W | 4.9V 1.4A 6.6W | 5.0V 1.2A 5.8W | 4.9V 1.4A 5.9W | 5.1V 1.2A 6.3W | 4.8V 1.3A 6.8W | 5.0V 1.2A 5.8W |
| Librem 5 | 4.4V 1.5A 6.7W | 4.6V 2.0A 9.2W | 4.8V 2.4A 11.2W | 12V 0.48A 5.8W | 5.0V 0.56A 2.7W | 5.1V 2.0A 10W | 4.8V 2.4A 11.2W | 5.0V 0.56A 2.7W |
| OnePlus6 | 5.0V 0.51A 2.5W | 5.0V 0.50A 2.5W | 5.0V 0.81A 4.0W | 5.0V 0.75A 3.7W | 5.0V 0.77A 3.7W | 5.0V 0.77A 3.9W | 5.0V 0.81A 4.0W | 5.0V 0.50A 2.5W |
| Best | 4.4V 1.5A 6.7W | 4.6V 2.0A 9.2W | 8.9V 2.1A 19W | 9.1V 2.7A 24W | 4.8V 2.3A 11.0W | 9.1V 2.6A 24W |
mrcal-show-projection-uncertainty
--method cross-reprojection-rrp-Jfp is available today, and works in the
usual moving-chessboard-stationary camera case. Fully boardless coming later.
mrcal-show-stereo-pair-diff tool reports an extrinsics+intrinsics
diff between two calibrations of a stereo pair; experimental
analyses/extrinsics-stability.py tool reports an extrinsics-only diff. These
are in contrast to the intrinsics-only uncertainty and diffs in the existing
mrcal-show-projection-diff and mrcal-show-projection-uncertainty tools.
Some documentation in the uncertainty and differencing pages.
_mrcal_triangulated_error(). This is demoed in
test/test-sfm-triangulated-points.py. And I've been using
_mrcal_triangulated_error() in structure-from-motion implementations within
other optimization routines.
/backup/serverA.domain/.sync/foo
/backup/serverA.domain/daily.0/foo
/backup/serverA.domain/daily.1/foo
/backup/serverA.domain/daily.2/foo
...
/backup/serverA.domain/daily.6/foo
/backup/serverA.domain/weekly.0/foo
/backup/serverA.domain/weekly.1/foo
...
/backup/serverA.domain/monthly.0/foo
/backup/serverA.domain/monthly.1/foo
...
/backup/serverA.domain/yearly.0/foo
I can browse and rescue files easily, going back in time when needed.
The rsnapshot project README explains more, there is a long rsnapshot HOWTO although I usually find the rsnapshot man page the easiest to digest.
I have stored multi-TB Git-LFS data on GitLab.com for some time. The yearly renewal is coming up, and the price for Git-LFS storage on GitLab.com is now excessive (~$10.000/year). I have reworked my work-flow and finally migrated debdistget to only store Git-LFS stubs on GitLab.com and push the real files to S3 object storage. The cost for this is barely measurable, I have yet to run into the 25/month warning threshold.
But how do you backup stuff stored in S3?
For some time, my S3 backup solution has been to run the minio-client mirror command to download all S3 objects to my laptop, and rely on rsnapshot to keep backups of this. While 4TB NVME s are relatively cheap, I ve felt that this disk and network churn on my laptop is unsatisfactory for quite some time.
What is a better approach?
I find S3 hosting sites fairly unreliable by design. Only a couple of clicks in your web browser and you have dropped 100TB of data. Or by someone else who steal your plaintext-equivalent cookie. Thus, I haven t really felt comfortable using any S3-based backup option. I prefer to self-host, although continously running a mirror job is not sufficient: if I accidentally drop the entire S3 object store, my mirror run will remove all files locally too.
The rsnapshot approach that allows going back in time and having data on self-managed servers feels superior to me.
What if we could use rsnapshot with a S3 client instead of rsync?
Someone else asked about this several years ago, and the suggestion was to use the fuse-based s3fs which sounded unreliable to me. After some experimentation, working around some hard-coded assumption in the rsnapshot implementation, I came up with a small configuration pattern and a wrapper tool to implement what I desired.
Here is my configuration snippet:
cmd_rsync /backup/s3/s3rsync
rsync_short_args -Q
rsync_long_args --json --remove
lockfile /backup/s3/rsnapshot.pid
snapshot_root /backup/s3
backup s3:://hetzner/debdistget-gnuinos ./debdistget-gnuinos
backup s3:://hetzner/debdistget-tacos ./debdistget-tacos
backup s3:://hetzner/debdistget-diffos ./debdistget-diffos
backup s3:://hetzner/debdistget-pureos ./debdistget-pureos
backup s3:://hetzner/debdistget-kali ./debdistget-kali
backup s3:://hetzner/debdistget-devuan ./debdistget-devuan
backup s3:://hetzner/debdistget-trisquel ./debdistget-trisquel
backup s3:://hetzner/debdistget-debian ./debdistget-debian
The idea is to save a backup of a couple of S3 buckets under /backup/s3/.
I have some scripts that take a complete rsnapshot.conf file and append my per-directory configuration so that this becomes a complete configuration. If you are curious how I roll this, backup-all invokes backup-one appending my rsnapshot.conf template with the snippet above.
The s3rsync wrapper script is the essential hack to convert rsnapshot s rsync parameters into something that talks S3 and the script is as follows:
#!/bin/sh
set -eu
S3ARG=
for ARG in "$@"; do
case $ARG in
s3:://*) S3ARG="$S3ARG "$(echo $ARG sed -e 's,s3:://,,');;
-Q*) ;;
*) S3ARG="$S3ARG $ARG";;
esac
done
echo /backup/s3/mc mirror $S3ARG
exec /backup/s3/mc mirror $S3ARG
It uses the minio-client tool. I first tried s3cmd but its sync command read all files to compute MD5 checksums every time you invoke it, which is very slow. The mc mirror command is blazingly fast since it only compare mtime s, just like rsync or git.
First you need to store credentials for your S3 bucket. These are stored in plaintext in ~/.mc/config.json which I find to be sloppy security practices, but I don t know of any better way to do this. Replace AKEY and SKEY with your access token and secret token from your S3 provider:
/backup/s3/mc alias set hetzner AKEY SKEY
If I invoke a sync job for a fully synced up directory the output looks like this:
root@hamster /backup# /run/current-system/profile/bin/rsnapshot -c /backup/s3/rsnapshot.conf -V sync
Setting locale to POSIX "C"
echo 1443 > /backup/s3/rsnapshot.pid
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-gnuinos \
/backup/s3/.sync//debdistget-gnuinos
/backup/s3/mc mirror --json --remove hetzner/debdistget-gnuinos /backup/s3/.sync//debdistget-gnuinos
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-tacos \
/backup/s3/.sync//debdistget-tacos
/backup/s3/mc mirror --json --remove hetzner/debdistget-tacos /backup/s3/.sync//debdistget-tacos
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-diffos \
/backup/s3/.sync//debdistget-diffos
/backup/s3/mc mirror --json --remove hetzner/debdistget-diffos /backup/s3/.sync//debdistget-diffos
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-pureos \
/backup/s3/.sync//debdistget-pureos
/backup/s3/mc mirror --json --remove hetzner/debdistget-pureos /backup/s3/.sync//debdistget-pureos
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-kali \
/backup/s3/.sync//debdistget-kali
/backup/s3/mc mirror --json --remove hetzner/debdistget-kali /backup/s3/.sync//debdistget-kali
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-devuan \
/backup/s3/.sync//debdistget-devuan
/backup/s3/mc mirror --json --remove hetzner/debdistget-devuan /backup/s3/.sync//debdistget-devuan
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-trisquel \
/backup/s3/.sync//debdistget-trisquel
/backup/s3/mc mirror --json --remove hetzner/debdistget-trisquel /backup/s3/.sync//debdistget-trisquel
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
/backup/s3/s3rsync -Qv --json --remove s3:://hetzner/debdistget-debian \
/backup/s3/.sync//debdistget-debian
/backup/s3/mc mirror --json --remove hetzner/debdistget-debian /backup/s3/.sync//debdistget-debian
"status":"success","total":0,"transferred":0,"duration":0,"speed":0
touch /backup/s3/.sync/
rm -f /backup/s3/rsnapshot.pid
/run/current-system/profile/bin/logger -p user.info -t rsnapshot[1443] \
/run/current-system/profile/bin/rsnapshot -c /backup/s3/rsnapshot.conf \
-V sync: completed successfully
root@hamster /backup#
You can tell from the paths that this machine runs Guix. This was the first production use of the Guix System for me, and the machine has been running since 2015 (with the occasional new hard drive). Before, I used rsnapshot on Debian, but some stable release of Debian dropped the rsnapshot package, paving the way for me to test Guix in production on a non-Internet exposed machine. Unfortunately, mc is not packaged in Guix, so you will have to install it from the MinIO Client GitHub page manually.
Running the daily rotation looks like this:
root@hamster /backup# /run/current-system/profile/bin/rsnapshot -c /backup/s3/rsnapshot.conf -V daily
Setting locale to POSIX "C"
echo 1549 > /backup/s3/rsnapshot.pid
mv /backup/s3/daily.5/ /backup/s3/daily.6/
mv /backup/s3/daily.4/ /backup/s3/daily.5/
mv /backup/s3/daily.3/ /backup/s3/daily.4/
mv /backup/s3/daily.2/ /backup/s3/daily.3/
mv /backup/s3/daily.1/ /backup/s3/daily.2/
mv /backup/s3/daily.0/ /backup/s3/daily.1/
/run/current-system/profile/bin/cp -al /backup/s3/.sync /backup/s3/daily.0
rm -f /backup/s3/rsnapshot.pid
/run/current-system/profile/bin/logger -p user.info -t rsnapshot[1549] \
/run/current-system/profile/bin/rsnapshot -c /backup/s3/rsnapshot.conf \
-V daily: completed successfully
root@hamster /backup#
Hopefully you will feel inspired to take backups of your S3 buckets now!
No man s Sky (or as it s known in our house, "spaceship game") is a space
exploration/sandbox game that was originally released 10 years ago. Back then I
tried it on my brother s PS4 but I couldn t get into it. In 2022 it launched
for the Nintendo Switch1 and the game finally clicked for me.
I play it very casually. I mostly don t play at all, except sometimes when
there are time-limited expeditions running, which I find refreshing, and
usually have some exclusives as a reward for play.
One of the many things you can do in the game is collect star ships. I started
keeping a list of notable ones I ve found, and I ve decided to occasionally
blog about them.
The Horizon Vector NX is a small sporty ship that players on Nintendo Switch
could claim within the first month or so after it launched. The colour scheme
resembles the original neon switch controllers. Although the ship type occurs
naturally in the game in other configurations, I think differently-painted
wings are unique to this ship.
For most of the last 4 years, my copy of this ship was confined to the Switch,
until November 2024, when they added cross-save capability to the game. I was
then able to access the ship when playing on Linux (or Mac).
Brunei entry stamp on my passport. Picture by Ravi Dwivedi, released under CC-BY-SA 4.0.
Our room in Brunei. Picture by Ravi Dwivedi, released under CC-BY-SA 4.0
Jame Asr Hassanil Bolkiah Mosque. Picture by Ravi Dwivedi, released under CC-BY-SA 4.0
Omar Ali Saifuddien Mosque. Picture by Ravi Dwivedi, released under CC-BY-SA 4.0
A shot of Brunei s countryside. Picture by Ravi Dwivedi, released under CC-BY-SA 4.0.
Via (I think) @mcc on the Fediverse, I
learned of GetMusic: a sort-of "clearing house"
for Free Bandcamp codes. I think the way it works is, some artists
release a limited set of download codes for their albums in order to
promote them, and GetMusic help them to keep track of that, and helps
listeners to discover them.
GetMusic mail me occasionally, and once they highlighted an album The Arcane &
Paranormal
Earth which
they described as "Post-Industrial in the vein of Coil and Nurse With Wound
with shades of Aphex Twin, Autechre and assorted film music."
Well that description hooked me immediately but I missed out on the code.
However, I sampled the album on Bandcamp directly a few times as well as
a few of his others (Ye Gods is a side-project of Antoni Maiovvi, which
itself is a pen-name) and liked them very much. I picked up the full
collection of Ye Gods albums in one go for 30% off.
Here's a stand-out track:
On Earth by Ye Gods
So I guess this service works! Although I didn't actually get a free code
in this instance, it promoted the artist, introduced me to something I really
liked and drove a sale.
Version 0.0.25 of RcppSpdlog arrived
on CRAN right now, and will be
uploaded to Debian and built for r2u shortly along with a
minimal refresh of the documentation site. RcppSpdlog
bundles spdlog, a
wonderful header-only C++ logging library with all the bells and
whistles you would want that was written by Gabi Melman, and also includes fmt by Victor Zverovich. You can learn
more at the nice package
documention site.
This release fixes a minuscule cosmetic issue from the previous
release a week ago. We rely on two #defines that R sets to
signal to spdlog that we
are building in the R context (which matters for the R-specific logging
sink, and picks up something Gabi added upon my suggestion at
the very start of this package). But I use the same
#defines to now check in Rcpp that we are building with R and, in
this case, wrongly conclude R headers have already been installed so Rcpp (incorrectly) nags about that. The
solution is to add two #undefine and proceed as normal
(with Rcpp controlling and taking
care of R header includion too) and that is what we do here. All good
now, no nags from a false positive.
The NEWS entry for this release follows.
Courtesy of my CRANberries, there is also a diffstat report detailing changes. More detailed information is on the RcppSpdlog page, or the package documention site.Changes in RcppSpdlog version 0.0.25 (2026-01-15)
- Ensure
#definesignaling R build (needed with spdlog) is unset before including R headers to not falsely triggering message from Rcpp
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.
A brand new release 0.1.15 of the RcppSimdJson
package is now on CRAN.
RcppSimdJson
wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire and collaborators. Via
very clever algorithmic engineering to obtain largely branch-free code,
coupled with modern C++ and newer compiler instructions, it results in
parsing gigabytes of JSON parsed per second which is quite
mindboggling. The best-case performance is faster than CPU speed as
use of parallel SIMD instructions and careful branch avoidance can lead
to less than one cpu cycle per byte parsed; see the video of the talk by Daniel Lemire
at QCon.
This version updates to the current 4.2.4 upstream release. It also
updates the RcppExports.cpp file with glue between C++
and R. We want move away from using Rf_error() (as
Rcpp::stop() is generally preferable). Packages (such as
this one) that are declaring an interface have an actual
Rf_error() call generated in RcppExports.cpp
which can protect which is what current Rcpp code
generation does. Long story short, a minor internal reason.
The short NEWS entry for this release follows.
Courtesy of my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo.Changes in version 0.1.15 (2026-01-14)
- simdjson was upgraded to version 4.2.4 (Dirk in #97
RcppExports.cppwas regenerated to aid a Rcpp transition- Standard maintenance updates for continuous integration and URLs
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.
dh-python to fix bugs and add features. This started
a month of attention
on dh-python, iterating through several bug fixes, and a couple of unfortunate
regressions.
dh-python is used by almost all packages containing Python (over 5000). Most
of these are very simple, but some are complex and use dh-python in unexpected
ways. It s hard to avoid almost any change (including obvious bug fixes) from
causing some unexpected knock-on behaviour. There is a fair amount of complexity
in dh-python, and some rather clever code, which can make it tricky to work on.
All of this means that good QA is important. Stefano spent some time
adding type annotations
and specialized types to make it easier to see what the code is doing and catch
mistakes. This has already made work on dh-python easier.
Now that Debusine has built-in repositories
and debdiff support, Stefano could quickly test the effects of changes on many
other packages. After each big change, he could upload dh-python to
a repository,
rebuild e.g. 50 Python packages with it, and see what differences appeared in
the output. Reviewing the diffs is still a manual process, but can be improved.
Stefano did a small test
on what it would take to replace direct setuptools setup.py calls with
PEP-517 (pyproject-style) builds. There is
more work to do here.
Python 3.14 as a supported version started in Debian unstable. To do this, we
update the list of supported versions in python3-defaults,
and then start rebuilding modules with C extensions from the leaves inwards.
This had already been tested in a PPA and Ubuntu, so many of the biggest
blocking compatibility issues with 3.14 had already been found and fixed. But
there are always new issues to discover.
Thanks to a number of people in the Debian Python team, we got through the first
bit of the transition fairly quickly. There are still a number of
open bugs
that need attention and many failed tests
blocking migration to testing.
Python 3.14.1 released just after we started the transition, and very soon
after, a follow-up 3.14.2 release came out to address a regression. We ran into
another regression in
Python 3.14.2.
ruby3.4 source package is already
available in experimental, also ruby-defaults added support to
Ruby 3.4. Lucas rebuilt all reverse dependencies against this new version of the
interpreter and published the results here.
Lucas also reached out to some stakeholders to coordinate the work.
Next steps are: 1) announcing the results to the whole team and asking for help
to fix packages failing to build against the new interpreter; 2) file bugs
against packages FTBFSing against Ruby 3.4 which are not fixed yet; 3) once we
have a low number of build failures against Ruby 3.4, ask the Debian Release
team to start the transition in unstable.
fail2ban powered firewall-level access limitation
for clients that display an abusive access pattern. This went through several
iterations, with some of them unfortunately blocking legitimate Debian
contributors, but the current state seems to strike a good balance between
blocking scrapers and not blocking real users. Please get in touch with the team
on the #debci OFTC channel if you are affected by this.
dose-builddebcheck to determine whether a package s dependencies can be
satisfied before attempting a build. About one third of Debian s packages fail
this check, so understanding the reasons is key to improving cross building.
Unfortunately, dose-builddebcheck stops after reporting the first problem and
does not display additional ones.
To address this, a greedy solver implemented in Python now examines each
build-dependency individually and can report multiple causes. dose-builddebcheck
is still used as a fall-back when the greedy solver does not identify any
problems. The report for bazel-bootstrap
is a lengthy example.
debhelper to its final installability test and builds a few more packages
required for installing it. It also now uses a variant of build-essential that
has been marked Multi-Arch: same
(see foundational work
from last year).
This in turn made the use of a non-default GCC version more difficult and
required more work to make it work for gcc-16 from experimental. Ongoing
archive changes temporarily regressed building fribidi and dash.
libselinux and groff have received patches for architecture specific changes
and libverto has been NMUed to remove the glib2.0 dependency.
python-authlib, python-mitogen, xdot.po-debconf-manager: added delete-package command,
show-information now uses properly formatted output (YAML), it now attaches
the translation on the bug reports for which a merge request has been opened too
long.po-debconf-manager but not
in the Debian l10n list.
Turns out that some packages had debian/po/templates.pot (appearing in
po-debconf-manager) but not the POTFILES.in file as expected.
Created a script
to find out which packages were in this or similar situation and
reported bugs.mbrola and festival) if using Orca speech synthesizer. Commented
a few issues and possible improvements in the debian-accessibility list.cmake_pkg_config builtin which is
not aware of the host architecture. He also
forwarded a Meson patch upstream.cups to fix a nasty bug that was
introduced by the latest security update.OpenSSH patch we ve been carrying
since 2008 to fix misleading verbose output from scp.astroid and pygments,
and wrote up the astroid case
on his blog.opencv
for the ffmpeg 8 transition.gnome-shell,
mutter, glib2.0).nocheck and nodoc
for the same git tree. Finally, Santiago provided
MR !685
to fix the documentation.
A recent article in The Atlantic makes the case that
very large language models effectively contain much of the works they're trained on.
This article is an attempt to popularize the insights in the recent academic paper
Extracting books from production language models from Ahmed et al.
The authors of the paper demonstrate convincingly that
well-known copyrighted textual material can be extracted from
the chatbot interfaces of popular commercial LLM services.
The Atlantic article cites a podcast quote about the Stable Diffusion AI image-generator model, saying "We took 100,000 gigabytes of images and compressed it to a two-gigabyte file that can re-create any of those and iterations of those".
By analogy, this suggests we might think of LLMs (which work on text, not the images handled by Stable Diffusion) as a form of lossy textual compression.
The entire text of Moby Dick, the canonical Big American Novel is merely 1.2MiB uncompressed (and less than 0.4MiB losslessly compressed with bzip2 -9).
It's not surprising to imagine that a model
with hundreds of billions of parameters might contain copies of these works.
Warning: The next paragraph contains fuzzy math with no real concrete engineering practice behind it!
Consider a hypothetical model with 100 billion parameters,
where each parameter is stored as a 16-bit floating point value.
The model weights would take 200 GB of storage.
If you were to fill the parameter space only with losslessly compressed copies of books like Moby Dick,
you could still fit half a million books, more than anyone can read in a lifetime.
And lossy compression is typically orders of magnitude less in size than lossless compression,
so we're talking about millions of works effectively encoded, with the acceptance of some
artifacts being injected in the output.
I first encountered this "compression" view of AI nearly three years ago, in Ted Chiang's insightful
ChatGPT is a Blurry JPEG of the Web.
I was suprised that The Atlantic article didn't cite Chiang's piece.
If you haven't read Ted Chiang, i strongly recommend his work,
and this piece is a great place to start.
Chiang aside, the more recent writing that focuses on the idea of
compressed works being "contained" in the model weights
seems to be used by people interested in wielding
esome sort of copyright claims against the AI companies that maintain or provide access to these models.
There are many many problems with AI today, but attacking AI companies based on copyright concerns
seems similar to going after Al Capone for tax evasion.
We should be much more concerned with the effect these projects have on
cultural homogeneity,
mental health,
labor rights,
privacy, and
social control
than whether they're violating copyright in some specific instance.
If you care about supporting open source software, and still use MySQL in 2026, you should switch to MariaDB like so many others have already done.
The number of git commits on github.com/mysql/mysql-server has been significantly declining in 2025. The screenshot below shows the state of git commits as of writing this in January 2026, and the picture should be alarming to anyone who cares about software being open source.
Due to newer MySQL versions deprecating many features, a lot of users also complained about significant struggles regarding both MySQL 5.7->8.0 and 8.0->8.4 upgrades. With few new features and heavy focus on code base cleanup and feature deprecation, it became obvious to many that Oracle had decided to just keep MySQL barely alive, and put all new relevant features (e.g. vector search) into Heatwave, Oracle s closed-source and cloud-only service for MySQL customers.
As it was evident that Oracle isn t investing in MySQL, Percona s Peter Zaitsev wrote Is Oracle Finally Killing MySQL in June 2024. At this time MySQL s popularity as ranked by DB-Engines had also started to tank hard, a trend that likely accelerates in 2026.
In September 2025 news reported that Oracle was reducing its workforce and that the MySQL staff was getting heavily reduced. Obviously this does not bode well for MySQL s future, and Peter Zaitsev posted already in November stats showing that the latest MySQL maintenance release contained fewer bug fixes than before.
apt/dnf/brew install mariadb-server.
Whatever you end up choosing, as long as it is not Oracle, you will be better off.
Team Rcpp is thrilled to share that an exciting new version 1.1.1 of
Rcpp is now on CRAN (and also
uploaded to Debian and already
built for r2u).
Having switchted to C++11 as the minimum standard in the
previous 1.1.0 release, this version takes full advantage of it and
removes a lot of conditional code catering to older standards
that no longer need to be supported. Consequently, the source tarball
shrinks by 39% from 3.11 mb to 1.88 mb. That is a big deal. (Size peaked
with Rcpp 1.0.12 two years ago at 3.43 mb; relative to its size we are
down 45% !!) Removing unused code also makes maintenance easier, and
quickens both compilation and installation in general.
This release continues as usual with the six-months January-July
cycle started with release
1.0.5 in July 2020. Interim snapshots are always available via the
r-universe page and
repo. We continue to strongly encourage the use of these development
released and their testing we tend to run our systems with them too.
Rcpp has long established itself
as the most popular way of enhancing R with C or C++ code. Right now,
3020 packages on CRAN depend on
Rcpp for making analytical code go
faster and further. On CRAN, 13.1% of all packages depend (directly) on
Rcpp, and 60.9% of all compiled
packages do. From the cloud mirror of CRAN (which is but a subset of all
CRAN downloads), Rcpp has been
downloaded 109.8 million times. The two published papers (also included
in the package as preprint vignettes) have, respectively, 2151 (JSS, 2011) and 405 (TAS, 2018)
citations, while the the book (Springer useR!,
2013) has another 715.
This time, I am not attempting to summarize the different changes.
The full list follows below and details all these changes, their
respective PRs and, if applicable, issue tickets. Big thanks from all of
us to all contributors!
Thanks to my CRANberries, you can also look at a diff to the previous interim release along with pre-releaseds 1.1.0.8, 1.1.0.8.1 and 1.1.0.8.2 that were needed because R-devel all of a sudden decided to move fast and break things. Not our doing. Questions, comments etc should go to the GitHub discussion section list]rcppdevellist off the R-Forge page. Bugs reports are welcome at the GitHub issue tracker as well. Both sections can be searched as well.Changes in Rcpp release version 1.1.1 (2026-01-08)
- Changes in Rcpp API:
- An unused old R function for a compiler version check has been removed after checking no known package uses it (Dirk in #1395)
- A narrowing warning is avoided via a cast (Dirk in #1398)
- Demangling checks have been simplified (I aki in #1401 addressing #1400)
- The treatment of signed zeros is now improved in the Sugar code (I aki in #1404)
- Preparations for phasing out use of
Rf_errorhave been made (I aki in #1407)- The long-deprecated function
loadRcppModules()has been removed (Dirk in #1416 closing #1415)- Some non-API includes from R were refactored to accommodate R-devel changes (I aki in #1418 addressing #1417)
- An accessor to
Rf_rnbetahas been removed (Dirk in #1419 also addressing #1420)- Code accessing non-API
Rf_findVarInFramenow usesR_getVarEx(Dirk in #1423 fixing #1421)- Code conditional on the R version now expects at least R 3.5.0; older code has been removed (Dirk in #1426 fixing #1425)
- The non-API
ATTRIBentry point to the R API is no longer used (Dirk in #1430 addressing #1429)- The unwind-protect mechanism is now used unconditionally (Dirk in #1437 closing #1436)
- Changes in Rcpp Attributes:
- The OpenMP plugin has been generalized for different macOS compiler installations (Kevin in #1414)
- Changes in Rcpp Documentation:
- Vignettes are now processed via a new "asis" processor adopted from R.rsp (Dirk in #1394 fixing #1393)
- R is now cited via its DOI (Dirk)
- A (very) stale help page has been removed (Dirk in #1428 fixing #1427)
- The main README.md was updated emphasizing r-universe in favor of the local drat repos (Dirk in #1431)
- Changes in Rcpp Deployment:
- A temporary change in R-devel concerning NA part in complex variables was accommodated, and then reverted (Dirk in #1399 fixing #1397)
- The macOS CI runners now use macos-14 (Dirk in #1405)
- A message is shown if
R.his included before Rcpp headers as this can lead to errors (Dirk in #1411 closing #1410)- Old helper functions use
message()to signal they are not used, deprecation and removal to follow (Dirk in #1413 closing #1412)- Three tests were being silenced following #1413 (Dirk in #1422)
- The heuristic whether to run all available tests was refined (Dirk in #1434 addressing #1433)
- Coverage has been tweaked via additional
#nocovtags (Dirk in #1435)- Non-release Changes:
- Two interim non-releases 1.1.0.8.1 and .2 were made in order to unblock CRAN due to changes in R-devel rather than Rcpp
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.
Welcome to the December 2025 from the Reproducible Builds project!
Our monthly reports outline what we ve been up to over the past month, highlighting items of news from elsewhere in the increasingly-important area of software supply-chain security. As ever, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website.
In Arch Linux this month, Robin Candau and Mark Hegreberg worked at making the Arch Linux WSL image bit-for-bit reproducible. Robin also shared some implementation details and future related work on our mailing list.
Continuing a series reported in these reports for March, April and July 2025 (etc.), Simon Josefsson has published another interesting article this month, itself a followup to a post Simon published in December 2024 regarding GNU Guix Container Images that are hosted on GitLab.
In Debian this month, Micha Lenk posted to the debian-backports-announce mailing list with the news that the Backports archive will now discard binaries generated and uploaded by maintainers: The benefit is that all binary packages [will] get built by the Debian buildds before we distribute them within the archive.
Felix Moessbauer of Siemens then filed a bug in the Debian bug tracker to signal their intention to package debsbom, a software bill of materials (SBOM) generator for distributions based on Debian. This generated a discussion on the bug inquiring about the output format as well as a question about how these SBOMs might be distributed.
Holger Levsen merged a number of significant changes written by Alper Nebi Yasak to the Debian Installer in order to improve its reproducibility. As noted in Alper s merge request, These are the reproducibility fixes I looked into before bookworm release, but was a bit afraid to send as it s just before the release, because the things like the xorriso conversion changes the content of the files to try to make them reproducible.
In addition, 76 reviews of Debian packages were added, 8 were updated and 27 were removed this month adding to our knowledge about identified issues. A new different_package_content_when_built_with_nocheck issue type was added by Holger Levsen. [ ]
Arnout Engelen posted to our mailing list reporting that they successfully reproduced the NixOS minimal installation ISO for the 25.11 release without relying on a pre-compiled package archive, with more details on their blog.
Lastly, Bernhard M. Wiedemann posted another openSUSE monthly update for his work there.
stampdalf allows you to run any command that modifies files in a directory tree, then automatically resets all timestamps back to their original values. Any new files created during command execution are set to [the UNIX epoch] or a custom timestamp via SOURCE_DATE_EPOCH.
The project s GitHub page helpfully reveals that the project is pronounced: stamp-dalf (stamp like time-stamp, dalf like Gandalf the wizard) as it s a wizard of time and stamps .)
Yogya Gamage and Benoit Baudry of Universit de Montr al, Canada together with Deepika Tiwari and Martin Monperrus of KTH Royal Institute of Technology, Sweden published a paper on The Design Space of Lockfiles Across Package Managers:
Most package managers also generate a lockfile, which records the exact set of resolved dependency versions. Lockfiles are used to reduce build times; to verify the integrity of resolved packages; and to support build reproducibility across environments and time. Despite these beneficial features, developers often struggle with their maintenance, usage, and interpretation. In this study, we unveil the major challenges related to lockfiles, such that future researchers and engineers can address them. [ ]A PDF of their paper is available online. Benoit Baudry also posted an announcement to our mailing list, which generated a number of replies.
Betul Gokkaya, Leonardo Aniello and Basel Halak of the University of Southampton then published a paper on the A taxonomy of attacks, mitigations and risk assessment strategies within the software supply chain:
While existing studies primarily focus on software supply chain attacks prevention and detection methods, there is a need for a broad overview of attacks and comprehensive risk assessment for software supply chain security. This study conducts a systematic literature review to fill this gap. By analyzing 96 papers published between 2015-2023, we identified 19 distinct SSC attacks, including 6 novel attacks highlighted in recent studies. Additionally, we developed 25 specific security controls and established a precisely mapped taxonomy that transparently links each control to one or more specific attacks. [ ]A PDF of the paper is available online via the article s canonical page.
Aman Sharma and Martin Monperrus of the KTH Royal Institute of Technology, Sweden along with Benoit Baudry of Universit de Montr al, Canada published a paper this month on Causes and Canonicalization of Unreproducible Builds in Java. The abstract of the paper is as follows:
[Achieving] reproducibility at scale remains difficult, especially in Java, due to a range of non-deterministic factors and caveats in the build process. In this work, we focus on reproducibility in Java-based software, archetypal of enterprise applications. We introduce a conceptual framework for reproducible builds, we analyze a large dataset from Reproducible Central, and we develop a novel taxonomy of six root causes of unreproducibility. [ ]A PDF of the paper is available online.
Once again, there were a number of improvements made to our website this month including:
golang-github-spf13-afero.golang-github-appleboy-easyssh-proxy.circlator.golang-github-jhoonb-archivex.golang-github-jonas-p-go-shp.golang-github-foxboron-go-uefi.in-toto-golang.lua-penlight.rust-fslock.fff.golang-github-notaryproject-notation-go.golang-github-google-go-tpm.golang-github-foxboron-go-tpm-keyfiles.goobook.fortran-regex.golang-github-yudai-gojsondiff.golang-github-tjfoc-gmsm.golang-github-otiai10-copy.golang-k8s-sigs-kustomize-cmd-config.golang-github-artyom-mtab.golang-k8s-sigs-release-utils.golang-github-theupdateframework-go-tuf.php-dompdf.golang-github-viant-toolbox.microbiomeutil.python-openstep-plist.rust-xdg.bibtexparser.plyara.golang-github-valyala-fasthttp.golang-github-issue9-identicon.golang-github-cue-lang-cue.sigstore-go.golang-github-apptainer-sif.golang-github-gin-gonic-gin.rust-rustpython-parser.golang-github-reviewdog-errorformat.geoalchemy2.golang-github-shenwei356-breader.golang-github-ulikunitz-xz.golang-mvdan-editorconfig.golang-github-digitorus-timestamp.golang-forgejo-forgejo-levelqueue.golang-github-kr-binarydist.golang-github-kshedden-dstream.golang-github-google-go-pkcs11.golang-github-akavel-rsrc.golang-github-go-macaron-toolbox.golang-goptlib.golang-github-dreamitgetit-statuscake.golang-github-google-go-attestation.python-pyshortcuts.graudit.golang-github-roaringbitmap-roaring.golang-github-linkedin-goavro.golang-github-cznic-ql.golang-github-muesli-termenv.golang-github-jung-kurt-gofpdf.tdiary.authselect.node-convert-source-map.zope.deferredimport.golang-k8s-apimachinery.kirigami (qml)libplasma (qml)powerdevil (qml)#reproducible-builds on irc.oftc.net.
rb-general@lists.reproducible-builds.org
This post is an unpublished review for Artificial Intelligence Play or break the deckAs a little disclaimer, I usually review books or articles written in English, and although I will offer this review to Computing Reviews as usual, it is likely it will not be published. The title of this book in Spanish is Inteligencia artificial: jugar o romper la baraja. I was pointed at this book, published last October by Margarita Padilla Garc a, a well known Free Software activist from Spain who has long worked on analyzing (and shaping) aspects of socio-technological change. As other books published by Traficantes de sue os, this book is published as Open Access, under a CC BY-NC license, and can be downloaded in full. I started casually looking at this book, with too long a backlog of material to read, but soon realized I could just not put it down: it completely captured me. This book presents several aspects of Artificial Intelligence (AI), written for a general, non-technical audience. Many books with a similar target have been published, but this one is quite unique; first of all, it is written in a personal, non-formal tone. Contrary to what s usual in my reading, the author made the explicit decision not to fill the book with references to her sources ( because searching on Internet, it s very easy to find things ), making the book easier to read linearly a decision I somewhat regret, but recognize helps develop the author s style. The book has seven sections, dealing with different aspects of AI. They are the Visions (historical framing of the development of AI); Spectacular (why do we feel AI to be so disrupting, digging particularly into game engines and search space); Strategies , explaining how multilayer neural networks work and linking the various branches of historic AI together, arriving at Natural Language Processing; On the inside , tackling technical details such as algorithms, the importance of training data, bias, discrimination; On the outside , presenting several example AI implementations with socio-ethical implications; Philosophy , presenting the works of Marx, Heidegger and Simondon in their relation with AI, work, justice, ownership; and Doing , presenting aspects of social activism in relation to AI. Each part ends with yet another personal note: Margarita Padilla includes a letter to one of her friends related to said part. Totalling 272 pages (A5, or roughly half-letter, format), this is a rather small book. I read it probably over a week. So, while this book does not provide lots of new information to me, the way how it was written, made it a very pleasing experience, and it will surely influence the way I understand or explain several concepts in this domain.
Log in for that account is temporary suspended while we perform maintenance. Please try again later.At this point, I suspected that the Riseup service itself was facing some issues. I asked a friend who had an account there if the service was up, and they said that it was. The issue seemed to be specific only to my account. I contacted Riseup support and informed them of the issue. They responded the next day (the 5th of January) saying:
The my-username-redacted account was found inviting another account that violated our terms of use. As a security measure we suspend all related accounts to ToS violations.(Before we continue, I would like to take a moment and reflect upon how nice it was to receive response from a human rather than an AI bot a trend that is unfortunately becoming the norm nowadays.) I didn t know who violated their ToS, so I asked which account violated their terms. Riseup told me:
username-redacted@riseup.net attempted to create aliases that could be abused to impersonate riseup itself.I asked a friend whom I invited a month before the incident, and they confirmed that the username belonged to them. When I asked what they did, they told me they tried creating aliases such as
floatup and risedown. I also asked Riseup which aliases violated their terms, but their support didn t answer this.
I explained to the Riseup support that the impersonation wasn t intentional, that the user hadn t sent any emails, and that I had been a user for more than 5 years and had donated to them in the past.
Furthermore, I suggested that they should block the creation of such aliases if they think the aliases violate their terms, like how email providers typically don t allow users to create admin@ or abuse@ email addresses.
After I explained myself, Riseup reinstated my account.
Update on the 10th of January 2025: My friend told me that the alias that violated Riseup s terms was cloudadmin and his account was reinstated on the 7th of January.
You must understand that we react [by] protecting our service, and therefore we cannot provide notice messages on the affected accounts. We need to act preventing any potential damage to the service that might affect the rest of the users, and that measure is not excessive (think on how abusers/spammers/scammers/etc could trick us and attempt any action before their account is suspended).This didn t address my concerns, so let s move on to the next section.
Next.