Russ Allbery: Review: Ghostdrift
| Series: | Finder Chronicles #4 |
| Publisher: | DAW |
| Copyright: | May 2024 |
| ISBN: | 0-7564-1888-7 |
| Format: | Kindle |
| Pages: | 378 |
| Series: | Finder Chronicles #4 |
| Publisher: | DAW |
| Copyright: | May 2024 |
| ISBN: | 0-7564-1888-7 |
| Format: | Kindle |
| Pages: | 378 |
Armadillo is a powerful
and expressive C++ template library for linear algebra and scientific
computing. It aims towards a good balance between speed and ease of use,
has a syntax deliberately close to Matlab, and is useful for algorithm
development directly in C++, or quick conversion of research code into
production environments. RcppArmadillo
integrates this library with the R environment and language and is
widely used by (currently) 1236 other packages on CRAN, downloaded 39 million times
(per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint
/ vignette) by Conrad and myself has been cited 620 times according
to Google Scholar.
This release brings a small upstream bug fix to the two
FFTW3-interfacing functions, something not likely to hit many CRAN packages.
The changes since the last and fairly recent CRAN release are summarised
below.
Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.Changes in RcppArmadillo version 14.4.1-1 (2025-03-27)
- Upgraded to Armadillo release 14.4.1 (Filtered Espresso)
- Fix for
fft()andifft()when using FFTW3 in multi-threaded contexts (such as OpenMP)
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.
Like each month, have a look at the work funded by Freexian s Debian LTS offering.

A new minor release 0.2.5 of our RcppRedis
package arrived on CRAN today.
RcppRedis
is one of several packages connecting R to the fabulous Redis in-memory datastructure store (and
much more). RcppRedis
does not pretend to be feature complete, but it may do some things
faster than the other interfaces, and also offers an optional coupling
with MessagePack binary
(de)serialization via RcppMsgPack. The
package has been deployed in production as a risk / monitoring tool on
a trading floor for several years. It also supports pub/sub
dissemination of streaming market data as per this
earlier example.
Given the changes around Redis, it is
worth stressing that the package works just as well with valkey and uses only hiredis which remains proper
open source.
This update is again somewhat mechanical as a few maintenance things
bubbled up in since the last release in the summer of 2023. As with
other packages, continuous integration was updated a few times as were
URLs and badges, and we update the use of our RApiSerialize.
And, just as we did today and yesterday with littler,
RQuantLib
and RDieHarder,
this addresses a nag from CRAN
about implicit bash dependency in configure.ac
(which we fixed in January as for the other packages but are under
deadline now). Last but not least the newly-added extension of
forbidden symbols in static libraries revealed that yes indeed I had
forgotten to set -DNDEBUG when building the embedded hiredis in fallback mode and
I now also converted its four uses of sprintf to
snprintf so we are clean there too.
The detailed changes list follows.
Courtesy of my CRANberries, there is also a diffstat report for this this release. More information is on the RcppRedis page and at the repository and its issue tracker.Changes in version 0.2.5 (2025-03-26)
- The continuous integration setup was updated several times
- Badges and URLs in README.md have been updated
- An updated interface from RApiSerialize is now used, and a versioned dependency on version 0.1.4 or later has been added
- The DESCRIPTION file now uses Authors@R
- Two possible bashisms have been converted in
configure.ac- The (fallback if needed) build of
libhiredis.anow sets-DNDEBUG, four uses ofsprintfconverted tosnprintf
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.
Are you a student aspiring to participate in the Google Summer of Code 2025? Would you like to improve the continuous integration pipeline used at salsa.debian.org, the Debian GitLab instance, to help improve the quality of tens of thousands of software packages in Debian?
This summer 2025, I and Emmanuel Arias will be participating as mentors in the GSoC program. We are available to mentor students who propose and develop improvements to the Salsa CI pipeline, as we are members of the Debian team that maintains it.
A post by Santiago Ruano Rinc n in the GitLab blog explains what Salsa CI is and its short history since inception in 2018. At the time of the article in fall 2023 there were 9000+ source packages in Debian using Salsa CI. Now in 2025 there are over 27,000 source packages in Debian using it, and since summer 2024 some Ubuntu developers have started using it for enhanced quality assurance of packaging changes before uploading new package revisions to Ubuntu. Personally, I have been using Salsa CI since its inception, and contributing as a team member since 2019. See my blog post about GitLab CI for MariaDB in Debian for a description of an advanced and extensive use case.
Helping Salsa CI is a great way to make a global impact, as it will help avoid regressions and improve the quality of Debian packages. The benefits reach far beyond just Debian, as it will also help hundreds of Debian derivatives, such as Ubuntu, Linux Mint, Tails, Purism PureOS, Pop!_OS, Zorin OS, Raspberry Pi OS, a large portion of Docker containers, and even the Windows Subsystem for Linux.
The Open Source Initiative has two classes of board seats: Affiliate seats, and Individual Member seats. In the upcoming election, each affiliate can nominate a candidate, and each affiliate can cast a vote for the Affiliate candidates, but there's only 1 Affiliate seat available. I initially expressed interest in being nominated as an Affiliate candidate via Debian. But since Bradley Kuhn is also running for an Affiliate seat with a similar platform to me, especially with regards to the OSAID, I decided to run as part of an aligned "ticket" as an Individual Member to avoid contention for the 1 Affiliate seat.Bradley and I discussed running on a similar ticket around 8/9pm Pacific, and I submitted my candidacy around 9pm PT on 17 February. I was dismayed when I received the following mail from Nick Vidal:Dear Luke,Nowhere on the "OSI s board of directors in 2025: details about the elections" page do they list a timezone for closure of nominations; they simply list Monday 17 February. The OSI's contact address is in California, so it seems arbitrary and capricious to retroactively define all of these processes as being governed by UTC.I was not able to participate in the "potential board director" info sessions accordingly, but people who attended heard that the importance of accommodating differing TZ's was discussed during the info session, and that OSI representatives mentioned they try to accommodate TZ's of everyone. This seems in sharp contrast with the above policy. I urge the OSI to reconsider this policy and allow me to stand for an Individual seat in the current cycle. Upd, N.B.: to people writing about this, I use they/them pronouns
Thank you for your interest in the OSI Board of Directors election. Unfortunately, we are unable to accept your application as it was submitted after the official deadline of Monday Feb 17 at 11:59 pm UTC. To ensure a fair process, we must adhere to the deadline for all candidates.
We appreciate your enthusiasm and encourage you to stay engaged with OSI s mission. We hope you ll consider applying in the future or contributing in other meaningful ways.
Best regards,
OSI Election Teams
dmidecode grep -A8 ^System Information
tells me that the Manufacturer is HP and Product Name is OMEN Transcend Gaming Laptop 14-fb0xxx
I m provisioning a new piece of hardware for my eng consultant and it s proving more difficult than I expected. I must admit guilt for some of this difficulty. Instead of installing using the debian installer on my keychain, I dd d the pv block device of the 16 inch 2023 version onto the partition set aside from it. I then rebooted into rescue mode and cleaned up the grub config, corrected the EFI boot partition s path in /etc/fstab, ran the grub installer from the rescue menu, and rebooted.
On the initial boot of the system, X or Wayland or whatever is supposed to be talking to this vast array of GPU hardware in this device, it s unable to do more than create a black screen on vt1. It s easy enough to switch to vt2 and get a shell on the installed system. So I m doing that and investigating what s changed in Trixie. It seems like it s pretty significant. Did they just throw out Keith Packard s and Behdad Esfahbod s work on font rendering? I don t understand what s happening in this effort to abstract to a simpler interface. I ll probably end up reading more about it.
In an effort to have Debian re-configure the system for Desktop use, I have uninstalled as many packages as I could find that were in the display and human interface category, or were firmware/drivers for devices not present in this Laptop s SoC. Some commands I used to clear these packages and re-install connamon follow:
dpkg -S /etc/X11
dpkg -S /usr/lib/firmware
apt-get purge $(dpkg -l grep -i \
-e gnome -e gtk -e x11-common -e xfonts- -e libvdpau -e dbus-user-session -e gpg-agent \
-e bluez -e colord -e cups -e fonts -e drm -e xf86 -e mesa -e nouveau -e cinnamon \
-e avahi -e gdk -e pixel -e desktop -e libreoffice -e x11 -e wayland -e xorg \
-e firmware-nvidia-graphics -e firmware-amd-graphics -e firmware-mediatek -e firmware-realtek \
awk ' print $2 ')
apt-get autoremove
apt-get purge $(dpkg -l grep '^r' awk ' print $2 ')
tasksel install cinnamon-desktop
And then I rebooted. When it came back up, I was greeted with a login prompt, and Trixie looks to be fully functional on this device, including the attached wifi radio, tethering to my android, and the thunderbolt-attached Marvell SFP+ enclosure.
I m also installing libvirt and fetched the DVD iso material for Debian, Ubuntu and Rocky in case we have a need of building VMs during the development process. These are the platforms that I target at work with gcp Dataproc, so I m pretty good at performing maintenance operation on them at this point.
Almost two years ago, Twitter launched encrypted direct messages. I wrote about their technical implementation at the time, and to the best of my knowledge nothing has changed. The short story is that the actual encryption primitives used are entirely normal and fine - messages are encrypted using AES, and the AES keys are exchanged via NIST P-256 elliptic curve asymmetric keys. The asymmetric keys are each associated with a specific device or browser owned by a user, so when you send a message to someone you encrypt the AES key with all of their asymmetric keys and then each device or browser can decrypt the message again. As long as the keys are managed appropriately, this is infeasible to break.
Armadillo is a powerful
and expressive C++ template library for linear algebra and scientific
computing. It aims towards a good balance between speed and ease of use,
has a syntax deliberately close to Matlab, and is useful for algorithm
development directly in C++, or quick conversion of research code into
production environments. RcppArmadillo
integrates this library with the R environment and language and is
widely used by (currently) 1234 other packages on CRAN, downloaded 38.8 million
times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint
/ vignette) by Conrad and myself has been cited 617 times according
to Google Scholar.
Conrad released a new
minor version 14.4.0 last month. That was preceding by several extensive
rounds of reverse-dependency checks covering the 1200+ packages at CRAN.
We eventually narrowed the impact down to just eight packages, and I
opened issue
#462 to manage the transition along with GitHub-only release
14.4.0-0 of RcppArmadillo.
Several maintainers responded very promptly and updated within days
this is truly appreciated. Yesterday the last package updated at CRAN coinciding nicely with our
planned / intended upload to CRAN one month after the release.
So this new release, at version -1, is now on CRAN. It brings the usual number
of small improvements to Armadillo itself as well as
updates to packaging.
The changes since the last CRAN release are summarised
below.
Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.Changes in RcppArmadillo version 14.4.0-1 (2025-03-17)
- CRAN release having given a few packages time to catch-up to small upstream change as discussed and managed in #462
- Updated bibliography, and small edits to sparse matrix vignette
- Switched continuous integration action to r-ci with implicit bootstrap
Changes in RcppArmadillo version 14.4.0-0 (2025-02-17) (GitHub Only)
- Upgraded to Armadillo release 14.4.0 (Filtered Espresso)
- Faster handling of
pow()andsquare()withinaccu()andsum()expressions- Faster
sort()andsort_index()for complex matrices- Expanded the field class with
.reshape()and.resize()member functions- More efficient handling of compound expressions by
sum(),reshape(),trans()- Better detection of vector expressions by
pow(),imag(),conj()- The package generator helper function now supports additional DESCRIPTIONs
- This release revealed a need for very minor changes for a handful reverse-dependency packages which will be organized via GitHub issue tracking
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.
libteora 1.2.0beta1 (2025 March 15)There are a few bugs still being investigated, and my plan is to wrap up a final 1.2.0 release two weekends from now. As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
- Bumped minor SONAME versions as methods changed constness of arguments.
- Updated libogg dependency to version 1.3.4 for ogg_uint64_t.
- Updated doxygen setup.
- Updated autotools setup and support scripts (#1467 #1800 #1987 #2318 #2320).
- Added support for RISC OS.
- Fixed mingw build (#2141).
- Improved ARM support.
- Converted SCons setup to work with Python 3.
- Introduced new configure options --enable-mem-constraint and --enable-gcc-sanitizers.
- Fixed all known compiler warnings and errors from gcc and clang.
- Improved examples for stability and correctness.
- Variuos speed, bug fixes and code quality improvements.
- Fixed build problem with Visual Studio (#2317).
- Avoids undefined bit shift of signed numbers (#2321, #2322).
- Avoids example encoder crash on bogus audio input (#2305).
- Fixed musl linking issue with asm enabled (#2287).
- Fixed some broken clamping in rate control (#2229).
- Added NULL check _tc and _setup even for data packets (#2279).
- Fixed mismatched oc_mb_fill_cmapping11 signature (#2068).
- Updated the documentation for theora_encode_comment() (#726).
- Adjusted build to Only link libcompat with dump_video (#1587).
- Corrected an operator precedence error in the visualization code (#1751).
- Fixed two spelling errors in the comments (#1804).
- Avoid negative bit shift operation in huffdec.c (CVE-2024-56431).
- Improved library documentation and specification text.
- Adjusted library dependencies so libtheoraenc do not depend on libtheoradec.
- Handle fallout from CVE-2017-14633 in libvorbis, check return value in encoder_example and transcoder_example.

offline-pki is a small Python application to manage an offline PKI.
It relies on yubikey-manager to manage YubiKeys and cryptography for
cryptographic operations not executed on the YubiKeys. The application has some
opinionated design choices. Notably, the cryptography is hard-coded to use NIST
P-384 elliptic curve.
The first step is to reset all your YubiKeys:
$ offline-pki yubikey reset This will reset the connected YubiKey. Are you sure? [y/N]: y New PIN code: Repeat for confirmation: New PUK code: Repeat for confirmation: New management key ('.' to generate a random one): WARNING[pki-yubikey] Using random management key: e8ffdce07a4e3bd5c0d803aa3948a9c36cfb86ed5a2d5cf533e97b088ae9e629 INFO[pki-yubikey] 0: Yubico YubiKey OTP+FIDO+CCID 00 00 INFO[pki-yubikey] SN: 23854514 INFO[yubikit.management] Device config written INFO[yubikit.piv] PIV application data reset performed INFO[yubikit.piv] Management key set INFO[yubikit.piv] New PUK set INFO[yubikit.piv] New PIN set INFO[pki-yubikey] YubiKey reset successful!
$ offline-pki certificate root --permitted example.com Management key for Root X: Plug YubiKey "Root X"... INFO[pki-yubikey] 0: Yubico YubiKey CCID 00 00 INFO[pki-yubikey] SN: 23854514 INFO[yubikit.piv] Data written to object slot 0x5fc10a INFO[yubikit.piv] Certificate written to slot 9C (SIGNATURE), compression=True INFO[yubikit.piv] Private key imported in slot 9C (SIGNATURE) of type ECCP384 Copy root certificate to another YubiKey? [y/N]: y Plug YubiKey "Root X"... INFO[pki-yubikey] 0: Yubico YubiKey CCID 00 00 INFO[pki-yubikey] SN: 23854514 INFO[yubikit.piv] Data written to object slot 0x5fc10a INFO[yubikit.piv] Certificate written to slot 9C (SIGNATURE), compression=True INFO[yubikit.piv] Private key imported in slot 9C (SIGNATURE) of type ECCP384 Copy root certificate to another YubiKey? [y/N]: n
$ offline-pki yubikey info INFO[pki-yubikey] 0: Yubico YubiKey CCID 00 00 INFO[pki-yubikey] SN: 23854514 INFO[pki-yubikey] Slot 9C (SIGNATURE): INFO[pki-yubikey] Private key type: ECCP384 INFO[pki-yubikey] Public key: INFO[pki-yubikey] Algorithm: secp384r1 INFO[pki-yubikey] Issuer: CN=Root CA INFO[pki-yubikey] Subject: CN=Root CA INFO[pki-yubikey] Serial: 1 INFO[pki-yubikey] Not before: 2024-07-05T18:17:19+00:00 INFO[pki-yubikey] Not after: 2044-06-30T18:17:19+00:00 INFO[pki-yubikey] PEM: -----BEGIN CERTIFICATE----- MIIBcjCB+aADAgECAgEBMAoGCCqGSM49BAMDMBIxEDAOBgNVBAMMB1Jvb3QgQ0Ew HhcNMjQwNzA1MTgxNzE5WhcNNDQwNjMwMTgxNzE5WjASMRAwDgYDVQQDDAdSb290 IENBMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAERg3Vir6cpEtB8Vgo5cAyBTkku/4w kXvhWlYZysz7+YzTcxIInZV6mpw61o8W+XbxZV6H6+3YHsr/IeigkK04/HJPi6+i zU5WJHeBJMqjj2No54Nsx6ep4OtNBMa/7T9foyMwITAPBgNVHRMBAf8EBTADAQH/ MA4GA1UdDwEB/wQEAwIBhjAKBggqhkjOPQQDAwNoADBlAjEAwYKy/L8leJyiZSnn xrY8xv8wkB9HL2TEAI6fC7gNc2bsISKFwMkyAwg+mKFKN2w7AjBRCtZKg4DZ2iUo 6c0BTXC9a3/28V5aydZj6rvx0JqbF/Ln5+RQL6wFMLoPIvCIiCU= -----END CERTIFICATE-----
offline-pki yubikey
intermediate and use it to sign certificates by providing a CSR to offline-pki
certificate sign. Be careful and inspect the CSR before signing it, as only the
subject name can be overridden. Check the documentation for more details.
Get the available options using the --help flag.

$ tio /dev/ttyUSB0 [16:40:44.546] tio v3.7 [16:40:44.546] Press ctrl-t q to quit [16:40:44.555] Connected to /dev/ttyUSB0 GXL:BL1:9ac50e:bb16dc;FEAT:ADFC318C:0;POC:1;RCY:0;SPI:0;0.0;CHK:0; TE: 36574 BL2 Built : 15:21:18, Aug 28 2019. gxl g1bf2b53 - luan.yuan@droid15-sz set vcck to 1120 mv set vddee to 1000 mv Board ID = 4 CPU clk: 1200MHz [ ]
offline-pki application, with shell completion,offline-pki application,# Execute the application locally nix run github:vincentbernat/offline-pki -- --help # Run the application inside a QEMU VM nix run github:vincentbernat/offline-pki\#qemu # Build a SD card for the Sweet Potato or for the Raspberry Pi nix build --system aarch64-linux github:vincentbernat/offline-pki\#sdcard.potato nix build --system aarch64-linux github:vincentbernat/offline-pki\#sdcard.generic # Get a development shell with the application nix develop github:vincentbernat/offline-pki
\xFF byte to the device. The
start bit is the only active-voltage bit being sent, and that produces my pulse.
I wrote this obvious C program:
#include <stdio.h> #include <stdlib.h> #include <stdbool.h> #include <sys/ioctl.h> #include <unistd.h> #include <fcntl.h> #include <termios.h> #include <stdint.h> #include <sys/time.h> static uint64_t gettimeofday_uint64() struct timeval tv; gettimeofday(&tv, NULL); return (uint64_t) tv.tv_sec * 1000000ULL + (uint64_t) tv.tv_usec; int main(int argc, char* argv[]) // open the serial device, and make it as raw as possible const char* device = "/dev/ttyS0"; const speed_t baud = B9600; int fd = open(device, O_WRONLY O_NOCTTY); tcflush(fd, TCIOFLUSH); struct termios options = .c_iflag = IGNBRK, .c_cflag = CS8 CREAD CLOCAL ; cfsetspeed(&options, baud); tcsetattr(fd, TCSANOW, &options); const uint64_t T_us = (uint64_t)(1e6 / 58.); const uint64_t t0 = gettimeofday_uint64(); for(int i=0; ; i++) const uint64_t t_target = t0 + T_us*i; const uint64_t t1 = gettimeofday_uint64(); if(t_target > t1) usleep(t_target - t1); write(fd, &((char) '\xff' ), 1); return 0;
write() call happens at 58Hz. I need these
pulses to be regular, so I need to also make sure that the time between each
userspace write() and when the edge actually hits the line is as short as
possible or, at least, stable.
Potential reasons for timing errors:
usleep() doesn't wake up exactly when it should. This is subject to the
Linux scheduler waking up the trigger process
write() almost certainly ends up scheduling a helper task to actually
write the \xFF to the hardware. This helper task is also subject to the
Linux scheduler waking it up.
./trigger program, and look at diagnostics while that's running.
I look at some device details:
# ls -lh /dev/ttyS0 crw-rw---- 1 root dialout 4, 64 Mar 6 18:11 /dev/ttyS0 # ls -lh /sys/dev/char/4:64/ total 0 -r--r--r-- 1 root root 4.0K Mar 6 16:51 close_delay -r--r--r-- 1 root root 4.0K Mar 6 16:51 closing_wait -rw-r--r-- 1 root root 4.0K Mar 6 16:51 console -r--r--r-- 1 root root 4.0K Mar 6 16:51 custom_divisor -r--r--r-- 1 root root 4.0K Mar 6 16:51 dev lrwxrwxrwx 1 root root 0 Mar 6 16:51 device -> ../../../0000:00:16.3:0.0 -r--r--r-- 1 root root 4.0K Mar 6 16:51 flags -r--r--r-- 1 root root 4.0K Mar 6 16:51 iomem_base -r--r--r-- 1 root root 4.0K Mar 6 16:51 iomem_reg_shift -r--r--r-- 1 root root 4.0K Mar 6 16:51 io_type -r--r--r-- 1 root root 4.0K Mar 6 16:51 irq -r--r--r-- 1 root root 4.0K Mar 6 16:51 line -r--r--r-- 1 root root 4.0K Mar 6 16:51 port drwxr-xr-x 2 root root 0 Mar 6 16:51 power -rw-r--r-- 1 root root 4.0K Mar 6 16:51 rx_trig_bytes lrwxrwxrwx 1 root root 0 Mar 6 16:51 subsystem -> ../../../../../../../class/tty -r--r--r-- 1 root root 4.0K Mar 6 16:51 type -r--r--r-- 1 root root 4.0K Mar 6 16:51 uartclk -rw-r--r-- 1 root root 4.0K Mar 6 16:51 uevent -r--r--r-- 1 root root 4.0K Mar 6 16:51 xmit_fifo_sizeUnsurprisingly, this is a part of the
tty subsystem. I don't want to spend the
time to really figure out how this works, so let me look at all the tty
kernel calls and also at all the kernel tasks scheduled by the trigger
process, since I suspect that the actual hardware poke is happening in a helper
task. I see this:
# bpftrace -e 'k:*tty* /comm=="trigger"/
printf("%d %d %s\n",pid,tid,probe);
t:sched:sched_wakeup /comm=="trigger"/
printf("switching to %s(%d); current backtrace:", args.comm, args.pid); print(kstack()); '
...
3397345 3397345 kprobe:tty_ioctl
3397345 3397345 kprobe:tty_check_change
3397345 3397345 kprobe:__tty_check_change
3397345 3397345 kprobe:tty_wait_until_sent
3397345 3397345 kprobe:tty_write
3397345 3397345 kprobe:file_tty_write.isra.0
3397345 3397345 kprobe:tty_ldisc_ref_wait
3397345 3397345 kprobe:n_tty_write
3397345 3397345 kprobe:tty_hung_up_p
switching to kworker/0:1(3400169); current backtrace:
ttwu_do_activate+268
ttwu_do_activate+268
try_to_wake_up+605
kick_pool+92
__queue_work.part.0+582
queue_work_on+101
rpm_resume+1398
__pm_runtime_resume+75
__uart_start+85
uart_write+150
n_tty_write+1012
file_tty_write.isra.0+373
vfs_write+656
ksys_write+109
do_syscall_64+130
entry_SYSCALL_64_after_hwframe+118
3397345 3397345 kprobe:tty_update_time
3397345 3397345 kprobe:tty_ldisc_deref
... repeated with each pulse ...
Looking at the sources I see that uart_write() calls __uart_start(), which
schedules a task to call serial_port_runtime_resume() which eventually calls
serial8250_tx_chars(), which calls some low-level functions to actually send
the bits.
I look at the time between two of those calls to quantify the scheduler latency:
pulserate=58 sudo zsh -c \ '( echo "# dt_write_ns dt_task_latency_ns"; bpftrace -q -e "k:vfs_write /comm==\"trigger\" && arg2==1/ \$t=nsecs(); if(@t0) @dt_write = \$t-@t0; @t0=\$t; k:serial8250_tx_chars /@dt_write/ \$t=nsecs(); printf(\"%d %d\\n\", @dt_write, \$t-@t0); " )' \ vnl-filter \ --stream -p dt_write_ms="dt_write_ns/1e6 - 1e3/$pulserate",dt_task_latency_ms=dt_task_latency_ns/1e6 \ feedgnuplot \ --stream \ --lines \ --points \ --xlen 200 \ --vnl \ --autolegend \ --xlabel 'Pulse index' \ --ylabel 'Latency (ms)'
write() call happens. This shows effect #1
from above: how promptly the trigger process wakes up
trigger task. This tells
the scheduler to wake that task up first, even if something else is currently
using the CPU. I do this:
sudo chrt -p 90 pidof trigger
6.12) the helper task is called kworker/N where N is
the CPU index. I tie the trigger process to cpu 0, and priorities all the
relevant helpers:
taskset -c 0 ./trigger 58 pgrep -f kworker/0 while read pid sudo chrt -p 90 $pid
-rt kernel. But all this doesn't matter if the serial hardware adds
unacceptable delays. Let's look.
Let's look at it with a logic analyzer. I use a saleae logic analyzer with
sigrok. The tool spits out the samples as it gets them, and an awk script
finds the edges and reports the timings to give me a realtime plot.
samplerate=500000; pulserate=58.; sigrok-cli -c samplerate=$samplerate -O csv --continuous -C D1 \ mawk -Winteractive \ "prev_logic==0 && \$0==1 \ iedge = NR; if(prev_iedge) di = iedge -prev_iedge; dt = di/$samplerate; print(dt*1000); prev_iedge = iedge; prev_logic=\$0; " feedgnuplot --stream --ylabel 'Period (ms)' --equation "1000./$pulserate title \"True $ pulserate Hz period\""
A few months ago I explained that one reason why this blog has become more quiet is that all my work on Lean is covered elsewhere.
This post is an exception, because it is an observation that is (arguably) interesting, but does not lead anywhere, so where else to put it than my own blog
Want to share your thoughts about this? Please join the discussion on the Lean community zulip!
List.map, then extra attention used to be necessary so that Lean can see that xs.map applies its argument only elements of the list xs. The usual idiom is to write xs.attach.map instead, where List.attach attaches to the list elements a proof that they are in that list. You can read more about this my Lean blog post on recursive definitions and our new shiny reference manual, look for Example Nested Recursion in Higher-order Functions .
To make this step less tedious I taught Lean to automatically rewrite xs.map to xs.attach.map (where suitable) within the construction of well-founded recursion, so that nested recursion just works (issue #5471). We already do such a rewriting to change if c then else to the dependent if h : c then else , but the attach-introduction is much more ambitious (the rewrites are not definitionally equal, there are higher-order arguments etc.) Rewriting the terms in a way that we can still prove the connection later when creating the equational lemmas is hairy at best. Also, we want the whole machinery to be extensible by the user, setting up their own higher order functions to add more facts to the context of the termination proof.
I implemented it like this (PR #6744) and it ships with 4.18.0, but in the course of this work I thought about a quite different and maybe better way to do this, and well-founded recursion in general:
fix
Recall that to use WellFounded.fix
WellFounded.fix : (hwf : WellFounded r) (F : (x : ) ((y : ) r y x C y) C x) (x : ) : C x
we have to rewrite the functorial of the recursive function, which naturally has type
F : ((y : ) C y) ((x : ) C x)
to the one above, where all recursive calls take the termination proof r y x. This is a fairly hairy operation, mangling the type of matcher s motives and whatnot.
Things are simpler for recursive definitions using the new partial_fixpoint machinery, where we use Lean.Order.fix
Lean.Order.fix : [CCPO ] (F : ) (hmono : monotone F) :
so the functorial s type is unmodified (here will be ((x : ) C x)), and everything else is in the propositional side-condition montone F. For this predicate we have a syntax-guided compositional tactic, and it s easily extensible, e.g. by
theorem monotone_mapM (f : m ) (xs : List ) (hmono : monotone f) :
monotone (fun x => xs.mapM (f x))
Once given, we don t care about the content of that proof. In particular proving the unfolding theorem only deals with the unmodified F that closely matches the function definition as written by the user. Much simpler!
List.map something like our List.map_congr_left
List.map_congr_left : (h : a l, f a = g a) :
List.map f l = List.map g l
This is because in Isabelle, too, the termination proofs is a side-condition that essentially states the functorial F calls its argument f only on smaller arguments .
F can alternatively implemented using the more restrictive argument. The following callsOn P F predicate can express that F calls its higher-order argument only on arguments that satisfy the predicate P:
section setup
variable : Sort u
variable : Sort v
variable : Sort w
def callsOn (P : Prop) (F : ( y, y) ) :=
(F': ( y, P y y) ), f, F' (fun y _ => f y) = F f
variable (R : Prop)
variable (F : ( y, y) ( x, x))
local infix:50 " " => R
def recursesVia : Prop := x, callsOn ( x) (fun f => F f x)
noncomputable def fix (wf : WellFounded R) (h : recursesVia R F) : ( x, x) :=
wf.fix (fun x => (h x).choose)
def fix_eq (wf : WellFounded R) h x :
fix R F wf h x = F (fix R F wf h) x := by
unfold fix
rw [wf.fix_eq]
apply (h x).choose_spec
This allows nice compositional lemmas to discharge callsOn predicates:
theorem callsOn_base (y : ) (hy : P y) :
callsOn P (fun (f : x, x) => f y) := by
exists fun f => f y hy
intros; rfl
@[simp]
theorem callsOn_const (x : ) :
callsOn P (fun (_ : x, x) => x) :=
fun _ => x, fun _ => rfl
theorem callsOn_app
: Sort uu : Sort ww
(F : ( y, y) ) -- can this also support dependent types?
(F : ( y, y) )
(h : callsOn P F )
(h : callsOn P F ) :
callsOn P (fun f => F f (F f)) := by
obtain F ', h := h
obtain F ', h := h
exists (fun f => F ' f (F ' f))
intros; simp_all
theorem callsOn_lam
: Sort uu
(F : ( y, y) ) -- can this also support dependent types?
(h : x, callsOn P (F x)) :
callsOn P (fun f x => F x f) := by
exists (fun f x => (h x).choose f)
intro f
ext x
apply (h x).choose_spec
theorem callsOn_app2
: Sort uu : Sort ww
(g : )
(F : ( y, y) ) -- can this also support dependent types?
(F : ( y, y) )
(h : callsOn P F )
(h : callsOn P F ) :
callsOn P (fun f => g (F f) (F f)) := by
apply_rules [callsOn_app, callsOn_const]
With this setup, we can have the following, possibly user-defined, lemma expressing that List.map calls its arguments only on elements of the list:
theorem callsOn_map ( : Type uu) ( : Type ww)
(P : Prop) (F : ( y, y) ) (xs : List )
(h : x, x xs callsOn P (fun f => F f x)) :
callsOn P (fun f => xs.map (fun x => F f x)) := by
suffices callsOn P (fun f => xs.attach.map (fun x, h => F f x)) by
simpa
apply callsOn_app
apply callsOn_app
apply callsOn_const
apply callsOn_lam
intro x', hx'
dsimp
exact (h x' hx')
apply callsOn_const
end setup
So here is the (manual) construction of a nested map for trees:
section examples
structure Tree ( : Type u) where
val :
cs : List (Tree )
-- essentially
-- def Tree.map (f : ) : Tree Tree :=
-- fun t => f t.val, t.cs.map Tree.map )
noncomputable def Tree.map (f : ) : Tree Tree :=
fix (sizeOf < sizeOf ) (fun map t => f t.val, t.cs.map map )
(InvImage.wf (sizeOf ) WellFoundedRelation.wf) < by
intro v, cs
dsimp only
apply callsOn_app2
apply callsOn_const
apply callsOn_map
intro t' ht'
apply callsOn_base
-- ht' : t' cs -- !
-- sizeOf t' < sizeOf val := v, cs := cs
decreasing_trivial
end examples
This makes me happy!
All details of the construction are now contained in a proof that can proceed by a syntax-driven tactic and that s easily and (likely robustly) extensible by the user. It also means that we can share a lot of code paths (e.g. everything related to equational theorems) between well-founded recursion and partial_fixpoint.
I wonder if this construction is really as powerful as our current one, or if there are certain (likely dependently typed) functions where this doesn t fit, but the above is dependent, so it looks good.
With this construction, functions defined by well-founded recursion will reduce even worse in the kernel, I assume. This may be a good thing.
F to return a proof, but since the extra assumptions (e.g. for ite or List.map) only exist in the termination proof, they are not available in F.
Oh wey, how anticlimactic.
An update to our package RcppNLoptExample
arrived on CRAN earlier today
marking the first update since the intial
release more than four year ago. The nloptr package, created by
Jelmer Ypma, has long been
providing an excellent R interface to NLopt, a very
comprehensive library for nonlinear optimization. In particular, Jelmer carefully exposed the API
entry points such that other R packages can rely on NLopt without having
to explicitly link to it (as one can rely on R providing
sufficient function calling and registration to make this possible by
referring back to nloptr
which naturally has the linking information and resolution). This
package demonstrates this in a simple-to-use Rcpp example package that can serve as a
stanza.
More recent NLopt versions appear
to have changed behaviour a little so that an example we relied upon in
simple unit test now converges to a marginally different numerical
value, so we adjusted a convergence treshold. Other than that we did a
number of the usual small updates to package metadata, to the README.md
file, and to continuous integration.
The (very short) NEWS entry follows:
Courtesy of my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo.Changes in version 0.0.2 (2025-03-09)
- Updated tolerance in simple test as newer upstream nlopt change behaviour ever so slightly leading to an other spurious failure
- Numerous small and standard updates to DESCRIPTION, README.md, badges, and continuous integration setup
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.
Update 20250309 13:20-03:00 - How to help
A friend of mine living in the USA sent me this link to help the flood victims: Support Bah a Blanca (Argentina) Flood Victims
Original blog post
These are not good news. In fact, much the contrary. Compared to the real issue, the fact that I'm not able to attend Embedded World at Nuremberg is, well, a detail. Or at least that's what I'm forcing myself to believe, as I REALLY wanted to be there. But mother nature said otherwise.
Park "D. Alberto Martinelli", Las Ca itas, Bah a Blanca (Google Maps)
Bah a Blanca , the city I live, has received a lot on rainfall. Really, a lot. Let me introduce the number like this: the previous highest recorded measurement was 170mm (6.69 inch)... in a month. Yesterday Friday 07 we had more than 400mm (15.75 inch) in 9 hours.
But those are just numbers. Some things are better seen in images.
I'll start with some soft ones.
Sink in Fournier street near Cambaceres (Google Maps)
I also happen to do figure skating in the same school of the 4 times world champions (where "world" means the whole world) Roller Dreams precision skating team - Instagram, from Club El Nacional. Our skating rink got severely damaged with the hail we had like 3 weeks ago (yes, we had hail too!!!). Now it's just impossible:
The "real" thing
Let's get to the heavy, heartbreaker part. I did go to downtown Bah a Blanca, but during night, so let me share some links, most of them in Spanish, but images are images:
Welcome to the second report in 2025 from the Reproducible Builds project. Our monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. As usual, however, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website.
Table of contents:
Similar to last year s event, there was considerable activity regarding Reproducible Builds at FOSDEM 2025, held on on 1st and 2nd February this year in Brussels, Belgium. We count at least four talks related to reproducible builds. (You can also read our news report from last year s event in which Holger Levsen presented in the main track.)
Jelle van der Waa, Holger Levsen and kpcyrd presented in the Distributions track on A Tale of several distros joining forces for a common goal. In this talk, three developers from two different Linux distributions (Arch Linux and Debian), discuss this goal which is, of course, reproducible builds. The presenters discuss both what is shared and different between the two efforts, touching on the history and future challenges alike. The slides of this talk are available to view, as is the full video (30m02s). The talk was also discussed on Hacker News.
Zbigniew J drzejewski-Szmek presented in the ever-popular Python track a on Rewriting .pyc files for fun and reproducibility, i.e. the bytecode files generated by Python in order to speed up module imports: It s been known for a while that those are not reproducible: on different architectures, the bytecode for exactly the same sources ends up slightly different. The slides of this talk are available, as is the full video (28m32s).
Lastly, Simon Tournier presented in the Open Research track on the confluence of GNU Guix and Software Heritage: Source Code Archiving to the Rescue of Reproducible Deployment. Simon s talk describes design and implementation we came up and reports on the archival coverage for package source code with data collected over five years. It opens to some remaining challenges toward a better open and reproducible research. The slides for the talk are available, as is the full video (23m17s).
Vagrant Cascadian presented at this year s PyCascades conference which was held on February 8th and 9th February in Portland, OR, USA. PyCascades is a regional instance of PyCon held in the Pacific Northwest. Vagrant s talk, entitled Re-Py-Ducible Builds caught the audience s attention with the following abstract:
Crank your Python best practices up to 11 with Reproducible Builds! This talk will explore Reproducible Builds by highlighting issues identified in Python projects, from the simple to the seemingly inscrutable. Reproducible Builds is basically the crazy idea that when you build something, and you build it again, you get the exact same thing or even more important, if someone else builds it, they get the exact same thing too.More info is available on the talk s page.
The last few months have seen the introduction of reproduce.debian.net. Announced first at the recent Debian MiniDebConf in Toulouse, reproduce.debian.net is an instance of rebuilderd operated by the Reproducible Builds project.
Powering this work is rebuilderd, our server which monitors the official package repositories of Linux distributions and attempt to reproduce the observed results there. This month, however, Holger Levsen:
riscv64 nodes to a total of 4, and added a new amd64 node added thanks to our (now 10-year sponsor), IONOS.
devscripts package, incorporating changes from Jochen Sprickerhof to the debrebuild script specifically to fix the handling the Rules-Requires-Root header in Debian source packages.
rust-libbz2-rs-sys, rust-actix-web, rust-actix-server, rust-actix-http, rust-actix-server, rust-actix-http, rust-actix-web-codegen and rust-time-tz) after they were prepared by kpcyrd :
sbuild package to:
Rules-Requires-Root.--root-owner-group to old versions of dpkg.rust-i8n (random HashMap order)starship/shadowpython-assertpy.terminaltables3.acme.sh.node-svgdotjs-svg.js.onevpl-intel-gpu.rocdbgapi.siege.pkg-rocm-tools.warewulf4 (embeds CPU core count)highlight (timestamp)arch-wiki-lite (timestamp)f3d (timestamp)jacktrip (timestamp)prometheus (timestamp)go (clear GOROOT for func ldShared when -trimpath is used)
In Debian, 17 reviews of Debian packages were added, 6 were updated and 8 were removed this month adding to our knowledge about identified issues.
Thanks to an investment from the Sovereign Tech Agency, the FreeBSD project s work on unprivileged and reproducible builds continued this month. Notable fixes include:
pkg (hash ordering)makefs (source filesystem inode number leakage)FreeBSD base system packages (timestamp)
Finally, Douglas DeMaio published an article on the openSUSE blog on announcing that the Reproducible-openSUSE (RBOS) Project Hits [Significant] Milestone. In particular:
The Reproducible-openSUSE (RBOS) project, which is a proof-of-concept fork of openSUSE, has reached a significant milestone after demonstrating a usable Linux distribution can be built with 100% bit-identical packages.This news was also announced on our mailing list by Bernhard M. Wiedemann, who also published another report for openSUSE as well.
diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 288 and 289 to Debian:
asar to DIFFOSCOPE_FAIL_TESTS_ON_MISSING_TOOLS in order to address Debian bug #1095057) [ ]CalledProcessError when calling html2text. [ ]1.14.1-2 was uploaded to Debian unstable by Holger Levsen.
There were a large number of improvements made to our website this month, including:
readdir component of Bernhard s own Unreproducible Package. [ ]
README file to document a couple of additional dependencies [ ][ ], as well as did more work on a future Getting Started guide page [ ][ ].
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In January, a number of changes were made by Holger Levsen, including:
riscv64 archicture nodes and integrate them elsewhere in our infrastructure. [ ][ ]riscv64 architecture nodes. [ ][ ][ ][ ][ ]/all/api/ API endpoints on reproduce.debian.net by altering the nginx configuration. [ ]
mmdebstrap package [ ] as well as updating some documentation [ ].
#reproducible-builds on irc.oftc.net.
rb-general@lists.reproducible-builds.org
#!/usr/bin/python3 import sys import os import re def report_solution_to_console(vars): for w in days_of_week: annotation = '' if human_annotate is not None: for s in shifts.keys(): m = re.match(rf' w - ', s) if not m: continue if vars[human_annotate][s].value(): annotation = f" ( human_annotate SCHEDULED)" break if not len(annotation): annotation = f" ( human_annotate OFF)" print(f" w annotation ") for s in shifts.keys(): m = re.match(rf' w - ', s) if not m: continue annotation = '' if human_annotate is not None: annotation = f" ( human_annotate shifts[s][human_annotate] )" print(f" ---- s[m.end():] annotation ") for h in humans: if vars[h][s].value(): print(f" h ( shifts[s][h] )") def report_solution_summary_to_console(vars): print("\nSUMMARY") for h in humans: print(f"-- h ") print(f" benefit: benefits[h].value():.3f ") counts = dict() for a in availabilities: counts[a] = 0 for s in shifts.keys(): if vars[h][s].value(): counts[shifts[s][h]] += 1 for a in availabilities: print(f" counts[a] a ") human_annotate = None days_of_week = ('SUNDAY', 'MONDAY', 'TUESDAY', 'WEDNESDAY', 'THURSDAY', 'FRIDAY', 'SATURDAY') humans = ['ALICE', 'BOB', 'CAROL', 'DAVID', 'EVE', 'FRANK', 'GRACE', 'HEIDI', 'IVAN', 'JUDY'] shifts = 'SUNDAY - SANDING 9:00 AM - 4:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'PREFERRED', 'DAVID': 'PREFERRED', 'EVE': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'DISFAVORED', 'HEIDI': 'DISFAVORED', 'IVAN': 'PREFERRED', 'JUDY': 'NEUTRAL' , 'WEDNESDAY - SAWING 7:30 AM - 2:30 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'PREFERRED', 'DAVID': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'NEUTRAL', 'HEIDI': 'DISFAVORED', 'IVAN': 'PREFERRED', 'EVE': 'REFUSED', 'JUDY': 'REFUSED' , 'THURSDAY - SANDING 9:00 AM - 4:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'PREFERRED', 'DAVID': 'PREFERRED', 'EVE': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'PREFERRED', 'HEIDI': 'DISFAVORED', 'IVAN': 'PREFERRED', 'JUDY': 'PREFERRED' , 'SATURDAY - SAWING 7:30 AM - 2:30 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'PREFERRED', 'DAVID': 'PREFERRED', 'FRANK': 'PREFERRED', 'HEIDI': 'DISFAVORED', 'IVAN': 'PREFERRED', 'EVE': 'REFUSED', 'JUDY': 'REFUSED', 'GRACE': 'REFUSED' , 'SUNDAY - SAWING 9:00 AM - 4:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'PREFERRED', 'DAVID': 'PREFERRED', 'EVE': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'DISFAVORED', 'IVAN': 'PREFERRED', 'JUDY': 'PREFERRED', 'HEIDI': 'REFUSED' , 'MONDAY - SAWING 9:00 AM - 4:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'PREFERRED', 'DAVID': 'PREFERRED', 'EVE': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'PREFERRED', 'IVAN': 'PREFERRED', 'JUDY': 'PREFERRED', 'HEIDI': 'REFUSED' , 'TUESDAY - SAWING 9:00 AM - 4:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'PREFERRED', 'DAVID': 'PREFERRED', 'EVE': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'NEUTRAL', 'IVAN': 'PREFERRED', 'JUDY': 'PREFERRED', 'HEIDI': 'REFUSED' , 'WEDNESDAY - PAINTING 7:30 AM - 2:30 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'NEUTRAL', 'HEIDI': 'DISFAVORED', 'IVAN': 'PREFERRED', 'EVE': 'REFUSED', 'JUDY': 'REFUSED', 'DAVID': 'REFUSED' , 'THURSDAY - SAWING 9:00 AM - 4:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'PREFERRED', 'DAVID': 'PREFERRED', 'EVE': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'PREFERRED', 'IVAN': 'PREFERRED', 'JUDY': 'PREFERRED', 'HEIDI': 'REFUSED' , 'FRIDAY - SAWING 9:00 AM - 4:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'PREFERRED', 'DAVID': 'PREFERRED', 'EVE': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'PREFERRED', 'IVAN': 'PREFERRED', 'JUDY': 'DISFAVORED', 'HEIDI': 'REFUSED' , 'SATURDAY - PAINTING 7:30 AM - 2:30 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'PREFERRED', 'FRANK': 'PREFERRED', 'HEIDI': 'DISFAVORED', 'IVAN': 'PREFERRED', 'EVE': 'REFUSED', 'JUDY': 'REFUSED', 'GRACE': 'REFUSED', 'DAVID': 'REFUSED' , 'SUNDAY - PAINTING 9:45 AM - 4:45 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'NEUTRAL', 'EVE': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'DISFAVORED', 'IVAN': 'PREFERRED', 'JUDY': 'PREFERRED', 'HEIDI': 'REFUSED', 'DAVID': 'REFUSED' , 'MONDAY - PAINTING 9:45 AM - 4:45 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'NEUTRAL', 'EVE': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'PREFERRED', 'IVAN': 'PREFERRED', 'JUDY': 'NEUTRAL', 'HEIDI': 'REFUSED', 'DAVID': 'REFUSED' , 'TUESDAY - PAINTING 9:45 AM - 4:45 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'NEUTRAL', 'EVE': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'NEUTRAL', 'IVAN': 'PREFERRED', 'JUDY': 'PREFERRED', 'HEIDI': 'REFUSED', 'DAVID': 'REFUSED' , 'WEDNESDAY - SANDING 9:45 AM - 4:45 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'NEUTRAL', 'DAVID': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'NEUTRAL', 'HEIDI': 'DISFAVORED', 'IVAN': 'PREFERRED', 'JUDY': 'NEUTRAL', 'EVE': 'REFUSED' , 'THURSDAY - PAINTING 9:45 AM - 4:45 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'NEUTRAL', 'EVE': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'NEUTRAL', 'IVAN': 'PREFERRED', 'JUDY': 'PREFERRED', 'HEIDI': 'REFUSED', 'DAVID': 'REFUSED' , 'FRIDAY - PAINTING 9:45 AM - 4:45 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'NEUTRAL', 'EVE': 'PREFERRED', 'FRANK': 'PREFERRED', 'GRACE': 'PREFERRED', 'IVAN': 'PREFERRED', 'JUDY': 'DISFAVORED', 'HEIDI': 'REFUSED', 'DAVID': 'REFUSED' , 'SATURDAY - SANDING 9:45 AM - 4:45 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'NEUTRAL', 'DAVID': 'PREFERRED', 'FRANK': 'PREFERRED', 'HEIDI': 'DISFAVORED', 'IVAN': 'PREFERRED', 'EVE': 'REFUSED', 'JUDY': 'REFUSED', 'GRACE': 'REFUSED' , 'SUNDAY - PAINTING 11:00 AM - 6:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'NEUTRAL', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'HEIDI': 'PREFERRED', 'IVAN': 'NEUTRAL', 'JUDY': 'NEUTRAL', 'DAVID': 'REFUSED' , 'MONDAY - PAINTING 12:00 PM - 7:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'NEUTRAL', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'PREFERRED', 'IVAN': 'NEUTRAL', 'JUDY': 'NEUTRAL', 'HEIDI': 'REFUSED', 'DAVID': 'REFUSED' , 'TUESDAY - PAINTING 12:00 PM - 7:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'NEUTRAL', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'IVAN': 'NEUTRAL', 'HEIDI': 'REFUSED', 'JUDY': 'REFUSED', 'DAVID': 'REFUSED' , 'WEDNESDAY - PAINTING 12:00 PM - 7:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'NEUTRAL', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'IVAN': 'NEUTRAL', 'JUDY': 'PREFERRED', 'EVE': 'REFUSED', 'HEIDI': 'REFUSED', 'DAVID': 'REFUSED' , 'THURSDAY - PAINTING 12:00 PM - 7:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'NEUTRAL', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'IVAN': 'NEUTRAL', 'JUDY': 'PREFERRED', 'HEIDI': 'REFUSED', 'DAVID': 'REFUSED' , 'FRIDAY - PAINTING 12:00 PM - 7:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'NEUTRAL', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'IVAN': 'NEUTRAL', 'JUDY': 'DISFAVORED', 'HEIDI': 'REFUSED', 'DAVID': 'REFUSED' , 'SATURDAY - PAINTING 12:00 PM - 7:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'NEUTRAL', 'FRANK': 'NEUTRAL', 'IVAN': 'NEUTRAL', 'JUDY': 'DISFAVORED', 'EVE': 'REFUSED', 'HEIDI': 'REFUSED', 'GRACE': 'REFUSED', 'DAVID': 'REFUSED' , 'SUNDAY - SAWING 12:00 PM - 7:00 PM': 'ALICE': 'PREFERRED', 'BOB': 'PREFERRED', 'CAROL': 'NEUTRAL', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'IVAN': 'NEUTRAL', 'JUDY': 'PREFERRED', 'HEIDI': 'REFUSED', 'DAVID': 'REFUSED' , 'MONDAY - SAWING 2:00 PM - 9:00 PM': 'ALICE': 'PREFERRED', 'BOB': 'PREFERRED', 'CAROL': 'DISFAVORED', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'IVAN': 'DISFAVORED', 'JUDY': 'DISFAVORED', 'HEIDI': 'REFUSED', 'DAVID': 'REFUSED' , 'TUESDAY - SAWING 2:00 PM - 9:00 PM': 'ALICE': 'PREFERRED', 'BOB': 'PREFERRED', 'CAROL': 'DISFAVORED', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'IVAN': 'DISFAVORED', 'HEIDI': 'REFUSED', 'JUDY': 'REFUSED', 'DAVID': 'REFUSED' , 'WEDNESDAY - SAWING 2:00 PM - 9:00 PM': 'ALICE': 'PREFERRED', 'BOB': 'PREFERRED', 'CAROL': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'IVAN': 'DISFAVORED', 'JUDY': 'DISFAVORED', 'EVE': 'REFUSED', 'HEIDI': 'REFUSED', 'DAVID': 'REFUSED' , 'THURSDAY - SAWING 2:00 PM - 9:00 PM': 'ALICE': 'PREFERRED', 'BOB': 'PREFERRED', 'CAROL': 'DISFAVORED', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'IVAN': 'DISFAVORED', 'JUDY': 'DISFAVORED', 'HEIDI': 'REFUSED', 'DAVID': 'REFUSED' , 'FRIDAY - SAWING 2:00 PM - 9:00 PM': 'ALICE': 'PREFERRED', 'BOB': 'PREFERRED', 'CAROL': 'DISFAVORED', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'IVAN': 'DISFAVORED', 'HEIDI': 'REFUSED', 'JUDY': 'REFUSED', 'DAVID': 'REFUSED' , 'SATURDAY - SAWING 2:00 PM - 9:00 PM': 'ALICE': 'PREFERRED', 'BOB': 'PREFERRED', 'CAROL': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'IVAN': 'DISFAVORED', 'JUDY': 'DISFAVORED', 'EVE': 'REFUSED', 'HEIDI': 'REFUSED', 'GRACE': 'REFUSED', 'DAVID': 'REFUSED' , 'SUNDAY - PAINTING 12:15 PM - 7:15 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'PREFERRED', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'HEIDI': 'NEUTRAL', 'IVAN': 'DISFAVORED', 'JUDY': 'NEUTRAL', 'DAVID': 'REFUSED' , 'MONDAY - PAINTING 2:00 PM - 9:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'DISFAVORED', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'HEIDI': 'NEUTRAL', 'IVAN': 'DISFAVORED', 'JUDY': 'DISFAVORED', 'DAVID': 'REFUSED' , 'TUESDAY - PAINTING 2:00 PM - 9:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'DISFAVORED', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'HEIDI': 'NEUTRAL', 'IVAN': 'DISFAVORED', 'JUDY': 'REFUSED', 'DAVID': 'REFUSED' , 'WEDNESDAY - PAINTING 2:00 PM - 9:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'HEIDI': 'NEUTRAL', 'IVAN': 'DISFAVORED', 'JUDY': 'DISFAVORED', 'EVE': 'REFUSED', 'DAVID': 'REFUSED' , 'THURSDAY - PAINTING 2:00 PM - 9:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'DISFAVORED', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'HEIDI': 'NEUTRAL', 'IVAN': 'DISFAVORED', 'JUDY': 'DISFAVORED', 'DAVID': 'REFUSED' , 'FRIDAY - PAINTING 2:00 PM - 9:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'DISFAVORED', 'EVE': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'GRACE': 'NEUTRAL', 'HEIDI': 'NEUTRAL', 'IVAN': 'DISFAVORED', 'JUDY': 'REFUSED', 'DAVID': 'REFUSED' , 'SATURDAY - PAINTING 2:00 PM - 9:00 PM': 'ALICE': 'NEUTRAL', 'BOB': 'NEUTRAL', 'CAROL': 'DISFAVORED', 'FRANK': 'NEUTRAL', 'HEIDI': 'NEUTRAL', 'IVAN': 'DISFAVORED', 'JUDY': 'DISFAVORED', 'EVE': 'REFUSED', 'GRACE': 'REFUSED', 'DAVID': 'REFUSED' availabilities = ['PREFERRED', 'NEUTRAL', 'DISFAVORED'] import pulp prob = pulp.LpProblem("Scheduling", pulp.LpMaximize) vars = pulp.LpVariable.dicts("Assignments", (humans, shifts.keys()), None,None, # bounds; unused, since these are binary variables pulp.LpBinary) # Everyone works at least 2 shifts Nshifts_min = 2 for h in humans: prob += ( pulp.lpSum([vars[h][s] for s in shifts.keys()]) >= Nshifts_min, f" h works at least Nshifts_min shifts", ) # each shift is ~ 8 hours, so I limit everyone to 40/8 = 5 shifts Nshifts_max = 5 for h in humans: prob += ( pulp.lpSum([vars[h][s] for s in shifts.keys()]) <= Nshifts_max, f" h works at most Nshifts_max shifts", ) # all shifts staffed and not double-staffed for s in shifts.keys(): prob += ( pulp.lpSum([vars[h][s] for h in humans]) == 1, f" s is staffed", ) # each human can work at most one shift on any given day for w in days_of_week: for h in humans: prob += ( pulp.lpSum([vars[h][s] for s in shifts.keys() if re.match(rf' w ',s)]) <= 1, f" h cannot be double-booked on w " ) #### Some explicit constraints; as an example # DAVID can't work any PAINTING shift and is off on Thu and Sun h = 'DAVID' prob += ( pulp.lpSum([vars[h][s] for s in shifts.keys() if re.search(r'- PAINTING',s)]) == 0, f" h can't work any PAINTING shift" ) prob += ( pulp.lpSum([vars[h][s] for s in shifts.keys() if re.match(r'THURSDAY SUNDAY',s)]) == 0, f" h is off on Thursday and Sunday" ) # Do not assign any "REFUSED" shifts for s in shifts.keys(): for h in humans: if shifts[s][h] == 'REFUSED': prob += ( vars[h][s] == 0, f" h is not available for s " ) # Objective. I try to maximize the "happiness". Each human sees each shift as # one of: # # PREFERRED # NEUTRAL # DISFAVORED # REFUSED # # I set a hard constraint to handle "REFUSED", and arbitrarily, I set these # benefit values for the others benefit_availability = dict() benefit_availability['PREFERRED'] = 3 benefit_availability['NEUTRAL'] = 2 benefit_availability['DISFAVORED'] = 1 # Not used, since this is a hard constraint. But the code needs this to be a # part of the benefit. I can ignore these in the code, but let's keep this # simple benefit_availability['REFUSED' ] = -1000 benefits = dict() for h in humans: benefits[h] = \ pulp.lpSum([vars[h][s] * benefit_availability[shifts[s][h]] \ for s in shifts.keys()]) benefit_total = \ pulp.lpSum([benefits[h] \ for h in humans]) prob += ( benefit_total, "happiness", ) prob.solve() if pulp.LpStatus[prob.status] == "Optimal": report_solution_to_console(vars) report_solution_summary_to_console(vars)
humans variable, and the shift schedule and the
workers' preferences are encoded in the shifts dict. The problem is defined by
a vars dict of dicts, each a boolean variable indicating whether a particular
worker is scheduled for a particular shift. We define a set of constraints to
these worker allocations to restrict ourselves to valid solutions. And among
these valid solutions, we try to find the one that maximizes some benefit
function, defined here as:
benefit_availability = dict() benefit_availability['PREFERRED'] = 3 benefit_availability['NEUTRAL'] = 2 benefit_availability['DISFAVORED'] = 1 benefits = dict() for h in humans: benefits[h] = \ pulp.lpSum([vars[h][s] * benefit_availability[shifts[s][h]] \ for s in shifts.keys()]) benefit_total = \ pulp.lpSum([benefits[h] \ for h in humans])
benefit_availability dict. With the above numbers, it's equally good for
somebody to have a NEUTRAL shift and a day off as it is for them to have
DISFAVORED shifts. If we really want to encourage the program to work people as
much as possible (days off discouraged), we'd want to raise the DISFAVORED
threshold.
I run this program and I get:
....
Result - Optimal solution found
Objective value: 108.00000000
Enumerated nodes: 0
Total iterations: 0
Time (CPU seconds): 0.01
Time (Wallclock seconds): 0.01
Option for printingOptions changed from normal to all
Total time (CPU seconds): 0.02 (Wallclock seconds): 0.02
SUNDAY
---- SANDING 9:00 AM - 4:00 PM
EVE (PREFERRED)
---- SAWING 9:00 AM - 4:00 PM
IVAN (PREFERRED)
---- PAINTING 9:45 AM - 4:45 PM
FRANK (PREFERRED)
---- PAINTING 11:00 AM - 6:00 PM
HEIDI (PREFERRED)
---- SAWING 12:00 PM - 7:00 PM
ALICE (PREFERRED)
---- PAINTING 12:15 PM - 7:15 PM
CAROL (PREFERRED)
MONDAY
---- SAWING 9:00 AM - 4:00 PM
DAVID (PREFERRED)
---- PAINTING 9:45 AM - 4:45 PM
IVAN (PREFERRED)
---- PAINTING 12:00 PM - 7:00 PM
GRACE (PREFERRED)
---- SAWING 2:00 PM - 9:00 PM
ALICE (PREFERRED)
---- PAINTING 2:00 PM - 9:00 PM
HEIDI (NEUTRAL)
TUESDAY
---- SAWING 9:00 AM - 4:00 PM
DAVID (PREFERRED)
---- PAINTING 9:45 AM - 4:45 PM
EVE (PREFERRED)
---- PAINTING 12:00 PM - 7:00 PM
FRANK (NEUTRAL)
---- SAWING 2:00 PM - 9:00 PM
BOB (PREFERRED)
---- PAINTING 2:00 PM - 9:00 PM
HEIDI (NEUTRAL)
WEDNESDAY
---- SAWING 7:30 AM - 2:30 PM
DAVID (PREFERRED)
---- PAINTING 7:30 AM - 2:30 PM
IVAN (PREFERRED)
---- SANDING 9:45 AM - 4:45 PM
FRANK (PREFERRED)
---- PAINTING 12:00 PM - 7:00 PM
JUDY (PREFERRED)
---- SAWING 2:00 PM - 9:00 PM
BOB (PREFERRED)
---- PAINTING 2:00 PM - 9:00 PM
ALICE (NEUTRAL)
THURSDAY
---- SANDING 9:00 AM - 4:00 PM
GRACE (PREFERRED)
---- SAWING 9:00 AM - 4:00 PM
CAROL (PREFERRED)
---- PAINTING 9:45 AM - 4:45 PM
EVE (PREFERRED)
---- PAINTING 12:00 PM - 7:00 PM
JUDY (PREFERRED)
---- SAWING 2:00 PM - 9:00 PM
BOB (PREFERRED)
---- PAINTING 2:00 PM - 9:00 PM
ALICE (NEUTRAL)
FRIDAY
---- SAWING 9:00 AM - 4:00 PM
DAVID (PREFERRED)
---- PAINTING 9:45 AM - 4:45 PM
FRANK (PREFERRED)
---- PAINTING 12:00 PM - 7:00 PM
GRACE (NEUTRAL)
---- SAWING 2:00 PM - 9:00 PM
BOB (PREFERRED)
---- PAINTING 2:00 PM - 9:00 PM
HEIDI (NEUTRAL)
SATURDAY
---- SAWING 7:30 AM - 2:30 PM
CAROL (PREFERRED)
---- PAINTING 7:30 AM - 2:30 PM
IVAN (PREFERRED)
---- SANDING 9:45 AM - 4:45 PM
DAVID (PREFERRED)
---- PAINTING 12:00 PM - 7:00 PM
FRANK (NEUTRAL)
---- SAWING 2:00 PM - 9:00 PM
ALICE (PREFERRED)
---- PAINTING 2:00 PM - 9:00 PM
BOB (NEUTRAL)
SUMMARY
-- ALICE
benefit: 13.000
3 PREFERRED
2 NEUTRAL
0 DISFAVORED
-- BOB
benefit: 14.000
4 PREFERRED
1 NEUTRAL
0 DISFAVORED
-- CAROL
benefit: 9.000
3 PREFERRED
0 NEUTRAL
0 DISFAVORED
-- DAVID
benefit: 15.000
5 PREFERRED
0 NEUTRAL
0 DISFAVORED
-- EVE
benefit: 9.000
3 PREFERRED
0 NEUTRAL
0 DISFAVORED
-- FRANK
benefit: 13.000
3 PREFERRED
2 NEUTRAL
0 DISFAVORED
-- GRACE
benefit: 8.000
2 PREFERRED
1 NEUTRAL
0 DISFAVORED
-- HEIDI
benefit: 9.000
1 PREFERRED
3 NEUTRAL
0 DISFAVORED
-- IVAN
benefit: 12.000
4 PREFERRED
0 NEUTRAL
0 DISFAVORED
-- JUDY
benefit: 6.000
2 PREFERRED
0 NEUTRAL
0 DISFAVORED
So we have a solution! We have 108 total benefit points. But it looks a bit
uneven: Judy only works 2 days, while some people work many more: David works 5
for instance. Why is that? I update the program with =human_annotate = 'JUDY'=,
run it again, and it tells me more about Judy's preferences:
Objective value: 108.00000000
Enumerated nodes: 0
Total iterations: 0
Time (CPU seconds): 0.01
Time (Wallclock seconds): 0.01
Option for printingOptions changed from normal to all
Total time (CPU seconds): 0.01 (Wallclock seconds): 0.02
SUNDAY (JUDY OFF)
---- SANDING 9:00 AM - 4:00 PM (JUDY NEUTRAL)
EVE (PREFERRED)
---- SAWING 9:00 AM - 4:00 PM (JUDY PREFERRED)
IVAN (PREFERRED)
---- PAINTING 9:45 AM - 4:45 PM (JUDY PREFERRED)
FRANK (PREFERRED)
---- PAINTING 11:00 AM - 6:00 PM (JUDY NEUTRAL)
HEIDI (PREFERRED)
---- SAWING 12:00 PM - 7:00 PM (JUDY PREFERRED)
ALICE (PREFERRED)
---- PAINTING 12:15 PM - 7:15 PM (JUDY NEUTRAL)
CAROL (PREFERRED)
MONDAY (JUDY OFF)
---- SAWING 9:00 AM - 4:00 PM (JUDY PREFERRED)
DAVID (PREFERRED)
---- PAINTING 9:45 AM - 4:45 PM (JUDY NEUTRAL)
IVAN (PREFERRED)
---- PAINTING 12:00 PM - 7:00 PM (JUDY NEUTRAL)
GRACE (PREFERRED)
---- SAWING 2:00 PM - 9:00 PM (JUDY DISFAVORED)
ALICE (PREFERRED)
---- PAINTING 2:00 PM - 9:00 PM (JUDY DISFAVORED)
HEIDI (NEUTRAL)
TUESDAY (JUDY OFF)
---- SAWING 9:00 AM - 4:00 PM (JUDY PREFERRED)
DAVID (PREFERRED)
---- PAINTING 9:45 AM - 4:45 PM (JUDY PREFERRED)
EVE (PREFERRED)
---- PAINTING 12:00 PM - 7:00 PM (JUDY REFUSED)
FRANK (NEUTRAL)
---- SAWING 2:00 PM - 9:00 PM (JUDY REFUSED)
BOB (PREFERRED)
---- PAINTING 2:00 PM - 9:00 PM (JUDY REFUSED)
HEIDI (NEUTRAL)
WEDNESDAY (JUDY SCHEDULED)
---- SAWING 7:30 AM - 2:30 PM (JUDY REFUSED)
DAVID (PREFERRED)
---- PAINTING 7:30 AM - 2:30 PM (JUDY REFUSED)
IVAN (PREFERRED)
---- SANDING 9:45 AM - 4:45 PM (JUDY NEUTRAL)
FRANK (PREFERRED)
---- PAINTING 12:00 PM - 7:00 PM (JUDY PREFERRED)
JUDY (PREFERRED)
---- SAWING 2:00 PM - 9:00 PM (JUDY DISFAVORED)
BOB (PREFERRED)
---- PAINTING 2:00 PM - 9:00 PM (JUDY DISFAVORED)
ALICE (NEUTRAL)
THURSDAY (JUDY SCHEDULED)
---- SANDING 9:00 AM - 4:00 PM (JUDY PREFERRED)
GRACE (PREFERRED)
---- SAWING 9:00 AM - 4:00 PM (JUDY PREFERRED)
CAROL (PREFERRED)
---- PAINTING 9:45 AM - 4:45 PM (JUDY PREFERRED)
EVE (PREFERRED)
---- PAINTING 12:00 PM - 7:00 PM (JUDY PREFERRED)
JUDY (PREFERRED)
---- SAWING 2:00 PM - 9:00 PM (JUDY DISFAVORED)
BOB (PREFERRED)
---- PAINTING 2:00 PM - 9:00 PM (JUDY DISFAVORED)
ALICE (NEUTRAL)
FRIDAY (JUDY OFF)
---- SAWING 9:00 AM - 4:00 PM (JUDY DISFAVORED)
DAVID (PREFERRED)
---- PAINTING 9:45 AM - 4:45 PM (JUDY DISFAVORED)
FRANK (PREFERRED)
---- PAINTING 12:00 PM - 7:00 PM (JUDY DISFAVORED)
GRACE (NEUTRAL)
---- SAWING 2:00 PM - 9:00 PM (JUDY REFUSED)
BOB (PREFERRED)
---- PAINTING 2:00 PM - 9:00 PM (JUDY REFUSED)
HEIDI (NEUTRAL)
SATURDAY (JUDY OFF)
---- SAWING 7:30 AM - 2:30 PM (JUDY REFUSED)
CAROL (PREFERRED)
---- PAINTING 7:30 AM - 2:30 PM (JUDY REFUSED)
IVAN (PREFERRED)
---- SANDING 9:45 AM - 4:45 PM (JUDY REFUSED)
DAVID (PREFERRED)
---- PAINTING 12:00 PM - 7:00 PM (JUDY DISFAVORED)
FRANK (NEUTRAL)
---- SAWING 2:00 PM - 9:00 PM (JUDY DISFAVORED)
ALICE (PREFERRED)
---- PAINTING 2:00 PM - 9:00 PM (JUDY DISFAVORED)
BOB (NEUTRAL)
SUMMARY
-- ALICE
benefit: 13.000
3 PREFERRED
2 NEUTRAL
0 DISFAVORED
-- BOB
benefit: 14.000
4 PREFERRED
1 NEUTRAL
0 DISFAVORED
-- CAROL
benefit: 9.000
3 PREFERRED
0 NEUTRAL
0 DISFAVORED
-- DAVID
benefit: 15.000
5 PREFERRED
0 NEUTRAL
0 DISFAVORED
-- EVE
benefit: 9.000
3 PREFERRED
0 NEUTRAL
0 DISFAVORED
-- FRANK
benefit: 13.000
3 PREFERRED
2 NEUTRAL
0 DISFAVORED
-- GRACE
benefit: 8.000
2 PREFERRED
1 NEUTRAL
0 DISFAVORED
-- HEIDI
benefit: 9.000
1 PREFERRED
3 NEUTRAL
0 DISFAVORED
-- IVAN
benefit: 12.000
4 PREFERRED
0 NEUTRAL
0 DISFAVORED
-- JUDY
benefit: 6.000
2 PREFERRED
0 NEUTRAL
0 DISFAVORED
This tells us that on Monday Judy does not work, although she marked the SAWING
shift as PREFERRED. Instead David got that shift. What would happen if David
gave that shift to Judy? He would lose 3 points, she would gain 3 points, and
the total would remain exactly the same at 108.
How would we favor a more even distribution? We need some sort of tie-break. I
want to add a nonlinearity to strongly disfavor people getting a low number of
shifts. But PuLP is very explicitly a linear programming solver, and cannot
solve nonlinear problems. Here we can get around this by enumerating each
specific case, and assigning it a nonlinear benefit function. The most obvious
approach is to define another set of boolean variables:
vars_Nshifts[human][N]. And then using them to add extra benefit terms, with
values nonlinearly related to Nshifts. Something like this:
benefit_boost_Nshifts = \ 2: -0.8, 3: -0.5, 4: -0.3, 5: -0.2 for h in humans: benefits[h] = \ ... + \ pulp.lpSum([vars_Nshifts[h][n] * benefit_boost_Nshifts[n] \ for n in benefit_boost_Nshifts.keys()])
vars_Nshifts[human][N] variables
to function as intended: they need to be binary indicators of whether a specific
person has that many shifts or not. That would need to be implemented with
constraints. Let's plot it like this:
#!/usr/bin/python3 import numpy as np import gnuplotlib as gp Nshifts_eq = 4 Nshifts_max = 10 Nshifts = np.arange(Nshifts_max+1) i0 = np.nonzero(Nshifts != Nshifts_eq)[0] i1 = np.nonzero(Nshifts == Nshifts_eq)[0] gp.plot( # True value: var_Nshifts4==0, Nshifts!=4 ( np.zeros(i0.shape), Nshifts[i0], dict(_with = 'points pt 7 ps 1 lc "red"') ), # True value: var_Nshifts4==1, Nshifts==4 ( np.ones(i1.shape), Nshifts[i1], dict(_with = 'points pt 7 ps 1 lc "red"') ), # False value: var_Nshifts4==1, Nshifts!=4 ( np.ones(i0.shape), Nshifts[i0], dict(_with = 'points pt 7 ps 1 lc "black"') ), # False value: var_Nshifts4==0, Nshifts==4 ( np.zeros(i1.shape), Nshifts[i1], dict(_with = 'points pt 7 ps 1 lc "black"') ), unset=('grid'), _set = (f'xtics ("(Nshifts== Nshifts_eq ) == 0" 0, "(Nshifts== Nshifts_eq ) == 1" 1)'), _xrange = (-0.1, 1.1), ylabel = "Nshifts", title = "Nshifts equality variable: not linearly separable", hardcopy = "/tmp/scheduling-Nshifts-eq.svg")
vars_Nshifts[h][4] variable (plotted on the x axis of this
plot) would need to be defined by a set of linear AND constraints to linearly
separate the true (red) values of this variable from the false (black) values.
As can be seen in this plot, this isn't possible. So this representation does
not work.
How do we fix it? We can use inequality variables instead. I define a different
set of variables vars_Nshifts_leq[human][N] that are 1 iff Nshifts <= N.
The equality variable from before can be expressed as a difference of these
inequality variables: vars_Nshifts[human][N] =
vars_Nshifts_leq[human][N]-vars_Nshifts_leq[human][N-1]
Can these vars_Nshifts_leq variables be defined by a set of linear AND
constraints? Yes:
#!/usr/bin/python3 import numpy as np import numpysane as nps import gnuplotlib as gp Nshifts_leq = 4 Nshifts_max = 10 Nshifts = np.arange(Nshifts_max+1) i0 = np.nonzero(Nshifts > Nshifts_leq)[0] i1 = np.nonzero(Nshifts <= Nshifts_leq)[0] def linear_slope_yintercept(xy0,xy1): m = (xy1[1] - xy0[1])/(xy1[0] - xy0[0]) b = xy1[1] - m * xy1[0] return np.array(( m, b )) x01 = np.arange(2) x01_one = nps.glue( nps.transpose(x01), np.ones((2,1)), axis=-1) y_lowerbound = nps.inner(x01_one, linear_slope_yintercept( np.array((0, Nshifts_leq+1)), np.array((1, 0)) )) y_upperbound = nps.inner(x01_one, linear_slope_yintercept( np.array((0, Nshifts_max)), np.array((1, Nshifts_leq)) )) y_lowerbound_check = (1-x01) * (Nshifts_leq+1) y_upperbound_check = Nshifts_max - x01*(Nshifts_max-Nshifts_leq) gp.plot( # True value: var_Nshifts_leq4==0, Nshifts>4 ( np.zeros(i0.shape), Nshifts[i0], dict(_with = 'points pt 7 ps 1 lc "red"') ), # True value: var_Nshifts_leq4==1, Nshifts<=4 ( np.ones(i1.shape), Nshifts[i1], dict(_with = 'points pt 7 ps 1 lc "red"') ), # False value: var_Nshifts_leq4==1, Nshifts>4 ( np.ones(i0.shape), Nshifts[i0], dict(_with = 'points pt 7 ps 1 lc "black"') ), # False value: var_Nshifts_leq4==0, Nshifts<=4 ( np.zeros(i1.shape), Nshifts[i1], dict(_with = 'points pt 7 ps 1 lc "black"') ), ( x01, y_lowerbound, y_upperbound, dict( _with = 'filledcurves lc "green"', tuplesize = 3) ), ( x01, nps.cat(y_lowerbound_check, y_upperbound_check), dict( _with = 'lines lc "green" lw 2', tuplesize = 2) ), unset=('grid'), _set = (f'xtics ("(Nshifts<= Nshifts_leq ) == 0" 0, "(Nshifts<= Nshifts_leq ) == 1" 1)', 'style fill transparent pattern 1'), _xrange = (-0.1, 1.1), ylabel = "Nshifts", title = "Nshifts inequality variable: linearly separable", hardcopy = "/tmp/scheduling-Nshifts-leq.svg")
# I want to favor people getting more extra shifts at the start to balance # things out: somebody getting one more shift on their pile shouldn't take # shifts away from under-utilized people benefit_boost_leq_bound = \ 2: .2, 3: .3, 4: .4, 5: .5 # Constrain vars_Nshifts_leq variables to do the right thing for h in humans: for b in benefit_boost_leq_bound.keys(): prob += (pulp.lpSum([vars[h][s] for s in shifts.keys()]) >= (1 - vars_Nshifts_leq[h][b])*(b+1), f" h at least b shifts: lower bound") prob += (pulp.lpSum([vars[h][s] for s in shifts.keys()]) <= Nshifts_max - vars_Nshifts_leq[h][b]*(Nshifts_max-b), f" h at least b shifts: upper bound") benefits = dict() for h in humans: benefits[h] = \ ... + \ pulp.lpSum([vars_Nshifts_leq[h][b] * benefit_boost_leq_bound[b] \ for b in benefit_boost_leq_bound.keys()])
Welcome to post 46 in the $R^4 series!
r2u, introduced less
than three years ago in post
#37, has become a runaway success. When I last tabulated downloads
in early January, we were already at 33 million downloads of binary CRAN packages across the three
Ubuntu LTS releases we support. These were exclusively for the amd64
platform of standard (Intel or AMD made) x64_64 cpus. Now we are happy
to announce that arm64 support has been added and is available!
Rblpapi among them) but
none exist for arm64. Such is life. We will try to fix builds as time
permits and report build issues to the respective upstream repos. Help
in that endeavour would be most welcome.
But all the big and slow compiles one may care about (hello
duckdb, hello arrow, ) are there. Which is
pretty exciting!
ubuntu-24.04-arm as the
platform, and use the r-ci or r2u-setup
actiions. A first test yaml exists
worked (though this last version had the arm64 runner commented out
again). (And no, arm64 was not faster than amd64. More tests
needed.)
For simple tests, Docker. The rocker/r-ubuntu:24.04
container exists for arm64 (see here), and one
can add r2u support as is done in
this Dockerfile which is used by the builds and available as
eddelbuettel/r2u_build:noble. I will add the standard
rocker/r2u:24.04 container (or equally
rocker/r2u:noble) in a day or two, I had not realised I
wasn t making them for arm64.
One a real machine such as a cloud instance or a proper instance,
just use the standard r2u script for noble aka 24.04
available here.
The key lines are the two lines
echo "deb [arch=amd64,arm64] https://r2u.stat.illinois.edu/ubuntu noble main" \
> /etc/apt/sources.list.d/cranapt.list
# ...
echo "deb [arch=amd64,arm64] https://cloud.r-project.org/bin/linux/ubuntu noble-cran40/" \
> /etc/apt/sources.list.d/cran_r.listapt entry and which are now arm64-aware.
After that, apt works as usual, and of course r2u works as
usual thanks also to bspm so you can just do, say
and enjoy the binaries rolling in. So give it a whirl if you have
access to such hardware. We look forward to feedback, suggestes, feature
requests or bug reports. Let us know how it goes!
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.
In today s digital landscape, social media is more than just a communication tool it is the primary medium for global discourse. Heads of state, corporate leaders and cultural influencers now broadcast their statements directly to the world, shaping public opinion in real time. However, the dominance of a few centralized platforms X/Twitter, Facebook and YouTube raises critical concerns about control, censorship and the monopolization of information. Those who control these networks effectively wield significant power over public discourse.
In response, a new wave of distributed social media platforms has emerged, each built on different decentralized protocols designed to provide greater autonomy, censorship resistance and user control. While Wikipedia maintains a comprehensive list of distributed social networking software and protocols, it does not cover recent blockchain-based systems, nor does it highlight which have the most potential for mainstream adoption.
This post explores the leading decentralized social media platforms and the protocols they are based on: Mastodon (ActivityPub), Bluesky (AT Protocol), Warpcast (Farcaster), Hey (Lens) and Primal (Nostr).
| Protocol | Identity System | Example | Storage model | Cost for end users | Potential |
|---|---|---|---|---|---|
| Mastodon | Tied to server domain | @ottok@mastodon.social |
Federated instances | Free (some instances charge) | High |
| Bluesky | Portable (DID) | ottoke.bsky.social |
Federated instances | Free | Moderate |
| Farcaster | ENS (Ethereum) | @ottok |
Blockchain + off-chain | Small gas fees | Moderate |
| Lens | NFT-based (Polygon) | @ottok |
Blockchain + off-chain | Small gas fees | Niche |
| Nostr | Cryptographic Keys | npub16lc6uhqpg6dnqajylkhwuh3j7ynhcnje508tt4v6703w9kjlv9vqzz4z7f |
Federated instances | Free (some instances charge) | Niche |
Mastodon was created in 2016 by Eugen Rochko, a German software developer who sought to provide a decentralized and user-controlled alternative to Twitter. It was built on the ActivityPub protocol, now standardized by W3C Social Web Working Group, to allow users to join independent servers while still communicating across the broader Mastodon network.
Mastodon operates on a federated model, where multiple independently run servers communicate via ActivityPub. Each server sets its own moderation policies, leading to a decentralized but fragmented experience. The servers can alternatively be called instances, relays or nodes, depending on what vocabulary a protocol has standardized on.
@username@instance.tld.
"@context": "https://www.w3.org/ns/activitystreams",
"type": "Create",
"actor": "https://mastodon.social/users/ottok",
"object":
"type": "Note",
"content": "Hello from #Mastodon!",
"published": "2025-03-03T12:00:00Z",
"to": ["https://www.w3.org/ns/activitystreams#Public"]
Interestingly, Bluesky was conceived within Twitter in 2019 by Twitter founder Jack Dorsey. After being incubated as a Twitter-funded project, it spun off as an independent Public Benefit LLC in February 2022 and launched its public beta in February 2023.
Bluesky runs on top of the Authenticated Transfer (AT) Protocol published at https://github.com/bluesky-social/atproto. The protocol enables portable identities and data ownership, meaning users can migrate between platforms while keeping their identity and content intact. In practice, however, there is only one popular server at the moment, which is Bluesky itself.
@user.bsky.social).
"repo": "did:plc:ottoke.bsky.social",
"collection": "app.bsky.feed.post",
"record":
"$type": "app.bsky.feed.post",
"text": "Hello from Bluesky!",
"createdAt": "2025-03-03T12:00:00Z",
"langs": ["en"]
"fid": 766579,
"username": "ottok",
"custodyAddress": "0x127853e48be3870172baa4215d63b6d815d18f21",
"connectedWallet": "0x3ebe43aa3ae5b891ca1577d9c49563c0cee8da88",
"text": "Hello from Farcaster!",
"publishedAt": 1709424000,
"replyTo": null,
"embeds": []
"profileId": "@ottok",
"contentURI": "ar://QmExampleHash",
"collectModule": "0x23b9467334bEb345aAa6fd1545538F3d54436e96",
"referenceModule": "0x0000000000000000000000000000000000000000",
"timestamp": 1709558400
npub...).
"id": "note1xyz...",
"pubkey": "npub1...",
"kind": 1,
"content": "Hello from Nostr!",
"created_at": 1709558400,
"tags": [],
"sig": "sig1..."
| Platform | Total Accounts | Active Users | Growth Trend |
|---|---|---|---|
| Mastodon | ~10 million | ~1 million | Steady |
| Bluesky | ~33 million | ~1 million | Steady |
| Nostr | ~41 million | ~20 thousand | Steady |
| Farcaster | ~850 thousand | ~50 thousand | Flat |
| Lens | ~140 thousand | ~20 thousand | Flat |
Next.