Search Results: "edd"

3 November 2025

Dirk Eddelbuettel: duckdb-mlpack 0.0.3: macOS binaries, unit tests, more outputs

A littler two weeks a short post announced the duckdb-mlpack as ML quacks : combining the powerful C++ machine learning library mlpack with the amazing analytical database engine duckdb. About a week ago another short post covered first extensions. We actually followed-up with release 0.0.3 days later, and never posted about it so this short note catches up. In release 0.0.3, we provide macOS binaries: following a known issue with one of the components, we apply a simple patch to enable the build. Next up are wasm and windows, if you know your way around these platforms please get in touch. Release 0.0.3 also added first unit tests, serializes the coefficients from the (regularized) linear regression into the output table. See see two previous posts linked above for details and background, the repo for code, issues and more, and the extension page for more about this duckdb community extension.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

27 October 2025

Dirk Eddelbuettel: #054: Faster r-ci Continuous Integration via r2u Container

Welcome to post 54 in the R4 series. The topic of continuous integration has been a recurrent theme here at the R4 series. Post #32 introducess r-ci, while post #41 brings r2u to r-ci, but does not show a matrix deployment. Post #45 describes the updated r-ci setup that is now the default and contains a macOS and Ubuntu matrix, where the latter relies on r2u to keep things fast, easy, reliable . Last but not least more recent post #52 shares a trick for ensuring coverage reports. Following #45, use of r-ci at for example GitHub Actions has seen steady use and very reliable performance. With the standard setup, a vanilla Ubuntu setup is changed into one supported by r2u. This requires downloading and installating a few Ubuntu packages, and has generally been fairly quick on the order of around fourty seconds. Now, the general variability of run-times for identical tasks in GitHub Actions is well documented by the results of the setup described in post #39 which still runs weekly. It runs the identical SQL query against a remote backend using two different package families. And lo and behold, the intra-method variability on unchanged code or setup and therefore due solely to system variability is about as large as the inter-method variability. In short, GitHub Actions performance varies randomly with significant variability. See the repo README.md for chart that updates weekly (and see #39 for background). Of late, this variability became more noticeable during standard GitHub Actions runs where it would regularly take more than two minutes of setup time before actual continuous integration work was done. Some caching seems to be in effect, so subsequent runs in the same repo seem faster and often came back to one minute or less. For lightweight and small packages, loosing two minutes to setup when the actual test time is a minute or less gets old fast. Looking around, we noticed that container use can be combined with matrix use. So we have now been deploying the following setup (not always over all the matrix elements though)
jobs:
  ci:
    strategy:
      matrix:
        include:
          -   name: container, os: ubuntu-latest, container: rocker/r2u4ci  
          -   name: macos,     os: macos-latest   
          -   name: ubuntu,    os: ubuntu-latest  

    runs-on: $  matrix.os  
    container: $  matrix.container  
GitHub Actions is smart enough to provide NULL for container in the two other cases, so container: $ matrix.container is ignored there. But when container is set as here for the ci-enhanced version of r2u (which adds a few binaries commonly needed such as git, curl, wget etc needed for CI) then the CI jobs runs inside the container. And thereby skips most of the setup time as the container is already prepared. This also required some small adjustments in the underlying shell script doing the work. To not disrupt standard deployment, we placed these into a release candidate / development version one can op into via an new variable dev_version
      - name: Setup
        uses: eddelbuettel/github-actions/r-ci@master
        with:
          dev_version: 'TRUE'
Everything else remains the same and works as before. But faster as much less time is spent on setup. You can see the actual full yaml file and actions in my repositories for rcpparmadillo and rcppmlpack-examples. Additional testing would be welcome, so feel free to deploy this in your actions now. Otherwise I will likely carry this over and make it the defaul in a few weeks time. It will still work as before but when the added container: line is used will run much faster thanks to rocker/r2u4ci being already set up for CI.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

26 October 2025

Dirk Eddelbuettel: duckdb-mlpack 0.0.2: mlpack is now a duckdb community extension

A couple of days ago in a short post, I announced duckdb-mlpack as ML quacks : combining the powerful C++ machine learning library mlpack with the amazing analytical database engine duckdb. See that post for more background. The duckdb-mlpack package is now a community extension joining an impressive list of existing extensions. This means duckdb builds and distributes duckdb-mlpack for all supported platforms allowing users to just install the resulting (signed) binary. (We currently only support Linux in both arm64 and amd64, adding macOS should be straightforward once we sort one build issue out. Windows and WASM should work too, with a little love and polish, as both duckdb and mlpack support them.) Given the binary build, a simple
INSTALL mlpack FROM community;
LOAD mlpack;
installs and loads the package. By the duckdb convention the code is stored per-user and per-version, so the first line needs to be executed only once per duckdb release used. The second line is then per session. We also extended the capabilities of duckdb-mlpack. While still a MVP stressing minimal viable product, the two supported methods adaBoost and (regularized) linear regression both serialize and store their model object permitting rapid prediction on new data as shown in the adaBoost example:
-- Perform adaBoost (using weak learner 'Perceptron' by default)
-- Read 'features' into 'X', 'labels' into 'Y', use optional parameters
-- from 'Z', and prepare model storage in 'M'
CREATE TABLE X AS SELECT * FROM read_csv("https://eddelbuettel.github.io/duckdb-mlpack/data/iris.csv");
CREATE TABLE Y AS SELECT * FROM read_csv("https://eddelbuettel.github.io/duckdb-mlpack/data/iris_labels.csv");
CREATE TABLE Z (name VARCHAR, value VARCHAR);
INSERT INTO Z VALUES ('iterations', '50'), ('tolerance', '1e-7');
CREATE TABLE M (json VARCHAR);

-- Train model for 'Y' on 'X' using parameters 'Z', store in 'M'
CREATE TEMP TABLE A AS SELECT * FROM mlpack_adaboost("X", "Y", "Z", "M");

-- Count by predicted group
SELECT COUNT(*) as n, predicted FROM A GROUP BY predicted;

-- Model 'M' can be used to predict
CREATE TABLE N (x1 DOUBLE, x2 DOUBLE, x3 DOUBLE, x4 DOUBLE);
-- inserting approximate column mean values
INSERT INTO N VALUES (5.843, 3.054, 3.759, 1.199);
-- inserting approximate column mean values, min values, max values
INSERT INTO N VALUES (5.843, 3.054, 3.759, 1.199), (4.3, 2.0, 1.0, 0.1), (7.9, 4.4, 6.9, 2.5);
-- and this predict one element each
SELECT * FROM mlpack_adaboost_pred("N", "M");
Ryan and I have some ideas for where to go from here, ideally towards autogenerating bindings for most (if not all) methods as is done for the mlpack language bindings. Anybody interested and willing to help should reach out to us.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

20 October 2025

Dirk Eddelbuettel: RcppArmadillo 15.2.0-0 on GitHub: New Upstream, Simpler OpenMP

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra and scientific computing. It aims towards a good balance between speed and ease of use, has a syntax deliberately close to Matlab, and is useful for algorithm development directly in C++, or quick conversion of research code into production environments. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 1270 other packages on CRAN, downloaded 42 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 650 times according to Google Scholar. This versions updates to the 15.2.0 upstream release made today. It brings a few changes over Armadillo 15.0 (see below for more). It follows the most recent RcppArmadillo 15.0.2-2 release and the Armadillo 15 upstream transition with its dual focus on moving on from C++11 and deprecation of a number of API access points. As we had a few releases last month to manage the transition, we will sit this upgrade out and not upload to CRAN in order to normalize our update cadence towards the desired about six in six months (that the CRAN Policy asks for). One can of course install as usual directly from the GitHub repository as well as from r-universe which also offers binaries for all CRAN platforms. The transition to Armadillo 15 appears to be going slowly but steadily. We had well over 300 packages with either a need to relax the C++11 setting and/or update away from now-deprecated API access points. That number has been cut in half thanks to a lot of work from a lot of package maintainers which is really appreciated! Of course, a lot remains to be done. Issues #489 and #491 contain the over sixty PRs and patches I prepared for all packages with at least one reverse dependency. Most (but not all) have aided in CRAN updates, some packages are still outstanding in terms of updates. As before meta-issue #475 regroups all the resources for the transition. If you, dear reader, have a package that is affected and I could be of assistance please do reach out. The other change we made is to greatly simplify the detection and setup of OpenMP. As before, we rely on configure to attempt compilation of a minimal OpenMP-using program in order to pass the success or failure onto Armadillo as a can-or-cannot use OpenMP. In the year 2025 one of the leading consumer brands still cannot ship an OS where this works out of the box, so we try to aide there. For all others systems, R actually covers this pretty well and has a reliable configuration variable that we rely upon. Just as we recommend for downstream users of the package. This setup should be robust, but is a change so by all means if you knowingly rely on OpenMP please test and report back. The detailed changes since the last CRAN release follow.

Changes in RcppArmadillo version 15.2.0-0 (2025-10-20) (GitHub Only)
  • Upgraded to Armadillo release 15.2.0 (Medium Roast Deluxe)
    • Added rande() for generating matrices with elements from exponential distributions
    • shift() has been deprecated in favour of circshift(), for consistency with Matlab/Octave
    • Reworked detection of aliasing, leading to more efficient compiled code
  • OpenMP detection in configure has been simplified

More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

17 October 2025

Dirk Eddelbuettel: ML quacks: Combining duckdb and mlpack

A side project I have been working on a little since last winter and which explores extending duckdb with mlpack is now public at the duckdb-mlpack repo. duckdb is an excellent small (as in runs as a self-contained binary ) database engine with both a focus on analytical payloads (OLAP rather than OLTP) and an impressive number of already bolted-on extensions (for example for cloud data access) delivered as a single-build C++ executable (or of course as a library used from other front-ends). mlpack is an excellent C++ library containing many/most machine learning algorithms, also built in a self-contained manner (or library) making it possible to build compact yet powerful binaries, or to embed (as opposed to other ML framework accessed from powerful but not lightweight run-times such as Python or R). The compact build aspect as well as the common build tools (C++, cmake) make these two a natural candidate for combining them. Moreover, duckdb is a champion of data access, management and control and the complementary machine learning insights and predictions offered by mlpack are fully complementary and hence fit this rather well. duckdb also has a very robust and active extension system. To use it, one starts from a template repository and its use this template button, runs a script and can then start experimenting. I have now grouped my initial start and test functions into a separate repository duckdb-example-extension to keep the duckdb-mlpack one focused on the extend to mlpack aspect. duckdb-mlpack is right an MVP , i.e. a minimally viable product (or demo). It just runs the adaboost classifier but does so on any dataset fitting the rectangular setup with columns of features (real valued) and a final column (integer valued) of labels. I had hope to use two select queries for both features and then labels but it turns a table function (returning a table of data from a query) can only run one select *. So the basic demo, also on the repo README is now to run the following script (where the SELECT * FROM mlpack_adaboost((SELECT * FROM D)); is the key invocation of the added functionality):
#!/bin/bash

cat <<EOF   build/release/duckdb
SET autoinstall_known_extensions=1;
SET autoload_known_extensions=1; # for httpfs

CREATE TEMP TABLE Xd AS SELECT * FROM read_csv("https://mlpack.org/datasets/iris.csv");
CREATE TEMP TABLE X AS SELECT row_number() OVER () AS id, * FROM Xd;
CREATE TEMP TABLE Yd AS SELECT * FROM read_csv("https://mlpack.org/datasets/iris_labels.csv");
CREATE TEMP TABLE Y AS SELECT row_number() OVER () AS id, CAST(column0 AS double) as label FROM Yd;
CREATE TEMP TABLE D AS SELECT * FROM X INNER JOIN Y ON X.id = Y.id;
ALTER TABLE D DROP id;
ALTER TABLE D DROP id_1;
CREATE TEMP TABLE A AS SELECT * FROM mlpack_adaboost((SELECT * FROM D));

SELECT COUNT(*) as n, predicted FROM A GROUP BY predicted;
EOF
to produce the following tabulation / group by:
./sampleCallRemote.sh
Misclassified: 1
 
    n     predicted  
  int64     int32    
 
     50           0  
     49           1  
     51           2  
 
$
(Note that this requires the httpfs extension. So when you build from a freshly created extension repository you may be ahead of the most recent release of duckdb by a few commits. It is easy to check out the most recent release tag (or maybe the one you are running for your local duckdb binary) to take advantage of the extensions you likely already have for that version. So here, and in the middle of October 2025, I picked v1.4.1 as I run duckdb version 1.4.1 on my box.) There are many other neat duckdb extensions. The core ones are regrouped here while a list of community extensions is here and here. For this (still more minimal) extension, I added a few TODO items to the README.md: Please reach out if you are interested in working on any of this.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

14 October 2025

Dirk Eddelbuettel: qlcal 0.0.17 on CRAN: Regular Update

The seventeenth release of the qlcal package arrivied at CRAN today, once again following a QuantLib release as 1.40 came out this morning. qlcal delivers the calendaring parts of QuantLib. It is provided (for the R package) as a set of included files, so the package is self-contained and does not depend on an external QuantLib library (which can be demanding to build). qlcal covers over sixty country / market calendars and can compute holiday lists, its complement (i.e. business day lists) and much more. Examples are in the README at the repository, the package page, and course at the CRAN package page. This releases mainly synchronizes qlcal with the QuantLib release 1.40. Only one country calendar got updated; the diffstat looks larger as the URL part of the copyright got updated throughout. We also updated the URL for the GPL-2 badge: when CRAN checks this, they always hit a timeout as the FSF server possibly keeps track of incoming requests; we now link to version from the R Licenses page to avoid this.

Changes in version 0.0.17 (2025-07-14)
  • Synchronized with QuantLib 1.40 released today
  • Calendar updates for Singapore
  • URL update in README.md

Courtesy of my CRANberries, there is a diffstat report for this release. See the project page and package documentation for more details, and more examples.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

12 October 2025

Dirk Eddelbuettel: RcppSpdlog 0.0.23 on CRAN: New Upstream

Version 0.0.23 of RcppSpdlog arrived on CRAN today (after a slight delay) and has been uploaded to Debian. RcppSpdlog bundles spdlog, a wonderful header-only C++ logging library with all the bells and whistles you would want that was written by Gabi Melman, and also includes fmt by Victor Zverovich. You can learn more at the nice package documention site. This release updates the code to the version 1.16.0 of spdlog which was released yesterday morning, and includes version 1.12.0 of fmt. We also converted the documentation site to now using mkdocs-material to altdoc (plus local style and production tweaks) rather than directly. I updated the package yesterday morning when spdlog was updated. But the passage was delayed for a day at CRAN as their machines still times out hitting the GPL-2 URL from the README.md badge, leading to a human to manually check the log assert the nothingburgerness of it. This timeout does not happen to me locally using the corresponding URL checker package. I pondered this in a r-package-devel thread and may just have to switch to using the R Project URL for the GPL-2 as this is in fact recurrning. The NEWS entry for this release follows.

Changes in RcppSpdlog version 0.0.23 (2025-10-11)
  • Upgraded to upstream release spdlog 1.16.0 (including fmt 12.0)
  • The mkdocs-material documentation site is now generated via altdoc

Courtesy of my CRANberries, there is also a diffstat report detailing changes. More detailed information is on the RcppSpdlog page, or the package documention site.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

10 October 2025

Reproducible Builds: Reproducible Builds in September 2025

Welcome to the September 2025 report from the Reproducible Builds project! Welcome to the very latest report from the Reproducible Builds project. Our monthly reports outline what we ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. As ever, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website. In this report:

  1. Reproducible Builds Summit 2025
  2. Can t we have nice things?
  3. Distribution work
  4. Tool development
  5. Reproducibility testing framework
  6. Upstream patches

Reproducible Builds Summit 2025 Please join us at the upcoming Reproducible Builds Summit, set to take place from October 28th 30th 2025 in Vienna, Austria! We are thrilled to host the eighth edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin, Hamburg and Athens. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving. If you re interesting in joining us this year, please make sure to read the event page which has more details about the event and location. Registration is open until 20th September 2025, and we are very much looking forward to seeing many readers of these reports there!

Can t we have nice things? Debian Developer Gunnar Wolf blogged that George V. Neville-Neil s Kode Vicious column in Communications of the ACM in which reproducible builds is mentioned without needing to introduce it (assuming familiarity across the computing industry and academia) . Titled, Can t we have nice things?, the article mentions:
Once the proper measurement points are known, we want to constrain the system such that what it does is simple enough to understand and easy to repeat. It is quite telling that the push for software that enables reproducible builds only really took off after an embarrassing widespread security issue ended up affecting the entire Internet. That there had already been 50 years of software development before anyone thought that introducing a few constraints might be a good idea is, well, let s just say it generates many emotions, none of them happy, fuzzy ones. [ ]

Distribution work In Debian this month, Johannes Starosta filed a bug against the debian-repro-status package, reporting that it does not work on Debian trixie. (An upstream bug report was also filed.) Furthermore, 17 reviews of Debian packages were added, 10 were updated and 14 were removed this month adding to our knowledge about identified issues. In March s report, we included the news that Fedora would aim for 99% package reproducibility. This change has now been deferred to Fedora 44 according to Phoronix. Lastly, Bernhard M. Wiedemann posted another openSUSE monthly update for their work there.

Tool development diffoscope version 306 was uploaded to Debian unstable by Chris Lamb. It included contributions already covered in previous months as well as some changes by Zbigniew J drzejewski-Szmek to address issues with the fdtump support [ ] and to move away from the deprecated codes.open method. [ ][ ] strip-nondeterminism version 1.15.0-1 was uploaded to Debian unstable by Chris Lamb. It included a contribution by Matwey Kornilov to add support for inline archive files for Erlang s escript [ ]. kpcyrd has released a new version of rebuilderd. As a quick recap, rebuilderd is an automatic build scheduler that tracks binary packages available in a Linux distribution and attempts to compile the official binary packages from their (purported) source code and dependencies. The code for in-toto attestations has been reworked, and the instances now feature a new endpoint that can be queried to fetch the list of public-keys an instance currently identifies itself by. [ ] Lastly, Holger Levsen bumped the Standards-Version field of disorderfs, with no changes needed. [ ][ ]

Reproducibility testing framework The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In August, however, a number of changes were made by Holger Levsen, including:
  • Setting up six new rebuilderd workers with 16 cores and 16 GB RAM each.
  • reproduce.debian.net-related:
    • Do not expose pending jobs; they are confusing without explaination. [ ]
    • Add a link to v1 API specification. [ ]
    • Drop rebuilderd-worker.conf on a node. [ ]
    • Allow manual scheduling for any architectures. [ ]
    • Update path to trixie graphs. [ ]
    • Use the same rebuilder-debian.sh script for all hosts. [ ]
    • Add all other suites to all other archs. [ ][ ][ ][ ]
    • Update SSH host keys for new hosts. [ ]
    • Move to the pull184 branch. [ ][ ][ ][ ][ ]
    • Only allow 20 GB cache for workers. [ ]
  • OpenWrt-related:
    • Grant developer aparcar full sudo control on the ionos30 node. [ ][ ]
  • Jenkins nodes:
    • Add a number of new nodes. [ ][ ][ ][ ][ ]
    • Dont expect /srv/workspace to exist on OSUOSL nodes. [ ]
    • Stop hardcoding IP addresses in munin.conf. [ ]
    • Add maintenance and health check jobs for new nodes. [ ]
    • Document slight changes in IONOS resources usage. [ ]
  • Misc:
    • Drop disabled Alpine Linux tests for good. [ ]
    • Move Debian live builds and some other Debian builds to the ionos10 node. [ ]
    • Cleanup some legacy support from releases before Debian trixie. [ ]
In addition, Jochen Sprickerhof made the following changes relating to reproduce.debian.net:
  • Do not expose pending jobs on the main site. [ ]
  • Switch the frontpage to reference Debian forky [ ], but do not attempt to build Debian forky on the armel architecture [ ].
  • Use consistent and up to date rebuilder-debian.sh script. [ ]
  • Fix supported worker architectures. [ ]
  • Add a basic excuses page. [ ]
  • Move to the pull184 branch. [ ][ ][ ][ ]
  • Fix a typo in the JavaScript. [ ]
  • Update front page for the new v1 API. [ ][ ]
Lastly, Roland Clobus did some maintenance relating to the reproducibility testing of the Debian Live images. [ ][ ][ ][ ]

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

Dirk Eddelbuettel: RcppArmadillo 15 CRAN Transition: Offering Office Hours

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra and scientific computing. It aims towards a good balance between speed and ease of use, has a syntax deliberately close to Matlab, and is useful for algorithm development directly in C++, or quick conversion of research code into production environments. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 1273 other packages on CRAN, downloaded 41.8 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 651 times according to Google Scholar. Armadillo 15 brought changes. We mentioned these in the 15.0.2-1 and 15.0.2-1 release blog posts: (The second point is a consequence of the first. Prior to C++14, deprecation notes were issue via a macro, and the macro was set up by Conrad in the common way of allowing an override, which we took advantage of in RcppArmadillo effectively shielding downstream package. In C++14 this is now an attribute, and those cannot be suppressed.) We tested this then-upcoming change extensively: Thirteen reverse dependency runs expoloring different settings and leading to the current package setup where an automatic fallback to the last Armadillo 14 release offers fallback for hardwired C++11 use and Armadillo 15 others. Given the 1200+ reverse deoendencies, this took considerable time. All this was also quite extensively discussed with CRAN (especially Kurt Hornik) and documented / controlled via a series of issue tickets starting with overall issue #475 covering the subissues: The sixty pull requests (or emailed patches) followed a suggestion by CRAN to rank-order packages affected by their reverse dependencies sorted in descending package count. Now, while this change from Armadillo 14 to 15 was happening, CRAN also tightened the C++11 requirement for packages and imposed a deadline for changes. In discussion, CRAN also convinced me that a deadline for the deprecation warning, now unmasked, was viable (and is in fairness commensurate with similar, earlier changes triggered by changes in the behaviour of either gcc/g++ or clang/clang++). So we now have two larger deadline campaigns affecting the package (and as always there are some others). These deadlines are coming close: October 17 for the C++11 transition, and October 23 for the deprecation warning. Now, as became clear preparing the sixty pull requests and patches, these changes are often relatively straightforward. For the former, remove the C++11 enforcement and the package will likely build without changes. For the latter, make the often simple (e.g. swith from arma::is_finite to std::isfinite) change. I did not encounter anything much more complicated yet. The number of affected packages approximated by looking at all packages with a reverse dependency on RcppArmadillo and having a deadline can be computed as
suppressMessages(library(data.table))
D <- setDT(tools::CRAN_package_db())
P <- data.table(Package=tools::package_dependencies("RcppArmadillo", reverse=TRUE, db=D)[[1]])
W <- merge(P, D, all.x=TRUE)[is.na(Deadline)==FALSE,c(1:2,38,68)]
W

W[, nrevdep := length(tools::package_dependencies(Package, reverse=TRUE, recursive=TRUE, db=D)[[1]]), by=Package]
W[order(-nrevdep)]
and has been declining steadily from over 350 to now under 200. For that a big and heartfelt Thank You! to all the maintainers who already addressed their package and uploaded updated packages to CRAN. That rocks, and is truly appreciated. Yet the number is still large. And while issues #489 and #491 show a number of pending packages that have merged but not uploaded (yet?) there are also all the other packages I have not been able to look at in detail. While preparing sixty PRs / patches was viable over a period of a good week, I cannot create these for all packages. So with that said, here is a different suggestion for help: All of next week, I will be holding open door open source office hours online two times each day (11:00h to 13:00h Central, 16:00h to 18:00h Central) which can be booked via this booking link for Monday to Friday next week in either fifteen or thirty minutes slots you can book. This should offer Google Meet video conferencing (with jitsi as an alternate, you should be able to control that) which should allow for screen sharing. (I cannot hookup Zoom as my default account has organization settings with a different calendar integration.) If you are reading this and have a package that still needs helps, I hope to see you in the Open Source Office Hours to aid in the RcppArmadillo package updates for your package. Please book a slot!

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

9 October 2025

Dirk Eddelbuettel: xptr 1.2.0 on CRAN: New(ly Adopted) Package!

Excited to share that xptr is back on CRAN! The xptr package helps to create, check, modify, use, share, external pointer objects. External pointers are used quite extensively throughout R to manage external resources such as datanbase connection objects and alike, and can be very useful to pass pointers to just about any C / C++ data structure around. While described in Writing R Extensions (notably Section 5.13), they can be a little bare-bones and so this package can be useful. It had been created by Randy Lai and maintained by him during 2017 to 2020, but then fell off CRAN. In work with nanoarrow and its clean and minimal Arrow interface xptr came in handy so I adopted it. Several extensions and updates have been added: (compiled) function registration, continuous integration, tests, refreshed and extended documentation as well as a print format extension useful for PyCapsule objects when passing via reticulate. The package documentation site was switched to altdoc driving the most excellent Material for MkDocs framework (providing my first test case of altdoc replacing my older local scripts; I should post some more about that ). The first NEWS entry follow.

Changes in version 1.2.0 (2025-10-03)
  • New maintainer
  • Compiled functions are now registered, .Call() adjusted
  • README.md and DESCRIPTION edited and update
  • Simple unit tests and continunous integration have been added
  • The package documentation site has been recreated using altdoc
  • All manual pages for functions now contain \value sections

For more, see the package page, the git repo or the documentation site.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

8 October 2025

Dirk Eddelbuettel: RPushbullet 0.3.5: Mostly Maintenance

RPpushbullet demo A new version 0.3.5 of the RPushbullet package arrived on CRAN. It marks the first release in 4 1/2 years for this mature and feature-stable package. RPushbullet interfaces the neat Pushbullet service for inter-device messaging, communication, and more. It lets you easily send (programmatic) alerts like the one to the left to your browser, phone, tablet, or all at once. This releases reflects mostly internal maintenance and updates to the documentation site, to continuous integration, to package metadata, and one code robustitication. See below for more details.

Changes in version 0.3.5 (2025-10-08)
  • URL and BugReports fields have been added to DESCRIPTION
  • The pbPost function deals more robustly with the case of multiple target emails
  • The continuous integration and the README badge have been updated
  • The DESCRIPTION file now use Authors@R
  • The (encrypted) unit test configuration has been adjusted to reflect the current set of active devices
  • The mkdocs-material documentation site is now generated via altdoc

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the repo where comments and suggestions are welcome.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

3 October 2025

Dirk Eddelbuettel: #053: Adding llvm Snapshots for R Package Testing

Welcome to post 53 in the R4 series. Continuing with posts #51 from Tuesday and #52 from Wednesday and their stated intent of posting some more here is another quick one. Earlier today I helped another package developer who came to the r-package-devel list asking for help with a build error on the Fedora machine at CRAN running recent / development clang. In such situations, the best first step is often to replicate the issue. As I pointed out on the list, the LLVM team behind clang maintains an apt repo at apt.llvm.org/ making it a good resource to add to Debian-based container such as Rocker r-base or the offical r-base (the two are in fact interchangeable, and I take care of both). A small pothole, however, is that the documentation at the top of apt.llvm.org site is a bit stale and behind two aspects that changed on current Debian systems (i.e. unstable/testing as used for r-base). First, apt now prefers files ending in .sources (in a nicer format) and second, it now really requires a key (which is good practice). As it took me a few minutes to regather how to meet both requirements, I reckoned I might as well script this. Et voil the following script does that:
#!/bin/sh

## Update does not hurt but is not strictly needed
#apt update --quiet --quiet
#apt upgrade --yes

## wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key   sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc
## or as we are root in container
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key > /etc/apt/trusted.gpg.d/apt.llvm.org.asc

cat <<EOF >/etc/apt/sources.list.d/llvm-dev.sources
Types: deb
URIs: http://apt.llvm.org/unstable/
# for clang-21 
#   Suites: llvm-toolchain-21
# for current clang
Suites: llvm-toolchain
Components: main
Signed-By: /etc/apt/trusted.gpg.d/apt.llvm.org.asc
EOF

test -d ~/.R   mkdir ~/.R
cat <<EOF >~/.R/Makevars
CLANGVER=-22
# CLANGLIB=-stdlib=libc++
CXX=clang++\$(CLANGVER) \$(CLANGLIB)
CXX11=clang++\$(CLANGVER) \$(CLANGLIB)
CXX14=clang++\$(CLANGVER) \$(CLANGLIB)
CXX17=clang++\$(CLANGVER) \$(CLANGLIB)
CXX20=clang++\$(CLANGVER) \$(CLANGLIB)
CC=clang\$(CLANGVER)
SHLIB_CXXLD=clang++\$(CLANGVER) \$(CLANGLIB)
EOF

apt update
apt install --yes clang-22
Once the script is run, one can test a package (or set of packages) against clang-22 and clang++-22. This may help R package developers. The script is also generic enough for other development communities who can ignore (or comment-out / delete) the bit about ~/.R/Makevars and deploy the compiler differently. Updating the softlink as apt-preferences does is one way and done in many GitHub Actions recipes. As we only need wget here a basic Debian container should work, possibly with the addition of wget. For R users r-base hits a decent sweet spot.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

2 October 2025

Dirk Eddelbuettel: tinythemes 0.0.4 at CRAN: Micro Maintenance

tinythemes demo A tiniest of tiny violins micro maintenance release 0.0.4 of our tinythemes arrived on CRAN today. tinythemes provides the theme_ipsum_rc() function from hrbrthemes by Bob Rudis in a zero (added) dependency way. A simple example is (also available as a demo inside the package) contrasts the default style (on the left) with the one added by this package (on the right): This version adjusts to the fact that hrbrthemes is no longer on CRAN so the help page cannot link to its documentation. No other changes were made. The set of changes since the last release follows.

Changes in tinythemes version 0.0.4 (2025-10-02)
  • No longer \link to now-archived hrbrthemes

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the repo where comments and suggestions are welcome.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

1 October 2025

Dirk Eddelbuettel: #052: Running r-ci with Coverage

Welcome to post 52 in the R4 series. Following up on the post #51 yesterday and its stated intent of posting some more here The r-ci setup (which was introduced in post #32 and updated in post #45) offers portable continuous integration which can take advantage of different backends: Github Actions, Azure Pipelines, GitLab, BitBuckets, and possibly as it only requires a basic Ubuntu shell after which it customizes itself and runs via shell script. Portably. Now, most of us, I suspect still use it with GitHub Actions but it is reassuring to know that one can take it elsewhere should the need or desire arise. One thing many repos did, and which stopped working reliably is coverage analysis. This is made easy by the covr package, and made fast, easy, reliable (as we quip) thanks to r2u. But transmission to codecov stopped working a while back so I had mostly commented it out in my repos, rendering the reports more and more stale. Which is not ideal. A few weeks ago I gave this another look, and it turns that that codecov now requires an API token to upload. Which one can generate in the user -> settings menu on their website under the access tab. Which in turn then needs to be stored in each repo wanting to upload. For Github, this is under settings -> secrets and variables -> actions as a repository secret . I suggest using CODECOV_TOKEN for its name. After that the three-line block in the yaml file can reference it as a secret as in the following snippet, now five lines, taken from one of my ci.yaml files:
      - name: Coverage
        if: $  matrix.os == 'ubuntu-latest'  
        run: ./run.sh coverage
        env:
          CODECOV_TOKEN: $  secrets.CODECOV_TOKEN  
It assigns the secret we stored on the website, references it by prefix secrets. and assigns it to the environment variable CODECOV_TOKEN. After this reports flow as one can see on repositories where I re-enabled this as for example here for RcppArmadillo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

30 September 2025

Dirk Eddelbuettel: #051: A Neat Little Rcpp Trick

Welcome to post 51 in the R4 series. A while back I realized I should really just post a little more as not all post have to be as deep and introspective as for example the recent-ish two cultures post #49. So this post is a neat little trick I (somewhat belatedly) realized somewhat recently. The context is the ongoing transition from (Rcpp)Armadillo 14.6.3 and earlier to (Rcpp)Armadillo 15.0.2 or later. (I need to write a bit more about that, but that may require a bit more time.) (And there are a total of seven (!!) issue tickets managing the transition with issue #475 being the main parent issue, please see there for more details.) In brief, the newer and current Armadillo no longer allows C++11 (which also means it no longer allowes suppression of deprecation warnings ). It so happens that around a decade ago packages were actively encouraged to move towards C++11 so many either set an explicit SystemRequirements: for it, or set CXX_STD=CXX11 in src/Makevars .win . CRAN has for some time now issued NOTEs asking for this to be removed, and more recently enforced this with actual deadlines. In RcppArmadillo I opted to accomodate old(er) packages (using this by-now anti-pattern) and flip to Armadillo 14.6.3 during a transition period. That is what the package does now: It gives you either Armadillo 14.6.3 in case C++11 was detected (or this legacy version was actively selected via a compile-time #define), or it uses Armadillo 15.0.2 or later. So this means we can have either one of two versions, and may want to know which one we have. Armadillo carries its own version macros, as many libraries or projects do (R of course included). Many many years ago (git blame points to sixteen and twelve for a revision) we added the following helper function to the package (full source here, we show it here without the full roxygen2 comment header)
// [[Rcpp::export]]
Rcpp::IntegerVector armadillo_version(bool single)  

    // These are declared as constexpr in Armadillo which actually does not define them
    // They are also defined as macros in arma_version.hpp so we just use that
    const unsigned int major = ARMA_VERSION_MAJOR;
    const unsigned int minor = ARMA_VERSION_MINOR;
    const unsigned int patch = ARMA_VERSION_PATCH;

    if (single)  
        return Rcpp::wrap(10000 * major + 100 * minor + patch) ;
      else  
        return Rcpp::IntegerVector::create(Rcpp::Named("major") = major,
                                           Rcpp::Named("minor") = minor,
                                           Rcpp::Named("patch") = patch);
     
 
It either returns a (named) vector of the standard major , minor , patch form of the common package versioning pattern, or a single integer which can used more easily in C(++) via preprocessor macros. And this being an Rcpp-using package, we can of course access either easily from R:
> library(RcppArmadillo)
> armadillo_version(FALSE)
major minor patch 
   15     0     2 
> armadillo_version(TRUE)
[1] 150002
>
Perfectly valid and truthful. But cumbersome at the R level. So when preparing for these (Rcpp)Armadillo changes in one of my package, I realized I could alter such a function and set the S3 type to package_version. (Full version of one such variant here)
// [[Rcpp::export]]
Rcpp::List armadilloVersion()  
    // create a vector of major, minor, patch
    auto ver = Rcpp::IntegerVector::create(ARMA_VERSION_MAJOR, ARMA_VERSION_MINOR, ARMA_VERSION_PATCH);
    // and place it in a list (as e.g. packageVersion() in R returns)
    auto lst = Rcpp::List::create(ver);
    // and class it as 'package_version' accessing print() etc methods
    lst.attr("class") = Rcpp::CharacterVector::create("package_version", "numeric_version");
    return lst;
 
Three statements each to and return the value. And now in R we can operate more easily on this (using three dots as I didn t export it from this package):
> RcppDE:::armadilloVersion()
[1]  15.0.2 
> RcppDE:::armadilloVersion() >= "15.0.0"
[1] TRUE
>
An object of class package_version inheriting from numeric_version can directly compare against a (human- but not normally machine-readable) string like 15.0.0 because the simple S3 class defines appropriate operators, as well as print() / format() methods as the first expression shows. It is these little things that make working with R so smooth, and we can easily (three statements !!) do so from Rcpp-based packages too. The underlying object really is merely a list containing a vector:
> unclass(RcppDE:::armadilloVersion())
[[1]]
[1] 15  0  2

> 
but the S3 glue around it makes it behave nicely. So next time you are working with an object you plan to return to R, consider classing it to take advantage of existing infrastructure (if it exists, of course). It s easy enough to do, and may smoothen the experience at the R side.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

25 September 2025

Dirk Eddelbuettel: tint 0.1.6 on CRAN: Maintenance

A new version 0.1.6 of the tint package arrived at CRAN today. tint provides a style not unlike Tufte for use in html and pdf documents created from markdown. The github repo shows several examples in its README, more as usual in the package documentation. This release addresses a small issue where in pdf mode, pandoc (3.2.1 or newer) needs a particular macros defined when static (premade) images or figure files are included. No other changes.

Changes in tint version 0.1.6 (2025-09-25)
  • An additional LaTeX command needed by pandoc (>= 3.2.1) has been defined for the two pdf variants.

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More information is on the tint page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

22 September 2025

Vincent Bernat: Akvorado release 2.0

Akvorado 2.0 was released today! Akvorado collects network flows with IPFIX and sFlow. It enriches flows and stores them in a ClickHouse database. Users can browse the data through a web console. This release introduces an important architectural change and other smaller improvements. Let s dive in!
$ git diff --shortstat v1.11.5
 493 files changed, 25015 insertions(+), 21135 deletions(-)

New outlet service The major change in Akvorado 2.0 is splitting the inlet service into two parts: the inlet and the outlet. Previously, the inlet handled all flow processing: receiving, decoding, and enrichment. Flows were then sent to Kafka for storage in ClickHouse:
Akvorado flow processing before the change: flows are received and processed by the inlet, sent to Kafka and stored in ClickHouse
Akvorado flow processing before the introduction of the outlet service
Network flows reach the inlet service using UDP, an unreliable protocol. The inlet must process them fast enough to avoid losing packets. To handle a high number of flows, the inlet spawns several sets of workers to receive flows, fetch metadata, and assemble enriched flows for Kafka. Many configuration options existed for scaling, which increased complexity for users. The code needed to avoid blocking at any cost, making the processing pipeline complex and sometimes unreliable, particularly the BMP receiver.1 Adding new features became difficult without making the problem worse.2 In Akvorado 2.0, the inlet receives flows and pushes them to Kafka without decoding them. The new outlet service handles the remaining tasks:
Akvorado flow processing after the change: flows are received by the inlet, sent to Kafka, processed by the outlet and inserted in ClickHouse
Akvorado flow processing after the introduction of the outlet service
This change goes beyond a simple split:3 the outlet now reads flows from Kafka and pushes them to ClickHouse, two tasks that Akvorado did not handle before. Flows are heavily batched to increase efficiency and reduce the load on ClickHouse using ch-go, a low-level Go client for ClickHouse. When batches are too small, asynchronous inserts are used (e20645). The number of outlet workers scales dynamically (e5a625) based on the target batch size and latency (50,000 flows and 5 seconds by default). This new architecture also allows us to simplify and optimize the code. The outlet fetches metadata synchronously (e20645). The BMP component becomes simpler by removing cooperative multitasking (3b9486). Reusing the same RawFlow object to decode protobuf-encoded flows from Kafka reduces pressure on the garbage collector (8b580f). The effect on Akvorado s overall performance was somewhat uncertain, but a user reported 35% lower CPU usage after migrating from the previous version, plus resolution of the long-standing BMP component issue.

Other changes This new version includes many miscellaneous changes, such as completion for source and destination ports (f92d2e), and automatic restart of the orchestrator service (0f72ff) when configuration changes to avoid a common pitfall for newcomers. Let s focus on some key areas for this release: observability, documentation, CI, Docker, Go, and JavaScript.

Observability Akvorado exposes metrics to provide visibility into the processing pipeline and help troubleshoot issues. These are available through Prometheus HTTP metrics endpoints, such as /api/v0/inlet/metrics. With the introduction of the outlet, many metrics moved. Some were also renamed (4c0b15) to match Prometheus best practices. Kafka consumer lag was added as a new metric (e3a778). If you do not have your own observability stack, the Docker Compose setup shipped with Akvorado provides one. You can enable it by activating the profiles introduced for this purpose (529a8f). The prometheus profile ships Prometheus to store metrics and Alloy to collect them (2b3c46, f81299, and 8eb7cd). Redis and Kafka metrics are collected through the exporter bundled with Alloy (560113). Other metrics are exposed using Prometheus metrics endpoints and are automatically fetched by Alloy with the help of some Docker labels, similar to what is done to configure Traefik. cAdvisor was also added (83d855) to provide some container-related metrics. The loki profile ships Loki to store logs (45c684). While Alloy can collect and ship logs to Loki, its parsing abilities are limited: I could not find a way to preserve all metadata associated with structured logs produced by many applications, including Akvorado. Vector replaces Alloy (95e201) and features a domain-specific language, VRL, to transform logs. Annoyingly, Vector currently cannot retrieve Docker logs from before it was started. Finally, the grafana profile ships Grafana, but the shipped dashboards are broken. This is planned for a future version.

Documentation The Docker Compose setup provided by Akvorado makes it easy to get the web interface up and running quickly. However, Akvorado requires a few mandatory steps to be functional. It ships with comprehensive documentation, including a chapter about troubleshooting problems. I hoped this documentation would reduce the support burden. It is difficult to know if it works. Happy users rarely report their success, while some users open discussions asking for help without reading much of the documentation. In this release, the documentation was significantly improved.
$ git diff --shortstat v1.11.5 -- console/data/docs
 10 files changed, 1873 insertions(+), 1203 deletions(-)
The documentation was updated (fc1028) to match Akvorado s new architecture. The troubleshooting section was rewritten (17a272). Instructions on how to improve ClickHouse performance when upgrading from versions earlier than 1.10.0 was added (5f1e9a). An LLM proofread the entire content (06e3f3). Developer-focused documentation was also improved (548bbb, e41bae, and 871fc5). From a usability perspective, table of content sections are now collapsable (c142e5). Admonitions help draw user attention to important points (8ac894).
Admonition in Akvorado documentation to ask a user not to open an issue or start a discussion before reading the documentation
Example of use of admonitions in Akvorado's documentation

Continuous integration This release includes efforts to speed up continuous integration on GitHub. Coverage and race tests run in parallel (6af216 and fa9e48). The Docker image builds during the tests but gets tagged only after they succeed (8b0dce).
GitHub workflow for CI with many jobs, some of them running in parallel, some not
GitHub workflow to test and build Akvorado
End-to-end tests (883e19) ensure the shipped Docker Compose setup works as expected. Hurl runs tests on various HTTP endpoints, particularly to verify metrics (42679b and 169fa9). For example:
## Test inlet has received NetFlow flows
GET http://127.0.0.1:8080/prometheus/api/v1/query
[Query]
query: sum(akvorado_inlet_flow_input_udp_packets_total job="akvorado-inlet",listener=":2055" )
HTTP 200
[Captures]
inlet_receivedflows: jsonpath "$.data.result[0].value[1]" toInt
[Asserts]
variable "inlet_receivedflows" > 10
## Test inlet has sent them to Kafka
GET http://127.0.0.1:8080/prometheus/api/v1/query
[Query]
query: sum(akvorado_inlet_kafka_sent_messages_total job="akvorado-inlet" )
HTTP 200
[Captures]
inlet_sentflows: jsonpath "$.data.result[0].value[1]" toInt
[Asserts]
variable "inlet_sentflows" >=   inlet_receivedflows  

Docker Akvorado ships with a comprehensive Docker Compose setup to help users get started quickly. It ensures a consistent deployment, eliminating many configuration-related issues. It also serves as a living documentation of the complete architecture. This release brings some small enhancements around Docker: Previously, many Docker images were pulled from the Bitnami Containers library. However, VMWare acquired Bitnami in 2019 and Broadcom acquired VMWare in 2023. As a result, Bitnami images were deprecated in less than a month. This was not really a surprise4. Previous versions of Akvorado had already started moving away from them. In this release, the Apache project s Kafka image replaces the Bitnami one (1eb382). Thanks to the switch to KRaft mode, Zookeeper is no longer needed (0a2ea1, 8a49ca, and f65d20). Akvorado s Docker images were previously compiled with Nix. However, building AArch64 images on x86-64 is slow because it relies on QEMU userland emulation. The updated Dockerfile uses multi-stage and multi-platform builds: one stage builds the JavaScript part on the host platform, one stage builds the Go part cross-compiled on the host platform, and the final stage assembles the image on top of a slim distroless image (268e95 and d526ca).
# This is a simplified version
FROM --platform=$BUILDPLATFORM node:20-alpine AS build-js
RUN apk add --no-cache make
WORKDIR /build
COPY console/frontend console/frontend
COPY Makefile .
RUN make console/data/frontend
FROM --platform=$BUILDPLATFORM golang:alpine AS build-go
RUN apk add --no-cache make curl zip
WORKDIR /build
COPY . .
COPY --from=build-js /build/console/data/frontend console/data/frontend
RUN go mod download
RUN make all-indep
ARG TARGETOS TARGETARCH TARGETVARIANT VERSION
RUN make
FROM gcr.io/distroless/static:latest
COPY --from=build-go /build/bin/akvorado /usr/local/bin/akvorado
ENTRYPOINT [ "/usr/local/bin/akvorado" ]
When building for multiple platforms with --platform linux/amd64,linux/arm64,linux/arm/v7, the build steps until the highlighted line execute only once for all platforms. This significantly speeds up the build. Akvorado now ships Docker images for these platforms: linux/amd64, linux/amd64/v3, linux/arm64, and linux/arm/v7. When requesting ghcr.io/akvorado/akvorado, Docker selects the best image for the current CPU. On x86-64, there are two choices. If your CPU is recent enough, Docker downloads linux/amd64/v3. This version contains additional optimizations and should run faster than the linux/amd64 version. It would be interesting to ship an image for linux/arm64/v8.2, but Docker does not support the same mechanism for AArch64 yet (792808).

Go This release includes many changes related to Go but not visible to the users.

Toolchain In the past, Akvorado supported the two latest Go versions, preventing immediate use of the latest enhancements. The goal was to allow users of stable distributions to use Go versions shipped with their distribution to compile Akvorado. However, this became frustrating when interesting features, like go tool, were released. Akvorado 2.0 requires Go 1.25 (77306d) but can be compiled with older toolchains by automatically downloading a newer one (94fb1c).5 Users can still override GOTOOLCHAIN to revert this decision. The recommended toolchain updates weekly through CI to ensure we get the latest minor release (5b11ec). This change also simplifies updates to newer versions: only go.mod needs updating. Thanks to this change, Akvorado now uses wg.Go() (77306d) and I have started converting some unit tests to the new test/synctest package (bd787e, 7016d8, and 159085).

Testing When testing equality, I use a helper function Diff() to display the differences when it fails:
got := input.Keys()
expected := []int 1, 2, 3 
if diff := helpers.Diff(got, expected); diff != ""  
    t.Fatalf("Keys() (-got, +want):\n%s", diff)
 
This function uses kylelemons/godebug. This package is no longer maintained and has some shortcomings: for example, by default, it does not compare struct private fields, which may cause unexpectedly successful tests. I replaced it with google/go-cmp, which is stricter and has better output (e2f1df).

Another package for Kafka Another change is the switch from Sarama to franz-go to interact with Kafka (756e4a and 2d26c5). The main motivation for this change is to get a better concurrency model. Sarama heavily relies on channels and it is difficult to understand the lifecycle of an object handed to this package. franz-go uses a more modern approach with callbacks6 that is both more performant and easier to understand. It also ships with a package to spawn fake Kafka broker clusters, which is more convenient than the mocking functions provided by Sarama.

Improved routing table for BMP To store its routing table, the BMP component used kentik/patricia, an implementation of a patricia tree focused on reducing garbage collection pressure. gaissmai/bart is a more recent alternative using an adaptation of [Donald Knuth s ART algorithm][] that promises better performance and delivers it: 90% faster lookups and 27% faster insertions (92ee2e and fdb65c). Unlike kentik/patricia, gaissmai/bart does not help efficiently store values attached to each prefix. I adapted the same approach as kentik/patricia to store route lists for each prefix: store a 32-bit index for each prefix, and use it to build a 64-bit index for looking up routes in a map. This leverages Go s efficient map structure. gaissmai/bart also supports a lockless routing table version, but this is not simple because we would need to extend this to the map storing the routes and to the interning mechanism. I also attempted to use Go s new unique package to replace the intern package included in Akvorado, but performance was worse.7

Miscellaneous Previous versions of Akvorado were using a custom Protobuf encoder for performance and flexibility. With the introduction of the outlet service, Akvorado only needs a simple static schema, so this code was removed. However, it is possible to enhance performance with planetscale/vtprotobuf (e49a74, and 8b580f). Moreover, the dependency on protoc, a C++ program, was somewhat annoying. Therefore, Akvorado now uses buf, written in Go, to convert a Protobuf schema into Go code (f4c879). Another small optimization to reduce the size of the Akvorado binary by 10 MB was to compress the static assets embedded in Akvorado in a ZIP file. It includes the ASN database, as well as the SVG images for the documentation. A small layer of code makes this change transparent (b1d638 and e69b91).

JavaScript Recently, two large supply-chain attacks hit the JavaScript ecosystem: one affecting the popular packages chalk and debug and another impacting the popular package @ctrl/tinycolor. These attacks also exist in other ecosystems, but JavaScript is a prime target due to heavy use of small third-party dependencies. The previous version of Akvorado relied on 653 dependencies. npm-run-all was removed (3424e8, 132 dependencies). patch-package was removed (625805 and e85ff0, 69 dependencies) by moving missing TypeScript definitions to env.d.ts. eslint was replaced with oxlint, a linter written in Rust (97fd8c, 125 dependencies, including the plugins). I switched from npm to Pnpm, an alternative package manager (fce383). Pnpm does not run install scripts by default8 and prevents installing packages that are too recent. It is also significantly faster.9 Node.js does not ship Pnpm but it ships Corepack, which allows us to use Pnpm without installing it. Pnpm can also list licenses used by each dependency, removing the need for license-compliance (a35ca8, 42 dependencies). For additional speed improvements, beyond switching to Pnpm and Oxlint, Vite was replaced with its faster Rolldown version (463827). After these changes, Akvorado only pulls 225 dependencies.

Next steps I would like to land three features in the next version of Akvorado:
  • Add Grafana dashboards to complete the observability stack. See issue #1906 for details.
  • Integrate OVH s Grafana plugin by providing a stable API for such integrations. Akvorado s web console would still be useful for browsing results, but if you want to build and share dashboards, you should switch to Grafana. See issue #1895.
  • Move some work currently done in ClickHouse (custom dictionaries, GeoIP and IP enrichment) back into the outlet service. This should give more flexibility for adding features like the one requested in issue #1030. See issue #2006.

I started working on splitting the inlet into two parts more than one year ago. I found more motivation in recent months, partly thanks to Claude Code, which I used as a rubber duck. Almost none of the produced code was kept:10 it is like an intern who does not learn.

  1. Many attempts were made to make the BMP component both performant and not blocking. See for example PR #254, PR #255, and PR #278. Despite these efforts, this component remained problematic for most users. See issue #1461 as an example.
  2. Some features have been pushed to ClickHouse to avoid the processing cost in the inlet. See for example PR #1059.
  3. This is the biggest commit:
    $ git show --shortstat ac68c5970e2c   tail -1
    231 files changed, 6474 insertions(+), 3877 deletions(-)
    
  4. Broadcom is known for its user-hostile moves. Look at what happened with VMWare.
  5. As a Debian developer, I dislike these mechanisms that circumvent the distribution package manager. The final straw came when Go 1.25 spent one month in the Debian NEW queue, an arbitrary mechanism I don t like at all.
  6. In the early years of Go, channels were heavily promoted. Sarama was designed during this period. A few years later, a more nuanced approach emerged. See notably Go channels are bad and you should feel bad.
  7. This should be investigated further, but my theory is that the intern package uses 32-bit integers, while unique uses 64-bit pointers. See commit 74e5ac.
  8. This is also possible with npm. See commit dab2f7.
  9. An even faster alternative is Bun, but it is less available.
  10. The exceptions are part of the code for the admonition blocks, the code for collapsing the table of content, and part of the documentation.

21 September 2025

Dirk Eddelbuettel: rcppmlpackexamples 0.0.1 on CRAN: New Package

mlpack is a fabulous project providing countless machine learning algorithms in clean and performant C++ code as a header-only library. This gives both high performance and the ability to run the code in resource-constrained environment such as embedded systems. Bindings to a number of other languages are available, and an examples repo provides examples. The project also has a mature R package on CRAN which offers the various algorithms directly in R. Sometimes, however, one might want to use the header-only C++ code in another R package. How to do that was not well documented. A user alerted me by email to this fact a few weeks ago, and this lead to both an updated mlpack release at CRAN and this package. In short, we show via three complete examples how to access the mlpack code in C++ code in this package, offering a re-usable stanza to start a different package from. The only other (header-only) dependencies are provided by CRAN packages RcppArmadillo and RcppEnsmallen wrapping, respectively, the linear algebra and optimization libraries used by mlpack. Courtesy of my CRANberries, there is also a a new package note (no diffstat report yet). More detailed information is on the rcppmlpackexamples page, or the github repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

19 September 2025

Dirk Eddelbuettel: RcppArmadillo 15.0.2-2 on CRAN: Transition to Armadillo 15

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra and scientific computing. It aims towards a good balance between speed and ease of use, has a syntax deliberately close to Matlab, and is useful for algorithm development directly in C++, or quick conversion of research code into production environments. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 1261 other packages on CRAN, downloaded 41.4 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 647 times according to Google Scholar. This versions updates the 15.0.2-1 release from last week. Following fairly extensive email discussions with CRAN, we are now accelerating the transition to Armadillo. When C++14 or newer is used (which after all is the default since R 4.1.0 released May 2021, see WRE Section 1.2.4), or when opted into, the newer Armadillo is selected. If on the other hand either C++11 is still forced, or the legacy version is explicitly selected (which currently one package at CRAN does), then Armadillo 14.6.3 is selected. Most packages will not see a difference and automatically switch to the newer Armadillo. However, some packages will see one or two types of warning. First, if C++11 is still actively selected via for examples CXX_STD then CRAN will nudge a change to a newer compilation standard (as they have been doing for some time already). Preferably the change should be to simply remove the constraint and let R pick the standard based on its version and compiler availability. These days that gives us C++17 in most cases; see WRE Section 1.2.4 for details. (Some packages may need C++14 or C++17 or C++20 explicitly and can also do so.) Second, some packages may see a deprecation warning. Up until Armadillo 14.6.3, the package suppressed these and you can still get that effect by opting into that version by setting -DARMA_USE_LEGACY. (However this route will be sunset eventually too.) But one really should update the code to the non-deprecated version. In a large number of cases this simply means switching from using arma::is_finite() (typically called on a scalar double) to calling std::isfinite(). But there are some other cases, and we will help as needed. If you maintain a package showing deprecation warnings, and are lost here and cannot workout the conversion to current coding styles, please open an issue at the RcppArmadillo repository (i.e. here) or in your own repository and tag me. I will also reach out to the maintainers of a smaller set of packages with more than one reverse dependency. A few small changes have been made internal packaging and documentation, a small synchronization with upstream for two commits since the 15.0.2 release, as well as a link to the ldlasb2 repository and its demonstration regarding some ill-stated benchmarks done elsewhere. The detailed changes since the last CRAN release follow.

Changes in RcppArmadillo version 15.0.2-2 (2025-09-18)
  • Minor update to skeleton Makevars,Makevars.win
  • Update README.md to mention ldlasb2 repository
  • Minor documentation update (#487)
  • Synchronized with Armadillo upstream (#488)
  • Refine Armadillo version selection in coordination with CRAN maintainers to support transition towards Armadillo 15.0.*

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

14 September 2025

Dirk Eddelbuettel: RcppSimdJson 0.1.14 on CRAN: New Upstream Major

A brand new release 0.1.14 of the RcppSimdJson package is now on CRAN. RcppSimdJson wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire and collaborators. Via very clever algorithmic engineering to obtain largely branch-free code, coupled with modern C++ and newer compiler instructions, it results in parsing gigabytes of JSON parsed per second which is quite mindboggling. The best-case performance is faster than CPU speed as use of parallel SIMD instructions and careful branch avoidance can lead to less than one cpu cycle per byte parsed; see the video of the talk by Daniel Lemire at QCon. This version includes the new major upstream release 4.0.0 with major new features including a builder for creating JSON from the C++ side objects. This is something a little orthogonal to the standard R usage of the package to parse and load JSON data but could still be of interest to some. The short NEWS entry for this release follows.

Changes in version 0.1.14 (2025-09-13)
  • simdjson was upgraded to version 4.0.0 (Dirk in #96
  • Continuous integration now relies a token for codecov.io

Courtesy of my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

Next.

Previous.