Search Results: "nate"

5 February 2022

Dirk Eddelbuettel: corels 0.0.4 on CRAN: M1 Update

An updated version of the corels package is now on CRAN! The Certifiably Optimal RulE ListS (Corels) learner provides interpretable decision rules with an optimality guarantee. The change is (just like the previous one) chiefly an update to configure.ac in order to ensure R on M1 macOS finds the locally-added GNU GMP. Our thanks to the infatiguable Brian Ripley for the heads-up even containing the two needed assignments to LD and CPPFLAGS.. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

4 February 2022

Ian Jackson: EUDCC QR codes vs NHS Travel barcodes vs TAC Verify

The EU Digital Covid Certificate scheme is a format for (digitally signed) vaccination status certificates. Not only EU countries participate - the UK is now a participant in this scheme. I am currently on my way to go skiing in the French Alps. So I needed a certificate that would be accepted in France. AFAICT the official way to do this is to get the international certificate from the NHS, and take it to a French pharmacy who will convert it into something suitably French. (AIUI the NHS international barcode is the same regardless of whether you get it via the NHS website, the NHS app, or a paper letter. NB that there is one barcode per vaccine dose so you have to get the right one - probably that means your booster since there s a 9 month rule!) I read on an forum somewhere that you could use the French TousAntiCovid app to convert the barcode. So I thought I would try that. The TousAntiCovid is Free Softare and on F-Droid, so I was happy to install and use it for this. I also used the French TAC Verify app to check to see what barcodes were accepted. (I found an official document addressed to French professionals recommending this as an option for verifying the status of visitors to one s establishment.) Unfortunately this involves a googlified phone, but one could use a burner phone or ask a friend who s bitten that bullet already. I discovered that, indeed: This made me curious. I used a QR code reader to decode both barcodes. The decodings were identical! A long string of guff starting HC1:. AIUI it is an encoded JWT. But there was a difference in the framing: Binary Eye reported that the NHS barcode used error correction level M (medium, aka 15%). The TousAntiCovid barcode used level L (low, 7%). I had my QR code software regenerate a QR code at level M for the data from the TousAntiCovid code. The result was a QR code which is identical (pixel-wise) to the one from the NHS. So the only difference is the error correction level. Curiously, both L (low, generated by TousAntiCovid, accepted by TAC Verify) and M (medium, generated by NHS, rejected by TAC Verify) are lower than the Q (25") recommended by what I think is the specification. This is all very odd. But the upshot is that I think you can convert the NHS international barcode into something that should work in France simply by passing it through any QR code software to re-encode it at error correction level L (7%). But if you re happy to use the TousAntiCovid app it s probably a good way to store them. I guess I ll find out when I get to France if the converted NHS barcodes work in real establishments. Thanks to the folks behind sanipasse.fr for publishing some helpful backround info and operating a Free Software backed public verification service. Footnote To compare the QR codes pixelwise, I roughly cropped the NHS PDF image using a GUI tool, and then on each of the two images used pnmcrop (to trim the border), pnmscale (to rescale the one-pixel-per-pixel output from Binary Eye) and pnmarith -difference to compare them (producing a pretty squirgly image showing just the pixel edges due to antialiasing).

comment count unavailable comments

2 February 2022

Norbert Preining: Mechanical keyboards: Pulsar PCMK

Mechanical keyboards the big fat rabbit hole you can disappear I started using mechanical keyboards about a year ago, with a Drevo Blademaster Pro (review coming up), but recently got a Pulsar PCML TKL keyboard in a build-it-yourself order.
The Drevo Blademaster Pro I am using is great, but doesn t allow changing switches at all. So I was contemplating getting a mechanical keyboard that allows for arbitrary switches. My biggest problem here is that I am used to the Japanese JIS layout which gives you a lot more keys which come in extremely handy in Emacs or when typing various languages. Fortunately, APlusX a Korean company manufactures a lot of gear under the Pulsar name, and supports also JIS layout. In addition, they have a great web site to customize your keyboard (layout, color, switch, keycaps) and send you a build-yourself kit for a very reasonable prize at least in Japan. So I got my first keyboard to put together myself . how was I nervous getting all the stuff out! Despite being a DIY keyboard, it is very easy (and they offer also pre-assembly options, too!). You don t need to solder the PCB or similar, the steps are more or less: (i) put in the switches, and (ii) add the key caps. I started with the first, putting the switches (I went with Kailh Box Brown tactile ones) into the PCB board
Well, that was easy at least I thought until I started testing the keys and realized that about 20 of them didn t work!! Pulling out the switches again I saw that I twisted a pin on each of them. One by one I straightened the pins and reinserted them very carefully. Lesson learned! At the end all the switches were on the board and reacted to key presses. Next step was adding the key caps. Again, those are not really special key caps, but simply style and sufficient for me. Of course I messed up 0 and O (which have different heights) and at first were confused about different arrow options etc, but since plugging in and pulling out key caps is very easy, at the end all the caps were in place.
With the final keyboard assembled (see top photo), I connected it to my Linux system and started typing around. And first of all, the typing experience was nice. The Kailh Box Brown switches have a bit stronger actuation point then the switches I have in the Blademaster Pro (which are Cherry MX Brown ones), but above all the sound is a bit deeper and thumbier , which really gives a nice feeling. The keyboard also allows changing the RGB lightening via the keyboard (color, pattern, speed, brightness etc). There is a configuration software for macros etc, unfortunately it only works on Windows (and I couldn t get it to work with Wine, either), a sour point One more negative point is that the LED backlight doesn t have a timeout, that is, it stays on all the time. The Drevo I have turns off after a configured number of seconds, and turns completely black something I really like and miss on the Pulsar. Another difference to the Blademaster is connectivity: While the Blademaster offers cable, bluetooth, and wireless (with dongle), the Pulsar only offers cable (USB-C). Not a real deal-breaker for me, since I use it at my desktop and really don t need wireless/bluetooth capabilities, but still. I have been using the Pulsar now for a few days without even touching the Drevo (besides comparing typing sounds and actuation points), and really like the Pulsar. I think it is hard to get a fully configurable and changeable mechanical keyboard for a similar prize. There is one last thing that I really really miss an ergonomic mechanical keyboard. Of course there are some, like the ErgoDox EZ or the Kinesis Advantage 2, but they don t offer JIS layout (and are very expensive). Then there is the Truly Ergonomic CLEAVE keyboard, which is really great, but quite expensive. I guess I have to dive down the rabbit hole even more and make my own PCB and ergonomic keyboard!

Ben Hutchings: CI for the Debian kernel team

Starting just after Christmas, I have been working on CI for all the kernel team's packages on Salsa. The salsa-ci-team has done great work on producing a common pipeline that is usable for most packages with minimal configuration. However, for some packages a lot more work was required. Linux I started with the most important package, linux itself. This now has about 1.1 GiB of source spread over 76,000 source files. That turns out to be a problem for the pipeline which currently puts unpacked source in artifacts - it is far beyond the limits of what Salsa allows. I worked around this by using a modified version of the extract-source and build jobs that use packed source package as the artifacts. The output of the build job is compatible with the common test jobs. The linux package also takes a lot of resources to build; around 80 minutes on the fastest PC I have at home (if ccache is not primed). Salsa's shared CI runners seem to be about 10 times slower than that, so it is completely unfeasible to even one full build in CI. Instead I defined a new build profile that includes only the smallest kernel configuration, without debug info, and the user-space packages. This still takes over an hour with the Salsa CI runners, but I don't think we can improve this much without losing a lot of code coverage. Our Git repository for linux also does not contain the upstream source, so the extract-source job has to fetch that. The common extract-source job uses origtargz to do that, and in case the orig tarball is not already in the archive this will run uscan. That led me to a new problem: our debian/watch file could only find tarballs linked from the front of www.kernel.org, and we're sometimes working with different upstream versions. There is actually no single page listing all tarball releases of Linux, and tarballs for release candidates are dynamically generated by CGit and unsigned. So I changed debian/watch to fetch from Git, which is what we were already doing with our own genorig.py script. Unfortunately, running uscan against a Git upstream, with some files excluded (as there are still a few upstream files we consider non-free), is about twice as slow as it could be. Since I had to modify the extract-source job anyway, I've continued using genorig.py there. A full build log for linux is over 200 MiB, and even with the reduced build profile it would be much longer than Salsa's limit of 2 4 MiB. I therefore opted to use the 'terse' build option (which translates to V=0), but made the builds of user-space tools ignore this option so that blhc could still do its work. (The kernel itself cannot use the same hardening options, so blhc is not useful there.) Finally, with the CI pipeline running, blhc and lintian showed a lot of problems that we hadn't been attending to. I've fixed all the blhc errors (with some careful suppressions), all the lintian errors, and the most straightforward lintian warnings. firmware-nonfree The firmware-nonfree package also has huge "source" (about 560 MB) and needed some of the same modifications, but is quick to build so did not require a special build profile. Running lintian over firmware-nonfree reminded me that I needed to sort out the unsuual and inconistent handling of machine-readable copyright information in this source package. I had already done most of that work on a private branch in 2020, so this is mostly ready but I still need to resolve a licensing issue with AppStream metadata. Other packages For kernel-handbook, there was already a trivial "CI" pipeline used to push static pages to the web site. I've replaced this with the common pipeline plus a job that will push the pages from each build on the master branch. For everything else, it was straightforward to enable the common pipeline with a little bit of configuration.

1 February 2022

Dirk Eddelbuettel: #35: apt install rstudio quarto

Welcome to the 35th post in the ravishingly rabiant R recommendations, or R4. Today s post is about apt and R tools. Many of us have been running RStudio off our local machines for as long as binaries have been provided. Which is by now probably a bit over a decade. Time flies. And as nice it is to have matching binaries, in my case in the .deb format used on Debian or Ubuntu, it is wee bit a painful to manually download a file and then install it. Twice the pain if you are lucky enough to be on a system where you can also run RStudio Server. And now three times as painful as you may need a matching quarto-cli binary for the nice quarto service. So wouldn t it be nice to have an apt-getable repo? And to autoMAGICall get updated versions when they are available? Oh yes. And I had been bugging JJ from day one. And JJ would almost listen intendly, nod briefly and firmly, and issue an assured we will look into it. Well, they are still looking Luckily, years ago, Carl wrote a helper script for our use in Rocker. I promptly adopted these and kept them in the littler examples directory as a pair of script getRStudioDesktop.r and getRStudioServer.r, later complemented by getQuartoCli.r. And I used these for years, somewhere between weekly and monthly. But it is still very manual: three script calls, one sudo dpkg -i call. And as our good friends at RStudio don t seem to be coming forward with a repo, I created one at GitHub thinking I could serve the files via GitHub Pages. Which of course I cannot as the .deb file for rstudio is well above the 100mb limit. So that seemed to be a bit of a setback. But after a bit of pondering, and recognizing that I am now in the fortunate position to have symmetric broadband access at home, I reckoned that until the bandwidth use gets excessive I will serve this as truly personal package archive (or tPPA) from here. Note that this is calibrated for my use so Ubuntu amd64 it is. Nothing else. And that it installs dailies . Which may cause issues for some people. You have warned. Reading tis paragraphs signifies agreement with the terms and limitations. Just kidding. A quick screenshot from an update earlier is here. Note that I use the fabulous wajig wrapper by Graham Williams here as my frontend to apt, dpkg and more as I have for even longer than I have use RStudio. Its use is tangential here; sudo apt upgrade would have done the same (and is essentially being called). And it demonstrates the main benefit: we are now automated as the cron scheduler launches an update of the PPA at which ever frequency you chose (currently twice a week for me) and after that it becomes part of the normal apt updates we do anyway (and which I do about daily). So that s main gist: automated apt upgrades of rstudio, rstudio-server, and quarto-cli. And you can find the underlying code in the GitHub repo ppa-rstudio which I put together a good week ago. I am currently updating the tPPA twice a week from crontab and have had two full upgrades already. And who knows, maybe with a bid of prodding RStudio may come around. One can always hope. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

27 January 2022

Dirk Eddelbuettel: td 0.0.6 on CRAN: Minor Bugfix

The td package for accessing the twelvedata API for financial data has been updated once more on CRAN and is now at version 0.0.6. The release comes in response to an email from CRAN who noticed (via r-devel) that I was sloppy (in one spot, it turns out) with a logical expression resulting in an expression of length greather than one. Fixed by wrapping an all() around it and the package was back at CRAN minutes later thanks to automated processing over their end. The NEWS entry follows.

Changes in version 0.0.6 (2022-01-26)
  • Correct one equality comparison by wrapping in all() to ensure a length-one logical expression is created

Courtesy of my CRANberries, there is a comparison to the previous release. For questions or comments use the issue tracker off the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

25 January 2022

Dirk Eddelbuettel: RcppArmadillo 0.10.8.1.0 on CRAN: Upstream Updates

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 950 other packages on CRAN, downloaded over 22.9 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint/vignette) by Conrad and myself has been cited 451 times according to Google Scholar. This release brings another upstream update 10.8.0, and first bug fix release 10.8.1. As updates by Conrad can come a little quicker than the desired monthly cadence CRAN aims for, we skipped the 10.8.0 release for CRAN only but of course generally provide them via the Rcpp drat repo as well as via general updates to the repo, and full reverse dependency testing (for which results are always logged here). The full set of changes (since the last CRAN release 0.10.7.5.0) follows.

Changes in RcppArmadillo version 0.10.8.1.0 (2022-01-23)
  • Upgraded to Armadillo release 10.8.1 (Realm Raider)
    • fix interaction between OpenBLAS and LAPACK
    • emit warning if find() is incorrectly used to locate NaN elements

Changes in RcppArmadillo version 0.10.8.0.0 (2022-01-02)
  • Upgraded to Armadillo release 10.8 (Realm Raider)
    • faster handling of symmetric matrices by pinv() and rank()
    • faster handling of diagonal matrices by inv_sympd(), pinv(), rank()
    • expanded norm() to handle integer vectors and matrices
    • added datum::tau to replace 2

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

23 January 2022

Matthieu Caneill: Debsources, python3, and funky file names

Rumors are running that python2 is not a thing anymore. Well, I'm certainly late to the party, but I'm happy to report that sources.debian.org is now running python3. Wait, it wasn't? Back when development started, python3 was very much a real language, but it was hard to adopt because it was not supported by many libraries. So python2 was chosen, meaning print-based debugging was used in lieu of print()-based debugging, and str were bytes, not unicode. And things were working just fine. One day python2 EOL was announced, with a date far in the future. Far enough to procrastinate for a long time. Combine this with a codebase that is stable enough to not see many commits, and the fact that Debsources is a volunteer-based project that happens at best on week-ends, and you end up with a dormant software and a missed deadline. But, as dormant as the codebase is, the instance hosted at sources.debian.org is very popular and gets 200k to 500k hits per day. Largely enough to be worth a proper maintenance and a transition to python3. Funky file names While transitioning to python3 and juggling left and right with str, bytes and unicode for internal objects, files, database entries and HTTP content, I stumbled upon a bug that has been there since day 1. Quick recap if you're unfamiliar with this tool: Debsources displays the content of the source packages in the Debian archive. In other words, it's a bit like GitHub, but for the Debian source code. And some pieces of software out there, that ended up in Debian packages, happen to contain files whose names can't be decoded to UTF-8. Interestingly enough, there's no such thing as a standard for file names: with a few exceptions that vary by operating system, any sequence of bytes can be a legit file name. And some sequences of bytes are not valid UTF-8. Of course those files are rare, and using ASCII characters to name a file is a much more common practice than using bytes in a non-UTF-8 character encoding. But when you deal with almost 100 million files on which you have no control (those files come from free software projects, and make their way into Debian without any renaming), it happens. Now back to the bug: when trying to display such a file through the web interface, it would crash because it can't convert the file name to UTF-8, which is needed for the HTML representation of the page. Bugfix An often valid approach when trying to represent invalid UTF-8 content is to ignore errors, and replace them with ? or . This is what Debsources actually does to display non-UTF-8 file content. Unfortunately, this best-effort approach is not suitable for file names, as file names are also identifiers in Debsources: among other places, they are part of URLs. If an URL were to use placeholder characters to replace those bytes, there would be no deterministic way to match it with a file on disk anymore. The representation of binary data into text is a known problem. Multiple lossless solutions exist, such as base64 and its variants, but URLs looking like https://sources.debian.org/src/Y293c2F5LzMuMDMtOS4yL2Nvd3NheS8= are not readable at all compared to https://sources.debian.org/src/cowsay/3.03-9.2/cowsay/. Plus, not backwards-compatible with all existing links. The solution I chose is to use double-percent encoding: this allows the representation of any byte in an URL, while keeping allowed characters unchanged - and preventing CGI gateways from trying to decode non-UTF-8 bytes. This is the best of both worlds: regular file names get to appear normally and are human-readable, and funky file names only have percent signs and hex numbers where needed. Here is an example of such an URL: https://sources.debian.org/src/aspell-is/0.51-0-4/%25EDslenska.alias/. Notice the %25ED to represent the percentage symbol itself (%25) followed by an invalid UTF-8 byte (%ED). Transitioning to this was quite a challenge, as those file names don't only appear in URLs, but also in web pages themselves, log files, database tables, etc. And everything was done with str: made sense in python2 when str were bytes, but not much in python3. What are those files? What's their network? I was wondering too. Let's list them!
import os
with open('non-utf-8-paths.bin', 'wb') as f:
    for root, folders, files in os.walk(b'/srv/sources.debian.org/sources/'):
        for path in folders + files:
            try:
                path.decode('utf-8')
            except UnicodeDecodeError:
                f.write(root + b'/' + path + b'\n')
Running this on the Debsources main instance, which hosts pretty much all Debian packages that were part of a Debian release, I could find 307 files (among a total of almost 100 million files). Without looking deep into them, they seem to fall into 2 categories: That last point hits home, as it was clearly lacking in Debsources. A funky file name is now part of its test suite. ;)

22 January 2022

Steve Kemp: Visiting the UK was difficult, but worth it

So in my previous post I mentioned that we were going to spend the Christmas period in the UK, which we did. We spent a couple of days there, meeting my parents, and family. We also persuaded my sister to drive us to Scarborough so that we could hang out on the beach for an afternoon. Finland has lots of lakes, but it doesn't have proper waves. So it was surprisingly good just to wade in the sea and see waves! Unfortunately our child was a wee bit too scared to ride on a donkey! Unfortunately upon our return to Finland we all tested positive for COVID-19, me first, then the child, and about three days later my wife. We had negative tests in advance of our flights home, so we figure that either the tests were broken, or we were infected in the airplane/airport. Thankfully things weren't too bad, we stayed indoors for the appropriate length of time, and a combination of a couple of neighbours and online shopping meant we didn't run out of food. Since I've been back home I've been automating AWS activities with aws-utils, and updating my simple host-automation system, marionette. Marionette is something that was inspired by puppet, the configuration management utility, but it runs upon localhost only. Despite the small number of integrated primitives it actually works surprisingly well, and although I don't expect it will ever become popular it was an interesting research project. The aws-utilities? They were specifically put together because I've worked in a few places where infrastructure is setup with terraform, or cloudformation, but there are always the odd thing that is configured manually. Typically we'll have an openvpn gateway which uses a manually maintained IP allow-list, or some admin-server which has a security-group maintained somewhat manually. Having the ability to update a bunch of rules with your external IP, as a single command, across a number of AWS accounts/roles, and a number of security-groups is an enormous time-saver when your home IP changes. I'd quite like to add more things to that collection, but there's no particular rush.

Dirk Eddelbuettel: qlcal 0.0.2 on CRAN: Updates

The second release of the still fairly new qlcal package arrivied at CRAN today. qlcal is based on the calendaring subset of QuantLib. It is provided (for the R package) as a set of included files, so the package is self-contained and does not depend on an external QuantLib library (which can be demanding to build). qlcal covers over sixty country / market calendars and can compute holiday lists, its complement (i.e. business day lists) and much more. This release brings a further package simplification from removing a few more files not needed for just calendaring, as well as an update 2022 calendar for China from the just-release 1.25 version of QuantLib.

Changes in version 0.0.2 (2022-01-21)
  • Further minimize set of files needed for calendaring
  • Update China calendar from QuantLib 1.25 release

See the project page and package documentation for more details, and more examples. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

20 January 2022

Dirk Eddelbuettel: RQuantLib 0.4.15: Regular Update

A new release 0.4.15 of RQuantLib arrived at CRAN earlier today, and has been uploaded to Debian as well. QuantLib is a very comprehensice free/open-source library for quantitative finance; RQuantLib connects it to the R environment and language. The release of RQuantLib comes four months after the previous release, and brings a momitor update for the just-released QuantLib 1.2.5 version along with a few small cleanups to calendars and daycounters.

Changes in RQuantLib version 0.4.15 (2022-01-19)
  • Changes in RQuantLib code:
    • Calendar support has been updated and completed to current QuantLib standards (Dirk in #161)
    • More daycounters have been added (Kai Lin in #163 fixing #162, #164)
    • The bonds pricers were update to changes in QuantLib 1.25 (Dirk)
  • Changes in RQuantLib package and setup:
    • Some package metadata was removed from the README.md (Dirk)

Courtesy of my CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the RQuantLib page. Questions, comments etc should go to the new rquantlib-devel mailing list. Issue tickets can be filed at the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

19 January 2022

Joerg Jaspert: Funny CPU usage

Munin plugin and it s CPU usage (shell fixup) So at work we do have a munin server running, and one of the graphs we do for every system is a network statistics one with a resolution of 1 second. That s a simple enough script to have, and it is working nicely - on 98% of our machines. You just don t notice the data gatherer at all, so that we also have some other graphs done with a 1 second resolution. For some, this really helps.

Basics The basic code for this is simple. There is a bunch of stuff to start the background gathering, some to print out the config, and some to hand out the data when munin wants it. Plenty standard. The interesting bit that goes wrong and uses too much CPU on one Linux Distribution is this:
run_acquire()  
   echo "$$" > $ pidfile 
   while :; do
     TSTAMP=$(date +%s)
     echo $ IFACE _tx.value $ TSTAMP :$(cat /sys/class/net/$ IFACE /statistics/tx_bytes ) >> $ cache 
     echo $ IFACE _rx.value $ TSTAMP :$(cat /sys/class/net/$ IFACE /statistics/rx_bytes ) >> $ cache 
     # Sleep for the rest of the second
     sleep 0.$(printf '%04d' $((10000 - 10#$(date +%4N))))
   done
 
That code works, and none of Debian wheezy, stretch and buster as well as RedHat 6 or 7 shows anything, it just works, no noticable load generated. Now, Oracle Linux 7 thinks differently. The above code run there generates between 8 and 15% CPU usage (on fairly recent Intel CPUs, but that shouldn t matter). (CPU usage measured with the highly accurate use of top and looking what it tells ) Whyever.

Fixing Ok, well, the code above isn t all the nicest shell, actually. There is room for improvement. But beware, the older the bash, the less one can fix it.
  • So, first of, there are two useless uses of cat. Bash can do that for us, just use the $(< /PATH/TO/FILE ) way.
  • Oh, Bash5 knows the epoch directly, we can replace the date call for the timestamp and use $ EPOCHSECONDS
  • Too bad Bash4 can t do that. But hey, it s builtin printf can help out, a nice TSTAMP=$(printf %(%s)T\n -1) works.
  • Unfortunately, Bash4.2 and later, not 4.1, and meh, we have a 4.1 system, so that has to stay with the date call there.
Taking that, we end up with 3 different possible versions, depending on the Bash on the system.
obtain5()  
  ## Purest bash version, Bash can tell us epochs directly
  echo $ IFACE _tx.value $ EPOCHSECONDS :$(</sys/class/net/$ IFACE /statistics/tx_bytes) >> $ cache 
  echo $ IFACE _rx.value $ EPOCHSECONDS :$(</sys/class/net/$ IFACE /statistics/rx_bytes) >> $ cache 
  # Sleep for the rest of the second
  sleep 0.$(printf '%04d' $((10000 - 10#$(date +%4N))))
 
obtain42()  
  ## Bash cant tell us epochs directly, but the builtin printf can
  TSTAMP=$(printf '%(%s)T\n' -1)
  echo $ IFACE _tx.value $ TSTAMP :$(</sys/class/net/$ IFACE /statistics/tx_bytes) >> $ cache 
  echo $ IFACE _rx.value $ TSTAMP :$(</sys/class/net/$ IFACE /statistics/rx_bytes) >> $ cache 
  # Sleep for the rest of the second
  sleep 0.$(printf '%04d' $((10000 - 10#$(date +%4N))))
 
obtain41()  
  ## Bash needs help from a tool to get epoch, means one exec() all the time
  TSTAMP=$(date +%s)
  echo $ IFACE _tx.value $ TSTAMP :$(</sys/class/net/$ IFACE /statistics/tx_bytes) >> $ cache 
  echo $ IFACE _rx.value $ TSTAMP :$(</sys/class/net/$ IFACE /statistics/rx_bytes) >> $ cache 
  # Sleep for the rest of the second
  sleep 0.$(printf '%04d' $((10000 - 10#$(date +%4N))))
 
run_acquire()  
   echo "$$" > $ pidfile 
   case $ BASH_VERSINFO[0]  in
     5) while :; do
          obtain5
        done
        ;;
     4) if [[ $ BASHVERSION[1]  -ge 2 ]]; then
          while :; do
            obtain42
          done
        else
          while :; do
            obtain41
          done
        fi
        ;;
   esac
 

Does it help? Oh yes, it does. Oracle Linux 7 appears to use Bash 4.2, so uses obtain42 and hey, removing one date and two cat calls, and it has a sane CPU usage of 0 (again, highly accurate number generated from top ). Appears OL7 is doing heck-what-do-i-know extra, when calling other tools, for whatever gains, removing that does help (who would have thought). (None of RedHat or Oracle Linux has SELinux turned on, so that one shouldn t bite. But it is clear OL7 doing something extra for everything that bash spawns.)

16 January 2022

Russ Allbery: Review: The Brightest Fell

Review: The Brightest Fell, by Seanan McGuire
Series: October Daye #11
Publisher: DAW
Copyright: 2017
ISBN: 0-698-18352-5
Format: Kindle
Pages: 353
This is the eleventh book in the October Daye urban fantasy series, not counting various novellas and side stories. You really cannot start here, particularly given how many ties this book has to the rest of the series. I would like to claim there's some sort of plan or strategy in how I read long series, but there are just a lot of books to read and then I get distracted and three years have gone by. The advantage of those pauses, at least for writing reviews, is that I return to the series with fresh eyes and more points of comparison. My first thought this time around was "oh, these books aren't that well written, are they," followed shortly thereafter by staying up past midnight reading just one more chapter. Plot summaries are essentially impossible this deep into a series, when even the names of the involved characters can be a bit of a spoiler. What I can say is that we finally get the long-awaited confrontation between Toby and her mother, although it comes in an unexpected (and unsatisfying) form. This fills in a few of the gaps in Toby's childhood, although there's not much there we didn't already know. It fills in considerably more details about the rest of Toby's family, most notably her pure-blood sister. The writing is indeed not great. This series is showing some of the signs I've seen in other authors (Mercedes Lackey, for instance) who wrote too many books per year to do each of them justice. I have complained before about McGuire's tendency to reuse the same basic plot structure, and this instance seemed particularly egregious. The book opens with Toby enjoying herself and her found family, feeling like they can finally relax. Then something horrible happens to people she cares about, forcing her to go solve the problem. This in theory requires her to work out some sort of puzzle, but in practice is fairly linear and obvious because, although I love Toby as a character, she can't puzzle her way out of a wet sack. Everything is (mostly) fixed in the end, but there's a high cost to pay, and everyone ends the book with more trauma. The best books of this series are the ones where McGuire manages to break with this formula. This is not one of them. The plot is literally on magical rails, since The Brightest Fell skips even pretending that Toby is an actual detective (although it establishes that she's apparently still working as one in the human world, a detail that I find baffling) and gives her a plot compass that tells her where to go. I don't really mind this since I read this series for emotional catharsis rather than Toby's ingenuity, but alas that's mostly missing here as well. There is a resolution of sorts, but it's the partial and conditional kind that doesn't include awful people getting their just deserts. This is also not a good series entry for world-building. McGuire has apparently been dropping hints for this plot back at least as far as Ashes of Honor. I like that sort of long-term texture to series like this, but the unfortunate impact on this book is a lot of revisiting of previous settings and very little in the way of new world-building. The bit with the pixies was very good; I wanted more of that, not the trip to an Ashes of Honor setting to pick up a loose end, or yet another significant scene in Borderland Books. As an aside, I wish authors would not put real people into their books as characters, even when it's with permission as I'm sure it was here. It's understandable to write a prominent local business into a story as part of the local color (although even then I would rather it not be a significant setting in the story), but having the actual owner and staff show up, even in brief cameos, feels creepy and weird to me. It also comes with some serious risks because real people are not characters under the author's control. (All the content warnings for that link, which is a news story from three years after this book was published.) So, with all those complaints, why did I stay up late reading just one more chapter? Part of the answer is that McGuire writes very grabby books, at least for me. Toby is a full-speed-ahead character who is constantly making things happen, and although the writing in this book had more than the usual amount of throat-clearing and rehashing of the same internal monologue, the plot still moved along at a reasonable clip. Another part of the answer is that I am all-in on these characters: I like them, I want them to be happy, and I want to know what's going to happen next. It helps that McGuire has slowly added characters over the course of a long series and given most of them a chance to shine. It helps even more that I like all of them as people, and I like the style of banter that McGuire writes. Also, significant screen time for the Luidaeg is never a bad thing. I think this was the weakest entry in the series in a while. It wrapped up some loose ends that I wasn't that interested in wrapping up, introduced a new conflict that it doesn't resolve, spent a bunch of time with a highly unpleasant character I didn't enjoy reading about, didn't break much new world-building ground, and needed way more faerie court politics. But some of the banter was excellent, the pixies and the Luidaeg were great, and I still care a lot about these characters. I am definitely still reading. Followed by Nights and Silences. Continuing a pattern from Once Broken Faith, the ebook version of The Brightest Fell includes a bonus novella. (I'm not sure if it's also present in the print version.) "Of Things Unknown": As is usual for the short fiction in this series, this is a side story from the perspective of someone other than Toby. In this case, that's April O'Leary, first introduced all the way back in A Local Habitation, and the novella focuses on loose ends from that novel. Loose ends are apparently the theme of this book. This was... fine. I like April, I enjoyed reading a story from her perspective, and I'm always curious to see how Toby looks from the outside. I thought the plot was strained and the resolution a bit too easy and painless, and I was not entirely convinced by April's internal thought processes. It felt like McGuire left some potential for greater plot complications on the table here, and I found it hard to shake the impression that this story was patching an error that McGuire felt she'd made in the much earlier novel. But it was nice to have an unambiguously happy ending after the more conditional ending of the main story. (6) Rating: 6 out of 10

14 January 2022

Norbert Preining: Future of my packages in Debian

After having been (again) demoted (timed perfectly to my round birthday!) based on flimsy arguments, I have been forced to rethink the level of contribution I want to do for Debian. Considering in particular that I have switched my main desktop to dual-boot into Arch Linux (all on the same btrfs fs with subvolumes, great!) and have run Arch now for several days exclusively, I think it is time to review the packages I am somehow responsible for (full list of packages). After about 20 years in Debian, time to send off quite some stuff that has accumulated over time. KDE/Plasma, frameworks, Gears, and related packages All these packages are group maintained, so there is not much to worry about. Furthermore, a few new faces have joined the team and are actively working on the packages, although mostly on Qt6. I guess that with me not taking action, frameworks, gears, and plasma will fall back over time (frameworks: Debian 5.88 versus current 5.90, gears: Debian 21.08 versus current 21.12, plasma uptodate at the moment). With respect to my packages on OBS, they will probably also go stale over time. Using Arch nowadays I lack the development tools necessary to build Debian packages, and above all, the motivation. I am sorry for all those who have learned to rely on my OBS packages over the last years, bringing modern and uptodate KDE/Plasma to Debian/stable, please direct your complaints at the responsible entities in Debian. Cinnamon As I have written already here, I have reduced my involvement quite a lot, and nowadays Fabio and Joshua are doing the work. But both are not even DM (AFAIR) and I am the only one doing uploads (I got DM upload permissions for it). But I am not sure how long I will continue doing this. This also means that in the near future, Cinnamon will also go stale. TeX related packages Hilmar has DM upload permissions and is very actively caring for the packages, so I don t see any source of concern here. New packages will need to find a new uploader, though. With myself also being part of upstream, I can surely help out in the future with difficult problems. Calibre and related packages Yokota-san (another DM I have sponsored) has DM upload permissions and is very actively caring for the packages, so also here there is not much of concern. Onedrive This is already badly outdated, and I recommend using the OBS builds which are current and provide binaries for Ubuntu and Debian for various versions. ROCm Here fortunately a new generation of developers has taken over maintenance and everything is going smoothly, much better than I could have done, yeah to that! Qalculate related packages These are group maintained, but unfortunately nobody else but me has touched the repos for quite some time. I fear that the packages will go stale rather soon. isync/mbsync I have recently salvaged this package, and use it daily, but I guess it needs to be orphaned sooner or later. CafeOBJ While I am also part of upstream here, I guess it will be orphaned. Julia Julia is group maintained, but unfortunately nobody else but me has touched the repo for quite some time, and we are already far behind the normal releases (and julia got removed from testing). While go stale/orphaned. I recommend installing upstream binaries. python-mechanize Another package that is group maintained in the Python team, but with only me as uploader I guess it will go stale and effectively be orphaned soon. xxhash Has already by orphaned. qpdfview No upstream development, so not much to do, but will be orphaned, too.

Dirk Eddelbuettel: Rcpp 1.0.8: Updated, Strict Headers

rcpp logo The Rcpp team is thrilled to share the news of the newest release 1.0.8 of Rcpp which hit CRAN today, and has already been uploaded to Debian as well. Windows and macOS builds should appear at CRAN in the next few days. This release continues with the six-months cycle started with release 1.0.5 in July 2020. As a reminder, interim dev or rc releases will alwasys be available in the Rcpp drat repo; this cycle there were once again seven (!!) times two as we also tested the modified header (more below). These rolling release tend to work just as well, and are also fully tested against all reverse-dependencies. Rcpp has become the most popular way of enhancing R with C or C++ code. Right now, around 2478 packages on CRAN depend on Rcpp for making analytical code go faster and further, along with 242 in BioConductor. This release finally brings a change we have worked on quite a bit over the last few months. The idea of enforcing the setting of STRICT_R_HEADERS was prososed years ago in 2016 and again in 2018. But making such a chance against a widely-deployed code base has repurcussions, and we were not ready then. Last April, this was revisited in issue #1158. Over the course of numerous lengthy runs of tests of a changed Rcpp package against (essentially) all reverse-dependencies (i.e. packages which use Rcpp) we identified ninetyfour packages in total which needed a change. We provided either a patch we emailed, or a GitHub pull request, to all ninetyfour. And we are happy to say that eighty cases were resolved via a new CRAN upload, with a seven more having merged the pull request but not yet uploaded. Hence, we could make the case to CRAN (who were always CC ed on the monthly nag emails we sent to maintainers of packages needing a change) that an upload was warranted. And after a brief period for their checks and inspection, our January 11 release of Rcpp 1.0.8 arrived on CRAN on January 13. So with that, a big and heartfelt Thank You! to all eighty maintainers for updating their packages to permit this change at the Rcpp end, to CRAN for the extra checking, and to everybody else who I bugged with the numerous emails and updated to the seemingly never-ending issue #1158. We all got this done, and that is a Good Thing (TM). Other than the aforementioned change which will not automatically set STRICT_R_HEADERS (unless opted out which one can), a number of nice pull request by a number of contributors are included in this release: The full list of details follows.

Changes in Rcpp release version 1.0.8 (2022-01-11)
  • Changes in Rcpp API:
    • STRICT_R_HEADERS is now enabled by default, see extensive discussion in #1158 closing #898.
    • A new #define allows default setting of finalizer calls for external pointers (I aki in #1180 closing #1108).
    • Rcpp:::CxxFlags() now quotes the include path generated, (Kevin in #1189 closing #1188).
    • New header files Rcpp/Light, Rcpp/Lighter, Rcpp/Lightest and default Rcpp/Rcpp for fine-grained access to features (and compilation time) (Dirk #1191 addressing #1168).
  • Changes in Rcpp Attributes:
    • A new option signature allows customization of function signatures (Travers Ching in #1184 and #1187 fixing #1182)
  • Changes in Rcpp Documentation:
    • The Rcpp FAQ has a new entry on how not to grow a vector (Dirk in #1167).
    • Some long-spurious calls to RNGSope have been removed from examples (Dirk in #1173 closing #1172).
    • DOI reference in the bibtex files have been updated per JSS request (Dirk in #1186).
  • Changes in Rcpp Deployment:
    • Some continuous integration components have been updated (Dirk in #1174, #1181, and #1190).

Thanks to my CRANberries, you can also look at a diff to the previous release. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. Bugs reports are welcome at the GitHub issue tracker as well (where one can also search among open or closed issues); questions are also welcome under rcpp tag at StackOverflow which also allows searching among the (currently) 2822 previous questions. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

13 January 2022

Daniel Lange: Leveling the playing field for non-native speakers

Wordle game screenshort of bash, grep and pipes

Updates 24.01.2022: What I love about the community is the playful creativity that inspires a game like Wordle and that in turn inspires others to create fun tools around it: Robert Reichel has reverse engineered the Wordle application, so in case you want to play tomorrow's word today .. you can. Or have that one guess "Genius" solution experience. JP Fosterson created a Wordle helper that is very much the Python version of my grep-foo above. In case you play regularly and can use a hand. And Tom Lockwood wrote a Wordle solver also in Python. He blogged about it and ... is pondering to rewrite things in Rust:

I ve decided to explore Rust for this, and so far what was taking 1GB of RAM in Python is taking, literally 1MB in Rust!
Welcome to 2022. 01.02.2022: OMG. Wordle has been bought by the New York Times for "for a price in the low seven figures" (Source). Joey Rees-Hill put it well in The Death of Wordle:
Today s Web is dominated by platforms. The average Web user will spend most of their time on large platforms such as Instagram, Facebook, Twitter, TikTok, Google Drive/Docs, YouTube, Netflix, Spotify, Gmail, and Google Calendar, along with sites operated by large publishers such as The New York Times or The Washington Post. [..]
The Web wasn t always this way. I m not old enough to remember this, but things weren t always so centralized. Web users might run their own small website, and certainly would visit a good variety of smaller sites. With the increasing availability of internet access, the Web has become incredibly commercialized, with a handful of companies concentrating Web activity on their own properties.
Wordle was a small site that gained popularity despite not being part of a corporate platform. It was wonderful to see an independent site gain attention for being simple and fun. Wordle was refreshingly free of attention-manipulating dark patterns and pushy monetization. That s why it s a shame to see it absorbed, to inevitably become just another feature of one large media company s portfolio.
Still kudos to Josh Wardle, a Million Pounds for Wordle. Well done! It was fun while it lasted. Let's see what the next Wordle will be. This one has just been absorbed into the borg collective.

11 January 2022

Russ Allbery: Review: Hench

Review: Hench, by Natalie Zina Walschots
Publisher: William Morrow
Copyright: September 2020
ISBN: 0-06-297859-4
Format: Kindle
Pages: 403
Anna Tromedlov is a hench, which means she does boring things for terrible people for money. Supervillains need a lot of labor to keep their bases and criminal organizations running, and they get that labor the same way everyone else does: through temporary agencies. Anna does spreadsheets, preferably from home on her couch. On-site work was terrifying and she tried to avoid it, but the lure of a long-term contract was too strong. The Electric Eel, despite being a creepy sleazeball, seemed to be a manageable problem. He needed some support at a press conference, which turns out to be code for being a diversity token in front of the camera, but all she should have to do is stand there. That's how Anna ends up holding the mind control device to the head of the mayor's kid when the superheroes attack, followed shortly by being thrown across the room by Supercollider. Left with a complex fracture of her leg that will take months to heal, a layoff notice and a fruit basket from Electric Eel's company, and a vaguely menacing hospital conversation with the police (including Supercollider in a transparent disguise) in which it's made clear to her that she is mistaken about Supercollider's hand-print on her thigh, Anna starts wondering just how much damage superheroes have done. The answer, when analyzed using the framework for natural disasters, is astonishingly high. Anna's resulting obsession with adding up the numbers leads to her starting a blog, the Injury Report, with a growing cult following. That, in turn, leads to a new job and a sponsor: the mysterious supervillain Leviathan. To review this book properly, I need to talk about Watchmen. One of the things that makes superheroes interesting culturally is the straightforwardness of their foundational appeal. The archetypal superhero story is an id story: an almost pure power fantasy aimed at teenage boys. Like other pulp mass media, they reflect the prevailing cultural myths of the era in which they're told. World War II superheroes are mostly all-American boy scouts who punch Nazis. 1960s superheroes are a more complex mix of outsider misfits with a moral code and sarcastic but earnestly ethical do-gooders. The superhero genre is vast, with numerous reinterpretations, deconstructions, and alternate perspectives, but its ur-story is a good versus evil struggle of individual action, in which exceptional people use their powers for good to defeat nefarious villains. Watchmen was not the first internal critique of the genre, but it was the one that everyone read in the 1980s and 1990s. It takes direct aim at that moral binary. The superheroes in Watchmen are not paragons of virtue (some of them are truly horrible people), and they have just as much messy entanglement with the world as the rest of us. It was superheroes re-imagined for the post-Vietnam, post-Watergate era, for the end of the Cold War when we were realizing how many lies about morality we had been told. But it still put superheroes and their struggles with morality at the center of the story. Hench is a superhero story for the modern neoliberal world of reality TV and power inequality in the way that Watchmen was a superhero story for the Iran-Contra era and the end of the Cold War. Whether our heroes have feet of clay is no longer a question. Today, a better question is whether the official heroes, the ones that are celebrated as triumphs of individual achievement, are anything but clay. Hench doesn't bother asking whether superheroes have fallen short of their ideal; that answer is obvious. What Hench asks instead is a question familiar to those living in a world full of televangelists, climate denialism, manipulative advertising, and Facebook: are superheroes anything more than a self-perpetuating scam? Has the good superheroes supposedly do ever outweighed the collateral damage? Do they care in the slightest about the people they're supposedly protecting? Or is the whole system of superheroes and supervillains a performance for an audience, one that chews up bystanders and spits them out mangled while delivering simplistic and unquestioned official morality? This sounds like a deeply cynical premise, but Hench is not a cynical book. It is cynical about superheroes, which is not the same thing. The brilliance of Walschots's approach is that Anna has a foot in both worlds. She works for a supervillain and, over the course of the book, gains access to real power within the world of superheroic battles. But she's also an ordinary person with ordinary problems: not enough money, rocky friendships, deep anger at the injustices of the world and the way people like her are discarded, and now a disability and PTSD. Walschots perfectly balances the tension between those worlds and maintains that tension straight to the end of the book. From the supervillain world, Anna draws support, resources, and a mission, but all of the hope, true morality, and heart of this book comes from the ordinary side. If you had the infrastructure of a supervillain at your disposal, what would you do with it? Anna's answer is to treat superheroes as a destructive force like climate change, and to do whatever she can to drive them out of the business and thus reduce their impact on the world. The tool she uses for that is psychological warfare: make them so miserable that they'll snap and do something too catastrophic to be covered up. And the raw material for that psychological warfare is data. That's the foot in the supervillain world. In descriptions of this book, her skills with data are often called her superpower. That's not exactly wrong, but the reason why she gains power and respect is only partly because of her data skills. Anna lives by the morality of the ordinary people world: you look out for your friends, you treat your co-workers with respect as long as they're not assholes, and you try to make life a bit better for the people around you. When Leviathan gives her the opportunity to put together a team, she finds people with skills she admires, funnels work to people who are good at it, and worries about the team dynamics. She treats the other ordinary employees of a supervillain as people, with lives and personalities and emotions and worth. She wins their respect. Then she uses their combined skills to destroy superhero lives. I was fascinated by the moral complexity in this book. Anna and her team do villainous things by the morality of the superheroic world (and, honestly, by the morality of most readers), including some things that result in people's deaths. By the end of the book, one could argue that Anna has been driven by revenge into becoming an unusual sort of supervillain. And yet, she treats the people around her so much better than either the heroes or the villains do. Anna is fiercely moral in all the ordinary person ways, and that leads directly to her becoming a villain in the superhero frame. Hench doesn't resolve that conflict; it just leaves it on the page for the reader to ponder. The best part about this book is that it's absurdly grabby, unpredictable, and full of narrative momentum. Walschots's pacing kept me up past midnight a couple of times and derailed other weekend plans so that I could keep reading. I had no idea where the plot was going even at the 80% mark. The ending is ambiguous and a bit uncomfortable, just like the morality throughout the book, but I liked it the more I thought about it. One caveat, unfortunately: Hench has some very graphic descriptions of violence and medical procedures, and there's an extended torture sequence with some incredibly gruesome body horror that I thought went on far too long and was unnecessary to the plot. If you're a bit squeamish like I am, there are some places where you'll want to skim, including one sequence that's annoyingly intermixed with important story developments. Otherwise, though, this is a truly excellent book. It has a memorable protagonist with a great first-person voice, an epic character arc of empowerment and revenge, a timely take on the superhero genre that uses it for sharp critique of neoliberal governance and reality TV morality, a fascinatingly ambiguous and unsettled moral stance, a gripping and unpredictable plot, and some thoroughly enjoyable competence porn. I had put off reading it because I was worried that it would be too cynical or dark, but apart from the unnecessary torture scene, it's not at all. Highly recommended. Rating: 9 out of 10

9 January 2022

Dirk Eddelbuettel: Rblpapi 0.3.13: Some Fixes and Documentation

A new version, now at 0.3.13, of the Rblpapi package just arrived at CRAN. Rblpapi provides a direct interface between R and the Bloomberg Terminal via the C++ API provided by Bloomberg (but note that a valid Bloomberg license and installation is required). This is the thirteenth release since the package first appeared on CRAN in 2016. It comprises the PRs from three different contributors (with special thanks once again to Michael Kerber), and extends test and documentation, and extends two function interfaces to control explicitly whether returned lists of length one should be simplified. The list of changes follow below.

Changes in Rblpapi version 0.3.13 (2022-01-09)
  • Add a test for bds (Michael Kerber in #352)
  • Add simplify argument (and option) to bdh and bds (Dirk in #354 closing #353, #351)
  • Improve documentation for bsearch (John in #357 closing #356)

Courtesy of my CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the Rblpapi page. Questions, comments etc should go to the issue tickets system at the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Russ Allbery: Review: Redemptor

Review: Redemptor, by Jordan Ifueko
Series: Raybearer #2
Publisher: Amulet Books
Copyright: 2021
ISBN: 1-68335-720-5
Format: Kindle
Pages: 328
Redemptor is the second half of a duology that started with Raybearer. You could read the first book without the second, but reading the second without the first will not make much sense. I'm going to be a bit elliptical in my plot description since there's a lot of potential for spoilers for the first book. Tarisai has reached a point of stability and power, but she's also committed herself to a goal, one that will right a great historical and ongoing injustice. She's also now in a position to both notice and potentially correct numerous other injustices in the structure of her society, and plans to start by defending those closest to her. But in the midst of her opening gambit to save someone she believes is unjustly imprisoned, the first murderous undead child appears, attacking both Tarisai's fragile sense of security and her self-esteem and self-worth. Before long, she's drowning in feelings of inadequacy and isolation, and her grand plans for reordering the world have turned into an anxiety loop of self-flagellating burnout. I so much wanted to like this book. Argh. I think I see what Ifueko was aiming for, and it's a worthy topic for a novel. In Raybearer, Tarisai got the sort of life that she previously could only imagine, but she's also the sort of person who shoulders massive obligations. Imposter syndrome, anxiety, overwork, and burnout are realistic risks, and are also important topics to write about. There are some nicely subtle touches buried in this story, such as the desire of her chosen family to have her present and happy without entirely understanding why she isn't, and without seeing the urgency that she sees in the world's injustice. The balancing act of being effective without overwhelming oneself is nearly impossible, and Tarisai has very little preparation or knowledgeable support. But this story is told with the subtlety of a sledgehammer, and in a way that felt forced rather than arising naturally from the characters. If the point of emphasis had been a disagreement with her closest circle over when and how much the world should be changed, I think this would be a better book. In the places where this drives the plot, it is a better book. But Ifueko instead externalizes anxiety and depression in the form of obviously manipulative demonic undead children who (mostly) only Tarisai can see, and it's just way too much. Her reactions are manipulated and sometimes externally imposed in a way that turns what should have been a character vs. self plot into a character vs. character plot in which the protagonist is very obviously making bad decisions and the antagonist is an uninteresting cliche. The largest problem I had with this book is that I found it thuddingly obvious, in part because the plot felt like it was on narrowly constrained rails to ensure it hit all of the required stops. When the characters didn't want the plot to go somewhere, they're sidelined, written out of the story, or otherwise forcibly overridden. Tarisai has to feel isolated, so all the people who, according to the events of the previous book and the established world-building rules, would not let her be isolated are pushed out of her life. When this breaks the rules of magic in this world, those rules are off-handedly altered. Characters that could have had their own growth arcs after Raybearer become static and less interesting, since there's no room for them in the plot. Instead, we get all new characters, which gives Redemptor a bit of a cast size problem. Underneath this, there is an occasional flash of great writing. Ifueko chooses to introduce a dozen mostly-new characters to an already large cast and I was still able to mostly keep them straight, which shows real authorial skill. She is very good with short bursts of characterization to make new characters feel fresh and interesting. Even the most irritating of the new characters (Crocodile, whose surprise twist I thought was obvious and predictable) is an interesting archetype to explore in a book about activism and activist burnout. I can see some pieces of a better book here. But I desperately wanted something to surprise me, for Tarisai or one of the other characters to take the plot in some totally unexpected direction the way that Raybearer did. It never happened. That leads directly to another complaint: I liked Raybearer in part because of the freshness of a different mythological system and a different storytelling tradition than what we typically get in fantasy novels. I was hoping for more of the same in Redemptor, which meant I was disappointed when I got a mix of Christianity and Greek mythology. As advertised by Raybearer, the central mythological crisis of Redemptor concerns the Underworld. This doesn't happen until about 80% into the book (which is also a bit of a problem; the ending felt rushed given how central it was to the plot), so I can't talk about it in detail without spoiling it. But what I think I can say is that unfortunately the religious connotations of the title are not an accident. Rather than something novel that builds on the excellent idea of the emi-ehran spirit animal, there is a lot of Christ symbolism mixed with an underworld that could have come from an Orpheus retelling. There's nothing inherently wrong with this (although the Christian bits landed poorly for me), but it wasn't what I was hoping for from the mythology of this world. I rarely talk much about the authors in fiction reviews. I prefer to let books stand on their own without trying too hard to divine the author's original intentions. But here, I think it's worth acknowledging Ifueko's afterword in which she says that writing Redemptor in the middle of a pandemic, major depression, and the George Floyd protests was the most difficult thing she'd ever done. I've seen authors write similar things in afterwords when the effect on the book was minimal or invisible, but I don't think that was the case here. Redemptor is furious, anxious, depressed, and at points despairing, and while it's okay for novels to be all of those things when it's under the author's control, here they felt like emotions that were imposed on the story from outside. Raybearer was an adventure story about found family and ethics that happened to involve a lot of politics. Redemptor is a story about political activism and governance, but written in a universe whose bones are set up for an adventure story. The mismatch bothered me throughout; not only did these not feel like the right characters to tell this story with, but the politics were too simple, too morally clear-cut, and too amenable to easy solutions for a good political fantasy. Raybearer focused its political attention on colonialism. That's a deep enough topic by itself to support a duology (or more), but Redemptor adds in property rights, land reform, economic and social disparity, unfair magical systems, and a grab bag of other issues, and it overwhelms the plot. There isn't space and time to support solutions with sufficient complexity to satisfyingly address the problems. Ifueko falls back on benevolent dictator solutions, and I understand why, but that's not the path to a satisfying resolution in an overtly political fantasy. This is the sort of sequel that leaves me wondering if I can recommend reading the first book and not the second, and that makes me sad. Redemptor is not without its occasional flashes of brilliance, but I did not have fun reading this book and I can't recommend the experience. That said, I think this is a book problem, not an author problem; I will happily read Ifueko's next novel, and I suspect it will be much better. Rating: 5 out of 10

Matthew Garrett: Pluton is not (currently) a threat to software freedom

At CES this week, Lenovo announced that their new Z-series laptops would ship with AMD processors that incorporate Microsoft's Pluton security chip. There's a fair degree of cynicism around whether Microsoft have the interests of the industry as a whole at heart or not, so unsurprisingly people have voiced concerns about Pluton allowing for platform lock-in and future devices no longer booting non-Windows operating systems. Based on what we currently know, I think those concerns are understandable but misplaced.

But first it's helpful to know what Pluton actually is, and that's hard because Microsoft haven't actually provided much in the way of technical detail. The best I've found is a discussion of Pluton in the context of Azure Sphere, Microsoft's IoT security platform. This, in association with the block diagrams on page 12 and 13 of this slidedeck, suggest that Pluton is a general purpose security processor in a similar vein to Google's Titan chip. It has a relatively low powered CPU core, an RNG, and various hardware cryptography engines - there's nothing terribly surprising here, and it's pretty much the same set of components that you'd find in a standard Trusted Platform Module of the sort shipped in pretty much every modern x86 PC. But unlike Titan, Pluton seems to have been designed with the explicit goal of being incorporated into other chips, rather than being a standalone component. In the Azure Sphere case, we see it directly incorporated into a Mediatek chip. In the Xbox Series devices, it's incorporated into the SoC. And now, we're seeing it arrive on general purpose AMD CPUs.

Microsoft's announcement says that Pluton can be shipped in three configurations:as the Trusted Platform Module; as a security processor used for non-TPM scenarios like platform resiliency; or OEMs can choose to ship with Pluton turned off. What we're likely to see to begin with is the former - Pluton will run firmware that exposes a Trusted Computing Group compatible TPM interface. This is almost identical to the status quo. Microsoft have required that all Windows certified hardware ship with a TPM for years now, but for cost reasons this is often not in the form of a separate hardware component. Instead, both Intel and AMD provide support for running the TPM stack on a component separate from the main execution cores on the system - for Intel, this TPM code runs on the Management Engine integrated into the chipset, and for AMD on the Platform Security Processor that's integrated into the CPU package itself.

So in this respect, Pluton changes very little; the only difference is that the TPM code is running on hardware dedicated to that purpose, rather than alongside other code. Importantly, in this mode Pluton will not do anything unless the system firmware or OS ask it to. Pluton cannot independently block the execution of any other code - it knows nothing about the code the CPU is executing unless explicitly told about it. What the OS can certainly do is ask Pluton to verify a signature before executing code, but the OS could also just verify that signature itself. Windows can already be configured to reject software that doesn't have a valid signature. If Microsoft wanted to enforce that they could just change the default today, there's no need to wait until everyone has hardware with Pluton built-in.

The two things that seem to cause people concerns are remote attestation and the fact that Microsoft will be able to ship firmware updates to Pluton via Windows Update. I've written about remote attestation before, so won't go into too many details here, but the short summary is that it's a mechanism that allows your system to prove to a remote site that it booted a specific set of code. What's important to note here is that the TPM (Pluton, in the scenario we're talking about) can't do this on its own - remote attestation can only be triggered with the aid of the operating system. Microsoft's Device Health Attestation is an example of remote attestation in action, and the technology definitely allows remote sites to refuse to grant you access unless you booted a specific set of software. But there are two important things to note here: first, remote attestation cannot prevent you from booting whatever software you want, and second, as evidenced by Microsoft already having a remote attestation product, you don't need Pluton to do this! Remote attestation has been possible since TPMs started shipping over two decades ago.

The other concern is Microsoft having control over the firmware updates. The context here is that TPMs are not magically free of bugs, and sometimes these can have security consequences. One example is Infineon TPMs producing weak RSA keys, a vulnerability that could be rectified by a firmware update to the TPM. Unfortunately these updates had to be issued by the device manufacturer rather than Infineon being able to do so directly. This meant users had to wait for their vendor to get around to shipping an update, something that might not happen at all if the machine was sufficiently old. From a security perspective, being able to ship firmware updates for the TPM without them having to go through the device manufacturer is a huge win.

Microsoft's obviously in a position to ship a firmware update that modifies the TPM's behaviour - there would be no technical barrier to them shipping code that resulted in the TPM just handing out your disk encryption secret on demand. But Microsoft already control the operating system, so they already have your disk encryption secret. There's no need for them to backdoor the TPM to give them something that the TPM's happy to give them anyway. If you don't trust Microsoft then you probably shouldn't be running Windows, and if you're not running Windows Microsoft can't update the firmware on your TPM.

So, as of now, Pluton running firmware that makes it look like a TPM just isn't a terribly interesting change to where we are already. It can't block you running software (either apps or operating systems). It doesn't enable any new privacy concerns. There's no mechanism for Microsoft to forcibly push updates to it if you're not running Windows.

Could this change in future? Potentially. Microsoft mention another use-case for Pluton "as a security processor used for non-TPM scenarios like platform resiliency", but don't go into any more detail. At this point, we don't know the full set of capabilities that Pluton has. Can it DMA? Could it play a role in firmware authentication? There are scenarios where, in theory, a component such as Pluton could be used in ways that would make it more difficult to run arbitrary code. It would be reassuring to hear more about what the non-TPM scenarios are expected to look like and what capabilities Pluton actually has.

But let's not lose sight of something more fundamental here. If Microsoft wanted to block free operating systems from new hardware, they could simply mandate that vendors remove the ability to disable secure boot or modify the key databases. If Microsoft wanted to prevent users from being able to run arbitrary applications, they could just ship an update to Windows that enforced signing requirements. If they want to be hostile to free software, they don't need Pluton to do it.

(Edit: it's been pointed out that I kind of gloss over the fact that remote attestation is a potential threat to free software, as it theoretically allows sites to block access based on which OS you're running. There's various reasons I don't think this is realistic - one is that there's just way too much variability in measurements for it to be practical to write a policy that's strict enough to offer useful guarantees without also blocking a number of legitimate users, and the other is that you can just pass the request through to a machine that is running the appropriate software and have it attest for you. The fact that nobody has actually bothered to use remote attestation for this purpose even though most consumer systems already ship with TPMs suggests that people generally agree with me on that)

comment count unavailable comments

Next.

Previous.