Search Results: "nate"

29 October 2022

Dirk Eddelbuettel: littler 0.3.17 on CRAN: Maintenance

max-heap image The eighteenth release of littler as a CRAN package just landed, following in the now sixteen year history (!!) as a package started by Jeff in 2006, and joined by me a few weeks later. littler is the first command-line interface for R as it predates Rscript. It allows for piping as well for shebang scripting via #!, uses command-line arguments more consistently and still starts faster. It also always loaded the methods package which Rscript only started to do in recent years. littler lives on Linux and Unix, has its difficulties on macOS due to yet-another-braindeadedness there (who ever thought case-insensitive filesystems as a default were a good idea?) and simply does not exist on Windows (yet the build system could be extended see RInside for an existence proof, and volunteers are welcome!). See the FAQ vignette on how to add it to your PATH. A few examples are highlighted at the Github repo, as well as in the examples vignette. This release, coming just a few weeks since the last release in August, heeds to clang-15 a updates one signature to a proper interface. It also contains one kindly contributed patch updating install2.r (and installBioc.r) to cope with a change in R-devel. The full change description follows.

Changes in littler version 0.3.17 (2022-10-29)
  • Changes in package
    • An internal function prototype was updated for clang-15.
  • Changes in examples
    • The install2.r and installBioc. were updated for an update in R-devel (Tatsuya Shima and Dirk in #104).

My CRANberries service provides a comparison to the previous release. Full details for the littler release are provided as usual at the ChangeLog page, and also on the package docs website. The code is available via the GitHub repo, from tarballs and now of course also from its CRAN page and via install.packages("littler"). Binary packages are available directly in Debian as well as soon via Ubuntu binaries at CRAN thanks to the tireless Michael Rutter. Comments and suggestions are welcome at the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dirk Eddelbuettel: gettz 0.0.5 on CRAN: Maintenance

A minor routine update 0.0.5 of gettz arrived on CRAN overnight. gettz provides a possible fallback in situations where Sys.timezone() fails to determine the system timezone. That happened when e.g. the file /etc/localtime somehow is not a link into the corresponding file with zoneinfo data in, say, /usr/share/zoneinfo. Since the package was written (in the fall of 2016), R added a similar extended heuristic approach itself. This release updates a function signature to satisfy the more stringent tests by clang-15, updates the GitHub Action checkout code to suppress a nag, and changes a few remaining http documentation links to https. As with the previous releses: No functional changes, no new code, or new features. Courtesy of my CRANberries, there is a comparison to the previous release. More information is on the gettz page. For questions or comments use the issue tracker off the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Fran ois Marier: Making the mounting of an encrypted /home optional on a home server

I have a computer that serves as a home server as well as a desktop machine. It has an encrypted home directory to protect user files and, in the default configuration, that unfortunately interferes with unattended reboots since someone needs to be present to enter the encryption password. Here's how I added a timeout and made /home optional on that machine. I started by adding a one-minute timeout on the password prompt by adding timeout=60 in my /etc/crypttab:
crypt  UUID=7e12c123-abcd-5555-8c40-900d1f8cc281  none  luks,timeout=60
then I made /home optional by adding nofail to the appropriate mount point in /etc/fstab:
/dev/mapper/crypt  /home  ext4  nodev,noatime,nosuid,nofail  0  2
Before that, the password prompt would timeout but the system would be unable to boot since one of the required partitions had failed to mount. Now, to ensure that I don't accidentally re-create home directories for users when the system is mounted without a /home, I made the /home directory on the non-encrypted drive read-only:
umount /home
cd /home
chmod a-w .
Finally, with all of this in place, I was now happy to configure the machine to automatically reboot after a kernel panic by putting the following in /etc/sysctl.d/local.conf:
# Automatic reboot 10 seconds after a kernel panic
kernel.panic = 10
since I know that the machine will come back up just fine and that all services will be running. I simply won't be able to log into that machine as any other user than root until I manually unlock and mount /home.

Russ Allbery: Review: The Last Continent

Review: The Last Continent, by Terry Pratchett
Series: Discworld #22
Publisher: Harper
Copyright: 1998
Printing: May 2014
ISBN: 0-06-228019-8
Format: Mass market
Pages: 392
This is the 22nd Discworld novel and follows Interesting Times in internal continuity. Like some of the other Rincewind novels, it stands alone well enough that you could arguably start reading here, but I have no idea why you'd want to. When we last saw Rincewind, he was being magically yanked out of the Agatean Empire. The intent was to swap him with a cannon and land him back in Ankh-Morpork, but an unfortunate expansion of the spell to three targets instead of two meant that a kangaroo had a very bad day. Ever since then, Rincewind has been trying to survive the highly inhospitable land of FourEcks (XXXX), so called because no one in Ankh-Morpork knows where it is. The faculty at the Unseen University didn't care enough about Rincewind to bother finding him until the Librarian fell sick. He's feverish and miserable, but worse, he's lost control of his morphic function, which means that he's randomly turning into other things and is unable to take care of the books. When those books are magical, this is dangerous. One possible solution is to stabilize the Librarian's form with a spell, but to do that they need his real name. The only person who might know it is the former assistant librarian: Rincewind. I am increasingly convinced that one of the difficulties in getting people hooked on Discworld is that the series starts with two Rincewind books, and the Rincewind books just aren't very good. The fundamental problem is that Rincewind isn't a character, he's a gag. Discworld starts out as mostly gags, but then the characterization elsewhere gets deeper, the character interactions become more complex, and Pratchett adds more and more philosophy. That, not the humor, is what I think makes these books worth reading. But none of this applies to Rincewind. By this point, he's been the protagonist of six novels, and still the only thing I know about him is that he runs away from everything. Other than that, he's just sort of... there. In the better Rincewind novels, some of the gap is filled by Twoflower, the Luggage, Cohen the barbarian, the Librarian (who sadly is out of commission for most of this book), or the Unseen University faculty. But they're all supporting characters. Most of them are also built around a single (if better) gag. As a result, the Rincewind books tend more towards joke collections than the rest of Discworld. There isn't a philosophical or characterization through line to hold them together. The Last Continent is, as you might have guessed, a parody of Australia. And by that I mean it's a mash-up of Crocodile Dundee, Mad Max, The Adventures of Priscilla, Queen of the Desert, and every dad joke about Australia that you've heard. Pratchett loves movie references and I do not love movie references, so there's always part of his books that doesn't click for me, but this one was just Too Much. Yes, everything in Australia is poisonous. Yes, Australians talk funny. Oh look, there's another twist on a Crocodile Dundee quote. Yes, yes, that's a knife. Gah. The Rincewind sections were either confusing (there's some sort of drug-trip kangaroo god because reasons) or cliched and boring. Sometimes both. The second plot, following the Unseen University faculty in their inept attempts to locate Rincewind, is better. Their bickering is still a bit one-trick and works better in the background of stronger characters (such as Death and Susan), but Pratchett does make their oblivious overconfidence entertaining. It's enough to sustain half of the book, but not enough to make up for the annoyances of the Rincewind plot. To his credit, I think Pratchett was really trying to say something interesting in this novel about Discworld metaphysics. There are bits in the Australian plot that clearly are references to Aboriginal beliefs, which I didn't entirely follow but which I'm glad were in there. The Unseen University faculty showing up in the middle of a creation myth and completely misunderstanding it was a good scene. But the overall story annoyed me and failed to hold my interest. I don't feel qualified to comment on the Priscilla scenes, since I've never seen the movie and have only a vague understanding of its role in trans history. I'm not sure his twists on the story quite worked, but I'm glad that Pratchett is exploring gender; that wasn't as common when these books were written. Overall, though, this was forgettable and often annoying. There are a few great lines and a few memorable bits in any Pratchett book, including this one, but the Rincewind books just aren't... good. Not like the rest of the series, at least. I will be very happy to get back to the witches in the next book. Followed in publication order by Carpe Jugulum, and later thematically by The Last Hero. Rating: 5 out of 10

28 October 2022

Antoine Beaupr : Debating VPN options

In my home lab(s), I have a handful of machines spread around a few points of presence, with mostly residential/commercial cable/DSL uplinks, which means, generally, NAT. This makes monitoring those devices kind of impossible. While I do punch holes for SSH, using jump hosts gets old quick, so I'm considering adding a virtual private network (a "VPN", not a VPN service) so that all machines can be reachable from everywhere. I see three ways this can work:
  1. a home-made Wireguard VPN, deployed with Puppet
  2. a Wireguard VPN overlay, with Tailscale or equivalent
  3. IPv6, native or with tunnels
So which one will it be?

Wireguard Puppet modules As is (unfortunately) typical with Puppet, I found multiple different modules to talk with Wireguard.
module score downloads release stars watch forks license docs contrib issue PR notes
halyard 3.1 1,807 2022-10-14 0 0 0 MIT no requires firewall and Configvault_Write modules?
voxpupuli 5.0 4,201 2022-10-01 2 23 7 AGPLv3 good 1/9 1/4 1/61 optionnally configures ferm, uses systemd-networkd, recommends systemd module with manage_systemd to true, purges unknown keys
abaranov 4.7 17,017 2021-08-20 9 3 38 MIT okay 1/17 4/7 4/28 requires pre-generated private keys
arrnorets 3.1 16,646 2020-12-28 1 2 1 Apache-2 okay 1 0 0 requires pre-generated private keys?
The voxpupuli module seems to be the most promising. The abaranov module is more popular and has more contributors, but it has more open issues and PRs. More critically, the voxpupuli module was written after the abaranov author didn't respond to a PR from the voxpupuli author trying to add more automation (namely private key management). It looks like setting up a wireguard network would be as simple as this on node A:
wireguard::interface   'wg0':
  source_addresses => ['2003:4f8:c17:4cf::1', '149.9.255.4'],
  public_key       => $facts['wireguard_pubkeys']['nodeB'],
  endpoint         => 'nodeB.example.com:53668',
  addresses        => [ 'Address' => '192.168.123.6/30', , 'Address' => 'fe80::beef:1/64' ,],
 
This configuration come from this pull request I sent to the module to document how to use that fact. Note that the addresses used here are examples that shouldn't be reused and do not confirm to RFC5737 ("IPv4 Address Blocks Reserved for Documentation", 192.0.2.0/24 (TEST-NET-1), 198.51.100.0/24 (TEST-NET-2), and 203.0.113.0/24 (TEST-NET-3)) or RFC3849 ("IPv6 Address Prefix Reserved for Documentation", 2001:DB8::/32), but that's another story. (To avoid boostrapping problems, the resubmit-facts configuration could be used so that other nodes facts are more immediately available.) One problem with the above approach is that you explicitly need to take care of routing, network topology, and addressing. This can get complicated quickly, especially if you have lots of devices, behind NAT, in multiple locations (which is basically my life at home, unfortunately). Concretely, basic Wireguard only support one peer behind NAT. There are some workarounds for this, but they generally imply a relay server of some sort, or some custom registry, it's kind of a mess. And this is where overlay networks like Tailscale come in.

Tailscale Tailscale is basically designed to deal with this problem. It's not fully opensource, but pretty close, and they have an interesting philosophy behind that. The client is opensource, and there is an opensource version of the server side, called headscale. They have recently (late 2022) hired the main headscale developer while promising to keep supporting it, which is pretty amazing. Tailscale provides an overlay network based on Wireguard, where each peer basically has a peer-to-peer encrypted connexion, with automatic key rotation. They also ship a multitude of applications and features on top of that like file sharing, keyless SSH access, and so on. The authentication layer is based on an existing SSO provider, you don't just register with Tailscale with new account, you login with Google, Microsoft, or GitHub (which, really, is still Microsoft). The Headscale server ships with many features out of that:
  • Full "base" support of Tailscale's features
  • Configurable DNS
    • Split DNS
    • MagicDNS (each user gets a name)
  • Node registration
    • Single-Sign-On (via Open ID Connect)
    • Pre authenticated key
  • Taildrop (File Sharing)
  • Access control lists
  • Support for multiple IP ranges in the tailnet
  • Dual stack (IPv4 and IPv6)
  • Routing advertising (including exit nodes)
  • Ephemeral nodes
  • Embedded DERP server (AKA NAT-to-NAT traversal)
Neither project (client or server) is in Debian (RFP 972439 for the client, none filed yet for the server), which makes deploying this for my use case rather problematic. Their install instructions are basically a curl bash but they also provide packages for various platforms. Their Debian install instructions are surprisingly good, and check most of the third party checklist we're trying to establish. (It's missing a pin.) There's also a Puppet module for tailscale, naturally. What I find a little disturbing with Tailscale is that you not only need to trust Tailscale with authorizing your devices, you also basically delegate that trust also to the SSO provider. So, in my case, GitHub (or anyone who compromises my account there) can penetrate the VPN. A little scary. Tailscale is also kind of an "all or nothing" thing. They have MagicDNS, file transfers, all sorts of things, but those things require you to hook up your resolver with Tailscale. In fact, Tailscale kind of assumes you will use their nameservers, and have suffered great lengths to figure out how to do that. And naturally, here, it doesn't seem to work reliably; my resolv.conf somehow gets replaced and the magic resolution of the ts.net domain fails. (I wonder why we can't opt in to just publicly resolve the ts.net domain. I don't care if someone can enumerate the private IP addreses or machines in use in my VPN, at least I don't care as much as fighting with resolv.conf everywhere.) Because I mostly have access to the routers on the networks I'm on, I don't think I'll be using tailscale in the long term. But it's pretty impressive stuff: in the time it took me to even review the Puppet modules to configure Wireguard (which is what I'll probably end up doing), I was up and running with Tailscale (but with a broken DNS, naturally). (And yes, basic Wireguard won't bring me DNS either, but at least I won't have to trust Tailscale's Debian packages, and Tailscale, and Microsoft, and GitHub with this thing.)

IPv6 IPv6 is actually what is supposed to solve this. Not NAT port forwarding crap, just real IPs everywhere. The problem is: even though IPv6 adoption is still growing, it's kind of reaching a plateau at around 40% world-wide, with Canada lagging behind at 34%. It doesn't help that major ISPs in Canada (e.g. Bell Canada, Videotron) don't care at all about IPv6 (e.g. Videotron in beta since 2011). So we can't rely on those companies to do the right thing here. The typical solution here is often to use a tunnel like HE's tunnelbroker.net. It's kind of tricky to configure, but once it's done, it works. You get end-to-end connectivity as long as everyone on the network is on IPv6. And that's really where the problem lies here; the second one of your nodes can't setup such a tunnel, you're kind of stuck and that tool completely breaks down. IPv6 tunnels also don't give you the kind of security a VPN provides as well, naturally. The other downside of a tunnel is you don't really get peer-to-peer connectivity: you go through the tunnel. So you can expect higher latencies and possibly lower bandwidth as well. Also, HE.net doesn't currently charge for this service (and they've been doing this for a long time), but this could change in the future (just like Tailscale, that said). Concretely, the latency difference is rather minimal, Google:
--- ipv6.l.google.com ping statistics ---
10 packets transmitted, 10 received, 0,00% packet loss, time 136,8ms
RTT[ms]: min = 13, median = 14, p(90) = 14, max = 15
--- google.com ping statistics ---
10 packets transmitted, 10 received, 0,00% packet loss, time 136,0ms
RTT[ms]: min = 13, median = 13, p(90) = 14, max = 14
In the case of GitHub, latency is actually lower, interestingly:
--- ipv6.github.com ping statistics ---
10 packets transmitted, 10 received, 0,00% packet loss, time 134,6ms
RTT[ms]: min = 13, median = 13, p(90) = 14, max = 14
--- github.com ping statistics ---
10 packets transmitted, 10 received, 0,00% packet loss, time 293,1ms
RTT[ms]: min = 29, median = 29, p(90) = 29, max = 30
That is because HE.net peers directly with my ISP and Fastly (which is behind GitHub.com's IPv6, apparently?), so it's only 6 hops away. While over IPv4, the ping goes over New York, before landing AWS's Ashburn, Virginia datacenters, for a whopping 13 hops... I managed setup a HE.net tunnel at home, because I also need IPv6 for other reasons (namely debugging at work). My first attempt at setting this up in the office failed, but now that I found the openwrt.org guide, it worked... for a while, and I was able to produce the above, encouraging, mini benchmarks. Unfortunately, a few minutes later, IPv6 just went down again. And the problem with that is that many programs (and especially OpenSSH) do not respect the Happy Eyeballs protocol (RFC 8305), which means various mysterious "hangs" at random times on random applications. It's kind of a terrible user experience, on top of breaking the one thing it's supposed to do, of course, which is to give me transparent access to all the nodes I maintain. Even worse, it would still be a problem for other remote nodes I might setup where I might not have acess to the router to setup the tunnel. It's also not absolutely clear what happens if you setup the same tunnel in two places... Presumably, something is smart enough to distribute only a part of the /48 block selectively, but I don't really feel like going that far, considering how flaky the setup is already.

Other options If this post sounds a little biased towards IPv6 and Wireguard, it's because it is. I would like everyone to migrate to IPv6 already, and Wireguard seems like a simple and sound system. I'm aware of many other options to make VPNs. So before anyone jumps in and says "but what about...", do know that I have personnally experimented with:
  • tinc: nice, automatic meshing, used for the Montreal mesh, serious design flaws in the crypto that make it generally unsafe to use; supposedly, v1.1 (or 2.0?) will fix this, but that's been promised for over a decade by now
  • ipsec, specifically strongswan: hard to configure (especially configure correctly!), harder even to debug, otherwise really nice because transparent (e.g. no need for special subnets), used at work, but also considering a replacement there because it's a major barrier to entry to train new staff
  • OpenVPN: mostly used as a client for [VPN service][]s like Riseup VPN or Mullvad, mostly relevant for client-server configurations, not really peer-to-peer, shared secrets or TLS, kind of an hassle to maintain, see also SoftEther for an alternative implementation
All of those solutions have significant problems and I do not wish to use any of those for this project. Also note that Tailscale is only one of many projects laid over Wireguard to do that kind of thing, see this LWN review for others (basically NetbBird, Firezone, and Netmaker).

Future work Those are options that came up after writing this post, and might warrant further examination in the future.
  • Meshbird, a "distributed private networking" with little information about how it actually works other than "encrypted with strong AES-256"
  • Nebula, "A scalable overlay networking tool with a focus on performance, simplicity and security", written by Slack people to replace IPsec, docs, runs as an overlay for Slack's 50k node network, only packaged in Debian experimental, lagging behind upstream (1.4.0, from May 2021 vs upstream's 1.6.1 from September 2022), requires a central CA, Golang, I'm in "wait and see" mode for now
  • n2n: "layer two VPN", seems packaged in Debian but inactive
  • ouroboros: "peer-to-peer packet network prototype", sounds and seems complicated
  • QuickTUN is interesting because it's just a small wrapper around NaCL, and it's in Debian... but maybe too obscure for my own good
  • unetd: Wireguard-based full mesh networking from OpenWRT, not in Debian
  • vpncloud: "high performance peer-to-peer mesh VPN over UDP supporting strong encryption, NAT traversal and a simple configuration", sounds interesting, not in Debian
  • Yggdrasil: actually a pretty good match for my use case, but I didn't think of it when starting the experiments here; packaged in Debian, with the Golang version planned, Puppet module; major caveat: nodes exposed publicly inside the global mesh unless configured otherwise (firewall suggested), requires port forwards, alpha status

Conclusion Right now, I'm going to deploy Wireguard tunnels with Puppet. It seems like kind of a pain in the back, but it's something I will be able to reuse for work, possibly completely replacing strongswan. I have another Puppet module for IPsec which I was planning to publish, but now I'm thinking I should just abort that and replace everything with Wireguard, assuming we still need VPNs at work in the future. (I have a number of reasons to believe we might not need any in the near future anyways...)

Dirk Eddelbuettel: RcppAnnoy 0.0.20 on CRAN: Maintenance

annoy image Another minor maintenance release, now at version 0.0.20, of RcppAnnoy has arrived on CRAN. RcppAnnoy is the Rcpp-based R integration of the nifty Annoy library by Erik Bernhardsson. Annoy is a small and lightweight C++ template header library for very fast approximate nearest neighbours originally developed to drive the Spotify music discovery algorithm. This release only contains internal changes to please, respectively clang-15 and (macOS) Xcode 14 (one of which PRed upstream too). No changes in package functionality. Detailed changes follow.

Changes in version 0.0.20 (2022-10-27)
  • Minor tweaks to appease clang-15 and Xcode 14

Courtesy of my CRANberries, there is also a diffstat report for this release. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

26 October 2022

Scarlett Gately Moore: KDE Gear snaps round 3!

While trying to stay warm in our first snow of the year, I got several apps tested and released in round 3 of https://www.scarlettgatelymoore.dev/kde-gear-snaps-round-2/ ! All of these are being ( re ) tested on both arm64 and amd64. Hence, the release is going slower than it will in the future. Thank you for your patience. More rounds of bug gardening has been done, if you find bugs, or know one of your bugs is related to it being a snap, please assign to me. https://bugs.kde.org Please consider a donation so I may bring many more apps your way. https://www.patreon.com/sgmoore I am working on one time donations.

Dirk Eddelbuettel: RQuantLib 0.4.17 on CRAN: Maintenance

A new release 0.4.17 of RQuantLib arrived at CRAN earlier today, and has been uploaded to Debian as well. QuantLib is a very comprehensice free/open-source library for quantitative finance; RQuantLib connects it to the R environment and language. The release of RQuantLib comes five months after the previous maintenance, and brings a somewhat humurous upgrade from a default C++ standard of C++11 to C++14. We waited so long for C++11 to become available for R (which happened eventually when g++ 4.9 was no longer the default on Windows) and now it has become a constraint!! QuantLib 1.28, released today actually switched to C++14 as a minimum required. R also supports this as the default, but we still had C++11 hardwired so this quick maintenance release does away with that.

Changes in RQuantLib version 0.4.17 (2022-10-25)
  • Switch compilation to C++14 which is required by QuantLib 1.28 and, while standard with R 4.2.*, may be needed for R 4.1.*

Courtesy of my CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the RQuantLib page. Questions, comments etc should go to the new rquantlib-devel mailing list. Issue tickets can be filed at the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

25 October 2022

Dirk Eddelbuettel: nanotime 0.3.7 on CRAN: Enhancements

A new version of our nanotime package arrived at CRAN today as version 0.3.7. nanotime relies on the RcppCCTZ package (as well as the RcppDate package for additional C++ operations) and offers efficient high(er) resolution time parsing and formatting up to nanosecond resolution, and the bit64 package for the actual integer64 arithmetic. Initially implemented using the S3 system, it has benefitted greatly from a rigorous refactoring by Leonardo who not only rejigged nanotime internals in S4 but also added new S4 types for periods, intervals and durations. This release adds a few more operators, plus some other fixes, that were contributed in several PRs by Trevor Davis. The NEWS snippet has the full details.

Changes in version 0.3.7 (2022-10-23)
  • Update mkdocs for material docs generator (Dirk in #102)
  • Use inherits() instead comparing to class() (Trevor Davis in #104)
  • Set default arguments in nanoduration() (Trevor Davis in #105)
  • Add as.nanoduration.difftime() support (Trevor Davis in #106)
  • Add +/- methods for nanotime and difftime objects (Trevor Davis in #110 closing #108, #107)

Thanks to my CRANberries there is also a diff to the previous version. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository and all documentation is provided at the nanotime documentation site. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

24 October 2022

Russ Allbery: Review: Nona the Ninth

Review: Nona the Ninth, by Tamsyn Muir
Series: The Locked Tomb #3
Publisher: Tordotcom
Copyright: 2022
ISBN: 1-250-85412-1
Format: Kindle
Pages: 480
Nona the Ninth is the third book of the Locked Tomb series and entirely pointless to read if you have not read the series to date. It completely spoils the previous books, assuming you would be able to figure out who these people were and why you should care about them. This is only for readers who are already invested. This series was originally supposed to be a trilogy, and this book was supposed to be Alecto the Ninth. Muir says in the acknowledgments, and has said at more length elsewhere, that Nona changed all of her plans and demanded her own book. Hence this book, postponing the end of the series and lengthening it to four books. After reading it, I understand why Muir decided to write a whole book about Nona. She's an interesting character in ways that wouldn't have come out if she was a small part of the concluding book. Unfortunately, it's also obvious that this book wasn't part of the plan. It's not entirely correct to say that Nona the Ninth is devoid of series plot, but the plot advances very little, and mostly at the end. Instead, we get Nona, who is physically a teenager who acts like someone several years younger, most of the time. She lives with her family (who I won't name to avoid spoilers for Harrow the Ninth), helps at a local school (although her level of understanding is about that of the students), and is a member of a kid's gang. She also has dreams every night about a woman with a painted face, dreams her family are very interested in. This sounds weirdly normal for this series, but Nona and her family live in a war-torn city full of fighting, refugees, and Blood of Eden operatives. The previous books of the series took place in the rarefied spaces of the Houses. Here we see a bit of the rest of the universe, although it's not obvious at first what we're looking at and who these people are. Absolutely no concession is made to the reader's fading memory, so expect to need either a re-read, help from friends with better memories, or quality time with a wiki. And, well, good luck with the latter if you've not already read this book, since the Locked Tomb Wiki has now been updated with spoilers for Nona. The other challenge, besides memory for the plot, is that this book is told from a tight third-person focus on Nona, and Nona is not a very reliable narrator. She doesn't lie, exactly, but she mostly doesn't understand what's going on, often doesn't care, and tends not to focus on what the reader is the most interested in. Nona is entirely uninterested in developing the series plot. Her focus is on her child friends (who are moderately interesting but not helpful if you're trying to figure out the rest of the story) and the other rhythms of a strange life that's normal to her. For me at least, that meant the first half of this book involved a lot of "what the heck is going on and why do I care about any of this?" I liked Harrow the Ninth a lot, despite how odd and ambiguous it was, but I was ready for revelations and plot coherence and was not thrilled by additional complexity, odd allusions, and half-revealed details. I didn't mind the layers of complexity added on by Harrow, but for me Nona was a bit too much and I started getting frustrated rather than intrigued. We do, at last, get most of the history of this universe, including the specific details of how John became God Emperor and how the Houses were founded. That happens in odd interludes with a forced and somewhat artificial writing style, but it's more straightforward and comprehensible than I feared at first. The pace of the story picks up considerably towards the end of the book, finally providing the plot momentum that I was hoping for. Unfortunately, it also gets more cryptic at the end of the book in ways that I didn't enjoy. The epilogue, which is vital to understanding the climax of the novel, took me three readings before I think I understood what happened. If you preferred the clarity of Gideon the Ninth, be warned that Nona is more like Harrow and Muir seems to be making the plot more cryptic as she goes. I am hoping this trend reverses in Alecto the Ninth. This book made me grumpy. Nona is okay as a character, but the characters in this series that I really like mostly do not appear or appear in heavily damaged and depressing forms. Muir does bring back a couple of my favorite characters, but then does something to them that's a major spoiler but that I think was intended to be a wonderful moment for them and instead left me completely cold and unhappy. There are still some great moments of humor, but overall it felt more strained. That said, I still had tons of fun discussing this book and its implications with friends who were reading it at the same time. I think that is the best way to read this series. Muir is being intentionally confusing and is inserting a blizzard of references. Some of them are pop culture jokes, but some of them are deep plot clues, and I'm not up to deciphering them all by myself. Working through them with other people is much more fun. (It also gives me an opportunity to feel smug about guessing correctly what was happening at the end of Harrow the Ninth, when I'm almost never the person who makes correct guesses about that sort of thing.) I think your opinion of this one will depend on how much you like Nona as a character, how much patience you have for the postponement of plot resolution, and how much tolerance you have for even more cryptic references. I'm still invested in this series until the end, but this was not my favorite installment. I suspect it (and the rest of the series) would benefit immensely from re-reading, but life is short and my reading backlog is long. What Muir is doing is interesting and has a lot of depth, but she's asking quite a lot of the reader. Content warning: Nona has an eating disorder, which occupied rather more of my mental space while reading this book than I was comfortable with. Followed by Alecto the Ninth, which does not have a publication date scheduled as of this writing. Rating: 7 out of 10

23 October 2022

Dirk Eddelbuettel: RDieHarder 0.2.4 on CRAN: Packaging Updates

An new version 0.2.4 of the random-number generator tester RDieHarder (based on the DieHarder suite developed / maintained by Robert Brown with contributions by David Bauer and myself along with other contributors) is now on CRAN. This release comes ten months after the previous release 0.2.3. It is once more related to R and requested CRAN changes as clang-15 brings additional warnings concerning -Wstrict-prototyping. This make use of C more solid, but it was a metric ton of work (see pull request #8). Thanks to CRANberries, you can also look at the most recent diff. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

20 October 2022

Dirk Eddelbuettel: qlcal 0.0.3 on CRAN: Maintenance and Updates

The third release of the still pretty new qlcal package arrivied at CRAN today. qlcal is based on the calendaring subset of QuantLib. It is provided (for the R package) as a set of included files, so the package is self-contained and does not depend on an external QuantLib library (which can be demanding to build). qlcal covers over sixty country / market calendars and can compute holiday lists, its complement (i.e. business day lists) and much more. This release fixes a small bug affecting one function, brings calendar updates from QuantLib 1.27 and 1.28, and applies a little bit of polish to appease clang++-15.

Changes in version 0.0.3 (2022-10-19)
  • Correct the isBusinessDay() functionality (Fixes #2)
  • Update Australia and Saudi Arabia calendars from QuantLib 1.27
  • Update United Kingdom calendar from QuantLib 1.28
  • Convert one source file to utf-8 to appease clang-15

See the project page and package documentation for more details, and more examples. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dirk Eddelbuettel: RcppQuantuccia 0.1.1 on CRAN: Maintenance

A minor release of RcppQuantuccia arrived on CRAN today. RcppQuantuccia started from the Quantuccia header-only subset / variant of QuantLib which it brings it to R. This project validated the idea of making the calendaring functionality of QuantLib available in a more compact and standalone project which we now do with qlcal which can be seen as a successor to this. This release merely updates a source file to proper encoding as clang++-15 would otherwise warn. The complete list changes for this release follows.

Changes in version 0.1.1 (2022-10-19)
  • Minor code reorganization splitting off calendars.cpp
  • Convert the Argentinian calendar sources files as utf-8 to appease clang++-15
  • Advertise the qlcal package as an alternative

Courtesy of CRANberries, there is also a diffstat report relative to the previous release. More information is on the RcppQuantuccia page. Issues and bugreports should go to the GitHub issue tracker. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

19 October 2022

Shirish Agarwal: Pune Rains, Uncosted Budgets, Hearing Loss Covid, Fracking

Pune Rains Lemme start with a slightly funny picture that tells as much about Pune, my city as anything else does.
Pune- Leave your attitude behind, we have our own
This and similar tags, puns and whatnot you will find if you are entering Pune from the road highway. You can also find similar similar symbols and Puns all over the city and they are partly sarcasm and ironic and partly the truth. Puneities work from the attitude that they know everything rather than nothing, including yours truly  . What is the basis of that or why is there such kind of confidence I have no clue or idea, it is what it is. Approximately 24 hrs. ago, apparently we had a cloudburst. What I came to know later is that we got 100 mm of rain. Sharing from local news site. Much more interesting was a thread made on Reddit where many people half-seriously asked where they can buy a boat. One of the reasons being even if it s October, in fact, we passed middle of October and it s still raining. Even today in the evening, it rained for quite a while. As I had shared in a few blog posts before, June where rains should have started, it didn t, it actually started late July or even August, so something has shifted. The current leadership does not believe in Anthropogenic Climate Change or human activity induced climate change even though that is a reality. I could share many links and even using the term above should give links to various studies. Most of the people who are opposed to it are either misinformed or influenced from the fossil fuel industry. Again, could share many links, but will share just one atm. I have talked to quite a few people about it but nobody has ever been able to give a convincing answer as to why GM had to crush the cars. Let s even take the argument that it was the worst manufactured car in history and there have been quite a few, have the others been crushed? If not, then the reason shared or given by most people sounds hollow. And if you look into it, they had an opportunity that they let go, and now most of them are scrambling and yet most of the legacy auto manufacturers will be out of existence if they don t get back into the game in the next 2-3 years. There have been a bunch of announcements but we are yet to see. The Chinese though have moved far ahead, although one has to remark that they have been doing that for the last decade, so they have a 10-year head start, hardly surprising then. But I need to get back to the subject, another gentleman on Reddit remarked that if you start to use boat, and others start to use boat, then the Govt. will tax it. In fact, somebody had shared the below the other day
Different types of taxes collected by GOI
Many of the taxes that I have shared above are by the Modi Govt. who came on the platform, manifesto that once they come to power they will reduce taxes for the common man, they have reduced taxes but only for the Corporates. For the common man, the taxes have only gone up, both direct tax and indirect tax. Any reference to the Tory party who have also done similar things and have also shared that it is labor who had done large expenditures even though they have been 8 years in power, I am sure for most is purely coincidental. Incidentally, that is the same tack that was taken even by the Republican party. They all like to give tax benefits to the 1% while for the rest is austerity claiming some reason, even if it has been proven to be false.
Corporate Tax Rate, Revenue Loss to Govt.
The figures mentioned above are findings of parliamentary panel so nobody can accuse of anybody having a bias. Also, I probably had shared this but still feel the need to re-share it as people still believe that 2G scam happened even though there are plenty of arguments I can share to prove how it was all a fabricated lie.
Vinod Rai Mafinama in Uttarakhand High Court.
Part 2 of the same Mafinama.
How pathetic Mr. Rai s understanding of economics is can be gauged from the fact that he was made Chairman of IDFC and subsequently had to be thrown out. That whole lie was engineered to throw UPA out and it worked. There are and have been many such coincidences happening over the last 8 years, parallel stories happening in India and UK. This was just yesterday, about a year back Air India was given back to the Tatas, There was controversy about the supposed auctions as only Indigo was the only other party allowed to be at auction but not allowed to buy but more as a spectator as they already have 60% of the Indian civil aviation market. And there was lot of cheering from the Govt. side that finally Air India has been bought home to its true owners, the Tatas. The Tatas too started cheering and sharing how they will take down all the workers, worker unions and everything will be happy glory within a year. In fact, just couple of days back they shared new plans. Btw for the takeover of Air India, they had bought loans from the Banks and they are in the category of too big to fail. As I have shared couple of times before, RBI has not shared any inspection reports of nationalized or private banks after 2013/14. While by law, RBI is supposed to do inspection reports every 3 months and share them in the Public domain. And if you ask any of their supporter, for everything they will say UPA did x or y, which only goes to show morally bankrupt the present Govt. is. Coming back to the topic, before I forget, the idea of sharing their plans is so that they can again borrow money from the banks. But that is not the only story. Just one day back, Smita Prakash, one of the biggest cheerleaders of the present Govt. (she is the boss at Asia News International (ANI)) posted how Air India had treated her sister and other 21 passengers. Basically, they had bought business tickets but the whole cabin was dirty, they complained and they were forced to sit in economy class, not just her cousin sister but the other 21 odd passengers too. Of course Ms Smita became calm as her sister was given free air tickets on Vistara and other goodies. Of course, after that she didn t post anything about the other 21 odd passengers after that. And yes, I understand she is supposed to be a reporter but as can be seen from the twitter thread, there is or was no follow up. Incidentally, she is one of many who has calling others about Revdi culture (freebies to the masses.) but guess that only applies to other people not her or her sister. Again, if there are any coincidences of similar nature in the UK or when Trump was P.M. of the U.S. they are just coincidental .

The Uncosted budget India and the UK have many parallels, it s mind boggling. Before we get into the nitty-gritties, saw something that would be of some interest to the people here.
For those who might not be able to see above, apparently there is place in UK called Tufton Street where there are quite a few organizations that are shadowy and whose finances are not known as to how they are financed. Ms. Truss and quite a few of the people in the cabinet are from the same shadowy organizations. Mr. Kwasi Kwarteng, the just-departed chancellor is and was part of the same group. Now even for me it was a new term to learn and understand what is an uncosted budget is. To make it much more easier I share the example using a common person who goes to the bank for a loan

Now M/S X wants a loan of say Rs. 1000/- for whatever reason. He/she/they go to the bank and asks give me a loan of say INR 1000/- The banks asks them to produce a statement of accounts to show what their financial position is. They produce a somewhat half-filled statement of accounts In which all liabilities are shown, but incomes are not. The bank says you already have so much liabilities, how are you gonna pay those, accounts have to be matched otherwise you are not solvent. M/s X adamantly refuses to do any changes citing that they don t need to. At this point, M/s X credit rating goes down and nobody in the market will give them a loan. At the same time, the assets they had held, their value also depreciates because it became known that they can t act responsibly. So whose to say whether or not M/s X has those number of assets and priced them accurately. But the drama doesn t end there, M/s X says this is the responsibility of actually Mr. Z ( cue The Bank of England) as they are my accountant/lawyer etc. M/s Z says as any lawyer/accountant should. This is not under my remit. If the clients either gives incomplete information or false information or whatever then it is their responsibility not mine. And in fact, the Chancellor is supposed to be the one who is given the responsibility of making the budget. The Chancellor is very similar to our Finance Minister. Because the UK has constitutional monarchy, I am guessing the terms are slightly different, otherwise the functionality seems to be the same. For two weeks, there was lot of chaos, lot of pension funds lost quite a bit in the market and in the end Mr. Kwasi Kwarteng was ousted out of the job. Incredibly, the same media and newspapers who had praised Mr. Kwarteng just few weeks back as the best Tory budget, they couldn t wait to bury him. And while I have attempted to simplify what happened, the best explanation of what has happened can be found in an article from the guardian. Speculation is rife in the UK as to who s ruling atm as the new Chancellor has reversed almost all the policies that Ms. Truss had bought and she is now more or less a figurehead. Mr. Hunt, the new chancellor doesn t have anybody behind him. Apparently, the gentleman wanted to throw his hat the ring in the Tory leadership contest that was held about a month back and he couldn t get 20 MP s to support him. Another thing that is different between UK and India is that in UK by law the PM has to answer questions put up to him or her by the opposition leaders. That is the way accountability is measured there. This is known as PMQ s or Prime Minister Questions and Answers. One can just go to YouTube or any streaming service and give Liz Truss and PMQ s and if they are interested of a certain date, give a date and they can see how she answered the questions thrown at her. Unfortunately, all she could do in both times were non-answers. In fact, the Tories seem to be using some of Labor s policies after they had bad-mouthed the same policies. Politics of right-wing both in the UK and the US seems so out of touch with the people whom they are supposed to protect and administer. An article about cyclists which is sort of half-truth, half irony shows how screwed up the policies are of the RW (right-wing). Now they are questions about the pensions triple-lock. Sadly, it is the working class who would suffer the most, most of the rich have already moved their money abroad several years ago. The Financial Times, did share a video about how things have been unfolding

Seems Ms. Truss forgot to add Financial Times in the list of anti-growth coalition she is so fond of. Also, the Tory party seems to want to create more tax havens in the UK and calling them investment zones. Of course, most of the top tax havens are situated around the UK itself. I wouldn t go more into as that would probably require its own article, although most of that information is all in public domain. Fracking I don t really want to take much time as the blog post has become long. There have been many articles written why Fracking is bad and that is why even the Tories had put in their Manifesto that they won t allow Fracking but apparently, today they are trying to reopen Fracking. And again how bad it is and can be found out by the article in Guardian.

18 October 2022

Dirk Eddelbuettel: RcppSimdJson 0.1.8 on CRAN: Maintenance

The RcppSimdJson package was just updated to release 0.1.8 today. RcppSimdJson wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire and collaborators. Via very clever algorithmic engineering to obtain largely branch-free code, coupled with modern C++ and newer compiler instructions, it results in parsing gigabytes of JSON parsed per second which is quite mindboggling. The best-case performance is faster than CPU speed as use of parallel SIMD instructions and careful branch avoidance can lead to less than one cpu cycle per byte parsed; see the video of the talk by Daniel Lemire at QCon. This release simply changes one statement to not trigger a warning under clang++-14. The very short NEWS entry for this release follows.

Changes in version 0.1.8 (2022-10-18)
  • Use the ' ' operator instead of ' ' on a set of booleans to appease 'clang-14'.

Courtesy of my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dirk Eddelbuettel: digest 0.6.30 on CRAN: More Package Maintenance

Release 0.6.30 of the digest package arrived at CRAN earlier today, and was just uploaded to Debian as well. digest creates hash digests of arbitrary R objects (using the md5, sha-1, sha-256, sha-512, crc32, xxhash32, xxhash64, murmur32, spookyhash, and blake3 algorithms) permitting easy comparison of R language objects. It is a mature and widely-used as many tasks may involve caching of objects for which it provides convenient general-purpose hash key generation to quickly identify the various objects. This release contains two tiny changes on old standard C code to appease the new / upcoming clang-15 release now used by CRAN in their forward-looking checks. My CRANberries provides the usual summary of changes to the previous version. For questions or comments use the issue tracker off the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

17 October 2022

Jeremy Bicha: Ubuntu bug fix anniversary

I first installed Ubuntu when Ubuntu 6.06 LTS Dapper Drake was released. I was brand new to Linux. This was Ubuntu s first LTS release; the very first release of Ubuntu was only a year and a half before. I was impressed by how usable and useful the system was. It soon became my primary home operating system and I wanted to help make it better. On October 15, 2009, I was helping test the release candidates ISOs for the Ubuntu 9.10 release. Specifically, I tested Edubuntu. Edubuntu has since been discontinued but at the time it was an official Ubuntu flavor preloaded with lots of education apps. One of those education apps was Moodle, an e-learning platform. When testing Moodle, I found that a default installation would make Moodle impossible to use locally. I figured out how to fix this issue. This was really exciting: I finally found an Ubuntu bug I knew how to fix. I filed the bug report. This was very late in the Ubuntu 9.10 release process and Ubuntu was in the Final Freeze state. In Final Freeze, every upload to packages included in the default install need to be individually approved by a member of the Ubuntu Release Team. Also, I didn t have upload rights to Ubuntu. Jordan Mantha (LaserJock), an Edubuntu maintainer, sponsored my bug fix upload. I also forwarded my patch to Debian. While trying to figure out what wasn t working with Moodle, I stumbled across a packaging bug. Edubuntu provided a choice of MySQL or PostgreSQL for the system default database. MySQL was the default, but if PostgreSQL were chosen instead, Moodle wouldn t work. I figured out how to fix this bug too a week later. Jordan sponsored this upload and Steve Langasek from the Release Team approved it so it also was able to be fixed before 9.10 was released. Although the first bug was new to 9.10 because of a behavior change in a low-level dependency, this PostgreSQL bug existed in stable Ubuntu releases. Therefore, I prepared Stable Release Updates for Ubuntu 9.04 and Ubuntu 8.04 LTS. Afterwards Six months later, I was able to attend my first Ubuntu Developer Summit. I was living in Bahrain (in the Middle East) at the time and a trip to Belgium seemed easier to me than if I were living in the United States where I usually live. This was the Ubuntu Developer Summit where planning for Ubuntu 10.10 took place. I like to believe that I helped with the naming since I added Maverick to the wiki page where people contribute suggestions. I did not apply for financial sponsorship to attend and I stayed in a budget hotel on the other side of Brussels. The event venue was on the outskirts of Brussels so there wasn t a direct bus or metro line to get there. I rented a car. I didn t yet have a smartphone and I had a LOT of trouble navigating to and from the site every day. I learned then that it s best to stay close to the conference site since a lot of the event is actually in the unstructured time in the evenings. Fortunately, I managed to arrive in time for Mark Shuttleworth s keynote where the Unity desktop was first announced. This was released in Ubuntu 10.10 in the Ubuntu Netbook Remix and became the default for Ubuntu Desktop in Ubuntu 11.04. Ubuntu s switch to Unity provided me with a huge opportunity. In April 2011, GNOME 3.0 was released. I wanted to try it but it wasn t yet packaged in Ubuntu or Debian. It was suggested that I could help work on packaging the major new version in a PPA. The PPA was convenient because I was able to get permission to upload there easier than being able to upload directly to Ubuntu. My contributions there then enabled me to get upload rights to the Ubuntu Desktop packages later that year. At a later Ubuntu Developer Summit, it was suggested that I start an official Ubuntu flavor for GNOME. So along with Tim Lunn (darkxst), I co-founded Ubuntu GNOME. Years later, Canonical stopped actively developing Unity; instead, Ubuntu GNOME was merged into Ubuntu Desktop. Along the way, I became an Ubuntu Core Developer and a Debian Developer. And in January 2022, I joined Canonical on the Desktop Team. This all still feels amazing to me. It took me a long time to be comfortable calling myself a developer! Conclusion My first Ubuntu bugfix was 13 years ago this week. Because Ubuntu historically uses alphabetical adjective animal release names, 13 years means that we have rolled around to the letter K again! Later today, we begin release candidate ISO testing for Ubuntu 22.10 Kinetic Kudu . I encourage you to help us test the release candidates and report bugs that you find. If you figure out how to fix a bug, we still sponsor bug fixes. If you are an Ubuntu contributor, I highly encourage you to attend an Ubuntu Summit if you can. The first Ubuntu Summit in years will be in 3 weeks in Prague, but the intent is for the Ubuntu Summits to be recurring events again.

16 October 2022

Colin Watson: Reproducible man-db databases

I ve released man-db 2.11.0 (announcement, NEWS), and uploaded it to Debian unstable. The biggest chunk of work here was fixing some extremely long-standing issues with how the database is built. Despite being in the package name, man-db s database is much less important than it used to be: most uses of man(1) haven t required it in a long time, and both hardware and software improvements mean that even some searches can be done by brute force without needing prior indexing. However, the database is still needed for the whatis(1) and apropos(1) commands. The database has a simple format - no relational structure here, it s just a simple key-value database using old-fashioned DBM-like interfaces and composing a few fields to form values - but there are a number of subtleties involved. The issues tend to amount to this: what does a manual page name mean? At first glance it might seem simple, because you have file names that look something like /usr/share/man/man1/ls.1.gz and that s obviously ls(1). Some pages are symlinks to other pages (which we track separately because it makes it easier to figure out which entries to update when the contents of the file system change), and sometimes multiple pages are even hard links to the same file. The real complications come with whatis references . Pages can list a bunch of names in their NAME section, and the historical expectation is that it should be possible to use those names as arguments to man(1) even if they don t also appear in the file system (although Debian policy has deprecated relying on this for some time). Not only does that mean that man(1) sometimes needs to consult the database, but it also means that the database is inherently more complicated, since a page might list something in its NAME section that conflicts with an actual file name in the file system, and now you need a priority system to resolve ambiguities. There are some other possible causes of ambiguity as well. The people working on reproducible builds in Debian branched out to the related challenge of reproducible installations some time ago: can you take a collection of packages, bootstrap a file system image from them, and reproduce that exact same image somewhere else? This is useful for the same sorts of reasons that reproducible builds are useful: it lets you verify that an image is built from the components it s supposed to be built from, and doesn t contain any other skulduggery by accident or design. One of the people working on this noticed that man-db s database files were an obstacle to that: in particular, the exact contents of the database seemed to depend on the order in which files were scanned when building it. The reporter proposed solving this by processing files in sorted order, but I wasn t keen on that approach: firstly because it would mean we could no longer process files in an order that makes it more efficient to read them all from disk (still valuable on rotational disks), but mostly because the differences seemed to point to other bugs. Having understood this, there then followed several late nights of very fiddly work on the details of how the database is maintained. None of this was conceptually difficult: it mainly amounted to ensuring that we maintain a consistent well-order for different entries that we might want to insert for a given database key, and that we consider the same names for insertion regardless of the order in which we encounter files. As usual, the tricky bit is making sure that we have the right data structures to support this. man-db is written in C which is not very well-supplied with built-in data structures, and originally much of the code was written in a style that tried to minimize memory allocations; this came at the cost of ownership and lifetime often being rather unclear, and it was often difficult to make changes without causing leaks or double-frees. Over the years I ve been gradually introducing better encapsulation to make things easier to follow, and I had to do another round of that here. There were also some problems with caching being done at slightly the wrong layer: we need to make use of a trace of the chain of links followed to resolve a page to its ultimate source file, but we were incorrectly caching that trace and reusing it for any link to the same file, with incorrect results in many cases. Oh, and after doing all that I found that the on-disk representation of a GDBM database is insertion-order-dependent, so I ended up having to manually reorganize the database at the end by reading it all in and writing it all back out in sorted order, which feels really weird to me coming from spending most of my time with PostgreSQL these days. Fortunately the database is small so this takes negligible time. None of this is particularly glamorous work, but it paid off:
# export SOURCE_DATE_EPOCH="$(date +%s)"
# mkdir emptydir disorder
# disorderfs --multi-user=yes --shuffle-dirents=yes --reverse-dirents=no emptydir disorder
# export TMPDIR="$(pwd)/disorder"
# mmdebstrap --variant=standard --hook-dir=/usr/share/mmdebstrap/hooks/merged-usr \
      unstable out1.tar
# mmdebstrap --variant=standard --hook-dir=/usr/share/mmdebstrap/hooks/merged-usr \
      unstable out2.tar
# cmp out1.tar out2.tar
# echo $?
0

Vincent Fourmond: Tutorial: analysis of multiwavelength fast kinetics data

The purpose of this post is to demonstrate a first approach to the analysis of multiwavelength kinetic data, like those obtained using stopped-flow data. To practice, we will use data that were acquired during the stopped flow practicals of the MetBio summer school from the FrenchBIC. During the practicals, the student monitored the reaction of myoglobin (in its Fe(III) state) with azide, which yields a fast and strong change in the absorbance spectrum of the protein, which was monitored using a diode array. The data is publicly available on zenodo. Aims of this tutorial The purpose of this tutorial is to teach you to use the free softwareQSoas to run a simple, multiwavelength exponential fit on the data, and to look at the results. This is not a kinetics lecture, so that it will not go in depth about the use of the exponential fit and its meaning. Getting started: loading the file First, make sure you have a working version of QSoas, you can download them (for free) there. Then download the data files from zenodo. We will work only on the data file Azide-1.25mm_001.dat, but of course, the purpose of this tutorial is to enable you to work on all of them. The data files contain the time evolution of the absorbance for all wavelengths, in a matrix format, in which each row correpond to a time point and each column to a wavelength. Start QSoas, and launch the command:
QSoas> load /comments='"'
Then, choose the Azide-1.25mm_001.dat data file. This should bring up a horizontal red line at the bottom of the data display, with X values between about 0 and 2.5. If you zoom on the red line with the mouse wheel, you'll realize it is data. The /comments='"' part is very important since it allows the extraction of the wavelength from the data. We will look at what it means another day. At this stage, you can look at the loaded data using the command:
QSoas> edit
You should have a window looking like this:
The rows each correspond to a data point displayed on the window below. The first column correspond to the X values, the second the Y values, and all the other ones to extra Y columns (they are not displayed by default). What is especially interesting is the first row, which contains a nan as the X value and what is obviously the wavelength for all the Y values. To tell that QSoas should take this line as the wavelength (which will be the perpendicular coordinate, the coordinate of the other direction of the matrix), first close the edit window and run:
QSoas> set-perp /from-row=0
Splitting and fitting Now, we have a single dataset containing a lot of Y columns. We want to fit all of them simultaneously with a (mono) exponential fit. For that, we first need to split the big matrix into a series of X,Y datasets (because fitting only works on the first Y). This is possible by running:
QSoas> expand /style=red-to-blue /flags=kinetics
Your screen should now look like this:
You're looking at the kinetics at all wavelengths at the same time (this may take some time to display on your computer, it is after all a rather large number of data points). The /style=red-to-blue is not strictly necessary, but it gives the red to blue color gradient which makes things easier to look at (and cooler !). The /flags=kinetics is there to attach a label (a flag) to the newly created datasets so we can easily manipulate all of them at the same time. Then it's time to fit, with the following command:
QSoas> mfit-exponential-decay flagged:kinetics
This should bring up a new window. After resizing it, you should have something that looks like this:
The bottom of the fit window is taken by the parameters, each with two checkboxes on the right to set them fixed (i.e. not determined by the fitting mechanism) and/or global (i.e. with a single value for all the datasets, here all the wavelengths). The top shows the current dataset along with the corresponding fit (in green), and, below, the residuals. You can change the dataset by clicking on the horizontal arrows or using Ctrl+PgUp or Ctrl+PgDown (keep holding it to scan fast). See the Z = 728.15 showing that QSoas has recognized that the currently displayed dataset corresponds to the wavelength 728.15. The equation fitted to the data is: $$y(x) = A_\infty + A_1 \times \exp -(x - x_0)/\tau_1$$ In this case, while the \(A_1\) and \(A_\infty\) parameters clearly depend on the wavelength, the time constant of evolution should be independent of wavelength (the process happens at a certain rate regardless of the wavelength we're analyzing), so that the \(\tau_1\) parameter should be common for all the datasets/wavelengths. Just click on the global checkbox at the right of the tau_1 parameter, make sure it is checked, and hit the Fit button... The fit should not take long (less than a minute), and then you end up with the results of the fits: all the parameters. The best way to look at the non global parameters like \(A_1\) and \(A_\infty\) is to use the Show Parameters item from the Parameters menu. Using it and clicking on A_inf too should give you a display like this one:
The A_inf parameter corresponds to the spectum at infinite time (of azide-bound heme), while the A_1 parameter corresponds to the difference spectrum between the initial (azide-free) and final (azide-bound) states. Now, the fit is finished, you can save the parameters if you want to reload them in a later fit by using the Parameters/Save menu item or export them in a form more suitable for plotting using Parameters/Export (although QSoas can also display and the parameters saved using Save). This concludes this first approach to fitting the data. What you can do is How to read the code above All the lines starting by QSoas> in the code areas above are meant to be typed into the QSoas command line (at the bottom of the window), and started by pressing enter at the end. You must remove the QSoas> bit. The other lines (when applicable) show you the response of QSoas, in the terminal just above the command-line. You may want to play with the QSoas tutorial to learn more about how to interact with QSoas. About QSoas QSoas is a powerful open source data analysis program that focuses on flexibility and powerful fitting capacities. It is released under the GNU General Public License. It is described in Fourmond, Anal. Chem., 2016, 88 (10), pp 5050 5052. Current version is 3.1. You can freely (and at no cost) download its source code or precompiled versions for MacOS and Windows there. Alternatively, you can clone from the GitHub repository.
Contact: find my email address there, or contact me on LinkedIn.

15 October 2022

Jonathan Dowland: podman generate

I've been working with and on container technology for seven years, but I still learn new things every day. Recently I read the excellent LWN article Docker and the OCI container ecosystem and this was news to me:
Running the docker CLI under a process supervisor only results in supervising the CLI process. This has a variety of consequences for users of these tools. For example, any attempt to limit a container's memory usage by running the CLI as a systemd service will fail; the limits will only apply to the CLI and its non-existent children. In addition, attempts to terminate a client process may not result in terminating all of the processes in the container.
Huh of course! I hadn't really thought about that. I run a small number of containers on my home system via docker (since I was using it at work) and systemd, and sometimes I have weird consistency issues. This explains them. Later:
As a result, Podman plays nicely with tools like systemd; using podman run with a process supervisor works as expected, because the processes inside the container are children of podman run. The developers of Podman encourage people to use it in this way by a command to generate systemd units for Podman containers.
Given the above, it seemed like a good idea to migrate my few local containers over to Podman. This was easy. The first part is copying the images from Docker's storage to Podman's. To do this, I used the skopeo tool:
sudo skopeo copy  docker-daemon,containers-storage :octoprint/octoprint:latest
(I want to launch these as a superuser, rather than use root-less containers, to isolate the containers from each other: rootless ones are forced to share a common namespace.) Once the images were copied over, I needed to start up a container instance via Podman. For most of them, running under Docker, I had volume-mounted the important bits for persistent data. I can do the same with Podman:
# podman run -d --rm \
    --name octoprint \
    -p 8080:80 \
    -v /archive/octoprint:/octoprint \
    --device /dev/ttyACM0:/dev/ttyACM0 \
    octoprint/octoprint
Once an instance was running, I can use podman generate to create a Systemd unit file which describes creating an equivalent container:
# cd /etc/systemd/system
# podman generate systemd octoprint \
    --new \
    --name \
    --files
For some of the containers I was running, there are a few more steps required: migrating docker volumes and configuring equivalents for private docker networks. But Podman's versions of those concepts are largely identical. Running a mixture of Podman and Docker containers side-by-side also meant renumbering some private addresses for migrated hosts whilst the original network was still up.

Next.

Previous.