Search Results: "nate"

6 April 2022

Thorsten Alteholz: My Debian Activities in March 2022

FTP master This month I accepted 332 and rejected 15 packages. This ratio gives a reason to hope. The overall number of packages that got accepted was 342. Debian LTS This was my ninety-third month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. This month my all in all workload has been 40h. During that time I did LTS and normal security uploads of: All my PU bugs for Buster and Bullseye, that accumulated over the last months, were part of the latest point release. So new ones have to be created now :-). I also continued to work on security support for golang packages. As a result #1008577 and #1008578 were the first real tests with a simple package. Debian ELTS This month was the forty-fifth ELTS month. During my allocated time I uploaded: Unfortunately uploads have to be done for younger releases first, so I had to withhold some uploads for ELTS. Hopefully they can be done in April. Probably this policy needs to be reconsidered. Last but not least I did some days of frontdesk duties. Debian Printing This month I uploaded new upstream versions or improved packaging of: In order to make the Debian Edu team happy, I uploaded a new version of cups-filters with an adapted Apparmor-file to Unstable and Bullseye. Debian Astro This month I uploaded new upstream versions or improved packaging of: Other stuff This month I uploaded new upstream versions or improved packaging of: In order to avoid an AUTORM of some Osmocom packages, I also had to NMU:

5 April 2022

Dirk Eddelbuettel: RcppSpdlog 0.0.8 on CRAN: Upstream Update

A new version 0.0.8 of RcppSpdlog is now on CRAN. RcppSpdlog bundles spdlog, a wonderful header-only C++ logging library with all the bells and whistles you would want that was written by Gabi Melman, and also includes fmt by Victor Zverovich. This release brings a new upstream release 1.10.0 of spdlog. The (minimal) NEWS entry for this release follows.

Changes in RcppSpdlog version 0.0.8 (2022-04-04)
  • Upgraded to upstream releases spdlog 1.10.0

Courtesy of my CRANberries, there is also a diffstat report. More detailed information is on the RcppSpdlog page, or the package documention site. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dirk Eddelbuettel: RcppArmadillo 0.11.0.0.0 on CRAN: Upstream Updates

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 972 other packages on CRAN, downloaded over 24 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 465 times according to Google Scholar. This release brings a new upstream release 11.0.0. We tested this very rigorously via three different RC release each of which got a full reverse-dependencies run (for which results are always logged here). The full set of changes (since the last CRAN release 0.10.8.1.0) follows.

Changes in RcppArmadillo version 0.11.0.0 (2022-04-04)
  • Upgraded to Armadillo release 11.0.0 (Creme Brulee)
    • added variants of inv() and inv_sympd() that provide rcond (reciprocal condition number)
    • expanded inv() and inv_sympd() with options inv_opts::tiny and inv_opts::allow_approx
    • stricter handling of singular matrices by inv() and inv_sympd()
    • stricter handling of non-sympd matrices by inv() and inv_sympd()
    • stricter handling of non-finitie matrices by pinv()
    • more robust handling of rank deficient matrices by solve()
    • faster handling of diagonal matrices by rcond()
    • changed eigs_sym() and eigs_gen() to use higher quality RNG
    • quantile() and median() will now throw an exception if given matrices/vectors have NaN elements
    • workaround for yet another bug in Intel MKL
  • Until May 2022, protect correction to Field behavior via define of RCPP_ARMADILLO_FIX_Field
  • If a LAPACK installation with missing complex routines is found (as e.g. Ubuntu using 3.9.0) then the LAPACK unit test is skipped.

Changes in RcppArmadillo version 0.10.8.2.0 (2022-02-01)
  • Upgraded to Armadillo release 10.8.2 (Realm Raider)
    • fix potential speed regression in pinv() and rank()

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Kees Cook: security things in Linux v5.10

Previously: v5.9 Linux v5.10 was released in December, 2020. Here s my summary of various security things that I found interesting: AMD SEV-ES
While guest VM memory encryption with AMD SEV has been supported for a while, Joerg Roedel, Thomas Lendacky, and others added register state encryption (SEV-ES). This means it s even harder for a VM host to reconstruct a guest VM s state. x86 static calls
Josh Poimboeuf and Peter Zijlstra implemented static calls for x86, which operates very similarly to the static branch infrastructure in the kernel. With static branches, an if/else choice can be hard-coded, instead of being run-time evaluated every time. Such branches can be updated too (the kernel just rewrites the code to switch around the branch ). All these principles apply to static calls as well, but they re for replacing indirect function calls (i.e. a call through a function pointer) with a direct call (i.e. a hard-coded call address). This eliminates the need for Spectre mitigations (e.g. RETPOLINE) for these indirect calls, and avoids a memory lookup for the pointer. For hot-path code (like the scheduler), this has a measurable performance impact. It also serves as a kind of Control Flow Integrity implementation: an indirect call got removed, and the potential destinations have been explicitly identified at compile-time. network RNG improvements
In an effort to improve the pseudo-random number generator used by the network subsystem (for things like port numbers and packet sequence numbers), Linux s home-grown pRNG has been replaced by the SipHash round function, and perturbed by (hopefully) hard-to-predict internal kernel states. This should make it very hard to brute force the internal state of the pRNG and make predictions about future random numbers just from examining network traffic. Similarly, ICMP s global rate limiter was adjusted to avoid leaking details of network state, as a start to fixing recent DNS Cache Poisoning attacks. SafeSetID handles GID
Thomas Cedeno improved the SafeSetID LSM to handle group IDs (which required teaching the kernel about which syscalls were actually performing setgid.) Like the earlier setuid policy, this lets the system owner define an explicit list of allowed group ID transitions under CAP_SETGID (instead of to just any group), providing a way to keep the power of granting this capability much more limited. (This isn t complete yet, though, since handling setgroups() is still needed.) improve kernel s internal checking of file contents
The kernel provides LSMs (like the Integrity subsystem) with details about files as they re loaded. (For example, loading modules, new kernel images for kexec, and firmware.) There wasn t very good coverage for cases where the contents were coming from things that weren t files. To deal with this, new hooks were added that allow the LSMs to introspect the contents directly, and to do partial reads. This will give the LSMs much finer grain visibility into these kinds of operations. set_fs removal continues
With the earlier work landed to free the core kernel code from set_fs(), Christoph Hellwig made it possible for set_fs() to be optional for an architecture. Subsequently, he then removed set_fs() entirely for x86, riscv, and powerpc. These architectures will now be free from the entire class of kernel address limit attacks that only needed to corrupt a single value in struct thead_info. sysfs_emit() replaces sprintf() in /sys
Joe Perches tackled one of the most common bug classes with sprintf() and snprintf() in /sys handlers by creating a new helper, sysfs_emit(). This will handle the cases where kernel code was not correctly dealing with the length results from sprintf() calls, which might lead to buffer overflows in the PAGE_SIZE buffer that /sys handlers operate on. With the helper in place, it was possible to start the refactoring of the many sprintf() callers. nosymfollow mount option
Mattias Nissler and Ross Zwisler implemented the nosymfollow mount option. This entirely disables symlink resolution for the given filesystem, similar to other mount options where noexec disallows execve(), nosuid disallows setid bits, and nodev disallows device files. Quoting the patch, it is useful as a defensive measure for systems that need to deal with untrusted file systems in privileged contexts. (i.e. for when /proc/sys/fs/protected_symlinks isn t a big enough hammer.) Chrome OS uses this option for its stateful filesystem, as symlink traversal as been a common attack-persistence vector. ARMv8.5 Memory Tagging Extension support
Vincenzo Frascino added support to arm64 for the coming Memory Tagging Extension, which will be available for ARMv8.5 and later chips. It provides 4 bits of tags (covering multiples of 16 byte spans of the address space). This is enough to deterministically eliminate all linear heap buffer overflow flaws (1 tag for free , and then rotate even values and odd values for neighboring allocations), which is probably one of the most common bugs being currently exploited. It also makes use-after-free and over/under indexing much more difficult for attackers (but still possible if the target s tag bits can be exposed). Maybe some day we can switch to 128 bit virtual memory addresses and have fully versioned allocations. But for now, 16 tag values is better than none, though we do still need to wait for anyone to actually be shipping ARMv8.5 hardware. fixes for flaws found by UBSAN
The work to make UBSAN generally usable under syzkaller continues to bear fruit, with various fixes all over the kernel for stuff like shift-out-of-bounds, divide-by-zero, and integer overflow. Seeing these kinds of patches land reinforces the the rationale of shifting the burden of these kinds of checks to the toolchain: these run-time bugs continue to pop up. flexible array conversions
The work on flexible array conversions continues. Gustavo A. R. Silva and others continued to grind on the conversions, getting the kernel ever closer to being able to enable the -Warray-bounds compiler flag and clear the path for saner bounds checking of array indexes and memcpy() usage. That s it for now! Please let me know if you think anything else needs some attention. Next up is Linux v5.11.

2022, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

1 April 2022

Dirk Eddelbuettel: RInside 0.2.17

A new release 0.2.17 of RInside arrived on CRAN and in Debian today. This is the first release in two years since the 0.2.16 release. RInside provides a set of convenience classes which facilitate embedding of R inside of C++ applications and programs, using the classes and functions provided by Rcpp. This release brings a minor update for R 4.2.0, due out later this month, and kindly provided by Tomas Kalibera. It updates one function declaration; we wrapped some conditioning around so that RInside continues to build with older releases too. Besides that minor updates were made to continuous integration and some of the package files. The short list of changes since the last release were:

Changes in RInside version 0.2.17 (2022-03-31)
  • A Windows-only patch for R 4.2.0 kindly provided by Tomas Kalibera was applied, and also conditioned on R (>= 4.2.0)
  • Continuous Integration setup was updated and now uses r-ci.
  • Several updates were made to README.md (badges etc) and DESCRIPTION

My CRANberries also provides a short report with changes from the previous release. More information is on the RInside page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page, or to issues tickets at the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Russ Allbery: Review: Princess Floralinda and the Forty-Flight Tower

Review: Princess Floralinda and the Forty-Flight Tower, by Tamsyn Muir
Publisher: Subterranean Press
Copyright: 2020
ISBN: 1-59606-992-9
Format: Kindle
Pages: 111
A witch put Princess Floralinda at the top of a forty-flight tower, but it wasn't personal. This is just what witches do, particularly with princesses with butter-coloured curls and sapphire-blue eyes. Princes would come from miles around to battle up the floors of the tower and rescue the princess. The witch even helpfully provided a golden sword, in case a prince didn't care that much about princesses. Floralinda was provided with water and milk, two loaves of bread, and an orange, all of them magically renewing, to sustain her while she waited. In retrospect, the dragon with diamond-encrusted scales on the first floor may have been a mistake. None of the princely endeavors ever saw the second floor. The diary that Floralinda found in her room indicated that she may not be the first princess to have failed to be rescued from this tower. Floralinda finally reaches the rather astonishing conclusion that she might have to venture down the tower herself, despite the goblins she was warned were on the 39th floor (not to mention all the other monsters). The result of that short adventure, after some fast thinking, a great deal of luck, and an unforeseen assist from her magical food, is a surprising number of dead goblins. Also seriously infected hand wounds, because it wouldn't be a Tamsyn Muir story without wasting illness and body horror. That probably would have been the end of Floralinda, except a storm blew a bottom-of-the-garden fairy in through the window, sufficiently injured that she and Floralinda were stuck with each other, at least temporarily. Cobweb, the fairy, is neither kind nor inclined to help Floralinda (particularly given that Floralinda is not a child whose mother is currently in hospital), but it is an amateur chemist and finds both Floralinda's tears and magical food intriguing. Cobweb's magic is also based on wishes, and after a few failed attempts, Floralinda manages to make a wish that takes hold. Whether she'll regret the results is another question. This is a fairly short novella by the same author as Gideon the Ninth, but it's in a different universe and quite different in tone. This summary doesn't capture the writing style, which is a hard-to-describe mix of fairy tale, children's story, and slightly archaic and long-winded sentence construction. This is probably easier to show with a quote:
"You are displaying a very small-minded attitude," said the fairy, who seemed genuinely grieved by this. "Consider the orange-peel, which by itself has many very nice properties. Now, if you had a more educated brain (I cannot consider myself educated; I have only attempted to better my situation) you would have immediately said, 'Why, if I had some liquor, or even very hot water, I could extract some oil from this orange-peel, which as everyone knows is antibacterial; that may well do my hands some good,' and you wouldn't be in such a stupid predicament."
On balance, I think this style worked. It occasionally annoyed me, but it has some charm. About halfway through, I was finding the story lightly entertaining, although I would have preferred a bit less grime, illness, and physical injury. Unfortunately, the rest of the story didn't work for me. The dynamic between Floralinda and Cobweb turns into a sort of D&D progression through monster fights, and while there are some creative twists to those fights, they become all of a sameness. And while I won't spoil the ending, it didn't work for me. I think I see what Muir was trying to do, and I have some intellectual appreciation for the idea, but it wasn't emotionally satisfying. I think my root problem with this story is that Muir sets up a rather interesting world, one in which witches artistically imprison princesses, and particularly bright princesses (with the help of amateur chemist fairies) can use the trappings of a magical tower in ways the witch never intended. I liked that; it has a lot of potential. But I didn't feel like that potential went anywhere satisfying. There is some relationship and characterization work, and it reached some resolution, but it didn't go as far as I wanted. And, most significantly, I found the end point the characters reached in relation to the world to be deeply unsatisfying and vaguely irritating. I wanted to like this more than I did. I think there's a story idea in here that I would have enjoyed more. Unfortunately, it's not the one that Muir wrote, and since so much of my problem is with the ending, I can't provide much guidance on whether someone else would like this story better (and why). But if the idea of taking apart a fairy-tale tower and repurposing the pieces sounds appealing, and if you get along better with Muir's illness motif than I do, you may enjoy this more than I did. Rating: 5 out of 10

Antoine Beaupr : Salvaged my first Debian package

I finally salvaged my first Debian package, python-invoke. As part of ITS 964718, I moved the package from the Openstack Team to the Python team. The Python team might not be super happy with it, because it's breaking some of its rules, but at least someone (ie. me) is actively working (and using) the package.

Wait what People not familiar with Debian will not understand anything in that first paragraph, so let me expand. Know-it-all Debian developers (you know who you are) can skip to the next section. Traditionally, the Debian project (my Linux-based operating system of choice) has prided itself on the self-managed, anarchistic organisation of its packaging. Each package maintainer is the lord of their little kingdom. Some maintainers like to accumulate lots of kingdoms to rule over. (Yes, it really doesn't sound like anarchism when you put it like that. Yes, it's complicated: there's a constitution and voting involved. And yes, we're old.) Therefore, it's really hard to make package maintainers do something they don't want. Typically, when things go south, someone makes a complaint to the Debian Technical Committee (CTTE) which is established by the Debian constitution to resolve such conflicts. The committee is appointed by the Debian Project leader, elected each year (and there's an election coming up if you haven't heard). Typically, the CTTE will then vote and formulate a decision. But here's the trick: maintainers are still free to do whatever they want after that, in a sense. It's not like the CTTE can just break down doors and force maintainers to type code. (I won't go into the details of the why of that, but it involves legal issues and, I think, something about the Turing halting problem. Or something like that.) Anyways. The point is all that is super heavy and no one wants to go there... (Know-it-all Debian developers, I know you are still reading this anyways and disagree with that statement, but please, please, make it true.) ... but sometimes, packages just get lost. Maintainers get distracted, or busy with something else. It's not that they want to abandon their packages. They love their little fiefdoms. It's just there was a famine or a war or something and everyone died, and they have better things to do than put up fences or whatever. So clever people in Debian found a better way of handling such problems than waging war in the poor old CTTE's backyard. It's called the Package Salvaging process. Through that process, a maintainer can propose to take over an existing package from another maintainer, if a certain set of condition are met and a specific process is followed. Normally, taking over another maintainer's package is basically a war declaration, rarely seen in the history of Debian (yes, I do think it happened!), as rowdy as ours is. But through this process, it seems we have found a fair way of going forward. The process is basically like this:
  1. file a bug proposing the change
  2. wait three weeks
  3. upload a package making the change, with another week delay
  4. you now have one more package to worry about
Easy right? It actually is! Process! It's magic! It will cure your babies and resurrect your cat!

So how did that go? It went well! The old maintainer was actually fine with the change because his team wasn't using the package anymore anyways. He asked to be kept as an uploader, which I was glad to oblige. (He replied a few months after the deadline, but I wasn't in a rush anyways, so that doesn't matter. It was polite for him to answer, even if, technically, I was already allowed to take it over.) What happened next is less shiny for me though. I totally forgot about the ITS, even after the maintainer reminded me of its existence. See, the thing is the ITS doesn't show up on my dashboard at all. So I totally forgot about it (yes, twice). In fact, the only reason I remembered it was that got into the process of formulating another ITS (1008753, trocla) and I was trying to figure out how to write the email. Then I remembered: "hey wait, I think I did this before!" followed by "oops, yes, I totally did this before and forgot for 9 months". So, not great. Also, the package is still not in a perfect shape. I was able to upload the upstream version that was pending 1.5.0 to clear out the ITS, basically. And then there's already two new upstream releases to upload, so I pushed 1.7.0 to experimental as well, for good measure. Unfortunately, I still can't enable tests because everything is on fire, as usual. But at least my kingdom is growing.

Appendix Just in case someone didn't notice the hyperbole, I'm not a monarchist promoting feudalism as a practice to manage a community. I do not intend to really "grow my kingdom" and I think the culture around "property" of "packages" is kind of absurd in Debian. I kind of wish it would go away. (Update: It has also been pointed out that I might have made Debian seem more confrontational than it actually is. And it's kind of true: most work and interactions in Debian actually go fine, it's only a minority of issues that degenerate into conflicts. It's just that they tend to take up a lot of space in the community, and I find that particularly draining. And I think think our "package ownership" culture is part of at least some of those problems.) Team maintenance, the LowNMU process, and low threshold adoption processes are all steps in the good direction, but they are all opt in. At least the package salvaging process is someone a little more ... uh... coercive? Or at least it allows the community to step in and do the right thing, in a sense. We'll see what happens with the coming wars around the recent tech committee decision, which are bound to touch on that topic. (Hint: our next drama is called "usrmerge".) Hopefully, LWN will make a brilliant article to sum it up for us so that I don't have to go through the inevitable debian-devel flamewar to figure it out. I already wrecked havoc on the #debian-devel IRC channel asking newbie questions so I won't stir that mud any further for now. (Update: LWN, of course, did make an article about usrmerge in Debian. I will read it soon and can then tell you know if it's brilliant, but they are typically spot on.)

31 March 2022

Matthew Garrett: ZTA doesn't solve all problems, but partial implementations solve fewer

Traditional network access controls work by assuming that something is trustworthy based on some other factor - for example, if a computer is on your office network, it's trustworthy because only trustworthy people should be able to gain physical access to plug something in. If you restrict access to your services to requests coming from trusted networks, then you can assert that it's coming from a trusted device.

Of course, this isn't necessarily true. A machine on your office network may be compromised. An attacker may obtain valid VPN credentials. Someone could leave a hostile device plugged in under a desk in a meeting room. Trust is being placed in devices that may not be trustworthy.

A Zero Trust Architecture (ZTA) is one where a device is granted no inherent trust. Instead, each access to a service is validated against some policy - if the policy is satisfied, the access is permitted. A typical implementation involves granting each device some sort of cryptographic identity (typically a TLS client certificate) and placing the protected services behind a proxy. The proxy verifies the device identity, queries another service to obtain the current device state (we'll come back to that in a moment), compares the state against a policy and either pass the request through to the service or reject it. Different services can have different policies (eg, you probably want a lax policy around whatever's hosting the documentation for how to fix your system if it's being refused access to something for being in the wrong state), and if you want you can also tie it to proof of user identity in some way.

From a user perspective, this is entirely transparent. The proxy is made available on the public internet, DNS for the services points to the proxy, and every time your users try to access the service they hit the proxy instead and (if everything's ok) gain access to it no matter which network they're on. There's no need to connect to a VPN first, and there's no worries about accidentally leaking information over the public internet instead of over a secure link.

It's also notable that traditional solutions tend to be all-or-nothing. If I have some services that are more sensitive than others, the only way I can really enforce this is by having multiple different VPNs and only granting access to sensitive services from specific VPNs. This obviously risks combinatorial explosion once I have more than a couple of policies, and it's a terrible user experience.

Overall, ZTA approaches provide more security and an improved user experience. So why are we still using VPNs? Primarily because this is all extremely difficult. Let's take a look at an extremely recent scenario. A device used by customer support technicians was compromised. The vendor in question has a solution that can tie authentication decisions to whether or not a device has a cryptographic identity. If this was in use, and if the cryptographic identity was tied to the device hardware (eg, by being generated in a TPM), the attacker would not simply be able to obtain the user credentials and log in from their own device. This is good - if the attacker wanted to maintain access to the service, they needed to stay on the device in question. This increases the probability of the monitoring tooling on the compromised device noticing them.

Unfortunately, the attacker simply disabled the monitoring tooling on the compromised device. If device state was being verified on each access then this would be noticed before too long - the last data received from the device would be flagged as too old, and the requests would no longer satisfy any reasonable access control policy. Instead, the device was assumed to be trustworthy simply because it could demonstrate its identity. There's an important point here: just because a device belongs to you doesn't mean it's a trustworthy device.

So, if ZTA approaches are so powerful and user-friendly, why aren't we all using one? There's a few problems, but the single biggest is that there's no standardised way to verify device state in any meaningful way. Remote Attestation can both prove device identity and the device boot state, but the only product on the market that does much with this is Microsoft's Device Health Attestation. DHA doesn't solve the broader problem of also reporting runtime state - it may be able to verify that endpoint monitoring was launched, but it doesn't make assertions about whether it's still running. Right now, people are left trying to scrape this information from whatever tooling they're running. The absence of any standardised approach to this problem means anyone who wants to deploy a strong ZTA has to integrate with whatever tooling they're already running, and that then increases the cost of migrating to any other tooling later.

But even device identity is hard! Knowing whether a machine should be given a certificate or not depends on knowing whether or not you own it, and inventory control is a surprisingly difficult problem in a lot of environments. It's not even just a matter of whether a machine should be given a certificate in the first place - if a machine is reported as lost or stolen, its trust should be revoked. Your inventory system needs to tie into your device state store in order to ensure that your proxies drop access.

And, worse, all of this depends on you being able to put stuff behind a proxy in the first place! If you're using third-party hosted services, that's a problem. In the absence of a proxy, trust decisions are probably made at login time. It's possible to tie user auth decisions to device identity and state (eg, a self-hosted SAML endpoint could do that before passing through to the actual ID provider), but that's still going to end up providing a bearer token of some sort that can potentially be exfiltrated, and will continue to be trusted even if the device state becomes invalid.

ZTA doesn't solve all problems, and there isn't a clear path to it doing so without significantly greater industry support. But a complete ZTA solution is significantly more powerful than a partial one. Verifying device identity is a step on the path to ZTA, but in the absence of device state verification it's only a step.

comment count unavailable comments

Russell Coker: Links March 2022

Anarcat wrote a great blog post about switching from OpenNTP to Chrony which gives a good overview of how NTP works and how accurate the different versions are [1]. Bleeping Computer has an amusing article about criminals who copied a lot of data from NVidia servers including specs of their latest products [2], they are threatening to release all the data if NVidia doesn t stop crippling their GPUs to make them unsuitable for crypto currency mining. I don t support these criminals, but I think NVidia should allow people who buy hardware to use their property as they choose. If cryptocurrency miners buy all the NVidia products then NVidia still makes the sales, they could even auction them to make more money. NPR has a disturbing article about the way execution by lethal injection works in the US [3]. It seems that most people die in an extremely unpleasant way. It makes the North Korean execution by anti-aircraft gun seem civilised. The DirtyPipe vulnerability is the latest serious security issue in the Linux kernel [4]. The report of how it was discovered is very interesting and should be read by all sysadmins. SE Linux will not save you from this as the vulnerability allows writing to read-only files like /etc/passwd. Politico has an insightful analysis of Putin, it s not good news he wants to conquer all territory that had ever been part of a Russian empire at any time in history [5]. The Guardian has an informative article about the EU s attempts to debunk Russian propaganda about Covid19 [6]. Fortunately the sanctions are reducing Russia s ability to do such things now. The Guardian has in interesting article about a project to use literary analysis to predict wars [7]. Funded by the German military but funding was cut after it was proven to work. The Fact Act is a proposal by David Brin for political changes in the US to involve scientists and statisticians in an official advisory role in the legislative process [8], it s an idea with a lot of potential. Technology Review has an interesting interview with the leader of the NSA s Research Directorate [9]. In 2008 the EFF posted a long and informative article about the RIAA s war against music fans [10]. I had followed a lot of the news about this when it was happening, but I still learnt some things from this article that I hadn t known at the time. Also considering past legal battles in the context of the current situation is useful. As an aside all the music I want to listen to is now on YouTube and youtube-dl works really well for me. The 1952 edition of Psychiatry: Journal of Interpersonal Relations has an interesting article On Cooling the Mark Out [11] which starts about how criminal gangs engaged in fraud try to make their victims come to terms with the loss in a way that doesn t involve the police. But it goes on to cover ways of dealing with loss of status in general. The layout is hacky with words broken by hyphens in the middle of lines as it appears to have been scanned from paper, converted to MS-Word, and from there to PDF. But it s worth it. The Internet Heist by Cory Doctorow is an insightful series of 3 articles about the MPAA (MAFIAA) attempts to take over all TV distribution in the US [12]. Wired has an interesting exerpt from the book Spies, Lies, and Algorithms: The History and Future of American Intelligence , by Amy B. Zegart [13]. Interesting summary of the open source intelligence systems (which have nothing to do with open source as free software). But it would be interesting to have an open source intelligence organisation along similar lines to open source software. The guy who tracks billionaire s private jets is an example of this.

29 March 2022

Dirk Eddelbuettel: RcppBDT 0.2.5: Maintenance

A minor maintenance release for the RcppBDT package is now on CRAN. The RcppBDT package is an early adopter of Rcpp and was one of the first packages utilizing Boost and its Date_Time library. The now more widely-used package anytime is a direct descentant of RcppBDT. Thanks again for the heads-up! This release mostly deals with a one-definition rule violation detected by link-time optimisation (which can be enable when configuring R itself at build time with --enable-lto). I confused myself into thinking Rcpp Modules may be at fault, but I aki was a little more awake than myself and noticed that I only needed to carry the (common) header RcppBDT.h to the file toPOSIXct.cpp added last summer. The NEWS entry follows:

Changes in version 0.2.5 (2022-03-29)
  • Ensure consistent compilation by ensuring RcppBDT.h is included in all files, this addresses an LTO/ODR issue
  • Correct one declaration in init.c
  • Minor additional cleanups

Courtesy of my CRANberries, there is also a diffstat report for this release. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Raphaël Hertzog: Join Freexian to help improve Debian

Freexian has set itself new ambitious goals in support of Debian and we would like to expand our team to help us reach those goals. We have drafted a mission statement to clarify our purpose and our values, and we hope to be able to attract talented software developers, entrepreneurs and Debian experts from our community.
Freexian s mission is to help Debian evolve to be the leading Linux distribution and a model to follow in the free software world.We want to achieve that by enabling passionate contributors to spend most of their time working on Debian and free software, combining lucrative work in support of enterprise customers, core contributions to Debian s processes, and personal goals.Extract from Freexian s mission statement
If Debian is an important part of your life, if improving Debian is something that you enjoy doing, if you have enough skills and Debian expertise to match some of the roles that we have described on our Join us page, then we would love to hear from you at gerants@freexian.com, so we can figure out a way to work together.

28 March 2022

Dirk Eddelbuettel: RcppCNPy 0.2.11

A minor maintenance release of the RcppCNPy package arrived on CRAN three days ago, but we skipped announcing it right then. RcppCNPy provides R with read and write access to NumPy files thanks to the cnpy library by Carl Rogers along with Rcpp for the glue to R. One of the vignettes created an issue at CRAN with one of the Python modules used, so we simply switched to pre-made vignettes just as we do for a few other packages. Other small changes that had accumulated since the previous release were a new section in the reticulate vignette as well as some more documentation on types as well as some updates to continuous integration and bdages.

Changes in version 0.2.11 (2022-03-24)
  • The reticulate vignette has new section on 3d-arrays.
  • Added clarification to the manual page that the default types are 32-bit integer and 64-bit double (as we are working with R here).
  • Several updates have been made to the CI system; it now runs r-ci.
  • The README.md was updated with some new badges.
  • The vignettes are now pre-made to avoid any external dependencies.

CRANberries also provides a diffstat report for the latest release. As always, feedback is welcome and the best place to start a discussion may be the GitHub issue tickets page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

27 March 2022

Russ Allbery: Review: A Song for a New Day

Review: A Song for a New Day, by Sarah Pinsker
Publisher: Berkley
Copyright: September 2019
ISBN: 1-9848-0259-3
Format: Kindle
Pages: 372
Luce Cannon was touring with a session band when the shutdown began. First came the hotel evacuation in the middle of the night due to bomb threats against every hotel in the state. Then came the stadium bombing just before they were ready to go on stage. Luce and most of the band performed anyway, with a volunteer crew and a shaken crowd. It was, people later decided, the last large stage show in the United States before the congregation laws shut down public gatherings. That was the end of Luce's expected career, and could have been the end of music, or at least public music. But Luce was stubborn and needed the music. Rosemary grew up in the aftermath: living at home with her parents well away from other people, attending school virtually, and then moving seamlessly into a virtual job for Superwally, the corporation that ran essentially everything. A good fix for some last-minute technical problems with StageHoloLive's ticketing system got her an upgraded VR hoodie and complimentary tickets to the first virtual concert she'd ever attended. She found the experience astonishing, prompting her to browse StageHoloLive job openings and then apply for a technical job and, on a whim, an artist recruiter role. That's how Rosemary found herself, quite nerve-wrackingly, traveling out into the unsafe world to look for underground musicians who could become StageHoloLive acts. A Song for a New Day was published in 2019 and had a moment of fame at the beginning of 2020, culminating in the Nebula Award for best novel, because it's about lockdowns, isolation, and the suppression of public performances. There's even a pandemic, although it's not a respiratory disease (it's some variety of smallpox or chicken pox) and is only a minor contributing factor to the lockdowns in this book. The primary impetus is random violence. Unfortunately, the subsequent two years have not been kind to this novel. Reading it in 2022, with the experience of the past two years fresh in my mind, was a frustrating and exasperating experience because the world setting is completely unbelievable. This is not entirely Pinsker's fault; this book was published in 2019, was not intended to be about our pandemic, and therefore could not reasonably predict its consequences. Still, it required significant effort to extract the premise of the book from the contradictory evidence of current affairs and salvage the pieces of it I still enjoyed. First, Pinsker's characters are the most astonishingly incurious and docile group of people I've seen in a recent political SF novel. This extends beyond the protagonists, where it could arguably be part of their characterization, to encompass the entire world (or at least the United States; the rest of the world does not appear in this book at all so far as I can recall). You may be wondering why someone bombs a stadium at the start of the book. If so, you are alone; this is not something anyone else sees any reason to be curious about. Why is random violence spiraling out of control? Is there some coordinated terrorist activity? Is there some social condition that has gotten markedly worse? Race riots? Climate crises? Wars? The only answer this book offers is a completely apathetic shrug. There is a hint at one point that the government may have theories that they're not communicating, but no one cares about that either. That leads to the second bizarre gap: for a book that hinges on political action, formal political structures are weirdly absent. Near the end of the book, one random person says that they have been inspired to run for office, which so far as I can tell is the first mention of elections in the entire book. The "government" passes congregation laws shutting down public gatherings and there are no protests, no arguments, no debate, but also no suppression, no laws against the press or free speech, no attempt to stop that debate. There's no attempt to build consensus for or against the laws, and no noticeable political campaigning. That's because there's no need. So far as one can tell from this story, literally everyone just shrugs and feels sad and vaguely compliant. Police officers exist and enforce laws, but changing those laws or defying them in other than tiny covert ways simply never occurs to anyone. This makes the book read a bit like a fatuous libertarian parody of a docile populace, but this is so obviously not the author's intent that it wouldn't be satisfying to read even as that. To be clear, this is not something that lasts only a few months in an emergency when everyone is still scared. This complete political docility and total incuriosity persists for enough years that Rosemary grows up within that mindset. The triggering event was a stadium bombing followed by an escalating series of random shootings and bombings. (The pandemic in the book only happens after everything is locked down and, apart from adding to Rosemary's agoraphobia and making people inconsistently obsessed with surface cleanliness, plays little role in the novel.) I lived through 9/11 and the Oklahoma City bombing in the US, other countries have been through more protracted and personally dangerous periods of violence (the Troubles come to mind), and never in human history has any country reacted to a shock of violence (or, for that matter, disease) like the US does in this book. At points it felt like one of those SF novels where the author is telling an apparently normal story and all the characters turn out to be aliens based on spiders or bats. I finally made sense of this by deciding that the author wasn't using sudden shocks like terrorism or pandemics as a model, even though that's what the book postulates. Instead, the model seems to be something implicitly tolerated and worked around: US school shootings, for instance, or the (incorrect but widespread) US belief in a rise of child kidnappings by strangers. The societal reaction here looks less like a public health or counter-terrorism response and more like suburban attitudes towards child-raising, where no child is ever left unattended for safety reasons but we routinely have school shootings no other country has at the same scale. We have been willing to radically (and ineffectually) alter the experience of childhood due to fears of external threat, and that's vaguely and superficially similar to the premise of this novel. What I think Pinsker still misses (and which the pandemic has made glaringly obvious) is the immense momentum of normality and the inability of adults to accept limitations on their own activities for very long. Even with school shootings, kids go to school in person. We now know that parts of society essentially collapse if they don't, and political pressure becomes intolerable. But by using school shootings as the model, I managed to view Pinsker's setup as an unrealistic but still potentially interesting SF extrapolation: a thought experiment that ignores countervailing pressures in order to exaggerate one aspect of society to an extreme. This is half of Pinsker's setup. The other half, which made less of a splash because it didn't have the same accident of timing, is the company Superwally: essentially "what if Amazon bought Walmart, Google, Facebook, Netflix, Disney, and Live Nation." This is a more typical SF extrapolation that left me with a few grumbles about realism, but that I'll accept as a plot device to talk about commercialization, monopolies, and surveillance capitalism. But here again, the complete absence of formal political structures in this book is not credible. Superwally achieves an all-pervasiveness that in other SF novels results in corporations taking over the role of national governments, but it still lobbies the government in much the same way and with about the same effectiveness as Amazon does in our world. I thought this directly undermined some parts of the end of the book. I simply did not believe that Superwally would be as benign and ineffectual as it is shown here. Those are a lot of complaints. I found reading the first half of this book to be an utterly miserable experience and only continued reading out of pure stubbornness and completionism. But the combination of the above-mentioned perspective shift and Pinsker's character focus did partly salvage the book for me. This is not a book about practical political change, even though it makes gestures in that direction. It's primarily a book about people, music, and personal connection, and Pinsker's portrayal of individual and community trust in all its complexity is the one thing the book gets right. Rosemary's character combines a sort of naive arrogance with self-justification in a way that I found very off-putting, but the pivot point of the book is the way in which Luce and her community extends trust to her anyway, as part of staying true to what they believe. The problem that I think Pinsker was trying to write about is atomization, which leads to social fragmentation into small trust networks with vast gulfs between them. Luce and Rosemary are both characters who are willing to bridge those gulfs in their own ways. Pinsker does an excellent job describing the benefits, the hurt, the misunderstandings, the risk, and the awkward process of building those bridges between communities that fundamentally do not understand each other. There's something deep here about the nature of solidarity, and how you need both people like Luce and people like Rosemary to build strong and effective communities. I've kept thinking about that part. It's also helpful for a community to have people who are curious about cause and effect, and who know how a bill becomes a law. It's hard to sum up this book, other than to say that I understand why it won a Nebula but it has deep world-building flaws that have become far more obvious over the past two years. Pinsker tries hard to capture the feeling of live music for both the listener and the performer and partly succeeded even for me, which probably means others will enjoy that part of the book immensely. The portrayal of the difficult dynamics of personal trust was the best part of the book for me, but you may have to build scaffolding and bracing for your world-building disbelief in order to get there. On the whole, I think A Song for a New Day is worth reading, but maybe not right now. If you do read it now, tell yourself at the start that this is absolutely not about the pandemic and that everything political in this book is a hugely simplified straw-man extrapolation, and hopefully you'll find the experience less frustrating than I found it. Rating: 6 out of 10

24 March 2022

Ingo Juergensmann: New Server NVMe Issues

My current server is somewhat aged. I bought it new in July 2014 with a 6-core Xeon E5-2630L, 32 GB RAM and 4x 3.5 hot-swappable drives. Gladly I had the opportunity to extend the memory to 128 GB RAM at no additional cost by using memory from my ex-employer. It also has 4x 2 TB WD Red HDDs with 5400 rpm hooked up to the SATA backplane, but unfortunately only two of them are SATA-3 with 6 Gbit/s. The new server is a used/refurbished Supermicro server with 2x 14-core Xeon E5-2683 and 256 GB RAM and 4x 3.5 hot-swappable drives. It also came with a Hardware-RAID SAS/SATA 8-port controller with BBU. I also ordered two slim drive kits (MCP-220-81504-0N & MCP-220-81506-0N) to be able to use 2x 3.5 slots for rotational HDDs as a cheap storage. Right now I added 2x 128 GB Supermicro SATA DOMs, 4x WD Red 4 TB SSDs and a Sonnet Fusion 4 4 Silent and 4x 1 TB Seagate Firecuda 520 NVMe disks. And here the issue starts: The NVMe should be capable of 4-5 GB/s, but they are connected to a PCIe 3.0 x16 port via the Sonnet Fusion 4 4, which itself features a PCIe bridge, so bifurbacation is not necessary. When doing some tests with bonnie++ I get around 1 GB/s transfer rates out of a RAID10 setup with all 4 NVMes. In fact, regardless of the RAID level there are only transfer rates of about 1 1.2 GB/s with bonnie++. (All software RAIDs with mdadm.) But also when constructing a RAID each NVMe gives around 300-600 MB/s in sync speed except for one exception: RAID1. Regardless of how many NVMe disks in a RAID1 setup the sync speed is up to 2.5 GB/s for each of the NVMe disks. So the lower transfer rates with bonnie++ or other RAID levels shouldn t be limited by bus speed nor by CPU speed. Alas, atop shows upto 100% CPU usage for all tests. I even tested In my understanding RAID10 should perform similar to RAID1 in terms of syncing and better and while bonnie++ tests (up to 2x write and 4x read speed compared to a single disk). For the bonnie++ tests I even made some tests that are available here. You can find the test parameters listed in the hostname column: Baldur is the hostname, then followed by the layout (near-2, far-2, offset-2), chunk size and concurrency of bonnie++. In the end there was no big impact of the chunk size of the RAID. So, now I m wondering what the reason for the slow performance of those 4x NVMe disks is? Bus speed of the PCIe 3.0 x16 shouldn t be the cause, because I assume that the software RAID will need to transfer the blocks in RAID1 as well as in RAID10 over the bus. Same goes for the CPU: the amount of CPU work should be roughly the same for RAID1 and for RAID10. RAID10 should even have an advantage because the blocks only need to be synced to 2 disks in a stripe set. Bonnie++ tests are a different topic for sure. But when testing reading with dd from the md-devices I only get around 1-1.5 GB/s as well. Even when using LVM RAID instead of LVM on top of md RAID. All NVMe disks are already set to 4k and IO scheduler is set to mq-deadline. Is there anything I could do to improve the performance of the NVMe disks? On the other head, pure transfer rates are not that important to a server that runs a dozen of VMs. Here the improved IOPS performance over rotation disks is a clear performance gain. But I m still curious if I could get maybe 2 GB/s out of a RAID10 setup with the NVMe disks. Then again having two independent RAID1 setups for MariaDB and for PostgreSQL databases might be a better choice over a single RAID10 setup?

23 March 2022

Matthew Garrett: AMD's Pluton implementation seems to be controllable

I've been digging through the firmware for an AMD laptop with a Ryzen 6000 that incorporates Pluton for the past couple of weeks, and I've got some rough conclusions. Note that these are extremely preliminary and may not be accurate, but I'm going to try to encourage others to look into this in more detail. For those of you at home, I'm using an image from here, specifically version 309. The installer is happy to run under Wine, and if you tell it to "Extract" rather than "Install" it'll leave a file sitting in C:\\DRIVERS\ASUS_GA402RK_309_BIOS_Update_20220322235241 which seems to have an additional 2K of header on it. Strip that and you should have something approximating a flash image.

Looking for UTF16 strings in this reveals something interesting:

Pluton (HSP) X86 Firmware Support
Enable/Disable X86 firmware HSP related code path, including AGESA HSP module, SBIOS HSP related drivers.
Auto - Depends on PcdAmdHspCoreEnable build value
NOTE: PSP directory entry 0xB BIT36 have the highest priority.
NOTE: This option will NOT put HSP hardware in disable state, to disable HSP hardware, you need setup PSP directory entry 0xB, BIT36 to 1.
// EntryValue[36] = 0: Enable, HSP core is enabled.
// EntryValue[36] = 1: Disable, HSP core is disabled then PSP will gate the HSP clock, no further PSP to HSP commands. System will boot without HSP.

"HSP" here means "Hardware Security Processor" - a generic term that refers to Pluton in this case. This is a configuration setting that determines whether Pluton is "enabled" or not - my interpretation of this is that it doesn't directly influence Pluton, but disables all mechanisms that would allow the OS to communicate with it. In this scenario, Pluton has its firmware loaded and could conceivably be functional if the OS knew how to speak to it directly, but the firmware will never speak to it itself. I took a quick look at the Windows drivers for Pluton and it looks like they won't do anything unless the firmware wants to expose Pluton, so this should mean that Windows will do nothing.

So what about the reference to "PSP directory entry 0xB BIT36 have the highest priority"? The PSP is the AMD Platform Security Processor - it's an ARM core on the CPU package that boots before the x86. The PSP firmware lives in the same flash image as the x86 firmware, so the PSP looks for a header that points it towards the firmware it should execute. This gives a pointer to a "directory" - a list of different object types and where they're located in flash (there's a description of this for slightly older AMDs here). Type 0xb is treated slightly specially. Where most types contain the address of where the actual object is, type 0xb contains a 64-bit value that's interpreted as enabling or disabling various features - something AMD calls "soft fusing" (Intel have something similar that involves setting bits in the Firmware Interface Table). The PSP looks at the bits that are set here and alters its behaviour. If bit 36 is set, the PSP tells Pluton to turn itself off and will no longer send any commands to it.

So, we have two mechanisms to disable Pluton - the PSP can tell it to turn itself off, or the x86 firmware can simply never speak to it or admit that it exists. Both of these imply that Pluton has started executing before it's shut down, so it's reasonable to wonder whether it can still do stuff. In the image I'm looking at, there's a blob starting at 0x0069b610 that appears to be firmware for Pluton - it contains chunks that appear to be the reference TPM2 implementation, and it broadly decompiles as valid ARM code. It should be viable to figure out whether it can do anything in the face of being "disabled" via either of the above mechanisms.

Unfortunately for me, the system I'm looking at does set bit 36 in the 0xb entry - as a result, Pluton is disabled before x86 code starts running and I can't investigate further in any straightforward way. The implication that the user-controllable mechanism for disabling Pluton merely disables x86 communication with it rather than turning it off entirely is a little concerning, although (assuming Pluton is behaving as a TPM rather than having an enhanced set of capabilities) skipping any firmware communication means the OS has no way to know what happened before it started running even if it has a mechanism to communicate with Pluton without firmware assistance. In that scenario it'd be viable to write a bootloader shim that just faked up the firmware measurements before handing control to the OS.

The bit 36 disabling mechanism seems more solid? Again, it should be possible to analyse the Pluton firmware to determine whether it actually pays attention to a disable command being sent. But even if it chooses to ignore that, if the PSP is in a position to just cut the clock to Pluton, it's not going to be able to do a lot. At that point we're trusting AMD rather than trusting Microsoft, but given that you're also trusting AMD to execute the code you're giving them to execute, it's hard to avoid placing trust in them.

Overall: I'm reasonably confident that systems that ship with Pluton disabled via setting bit 36 in the soft fuses are going to disable it sufficiently hard that the OS can't do anything about it. Systems that give the user an option to enable or disable it are a little less clear in that respect, and it's possible (but not yet demonstrated) that an OS could communicate with Pluton anyway. However, if that's true, and if the firmware never communicates with Pluton itself, the user could install a stub loader in UEFI that mimicks the firmware behaviour and leaves the OS thinking everything was good when it absolutely is not.

So, assuming that Pluton in its current form on AMD has no capabilities outside those we know about, the disabling mechanisms are probably good enough. It's tough to make a firm statement on this before I have access to a system that doesn't just disable it immediately, so stay tuned for updates.

comment count unavailable comments

21 March 2022

Gunnar Wolf: Long, long, long live Emacs after 39 years

Reading Planet Debian (see, Sam, we are still having a conversation over there? ), I read Anarcat s 20+ years of Emacs. And.. Well, should I brag contribute to the discussion? Of course, why not? Emacs is the first computer program I can name that I ever learnt to use to do something minimally useful. 39 years ago.
From the Space Cadet keyboard that (obviously ) influenced Emacs early design
The Emacs editor was born, according to Wikipedia, in 1976, same year as myself. I am clearly not among its first users. It was already a well-established citizen when I first learnt it; I am fortunate to be the son of a Physics researcher at UNAM, My father used to take me to his institute after he noticed how I was attracted to computers; we would usually spend some hours there between 7 and 11PM on Friday nights. His institute had a computer room where they had very sweet gear: Some 10 Heathkit terminals quite similar to this one: The terminals were connected (via individual switches) to both a PDP-11 and a Foonly F2 computers. The room also had a beautiful thermal printer, a beautiful Tektronix vectorial graphics output terminal, and some other stuff. The main user for my father was to typeset some books; he had recently (1979) published Integral Transforms in Science and Engineering (that must be my first mention in scientific literature), and I remember he was working on the proceedings of a conference he held in Oaxtepec (the account he used in the system was oax, not his usual kbw, which he lent me). He was also working on Manual de Lenguaje y Tipograf a Cient fica en Castellano, where you can see some examples of TeX; due to a hardware crash, the book has the rare privilege of being a direct copy of the output of the thermal printer: It was not possible to produce a higher resolution copy for several years But it is fun and interesting to see what we were able to produce with in-house tools back in 1985! So, what could he teach me so I could use the computers while he worked? TeX, of course. No, no LaTeX (that was published in 1984). LaTeX is a set of macros developed initially by Leslie Lamport, used to make TeX easier; TeX was developed by Donald Knuth, and if I have this information correct, it was Knuth himself who installed and demonstrated TeX in the Foonly computer, during a visit to UNAM. Now, after 39 years hammering at Emacs buffers Have I grown extra fingers? Nope. I cannot even write decent elisp code, and can barely read it. I do use org-mode (a lot!) and love it; I have written basically five books, many articles and lots of presentations and minor documents with it. But I don t read my mail or handle my git from Emacs. I could say, I m a relatively newbie after almost four decades. Four decades When we got a PC in 1986, my father got the people at the Institute to get him memacs (micro-emacs). There was probably a ten year period I barely used any emacs, but always recognized it. My fingers hve memorized a dozen or so movement commands, and a similar number of file management commands. And yes, Emacs and TeX are still the main tools I use day to day.

Antoine Beaupr : 20+ years of Emacs

I enjoyed reading this article named "22 years of Emacs" recently. It's kind of fascinating, because I realised I don't exactly know for how long I've been using Emacs. It's lost in the mists of history. If I would have to venture a guess, it was back in the "early days", which in that history is mapped around 1996-1997, when I installed my very own "PC" with FreeBSD 2.2.x and painstakingly managed to make XFree86 run on it. Modelines. Those were the days... But I digress.

I am old... The only formal timestamp I can put is that my rebuilt .emacs.d git repository has its first commit in 2002. Some people reading this may be born after that time. This means I'm at least significantly older than those people, to put things gently. Clever history nerds will notice that the commit is obviously fake: Git itself did not exist until 2005. But ah-ah! I was already managing my home directory with CVS in 2001! I converted that repository into git some time in 2009, and therefore you can see all my embarrassing history, including changes from two decades ago. That includes my first known .emacs file which is just bizarre to read right now: 200 lines, most of which are "customize" stuff. Compare with the current, 1000+ lines init.el which is also still kind of a mess, but actually shares very little with the original, thankfully. All this to say that in those years (decades, really) of using Emacs, I have had a very different experience than credmp who wrote packages, sent patches, and got name dropping from other developers. My experience is just struggling to keep up with everything, in general, but also in Emacs.

... and Emacs is too fast for me It might sound odd to say, but Emacs is actually moving pretty fast right now. A lot of new packages are coming out, and I can hardly keep up.
  • I am not using org mode, but did use it for time (and task) tracking for a while (and for invoicing too, funky stuff).
  • I am not using mu4e, but maybe I'm using something better (notmuch) and yes, I am reading my mail in Emacs, which I find questionable from a security perspective. (Sandboxing untrusted inputs? Anyone?)
  • I am using magit, but only when coding, so I do end up using git on the command line quite a bit anyways.
  • I do have which-key enabled, and reading about it reminded me I wanted to turn it off because it's kind of noisy and I never remember I can actually use it for anything. Or, in other words, I don't even remember the prefix key or, when I do, there's too many possible commands after for it to be useful.
  • I haven't setup lsp-mode, let alone Eglot, which I just learned about reading the article. I thought I would be super shiny and cool by setting up LSP instead of the (dying?) elpy package, but I never got around to it. And now it seems lsp-mode is uncool and I should really do eglot instead, and that doesn't help. UPDATE: I finally got tired and switched to lsp-mode. The main reason for choosing it over eglot is that it's in Debian (and eglot is not). (Apparently, eglot has more chance of being upstreamed, "when it's done", but I guess I'll cross that bridge when I get there.) lsp-mode feels slower than elpy but I haven't done any of the performance tuning and this will improve even more with native compilation (see below). I already had lsp-mode partially setup in Emacs so I only had to do this small tweak to switch and change the prefix key (because s-l or mod is used by my window manager). I also had to pin LSP packages to bookworm here and here.
  • I am not using projectile. It's on some of my numerous todo lists somewhere, surely. I suspect it's important to getting my projects organised, but I still live halfway between the terminal and Emacs, so it's not quite clear what I would gain.
  • I had to ask what native compilation was or why it mattered the first time I heard of it. And when I saw it again in the article, I had to click through to remember.
Overall, I feel there's a lot of cool stuff in Emacs out there. But I can't quite tell what's the best of which. I can barely remember which completion mechanism I use (company, maybe?) or what makes my mini-buffer completion work the way it does. Everything is lost in piles of customize and .emacs hacks that is constantly changing. Because a lot is in third-party packages, there are often many different options and it's hard to tell which one we should be using.

... or at least fast enough And really, Emacs feels fast enough for me. When I started, I was running Emacs on a Pentium I, 166MHz, with 8MB of RAM (eventually upgraded to 32MB, whoohoo!). Back in those days, the joke was that EMACS was an acronym for "Eight Megs, Always Scratching" and now that I write this down, I realize it's actually "Eight Megs, and Constantly Swapping", which doesn't sound as nice because you could actually hear Emacs running on those old hard drives back in the days. It would make a "scratching" noise as the hard drive heads would scramble maniacally to swap pages in and out of swap to make room for the memory-hungry editor. Now Emacs is pretty far down the list of processes in top(1) regardless of how you look at it. It's using 97MB of resident memory and close to 400MB of virtual memory, which does sound like an awful lot compared to my first computer... But it's absolutely nothing compared to things like Signal-desktop, which somehow manages to map a whopping 20.5GB virtual memory. (That's twenty Gigabytes of memory for old timers or time travelers from the past, and yes, that is now a thing.) I'm not exactly sure how much resident memory it uses (because it forks multiple processes), probably somewhere around 300MB of resident memory. Firefox also uses gigabytes of that good stuff, also spread around the multiple processes, per tab. Emacs "feels" super fast. Typing latency is noticeably better in Emacs than my web browser, and even beats most terminal emulators. It gets a little worse when font-locking is enabled, unfortunately, but it's still feels much better. And all my old stuff still works in Emacs, amazingly. (Good luck with your old Netscape or ICQ configuration from 2000.) I feel like an oldie, using Emacs, but I'm really happy to see younger people using it, and learning it, and especially improving it. If anything, one direction I would like to see it go is closer to what web browsers are doing (yes, I know how bad that sounds) and get better isolation between tasks. An attack on my email client shouldn't be able to edit my Puppet code, and/or all files on my system, for example. And I know, fundamentally, that's a really hard challenge in Emacs. But if you're going to treat your editor as your operating system (or vice versa, I lost track of where we are now that there's an Emacs Window Manager, which I do not use), at least we should get that kind of security. Otherwise I'll have to find a new mail client, and that's really something I try to limit to once a decade or so.

17 March 2022

Dirk Eddelbuettel: Rcpp 1.0.8.3: Hotfixing Hotfix

rcpp logo An even newer hot-fix release 1.0.8.3 of Rcpp follows the 1.0.8.2 release of a few days ago and got to CRAN this morning. A Debian upload will follow shortly, and Windows and macOS binaries will appear at CRAN in the next few days. This release again breaks with the six-months cycle started with release 1.0.5 in July 2020. When we addressed the CRAN request in 1.0.8.2 we forgot to dial testing down to their desired level (as three-part release numbers do automagically for us, whereas four-part do not). This is now taken care of, along with the hot-fix that was in 1.0.8.2 already. Rcpp has become the most popular way of enhancing R with C or C++ code. Right now, around 2522 packages on CRAN depend on Rcpp for making analytical code go faster and further, along with 239 in BioConductor. The full list of details for these two interim releases (and hence all changes accumulated since the last regular release, 1.0.8 in January) follows.

Changes in Rcpp hotfix release version 1.0.8.3 (2022-03-14)
  • Changes in Rcpp API:
    • Accomodate C++98 compilation by adjusting attributes.cpp (Dirk in #1193 fixing #1192)
    • Accomodate newest compilers replacing deprecated std::unary_function and std::binary_function with std::function (Dirk in #1202 fixing #1201 and CRAN request)
  • Changes in Rcpp Documentation:
    • Adjust one overflowing column (Bill Denney in #1196 fixing #1195)
  • Changes in Rcpp Deployment:
    • Accomodate four digit version numbers in unit test (Dirk)
    • Do not run complete test suite to limit test time to CRAN preference (Dirk)

Thanks to my CRANberries, you can also look at a diff to the previous release. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. Bugs reports are welcome at the GitHub issue tracker as well (where one can also search among open or closed issues); questions are also welcome under rcpp tag at StackOverflow which also allows searching among the (currently) 2843 previous questions. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

12 March 2022

Dirk Eddelbuettel: RcppGSL 0.3.11: Small Maintenance

A new release 0.3.11 of RcppGSL is now on CRAN. The RcppGSL package provides an interface from R to the GNU GSL by relying on the Rcpp package. This release updates src/Makefile.ucrt to use the RTools42 libraries. Details follow from the NEWS file.

Changes in version 0.3.11 (2022-03-12)
  • The UCRT Makefile was updated
  • Minor edits to README.md were made

Courtesy of CRANberries, a summary of changes in the most recent release is also available. More information is on the RcppGSL page. Questions, comments etc should go to the issue tickets at the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dirk Eddelbuettel: Rcpp 1.0.8.2: Hotfix release per CRAN request

rcpp logo A new hot-fix release 1.0.8.2 of Rcpp just got to CRAN. It will also be uploaded to Debian shortly, and Windows and macOS binaries will appear at CRAN in the next few days. This release breaks with the six-months cycle started with release 1.0.5 in July 2020 as CRAN desired an update to silence nags from the newest clang version which turned a little loud over a feature deprecated in C++11 (namely std::unary_function() and std::binary_function()). This was easy to replace with std::function() which we did. The release also contains a minor bugfix relative to 1.0.8 and C++98 builds, and minor correction to one pdf vignette. The release was fully tested by us and CRAN as usual against all reverse dependencies. Rcpp has become the most popular way of enhancing R with C or C++ code. Right now, around 2519 packages on CRAN depend on Rcpp for making analytical code go faster and further, along with 239 in BioConductor. The full list of details for this interim release follows.

Changes in Rcpp hotfix release version 1.0.8.2 (2022-03-10)
  • Changes in Rcpp API:
    • Accomodate C++98 compilation by adjusting attributes.cpp (Dirk in #1193 fixing #1192)
    • Accomodate newest compilers replacing deprecated std::unary_function and std::binary_function with std::function (Dirk in #1202 fixing #1201 and CRAN request)
  • Changes in Rcpp Documentation:
    • Adjust one overflowing column (Bill Denney in #1196 fixing #1195)
  • Changes in Rcpp Deployment:
    • Accomodate four digit version numbers in unit test (Dirk)

Thanks to my CRANberries, you can also look at a diff to the previous release. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. Bugs reports are welcome at the GitHub issue tracker as well (where one can also search among open or closed issues); questions are also welcome under rcpp tag at StackOverflow which also allows searching among the (currently) 2843 previous questions. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.