Search Results: "nate"

27 April 2020

Dirk Eddelbuettel: #26: Upgrading to R 4.0.0

Welcome to the 26th post in the rationally regularized R revelations series, or R4 for short. R 4.0.0 was released two days ago, and a casual glance at some social media conversations appears to suggest quite some confusion, almost certainly some misunderstandings, and possibly also a fair amount of fear, uncertainty, and doubt about the process. So I thought I could show how I upgrade my own main workstation, live and in colour without a safety net. (Almost: I did upgrade my laptop yesterday which went swimmingly, if more slowly.) So here is a fresh video about upgrading to R 4.0.0, with some support slides as usual:

The slides used in the video are at this link. A few quick follow-ups to the live nature of this. The pbdZMQ package did in fact install smoothly once the (Ubuntu) -dev packages for Zero MQ were (re-)installed; then IRkernel also followed. BioConductor completed once I realized that GOSemSim needed the annotation package GO.db to be updated, that allowed MNF to install. So the only bug, really, was the circular depdency between pkgload and testthat. Overall, not bad at all for a quick afternoon session! And as mentioned, if you are interested and have questions concerning use of R on a .deb based system like Debain or Ubuntu (or Mint or ), the r-sig-debian list is a very good and friendly place to ask them. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

26 April 2020

Enrico Zini: Some Italian women

Artemisia Gentileschi - Wikipedia
art history people archive.org
Artemisia Lomi or Artemisia Gentileschi (US: / d nt l ski, -ti -/, Italian: [arte mi zja d enti leski]; July 8, 1593 c. 1656) was an Italian Baroque painter, now considered one of the most accomplished seventeenth-century artists working in the dramatic style of Caravaggio. In an era when women had few opportunities to pursue artistic training or work as professional artists, Artemisia was the first woman to become a member of the Accademia di Arte del Disegno in Florence and had an international clientele.
Maria Pellegrina Amoretti (1756 1787), was an Italian lawyer. She is referred to as the first woman to graduate in law in Italy, and the third woman to earn a degree.
Laura Maria Caterina Bassi (October 1711 20 February 1778) was an Italian physicist and academic. She received a doctoral degree in Philosophy from the University of Bologna in May 1732. She was the first woman to earn a professorship in physics at a university. She is recognized as the first woman in the world to be appointed a university chair in a scientific field of studies. Bassi contributed immensely to the field of science while also helping to spread the study of Newtonian mechanics through Italy.
Maria Gaetana Agnesi (UK: / n je zi/ an-YAY-zee,[1] US: / n -/ ahn-,[2][3] Italian: [ma ri a ae ta na a zi, - e z-];[4] 16 May 1718 9 January 1799) was an Italian mathematician, philosopher, theologian, and humanitarian. She was the first woman to write a mathematics handbook and the first woman appointed as a mathematics professor at a university.[5]
Elena Lucrezia Cornaro Piscopia (US: /k r n ro p sko pi /,[4] Italian: [ lena lu kr ttsja kor na ro pi sk pja]) or Elena Lucrezia Corner (Italian: [kor n r]; 5 June 1646 26 July 1684), also known in English as Helen Cornaro, was a Venetian philosopher of noble descent who in 1678 became one of the first women to receive an academic degree from a university, and the first to receive a Doctor of Philosophy degree.
Maria Tecla Artemisia Montessori (/ m nt s ri/ MON-tiss-OR-ee, Italian: [ma ri a montes s ri]; August 31, 1870 May 6, 1952) was an Italian physician and educator best known for the philosophy of education that bears her name, and her writing on scientific pedagogy. At an early age, Montessori broke gender barriers and expectations when she enrolled in classes at an all-boys technical school, with hopes of becoming an engineer. She soon had a change of heart and began medical school at the Sapienza University of Rome, where she graduated with honors in 1896. Her educational method is still in use today in many public and private schools throughout the world.
Rita Levi-Montalcini OMRI OMCA (US: / le vi mo nt l t i ni, l v-, li vi m nt l -/, Italian: [ ri ta l vi montal t i ni]; 22 April 1909 30 December 2012) was an Italian Nobel laureate, honored for her work in neurobiology. She was awarded the 1986 Nobel Prize in Physiology or Medicine jointly with colleague Stanley Cohen for the discovery of nerve growth factor (NGF). From 2001 until her death, she also served in the Italian Senate as a Senator for Life. This honor was given due to her significant scientific contributions. On 22 April 2009, she became the first Nobel laureate ever to reach the age of 100, and the event was feted with a party at Rome's City Hall. At the time of her death, she was the oldest living Nobel laureate.
Margherita Hack Knight Grand Cross OMRI (Italian: [mar e ri ta (h)ak]; 12 June 1922 29 June 2013) was an Italian astrophysicist and scientific disseminator. The asteroid 8558 Hack, discovered in 1995, was named in her honour.
Samantha Cristoforetti (Italian pronunciation: [sa manta kristofo retti]; born 26 April 1977, in Milan) is an Italian European Space Agency astronaut, former Italian Air Force pilot and engineer. She holds the record for the longest uninterrupted spaceflight by a European astronaut (199 days, 16 hours), and until June 2017 held the record for the longest single space flight by a woman until this was broken by Peggy Whitson and later by Christina Koch. She is also the first Italian woman in space. Samantha Cristoforetti is also known as the first person who brewed an espresso in space.

Russ Allbery: Review: The Last Goodbye

Review: The Last Goodbye, by Matt Potter
Publisher: Silvertail
Copyright: 2014-2016
Printing: 2016
ISBN: 1-909269-42-5
Format: Kindle
Pages: 308
In the contested space between the interested amateur and the trained expert lies the enthusiast. A topic has, for often inexplicable reasons, caught fire in their thoughts and become a mild obsession. They may not have formal training or systematic conceptual grounding beneath their interest, but they partly make up for that lack with sustained fascination. They research widely and obscurely about their particular focus, develop novel theories, see distinctions and classifications that few others would bother to investigate, and present their discoveries to anyone who will stand still long enough. And occasionally, when that interest happens to coincide with writing skill, they produce some surprisingly excellent essays or books. Matt Potter's enthusiasm is resignation letters.
Every damaging resignation letter, every cornered truth attack, every out-of-control speech by a former friend, is more than just an inconvenience, to be countered with positive spin and internal memos: it's an open challenge to the official version of the story, the perfectly controlled brand. They are breaks in an otherwise perfectly planned, smoothly executed narrative of the powerful. Holes in the program code. A rare, irresistible chance to hack into history's shiny, unstoppable operation.
The Last Goodbye: A History of the World in Resignation Letters is not, in truth, a history of the world. It is, first and foremost, a taxonomy, because there are types of resignation letters. The opening chapter, the truth bomb, is the type that one would expect upon discovering that someone wrote a book on the topic (that wasn't advice on how to write a good one). But there are other types, less heavy on the fireworks but just as fascinating. The unquotable expert construction. The knife in the back. The incoherent scream of rage. But also the surprisingly gentle and graceful conclusion.
It is the question that the letters themselves try in vain to answer, over and over again even as they explain, analyse, protest and bear witness to a million other details. The question is: Why? All the forces in the universe stack up against unburdening ourselves in a resignation letter. Professionally, it can be suicide. In practical terms, it is often self-defeating. Self-help books coach against unleashing its force; colleagues and confidantes urge caution, self-restraint. And yet we do it, and damn the consequences. We have no choice but to speak in sorrow, love, grief, cold anger, thirst for revenge, wounded pride, the pain of injustice, loyalty, pangs of regret, throes of vengeful madness, deluded righteousness, panic, black distress, isolation, ecstasies of martyrdom, and a million other shades of human extremity we need to say our piece even as we leave the stage.
The risk of the enthusiast's book is that the lack of structural grounding can leave the conclusions unsupported. A fair critique of this book is that it contains a lot of amateur sociology. Potter has a journalist's eye for motive and narrative, but some of his conclusions may not be warranted. But he compensates for the lack of rigor with, well, enthusiasm. Potter is fascinated by resignation letters and the insight they offer, and that fascination is irresistibly contagious. It's probably obvious that the chapters on truth bombs, fuck yous, and knives in the back have visceral appeal. The resignation letter as a force of truth-telling, as the revenge of a disregarded peon, as a crack in the alliance between the powerful that may let some truth shine through, is what got me to buy this book. And Potter doesn't disappoint; he quotes from both famous and nearly unknown examples, dissects the writing and the intent, and gives us a ringside seat to a few effective acts of revenge. That's not the best part of this book, though. The chapter that I will remember the longest is Potter's dissection of the constructed resignation letter. The carefully drafted public relations statement, the bland formality, the attempt to make a news story disappear. The conversation-ender. It's a truism that any area of human endeavor involves more expertise than those who have only observed it from the outside will realize, but I had never thought to apply that principle to the ghost-written resignation letter. The careful non-apology, the declaration that one has "become a distraction," the tell-tale phrasing of "spending more time with my family" is offensive in its bland dishonesty. But Potter shows that the blandness is expertly constructed to destroy quotability. Those statements are nearly impossible to remember or report on because they have been built that way: nouns carefully separated from verbs, all force dissipated by circuities and modifiers, and subtle grammatical errors introduced to discourage written news from including direct quotes. Potter's journalism background shines here because he can describe the effect on news reporting. He walks the reader through the construction to reveal that the writing is not incompetent but instead is skillfully bad in a way that causes one's attention to skitter off of it. The letter vanishes into its own vagueness. The goal is to smother the story in such mediocrity that it becomes forgettable. And it works. I've written several resignation letters of my own. Somewhat unusually, I've even sent several of them, although (as Potter says is typical) fewer than I've written. I've even written (and sent) the sort of resignation letter that every career advisor will say to never send. Potter's discussion of the motives and thought process behind those letters rang true for me. It's a raw and very human moment, one is never in as much control of it as one wishes, the cracks and emotions break through in the writing, and often those letters are trying to do far too many things at the same time. But it's also a moment in which one can say something important and have others listen, which can be weirdly challenging to do in the normal course of a job. Potter ends this book beautifully by looking at resignation letters that break or transcend the mold of most of the others he's examined: letters where the author seems to have found some peace and internal understanding and expresses that simply and straightforwardly. I found this surprisingly touching after the emotional roller-coaster of the rest of the book, and a lovely note on which to end. This is a great book. Potter has a good eye for language and the emotion encoded in it, a bracing preference for the common worker or employee over the manager or politician, and the skill to produce some memorable turns of phrase. Most importantly, he has the enthusiast's love of the topic. Even if you don't care about resignation letters going in, it will be hard to avoid some fascination with them by the end of this book. Recommended. This book was originally published as F*ck You and Goodbye in the UK. Rating: 8 out of 10

25 April 2020

Reinhard Tartler: Building Packages with Buildah in Debian

1 Building Debian Packages with buildah
Building packages in Debian seems to be a solved problem. But is it? At the bottom, installing the dpkg-dev package provides all the basic tools needed. Assuming that you already succeeded with creating the necessary packaging metadata (i.e., debian/changelog, debian/control, debian/copyright, etc., and there are great helper tools for this such ash dh-make, dh-make-golang, etc.,) it should be as simple as invoking the dpkg-buildpackage tool. So what's the big deal here?

The issue is that dpkg-buildpackage expects to be called with an appropriately setup build context, that is, it needs to be called in an environment that satisfies all build dependencies on the system. Let's say you are building a package for Debian unstable on your Debian stable system (this is the common scenario for the official Debian build machines), you would need your build to link against libraries in unstable, not stable. So how to tell the package build process where to find its dependencies?

The answer (in Debian and many other Linux distributions) is you do not at all. This is actually a somewhat surprising answer for software developers without a Linux distribution development background1. Instead, chroots "simulate" an environment that has all dependencies that we want to build against at the system locations, that is. /usr/lib, etc.

Chroots are basically full system installations in a subdirectory that includes system and application libraries. In order to use them, a package build needs to use the chroot(2) system call, which is a privileged operation. Also creating these system installations is a somewhat finicky process. In Debian, we have tools that make this process easier, the most popular ones are probably pbuilder(1)2 and sbuild(1)3. Still, they are somewhat clumsy to use, add significant levels of abstraction in the sense that they do quite a bit of magic behind the scenes to work hide the fact that privileged operations (need root to run chroot(2), etc.) are required. They are also somewhat brittle, for instance, if a build process is aborted (SIGKILL, or system crash), you may end up with temporary directories and files under userids other than your own that again may require root-privileges to cleanup.

What if there was an easy way to do all of the above without any process running as root? Enter rootless buildah.

Modern Linux container technologies allow unprivileged users to "untie" a process from the system (cf. the unshare(2) system call). This means that a regular user may run a process (and its child processes) in an "environment" where system calls behave differently and provide a configurable amount of isolation levels. This article demonstrates a novel build tool buildah, which is:
  • easy to setup build environment from the command line
  • secure, as no process needs to run as root
  • simple in architecture, requires no running daemon like for example docker
  • convenient to use: you can debug your debian/rules interactively
Architecturally, buildah is written in golang and compiled as a (mostly) statically linked executable. It builds on top a number of libraries written in golang, including github.com/containers/storage and github.com/containers/image. The overlay functionality is provided by the fuse-overlayfs(1) utility.

1.1 Preparation:
The Kernel in Debian bullseye (and in buster, and recent Ubuntu kernels to the best of my knowledge) do support usernamespaces, but leave them disabled by default. Here is how to enable them:
echo kernel.unprivileged_userns_clone = 1   sudo tee -a /etc/sysctl.d/containers.conf
systemctl -p /etc/sysctl.d/containers.conf
I have to admit that I'm not entirely sure why the Debian kernels don't enable usernamespaces by default. I've found a reference on stackexchange4 that claims this disables some "hardening" features in the Debian kernel. I also understand this step is not necessary if you chose to compile and run a vanilla upstream kernel. I'd appreciate a better reference and am happy to update this text.
$ c=$(buildah from docker.io/debian:sid)
Getting image source signatures
Copying blob 2bbc6b8c460d done
Copying config 9b90abe801 done
Writing manifest to image destination
Storing signatures
This command downloads the image from docker.io and stores it locally in your home directory. Let's install essential utilities for building Debian packages:
1: buildah run $c apt-get update -qq
2: buildah run $c apt-get install dpkg-dev -y
3: buildah config --workingdir /src $c
4: buildah commit $c dpkg-dev
The command on line 1 and 2 execute the installation of compilers and dpkg development tools such as dpkg-buildpackage, etc. The buildah config command in line 4 arranges that whenever you start a shell in the container, the current working directory in the container is in /src. Don't worry about this location not existing yet, we will make sources from your host system available there. The last command creates an OCI image with the name dpkg-dev. BTW, the name you use for the image in the commit command can be used in podman (but not the "containers"). See 5 and 6 for a comparison between podman and buildah.
buildah images -a
This output might look like this:
  REPOSITORY                 TAG      IMAGE          ID   CREATED   SIZE             
localhost/dpkg-dev latest b85c34f95d3e 16 seconds ago 406 MB
docker.io/library/debian sid 9b90abe801db 11 hours ago 124 MB

1.2 Running a package build
Now we have a working container with the reference in the variable $c. To use it conveniently with source packages that I have stored in /srv/scratch/packages/containers, let's introduce a shell alias r like this:
alias r='buildah run --tty -v /srv/scratch/packages/containers:/src $c '
This allows you to easily execute commands in that container:
r pwd
r bash
The last command will give you an interactive shell that we'll be using for building packages!
siretart@x1:~/scratch/packages/containers/libpod$ r bash

root@x1:/src# cd golang-github-openshift-imagebuilder

root@x1:/src/golang-github-openshift-imagebuilder# dpkg-checkbuilddeps
dpkg-checkbuilddeps: error: Unmet build dependencies: debhelper (>= 11) dh-golang golang-any golang-github-containers-storage-dev (>= 1.11) golang-github-docker-docker-dev (>= 18.09.3+dfsg1) golang-github-fsouza-go-dockerclient-dev golang-glog-dev

root@x1:/src/golang-github-openshift-imagebuilder# apt-get build-dep -y .
Note, using directory '.' to get the build dependencies
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
[...]

root@x1:/src/golang-github-openshift-imagebuilder# dpkg-checkbuilddeps

root@x1:/src/golang-github-openshift-imagebuilder#
Now we have a environment that has all build-dependencies available and we are ready to build the package:
root@x1:/src/golang-github-openshift-imagebuilder# dpkg-buildpackage -us -uc -b
Assuming the package builds, the package build results are placed in /src inside the container, and are visible at ~/scratch/packages/containers on the host. There you can inspect, extract or even install them. The latter part allows you to interactively rebuild packages against updated dependencies, without the need of setting up an apt archive or similar.

2 Availability
The buildah(1) tool is available in debian/bullseye since 2019-12-17. Since is a golang binary, it only links dynamically against system libraries such as libselinux and libseccomp, which are all available in buster-backports. I'd expect buildah to just work on a debian/buster system as well provided you install those system libraries and possibly the backported Linux kernel.

Keeping the package at the latest upstream version is challenging because of its fast development pace that picks up new dependencies on golang libraries and new versions of existing libraries with every new upstream release. In general, this requires updating several source packages, and in many cases also uploading new source packages to the Debian archive that need to be processed by the Debian ftp-masters.

As an exercise, I suggest to install the buildah package from bullseye, 'git clone' the packaging repository from https://salsa.debian.org/go-team/packages/golang-github-containers-buildah and build the latest version yourself. Note, I would expect the above to even work on a Fedora laptop.

The Debian packages have not swept into the Ubuntu distributions yet, but I expect them to be included in the Ubuntu 20.10 release. In the mean time, ubuntu users can install the package that are provided by the upstream maintainers in the "Project Atomic" PPA at https://launchpad.net/~projectatomic/+archive/ubuntu/ppa

3 Related Tools
The buildah tool is accompanied with two "sister" tools:

The Skopeo package provides tooling to work with remote images registries. This allows you download, upload images to remote registries, convert container images between different formats. It has been available in Debian since 2020-04-20.

The podman tool is a 'docker' replacement. It provides a command-line interface that mimics the original docker command to an extend that a user familiar with docker might want to place this in their ~/.bashrc file:
alias docker='podman'
Unfortunately, at the time of writing podman is still being processed by the ftp-masters since 01-03-2020. At this point, I recommend building the package from our salsa repository at https://salsa.debian.org/debian/libpod

4 Conclusion
Building packages in the right build context is a fairly technical issue for which many tools have been written for. They come with different trade-off when it comes to usability. Containers promise a secure alternative to the tried and proven chroot-based approaches, and the buildah makes using this technology very easy to use.

I'd love to get into a conversation with you on how these tools work for you, and would like to encourage participation and assistance with keeping the complicated software stack up-to-date in Debian (and by extension, in derived distribution such as Ubuntu, etc.).

Footnotes:
1
At my day-job, we build millions of lines of C++ code on Solaris10 and AIX6, where concepts such as "chroots" are restricted to the super user 'root' and is therefore not available to developers, not even through wrappers. Instead, libraries and headers are installed into "refroots", that is, subdirectories that mimic the structure of the "sysroot" directories that are used in the embedded Linux community for cross-compiling packages, and we use Makefiles that set include flags (-I rules and -L flags) to tell the compiler and linker where to look.

24 April 2020

Norbert Preining: (Working) Unison Sync between for Debian/Buster and Sid

Since many years I am a big big fan of Unison. It is a two-way synchronization tool, meaning that changes on both sides are carried over as long as there is no conflict. Unison is the bread and butter for keeping my computers (main desktop, laptop, server) in sync. There is only one problem with Unison it is extremely picky about with what kind of Unison on the other side it speaks: Even the very same version, compiled with a slightly different OCaml compiler version, already produces incompatibilities. And with me having to sync between Debian/Buster (my server) and Debian/sid (my other machines), this always ended in tears. But heyhoh! the pain is over! My solution up to now was to keep a Unison binary from Buster on my Sid system, and used that binary when syncing to Buster server, but the normal Unison binary when syncing between two sid computers. This created lots of pain, because, when using a incompatible Unison binary, already a simple connect without a sync trial messed up the state files and required a complete resync. And so it happened often that I used the wrong binary, and had to resync a few gigabyte, what a pain. The Debian maintainer is working with upstream to improve this rather unfortunate situation, so maybe this will change in the future, but I need a solution now. BTW, thanks a lot to the excellent work of the current maintainers, setting up the new package was a blaze and only needed a bit of adjustments for the changed layout of recent sources. Big kudos to the maintainers of unison!!! So I have ventured out to create my own Unison package for Debian. And what I mean for Debian/Buster and Debian/sid, so that they are compatible, by using the very same compiler and the very same source and the very same dependencies to build the binaries. Fortunately, the lessons learned from my KDE/Plasma updates for Debian helped, and with a bit of magic and the nice service from the openSUSE build service, I now have packages for Unison ready for Debian/Buster and Debian/sid that are compatible, and in addition also the latest version, not the 5 year old package that is currently Debian. So here it is, the Debian Unison repository, binaries of (currently) Unison 2.51.2 are available for both i386 and amd64: For Debian/Buster
deb http://download.opensuse.org/repositories/home:/npreining:/debian-unison/Debian_10  ./
For Debian/sid:
deb http://download.opensuse.org/repositories/home:/npreining:/debian-unison/Debian_Unstable  ./
To use this repository, you need to import my OBS key obs-npreining.asc, best put it as is into /etc/apt/trusted.gpg.d/obs-npreining.asc. The sources are available at the OBS project debian-unison. For those interested, I had to backport a few package to Buster to ensure that the builds agree: debhelper (because dh_dwz is buggy in Buster), dh-ocaml, findlib, lablgl, lablgtk2, labltk, and ocaml. These packages are available in the OBS debian-ocaml-backports repository, and are also apt accessible if necessary. I guess you can conjure up the correct line ;-). That s it. I can now forget about all the differences between versions and happily sync between Buster and sid. Enjoy.

21 April 2020

Matthew Garrett: Linux kernel lockdown, integrity, and confidentiality

The Linux kernel lockdown patches were merged into the 5.4 kernel last year, which means they're now part of multiple distributions. For me this was a 7-year journey, which means it's easy to forget that others aren't as invested in the code as I am. Here's what these patches are intended to achieve, why they're implemented in the current form and what people should take into account when deploying the feature.

Root is a user - a privileged user, but nevertheless a user. Root is not identical to the kernel. Processes running as root still can't dereference addresses that belong to the kernel, are still subject to the whims of the scheduler and so on. But historically that boundary has been very porous. Various interfaces make it straightforward for root to modify kernel code (such as loading modules or using /dev/mem), while others make it less straightforward (being able to load new ACPI tables that can cause the ACPI interpreter to overwrite the kernel, for instance). In the past that wasn't seen as a significant issue, since there were no widely deployed mechanisms for verifying the integrity of the kernel in the first place. But once UEFI secure boot became widely deployed, this was a problem. If you verify your boot chain but allow root to modify that kernel, the benefits of the verified boot chain are significantly reduced. Even if root can't modify the on-disk kernel, root can just hot-patch the kernel and then make this persistent by dropping a binary that repeats the process on system boot.

Lockdown is intended as a mechanism to avoid that, by providing an optional policy that closes off interfaces that allow root to modify the kernel. This was the sole purpose of the original implementation, which maps to the "integrity" mode that's present in the current implementation. Kernels that boot in lockdown integrity mode prevent even root from using these interfaces, increasing assurances that the running kernel corresponds to the booted kernel. But lockdown's functionality has been extended since then. There are some use cases where preventing root from being able to modify the kernel isn't enough - the kernel may hold secret information that even root shouldn't be permitted to see (such as the EVM signing key that can be used to prevent offline file modification), and the integrity mode doesn't prevent that. This is where lockdown's confidentiality mode comes in. Confidentiality mode is a superset of integrity mode, with additional restrictions on root's ability to use features that would allow the inspection of any kernel memory that could contain secrets.

Unfortunately right now we don't have strong mechanisms for marking which bits of kernel memory contain secrets, so in order to achieve that we end up blocking access to all kernel memory. Unsurprisingly, this compromises people's ability to inspect the kernel for entirely legitimate reasons, such as using the various mechanisms that allow tracing and probing of the kernel.

How can we solve this? There's a few ways:
  1. Introduce a mechanism to tag memory containing secrets, and only restrict accesses to this. I've tried to do something similar for userland and it turns out to be hard, but this is probably the best long-term solution.
  2. Add support for privileged applications with an appropriate signature that implement policy on the userland side. This is actually possible already, though not straightforward. Lockdown is implemented in the LSM layer, which means the policy can be imposed using any other existing LSM. As an example, we could use SELinux to impose the confidentiality restrictions on most processes but permit processes with a specific SELinux context to use them, and then use EVM to ensure that any process running in that context has a legitimate signature. This is quite a few hoops for a general purpose distribution to jump through.
  3. Don't use confidentiality mode in general purpose distributions. The attacks it protects against are mostly against special-purpose use cases, and they can enable it themselves.

My recommendation is for (3), and I'd encourage general purpose distributions that enable lockdown to do so only in integrity mode rather than confidentiality mode. The cost of confidentiality mode is just too high compared to the benefits it provides. People who need confidentiality mode probably already know that they do, and should be in a position to enable it themselves and handle the consequences.

comment count unavailable comments

Debian Project Leader: Bits from the new DPL

My fellow Debianites, It's been one month, one week and one day since I decided to run for this DPL term. The Debian community has been through a variety of interesting times during the last decade, and instead of focusing on grand, sweeping changes for Debian, core to my DPL campaign was to establish a sense of normality and stability so that we can work on community building, continue to focus on technical excellence and serve our users the best we can. Thing don't always work out as we plan, and for many of us, Debian recently had to take a back seat to personal priorities. Back when I posted my intention to run, there were 125 260 confirmed cases of COVID-19 globally. Today, that number is 20 times higher, with the actual infected number likely to be significantly higher. A large number of us are under lock-down, where we not only fear the disease and its effect on local hospitals and how it will affect our loved ones, but also our very livelihoods and the future of our local businesses and industry. I don't mean to be gloomy with the statement above, I am after all, an optimist - but unfortunately it does get even worse. Governments and corporations around the world have started to take advantage of COVID-19 in negative ways and are making large sweeping changes that undermine the privacy and rights of individuals everywhere. For many reasons, including those above, I believe that the Debian project is more important and relevant now than it's ever been before. The world needs a free, general purpose operating system, unburdened by the needs of profit, which puts the needs of its users first, providing a safe and secure platform for the computing needs of the masses. While we can't control or fix all the problems in the world, we can control our response to it, and be part of the solutions that bring the change we want to see. During my term as DPL, I will be available to help with problems in our community to the maximum extent that my time permits. If we help ourselves, we will be in a better position to help others. If you (or your team) get stuck and are in need of help, then please do not hessitate to e-mail me. A few thank-yous As incoming DPL, I'd like to thank Sam Hartman on behalf of the project for the work that he's done over the last year as DPL. It's a tremendous time commitment that requires constant attention to detail. On Sunday, Sam and I had a handover meeting where we discussed various DPL responsibilities including finances, delegations (including specifics of some delegations), legal matters, outreach and other questions I had. I'd also like to thank Sam for taking the time to do this. Thank you to Sruthi Chadran and Brian Gupta who took the time to also run for DPL this year. Both candidates brought important issues to the forefront and I hope to work with both of them on those in the near future. DPL Blog Today, I've started a new blog for the Debian Project Leader to help facilitate more frequent communication, and to reach a wider audience via Planet Debian. This will contain supplemental information to what I send to the debian-devel-announce mailing list. Want to help? In my platform, I listed some key areas that I'd like to work on. My work won't be limited to those, but it should give you some idea of the type of DPL that I'll be. If you'd like to get involved, feel free to join the #debian-dpl channel on the oftc IRC network, and please introduce yourself along with any areas of interest that you'd like to contribute to.

Debian Project Leader: Bits from the new DPL

My fellow Debianites, It's been one month, one week and one day since I decided to run for this DPL term. The Debian community has been through a variety of interesting times during the last decade, and instead of focusing on grand, sweeping changes for Debian, core to my DPL campaign was to establish a sense of normality and stability so that we can work on community building, continue to focus on technical excellence and serve our users the best we can. Thing don't always work out as we plan, and for many of us, Debian recently had to take a back seat to personal priorities. Back when I posted my intention to run, there were 125 260 confirmed cases of COVID-19 globally. Today, that number is 20 times higher, with the actual infected number likely to be significantly higher. A large number of us are under lock-down, where we not only fear the disease and its effect on local hospitals and how it will affect our loved ones, but also our very livelihoods and the future of our local businesses and industry. I don't mean to be gloomy with the statement above, I am after all, an optimist - but unfortunately it does get even worse. Governments and corporations around the world have started to take advantage of COVID-19 in negative ways and are making large sweeping changes that undermine the privacy and rights of individuals everywhere. For many reasons, including those above, I believe that the Debian project is more important and relevant now than it's ever been before. The world needs a free, general purpose operating system, unburdened by the needs of profit, which puts the needs of its users first, providing a safe and secure platform for the computing needs of the masses. While we can't control or fix all the problems in the world, we can control our response to it, and be part of the solutions that bring the change we want to see. During my term as DPL, I will be available to help with problems in our community to the maximum extent that my time permits. If we help ourselves, we will be in a better position to help others. If you (or your team) get stuck and are in need of help, then please do not hessitate to e-mail me. A few thank-yous As incoming DPL, I'd like to thank Sam Hartman on behalf of the project for the work that he's done over the last year as DPL. It's a tremendous time commitment that requires constant attention to detail. On Sunday, Sam and I had a handover meeting where we discussed various DPL responsibilities including finances, delegations (including specifics of some delegations), legal matters, outreach and other questions I had. I'd also like to thank Sam for taking the time to do this. Thank you to Sruthi Chadran and Brian Gupta who took the time to also run for DPL this year. Both candidates brought important issues to the forefront and I hope to work with both of them on those in the near future. DPL Blog Today, I've started a new blog for the Debian Project Leader to help facilitate more frequent communication, and to reach a wider audience via Planet Debian. This will contain supplemental information to what I send to the debian-devel-announce mailing list. Want to help? In my platform, I listed some key areas that I'd like to work on. My work won't be limited to those, but it should give you some idea of the type of DPL that I'll be. If you'd like to get involved, feel free to join the #debian-dpl channel on the oftc IRC network, and please introduce yourself along with any areas of interest that you'd like to contribute to.

19 April 2020

Dirk Eddelbuettel: prrd 0.0.3: More improvements

Back in early 2018, the prrd package was introduced as release 0.0.1, uploaded to CRAN, and updated once as release 0.0.2. I have used it extensively for every CRAN release of Rcpp, RcppArmadillo, RcppEigen, BH, and possibly others. The idea of prrd is simple, and described in some more detail on its webpage and its GitHub repo. Reverse dependency checks are an important part of package development and is easily done in a (serial) loop. But these checks are also generally embarassingly parallel as there is no or little interdependency between them (besides maybe shared build depedencies). See the following (dated) screenshot (running six parallel workers, arranged in split byobu session). Since the two early releases, I made a few changes to e.g. the analysis of package checks and their summaries (but never pushed these) as well as a few usability enhancements. This release regroups these. The release is summarised in the NEWS entry:

Changes in prrd version 0.0.3 (2020-04-19)
  • Extended functionality summarise() with optional use of package foghorn (Dirk)
  • New scripts enqueueDepends, dequeueDepends and summmariseJobs (Dirk)
  • Added tweaks and enhancements throughout (Dirk)
  • The default queue location can now be set via options (Dirk)
  • Meta-data on package, version and date is now stored, and shown on dequeue startup (Dirk)
  • The README.md was clarified and improved (Dirk, Avraham Adler in #8, Robert Nuske in #10)

CRANberries provides the usual summary of changes to the previous version. See the aforementioned webpage and its repo for details. For more questions or comments use the issue tracker off the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Russ Allbery: Review: Beyond the Galaxy

Review: Beyond the Galaxy, by Ethan Siegel
Publisher: World Scientific
Copyright: 2016
ISBN: 981-4667-16-1
Format: Kindle
Pages: 370
In the preface of this book, Ethan Siegel recounts his experience as a new college professor in 2009, thrilled to learn that he would be teaching the college's introductory astronomy class. The only disappointment was that he could not find a textbook that described astronomy the way that he wanted to teach it. Siegel did not want to simply describe the current astronomical understanding. He wanted to start at the beginning and tell a story. What questions did we start with? How did we answer them? What new questions did those answers provoke? I enjoyed Sabine Hossenfelder's Lost in Math, but I'm even happier that she included in it a recommendation of Beyond the Galaxy as a good introduction to modern astronomy. I am completely in agreement with Siegel: Astronomy is interesting however it is described, but the best presentation for the interested lay person who does not intend to become a professional astronomy is chronological. (I feel the same way about physics and mathematics.) The narrative structure of scientific discovery draws me in every time. It's so satisfying to see our knowledge growing, breakthrough by breakthrough, only to meet the next puzzling inconsistency. It also is a natural fit for bringing someone up to speed on a field, since the history of the field starts with simpler questions and problems, and the complications of later discoveries build on the gaps left by earlier theories. Siegel tells that story well. Beyond the Galaxy starts with observations of seasonal changes in the path of the Sun through the sky and phases of the moon, quickly moves on to Eratosthenes's calculation of the circumference of the Earth, and continues at pace through early astronomy. The first chapter brings the reader up to the start of the 20th century, after which Siegel slows down and spends the remaining 10 chapters on the last 100 years: special and general relativity, the expansion of the universe, the Big Bang, dark matter, dark energy, the inflationary period, and more. For me, this was perfect. The story of early astronomy and the fight over the heliocentric model of the universe is worth telling, but it's been told many times before and features prominently in history books outside of science classes. The weight of material in this book matched my interest: What new developments have there been after general relativity and the Big Bang theory? What inconsistencies are dark matter and dark energy explaining? What is the inflationary period and how did it modify the Big Bang theory? (My half-formed impression of this part of astronomy from news articles turned out to be almost entirely wrong.) And how do we know all of this? The question of how we know as much as we do about the early origins of the universe turned out to be far more interesting than I had realized. It's one of those pieces of science where, when given only the conclusions, it's hard to imagine how our theories could be as complete as they are. Siegel is a bit repetitive in how he talks about theory development and occasionally skims over the details to avoid getting too deep into techniques and math, but he does a great job explaining the predictions of a theory, the design of experiments to prove or disprove those predictions, and the implications for the boundaries of what we know. This is the first time someone has explained the theorized origin of matter-antimatter asymmetry in sufficient detail that I was able to understand and internalize it. Dark matter and dark energy are the current topics of astronomy that get the most press, and the coverage of both is very good here. I was particularly fascinated by the link between dark matter and the superstructures of the universe that we've discovered. Astronomy does scale in a way that no other science can manage, and the scale of structure we've discovered and been able to map has gone up considerably since the last time I refreshed my astronomical knowledge. If you're looking for a math-light, comprehensive introduction to modern astronomy, I can wholeheartedly recommend this book. That goes double if you, like me, prefer a chronological view of scientific discovery so that you can understand what questions people were trying to answer and how our understanding changed over time. One note on format: Astronomy being what it is, this book is very heavy on pictures. It is not at all suitable for a dedicated ebook reader. The Kindle tries, but the combination of a small screen and black and white e-ink fail miserably. I switched to reading on a tablet, and that was a far better experience. The Kindle app still struggles to put the figures next to the text that discusses them, but that's a typesetting problem that I often have with physical books as well. If you don't have a tablet and you aren't willing to read on a computer screen, you probably want a physical copy of this one (which, sadly, means academic textbook pricing). Rating: 8 out of 10

17 April 2020

Julien Danjou: Being in Charge

Being in ChargeIf you never heard of the 10x engineer myth, it's a pretty great concept. It boils down to the idea where an engineer could be 10x more efficient than a random engineer. I find this fantastically twisted.Last week, I sat and chat with Alexis Monville in Le Podcast a podcast that equips you to make positive change in your organization. We talked about that 10x Engineer myth, and from there we digressed on how to grow your career and handle the different aspects of it.This was a very interesting exchange. Alexis is actually going to publish a new book next month (May 2020) entitled I am a Software Engineer and I am in charge.
Being in Charge
Lucky me, this week, I had the chance to be able to read the book before everybody else which means I actually read after our recording. I understood why Alexis said that a lot of what I was talking about during our podcast resonated with him. I send a detailed review of the book to Alexis and Michael if you're curious. I'm definitely recommending this book if you want to stop complaining about your job and start understanding how to pull the strings.I wish I had this book available 10 years ago!

14 April 2020

Emmanuel Kasper: Recommended keyboard settings for Productivity and Usability, for European Programmers

TLDR: setxkbmap -layout us -variant altgr-intl and become a happier programmer.
The case for QWERTY for European ProgrammersIf you re working on Unix / Linux, or C based programming languages, it can make sense to switch to the qwerty(us) keyboard layout. Why ?
Unix, C, Perl, Java, and most of programming languages have been conceived on QWERTY keyboards.
So when the designers choose special characters to use for the language synthax, they simply choose what was easy to access on their own keyboard. This has been historically documented for the vi editor.

To give an example, using an Unix shell you have to type the dot . and slash / symbols quite often to navigate the filesystem. The two keys producing these symbols, are nicely aligned on a QWERTY layout and do not require a key combination to be entered. So you can quickly enter something like ../.. using a single hand.
Now using a QWERTZ layout, like in Germany / Austria, you have the . symbol easily accessible, but you need to combine two keys ( Shift + 7 ) to get a / .
And if you are a poor soul using an AZERTY layout, to get the . and / symbol you need each time a key combo.
The need of key combos is bad not only for speed (multiple keys to lookup) but also for usability, as you have to stretch your fingers to reach the key if using a single hand, provoking repetitive strain injury. You might be smiling but this is commonly known amongst Emacs Users, due to the prominent use of commands using Ctrl and Alt combos, and led to the creation of an Emacs Ergonomic wiki.

This goes as well for many symbol commonly used in programming languages, think for instance about the semicolon ; for terminating a statement, the [ , and ] symbols for defining an array, and the backslash \ for escaping.
All these keys are accessible via a single keypress on qwerty and require key combos on qwertz and azerty. No wonder Linux, Minix and BSD were invented on non-azerty layouts: in France we were still busy typing the path to the source code, when in other parts of the world people already had the file open in their editor.
You don t need to throw away your existing keyboards when learning the qwerty (us) layout: for a couple of euros, you can find on ebay alternate keys stickers to put on your laptop.
Accessing keys with diacritics with the AltGr International variantNow if you want to switch to a qwerty keyboard layout, and you re French or German, you might wonder how to access the characters with diacritics, the and of French and and of German.
Fortunately there is a very clever keyboard variant for the us layout who uses the AltGr key, to make all these keys accessible, just hiding them behind the AltGr key.
Need ? that s just AltGr + e away. Needs ? That s just AltGr + , Needs ? AltGr +p
See the coolness of that ? You can type all international diacritics in an easy way, and there are even keyboard stickers for that too.

Now how to access to this layout of wonder ?
You can configure the layout and variant in Xorg, in debian/ubuntu that would be entering in /etc/default/keyboard
XKBLAYOUT="us"
XKBVARIANT="altgr-intl"
After restarting the X server, you can check that the settings have been applied with
setxkbmap -print -verbose 10
If using Gnome, you can also set the keyboard layout and variant by changing the schema org.gnome.desktop-inputsources, which will override the desktop-agnostic settings of /etc/default/keyboard.
For this you can either call
dconf write /org/gnome/desktop/input-sources/sources "[('xkb', 'us+altgr-intl')]"
or navigate with the gui tool dconf-settings to org.gnome.desktop-inputsources and set the value there.

If you want to further improve your keyboard layout, you can also have a look at swapping Ctrl and Alt, as described here (French language article) Happy hacking !

Dirk Eddelbuettel: RcppArmadillo 0.9.860.2.0

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 706 other packages on CRAN. A new upstream release 9.860.2 of Armadillo was just released. The theme of convergence continues; the previous release increased the minor from 800 to 850, now we are at 860. We first wrapped this up as version 0.9.859.1.0, but it turned out to have been held back by a buglet between R 4.0.0 and Rcpp which the recent patch release fixed (along with other woes on old R or non-CRAN-alike macOS). It then turns out that the new (upstream) version 9.860.1 had a minor bug which I missed as I reverse-depends checked the prior version. Doh. My thanks, as always, to CRAN for spotting this. The fix was added upstream and we have 9.860.2 as RcppArmadillo 0.9.860.2.0. Changes in the new release are noted below.

Changes in RcppArmadillo version 0.9.860.2.0 (2020-04-13)
  • Upgraded to Armadillo release 9.860.2 (Roasted Mocha Fix)
    • Added powmat()
    • Faster access to columns in sparse submatrix views
    • Faster handling of relational expressions by accu()
    • Faster handling of sympd matrices by expmat(), logmat(), sqrtmat()
    • Workaround for save/load issues with HDF5 v1.12
  • Vignettes are now pre-made and include (#285)
  • Two test files are now skipped on 32-bit Windows

Courtesy of CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dirk Eddelbuettel: gettz 0.0.4

A minor routine update 0.0.4 of gettz arrived on CRAN overnight. gettz provides a possible fallback in situations where Sys.timezone() fails to determine the system timezone. That happened when e.g. the file /etc/localtime somehow is not a link into the corresponding file with zoneinfo data in, say, /usr/share/zoneinfo. Since the package was written (in the fall of 2016), R added a similar extended heuristic approach itself. This release adds registration of the compiled routine via R_registerRoutines() and R_useDynamicSymbols(), adds .registration=TRUE to useDynLib() in NAMESPACE, and uses an unquoted symbol in .Call(). Two new badges were added to the README.md as well. And as in the previous release in 2016: No new code, or new features. Courtesy of CRANberries, there is a comparison to the previous release. More information is on the gettz page. For questions or comments use the issue tracker off the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

13 April 2020

Giovanni Mascellani: DKIM for Debian Developers

What is DKIM? DKIM (DomainKeys Identified Mail), as Wikipedia puts it, "is an email authentication method designed to detect forged sender addresses in emails (email spoofing), a technique often used in phishing and email spam". More prosaically, one of the reasons email spam is so abundant is that, given a certain email message, there is no simple way to know for certain who sent it and how reputable they are. So even if people having addresses @debian.org are very nice and well-behaving, any random spammer can easily send emails from whatever@debian.org, and even if you trust people from @debian.org you cannot easily configure your antispam filter to just accept all emails from @debian.org, because spammers would get in too. Since nearly ten years DKIM is there to help you. If you send an email from @debian.org with DKIM, it will have a header like this:
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=debian.org;
    s=vps.gio.user; t=1586779391;
    bh=B6tckJy2cynGjNRdm3lhFDrp0tD7fF8hS4x0FCfLADo=;
    h=From:Subject:To:Date:From;
    b=H4EDlATxVm7XNqPy2x7IqCchBUz1SxFtUSstB23BAsdyTKJIohM0O4RRWhrQX+pqE
     prPVhzcfNALMwlfExNE69940Q6pMCuYsoxNQjU7Jl/UX1q6PGqdVSO+mKv/aEI+N49
     vvYNgPJNLaAFnYqbWCPI8mNskLHLe2VFYjSjE4GJFOxl9o2Gpe9f5035FYPJ/hnqBF
     XPnZq7Osd9UtBrBq8agEooTCZHbNFSyiXdS0qp1ts7HAo/rfrBfbQSk39fOOQ5GbjV
     6FehkN4GAXFNoFnjfmjrVDJC6hvA8m0tJHbmZrNQS0ljG/SyffW4OTlzFzu4jOmDNi
     UHLnEgT07eucw==
The field d=debian.org is the domain this email claims to be from and the fields bh= and b= are a cryptographic public key signature certifying this fact. How do I check that the email is actually from @debian.org? I use the selector s=vps.gio.user to fetch the public key via DNS, and then use the public key to verify the signature.
$ host -t TXT vps.gio.user._domainkey.debian.org
vps.gio.user._domainkey.debian.org descriptive text "v=DKIM1; k=rsa; s=email; h=sha256; p=" "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsM/W/kxtKWT58Eak0cfm/ntvurfbkkvugrG2jfvSMnHHkFyfJ34Xvn/HhQPLwX1QsjhuLV+tW+BQtxY7jxSABCee6nHQRBrpDej1t86ubw3CSrxcg1mzJI5BbL8un0cwYoBtUvhCYAZKarv1W2otCGs43L0s" "GtEqqtmYN/hIVVm4FcqeYS1cYrZxDsjPzCEocpYBhqHh1MTeUEddVmPHKZswzvllaWF0mgIXrfDNAE0LiX39aFKWtgvflrYFKiL4hCDnBcP2Mr71TVblfDY0wEdAEbGEJqHR1SxvWyn0UU1ZL4vTcylB/KJuV2gMhznOjbnQ6cjAhr2JYpweTYzz3wIDAQAB"
There it is! Debian declares in its DNS record that that key is authorized to sign outbound email from @debian.org. The spammer hopefully does not have access to Debian's DKIM keys, and they cannot sign emails. Many large and small email services have already deployed DKIM since years, while most @debian.org emails still do not use it. Why not? Because people send @debian.org emails from many different servers. Basically, every DD used their @debian.org address sends email from their own mail server, and those mail servers (fortunately) do not have access to Debian's DNS record to install their DKIM keys. Well, that was true until yesterday! :-) A few weeks ago I poked DSA asking to allow any Debian Developer to install their DKIM keys, so that DDs could use DKIM to sign their emails and hopefully reduce the amount of spam sent from @debian.org. They have done it (thank you DSA very much, especially adsb), and now it is possible to use it! How do I configure it? I will not write here a full DKIM tutorial, there are many around. You have to use opendkim-genkey to generate a key and then configure your mail server to use opendkim to digitally sign outbound email. There are a few Debian-specific things you have to care about, though. First the have to choose a selector, which is a string used to distinguish many DKIM keys belonging to the same domain. Debian allows you to installa a key whose selector is <something>.<uid>.user, where <uid> is your Debian uid (this is done both for namespacing reasons and for exposing who might be abusing the system). So check carefully that your selector has this form. Then you cannot edit directly Debian's DNS record. But you can use the email-LDAP gateway on db.debian.org to install your key in a way similar to how entries in debian.net are handled (see the updated documentation). Specifically, suppose that opendkim-genkey generated the following thing for selector vps.gio.user and domain debian.org:
vps.gio.user._domainkey IN  TXT ( "v=DKIM1; h=sha256; k=rsa; "
      "p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsM/W/kxtKWT58Eak0cfm/ntvurfbkkvugrG2jfvSMnHHkFyfJ34Xvn/HhQPLwX1QsjhuLV+tW+BQtxY7jxSABCee6nHQRBrpDej1t86ubw3CSrxcg1mzJI5BbL8un0cwYoBtUvhCYAZKarv1W2otCGs43L0sGtEqqtmYN/hIVVm4FcqeYS1cYrZxDsjPzCEocpYBhqHh1MTeUE"
      "ddVmPHKZswzvllaWF0mgIXrfDNAE0LiX39aFKWtgvflrYFKiL4hCDnBcP2Mr71TVblfDY0wEdAEbGEJqHR1SxvWyn0UU1ZL4vTcylB/KJuV2gMhznOjbnQ6cjAhr2JYpweTYzz3wIDAQAB" )  ; ----- DKIM key vps.gio.user for debian.org
Then you have to carefully copy the content of the p= field (without being fooled by it being split between different strings) and construct a request of the form:
dkimPubKey: vps.gio.user MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsM/W/kxtKWT58Eak0cfm/ntvurfbkkvugrG2jfvSMnHHkFyfJ34Xvn/HhQPLwX1QsjhuLV+tW+BQtxY7jxSABCee6nHQRBrpDej1t86ubw3CSrxcg1mzJI5BbL8un0cwYoBtUvhCYAZKarv1W2otCGs43L0sGtEqqtmYN/hIVVm4FcqeYS1cYrZxDsjPzCEocpYBhqHh1MTeUEddVmPHKZswzvllaWF0mgIXrfDNAE0LiX39aFKWtgvflrYFKiL4hCDnBcP2Mr71TVblfDY0wEdAEbGEJqHR1SxvWyn0UU1ZL4vTcylB/KJuV2gMhznOjbnQ6cjAhr2JYpweTYzz3wIDAQAB
and then send it GPG-signed to changes@db.debian.org:
echo 'dkimPubKey: vps.gio.user blahblahblah'   gpg --clearsign   mail changes@db.debian.org
Then use host -t TXT vps.gio.user._domainkey.debian.org to chech the key gets published (it will probably take some minutes/hours, I don't know). Once it is published, you can enable DKIM in you mail server and your email will be signed. Congratulations, you will not look like a spammer any more! You can send an email to check-auth@verifier.port25.com to check that your setup is correct. They will reply with a report, including the success of DKIM test. Notice that currently Debian's setup only allows you to use RSA DKIM keys and doesn't allow you to set other DKIM fields (but you probably won't need to set them). EDIT DSA made an official announcement about DKIM support, which you might want to check out as well, together with its links. EDIT 2 Now ed25519 keys are supported, the syntax for specifying keys on LDAP is a little bit more flexible and you can also insert CNAME records. See the official documentation for the updated details. So we have solved our problems with spam? Ha, no! DKIM is only a small step. Useful, also because it enable other steps to be taken in the future, but small. In particular, DKIM enables you to say: "This particular email actually comes from @debian.org", but doesn't tell anybody what to do with emails that are not signed. A third-party mail server might wonder whether @debian.org emails are actually supposed to be signed or not. There is another standard for dealing with that, which is called DMARD, and I believe that Debian should eventually use it, but not now: the problem is that currently virtually no email from @debian.org is signed with DKIM, so if DMARC was enabled other mail servers would start to nuke all @debian.org emails, except those which are already signed, a minority. If people and services sending emails from @debian.org will start configuring DKIM on their servers, which is now possible, it will eventually come a time when DMARC can be enabled, and spammers will find themselves unable to send forged @debian.org emails. We are not there yet, but todays we are a little step closer than yesterday. Also, notice that having DKIM on @debian.org only counters spam pretending to be from @debian.org, but there is much more. The policy on what to accept is mostly independent on that on what you send. However, knowing that @debian.org emails have DKIM and DMARC would mean that we can set our spam filters to be more aggressive in general, but whitelist official Debian Developers and services. And the same can be done for other domains using DKIM and DMARC. Finally, notice that some incompatibilities between DKIM and mailing lists are known, and do not have a definitive answer yet. Basically, most mailing list engines modify either the body of the headers in forwarded emails, which means that DKIM does not validate any more. There are many proposed solutions, possibly none completely satisfying, but since spam is not very satisfying as well, something will have to be worked out. I wrote a lot already, though, so I wont't discuss this here.

Matthew Garrett: Implementing support for advanced DPTF policy in Linux

Intel's Dynamic Platform and Thermal Framework (DPTF) is a feature that's becoming increasingly common on highly portable Intel-based devices. The adaptive policy it implements is based around the idea that thermal management of a system is becoming increasingly complicated - the appropriate set of cooling constraints to place on a system may differ based on a whole bunch of criteria (eg, if a tablet is being held vertically rather than lying on a table, it's probably going to be able to dissipate heat more effectively, so you should impose different constraints). One way of providing these criteria to the OS is to embed them in the system firmware, allowing an OS-level agent to read that and then incorporate OS-level knowledge into a final policy decision.

Unfortunately, while Intel have released some amount of support for DPTF on Linux, they haven't included support for the adaptive policy. And even more annoyingly, many modern laptops run in a heavily conservative thermal state if the OS doesn't support the adaptive policy, meaning that the CPU throttles down extremely quickly and the laptop runs excessively slowly.

It's been a while since I really got stuck into a laptop reverse engineering project, and I don't have much else to do right now, so I've been working on this. It's been a combination of examining what source Intel have released, reverse engineering the Windows code and staring hard at hex dumps until they made some sort of sense. Here's where I am.

There's two main components to the adaptive policy - the adaptive conditions table (APCT) and the adaptive actions table (APAT). The adaptive conditions table contains a set of condition sets, with up to 10 conditions in each condition set. A condition is something like "is the battery above a certain charge", "is this temperature sensor below a certain value", "is the lid open or closed", "is the machine upright or horizontal" and so on. Each condition set is evaluated in turn - if all the conditions evaluate to true, the condition set's target is implemented. If not, we move onto the next condition set. There will typically be a fallback condition set to catch the case where none of the other condition sets evaluate to true.

The action table contains sets of actions associated with a specific target. Once we've picked a target by evaluating the conditions, we execute the actions that have a corresponding target. Actions are things like "Set the CPU power limit to this value" or "Load a passive policy table". Passive policy tables are simply tables associating sensors with devices and an associated temperature limit. If the limit is exceeded, the associated device should be asked to reduce its heat output until the situation is resolved.

There's a couple of twists. The first is the OEM conditions. These are conditions that refer to values that are exposed by the firmware and are otherwise entirely opaque - the firmware knows what these mean, but we don't, so conditions that rely on these values are magical. They could be temperature, they could be power consumption, they could be SKU variations. We just don't know. The other is that older versions of the APCT table didn't include a reference to a device - ie, if you specified a condition based on a temperature, you had no way to express which temperature sensor to use. So, instead, you specified a condition that's greater than 0x10000, which tells the agent to look at the APPC table to extract the device and the appropriate actual condition.

Intel already have a Linux app called Thermal Daemon that implements a subset of this - you're supposed to run the binary-only dptfxtract against your firmware to parse a few bits of the DPTF tables, and it writes out an XML file that Thermal Daemon makes use of. Unfortunately it doesn't handle most of the more interesting bits of the adaptive performance policy, so I've spent the past couple of days extending it to do so and to remove the proprietary dependency.

My current work is here - it requires a couple of kernel patches (that are in the patches directory), and it only supports a very small subset of the possible conditions. It's also entirely possible that it'll do something inappropriate and cause your computer to melt - none of this is publicly documented, I don't have access to the spec and you're relying on my best guesses in a lot of places. But it seems to behave roughly as expected on the one test machine I have here, so time to get some wider testing?

comment count unavailable comments

12 April 2020

Dirk Eddelbuettel: #25: Test, test, test, those R 4.0.0 binaries with Ubuntu 20.04 and Rocker

Welcome to the 25nd post in the randomly recurring R recitations series, or R4 for short. Just yesterday, we posted a short post along with a video and supporting slides. It covered how to test the soon-to-be-released R 4.0.0 on a custom Ubuntu 18.04 Rocker container. A container for Ubuntu 20.04, which is itself in final beta stages, was being built while the video was made. As it is available now, we created a quick follow-up video showing the use under Ubuntu 20.04:

The updated supporting slides from the video are still at this link. What we showed in both videos does of course also work directly on Ubuntu (or Debian, using those source repos) installations; the commands shown in the Rocker use case generally apply equally to a normal installation. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dirk Eddelbuettel: #24: Test, test, test, those R 4.0.0 binaries with Ubuntu and Rocker

Welcome to the 24nd post in the relentlessly regular R ravings series, or R4 for short. R 4.0.0 will be released in less than two weeks, and testing is very important. I had uploaded two alpha release builds (at the end of March and a good week ago) as well as a first beta release yesterday, all to the Debian experimental distribution (as you can see here) tracking the release schedule set by Peter Dalgaard. Because R 4.0.0 will require reinstallation of all packages, it makes some sense to use a spare machine. Or a Docker container. So to support that latter mode, I have now complemented the binaries created from the r-base source package with all base and recommended packages, providing a starting point for actually running simple tests. Which is what we do in the video, using again the R on Ubuntu (18.04) Rocker container:

Slides from the video are at this link. This container based on 18.04 is described here on the Docker Hub; a new 20.04 container with the pre-release of the next Ubuntu LTS should be there shortly once it leaves the build queue. What we showed does of course also work on direct Ubuntu (or Debian, using those source repos) installations; the commands shown in the Rocker use case generally apply equally to a normal installation. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

10 April 2020

Dirk Eddelbuettel: Rcpp 1.0.4.6: Bug fix interim version

rcpp logo Rcpp 1.0.4 was released on March 17, following the usual sequence of fairly involved reverse-depends check along with a call for community testing issued weeks before the release. In that email I specifically pleaded with folks to pretty-please test non-standard setups:
It would be particularly beneficial if those with unsual build dependencies tested it as we would increase overall coverage beyond what I get from testing against 1800+ CRAN packages. BioConductor would also be welcome.
Alas, you can t always get what you want. Shortly after the release we were made aware that the two (large) pull request at the book ends of the 1.0.3 to 1.0.4 release period created trouble. Of these two, the earliest PR in the 1.0.4 release upset older-than-CRAN-tested installation, i.e. R 3.3.0 or before. (Why you d want to run R 3.3.* when R 3.6.3 is current is something I will never understand, but so be it.) This got addressed in two new PRs. And the matching last PR had a bit of sloppyness leaving just about everyone alone, but not all those macbook-wearing data scientists when using newer macOS SDKs not used by CRAN. In other words, unsual setups. But boy, do those folks have an ability to complain. Again, two quick PRs later that was addressed. Along came a minor PR with two more Rcpp::Shield<> uses (as life is too short to manually count PROTECT and UNPROTECT). And then a real issue between R 4.0.0 and Rcpp first noticed with RcppParallel builds on Windows but then also affecting RcppArmadillo. Another quickly issued fix. So by now the count is up to six, and we arrived at Rcpp 1.0.4.6. Which is now on CRAN, after having sat there for nearly a full week, and of course with no reason given. Because the powers that be move in mysterious ways. And don t answer to earthlings like us. As may transpire here, I am little tired from all this. I think we can do better, and I think we damn well should, or I may as well throw in the towel and just release to the drat repo where each of the six interim versions was available for all to take as soon as it materialized. Anyway, here is the state of things. Rcpp has become the most popular way of enhancing R with C or C++ code. As of today, 1897 packages on CRAN depend on Rcpp for making analytical code go faster and further, along with 191 in BioConductor. And per the (partial) logs of CRAN downloads, we are running steasy at one millions downloads per month. The changes for this interim version are summarized below.

Changes in Rcpp patch release version 1.0.4.6 (2020-04-02)
  • Changes in Rcpp API:
    • The exception handler code in #1043 was updated to ensure proper include behavior (Kevin in #1047 fixing #1046).
    • A missing Rcpp_list6 definition was added to support R 3.3.* builds (Davis Vaughan in #1049 fixing #1048).
    • Missing Rcpp_list 2,3,4,5 definition were added to the Rcpp namespace (Dirk in #1054 fixing #1053).
    • A further updated corrected the header include and provided a missing else branch (Mattias Ellert in #1055).
    • Two more assignments are protect with Rcpp::Shield (Dirk in #1059)
  • Changes in Rcpp Attributes:
    • Empty strings are not passed to R CMD SHLIB which was seen with R 4.0.0 on Windows (Kevin in #1062 fixing #1061).
  • Changes in Rcpp Deployment:
    • Travis CI unit tests now run a matrix over the versions of R also tested at CRAN (rel/dev/oldrel/oldoldrel), and coverage runs in parallel for a net speed-up (Dirk in #1056 and #1057).

Thanks to CRANberries, you can also look at a diff to the previous release. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. Bugs reports are welcome at the GitHub issue tracker as well (where one can also search among open or closed issues); questions are also welcome under rcpp tag at StackOverflow which also allows searching among the (currently) 2356 previous questions. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

9 April 2020

Antoine Beaupr : Mumble dreams

With everyone switching to remote tools for social distancing, I've been using Mumble more and more. That's partly by choice -- I don't like videoconferencing much, frankly -- and partly by necessity: sometimes my web browser fails and Mumble is generally more reliable. Some friend on a mailing list recently asked "shouldn't we make Mumble better?" and opened the door for me to go on a long "can I get a pony?" email. Because I doubt anyone on that mailing list has the time or capacity to actually fix those issues, I figured I would copy this to a broader audience in the hope that someone else would pick it up.

Why Mumble rocks Before I go on with the UI critique, I should show why care: Mumble is awesome. When you do manage to configure it correctly, Mumble just works; it's highly reliable. It uses little CPU, both on the client and the server side, and can have rooms with tens if not hundreds of participants. The server can be easily installed and configured: there's a Debian package and resource requirements are minimal. It's basically network-bound. There are at least three server implementations, the official one called Murmur, the minimalist umurmur and Grumble, a Go rewrite. It has great quality: echo canceling, when correctly configured, is solid and latency is minimal. It has "overlays" so you can use it while gaming or demo'ing in full screen while still having an idea of who's talking. It also supports positional audio for gaming that integrates with popular games like Counterstrike or Half-Life. It's moderately secure: it doesn't support end-to-end encryption, but client/server communication is encrypted with TLS. It supports a server password and some moderation mechanisms.

UI improvements Mumble should be smarter about a bunch of things. Having all those settings is nice for geeky control freaks, but it makes the configuration absolutely unusable for most people. Hide most settings by default, and make better defaults. Specifically, those should be on by default:
  • RNNoise
  • echo cancellation (the proper "monitor" channels)
  • pre-configured shortcut for PTT (Push To Talk) -- right-shift is my favorite
  • "double-PTT" to hold it enabled
  • be more silent by default (I understand why it would want to do voice synthesis, but it would need to be much better at it before it's default)
The echo test should be more accessible, one or two clicks away from the main UI. I have only found out about that feature when someone told me where to find it. This basically means to take it out of the settings page and into its own dialog. The basic UI should be much simpler. It could look something like Jitsi: just one giant mute button with a list of speakers. Basically:
  1. Take that status bar and make it use the entire space of the main window
  2. Push the chat and room list dialog to separate, optional dialog (e.g. the room list could be a popup on login, but we don't need to continuously see the damn thing)
  3. Show the name of the person talking in the main UI, along with other speakers (Big Blue Button does this well: just a label that fades away with time after a person talks)
Some features could be better explained. For example, the "overlay" feature makes no sense at all for most users. It only makes sense when you're a gamer and use Mumble alongside another full-screen program, to show you who's talking. Improved authentication. The current authentication systems in Mumble are somewhat limited: the server can have a shared password to get access to it, and from there it's pretty much free-for-all. There are client certificates but those are hard to understand and the most common usage scenario is that someone manages to configure it once, forgets about it and then cannot login again with the same username. It should be easier to get the audio right. Now, to be fair, this is hard to do in any setup, and Mumble is only a part of this. There are way too many moving parts in Linux for this to be easy: between your hardware, ALSA drivers, Pulseaudio mixers and Mumble, too many things can go wrong. So this is a general problem when doing multimedia in general, and the Linux ecosystem in particular, but Mumble is especially hard to configure in there. Improved speaker stats. When you right-click on a user in Mumble, you get detailed stats about the user: packet loss, latency, bandwidth, codecs... It's pretty neat. But that is hard to parse for a user. Jitsi, in contrast, shows a neat little "bar graph" (similar to what you get on a cell phone) with a color code to show network conditions for that user. Then you can drill down to show more information. Having that info for the user would be really useful to figure out which user is causing that echo or latency. Heck, while I'm dreaming, we could do the same thing Jitsi and tell the user when we detect too much noise on their side and suggest muting! There's probably more UI issues, but at that point you have basically rebuilt the entire user interface. This problem is hard to fix because UX people are unlikely to have the skills required to hack at an (old) Qt app, and C++ hackers are unlikely to have the best UX skills...

Missing features Video. It has been on the roadmap since 2011, so I'm not holding my breath. It is, obviously, the key feature missing from the software when compared to other conferencing tools and it's nice to see they are considering it. Screensharing and whiteboarding would also be a nice addition. Unfortunately, all that is a huge undertaking and it's unlikely to happen in the short term. And even if it does, it's possible hard-core Mumble users would be really upset at the change... A good web app -- a major blocker to the adoption of Mumble is the need for that complex app. If users could join just with a web browser, adoption would be much easier. There is a web app called mumble-web out there, but it seems to work only for listening as there are numerous problems with recording: quality issues, audio glitches, voice activation, voice activation.. The CCC seems to be using that app to stream talk translation, so that part supposedly works correctly. Dial-in -- allow plain old telephones to call into conferences. There seems to be a program called mumsi that can do this, but it's unmaintained and it's unclear if any of the forks work at all. Update: according to samba, mumsi works, but sometimes freezes and needs to be restarted. Each SIP account shows up as a bot that comes up when someone calls the number. It supports multiple callers, although apparently mumsi crashes after a while with 4 callers. A comment here also mentioned there's a fork that mentions using a "pin" as well for dialing in.

Caveats Now the above will probably not happen soon. Unfortunately, Mumble has had trouble with their release process recently. It took them a long time to even agree on releasing 1.3, and when they did agree, it took them a long time again to actually do the release. There has been much more activity on the Mumble client and web app recently, so hopefully I will be proven wrong. The 1.3.1 release actually came out recently(correction:) is actually being worked on, which is encouraging. All in all, mumble has some deeply ingrained UI limitations. it's built like an app from the 1990, all the way down to the menu system and "status bar" buttons. It's definitely not intuitive for a new user and while there's an audio wizard that can help you get started, it doesn't always work and can be confusing in itself. I understand that I'm just this guy saying "please make this for me ktxbye". I'm not writing this as a critic of Mumble: I love the little guy, the underdog. Mumble has been around forever and it kicks ass. I'm writing this in a spirit of solidarity, in the hope the feedback can be useful and to provide useful guidelines on how things could be improved. I wish I had the time to do this myself and actually help the project beyond just writing, but unfortunately the reality is I'm a poor UI designer and I have little time to contribute to more software projects. So hopefully someone could take those ideas and make Mumble even greater. And if not, we'll just have to live with it. Thanks to all the Mumble developers who, over all those years, managed to make and maintain such an awesome product. You rock!

Update: a chat with Mumble folks Update: it seems the idea of simplifying the Mumble interface will take some time to sink in. After presenting this article in the #mumble Freenode IRC channel, it became obvious that having a more usable interface is not a priority. To put it in the words of a participant in the channel:
If someone is too stupid to use a piece of software than they should probably use something else or have someone capable of critical thinking set it up for them. -- meep
So I guess "I'm with stupid": I do not believe we should make software only for "smart" people because that leads to overly complicated and over-engineered user interfaces that are needlessly hard to use and configure. Mumble is actually an excellent example of such programs: powerful software, but too hard to use to actually reach critical mass. Thankfully, meep is not a Mumble developer and does not actually represent the official position of the Mumble project. A core developer (not representing the project either) was more nuanced:
The UI being not up to date is a matter of taste. The kind of UI you are describing is the kind of UI I certainly don't want. More gernally though I agree that the UI needs some rework and especially: Flexibility so that each user can (ideally) make it appear as (s)he wants. Before this can happen though, we'll have to rewrite a fair bit of code as we'll have to separate backend and frontend from one another. -- krzmbrzl
So this will take time. The fact that Qt seems to be moving back to closed-source does not help the situation with Mumble's GUI development, unfortunately. But the other hand, maybe it will provide the necessary "kick" to write a new, simpler GUI.

Next.

Previous.