Search Results: "ag"

18 December 2022

Ian Jackson: Rust needs #[throws]

tl;dr: Ok-wrapping as needed in today s Rust is a significant distraction, because there are multiple ways to do it. They are all slightly awkward in different ways, so are least-bad in different situations. You must choose a way for every fallible function, and sometimes change a function from one pattern to another. Rust really needs #[throws] as a first-class language feature. Code using #[throws] is simpler and clearer. Please try out withoutboats s fehler. I think you will like it. Contents

A recent personal experience in coding style Ever since I read withoutboats s 2020 article about fehler, I have been using it in most of my personal projects. For Reasons I recently had a go at eliminating the dependency on fehler from Hippotat. So, I made a branch, deleted the dependency and imports, and started on the whack-a-mole with the compiler errors. After about a half hour of this, I was starting to feel queasy. After an hour I had decided that basically everything I was doing was making the code worse. And, bizarrely, I kept having to make individual decisons about what idiom to use in each place. I couldn t face it any more. After sleeping on the question I decided that Hippotat would be in Debian with fehler, or not at all. Happily the Debian Rust Team generously helped me out, so the answer is that fehler is now in Debian, so it s fine. For me this experience, of trying to convert Rust-with-#[throws] to Rust-without-#[throws] brought the Ok wrapping problem into sharp focus. What is Ok wrapping? Intro to Rust error handling (You can skip this section if you re already a seasoned Rust programer.) In Rust, fallibility is represented by functions that return Result<SuccessValue, Error>: this is a generic type, representing either whatever SuccessValue is (in the Ok variant of the data-bearing enum) or some Error (in the Err variant). For example, std::fs::read_to_string, which takes a filename and returns the contents of the named file, returns Result<String, std::io::Error>. This is a nice and typesafe formulation of, and generalisation of, the traditional C practice, where a function indicates in its return value whether it succeeded, and errors are indicated with an error code. Result is part of the standard library and there are convenient facilities for checking for errors, extracting successful results, and so on. In particular, Rust has the postfix ? operator, which, when applied to a Result, does one of two things: if the Result was Ok, it yields the inner successful value; if the Result was Err, it returns early from the current function, returning an Err in turn to the caller. This means you can write things like this:

    let input_data = std::fs::read_to_string(input_file)?;

and the error handling is pretty automatic. You get a compiler warning, or a type error, if you forget the ?, so you can t accidentally ignore errors. But, there is a downside. When you are returning a successful outcome from your function, you must convert it into a Result. After all, your fallible function has return type Result<SuccessValue, Error>, which is a different type to SuccessValue. So, for example, inside std::fs::read_to_string, we see this:

        let mut string = String::new();
        file.read_to_string(&mut string)?;
        Ok(string)

string has type String; fs::read_to_string must return Result<String, ..>, so at the end of the function we must return Ok(string). This applies to return statements, too: if you want an early successful return from a fallible function, you must write return Ok(whatever). This is particularly annoying for functions that don t actually return a nontrivial value. Normally, when you write a function that doesn t return a value you don t write the return type. The compiler interprets this as syntactic sugar for -> (), ie, that the function returns (), the empty tuple, used in Rust as a dummy value in these kind of situations. A block ( ... ) whose last statement ends in a ; has type (). So, when you fall off the end of a function, the return value is (), without you having to write it. So you simply leave out the stuff in your program about the return value, and your function doesn t have one (i.e. it returns ()). But, a function which either fails with an error, or completes successfuly without returning anything, has return type Result<(), Error>. At the end of such a function, you must explicitly provide the success value. After all, if you just fall off the end of a block, it means the block has value (), which is not of type Result<(), Error>. So the fallible function must end with Ok(()), as we see in the example for std::fs::read_to_string. A minor inconvenience, or a significant distraction? I think the need for Ok-wrapping on all success paths from fallible functions is generally regarded as just a minor inconvenience. Certainly the experienced Rust programmer gets very used to it. However, while trying to remove fehler s #[throws] from Hippotat, I noticed something that is evident in codebases using vanilla Rust (without fehler) but which goes un-remarked. There are multiple ways to write the Ok-wrapping, and the different ways are appropriate in different situations. See the following examples, all taken from a real codebase. (And it s not just me: I do all of these in different places, - when I don t have fehler available - but all these examples are from code written by others.) Idioms for Ok-wrapping - a bestiary Wrap just a returned variable binding If you have the return value in a variable, you can write Ok(reval) at the end of the function, instead of retval.

    pub fn take_until(&mut self, term: u8) -> Result<&'a [u8]>  
        // several lines of code
        Ok(result)

If the returned value is not already bound to variable, making a function fallible might mean choosing to bind it to a variable. Wrap a nontrivial return expression Even if it s not just a variable, you can wrap the expression which computes the returned value. This is often done if the returned value is a struct literal:

    fn take_from(r: &mut Reader<'_>) -> Result<Self>  
        // several lines of code
        Ok(AuthChallenge   challenge, methods  )

Introduce Ok(()) at the end For functions returning Result<()>, you can write Ok(()). This is usual, but not ubiquitous, since sometimes you can omit it. Wrap the whole body If you don t have the return value in a variable, you can wrap the whole body of the function in Ok( ). Whether this is a good idea depends on how big and complex the body is.

    fn from_str(s: &str) -> std::result::Result<Self, Self::Err>  
        Ok(match s  
            "Authority" => RelayFlags::AUTHORITY,
            // many other branches
            _ => RelayFlags::empty(),
         )

Omit the wrap when calling fallible sub-functions If your function wraps another function call of the same return and error type, you don t need to write the Ok at all. Instead, you can simply call the function and not apply ?. You can do this even if your function selects between a number of different sub-functions to call:

    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result  
        if flags::unsafe_logging_enabled()  
            std::fmt::Display::fmt(&self.0, f)
          else  
            self.0.display_redacted(f)

But this doesn t work if the returned error type isn t the same, but needs the autoconversion implied by the ? operator. Convert a fallible sub-function error with Ok( ... ?) If the final thing a function does is chain to another fallible function, but with a different error type, the error must be converted somehow. This can be done with ?.

     fn try_from(v: i32) -> Result<Self, Error>  
         Ok(Percentage::new(v.try_into()?))

Convert a fallible sub-function error with .map_err Or, rarely, people solve the same problem by converting explicitly with .map_err:

     pub fn create_unbootstrapped(self) -> Result<TorClient<R>>  
         // several lines of code
         TorClient::create_inner(
             // several parameters
         )
         .map_err(ErrorDetail::into)

What is to be done, then? The fehler library is in excellent taste and has the answer. With fehler:

Whether a function is fallible, and what it s error type is, is specified in one place. It is not entangled with the main return value type, nor with the success return paths.
So the success paths out of a function are not specially marked with error handling boilerplate. The end of function return value, and the expression after return, are automatically wrapped up in Ok. So the body of a fallible function is just like the body of an infallible one, except for places where error handling is actually involved.
Error returns occur through ? error chaining, and with a new explicit syntax for error return.
We usually talk about the error we are possibly returning, and avoid talking about Result unless we need to.

fehler provides:

An attribute macro #[throws(ErrorType)] to make a function fallible in this way.
A macro throws!(error) for explicitly failing.

This is precisely correct. It is very ergonomic. Consequences include:

One does not need to decide where to put the Ok-wrapping, since it s automatic rather than explicitly written out.
Specifically, what idiom to adopt in the body (for example write!(...)?; vs write!(...) in a formatter) does not depend on whether the error needs converting, how complex the body is, and whether the final expression in the function is itself fallible.
Making an infallible function fallible involves only adding #[throws] to its definition, and ? to its call sites. One does not need to edit the body, or the return type.
Changing the error returned by a function to a suitably compatible different error type does not involve changing the function body.
There is no need for a local Result alias shadowing std::result::Result, which means that when one needs to speak of Result explciitly, the code is clearer.

Limitations of fehler But, fehler is a Rust procedural macro, so it cannot get everything right. Sadly there are some wrinkles.

You can t write #[throws] on a closure.
Sometimes you can get quite poor error messages if you have a sufficiently broken function body.
Code inside a macro call isn t properly visible to fehler so sometimes return statements inside macro calls are untreated. This will lead to a type error, so isn t a correctness hazard, but it can be nuisance if you like other syntax extensions eg if_chain.
#[must_use] #[throws(Error)] fn obtain() -> Thing; ought to mean that Thing must be used, not the Result<Thing, Error>.

But, Rust-with-#[throws] is so much nicer a language than Rust-with-mandatory-Ok-wrapping, that these are minor inconveniences. Please can we have #[throws] in the Rust language This ought to be part of the language, not a macro library. In the compiler, it would be possible to get the all the corner cases right. It would make the feature available to everyone, and it would quickly become idiomatic Rust throughout the community. It is evident from reading writings from the time, particularly those from withoutboats, that there were significant objections to automatic Ok-wrapping. It seems to have become quite political, and some folks burned out on the topic. Perhaps, now, a couple of years later, we can revisit this area and solve this problem in the language itself ? Explicitness An argument I have seen made against automatic Ok-wrapping, and, in general, against any kind of useful language affordance, is that it makes things less explicit. But this argument is fundamentally wrong for Ok-wrapping. Explicitness is not an unalloyed good. We humans have only limited attention. We need to focus that attention where it is actually needed. So explicitness is good in situtions where what is going on is unusual; or would otherwise be hard to read; or is tricky or error-prone. Generally: explicitness is good for things where we need to direct humans attention. But Ok-wrapping is ubiquitous in fallible Rust code. The compiler mechanisms and type systems almost completely defend against mistakes. All but the most novice programmer knows what s going on, and the very novice programmer doesn t need to. Rust s error handling arrangments are designed specifically so that we can avoid worrying about fallibility unless necessary except for the Ok-wrapping. Explicitness about Ok-wrapping directs our attention away from whatever other things the code is doing: it is a distraction. So, explicitness about Ok-wrapping is a bad thing. Appendix - examples showning code with Ok wrapping is worse than code using #[throws] Observe these diffs, from my abandoned attempt to remove the fehler dependency from Hippotat. I have a type alias AE for the usual error type (AE stands for anyhow::Error). In the non-#[throws] code, I end up with a type alias AR<T> for Result<T, AE>, which I think is more opaque but at least that avoids typing out -> Result< , AE> a thousand times. Some people like to have a local Result alias, but that means that the standard Result has to be referred to as StdResult or std::result::Result.

With `fehler` and `#[throws]`	Vanilla Rust, `Result<>`, mandatory `Ok-wrapping`

Return value clearer, error return less wordy:
`impl Parseable for Secret`	`impl Parseable for Secret`
`#[throws(AE)]`
`fn parse(s: Option<&str>) -> Self`	`fn parse(s: Option<&str>) -> AR<Self>`
`let s = s.value()?;`	`let s = s.value()?;`
`if s.is_empty() throw!(anyhow!( secret value cannot be empty ))`	`if s.is_empty() return Err(anyhow!( secret value cannot be empty ))`
`Secret(s.into())`	`Ok(Secret(s.into()))`


No need to wrap whole match statement in `Ok( ):`
`#[throws(AE)]`
`pub fn client<T>(&self, key: & static str, skl: SKL) -> T`	`pub fn client<T>(&self, key: & static str, skl: SKL) -> AR<T>`
`where T: Parseable + Default`	`where T: Parseable + Default`
`match self.end`	`Ok(match self.end`
`LinkEnd::Client => self.ordinary(key, skl)?,`	`LinkEnd::Client => self.ordinary(key, skl)?,`
`LinkEnd::Server => default(),`	`LinkEnd::Server => default(),`
	`)`

Return value and `Ok(())` entirely replaced by `#[throws]`:
`impl Display for Loc`	`impl Display for Loc`
`#[throws(fmt::Error)]`
`fn fmt(&self, f: &mut fmt::Formatter)`	`fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result`
`write!(f, :? : , &self.file, self.lno)?;`	`write!(f, :? : , &self.file, self.lno)?;`
`if let Some(s) = &self.section`	`if let Some(s) = &self.section`
`write!(f, )?;`	`write!(f, )?;`


	`Ok(())`

Call to `write!` now looks the same as in more complex case shown above:
`impl Debug for Secret`	`impl Debug for Secret`
`#[throws(fmt::Error)]`
`fn fmt(&self, f: &mut fmt::Formatter)`	`fn fmt(&self, f: &mut fmt::Formatter)-> fmt::Result`
`write!(f, "Secret(***)")?;`	`write!(f, "Secret(***)")`


Much tiresome `return Ok()` noise removed:
`impl FromStr for SectionName`	`impl FromStr for SectionName`
`type Err = AE;`	`type Err = AE;`
`#[throws(AE)]`
`fn from_str(s: &str) -> Self`	`fn from_str(s: &str) ->AR< Self>`
`match s`	`match s`
`COMMON => return SN::Common,`	`COMMON => return Ok(SN::Common),`
`LIMIT => return SN::GlobalLimit,`	`LIMIT => return Ok(SN::GlobalLimit),`
`_ =>`	`_ =>`
`;`	`;`
`if let Ok(n@ ServerName(_)) = s.parse() return SN::Server(n)`	`if let Ok(n@ ServerName(_)) = s.parse() return Ok(SN::Server(n))`
`if let Ok(n@ ClientName(_)) = s.parse() return SN::Client(n)`	`if let Ok(n@ ClientName(_)) = s.parse() return Ok(SN::Client(n))`

`if client == LIMIT return SN::ServerLimit(server)`	`if client == LIMIT return Ok(SN::ServerLimit(server))`
`let client = client.parse().context( client name in link section name )?;`	`let client = client.parse().context( client name in link section name )?;`
`SN::Link(LinkName server, client )`	`Ok(SN::Link(LinkName server, client ))`

edited 2022-12-18 19:58 UTC to improve, and 2022-12-18 23:28 to fix, formatting

comments

Bastian Venthur: The State of Python Packaging in 2022

Every year or so, I revisit the current best practices for Python packaging. This was my summary for 2021 here s the update for 2022. PyPA PyPA is still the place to go for information, best practices and tutorials for packaging Python projects. My only criticism from last year, namely that PyPA was heavily biased towards their own tooling (e.g. pipenv), has been addressed: the tool recommendations section lists now several tools for the same purpose with their own ones not necessarily being the first anymore. setup.py, setup.cfg, requirements.txt, Pipfile, pyproject.toml oh my! This is the reason why I m revisiting the documentation every year, to see what s the current way to go. Good progress has been made since last year: Bye setup.py and setup.cfg hello pyproject.toml pyproject.toml finally got mature enough to replace setup.py and setup.cfg in most cases. Recent versions of setuptools and pip now fully support pyproject.toml and even PyPA s packaging tutorial completely switched their example project from away setup.py towards pyproject.toml, making it an official recommendation. So, now you can replace your setup.py with pyproject.toml. If you had already some kind of declarative configuration in setup.cfg you can move that as well into pyproject.toml. Most tools, like pypy or pytest also support configuration in pyproject.toml (flake8 being a notable exception ) so there s no reason to keep setup.cfg around anymore. Actually, if you migrate to pyproject.toml it is best to do it properly and remove setup.py and setup.cfg as setuptools behaves a bit buggy when building a package that has either of them and the pyproject.toml. requirements.txt requirements.txt are still needed if you develop a deployable application (vs. a library) and want to provide pinned dependencies, i.e. with the specific versions that you ve tested your application with. Usually, the list of requirements in requirements.txt is the same as defined in pyproject.toml, but with pinned versions. Pipfile + Pipfile.lock I still do completely ignore Pipfile and Pipfile.lock as they are only supported by pipenv and not backed by any standard. Summary The major change this year was the proper support of pyproject.toml. I am slowly replacing all setup.py and setup.cfg in my projects with pyproject.toml and haven t discovered any issues yet. Even packaging those packages as Debian packages is well-supported by Debian s tooling. I m still running a quite boring stack based on pyproject.toml and requirements.txt, ignoring more advanced tools like poetry or such for dependency management. My build-system defined in pyproject.toml requires setuptools and I m using build for building and twine for uploading. Since PyPA changed the packaging tutorial towards pyproject.toml and away from setup.py, I think we will slowly see setup.py and setup.cfg go away over the years. Speaking of PyPA, I m happy that they changed their attitude towards a more unbiased recommendation of tooling.

Russell Coker: Wall Facers

I m currently reading the second book of the TriSolar Sci-Fi series by Cixin Liu, I ve only just started it so this post can t have spoilers for it and I will also only have minimal spoilers for the first book (nothing more than you will get from pop culture references to it). In the second book there are people called Wall Facers who have broad powers to shape the course of the Human response to an alien invasion in 400+ years time. The idea is that as the aliens have an ability to see everything that can be seen on Earth any ideas that leave the brain of one person can be snooped on, so if some people act independently without communicating their plans they can take the aliens by surprise. While that is probably going to work out well in the books history in general seems to show that people who act independently without any useful feedback from others tend to perform poorly, every king and dictator seems to demonstrate this. Efficient Work I ve been thinking about what I would do if I had significant powers to guide the response to an alien threat in some hundreds of years. The first thing to do would be to get all people working as efficiently as possible. Without the imminent threat of alien invasion we can have debates about how much time to spend working vs leisure time. Should we make 24 hours per week the new normal work week? But if the threat of annihilation is looming then the discussion should be about how to get as many people as possible working as much as possible. Currently 1/4 of the world population lack access to safe drinking water [1], there s a plan to achieve universal and equitable access to safe and affordable drinking water for all by 2030 . But 2030 isn t soon enough, another 8 years where 1/4 of children born won t reach their potential due to poor water is unacceptable. Currently 13% of the world population don t have access to electricity and 40% don t have access to clean fuels for cooking [2]. Lack of energy access reduces health and opportunities for education. Healthcare is another major obstacle to human development and therefore economic development. Even some allegedly first-world countries like the US lack universal affordable healthcare. I think we could reasonably get safe water to 99% of the world population before 2025 if we tried hard (IE applied a small fraction of the resources of a single war to it). Getting electricity to 95% of the world population and clean cooking fuels to 90% of the world population are probably achievable goals for 2025 as well. Healthcare is a slightly harder problem as we need to train more nurses and doctors. A registered nurse apparently needs 3 years of training after completing high school. We may have to improve high schools to get more students up to the standard of nursing degrees. If it takes 3 years to improve schools in year 9+ and then 3 years to get more high school graduates that would mean that it would take about 9 years to get an increase in nurses. Doing this would require increasing the capacity of universities and making university almost free (as it was for decades). So in about 2031 we could start sending a significant number of nurses from developed countries to help out developing countries. Becoming a doctor apparently requires 8 years of study plus a minimum of 3 years residency . So if doctors were entirely trained in first world countries then we wouldn t be able to send many doctors to developing countries until 2039. If the residency was performed in other countries then it could be as early as 2036. According to the WHO currently only half the world s population have adequate healthcare [3]. To get adequate healthcare to the world we need to more than double the number of doctors because currently we don t have enough in countries with decent healthcare systems such as Australia. It would probably take to at least 2060 to get enough doctors trained. The end goal of course would be to have every country able to train enough of it s citizens to provide all medical services, but countries that have serious widespread healthcare problems that reduce the number of people who can pursue higher education will have difficulty in that until some of the healthcare problems are alleviated. Education Obviously education is important to all achievements. Currently education seems very poorly run, it is possible to create a school system that teaches children effectively without the bullying that is common in Australia and without the sort of pressure that South Korea is infamous for. One of the main issues to resolve with the school system is the idea that everyone should learn at the same speed, that goal can only be achieved by making the majority of the students learn slowly. Students should be able to freely skip ahead as their skill permits and finish school at any age. Also high school isn t for everyone, the tech schools that teach trades need to be brought back. Deceiving Aliens A plot point in the TriSolar series is that the aliens can see each other s thoughts, the local communication (their equivalent to talking) is based on reading each other s thoughts without the possibility of deception. While deceptive written communication is potentially possible for them they haven t developed skills in that area. As a first step towards exploiting this humans could focus more on linguistic development that increases language complexity, such as the way the English language adopts words from other languages and gives them slightly different meanings for example the difference between driver and chauffeur and the difference between dog and hound is not obvious to many Europeans who otherwise speak English fluently. When involved in conversation it s possible to convey meaning without directly stating things, this is used extensively by people who are interested in security. My observations of this are based on conversations with people who do government work, but I imagine that criminal organisations also do similar things for similar reasons. An increased focus on poetry in schools might be helpful in developing skills for conveying ideas to people who think in human ways where the message is unclear to non-humans who have no experience of deception. I wonder whether the ability to understand human poetry would make aliens less hostile to humans, if they can think like us then they would be less likely to want to exterminate us. Poker is a game that depends on the ability to deceive others, I ve never been any good at it. I wonder if making it part of the school curriculum would help improve the overall human ability to deceive aliens. I don t think that such schools would become dens of sociopathy as depicted in Kakegurui, but it might have some negative results. Spreading education to a larger portion of the world s population requires more use of electronic education. Anything learned via text can be more easily assimilated by aliens than things that are learned directly from other people. For high school and the basics of a university degree this is fine. But for more advanced education it seems that having a large face to face component might help keep the value away from the aliens. More Ideas? What do you think I missed on this list? I wasn t trying to list every possibility, just the more important ones. Also for any goals other than increasing inequality for it s own sake we should improve health and education for the world.

Freexian Collaborators: Monthly report about Debian Long Term Support, November 2022 (by Anton Gladky)

Like each month, have a look at the work funded by Freexian s Debian LTS offering.

Debian LTS contributors In November, 15 contributors have been paid to work on Debian LTS, their reports are available:

Abhijith PA did 0.0h (out of 14.0h assigned), thus carrying over 14.0h to the next month.

Anton Gladky did 6.0h (out of 15.0h assigned), thus carrying over 9.0h to the next month.

Ben Hutchings did 9.0h (out of 24.0h assigned), thus carrying over 15.0h to the next month.

Chris Lamb did 18.0h (out of 18.0h assigned).

Dominik George did 10.0h (out of 0h assigned and 24.0h from previous period), thus carrying over 14.0h to the next month.

Emilio Pozuelo Monfort did 0.0h (out of 38.0h assigned and 19.5h from previous period), thus carrying over 57.5h to the next month.

Enrico Zini did 0.0h (out of 0h assigned and 8.0h from previous period), thus carrying over 8.0h to the next month.

Helmut Grohne did 17.5h (out of 20.0h assigned).

Markus Koschany did 40.0h (out of 40.0h assigned).

Ola Lundqvist did 7.5h (out of 11.0h assigned and 5.0h from previous period), thus carrying over 8.5h to the next month.

Roberto C. S nchez did 20.25h (out of 0.75h assigned and 31.25h from previous period), thus carrying over 11.75h to the next month.

Stefano Rivera did 2.5h (out of 0h assigned and 17.0h from previous period), thus carrying over 14.5h to the next month.

Sylvain Beucler did 35.5h (out of 23.0h assigned and 34.5h from previous period), thus carrying over 22.0h to the next month.

Thorsten Alteholz did 14.0h (out of 14.0h assigned).

Utkarsh Gupta did 41.0h (out of 32.5h assigned and 25.0h from previous period), thus carrying over 16.5h to the next month.

Evolution of the situation In November, we released 43 DLAs, fixing 183 CVEs. We currently have 63 packages in dla-needed.txt that are waiting for updates, which is 19 fewer than the previous month. We re excited to announce that two Debian Developers Tobias Frost and Guilhem Moulin, have completed the on-boarding process and will begin contributing to LTS as of December 2022. Welcome aboard!

Thanks to our sponsors Sponsors that joined recently are in bold.

Platinum sponsors:

TOSHIBA (for 87 months)

GitHub (for 78 months)

Civil Infrastructure Platform (CIP) (for 55 months)

Gold sponsors:

Roche Diagnostics International AG (for 98 months)

Linode (for 92 months)

Babiel GmbH (for 81 months)

Plat Home (for 81 months)

University of Oxford (for 37 months)

Deveryware (for 24 months)

VyOS Inc (for 19 months)

EDF SA (for 8 months)

Silver sponsors:

Domeneshop AS (for 102 months)

Nantes M tropole (for 96 months)

Univention GmbH (for 88 months)

Universit Jean Monnet de St Etienne (for 88 months)

Ribbon Communications, Inc. (for 82 months)

Exonet B.V. (for 72 months)

Leibniz Rechenzentrum (for 66 months)

CINECA (for 55 months)

Minist re de l Europe et des Affaires trang res (for 49 months)

Cloudways Ltd (for 39 months)

Dinahosting SL (for 37 months)

Bauer Xcel Media Deutschland KG (for 31 months)

Platform.sh (for 31 months)

MOXA INC. (for 25 months)

sipgate GmbH (for 22 months)

OVH US LLC (for 20 months)

Tilburg University (for 20 months)

GSI Helmholtzzentrum f r Schwerionenforschung GmbH (for 12 months)

Soliton Systems K.K. (for 9 months)

Bronze sponsors:

Evolix (for 103 months)

Seznam.cz, a.s. (for 103 months)

Intevation GmbH (for 100 months)

Linuxhotel GmbH (for 100 months)

Daevel SARL (for 98 months)

Bitfolk LTD (for 97 months)

Megaspace Internet Services GmbH (for 97 months)

Greenbone Networks GmbH (for 96 months)

NUMLOG (for 96 months)

WinGo AG (for 96 months)

Ecole Centrale de Nantes - LHEEA (for 92 months)

Entr ouvert (for 87 months)

Adfinis AG (for 84 months)

GNI MEDIA (for 79 months)

Laboratoire LEGI - UMR 5519 / CNRS (for 79 months)

Tesorion (for 79 months)

Bearstech (for 70 months)

LiHAS (for 70 months)

Catalyst IT Ltd (for 65 months)

Supagro (for 60 months)

Demarcq SAS (for 59 months)

Universit Grenoble Alpes (for 45 months)

TouchWeb SAS (for 37 months)

SPiN AG (for 34 months)

CoreFiling (for 29 months)

Institut des sciences cognitives Marc Jeannerod (for 24 months)

Observatoire des Sciences de l Univers de Grenoble (for 21 months)

Tem Innovations GmbH (for 16 months)

WordFinder.pro (for 15 months)

CNRS DT INSU R sif (for 14 months)

Alter Way (for 7 months)

16 December 2022

Charles Plessy: Bad words in Debian.

A discussion on the debian-project mailing list caught my attention to an Italian word meaning something like would you be so kind to please go somewhere else? , but in a more direct and vulgar manner. I then used http://codesearch.debian.net to study its usage more in detail. I found it in:

the source code of XEmacs;
a list of bad words to filter conversations in BZflag;
the random sentence generator PolyGen;
the source code of the board game Tagua;
a database of offensive fortunes for the educative platform WIMS;
the crossword game parolottero;
a database of offensive fortunes for login screens and email signatures;
a list of (too) frequent passwords;
a source code comment aimed at somebody called Wolf;
a collection of rude gestures in the xwrists package.

That was a refreshing and pleasant recreation in the Debian package universe.

Simon Josefsson: Guix 1.4 on NV41PZ

On the shortlist of things to try on my new laptop has been Guix. I have been using Guix on my rsnapshot-based backup server since 2018, and experimented using it on a second laptop but never on my primary daily work machine. The main difference with Guix for me, compared to Debian (or Trisquel), is that Guix follows a rolling release model, even though they prepare stable versioned installation images once in a while. It seems the trend for operating system software releases is to either following a Long-Term-Support approach or adopt a rolling approach. Historically I have found that the rolling release approach, such as following Debian testing, has lead to unreliable systems, since little focus was given to system integration stability. This probably changed in the last 10 years or so, and today add-on systems like Homebrew on macOS gives me access to modern releases of free software easily. While I am likely to stay with LTS releases of GNU/Linux on many systems, the experience with rolling Guix (with unattended-upgrades from a cron job to pull in new code continously) on my backup servers has been smooth: no need for re-installation or debugging of installations for over four years! I tried the Guix 1.4 rc2 installation image on top of my previous Trisquel 11 installation; following the guided Guix installation menus was simple. I installed using wired network, since the WiFi dongle I had did not automatically become available. I put the Guix system on a separate partition, that I left empty when I installed Trisquel, and mounted the same /home that I used for Trisquel. Everything booted fine, and while I had some issues doing guix pull followed by guix system reconfigure /etc/config.scm I eventually got it working by using --allow-downgrade once. I believe this was a symptom of using a release candidate installation image. Guix did not auto-detect Trisquel or set up a Grub boot menu for it, and I have been unable to come up with the right Guix bootloader magic to add a Trisquel boot item again. Fortunately, the EFI boot choser allows me to boot Trisquel again. Guix 1.4 uses Linux-libre 6.0 which is newer than Trisquel 11 s Linux-libre 5.15. The WiFi dongle worked automatically once the system was installed. I will continue to tweak the default system configuration that was generated, it seems a standard GNOME installation does not include Evolution on Guix. Everything else I have tested works fine, including closing the lid and suspend and then resume, however the builtin webcam has a distorted image which does not happen on Trisquel. All in all, it seems the resulting system would be usable enough for me. I will be switching between Trisquel and Guix, but expect to spend most of time for daily work within Trisquel because it gives me the stable Debian-like environment that I ve been used to for ~20 years. Sharing the same /home between Trisquel and Guix may have been a mistake: GNOME handles this badly, and the dock will only contain the lowest-common-denominator of available applications, with the rest removed permanently.

15 December 2022

Reproducible Builds: Supporter spotlight: David A. Wheeler on supply chain security

The Reproducible Builds project relies on several projects, supporters and sponsors for financial support, but they are also valued as ambassadors who spread the word about our project and the work that we do. This is the sixth instalment in a series featuring the projects, companies and individuals who support the Reproducible Builds project. We started this series by featuring the Civil Infrastructure Platform project and followed this up with a post about the Ford Foundation as well as a recent ones about ARDC, the Google Open Source Security Team (GOSST), Jan Nieuwenhuizen on Bootstrappable Builds, GNU Mes and GNU Guix and Hans-Christoph Steiner of the F-Droid project. Today, however, we will be talking with David A. Wheeler, the Director of Open Source Supply Chain Security at the Linux Foundation.

Holger Levsen: Welcome, David, thanks for taking the time to talk with us today. First, could you briefly tell me about yourself? David: Sure! I m David A. Wheeler and I work for the Linux Foundation as the Director of Open Source Supply Chain Security. That just means that my job is to help open source software projects improve their security, including its development, build, distribution, and incorporation in larger works, all the way out to its eventual use by end-users. In my copious free time I also teach at George Mason University (GMU); in particular, I teach a graduate course on how to design and implement secure software. My background is technical. I have a Bachelor s in Electronics Engineering, a Master s in Computer Science and a PhD in Information Technology. My PhD dissertation is connected to reproducible builds. My PhD dissertation was on countering the Trusting Trust attack, an attack that subverts fundamental build system tools such as compilers. The attack was discovered by Karger & Schell in the 1970s, and later demonstrated & popularized by Ken Thompson. In my dissertation on trusting trust I showed that a process called Diverse Double-Compiling (DDC) could detect trusting trust attacks. That process is a specialized kind of reproducible build specifically designed to detect trusting trust style attacks. In addition, countering the trusting trust attack primarily becomes more important only when reproducible builds become more common. Reproducible builds enable detection of build-time subversions. Most attackers wouldn t bother with a trusting trust attack if they could just directly use a build-time subversion of the software they actually want to subvert.
Holger: Thanks for taking the time to introduce yourself to us. What do you think are the biggest challenges today in computing? There are many big challenges in computing today. For example:

Lack of resilience & capacity in chip fabrication. Fabs are extraordinarily expensive, and at the high end continue to have technological advancement. As a result, supply is failing to meet demand, and geopolitical issues raise further concerns. We ve seen cars, gaming consoles and many other devices unable to be delivered due to chip shortages. More fabs are being built, and some politicians are raising concerns, but it s unclear that current efforts will be enough.
Lack of enough developers able to develop the software that people & organizations need. Computers are far faster, and open source software has made software reuse incredibly easy. However, organizations still struggle to automate many tasks. The bottleneck is the lack of enough talented developers able to convert ideas into working software. Low-code and no-code approaches help in specialized areas, just like all previous automate the programmer efforts of the last 60 years, but there s no reason to believe they will help enough.
Large scale of software. Small systems are easier to develop & maintain, but today s systems increasingly get bigger to meet users needs & are much harder to manage. Even small embedded systems are often supported by huge back-end systems.
Ending tail of Moore s law & rise of smartphones. Historically people would just wait a few years for their software to speed up, but Moore s law is petering out, and smartphones are necessarily limited by power & size limits. As a result, software developers can t wait for the hardware to save their slow systems; they must redesign. Switching to faster languages, or using multiple processors, is much more difficult than waiting for performance problems to disappear.
Continuous change in interfaces. Developers continuously find reasons to change component interfaces: perhaps they re too inflexible, too hard to use, and so on. But now that developers are reusing hundreds, thousands, or tens of thousands of components, managing the continuous change of the reused components is challenging. Package managers make updating easy but don t automatically handle interface changes. I think this is mostly a self-inflicted problem most components could support old interfaces (like the Linux kernel does) but because it s often not acknowledged as a problem, it s often not addressed.
Security & privacy. Decades ago there were fewer computers and most computers weren t connected to a network. Today things are different. Criminals have found many ways to attack computer systems to make money, and nation-states have found many ways to attack computer systems for their own reasons. Attackers now have very strong motivations to perform attacks. Yet many developers aren t told how to develop software that resists attacks, nor how to protect their supply chains. Operations try to monitor and recover from attacks, but their job is difficult due to inadequately secure software that doesn t support those monitoring & recovery efforts well either. The results are terrible security.

Holger: Do you think reproducible builds are an important part in secure computing today already? David: Yes, but first let s put things in context. Today, when attackers exploit software vulnerabilities, they re primarily exploiting unintentional vulnerabilities that were created by the software developers. There are a lot of efforts to counter this:

Train & education developers in how to develop secure software. The OpenSSF provides a free course on how to do that (full disclosure: I m the author). Take that course or something like it!
Add tools to your CI pipeline to detect potential vulnerabilities. Yes, they have false positives and false negatives, so you have to also use your brain but that just means you need to be smart about using tools, instead of not using them.
Get projects & organizations to update the components they use, since often the vulnerabilities are well-known publicly (e.g., Equifax in 2017). Add some tools to your development process to warn you about components with known vulnerabilities! GitHub & GitLab both provide tools to do this, and there are many other tools.
When starting new projects, try to use memory-safe languages. On average 70% of the vulnerabilities in Chrome and in Microsoft are from memory safety problems; using a memory-safe language eliminates most of them.

We re just starting to get better at this, which is good. However, attackers always try to attack the easiest target. As our deployed software has started to be hardened against attack, attackers have dramatically increased their attacks on the software supply chain (Sonatype found in 2022 that there s been a 742% increase year-over-year). The software supply chain hasn t historically gotten much attention, making it the easy target. There are simple supply chain attacks with simple solutions:

In almost every year the top attack has been typosquatting. In typo squatting, an attacker creates packages with almost the right name. This is an easy attack to counter developers just need to double-check the name of a package before adding it. But we aren t warning developers enough about it! For more information, see papers such as the Backstabber s Knife Collection.
Last year the top software supply chain attack was dependency confusion convincing projects to use the wrong repo for a given package. There are simple solutions to this, such as specifying the package source and/or requiring a cryptographic hash to match.
Some attacks involve takeovers of developer accounts. In almost all cases, these are caused by stolen passwords. Using a multi-factor authentication (MFA) token eliminates stolen password attacks, which is why several repositories are starting to require MFA tokens in some cases.

Unfortunately, attackers know there are other lines of attack. One of the most dangerous is subverted build systems, as demonstrated by the subversion of SolarWinds Orion system. In a subverted build system, developers can review the software source code all day and see no problem, because there is no problem there. Instead, the process to convert source code into the code people run, called the build system , is subverted by an attacker. One solution for countering subverted build systems is to make the build systems harder to attack. That s a good thing to do, but you can never be confident that it was good enough . How can you be sure it s not subverted, if there s no way to know? A stronger defense against subverted build systems is the idea of verified reproducible builds. A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts. A build is verified if multiple different parties verify that they get the same result for that situation. When you have a verified reproducible build, either all the parties colluded (and you could always double-check it yourself), or the build process isn t subverted. There is one last turtle: What if the build system tools or machines are subverted themselves? This is not a common attack today, but it s important to know if we can address them when the time comes. The good news is that we can address this. For some situations reproducible builds can also counter such attacks. If there s a loop (that is, a compiler is used to generate itself), that s called the trusting trust attack, and that is more challenging. Thankfully, the trusting trust attack has been known about for decades and there are known solutions. The diverse double-compiling (DDC) process that I explained in my PhD dissertation, as well as the bootstrappable builds process, can both counter trusting trust attacks in the software space. So there is no reason to lose hope: there is a bottom turtle , as it were.
Holger: Thankfully, this has all slowly started to change and supply chain issues are now widely discussed, as evident by efforts like Securing the Software Supply Chain: Recommended Practices Guide for Developers which you shared on our mailing list. In there, Reproducible Builds are mentioned as recommended advanced practice, which is both pretty cool (we ve come a long way!), but to me it also sounds like this will take another decade until it s become standard normal procedure. Do you agree on that timeline? David: I don t think there will be any particular timeframe. Different projects and ecosystems will move at different speeds. I wouldn t be surprised if it took a decade or so for them to become relatively common there are good reasons for that. Today the most common kinds of attacks based on software vulnerabilities still involve unintentional vulnerabilities in operational systems. Attackers are starting to apply supply chain attacks, but the top such attacks today are typosquatting (creating packages with similar names) and dependency confusion) (convincing projects to download packages from the wrong repositories). Reproducible builds don t counter those kinds of attacks, they counter subverted builds. It s important to eventually have verified reproducible builds, but understandably other issues are currently getting prioritized first. That said, reproducible builds are important long term. Many people are working on countering unintentional vulnerabilities and the most common kinds of supply chain attacks. As these other threats are countered, attackers will increasingly target build systems. Attackers always go for the weakest link. We will eventually need verified reproducible builds in many situations, and it ll take a while to get build systems able to widely perform reproducible builds, so we need to start that work now. That s true for anything where you know you ll need it but it will take a long time to get ready you need to start now.
Holger: What are your suggestions to accelerate adoption? David: Reproducible builds need to be:

Easy (ideally automatic). Tools need to be modified so that reproducible builds are the default or at least easier to do.
Transparent to projects & potential users. Many projects have no idea that their results aren t reproducible, and many potential users of the project don t know either. That information needs to be obvious. I ve proposed that the OpenSSF Dashboard SIG try to reproduce builds, for at least some packages, to make it more obvious to everyone when a project isn t reproducible. I don t know if that will happen in that particular case, but the point is to help people learn that information as soon as possible.
Deployed. Experiments are great, but experiments showing that a project could be reproducible are inadequate. We need the projects that people use to be reproducible.

I think there s a snowball effect. Once many projects packages are reproducible, it will be easier to convince other projects to make their packages reproducible. I also think there should be some prioritization. If a package is in wide use (e.g., part of minimum set of packages for a widely-used Linux distribution or framework), its reproducibility should be a special focus. If a package is vital for supporting some societally important critical infrastructure (e.g., running dams), it should also be considered important. You can then work on the ones that are less important over time.
Holger: How is the Best Practices Badge going? How many projects are participating and how many are missing? David: It s going very well. You can see some automatically-generated statistics, showing we have over 5,000 projects, adding more than 1/day on average. We have more than 900 projects that have earned at least the passing badge level.
Holger: How many of the projects participating in the Best Practices badge engaging with reproducible builds? David: As of this writing there are 168 projects that report meeting the reproducible builds criterion. That s a relatively small percentage of projects. However, note that this criterion (labelled build_reproducible) is only required for the gold badge. It s not required for the passing or silver level badge. Currently we ve been strategically focused on getting projects to at least earn a passing badge, and less on earning silver or gold badges. We would love for all projects to get earn a silver or gold badge, of course, but our theory is that projects that can t even earn a passing badge present the most risk to their users. That said, there are some projects we especially want to see implementing higher badge levels. Those include projects that are very widely used, so that vulnerabilities in them can impact many systems. Examples of such projects include the Linux kernel and curl. In addition, some projects are used within systems where it s important to society that they not have serious security vulnerabilities. Examples include projects used by chemical manufacturers, financial systems and weapons. We definitely encourage any of those kinds of projects to earn higher badge levels.
Holger: Many thanks for this interview, David, and for all of your work at the Linux Foundation and elsewhere!

For more information about the Reproducible Builds project, please see our website at reproducible-builds.org. If you are interested in ensuring the ongoing security of the software that underpins our civilisation and wish to sponsor the Reproducible Builds project, please reach out to the project by emailing contact@reproducible-builds.org.

Russell Coker: Pixel 6A

I have just bought a Pixel 6A [1] for my wife. It s one of the latest Google phones that was released almost at the same time as the Pixel 7 series, so if you want to spend a lot of money on a phone that s the latest and greatest then the Pixel 7 and Pixel 7 Pro are the options, but if you want to save some money and don t need something really high end then the Pixel 6A is a good option. The one I bought cost $550 when I bought it from Google which seemed like a good deal when it was advertised as being discounted from $750. Later I discovered that other retailers were selling it for $500 or for $550 when bundled with a Chromecast. Also one of the other retailers was a company that I could get discount gift cards for. So this is the type of item you should really shop around for. It doesn t come with a charger so you don t have the gray-market disadvantage of getting yet another charger that doesn t fit the sockets in your country. The main new feature of this phone is a fingerprint scanner built in to the screen. I don t think this is a good thing, sure it s neat technology to both display pictures and read fingerprints through the same piece of glass but there are benefits to using a different location. Huawei phones with a fingerprint reader at the rear of the phone allow the user to answer calls, drag down the notifications list, and scroll sideways via touching the fingerprint reader. So the Pixel 6A clearly gives less functionality than some Huawei phones in this regard. One major annoyance with the phone is a combination of the phone hardware and Android 13 which gives no back arrow (swipe sideways) and no button to get the task list (swipe upwards at the correct speed from the bottom of the screen). The Android 12 way of swiping up from the bottom of the screen to get buttons for back, task-list, and home-screen is much better IMHO. Hopefully they will release a software update to make this a configuration option. Generally this is a nice phone, but the lack of buttons for back and task-list is annoying. Maybe the lack of buttons is something you can get used to after using it for a while. But millions of people just taking whatever companies like Google offer isn t what I imagined when I first hoped for a large portion of the world using Linux. On PCs we have a choice between KDE, GNOME, and other UIs. It would be nice if we had similar choices on phones.

[1] https://en.wikipedia.org/wiki/Pixel_6a

Abhijith PA: Running PostmarketOS on my phone

Couple of weeks back I installed PostmarketOS on my idle phone Leeco Le 1s , which was paper weight for some time now. It all started with a roadtrip to Pondicherry (I will soon write about this trip). As I was sitting on the front seat where Praveen s Librem 5 kept charing on the car dashboard. And we had a small discussion about PostmarketOS and how much new ports are available now. My idle phone came to my mind. After reaching home I started setting up porting pmOS to this device. Going through pmOS website, to my surprise there is already a port for this device. The OEM unlock is quite easy even though a little hiccup at begin (I suspect it is solely of my cable). The Xiaomi users knows the pain of unlocking bootloader. With the pmOS community preferred practice of using pmbootstrap, I built image for my device, flashed it. And phone boot stopped with pmOS logo. I thought I went to bootloop. I tried sxmo, xfce4, everything same nothing happening after boot logo splash. The pmOS troubleshooting wiki is quite good. They have documented most issues. Though screen is stuck I can still ssh to the phone. From wiki I came to know its a screen refresh problem. I installed msm-fb-refresher package and ran it as a daemon. Voila, I have a mate desktop. Mate

Mate desktop is not at all touch friendly even though I tried to scaled up to read and tap things on the screen. Wiki suggest xfce4 is little more touch friendly. I started moving to xfce. Then again same problem, stuck at boot loop. This time it was with lightdm, I turned off CanGraphical issue warning and now I have xfce desktop. pmos xfce seems great comparing with mate. Nothing works as of now from a Mobile phone point of view. The phone maintainer says the battery is working, but I couldn t get it working. Its always in battery mode and 50% status. I thought the hardware buttons will never work. But with xev, I can see the hardware key event triggering.

Dirk Eddelbuettel: spdl 0.0.3 on CRAN: Adding File Logger

A second update to the still-new package spdl is now om CRAN, and in Debian. The key focus of spdl is a offering the same interface from both R and C++ for logging by relying on spdlog via my RcppSpdlog package. This release add support for a simple filesetup() initialiser to direct logging output to a file. For now the console logger and the file logger are exclusive, if there is interest we could borrow a page from upstream and combine them. The short NEWS entry follows.

Changes in spld version 0.0.3 (2022-12-14)

Add filesetup method

Courtesy of my CRANberries, there is also a diffstat report. More detailed information is on the spdl page. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

14 December 2022

Russ Allbery: Review: Contact

Review: Contact, by Carl Sagan

Publisher:	Pocket Books
Copyright:	1985
Printing:	October 1986
ISBN:	0-671-43422-5
Format:	Mass market
Pages:	434

Contact is a standalone first-contact science fiction novel. Carl Sagan (1934 1996) was best known as a non-fiction writer, astronomer, and narrator of the PBS non-fiction program Cosmos. This is his first and only novel. Ellie Arroway is the director of Project Argus, a radio telescope array in the New Mexico desert whose primary mission is SETI: the search for extra-terrestrial intelligence by scanning the skies for unexpected radio signals. Its assignment to SETI is controversial; there are radio astronomy projects waiting, and although 25% of the telescope time is assigned to non-SETI projects, some astronomers think the SETI mission should be scrapped and Argus fully diverted to more useful research. That changes overnight when Argus picks up a signal from Vega, binary pulses representing the sequence of prime numbers. The signal of course doesn't stop when the Earth rotates, so Ellie and her team quickly notify every radio observatory they can get hold of to follow the signal as it passes out of their line of sight. Before long, nearly every country with a radio telescope is involved, and Russian help is particularly vital since they have ship-mounted equipment. The US military and intelligence establishment isn't happy about this and make a few attempts to shove the genie back into the bottle and keep any further discoveries secret, without a lot of success. (Sagan did not anticipate the end of the Cold War, and yet ironically relations with the Russians in his version of the 1990s are warmer by far than they are today. Not that this makes the military types any happier.) For better or worse, making sense of the alien signal becomes a global project. You may be familiar with this book through its 1997 movie adaptation starring Jodie Foster. What I didn't know before reading this book is that it started life as a movie treatment, co-written with Ann Druyan, in 1979. When the movie stalled, Sagan expanded it into a novel. (Given the thanks to Druyan in the author's note, it may not be far wrong to name her as a co-author.) If you've seen the movie, you will have a good idea of what will happen, but the book gives the project a more realistic international scope. Ellie has colleagues carefully selected from all over the world, including for the climactic moment of the story. The biggest problem with Contact as a novel is that Sagan is a non-fiction writer who didn't really know how to write a novel. The long, detailed descriptions of the science and the astronomical equipment fit a certain type of SF story, but the descriptions of the characters, even Ellie, are equally detailed and yet use the same style. The book starts with an account of Ellie's childhood and path into science written like a biography or a magazine profile, not like a novel in which she's the protagonist. The same is true of the other characters: we get characterization of a sort, but the tone ranges from Wikipedia article to long-form essay and never quite feels like a story. Ellie is the most interesting character in the book, partly because the way Sagan writes her is both distant but oddly compelling. Sagan (or perhaps Druyan?) tries hard to show what life is like for a woman born in the middle of the 20th century who is interested in science and I think mostly succeeds, although Ellie's reactions to sexism felt weirdly emotionless. The descriptions of her relationships are even odder and the parts where this book felt the least like a novel, but Sagan does sell some of that tone as reflective of Ellie's personality. She's a scientist, the work is the center of her life, and everything else, even when important, is secondary. It doesn't entirely make the writing style work, but it helps. Sagan does attempt to give Ellie a personal development arc related to her childhood and her relationships with her father and step-father. I thought the conclusion to that was neither believable nor anywhere near as important as Sagan thought it was, which was off-putting. Better were her ongoing arguments with evangelical Christians, one of whom is a close-minded ass and the other of which is a far more interesting character. They felt wedged into this book, and I'm dubious a realistic version of Ellie would have been the person to have those debates, but it's a subject Sagan clearly has deep personal interest in and that shows in how they're written. The other problem with Contact as a novel is that Sagan does not take science fiction seriously as a genre, instead treating it as a way to explore a thought experiment. To a science fiction reader, or at least to this science fiction reader, the interesting bits of this story involve the aliens. Those are not the bits Sagan is interested in. His attention is on how this sort of contact, and project, would affect humanity and human politics. We do get some more traditional science fiction near the end of the book, but Sagan then immediately backs away from it, minimizes that part of the story, and focuses exclusively on the emotional and philosophical implications for humans of his thought experiment. Since I found his philosophical musings about agnosticism and wonder and discovery less interesting than the actual science fiction bits, I found this somewhat annoying. The ending felt a bit more like a cheap trick than a satisfying conclusion. Interestingly, this entire novel is set in an alternate universe, for reasons entirely unexplained (at least that I noticed) in the book. It's set in the late 1990s but was written in 1985, so of course this is an alternate future, but the 1985 of this world still isn't ours. Yuri Gagarin was the first man to set foot on the moon, and the space program and the Cold War developed in subtly different ways. I'm not sure why Sagan made that choice, but it felt to me like he was separating his thought experiment farther from our world to give the ending more plausible deniability. There are, at the time of the novel, permanent orbital colonies for (mostly) rich people, because living in space turns out to greatly extend human lifespans. That gives Sagan an opportunity to wax poetic about the life-altering effects of seeing Earth from space, which in his alternate timeline rapidly sped up nuclear disarmament and made the rich more protective of the planet. This is an old bit of space boosterism that isn't as common these days, mostly because it's become abundantly clear that human psychology doesn't work this way. Sadly, rich sociopaths remain sociopaths even when you send them into space. I was a bit torn between finding Sagan's predictions charmingly hopeful and annoyingly daft. I don't think this novel is very successful as a novel. It's much longer than it needs to be and long parts of it drag. But it's still oddly readable; even knowing the rough shape of the ending in advance, I found it hard to put down once the plot properly kicks into gear about two-thirds of the way through. There's a lot in here that I'd argue with Sagan about, but he's thoughtful and makes a serious attempt to work out the political and scientific ramifications of such a discovery in detail. Despite the dry tone, he does a surprisingly good job capturing the anticipation and excitement of a very expensive and risky scientific experiment. I'm not sure I would recommend this book to anyone, but I'm also the person who found Gregory Benford's Timescape to be boring and tedious, despite its rave reviews as a science fiction novel about the practice of science. If that sort of book is more your jam, you may like Contact better than I did. Rating: 6 out of 10

13 December 2022

Dirk Eddelbuettel: RcppSpdlog 0.0.11 on CRAN: Small Enhancement

Version 0.0.11 of RcppSpdlog is now on CRAN and in Debian. RcppSpdlog bundles spdlog, a wonderful header-only C++ logging library with all the bells and whistles you would want that was written by Gabi Melman, and also includes fmt by Victor Zverovich. This release adds support for a basic file logger as a alternative to the console logger. This can be helpful with code which suppresses or hides console output as for example unit test code does. We also expose the formatting helper function for direct use at the C level from other packages, and mention the handy wrapper spdl in the README. The NEWS entry for this release follows.

Changes in RcppSpdlog version 0.0.11 (2022-12-13)

Export the formatter at C level

Mention package spdl in README.md

Support simple file-based logger

Courtesy of my CRANberries, there is also a diffstat report. More detailed information is on the RcppSpdlog page, or the package documention site. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Matthew Garrett: Trying to remove the need to trust cloud providers

First up: what I'm covering here is probably not relevant for most people. That's ok! Different situations have different threat models, and if what I'm talking about here doesn't feel like you have to worry about it, that's great! Your life is easier as a result. But I have worked in situations where we had to care about some of the scenarios I'm going to describe here, and the technologies I'm going to talk about here solve a bunch of these problems.

So. You run a typical VM in the cloud. Who has access to that VM? Well, firstly, anyone who has the ability to log into the host machine with administrative capabilities. With enough effort, perhaps also anyone who has physical access to the host machine. But the hypervisor also has the ability to inspect what's running inside a VM, so anyone with the ability to install a backdoor into the hypervisor could theoretically target you. And who's to say the cloud platform launched the correct image in the first place? The control plane could have introduced a backdoor into your image and run that instead. Or the javascript running in the web UI that you used to configure the instance could have selected a different image without telling you. Anyone with the ability to get a (cleverly obfuscated) backdoor introduced into quite a lot of code could achieve that. Obviously you'd hope that everyone working for a cloud provider is honest, and you'd also hope that their security policies are good and that all code is well reviewed before being committed. But when you have several thousand people working on various components of a cloud platform, there's always the potential for something to slip up.

Let's imagine a large enterprise with a whole bunch of laptops used by developers. If someone has the ability to push a new package to one of those laptops, they're in a good position to obtain credentials belonging to the user of that laptop. That means anyone with that ability effectively has the ability to obtain arbitrary other privileges - they just need to target someone with the privilege they want. You can largely mitigate this by ensuring that the group of people able to do this is as small as possible, and put technical barriers in place to prevent them from pushing new packages unilaterally.

Now imagine this in the cloud scenario. Anyone able to interfere with the control plane (either directly or by getting code accepted that alters its behaviour) is in a position to obtain credentials belonging to anyone running in that cloud. That's probably a much larger set of people than have the ability to push stuff to laptops, but they have much the same level of power. You'll obviously have a whole bunch of processes and policies and oversights to make it difficult for a compromised user to do such a thing, but if you're a high enough profile target it's a plausible scenario.

How can we avoid this? The easiest way is to take the people able to interfere with the control plane out of the loop. The hypervisor knows what it booted, and if there's a mechanism for the VM to pass that information to a user in a trusted way, you'll be able to detect the control plane handing over the wrong image. This can be achieved using trusted boot. The hypervisor-provided firmware performs a "measurement" (basically a cryptographic hash of some data) of what it's booting, storing that information in a virtualised TPM. This TPM can later provide a signed copy of the measurements on demand. A remote system can look at these measurements and determine whether the system is trustworthy - if a modified image had been provided, the measurements would be different. As long as the hypervisor is trustworthy, it doesn't matter whether or not the control plane is - you can detect whether you were given the correct OS image, and you can build your trust on top of that.

(Of course, this depends on you being able to verify the key used to sign those measurements. On real hardware the TPM has a certificate that chains back to the manufacturer and uniquely identifies the TPM. On cloud platforms you typically have to retrieve the public key via the metadata channel, which means you're trusting the control plane to give you information about the hypervisor in order to verify what the control plane gave to the hypervisor. This is suboptimal, even though realistically the number of moving parts in that part of the control plane is much smaller than the number involved in provisioning the instance in the first place, so an attacker managing to compromise both is less realistic. Still, AWS doesn't even give you that, which does make it all rather more complicated)

Ok, so we can (largely) decouple our trust in the VM from having to trust the control plane. But we're still relying on the hypervisor to provide those attestations. What if the hypervisor isn't trustworthy? This sounds somewhat ridiculous (if you can't run a trusted app on top of an untrusted OS, how can you run a trusted OS on top of an untrusted hypervisor?), but AMD actually have a solution for that. SEV ("Secure Encrypted Virtualisation") is a technology where (handwavily) an encryption key is generated when a new VM is created, and the memory belonging to that VM is encrypted with that key. The hypervisor has no access to that encryption key, and any access to memory initiated by the hypervisor will only see the encrypted content. This means that nobody with the ability to tamper with the hypervisor can see what's going on inside the OS (and also means that nobody with physical access can either, so that's another threat dealt with).

But how do we know that the hypervisor set this up, and how do we know that the correct image was booted? SEV has support for a "Launch attestation", a CPU generated signed statement that it booted the current VM with SEV enabled. But it goes further than that! The attestation includes a measurement of what was booted, which means we don't need to trust the hypervisor at all - the CPU itself will tell us what image we were given. Perfect.

Except, well. There's a few problems. AWS just doesn't have any VMs that implement SEV yet (there are bare metal instances that do, but obviously you're building your own infrastructure to make that work). Google only seem to provide the launch measurement via the logging service - and they only include the parsed out data, not the original measurement. So, we still have to trust (a subset of) the control plane. Azure provides it via a separate attestation service, but again it doesn't seem to provide the raw attestation and so you're still trusting the attestation service. For the newest generation of SEV, SEV-SNP, this is less of a big deal because the guest can provide its own attestation. But Google doesn't offer SEV-SNP hardware yet, and the driver you need for this only shipped in Linux 5.19 and Azure's SEV Ubuntu images only offer up to 5.15 at the moment, so making use of that means you're putting your own image together at the moment.

And there's one other kind of major problem. A normal VM image provides a bootloader and a kernel and a filesystem. That bootloader needs to run on something. That "something" is typically hypervisor-provided "firmware" - for instance, OVMF. This probably has some level of cloud vendor patching, and they probably don't ship the source for it. You're just having to trust that the firmware is trustworthy, and we're talking about trying to avoid placing trust in the cloud provider. Azure has a private beta allowing users to upload images that include their own firmware, meaning that all the code you trust (outside the CPU itself) can be provided by the user, and once that's GA it ought to be possible to boot Azure VMs without having to trust any Microsoft-provided code.

Well, mostly. As AMD admit, SEV isn't guaranteed to be resistant to certain microarchitectural attacks. This is still much more restrictive than the status quo where the hypervisor could just read arbitrary content out of the VM whenever it wanted to, but it's still not ideal. Which, to be fair, is where we are with CPUs in general.

(Thanks to Leonard Cohnen who gave me a bunch of excellent pointers on this stuff while I was digging through it yesterday)

comments

12 December 2022

Dirk Eddelbuettel: digest 0.6.31 on CRAN: snprintf Update

Release 0.6.31 of the digest package arrived at CRAN this weekend, and is being uploaded to Debian as well. digest creates hash digests of arbitrary R objects (using the md5, sha-1, sha-256, sha-512, crc32, xxhash32, xxhash64, murmur32, spookyhash, and blake3 algorithms) permitting easy comparison of R language objects. It is a mature and widely-used as many tasks may involve caching of objects for which it provides convenient general-purpose hash key generation to quickly identify the various objects. This release contains the tiny change (in maybe a dozen places) triggered recently by macOS and now checked for by r-devel which consists of replacing (v)sprintf calls with (v)snprintf. No more, no less. My CRANberries provides the usual summary of changes to the previous version. For questions or comments use the issue tracker off the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Jonathan McDowell: Setting up FreshRSS in a subdirectory

Ever since the demise of Google Reader I have been looking for a suitable replacement RSS reader. In the past I used to use Liferea but that was when I used a single desktop machine; these days I want to be able to read on my phone and multiple machines. I moved to Feedly and it s been mostly ok, but I m hitting the limit of feeds available in the free tier, and $72/year is a bit more than I can justify to myself. Especially when I have machines already available to me where I could self host something. The problem, of course, is what to host. It seems the best options are all written in PHP, so I had to get over my adverse knee-jerk reaction to that. I ended up on FreshRSS but if it hadn t worked out I d have tried TinyTinyRSS. Of course I m hosting on Debian, and the machine I chose to use was already running nginx and PostgreSQL. So I needed to install PHP:

$ sudo apt install php7.4-fpm php-curl php-gmp php-intl php-mbstring \
	php-pgsql php-xml php-zip

I put my FreshRSS install in /srv/freshrss so I grabbed the 1.20.2 release from GitHub (actually 1.20.1 at the time, but I ve upgraded to the latest since) and untared it in there. I gave www-data access to the data directory (sudo chown -R www-data /srv/freshrss/data) (yes, yes, I could have created a new user specifically for FreshRSS, but I ve chosen not to for now). There s no actual need to configure things up on the filesystem, you can do the initial setup from the web interface. Which is where the trouble came. I ve been an Apache user since 1998 and as a result it s what I know and what I go to. nginx is new to me. And I wanted my FreshRSS instance to live in a subdirectory of an existing TLS-enabled host, rather than have it s own hostname. Now, at least FreshRSS copes with this (unlike far too many other projects), you just have to configure your webserver correctly. Which took me more experimentation than I d like, but I ve ended up with the following snippet:

    # PHP files handling
    location ~ ^/freshrss/.+?\.php(/.*)?$  
        root /srv/freshrss/p;
        fastcgi_pass unix:/run/php/php-fpm.sock;
        fastcgi_split_path_info ^/freshrss(/.+\.php)(/.*)?$;
        set $path_info $fastcgi_path_info;
        fastcgi_param PATH_INFO $path_info;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
     
    location ~ ^/freshrss(/.*)?$  
        root /srv/freshrss/p;
        try_files $1 /freshrss$1/index.php$is_args$args;
     

Other than the addition of the freshrss prefix this ends up differing slightly from the FreshRSS webserver configuration example. I ended up having to make the path info on the fastcgi_split_path_info optional, and my try_files in the bare directory location directive needed $is_args$args added or I just ended up in a redirect loop because the session parameters didn t get passed through. I m sure there s a better way to do it, but I did a bunch of searching and this is how I ended up making it work. Before firing up the web configuration I created a suitable database:

$ sudo -Hu postgres psql
psql (13.8 (Debian 13.8-0+deb11u1))
Type "help" for help.
postgres=# create database freshrss;
CREATE DATABASE
postgres=# create user freshrss with encrypted password 'hunter2';
CREATE ROLE
postgres=# grant all privileges on database freshrss to freshrss;
GRANT
postgres=# \q

I ran through the local configuration, creating myself a user and adding some feeds, then created a cronjob to fetch updates hourly and keep a log:

# mkdir /var/log/freshrss
# chown :www-data /var/log/freshrss
# chmod 775 /var/log/freshrss
# cat > /etc/cron.d/freshrss-refresh <EOF
33 * * * * www-data /srv/freshrss/app/actualize_script.php > /var/log/freshrss/update-$(date --iso-8601=minutes).log 2>&1
EOF

Experiences so far? Reasonably happy. The interface seems snappy enough, and works well both on mobile and desktop. I m only running a single user instance at present, but am considering opening it up to some other folk and will see how that scales. And it clearly indicated a number of my feeds that were broken, so I ve cleaned some up that are still around and deleted the missing ones. Now I just need to figure out what else I should be subscribed to that I ve been putting off due to the Feedly limit!

Vasudev Kamath: Installing Debian from GRML Live CD

I had bought a Thinkpad E470 laptop back in 2018 which was lying unused for quite some time. Recently when I wanted to use it, I found that the keyboard is not working, especially some keys and after some time the laptop will hang in Lenovo boot screen. I came back to Bangalore almost after 2 years from my hometown (WFH due to Covid) and thought it was the right time to get my laptop back to normal working state. After getting the keyboard replaced I noticed that 1TB HDD is no longer fast enough for my taste!. I've to admit I never thought I would start disliking HDD so quickly thanks to modern SSD based work laptops. So as a second upgrade I got the HDD removed from my laptop and got a 240G SSD. Yeah I know its reduction from my original size but I intend to continue using my old HDD via USB SATA enclosure as an external HDD which can house the extra data which I need to save. So now that I've a SSD I need to install Debian Unstable again on it and this is where I tried something new. My colleague (name redacted on request) suggested to me use GRML live CD and install Debian via debootstrap. And after giving a thought I decided to try this out. Some reason for going ahead with this are listed below

Debian Installer does not support a proper BTRFS based root file system. It just allows btrfs as root but no subvolume support. Also I'm not sure about the luks support with btrfs as root.
I also wanted to give a try to systemd-boot as my laptop is UEFI capable and I've slowly started disliking Grub.
I really hate installing task-kde-desktop (Yeah you read it right, I've switched to be a KDE user for quite some time) which will pull tons of unwanted stuff and bloat. Well it's not just task-kde-desktop but any other task-desktop package does similar and I don't want to have too much of unused stuff and services running.

Disk Preparation As a first step I went to GRML website and downloaded current pre-release. Frankly, I'm using GRML for first time and I was not sure what to expect. When I booted it up I was bit taken a back to see its console based and I did not have a wired lan just a plain wireless dongle (Jiofi device) and was wondering what it will take to connect. But surprisingly curses based UI was pretty much straight forward to allow me to connect to Wifi AP. Another thing was the rescue CD had non-free firmware as the laptop was using ath10k device and needed non-free blobs to operate. Once I got shell prompt in rescue CD first thing I did was to reconfigure console-setup to increase font size which was very very small on default boot. Once that is done I did the following to create a 1G (FAT32) partition for EFI.

parted -a optimal -s /dev/sda mklabel gpt
parted -a optimal -s /dev/sda mkpart primary vfat 0% 1G
parted -a optimal -s /dev/sda set 1 esp on
mkfs.vfat -n boot_disk -F 32 /dev/sda1

So here is what I did: created a 1G vfat type partition and set the esp flag on it. This will be mounted to /boot/efi for systemd-boot. Next I created a single partition on the rest of the available free disk which will be used as the root file system. Next I encrypted the root parition using LUKS and then created the BTRFS file system on top of it.

cryptsetup luksFormat /dev/sda2
cryptsetup luksOpen /dev/sda2 ENC
mkfs.btrfs -L root_disk /dev/mapper/ENC

Next is to create subvolumes in BTRFS. I followed suggestion by colleague and created a top-level @ as subvolume below which created @/home @/var/log @/opt . Also enabled compression with zstd and level of 1 to avoid battery drain. Finally marked the @ as default subvolume to avoid adding it to fstab entry.

mount -o compress=zstd:1 /dev/mapper/ENC /mnt
btrfs subvol create /mnt/@
cd /mnt/@
btrfs subvol create ./home
btrfs subvol create ./opt
mkdir -p var
btrfs subvol create ./var/log
btrfs suvol set-default /mnt/@

Bootstrapping Debian Now that root disk is prepared next step was to bootstrap the root file system. I used debootstrap for this job. One thing I missed here from installer was ability to preseed. I tried looking around to figure out if we can preseed debootstrap but did not find much. If you know the procedure do point it to me.

cd /mnt/
debootstrap --include=dbus,locales,tzdata unstable @/ http://deb.debian.org/debian

Well this just gets a bare minimal installation of Debian I need to install rest of the things post this step manually by chroot into target folder @/. I like the grml-chroot command for chroot purpose, it does most of the job of mounting all required directory like /dev/ /proc /sys etc. But before entering chroot I need to mount the ESP partition we created to /boot/efi so that I can finalize the installation of kernel and systemd-boot.

umount /mnt
mount -o compress=zstd:1 /dev/mapper/ENC /mnt
mkdir -p /mnt/boot/efi
mount /dev/sda1 /mnt/boot/efi
grml-chroot /mnt /bin/bash

I remounted the root subvolume @ directly to /mnt now, remember I made @ as default subvolume before. I also mounted ESP partition with FAT32 file system to /boot/efi. Finally I used grml-chroot to get into chroot of newly bootstrapped file system. Now I will install the kernel and minimal KDE desktop installation and configure locales and time zone data for the new system. I wanted to use dracut instead of default initramfs-tools for initrd. I also need to install cryptsetup and btrfs-progs so I can decrypt and really boot into my new system.

apt-get update
apt-get install linux-image-amd64 dracut openssh-client \
                        kde-plasma-desktop plasma-workspace-wayland \
                        plasma-nm cryptsetup btrfs-progs sudo

Next is setting up crypttab and fstab entries for new system. Following entry is added to fstab

LABEL="root_disk" / btrfs defaults,compress=zstd:1 0 0

And the crypttab entry

ENCRYPTED_ROOT UUID=xxxx none discard,x-initrd.attach

I've not written actual UUID above this is just for the purpose of showing the content of /etc/crypttab. Once these entries are added we need to recreate initrd. I just reconfigured the installed kernel package for retriggerring the recreation of initrd using dracut. .. Reconfiguration was locales is done by editing /etc/locales.gen to uncomment en_US.UTF-8 and writing /etc/timezone with Asia/Kolkata. I used DEBIAN_FRONTEND=noninteractive to avoid another prompt asking for locale and timezone information.

export DEBIAN_FRONTEND=noninteractive
dpkg-reconfigure locales
dpkg-reconfigure tzdata

Added my user using adduser command and also set the root password as well. Added my user to sudo group so I can use sudo to elevate privileges.

Setting up systemd-boot So now basic usable system is ready last part is enabling the systemd-boot configuration as I'm not gonna use grub. I did following to install systemd-boot. Frankly I'm not expert of this it was colleague's suggestion. Before installing the systemd-boot I had to setup kernel command line. This can be done by writing command line to /etc/kernel/cmdline with following contents.

systemd.gpt_auto=no quiet root=LABEL=root_disk

I'm disabling systemd-gpt-generator to avoid race condition between crypttab entry and auto generated entry by systemd. I faced this mainly because of my stupidity of not adding entry root=LABEL=root_disk

apt-get install -y systemd-boot
bootctl install --make-entry-directory=yes --entry-token=machine-id
dpkg-reconfigure linux-image-6.0.0-5-amd64

Finally exit from the chroot and reboot into the freshly installed system. systemd-boot already ships a hook file zz-systemd-boot under /etc/kernel so its pretty much usable without any manual intervention. Previously after kernel installation we had to manually update kernel image in efi partitions using bootctl

Conclussion Though installing from live image is not new and debian-installer also does the same only difference is more control over installation and doing things which is installer is not letting you do (or should I say is not part of default installation?). If properly automated using scripts we can leverage this to do custom installation in large scale environments. I know there is FAI but I've not explored it and felt there is too much to setup for a simple installations with specific requirements. So finally I've a system with Debian which differs from default Debian installation :-). I should thank my colleague for rekindling nerd inside me who had stopped experimenting quite a long time back.

Russ Allbery: Review: The Unbroken

Review: The Unbroken, by C.L. Clark

Series:	Magic of the Lost #1
Publisher:	Orbit
Copyright:	March 2021
ISBN:	0-316-54267-9
Format:	Kindle
Pages:	490

The Unbroken is the first book of a projected fantasy trilogy. It is C.L. Clark's first novel. Lieutenant Touraine is one of the Sands, the derogatory name for the Balladairan Colonial Brigade. She, like the others of her squad, are conscript soldiers, kidnapped by the Balladairan Empire from their colonies as children and beaten into "civilized" behavior by Balladairan training. They fought in the Balladairan war against the Taargens. Now, they've been reassigned to El-Wast, capital city of Qaz l, the foremost of the southern colonies. The place where Touraine was born, from which she was taken at the age of five. Balladaire is not France and Qaz l is not Algeria, but the parallels are obvious and strongly implied by the map and the climates. Touraine and her squad are part of the forces accompanying Princess Luca, the crown princess of the Balladairan Empire, who has been sent to take charge of Qaz l and quell a rebellion. Luca's parents died in the Withering, the latest round of a recurrent plague that haunts Balladaire. She is the rightful heir, but her uncle rules as regent and is reluctant to give her the throne. Qaz l is where she is to prove herself. If she can bring the colony in line, she can prove that she's ready to rule: her birthright and her destiny. The Qaz li are uninterested in being part of Luca's grand plan of personal accomplishment. She steps off her ship into an assassination attempt, foiled by Touraine's sharp eyes and quick reactions, which brings the Sand to the princess's attention. Touraine's reward is to be assigned the execution of the captured rebels, one of whom recognizes her and names her mother before he dies. This sets up the core of the plot: Qaz li rebellion against an oppressive colonial empire, Luca's attempt to use the colony as a political stepping stone, and Touraine caught in between. One of the reasons why I am happy to see increased diversity in SFF authors is that the way we tell stories is shaped by our cultural upbringing. I was taught to tell stories about colonialism and rebellion in a specific ideological shape. It's hard to describe briefly, but the core idea is that being under the rule of someone else is unnatural as well as being an injustice. It's a deviation from the way the world should work, something unexpected that is inherently unstable. Once people unite to overthrow their oppressors, eventual success is inevitable; it's not only right or moral, it's the natural path of history. This is what you get when you try to peel the supremacy part away from white supremacy but leave the unshakable self-confidence and bedrock assumption that the universe cares what we think. We were also taught that rebellion is primarily ideological. One may be motivated by personal injustice, but the correct use of that injustice is to subsume it into concepts such as freedom and democracy. Those concepts are more "real" in some foundational sense, more central to the right functioning of the world, than individual circumstance. When the now-dominant group tells stories of long-ago revolution, there is no personal experience of oppression and survival in which to ground the story; instead, it's linked to anticipatory fear in the reader, to the idea that one's privileges could be taken away by a foreign oppressor and that the counter to this threat is ideological unity. Obviously, not every white fantasy author uses this story shape, but the tendency runs deep because we're taught it young. You can see it everywhere in fantasy, from Lord of the Rings to Tigana. The Unbroken uses a much different story shape, and I don't think it's a coincidence that the author is Black. Touraine is not sympathetic to the Qaz li. These are not her people and this is not her life. She went through hell in Balladairan schools, but she won a place, however tenuous. Her personal role model is General Cantic, the Balladairan Blood General who was also one of her instructors. Cantic is hard as nails, unforgiving, unbending, and probably a war criminal, but also the embodiment of a military ethic. She is tough but fair with the conscript soldiers. She doesn't put a stop to their harassment by the regular Balladairan troops, but neither does she let it go too far. Cantic has power, she knows how to keep it, and there is a place for Touraine in Cantic's world. And, critically, that place is not just hers: it's one she shares with her squad. Touraine's primary loyalty is not to Balladaire or to Qaz l. It's to the Sands. Her soldiers are neither one thing nor the other, and they disagree vehemently among themselves about what Qaz l and their other colonial homes should be to them, but they learned together, fought together, and died together. That theme is woven throughout The Unbroken: personal bonds, third and fourth loyalties, and practical ethics of survival that complicate and contradict simple dichotomies of oppressor and oppressed. Touraine is repeatedly offered ideological motives that the protagonist in the typical story shape would adopt. And she repeatedly rejects them for personal bonds: trying to keep her people safe, in a world that is not looking out for them. The consequence is that this book tears Touraine apart. She tries to walk a precarious path between Luca, the Qaz li, Cantic, and the Sands, and she falls off that path a lot. Each time I thought I knew where this book was going, there's another reversal, often brutal. I tend to be a happily-ever-after reader who wants the protagonist to get everything they need, so this isn't my normal fare. The amount of hell that Touraine goes through made for difficult reading, worse because much of it is due to her own mistakes or betrayals. But Clark makes those decisions believable given the impossible position Touraine is in and the lack of role models she has for making other choices. She's set up to fail, and the price of small victories is to have no one understand the decisions that she makes, or to believe her motives. Luca is the other viewpoint character of the book (and yes, this is also a love affair, which complicates both of their loyalties). She is the heroine of a more typical genre fantasy novel: the outsider princess with a physical disability and a razor-sharp mind, ambitious but fair (at least in her own mind), with a trusted bodyguard advisor who also knew her father and a sincere desire to be kinder and more even-handed in her governance of the colony. All of this is real; Luca is a protagonist, and the reader is not being set up to dislike her. But compared to Touraine's grappling with identity, loyalty, and ethics, Luca is never in any real danger, and her concerns start to feel too calculated and superficial. It's hard to be seriously invested in whether Luca proves herself or gets her throne when people are being slaughtered and abused. This, I think, is the best part of this book. Clark tells a traditional ideological fantasy of learning to be a good ruler, but she puts it alongside a much deeper and more complex story of multi-faceted oppression. She has the two protagonists fall in love with each other and challenges them to understand each other, and Luca does not come off well in this comparison. Touraine is frustrated, impulsive, physical, and sometimes has catastrophically poor judgment. Luca is analytical and calculating, and in most ways understands the political dynamics far better than Touraine. We know how this story usually goes: Luca sees Touraine's brilliance and lifts her out of the ranks into a role of importance and influence, which Touraine should reward with loyalty. But Touraine's world is more real, more grounded, and more authentic, and both Touraine and the reader know what Luca could offer is contingent and comes with a higher price than Luca understands. (Incidentally, the cover of The Unbroken, designed by Lauren Panepinto with art by Tommy Arnold, is astonishingly good at capturing both Touraine's character and the overall feeling of the book. Here's a larger version.) The writing is good but uneven. Clark loves reversals, and they did keep me reading, but I think there were too many of them. By the end of the book, the escalation of betrayals and setbacks was more exhausting than exciting, and I'd stopped trusting anything good would last. (Admittedly, this is an accurate reflection of how Touraine felt.) Touraine's inner monologue also gets a bit repetitive when she's thrashing in the jaws of an emotional trap. I think some of this is first-novel problems of over-explaining emotional states and character reasoning, but these problems combine to make the book feel a bit over-long. I'm also not in love with the ending. It's perhaps the one place in the book where I am more cynical about the politics than Clark is, although she does lay the groundwork for it. But this book is also full of places small and large where it goes a different direction than most fantasy and is better for it. I think my favorite small moment is Touraine's quiet refusal to defend herself against certain insinuations. This is such a beautiful bit of characterization; she knows she won't be believed anyway, and refuses to demean herself by trying. I'm not sure I can recommend this book unconditionally, since I think you have to be in the mood for it, but it's one of the most thoughtful and nuanced looks at colonialism and rebellion I can remember seeing in fantasy. I found it frustrating in places, but I'm also still thinking about it. If you're looking for a political fantasy with teeth, you could do a lot worse, although expect to come out the other side a bit battered and bruised. Followed by The Faithless, and I have no idea where Clark is going to go with the second book. I suppose I'll have to read and find out. Content note: In addition to a lot of violence, gore, and death, including significant character death, there's also a major plague. If you're not feeling up to reading about panic caused by contageous illness, proceed with caution. Rating: 7 out of 10

11 December 2022

Vincent Bernat: Akvorado: a flow collector, enricher, and visualizer

Earlier this year, we released Akvorado, a flow collector, enricher, and visualizer. It receives network flows from your routers using either NetFlow v9, IPFIX, or sFlow. Several pieces of information are added, like GeoIP and interface names. The flows are exported to Apache Kafka, a distributed queue, then stored inside ClickHouse, a column-oriented database. A web frontend is provided to run queries. A live version is available for you to play.

Akvorado web interface displays the result of a simple query using stacked areas

Several alternatives exist:

Kentik, a popular cloud-based solution,
ElastiFlow, a self-hosted proprietary solution, or
your own assembly of open source solutions by picking a flow collector (pmacct, GoFlow2, or vFlow), a distributed queue¹ (Apache Kafka or RabbitMQ), a database (ClickHouse, Elasticsearch, or Apache Pinot), and a web frontend (Grafana, Kibana, or Apache Superset).

Akvorado differentiates itself from these solutions because:

it is open source (licensed under the AGPLv3 license), and
it bundles flow collection, storage, and a web interface into a single product.

The proposed deployment solution relies on Docker Compose to set up Akvorado, Zookeeper, Kafka, and ClickHouse. I hope it should be enough for anyone to get started quickly. Akvorado is performant enough to handle 100 000 flows per second with 64 GB of RAM and 24 vCPU. With 2 TB of disk, you should expect to keep data for a few years. I spent some time writing a fairly complete documentation. It seems redundant to repeat its content in this blog post. There is also a section about its internal design if you are interested in how it is built. I also did a FRnOG presentation earlier this year, and a ClickHouse meetup presentation, which focuses more on how ClickHouse is used. I plan to write more detailed articles on specific aspects of Akvorado. Stay tuned!

While the collector could write directly to the database, the queue buffers flows if the database is unavailable. It also enables you to process flows with another piece of software (like an anti-DDoS system).

Russ Allbery: Review: The Fifth Elephant

Review: The Fifth Elephant, by Terry Pratchett

Series:	Discworld #24
Publisher:	Harper
Copyright:	2000
Printing:	May 2014
ISBN:	0-06-228013-9
Format:	Mass market
Pages:	455

The Fifth Elephant is the 24th Discworld and fifth Watch novel, and largely assumes you know who the main characters are. This is not a good place to start. The dwarves are electing a new king. The resulting political conflict is spilling over into the streets of Ankh-Morpork, but that's not the primary problem. First, the replica Scone of Stone, a dwarven artifact used to crown the Low King of the Dwarves, is stolen from the Dwarf Bread Museum. Then, Vimes is dispatched to berwald, ostensibly to negotiate increased fat exports with the new dwarven king. And then Angua disappears, apparently headed towards her childhood home in berwald, which immediately prompts Carrot to resign and head after her. The City Watch is left in the hands of now-promoted Captain Colon. We see lots of Lady Sybil for the first time since Guards! Guards!, and there's a substantial secondary plot with Angua and Carrot and a tertiary plot with Colon making a complete mess of things back home, but this is mostly a Vimes novel. As usual, Vetinari is pushing him outside of his comfort zone, but he's not seriously expecting Vimes to act like an ambassador. He's expecting Vimes to act like a policeman, even though he's way outside his jurisdiction. This time, that means untangling a messy three-sided political situation involving the dwarves, the werewolves, and the vampires. There is some Igor dialogue in this book, but thankfully Pratchett toned it down a lot and it never started to bother me. I do enjoy Pratchett throwing Vimes and his suspicious morality at political problems and watching him go at them sideways. Vimes's definition of crimes is just broad enough to get him fully invested in a problem, but too narrow to give him much patience with the diplomatic maneuvering. It makes him an unpredictable diplomat in a clash of cultures way that's fun to read about. Cheery and Detritus are great traveling companions for this, since both of them also unsettle the dwarves in wildly different ways. I also have to admit that Pratchett is doing more interesting things with the Angua and Carrot relationship than I had feared. In previous books, I was getting tired of their lack of communication and wasn't buying the justifications for it, but I think I finally understand why the communication barriers are there. It's not that Angua refuses to talk to Carrot (although there's still a bit of that going on). It's that Carrot's attitude towards the world is very strange, and gets stranger the closer you are to him. Carrot has always been the character who is too earnest and straightforward and good for Ankh-Morpork and yet somehow makes it work, but Pratchett is doing something even more interesting with the concept of nobility. A sufficiently overwhelming level of heroic ethics becomes almost alien, so contrary to how people normally think that it can make conversations baffling. It's not that Carrot is perfect (sometimes he does very dumb things), it's that his natural behavior follows a set of ethics that humans like to pretend they follow but actually don't and never would entirely. His character should be a boring cliche or an over-the-top parody, and yet he isn't at all. But Carrot's part is mostly a side plot. Even more than Jingo, The Fifth Elephant is establishing Vimes as a force to be reckoned with, even if you take him outside his familiar city. He is in so many ways the opposite of Vetinari, and yet he's a tool that Vetinari is extremely good at using. Colon of course is a total disaster as the head of the Watch, and that's mostly because Colon should never be more than a sergeant, but it's also because even when he's taking the same action as Vimes, he's not doing it for the same reasons or with the same stubborn core of basic morality and loyalty that's under Vimes's suspicious conservatism. The characterization in the Watch novels doesn't seem that subtle or deep at first, but it accumulates over the course of the series in a way that I think is more effective than any of the other story strands. Vetinari, Vimes, and Carrot all represent "right," or at least order, in overlapping stories of right versus wrong, but they do so in radically different ways and with radically different goals. Each time one of them seems ascendant, each time one of their approaches seems more clearly correct, Pratchett throws them at a problem where a different approach is required. It's a great reading experience. This was one of the better Discworld novels even though I found the villains to be a bit tedious and stupid. Recommended. Followed by The Truth in publication order. The next Watch novel is Night Watch. Rating: 8 out of 10

10 December 2022

Simon Josefsson: Trisquel 11 on NV41PZ: First impressions

My NovaCustom NV41PZ laptop arrived a couple of days ago, and today I had some time to install it. You may want to read about my purchasing decision process first. I expected a rough ride to get it to work, given the number of people claiming that modern laptops can t run fully free operating systems. I first tried the Trisquel 10 live DVD and it booted fine including network, but the mouse trackpad did not work. Before investigating it, I noticed a forum thread about Trisquel 11 beta3 images, and being based on Ubuntu 22.04 LTS and has Linux-libre 5.15 it seemed better to start with more modern software. After installing through the live DVD successfully, I realized I didn t like MATE but wanted to keep using GNOME. I reverted back to installing a minimal environment through the netinst image, and manually installed GNOME (apt-get install gnome) since I prefer that over MATE, together with a bunch of other packages. I ve been running it for a couple of hours now, and here is a brief summary of the hardware components that works.

CPU	Alder Lake Intel i7-1260P
Memory	2x32GB Kingston DDR4 SODIMM 3200MHz
Storage	Samsung 980 Pro 2TB NVME
BIOS	Dasharo Coreboot
Graphics	Intel Xe
Screen (internal)	14 1920 1080
Screen (HDMI)	Connected to Dell 27 2560 1440
Screen (USB-C)	Connected to Dell 27 2560 1440 via Wavlink port extender
Webcam	Builtin 1MP Camera
Microphone	Intel Alder Lake
Keyboard	ISO layout, all function keys working
Mouse	Trackpad, tap clicking and gestures
Ethernet RJ45	Realtek RTL8111/8168/8411 with r8169 driver
Memory card	O2 Micro comes up as /dev/mmcblk0
Docking station	Wavlink 4xUSB, 2xHDMI, DP, RJ45,
Connectivity	USB-A, USB-C
Audio	Intel Alder Lake

Hardware components and status So what s not working? Unfortunately, NovaCustom does not offer any WiFi or Bluetooth module that is compatible with Trisquel, so the AX211 (1675x) Wifi/Bluetooth card in it is just dead weight. I imagine it would be possible to get the card to work if non-free firmware is loaded. I don t need Bluetooth right now, and use the Technoetic N-150 USB WiFi dongle when I m not connected to wired network. Compared against my X201, the following factors have improved.

Faster CPU benchmark suggests it is 8 times faster than my old i7-620M. While it feels snappier it is not a huge difference. While NVMe should improve SSD performance, benchmark wise the NVMe 980Pro only seems around 2-3 faster than the SATA-based 860 Evo. Going from 6GB to 64GB is 10 times more memory, which is useful for disk caching.
BIOS is free software.
EC firmware is free.
Operating system follows the FSDG.

I m still unhappy about the following properties with both the NV41PZ and the X201.

CPU microcode is not available under free license.
Intel Mangement Engine is still present in the CPU.
No builtin WiFi/Bluetooth that works with free software.
Some other secondary processors (e.g., disk or screen) may be running non-free software but at least none requires non-free firmware.

Hopefully my next laptop will have improved on this further. I hope to be able to resolve the WiFi part by replacing the WiFi module, there appears to be options available but I have not tested them on this laptop yet. Does anyone know of a combined WiFi and Bluetooth M.2 module that would work on Trisquel? While I haven t put the laptop to heavy testing yet, everything that I would expect a laptop to be able to do seems to work fine. Including writing this blog post!

Next.

Previous.

Search Results: "ag"

18 December 2022

Ian Jackson: Rust needs #[throws]

Bastian Venthur: The State of Python Packaging in 2022

Russell Coker: Wall Facers

Freexian Collaborators: Monthly report about Debian Long Term Support, November 2022 (by Anton Gladky)

16 December 2022

Charles Plessy: Bad words in Debian.

Simon Josefsson: Guix 1.4 on NV41PZ

15 December 2022

Reproducible Builds: Supporter spotlight: David A. Wheeler on supply chain security

Russell Coker: Pixel 6A

Abhijith PA: Running PostmarketOS on my phone

Dirk Eddelbuettel: spdl 0.0.3 on CRAN: Adding File Logger

Changes in spld version 0.0.3 (2022-12-14)

Add `filesetup` method

14 December 2022

Russ Allbery: Review: Contact

13 December 2022

Dirk Eddelbuettel: RcppSpdlog 0.0.11 on CRAN: Small Enhancement

Changes in RcppSpdlog version 0.0.11 (2022-12-13)

Export the formatter at C level

Mention package spdl in README.md

Support simple file-based logger

Matthew Garrett: Trying to remove the need to trust cloud providers

12 December 2022

Dirk Eddelbuettel: digest 0.6.31 on CRAN: snprintf Update

Jonathan McDowell: Setting up FreshRSS in a subdirectory

Vasudev Kamath: Installing Debian from GRML Live CD

Russ Allbery: Review: The Unbroken

11 December 2022

Vincent Bernat: Akvorado: a flow collector, enricher, and visualizer

Russ Allbery: Review: The Fifth Elephant

10 December 2022

Simon Josefsson: Trisquel 11 on NV41PZ: First impressions

Search Results: "ag"

18 December 2022

16 December 2022

15 December 2022

Changes in spld version 0.0.3 (2022-12-14) Add filesetup method

14 December 2022

13 December 2022

Changes in RcppSpdlog version 0.0.11 (2022-12-13) Export the formatter at C level Mention package spdl in README.md Support simple file-based logger

12 December 2022

11 December 2022

10 December 2022

Changes in spld version 0.0.3 (2022-12-14)

Add `filesetup` method

Changes in RcppSpdlog version 0.0.11 (2022-12-13)

Export the formatter at C level

Mention package spdl in README.md

Support simple file-based logger