Search Results: "ag"

16 October 2022

Vincent Fourmond: Tutorial: analysis of multiwavelength fast kinetics data

The purpose of this post is to demonstrate a first approach to the analysis of multiwavelength kinetic data, like those obtained using stopped-flow data. To practice, we will use data that were acquired during the stopped flow practicals of the MetBio summer school from the FrenchBIC. During the practicals, the student monitored the reaction of myoglobin (in its Fe(III) state) with azide, which yields a fast and strong change in the absorbance spectrum of the protein, which was monitored using a diode array. The data is publicly available on zenodo. Aims of this tutorial The purpose of this tutorial is to teach you to use the free softwareQSoas to run a simple, multiwavelength exponential fit on the data, and to look at the results. This is not a kinetics lecture, so that it will not go in depth about the use of the exponential fit and its meaning. Getting started: loading the file First, make sure you have a working version of QSoas, you can download them (for free) there. Then download the data files from zenodo. We will work only on the data file Azide-1.25mm_001.dat, but of course, the purpose of this tutorial is to enable you to work on all of them. The data files contain the time evolution of the absorbance for all wavelengths, in a matrix format, in which each row correpond to a time point and each column to a wavelength. Start QSoas, and launch the command:

QSoas> load /comments='"'

Then, choose the Azide-1.25mm_001.dat data file. This should bring up a horizontal red line at the bottom of the data display, with X values between about 0 and 2.5. If you zoom on the red line with the mouse wheel, you'll realize it is data. The /comments='"' part is very important since it allows the extraction of the wavelength from the data. We will look at what it means another day. At this stage, you can look at the loaded data using the command:

QSoas> edit

You should have a window looking like this:

The rows each correspond to a data point displayed on the window below. The first column correspond to the X values, the second the Y values, and all the other ones to extra Y columns (they are not displayed by default). What is especially interesting is the first row, which contains a nan as the X value and what is obviously the wavelength for all the Y values. To tell that QSoas should take this line as the wavelength (which will be the perpendicular coordinate, the coordinate of the other direction of the matrix), first close the edit window and run:

QSoas> set-perp /from-row=0

Splitting and fitting Now, we have a single dataset containing a lot of Y columns. We want to fit all of them simultaneously with a (mono) exponential fit. For that, we first need to split the big matrix into a series of X,Y datasets (because fitting only works on the first Y). This is possible by running:

QSoas> expand /style=red-to-blue /flags=kinetics

Your screen should now look like this:

You're looking at the kinetics at all wavelengths at the same time (this may take some time to display on your computer, it is after all a rather large number of data points). The /style=red-to-blue is not strictly necessary, but it gives the red to blue color gradient which makes things easier to look at (and cooler !). The /flags=kinetics is there to attach a label (a flag) to the newly created datasets so we can easily manipulate all of them at the same time. Then it's time to fit, with the following command:

QSoas> mfit-exponential-decay flagged:kinetics

This should bring up a new window. After resizing it, you should have something that looks like this:

The bottom of the fit window is taken by the parameters, each with two checkboxes on the right to set them fixed (i.e. not determined by the fitting mechanism) and/or global (i.e. with a single value for all the datasets, here all the wavelengths). The top shows the current dataset along with the corresponding fit (in green), and, below, the residuals. You can change the dataset by clicking on the horizontal arrows or using Ctrl+PgUp or Ctrl+PgDown (keep holding it to scan fast). See the Z = 728.15 showing that QSoas has recognized that the currently displayed dataset corresponds to the wavelength 728.15. The equation fitted to the data is: $$y(x) = A_\infty + A_1 \times \exp -(x - x_0)/\tau_1$$ In this case, while the $A_1$ and $A_\infty$ parameters clearly depend on the wavelength, the time constant of evolution should be independent of wavelength (the process happens at a certain rate regardless of the wavelength we're analyzing), so that the $\tau_1$ parameter should be common for all the datasets/wavelengths. Just click on the global checkbox at the right of the tau_1 parameter, make sure it is checked, and hit the Fit button... The fit should not take long (less than a minute), and then you end up with the results of the fits: all the parameters. The best way to look at the non global parameters like $A_1$ and $A_\infty$ is to use the Show Parameters item from the Parameters menu. Using it and clicking on A_inf too should give you a display like this one:

The A_inf parameter corresponds to the spectum at infinite time (of azide-bound heme), while the A_1 parameter corresponds to the difference spectrum between the initial (azide-free) and final (azide-bound) states. Now, the fit is finished, you can save the parameters if you want to reload them in a later fit by using the Parameters/Save menu item or export them in a form more suitable for plotting using Parameters/Export (although QSoas can also display and the parameters saved using Save). This concludes this first approach to fitting the data. What you can do is

look at the depence of the tau_1 parameter as a function of the azide concentration;

try fitting more than one exponential, using for instance:

QSoas> mfit-exponential-decay /exponentials=2 flagged:kinetics

How to read the code above All the lines starting by QSoas> in the code areas above are meant to be typed into the QSoas command line (at the bottom of the window), and started by pressing enter at the end. You must remove the QSoas> bit. The other lines (when applicable) show you the response of QSoas, in the terminal just above the command-line. You may want to play with the QSoas tutorial to learn more about how to interact with QSoas. About QSoas QSoas is a powerful open source data analysis program that focuses on flexibility and powerful fitting capacities. It is released under the GNU General Public License. It is described in Fourmond, Anal. Chem., 2016, 88 (10), pp 5050 5052. Current version is 3.1. You can freely (and at no cost) download its source code or precompiled versions for MacOS and Windows there. Alternatively, you can clone from the GitHub repository.
Contact: find my email address there, or contact me on LinkedIn.

kpcyrd: updlockfiles: Manage lockfiles in PKGBUILDs for upstreams that don't ship them

I ve released a new tool to manage lockfiles for Arch Linux packages that can t use a lockfile from the official upstream release. It integrates closely with other Arch Linux tooling like updpkgsums that s already used to pin the content of build inputs in PKGBUILD. To use this, the downstream lockfile becomes an additional source input in the source= array of our PKGBUILD (this is already the case for some packages).

source=("git+https://github.com/vimeo/psalm.git#commit=$ _commit "
        "composer.lock")

You would then add a new function named updlockfiles that can generate new lockfiles and copies them into $outdir, and a prepare function to copy the lockfile in the right place:

 prepare()  
   cd $ pkgname 
   cp ../composer.lock .
 
updlockfiles()  
  cd $ pkgname 
  rm -f composer.lock
  composer update
  cp composer.lock "$ outdir /"
 

To update the package to the latest (compatible) patch level simply run:

updlockfiles

This can also be used in case upstreams lockfile has vulnerable dependencies that you want to patch downstream. For more detailed instructions see the readme.

Thanks This work is currently crowd-funded on github sponsors. I d like to thank @SantiagoTorres, @repi and @rgacogne for their support in particular.

15 October 2022

Jonathan Dowland: podman generate

I've been working with and on container technology for seven years, but I still learn new things every day. Recently I read the excellent LWN article Docker and the OCI container ecosystem and this was news to me:

Running the docker CLI under a process supervisor only results in supervising the CLI process. This has a variety of consequences for users of these tools. For example, any attempt to limit a container's memory usage by running the CLI as a systemd service will fail; the limits will only apply to the CLI and its non-existent children. In addition, attempts to terminate a client process may not result in terminating all of the processes in the container.

Huh of course! I hadn't really thought about that. I run a small number of containers on my home system via docker (since I was using it at work) and systemd, and sometimes I have weird consistency issues. This explains them. Later:

As a result, Podman plays nicely with tools like systemd; using podman run with a process supervisor works as expected, because the processes inside the container are children of podman run. The developers of Podman encourage people to use it in this way by a command to generate systemd units for Podman containers.

Given the above, it seemed like a good idea to migrate my few local containers over to Podman. This was easy. The first part is copying the images from Docker's storage to Podman's. To do this, I used the skopeo tool:

sudo skopeo copy  docker-daemon,containers-storage :octoprint/octoprint:latest

(I want to launch these as a superuser, rather than use root-less containers, to isolate the containers from each other: rootless ones are forced to share a common namespace.) Once the images were copied over, I needed to start up a container instance via Podman. For most of them, running under Docker, I had volume-mounted the important bits for persistent data. I can do the same with Podman:

# podman run -d --rm \
    --name octoprint \
    -p 8080:80 \
    -v /archive/octoprint:/octoprint \
    --device /dev/ttyACM0:/dev/ttyACM0 \
    octoprint/octoprint

Once an instance was running, I can use podman generate to create a Systemd unit file which describes creating an equivalent container:

# cd /etc/systemd/system
# podman generate systemd octoprint \
    --new \
    --name \
    --files

For some of the containers I was running, there are a few more steps required: migrating docker volumes and configuring equivalents for private docker networks. But Podman's versions of those concepts are largely identical. Running a mixture of Podman and Docker containers side-by-side also meant renumbering some private addresses for migrated hosts whilst the original network was still up.

14 October 2022

Shirish Agarwal: Dowry, Racism, Railways

Dowry Few days back, had posted about the movie Raksha Bandhan and whatever I felt about it. Sadly, just couple of days back, somebody shared this link. Part of me was shocked and part of me was not. Couple of acquaintances of mine in the past had said the same thing for their daughters. And in such situations you are generally left speechless because you don t know what the right thing to do is. If he has shared it with you being an outsider, how many times he must have told the same to their wife and daughters? And from what little I have gathered in life, many people have justified it on similar lines. And while the protests were there, sadly the book was not removed. Now if nurses are reading such literature, how their thought process might be forming, you can tell :(. And these are the ones whom we call for when we are sick and tired :(. And I have not taken into account how the girls/women themselves might be feeling. There are similar things in another country but probably not the same, nor the same motivations though although feeling helplessness in both would be a common thing. But such statements are not alone. Another gentleman in slightly different context shared this as well

The above is a statement shared in a book recommended for CTET (Central Teacher s Eligibility Test that became mandatory to be taken as the RTE (Right To Education) Act came in.). The statement says People from cold places are white, beautiful, well-built, healthy and wise. And people from hot places are black, irritable and of violent nature. Now while I can agree with one part of the statement that people residing in colder regions are more fair than others but there are loads of other factors that determine fairness or skin color/skin pigmentation. After a bit of search came to know that this and similar articulation have been made in an idea/work called Environmental Determinism . Now if you look at that page, you would realize this was what colonialism is and was all about. The idea that the white man had god-given right to rule over others. Similarly, if you are fair, you can lord over others. Seems simplistic, but yet it has a powerful hold on many people in India. Forget the common man, this thinking is and was applicable to some of our better-known Freedom fighters. Pune s own Bal Gangadhar Tilak The Artic Home to the Vedas. It sort of talks about Aryans and how they invaded India and became settled here. I haven t read or have access to the book so have to rely on third-party sources. The reason I m sharing all this is that the right-wing has been doing this myth-making for sometime now and unless and until you put a light on it, it will continue to perpetuate . For those who have read this blog, do know that India is and has been in casteism from ever. They even took the fair comment and applied it to all Brahmins. According to them, all Brahmins are fair and hence have god-given right to lord over others. What is called the Eton boy s network serves the same in this casteism. The only solution is those idea under limelight and investigate. To take the above, how does one prove that all fair people are wise and peaceful while all people black and brown are violent. If that is so, how does one count for Mahatma Gandhi, Martin Luther King Junior, Nelson Mandela, Michael Jackson the list is probably endless. And not to forget that when Mahatma Gandhiji did his nonviolent movements either in India or in South Africa, both black and brown people in millions took part. Similar examples of Martin Luther King Jr. I know and read of so many non-violent civl movements that took place in the U.S. For e.g. Rosa Parks and the Montgomery Bus Boycott. So just based on these examples, one can conclude that at least the part about the fair having exclusive rights to being fair and noble is not correct. Now as far as violence goes, while every race, every community has had done violence in the past or been a victim of the same. So no one is and can be blameless, although in light of the above statement, the question can argumentated as to who were the Vikings? Both popular imagination and serious history shares stories about Vikings. The Vikings were somewhat nomadic in nature even though they had permanent settlements but even then they went on raids, raped women, captured both men and women and sold them at slaves. So they are what pirates came to be, but not the kind Hollywood romanticizes about. Europe in itself has been a tale in conflict since time immemorial. It is only after the formation of EU that most of these countries stopped fighting each other From a historical point perspective, it is too new. So even the part of fair being non-violent dies in face of this evidence. I could go on but this is enough on that topic.

Railways and Industrial Action around the World. While I have shared about Railways so many times on this blog, it continues to fascinate me that how people don t understand the first things about Railways. For e.g. Railways is a natural monopoly. What that means is and you can look at all and any type of privatization around the world, you will see it is a monopoly. Unlike the road or Skies, Railways is and would always be limited by infrastructure and the ability to have new infrastructure. Unlike in road or Skies (even they have their limits) you cannot run train services on a whim. At any particular point in time, only a single train could and should occupy a stretch of Railway network. You could have more trains on one line, but then the likelihood of front or rear-end collisions becomes a real possibility. You also need all sorts of good and reliable communications, redundant infrastructure so if one thing fails then you have something in place. The reason being a single train can carry anywhere from 2000 to 5000 passengers or more. While this is true of Indian Railways, Railways around the world would probably have some sort of similar numbers.It is in this light that I share the below videos.
To be more precise, see the fuller video
Now to give context to the recording above, Mike Lynch is the general secretary at RMT. For those who came in late, both UK and the U.S. have been threatened by railway strikes. And the reason for the strikes or threat of strikes is similar. Now from the company perspective, all they care is to invest less and make the most profits that can be given to equity shareholders. At the same time, they have freezed the salaries of railway workers for the last 3 years. While the politicians who were asking the questions, apparently gave themselves raise twice this year. They are asking them to negotiate at 8% while inflation in the UK has been 12.3% and projected to go higher. And it is not only the money. Since the 1980s when UK privatized the Railways, they stopped investing in the infrastructure. And that meant that the UK Railway infrastructure over period of time started getting behind and is even behind say Indian Railways which used to provide most bang for the buck. And Indian Railways is far from ideal. Ironically, most of the operators on UK are nationalized Railways of France, Germany etc. but after the hard Brexit, they too are mulling to cut their operations short, they have too There is also the EU Entry/Exit system that would come next year. Why am I sharing about what is happening in UK Rail, because the Indian Government wants to follow the same thing, and fooling the public into saying we would do it better. What inevitably will happen is that ticket prices go up, people no longer use the service, the number of services go down and eventually they are cancelled. This has happened both in Indian Railways as well as Airlines. In fact, GOI just recently announced a credit scheme just a few days back to help Airlines stay afloat. I was chatting with a friend who had come down to Pune from Chennai and the round-trip cost him INR 15k/- on that single trip alone. We reminisced how a few years ago, 8 years to be precise, we could buy an Air ticket for 2.5k/- just a few days before the trip and did it. I remember doing/experiencing at least a dozen odd trips via air in the years before 2014. My friend used to come to Pune, almost every weekend because he could afford it, now he can t do that. And these are people who are in the above 5-10% of the population. And this is not just in UK, but also in the United States. There is one big difference though, the U.S. is mainly a freight carrier while the UK Railway Operations are mostly passenger based. What was and is interesting that Scotland had to nationalize their services as they realized the Operators cannot or will not function when they were most needed. Most of the public even in the UK seem to want a nationalized rail service, at least their polls say so. So, it would definitely be interesting to see what happens in the UK next year. In the end, I know I promised to share about books, but the above incidents have just been too fascinating to not just share the news but also share what I think about them. Free markets function good where there is competition, for example what is and has been happening in China for EV s but not where you have natural monopolies. In all Railway privatization, you have to handover the area to one person, then they have no motivation. If you have multiple operators, then there would always be haggling as to who will run the train and at what time. In either scenario, it doesn t work and raises prices while not delivering anything better I do take examples from UK because lot of things are India are still the legacy of the British. The whole civil department that was created in 1953 is/was a copy of the British civil department at that time and it is to this day. P.S. Just came to know that the UK Chancellor Kwasi Kwarteng was just sacked as UK Chancellor. I do commend Truss for facing the press even though she might be dumped a week later unlike our PM who hasn t faced a single press conference in the last 8 odd years.

https://www.youtube.com/watch?v=oTP6ogBqU7of The difference in Indian and UK politics seems to be that the English are now asking questions while here in India, most people are still sleeping without a care in the world. Another thing to note Minidebconf Palakkad is gonna happen 12-13th November 2022. I am probably not gonna go but would request everyone who wants to do something in free software to attend it. I am not sure whether I would be of any use like this and also when I get back, it would be an empty house. But for people young and old, who want to do anything with free/open source software it is a chance not to be missed. Registration of the same closes on 1st of November 2022. All the best, break a leg Just read this, beautifully done.

Simon Josefsson: On language bindings & Relaunching Guile-GnuTLS

The Guile bindings for GnuTLS has been part of GnuTLS since spring 2007 when Ludovic Court s contributed it after some initial discussion. I have been looking into getting back to do GnuTLS coding, and during a recent GnuTLS meeting one topic was Guile bindings. It seemed like a fairly self-contained project to pick up on. It is interesting to re-read the old thread when this work was included: some of the concerns brought up there now have track record to be evaluated on. My opinion that the cost of introducing a new project per language binding today is smaller than the cost of maintaining language bindings as part of the core project. I believe the cost/benefit ratio has changed during the past 15 years: introducing a new project used to come with a significant cost but this is no longer the case, as tooling and processes for packaging have improved. I have had similar experience with Java, C# and Emacs Lisp bindings for GNU Libidn as well, where maintaining them centralized slow down the pace of updates. Andreas Metzler pointed to a similar conclusion reached by Russ Allbery. There are many ways to separate a project into two projects; just copying the files into a new git repository would have been the simplest and was my original plan. However Ludo mentioned git-filter-branch in an email, and the idea of keeping all git history for some of the relevant files seemed worth pursuing to me. I quickly found git-filter-repo which appears to be the recommend approach, and experimenting with it I found a way to filter out the GnuTLS repo into a small git repository that Guile-GnuTLS could be based on. The commands I used were the following, if you want to reproduce things.

$ git clone https://gitlab.com/gnutls/gnutls.git guile-gnutls
$ cd guile-gnutls/
$ git checkout f5dcbdb46df52458e3756193c2a23bf558a3ecfd
$ git-filter-repo --path guile/ --path m4/guile.m4 --path doc/gnutls-guile.texi --path doc/extract-guile-c-doc.scm --path doc/cha-copying.texi --path doc/fdl-1.3.texi

I debated with myself back and forth whether to include some files that would be named the same in the new repository but would share little to no similar lines, for example configure.ac, Makefile.am not to mention README and NEWS. Initially I thought it would be nice to preserve the history for all lines that went into the new project, but this is a subjective judgement call. What brought me over to a more minimal approach was that the contributor history and attribution would be quite strange for the new repository: Should Guile-GnuTLS attribute the work of the thousands of commits to configure.ac which had nothing to do with Guile? Should the people who wrote that be mentioned as contributor of Guile-GnuTLS? I think not. The next step was to get a reasonable GitLab CI/CD pipeline up, to make sure the project builds on some free GNU/Linux distributions like Trisquel and PureOS as well as the usual non-free distributions like Debian and Fedora to have coverage of dpkg and rpm based distributions. I included builds on Alpine and ArchLinux as well, because they tend to trigger other portability issues. I wish there were GNU Guix docker images available for easy testing on that platform as well. The GitLab CI/CD rules for a project like this are fairly simple. To get things out of the door, I tagged the result as v3.7.9 and published a GitLab release page for Guile-GnuTLS that includes OpenPGP-signed source tarballs manually uploaded built on my laptop. The URLs for these tarballs are not very pleasant to work with, and discovering new releases automatically appears unreliable, but I don t know of a better approach. To finish this project, I have proposed a GnuTLS merge request to remove all Guile-related parts from the GnuTLS core. Doing some GnuTLS-related work again felt nice, it was quite some time ago so thank you for giving me this opportunity. Thoughts or comments? Happy hacking!

Gunnar Wolf: Learning some Rust with Lars!

A couple of weeks ago, I read a blog post by former Debian Developer Lars Wirzenius offering a free basic (6hr) course on the Rust language to interested free software and open source software programmers. I know Lars offers training courses in programming, and besides knowing him for ~20 years and being proud to consider us to be friends, have worked with him in a couple of projects (i.e. he is upstream for vmdb2, which I maintain in Debian and use for generating the Raspberry Pi Debian images) He is a talented programmer, and a fun guy to be around.

I was admitted to the first cohort of students of this course (please note I m not committing him to run free courses ever again! He has said he would consider doing so, specially to offer a different time better suited for people in Asia). I have wanted to learn some Rust for quite some time. About a year ago, I bought a copy of The Rust Programming Language, the canonical book for learning the language, and started reading it But lacked motivation and lost steam halfway through, and without having done even a simple real project beyond the simple book exercises. How has this been? I have enjoyed the course. I must admit I did expect it to be more hands-on from the beginning, but Rust is such a large language and it introduces so many new, surprising concepts. Session two did have two somewhat simple hands-on challenges; by saying they were somewhat simple does not mean we didn t have to sweat to get them to compile and work correctly! I know we will finish this Saturday, and I ll still be a complete newbie to Rust. I know the only real way to wrap my head around a language is to actually have a project that uses it And I have some ideas in mind. However, I don t really feel confident to approach an already existing project and start meddling with it, trying to contribute. What does Rust have that makes it so different? Bufff Variable ownership (borrow checking) and values lifetimes are the most obvious salient idea, but they are relatively simple, as you just cannot forget about them. But understanding (and adopting) idiomatic constructs such as the pervasive use of enums, understanding that errors always have to be catered for by using expect() and Result<T,E> It will take some time to be at ease developing in it, if I ever reach that stage! Oh, FWIW Interested related reading. I am halfway through an interesting article, published in March in the Communications of the ACM magazine, titled Here We Go Again: Why Is It Difficult for Developers to Learn Another Programming Language? , that presents an interesting point we don t always consider: If I m a proficient programmer in the X programming language and want to use the Y programming language, learning it Should be easier for me than for the casual bystander, or not? After all, I already have a background in programming! But it happens that mental constructs we build for a given language might hamper our learning of a very different one. This article presents three interesting research questions:

Does cross-language interference occur?
How do experienced programmers learn new languages?
What do experienced programmers find confusing in new languages?

I m far from reaching the conclusions, but so far, it s been a most interesting read. Anyway, to wrap up Thanks Lars! I am learning (although at a pace that is not magically quick But I am aware of the steep learning curve of the language) quite a bit of a very interesting topic, and I m also enjoying the time I spend in front of my computer on Saturday.

13 October 2022

Dirk Eddelbuettel: GitHub Streak: Round Nine

Eight years ago I referenced the Seinfeld Streak used in an earlier post of regular updates to to the Rcpp Gallery:

This is sometimes called Jerry Seinfeld s secret to productivity: Just keep at it. Don t break the streak.

and then showed the first chart of GitHub streaking 366 days:

github activity october 2013 to october 2014

And seven years ago a first follow-up appeared in this post about 731 days:

github activity october 2014 to october 2015

And six years ago we had a followup at 1096 days

github activity october 2015 to october 2016

And five years ago we had another one marking 1461 days

github activity october 2016 to october 2017

And four years ago another one for 1826 days

github activity october 2017 to october 2018

And three years ago another one bringing it to 2191 days

github activity october 2018 to october 2019

And two years ago another one bringing it to 2557 days

github activity october 2019 to october 2020

And last year another one bringing it to 2922 days

github activity october 2020 to october 2021

And as today is October 12 here is the newest one from 2021 to 2022 one bringing it 3287 days:

github activity october 2021 to october 2022

As always, special thanks go to Alessandro Pezz for the Chrome add-on GithubOriginalStreak. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

11 October 2022

Ben Hutchings: Debian LTS work, August-September 2022

I have continued to work for Freexian on Debian LTS. In August I carried over 21 hours from July, and worked 13 hours. In September I was assigned an additional 17 hours, and worked 16 hours. I will carry over 9 hours into October. In August, Debian 10 "buster" entered LTS status. I spent some time on the backport of Linux 5.10 for buster. While this previously existed in buster-backports, further changes were required to add it as an alternative kernel version in buster-security, particularly around code signing. When that was complete, I issued DLA-3102-1. I also prepared and uploaded an update to the linux (4.19) package. I issued DLA-3131-1 for these changes.

10 October 2022

Ian Jackson: Skipping releases when upgrading Debian systems

Debian does not officially support upgrading from earlier than the previous stable release: you re not supposed to skip releases. Instead, you re supposed to upgrade to each intervening major release in turn. However, skipping intervening releases does, in fact, often work quite well. Apparently, this is surprising to many people, even Debian insiders. I was encouraged to write about it some more. My personal experience I have three conventionally-managed personal server systems (by which I mean systems which aren t reprovisioned by some kind of automation). Of these at least two have been skip upgraded at least once: The one I don t think I ve skip-upgraded (at least, not recently) is my house network manager (and now VM host) which I try to keep to a minimum in terms of functionality and which I keep quite up to date. It was crossgraded from i386 (32-bit) to amd64 (64-bit) fairly recently, which is a thing that Debian isn t sure it supports. The crossgrade was done a hurry and without any planning, prompted by Spectre et al suddenly requiring big changes to Xen. But it went well enough. My home does random stuff server (media server, web cache, printing, DNS, backups etc.), has etckeeper records starting in 2015. I upgraded directly from jessie (Debian 8) to buster (Debian 10). I think it has probably had earlier skip upgrade(s): the oldest file in /etc is from December 1996 and I have been doing occasional skip upgrades as long as I can remember. And of course there s chiark, which is one of the oldest Debian installs in existence. I wrote about the most recent upgrade, where I went directly from jessie i386 ELTS (32-bit Debian 8) to bulleye amd64 (64-bit Debian 11). That was a very extreme case which required significant planning and pre-testing, since the package dependencies were in no way sufficient for the proper ordering. But, I don t normally go to such lengths. Normally, even on chiark, I just edit the sources.list and see what apt proposes to do. I often skip upgrade chiark because I tend to defer risky-looking upgrades partly in the hope of others fixing the bugs while I wait :-), and partly just because change is disruptive and amortising it is very helpful both to me and my users. I have some records of chiark s upgrades from my announcements to users. As well as the recent skip skip up cross grade, direct , I definitely did a skip upgrade of chiark from squeeze (Debian 6) to jessie (Debian 8). It appears that the previous skip upgrade on chiark was rex (Debian 1.2) to hamm (Debian 2.0). I don t think it s usual for me to choose to do a multi-release upgrade the officially supported way, in two (or more) stages, on a server. I have done that on systems with a GUI desktop setup, but even then I usually skip the intermediate reboot(s). When to skip upgrade (and what precautions to take) I m certainly not saying that everyone ought to be doing this routinely. Most users with a Debian install that is older than oldstable probably ought to reinstall it, or do the two-stage upgrade. Skip upgrading almost always runs into some kind of trouble (albeit, usually trouble that isn t particularly hard to fix if you know what you re doing). However, officially supported non-skip upgrades go wrong too. Doing a two-or-more-releases upgrade via the intermediate releases can expose you to significant bugs in the intermediate releases, which were later fixed. Because Debian s users and downstreams are cautious, and Debian itself can be slow, it is common for bugs to appear for one release and then be fixed only in the next. Paradoxically, this seems to be especially true with the kind of big and scary changes where you d naively think the upgrade mechanisms would break if you skipped the release where the change first came in. I would not recommend a skip upgrade to someone who is not a competent Debian administrator, with good familiarity with Debian package management, including use of dpkg directly to fix things up. You should have a mental toolkit of manual bug workaround techniques. I always should make sure that I have rescue media (and in the case of a remote system, full remote access including ability to boot a different image), although I don t often need it. And, when considering a skip upgrade, you should be aware of the major changes that have occurred in Debian. Skip upgrading is more likely to be a good idea with a complex and highly customised system: a fairly vanilla install is not likely to encounter problems during a two-stage update. (And, a vanilla system can be upgraded by reinstalling.) I haven t recently skip upgraded a laptop or workstation. I doubt I would attempt it; modern desktop software seems to take a much harder line about breaking things that are officially unsupported, and generally trying to force everyone into the preferred mold. A request to Debian maintainers I would like to encourage Debian maintainers to defer removing upgrade compatibility machinery until it is actually getting in the way, or has become hazardous, or many years obsolete. Examples of the kinds of things which it would be nice to keep, and which do not usually cause much inconvenience to retain, are dependency declarations (particularly, alternatives), and (many) transitional fragments in maintainer scripts. If you find yourself needing to either delete some compatibility feature, or refactor/reorganise it, I think it is probably best to delete it. If you modify it significantly, the resulting thing (which won t be tested until someone uses it in anger) is quite likely to have bugs which make it go wrong more badly (or, more confusingly) than the breakage that would happen without it. Obviously this is all a judgement call. I m not saying Debian should formally support skip upgrades, to the extent of (further) slowing down important improvements. Nor am I asking for any change to the routine approach to (for example) transitional packages (i.e. the technique for ensuring continuity of installation when a package name changes). We try to make release upgrades work perfectly; but skip upgrades don t have to work perfectly to be valuable. Retaining compatibility code can also make it easier to provide official backports, and it probably helps downstreams with different release schedules. The fact that maintainers do in practice often defer removing compatibility code provides useful flexibility and options to at least some people. So it would be nice if you d at least not go out of your way to break it. Building on older releases I would also like to encourage maintainers to provide source packages in Debian unstable that will still build on older releases, where this isn t too hard and the resulting binaries might be basically functional. Speaking personally, it s not uncommon for me to rebuild packages from unstable and install them on much older releases. This is another thing that is not officially supported, but which often works well. I m not saying to contort your build system, or delay progress. You ll definitely want to depend on a recentish debhelper. But, for example, retaining old build-dependency alternatives is nice. Retaining them doesn t constitute a promise that it works - it just makes life slightly easier for someone who is going off piste. If you know you have users on multiple distros or multiple releases, and wish to fully support them, you can go further, of course. Many of my own packages are directly buildable, or even directly installable, on older releases.

comments

Joachim Breitner: rec-def: Minesweeper case study

I m on the train back from MuniHac, where I gave a talk about the rec-def library that I have excessively blogged about recently (here, here, here and here). I got quite flattering comments about that talk, so if you want to see if they were sincere, I suggest you watch the recording of Getting recursive definitions off their bottoms (but it s not necessary for the following). After the talk, Franz Thoma approached me and told me a story of how we was once implementing the game Minesweeper in Haskell, and in particular the part of the logic where, after the user has uncovered a field, the game would automatically uncover all fields that are next to a neutral field, i.e. one with zero adjacent bombs. He was using a comonadic data structure, which makes a context-dependent parallel computation such as uncovering one field quite natural, and was hoping that using a suitable fix-point operator, he can elegantly obtain not just the next step, but directly the result of recursively uncovering all these fields. But, much to his disappointment, that did not work out: Due to the recursion inherent in that definition, a knot-tying fixed-point operator will lead to a cyclic definition.

He was wondering if the rec-def library could have helped him, and we sat down to find out, and this is the tale of this blog post. I will avoid the comonadic abstractions and program it more naively, though, to not lose too many readers along the way. Maybe read Chris Penner s blog post and Finch s functional pearl Getting a Quick Fix on Comonads if you are curious about that angle.

Minesweeper setup Let s start with defining a suitable data type for the grid of the minesweeper board. I ll use the Array data type, it s Ix-based indexing is quite useful for grids:

import Data.Array

type C = (Int,Int)
type Grid a = Array C a

The library lacks a function to generate an array from a generating function, but it is easy to add:

genArray :: Ix i => (i,i) -> (i -> e) -> Array i e
genArray r f = listArray r $ map f $ range r

Let s also fix the size of the board, as a pair of lower and upper bounds (this is the format that the Ix type class needs):

size :: (C,C)
size = ((0,0), (3,3))

Now board is simply a grid of boolean values, with True indicating that a bomb is there:

type Board = Grid Bool

board1 :: Board
board1 = listArray size
  [ False, False, False, False
  , True,  False, False, False
  , True,  False, False, False
  , False, False, False, False
  ]

It would be nice to be able to see these board in a nicer way. So let us write A function that prints a grid, including a frame, given a function that prints something for each coordinate. Together with a function that prints a bomb (as *), we can print the board:

pGrid :: (C -> String) -> String
pGrid p = unlines
    [ concat [ p' (y,x)   x <- [lx-1 .. ux+1] ]
      y  <- [ly-1 .. uy+1] ]
  where
    ((lx,ly),(ux,uy)) = size

    p' c   inRange size c = p c
    p'  _ = "#"

pBombs :: Board -> String
pBombs b = pGrid $ \c -> if b ! c then "*" else " "

The expression b ! c looks up a the coordinate in the array, and is True when there is a bomb at that coordinate. So here is our board, with two bombs:

ghci> putStrLn $ pBombs board1
######
#    #
#*   #
#*   #
#    #
######

But that s not what we want to show to the user: Every field should have have a number that indicates the number of bombs in the surrounding fields. To that end, we first define a function that takes a coordinate, and returns all adjacent coordinates. This also takes care of the border, using inRange:

neighbors :: C -> [C]
neighbors (x,y) =
    [ c
      (dx, dy) <- range ((-1,-1), (1,1))
    , (dx, dy) /= (0,0)
    , let c = (x + dx, y + dy)
    , inRange size c
    ]

With that, we can calculate what to display in each cell a bomb, or a number:

data H = Bomb   Hint Int deriving Eq

hint :: Board -> C -> H
hint b c   b ! c = Bomb
hint b c = Hint $ sum [ 1   c' <- neighbors c, b ! c' ]

With a suitable printing function, we can now see the full board:

hint :: Board -> C -> H
hint b c   b ! c = Bomb
hint b c = Hint $ sum [ 1   c' <- neighbors c, b ! c' ]

pCell :: Board -> C -> String
pCell b c = case hint b c of
    Bomb -> "*"
    Hint 0 -> " "
    Hint n -> show n

pBoard :: Board -> String
pBoard b = pGrid (pCell b)

And here it is:

ghci> putStrLn $ pBoard board1
######
#11  #
#*2  #
#*2  #
#11  #
######

Next we have to add masks: We need to keep track of which fields the user already sees. We again use a grid of booleans, and define a function to print a board with the masked fields hidden behind ?:

type Mask = Grid Bool

mask1 :: Mask
mask1 = listArray size
  [ True,  True,  True,  False
  , False, False, False, False
  , False, False, False, False
  , False, False, False, False
  ]

pMasked :: Board -> Mask -> String
pMasked b m = pGrid $ \c -> if m ! c then pCell b c else "?"

So this is what the user would see

ghci> putStrLn $ pMasked board1 mask1
######
#11 ?#
#????#
#????#
#????#
######

Uncovering some fields With that setup in place, we now implement the piece of logic we care about: Uncovering all fields that are next to a neutral field. Here is the first attempt:

solve0 :: Board -> Mask -> Mask
solve0 b m0 = m1
  where
    m1 :: Mask
    m1 = genArray size $ \c ->
      m0 ! c   or [ m0 ! c'   c' <- neighbors c, hint b c' == Hint 0 ]

The idea is that we calculate the new mask m1 from the old one m0 by the following logic: A field is visible if it was visible before (m0 ! c), or if any of its neighboring, neutral fields are visible. This works so far: I uncovered the three fields next to the one neutral visible field:

ghci> putStrLn $ pMasked board1 $ solve0 board1 mask1
######
#11  #
#?2  #
#????#
#????#
######

But that s not quite what we want: We want to keep doing that to uncover all fields.

Uncovering all fields So what happens if we change the logic to: A field is visible if it was visible before (m0 ! c), or if any of its neighboring, neutral fields will be visible. In the code, this is just a single character change: Instead of looking at m0 to see if a neighbor is visible, we look at m1:

solve1 :: Board -> Mask -> Mask
solve1 b m0 = m1
  where
    m1 :: Mask
    m1 = genArray size $ \c ->
      m0 ! c   or [ m1 ! c'   c' <- neighbors c, hint b c' == Hint 0 ]

(This is roughly what happened when Franz started to use the kfix comonadic fixed-point operator in his code, I believe.) Does it work? It seems so:

ghci> putStrLn $ pMasked board1 $ solve1 board1 mask1
######
#11  #
#?2  #
#?2  #
#?1  #
######

Amazing, isn t it! Unfortunately, it seems to work by accident. If I start with a different mask:

mask2 :: Mask
mask2 = listArray size
  [ True,  True,  False, False
  , False, False, False, False
  , False, False, False, False
  , False, False, False, True
  ]

which looks as follows:

ghci> putStrLn $ pMasked board1 mask2
######
#11??#
#????#
#????#
#??? #
######

Then our solve1 function does not work, and just sits there:

ghci> putStrLn $ pMasked board1 $ solve1 board1 mask2
######
#11^CInterrupted.

Why did it work before, but now now? It fails to work because as the code tries to figure out if a field, it needs to know if the next field will be uncovered. But to figure that out, it needs to know if the present field will be uncovered. With the normal boolean connectives ( and or), this does not make progress. It worked with mask1 more or less by accident: None of the fields on in the first column don t have neutral neighbors, so nothing happens there. And for all the fields in the third and forth column, the code will know for sure that they will be uncovered based on their upper neighbors, which come first in the neighbors list, and due to the short-circuting properties of , it doesn t have to look at the later cells, and the vicious cycle is avoided.

rec-def to the rescue This is where rec-def comes in: By using the RBool type in m1 instead of plain Bool, the recursive self-reference is not a problem, and it simply works:

import qualified Data.Recursive.Bool as RB

solve2 :: Board -> Mask -> Mask
solve2 b m0 = fmap RB.get m1
  where
    m1 :: Grid RB.RBool
    m1 = genArray size $ \c ->
      RB.mk (m0 ! c) RB.  RB.or [ m1 ! c'   c' <- neighbors c, hint b c' == Hint 0 ]

Note that I did not change the algorithm, or the self-reference through m1; I just replaced Bool with RBool, with RB. and or with RB.or. And used RB.get at the end to get a normal boolean out. And , here we go:

ghci> putStrLn $ pMasked board1 $ solve2 board1 mask2
######
#11  #
#?2  #
#?2  #
#?1  #
######

That s the end of this repetition of let s look at a tying-the-knot-problem and see how rec-def helps , which always end up a bit anti-climatic because it just works , at least in these cases. Hope you enjoyed it nevertheless.

9 October 2022

Jonathan Dowland: Focus writing with (despite) LaTeX

LaTeX the age-old typesetting system makes me angry. Not because it's bad. To clarify, not because there's something better. But because there should be. When writing a document using LaTeX, if you are prone to procrastination it can be very difficult to focus on the task at hand, because there are so many yaks to shave. Here's a few points of advice.

format the document source for legible reading. Yes, it's the input to the typesetter, and yes, the output of the typesetter needs to be legible. But it's worth making the input easy to read, too. Because
avoid rebuilding your rendered document too often. It's slow, it takes you out of the activity of writing, and it throws up lots of opportunities to get distracted by some rendering nit that you didn't realise would happen.
Unless you are very good at manoeuvring around long documents, liberally split them up. I think it's fine to have sections in their own source files.
Machine-assisted moving around documents is good. If you use (neo)vim, you can tweak exuberant-ctags to generate more useful tags for LaTeX documents than what you get OOTB, including jumping to \label s and the BibTeX source of \cite s. See this stackoverflow post.
If you use syntax highlighting in your editor, take a long, hard look at what it's drawing attention to. It's not your text, that's for sure. Is it worth having it on? Consider turning it off. Or (yak shaving beware!) tweak it to de-emphasise things, instead of emphasising them. One small example for (neo)vim, to change tokens recognised as being "todo" to match the styling used for comments (which is normally de-emphasised):
```
hi def link texTodo Comment
```

In a nutshell, I think it's wise to move much document reviewing work back into the editor rather than the rendered document, at least in the early stages of a section. And to do that, you need the document to be as legible as possible in the editor. The important stuff is the text you write, not the TeX macros you've sprinkled around to format it. A few tips I benefit from in terms of source formatting:

I stick a line of 78 '%' characters between each section and sub-section. This helps to visually break them up and finding them in a scroll-past is quicker.
I indent as much of the content as I can in each chapter/section/subsection (however deep I go in sections) to tie them to the section they belong to and see at a glance how deep I am in subsections, just like with source code. The exception is environments that I can't due to other tool limitations: I have code excerpts demarked by \begin code /\end code which are executed by Haskell's GHCi interpreter, and the indentation can interfere with Haskell's indentation rules.
For large documents (like a thesis), I have little helper "standalone" .tex files whose purpose is to let me build just one chapter or section at a time.
I'm fairly sure I'll settle on a serif font for my final document. But I have found that a sans-serif font is easier on my eyes on-screen. YMMV.

Of course, you need to review the rendered document too! I like to bounce that to a tablet with a pen/stylus/pencil and review it in a different environment to where I write. I then end up with a long list of scrawled notes, and a third distinct activity, back at the writing desk, is to systematically go through them and apply some GTD-style thinking to them: can I fix it in a few seconds? Do it straight away. Delegate it? Unlikely Defer it? transfer the review note into another system of record (such as LaTeX \\todo ). And finally

Don't stop to write a blog post about it.

Emmanuel Kasper: Markdown CMS or Wiki for smallish website

Following my markdown craze, I am slowly starting to move my Dokuwiki based homesite to Grav, a flat file Markdown CMS. PHP will be always be PHP, but the documentation and usage seem sound (all config is either via an admin panel or editing YAML files) and it has professional support. I intend to use this Debian based Dockerfile and Podman to deploy Grav. Pandoc the talented document converter, hat support for Dokuwiki syntax, Markdown and PHP Markdown extra, so I expected limited hurdles when converting the data.

Sergio Talens-Oliag: Shared networking for Virtual Machines and Containers

This entry explains how I have configured a linux bridge, dnsmasq and iptables to be able to run and communicate different virtualization systems and containers on laptops running Debian GNU/Linux. I ve used different variations of this setup for a long time with VirtualBox and KVM for the Virtual Machines and Linux-VServer, OpenVZ, LXC and lately Docker or Podman for the Containers.

Required packagesI m running Debian Sid with systemd and network-manager to configure the WiFi and Ethernet interfaces, but for the bridge I use bridge-utils with ifupdown (as I said this setup is old, I guess ifupdow2 and ifupdown-ng will work too). To start and stop the DNS and DHCP services and add NAT rules when the bridge is brought up or down I execute a script that uses:

ip from iproute2 to get the network information,
dnsmasq to provide the DNS and DHCP services (currently only the dnsmasq-base package is needed and it is recommended by network-manager, so it is probably installed),
iptables to configure NAT (for now docker kind of forces me to keep using iptables, but at some point I d like to move to nftables).

To make sure you have everything installed you can run the following command:

sudo apt install bridge-utils dnsmasq-base ifupdown iproute2 iptables

Bridge configurationThe bridge configuration for ifupdow is available on the file /etc/network/interfaces.d/vmbr0:

# Virtual servers NAT Bridge
auto vmbr0
iface vmbr0 inet static
    address         10.0.4.1
    network         10.0.4.0
    netmask         255.255.255.0
    broadcast       10.0.4.255
    bridge_ports    none
    bridge_maxwait  0
    up              /usr/local/sbin/vmbridge $ IFACE  start nat
    pre-down        /usr/local/sbin/vmbridge $ IFACE  stop nat

Warning: To use a separate file with ifupdown make sure that /etc/network/interfaces contains the line:

source /etc/network/interfaces.d/*

or add its contents to /etc/network/interfaces directly, if you prefer.

This configuration creates a bridge with the address 10.0.4.1 and assumes that the machines connected to it will use the 10.0.4.0/24 network; you can change the network address if you want, as long as you use a private range and it does not collide with networks used in your Virtual Machines all should be OK. The vmbridge script is used to start the dnsmasq server and setup the NAT rules when the interface is brought up and remove the firewall rules and stop the dnsmasq server when it is brought down.

The vmbridge scriptThe vmbridge script launches an instance of dnsmasq that binds to the bridge interface (vmbr0 in our case) that is used as DNS and DHCP server. The DNS server reads the /etc/hosts file to publish local DNS names and forwards all the other requests to the the dnsmasq server launched by NetworkManager that is listening on the loopback interface. As this server already does catching we disable it for our server, with the added advantage that, if we change networks, new requests go to the new resolvers because the DNS server handled by NetworkManager gets restarted and flushes its cache (this is useful if we connect to a new network that has internal DNS servers that are configured to do split DNS for internal services; if we use this model all requests get the internal address as soon as the DNS server is queried again). The DHCP server is configured to provide IPs to unknown hosts for a sub range of the addresses on the bridge network and use fixed IPs if the /etc/ethers file has a MAC with a matching hostname on the /etc/hosts file. To make things work with old DHCP clients the script also adds checksums to the DHCP packets using iptables (when the interface is not linked to a physical device the kernel does not add checksums, but we can fix it adding a rule on the mangle table). If we want external connectivity we can pass the nat argument and then the script creates a MASQUERADE rule for the bridge network and enables IP forwarding. The script source code is the following:

/usr/local/sbin/vmbridge

#!/bin/sh
set -e
# ---------
# VARIABLES
# ---------
LOCAL_DOMAIN="vmnet"
MIN_IP_LEASE="192"
MAX_IP_LEASE="223"
# ---------
# FUNCTIONS
# ---------
get_net()  
  NET="$(
    ip a ls "$ BRIDGE " 2>/dev/null   sed -ne 's/^.*inet \(.*\) brd.*$/\1/p'
  )"
  [ "$NET" ]   return 1
 
checksum_fix_start()  
  iptables -t mangle -A POSTROUTING -o "$ BRIDGE " -p udp --dport 68 \
    -j CHECKSUM --checksum-fill 2>/dev/null   true
 
checksum_fix_stop()  
  iptables -t mangle -D POSTROUTING -o "$ BRIDGE " -p udp --dport 68 \
    -j CHECKSUM --checksum-fill 2>/dev/null   true
 
nat_start()  
  [ "$NAT" = "yes" ]   return 0
  # Configure NAT
  iptables -t nat -A POSTROUTING -s "$ NET " ! -d "$ NET " -j MASQUERADE
  # Enable forwarding (just in case)
  echo 1 >/proc/sys/net/ipv4/ip_forward
 
nat_stop()  
  [ "$NAT" = "yes" ]   return 0
  iptables -t nat -D POSTROUTING -s "$ NET " ! -d "$ NET " \
    -j MASQUERADE 2>/dev/null   true
 
do_start()  
  # Bridge address
  _addr="$ NET%%/* "
  # DNS leases (between .MIN_IP_LEASE and .MAX_IP_LEASE)
  _dhcp_range="$ _addr%.* .$ MIN_IP_LEASE ,$ _addr%.* .$ MAX_IP_LEASE "
  # Bridge mtu
  _mtu="$(
    ip link show dev "$ BRIDGE "  
      sed -n -e '/mtu/   s/^.*mtu \([0-9]\+\).*$/\1/p  '
  )"
  # Compute extra dnsmasq options
  dnsmasq_extra_opts=""
  # Disable gateway when not using NAT
  if [ "$NAT" != "yes" ]; then
    dnsmasq_extra_opts="$dnsmasq_extra_opts --dhcp-option=3"
  fi
  # Adjust MTU size if needed
  if [ -n "$_mtu" ] && [ "$_mtu" -ne "1500" ]; then
    dnsmasq_extra_opts="$dnsmasq_extra_opts --dhcp-option=26,$_mtu"
  fi
  # shellcheck disable=SC2086
  dnsmasq --bind-interfaces \
    --cache-size="0" \
    --conf-file="/dev/null" \
    --dhcp-authoritative \
    --dhcp-leasefile="/var/lib/misc/dnsmasq.$ BRIDGE .leases" \
    --dhcp-no-override \
    --dhcp-range "$ _dhcp_range " \
    --domain="$ LOCAL_DOMAIN " \
    --except-interface="lo" \
    --expand-hosts \
    --interface="$ BRIDGE " \
    --listen-address "$ _addr " \
    --no-resolv \
    --pid-file="$ PIDF " \
    --read-ethers \
    --server="127.0.0.1" \
    $dnsmasq_extra_opts
  checksum_fix_start
  nat_start
 
do_stop()  
  nat_stop
  checksum_fix_stop
  if [ -f "$ PIDF " ]; then
    kill "$(cat "$ PIDF ")"   true
    rm -f "$ PIDF "
  fi
 
do_status()  
  if [ -f "$ PIDF " ] && kill -HUP "$(cat "$ PIDF ")"; then
    echo "dnsmasq RUNNING"
  else
    echo "dnsmasq NOT running"
  fi
 
do_reload()  
  [ -f "$ PIDF " ] && kill -HUP "$(cat "$ PIDF ")"
 
usage()  
  echo "Uso: $0 BRIDGE (start stop [nat]) status reload"
  exit 1
 
# ----
# MAIN
# ----
[ "$#" -ge "2" ]   usage
BRIDGE="$1"
OPTION="$2"
shift 2
NAT="no"
for arg in "$@"; do
  case "$arg" in
  nat) NAT="yes" ;;
  *) echo "Unknown arg '$arg'" && exit 1 ;;
  esac
done
PIDF="/var/run/vmbridge-$ BRIDGE -dnsmasq.pid"
case "$OPTION" in
start) get_net && do_start ;;
stop) get_net && do_stop ;;
status) do_status ;;
reload) get_net && do_reload ;;
*) echo "Unknown command '$OPTION'" && exit 1 ;;
esac
# vim: ts=2:sw=2:et:ai:sts=2

NetworkManager ConfigurationThe default /etc/NetworkManager/NetworkManager.conf file has the following contents:

[main]
plugins=ifupdown,keyfile
[ifupdown]
managed=false

Which means that it will leave interfaces managed by ifupdown alone and, by default, will send the connection DNS configuration to systemd-resolved if it is installed. As we want to use dnsmasq for DNS resolution, but we don t want NetworkManager to modify our /etc/resolv.conf we are going to add the following file (/etc/NetworkManager/conf.d/dnsmasq.conf) to our system:

/etc/NetworkManager/conf.d/dnsmasq.conf

[main]
dns=dnsmasq
rc-manager=unmanaged

and restart the NetworkManager service:

$ sudo systemctl restart NetworkManager.service

From now on the NetworkManager will start a dnsmasq service that queries the servers provided by the DHCP servers we connect to on 127.0.0.1:53 but will not touch our /etc/resolv.conf file.

Configuring systemd-resolvedIf we start using our own name server but our system has systemd-resolved installed we will no longer need or use the DNS stub; programs using it will use our dnsmasq server directly now, but we keep running systemd-resolved for the host programs that use its native api or access it through /etc/nsswitch.conf (when libnss-resolve is installed). To disable the stub we add a /etc/systemd/resolved.conf.d/disable-stub.conf file to our machine with the following content:

# Disable the DNS Stub Listener, we use our own dnsmasq
[Resolve]
DNSStubListener=no

and restart the systemd-resolved to make sure that the stub is stopped:

$ sudo systemctl restart systemd-resolved.service

Adjusting /etc/resolv.confFirst we remove the existing /etc/resolv.conf file (it does not matter if it is a link or a regular file) and then create a new one that contains at least the following line (we can add a search line if is useful for us):

nameserver 10.0.4.1

From now on we will be using the dnsmasq server launched when we bring up the vmbr0 for multiple systems:

as our main DNS server from the host (if we use the standard /etc/nsswitch.conf and libnss-resolve is installed it is queried first, but the systemd-resolved uses it as forwarder by default if needed),
as the DNS server of the Virtual Machines or containers that use DHCP for network configuration and attach their virtual interfaces to our bridge,
as the DNS server of docker containers that get the DNS information from /etc/resolv.conf (if we have entries that use loopback addresses the containers that don t use the host network tend to fail, as those addresses inside the running containers are not linked to the loopback device of the host).

TestingAfter all the configuration files and scripts are in place we just need to bring up the bridge interface and check that everything works:

$ # Bring interface up
$ sudo ifup vmbr0
$ # Check that it is available
$ ip a ls dev vmbr0
4: vmbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
          group default qlen 1000
    link/ether 0a:b8:ef:b8:07:6c brd ff:ff:ff:ff:ff:ff
    inet 10.0.4.1/24 brd 10.0.4.255 scope global vmbr0
       valid_lft forever preferred_lft forever
$ # View the listening ports used by our dnsmasq servers
$ sudo ss -tulpan   grep dnsmasq
udp UNCONN 0 0  127.0.0.1:53     0.0.0.0:* users:(("dnsmasq",pid=1733930,fd=4))
udp UNCONN 0 0  10.0.4.1:53      0.0.0.0:* users:(("dnsmasq",pid=1705267,fd=6))
udp UNCONN 0 0  0.0.0.0%vmbr0:67 0.0.0.0:* users:(("dnsmasq",pid=1705267,fd=4))
tcp LISTEN 0 32 10.0.4.1:53      0.0.0.0:* users:(("dnsmasq",pid=1705267,fd=7))
tcp LISTEN 0 32 127.0.0.1:53     0.0.0.0:* users:(("dnsmasq",pid=1733930,fd=5))
$ # Verify that the DNS server works on the vmbr0 address
$ host www.debian.org 10.0.4.1
Name: 10.0.4.1
Address: 10.0.4.1#53
Aliases:
www.debian.org has address 130.89.148.77
www.debian.org has IPv6 address 2001:67c:2564:a119::77

Managing running systemsIf we want to update DNS entries and/or MAC addresses we can edit the /etc/hosts and /etc/ethers files and reload the dnsmasq configuration using the vmbridge script:

$ sudo /usr/local/sbin/vmbridge vmbr0 reload

That call sends a signal to the running dnsmasq server and it reloads the files; after that we can refresh the DHCP addresses from the client machines or start using the new DNS names immediately.

8 October 2022

Russ Allbery: Tie::ShadowHash 2.01

Tie::ShadowHash is a small Perl module that allows one to stack an in-memory modifiable hash on top of a read-only hash obtained from anywhere you can get a hash in Perl (including a tied hash), functioning much like an overlay file system with the same benefits. This release just (hopefully) fixes a few test suite failures, and is probably not worth announcing except that it bugs me to not announce each software release. You can get the latest version from CPAN or the Tie::ShadowHash distribution page.

Steve Kemp: Trivial benchmarks of toy languages

Over the past few months (years?) I've posted on my blog about the various toy interpreters I've written. I've used a couple of scripting languages/engines in my professional career, but in public I think I've implemented

BASIC
- https://github.com/skx/gobasic
evalfilter
- https://github.com/skx/evalfilter
- (Basically the same as the bytecode VM of Monkey).
FORTH
- https://github.com/skx/foth
Lisp
- https://github.com/skx/yal
Monkey
- https://github.com/skx/monkey
TCL
- https://github.com/skx/critical

Each of these works in similar ways, and each of these filled a minor niche, or helped me learn something new. But of course there's always a question:

Which is fastest?

In the real world? It just doesn't matter. For me. But I was curious, so I hacked up a simple benchmark of calculating 12! (i.e. The factorial of 12). The specific timings will vary based on the system which runs the test(s), but there's no threading involved so the relative performance is probably comparable. Anyway the benchmark is simple, and I did it "fairly". By that I mean that I didn't try to optimize any particular test-implementation, I just wrote it in a way that felt natural. The results? Evalfilter wins, because it compiles the program into bytecode, which can be executed pretty quickly. But I was actually shocked ("I wrote a benchmark; The results will blow your mind!") at the second and third result:

BenchmarkEvalFilterFactorial-4      61542     17458 ns/op
BenchmarkFothFactorial-4            44751     26275 ns/op
BenchmarkBASICFactorial-4           36735     32090 ns/op
BenchmarkMonkeyFactorial-4          14446     85061 ns/op
BenchmarkYALFactorial-4              2607    456757 ns/op
BenchmarkTCLFactorial-4               292   4085301 ns/op

here we see that FOTH, my FORTH implementation, comes second. I guess this is an efficient interpreter too, bacause that too is essentially "bytecode". (Looking up words in a dictionary, which really maps to indexes to other words. The stack operations are reasonably simple and fast too.) Number three? BASIC? I expected better from the other implementations to be honest. BASIC doesn't even use an AST (in my implementation), just walks tokens. I figured the TCO implemented by my lisp would make that number three. Anyway the numbers mean nothing. Really. But still interesting.

7 October 2022

Reproducible Builds: Reproducible Builds in September 2022

Welcome to the September 2022 report from the Reproducible Builds project! In our reports we try to outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. If you are interested in contributing to the project, please visit our Contribute page on our website.

David A. Wheeler reported to us that the US National Security Agency (NSA), Cybersecurity and Infrastructure Security Agency (CISA) and the Office of the Director of National Intelligence (ODNI) have released a document called Securing the Software Supply Chain: Recommended Practices Guide for Developers (PDF). As David remarked in his post to our mailing list, it expressly recommends having reproducible builds as part of advanced recommended mitigations . The publication of this document has been accompanied by a press release.

Holger Levsen was made aware of a small Microsoft project called oss-reproducible. Part of, OSSGadget, a larger collection of tools for analyzing open source packages , the purpose of oss-reproducible is to:

analyze open source packages for reproducibility. We start with an existing package (for example, the NPM left-pad package, version 1.3.0), and we try to answer the question, Do the package contents authentically reflect the purported source code?

More details can be found in the README.md file within the code repository.

David A. Wheeler also pointed out that there are some potential upcoming changes to the OpenSSF Best Practices badge for open source software in relation to reproducibility. Whilst the badge programme has three certification levels ( passing , silver and gold ), the gold level includes the criterion that The project MUST have a reproducible build . David reported that some projects have argued that this reproducibility criterion should be slightly relaxed as outlined in an issue on the best-practices-badge GitHub project. Essentially, though, the claim is that the reproducibility requirement doesn t make sense for projects that do not release built software, and that timestamp differences by themselves don t necessarily indicate malicious changes. Numerous pragmatic problems around excluding timestamps were raised in the discussion of the issue.

Sonatype, a pioneer of software supply chain management , issued a press release month to report that they had found:

[ ] a massive year-over-year increase in cyberattacks aimed at open source project ecosystems. According to early data from Sonatype s 8th annual State of the Software Supply Chain Report, which will be released in full this October, Sonatype has recorded an average 700% jump in repository attacks over the last three years.

More information is available in the press release.

A number of changes were made to the Reproducible Builds website and documentation this month, including Chris Lamb adding a redirect from /projects/ to /who/ in order to keep old or archived links working [ ], Jelle van der Waa added a Rust programming language example for SOURCE_DATE_EPOCH [ ][ ] and Mattia Rizzolo included Protocol Labs amongst our project-level sponsors [ ].

Debian There was a large amount of reproducibility work taking place within Debian this month:

The `nfft` source package was removed from the archive, and now all packages in Debian bookworm now have a corresponding `.buildinfo` file. This can be confirmed and tracked on the associated page on the tests.reproducible-builds.org site.

Vagrant Cascadian announced on our mailing list an informal online sprint to help clear the huge backlog of reproducible builds patches submitted by performing NMU (Non-Maintainer Uploads). The first such sprint took place on September 22nd with the following results:

Holger Levsen:

Mailed #1010957 in `man-db` asking for an update and whether to remove the patch tag for now. This was subsequently removed and the maintainer started to address the issue.

Uploaded `gmp` to `DELAYED/15`, fixing #1009931.

Emailed #1017372 in `plymouth` and asked for the maintainer s opinion on the patch. This resulted in the maintainer improving Vagrant s original patch (and uploading it) as well as filing an issue upstream.

Uploaded `time` to `DELAYED/15`, fixing #983202.

Vagrant Cascadian:

Verify and updated patch for `mylvmbackup` (#782318)

Verified/updated patches for `libranlip`. (#788000, #846975 & #1007137)

Uploaded `libranlip` to `DELAYED/10`.

Verified patch for `cclive`. (#824501)

Uploaded `cclive` to `DELAYED/10`.

Vagrant was unable to reproduce the underlying issue within #791423 (`linuxtv-dvb-apps`) and so the bug was marked as done .

Researched #794398 (in `clhep`).

The plan is to repeat these sprints every two weeks, with the next taking place on Thursday October 6th at 16:00 UTC on the `#debian-reproducible` IRC channel.

Roland Clobus posted his 13th update of the status of reproducible Debian ISO images on our mailing list. During the last month, Roland ensured that the live images are now automatically fed to openQA for automated testing after they have been shown to be reproducible. Additionally Roland asked on the debian-devel mailing list about a way to determine the canonical timestamp of the Debian archive. [ ]

Following up on last month s work on reproducible bootstrapping, Holger Levsen filed two bugs against the debootstrap and cdebootstrap utilities. (#1019697 & #1019698)

Lastly, 44 reviews of Debian packages were added, 91 were updated and 17 were removed this month adding to our knowledge about identified issues. A number of issue types have been updated too, including the descriptions of `cmake_rpath_contains_build_path` [ ], `nondeterministic_version_generated_by_python_param` [ ] and `timestamps_in_documentation_generated_by_org_mode` [ ]. Furthermore, two new issue types were created: `build_path_used_to_determine_version_or_package_name` [ ] and `captures_build_path_via_cmake_variables` [ ].

Other distributions In openSUSE, Bernhard M. Wiedemann published his usual openSUSE monthly report.

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions `222` and `223` to Debian, as well as made the following changes:

The `cbfstools` utility is now provided in Debian via the `coreboot-utils` package so we can enable that functionality within Debian. [ ]

Looked into Mach-O support.

Fixed the try.diffoscope.org service by addressing a compatibility issue between `glibc`/`seccomp` that was preventing the Docker-contained diffoscope instance from spawning any external processes whatsoever [ ]. I also updated the `requirements.txt` file, as some of the specified packages were no longer available [ ][ ].

In addition Jelle van der Waa added support for `file` version 5.43 [ ] and Mattia Rizzolo updated the packaging:

Also include `coreboot-utils` in the `Build-Depends` and `Test-Depends` fields so that it is available for tests. [ ]

Use pep517 and pip to load the requirements. [ ]

Remove packages in `Breaks`/`Replaces` that have been obsoleted since the release of Debian bullseye. [ ]

Reprotest reprotest is our end-user tool to build the same source code twice in widely and deliberate different environments, and checking whether the binaries produced by the builds have any differences. This month, reprotest version `0.7.22` was uploaded to Debian unstable by Holger Levsen, which included the following changes by Philip Hands:

Actually ensure that the `setarch(8)` utility can actually execute before including an architecture to test. [ ]

Include all files matching `.deb` in the default `artifact_pattern` in order to archive all results of the build. [ ]

Emit an error when building the Debian package if the Debian packaging version does not patch the Python version of reprotest. [ ]

Remove an unneeded invocation of the `head(1)` utility. [ ]

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann (18 bugs):

`DateTime` (fails to build in 2038)

`FreeRCT` (date-related issue)

`clanlib1` (filesystem ordering)

`cli` (fails to build in 2038)

`deepin-gettext-tools` (patch+version update toolchain sort python glob)

`mariadb` (fails to build in 2038)

`mercurial` (fails to build in 2038)

`mirrormagic` (parallelism-related issue)

`ocaml-extlib` (parallelism-related issue)

`python-xmlrpc/python-softlayer` (fails to build in 2038)

`python` (fails to build in 2038)

`q3rally` (zip-related issue)

`rnd_jue` (parallelism-related issue)

`rsync` (workaround an issue in GCC 7.x)

`scons` (`SOURCE_DATE_EPOCH`-related issue)

`stratagus` (date-related issue)

`triplane` (nondeterminism caused by uninitialised memory)

`tyrquake` (date-related issue)

Chris Lamb:

#1019382 filed against `gnome-online-accounts`.

There was renewed activity on a reproducibility-related bug in the Sphinx documentation tool this month. Originally filed in October 2021 by Chris Lamb, the bug in question relates to contents of the `LANGUAGE` environment variable inconsistently affecting the output of `objects.inv` files.

Jelle van der Waa:

`mp4v2` (date-related issue)

`mm-common` (uid/gid issue)

`aardvark-dns` (date-related issue)

Vagrant Cascadian (70 bugs!):

#1020648 filed against `extrepo-data`.

#1020650 filed against `tmpreaper`.

#1020651 filed against `xmlrpc-epi`.

#1020653 filed against `pal`.

#1020656 filed against `nvram-wakeup`.

#1020657 filed against `netris`.

#1020658 filed against `netpbm-free`.

#1020659 filed against `lookup`.

#1020660 filed against `logtools`.

#1020661 filed against `libid3tag`.

#1020662 filed against `log4cpp`.

#1020665 filed against `libimage-imlib2-perl`.

#1020668 filed against `jnettop`.

#1020670 filed against `gwaei`.

#1020671 filed against `ipfm`.

#1020672 filed against `tarlz`.

#1020673 filed against `w3cam`.

#1020674 filed against `ifstat`.

#1020715 filed against `xserver-xorg-input-joystick`.

#1020719 filed against `chibicc`.

#1020723 filed against `python-omegaconf`.

#1020724 and #1020725 filed against `snapper`.

#1020736 filed against `libreswan`.

#1020743 filed against `pure-ftpd`.

#1020748 filed against `xcolmix`.

#1020749 filed against `gigalomania`.

#1020750 filed against `xjump`.

#1020751 filed against `waili`.

#1020752 filed against `sjeng`.

#1020753 filed against `seqtk`.

#1020754 filed against `shapetools`.

#1020755 filed against `rotter`.

#1020756 filed against `rakarrack`.

#1020757 filed against `rig`.

#1020759 filed against `postal`.

#1020798 filed against `netkit-rsh`.

#1020800 filed against `libapache-mod-evasive`.

#1020804 filed against `paxctl`.

#1020805 filed against `png23d`.

#1020806 filed against `perl-byacc`.

#1020807 filed against `poster`.

#1020808 filed against `powerdebug`.

#1020809 filed against `aespipe`.

#1020810 filed against `aewm++-goodies`.

#1020811 filed against `apache-upload-progress-module`.

#1020812 filed against `ascii2binary`.

#1020813 filed against `bible-kjv`.

#1020814 filed against `dradio`.

#1020815 filed against `libapache2-mod-python`.

#1020816 filed against `tempest-for-eliza`.

#1020817 filed against `aplus-fsf`.

#1020866 filed against `wrapsrv`.

#1020867 filed against `uclibc`.

#1020870 filed against `xppaut`.

#1020872 filed against `xvier`.

#1020873 filed against `xserver-xorg-video-glide`.

#1020875 filed against `z80asm`.

#1020876 filed against `yaskkserv`.

#1020877 filed against `edid-decode`.

#1020878 filed against `dustmite`.

#1020879 filed against `dustmite`.

#1020880 filed against `libapache2-mod-authnz-pam`.

#1020881 filed against `kafs-client`.

#1020882 filed against `yaku-ns`.

#1020884 filed against `bplay`.

#1020886 filed against `chise-base`.

#1020887 filed against `checkpw`.

#1020888 filed against `clamz`.

#1020889 filed against `libapache2-mod-auth-pgsql`.

Testing framework The Reproducible Builds project runs a significant testing framework at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. This month, however, the following changes were made:

Holger Levsen:

Add a job to build reprotest from Git [ ] and use the correct Git branch when building it [ ].

Mattia Rizzolo:

Enable syncing of results from building live Debian ISO images. [ ]

Use `scp -p` in order to preserve modification times when syncing live ISO images. [ ]

Apply the shellcheck shell script analysis tool. [ ]

In a build node wrapper script, remove some debugging code which was messing up calling `scp(1)` correctly [ ] and consquently add support to use both `scp -p` and regular `scp` [ ].

Roland Clobus:

Track and handle the case where the Debian archive gets updated between two live image builds. [ ]

Remove a call to `sudo(1)` as it is not (or no longer) required to delete old live-build results. [ ]

Contact As ever, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mailing list: `rb-general@lists.reproducible-builds.org`

Edward Betts: Fish shell now has underscore as a number separator (my feature request)

In November 2021 I filed a feature request for the fish shell to add underscore as a thousand separator in numbers. My feature request has been implemented and is available in fish 3.5.0, released 16 June 2022. The fish shell supports mathematical operations using the math command.

edward@x1c9 ~> math 2_000 + 22
2022
edward@x1c9 ~>

The underscore can be used as a thousand separator, but there are other uses for a number separator. Here's a list taken from a post by Mathias Bynens about the number separator in JavaScript:

// A decimal integer literal with its digits grouped per thousand:
1_000_000_000_000
// A decimal literal with its digits grouped per thousand:
1_000_000.220_720
// A binary integer literal with its bits grouped per octet:
0b01010110_00111000
// A binary integer literal with its bits grouped per nibble:
0b0101_0110_0011_1000
// A hexadecimal integer literal with its digits grouped by byte:
0x40_76_38_6A_73
// A BigInt literal with its digits grouped per thousand:
4_642_473_943_484_686_707n

Programming languages are gradually adding a number separator to their syntax, I think Perl was the first. Most are languages use underscore, but C++ 14 uses an apostrophe for the number separator.

Reproducible Builds (diffoscope): diffoscope 224 released

The diffoscope maintainers are pleased to announce the release of diffoscope version 224. This version includes the following changes:

[ Mattia Rizzolo ]
* Fix rlib test failure with LLVM 15. Thanks to Gianfranco Costamagna
  (locutusofborg) for the patch.

You find out more by visiting the project homepage.

6 October 2022

Dirk Eddelbuettel: Rblpapi 0.3.14: Updates and Extensions

Version 0.3.14 of the Rblpapi package arrived on CRAN earlier today. Rblpapi provides a direct interface between R and the Bloomberg Terminal via the C++ API provided by Bloomberg (but note that a valid Bloomberg license and installation is required). This is the fourteenth release since the package first appeared on CRAN in 2016. It comprises a nice PR from Robert Harlow extending support to B-PIPE authentication (for those who have it) along with a few fixes made since the last release in January. The last one provided from a kind assist by Tomas Kalibera who pointed out how to overcome an absolute rpath dynamic linker instruction (and as I noticed noticed something I already did in another package ah well) so that we no longer require StagedInstall: yes. The detailed list of changes follow below.

Changes in Rblpapi version 0.3.14 (2022-10-05)

Build configuration was generalized to consider local copies of library and headers (Dirk in #362)

A ticker symbol was corrected (Dirk in #368 addressing an issue #366 and #367)

Support for B-PIPE was added (Robert Harlow in #369 closing #342)

The package no longer requires staged installation thanks to an assist from Tomas Kalibera (Dirk in #373)

The retired package fts is no longer suggested (Dirk in #374 closing #372)

Courtesy of my CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the Rblpapi page. Questions, comments etc should go to the issue tickets system at the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Matthew Garrett: Cloud desktops aren't as good as you'd think

Fast laptops are expensive, cheap laptops are slow. But even a fast laptop is slower than a decent workstation, and if your developers want a local build environment they're probably going to want a decent workstation. They'll want a fast (and expensive) laptop as well, though, because they're not going to carry their workstation home with them and obviously you expect them to be able to work from home. And in two or three years they'll probably want a new laptop and a new workstation, and that's even more money. Not to mention the risks associated with them doing development work on their laptop and then drunkenly leaving it in a bar or having it stolen or the contents being copied off it while they're passing through immigration at an airport. Surely there's a better way?

This is the thinking that leads to "Let's give developers a Chromebook and a VM running in the cloud". And it's an appealing option! You spend far less on the laptop, and the VM is probably cheaper than the workstation - you can shut it down when it's idle, you can upgrade it to have more CPUs and RAM as necessary, and you get to impose all sorts of additional neat security policies because you have full control over the network. You can run a full desktop environment on the VM, stream it to a cheap laptop, and get the fast workstation experience on something that weighs about a kilogram. Your developers get the benefit of a fast machine wherever they are, and everyone's happy.

But having worked at more than one company that's tried this approach, my experience is that very few people end up happy. I'm going to give a few reasons here, but I can't guarantee that they cover everything - and, to be clear, many (possibly most) of the reasons I'm going to describe aren't impossible to fix, they're simply not priorities. I'm also going to restrict this discussion to the case of "We run a full graphical environment on the VM, and stream that to the laptop" - an approach that only offers SSH access is much more manageable, but also significantly more restricted in certain ways. With those details mentioned, let's begin.

The first thing to note is that the overall experience is heavily tied to the protocol you use for the remote display. Chrome Remote Desktop is extremely appealing from a simplicity perspective, but is also lacking some extremely key features (eg, letting you use multiple displays on the local system), so from a developer perspective it's suboptimal. If you read the rest of this post and want to try this anyway, spend some time working with your users to find out what their requirements are and figure out which technology best suits them.

Second, let's talk about GPUs. Trying to run a modern desktop environment without any GPU acceleration is going to be a miserable experience. Sure, throwing enough CPU at the problem will get you past the worst of this, but you're still going to end up with users who need to do 3D visualisation, or who are doing VR development, or who expect WebGL to work without burning up every single one of the CPU cores you so graciously allocated to their VM. Cloud providers will happily give you GPU instances, but that's going to cost more and you're going to need to re-run your numbers to verify that this is still a financial win. "But most of my users don't need that!" you might say, and we'll get to that later on.

Next! Video quality! This seems like a trivial point, but if you're giving your users a VM as their primary interface, then they're going to do things like try to use Youtube inside it because there's a conference presentation that's relevant to their interests. The obvious solution here is "Do your video streaming in a browser on the local system, not on the VM" but from personal experience that's a super awkward pain point! If I click on a link inside the VM it's going to open a browser there, and now I have a browser in the VM and a local browser and which of them contains the tab I'm looking for WHO CAN SAY. So your users are going to watch stuff inside their VM, and re-compressing decompressed video is going to look like shit unless you're throwing a huge amount of bandwidth at the problem. And this is ignoring the additional irritation of your browser being unreadable while you're rapidly scrolling through pages, or terminal output from build processes being a muddy blur of artifacts, or the corner case of "I work for Youtube and I need to be able to examine 4K streams to determine whether changes have resulted in a degraded experience" which is a very real job and one that becomes impossible when you pass their lovingly crafted optimisations through whatever codec your remote desktop protocol has decided to pick based on some random guesses about the local network, and look everyone is going to have a bad time.

The browser experience. As mentioned before, you'll have local browsers and remote browsers. Do they have the same security policy? Who knows! Are all the third party services you depend on going to be ok with the same user being logged in from two different IPs simultaneously because they lost track of which browser they had an open session in? Who knows! Are your users going to become frustrated? Who knows oh wait no I know the answer to this one, it's "yes".

Accessibility! More of your users than you expect rely on various accessibility interfaces, be those mechanisms for increasing contrast, screen magnifiers, text-to-speech, speech-to-text, alternative input mechanisms and so on. And you probably don't know this, but most of these mechanisms involve having accessibility software be able to introspect the UI of applications in order to provide appropriate input or expose available options and the like. So, I'm running a local text-to-speech agent. How does it know what's happening in the remote VM? It doesn't because it's just getting an a/v stream, so you need to run another accessibility stack inside the remote VM and the two of them are unaware of each others existence and this works just as badly as you'd think. Alternative input mechanism? Good fucking luck with that, you're at best going to fall back to "Send synthesized keyboard inputs" and that is nowhere near as good as "Set the contents of this text box to this unicode string" and yeah I used to work on accessibility software maybe you can tell. And how is the VM going to send data to a braille output device? Anyway, good luck with the lawsuits over arbitrarily making life harder for a bunch of members of a protected class.

One of the benefits here is supposed to be a security improvement, so let's talk about WebAuthn. I'm a big fan of WebAuthn, given that it's a multi-factor authentication mechanism that actually does a good job of protecting against phishing, but if my users are running stuff inside a VM, how do I use it? If you work at Google there's a solution, but that does mean limiting yourself to Chrome Remote Desktop (there are extremely good reasons why this isn't generally available). Microsoft have apparently just specced a mechanism for doing this over RDP, but otherwise you're left doing stuff like forwarding USB over IP, and that means that your USB WebAuthn no longer works locally. It also doesn't work for any other type of WebAuthn token, such as a bluetooth device, or an Apple TouchID sensor, or any of the Windows Hello support. If you're planning on moving to WebAuthn and also planning on moving to remote VM desktops, you're going to have a bad time.

That's the stuff that comes to mind immediately. And sure, maybe each of these issues is irrelevant to most of your users. But the actual question you need to ask is what percentage of your users will hit one or more of these, because if that's more than an insignificant percentage you'll still be staffing all the teams that dealt with hardware, handling local OS installs, worrying about lost or stolen devices, and the glorious future of just being able to stop worrying about this is going to be gone and the financial benefits you promised would appear are probably not going to work out in the same way.

A lot of this falls back to the usual story of corporate IT - understand the needs of your users and whether what you're proposing actually meets them. Almost everything I've described here is a corner case, but if your company is larger than about 20 people there's a high probability that at least one person is going to fall into at least one of these corner cases. You're going to need to spend a lot of time understanding your user population to have a real understanding of what the actual costs are here, and I haven't seen anyone do that work before trying to launch this and (inevitably) going back to just giving people actual computers.

There are alternatives! Modern IDEs tend to support SSHing out to remote hosts to perform builds there, so as long as you're ok with source code being visible on laptops you can at least shift the "I need a workstation with a bunch of CPU" problem out to the cloud. The laptops are going to need to be more expensive because they're also going to need to run more software locally, but it wouldn't surprise me if this ends up being cheaper than the full-on cloud desktop experience in most cases.

Overall, the most important thing to take into account here is that your users almost certainly have more use cases than you expect, and this sort of change is going to have direct impact on the workflow of every single one of your users. Make sure you know how much that's going to be, and take that into consideration when suggesting it'll save you money.

comments

Next.

Previous.

Search Results: "ag"

16 October 2022

Thanks This work is currently crowd-funded on github sponsors. I d like to thank @SantiagoTorres, @repi and @rgacogne for their support in particular.

15 October 2022

14 October 2022

13 October 2022

11 October 2022

10 October 2022

9 October 2022

Warning: To use a separate file with ifupdown make sure that /etc/network/interfaces contains the line: source /etc/network/interfaces.d/* or add its contents to /etc/network/interfaces directly, if you prefer.

8 October 2022

7 October 2022

Other distributions In openSUSE, Bernhard M. Wiedemann published his usual openSUSE monthly report.

6 October 2022

Warning: To use a separate file with `ifupdown` make sure that `/etc/network/interfaces` contains the line:
`source /etc/network/interfaces.d/*`
or add its contents to `/etc/network/interfaces` directly, if you prefer.