What is this thing? When you first open a command-line (note that I use the terms command-line and shell interchangably here, they re basically the same, but command-line is the more general term, and shell is the name for the program that executes commands for you) you ll see something like this: This line is called a command prompt and it tells you four pieces of information:

1. jaadams: The username of the user that is currently running this shell.
2. bg7: The name of the computer that this shell is running on, important for when you start accessing shells on remote machines.
3. /tmp/thisfolder: The folder or directory that your shell is currently running in. Like a file explorer (like Window s Explorer or Mac s Finder) a shell always has a working directory, from which all relative paths (see sidenote below) are resolved.

When you first opened a shell, however, you might notice that is looks more like this: This is a shorthand notation that the shell uses to make this output shorter when possible. ~ stands for your home directory, usually /home/<username>. Like C:\Users\<username>\ on Windows or /Users/<username> on Mac, this directory is where all your files should go by default. Thus a command prompt like this: actually tells you that you are currently in the /home/jaadams/Downloads directory.

Sidenote: The Unix Filesystem and Relative Paths folders on Linux and other Unix-derived systems like MacOS are usually called directories. These directories are represented by paths, strings that indicate where the directory is on the filesystem. The one unusual part is the so-called root directory . All files are stored in this folder or directories under it. Its path is just / and there are no directories above it. For example, the directory called home typically contains all user directories. This is stored in the root directory, and each users specific data is stored in a directory named after that user under home. Thus, the home directory of the user jacob is typically /home/jacob, the directory jacob under the home directory stored in the root directory /. If you re interested in more details about what goes in what directory, man hier has the basics and the Filesystem Hierarchy Standard governs the layout of the filesystem on most Linux distributions. You don t always have to use the full path, however. If the path does not begin with a /, it is assumed that the path actually begins with the path of the current directory. So if you use a path like my/folders/here, and you re in the /home/jacob directory, the path will be treated like /home/jacob/my/folders/here. Each folder also contains the symbolic links .. and .. Symbolic links are a very powerful kind of file that is actually a reference to another file. .. always represents the parent directory of the current directory, so /home/jacob/.. links to /home/. . always links to the current directory, so /home/jacob/. links to /home/jacob.

Running commands To run a command from the command prompt, you type its name and then usually some arguments to tell it what to do. For example, the echo command displays the text passed as arguments.

jacob@lovelace/home/jacob$ echo hello world
hello world

Arguments to commands are space-separated, so in the previous example hello is the first argument and world is the second. If you need an argument to contain spaces, you ll want to put quotes around it, echo "like so". Certain arguments are called flags , or options (options if they take another argument, flags otherwise) usually prefixed with a hyphen, and they change the way a program operates. For example, the ls command outputs the contents of a directory passed as an argument, but if you add -l before the directory, it will give you more details on the files in that directory.

jacob@lovelace/tmp/test$ ls /tmp/test
1  2  3  4  5  6
jacob@lovelace/tmp/test$ ls -l /tmp/test
total 0
-rw-r--r-- 1 jacob jacob 0 Aug 26 22:06 1
-rw-r--r-- 1 jacob jacob 0 Aug 26 22:06 2
-rw-r--r-- 1 jacob jacob 0 Aug 26 22:06 3
-rw-r--r-- 1 jacob jacob 0 Aug 26 22:06 4
-rw-r--r-- 1 jacob jacob 0 Aug 26 22:06 5
-rw-r--r-- 1 jacob jacob 0 Aug 26 22:06 6
jacob@lovelace/tmp/test$

Most commands take different flags to change their behavior in various ways.

File Management

• cd <path>: Change the current directory of the running shell to <path>.
• ls <path>: Output the contents of <path>. If no path is passed, it prints the contents of the current directory.
• touch <filename>: create an new empty file called <filename>. Used on an existing file, it updates the file s last accessed and modified times. Most text editors can also create a new file for you, which is probably more useful.
• mkdir <directory>: Create a new folder/directory at path <directory>.
• mv <src> <dest>: Move a file or directory at path <src> to <dest>.
• cp <src> <dest>: Copy a file or directory at path <src> to <dest>.
• rm <file>: Remove a file at path <file>.
• zip -r <zipfile> <contents...>: Create a zip file <zipfile> with contents <contents>. <contents> can be multiple arguments, and you ll usually want to use the -r argument when including directories in your zipfile, as otherwise only the directory will be included and not the files and directories within it.

Searching

• grep <thing> <file>: Look for the string <thing> in <file>. If no <file> is passed it searches standard input.
• find <path> -name <name>: Find a file or directory called <name> somwhere under <path>. This command is actually very powerful, but also very complex. For example you can delete all files in a directory older than 30 days with:

  find -mtime +30 -exec rm  \;
  

• locate <name>: A much easier to use command to find a file with a given name, but it is not usually installed by default.

Outputting Files

• cat <files...>: Output (concatenate) all the files passed as arguments.
• head <file>: Output the beginning of <file>
• tail <file>: Output the end of <file>

How to Find the Right Command All commands (at least on sane Linux distributions like Debian or Ubuntu) are documented with a manual page, in man section 1 (for more information on manual sections, run man intro). This can be accessed using man <command> You can search for the right command using the -k flag, as in man -k <search>. You can also view manual pages in your browser, on sites like https://manpages.debian.org or https://linux.die.net/man. This is not always helpful, however, because some command s descriptions are not particularly useful, and also there are a lot of manual pages, which can make searching for a specific one difficult. For example, finding the right command to search inside text files is quite difficult via man (it s grep). When you can t find what you need with man I recommend falling back to searching the Internet. There are lots of bad Linux tutorials out there, but here are some reputable sources I recommend:

• https://www.cyberciti.biz: nixCraft has excellent tutorials on all things Linux
• Hosting providers like Digital Ocean or Linode: Good intro documentation, but can sometimes be outdated
• https://tldp.org: The Linux Documentation project is great, but it can also be a little outdated sometimes.
• https://stackoverflow.com: Oftentimes has great answers, but quality varies wildly since anyone can answer.

These are certainly not the only options but they re the sources I would recommend when available.

How to Read a Manual Page Manual pages consist of a series of sections, each with a specific purpose. Instead of attempting to write my own description here, I m going to borrow the excellent one from The Linux Documentation Project

The NAME section is the only required section. Man pages without a name section are as useful as refrigerators at the north pole. This section also has a standardized format consisting of a comma-separated list of program or function names, followed by a dash, followed by a short (usually one line) description of the functionality the program (or function, or file) is supposed to provide. By means of makewhatis(8), the name sections make it into the whatis database files. Makewhatis is the reason the name section must exist, and why it must adhere to the format I described. (Formatting explanation cut for brevity) The SYNOPSIS section is intended to give a short overview on available program options. For functions this sections lists corresponding include files and the prototype so the programmer knows the type and number of arguments as well as the return type. The DESCRIPTION section eloquently explains why your sequence of 0s and 1s is worth anything at all. Here s where you write down all your knowledge. This is the Hall Of Fame. Win other programmers and users admiration by making this section the source of reliable and detailed information. Explain what the arguments are for, the file format, what algorithms do the dirty jobs. The OPTIONS section gives a description of how each option affects program behaviour. You knew that, didn t you? The FILES section lists files the program or function uses. For example, it lists configuration files, startup files, and files the program directly operates on. (Cut details about installing files) The ENVIRONMENT section lists all environment variables that affect your program or function and tells how, of course. Most commonly the variables will hold pathnames, filenames or default options. The DIAGNOSTICS section should give an overview of the most common error messages from your program and how to cope with them. There s no need to explain system error error messages (from perror(3)) or fatal signals (from psignal(3)) as they can appear during execution of any program. The BUGS section should ideally be non-existent. If you re brave, you can describe here the limitations, known inconveniences and features that others may regard as misfeatures. If you re not so brave, rename it the TO DO section ;-) The AUTHOR section is nice to have in case there are gross errors in the documentation or program behaviour (Bzzt!) and you want to mail a bug report. The SEE ALSO section is a list of related man pages in alphabetical order. Conventionally, it is the last section.

Remote Access One of the more powerful uses of the shell is through ssh, the secure shell. This allows you to remotely connect to another computer and run a shell on that machine:

user@host:~$ ssh other@example.com
other@example:~$

The prompt changes to reflect the change in user and host, as you can see in the example above. This allows you to work in a shell on that machine as if it was right in front of you.

Moving Files Between Machines There are several ways you can move files between machines over ssh. The first and easiest is scp, which works much like the cp command except that paths can also take a user@host argument to move files across computers. For example, if you wanted to move a file test.txt to your home directory on another machine, the command would look like:

scp test.txt other@example.com:

(The home directory is the default path) Otherwise you can move files by reversing the order of the arguments and put a path after the colon to move files from another directory on the remote host. For example, if you wanted to fetch the file /etc/issue.net from example.com:

scp other@example.com:/etc/issue.net .

Another option is the sftp command, which gives you a very simple shell-like interface in which you can cd and ls, before either puting files onto the local machine or geting files off of it. The final and most powerful option is rsync which syncs the contents of one directory to another, and doesn t copy files that haven t changed. It s powerful and complex, however, so I recommend reading the USAGE section of its man page.

Long-Running Commands The one problem with ssh is that it will stop any command running in your shell when you disconnect. If you want to leave something on and come back later then this can be a problem. This is where terminal multiplexers come in. tmux and screen both allow you to run a shell in a safe environment where it will continue even if you disconnect from it. You do this by running the command without any arguments, i.e. just tmux or just screen. In tmux you can disconnect from the current session by pressing Ctrl+b then d, and reattach with the tmux attach command. screen works similarly, but with Ctrl+a instead of b and screen -r to reattach.

Command Inputs and Outputs Arguments are not the only way to pass input to a command. They can also take input from what s called standard input , which the shell usually connects to your keyboard. Output can go to two places, standard output and standard error, both of which are directed to the screen by default.

Redirecting I/O Note that I said above that standard input/output/error are only usually connected to the keyboard and the terminal? This is because you can redirect them to other places with the shell operators <, > and the very powerful .

File redirects The operators < and > redirect the input and output of a command to a file. For example, if you wanted a file called list.txt that contained a list of all the files in a directory /this/one/here you could use:

ls /this/one/here > list.txt

Pipelines The pipe character, , allows you to direct the output of one command into the input of another. This can be very powerful. For example, the following pipeline lists the contents of the current directory searches for the string test , then counts the number of results. (wc -l counts the number of lines in its input)

ls   grep test   wc -l

For a better, but even more contrived example, say you have a file myfile, with a bunch of lines of potentially duplicated and unsorted data

test
test
1234
4567
1234

You can sort it and output only the unique lines with sort and uniq:

$ uniq < myfile   sort
1234
1234
4567
test

Save Yourself Some Typing: Globs and Tab-Completion Sometimes you don t want to type out the whole filename when writing out a command. The shell can help you here by autocompleting when you press the tab key. If you have a whole bunch of files with the same suffix, you can refer to them when writing arguments as *.suffix. This also works with prefixes, prefix*, and in fact you can put a * anywhere, *middle*. The shell will expand that * into all the files in that directory that match your criteria (ending with a specific suffix, starting with a specific prefix, and so on) and pass each file as a separate argument to the command. For example, if I have a series of files called 1.txt, 2.txt, and so on up to 9, each containing just the number for which it s named, I could use cat to output all of them like so:

jacob@lovelace/tmp/numbers$ ls
1.txt  2.txt  3.txt  4.txt  5.txt  6.txt  7.txt  8.txt	9.txt
jacob@lovelace/tmp/numbers$ cat *.txt
1
2
3
4
5
6
7
8
9

Also the ~ shorthand mentioned above that refers to your home directory can be used when passing a path as an argument to a command.

Ifs and For loops The files in the above example were generated with the following shell commands:

for i in 1 2 3 4 5 6 7 8 9
do
echo $i > $i.txt
done

But I ll have to save variables, conditionals and loops for another day because this is already too long. Needless to say the shell is a full programming language, although a very ugly and dangerous one.

Changes in RcppSMC version 0.2.2 (2020-08-30)

• Package helper files .editorconfig added (Adam in #43).
• Change const correctness and add return (Leah in #44).
• Updates to continuous integration and R versions used (Dirk)
• Accomodate CRAN request, other updates to CRAN Policy (Dirk in #49 fixing #48).

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dealing with the void during MiniDebConf Online #1 Between 28 and 31 May this year, we set out to create our first ever online MiniDebConf for Debian. Many people have been meaning to do something similar for a long time, but it just didn t work out yet. With many of us being in lock down due to COVID-19, and with the strong possibility looming that DebConf20 might have had to become an online event, we rushed towards organising the first ever Online MiniDebConf and put together some form of usable video stack for it. I could go into all kinds of details on the above, but this post is about a bug that lead to a pretty nifty feature for DebConf20. The tool that we use to capture Jitsi calls is called Jibri (Jitsi Broadcasting Infrustructure). It had a bug (well, bug for us, but it s an upstream feature) where Jibri would hang up after 30s of complete silence, because it would assume that the call has ended and that the worker can be freed up again. This would result in the stream being ended at the end of every talk, so before the next talk, someone would have to remember to press play again in their media player or on the video player on the stream page. Hrmph. Easy solution on the morning that the conference starts? I was testing a Debian Live image the night before in a KVM and thought that I might as well just start a Jitsi call from there and keep a steady stream of silence so that Jibri doesn t hang up. It worked! But the black screen and silence on stream was a bit eery. Because this event was so experimental in nature, and because we were on such an incredibly tight timeline, we opted not to seek sponsors for this event, so there was no sponsors loop that we d usually stream during a DebConf event. Then I thought Ah! I could just show the schedule! .

DebConf20 Moves Online For DebConf, we usually show a sponsors loop in between sessions. It s great that we give our sponsors visibility here, but in reality people see the sponsors loop and think Talk over! and then they look away. It s also completely silent and doesn t provide any additional useful information. I was wondering how I could take our lessons from MDCO#1 and integrate our new tricks with the sponsors loop. That is, add the schedule, time, some space to type announcements on the screen and also add some loopable music to it. I used OBS before in making my videos, and like the flexibility it provides when working with scenes and sources. A scene is what you would think of as a screen or a document with its own collection of sources or elements. For example, a scene might contain sources such as a logo, clock, video, image, etc. A scene can also contain another scene. This is useful if you want to contain a banner or play some background music that is shared between scenes.

The Loopy Loop Music The two mini albums that mostly played during the first few days were just a copy and paste from the MDCO#1 music, which was:

• Captive Portal What is
• Cinema Noir Children kill ants

• Foam Rubber (used in Nattie s and Urbec s shoutouts, and other places)
• Metaltania (used in Luna s shoutout)
• Flying Jack (used in Pollo s epic videoteam call for help)
• Study and relax (used in Rhonda s poetry night announcement)
• City sunshine (used in gregoa s coffee shoutout and then in several more after)
• Other tracks played in the loop from freepd.com included Brothers Unite, Ice and Snow, Parhelion, Shining stars and The Lagoon.

The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid packages maintained in Git. This post is part of a series about the progress of the Janitor. Last week s blog post documented how there are now over 30,000 lintian issues that have been fixed in git packaging repositories by the Janitor. It's important to note that any fixes from the Janitor that make it into a Git packaging repository will also need to be uploaded to the Debian archive. This currently requires that a Debian packager clones the repository and builds and uploads the package. Until a change makes it into the archive, users of Debian will unfortunately not see the benefits of improvements made by the Janitor. 82% of the 30,000 changes from the Janitor that have made it into a Git repository have not yet been uploaded, although changes do slowly trickle in as maintainers make other changes to packages and upload them along with the lintian fixes from the Janitor. This is not just true for changes from the Janitor, but for all sorts of other smaller improvements as well. However, the process of cloning and building git repositories and uploading the resulting packages to the Debian archive is fairly time-consuming and it s probably not worth the time of developers to follow up every change from the Janitor with a labour-intensive upload to the archive. It would be great if it was easier to trigger uploads from git commits. Projects like tag2upload will hopefully help, and make it more likely that changes end up in the Debian archive. The majority packages do get at least one new source version upload per release, so most changes will eventually make it into the archive.

For more information about the Janitor's lintian-fixes efforts, see the landing page.

Changes in anytime version 0.3.9 (2020-08-26)

• Skip one test file that is impossible to run across different CRAN setups, and life is definitely too short for these games.
• Change remaining http:// to https:// because, well, you know.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Background CRAN has been a cornerstone of the success of R in recent years. As a well-maintained repository with stringent quality control, it ensures users have access to highest-quality statistical / analytical software that just works . Users on Windows and macOS also benefit from faster installation via pre-compiled binary packages. Linux users generally install from source, which can be more tedious and, often, much slower. Those who know where to look have had access to (at least some) binaries for years as well (and one of us blogged and vlogged about this at length). Debian users get close to 1000 CRAN and BioConductor packages (and, true to Debian form, for well over a dozen hardware platforms). Michael Rutter maintains a PPA with 4600 binaries for three different Ubuntu flavors (see c2d4u4.0+). More recently, Fedora joined the party with 16000 (!!) binaries, essentially all of CRAN, via a Copr repository (see iucar/cran). The buzz currently is however with RSPM, a new package manager by RStudio. An audacious project, it provides binaries for several Linux distributions and releases. It has already been tested in many RStudio Cloud sessions (including with some of our students) as well as some CI integrations. RSPM cuts across and takes the breadth of CRAN across several Linux distributions, bringing installation of pre-built CRAN packages a binaries under their normal CRAN package names. Another nice touch is the integration with install.packages(): these binaries are installed in a way that is natural for R users but as binaries. It is however entirely disconnected from the system package management. This means that the installation of a package requiring an external library may succeed and still fail, as a required library simply cannot be pulled in directly by RSPM. So what is needed is a combination. We want binaries that are aware of their system dependencies but accessible directly from R just like RSPM offers it. Enter BSPM the Bridge to System Package Manager package (also on CRAN). The first illustration (using Ubuntu 18.04) shows RSPM on the left, and BSPM on the right, both installing the graphics package Cairo (and both using custom Rocker containers). This fails for RSPM as no binary is present and a source build fails for the familiar lack of a -dev package. It proceeds just fine on the right under BSPM. A second illustration shows once again RSPM on the left, and BSPM on the right (this time on Fedora), both installing the units package without a required system dependency. The installation of units works for BSPM as the dependency libudunits is brought in, but fails under RSPM. The binary installation succeeds in both cases, but the missing dependency (the UDUNITS2 library) is brought in only by BSPM. Consequently, the package fails to load under RSPM.

Summary To conclude, highlights of BSPM are:

• direct installation of binary packages from R via R commands under their normal CRAN names (just like RSPM);
• full integration with the system package manager, delivering system installations (improving upon RSPM);
• full dependency resolution for R packages, including system requirements (improving upon RSPM).

This offers easy, reliable, fast installation of R packages, and we invite you to pick all three. We recommend usage with either Ubuntu with the 4.6k packages via the Rutter PPA, or Fedora via the even more complete Copr repository (which already includes a specially-tailored version of BSPM called CoprManager). We hope this short note wets your appetite to learn more about bspm (which is itself on CRAN) and the two sets of Rocker containers shown. The rocker/r-rspm container comes in two two flavours for Ubuntu 18.04 and 20.04. Similarly, the rocker/r-bspm container comes in the same two two flavours for Ubuntu 18.04 and 20.04, as well as in a Debian testing variant. Feedback is appreciated at the bspm or rocker issue trackers. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

• LoopDocs: https://github.com/LoopKit/loopdocs/pull/191
• Nightscout: https://github.com/nightscout/cgm-remote-monitor/pull/5893

Setting up Nightscout If you want to setup Nightscout from scratch using MongoDB Atlas, please follow this modified guide. However, note that you will have to make one modification to the steps in the guide. At the start of Step 4, you will need to go to this repository instead: https://github.com/aviau/cgm-remote-monitor. This is my own version of Nightscout and it contains small modifications that will allow you to set it up with MongoDB Atlas easily. I will keep this blog post updated as I receive feedback. Come back here for more instructions.

Mac If you have a Mac, all you need to do is open Terminal.app, which is usually found under /Applications/Utilities. Note that Mac now defaults to zsh instead of bash, which is usually the shell used on Linux. This shouldn t matter, but it s something you should be aware of.

Windows On Windows, things are much more complex. There s always Powershell, but if you want a true Unix shell experience like you d get on Linux, you ll need to install the Windows Subsystem for Linux. This allows you to run Linux programs on your Windows 10 computer. This boils down to opening Powershell (open the start menu, search for powershell ) as an administrator (right-click, then Run as Administrator, then click Yes or enter an administrator s password when the UAC prompt appears). In this new Powershell window, you need to run the following command to enable WSL:

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart

When this command executes successfully you should see the following output: After this output appears, you ll need to reboot your machine before you can continue. Once your machine is rebooted, you need to install a Linux distribution. Different communities and companies ship their own versions of Linux and the various services and utilities required to use it, and these versions are called distributions. If you re not sure which one to use, I would recommend Ubuntu, as it was the distribution first integrated into WSL, and it s a common distribution for new users. After installing your chosen distribution, you ll need to perform the first-time setup. You ll just need to run it, as it is now installed as a program on your computer, and then it will walk you through the setup process, which requires you to create a new user account for your Linux distribution. This does not need to be the same as your Windows username and password, and it s probably safer if it isn t. You ll need to remember that password for running administrative commands with sudo.

Introducing cryptsetup-suspend Today, we're introducing cryptsetup-suspend, whose job is to protect the content of your harddrives while the system is sleeping. TL;DR:

• You can lock your encrypted harddrives during suspend mode by installing cryptsetup-suspend
• For cryptsetup-suspend to work properly, at least Linux kernel 5.6 is required
• We hope that in a bright future, everything will be available out-of-the-box in Debian and it's derivatives

Before: After:

Table of contents

What does this mean and why should you care about it? If you don't use full-disk encryption, don't read any further. Instead, think about, what will happen if you lose your notebook on the train, a random person picks it up and browses through all your personal pictures, e-mails, and tax records. Then encrypt your system and come back. If you believe full-disk encryption is necessary, you might know that it only works when your machine is powered off. Once you turn on the machine and decrypt your harddrive, your encryption key stays in RAM and can potentially be extracted by malicious software or physical access. Even if these attacks are non-trivial, it's enough to worry about. If an attacker is able to extract your disk encryption keys from memory, they're able to read the content of your disk in return. Sadly, in 2020, we hardly power off our laptops anymore. The sleep mode, also known as "suspend mode", is just too convenient. Just close the lid to freeze the system state and lift it anytime later in order to continue. Well, convenience usually comes with a cost: during suspend mode, your system memory is kept powered, all your data - including your encryption keys - stays there, waiting to be extracted by a malicious person. Unfortunately, there are practical attacks to extract the data of your powered memory. Cryptsetup-suspend expands the protection of your full-disk encryption to all those times when your computer sleeps in suspend mode. Cryptsetup-suspend utilizes the suspend feature of LUKS volumes and integrates it with your Debian system. Encryption keys are evicted from memory before suspend mode and the volumes have to be re-opened after resuming - potentially prompting for the required passphrases. By now, we have a working prototype which we want to introduce today. We did quite some testing, both on virtualized and bare-metal Debian and Ubuntu systems, with and without graphical stack, so we dare to unseal and set free the project and ask you - the community - to test, review, criticize and give feedback. Here's a screencast of cryptsetup-suspend in action:

State of the implementation: where are we? If you're interested in the technical details, here's how cryptsetup-suspend works internally. It basically consists of three parts:

1. cryptsetup-suspend: A C program that takes a list of LUKS devices as arguments, suspends them via luksSuspend and suspends the system afterwards. Also, it tries to reserve some memory for decryption, which we'll explain below.
2. cryptsetup-suspend-wrapper: A shell wrapper script which works the following way:
   1. Extract the initramfs into a ramfs
   2. Run (systemd) pre-suspend scripts, stop udev, freeze almost all cgroups
   3. Chroot into the ramfs and run cryptsetup-suspend
   4. Resume initramfs devices inside chroot after resume
   5. Resume non-initramfs devices outside chroot
   6. Thaw groups, start udev, run (systemd) post-suspend scripts
   7. Unmount the ramfs
3. A systemd unit drop-in file overriding the Exec property of systemd-suspend.service so that it invokes the script cryptsetup-suspend-wrapper.

Reusing large parts of the existing cryptsetup-initramfs implementation has some positive side-effects: Out-of-the-box, we support all LUKS block device setups that have been supported by the Debian cryptsetup packages before. Freezing most processes/cgroups is necessary to prevent possible race-conditions and dead-locks after the system resumes. Processes will try to access data on the locked/suspended block devices eventually leading to buffer overflows and data loss.

Technical challenges and caveats

• Dead-locks at suspend: In order to prevent possible dead-locks between suspending the encrypted LUKS disks and suspending the system, we have to tell the Linux kernel to not sync() before going to sleep. A corresponding patch got accepted upstream in Linux 5.6. See section What about the kernel patch? below for details.
• Race conditions at resume: Likewise, there's a risk of race conditions between resuming the system and unlocking the encypted LUKS disks. We went with freezing as many processes as possible as a counter measurement. See last part of section State of the implementation: where are we? for details.
• Memory management: Memory management is definitely a challenge. Unlocking disks might require a lot of memory (if key derivation function is argon2i) and the swap device most likely is locked at that time. See section All that matters to me is the memories! below for details.
• systemd dependency: Our implementation depends on systemd. It uses a unit drop-in file for systemd-suspend.service for hooking into the system suspend process and depends on systemds cgroup management to freeze and thaw processes. If you're using a different init system, sorry, you're currently out of luck.

What about the kernel patch? The problem is simple: the Linux kernel suspend implementation enforces a final filesystem sync() before the system goes to sleep in order to prevent potential data loss. While that's sensible in most scenarios, it may result in dead-locks in our situation, since the block device that holds the filesystem is already suspended. The fssync() call will block forever as it waits for the block device to finish the sync() operation. So we need a way to conditionally disable this sync() call in the Linux kernel resume function. That's what our patch does, by introducing a run-time switch at /sys/power/sync_on_suspend, but it only got accepted into the Linux kernel recently and was first released with Linux kernel 5.6. Since release 4.3, the Linux kernel at least provides a build-time flag to disable the sync(): CONFIG_SUSPEND_SKIP_SYNC (that was called SUSPEND_SKIP_SYNC first and renamed to CONFIG_SUSPEND_SKIP_SYNC in kernel release 4.9). Enabling this flag at build-time protects you against the dead locks perfectly well. But while that works on an individual basis, it's a non-option for the distribution Linux kernel defaults. In most cases you still want the sync() to happen, except if you have user-space code that takes care of the sync() just before suspending your system - just like our cryptsetup-suspend implementation does. So in order to properly test cryptsetup-suspend, you're strongly advised to run Linux kernel 5.6 or newer. Fortunately, Linux 5.6 is available in buster-backports thanks to the Debian Kernel Team.

All that matters to me is the memories! One of the tricky parts is memory management. Since version 2, LUKS uses argon2i as default key derivation function. Argon2i is a memory-hard hash function and LUKS2 assigns the minimum of half of your systems memory or 1 GB to unlocking your device. While this is usually unproblematic during system boot - there's not much in the system memory anyway - it can become problematic when suspending. When cryptsetup tries to unlock a device and wants 1 GB of memory for this, but everything is already occupied by your browser and video player, there's only two options what to do:

1. Kill a process to free some memory
2. Move some of the data from memory to swap space

The first option is certainly not what you expect when suspending your system. The second option is impossible, because swap is located on your harddrive which we have locked before. Our current solution is to allocate the memory after freezing the other processes, but before locking the disks. At this time, the system can still move data to swap, but it won't be accessed anymore. We then release the memory just in time for cryptsetup to claim it again. The implementation of this is still subject to change.

What's missing: A proper user interface As mentioned before, we consider cryptsetup-suspend usable, but it certainly still has bugs and shortcomings. The most obvious one is lack of a proper user interface. Currently, we switch over to a tty command-line interface to prompt for passphrases when unlocking the LUKS devices. It certainly would be better to replace this with a graphical user interface later, probably by using plymouth or something alike. Unfortunately, it seems rather impossible to spawn a real graphical environment for the passphrase prompt. That would imply to load the full graphical stack into the ramfs, raising the required amount of memory significantly. Lack of memory is currently our biggest concern and source of trouble. We'd definitely appreciate to learn about your ideas how to improve the user experience here.

Let's get practical: how to use TL;DR: On Debian Bullseye (Testing), all you need to do is to install the cryptsetup-suspend package from experimental. It's not necessary to upgrade the other cryptsetup packages. On Debian Buster, cryptsetup packages from backports are required.

1. First, be sure that you're running Linux kernel 5.6 or newer. For Buster systems, it's available in buster-backports.
2. Second, if you're on Debian Buster, install the cryptsetup 2:2.3.3-2~bpo10+1 packages from buster-backports.
3. Third, install the cryptsetup-suspend package from experimental. Beware that cryptsetup-suspend depends on cryptsetup-initramfs (>= 2:2.3.3-1~). Either you need the cryptsetup packages from testing/unstable, or the backports from buster-backports.
4. Now that you have the cryptsetup-suspend package installed, everything should be in place: Just send your system to sleep. It should switch to a virtual text terminal before going to sleep, ask for a passphrase to unlock your encrypted disk(s) after resume and switch back to your former working environment (most likely your graphical desktop environment) afterwards.

Security considerations Suspending LUKS devices basically means to remove the corresponding encryption keys from system memory. This protects against all sort of attacks trying to read them from there, e.g. cold-boot attacks. But, cryptsetup-suspend only protects the encryption keys of your LUKS devices. Most likely there's more sensitive data in system memory, like all kinds of private keys (e.g. OpenPGP, OpenSSH) or documents with sensitive content. We hope that the community will help improve this situation by providing useful pre-/post-suspend scripts. A positive example is KeepassXC, which is able to lock itself when going to suspend mode.

Related and similar projects

• Systemd Homed: systemd recently got a new feature to manage home directories. It brings support for encrypting your home directory within a LUKS2 container and for suspending the LUKS2 container during system sleep. It all is limited to systemd home directories though and doesn't help with other LUKS devices.
• There's several earlier luks-suspend-* implementations. Unfortunately, neither of them deal with all the Technical challenges and caveats we discovered. Still we'd like to mention some of them:
• debian-luks-suspend
• go-luks-suspend
• arch-luks-suspend

Links

• Our Linux kernel patch: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c052bf82c6b00ca27aab0859addc4b3159dfd3a4
• The debian/experimental branch of our Debian cryptsetup packaging repository, containing cryptsetup-suspend related changes: https://salsa.debian.org/cryptsetup-team/cryptsetup/-/tree/debian/experimental
• cryptsetup-suspend manpage: https://salsa.debian.org/cryptsetup-team/cryptsetup/-/blob/debian/experimental/debian/doc/cryptsetup-suspend.xml

Feedback and Comments We'd be more than happy to learn about your thoughts on cryptsetup-suspend. For specific issues, don't hesitate to open a bugreport against cryptsetup-suspend. You can also reach us via mail - see the next section for contact addresses. Last but not least, comments below the blogpost work as well.

Authors

• Tim (tim at systemli.org)
• Jonas (jonas at freesources.org)

Bits from the DPL Time: Aug 24 (Mon): 16:00 UTC. The traditional DebConf talk from the DPL, where we take a look at the state of the Debian project and where we re heading. This talk is pre-recorded, but there will be a few minutes after the talk for questions. https://debconf20.debconf.org/talks/9-bits-from-the-dpl/

Leadership in Debian BOF/Panel Time: Aug 27 (Thu): 18:00 UTC. In this session, we will host a panel of people who hold (or who have held) leadership positions within Debian. We ll go through a few questions for the panel and then continue with open questions and discussion. https://debconf20.debconf.org/talks/46-leadership-in-debian-bofpanel/

Local Teams Time: Aug 29 (Sat): 19:00 UTC. We already have a number of large and very successful Debian Local Groups (Debian France, Debian Brazil and Debian Taiwan, just to name a few), but what can we do to help support upcoming local groups or help spark interest in more parts of the world? In this BoF, we ll discuss the possibility of setting up a local group support team or a new delegation that will keep track of local teams, manage budgets and get new local teams bootstrapped. https://debconf20.debconf.org/talks/50-local-teams/

• Ensuring that the code works as intended
• Ensuring that the task was fully accomplished and no detail left out
• Ensuring that no security issues have been introduced
• Making sure the code fits the practices of the team and is understandable and maintainable by others
• Sharing insights, transferring knowledge between code author and code reviewer
• Helping the code author to learn to write better code

• You run Linux on your computer (let's assume a Debian-like distro).
• You want to send emails from your terminal.

• Your email account is a GMail one.
• Your desktop environment is GNOME.

• it will basically work, GMail won't block you because it thinks that you're trying to sign in from an unknown device, a weird location or whatever.
• your main GMail password remains secret, you won't have to write it down in any configuration file or anywhere else.
• you can change your main GMail password, no breakage, apps will still work as each of them use their own passwords.
• you can revoke an app password anytime, without impacting anything else.

sudo apt install msmtp

# Write a dummy email
cat << EOF > message.txt
From: YOUR_LOGIN@gmail.com
To: SOMEONE_ELSE@SOMEWHERE_ELSE.com
Subject: Cafe Sua Da
Iced-coffee with condensed milk
EOF
# Send it
cat message.txt   msmtp \
    --auth=on --tls=on \
    --host smtp.gmail.com \
    --port 587 \
    --user YOUR_LOGIN \
    --read-envelope-from \
    --read-recipients
# msmtp prompts you for your password:
# this is where goes the app password!

• --auth=on because we want to authenticate with the server.
• --tls=on because we want to make sure that the communication with the server is encrypted.
• --host and --port tells where to find the server. If you don't use GMail, adjust that accordingly.
• --user is obviously your GMail username.

cat << 'EOF' > ~/.msmtprc
defaults
tls on
account gmail
auth on
host smtp.gmail.com
port 587
user YOUR_LOGIN
from YOUR_LOGIN@gmail.com
account default : gmail
EOF

• under defaults are the default values for all the following accounts.
• under account are the settings specific to this account, until another account line is found.
• finally, the last line defines which account is the default.

# Write a dummy email. Note that the
# header 'From:' is no longer needed,
# it's already in '~/.msmtprc'.
cat << 'EOF' > message.txt
To: SOMEONE_ELSE@SOMEWHERE_ELSE.com
Subject: Flat White
The milky way for coffee
EOF
# Send it
cat message.txt   msmtp \
    --account default \
    --read-recipients

msmtp -t < message.txt

sudo apt install libsecret-tools

secret-tool store --label msmtp \
    host smtp.gmail.com \
    service smtp \
    user YOUR_LOGIN

msmtp -t < message.txt

the built-in default is to search for sendmail in /usr/sbin, /usr/lib and $PATH if such program is available

--smtp-server=
[...] Alternatively it can specify a full pathname of a sendmail-like program instead; the program must support the -i option.

[sendemail]
    smtpserver = /usr/bin/msmtp

--sendmail=SENDMAILCMD
Specify the sendmail command [...] Default is /usr/sbin/sendmail.

--mta=MTA
Specify an alternate MTA, instead of /usr/sbin/sendmail (the default).

sudo apt install msmtp-mta

Our thoughts about cooperation aspects of doing things together. Sometimes in Debian we do work together with others, and sometimes we are a number of people who work alone, and happen to all upload their work in the same place. In times when we have needed to take important decisions together, this distinction has become crucial, and some of us might have found that we were not as good at cooperation as we would have thought. This talk is intended for everyone who is part of a larger community. We will show concepts and tools that we think could help understand and shape cooperation.

• full quality
• low quality

• Carlo Maria Cipolla's graph
• Thomas Kilmann Conflict Mode Instrument
• Kahler's Drivers

• Carlo M. Cipolla - Wikipedia
• Thomas Kilmann Conflict Mode Instrument - Wikipedia
• The Evolution of Trust
• Moral behavior in animals
• Taibi Kahler - Wikipedia
• Kahler's Drivers questionnaire / Antreiber Test - English and German
• Alison Green Coaching - Behavioural Drivers Questionnaire
• Founder's syndrome - Wikipedia
• Peter principle - Wikipedia
• Benevolent dictator for life - Wikipedia
• Difficult Conversations: How to Discuss What Matters Most
• Getting to Yes - Wikipedia
• Webcomics - Pepper&Carrot

How can we still have a good code review process without making it a "you need to be perfect" scenario? I often find picky code reviews help me write better code.

The Debian Janitor is an automated system that commits fixes for (minor) issues in Debian packages that can be fixed by software. It gradually started proposing merges in early December. The first set of changes sent out ran lintian-brush on sid packages maintained in Git. This post is part of a series about the progress of the Janitor.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

#!/usr/bin/python3
from debmutate.control import ControlEditor
from lintian_brush.fixer import report_result, fixed_lintian_tag
with ControlEditor() as updater:
    for para in updater.paragraphs:
        if para.get("Priority") == "extra":
            para["Priority"] = "optional"
            fixed_lintian_tag(
                para, 'priority-extra-is-replaced-by-priority-optional')
report_result("Change priority extra to priority optional.")

Dmitry and Lisandro.Note 20200818 12:12 ARST: I was asked if the move has anything to do with code quality or licensing. The answer is a huge no, Qt is a **great** project which we love. As stated before it's mostly about lack of free time to properly maintain it.

The WITH query will generally be evaluated as written, without suppression of rows that the parent query might discard afterwards.

• New features don't gain a backpatch. This is a project policy. Of course this is not a new feature, and if it is an exception should be made. This was explained clearly in the thread.
• I'm not sure the "we don't want to upgrade application code at the same time as the database" is really tenable. This is quite an astonishing statement, particularly given the multiple users who said they wanted to do precisely that.
• I think we could find cases where we caused worse breaks between major versions. Paraphrasing: "We've done worse things in the past so we should do this bad thing too". WTF?
• One disadvantage is that this will increase confusion for users, who'll get used to the behavior on 12, and then they'll get confused on older releases. This seems highly contrived. Surely the number of people who are likely to suffer this confusion is tiny. Providing the new syntax in old versions (including of course the appropriate changes to the docs everywhere) might well make such confusion less rather than more likely.
• [Poster is concerned about] 11.6 and up allowing a syntax that 11.0-11.5 don't. People are likely to write code relying on this and then be surprised when it doesn't work on a slightly older server. And, similarly: we'll then have a lot more behavior differences between minor releases. Again this seems a contrived and unconvincing objection. As that first poster even notes: Still, is that so much different from cases where we fix a bug that prevented some statement from working? No, it isn't.
• if we started looking we'd find many changes every year that we could justify partially or completely back-porting on similar grounds ... we'll certainly screw it up sometimes. This is a slippery slope argument. But there is no slippery slope: in particular, the proposed change does not change any of the substantive database logic, and the upstream developers will hardly have any difficulty rejecting future more risky backport proposals.
• if you insist on using the same code with pre-12 and post-12 releases, this should be achievable (at least in most cases) by using the "offset 0" trick. What? First I had heard of it but this in fact turns out to be true! Read more about this, below...

I think OFFSET 0 is the better approach since it's more obviously a hack showing that something weird is going on, and it's unlikely we'll ever change the optimiser behaviour around OFFSET 0 ... wheras hopefully CTEs will become inlineable at some point CTEs became inlineable by default in PostgreSQL 12.