Russ Allbery: Review: The House in the Cerulean Sea
| Publisher: | Tor |
| Copyright: | March 2020 |
| ISBN: | 1-250-21732-6 |
| Format: | Kindle |
| Pages: | 398 |
| Publisher: | Tor |
| Copyright: | March 2020 |
| ISBN: | 1-250-21732-6 |
| Format: | Kindle |
| Pages: | 398 |
Despite having worked on a number of ARM platforms I ve never actually had an ARM based development box at home. I have a Raspberry Pi B Classic (the original 256MB rev 0002 variant) a coworker gave me some years ago, but it s not what you d choose for a build machine and generally gets used as a self contained TFTP/console server for hooking up to devices under test. Mostly I ve been able to do kernel development with the cross compilers already built as part of Debian, and either use pre-built images or Debian directly when I need userland pieces. At a previous job I had a Marvell MACCHIATObin available to me, which works out as a nice platform - quad core A72 @ 2GHz with 16GB RAM, proper SATA and a PCIe slot. However they re still a bit pricey for a casual home machine. I really like the look of the HoneyComb LX2 - 16 A72 cores, up to 64GB RAM - but it s even more expensive.
So when I saw the existence of the 8GB Raspberry Pi 4 I was interested. Firstly, the Pi 4 is a proper 64 bit device (my existing Pi B is ARMv6 which means it needs to run Raspbian instead of native Debian armhf), capable of running an upstream kernel and unmodified Debian userspace. Secondly the Pi 4 has a USB 3 controller sitting on a PCIe bus rather than just the limited SoC USB 2 controller. It s not SATA, but it s still a fairly decent method of attaching some storage that s faster/more reliable than an SD card. Finally 8GB RAM is starting to get to a decent amount - for a headless build box 4GB is probably generally enough, but I wanted some headroom.
The Pi comes as a bare board, so I needed a case. Ideally I wanted something self contained that could take the Pi, provide a USB/SATA adaptor and take the drive too. I came across the pre-order for the DeskPi Pro, decided it was the sort of thing I was after, and ordered one towards the end of September. It finally arrived at the start of December, at which point I got round to ordering a Pi 4 from CPC.
Total cost ~ 120 for the case + Pi.
I managed to break a USB port on the Desk Pi. It has a pair of forward facing ports, I plugged my wireless keyboard dongle into it and when trying to remove it the solid spacer bit in the socket broke off. I ve never had this happen to me before and I ve been using USB devices for 20 years, so I m putting the blame on a shoddy socket.
The first drive I tried was an old Crucial M500 mSATA device. I have an adaptor that makes it look like a normal 2.5 drive so I used that. Unfortunately it resulted in a boot loop; the Pi would boot its initial firmware, try to talk to the drive and then reboot before even loading Linux. The DeskPi Pro comes with an m2 adaptor and I had a spare m2 drive, so I tried that and it all worked fine. This might just be power issues, but it was an unfortunate experience especially after the USB port had broken off.
(Given I ended up using an M.2 drive another case option would have been the Argon ONE M.2, which is a bit more compact.)
The case is a little snug; I was worried I was going to damage things as I slid it in. Additionally the construction process is a little involved. There s a good set of instructions, but there are a lot of pieces and screws involved. This includes a couple of FFC cables to join things up. I think this is because they ve attempted to make a compact case rather than allowing a little extra room, and it does have the advantage that once assembled it feels robust without anything loose in it.
I hate the need for an external USB3 dongle to bridge from the Pi to the USB/SATA adaptor. All the cases I ve seen with an internal drive bay have to do this, because the USB3 isn t brought out internally by the Pi, but it just looks ugly to me. It s hidden at the back, but meh.
Fan control is via a USB/serial device, which is fine, but it attaches to the USB C power port which defaults to being a USB peripheral. Raspbian based kernels support device tree overlays which allows easy reconfiguration to host mode, but for a Debian based system I ended up rolling my own dtb file. I changed
#include "bcm283x-rpi-usb-peripheral.dtsi"
#include "bcm283x-rpi-usb-host.dtsi"
arch/arm/boot/dts/bcm2711-rpi-4-b.dts and then I did:
cpp -nostdinc -I include -I arch -undef -x assembler-with-cpp \
arch/arm/boot/dts/bcm2711-rpi-4-b.dts > rpi4.preprocessed
dtc -I dts -O dtb rpi4.preprocessed -o bcm2711-rpi-4-b.dtb
bcm2711-rpi-4-b.dtb file replaced the one in /boot/firmware. This isn t a necessary step if you don t want to use the cooling fan in the case, or the front USB ports, and it s not really anyone s fault, but it was an annoying extra step to have to figure out.
The DeskPi came with a microSD card that was supposed to have RaspiOS already on it. It didn t, it was blank. In my case that was fine, because I wanted to use Debian, but it was a minor niggle.
apt upgrade and got updated to the Buster 10.7 release, as well as the latest 5.9 backport kernel, and everything came back without effort after a reboot. It s lovely to be able to run Debian on this device without having to futz around with self-compiled kernels.
The DeskPi makes a lot of effort to route things externally. The SD slot is brought out to the front, making it easy to fiddle with the card contents without having to open the case to replace it. All the important ports are brought out to the back either through orientation of the Pi, or extenders in the case. That means the built in Pi USB ports, the HDMI sockets (conveniently converted to full size internally), an audio jack and a USB-C power port. The aforementioned USB3 dongle for the bridge to the drive is the only external thing that s annoying.
Thermally things seem good too. I haven t done a full torture test yet, but with the fan off the system is sitting at about 40 C while fairly idle. Some loops in bash that push load up to above 2 get the temperature up to 46 C or so, and turning the fan on brings it down to 40 C again. It s audible, but quieter than my laptop and not annoying.
I liked the way the case came with everything I needed other than the Pi 4 and a suitable disk drive. There was an included PSU (a proper USB-C PD device, UK plug), the heatsink/fan is there, the USB/SATA converter is there and even an SD card is provided (though that s just because I had a pre-order).
Speaking of the SD, I only needed it for initial setup. Recent Pi 4 bootloaders are capable of booting directly from USB mass storage devices. So I upgraded using the RPi EEPROM Recovery image (which just needs extracted to the SD FAT partition, no need for anything complicated - boot with it and the screen goes all green and you know it s ok), then created a FAT partition at the start of the drive for the kernel / bootloader config and a regular EXT4 partition for root. Copies everything over, updated paths, took out the SD and it all just works happily.
Once a year folks interested in Netfilter technologies gather together to discuss past, ongoing and
future works. The Netfilter Workshop is an opportunity to share and discuss new ideas, the state of
the project, bring people together to work & hack and to put faces to people who otherwise are just
email names. This is an event that has been happening since at least 2001, so we are
talking about a genuine community thing here.
It was decided there would be an online format, split in 3 short meetings, once per week on
Fridays. I was unable to attend the first session on 2020-11-06 due to scheduling conflict, but I
made it to the sessions on 2020-11-13 and 2020-11-20. I would say the sessions were joined by about
8 to 10 people, depending on the day. This post is a summary with some notes on what happened in
this edition, with no special order.
Pablo did the classical review of all the changes and updates that happened in all the Netfilter
project software components since last workshop. I was unable to watch this presentation, so I have
nothing special to comment. However, I ve been following the development of the project very
closely, and there are several interesting things going on, some of them commented below.
Florian Westphal brought to the table status on some open/pending work for mptcp option matching,
systemd integration and finally interfacing from nft with cgroupv2. I was unable to participate in
the talk for the first two items, so I cannot comment a lot more. On the cgroupv2 side, several
options were evaluated to how to match them, identification methods, the hierarchical tree that
cgroups present, etc. We will have to wait a bit more to see how the final implementation looks
like.
Also, Florian presented his concerns on conntrack hash collisions. There are no real-world known
issues at the moment, but there is an old paper that suggests we should keep and eye on
this and introduce improvements to prevent future DoS attack vectors. Florian mentioned
these attacks are not practical at the moment, but who knows in a few years. He wants to explore
introducing RB trees for conntrack. It will probably be a rbtree structure of hash
tables in order to keep supporting parallel insertions. He was encouraged by others to go ahead and
play/explore with this.
Phil Sutter shared his past and future iptables development efforts. He highlighted fixed bugs and
his short/midterm TODO list. I know Phil has been busy lately fixing iptables-legacy/iptables-nft
incompatibilities. Basically addressing annoying bugs discovered by all ruleset managers out there
(kubernetes, docker, openstack neutron, etc). Lots of work has been done to improve the situation;
moreover I myself reported, or forwarded from the Debian bug tracker, several bugs. Anyway I was
unable to attend this talk, only learnt a few bits in the following sessions, so I don t have a lot
to comment here.
But when I was fully present, I was asked by Phil about the status of netfilter components in
Debian, and future plans. I shared my information. The idea for the next Debian stable release is
to don t include iptables in the installer, and include nftables instead. Since Debian
Buster, nftables is the default firewalling tool anyway. He shared the plans for the RedHat-related
ecosystem, and we were able to confirm that we are basically in sync.
Pablo commented on the latest Netfilter flowtable enhancements happening. Using the flowtable
infrastructure, one can create kernel network bypasses to speed up packet throughput. The latest
changes are aimed for bridge and VLAN enabled setups. The flowtable component will now know how
to bypass in these 2 network architectures as well as the previously supported ingress hook. This
is basically aimed for virtual machines and containers scenarios. There was some debate on use
cases and supported setups. I commented that a bunch of virtual machines connected to a classic
linux bridge and then doing NAT is basically what Openstack Neutron does, specifically in DVR
setups. Same can be found in some container-based environments. Early/simple benchmarks done by
Pablo suggest there could be huge performance improvements for those use cases.
There was some inevitable comparison of this approach to what others, like DPDK/XDP can do. A
point was raised about this being a more generic and operating system-integrated solution, which
should make it more extensible and easier to use.
Stefano Bravio commented on several open topics for nftables that he is interested on working on.
One of them, issues related to concatenations + vmap issues. He also addressed concerns with
people s expectations when migrating from ipset to nftables. There are several corner features in
ipset that aren t currently supported in nftables, and we should document them. Stefano is also
wondering about some tools to help in the migration. A translation layer like there is in place
for iptables. Eric Gaver commented there are a couple of semantics that will not be suitable for
translation, such as global sets, or sets of sets. But ipset is way simpler than iptables, so a
translation mechanism should probably be created. In any case, there was agreement that anything
that helps people migrate is more than welcome, even if it doesn t support 100% of the use cases.
Stefano is planning to write documentation in the nftables wiki on how the pipapo algorithm
works and the supported use cases. Other plans by Stefano include to work on some optimisations for
faster matches. He mentioned using architecture specific instruction to speed up sets operations,
like lookups.
Finally, he commented that some folks working with eBPF have showed interest in reusing some parts
of the nftables sets infrastructure (pipapo) because they have detected performance issues in their
own data structures in some cases. It is not clear how to best achieve it, how to better bridge the
two things together. Probably the ideal is to generalize the pipapo data structures and integrate
it into the generic bitmap library or something which can be used by anyone. Anyway, he hopes to
get some more time to focus on Netfilter stuff begining with the next year, in a couple of months.
Moving a bit away from the pure software development topics, Pablo commented on the
netfilter.org infrastructure. Right now the servers are running on gandi.net,
on virtual machines that are being basically donated to us. He pointed that the plan is to
simplify the infrastructure. For that reason, for example, FTP services has been shut down. Rsync
services have been shut down as well, so basically we no longer have a mirrors infrastructure. The
bugzilla and wikis we have need some attention, given they are old software pieces, and we need
to migrate them to be more modern. Finally, the new logo that was created was presented.
Later on, we spent a good chunk of the meeting discussing options on how to address the inevitable
iptables deprecation situation. There are some open questions, and we discussed several approaches.
From doing nothing at all, which means keeping the current status-quo, to setting a deadline date
for the deprecation like the python community did with python2. I personally like this deadline
idea, but it is perceived like a negative push by other. We all agree that the current do nothing
approach is not sustainable either. Probably the way to go is basically to be more informative. We
need to clearly communicate that choosing iptables for anything in 2020 is a bad idea. There are
additional initiatives to help on this topic, without being too aggressive. A FAQ will probably be
introduced. Eric Garver suggested we should bring nftables front and center. Given the website
still mentions iptables everywhere, we will probably refresh the web content, introduce additional
informative banners and similar things.
There was an interesting talk on the topic of nft table ownership. The idea is to attach a table,
and all the child objects, to a process. Then, we prevent any modifications to the table or the
child objects by external entities. Basically, allocating and locking a table for a certain
netlink socket. This is a nice way for ruleset managers, like firewalld, to ensure
they have full control of what s happening to their ruleset, reducing the chances for ending with
an inconsistent configuration. There is a proof-of-concept patch by Pablo to support this, and Eric
mentioned he is pretty much interested in any improvements to support this use case.
The final time block in the final session day was dedicated to talk about the next workshop.
We are all very happy we could meet. Meeting virtually is way easier (and cheaper) than in person.
Perhaps we can make it online every 3 or 6 months instead of, or in addition to, one big annual
physical event. We will see what happens next year.
That s all on my side!
set-meta command:
QSoas> set-meta pH 7.5This command sets the meta-data
pH to the value 7.5. Keep in mind that QSoas does not know anything about the meaning of the meta-data[1]. It can keep track of the meta-data you give, and manipulate them, but it will not interpret them for you. You can set several meta-data by repeating calls to set-meta, and you can display the meta-data attached to a dataset using the command show. Here is an example:
QSoas> generate-buffer 0 10 QSoas> set-meta pH 7.5 QSoas> set-meta sample "My sample" QSoas> show 0 Dataset generated.dat: 2 cols, 1000 rows, 1 segments, #0 Flags: Meta-data: pH = 7.5 sample = My sampleNote here the use of quotes around
My sample since there is a space inside the value.
Using meta-data
There are many ways to use meta-data in QSoas. In this post, we will discuss just one: using meta-data in the output file. The output file can collect data from several commands, like peak data, statistics and so on. For instance, each time the command 1 is run, a line with the information about the largest peak of the current dataset is written to the output file. It is possible to automatically add meta-data to those lines by using the /meta= option of the output command. Just listing the names of the meta-data will add them to each line of the output file.
As a full example, we'll see how one can take advantage of meta-data to determine the position of the peak of the function \(x^2 \exp (-a\,x)\) depends on \(a\). For that, we first create a script that generates the function for a certain value of \(a\), sets the meta-data a to the corresponding value, and find the peak. Let's call this file do-one.cmds (all the script files can be found in the GitHub repository):
generate-buffer 0 20 x**2*exp(-x*$ 1 ) set-meta a $ 1 1This script takes a single argument, the value of \(a\), generates the appropriate dataset, sets the meta-data
a and writes the data about the largest (and only in this case) peak to the output file. Let's now run this script with 1 as an argument:
QSoas> @ do-one.cmds 1This command generates a file
out.dat containing the following data:
## buffer what x y index width left_width right_width area generated.dat max 2.002002002 0.541340590883 100 3.4034034034 1.24124124124 2.162162162161.99999908761This gives various information about the peak found: the name of the dataset it was found in, whether it's a maximum or minimum, the x and y positions of the peak, the index in the file, the widths of the peak and its area. We are interested here mainly in the
x position.
Then, we just run this script for several values of \(a\) using run-for-each, and in particular the option /range-type=lin that makes it interpret values like 0.5..5:80 as 80 values evenly spread between 0.5 and 5. The script is called
run-all.cmds:
output peaks.dat /overwrite=true /meta=a run-for-each do-one.cmds /range-type=lin 0.5..5:80 V all /style=red-to-blueThe first line sets up the output to the output file
peaks.dat. The option /meta=a makes sure the meta a is added to each line of the output file, and /overwrite=true make sure the file is overwritten just before the first data is written to it, in order to avoid accumulating the results of different runs of the script. The last line just displays all the curves with a color gradient. It looks like this:
Running this script (with @ run-all.cmds) creates a new file peaks.dat, whose first line looks like this:
## buffer what x y index width left_width right_width area aThe column
x (the 3rd) contains the position of the peaks, and the column a (the 10th) contains the meta a (this column wasn't present in the output we described above, because we had not used yet the output /meta=a command). Therefore, to load the peak position as a function of a, one has just to run:
QSoas> load peaks.dat /columns=10,3This looks like this: Et voil ! To train further, you can:
[1] this is not exactly true. For instance, some commands like unwrap interpret the sr meta-data as a voltammetric scan rate if it is present. But this is the exception.
About QSoasQSoas is a powerful open source data analysis program that focuses on flexibility and powerful fitting capacities. It is released under the GNU General Public License. It is described in Fourmond, Anal. Chem., 2016, 88 (10), pp 5050 5052. Current version is 2.2. You can download its source code there (or clone from the GitHub repository) and compile it yourself, or buy precompiled versions for MacOS and Windows there.
word_t peek(word_t *addr, int *success);
return *addr, except that if
addr is not mapped it doesn't crash, and if success is not NULL it
is set to 0 or 1 to indicate that addr was not mapped or
mapped. If addr was not mapped the return value is meaningless.
I won't explain it in detail to leave you some fun. Basically the idea
is to install a handler for SIGSEGV: if the address is invalid, the
handler is called, which basically fixes everything by advancing a
little bit the instruction pointer, in order to skip the faulting
instruction. The dereferencing instruction is written as hardcoded
Assembly bytes, so that I know exactly how many bytes I need to skip.
Of course this is very architecture-dependent: I wrote the i386 and
amd64 variants (no x32). And I don't guarantee there are no bugs
or subtelties!
Another solution would have been to just parse /proc/self/maps
before dereferencing and check whether the pointer is in a mapped
area, but it would have suffered of a
TOCTTOU
problem: another thread might have changed the mappings between the
time when /proc/self/maps was parsed and when the pointer was
dereferenced (also, parsing that file can take a relatively long
amount of time). Another less architecture-dependent but still not
pure-C approach would have been to establish a setjmp before
attempting the dereference and longjmp-ing back from the signal
handler (but again you would need to use different setjmp contexts
in different threads to exclude race conditions).
Have fun! (and again, don't try this in real programs)
EDIT I realized I should specify the language for source code
highlighting to work decently. Now it's better!
EDIT 2 I also realized that my version of peek has problems when
there are other threads, because signal actions are per-process, not
per-thread (as I initially thought). See the comments for a better
version (though not perfect).
#define _GNU_SOURCE
#include <stdint.h>
#include <signal.h>
#include <assert.h>
#include <stdlib.h>
#include <stdio.h>
#include <ucontext.h>
#ifdef __i386__
typedef uint32_t word_t;
#define IP_REG REG_EIP
#define IP_REG_SKIP 3
#define READ_CODE __asm__ __volatile__(".byte 0x8b, 0x03\n" /* mov (%ebx), %eax */ \
".byte 0x41\n" /* inc %ecx */ \
: "=a"(ret), "=c"(tmp) : "b"(addr), "c"(tmp));
#endif
#ifdef __x86_64__
typedef uint64_t word_t;
#define IP_REG REG_RIP
#define IP_REG_SKIP 6
#define READ_CODE __asm__ __volatile__(".byte 0x48, 0x8b, 0x03\n" /* mov (%rbx), %rax */ \
".byte 0x48, 0xff, 0xc1\n" /* inc %rcx */ \
: "=a"(ret), "=c"(tmp) : "b"(addr), "c"(tmp));
#endif
static void segv_action(int sig, siginfo_t *info, void *ucontext)
(void) sig;
(void) info;
ucontext_t *uctx = (ucontext_t*) ucontext;
uctx->uc_mcontext.gregs[IP_REG] += IP_REG_SKIP;
struct sigaction peek_sigaction =
.sa_sigaction = segv_action,
.sa_flags = SA_SIGINFO,
.sa_mask = 0,
;
word_t peek(word_t *addr, int *success)
word_t ret;
int tmp, res;
struct sigaction prev_act;
res = sigaction(SIGSEGV, &peek_sigaction, &prev_act);
assert(res == 0);
tmp = 0;
READ_CODE
res = sigaction(SIGSEGV, &prev_act, NULL);
assert(res == 0);
if (success)
*success = tmp;
return ret;
int main()
int success;
word_t number = 22;
word_t value;
number = 22;
value = peek(&number, &success);
printf("%d %d\n", success, value);
value = peek(NULL, &success);
printf("%d %d\n", success, value);
value = peek((word_t*)0x1234, &success);
printf("%d %d\n", success, value);
return 0;
Uploaders field on packages with other maintainersunchecked,
dinstall, possibly others) which run on fixed schedules (e.g. dinstall runs
at 01:52 UTC, 07:52 UTC, 13:52 UTC and 19:52 UTC).
Depending on timing, I estimated that you might wait for over 7 hours (!!)
before your package is actually installable.
What s worse for me is that feedback to your upload is asynchronous. I like to
do one thing, be done with it, move to the next thing. The current setup
requires a many-minute wait and costly task switch for no good technical
reason. You might think a few minutes aren t a big deal, but when all the time I
can spend on Debian per day is measured in minutes, this makes a huge difference
in perceived productivity and fun.
The last communication I can find about speeding up this process is ganneff s
post from 2008.
How would things look like in a better world?
reportbug(1)
or manually dealing with attachments is a rather big hurdle.
For reasons I don t understand, every interaction with debbugs results in many
different email threads.
Aside from the technical implementation, I also can never remember the different
ways that Debian uses pseudo-packages for bugs and processes. I need them rarely
enough to establish a mental model of how they are set up, or working memory of
how they are used, but frequently enough to be annoyed by this.
How would things look like in a better world?
psql(1). This
is because maintainer scripts modify the alternatives database by calling shell
scripts. Without actually installing a package, you cannot know which changes it
does to the alternatives database.
pk4 needs to maintain its own cache to look up
package metadata based on the package name. Other tools parse the apt database
from scratch on every invocation. A proper database format, or at least a binary
interchange format, would go a long way.
Debian Code Search wants to ingest new
packages as quickly as possible. There used to be a
fedmsg instance for Debian, but it no
longer seems to exist. It is unclear where to get notifications from for new
packages, and where best to fetch those packages.
I have a lot of respect for Nadia Eghbal, partly because I can't help to be
jealous of her work on the economics of Free Software1. If you are not
already familiar with Eghbal, she is the author of Roads and Bridges: The
Unseen Labor Behind Our Digital Infrastructure, a great technical
report published for the Ford Foundation in 2016. You may also have caught her
excellent keynote at LCA 2017, entitled Consider the Maintainer.
Her latest book, Working in Public: The Making and Maintenance of Open Source
Software, published by Stripe Press a few months ago, is a great read and if
this topic interests you, I highly recommend it.
The book itself is simply gorgeous; bright orange, textured hardcover binding,
thick paper, wonderful typesetting it has everything to please. Well, nearly
everything. Sadly, it is only available on Amazon, exclusively in the United
States. A real let down for a book on Free and Open Source Software.
The book is divided in five chapters, namely:
Contrary to what I was expecting, the book feels more like an extension of the
LCA keynote I previously mentioned than Roads and Bridges. Indeed, as made
apparent by the following quote, Eghbal doesn't believe funding to be the
primary problem of FOSS anymore:
We still don't have a common understanding about who's doing the work, why they do it, and what work needs to be done. Only when we understand the underlying behavioral dynamics of open source today, and how it differs from its early origins, can we figure out where money fits in. Otherwise, we're just flinging wet paper towels at a brick wall, hoping that something sticks. p.184That is to say, the behavior of maintainers and the challenges they face not the eternal money problem is the real topic of this book. And it feels refreshing. When was the last time you read something on the economics of Free Software without it being mostly about what licences projects should pick and how business models can be tacked on them? I certainly can't. To be clear, I'm not sure I agree with Eghbal on this. Her having worked at Github for a few years and having interviewed mostly people in the Ruby on Rails and Javascript communities certainly shows in the form of a strong selection bias. As she herself admits, this is a book on how software on Github is produced. As much as this choice irks me (the Free Software community certainly cannot be reduced to Github), this exercise had the merit of forcing me to look at my own selection biases. As such, reading Working in Public did to me something I wasn't expecting it to do: it broke my Free Software echo chamber. Although I consider myself very familiar with the world of Free and Open Source Software, I now understand my somewhat ill-advised contempt for certain programming languages mostly JS skewed my understanding of what FOSS in 2020 really is. My Free Software world very much revolves around Debian, a project with a strong and opinionated view of Free Software, rooted in a historical and political understanding of the term. This, Eghbal argues, is not the case for a large swat of developers anymore. They are The Github Generation, people attached to Github as a platform first and foremost, and who feel "Open Source" is just a convenient way to make things. Although I could intellectualise this, before reading the book, I didn't really grok how communities akin to npm have been reshaping the modern FOSS ecosystem and how different they are from Debian itself. To be honest, I am not sure I like this tangent and it is certainly part of the reason why I had a tendency to dismiss it as a fringe movement I could safely ignore. Thanks to Nadia Eghbal, I come out of this reading more humble and certainly reminded that FOSS' heterogeneity is real and should not be idly dismissed. This book is rich in content and although I could go on (my personal notes clock-in at around 2000 words and I certainly disagree with a number of things), I'll stop here for now. Go and grab a copy already!
A couple of years ago, I moved into a new flat that comes with RJ45 sockets wired for 10 Gigabit (but currently offering 1 Gigabit) Ethernet.This also meant changing the settings on my router box for my new ISP.I took this opportunity to review my router's other settings too. I'll be blogging about these over the next few posts.
GRUB fine-tuning
One thing that had been annoying me ever since Debian migrated to systemd as /sbin/init is that boot message verbosity hasn't been the same. Previously, the cmdline option quiet merely suppressed the kernel's output to the bootscreen, but left the daemon startup messages alone. Not anymore. Nowadays, quiet produces a blank screen.
After some googling, I found the solution to that:
GRUB_CMDLINE_LINUX_DEFAULT="noquiet loglevel=5"
The former restores daemon startup messages, while the later makes the kernel output only significant notices or more serious messages. On most of my hosts, it mostly reports inconsistencies in the ACPI configuration of the BIOS.
Another setting I find useful is a reboot delay in case a kernel panic happens:
GRUB_CMDLINE_LINUX="panic=15"
This gives me enough time to snap a picture of the screen output to attach to the bug report that will follow.
issoPackage = with pkgs.python3Packages; buildPythonPackage rec pname = "isso"; version = "custom"; src = pkgs.fetchFromGitHub # Use my fork owner = "vincentbernat"; repo = pname; rev = "vbe/master"; sha256 = "0vkkvjcvcjcdzdj73qig32hqgjly8n3ln2djzmhshc04i6g9z07j"; ; propagatedBuildInputs = [ itsdangerous jinja2 misaka html5lib werkzeug bleach flask-caching ]; buildInputs = [ cffi ]; checkInputs = [ nose ]; checkPhase = '' $ python.interpreter setup.py nosetests ''; ;
"$ issoEnv /bin/gunicorn", like with a virtual
environment.
issoEnv = pkgs.python3.buildEnv.override extraLibs = [ issoPackage pkgs.python3Packages.gunicorn pkgs.python3Packages.gevent ]; ;
issoConfig = pkgs.writeText "isso.conf" '' [general] dbpath = /db/comments.db host = https://vincent.bernat.ch http://localhost:8080 notify = smtp [ ] '';
Dockerfile:
issoDockerImage = pkgs.dockerTools.buildImage name = "isso"; tag = "latest"; extraCommands = '' mkdir -p db ''; config = Cmd = [ "$ issoEnv /bin/gunicorn" "--name" "isso" "--bind" "0.0.0.0:$ port " "--worker-class" "gevent" "--workers" "2" "--worker-tmp-dir" "/dev/shm" "--preload" "isso.run" ]; Env = [ "ISSO_SETTINGS=$ issoConfig " "SSL_CERT_FILE=$ pkgs.cacert /etc/ssl/certs/ca-bundle.crt" ]; ; ;
issoEnv derivation in config.Cmd, the
whole derivation, including Isso and Gunicorn, is copied inside the
Docker image. The same applies for issoConfig, the configuration
file we created earlier, and pkgs.cacert, the derivation containing
trusted root certificates. The resulting image is 171 MB once
installed, which is comparable to the Debian Buster image generated by
the official Dockerfile.
NixOS features an abstraction to run Docker containers. It is not
currently documented in NixOS manual but you can look at the source
code of the module for the available options. I choose to use
Podman instead of Docker as the backend because it does not
require running an additional daemon.
virtualisation.oci-containers = backend = "podman"; containers = isso = image = "isso"; imageFile = issoDockerImage; ports = ["127.0.0.1:$ port :$ port "]; volumes = [ "/var/db/isso:/db" ]; ; ; ;
$ systemctl status podman-isso.service podman-isso.service Loaded: loaded (/nix/store/a66gzqqwcdzbh99sz8zz5l5xl8r8ag7w-unit-> Active: active (running) since Sun 2020-11-01 16:04:16 UTC; 4min 44s ago Process: 14564 ExecStartPre=/nix/store/95zfn4vg4867gzxz1gw7nxayqcl> Main PID: 14697 (podman) IP: 0B in, 0B out Tasks: 10 (limit: 2313) Memory: 221.3M CPU: 10.058s CGroup: /system.slice/podman-isso.service 14697 /nix/store/pn52xgn1wb2vr2kirq3xj8ij0rys35mf-podma> 14802 /nix/store/7vsba54k6ag4cfsfp95rvjzqf6rab865-conmo> nov. 01 16:04:17 web03 podman[14697]: container init (image=localhost/isso:latest) nov. 01 16:04:17 web03 podman[14697]: container start (image=localhost/isso:latest) nov. 01 16:04:17 web03 podman[14697]: container attach (image=localhost/isso:latest) nov. 01 16:04:19 web03 conmon[14802]: INFO: connected to SMTP server nov. 01 16:04:19 web03 conmon[14802]: INFO: connected to https://vincent.bernat.ch nov. 01 16:04:19 web03 conmon[14802]: [INFO] Starting gunicorn 20.0.4 nov. 01 16:04:19 web03 conmon[14802]: [INFO] Listening at: http://0.0.0.0:8080 (1) nov. 01 16:04:19 web03 conmon[14802]: [INFO] Using worker: gevent nov. 01 16:04:19 web03 conmon[14802]: [INFO] Booting worker with pid: 8 nov. 01 16:04:19 web03 conmon[14802]: [INFO] Booting worker with pid: 9
comments.luffy.cx to the container. NixOS provides a simple
integration to grab a Let s Encrypt certificate.
services.nginx.virtualHosts."comments.luffy.cx" = root = "/data/webserver/comments.luffy.cx"; enableACME = true; forceSSL = true; extraConfig = '' access_log /var/log/nginx/comments.luffy.cx.log anonymous; ''; locations."/" = proxyPass = "http://127.0.0.1:$ port "; extraConfig = '' proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_hide_header Set-Cookie; proxy_hide_header X-Set-Cookie; proxy_ignore_headers Set-Cookie; ''; ; ; security.acme.certs."comments.luffy.cx" = email = lib.concatStringsSep "@" [ "letsencrypt" "vincent.bernat.ch" ]; ;
rr.ntt.net or whois.radb.net can be a bottleneck. Updating many
filters may take several tens of minutes, depending on the load:
$ time bgpq4 -h whois.radb.net AS-HURRICANE wc -l 909869 1.96s user 0.15s system 2% cpu 1:17.64 total $ time bgpq4 -h rr.ntt.net AS-HURRICANE wc -l 927865 1.86s user 0.08s system 12% cpu 14.098 total
$ git clone https://github.com/vincentbernat/irrd-legacy.git -b blade/master $ cd irrd-legacy $ docker build . -t irrd-snapshot:latest [ ] Successfully built 58c3e83a1d18 Successfully tagged irrd-snapshot:latest $ docker container run --rm --detach --publish=43:43 irrd-snapshot 4879cfe7413075a0c217089dcac91ed356424c6b88808d8fcb01dc00eafcc8c7 $ time bgpq4 -h localhost AS-HURRICANE wc -l 904137 1.72s user 0.11s system 96% cpu 1.881 total
rr.ntt.net:
NTTCOM, RADB, RIPE, ALTDB, BELL, LEVEL3, RGNET, APNIC, JPIRR, ARIN,
BBOI, TC, AFRINIC, ARIN-WHOIS, and REGISTROBR. However, it misses
RPKI.2 Feel free to adapt!
The image can be scheduled to be rebuilt daily or weekly, depending on
your needs. The repository includes a .gitlab-ci.yaml
file automating the build and triggering the compilation
of all filters by your CI/CD upon success.
rpki-client and
transformed into RPSL objects to be imported in IRRd.
whois and
bgpq4. The first one allows you to do a query with the WHOIS
protocol:
$ whois -BrG 2a0a:e805:400::/40 [ ] inet6num: 2a0a:e805:400::/40 netname: FR-BLADE-CUSTOMERS-DE country: DE geoloc: 50.1109 8.6821 admin-c: BN2763-RIPE tech-c: BN2763-RIPE status: ASSIGNED mnt-by: fr-blade-1-mnt remarks: synced with cmdb created: 2020-05-19T08:04:58Z last-modified: 2020-05-19T08:04:58Z source: RIPE route6: 2a0a:e805:400::/40 descr: Blade IPv6 - AMS1 origin: AS64476 mnt-by: fr-blade-1-mnt remarks: synced with cmdb created: 2019-10-01T08:19:34Z last-modified: 2020-05-19T08:05:00Z source: RIPE
$ bgpq4 -6 -S RIPE -b AS64476 NN = [ 2a0a:e805::/40, 2a0a:e805:100::/40, 2a0a:e805:300::/40, 2a0a:e805:400::/40, 2a0a:e805:500::/40 ];
Notice I recommend that you read Writing a custom Ansible module as an introduction, as well as Syncing MySQL tables for a more instructive example.
- name: prepare RIPE objects irr_sync: irr: RIPE mntner: fr-blade-1-mnt source: whois-ripe.txt register: irr
- name: sign RIPE objects shell: cmd: gpg --batch --user noc@example.com --clearsign stdin: " irr.objects " register: signed check_mode: false changed_when: false - name: update RIPE objects by email mail: subject: "NEW: update for RIPE" from: noc@example.com to: "auto-dbm@ripe.net" cc: noc@example.com host: smtp.example.com port: 25 charset: us-ascii body: " signed.stdout "
key-cert object and adding it as a valid authentication
method for the corresponding mntner object:
key-cert: PGPKEY-A791AAAB certif: -----BEGIN PGP PUBLIC KEY BLOCK----- certif: certif: mQGNBF8TLY8BDADEwP3a6/vRhEERBIaPUAFnr23zKCNt5YhWRZyt50mKq1RmQBBY [ ] certif: -----END PGP PUBLIC KEY BLOCK----- mnt-by: fr-blade-1-mnt source: RIPE mntner: fr-blade-1-mnt [ ] auth: PGPKEY-A791AAAB mnt-by: fr-blade-1-mnt source: RIPE
module_args = dict( irr=dict(type='str', required=True), mntner=dict(type='str', required=True), source=dict(type='path', required=True), ) result = dict( changed=False, ) module = AnsibleModule( argument_spec=module_args, supports_check_mode=True )
whois command to retrieve all
the objects from the provided maintainer.
# Per-IRR variations: # - whois server whois = 'ARIN': 'rr.arin.net', 'RIPE': 'whois.ripe.net', 'APNIC': 'whois.apnic.net' # - whois options options = 'ARIN': ['-r'], 'RIPE': ['-BrG'], 'APNIC': ['-BrG'] # - objects excluded from synchronization excluded = ["domain"] if irr == "ARIN": # ARIN does not return these objects excluded.extend([ "key-cert", "mntner", ]) # Grab existing objects args = ["-h", whois[irr], "-s", irr, *options[irr], "-i", "mnt-by", module.params['mntner']] proc = subprocess.run("whois", *args, capture_output=True) if proc.returncode != 0: raise AnsibleError( f"unable to query whois: args ") output = proc.stdout.decode('ascii') got = extract(output, excluded)
whois command and the
objects to exclude from synchronization. The second part invokes the
whois command, requesting all objects whose mnt-by field is the
provided maintainer. Here is an example of output:
$ whois -h whois.ripe.net -s RIPE -BrG -i mnt-by fr-blade-1-mnt [ ] inet6num: 2a0a:e805:300::/40 netname: FR-BLADE-CUSTOMERS-FR country: FR geoloc: 48.8566 2.3522 admin-c: BN2763-RIPE tech-c: BN2763-RIPE status: ASSIGNED mnt-by: fr-blade-1-mnt remarks: synced with cmdb created: 2020-05-19T08:04:59Z last-modified: 2020-05-19T08:04:59Z source: RIPE [ ] route6: 2a0a:e805:300::/40 descr: Blade IPv6 - PA1 origin: AS64476 mnt-by: fr-blade-1-mnt remarks: synced with cmdb created: 2019-10-01T08:19:34Z last-modified: 2020-05-19T08:05:00Z source: RIPE [ ]
extract() function. It parses and
normalizes the results into a dictionary mapping object names to
objects. We store the result in the got variable.
def extract(raw, excluded): """Extract objects.""" # First step, remove comments and unwanted lines objects = "\n".join([obj for obj in raw.split("\n") if not obj.startswith(( "#", "%", ))]) # Second step, split objects objects = [RPSLObject(obj.strip()) for obj in re.split(r"\n\n+", objects) if obj.strip() and not obj.startswith( tuple(f" x :" for x in excluded))] # Last step, put objects in a dict objects = repr(obj): obj for obj in objects return objects
RPSLObject() is a class enabling normalization and comparison of
objects. Look at the module code for more details.
>>> output=""" ... inet6num: 2a0a:e805:300::/40 ... [ ] ... """ >>> pprint( k: str(v) for k,v in extract(output, excluded=[]) ) '<Object:inet6num:2a0a:e805:300::/40>': 'inet6num: 2a0a:e805:300::/40\n' 'netname: FR-BLADE-CUSTOMERS-FR\n' 'country: FR\n' 'geoloc: 48.8566 2.3522\n' 'admin-c: BN2763-RIPE\n' 'tech-c: BN2763-RIPE\n' 'status: ASSIGNED\n' 'mnt-by: fr-blade-1-mnt\n' 'remarks: synced with cmdb\n' 'source: RIPE', '<Object:route6:2a0a:e805:300::/40>': 'route6: 2a0a:e805:300::/40\n' 'descr: Blade IPv6 - PA1\n' 'origin: AS64476\n' 'mnt-by: fr-blade-1-mnt\n' 'remarks: synced with cmdb\n' 'source: RIPE'
wanted dictionary using the same structure, thanks
to the extract() function we can use verbatim:
with open(module.params['source']) as f: source = f.read() wanted = extract(source, excluded)
got and wanted to build the diff
object:
if got != wanted: result['changed'] = True if module._diff: result['diff'] = [ dict(before_header=k, after_header=k, before=str(got.get(k, "")), after=str(wanted.get(k, ""))) for k in set((*wanted.keys(), *got.keys())) if k not in wanted or k not in got or wanted[k] != got[k]]
source variable) and let
the IRR ignore unmodified objects. We also append the objects to be
deleted by adding a delete: attribute to each them them.
# We send all source objects and deleted objects. deleted_mark = f" 'delete:':16 deleted by CMDB" deleted = "\n\n".join([f" got[k].raw \n deleted_mark " for k in got if k not in wanted]) result['objects'] = f" source \n\n deleted " module.exit_json(**result)
--diff and --check flags. It does not return
anything if no change is detected. It can work with APNIC, RIPE and
ARIN. It is not perfect: it may not detect some changes,3 it
is not able to modify objects not owned by the provided
maintainer4 and some attributes cannot be modified, requiring
to manually delete and recreate the updated object.5 However,
this module should automate 95% of your needs.
key-cert and mntner objects
and therefore we cannot detect changes in them. It is also not
possible to detect changes to the auth mechanisms of a mntner
object.
inetnum object requires
deleting and recreating the object.
mipsel architecture and
so that's not an option.
fsck.ext4 -pf /dev/sde2
which failed for some reason and so I moved to an interactive repair:
fsck.ext4 -f /dev/sde2
Once all of the errors were fixed, I ran a full surface scan to update the
list of bad blocks:
fsck.ext4 -c /dev/sde2
Finally, I forced another check to make sure that everything was fixed at
the filesystem level:
fsck.ext4 -f /dev/sde2
dmesg log:
scsi 8:0:0:0: Direct-Access KINGSTON SA400S37120 SBFK PQ: 0 ANSI: 6
sd 8:0:0:0: Attached scsi generic sg4 type 0
sd 8:0:0:0: [sde] 234441644 512-byte logical blocks: (120 GB/112 GiB)
sd 8:0:0:0: [sde] Write Protect is off
sd 8:0:0:0: [sde] Mode Sense: 31 00 00 00
sd 8:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 8:0:0:0: [sde] Optimal transfer size 33553920 bytes
Alternate GPT is invalid, using primary GPT.
sde: sde1 sde2
I therefore checked to see if the partition table looked fine and got the
following:
$ fdisk -l /dev/sde
GPT PMBR size mismatch (234441643 != 234441647) will be corrected by write.
The backup GPT table is not on the end of the device. This problem will be corrected by write.
Disk /dev/sde: 111.8 GiB, 120034123776 bytes, 234441648 sectors
Disk model: KINGSTON SA400S3
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 799CD830-526B-42CE-8EE7-8C94EF098D46
Device Start End Sectors Size Type
/dev/sde1 2048 8390655 8388608 4G Linux swap
/dev/sde2 8390656 234441614 226050959 107.8G Linux filesystem
It turns out that all I had to do, since only the backup / alternate GPT
partition table was corrupt and the primary one was fine, was to re-write
the partition table:
$ fdisk /dev/sde
Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
GPT PMBR size mismatch (234441643 != 234441647) will be corrected by write.
The backup GPT table is not on the end of the device. This problem will be corrected by write.
Command (m for help): w
The partition table has been altered.
Syncing disks.
smartctl -a /dev/sda
That didn't show any errors and so I kicked off an extended test:
smartctl -t long /dev/sda
which ran for 30 minutes and then passed without any errors.
The mystery remains unsolved.
Previously: v5.6
Linux v5.7 was released at the end of May. Here s my summary of various security things that caught my attention:
arm64 kernel pointer authenticationPACIASP which protects the saved stack pointer, similar to the existing CONFIG_STACKPROTECTOR feature, only faster. This also paves the way to sign and check pointers stored in the heap, as a way to defeat function pointer overwrites in those memory regions too. Since the behavior is different from the traditional stack protector, Amit Daniel Kachhap added an LKDTM test for PAC as well.
BPF LSMexecve() deadlock refactoringexec_id counter to 64 bits to avoid the possibility of the counter wrapping and allowing an attacker to send arbitrary signals to processes they normally shouldn t be able to.
slub freelist obfuscation improvementsCONFIG_SLAB_FREELIST_HARDENED s freelist pointer content obfuscation, I improved their bit diffusion, which makes attacks require significantly more memory content exposures to defeat the obfuscation. As part of the conversation, Vitaly Nikolenko pointed out that the freelist pointer s location made it relatively easy to target too (for either disclosures or overwrites), so I moved it away from the edge of the slab, making it harder to reach through small-sized overflows (which usually target the freelist pointer). As it turns out, there were a few assumptions in the kernel about the location of the freelist pointer, which had to also get cleaned up.
RISCV page table dumpingPTDUMP_DEBUGFS), visible in /sys/kernel/debug/kernel_page_tables.
array index bounds checkingint foo[5]; ... foo[8] = bar;The long version gets complicated by the evolution of flexible array structure members, so we ll pause for a moment and skim the surface of this topic. While things like
CONFIG_FORTIFY_SOURCE try to catch these kinds of cases in the memcpy() and strcpy() family of functions, it doesn t catch it in open-coded array indexing, as seen in the code above. GCC has a warning (-Warray-bounds) for these cases, but it was disabled by Linus because of all the false positives seen due to fake flexible array members. Before flexible arrays were standardized, GNU C supported zero sized array members. And before that, C code would use a 1-element array. These were all designed so that some structure could be the header in front of some data blob that could be addressable through the last structure member:
/* 1-element array */
struct foo
...
char contents[1];
;
/* GNU C extension: 0-element array */
struct foo
...
char contents[0];
;
/* C standard: flexible array */
struct foo
...
char contents[];
;
instance = kmalloc(sizeof(struct foo) + content_size);
Converting all the zero- and one-element array members to flexible arrays is one of Gustavo A. R. Silva s goals, and hundreds of these changes started landing. Once fixed, -Warray-bounds can be re-enabled. Much more detail can be found in the kernel s deprecation docs.
However, that will only catch the visible at compile time cases. For runtime checking, the Undefined Behavior Sanitizer has an option for adding runtime array bounds checking for catching things like this where the compiler cannot perform a static analysis of the index values:
int foo[5];
...
for (i = 0; i < some_argument; i++)
...
foo[i] = bar;
...
It was, however, not separate (via kernel Kconfig) until Elena Petrova and I split it out into CONFIG_UBSAN_BOUNDS, which is fast enough for production kernel use. With this enabled, it's now possible to instrument the kernel to catch these conditions, which seem to come up with some regularity in Wi-Fi and Bluetooth drivers for some reason. Since UBSAN (and the other Sanitizers) only WARN() by default, system owners need to set panic_on_warn=1 too if they want to defend against attacks targeting these kinds of flaws. Because of this, and to avoid bloating the kernel image with all the warning messages, I introduced CONFIG_UBSAN_TRAP which effectively turns these conditions into a BUG() without needing additional sysctl settings.
Fixing "additive" snprintf() usagesprintf()'s return value to increment a pointer into a string, and build a string with more sprintf() calls:
/* safe if strlen(foo) + 1 < sizeof(string) */ wrote = sprintf(string, "Foo: %s\n", foo); /* overflows if strlen(foo) + strlen(bar) > sizeof(string) */ wrote += sprintf(string + wrote, "Bar: %s\n", bar); /* writing way beyond the end of "string" now ... */ wrote += sprintf(string + wrote, "Baz: %s\n", baz);The risk is that if these calls eventually walk off the end of the
string buffer, it will start writing into other memory and create some bad situations. Switching these to snprintf() does not, however, make anything safer, since snprintf() returns how much it would have written:
/* safe, assuming available <= sizeof(string), and for this example * assume strlen(foo) < sizeof(string) */ wrote = snprintf(string, available, "Foo: %s\n", foo); /* if (strlen(bar) > available - wrote), this is still safe since the * write into "string" will be truncated, but now "wrote" has been * incremented by how much snprintf() *would* have written, so "wrote" * is now larger than "available". */ wrote += snprintf(string + wrote, available - wrote, "Bar: %s\n", bar); /* string + wrote is beyond the end of string, and availabe - wrote wraps * around to a giant positive value, making the write effectively * unbounded. */ wrote += snprintf(string + wrote, available - wrote, "Baz: %s\n", baz);So while the first overflowing call would be safe, the next one would be targeting beyond the end of the array and the size calculation will have wrapped around to a giant limit. Replacing this idiom with
scnprintf() solves the issue because it only reports what was actually written. To this end, Takashi Iwai has been landing a bunch scnprintf() fixes.
That's it for now! Let me know if there is anything else you think I should mention here. Next up: Linux v5.8.
2020, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
vrrp_instance gateway1 state BACKUP # interface eth0 # virtual_router_id 12 # priority 101 # virtual_ipaddress 2001:db8:ff/64
state keyword in instructs Keepalived to not take the leader
role when starting. Otherwise, incoming nodes create a temporary
disruption by taking over the IP address until the election settles.
The interface keyword in defines the interface for sending and
receiving VRRP packets. It is also the default interface to configure
the virtual IP address. The virtual_router_id directive in is
common to all nodes sharing the virtual IP. The priority keyword in
helps choosing which router will be elected as leader. If you need
more information around Keepalived, be sure to check the
documentation.
VRRP design is tied to Ethernet networks and requires a
multicast-enabled network for communication between nodes. In some
environments, notably public clouds, multicast is unavailable. In this
case, Keepalived can send VRRP packets using unicast:
vrrp_instance gateway1 state BACKUP interface eth0 virtual_router_id 12 priority 101 unicast_peer 2001:db8::11 2001:db8::12 virtual_ipaddress 2001:db8:ff/64 dev lo
notify_* scripts.
Until version 2.21 (not released yet), the interface directive is
mandatory and Keepalived will transmit and receive VRRP packets on
this interface only. If peers are reachable through several
interfaces, like on a BGP on the host setup, you need a
workaround. A simple one is to use a VXLAN interface:
$ ip -6 link add keepalived6 type vxlan id 6 dstport 4789 local 2001:db8::10 nolearning $ bridge fdb append 00:00:00:00:00:00 dev keepalived6 dst 2001:db8::11 $ bridge fdb append 00:00:00:00:00:00 dev keepalived6 dst 2001:db8::12 $ ip link set up dev keepalived6
vrrp_instance gateway1 state BACKUP interface keepalived6 mcast_src_ip 2001:db8::10 virtual_router_id 12 priority 101 virtual_ipaddress 2001:db8:ff/64 dev lo
unicast_peer can be used without
the interface directive. I think using VXLAN is still a
neat trick applicable to other situations where communication using
broadcast or multicast is needed, while the underlying network provide
no support for this.
netbox.netbox collection from Ansible Galaxy
provides several modules to update NetBox objects:
- name: create a device in NetBox netbox_device: netbox_url: http://netbox.local netbox_token: s3cret data: name: to3-p14.sfo1.example.com device_type: QFX5110-48S device_role: Compute Switch site: SFO1
Notice I recommend that you read Writing a custom Ansible module as an introduction, as well as Syncing MySQL tables for a first simpler example.
netbox_sync: source: netbox.yaml api: https://netbox.example.com token: s3cret
devices: ad2-p6.sfo1.example.com: datacenter: sfo1 manufacturer: Cisco model: Catalyst 2960G-48TC-L role: net_tor_oob_switch to1-p6.sfo1.example.com: datacenter: sfo1 manufacturer: Juniper model: QFX5110-48S role: net_tor_gpu_switch # [ ] ips: - device: ad2-p6.example.com ip: 172.31.115.18/21 interface: oob - device: to1-p6.example.com ip: 172.31.115.33/21 interface: oob - device: to1-p6.example.com ip: 172.31.254.33/32 interface: lo0.0 # [ ]
cmdb. Most objects in NetBox accept tags.
module_args = dict( source=dict(type='path', required=True), api=dict(type='str', required=True), token=dict(type='str', required=True, no_log=True), max_workers=dict(type='int', required=False, default=10) ) result = dict( changed=False ) module = AnsibleModule( argument_spec=module_args, supports_check_mode=True )
Synchronizer abstract class. For
each kind of object, a concrete class is built with the appropriate
class attributes to tune its behaviour and a wanted() method to
provide the objects we want.
I am not explaining the abstract class code here. Have a look at the
source if you want.
class SyncTags(Synchronizer): app = "extras" table = "tags" key = "name" def wanted(self): return "cmdb": dict( slug="cmdb", color="8bc34a", description="synced by network CMDB")
app and table attributes defines the NetBox objects we want
to manipulate. The key attribute is used to determine how to lookup
for existing objects. In this example, we want to lookup tags using
their names.
The wanted() method is expected to return a dictionary mapping
object keys to the list of wanted attributes. Here, the keys are tag
names and we create only one tag, cmdb, with the provided slug,
color and description. This is the tag we will use to mark the objects
we create or modify.
If the tag does not exist, it is created. If it exists, the provided
attributes are updated. Other attributes are left untouched.
We also want to create a specific tenant for objects accepting such an
attribute (devices and IP addresses):
class SyncTenants(Synchronizer): app = "tenancy" table = "tenants" key = "name" def wanted(self): return "Network": dict(slug="network", description="Network team")
wanted() method uses the information provided in the YAML file: it
walks the devices and builds a set of datacenter names.
class SyncSites(Synchronizer): app = "dcim" table = "sites" key = "name" only_on_create = ("status", "slug") def wanted(self): result = set(details["datacenter"] for details in self.source['devices'].values() if "datacenter" in details) return k: dict(slug=k, status="planned") for k in result
only_on_create attribute, the specified
attributes are not updated if they are different. The goal of this
synchronizer is mostly to collect the references to the different
sites for other objects.
>>> pprint(SyncSites(**sync_args).wanted()) 'sfo1': 'slug': 'sfo1', 'status': 'planned' , 'chi1': 'slug': 'chi1', 'status': 'planned' , 'nyc1': 'slug': 'nyc1', 'status': 'planned'
only_on_create attribute:
class SyncManufacturers(Synchronizer): app = "dcim" table = "manufacturers" key = "name" def wanted(self): result = set(details["manufacturer"] for details in self.source['devices'].values() if "manufacturer" in details) return k: "slug": slugify(k) for k in result
foreign attribute linking
a NetBox attribute to the synchronizer handling it.
class SyncDeviceTypes(Synchronizer): app = "dcim" table = "device_types" key = "model" foreign = "manufacturer": SyncManufacturers def wanted(self): result = set((details["manufacturer"], details["model"]) for details in self.source['devices'].values() if "model" in details) return k[1]: dict(manufacturer=k[0], slug=slugify(k[1])) for k in result
wanted() method refers to the manufacturer using its key
attribute. In this case, this is the manufacturer name.
>>> pprint(SyncManufacturers(**sync_args).wanted()) 'Cisco': 'slug': 'cisco' , 'Dell': 'slug': 'dell' , 'Juniper': 'slug': 'juniper' >>> pprint(SyncDeviceTypes(**sync_args).wanted()) 'ASR 9001': 'manufacturer': 'Cisco', 'slug': 'asr-9001' , 'Catalyst 2960G-48TC-L': 'manufacturer': 'Cisco', 'slug': 'catalyst-2960g-48tc-l' , 'MX10003': 'manufacturer': 'Juniper', 'slug': 'mx10003' , 'QFX10002-36Q': 'manufacturer': 'Juniper', 'slug': 'qfx10002-36q' , 'QFX10002-72Q': 'manufacturer': 'Juniper', 'slug': 'qfx10002-72q' , 'QFX5110-32Q': 'manufacturer': 'Juniper', 'slug': 'qfx5110-32q' , 'QFX5110-48S': 'manufacturer': 'Juniper', 'slug': 'qfx5110-48s' , 'QFX5200-32C': 'manufacturer': 'Juniper', 'slug': 'qfx5200-32c' , 'S4048-ON': 'manufacturer': 'Dell', 'slug': 's4048-on' , 'S6010-ON': 'manufacturer': 'Dell', 'slug': 's6010-on'
class SyncDeviceRoles(Synchronizer): app = "dcim" table = "device_roles" key = "name" def wanted(self): result = set(details["role"] for details in self.source['devices'].values() if "role" in details) return k: dict(slug=slugify(k), color="8bc34a") for k in result
class SyncDevices(Synchronizer): app = "dcim" table = "devices" key = "name" foreign = "device_role": SyncDeviceRoles, "device_type": SyncDeviceTypes, "site": SyncSites, "tenant": SyncTenants remove_unused = 10 def wanted(self): return name: dict(device_role=details["role"], device_type=details["model"], site=details["datacenter"], tenant="Network") for name, details in self.source['devices'].items() if "datacenter", "model", "role" <= set(details.keys())
remove_unused attribute is a safety implemented to fail if we
have to delete more than 10 devices: this may be the indication there
is a bug somewhere, unless one of your datacenter suddenly caught
fire.
>>> pprint(SyncDevices(**sync_args).wanted()) 'ad2-p6.sfo1.example.com': 'device_role': 'net_tor_oob_switch', 'device_type': 'Catalyst 2960G-48TC-L', 'site': 'sfo1', 'tenant': 'Network' , 'to1-p6.sfo1.example.com': 'device_role': 'net_tor_gpu_switch', 'device_type': 'QFX5110-48S', 'site': 'sfo1', 'tenant': 'Network' , [ ]
class SyncIPs(Synchronizer): app = "ipam" table = "ip-addresses" key = "address" foreign = "tenant": SyncTenants remove_unused = 1000 def wanted(self): wanted = for details in self.source['ips']: if details['ip'] in wanted: wanted[details['ip']]['description'] = \ f" details['device'] (and others)" else: wanted[details['ip']] = dict( tenant="Network", status="active", dns_name="", # information is present in DNS description=f" details['device'] : details['interface'] ", role=None, vrf=None) return wanted
cmdb, then those
already attached to an interface:
def get(self, key): """Grab IP address from NetBox.""" # There may be duplicate. We need to grab the "best". results = super(Synchronizer, self).get(key) if len(results) == 0: return None if len(results) == 1: return results[0] scores = [0]*len(results) for idx, result in enumerate(results): if "cmdb" in result.tags: scores[idx] += 10 if result.interface is not None: scores[idx] += 5 return sorted(zip(scores, results), reverse=True, key=lambda k: k[0])[0][1]
source = yaml.safe_load(open(module.params['source'])) netbox = pynetbox.api(module.params['api'], token=module.params['token']) sync_args = dict( module=module, netbox=netbox, source=source, before= , after= ) synchronizers = [synchronizer(**sync_args) for synchronizer in [ SyncTags, SyncTenants, SyncSites, SyncManufacturers, SyncDeviceTypes, SyncDeviceRoles, SyncDevices, SyncIPs ]]
prepare() method whose goal is to compute
the current and wanted states. It returns True in case of a
difference:
# Check what needs to be synchronized try: for synchronizer in synchronizers: result['changed'] = synchronizer.prepare() except AnsibleError as e: result['msg'] = e.message module.fail_json(**result)
sync_args["before"][table] and
sync_args["after"][table] where table is the name of the table for
a given NetBox object type. The diff object is a bit elaborate as
it is built table by table. This enables Ansible to display the name
of each table before the diff representation:
# Compute the diff if module._diff and result['changed']: result['diff'] = [ dict( before_header=table, after_header=table, before=yaml.safe_dump(sync_args["before"][table]), after=yaml.safe_dump(sync_args["after"][table])) for table in sync_args["after"] if sync_args["before"][table] != sync_args["after"][table] ] # Stop here if check mode is enabled or if no change if module.check_mode or not result['changed']: module.exit_json(**result)
synchronize() method to apply
changes and a cleanup() method to delete unwanted objects. Order is
important due to the relation between the objects.
# Synchronize for synchronizer in synchronizers: synchronizer.synchronize() for synchronizer in synchronizers[::-1]: synchronizer.cleanup() module.exit_json(**result)
netbox.netbox collection, the logic is written in
Python instead of trying to glue Ansible tasks together. I believe
this is both more flexible and easier to read, notably when trying to
delete outdated objects. While I did not test it, it should also be
faster. An alternative would have been to reuse code from the
netbox.netbox collection, as it contains similar primitives.
Unfortunately, I didn t think of it until now.
In our monthly reports, we summarise the things that we have been up to over the past month. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced from the original free software source code to the pre-compiled binaries we install on our systems. If you re interested in contributing to the project, please visit our main website.
One hiccup we ve encountered in SecureDrop development is that not all Python wheels can be built reproducibly. We ship multiple (Python) projects in Debian packages, with Python dependencies included in those packages as wheels. In order for our Debian packages to be reproducible, we need that wheel build process to also be reproducibleParallel to this, transparencylog.com was also launched, a service that verifies the contents of URLs against a publicly recorded cryptographic log. It keeps an append-only log of the cryptographic digests of all URLs it has seen. (GitHub repo)
On 18th September, Bernhard M. Wiedemann will give a presentation in German, titled Wie reproducible builds Software sicherer machen ( How reproducible builds make software more secure ) at the Internet Security Digital Days 2020 conference.
Holger gave a talk titled Reproducing Bullseye in practice , focusing on independently verifying that the binaries distributed from ftp.debian.org are made from their claimed sources. It also served as a general update on the status of reproducible builds within Debian. The video (145 MB) and slides are available.
There were also a number of other talks that involved Reproducible Builds too. For example, the Malayalam language mini-conference had a talk titled , ? ( I want to join Debian, what should I do? ) presented by Praveen Arimbrathodiyil, the Clojure Packaging Team BoF session led by Elana Hashman, as well as Where is Salsa CI right now? that was on the topic of Salsa, the collaborative development server that Debian uses to provide the necessary tools for package maintainers, packaging teams and so on.
Jonathan Bustillos (Jathan) also gave a talk in Spanish titled Un camino verificable desde el origen hasta el binario ( A verifiable path from source to binary ). (Video, 88MB)
After many years of development work, the compiler for the Rust programming language now generates reproducible binary code. This generated some general discussion on Reddit on the topic of reproducibility in general.
Paul Spooren posted a request for comments to OpenWrt s openwrt-devel mailing list asking for clarification on when to raise the PKG_RELEASE identifier of a package. This is needed in order to successfully perform rebuilds in a reproducible builds context.
In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update.
Chris Lamb provided some comments and pointers on an upstream issue regarding the reproducibility of a Snap / SquashFS archive file. [ ]
.buildinfo build certificates have been tainted on the official Debian build servers, as these environments have files underneath the /usr/local/sbin directory [ ]. He also filed against bug for debrebuild after spotting that it can fail to download packages from snapshot.debian.org [ ].
This month, several issues were uncovered (or assisted) due to the efforts of reproducible builds.
For instance, Debian bug #968710 was filed by Simon McVittie, which describes a problem with detached debug symbol files (required to generate a traceback) that is unlikely to have been discovered without reproducible builds. In addition, Jelmer Vernooij called attention that the new Debian Janitor tool is using the property of reproducibility (as well as diffoscope when applying archive-wide changes to Debian:
New merge proposals also include a link to the diffoscope diff between a vanilla build and the build with changes. Unfortunately these can be a bit noisy for packages that are not reproducible yet, due to the difference in build environment between the two builds. [ ]56 reviews of Debian packages were added, 38 were updated and 24 were removed this month adding to our knowledge about identified issues. Specifically, Chris Lamb added and categorised the
nondeterministic_version_generated_by_python_param and the lessc_nondeterministic_keys toolchain issues. [ ][ ]
Holger Levsen sponsored Lukas Puehringer s upload of the python-securesystemslib pacage, which is a dependency of in-toto, a framework to secure the integrity of software supply chains. [ ]
Lastly, Chris Lamb further refined his merge request against the debian-installer component to allow all arguments from sources.list files (such as [check-valid-until=no]) in order that we can test the reproducibility of the installer images on the Reproducible Builds own testing infrastructure and sent a ping to the team that maintains that code.
asymptote (shell/Perl date)getfem (embeds datetime and user, submitted via email)getdp (hostname and user)getdp (user)guix (disable parallelism)httpcomponents-client (Java documentation generator readdir order)kuberlr (date)lal (date and time issue, submitted via email)libmesh (host)OBS (discuss how to track old build prjconf metadata in buildinfo)openblas (disable CPU detection)openfoam-selector (date)perl (toolchain, date)python-blosc (CPU detection)python-eventlet (fails to build far in the future)rna-star (date and hostname)trilinos (date)xz/b4 (workaround CPU count influencing output, reported upstream)json-c (forwarded upstream).nmh.golang-gonum-v1-plot.chirp.pixelmed-codec.debhelper.muroar.serd.pencil2d.tpot.evolution.aflplusplus.plexus-archiver (timezone/DST issue)libjpeg-turbo.jack-audio-connection-kit.glusterfs.155, 156, 157 and 158 to Debian:
<!ENTITY> declarations inside the Document Type Definition (DTD), or when a DTD or entity references an external resource. (#212)pgpdump(1) can successfully parse some binary files, so check that the parsed output contains something sensible before accepting it. [ ]gnumeric from the Debian build-dependencies as it has been removed from the testing distribution. (#968742)fallback_recognises to prevent matching .xsb binary XML files.file(1) returns data . (#211)ppudump version does not match our file header. [ ]repr(object) output in Calling external command messages. [ ].ppu files requires ppudump version 3.2.0 or higher. [ ]setup.py that diffoscope works with Python version 3.8 [ ] and Frazer Clews applied some Pylint suggestions [ ] and removed some deprecated methods [ ].
This month, Chris Lamb updated the main Reproducible Builds website and documentation to:
SOURCE_DATE_EPOCH age. [ ]
The Reproducible Builds project operate a Jenkins-based testing framework to power tests.reproducible-builds.org. This month, Holger Levsen made the following changes:
arm64 architecture. [ ]armhf. [ ][ ][ ]ruby-jekyll-polyglot package to needed for the recently-added internationalisation and translation support on the Reproducible Builds website. [ ]jenkins.debian.net, www.reproducible-builds.org, diffoscope.org, buildinfos.debian.net, etc.). [ ][ ][ ][ ][ ]arm64 architecture anymore. [ ]If you think you know how to spread the word about reproducibility in the context of Bitcoin wallets through WalletScrutiny, your contributions are highly welcome on this PR [ ]Julien Lepiller posted to the list linking to a blog post by Tavis Ormandy titled You don t need reproducible builds. Morten Linderud (foxboron) responded with a clear rebuttal that Tavis was only considering the narrow use-case of proprietary vendors and closed-source software. He additionally noted that the criticism that reproducible builds cannot prevent against backdoors being deliberately introduced into the upstream source ( bugdoors ) are decidedly (and deliberately) outside the scope of reproducible builds to begin with. Chris Lamb included the Reproducible Builds mailing list in a wider discussion regarding a tentative proposal to include
.buildinfo files in .deb packages, adding his remarks regarding requiring a custom tool in order to determine whether generated build artifacts are identical in a reproducible context. [ ]
Jonathan Bustillos (Jathan) posted a quick email to the list requesting whether there was a list of To do tasks in Reproducible Builds.
Lastly, Chris Lamb responded at length to a query regarding the status of reproducible builds for Debian ISO or installation images. He noted that most of the technical work has been performed but there are at least four issues until they can be generally advertised as such . He pointed that the privacy-oriented Tails operation system, which is based directly on Debian, has had reproducible builds for a number of years now. [ ]
#reproducible-builds on irc.oftc.net.
rb-general@lists.reproducible-builds.org
cisco.iosxr collection from Ansible Galaxy
provides an iosxr_user module to manage local users,
along with their SSH keys. However, the module is quite slow, do not
display a diff for changed SSH keys, never signal change when a key is
modified, and does not delete obsolete keys. Let s write a custom
Ansible module managing only the SSH keys while fixing these issues.
Notice I recommend that you read Writing a custom Ansible module as an introduction.
$ awk ' print $2 ' id_rsa.pub \ base64 -d \ > publickey_vincent.raw
harddisk:/publickey_vincent.raw
and import it for the current user with the following IOS command:
crypto key import authentication rsa harddisk:/publickey_vincent.b64
root-system group:
username vincent group root-lr group root-system
admin crypto key import authentication rsa username cedric harddisk:/publickey_cedric.b64
iosxr_users: keys: vincent: ssh-rsa AAAAB3NzaC1yc2EAA[ ]ymh+YrVWLZMJR cedric: ssh-rsa AAAAB3NzaC1yc2EAA[ ]RShPA8w/8eC0n
iosxr_user module, our custom module only handles SSH
keys, one per user. Therefore, the user definitions have to already
exist in the running configuration.1 Moreover, the user defined
in ansible_user needs to be in the root-system group. The
cisco.iosxr collection must also be installed as the module relies
on its code.
When running the module, ansible_connection needs to be set to
network_cli and ansible_network_os to iosxr. These variables are
usually defined in the inventory.
module_args = dict( keys=dict(type='dict', elements='str', required=True), ) module = AnsibleModule( argument_spec=module_args, supports_check_mode=True ) result = dict( changed=False )
# show crypto key authentication rsa all Key label: vincent Type : RSA public key authentication Size : 2048 Imported : 16:17:08 UTC Tue Aug 11 2020 Data : 30820122 300D0609 2A864886 F70D0101 01050003 82010F00 3082010A 02820101 00D81E5B A73D82F3 77B1E4B5 949FB245 60FB9167 7CD03AB7 ADDE7AFE A0B83174 A33EC0E6 1C887E02 2338367A 8A1DB0CE 0C3FBC51 15723AEB 07F301A4 B1A9961A 2D00DBBD 2ABFC831 B0B25932 05B3BC30 B9514EA1 3DC22CBD DDCA6F02 026DBBB6 EE3CFADA AFA86F52 CAE7620D 17C3582B 4422D24F D68698A5 52ED1E9E 8E41F062 7DE81015 F33AD486 C14D0BB1 68C65259 F9FD8A37 8DE52ED0 7B36E005 8C58516B 7EA6C29A EEE0833B 42714618 50B3FFAC 15DBE3EF 8DA5D337 68DAECB9 904DE520 2D627CEA 67E6434F E974CF6D 952AB2AB F074FBA3 3FB9B9CC A0CD0ADC 6E0CDB2A 6A1CFEBA E97AF5A9 1FE41F6C 92E1F522 673E1A5F 69C68E11 4A13C0F3 0FFC782D 27020301 0001 [ ]
ansible_collections.cisco.iosxr.plugins.module_utils.network.iosxr.iosxr
contains a run_commands() function we can use:
command = "show crypto key authentication rsa all" out = run_commands(module, command) out = out[0].replace(' \n', '\n')
template = r""" Value Required Label (\w+) Value Required,List Data ([A-F0-9 ]+) Start ^Key label: $ Label ^Data\s+: -> GetData GetData ^ $ Data ^$$ -> Record Start """.lstrip() re_table = textfsm.TextFSM(io.StringIO(template)) got = data[0]: "".join(data[1]).replace(' ', '') for data in re_table.ParseText(out)
got is a dictionary associating key labels, considered as usernames,
with a hexadecimal representation of the public key currently
installed. It looks like this:
>>> pprint(got) 'alfred': '30820122300D0609[ ]6F0203010001', 'cedric': '30820122300D0609[ ]710203010001', 'vincent': '30820122300D0609[ ]270203010001'
wanted dictionary using the same structure. In
module.params['keys'], we have a dictionary associating usernames
to public SSH keys in the OpenSSH format:
>>> pprint(module.params['keys']) 'cedric': 'ssh-rsa AAAAB3NzaC1yc2[ ]', 'vincent': 'ssh-rsa AAAAB3NzaC1yc2[ ]'
ssh-keygen command and some glue can do the
conversion:2
$ ssh-keygen -f id_rsa.pub -e -mPKCS8 \ grep -v '^---' \ base64 -d \ hexdump -e '4/1 "%0.2X"' 30820122300D06092[ ]782D270203010001
ssh2cisco() function doing that, we can build the
wanted dictionary:
wanted = k: ssh2cisco(v) for k, v in module.params['keys'].items()
got and wanted when not running with check
mode. The part comparing got and wanted is taken verbatim from the
skeleton module:
if got != wanted: result['changed'] = True result['diff'] = dict( before=yaml.safe_dump(got), after=yaml.safe_dump(wanted) ) if module.check_mode or not result['changed']: module.exit_json(**result)
get_connection() and
copy_file() functions from
ansible_collections.cisco.iosxr.plugins.module_utils.network.iosxr.iosxr.
conn = get_connection(module) for user in wanted: if user not in got or wanted[user] != got[user]: dst = f"/harddisk:/publickey_ user .raw" with tempfile.NamedTemporaryFile() as src: decoded = base64.b64decode( module.params['keys'][user].split()[1]) src.write(decoded) src.flush() copy_file(module, src.name, dst) command = ("admin crypto key import authentication rsa " f"username user dst ") conn.send_command(command, prompt="yes/no", answer="yes")
for user in got: if user not in wanted: command = ("admin crypto key zeroize authentication rsa " f"username user ") conn.send_command(command, prompt="yes/no", answer="yes")
iosxr_user module, this one displays a diff when
running with --diff, correctly signals a change, is faster,
3 and deletes unwanted SSH keys. However, it is unable to
create users and cannot configure passwords or multiple SSH keys.
ssh-keygen, the format
used by Cisco is not PKCS#8. This is the ASN.1 representation of a
Subject Public Key Info structure, as defined in RFC 2459.
Moreover, PKCS#8 is a format for a private key, not a public one.
Next.