Launchpad still requires Python 2, which in 2020
is a bit of a problem.
Unlike a lot of the rest of 2020, though, there s good reason to be
optimistic about progress.
I ve been porting Python 2 code to Python 3 on and off for a long time, from
back when I was on the Ubuntu Foundations team and maintaining things like
the Ubiquity installer. When I moved to
Launchpad in 2015 it was certainly on my mind that this was a large body of
code still stuck on Python 2. One option would have been to just accept
that and leave it as it is, maybe doing more backporting work over time as
support for Python 2 fades away. I ve long been of the opinion that this
would doom Launchpad to being unmaintainable in the long run, and since I
genuinely love working on Launchpad - I find it an incredibly rewarding
project - this wasn t something I was willing to accept. We re already
seeing some of our important dependencies dropping support for Python 2,
which is perfectly reasonable on their terms but which is starting to become
a genuine obstacle to delivering important features when we need new
features from newer versions of those dependencies. It also looks as though
it may be difficult for us to run on Ubuntu 20.04 LTS (we re currently on
16.04, with an upgrade to 18.04 in progress) as long as we still require
Python 2, since we have some system dependencies that 20.04 no longer
provides. And then there are exciting new features like type
hints and
async/await that we d like
to be able to use.
However, until last year there were so many blockers that even considering a
port was barely conceivable. What changed in 2019 was sorting out a
trifecta of core dependencies. We ported our
database layer, Storm. We
upgraded
to modern versions of our Zope Toolkit dependencies
(after contributing various fixes upstream, including some substantial
changes to Zope s test runner
that we d carried as local patches for some years). And we
ported
our Bazaar code hosting infrastructure to
Breezy. With all that in place, a port
seemed more of a realistic possibility.
Still, even with this, it was never going to be a matter of just following
some standard porting advice and calling it
good. Launchpad has almost a million lines of Python code in its main git
tree, and around 250 dependencies of
which a number are quite Launchpad-specific. In a project that size, not
only is following standard porting advice an extremely time-consuming task
in its own right, but just about every strange corner case is going to show
up somewhere. (Did you know that StringIO.StringIO(None) and
io.StringIO(None) do different things even after you account for the
native string vs. Unicode text difference? How about the behaviour of
.union() on a subclass of
frozenset?)
Launchpad s test suite is fortunately extremely thorough, but even just
starting up the test suite involves importing most of the data model code,
so before you can start taking advantage of it you have to make a large
fraction of the codebase be at least syntactically-correct Python 3 code and
use only modules that exist in Python 3 while still working in Python 2; in
a project this size that turns out to be a large effort on its own, and can
be quite
risky
in places.
Canonical s product engineering teams work on a six-month cycle, but it just
isn t possible to cram this sort of thing into six months unless you do
literally nothing else, and please can we put all feature development on
hold while we run to stand still is a pretty tough sell to even the most
understanding management. Fortunately, we ve been able to grow the
Launchpad team in the last year or so,
and so it s been possible to put Python 3 on our roadmap on the
understanding that we aren t going to get all the way there in one cycle,
while still being able to do other substantial feature development work as well.
So, with all that preamble, what have we done this cycle? We ve taken a
two-pronged approach. From one end, we identified 147 classes that needed
to be ported away from some compatibility code in our database layer that
was substantially less friendly to Python 3: we ve ported 38 of those, so
there s clearly a fair bit more to do, but we were able to distribute this
work out among the team quite effectively. From the other end, it was clear
that it would be very inefficient to do general porting work when any
attempt to even run the test suite would run straight into the same crashes
in the same order, so I set myself a target of getting the test suite to
start up, and started hacking on an enormous git
branch that I never
expected to try to land directly: instead, I felt free to commit just about
anything that looked reasonable and moved things forward even if it was very
rough, and every so often went back to tidy things up and cherry-pick
individual commits into a form that included some kind of explanation and
passed existing tests so that I could propose them for review.
This strategy has been dramatically more successful than anything I ve tried
before at this scale. So far this cycle, considering only Launchpad s main
git tree, we ve landed 137 Python-3-relevant merge proposals for a total of
39552 lines of git diff output, keeping our existing tests passing along
the way and deploying incrementally to production. We have about 27000 more
lines of patch at varying degrees of quality to tidy up and merge. Our main
development branch is only perhaps 10 or 20 more patches away from the test
suite being able to start up, at which point we ll be able to get a buildbot
running so that multiple developers can work on this much more easily and
see the effect of their work. With the full unlanded patch stack, about 75%
of the test suite passes on Python 3! This still leaves a long tail of
several thousand tests to figure out and fix, but it s a much more
incrementally-tractable kind of problem than where we started.
Finally: the funniest (to me) bug I ve encountered in this effort was the
one I encountered in the test runner and fixed in
zopefoundation/zope.testrunner#106:
IDs of failing tests were written to a pipe, so if you have a test suite
that s large enough and broken enough then eventually that pipe would reach
its capacity and your test runner would just give up and hang. Pretty
annoying when it meant an overnight test run didn t give useful results, but
also eloquent commentary of sorts.
Amateur Radio Digital Communications (ARDC) is a
non-profit that was formed to further research and experimentation with digital
communications using radio, with a goal of advancing the state of the art of
amateur radio and to educate radio operators in these techniques.
It does this by managing the allocation of network resources, encouraging
research and experimentation with networking protocols and equipment,
publishing technical articles and number of other activities to promote the
public good of amateur radio and other related fields. ARDC has recently begun
to contribute funding to organisations, groups, individuals and projects
towards these and related goals, and their grant to the Reproducible Builds
project is part of this new initiative.
Amateur radio is an entirely volunteer activity performed by knowledgeable
hobbyists who have proven their ability by passing the appropriate government
examinations. No remuneration is permitted. Ham radio, as it is also known,
has proven its value in advancements of the state of the communications arts,
as well as in public service during disasters and in times of emergency.
For more information about ARDC, please see their website at
ampr.org.
One of the original promises of open source software was that peer review would
result in greater end-user security and stability of our digital ecosystem.
However, although it is theoretically possible to inspect and build the
original source code in order to avoid maliciously-inserted flaws, almost all
software today is distributed in prepackaged form.
This disconnect allows third-parties to compromise systems by injecting code
into seemingly secure software during the build process, as well as by
manipulating copies distributed from app stores and other package repositories.
In order to address this, Reproducible builds are a set of software
development practices, ideas and tools that create an independently-verifiable
path from the original source code, all the way to what is actually running on
our machines. Reproducible builds can reveal the injection of backdoors
introduced by the hacking of developers own computers, build servers and
package repositories, but can also expose where volunteers or companies have
been coerced into making changes via blackmail, government order, and so on.
A world without reproducible builds is a world where our digital infrastructure
cannot be trusted and where online communities are slower to grow, collaborate
less and are increasingly fragile. Without reproducible builds, we leave space
for greater encroachments on our liberties both by individuals as well as
powerful, unaccountable actors such as governments, large corporations and
autocratic regimes.
The Reproducible Builds project began as a project within the Debian community,
but is now working with many crucial and well-known free software projects such
as Coreboot, openSUSE, OpenWrt, Tails, GNU Guix, Arch Linux, Tor, and many
others. It is now an entirely Linux distribution independent effort and serves
as the central clearing house for all issues related to securing build
systems and software supply chains of all kinds.
For more about the Reproducible Builds project, please see their website at
reproducible-builds.org.
QSoas> cd QSoas> l Km-CODH-IV.dat QSoas> RFirst fitThen, to fit the above equation to the data, the simplest is to take advantage of the time-dependent parameters features of QSoas. Run simply:
QSoas> fit-arb im/(1+km/s) /with=s:1,expThis simply launches the fit interface to fit the exact equations above. The
im/(1+km/s) is simply the translation of the Michaelis-Menten equation above, and the /with=s:1,exp specifies that s is the result of the sum of 1 exponential like for the definition of above. Then, load the Km-CODH-IV.params parameter files (using the "Parameters.../Load from file" action at the bottom, or the Ctrl+L keyboard shortcut). Your window should now look like this:
To fit the data, just hit the "Fit" button ! (or Ctrl+F).
Including an offset
The fit is not bad, but not perfect. In particular, it is easy to see why: the current predicted by the fit goes to 0 at large times, but the actual current is below 0. We need therefore to include an offset to take this into consideration. Close the fit window, and re-run a fit, but now with this command:
QSoas> fit-arb im/(1+km/s)+io /with=s:1,expNotice the
+io bit that corresponds to the addition of an offset current. Load again the base parameters, run the fit again... Your fit window show now look like:
See how the offset current is now much better taken into account. Let's talk a bit more about the parameters:
im is \(i_m\), the maximal current, around 120 nA (which matches the magnitude of the original current).io is the offset current, around -3nA.km is the \(K_m\), expressed in the same units as s_1, the first "injected" value of s (we used 50 because the injection is 50 M CO). That means the value of \(K_m\) determined by this fit is around 9 nM ! (but see below).s_t_1 is the time of the injection of CO (it was in the parameter files you loaded).s_tau_1 is the time constant of departure of CO, noted \(\tau\) in the equations above.
transport.rb Ruby file that contains the definition of the itrpt function, and re-launch the fit window using:
QSoas> ruby-run transport.rb QSoas> fit-arb itrprt(s,km,nFAm,nFAmu)+io /with=s:1,expLoad again the parameter file... but this time you'll have to play a bit more with the starting parameters for QSoas to find the right values when you fit. Here are some tips:
io, no parameter should be negative;ruby...ruby end block. Probably the most useful possibility is to define elaborated functions directly from within QSoas, or, preferable, from within a script; this is an alternative to defining a function in a completely separated Ruby-only file using ruby-run. For instance, you can define a function for plain Michaelis-Menten kinetics with a file containing:ruby def my_func(x, vm, km) return vm/(1 + km/x) end ruby end
my_func with three parameters, , (vm) and (km), with the formula: QSoas> eval my_func(1.0,1.0,1.0) => 0.5 QSoas> eval my_func(1e4,1.0,1.0) => 0.999900009999
x = 1.0, vm = 1.0 and km = 1.0. For , the result is (here 0.5). For , the result is almost . You can use the newly defined my_func in any place you would use any ruby code, such as in the optional argument to generate-buffer, or for arbitrary fits:QSoas> generate-buffer 0 10 my_func(x,3.0,0.6) QSoas> fit-arb my_func(x,vm,km)
my_func, just run the ruby code again with a new definition, such as:ruby def my_func(x, vm, km) return vm/(1 + km/x**2) end ruby endThe previous version is just erased, and all new uses of
my_func will refer to your new definition.qsoas-goodies github repository for more goodies !1.0. They are also freely available for download from the downloads page.VIRTFS="-virtfs local,path=/vmstore/virtfs,security_model=mapped-xattr,id=zz,writeout=immediate,fmode=0600,dmode=0700,mount_tag=zz" VIRTFS="-virtfs local,path=/vmstore/virtfs,security_model=passthrough,id=zz,writeout=immediate,mount_tag=zz"Above are the 2 configuration snippets I tried on the server side. The first uses mapped xattrs (which means that all files will have the same UID/GID and on the host XATTRs will be used for storing the Unix permissions) and the second uses passthrough which requires KVM to run as root and gives the same permissions on the host as on the VM. The advantages of passthrough are better performance through writing less metadata and having the same permissions in host and VM. The advantages of mapped XATTRs are running KVM/Qemu as non-root and not having a SUID file in the VM imply a SUID file in the host. Here is the link to Bonnie++ output comparing Ext3 on a KVM block device (stored on a regular file in a BTRFS RAID-1 filesystem on 2 SSDs on the host), a NFS share from the host from the same BTRFS filesystem, and virtfs shares of the same filesystem. The only tests that Ext3 doesn t win are some of the latency tests, latency is based on the worst-case not the average. I expected Ext3 to win most tests, but didn t expect it to lose any latency tests. Here is a link to Bonnie++ output comparing just NFS and Virtfs. It s obvious that Virtfs compares poorly, giving about half the performance on many tests. Surprisingly the only tests where Virtfs compared well to NFS were the file creation tests which I expected Virtfs with mapped XATTRs to do poorly due to the extra metadata. Here is a link to Bonnie++ output comparing only Virtfs. The options are mapped XATTRs with default msize, mapped XATTRs with 512k msize (I don t know if this made a difference, the results are within the range of random differences), and passthrough. There s an obvious performance benefit in passthrough for the small file tests due to the less metadata overhead, but as creating small files isn t a bottleneck on most systems a 20% to 30% improvement in that area probably doesn t matter much. The result from the random seeks test in passthrough is unusual, I ll have to do more testing on that. SE Linux On Virtfs the XATTR used for SE Linux labels is passed through to the host. So every label used in a VM has to be valid on the host and accessible to the context of the KVM/Qemu process. That s not really an option so you have to use the context mount option. Having the mapped XATTR mode work for SE Linux labels is a necessary feature. Conclusion The msize mount option in the VM doesn t appear to do anything and it doesn t appear in /proc/mounts, I don t know if it s even supported in the kernel I m using. The passthrough and mapped XATTR modes give near enough performance that there doesn t seem to be a benefit of one over the other. NFS gives significant performance benefits over Virtfs while also using less CPU time in the VM. It has the issue of files named .nfs* hanging around if the VM crashes while programs were using deleted files. It s also more well known, ask for help with an NFS problem and you are more likely to get advice than when asking for help with a virtfs problem. Virtfs might be a better option for accessing databases than NFS due to it s internal operation probably being a better map to Unix filesystem semantics, but running database servers on the host is probably a better choice anyway. Virtfs generally doesn t seem to be worth using. I had hoped for performance that was better than NFS but the only benefit I seemed to get was avoiding the .nfs* file issue. The best options for storage for a KVM/Qemu VM seem to be Ext3 for files that are only used on one VM and for which the size won t change suddenly or unexpectedly (particularly the root filesystem) and NFS for everything else.
| Series: | Sun Chronicles #1 |
| Publisher: | Tor |
| Copyright: | 2020 |
| ISBN: | 1-250-19725-2 |
| Format: | Kindle |
| Pages: | 526 |
Previously: v5.6
Linux v5.7 was released at the end of May. Here s my summary of various security things that caught my attention:
arm64 kernel pointer authenticationPACIASP which protects the saved stack pointer, similar to the existing CONFIG_STACKPROTECTOR feature, only faster. This also paves the way to sign and check pointers stored in the heap, as a way to defeat function pointer overwrites in those memory regions too. Since the behavior is different from the traditional stack protector, Amit Daniel Kachhap added an LKDTM test for PAC as well.
BPF LSMexecve() deadlock refactoringexec_id counter to 64 bits to avoid the possibility of the counter wrapping and allowing an attacker to send arbitrary signals to processes they normally shouldn t be able to.
slub freelist obfuscation improvementsCONFIG_SLAB_FREELIST_HARDENED s freelist pointer content obfuscation, I improved their bit diffusion, which makes attacks require significantly more memory content exposures to defeat the obfuscation. As part of the conversation, Vitaly Nikolenko pointed out that the freelist pointer s location made it relatively easy to target too (for either disclosures or overwrites), so I moved it away from the edge of the slab, making it harder to reach through small-sized overflows (which usually target the freelist pointer). As it turns out, there were a few assumptions in the kernel about the location of the freelist pointer, which had to also get cleaned up.
RISCV page table dumpingPTDUMP_DEBUGFS), visible in /sys/kernel/debug/kernel_page_tables.
array index bounds checkingint foo[5]; ... foo[8] = bar;The long version gets complicated by the evolution of flexible array structure members, so we ll pause for a moment and skim the surface of this topic. While things like
CONFIG_FORTIFY_SOURCE try to catch these kinds of cases in the memcpy() and strcpy() family of functions, it doesn t catch it in open-coded array indexing, as seen in the code above. GCC has a warning (-Warray-bounds) for these cases, but it was disabled by Linus because of all the false positives seen due to fake flexible array members. Before flexible arrays were standardized, GNU C supported zero sized array members. And before that, C code would use a 1-element array. These were all designed so that some structure could be the header in front of some data blob that could be addressable through the last structure member:
/* 1-element array */
struct foo
...
char contents[1];
;
/* GNU C extension: 0-element array */
struct foo
...
char contents[0];
;
/* C standard: flexible array */
struct foo
...
char contents[];
;
instance = kmalloc(sizeof(struct foo) + content_size);
Converting all the zero- and one-element array members to flexible arrays is one of Gustavo A. R. Silva s goals, and hundreds of these changes started landing. Once fixed, -Warray-bounds can be re-enabled. Much more detail can be found in the kernel s deprecation docs.
However, that will only catch the visible at compile time cases. For runtime checking, the Undefined Behavior Sanitizer has an option for adding runtime array bounds checking for catching things like this where the compiler cannot perform a static analysis of the index values:
int foo[5];
...
for (i = 0; i < some_argument; i++)
...
foo[i] = bar;
...
It was, however, not separate (via kernel Kconfig) until Elena Petrova and I split it out into CONFIG_UBSAN_BOUNDS, which is fast enough for production kernel use. With this enabled, it's now possible to instrument the kernel to catch these conditions, which seem to come up with some regularity in Wi-Fi and Bluetooth drivers for some reason. Since UBSAN (and the other Sanitizers) only WARN() by default, system owners need to set panic_on_warn=1 too if they want to defend against attacks targeting these kinds of flaws. Because of this, and to avoid bloating the kernel image with all the warning messages, I introduced CONFIG_UBSAN_TRAP which effectively turns these conditions into a BUG() without needing additional sysctl settings.
Fixing "additive" snprintf() usagesprintf()'s return value to increment a pointer into a string, and build a string with more sprintf() calls:
/* safe if strlen(foo) + 1 < sizeof(string) */ wrote = sprintf(string, "Foo: %s\n", foo); /* overflows if strlen(foo) + strlen(bar) > sizeof(string) */ wrote += sprintf(string + wrote, "Bar: %s\n", bar); /* writing way beyond the end of "string" now ... */ wrote += sprintf(string + wrote, "Baz: %s\n", baz);The risk is that if these calls eventually walk off the end of the
string buffer, it will start writing into other memory and create some bad situations. Switching these to snprintf() does not, however, make anything safer, since snprintf() returns how much it would have written:
/* safe, assuming available <= sizeof(string), and for this example * assume strlen(foo) < sizeof(string) */ wrote = snprintf(string, available, "Foo: %s\n", foo); /* if (strlen(bar) > available - wrote), this is still safe since the * write into "string" will be truncated, but now "wrote" has been * incremented by how much snprintf() *would* have written, so "wrote" * is now larger than "available". */ wrote += snprintf(string + wrote, available - wrote, "Bar: %s\n", bar); /* string + wrote is beyond the end of string, and availabe - wrote wraps * around to a giant positive value, making the write effectively * unbounded. */ wrote += snprintf(string + wrote, available - wrote, "Baz: %s\n", baz);So while the first overflowing call would be safe, the next one would be targeting beyond the end of the array and the size calculation will have wrapped around to a giant limit. Replacing this idiom with
scnprintf() solves the issue because it only reports what was actually written. To this end, Takashi Iwai has been landing a bunch scnprintf() fixes.
That's it for now! Let me know if there is anything else you think I should mention here. Next up: Linux v5.8.
2020, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
SUBSCRIBE_FORM_SECRET set in your mailman configuration:
SECRET=$(tr -dc 'A-Za-z0-9' < /dev/urandom head -c 30)'
echo "SUBSCRIBE_FORM_SECRET = '$SECRET'" >> /etc/mailman/mm.cfg
This will add a magic token to all forms in the Mailman web forms that
will force the attacker to at least get a token before asking for
registration. There are, of course, other ways of performing the
attack then, but it's more expensive than a single request for the
attacker and keeps most of the junk out.
RewriteMap hosts-deny txt:/etc/apache2/blocklist.txt
RewriteCond $ hosts-deny:% REMOTE_ADDR NOT-FOUND !=NOT-FOUND [OR]
RewriteCond $ hosts-deny:% REMOTE_HOST NOT-FOUND !=NOT-FOUND [OR]
RewriteCond % HTTP_REFERER !^https://lists.torproject.org/$ [NC]
RewriteRule ^/cgi-bin/mailman/subscribe/ - [F]
# see also https://www.w3.org/TR/referrer-policy/#referrer-policy-origin
Header always set Referrer-Policy "origin"
I kept those restrictions in place because it keeps the spammers from
even hitting the Mailman CGI, which is useful to preserve our server
resources. But if "they" escalate with smarter crawlers, the block
list will still be useful.
You can use this query to extract the top 10 IP addresses used for
subscription attempts:
awk ' print $NF ' /var/log/mailman/subscribe sort uniq -c sort -n tail -10 awk ' print $2 " " $1 '
Note that this might include email-based registration, but in our logs
those are extremely rare: only two in three weeks, out of over
73,000 requests. I also use this to keep an eye on the logs:
tail -f /var/log/mailman/subscribe /var/log/apache2/lists.torproject.org-access.log grep -v 'GET /pipermail/'
The server-side mitigations might also be useful if you happen to run
an extremely old version of Mailman, that is pre-2.1.18, but it's now
over 6 years old and part of every supported Debian release out there
(all the way back to Debian 8 jessie).
POST request to /mailman/subscribe/LISTNAME to
have Mailman send an email to the user. In the old "Internet is for
nice people" universe, that wasn't a problem: all it does is ask the
victim if they want to subscribe to LISTNAME. Innocuous, right?
But in the brave, new, post-Eternal-September, "Internet is for
stupid" universe, some assholes think it's a good idea to make a form
that collects hundreds of mailing list URLs and spam them through an
iframe. To see what that looks like, you can look at the rendered
source code behind samedyfreeday.co.uk (not linking to avoid
promoting it). That site does what is basically a distributed
cross-site scripting attack against Mailman servers.
Obviously, CSRF protection should be enabled by default in Mailman,
but there you go. Hopefully this will help some folks...
(The latest Mailman 3 release doesn't suffer from such idiotic
defaults and ships with proper CSRF protection out of the box.)
I upgraded my home internet connection to fibre (FTTP) last October. I m still on an 80M/20M service, so it s no faster than my old VDSL FTTC connection was, and as a result for a long time I continued to use my HomeHub 5A running OpenWRT. However the FTTP ONT meant I was using up an additional ethernet port on the router, and I was already short, so I ended up with a GigE switch in use as well. Also my wifi is handled by a UniFi, which takes its power via Power-over-Ethernet. That mean I had a router, a switch and a PoE injector all in close proximity. I wanted to reduce the number of devices, and ideally upgrade to something that could scale once I decide to upgrade my FTTP service speed.
Looking around I found the MikroTik RB3011UiAS-RM, which is a rack mountable device with 10 GigE ports (plus an SFP slot) and a dual core Qualcomm IPQ8064 ARM powering it. There s 1G RAM and 128MB NAND flash, as well as a USB3 port. It also has PoE support. On paper it seemed like an ideal device. I wasn t particularly interested in running RouterOS on it (the provided software), but that s based on Linux and there was some work going on within OpenWRT to add support, so it seemed like a worthwhile platform to experiment with (what, you expected this to be about me buying an off the shelf device and using it with only the supplied software?). As an added bonus a friend said he had one he wasn t using, and was happy to sell it to me for a bargain price.
I did try out RouterOS to start with, but I didn t find it particularly compelling. I m comfortable configuring firewalling and routing at a Linux command line, and I run some additional services on the router like my MQTT broker, and mqtt-arp, my wifi device presence monitor. I could move things around such that they ran on the house server, but I consider them core services and as a result am happier with them on the router.
The first step was to get something booting on the router. Luckily it has an RJ45 serial console port on the back, and a reasonably featured bootloader that can manage to boot via tftp over the network. It wants an ELF binary rather than a plain kernel, but Sergey Sergeev had done the hard work of getting u-boot working for the IPQ8064, which mean I could just build normal u-boot images to try out.
Linux upstream already had basic support for a lot of the pieces I was interested in. There s a slight fudge around AUTO_ZRELADDR because the network coprocessors want a chunk of memory at the start of RAM, but there s ongoing discussions about how to handle this cleanly that I m hopeful will eventually mean I can drop that hack. Serial, ethernet, the QCA8337 switches (2 sets of 5 ports, tied to different GigE devices on the processor) and the internal NOR all had drivers, so it was a matter of crafting an appropriate DTB to get them working. That left niggles.
First, the second switch is hooked up via SGMII. It turned out the IPQ806x stmmac driver didn t initialise the clocks in this mode correctly, and neither did the qca8k switch driver. So I need to fix up both of those (Sergey had handled the stmmac driver, so I just had to clean up and submit his patch). Next it turned out the driver for talking to the Qualcomm firmware (SCM) had been updated in a way that broke the old method needed on the IPQ8064. Some git archaeology figured that one out and provided a solution. Ansuel Smith helpfully provided the DWC3 PHY driver for the USB port. That got me to the point I could put a Debian armhf image onto a USB stick and mount that as root, which made debugging much easier.
At this point I started to play with configuring up the device to actually act as a router. I make use of a number of VLANs on my home network, so I wanted to make sure I could support those. Turned out the stmmac driver wasn t happy reconfiguring its MTU because the IPQ8064 driver doesn t configure the FIFO sizes. I found what seem to be the correct values and plumbed them in. Then the qca8k driver only supported port bridging. I wanted the ability to have a trunk port to connect to the upstairs switch, while also having ports that only had a single VLAN for local devices. And I wanted the switch to handle this rather than requiring the CPU to bridge the traffic. Thankfully it s easy to find a copy of the QCA8337 datasheet and the kernel Distributed Switch Architecture is pretty flexible, so I was able to implement the necessary support.
I stuck with Debian on the USB stick for actually putting the device into production. It makes it easier to fix things up if necessary, and the USB stick allows for a full Debian install which would be tricky on the 128M of internal NAND. That means I can use things like nftables for my firewalling, and use the standard Debian packages for things like collectd and mosquitto. Plus for debug I can fire up things like tcpdump or tshark. Which ended up being useful because when I put the device into production I started having weird IPv6 issues that turned out to be a lack of proper Ethernet multicast filter support in the IPQ806x ethernet device. The driver would try and setup the multicast filter for the IPv6 NDP related packets, but it wouldn t actually work. The fix was to fall back to just receiving all multicast packets - this is what the vendor driver does.
Most of this work will be present once the 5.9 kernel is released - the basics are already in 5.8. Currently not queued up that I can think of are the following:
| Publisher: | The New Press |
| Copyright: | 2017 |
| Printing: | 2018 |
| ISBN: | 1-62097-472-X |
| Format: | Kindle |
| Pages: | 217 |
Following up on the initial RcppSpdlog 0.0.1 release earlier this week, we are pumped to announce release 0.0.2. It contains upstream version 1.8.0 for spdlog which utilizes (among other things) a new feature in the embedded fmt library, namely completely automated formatting of high resolution time stamps which allows for gems like this (taken from this file in the package and edited down for brevity):
spdlog::stopwatch sw; // instantiate a stop watch
// some other code
sp->info("Elapsed time: ", sw);stopwatch object, and simply references it. The rest, as they say, is magic. And we get tic / toc alike behaviour in modern C++ at essentially no cost us (as code authors). So nice. Output from the (included in the package) function exampleRsink() (again edited down just a little):
R> RcppSpdlog::exampleRsink()
[13:03:23.845114] [fromR] [I] [thread 793264] Welcome to spdlog!
[...]
[13:03:23.845205] [fromR] [I] [thread 793264] Elapsed time: 0.000103611
[...]
[13:03:23.845281] [fromR] [I] [thread 793264] Elapsed time: 0.00018203
R> Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppSpdlog page. The only sour grapes, again, are once more over the CRAN processing. And just how 0.0.1 was delayed for no good reason for three weeks, 0.0.2 was delayed by three days just because well that is how CRAN rules sometimes. I d be even more mad if I had an alternative but I don t. We remain grateful for all they do but they really could have let this one through even at one-day update delta. Ah well, now we re three days wiser and of course nothing changed in the package. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.Changes in RcppSpdlog version 0.0.2 (2020-09-17)
- Upgraded to upstream release 1.8.0
- Switched Travis CI to using BSPM, also test on macOS
- Added 'stopwatch' use to main R sink example
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
My webcam has broken, something that I have been using a lot during the last few months for some reason.
A friend of mine suggested that I use the mic and camera on my mobile phone instead. There is a simple app droidcam that makes the phone behave as a simple webcam, it also has a client application to run on your PC to capture the web-stream and present it as a video device. All well and good but I would like to keep propitiatory software off my PCs (I have a hard enough time accepting it on a phone but I have to draw a line somewhere).
I decided that there had to be a simple way to ingest that stream and present it as a local video device on a Linux box. It turns out that it is a lot simpler than I thought it would be. I had it working within 10 minutes!
Packages needed: ffmpeg, v4l2loopback-utils
sudo apt-get install ffmpeg v4l2loopback-utilsStart the loop-back device:
sudo modprobe v4l2loopbackIngest video and present on loop-back device:
ffmpeg -re -i <URL_VideoStream> -vcodec rawvideo -pix_fmt yuv420p -f v4l2 /dev/video0Where
-re
Read input at native frame rate. Mainly used to simulate a grab
device, or live input stream (e.g. when reading from a file). Should
not be used with actual grab devices or live input streams (where it
can cause packet loss). By default ffmpeg attempts to read the
input(s) as fast as possible. This option will slow down the reading
of the input(s) to the native frame rate of the input(s). It is
useful for real-time output (e.g. live streaming).
-i <source>
In this case my phone running droidcam http://10.14.1.100:4747/video
-vcodec rawvideo
Select the output video codec to raw video (as expected for /dev/video#)
-pix_fmt yuv420p
Set pixel format. in this example tell ffmpeg that the video will
be yuv colour space at 420p resolution
-f v4l2 /dev/video0
Force the output to be in video for Linux 2 (v4l2) format bound to
device /dev/video0
vrrp_instance gateway1 state BACKUP # interface eth0 # virtual_router_id 12 # priority 101 # virtual_ipaddress 2001:db8:ff/64
state keyword in instructs Keepalived to not take the leader
role when starting. Otherwise, incoming nodes create a temporary
disruption by taking over the IP address until the election settles.
The interface keyword in defines the interface for sending and
receiving VRRP packets. It is also the default interface to configure
the virtual IP address. The virtual_router_id directive in is
common to all nodes sharing the virtual IP. The priority keyword in
helps choosing which router will be elected as leader. If you need
more information around Keepalived, be sure to check the
documentation.
VRRP design is tied to Ethernet networks and requires a
multicast-enabled network for communication between nodes. In some
environments, notably public clouds, multicast is unavailable. In this
case, Keepalived can send VRRP packets using unicast:
vrrp_instance gateway1 state BACKUP interface eth0 virtual_router_id 12 priority 101 unicast_peer 2001:db8::11 2001:db8::12 virtual_ipaddress 2001:db8:ff/64 dev lo
notify_* scripts.
Until version 2.21 (not released yet), the interface directive is
mandatory and Keepalived will transmit and receive VRRP packets on
this interface only. If peers are reachable through several
interfaces, like on a BGP on the host setup, you need a
workaround. A simple one is to use a VXLAN interface:
$ ip -6 link add keepalived6 type vxlan id 6 dstport 4789 local 2001:db8::10 nolearning $ bridge fdb append 00:00:00:00:00:00 dev keepalived6 dst 2001:db8::11 $ bridge fdb append 00:00:00:00:00:00 dev keepalived6 dst 2001:db8::12 $ ip link set up dev keepalived6
vrrp_instance gateway1 state BACKUP interface keepalived6 mcast_src_ip 2001:db8::10 virtual_router_id 12 priority 101 virtual_ipaddress 2001:db8:ff/64 dev lo
unicast_peer can be used without
the interface directive. I think using VXLAN is still a
neat trick applicable to other situations where communication using
broadcast or multicast is needed, while the underlying network provide
no support for this.
netbox.netbox collection from Ansible Galaxy
provides several modules to update NetBox objects:
- name: create a device in NetBox netbox_device: netbox_url: http://netbox.local netbox_token: s3cret data: name: to3-p14.sfo1.example.com device_type: QFX5110-48S device_role: Compute Switch site: SFO1
Notice I recommend that you read Writing a custom Ansible module as an introduction, as well as Syncing MySQL tables for a first simpler example.
netbox_sync: source: netbox.yaml api: https://netbox.example.com token: s3cret
devices: ad2-p6.sfo1.example.com: datacenter: sfo1 manufacturer: Cisco model: Catalyst 2960G-48TC-L role: net_tor_oob_switch to1-p6.sfo1.example.com: datacenter: sfo1 manufacturer: Juniper model: QFX5110-48S role: net_tor_gpu_switch # [ ] ips: - device: ad2-p6.example.com ip: 172.31.115.18/21 interface: oob - device: to1-p6.example.com ip: 172.31.115.33/21 interface: oob - device: to1-p6.example.com ip: 172.31.254.33/32 interface: lo0.0 # [ ]
cmdb. Most objects in NetBox accept tags.
module_args = dict( source=dict(type='path', required=True), api=dict(type='str', required=True), token=dict(type='str', required=True, no_log=True), max_workers=dict(type='int', required=False, default=10) ) result = dict( changed=False ) module = AnsibleModule( argument_spec=module_args, supports_check_mode=True )
Synchronizer abstract class. For
each kind of object, a concrete class is built with the appropriate
class attributes to tune its behaviour and a wanted() method to
provide the objects we want.
I am not explaining the abstract class code here. Have a look at the
source if you want.
class SyncTags(Synchronizer): app = "extras" table = "tags" key = "name" def wanted(self): return "cmdb": dict( slug="cmdb", color="8bc34a", description="synced by network CMDB")
app and table attributes defines the NetBox objects we want
to manipulate. The key attribute is used to determine how to lookup
for existing objects. In this example, we want to lookup tags using
their names.
The wanted() method is expected to return a dictionary mapping
object keys to the list of wanted attributes. Here, the keys are tag
names and we create only one tag, cmdb, with the provided slug,
color and description. This is the tag we will use to mark the objects
we create or modify.
If the tag does not exist, it is created. If it exists, the provided
attributes are updated. Other attributes are left untouched.
We also want to create a specific tenant for objects accepting such an
attribute (devices and IP addresses):
class SyncTenants(Synchronizer): app = "tenancy" table = "tenants" key = "name" def wanted(self): return "Network": dict(slug="network", description="Network team")
wanted() method uses the information provided in the YAML file: it
walks the devices and builds a set of datacenter names.
class SyncSites(Synchronizer): app = "dcim" table = "sites" key = "name" only_on_create = ("status", "slug") def wanted(self): result = set(details["datacenter"] for details in self.source['devices'].values() if "datacenter" in details) return k: dict(slug=k, status="planned") for k in result
only_on_create attribute, the specified
attributes are not updated if they are different. The goal of this
synchronizer is mostly to collect the references to the different
sites for other objects.
>>> pprint(SyncSites(**sync_args).wanted()) 'sfo1': 'slug': 'sfo1', 'status': 'planned' , 'chi1': 'slug': 'chi1', 'status': 'planned' , 'nyc1': 'slug': 'nyc1', 'status': 'planned'
only_on_create attribute:
class SyncManufacturers(Synchronizer): app = "dcim" table = "manufacturers" key = "name" def wanted(self): result = set(details["manufacturer"] for details in self.source['devices'].values() if "manufacturer" in details) return k: "slug": slugify(k) for k in result
foreign attribute linking
a NetBox attribute to the synchronizer handling it.
class SyncDeviceTypes(Synchronizer): app = "dcim" table = "device_types" key = "model" foreign = "manufacturer": SyncManufacturers def wanted(self): result = set((details["manufacturer"], details["model"]) for details in self.source['devices'].values() if "model" in details) return k[1]: dict(manufacturer=k[0], slug=slugify(k[1])) for k in result
wanted() method refers to the manufacturer using its key
attribute. In this case, this is the manufacturer name.
>>> pprint(SyncManufacturers(**sync_args).wanted()) 'Cisco': 'slug': 'cisco' , 'Dell': 'slug': 'dell' , 'Juniper': 'slug': 'juniper' >>> pprint(SyncDeviceTypes(**sync_args).wanted()) 'ASR 9001': 'manufacturer': 'Cisco', 'slug': 'asr-9001' , 'Catalyst 2960G-48TC-L': 'manufacturer': 'Cisco', 'slug': 'catalyst-2960g-48tc-l' , 'MX10003': 'manufacturer': 'Juniper', 'slug': 'mx10003' , 'QFX10002-36Q': 'manufacturer': 'Juniper', 'slug': 'qfx10002-36q' , 'QFX10002-72Q': 'manufacturer': 'Juniper', 'slug': 'qfx10002-72q' , 'QFX5110-32Q': 'manufacturer': 'Juniper', 'slug': 'qfx5110-32q' , 'QFX5110-48S': 'manufacturer': 'Juniper', 'slug': 'qfx5110-48s' , 'QFX5200-32C': 'manufacturer': 'Juniper', 'slug': 'qfx5200-32c' , 'S4048-ON': 'manufacturer': 'Dell', 'slug': 's4048-on' , 'S6010-ON': 'manufacturer': 'Dell', 'slug': 's6010-on'
class SyncDeviceRoles(Synchronizer): app = "dcim" table = "device_roles" key = "name" def wanted(self): result = set(details["role"] for details in self.source['devices'].values() if "role" in details) return k: dict(slug=slugify(k), color="8bc34a") for k in result
class SyncDevices(Synchronizer): app = "dcim" table = "devices" key = "name" foreign = "device_role": SyncDeviceRoles, "device_type": SyncDeviceTypes, "site": SyncSites, "tenant": SyncTenants remove_unused = 10 def wanted(self): return name: dict(device_role=details["role"], device_type=details["model"], site=details["datacenter"], tenant="Network") for name, details in self.source['devices'].items() if "datacenter", "model", "role" <= set(details.keys())
remove_unused attribute is a safety implemented to fail if we
have to delete more than 10 devices: this may be the indication there
is a bug somewhere, unless one of your datacenter suddenly caught
fire.
>>> pprint(SyncDevices(**sync_args).wanted()) 'ad2-p6.sfo1.example.com': 'device_role': 'net_tor_oob_switch', 'device_type': 'Catalyst 2960G-48TC-L', 'site': 'sfo1', 'tenant': 'Network' , 'to1-p6.sfo1.example.com': 'device_role': 'net_tor_gpu_switch', 'device_type': 'QFX5110-48S', 'site': 'sfo1', 'tenant': 'Network' , [ ]
class SyncIPs(Synchronizer): app = "ipam" table = "ip-addresses" key = "address" foreign = "tenant": SyncTenants remove_unused = 1000 def wanted(self): wanted = for details in self.source['ips']: if details['ip'] in wanted: wanted[details['ip']]['description'] = \ f" details['device'] (and others)" else: wanted[details['ip']] = dict( tenant="Network", status="active", dns_name="", # information is present in DNS description=f" details['device'] : details['interface'] ", role=None, vrf=None) return wanted
cmdb, then those
already attached to an interface:
def get(self, key): """Grab IP address from NetBox.""" # There may be duplicate. We need to grab the "best". results = super(Synchronizer, self).get(key) if len(results) == 0: return None if len(results) == 1: return results[0] scores = [0]*len(results) for idx, result in enumerate(results): if "cmdb" in result.tags: scores[idx] += 10 if result.interface is not None: scores[idx] += 5 return sorted(zip(scores, results), reverse=True, key=lambda k: k[0])[0][1]
source = yaml.safe_load(open(module.params['source'])) netbox = pynetbox.api(module.params['api'], token=module.params['token']) sync_args = dict( module=module, netbox=netbox, source=source, before= , after= ) synchronizers = [synchronizer(**sync_args) for synchronizer in [ SyncTags, SyncTenants, SyncSites, SyncManufacturers, SyncDeviceTypes, SyncDeviceRoles, SyncDevices, SyncIPs ]]
prepare() method whose goal is to compute
the current and wanted states. It returns True in case of a
difference:
# Check what needs to be synchronized try: for synchronizer in synchronizers: result['changed'] = synchronizer.prepare() except AnsibleError as e: result['msg'] = e.message module.fail_json(**result)
sync_args["before"][table] and
sync_args["after"][table] where table is the name of the table for
a given NetBox object type. The diff object is a bit elaborate as
it is built table by table. This enables Ansible to display the name
of each table before the diff representation:
# Compute the diff if module._diff and result['changed']: result['diff'] = [ dict( before_header=table, after_header=table, before=yaml.safe_dump(sync_args["before"][table]), after=yaml.safe_dump(sync_args["after"][table])) for table in sync_args["after"] if sync_args["before"][table] != sync_args["after"][table] ] # Stop here if check mode is enabled or if no change if module.check_mode or not result['changed']: module.exit_json(**result)
synchronize() method to apply
changes and a cleanup() method to delete unwanted objects. Order is
important due to the relation between the objects.
# Synchronize for synchronizer in synchronizers: synchronizer.synchronize() for synchronizer in synchronizers[::-1]: synchronizer.cleanup() module.exit_json(**result)
netbox.netbox collection, the logic is written in
Python instead of trying to glue Ansible tasks together. I believe
this is both more flexible and easier to read, notably when trying to
delete outdated objects. While I did not test it, it should also be
faster. An alternative would have been to reuse code from the
netbox.netbox collection, as it contains similar primitives.
Unfortunately, I didn t think of it until now.
The following contributors were added as Debian Maintainers in the last two months:
Next.