--gem-install layoutand fixing autopkgtest.
Breaks+Replacesfor ruby-sass-rails (#952682 and #954544).
cut -d'-' -f1to just show upstream version.
encryptorfor Ruby2.7 test failures.
image_sciencefor removing relative paths.
ffi-yajlfor Ruby2.7 test failures.
aggregatefor simply using
aggregatefor modernizing CI and adding Ruby 2.5 and 2.7 support.
espeak-rubyfor Ruby2.7 test failures.
tty-whichfor test failures in general.
packablefor Ruby2.7 test failures. PR #12 has been proposed.
growlfor test failures and proposed an initial patch.
hugo-coder, asking to enable RSS feed for blogs.
hugo-coderfor fixing the above issue myself.
arbrefor requesting a release.
combustion, asking to relax dependency on rubocop. Fixed in this commit.
ffi-compilerfor requesting to fix homepage and license.
gographvizfor requesting a release.
crb-blast, suggesting compatability with bio 2.0.x.
uniform_notifierfor asking to drop the use of ruby-growl.
polybar, adding installation instructions on Debian systems.
lualatexis now based on LuaHBTeX, meaning that one can use the HarfBuzz renderer which in particular for complicated scripts (Tibetan, Bengali, ) works better than the Lua-based renderer. Note that
luatexitself remains normal LuaTeX, only the luaLAtex one uses LuaHBTeX.
foobar.tar.xzwhile the name never changed. We have now changed all the infrastructure and TeX Live Manager to work with versioned containers
foobar.rNNNNN.tar.xz. This should help quite some distributors!
\inputprimitive in all TeX engines, including
tex, now also accepts a group-delimited lename argument, as a system-dependent extension. The usage with a standard space/token-delimited lename is completely unchanged. The group-delimited argument was previously implemented in LuaTeX; now it is available in all engines. ASCII double quote characters ( ) are removed from the lename, but it is otherwise left unchanged after tokenization. This does not currently a ect LaTeX s
\inputcommand, as that is a macro rede nition of the standard
mf, and all other engines, to support arbitrary con guration settings on the command line.
\iffontchar. For eupTeX only:
\currentcjktoken. LuaTeX: Integration with HarfBuzz library, available as new engines
luajithbtex. New primitives:
\eTeXglueshrinkorder. pdfTeX: New primitive
\pdfmajorversion; this merely changes the version number in the PDF output; it has no e ect on any PDF content.
\pdfximageand similar now search for image les in the same way as
\openin. pTeX: New primitives
\iftfont. Also in epTeX, upTeX, eupTeX. XeTeX: Fixes for
\pdfsavepos. Dvips: Output encodings for bitmap fonts, for better copy/paste capabilities (https://tug.org/TUGboat/tb40-2/tb125rokicki-type3search.pdf). MacTeX: MacTeX and
x86_64-darwinnow require 10.13 or higher (High Sierra, Mojave, and Catalina);
x86_64-darwinlegacysupports 10.6 and newer. MacTeX is notarized and command line programs have hardened runtimes, as now required by Apple for install packages. BibDesk and TeX Live Utility are not in MacTeX because they are not notarized, but a README le lists urls where they can be obtained. tlmgr and infrastructure:
tlmgr check texmfdbs, to to check consistency of
!!speci cations for each tree.
tlnet/archive/pkgname.rNNN.tar.xz; should be invisible to users, but a notable change in distribution.
catalogue-dateinformation no longer propagated from the TeX Catalogue, since it was often unrelated to package updates.
... something eerily similar to the infamous curl pipe bash method which I often decry. As a short-term workaround, I relied on the SHA-1 checksum of the repository to make sure I have the right code, by running this both on a "trusted" (ie. "local") repository and the remote, then visually comparing the output:
git clone REPO ./REPO/bootstrap.sh
One problem with this approach is that SHA-1 is now considered as flawed as MD5 so it can't be used as an authentication mechanism anymore. It's also fundamentally difficult to compare hashes for humans. The other flaw with comparing local and remote checksums is that we assume we trust the local repository. But how can I trust that repository? I can either:
$ git show-ref master 9f9a9d70dd1f1e84dec69a12ebc536c1f05aed1c refs/heads/master
That's the output of
commit b3c538898b0ed4e31da27fc9ca22cb55e1de0000 Author: Antoine Beaupr <firstname.lastname@example.org> Date: Mon Mar 16 14:37:28 2020 -0400 fix test autoloading pytest only looks for file names matching test by default. We inline tests inside the source code directly, so hijack that. diff --git a/fabric_tpa/pytest.ini b/fabric_tpa/pytest.ini new file mode 100644 index 0000000..71004ea --- /dev/null +++ b/fabric_tpa/pytest.ini @@ -0,0 +1,3 @@ +[pytest] +# we inline tests directly in the source code +python_files = *.py
git log -pin my local repository. I signed that commit, yet
git logis not telling me anything special. To check the signature, I need something special:
--show-signature, which looks like this:
Can you tell if this is a valid signature? If you speak a little french, maybe you can! But even if you would, you are unlikely to see that output on your own computer. What you would see instead is:
commit b3c538898b0ed4e31da27fc9ca22cb55e1de0000 gpg: Signature faite le lun 16 mar 2020 14:37:53 EDT gpg: avec la clef RSA 7B164204D096723B019635AB3EA1DDDDB261D97B gpg: Bonne signature de Antoine Beaupr <email@example.com> [ultime] gpg: alias Antoine Beaupr <firstname.lastname@example.org> [ultime] gpg: alias Antoine Beaupr <email@example.com> [ultime] gpg: alias Antoine Beaupr <firstname.lastname@example.org> [ultime] gpg: alias Antoine Beaupr <email@example.com> [ultime] Author: Antoine Beaupr <firstname.lastname@example.org> Date: Mon Mar 16 14:37:28 2020 -0400 fix test autoloading pytest only looks for file names matching test by default. We inline tests inside the source code directly, so hijack that.
commit b3c538898b0ed4e31da27fc9ca22cb55e1de0000 gpg: Signature made Mon Mar 16 14:37:53 2020 EDT gpg: using RSA key 7B164204D096723B019635AB3EA1DDDDB261D97B gpg: Can't check signature: No public key Author: Antoine Beaupr <email@example.com> Date: Mon Mar 16 14:37:28 2020 -0400 fix test autoloading pytest only looks for file names matching test by default. We inline tests inside the source code directly, so hijack that.
Can't check signature: No public key. No public key. Because of course you would see that. Why would you have my key lying around, unless you're me. Or, to put it another way, why would that server I'm installing from scratch have a copy of my OpenPGP certificate? Because I'm a Debian developer, my key is actually part of the 800 keys in the debian-keyring package, signed by the APT repositories. So I have a trust path. But that won't work for someone who is not a Debian developer. It will also stop working when my key expires in that repository, as it already has on Debian buster (current stable). So I can't assume I have a trust path there either. One could work with a trusted keyring like we do in the Tor and Debian project, and only work inside that project, that said. But I still feel uncomfortable with those commands. Both
git showwill happily succeed (return code 0 in the shell) even though the signature verification failed on the commits. Same with
git merge, which will happily push your branch ahead even if the remote has unsigned or badly signed commits. To actually verify commits (or tags), you need the git verify-commit (or git verify-tag) command, which seems to do the right thing:
At least it fails with some error code (
$ LANG=C.UTF-8 git verify-commit b3c538898b0ed4e31da27fc9ca22cb55e1de0000 gpg: Signature made Mon Mar 16 14:37:53 2020 EDT gpg: using RSA key 7B164204D096723B019635AB3EA1DDDDB261D97B gpg: Can't check signature: No public key $
1, above). But it's not flexible: I can't use it to verify that a "trusted" developer (say one that is in a trusted keyring) signed a given commit. Also, it is not clear what a failure means. Is a signature by an expired certificate okay? What if the key is signed by some random key in my personal keyring? Why should that be trusted?
In other words, even if git implements the arcane GnuPG dialect just so, and would allow us to setup the trust chain just right, and would give us meaningful and workable error messages, it still would fail because it's still stuck in SHA-1. There is work underway to fix that, but in February 2020, Jonathan Corbet described that work as being in a "relatively unstable state", which is hardly something I would like to trust to verify code. Also, when you clone a fresh new repository, you might get an entirely different repository, with a different root and set of commits. The concept of "validity" of a commit, in itself, is hard to establish in this case, because an hostile server could put you backwards in time, on a different branch, or even on an entirely different repository. Git will warn you about a different repository root with
<anarcat>i'd like to integrate pgp signing into tor's coding practices more, but so far, my approach has been "sign commits" and the verify step was "TBD"
<dkg>that's the main reason i've been reluctant to sign git commits. i haven't heard anyone offer a better subsequent step. if torproject could outline something useful, then i'd be less averse to the practice. i'm also pretty sad that git remains stuck on sha1, esp. given the recent demonstrations. all the fancy strong signatures you can make in git won't matter if the underlying git repo gets changed out from under the signature due to sha1's weakness
warning: no common commitsbut that's easy to miss. And complete branch switches, rebases and resets from upstream are hardly more noticeable: only a tiny plus sign (
+) instead of a star (
*) will tell you that a reset happened, along with a warning (
forced update) on the same line. Miss those and your git history can be compromised.
git tag -s. It's not designed to sign commits (it only verifies tags) but at least it uses a stronger algorithm (SHA-512) to checksum the tree, and will include everything in that tree, including blobs. If that sounds expensive to you, don't worry too much: it takes about 5 seconds to tag the Linux kernel, according to the author. Unfortunately, that checksum is then signed with GnuPG, in a manner similar to git itself, in that it exposes GnuPG output (which can be confusing) and is likely similarly vulnerable to mis-implementation of the GnuPG dialect as git itself. It also does not allow you to specify a keyring to verify against, so you need to trust GnuPG to make sense of the garbage that lives in your personal keyring (and, trust me, it doesn't). And besides,
git-evtagis fundamentally the same as signed git tags: checksum everything and sign with GnuPG. The difference is it uses SHA-512 instead of SHA-1, but that's something git will eventually fix itself anyways.
git-send-emailand teach git tools to recognize that (e.g.
git-am) at least if you're going to keep using OpenPGP anyways. And furthermore, it doesn't resolve the problems associated with verifying a full archive either, as it only attests "patches".
In other words, unless you have a repository that has frequent commits (either because of activity or by a bot generating fake commits), you have to rely on the central server to decide what "the latest version" is. This is the kind of problems that binary package distribution systems like APT and TUF solve correctly. Unfortunately, those don't apply to source code distribution, at least not in git form: TUF only deals with "repositories" and binary packages, and APT only deals with binary packages and source tarballs. That said, there's actually no reason why git could not support the TUF specification. Maybe TUF could be the solution to ensure end-to-end cryptographic integrity of the source code itself. OpenPGP-signed tarballs are nice, and signed git tags can be useful, but from my experience, a lot of OpenPGP (or, more accurately, GnuPG) derived tools are brittle and do not offer clear guarantees, and definitely not to the level that TUF tries to address. This would require changes on the git servers and clients, but I think it would be worth it.
<dkg>anarcat: even if you could do exactly what you describe, there are still some interesting wrinkles that i think would be problems for you. the big one: "git repo's latest commits" is a loophole big enough to drive a truck through. if your adversary controls that repo, then they get to decide which commits to include in the repo. (since every git repo is a view into the same git repo, just some have more commits than others)
gpg.minTrustLevelto "tell various signature verification codepaths the required minimum trust level", presumably to control how Git will treat keys in your keyrings, assuming the "trust database" is valid and up to date. For an interesting narrative of how "normal" (without PGP) git verification can fail, see also A Git Horror Story: Repository Integrity With Signed Commits.
Many voices arise now and then against risks linked to the Github use by Free Software projects. Yet the infatuation for the collaborative forge of the Octocat Californian start-ups doesn t seem to fade away.
These recent years, Github and its services take an important role in software engineering as they are seen as easy to use, efficient for a daily workload with interesting functions in enterprise collaborative workflow or amid a Free Software project. What are the arguments against using its services and are they valid? We will list them first, then we ll examine their validity.
The Github application belongs to a single entity, Github Inc, a US company which manage it alone. So, a unique company under US legislation manages the access to most of Free Software application code sources, which may be a problem with groups using it when a code source is no longer available, for political or technical reason.
This centralization leads to another trouble: as it obtained critical mass, it becomes more and more difficult not having a Github account. People who don t use Github, by choice or not, are becoming a silent minority. It is now fashionable to use Github, and not doing so is seen as out of date . The same phenomenon is a classic, and even the norm, for proprietary social networks (Facebook, Twitter, Instagram).
When you interact with Github, you are using a proprietary software, with no access to its source code and which may not work the way you think it is. It is a problem at different levels. First, ideologically, but foremost in practice. In the Github case, we send them code we can control outside of their interface. We also send them personal information (profile, Github interactions). And mostly, Github forces any project which goes through the US platform to use a crucial proprietary tools: its bug tracking system.
Working with Github interface seems easy and intuitive to most. Lots of companies now use it as a source repository, and many developers leaving a company find the same Github working environment in the next one. This pervasive presence of Github in free software development environment is a part of the uniformization of said developers working space.
As said above, nowadays, Github is the main repository of Free Software source code. As such it is a favorite target for cyberattacks. DDOS hit it in March and August 2015. On December 15, 2015, an outage led to the inaccessibility of 5% of the repositories. The same occurred on November 15. And these are only the incident reported by Github itself. One can imagine that the mean outage rate of the platform is underestimated.
One of the best examples is the npmgate. Any company could legally demand that Github take down some source code from its repository, which could create a chain reaction and blocking the development of many Free Software projects, as suffered the Node.js community from the decisions of Npm, Inc, the company managing npm.
Github didn t appear out of the blue. In his time, its predecessor, SourceForge, was also extremely popular.
Heavily centralized, based on strong interaction with the community, SourceForge is now seen as an aging SAAS (Software As A Service) and sees most of its customers fleeing to Github. Which creates lots of hurdles for those who stayed. The Gimp project suffered from spams and terrible advertising, which led to the departure of the VLC project, then from installers corrupted with adwares instead of the official Gimp installer for Windows. And finally, the Project Gimp s SourceForge account was hacked by SourceForge team itself!
These are very recent examples of what can do a commercial entity when it is under its stakeholders pressure. It is vital to really understand what it means to trust them with data and exchange centralization, where it could have tremendous repercussion on the day-to-day life and the habits of the Free Software and open source community.
Mostly based on ideology, this point deals with the definition every member of the community gives to Free Software and open source. Mostly about one thing: is it viral or not? Or GPL vs MIT/BSD.
Those on the side of the viral Free Software will have trouble to use a proprietary software as this last one shouldn t even exist. It must be assimilated, to quote Star Trek, as it is a connected black box, endangering privacy, corrupting for profit our uses and restrain our freedom to use as we re pleased what we own, etc.
Those on the side of complete freedom have no qualms using proprietary software as their very existence is a consequence of freedom without restriction. They even agree that code they developed may be a part of proprietary software, which is quite a common occurrence. This part of the Free Software community has no qualm using Github, which is well within their ideology parameters. Just take a look at the Janson amphitheater during Fosdem and check how many Apple laptops running on macOS are around.
Even without ideological consideration, and just focusing on Github infrastructure, the bug tracking system is a major problem by itself.
Bug report builds the memory of Free Software projects. It is the entrance point for new contributors, the place to find bug reporting, requests for new functions, etc. The project history can t be limited only to the code. It s very common to find bug reports when you copy and paste an error message in a search engine. Not their historical importance is precious for the project itself, but also for its present and future users.
Github gives the ability to extract bug reports through its API. What would happen if Github is down or if the platform doesn t support this feature anymore? In my opinion, not that many projects ever thought of this outcome. How could they move all the data generated by Github into a new bug tracking system? One old example now is Astrid, a TODO list bought by Yahoo a few years ago. Very popular, it grew fast until it was closed overnight, with only a few weeks for its users to extract their data. It was only a to-do list. The same situation with Github would be tremendously difficult to manage for several projects if they even have the ability to deal with it. Code would still be available and could still live somewhere else, but the project memory would be lost. A project like Debian has today more than 800,000 bug reports, which are a data treasure trove about problems solved, function requests and where the development stand on each. The developers of the Cpython project have anticipated the problem and decided not to use Github bug tracking systems.
Another thing we could lose if Github suddenly disappear: all the work currently done regarding the push requests (aka PRs). This Github function gives the ability to clone one project s Github repository, to modify it to fit your needs, then to offer your own modification to the original repository. The original repository s owner will then review said modification, and if he or she agrees with them will fuse them into the original repository. As such, it s one of the main advantages of Github, since it can be done easily through its graphic interface.
However reviewing all the PRs may be quite long, and most of the successful projects have several ongoing PRs. And this PRs and/or the proprietary bug tracking system are commonly used as a platform for comment and discussion between developers.
Code itself is not lost if Github is down (except one specific situation as seen below), but the peer review works materialized in the PRs and the bug tracking system is lost. Let s remember than the PR mechanism let you clone and modify projects and then generate PRs directly from its proprietary web interface without downloading a single code line on your computer. In this particular case, if Github is down, all the code and the work in progress is lost. Some also use Github as a bookmark place. They follow their favorite projects activity through the Watch function. This technological watch style of data collection would also be lost if Github is down.
The Free Software community is walking a thigh rope between normalization needed for an easier interoperability between its products and an attraction for novelty led by a strong need for differentiation from what is already there.
Github popularized the use of Git, a great tool now used through various sectors far away from its original programming field. Step by step, Git is now so prominent it s almost impossible to even think to another source control manager, even if awesome alternate solutions, unfortunately not as popular, exist as Mercurial.
A new Free Software project is now a Git repository on Github with README.md added as a quick description. All the other solutions are ostracized? How? None or very few potential contributors would notice said projects. It seems very difficult now to encourage potential contributors into learning a new source control manager AND a new forge for every project they want to contribute. Which was a basic requirement a few years ago. It s quite sad because Github, offering an original experience to its users, cut them out of a whole possibility realm. Maybe Github is one of the best web versioning control systems. But being the main one doesn t let room for a new competitor to grow. And it let Github initiate development newcomers into a narrow function set, totally unrelated to the strength of the Git tool itself.
Fight against centralization is a main part of the Free Software ideology as centralization strengthens the power of those who manage it and who through it control those who are managed by it. Uniformization allergies born against main software companies and their wishes to impose a closed commercial software world was for a long time the main fuel for innovation thirst and intelligent alternative development. As we said above, part of the Free Software community was built as a reaction to proprietary software and their threat. The other part, without hoping for their disappearance, still chose a development model opposite to proprietary software, at least in the beginning, as now there s more and more bridges between the two.
The Github effect is a morbid one because of its consequences: at least centralization, uniformization, proprietary software usage as their bug tracking system. But some years ago the Dear Github buzz showed one more side effect, one I ve never thought about: laziness. For those who don t know what it is about, this letter is a complaint from several spokespersons from several Free Software projects which demand to Github team to finally implement, after years of polite asking, new functions. Since when Free Software project facing a roadblock request for clemency and don t build themselves the path they need? When Torvalds was involved in the Bitkeeper problem and the Linux kernel development team couldn t use anymore their revision control software, he developed Git. The mere fact of not being able to use one tool or functions lacking is the main motivation to seek alternative solutions and, as such, of the Free Software movement. Every Free Software community member able to code should have this reflex. You don t like what Github offers? Switch to Gitlab. You don t like it Gitlab? Improve it or make your own solution.
Let s be crystal clear. I ve never said that every Free Software developers blocked should code his or her own alternative. We all have our own priorities, and some of us even like their beauty sleep, including me. But, to see that this open letter to Github has 1340 names attached to it, among them some spokespersons for major Free Software project showed me that need, willpower and strength to code a replacement are here. Maybe said replacement will be born from this letter, it would be the best outcome of this buzz.
In the end, Github usage is just another example of Internet usage massification. As Internet users are bound to go to massively centralized social network as Facebook or Twitter, developers are following the same path with Github. Even if a large fraction of developers realize the threat linked this centralized and proprietary organization, the whole community is following this centralization and uniformization trend. Github service is useful, free or with a reasonable price (depending on the functions you need) easy to use and up most of the time. Why would we try something else? Maybe because others are using us while we are savoring the convenience? The Free Software community seems to be quite sleepy to me.About Me Carl Chenet, Free Software Indie Hacker, founder of the French-speaking Hacker News-like Journal du hacker. Follow me on social networks
Documentation/security/tree had been converted yet. I took the opportunity to take a few passes at formatting the existing documentation and, at Jon Corbet s recommendation, split it up between end-user documentation (which is mainly how to use LSMs) and developer documentation (which is mainly how to use various internal APIs). A bunch of these docs need some updating, so maybe with the improved visibility, they ll get some extra attention. CONFIG_REFCOUNT_FULL
refcount_tAPI in v4.11, Elena Reshetova (with Hans Liljestrand and David Windsor) has been systematically replacing
atomic_treference counters with
refcount_t. As of v4.13, there are now close to 125 conversions with many more to come. However, there were concerns over the performance characteristics of the
refcount_timplementation from the maintainers of the net, mm, and block subsystems. In order to assuage these concerns and help the conversion progress continue, I added an unchecked
refcount_timplementation (identical to the earlier
atomic_timplementation) as the default, with the fully checked implementation now available under
CONFIG_REFCOUNT_FULL. The plan is that for v4.14 and beyond, the kernel can grow per-architecture implementations of
refcount_tthat have performance characteristics on par with
atomic_t(as done in grsecurity s PAX_REFCOUNT). CONFIG_FORTIFY_SOURCE
FORTIFY_SOURCEcompile-time and run-time protection for finding overflows in the common string (e.g.
strcmp) and memory (e.g.
memcmp) functions. The idea is that since the compiler already knows the size of many of the buffer arguments used by these functions, it can already build in checks for buffer overflows. When all the sizes are known at compile time, this can actually allow the compiler to fail the build instead of continuing with a proven overflow. When only some of the sizes are known (e.g. destination size is known at compile-time, but source size is only known at run-time) run-time checks are added to catch any cases where an overflow might happen. Adding this found several places where minor leaks were happening, and Daniel and I chased down fixes for them. One interesting note about this protection is that is only examines the size of the whole object for its size (via
__builtin_object_size(..., 0)). If you have a string within a structure,
CONFIG_FORTIFY_SOURCEas currently implemented will make sure only that you can t copy beyond the structure (but therefore, you can still overflow the string within the structure). The next step in enhancing this protection is to switch from 0 (above) to 1, which will use the closest surrounding subobject (e.g. the string). However, there are a lot of cases where the kernel intentionally copies across multiple structure fields, which means more fixes before this higher level can be enabled. NULL-prefixed stack canary
strcpy), since they will either stop an overflowing read at the NULL byte, or be unable to write a NULL byte, thereby always triggering the canary check. This does reduce the entropy from 64 bits to 56 bits for overflow cases where NULL bytes can be written (e.g.
memcpy), but the trade-off is worth it. (Besdies, x86_64 s canary was 32-bits until recently.) IPC refactoring
__randomize_layout). v4.14 will also have the automatic mode enabled, which randomizes all structures that contain only function pointers. A large number of fixes to support randstruct have been landing from v4.10 through v4.13, most of which were already identified and fixed by grsecurity, but many were novel, either in newly added drivers, as whitelisted cross-structure casts, refactorings (like IPC noted above), or in a corner case on ARM found during upstream testing. lower ELF_ET_DYN_BASE
ELF_ET_DYN_BASE(the lowest possible random position of a PIE executable in memory) was already so high in the memory layout (specifically, 2/3rds of the way through the address space). Fixing this required teaching the ELF loader how to load interpreters as shared objects in the mmap region instead of as a PIE executable (to avoid potentially colliding with the binary it was loading). As a result, the PIE default could be moved down to ET_EXEC (0x400000) on 32-bit, entirely avoiding the subset of Stack Clash attacks. 64-bit could be moved to just above the 32-bit address space (0x100000000), leaving the entire 32-bit region open for VMs to do 32-bit addressing, but late in the cycle it was discovered that Address Sanitizer couldn t handle it moving. With most of the Stack Clash risk only applicable to 32-bit, fixing 64-bit has been deferred until there is a way to teach Address Sanitizer how to load itself as a shared object instead of as a PIE binary. early device randomness
2017, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
struct fib6_node. When a node has the
RTN_RTINFOflag set, it embeds a pointer to a
struct rt6_infocontaining information about the next-hop. The
fib6_lookup_1()function walks the radix tree in two steps:
static struct fib6_node *fib6_lookup_1(struct fib6_node *root, struct in6_addr *addr) struct fib6_node *fn; __be32 dir; /* Step 1: locate potential candidate */ fn = root; for (;;) struct fib6_node *next; dir = addr_bit_set(addr, fn->fn_bit); next = dir ? fn->right : fn->left; if (next) fn = next; continue; break; /* Step 2: check prefix and backtrack if needed */ while (fn) if (fn->fn_flags & RTN_RTINFO) struct rt6key *key; key = fn->leaf->rt6i_dst; if (ipv6_prefix_equal(&key->addr, addr, key->plen)) if (fn->fn_flags & RTN_RTINFO) return fn; if (fn->fn_flags & RTN_ROOT) break; fn = fn->parent; return NULL;
2001:db8:3::1would get those two cache entries:
$ ip -6 route show cache 2001:db8:1::1 dev r2-r1 metric 0 cache 2001:db8:3::1 via 2001:db8:2::2 dev r2-r3 metric 0 cache
ip6_dst_gc()function controlled by the following parameters:
$ sysctl -a grep -F net.ipv6.route net.ipv6.route.gc_elasticity = 9 net.ipv6.route.gc_interval = 30 net.ipv6.route.gc_min_interval = 0 net.ipv6.route.gc_min_interval_ms = 500 net.ipv6.route.gc_thresh = 1024 net.ipv6.route.gc_timeout = 60 net.ipv6.route.max_size = 4096 net.ipv6.route.mtu_expires = 600
gc_elasticity) after a 500 ms pause. Starting from Linux 4.2 (commit 45e4fd26683c), only a PMTU exception would create a cache entry. A router doesn t have to handle those exceptions, so only hosts would get cache entries. And they should be pretty rare. Martin KaFai Lau explains:
Out of all IPv6Here is how a cache entry with a PMTU exception looks like:
RTF_CACHEroutes that are created, the percentage that has a different MTU is very small. In one of our end-user facing proxy server, only 1k out of 80k
RTF_CACHEroutes have a smaller MTU. For our DC traffic, there is no MTU exception.
$ ip -6 route show cache 2001:db8:1::50 via 2001:db8:1::13 dev out6 metric 0 cache expires 573sec mtu 1400 pref medium
ip6_route_output_flags()function and correlate the results with the radix tree size: Getting meaningful results is challenging due to the size of the address space. None of the scenarios have a fallback route and we only measure time for successful hits4. For the full view scenario, only the range from
2a06::/16is scanned (it contains more than half of the routes). For the /128 scenario, the whole /108 subnet is scanned. For the /48 scenario, the range from the first /48 to the last one is scanned. For each range, 5000 addresses are picked semi-randomly. This operation is repeated until we get 5000 hits or until 1 million tests have been executed. The relation between the maximum depth and the lookup time is incomplete and I can t explain the difference of performance between the different densities of the /48 scenario. We can extract two important performance points:
fib_lookup()and IPv6 s
ip6_route_output_flags()functions have a fixed cost implied by the evaluation of routing rules, IPv4 has several optimizations when the rules are left unmodified5. Those optimizations are removed on the first modification. If we cancel those optimizations, the lookup time for IPv4 is impacted by about 30 ns. This still leaves a 100 ns difference with IPv6 to be explained. Let s compare how time is spent in each lookup function. Here is a CPU flamegraph for IPv4 s
fib_lookup(): Only 50% of the time is spent in the actual route lookup. The remaining time is spent evaluating the routing rules (about 30 ns). This ratio is dependent on the number of routes we inserted (only 1000 in this example). It should be noted the
fib_table_lookup()function is executed twice: once with the local routing table and once with the main routing table. The equivalent flamegraph for IPv6 s
ip6_route_output_flags()is depicted below: Here is an approximate breakdown on the time spent:
CONFIG_IPV6_SUBTREESoptions enabled. Some other unrelated options are enabled to be able to boot them in a virtual machine and run the benchmark. There are three notable performance changes:
struct rt6_info(commit 887c95cc1da5). This should have lead to a performance increase. The small regression may be due to cache-related issues.
struct fib6_node) and routing information (
struct rt6_info) are allocated with the slab allocator7. It is therefore possible to extract the information from
/proc/slabinfowhen the kernel is booted with the
# sed -ne 2p -e '/^ip6_dst/p' -e '/^fib6_nodes/p' /proc/slabinfo cut -f1 -d: name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> fib6_nodes 76101 76104 64 63 1 ip6_dst_cache 40090 40090 384 10 1
struct rt6_infomatches the number of routes while the number of nodes is roughly twice the number of routes: The memory usage is therefore quite predictable and reasonable, as even a small single-board computer can support several full views (20 MiB for each): The LPC-trie used for IPv4 is more efficient: when 512 MiB of memory is needed for IPv6 to store 1 million routes, only 128 MiB are needed for IPv4. The difference is mainly due to the size of
struct rt6_info(336 bytes) compared to the size of IPv4 s
struct fib_alias(48 bytes): IPv4 puts most information about next-hops in
struct fib_infostructures that are shared with many entries.
CONFIG_IPV6_MULTIPLE_TABLESoption incurs a fixed penalty of 100 ns by lookup,
ip -6 route add 2001:db8:1::/64 \ from 2001:db8:3::/64 \ via fe80::1 \ dev eth0
CONFIG_IP_MULTIPLE_TABLESoptions enabled (however, no IP rules are used). Some other unrelated options are enabled to be able to boot them in a virtual machine and run the benchmark. The measurements are done in a virtual machine with one vCPU2. The host is an Intel Core i5-4670K and the CPU governor was set to performance . The benchmark is single-threaded. Implemented as a kernel module, it calls
fib_lookup()with various destinations in 100,000 timed iterations and keeps the median. Timings of individual runs are computed from the TSC (and converted to nanoseconds by assuming a constant clock). The following kernel versions bring a notable performance improvement:
CONFIG_IP_MULTIPLE_TABLESoption doesn t impact the performances unless some IP rules are configured. This version also removes the route cache (commit 5e9965c15ba8). However, this has no effect on the benchmark as it directly calls
fib_lookup()which doesn t involve the cache.
fib_lookup()function. From essential information about the datagram (source and destination IP addresses, interfaces, firewall mark, ), this function should quickly provide a decision. Some possible options are:
$ ip route show scope global table 100 default via 203.0.113.5 dev out2 192.0.2.0/25 nexthop via 203.0.113.7 dev out3 weight 1 nexthop via 203.0.113.9 dev out4 weight 1 192.0.2.47 via 203.0.113.3 dev out1 192.0.2.48 via 203.0.113.3 dev out1 192.0.2.49 via 203.0.113.3 dev out1 192.0.2.50 via 203.0.113.3 dev out1
|Destination IP||Next hop|
192.0.2.50, we will find the result in the corresponding leaf (at depth 32). However for
192.0.2.51, we will reach
192.0.2.50/31but there is no second child. Therefore, we backtrack until the
192.0.2.0/25routing entry. Adding and removing routes is quite easy. From a performance point of view, the lookup is done in constant time relative to the number of routes (due to maximum depth being capped to 32). Quagga is an example of routing software still using this simple approach.
struct fib_tablerepresents a routing table,
struct trierepresents a complete trie,
struct key_vectorrepresents either an internal node (when
bitsis not zero) or a leaf,
struct fib_infocontains the characteristics shared by several routes (like a next-hop gateway and an output interface),
struct fib_aliasis the glue between the leaves and the
$ cat /proc/net/fib_trie Id 100: +-- 0.0.0.0/0 2 0 2 -- 0.0.0.0 /0 universe UNICAST +-- 192.0.2.0/26 2 0 1 -- 192.0.2.0 /25 universe UNICAST -- 192.0.2.47 /32 universe UNICAST +-- 192.0.2.48/30 2 0 1 -- 192.0.2.48 /32 universe UNICAST -- 192.0.2.49 /32 universe UNICAST -- 192.0.2.50 /32 universe UNICAST [...]
CONFIG_IP_FIB_TRIE_STATS, some interesting statistics are available in
$ cat /proc/net/fib_triestat Basic info: size of leaf: 48 bytes, size of tnode: 40 bytes. Id 100: Aver depth: 2.33 Max depth: 3 Leaves: 6 Prefixes: 6 Internal nodes: 3 2: 3 Pointers: 12 Null ptrs: 4 Total size: 1 kB [...]
fib_lookup()function: The lookup time is loosely tied to the maximum depth. When the routing table is densily populated, the maximum depth is low and the lookup times are fast. When forwarding at 10 Gbps, the time budget for a packet would be about 50 ns. Since this is also the time needed for the route lookup alone in some cases, we wouldn t be able to forward at line rate with only one core. Nonetheless, the results are pretty good and they are expected to scale linearly with the number of cores. The measurements are done with a Linux kernel 4.11 from Debian unstable. I have gathered performance metrics accross kernel versions in Performance progression of IPv4 route lookup on Linux . Another interesting figure is the time it takes to insert all those routes into the kernel. Linux is also quite efficient in this area since you can insert 2 million routes in less than 10 seconds:
/proc/net/fib_triestat. The statistic provided doesn t account for the
fib_infostructures, but you should only have a handful of them (one for each possible next-hop). As you can see on the graph below, the memory use is linear with the number of routes inserted, whatever the shape of the routes is. The results are quite good. With only 256 MiB, about 2 million routes can be stored!
CONFIG_IP_MULTIPLE_TABLES, Linux supports several routing tables and has a system of configurable rules to select the table to use. These rules can be configured with
ip rule. By default, there are three of them:
$ ip rule show 0: from all lookup local 32766: from all lookup main 32767: from all lookup default
localtable. If it doesn t find one, it will lookup in the
maintable and at last resort, the
localtable contains routes for local delivery:
$ ip route show table local broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 broadcast 192.168.117.0 dev eno1 proto kernel scope link src 192.168.117.55 local 192.168.117.55 dev eno1 proto kernel scope host src 192.168.117.55 broadcast 192.168.117.63 dev eno1 proto kernel scope link src 192.168.117.55
192.168.117.55was configured on the
eno1interface, the kernel automatically added the appropriate routes:
192.168.117.55for local unicast delivery to the IP address,
192.168.117.255for broadcast delivery to the broadcast address,
192.168.117.0for broadcast delivery to the network address.
127.0.0.1was configured on the loopback interface, the same kind of routes were added to the
localtable. However, a loopback address receives a special treatment and the kernel also adds the whole subnet to the
localtable. As a result, you can ping any IP in
$ ping -c1 127.42.42.42 PING 127.42.42.42 (127.42.42.42) 56(84) bytes of data. 64 bytes from 127.42.42.42: icmp_seq=1 ttl=64 time=0.039 ms --- 127.42.42.42 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.039/0.039/0.039/0.000 ms
maintable usually contains all the other routes:
$ ip route show table main default via 192.168.117.1 dev eno1 proto static metric 100 192.168.117.0/26 dev eno1 proto kernel scope link src 192.168.117.55 metric 100
defaultroute has been configured by some DHCP daemon. The connected route (
scope link) has been automatically added by the kernel (
proto kernel) when configuring an IP address on the
defaulttable is empty and has little use. It has been kept when the current incarnation of advanced routing has been introduced in Linux 2.1.68 after a first tentative using classes in Linux 2.1.156.
localtables are merged and the lookup is done with this single table (and the
defaulttable if not empty). Moreover, since Linux 3.0 (commit f4530fa574df), without specific rules, there is no performance hit when enabling the support for multiple routing tables. However, as soon as you add new rules, some CPU cycles will be spent for each datagram to evaluate them. Here is a couple of graphs demonstrating the impact of routing rules on lookup times: For some reason, the relation is linear when the number of rules is between 1 and 100 but the slope increases noticeably past this threshold. The second graph highlights the negative impact of the first rule (about 30 ns). A common use of rules is to create virtual routers: interfaces are segregated into domains and when a datagram enters through an interface from domain A, it should use routing table A:
# ip rule add iif vlan457 table 10 # ip rule add iif vlan457 blackhole # ip rule add iif vlan458 table 20 # ip rule add iif vlan458 blackhole
# ip route add blackhole default metric 9999 table 10 # ip route add blackhole default metric 9999 table 20 # ip rule add iif vlan457 table 10 # ip rule add iif vlan458 table 20
# ip link add vrf-A type vrf table 10 # ip link set dev vrf-A up # ip link add vrf-B type vrf table 20 # ip link set dev vrf-B up # ip link set dev vlan457 master vrf-A # ip link set dev vlan458 master vrf-B # ip rule show 0: from all lookup local 1000: from all lookup [l3mdev-table] 32766: from all lookup main 32767: from all lookup default
l3mdev-tablerule was automatically added when configuring the first VRF interface. This rule will select the routing table associated to the VRF owning the input (or output) interface. VRF was introduced in Linux 4.3 (commit 193125dbd8eb), the performance was greatly enhanced in Linux 4.8 (commit 7889681f4a6c) and the special routing rule was also introduced in Linux 4.8 (commit 96c63fa7393d, commit 1aa6c4f6b8cd). You can find more details about it in the kernel documentation.
key_vectorstructure is embedded into a
tnodestructure. This structure contains information rarely used during lookup, notably the reference to the parent that is usually not needed for backtracking as Linux keeps the nearest candidate in a variable.
struct fib_aliasis a list). The number of prefixes can therefore be greater than the number of leaves. The system also keeps statistics about the distribution of the internal nodes relative to the number of bits they handle. In our example, all the three internal nodes are handling 2 bits.
To verify the timestamp, you first need to download the public key of the trusted timestamp service, for example using this command:openssl ts -query -data "$inputfile" -cert -sha256 -no_nonce \ curl -s -H "Content-Type: application/timestamp-query" \ --data-binary "@-" http://zeitstempel.dfn.de > $sha256.tsr
Note, the public key should be stored alongside the timestamps in the archive to make sure it is also available 100 years from now. It is probably a good idea to standardise how and were to store such public keys, to make it easier to find for those trying to verify documents 100 or 1000 years from now. :) The verification itself is a simple openssl command:wget -O ca-cert.txt \ https://pki.pca.dfn.de/global-services-ca/pub/cacert/chain.txt
Is there any reason this approach would not work? Is it somehow against the Noark 5 specification?openssl ts -verify -data $inputfile -in $sha256.tsr \ -CAfile ca-cert.txt -text
You can see here how the fonds (arkiv) and serie (arkivdel) only had one option, while the user need to choose which file (mappe) to use among the two created by the API tester. The archive-pdf tool can be found in the git repository for the API tester. In the project, I have been mostly working on the API tester so far, while getting to know the code base. The API tester currently use the HATEOAS links to traverse the entire exposed service API and verify that the exposed operations and objects match the specification, as well as trying to create objects holding metadata and uploading a simple XML file to store. The tester has proved very useful for finding flaws in our implementation, as well as flaws in the reference site and the specification. The test document I uploaded is a summary of all the specification defects we have collected so far while implementing the web service. There are several unclear and conflicting parts of the specification, and we have started writing down the questions we get from implementing it. We use a format inspired by how The Austin Group collect defect reports for the POSIX standard with their instructions for the MANTIS defect tracker system, in lack of an official way to structure defect reports for Noark 5 (our first submitted defect report was a request for a procedure for submitting defect reports :). The Nikita project is implemented using Java and Spring, and is fairly easy to get up and running using Docker containers for those that want to test the current code base. The API tester is implemented in Python.~/src//noark5-tester$ ./archive-pdf mangelmelding/mangler.pdf using arkiv: Title of the test fonds created 2017-03-18T23:49:32.103446 using arkivdel: Title of the test series created 2017-03-18T23:49:32.103446 0 - Title of the test case file created 2017-03-18T23:49:32.103446 1 - Title of the test file created 2017-03-18T23:49:32.103446 Select which mappe you want (or search term): 0 Uploading mangelmelding/mangler.pdf PDF title: Mangler i spesifikasjonsdokumentet for NOARK 5 Tjenestegrensesnitt File 2017/1: Title of the test case file created 2017-03-18T23:49:32.103446 ~/src//noark5-tester$
Four keynote speakers will anchor the event. Kade Crockford, director of the Technology for Liberty program of the American Civil Liberties Union of Massachusetts, will kick things off on Saturday morning by sharing how technologists can enlist in the growing fight for civil liberties. On Saturday night, Free Software Foundation president Richard Stallman will present the Free Software Awards and discuss pressing threats and important opportunities for software freedom. Day two will begin with Cory Doctorow, science fiction author and special consultant to the Electronic Frontier Foundation, revealing how to eradicate all Digital Restrictions Management (DRM) in a decade. The conference will draw to a close with Sumana Harihareswara, leader, speaker, and advocate for free software and communities, giving a talk entitled "Lessons, Myths, and Lenses: What I Wish I'd Known in 1998." That's not all. We'll hear about the GNU philosophy from Marianne Corvellec of the French free software organization April, Joey Hess will touch on encryption with a talk about backing up your GPG keys, and Denver Gingerich will update us on a crucial free software need: the mobile phone. Others will look at ways to grow the free software movement: through cross-pollination with other activist movements, removal of barriers to free software use and contribution, and new ideas for free software as paid work.-- Here's a sneak peek at LibrePlanet 2017: Register today! I'll be giving some varient of the keysafe talk from Linux.Conf.Au. By the way, videos of my keysafe and propellor talks at Linux.Conf.Au are now available, see the talks page.
Before the 2.6 series, there was a stable branch (2.4) where only relatively minor and safe changes were merged, and an unstable branch (2.5), where bigger changes and cleanups were allowed. Both of these branches had been maintained by the same set of people, led by Torvalds. This meant that users would always have a well-tested 2.4 version with the latest security and bug fixes to use, though they would have to wait for the features which went into the 2.5 branch. The downside of this was that the stable kernel ended up so far behind that it no longer supported recent hardware and lacked needed features. In the late 2.5 kernel series, some maintainers elected to try backporting of their changes to the stable kernel series, which resulted in bugs being introduced into the 2.4 kernel series. The 2.5 branch was then eventually declared stable and renamed to 2.6. But instead of opening an unstable 2.7 branch, the kernel developers decided to continue putting major changes into the 2.6 branch, which would then be released at a pace faster than 2.4.x but slower than 2.5.x. This had the desirable effect of making new features more quickly available and getting more testing of the new code, which was added in smaller batches and easier to test. Then, in the Ruby community. In 2007, Ruby 1.8.6 was the stable version of Ruby. Ruby 1.9.0 was released on 2007-12-26, without being declared stable, as a snapshot from Ruby s trunk branch, and most of the development s attention moved to 1.9.x. On 2009-01-31, Ruby 1.9.1 was the first release of the 1.9 branch to be declared stable. But at the same time, the disruptive changes introduced in Ruby 1.9 made users stay with Ruby 1.8, as many libraries (gems) remained incompatible with Ruby 1.9.x. Debian provided packages for both branches of Ruby in Squeeze (2011) but only changed the default to 1.9 in 2012 (in a stable release with Wheezy 2013). Finally, in the Python community. Similarly to what happened with Ruby 1.9, Python 3.0 was released in December 2008. Releases from the 3.x branch have been shipped in Debian Squeeze (3.1), Wheezy (3.2), Jessie (3.4). But the python command still points to 2.7 (I don t think that there are plans to make it point to 3.x, making python 3.x essentially a different language), and there are talks about really getting rid of Python 2.7 in Buster (Stretch+1, Jessie+2). In retrospect, and looking at what those projects have been doing in recent years, it is probably a better idea to break early, break often, and fix a constant stream of breakages, on a regular basis, even if that means temporarily exposing breakage to users, and spending more time seeking strategies to limit the damage caused by introducing breakage. What also changed since the time those branches were introduced is the increased popularity of automated testing and continuous integration, which makes it easier to measure breakage caused by disruptive changes. Distributions are in a good position to help here, by being able to provide early feedback to upstream projects about potentially disruptive changes. And distributions also have good motivations to help here, because it is usually not a great solution to ship two incompatible branches of the same project. (I wonder if there are other occurrences of the same pattern?) Update: There s a discussion about this post on HN
__latent_entropy. These functions have their branches and loops adjusted to mix random values (selected at build time) into a global entropy gathering variable. Since the branch and loop ordering is very specific to boot conditions, CPU quirks, memory layout, etc, this provides some additional uncertainty to the kernel s entropy pool. Since the entropy actually gathered is hard to measure, no entropy is credited , but rather used to mix the existing pool further. Probably the best place to enable this plugin is on small devices without other strong sources of entropy. vmapped kernel stack and thread_info relocation on x86 Normally, kernel stacks are mapped together in memory. This meant that attackers could use forms of stack exhaustion (or stack buffer overflows) to reach past the end of a stack and start writing over another process s stack. This is bad, and one way to stop it is to provide guard pages between stacks, which is provided by
vmalloced memory. Andy Lutomirski did a bunch of work to move to vmapped kernel stack via
CONFIG_VMAP_STACKon x86_64. Now when writing past the end of the stack, the kernel will immediately fault instead of just continuing to blindly write. Related to this, the kernel was storing
thread_info(which contained sensitive values like
addr_limit) at the bottom of the kernel stack, which was an easy target for attackers to hit. Between a combination of explicitly moving targets out of
thread_info, removing needless fields, and entirely moving
thread_infooff the stack, Andy Lutomirski and Linus Torvalds created
CONFIG_THREAD_INFO_IN_TASKfor x86. CONFIG_DEBUG_RODATA mandatory on arm64 As recently done for x86, Mark Rutland made CONFIG_DEBUG_RODATA mandatory on arm64. This feature controls whether the kernel enforces proper memory protections on its own memory regions (code memory is executable and read-only, read-only data is actually read-only and non-executable, and writable data is non-executable). This protection is a fundamental security primitive for kernel self-protection, so there s no reason to make the protection optional.
random_page()cleanup Cleaning up the code around the userspace ASLR implementations makes them easier to reason about. This has been happening for things like the recent consolidation on
ET_DYNand during the addition of the entropy sysctl. Both uncovered some awkward uses of
get_random_int()(or similar) in and around
arch_mmap_rnd()(which is used for
mmap(and therefore shared library) and PIE ASLR), as well as in
randomize_stack_top()(which is used for stack ASLR). Jason Cooper cleaned things up further by doing away with
randomize_range()entirely and replacing it with the saner
random_page(), making the per-architecture
brkASLR) much easier to understand. That s it for now! Let me know if there are other fun things to call attention to in v4.9.
2016, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
uname -a). It is unclear why those patches are necessary since the ARMv7 Armada 385 CPU has been supported in Linux since at least 4.2-rc1, but it is common for OpenWRT ports to ship patches to the kernel, either to backport missing functionality or perform some optimization. There has been some pressure from backers to petition Turris to "speedup the process of upstreaming Omnia support to OpenWrt". It could be that the team is too busy with delivering the devices already ordered to complete that process at this point. The software is available on the CZ-NIC GitHub repository and the actual Linux patches can be found here and here. CZ.NIC also operates a private GitLab instance where more software is available. There is technically no reason why you wouldn't be able to run your own distribution on the Omnia router: OpenWRT development snapshots should be able to run on the Omnia hardware and some people have installed Debian on Omnia. It may require some customization (e.g. the kernel) to make sure the Omnia hardware is correctly supported. Most people seem to prefer to run TurrisOS because of the extra features. The hardware itself is also free and open for the most part. There is a binary blob needed for the 5GHz wireless card, which seems to be the only proprietary component on the board. The schematics of the device are available through the Omnia wiki, but oddly not in the GitHub repository like the rest of the software.
Note: this article first appeared in the Linux Weekly News.
unzip angler-nrd90u-factory-7c9b6a2b.zip cd angler-nrd90u/ unzip image-angler-nrd90u.zip
fastboot flash bootloader bootloader-angler-angler-03.58.img fastboot reboot-bootloader sleep 5 fastboot flash radio radio-angler-angler-03.72.img fastboot reboot-bootloader sleep 5 fastboot erase system fastboot flash system system.img fastboot erase boot fastboot flash boot boot.img fastboot erase cache fastboot flash cache cache.img fastboot erase vendor fastboot flash vendor vendor.img fastboot erase recovery fastboot flash recovery recovery.img fastboot reboot
fastboot erase recovery fastboot flash recovery twrp-3.0.2-2-angler.img fastboot reboot-bootloader