| Series: | Mage Winds #2 |
| Publisher: | DAW |
| Copyright: | August 1993 |
| ISBN: | 0-88677-563-9 |
| Format: | Mass market |
| Pages: | 475 |
An new version 0.0.8 of RcppAnnoy, our Rcpp-based R integration of the nifty Annoy library by Erik, is now on CRAN. Annoy is a small, fast, and lightweight C++ template header library for approximate nearest neighbours.
This release pulls in a few suggested changes which had piled up since the last release.
Changes in this version are summarized here:
Courtesy of CRANberries, there is also a diffstat report for this release.Changes in version 0.0.8 (2016-10-01)
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
I sent this email to debian-private a few days ago, on the 10th anniversary of my Debian account creation:
Date: Fri, 14 Aug 2015 19:37:20 +0200 From: David Moreno To: debian-private@lists.debian.org Subject: Retiring from Debian User-Agent: Mutt/1.5.23 (2014-03-12) [-- PGP output follows (current time: Sun 23 Aug 2015 06:18:36 PM CEST) --] gpg: Signature made Fri 14 Aug 2015 07:37:20 PM CEST using RSA key ID 4DADEC2F gpg: Good signature from "David Moreno " gpg: aka "David Moreno " gpg: aka "David Moreno (1984-08-08) " [-- End of PGP output --] [-- The following data is signed --] Hi, Ten years ago today (2005-08-14) my account was created: https://nm.debian.org/public/person/damog Today, I don't feel like Debian represents me and neither do I represent the project anymore. I had tried over the last couple of years to retake my involvement but lack of motivation and time always got on the way, so the right thing to do for me is to officially retire and gtfo. I certainly learned a bunch from dozens of Debian people over these many years, and I'm nothing but grateful with all of them; I will for sure carry the project close to my heart as I carry it with the Debian swirl I still have tattooed on my back ;) http://damog.net/blog/2005/06/29/debian-tattoo/ I have three packages left that have not been updated in forever and you can consider orphaned now: gcolor2, libperl6-say-perl and libxml-treepp-perl. With all best wishes, David Moreno. http://damog.net/ [-- End of signed data --]I received a couple of questions about my decision here. I basically don t feel like Debian represents my interests and neither do I represent the project this doesn t mean I don t believe in free software, to the contrary. I think some of the best software advancements we ve made as society are thanks to it. I don t necessarily believe on how the project has evolved itself, whether that has been the right way, to regain relevancy and dominance, and if it s remained primarily a way to feed dogmatism versus pragmatism. This is the perfect example of a tragic consequence. I was very happy to learn that the current Debian Conference being held in Germany got the highest attendance ever, hopefully that can be utilized in a significant and useful way. Regardless, my contributions to Debian were never noteworthy so it s also not that big of a deal. I just need to close cycles myself and move forward, and the ten year anniversary looked like a significant mark for that. Poke me in case you wanna discuss some more. I ll always be happy to. Specially over beer :) Peace.


Previously: v4.5. The v4.6 Linux kernel release included a bunch of stuff, with much more of it under the KSPP umbrella.
seccomp support for parisc
Helge Deller added seccomp support for parisc, which including plumbing support for PTRACE_GETREGSET to get the self-tests working.
x86 32-bit mmap ASLR vs unlimited stack fixed
Hector Marco-Gisbert removed a long-standing limitation to mmap ASLR on 32-bit x86, where setting an unlimited stack (e.g. ulimit -s unlimited ) would turn off mmap ASLR (which provided a way to bypass ASLR when executing setuid processes). Given that ASLR entropy can now be controlled directly (see the v4.5 post), and that the cases where this created an actual problem are very rare, means that if a system sees collisions between unlimited stack and mmap ASLR, they can just adjust the 32-bit ASLR entropy instead.
x86 execute-only memory
Dave Hansen added Protection Key support for future x86 CPUs and, as part of this, implemented support for execute only memory in user-space. On pkeys-supporting CPUs, using mmap(..., PROT_EXEC) (i.e. without PROT_READ) will mean that the memory can be executed but cannot be read (or written). This provides some mitigation against automated ROP gadget finding where an executable is read out of memory to find places that can be used to build a malicious execution path. Using this will require changing some linker behavior (to avoid putting data in executable areas), but seems to otherwise Just Work. I m looking forward to either emulated QEmu support or access to one of these fancy CPUs.
CONFIG_DEBUG_RODATA enabled by default on arm and arm64, and mandatory on x86
Ard Biesheuvel (arm64) and I (arm) made the poorly-named CONFIG_DEBUG_RODATA enabled by default. This feature controls whether the kernel enforces proper memory protections on its own memory regions (code memory is executable and read-only, read-only data is actually read-only and non-executable, and writable data is non-executable). This protection is a fundamental security primitive for kernel self-protection, so making it on-by-default is required to start any kind of attack surface reduction within the kernel.
On x86 CONFIG_DEBUG_RODATA was already enabled by default, but, at Ingo Molnar s suggestion, I made it mandatory: CONFIG_DEBUG_RODATA cannot be turned off on x86. I expect we ll get there with arm and arm64 too, but the protection is still somewhat new on these architectures, so it s reasonable to continue to leave an out for developers that find themselves tripping over it.
arm64 KASLR text base offset
Ard Biesheuvel reworked a ton of arm64 infrastructure to support kernel relocation and, building on that, Kernel Address Space Layout Randomization of the kernel text base offset (and module base offset). As with x86 text base KASLR, this is a probabilistic defense that raises the bar for kernel attacks where finding the KASLR offset must be added to the chain of exploits used for a successful attack. One big difference from x86 is that the entropy for the KASLR must come either from Device Tree (in the /chosen/kaslr-seed property) or from UEFI (via EFI_RNG_PROTOCOL), so if you re building arm64 devices, make sure you have a strong source of early-boot entropy that you can expose through your boot-firmware or boot-loader.
zero-poison after free
Laura Abbott reworked a bunch of the kernel memory management debugging code to add zeroing of freed memory, similar to PaX/Grsecurity s PAX_MEMORY_SANITIZE feature. This feature means that memory is cleared at free, wiping any sensitive data so it doesn t have an opportunity to leak in various ways (e.g. accidentally uninitialized structures or padding), and that certain types of use-after-free flaws cannot be exploited since the memory has been wiped. To take things even a step further, the poisoning can be verified at allocation time to make sure that nothing wrote to it between free and allocation (called sanity checking ), which can catch another small subset of flaws.
To understand the pieces of this, it s worth describing that the kernel s higher level allocator, the page allocator (e.g. __get_free_pages()) is used by the finer-grained slab allocator (e.g. kmem_cache_alloc(), kmalloc()). Poisoning is handled separately in both allocators. The zero-poisoning happens at the page allocator level. Since the slab allocators tend to do their own allocation/freeing, their poisoning happens separately (since on slab free nothing has been freed up to the page allocator).
Only limited performance tuning has been done, so the penalty is rather high at the moment, at about 9% when doing a kernel build workload. Future work will include some exclusion of frequently-freed caches (similar to PAX_MEMORY_SANITIZE), and making the options entirely CONFIG controlled (right now both CONFIGs are needed to build in the code, and a kernel command line is needed to activate it). Performing the sanity checking (mentioned above) adds another roughly 3% penalty. In the general case (and once the performance of the poisoning is improved), the security value of the sanity checking isn t worth the performance trade-off.
Tests for the features can be found in lkdtm as READ_AFTER_FREE and READ_BUDDY_AFTER_FREE. If you re feeling especially paranoid and have enabled sanity-checking, WRITE_AFTER_FREE and WRITE_BUDDY_AFTER_FREE can test these as well.
To perform zero-poisoning of page allocations and (currently non-zero) poisoning of slab allocations, build with:
CONFIG_DEBUG_PAGEALLOC=nand enable the page allocator poisoning and slab allocator poisoning at boot with this on the kernel command line:
CONFIG_PAGE_POISONING=y
CONFIG_PAGE_POISONING_NO_SANITY=y
CONFIG_PAGE_POISONING_ZERO=y
CONFIG_SLUB_DEBUG=y
page_poison=on slub_debug=PTo add sanity-checking, change
PAGE_POISONING_NO_SANITY=n, and add F to slub_debug as slub_debug=PF .
read-only after init
I added the infrastructure to support making certain kernel memory read-only after kernel initialization (inspired by a small part of PaX/Grsecurity s KERNEXEC functionality). The goal is to continue to reduce the attack surface within the kernel by making even more of the memory, especially function pointer tables, read-only (which depends on CONFIG_DEBUG_RODATA above).
Function pointer tables (and similar structures) are frequently targeted by attackers when redirecting execution. While many are already declared const in the kernel source code, making them read-only (and therefore unavailable to attackers) for their entire lifetime, there is a class of variables that get initialized during kernel (and module) start-up (i.e. written to during functions that are marked __init ) and then never (intentionally) written to again. Some examples are things like the VDSO, vector tables, arch-specific callbacks, etc.
As it turns out, most architectures with kernel memory protection already delay making their data read-only until after __init (see mark_rodata_ro()), so it s trivial to declare a new data section ( .data..ro_after_init ) and add it to the existing read-only data section ( .rodata ). Kernel structures can be annotated with the new section (via the __ro_after_init macro), and they ll become read-only once boot has finished.
The next step for attack surface reduction infrastructure will be to create a kernel memory region that is passively read-only, but can be made temporarily writable (by a single un-preemptable CPU), for storing sensitive structures that are written to only very rarely. Once this is done, much more of the kernel s attack surface can be made read-only for the majority of its lifetime.
As people identify places where __ro_after_init can be used, we can grow the protection. A good place to start is to look through the PaX/Grsecurity patch to find uses of __read_only on variables that are only written to during __init functions. The rest are places that will need the temporarily-writable infrastructure (PaX/Grsecurity uses pax_open_kernel()/pax_close_kernel() for these).
That s it for v4.6, next up will be v4.7!
2016, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Surprising to learn it's already October.
I wanted to write another app using OAUTH but after having learnt how to write an app using OAUTH end-to-end, for web and command-line, it's still tedious.
| Series: | Sarantine Mosaic #1 |
| Publisher: | HarperPrism |
| Copyright: | 1998 |
| Printing: | January 2000 |
| ISBN: | 0-06-105990-0 |
| Format: | Mass market |
| Pages: | 533 |
Here is my monthly update covering what I have been doing in the free software world (previous month):
diffoscope is our "diff on steroids" that will not only recursively unpack archives but will transform binary formats into human-readable forms in order to compare them.
disorderfs is our FUSE filesystem that deliberately introduces nondeterminism into the results of system calls such as readdir(3).
strip-nondeterminism is our tool to remove specific information from a completed build.
I attended the Debian Bug Squashing Party (BSP) in Salzburg, Austria where 70+ "release-critical" bugs were fixed. Sincere thanks to Bernd Zeimetz (bzed) for organising and Conova for sponsoring/hosting. The event was covered by the Salzburg Cityguide.
I attended NixOS user group in London, England.
Niels Thykier granted me commit access to the Lintian Git repository and I added myself to the debian/copyright there.
Filed an ITP for the roughtime secure time synchronisation client and server. This is blocked on packaging the Bazel build system. (#838416)
At the end of August 2012, I announced my
Early Retirement
from HP. Two years later, my friend and former boss Martin Fink
successfully recruited me to return to what later became
Hewlett Packard Enterprise, as an
HPE Fellow
working on open source strategy in his Office of the CTO.
I'm proud of what I was was able to accomplish in the 25 months since then,
but recent efforts to "simplify" HPE actually made things complicated for
me. Between the announcement in late June that
Martin intended to retire himself,
and the two major spin-merger announcements involving
Enterprise Services
and
Software... well...
The bottom line is that today, 30 September 2016, is my last day at HPE.
My plan is to "return to retirement" and work on some fun projects with my wife
now that we are "empty nesters". I do intend to remain involved in the Free
Software and open hardware worlds, but whether that might eventually involve
further employment is something I'm going to try and avoid thinking about
for a while...
There is a rocket launch scheduled nearby this weekend, after all!
A pure maintenance release 0.2.6 of the gcbd package is now on CRAN. The gcbd proposes a benchmarking framework for LAPACK and BLAS operations (as the library can exchanged in a plug-and-play sense on suitable OSs) and records result in local database. Recent / upcoming changes to DBMI and RSQLite let me to update the package; there are no actual functionality changes in this release.
CRANberries also provides a diffstat report for the latest release.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
A new version of the RcppCNPy package arrived on CRAN a few days ago.
RcppCNPy provides R with read and write access to NumPy files thanks to the cnpy library by Carl Rogers.
This new release reflects all the suggestions and comments I received during the review process for the Journal of Open Source Software submission. I am happy to say that about twenty-nine days after I submitted, the paper was accepted and is now published.
Changes in version 0.2.6 (2016-09-25)CRANberries also provides a diffstat report for the latest release. As always, feedback is welcome and the best place to start a discussion may be the GitHub issue tickets page.
- Expanded documentation in README.md
- Added examples to help page
- Added CITATION file for JOSS paper
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Previously: v4.4. Some things I found interesting in the Linux kernel v4.5:
CONFIG_IO_STRICT_DEVMEM
The CONFIG_STRICT_DEVMEM setting that has existed for a long time already protects system RAM from being accessible through the /dev/mem device node to root in user-space. Dan Williams added CONFIG_IO_STRICT_DEVMEM to extend this so that if a kernel driver has reserved a device memory region for use, it will become unavailable to /dev/mem also. The reservation in the kernel was to keep other kernel things from using the memory, so this is just common sense to make sure user-space can t stomp on it either. Everyone should have this enabled. (And if you have a system where you discover you need IO memory access from userspace, you can boot with iomem=relaxed to disable this at runtime.)
If you re looking to create a very bright line between user-space having access to device memory, it s worth noting that if a device driver is a module, a malicious root user can just unload the module (freeing the kernel memory reservation), fiddle with the device memory, and then reload the driver module. So either just leave out /dev/mem entirely (not currently possible with upstream), build a monolithic kernel (no modules), or otherwise block (un)loading of modules (/proc/sys/kernel/modules_disabled).
ptrace fsuid checking
Jann Horn fixed some corner-cases in how ptrace access checks were handled on special files in /proc. For example, prior to this fix, if a setuid process temporarily dropped privileges to perform actions as a regular user, the ptrace checks would not notice the reduced privilege, possibly allowing a regular user to trick a privileged process into disclosing things out of /proc (ASLR offsets, restricted directories, etc) that they normally would be restricted from seeing.
ASLR entropy sysctl
Daniel Cashman standardized the way architectures declare their maximum user-space ASLR entropy (CONFIG_ARCH_MMAP_RND_BITS_MAX) and then created a sysctl (/proc/sys/vm/mmap_rnd_bits) so that system owners could crank up entropy. For example, the default entropy on 32-bit ARM was 8 bits, but the maximum could be as much as 16. If your 64-bit kernel is built with CONFIG_COMPAT, there s a compat version of the sysctl as well, for controlling the ASLR entropy of 32-bit processes: /proc/sys/vm/mmap_rnd_compat_bits.
Here s how to crank your entropy to the max, without regard to what architecture you re on:
for i in "" "compat_"; do f=/proc/sys/vm/mmap_rnd_$ i bits; n=$(cat $f); while echo $n > $f ; do n=$(( n + 1 )); done; donestrict sysctl writes Two years ago I added a sysctl for treating sysctl writes more like regular files (i.e. what s written first is what appears at the start), rather than like a ring-buffer (what s written last is what appears first). At the time it wasn t clear what might break if this was enabled, so a WARN was added to the kernel. Since only one such string showed up in searches over the last two years, the strict writing mode was made the default. The setting remains available as
/proc/sys/kernel/sysctl_writes_strict.
seccomp UM support
Micka l Sala n added seccomp support (and selftests) for user-mode Linux. Moar architectures!
seccomp NNP vs TSYNC fix
Jann Horn noticed and fixed a problem where if a seccomp filter was already in place on a process (after being installed by a privileged process like systemd, a container launcher, etc) then the setting of the no new privs flag could be bypassed when adding filters with the SECCOMP_FILTER_FLAG_TSYNC flag set. Bypassing NNP meant it might be possible to trick a buggy setuid program into doing things as root after a seccomp filter forced a privilege drop to fail (generally referred to as the sendmail setuid flaw ). With NNP set, a setuid program can t be run in the first place.
That s it! Next I ll cover v4.6
Edit: Added notes about iomem=
2016, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
In the perennial search of the meaning of life and the fundamental laws that govern nature, man was always faced for millennia with the mysterious concept of emptiness. What is emptiness? Does it really exist in nature? Is emptiness the non-being, as theorized by Parmenides?
Until the early years of the last century, technology had not yet been able to equip scientists with the necessary tools to investigate the innermost structure of matter, so the concept of emptiness was always faced with insights and metaphors that led, over the centuries, to a broad philosophical debate.
For the ancient atomist Greek philosophers, the existence of emptiness was not only possible but had become a necessity, becoming the ontological principle for the existence of being: for them, actually, the emptiness that permeates the atoms is what allows movement.
<Read More [by Fabio Marzocca]>
A new postdoc student arrived at our department this semester, and
after learning that he uses GNU/Linux for all his computing, I invited
him along to TFUG. During some
of our meetings people asked how could I do X on my GNU/Linux
desktop? and, jokingly, the postdoc would respond the answer to your
question is do you really need to do that? Sometimes the more
experienced GNU/Linux users at the table would respond to questions by
suggesting that the user should simply give up on doing X, and the
postdoc would slap his thigh and laugh and say see? I told you
that s the answer!
The phenomenon here is that people who have at some point made a
commitment to at least try to use GNU/Linux for all their computing
quickly find that they have come to value using GNU/Linux more than
they value engaging in certain activities that only work well/at all
under a proprietary operating system. I think that this is because
they get used to being treated with respect by their computer. And
indeed, one of the reasons I ve almost entirely given up on computer
gaming is that computer games are non-free software. Are you sure
you need to do that? starts sounding like a genuine question rather
than simply a polite way of saying that what someone wants to do can t
be achieved.
I suggest that this is a blessing in disguise. The majority of the
things that you can only do under a proprietary operating system are
things that it would be good for you if you were to switch them out
for other activities. I m not suggesting that switching to a
GNU/Linux is a good way to give up on the entertainment industry.
It s a good way of moderating your engagement with the entertainment
industry. Rather than logging onto Netflix, you might instead pop in
a DVD of a movie. You can still engage with contemporary popular
culture, but the technical barriers give you an opportunity to
moderate your consumption: once you ve finished watching the movie,
the software won t try to get you to watch something else by making a
calculation as to what you re most likely to assent to watching next
based on what you ve watched before. For this behaviour of the
Netflix software is just another example of non-free software working
against its user s interests: watching a movie is good for you, but
binge-watching a TV series probably isn t. In cases like this, living
in the world of Free Software makes it easier to engage with media
healthily.
Diffoscope is a diff utility which recursively unpacks archives, ISOs, etc., transforming a wide variety of files into human-readable forms before comparison instead of simply showing the raw difference in hexadecimal.
I recently added a progress bar when diffoscope is run on a terminal:
Note that as diffoscope can, at any point, encounter an archive or format that requires unpacking, the progress will always be approximate and may even appear to go "backwards".
The implementation, available in version 61, is simple (see #1, #2, #3 & #4) but takes into account of a number of subtleties by using context managers to correctly track the state throughout.
Previously: v4.3. Continuing with interesting security things in the Linux kernel, here s v4.4. As before, if you think there s stuff I missed that should get some attention, please let me know.
seccomp Checkpoint/Restore-In-Userspace
Tycho Andersen added a way to extract and restore seccomp filters from running processes via PTRACE_SECCOMP_GET_FILTER under CONFIG_CHECKPOINT_RESTORE. This is a continuation of his work (that I failed to mention in my prior post) from v4.3, which introduced a way to suspend and resume seccomp filters. As I mentioned at the time (and for which he continues to quote me) this feature gives me the creeps. :)
x86 W^X detection
Stephen Smalley noticed that there was still a range of kernel memory (just past the end of the kernel code itself) that was incorrectly marked writable and executable, defeating the point of CONFIG_DEBUG_RODATA which seeks to eliminate these kinds of memory ranges. He corrected this in v4.3 and added CONFIG_DEBUG_WX in v4.4 which performs a scan of memory at boot time and yells loudly if unexpected memory protection are found. To nobody s delight, it was shortly discovered the UEFI leaves chunks of memory in this state too, which posed an ugly-to-solve problem (which Matt Fleming addressed in v4.6).
x86_64 vsyscall CONFIG
I introduced a way to control the mode of the x86_64 vsyscall with a build-time CONFIG selection, though the choice I really care about is CONFIG_LEGACY_VSYSCALL_NONE, to force the vsyscall memory region off by default. The vsyscall memory region was always mapped into process memory at a fixed location, and it originally posed a security risk as a ROP gadget execution target. The vsyscall emulation mode was added to mitigate the problem, but it still left fixed-position static memory content in all processes, which could still pose a security risk. The good news is that glibc since version 2.15 doesn t need vsyscall at all, so it can just be removed entirely. Any kernel built this way that discovered they needed to support a pre-2.15 glibc could still re-enable it at the kernel command line with vsyscall=emulate .
That s it for v4.4. Tune in tomorrow for v4.5!
2016, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Next.