Search Results: "edd"

1 October 2016

Kees Cook: security things in Linux v4.6

Previously: v4.5. The v4.6 Linux kernel release included a bunch of stuff, with much more of it under the KSPP umbrella. seccomp support for parisc Helge Deller added seccomp support for parisc, which including plumbing support for PTRACE_GETREGSET to get the self-tests working. x86 32-bit mmap ASLR vs unlimited stack fixed Hector Marco-Gisbert removed a long-standing limitation to mmap ASLR on 32-bit x86, where setting an unlimited stack (e.g. ulimit -s unlimited ) would turn off mmap ASLR (which provided a way to bypass ASLR when executing setuid processes). Given that ASLR entropy can now be controlled directly (see the v4.5 post), and that the cases where this created an actual problem are very rare, means that if a system sees collisions between unlimited stack and mmap ASLR, they can just adjust the 32-bit ASLR entropy instead. x86 execute-only memory Dave Hansen added Protection Key support for future x86 CPUs and, as part of this, implemented support for execute only memory in user-space. On pkeys-supporting CPUs, using mmap(..., PROT_EXEC) (i.e. without PROT_READ) will mean that the memory can be executed but cannot be read (or written). This provides some mitigation against automated ROP gadget finding where an executable is read out of memory to find places that can be used to build a malicious execution path. Using this will require changing some linker behavior (to avoid putting data in executable areas), but seems to otherwise Just Work. I m looking forward to either emulated QEmu support or access to one of these fancy CPUs. CONFIG_DEBUG_RODATA enabled by default on arm and arm64, and mandatory on x86 Ard Biesheuvel (arm64) and I (arm) made the poorly-named CONFIG_DEBUG_RODATA enabled by default. This feature controls whether the kernel enforces proper memory protections on its own memory regions (code memory is executable and read-only, read-only data is actually read-only and non-executable, and writable data is non-executable). This protection is a fundamental security primitive for kernel self-protection, so making it on-by-default is required to start any kind of attack surface reduction within the kernel. On x86 CONFIG_DEBUG_RODATA was already enabled by default, but, at Ingo Molnar s suggestion, I made it mandatory: CONFIG_DEBUG_RODATA cannot be turned off on x86. I expect we ll get there with arm and arm64 too, but the protection is still somewhat new on these architectures, so it s reasonable to continue to leave an out for developers that find themselves tripping over it. arm64 KASLR text base offset Ard Biesheuvel reworked a ton of arm64 infrastructure to support kernel relocation and, building on that, Kernel Address Space Layout Randomization of the kernel text base offset (and module base offset). As with x86 text base KASLR, this is a probabilistic defense that raises the bar for kernel attacks where finding the KASLR offset must be added to the chain of exploits used for a successful attack. One big difference from x86 is that the entropy for the KASLR must come either from Device Tree (in the /chosen/kaslr-seed property) or from UEFI (via EFI_RNG_PROTOCOL), so if you re building arm64 devices, make sure you have a strong source of early-boot entropy that you can expose through your boot-firmware or boot-loader. zero-poison after free Laura Abbott reworked a bunch of the kernel memory management debugging code to add zeroing of freed memory, similar to PaX/Grsecurity s PAX_MEMORY_SANITIZE feature. This feature means that memory is cleared at free, wiping any sensitive data so it doesn t have an opportunity to leak in various ways (e.g. accidentally uninitialized structures or padding), and that certain types of use-after-free flaws cannot be exploited since the memory has been wiped. To take things even a step further, the poisoning can be verified at allocation time to make sure that nothing wrote to it between free and allocation (called sanity checking ), which can catch another small subset of flaws. To understand the pieces of this, it s worth describing that the kernel s higher level allocator, the page allocator (e.g. __get_free_pages()) is used by the finer-grained slab allocator (e.g. kmem_cache_alloc(), kmalloc()). Poisoning is handled separately in both allocators. The zero-poisoning happens at the page allocator level. Since the slab allocators tend to do their own allocation/freeing, their poisoning happens separately (since on slab free nothing has been freed up to the page allocator). Only limited performance tuning has been done, so the penalty is rather high at the moment, at about 9% when doing a kernel build workload. Future work will include some exclusion of frequently-freed caches (similar to PAX_MEMORY_SANITIZE), and making the options entirely CONFIG controlled (right now both CONFIGs are needed to build in the code, and a kernel command line is needed to activate it). Performing the sanity checking (mentioned above) adds another roughly 3% penalty. In the general case (and once the performance of the poisoning is improved), the security value of the sanity checking isn t worth the performance trade-off. Tests for the features can be found in lkdtm as READ_AFTER_FREE and READ_BUDDY_AFTER_FREE. If you re feeling especially paranoid and have enabled sanity-checking, WRITE_AFTER_FREE and WRITE_BUDDY_AFTER_FREE can test these as well. To perform zero-poisoning of page allocations and (currently non-zero) poisoning of slab allocations, build with:
CONFIG_DEBUG_PAGEALLOC=n
CONFIG_PAGE_POISONING=y
CONFIG_PAGE_POISONING_NO_SANITY=y
CONFIG_PAGE_POISONING_ZERO=y
CONFIG_SLUB_DEBUG=y
and enable the page allocator poisoning and slab allocator poisoning at boot with this on the kernel command line:
page_poison=on slub_debug=P
To add sanity-checking, change PAGE_POISONING_NO_SANITY=n, and add F to slub_debug as slub_debug=PF . read-only after init I added the infrastructure to support making certain kernel memory read-only after kernel initialization (inspired by a small part of PaX/Grsecurity s KERNEXEC functionality). The goal is to continue to reduce the attack surface within the kernel by making even more of the memory, especially function pointer tables, read-only (which depends on CONFIG_DEBUG_RODATA above). Function pointer tables (and similar structures) are frequently targeted by attackers when redirecting execution. While many are already declared const in the kernel source code, making them read-only (and therefore unavailable to attackers) for their entire lifetime, there is a class of variables that get initialized during kernel (and module) start-up (i.e. written to during functions that are marked __init ) and then never (intentionally) written to again. Some examples are things like the VDSO, vector tables, arch-specific callbacks, etc. As it turns out, most architectures with kernel memory protection already delay making their data read-only until after __init (see mark_rodata_ro()), so it s trivial to declare a new data section ( .data..ro_after_init ) and add it to the existing read-only data section ( .rodata ). Kernel structures can be annotated with the new section (via the __ro_after_init macro), and they ll become read-only once boot has finished. The next step for attack surface reduction infrastructure will be to create a kernel memory region that is passively read-only, but can be made temporarily writable (by a single un-preemptable CPU), for storing sensitive structures that are written to only very rarely. Once this is done, much more of the kernel s attack surface can be made read-only for the majority of its lifetime. As people identify places where __ro_after_init can be used, we can grow the protection. A good place to start is to look through the PaX/Grsecurity patch to find uses of __read_only on variables that are only written to during __init functions. The rest are places that will need the temporarily-writable infrastructure (PaX/Grsecurity uses pax_open_kernel()/pax_close_kernel() for these). That s it for v4.6, next up will be v4.7!

2016, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

29 September 2016

Dirk Eddelbuettel: gcbd 0.2.6

A pure maintenance release 0.2.6 of the gcbd package is now on CRAN. The gcbd proposes a benchmarking framework for LAPACK and BLAS operations (as the library can exchanged in a plug-and-play sense on suitable OSs) and records result in local database. Recent / upcoming changes to DBMI and RSQLite let me to update the package; there are no actual functionality changes in this release. CRANberries also provides a diffstat report for the latest release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dirk Eddelbuettel: RcppCNPy 0.2.6

A new version of the RcppCNPy package arrived on CRAN a few days ago. RcppCNPy provides R with read and write access to NumPy files thanks to the cnpy library by Carl Rogers. This new release reflects all the suggestions and comments I received during the review process for the Journal of Open Source Software submission. I am happy to say that about twenty-nine days after I submitted, the paper was accepted and is now published.
Changes in version 0.2.6 (2016-09-25)
  • Expanded documentation in README.md
  • Added examples to help page
  • Added CITATION file for JOSS paper
CRANberries also provides a diffstat report for the latest release. As always, feedback is welcome and the best place to start a discussion may be the GitHub issue tickets page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

24 September 2016

Dirk Eddelbuettel: tint 0.0.1: Tint Is Not Tufte

A new experimental package is now on the ghrr drat. It is named tint which stands for Tint Is Not Tufte. It provides an alternative for Tufte-style html presentation. I wrote a bit more on the package page and the README in the repo -- so go read this. Here is just a little teaser of what it looks like:
and the full underlying document is available too. For questions or comments use the issue tracker off the GitHub repo. The package may be short-lived as its functionality may end up inside the tufte package.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

16 September 2016

Dirk Eddelbuettel: anytime 0.0.2: Added functionality

anytime arrived on CRAN via release 0.0.1 a good two days ago. anytime aims to convert anything in integer, numeric, character, factor, ordered, ... format to POSIXct (or Date) objects. This new release 0.0.2 adds two new functions to gather conversion formats -- and set new ones. It also fixed a minor build bug, and robustifies a conversion which was seen to be not quite right under some time zones. The NEWS file summarises the release:
Changes in Rcpp version 0.0.2 (2016-09-15)
  • Refactored to use a simple class wrapped around two vector with (string) formats and locales; this allow for adding formats; also adds accessor for formats (#4, closes #1 and #3).
  • New function addFormats() and getFormats().
  • Relaxed one tests which showed problems on some platforms.
  • Added as.POSIXlt() step to anydate() ensuring all POSIXlt components are set (#6 fixing #5).
Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

13 September 2016

Dirk Eddelbuettel: anytime 0.0.1: New package for 'anything' to POSIXct (or Date)

anytime just arrived on CRAN as a very first release 0.0.1. So why (yet another) package dealing with dates and times? R excels at computing with dates, and times. By using typed representation we not only get all that functionality but also of the added safety stemming from proper representation. But there is a small nuisance cost: How often have we each told as.POSIXct() that the origin is epoch '1970-01-01'? Do we have to say it a million more times? Similarly, when parsing dates that are in some recogniseable form of the YYYYMMDD format, do we really have to manually convert from integer or numeric or factor or ordered to character first? Having one of several common separators and/or date / time month forms (YYYY-MM-DD, YYYY/MM/DD, YYYYMMDD, YYYY-mon-DD and so on, with or without times, with or without textual months and so on), do we really need a format string? anytime() aims to help as a small general purpose converter returning a proper POSIXct (or Date) object nomatter the input (provided it was somewhat parseable), relying on Boost date_time for the (efficient, performant) conversion. See some examples on the anytime page or the GitHub README.md, or in the screenshot below. And then just give it try! anytime examples For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

12 September 2016

Shirish Agarwal: mtpfs, feh and not being able to share the debconf experience.

I have been sick for about 2 weeks now hence haven t written. I had joint pains and still am weak. There has been lot of reports of malaria, chikungunya and dengue fever around the city. The only thing I came to know is how lucky I am to be able to move around on 2 legs and how powerless and debilitating it feels when you can t move. In the interim I saw Me Before You and after going through my most miniscule experience, I could relate with Will Taylor s character. If I was in his place, I would probably make the same choices. But my issues are and were slightly different. Last month I was supposed to share my debconf experience in the local PLUG meet. For that purpose, I took some pictures from my phone on a pen-drive to share. But when reached the venue, found out that I had forgotten to take the pen-drive. What I had also done is used the mogrify command from the imagemagick stable to lossy compress the images on the pen-drive so it is easier on image viewers. But that was not to be and at the last moment had to use my phone plugged into the USB drive of the lappy and show some pictures. This was not good. I had known that it was mounted somewhere but hadn t looked at where. After coming back home, it took me hardly 10 minutes to find out where it was mounted. It is not mounted under /media/shirish but under /run/user/1000/gvfs . If I do list under it shows mtp:host=%5Busb%3A005%2C007%5D . I didn t need any packages under debian to make it work. Interestingly, the only image viewer which seems to be able to work with all the images is feh which is a command-line image viewer in Debian. [$] aptitude show feh
Package: feh
Version: 2.16.2-1
State: installed
Automatically installed: no
Priority: optional
Section: graphics
Maintainer: Debian PhotoTools Maintainers
Architecture: amd64
Uncompressed Size: 391 k
Depends: libc6 (>= 2.15), libcurl3 (>= 7.16.2), libexif12 (>= 0.6.21-1~), libimlib2 (>= 1.4.5), libpng16-16 (>= 1.6.2-1), libx11-6, libxinerama1
Recommends: libjpeg-progs
Description: imlib2 based image viewer
feh is a fast, lightweight image viewer which uses imlib2. It is commandline-driven and supports multiple images through slideshows, thumbnail
browsing or multiple windows, and montages or index prints (using TrueType fonts to display file info). Advanced features include fast dynamic
zooming, progressive loading, loading via HTTP (with reload support for watching webcams), recursive file opening (slideshow of a directory
hierarchy), and mouse wheel/keyboard control.
Homepage: http://feh.finalrewind.org/
I did try various things to get it to mount under /media/shirish/ but as of date have no luck. Am running Android 6.0 Marshmallow and have enabled USB debugging with help from my friend Akshat . I even changed the /etc/fuse.conf options but even that didn t work. #cat /etc/fuse.conf
[sudo] password for shirish:
# /etc/fuse.conf - Configuration file for Filesystem in Userspace (FUSE)
# Set the maximum number of FUSE mounts allowed to non-root users.
# The default is 1000.
mount_max = 1
# Allow non-root users to specify the allow_other or allow_root mount options.
user_allow_other
One way which I haven t explored is adding/making an entry into /etc/fstab. If anybody knows of a solution which doesn t involve changing content of /etc/fstab. At the same time you are able to get the card and phone directories mounted under /media// , in my case /media/shirish would be interested to know. I would like the /etc/fstab to remain as it is. I am using Samsung J5 (unrooted) Btw I tried all the mtpfs packages in Debian testing but without any meaningful change Look forward to tips.
Filed under: Miscellenous Tagged: #Android, #Debconf16, #debian, #mptfs, feh, FUSE, PLUG

10 September 2016

Dirk Eddelbuettel: New package gettz on CRAN

gettz is now on CRAN in its initial release 0.0.1. It provides a possible fallback in situations where Sys.timezone() fails to determine the system timezone. That can happen when e.g. the file /etc/localtime somehow is not a link into the corresponding file with zoneinfo data in, say, /usr/share/zoneinfo. Duane McCully provided a nice StackOverflow answer with code that offers fallbacks via /etc/timezone (on Debian/Ubuntu) or /etc/sysconfig/clock (on RedHat/CentOS/Fedora, and rumour has it, BSD* systems) or /etc/TIMEZONE (on Solaris). The gettz micro-package essentially encodes that approach so that we have an optional fallback when Sys.timezone() comes up empty. In the previous paragraph, note the stark absense of OS X where there seems nothing to query, and of course Windows. Contributions for either would be welcome. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

9 September 2016

Dirk Eddelbuettel: RProtoBuf 0.4.6: bugfix update

Relatively quickly after version 0.4.5 of RProtoBuf was released, we have a new version 0.4.6 to announce which appeared on CRAN today. RProtoBuf provides R bindings for the Google Protocol Buffers ("Protobuf") data encoding and serialization library used and released by Google, and deployed as a language and operating-system agnostic protocol by numerous projects. This version contains a contributed bug-fix pull request covering conversion of zero-length vectors, and adding native support for S4 objects. At the request / suggestion of the CRAN maintainers, it also uncomments a LaTeX macro in the vignette (corresponding to our recent JSS paper paper) which older R versions do not (yet) have in their jss.cls file.
Changes in RProtoBuf version 0.4.6 (2016-09-08)
  • Support for serializing zero-length objects was added (PR #18 addressing #13)
  • S4 objects are natively encoded (also PR #18)
  • The vignette based on the JSS paper no longer uses a macro available only with the R-devel version of jss.cls, and hence builds on all R versions
CRANberries also provides a diff to the previous release. The RProtoBuf page has an older package vignette, a 'quick' overview vignette, a unit test summary vignette, and the pre-print for the JSS paper. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

6 September 2016

Andrew Shadura: Manual control of OpenEmbedded -dbg packages

In December last year, OpenEmbebbed introduced automatic debug packages. Prior to that, you d need to manually construct FILES_$ PN -dbg variable in your recipe. If you need to retain manual control over precisely what does into debug packages, set an undocumented NOAUTOPACKAGEDEBUG variable to 1, the same way Qt recipe does:
NOAUTOPACKAGEDEBUG = "1"
FILES_$ PN -dev = "$ includedir /$ QT_DIR_NAME /Qt/*"
FILES_$ PN -dbg = "/usr/src/debug/"
FILES_$ QT_BASE_NAME -demos-doc = "$ docdir /$ QT_DIR_NAME /qch/qt.qch"
P.S. Knowing this would have saved me and my colleagues days of work.

5 September 2016

Raphaël Hertzog: My Free Software Activities in August 2016

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. This months is rather light since I was away in vacation for two weeks. Kali related work The new pkg-security team is working full steam and I reviewed/sponsored many packages during the month: polenum, accheck, braa, t50, ncrack, websploit. I filed bug #834515 against sbuild since sbuild-createchroot was no longer usable for kali-rolling due to the embedded dash. That misfeature has been reverted and implemented through an explicit option. I brought the attention of ftpmasters on #832163 since we had unexpected packages in the standard section (they have been discovered in the Kali live ISO while we did not want them). I uploaded two fontconfig NMU to finally push to Debian a somewhat cleaner fix for the problem of various captions being displayed as squares after a font upgrade (see #828037 and #835142). I tested (twice) a live-build patch from Adrian Gibanel Lopez implementing EFI boot with grub and merged it into the official git repository (see #731709). I filed bug #835983 on python-pypdf2 since it has an invalid dependency forbidding co-installation with python-pypdf. I orphaned splint since its maintainer was missing in action (MIA) and immediately made a QA upload to fix the RC bug which kicked it out of testing (this package is a build dependency of a Kali package). django-jsonfield I wrote a patch to make python-django-jsonfield compatible with Django 1.10 (#828668) and I committed that patch in the upstream repository. Distro Tracker I made some changes to make the codebase compatible with Django 1.10 (and added Django 1.10 to the tox test matrix). I added a Debian Maintainer Dashboard link next to people s name on request of Lucas Nussbaum (#830548). I made a preliminary review of Paul Wise s patch to add multiarch hints (#833623) and improved the handling of the mailbot when it gets MIME Headers referencing an unknown charset (like cp-850 , Python only knows of cp850 ) I also helped Peter Palfrader to enabled a .onion address for tracker.debian.org, see onion.debian.org for the full list of services available over Tor. Misc stuff I updated my letsencrypt.sh salt formula to work with the latest version of letsencrypt.sh (0.2.0) I merged updated translations for the Debian Administrator s Handbook from weblate.org and uploaded a new version to Debian. Thanks See you next month for a new summary of my activities.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Dirk Eddelbuettel: Rcpp 0.12.7: More updates

The seventh update in the 0.12.* series of Rcpp just arrived on the CRAN network for GNU R as well as in Debian. This 0.12.7 release follows the 0.12.0 release from late July, the 0.12.1 release in September, the 0.12.2 release in November, the 0.12.3 release in January, the 0.12.4 release in March, the 0.12.5 release in May, and the 0.12.6 release in July --- making it the eleventh release at the steady bi-montly release frequency. Keeping with the established pattern, this is again more of a maintenance release which addresses small bugs, nuisances or documentation issues without adding any major new features. One issue that got to a few people was our casual use of NORET in the definition of Rcpp::stop(). We had (ahem) overlooked that NORET is only defined by R 3.2.0 or later, and several folks trying to build on older releases of R (why?) got bitten. Well, at least we have a new record for most frequently reported bug ... Kidding aside, this is now fixed. Rcpp has become the most popular way of enhancing GNU R with C or C++ code. As of today, 759 packages on CRAN depend on Rcpp for making analytical code go faster and further. That is up by well over fifty packages since the last release in mid-July! We are once again fortunate to have a number of pull request, from first-timers to regulars. James "coatless" Balamuta in particular relentlessly pushed for better documentation and cleanup of numerous dangling issue tickets. Artem Klevtsov also contributed again. Qiang, Kevin and I also got some changes in for the Rcpp Core team. More details are again below.
Changes in Rcpp version 0.12.7 (2016-09-04)
  • Changes in Rcpp API:
    • The NORET macro is now defined if it was not already defined by R itself (Kevin fixing issue #512).
    • Environment functions get() & find() now accept a Symbol (James Balamuta in #513 addressing issue #326).
    • Several uses of Rf_eval were replaced by the preferred Rcpp::Rcpp_eval (Qiang in PR #523 closing #498).
    • Improved Autogeneration Warning for RcppExports (James Balamuta in #528 addressing issue #526).
    • Fixed invalid C++ prefix identifiers in auto-generated code (James Balamuta in #528 and #531 addressing issue #387; Simon Dirmeier in #548).
    • String constructors now set default UTF-8 encoding (Qiang Kou in #529 fixing #263).
    • Add variadic variants of the RCPP_RETURN_VECTOR and RCPP_RETURN_MATRIX macro when C++11 compiler used (Artem Klevtsov in #537 fixing #38).
  • Changes in Rcpp build system
    • Travis CI is now driven via run.sh from our fork, and deploys all packages as .deb binaries using our PPA where needed (Dirk in #540 addressing issue #517).
  • Changes in Rcpp unit tests
    • New unit tests for random number generators the R namespace which call the standalone Rmath library. (James Balamuta in #514 addressing issue #28).
  • Changes in Rcpp Examples:
    • Examples that used cxxfunction() from the inline package have been rewritten to use either sourceCpp() or cppFunction() (James Balamuta in #541, #535, #534, and #532 addressing issue #56).
Thanks to CRANberries, you can also look at a diff to the previous release. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

4 September 2016

Steinar H. Gunderson: Multitrack audio in Nageru 1.4.0

Even though the Chess Olympiad takes some attention right now, development on Nageru (my live video mixer) has continued steadily throughout since the 1.3.0 release. I wanted to take a little time to talk about the upcoming 1.4.0 release, and why things are as they are; writing down things often make them a bit clearer. Every major release of Nageru has had a specific primary focus: 1.0.0 was about just getting everything to work, 1.1.0 was about NVIDIA support for more oomph, 1.2.0 was about stabilization and polish (and added support for Blackmagic's PCI cards as a nice little bonus), and 1.3.0 was about x264 integration. For 1.4.0, I wanted to work on multitrack audio and mixing. Audio has always been a clear focus for Nageru, and for good reason; video is 90% about audio, and it's sorely neglected in most amateur productions (not to mention that processing tools are nearly non-existant in most free or cheap software video mixers). Right from the get-go, it's had a chain with proper leveling, compressors and most importantly visual monitoring, so that you know when things are not as they should be. However, it was also written with an assumption that there would be a single audio input source (one of the cameras), and that's going to change. Single-input is less of a showstopper than one'd think at first; you can work around it by buying a mixer, plug everything into that and then feed that signal into the computer. However, there are a few downsides: If you want camera audio, you'll need to pull more cable from each camera (or have SDI de-embedders). Your mixer is likely to require an extra power supply, and that means yet more cable (any decent USB video card can power over USB, so why shouldn't your audio?). You'll need to buy and transport yet another device. And so on. (If you already have a PA mixer, of course you can use it, but just reusing the PA mix as a stream mix rarely gives the best results, and mixing on an aux bus gives very little flexibility.) So for 1.4.0, I wanted something to get essentially the processing equivalent of a mid-range mixer. But even though my education is in DSP, my experience with mixers is rather limited, so I did the only reasonable thing and went over to a friend who's also an (award-winning) audio engineer. (It turns out that everything on a mixer is the way it is for a pretty good reason, tuned through 50+ years of collective audio experience. If you just try to make up something on your own without understanding what's going on, you have a 0.001% chance of stumbling upon some genius new way of doing things by accident, and a significantly larger chance than that of messing things up.) After some back and forth, we figured out a reasonable set of basic tools that would be useful in the right hands, and not too confusing for a beginner. So let's have a look at the new controls you get: Nageru expanded audio control view There's one set of these controls for each bus. (This is the expanded view; there's also a compact view that has only the meters and the fader, which is what you'll typically want to use during the run itself the expanded view is for setup and tweaking.) A bus in Nageru is a pair of channels (left/right), sourced from a video capture or ALSA card. The channel mapping is flexible; my USB sound card has 18 channels, for instance, and you can use that to make several buses. Each bus has a name (here I named it very creatively Main , but in a real setting you might want something like Blue microphone or Speaker PC ), which is just for convenience; it doesn't mean much. The most important parts of the mix are given the most screen estate, so even though the way through the signal chain is left-to-right top-to-bottom, I'll go over it in the opposite direction. By far the most important part is the audio level, so the fader naturally is very prominent. (Note that the scale is nonlinear; you want more resolution in the most important area.) Changing a fader with the mouse or keyboard is possible, and probably most people will be doing that, but Nageru will also support USB faders. These usually speak MIDI, for historical reasons, and there are some UI challenges when they're all so different, but you can get really small ones if you want to get that tactile feel without blowing up your budget or getting a bigger backpack. Then there's the meter to the left of that. Nageru already has R128 level meters in the mastering section (not shown here, but generally unchanged from 1.3.0), and those are kept as-is, but for each bus, you don't want to know loudness; you want to know recording levels, so you want a peak meter, not a loudness meter. In particular, you don't want the bus to send clipped data to the master (which would happen if you set it too high); Nageru can handle this situation pretty well (unlike most digital mixers, it mixes in full 32-bit floating-point so there's no internal clipping, and there's a limiter on the master by default), but it's still not a good place to be in, so you can see that being marked in red in this example. The meter doubles as an input peak check during setup; if you turn off all the effects and set the fader to neutral, you can see if the input hits peak or not, and then adjust it down. (Also, you can see here that I only have audio in the left channel; I'd better check my connections, or perhaps just use mono, by setting the right channel on the bus mapping to the same input as the left one.) The compressor (now moved from the mastering section to each bus) should be well-known for those using 1.3.0, but in this view, it also has a reduction meter, so that you can see whether it kicks in or not. Most casual users would want to just leave the gain staging and compressor settings alone, but a skilled audio engineer will know how to adjust these to each speaker's antics (some speak at a pretty even volume and thus can get a bit of headroom, while some are much more variable and need tighter settings). Finally (or, well, first), there's the EQ section. The lo-cut is again well-known from 1.3.0 (the cutoff frequency is the same across all buses), but there's now also a simple three-band EQ per bus. Simply ask the speaker to talk normally for a bit, and tweak the controls until it sounds good. People have different voices and different ways of holding the microphone, and if you have a reasonable ear, you can use the EQ to your advantage to make them sound a little more even on the stream. Either that, or just put it in neutral, and the entire EQ code will be bypassed. The code is making pretty good progress; all the DSP stuff is done (save for some optimizations I want to do in zita-resampler, now that the discussion flow is started again), and in theory, one could use it already as-is. However, there's a fair amount of gnarly support code that still needs to be written: In particular, I need to do some refactoring to support ALSA hotplug (you don't want your entire stream to go down just because an USB soundcard dropped out for a split second), and similarly some serialization of saving/loading bus mappings. It's not exactly rocket science, but all the code still needs to be written, and there are a number of corner cases to think of. If you want to peek, the code is in the multichannel_audio branch, but beware; I rebase/squash it pretty frequently, so if you pull from it, expect frequent git breakage.

2 September 2016

Steve McIntyre: And about time, too!

A couple of weekends back, we had an awesome time at Jonathan and Charlene's wedding. I'd have blogged sooner, but I had to wait for this photo of the happy couple with the Debian gang... :-) Careful! Unfortunately, somebody let Charlene hide at the back! Follow the link for more photos...

30 August 2016

Joachim Breitner: Explicit vertical alignment in Haskell

Chris Done s automatic Haskell formatter hindent is released in a new version, and getting quite a bit of deserved attention. He is polling the Haskell programmers on whether two or four spaces are the right indentation. But that is just cosmetics I am in principle very much in favor of automatic formatting, and I hope that a tool like hindent will eventually be better at formatting code than a human. But it currently is not there yet. Code is literature meant to be read, and good code goes at length to be easily readable, and formatting can carry semantic information. The Haskell syntax was (at least I get that impression) designed to allow the authors to write nicely looking, easy to understand code. One important tool here is vertical alignment of corresponding concepts on different lines. Compare
maze :: Integer -> Integer -> Integer
maze x y
  abs x > 4    abs y > 4  = 0
  abs x == 4   abs y == 4 = 1
  x ==  2    && y <= 0     = 1
  x ==  3    && y <= 0     = 3
  x >= -2    && y == 0     = 4
  otherwise                = 2
with
maze :: Integer -> Integer -> Integer
maze x y
  abs x > 4   abs y > 4 = 0
  abs x == 4   abs y == 4 = 1
  x == 2 && y <= 0 = 1
  x == 3 && y <= 0 = 3
  x >= -2 && y == 0 = 4
  otherwise = 2
The former is a quick to grasp specification, the latter (the output of hindent at the moment) is a desert of numbers and operators. I see two ways forward: What could such ways be? (This post is cross-posted on reddit.) Update (2016-09-05) Shortly after this post, the Haskell formatter brittany gets released, which supports vertial alignment. Yay!

Dirk Eddelbuettel: RProtoBuf 0.4.5: now with protobuf v2 and v3!

A few short weeks after the 0.4.4 release of RProtoBuf, we are happy to announce a new version 0.4.5 which appeared on CRAN earlier today. RProtoBuf provides R bindings for the Google Protocol Buffers ("Protobuf") data encoding library used and released by Google, and deployed as a language and operating-system agnostic protocol by numerous projects. This release brings support to the recently-release 'version 3' Protocol Buffers standard, used e.g. by the (very exciting) gRPC project (which was just released as version 1.0). RProtoBuf continues to supportv 'version 2' but now also cleanly support 'version 3'.
Changes in RProtoBuf version 0.4.5 (2016-08-29)
  • Support for version 3 of the Protcol Buffers API
  • Added 'syntax = "proto2";' to all proto files (PR #17)
  • Updated Travis CI script to test against both versions 2 and 3 using custom-built .deb packages of version 3 (PR #16)
  • Improved build system with support for custom CXXFLAGS (Craig Radcliffe in PR #15)
CRANberries also provides a diff to the previous release. The RProtoBuf page has an older package vignette, a 'quick' overview vignette, a unit test summary vignette, and the pre-print for the JSS paper. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

29 August 2016

Russell Coker: Monitoring of Monitoring

I was recently asked to get data from a computer that controlled security cameras after a crime had been committed. Due to the potential issues I refused to collect the computer and insisted on performing the work at the office of the company in question. Hard drives are vulnerable to damage from vibration and there is always a risk involved in moving hard drives or systems containing them. A hard drive with evidence of a crime provides additional potential complications. So I wanted to stay within view of the man who commissioned the work just so there could be no misunderstanding. The system had a single IDE disk. The fact that it had an IDE disk is an indication of the age of the system. One of the benefits of SATA over IDE is that swapping disks is much easier, SATA is designed for hot-swap and even systems that don t support hot-swap will have less risk of mechanical damage when changing disks if SATA is used instead of IDE. For an appliance type system where a disk might be expected to be changed by someone who s not a sysadmin SATA provides more benefits over IDE than for some other use cases. I connected the IDE disk to a USB-IDE device so I could read it from my laptop. But the disk just made repeated buzzing sounds while failing to spin up. This is an indication that the drive was probably experiencing stiction which is where the heads stick to the platters and the drive motor isn t strong enough to pull them off. In some cases hitting a drive will get it working again, but I m certainly not going to hit a drive that might be subject to legal action! I recommended referring the drive to a data recovery company. The probability of getting useful data from the disk in question seems very low. It could be that the drive had stiction for months or years. If the drive is recovered it might turn out to have data from years ago and not the recent data that is desired. It is possible that the drive only got stiction after being turned off, but I ll probably never know. Doing it Properly Ever since RAID was introduced there was never an excuse for having a single disk on it s own with important data. Linux Software RAID didn t support online rebuild when 10G was a large disk. But since the late 90 s it has worked well and there s no reason not to use it. The probability of a single IDE disk surviving long enough on it s own to capture useful security data is not particularly good. Even with 2 disks in a RAID-1 configuration there is a chance of data loss. Many years ago I ran a server at my parents house with 2 disks in a RAID-1 and both disks had errors on one hot summer. I wrote a program that s like ddrescue but which would read from the second disk if the first gave a read error and ended up not losing any important data AFAIK. BTRFS has some potential benefits for recovering from such situations but I don t recommend deploying BTRFS in embedded systems any time soon. Monitoring is a requirement for reliable operation. For desktop systems you can get by without specific monitoring, but that is because you are effectively relying on the user monitoring it themself. Since I started using mon (which is very easy to setup) I ve had it notify me of some problems with my laptop that I wouldn t have otherwise noticed. I think that ideally for desktop systems you should have monitoring of disk space, temperature, and certain critical daemons that need to be running but which the user wouldn t immediately notice if they crashed (such as cron and syslogd). There are some companies that provide 3G SIMs for embedded/IoT applications with rates that are significantly cheaper than any of the usual phone/tablet plans if you use small amounts of data or SMS. For a reliable CCTV system the best thing to do would be to have a monitoring contract and have the monitoring system trigger an event if there s a problem with the hard drive etc and also if the system fails to send a I m OK message for a certain period of time. I don t know if people are selling CCTV systems without monitoring to compete on price or if companies are cancelling monitoring contracts to save money. But whichever is happening it s significantly reducing the value derived from monitoring.

28 August 2016

Dirk Eddelbuettel: rfoaas 1.1.0

rfoaas greed example FOAAS upstream came out with release 1.1.0 earlier last week. Tyler Hunt was kind enough to provide an almost immediate pull request adding support for the extended capabilties. Following yesterday's upload we now have version 1.1.0 of rfoaas on CRAN. It brings six more accessors: maybe(), blackadder(), horse(), deraadt(), problem(), cocksplat() and too(). As usual, CRANberries provides a diff to the previous CRAN release. Questions, comments etc should go to the GitHub issue tracker. More background information is on the project page as well as on the github repo

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

27 August 2016

Dirk Eddelbuettel: RcppCNPy 0.2.5

A maintenance release of the RcppCNPy package is now on CRAN. RcppCNPy provides R with read and write access to NumPy files thanks to the cnpy library by Carl Rogers. This release just updates a few package internals such as the vignette, the DESCRIPTION and the README.md file as well as under Travis script used for continuous integration.
Changes in version 0.2.5 (2016-08-26)
  • Synchronized code with the cnpy repository
  • Updated vignette
  • Expanded DESCRIPTION
CRANberries also provides a diffstat report for the latest release. As always, feedback is welcome and the best place to start a discussion may be the GitHub issue tickets page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

26 August 2016

Dirk Eddelbuettel: RcppArmadillo 0.7.400.2.0

armadillo image Another Armadillo 7.* release -- now at 7.400. We skipped the 7.300.* serie release as it came too soon after our most recent CRAN release. Releasing RcppArmadillo 0.7.400.2.0 now keeps us at the (roughly monthly) cadence which works as a good compromise between getting updates out at Conrad's sometimes frantic pace, while keeping CRAN (and Debian) uploads to about once per month. So we may continue the pattern of helping Conrad with thorough regression tests by building against all (by now 253 (!!)) CRAN dependencies, but keeping release at the GitHub repo and only uploading to CRAN at most once a month. Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. The new upstream release adds new more helper functions. Detailed changes in this release relative to the previous CRAN release are as follows:
Changes in RcppArmadillo version 0.7.400.2.0 (2016-08-24)
  • Upgraded to Armadillo release 7.400.2 (Feral Winter Deluxe)
    • added expmat_sym(), logmat_sympd(), sqrtmat_sympd()
    • added .replace()
Changes in RcppArmadillo version 0.7.300.1.0 (2016-07-30)
  • Upgraded to Armadillo release 7.300.1
    • added index_min() and index_max() standalone functions
    • expanded .subvec() to accept size() arguments
    • more robust handling of non-square matrices by lu()
Courtesy of CRANberries, there is a diffstat report. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.