impl TransportStream for TcpStream  

• TLS-backed connections: a new TransportStream implementation that wraps a TCP socket into a TLS wrapper. This will probably rely on OpenSSL bindings.
• Support for all HTTP/2 frames: only the three vital frames are implemented so far; other frames are a prerequisite to implementing properly closing streams, handling closed streams etc.
• Signaling stream- and connection-level errors to the peer
• Flow control: right now this aspect is completely ignored
• Server push: the idea is that multithreaded clients could handle pushed responses as soon as they arive, whereas single-threaded ones would be allowed to fetch a list of them every so often.

Changes in Rcpp version 0.11.5 (2015-03-04)

Changes in Rcpp API:

• An error handler for tinyformat was defined to prevent the assert() macro from spilling.
• The Rcpp::warning function was added as a wrapper for Rf_warning.
• The XPtr class was extended with new checked_get and release functions as well as improved behavior (throw an exception rather than crash) when a NULL external pointer is dereferenced.
• R code is evaluated within an R_toplevelExec block to prevent user interrupts from bypassing C++ destructors on the stack.
• The Rcpp::Environment constructor can now use a supplied parent environment.
• The Rcpp::Function constructor can now use a supplied environment or namespace.
• The attributes_hidden macro from R is used to shield internal functions; the R_ext/Visibility.h header is now included as well.
• A Rcpp::print function was added as a wrapper around Rf_PrintValue.

Changes in Rcpp Attributes:

• The pkg_types.h file is now included in RcppExports.cpp if it is present in either the inst/include or src.
• sourceCpp was modified to allow includes of local files (e.g. #include "foo.hpp"). Implementation files (.cc; .cpp) corresponding to local includes are also automatically built if they exist.
• The generated attributes code was simplified with respect to RNGScope and now uses RObject and its destructor rather than SEXP protect/unprotect.
• Support addition of the rng parameter in Rcpp::export to suppress the otherwise automatic inclusion of RNGScope in generated code.
• Attributes code was made more robust and can e.g. no longer recurse.
• Version 3.2 of the Rtools is now correctly detected as well.
• Allow 'R' to come immediately after '***' for defining embedded R code chunks in sourceCpp.
• The attributes vignette has been updated with documentation on new features added over the past several releases.

Changes in Rcpp tests:

• On Travis CI, all build dependencies are installed as binary .deb packages resulting in faster tests.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

cat >>/etc/default/tomcat7 <<EOF
LC_CTYPE=C.UTF-8
export LC_CTYPE
EOF

Introduction The drat package was released earlier this month, and described in a first blog post. I received some helpful feedback about what works and what doesn't. For example, Jenny Bryan pointed out that I was not making a clear enough distinction between the role of using drat to publish code, and using drat to receive/install code. Very fair point, and somewhat tricky as R aims to blur the line between being a user and developer of statistical analyses, and hence packages. Many of us are both. Both the main point is well taken, and this note aims to clarify this issue a little by focusing on the former. Another point make by Jenny concerns the double use of repository. And indeed, I conflated repository (in the sense of a GitHub code repository) with repository for a package store used by a package manager. The former, a GitHub repository, is something we use to implement a personal drat with: A GitHub repository happens to be uniquely identifiable just by its account name, and given an (optional) gh-pages branch also offers a stable and performant webserver we use to deliver packages for R. A (personal) code repository on the other hand is something we implement somewhere---possibly via drat which supports local directories, possibly on a network share, as well as anywhere web-accessible, e.g. via a GitHub repository. It is a little confusing, but I will aim to make the distinction clearer.

Just once: Setting up a drat repository So let us for the remainder of this post assume the role of a code publisher. Assume you have a package you would like to make available, which may not be on CRAN and for which you would like to make installation by others easier via drat. The example below will use an interim version of drat which I pushed out yesterday (after fixing a bug noticed when pushing the very new RcppAPT package). For the following, all we assume (apart from having a package to publish) is that you have a drat directory setup within your git / GitHub repository. This is not an onerous restriction. First off, you don't have to use git or GitHub to publish via drat: local file stores and other web servers work just as well (and are documented). GitHub simply makes it easiest. Second, bootstrapping one is trivial: just fork my drat GitHub repository and then create a local clone of the fork. There is one additional requirement: you need a gh-pages branch. Using the fork-and-clone approach ensures this. Otherwise, if you know your way around git you already know how to create a gh-pages branch. Enough of the prerequisities. And on towards real fun. Let's ensure we are in the gh-pages branch:

edd@max:~/git/drat(master)$ git checkout gh-pages
Switched to branch 'gh-pages'
Your branch is up-to-date with 'origin/gh-pages'.
edd@max:~/git/drat(gh-pages)$ 

Publish: Run one drat command to insert a package Now, let us assume you have a package to publish. In my case this was version 0.0.1.2 of drat itself as it contains a fix for the very command I am showing here. So if you want to run this, ensure you have this version of drat as the CRAN version is currently behind at release 0.0.1 (though I plan to correct that in the next few days). To publish an R package into a code repository created via drat running on a drat GitHub repository, just run insertPackage(packagefile) which we show here with the optional commit=TRUE. The path to the package can be absolute are relative; the easists is often to go up one directory from the sources to where R CMD build ... has created the package file.

edd@max:~/git$ Rscript -e 'library(drat); insertPackage("drat_0.0.1.2.tar.gz", commit=TRUE)'
[gh-pages 0d2093a] adding drat_0.0.1.2.tar.gz to drat
 3 files changed, 2 insertions(+), 2 deletions(-)
 create mode 100644 src/contrib/drat_0.0.1.2.tar.gz
Counting objects: 7, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 7.37 KiB   0 bytes/s, done.
Total 7 (delta 1), reused 0 (delta 0)
To git@github.com:eddelbuettel/drat.git
   206d2fa..0d2093a  gh-pages -> gh-pages
edd@max:~/git$ 

You can equally well run this as insertPackage("drat_0.0.1.2.tar.gz"), then inspect the repo and only then run the git commands add, commit and push. Also note that future versions of drat will most likely support git operations directly by relying on the very promising git2r package. But this just affect package internals, the user-facing call of e.g. insertPackage("drat_0.0.1.2.tar.gz", commit=TRUE) will remain unchanged. And in a nutshell that really is all there is to it. With the newly drat-ed package pushed to your GitHub repository with a single function call), it is available via the automatically-provided gh-pages webserver access to anyone in the world. All they need to do is to point R's package management code (which is built into R itself and used for e.g._ CRAN and BioConductor R package repositories) to the new repo---and that is also just a single drat command. We showed this in the first blog post and may expand on it again in a follow-up. So in summary, that really is all there is to it. After a one-time setup / ensuring you are on the gh-pages branch, all it takes is a single function call from the drat package to publish your package to your drat GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

R> hasPackages(c("r-cran-rcpp", "r-cran-rcppapt"))
   r-cran-rcpp r-cran-rcppapt 
          TRUE          FALSE 

R> library(RcppAPT)
R> getPackages("^r-base-c.")
          Package      Installed       Section
1 r-base-core-dbg 3.1.2-1utopic0 universe/math
2 r-base-core-dbg           <NA> universe/math
3     r-base-core 3.1.2-1utopic0 universe/math
4     r-base-core           <NA> universe/math
R> 

R> dim(getPackages())
[1] 104431      3
R> 

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

• A lot of logic was moved out of harness, meaning it is now closer to becoming a simple dumb task scheduler. With the logic being moved out in separate processes, harness now hogs vastly less memory that I cannot convince perl to release to the OS. On lilburn.debian.org vastly less is on the order of reducing 700ish MB to 32 MB .
• All important metadata was moved into the harness state-cache , which is a simple YAML file. This means that Lintian laboratory is no longer a data store. This change causes a lot of very positive side effects.
• With all metadata now stored in a single file, we can now do atomic updates of the data store. That said, this change itself does not enable us to run multiple lintian s in parallel.
• As the lintian laboratory is no longer a data store, we can now do our processing in throw away laboratories like the regular lintian user does. As the permanent laboratory is the primary source of failure, this removes an entire class of possible problems.

• Packages can now be up to date in the generated reports. Previously, they would always be listed as out of date even if they were up to date. This is the only end user/website-visitor visible change in all of this (besides the graphs are now working again \o/).
• The size of the harness work list is no longer based on the number of changes to the archive.
• The size of the harness work list can now be changed with a command line option and is no longer hard coded to 1024. However, the time limit remains hard coded for now.
• The full run (and clean run ) now simply marks everything out-of-date and processes its (new) backlog over the next (many) harness runs. Accordingly, a full-run no longer causes lintian to run 5-7 days on lilburn.d.o before getting an update to the website. Instead we now get incremental updates.
• The harness.log now features status updates from lintian as they happen with processed X successfully or error processing Y plus a little wall time benchmark. With this little feature I filed no less than 3 bugs against lintian 2 of which are fixed in git. The last remains unfixed but can only be triggered in Debian stable.
• It is now possible with throw-away labs to terminate the lintian part of a reporting run early with minimal lost processing. Since the lintian-harness is regular fed status updates from lintian, we can now mark successfully completed entries as done even if lintian does not complete its work list. Caveat: There may be minor inaccuracies in the generated report for the particular package lintian was processing when it was interrupted. This will fix itself when the package is reprocessed again.
• It is now vastly easier to collect new meta data to be used in the reports. Previously, they had to be included in the laboratory and extracted from there. Now, we just have to fit it into a YAML file. In fact, I have been considering to add the wall time and make a top X slowest page.
• It is now possible to generate the html pages with only a state-cache and the lintian.log file. Previously, it also required a populated lintian laboratory.

Why Johnny Doesn t Read When someone needs to know something, they might go look for some documentation, or they might ask someone else or just guess wildly. The behaviour look for documentation is often reinforced negatively, by the result documentation doesn t exist . At the same time, the behaviours ask someone and guess wildly are positively reinforced, by the results I get my question answered and/or at least I can get on with my work . Over time, people optimise their behaviour by skipping the look for documentation step, and just go straight to asking other people (or guessing wildly).

Why Johnny Doesn t Write When someone writes documentation, they re hoping that people will read it and not have to ask them questions in order to be productive and do the right thing. Hence, the behaviour write documentation is negatively reinforced by the results I still get asked questions , and nobody does things the right way around here, dammit! Worse, though, is that there is very little positive reinforcement for the author: when someone does read the docs, and thus doesn t ask a question, the author almost certainly doesn t know they dodged a bullet. Similarly, when someone does things the right way, it s unlikely that anyone will notice. It s only the mistakes that catch the attention. Given that the experience of writing documentation tends to skew towards the negative, it s not surprising that eventually, the time spent writing documentation is reallocated to other, more utility-producing activities.

Death Spiral The combination of these two situations is self-reinforcing. While a suitably motivated reader might start by strictly looking for documentation, or an author initially be enthused to always fully documenting their work, over time the reflex will be for readers to just go ask someone, because there s never any documentation! , and for authors to not write documentation because nobody bothers to read what I write anyway! . It is important to recognise that this iterative feedback loop is the natural state of the reader/author ecosystem, resulting in something akin to thermodynamic entropy. To avoid the system descending into chaos, energy needs to be constantly applied to keep the system in order.

The Solution Effective methods for avoiding the vicious circle can be derived from the things that cause it. Change the forces that apply themselves to readers and authors, and they will behave differently. On the reader s side, the most effective way to encourage people to read documentation is for it to consistently exist. This means that those in control of a project or system mustn t consider something done until the documentation is in a good state. Patches shouldn t be landed, and releases shouldn t be made, unless the documentation is altered to match the functional changes being made. Yes, this requires discipline, which is just a form of energy application to prevent entropic decay. Writing documentation should be an explicit and well-understood part of somebody s job description. Whoever is responsible for documentation needs to be given the time to do it properly. Writing well takes time and mental energy, and that time needs to be factored into the plans. Never forget that skimping on documentation, like short-changing QA or customer support, is a false economy that will cost more in the long term than it saves in the short term. Even if the documentation exists, though, some people are going to tend towards asking people rather than consulting the documentation. This isn t a moral failing on their part, but only happens when they believe that asking someone is more beneficial to them than going to the documentation. To change the behaviour, you need to change the belief. You could change the belief by increasing the cost of asking. You could fire (or hellban) anyone who ever asks a question that is answered in the documentation. But you shouldn t. You could yell RTFM! at everyone who asks a question. Thankfully that s one acronym that s falling out of favour. Alternately, you can reduce the cost of getting the answer from the documentation. Possibly the largest single productivity boost for programmers, for example, has been the existence of Google. Whatever your problem, there s a pretty good chance that a search or two will find a solution. For your private documentation, you probably don t have the power of Google available, but decent full-text search systems are available. Use them. Finally, authors would benefit from more positive reinforcement. If you find good documentation, let the author know! It requires a lot of effort (comparatively) to look up an author s contact details and send them a nice e-mail. The like button is a more low-energy way of achieving a similar outcome you click the button, and the author gets a warm, fuzzy feeling. If your internal documentation system doesn t have some way to close the loop and let readers easily give authors a bit of kudos, fix it so it does. Heck, even if authors just know that a page they wrote was loaded N times in the past week, that s better than the current situation, in which deafening silence persists, punctuated by the occasional plaintive cry of Hey, do you know how to ? . Do you have any other ideas for how to encourage readers to read, and for authors to write?

or Diaspora*

• Brebis was renamed Backup Checker, the last one being more explicit.
• Mercurial ,the distributed version control system of the project, was replaced by Git.
• The project switched from a self hosted old Redmine to GitHub. Here is the GitHub project page.

• The default behaviour calculated the hash sums of every files in the archive or the tree of files, this was discontinued because of poor performances while using Backup Checker on archives of large size.
• You can force the old behaviour by using the new hashes option.
• The new exceptions-file option allows the user to provide a list of files inside the archive in order to compute their hash sums.
• The documentation of the project is now available on Readthedocs.

• CC1200 radio chip is about 5dB more sensitive than TeleDongle's CC1111.
• LPC11U14 CPU can be reprogrammed over the USB link.

• Make sure the startup flight computer beeps are consistent. Sometimes, it was taking long enough to beep out the battery voltage that the flight computer state was changing in the middle, causing a bit of confusion.
• Change TeleDongle's LED indicators during telemetry reception. The green LED blinks on successful packet reception, and the red LED blinks when a packet with an invalid checksum is received.
• The SPI driver used in both TeleDongle v3 and TeleGPS has been rewritten to avoid locking up under heavy CPU load. If you've got a TeleGPS board, you'll want to reflash with new firmware.

• AltosUI can now compute and display tilt angle when graphing eeprom log data from TeleMega and EasyMega.
• The AltosUI tool window is shown when starting with a data file. This way, when you double-click on a file in the file manager, you'll get the whole AltosUI interface, rather than just the graphing window.
• At the end of replaying an old log file, stick 'done' in the Age field so you can tell the recording is over.

• Use a different Windows API to discover USB device ids. This works better on my new HP Windows 7 machine. Maybe it will work better for other people too?
• Look in more places in the Windows registry to try and find the installed Java version. It appears that the default Java download from Oracle is a 32-bit version? In any case, that version sticks its install information in a different spot in the registry.
• Fix file associations on Windows when Java isn't installed in the system root.
• Make 'Scan Channels' work better with new AltOS firmware which only reports device configuration information once ever five seconds.

or Diaspora*

• Brebis was renamed Backup Checker, the last one being more explicit.
• Mercurial ,the distributed version control system of the project, was replaced by Git.
• The project switched from a self hosted old Redmine to GitHub. Here is the GitHub project page.

• The default behaviour calculated the hash sums of every files in the archive or the tree of files, this was discontinued because of poor performances while using Backup Checker on archives of large size.
• You can force the old behaviour by using the new hashes option.
• The new exceptions-file option allows the user to provide a list of files inside the archive in order to compute their hash sums.
• The documentation of the project is now available on Readthedocs.

$ hash -d -- -lldpd=/home/bernat/code/deezer/lldpd
$ echo ~-lldpd/README.md
/home/bernat/code/deezer/lldpd/README.md
$ head -n1 ~-lldpd/README.md
lldpd: implementation of IEEE 802.1ab (LLDP)

for link ($MARKPATH/*(N@))  
    hash -d -- -$ link:t =$ link:A 
 

$ pwd
/home/bernat/code/deezer/lldpd/src/lib
$ echo $ (%):-%~ 
~-lldpd/src/lib

bookmark()  
    if (( $# == 0 )); then
        # When no arguments are provided, just display existing
        # bookmarks
        for link in $MARKPATH/*(N@); do
            local markname="$fg[green]$ link:t $reset_color"
            local markpath="$fg[blue]$ link:A $reset_color"
            printf "%-30s -> %s\n" $markname $markpath
        done
    else
        # Otherwise, we may want to add a bookmark or delete an
        # existing one.
        local -a delete
        zparseopts -D d=delete
        if (( $+delete[1] )); then
            # With  -d , we delete an existing bookmark
            command rm $MARKPATH/$1
        else
            # Otherwise, add a bookmark to the current
            # directory. The first argument is the bookmark
            # name.  .  is special and means the bookmark should
            # be named after the current directory.
            local name=$1
            [ $name == "." ] && name=$ PWD:t 
            ln -s $PWD $MARKPATH/$name
        fi
    fi
 

Dynamic named directories Another (more complex) way to achieve the same thing is using dynamic named directories. I was initially using this solution but it is far more complex. This section is only here for historical reason. You can find the complete implementation in my GitHub repository. During file name expansion, a ~ followed by a string in square brackets is provided to the zsh_directory_name() function which will eventually reply with a directory name. This feature can be used to implement directory bookmarks:

$ cd ~[@lldpd]
$ pwd
/home/bernat/code/deezer/lldpd
$ echo ~[@lldpd]/README.md
/home/bernat/code/deezer/lldpd/README.md
$ head -n1 ~[@lldpd]/README.md
lldpd: implementation of IEEE 802.1ab (LLDP)

Like for static named directories, because ~[@lldpd] is substituted during file name expansion, it is possible to use it in any command like a regular directory.

Basic implementation Bookmarks are still kept into a dedicated directory, $MARKPATH and are still symbolic links. Here is how the core feature is implemented:

_bookmark_directory_name()  
    emulate -L zsh #  
    setopt extendedglob
    case $1 in
        n)
            [ [ $2 != ( #b)"@"(?*) ]] && return 1 #  
            typeset -ga reply
            reply=($ $ :-$MARKPATH/$match[1] :A ) #  
            return 0
            ;;
        *)
            return 1
            ;;
    esac
    return 0
 
add-zsh-hook zsh_directory_name _bookmark_directory_name

zsh_directory_name() is a function accepting hooks²: instead of defining it directly, we define another function and register it as a hook with add-zsh-hook. The hook is expected to handle different situations. The first one is to be able to transform a dynamic name into a regular directory name. In this case, the first parameter of the function is n and the second one is the dynamic name. In , the call to emulate will restore the pristine behaviour of Zsh and also ensure that any option set in the scope of the function will not have an impact outside. The function can then be reused safely in another environment. In , we check that the dynamic name starts with @ followed by at least one character. Otherwise, we declare we don t know how to handle it. Another hook will get the chance to do something. (#b) is a globbing flag. It activates backreferences for parenthesised groups. When a match is found, it is stored as an array, $match. In , we build the reply. We could have just returned $MARKPATH/$match[1] but to hide the symbolic link mechanism, we use the A modifier to ask Zsh to resolve symbolic links if possible. Zsh allows nested substitutions. It is therefore possible to use modifiers and flags on anything. $ :-$MARKPATH/$match[1] is a common trick to turn $MARKPATH/$match[1] into a parameter substitution and be able to apply the A modifier on it.

Completion Zsh is also able to ask for completion of a dynamic directory name. In this case, the completion system calls the hook function with c as the first argument.

_bookmark_directory_name()  
    # [...]
    case $1 in
        c)
            # Completion
            local expl
            local -a dirs
            dirs=($MARKPATH/*(N@:t)) #  
            dirs=("@"$ ^dirs ) #  
            _wanted dynamic-dirs expl 'bookmarked directory' compadd -S\] -a dirs
            return
            ;;
        # [...]
    esac
    # [...]
 

First, in , we create a list of possible bookmarks. In *(N@:t), N@ is a glob qualifier. N allows us to not return nothing if there is no match (otherwise, we would get an error) while @ only returns symbolic links. t is a modifier which will remove all leading pathname components. This is equivalent to use basename or $ something##*/ in POSIX shells but it plays nice with glob expressions. In , we just add @ before each bookmark name. If we have b1, b2 and b3 as bookmarks, $ ^dirs expands to b1,b2,b3 and therefore "@"$ ^dirs expands to the (@b1 @b2 @b3) array. The result is then feeded into the completion system.

Prompt expansion Many people put the name of the current directory in their prompt. It would be nice to have the bookmark name instead of the full name when we are below a bookmarked directory. That s also possible!

$ pwd
/home/bernat/code/deezer/lldpd/src/lib
$ echo $ (%):-%~ 
~[@lldpd]/src/lib

The prompt expansion system calls the hook function with d as first argument and the file name to transform.

_bookmark_directory_name()  
    # [...]
    case $1 in
        d)
            local link slink
            local -A links
            for link ($MARKPATH/*(N@))  
                links[$  #link:A $'\0'$ link:A ]=$ link:t  #  
             
            for slink ($ (@On)$ (k)links )  
                link=$ slink#*$'\0'  #  
                if [ [ $2 = ( #b)($ link )( /*) ]]; then
                    typeset -ga reply
                    reply=("@"$ links[$slink]  $(( $  #match[1]  )) )
                    return 0
                fi
             
            return 1
            ;;
        # [...]
    esac
    # [...]
 

OK. This is some black Zsh wizardry. Feel free to skip the explanation. This is a bit complex because we want to substitute the most specific bookmark, hence the need to sort bookmarks by their target lengths. In , the associative array $links is created by iterating on each symbolic link ($link) in the $MARKPATH directory. The goal is to map a target directory with the matching bookmark name. However, we need to iterate on this map from the longest to the shortest key. To achieve that, we prepend each key with its length. Remember, $ link:A is the absolute path with symbolic links resolved. So, $ #link:A is the length of this path. We concatenate the length of the target directory with the target directory name and use $'\0' as a separator because this is the only safe character for this purpose. The result is mapped to the bookmark name. The second loop is an iteration on the keys of the associative array $links (thanks to the use of the k parameter flag in $ (k)links ). Those keys are turned into an array (@ parameter flag) and sorted numerically in descending order (On parameter flag). Since the keys are directory names prefixed by their lengths, the first match will be the longest one. In , we extract the directory name from the key by removing the length and the null character at the beginning. Then, we check if the extracted directory name matches the file name we have been provided. Again, (#b) just activates backreferences. With extended globbing, we can use the or operator, . So, when either the file name matches exactly the directory name or is somewhere deeper, we create the reply which is an array whose first member is the bookmark name and the second member is the untranslated part of the file name.

Easy typing Typing ~[@ is cumbersome. Hopefully, Zsh line editor can be extended with additional bindings. The following snippet will substitute @@ (if typed without a pause) by ~[@:

vbe-insert-bookmark()  
    emulate -L zsh
    LBUFFER=$ LBUFFER "~[@"
 
zle -N vbe-insert-bookmark
bindkey '@@' vbe-insert-bookmark

In combination with the autocd option and completion, it is quite easy to jump to a bookmarked directory.

---

1. For example:

   $ lldpd=/home/bernat/code/deezer/lldpd
   $ echo $lldpd/README.md
   /home/bernat/code/deezer/lldpd/README.md
   $ head -n1 $lldpd/README.md
   lldpd: implementation of IEEE 802.1ab (LLDP)
   

   The drawback is that you don t have a separate namespace for your bookmarks. You can still use a special prefix for that. Also, no prompt expansion.
2. Other functions accepting hooks are chpwd() or precmd().