Search Results: "Dirk Eddelbuettel"

2 December 2012

Dirk Eddelbuettel: RQuantLib 0.3.9

A minor feature release RQuantLib 0.3.9 is now on CRAN and in Debian. RQuantLib combines (some of) the quantitative analytics of QuantLib with the R statistical computing environment and language. Bryan Lewis had suggested to enable another pricing engine for American Options in order to get (at least some) Greeks. This is now supported by picking engine="CrankNicolson" as shown in the default example for the AmericanOption function:
R> library(RQuantLib)
R> example(AmericanOption)
AmrcnOR> # simple call with unnamed parameters
AmrcnOR> AmericanOption("call", 100, 100, 0.02, 0.03, 0.5, 0.4)
Concise summary of valuation for AmericanOption 
  value   delta   gamma    vega   theta     rho  divRho 
11.3648      NA      NA      NA      NA      NA      NA 
AmrcnOR> # simple call with some explicit parameters
AmrcnOR> AmericanOption("put", strike=100, volatility=0.4, 100, 0.02, 0.03, 0.5)
Concise summary of valuation for AmericanOption 
  value   delta   gamma    vega   theta     rho  divRho 
10.9174      NA      NA      NA      NA      NA      NA 
AmrcnOR> # simple call with unnamed parameters, using Crank-Nicolons
AmrcnOR> AmericanOption("put", strike=100, volatility=0.4, 100, 0.02, 0.03, 0.5, engine="CrankNicolson")
Concise summary of valuation for AmericanOption 
  value   delta   gamma    vega   theta     rho  divRho 
10.9173 -0.4358  0.0140      NA      NA      NA      NA 
R> 
Thanks to CRANberries, there is also a diff to the previous release 0.3.8. Full changelog details, examples and more details about this package are at my RQuantLib page.

28 November 2012

Dirk Eddelbuettel: Rcpp 0.10.1

A the new Rcpp release 0.10.1 arrived this morning on CRAN (as already has Windows binaries) and in Debian. This is a follow-up to the recent 0.10.0 release which extends the exciting new Rcpp-attributes and Rcpp-sugar work further, and as in a number of other areas as detailed below in the NEWS sections. This release brings an change to some of the binary interfaces. If you have packages using Rcpp, you will most likely have to reinstall them from source. Some change were made to const correctness as well as other aspects, and it seems that we have temporarily broken the excellent RcppEigen and RcppOctave packages. We are looking into this, and are sorry about the bug. The complete NEWS entry for 0.10.1 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.
Changes in Rcpp version 0.10.1 (2012-11-26)
  • Changes in Rcpp sugar:
    • New functions: setdiff, union_, intersect setequal, in, min, max, range, match, table, duplicated
    • New function: clamp which combines pmin and pmax, e.g. clamp( a, x, b) is the same as pmax( b, pmin(x, a) )
    • New function: self_match which implements something similar to match( x, unique( x ) )
  • Changes in Rcpp API:
    • The Vector template class (hence NumericVector ...) get the is_na and the get_na static methods.
    • New helper class no_init that can be used to create a vector without initializing its data, e.g. : IntegerVector out = no_init(n) ;
    • New exception constructor requiring only a message; stop function to throw an exception
    • DataFrame gains a nrows method
  • Changes in Rcpp attributes:
    • Ability to embed R code chunks (via specially formatted block comments) in C++ source files.
    • Allow specification of argument defaults for exported functions.
    • New scheme for more flexible mixing of generated and user composed C++ headers.
    • Print warning if no export attributes are found in source file.
    • Updated vignette with additional documentation on exposing C++ interfaces from packages and signaling errors.
  • Changes in Rcpp modules:
    • Enclose .External invocations in BEGIN_RCPP/END_RCPP
  • Changes in R code :
    • New function areMacrosDefined
    • Additions to Rcpp.package.skeleton:
      • attributes parameter to generate a version of rcpp_hello_world that uses Rcpp::export.
      • cpp_files parameter to provide a list of C++ files to include the in the src directory of the package.
  • Miscellaneous changes:
    • New example 'pi simulation' using R and C++ via Rcpp attributes
Thanks to CRANberries, you can also look at a diff to the previous release 0.10.0. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

25 November 2012

Dirk Eddelbuettel: digest 0.6.0

A new version of the digest package (which generates hash function summaries for arbitrary (and possibly nested) R objects using any of the standard md5, sha-1, sha-256, sha-512 or crc32 algorithms) is now on CRAN, and I will push the corresponding version into Debian in a moment. For this release, Hannes Muehleisen added support for sha-512 using an older standalone function by Aaron D. Gifford which I had to whip into slightly more portable shape to work on Windows. (Hint: uint32_t from stdint.h, not u_int32_t) CRANberries provides the usual summary of changes to version 0.5.2. As usual, our package is available via the R-Forge page leading to svn and tarball access, my digest page and the local directory here.

21 November 2012

Dirk Eddelbuettel: Rcpp attributes: A simple example 'making pi'

We introduced Rcpp 0.10.0 with a number of very nice new features a few days ago, and the activity on the rcpp-devel mailing list has been pretty responsive which is awesome. But because few things beat a nice example, this post tries to build some more excitement. We will illustrate how Rcpp attributes makes it really easy to add C++ code to R session, and that that code is as easy to grasp as R code. Our motivating example is everybody's favourite introduction to Monte Carlo simulation: estimating . A common method uses the fact the unit circle has a surface area equal to . We draw two uniform random numbers x and y, each between zero and one. We then check for the distance of the corresponding point (x,y) relative to the origin. If less than one (or equal), it is in the circle (or on it); if more than one it is outside. As the first quadrant is a quarter of a square of area one, the area of the whole circle is -- so our first quadrant approximates over four. The following figure, kindly borrowed from Wikipedia with full attribution and credit, illustrates this: Example of simulating pi Now, a vectorized version (drawing N such pairs at once) of this approach is provided by the following R function.
piR <- function(N)  
    x <- runif(N)
    y <- runif(N)
    d <- sqrt(x^2 + y^2)
    return(4 * sum(d < 1.0) / N)
 
And in C++ we can write almost exactly the same function thanks the Rcpp sugar vectorisation available via Rcpp:
#include <Rcpp.h>
using namespace Rcpp;
// [[Rcpp::export]]
double piSugar(const int N)  
    RNGScope scope;		// ensure RNG gets set/reset
    NumericVector x = runif(N);
    NumericVector y = runif(N);
    NumericVector d = sqrt(x*x + y*y);
    return 4.0 * sum(d < 1.0) / N;
 
Sure, there are small differences: C++ is statically typed, R is not. We need one include file for declaration, and we need one instantiation of the RNGScope object to ensure random number draws remain coordinated between the calling R process and the C++ function calling into its (compiled C-code based) random number generators. That way we even get the exact same draws for the same seed. But the basic approach is identical: draw a vector x and vector y, compute the distance to the origin and then obtain the proportion within the unit circle -- which we scale by four. Same idea, same vectorised implementation in C++. But the real key here is the one short line with the [[Rcpp::export]] attribute. This is all it takes (along with sourceCpp() from Rcpp 0.10.0) to get the C++ code into R. The full example (which assumes the C++ file is saved as piSugar.cpp in the same directory) is now:
#!/usr/bin/r
library(Rcpp)
library(rbenchmark)
piR <- function(N)  
    x <- runif(N)
    y <- runif(N)
    d <- sqrt(x^2 + y^2)
    return(4 * sum(d < 1.0) / N)
 
sourceCpp("piSugar.cpp")
N <- 1e6
set.seed(42)
resR <- piR(N)
set.seed(42)
resCpp <- piSugar(N)
## important: check results are identical with RNG seeded
stopifnot(identical(resR, resCpp))
res <- benchmark(piR(N), piSugar(N), order="relative")
print(res[,1:4])
and it does a few things: set up the R function, source the C++ function (and presto: we have a callable C++ function just like that), compute two simulations given the same seed and ensure they are in fact identical -- and proceed to compare the timing in a benchmarking exercise. That last aspect is not even that important -- we end up being almost-but-not-quite twice as fast on my machine for different values of N. The real takeaway here is the ease with which we can get a C++ function into R --- and the new process completely takes care of passing parameters in, results out, and does the compilation, linking and loading. More details about Rcpp attributes are in the new vignette. Now enjoy the .

Update:One somewhat bad typo fixed.

Update:Corrected one background tag.

17 November 2012

Dirk Eddelbuettel: RcppArmadillo 0.3.4.4

A minor bug-fix release 3.4.4 of Armadillo came out upstream a few days ago. RcppArmadillo, our wrapper for R and Armadillo, is now on CRAN with its corresponding version 0.3.4.4. No R level or interface changes were made and the upstream changes are summarized below.
Changes in RcppArmadillo version 0.3.4.4 (2012-11-15)
  • Upgraded to Armadillo release 3.4.4
    • fix for handling complex numbers by sparse matrices
    • fix for minor memory leak by sparse matrices
Courtesy of CRANberries, there is also a diffstat report for 0.3.4.4 relative to 0.3.4.3 As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

15 November 2012

Dirk Eddelbuettel: Rcpp and the new R:: namespace for Rmath.h

We released Rcpp 0.10.0 earlier today. This post will just provide a simple example for one of the smaller new features -- the new namespace for functions from Rmath.h -- and illustrate one of the key features (Rcpp attributes) in passing. R, as a statistical language and environment, has very well written and tested statistical distribution functions providing probability density, cumulative distribution, quantiles and random number draws for dozens of common and not so common distribution functions. This code is used inside R, and available for use from standalone C or C++ programs via the standalone R math library which Debian / Ubuntu have as a package r-mathlib (and which can be built from R sources). User sometimes write code against this interface, and then want to combine the code with other code, possibly even with Rcpp. We allowed for this, but it required a bit of an ugly interface. R provides a C interface; these have no namespaces. Identifiers can clash, and to be safe one can enable a generic prefix Rf_. So functions which could clash such as length or error become Rf_length and Rf_error and are less likely to conflict with symbols from other libraries. Unfortunately, the side-effect is that calling, say, the probability distribution function for the Normal distribution becomes Rf_pnorm5() (with the 5 denoting the five parameters: quantile, mean, std.deviation, lowerTail, logValue). Not pretty, and not obvious. So one of the things we added was another layer of indirection by adding a namespace R with a bunch of inline'd wrapper functions (as well as several handful of unit tests to make sure we avoided typos and argument transposition and what not). The short example below shows this for a simple function taking a vector, and returning its pnorm computed three different ways:
#include <Rcpp.h>
// [[Rcpp::export]]
Rcpp::DataFrame mypnorm(Rcpp::NumericVector x)  
    int n = x.size();
    Rcpp::NumericVector y1(n), y2(n), y3(n);
    for (int i=0; i<n; i++)  
        // the way we used to do this
        y1[i] = ::Rf_pnorm5(x[i], 0.0, 1.0, 1, 0);
        // the way we can do it now
        y2[i] = R::pnorm(x[i], 0.0, 1.0, 1, 0);
     
    // or using Rcpp sugar in one go
    y3 = Rcpp::pnorm(x);
    return Rcpp::DataFrame::create(Rcpp::Named("Rold")  = y1,
                                   Rcpp::Named("Rnew")  = y2,
                                   Rcpp::Named("sugar") = y3);
 
This example also uses the new Rcpp attributes described briefly in the announcement blog post and of course in more detail in the corresponding vignette. Let us just state here that we simply provide a complete C++ function, using standard Rcpp types -- along with one 'attribute' declaration of an export via Rcpp. That's it -- even easier than using inline. Now in R we simply do
R> sourceCpp("mypnorm.cpp")
to obtain a callable R function with the C++ code just shown behind it. No Makefile, no command-line tool invocation -- nothing but a single call to sourceCpp() which takes care of things --- and brings us a compiled C++ function to R just given the source file with its attribute declaration. We can now use the new function to compute the probaility distribution both the old way, the new way with the 'cleaner' R::pnorm(), and of course the Rcpp sugar way in a single call. We build a data frame in C++, and assert that all three variants are the same:
R> x <- seq(0, 1, length=1e3)
R> res <- mypnorm(x)
R> head(res)
      Rold     Rnew    sugar
1 0.500000 0.500000 0.500000
2 0.500399 0.500399 0.500399
3 0.500799 0.500799 0.500799
4 0.501198 0.501198 0.501198
5 0.501597 0.501597 0.501597
6 0.501997 0.501997 0.501997
R> all.equal(res[,1], res[,2], res[,3])
[1] TRUE
R> 
This example hopefully helped to illustrate how Rcpp 0.10.0 brings both something really powerful (Rcpp attributes -- more on this another time, hopefully) and convenient in the new namespace for statistical functions.

14 November 2012

Dirk Eddelbuettel: Rcpp 0.10.0

Rcpp release 0.10.0 is now on CRAN and being uploaded to Debian. This is a new feature release, and we are very exciting about the changes, notably Rcpp attributes which make using C++ from R even easier than inline (see below as well as the new vignette for details and first examples), the extensions to Rcpp modules (see below) and more as for example new Rcpp sugar functions, a new error output device syncing to R, and a new namespace R> for the statistical functions from Rmath.h. The complete NEWS entry for 0.10.0 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.
Changes in Rcpp version 0.10.0 (2012-11-13)
  • Support for C++11 style attributes (embedded in comments) to enable use of C++ within interactive sessions and to automatically generate module declarations for packages:
    • Rcpp::export attribute to export a C++ function to R
    • sourceCpp() function to source exported functions from a file
    • cppFunction() and evalCpp() functions for inline declarations and execution
    • compileAttribtes() function to generate Rcpp modules from exported functions within a package
    • Rcpp::depends attribute for specifying additional build dependencies for sourceCpp()
    • Rcpp::interfaces attribute to specify the external bindings compileAttributes() should generate (defaults to R-only but a C++ include file using R_GetCCallable can also be generated)
    • New vignette "Rcpp-attribute"
  • Rcpp modules feature set has been expanded:
    • Functions and methods can now return objects from classes that are exposed through modules. This uses the make_new_object template internally. This feature requires that some class traits are declared to indicate Rcpp's wrap/as system that these classes are covered by modules. The macro RCPP_EXPOSED_CLASS and RCPP_EXPOSED_CLASS_NODECL can be used to declared these type traits.
    • Classes exposed through modules can also be used as parameters of exposed functions or methods.
    • Exposed classes can declare factories with ".factory". A factory is a c++ function that returns a pointer to the target class. It is assumed that these objects are allocated with new on the factory. On the R side, factories are called just like other constructors, with the "new" function. This feature allows an alternative way to construct objects.
    • "converter" can be used to declare a way to convert an object of a type to another type. This gets translated to the appropriate "as" method on the R side.
    • Inheritance. A class can now declare that it inherits from another class with the .derives<Parent>( "Parent" ) notation. As a result the exposed class gains methods and properties (fields) from its parent class.
  • New sugar functions:
    • which_min implements which.min. Traversing the sugar expression and returning the index of the first time the minimum value is found.
    • which_max idem
    • unique uses unordered_set to find unique values. In particular, the version for CharacterVector is found to be more efficient than R's version
    • sort_unique calculates unique values and then sorts them.
  • Improvements to output facilities:
    • Implemented sync() so that flushing output streams works
    • Added Rcerr output stream (forwarding to REprintf)
  • Provide a namespace 'R' for the standalone Rmath library so that Rcpp users can access those functions too; also added unit tests
  • Development releases sets variable RunAllRcppTests to yes to run all tests (unless it was alredy set to 'no'); CRAN releases do not and still require setting which helps with the desired CRAN default of less testing at the CRAN server farm.
Thanks to CRANberries, you can also look at a diff to the previous release 0.9.15. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Update: One link corrected.

5 November 2012

Dirk Eddelbuettel: RInside 0.2.9

A new version 0.2.9 of RInside arrived on CRAN earlier today; Windows binaries have already been built too. RInside provides a set of convenience classes which facilitate embedding of R inside of C++ applications and programs, using the classes and functions provided by the Rcpp R and C++ integration package. This release adds a few new features as detailed in the extract from the NEWS below. A key new feature is the added support for resilience to bad user input based on discussions and an initial (but altered) patch by Theodore Lytras. There are a few ways that this can be deployedon so we also added two more example programs detailing it. The featured Qt example and the Wt example have been updated to use this too. Another feature that may be quite useful for some is the additonal attempt to find a value for R_HOME if none has been set. The value is typically found at compile time of the RInside package which poses a problem for those using the Windows build -- which is shipped as a binary reflecting the value of the build system. One alternative has always been to build the package locally too to get the local value, or to set it explicitly. But because the error behaviour a cryptic message of Cannot open base package is confusing to many, we now try to call the R function which gets this value from the registry. This may need more tweaking and testing, and if you use RInside on the Windows platform I would appreciate feedback. The complete list of changes since the last release are summarized below in the corresponding NEWS file entry:
Changes in RInside version 0.2.9 (2012-11-04)
  • Applied (modified) patch by Theodore Lytras which lets RInside recover from some parsing errors and makes RInside applications more tolerant of errors
  • Added non-throwing variants of parseEval() and parseEvalQ()
  • Modified Qt and Wt examples of density estimation applications to be much more resilient to bad user input
  • On Windows, have RInside use R's get_R_HOME() function to get R_HOME value from registry if not set by user
  • Added note to examples/standard/Makefile.win that R_HOME may need to be set to run the executables so either export your local value, or re-install RInside from source to have it reflected in the library build of libRinside
  • Updated CMake build support for standard, armadillo and eigen
  • Improved CMake builds of examples/standard, examples/eigen and examples/armadillo by detecting architecture
CRANberries also provides a short report with changes from the previous release. More information is on the RInside page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

26 October 2012

Dirk Eddelbuettel: Accelerating R code: Computing Implied Volatilities Orders of Magnitude Faster

This blog, together with Romain's, is one of the main homes of stories about how Rcpp can help with getting code to run faster in the context of the R system for statistical programming and analysis. By making it easier to get already existing C or C++ code to R, or equally to extend R with new C++ code, Rcpp can help in getting stuff done. And it is often fairly straightforward to do so. In this context, I have a nice new example. And for once, it is work-related. I generally cannot share too much of what we do there as this is, well, proprietary, but I have this nice new example. The other day, I was constructing (large) time series of implied volatilities. Implied volatilities can be thought of as the complement to an option's price: given a price (and all other observables which can be thought of as fixed), we compute an implied volatility price (typically via the standard Black-Scholes model). Given a changed implied volatility, we infer a new price -- see this Wikipedia page for more details. In essence, it opens the door to all sorts of arbitrage and relative value pricing adventures. Now, we observe prices fairly frequently to create somewhat sizeable time series of option prices. And each price corresponds to one matching implied volatility, and for each such price we have to solve a small and straightforward optimization problem: to compute the implied volatility given the price. This is usually done with an iterative root finder. The problem comes from the fact that we have to do this (i) over and over and over for large data sets, and (ii) that there are a number of callbacks from the (generic) solver to the (standard) option pricer. So our first approach was to just call the corresponding function GBSVolatility from the fOption package from the trusted Rmetrics project by Diethelm Wuertz et al. This worked fine, but even with the usual tricks of splitting over multiple cores/machines, it simply took too long for the resolution and data amount we desired. One of the problems is that this function (which uses the proper uniroot optimizer in R) is not inefficient per se, but simply makes to many function call back to the option pricer as can be seen from a quick glance at the code. The helper function .fGBSVolatility gets called time and time again:
R> GBSVolatility
function (price, TypeFlag = c("c", "p"), S, X, Time, r, b, tol = .Machine$double.eps, 
    maxiter = 10000) 
 
    TypeFlag = TypeFlag[1]
    volatility = uniroot(.fGBSVolatility, interval = c(-10, 10), 
        price = price, TypeFlag = TypeFlag, S = S, X = X, Time = Time, 
        r = r, b = b, tol = tol, maxiter = maxiter)$root
    volatility
 
<environment: namespace:fOptions>
R> 
R> .fGBSVolatility
function (x, price, TypeFlag, S, X, Time, r, b, ...) 
 
    GBS = GBSOption(TypeFlag = TypeFlag, S = S, X = X, Time = Time, 
        r = r, b = b, sigma = x)@price
    price - GBS
 
<environment: namespace:fOptions>
So the next idea was to try the corresponding function from my RQuantLib package which brings (parts of) QuantLib to R. That was seen as been lots faster already. Now, QuantLib is pretty big and so is RQuantLib, and we felt it may not make sense to install it on a number of machines just for this simple problem. So one evening this week I noodled around for an hour or two and combined (i) a basic Black/Scholes calculation and (ii) a standard univariate zero finder (both of which can be found or described in numerous places) to minimize the difference between the observed price and the price given an implied volatility. With about one hundred lines in C++, I had something which felt fast enough. So today I hooked this into R via a two-line wrapper in quickly-created package using Rcpp. I had one more advantage here. For our time series problem, the majority of the parameters (strike, time to maturity, rate, ...) are fixed, so we can structure the problem to be vectorised right from the start. I cannot share the code or more the details of my new implementation. However, both GBSVolatility and EuropeanOprionImpliedVolatility are on CRAN (and as I happen to maintain these for Debian, also just one sudo apt-get install r-cran-foptions r-cran-rquantlib away if you're on Debian or Ubuntu). And writing the other solver is really not that involved. Anyway, here is the result, courtesy of a quick run via the rbenchmark package. We create a vector of length 500; the implied volatility computation will be performed at each point (and yes, our time series are much longer indeed). This is replicated 100 times (as is the default for rbenchmark) for each of the three approaches:
xyz@xxxxxxxx:~$ r xxxxR/packages/xxxxOptions/demo/timing.R
    test replications elapsed  relative user.self sys.self user.child sys.child
3 zzz(X)          100   0.038     1.000     0.040    0.000          0         0
2 RQL(X)          100   3.657    96.237     3.596    0.060          0         0
1 fOp(X)          100 448.060 11791.053   446.644    1.436          0         0
xyz@xxxxxxxx:~$ 
The new local solution is denoted by zzz(X). It is already orders of magnitude faster than the RQL(x) function using RQuantLib (which is, I presume, due to my custom solution internalising the loop). And the new approach is a laughable amount faster than the basic approach (shown as fOp) via fOptions. For one hundred replications of solving implied volatilities for all elements of a vector of size 500, the slow solution takes about 7.5 minutes --- while the fast solution takes 38 milliseconds. Which comes to a relative gain of over 11,000. So sitting down with your C++ compiler to craft a quick one-hundred lines, combining two well-known and tested methods, can reap sizeable benefits. And Rcpp makes it trivial to call this from R.

14 October 2012

Dirk Eddelbuettel: Rcpp 0.9.15

Rcpp release 0.9.15 is now on CRAN and being uploaded to Debian. Martin Morgan provided a clever fix for a header search needed between clang++ (especially on OS X) and g++ (which still provided libstdc++ and headers for clang++). This should hopefully put the clang issues to bed. Ben North noticed an unprotected string conversion when exception messages are turned into R errors which got fixed, and I expanded the coverage of Date (and Datetime) types to deal properly with non-finite values NA, NaN and Inf. The complete NEWS entry for 0.9.15 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.
Changes in Rcpp version 0.9.15 (2012-10-13)
  • Untangling the clang++ build issue about the location of the exceptions header by directly checking for the include file an approach provided by Martin Morgan in a kindly contributed patch as unit tests for them.
  • The Date and Datetime types now correctly handles NA, NaN and Inf representation; the Date type switched to an internal representation via double
  • Added Date and Datetime unit tests for the new features
  • An additional PROTECT was added for parsing exception messages before returning them to R, following a report by Ben North
Thanks to CRANberries, you can also look at a diff to the previous release 0.9.14. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

5 October 2012

Dirk Eddelbuettel: RcppArmadillo 0.3.4.3

Another bug-fix release of Armadillo, now at version 3.4.3 whike the 3.4.* stabilizes, and with it a version 0.3.4.3 of RcppArmadillo, our wrapper for R and Armadillo. The new version is already on CRAN as of earlier today. Once again no R level or interface changes were, the upstream changes are summarized below.
Changes in RcppArmadillo version 0.3.4.3 (2012-10-04)
  • Upgraded to Armadillo release 3.4.3
    • fix for aliasing issue in diagmat()
    • fix for speye() signature
Courtesy of CRANberries, there is also a diffstat report for 0.3.4.3 relative to 0.3.4.2 As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

Dirk Eddelbuettel: RProtoBuf 0.2.6

Release 0.2.6 of RProtoBuf arrived on CRAN earlier this morning. RProtoBuf provides GNU R bindings for the Google Protobuf data encoding library used and released by Google. This release was once more driven largely by Murray whom we have now added among the list of authors of the package too. The NEWS file entry follows below:
Changes in version 0.2.6 (2012-10-04)
  • Applied several more patches by Murray to
    • correct '_' and '__' mismatches in wrapper calls
    • update a few manual pages for style, and add examples
    • fix a bug where NAs were silently treated as TRUE for logical/bool types
    • fix a bug that caused crashes when adding vectors to optional fields
    • fix bugs in readASCII that returned empty protocol buffers when the file or connection could not be opened
    • distinguish between non-existant and not-set fieldswith has() by returning NULL in the former case.
    • fix a bug that caused non-deterministic behavior when setting a repeated message field in a protobuf to a single Message.
    • add unit tests for all of the above.
  • Added Murray to Authors: field in DESCRIPTION
  • Removed old and unconvincing example on RProtoBuf for storage and serialization in an imagined HighFrequencyFinance context
CRANberries also provides a diff to the previous release 0.2.5. More information is at the RProtoBuf page which has a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. Questions, comments etc should go to the rprotobuf mailing list off the RProtoBuf page at R-Forge.

1 October 2012

Dirk Eddelbuettel: Rcpp 0.9.14

Another release of Rcpp has just appeared on CRAN and was just uploaded to Debian. It addresses yet another issue we had on OS X and should hopefully put the build issues to rest. Three new (vectorized) sugar functions were added, along with some new regression tests and more. The complete NEWS entry for 0.9.14 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.
Changes in Rcpp version 0.9.14 (2012-09-30)
  • Added new Rcpp sugar functions trunc(), round() and signif(), as well as unit tests for them
  • Be more conservative about where we support clang++ and the inclusion of exception_defines.h and prevent this from being attempted on OS X where it failed for clang 3.1
  • Corrected a typo in Module.h which now again permits use of finalizers
  • Small correction for (unexported) bib() function (which provides a path to the bibtex file that ships with Rcpp)
  • Converted NEWS to NEWS.Rd
Thanks to CRANberries, you can also look at a diff to the previous release 0.9.13. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

26 September 2012

Dirk Eddelbuettel: RcppArmadillo 0.3.4.2

The development of Armadillo 3.4.* continues with bug fixes and more sparse matrix support. Conrad release 3.4.2 this morning. I wrapped up the corresponding RcppArmadillo 0.3.4.2 before leaving for work, and this version should now have all CRAN mirrors. Once again no R level or interface changes were, the upstream changes are summarized below.
Changes in RcppArmadillo version 0.3.4.2 (2012-09-25)
  • Upgraded to Armadillo release 3.4.2
    • minor fixes for handling sparse submatrix views
    • minor speedups for sparse matrices
Courtesy of CRANberries, there is also a diffstat report for 0.3.4.2 relative to 0.3.4.1 As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

18 September 2012

Dirk Eddelbuettel: RcppArmadillo 0.3.4.1

Conrad released the bug-fix release 3.4.1 of Armadillo earlier today, and the corresponding RcppArmadillo package 0.3.4.1 is already on CRAN. No R level or interface changes were, the upstream changes are summarized below.
Changes in RcppArmadillo version 0.3.4.1 (2012-09-18)
  • Upgraded to Armadillo release 3.4.1
    • workaround for a bug in the Mac OS X accelerate framework
    • fixes for handling empty sparse matrices
    • added documentation for saving & loading matrices in HDF5 format
    • faster dot() and cdot() for complex numbers
Courtesy of CRANberries, there is also a diffstat report for 0.3.4.1 relative to 0.3.4.0 As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

8 September 2012

Dirk Eddelbuettel: RInside 0.2.8

This morning version 0.2.8 of RInside arrived on the CRAN sites. RInside provides a set of convenience classes which facilitate embedding of R inside of C++ applications and programs, using the classes and functions provided by the Rcpp R and C++ integration package. This release adds no new features but improves the build process a little and should make use on Windows a little easier. All changes since the last release are summarized below in the NEWS file entry:
Changes in RInside version 0.2.8 (2012-09-07)
  • Added CMake build support for armadillo and eigen examples, once again kindly contributed by Peter Aberline
  • Corrected Windows package build to always generate a 64 bit static library too
  • Updated package build to no longer require configire and configure.win to update the two header file supplying compile-time information; tightened build dependencies on headers in Makevars / Makevars.win
  • Improved examples/standard/Makefile.win by detecting architecture
CRANberries also provides a short report with changes from the previous release. More information is on the RInside page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

7 September 2012

Dirk Eddelbuettel: RcppArmadillo 0.3.4.0

A new major released of Armadillo came out earlier today. I prepared the corresponding RcppArmadillo package 0.3.4.0 which also arrived on CRAN earlier today. This released contains a few performance improvements, the beginnings of support of sparse matrices and more, see below. We also post the NEWS entry for the beta release which was prepared, but not uploaded to CRAN to minimise the upload frequency there. On the RcppArmadillo side, two enhancements were made for the fastLm() function for faster linear model fits.
Changes in RcppArmadillo version 0.3.4.0 (2012-09-06)
  • Upgraded to Armadillo release 3.4.0 (Ku De Ta)
    • added economical QR decomposition: qr_econ()
    • added .each_col() & .each_row() for vector operations repeated on each column or row
    • added preliminary support for sparse matrices, contributed by Ryan Curtin et al. (Georgia Institute of Technology)
    • faster singular value decomposition via divide-and-conquer algorithm
    • faster .randn()
  • NEWS file converted to Rd format
Changes in RcppArmadillo version 0.3.3.91 (2012-08-30)
  • Upgraded to Armadillo release 3.3.91
    • faster singular value decomposition via "divide and conquer" algorithm
    • added economical QR decomposition: qr_econ()
    • added .each_col() & .each_row() for vector operations repeated on each column or row
    • added preliminary support for sparse matrices, contributed by Ryan Curtin, James Cline and Matthew Amidon (Georgia Institute of Technology)
  • Corrected summary method to deal with the no intercept case when using a formula; also display residual summary() statistics
  • Expanded unit tests for fastLm
Courtesy of CRANberries, there is also a diffstat report for 0.3.4.0 relative to 0.3.2.4 As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

2 September 2012

Dirk Eddelbuettel: Faster creation of binomial matrices

Scott Chamberlain blogged about faster creation of binomial matrices the other day, and even referred to our RcppArmadillo package as a possible solution (though claiming he didn't get it to work, tst tst -- that is what the rcpp-devel list is here to help with). The post also fell short of a good aggregated timing comparison for which we love the rbenchmark package. So in order to rectify this, and to see what we can do here with Rcpp, a quick post revisiting the issue. As preliminaries, we need to load three packages: inline to create compiled code on the fly (which, I should mention, is also used together with Rcpp by the Stan / RStan MCMC sampler which is creating some buzz this week), the compiler package included with R to create byte-compiled code and lastly the aforementioned rbenchmark package to do the timings. We also set row and column dimension, and set them a little higher than the original example to actually have something measurable:
library(inline)
library(compiler)
library(rbenchmark)
n <- 500
k <- 100
The first suggestion was the one by Scott himself. We will wrap this one, and all the following ones, in a function so that all approaches are comparable as being in a function of two dimension arguments:
scott <- function(N, K)  
    mm <- matrix(0, N, K)
    apply(mm, c(1, 2), function(x) sample(c(0, 1), 1))
 
scottComp <- cmpfun(scott)
We also immediatly compute a byte-compiled version (just because we now can) to see if this helps at all with the code. As there are no (explicit !) loops, we do not expect a big pickup. Scott's function works, but sweeps the sample() function across all rows and columns which is probably going to be (relatively) expensive. Next is the first improvement suggested to Scott which came from Ted Hart.
ted <- function(N, K)  
    matrix(rbinom(N * K, 1, 0.5), ncol = K, nrow = N)
 
This is quite a bit smarter as it vectorises the approach, generating N times K elements at once which are then reshaped into a matrix. Another suggestion came from David Smith as well as Rafael Maia. We rewrite it slightly to make it a function with two arguments for the desired dimensions:
david <- function(m, n)  
    matrix(sample(0:1, m * n, replace = TRUE), m, n)
 
This is very clever as it uses sample() over zero and one rather than making (expensive) draws from random number generator. Next we have a version from Luis Apiolaza:
luis <- function(m, n)  
     round(matrix(runif(m * n), m, n))
 
It draws from a random uniform and rounds to zero and one, rather than deploying the binomial. Then we have the version using RcppArmadillo hinted at by Scott, but with actual arguments and a correction for row/column dimensions. Thanks to inline we can write the C++ code as an R character string; inline takes care of everything and we end up with C++-based solution directly callable from R:
arma <- cxxfunction(signature(ns="integer", ks="integer"), plugin = "RcppArmadillo", body='
   int n = Rcpp::as<int>(ns);
   int k = Rcpp::as<int>(ks);
   return wrap(arma::randu(n, k));
')
This works, and is pretty fast. The only problem is that it answers the wrong question as it returns U(0,1) draws and not binomials. We need to truncate or round. So a corrected version is
armaFloor <- cxxfunction(signature(ns="integer", ks="integer"), plugin = "RcppArmadillo", body='
   int n = Rcpp::as<int>(ns);
   int k = Rcpp::as<int>(ks);
   return wrap(arma::floor(arma::randu(n, k) + 0.5));
')
which uses the the old rounding approximation of adding 1/2 before truncating. With Armadillo in the picture, we do wonder how Rcpp sugar would do. Rcpp sugar, described in one of the eight vignettes of the Rcpp package, is using template meta-programming to provide R-like expressiveness (aka "syntactic sugar") at the C++ level. In particular, it gives access to R's RNG functions using the exact same RNGs as R making the results directly substitutable (whereas Armadillo uses its own RNG).
sugar <- cxxfunction(signature(ns="integer", ks="integer"), plugin = "Rcpp", body='
   int n = Rcpp::as<int>(ns);
   int k = Rcpp::as<int>(ks);
   Rcpp::RNGScope tmp;
   Rcpp::NumericVector draws = Rcpp::runif(n*k);
   return Rcpp::NumericMatrix(n, k, draws.begin());
')
Here Rcpp::RNGScope deals with setting/resetting the R RNG state. This draws a vector of N time K uniforms similar to Luis' function -- and just like Luis' R function does so without looping -- and then shapes a matrix of dimension N by K from it. And it does of course have the same problem as the RcppArmadillo approach earlier and we can use the same solution:
sugarFloor <- cxxfunction(signature(ns="integer", ks="integer"), plugin = "Rcpp", body='
   int n = Rcpp::as<int>(ns);
   int k = Rcpp::as<int>(ks);
   Rcpp::RNGScope tmp;
   Rcpp::NumericVector draws = Rcpp::floor(Rcpp::runif(n*k)+0.5);
   return Rcpp::NumericMatrix(n, k, draws.begin());
')
Now that we have all the pieces in place, we can compare:
res <- benchmark(scott(n, k), scottComp(n,k),
                 ted(n, k), david(n, k), luis(n, k),
                 arma(n, k), sugar(n,k),
                 armaFloor(n, k), sugarFloor(n, k),
                 order="relative", replications=100)
print(res[,1:4])
With all the above code example in a small R script we call via littler, we get
edd@max:~/svn/rcpp/pkg$ r /tmp/scott.r 
Loading required package: methods
              test replications elapsed   relative
7      sugar(n, k)          100   0.072   1.000000
9 sugarFloor(n, k)          100   0.088   1.222222
6       arma(n, k)          100   0.126   1.750000
4      david(n, k)          100   0.136   1.888889
8  armaFloor(n, k)          100   0.138   1.916667
3        ted(n, k)          100   0.384   5.333333
5       luis(n, k)          100   0.410   5.694444
1      scott(n, k)          100  33.045 458.958333
2  scottComp(n, k)          100  33.767 468.986111
We can see several takeaways: Thanks to Scott and everybody for suggesting this interesting problem. Trying the rbinom() Rcpp sugar function, or implementing sample() at the C++ level is, as the saying goes, left as an exercise to the reader.

17 August 2012

Dirk Eddelbuettel: Follow-up to Counting CRAN Package Depends, Imports and LinkingTo

A few days ago, I blogged about visualizing CRAN dependency ranks which turned out to be a somewhat popular post. David Smith followed-up at the REvo blog suggesting to exclude packages already shipping with R (which is indicated by their 'Recommended' priority). Good idea! So here is an updated version, where we limit the display to the top twenty packages counted by reverse 'Depends:', and excluding those already shipping with R such as MASS, lattice, survival, Matrix, or nlme. CRAN package chart of Reverse Depends relations excluding Recommended packages The mvtnorm package is still out by a wide margin, but we can note that (cough, cough) our Rcpp package for seamless R and C++ is now tied for second with the coda package for MCMC analysis. Also of note is the fact that CRAN keeps growing relentlessly and moved from 3969 packages to 3981 packages in the space of these few days... Lastly, I have been asked about the code and/or data behind this. It is really pretty simply as the main data.frame can be had from CRAN (where I also found the initial few lines to load it). After that, one only needs a little bit of subsetting as shown below. I look forward to seeing other people riff on this data set.
#!/usr/bin/r
##
## Initial db downloand from http://developer.r-project.org/CRAN/Scripts/depends.R and adapted
require("tools")
## this function is essentially the same as R Core's from the URL
## http://developer.r-project.org/CRAN/Scripts/depends.R
getDB <- function()  
    contrib.url(getOption("repos")["CRAN"], "source") # trigger chooseCRANmirror() if required
    description <- sprintf("%s/web/packages/packages.rds", getOption("repos")["CRAN"])
    con <- if(substring(description, 1L, 7L) == "file://")  
        file(description, "rb")
      else  
        url(description, "rb")
     
    on.exit(close(con))
    db <- readRDS(gzcon(con))
    rownames(db) <- db[,"Package"]
    db
 
db <- getDB()
## count packages
getCounts <- function(db, col)  
    foo <- sapply(db[,col],
                  function(s)   if (is.na(s)) NA else length(strsplit(s, ",")[[1]])   )
 
## build a data.frame with the number of entries for reverse depends, reverse imports,
## reverse linkingto and reverse suggests; also keep Recommended status
ddall <- data.frame(pkg=db[,1],
                    RDepends=getCounts(db, "Reverse depends"),
                    RImports=getCounts(db, "Reverse imports"),
                    RLinkingTo=getCounts(db, "Reverse linking to"),
                    RSuggests=getCounts(db, "Reverse suggests"),
                    Recommended=db[,"Priority"]=="recommended"
                    )
## Subset to non-Recommended packages as in David Smith's follow-up post
dd <- subset(ddall, is.na(ddall[,"Recommended"])   ddall[,"Recommended"] != TRUE)
labeltxt <- paste("Analysis as of", format(Sys.Date(), "%d %b %Y"),
                  "covering", nrow(db), "total CRAN packages")
cutOff <- 20
doPNG <- TRUE
if (doPNG) png("/tmp/CRAN_ReverseDepends.png", width=600, heigh=600)
z <- dd[head(order(dd[,2], decreasing=TRUE), cutOff),c(1,2)]
dotchart(z[,2], labels=z[,1], cex=1, pch=19,
         main="CRAN Packages sorted by Reverse Depends:",
         sub=paste("Limited to top", cutOff, "packages, excluding 'Recommended' ones shipped with R"),
         xlab=labeltxt)
if (doPNG) dev.off()
if (doPNG) png("/tmp/CRAN_ReverseImports.png", width=600, heigh=600)
z <- dd[head(order(dd[,3], decreasing=TRUE), cutOff),c(1,3)]
dotchart(z[,2], labels=z[,1], cex=1, pch=19,
         main="CRAN Packages sorted by Reverse Imports:",
         sub=paste("Limited to top", cutOff, "packages, excluding 'Recommended' ones shipped with R"),
         xlab=labeltxt)
if (doPNG) dev.off()
# no cutOff but rather a na.omit
if (doPNG) png("/tmp/CRAN_ReverseLinkingTo.png", width=600, heigh=600)
z <- na.omit(dd[head(order(dd[,4], decreasing=TRUE), 30),c(1,4)])
dotchart(z[,2], labels=z[,1], pch=19,
         main="CRAN Packages sorted by Reverse LinkingTo:",
         xlab=labeltxt)
if (doPNG) dev.off()

14 August 2012

Dirk Eddelbuettel: RInside 0.2.7

A new version 0.2.7 of RInside is now available via CRAN. RInside provides a set of convenience classes which facilitate embedding of R inside of C++ applications and programs, using the classes and functions provided by the Rcpp R and C++ integration package. This release adds two new examples subdirectories demonstrating use of RInside with, respectively, RcppArmadillo and RcppEigen. We extended the 'web application' example using the Wt toolkit by adding CSS and XML support, and added another new example motivated by a StackOverflow question. CMake support has been added for Windows as well thanks to Peter Aberline---he also contributed CMake code for the two new example directories but that contribution made it only into SVN and not this release. All changes since the last release are summarized below in the NEWS file entry:
Changes in RInside version 0.2.7 (2012-08-12)
  • New fifth examples subdirectory 'armadillo' with two new examples showing how to combine RInside with RcppArmadillo
  • New sixth examples subdirectory 'eigen' with two new examples showing how to combine RInside with RcppEigen
  • Prettified the Wt example 'web application' with CSS use, also added and XML file with simple headers and description text
  • New example rinside_sample12 motivated by StackOverflow question on using sample() from C
  • Added CMake build support on Windows for the examples
CRANberries also provides a short report with changes from the previous release. More information is on the RInside page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

Next.

Previous.