This blog, together with
Romain's, is one of the main homes of stories about how
Rcpp can help with getting code to run faster in the context of the
R system for statistical programming and analysis. By making it easier to get already existing C or C++ code
to R, or equally to extend R with new C++ code,
Rcpp can help in
getting stuff done.
And it is often fairly straightforward to do so.
In this context, I have a nice new example. And for once, it is work-related. I generally cannot share too much of what we do there as this is, well,
proprietary, but I have this nice new example. The other day, I was constructing (large) time series of implied volatilities. Implied volatilities
can be thought of as the complement to an option's price: given a price (and all other observables which can be thought of as fixed), we compute an
implied volatility price (typically via the standard
Black-Scholes model).
Given a changed implied volatility, we infer a new price -- see
this
Wikipedia page for more details. In essence, it opens the door to all sorts of arbitrage and relative value pricing adventures.
Now, we observe prices fairly frequently to create somewhat sizeable time series of option prices. And each price corresponds to one matching implied
volatility, and for each such price we have to solve a small and straightforward optimization problem: to compute the implied volatility given the
price. This is usually done with an iterative root finder.
The problem comes from the fact that we have to do this (i) over and over and over for large data sets, and (ii) that there are a number of callbacks
from the (generic) solver to the (standard) option pricer.
So our first approach was to just call the corresponding function
GBSVolatility from the
fOption package from the trusted
Rmetrics project by Diethelm Wuertz et al. This worked fine, but even with the usual tricks of splitting over
multiple cores/machines, it simply took too long for the resolution and data amount we desired. One of the problems is that this function (which
uses the proper
uniroot optimizer in R) is not inefficient per se, but simply makes to many function call back to the option pricer as
can be seen from a quick glance at the code. The helper function
.fGBSVolatility gets called time and time again:
R> GBSVolatility
function (price, TypeFlag = c("c", "p"), S, X, Time, r, b, tol = .Machine$double.eps,
maxiter = 10000)
TypeFlag = TypeFlag[1]
volatility = uniroot(.fGBSVolatility, interval = c(-10, 10),
price = price, TypeFlag = TypeFlag, S = S, X = X, Time = Time,
r = r, b = b, tol = tol, maxiter = maxiter)$root
volatility
<environment: namespace:fOptions>
R>
R> .fGBSVolatility
function (x, price, TypeFlag, S, X, Time, r, b, ...)
GBS = GBSOption(TypeFlag = TypeFlag, S = S, X = X, Time = Time,
r = r, b = b, sigma = x)@price
price - GBS
<environment: namespace:fOptions>
So the next idea was to try the corresponding function from my
RQuantLib package which
brings (parts of)
QuantLib to R. That was seen as been lots faster already. Now, QuantLib is pretty big and so
is RQuantLib, and we felt it may not make sense to install it on a number of machines just for this simple problem. So one evening this week I
noodled around for an hour or two and combined (i) a basic Black/Scholes calculation and (ii) a standard univariate zero finder (both of which can be
found or described in numerous places) to minimize the difference between the observed price and the price given an implied volatility. With about
one hundred lines in C++, I had something which felt fast enough. So today I hooked this into R via a two-line wrapper in quickly-created package using
Rcpp.
I had one more advantage here. For our time series problem, the majority of the parameters (strike, time to maturity, rate, ...) are fixed, so we can
structure the problem to be vectorised right from the start. I cannot share the code or more the details of my new implementation. However, both
GBSVolatility and
EuropeanOprionImpliedVolatility are on CRAN (and as I happen to maintain these for
Debian, also just one
sudo apt-get install r-cran-foptions r-cran-rquantlib away if you're
on
Debian or
Ubuntu). And writing the other solver is really not that
involved.
Anyway, here is the result, courtesy of a quick run via the
rbenchmark package. We create a vector of length
500; the implied volatility computation will be performed at each point (and yes, our time series are much longer indeed). This is replicated 100
times (as is the default for
rbenchmark) for each of the three approaches:
xyz@xxxxxxxx:~$ r xxxxR/packages/xxxxOptions/demo/timing.R
test replications elapsed relative user.self sys.self user.child sys.child
3 zzz(X) 100 0.038 1.000 0.040 0.000 0 0
2 RQL(X) 100 3.657 96.237 3.596 0.060 0 0
1 fOp(X) 100 448.060 11791.053 446.644 1.436 0 0
xyz@xxxxxxxx:~$
The new local solution is denoted by
zzz(X). It is already orders of magnitude faster than the
RQL(x) function using
RQuantLib (which is, I presume, due to my custom solution internalising the loop). And
the new approach is a laughable amount faster than the basic approach (shown as
fOp) via
fOptions. For one hundred replications of solving implied volatilities for
all elements of a vector of size 500, the slow solution takes about 7.5 minutes --- while the fast solution takes 38 milliseconds. Which comes to a
relative gain of over 11,000.
So sitting down with your C++ compiler to craft a quick one-hundred lines, combining two well-known and tested methods, can reap sizeable benefits. And
Rcpp makes it trivial to call this from
R.
My friend Martijn from 
I ve been a wannabe GTD afficionado for some years. I ve wanted to do it,
but managing lists has always been something that has too much friction,
overhead, or whatever. Finally, though, I think I might have found a way to
manage lists that works.
My use-case isn t unique, although I will concede I m perhaps being more
dogmatic than most. I want something that:
Ever since 
I've had a few weeks with very little time spent on Debian. I did set aside
some time to try to find and fix an RC bug, to help the release. Steven C
commented to let me know about the 
A few days ago, 
Synaps
Synaps is written in around 7k lines of Python (with 28 % of which are
comments), reuses at least one common module of OpenStack
(openstack.common.cfg) and copy some modules from Nova. One thing that
strikes me, is that there seems to be only a few unit tests compared to most
OpenStack projects. Also, many parts of the code and documentation contains
text written in korean, which won't be very helpful for most people! :-)
It uses some external technologies, like 
Overlap with Heat
For now, there's not a lot of overlap with Heat, because Heat does not
implement completely the CloudWatch API. Heat actually still misses a lot of
the CloudWatch functions. But as soon as it will implement the CloudWatch
API completely, the overlap will be complete with Synaps in this regard.
One divergence point however, is that Heat uses RPC to access data from the
storage back-end via its engine (the central daemon), whereas Synaps
directly connects to Cassandra. Also, Heat relies on SQLAlchemy, like most
OpenStack projects needing a database.
Overlap with Ceilometer
One of the goal of Ceilometer is to provide data probes and pollsters for
all OpenStack components (Nova, Swift, Quantum ) whereas Synaps let the
OpenStack users to put any kind of metric inside it, and therefore doesn't
provide anything for now.
But the storage of metrics is the main common point between Synaps and
Ceilometer. Synaps chose only one technology, Cassandra, to store its
metrics, whereas Ceilometer took care of building an abstraction layer for
the storage engine. Ceilometer currently allows an operator to use SQL or
MongoDB, but Cassandra could likely be added.
Data metric consolidation is done by Synaps. This makes sense, since Synaps
don't need to have the full data history to trigger alarms. On the opposite,
Ceilometer needs to have a full history to allow things like billing, and
don't do any aggregation on data.
Also, in Synaps, the data analysis is done using Pandas. This means the data
used are retrieved from the Cassandra back-end, and then transformed by
Pandas inside Synaps in something else. It's likely that in such a case,
Synaps should use CQL to achieve that. Ceilometer manipulates the data near
their storage: it means that the computation are done by back-end to be
efficient (SQL, mapreduce ).
Conclusion
Considering Samsung open-sourced Synaps late in the development process, I
don't feel like they aimed to have it becoming a core component. This is
always sad, because the effort put into this implementation are big and it
would have probably little to add some abstraction layers to follow what
other OpenStack projects do. But this takes time and energy, and it's
understandable that Samsung didn't want to achieve this in a short time
frame.
There's a part of the code and architecture that overlaps with Ceilometer
and Heat. Ceilometer is becoming a specialized point to store data metrics
from any source: so it's sad, but understandable, that Synaps did not tried
to reuse it. Fortunately, Heat is working with Ceilometer to achieve exactly
that. This means OpenStack would have only one metrics storage point, used
for billing, for monitoring and alarming.
Therefore, I think Synaps is an implementation of CloudWatch that should be
looked at as an inspiration for Heat and Ceilometer to build a better and
more integrated solution!
Christopher 
It's been 
That was not my target in today's marathon (my second one in the
year). The target was breaking my 3h38'45" record, set on June 10th in
Caen.
And, well, I did it. By more than 5 minutes \o/. Of course, the
To experiment with network stuff, I was using
I recently read this book and it was enlightening -- that is the least I can say. It is organized into a set of recipes addressing the most common problems you have with puppet. I think the whole point is about being practical and it gives you various ways to achieve the same goal, depending if you want to do a quick and dirty hack or pick a long term solution. The whole book aims at being practical and it achieves this goal pretty well. However, it is not for beginners. Puppet itself is not for beginner and you should already have written and run into problem to really take advantage of this book.
I read the book in 5 days (only while commuting), and at each page I thought : "OMG, they do that like this" or "of course, you should use git + that" and wait impatiently coming back home to test all this stuff. The book is really about technical details and how to organize yourself to write nice Puppet classes. I have been an "irregular" user of Puppet since 4 years. I was mainly running it to distribute files on all my computers. Since 3 weeks, I am applying the recipes found in this book and it is just like discovering that you own something worth a million euros.
Let look at some example:
