Search Results: "francois"

17 January 2011

Dirk Eddelbuettel: Keeping simple things simple

My friend Jeff deserves a sincere congratulation for finally unveiling his rebranded R consultancy Lemnica. One notable feature of the new website is a section called esoteric R which discusses less frequently-visited corners of the R world. It even boasts its own CRAN package esotericR with the example sources. esoteric R currently holds two articles. Jeff had sent me the one about introducing closures a while back, and I like it and may comment at another time. What caught me by surprise when Lemnica finally opened was the other article: R calling C. It is a fine article motivated by all the usual reasons that are e.g. mentioned in the Google Tech Talk which Romain and I gave last October about our work around Rcpp. But it is just not simple. Allow me to explain. When Jeff showed this C language file
#include <R.h>
#include <Rinternals.h>
SEXP esoteric_rev (SEXP x)  
  SEXP res;
  int i, r, P=0;
  PROTECT(res = allocVector(REALSXP, length(x))); P++;
  for(i=length(x), r=0; i>0; i--, r++)  
     REAL(res)[r] = REAL(x)[i-1];
   
  copyMostAttrib(x, res);
  UNPROTECT(P);
  return res;
 
and then needs several paragraphs to explain what is going on, what is needed to compile and then how to load it --- I simply could not resist. Almost immediately, I emailed back to him something as simple as this using both our Rcpp package as well as the wonderful inline package by Oleg which Romain and I more or less adopted:
library(inline)  ## for cxxfunction()
src <- 'Rcpp::NumericVector x = Rcpp::NumericVector(xs);
        std::reverse(x.begin(), x.end());
        return(x);'
fun <- cxxfunction(signature(xs="numeric"), body=src, plugin="Rcpp")
fun( seq(0, 1, 0.1) )
Here we load inline, and then define a three-line C++ program using facilities from our Rcpp package. All we need to revert a vector is to first access its R object in C++ by instantiating the R vector as a NumericVector. These C++ classes then provide iterators which are compatible with the Standard Template Library (STL). So we simply call the STL function reverse pointing the beginning and end of the vector, and are done! Rcpp then allows us the return the C++ vector which it turns into an R vector. Efficient in-place reversal, just like Jeff had motivated, in three lines. Best of all, we can execute this from within R itself:
R> library(inline)  ## for cxxfunction()
R> src <- 'Rcpp::NumericVector x = Rcpp::NumericVector(xs);
+         std::reverse(x.begin(), x.end());
+         return(x);'
R> fun <- cxxfunction(signature(xs="numeric"), body=src, plugin="Rcpp")
R> fun( seq(0, 1, 0.1) )
 [1] 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0
R>
Lastly, Jeff shows a more complete example wherein a new vector is created, and any potential attributes are copied as well. Naturally, we can do that too. First, we used clone() to make a deep copy (ie forcing creation of a new object rather than a mere proxy) and use the same R API function he accessed---but it our case both prefixed with ::Rf_ for R remapping (to protect clashed with other functions with identical names) and a global namespace identifier (as it is a global C function from R).
R> library(inline)
R> src <- 'Rcpp::NumericVector x = Rcpp::clone<Rcpp::NumericVector>(xs);
+         std::reverse(x.begin(), x.end());
+         ::Rf_copyMostAttrib(xs, x);
+         return(x);'
R> fun <- cxxfunction(signature(xs="numeric"), body=src, plugin="Rcpp")
R> obj <- structure(seq(0, 1, 0.1), obligatory="hello, world!")
R> fun(obj)
 [1] 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0
attr(,"obligatory")
[1] "hello, world!"
R> obj
 [1] 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
attr(,"obligatory")
[1] "hello, world!"
R>
Both the obj variable and the new copy contain the desired data attribute, the new copy is reversed, the original is untouched---and all in four lines of C++ called via one inline call. I have now been going on for over one hundred lines yet I never had to mention memory management, pointers, PROTECT or other components of the R API for C. Hopefully, this short writeup provided an idea of why Romain and I think Rcpp is the way to go for creating C/C++ functions for extending and enhancing R.

12 January 2011

Julien Danjou: Emacs snapshot Debian packages

I've decided to take over the maintenance of the unofficial emacs-snapshot Debian packages that were maintained by Romain Francoise. They are available on a dedicated page. Flattr this

25 December 2010

Dirk Eddelbuettel: Rcpp 0.9.0 announcement

The text below went out as a post to the r-packages list a few days ago, but I thought it would make sense to post it on the blog too. So with a little html markup... Summary Version 0.9.0 of the Rcpp package is now on CRAN and its mirrors. This release marks another step in the development of the package, and a few key points are highlighted below. More details are in the NEWS and ChangeLog files included in the package. Overview Rcpp is an R package and associated C++ library that facilitates integration of C++ code in R packages. The package features a complete set of C++ classes (Rcpp::IntegerVector, Rcpp:NumericVector, Rcpp::Function, Rcpp::Environment, ...) that makes it easier to manipulate R objects of matching types (integer vectors, functions, environments, etc ...). Rcpp takes advantage of C++ language features such as the explicit constructor / destructor lifecycle of objects to manage garbage collection automatically and transparently. We believe this is a major improvement over use of PROTECT / UNPROTECT. When an Rcpp object is created, it protects the underlying SEXP so that the garbage collector does not attempt to reclaim the memory. This protection is withdrawn when the object goes out of scope. Moreover, users generally do not need to manage memory directly (via calls to new / delete or malloc / free) as this is done by the Rcpp classes or the corresponding STL containers. A few key points about Rcpp:

Several key features were added during the 0.8.* cycles and are described below. Rcpp sugar Rcpp now provides syntactic sugar: vectorised expressions at the C++ level which are motivated by the corresponding R expressions. This covers operators (binary arithmetic, binary logical, unary), functions (producing single logical results, mathematical functions and d/p/q/r statistical functions). Examples comprises anything from ifelse() to pmin()/pmax() or A really simply example is a function
    SEXP foo( SEXP xx, SEXP yy) 
        NumericVector x(xx), y(yy) ;
        return ifelse( x < y, x*x, -(y*y) ) ;
     
which deploys the sugar 'ifelse' function modeled after the corresponding R function. Another simple example is
    double square( double x) 
        return x*x ;
     
    SEXP foo( SEXP xx ) 
        NumericVector x(xx) ;
        return sapply( x, square ) ;
     
where use the sugar function 'sapply' to sweep a simple C++ function which operates elementwise across the supplied vector. The Rcpp-sugar vignette describes sugar in more detail. Rcpp modules Rcpp modules are inspired by Boost.Python and make exposing C++ functions or classes to R even easier. A first illustration is provided by this simple C++ code snippet
    const char* hello( const std::string& who ) 
        std::string result( "hello " ) ;
        result += who ;
        return result.c_str() ;
     
    RCPP_MODULE(yada) 
        using namespace Rcpp ;
        function( "hello", &hello ) ;
     
which (after compiling and loading) we can access in R as
    yada <- Module( "yada" )
    yada$hello( "world" )
In a similar way, C++ classes can be exposed very easily. Rcpp modules are also described in more detail in their own vignette. Reference Classes R release 2.12.0 introduced Reference Classes. These are formal S4 classes with the corresponding dispatch method, but passed by reference and easy to use. Reference Classes can also be exposed to R by using Rcpp modules. Extension packackages The RcppArmadillo package permits use of the advanced C++ library 'Armadillo, a C++ linear algebra library aiming towards a good balance between speed and ease of use, providing integer, floating point and complex matrices and vectors with lapack / blas support via R. Armadillo uses templates for a delayed evaluation approach is employed (during compile time) to combine several operations into one and reduce (or eliminate) the need for temporaries. Armadillo is useful if C++ has been decided as the language of choice, rather than another language like Matlab or Octave, and aims to be as expressive as the former. Via Rcpp and RcppArmadillo, R users now have easy access to this functionality. Examples are provided in the RcppArmadillo package. The RcppGSL package permits easy use of the GNU Scientific Library (GSL), a collection of numerical routines for scientifc computing. It is particularly useful for C and C++ programs as it provides a standard C interface to a wide range of mathematical routines such as special functions, permutations, combinations, fast fourier transforms, eigensystems, random numbers, quadrature, random distributions, quasi-random sequences, Monte Carlo integration, N-tuples, differential equations, simulated annealing, numerical differentiation, interpolation, series acceleration, Chebyshev approximations, root-finding, discrete Hankel transforms physical constants, basis splines and wavelets. There are over 1000 functions in total with an extensive test suite. The RcppGSL package provides an easy-to-use interface between GSL data structures and R using concepts from Rcpp. The RcppGSL package also contains a vignette with more documentation. Legacy 'classic' API Packages still using code interfacing the initial 'classic' Rcpp API are encouraged to migrate to the new API. Should a code transition not be possible, backwards compatibility is provided by the RcppClassic package released alongside Rcpp 0.9.0. By including RcppClassic.h and building against the RcppClassic package and library, vintage code can remain operational using the classic API. The short vignette in the RcppClassic package has more details. Documentation The package contains a total of eight vignettes the first of which provides a short and succinct introduction to the Rcpp package along with several motivating examples. Links Support Questions about Rcpp should be directed to the Rcpp-devel mailing list https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Dirk Eddelbuettel, Romain Francois, Doug Bates and John Chambers
December 2010

19 December 2010

Francois Marier: Peer-to-peer video-conferencing using free software

I was looking for a simple free software solution which would allow me to have a video call with someone else (I don't care about sound since I've already got that working through Asterisk) and I ended up writing a Gstreamer-based poor man's videoconf solution because I wasn't satisfied with the other options I considered.

EmpathyEmpathy was my first choice since it seems to be the preferred GNOME communication software nowadays.

While the quality of the video was excellent, the latency between New Zealand and Canada was unbearable: a full 6 seconds. I suspect that this is due to the fact that it runs everything through the Google Talk STUN server and I couldn't find how to force it to go directly from one host to the other.

EkigaEkiga was my second choice since I had used it succesfully in the past.

It was not too bad latency-wise, but the quality of the video was not as good as Empathy (it was smaller and choppier). Also, given that it was running over SIP, it was interfering with my VoIP phone.

Direct peer-to-peer streamingGiven that I wasn't gonna use the voice features of these video-conference tools, I figured that there must be an easy way to just stream video from one peer to the other. That's when I thought of looking into Gstreamer (apt-get install gstreamer0.10-tools on Debian/Ubuntu).

To stream video from my webcam onto port 5000, I ran:

gst-launch v4l2src device=/dev/video0 ! videorate ! video/x-raw-yuv,width=640,height=480,framerate=6/1 ! jpegenc quality=30 ! multipartmux ! tcpserversink port=5000

which is the best I could do within 85 kbps (100-120 kbps is about the maximum reliable synchronous bandwidth I get between New Zealand and Canada):On the other computer, I simply ran this to connect and display the remote stream:

gst-launch tcpclientsrc host=stream.example.com port=5000 ! multipartdemux ! jpegdec ! autovideosink

Then I swapped the roles around to also stream video the other way around. That's it: two-way peer-to-peer video link!

Small tweaks to the Gstreamer pipelineThere are quite a few plugins that can be used within Gstreamer pipelines.

If you have problems with autovideosink refusing to load (I did on one of the two computers), you can also install the gstreamer0.10-sdl package and replace autovideosink with sdlvideosink:

gst-launch tcpclientsrc host=example.com port=5000 ! multipartdemux ! jpegdec ! sdlvideosink

Another change I had to make on one of the machines was to flip the image coming out of the webcam (which insists on giving me a mirror image instead of acting like a real camera):

gst-launch v4l2src device=/dev/video0 ! videorate ! video/x-raw-yuv,width=640,height=480,framerate=6/1 ! videoflip method=horizontal-flip ! jpegenc quality=30 ! multipartmux ! tcpserversink port=5000

Possible improvementsI got down to about 1-2 seconds of latency, which isn't bad considering the processing to be done and the distance bits have to travel, but I would love to further reduce this.

Using jpegenc was a lot better than theoraenc which added an extra 3-4 seconds of latency. Is there a better codec I should be using?

Another thing I thought of trying was to switch from TCP to UDP. I'm currently using tcpserversink and tcpclientsrc but since I don't care about having a few dropped frames, maybe I should look into the udp and rtp plugins. It seems like it might help but it also seems to be quite a bit more complicated and I have yet to find an easy way to make use of the RTP stack in Gstreamer.

Please feel free leave a comment if you can suggest ways of improving my quick 'n dirty solution.

14 December 2010

Dirk Eddelbuettel: RcppDE 0.1.0

A new package RcppDE has been uploaded in a first version 0.1.0 to CRAN. It provides differential evolution optimisation---a variant of stochastic optimisation that is similar to genetic algorithms but particularly suitable for the floating-point representations common in numerical optimisation. It builds of on the nice DEoptim package by Ardia et al, but reimplements the algorithm in C++ (rather than C) using a large serving of Rcpp and RcppArmadillo. I worked on this on for a few evenings and weekends in October and November and then spent a few more evenings writing a paper / vignette (which is finished as a very first draft now) about it. This was an interesting and captivating problem as I had worked on genetic algorithms going back quite some time to the beginning and then again the end of graduate school (and traces of that early work are near the bottom of my presentations page). So what got me started? DEoptim is a really nice package, but it is implemented in old-school C. There is nothing wrong with that per se, but at the same time that I was wrestling with GAs, I also taught myself C++ which, to put it simply, offers a few more choices to the programmer. I like having those choices. And with all the work that Romain and I have put into Rcpp, I was curious how far I could push this cart if I were to move it along. I made a bet with myself starting from the old saw shorter, easier, faster: pick any two. Would it be possible to achieve all three of these goals? DEoptim, and I take version 2.0-7 as my reference point here, is pretty efficiently yet verbosely coded. Copying a vector takes a loop with an assignment for each element, copying a matrix does the same using two loops. Replacing that with a single statement in C++ is pretty easy. We also have a few little optimisations behind the scenes here and there in Rcpp: would all that be enough to move the needle in terms of performance? And the same time, DEoptim is also full of the uses of the old R API which we often point to in the Rcpp documentation so fixing readibility should be a relatively low-hanging fruit. To cut a long story short, I was able to reduce code size quite easily by using a combination of C++ and Rcpp idioms. I was also able to get to faster: the paper / vignette demostrates consistent speed improvements on all setups that I tested (three standard functions on three small and three larger parameter vectors). More important speed gains were achieved by allowing use of objective functions that are written in C++ which again is both possible and easy thanks to Rcpp. That leaves easier to prove: adding compiled objective functions is one indication; further proof could be provided by, say, moving the inner loop to parallel execution thanks to Open MP which I may attempt over the next few months. So far I'd like to give myself about half a point here. So not quite yet shorter, easier, faster: pick any three, but working on it. Over the next few days I may try to follow up with a blog post or two contrasting some code examples and maybe showing a chart from the vignette.

8 December 2010

Dirk Eddelbuettel: inline 0.3.8

Romain pushed verion 0.3.8 of inline to CRAN earlier today, and I just updated the Debian package. This version adds an internal performance enhancement which is obtained by making due with fewer reads. The short NEWS file entry follows:
0.3.8   2010-12-07
    o   faster cfunction and cxxfunction by loading and resolving the routine
        at "compile" time

1 December 2010

Dirk Eddelbuettel: RcppGSL 0.1.0

Earlier in the year, Romain and I did a bunch of initial work on a wrapper from R to the GNU GSL by way of our Rcpp package for seamless R and C++ integration. But other work kept us busy and this fell a little to the side. We have now found some time to finish this work for a first release, together with a nicely detailed eleven page package vignette. As of today, the package is now a CRAN package, and Romain already posted a nice announcement on his blog and on the rcpp-devel list. So what does RcppGSL do? I gave the package its own webpage here as well and listed these points as key features of RcppGSL: Also provided is a simple example which is a simple implementation of a column norm (which we could easily compute directly in R, but we are simply re-using an example from Section 8.4.14 of the GSL manual):
#include <RcppGSL.h>
#include <gsl/gsl_matrix.h>
#include <gsl/gsl_blas.h>
extern "C" SEXP colNorm(SEXP sM)  
  try  
        RcppGSL::matrix<double> M = sM;     // create gsl data structures from SEXP
        int k = M.ncol();
        Rcpp::NumericVector n(k);           // to store results
        for (int j = 0; j < k; j++)  
            RcppGSL::vector_view<double> colview = gsl_matrix_column (M, j);
            n[j] = gsl_blas_dnrm2(colview);
         
        M.free() ;
        return n;                           // return vector
    catch( std::exception &ex )  
        forward_exception_to_r( ex );
    catch(...)  
        ::Rf_error( "c++ exception (unknown reason)" );
   
  return R_NilValue; // -Wall
 
This example function is implemented in an example package contained in the RcppGSL package itself -- so that users have a complete stanza to use in their packages. This will then build a user package on Linux, OS X and Windows provided the GSL is installed (and on Windows you have to do all the extra steps of defining an environment variable pointing to and of course install Rtools to build in the first place---Linux and OS X are so much easier for development). Another complete example is in the package itself and provides a faster (compiled) alternative to the standard lm() function in R; this example is the continuation of the same example I had in several versions of my Intro to HPC with R tutorials and in the Rcpp package itself as an early example. We will try to touch base with CRAN package authors using both GSL and Rcpp to see how this can help them. The API in our package may well be incomplete, but we are always happy to try to respond to requests for additional features brought to our attention, preferably via the rcpp-devel list. More information is on the RcppGSL page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

3 November 2010

Dirk Eddelbuettel: Rcpp 0.8.8

A bug-fix release 0.8.8 of Rcpp is now available. It is awaiting processing at CRAN, and will be uploaded to Debian once processed at CRAN. In the meantime, sources are available from my local directory here. This release follows on the heels of 0.8.7, but contains fixes for a few small things Romain and I had noticed over the last two weeks since releasing 0.8.7 and contains only a small number of new tweaks. The NEWS entry follows below:
0.8.8   2010-11-01
    o   New syntactic shortcut to extract rows and columns of a Matrix. 
        x(i,_) extracts the i-th row and x(_,i) extracts the i-th column. 
    
    o   Matrix indexing is more efficient. However, faster indexing is
        disabled if g++ 4.5.0 or later is used.
    o   A few new Rcpp operators such as cumsum, operator=(sugar)
    o   Variety of bug fixes:
        - column indexing was incorrect in some cases
        - compilation using clang/llvm (thanks to Karl Millar for the patch)
        - instantation order of Module corrected
        - POSIXct, POSIXt now correctly ordered for R 2.12.0 
As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

2 November 2010

Francois Marier: RAID1 alternative for SSD drives

I recently added a solid-state drive to my desktop computer to take advantage of the performance boost rumored to come with these drives. For reliability reasons, I've always tried to use software RAID1 to avoid having to reinstall my machine from backups should a hard drive fail. While this strategy is fairly cheap with regular hard drives, it's not really workable with SSD drives which are still an order of magnitude more expensive.

The strategy I settled on is this one:This setup has the benefit of using a very small SSD to speed up the main partition while keeping all important data on the larger mirrored drives.

Resetting the SSDThe first thing I did, given that I purchased a second-hand drive, was to completely erase the drive and mark all sectors as empty using an ATA secure erase. Because SSDs have a tendency to get slower as data is added to them, it is necessary to clear the drive in a way that will let the controller know that every byte is now free to be used again.

There is a lot of advice on the web on how to do this and many tutorials refer to an old piece of software called Secure Erase. There is a much better solution on Linux: issuing the commands directly using hdparm.

Partitioning the SSDOnce the drive is empty, it's time to create partitions on it. I'm not sure how important it is to align the partitions to the SSD erase block size on newer drives, but I decided to follow Ted Ts'o's instructions anyways.

Another thing I did is leave 20% of the drive unpartitioned. I've often read that SSDs are faster the more free space they have so I figured that limiting myself to 80% of the drive should help the drive maintain its peak performance over time. In fact, I've heard that extra unused unpartitionable space is one of the main differences between the value and extreme series of Intel SSDs. I'd love to see an official confirmation of this from Intel of course!

Keeping the RAID1 array in sync with the SSDOnce I added the solid-state drive to my computer and copied my root partition on it, I adjusted my fstab and grub settings to boot from that drive. I also setup the following cron job (running twice daily) to keep a copy of my root partition on the old RAID1 drives (mounted on /mnt):
nice ionice -c3 rsync -aHx --delete --exclude=/proc/* --exclude=/sys/* --exclude=/tmp/* --exclude=/home/* --exclude=/mnt/* --exclude=/lost+found/* --exclude=/data/* /* /mnt/

Tuning the SSDFinally, after reading this excellent LWN article, I decided to tune the SSD drive (/dev/sda) by adjusting three things:


Is there anything else I should be doing to make sure I get the most out of my SSD?

28 October 2010

Dirk Eddelbuettel: Google Tech Talk on Integrating R and C++: video and slides

Last Friday, Romain and I were guests of the R intergrouplet (what an adorable name!) at Google's headquarter in Mountain View. This arose out of discussions following useR! 2010 where we met Google's Murray Stokely. There appears to be ever increasing use of R at Google, and so it was a great opportunity to give a Google Tech Talk about R and C++ integration --- centered around our Rcpp, RInside and RProtoBuf packages which facilitate interoperability between R and C++. A video recording of our ninety-minute talk is already available via the YouTube channel for Google Tech Talks. The (large) pdf with slides (which Romain had already posted on slideshare) is also available from my presentations page. The remainder of the weekend was nice too (with the notably exception of the extremly sucky weather). We got to to spend some time at the Google Summer of Code Mentor Summit which is always a fun event and a great way to meet other open source folks in person. And we also took one afternoon off to spend some with John Chambers discussing further work involving Rcpp and the new ReferenceClasses that appeared in the just-released R version 2.12.0. This should be a nice avenue to further integrate R and C++ in the near future.

18 October 2010

Francois Marier: Manipulating debconf settings on the command line

It's not very easy to find information on how to adjust debconf settings after a package has been installed and configured. Most of the information out there is for Debian developers wanting to add support for debconf in their maintainer scripts.

I ran into the problem of being unable to change a package's configuration options through dpkg-reconfigure and I found the following commands to do it manually:

debconf-show packagename

to show the list of debconf values that a package has stored,

echo "get packagename/pgsql/app-pass" debconf-communicate

to query the current value of an option in the debconf database, and

echo "set packagename/pgsql/app-pass password1" debconf-communicate

to change that value.

I'm not convinced that this is the easiest way for system administrators to manually lookup and modify debconf options, but that's the best I could find at the time.

12 September 2010

Alastair McKinstry: Exoclimes: the diversity of planetary scientists

I'm just back from ExoClimes 2010: Exploring the Diversity of Planetary Atmospheres. An excellent conference: the PDFs of the talks and posters are now online, and they are putting the videos of the talks up soon. But in particular the organizers deserves thanks for bringing exoplanetary scientists and observers together with climate modelers doing Earth (and Mars, Titan, Venus, ...) models.
Model complexity graph Peter Cox on model complexity
The last talk on Friday was by Peter Cox on Climate change and exoplanet sciences that was far better than expected for the "graveyard shift". One theme of the conference was the need for a 'heirarchy' of models, from simple energy-balance models to full circulation (GCM) models: using progressively more complex models to understand more bits of whats going on. Exoplanet workers mostly use simpler models, progressing now to GCMs, while Earth modellers are moving beyond GCMs to "Earth system" models including biology, etc. Peter pointed out the two styles of work: the exoplanet modelers are short of data, and risk being too speculative. We know little of what the planets are like, and concentrate on implementing physics in the models to see what they might be like. Earth modelers on the other hand are if anything swamped with data: the tendency here is to make the model fit the data, by adjusting parameters until it does so. The danger of this approach is that the model will then not work away from current present-Earth conditions. Tim Lenton pointed out some work that was done with the Met Office model, where they took the radiative transfer part of the model and tested it for other planets, and paleo-Earth conditions. The model blew up : it wasn't capable of x2 or x4 current CO2 levels. (This has since been corrected). Over dinner there were interesting discussions on the different styles within the communities. While the underlying GCMs used come from the Earth sciences, its quite common within the exoplanetary community for a researcher to work on all parts of the model: dynamics one day, radiative transfer the next. In Earth climate work people have become more specialized and someone is a 'radiative transfer' person, and won't touch other parts of the code (even if they can follow them in the huge codes we have today!). On the other hand, there is a greater tradition of model inter-comparison in Earth sciences, where we compare the model outputs to each other for some known test cases ( Held & Suarez, the CMIP5 project, etc.) Apart from some initial work by Emily Rauscher, little has been done on this in exoplanetary models; it was agreed more of this would be a good idea. Radiative transfer (the interaction of 'sunlight' with the atmosphere, where it gets absorbed, scattered and re-radiated) in particular seems to be an area that could benefit from this. In this middle ground Francois Forget showed the work on the LMDZ model and applying GCMs to terrestrial planets. They've successfully applied this model to Mars, Titan, and partially to Venus (a much tougher problem, due to its heavy clouds giving a long radiative timescale). There are problems with correctly explaining super-rotation though. This is where the atmosphere rotates faster than the planet: on Venus for example the planet rotates every 243 days, while the clouds rotate around the planet every 4 days. Sebastian Lebonnois described the possible mechanisms for Venus and Titan; Johnathan Mitchell so did some interesting work on this recently. Different regimes are involved for different rotation rates of the planet. Ralph Lorenz pointed out the lack of "real paleo-Earth" climate work at the moment. While geology has inspired a lot of work on the atmospheric composition, what with the different gas mixtures (meaning earth-model radiative transfer codes don't work) and the faster dynamics meaning super-rotation could apply (Earth's day was about 8 hours long in the Archean era), we don't have a model of the climate yet. It looks like we should treat Earth as an exoplanet. Tags , , ,

Francois Marier: Setting up your own DNSSEC-aware resolver using Unbound

Now that the root DNS servers are signed, I thought it was time I started using DNSSEC on my own PC. However, not wanting to wait for my ISP to enable it, I decided to setup a private recursive DNS resolver for myself using Unbound.

Installing UnboundBeing already packaged in Debian and Ubuntu, unbound is only an apt-get away:
apt-get install unbound
though if you are running lenny, I suggest you grab the latest backport.

Once unbound is installed, follow these instructions to enable DNSSEC.

Optional settingsIn my /etc/unbound/unbound.conf, I enabled the following security options:
harden-referral-path: yes
use-caps-for-id: yes
and turned on prefetching to hopefully keep in cache the sites I visit regularly:
prefetch: yes
prefetch-key: yes
Finally, I also enabled statistics:
extended-statistics: yes
control-enable: yes
control-interface: 127.0.0.1
and ran sudo unbound-control-setup to generate the necessary keys.

Once unbound is restarted (sudo /etc/init.d/unbound restart) stats can be queried to make sure that the DNS resolver is working:
unbound-control stats

Overriding DHCP settingsIn order to use my own unbound server for DNS lookups and not the one received via DHCP, I added this line to /etc/dhcp/dhclient.conf:
supersede domain-name-servers 127.0.0.1;
and restarted dhclient:
sudo killall dhclient
sudo killall dhclient
sudo /etc/init.d/network-manager restart
If you're not using DHCP, then you simply need to put this in your /etc/resolv.conf:
nameserver 127.0.0.1

Testing DNSSEC resolutionOnce everything is configured properly, the best way I found to test that this setup was actually working is to use a web browser to visit these sites:
and using dig:
$ dig +dnssec A www.dnssec.cz   grep ad
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 3, ADDITIONAL: 1

Are there any other ways of making sure that DNSSEC is fully functional?

28 August 2010

Romain Francoise: An update on md5sums, and Debian's growth

Back in August 2007 I looked at the state of embedded md5sums in Debian packages and found that approximately 3% of the files in the archive didn't have checksums. Three years later, things have improved: only 0.76% of the archive is now missing checksums (sid, main/contrib/non-free). (See this lintian report for the list of affected packages.) Since then there's also been various discussions on this subject and there is now a policy bug open to make md5sums a requirement (at the "should" level). There is also a wishlist bug against debhelper to turn dh_md5sums into dh_checksums with a stronger hash algorithm, but MD5 still being good enough for simple integrity checking, it seems rather pointless to upgrade the algorithm without a trust path in the form of in-package signatures ala RPM... Anyway, what's perhaps more surprising is the growth of the distribution in only three years: sid has gone from 20774 to 30314 packages, a 45% increase. Similarly, the number of regular files has gone from approximately 2 million to just above 2.9 million. Indeed, looking at our last five releases, the distribution's growth is impressive: Whether or not that is a good thing is, of course, yet to be determined. As a data point, I used the UDD to know how many of these thousands of packages are actually used, and to my surprise, 22321 binary packages have a popcon installation count that is less than 500! (By comparison, dpkg's installation count is 89393.) So while each new release adds lots of packages, the majority of them have very few users. (If you want to check yourself, the query I used is select p.package, version, insts from packages p, popcon where (p.architecture = 'i386' or p.architecture = 'all') and p.release = 'squeeze' and p.package = popcon.package and popcon.insts < 500 order by insts;.)

24 August 2010

Francois Marier: Combining multiple commits into one using git rebase

git rebase provides a simple way of combining multiple commits into a single one. However using rebase to squash an entire branch down to a single commit is not completely straightforward.

Squashing normal commitsUsing the following repository:
$ git log --oneline
c172641 Fix second file
24f5ad2 Another file
97c9d7d Add first file
we can combine the last two commits (c172641 and 24f5ad2) by rebasing up to the first commit:
$ git rebase -i 97c9d7d
and specify the following commands in the interactive rebase screen:
pick 24f5ad2 Another file
squash c172641 Fix second file
which will rewrite the history into this:
$ git log --oneline
1a9d5e4 Another file
97c9d7d Add first file

Rebasing the initial commitTrying to include the initial commit in the interactive rebase screen will return this error:
$ git rebase -i 97c9d7d^
fatal: Needed a single revision
Invalid base
and squashing the top commit in the interactive rebase screen:
$ git rebase -i 97c9d7d

squash 24f5ad2 Another file
squash c172641 Fix second file
will return this error:
Cannot 'squash' without a previous commit
So we need to use a different approach to deal with the initial commit.

Amending the initial commitHere is an alternative to rebase which will work on commits that don't have a parent.

Taking the previously rebased branch:
$ git log --oneline
1a9d5e4 Another file
97c9d7d Add first file
we can rewind the branch to the initial commit:
$ git reset 97c9d7d
$ git log --oneline
97c9d7d Add first file
without losing any of the changes introduced in 1a9d5e4 (shown here as uncommitted changes):
$ git status
# On branch master
# Changed but not updated:
# (use "git add ..." to update what will be committed)
# (use "git checkout -- ..." to discard changes in working directory)
#
# modified: file1
#
# Untracked files:
# (use "git add ..." to include in what will be committed)
#
# file2
no changes added to commit (use "git add" and/or "git commit -a")
Then we can reopen commit 97c9d7d and add the changes present in the working directory:
$ git add .
$ gitc -a --amend -m "Initial version"
which will finally give us a fully squashed branch:
$ git log --oneline
fcb85fb Initial version

$ git status
# On branch master
nothing to commit (working directory clean)

15 August 2010

Kai Hendry: Debian powered Web applications

Blars dining Francois Freeing the Cloud talk at Debconf10, showed members of the Debian community keen to help see opensource Web applications. The proposed open&free implementation of Gravatar is a great example, whereby a user uploads an image of him/herself which corresponds to a hash of their email address. In order to make this Web "cloud" service rock, super quick downloads of that image are needed. Debian knows about mirroring data to distribute data amongst free software agents. And we also now know about CDN, aka content delivery networks and GEO DNS, which point to your local mirror. So lets pool our resources and offer CDN Web space (100MB?) to free software Web application projects. This will make the opensource Web services competitive in all important page load times and perhaps make it easier for user contributed data to be exported and "open". I have Debian server space in Germany I'm happy to contribute to Francois's avatar project and other similar opensource Web apps. If you think this is a good idea, please get in touch.

3 August 2010

Dirk Eddelbuettel: inline 0.3.6

A couple of days ago, Romain released inline release 0.3.6 to CRAN. This is a maintenance release with no user-visible changes. However, as it captures compiler errors more directly, it should help us debug Rcpp on recalcitrant platforms such as Solaris with suncc where we have no shell access and no build robot (though that may be changing with the rumoured bin-builder). More details on the release at Romain's blog.

24 July 2010

Dirk Eddelbuettel: useR 2010 at NIST in Gaithersburg

This past week, the annual R user conference useR! 2010 took place at the National Institute of Standards and Technology (NIST) in Gaithersburg, MD (which is a tad northwest of Washington, DC). Kate Mullen and her team of local organizers did a truly tremendous job in putting together a very smooth conference attended by almost 500 people. It is always nice to meet so many other R contributors and users in person. And needless to say it's also just plain fun to hang out with these folks. As at the preceding useR! 2008 in Dortmund and useR! 2009 in Rennes, I presented a three-hour tutorial on high-performance computing with R. This covers scripting/automation, profiling, vectorisation, interfacing compiled code, parallel computing and large-memory approaches. The slides, as well as a condensed 2-up version, are now on my presentations page. On Wednesday, Romain and I had a chance to talk about recent work on Rcpp, our R and C++ integration. Thursday, we followed up with a presentation on RProtoBuf -- a project integrating Google's Protocol Buffers with R which much to our delight already seems to be in use at Google itself! It was quite fun to do these two talks jointly with Romain. But my other coauthor Khanh had to be at a conference related to his actual PhD work. So on Friday it was just me to give a presentation about RQuantLib which brings QuantLib to R. Slides from all these talks have now been added to my presentations page. I will also upload them via the conference form so that they can be part of the conference's collection of presentations which should be forthcoming.

20 July 2010

Francois Marier: Cherry-picking a range of git commits

The cherry-pick command in git allows you to copy commits from one branch to another, one commit at a time. In order to copy more than one commit at once, you need a different approach.

Cherry-picking a single commitSay we have the following repository composed of three branches (master, feature1 and stable):

$ git tree --all
* d9484311 (HEAD, master) Delete test file
* 4d4a0da8 Add a test file
* 5753515c (stable) Add a license
* 4b95278e Add readme file
/
* a37658bd (feature1) Add fourth file
* a7785c10 Add lines to 3rd file
* 7f545188 Add third file
* 2bca593b Add line to second file
* 0c13e436 Add second file
/
* d3199755 Add a line
* b58d925c Initial commit

The "git tree" command is an alias I defined in my ~/.gitconfig:

[alias]
tree = log --oneline --decorate --graph

To copy the license file (commit 5753515c) to the master branch then we simply need to run:

$ git checkout master
$ git cherry-pick 5753515c
Finished one cherry-pick.
[master 08ff7d4] Add a license
1 files changed, 676 insertions(+), 0 deletions(-)
create mode 100644 COPYING

and the repository now looks like this:

$ git tree --all
* 08ff7d4a4 (HEAD, master) Add a license
* d94843113 Delete test file
* 4d4a0da88 Add a test file
* 5753515c (stable) Add a license
* 4b95278e Add readme file
/
* a37658bd (feature1) Add fourth file
* a7785c10 Add lines to 3rd file
* 7f545188 Add third file
* 2bca593b Add line to second file
* 0c13e436 Add second file
/
* d3199755 Add a line
* b58d925c Initial commit

Cherry-picking a range of commits
In order to only take the third file (commits a7785c10 and 7f545188) from the feature1 branch and add it to the stable branch, I could cherry-pick each commit separately, but there is a faster way if you need to cherry-pick a large range of commits.

First of all, let's create a new branch which ends on the last commit we want to cherry-pick:

$ git branch tempbranch a7785c10
$ git tree --all
* 08ff7d4a (HEAD, master) Add a license
* d9484311 Delete test file
* 4d4a0da8 Add a test file
* 5753515c (stable) Add a license
* 4b95278e Add readme file
/
* a37658bd (feature1) Add fourth file
* a7785c10 (tempbranch) Add lines to 3rd file
* 7f545188 Add third file
* 2bca593b Add line to second file
* 0c13e436 Add second file
/
* d3199755 Add a line
* b58d925c Initial commit

Now we'll rebase that temporary branch on top of the stable branch:

$ git rebase --onto stable 7f545188^ tempbranch
First, rewinding head to replay your work on top of it...
Applying: Add third file
Applying: Add lines to 3rd file
$ git tree --all
* ec488677 (HEAD, tempbranch) Add lines to 3rd file
* a85e5281 Add third file
* 5753515c (stable) Add a license
* 4b95278e Add readme file
* 08ff7d4a (master) Add a license
* d9484311 Delete test file
* 4d4a0da8 Add a test file
/
* a37658bd (feature1) Add fourth file
* a7785c10 Add lines to 3rd file
* 7f545188 Add third file
* 2bca593b Add line to second file
* 0c13e436 Add second file
/
* d3199755 Add a line
* b58d925c Initial commit

All that's left to do is to make stable point to the top commit of tempbranch and delete the old branch:

$ git checkout stable
Switched to branch 'stable'
$ git reset --hard tempbranch
HEAD is now at ec48867 Add lines to 3rd file
$ git tree --all
* ec488677 (HEAD, tempbranch, stable) Add lines to 3rd file
* a85e5281 Add third file
* 5753515c Add a license
* 4b95278e Add readme file
* 08ff7d4a (master) Add a license
* d9484311 Delete test file
* 4d4a0da8 Add a test file
/
* a37658bd (feature1) Add fourth file
* a7785c10 Add lines to 3rd file
* 7f545188 Add third file
* 2bca593b Add line to second file
* 0c13e436 Add second file
/
* d3199755 Add a line
* b58d925c Initial commit
$ git branch -d tempbranch
Deleted branch tempbranch (was ec48867).

It would be nice to be able to do it without having to use a temporary branch, but it still beats cherry-picking everything manually.

Another approachAnother way to achieve this is to use the format-patch command to output patches for the commits you are interested in copying to another branch and then using the am command to apply them all to the target branch:

$ git format-patch 7f545188^..a7785c10
0001-Add-third-file.patch
0002-Add-lines-to-3rd-file.patch
$ git am *.patch

Update: looking forward to git 1.7.2According to a few people who were nice to point this out in a comment, version 1.7.2 of git, which is going to be released soon, will have support for this in cherry-pick:

git cherry-pick 7f545188^..a7785c10

16 July 2010

Dirk Eddelbuettel: Rcpp 0.8.4

Romain and I wrapped up release 0.8.4 of Rcpp last Friday. However, given the time of year, it only appeared on CRAN this morning, and then only after some prodding as CRAN processing is more or less closed this week and probably next. This release builds upon release 0.8.3. Highlights include changes to the sugar framework for highly expressive C++ constructs which gained new vector function as well as a first set of matrix function. As well, unit tests have been reorganised in such a way that we end up with a lot fewer compilations (but of several files at once) which reaps significant speed gains. Date calculation now use the same mktime() function R itself uses (and which comes from Arthur Olson's tzone library). The NEWS entry follows below:
0.8.4   2010-07-09
    o   new sugar vector functions: rep, rep_len, rep_each, rev, head, tail,
        diag
        
    o   sugar has been extended to matrices: The Matrix class now extends the 
        Matrix_Base template that implements CRTP. Currently sugar functions 
        for matrices are: outer, col, row, lower_tri, upper_tri, diag
    o   The unit tests have been reorganised into fewer files with one call
        each to cxxfunction() (covering multiple tests) resulting in a
        significant speedup
    o   The Date class now uses the same mktime() replacement that R uses
        (based on original code from the timezone library by Arthur Olson)
        permitting wide dates ranges on all operating systems
    o   The FastLM example has been updated, a new benchmark based on the
        historical Longley data set has been added
    o   RcppStringVector now uses std::vector<std::string> internally
    
    o   setting the .Data slot of S4 objects did not work properly
As always, even fuller details are in Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Next.

Previous.