Request Tracker (also know as RT), the defacto Open Source ticket tracking software, has a special status value to get rid of spam tickets: deleted. It does exactly what it's supposed to do, that is get rid of these tickets quickly.

Unfortunately, it's very easy to accidentally delete a bunch of non-spam tickets via the bulk update interface. Restoring these deleted tickets is not straightforward.

Good news: none of the lost tickets were removed from the database.
Bad news: you can't search for deleted tickets.

Finding the lost ticketsIf you have a copy of the page before you did the bulk update (perhaps in your browser cache?) then you've got all you need. Otherwise, you'll need to access the database directly to get a list of ticket IDs.

First of all, find the user ID of the person who marked of all these tickets as deleted (42 in this example), then issue the following query:

 SELECT id, lastupdated, subject
 FROM tickets
WHERE status = 'deleted' AND
      lastupdatedby = 42
ORDER BY lastupdated;

You may want to filter some more using the lastupdated timestamp if you know roughly when the deletion occurred.

Restoring deleted ticketsSince all of the tickets are still in the database, all you need to do is to go to each of them directly (e.g. http://www.example.com/Ticket/Display.html?id=12345 for Ticket #12345) and change the status back to "open".

Request Tracker is a really neat support tool, but one of the common complaints I heard from people using it during a previous project was that it was pretty slow.

There wasn't much we could do about the (overloaded) server it was running on, but I found that enabling mod_deflate really helped.

After watching this great video though, I was inspired to look into it a bit more, focussing this time on latency.

Description of tests

• RT 3.6.7-5+lenny4
• Running on a Debian Lenny vserser.
• Server is on the LAN.
• Firefox 3.6.6 client (with Firebug 1.5.0)

Also note that I was looking for the "best case" for each of the different configurations and so each screenshot was taken after reloading the homepage 10-20 times to maximize cache hits (thanks in large part to mod_expires).

(Is there a nice automated way of measuring average latency?)

Stock RT 3.6Using the default apache2-modperl2 config file (as supplied by RT), here's what the homepage (logged in as root) looked like before I changed anything:



The purple section here indicates the time spent waiting for the server. This shows that the server (running Mason inside mod_perl) is doing quite a bit of processing, including a lot more work than you'd expect while serving static files. It's quite impressive to see how fast the images are being served (directly by Apache) in comparison with the Javascript and the CSS files (which go through Mason).

The reason while Javascript and CSS files have to be served by mod_perl is that they are in fact templates. They contain a few Mason variables which must be substituted before being served.

Looking into it further though, all of these replacements have to do with variables defined in RT_SiteConfig.pm (mostly the install path). Here's an example:

var path = "" ? "" : "/";

which gets turned into:

var path = "/rt" ? "/rt" : "/";

So as long as these paths don't change, then there is no need to re-generate these files.

Static Javascript and CSSThis next diagram was produced after configuring Apache to serve all Javascript and CSS files directly from Apache:



The way I did that (without modifying any of the original files) was by saving the Javascript/CSS sent to the browser and using mod_rewrite rules to serve these files instead of the original templated ones:

# Serve static files directly
RewriteRule ^/usr/share/request-tracker3.6/html/NoAuth/js/ahah.js$ /var/www/rt/ahah.js
RewriteRule ^/usr/share/request-tracker3.6/html/NoAuth/js/cascaded.js$ /var/www/rt/cascaded.js
RewriteRule ^/usr/share/request-tracker3.6/html/NoAuth/js/class.js$ /var/www/rt/class.js
RewriteRule ^/usr/share/request-tracker3.6/html/NoAuth/js/combobox.js$ /var/www/rt/combobox.js
RewriteRule ^/usr/share/request-tracker3.6/html/NoAuth/js/list.js$ /var/www/rt/list.js
RewriteRule ^/usr/share/request-tracker3.6/html/NoAuth/js/titlebox-state.js$ /var/www/rt/titlebox-state.js
RewriteRule ^/usr/share/request-tracker3.6/html/NoAuth/js/util.js$ /var/www/rt/util.js
RewriteRule ^/usr/share/request-tracker3.6/html/NoAuth/css/3.5-default/main-squished.css$ /var/www/rt/main-squished.css
RewriteRule ^/usr/share/request-tracker3.6/html/NoAuth/css/print.css$ /var/www/rt/print.css
RewriteRule ^/usr/share/request-tracker3.6/html/NoAuth/webrtfm.css$ /var/www/rt/webrtfm.css

Removing unnecessary images

Finally, one thing I noticed from this last graph is that the rounded corners in the theme require a number of small images. While these don't take a whole lot of bandwidth, they do require quite a bit of back and forth between the browser and the server.

So I replaced all of the "rounded corner" images in the main-squished.css file with the following CSS attributes:

-moz-border-radius-topleft: 8px;
-moz-border-radius-topright: 8px;
-moz-border-radius-bottomleft: 8px;
-moz-border-radius-bottomright: 8px;
-webkit-border-top-left-radius: 8px;
-webkit-border-top-right-radius: 8px;
-webkit-border-bottom-left-radius: 8px;
-webkit-border-bottom-right-radius: 8px;

(yes, Internet Explorer users probably don't get the rounded corners... oh well)

This eliminated a number of roundtrips and shaved off a few more milliseconds:



Merging CSS and Javascript filesBy this stage, the pages are pretty snappy so there is not much to be gained anymore, but I figured I'd try to reduce the latency a bit more by combining all Javascript files into one (and doing the same for CSS files with the exception of print.css). This is what I got:



(Note that I also took the opportunity to minify both squished files to reduce the filesize.)

Not a huge improvement and I unfortunately had to copy quite a few Mason templates from /usr/share/request-tracker3.6/html/ to /usr/local/share/request-tracker3.6/html/ and then replace all of the script tags with a single one in html/Elements/Header.

Others things to look intoI've stopped here, but there might be ways to further reduce the processing time on the server (hence the latency) by tuning mod_perl/Mason or Postgres. The RT wiki also has a few pointers.

Replacing Apache with Nginx (which means moving to FastCGI) was something I considered, but after trying it out, it turned out that it would add about 100ms of extra latency.

Feel free to leave a comment if you've found something else that makes a big difference on your site.

If you have removed a file (or part of a file) from git, it's not immediately obvious how to query its history. Here are two ways to deal with deleted content in git.

Commit history of a deleted fileIf we take the following two files:

$ ls
file1  file2

and then decide to delete one of them:

$ git rm file2
rm 'file2'
$ git commit -m "Delete a file"
[deletefile 87fadb9] Delete a file
1 files changed, 0 insertions(+), 1 deletions(-)
delete mode 100644 file2

To see the commit history of that file, you can't do it the usual way:

$ git log file2
fatal: ambiguous argument 'file2': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions

Instead, you need to do this:

$ git log -- file2

Finding the commit that deleted a lineFinding the commit that deleted a line is slightly more complicated. Unfortunately, we can't really use git blame for that. All we can do with git blame is to find the last commit which contained the deleted line.

So if we add the following file:

$ cat file3
one
two
three
$ git add file3
$ git commit -a -m "Add a third file"
[master e62ace6] Add a third file
1 files changed, 3 insertions(+), 0 deletions(-)
create mode 100644 file3

and remove the second line:

$ cat file3
one
three
$ git commit -a -m "Remove a line"
[removeline f3eb691] Remove a line
1 files changed, 0 insertions(+), 1 deletions(-)

then we can use git blame see what was the last revision to contain each line:

$ git blame --reverse HEAD^..HEAD file3
f3eb691d (Francois 2010-07-04  1) one
^e62ace6 (Francois 2010-07-04  2) two
f3eb691d (Francois 2010-07-04  3) three

Finding the commit that deleted that file requires using git log to search for the text contained on that deleted line:

$ git log --oneline -S'two' file3
f3eb691 Remove a line
e62ace6 Add a third file

There are lots of excellent tools to help web developers optimise their websites.

Here are two simple things you have no excuse for overlooking on your next project.
HTML, XML, Javascript and CSS filesOne of the easiest ways to speed up a website (often to a surprising degree) is to turn on compression of plaintext content through facilities like mod_deflate or the Gzip Module.

Here's the Apache configuration file I normally use:

<ifmodule>AddOutputFilterByType DEFLATE text/html text/plain text/css text/javascript text/xml application/x-javascript application/javascript
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch "MSIE 6" no-gzip dont-vary
BrowserMatch ^Mozilla/4\.0[678] no-gzip</ifmodule>

ImagesAs far as images go, the following tools will reduce file sizes through lossless compression (i.e. with no visual changes at all):

• gifsicle -O2 -b image.gif
• jpegoptim -p --strip-all image.jpg
• optipng -o7 -q image.png

(An alternative to optipng is pngcrush.)

Note that the --strip-all argument to jpegoptim will remove any EXIF/comments tags that may be present.

After dropping and restoring a very large database a few times just to refresh the data in a single table, I thought that there must be an easier way to do this. I was right, you can restore a single table.

If you are starting with a live database, you can simply use pg_dump to backup only one table:

pg_dump --data-only --table=tablename sourcedb > onetable.pg

which you can then restore in the other database:

psql destdb < onetable.pg

But even if all you've got is a full dump of the source database, you can still restore that single table by simply extracting it out of the large dump first:

pg_restore --data-only --table=tablename fulldump.pg > onetable.pg

before restoring it as shown above, using psql.

Early this morning I sent Rcpp version 0.8.1 off to CRAN and Debian. In the meantime, Romain has already provided a very nice blog post about it. There are a few fairly visible new things in this release. As we want to focus the next few minor releases on completing the documentation, we started by adding a total of four (!!) new vignettes:

• Rcpp-package showing how to write your own package using Rcpp,
• Rcpp-FAQ addressing several frequently asked questions,
• Rcpp-modules discussing how to expose C++ functions and modules with ease using an idea borrowed from Boost::Python, and
• Rcpp-extending detailing the steps needed to extend Rcpp with user-provided or third-party classes,

The most interesting new feature is what we call Rcpp modules and is modeled after Boost::Python. This makes it pretty easy to expose C++ functions and classes to R -- without having to write glue code. This is pretty new and may change a tad over the coming releases, but it is also quite exciting. Other changes concern more improvements for use of inline which should now allow packages like our RcppArmadillo to be used with it, and some bug fixes. The full NEWS entry for this release follows below:

0.8.1   2010-06-08
    o   This release adds Rcpp modules. An Rcpp module is a collection of
        internal (C++) functions and classes that are exposed to R. This
        functionality has been inspired by Boost.Python.
        
        Modules are created internally using the RCPP_MODULE macro and
        retrieved in the R side with the Module function. This is a preview 
        release of the module functionality, which will keep improving until
        the Rcpp 0.9.0 release. 
        The new vignette "Rcpp-modules" documents the current feature set of
        Rcpp modules.
        
    o   The new vignette "Rcpp-package" details the steps involved in making a
        package that uses Rcpp.
    o   The new vignette "Rcpp-FAQ" collects a number of frequently asked
        questions and answers about Rcpp.
    o   The new vignette "Rcpp-extending" documents how to extend Rcpp
        with user defined types or types from third party libraries. Based on
        our experience with RcppArmadillo
        
    o   Rcpp.package.skeleton has been improved to generate a package using 
        an Rcpp module, controlled by the "module" argument
    o   Evaluating a call inside an environment did not work properly
        
    o   cppfunction has been withdrawn since the introduction of the more
        flexible cxxfunction in the inline package (0.3.5). Rcpp no longer
        depends on inline since many uses of Rcpp do not require inline at
        all. We still use inline for unit tests but this is now handled
        locally in the unit tests loader runTests.R. 
        Users of the now-withdrawn function cppfunction can redefine it as:
        
           cppfunction <- function(...) cxxfunction( ..., plugin = "Rcpp" )
    o   Support for std::complex was incomplete and has been enhanced.
    o   The methods XPtr<t>::getTag and XPtr<t>::getProtected are deprecated, 
        and will be removed in Rcpp 0.8.2. The methods tag() and prot() should
        be used instead. tag() and prot() support both LHS and RHS use. 
    o   END_RCPP now returns the R Nil values; new macro VOID_END_RCPP
        replicates prior behabiour
        

As always, even fuller details are in Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Yesterday morning, Romain pushed inline release 0.3.5 to CRAN. This is some ways a continuation of the 0.3.4 release I had made in December. That release had opened the door for the wide use of inline in our Rcpp package. And just how Rcpp has grown, we now have needs beyond the initial change. See the post on Romain's blog for details, but in a nutshell we are now gaining

• cxxfunction which extends cfunction further for C++ use and, among other things, adds a plugin system we can use from RcppArmadillo to permit use of inline
• package.skeleton which makes it easy to carry a function that one has prototyped with inline over into its own package -- and how to do that was a question at my most recent Rcpp talk in Vienna), and
• getDynLib which Romain will use to great effect in the next version of Rcpp to provide something not unlike Boost::Python. Stay tuned!

Last but not least, our thanks to Oleg Sklyar for letting us extend his amazing inline package for use by Rcpp.

Romain and I are happy to announce the release of Rcpp version 0.8.0. It has been uploaded to CRAN. A Debian upload is delayed until the now-required inline package is accepted into Debian. The source package is also available from here. This release brings a number of changes that are detailed below. Of particular interest may be the much more robust treatment of exceptions, the new classes for data frames and formulae, and the availability of the new helper function cppfunction for use with inline. Also of note is the new support for the 'LinkingTo' directive with which packages using Rcpp will get automatic access to the header files. An announcement email went to the r-packages list (ETH Zuerich, Gmane); Romain also blogged about the release. The full NEWS entry for this release follows below:

0.8.0   2010-05-17
    o   All Rcpp headers have been moved to the inst/include directory,
	allowing use of 'LinkingTo: Rcpp'. But the Makevars and Makevars.win
	are still needed to link against the user library.
    o   Automatic exception forwarding has been withdrawn because of
	portability issues (as it did not work on the Windows platform).
	Exception forwarding is still possible but is now based on explicit
	code of the form:
	
	  try  
	    // user code
	    catch( std::exception& __ex__)  
	    forward_exception_to_r( __ex___ ) ;
	   
	
	Alternatively, the macro BEGIN_RCPP and END_RCPP can use used to enclose
	code so that it captures exceptions and forward them to R. 
	
	  BEGIN_RCPP
	  // user code
	  END_RCPP
	
    o   new __experimental__ macros 
	
	The macros RCPP_FUNCTION_0, ..., RCPP_FUNCTION_65 to help creating C++
	functions hiding some code repetition: 
	
	  RCPP_FUNCTION_2( int, foobar, int x, int y) 
	    return x + y ;
	   
	The first argument is the output type, the second argument is the 
	name of the function, and the other arguments are arguments of the C++ 
	function. Behind the scenes, the RCPP_FUNCTION_2 macro creates 
	an intermediate function compatible with the .Call interface and handles 
	exceptions
	
	Similarly, the macros RCPP_FUNCTION_VOID_0, ..., RCPP_FUNCTION_VOID_65 
	can be used when the C++ function to create returns void. The generated
	R function will return R_NilValue in this case.
	
	  RCPP_FUNCTION_VOID_2( foobar, std::string foo ) 
	    // do something with foo
	   
	  
	The macro RCPP_XP_FIELD_GET generates a .Call compatible function that
	can be used to access the value of a field of a class handled by an 
	external pointer. For example with a class like this: 
	
	  class Foo 
		public:
			int bar ;
	   
	
	  RCPP_XP_FIELD_GET( Foo_bar_get, Foo, bar ) ;
	  
	RCPP_XP_FIELD_GET will generate the .Call compatible function called
	Foo_bar_get that can be used to retrieved the value of bar.
	
	
	The macro RCPP_FIELD_SET generates a .Call compatible function that 
	can be used to set the value of a field. For example:
	
	  RCPP_XP_FIELD_SET( Foo_bar_set, Foo, bar ) ;
	
	generates the .Call compatible function called "Foo_bar_set" that 
	can be used to set the value of bar
	
	
	The macro RCPP_XP_FIELD generates both getter and setter. For example
	
	  RCPP_XP_FIELD( Foo_bar, Foo, bar )
	  
	generates the .Call compatible Foo_bar_get and Foo_bar_set using the 
	macros RCPP_XP_FIELD_GET and RCPP_XP_FIELD_SET previously described
	
	  
	The macros RCPP_XP_METHOD_0, ..., RCPP_XP_METHOD_65 faciliate 
	calling a method of an object that is stored in an external pointer. For 
	example: 
	
	  RCPP_XP_METHOD_0( foobar, std::vector<int> , size )
	
	creates the .Call compatible function called foobar that calls the
	size method of the std::vector<int> class. This uses the Rcpp::XPtr<
	std::vector<int> > class.
	
	The macros RCPP_XP_METHOD_CAST_0, ... is similar but the result of
	the method called is first passed to another function before being
	wrapped to a SEXP.  For example, if one wanted the result as a double
	
	   RCPP_XP_METHOD_CAST_0( foobar, std::vector<int> , size, double )
	
	The macros RCPP_XP_METHOD_VOID_0, ... are used when calling the
	method is only used for its side effect.
	
	  RCPP_XP_METHOD_VOID_1( foobar, std::vector<int>, push_back ) 
	
	Assuming xp is an external pointer to a std::vector<int>, this could
	be called like this :
	
	  .Call( "foobar", xp, 2L )
	
    o	Rcpp now depends on inline (>= 0.3.4)
	
    o   A new R function "cppfunction" was added which invokes cfunction from
	inline with focus on Rcpp usage (enforcing .Call, adding the Rcpp
	namespace, set up exception forwarding). cppfunction uses BEGIN_RCPP
	and END_RCPP macros to enclose the user code
    o	new class Rcpp::Formula to help building formulae in C++
    
    o   new class Rcpp::DataFrame to help building data frames in C++
	
    o   Rcpp.package.skeleton gains an argument "example_code" and can now be
	used with an empty list, so that only the skeleton is generated. It
	has also been reworked to show how to use LinkingTo: Rcpp
	
    o   wrap now supports containers of the following types: long, long double,
	unsigned long, short and unsigned short which are silently converted
	to the most acceptable R type.
	
    o	Revert to not double-quote protecting the path on Windows as this
	breaks backticks expansion used n Makevars.win etc
        
    o   Exceptions classes have been moved out of Rcpp classes,
	e.g. Rcpp::RObject::not_a_matrix is now Rcpp::not_a_matrix

As always, even fuller details are in Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Conference Management systems are web applications designed to make the lives of conference organisers easier. They usually include features such as registration, payment, paper submission and review, scheduling and publishing of announcements.

Since Wikipedia is apparently not a place for such a "spammy list," I figured I should at least post this here:

Name	Used by	License	Programming Language
A Conference Toolkit	YAPC::Eu	Artistic License	Perl
ConMan	Utah Open Source Conference, Texax Linux Fest	GPL	Python (Django)
Open Conference Systems	various academic conferences	GPL	PHP
OpenConferenceWare	Open Source Bridge	MIT	Ruby on Rails
Pentabarf	DebConf, FOSDEM, CCC	GPL	Ruby on Rails
SCALEreg	SCALE	GPL	Python (Django)
Zookeepr	linux.conf.au	GPL	Python (Pylons)

If you know of any other Open Source systems, please leave a comment!

I recently bought a Belkin 600VA UPS (model F6S600auUSB) and here's what I had to do to setup the monitoring and automatic shutdown on Ubuntu 9.10 (karmic). (This procedure should work on recent versions of Debian as well.)

This UPS comes with a proprietary monitoring tool (written in Java) and you can find instructions to get this working on the Ubuntu forums, but I was looking for a free solution that would integrate well with the rest of the system. So after reading this blog post I decided to go with the Network UPS Tools project:

$ apt-get install nut

Once the nut package is installed, I edited /etc/nut/nut.conf to set:

MODE=standalone

and created the following files:

$ vim /nut/ups.conf

[belkinusb]
driver = megatec_usb
port = auto
desc = "Belkin UPS, USB interface"

$ vim /etc/nut/upsd.conf

# MAXAGE 15
# LISTEN 127.0.0.1 3493
# MAXCONN 1024

$ vim /etc/nut/upsd.users

[local_mon]
password = MYPASSWORD
upsmon master

$ vim /etc/nut/upsmon.conf

MONITOR belkinusb@localhost 1 local_mon MYPASSWORD master
POWERDOWNFLAG /etc/killpower
SHUTDOWNCMD "/sbin/shutdown -h +0"


Then all that was left to do was to restart nut:

$ /etc/init.d/nut restart

and check syslog for any errors:

$ tail /var/log/syslog

While nut is running, it will monitor the UPS and report any power problems to syslog. Once the UPS is running on battery, it will make sure the computer is safely shut down before power runs out.

Hopefully future versions of GNOME Power Manager will be able to integrate with nut directly and display battery information through its notification icon.

We spent last week in the LA area and had a generally good time out west. I was able to sneak in two talks and a group discussion, thanks to the help by Jan de Leeuw (and everybody at UCLA's Stats department) as well as by Szilard Pafka representing the LA R User's Group. Pdf files for the slides for the talks are now on my presentations page in both a compact handout and presentation slide version (where the content is identical; if in doubt use the first file). The talks centered around R and C++ integration using both Rcpp and RInside and summarise where both projects stand after all the recent work Romain and I put in over the last few months. The presentations went fairly well; I received some favourable comments. Szilard and the R User Group had also suggested a group discussion about CRAN, its growth and how to maximise its usefulness. Given my CRANberries feed, my work on the CRAN Task Views for Empirical Finance and High-Performance Computing with R as well as our cran2deb binary package generator, I had some views and ideas that helped frame the discussion which turned out to very useful and informed. So maybe we should do this User Group thing in Chicago too! Special thanks to Jan de Leeuw and Szilard Pafka for organising the meeting, talks and discussion.

A git repository is primarily intended to store multiple branches of a single program or component. The underlying system however is much more flexible. Here are two ways to add files which are related to the project but outside the normal history.
Storing a single file all by itselfThis intermediate trick allows you to store a single file into a git repository without the file being in any of the branches.

First of all, let's create a new file:

echo "Hi" > message.txt

and store it in git:

git hash-object -w message.txt
b14df6442ea5a1b382985a6549b85d435376c351

At this stage, the file is stored within the git repository but there is no other way to get to it other than using the hash:

git cat-file blob b14df6442ea5a1b382985a6549b85d435376c351
Hi

A good way to point to the stored contents is to use a tag:

git tag message.txt b14df6442ea5a1b382985a6549b85d435376c351

Now you can access your file this way:

git cat-file blob message.txt
Hi

Creating an empty branchAs seen in this git screencast or in the Git Community Book, you can create a branch without a parent or any initial contents:

git symbolic-ref HEAD refs/heads/foo
rm .git/index
git clean -fdx

To finish it off, add what you were planning on storing there and commit it:

git add myfile
git commit -m 'Initial commit'


The shiny new 0.2.2 release of RInside has just been uploaded to CRAN; it should hit mirrors tommorow. Sources are also at my RInside page. RInside is a set of convenience classes to facilitate embedding of R inside of C++ applications. It works particularly well with Rcpp and now depends on it. This is the first release since version 0.2.1 in early January. Romain and I made numerous changes to Rcpp in the meantime. With this release, RInside is starting to catch up by taking advantage of many new automatic (templated) type converters. We have updated the existing examples, and added several new ones. These are all visibile directly via the Doxygen-generated documentation under the Files heading. Two examples are also shown directly on the RInside page. Also added are new examples showing how to use RInside to embed R inside C++ applications using MPI for parallel computing. This was contributed via two examples files by Jianping Hua, and we reworked the examples slightly (and added two variants that use MPI's C++ API). As it is so short, here is the basic 'Hello, World' example now showing the simpler Rcpp-based variable assignment:

// -*- mode: C++; c-indent-level: 4; c-basic-offset: 4;  tab-width: 8; -*-
//
// Simple example showing how to do the standard 'hello, world' using embedded R
//
// Copyright (C) 2009 Dirk Eddelbuettel
// Copyright (C) 2010 Dirk Eddelbuettel and Romain Francois
//
// GPL'ed
#include <RInside.h>                    // for the embedded R via RInside

int main(int argc, char *argv[])  
    RInside R(argc, argv);              // create an embedded R instance
    R["txt"] = "Hello, world!\n";	// assign a char* (string) to 'txt'
    R.parseEvalQ("cat(txt)");           // eval the init string, ignoring any returns
    exit(0);
 

One minor setback is that the examples currently segfault on Windows. That may be an issue with linking and class instantiation or something related. Romain and I focus much more on Linux and OS X, so this has not gotten a lot of attention. Debugging help would be appreciated.

Besides the new RcppExamples, another new package RcppArmadillo got spun out of Rcpp with the recent release 0.7.8 of Rcpp. Romain and I already had an example of a simple but fast linear model fit using the (very clever) Armadillo C++ library by Conrad Sanderson. In fact, I had used this as a motivational example of why Rcpp rocks in a recent talk to the ACM chapter at U of Chicago which, thanks to David Smith at REvo, got some further exposure. Now this example is more refined as further glue got added. Given that both Armadillo and Rcpp make use of C++ templates, the actual amount of code in RcppArmadillo is not that large: just over 200 lines in a header file, and a little less for some testing accessor and example functions in a source file. And this makes for some really nice example code: the 'fast regression' example becomes this (where I simply removed two blocks with conditional on the Armadillo version):

#include <RcppArmadillo.h>
extern "C" SEXP fastLm(SEXP ys, SEXP Xs)  
    Rcpp::NumericVector yr(ys);			// creates Rcpp vector from SEXP
    Rcpp::NumericMatrix Xr(Xs);			// creates Rcpp matrix from SEXP
    int n = Xr.nrow(), k = Xr.ncol();
    arma::mat X(Xr.begin(), n, k, false);   	// reuses memory and avoids extra copy
    arma::colvec y(yr.begin(), yr.size(), false);
    arma::colvec coef = solve(X, y);            // fit model y ~ X
    arma::colvec resid = y - X*coef; 		// residuals
    double sig2 = arma::as_scalar( trans(resid)*resid/(n-k) );
    						// std.error of estimate
    arma::colvec stderrest = sqrt( sig2 * diagvec( arma::inv(arma::trans(X)*X)) );
    Rcpp::Pairlist res(Rcpp::Named( "coefficients", coef),
                       Rcpp::Named( "stderr", stderrest));
    return res;
 

No extra copies! Armadillo instantiates directly from the underlying R objects for the vector and matrix, solves the regression equations, computes the standard error of the estimates and returns the two vectors. Leaving us to write about eleven lines of code. Moreover, as Armadillo is well designed and uses template meta-programming to avoid extra copies (see these lecture notes for details), it is about as efficient as it can be (and will use Atlas or other BLAS where available). And, this is just one example. Rcpp should be suitable for other C++ libraries, and provides an easy to use seamless interface between C++ and R. However, we should note that (at about the last minute) we found out about some unit test failures in OS X as well as some issues in a Debian chroot -- cran2deb ran into some build issues on i386 and amd64 in the testing chroot even this 'it all works' swimmingly on our Debian, Ubuntu and Fedora build environments. A follow-up with fixes for either Rcpp and/or RcppArmadillo appears likely.

Update: The build issues seems to be with 64-bit systems and everything appears cool in 32-bit.

Version 0.1.0 of RcppExamples, a simple demo package for Rcpp should appear on CRAN some time tomorrow. As mentioned in the post about release 0.7.8 of Rcpp, Romain and I carved this out of Rcpp itself to provide a cleaner separation of code that implements our R / C++ interfaces (which remain in Rcpp) and code that illustrates how to use it --- which is now in RcppExamples. This also provides an easier template for people wanting to use Rcpp in their packages as it will be easier to wrap one's head around the much smaller RcppExamples package. A simple example (using the newer API) may illustrate this:

#include <Rcpp.h>
RcppExport SEXP newRcppVectorExample(SEXP vector)  
    Rcpp::NumericVector orig(vector);			// keep a copy (as the classic version does)
    Rcpp::NumericVector vec(orig.size());		// create a target vector of the same size
    // we could query size via
    //   int n = vec.size();
    // and loop over the vector, but using the STL is so much nicer
    // so we use a STL transform() algorithm on each element
    std::transform(orig.begin(), orig.end(), vec.begin(), sqrt);
    Rcpp::Pairlist res(Rcpp::Named( "result", vec),
                       Rcpp::Named( "original", orig));
    return res;
 

With essentially five lines of code, we provide a function that takes any numeric vector and returns both the original vector and a tranformed version---here by applying a square root operation. Even the looping along the vector is implicit thanks to the generic programming idioms of the Standard Template Library. Nicer still, even on misuse, exceptions get caught cleanly and we get returned to the R prompt without any explicit coding on the part of the user:

R> library(RcppExamples)
Loading required package: Rcpp
R> print(RcppVectorExample( 1:5, "new" )) # select new API
$result
[1] 1.000 1.414 1.732 2.000 2.236
$original
[1] 1 2 3 4 5
R> RcppVectorExample( c("foo", "bar"), "new" )
Error in RcppVectorExample(c("foo", "bar"), "new") :
  not compatible with INTSXP
R>

There is also analogous code for the older API in the package, but it is about three times as long, has to loop over the vector and needs to set up the execption handling explicitly. As of right now, RcppExamples does not document every class but it should already provide a fairly decent start for using Rcpp. And many more actual usage examples are ... in the over two-hundred unit tests in Rcpp.

Update: Now actually showing new rather than classic API.

The new 0.7.6 release of Rcpp, our set of R / C++ interface classes, is now at CRAN and Debian. This comes just a few days after 0.7.5 as we had made a mistake in Makefile.win which is now fixed. A few other things sneaked in while were at it, see the snippet from the NEWS file below or look at Romain's blog where he highlights named-based indexing in vectors and the addition of iterator as well as begin() and end() members that now allow the use of STL algorithms on our R objects which is nifty. The changes are summarised below in the NEWS file snippet, more details are in the ChangeLog as well.

0.7.6   2010-02-12
    o   SEXP_Vector (and ExpressionVector and GenericVector, a.k.a List) now
	have methods push_front, push_back and insert that are templated
    o   SEXP_Vector now has int- and range-valued erase() members
    o   Environment class has a default constructor (for RInside)
    o   SEXP_Vector_Base factored out of SEXP_Vector (Effect. C++ #44)
    o   SEXP_Vector_Base::iterator added as well as begin() and end()
        so that STL algorithms can be applied to Rcpp objects
    o   CharacterVector gains a random access iterator, begin() and end() to
	support STL algorithmsl; iterator dereferences to a StringProxy
    o   Restore Windows build; successfully tested on 32 and 64 bit;
    o   Small fixes to inst/skeleton files for bootstrapping a package
    o   RObject::asFoo deprecated in favour of Rcpp::as<foo>

As always, even fuller details are in the ChangeLog on the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

A new release of our Rcpp R / C++ interface classes is now out, the version number is 0.7.5. It comes on the heels of the release 0.7.4 and keeps with our semi-frantic schedule of releases every ten or so days going. The package is now on CRAN and Debian, and mirrors start to get the new versions. As before, my local page provides more details and Romain's blog is always worth watching too. The changes are summarised below in the NEWS file snippet, more details are in the ChangeLog as well.

0.7.5	2010-02-08
    o 	wrap has been much improved. wrappable types now are :
    	- primitive types : int, double, Rbyte, Rcomplex, float, bool
    	- std::string
    	- STL containers which have iterators over wrappable types:
    	  (e.g. std::vector<t>, std::deque<t>, std::list<t>, etc ...). 
    	- STL maps keyed by std::string, e.g std::map<std::string>
    	- classes that have implicit conversion to SEXP
    	- classes for which the wrap template if fully or partly specialized
    	This allows composition, so for example this class is wrappable: 
    	std::vector< std::map<std::string> > (if T is wrappable)
    	
    o 	The range based version of wrap is now exposed at the Rcpp::
    	level with the following interface : 
    	Rcpp::wrap( InputIterator first, InputIterator last )
    	This is dispatched internally to the most appropriate implementation
    	using traits
    o	a new namespace Rcpp::traits has been added to host the various
    	type traits used by wrap
    o 	The doxygen documentation now shows the examples
    o 	A new file inst/THANKS acknowledges the kind help we got from others
    o	The RcppSexp has been removed from the library.
    
    o 	The methods RObject::asFoo are deprecated and will be removed
    	in the next version. The alternative is to use as<foo>.
    o	The method RObject::slot can now be used to get or set the 
    	associated slot. This is one more example of the proxy pattern
    	
    o	Rcpp::VectorBase gains a names() method that allows getting/setting
    	the names of a vector. This is yet another example of the 
    	proxy pattern.
    	
    o	Rcpp::DottedPair gains templated operator<< and operator>> that 
    	allow wrap and push_back or wrap and push_front of an object
    	
    o	Rcpp::DottedPair, Rcpp::Language, Rcpp::Pairlist are less
    	dependent on C++0x features. They gain constructors with up
    	to 5 templated arguments. 5 was choosed arbitrarily and might 
    	be updated upon request.
    	
    o	function calls by the Rcpp::Function class is less dependent
    	on C++0x. It is now possible to call a function with up to 
    	5 templated arguments (candidate for implicit wrap)
    	
    o	added support for 64-bit Windows (thanks to Brian Ripley and Uwe Ligges)

As always, even fuller details are in the ChangeLog on the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

git archive provides an easy way of producing a tarball directly from a project's git branch.

For example, this is what we use to build the Mahara tarballs:

git archive --format=tar --prefix=mahara-$ VERSION / $ RELEASETAG bzip2 -9 > $ CURRENTDIR /mahara-$ RELEASE .tar.bz2

If you do this however, you end up with the entire contents of the git branch, including potentially undesirable files like .gitignore.

There is an easy, though not very well-documented, way of specifying files to exclude from such exports: gitattributes.

This is what the Mahara .gitattributes file looks like:

/test export-ignore
.gitattributes export-ignore
.gitignore export-ignore

With this file in the root directory of our repository, tarballs we generate using git archive no longer contain the selenium tests or the git config files.

If you start playing with this feature however, make sure you commit the .gitattributes file to your repository before running git archive. Otherwise the settings will not be picked up by git archive.

Romain uploaded our first release of RProtoBuf to CRAN yesterday. RProtoBuf provides bindings for GNU R to the Google Protobuf implementation. Google Protobuf is (and I quote) a way of encoding structured data in an efficient yet extensible format that is used for almost all internal RPC protocols and file formats at Google. RProtoBuf had a funny start. I had blogged about the 12 hour passage from proof of concept to R-Forge project following the ORD session hackfest in October. What happened next was as good. Romain emailed within hours of the blog post and reminded me of a similar project that is part of Saptarshi Guha's RHIPE R/Hadoop implementation. So the three of us--Romain, Saptarshi and I---started emailing and before long it becomes clear that Romain is both rather intrigued by this (whereas Saptarshi has slightly different needs for the inner workings of his Hadoop bindings) and was able to devote some time to it. So the code kept growing and growing at a fairly rapid clip. Til that stopped as we switched to working feverishly on Rcpp to both support the needs of this project, and to implement ideas we had while working on this. That now lead to the point where Rcpp is maturing in terms of features, so we will probably have time come back to more work on RProtoBuf to take advantage of the nice templated autoconversions we now have in Rcpp. Oddly enough, the initial blog post seemed to anticipate changes in Rcpp. Anyway -- RProtoBuf is finally here and it already does a fair amount of magic based of code reflection using the proto files. The Google documentation has a simple example of a 'person' entry in an 'addressbook' which, when translated to R, goes like this:

R> library( RProtoBuf )                      ## load the package
R> readProtoFiles( "addressbook.proto" )     ## acquire protobuf information
R> bob <- new( tutorial.Person,              ## create new object
+   email = "bob@example.com",
+   name = "Bob",
+   id = 123 )
R> writeLines( bob$toString() )              ## serialize to stdout
name: "Bob"
id: 123
email: "bob@example.com"
R> bob$email                                 ## access and/or override
[1] "bob@example.com"
R> bob$id <- 5
R> bob$id
[1] 5
R> serialize( bob, "person.pb" )             ## serialize to compact binary format

There is more information at the RProtoBuf page, and we already have a draft package vignette, a 'quick' overview vignette and a unit test summary vignette. More changes should be forthcoming as Romain and I find time to code them up. Feedback is as always welcome.

Yesterday, and about nine days after release 0.7.3 of Rcpp (a set of R / C++ interface classes), Romain and I released version 0.7.4. It has been uploaded to CRAN and Debian, and mirrors should already have new versions. As before, my local page is also available for downloads and some more details. The release once again combines a number of necessary fixes with numerous new features:

• Building on OS X did not support multi-arch, and we are grateful for Simon who once again came to the rescue. Things should be fine now. The big take-away is that under no circumstances, include either a file configure or src/Makefile if you want multi-arch builds for free. As Rcpp is effectively a library to be used by other packages, this mattered.
• We added a file NEWS from which I include the relevant section below.
• Much more code re-organisation and enhancement making passage of various C++ types even easier -- see the NEWS entry below.
• More unit tests, now including ones for the 'old Rcpp API'.

Post-release, I also reworked the doxygen setup slightly so that all examples are now browseable, and the whole documentation is now searchable as well. Lastly, we had a remaining Windows build issue. Also, Brian Ripley and Uwe Ligges kindly sent us a small patch supporting the new Windows 64-bit builds using the new MinGW 64-bit compiler for Windows -- so release 0.7.5 may follow in due course. The NEWS file entry for release 0.7.4 is as follows:

0.7.4	2010-01-30
    o	matrix matrix-like indexing using operator() for all vector 
    	types : IntegerVector, NumericVector, RawVector, CharacterVector
    	LogicalVector, GenericVector and ExpressionVector. 
    o	new class Rcpp::Dimension to support creation of vectors with 
    	dimensions. All vector classes gain a constructor taking a 
    	Dimension reference.
    o	an intermediate template class "SimpleVector" has been added. All
    	simple vector classes are now generated from the SimpleVector 
    	template : IntegerVector, NumericVector, RawVector, CharacterVector
    	LogicalVector.
    o	an intermediate template class "SEXP_Vector" has been added to 
    	generate GenericVector and ExpressionVector.
    o	the clone template function was introduced to explicitely
    	clone an RObject by duplicating the SEXP it encapsulates.
    o	even smarter wrap programming using traits and template
        meta-programming using a private header to be include only
        RcppCommon.h
    o 	the as template is now smarter. The template now attempts to 
    	build an object of the requested template parameter T by using the
    	constructor for the type taking a SEXP. This allows third party code
    	to create a class Foo with a constructor Foo(SEXP) to have 
    	as<foo> for free.
    o	wrap becomes a template. For an object of type T, wrap<t> uses
    	implicit conversion to SEXP to first convert the object to a SEXP
    	and then uses the wrap(SEXP) function. This allows third party 
    	code creating a class Bar with an operator SEXP() to have 
    	wrap for free.
    o	all specializations of wrap :  wrap<double>, wrap< vector<double> >
    	use coercion to deal with missing values (NA) appropriately.
    o	configure has been withdrawn. C++0x features can now be activated
    	by setting the RCPP_CXX0X environment variable to "yes".
    o	new template r_cast<int> to facilitate conversion of one SEXP
    	type to another. This is mostly intended for internal use and 
    	is used on all vector classes
    o	Environment now takes advantage of the augmented smartness
    	of as and wrap templates. If as<foo> makes sense, one can 
    	directly extract a Foo from the environment. If wrap<bar> makes
    	sense then one can insert a Bar directly into the environment. 
    	Foo foo = env["x"] ;  /* as<foo> is used */
	Bar bar ;
	env["y"] = bar ;      /* wrap<foo> is used */    	
    o	Environment::assign becomes a template and also uses wrap to 
    	create a suitable SEXP
    o	Many more unit tests for the new features; also added unit tests
        for older API

As always, even fuller details are in the ChangeLog on the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page