Besides the new RcppExamples, another new package RcppArmadillo got spun out of Rcpp with the recent release 0.7.8 of Rcpp. Romain and I already had an example of a simple but fast linear model fit using the (very clever) Armadillo C++ library by Conrad Sanderson. In fact, I had used this as a motivational example of why Rcpp rocks in a recent talk to the ACM chapter at U of Chicago which, thanks to David Smith at REvo, got some further exposure. Now this example is more refined as further glue got added. Given that both Armadillo and Rcpp make use of C++ templates, the actual amount of code in RcppArmadillo is not that large: just over 200 lines in a header file, and a little less for some testing accessor and example functions in a source file. And this makes for some really nice example code: the 'fast regression' example becomes this (where I simply removed two blocks with conditional on the Armadillo version):

#include <RcppArmadillo.h>
extern "C" SEXP fastLm(SEXP ys, SEXP Xs)  
    Rcpp::NumericVector yr(ys);			// creates Rcpp vector from SEXP
    Rcpp::NumericMatrix Xr(Xs);			// creates Rcpp matrix from SEXP
    int n = Xr.nrow(), k = Xr.ncol();
    arma::mat X(Xr.begin(), n, k, false);   	// reuses memory and avoids extra copy
    arma::colvec y(yr.begin(), yr.size(), false);
    arma::colvec coef = solve(X, y);            // fit model y ~ X
    arma::colvec resid = y - X*coef; 		// residuals
    double sig2 = arma::as_scalar( trans(resid)*resid/(n-k) );
    						// std.error of estimate
    arma::colvec stderrest = sqrt( sig2 * diagvec( arma::inv(arma::trans(X)*X)) );
    Rcpp::Pairlist res(Rcpp::Named( "coefficients", coef),
                       Rcpp::Named( "stderr", stderrest));
    return res;
 

No extra copies! Armadillo instantiates directly from the underlying R objects for the vector and matrix, solves the regression equations, computes the standard error of the estimates and returns the two vectors. Leaving us to write about eleven lines of code. Moreover, as Armadillo is well designed and uses template meta-programming to avoid extra copies (see these lecture notes for details), it is about as efficient as it can be (and will use Atlas or other BLAS where available). And, this is just one example. Rcpp should be suitable for other C++ libraries, and provides an easy to use seamless interface between C++ and R. However, we should note that (at about the last minute) we found out about some unit test failures in OS X as well as some issues in a Debian chroot -- cran2deb ran into some build issues on i386 and amd64 in the testing chroot even this 'it all works' swimmingly on our Debian, Ubuntu and Fedora build environments. A follow-up with fixes for either Rcpp and/or RcppArmadillo appears likely.

Update: The build issues seems to be with 64-bit systems and everything appears cool in 32-bit.

I m happy to announce that after the filing of an Intent to Package and nearly 2 years of work, IronRuby 0.9, IronPython 2.6b2, and the DLR are now in Debian. To my knowledge, this is the first package in Debian with direct and active upstream support from Microsoft. Kudos for this release go to Jo Sheilds (package sponsorship & mentoring), Mirco Bauer (package sponsorship & mentoring), Matthias Klose (IronPython package review), Ivan Porto Carrero (IronRuby build/test support), Jim Deville (IronRuby build/test support), Jimmy Schementi (upstream point of contact @ Microsoft), Dino Viehland (IronPython build/test support), Michael Foord (IronPython build/test support), Marek Safar (mono c# compiler support), Ankit Jain (xbuild support), the folks on OFTC s #debian-cli, Freenode s #ironruby and GimpNet s #mono, and the folks on the IronRuby and IronPython mailing lists. This is my first package in Debian, too. I m pretty ecstatic ;)

Today I took a break before my usual lunchtime to go to the movies Boogie el Aceitoso was on at 13:00 (and not at the more usual, late screenings). Oily Boogie is a great antihero drawn by the much beloved Roberto El Negro Fontanarrosa, a very widely known Argentinian humorist/cartoonist. I got acquinted with Boogie as during the 80s-90s my parents were asiduous readers of Proceso, a weekly political analysis magazine which included one of his cartoons at the last page. Boogie is a pathological ex-Vietnam, ex-Laos ex-El Salvador, ex-Gulf War, ex-(whatever comes next) USA soldier, who deals with the local mafias whenever he is not active. Brutal, often seen as inhuman. I remember reading it without really understanding its nonsensical violence at first. And, as I said, Fontanarrosa is a very loved cartoonist - In Mexico I think we were much more acquinted with Boogie than with Inodoro Pereyra, and still, Fontanarrosa's death in 2007 was very heartfelt here. About the movie: I found it to be very good, of course, knowing what to expect. Most lines are short, screen adequations of various cartoons along Boogie's long life as a thug. I specially liked the animation technique I know very little about the subject, but it mixed quite naturally and constantly obvious still, cartoony characters with vivid, photo-based items. It creates a completely believable atmosphere inside the absolute amoral, selfish and (fortunately!) grossly exagerated and impossible world of Boogie. I sometimes feel somewhat stupid when writing in English for a mostly Spanish-speaking audience Still, if you see Boogie in a movie theater, don't hesitate and go. As always, with non-top-selling, non-Hollywood movies, it is quite probable it will not be showing for long.

Version 0.1.0 of RcppExamples, a simple demo package for Rcpp should appear on CRAN some time tomorrow. As mentioned in the post about release 0.7.8 of Rcpp, Romain and I carved this out of Rcpp itself to provide a cleaner separation of code that implements our R / C++ interfaces (which remain in Rcpp) and code that illustrates how to use it --- which is now in RcppExamples. This also provides an easier template for people wanting to use Rcpp in their packages as it will be easier to wrap one's head around the much smaller RcppExamples package. A simple example (using the newer API) may illustrate this:

#include <Rcpp.h>
RcppExport SEXP newRcppVectorExample(SEXP vector)  
    Rcpp::NumericVector orig(vector);			// keep a copy (as the classic version does)
    Rcpp::NumericVector vec(orig.size());		// create a target vector of the same size
    // we could query size via
    //   int n = vec.size();
    // and loop over the vector, but using the STL is so much nicer
    // so we use a STL transform() algorithm on each element
    std::transform(orig.begin(), orig.end(), vec.begin(), sqrt);
    Rcpp::Pairlist res(Rcpp::Named( "result", vec),
                       Rcpp::Named( "original", orig));
    return res;
 

With essentially five lines of code, we provide a function that takes any numeric vector and returns both the original vector and a tranformed version---here by applying a square root operation. Even the looping along the vector is implicit thanks to the generic programming idioms of the Standard Template Library. Nicer still, even on misuse, exceptions get caught cleanly and we get returned to the R prompt without any explicit coding on the part of the user:

R> library(RcppExamples)
Loading required package: Rcpp
R> print(RcppVectorExample( 1:5, "new" )) # select new API
$result
[1] 1.000 1.414 1.732 2.000 2.236
$original
[1] 1 2 3 4 5
R> RcppVectorExample( c("foo", "bar"), "new" )
Error in RcppVectorExample(c("foo", "bar"), "new") :
  not compatible with INTSXP
R>

There is also analogous code for the older API in the package, but it is about three times as long, has to loop over the vector and needs to set up the execption handling explicitly. As of right now, RcppExamples does not document every class but it should already provide a fairly decent start for using Rcpp. And many more actual usage examples are ... in the over two-hundred unit tests in Rcpp.

Update: Now actually showing new rather than classic API.

I'm looking for a way to do free hosting. But I mean free as in freedom, not free as in beer. Let me explain.

When I'm using a piece of free and open-source software such as OpenOffice.org, Evolution, or anything else, I have certain possibilities, freedoms if you will, of interacting with the software beyond just consuming it. I can look at the source code to study how it works. I can rebuild it to have a higher degree of confidence that I'm actually running that code. I can fix a bug or create an enhancement. I can send the patch upstream and wait for the next release, or in important cases I can create a local build. With the emerge of new project hosting sites such as GitHub, it's getting even easier to share one's modifications so others can use them. And so on.

As a lot of software moves to the web, how will this work in the future? There are those that say that it won't, and that it will be a big problem, and that's why you shouldn't use such services. Which is what probably a lot of free-software conscious users are doing right now. But I think that in the longer run, resisting isn't going to win over the masses to free software.

First of all, of course, the software would need to be written. So a free web office suite, a free web mail suite that matches the capabilities of the leading nonfree provider, and so on. We have good starts with Identi.ca and OpenStreetMap, for example, but we'd need a lot more. Then you throw it on a machine, and people can use it. Now as a user of this service, how do I get the source code? Of course you could offer a tarball for download, and that is the approach that the AGPL license takes. One problem with that is, if you are used to apt-get source or something similar for getting the source, everyone putting a tarball on their web site in a different place isn't going to make you happy. A standardized packaging-type thing ought to be wrapped around that. Another problem is that even if you trust the site's operator that that's the source code that's actually running on your site (even without malice, it could for example be outdated against the deployed version), it probably won't contain the local configuration files and setup scripts that would allow me to duplicate the service. And if I just want to study how the program is running in actuality, there is not much I can do.

Giving everyone SSH access to the box is probably not a good idea, and won't really solve all the issues anyway. In the future, when virtualization is standardized, ubiquitous, and awesome, one might imagine that a packaging of a web service won't be "put these files on the file system and reload a daemon" but instead "put these files and this configuration on that machine and activate it". This might give rise to a new generation of Linux "distributors". Getting the source tarball or "source package" might then involve getting a snapshot of that image, which you can examine, modify, and redeploy elsewhere. That could work for OpenStreetMap, for example, modulo the space and time required for their massive database. (But you might chose to fork only the code, not the data.) But it won't be easy to do the right thing in many cases, because with a web service, there is usually other people's data on the machine as well, which would need to be masked out or something. Maybe this really can't be done correctly, and the future will be more distributed, like in the way Jabber attempted to supplant centralized services such as ICQ. Distributed web mail makes sense, distributed OpenStreetMap perhaps less so.

Ideas anyone? Does anyone perhaps have experiences with running a web service that attempts to give users the freedoms and practical benefits that are usually associated with locally installed software?

There is an interesting conversation over at about the nature of peer production, and crowd based production over at PBS. Thankfully folks right off the bat noted that the types of activities they are addressing that range from 4chan to open source are so freaken distinct that perhaps it is not all that useful to use one moniker for them. The comments I am most fascinated by are danah s who notes:

We assume that the collective voice will be populist and, more importantly, that it will reflect the diversity of the populous. Yet, as we ve seen time and time again, certain values and attitudes and voices are over-represented in crowd-sourced activities. Who is looking out for those who aren t represented? In what ways are we reinforcing structural inequalities? What are the implications of this?

And then Clay s response:

So, to re-ask your question in a non-rhetorical way, under what
circumstances would we want to make the population of Deviant Art,
say, less white, or Linux less male, and if we wanted to do so, what
would need to happen?

What I find interesting about this discussion (and will be talking about this topic here, next week) is there not enough recognition of two related things: 1) the efforts are there (more on this soon) 2) that perhaps hacking and F/OSS in particular are not fully accessible to all and everyone because they are full-fledged, full-bodied, cultural worlds and all cultural worlds are to some degree not fully accessible and transparent for there are built on particularities, often invisible and unarticulated, forms of value. That is, just as some norms and values of Indo-Guyanese to take one random example, are not of my world, so too is hacking partially inaccessible for the fact that it is culturally configured. But I am starting to suspect that the culture-ness of these domains are often overlooked because they are overwhelmingly white, male, and chock full of computers (and so economically lucrative). All three, I suspect are (incorrectly) seen as lacking culture, as domains of rationality. (I stand rightly corrected and also forget this very fact, though I know it well from all the Brazil/Latin America Debconfs, as this diversity gets a bit lost from a pure US-European perspective, which I was assuming). Other historical factors have also produced certain distortions that don t allow us to see (easily at least) these worlds as culture-full. First is the fact that so many folks outside of this world lobbed onto F/OSS for being radical (and this is partially right in so far as its challenge to intellectual property can be seen as radical). But the portrayal or mere suggestion of these worlds as uber-democratic and populist, made people expect these groups to behave as radical egalitarian collectives. For the most part, they don t and yet never portrayed their own politics and forms of organization as such (openness comes in the form of code and technical merit). But this vision stuck and when some folks realized that larger projects, for example are very organized (which many people addressed only very late), have hierarchies (which are flexible and also allow them to function, which is I think is a good thing), and are not as diverse, there was deep disappointment that they did not conform to the sense that there was something extremely radical going on as opposed to a cultural group really into producing free software. But if I am offering a cultural alibi of sorts in which barriers to participation are to some degree a function of culture, one of the great things about the norms, values, ideas that compose culture is that there are dynamic and changing. They are alive and historical. They are pushed and pulled upon by insiders and outsiders based on wider social values. And there is an answer to these questions about diversity for there has been a dramatic, noticeable, and noteworthy push within this world, one that really started to coalesce I would say in the last year or so, to address these issues and it ranges from Python s mammoth efforts at addressing diversity (and I have been told that there was a great speech on the topic at Pycon recently), the geek feminism wiki, and smaller but increasingly common efforts such as Libre Planet s women s caucus and their funding of women to participate. So while I do think that culture goes at least part of the way to explain why these worlds are not fully open for culture limits this very domain has grown dissatisfied with its representational make-up and are leading some efforts for cultural change.

There is an interesting conversation over at about the nature of peer production, and crowd based production over at PBS. Thankfully folks right off the bat noted that the types of activities they are addressing that range from 4chan to open source are so freaken distinct that perhaps it is not all that useful to use one moniker for them. The comments I am most fascinated by are danah s who notes:

We assume that the collective voice will be populist and, more importantly, that it will reflect the diversity of the populous. Yet, as we ve seen time and time again, certain values and attitudes and voices are over-represented in crowd-sourced activities. Who is looking out for those who aren t represented? In what ways are we reinforcing structural inequalities? What are the implications of this?

And then Clay s response:

So, to re-ask your question in a non-rhetorical way, under what
circumstances would we want to make the population of Deviant Art,
say, less white, or Linux less male, and if we wanted to do so, what
would need to happen?

What I find interesting about this discussion (and will be talking about this topic here, next week) is there not enough recognition of two related things: 1) the efforts are there (more on this soon) 2) that perhaps hacking and F/OSS in particular are not fully accessible to all and everyone because they are full-fledged, full-bodied, cultural worlds and all cultural worlds are to some degree not fully accessible and transparent for there are built on particularities, often invisible and unarticulated, forms of value. That is, just as some norms and values of Indo-Guyanese to take one random example, are not of my world, so too is hacking partially inaccessible for the fact that it is culturally configured. But I am starting to suspect that the culture-ness of these domains are often overlooked because they are overwhelmingly white, male, and chock full of computers (and so economically lucrative). All three, I suspect are (incorrectly) seen as lacking culture, as domains of rationality. (I stand rightly corrected and also forget this very fact, though I know it well from all the Brazil/Latin America Debconfs, as this diversity gets a bit lost from a pure US-European perspective, which I was assuming). Other historical factors have also produced certain distortions that don t allow us to see (easily at least) these worlds as culture-full. First is the fact that so many folks outside of this world lobbed onto F/OSS for being radical (and this is partially right in so far as its challenge to intellectual property can be seen as radical). But the portrayal or mere suggestion of these worlds as uber-democratic and populist, made people expect these groups to behave as radical egalitarian collectives. For the most part, they don t and yet never portrayed their own politics and forms of organization as such (openness comes in the form of code and technical merit). But this vision stuck and when some folks realized that larger projects, for example are very organized (which many people addressed only very late), have hierarchies (which are flexible and also allow them to function, which is I think is a good thing), and are not as diverse, there was deep disappointment that they did not conform to the sense that there was something extremely radical going on as opposed to a cultural group really into producing free software. But if I am offering a cultural alibi of sorts in which barriers to participation are to some degree a function of culture, one of the great things about the norms, values, ideas that compose culture is that there are dynamic and changing. They are alive and historical. They are pushed and pulled upon by insiders and outsiders based on wider social values. And there is an answer to these questions about diversity for there has been a dramatic, noticeable, and noteworthy push within this world, one that really started to coalesce I would say in the last year or so, to address these issues and it ranges from Python s mammoth efforts at addressing diversity (and I have been told that there was a great speech on the topic at Pycon recently), the geek feminism wiki, and smaller but increasingly common efforts such as Libre Planet s women s caucus and their funding of women to participate. So while I do think that culture goes at least part of the way to explain why these worlds are not fully open for culture limits this very domain has grown dissatisfied with its representational make-up and are leading some efforts for cultural change.

Version 0.7.8 of the Rcpp R / C++ interface classes is now on CRAN and in Debian. As of right now. Debian has already built packages for eight more architectures; and CRAN has built the Windows binary. Oh, and cran2deb had Debian packages for 'testing' before I was done with the blog entry. This is a minor feature release based on a over three weeks of changes that are summarised below in the extract from the NEWS file. Some noteworthy highlights are

• something that isn't there: we have split most of the example code and their manual pages off into a new package RcppExamples which can now be released given that 0.7.8 is out
• another new package RcppArmadillo will also be forthcoming shortly: it shows how to use Rcpp with Conrad Sanderson's excellent Armadillo C++ library for linear algebra; this required some internal code changes to seamlessly pass data from R via Rcpp to Armadillo and back;
• there is a new example fastLm using Armadillo for faster (than lm() or lm.fit()) linear model fits
• yet more internal improvements to the class hierarchy as detailed below; more support for STL iterators and algorithms;
• more build fixes; paths with spaces in the name should now be tolerated
• and last but not least a new introduction / overview vignette based on a just-submitted paper on Rcpp.

The full NEWS entry for this release follows:

0.7.8   2010-03-09
    o	All vector classes are now generated from the same template class
    	Rcpp::Vector<int rtype="RTYPE"> where RTYPE is one of LGLSXP, RAWSXP, STRSXP,
    	INTSXP, REALSXP, CPLXSXP, VECSXP and EXPRSXP. typedef are still 
    	available : IntegerVector, ... All vector classes gain methods 
    	inspired from the std::vector template : push_back, push_front, 
    	erase, insert
    	
    o	New template class Rcpp::Matrix<rtype> deriving from 
    	Rcpp::Vector<rtype>. These classes have the same functionality
    	as Vector but have a different set of constructors which checks
    	that the input SEXP is a matrix. Matrix<rtype> however does/can not
    	guarantee that the object will allways be a matrix. typedef 
    	are defined for convenience: Matrix<intsxp> is IntegerMatrix, etc...
    	
    o	New class Rcpp::Row<int rtype="RTYPE"> that represents a row of a matrix
    	of the same type. Row contains a reference to the underlying 
    	Vector and exposes a nested iterator type that allows use of 
    	STL algorithms on each element of a matrix row. The Vector class
    	gains a row(int) method that returns a Row instance. Usage 
    	examples are available in the runit.Row.R unit test file
    	
    o	New class Rcpp::Column<int rtype="RTYPE"> that represents a column of a 
    	matrix. (similar to Rcpp::Row<int rtype="RTYPE">). Usage examples are 
    	available in the runit.Column.R unit test file
    o	The Rcpp::as template function has been reworked to be more 
    	generic. It now handles more STL containers, such as deque and 
    	list, and the genericity can be used to implement as for more
    	types. The package RcppArmadillo has examples of this
    o   new template class Rcpp::fixed_call that can be used in STL algorithms
	such as std::generate.
    o	RcppExample et al have been moved to a new package RcppExamples;
        src/Makevars and src/Makevars.win simplified accordingly
    o	New class Rcpp::StringTransformer and helper function 
    	Rcpp::make_string_transformer that can be used to create a function
    	that transforms a string character by character. For example
    	Rcpp::make_string_transformer(tolower) transforms each character
    	using tolower. The RcppExamples package has an example of this.
        
    o	Improved src/Makevars.win thanks to Brian Ripley
    o	New examples for 'fast lm' using compiled code: 
        - using GNU GSL and a C interface
        - using Armadillo (http://arma.sf.net) and a C++ interface
        Armadillo is seen as faster for lack of extra copying
    o	A new package RcppArmadillo (to be released shortly) now serves 
        as a concrete example on how to extend Rcpp to work with a modern 
	C++ library such as the heavily-templated Armadillo library
    o	Added a new vignette 'Rcpp-introduction' based on a just-submitted 
        overview article on Rcpp

As always, even fuller details are in the ChangeLog on the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Update: Two links corrected.

By the time this blog entry goes live I'll be running upon my new machine. The migration process was mostly straightfoward and followed my plan:

• Using my existing desktop system as a PXE server to install Lenny over the network.
• Copied over important directories.
• Restored backups.
• Turned off old machine.

Of course it wasn't that simple in practise, as previously mentioned the whole reason I was looking for a new machine was because the software RAID upon my old desktop was failing - One of the two drives was completely dead. As I'd feared the second drive failed partway through my migration. But thankfully I'd copied off the important stuff before then, and the backups I have off-site mostly covered everything else. (The things I lost were things I can find again such as ~/Music, ~/Videos. On the one hand they're too large to backup, on the other hand I should probably do it next time as they never change.) Unfortunately the version of X in Lenny refused to work with the GeForce G210 video card I had. To be more correct using the Vesa driver I could get a picture and a smooth desktop, but when watching videos with xine I got maybe two frames a second. Both the open nv driver and the closed nvidia driver failed to support the card - so I swapped hardware, and I'm now running with the GeForce 7300 GS card from my previous desktop. This allows me to watch videos at full-screen with no issues. (Desktop size is 1600x1200 FWIW). So now it's just a matter of tweaking the system. I've installed enough to be useful:

• miredo - So I have IPv6 connectivity despite Virgion.
• squid - So that I have a decent cache for surfing.
• pdnsd - So I have a caching nameserver and am not at the whim of Virgin.
• kvm - So I can setup scratch machines for play.

I've still got to setup pbuilder, but that'll be done shortly, and I've installed backported packages such that I can watch youtube videos. I'm currently running firefox from lenny but I expect that will change soon enough - not least because that version fails to support "adblockplus", only "adblock". Two partitions md0 for /boot and md1 used as LVM, from which I've taken /, /home, etc:

Filesystem                      Size    Used    Avail Use% Mounted on
/dev/mapper/birthday--vol-root   9.9G     2.8G   6.6G  30% /
/dev/mapper/birthday--vol-home   22G      4.3G  16G    22% /home
/dev/mapper/birthday--vol-music  127G    43G    78G    36% /mnt/music
/dev/md0                         988M    38M    901M    4% /boot
/dev/mapper/birthday--vol-kvm    22G      8.8G  12G    44% /mnt/kvm
/dev/sdg1                        163G    143G   12G    93% /media/disk
skx@birthday:~/hg/blog/data$

skx@birthday:~/hg/blog/data$ sudo pvs
[sudo] password for skx:
  PV         VG           Fmt  Attr PSize   PFree
  /dev/md1   birthday-vol lvm2 a-   464.82G 274.51G

Update: Three irritations with this machine:

1. As supplied the BIOS was set with "USB Mouse" and "USB Keyboard" set to "disabled". I had to beg the loan of a keyboard from a neighbour.
2. As supplied the BIOS had virtualisation set to "disabled". Not a huge shock, but it caught me out regardless.
3. As supplied the system had only a single SATA power connector. Annoying given that the motherboard is advertised as having "onboard RAID" and I'd purchased it with two hard drives. Happily I had a spare adaptor to hand.

I'd still recommend Novatech, but the last point had me swearing for a few minutes until I realised I did have a spare adaptor in the house. ObFilm: Chitty Chitty Bang Bang

Just a kind thank you note to the gThumb authors for completely breaking my workflow by re-inventing the paradigm used to save imported pictures. Until now, all my pictures landed in a predictable location, using a predictable filename pattern that was easily searchable. Not anymore. Now, files land into my home directory, according to some recursive folder pattern that further complicates searching for files and requires a few more clicks to accomplish. Dammit! Couldn't you at least make this configurable, so that those of us who prefer to retain the old paradigm can?! Note: re-inventing an application's paradigms is always a very bad idea. If you're a software developer who is reading this, please keep it in mind and go scratch your itch to change the world somewhere else. Thank you. Post Scriptum Many thanks to Damon Lynch for pointing me to his own professional picture importer called Rapid. This is an extremely configurable importing tool and, lo and behold, Damon even offers builds for Ubuntu via his PPA! Still, the consequence of this mess is that migrating to Rapid means that I'll be loosing gThumb's simple but extremely efficient editing tools. To me, one strength of gTthumb was this unique combination of picture importing with basic editing tools. Now, I'm forced to split these interconnected tasks, simply because someone chose to completely rethink gThumb's paradigms. I'm of course aware of Gimp's existence, but repeated attempts at mastering it made me conclude that it's entirely the wrong software for my needs and essentially overkill. By contrast, gThumb offers just enough tools to enable someone to crop images to useful sizes and to adjust color balances in easy steps; it does the job without hassle, which is not the case with Gimp. Thinking out loud, it is precisely on days like these that the urge to create my own Linux distribution keeps on coming to mind. Retaining consistent paradigms in the desktop environment and applications that I use, not to mention maintaining the number of duplicate libraries to a bare minimal, has been a constant struggle and, noticing how some developers' urge to re-invent the wheel every other day, using whatever new programming language of the day, persistently takes precedence over keeping system resource consumption to a bare minimum and over preserving user sanity, I'm slowly coming to the conclusion that Free Software has veered way too far into the bazaar and urgently needs a copious amount of cathedral to make it usable for mere mortals again. Some dpkg-based distribution where the only scripting language allowed is Bourne shell and the only programming languages C or C++ comes to mind. Of course, this would also require porting popular application from e.g. Java, Python, etc. which would be a colossal amount of work. Still, I think that the time has come for this to happen. As an added bonus, this would make applications usable again on embedded devices with spartan CPU, RAM and storage resources, so this project could generate huge benefits to the embedded Linux industry. Based on my experience at my previous jobs, I have a rather clear picture (pun intended) of what needs to be done and of who I would hire to make it happen. What I'm missing are investors. Who's with me?

I am still being plagued by the burdens of real life so most development is taking place after the kids ("It should have mermaids! And Spiderman!") have gone to sleep. By the time I finished up yesterday, I was too tired to blog so this is is a combined report on the last two days. I decided not to bother with Javascript as I want to try a different project for the plasmoid competition. So I'm back to C++ and ncurses. At this point my '@' can move across the dungeon which at the moment a very simplistic 10 10 grid where every space is a 'room'. Rooms are populated with monsters, treasure and items though at the moment only the monsters can do anything and by anything I mean fighting. My combat system is based on Solo Dungeon Bash and pretty challenging I think. Hopefully by tomorrow, A Lively Fisting will be playable though I really need to pick up the pace if I want to finish on time.

Since I last mentioned music back in January I have accumulated another ten albums and unlike last time where there were only a couple of stand outs, this time I have he opposite problem.

Well my last post elicited a response from Mr Lougher the squashfs author. Just not one I was expecting. Apparently he did receive one of my emails (I sent five in total) to which he has not replied as I have accidentally come across as critical.

Temporarily disabling file caching Does it happen to you that you cp a big, big file (say, similar in order of magnitude to the amount of RAM) and the system becomes rather unusable? It looks like Linux is saying "let's cache this", and as you copy it will try to free more and more ram in order to cache the big file you're copying. In the end, all the RAM is full with file data that you are not going to need. This varies according to how /proc/sys/vm/swappiness is set. I learnt about posix_fadvise and I tried to play with it. The result is this preloadable library that hooks into open(2) and fadvises everything as POSIX_FADV_DONTNEED. It is all rather awkward. fadvise in that way will discard existing cache pages if the file is already cached, which is too much. Ideally one would like to say "don't cache this because of me" without stepping on the feet of other system activities. Also, I found I need to also hook into write(2) and run fadvise after every single write, because you can't fadvise a file to be written in its entirety, unless you pass fadvise the file size in advance. But the size of the output file cannot be known by the preloaded library, so meh. So, now I can run: nocache cp bigfile someplace/ without trashing the existing caches. I can also run nocache tar zxf foo.tar.gz and so on. I wish, of course, that there were no need to do so in the first place. Here is the nocache library source code, for reference:

/*
 * nocache - LD_PRELOAD library to fadvise written files to not be cached
 *
 * Copyright (C) 2009--2010 Enrico Zini <enrico@enricozini.org>
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program; if not, write to the Free Software
 * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 */
#define _XOPEN_SOURCE 600
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <dlfcn.h>
#include <stdarg.h>
#include <errno.h>
#include <stdio.h>
typedef int (*open_t)(const char*, int, ...);
typedef int (*write_t)(int fd, const void *buf, size_t count);
int open(const char *pathname, int flags, ...)
 
    static open_t func = 0;
    int res;
    if (!func)
        func = (open_t)dlsym(RTLD_NEXT, "open");
    // Note: I wanted to add O_DIRECT, but it imposes restriction on buffer
    // alignment
    if (flags & O_CREAT)
     
        va_list ap;
        va_start(ap, flags);
        mode_t mode = va_arg(ap, mode_t);
        res = func(pathname, flags, mode);
        va_end(ap);
      else
        res = func(pathname, flags);
    if (res >= 0)
     
        int saved_errno = errno;
        int z = posix_fadvise(res, 0, 0, POSIX_FADV_DONTNEED);
        if (z != 0) fprintf(stderr, "Cannot fadvise on %s: %m\n", pathname);
        errno = saved_errno;
     
    return res;
 
int write(int fd, const void *buf, size_t count)
 
    static write_t func = 0;
    int res;
    if (!func)
        func = (write_t)dlsym(RTLD_NEXT, "write");
    res = func(fd, buf, count);
    if (res > 0)
     
        int saved_errno = errno;
        int z = posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED);
        if (z != 0) fprintf(stderr, "Cannot fadvise during write: %m\n");
        errno = saved_errno;
     
    return res;
 

Updates Steve Schnepp writes:

Robert Love did a O_STREAMING patch for 2.4. It wasn't merged in 2.6 since POSIX_FADV_NOREUSE should be used instead. But unfortunatly it's currently mapped either as WILLNEED or as a noop. It seems that there is a google code project that has spawned to control this.

The Threat Bruce Schneier s blog post about the Mariposa Botnet has an interesting discussion in the comments about how to make a secure system [1]. Note that the threat is considered to be remote attackers, that means viruses and trojan horses which includes infected files run from USB devices (IE you aren t safe just because you aren t on the Internet). The threat we are considering is not people who can replace hardware in the computer (people who have physical access to it which includes people who have access to where it is located or who are employed to repair it). This is the most common case, the risk involved in stealing a typical PC is far greater than the whatever benefit might be obtained from the data on it a typical computer user is at risk of theft only for the resale value of a second-hand computer. So the question is, how do can we most effectively use free software to protect against such threats? The first restriction is that the hardware in common use is cheap and has little special functionality for security. Systems that have a TPM seem unlikely to provide a useful benefit due to the TPM being designed more for Digital Restrictions Management than for protecting the user and due to TPM not being widely enough used. The BIOS and the Bootloader It seems that the first thing that is needed is a BIOS that is reliable. If an attacker manages to replace the BIOS then it could do exciting things like modifying the code of the kernel at boot time. It seems quite plausible for the real-mode boot loader code to be run in a VM86 session and to then have it s memory modified before it starts switches to protected mode. Every BIOS update is a potential attack. Coreboot replaces the default PC BIOS, it initialises the basic hardware and then executes an OS kernel or boot loader [2] (the Coreboot Wikipedia page has a good summary). The hardest part of the system startup process is initialising the hardware, Coreboot has that solved for 213 different motherboards. If engineers were allowed to freely design hardware without interference then probably a significant portion of the computers in the market would have a little switch to disable the write line for the flash BIOS. I heard a rumor that in the days of 286 systems a vendor of a secure OS shipped a scalpel to disable the hardware ability to leave protected mode, cutting a track on the motherboard is probably still an option. Usually once a system is working you don t want to upgrade the BIOS. One of the payloads for Coreboot is GRUB. The Grub Feature Requests page has as it s first entry Option to check signatures of the bootchain up to the cryptsetup/luksOpen: MBR, grub partition, kernel, initramfs [3]. Presumably this would allow a GPG signature to be checked so that a kernel and initrd would only be used if they came from a known good source. With this feature we could only boot a known good kernel. How to run User Space The next issue is how to run the user-space. There has been no shortage of Linux kernel exploits and I think it s reasonable to assume that there will continue to be a large number of exploits. Some of the kernel flaws will be known by the bad guys for some time before there are patches, some of them will have patches which don t get applied as quickly as desired. I think we have to assume that the Linux kernel will be compromised. Therefore the regular user applications can t be run against a kernel that has direct hardware access. It seems to me that the best way to go is to have the Linux kernel run in a virtual environment such as Xen or KVM. That means you have a hypervisor (Xen+Linux or Linux+KVM+QEMU) that controls the hardware and creates the environment for the OS image that the user interacts with. The hypervisor could create multiple virtual machines for different levels of data in a similar manner to the NSA NetTop project, not that this is really a required part of solving the general secure Internet terminal problem but as it would be a tiny bit of extra work you might as well do it. One problem with using a hypervisor is that the video hardware tends to want to use features such as bus-mastering to give best performance. Apparently KVM has IOMMU support so it should be possible to grant a virtual machine enough hardware access to run 3D graphics at full speed without allowing it to break free. Maintaining the Virtual Machine Image Google has a good design for the ChromiumOS in terms of security [4]. They are using CGroups [5] to control access to device nodes in jails, RAM, CPU time, and other resources. They also have some intrusion detection which can prompt a user to perform a hardware reset. Some of the features would need to be implemented in a different manner for a full desktop system but most of the Google design features would work well. For an OS running in a virtual machine when an intrusion is detected it would be best to have the hypervisor receive a message by some defined interface (maybe a line of text printed on the console ) and then terminate and restart the virtual machine. Dumping the entire address space of the virtual machine would be a good idea too, with typical RAM sizes at around 4G for laptops and desktops and typical storage sizes at around 200G for laptops and 2T for new desktops it should be easy to store a few dumps in case they are needed. The amount of data received by a typical ADSL link is not that great. Apart from the occasional big thing (like downloading a movie or listening to Internet radio for a long time) most data transfers are from casual web browsing which doesn t involve that much data. A hypervisor could potentially store the last few gigabytes of data that were received which would then permit forensic analysis if the virtual machine was believed to be compromised. With cheap SATA disks in excess of 1TB it would be conceivable to store the last few years of data transfer (with downloaded movies excluded) but such long-term storage would probably involve risks that would outweigh the rewards, probably storing no more than 24 hours of data would be best. Finally in terms of applying updates and installing new software the only way to do this would be via the hypervisor as you don t want any part of the virtual machine to be able to write to it s data files or programs. So if the user selects to install a new application then the request please install application X would have to be passed to the hypervisor. After the application is installed a reboot of the virtual machine would be needed to apply the change. This is a common experience for mobile phones (where you even have to reboot if the telco changes some of their network settings) and it s something that MS-Windows users have become used to but it would get a negative reaction from the more skilled Linux users. Would this be Accepted? The question is, if we built this would people want to use it? The NetTop functionality of having two OSs interchangeable on the one desktop would attract some people. But most users don t desire greater security and would find some reason to avoid this. They would claim that it lowered the performance (even for aspects of performance where benchmarks revealed no difference) and claim that they don t need it. At this time it seems that computer security isn t regarded as a big enough problem for users. It seems that the same people who will avoid catching a train because one mugging made it to the TV news will happily keep using insecure computers in spite of the huge number of cases of fraud that are reported all the time.

• [1] http://www.schneier.com/blog/archives/2010/03/mariposa_botnet.html
• [2] http://www.coreboot.org/Welcome_to_coreboot
• [3] http://grub.enbug.org/FeatureRequests
• [4] http://sites.google.com/a/chromium.org/dev/chromium-os/chromiumos-design-docs/system-hardening
• [5] http://lxr.linux.no/linux+v2.6.30/Documentation/cgroups/

I've made a quick release of dh-make, now up to 0.52. Besides a minor dh_make.1 manual page fix this release will put the right value in the debian/source/format file. 0.51 will make it a native source package no matter what flags you use.

If you get weird lintian errors about native source formats and version numbers wrong on your brand-new Debian package you just made, you might of been bitten by this bug.

That leaves a few bugs left, there are two I need help with:

• Bug 328692 - If you have a plan ASCII name like me, then your name looks fine in the dh-make generated man pages. This bug requires converting names that aren't plain ASCII (e.g. that use UTF-8) into something groff understands. Someone suggested decomposed unicode but unicode and groff are pretty much a black art to me, let alone combining them.
• Bug 533117 - This one is all about making dh-make make your python packages. You'll need to understand the new debhelper v7 rules files. Again, python is not something I use so the bug is stuck here.
  

Related articles by Zemanta

• UTF-8: Bits, Bytes, and Benefits (research.swtch.com)

I just updated to the latest beta of Chrome, and the backspace key stopped working as a keyboard shortcut for the Back button. After a few times of stabbing the backspace key and not getting the result I wanted, I decided to go looking into what was going on here. It looks like it all started with bug 30699, where someone didn't like the default behaviour. That led to bug 36533, when the people (like me) noticed the functionality they were relying on disappeared. Now I fully understand that Backspace == Back is not the default behaviour of Firefox (on Linux), but it is a configurable option, and I'd had it enabled there for years. I think it all started with when I migrated from Windows to Linux. It's normal for Backspace == Back with IE and I think Firefox for Windows, and I've just developed the muscle memory for it, and I've never had a problem like what the submitter of bug 30699 was complaining about. I look forward to it becoming a configurable option in Chrome.

I must confess I don't remember who I got this invitation from. Anyway, if you are in the right geographic area, you might be interested. I will try to participate: This is a year-long seminar that will be held the second Thursday every month at Fonoteca Nacional (a place I have wanted to visit for a long time!), in Barrio de Santa Catarina, Coyoac n. Among the organizers they have Creative Commons Mexico. Free entrance (but limited space - so they ask interested people to confirm their presence by mail to bvallarta@conaculta.gob.mx). [update] I went with Pooka to the first session. We arrived almost 1hr late (due to me mistaking the schedule :-/ ) but it was interesting. Of course, quite biased towards the Google viewpoints, but interesting. We got the program for the next sessions So, mostly for myself to keep handy, here it goes:

Date	Title	Speakers
2010-03-11	Google and copyright	Manuel Tamez, Hugo Contreras, Mar a Fernanda Mendoza
2010-04-08	Generalities about rights on intelectual property	Jes s Parets, Guillermo Sol rzano, Jorge Mier y Concha
2010-05-13	Copyright's nature and competent authorites	Carmen Arteaga, Luis Schmidt, C sar Callejas
2010-06-10	Moral and patrimonial rights	Guillermo Pous, Eduardo de la Parra, Ram n Ob n
2010-07-08	Reproduction rights for audible material	lvaro Hegewisch, scar Javier Solorio, Marco Antonio Morales, Jos Ram n C rdeno
2010-08-12	Licenses and patrimonial right transmission. Works for hire, works done under laboral relationship, or carried out in official service	Dolores Franco, Jes s Mej a, Ra l Pastor
2010-09-09	Limits to explotation rights and literary plagiarism	Carmen Arteaga, Juan Ram n Ob n, Jorge Mier y Concha, C sar Callejas
2010-10-14	Copyright in a digital setting	Jes s Parets, Gast n Esquivel
2010-11-11	Law-regulated intelectual property rights	Rosalba Elizalde, Salvador Ortega, Gast n Esquivel, Manrique Moheno
2010-12-09	International protection and collective gestive societies	Horacio Rangel, Luis Schmidt, Jes s Mej a

After a long time of testing and just plain other non-software writing stuff, I'm happy to announce Gjay 0.3.0 is released. This is my first release of Chuck Groom's code and hopefully it will work for you too.

The Debian packages will be out shortly after some building and testing. If you have a 64-bit computer it now works with 64 bits fine (ie on my amd64).

It still needs some work, I'd like it to interact with more than audacious as the sound player. Also if you know how to in one of the sound libraries stream wav, ogg or mp3 files correctly I'd like to hear from you. Currently gjay just uses the same old fork to mpg321 method, but idealy I'd like it to use the libraries directly.

For the first time since I got involved in Ruby maintenance in Debian & Ubuntu, we are seriously at risk of not being able to release working Ruby interpreters in both Ubuntu Lucid and Debian Squeeze, because of two pretty serious issues. Ruby 1.8(.7.249) randomly hangs with eglibc 2.11 That issue was found by Google folks, because Puppet was hanging. After a lot of (quite heated) discussion, it was found that the problem is caused by something in eglibc 2.11: the problem can t be reproduced in a Debian unstable chroot (with eglibc 2.10), but is reproducible if you just upgrade glibc to 2.11 (which is in Debian experimental). It is not clear at all whether it is a glibc bug, or a ruby bug: Ruby has shown in the past that it was making some assumptions on the behaviour of pthread. It is possible that a valid change in glibc conflicts with an assumption made by Ruby. Still, I wonder why Ubuntu chose to release with eglibc 2.11, while Debian still only has 2.10 in unstable, and Ubuntu doesn t really have a lot of people with deep glibc knowledge. (Update: I assumed we would release squeeze with 2.10, but the eglibc maintainers plan to release 2.11 in squeeze.) That issue is tracked in Launchpad as bug 520715 (which was reassigned to eglibc), and in Ruby s redmine as bug 2739. Debian Squeeze isn t affected by that issue, but we are likely to get bitten as well after Squeeze if a solution isn t found by then. As Ruby maintainers in Debian/Ubuntu, we can t do much about this issue, because it requires either a lot of glibc knowledge, or a lot of understanding of the Ruby threading code. Ruby 1.9(.1.37 6,8 ) s test suite hangs on Sparc The development branch of Ruby (declared stable by the interpreter developers, but not considered as such by the third-party libraries developers) also has interesting issues. Its test suite hangs on sparc, but only (apparently) on specific CPUs/kernel versions. Here again, it is not clear whether it is a Ruby issue, or a kernel issue, for the same reasons as above. It is also possible that it is fixed in Linux 2.6.32, but the Debian build daemons run the lenny kernel, so we can t check that. This is tracked as Debian bug 565765, and is not tracked in Ruby s redmine, because Sparc is not supported by the upstream Ruby developers. This issue really requires Sparc knowledge (and access), so as Ruby maintainers, again, we can t do much, besides coordinating and poking the various people that can help.