The other day, I found myself confronted with a large number of large files. Which were presented in (gzip-)compressed ascii format---which R reads directly via gzfile() connections---as well as (compressed) numpy files. The numpy can be read very efficiently into Python. We can do the same in R via save() and load(), of course. But the trouble is that you need to read them first. And reading hundreds of megabytes from ascii is slow, no matter which language you use. Concerning R, I poked aound scan(), played with the colClasses argument and looked at the recent LaF package written just for this purpose. And all these solutions were still orders of magnitude slower than reading numpy. Which is no surprise as it is really hard to beat binary formats when you have to parse countless ascii tokens. So the obvious next idea was to read the numpy file in Python, and to write a simple binary format. One helpful feature with this data set was that it contained only regular (rectangular) matrices of floats. So we could just store two integers for the dimensions, followed by the total data in either one large binary blob, or a sequence of column vectors. But one minor trouble was that the Intertubes lead to no easy solution to unpack the numpy format. StackOverflow had plenty of question around this topic converned with, say, how to serialize in language-independent way. But no converters. And nobody local knew how to undo the "pickle" format underlying numpy. But a remote friend did: Laurent, well-known for his Rpy2 package, pointed me towards using the struct module and steered me towards the solution shown below. So a shameless plug: if you need a very experienced Python or R consultant for sciece work, consider his consulting firm. Finally, to round out this post, let's show the simple solution we crafted so that the next guy searching the Intertubes will have an easier. Let us start with a minimal Python program writing numpy data to disk:

#!/usr/bin/env python
#
# simple example for creating numpy data to demonstrate converter
import numpy as np
# simple float array
a = np.arange(15).reshape(3,5) * 1.1
outfile = "/tmp/data.npy"
np.save(outfile, a)

Next, the simple Python converter to create a binary file containing two integers for row and column dimension, followed by row times columns of floats:

#!/usr/bin/python
#
# read a numpy file, and write a simple binary file containing
#   two integers 'n' and 'k' for rows and columns
#   n times k floats with the actual matrix
# which can be read by any application or language that can read binary
 
import struct
import numpy as np
inputfile = "/tmp/data.npy"
outputfile = "/tmp/data.bin"
# load from the file
mat = np.load(inputfile)
# create a binary file
binfile = file(outputfile, 'wb')
# and write out two integers with the row and column dimension
header = struct.pack('2I', mat.shape[0], mat.shape[1])
binfile.write(header)
# then loop over columns and write each
for i in range(mat.shape[1]):
    data = struct.pack('%id' % mat.shape[0], *mat[:,i])
    binfile.write(data)
binfile.close()

Lastly, a quick littler script showing how R can read the data in a handful of lines:

#!/usr/bin/r
infile <- "/tmp/data.bin"
con <- file(infile, "rb")
dim <- readBin(con, "integer", 2)
Mat <- matrix( readBin(con, "numeric", prod(dim)), dim[1], dim[2])
close(con)
print(Mat)

That did the job---and I already used to converter to read a few weeks worth of data for further analysis in R. This obviously isn't the last word on possible solutions as the additional temporary file can be wasteful (unless it forms a cache for data read multiple times). If someone has nice solutions, please don't hold back and contact me. Thanks again to Laurent for the winning suggestion concerning struct, and help in getting the examples shown here to work.

I've been using Twitter's bootstrap library recently to build this Web site, and wondered how to be able to use the bootstrap-tab Javascript plugin in a bookmark friendly manner. I ended up with a simple solution. These are my first steps in Javascript and front-end manipulation, and it's really not my area of expertise, so don't be harsh. <figure>

function bootstrap_tab_bookmark (selector)   if (selector == undefined)  
    selector = "";  
    /* Automagically jump on good tab based on anchor */
    $(document).ready(function()  
        url = document.location.href.split('#');
        if(url[1] != undefined)  
            $(selector + '[href=#'+url[1]+']').tab('show');
         
     );
    var update_location = function (event)  
        document.location.hash = this.getAttribute("href");
     
    /* Update hash based on tab */
    $(selector + "[data-toggle=pill]").click(update_location);
    $(selector + "[data-toggle=tab]").click(update_location);
 

</figure> All you need is to use and call this function with a selector (only useful if you have several tabs/pills divisions) when the document is ready. The first part takes care of showing the good tab based on the hash contained in the URL. The second part takes care of changing the document location to add the current tab to it when the user clicks.

The bug-fix in version 0.9.12 of Rcpp turned out to be incomplete, so a new version 0.9.13 is now on CRAN and will get to Debian shortly. The Rcpp::Enviroment constructor is now properly fixed (using the global environment as a default value). As well, a new #define was created to detect the clang++ compiler before/after version 3.0.0 as one exceptions header changed location. Unit tests files were also once more updated. The complete NEWS entry for 0.9.13 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.

0.9.13  2012-06-28
    o   Truly corrected Rcpp::Environment class by having default constructor
        use the global environment, and removing the default argument of
        global environment from the SEXP constructor
    o   Added tests for clang++ version to include bits/exception_defines.h
        for versions 3.0 or higher (similar to g++ 4.6.0 or later), needed to
        include one particular exceptions header
    o   Made more regression tests conditional on the RunAllRcppTests to come
        closer to the CRAN mandate of running tests in sixty seconds
    o   Updated unit test wrapper tests/doRUnit.R as well as unitTests/runTests.R

Thanks to CRANberries, you can also look at a diff to the previous release 0.9.12. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Starting next week, I'll start at new job. I will still stay at SUSE, so it's not that big change, but still it is new challenge for me. I'll be moving out of L3 department, where I've spend almost four years. L3 is really a great team where man can learn basically anything, simply because you can meet there anything from broken swap on S390 up to wrong icons in LibreOffice :-). The new position is inside Security team, still I'll be mostly focused on writing some internal tools, rather than doing security work (at least for first year or so). The current plan is base on work I've done for L3 tools and use Django for the web interface, but that's just a plan for now and might change in the future. Anyway looking forward to new challenges at new job.

Filed under: Django English Suse 0 comments Flattr this!

OASIS is a tool to integrate a configure, build and install system in your OCaml project. It helps to create standard entry points in your build system and allows external tools to analyse your project easily. It is the building brick of OASIS-DB, a CPAN for OCaml. This release takes almost 18 months to complete. This is too long and I will talk in another blog post on the way I am trying to improve this right now (esp. using continuous integration). This new release fixes a small bug (1 line) that prevents setup.ml to run with OCaml 4.00. If you have projects that was generated with a former release consider upgrading to OASIS 0.3.0. There are several big new stuff that comes with this release. It now supports Pack: field for libraries which allows to pack your library using -for-pack and so on. We are also compiling .cmxs (dynlink object for native) by default and we publish them in the META file. This feature was implemented in order to get more libraries to provide .cmxs and to help project like Ocsigen to take advantage of this. If you want to get rid of this at configure time, you can use ./configure --override native_dynlink false. We introduce two new default flags: --enable-tests and --disable-docs for configure. These are implicit flags that define if we will run Test sections or compile Document sections. They are especially useful to reduce the number of dependencies, because dependencies of Test will be excluded by default. We recommend to set Build$: flag(tests) to any Library or Executable sections that are only useful for tests. This allows you to really cut down the number of dependencies. The last change I want to introduce is about the old setup-dev subcommand which is now deprecated. It has been replaced by 2 different update schemes. I am pretty excited by this feature which in fact comes from OASIS user (esp. Lwt project). The former scheme was to have a big 'setup.ml' that always call the command oasis to update. This was complex and not very useful. We now have 2 mode: dynamic and weak. dynamic allow you to have a small setup.ml and to keep your VCS history clean but you need to install OASIS. weak need a big setup.ml but only need to call the command oasis if someone change something in _oasis. This mode is targeted to project that wants to be able to checkout from VCS an OCaml project without installing OASIS. The difference generated by weak mode doesn't pollute too much the VCS history because most the time they make sense. For example, you upgrade your package version number in _oasis and it produces a change with 6 lines where the version number changes in every META, setup.ml files and so on. This version has been tested on:

• Linux (Debian Sid and Debian Squeeze with GODI, OCaml 3.12)
• Windows Vista with MS Visual Studio 2008 toolchain, OCaml 3.12
• Mac OS X, OCaml 3.12 from GODI

You can download it here or use your favorite package manager: Debian (UPDATE: pending upload)

$ apt-get install -t experimental oasis

GODI

$ godi_console perform -build apps-oasis

odb.ml

$ ocaml odb.ml oasis

Here is the complete changelog: EXTREMLY IMPORTANT changes (read this)

• Fix bug with scanf %S@\n for ocaml 4.00. We were unfortunetaly using an undocumented tolerance of Scanf in the previous version. You should consider making new release using this version that fixed this.

PACKAGES uploaded to oasis-db will be automatically "derived" before OCaml 4.00 release (i.e. oUnit v1.1.1 will be regenerated with this new version as oUnit v1.1.1~oasis1). PACKAGES not uploaded to oasis-db need to be regenerated. In order not to break 3rd party tools that consider a tarball constant, I recommend to create a new version. Thanks to INRIA OCaml team for synchronizing with us on this point. Major changes:

• Handle the field Pack: true to be able to create packed libraries. It also installs .mli files for documentation into the target directory. The pack option is only supported for OASISFormat: 0.3, you will need to update the version of your _oasis file to match it.

• Introduce --enable-tests to disable tests and docs at oasis level. It seems a very common pattern to have a Flag tests to turn off by default the tests. This is now define as a standard var and you should remove you previous Flag tests but you can continue to use flag(tests) where needed.

You can now have the following example:

     ...
     Executable test_exec
       Install: false
       Build$: flag(tests)
       MainIs: testExec.ml
       BuildDepends: oUnit
     
     Test main
       Command: $test_exec
       TestTools: test_exec
     ...

The Run$: flag(tests) is implicit for the section Test main. The default value is false for tests. If all the executable for test are flagged correctly (Build$: flag(tests)), you'll get rid of the dependency on oUnit. It works the same for documentation, however the default is true. (Closes: #866)

• Allow to define interdependent flags

In order to allow interdependent flags, we transform back lazy values into unit -> string functions. This allows to change a flag value on the command line and to update all the dependent values. (Closes: #827, #938)

• Deprecate the subcommand oasis setup-dev in favor of oasis setup -setup-update none weak dynamic

It defines different ways to manage the auto-update of setup.ml:

• none: this is the default mode, and the one you should use when distributing tarballs. No update are performed at al.
• weak: the update is only triggered when something change in _oasis , we keep all files generated
• dynamic: the content of 'setup.ml' is ultra small (<2kB) and we only keep a small setup.ml, Makefile and configure.

The choice between weak and dynamic depends on your need with regard to VCS and to the presence of oasis. The weak allow to checkout the project from VCS and be able to work on it, without the need of installing 'oasis' as long as you don't change the file _oasis. But it clutter your VCS history with changes to the build system each time you change something in _oasis. The 'dynamic' mode gives you no VCS history pollution but makes mandatory to have installed oasis libraries.

• Don't copy executable in ocamlbuild

Avoid copying executable to their real name. It helps to call ocamlbuild a single time for the whole build rather than calling it n time (n = number of executable sections) and copying resulting exec. This speeds up the build process because ocamlbuild doesn't have to compute/scan dependencies each time. The drawback is that you have to use $foo when you want to call Executable foo, because $foo will be '_build/.../main.byte'.

• Change the way we parse command line like option (CCOpt, CCLib and the like). We have implemented a real POSIX command line parser, except that variables are processed by Buffer.add_substitute (except if correctly escaped, using Buffer.add_substitute escaping).

For example:

CCOpt: -DEXTERNAL_EXP10 -L/sw/lib "-framework vecLib"

Will be parsed correctly and outputed according to target OS.

• Minimize the dependencies of the project.

In order to ease building oasis, we have minimize the number of dependencies. You only need to install ocamlmod, ocamlify and ocaml-data-notation for a standard build without tests. Dependencies on pcre, extlib and ocamlgraph has been dropped. The remaining dependencies are hidden behind a flag tests.

• Implement proposal for handling plugin in META, using "plugin" extra directives

OASIS now produces .cmxs file by default and add them to META. Now a META looks like:

     ...
     archive(byte) = "oasis.cma"
     archive(byte, plugin) = "oasis.cma"
     archive(native) = "oasis.cmxa"
     archive(native, plugin) = "oasis.cmxs"
     ...

This will ultimately help to generate automatically .cmxs for all oasis enabled projects. We hope that this new feature will improve dynamic linking use in OCaml (esp. for project like Ocsigen). Other changes

• Produce only subcommands specific helps when asked (Closes: #783)
• Use pager for long textual output (Closes: #782)
• Give the help summary directly when running just "oasis" (Closes: #781)
• Remove pwd filename prefix to avoid too long command line on Windows (Closes: #1007)
• Install required .cmx when installing libraries (Closes: #686, #885)
• Retain blanks at the beginning of line in freeform (Closes: #811)
• Allow PreConfCommand to create setup.data with any variables inside (Closes: #823)
• Allow to use setup.data in shell script and Makefile (Closes: #982)
• Fix execute permissions (Closes: #980)
• Drop extra version in ocaml version (Closes: #964, #888)
• Always output info messages on stderr (Closes: #785)
• Fix interversion in version comparator lexer (Closes: #808)
• Document the fact that setup.data is mandatory (Closes: #946)
• configure translates getopt command line to Arg (Closes: #778, #804)
• Generate valid META when there are line breaks in synopsis (Closes: #933)
• Keep owner of generated files (Closes: #793)
• Set -I +threads for ocamlfind ocamldoc in ocamlbuild (Closes: #949)
• Exclude VCS directories by default (Closes: #847)
• Print the configure summary in the "normal" order.
• Merge findlib virtual containers with the same name
• Synchronize plugin data for section
• Improve documentation
• Verify validity of input when parsing in quickstart (Closes: #797)
• Fix stack overflow when there are twice the same library (Closes: #1130)
• Parse license examples for DEP5 specification (Closes: #1102)
• Be ultra-liberal regarding version syntax (allow '_' and ' ' in the version).
• Allow to add extra content to META files (XMETAExtraLines)
• Allow to use executable name that contains "-" as a variable (replacing "-" by "_").

Thanks to Anil Madhavapeddy, Pierre Chambart, Christophe Troestler, Jeremie Dimino, Ronan Le Hy, Yaron Minsky and Till Varoquaux for their help with this release. Also thanks to all the testers of the numerous release candidates. This was a long work and each time a tester has downloaded oasis has helped me to know that I was working for someone.

The DebConf12 schedule is finally available! As usual, the DebConf conference can only be finalized a couple of days before DebConf itself starts. But hey Take a look at it, and start planning your activities! Not final, never final Remember that, even now that the conference schedule is announced, you can still submit talks to DebConf, following what we call ad-hoc talks. What are the steps to do it?

1. Check the schedule and find the timeslot (day, hour and room) you want to talk in
2. Register your event in Pentabarf, and note the generated event ID (the last number in the URL after the submission is accepted)
3. Send a mail to schedule@debconf.org detailing the event ID, title (just to confirm it), and the requested timeslot
4. All requests _must_ be received at least 24hr before the requested event for the scheduling team to act accordingly and for the video team to be able to plan on resources to cover the requested session
5. Video team coverage (event recording plus streaming on a best-effort basis) will be given to activities scheduled on the Aula Magna and Roberto Ter n auditoriums.
6. We will try to grant requests on a first-come-first-serve basis. If two events are requested for the same timeslot, the first will have precedence, and we will schedule the second as close as possible to it. We will notify the submitters if this happens, of course.

Conference tracks In DebConf12, many talks are organized around the following tracks:

DebianDay

This track is meant for the local (Nicaraguan/Central American) community to be attracted to Debian. It does not usually have technically minded talks; the DebConf general audience is, of course, invited to take part in it, as there will surely be interesting angles on how Debian is used in different use-cases, and there will be interesting Q&A session with the audience.

Getting involved in Debian

Central America is a very special region. It has vibrant Free Software user groups and communities, among which there is an important interest to get involved in Debian.
This track will have some talks in Spanish, some talks in English, explaining different Debian processes and tools, aiming at attracting locals to contribute to Debian.

Debian for the cloud

All about virtualization. Management infrastructure, details on virtual machine engines, insight on how to tweak your kernel for optimum virtualized performance, and of course how to get it all humming perfectly with your Debian packages.

Building and porting

With an every day stronger transit to mobile computing, we are seeing a renaissance of porting. At the same time, we finally have multiarch as part of Debian. And were that not to be enough, we finally have more than one compiler that can be used for most of the software in the archive. Put all this together, and you will have a very interesting track.

Social activities

DebConf is as much about technical stuff as it is about building closer ties within the members of its community. Eating cheese+wine, going on a trip to the guts of a volcano, having a nice banquet Do you want to add an activity to this pseudo-track?

Skills exchange

Knowledge can also be built by all of us putting our grain of sand. We first had explicit skills exchange sessions in Bosnia last year, and they were very well received. Feel free to propose similar sessions!

And Well, three days to DebCamp and counting. See you Real Soon Now in Managua!

I've been too busy at home, but finally I found time to wrap up another interview with the people behind Debian Edu and Skolelinux. This time we get to know Jos Luis Redrejo Rodr guez, one of our great helpers from Spain. His effort was the reason we added support for several desktop types (KDE, Gnome and most recently LXDE) in Debian Edu, and have all of these available in the recently published Debian Edu Squeeze version. Who are you, and how do you spend your days? I'm a father, teacher and engineer who is working for the Education ministry of the Region of Extremadura (Spain) in the implementation of ICT in schools How did you get in contact with the Skolelinux/Debian Edu project? At 2006, I verified that both, we in Extremadura and Skolelinux project, had been working in parallel for some years, doing very similar things, using very similar tools and with similar targets, so I decided it was time to join forces as much as possible. What do you see as the advantages of Skolelinux/Debian Edu? A community of highly skilled experts working together, with a really open schema of collaboration and work. I really love the concepts of Do-ocracy and Merit-ocracy and the way these concepts are been used everyday inside Debian Edu. What do you see as the disadvantages of Skolelinux/Debian Edu? Sometimes the differences in the implementations, laws or economical and technical resources in the different countries don't allow us to agree in the same solution for all of us, and several approaches are needed, what is a waste of effort. Also, there is a lack of more man power to be able to follow the fast evolution of the technologies in school. Which free software do you use daily? Debian, of course, and due to my kind of job I am most of my time between Iceweasel, Geany and Terminator. Which strategy do you believe is the right one to use to get schools to use free software? I think there is not a single strategy because there are very different scenarios: schools with mixed proprietary and free environments, schools using only workstations, other schools using laptops, netbooks, tablets, interactive white-boards, etc. Also the range of ages of the students is very broad and you can not use the same solutions for primary schools and secondary or even universities. So different strategies are needed. But, looking at these differences, and looking back to the things we've done and implemented, and the places were we have spent most of our forces, I think we should focus as much as possible in free multi-platform environments, using only standards tools, and moving more and more to Internet or network solutions that could be deployed using wireless. I think we'll see more and more personal devices in the schools, devices the students and teachers will take home with them, so the solutions must be able to be taken at home and continue working there.

Review: Quiet, by Susan Cain

Publisher:	Crown
Copyright:	2012
ISBN:	0-307-45220-4
Format:	Kindle
Pages:	263

I've always been an introvert. This is something that surprises some people when they first meet me since they equate introversion with shyness, and I'm not at all shy. It surprises others because I'm quite vocal and outspoken in meetings, but introversion also doesn't imply a lack of self-confidence. I can run meetings, give presentations, and argue my perspective in front of just about anyone, but I don't like parties, I crave time by myself, and I could happily go for weeks without seeing another human being. I'm an introvert because I find people draining rather than invigorating, written communication far easier and more comfortable than spoken, and superficial social contact more irritating and frustrating than enjoyable. If you think think that means there may be something wrong with me, or that I would be happier if "drawn out of my shell," I wish you would read this book. But I suspect its core audience will be people like me: those who are tired of being pushed to conform with extrovert beliefs about social interaction, those who are deeply disgusted by the word "antisocial" or feel pangs of irrational guilt when hearing it, or those who just want to read an examination of interpersonal interactions that, for once, is written by and about people who like quiet and solitude just like they do. I first encountered Susan Cain via her TED talk, which I think is both the best possible summary of this book and the best advertisement for it. If you've not already seen it, watch it; it's one of the best TED talks I've seen, good enough that I've watched it three times. If you then want more of the same, buy Quiet. Quiet has, I think, three messages. First, it's a tour of the science: what is introversion and extroversion? Is there evidence that these are real physiological differences? (Spoiler: yes.) What do we know about introversion? How do we know those things what experiments have been done and what methods have been used? Here, it's a good general introduction, although Cain is careful to point out that it only scratches the surface and there's much more scientific depth. For example, she touches on the connections between introversion and sensitivity to stimulus and points out that they're two separate, if related, categorizations, but doesn't have the space here to clarify the distinctions and tease them apart. But she lays a reasonable foundation, particularly in defense of introversion as a natural, physiologically grounded, scientifically analyzable, common, and healthy way of interacting with the world. (For those who are curious about the distinctions between introversion and sensitivity, and the argument that most of what Cain says here about introversion is actually about sensitivity, see the blog post by Elaine Aron.) The second message, the one that resonated with me the most, was Cain's passionate defense of introversion. Business culture (at least in the United States, which is what both Cain and I know) is strongly biased towards extroversion; at least faking extroversion seems to be required for some career advancement. Extrovert culture dominates politics and most public discourse. It's common to find people who consider introversion, particularly in children, to be a sign of unhappiness, poor social adjustment, psychological problems, or other issues that should be "cured" or changed. Cain's gentle but firm passion in defense of introversion is a breath of fresh air. She attacks open plan offices, the current obsession with group learning and social school settings, and the modern group-think bias towards collaboration over solitude and concentration, and she does that with a combination of polite frustration and the conclusions of multiple studies. Introverts will be cheering as she constructs solid arguments and musters evidence against things that we've always found miserable and then been told we were wrong, short-sighted, or socially inept for finding miserable. I am so utterly on her side in this argument that I have no way of knowing how persuasive it will be, but it's lovely just to hear someone put into words what I feel. This defense does skew the book. Quiet is not, and does not purport to be, an even-handed presentation of introversion and extroversion. It's written proudly and unabashedly from the introvert's point of view. I'm fine with that: I, like Cain, think the US is saturated in extrovert perspectives and extrovert advice, particularly in the business world, and could use some balancing by activism from the other perspective. But be aware that this is not the book to look to for an objective study of all angles of the introvert/extrovert dichotomy, and I'm not sure her descriptions of extroversion are entirely fair or analogous to those of introversion. The extroversion described here seems somewhat extreme to me. I'm dubious how many extroverts would recognize themselves in it, which partly undermines the argument. The third message of the book, once Cain has won the introvert's heart, is some advice on how to be a proud introvert, to make the space and find the quiet that one desires, and to balance that against places where one may want and need to act like an extrovert for a while. Cain thankfully does not try to make this too much of the book, nor does she hold up any particular approach as The Answer. All the answers are going to be individual. But she does offer some food for thought, particularly around how to be conscious of and make delibrate choices about one's energy expenditures and one's recharge space. She also captures beautifully something that I've not seen explained this well before: the relief that an introvert can feel in the company of an extrovert who helps navigate social situations, make conversation, and keep discussions going until they can reach the depth and comfort level where the introvert can engage. I wish anyone in a position of authority over social situations would read this book, or at least watch the TED talk and be aware of the issues. Particularly managers, since (at least in my relatively limited experience) workplace culture is so far skewed towards extroversion that it can be toxic to introverts. Many of the techniques used by extrovert managers, and the goals and advice they give their employees, are simply wrong for introverts, and even damaging. Cain speaks well to the difficulties of empathy between people with very different interaction preferences, such as the problems with extroverts trying to "draw out" introverts who have hit social overload (or sensitive people who have hit stimulus overload). She also discusses something that I'd previously not thought about, namely how the pressure towards extroversion leads people to act extroverted even when naturally introverted, and how it's therefore very difficult to tell from behavior (and sometimes even to tell internally!) what one's natural interaction style is. But mostly I recommend this book if you're an introvert, if the TED talk linked above speaks to you. Even if we can't convince the world to respect introversion more, or at least stop treating it as abnormal, it's a lovely feeling to read a book from someone who gets it. Who understands. Who fills a book with great stories about introverts and how they construct their worlds and create quiet space in which to be themselves. Rating: 9 out of 10

A bug-fix release 0.9.12 of Rcpp arrived earlier today on CRAN and is now in Debian too. This fixes a minor snafu with the Rcpp::Enviroment constructor following a small change made for 0.9.11. It also reduces the number of unit tests running by default as CRAN was complaining that it took too long to run the checks. I am not impressed---tests are for running, not skipping. Lastly, the cleanup needed to be extended to another directory. The complete NEWS entry for 0.9.12 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.

0.9.12  2012-06-23
    o   Corrected Rcpp::Environment class by removing (empty) ctor following
        rev3592 (on May 2) where default argument for ctor was moved
    o   Unit testing now supports argument --allTests to impose that all
        tests are executed; otherwise some expensive tests are skipped. This
        is arguably not the right thing to do, but CRAN maintainers insist 
        on faster tests.
    o   The cleanup script now also considers inst/unitTests/testRcppClass/src 

Thanks to CRANberries, you can also look at a diff to the previous release 0.9.11. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Today, photo-uploader just got it's new maintainer. I've basically lost interest in maintaining it two years ago, what resulted in dropping package from Debian year ago. I thought this is end of the software, but I seem to be wrong. As a consequence of Debian bug #677220 Andrew Shadura stepped up as an upstream maintainer. At least Imageshack support is now working in git and maybe some new features will come or the package will be reintroduced to Debian. But that's all up to Andrew. Good luck!

Filed under: English Photo-Uploader 0 comments Flattr this!

I ve begun work on desktop-base to get it ready for wheezy. As many of you know, we saw the SpaceFun theme with Squeeze, and currently Wheezy as well. Rather then see the same theme twice, I ve taken the liberty of updating desktop-base to use the Joy theme proposed for Wheezy. My announcement can be seen on lists.debian.org, and details can be found there. Rather then drone on, check out what s been done so far. We ll be going from this: To: Game on!

I ve just uploaded a new SE Linux policy for Debian/Wheezy. It now works correctly with systemd and Chromium, two significant features that I wanted for Wheezy. Now it turns out that we have until the end of the month for Wheezy updates, so I may get another version of the policy uploaded before then. If so it will only be for relatively minor changes, I think that most SE Linux users would be reasonably happy with policy the way it is. Anything that doesn t work now can probably be solved by local configuration changes. execmem The current version of KDE in Debian is 4.8.4, it seems that large parts of the KDE environment depend on execmem access, this includes kwin and plasma-desktop. Basically there is no possibility of having a KDE desktop environment without those programs and therefore KDE depends on execmem access. Debugging this is difficult as the important programs SEGV when denied execmem access and the KDE crash handler really gets in the way of debugging it running /usr/bin/plasma-desktop results in the process forking a child and detaching from the gdb session. The most clear example of an execmem issue in KDE is from the program /usr/lib/kde4/libexec/kwin_opengl_test which gives the following error:
LLVM ERROR: Allocation failed when allocating new memory in the JIT
Can t allocate RWX Memory: Permission denied To make this work you run the command setsebool -P allow_execmem 1 which gives many domains the ability to create writable-executable memory regions. I raised this issue for discussion on the SE Linux mailing list and Hinnerk van Bruinehsen wrote an informative message in response summarising the situation [1]. It seems that it s possible to compile some of the programs in question to not use the JIT and therefore not require such access and there is a build option in Gentoo to allow it. But it s impractically difficult for me to fork KDE in Debian so the only option is to recommend that people enable the allow_execmem boolean for Debian desktop systems running SE Linux.

• [1] http://marc.info/?l=selinux&m=134011618909818&w=2

Related posts:

1. /run and SE Linux Policy Currently Debian/Unstable is going through a transition to using /run...
2. An Update on DKIM Signing and SE Linux Policy In my previous post about DKIM [1] I forgot to...
3. New SE Linux Policy for Squeeze I have just uploaded refpolicy version 0.2.20100524-1 to Unstable. This...

I am trying to get used to fish as a default shell. I like some things in it, but can not quite get used to other things, so I wonder maybe I am cooking it wrong? So here are things that I could not find a solution for, while switching from bash.

• bash$ DISPLAY=:1 gedit very simple thing, how do I set and environment variable for just one command execution in the most simple possible way.
• bash$ scp hostname:somedir/*.jpg . when I try to run this in fish, for some reason I get fish: Warning: No match for wildcard hostname:somedir/*.jpg . The command will not be executed. and I could not figure out how to actually run the command in a quick Googling.
• I have (almost) come to terms that while and for syntax is different, but I still wonder maybe there is some bash compatibility mode that I missed that would allow me to use my old ways?

I do like the colors, the shortened cwd in the prompt, the univeral history and univeral variables, lower memory usage and funkier completions. Any other fish features people like? Especially I am interested in features that are not enabled by default.

Two years ago, Jacob (then 3) and I built his first computer together. I installed Debian on it, but never put a GUI on the thing. It s command-line, and has provided lots of enjoyment off and on over the last couple of years. I ve written extensively about what our boys like to do, and the delight they have at learning things on the command line. The looks of shock I get from people when I explain, as if it s perfectly natural, that my child has been able to log in by himself to a Linux shell since age 3, are amusing and astounding. Especially considering that it is really not that hard. Instead of learning how to run an Xbox, he s learned how to run bash. I like that. Lately, Jacob (now 5) hasn t been spending much time with it. He isn t really at a stage where he wants to push his limits too far, I think, but yet also gets bored with the familiar. So I thought it was time to introduce a GUI in a limited fashion, perhaps to let him download photos and video from his Vtech toy camera (that takes real low-res photos and videos which can be downloaded over a USB1 link). He s familiar with the concept, at least somewhat, having seen GUIs on Terah s computer (Gnome 2) and mine (xfce4 + xmonad). So last night, Oliver (age 2) and I went down to the basement on a mouse-finding expedition. Sure enough, I had an old PS/2 mouse down there that would work fine. The boys both helped string it through the desk up on our play room, and were tremendously excited to see the red light underneath it when the computer came on. Barely able to contain the excitement, really. A bit like I remember being when I got my first mouse (at a bit of an older age, I suppose.) I helped him them in as root for the very first time. (Jacob typed root , and I typed the password, and provided the explanation for why we were telling the computer we were root .) Jacob and Oliver alternated typing bits of some apt-get command lines. Then while we waited for software to download, I had to answer repeated questions of how soon will the mouse work? and what does install mean? Finally it was there, and I told Jacob to type startx. I intentionally did not install a display manager; more on that later. He pressed Enter, the screen went blank for about 5 seconds, and then X appeared. Excited can t begin to describe how they acted. They took turns playing with the mouse. They loved how the trash can icon (I started with XFCE) showed trash IN the trash can. But they are just learning the mouse, and there s a lot about a typical GUI that is unfriendly to someone that isn t yet proficient with a mouse. The close buttons are disappointingly small, things can be too easily dragged on and off the panel and menus. When I sat down to think about it, the typical GUI design does not present a very good it always works the same interface that would be good for a child. And then it occurred to me: the perfect GUI for a child would be simply xmonad (a tiling window manager that can be controlled almost entirely by keyboard and has no need for mouse movements in most cases.) No desktop environment, no file manager in the root window. Just a window manager in the classic X way. Of course! So after the boys were in bed, I installed xmonad. I gave Jacob s account a simple .xsession that starts a terminal and xmonad. Today, Jacob informed me that he wanted his computer to look just like yours. Playing right into my hands, that was! But when he excitedly typed startx, he said it wasn t just like mine. Uh oh. Turns out he wanted the same wallpaper as my computer uses. Whew. We found it, I figured out that xli(1) loads it in the root window, and so I added a third line to .xsession. More delight unlocked! Jacob mastered the basics of xmonad really quickly. Alt-Shift-C to close a window. Alt-Shift-Q to quit back to the big black screen . Alt-Shift-Enter to get a terminal window. We launched thunar (the XFCE file manager) and plugged in his camera. He had a good deal of fun looking at photos and videos from it. But then I dropped the true highlight of the day for him: I offered to install Tuxpaint for him. That s probably his favorite program of all time. He watched impatiently as apt-get counted down 1m30s for tuxpaint and its libraries. Then we launched it, and he wanted to skip supper so he could keep playing Tuxpaint on my VERY OWN COMPUTER! I d been debating how to introduce GUIs for a very long time. It has not escaped my attention that children that used Commodores or TRS-80s or DOS knew a lot more about how their computers worked, on average, than those of the same age that use Windows or MacOS. I didn t want our boys to skip an entire phase of learning how their technology works. I am pleased with this solution; they still run commands to launch things, yet get to play with more than text-based programs. At bedtime, Jacob asked me, very seriously: Dad, how do I start tuxpaint again? First you log in and type startx. Then you can use the mouse. Jacob nods, a contemplative look on his face.. Then, I continue, you type tuxpaint in the terminal, and it comes right up. Jacob nodded very seriously a second time, as if committing this very important information to long-term memory. Then gave a single excited clap, yelled Great! , and dashed off.

Sarah and I are running in the 5K Packard Summer Scamper on Saturday. Sarah's been working on her fitness for a few months now, and I belatedly started working on mine about a 6 weeks ago. (Sarah's fund raising, so if you'd like to sponsor her, her page is here) I've been using the Couch to 5K program to get my fitness up, but I haven't managed to stick to it exactly, due to a combination of injury and wet weather. I have still managed to get myself to the point where I can run for 28 minutes solidly, which is close to 5K (I'm doing my last training run tomorrow, so I'll see what the distance comes out to). I've been doing up to three runs a week at lunchtime around the Shoreline Amphitheatre area, using my Galaxy Nexus and some Bluetooth headphones, and I've got a pretty nice little setup. I'm using Pandora for some music to listen to, C25K Pro for the actual running intervals. It has voice prompts that tell me when to run, or to start a walk interval. Finally, I'm using Strava Run for recording the run and getting some additional statistics during the run (kilometre markers, split times and time elapsed). These also come through as voice prompts. I'm not currently wearing a watch, and I want to keep track of the wall time, so I've also found Audio Clock, which verbally announces the time (I've set it to do it every 15 minutes that a Bluetooth headset is connected). What I really like about all of this is that I get spoken voice prompts and information, seamlessly mixed in with the music from Pandora, all while I'm running. I've got my phone strapped to my arm, and I don't have to look at it during the run, and there's no wires. I'm a huge personal metrics junkie, and that was why I previously eschewed running outdoors. I wanted that information on the treadmill display so I could push myself that little bit further. Sure, I'm not getting the constant information as much as if I had a treadmill display in front of me the whole time, but I'll settle for running in the great outdoors and getting some feedback every kilometre (if I get really desperate, Strava will do it every half kilometre). Not to diss optical augmented reality efforts too much, but I'd rather go for a combination of discreet audio announcements through a Bluetooth earpiece, perhaps complemented with something I choose to look at like a Pebble watch, than an in-your-face pair of glasses. My Bluetooth headphones are the around the back of the head variety that aren't going to bounce off during a run. I've really enjoyed doing the runs during the week, and when I was doing the full 3 runs a week of a given week of the Couch to 5K program, I'd really notice a difference by run number 3. I'm currently planning on sticking with doing a 5K run hopefully at least twice a week after this weekend. It's certainly done wonders for my waistline.

I have an interesting problem: How do I shoehorn "hired by The Internet for a full year to work on Free Software" into my resume? Yes, the git-annex Kickstarter went well. :) I had asked for enough to get by for three months. Multiple people thought I should instead work on it for a full year and really do the idea justice. Jason Scott was especially enthusiastic about this idea. So I added new goals and eventually it got there. Don Marti thinks the success of my Kickstarter validates crowdfunding for Free Software. Hard to say; this is not the first Free Software to be funded on Kickstarter. Remember Diaspora? Here's what I think worked to make this a success:

• I have a pretty good reach with this blog. I reached my original goal in the first 24 hours, and during that time, probably 75% of contributions were from people I know, people who use git-annex already, or people who probably read this blog. Oh, and these contributors were amazingly generous.
• I had a small, realistic, easily acheivable goal. This ensured my project was quickly visible in "successful projects" on Kickstarter, and stayed visible. If I had asked for a year up front, I might not have fared as well. It also led to my project being a "Staff Pick" for a week on Kickstarter, which exposed it to a much wider audience. In the end, nearly half my funding came from people who stumbled over the project on Kickstarter.
• The git-annex assistant is an easy idea to grasp, at varying levels of technical expertise. It can be explained by analogy to DropBox, or as something on top of git, or as an approach to avoid to cloud vendor lockin. Most of my previous work would be much harder to explain to a broad audience in a Kickstarter. But this still appeals to very technical audiences too. I hit a sweet spot here.
• I'm enhancing software I've already written. This made my Kickstarter a lot less vaporware than some other software projects on Kickstarter. I even had a branch in git where I'd made sure I could pull off the basic idea of tying git-annex and inotify together.
• I put in a lot of time on the Kickstarter side. My 3 minute video, amuaturish as it is, took 3 solid days work to put together. (And 14 thousand people watched it... eep!) I added new and better rewards, posted fairly frequent updates, answered numerous questions, etc.
• I managed to have at least some Kickstarter rewards that are connected to the project is relevant ways. This is a hard part of Kickstarter for Free Software; just giving backers a copy of the software is not an incentive for most of them. A credits file mention pulled in a massive 50% of all backers, but they were mostly causual backers. On the other end, 30% of funds came from USB keychains, which will be a high-quality reward and has a real use case with git-annex. The surprising, but gratifying part of the rewards was that 30% of funds came from rewards that were essentially "participate in this free software project" -- ie, "help set my goals" and "beta tester". It's cool to see people see real value in participating in Free Software.
• I was flexible when users asked for more. I only hope I can deliver on the Android port. Its gonna be a real challange. I even eventually agreed to spend a month trying to port it to Windows. (I refused to develop an IOS port even though it could have probably increased my funding; Steve's controlling ghost and I don't get along.)
• It seemed to help the slope of the graph when I started actually working on the project, while the Kickstarter was still ongoing. I'd reached my original goal, so why not?

I've already put two weeks work into developing the git-annex assistant. I'm blogging about my progress every day on its development blog. The first new feature, a git annex watch command that automatically commits changes to git, will be in the next release of git-annex. I can already tell this is going to be a year of hard work, difficult problems, and great results. Thank you for helping make it happen.

(This blog post is a more firm version of a series of tweets and forum posts I made a few weeks ago. This should also be considered a refresh of this post by Mirco Bauer a few years ago) It has been said that Mono is bloated, and that people should use lighter frameworks that don t pull in hundreds of meg . How much basis in reality is there for those comments? Well, let s do a little thought exercise. How much space is the minimum space needed for several languages to work? For this experiment, we will be looking at the required installation size on disk of several language frameworks, each time installing the bare minimum for a command-line hello world app in that language to run. For example, the Ruby result is the minimum install required to run:

puts 'Hello world'

and the Mono result is the minimum install required to run the following code, which has been compiled on a different machine using dmcs :

public class Hello1
 
   public static void Main()
    
      System.Console.WriteLine("Hello, World!");
    
 

Obviously, for dynamic languages like Ruby and Python, there is no compiler step needed. For languages like C# and Java, it is fair to compile these on a different machine as end-users of packages running these frameworks receive binaries, not source they do not need a development environment. So, on to the test environment. This is a bare bootstrapped AMD64 Debian Unstable system, as of today (i.e. debootstrap sid /tmp/badger ). Its install size is 275MB. So how much space? Ruby 1.9 Test code is as follows:

puts 'Hello world'

Install requirements are as follows:

root@dream:/# aptitude install ruby1.9.1
The following NEW packages will be installed:
  libffi5 a  libruby1.9.1 a  libyaml-0-2 a  ruby1.9.1 
0 packages upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 4703 kB of archives. After unpacking 13.1 MB will be used.

OpenJDK 7 Test code is as follows:

class HelloWorldApp  
    public static void main(String[] args)  
        System.out.println("Hello world");
     
 

Install requirements are as follows:

root@dream:/# aptitude install openjdk-7-jre-headless
The following NEW packages will be installed:
  ca-certificates a  ca-certificates-java a  dbus a  fontconfig-config a  
  icedtea-7-jre-cacao a  icedtea-7-jre-jamvm a  java-common a  
  krb5-locales a  libavahi-client3 a  libavahi-common-data a  
  libavahi-common3 a  libcap2 a  libcups2 a  libdbus-1-3 a  libexpat1 a  
  libffi5 a  libfontconfig1 a  libfreetype6 a  libglib2.0-0 a  
  libglib2.0-data a  libgssapi-krb5-2 a  libjpeg8 a  libk5crypto3 a  
  libkeyutils1 a  libkrb5-3 a  libkrb5support0 a  liblcms2-2 a  libnspr4 a  
  libnss3 libnss3-1d a  libpcre3 a  libpcsclite1 a  libsystemd-login0 a  
  libxml2 a  openjdk-7-jre-headless openjdk-7-jre-lib a  openssl a  
  sgml-base a  shared-mime-info a  ttf-dejavu-core a  tzdata-java a  ucf a  
  xml-core a  
0 packages upgraded, 43 newly installed, 0 to remove and 0 not upgraded.
Need to get 51.5 MB of archives. After unpacking 137 MB will be used.

Python 3.2 Test code is as follows:

print ("Hello world")

Install requirements are as follows:

root@dream:/# aptitude install python3.2-minimal
The following NEW packages will be installed:
  file a  libexpat1 a  libffi5 a  libmagic1 a  mime-support a  python3.2 a  python3.2-minimal 
0 packages upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 4853 kB of archives. After unpacking 17.7 MB will be used.

Mono 2.10 (C#) Test code is as follows:

public class Hello
 
   public static void Main()
    
      System.Console.WriteLine("Hello world");
    
 

Install requirements are as follows:

root@dream:/# aptitude install mono-runtime
The following NEW packages will be installed:
  binfmt-support a  cli-common a  libmono-corlib4.0-cil a  libmono-i18n-west4.0-cil a  libmono-i18n4.0-cil a  
  libmono-security4.0-cil a  libmono-system-configuration4.0-cil a  libmono-system-security4.0-cil a  
  libmono-system-xml4.0-cil a  libmono-system4.0-cil a  mono-4.0-gac a  mono-gac a  mono-runtime 
0 packages upgraded, 13 newly installed, 0 to remove and 0 not upgraded.
Need to get 4383 kB of archives. After unpacking 11.5 MB will be used.

Of course, every time I try to make the argument that Mono is much leaner than what people propose as an alternative, the anti-Mono brigade go on the defensive and insist that nobody in their right minds uses Python or Ruby either everything ever should be written in Qt. So, hypothetically Qt 4.8 (C++) Test code is as follows:

#include <QTextStream>
int main(int argc, char **argv)
 
   QTextStream s(stdout);
   s << "Hello, world!\n";
   return 0;
 

Install requirements are as follows:

root@dream:/# aptitude install libqtcore4
The following NEW packages will be installed:
  libffi5 a  libglib2.0-0 a  libglib2.0-data a  libpcre3 a  libqtcore4 
  libxml2 a  sgml-base a  shared-mime-info a  xml-core a  
0 packages upgraded, 9 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.3 MB of archives. After unpacking 27.8 MB will be used.

Presented without comment for your consideration. Edit: There are complaints that these numbers are misleading, because they show default package manager behaviour, rather than what happens when passing extra flags or config file settings to Apt. When disabling installation of Recommends: packages, the numbers change as follows:

• openjdk-7-jre-headless 137 MB -> 116 MB
• ruby1.9.1 13.1 MB -> 13.1 MB
• python3.2-minimal 17.7 MB -> 6.1 MB
• mono-runtime 11.5 MB -> 10.9 MB
• libqtcore4 27.8 MB -> 13.7 MB

A post by Gunnar reminded me that I wanted to blog about this: e-voting in Belgium is going to change slightly. Previously, e-voting worked with a computer with a lightpen on which you would enter your vote, a magnetic card that would contain your vote in encoded form, and a magnetic card reader that would count the votes. After more than a decade when I was first eligible to vote back in 1996, it had already been implemented in the city of Antwerp, where my vote would be cast, so I've never voted on paper it had still not been rolled out everywhere, and some of the computers used for the voting were now starting to show their age. A few days ago on the news, they reported that the e-voting system was to change so that rather than magnetic cards, it would use pieces of paper. This way, voters would be able to check that the information on the piece of paper conforms with their vote. Investigating what this means, I found this website by the federal government, where they have some pictures of a test run of the new system which they ran in October last year. Rather than using magnetic cards, the new system uses paper slips with the voted information in two forms: once as a (huge) QR code, and once as a human-readable text. In theory, this is a more transparent system than using a magnetic card, and that's a good thing; additionally, it allows for a traditional recount (by people, not computers) should the need ever arise. However, I do still see some problems:

• The website says you need to first fold the slip, and then unfold it when you get at the scanner so you can scan your barcode. This risks violating the secrecy of the vote, as people will be able to glance what other people have voted as they unfold the slip. I'm not sure how this can be fixed.
• If the vote is printed on the paper slip in two forms, you need to be able to guarantee in some way that the two forms are the exact same information. People with smartphones can scan the QR code and compare it to the human-readable code, but not everyone has one of those.

Having said that, I do think the new system is an improvement over the old (in fact, it's exactly the kind of system I had thought about myself earlier when thinking about ways to improve e-voting); and if the two problems are solved in some way (it's not impossible to do that, just requires some clever thinking), I believe it has potential as a way to implement electronic voting in a good way.

I will sound monothematic, but I have been devoting quite a bit of work to this topic lately: Trying to stop the advance of e-voting in Mexico, Latin America and the world. Why trying to stop it? Isn't technology supposed to help us, to get trustable processes? Yes, it's supposed to... but it just cannot achieve it, no matter how hard it is tried I won't get into explanations in this blog post, but there is plenty of information. Feel free to ask me for further details. Anyway Yesterday (Sunday, 2012-06-17) was the fifth simulated voting that will lead to the first wide-scale deployment of electronic voting booths in my country: About 10% of the population of the state of Jalisco (that means, ~500,000 people) will cast their votes on July 1st electronically. This particular case illustrates how simulated votings can be used to forge a lie: Pounce Consulting, the company that won the e-voting project for IEPC (Jalisco's voting authority), delivered their booths over 40 days late, just before the deadline for the project to be canceled. Oh, and by the way, it's the same company that just failed to deliver on time for another planned local authority (10% of the booths in the Federal District, where I live, where fortunately 100% of the votes will be cast on traditional, auditable and cheap paper). After this delay, five voting simulations were programmed, to get the local population acquinted with them. The first ones just failed to get the population's interest and had close to 40% failure rates (mainly regarding transmission). Several other "minor details" were reported, including mechanical details that allowed subsequent voters to see the vote of who had just left. Anyway, making long story short: The fifth and last simulation was held yesterday. Officially, it was finally successful (about time). As these booths include the "facilities" to communicate the results via the cellular network, but the populations where they are to be deployed do not yet have cellular coverage, 10% of the booths will have to be carried back to the Districtal Header (that can be a ~10hr trip) to be counted. Also, in all places, traditional paper stationery and paraphernalia will be printed just in case it is needed (and when will they now? When half of the votes are cast and lost?) Anyway... e-voting is still in its first stage in Mexico. Right now, I'm sure, no attempts to rig the election will be made (centrally). But every effort will be made (as it has been made) to dismiss the obviously big and nontrivial ways it has failed and will fail, and any problems will be labeled as "minor". And probably by 2018 we will be facing many more states (even nationwide) deployments. But propaganda fails to see the obvious: E-voting is more expensive, more complicated, leads to more possible failure states. E-voting should not be deployed in large-scale (i.e. more than a couple of hundred voters) elections. Electronic voting is insecure, violates secrecy, allows for fraud. No matter how many locks are put into it.