Search Results: "xam"

28 November 2012

Alberto Garc a: QEMU and open hardware: SPEC and FMC TDC

Working with open hardware Some weeks ago at LinuxCon EU in Barcelona I talked about how to use QEMU to improve the reliability of device drivers. At Igalia we have been using this for some projects. One of them is the Linux IndustryPack driver. For this project I virtualized two boards: the TEWS TPCI200 PCI carrier and the GE IP-Octal 232 module. This work helped us find some bugs in the device driver and improve its quality. Now, those two boards are examples of products available in the market. But fortunately we can use the same approach to develop for hardware that doesn t exist yet, or is still in a prototype phase. Such is the case of a project we are working on: adding Linux support for this FMC Time-to-digital converter.
FMC TDC
This piece of hardware is designed by CERN and is published under the CERN Open Hardware Licence, which, in their own words is to hardware what the General Public Licence (GPL) is to software . The Open Hardware repository hosts a number of projects that have been published under this license. Why we use QEMU So we are developing the device driver for this hardware, as my colleague Samuel explains in his blog. I m the responsible of virtualizing it using QEMU. There are two main reasons why we want to do this:
  1. Limited availability of the hardware: although the specification is pretty much ready, this is still a prototype. The board is not (yet) commercially available. With virtual hardware, the whole development team can have as many boards as it needs.
  2. Testing: we can test the software against the virtual driver, force all kinds of conditions and scenarios, including the ones that would probably require us to physically damage the board.
While the first point might be the most obvious one, testing the software is actually the one we re more interested in. My colleague Miguel wrote a detailed blog post on how we have been using QEMU to do testing. Writing the virtual hardware Writing a virtual version of a particular piece of hardware for this purpose is not as hard as it might look. First, the point is not to reproduce accurately how the hardware works, but rather how it behaves from the operating system point of view: the hardware is a black box that the OS talks to. Second, it s not necessary to have a complete emulation of the hardware, there s no need to support every single feature, particularly if your software is not going to use it. The emulation can start with the basic functionality and then grow as needed. The FMC TDC, for example, is an FMC card which is in our case connected to a PCIe bridge called SPEC (also available in the Open Hardware repository). We need to emulate both cards in order to have a working system, but the emulation is, at the moment, treating both as if they were just one, which makes it a bit easier to have a prototype and from the device driver point of view doesn t really make a difference. Later the emulation can be split in two as I did with with TPCI200 and IP-Octal 232. This would allow us to support more FMC hardware without having to rewrite the bridging code. There s also code in the emulation to force different kind of scenarios that we are using to test if the driver behaves as expected and handles errors correctly. Those tests include the simulation of input in the any of the lines, simulation of noise, DMA errors, etc.
Tests
And we have written a set of test cases and a continuous integration system, so the driver is automatically tested every time the code is updated. If you want details on this I recommend you again to read Miguel s post.

Dirk Eddelbuettel: Rcpp 0.10.1

A the new Rcpp release 0.10.1 arrived this morning on CRAN (as already has Windows binaries) and in Debian. This is a follow-up to the recent 0.10.0 release which extends the exciting new Rcpp-attributes and Rcpp-sugar work further, and as in a number of other areas as detailed below in the NEWS sections. This release brings an change to some of the binary interfaces. If you have packages using Rcpp, you will most likely have to reinstall them from source. Some change were made to const correctness as well as other aspects, and it seems that we have temporarily broken the excellent RcppEigen and RcppOctave packages. We are looking into this, and are sorry about the bug. The complete NEWS entry for 0.10.1 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.
Changes in Rcpp version 0.10.1 (2012-11-26)
  • Changes in Rcpp sugar:
    • New functions: setdiff, union_, intersect setequal, in, min, max, range, match, table, duplicated
    • New function: clamp which combines pmin and pmax, e.g. clamp( a, x, b) is the same as pmax( b, pmin(x, a) )
    • New function: self_match which implements something similar to match( x, unique( x ) )
  • Changes in Rcpp API:
    • The Vector template class (hence NumericVector ...) get the is_na and the get_na static methods.
    • New helper class no_init that can be used to create a vector without initializing its data, e.g. : IntegerVector out = no_init(n) ;
    • New exception constructor requiring only a message; stop function to throw an exception
    • DataFrame gains a nrows method
  • Changes in Rcpp attributes:
    • Ability to embed R code chunks (via specially formatted block comments) in C++ source files.
    • Allow specification of argument defaults for exported functions.
    • New scheme for more flexible mixing of generated and user composed C++ headers.
    • Print warning if no export attributes are found in source file.
    • Updated vignette with additional documentation on exposing C++ interfaces from packages and signaling errors.
  • Changes in Rcpp modules:
    • Enclose .External invocations in BEGIN_RCPP/END_RCPP
  • Changes in R code :
    • New function areMacrosDefined
    • Additions to Rcpp.package.skeleton:
      • attributes parameter to generate a version of rcpp_hello_world that uses Rcpp::export.
      • cpp_files parameter to provide a list of C++ files to include the in the src directory of the package.
  • Miscellaneous changes:
    • New example 'pi simulation' using R and C++ via Rcpp attributes
Thanks to CRANberries, you can also look at a diff to the previous release 0.10.0. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

27 November 2012

Daniel Kahn Gillmor: more proprietary workarounds, sigh

In supporting a labful of Debian GNU/Linux machines with NFS-mounted home directories, i find some of my users demand a few proprietary programs. Adobe Flash is one of the most demanded, in particular because some popular streaming video services (like Amazon Prime and Hulu) seem to require it. I'm not a fan of proprietary network services, but i'm happy to see that Amazon Prime takes Linux support seriously enough to direct users to Adobe's Linux Flash "Protected Content" troubleshooting page (Amazon Prime's rival NetFlix, by comparison, has an abysmal record on this platform). Of course, none of this will work on any platform but i386, since the flash player is proprietary software and its proprietors have shown no interest in porting it or letting others port it :( One of the main issues with proprietary network services is their inclination to view their customer as their adversary, as evidenced by various DRM schemes. In two examples, the Flash Player's DRM module appears to arbitrarily break when you use one home directory across multiple machines. Also, the DRM module appears to depend on HAL, which is being deprecated by most of the common distributions. Why bother with this kind of gratuitous breakage? We know that video streaming can and does work fine without DRM. With modern browsers, freely-formatted video, and HTML5 video tags, video just works, and it works under the control of the user, on any platform. But Flash appears to throw up unnecessary hurdles, requiring not only proprietary code, but deprecated subsystems and fiddly workarounds to get it functional. I'm reminded of Mako's concept of "antifeatures" -- how much engineering time and effort went into making this system actually be less stable and reliable than it would have otherwise been? How could that work have been better-directed? Tags: antifeatures, flash, hal, proprietary, streaming

25 November 2012

Michael Stapelberg: RC bugs

I recently worked on the following RC bugs: I have to say that I find the BTS to be very hard to read from the perspective of a newcomer who wants to help squashing RC bugs, both from a technical and a social standpoint. As an example for the technical side: blocking bugs are very easy to miss, so innocent bugreports which seem actionable turn out to just wait for another bug with lots of discussion. Also, it seems like the best interface to the BTS for RC bugs is UDD, so RichiH s RC report was quite helpful. For the social side: Bugreports often are in a state in which it is not clear what the next course of action is, e.g. #634261 (the bug is not considered RC, but still has severity: grave) or #684633 (dropping from testing suggested but not clear whether a removal request has been filed or not). I m aware that this is very vague critic. I suppose what I am missing is a way in which we can make the list of RC bugs more actionable. When I am browsing through RC bugs, I often open bugreports and figure out that there s nothing I can do. Possibly a way to improve this situation is by letting people claim ownership of a bug and then filtering for un-owned bugs and those which have an owner but have not seen any activity for some period of time.

24 November 2012

Richard Hartmann: Release Critical Bug report for Week 47

The gap As you can see below, the gap between this release cycle and the last is closing slowly but steadily. The graph Pawel Sadkowski published a very nice graph up to and including week 46. The gang This weekend, there are BSPs in Essen (Germany), Munich, Paris, and Tokyo. They are all hanging out on irc.debian.org (OFTC) in #debian-bugs. The gist The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the squeeze cycle ?
Week Squeeze Wheezy Diff
43 284 (213+71) 468 (332+136) +184 (+119/+65)
44 261 (201+60) 408 (265+143) +147 (+64/+83)
45 261 (205+56) 425 (291+134) +164 (+86/+78)
46 271 (200+71) 401 (258+143) +130 (+58/+72)
47 283 (209+74) 366 (221+145) +83 (+12/+71)
48 256 (177+79)
49 256 (180+76)
50 204 (148+56)
51 124 (124+54)
52 115 (78+37)
1 93 (60+33)
2 82 (46+36)
3 25 (15+10)
4 14 (8+6)
5 2 (0+2)
6 release!

Russ Allbery: Review: Dune

Review: Dune, by Frank Herbert
Series: Dune #1
Publisher: Ace
Copyright: 1965
Printing: September 1990
ISBN: 0-441-17271-7
Format: Mass market
Pages: 537
The Atreides family, distant cousins to the imperial family, have ruled the planet of Caladan for twenty generations. Caladan is a wet farming world, comfortable and pleasant, but not horribly important. But House Atreides is feuding with House Harkonnen, and, at the start of Dune, that feud maneuvers Duke Leto into giving up his holdings and moving his family to take possession of Arrakis. Arrakis is a desert planet, previously controlled by Baron Harkonnen. It is unrelentingly hostile, home to smugglers and dangerous local desert dwellers called Fremen. But it's also one of the most important planets in the galaxy, since it's the sole origin of the chemical called melange, or spice. Spice permits a limited form of prescience, which allows the navigators of the Spacing Guild to successfully steer ships across the interstellar void. Arrakis's production of spice is what makes interstellar travel, and therefore all of interstellar civilization, possible. Dune is the story of Paul Atreides, son and heir to Duke Leto Atreides. His mother, Lady Jessica, is one of the Bene Gesserit, a secretive order of women devoted to mental and physical discipline and to the long-term genetic improvement of mankind. He is not supposed to exist; Lady Jessica was supposed to only bear a daughter of Leto. But he may be something special, the long-sought (but also dangerous) Kwisatz Haderach who can unite male and female Bene Gesserit powers. The Bene Gesserit take great interest in him from the start of Dune. More surprisingly, so do the Fremen of Arrakis; from the moment he arrives there, he seems to be fulfilling prophecies of theirs that are partly, but not entirely, ones planted by the Bene Gesserit long ago. The feud with the Harkonnens, the unstable place of Arrakis in galactic politics, the dreams of the Fremen and the Imperial ecologist on Arrakis of terraforming, Bene Gesserit plans, Paul's abilities, and the legends of the Fremen all combine in a complex mix of politics, battle, and clashes of culture. Dune is an acknowledged SF masterpiece, one of the best-known classics of the genre. It's usually found in short lists of the best SF novels ever written. It spawned five sequels by Frank Herbert (about which more in a moment), as well as numerous additional sequels and prequels by Kevin J. Anderson and Brian Herbert. It's been adopted for the screen twice, not to mention board games, video games, and numerous other projects. This is my second reading, the first in about twenty years, but the story was still immediately familiar from having seen films and having discussed and read about the universe. This is not science fiction in any strict sense. Dune is science fiction in the same way that Star Wars is: a futuristic gloss on top of power structures inspired by feudalism, heavily mixed with mysticism, mental powers, magic, and implausible but convenient science that creates the story effects the author wants. Both Bene Gesserit powers in general and Paul's abilities in particular are effectively magic. There is some hand-waving explanation of their ability to verbally control other people as taking advantage of specific pitches and intonations that people are vulnerable to, but it's effectively spell-casting (and is a direct inspiration for Jedi mind tricks). All of the mysticism (and there's quite a lot of it in Dune, including race memory, precognition, and even molecular transformation) resembles the Force from Star Wars more than anything scientific. Dune is epic fantasy told on a science fiction stage, complete with a young protagonist coming into his powers and dangerous and sometimes hostile mentors. What Dune gets right, and what has put it so high in the pantheon of great science fiction, is the world building. Herbert sets the story tens of thousands of years into the future of humanity and then effectively projects the feeling of deep history over everything in the novel. This is the kind of book that has appendices with more background information; more to the point, it's the kind of book where you may actually read them out of curiosity. Mankind has a vast interstellar empire (Herbert's universe, like Asimov's Foundation universe, admits no aliens) governed by a system akin to the early British monarchy. An emperor rules in balance with the Great Houses, who meet in a sort of parliament. But against both is a third force: the Spacing Guild, who maintains a monopoly over all interstellar travel. (And the Bene Gesserit form an underground, secretive fourth power base.) Herbert plays with vast swaths of time and great forces of history as well as very good epic fantasy and better than nearly all SF I've read. The detailed world-building is equally good. Nearly all of Dune takes place on the desert planet of Arrakis, which has a lovingly-described ecology and local culture built entirely around scarcity of water. (The details of that ecology are much of the plot and mystery of the book, so I won't spoil them further.) While I doubt the precise details hold up to close scientific scrutiny, this is an obvious precursor to the great ecological stories of later SF, such as Kim Stanley Robinson's Mars trilogy. The details all feel right and hang together in satisfying ways, while also generating the great Sand Worms of Arrakis, a key ingredient in several of the best set pieces in the history of SF. This is the sort of book where the fascinating details and discoveries about the world do as much to keep one turning the pages as the plot, although the plot is also satisfyingly twisty and tense. Unfortunately, Dune doesn't get everything right. The amount of mysticism involved is a bit much, and at times the drug-trip mystical experiences of viewpoint characters turn into excessively purple prose and nearly incomprehensible descriptions. Those mystical experiences also involve race and genetic memory, a concept that's just scientific enough to be unbelievable. A few of the other scentific cheats are also rather blantant; for example, Herbert constructs an elaborate, artificial technology of shielding that seems designed primarily as an excuse to add sword combat to a futuristic story, and I have always struggled to suspend disbelief about the way lasers and shields interact in Dune. The Spacer Guild's monopoly on interstellar travel can be explained; their monopoly on local orbital space, or even the high stratosphere, both vital to allow certain things on Arrakis to remain secret, are much more dubious. Herbert mostly doesn't try to explain these things, and as with Star Wars the less explained the cheats are, the better they work as part of the story. But the technological background doesn't hold up against much examination. Worse, for me, is the general quality of the writing. Herbert does some things very well, such as world-building, and avoids awkward infodumps. Characterization and pacing are both fairly solid; he does a good job with Paul and Jessica in particular, and I've always liked the Fremen. But he wants to put the reader in everyone's head, frequently by giving character thoughts as italicized dialogue, and to enable that he uses a perspective that I always find distracting. Most fiction is written in tight third person. This means that the viewpoint character for any given section of the book is referred to in the third person, like all the other characters, but the reader has special access to their thoughts and emotions. We get to know what they're really thinking and feeling, not just the impressions they give to others, while the non-viewpoint characters are shown only from external appearances and the thoughts of the viewpoint character. Some books hold to the same viewpoint character throughout, but more commonly books move between viewpoint characters at scene breaks to provide more angles on the book's events. First person, in which the story is told by a specific character as if they were telling a story or writing it down, is the most common alternative. Third person objective, in which we don't get any special insight into the internal thoughts of any of the characters, is less common but still unsurprising. Dune does not use any of those perspectives. Instead, Dune uses wandering third-person omniscient, in which we get the inner thoughts and emotions of a character in a scene and then a few lines later the inner thoughts and emotions of a different character. This is the sort of thing that may or may not bug you depending on how much you've read, how deep the expectations of perspective are ingrained, and how much you notice perspective. It drives me nuts. I subconsciously align with the viewpoint character of a section, and pay attention to the ways that authors indicate which character will be the viewpoint character at the start of a scene. Herbert's constant flitting from character to character makes me dizzy. We get the verbatim thoughts of everyone almost indiscriminately, making me feel like I'm randomly hopscotching through the scene. For me, this does two things: it hurts my ability to get engrossed in the story, since I'm constantly thrown out of my normal reading mode when the viewpoint unexpectedly shifts, and it makes the writing feel repetitive. One keeps hearing about the same thing from multiple perspectives, and at times the story bogs down in everyone's internal dialogues rather than showing character reactions and letting the reader draw their own conclusions. I think it tries for a cinematic perspective, but ends up making the story feel muddled. The other flaw, which I didn't notice originally but which leaped out at me during this re-read, is that Herbert's world-building uses quite a few stereotypes. The most notorious, and most widely discussed, is of course the Fremen. Herbert draws heavily on Arab and Islamic culture even beyond the obvious similarities of people living in a harsh, arid climate. He borrows some rather loaded terms and cultural markers, such as jihad, to construct a culture of potential religious fanatics. This is not all bad; the Fremen are clearly portrayed as the good guys, which is a refreshing change from more typical current portrayals of Islam. But it becomes clear that they have aligned their entire culture around influences from outside, and the whole plot of Dune can be fairly characterized as an instance of "what these people need is a white man." Paul (and Kynes before him) joins their culture as well, but Paul becomes a better native than the natives, while simultaneously bringing his outside perspective. It's the sort of plot that is more widely noticed today than it would have been in 1965. Another major example of this, and one that I found more blatant, is that Herbert turns the Harkonnen into hissable, one-sided villains and uses some nasty stereotypes to do it. The insane torturer is consistently and repeatedly described as effeminate, fat is used as a marker of moral inferiority and evil, and the primary villain is homosexual and prefers drugged young male slaves. Here too, this sort of characterization short-cut was more common in 1965, but it's not appealing and makes the (already rather camp) scenes set among the Harkonnen even less enjoyable. Less clear-cut is the way women are handled throughout Dune. I do have to give Herbert some credit, particularly for the era in which he was writing. There are powerful female characters in Dune, including both Jessica and Alia, who have their own independent power and successfully pursue their own agendas throughout. The effectively all-female Bene Gesserit is a major political power in the story and is treated by the other players with respect as well as fear. But it's hard not to also notice the general position of women as subservient to men, not only in the general culture of the Great Houses but also in the more positively-portrayed Fremen culture. Indeed, the subservience of women is even worse in Fremen culture, where they're treated like property and where being killed by a woman is a sign of shame. Again, Herbert deserves some credit for doing better than a lot of 1960s fiction, but the sexism fairy has still been at work here. None of these flaws change the fact that Dune is a masterpiece. Herbert brings together history, world building, ecology, politics, and a compelling coming-of-age story about a messiah figure into a fast-paced, sweeping epic with a thoroughly satisfying conclusion. I think they do make it a flawed masterpiece, but it's still one of those SF novels that everyone should read at least once. Sadly, it's also a masterpiece that I think has suffered from its own success in the form of sequels, prequels, and a ton of supporting material. This is one of the problems that truly excellent world building can lead to. Human history is fractal: any specific detail can be examined in more depth and will usually lead (provided that information is available at all) to even more fascinating detail. The best world building conveys that impression of depth. That's what Herbert achieves here with hints, notes, and asides: the sense that galactic history is a vast ediface with the same fractal complexity as real human history. It makes for a compelling background, but it also inspires people to dig into that background and flesh out all of the details the way that we do with human history. But this doesn't actually work; invented history created by one person simply cannot be fractal in the same way. Human history is endlessly complex because it was generated by the complex interactions of many people. Invented history is an illusion that hints at complexity by building the same surface, but one mind, or even a small number of minds, cannot generate the same depth. The result is that if one digs too deep, one removes that convincing surface and ends up with a mundane, simplistic, and unsatisfyingly fake set of events. I think that's what's happened with all of the supporting material that's been written around Dune since its original publication. Dune is of a piece, a single story that's deeply enjoyable on its own terms and leaves the reader with a satisfying impression of complexity. The systemic excavation of that complexity lessens it and reveals too much of the illusion. Yes, I want to know more about the Butlerian Jihad, but that's the point: the wanting is the sign of succesful crafting of imagined history. Reading the definitive account is more likely to leave me unsatisfied than to lead to the recursive curiosity that human history can create. The sequels to Dune written by Herbert himself are, for me, another matter. Some reviewers level the same criticism at them: that Herbert dives too far into background best left unexplored. But they have the advantage of moving forward, telling more of the story set off by Paul, and the end of Dune is a clear setup for a sequel. One of Paul's goals throughout most of the book has been left unaccomplished. I don't think Herbert dove too deep into his creation; rather, my problem with his sequels (all of which I've read, although it's been some years now) is that he took the story in a direction that I actively disliked and found painful to read. Regardless, the general consensus is that the sequels aren't as good as the original, and while Dune doesn't fully resolve its story, it's complete enough that it's possible to stop here. Stopping is the general recommendation, although I still may re-read and review the sequels at some point. Followed by Dune Messiah. Rating: 8 out of 10

21 November 2012

Axel Beckert: Suggestions for the GNOME Team

Thanks to Erich Schubert s blog posting on Planet Debian I became aware of the 2012 GNOME User Survey at Phoronix. Like back in 2006 I still use some GNOME applications, so I do consider myself as GNOME user in the widest sense and hence I filled out that survey. Additionally I have to live with GNOME 3 as a system administrator of workstations, and that s some kind of usage, too. ;-) The last question in the survey was Do you have any comments or suggestions for the GNOME team? Sure I have. And since I tried to give constructive feedback instead of only ranting, here s my answer to that question as I submitted it in the survey, too, just spiced up with some hyperlinks and highlighting:
Don t try to change the users. Give the users more possibilities to change GNOME if they don t agree with your own preferences and decisions. (The trend to castrate the user was already starting with GNOME 2 and GNOME 3 made that worse IMHO.) If you really think that you need less configurability because some non-power-users are confused or challenged by too many choices, then please give the other users at least the chance to enable more configuration options. A very good example in that hindsight was Kazehakase (RIP) who offered several user interfaces (novice, intermediate and power user or such). The popular text-mode web browser Lynx does the same, too, btw. GNOME lost me mostly with the change to GNOME 2. The switch from Galeon 1.2 to 1.3/2.0 was horrible and the later switch to Epiphany made things even worse on the browser side. My short trip to GNOME as desktop environment ended with moving back to FVWM (configurable without tons of clicking, especially after moving to some other computer) and for the browser I moved on to Kazehakase back then. Nowadays I m living very well with Awesome and Ratpoison as window managers, Conkeror as web browser (which are all very configurable) and a few selected GNOME applications like Liferea (luckily still quite configurable despite I miss Gecko s about:config since the switch to WebKit), GUCharmap and Gnumeric. For people switching from Windows I nowadays recommend XFCE or maybe LXDE on low-end computers. I likely would recommend GNOME 2, too, if it still would exist. With regards to MATE I m skeptical about its persistance and future, but I m glad it exists as it solves a lot of problems and brings in just a few new ones. Cinnamon as well as SolusOS are based on the current GNOME libraries and are very likely the more persistent projects, but also very likely have the very same multi-head issues we re all barfing about at work with Ubuntu Precise. (Heck, am I glad that I use Awesome at work, too, and all four screens work perfectly as they did with FVWM before.)
Thanks to Dirk Deimeke for his (German written) pointer to Marcus Moeller s interview with Ikey Doherty (in German, too) about his Debian-/GNOME-based distribution SolusOS.

Dirk Eddelbuettel: Rcpp attributes: A simple example 'making pi'

We introduced Rcpp 0.10.0 with a number of very nice new features a few days ago, and the activity on the rcpp-devel mailing list has been pretty responsive which is awesome. But because few things beat a nice example, this post tries to build some more excitement. We will illustrate how Rcpp attributes makes it really easy to add C++ code to R session, and that that code is as easy to grasp as R code. Our motivating example is everybody's favourite introduction to Monte Carlo simulation: estimating . A common method uses the fact the unit circle has a surface area equal to . We draw two uniform random numbers x and y, each between zero and one. We then check for the distance of the corresponding point (x,y) relative to the origin. If less than one (or equal), it is in the circle (or on it); if more than one it is outside. As the first quadrant is a quarter of a square of area one, the area of the whole circle is -- so our first quadrant approximates over four. The following figure, kindly borrowed from Wikipedia with full attribution and credit, illustrates this: Example of simulating pi Now, a vectorized version (drawing N such pairs at once) of this approach is provided by the following R function.
piR <- function(N)  
    x <- runif(N)
    y <- runif(N)
    d <- sqrt(x^2 + y^2)
    return(4 * sum(d < 1.0) / N)
 
And in C++ we can write almost exactly the same function thanks the Rcpp sugar vectorisation available via Rcpp:
#include <Rcpp.h>
using namespace Rcpp;
// [[Rcpp::export]]
double piSugar(const int N)  
    RNGScope scope;		// ensure RNG gets set/reset
    NumericVector x = runif(N);
    NumericVector y = runif(N);
    NumericVector d = sqrt(x*x + y*y);
    return 4.0 * sum(d < 1.0) / N;
 
Sure, there are small differences: C++ is statically typed, R is not. We need one include file for declaration, and we need one instantiation of the RNGScope object to ensure random number draws remain coordinated between the calling R process and the C++ function calling into its (compiled C-code based) random number generators. That way we even get the exact same draws for the same seed. But the basic approach is identical: draw a vector x and vector y, compute the distance to the origin and then obtain the proportion within the unit circle -- which we scale by four. Same idea, same vectorised implementation in C++. But the real key here is the one short line with the [[Rcpp::export]] attribute. This is all it takes (along with sourceCpp() from Rcpp 0.10.0) to get the C++ code into R. The full example (which assumes the C++ file is saved as piSugar.cpp in the same directory) is now:
#!/usr/bin/r
library(Rcpp)
library(rbenchmark)
piR <- function(N)  
    x <- runif(N)
    y <- runif(N)
    d <- sqrt(x^2 + y^2)
    return(4 * sum(d < 1.0) / N)
 
sourceCpp("piSugar.cpp")
N <- 1e6
set.seed(42)
resR <- piR(N)
set.seed(42)
resCpp <- piSugar(N)
## important: check results are identical with RNG seeded
stopifnot(identical(resR, resCpp))
res <- benchmark(piR(N), piSugar(N), order="relative")
print(res[,1:4])
and it does a few things: set up the R function, source the C++ function (and presto: we have a callable C++ function just like that), compute two simulations given the same seed and ensure they are in fact identical -- and proceed to compare the timing in a benchmarking exercise. That last aspect is not even that important -- we end up being almost-but-not-quite twice as fast on my machine for different values of N. The real takeaway here is the ease with which we can get a C++ function into R --- and the new process completely takes care of passing parameters in, results out, and does the compilation, linking and loading. More details about Rcpp attributes are in the new vignette. Now enjoy the .

Update:One somewhat bad typo fixed.

Update:Corrected one background tag.

20 November 2012

Jon Dowland: Waterstones

In May this year, in a desperate bid to bail water out of a sinking ship, HMV group sold off the Waterstones chain of bookstores. I'm fond of reading so naturally I visit Waterstones from time to time. (but not exclusively.) For many years, their promotions were unanimously structured as "3 for 2" offers: find three books in the promotion, in-store, and get the cheapest of the three for free. I hated these promotions. I always found it very challenging to find three books in the promotion that I wanted. If I was visiting to buy a specific book and found it in the promotion, on more than one occasion I left the shop empty handed: I wasn't prepared to pay full price for a book in a promotion and not get the benefit of the promotion. Since being sold, Waterstones seem to have made a lot of changes. Many new hardbacks from popular authors have debuted at a discount for a short period (Rowling, Pratchett). I've found many more discounted books not tied to others in 3-for-2-style offers, sometimes window-display stock at half price, a month or so after release. They started a loyalty-card scheme, with a 10 gift card the prize you reach once you've spent 100. (I wonder if such schemes are examples of a token economy?) Finally, they've tentatively started to explore putting cafes in their branches. The one in Newcastle is best described as 'fledgling', but it's nice to have one more place to escape for a cup of tea at lunchtimes. The cafe, as well as a very well stocked collection of magazines, were the two killer features of the sadly- defunct Borders bookseller chain. For years before we got our own Borders in the North East, I used to enjoy the York branch if I was passing. I actually managed to fill a loyalty card this year, so clearly their change in attitude has worked on me! I'm generally not a fan of gift cards, especially in today's climate, so I quickly spent it. I hold an American Express credit card, which earns me cashbask on purchases which I make with it. I think the rate is about 1%. However, I can opt to receive the cashback in the form of gift vouchers. If I do so, I get them to the value of 110% my cashback balance. One of the retailers in on the scheme are Waterstones. I'm tempted

18 November 2012

Jamie McClelland: New Privacy Arguments are Needed

Another decent article on online privacy was published by the NYT this morning. The article documents the collection and centralized aggregation of consumer data online. It ends with a strong message about consumers becoming the "product" - which echoes what Noam Chomsky and Ben Bagdikian have been saying for decades. The problem with this critique is that it doesn't go far enough. Chomsky and Bagdikian didn't say readers are the product of modern journalism and that's bad because nobody wants to be a product. They critiqued the model because it produces terrible journalism and democracy suffers as a result. Similarly, the online trading of audiences and the invasion of our privacy that results has more important implications than "don't be a sucker" which seems to be the gist of the privacy argument these days. Honestly, I don't stay awake at night worrying that Nike knows I'm looking for a pair of sneakers, or even that they know my income. I do stay awake at night worrying that people are tagging my photo on Facebook, which could allow the New York Police Dept to submit a photo of protesters to Facebook and get a list of names and addresses of the people in the photo. Or it could allow the police to track my movements via existing networks of surveillance cameras by matching my image to my name. Would that require a search warrant? How would that impact my trust in my government to know that my movements are being tracked? Or worse, to know they might be tracked but I'll never know if they are or aren't? Many governments (including the US) have sordid histories of infiltrating political organizations and intentionally sowing dissent. Having access to activists' purchasing habits provides a treasure trove of material for this purpose that previously was labor intensive for cops to collect. It also provides leverage during interrogation, a time of heightened emotional vulnerability where the mere mention of a private detail of one's life can have a powerful impact. How can we prevent law enforcement from having access to these details of our lives? And then there's the dragnet. A central database of consumer habits provides a tantalizing collection of information that could be searched for people who fit a profile of a crime. Is solving the crime at hand worth the invasion of privacy that results from people being investigating only based on their consumer habits? How would we measure the impact an investigation like this would have on our confidence in democracy and government? The corporate advertisers promoting this kind of data collection argue that these problems are the price of progress. However, the ideology behind these technology developments is capitalism, not some intrinsic aspect of the Internet or network communications. Historically, the Internet has developed based on a drive toward de-centralization. Email and web sites, the bedrock of Internet activity, are based on these principles: anyone can add their own email server or web server and everyone can seamlessly access it. In contrast, the logic of capitalism as it relates to the Internet, is one of centralization and aggregating data. Facebook and Google want you to use their platforms for all your online activities so they have as complete a picture as possible of what you like and do. As the New York Times article describes, companies like Rubicon want to track all your movements and aggregate that data for profit. We have a choice. We can carefully evaluate our use of corporate services and consider the implications it has not only on our lives but on society as a whole. And, we can choose to use and support services that promote open standards and interoperability that will respect our personal privacy values. Want a practical example? Friendica is software designed to allow you to post status alerts, photos, videos and other information about yourself in a way similar to Facebook. However, it has one big difference: you can host your account on any server running the software anywhere in the world. You can still "friend" people on different servers, but we don't have to all trust the same organization to host it. Furthermore, Friendica allows you to post once and automatically cross-post to your Twitter, Facebook or other similar services. Sound good? You are welcome to try out an account on the May First/People Link install: https://friends.mayfirst.org/.

17 November 2012

Antoine Beaupr : First stable release of the Drupal OpenID Provider

A short announcement to mark the release of the first stable OpenID provider for Drupal. Started by James Walker around 4 years ago, I have since then got commit access and taken care of the module for the last 2 years. And while it was not an easy task, with the help of the community, this module was able to mature to a fully featured, flexible and standard implementation during that time. One has to wonder however, is it too late for OpenID?

A bit of history Drupal was one of the pioneers in OpenID support. Back in 2006 when the protocol was still in its infancy (less than a year old!), Drupal was adding OpenID support to its 4.7 release through a contrib module and two years later, the Drupal 6.0 release was sporting OpenID in core, although not without significant interoperability issues. Still, we could say that rapid adoption by such a popular CMS as Drupal really boosted OpenID's adoption. At about the same time, Yahoo, Sourceforge, Myspace, Microsoft and Google would all join the OpenID bandwagon. On the OpenID "provider" side of things, things were not so great however. Back when I started experimenting with this, it wasn't actually that easy to host your own OpenID provider, to allow your users to login to other OpenID-enabled sites. While there are a lot of different libraries to provide OpenID support in your application there was not (and still isn't) any software that gives you an OpenID provider and is dedicated to that. No official reference implementation either, other than the code from the one of the founding companies of OpenID, JanRain. There was an OpenID provider module in Drupal, but there was no official release done until a two beta releases in 2010.

Koumbit's community involvement It is about that time I started getting involved in the project. Koumbit needed an easy way for our workers to log into multiple sites without too much trouble, so I started looking at OpenID provider implementations. The lack of dedicated software somehow made the Drupal implementation a natural fit, and I started experimenting with the module, filing bugs and reviewing patches, as there were a lot of interoperability and design issues with the provider. After a while, walkah was kind enough to give me full access to the project and I started doing regular releases (every few months), culminating in today's official 1.0 release for both Drupal 6 and Drupal 7, which I am very proud and happy about. While I am here, I want to send many thanks to the people behind the Drupal 7 port: paranojik, mikecar, xamanu and testing from neizod. A lot of issues were also patched, reviewed and tested by numerous contributors that are unfortunately difficult to track down other than reading the commit log. So we at Koumbit have been running our own OpenID provider for a good while now, and it has the key advantage of being connected to our LDAP directory (with the D6 LDAP integration module so it is, in a sense, a central point for authentication at Koumbit. Very often people don't clearly understand what their LDAP password does, and I can easily remind them "it's the OpenID thing", or "try your password in the OpenID site" if they think they have forgotten it. So much goes on in the web browser these days that it makes everything much easier.

The problem with Drupal's OpenID provider However, it is kind of embarrassing that it took so long. More than 6 years after the OpenID protocol was written, we have a stable OpenID provider. Now, this may be because I am overly conservative about stable releases, but I don't think so: the OpenID provider module had, for a long time, serious interoperability issues with OpenID sites. There was also a clear lack of features on the OpenID provider, a gap that various third party modules, now documented on the OpenID provider homepage are now trying to fill. There is also serious security issues with the protocol itself. The complexity and lack of clarity of the OpenID specification have rendered my work quite difficult at time, as the protocol has significant ambiguities and no clear reference implementation. OpenID is unfortunately renowned to be vulnerable to phishing and while Koumbit's use of the HTTPS protocol alleviates some of those issues (especially the redirect session hijack), tools like sslstrip can bypass those protections easily. The privacy aspect is also problematic, as the user is usually not free to choose which information from his or her profile is divulged to the site they are logging into. Of course those issues may be overlooked if the security trade off is not worth it. In the case of Koumbit, we use OpenID on a lot of client sites, but usually nothing critical, and we train our users to pay attention to the HTTPS icon when login in their OpenID account.

Are there alternatives? But unfortuantely, the reality is that there are not really any alternatives to OpenID out there. Mozilla's new BrowserID initiative (now rebranded Persona, boy those guys sure love rebranding..) is interesting, but actually has even more flaws than OpenID itself. Not only is it not resolving the phishing problem, it actually creates new problems, including my personal favorite, trusting server-side Javascript code, which OpenID is not using that much. I should also mention the Shibboleth module which has a surprising number of installs recorded on Drupal.org, but that depends on the much more complicated Shibboleth federation, for which there is no server implementation for Drupal, from what I understand. Oauth, on the other end, is even more widely used, and has server-side components like Oauth Login Provider and Services 3.x. We could also simply give up and use our souls stored at Facebook, Google and Twitter for authentication everywhere. Maybe OpenID opened the door for those corporations to see that there is a real opportunity in leveraging user's profile to track their online activity. It is not something that OpenID was necessarily designed for but it's an idea that certainly permeates Facebook connect and "like" buttons or Twitter and Google online business strategies, where you are the product sold to advertisers. I personally believe we should keep the struggle for distributed and standard identity mechanisms for the web that respect users' privacy. With Drupal's OpenID Provider, it is possible to create a secure, privacy-aware identity provider, although I still have to perform that damn security audit. Experiments with those third party modules should also be encouraged and I welcome any feedback on those modules. While I do not plan to attend OpenID conferences or do significant development on the OpenID provider right now, I will certainly make sure Koumbit keeps its commitment to maintaining stewardship on the OpenID provider module in Drupal. I encourage people to try it out, submit issues and patches to make it even better. Long life to OpenID! PS: if people are interested in funding that security audit, I would be happy to perform this through my regular work at Koumbit.org, just file a request at services@koumbit.org.

Sylvestre Ledru: Mini Debconf & merchandising

Next week end, November 24th and 25th, the mini Debconf will take place at EPITA in Paris. Mehdi Dogguy and I worked hard on the organization and on building a great program for this 2012 conference. This mini debconf will cover many subjects like Gnome (both as upstream and downstream), the Release team, how the Linux is packaged in Debian, etc.
The keynote will be "Free software and Debian, 20 years after" by Roberto Di Cosmo. Also, some merchandising will be proposed during the event by the association Debian France. Here is a quick list: Polos
long-sleeved polo
Long-sleeved polo

40 euros - 10 pieces Sleeveless polo
Sleeveless polo

25 euros - 100 pieces Thanks to Tanguy Ortolo for taking take of the order. Buff
Buff Debian
Buff Debian

18 euros - 75 pieces. It is described as Original Multifunctional Headwear. Real life example in video.
I love them! Finally, we will sell two kinds of Debian branded Sticker Portable:
Sticker Debian
Sticker Debian

1 euro - 100 pieces. They can be also ordered on the it2l website. Thanks to J r me Lemaire for providing these great products. If these products are successful, we will probably produce more for FOSDEM! For more information about the conference:
The official website
Wiki page for subscription

16 November 2012

Richard Hartmann: Release Critical Bug report for Week 46

In case you care, there's a short discussion about when to publish this info and how it relates to DWN on the Debian publicity mailing list. The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the squeeze cycle ? Yes, how indeed. There's not automatic way to generate this data, yet. Stay tuned for now.

Paul Tagliamonte: pathfinding debian's social connections

As a followup to my toy over at debtree, I ve taken some suggestions from my buddy Arno and found some time in the hotel during a visit for work up to Canada with the Open North folks hacking on open13. The hack is called debfriends (I need to purge personal details from the source tree before I can push it to VCS, since it uses the nm.d.o dump. Sorry, I should have been better about that.) I ve posted a cute front-end on my staging machine to play with over at graph.lucifer.pault.ag There s also an API, but it has all the data in the NM dump exposed. If you re a DD who will treat this data right, I can add a key (it s not secure, like, at all). I m going to publish the source ASAP so that others can stand up an instance if they d like. It s a cute toy, though! In the tradition of using nhandler for my examples, here s an example path from pabs to Nathan: pabs > nhandler Also, we re a shockingly well connected bunch. Well done :) Feel free to play with it!

Axel Beckert: Useful but Unknown Unix Tools: dwdiff better than wdiff + colordiff

A year ago I wrote in Useful but Unknown Unix Tools: How wdiff and colordiff help to choose the right Swiss Army Knife about using wdiff and colordiff together. Colordiff ed wdiff output looks like this:
$ wdiff foobar.txt barfoo.txt   colordiff
[-foo-]bar fnord
gnarz hurz quux
bla  +foo+  fasel
But if you have colour, why still having these hard to read wdiff markers still in the text? There exists a tool named dwdiff which can do word diffs in colour without textual markers and with even less to type (and without being git diff --color-words ;-). Actually it looks like git diff --color-words, just without the git:
$ dwdiff -c foobar.txt barfoo.txt
foo bar fnord
gnarz hurz quux
bla foo fasel
Another cool thing about dwdiff (and its name giving feature) is that you can defined what you consider whitespace, i.e. which character(s) delimit the words. So lets do the example above again, but this time declare that f is considered the only whitespace character:
$ dwdiff -W f -c foobar.txt barfoo.txt
foo bar bar fnord
gnarz hurz quux
bla foo fasel
dwdiff can also show line numbers:
$ dwdiff -c -L foobar.txt barfoo.txt
   1:1    foo bar fnord
   2:2    gnarz hurz quux
   3:3    bla foo fasel
$ dwdiff -c -L foobar.txt quux.txt
   1:1    foo bar fnord
   1:2    foobar floedeldoe
   2:3    gnarz hurz quux
   3:4    bla foo fasel
(coloured shell screenshots by aha)

15 November 2012

Matthew Garrett: More in the series of bizarre UEFI bugs

A (well, now former) coworker let me know about a problem he was having with a Lenovo Thinkcentre M92p. It booted Fedora UEFI install media fine, but after an apparently successful installation refused to boot. UEFI installs on Windows worked perfectly. Secure Boot was quickly ruled out, but this could still have been a number of things. The most interesting observation was that the Fedora boot option didn't appear in the firmware boot menu at all, but Windows did. We spent a little while comparing the variable contents, gradually ruling out potential issues - Linux was writing an entry that had an extra 6 bytes in a structure, for instance[1], and a sufficiently paranoid firmware implementation may have been tripping up on that. Fixing that didn't help, though. Finally we tried just taking the Windows entry and changing the descriptive string. And it broke.

Every UEFI boot entry has a descriptive string. This is used by the firmware when it's presenting a menu to users - instead of "Hard drive 0" and "USB drive 3", the firmware can list "Windows Boot Manager" and "Fedora Linux". There's no reason at all for the firmware to be parsing these strings. But the evidence seemed pretty strong - given two identical boot entries, one saying "Windows Boot Manager" and one not, only the first would work. At this point I downloaded a copy of the firmware and started poking at it. Turns out that yes, actually, there is a function that compares the descriptive string against "Windows Boot Manager" and appears to return an error if it doesn't match. What's stranger is that it also checks for "Red Hat Enterprise Linux" and lets that one work as well.

This is, obviously, bizarre. A vendor appears to have actually written additional code to check whether an OS claims to be Windows before it'll let it boot. Someone then presumably tested booting RHEL on it and discovered that it didn't work. Rather than take out that check, they then addded another check to let RHEL boot as well. We haven't yet verified whether this is an absolute string match or whether a prefix of "Red Hat Enterprise Linux" is sufficient, and further examination of the code may reveal further workarounds. For now, if you want to run Fedora[2] on these systems you're probably best off changing the firmware to perform a legacy boot.

[1] src/include/efi.h: uint8_t padding[6]; /* Emperically needed */, says the efibootmgr source code. Unhelpful.
[2] Or Ubuntu, or Suse, or

comment count unavailable comments

Dirk Eddelbuettel: Rcpp and the new R:: namespace for Rmath.h

We released Rcpp 0.10.0 earlier today. This post will just provide a simple example for one of the smaller new features -- the new namespace for functions from Rmath.h -- and illustrate one of the key features (Rcpp attributes) in passing. R, as a statistical language and environment, has very well written and tested statistical distribution functions providing probability density, cumulative distribution, quantiles and random number draws for dozens of common and not so common distribution functions. This code is used inside R, and available for use from standalone C or C++ programs via the standalone R math library which Debian / Ubuntu have as a package r-mathlib (and which can be built from R sources). User sometimes write code against this interface, and then want to combine the code with other code, possibly even with Rcpp. We allowed for this, but it required a bit of an ugly interface. R provides a C interface; these have no namespaces. Identifiers can clash, and to be safe one can enable a generic prefix Rf_. So functions which could clash such as length or error become Rf_length and Rf_error and are less likely to conflict with symbols from other libraries. Unfortunately, the side-effect is that calling, say, the probability distribution function for the Normal distribution becomes Rf_pnorm5() (with the 5 denoting the five parameters: quantile, mean, std.deviation, lowerTail, logValue). Not pretty, and not obvious. So one of the things we added was another layer of indirection by adding a namespace R with a bunch of inline'd wrapper functions (as well as several handful of unit tests to make sure we avoided typos and argument transposition and what not). The short example below shows this for a simple function taking a vector, and returning its pnorm computed three different ways:
#include <Rcpp.h>
// [[Rcpp::export]]
Rcpp::DataFrame mypnorm(Rcpp::NumericVector x)  
    int n = x.size();
    Rcpp::NumericVector y1(n), y2(n), y3(n);
    for (int i=0; i<n; i++)  
        // the way we used to do this
        y1[i] = ::Rf_pnorm5(x[i], 0.0, 1.0, 1, 0);
        // the way we can do it now
        y2[i] = R::pnorm(x[i], 0.0, 1.0, 1, 0);
     
    // or using Rcpp sugar in one go
    y3 = Rcpp::pnorm(x);
    return Rcpp::DataFrame::create(Rcpp::Named("Rold")  = y1,
                                   Rcpp::Named("Rnew")  = y2,
                                   Rcpp::Named("sugar") = y3);
 
This example also uses the new Rcpp attributes described briefly in the announcement blog post and of course in more detail in the corresponding vignette. Let us just state here that we simply provide a complete C++ function, using standard Rcpp types -- along with one 'attribute' declaration of an export via Rcpp. That's it -- even easier than using inline. Now in R we simply do
R> sourceCpp("mypnorm.cpp")
to obtain a callable R function with the C++ code just shown behind it. No Makefile, no command-line tool invocation -- nothing but a single call to sourceCpp() which takes care of things --- and brings us a compiled C++ function to R just given the source file with its attribute declaration. We can now use the new function to compute the probaility distribution both the old way, the new way with the 'cleaner' R::pnorm(), and of course the Rcpp sugar way in a single call. We build a data frame in C++, and assert that all three variants are the same:
R> x <- seq(0, 1, length=1e3)
R> res <- mypnorm(x)
R> head(res)
      Rold     Rnew    sugar
1 0.500000 0.500000 0.500000
2 0.500399 0.500399 0.500399
3 0.500799 0.500799 0.500799
4 0.501198 0.501198 0.501198
5 0.501597 0.501597 0.501597
6 0.501997 0.501997 0.501997
R> all.equal(res[,1], res[,2], res[,3])
[1] TRUE
R> 
This example hopefully helped to illustrate how Rcpp 0.10.0 brings both something really powerful (Rcpp attributes -- more on this another time, hopefully) and convenient in the new namespace for statistical functions.

Axel Beckert: Tools to handle archives conveniently

TL;DR: There s a summary at the end of the article. Today I wanted to see why a dependency in a .deb-package from an external APT repository changed so that it became uninstallable. While dpkg-deb --info foobar.deb easily shows the control information, the changelog is in the filesystem part of the package. I could extract that one dpkg-deb, too, but I d have to extract either to some temporary directory or pipe it into tar which then can extract a single file from the archive and sent it to STDOUT:
dpkg-deb --fsys-tarfile foobar.deb   tar xOf - ./usr/share/doc/foobar/changelog.Debian.gz   zless
But that s tedious to type. The following command is clearly less to type and way easier to remember:
acat foobar.deb ./usr/share/doc/foobar/changelog.Debian.gz   zless
acat stands for archive cat is part of the atool suite of commands:
als
lists files in an archive.
$ als foobar.tgz
drwxr-xr-x abe/abe           0 2012-11-15 00:19 foobar/
-rw-r--r-- abe/abe          13 2012-11-15 00:20 foobar/bar
-rw-r--r-- abe/abe          13 2012-11-15 00:20 foobar/foo
acat
extracts files in an archive to standard out.
$ acat foobar.tgz foobar/foo foobar/bar
foobar/bar
bar contents
foobar/foo
foo contents
adiff
generates a diff between two archives using diff(1).
$ als quux.zip
Archive:  quux.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2012-11-15 00:23   quux/
       16  2012-11-15 00:22   quux/foo
       13  2012-11-15 00:20   quux/bar
---------                     -------
       29                     3 files
$ adiff foobar.tgz quux.zip
diff -ru Unpack-3594/foobar/foo Unpack-7862/quux/foo
--- Unpack-3594/foobar/foo      2012-11-15 00:20:46.000000000 +0100
+++ Unpack-7862/quux/foo        2012-11-15 00:22:56.000000000 +0100
@@ -1 +1 @@
-foo contents
+foobar contents
arepack
repacks archives to a different format. It does this by first extracting all files of the old archive into a temporary directory, then packing all files extracted to that directory to the new archive. Use the --each (-e) option in combination with --format (-F) to repack multiple archives using a single invocation of atool. Note that arepack will not remove the old archive.
$ arepack foobar.tgz foobar.txz
foobar.tgz: extracted to  Unpack-7121/foobar'
foobar.txz: grew 36 bytes
apack
creates archives (or compresses files). If no file arguments are specified, filenames to add are read from standard in.
aunpack
extracts files from an archive. Often one wants to extract all files in an archive to a single subdirectory. However, some archives contain multiple files in their root directories. The aunpack program overcomes this problem by first extracting files to a unique (temporary) directory, and then moving its contents back if possible. This also prevents local files from being overwritten by mistake.
(atool subcommand descriptions from the atool man page which is licensed under GPLv3+. Examples by me.) I though miss the existence of an agrep subcommand. Guess why? atool supports a wealth of archive types: tar (gzip-, bzip-, bzip2-, compress-/Z-, lzip-, lzop-, xz-, and 7zip-compressed), zip, jar/war, rar, lha/lzh, 7zip, alzip/alz, ace, ar, arj, arc, rpm, deb, cab, gzip, bzip, bzip2, compress/Z, lzip, lzop, xz, rzip, lrzip and cpio. (Not all subcommands support all archive types.) Similar Utilities There are some utilities which cover parts of what atool does, too: Tools from the mtools package Yes, they come from the handle MS-DOS floppy disks tool package, don t ask me why. :-)
uz
gunzips and extracts a gzip d tar d archives
Advantage over aunpack: Less to type. :-)
Disadvantage compared to aunpack: Supports only one archive format.
lz
gunzips and shows a listing of a gzip d tar d archive
Advantage over als: One character less to type. :-)
Disadvantage compared to als: Supports only one archive format.
unp unp extracts one or more files given as arguments on the command line.
$ unp -s
Known archive formats and tools:
7z:           p7zip or p7zip-full
ace:          unace
ar,deb:       binutils
arj:          arj
bz2:          bzip2
cab:          cabextract
chm:          libchm-bin or archmage
cpio,afio:    cpio or afio
dat:          tnef
dms:          xdms
exe:          maybe orange or unzip or unrar or unarj or lha 
gz:           gzip
hqx:          macutils
lha,lzh:      lha
lz:           lzip
lzma:         xz-utils or lzma
lzo:          lzop
lzx:          unlzx
mbox:         formail and mpack
pmd:          ppmd
rar:          rar or unrar or unrar-free
rpm:          rpm2cpio and cpio
sea,sea.bin:  macutils
shar:         sharutils
tar:          tar
tar.bz2,tbz2: tar with bzip2
tar.lzip:     tar with lzip
tar.lzop,tzo: tar with lzop
tar.xz,txz:   tar with xz-utils
tar.z:        tar with compress
tgz,tar.gz:   tar with gzip
uu:           sharutils
xz:           xz-utils
zip,cbz,cbr,jar,war,ear,xpi,adf: unzip
zoo:          zoo
So it s very similar to aunpack, just a shorter command and it supports some more exotic archive formats which atool doesn t support. Also part of the unp package is ucat which does more or less the same as acat, just with unp as backend. dtrx From the man page of dtrx:
In addition to providing one command to extract many different archive types, dtrx also aids the user by extracting contents consistently. By default, everything will be written to a dedicated directory that s named after the archive. dtrx will also change the permissions to ensure that the owner can read and write all those files. Supported archive formats: tar, zip (including self-extracting .exe files), cpio, rpm, deb, gem, 7z, cab, rar, and InstallShield. It can also decompress files compressed with gzip, bzip2, lzma, or compress.
dtrx -l lists the contents of an archive, i.e. works like als or lz. dtrx has two features not present in the other tools mentioned so far: Unfortunately you can t mix those two features. But you can use the following tool for that purpose: deepfind deepfind is a command from the package strigi-utils and recursively lists files in archives, including archives in archives. I ve already written a detailed blog-posting about deepfind and its friend deepgrep. tardiff tardiff was written to check what changed in source code tarballs from one release to another. By default it just lists the differences in the file lists, not in the files contents and hence works different than adiff. Summary atool and friends are probably the first choice when it comes to DWIM archive handling, also because they have an easy to remember subcommand scheme. uz and lz and the shortest way to extract or list the contents of a .tar.gz file. But nothing more. And you have to install mtools even if you don t have a floppy drive. unp comes in handy for exotic archive formats atool doesn t support. And it s way easier to remember and type than aunpack. dtrx is neat if you want to extract archives in archives or if you want to extract metadata from some package files with just a few keystrokes. For listing all files in recursive archives, use deepfind.

14 November 2012

Dirk Eddelbuettel: Rcpp 0.10.0

Rcpp release 0.10.0 is now on CRAN and being uploaded to Debian. This is a new feature release, and we are very exciting about the changes, notably Rcpp attributes which make using C++ from R even easier than inline (see below as well as the new vignette for details and first examples), the extensions to Rcpp modules (see below) and more as for example new Rcpp sugar functions, a new error output device syncing to R, and a new namespace R> for the statistical functions from Rmath.h. The complete NEWS entry for 0.10.0 is below; more details are in the ChangeLog file in the package and on the Rcpp Changelog page.
Changes in Rcpp version 0.10.0 (2012-11-13)
  • Support for C++11 style attributes (embedded in comments) to enable use of C++ within interactive sessions and to automatically generate module declarations for packages:
    • Rcpp::export attribute to export a C++ function to R
    • sourceCpp() function to source exported functions from a file
    • cppFunction() and evalCpp() functions for inline declarations and execution
    • compileAttribtes() function to generate Rcpp modules from exported functions within a package
    • Rcpp::depends attribute for specifying additional build dependencies for sourceCpp()
    • Rcpp::interfaces attribute to specify the external bindings compileAttributes() should generate (defaults to R-only but a C++ include file using R_GetCCallable can also be generated)
    • New vignette "Rcpp-attribute"
  • Rcpp modules feature set has been expanded:
    • Functions and methods can now return objects from classes that are exposed through modules. This uses the make_new_object template internally. This feature requires that some class traits are declared to indicate Rcpp's wrap/as system that these classes are covered by modules. The macro RCPP_EXPOSED_CLASS and RCPP_EXPOSED_CLASS_NODECL can be used to declared these type traits.
    • Classes exposed through modules can also be used as parameters of exposed functions or methods.
    • Exposed classes can declare factories with ".factory". A factory is a c++ function that returns a pointer to the target class. It is assumed that these objects are allocated with new on the factory. On the R side, factories are called just like other constructors, with the "new" function. This feature allows an alternative way to construct objects.
    • "converter" can be used to declare a way to convert an object of a type to another type. This gets translated to the appropriate "as" method on the R side.
    • Inheritance. A class can now declare that it inherits from another class with the .derives<Parent>( "Parent" ) notation. As a result the exposed class gains methods and properties (fields) from its parent class.
  • New sugar functions:
    • which_min implements which.min. Traversing the sugar expression and returning the index of the first time the minimum value is found.
    • which_max idem
    • unique uses unordered_set to find unique values. In particular, the version for CharacterVector is found to be more efficient than R's version
    • sort_unique calculates unique values and then sorts them.
  • Improvements to output facilities:
    • Implemented sync() so that flushing output streams works
    • Added Rcerr output stream (forwarding to REprintf)
  • Provide a namespace 'R' for the standalone Rmath library so that Rcpp users can access those functions too; also added unit tests
  • Development releases sets variable RunAllRcppTests to yes to run all tests (unless it was alredy set to 'no'); CRAN releases do not and still require setting which helps with the desired CRAN default of less testing at the CRAN server farm.
Thanks to CRANberries, you can also look at a diff to the previous release 0.9.15. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Update: One link corrected.

Michal &#268;iha&#345;: Key signing party in Prague

Tomorrow (Thursday 15th November), there will be a PGP key signing and CAcert assurance party in Prague. It will be held in SUSE offices and it will start at 17:00. If you are interesting in having your PGP key signed, bring papers with your key fingerprint (you can use for example these slips) and some government issued ID. For CAcert assurance it is recommended to bring printed out assurance papers (you can get them from CAcert website) and of course IDs as well. You can find more information on Facebook or Google+ event pages.

Filed under: English Suse 2 comments Flattr this!

Next.

Previous.