Search Results: "nate"

22 November 2012

Ian Wienand: Upgrading Samsung Series 9 Laptop BIOS without Windows

Having recently acquired a Samsung Series 9 laptop, I'd like to update the BIOS. This is proving rather difficult without Windows. The first thing you download from the Samsung website isn't actually BIOS update at all; it's a Windows application which then goes and finds a BIOS update for your machine. Some time with a Windows VM, tcpdump and you'll end up at http://sbuservice.samsungmobile.com/. I'll leave interested parties to play around at that website, but with a little fiddling you can find the BIOS you're looking for. In this case, it is ITEM_20121024_754_WIN_P08AAH.exe which is the latest version, P08AAH. Unfortunately, this still won't run in any useful way -- in a virtual-machine it errors out saying the battery isn't connected. On previous endeavors such as this, running various incarnations of zip on the .exe file has found interesting components, but not in this case. Not to be deterred, some work with strings will yield the following

$ strings ./ITEM_20121024_754_WIN_P08AAH.exe   grep Copyright
inflate 1.2.7 Copyright 1995-2012 Mark Adler
deflate 1.2.7 Copyright 1995-2012 Jean-loup Gailly and Mark Adler

Now this is interesting, because it tells us that the components aren't using some fancy compression scheme we have no hope of figuring out. Checking RFC1952 we see the following header mentioned

Each member has the following structure:
+---+---+---+---+---+---+---+---+---+---+
 ID1 ID2 CM  FLG      MTIME      XFL OS   (more-->)
+---+---+---+---+---+---+---+---+---+---+

We can easily look for that in the file. Expanding the values, what we're looking for is 0x1f 0x8b 0x08 0x0 0x0 0x0 0x0 0x0 0x0 0x0 which represents the compressed stream (the OS byte is 0xb for NTFS which we can just ignore). A quick hack will get that done and give us the offsets we need

$ ./look-for-headers
Found potential header at 455212
Found potential header at 677136
Found potential header at 1293930
Found potential header at 1294980
Found potential header at 2771193
Found potential header at 2779874
Found potential header at 2789142
Found potential header at 2935778
Found potential header at 3078639
Found potential header at 3083039
Found potential header at 3338552
Found potential header at 3339443
Found potential header at 3352440
Found potential header at 3366757
Found potential header at 3391734
Found potential header at 3444255
Found potential header at 3494372

So, we can feed this into a shell script and extract the streams

FILE=ITEM_20121024_754_WIN_P08AAH.exe
for offset in 455212 677136 1293930 1294980 \
    2771193 2779874 2789142 2935778 3078639 \
    3083039 3338552 3339443 3352440 3366757 \
    3391734 3444255 3494372
do
    echo "create $offset.gz"
    dd if=$FILE  bs=1 skip=$offset   zcat > $offset.data
done

We don't have to worry about the end-point, because zcat will handily stop outputting and ignore junk at the end of the stream. That will yield the following output files:

$ md5sum *
c014e900b02ae39d6e574c63154809b2  1293930.data
982747cd5215c7d51fa1fba255557fef  1294980.data
942c6a8332d5dd06d8f4b2a9cb386ff4  2771193.data
d98d2f80b94f70780b46d1f079a38d93  2779874.data
33ce3174b0e205f58c7cedc61adc6f32  2789142.data
1d6851288f40fa95895b2fadee8a8b82  2935778.data
3a0a9b2fa9b40bdf5721068acde351ce  3078639.data
fc166141d06934e9c7238326f18a4e68  3083039.data
8cf7f86f84afa81363d3cef44b257d50  3338552.data
85e09e18fd495599041e13b4197e1746  3339443.data
0c72004eae0131d52817f23faf8a46a7  3352440.data
26347f93721e7ac49134a17104f56090  3366757.data
26973e301da4948a0016765e40700e8c  3391734.data
c8e4675cda3aa367d523a471bb205fba  3444255.data
7bf5ea753d4cc056b9462a02ac51b160  3494372.data
a2c6562f7f43aaa0e636cc1995b7ee3d  455212.data
d5c91b007dac3a5c6317f8a8833f9304  677136.data
65c0dcd1e7e9f8f87a8e9fb706eb8768  ITEM_20121024_754_WIN_P08AAH.exe
$ file *
1293930.data:                     MS-DOS executable
1294980.data:                     data
2771193.data:                     PE32 executable (native) Intel 80386, for MS Windows
2779874.data:                     PE32+ executable (native) x86-64, for MS Windows
2789142.data:                     PE32 executable (console) Intel 80386 (stripped to external PDB), for MS Windows
2935778.data:                     PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows
3078639.data:                     MS-DOS executable
3083039.data:                     PE32 executable (GUI) Intel 80386, for MS Windows
3338552.data:                     ASCII text, with CRLF line terminators
3339443.data:                     PE32 executable (native) Intel 80386, for MS Windows
3352440.data:                     PE32+ executable (native) x86-64, for MS Windows
3366757.data:                     PE32 executable (DLL) (GUI) Intel 80386, for MS Windows
3391734.data:                     PE32 executable (GUI) Intel 80386, for MS Windows
3444255.data:                     PE32 executable (GUI) Intel 80386, for MS Windows
3494372.data:                     PE32 executable (DLL) (GUI) Intel 80386, for MS Windows
455212.data:                      PE32 executable (DLL) (console) Intel 80386, for MS Windows
677136.data:                      data
ITEM_20121024_754_WIN_P08AAH.exe: PE32 executable (GUI) Intel 80386, for MS Windows

This is where I get a bit stuck. There is clearly a bunch of WinFlash stuff that Wine kind-of recognises, but so far I haven't managed to figure out how I might go about flashing this from a DOS boot disk. So far I know it's a "Phoenix SecureCore Tiano" based BIOS, and there is some indication that there is a UEFI app to flash the BIOS, but unfortunately I didn't set the laptop up with UEFI (note to anyone setting this up; probably a good idea to do that). I believe that if you have Windows, you can halt the install part-way through and see these files extracted in a temp directory. Possibly if we can put some file-names to the blobs it might help. There may also be a much easier way to do this that I'm completely missing. If anything comes up, I'll update this post; hopefully a future Googlenaut who enjoys BIOS hacking will drop a comment... Update 2012-11-27 I'm unfortunately less optimistic about being able to update this. Certainly the files identified above as MS-DOS executable are EFI binaries; however they are both 32-bit binaries. You can easily get an EFI shell up and running by putting a 64-bit EFI shell binary on a FAT formatted USB disk in the efi/boot/bootx64.efi but it will not run either of those binaries, presumably because they're 32-bit. A 32-bit EFI binary doesn't seem to be recognised for boot, so no luck there. A comment also mentioned some issues with Samsung's and UEFI boot in this bug though it's not clear what models are affected. Another comment mentioned binwalk which also does a nice job of finding the embedded zip streams.

$ strings ./1293930.data
...
RSDS
D:\flash\Source\ENV\src\Efi863\edk\phoenix\edk\Sample\Platform\DUET\ldr16\IA32\oemslp20.pdb
oemslp20.dll
InitializeDriver
2n3u3
$ strings ./3078639.data
en-us;zh-Hans
slp20.marker
slp20.size
slp20.offset
slp20.offset2
read marker fail!
Marker file '%s''s size(%d)!=%d
Not a valided OemMarker!
Update image data error.
marker file is updated.
marker file size too big!
Not a valided OemMarker in ROM.
Update image data error.
SLP marker is reserved.
-PROT
Prot flag is found, mProgFlag is true.
Prot flag is not found, mProgFlag is false.
Please specify input file name.
-PROT
slpFile
OEM Activation 2.0 processing
Failed to Locate Flash Support protocol.
...
D:\flash\Source\ENV\src\Efi863\edk\phoenix\edk\Sample\Platform\DUET\ldr16\IA32\SLP20.pdb

21 November 2012

Erich Schubert: Phoronix GNOME user survey

While not everybody likes Phoronix (common complaints include tabloid journalism), they are doing a GNOME user survey again this year. If you are concerned about Linux on the desktop, you might want to participate; it is not particularly long.

Unfortunately, "the GNOME Foundation still isn't interested in having a user survey", and may again ignore the results; and already last year you could see a lot of articles along the lines of The Survey That GNOME Would Rather Ignore. One more reason to fill it out.

Dirk Eddelbuettel: Rcpp attributes: A simple example 'making pi'

We introduced Rcpp 0.10.0 with a number of very nice new features a few days ago, and the activity on the rcpp-devel mailing list has been pretty responsive which is awesome. But because few things beat a nice example, this post tries to build some more excitement. We will illustrate how Rcpp attributes makes it really easy to add C++ code to R session, and that that code is as easy to grasp as R code. Our motivating example is everybody's favourite introduction to Monte Carlo simulation: estimating . A common method uses the fact the unit circle has a surface area equal to . We draw two uniform random numbers x and y, each between zero and one. We then check for the distance of the corresponding point (x,y) relative to the origin. If less than one (or equal), it is in the circle (or on it); if more than one it is outside. As the first quadrant is a quarter of a square of area one, the area of the whole circle is -- so our first quadrant approximates over four. The following figure, kindly borrowed from Wikipedia with full attribution and credit, illustrates this: Example of simulating pi

Now, a vectorized version (drawing N such pairs at once) of this approach is provided by the following R function.

piR <- function(N)  
    x <- runif(N)
    y <- runif(N)
    d <- sqrt(x^2 + y^2)
    return(4 * sum(d < 1.0) / N)

And in C++ we can write almost exactly the same function thanks the Rcpp sugar vectorisation available via Rcpp:

#include <Rcpp.h>
using namespace Rcpp;
// [[Rcpp::export]]
double piSugar(const int N)  
    RNGScope scope;		// ensure RNG gets set/reset
    NumericVector x = runif(N);
    NumericVector y = runif(N);
    NumericVector d = sqrt(x*x + y*y);
    return 4.0 * sum(d < 1.0) / N;

Sure, there are small differences: C++ is statically typed, R is not. We need one include file for declaration, and we need one instantiation of the RNGScope object to ensure random number draws remain coordinated between the calling R process and the C++ function calling into its (compiled C-code based) random number generators. That way we even get the exact same draws for the same seed. But the basic approach is identical: draw a vector x and vector y, compute the distance to the origin and then obtain the proportion within the unit circle -- which we scale by four. Same idea, same vectorised implementation in C++. But the real key here is the one short line with the [[Rcpp::export]] attribute. This is all it takes (along with sourceCpp() from Rcpp 0.10.0) to get the C++ code into R. The full example (which assumes the C++ file is saved as piSugar.cpp in the same directory) is now:

#!/usr/bin/r
library(Rcpp)
library(rbenchmark)
piR <- function(N)  
    x <- runif(N)
    y <- runif(N)
    d <- sqrt(x^2 + y^2)
    return(4 * sum(d < 1.0) / N)
 
sourceCpp("piSugar.cpp")
N <- 1e6
set.seed(42)
resR <- piR(N)
set.seed(42)
resCpp <- piSugar(N)
## important: check results are identical with RNG seeded
stopifnot(identical(resR, resCpp))
res <- benchmark(piR(N), piSugar(N), order="relative")
print(res[,1:4])

and it does a few things: set up the R function, source the C++ function (and presto: we have a callable C++ function just like that), compute two simulations given the same seed and ensure they are in fact identical -- and proceed to compare the timing in a benchmarking exercise. That last aspect is not even that important -- we end up being almost-but-not-quite twice as fast on my machine for different values of N. The real takeaway here is the ease with which we can get a C++ function into R --- and the new process completely takes care of passing parameters in, results out, and does the compilation, linking and loading. More details about Rcpp attributes are in the new vignette. Now enjoy the .

Update:One somewhat bad typo fixed.

Update:Corrected one background tag.

Axel Beckert: zutils: zcat and friends on Steroids

I recently wrote about tools to handle archives conveniently. If you just have to handle compressed text files, there are some widely known shortcut commands to mimic common commands on files compressed with a specific compression format.

	gzip	bzip2	lzma	xz
cat	zcat	bzcat	lzcat	xzcat
cmp	zcmp	bzcmp	lzcmp	xzcmp
diff	zdiff	bzdiff	lzdiff	xzdiff
grep	zgrep	bzgrep	lzgrep	xzgrep
egrep	zegrep	bzegrep	lzegrep	xzegrep
fgrep	zfgrep	bzfgrep	lzfgrep	xzfgrep
more	zmore	bzmore	lzmore	xzmore
less	zless	bzless	lzless	xzless

In Debian and derivatives, those tools are part of the according package for that compression utility, i.e. the zcat command is part of the gzip package and the xzfgrep command is part of the xz-utils package. But despite this matrix is quite easy to remember, the situation has a few drawbacks:

Those tools can only handle the format they re written for (which btw. means that all xz-tools can also handle lzma-compressed files as lzma is xz s predecessor)
zcat and the other cat variants can t even recognize non-compressed files and throw an error instead of just showing their contents.
I always tend to think that lzcat and friends are for lzip-based compression as xzcat can handle lzma-compressed files anyway.

This is where the zutils project comes in: zutils provides the functionality of most of these utilities, too, but with one big difference: You don t have to remember, think about or type which compression method has been used for your data, just use zcat, zcmp, zdiff, zgrep, zegrep, or zfgrep and it works independently of what compression method has been used if any or if there are different compression types mixed in the parameters to the same command:

$ zfgrep foobar bla.txt fnord.gz hurz.xz quux.lz bar.lzma

Especially if you use logrotate and let logrotate compress old logs, it s very comfortable that one command suffices to concatenate all the available logfiles, including the current uncompressed one:

$ zcat /var/log/syslog*

Additionally, zutils versions of these tools also support lzip-compressed files. The zutils package is available in Debian starting with Wheezy and in Ubuntu since Oneiric. When being installed, it replaces the original z* utilities from the gzip package by diverting them away. The only drawback so far is that there neither a zless nor a zmore utility from the zutils project, so

zless bla.txt fnord.gz hurz.xz quux.lz
bar.lzma

will not work as expected even after installing zutils as it is still the one from the gzip package and hence it will show you just the first two files in plain text, but not the remaining ones.

20 November 2012

Pablo Lorenzzoni: Decision-making by flipping a coin

Yesterday I was discussing in an online board how to break a decision deadlock in life. There are real deadlocks, but I don t think they are very frequent most of the time, doing a pros and cons analysis is enough to decide what to do. Sometimes our judgement is impaired by lack of objectivity or by our inability to see things from a different point of view (it s hard to think straight when everything seems to be falling apart around you); talking to a friend or relative can help in these situations. But there are times when nothing helps. When you are really stuck and nothing seems to break the deadlock. At these times I give it one or two days, sleep on the issue and if I cannot come up with a decision I assume the alternatives are equivalent to me and just flip a coin. Of course, this will do if you can postpone the decision, giving yourself (and your inner self) time to decide on a course-of-action. But is there some way to speed up the process? During the discussion, someone came up with a rather smart quote by Rothstein character in Boardwalk Empire TV series:

Flip a coin. When it s in the air, you ll know what side you re hoping for.

I found it an interesting way to give your inner self an ultimatum: decide or luck will decide instead. I ve never did it like that, for I would not flip a coin unless it s my last resort. Being such a clever psychological idea, I doubted it originated in the TV series itself, so I went on pursue of the original idea. I found a Danish poet and mathematician called Piet Hein, who wrote a poem about it circa 1969:

A PSYCHOLOGICAL TIP Whenever you re called on to make up your mind,
and you re hampered by not having any,
the best way to solve the dilemma, you ll find,
is simply by spinning a penny.
No not so that chance shall decide the affair
while you re passively standing there moping;
but the moment the penny is up in the air,
you suddenly know what you re hoping.

And also, there are a Donald Duck comic called Flip Decision circa 1953 that introduces Flipism philosophy, which supports a rather radical anecdotal variant: make all decisions by flipping a coin. Piet Hein or Rothstein character idea is much better than Flipism, of course. Does anyone knows any older sources of the same idea?

19 November 2012

Steve Langasek: Pflaumenkuchen

This was a good year for plums in the garden, both for the yellow plums and for the Italian prunes - enough so that it took some doing to figure out what to do with them all. Since I'm not in a hurry to set up a still and make slivovic, and you can only pawn so many plums off on friends and neighbors, I had on the order of 15 pounds of Italian prunes to dispense with. With our change of diet to eliminate extra carbs, Patty and I have both been experimenting with reduced-carb desserts in the kitchen. And I've always been fond of central European (e.g., German) desserts, which tend to be sweetened much more lightly than American equivalents. Indeed, my earliest impression of "coffee cake" comes from the home of an elderly couple who were friends of the family, who served a delicious plum cake in their home. She was from Bavaria, so I guess she probably wouldn't have called it Zwetchgendatschi like the Austrians do (if they even really call it that; I have my suspicions that this is a made-up name designed to poke fun at the foreigners). Maybe she would have called it Pflaumenkuchen. The Internet calls it "Polish plum cake". Whatever you call it, it's tasty, and I aimed to recreate it at home. The plums are now nearly a month gone, but after multiple iterations the cake turned out well enough that I feel compelled to share it with the Internet. Since we have friends who are dairy-intolerant, this particular recipe is tailored to be low-carb and dairy-free. It doesn't lose anything in the taste for being made with a restricted set of ingredients. The original recipe did call for a streusel, and a non-dairy streusel is pretty pointless so this recipe simply omits it. Metric cooks, avert your eyes. ;)

1 c almond flour

1 tsp baking powder

tsp salt

c Steviva blend

3 tbsp grapeseed oil

c milk

1 large egg

12 fresh plums, pitted and halved

tsp cloves

Lightly coat an 8"x8" pan with cooking spray. Heat oven to 350 degrees. In medium bowl, combine flour, baking powder, salt and Steviva. Add grapeseed oil, milk and egg. Beat at medium speed 4 minutes. Pour batter into prepared pan. Place plums on top, cut side up, pushing down slightly into batter. Sprinkle cloves over the plums. Bake about 50 minutes or until a toothpick tests clean. Let the cake cool in the pan so the plum juices will be reabsorbed to create a moist cake. Sprinkle with confectioners' sugar, if desired, and cut into 16 squares.

[edit: the Internet is bad at me for my badly spelled german. Spelling corrected. ;)]

Steve Langasek: Pflaumkuchen

This was a good year for plums in the garden, both for the yellow plums and for the Italian prunes - enough so that it took some doing to figure out what to do with them all. Since I'm not in a hurry to set up a still and make slivovic, and you can only pawn so many plums off on friends and neighbors, I had on the order of 15 pounds of Italian prunes to dispense with. With our change of diet to eliminate extra carbs, Patty and I have both been experimenting with reduced-carb desserts in the kitchen. And I've always been fond of central European (e.g., German) desserts, which tend to be sweetened much more lightly than American equivalents. Indeed, my earliest impression of "coffee cake" comes from the home of an elderly couple who were friends of the family, who served a delicious plum cake in their home. She was from Bavaria, so I guess she probably wouldn't have called it Zwetchgendatschi like the Austrians do (if they even really call it that; I have my suspicions that this is a made-up name designed to poke fun at the foreigners). Maybe she would have called it Pflaumkuchen. The Internet calls it "Polish plum cake". Whatever you call it, it's tasty, and I aimed to recreate it at home. The plums are now nearly a month gone, but after multiple iterations the cake turned out well enough that I feel compelled to share it with the Internet. Since we have friends who are dairy-intolerant, this particular recipe is tailored to be low-carb and dairy-free. It doesn't lose anything in the taste for being made with a restricted set of ingredients. The original recipe did call for a streusel, and a non-dairy streusel is pretty pointless so this recipe simply omits it. Metric cooks, avert your eyes. ;)

1 c almond flour

1 tsp baking powder

tsp salt

c Steviva blend

3 tbsp grapeseed oil

c milk

1 large egg

12 fresh plums, pitted and halved

tsp cloves

Lightly coat an 8"x8" pan with cooking spray. Heat oven to 350 degrees. In medium bowl, combine flour, baking powder, salt and Steviva. Add grapeseed oil, milk and egg. Beat at medium speed 4 minutes. Pour batter into prepared pan. Place plums on top, cut side up, pushing down slightly into batter. Sprinkle cloves over the plums. Bake about 50 minutes or until a toothpick tests clean. Let the cake cool in the pan so the plum juices will be reabsorbed to create a moist cake. Sprinkle with confectioners' sugar, if desired, and cut into 16 squares.

18 November 2012

Sven Mueller: Ingress

As some may have noticed, Google launched something that is unusual for the company: A game. A mobile alternate reality game, to be precise. See allthingsd article for a bit more information. In my opinion this game is most interesting to people who are mildly interested in geo-caching. You spend most of the gaming time either looking for the right portals (geo-location that is an interesting place for one reason or another, many located at sculptures, murals or (more scarcely) interesting local businesses), or trying to solve the puzzles that appear in the game itself or on http://www.nianticproject.com/. I was one of the beta testers and really enjoy the game, especially in areas with more players. As the game is most fun when people coordinate a bit, I started a google group for the munich area for one of the two teams (the enlightenment), but I will only let people enter that group who can be verified to be on that team. On a side note: I still have a few invites to the game left, so if you are interested, please leave a comment and enter your gmail.com email address (you need a Google account to play anyway) while doing so. I will send out invites in the order I received the requests (which I admit might not always be the same in which they were submitted. First come first serve and no guarantee to get an invitation though. Your email address you enter when commenting isn t shown to anyone and I don t share them with anyone but the game (and thus Google). EDIT: So short time went past, but I already ran out of invites (note that I didn t approve comments that really only were like please send me an invite , but still did sent one out). Feel free to still ask me for an invite, I will send some out as I get new ones, but this might take anything between a few days and some weeks. PS: For the lucky ones who got invites: You don t strictly need to do the training offered when you start the game for the first time, but I strongly encourage you to do so.
PPS: I m opening my group to any enlightened players in Germany. I will share some useful information about some game details there in the near future (nothing company confidential of course, just information anyone can gather in the game or from the published information). EDIT: As noted before, I don t have any more invites to give away. As I still get like 4 requests for invites per day, and now have more than 30 unfulfilled requests pending (with no new invites to give in sight), I m now disabling comments. Disclaimer: I work for Google since a few months ago, but this post solely represents my own opinions and hasn t been endorsed by Google in any way.

Neil Williams: Long term maintenance of perl cross-build support in Debian

After prompts from Wookey and Steve McIntyre, I decided to look at #285559 and #633884 for perl cross-build support and then port that support forward to the current perl in Wheezy and on to the version of perl currently in experimental. The first patch is for perl 5.8, the second for perl 5.12, neither of which is available currently in Debian. snapshot.debian.org provided the 5.12 source but then that no longer cross-builds with the patch.

The problem, as with any cross build, is that the build must avoid trying to execute binaries compiled within the build to achieve the test results required by ./configure (or in the case of perl, Configure). dpkg-cross has one collection of cache values but the intention was always to migrate the package-specific support data into the packages themselves and keep the architecture-specific data in dpkg-cross or dpkg-dev. Therefore, the approach taken in #633884 would be correct, if only there was a way of ensuring that the cached values remain in sync with the relevant Debian package.

I'll note here that I am aware of other ways of cross-building perl, this is particularly concerned with cross-building the Debian configuration of perl as a Debian package and using Debian or Emdebian cross-compilers. After all, the objective is to support bootstrapping Debian onto new architectures. However, I fully expect this to be just as usable with Ubuntu packages of perl compiled with, e.g. Linaro cross-compilers but I haven't yet looked at the differences between perl in Debian vs Ubuntu in any detail.

I've just got perl 5.14.2 cross-building for armel using the Emdebian gcc-4.4 cross-compiler (4.4.5-8) on a Debian sid amd64 machine without errors (it needs testing, which I'll look at later), so now is the time to document how it is done and what needs to be fixed. I've already discussed part of this with the current perl maintainers in Debian and, subject to just how the update mechanism works, have outline approval for pushing these changes into the Debian package and working with upstream where appropriate. The cache data itself might live in a separate source package which will use a strict dependency on perl to ensure that it remains in sync with the version which it can cross-build. Alternatively, if I can correctly partition the cache data between architecture-specific (and therefore generated from the existing files) and package_$version specific, then it may be possible to push a much smaller patch into the Debian perl package. This would start with some common data, calculate the arch-specific data and then look for some version-specific data, gleaned from Debian porter boxes whilst the version is in Debian experimental.

The key point is that I've offered to provide this support for the long term, ensuring that we don't end up with future stable releases of Debian having a perl package which cannot be cross-built. (To achieve that, we will also end up with versions of perl in Debian testing which also cross-build.)

This cross-build is still using dpkg-cross paths, not MultiArch paths, and this will need to be changed eventually. (e.g. by the source package providing two binaries, one which uses MultiArch and one which expects dpkg-cross paths.) The changes include patches for the upstream Makefile.SH, debian/rules and the cache data itself. Depending on where the cache data finally lives, the new support might or might not use the upstream Cross/ directory as the current contents date from the Zaurus support and don't appear to be that useful for current versions of perl.

The cache data itself has several problems:

It is tightly tied to the version of perl which generated it.

It is, as expected, architecture-dependent

It is, unfortunately, very sensitive to the specific configuration used by the distribution itself

That last point is important because it means that the cache data is not useful upstream as a block. It also means that generating the cache data for a specific Debian package means running the generation code on the native architecture with all of the Debian build-dependencies installed for the full perl build. This is going to complicate the use of this method for new architectures like arm64.

My objective for the long term maintenance of this code is to create sufficient data that a new architecture can be bootstrapped by judicious use of some form of template. Quite how that works out, only time will tell. I expect that this will involve isolating the data which is truly architecture-specific which doesn't change between perl versions from the data which is related to the tests for build-dependencies which does change between perl versions and then work out how to deal with any remainder. A new architecture for a specific perl version should then just be a case of populating the arch-specific data such as the size of a pointer/char and the format specifiers for long long etc. alongside the existing (and correct) data for the current version of perl.

Generating the cache data natively

The perl build repeats twice (three builds in total) and each build provides and requires slightly different cache data - static, debug and shared. Therefore, the maintenance code will need to provide a script which can run the correct configuration step for each mode, copy out the cache data for each one and clean up. The script will need to run inside a buildd chroot on a porter box (I'm looking at using abel.debian.org and harris.debian.org for this work so far) so that the derived data matches what the corresponding Debian native build would use. The data then needs slight modification - typically to replace the absolute paths with PERL_BUILD_DIR. It may also be necessary to change the value of cc, ranlib and other compiler-related values to the relevant cross-compiler executables. That should be possible to arrange within the build of the cache data support package itself, allowing new cache files to be dropped in directly from the porter box.

The configuration step may need to be optimised within debian/rules of perl itself as it currently proceeds on from the bare configuration to do some actual building but I need to compare the data to see if a bare config is modified later. The test step can be omitted already. Each step is performed as:



DEB_BUILD_OPTIONS="nocheck" fakeroot debian/rules perl.static

That is repeated for perl.debug and libperl.so.$(VERSION) where $VERSION comes from :



/bin/bash debian/config.debian --full-version

The files to be copied out are:

bitcount.h - architecture-specific (may be identical between similar architectures like armel & armhf)

config.h - architecture & build specific, config.h.debug, config.h.static, config.h.shared.

config.sh - architecture & build specific config.sh.debug, config.sh.static, config.sh.shared.

uudmap.h - architecture-specific (may be identical between similar architectures like armel & armhf)

There is a lot of scope for templating of some form here, e.g. config.h.debug is 4,686 lines long but only 41 of those lines differ between amd64 and armhf for the same version of perl (and some of those can be identified from existing architecture-specific constants) which should make for a much smaller patch.

Architecture-specific cache data for perl

So far, the existing patches only deal with armel and armhf. If I compare the differences between armel & armhf, it comes down to:

compiler names (config.h.*)

architecture names (config.sh.*)

architecture-based paths (config.sh.*)

However, comparing armel and armhf doesn't provide sufficient info for deriving arm64 or mips etc. Comparing the same versions for armhf and amd64 shows the range of differences more clearly. Typical architecture differences exist like the size of a long, flags to denote if the compiler can cast negative floats to 32bit ints and the sprintf format specifier strings for handling floats and doubles. The data also includes some less expected ones like:



armhf: #define Netdb_host_t		const void * /**/

amd64: #define Netdb_host_t		char * /**/

I'm not at all sure why that is arch-specific - if anyone knows, email codehelp @ d.o - same address if anyone fancies helping out ....

Cross-builds and debclean

When playing with the cross-build, remember to use the cross-build clean support, not just debclean:



dpkg-architecture -aarmel -c fakeroot debian/rules clean

That wasted quite a bit of my time initially with having to blow away the entire tree, unpack it from original apt sources and repatch it. (Once Wheezy is out, may actually investigate getting debclean to support the -a switch).

OK, that's an introduction, I'm planning on pushing the cross-build support code onto github soon-ish and doing some testing of the cross-built perl binaries in a chroot on an armel box. I'll detail that in another blog post when it's available.

Next step is to look at perl 5.16 and then current perl upstream git to see how to get Makefile.SH fixed for the long term.

Mike Hommey: Debian EFI mode boot on a Macbook Pro, without rEFIt

Diego s post got me to switch from grub-pc to grub-efi to boot Debian on my Macbook Pro. But I wanted to go further: getting rid of rEFIt. rEFIt is a pretty useful piece of software, but it s essentially dead. There is the rEFInd fork, which keeps it up-to-date, but it doesn t really help with FileVault. Moreover, the boot sequence for a Linux distro with rEFIt/rEFInd looks like: Apple EFI firmware rEFIt/rEFInd GRUB Linux kernel. Each intermediate step adding its own timeout, so rEFIt/rEFInd can be seen as not-so-useful intermediate step. Thankfully, Matthew Garrett did all the research to allow to directly boot GRUB from the Apple EFI firmware. Unfortunately, his blog post didn t have much actual detail on how to do it. So here it is, for a Debian system:

Install a few packages you ll need in this process:
```
# apt-get install hfsprogs icnsutils
```

Create a small HFS+ partition. I have a 9MB one, but it s only filled by about 500K, so even smaller should work too. If, like me, you were previously using grub-pc, you probably have a GRUB partition, you can repurpose it. In gdisk, it looks like this:

Number  Start (sector)    End (sector)  Size       Code  Name
   5       235284480       235302943   9.0 MiB     AF00  Apple HFS/HFS+
Partition GUID code: 48465300-0000-11AA-AA11-00306543ECAC (Apple HFS/HFS+)
Partition unique GUID: AD1F5465-B777-4178-AC4D-1DE8B2EB1B4B
First sector: 235284480 (at 112.2 GiB)
Last sector: 235302943 (at 112.2 GiB)
Partition size: 18464 sectors (9.0 MiB)
Attribute flags: 0000000000000000
Partition name: 'Apple HFS/HFS+'

Create a HFS+ filesystem on that partition:

# mkfs.hfsplus /dev/sda5 -v Debian
(replace /dev/sda5 with whatever your partition is)

Add a fstab entry for that filesystem:

# echo $(blkid -o export -s UUID /dev/sda5) /boot/efi auto defaults 0 0 >> /etc/fstab

Mount the filesystem:
```
# mkdir /boot/efi
# mount /boot/efi
```

Edit /usr/sbin/grub-install, look for xfat , and remove the block of code that looks like:

if test "x$efi_fs" = xfat; then :; else
    echo "$ efidir  doesn't look like an EFI partition." 1>&2
    efidir=
fi

Run grub-install. At this point, there should be a /boot/efi/EFI/debian/grubx64.efi file (if using grub-efi-amd64).
Create a /boot/efi/System/Library/CoreServices directory:
```
# mkdir -p /boot/efi/System/Library/CoreServices
```

Create a hard link:

# ln /boot/efi/EFI/debian/grubx64.efi /boot/efi/System/Library/CoreServices/boot.efi

Create a dummy mach_kernel file:

# echo "This file is required for booting" > /boot/efi/mach_kernel

Grab the mactel-boot source code, unpack and build it:

# wget http://www.codon.org.uk/~mjg59/mactel-boot/mactel-boot-0.9.tar.bz2
# tar -jxf mactel-boot-0.9.tar.bz2
# cd mactel-boot-0.9
# make PRODUCTVERSION=Debian

Copy the SystemVersion.plist file:

# cp SystemVersion.plist /boot/efi/System/Library/CoreServices/

Bless the boot file:

# ./hfs-bless /boot/efi/System/Library/CoreServices/boot.efi

(optional) Add an icon:

# rsvg-convert -w 128 -h 128 -o /tmp/debian.png /usr/share/reportbug/debian-swirl.svg
# png2icns /boot/efi/.VolumeIcon.icns /tmp/debian.png
# rm /tmp/debian.png

Now, the Apple Boot Manager, shown when holding down the option key when booting the Macbook Pro, looks like this:

And the Startup disk preferences dialog under OSX, like this:

17 November 2012

Antoine Beaupr : First stable release of the Drupal OpenID Provider

A short announcement to mark the release of the first stable OpenID provider for Drupal. Started by James Walker around 4 years ago, I have since then got commit access and taken care of the module for the last 2 years. And while it was not an easy task, with the help of the community, this module was able to mature to a fully featured, flexible and standard implementation during that time. One has to wonder however, is it too late for OpenID?

A bit of history Drupal was one of the pioneers in OpenID support. Back in 2006 when the protocol was still in its infancy (less than a year old!), Drupal was adding OpenID support to its 4.7 release through a contrib module and two years later, the Drupal 6.0 release was sporting OpenID in core, although not without significant interoperability issues. Still, we could say that rapid adoption by such a popular CMS as Drupal really boosted OpenID's adoption. At about the same time, Yahoo, Sourceforge, Myspace, Microsoft and Google would all join the OpenID bandwagon. On the OpenID "provider" side of things, things were not so great however. Back when I started experimenting with this, it wasn't actually that easy to host your own OpenID provider, to allow your users to login to other OpenID-enabled sites. While there are a lot of different libraries to provide OpenID support in your application there was not (and still isn't) any software that gives you an OpenID provider and is dedicated to that. No official reference implementation either, other than the code from the one of the founding companies of OpenID, JanRain. There was an OpenID provider module in Drupal, but there was no official release done until a two beta releases in 2010.

Koumbit's community involvement It is about that time I started getting involved in the project. Koumbit needed an easy way for our workers to log into multiple sites without too much trouble, so I started looking at OpenID provider implementations. The lack of dedicated software somehow made the Drupal implementation a natural fit, and I started experimenting with the module, filing bugs and reviewing patches, as there were a lot of interoperability and design issues with the provider. After a while, walkah was kind enough to give me full access to the project and I started doing regular releases (every few months), culminating in today's official 1.0 release for both Drupal 6 and Drupal 7, which I am very proud and happy about. While I am here, I want to send many thanks to the people behind the Drupal 7 port: paranojik, mikecar, xamanu and testing from neizod. A lot of issues were also patched, reviewed and tested by numerous contributors that are unfortunately difficult to track down other than reading the commit log. So we at Koumbit have been running our own OpenID provider for a good while now, and it has the key advantage of being connected to our LDAP directory (with the D6 LDAP integration module so it is, in a sense, a central point for authentication at Koumbit. Very often people don't clearly understand what their LDAP password does, and I can easily remind them "it's the OpenID thing", or "try your password in the OpenID site" if they think they have forgotten it. So much goes on in the web browser these days that it makes everything much easier.

The problem with Drupal's OpenID provider However, it is kind of embarrassing that it took so long. More than 6 years after the OpenID protocol was written, we have a stable OpenID provider. Now, this may be because I am overly conservative about stable releases, but I don't think so: the OpenID provider module had, for a long time, serious interoperability issues with OpenID sites. There was also a clear lack of features on the OpenID provider, a gap that various third party modules, now documented on the OpenID provider homepage are now trying to fill. There is also serious security issues with the protocol itself. The complexity and lack of clarity of the OpenID specification have rendered my work quite difficult at time, as the protocol has significant ambiguities and no clear reference implementation. OpenID is unfortunately renowned to be vulnerable to phishing and while Koumbit's use of the HTTPS protocol alleviates some of those issues (especially the redirect session hijack), tools like sslstrip can bypass those protections easily. The privacy aspect is also problematic, as the user is usually not free to choose which information from his or her profile is divulged to the site they are logging into. Of course those issues may be overlooked if the security trade off is not worth it. In the case of Koumbit, we use OpenID on a lot of client sites, but usually nothing critical, and we train our users to pay attention to the HTTPS icon when login in their OpenID account.

Are there alternatives? But unfortuantely, the reality is that there are not really any alternatives to OpenID out there. Mozilla's new BrowserID initiative (now rebranded Persona, boy those guys sure love rebranding..) is interesting, but actually has even more flaws than OpenID itself. Not only is it not resolving the phishing problem, it actually creates new problems, including my personal favorite, trusting server-side Javascript code, which OpenID is not using that much. I should also mention the Shibboleth module which has a surprising number of installs recorded on Drupal.org, but that depends on the much more complicated Shibboleth federation, for which there is no server implementation for Drupal, from what I understand. Oauth, on the other end, is even more widely used, and has server-side components like Oauth Login Provider and Services 3.x. We could also simply give up and use our souls stored at Facebook, Google and Twitter for authentication everywhere. Maybe OpenID opened the door for those corporations to see that there is a real opportunity in leveraging user's profile to track their online activity. It is not something that OpenID was necessarily designed for but it's an idea that certainly permeates Facebook connect and "like" buttons or Twitter and Google online business strategies, where you are the product sold to advertisers. I personally believe we should keep the struggle for distributed and standard identity mechanisms for the web that respect users' privacy. With Drupal's OpenID Provider, it is possible to create a secure, privacy-aware identity provider, although I still have to perform that damn security audit. Experiments with those third party modules should also be encouraged and I welcome any feedback on those modules. While I do not plan to attend OpenID conferences or do significant development on the OpenID provider right now, I will certainly make sure Koumbit keeps its commitment to maintaining stewardship on the OpenID provider module in Drupal. I encourage people to try it out, submit issues and patches to make it even better. Long life to OpenID! PS: if people are interested in funding that security audit, I would be happy to perform this through my regular work at Koumbit.org, just file a request at services@koumbit.org.

Michal Čihař: phpMyAdmin documentation now in Spanish

Since converting the phpMyAdmin documentation to new format and updating it's content, I was quite frightened how this will affect translated versions of it. Fortunately giving it more attractive format and more advertisement lead to increased contributions. Since Friday, we already have one language which has reached 100% - Spanish. Mat as Bellone simply did there awesome job and went through all fuzzy and new strings in just few days. Now I'm looking forward to other contributors as well, I hope we will be able to deliver translated manual for phpMyAdmin 4.0 in many languages. I'll definitely try to translate to Czech, but any help with that is welcome.

Filed under: English Phpmyadmin 2 comments Flattr this!

Russell Coker: Geeky Jeans

It s likely that most people make things like comfort, style, and price the main criteria when purchasing clothes. But there are other things that can be more important such as the ability to fit a phone in the pocket. My last pair of jeans was from Rivers (one of the more affordable Australian clothing stores which also has online sales) [1]. They were the long leg version and have front pockets that are 28cm deep (measured from the bottom of the pocket to the lowest part of the lip) and 15cm wide (a 15cm ruler will barely fit sideways in the pocket). I ve just got some new jeans from Rivers which are the regular leg length, they have front pockets which are 21cm deep and slightly more than 16cm wide. The old pair could fit a Nexus 7 tablet in the front pocket. The new pair should more easily fit such a tablet in the pocket but it might be less comfortable to walk with the tablet in the pocket. I don t plan to try using my front pocket for a tablet (I d be more likely to use a backpack or my Scott e vest [2]), but a Galaxy Note 2 (which is about the largest device that most people would want in their pocket) would fit nicely. I find the Rivers jeans to be quite comfortable and I like the way they look. They also only cost $25 online or $30 in the store. When I bought my first pair before I even had a 4 phone they were good value and they will be even better value early next year when I get a bigger phone. Even though Rivers jeans may wear out faster than more expensive brands, for $30 it s easy to just buy a few pairs at a time. During an email discussion of geeky clothing the issue of women s clothes having fewer and smaller pockets was raised. Unfortunately I didn t think to measure the pockets in women s jeans when I was at the Rivers to discover whether they have big pockets too. I ll do that next time I m in the area.

[1] http://rivers.com.au/
[2] http://www.scottevest.com/

16 November 2012

Diego Escalante Urrelo: EFI mode boot on Macbook3,1 with Debian

After my last post I got curious about booting the Macbook in EFI mode. Because, well, why not? So I googled for a while and found this blog post which served as a great reference and comment on two other. I understand EFI is somewhat buggy and what works for me might not work for you. So be careful, have a rescue disk at hand. On that topic, I should say that I could not get this to work with Debian sid s 3.2 kernel, but 3.5 from experimental worked fine for me.

Hopefully, you after booting EFI succesfully. My main question before doing this was What works? , I had heard some weird rumours and mentions that you had to run X in framebuffer mode and weird stuff like that. This is bullshit. Everything works fine, with the same standard software and setup. Features:

GNOME Shell, hence acceleration, works
All drivers work: wireless, bluetooth, etc.
No need to force AHCI mode on the disk controller
No graphic flicker during the boot process
A feeling of catching up with the times

So, here we go. I am posting my notes in shell-like since I find little benefit in turning them into an essay or something more textual. Good luck! # Install grub-efi-$arch.
# Usually $arch depends on your processor.
# Best bet: you can use OSX to check for this:
#
# $ ioreg -l -p IODeviceTree grep firmware-abi
# It will print EFI32, or EFI64. You know what to do.
$ sudo apt-get install grub-efi-amd64 # Now we need to mount the system EFI partition for GRUB2 to install
# itself on it.
# NOTE: you have to do this, installing the package is NOT enough.

$ sudo mkdir /boot/efi

$ sudo mount /dev/sda1 /boot/efi

# Did you get the right partition?
# It should have an EFI directory with an APPLE directory inside.
# DO NOT DELETE THE APPLE DIRECTORY, PRICK.

$ ls /boot/efi

EFI/

$ ls /boot/efi/EFI/

APPLE/

# Create a new directory, debian will do.
# Run grub-install so GRUB2 drops its EFI code in the new directory.

$ sudo mkdir /boot/efi/EFI/debian

$ sudo grub-install

$ ls /boot/efi/EFI/

$ ls /boot/efi/EFI/debian/

# Help rEFIt know this is a GNU/Linux boot.
# rEFIt assigns icons depending on the name of the EFI file, grub64x is not in its list
$ sudo mv /boot/efi/EFI/debian/grub64x.efi /boot/efi/EFI/debian/e.efi # IMPORTANT: Add the proper modeset.
# For EFI-only systems: efi_uga
# For UEFI systems: efi_gop
# Without this you won t have any graphics.

$ sudo vim /etc/default/grub

...

GRUB_VIDEO_BACKEND="efi_uga"

...

# Update grub, reboot. Hope that everything worked.
$ sudo update-grub Extra credit
You can clean your MBR from the old GRUB2-pc installation (if you installed it to the hybryd-MBR, like a sane person) from OSX:
$ sudo fdisk -u /dev/disk(tab-to-complete, verify it is the right disk) (Read the man page for your fdisk to be sure -u still means fix MBR while keeping partitions )

Michal Čihař: Key signing done

As I've mentioned before, there was a keysigning party in Prague yesterday. Though we were afraid that not so many people will arrive, in the end I have signed 19 keys and assured 13 people for CAcert. The nice outcome of this is that I can now give up to 25 points when assuring for CAcert and altogether we are able to give you over 50 points in the Prague SUSE office. Unfortunately this also reminded me my problems with Unicode on CAcert and quick look at the bug tracker showed me that my bugs 769 and 770 are still open, valid and did not receive any attention :-(. PS: Of course after key signing, we had some fun with the openSUSE beer.

Filed under: English Suse 0 comments Flattr this!

Julien Danjou: Logitech Unifying devices support in UPower

A few months ago, I wrote about my reverse engineering attempt to Logitech Unifying devices. Back then, I concluded my post with big hopes on the future after receiving a document with some part of the specification of the HID++ 2.0 from Logitech. A couple of weeks ago, some of my summer work has been merged to UPower, adding battery support for some Logitech devices. HID++ UPower

As I discovered late in my first reverse engineering attempt, Logitech developed a custom HID protocol named HID++. This protocol exists in two versions, 1.0 and 2.0. Some devices talk with version 1 of the protocol (like my M705 mouse) and some others talk with version 2 of the protocol (like my K750 keyboard). Recently, I've been able to be in touch with a Logitech engineer who worked on the Linux support for the Unifying receiver, and he has been really helpful and exposed me some details about this protocol. Logitech made the decision to publish their HID++ specification publicly about a year ago, but still didn't do it. The internal review needed to publish such documents hasn't be done yet. The only published draft is just an extract of the specification, with even some typo in it as I discovered. Some other documents have been recently published, but I didn't have the time to review them. They contains HID++ 1.0 specifications and some details I asked for about the K750 keyboard. UPower support UPower

It took me sometime to get a full understanding of the protocol, its different version etc. After reverse engineering my K750 keyboard, I've also reverse engineered the data stream used to get my M705 mouse battery status. I've also received some information about the HID++ 1.0 protocol, so I've been able to discover a bit more on what the packets mean. Most of my discoveries are now used to do proper #define in up-lg-unifying.c so the code makes more sense. My first patch implements a new property for UPower devices, named luminosity, that use with K750 keyboard to report the light level received. The second patch add support for Logitech Unifying devices (over USB only) and should work with at least Logitech M705 and K750 devices. This should be available with the next version of UPower, which should be 0.9.19. gnome-power-statistics for K750

So far, Logitech has been kind enough to help me understanding part of the protocol and even sent me a few devices so I can play and test my work with them. Unfortunately, this will probably requires some work and time, and so far Logitech was not able to help with that. There should be enough information out there to at least add support for battery to HID++ 2.0 devices, and probably a few other things too. I hope I'd get the time do this at some point, but feel free to beat me in this race!

15 November 2012

Ian Campbell: Xen Microcode Loading in Wheezy

No sooner had Henrique de Moraes Holschuh announced the availability of improved microcode updating in Wheezy (sadly non-free) than Stephan Seitz pointed out that this support was broken for systems running Xen, which is unfortunate since as Henrique notes the impact of not applying such an update could be pretty much anything. There's a bit of a saga here: a patch has been available for several years which adds Xen support to the kernel's microcode driver in a nicely self contained way. However the Linux maintainers have decided that loading microcode from the kernel is too late and it should be done earlier (e.g. by the bootloader, or some shim between the bootloader and the kernel) in order to be effective and on that basis rejected the Xen patch implementing the existing kernel based scheme. In the meantime Xen has implemented early loading of microcode via an additional multiboot module passed by the bootloader and this was released in Xen 4.2 in September. (so far I've not seen any progress on making any change to the native side but that's by the by). However Wheezy is going to be shipping with Xen 4.1 which doesn't yet have this support in it. So we are basically faced with two choices, either we backport the support from Xen 4.2 or we apply the kernel patch. The downside of backporting rather than taking the kernel patch is that the backport is non-trivial (although not totally unwieldy) while the kernel patch is pretty much entirely self-contained. The other downside of early the early loading scheme in general (for either native or Xen) is the need to update tools (e.g. update-grub) as well as modifying the microcode packaging to put the files somewhere that the bootloader can see them (so /boot). This certainly can't happen for Wheezy at this stage which basically rules out the backport approach. So, the upshot is I've filed Debian bug #693053 and applied the kernel patch to our tree. It's all a bit unsatisfactory but at least microcode loading will work for users running Xen on Wheezy. With any luck for Jessie the native early microcode loading code will have landed as well and the pain of the infrastructure updates can be shared.

Dirk Eddelbuettel: Rcpp and the new R:: namespace for Rmath.h

We released Rcpp 0.10.0 earlier today. This post will just provide a simple example for one of the smaller new features -- the new namespace for functions from Rmath.h -- and illustrate one of the key features (Rcpp attributes) in passing. R, as a statistical language and environment, has very well written and tested statistical distribution functions providing probability density, cumulative distribution, quantiles and random number draws for dozens of common and not so common distribution functions. This code is used inside R, and available for use from standalone C or C++ programs via the standalone R math library which Debian / Ubuntu have as a package r-mathlib (and which can be built from R sources). User sometimes write code against this interface, and then want to combine the code with other code, possibly even with Rcpp. We allowed for this, but it required a bit of an ugly interface. R provides a C interface; these have no namespaces. Identifiers can clash, and to be safe one can enable a generic prefix Rf_. So functions which could clash such as length or error become Rf_length and Rf_error and are less likely to conflict with symbols from other libraries. Unfortunately, the side-effect is that calling, say, the probability distribution function for the Normal distribution becomes Rf_pnorm5() (with the 5 denoting the five parameters: quantile, mean, std.deviation, lowerTail, logValue). Not pretty, and not obvious. So one of the things we added was another layer of indirection by adding a namespace R with a bunch of inline'd wrapper functions (as well as several handful of unit tests to make sure we avoided typos and argument transposition and what not). The short example below shows this for a simple function taking a vector, and returning its pnorm computed three different ways:

#include <Rcpp.h>
// [[Rcpp::export]]
Rcpp::DataFrame mypnorm(Rcpp::NumericVector x)  
    int n = x.size();
    Rcpp::NumericVector y1(n), y2(n), y3(n);
    for (int i=0; i<n; i++)  
        // the way we used to do this
        y1[i] = ::Rf_pnorm5(x[i], 0.0, 1.0, 1, 0);
        // the way we can do it now
        y2[i] = R::pnorm(x[i], 0.0, 1.0, 1, 0);
     
    // or using Rcpp sugar in one go
    y3 = Rcpp::pnorm(x);
    return Rcpp::DataFrame::create(Rcpp::Named("Rold")  = y1,
                                   Rcpp::Named("Rnew")  = y2,
                                   Rcpp::Named("sugar") = y3);

This example also uses the new Rcpp attributes described briefly in the announcement blog post and of course in more detail in the corresponding vignette. Let us just state here that we simply provide a complete C++ function, using standard Rcpp types -- along with one 'attribute' declaration of an export via Rcpp. That's it -- even easier than using inline. Now in R we simply do

R> sourceCpp("mypnorm.cpp")

to obtain a callable R function with the C++ code just shown behind it. No Makefile, no command-line tool invocation -- nothing but a single call to sourceCpp() which takes care of things --- and brings us a compiled C++ function to R just given the source file with its attribute declaration. We can now use the new function to compute the probaility distribution both the old way, the new way with the 'cleaner' R::pnorm(), and of course the Rcpp sugar way in a single call. We build a data frame in C++, and assert that all three variants are the same:

R> x <- seq(0, 1, length=1e3)
R> res <- mypnorm(x)
R> head(res)
      Rold     Rnew    sugar
1 0.500000 0.500000 0.500000
2 0.500399 0.500399 0.500399
3 0.500799 0.500799 0.500799
4 0.501198 0.501198 0.501198
5 0.501597 0.501597 0.501597
6 0.501997 0.501997 0.501997
R> all.equal(res[,1], res[,2], res[,3])
[1] TRUE
R>

This example hopefully helped to illustrate how Rcpp 0.10.0 brings both something really powerful (Rcpp attributes -- more on this another time, hopefully) and convenient in the new namespace for statistical functions.

Axel Beckert: Tools to handle archives conveniently

TL;DR: There s a summary at the end of the article. Today I wanted to see why a dependency in a .deb-package from an external APT repository changed so that it became uninstallable. While dpkg-deb --info foobar.deb easily shows the control information, the changelog is in the filesystem part of the package. I could extract that one dpkg-deb, too, but I d have to extract either to some temporary directory or pipe it into tar which then can extract a single file from the archive and sent it to STDOUT:

dpkg-deb --fsys-tarfile foobar.deb   tar xOf - ./usr/share/doc/foobar/changelog.Debian.gz   zless

But that s tedious to type. The following command is clearly less to type and way easier to remember:

acat foobar.deb ./usr/share/doc/foobar/changelog.Debian.gz   zless

acat stands for archive cat is part of the atool suite of commands:

als

lists files in an archive.

$ als foobar.tgz
drwxr-xr-x abe/abe           0 2012-11-15 00:19 foobar/
-rw-r--r-- abe/abe          13 2012-11-15 00:20 foobar/bar
-rw-r--r-- abe/abe          13 2012-11-15 00:20 foobar/foo

acat

extracts files in an archive to standard out.

$ acat foobar.tgz foobar/foo foobar/bar
foobar/bar
bar contents
foobar/foo
foo contents

adiff

generates a diff between two archives using diff(1).

$ als quux.zip
Archive:  quux.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2012-11-15 00:23   quux/
       16  2012-11-15 00:22   quux/foo
       13  2012-11-15 00:20   quux/bar
---------                     -------
       29                     3 files
$ adiff foobar.tgz quux.zip
diff -ru Unpack-3594/foobar/foo Unpack-7862/quux/foo
--- Unpack-3594/foobar/foo      2012-11-15 00:20:46.000000000 +0100
+++ Unpack-7862/quux/foo        2012-11-15 00:22:56.000000000 +0100
@@ -1 +1 @@
-foo contents
+foobar contents

arepack

repacks archives to a different format. It does this by first extracting all files of the old archive into a temporary directory, then packing all files extracted to that directory to the new archive. Use the --each (-e) option in combination with --format (-F) to repack multiple archives using a single invocation of atool. Note that arepack will not remove the old archive.

$ arepack foobar.tgz foobar.txz
foobar.tgz: extracted to  Unpack-7121/foobar'
foobar.txz: grew 36 bytes

apack

creates archives (or compresses files). If no file arguments are specified, filenames to add are read from standard in.

aunpack

extracts files from an archive. Often one wants to extract all files in an archive to a single subdirectory. However, some archives contain multiple files in their root directories. The aunpack program overcomes this problem by first extracting files to a unique (temporary) directory, and then moving its contents back if possible. This also prevents local files from being overwritten by mistake.

(atool subcommand descriptions from the atool man page which is licensed under GPLv3+. Examples by me.) I though miss the existence of an agrep subcommand. Guess why? atool supports a wealth of archive types: tar (gzip-, bzip-, bzip2-, compress-/Z-, lzip-, lzop-, xz-, and 7zip-compressed), zip, jar/war, rar, lha/lzh, 7zip, alzip/alz, ace, ar, arj, arc, rpm, deb, cab, gzip, bzip, bzip2, compress/Z, lzip, lzop, xz, rzip, lrzip and cpio. (Not all subcommands support all archive types.) Similar Utilities There are some utilities which cover parts of what atool does, too: Tools from the mtools package Yes, they come from the handle MS-DOS floppy disks tool package, don t ask me why. :-)

uz: gunzips and extracts a gzip d tar d archives; Advantage over aunpack: Less to type. :-); Disadvantage compared to aunpack: Supports only one archive format.
lz: gunzips and shows a listing of a gzip d tar d archive; Advantage over als: One character less to type. :-); Disadvantage compared to als: Supports only one archive format.

unp unp extracts one or more files given as arguments on the command line.

$ unp -s
Known archive formats and tools:
7z:           p7zip or p7zip-full
ace:          unace
ar,deb:       binutils
arj:          arj
bz2:          bzip2
cab:          cabextract
chm:          libchm-bin or archmage
cpio,afio:    cpio or afio
dat:          tnef
dms:          xdms
exe:          maybe orange or unzip or unrar or unarj or lha 
gz:           gzip
hqx:          macutils
lha,lzh:      lha
lz:           lzip
lzma:         xz-utils or lzma
lzo:          lzop
lzx:          unlzx
mbox:         formail and mpack
pmd:          ppmd
rar:          rar or unrar or unrar-free
rpm:          rpm2cpio and cpio
sea,sea.bin:  macutils
shar:         sharutils
tar:          tar
tar.bz2,tbz2: tar with bzip2
tar.lzip:     tar with lzip
tar.lzop,tzo: tar with lzop
tar.xz,txz:   tar with xz-utils
tar.z:        tar with compress
tgz,tar.gz:   tar with gzip
uu:           sharutils
xz:           xz-utils
zip,cbz,cbr,jar,war,ear,xpi,adf: unzip
zoo:          zoo

So it s very similar to aunpack, just a shorter command and it supports some more exotic archive formats which atool doesn t support. Also part of the unp package is ucat which does more or less the same as acat, just with unp as backend. dtrx From the man page of dtrx:

In addition to providing one command to extract many different archive types, dtrx also aids the user by extracting contents consistently. By default, everything will be written to a dedicated directory that s named after the archive. dtrx will also change the permissions to ensure that the owner can read and write all those files. Supported archive formats: tar, zip (including self-extracting .exe files), cpio, rpm, deb, gem, 7z, cab, rar, and InstallShield. It can also decompress files compressed with gzip, bzip2, lzma, or compress.

dtrx -l lists the contents of an archive, i.e. works like als or lz. dtrx has two features not present in the other tools mentioned so far:

It can extract metadata instead of the normal contents from .deb and .gem files.
It can extract archives recursively, i.e. can extract archives inside of archives.

Unfortunately you can t mix those two features. But you can use the following tool for that purpose: deepfind deepfind is a command from the package strigi-utils and recursively lists files in archives, including archives in archives. I ve already written a detailed blog-posting about deepfind and its friend deepgrep. tardiff tardiff was written to check what changed in source code tarballs from one release to another. By default it just lists the differences in the file lists, not in the files contents and hence works different than adiff. Summary atool and friends are probably the first choice when it comes to DWIM archive handling, also because they have an easy to remember subcommand scheme. uz and lz and the shortest way to extract or list the contents of a .tar.gz file. But nothing more. And you have to install mtools even if you don t have a floppy drive. unp comes in handy for exotic archive formats atool doesn t support. And it s way easier to remember and type than aunpack. dtrx is neat if you want to extract archives in archives or if you want to extract metadata from some package files with just a few keystrokes. For listing all files in recursive archives, use deepfind.

13 November 2012

Steve McIntyre: Spoiling my vote

Ever since I passed the age of 18, I've used my legal right in the UK to vote in every poll I could: local council, British Parliament, European Parliament and even the messed-up referendum on the Alternative Vote that we had last year. In each of those cases, I made a point of considering my available options and I voted accordingly. I went along and marked my ballot paper in person where possible, or I filled in a postal vote for the last Westminster election in 2010 as I was on vacation in Florida when it was held. Unfortunately, it seems that my diligence in voting makes me unusual amongst the UK population these days. The long-term trend for voter turnout is downwards (as you can see). However, the latest election that's happening in the UK (well, England and Wales outside London) this week is for the newly-created posts of Police and Crime Commissioners. It's a textbook example of how not to organise an election, as pointed out eloquently by the Electoral Reform Society. This election has been incredibly badly managed and publicised by the Government, and lots of early polls are suggesting a tiny turnout of less than 20%. Despite the political crap about "democracy" being spouted by some Tory ministers, it's clear that very few people want this change in how the police are run, and it has simply been imposed from the top. Too few people care about the results of these elections for them to be valid - most people don't want the police politicised. After a lot of thought about the issue, I've decided how I'm going to vote this time. For the first time ever, I'm explicitly going to turn up and spoil my ballot paper as a protest - I do not want this crap. I urge other people in England and Wales to consider doing the same.

Next.

Previous.