The other day, I found myself confronted with a large number of large
files. Which were presented in (gzip-)compressed ascii format---which R reads
directly via
gzfile() connections---as well as (compressed)
numpy files.
The numpy can be read very efficiently into Python. We can do the same in R
via
save() and
load(), of course. But the trouble
is that you need to read them first. And reading hundreds of megabytes from ascii is
slow, no matter which language you use. Concerning R, I poked aound
scan(),
played with the
colClasses argument and looked at the recent
LaF package written just for
this purpose. And all these solutions were still orders of magnitude slower than
reading numpy. Which is no surprise as it is
really hard to beat binary
formats when you have to parse countless ascii tokens.
So the obvious next idea was to read the numpy file in Python, and to write a simple
binary format. One helpful feature with this data set was that it contained
only regular (rectangular) matrices of floats. So we could just store two integers for
the dimensions, followed by the total data in either one large binary blob,
or a sequence of column vectors.
But one minor trouble was that the Intertubes lead to no easy solution to
unpack the numpy format.
StackOverflow
had plenty of question around this topic converned with, say, how to
serialize in language-independent way. But no converters. And nobody
local knew how to undo the "pickle" format underlying numpy.
But a remote friend did:
Laurent,
well-known for his
Rpy2
package, pointed me towards using the
struct module and steered
me towards the solution shown below. So a shameless plug: if you need a very
experienced Python or R consultant for sciece work, consider
his consulting firm.
Finally, to round out this post, let's show the simple solution we crafted so that the
next guy searching the Intertubes will have an easier. Let us start with a
minimal Python program writing numpy data to disk:
#!/usr/bin/env python
#
# simple example for creating numpy data to demonstrate converter
import numpy as np
# simple float array
a = np.arange(15).reshape(3,5) * 1.1
outfile = "/tmp/data.npy"
np.save(outfile, a)
Next, the simple Python converter to create a binary file containing two
integers for row and column dimension, followed by row times columns of
floats:
#!/usr/bin/python
#
# read a numpy file, and write a simple binary file containing
# two integers 'n' and 'k' for rows and columns
# n times k floats with the actual matrix
# which can be read by any application or language that can read binary
import struct
import numpy as np
inputfile = "/tmp/data.npy"
outputfile = "/tmp/data.bin"
# load from the file
mat = np.load(inputfile)
# create a binary file
binfile = file(outputfile, 'wb')
# and write out two integers with the row and column dimension
header = struct.pack('2I', mat.shape[0], mat.shape[1])
binfile.write(header)
# then loop over columns and write each
for i in range(mat.shape[1]):
data = struct.pack('%id' % mat.shape[0], *mat[:,i])
binfile.write(data)
binfile.close()
Lastly, a quick
littler script
showing how R can read the data in a handful of lines:
#!/usr/bin/r
infile <- "/tmp/data.bin"
con <- file(infile, "rb")
dim <- readBin(con, "integer", 2)
Mat <- matrix( readBin(con, "numeric", prod(dim)), dim[1], dim[2])
close(con)
print(Mat)
That did the job---and I already used to converter to read a few weeks worth
of data for further analysis in R. This obviously isn't the last word on
possible solutions as the additional temporary file can be wasteful (unless
it forms a cache for data read multiple times). If someone has nice
solutions, please don't hold back and contact me. Thanks again to Laurent
for the winning suggestion concerning
struct, and help in
getting the examples shown here to work.
So I slept in the Sheraton Heathrow hotel last night. It's a
half-an-hour walk away from the airport, or a five minute bus drive.
If you go to any service desk and ask for the best way to get to the
hotel, they'll offer you a ticket for the 'HotelHoppa' service, which
will stop right in front of your hotel entrance, and costs 4 for a
one-way trip (even though the bus sides declare a 4.5 charge)
However, if you take the regular "London United" bus service (which
drives all over London, not just near the airport like the HotelHoppa
service) then the same trip is completely free of charge. You may have
to walk about a hundred yards more to reach your hotel, since the bus
will stop at a regular bus stop rather than at the hotel entrance, but
that's hardly a problem.
What a ripoff.
So, to clarify, don't take this bus:
Instead, take this bus:
This message brought by your friendly neighbourhood anti-ripoff
service.
While it would have been easy to attribute the performance gain in 2.5.10 on the new parallelization improvements, it is simply not the case. These improvements only apply to running collections when checking multiple packages. On my machine, the parallelization limit for a package is effectively determined by the dependencies between the collections on my machine.
Instead the improvements comes from reducing the number of system(3) (or fork+exec) calls Lintian does. Mostly through using xargs more, even if it meant slightly more complex code. But also, libperlio-gzip-perl shaved off a couple of seconds on binaries check.
But as I said, linux-image is not your average package . Most of the improvements mentioned here are hardly visible on other packages. So let s have a look at some more other bottlenecks. In my experience the following are the worst offenders :
Hope you used it well.
Already chilled by 
Do you ever have those afternoons you get a great idea and you've all the evening time for that task. The task is a relaxing one and won't need much attention, and you can watch a movie or something. But then, it happens that the evening turns into night as you realize a couple of little details adding complexity to the idea, and the task turns out to be much more invasive to your evening than you thought?
wheezy is frozen, the outside temperature in INN at midnight is 30 C,
& in 3 hours my trip to MGA starts. & here's the list of RC
bugs; this week all but one were sponsored uploads where the BTS already
contained a complete debdiff; thanks guys!
DebCamp starts tomorrow and the local team is relaxing and enjoying some free time with a cup of Debian Coffee. See you soon in Managua! (In case you wonder yes!!! both the Debian Coffee and the DebConf coffee mug will be for sale during DebConf12 in Managua)
Lately (as I earlier pointed out on my 
I got an awfull bug with npm after installing Node.js 0.8.0 and trying to do an 
Whilst nobody was looking, Mono became the most important framework in the world of games.
And I mean that with only a smidge more than my usual levels of hyperbole. It really is fantastically important, as far as game development goes.
I blogged a long time ago about its use inside EA s multi-million-selling The Sims 3. And the massive performance gains in Second Life when they switched from their in-house scripting language to Mono are well documented. But I m not talking about those, I m talking about the now, the recent stuff.
First up is Unity 3D. Not to be confused with the Ubuntu user interface, Unity 3D is a proprietary game development tool which began life for Mac, and has now managed to become 
The 
I've been using 
Starting next week, I'll start at new job. I will still stay at 