So it happened: the Bug Squashing Party in Essen is done. And depending on your point of view at least I would call it a success.
We had in total fourteen participants. The number of release-critical bugreports touched and processed is not clear, but over 100 bugreports were looked at and either got comments added for clearification, maintainer pinged, unblock requests filed or already filed ones noted down, or NMUed. Given that we are in freeze since a while now and many easybugreports already squashed by those who aim for a daily fix, I consider this impressive. There is also the point that we managed to get some people involved that didn't consider themself
techyenough to be of help. On the contrary, they were a great help on checking these bugreports with analyzing the discussions in some lengthy bugreports and upstream bug trackers, or trying to reproduce some issues. I can just hope that the weekend left an impression deep enough to keep them in the boat. All in all, I'm quite pleased of the outcome, especially in the light of several other events catching the interest of potential participants which were going on at the same weekend. Thanks to all the people that stumbled by, and I am looking forward to maybe having another BSP at some point, but then I hope to not clash with so many other events. Thanks!
my short list of contributions to RC bug fixing during the last week:
---------------------------------------------------------------------------------
/ Pop-up advertisement. Breaking news. Help the Release Team [0] by reviewing \
pending unblock requests. Oh, and by fixing RC bugs too.
[0]: http://nthykier.wordpress.com/2012/12/09/wheezy-release-progress-december/
\---------------------------------------------------------------------------------/
Dear Project Members, here is another (delayed) monthly report
of DPL activities, this time for November 2012.
Highlights
master:/srv/leader/news/bits-from-the-DPL.txt.201211
$ dget -d http://.../pkg_testing-version_arch.dsc
$ dget -d http://.../pkg_sid-version_arch.dsc
$ debdiff pkg_testing-version_arch.dsc pkg_sid-version_arch.dsc
Following the Rcpp 0.10.0 release, I had written about
simulating pi easily by using the wonderful new
Rcpp Attributes feature.
Now with Rcpp 0.10.1 released a good week ago, it is time to look
at how Rcpp Attributes can help with external libraries. As this posts aims to show, it is a breeze!
One key aspect is the use of the plugins for the inline package. They provide
something akin to a callback mechanism so that compilation and linking steps can be informed about header and library locations and names.
We are going to illustrate this with an example from GNU Scientific Library (GSL). The example I picked
uses B-Spline estimation from the GSL. This is a little redundant as R has its own spline routines and package, but serves well as a simple
illustration---and, by reproducing an existing example, followed an established path. So we will look at
Section 39.7 of the GSL manual which has a
complete example as a standalone C program, generating both the data and the fit via cubic B-splines.
We can decompose this two parts: data generation, and fitting. We will provide one function each, and then use both from R. These two
function will follow the
aforementioned example from Section 39.7
somewhat closely.
We start with the first function to generate the data.
We include a few header files, define (in what is common for C programs) a few constants and then define a single function// [[Rcpp::depends(RcppGSL)]] #include <RcppGSL.h> #include <gsl/gsl_bspline.h> #include <gsl/gsl_multifit.h> #include <gsl/gsl_rng.h> #include <gsl/gsl_randist.h> #include <gsl/gsl_statistics.h> const int N = 200; // number of data points to fit const int NCOEFFS = 12; // number of fit coefficients */ const int NBREAK = (NCOEFFS - 2); // nbreak = ncoeffs + 2 - k = ncoeffs - 2 since k = 4 */ // [[Rcpp::export]] Rcpp::List genData() const size_t n = N; size_t i; double dy; gsl_rng *r; RcppGSL::vector<double> w(n), x(n), y(n); gsl_rng_env_setup(); r = gsl_rng_alloc(gsl_rng_default); //printf("#m=0,S=0\n"); /* this is the data to be fitted */ for (i = 0; i < n; ++i) double sigma; double xi = (15.0 / (N - 1)) * i; double yi = cos(xi) * exp(-0.1 * xi); sigma = 0.1 * yi; dy = gsl_ran_gaussian(r, sigma); yi += dy; gsl_vector_set(x, i, xi); gsl_vector_set(y, i, yi); gsl_vector_set(w, i, 1.0 / (sigma * sigma)); //printf("%f %f\n", xi, yi); Rcpp::DataFrame res = Rcpp::DataFrame::create(Rcpp::Named("x") = x, Rcpp::Named("y") = y, Rcpp::Named("w") = w); x.free(); y.free(); w.free(); gsl_rng_free(r); return(res);
genData() which returns and Rcpp::List as a list object to R. A primary importance here are the two attributes:
one to declare a dependence on the RcppGSL package, and one to declare the
export of the data generator function. That is all it takes! The plugin of the
RcppGSL will provide information about the headers and library, and Rcpp
Attributes will do the rest.
The core of the function is fairly self-explanatory, and closely follows the
original example. Space gets
allocated, the RNG is setup and a simple functional form generates some data plus noise (see below). In the original, the data is written to
the standard output; here we return it to R as three columns in a data.frame object familiar to R users. We then free the GSL
vectors; this manual step is needed as they are implemented as C vectors which do not have a destructor.
Next, we can turn the fitting function.
The second function closely follows the second part of the GSL example and, given the input data, fits the output data. Data structures are setup, the spline basis is created, data is fit and then the fit is evaluated at a number of points. These two vectors are returned along with two goodness of fit measures. We only need to load the Rcpp package and source a file containing the two snippets shown above, and we are ready to deploy this:// [[Rcpp::export]] Rcpp::List fitData(Rcpp::DataFrame ds) const size_t ncoeffs = NCOEFFS; const size_t nbreak = NBREAK; const size_t n = N; size_t i, j; Rcpp::DataFrame D(ds); // construct the data.frame object RcppGSL::vector<double> y = D["y"]; // access columns by name, RcppGSL::vector<double> x = D["x"]; // assigning to GSL vectors RcppGSL::vector<double> w = D["w"]; gsl_bspline_workspace *bw; gsl_vector *B; gsl_vector *c; gsl_matrix *X, *cov; gsl_multifit_linear_workspace *mw; double chisq, Rsq, dof, tss; bw = gsl_bspline_alloc(4, nbreak); // allocate a cubic bspline workspace (k = 4) B = gsl_vector_alloc(ncoeffs); X = gsl_matrix_alloc(n, ncoeffs); c = gsl_vector_alloc(ncoeffs); cov = gsl_matrix_alloc(ncoeffs, ncoeffs); mw = gsl_multifit_linear_alloc(n, ncoeffs); gsl_bspline_knots_uniform(0.0, 15.0, bw); // use uniform breakpoints on [0, 15] for (i = 0; i < n; ++i) // construct the fit matrix X double xi = gsl_vector_get(x, i); gsl_bspline_eval(xi, B, bw); // compute B_j(xi) for all j for (j = 0; j < ncoeffs; ++j) // fill in row i of X double Bj = gsl_vector_get(B, j); gsl_matrix_set(X, i, j, Bj); gsl_multifit_wlinear(X, w, y, c, cov, &chisq, mw); // do the fit dof = n - ncoeffs; tss = gsl_stats_wtss(w->data, 1, y->data, 1, y->size); Rsq = 1.0 - chisq / tss; Rcpp::NumericVector FX(151), FY(151); // output the smoothed curve double xi, yi, yerr; for (xi = 0.0, i=0; xi < 15.0; xi += 0.1, i++) gsl_bspline_eval(xi, B, bw); gsl_multifit_linear_est(B, c, cov, &yi, &yerr); FX[i] = xi; FY[i] = yi; Rcpp::List res = Rcpp::List::create(Rcpp::Named("X")=FX, Rcpp::Named("Y")=FY, Rcpp::Named("chisqdof")=Rcpp::wrap(chisq/dof), Rcpp::Named("rsq")=Rcpp::wrap(Rsq)); gsl_bspline_free(bw); gsl_vector_free(B); gsl_matrix_free(X); gsl_vector_free(c); gsl_matrix_free(cov); gsl_multifit_linear_free(mw); y.free(); x.free(); w.free(); return(res);
And with that, we generate a chart such aslibrary(Rcpp) sourceCpp("bSpline.cpp") # compile two functions dat <- genData() # generate the data fit <- fitData(dat) # fit the model, returns matrix and gof measures
via a simple four lines, or as much as it took to create the C++ functions, generate the data and fit it!
The RcppArmadillo and RcppEigen package support plugin use in the same way. Add an attribute to export a function, and an attribute for the depends -- and you're done. Extending R with (potentially much faster) C++ code has never been easier, and opens a whole new set of doors.op <- par(mar=c(3,3,1,1)) plot(dat[,"x"], dat[,"y"], pch=19, col="#00000044") lines(fit[[1]], fit[[2]], col="orange", lwd=2) par(op)
Russ makes a good point about a nuance of practical jokes that makes them special, the characteristic of involving a third party which is the target of the joke, and how that relates to consent. He is right, consent is an important fact, I acknowledge that. Russ also says at the end of his post that being an asshole doesn't make you responsible for somebody else's decisions, I ackowledge that, too. I even somewhat agree to the position that the employer might fire you on assholery reasons, but in this particular case, I think that's a hypocrite position. If assholery was an issue, it should have been before, too, and I don't think the employer thought of it before this sad even, as long as it meant more money for them. I think this only shows how cynical they are, the cost analysis for them showed them they now might lose more money by not firing, than by keeping them. It's just cynical.
| Series: | Major Ariane Kedros #1 |
| Publisher: | Roc |
| Copyright: | December 2008 |
| ISBN: | 0-451-46245-9 |
| Format: | Mass market |
| Pages: | 324 |
Conrad launched the 3.6 series of
Armadillo earlier today with a first 3.6.0 release.
So RcppArmadillo, our wrapper for R and Armadillo, is now on
CRAN with its corresponding version 0.3.6.0. No R level or interface changes
were needed, and the upstream changes are summarized below.
Changes in RcppArmadillo version 0.3.6.0 (2012-12-07)Courtesy of CRANberries, there is also a diffstat report for the most recent release As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.
- Upgraded to Armadillo release Version 3.6.0 (Piazza del Duomo)
- faster handling of compound expressions with submatrices and subcubes
- added support for loading matrices as text files with
NaNandInfelements- added
stable_sort_index(), which preserves the relative order of elements with equivalent values- added handling of sparse matrices by
mean(),var(),norm(),abs(),square(),sqrt()- added saving and loading of sparse matrices in arma_binary format
I want to start this post by saying two things:| Week | Squeeze | Wheezy | Diff |
|---|---|---|---|
| 43 | 284 (213+71) | 468 (332+136) | +184 (+119/+65) |
| 44 | 261 (201+60) | 408 (265+143) | +147 (+64/+83) |
| 45 | 261 (205+56) | 425 (291+134) | +164 (+86/+78) |
| 46 | 271 (200+71) | 401 (258+143) | +130 (+58/+72) |
| 47 | 283 (209+74) | 366 (221+145) | +83 (+12/+71) |
| 48 | 256 (177+79) | 378 (230+148) | +122 (+53/+69) |
| 49 | 256 (180+76) | 360 (216+155) | +104 (+36/+79) |
| 50 | 204 (148+56) | ||
| 51 | 124 (124+54) | ||
| 52 | 115 (78+37) | ||
| 1 | 93 (60+33) | ||
| 2 | 82 (46+36) | ||
| 3 | 25 (15+10) | ||
| 4 | 14 (8+6) | ||
| 5 | 2 (0+2) | ||
| 6 | release! |
Before we begin, everyone should read up on hashtables and what open
addressing / closed hashing is. The context is lines 111 190 of Python s
Objects/dictobject.c as of today (so we get the line numbers straight).
(I ve reworded this wlog entry a bit; I originally wrote it too late at
night for it to read coherent.) Basically, I ve got an application where
I d like to use a hashtable for a number of things not as generic as
Python, and with focus on small footprint. I d like to offer associative
arrays in a scripting language, where the keys are always arbitrary byte
strings excluding NUL. Also, I d like to use the hashtable as backend for
indexed arrays, where the keys are uint32_t and the usual use
case is sequential. Finally, I m using it for several internal tables,
such as a list of keywords, one of builtins, one of special variables,
etc. which is a reason for me to not use a self-balancing binary tree
as data structure (reading further below might suggest that, but getting
a sorted list of hashtable keys is not the focus, though not unimportant).
Why is the shift on perturb done after its first use? In my experiments (using 32-bit width everywhere), for the pathological case of an 8-element (i = 3) table with three entries 0, 0x40000000 and 0x800000000, the second round yields 1 for all three, so it cannot have to do with the upper bits. My lookup looks like:
mask = 2 - 1; j = perturb = hash(key); goto find_first_slot; find_next_slot: j = (j << 2) + j + perturb + 1; perturb >>= PERTURB_SHIFT; /* FALLTHROUGH */ find_first_slot: entry = table[j & mask]; if (!match(entry)) goto find_next_empty_slot;
This means that my first check is always the bare hash (so only do it if needed is no reason) and, since I m using gotos, I could just move the perturb >>= PERTURB_SHIFT; line before the line recalculating the next j to use. This seems to make more sense, even in the face of Python. (I actually looked at the Python file s comments again today because I thought to use a different resolution, but they have a good rationale for using the multiplication by 5.) Why can t we just use i as the PERTURB_SHIFT? Sure, this changes a shift-right by a constant, which can possibly be encoded as immediate value in assembly (unless you re on a pre-80186, which can only do SHR AX,1 and SHR AX,CL but not SHR AX,4, but that s outside of mksh s scope) into a right-shift by a variable, but i is already known, and I think the behaviour is better (it wouldn t eat any bits; assume the same 8-entry hashtable and pathologic keys 0, 8 and 16). Again: who do I think I am to go against the wisdom of the Python people, who seem to have shed more thought on this than everyone else I saw, asked, read about (including Spammipedia). That s why I m asking here. On that reference: I don t support spammers or people nagging for donations or premium accounts, like Xing and Groundspeak/Geocaching.COM, at all. In fact, I urge others to do the same, so it really hurts them; it may be their business model, but not if they spam me. Besides, OpenCaching.DE exists. Another thing is: to avoid CVE-2011-4815, I m randomising the hash used, with one seed value per hashtable, changed before a resize operation. I originally thought to seed it with nonzero, but then I have to rehash on hashtable resize, so I ll be XORing the final hash value instead (thanks ciruZ for the idea). I m thinking of omitting that for indexed arrays, as an attacker almost certainly cannot determine the keys there. (To directly use the indexed array keys, which are already uint32_t, as hashes makes using i from even more important.) The hash I m using is a modified Jenkins one-at-a-time called NZAAT: it s my new generic standard n n-cryptographic hash, and the changes are thus: while adding a byte, another increment of the hash is done (so NUL counts), and the finaliser got prefixed with the shift-left-add+shift-right-xor sequence of the adder (but not adding any value or the +1), to get best avalanche for all bytes. I actually compiled several versions of Hash.cs on a Windows VM at work to analyse the original one-at-a-time and all of my modifications; these turned out to be the simplest ones (I originally had added 0x100 instead of 1, but the effect was the same, and +1 is usually cheaper on most CPUs). Also, to avoid people being able to get to the seed, a user will always get only a sorted list of hashtable keys (numeric for indexed arrays, ASCIIbetically otherwise; see also my thoughts on JSON from the previous wlog entry). What algorithm do I use? For strings, comparisons are much more expensive, so I d like to keep them low. Memory use is also a factor; allocating one large(r) block is better than many small ones due to the pool allocator overhead and due to portability to ancient Unic s (which is another reason for me to use a hashtable which is a small struct plus an array of pointers, and then pass the list of keys as array of string pointers, instead of a tree). For both reasons, I m thinking a relatively simple MergeSort: I need to allocate the result array anyway, so I can just get two and free the one that isn t the end result, and it s AFAICT the cheapest on comparisons other than Tree Sort (which nobody really seems to use, and which would effect to using a balanced binary tree again). Since keys are unique, stability and duplicate handling is never an issue. I d like to use only one algorithm and one data structure, not a combination, as compactness is a design goal. Please drop your thoughts on Freenode, e.g. by /msg MemoServ send mirabilos your text here or per eMail to the domains debian, freewrt or mirbsd, which are organisations, with the localpart tg. Or just contact me as usual, if you re already acquainted. Or lookup 0xE99007E0. Thanks in advance! (Especially, Python Developers thoughts are welcome.)
For more than ten years now, I am a regular and happy user of GnuCash, the free accountancy program. Especially the integrated online banking support for HBCI that allows me not only to fetch my transactions, but also to transfer money using a cryptographic makes it a great tool. Nevertheless, it has shortcomings, especially when it comes to reports. And for some reason I never felt the urge to hack on that C/Guile code (although I recently hacked a bit on the UI).
A few days ago, I pushed out version 2.11.14 of the Geode X.org driver. This is the driver used by the OLPC XO-1 and by a plethora of low-power desktops, micro notebooks and thin clients.
So we finally made it, and sent out an
official announcement for
apt.postgresql.org.
This new repository hosts packages for all PostgreSQL server versions (at the
moment 8.3, 8.4, 9.0, 9.1, 9.2) for several Debian/Ubuntu distributions
(squeeze, wheezy, sid, precise) on two architectures (amd64, i386). Now add
packages for extension modules on top of all these, and you get a really large
amount of binaries from a small number of sources. Right now there's 1670 .deb
files and 148 .dsc files, but the .dsc count includes variants that only differ
in the version number per distribution (we attach .pgdg60+1 for squeeze
packages, .pgdg70+1 for wheezy and so on), so the real number of different
sources is rather something like 81, with 38 distinct source package names.
Dimitri Fontaine,
Magnus Hagander, and I have been working
on this since I first presented the idea at PGconf.EU 2011 in Amsterdam. We now
have a Jenkins server building all the packages, an archive server with the
master repository, and a feed that syncs the repository to the postgresql.org
FTP (well, mostly http) server.
If you were previously using pgapt.debian.net, that's the same archive as on
apt.postgresql.org (one rsync away). Please update your sources.list to point
to apt.postgresql.org, I'll shut down the archive at that location at the end
of January.
Here's the Quickstart
instructions from the Wiki page:
Import the repository key from http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc:
wget -O - http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc sudo apt-key add -Edit /etc/apt/sources.list.d/pgdg.list. The distributions are called codename-pgdg. In the example, replace squeeze with the actual distribution you are using:
deb http://apt.postgresql.org/pub/repos/apt/ squeeze-pgdg mainConfigure apt's package pinning to prefer the PGDG packages over the Debian ones in /etc/apt/preferences.d/pgdg.pref:
Package: * Pin: release o=apt.postgresql.org Pin-Priority: 500Note: this will replace all your Debian/Ubuntu packages with available packages from the PGDG repository. If you do not want this, skip this step. Update the package lists, and install the pgdg-keyring package to automatically get repository key updates:
apt-get update apt-get install pgdg-keyring
Google Latitude history dumps come in KML, and KML isn t super compatible with existing GIS tools. I wrote a quick snippit to output CSV from a KML dump on google latitude.
from lxml import etree
import sys
root = etree.fromstring(open(sys.argv[1], 'r').read())
track = root.xpath("//*[local-name()='Track']")[0]
coord = None
when = None
print "time,lat,lon"
for node in track.iter():
name = node.xpath("local-name()")
if name == 'coord':
lon, lat, z = node.xpath("text()")[0].split()
coord =
"lat": lat,
"lon": lon,
"zee": z
elif name == 'when':
when = node.xpath("text()")[0]
if coord and when:
print "%s,%s,%s" % (
when,
coord['lat'],
coord['lon']
)
coord, when = None, None
P.S. namespaces in lxml is nasty.
Next.