CURP: Clave nica del Registro Poblacional, or Unique Population Registry Code. This is a (hopefully) unique, 18 character long, string identifying each Mexican - I won't get into the technicalities, but serve yourselves. By following such a convoluted scheme, the authorities should have ensured a biunivocal relation between each person and their CURP number. Well, at some point, due to a bureaucratical mishap completely outside my hands (my patron registered me with the wrong birth date, as you can see in the page at the bottom), I got two CURPs - And the procedure to fix it was far from trivial. Last Sunday, I entered the Consulta CURP system to print a copy of the official document. Much to my surprise... It answered that I was not registered! A couple of minutes later, I tried again, and succeeded. But I could not refrain from printing my Certificate of non-registration. I guess their system follows a strong-but-stupid scheme such as:
So, right, if a user submits a query during the system maintenance window (after all, it was Sunday after 23:00), the system will fail to connect to the database (or whatever), raise an exception, trap it, and... Well, you no longer exist, thanks for playing!

On year ago, we (the mancoosi team) published a comparison study regarding the state of the art of dependency solving in debian. As few noticed, the data presented had few glitches that I promised to fix. So we've repeated our tests using exactly the same data we used one year ago, but now using the latest available versions of all package managers as available in debian unstable. During the last year, three out of four solver that we evaluated release a major upgrade so I expected many improvements in performances and accuracy.

• apt-get 0.8.10 -> 0.9.7
• aptitude 0.6.3 -> 0.6.7
• smart 1.3-1 -> 1.4
• cupt 1.5.14.1 -> 2.5.6

Mpm, our test-bench for new technologies, changed quite a bit under the wood as a consequence of the evolution of apt-cudf and the recent work done in apt-get to integrate external cudf dependency solvers. Overall the results of our study are not changed. All solvers but mpm, that is based on aspcud, are not scalable as the number of packages (and alternatives) grows. It seems that Smart is the solver that does not give up, incurring in a timeout (fixed at 60 seconds) most of the time. Aptitude is the solver that tried to give you a solution, doesn't matter what and as result providing solutions that do not satisfy the user request for one reason or the other. Apt-get does surprisingly well, but it gives up pretty often showing the incomplete nature of it's internal solver. Cupt sometimes timeout, sometimes gives up, but when it is able to provide an answer it is usually optimal and it is very fast ... Mpm consistently finds an optimal solution, but sometimes it takes really a long time to do it. Since mpm is written in python and not optimized for speed this is not a big problem for us. The technology used by mpm is now integrated in apt-get and I hope this will alleviate this problem. All the details of our study can be found one the Mancoosi Website as usual with a lot of details. For example here you can find the results when mixing four major releases : sarge-etch-lenny-squeeze. Comments are more then welcome.

What, haven't you heard about the WikiLovesMonuments photo contest around cultural heritage? Copying from its web page,

Wiki Loves Monuments is an international photo contest around cultural heritage monuments in September. Starting from the Netherlands in 2010 and organized on a European level in 2011, we go global in 2012!

I heard about this initiative in Iv n Mart nez's Wikimedia talk at COSIT 2012, held last week in Coatzacoalcos, Veracruz (I intend to write a bit more regarding COSIT later on). I loved the idea, and intend to participate Not because I take great pictures (I don't, and I usually take them using my aging phone, which gives decent results but nothing beyond that), but because I love to move by bike in the city, and it's one of the best ways to roll in front of some of them. But more on me later Back to the topic! WikiLovesMonuments aims to improve on Wikimedia's (the organization behind Wikipedia and several other Free Culture reference projects) coverage of important landmarks all over the world. To do so, they are offering a trip to attend WikiMania 2013 in Hong Kong to the first place winner, and other "photography-related" prizes to the other winners. So, back to me: My motivation to enter the contest is to help Wikimedia. I know my shots won't be top-notch (although they will be the best I can do). I enjoy biking in my city, and often go not too far from many of the listed monuments. I am amazed at the number of monuments still pending in my area (of course, it's not by mistake this is called "La ciudad de los palacios", The city of the palaces) Surely some of the readers of this post will have (or will find easy to take) some photos to add. Of course, I'll try to focus on the missing monuments, but if you are a good photographer, you might want to submit a better version for a monument that's already there. So, some pointers, from what's closest to what's farthest from me:

• List of monuments in Mexico city
• Map with the list of monuments (Centered in Mexico City; other places in Mexico and other countries in the contest are a click away)
• Links to the monuments listing by state in Mexico
• GoogleMaps-based map with the monuments for Mexico City pin-pointed (similar maps are found at the other list of monuments pages)
• Android user? Android application pin-pointing you the nearest monuments

At least for Mexico, the listings are taken from the National Institute for Anthropology and History (INAH)'s Public registry of archaeological zones and monuments. So, I cannot wait to start my biking session today to get some good end-of-summer evening sun and get some pictures taken! :-D

It is a frequently asked Perl question how to write a regular expression that matches anything but something . The usual answer is that it can t be done or that you need to use the !~ operator instead of the =~ operator. But when you are using API s that passes regular expressions around as objects this might not be entirely helpful answers. By digging a bit around in the scarier parts of the perlre manual page I worked out this quite Perl specific solution: qr/(?:(?:something(*COMMIT)(*FAIL)) .)*/. I am not sure that it works in all possible corner cases and I certainly don t want to write the much needed comment that makes the code using the regexp maintainable. In the end it might be possible, but it is just one of those Now you have two problems scenarios.

Scott Chamberlain blogged about faster creation of binomial matrices the other day, and even referred to our RcppArmadillo package as a possible solution (though claiming he didn't get it to work, tst tst -- that is what the rcpp-devel list is here to help with). The post also fell short of a good aggregated timing comparison for which we love the rbenchmark package. So in order to rectify this, and to see what we can do here with Rcpp, a quick post revisiting the issue. As preliminaries, we need to load three packages: inline to create compiled code on the fly (which, I should mention, is also used together with Rcpp by the Stan / RStan MCMC sampler which is creating some buzz this week), the compiler package included with R to create byte-compiled code and lastly the aforementioned rbenchmark package to do the timings. We also set row and column dimension, and set them a little higher than the original example to actually have something measurable:

library(inline)
library(compiler)
library(rbenchmark)
n <- 500
k <- 100

The first suggestion was the one by Scott himself. We will wrap this one, and all the following ones, in a function so that all approaches are comparable as being in a function of two dimension arguments:

scott <- function(N, K)  
    mm <- matrix(0, N, K)
    apply(mm, c(1, 2), function(x) sample(c(0, 1), 1))
 
scottComp <- cmpfun(scott)

We also immediatly compute a byte-compiled version (just because we now can) to see if this helps at all with the code. As there are no (explicit !) loops, we do not expect a big pickup. Scott's function works, but sweeps the sample() function across all rows and columns which is probably going to be (relatively) expensive. Next is the first improvement suggested to Scott which came from Ted Hart.

ted <- function(N, K)  
    matrix(rbinom(N * K, 1, 0.5), ncol = K, nrow = N)
 

This is quite a bit smarter as it vectorises the approach, generating N times K elements at once which are then reshaped into a matrix. Another suggestion came from David Smith as well as Rafael Maia. We rewrite it slightly to make it a function with two arguments for the desired dimensions:

david <- function(m, n)  
    matrix(sample(0:1, m * n, replace = TRUE), m, n)
 

This is very clever as it uses sample() over zero and one rather than making (expensive) draws from random number generator. Next we have a version from Luis Apiolaza:

luis <- function(m, n)  
     round(matrix(runif(m * n), m, n))
 

It draws from a random uniform and rounds to zero and one, rather than deploying the binomial. Then we have the version using RcppArmadillo hinted at by Scott, but with actual arguments and a correction for row/column dimensions. Thanks to inline we can write the C++ code as an R character string; inline takes care of everything and we end up with C++-based solution directly callable from R:

arma <- cxxfunction(signature(ns="integer", ks="integer"), plugin = "RcppArmadillo", body='
   int n = Rcpp::as<int>(ns);
   int k = Rcpp::as<int>(ks);
   return wrap(arma::randu(n, k));
')

This works, and is pretty fast. The only problem is that it answers the wrong question as it returns U(0,1) draws and not binomials. We need to truncate or round. So a corrected version is

armaFloor <- cxxfunction(signature(ns="integer", ks="integer"), plugin = "RcppArmadillo", body='
   int n = Rcpp::as<int>(ns);
   int k = Rcpp::as<int>(ks);
   return wrap(arma::floor(arma::randu(n, k) + 0.5));
')

which uses the the old rounding approximation of adding 1/2 before truncating. With Armadillo in the picture, we do wonder how Rcpp sugar would do. Rcpp sugar, described in one of the eight vignettes of the Rcpp package, is using template meta-programming to provide R-like expressiveness (aka "syntactic sugar") at the C++ level. In particular, it gives access to R's RNG functions using the exact same RNGs as R making the results directly substitutable (whereas Armadillo uses its own RNG).

sugar <- cxxfunction(signature(ns="integer", ks="integer"), plugin = "Rcpp", body='
   int n = Rcpp::as<int>(ns);
   int k = Rcpp::as<int>(ks);
   Rcpp::RNGScope tmp;
   Rcpp::NumericVector draws = Rcpp::runif(n*k);
   return Rcpp::NumericMatrix(n, k, draws.begin());
')

Here Rcpp::RNGScope deals with setting/resetting the R RNG state. This draws a vector of N time K uniforms similar to Luis' function -- and just like Luis' R function does so without looping -- and then shapes a matrix of dimension N by K from it. And it does of course have the same problem as the RcppArmadillo approach earlier and we can use the same solution:

sugarFloor <- cxxfunction(signature(ns="integer", ks="integer"), plugin = "Rcpp", body='
   int n = Rcpp::as<int>(ns);
   int k = Rcpp::as<int>(ks);
   Rcpp::RNGScope tmp;
   Rcpp::NumericVector draws = Rcpp::floor(Rcpp::runif(n*k)+0.5);
   return Rcpp::NumericMatrix(n, k, draws.begin());
')

Now that we have all the pieces in place, we can compare:

res <- benchmark(scott(n, k), scottComp(n,k),
                 ted(n, k), david(n, k), luis(n, k),
                 arma(n, k), sugar(n,k),
                 armaFloor(n, k), sugarFloor(n, k),
                 order="relative", replications=100)
print(res[,1:4])

With all the above code example in a small R script we call via littler, we get

edd@max:~/svn/rcpp/pkg$ r /tmp/scott.r 
Loading required package: methods
              test replications elapsed   relative
7      sugar(n, k)          100   0.072   1.000000
9 sugarFloor(n, k)          100   0.088   1.222222
6       arma(n, k)          100   0.126   1.750000
4      david(n, k)          100   0.136   1.888889
8  armaFloor(n, k)          100   0.138   1.916667
3        ted(n, k)          100   0.384   5.333333
5       luis(n, k)          100   0.410   5.694444
1      scott(n, k)          100  33.045 458.958333
2  scottComp(n, k)          100  33.767 468.986111

We can see several takeaways:

• Rcpp sugar wins, which is something we have seen in previous posts on this blog. One hundred replication take only 72 milliseconds (or 88 in the corrected version) --- less than one millisecond per matrix creation.
• RcppArmadillo does well too, and I presume that the small difference is due not to code in Armadillo but the fact that we need one more 'mapping' of data types on the way back to R
• The sample() idea by David and Rafael is very, very fast too. This proves once again that well-written R code can be competitive. It also suggest how to make the C++ solution by foregoing (expensive) RNG draws in favour of sampling
• The approaches by Ted and Luis are also pretty good. In practice, the are probably good enough.
• Scott's function is not looking so hot (particularly as we increased the problem dimensions) and byte-compilation does not help at all.

Thanks to Scott and everybody for suggesting this interesting problem. Trying the rbinom() Rcpp sugar function, or implementing sample() at the C++ level is, as the saying goes, left as an exercise to the reader.

In late July, the Awesome Foundations invited me to participate in an interesting conversation about open brands at their conference. Awesome is a young collection of organizations struggling with the idea of if, and how, they want to try to control who gets call themselves Awesome. I was asked to talk about how the free software community approaches the issue. Guidance from free software is surprisingly unclear. I have watched and participated in struggles over issues of branding in every successful free software project I've worked in. Many years ago, Greg Pomerantz and I wrote a draft trademark policy for the Debian distribution over a couple beers. Over the last year, I've been working with Debian Project Leader Stefano Zacchiroli and lawyers at the Software Freedom Law Center to help draft a trademark policy for the Debian project. Through that process, I've come up with three principles which I think lead to more clear discussion about whether a free culture or free software should register a trademark and, if they do, how they should think about licensing it. I've listed those principles below in order of importance. 1. We want people to use our brands. Conversation about trademarks seem to turn into an exercise in imagining all the horrible ways in which a brand might be misused. This is silly and wrong. It is worth being extremely clear on this point: Our problem is not that people will misuse our brands. Our problem is that not enough people will use them at all. The most important goal of a trademark policy should be to make legitimate use possible and easy. We want people to make t-shirts with our logos. We want people to write books about our products. We want people to create user groups and hold conferences. We want people to use, talk about, and promote our projects both commercially and non-commercially. Trademarks will limit the diffusion of our brand and, in that way, will hurt our projects. Sometimes, after carefully considering these drawbacks, we think the trade-off is worth making. And sometimes it is. However, projects are generally overly risk averse and, as a result, almost always err on the side of too much control. I am confident that free software and free culture projects' desire to control their brands has done more damage than all brand misuse put together. 2. We want our projects to be able to evolve. The creation of a trademark puts legal power to control a brand in the hands of an individual, firm, or a non-profit. Although it might not seem like such a big deal, this power is, fundamentally, the ability to determine what a project is and is not. By doing this, it creates a single point of failure and a new position of authority and, in that process, limits projects' ability to shift and grow organically over time. I've heard that in US politics, there is no trademark for the terms Republican or Democrat and that you do not need permission to create an organization that claims to be part of either party. And that does not mean that everybody is confused. Through social and organizational structures, it is clear who is in, who is out, and who is on the fringes. More importantly, this structure allows for new branches and groups outside of the orthodoxy to grow and develop on the margins. Both parties have been around since the nineteenth century, have swapped places on the political spectrum on a large number of issues, and have played host to major internal ideological disagreements. Almost any organization should aspire to such longevity, internal debate, and flexibility. 3. We should not confuse our communities. Although they are often abused, trademarks are fundamentally pro-consumer. The point of legally protected brands is to help consumers from being confused as the source of a product or service. Users might love software from the Debian project, or might hate it, but it's nice for them to be able to know that they're getting "Debian Quality" when they download a distribution. Of course, legally protected trademarks aren't the only way to ensure this. Domains names, internal policies, and laws against fraud and misrepresentation all serve this purpose as well. The Open Source Initiative applied for a trademark on the term open source and had their application rejected. The lack of a registered trademark has not kept folks from policing use of the term. Folks try to call their stuff "open source" when it is not and are kept in line by a community of folks who know better. And since lawyers are rarely involved, it is hardly clear that a registered trademark would help in the vast majority of these these situations. It is also the case that most free software/culture organizations lack the money, lawyers, or time, to enforce trademarks in any case. Keeping your communities of users and developers clear on what is, and what isn't, your product and your project is deeply important. But how we choose to do this is something we should never take for granted.

not my Best RCBW Week Ever. anyway, here's what I've done:

• #669625 gummi: "gummi: Gummi should depend on enchant"
  add missing dependency, upload to DELAYED/2
• #684433 gdal-bin: "gdal-bin: FTBFS when ruby1.9.1 installed"
  add info to the bug report
• #684778 pythontracer: "pythontracer: missing dependency on python-gtk2"
  add missing dependency, upload to DELAYED/2
• #684808 src:mapserver: "mapserver: FTBFS: ld: cannot find -lstdc++"
  add missing build dependency (hint from Arto Jantunen), upload to DELAYED/2
• #684833 src:argyll: "argyll: FTBFS: ld: cannot find -lconv"
  verify bug unreproducibility
• #684837 src:gst-buzztard: "gst-buzztard: FTBFS: plugin.c:403:3: error: too many arguments to function 'bml_setup'"
  reproduce bug and add info to the BTS
• #684855 src:fusioninventory-agent: "fusioninventory-agent: FTBFS: tests failed"
  some unsuccessful debugging
• #684860 src:ruby-mocha: "ruby-mocha: FTBFS: tests failed"
  verify bug unreproducibility

I claim that the way we conduct discussions on debian-devel has gone quickly downhill this year. The discussions around the default init implementation are particularly sad examples of how Debian debates technical topics. I've whined about this before, and people have suggested various things. Here's a list of things that I think we can conclude do not work:

• "Do not feed the troll" does not work, because there is always someone who can't resist responding. Also, some of the bad behavior is not trolling. Further, having everyone ignore people who behave badly does not give negative feedback for bad behavior, and does not build shared social norms for what is acceptable behavior.
• Unorganized private feedback (people mailing in response to public list posts) sometimes works, but it's a bit hit-and-miss, and again does not help build shared social norms. In other words, new members of the community see that others behave badly, and then nothing happens as far as they can see, whereas in private there may be quite a lot of activity.
• Up/down-voting a la Reddit or Stack Overflow works on those sites, but it is not the only thing that keeps their sites in shape. It also works badly on mailing lists, I think, and we can't very well switch away from the lists for all sorts of reasons. Voting can also result in a stronger "echo chamber", where the popular people or opinions are upvoted, regardless of technical merit.

Things that can work:

• Real-life meetings between participants. Debconf, sprints, FOSDEM, other such conferences. Unfortunately, this is expensive, and we can't reach everyone.
• Active public, but constructive confrontation when people behave badly. This needs good social skills to be done well.

Things that might be worthwhile to try:

• Changes to how our mailing list software works. Not sure what they would be, but Clay Shirky's Group as User: Flaming and the Design of Social Software may be a good source of inspiration.
• If not technical changes to the list software, possibly ways in which we use the tools we have. For example, a friend of mine suggested a while ago that when there is a topic on which consensus is difficult to achieve, each side of the debate (there may be any number) writes up their position and arguments for it and against the others on a wiki page. This might avoid going in circles in the discussions, repeating the same arguments over and over, during weeks and months of discussion.
• Public, but anonymous, feedback tools, a la Reddit and Stack Overflow. Perhaps as simple as adding "+1" and "-1" buttons to the web archive pages for messages, along the "Spam" button, and then publishing aggregate statistics? (A voting system, if you wish.)
• A consensus on the rough bounds of acceptable behavior and some group of people who are delegated the authority to enforce that.

The current situation, where a few hard-nosed, uncompromising people can prevent any rational discussion about init systems cannot be allowed to continue.

ELKI is a data mining software project that I have been working on for the last years as part of my PhD research. It is open source (AGPL-3 licensed) and avilable as both a Debian package and Ubuntu package in the official repositories. So a simple aptitude install elki should get you going and give you a menu entry for ELKI. These packages come with the scripts elki to launch the MiniGUI and elki-cli to run from command line. The key feature that sets ELKI apart from existing open source tools used in data mining (e.g. Weka and R) is that it has support for index structures to speed up algorithms, and a very modular architecture that allows various combinations of data types, distance functions, index structures and algorithms. When looking for performance regressions and optimization potential in ELKI, I recently ran some benchmarks on a data set with 110250 images described by 8 dimensional color histograms. This is a decently sized dataset: it takes long enough (usually in the range of 1-10 minutes) to measure true hotspots. When including Weka and R in the comarison I was quite surprised: our k-means implementation runs at the same speed as Rs implementation in C (and around twice that of the more flexible "flexclus" version). For some of the key agorithms (DBSCAN, OPTICS, LOF) we are an order of magnitude faster than Weka and R, and adding index support speeds up the computation by another factor of 5-10x. In the most extreme case - DBSCAN in Weka vs. DBSCAN with R-tree in ELKI - the speedup was a factor of 330x, or 2 minutes (ELKI) as opposed to 11 hours (Weka).
The reason why I was suprised is that I expected ELKI to perform much worse. It is written in Java (as opposed to R's kmeans, which is in C), uses a very flexible architecture which for example does not assume distances to be of type double and just has a lot of glue code inbetween. However, obviously, the Java Hotspot compiler actually lives up to its expectations and manages to inline the whole distance computations into k-means, and then compiles it at a level comparable to C. R executes vectorized operations quite fast, but on non-native code as in the LOF example it can become quite slow, too. (I would not take Weka as reference, in particular with DBSCAN and OPTICS there seems to be something seriously broken. Judging from a quick look at it, the OPTICS implementation actually is not even complete, and both implementations actually copy all data out of Weka into a custom linear database, process it there, then feed back the result into Weka. They should just drop that "extension" altogether. The much newer and Weka-like LOF module is much more comparable.) Note that we also have a different focus than Weka. Weka is really popular for machine learning, in particular for classification. In ELKI, we do not have a single classification algorithm because there is Weka for that. Instead, ELKI focuses on cluster analysis and outlier detection. And ELKI has a lot of algorithms in this domain, I dare to say the largest collection. In particular, they are all in the same framework, so they can be easily compared. R does of course have an impressive collection in CRAN, but in the end they do not really fit together. Anyway, ELKI is a cool research project. It keeps on growing, we have a number of students writing extensions as part of their thesis. It has been extremely helpful for me in my own research, as I could quickly prototype some algorithms, then try different combinations and use my existing evaluation and benchmarking. You need some time to get started (largely because of the modular architecture, Java generics and such hurdles), but then it is a very powerful research tool. But there are just many more algorithms, published sometime, somewhere, but barely with source code available. We'd love to get all these published algorithms into ELKI, so researchers can try them out. And enhance them. And use them for their actual data. So far, ELKI was mostly used for algorithmic research, but it's starting to move out into the "real" world. More and more people that are not computer scientists start using ELKI to analyze their data. Because it has algorithms that no other tools have. I tried to get ELKI into the "Google Summer of Code", but it was not accepted. But I'd really like to see it gain more traction outside the university world. There are a number of cool projects associated with ELKI that I will not be able to do myself the next years, unfortunately.

• A web browser frontend would be cool. Maybe even connected to Google Refine, using Refine for preprocessing the data, then migrating it into ELKI for analysis. The current visualization engine of ELKI is using SVG - this should be fairly easy to port into the web browser. Likely, the web browers will even be faster than the current Apache Batik renderer.
• Visual programming frontend. Weka, RapidMiner, Orange: they all have visual programming style UIs. This seems to work quite well to model the data flow within the analysis. I'd love to see this for ELKI, too.
• Cluster/Cloud backend. ELKI can already handle fairly large data sets on a big enough system. If someone spends extra effort on the index structures, the data won't even need to fit into main memory anymore. Yet, everybody now wants "big data", and parallel computation probably is the future. I'm currently working on some first Hadoop YARN based experiments with ELKI. But this is a huge field, turning this into true "elk yarn". I will likely only lay some foundations (unless I get funding to continue as a PostDoc on this project. I sure hope to get to do at least a few years of postdoc somewhere, as I really enjoy working with students on this kind of project)
• New visualization engine. The current visualization engine, based on Apache Batik and SVG is quite okay. It does what I need, which is to get a quick glance at the results and the ability to export them for publications in print quality. (in particular, I can easily edit the SVG files with Inkscape) But it is not really something fancy (although we have a lot of cool visualizations). And it is slow. I havn't found a portable and fast graphics toolkit for Java yet that can produce SVG files. There is a lot of hype around processing, for example, but it seems to be too much about art for me. In fact, I'd love to use either something like Clutter or Cairo. But getting them to work for Windows and Mac OSX will likely be a pain.
• Human Computer Interaction (HCI). This is in my opinion the biggest challenge we are facing with all the "big data" stuff. If you really go into big data (and not just run Hadoop and Mahout on a single system; yes - a lot of people seem to do this), you will at some point need to go beyond just crunching the numbers. So far, the challenges that we are tackling are largely data summarization and selection. TeraSort is a cool project, and a challenge. Yet, what do you actually get from sorting this large amount of data? What do you get from running k-means on a terabyte? When doing data mining on a small data set, you quickly learn that the main challenge actually is preprocessing the data and choosing parameters the right way so that your result is not bogus. Unless you are doing simple prediction tasks, you often don't have a clearly defined objective. Sure, when predicting churn rates, you can hope to just throw all the data into a big cloud and hope you get some enlightement out. But when you are doing cluster analysis or outlier detection - unsupervised methods - the actual objective by definition cannot be hardcoded into a computer. The key objective then is learn something new on the data set. But if you want to have your user learn something on the data set, you will have to have the user guide the whole process, and you will have to present results to the user. Which gets immensely more difficult with larger data. Big data just does no longer look like this. And neither are the algorithms as simple as k-means or hierarchical clustering. Hierarchical clustering is good for teaching the basic ideas of clustering. But you will not be using a dendrogram for a huge data set. Plus, it has a naive complexity of O(n^3) and for some special cases O(n^2) - too slow for truly big data.
  For the "big data future" once we get over all the excitement of being able to just somehow crunch these numbers we will need to seriously look into what to do with the results (in particular, how to present them to the user), and how to make the algorithms accessible and usable for non-techies. Right now, you cannot expect a social sciences researcher to be able to use a Hadoop cluster. Yet to make sense of the results. But if you are a smart guy to actually solve this, and open up "big data processing" to the average non-IT user, this will be big.
• Oh, and of course there are just hundreds of algorithms not yet available (accessible) as open source. Not in ELKI, and usually not anywhere else either. Just to name a few from my wishlist (I could probably implement many of them in a few hours in ELKI, but I don't have the time to do so myself, plus they are good student or starter project to get used to ELKI): BIRCH, CLARA, CLARANS, CLINK, COBWEB, CURE, DOC, DUSC, EDSC, INSCY, MAFIA, P3C, SCHISM, STATPC, SURFING, ... just to name a few.

If you are a researcher in cluster analysis or outlier detection, consider contributing your algorithms to ELKI. Spend some time optimizing them, adding some documentation. Because, if ELKI keeps on growing and gaining popularity, it will be the future benchmark platform. And this can give you citations, which are somewhat the currency of science these days. Algorithms available in the major toolkits just do get cited more, because people compare to them. See this list for an overview of work cited by ELKI - scientific work that we reimplemented at least to some extend for ELKI. It is one of the services that we provide with ELKI for researchers: not only the algorithm, but also the appropriate citation.

There are so many sites trying to vie for Google results for any and every town in existance in the United States that they are crowding out useful information. They get their list of towns from census data or similar sources, and generate pages for every single entry they find, no matter how small. During the weekend, I was poking around, looking for information on a not-quite-ghost town called Lonerock, in Oregon:
http://en.wikipedia.org/wiki/Lonerock,_Oregon
I love to visit out of the way places like that when I'm back in Oregon, which we hope to do next summer. So I was curious about it - it appears quite isolated, but seems like it might be worth a look. According to the Wikipedia article it is a very small town, with a population of 24 people. And yet, if you look at the Google results for it, you find a few good links at first, and then:

• Any number of results with "city information", which is just census data with a bit of fiddling.
• Current local time. Gee, that's great to know.
• Various 'homes for sale' sites: none listed.
• Horses for sale "in Lonerock", but it turns out they're all in Heppner (30 miles away) or farther.
• Climate statistics.
• Bicycles for sale in Lonerock. Surprisingly, none of those, either.
• Attorneys at law in Lonerock. I think they ran them all out of town: there are none.
• Truck scales and weigh stations in Lonerock. You have to drive at least 50 miles to get to the nearest one.
• Hotels in Lonerock: nope, none of those either.
• Various Cable and Internet offerings (I think in reality they use this: http://en.wikipedia.org/wiki/IP_over_Avian_Carriers )
• Doctors in Lonerock: Neurologists, Gastroenterologists and even Nephrologists. Turns out there aren't any in town.
• Car Rentals in Lonerock. Just in case you fly in to any of the Airports in Lonerock, and want to leave rather than staying in the Hotels in Lonerock.
• Apartments for rent in Lonerock. A town of 24 people in the middle of the wide open west is not the kind of place you're likely to live in an apartment...
• Relocation guides for Lonerock.

And so on and so forth. Scattered in the middle, you can even find a few articles and photo pages by people who actually had something to say or show about the place, but finding them amidst all the crud is not an easy task. To me this seems surprising - it doesn't seem like it'd be that hard to make sites that just happen to have entries for every town in the entire nation rank a bit lower than things written by human beings. Google - have a look at Lonerock, and see if you can use it as a way to seperate the wheat from the chaff!

One major change in GNOME 3.6 is Pango s shaper engines replacement with HarfBuzz. Only language engines (for word break analysis, for example) are retained. So, I m checking how this affects Thai/Lao rendering and what to do next. Over all, Behdad has put a good effort to make it right. Most Uniscribe behaviors have been achieved for compatibility. He even cares enough to cover some widespread Thai fonts in which the language tag 'latn' is used instead of 'thai', as seen in Mozilla #719366. Unfortunately, this font set has been declared as standard fonts in official documents. The workaround seems inevitable. Supported Fonts In my experiments with some existing Thai OpenType fonts, the new Pango still renders well without regression. Loma font from fonts-tlwg (glyph positioning with GPOS, rearrangement with GSUB): Arundina Sans font from Fonts-SIPA-Arundina (positioning by substitution, only GSUB, no GPOS): But for legacy fonts without OpenType features, it renders badly: In addition, according to Behdad, PUA glyphs in legacy fonts are not supported yet. This means there will be regression on fonts designed for Windows XP or below. But modern fonts designed for Windows 7 should be fine. Changes on Bugs The engine replacement from scratch certainly affects existing bugs. Some become obsolete, some still remain. Here are the summary for Thai/Lao engine, as resolved upstream: Closed bugs:

• GNOME #616495 (Debian #620001) regarding Lao MAI KONG rendering, which was caused by a flaw in my code. I have proposed a patch for a while, but no action is taken yet. And patched debs have been locally distributed as a workaround. However, with HarfBuzz replacement, the bug has now gone.
• GNOME #378001 regarding minority languages supports. I hadn t worked on this because I was waiting for WTT 3.0, a local standard, to be drafted. Anyway, with HarfBuzz replacement, the old WTT 2.0 clustering has been dumped and replaced with Unicode guidelines. Therefore, I assume it should be now possible to render minority languages with Thai script, provided that the font has the required OpenType features.
• GNOME #393307, #677090 regarding wrong rendering of zero-width marks like ZWJ, ZWSP. This bug has also been dumped with the HarfBuzz replacement.

Questionable bugs:

• GNOME #583718 (Debian #620002) regarding the rendition of Thai SARA AM (U+0E33) on VTE with excessive dotted circle. So far, I have disagreed with Behdad whether this bug should be treated along with Indic scripts. IMO, there is an easy path for Thai by rendering monospace fonts differently, which is also in accordance with widespread practice everywhere else, albeit XTerm, Emacs, or framebuffer TTYs. But Behdad doesn t like the idea and insists that it should be treated along with Indic scripts, which would complicate things a lot. So, the bug has been there for many years. Meanwhile, I have also provided a workaround in the aforementioned patched debs. BTW, the situation has been changed a little bit after the HarfBuzz replacement. Firstly, let s see the problem with current Pango: One can easily spot the dotted circle glitches. And here is how I workaround it, which is like how it's rendered on other terminals: With the HarfBuzz replacement, here is how it renders: That is, although it s still wrong, it s more tolerable. So, the question for users is: Could they tolerate this until VTE is redesigned for Indic scripts supports?

Remaining bugs:

• GNOME #576156 (Debian #620004) regarding weird cursor movements caused by Unicode UAX #29. Many amendment efforts have been pushed to Unicode from different sources, until it was finally accepted in Unicode 6.1.0. However, no action has been taken in Pango yet. We still have to push it further. Again, a fix has also been provided in the patched debs.

FreedomBox 0.1 Ian Sullivan posted a release announcement today regarding the first FreedomBox public release, which we're calling 0.1. This release is, frankly, long overdue. Getting here has been harder and taken much longer than any of us intended. Fortunately, now that we've reached this milestone, I believe that making regular progress towards the vision we have for a "1.0" release will get easier. FreedomBox is far from the only thing that I'll be working on after my upcoming early retirement from HP, but hopefully I will succeed in putting far more time towards the project soon than I have been able to in the past.

From Ken Starks, of Helios Project / Reglue

From the desk of just-so-you-know.....

Thomas Knight and I have agreed that the fund drive to aid in paying for necessary surgery for me should be halted. I will do a full blog about this entire effort this Monday but suffice it to say, you have donated more than enough to take care of my immediate needs and those down the road. Asking for anything more would be taking advantage of a loving and generous community. While it is far from adequate, the only thing I can offer you is my eternal thanks. It just seems so....small of a thing to give in return.

They will not stop the Indiegogo campaign until it reaches its time limit unfortunately so that will remain in effect for the next 30-some hours. Please do not donate anything further as I have more than enough to cover my expenses. I will explain more about that on my blog Monday.

If you helped spread the word about this campaign, please pass this posting along with the same intensity. I would deeply appreciate it.

See also now: update from Ken:

http://linuxlock.blogspot.co.uk/2012/08/of-roller-coasters-cancer-and-linux.html
==

Donations for Ken himself may be over: the work of Helios/Reglue still remains good, important and useful - the world benefits for investment in education. I would urge that this is still a worthwhile charity to contribute to as you get Linux into the hands of young people and the disadvantaged who can then USE it.

For anybody that doesn't know - Ken Starks is the man behind the Helios Project - recycling computers, putting Linux on them, donating them to youngsters and the needy in Austin Texas.

He has a cancer diagnosis but his disability pension is enough to defeat his state provided insurance. Now he himself needs the help of the community.

More details here: http://linuxlock.blogspot.co.uk/2012/08/something-ive-been-putting-off-all-day.html and his partner's post above.

and here: Thomas A. Knight, who is setting up donations.

http://thomasaknight.com/blog.php?id=71

UPDATE:

There is now also a donate button at the Blog of Helios (linuxlock.blogspot.com).

Enough has been raised so far to buy Ken an appropriate monitor - because of the effects of radiotherapy, he's been unable to tolerate using a monitor with backlighting.

One donor has also stepped in to buy a month's drug treatment.

The initial target -US $5000 - has been met but The cost of an operating theatre session is at least US $50,000. All donations continue to be gratefully received.

Should efforts for raising funds for Ken himself be unsuccessful, all surplus funds raised will be ploughed back into the Helios Project (now known as Reglue following a recent merger with another local computer charity).

The Helios Project has been associated with Software in the Public Interest [SPI - also the umbrella organisation supporting Debian] for some years.

UPDATE2 - From Thomas A. Knight's blog referenced above:

By Thomas A. Knight on Sun 19 Aug 2012 07:33:49 am $11,000 so far, and still going strong. The outpouring of support for this is amazing. My most humble thanks to all contributors.

Even though we are still far from our goal, this has been far more successful than I ever thought possible. Thank you.

UPDATE3 - From Ken Starks, Sunday 26 August 2012

The amount of money needed has been raised. Donations to Helios Project/Reglue are welcome, as ever - but Ken's immediate needs are satisfactorily met and he himself doesn't wish to profit further from the community's goodwill. As he says, he can't thank us enough
More news at http://linuxlock.blogspot.com is due on 27/08/2012

good news: I'm seeing more & more people contributing to RC bugs in the BTS. here are my own contributions for the past week:

• #656552 zope2.12-sandbox: "zope2.12-sandbox: fails to install, remove and install again"
  add info to bug report
• #670405 ekiga: "ekiga: Ekiga crashes on startup"
  add info to bug report
• #673370 centerim: "centerim: Crash at startup (suspect login msn)"
  add info to bug report, then close on submitter's behalf
• #683061 ntp: "ntp: missing init script dependency on $named"
  sponsor NMU from Ivo De Decker (fix init script), upload to DELAYED/2, then cancelled by maintainer after severity change
• #684019 python-liblcms: "python-liblcms: static library in /usr/share"
  re-enable dropping of .a and .la files from python package, upload to DELAYED/2
• #684024 src:python-adodb: "python-adodb: broken binary-indep target"
  apply patches from Evgeni Golov (d/rules targets) and Jakub Wilk (python-support handling), upload to DELAYED/2
• #684078 wv2: "calligra: Buffer overflow"
  apply modified patch from KDE, upload to DELAYED/2
• #684150 python-wxgtk2.8: "python-wxgtk2.8 breaks upgrade from squeeze to wheezy"
  add info to bug report
• #684180 angband: "angband: removes files that were installed by another package: /usr/share/angband/*"
  don't remove directory belonging to other package in postrm, upload to DELAYED/2
• #684488 puppet-lint: "puppet-lint not working with ruby1.9.1"
  send a patch to the BTS

[ once again, I am translating somebody else's material In this case, my good Costa Rican friend Carolina Flores. Please excuse my stylistic mistakes My English is far from native as you well know. But this material is worth sharing, and worth investing some tens of minutes doing a quick translation. If you can read Spanish, go read Caro's original entry ] Have you been to a music record store lately? I did so last Saturday, as a mere excercise. I was not planning on buying anything but I wanted to monitor things and confirm my suspicions. What was I suspicious of? First, that I would only find old records. And so it was: The only recent record I found was ...little broken hearts by Norah Jones. the second, that I would only find music for over 50 year old people. so it was: Were I there to look for a present for my father, I would have walked out with 10 good records. Third, that in the store nothing worth commenting would happen. About that last point, I should point out it was around 10 AM and the store had just opened its doors. Lets concede the benefit of doubt. I don't think many of you will remember, but in Barrio La California (where there is now a beauty parlour, almost in front of AM.PM) there was Auco Disco. In Auco Disco there was a guy specialized in rock (Mauricio Alice) and another one specialized in jazz (I don't remember his name). In that record you could always find rare records, but if they were not there, at least you were sure to find somebody to say: "No, we don't have that, but that's an excellent record, it's the best that [insert group here] have ever recorded because just afterwards they switched their guitar player, they had gone a bit south but with that record they are flying. But no, we don't have it; I can recommend you this record by [insert another group] because it has a guitar solo in track six that is amazing". It would happen more or less like that, which means, one would arrive to Auco Disco at 10 AM and leave around 5 PM with three new records, after having listened to a spectacular music selection. What happened to those stores? Were they killed by The Pirate Bay? That's the simplistic answer from the recording industry! The answer is that those stores never got anything from the industry but an invoice. The industry specially in prescindible markets such as ours was limited to hiring artists, taking care of them recording a sellable product, producing the object called record, and that's it. The more commercial radio stations were paid to program those songs as it cannot be casual that "Mosa, mosa" is the summer hit in all of Latin America, can it? but, record stores? Nothing. Lets carry on with that idea: Radio stations are paid to program said music. This idea should not lead us to believe that recording companies are to blame for bad taste. I won't reveal my sources, but I know the success of the "Locura autom tica" song by La Secta was a real example. Nobody paid for it. That song got to the number one because of its own merits(?) (you don't know the effort it took to find that thing, I cannot recommend it to you). Same thing happens with other stations that don't program reggaet n, that try to save the species, and where they play what we do like. But the thing is, everything we like is not available in any record store in this country. Then, even if we wanted to buy a record or give it as a gift to somebody, it is plain impossible. And don't tell me it's the same to present as a gift a link or a CD full of downloaded MP3 as it is to give a record with cover and booklet, wrapped in gift paper. I might be old-school, but the fetish object record still exists, not only because of its cover, but because of its sound. A 3MB MP3 is akin to drinking coffee dripping from a bag that has been used eight times with the same coffee beans. That format is the worst that has ever happened to music, and if we had any bit dignity we would never purchase digital files in Amazon or iTunes safe for MP3 with an acceptable compression level. That, if we could buy them, because not only that is allowed to us. As the musical industry has no interest in resolving ITS problem (that is not our problem, it is those companies') it has not even been resolved how to charge for a MP3 download that includes import fees (well, if downloading from here a MP3 from a USA-based file server can be considered importing goods into Costa Rica!!!) so we don't have to get dizzy entering into the nineties to Titi Online to discover there is nothing by Muse, Andrew Bird, The Killers, Death Cab for Cutie, Paramore, Bj rk... (believe me, I looked them all up, even Norah jones and La Secta. They also were not there). This all leads me to the question, which I present with all due respect (NOT): What the fuck do they expect us to do??? It is outrageous; above all because in the best case they will sell us a watery coffee download that won't allow us to get all the details a vinyl or less compression would give us. In the worst case, post-MP3 groups will end up recording music with no harmonics or hidden sounds, because, what for? Nobody will hear it. They even admit it: "Some musicians and audio engineers say the MP3 format is changing the way studios mix their recordings. They say the MP3 format "flattens" dynamics differences in tone and volume in a song. As a result, a great deal of new music sounds very much alike, and there is nothing as focusing to create a dynamic listening experience. Why working so hard in creating complex sound if nobody can detect it?" (Rolling Stone, The Death of High fidelity, December 26 2007, taken from here). That's why I am not surprised by Adri n's post regarding the sales of old records. The price has nothing to do with it. The causes are related to the fetish object record and what it means or does not mean for people that have never purchased one. Adri n also asks if somebody here keeps buying records. I answered that I would if the stores sold anything I like. I do it even after the nausea I feel while reading "This phonogram is an intellectual work protected in favor of its producer COPYING IT IN WHOLE OR PARTIALLY IS FORBIDDEN " (like that, uppercased, yelling to whoever is only guilty of having bought a record and defending the producer, not the artist). But I am sure that almost nobody buys records because doing so is no longer a gratifying experience; because if buying a record is clicking to wait 15 days for it to reach the mailbox, we prefer clicking on the download link. But there is another reason for people not to buy records anymore. In one of my talks regarding the dictatorship of the all rights reserved, I asked the 30 twenty-something-year-old students if any of them had ever bought a record. One answered he had, because he is an author and performer (cantautor in Spanish) and understands the effort that producing a record entails. The rest of them had never done so. Is it possible that said young people have never listened to real music? Is it possible that, were it not for concerts, what they consider music is a set of washed-out MP3 that are about to fill up 1TB of their computer? Does people no longer buy records because they cannot differentiate one sound from the other? It is not very clear for me where do I want to get to. The recording industry is despicable. An industry that instead of innovating sets its energy on suing adolescents for downloading songs, trying to pass laws restricting our freedoms in Internet, putting up DRMs making us hostages to our devices* and forcing us to listen just the aroma of music, deserves my whole contempt. If we add ot this that said industry won't allow us to legally download their breadcrumbs because it has not understood that Internet does not need a van crossing borders, besides my contempt they deserve my pity and my heartfelt condolence. But the condolence is also for music, real music, that which is not compressed under the shoe using a terrible format. It is also for independent musicians that have not realized that begging for a bit of space to that industry they just add to themselves the "despicable" tag, given they deserve the fruits of their work to enter their bank account. However, there are good things that have come out of this absurdity. Be it for those that have joined projects such as Aut mata (even if it is in MP3) and for dreams come true such as Musopen (that have achieved that the music that's in theory Public Domain becomes so in practice as well). Good for the Electronic Frontier Foundation and the list of lawyers willing to defend people accused of ilegally downloading music in the USA. Good for the Creative Commons licenses that allow free sharing. All those are growing solutions, although none of them allows me to buy the Panamanian Carlos M ndez's record. Thankfully, a friend of mine who knows I will never give a dime to Apple, bought the files for me in iTunes. I thank him deeply, although I would have prefered to go to Auco Disco and have Mauricio tell me that the 2007 EP I have from Carlos is better than the record he did on 2009. * My devices don't have DRM because I use free software. I also use the ogg file format. Image by verbeeldingskr8

Attachment	Size
Full video (Ogg Vorbis+Theora encoded)	81.19 MB

it's sunday evening again, & again, here comes my report about activities around RC bugs:

• #602668 libdevel-bt-perl: "libdevel-bt-perl: FTBFS on armel"
  debugging (pkg-perl)
• #636805 viewvc: "viewvc runs extremely slowly (~15s per page)"
  add backported upstream patch, upload to DELAYED/2
• #646540 phpgacl: "phpgacl: uninstallable in sid, needs to transition to smarty3"
  fix smarty path variable, upload to DELAYED/2
• #680790 libio-async-perl: "libio-async-loop-epoll-perl: FTBFS: tests failed"
  add backported upstream patch (pkg-perl)
• #680806 src:libio-async-loop-glib-perl: "libio-async-loop-glib-perl: FTBFS: tests failed"
  add patch provided by upstream (pkg-perl)
• #681903 melange-client: "melange-client: missing dependency on python-pkg-resources"
  add missing dependency and a patch provided by Jakub Wilk, upload to DELAYED/2
• #682517 ocfs2console: "ocfs2console: DeprecationWarning at start prevent usage"
  review patch in BTS, then upload fix prepared by S bastien Villemot to DELAYED/2, then fixed in quick maintainer upload
• #682631 python-distutils-extra: "linaro-image-tools: FTBFS: AttributeError: 'tuple' object has no attribute 'major'"
  send debdiff to BTS, then fixed in a maintainer upload
• #682636 src:racket: "racket: FTBFS: raco setup: error: during making for scribblings/tools"
  bug manipulation, bug closed by maintainer afterwards
• #682825 phpgacl: "phpgacl: creates world writable directory /usr/share/phpgacl/admin/templates_c"
  fix permissions in postinst, upload to DELAYED/2
• #683284 graphicsmagick: "CVE-2012-3438"
  apply upstream patch, upload to DELAYED/2
• #683925 src:ocp: "FTBFS with gcc-4.7 on i386"
  add information to bug report
• #683927 libcloud: "CVE-2012-3446: MITM vulnerability in TLS/SSL certificates verification"
  add backported patch from upstream, upload to DELAYED/2

In my spare time, I currently work on a Norwegian docbook version of the 2004 book Free Culture by Lawrence Lessig, to get a Norwegian text explaining the problems with the copyright law I can give to my parents and others that are reluctant to read an English book. It is a marvellous set of examples on how the ever expanding copyright regulations hurt culture and society. When the translation is done, I hope to find funding to print and ship a copy to all the members of the Norwegian parliament, before they sit down to debate the latest revisions to the Norwegian copyright law. This summer I called for volunteers to help me, and I have been able to secure the valuable contribution from at least one other Norwegian. Two days ago, we finally broke the 50% mark. Then more than 50% of the number of strings to translate (normally paragraphs, but also titles and index entries are also counted). All parts from the beginning up to and including chapter four is translated. So is chapters six, seven and the conclusion. I created a graph to show the progress: The number of strings to translate increase as I insert the index entries into the docbook. They were missing with the docbook version I initially started with. There are still quite a few index entries missing, but everyone starting with A, B, O, Z and Y are done. I currently focus on completing the index entries, to get a complete english version of the docbook source. There is still need for translators and people with docbook knowledge, to be able to get a good looking book (I still struggle with dblatex, xmlto and docbook-xsl) as well as to do the draft translation and proof reading. And I would like the figures to be redrawn as SVGs to make it easy to translate them. Any SVG master around? I am sure there are some legal terms that are unfamiliar to me. If you want to help, please get in touch, and check out the project files currently available from github. If you are curious what the translated book currently look like, the updated PDF and EPUB are published on github. The HTML version is published as well, but github hand it out with MIME type text/plain, confusing browsers, so I saw no point in linking to that version.