I hope I can get spots for 1-2 students to help implementing additional
state of the art methods, to allow for even more broad comparisons.
Acceptance notification will be tomorrow. I have no idea how high out chances
are of getting accepted. We're open source, as part of the university we have
a proven record of educating students (in fact, around two dozen have by now
contributed to ELKI, although not all of this has been "released" yet). We're
rather small compared to e.g. Gnome, Debian or Apache and just a few years
old. But I believe we are trying to fill an important gap in the research and
open source interaction: way too many algorithms get published in science, but
are never made available as source code to test and compare. This is where
we try to step in: make many largely overlooked methods available. Of course
we also have k-means (who hasn't?), but there is much more than just
k-means! And there are plenty of methods often cited in scientific literature,
yet nobody seems to have a working implementation of them ...
While most people equal data mining to prediction and classification (both
of which are actually more machine learning topics), ELKI is strong on
cluster analysis and outlier detection as well as index structures. Plus, it is
much more flexible than other frameworks. For example, we allow almost
arbitrary combinations of algorithms and distance functions. So with our tool,
you can easily test your own distance function by plugging it into various
algorithms. Otherwise we could just have used Weka.
The other reason why we did not extend Weka (or used R) is that we
actually wanted to not just implement some algorithms, but actually be able
to study the effects of index structures on the algorithms. And in
my opinion, this is actually the key difference between true data mining and
ML, AI or "statistics". In ML and statistics, the key objective is the
result quality. Which is often rather easy to measure, too. For "full" data
mining, one also needs to consider all the issues of actually managing
the data, indexing it to accelerate computations. Plus, the prime objective
is to discover something new, that you did not know before.
Of course these things cannot be completely separated. You can of course
discover patterns and rules in the data that will allow you to make good
predictions. But data mining is not just prediction and classification!
The next release of ELKI, 0.5, will be released in early april. ELKI 0.4
is already available in Debian testing and Ubuntu. The next release has a focus
on comparing clustering results, including an experimental visualization for
clustering differences, which will be presented at the ICDE 2012
conference.
On the big to-do list is a lot of stuff. In particular, ELKI is currently
mostly useful for research, as it is too difficult to use for the average
business guy wanting to do data mining. You can think of it more as a
prototyping system: try out what works for you, then implement that within your
larger project in an integrated and optimized way. The other big thing that
I'm unhappy with is the visualization speed. SVG is great for print export,
but Batik can be really slow, and the XML-DOM-API isn't exactly "accessible"
to students wanting to add new visualizations. Right now, it is all useful and
okay for my kind of experiments, but it could be so much more if we could
solve the speed (and memory) issues of the pure SVG based approach. I'd love
to see something much faster here, but with a SVG export for editing and
printing.
A new version of the digest package (which generates hash function summaries for
arbitrary (and possibly nested) R objects using any of the standard md5, sha-1, sha-256 or crc32 algorithms)
is now on CRAN.
Murray Stokely noticed a corner case where a file that was inaccessible lead
to a segmentation fault. Thanks to his patch, this is now caught and
flagged. He even added a test for it and fixed another one of my idioms. Nice
work, so keep the patches coming folks :)
CRANberries
provides the usual summary of changes to version
0.5.1.
As usual, our package is available via
the R-Forge page
leading to svn and tarball access, my digest page and the
local directory here.
I was playing with my new Laptop (a 
After a long silence, I updated the bpo Debian-installer for Debian GNU/Linux 6.0.4, Squeeze. You know, you can take it from 
Continuing with fast release pace for 
Thanks to Daniel Silverstone, pepperfish.net is now hosting
a new mailing list for Obnam. See the 
Some time ago I wrote about the 
One of the earliest features I wrote for the Debian bug tracking
system (Debbugs) after joining the team was support for forcibly
merging bugs. Originally, merging two bugs required that the bugs be
in exactly the same state before merging them; 
Over the past few weeks, I upgraded my Ubuntu hosts to the Precise release due to be published next month. While the upgrade on my desktops from Oneiric to Precise went mostly well, the upgrade of my terminal server from Lucid to Precise was more daunting. In short: too many packages leave an undue amount of cruft behind, namely tons of obsolete configuration files that are not correctly handled by maintainer scripts, to either delete obsolete files or to rename them to the currently valid filename. At Debian, these oversights are periodically monitored using piuparts and reported to the package maintainer, but Ubuntu doesn't seem to offer this sort of automated Quality Assurance process yet. Perhaps this would be something worth developing?
PS: For those who wonder how I spotted those obsolete configuration files, I simply enabled the 