I've recently returned from
ICFP 2010 (
International
conference on functional programming). This was the second time I
went there, and like last year, it was a very interesting
conference. This year, I've even managed to do one keysigning with a
fellow Debian Developer (hello Sylvain!)
While there were many talks, I only mention a few key ones, that I
have found very cool.
Note: I'm a layman in this field. The text below is my own
interpretation of the talks and contains many errors and wrong
facts corrections are welcome!
Workshop on ML
I wasn't present at this workshop, but other people pointed me to this
very cool talk:
Mirage project
The abstract, to be found at
http://www.cs.rit.edu/~mtf/ml2010/program.html#Program (search for
Mirage), says:
Our framework, dubbed Mirage, extends the Objective Caml language with
I/O extensions and a custom runtime to emit binaries that execute as a
paravirtual operating system directly under Xen.
The custom run-time is significantly simpler than a general-purpose
operating system, with a static single-process 64-bit address space,
and "zero-copy" I/O that works directly with the OCaml garbage
collector. As a result, Mirage applications exhibit significant
performance speedups for I/O and memory handling versus the same code
running under Linux/Xen.
It's extremely interesting, IMHO. You write your code as a
more-or-less normal OCaml program, but you compile it with this
special tool-chain, and you boot it directly on ton of Xen, as a domU,
without the intermediate OS.
The project Git repo is at
http://github.com/avsm/mirage, and that
links a bit older presentation
(
http://anil.recoil.org/papers/2010-hotcloud-lamp.pdf) which IMHO is
a required read for anyone interested in cloud-related stuff.
ICFP proper
ICFP, the main conference, is as usual, full of math, designed to
scare people away. Err, I mean, designed to tell about new
developments in this field.
The papers are gathered in this reddit post:
http://www.reddit.com/r/haskell/comments/dji4v/papers_from_icfp_2010/.
XenServer
The paper
http://anil.recoil.org/papers/2010-icfp-xen.pdf describes
the XenServer project, and more specifically the management interface,
which is written entirely in OCaml, by a team of ~10 people.
They also have a bit of bin-packing code
(
http://xenbits.xen.org/xapi/xen-api.hg?file/95b9e4f1b9dd/ocaml/xapi/binpack.ml),
but much simpler than the
ganeti-htools project on which I'm working
(no balancing or similar). On the other hand, it is interesting to see
that they committed themselves to write the entire management suite in
OCaml, as opposed to our Python plus just a few bits of Haskell. I
envy them, very much so
The fun tidbit was how stable they found the OCaml tool-chain. In
three years, they had just one bug related to this (the tool-chain), a
segfault due to wrong formatting code in the handling path of a
seldom-triggered Python exception, which led to a
%s to be
translated as is to C code, which meant
printf got the wrong
specifiers. So in an OCaml project, a serious bug was due to lack of
type checking in string formatting in a Python component
Overall I have found interesting parallels between their experience
and my own with Haskell; from is this language/tool-chain stable
enough for use to concerns about using FP for a significant project
(but of course, orders of magnitude difference between their case and
mine).
Super-compilation
There were two papers on this topic;
one at ICFP
proper and
one
at the Haskell symposium.
If I understood things right, super-compilation is the opposite of lazy
evaluation. But let's start a bit back.
First, programs written in a functional languages contain many pure
function. Pure functions, being side-effect-free, exhibit many
interesting properties, and one of them is that they permit decoupling
of the computation of a value from the declaration of such computation.
One technique derived from this is lazy computations. Lazy computation
means that even though you write
y = sin(x) + 1, the actual call to
sin and the increment is not actually done until you
need the
actual value of
y (not just for storing it in
y, that is).
Super-compilation is the opposite of this. While laziness postpones
doing a computation to as late as possible, in the hope it won't be
needed at all, super-compilation runs the actual computation way ahead
of time, even before the program starts, that is at compile
time. Since programs use input data at runtime, which is not known at
compile time, the trick is to pre-compute as much as possible at this
stage, given incomplete information; and the papers showed that it's
possible to pre-compute much more than you'd usually believe.
Optimising compilers already do some form of CAF (constant applicative
form) optimisation, but the techniques presented in both papers are
going beyond simple CAFs. The details are in the papers, and are
complicated, but one simple example is converting this:
ones = map inc [1..]
(note that is an infinite list, and you don't want to fully compute
that at any time, neither at compile nor at runtime), into:
ones = [2..]
The details on how this is done depend on the exact super-compilation
implementation, and not all implementations can do the same
optimisations, but in the presentations at this ICFP the idea was to
split things into static/non-static parts, and build on that, or to do
partial evaluation, etc.
The result is that super-compilation shows significant increases in
speed (in some synthetic benchmarks up to 80%), with a few benchmarks
not being sped up at all, but most are. The downside is an explosion
in code size, as many more things are pre-computed. Still, as a
research topic, this is very interesting.
My own paper
Surprisingly, it seems I survived giving my talk (just an experience
report, not actually fancy stuff like the rest). Either the people
actually found it interesting (less likely), or they were very polite
(more probably). Anyway, the paper is on the
papers page
(shameless self-promotion) (if you care).
Haskell symposium
This is co-located with ICFP, and is more practical than ICFP (which
is, as I said, way too much math). The papers are again available on
reddit:
http://www.reddit.com/r/haskell/comments/dkzn4/papers_from_the_haskell_symposium_2010/.
Orc: Concurrent Orchestration in Haskell
How to easily write code for synchronising concurrent work? As an
example, how complicated is to solve this problem:
[from the Orc literature] we might wish to contact two airlines
simultaneously seeking price quotes. If either quote comes back
below a threshold price, say $300, then let's buy a ticket
immediately. On the other hand if both quotes exceed the
threshold, then buy the cheapest ticket. Additionally, buy a
ticket if the other airline does not give a timely quote, or
notify the user if neither airline provides a timely quote.
Now, in my experience, this would require manual management and
synchronisation of threads, and for example in Python the early abort
of querying the price from airline B if airline A returned a good
price would be non-trivial; in addition, adding two extra conditions
would be again not a simple composition of rules. In Orc however, it
takes all of 10 lines of code to do this, safely. I won't copy the
code here, see the
paper for
details, or the
API docs.
And the best part is, this is just a
cabal install orc away! I'll be
very interested to see how I can use this in practice, as it is seems
indeed a very powerful library.
Nikola: Embedding Compiled GPU Functions in Haskell
Interesting
project. Using
almost 100% Haskell code, and it marshalls the data to/from the GPU;
in the example given, the performance is as good as hand-written CUDA
code of course, this doesn't happen in all cases. It it's interesting
how they manage to compile the GPU-related code to actual object code
at Haskell compile time, which makes it transparent for the user.
Given the prevalence of high-speed GPUs, this could be a nice way to
utilise them without learning yet another language (or two, since
NVidia and ATI have different stacks).
They've also developed some interesting techniques (e.g. manual
control of sharing, to prevent inlining) as part of their project.
Scalable Event Handling for GHC
This is a
paper
co-authored by Johan Tibell of Google Switzerland and Bryan O'Sullivan
of Real World Haskell fame.
Very interesting serving 20K QPS from a Haskell process while having
16K+ idle clients connected:
[ ] the performance of the epoll back end does not begin to fall
off significantly until we have 50,000 idle connections open.
The talk started with the idea that we're on the path to serving 10
clients from a single machine, and how to improve Haskell's I/O
manager to achieve (almost) 10 concurrent clients. The slides (which
I don't have a link for) were a bit more cool on numbers than the
paper.
Oh, and this will come with GHC 7.0, for free
An LLVM Backend For GHC
The eternal fight to get better than the C backend in other words, to
not have to translate Haskell to C It is interesting to see how many
languages target LLVM nowadays, as opposed to plain old C via gcc. A
fun trivia item was that, according to the author, there are many
features in LLVM that are not actually used by 90% of the languages
targeting it (e.g. the garbage collector) (if I heard correctly in the
talk).
The results are interesting, although not too much overall speedup is
gained; this seems to be due to the hard-to-optimise Cmm intermediate
language inside GHC. In any case, a few benchmarks get more than 10%
speedup, even in the current form.
This will also come in GHC 7.0, although not enabled by default;
probably needs to mature some more
Conclusion
I've enjoyed this conference a lot, and I've learned a lot from
it. Given that many of the talks are about research topics, it's
harder to apply them in day-to-day work, but I think the pay-offs will
be long term rather than immediate.
Many people know Ultimate Debian Database already, and do use of it to create useful QA tools, or other Debian related activities. There is another database which is widely used by all of us, every day, and runs behind the scenes without being noticed. It is called projectb, the huge database of the Debian archive.
projectb is the place where dak stores all the information it needs to run the archive (including but not limited to source packages, related binaries, suite definitions and associations, overrides, and so on), and it tells dak to hit you with a REJECT if you try to upload a package twice, or if you do something The Wrong Way. OK OK, it eventually allows your packages to be accepted too, sometimes it also behaves good 
projectb is hosted on ftp-master.debian.org. Most fellow Debian Developers cannot login over there because access is restricted to FTP and Release people only, but can enjoy its clone on merkel.debian.org [1]. Please note that clone is synced on a regular basis, but it could be a little bit not up-to-date with the very latest changes. Just run the following:
This query displays which dpkg version known suites ship:
It s often said that KDE and GNOME are too bloated, too complex, too slow, or whatever. I won t deny that these critics are often justified, and some parts of the code are badly designed. But there can also be reasons behind this bloat: they are called features.
When you want to mount an encrypted USB disk, you can write your own script and even 
A while ago, I found the nice arcade game 
Today I decided to 
The Backports Team is pleased to announce the availability of a new suite on 
Debian s testing distribution is where Debian developers prepare the next
stable distribution. While this is still its main purpose, many users have
adopted this version of Debian because it offers them a good trade-off between
stability and freshness. But there are downsides to using this distribution
and the Constantly Usable Testing (CUT) project aims to resolve those.
This article will present the project and the challenges involved to make
it happen.
About Debian unstable & testing
Debian unstable is the distribution where developers upload new versions of
their packages. It happens frequently that some packages are not installable due
to changes in other packages or due to transitions not yet completed.
Debian testing, on the contrary, is managed by a tool that ensures the
consistency of the whole distribution: it picks updates from unstable only if
the package has been enough tested (10 days usually), if it s free of new
release-critical bugs, if it s available on all supported architectures, and if
it doesn t break any other package already present in testing. The Release
Team (RT) controls this tool and provide hints to help it find a set
of packages that can flow from unstable to testing.
Those rules also ensure that the packages that flow into testing are
reasonably free of show-stopper bugs (like a system that doesn t boot, or
X that doesn t work at all). This makes it very attractive to users
who like to regularly get new upstream versions of their software without
dealing with the biggest problems associated to them. This is all very
attractive, yet several Debian developers advise people to not use
testing. Why is that?
Known problems with testing
Disappearing software
The release team use this distribution to prepare the next
stable release and from time to time they remove packages from it. Either
because it s needed to ensure that other packages can migrate from
unstable to testing, or because they have long-standing release-critical
bugs without progress towards a resolution. It also happens that they
remove packages on request of the maintainers because they believe
that the current version of the software cannot be supported
(security-wise) for 2 years or more. The security team also regularly
issues such requests.
Long delays for security and important fixes
Despite the 10-day delay in unstable, there are always some annoying
bugs (and security bugs are no exceptions) that are only discovered when
the package already has migrated to testing. The maintainer might be quick to
upload a fixed package in unstable, and might even raise the urgency to
allow the package to migrate sooner, but if the packages gets entangled in a
large ongoing transition, it will not migrate before the transition is
completed. Sometimes it can take weeks for that to happen.
This delay can be avoided by doing direct uploads to testing (through
testing-proposed-updates) but this is almost never used, except
during a freeze, where targeted bugfixes are the norm.
Not always installable
With testing evolving daily, updates sometimes break
the last installation images available (in particular netboot images
that get everything from the network). The debian-installer (d-i) packages
are usually quickly fixed but they don t move to testing automatically
because the new combination of d-i packages has not necessarily been
validated yet. 
Following up on my
This chart--in levels rather than using logarithmic axes is done
When I took over 
Recently, Google started changing the web in some interesting ways. A lot by
contributing code. It started with projects such as the Chrome web browser,
which quickly achieved a market shares over 10%, which is more than Apple
Safari and Opera together.And the good side, Chrome did not actually add "yet another incompatible
platform", but by being closely related to Safari (or more precisely WebKit),
it is not that different.Other stuff that belongs into this category is the Google Web Toolkit and the
closure compiler (which essentially is a JavaScript code checker and optimizer).But recently, Google goes a step further. They started tackling internet
file formats. First they proposed a new Video codec known as WebM or VP8.
Now they extracted an image format out of this codec, known as WebP.There is some