I've recently returned from ICFP 2010 (International conference on functional programming). This was the second time I went there, and like last year, it was a very interesting conference. This year, I've even managed to do one keysigning with a fellow Debian Developer (hello Sylvain!) While there were many talks, I only mention a few key ones, that I have found very cool. Note: I'm a layman in this field. The text below is my own interpretation of the talks and contains many errors and wrong facts corrections are welcome! Workshop on ML I wasn't present at this workshop, but other people pointed me to this very cool talk: Mirage project The abstract, to be found at http://www.cs.rit.edu/~mtf/ml2010/program.html#Program (search for Mirage), says:

Our framework, dubbed Mirage, extends the Objective Caml language with I/O extensions and a custom runtime to emit binaries that execute as a paravirtual operating system directly under Xen. The custom run-time is significantly simpler than a general-purpose operating system, with a static single-process 64-bit address space, and "zero-copy" I/O that works directly with the OCaml garbage collector. As a result, Mirage applications exhibit significant performance speedups for I/O and memory handling versus the same code running under Linux/Xen.

It's extremely interesting, IMHO. You write your code as a more-or-less normal OCaml program, but you compile it with this special tool-chain, and you boot it directly on ton of Xen, as a domU, without the intermediate OS. The project Git repo is at http://github.com/avsm/mirage, and that links a bit older presentation (http://anil.recoil.org/papers/2010-hotcloud-lamp.pdf) which IMHO is a required read for anyone interested in cloud-related stuff. ICFP proper ICFP, the main conference, is as usual, full of math, designed to scare people away. Err, I mean, designed to tell about new developments in this field. The papers are gathered in this reddit post: http://www.reddit.com/r/haskell/comments/dji4v/papers_from_icfp_2010/. XenServer The paper http://anil.recoil.org/papers/2010-icfp-xen.pdf describes the XenServer project, and more specifically the management interface, which is written entirely in OCaml, by a team of ~10 people. They also have a bit of bin-packing code (http://xenbits.xen.org/xapi/xen-api.hg?file/95b9e4f1b9dd/ocaml/xapi/binpack.ml), but much simpler than the ganeti-htools project on which I'm working (no balancing or similar). On the other hand, it is interesting to see that they committed themselves to write the entire management suite in OCaml, as opposed to our Python plus just a few bits of Haskell. I envy them, very much so The fun tidbit was how stable they found the OCaml tool-chain. In three years, they had just one bug related to this (the tool-chain), a segfault due to wrong formatting code in the handling path of a seldom-triggered Python exception, which led to a %s to be translated as is to C code, which meant printf got the wrong specifiers. So in an OCaml project, a serious bug was due to lack of type checking in string formatting in a Python component Overall I have found interesting parallels between their experience and my own with Haskell; from is this language/tool-chain stable enough for use to concerns about using FP for a significant project (but of course, orders of magnitude difference between their case and mine). Super-compilation There were two papers on this topic; one at ICFP proper and one at the Haskell symposium. If I understood things right, super-compilation is the opposite of lazy evaluation. But let's start a bit back. First, programs written in a functional languages contain many pure function. Pure functions, being side-effect-free, exhibit many interesting properties, and one of them is that they permit decoupling of the computation of a value from the declaration of such computation. One technique derived from this is lazy computations. Lazy computation means that even though you write y = sin(x) + 1, the actual call to sin and the increment is not actually done until you need the actual value of y (not just for storing it in y, that is). Super-compilation is the opposite of this. While laziness postpones doing a computation to as late as possible, in the hope it won't be needed at all, super-compilation runs the actual computation way ahead of time, even before the program starts, that is at compile time. Since programs use input data at runtime, which is not known at compile time, the trick is to pre-compute as much as possible at this stage, given incomplete information; and the papers showed that it's possible to pre-compute much more than you'd usually believe. Optimising compilers already do some form of CAF (constant applicative form) optimisation, but the techniques presented in both papers are going beyond simple CAFs. The details are in the papers, and are complicated, but one simple example is converting this:

ones = map inc [1..]

(note that is an infinite list, and you don't want to fully compute that at any time, neither at compile nor at runtime), into:

ones = [2..]

The details on how this is done depend on the exact super-compilation implementation, and not all implementations can do the same optimisations, but in the presentations at this ICFP the idea was to split things into static/non-static parts, and build on that, or to do partial evaluation, etc. The result is that super-compilation shows significant increases in speed (in some synthetic benchmarks up to 80%), with a few benchmarks not being sped up at all, but most are. The downside is an explosion in code size, as many more things are pre-computed. Still, as a research topic, this is very interesting. My own paper Surprisingly, it seems I survived giving my talk (just an experience report, not actually fancy stuff like the rest). Either the people actually found it interesting (less likely), or they were very polite (more probably). Anyway, the paper is on the papers page (shameless self-promotion) (if you care). Haskell symposium This is co-located with ICFP, and is more practical than ICFP (which is, as I said, way too much math). The papers are again available on reddit: http://www.reddit.com/r/haskell/comments/dkzn4/papers_from_the_haskell_symposium_2010/. Orc: Concurrent Orchestration in Haskell How to easily write code for synchronising concurrent work? As an example, how complicated is to solve this problem:

[from the Orc literature] we might wish to contact two airlines simultaneously seeking price quotes. If either quote comes back below a threshold price, say $300, then let's buy a ticket immediately. On the other hand if both quotes exceed the threshold, then buy the cheapest ticket. Additionally, buy a ticket if the other airline does not give a timely quote, or notify the user if neither airline provides a timely quote.

Now, in my experience, this would require manual management and synchronisation of threads, and for example in Python the early abort of querying the price from airline B if airline A returned a good price would be non-trivial; in addition, adding two extra conditions would be again not a simple composition of rules. In Orc however, it takes all of 10 lines of code to do this, safely. I won't copy the code here, see the paper for details, or the API docs. And the best part is, this is just a cabal install orc away! I'll be very interested to see how I can use this in practice, as it is seems indeed a very powerful library. Nikola: Embedding Compiled GPU Functions in Haskell Interesting project. Using almost 100% Haskell code, and it marshalls the data to/from the GPU; in the example given, the performance is as good as hand-written CUDA code of course, this doesn't happen in all cases. It it's interesting how they manage to compile the GPU-related code to actual object code at Haskell compile time, which makes it transparent for the user. Given the prevalence of high-speed GPUs, this could be a nice way to utilise them without learning yet another language (or two, since NVidia and ATI have different stacks). They've also developed some interesting techniques (e.g. manual control of sharing, to prevent inlining) as part of their project. Scalable Event Handling for GHC This is a paper co-authored by Johan Tibell of Google Switzerland and Bryan O'Sullivan of Real World Haskell fame. Very interesting serving 20K QPS from a Haskell process while having 16K+ idle clients connected:

[ ] the performance of the epoll back end does not begin to fall off significantly until we have 50,000 idle connections open.

The talk started with the idea that we're on the path to serving 10 clients from a single machine, and how to improve Haskell's I/O manager to achieve (almost) 10 concurrent clients. The slides (which I don't have a link for) were a bit more cool on numbers than the paper. Oh, and this will come with GHC 7.0, for free An LLVM Backend For GHC The eternal fight to get better than the C backend in other words, to not have to translate Haskell to C It is interesting to see how many languages target LLVM nowadays, as opposed to plain old C via gcc. A fun trivia item was that, according to the author, there are many features in LLVM that are not actually used by 90% of the languages targeting it (e.g. the garbage collector) (if I heard correctly in the talk). The results are interesting, although not too much overall speedup is gained; this seems to be due to the hard-to-optimise Cmm intermediate language inside GHC. In any case, a few benchmarks get more than 10% speedup, even in the current form. This will also come in GHC 7.0, although not enabled by default; probably needs to mature some more Conclusion I've enjoyed this conference a lot, and I've learned a lot from it. Given that many of the talks are about research topics, it's harder to apply them in day-to-day work, but I think the pay-offs will be long term rather than immediate.

I have developed an affection for crime drama - on Sunday nights, the TV channel Five USA shows something like four hours of CSI, back to back. There's obviously something fascinating about the problem-solving process... and there's always the too-good-to-be-true computer interfaces to laugh at. (I also watch The Mentalist, NCIS and Law & Order, for variety.) And also, on CSI:NY and Law & Order, I get to spot the places in New York I visited at DebConf. So yes, problem solving. Sometimes the real thing can be even more compelling. And who wants to slump in front of the TV for four hours, anyway? Much better to sit up in front of the computer for... erm... a lot longer.

• Bug #525549 - gnucash sometimes removes the account data file

This was a more challenging bug - I had to break out gdb and everything. Turns out string manipulation with C is still error-prone. First reported upstream in October 2008 - quite a mystery. NMU to follow later in the week, since there are two other serious bugs open for gnucash.

When I took over man-db in 2001, one of the major problems that became evident after maintaining it for a while was the way it handled subprocesses. The nature of man and friends means that it spends a lot of time calling sequences of programs such as zsoelim < input-file tbl nroff -mandoc -Tutf8. Back then, it was using C library facilities such as system and popen for all this, and I had to deal with several bugs where those functions were being called with untrusted input as arguments without properly escaping metacharacters. Of course it was possible to chase around every such call inserting appropriate escaping functions, but this was always bound to be error-prone and one of the tasks that rapidly became important to me was arranging to start subprocesses in a way that was fundamentally immune to this kind of bug. In higher-level languages, there are usually standard constructs which are safer than just passing a command line to the shell. For example, in Perl you can use system([$command, $arg1, $arg2, ...]) to invoke a program with arguments without the interference of the shell, and perlipc(1) describes various facilities for connecting them together. In Python, the subprocess module allows you to create pipelines easily and safely (as long as you remember the SIGPIPE gotcha). C has the fork and execve primitives, but assembling these to construct full-blown pipelines correctly is difficult and error-prone, so many programmers don't bother and use the simple but unsafe library facilities instead. I wrote a couple of thousand lines of library code in man-db to address this problem, loosely and now quite distantly based on code in groff. In the following examples, function names starting with command_, pipeline_, or decompress_ are real functions in the library, while any other function names are pseudocode. Constructing the simplified example pipeline from my first paragraph using this library looks like this:

pipeline *p;
int status;
p = pipeline_new ();
p->want_infile = "input-file";
pipeline_command_args (p, "zsoelim", NULL);
pipeline_command_args (p, "tbl", NULL);
pipeline_command_args (p, "nroff", "-mandoc", "-Tutf8", NULL);
pipeline_start (p);
status = pipeline_wait (p);
pipeline_free (p);

You might want to construct a command more dynamically:

command *manconv = command_new_args ("manconv", "-f", from_code,
                                     "-t", "UTF-8", NULL);
if (quiet)
	command_arg (manconv, "-q");
pipeline_command (p, manconv);

Perhaps you want an environment variable set only while running a certain command:

command *less = command_new ("less");
command_setenv (less, "LESSCHARSET", lesscharset);

You might find yourself needing to pass the output of one pipeline to several other pipelines, in a "tee" arrangement:

pipeline *source, *sink1, *sink2;
source = make_source ();
sink1 = make_sink1 ();
sink2 = make_sink2 ();
pipeline_connect (source, sink1, sink2, NULL);
/* Pump data among these pipelines until there's nothing left. */
pipeline_pump (source, sink1, sink2, NULL);
pipeline_free (sink2);
pipeline_free (sink1);
pipeline_free (source);

Maybe one of your commands is actually an in-process function, rather than an external program:

command *inproc = command_new_function ("in-process", &func, NULL, NULL);
pipeline_command (p, inproc);

Sometimes your program needs to consume the output of a pipeline, rather than sending it all to some other subprocess:

pipeline *p = make_pipeline ();
const char *line;
line = pipeline_peekline (p);
if (!strstr (line, "coding: UTF-8"))
	printf ("Unicode text follows:\n");
while (line = pipeline_readline (p))
	printf ("  %s", line);
pipeline_free (p);

man-db deals with compressed files a lot, so I wrote an add-on library for opening compressed files (which is somewhat man-db-specific, but the implementation wasn't difficult given the underlying library):

pipeline *decomp_file = decompress_open (compressed_filename);
pipeline *decomp_stdin = decompress_fdopen (fileno (stdin));

This library has been in production in man-db for over five years now. The very careful signal handling code has been reviewed independently and the whole thing has been run through multiple static analysis tools, although I would always welcome more review; in particular I have no idea what it would take to make it safe for use in threaded programs since I generally avoid threading wherever possible. There have been a handful of bugs, which I've fixed promptly, and I've added various new features to support particular requirements of man-db (though in as general a way as possible). Every so often I see somebody asking about subprocess handling in C, and I wonder if I should split this library out into a standalone package so that it can be used elsewhere. Web searches for things like "pipeline library" and "libpipeline" don't reveal anything that's a particularly close match for what I have. The licensing would be GPLv2 or later; this isn't likely to be negotiable since some of the original code wasn't mine and in any case I don't feel particularly bad about giving an advantage to GPLed programs. For more details on the interface, the header file is well-commented. Is there enough interest in this to make the effort of producing a separate library package worthwhile? As well as the general effort of creating a new package, I'd need to do some work to disentangle it from a few bits and pieces specific to man-db. If you maintain a specific package that could use this and you're interested, please contact me with details, mentioning any extensions you think you'd need. I intentionally haven't enabled comments on my blog for various reasons, but you can e-mail me at cjwatson at debian.org or man-db-devel at nongnu.org.

• There is no Plan B: why the IPv4-to-IPv6 transition will be ugly
• Scanner looking under clothes has already been misused at Heathrow.
• Landell Webcasting - interesting alternative for <ahref>DVSwitch with simple setup.

It's been raining this weekend in Southampton. This afternoon's not been too bad - I went for a walk around the city walls. Just a few hundred years ago, the River Test would have come right up to them, and my flat would have been in a defensive moat full of sea water, I think. I re-read zack's rcbw page, and noticed the section on 'blog posts' - there is also a propaganda element to the idea, in improving Debian's internal culture w.r.t. NMUs. So with my grey Sunday afternoon, I have uploaded a couple of NMUs for RC bugs:

• Bug #585614 - a small missing dependency from mp3splt-gtk.
• Bug #591547 - pkg-config compatibility with autoconf >= 2.66

I am surprised that there are still such easy pickings from the RC bug list at this point in the release cycle. This Debian Developer lark is a piece of cake, innit?

Steve Jobs is a name that comes up a lot when talking to businesspeople, especially in the technology industry. His ideas, his background, his companies, their products, and his personal style are intertwined in the folklore of tech. I have no idea whether most of it is fiction or not, and I write this with apologies to Mr. Jobs for using him as shorthand. I have never met him, and my point here has nothing to do with him personally. What I want to discuss is the behavior of people who invoke the myth of Steve Jobs. In my (entirely subjective) experience, it seems to me that there is a pattern which comes up again and again: People seem to want to discuss and emulate the worst of his alleged qualities. Jobs has been characterized as abusive to his employees, dismissive of his business partners, harshly critical of mistakes, punishingly secretive, and otherwise extremely difficult to work with. Somehow, it is these qualities which are put forward as worthy of discussion, inspiration and emulation. Is this a simple case of confusing correlation with causation? Do people believe that Steve Jobs is successful because of these traits? Perhaps it is a way of coping with one s own character flaws: if Jobs can get away with such misbehavior, then perhaps we can be excused from trying to improve ourselves. Or is there something more subtle going on here? Maybe this observation is an effect of my own cognitive biases, as it is only anecdotal. As with any successful person, Jobs surely has qualities and techniques which are worthy of study, and perhaps even emulation. Although direct comparison can be problematic, luminaries like Jobs can provide valuable inspiration. I d just like to hear more about what they re doing right. Perhaps this is an argument for drawing inspiration from people you know personally, rather than from second-hand or fictitious accounts of someone else s life. I ve been fortunate to be able to work with many different people, each with their own strengths, weaknesses and style. I ve seen those characteristics first-hand, so I also have the context to understand why they were successful in particular situations. If there s one thing I ve learned about leadership, it s that it s highly context-sensitive: what worked well in one situation can be disastrous in another. Is your company really that much like Apple? Probably not.

shapado.debian.net Since a few hours, a Shapado instance for Debian is available at: Shapado is a Question and Answer (QA) website, meant to provide user support to the community, by the community. Another way of seeing it is as a skill exchange website: users can ask questions about stuff they don't know and other users, possiby more expert on that topic, can answer those questions; everyone can then use peer review to evaluate and sort answers. Several proprietary web applications are based on the above notions and have proven useful for user support in several contexts, including other Free Software distributions. Shapado on the other hand is Free Software released under AGPL; as such it's the best match to give a try to something similar in the context of Debian. Shapado has also a couple of other niceties, e.g. offers login via OpenID and supports questions/answers in multiple languages. Will it work for us?
Will it not?
It's up to the community. This kind of user support works when it reaches a critical mass of both "newbie" users in need of help and of "expert" users willing to help them. No matter your category, give it a try: go to the website and look around, you'll probably find a question to ask, a question to answer, or an answer to evaluate. Acknowledgements

• the Shapado developers
• Patrick Aljord for hosting support
• Fernando C. Estrada and Luis Uribe for admin / moderation work (by the way, they are looking for help)
• Jorge Castro for pinging me about this idea

---

Update: add mention of multi-language support

For some time, I've had this idea brewing for how the 'net would work without DNS. Initially I was mostly thinking of how to get rid of the dark side of the registrar business, but it appears that maybe it could help with security in general too. It may not be feasible at all, or I may have forgotten extremely crucial points. I'm not a DNS guru, so bear with me and consider this as a thought experiment. This is, however, a proposal for how the Internet could be made truly decentralized, or acentric. Made so, because it currently is very centralized on at least two counts: the DNS is an inherently hierarchical structure of authority; and so is the situation with the SSL certificates used for securing websites. Let's start with DNS, if you will. Basically, the idea is to replace the current Domain Name System, which is centralised in nature (even if subdomains can be delegated) with a fully decentralised system with no central authority. The point of current DNS is to map names to IP addresses in a unique way. It currently works by registering a this name is managed by such-and-such server, ask for details at such-and-such IP address message. Of course, since there's only one namespace (or a limited number thereof, if you consider that TLDs are different namespaces), there needs to be a central authority to resolve conflicts, because some names will be more popular than others. I propose to start with a cryptographic key pair (think GPG key). Anyone can create their key pair; we assume that it's extremely unlikely for two people to generate the same keys, and that it's also very hard for two people to generate keys with the same fingerprint. By (being the only one) knowing the secret part of a key, I am the authority on what can be signed with it. I could for instance sign records that say that www points to this set of IP addresses, and smtp points to that other address. Within my area of authority, which is to say within my domain. Which is identified by the fingerprint of my key pair. What do we have so far? A mechanism that allows in a decentralised way to verify who's authoritative on the F5566852EB92BD779CF137190EA756B5144843F5 domain . In DNS terms, more probably F55...3F5.some-tld. Now let's publish the signed records on a decentralised peer-to-peer network. I'm not up to date with the latest trends in that domain, but I think something like Gnutella or Freenet would qualify, since it doesn't need central hubs but only an initial list of nodes, and it does automatic discovery of new nodes. Projects such as the FreedomBox (name likely to be changed) will probably help with that. Anyway, when looking up www.F55...3F5.some-tld, the resolver gets the signed record (and the public part of the key) from the P2P network, checks the signature, and uses the contents of the record to determine the IP address. No central or delegated registry involved at any point. Lesser risk of a single point of failure, which means better resilience against outages. Also, a better resilience against DNS-based censorship, be it promoted by non-democratic states or by greedy corporations. Of course, F5566852EB92BD779CF137190EA756B5144843F5 isn't a terribly easy name to remember or transmit. But the string of hex could be presented to the user in some other form easier to recognize for people without geek super-powers. My favourite would be the bibi-binary representation, but I'm told that the bubble-babble method works equally well. It would still be unreasonable to expect people to remember such names, and especially to type them into their web browsers or e-mail agents. A possible solution to that would be to use a short form, such as 144843F5 (or its bibi-binary equivalent), complemented by a visual representation of the full fingerprint such as the one SSH does. That pair could be displayed on business cards, email signatures, glossy brochures, advertisements and so on, and allow users to just type the strange word or flash the 2D barcode, check that the funny picture matches (in case the strange word returns several results), and get the website they want. Which they can of course bookmark under a name they can remember, probably even the same name that everyone else uses to bookmark that website. And nothing precludes convenient directories mapping names to full URLs (like search engines currently do) as long as users check that the visual representation of the site they go to matches what they've seen elsewhere. That elsewhere could, of course, take advantage of the natural web of trust that emerges in of GPG-using communities. If I'm directed at a website whose key I don't immediately recognize, then I can see who trusts that key to be legitimate (by checking the signatures), and who trusts them, and so on. Ultimately, the decision is mine to make, and mine only. In any case, once I know what URL I want to visit, I can be sure this DNS replacement will give me the right IP address (or addresses). DNS servers replying with advertisement sites instead of correct ones, and DNS-based filtering, will be things of the past. Of course, having the right IP address only solves the attacks on the DNS part. Eavesdropping can also happen by diverting traffic at the IP level. But we have a trustable way to know information related to a domain, so let's use it: the signed record can also contain the fingerprint of cryptographic keys (SSL or SSH or whatever). So we can be sure we're actually talking to the correct server, rather than to some man-in-the-middle spy. With the same guarantees of correctness as the ones for the DNS, backed by the web of trust and out-of-band validation. Again, we get rid of the hierarchy of certification authorities , some of which have been known for helping organizations get at the (supposedly secret) data people exchange over the net. To summarize: with this scheme, if I know the site I want to access is F55...3F5, then I can be certain that a) I get the right IP address for it and b) I'm actually talking to it. And nobody in-between can intercept the traffic, if the site is properly administrated and its private key is kept secure. And from the site admin's point of view, there's no need to trust DNS registrars and SSL certification authorities not to fool around with the data, or blackmail me into paying increasing fees for keeping my URL working or my SSL key certified. There's still the problem of the allocation for the IP addresses themselves, but two major parts of the centralized and hierarchical net can be gotten rid of. There are proposals attempting to fix part of the problems already, but they're incomplete. DNSSEC only solves the authentication of DNS records, not the centralisation problem. Monkeysphere aims at adding a web of trust to the SSL certificates system, but the CA chain problem seems to persist (although I must admit I'm not up to speed with the actual details). This is of course only a rough draft, full of technical considerations yet certainly not exhaustive. I've thought about it though, and I believe there are no theoretical problems with the implementation of such a scheme (apart for the handling of expiration and revocations, probably). The main obstacle to mass-adoption I can envision is the UI part, and the entrenched habit of just typing a keyword into a search engine and implicitly trusting its results, then blindly trusting the website itself; but even if mass-adoption isn't reached, I can't see anything wrong with giving those who want to be careful a more secure Internet.

A new month has just begun, it s again time for a few suggestions of free software projects to flattr. This is Flattr FOSS in action. Enough said, let s start with my suggestions:

1. The battle for Wesnoth is a great turn-based tactical strategy game (I spent countless hours on it a few years ago). It s been a long time I have not played on my computer, but opensource games should be encouraged. Lack of good games is one of the few recurring complaints that come back from people trying out Linux for the first time. And The battle for Wesnoth is a very active project, so there s no reason to not support them.
2. Shutter is the next-generation tool to take screenshots, it s quite popular at least in Ubuntu and it can easily replace the default gnome-screenshot application. You can apply various effects on the fly, and upload the resulting pictures directly on image hosting sites (Flickr, Picasa, etc.).
3. Awesome is a highly configurable window-manager for X. I use it as a tiling window manager (windows are arranged to always fill the entire screen). Its configuration file consists of LUA code so it s for power-users mainly Julien Danjou started Awesome but he has also written many other small nifty tools, discover them in his Flattr profile.
4. Sparkleshare is a new collaboration tool that will transparently synchronize a folder between several computers/persons and inform you in real-time of changes. Under the hood, it uses public Git hosting sites (like Github or Gitorious) to store and exchange the data. In some aspects, it s like Dropbox s shared folders. You can also read Linux Weekly News review of it.
5. My last suggestion is to support Harald Welte and his projects. Click here to see his Flattr profile. He is someone in the free software world. First of all he s a famous kernel hacker, he wrote much of the firewalling code (known as netfilter). He also initiated gpl-violations.org and ensured that the GNU General Public License was respected when he found out companies that failed to meet the terms of the license. Lately he has been reverse-engineering the whole GSM stack pointing out security problems as he discovers them with the goal to provide a free implementation of everything (first results are in projects OsmocomBB and OpenBSC). Isn t that impressive?

That s it for this month. By the way, did you tell your friends how easy it is to support free software with Flattr? Share this article with them and let them join Flattr FOSS too. No comment Support my work

There was a lot of news coverage like Credit card fees being imposed by councils, says Which? earlier this week. People seemed outraged that councils were making a small surplus over the year in total. Interviewees suggested that councils should just include the cost of taking payments in the amount charged. Of course, councils are sort-of not-for-profit, so they ll just set prices to include the credit card fees and make a small surplus on other cheaper forms of payment over the year in total. So everyone who uses bank transfers will pay more because credit card fees are high! Will that be next year s scandal? Presumably that s what happens in France, Germany and other countries where sellers can t charge more for credit card payment. That, or you just don t accept credit cards, like many German retailers never used to. What else can they do? It s a bit odd of Visa to claim it costs more to accept other forms of payment. Cheques are horrendously expensive if you only get a few, but seem to get cheaper if handled in bulk. Domestic bank transfers and cash also seem cheaper than Visa to me. Sellers seem to be asked explicitly to assume much more fraud risk in order to accept credit cards, too. Mind you, with the amazingly slack approach of most US retailers to cards (no chip-and-PIN and they usually don t bother to even check the signature I cut up the card I used in the USA when I arrived home), I m surprised there s not more card fraud. I m also a bit surprised Bath and North East Somerset (which neighbours my home district) is paying 2% on credit card transactions. Our co-op has been offered deals about as good as that and we don t do anything like as many transactions. We ve not yet taken one up because the setup paperwork is a pain and the setup and maintenance fees usually suck. It feels like it doesn t matter what you do with money, bankers still get rich. Even though I keep looking (like in 2007), it ll probably happen: we need to act again, because some overseas customers seem unwilling or unable to register with a site to pay (and some payment sites have ethical questions too), while international bank transfers are even more expensive and an even bigger pain. Besides card payments, we re considering opening a currency trading account and a Euro bank account. Anyone done these and do you think they re worth it or not? And do any Eurozone banks allow a UK company to open an account without a member visiting in person? I ve found few possibilities, after looking at banks in seven countries so far.

After being noticed in blog comment about another theme for phpMyAdmin, I started to look for other hidden contributions for phpMyAdmin and I was surprised how many people did not even try to submit their code upstream. Have you ever considered trying to contact upstream? Quick search on SourceForge.net revealed: VisualMCD for phpMyAdmin

An Add-on for phpMyAdmin which enables user to create PDF Schema and Relation with a simple click and mouse moves. No more fastidious typing of table coords needed to create the PDF Schema.

Even though the SF.net history says it was released 440 days ago, it is actually dated back to 2003 and it obviously is not compatible with current code base. Also we have similar feature for few years now, so it was just duplicated effort. phpMyAdmin theme: Paradice

Paradice is a Theme for the popular web-based database administration tool phpMyAdmin. You will find the development version here. All official releases will be made by the phpMyAdmin project.

This theme is just developed externally, but you can get it from our themes page phpMyAdmin persian project

phpMyAdmin persian project

Something what seems to be dead from beginning. Anyway if you want to translate, please use our translation server. MySQL Form Generator pour PHPMyAdmin

C'est un outil pour crr un site autour d'une base. Un gnrateur de formulaire. Sur base d'une DB Mysql.Projet prvu pour tre intgr PhpMyAdmin.mysql -> formulaire HTML+javascript avec le code PHP pour trater les soumissions de donnes.

Something we still don't have in phpMyAdmin, however the project is inactive for more than year. pmahomme

Clean, modern and easy to use phpMyAdmin theme.

Theme which started this search, however it seems to fail for me right now. phpMyDesigner

Plugin for phpMyAdmin. PhpMyDesigner is a tool written in PHP and Ajax. This CASE with WEB interface. ( MySQL )

This actually does not fit into this blog, because it is one of projects that got merged and are now part of our code base. phpMySchema

phpMySchema is a tool written in PHP for generating an ERD or otherwise graphical view of your MySQL databse over the web. It is being developed to be used as a standalone script and as a module for phpMyAdmin.

Generally it looks like an interesting addition, however we can quite well compete with this after this year's GSoC, which added ability to export schema in various formats.

Filed under: Phpmyadmin 0 comments

Lucas has written two long, insightful posts on the frustration about the social aspects of integrating Ruby stuff Debian The first one, on September 12 and the second one, on September 29. I cannot really blame (thoroughly) the Ruby guys for their position. After all, they have a vibrant community, and they are advancing great pieces of work. And they know who that code is meant for Fellow programmers. And yes, although it is a pain to follow their API changes (and several of the Gems I regularly use do often get refactorings and functionality enhancements which break compatibility but introduce very nice new features), they say that's solved with one of Gems' main features being the simultaneous installability of different versions. The key difference in Debian's worldview with Ruby's is they cater to Fellow programmers. Even leaving aside heaps of different positions and worldview/mindset, we have a fundamental difference: Debian cares about its users, whatever that means. So, our users should not even care what language a given application is implemented in They should only care that it works. We, as packagers, should take care of all the infrastructural stuff. And yes, that's where we find the conflicting spot: We don't want to ship many versions of a system library (that in this case would be a Gem). Specially if later versions fix known bugs in earlier versions and backports are not available or supported. Specially if upstream authors' only response to a bug in an older release will be "upgrade and rewrite whatever breaks in your application". As an example of this, I am not currently updating the gems I maintain, as Debian is on a freeze to get out the next stable release. Or if at all, I am targetting those uploads to our Experimental branch, in order not to create a huge backlog for me when the freeze is over (just a series of rebuilds targetted at unstable). And yes, I will have to be responsible for any bugs that will most likely not be supported by most of my upstreams during the next ~2 years. That's the role of a Linux distribution. And yes, as Lucas writes in the comments he got as responses to the first post This dissonance comes in no small part because the Ruby developer community is mostly made from non-linuxers. People coming from a background where (mostly propietary) applications bundle up everything they need, where static linking is more popular than dynamic libraries, where there is no coordination between parts of the system are much less likely to understand our work. And yes, the Perl community is a joy to work with in this regard. And that's the same I understand from the Python one. Because of their origins and where their main strength was grown and remains. PS - And yes, I will join the flock of people saying that... The specific person that attacked your work is a great programmer, but well known as intolerant and obnoxious. Fortunately, even if our respective cultures fail to mix so much, most of our interactions just end with a "sigh" of lack of understanding, and not with the flames you got targetted with :-/

In my Ruby packaging in Debian and Ubuntu: Mythbusting and FAQ, I wrote The Ruby developers (or expert users) community is generally very hostile towards Debian. Many harsh words and insults are used in discussions mentioning Debian in the ruby-core@ mailing list, and there are frequent recommendations to avoid the Debian packages and install from sources, which is quite demotivating (Actually, what prompted that blog post is someone calling Debian-specific changes unforgivable). Zed A. Shaw just provided a very good example of that. Quoting the most juicy parts:

It s simply a tactic to make sure that you are stuck on Debian.

If we look at this problem from an open source perspective, Debian is at a minimum being a horrible citizen.

What is happening is control is not aligned with responsibility. Debian has taken control of the software package and recrafted it for their very narrow set of system administrator users. However, they refuse to take responsibility for the defect and quality problems they create. Instead I end up being responsible for their mistakes, which is wrong.

If Debian is in control, then Debian should be made responsible. That s the only way to get them to change it. If they don t want to be responsible, then they should let me be in control.

I m imagining a mass petition, maybe some funny ad campaigns, in-person confrontations meant to embarrass package maintainers, SEO tricks to get people off Debian, promotion of any alternative to Debian, anything to make them pay, apologize, and listen.

If you write software and you re sick of Debian screwing up your gear, then shoot me ideas for how we can make this happen. I think that if enough of us band together to make Debian responsible for its actions we can actually get them to start asking us before they package our software to find out how we think it should be packaged.

On the technical part of the post: The problem that Zed is experiencing is unknown to me, and also to the Debian and Ubuntu bug trackers. To my knowledge, rubygems doesn t not require openssl or net/scp to work, and a simple grep through the sources kind-of confirms that. But of course, that blog post was written as a rant, not as a proper bug report. My best guess is that the gem that Zed is installing requires openssl and net/scp. But of course, rubygems can t just depend on all the libraries that applications or libraries installed via rubygems could possibly require. Regarding openssl, we had a problem in Ruby because of the licensing situation of Ruby (dual licensed GPLv2+ and Ruby license, linked with openssl, which might raise some eyebrows), so it was split off the main standard library package until recently (and, apparently, in the version Zed is using). For net/scp, it is not part of the standard library, and not a dependency of rubygems, so I don t see why Zed expects it to be installed. On the rest of the post: Following my last post, several people volunteered to help with Ruby on Debian (and Ubuntu). In particular, Laurent Arnoud has been doing some fantastic work fixing the pkg-ruby-extras team s RC bugs. Also, all the messages of support both in the comments of my last post were very much appreciated. But I m really wondering why the Ruby community generates so many poisonous people. All my work on Debian is done on a volunteer basis, and it s really hard to get so much shit from it.

So I never had the chance to see Weeping Tile in Concert, but this year I have managed to see both Luther Wright in concert and Sarah Harmer. That and I found "The show 1995" channel on youtube which does have weeping tile in concert. And Uncle Remus.

Can you believe it? I still talk to people who don t know how to play blackjack. There must be something in the mechanics or tableau of blackjack that people find intimidating; for some reason, people can be a little cautious about taking the first move.
Photo courtesy of Winquerque on Net via Flickr. Clearly, there s no cause to hesitate. Players find quite quickly that it is actually not hard to figure out, and the strategies aren t even terribly complicated. For sure, if you re used to playing online slots, you are probably going to need to work a little harder to master the nuts and bolts. But it s still nothing like getting to grips with Texas Hold em from the beginning, for example!. Quite frankly, you very likely already know a fair portion of what most players have to know. I am able to give you the essentials in one simple little paragraph: If the card has aparticular number, that s the amount it will be worth. If it doesn t, it s worth 10. The only exception to this rule are Aces, which are worth either 1 or 11 (more about that later). Say hit when you have to get closer to 21, and/or stand when you don t. If you are closest to 21, you win. If you bust (get more than 21), you lose. Getting dealt blackjack (a natural 21 from the first two cards), makes you an instant winner (well, you and the dealer can also tie, or push ). Got it? OK, so it is not completely all you need to know. To begin with, most professional blackjack fans understand all too well that your goal has less to do with trying to get to 21, and just about everything to do with defeating an opponent. The dealer will not have the luxury of actually playing , precisely. Whether you re playing at a land-based casino or online (speaking of which, for additional info on USA casino players, all you have to do is click the link!), the house rules specify each individual play they can make. Mainly, she or he needs to take another card until their cards add up to the house rule limit (generally 17 and over). Of course, one of you will get an ill-fated draw and find themselves busted , but this decides a fewer number of games than you might expect. The tricky aspect of blackjack is the decision that you have to make if the cards add up somewhere around 12 and 16 (approximately, subject to other details). If you only had your own cards to consider, it would just be nothing but chance. Don t get me wrong, dumb luck has quite a lot to do with it. It is a card game, of course, and all card games are based on luck to some extent. But the serious gamer recognizes that luck is essentially a code word for statistics ; if you know what you have, and you possess a clue about the dealer s cards, you may make a knowledgeable judgment about which cards you re more inclined to end up with if you decide to hit another time. The most useful clue that you require to evaluate whether to play risky or conservative (sorry, I meant loose or tight ) is the dealer s upcard. Well, that s all for now, but for more information, you can always take a look at our friends at USA online casinos accepting Webmoney , who will get you up to speed. A bit later, we re going to get down to the serious nitty-gritty: doubling down, splitting, and much much more. Until then !

Many Unix daemons use a technique known as pre-forking . This means that to save the amount of time taken to fork a child process they will keep a pool of processes waiting for work to come in. When a job arrives then one of the existing processes is used and the overhead of the fork() system call is saved. I decided to write a little benchmark to see how much overhead a fork() really has. I wrote the below program (which is released under the GPL 3.0 license) to test this. It gives the performance of a fork() operation followed by a waitpid() operation in fork()s per second and also the performance of running a trivial program via system which uses /bin/sh to execute the given command. On my Thinkpad T61 with a Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz I could get 2429.85 forks per second when running Linux 2.6.32 in 64bit mode. On a Thinkpad T20 with a 500MHz P3 CPU I could get 341.74 forks per second. In both cases is seems that the number of forks per second is significantly greater than the number of real-world requests. If each request on average took one disk seek then neither system would have the fork performance as any sort of bottleneck. Also if each request took more than a couple of milliseconds of CPU time on the T7500 or 10ms of CPU time on the 500MHz P3 then the benefits of pre-forking would be very small. Finally it s worth noting that the overhead of fork() + waitpid() in a loop will not be the same as the overhead of just fork()ing off processes and calling waitpid() when there s nothing else to do. I had a brief look at some of my servers to see how many operations they perform. One busy front-end mail server has about 3,000,000 log entries in mail.log per day, that is about 35 per second. These log entries include calling SpamAssassin and Clamav, which are fairly heavy operations. The system in question averages one Intel(R) Xeon(R) CPU L5420 @ 2.50GHz core being used 24*7, I can t do a good benchmark run on that system as it s always busy but I think it s reasonable to assume for the sake of discussion that it s about the same speed as the T7500 (it may be 5* faster, but that won t change things much). At 2429 forks per second (or 0.4ms per fork/wait) if that time is entirely reduced to zero that won t make any noticeable difference to a system that has an average operation taking 1000/35= 28ms! Now if a daemon was to use fork() + system() to launch a child process (which is a really slow way of doing it) then the T7500 gets 248.51 fork()+system() operations per second with bash and 305.63 per second with dash. The P3-500 gets 24.48 with bash and 33.06 with dash. So it seems that if every log entry on my busy mail server involved using a fork()+system() operation and it was replaced to use pre-forked daemons then it might be possible to save almost 10% of the CPU time on that system in question. Now it is theoretically possible that the setup of a daemon process can take more CPU time than fork()+system(). EG a daemon could have some really complex data structures to initialise. If the structures in question were initialised in the same way for each request then a viable design would be to have the master process initialise all the data which would then be inherited by the children. The only way I can imagine for a daemon child process to take any significant amount of time on modern hardware is for it to generate a session encryption key, and there s really nothing stopping a single master process from generating several such keys in advance and then passing them to child processes as needed. In conclusion I think that the meme about pre-forking is based on hardware that was used at a time when a 500MHz 32bit system (like my ancient Thinkpad T20) was unimaginably fast and when operating systems were less efficient than a modern Linux kernel. The only corner case might be daemons which do relatively simple CPU bound operations such as serving static files from a web server where the data all fits into the system cache, but even then I expect that the benefit is a lot smaller than most people think and the number of pre-forked processes is probably best kept very low. One final thing to note is that if you compare fork()+exec() with an operation to instruct a running daemon (via Unix domain sockets perhaps) to provide access to a new child (which may be pre-forked or may be forked on demand) then you have the potential to save a moderate amount of CPU time. The initialisation of a new process has some overhead that is greater than calling fork(), and when you fork() a new process there are usually lots of data structures which are not written after that time which means that on Linux they remain as shared memory and thus reduce the system memory use (and improve cache efficiency when they are read). #include <unistd.h>
#include <stdio.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <stdlib.h> #define NUM_FORKS 10000
#define NUM_SHELLS 1000 int main()

struct timeval start, end;
if(gettimeofday(&start, NULL) == -1)

fprintf(stderr, "Can't get time of day\n");
return 1;
int i = 0;
while(i < NUM_FORKS)

pid_t pid = fork();
if(pid == 0)
return 0;
if(pid > 0)

int status;
pid_t rc = waitpid(-1, &status, 0);
if(rc != pid)

fprintf(stderr, "waidpid() failed\n");
return 1;


else

fprintf(stderr, "fork() failed\n");
return 1;

i++;
if(gettimeofday(&end, NULL) == -1)

fprintf(stderr, "Can't get time of day\n");
return 1;
printf("%.2f fork()s per second\n", double(NUM_FORKS)/(double(end.tv_sec start.tv_sec) + double(end.tv_usec start.tv_usec) / 1000000.0) ); if(gettimeofday(&start, NULL) == -1)

fprintf(stderr, "Can't get time of day\n");
return 1;
i = 0;
while(i < NUM_SHELLS)

pid_t pid = fork();
if(pid == 0)

if(system("id > /dev/null") == -1)
fprintf(stderr, "system() failed\n");
return 0;

if(pid > 0)

int status;
pid_t rc = waitpid(-1, &status, 0);
if(rc != pid)

fprintf(stderr, "waidpid() failed\n");
return 1;


else

fprintf(stderr, "fork() failed\n");
return 1;

i++;
if(gettimeofday(&end, NULL) == -1)

fprintf(stderr, "Can't get time of day\n");
return 1;
printf("%.2f fork() and system() calls per second\n", double(NUM_SHELLS)/(double(end.tv_sec start.tv_sec) + double(end.tv_usec start.tv_usec) / 1000000.0) );
return 0;

Daily builds As of a month ago there are Ubuntu archives with fresh packages of Samba 4 and Openchange, built on a daily basis day from the latest upstream revision. This means that it is now possible to run a version of Samba 4 that is less than 24 hours old, without having to know how to extract source code from the version control system that upstream is using, without having to know how to build and install an application from source, but perhaps most importantly: without having to go through the tedious process of manually updating the source code and rebuilding. OpenChange is tightly coupled to Samba 4, so installing a new version of OpenChange usually involves installing a new version of Samba 4 as well. To make matters more confusing, the two projects use different version control systems (Samba 4 is in Git, while OpenChange is in Subversion) and different build systems (Samba 4 uses waf, OpenChange uses autoconf and make). I have been involved in Samba 4 and OpenChange as an upstream developer and more recently also as a packager for both Debian and Ubuntu. As an upstream developer for both these projects it is important for me that users can easily run the development versions. It makes it possible for interested users to confirm the fixes for issues they have reported and to test new features. The more users run the development version, the more confident I can be as a developer that doing a release will not cause any unexpected surprises. As a packager it is useful to know when there are upstream changes that are going to break my package with the next release.

Recipes The daily builds work using so-called recipes which describe how to build a Debian source package from a set of Bazaar branches. For example, the Samba 4 recipe looks like this:

# bzr-builder format 0.2 deb-version 4.0.0~alpha14~bzr revno ~ppa revno:packaging + revno:debian 
lp:samba
merge debian lp:~samba-team/samba/unstable
merge packaging lp:~samba-team/samba/4.0-ppa-maverick

This dictates that a source package should be built by taking the upstream Samba branch and merging the Debian packaging and some recipe-specific tweaking. The last bit on the first line indicates the version string to be used when generating a changelog entry for the daily build. Every night Launchpad (through bzr-builder) merges these branches and attempts to build the resulting source package, e-mailing me in case of build problems. Generally I fix issues that come up by committing directly to upstream VCS or to the Debian packaging branch. There is no overhead in maintaining the daily build after I've set it up. For more information on creating source package recipes, see getting started.

Toolchain The entire toolchain that does the daily package builds for Ubuntu is Free Software, and I have contributed to various bits of that toolchain over the years. It's exciting to see everything come together.

Soyuz Launchpad consists of multiple pillars - one of those pillars is Soyuz, which I hack on as part of my day job at Canonical. Soyuz is responsible for the archive management and package building. Debian source packages (a combination of upstream source code and packaging metadata) get uploaded by users and then built for various architectures on our buildfarm and published to the Ubuntu archive or to users personal package archives.

Launchpad-code Another pillar of Launchpad is Launchpad-code, which is responsible for the hosting and management of version control branches. Launchpad users can either host their branches on Launchpad directly or mirror branches (either native Bazaar branches or branches in a foreign format such as Subversion, Git or Mercurial). The mirrorring of native and foreign branches happens using standard Bazaar API's. In the case of Samba and OpenChange we import the branches of the upstream projects (Samba is in Git, OpenChange is in Subversion) and the packaging for both projects is in Bazaar. Launchad-code calls out to Bazaar to do the actual mirrorring. Over the last few years I have done a lot of work to improve Bazaars support for foreign branches, in particular on supporting Subversion, Git and Mercurial. As the code mirrorring in Launchpad is one of the biggest users of bzr-svn and bzr-git it has helped find some of the more obscure bugs in those plugins over the last few years, to the point where there are only a handful of issues with Git and Subversion imports left.

bzr-git and dulwich bzr-git provides transparent access to Git repositories from within Bazaar and is built on top of Dulwich. Dulwich is a Python library that provides access to the Git file formats and protocols that is completely independent of Bazaar. James Westby originally started it and I adopted it for bzr-git and further extended it. There are now several other projects that use it as well, including hg-git, and rabbitvcs. Apart from James and me almost two dozen other people have contributed it so far.

bzr-svn and subvertpy bzr-svn provides transparant access to Subversion repositories in Bazaar. When I grew frustrated with the existing Subversion Python bindings for various reasons, I decided to create independent Python bindings for Subversion from scratch. These bindings have since been split out into a separate project - subvertpy - and other projects have since also started using them, e.g. hgsubversion and basie.

Using the daily builds To use the Samba 4 and OpenChange daily builds (Ubuntu Maverick only for now), run:

$ apt-add-repository ppa:samba-team/ppa
$ apt-add-repository ppa:openchange/daily-builds

cp: Karnivool - Themata

---

This year, I have moved to a new country two times. This sharply brings to attention the pain and cost of owning things. It is not just when carrying maximally filled suitcases across airports that things are bothersome. The more things you have, the bigger a home you need, and the more effort you need to spend on cleaning things and home. A meme of sorts in recent years has been surviving on 100 things only. That sounds like very little, but it looks doable, if a bit extreme. I am sorely tempted to try this, but, well, I am not like Joey, living in yurts, and using only a miniature netbook for my computing needs. But perhaps that's the problem? I am too accustomed to my stuff, and it is hard to give up things, even when I don't really need them much. Like, USB hard disks. I like to have a Debian mirror locally, so I have a disk which has a couple of hundred gigabytes of Debian packages. However, I don't actually use that much. Would it make more sense to rely on servers hosted elsewhere? While those might be part of the 100 things count, at least I wouldn't have to carry them, ever. But then I'd need to have Internet access, and that's also difficult, especially in the early months of living in a new country. Things would be so much easier if I could just buy a Scottish island, with a castle, fix that up with some broadband and weekly grocery deliveries, and never go anywhere again.

I use to take my (company) notebook with me on business travels.
Two times I now had the unlucky situation that something bad happened to it on such an occassion. Whenever you get in the situation that you need to reinstall your system in a hotel room you'll might have the same wish that I got: A way to quickly bring the system in a state where I could work with it.

Well, I used FAI a while back for a customer. Its a real great tool for automated installations and I really prefer it over debian-installer preseeding. Apart from the fact that the partitioning is way easier it also gives me the power to complete the whole installation up to a point where I've got almost nothing to do anymore. It also features an installation completely from CD or USB-Stick which makes it suitable for me.

However, my notebook installation has a little "caveat" which made that a little bit more harder as previously thought. As it is a notebook and I carry company data on it it has to be encrypted. Disk encryption at a whole.
The stable FAI version does not support this.
The problem is: The current support for crypto in setup-storage (FAIs disk setup tool) is not very far. Supported is the creating of a LUKS container with a keyfile, saving this keyfile to the FAI $LOGDIR and creating a crypttab.
Unfortunately for a root filesystem this would leave us with an unbootable system, because this requires manual interaction. And on the other hand using a keyfile for a cryptoroot is a no-go anyway. We want a passphrase.
On a side-note: cryptoroot support with a keyfile is more complex than with a passphrase, as you have to provide a script that knows how to get to the key.

So I started experiments with scripts in the FAI-configuration that added a passphrase, changed the crypttab and recreated the crypttab. That worked, although it was very ugly.
But due to a good coorperation with Michael Tautschnig, a FAI- and Debian-Developer, on this, the FAI experimental version 4.0~beta2+experimental18
now supports LUKS-volumes with a passphrase that can be specified in the disk_config.

Now its actually possible to setup a system like mine with FAI out-of-the-box. One thing (apart from the FAI configuration and setup as you want and need it) has to be done, anyway:
The initrd support of cryptsetup requires busybox (otherwise you will see a lot of "command not found" errors and you system won't boot) and it requires initramfs-tools, which is standard nowadays.
So you have to make sure that these packages are in your package config!

So now I can define a FAI-profile for my notebook, create a partial fai mirror with the packages it needs and put all this together on an USB stick with fai-cd (don't worry about the name, it can be used to create ISO images as well). I can carry this with me and if I need it I stick it into my notebook and let FAI automatically reinstall my system. Nice :)

Update: Somebody asked me, weither he understood me right, that I'd put my LUKS passphrase on a FAI usbstick clear-text. Obviously, the answer is and should be NO. What I do and what I'd suggest to others: Use a default passphrase in the FAI configuration, install with it - after all on a fresh installation there is not much to protect - and once it is finished *change* the passphrase to something secure by adding a new keyslot and removing the old.