Fresh up on arxiv.org is a paper by Kevin Heng, Kristen Menou and Peter Phillipps: Atmospheric circulation of tidally-locked exoplanets: a suite of benchmark tests for dynamical solvers. Kristen Menou was at exoclimes 2010 where one of the background topics was how to compare and validate our various models in exoplanetary climates. Earth-based climate modellers have test cases such as Held-Suarez which are used to compare the models, but to date no equivalents are used for exoplanetary models. This work fills that gap for Tidally-locked atmospheres (similar to Zarmina for example, and some 'Hot Jupiters'. I'm particularly interested in the tidally-locked Earth case; it's relevant for my own PhD work. Incidentally, for a good summary of the different ways we test climate models, see Steve Easterbrook's recent blog entry; he's been doing a tour of major modelling centres, learning how they do things. Worth a look. Tags exoplanets, climate, benchmarks

After pushing for completion to the Portuguese l10n team, I did the same work for Czech and Spanish teams, identifying what's missing for them to reach 100%. Doing this, I sent 15 intents to NMU various packages..dozens of mails to debian-l10n-czech and debian-l10n-spanish mailing lists. And I even "translated" debconf templates to Czech and Vietnamese for two packages..:-) (don't worry, these were trivial updates, of course. I still don't speak, read or write Czech and Vietnamese)

As some of you might know, I'm an OpenStreetMaper. In the last month, during those bits of spare time I had, I wrote a set of python scripts which compute some statistics over an OSM dump. I do this by parsing the whole XML tree, and national dumps are pretty huge (italy.osm is ~4G nowadays, and is not as well-mapped as Germany!), so I needed a way to do this without creating a memory-hungry beast. With Python, I could succesfully do this:

#!/usr/bin/python
# -*- coding: utf-8 -*-
import xml.etree.cElementTree as etree
def parse(filename):
    source = open(filename)
    context = etree.iterparse(source, events=("start", "end"))
    context = iter(context)
    event, root = context.next()
    for event, elem in context:
        if event == "end":
            root.clear()
            continue
        print elem.tag
if __name__ == '__main__':
    import sys
    parse(sys.argv[1])

And here the results:

$ /usr/bin/time ./parsexml.py ~/osmstats/dumps/italy.osm.bz2.20100912.out 1>/dev/null
413.22user 6.15system 8:57.80elapsed 77%CPU (0avgtext+0avgdata 21200maxresident)k
7739552inputs+0outputs (10major+1470minor)pagefaults 0swaps

However, since some time, I wanted to learn Vala, so I tried to do this very same task with it. Here's the code:

using Xml;
class XmlParser  
    public void parse_file(string path)  
        var handler = SAXHandler();
        void* user_data = null;
        handler.startElement = start_element;
        handler.user_parse_file(user_data, path);
     
    public void start_element(string name, string[] attr)  
        stdout.printf("%s\n", name);
     
 
int main(string[] args)  
    Parser.init();
    var parser = new XmlParser();
    parser.parse_file(args[1]);
    Parser.cleanup();
    return 0;
 

You need to compile it with: valac --pkg libxml-2.0 xml.vala And here's the result:

$ /usr/bin/time ./xml ~/osmstats/dumps/italy.osm.bz2.20100912.out 1>/dev/null
122.01user 4.03system 3:14.61elapsed 64%CPU (0avgtext+0avgdata 6352maxresident)k
7738984inputs+0outputs (0major+461minor)pagefaults 0swaps

Both these codes are, however, CPU-hungry. But at least they don't swap Needless to say that I'm planning to switch my scripts to Vala for 2.0.

I should probably have researched the batteries in this old solar-powered house before I installed a charge controller. I had assumed that the 24 6-volt batteries in the house's battery bay were standard lead-acid deep cycle batteries. It wasn't until I stumbled over a receipt from 1997 that I learned that the batteries are really NiCads. Specifically Saft model STM5-180. Thirty of these were purchased in 1997 from TVA in Chatanooga, for $350 total. An amazing price, since a single deep cycle battery is in that ballpark, new. These batteries were used, they had been in a "bus" -- perhaps it was a 1995 Chrysler TEVan, or maybe one of the pilot electric buses running there in the late 90's. They were probably not in this '79 VW TVA Bus. All mentions of these batteries I can find involve electric vehicles -- it's unusual for them to be used to power a house. Being vented NiCads probably accounts for these batteries' long useful life -- surely at least 15 years. Still, with only twenty-four good ones left, they are probably toward the end of their lifespan and need to be taken care of in order to last. Once I realized they were NiCads, I knew the charge controller was charging them wrong. NiCads like to be charged at a higher voltage than the 13 or so volts used for lead acid. I would have liked to charge them at 16 volts, but that would feed back through the house wiring, and could fry 12 volt stuff. Checking ratings, 15 volts seemed the highest voltage I could risk. Coming back a week later, I found the batteries charged up to 13.4 volts. And they are now working great. Through several cloudy days, we had all the power we needed. And when the sun was out fully, I sometimes saw the solar panels charging the batteries at 125 watts -- fully half of the panels' rated capacity, and much better than before. The batteries start each evening at 13.4 volts, and only drop to 13.3 by morning. We started using electric lights more, and then just leaving them on all evening, and the nslu2 online all night, and the batteries remained at the magic 13.3 in the morning. This is because NiCad batteries have a near-constant voltage until they are perhaps 30% discharged. In other words, I had been charging them less than half full before. I have figured out how to combine the two banks the batteries are in into one large bank, and once I get the cables to do that, I hope to have battery capacity to get through up to a week of solid clouds in midwinter.

battery box and solar panel

My phone contract ends in a few months, so I m looking at getting a new Android phone. I want a big Android phone (in both physical size and resolution) that has a physical keyboard, a digital compass, A-GPS and at least a 5MP camera with geo-tagging. I want to be able to read PDF files and run ssh sessions, so a big screen is required and a physical keyboard avoids wasting screen space for a soft-keyboard. My pockets will fit something about 10.5cm wide by 17cm high but I don t expect anyone to manufacture such a large phone. High resolution is a good thing too, it seems that the best available at the moment is 854*480 (with 800*480 being reasonably common). I want Wifi and all the 3G and GSM data transfer standards. It would be ideal to have a phone with the dual networking stack needed to do both voice and data at the same time. I m not interested in anything that runs a version of Android older than 2.2 as native tethering is important. An option to upgrade to post 2.2 would be a really good thing. Here are the nearest options I could find:

Phone	Resolution	Screen Size (inches)	Camera Resolution	Notes
Motorola Milestone	854*480	3.7	5MP
Motorola Droid	854*480	3.7	5MP
LG VS 740	800*480	3.2	3.2MP	no GPS or compass
Lenovo LePhone	800*480	3.7	3MP	no GPS or compass

It seems that Motorola makes the phones that best suit my needs, does anyone know of any better options?

In the last two cases where I created FAI configurations for different systems I needed to install packages which are not part of the Debian repositories.
Unfortunately FAI does not provide a standard way to supply the installation with custom packages so I need to figure out a way on my own.

I have the following requirements in the solution:

• Packages can be configured in package_config files as usual
• Packages will be installed during the normal package installation, no crude hack (like 'dpkg -i' and 'apt-get -f install' after usual package installation)

It would be easy if a repository reachable by the installed system would already exist. Unfortunately it does not. But.. aha! So here is the idea:

1. In the FAI configuration space add a new directory
   'packages'
2. We will use reprepro for the repositiory. You should install it where ever you prepare the configuration space. It would also be an option to install it into the NFSROOT and run it from there, but I decided to pre-prepare it (but this would be a clean solution if one of the FAI developers decided to implement a standard way, I guess).
3. Create a directory packages/conf and therein a file distribution with this content:
   

   Origin: Debian
   Label: Debian-All
   Suite: stable
   Codename: lenny
   Version: 5.0
   Architectures: i386 amd64
   Components: main
   Description: Debian Lenny

   Obviously you should adapt at least the Architectures line to whatever you need. Otherwise adapt what you need.
   
4. Now you can add packages, with reprepro includedeb lenny /path/to/debfile from within that directory, so cd to it, before.
   The result is, that the packages directory becomes a full-fledged Debian APT repository which can be used with apt by creating a line similar to the following:
   

   deb file:/var/lib/fai/config/packages lenny main contrib non-free

6. And thats what we'll do. In order to make this happen before any software gets installed, we create a hook. It needs to be called instsoft.CLASS where CLASS is a class that is used for the fai install we are working on. It basically needs to contain something like (+ the usual shell script stuff):
   
   

   $ROOTCMD ainsl /etc/apt/sources.list "deb file:/var/lib/fai/config/packages lenny main contrib non-free"
   $ROOTCMD apt-get update

   This should add the repository the sources.list in the /target and run apt-get update.
   

Thats it. This should be all to make the repo available to the usual installation. That means if your package_config files contain packages which are only part of the custom repo they will now be able to be installed.

(This might probably not be a copy and paste howto. It just describes the rough steps needed to do and might contain errors. So don't blame me if it doesn't work 100% this way. ;-)

Debian 6.0 squeeze will be the first GNU/Linux distribution release ever to offer comprehensive support for magnetic resonance imaging (MRI) based neuroimaging research. It comes with up-to-date software for structural image analysis (e.g. ants), diffusion imaging and tractography (e.g. mrtrix), stimulus delivery (e.g. psychopy), MRI sequence development (e.g. odin), as well as a number of versatile data processing and analysis suites (e.g. nipype). Moreover, this release will have built-in support for all major neuroimaging data formats.

Please see the Debian Science and Debian Med task pages for a comprehensive list of included software and the NeuroDebian webpage for further information.

NeuroDebian at the Society for Neuroscience meeting 2010

The NeuroDebian team will run a Debian booth at the Society for Neuroscience meeting (SfN2010) that will take place November 13-17 in San Diego, USA. The annual meeting of the Society for Neuroscience is one of the largest neuroscience conferences in the world, with over 30,000 attendees. Researchers, clinicians, and leading experts discuss the latest findings about the brain, nervous system, and related disorders.

If you are a Debian enthusiast (developer, contributor, evangelist) and reside near San Diego (or have time and funds for travel/lounge), or already planing to attend SfN 2010, please help us to make the Debian booth at SfN shine. Please contact the NeuroDebian team at team@neuro.debian.net

If you are going to SfN2010, come talk to us at booth #3815.

Contributed by: Michael Hanke and Yaroslav Halchenko

I've recently got a mail from one of my faithful reader, asking why not considering notmuch. Actually, I think notmuch already exists in a better way, and that's called IMAP and Sieve. What notmuch does, is tagging your mail with tags (obviously), based on filtering rules you write. The big downside is that you have to tag all your mails on your computer. And if you use several computers, you'll have to tag several times your mails. And you'll have to find a way to maintain your rules to be identical on all your computers. That does not scale. Using Sieve for mail filtering, one can do already that actually, and much more. A notmuch rule like:

notmuch tag +intel from:intel.com and not tag:intel

Can be written as a Sieve rule like:

if address :all :contains "From" "intel.com"  
        addflag "intel";
 

The flags extension for Sieve is explained in RFC5232. The Sieve based solution has the advantage of being treated server side, and therefore not subject to multiple or different MUA usages. It's also fast, if you use a good IMAP server like Dovecot, which has indexing, etc. Furthermore, Sieve can obviously do a lot more than tagging, like splitting into different mailboxes, filtering with regexp usage, vacation, etc. And if you want to fetch your mail locally, you can synchronize the IMAP box entirely with any software able to (like OfflineIMAP). Now, what's probably missing, is a correct support for IMAP flags on various MUA around. But that's not something notmuch helps to solve either. :-)

Roland Mas writes an interesting article about decentralized naming, in which he says:

Monkeysphere aims at adding a web of trust to the SSL certificates system, but the CA chain problem seems to persist (although I must admit I'm not up to speed with the actual details).

Since i'm one of the Monkeysphere developers, i figure i should respond! Let me clarify that Monkeysphere doesn't just work in places where X.509 (the SSL certificate system) works. It works in other places too (like SSH connections). And I don't think that the CA chain problem that remains in Monkeysphere is anything like the dangerous mess that common X.509 usage has given us. I do think that at some level, people need to think about who is introducing them to other people -- visual or human-comprehensible representations of public key material are notoriously difficult to make unspoofable. On the subject of distributed naming: OpenPGP already allows distributed naming: every participant in the WoT is allowed to assert that any given key maps to any given identity. Duplicates and disagreements can exist just fine. How an entity decides to certify another entity's ID without a consensus global namespace is a tough one, though. If i've always been known as "John Smith" to my friends, and someone else has also been known as "John Smith" to his friends, our friends aren't actually disagreeing or in conflict -- it's just that neither of us has a unique name. The trouble comes when someone new wants to find "John Smith" -- which of us should they treat as the "correct" one? I think the right answer probably has to do with who they're actually looking for, which has to do with why they're looking for someone named "John Smith". If they're looking for John Smith because the word on the street is that John Smith is a good knitter and they need a pair of socks, they can just examine what information we each publish about ourselves, and decide on a sock-by-sock basis which of us best suits their needs. But if they're looking for "John Smith" because their cousin said "hey, i know this guy John Smith. I think you would like to argue politics over a beer with him", then what matters is the introduction. And OpenPGP handles that just fine -- if their cousin has only ever met a single John Smith, that's the right one. If their cousin has met several John Smiths, then the searcher would do well to ask their cousin some variant of "hey, do you mean John Smith or John Smith ", or even "do you mean the John Smith who Molly has met, or the one who Charles has met?" (assuming that Molly and Charles have each only certified one John Smith in common with the cousin, and not the same one as each other), or to get a real-time introduction to a particular John Smith, where his specific key is somehow recordable by the searcher for future conversations (or beer drinking). This is what we do in the real world anyway. We currently lack good UIs for doing this over the network, but the certification infrastructure is in place already. What we're lacking in infrastructure, though, is a way to have a distributed addressing. Roland's proposal was to publish addresses corresponding to cryptographic identities within some DNS zone, or in freenet or gnutella. Another approach (piggybacking on existing infrastructure) would be to include IP address information in the OpenPGP self-certification, so the holder of the name could claim exactly their own IP address. This could be distributed through the keyserver network, just like other updates are today, and it could be done simply and immediately with a well-defined OpenPGP notation. I'd be happy to talk to interested people about how to specify such a notation, and what possible corner cases we might run into. Drop a note here, or mail the Monkeysphere mailing list or hop onto #monkeysphere on irc.oftc.netTags: authentication, distributed naming, identity, monkeysphere

What's the best way to run PHP in Debian in a shared environment? Years ago May First/People Link switched from the old reliable modphp to suPHP because modphp ran all php scripts as the same user, meaning (among other things) that any site could access the database configurations of any other site. suPHP was great, however, after several years we've decided to replace it because:

• As our members increasingly use Drupal and Drupal sites grow increasingly more complex, suPHP's performance has become a problem.
• suPHP executes with the permissions of the owner of the file it is executing. From a zero-conf perspective, this features makes suPHP very easy to setup. However, from a security perspective, it forces you to run your php script with the permission to delete itself. Not optimal.

We first turned to fcgid. We tried it out on a couple individual web sites (fcgid can run alongside suPHP), and, after a week without much problems, we implemented it on an entire server. Out of the box, on a shared server with several dozen web sites, the most notable resource utilization difference between fcgid and suPHP is the increase in memory usage. Unlike suPHP which dies after each requests, fcgid gains a performance boost by sticking around (with either 2 or 3 processes living on) to server future requests. If you have several dozen web sites that aren't high traffic sites, it means they may each have up to the three processes staking out memory that would otherwise be free with suPHP. We addressed the memory problem by adding the following to /etc/apache2/mods-available/fcgid.conf:

DefaultMinClassProcessCount 0
IdleTimeout 60
IdleScanInterval 60

These directions tell fcgid to kill processes that have been idle for more than 60 seconds. That means we still get the performance gain from a site getting more than one page view per minute, but otherwise we re-claim the memory. Before switching our remaining servers we decided to do more extensive testing to ensure we would be getting the performance and resource gains we were hoping. Also, we decided to consider mpm-itk, which is a variation of mpm-prefork that runs each virtual host with a per-vhost configured user and group (like suexec, but for the entire virtual host). Since it's a variation of mpm-prefork, it's safe to run it with the more resource friendly mod_php. We used the apache benchmarking tool ab. And we tested on a virtual machine (running on my laptop) with 1GB of RAM. The test web site we used was a copy of the May First/People Link site (we used the home page for the test), which is running Drupal 6. It's a relatively simple Drupal site, however, we turned Drupal caching off to better simulate PHP processing usage likely to be found on our members' sites. We ran tests against the following configurations:

• Plain mpm-prefork with mod_php and apc (as a standard to strive for)
• Both mpm-worker with suPHP and mpm-prefork with suPHP (how much of a difference does worker vs prefork make with a Drupal site?)
• mpm-worker/fcgid/suexec
• mpm-itk with mod_php

The tests were run with:

• 1 request (in the results, this test is labelled 1:1)
• 20 requests, 5 concurrent (20:5)
• 50 requests, 10 concurrent (50:10)
• 70 requests, 10 concurrent (70:10)

We originally tried, for the last test, 100 requests/25 concurrent but it caused ab to time out on the mpm-worker/fcgid/suexec configuration. We then lowered it to 75/15. Still timed out (load on our test server went way up). The tests paused 120 seconds between testing each environment to give the server load a chance to settle down. And we paused 30 seconds between each test (not enough time for the fcgid processes to be killed, but enough time for all the processes to hopefully complete their requests). The tests are scripted and the full results are available. You can also see the various vhost configurations (test.mouse, test.mouse.itk, test.mouse.suexec.fcgid), and the fcgid.conf. To summarize the report ... if you are running one web site on a server, there's little argument for using anything other than mod_php and mpm-prefork - the performance of this combination far exceeds any other combination. I'm scratching my head over the the vastly different conclusions reached by 2bits. They didn't publish their tests so I'm not sure exactly how to compare. Maybe they have a lot more static content? Or - maybe there's a mistake in our tests? Open to ideas. suPHP seems to be left in the dust compared with either fcgid/suexec or mpm-itk. Given that it's also less flexible, there's not much going for it. And lastly... fcgid/suexec and mpm-itk seem comparable. That was a surprise. We were expecting mpm-itk to perform better because it's running mod_php. Given the similarity in performance, fcgid/mpm-worker seems like a much more well-tested and well-supported approach for a shared server. One side note... originally, out of laziness, we tested against the default Drupal install.php page. Less static content and much less PHP processing. In these tests, with low loads, the results provide different numbers, but the comparisons between environment were similar ... except with the last test. With the 70 requests/10 concurrent test, fcgid/suexec dramatically out-performed everyone (by a factor or more than 2). That's probably because no new cgi processes had to be spawned and the effect of mod_php using APC was reduced because there was very little php code to process.

I've just uploaded git-dpm 0.3.0-1 packages to experimental. Apart from many bugfixes (which I will also take a look if I can make an 0.2.1 version targeting squeeze, though the freeze requirements tend to get tighter and tighter, so I may already be too late), the biggest improvement is the newly added git-dpm dch command to spawn dch and then extracts something to be used for the git commit message (I prefer to have more control over debian/changelog, so I prefer this way over the other direction).

I'm in hospital, taking an enforced rest from work and normal life.with much too much time to think. Luckily I have relatively slow 'Net access.

I've just spent about three weeks in a specialist orthopaedic hospital which is also a centre of excellence, a research and teaching hospital. Some lessons for FLOSS:

Research methods matter: open communication matters more

The hospital was apparently doing various research projects, some minor, some major: whether cooling a knee joint in surgery improved recovery and aftercare, whether complete pain relief and comprehensive analgesia stimulated recovery, trying to standardise on a pain score to measure subjective pain - " On a scale of 1-10 how does it feel?"

The knee joint research was being written up and would be shared worldwide - if it proved worthwhile, hundreds of patients could benefit. If it wasn't worthwhile, it would still be written up just as comprehensively with testing methodology, evidence from double-blind trials or whatever as appropriate to save someone else from having to re-invent the wheel.

The analgesia study sounded just like the application of a doctor or nurse's intuitive commn sense and experience - but it was being carried out scientifically and measured. also answering supplementary questions like "If pain is sometimes a helpful indicator of problems, how should you guage reactions when all pain is dulled?" Additionally, the ward had dispensations to use different and more flexible drug regimes to include the use of oral morphine derivatives. Again, everything was being noted, measured and shared.

Here are the Debian users who have submitted a request to appear in our website so far in this year. Please welcome them into our Project!

1dental.com: Commercial company from US.

A.S.D. Irishmassa American Football Team: Non-profit organization from Italy

Centro de Comunicaci n Cient fica, Universidad de Buenos Aires: University from Argentina.

Creppolo, crepes e piadine: Commercial company from Italy.

Eye Of The Beholder BBS: Non-profit organization from Spain.

IT Services, Medhurst Communications Ltd.: Commercial company from UK.

JSC VS: Commercial company from Russia.

King Abdulaziz City for Science and Technology (KACST): Academic section from Saudi Arabia

Lumut MARA Vocational Institute: Educational institution from Malaysia.

Madkom Ltd.: Commercial company from Poland.

Media-X Systems Inc.: Commercial company from Canada.

Microbiology laboratory. Science Faculty. UNAM: University from Mexico.

Nimbus Hookah Lounge: Commercial company from US.

Nomura Technical Management Office Ltd.: Commercial company from Japan.

oriented.net Web Hosting: Commercial company from Switzerland.

Parkyeri: Software development company from Turkey.

Pastoral da Crian a: Non-profit organization from Brazil.

XSoft Ltd.: Software development company from Bulgaria.

Ziehm Computer Repair: Commercial company from US.

And here are some of their reasons to choose Debian:

We have chosen Debian over other alternatives; because of its reliability and easiness of maintenance.

The decision was based on the selection of additional expandable components, availability of distribution, a regular cycle for issuing amendments, stability and support solutions for a wide range of equipment for a dedicated server solutions.

Stability is important for us.

The main reason for choosing Debian over other distributions is the package management, security, number of available packages, ease-of-use, stability, amount of quality documentation, and cost.

We choose Debian to easen the burden of system administration. Also because Debian has very well maintaned packages and for high security.

Debian is powerful and keeping originality of Linux.

We find the software that comes with Debian very useful and easy to use since we are not specialists in computer matters.

Debian is the most powerful Linux distribution! The best choice for our jobs, our servers and our lifestyle.

We chose Debian because it is stable, simple and fast to install and to use, and because it is Free Software.

We are proud to say that we use Debian/GNU Linux on more than 90% of our servers and this simply because of it s stability, security and great packaging system.

Debian offered the perfect balance between security, stability and ease of administration.

We chose Debian for its rock-solid stability, ease of installation and configuration, and awesome ability to not only run on old, second-hand hardware, but to turn that hardware into powerful production machines perfect for our needs. We also like Debian s Social Contract. None of the other options were as successful in this implementation.

It is refreshing and enjoyable to have an operating system which gives users easy access.

Thanks very much for the hard work, Webmaster Team.

When I am writing software sometimes XKCD is accurate!

Has anyone experience with using Google Apps?The server I'm currently using for my private email will be shut down soon, so I need to decide upon a future way of handling my personal email. I have low requirements, they key feature I need is a good spam filter; Google Apps can certainly provide this. Since I'm the single user, the free "standard" edition should be easily good enough.Obviously, since I don't have a reachable mail server myself anymore, I'd either have to rent a server just for my personal email, or use some hosted solution like Gmail, even if it is not open source.So, did you try it? Did you return to a different solution?

• Privacy issues of using Google
• Free service limitations (50 acounts, sufficient for me)
• Good spam filter
• Good Webmail
• High availability
• IMAP support (at least to some extend
• Android support - my next phone likely will be an Android device

Any points I'm missing? Email me: erich (a) debian orgUpdate: I have received a lot of feedback, much of which is very positive on Gmail/Google Apps for personal use.

Richard Baraniuk gave an interesting TED talk about Open Source Learning [1]. His project named Connexions which is dedicated to the purpose of creating Creative Commons free textbooks is a leader in this space [2]. He spoke about Catherine Schmidt-Jones who wrote 197 modules and 12 courses on music [3], that s a very significant amount of work! He also mentioned the translation of the work into other languages. I wonder how well the changes get merged back across the language divide. We have ongoing disputes in the free software community about whether various organisations do enough work to send patches back upstream, this seems likely to be more of a problem in situations where most of the upstream authors can t even understand the language in which the changes are written and when the changes involve something a lot more subtle than an change to an algorithm. This would be particularly difficult for Chinese and Japanese as those languages seem to lack quality automatic translation. He mentioned Teachers Without Borders [4] in passing. Obviously an organisation that wants to bring education to some of the poorer parts of the world can t have a curriculum that involves $250 of text books per year for a high school student (which was about what my parents paid when I was in year 12) or $500 of text books per year for a university student (which might be a low estimate for some courses as a single text can cost more than $120). Free content and on-demand printing (or viewing PDF files on a OLPC system) can dramatically lower the cost of education. It s widely believed that free content without the ability to remix is cultural imperialism. This is apparently one of the reasons that the connexions project is based on the Creative Commons Attribution license [5]. So anyone anywhere can translate it, make a derivative work, or collate parts of it with other work. I expect that another factor is the great lack of success of all the various schemes that involve people contributing content for a share of the profits, the profits just don t match the amount of work involved. Philanthropy and reputation seem to be the only suitable motivating factors for contributing to such projects. One of the stated benefits of the project is to have computer based content with live examples of equations. Sometimes it is possible to just look at an equation and know what it means, but often more explanation is required. The ability to click on an equation, plug in different values and have them automatically calculated and possibly graphed if appropriate can make things a lot easier. Even if the result is merely what would be provided by reading a text book and spending a few minutes with a scientific calculator the result should be a lot better in terms of learning as the time required to operate a calculator can break the student s concentration. Even better it s possible to have dynamic explanations tailored to the user s demand. To try this out I searched on Ohm s Law (something that seems to be unknown by many people on the Internet who claim to understand electricity). I was directed to an off-site page which used Flash to display a tutorial on Ohm s Law, the tutorial was quite good but it does seem to depart from the free content mission of the project to direct people off-site to proprietary content which uses a proprietary delivery system. I think that the Connexions project could do without links to sites such as college-cram.com. One of the most important features of the project is peer review lenses . The High Performance Computing Lens [6] has some good content and will be of interest to many people in the free software community but again it requires Flash. One final nit is the search engine which is slow and not very useful. A search for engine returned lots of hits about engineering which isn t useful if you want to learn about how engines work. But generally this is a great project, it seems to be doing a lot of good and it s got enough content to encourage other people and organisations to get involved. It would be good to get some text books about free software on there!

• [1] http://www.ted.com/talks/lang/eng/richard_baraniuk_on_open_source_learning.html
• [2] http://cnx.org/
• [3] http://cnx.org/member_profile/Catherine
• [4] http://www.teacherswithoutborders.org/
• [5] http://creativecommons.org/licenses/by/2.0/

Debian s testing distribution is where Debian developers prepare the next stable distribution. While this is still its main purpose, many users have adopted this version of Debian because it offers them a good trade-off between stability and freshness. But there are downsides to using this distribution and the Constantly Usable Testing (CUT) project aims to resolve those. This article will present the project and the challenges involved to make it happen. About Debian unstable & testing Debian unstable is the distribution where developers upload new versions of their packages. It happens frequently that some packages are not installable due to changes in other packages or due to transitions not yet completed. Debian testing, on the contrary, is managed by a tool that ensures the consistency of the whole distribution: it picks updates from unstable only if the package has been enough tested (10 days usually), if it s free of new release-critical bugs, if it s available on all supported architectures, and if it doesn t break any other package already present in testing. The Release Team (RT) controls this tool and provide hints to help it find a set of packages that can flow from unstable to testing. Those rules also ensure that the packages that flow into testing are reasonably free of show-stopper bugs (like a system that doesn t boot, or X that doesn t work at all). This makes it very attractive to users who like to regularly get new upstream versions of their software without dealing with the biggest problems associated to them. This is all very attractive, yet several Debian developers advise people to not use testing. Why is that? Known problems with testing Disappearing software The release team use this distribution to prepare the next stable release and from time to time they remove packages from it. Either because it s needed to ensure that other packages can migrate from unstable to testing, or because they have long-standing release-critical bugs without progress towards a resolution. It also happens that they remove packages on request of the maintainers because they believe that the current version of the software cannot be supported (security-wise) for 2 years or more. The security team also regularly issues such requests. Long delays for security and important fixes Despite the 10-day delay in unstable, there are always some annoying bugs (and security bugs are no exceptions) that are only discovered when the package already has migrated to testing. The maintainer might be quick to upload a fixed package in unstable, and might even raise the urgency to allow the package to migrate sooner, but if the packages gets entangled in a large ongoing transition, it will not migrate before the transition is completed. Sometimes it can take weeks for that to happen. This delay can be avoided by doing direct uploads to testing (through testing-proposed-updates) but this is almost never used, except during a freeze, where targeted bugfixes are the norm. Not always installable With testing evolving daily, updates sometimes break the last installation images available (in particular netboot images that get everything from the network). The debian-installer (d-i) packages are usually quickly fixed but they don t move to testing automatically because the new combination of d-i packages has not necessarily been validated yet. Colin Watson sums up the problem:

Getting new installer code into testing takes too long, and problems remain unfixed in testing for too long. [...] The problem with d-i development at the moment is more that we re very slow at producing new d-i *releases*. [...] Your choices right now are to work with stable (too old), testing (would be nice except for the way sometimes it breaks and then it tends to take a week to fix anything), unstable (breaks all the time).

CUT s history CUT finds its root in an old proposal by Joey Hess: it introduces the idea that the stable release is not Debian s sole product and that testing could become with some work a suitable choice for end-users. Nobody took on that work and there was no visible progress in the last 3 years. But recently Joey brought up CUT again on the debian-devel mailing list and Stefano Zacchiroli (the Debian project leader) challenged him to setup a BoF on CUT for Debconf10. It turned out to be one of the most heavily attended BoF (video recording is here), there is clearly a lot of interest in the topic. There s now a dedicated wiki and an Alioth project with a mailing list. The rest of this article tries to summarize the various options discussed and how they re supposed to address the problems identified. The ideas behind CUT Among all the ideas, there are two main approaches that have been discussed. The first is to regularly snapshot testing at points where it is known to work reasonably well (those snapshots would be named cut ). The second is to build an improved testing distribution tailored to the needs of users who want a working distribution with daily updates, its name would be rolling . Regular snapshots of testing There s general agreement that regular snapshots of testing are required: it s the only way to ensure that the generated installation media will continue to work until the next snapshot. If tests of the snapshot do not reveal any major problem, then it becomes the latest cut . For clarity, the official codename would be date based: e.g. cut-2010-09 would be the cut taken during September 2010. While the frequency has not been fixed yet, the goal is clearly to be on the aggressive side: at the very least every 6 months, but every month has been suggested as well. In order to reach a decision, many aspects have to be balanced. One of them (and possibly the most important) is the security support. Given that the security team is already overworked, it s difficult to put more work on their shoulders by declaring that cuts will be supported like any stable release. No official security support sounds bad but it s not necessarily so problematic as one might imagine. Testing s security record is generally better than stable s one (see the security tracker) because fixes flow in naturally with new upstream versions. Stable still get fixes for very important security issues sooner than testing, but on the whole there are less known security-related problems in testing than in stable. Since it s only a question of time until the fixed version comes naturally from upstream, more frequent cut releases means that users get security fixes sooner. But Stefan Fritsch, who used to be involved in the Debian testing security team, has also experienced the downside for anyone who tries to contribute security updates:

The updates to testing-security usually stay useful only for a few weeks, until a fixed version migrates from unstable. In stable, the updates stay around for a few years, which gives a higher motivation to spend time on preparing them.

So if it s difficult to form a dedicated security team, the work of providing security updates comes back to the package maintainer. They are usually quite quick to upload fixed packages in unstable but tend to not monitor whether the packages migrate to testing. They can t be blamed because testing was created to prepare the next stable release and there is thus no urgency to get the fix in as long as it makes it before the release. CUT can help in this regard precisely because it changes this assumption: there will be users of the testing packages and they deserve to get security fixes much like the stable users. Another aspect to consider when picking a release frequency is the amount of associated work that comes with any official release: testing upgrades from the previous version, writing release notes and preparing installation images. It seems difficult to do this every month. With this frequency it s also impossible to have a new major kernel release for each cut (since they tend to come out only every 2 to 3 months) and the new hardware support that it brings is something worthwhile to many users. In summary, regular snapshots address the not always installable problem and changes the perception of maintainers towards testing, so that hopefully they care more of security updates in that distribution (and in cuts). But they do not solve the problem of disappearing packages. Something else is needed to fix that problem. A new rolling distribution? Lucas Nussbaum pointed out that regular snapshots of Debian is not really a new concept:

How would this differentiate from other distributions doing 6-month release cycles, and in particular Ubuntu, which can already be seen as Debian snapshots (+ added value)?

In Lucas s eyes, CUT becomes interesting if it can provide a rolling distribution (like testing) with a constant flux of new upstream releases . For him, that would be something quite unique in the Free Software world . The snapshots would be used as starting point for the initial installation, but the installed system would point to the rolling distribution and users would then upgrade as often as they want. In this scenario, security support for the snapshots is not so important, what matters is the state of the rolling distribution. If testing were used as the rolling distribution, the problem of disappearing packages would not be fixed. That s why there have been discussions of introducing a new distribution named rolling that would work like testing but with adapted rules, and the cuts would then be snapshots of rolling instead of testing. The basic proposal is to make a copy of testing and to re-add the packages which have been removed because they are not suited for a long term release while they are perfectly acceptable for a constantly updated release (the most recent example being Chromium). Then it s possible to go one step further: during freeze, testing is no longer automatically updated which makes it inappropriate to feed the rolling distribution. That s why rolling would be reconfigured to grab updates from unstable (but using the same rules than testing). Given the frequent releases, it s likely that only a subset of architectures would be officially supported. This is not a real problem because the users who want bleeding edge software tends to be desktop users on mainly i386/amd64 (and maybe armel for tablets and similar mobile products). This choice if made opens up the door to even more possibilities: if rolling is configured exactly like testing but with only a subset of the architectures, it s likely that some packages migrate to rolling before testing when non-mainstream architectures are lagging in terms of auto-building (or have toolchain problems). While being ahead of testing can be positive for the users, it s also problematic on several levels. First, managing rolling becomes much more complicated because the transition management work done by the release team can t be reused as-is. Then it introduces competition between both distributions which can make it more difficult to get a stable release out, for example if maintainers stop caring of the migration to testing once the migration to rolling has been completed. The rolling distribution is certainly a good idea but the rules governing it must be designed to avoid any conflict with the process of releasing a stable distribution. Lastly, the mere existence of rolling would finally fix the marketing problem plaguing testing: the name rolling does not suggest that the software is not yet ready for prime time. Conclusion Whether CUT will be implemented remains to be seen, but it s off for a good start: ftpmaster Joerg Jaspert said that the new archive server can cope with a new distribution, and there s now a proposal shaping up. The project might start quickly: there is already an implementation plan for the snapshot side of the project. The rolling distribution can always be introduced later, once it is ready. Both approaches can complement each other and provide something useful to different kind of users. The global proposal is certainly appealing: it would address the concerns of obsolescence of Debian s stable release by making intermediary releases. Anyone needing something more recent for hardware support can start by installing a cut and follow the subsequent releases until the next stable version. And users who always want the latest version of every software could use rolling after having installed a cut. From a user point of view, there are similarities with the mix of usual and long term releases of Ubuntu. But from the development side, the process followed would be quite different, and the constraints imposed by having a constantly usable distribution are stronger: any wide-scale change must be designed in a way that it can happen progressively in a transparent manner for the user.

This article was first published in Linux Weekly News. If you want to see more articles like this, join Flattr and click on the flattr button below every article that you like.

21 comments Support my work

I've recently returned from ICFP 2010 (International conference on functional programming). This was the second time I went there, and like last year, it was a very interesting conference. This year, I've even managed to do one keysigning with a fellow Debian Developer (hello Sylvain!) While there were many talks, I only mention a few key ones, that I have found very cool. Note: I'm a layman in this field. The text below is my own interpretation of the talks and contains many errors and wrong facts corrections are welcome! Workshop on ML I wasn't present at this workshop, but other people pointed me to this very cool talk: Mirage project The abstract, to be found at http://www.cs.rit.edu/~mtf/ml2010/program.html#Program (search for Mirage), says:

Our framework, dubbed Mirage, extends the Objective Caml language with I/O extensions and a custom runtime to emit binaries that execute as a paravirtual operating system directly under Xen. The custom run-time is significantly simpler than a general-purpose operating system, with a static single-process 64-bit address space, and "zero-copy" I/O that works directly with the OCaml garbage collector. As a result, Mirage applications exhibit significant performance speedups for I/O and memory handling versus the same code running under Linux/Xen.

It's extremely interesting, IMHO. You write your code as a more-or-less normal OCaml program, but you compile it with this special tool-chain, and you boot it directly on ton of Xen, as a domU, without the intermediate OS. The project Git repo is at http://github.com/avsm/mirage, and that links a bit older presentation (http://anil.recoil.org/papers/2010-hotcloud-lamp.pdf) which IMHO is a required read for anyone interested in cloud-related stuff. ICFP proper ICFP, the main conference, is as usual, full of math, designed to scare people away. Err, I mean, designed to tell about new developments in this field. The papers are gathered in this reddit post: http://www.reddit.com/r/haskell/comments/dji4v/papers_from_icfp_2010/. XenServer The paper http://anil.recoil.org/papers/2010-icfp-xen.pdf describes the XenServer project, and more specifically the management interface, which is written entirely in OCaml, by a team of ~10 people. They also have a bit of bin-packing code (http://xenbits.xen.org/xapi/xen-api.hg?file/95b9e4f1b9dd/ocaml/xapi/binpack.ml), but much simpler than the ganeti-htools project on which I'm working (no balancing or similar). On the other hand, it is interesting to see that they committed themselves to write the entire management suite in OCaml, as opposed to our Python plus just a few bits of Haskell. I envy them, very much so The fun tidbit was how stable they found the OCaml tool-chain. In three years, they had just one bug related to this (the tool-chain), a segfault due to wrong formatting code in the handling path of a seldom-triggered Python exception, which led to a %s to be translated as is to C code, which meant printf got the wrong specifiers. So in an OCaml project, a serious bug was due to lack of type checking in string formatting in a Python component Overall I have found interesting parallels between their experience and my own with Haskell; from is this language/tool-chain stable enough for use to concerns about using FP for a significant project (but of course, orders of magnitude difference between their case and mine). Super-compilation There were two papers on this topic; one at ICFP proper and one at the Haskell symposium. If I understood things right, super-compilation is the opposite of lazy evaluation. But let's start a bit back. First, programs written in a functional languages contain many pure function. Pure functions, being side-effect-free, exhibit many interesting properties, and one of them is that they permit decoupling of the computation of a value from the declaration of such computation. One technique derived from this is lazy computations. Lazy computation means that even though you write y = sin(x) + 1, the actual call to sin and the increment is not actually done until you need the actual value of y (not just for storing it in y, that is). Super-compilation is the opposite of this. While laziness postpones doing a computation to as late as possible, in the hope it won't be needed at all, super-compilation runs the actual computation way ahead of time, even before the program starts, that is at compile time. Since programs use input data at runtime, which is not known at compile time, the trick is to pre-compute as much as possible at this stage, given incomplete information; and the papers showed that it's possible to pre-compute much more than you'd usually believe. Optimising compilers already do some form of CAF (constant applicative form) optimisation, but the techniques presented in both papers are going beyond simple CAFs. The details are in the papers, and are complicated, but one simple example is converting this:

ones = map inc [1..]

(note that is an infinite list, and you don't want to fully compute that at any time, neither at compile nor at runtime), into:

ones = [2..]

The details on how this is done depend on the exact super-compilation implementation, and not all implementations can do the same optimisations, but in the presentations at this ICFP the idea was to split things into static/non-static parts, and build on that, or to do partial evaluation, etc. The result is that super-compilation shows significant increases in speed (in some synthetic benchmarks up to 80%), with a few benchmarks not being sped up at all, but most are. The downside is an explosion in code size, as many more things are pre-computed. Still, as a research topic, this is very interesting. My own paper Surprisingly, it seems I survived giving my talk (just an experience report, not actually fancy stuff like the rest). Either the people actually found it interesting (less likely), or they were very polite (more probably). Anyway, the paper is on the papers page (shameless self-promotion) (if you care). Haskell symposium This is co-located with ICFP, and is more practical than ICFP (which is, as I said, way too much math). The papers are again available on reddit: http://www.reddit.com/r/haskell/comments/dkzn4/papers_from_the_haskell_symposium_2010/. Orc: Concurrent Orchestration in Haskell How to easily write code for synchronising concurrent work? As an example, how complicated is to solve this problem:

[from the Orc literature] we might wish to contact two airlines simultaneously seeking price quotes. If either quote comes back below a threshold price, say $300, then let's buy a ticket immediately. On the other hand if both quotes exceed the threshold, then buy the cheapest ticket. Additionally, buy a ticket if the other airline does not give a timely quote, or notify the user if neither airline provides a timely quote.

Now, in my experience, this would require manual management and synchronisation of threads, and for example in Python the early abort of querying the price from airline B if airline A returned a good price would be non-trivial; in addition, adding two extra conditions would be again not a simple composition of rules. In Orc however, it takes all of 10 lines of code to do this, safely. I won't copy the code here, see the paper for details, or the API docs. And the best part is, this is just a cabal install orc away! I'll be very interested to see how I can use this in practice, as it is seems indeed a very powerful library. Nikola: Embedding Compiled GPU Functions in Haskell Interesting project. Using almost 100% Haskell code, and it marshalls the data to/from the GPU; in the example given, the performance is as good as hand-written CUDA code of course, this doesn't happen in all cases. It it's interesting how they manage to compile the GPU-related code to actual object code at Haskell compile time, which makes it transparent for the user. Given the prevalence of high-speed GPUs, this could be a nice way to utilise them without learning yet another language (or two, since NVidia and ATI have different stacks). They've also developed some interesting techniques (e.g. manual control of sharing, to prevent inlining) as part of their project. Scalable Event Handling for GHC This is a paper co-authored by Johan Tibell of Google Switzerland and Bryan O'Sullivan of Real World Haskell fame. Very interesting serving 20K QPS from a Haskell process while having 16K+ idle clients connected:

[ ] the performance of the epoll back end does not begin to fall off significantly until we have 50,000 idle connections open.

The talk started with the idea that we're on the path to serving 10 clients from a single machine, and how to improve Haskell's I/O manager to achieve (almost) 10 concurrent clients. The slides (which I don't have a link for) were a bit more cool on numbers than the paper. Oh, and this will come with GHC 7.0, for free An LLVM Backend For GHC The eternal fight to get better than the C backend in other words, to not have to translate Haskell to C It is interesting to see how many languages target LLVM nowadays, as opposed to plain old C via gcc. A fun trivia item was that, according to the author, there are many features in LLVM that are not actually used by 90% of the languages targeting it (e.g. the garbage collector) (if I heard correctly in the talk). The results are interesting, although not too much overall speedup is gained; this seems to be due to the hard-to-optimise Cmm intermediate language inside GHC. In any case, a few benchmarks get more than 10% speedup, even in the current form. This will also come in GHC 7.0, although not enabled by default; probably needs to mature some more Conclusion I've enjoyed this conference a lot, and I've learned a lot from it. Given that many of the talks are about research topics, it's harder to apply them in day-to-day work, but I think the pay-offs will be long term rather than immediate.

Recently, Google started changing the web in some interesting ways. A lot by contributing code. It started with projects such as the Chrome web browser, which quickly achieved a market shares over 10%, which is more than Apple Safari and Opera together.And the good side, Chrome did not actually add "yet another incompatible platform", but by being closely related to Safari (or more precisely WebKit), it is not that different.Other stuff that belongs into this category is the Google Web Toolkit and the closure compiler (which essentially is a JavaScript code checker and optimizer).But recently, Google goes a step further. They started tackling internet file formats. First they proposed a new Video codec known as WebM or VP8. Now they extracted an image format out of this codec, known as WebP.There is some serious doubt on the quality, the images do appear to be a bit blurry and lose details. I concur with JPEG even being a bit better, despite the block artefacts. The blurry results of WebP are not really convincing.The h.264 based image looks best to me, but H.264 is patent-encumbered for all I know, so we might not see this new format in wide use until 2028. This already ruled out other image formats such as JBIG or FIF. Or look at audio formats: we're still using MP3 everywhere, because everyone tries to push another patent-encumbered audio format that the others are not going to adopt. So MP3 remains the only thing widely accepted (Ogg Vorbis is nice, but it looks as if companies such as Apple are not going to adopt it, probably because they have interest in other formats. Maybe the Ogg Vorbis support in Chrome will help here on the long run).In particular, why would you pay royalties for a tiny bit of image quality, given the constantly sinking prices of bandwidth? The H.264 patents may make a difference on video data, they probably won't pay of for still images.Given that WebM is sometimes claimed to be a simplified H.264 (minus the patent issues?), WebP probably is as good as we can get to the H.264 based quality without running into the same patent issues?But enough on the image data. There is one thing, I'd really like to see Google change on the web. The worst thing about the web is called JavaScript (or more accurately, ECMAScript). There is just so much wrong with this language that we really need to replace it ASAP.Some things that are wrong with JavaScript:

• The syntax is about as crazy as Perl, anything could be valid code.
• There a lot of syntax quirks and oddities. Try to test for a variable storing an Array ...
• It lacks any reasonable Standard Library - which is a key strength of modern languages such as Java, Python, C# - and which is why everbody is using one of the thousand JavaScript frameworks such as Dojo, Mojikit, Google Closure Library, Prototype, jQuery, MooTools, Yahoo UI Library, ...
• Due to the lack of standard functionality, the JavaScript development cost is a lot higher than needed, development time is bad for the triviality of most JavaScript tasks, and the file size is even worse than JPEG vs. WebP ...

So my biggest wishlist item of Google to contribute to the Web via Chrome and Firefox is a new web scripting language. As much as I love python, the whitespace syntax of Python doesn't work for the web (we need to be able to store complex statements in HTML attributes without newlines), so we'll have to look for something else. Maybe Ruby will work?