Award winning code Me and Yuwei had a fun day at hhhmcr (#hhhmcr) and even managed to put together a prototype that won the first prize \o/ We played with the gmp24 dataset kindly extracted from Twitter by Michael Brunton-Spall of the Guardian into a convenient JSON dataset. The idea was to find ways of making it easier to look at the data and making sense of it. This is the story of what we did, including the code we wrote. The original dataset has several JSON files, so the first task was to put them all together:

#!/usr/bin/python
# Merge the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import os
res = []
for f in os.listdir("."):
    if not f.startswith("gmp24"): continue
    data = open(f).read().strip()
    if data == "[]": continue
    parsed = simplejson.loads(data)
    res.extend(parsed)
print simplejson.dumps(res)

The results however were not ordered by date, as GMP had to use several accounts to twit because Twitter was putting Greather Manchester Police into jail for generating too much traffic. There would be quite a bit to write about that, but let's stick to our work. Here is code to sort the JSON data by time:

#!/usr/bin/python
# Sort the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import sys
import datetime as dt
all_recs = simplejson.load(sys.stdin)
all_recs.sort(key=lambda x: dt.datetime.strptime(x["created_at"], "%a %b %d %H:%M:%S +0000 %Y"))
simplejson.dump(all_recs, sys.stdout)

I then wanted to play with Tf-idf for extracting the most important words of every tweet:

#!/usr/bin/python
# tfifd - Annotate JSON elements with Tf-idf extracted keywords
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import sys, math
import simplejson
import re
# Read all the twits
records = simplejson.load(sys.stdin)
# All the twits by ID
byid = dict(((x["id"], x) for x in records))
# Stopwords we ignore
stopwords = set(["by", "it", "and", "of", "in", "a", "to"])
# Tokenising engine
re_num = re.compile(r"^\d+$")
re_word = re.compile(r"(\w+)")
def tokenise(tweet):
    "Extract tokens from a tweet"
    for tok in tweet["text"].split():
        tok = tok.strip().lower()
        if re_num.match(tok): continue
        mo = re_word.match(tok)
        if not mo: continue
        if mo.group(1) in stopwords: continue
        yield mo.group(1)
# Extract tokens from tweets
tokenised = dict(((x["id"], list(tokenise(x))) for x in records))
# Aggregate token counts
aggregated =  
for d in byid.iterkeys():
    for t in tokenised[d]:
        if t in aggregated:
            aggregated[t] += 1
        else:
            aggregated[t] = 1
def tfidf(doc, tok):
    "Compute TFIDF score of a token in a document"
    return doc.count(tok) * math.log(float(len(byid)) / aggregated[tok])
# Annotate tweets with keywords
res = []
for name, tweet in byid.iteritems():
    doc = tokenised[name]
    keywords = sorted(set(doc), key=lambda tok: tfidf(doc, tok), reverse=True)[:5]
    tweet["keywords"] = keywords
    res.append(tweet)
simplejson.dump(res, sys.stdout)

I thought this was producing a nice summary of every tweet but nobody was particularly interested, so we moved on to adding categories to tweet. Thanks to Yuwei who put together some useful keyword sets, we managed to annotate each tweet with a place name (i.e. "Stockport"), a social place name (i.e. "pub", "bank") and a social category (i.e. "man", "woman", "landlord"...) The code is simple; the biggest work in it was the dictionary of keywords:

#!/usr/bin/python
# categorise - Annotate JSON elements with categories
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
# Copyright (C) 2010  Yuwei Lin <yuwei@ylin.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import sys, math
import simplejson
import re
# Electoral wards from http://en.wikipedia.org/wiki/List_of_electoral_wards_in_Greater_Manchester
placenames = ["Altrincham", "Sale West",
"Altrincham", "Ashton upon Mersey", "Bowdon", "Broadheath", "Hale Barns", "Hale Central", "St Mary", "Timperley", "Village",
"Ashton-under-Lyne",
"Ashton Hurst", "Ashton St Michael", "Ashton Waterloo", "Droylsden East", "Droylsden West", "Failsworth East", "Failsworth West", "St Peter",
"Blackley", "Broughton",
"Broughton", "Charlestown", "Cheetham", "Crumpsall", "Harpurhey", "Higher Blackley", "Kersal",
"Bolton North East",
"Astley Bridge", "Bradshaw", "Breightmet", "Bromley Cross", "Crompton", "Halliwell", "Tonge with the Haulgh",
"Bolton South East",
"Farnworth", "Great Lever", "Harper Green", "Hulton", "Kearsley", "Little Lever", "Darcy Lever", "Rumworth",
"Bolton West",
"Atherton", "Heaton", "Lostock", "Horwich", "Blackrod", "Horwich North East", "Smithills", "Westhoughton North", "Chew Moor", "Westhoughton South",
"Bury North",
"Church", "East", "Elton", "Moorside", "North Manor", "Ramsbottom", "Redvales", "Tottington",
"Bury South",
"Besses", "Holyrood", "Pilkington Park", "Radcliffe East", "Radcliffe North", "Radcliffe West", "St Mary", "Sedgley", "Unsworth",
"Cheadle",
"Bramhall North", "Bramhall South", "Cheadle", "Gatley", "Cheadle Hulme North", "Cheadle Hulme South", "Heald Green", "Stepping Hill",
"Denton", "Reddish",
"Audenshaw", "Denton North East", "Denton South", "Denton West", "Dukinfield", "Reddish North", "Reddish South",
"Hazel Grove",
"Bredbury", "Woodley", "Bredbury Green", "Romiley", "Hazel Grove", "Marple North", "Marple South", "Offerton",
"Heywood", "Middleton",
"Bamford", "Castleton", "East Middleton", "Hopwood Hall", "Norden", "North Heywood", "North Middleton", "South Middleton", "West Heywood", "West Middleton",
"Leigh",
"Astley Mosley Common", "Atherleigh", "Golborne", "Lowton West", "Leigh East", "Leigh South", "Leigh West", "Lowton East", "Tyldesley",
"Makerfield",
"Abram", "Ashton", "Bryn", "Hindley", "Hindley Green", "Orrell", "Winstanley", "Worsley Mesnes",
"Manchester Central",
"Ancoats", "Clayton", "Ardwick", "Bradford", "City Centre", "Hulme", "Miles Platting", "Newton Heath", "Moss Side", "Moston",
"Manchester", "Gorton",
"Fallowfield", "Gorton North", "Gorton South", "Levenshulme", "Longsight", "Rusholme", "Whalley Range",
"Manchester", "Withington",
"Burnage", "Chorlton", "Chorlton Park", "Didsbury East", "Didsbury West", "Old Moat", "Withington",
"Oldham East", "Saddleworth",
"Alexandra", "Crompton", "Saddleworth North", "Saddleworth South", "Saddleworth West", "Lees", "St James", "St Mary", "Shaw", "Waterhead",
"Oldham West", "Royton",
"Chadderton Central", "Chadderton North", "Chadderton South", "Coldhurst", "Hollinwood", "Medlock Vale", "Royton North", "Royton South", "Werneth",
"Rochdale",
"Balderstone", "Kirkholt", "Central Rochdale", "Healey", "Kingsway", "Littleborough Lakeside", "Milkstone", "Deeplish", "Milnrow", "Newhey", "Smallbridge", "Firgrove", "Spotland", "Falinge", "Wardle", "West Littleborough",
"Salford", "Eccles",
"Claremont", "Eccles", "Irwell Riverside", "Langworthy", "Ordsall", "Pendlebury", "Swinton North", "Swinton South", "Weaste", "Seedley",
"Stalybridge", "Hyde",
"Dukinfield Stalybridge", "Hyde Godley", "Hyde Newton", "Hyde Werneth", "Longdendale", "Mossley", "Stalybridge North", "Stalybridge South",
"Stockport",
"Brinnington", "Central", "Davenport", "Cale Green", "Edgeley", "Cheadle Heath", "Heatons North", "Heatons South", "Manor",
"Stretford", "Urmston",
"Bucklow-St Martins", "Clifford", "Davyhulme East", "Davyhulme West", "Flixton", "Gorse Hill", "Longford", "Stretford", "Urmston",
"Wigan",
"Aspull New Springs Whelley", "Douglas", "Ince", "Pemberton", "Shevington with Lower Ground", "Standish with Langtree", "Wigan Central", "Wigan West",
"Worsley", "Eccles South",
"Barton", "Boothstown", "Ellenbrook", "Cadishead", "Irlam", "Little Hulton", "Walkden North", "Walkden South", "Winton", "Worsley",
"Wythenshawe", "Sale East",
"Baguley", "Brooklands", "Northenden", "Priory", "Sale Moor", "Sharston", "Woodhouse Park"]
# Manual coding from Yuwei
placenames.extend(["City centre", "Tameside", "Oldham", "Bury", "Bolton",
"Trafford", "Pendleton", "New Moston", "Denton", "Eccles", "Leigh", "Benchill",
"Prestwich", "Sale", "Kearsley", ])
placenames.extend(["Trafford", "Bolton", "Stockport", "Levenshulme", "Gorton",
"Tameside", "Blackley", "City centre", "Airport", "South Manchester",
"Rochdale", "Chorlton", "Uppermill", "Castleton", "Stalybridge", "Ashton",
"Chadderton", "Bury", "Ancoats", "Whalley Range", "West Yorkshire",
"Fallowfield", "New Moston", "Denton", "Stretford", "Eccles", "Pendleton",
"Leigh", "Altrincham", "Sale", "Prestwich", "Kearsley", "Hulme", "Withington",
"Moss Side", "Milnrow", "outskirt of Manchester City Centre", "Newton Heath",
"Wythenshawe", "Mancunian Way", "M60", "A6", "Droylesden", "M56", "Timperley",
"Higher Ince", "Clayton", "Higher Blackley", "Lowton", "Droylsden",
"Partington", "Cheetham Hill", "Benchill", "Longsight", "Didsbury",
"Westhoughton"])
# Social categories from Yuwei
soccat = ["man", "woman", "men", "women", "youth", "teenager", "elderly",
"patient", "taxi driver", "neighbour", "male", "tenant", "landlord", "child",
"children", "immigrant", "female", "workmen", "boy", "girl", "foster parents",
"next of kin"]
for i in range(100):
    soccat.append("%d-year-old" % i)
    soccat.append("%d-years-old" % i)
# Types of social locations from Yuwei
socloc = ["car park", "park", "pub", "club", "shop", "premises", "bus stop",
"property", "credit card", "supermarket", "garden", "phone box", "theatre",
"toilet", "building site", "Crown court", "hard shoulder", "telephone kiosk",
"hotel", "restaurant", "cafe", "petrol station", "bank", "school",
"university"]
extras =   "placename": placenames, "soccat": soccat, "socloc": socloc  
# Normalise keyword lists
for k, v in extras.iteritems():
    # Remove duplicates
    v = list(set(v))
    # Sort by length
    v.sort(key=lambda x:len(x), reverse=True)
# Add keywords
def add_categories(tweet):
    text = tweet["text"].lower()
    for field, categories in extras.iteritems():
        for cat in categories:
            if cat.lower() in text:
                tweet[field] = cat
                break
    return tweet
# Read all the twits
records = (add_categories(x) for x in simplejson.load(sys.stdin))
simplejson.dump(list(records), sys.stdout)

All these scripts form a nice processing chain: each script takes a list of JSON records, adds some bit and passes it on. In order to see what we have so far, here is a simple script to convert the JSON twits to CSV so they can be viewed in a spreadsheet:

#!/usr/bin/python
# Convert the JSON twits to CSV
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import sys
import csv
rows = ["id", "created_at", "text", "keywords", "placename"]
writer = csv.writer(sys.stdout)
for rec in simplejson.load(sys.stdin):
    rec["keywords"] = " ".join(rec["keywords"])
    rec["placename"] = rec.get("placename", "")
    writer.writerow([rec[row] for row in rows])

At this point we were coming up with lots of questions: "were there more reports on women or men?", "which place had most incidents?", "what were the incidents involving animals?"... Time to bring Xapian into play. This script reads all the JSON tweets and builds a Xapian index with them:

#!/usr/bin/python
# toxapian - Index JSON tweets in Xapian
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import simplejson
import sys
import os, os.path
import xapian
DBNAME = sys.argv[1]
db = xapian.WritableDatabase(DBNAME, xapian.DB_CREATE_OR_OPEN)
stemmer = xapian.Stem("english")
indexer = xapian.TermGenerator()
indexer.set_stemmer(stemmer)
indexer.set_database(db)
data = simplejson.load(sys.stdin)
for rec in data:
    doc = xapian.Document()
    doc.set_data(str(rec["id"]))
    indexer.set_document(doc)
    indexer.index_text_without_positions(rec["text"])
    # Index categories as categories
    if "placename" in rec:
        doc.add_boolean_term("XP" + rec["placename"].lower())
    if "soccat" in rec:
        doc.add_boolean_term("XS" + rec["soccat"].lower())
    if "socloc" in rec:
        doc.add_boolean_term("XL" + rec["socloc"].lower())
    db.add_document(doc)
db.flush()
# Also save the whole dataset so we know where to find it later if we want to
# show the details of an entry
simplejson.dump(data, open(os.path.join(DBNAME, "all.json"), "w"))

And this is a simple command line tool to query to the database:

#!/usr/bin/python
# xgrep - Command line tool to query the GMP24 tweet Xapian database
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import simplejson
import sys
import os, os.path
import xapian
DBNAME = sys.argv[1]
db = xapian.Database(DBNAME)
stem = xapian.Stem("english")
qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
qp.add_boolean_prefix("place", "XP")
qp.add_boolean_prefix("soc", "XS")
qp.add_boolean_prefix("loc", "XL")
query = qp.parse_query(sys.argv[2],
    xapian.QueryParser.FLAG_BOOLEAN  
    xapian.QueryParser.FLAG_LOVEHATE  
    xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE  
    xapian.QueryParser.FLAG_WILDCARD  
    xapian.QueryParser.FLAG_PURE_NOT  
    xapian.QueryParser.FLAG_SPELLING_CORRECTION  
    xapian.QueryParser.FLAG_AUTO_SYNONYMS)
enquire = xapian.Enquire(db)
enquire.set_query(query)
count = 40
matches = enquire.get_mset(0, count)
estimated = matches.get_matches_estimated()
print "%d/%d results" % (matches.size(), estimated)
data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))
for m in matches:
    rec = data[m.document.get_data()]
    print rec["text"]
print "%d/%d results" % (matches.size(), matches.get_matches_estimated())
total = db.get_doccount()
estimated = matches.get_matches_estimated()
print "%d results over %d documents, %d%%" % (estimated, total, estimated * 100 / total)

Neat! Now that we have a proper index that supports all sort of cool things, like stemming, tag clouds, full text search with complex queries, lookup of similar documents, suggest keywords and so on, it was just fair to put together a web service to share it with other people at the event. It helped that I had already written similar code for apt-xapian-index and dde before. Here is the server, quickly built on bottle. The very last line starts the server and it is where you can configure the listening interface and port.

#!/usr/bin/python
# xserve - Make the GMP24 tweet Xapian database available on the web
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import bottle
from bottle import route, post
from cStringIO import StringIO
import cPickle as pickle
import simplejson
import sys
import os, os.path
import xapian
import urllib
import math
bottle.debug(True)
DBNAME = sys.argv[1]
QUERYLOG = os.path.join(DBNAME, "queries.txt")
data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))
prefixes =   "place": "XP", "soc": "XS", "loc": "XL"  
prefix_desc =   "place": "Place name", "soc": "Social category", "loc": "Social location"  
db = xapian.Database(DBNAME)
stem = xapian.Stem("english")
qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
for k, v in prefixes.iteritems():
    qp.add_boolean_prefix(k, v)
def make_query(qstring):
    return qp.parse_query(qstring,
        xapian.QueryParser.FLAG_BOOLEAN  
        xapian.QueryParser.FLAG_LOVEHATE  
        xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE  
        xapian.QueryParser.FLAG_WILDCARD  
        xapian.QueryParser.FLAG_PURE_NOT  
        xapian.QueryParser.FLAG_SPELLING_CORRECTION  
        xapian.QueryParser.FLAG_AUTO_SYNONYMS)
@route("/")
def index():
    query = urllib.unquote_plus(bottle.request.GET.get("q", ""))
    out = StringIO()
    print >>out, '''
<html>
<head>
<title>Query</title>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
<script type="text/javascript">
$(function() 
    $("#queryfield")[0].focus()
 )
</script>
</head>
<body>
<h1>Search</h1>
<form method="POST" action="/query">
Keywords: <input type="text" name="query" value="%s" id="queryfield">
<input type="submit">
<a href="http://xapian.org/docs/queryparser.html">Help</a>
</form>''' % query
    print >>out, '''
<p>Example: "car place:wigan"</p>

<p>Available prefixes:</p>

<ul>
'''
    for pfx in prefixes.keys():
        print >>out, "<li><a href='/catinfo/%s'>%s - %s</a></li>" % (pfx, pfx, prefix_desc[pfx])
    print >>out, '''
</ul>
'''
    oldqueries = []
    if os.path.exists(QUERYLOG):
        total = db.get_doccount()
        fd = open(QUERYLOG, "r")
        while True:
            try:
                q = pickle.load(fd)
            except EOFError:
                break
            oldqueries.append(q)
        fd.close()
        def print_query(q):
            count = q["count"]
            print >>out, "<li><a href='/query?query=%s'>%s (%d/%d %.2f%%)</a></li>" % (urllib.quote_plus(q["q"]), q["q"], count, total, count * 100.0 / total)
        print >>out, "<p>Last 10 queries:</p><ul>"
        for q in oldqueries[:-10:-1]:
            print_query(q)
        print >>out, "</ul>"
        # Remove duplicates
        oldqueries = dict(((x["q"], x) for x in oldqueries)).values()
        print >>out, "<table>"
        print >>out, "<tr><th>10 queries with most results</th><th>10 queries with least results</th></tr>"
        print >>out, "<tr><td>"
        print >>out, "<ul>"
        oldqueries.sort(key=lambda x:x["count"], reverse=True)
        for q in oldqueries[:10]:
            print_query(q)
        print >>out, "</ul>"
        print >>out, "</td><td>"
        print >>out, "<ul>"
        nonempty = [x for x in oldqueries if x["count"] > 0]
        nonempty.sort(key=lambda x:x["count"])
        for q in nonempty[:10]:
            print_query(q)
        print >>out, "</ul>"
        print >>out, "</td></tr>"
        print >>out, "</table>"
    print >>out, '''
</body>
</html>'''
    return out.getvalue()
@route("/query")
@route("/query/")
@post("/query")
@post("/query/")
def query():
    query = bottle.request.POST.get("query", bottle.request.GET.get("query", ""))
    enquire = xapian.Enquire(db)
    enquire.set_query(make_query(query))
    count = 40
    matches = enquire.get_mset(0, count)
    estimated = matches.get_matches_estimated()
    total = db.get_doccount()
    out = StringIO()
    print >>out, '''
<html>
<head><title>Results</title></head>
<body>
<h1>Results for "<b>%s</b>"</h1>
''' % query
    if estimated == 0:
        print >>out, "No results found."
    else:
        # Give as results the first 30 documents; also use them as the key
        # ones to use to compute relevant terms
        rset = xapian.RSet()
        for m in enquire.get_mset(0, 30):
            rset.add_document(m.document.get_docid())
        # Compute the tag cloud
        class NonTagFilter(xapian.ExpandDecider):
            def __call__(self, term):
                return not term[0].isupper() and not term[0].isdigit()
        cloud = []
        maxscore = None
        for res in enquire.get_eset(40, rset, NonTagFilter()):
            # Normalise the score in the interval [0, 1]
            weight = math.log(res.weight)
            if maxscore == None: maxscore = weight
            tag = res.term
            cloud.append([tag, float(weight) / maxscore])
        max_weight = cloud[0][1]
        min_weight = cloud[-1][1]
        cloud.sort(key=lambda x:x[0])
        def mklink(query, term):
            return "/query?query=%s" % urllib.quote_plus(query + " and " + term)
        print >>out, "<h2>Tag cloud</h2>"
        print >>out, "<blockquote>"
        for term, weight in cloud:
            size = 100 + 100.0 * (weight - min_weight) / (max_weight - min_weight)
            print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(query, term), size, term)
        print >>out, "</blockquote>"
        print >>out, "<h2>Results</h2>"
        print >>out, "<p><a href='/'>Search again</a></p>"
        print >>out, "<p>%d results over %d documents, %.2f%%</p>" % (estimated, total, estimated * 100.0 / total)
        print >>out, "<p>%d/%d results</p>" % (matches.size(), estimated)
        print >>out, "<ul>"
        for m in matches:
            rec = data[m.document.get_data()]
            print >>out, "<li><a href='/item/%s'>%s</a></li>" % (rec["id"], rec["text"])
        print >>out, "</ul>"
        fd = open(QUERYLOG, "a")
        qinfo = dict(q=query, count=estimated)
        pickle.dump(qinfo, fd)
        fd.close()
    print >>out, '''
<a href="/">Search again</a>

</body>
</html>'''
    return out.getvalue()
@route("/item/:id")
@route("/item/:id/")
def show(id):
    rec = data[id]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Result %s</title></head>
<body>
<h1>Raw JSON record for twit %s</h1>
<pre>''' % (rec["id"], rec["id"])
    print >>out, simplejson.dumps(rec, indent=" ")
    print >>out, '''
</pre>
</body>
</html>'''
    return out.getvalue()
@route("/catinfo/:name")
@route("/catinfo/:name/")
def catinfo(name):
    prefix = prefixes[name]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Values for %s</title></head>
<body>
''' % name
    terms = [(x.term[len(prefix):], db.get_termfreq(x.term)) for x in db.allterms(prefix)]
    terms.sort(key=lambda x:x[1], reverse=True)
    freq_min = terms[0][1]
    freq_max = terms[-1][1]
    def mklink(name, term):
        return "/query?query=%s" % urllib.quote_plus(name + ":" + term)
    # Build tag cloud
    print >>out, "<h1>Tag cloud</h1>"
    print >>out, "<blockquote>"
    for term, freq in sorted(terms[:20], key=lambda x:x[0]):
        size = 100 + 100.0 * (freq - freq_min) / (freq_max - freq_min)
        print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(name, term), size, term)
    print >>out, "</blockquote>"
    print >>out, "<h1>All terms</h1>"
    print >>out, "<table>"
    print >>out, "<tr><th>Occurrences</th><th>Name</th></tr>"
    for term, freq in terms:
        print >>out, "<tr><td>%d</td><td><a href='/query?query=%s'>%s</a></td></tr>" % (freq, urllib.quote_plus(name + ":" + term), term)
    print >>out, "</table>"
    print >>out, '''
</body>
</html>'''
    return out.getvalue()
# Change here for bind host and port
bottle.run(host="0.0.0.0", port=8024)

...and then we presented our work and ended up winning the contest. This was the story of how we wrote this set of award winning code.

I took the train from Charlottesville to New York on Saturday. With this year's DebConf in the USA, I finally have a chance to experience the conference from a local perspective, without jetlag, or a language barrier. I'm still working on that. Part of me thinks I am in a foreign country here. :) Actually, the neighboorhood around Columbia University is one I have ties to. Here is my Mom's old apartment, a few short blocks from the main DebConf venue. And here is family friend Ruth Shereff's memorial garden in the Broadway median. DebConf has been outstanding, and very exhausting. Lars and I have been spending a lot of time here putting the final touches on something to be announced soon.

I just posted this announcement culminating several months of work with the next Debian Edu release. Not nearly done, but one major step completed.

This is the first test release based on Squeeze. The focus of this release is to test the user application selection. To have a look, install the standalone profile and let the developers know if the set of installed packages i.e. applications should be modified. If some user application is missing, or if there are some applications that no longer make sense to be included in Debian Edu, please let us know. Also, if a useful application is missing the translation for your language of choice, please let us know too. In addition, feedback and help to polish the desktop (menus, artwork, starters, etc.) is appreciated. We would like to ship a nice and handy KDE4 desktop targeted for schools out of the box. The other profiles should be installable, but there is a lot more work left to be done before they are ready, so do not expect to much. Changes compared to the lenny based version

• Everything from Debian Squeeze
  • Desktop environment KDE 4.4 => the new KDE desktop in combination with some new artwork
  • Web browser Iceweasel 3.5
  • OpenOffice.org 3.2
  • Educational toolbox GCompris 9.3
  • Music creator Rosegarden 10.04.2
  • Image editor Gimp 2.6.10
  • Virtual universe Celestia 1.6.0
  • Virtual stargazer Stellarium 0.10.4
  • 3D modeler Blender 2.49.2 (new application)
  • Video editor Kdenlive 0.7.7 (new application)
• Now using Kerberos for password checking (migration not finished). Enabled for:
  • PAM
  • LDAP
  • IMAP
  • SMTP (sender verification)
• New experimental roaming workstation profile for laptops.
• Show welcome page to users when they first log in. The URL is fetched from LDAP.
• New LXDE desktop option, in addition to KDE (default) and Gnome.
• General cleanup (not finished)

The following features are not working as they should

• No web based administration tool for creating users and groups. The scripts ldap-createuser-krb and ldap-add-user-to-group can be used for testing.
• DVD installs are missing debian-installer images for the PXE boot, and do not set up the PXE menu on eth0 because of this. LTSP clients should still boot from eth1 on thin client servers.
• The restructured KDE menu is not implemented.
• The LDAP server setup need to be reviewed for security.
• The LDAP directory structure need to be reworked.
• Different sets of packages are installed when using the DVD and the netinst CD. More packages are installed using the netinst CD.
• The jackd package fail to install. This is believed to be caused by some ongoing transition, and hopefully should be solved soon. The jackd1 package can be installed manually for those that need it.
• Some packages lack translations. See http://wiki.debian.org/DebianEdu/Status/Squeeze for updated status, and help out with translations.

To download this multiarch netinstall release you can use

• ftp://ftp.skolelinux.org/skolelinux-cd/squeeze-alpha/debian-edu-6.0.0+edua0-CD.iso
• http://ftp.skolelinux.org/skolelinux-cd/squeeze-alpha/debian-edu-6.0.0+edua0-CD.iso
• rsync -avzP ftp.skolelinux.org::skolelinux-cd/squeeze-alpha/debian-edu-6.0.0+edua0-CD.iso

To download this multiarch dvd release you can use

• ftp://ftp.skolelinux.org/skolelinux-cd/squeeze-alpha/debian-edu-6.0.0+edua0-DVD.iso
• http://ftp.skolelinux.org/skolelinux-cd/squeeze-alpha/debian-edu-6.0.0+edua0-DVD.iso
• rsync -avzP ftp.skolelinux.org::skolelinux-cd/squeeze-alpha/debian-edu-6.0.0+edua0-DVD.iso

There is no source DVD available yet. It will be prepared when we get closer to the final release. The MD5SUM of these images are

• 3dbf45d59f42a53518b6e3c9ec3b5eb6 debian-edu-6.0.0+edua0-CD.iso
• 22f2cbfce281d1c6e478be452638675d debian-edu-6.0.0+edua0-DVD.iso

The SHA1SUM of these images are

• c53d1b69b40cf37cd27aefaf33f6f6a3821bedf0 debian-edu-6.0.0+edua0-CD.iso
• 2ec29d7db676d59d32197b05c277ffe16348376c debian-edu-6.0.0+edua0-DVD.iso

How to report bugs: http://wiki.debian.org/DebianEdu/HowTo/ReportBugsInBugzilla Please direct replies to debian-edu@lists.debian.org

Bug-Squashing-Party in Munich We are organizing another BSP in Munich on the weekend of 17th/18th July. Like the last BSP, it will take place in the LiMux office in the city center. See the coordination page for further information and directions and in order to sign up for it. Food and drinks for the event are partly sponsored by the City of Munich and this amount (100 ) has been again matched by credativ GmbH. Contributions by non-DDs are welcome as always; the BSP will be attended by several Debian Developers who will be able to upload fixes. Attending BSPs is a great way to get involved in the Debian community! We cannot guarantee crash space to people from outside Munich at this point, so if you want to attend please contact the Munich list (area-muc@teams.debian.net) beforehand or arrange your own accomodation. We probably start the BSP at some point after 6PM on Friday evening already, but the main action will be on Saturday and Sunday. As usual, people should bring their notebooks and possibly an ethernet cable. Wireless will be present as well, but a certain bandwidth cannot be guaranteed. Bug-Squashing Introduction I also plan to give an introduction to bug-squashing at this month's Garching Debian Stammtisch, which will take place next Wednesday, July 14th in the campus-cneipe (or its beergarden) as usual (the monthly Stammtisch is always on the second Wednesday of the month). If you study/work in Garching and would like to attend the BSP, come to the Stammtisch for a tutorial (and beer)! Debian Packaging Tutorial Some time ago, I gave a tutorial on Debian packaging in the TechTalk series of the Open-Source- School. The audience seemed to be pleased with the talk, and as more people requested to attend than there were seats in the room, we decided to reprise the event on September 7th. Attendence is free, but you have to register (see the event page). There are still some places left at this point.

Terah has been having thyroid troubles for some time now (you can read about it on her blog). Today was the day for it to be removed. This procedure is somewhat delicate. Running through each half of the thyroid are nerves that control the vocal chords. Also embedded in each half are tiny parathyroids, which regulate the body s calcium levels. Losing either of these things in one half of the thyroid is no big deal, but losing either of them entirely can be serious. For this reason, the general plan was for Terah to have half of her thyroid out, unless it was obvious in surgery that it would be a health risk to leave the other half in, or that it was obvious that the removal of the first half had successfully preserved the nerves and parathyroids. Jacob and Oliver spent the night with my parents last night an event that Jacob was incredibly excited about. I came home last night to find him running in circles in the kitchen. I had this conversation with him: Me: Jacob, are you excited about something? Jacob: Yes! Me: What are you excited about? Jacob: I don t know. (He usually says this when he doesn t want to say, but I figured it was his pending overnight stay) .. a minute later .. Jacob: I am excited about something blue. Me: [genuinely confused] Hmmm . I don t know what blue thing you might be excited about. Can you tell me? Jacob: A blue house! Me: Oh, is it grandma and grandpa s blue house? Jacob: YEAH! Now you ve got it! I m so excited! He was still running in circles the whole time. So Jacob and Oliver left, and Terah and I worked on the house. This morning, we arrived at the hospital at 6:15 to check in. At usual, the staff there was incredibly friendly and helpful. And two of them made connections: one of the intake nurses had taught at the college where my dad works, and another nurse knows his cousin. Living where we do, we were not surprised by this. (You sort of wonder how many connections you ll make at a place like this.) The surgery area is right next to the ER, and people waiting for people in surgery normally wait there. Having got my fill of the TV playing infomercials for the morning in the 5 minutes we sat there, I arranged to wait in the cafeteria (which is large and quiet in the middle of the morning) and they just called my cell phone when Terah was done. I was with Terah for as long as possible, then ate breakfast in the cafeteria, walked over to the doctor s office next door for my weekly allergy shots, and then waited with my Kindle in the cafeteria. They called, and a little while later the surgeon explained how the surgery went: far easier than they had expected. He was confident the left side surgery went well, and so took out the whole thing and was confident the right side removal also went well. I went to wait in Terah s room for her to exit recovery, and our pastor and his daughter showed up to wait too. When she arrived, she was very tired but said a few words before going to sleep. Our pastor gave the two of us a minute before coming in, and read a selection from Psalm 139 before leaving. I then mostly waited with my Kindle some more until she woke up. I brought back some lunch from the cafeteria, and eventually Terah woke up. We chatted just a bit (it was mostly me talking). I went to pick up the boys, and brought them to the hospital. They were glad to see her. But then, the dilemma: I had promised Jacob that we would eat in the hospital restaurant , but it was only 4 and they didn t start serving supper until 5. Jacob somehow remembered that we went exploring in the hospital when Oliver was born 11 months ago, and had been talking about doing this already. So off we went: Jacob sometimes helping me push Oliver s stroller, sometimes running on ahead, enjoying himself and spelling out letters on big signs on the walls. We got a whole bunch of smiles and compliments. Oliver was laughing, and Jacob was happy and well-behaved. We went towards the cafeteria at 4:30 and I got Jacob some of their free lemonade. He was complaining about not eating, and I remembered that he noticed a nurse walking out of there with an ice cream cone while we were exploring. Obviously the soft-serve machine was already up, so I got him and me ice cream cones. I m sure he ll be talking about the hospital ice cream for days. Anyhow, it took him about half an hour to eat it, so it was perfect timing for supper. We ate, went back to see Terah, and then headed out. I do believe that these may be the only two boys I know of that actually think of a hospital as a fun place to go. It wouldn t surprise me at all if Jacob asks to go back there. Both boys fell asleep on the way home. Jacob, unfortunately, had a rather significant accident, possibly due to the large amounts of free lemonade he insisted on drinking. And it was hot outside. So I carried Jacob in, and he sort of groggily looked at me. I suggested he take a nap on the rug in the kitchen (his room is upstairs) while I bring Oliver in. He sort of collapsed onto the rug and was back asleep a few seconds later. I got Oliver in, and brought down new shorts and an overnight pullup for Jacob. Put them on, had him use the toilet, and got him in bed. By that time, Oliver was awake and hungry. So I got him fed, and neither one of them quite went to sleep easily but they did. I watered the flowers and garden, did a load of laundry, etc. Jacob had been worried about the surgery, though had trouble verbalizing it. I think he s better about it now, though some of his trouble falling asleep tonight was probably still worry about that. Hopefully he ll be better once Terah is back home, which should hopefully be tomorrow.

The fact that it's only two weeks feels totally surreal. I think it's the feeding approximately every 3 hours. That's approximately 100 interactions that we've had with her in only 2 weeks. We had her two week weight check at the paediatrician's today. She's now weighing 6 lbs 12 oz (3061 grams), so she's made back her birth weight and then some. We're still largely bottle feeding her with pumped breast milk. The paediatrician said that once she's 7 pounds she should have enough strength to breastfeed exclusively, so we'll get a lactation consultant involved if that isn't the case in another week or so. Other than that, everything's going swimmingly. She eats, sleeps and poops. She sleeps amazingly well. I keep telling myself it won't last, but we're making the most of not needing to tip-toe around the house. In the last week, we had a new stove installed, which involved the contractor using a hammer drill to install the anti-tip brackets. He also used a drop-saw to modify one of the drawers. Didn't bother her. We had some shutters installed in the kitchen, which involved a lot of cordless drilling. Also didn't bother her. The gardeners for the complex were mowing the lawn and using a leaf blower right outside her window. Not a peep. Today, after the paediatrician's appointment, we dropped in to Palo Alto Animal Services, the animal shelter where Sarah has volunteered for the last 4.5 years, to show them Zoe. They threw a bit of a surprise baby shower, and gave us a ton of baby clothes, which was very nice and totally unexpected. Zoe slept in her car seat on the table in the lunch room while a good dozen people or so made a lot of noise around her, and she was oblivious to it all. Sarah's Mum arrives on Wednesday. Tomorrow will be a busy day of final cleaning up in preparation.

Today all across America, many are experiencing a soft form of hell, anxiety, and constant cursing for it is tax day. Thankfully last night I squared those away (more or less). Today is also the last day to register forDebconf sponsored registration. Even if there is a chance you can t make it, it is worth filling out the form just in case. For those spending a little extra time in NYC or planning on sneaking out to take a whirl in the city, here are some more fun options. NYC is known to be a cultural cornucopia of sorts and one reason are the many museums. I just found a twitter list listing those museums with a feed. Probably one of my favorite museums is located in the far north of Manhattan, the Cloisters. It is a haven, an oasis with small beautiful gardens, as well large stone rooms and buildings filled with medieval art, tapestry, and manuscripts. If you need to get away from people, chatter, and computers, this might be a good place to visit. Museums are easy enough to find but there are other spots in NYC that may be a little harder to spot. If you want to hit Chinatown for dim sum, I would recommend Jing Fong, which is mind blowing largely because of its LARGE (like there are escalators that take you up) room, which gets pretty darn packed on the weekend. If you want something that involves not just food but involves sweat, water, food, sun, hot rocks, body scrubs, funny looking pink, orange and blue uniforms and all in a cross-cultural context, then Spa Castle is the place to go to. Just check out the website and you will see what I am talking about. It s family friendly, has a cornucopia of saunas (filled with jade, gold, ice, Tibetan bricks), some decent Korean food, among many other attractions. In general is one of the best deals in NY.

Today all across America, many are experiencing a soft form of hell, anxiety, and constant cursing for it is tax day. Thankfully last night I squared those away (more or less). Today is also the last day to register forDebconf sponsored registration. Even if there is a chance you can t make it, it is worth filling out the form just in case. For those spending a little extra time in NYC or planning on sneaking out to take a whirl in the city, here are some more fun options. NYC is known to be a cultural cornucopia of sorts and one reason are the many museums. I just found a twitter list listing those museums with a feed. Probably one of my favorite museums is located in the far north of Manhattan, the Cloisters. It is a haven, an oasis with small beautiful gardens, as well large stone rooms and buildings filled with medieval art, tapestry, and manuscripts. If you need to get away from people, chatter, and computers, this might be a good place to visit. Museums are easy enough to find but there are other spots in NYC that may be a little harder to spot. If you want to hit Chinatown for dim sum, I would recommend Jing Fong, which is mind blowing largely because of its LARGE (like there are escalators that take you up) room, which gets pretty darn packed on the weekend. If you want something that involves not just food but involves sweat, water, food, sun, hot rocks, body scrubs, funny looking pink, orange and blue uniforms and all in a cross-cultural context, then Spa Castle is the place to go to. Just check out the website and you will see what I am talking about. It s family friendly, has a cornucopia of saunas (filled with jade, gold, ice, Tibetan bricks), some decent Korean food, among many other attractions. In general is one of the best deals in NY.

A truckload of compost was dropped off today. Went out this evening to put together a raised bed for it, and when I looked up, I saw a wonderful surprise. I've been trying to catch this groundhog for ages. It's still fat from last year's garden and the whole other contents of the yard, and also, I think, pea shoots that came up recently. But it would not set foot in the trap for any bait, and I had given up (again) and planned to put the trap away this evening. I came out at just the right instant, to confuse the beast, already stressed by the appearance of big piles of dirt, to dodging for the wrong hiding place. It's now enjoying its new home out at the lake, and I'm looking forward to enjoying vegetables.

Kiwis have to be one of the cutest birds of all time. A bit chubby and clumsy looking, they are (to me, at least), walking, breathing icon of furry cuteness. Truth be told, I did not see one live in action during my recent trip to New Zealand, but I did see a Weka bird which is pretty similar. (Micah and I just consulted the Internets and agreed that we did in fact see cherubic Kiwi not a Weka. In fact, Micah almost ran one over on the west coast of the southern island but his quick reflexes born from years of saving servers while typing saved the bird. So as you can tell by now I finally got to go to New Zealand for a few weeks, even able to travel the south island before attending THE Linux conference of the region, LCA where I gave a talk (video to be posted when available). I jumped at the opportunity and was thrilled to find out that I could actually spend some time traveling around before the intense week long conference. So on January 5th I headed down with Micah to what I have found out is one of the windiest cities on earth ( Windy Wellington ) for a few days before heading to the south island to do a loop on the northern half of the island. I wanted to conquer the mountains made famous by a certain set of movies, but last semester was all about sitting in front of the computer, a sort of agonizing life of the mind, which did not prepare me physically for mountain trekking or as it call it there, tramping. So we decided to stick close to the coast (with one transalpine train that serenaded us through some awesome mountain passes) and hike along the coast, snorkel and swim, and kayak. After Andrew showed us around Wellington, we headed to the sea town of Kaikoura brimming with sea life, such as seals, dolphins, and birds. The two highlights there were our snorkeling trip and a hike along the peninsula. Originally we were going to go diving but I arrived to NZ with a head cold so we opted for the sinus friendlier version and snorkeled in the massive kelp forest, which was strikingly beautiful (I am used to tropical reefs and waters). There happened to be a lot of seals and given their curious nature, they would dart over right next to you and play. One seal I called him the twirler was chubby and mellow and twirled next to me for a long while. The lady seals were sleeker and faster and liked to dive deep down and the zoom up and surprise you. As someone who has spent a lot of time underwater, I found it fun and weird to be with animals that pay attention to you given that most of the sea life ignores you. The hike in Kaikoura was fantastic as well. You go high on a bluff and stare down at this water sparkling with the most beautiful blue hues brought out even more majestically by the white rocks. If I were to go back I would hike along the water and up above on the bluff. After a brief stopover in Christchurch to visit a friend, we took the transalpine train (across Arthur s pass) which was superbly beautiful and arrived on the west coast whose seas are far more rough, rocky, andtemperamental than on the east coast. The vegetation is lush as lush can be and you get the feeling that dinosaurs would have been a happy lot in these parts. The cool part about this area are the many rivers that feed into the ocean (we spent an afternoon swimming and exploring the caves in one) and then what are called the pancake rocks, which are odd geological creatures that really amaze. We then made our way to the Marlborough Sounds area, which includes the city of Nelson (apparently the sunniest in New Zealand) as well as many olive and grape vineyards in outlying areas. We spent some time in the Alpine lakes region hiking but more of the time on the coast. The highlight for me and it is a place I would love to go back to is the Abel Tasman National Park. We spent time hiking and kayaking there and you can even hike down the coast for 5-6 days (there are excellent camping facilities along the way). The coast is simply majestic with alpine like forest conditions (unfortunately a cool old tree whose name I can t remember is no longer common). This forest is situated along the most stunning waters, an electric but totally clear turquoise blue deceivingly inviting because the water was quite cold (but not as cold as the Oregon coast over the summer, which I went swimming in only once after weeks of bike travel). Along with the blue, parts of the coastal waters were a stunning clear green that I have never seen despite my many sea travels. If you ever plan on going to the South Island, I would not miss this area and might even try to spend a few days or more hiking the coastal trails and listening to the cool-as/sweet as birds, one of which has double vocal cords so it sings what sounds to be like totally electronic bird calls. We returned via the same ferry a massive 7 story ship from Picton to Wellington but the conditions were far far worse during our return trip. When we got out of the sounds, we hit the Cook Straight, notorious for being rough and tumble. And indeed, once we hit the waves, I felt like I was no longer in a boat but amusement park ride, the bow of the ship thrown high only to fall hard on upon the waves. It was sort of fun or fun for about 10 minutes until people, most especially kids, started yaking all over the place. It was really the most flagrant yakfest I have ever experienced (I mean it is a 7 story ferry, you can fit a lot of people on that boat) and that made the sail particularly difficult. I have pretty strong sea legs, never once getting sick sick during the year I spent at sea but massive yaking will bring out the sea sick feeling in even the hardiest of sea souls and I was thrilled to arrive in Wellington. Wellington is the only large city I spent any considerable time in and it is a pretty awesome one. The city center area is flat and by the water, with a magnificent area filled with museums, the national library, parks, breweries, and piers where kids like to jump off high things. The botanical gardens were the most beautiful I have been to and since I am a flower nut, I was in high heaven. In this area there are a ridiculous number of great coffee shops (NYC has a thing or two to learn as we don t have enough, I suspect it has something to do with insane real estate prices). The city is surrounded by hills populated by houses, some of the precipitously perched on cliffs that require lifts to get to! It was a wonderful home base for what was a sweet as conference (as they would say down there). And I hope to write about it sometime soon.

Kiwis have to be one of the cutest birds of all time. A bit chubby and clumsy looking, they are (to me, at least), walking, breathing icon of furry cuteness. Truth be told, I did not see one live in action during my recent trip to New Zealand, but I did see a Weka bird which is pretty similar. (Micah and I just consulted the Internets and agreed that we did in fact see cherubic Kiwi not a Weka. In fact, Micah almost ran one over on the west coast of the southern island but his quick reflexes born from years of saving servers while typing saved the bird. So as you can tell by now I finally got to go to New Zealand for a few weeks, even able to travel the south island before attending THE Linux conference of the region, LCA where I gave a talk (video to be posted when available). I jumped at the opportunity and was thrilled to find out that I could actually spend some time traveling around before the intense week long conference. So on January 5th I headed down with Micah to what I have found out is one of the windiest cities on earth ( Windy Wellington ) for a few days before heading to the south island to do a loop on the northern half of the island. I wanted to conquer the mountains made famous by a certain set of movies, but last semester was all about sitting in front of the computer, a sort of agonizing life of the mind, which did not prepare me physically for mountain trekking or as it call it there, tramping. So we decided to stick close to the coast (with one transalpine train that serenaded us through some awesome mountain passes) and hike along the coast, snorkel and swim, and kayak. After Andrew showed us around Wellington, we headed to the sea town of Kaikoura brimming with sea life, such as seals, dolphins, and birds. The two highlights there were our snorkeling trip and a hike along the peninsula. Originally we were going to go diving but I arrived to NZ with a head cold so we opted for the sinus friendlier version and snorkeled in the massive kelp forest, which was strikingly beautiful (I am used to tropical reefs and waters). There happened to be a lot of seals and given their curious nature, they would dart over right next to you and play. One seal I called him the twirler was chubby and mellow and twirled next to me for a long while. The lady seals were sleeker and faster and liked to dive deep down and the zoom up and surprise you. As someone who has spent a lot of time underwater, I found it fun and weird to be with animals that pay attention to you given that most of the sea life ignores you. The hike in Kaikoura was fantastic as well. You go high on a bluff and stare down at this water sparkling with the most beautiful blue hues brought out even more majestically by the white rocks. If I were to go back I would hike along the water and up above on the bluff. After a brief stopover in Christchurch to visit a friend, we took the transalpine train (across Arthur s pass) which was superbly beautiful and arrived on the west coast whose seas are far more rough, rocky, andtemperamental than on the east coast. The vegetation is lush as lush can be and you get the feeling that dinosaurs would have been a happy lot in these parts. The cool part about this area are the many rivers that feed into the ocean (we spent an afternoon swimming and exploring the caves in one) and then what are called the pancake rocks, which are odd geological creatures that really amaze. We then made our way to the Marlborough Sounds area, which includes the city of Nelson (apparently the sunniest in New Zealand) as well as many olive and grape vineyards in outlying areas. We spent some time in the Alpine lakes region hiking but more of the time on the coast. The highlight for me and it is a place I would love to go back to is the Abel Tasman National Park. We spent time hiking and kayaking there and you can even hike down the coast for 5-6 days (there are excellent camping facilities along the way). The coast is simply majestic with alpine like forest conditions (unfortunately a cool old tree whose name I can t remember is no longer common). This forest is situated along the most stunning waters, an electric but totally clear turquoise blue deceivingly inviting because the water was quite cold (but not as cold as the Oregon coast over the summer, which I went swimming in only once after weeks of bike travel). Along with the blue, parts of the coastal waters were a stunning clear green that I have never seen despite my many sea travels. If you ever plan on going to the South Island, I would not miss this area and might even try to spend a few days or more hiking the coastal trails and listening to the cool-as/sweet as birds, one of which has double vocal cords so it sings what sounds to be like totally electronic bird calls. We returned via the same ferry a massive 7 story ship from Picton to Wellington but the conditions were far far worse during our return trip. When we got out of the sounds, we hit the Cook Straight, notorious for being rough and tumble. And indeed, once we hit the waves, I felt like I was no longer in a boat but amusement park ride, the bow of the ship thrown high only to fall hard on upon the waves. It was sort of fun or fun for about 10 minutes until people, most especially kids, started yaking all over the place. It was really the most flagrant yakfest I have ever experienced (I mean it is a 7 story ferry, you can fit a lot of people on that boat) and that made the sail particularly difficult. I have pretty strong sea legs, never once getting sick sick during the year I spent at sea but massive yaking will bring out the sea sick feeling in even the hardiest of sea souls and I was thrilled to arrive in Wellington. Wellington is the only large city I spent any considerable time in and it is a pretty awesome one. The city center area is flat and by the water, with a magnificent area filled with museums, the national library, parks, breweries, and piers where kids like to jump off high things. The botanical gardens were the most beautiful I have been to and since I am a flower nut, I was in high heaven. In this area there are a ridiculous number of great coffee shops (NYC has a thing or two to learn as we don t have enough, I suspect it has something to do with insane real estate prices). The city is surrounded by hills populated by houses, some of the precipitously perched on cliffs that require lifts to get to! It was a wonderful home base for what was a sweet as conference (as they would say down there). And I hope to write about it sometime soon.

renovate

v 1: restore to a previous or better condition
  2: make brighter and prettier

remodel

v 1: do over, as of (part of) a house
  2: cast or model anew

I much prefer the term renovate to remodel. Remodelling to me feels more like tearing down internal walls and changing the actual layout of the place. But we're in America, so remodelling it is. We've had a bit of (mostly self-inflicted) scope creep. It started with just wanting to get some Ethernet cabling installed, then we decided to get recessed canned lighting (the whole lamp thing over here drives us nuts), and then we discovered that acoustic ("popcorn") ceilings are a haven for dust mites (we're both allergic) so it made sense to get the ceilings redone while they were putting the lights in. We're also getting the light switches replaced, and new childproof power outlets. We also got the electrical panel upgraded. Removing popcorn ceilings is somewhat hilarious. They put down plastic everywhere, then have at it with a garden hose, and then it just scrapes off. The ceilings they've completely finished already look a million bucks, so we're very happy with the decision to do that. The Ethernet cabling installation caused a lot of wall carnage, so we're going to get the walls repainted as well. Then we can get the carpets cleaned and stain proofed, and then we can start thinking about trying to move some non-essential stuff in. Hopefully it'll all be done by next weekend. The kitchen cabinets have a six week lead time, so the kitchen remodelling will have to be done while we're living there. Unfortunately some unexpectedly time demanding stuff has come up at work, right in the middle of this, so Sarah's had to do the bulk of the running around to sort out contractors for all of the work, and I've been stuck at work until all hours, 7 days a week. To cap it off, I've managed to come down with a cold, and feel quite crappy. NyQuil consumption is ensuing.

Our lease runs out at the end of January, and we're thinking about moving to a bigger place. In the four years we've been here (yeah, we just crossed the four year mark the other day) we've often thought about trying to buy as well. We go through these phases where we really feel like buying, then we run the numbers and run screaming back to the warm bosom of renting. A couple of months ago we had the most serious foray into buying. We'd just checked out the models for Mondrian and we really liked the floorplan for Bleu, and found the price to be the least breathtaking of anything we'd looked at in the Bay Area. I got as far as talking to mortgage brokers and running the numbers, and the things that killed it for us were the property taxes and homeowners association fees. The monthly repayments would have been doable, but it'd have really been a ball and chain. We're over here to see the country as much as anything else, and if the mortgage is going to be a significant impedance on our ability to travel, then there isn't really any point in doing it. So we sadly passed up on Mondrian. The three bedroom townhouses in our current complex are going for around the $2500 a month mark, which is a pretty serious jump on what we're paying now for our two bedroom one, so Sarah's been scouring Craigslist for anything better. She found a 2 bedroom plus loft condo being rented privately in Mountain View, which we took at look at on Wednesday. The immediately downsides are it's older (the kitchen and bathroom are really a bit dated) and has no data cabling (this is something I've really loved about our current place) and no microwave oven included. The upsides are it's significantly bigger (about 500 square feet larger), the kitchen has heaps of cupboards, it has a washer and dryer, a lock-up garage, a small, fairly private yard (the rent includes a gardener), and it has what looks like a communal garden bed (the thing that really caught my eye were the compost bins). So I think overall, as long as we can live with the kitchen and bathroom, it's going to be an improvement on where we are now. The windows are all double glazed, so it should be fairly well insulated. It's got a gas furnace and gas hot water, and I think the landlord pays for the water, so I think the utilities would at best come out the same as what we're paying here. We've decided to put an application in for it and see what happens. The landlord is living overseas, so we're dealing with a real estate agent for the letting. Apparently we'd be paying the rent via PayPal or something. He's got a home warranty arrangement for maintenance, which sounds like it'll be pretty good. Meanwhile, Sarah was scouring Redfin and found a house nearby that is for sale (a short sale), which is pretty reasonably priced. We've called up a real estate agent, and we're taking a look at it tomorrow, just because we can.

It seems that despite having a very important work deadline today I am destined to be interrupted every five minutes. Unfortunately I need to concentrate to get this done. And while i managed to get "in the zone" earlier, now I cannot string two words together.

This seems to be a recurring theme recently, my productivity is horribly low because I get interrupted all the time for "can you just" jobs. I turn the phone ringer off and hide my email window and I get called on my mobile and get asked why don't you answer your phone?

Its not so much the time lost to answering the specific query or doing the job but more my context switch time to restore to my previous task becomes huge. Somehow my switching time is absolutely non linear and today it has become so large I have decided to dump state altogether.

Because I know I have to do the school run soon and the interruptions continue I have abandoned the critical task altogether for now in the hope I can get back to it fresh. Do others get this I wonder? or is it a personal fault I should strive to fix? I know I used to be able to deal with this sort of thing with much less trouble, maybe I am getting less flexible as I age?

Fortunately (umm, you know what I mean) the Entropy Key software has some cleanups required which is an easy job and does not matter if I get interrupted so I do have a productive task to complete.

Oh speaking of the entropy key Niel blogged something silly I created the other day. I had a senior moment and filked the children's nursery rhyme "If You're happy and you know it" slapping some guitar chords on it.

It seems I cannot simply leave well enough alone and last Friday night (with the help of Mark Hymers) I made it a whole lot worse. I present to you the score of "If you're happy with your ekey, blog your praise" for piano and guitar. In fact I wrote this out in rosegarden so it can be turned into a midi file etc. all the source files are available. It is released under a CC license so I suppose someone could put the lyrics back to the proper nursery rhyme and use it for something less tacky.

I m back from Dallas, where I was invited at the Ubuntu Developer Summit for Lucid. I spent a great week there ; the event was extremely well organized (by organizing them every 6 months, you are probably able to gather a lot of experience!). Of course, after all I had heard about people hugging each other all the time in the Ubuntu community, I was a bit worried, especially with the flu spreading! But there are lots of fantastic people around Ubuntu, and it was a very nice opportunity to be able to meet them all. Since it was my first UDS (I was at FOSSCAMP in Prague a few years ago, but didn t stay for UDS back then), I was not really sure of what to expect. I was very pleasantly surprised. UDS vs Debconf UDS is very different from Debconf. In Debconf, we do three different kind of things:

• we hack, mostly on our own, in the hacklabs
• during talks, we report on what we have done recently
• during some talks or BOFs, we discuss future (possible) changes

In UDS, the main focus is on the third point: most of the sessions are about discussing what will be implemented for the next release. All of the relevant developers are in the same room to discuss possible problems, and the outcome of each session is usually a detailed plan, with a list of action items. It s a very nice way to ensure that changes are well thought, and allows making large-scale changes in Ubuntu very easily (you don t spend weeks arguing about them on mailing lists). Of course, it s probably also helped by the fact that there s a company behind Ubuntu, with a set of large teams (kernel, foundations, desktop, etc), which helps transfering trust (not everybody feel like they have to participate in each discussion, even when they affect the whole distribution: the team in charge is trusted by the rest of the project). On the other hand (yeah, let s be negative for a while) it doesn t really help spreading information between Ubuntu developers: it s often a bit difficult to get the global view of what is happening inside Ubuntu, especially since lots of things are discussed on IRC. Collaboration between Debian and Ubuntu During the week, I mainly was interested on collaboration between Debian and Ubuntu. There s a strong focus on doing the right thing wrt Debian (and also other upstreams). Then, of course, Ubuntu also has an agenda, which sometimes requires moving very fast on some things, or making compromises between technical purity and pragmatism. But the willingness to have common foundations between Debian Squeeze and Ubuntu Lucid will surely benefit both distros. Quality Assurance On the QA front, I am planning to do archive rebuilds for Ubuntu as well (fixing FTBFS is an easy way to start contributing to Ubuntu or Debian, and having those bugs fixed in Lucid would benefit Debian as well, by having patches already prepared). I also had a session with the QA team, where I gave an overview of what we are doing in the Debian QA group, to discuss opportunities for collaboration. The Ubuntu QA team focuses more on testing (with automated or manual testing) and bug triaging than archive quality that part is left to the MOTUs and the release managers. (About MOTUs, I liked how what they do was described as long-tail maintenance, landscape gardening or terraforming. That gives a good idea of what it s about). Ultimate Debian Database On the Ultimate Debian Database front, I did a plenary talk to try to demonstrate how UDD could be useful to Ubuntu as well, and, with Jorge Castro, we examined some metrics of Ubuntu s giving back to Debian. I also talked with the Launchpad team to try to resolve my long standing pretty please provide an export of Ubuntu bugs, so I can easily import them in UDD! issue. Ubuntu and ARM ARM netbooks and smartbooks (mix between netbooks and smartphones) are coming, and Ubuntu is clearly very well positioned to play an important role on that market. There was a whole track about ARM support, with lots of changes that will be done for Lucid. Let s all hope that Ubuntu-powered ARM netbooks win that market, so we don t reproduce the failures of the non-ARM netbooks. Distributed Development James Westby has been working on a set of tools to be able to work on Ubuntu packaging using bzr. The point is not to store the canonical source for Ubuntu packages in bzr (well, at least it s not the plan yet), but to provide a set of branches to make it easier to merge or cherry-pick from Debian. The resulting workflow looks extremely nice, with lots of syntaxic sugar. And even better, he assured me that his code is portable to Git ;)
Using his work, merging Ubuntu-specific changes in a new version of a Debian package basically means pulling from lp:ubuntu/foopkg, merging from lp:debian/sid/foopkg, and you are done!
As a bonus, we (Debian) would get bzr branches with the history of packages (kind of bzr-powered snapshots.debian.org).
James project is not completely ready yet, but should be very soon. It s already basically usable, apparently. Conclusion Ubuntu has clearly gone a very long way since 2004. Everything looks very well organized and polished, and gives the impression of a big machine that nothing can stop. With Cloud Computing and now ARM netbooks, Ubuntu has proven to be able to adapt to the current trends and attract a lot of visibility. It is great news for Free Software, but also proposes an interesting challenge to Debian: of course, it s nice that a Debian-based distro is in that position, but will Debian manage to stay relevant, or are we just going to be the technically-pure distro without many users that serves as a package supermarket for Ubuntu?

Forgive me if I'm missing a nuance here, but consider this quote from the article "Developing for the Apple iPhone using Flash":

We created a new compiler front end that allowed LLVM to understand ActionScript 3 and used its existing ARM back end to output native ARM assembly code. We call this Ahead of Time (AOT) compilation in contrast to the way Adobe Flash Player and Adobe AIR function on the desktop using Just in Time (JIT) compilation.

So er, this new-fangled non-JIT technology is important enough to get it's own three-letter acronym: AOT. Perhaps at some point it might be added to the already crowded wikipedia 'AOT' article. But isn't this essentially a garden-variety Compiler, the same technology we've enjoyed since 1957? Please don't misinterpret this as a slight at LLVM, which is very a very promising piece of software.

I read the news about the launch of Ubuntu User by Linux New Media AG with some sadness. I believe that free software is stronger because of the informal cooperation between different distributions that usually do different jobs. It lets us see what s growing in our neighbour s garden. I m a debian developer, but I don t subscribe to many debian-specific products or services because I like the broader view. Also, at a generally-economically-weak time when some more general publications seem to be dying and restarting, concentrating on the Ubuntu users seems like it probably won t be a big enough audience to sustain a title. The temporary loss of readers could also push other multi-distribution magazines over the brink. Now, I use and support Ubuntu systems from time to time (among others), but it feels like Ubuntu attracts this sort of anti-social single-distribution concentration more often than any other current one. Its users seem even more fixed on it than Gentoo s users, which is almost the opposite of what I expect from a distribution which claims Ubuntu and Free Software are about collaboration and working together . So why does this happen? Are any sociologists researching whether there are really narrower horizons among Ubuntu users than others? Have any Ubuntu community leaders questioned the lack of collaboration with Free Software in launching an Ubuntu-only magazine in a recession?

Sometimes I really, really hate London. A trip to London on Saturday, in theory: Leave Ely 16:26, arrive Kings Cross 17:34, change to Piccadilly line and arrive Covent Garden at 17:54. Dinner then the 21:52 train back home, arriving 23:10. A trip to London on Saturday, in practise. Leave Ely 16:26, arrive Kings Cross 10 minutes late. Change to Piccadilly line and stand outside the closed barriers for 15 minutes because of overcrowding. Give up on the tube, catch a number 59 bus to Aldwych: 10 minutes to reach Euston (faster to walk) and gave up after sitting in gridlock on Russel Square for 15 minutes. Eventually get rather empty Piccadilly line from Russel Square to Covent Garden, arriving 18:50. Oh, and then the restaurant said it would be a two hour wait for a table for four. That said, the return journey wasn't exactly a barrel of laughs. The tube was behaving so that took the expected ten minutes, but then we just missed the train back home and the next train wasn't for over an hour. Jump into a taxi to Liverpool Street to catch the 22:26... to discover there are engineering works and we'd have to get a bus for two hours. Attempt to get the tube back to Kings Cross... more engineering works so that was out. Another taxi back to Kings Cross and we finally get on the last train home, arriving at 00:35. Just for extra fun I'm in London for the Moblin 2.0 Release Party on Monday and there are yet more engineering works, so if I miss the 22:15 I'll be on a bus for half the journey. Stab stab stab.

I'm moving to the US on Thursday, so I will be here on Wednesday evening from about 7. If your presence is unlikely to make me stupifyingly angry, feel free to join me.

This weekend I implemented an idea I had some weeks ago: sending regular mails with the translation status and the contents status of debian-edu-doc to the Debian Edu mailinglist, in the hope this would motivate people to finish the documentation and its translations. Currently the package contains four different manuals, translated in up to 6 languages: the old Debian Edu etch manual, the new Debian Edu lenny manual and manuals for teachers using Rosegarden and Audicity. All four are being written on wiki.debian.org, then exported to Docbook, then .po files are created with the great po4a tool suite. Those .po files are kept in svn of the the debian-edu-doc package, for translators to edit. Detecting missing or half finished content is done by putting the string "FIXME" plus a description what to fix in the wiki text. Then a small script is run by cron every 2nd and 4th sunday of the month, which outputs the translation status and greps for FIXME and formats this nicely. The result looks like this:

The (translated) debian-edu-lenny manual as PDF or HTML is available at http://maintainer.skolelinux.org/debian-edu-doc/

To understand this mail better, please read /usr/share/doc/debian-edu-doc/README.
This mail is automatically send by a cronjob run by Holger Levsen every two weeks. Please send feedback, suggestions, flames and cookies via this list.


debian-edu-lenny-manual.de.po: 545 translated messages, 193 fuzzy translations, 220 untranslated messages.
debian-edu-lenny-manual.es.po: 85 translated messages, 204 fuzzy translations, 669 untranslated messages.
debian-edu-lenny-manual.fr.po: 447 translated messages, 279 fuzzy translations, 232 untranslated messages.
debian-edu-lenny-manual.it.po: debian-edu-lenny-manual.it.po:7683: end-of-line within string
debian-edu-lenny-manual.nb.po: 711 translated messages, 159 fuzzy translations, 88 untranslated messages.
----------------------------------
http://wiki.debian.org/DebianEdu/Documentation/Lenny//Features
FIXME: describe new features in 5.0.1+edu0 here.
----------------------------------
http://wiki.debian.org/DebianEdu/Documentation/Lenny//Requirements
FIXME: a description of main-server and thinclient-server is missing.
----------------------------------
http://wiki.debian.org/DebianEdu/Documentation/Lenny//Installation
FIXME: probably each paragraph on this page needs to be (slightly) rewritten for lenny.
FIXME: describe PXE installation here, if its only two sentences, then they should be here.
FIXME: path to preseed file location missing
----------------------------------
[...]
====================
24 FIXMEs left to fix
====================


I'm quite happy with the result, now we we just need to fix those FIXMEs :-)

The same tools will probably be used for the illustrated Lenny installation guide and maybe others. I've described the requiered structure for such a manual on wiki.d.o in an email to debian-boot (as the discussion about that guide happened there), in case you are interested.

In mostly unrelated news, I've started running everyday again six days ago (slowly running a bit further each day) and yesterday I've also started with doing gymnastics. It was high time to realize and to change some stuff in my life, and it looks like I finally heard the call. And doing sports is by far not the only thing I want to do very differently again, it's just the most visible atm and it's also quite easy to do. Also I can feel it in my legs today :-) Now I "just" need to keep going forward, change is a continuous process, and nothing to be done with. Ever. The day I stop changing myself, the day I stop learning, I consider myself to be dead. I'm glad I woke up, thanks for contributing to making me wake up and for bearing with me sometimes.