Search Results: "stone"

28 October 2010

John Goerzen: Rudy Schmidt & Time Capsules

This evening, I arrived at church for mens chorus practice. I was surprised to see this sign on the door:
Rudy Schmidt died rather unexpectedly, but peacefully on Wednesday morning at his home .
One of the benefits of living in a small community is that I get to know people of all ages around here. Not just people my own age, or coworkers. A few years ago, I worked on a history project for our church s centennial. Back in the 1960s, Rudy was one of the people in charge of building our current church. So he was a person I was interested in visiting with. Here he is, on the left, during the 1964 groundbreaking ceremony for the new building (the old one is visible in the background). 022 stamped may 1964cropped.jpg I and a few others went to his home one time a few years ago and we had a wonderful evening. Rudy shared all sorts of stories with us which I am happy to say I recorded. But moreover, Rudy was an avid photographer. There were some rare and brilliant color slides of the church being built in his collection. He let me borrow and scan some of them. Here are my two favorites: 015 stamped july 1964.jpg 016 stamped oct 1964.jpg Back in 1965, there was a ceremony at the setting of the cornerstone of Tabor Church. The people at the time assembled some papers and memorabilia and put them in a time capsule. Rudy Schmidt sealed the capsule and it was placed in the spot for it behind the cornerstone. Here is a photo of that ceremony. 013 stamped nov 1964.jpg 44 years later, on the occasion of Tabor s centennial, it was time to open the time capsule. The other building committee people, such as Jake Koehn, had passed away, so Rudy was the one to open it. He gave a few words in front of the large crowd, and mentioned that when he sealed it away, he smelled some smoke when he soldered it shut. Jake Koehn told him, Don t worry, nobody will ever see it again anyway and they had a little laugh about it. Now he was going to open it up and see if anything survived. img_6173.jpg It all did just a few minor burn marks. We enjoyed looking at all the items. Then the church placed some new items into the capsule. It was closed up again, to be opened in maybe 50 years. With luck, I ll still be around to see it opened again. Even if I m not, maybe Jacob and Oliver will be able to. And so it seems very fitting that the last hands to touch that box for the next 50 years were Rudy Schmidt s, when he placed it back in its spot on Oct. 12, 2008. img_6243.jpg

27 October 2010

Daniel Stone: how do i progammatically invoke aero peek on a window?

Whenever someone asks how to force their window to be raised to the top in X, because their program is so great that their users shouldn't be allowed to even see other programs, the answer looks a lot like the Windows equivalent:
Yes, I realize you wrote a program so awesome that all other programs pale in comparison, and that part of your mission is to make all the other programs literally pale in comparison to your program. Sorry. Maybe you can meet up with that other program that is the most awesome program in the history of the universe and share your sorrows over a beer.

15 October 2010

Enrico Zini: Award winning code

Award winning code Me and Yuwei had a fun day at hhhmcr (#hhhmcr) and even managed to put together a prototype that won the first prize \o/ We played with the gmp24 dataset kindly extracted from Twitter by Michael Brunton-Spall of the Guardian into a convenient JSON dataset. The idea was to find ways of making it easier to look at the data and making sense of it. This is the story of what we did, including the code we wrote. The original dataset has several JSON files, so the first task was to put them all together:
#!/usr/bin/python
# Merge the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import os
res = []
for f in os.listdir("."):
    if not f.startswith("gmp24"): continue
    data = open(f).read().strip()
    if data == "[]": continue
    parsed = simplejson.loads(data)
    res.extend(parsed)
print simplejson.dumps(res)
The results however were not ordered by date, as GMP had to use several accounts to twit because Twitter was putting Greather Manchester Police into jail for generating too much traffic. There would be quite a bit to write about that, but let's stick to our work. Here is code to sort the JSON data by time:
#!/usr/bin/python
# Sort the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import sys
import datetime as dt
all_recs = simplejson.load(sys.stdin)
all_recs.sort(key=lambda x: dt.datetime.strptime(x["created_at"], "%a %b %d %H:%M:%S +0000 %Y"))
simplejson.dump(all_recs, sys.stdout)
I then wanted to play with Tf-idf for extracting the most important words of every tweet:
#!/usr/bin/python
# tfifd - Annotate JSON elements with Tf-idf extracted keywords
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import sys, math
import simplejson
import re
# Read all the twits
records = simplejson.load(sys.stdin)
# All the twits by ID
byid = dict(((x["id"], x) for x in records))
# Stopwords we ignore
stopwords = set(["by", "it", "and", "of", "in", "a", "to"])
# Tokenising engine
re_num = re.compile(r"^\d+$")
re_word = re.compile(r"(\w+)")
def tokenise(tweet):
    "Extract tokens from a tweet"
    for tok in tweet["text"].split():
        tok = tok.strip().lower()
        if re_num.match(tok): continue
        mo = re_word.match(tok)
        if not mo: continue
        if mo.group(1) in stopwords: continue
        yield mo.group(1)
# Extract tokens from tweets
tokenised = dict(((x["id"], list(tokenise(x))) for x in records))
# Aggregate token counts
aggregated =  
for d in byid.iterkeys():
    for t in tokenised[d]:
        if t in aggregated:
            aggregated[t] += 1
        else:
            aggregated[t] = 1
def tfidf(doc, tok):
    "Compute TFIDF score of a token in a document"
    return doc.count(tok) * math.log(float(len(byid)) / aggregated[tok])
# Annotate tweets with keywords
res = []
for name, tweet in byid.iteritems():
    doc = tokenised[name]
    keywords = sorted(set(doc), key=lambda tok: tfidf(doc, tok), reverse=True)[:5]
    tweet["keywords"] = keywords
    res.append(tweet)
simplejson.dump(res, sys.stdout)
I thought this was producing a nice summary of every tweet but nobody was particularly interested, so we moved on to adding categories to tweet. Thanks to Yuwei who put together some useful keyword sets, we managed to annotate each tweet with a place name (i.e. "Stockport"), a social place name (i.e. "pub", "bank") and a social category (i.e. "man", "woman", "landlord"...) The code is simple; the biggest work in it was the dictionary of keywords:
#!/usr/bin/python
# categorise - Annotate JSON elements with categories
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
# Copyright (C) 2010  Yuwei Lin <yuwei@ylin.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import sys, math
import simplejson
import re
# Electoral wards from http://en.wikipedia.org/wiki/List_of_electoral_wards_in_Greater_Manchester
placenames = ["Altrincham", "Sale West",
"Altrincham", "Ashton upon Mersey", "Bowdon", "Broadheath", "Hale Barns", "Hale Central", "St Mary", "Timperley", "Village",
"Ashton-under-Lyne",
"Ashton Hurst", "Ashton St Michael", "Ashton Waterloo", "Droylsden East", "Droylsden West", "Failsworth East", "Failsworth West", "St Peter",
"Blackley", "Broughton",
"Broughton", "Charlestown", "Cheetham", "Crumpsall", "Harpurhey", "Higher Blackley", "Kersal",
"Bolton North East",
"Astley Bridge", "Bradshaw", "Breightmet", "Bromley Cross", "Crompton", "Halliwell", "Tonge with the Haulgh",
"Bolton South East",
"Farnworth", "Great Lever", "Harper Green", "Hulton", "Kearsley", "Little Lever", "Darcy Lever", "Rumworth",
"Bolton West",
"Atherton", "Heaton", "Lostock", "Horwich", "Blackrod", "Horwich North East", "Smithills", "Westhoughton North", "Chew Moor", "Westhoughton South",
"Bury North",
"Church", "East", "Elton", "Moorside", "North Manor", "Ramsbottom", "Redvales", "Tottington",
"Bury South",
"Besses", "Holyrood", "Pilkington Park", "Radcliffe East", "Radcliffe North", "Radcliffe West", "St Mary", "Sedgley", "Unsworth",
"Cheadle",
"Bramhall North", "Bramhall South", "Cheadle", "Gatley", "Cheadle Hulme North", "Cheadle Hulme South", "Heald Green", "Stepping Hill",
"Denton", "Reddish",
"Audenshaw", "Denton North East", "Denton South", "Denton West", "Dukinfield", "Reddish North", "Reddish South",
"Hazel Grove",
"Bredbury", "Woodley", "Bredbury Green", "Romiley", "Hazel Grove", "Marple North", "Marple South", "Offerton",
"Heywood", "Middleton",
"Bamford", "Castleton", "East Middleton", "Hopwood Hall", "Norden", "North Heywood", "North Middleton", "South Middleton", "West Heywood", "West Middleton",
"Leigh",
"Astley Mosley Common", "Atherleigh", "Golborne", "Lowton West", "Leigh East", "Leigh South", "Leigh West", "Lowton East", "Tyldesley",
"Makerfield",
"Abram", "Ashton", "Bryn", "Hindley", "Hindley Green", "Orrell", "Winstanley", "Worsley Mesnes",
"Manchester Central",
"Ancoats", "Clayton", "Ardwick", "Bradford", "City Centre", "Hulme", "Miles Platting", "Newton Heath", "Moss Side", "Moston",
"Manchester", "Gorton",
"Fallowfield", "Gorton North", "Gorton South", "Levenshulme", "Longsight", "Rusholme", "Whalley Range",
"Manchester", "Withington",
"Burnage", "Chorlton", "Chorlton Park", "Didsbury East", "Didsbury West", "Old Moat", "Withington",
"Oldham East", "Saddleworth",
"Alexandra", "Crompton", "Saddleworth North", "Saddleworth South", "Saddleworth West", "Lees", "St James", "St Mary", "Shaw", "Waterhead",
"Oldham West", "Royton",
"Chadderton Central", "Chadderton North", "Chadderton South", "Coldhurst", "Hollinwood", "Medlock Vale", "Royton North", "Royton South", "Werneth",
"Rochdale",
"Balderstone", "Kirkholt", "Central Rochdale", "Healey", "Kingsway", "Littleborough Lakeside", "Milkstone", "Deeplish", "Milnrow", "Newhey", "Smallbridge", "Firgrove", "Spotland", "Falinge", "Wardle", "West Littleborough",
"Salford", "Eccles",
"Claremont", "Eccles", "Irwell Riverside", "Langworthy", "Ordsall", "Pendlebury", "Swinton North", "Swinton South", "Weaste", "Seedley",
"Stalybridge", "Hyde",
"Dukinfield Stalybridge", "Hyde Godley", "Hyde Newton", "Hyde Werneth", "Longdendale", "Mossley", "Stalybridge North", "Stalybridge South",
"Stockport",
"Brinnington", "Central", "Davenport", "Cale Green", "Edgeley", "Cheadle Heath", "Heatons North", "Heatons South", "Manor",
"Stretford", "Urmston",
"Bucklow-St Martins", "Clifford", "Davyhulme East", "Davyhulme West", "Flixton", "Gorse Hill", "Longford", "Stretford", "Urmston",
"Wigan",
"Aspull New Springs Whelley", "Douglas", "Ince", "Pemberton", "Shevington with Lower Ground", "Standish with Langtree", "Wigan Central", "Wigan West",
"Worsley", "Eccles South",
"Barton", "Boothstown", "Ellenbrook", "Cadishead", "Irlam", "Little Hulton", "Walkden North", "Walkden South", "Winton", "Worsley",
"Wythenshawe", "Sale East",
"Baguley", "Brooklands", "Northenden", "Priory", "Sale Moor", "Sharston", "Woodhouse Park"]
# Manual coding from Yuwei
placenames.extend(["City centre", "Tameside", "Oldham", "Bury", "Bolton",
"Trafford", "Pendleton", "New Moston", "Denton", "Eccles", "Leigh", "Benchill",
"Prestwich", "Sale", "Kearsley", ])
placenames.extend(["Trafford", "Bolton", "Stockport", "Levenshulme", "Gorton",
"Tameside", "Blackley", "City centre", "Airport", "South Manchester",
"Rochdale", "Chorlton", "Uppermill", "Castleton", "Stalybridge", "Ashton",
"Chadderton", "Bury", "Ancoats", "Whalley Range", "West Yorkshire",
"Fallowfield", "New Moston", "Denton", "Stretford", "Eccles", "Pendleton",
"Leigh", "Altrincham", "Sale", "Prestwich", "Kearsley", "Hulme", "Withington",
"Moss Side", "Milnrow", "outskirt of Manchester City Centre", "Newton Heath",
"Wythenshawe", "Mancunian Way", "M60", "A6", "Droylesden", "M56", "Timperley",
"Higher Ince", "Clayton", "Higher Blackley", "Lowton", "Droylsden",
"Partington", "Cheetham Hill", "Benchill", "Longsight", "Didsbury",
"Westhoughton"])
# Social categories from Yuwei
soccat = ["man", "woman", "men", "women", "youth", "teenager", "elderly",
"patient", "taxi driver", "neighbour", "male", "tenant", "landlord", "child",
"children", "immigrant", "female", "workmen", "boy", "girl", "foster parents",
"next of kin"]
for i in range(100):
    soccat.append("%d-year-old" % i)
    soccat.append("%d-years-old" % i)
# Types of social locations from Yuwei
socloc = ["car park", "park", "pub", "club", "shop", "premises", "bus stop",
"property", "credit card", "supermarket", "garden", "phone box", "theatre",
"toilet", "building site", "Crown court", "hard shoulder", "telephone kiosk",
"hotel", "restaurant", "cafe", "petrol station", "bank", "school",
"university"]
extras =   "placename": placenames, "soccat": soccat, "socloc": socloc  
# Normalise keyword lists
for k, v in extras.iteritems():
    # Remove duplicates
    v = list(set(v))
    # Sort by length
    v.sort(key=lambda x:len(x), reverse=True)
# Add keywords
def add_categories(tweet):
    text = tweet["text"].lower()
    for field, categories in extras.iteritems():
        for cat in categories:
            if cat.lower() in text:
                tweet[field] = cat
                break
    return tweet
# Read all the twits
records = (add_categories(x) for x in simplejson.load(sys.stdin))
simplejson.dump(list(records), sys.stdout)
All these scripts form a nice processing chain: each script takes a list of JSON records, adds some bit and passes it on. In order to see what we have so far, here is a simple script to convert the JSON twits to CSV so they can be viewed in a spreadsheet:
#!/usr/bin/python
# Convert the JSON twits to CSV
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import sys
import csv
rows = ["id", "created_at", "text", "keywords", "placename"]
writer = csv.writer(sys.stdout)
for rec in simplejson.load(sys.stdin):
    rec["keywords"] = " ".join(rec["keywords"])
    rec["placename"] = rec.get("placename", "")
    writer.writerow([rec[row] for row in rows])
At this point we were coming up with lots of questions: "were there more reports on women or men?", "which place had most incidents?", "what were the incidents involving animals?"... Time to bring Xapian into play. This script reads all the JSON tweets and builds a Xapian index with them:
#!/usr/bin/python
# toxapian - Index JSON tweets in Xapian
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import simplejson
import sys
import os, os.path
import xapian
DBNAME = sys.argv[1]
db = xapian.WritableDatabase(DBNAME, xapian.DB_CREATE_OR_OPEN)
stemmer = xapian.Stem("english")
indexer = xapian.TermGenerator()
indexer.set_stemmer(stemmer)
indexer.set_database(db)
data = simplejson.load(sys.stdin)
for rec in data:
    doc = xapian.Document()
    doc.set_data(str(rec["id"]))
    indexer.set_document(doc)
    indexer.index_text_without_positions(rec["text"])
    # Index categories as categories
    if "placename" in rec:
        doc.add_boolean_term("XP" + rec["placename"].lower())
    if "soccat" in rec:
        doc.add_boolean_term("XS" + rec["soccat"].lower())
    if "socloc" in rec:
        doc.add_boolean_term("XL" + rec["socloc"].lower())
    db.add_document(doc)
db.flush()
# Also save the whole dataset so we know where to find it later if we want to
# show the details of an entry
simplejson.dump(data, open(os.path.join(DBNAME, "all.json"), "w"))
And this is a simple command line tool to query to the database:
#!/usr/bin/python
# xgrep - Command line tool to query the GMP24 tweet Xapian database
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import simplejson
import sys
import os, os.path
import xapian
DBNAME = sys.argv[1]
db = xapian.Database(DBNAME)
stem = xapian.Stem("english")
qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
qp.add_boolean_prefix("place", "XP")
qp.add_boolean_prefix("soc", "XS")
qp.add_boolean_prefix("loc", "XL")
query = qp.parse_query(sys.argv[2],
    xapian.QueryParser.FLAG_BOOLEAN  
    xapian.QueryParser.FLAG_LOVEHATE  
    xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE  
    xapian.QueryParser.FLAG_WILDCARD  
    xapian.QueryParser.FLAG_PURE_NOT  
    xapian.QueryParser.FLAG_SPELLING_CORRECTION  
    xapian.QueryParser.FLAG_AUTO_SYNONYMS)
enquire = xapian.Enquire(db)
enquire.set_query(query)
count = 40
matches = enquire.get_mset(0, count)
estimated = matches.get_matches_estimated()
print "%d/%d results" % (matches.size(), estimated)
data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))
for m in matches:
    rec = data[m.document.get_data()]
    print rec["text"]
print "%d/%d results" % (matches.size(), matches.get_matches_estimated())
total = db.get_doccount()
estimated = matches.get_matches_estimated()
print "%d results over %d documents, %d%%" % (estimated, total, estimated * 100 / total)
Neat! Now that we have a proper index that supports all sort of cool things, like stemming, tag clouds, full text search with complex queries, lookup of similar documents, suggest keywords and so on, it was just fair to put together a web service to share it with other people at the event. It helped that I had already written similar code for apt-xapian-index and dde before. Here is the server, quickly built on bottle. The very last line starts the server and it is where you can configure the listening interface and port.
#!/usr/bin/python
# xserve - Make the GMP24 tweet Xapian database available on the web
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import bottle
from bottle import route, post
from cStringIO import StringIO
import cPickle as pickle
import simplejson
import sys
import os, os.path
import xapian
import urllib
import math
bottle.debug(True)
DBNAME = sys.argv[1]
QUERYLOG = os.path.join(DBNAME, "queries.txt")
data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))
prefixes =   "place": "XP", "soc": "XS", "loc": "XL"  
prefix_desc =   "place": "Place name", "soc": "Social category", "loc": "Social location"  
db = xapian.Database(DBNAME)
stem = xapian.Stem("english")
qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
for k, v in prefixes.iteritems():
    qp.add_boolean_prefix(k, v)
def make_query(qstring):
    return qp.parse_query(qstring,
        xapian.QueryParser.FLAG_BOOLEAN  
        xapian.QueryParser.FLAG_LOVEHATE  
        xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE  
        xapian.QueryParser.FLAG_WILDCARD  
        xapian.QueryParser.FLAG_PURE_NOT  
        xapian.QueryParser.FLAG_SPELLING_CORRECTION  
        xapian.QueryParser.FLAG_AUTO_SYNONYMS)
@route("/")
def index():
    query = urllib.unquote_plus(bottle.request.GET.get("q", ""))
    out = StringIO()
    print >>out, '''
<html>
<head>
<title>Query</title>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
<script type="text/javascript">
$(function() 
    $("#queryfield")[0].focus()
 )
</script>
</head>
<body>
<h1>Search</h1>
<form method="POST" action="/query">
Keywords: <input type="text" name="query" value="%s" id="queryfield">
<input type="submit">
<a href="http://xapian.org/docs/queryparser.html">Help</a>
</form>''' % query
    print >>out, '''
<p>Example: "car place:wigan"</p>

<p>Available prefixes:</p>

<ul>
'''
    for pfx in prefixes.keys():
        print >>out, "<li><a href='/catinfo/%s'>%s - %s</a></li>" % (pfx, pfx, prefix_desc[pfx])
    print >>out, '''
</ul>
'''
    oldqueries = []
    if os.path.exists(QUERYLOG):
        total = db.get_doccount()
        fd = open(QUERYLOG, "r")
        while True:
            try:
                q = pickle.load(fd)
            except EOFError:
                break
            oldqueries.append(q)
        fd.close()
        def print_query(q):
            count = q["count"]
            print >>out, "<li><a href='/query?query=%s'>%s (%d/%d %.2f%%)</a></li>" % (urllib.quote_plus(q["q"]), q["q"], count, total, count * 100.0 / total)
        print >>out, "<p>Last 10 queries:</p><ul>"
        for q in oldqueries[:-10:-1]:
            print_query(q)
        print >>out, "</ul>"
        # Remove duplicates
        oldqueries = dict(((x["q"], x) for x in oldqueries)).values()
        print >>out, "<table>"
        print >>out, "<tr><th>10 queries with most results</th><th>10 queries with least results</th></tr>"
        print >>out, "<tr><td>"
        print >>out, "<ul>"
        oldqueries.sort(key=lambda x:x["count"], reverse=True)
        for q in oldqueries[:10]:
            print_query(q)
        print >>out, "</ul>"
        print >>out, "</td><td>"
        print >>out, "<ul>"
        nonempty = [x for x in oldqueries if x["count"] > 0]
        nonempty.sort(key=lambda x:x["count"])
        for q in nonempty[:10]:
            print_query(q)
        print >>out, "</ul>"
        print >>out, "</td></tr>"
        print >>out, "</table>"
    print >>out, '''
</body>
</html>'''
    return out.getvalue()
@route("/query")
@route("/query/")
@post("/query")
@post("/query/")
def query():
    query = bottle.request.POST.get("query", bottle.request.GET.get("query", ""))
    enquire = xapian.Enquire(db)
    enquire.set_query(make_query(query))
    count = 40
    matches = enquire.get_mset(0, count)
    estimated = matches.get_matches_estimated()
    total = db.get_doccount()
    out = StringIO()
    print >>out, '''
<html>
<head><title>Results</title></head>
<body>
<h1>Results for "<b>%s</b>"</h1>
''' % query
    if estimated == 0:
        print >>out, "No results found."
    else:
        # Give as results the first 30 documents; also use them as the key
        # ones to use to compute relevant terms
        rset = xapian.RSet()
        for m in enquire.get_mset(0, 30):
            rset.add_document(m.document.get_docid())
        # Compute the tag cloud
        class NonTagFilter(xapian.ExpandDecider):
            def __call__(self, term):
                return not term[0].isupper() and not term[0].isdigit()
        cloud = []
        maxscore = None
        for res in enquire.get_eset(40, rset, NonTagFilter()):
            # Normalise the score in the interval [0, 1]
            weight = math.log(res.weight)
            if maxscore == None: maxscore = weight
            tag = res.term
            cloud.append([tag, float(weight) / maxscore])
        max_weight = cloud[0][1]
        min_weight = cloud[-1][1]
        cloud.sort(key=lambda x:x[0])
        def mklink(query, term):
            return "/query?query=%s" % urllib.quote_plus(query + " and " + term)
        print >>out, "<h2>Tag cloud</h2>"
        print >>out, "<blockquote>"
        for term, weight in cloud:
            size = 100 + 100.0 * (weight - min_weight) / (max_weight - min_weight)
            print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(query, term), size, term)
        print >>out, "</blockquote>"
        print >>out, "<h2>Results</h2>"
        print >>out, "<p><a href='/'>Search again</a></p>"
        print >>out, "<p>%d results over %d documents, %.2f%%</p>" % (estimated, total, estimated * 100.0 / total)
        print >>out, "<p>%d/%d results</p>" % (matches.size(), estimated)
        print >>out, "<ul>"
        for m in matches:
            rec = data[m.document.get_data()]
            print >>out, "<li><a href='/item/%s'>%s</a></li>" % (rec["id"], rec["text"])
        print >>out, "</ul>"
        fd = open(QUERYLOG, "a")
        qinfo = dict(q=query, count=estimated)
        pickle.dump(qinfo, fd)
        fd.close()
    print >>out, '''
<a href="/">Search again</a>

</body>
</html>'''
    return out.getvalue()
@route("/item/:id")
@route("/item/:id/")
def show(id):
    rec = data[id]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Result %s</title></head>
<body>
<h1>Raw JSON record for twit %s</h1>
<pre>''' % (rec["id"], rec["id"])
    print >>out, simplejson.dumps(rec, indent=" ")
    print >>out, '''
</pre>
</body>
</html>'''
    return out.getvalue()
@route("/catinfo/:name")
@route("/catinfo/:name/")
def catinfo(name):
    prefix = prefixes[name]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Values for %s</title></head>
<body>
''' % name
    terms = [(x.term[len(prefix):], db.get_termfreq(x.term)) for x in db.allterms(prefix)]
    terms.sort(key=lambda x:x[1], reverse=True)
    freq_min = terms[0][1]
    freq_max = terms[-1][1]
    def mklink(name, term):
        return "/query?query=%s" % urllib.quote_plus(name + ":" + term)
    # Build tag cloud
    print >>out, "<h1>Tag cloud</h1>"
    print >>out, "<blockquote>"
    for term, freq in sorted(terms[:20], key=lambda x:x[0]):
        size = 100 + 100.0 * (freq - freq_min) / (freq_max - freq_min)
        print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(name, term), size, term)
    print >>out, "</blockquote>"
    print >>out, "<h1>All terms</h1>"
    print >>out, "<table>"
    print >>out, "<tr><th>Occurrences</th><th>Name</th></tr>"
    for term, freq in terms:
        print >>out, "<tr><td>%d</td><td><a href='/query?query=%s'>%s</a></td></tr>" % (freq, urllib.quote_plus(name + ":" + term), term)
    print >>out, "</table>"
    print >>out, '''
</body>
</html>'''
    return out.getvalue()
# Change here for bind host and port
bottle.run(host="0.0.0.0", port=8024)
...and then we presented our work and ended up winning the contest. This was the story of how we wrote this set of award winning code.

8 October 2010

Russell Coker: Choosing an Android Phone

My phone contract ends in a few months, so I m looking at getting a new Android phone. I want a big Android phone (in both physical size and resolution) that has a physical keyboard, a digital compass, A-GPS and at least a 5MP camera with geo-tagging. I want to be able to read PDF files and run ssh sessions, so a big screen is required and a physical keyboard avoids wasting screen space for a soft-keyboard. My pockets will fit something about 10.5cm wide by 17cm high but I don t expect anyone to manufacture such a large phone. High resolution is a good thing too, it seems that the best available at the moment is 854*480 (with 800*480 being reasonably common). I want Wifi and all the 3G and GSM data transfer standards. It would be ideal to have a phone with the dual networking stack needed to do both voice and data at the same time. I m not interested in anything that runs a version of Android older than 2.2 as native tethering is important. An option to upgrade to post 2.2 would be a really good thing. Here are the nearest options I could find:
Phone Resolution Screen Size (inches) Camera Resolution Notes
Motorola Milestone 854*480 3.7 5MP
Motorola Droid 854*480 3.7 5MP
LG VS 740 800*480 3.2 3.2MP no GPS or compass
Lenovo LePhone 800*480 3.7 3MP no GPS or compass
It seems that Motorola makes the phones that best suit my needs, does anyone know of any better options?

3 October 2010

Matt Zimmerman: Tips for frequent international travel

I travel pretty regularly, about 35% so far in 2010. When it goes wrong, travel can be exhausting, frustrating, complicated, stressful and even debilitating. I m always looking for ways to make my trips run more smoothly. On a recent flight to Taipei, I wrote down a few of the techniques which I ve successfully put into practice and found helpful. This is not an exhaustive list; I ve omitted a lot of the common and obvious tips I ve seen elsewhere.
  1. Make a packing list. This one may be obvious, but a lot of people neglect it. Perhaps they think making lists is boring and fussy, but really, it isn t. Without a packing list, it s easy to forget to do the things which will make your trip better. Use it every time, and bring a copy with you (or store it online) so you can add the things you wish you had brought or done. A simple, ever-improving packing list is the most effective technique I have found for making travel less stressful and more enjoyable.
  2. Carry a water bottle with a tight-fitting lid. I use a 32oz Nalgene bottle, which fits nicely into the seat next to me or under an armrest, and gives me enough water for even the longest flights. I fill it up after passing through security, at a cafe, bar or lounge, and generally decline the beverages offered by the cabin crew. Staying hydrated helps me feel better during the flight, and leaves me with less malaise when I arrive. I don t need to manage a tray table or armrest full of cups and other debris, so I can sit more comfortably, with the tray table folded away.
  3. Consolidate essential items using multipurpose equipment. For example, invest in a power adapter which has USB sockets onboard, and carry USB cables instead of wall chargers. Versatile items like this save on space and weight. I can charge two devices this way, but the equipment is smaller and lighter than even a single wall charger.
  4. Learn how to sleep on an airplane. Getting some sleep on a long flight really helps to offset the effects of traveling. There are several resources out there with practical advice on how to do it. One thing which really helped me was to buy a high quality eye mask which blocks out all of the light in the cabin. The one I use looks a little funny and is not cheap, but is very comfortable and effective. It s made of memory foam with a soft, washable cover and works much better than the ones the airlines give away for free. I no longer bother with a neck pillow, and use the flaps built into the seat to lean my head against. I m surprised at how many people don t know about this common aricraft feature: virtually every long-haul seat has something like this, even in economy, though it may not be obvious how to use it.
  5. Buy duplicates of things like toiletries, and keep them in your travel kit so you don t need to pack your everyday items (and risk forgetting them) each time. The less packing you need to do, the less time it will take, and the less opportunity there is for mistakes. This also saves time unpacking when you get home, and lets you buy a smaller size of the item where available.
  6. Optimize border crossings Carry the forms you ll need for customs, immigration, etc. in your carry-on. They don t always provide them at the counter or on board the plane, and it s a hassle to rush to fill in the form at the last minute. If you have a few of them with you, you can fill them out early (perhaps even before you fly) and then hustle to the front of the queue. For countries you enter frequently (especially your home country), programs like Global Entry (US) and IRIS (UK) will save you a lot of time by allowing you to use an automated kiosk to cross the border.

29 September 2010

Russell Coker: Links September 2010

Kevin Stone gave an interesting TED talk about the biological future of joint replacement [1]. Using stem cells and animal tissue which has been prepared to destroy the chemicals that trigger immune responses the tissues can regrow. Replacing joints with titanium and ceramic lets people walk again, regrowing them with Kevin s methods allows them to compete at the highest levels of sporting contests! Derek Sivers gave a brief and interesting TED talk advising people to keep their goals secret if they want to achieve them [2]. The Parrot AR.Drone is an interesting device [3], it s a four-propeller helicopter that is controlled by WiFi. Apparently there is little security (it binds to the first WiFi client it sees) which is a significant down-side. It returns a video feed to the controlling iPhone as it flys and can hover when it loses it s connection. It will be interesting to see when people write software for other devices (Android etc). Also I wonder whether there will be open source weapons kits for it. If you could have those devices use either a nerf gun or a lance to attack each other s turbines then you could have drone jousting. Don Marti had an interesting new idea for a crime game [4]. The main concept is to use the element of mistrust that is present in real criminal gangs. The new guy you invite to join a job might inform the police and you won t know for sure. Sometimes a heist will be discovered by the police through bad luck (or good police work) and you will wonder whether there was an informant. The aim is for a simple game design and with the complexity in email discussions between the players. The C64 isn t dead, it s even on the Internet [5], an original C64 is running a web site! Tan Li gave an interesting TED talk about a new headset to read brain-waves [6]. The device in question can be applied in minutes, requires no gel or abrasion of the scalp, connects to the computer wirelessly and is relatively cheap at $300US. The developer s kit (which I think includes a headset) is $500US. I wonder if the community can develop a cheaper version of this which is more open. Lisa Margonelli gave an interesting TED talk about the politics of oil [7]. One of her insightful points is that the subsidies for oil should be shifted from the oil industry to middle-class consumers. But she goes off track a bit by suggesting gradual oil tax increases until 2020, according to the best estimates of groups such as the CSIRO they won t need to have taxes to give a high oil price in 2020! She is aiming for a 20% reduction in petrol use by 2020, but I m not aware of any serious group of scientists who have evidence to suggest that the production capacity in 2020 will be close to 80% what it is now. Slate has a good article about The Innocence Project which uses DNA tests to overturn false convictions [8], it s scarey how badly the justice system works. Rachel Sussman gave an interesting talk about the World s Oldest Living Things [9], nothing less than 2000 years old is included. Nicholas Negroponte gave an interesting EG 2007 talk about the OLPC project [10]. While some of the content about OLPC production is a little dated the OLPC history is relevant and his personal opinions regarding the benefits that children receive from computers are insightful. Jayne Poynter gave an interesting TED talk about life in Biosphere 2 [11]. Her work on mini-biospheres is also interesting. Let s hope we get a permanent Biosphere running outside the Earth sometime soon. Sheryl WuDunn gave an informative TED talk titled Our Century s Greatest Injustice about the plight of women in most of the world [12]. Daniel Kahn Gillmor wrote a good article about the use of ssh-agent [13]. You really should use it to secure your ssh keys. Mark Shuttleworth has described the development process for the Ubuntu font [14]. This is a very interesting project and IMHO one of the most significant things that Ubuntu has done. Prior to this an entirely closed development process has been used for font development. Now we are getting a major new font family developed with a free and open process and with some new legal developments for a font license! One thing to note that this project appears to have involved a lot of work from professional font designers, it sounds like Canonical spent a significant amount of money on this project.

28 September 2010

Lior Kaplan: Welcome The Document Foundation and LibreOffice

For those who haven t heard: OpenOffice.org Community announces The Document Foundation
The community of volunteers developing and promoting OpenOffice.org sets up an independent Foundation to drive the further growth of the project The Internet, September 28, 2010 The community of volunteers who develop and promote OpenOffice.org, the leading free office software, announce a major change in the project s structure. After ten years successful growth with Sun Microsystems as founding and principle sponsor, the project launches an independent foundation called The Document Foundation , to fulfil the promise of independence written in the original charter. The Foundation will be the cornerstone of a new ecosystem where individuals and organisations can contribute to and benefit from the availability of a truly free office suite. It will generate increased competition and choice for the benefit of customers and drive innovation in the office suite market. From now on, the OpenOffice.org community will be known as The Document Foundation . Oracle, who acquired OpenOffice.org assets as a result of its acquisition of Sun Microsystems, has been invited to become a member of the new Foundation, and donate the brand the community has grown during the past ten years. Pending this decision, the brand LibreOffice has been chosen for the software going forward. The Document Foundation is the result of a collective effort by leading independent members of the OpenOffice.org community, including several project leads and key members of the Community Council. It will be led initially by a Steering Committee of developers and national language projects managers. The Foundation aims to lower the barrier of adoption for both users and developers, to make LibreOffice the most accessible office suite ever. The Foundation will coordinate and oversee the development of LibreOffice, which is available in beta version at the placeholder site: http://www.libreoffice.org. Developers are invited to join the project and contribute to the code in the new friendly and open environment, to shape the future of office productivity suites alongside contributors who translate, test, document, support, and promote the software. Speaking for the group of volunteers, Sophie Gautier a veteran of the community and the former maintainer of the French speaking language project has declared: We believe that the Foundation is a key step for the evolution of the free office suite, as it liberates the development of the code and the evolution of the project from the constraints represented by the commercial interests of a single company. Free software advocates around the world have the extraordinary opportunity of joining the group of founding members today, to write a completely new chapter in the history of FLOSS . [...]
So welcome The Document Foundation and LibreOffice. I already can see how does the community benefits from this due to the merge with go-oo:
Q: What does this announcement mean to other derivatives of OpenOffice.org?
A: We want The Document Foundation to be open to code contributions from as many people as possible. We are delighted to announce that the enhancements produced by the Go-OOo team will be merged into LibreOffice, effective immediately. We hope that others will follow suit.
(from http://www.documentfoundation.org/faq/) So, will Oracle contribute the OpenOffice.org trademark to the community ? Time will tell
Filed under: LibreOffice, Openoffice.org, Proud to use free software

18 September 2010

Evgeni Golov: How to bypass DNS-based spam-filters using DNS

I ve been sitting on this one since more than a month. I ve contacted upstream on the 19.08.2010 and the Debian maintainer on the 02.09.2010. No reaction from them till today, and no, my spamfilter does not eat mail ;) Still, I won t tell you the name of the software (but you could easily guess or check ) So let s put the grey hat on and begin ;) Foreword: when I write spam-filter, I mean some DNSBL/SPF/blah filter, not SpamAssassin, crm114 or other content-based filters. common spam-filters There are plenty of spam-filters out there which work after the following schema:
for check in checks:
    if check(mail) == BAD:
        reject(mail)
        break
And the checks often look like this:
def check(mail):
    errors = False
    result = some_magic(mail)
    if not errors: return result
    else: return GOOD
Here some_magic is a function that do the actual checks (DNSBL lookups etc) and which stores possible happened errors in errors. If an error occurred, it s safer to say the mail was good (or at least not bad) than bad. the problem This looks like a sane approach on the first sight: iterate over all checks and execute them until a mail is found to be spam (and the corresponding check did not end up in an error (imagine some weird DNS error for DNSBL or SPF checks here)). But then the author realizes that the errors might be global , like with DNS: when the first check notices that it can t resolve anything (ie. it ran into a DNS timeout three times in a row), the second check (which also utilizes DNS queries) will most probably result in an error too. After this he adds a global counter for DNS errors and skips all the checks that use DNS if this counter is >=3. Great idea , you say, saving resources is good . Bad idea , I say The reason is simple: DNSError != DNSError. Why? The local resolver might be broken, then every DNS query will end up with an error and DNS-based checks should be really disabled. The first three (out of eg. ten) DNSBLs might be down, but that s not a reason to skip the other seven and the SPF check. Etc Now let s assume all configured DNSBLs are working properly, so is the local resolver and the checks are:
checks = [check_HELO, check_SPF, check_DNSBL]
How could one attack this filter? the attack Set up the following zone
spam.example.com IN NS ns1.example.com
spam.example.com IN NS ns2.example.com
spam.example.com IN NS ns3.example.com
spam2.example.com IN MX mx1.spam.example.com
spam2.example.com IN MX mx2.spam.example.com
spam2.example.com IN MX mx3.spam.example.com
spam2.example.com IN MX mx4.spam.example.com
ns1.example.com IN A 10.1.1.1
ns2.example.com IN A 10.2.2.2
ns3.example.com IN A 10.3.3.3
and greet the to-be-spammed-MTA with EHLO client1.spam.example.com
What happens? The spam-filter tries to resolve client1.spam.example.com in check_HELO to check whether the hostname matches the IP-address the connection is coming from, fails three times (there are no reachable DNS servers in 10.0.0.0/8) and continues with the next two checks, but these are skipped because of previous DNS errors. As no checks could identify the mail as being spam, the mail is delivered to the users mailbox. With some creativity this can also be used in SPF records (via spam2.example.com), sender-domains etc. conclusion If you want to save resources by skipping DNS-based checks after another (DNS-based) check failed don t! Think whether this is really the same error you are seeing Additionally this can be used as a sort of DoS against the not said software: according to the logs each of my tries needed between 30 and 36 seconds (!) to complete imagine some hundred simultaneous requests the end You think I did something wrong with that post? Throw the first stone then, I don t care.

7 September 2010

Brett Parker: Life, the Universe, and Everything (or something)

Forgive me readers, for I have sinned... or at least, not blogged for a looooong time. I've been reasonably busy, stopped working for Runtime Collective and started contracting for a small company in Portsmouth, it's been somewhat hectic, and the commute isn't brilliant, but it has given me a lot of time for reading... Really, mostly a lot of my time during the week is commuting, working, reading and sleeping, haven't had much time for anything else, and have mostly been knackered... I'm getting in to the habit now, though, so am getting a bit more time for getting on with other things. I have, however, gained a lot more fiction books in the last few months, and am averaging 2 or 3 books a week at the moment. Most enjoyed the first 2 books of the Demon Trilogy by Peter V Brett - have to wait till 2012 to get the last book, though - which is a loooong time! Actually, I seem to have made a habit of ending up starting trilogys recently where not all of the books are published, in the case of Feed by Mira Grant I managed to really screw up - not even the second book is due out until next year - but the first is really quite good. I've also got the first book of Mark C Newton's Legend of the Red Sun books, really enjoyed that, but am waiting for the second in paperback, as they're much easier to carry around and use on the tube. Have read the Millennium trilogy, and thoroughly enjoyed it. The first film, although a reasonable adaptation of the book, misses some good parts of the book, and doesn't leave you feeling as if you know the characters as much. Am also planning on working through all the Laundry books by Charles Stross, I picked up The Fuller Memorandum on a whim (OK, it was on a 2 for 1 in Waterstones...) and really enjoyed it, so am going to get the rest (when I start getting low on books again - I've currently got another 5 queued up, so it might not be for a couple of weeks). So, other than that, what have I been up to? Well, played the last bits of DLC for Assassins Creed 2 on the PS3, and thoroughly looking forward to the release of AC3, played most of the way through Lego Harry Potter, wandered about and saw people... you know, the usual things! Oh, and of course, August bank holiday weekend (28-30) were spent with good peoples at the ever fantastic Debian UK BBQ where much Nero was imbibed, many people spoken to, and some gpg signing done - as always many thanks to Steve McIntyre for a fantastic weekend, to Collabora and Mythic Beasts for the beer, and to antibodyMX and Coding Craft Ltd for the food (Oi, Pashley , stop being crap and sort out your company website...).

13 August 2010

Daniel Stone: lca 2011 cfp closing soon!

Just a reminder that the linux.conf.au 2011 call for papers is closing THIS SATURDAY, 14th August! If you haven't already submitted your talk, get in now while you still can ...

4 August 2010

Andrew Pollock: [life] Zoe at 3 months

Again, there's been this surreal time warp effect. 3 months has passed, and it feels like both a long time and a blink of an eye. It more feels like a long time though. I've mostly forgotten what it was like before Zoe came along. Zoe is growing well. She weighs around 11 lbs 11 oz (5.3 kg), and I swear she's getting very long. When I'm holding her in my arms, there's legs everywhere. She started being able to roll from her stomach onto her back at 2 months, and her neck strength is pretty good now. I guess her next significant milestone will be being able to sit up properly. When she's on her tummy, she can prop herself up on her elbows and look around. She smiles all the time. We had a few nights of 8 hours of sleep, then everything went completely out the window, and we had about a week of very disrupted sleep at night. I think she's starting to settle into a routine now though, with a few 5 hour stints the last few nights. Her US passport arrived yesterday. We still need to get everything together to lodge her Australian citizenship application, which is the first step towards getting her an Australian passport. Sarah's back studying now (two subjects this semester), which is proving to be a bit of a juggling act. Zoe with her shiny new US passport

2 August 2010

Matt Zimmerman: DebConf 10: Day 2

Today was the first day of DebConf proper, where all of the sessions were aimed at project participants. Bits from the DPL (Stefano Zacchiroli) Stefano delivered an excellent address to the Debian project. As Project Leader, he offered a perspective on how far Debian has come, raised some of the key questions facing Debian today, and challenged the project to move forward and improve in several important ways. He asked the audience: Is Debian better than other distributions? Is Debian still relevant? Why/how? Having asked this question on identi.ca and Twitter recently, he presented a summary. There was a fairly standard list of technical concerns, but also: He pointed out some areas which we would like to see improve, including: All in all, I thought this was an accurate, timely and inspirational message for the project, and the talk is worth watching for any current or prospective contributor to Debian. Debian Policy BoF (Russ Albery) Russ facilitated a discussion about the Debian policy document itself and the process for managing it. He has recently put in a lot of time working on the backlog (down from 160+ to 120), but this is not sustainable for him, and help is needed. There was a wide-ranging discussion of possible improvements including: There was also some discussion in passing of the long-standing confusion (presumably among people new to the project) with regard to how policy is established. In Debian, best practices are first implemented in packages, then documented in policy (not the reverse). Sometimes, improvements are suggested at the policy level, when they need to start elsewhere. I m not very familiar with how the policy manual is maintained at present, but listening to the discussion, it sounded like it might help to extend the process to include the implementation stage. This would allow standards improvements to be tracked all the way through from concept, to implementation, to documentation. The Java Packaging Nightmare (Torsten Werner) Torsten described the current state of Java packaging in Debian and the general problems involved, including licensing issues, build system challenges (e.g. maven) and dependency management. His slides were information-dense, so I didn t take a lot of notes. His presentation inspired a lively discussion about why upstream developers of Java applications and libraries often do not engage with Debian. Suggested reasons included: Collaboration between Ubuntu and Debian (Jorge Castro) Jorge talked about the connections between Debian and Ubuntu, how people in the projects perceive each other, and how to foster good relationships between developers. He talked about past efforts to quantify collaboration between the projects, but the focus is now on building personal relationships. There were many good questions and comments afterward, and I m looking forward to the Debian derivatives BoF session tomorrow to get into more detail. Tonight is the traditional wine and cheese party. When this tradition started, I was one of just a handful of people in a room with some cheese and paper plates, but it s now a large social gathering with contributions of cheese and wine from around the world. I m looking forward to it.

19 July 2010

Daniel Stone: sgi summer interns write like

So, #collabora was putting chunks of code into iwritelike.com, and I decided to join in the fun. While Telepathy, GStreamer and other similarly soft projects are all apparently written like Dan Brown or David Foster Wallace, I put a chunk of xkbActions.c into iwritelike.com, and apparently it's written like Chuck Norris. Brilliant.

18 July 2010

Rog rio Brito: I am going to DebConf 10

I just bought my tickets today to NYC and I am glady to say that I am going to Debconf 10.
I am going to DebConf10

I am going to DebConf10

The tickets have this information: This represents a personal milestone for me, as I have many plans for the conference. In fact, this will be a trip of many firsts for me: I am thankful for the DebConf team sponsoring both accomodation and food. It is highly appreciated. See you in NYC! Any hints that you may happen to have are warmly welcome.

8 July 2010

Aigars Mahinovs: Phoning it in

Currently I own and use an iPhone 3G. I bought it almost two years ago, when the local phone provider LMT started offering the iPhone legally. I had a pretty good experience with it most of the time, but now it is showing its age: I will try to send my iPhone in for repair hoping to extend its usefulness, but frankly it looks that I might have to get a new phone at or before Debconf. So I am considering my options for a new phone and so far I have 3 main options each with some sub-options. The main options are: iPhone 4, MeeGo and Android. Each choice has its benefits and drawbacks and each also has several sub-options related either to specific phone models or to purchase methods. What I need is a phone with: calling function, contact information synchronised to Google, support for Google Mail/Contacts/Calendar simultaneously with one Exchange account also providing mail/contacts/calendars, music, audiobooks saving my place in the audio book and in the list of books, fully featured Twitter client with Twitpic and location support, encrypted password storage application (preferably open source), GPS with maps, ability to download maps for offline use, Geocaching support with offline cache info and logging, Skype client, ability to write in Latvian, English and Russian in all applications, reliable backup and restore of phones data including settings and data of installed apps. Optional features: last.fm streaming and reporting support, ebook applications with purchasable ebooks, high quality video and photo recording (for a phone), ability for me to write applications on the phone, ssh client, wireless data upload to the phone, ability to change the phones battery without loosing data in the phone (on the go), compass, physical keyboard, confidence in getting software upgrades and developer interest for the next 2 years. I ll try to describe all of the options here to organise my thoughts and maybe also help someone else make the choice. Option 1: iPhone 4 To get the iPhone 4 I have two options: LMT and unlocked. The official arrival of iPhone 4 to Latvia is expected in September and it will likely be sold by LMT for the same prices as iPhone 3GS is now. If I choose this scenario, I will need to survive with my semi-broken iPhone for 2-3 more months. Total price over 2 years (LVL): 131 (purchase price) + 24 * 8 (surcharge) = 323 LVL. Note: the surcharge is the extra cost of the iPhone call plan compared to a comparable regular plan (Vien dais 7 + Internets telefon 5). If I go and purchase an unlocked iPhone 4 in a country where such units are sold (UK is the most easily accessible choice, because a few of my workmates travel weekly to and from UK) then the total price over 2 years would be the purchase price: 499 GBP + 65 GBP (2 year warranty extension) = 505 LVL Now lets go over the general benefits and downsides of the iPhone 4, first the positives: Drawbacks: Option 2: Maemo/MeeGo Some of my colleagues at work (in the Riga office of Accenture) have N900 phones and I have been exposed to people with raw enthusiasm towards the Maemo platform ever since the Debconf in Helsinki, where we saw the first N700 devices in the hands of some lucky Debian/Nokia people using it as an Internet tablet. N900 has been a strong leap forward for the platform, before its head was teleported sideways by the whole MeeGo merger/debackle. Again I have two options here: Get a N900 either here in Latvia for 325 LVL (that would be the total cost over 2 years) or get it from USA during Debconf for 399 USD (+129 USD for 2 year warranty) = 317 LVL. And additional option would be to wait until October/November and then buy the new Nokia MeeGo device, rumoured to be N9-00 and likely to be around the same price as N900 was when it was introduced (around 500 LVL). Benefits: Drawbacks: Option 3 Android After looking trough the options for Android phones, I ve narrowed the selection to the ones that are either available from local carriers or can be easily gotten unlocked also the phones need to have announced plans to have at least Android 2.2 version. Currently the choice is limited to HTC Desire (from carrier or unlocked) and Samsung Galaxy S (unlocked). The prices over 2 years break down as follows: HTC Desire (locked to LMT): 69 + 9*24 = 285 LVL
HTC Desire (locked to Bite): 199 + 2*24 = 247 LVL (Note: only 500Mb of data per month available)
HTC Desire (locked to Tele2): 179 + 6*24 = 323 LVL
Note: all the above options would also require me to pay a 45 LVL early termination fee on my current iPhone contract if I choose to do this before September.
HTC Desire (unlocked): 311 LVL
Samsung Galaxy S (unlocked): 350 LVL (import from Germany) Benefits: Drawbacks: Please correct me if I am wrong with something!!! And also it would be cool if you expressed your own opinions in the comments. Currently I am very undecided about what I am going to do, but after completing this entry I am leaning towards an unlocked HTC Desire. Update:
Olivier Cr te shares a N900 vs. Nexus One experience Edit 01-07-2010: Handed my iPhone in for a warranty repair, got a dumbphone Nokia as a loaner during repairs. Very surprised about how long a battery can last on a low powered and cheap phone. In the mean time from all the comments here and elsewhere I am starting to see that I am too annoyed with the iPhone platform (mostly iTunes) to stay there and that I am also not convinced in the direction MeeGo is going (a lot of community developers are annoyed and are jumping ship), so N900 is also out. I will need to see 2-3 MeeGo smartphones and also see how Nokia will treat their smartphone users and developers on this platform, in the long term, before I ll be ready to trust them with my money. Therefore, my choice is becoming pretty clear look for the best Android phone (after my iPhone dies). We have this thing at Accenture, where we can get company phones at a discount and HTC Desire is on that list. If I can get that, it would slash 100 LVL (almost a third of the price!) off it, but to get there I either need to get a promotion (rather rare after just one year with the company) or prove the necessity of the phone for the needs of my project (kinda hard currently). Well, the situation will be a bit clearer next week when I get my iPhone back. (Apparently the Latvia s largest holiday Midsummer Festivities or J i/L go with tons of traditional outdoor activities, caused a lot of phones to become broken and there is a backlog of warranty service :) ) So at this point my plan is: get my iPhone repaired and get a promotion or a project where I can justify a company-paid HTC Desire and then hope that Nokia/Intel really get their stuff together and make MeeGo a great smartphone platform over the next year or two. Update: Got my iPhone 3G back from the warranty . It is in quotes because they just gave me a new iPhone 3G. The battery lasts for 3 days of minimal use and it has not crashed yet. The recovery process was a pain again caused by iTunes: iTunes refused to restore my backup onto the new phone, because my backup was made on a iOS 4.0, while the new phone I got from the warranty has 3.1.3 on it. So I had to set this phone as a new phone, initialise it, make a backup, upgrade to 4.0, restore from the (empty) backup, then reset the phone to factory settings again and only then I could start restoring data from my original (pre-warranty) backup and start copying my data on to the device. All in all it took nearly 3 hours if iTunes doing something and almost a dozen reboots of the phone. Almost half of that time was spent doing completely unnecessary steps to work around the fact that iTunes is braindead. I ll continue using my iPhone for now, keeping my eyes on the newest Android phones and also waiting for the long-promised first MeeGo phone from Nokia.

2 July 2010

Daniel Stone: xhost plus considered harmful

So, flipping through the otherwise good MeeGo SDK setup instructions, I was pretty disheartened to see this:
Next, configure X on the host machine to enable the Simulator (running from the chroot) to access the display of the normal user:

xhost +local:

This gives everyone who can execute anything on your machine full permission to rip your credit card numbers out of Chromium and your password out of gnome-terminal. Sweet. This comes up all the time, so man wouldn't it be great if there was some kind of better way?

xhost +SI:localuser:usernametogiveaccesstogoeshere

I'm pretty sure this has been available in every single X.Org release since 6.8 or so, so next time you see someone advocating something as daft as the above, please point them towards the correct way. Thanks.

(Pushed it with the wrong date originally; couldn't fix it without bumping it to the top again. Sorry!)

30 June 2010

Vincent Sanders: Programmers are suckers for a meme

Many Open Source projects have and IRC # for developers. The NetSurf project is no different. During a discussion someone jokingly suggested that one contributor should be asked to take the FizzBuzz test. Can you guess what happened next?

Ten minutes later Michael Drake posted this solid example in C which is where it all ought to have ended.

<iframe src="http://pastebin.com/embed_iframe.php?i=MesmvSjk" style="border:none;width:100%;height:200px;"></iframe>

Being programmers, of course this had a predictable result. The original question, the reason for asking it and any the serious point being made in the original article were discarded. Just so everyone (including myself) could play silly buggers over our lunch break. Coders, it would seem, simply like to produce a solution even if it is only for fun.

None of these programs took more than ten minutes (except the JAVA monstrosity), are reproduced with permission and I am to blame for none of them (ok maybe just the one ;-).

First up was Rob Kendrick with the classic solution in C (his day job is as a support team lead which seems to make those programmers who cannot do this seem even more scary bad.)

<iframe src="http://pastebin.com/embed_iframe.php?i=EaQ2414U" style="border:none;width:100%;height:200px;"></iframe>

Next was Daniel Silverstone who turned this Lua solution out very quickly and berated the rest of us for not following the rules ;-)

<iframe src="http://pastebin.com/embed_iframe.php?i=cYY0YDYM" style="border:none;width:100%;height:220px;"></iframe>

The final C solution was my own uber silly sieve implementation

<iframe src="http://pastebin.com/embed_iframe.php?i=5w5BhKt3" style="border:none;width:100%;height:200px;"></iframe>

Peter Howkins decided the world required a solution in PHP

<iframe src="http://pastebin.com/embed_iframe.php?i=h2aQ5eXM" style="border:none;width:100%;height:200px;"></iframe>

When pointed out that his solution stopped at 50 he presented this vastly superior and obviously idiomatic solution

<iframe src="http://pastebin.com/embed_iframe.php?i=k8T6VtJu" style="border:none;width:100%;height:200px;"></iframe>

Finally after a long time James Shaw caused mental anguish and wailing with this abomination unto Nuggin.

<iframe src="http://pastebin.com/embed_iframe.php?i=GFeHdJdr" style="border:none;width:100%;height:200px;"></iframe>

With Luck everyone has now got it out of their system and we will never have to put up with this again (yeah right). And now you also know why Open Source projects sometimes take ages to release ;-)

29 June 2010

Daniel Stone: xhost plus considered harmful

So, flipping through the otherwise good MeeGo SDK setup instructions, I was pretty disheartened to see this:
Next, configure X on the host machine to enable the Simulator (running from the chroot) to access the display of the normal user:

xhost +local:

This gives everyone who can execute anything on your machine full permission to rip your credit card numbers out of Chromium and your password out of gnome-terminal. Sweet. This comes up all the time, so man wouldn't it be great if there was some kind of better way?

xhost +SI:localuser:usernametogiveaccesstogoeshere

I'm pretty sure this has been available in every single X.Org release since 6.8 or so, so next time you see someone advocating something as daft as the above, please point them towards the correct way. Thanks.

7 June 2010

Daniel Stone: and one more thing ...

So, I spent an hour or two this afternoon following the iPhone 4 liveblog. It all looked fairly compelling (the screen!), right up until Steve's 'and one more thing': video calling.



HELLO I'M IN 2007, CAN YOU HEAR ME
(Photo of the Nokia N800, which shipped in January 2007, from rnair.)

The moral of the story? If you want to be four years ahead of the WWDC closing bombshell, email sales@collabora.co.uk. :)


PS: The 2010 'fuck it, we're going to fivefour blades' version; we also had a six-way video call going earlier today.

27 May 2010

Vincent Sanders: Ex Phone

I went to the Linux Plumbers Conference (LPC) last year (which was very interesting and productive). While I was there Qualcomm were in attendance giving out penguin mints and running a competition. The lucky winners of this draw were to receive an ADP1 Android phone. So I dutifully filled in the entry form, handed it in and thought no more about it.

In the break after attending a particularly interesting workshop on the last day of the conference several other people congratulated me on winning the Qualcomm draw, this was the first I knew about it! I went in search of the nice people from Qualcomm and sure enough they handed over an ADP1 after getting a couple of photos.

Once I returned home I switched to using ADP1 as my phone and started experimenting with Android kernel stuff. Then one day the Wi-Fi stopped working which while odd could be overcome by repeatedly unloading and reloading the driver until it worked once more. Then one day the USB stopped working, no more ADB, no more console, no more debugging, no more hacking.

Then one day it turned itself off and never came back. I have been forced to return to using a hand me down, very kindly given to me by Daniel Silverstone. Alas it is not a smart phone of any kind and my finances do not allow for me to spend the money to replace it.

I have repeatedly tried to contact Qualcomm open source to see if there is any kind of warranty I might be able to use to get the phone repaired alas all the contact addresses I have are now simply returning SMTP errors.

So this is pretty much a tale without a happy ending, unless anyone out there knows the right people to contact? Perhaps someone at Google maybe? I only revisit the subject at all because of the recent announcement of an Android 2.1 based edition for these devices which reminded me I like to play with these things.

Next.

Previous.