Here's a quick howto for local map rendering, using python, mapnik and data from OpenStreetMap. This tutorial shows how to create a stylesheet for mapnik from scratch, if you're looking for how to tweak OpenStreetMap's mapnik rendering, I plan to write a tutorial on PostGIS-driven rendering at a later time. This post is not for those who want to make a local slippy map, nor those who want to render different areas of big dumps (think of big national dumps). We'll render some features of a residential area of a city with buildings mapped. We'll get a small dump of the area for this, using the XAPI. First, you need to install mapnik. On Debian-like systems, it should be as easy as:

# apt-get install python-mapnik

Then, you need some python code to use the mapnik library. Let's call it render.py:

#!/usr/bin/python
import mapnik
###
# Configuration
###
style = 'style.xml'
output = 'output.svg'
width = 1280
height = 800
bbox = '12.58664,37.65759,12.5945,37.66325'
###
# Don't touch below!
###
bbox = bbox.split(',')
ll = mapnik.Coord(float(bbox[0]), float(bbox[1]))
tr = mapnik.Coord(float(bbox[2]), float(bbox[3]))
mymap = mapnik.Map(width, height)
mapnik.load_map(mymap, style)
map_bbox = mapnik.Envelope(ll, tr)
mymap.zoom_to_box(map_bbox)
mapnik.render_to_file(mymap, output)

For the purposes of this post, we're going to create a 1280x800 map -- adjust that to suit your needs -- and we specified a bounding box corresponding to that area. That's composed as west,south,east,north coordinates. Be sure to get them right :-). Now we need to write a Mapnik stylesheet, we'll call it style.xml. That's a XML document, you can find the documentation at the official website. The skeleton of a stylesheet is as follows:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE Map>
<Map bgcolor="lightblue" srs="+proj=latlong +datum=WGS84">
<Style name="...">
...
</Style>
<Layer name="...">
 <Datasource>
  ...
 </Datasource>
</Layer>
</Map>

The stylesheet has an arbitrary number of Style elements and an arbitrary number of Layer elements. The former class of elements define the appearance of the features, while the latter defines where to get data from. Each layer has a Datasource child: here's where you say where to get your data from. We're going to use the osm input plugin, which lets you parse the dump directly. This has some drawbacks:

• not scalable for big dumps
• for each bbox you want to render, the full dump has to be re-parsed

But works great for small dumps. In this tutorial, we want to pretty render a residential area, so we first need to render the road network. This will mainly be made of highway=residential roads. Each style can have one or more Rule element. These define what elements that particular style should apply to. Each rule can have different rendering settings. The given dump has different roads: residential, unclassified, tertiary and secondary. Let's add a rule for each. Here's the first of them:

<Rule>
 <Filter>[highway] = 'residential'</Filter>
 <LineSymbolizer>
  <CssParameter name="stroke">#ffffff</CssParameter>
  <CssParameter name="stroke-width">5</CssParameter>
  <CssParameter name="stroke-linejoin">round</CssParameter>
  <CssParameter name="stroke-linecap">round</CssParameter>
 </LineSymbolizer>
 <TextSymbolizer name="name" face_name="DejaVu Sans Book" size="8" fill="#000" halo_radius="1" spacing="300" placement="line" />
</Rule>

It is quite straightforward: you define what elements the rule should apply to, and then add some symbolizer. For a detailed description of each symbolizer, please read the documentation. The example dump also has a railway and some steps. To render them, we use the stroke-dasharray property:

<Rule>
 <Filter>[highway] = 'steps'</Filter>
 <LineSymbolizer>
  <CssParameter name="stroke">#ff0000</CssParameter>
  <CssParameter name="stroke-width">5.0</CssParameter>
  <CssParameter name="stroke-dasharray">2,1</CssParameter>
 </LineSymbolizer>
</Rule>
<Rule>
 <Filter>[railway] = 'rail'</Filter>
 <LineSymbolizer>
  <CssParameter name="stroke">#222222</CssParameter>
  <CssParameter name="stroke-width">3</CssParameter>
  <CssParameter name="stroke-linejoin">round</CssParameter>
 </LineSymbolizer>
 <LineSymbolizer>
  <CssParameter name="stroke">white</CssParameter>
 <CssParameter name="stroke-width">1</CssParameter>
  <CssParameter name="stroke-linejoin">round</CssParameter>
  <CssParameter name="stroke-dasharray">10,20</CssParameter>
 </LineSymbolizer>
</Rule>

Also here, it is quite simple. The values given to stroke-dasharray determine the size of the dashes, and they can be alternated. So, for the steps we have 2px of #ff0000 and 1px of trasparency, while for the rail we have 10px of white and 20px of #222222. What if we want to add some markers to the map? Easy, just use another symbolizer, called PointSymbolizer:

<Rule>
 <Filter>[highway] = 'stop'</Filter>
 <PointSymbolizer width='10' height='22' file='stop.png' type='png' allow_overlap='true' opacity='1.0' />
</Rule>

Here we mark stop signals with a pre-rendered stop.png (only png and tiff are currently supported, and svg support is only in the development version). What to do next? Well, we can make this map really nice. We'll render buildings with a pseudo-3D effect, which, however, does not reflect real heights -- but it's rather nice to see. How to achieve this? BuildingSymbolizer is the answer.

<Rule>
 <Filter>[building] &lt;&gt; ''</Filter>
 <BuildingSymbolizer>
  <CssParameter name="fill">#00ff00</CssParameter>
  <CssParameter name="fill-opacity">0.4</CssParameter>
  <CssParameter name="height">0.00007</CssParameter>
 </BuildingSymbolizer>
</Rule>

Here we just draw building=*. Regarding the height, you need to play a bit -- it took a while before I found a good value of it. One thing missing in this stylesheet, is the rendering of coastlines. Since we don't have a coastline here, I'm skipping it, but the easy way is: use another rule. This is really just a trick; OpenStreetMap renders the coastline in a different way respect to other features, that's why I'm using this trick. I'm not posting the whole stylesheet, but you can download it. You can also get the script, the OSM dump and the stop.png image. Now, you should be able to just do:

$ ./render.py

and you will have your map in output.svg Here's what you would get if you use the files provided above (click to see the original SVG version):

I m currently working on something which will eventually amount to a re-bootstrapping of sorts of the Debian/m68k (Linux) port patches to the Linux kernel, gcc, etc. are prepared (some have been accepted into upstream or the packages already). I will probably have more, once the compile processes finish, anyway (even emulated, it s slow). I think that, once I get past that TLS (thread-local storage, needed by eglibc) migration, I will try to find out a list of packages needed for debootstrap (AFAICT: all packages marked Essential, or of priority important, and all marked Build-Essential (for the *-builder variant), and their dependencies (although I ll substitute sysv-rc with file-rc, which is better and needs less deps)) and pull arch:all from sid, then build the rest myself using a consistent snapshot of sid possibly with patches going to unreleased. Then I can use cowbuilder to make cleaner packages, which can eventually be uploaded (once I get enough to get a buildd running kernel, bootloader, etc) binNMUs are way to go here I suppose. I will only upload once it s self-hosted, installable (seen by edos-debcheck), clean, etc. (i.e. I ll rebuild all binaries) and probably keep a bootstrap repo around (until m68k caught up) so that unstable (possibly amended by unreleased for a while) will not again become uninstallable, e.g. if arch:all packages change their dependencies (Python, gcc-defaults are some I ve seen). That bootstrap repository is needed anyway because debootstrap can t install from two separate repositories (unstable+unreleased for example). Progress is slow because I try to keep as close to official packages as possible, refuse to cross-compile, and try to produce uploadable if possible packages all the time. Getting patches into packages, so that I can build from unstable, instead of debian-ports.org unreleased, has proven time-consuming and occasionally frustrating as well. Although I would like to thank the people who helped me on the way already. (I am not naming any in fear of forgetting some, but you know who you are ) They are among the Debian (gcc, kernel, m68k) and Linux-68k crowd. (Why does genattrtab in gcc-4.4 take 3 hours when it took less than half an hour in gcc-4.3 anyway?) I m also still working on mksh and some Python ISO hacks for mika and some minor stuff, and further cleaning up MirBSD.

Python-popcon is a small Python library which allows to query Debian s popcon database. The usage is very trivial:
>>> import popcon >>> popcon.package('python-popcon')
'python-popcon': 2 >>> popcon.package('icedove', 'iceweasel')
'icedove': 12140, 'iceweasel': 45666 >>> popcon.package('foobarbaz')

You can call the package method with an arbitrary number of package names and it will return a dictionary with package name : popcon value mappings for all packages found. If a package is not found in the popcon database it will not be in the resulting dictionary or if it was the only package in the method call, the method will return an empty dictionary. The popcon value is the number of installations according to Debian s Popularity Contest service. There is also a second method which gives some more information
>>> popcon.package_raw('icedove', 'iceweasel')
'icedove': [8065, 2195, 1879, 1], iceweasel : [27857, 10681, 7120, 8]
It returns a list of [vote, old, recent, no-files] for the given package. How does it work? Upon a query, popcon downloads http://popcon.debian.org/all-popcon-results.txt.gz, extracts and parses the file and returns the desired information. Since downloading and extraction this file is expensive, it saves the file and tries to re-use it is not older than 7 days. So while the first call of one of the above methods can take a few seconds, the result will appear almost instantaneously for the next week until the cache file expired. That s it, no rocket science but a convenient way to get popcon information from within python. The package python-popcon is available in Debian/Sid for a few days.

Before I upgraded from Lenny to Squeeze, all my CGI scripts were working properly. The CGI scripts has been executed by Apache and resulted in rendered webpage. After the upgrade to Squeeze all those CGI scripts stopped been executed but instead started to get displayed as a plain text. Common to all those CGI scripts is that they have .phtml as suffix, but the bang path/shebang consists of "#!/usr/bin/python". As an example you can have a look at Buildd.Net and its scripts like this one. The section in Apaches config looks like this:

<Directory /home/builddnet/unstable/WWW/cgi/>
Options -Indexes +ExecCGI +FollowSymLinks
AddHandler cgi-script .cgi .sh .pl .py .phtml
Order allow,deny
Allow from all
</Directory>

So, I would expect that the CGI script handler will get executed when loading a *.phtml file and that the shebang would be honoured. Funny enough: when I rename that script to *.cgi it works. I haven't figured out yet, what causes this beahviour or what have changed during the upgrade - and how to revert to the old behaviour. So, dear lazyweb, can you give me some hints and pointers?

We live in a nat-ed / firewall-ed world. Almost all DSL providers don't give public IPs and when the do, they are often behind a draconian firewall. In this context having an emergency remote shell, despite not fast and not public is very handy. A simple way to solve this problem it to create a hidden server on the tor network and the access the shell from anywhere in the world without caring of change of IPs, routing, dns or anything else. On debian you can just install tor from the official repository. Since Tor is not available in ubuntu (but it is available on debian), we need to get it directly from the tor website. There is a nice write-up on the ubuntu site : https://help.ubuntu.com/community/Tor . And these are the details on the tor website. So we add this to apt.sources and the we aptitude install tor .The package will install and run by default the tor daemon. Next step is to edit /etc/tor/torrc to add the proxy server. Remember also to install the openssh-server server if you don't have it already. And this is it. In the directory /var/lib/tor/ssh/ you will find a file with the hostname on the TOR ring that you have to use to connect to you new hidden server. On the client side we need to aptitude install connect-proxy. It's a simple tool to tunnel ssh through a socks5 connection. Now you are ready to test. In your ~/.ssh/config you can simply add something like and then ssh-way youronionhost.onion server . The connection will be veeeeery slow since you are going through different layers of encryption and indirection. You should also check the hostid of your server before connecting and dropping in a pub-key as you should never trust your friendly TOR providers (US govt, Chinese gvt, Iranian govt, etc ...). For emergency is actually pretty handy. For anything else if will make you die of boredom ...

Update Ahhh . It seems I've assumed that since TOR was not available on ubuntu, this was the same on debian. Tor is definitely available on debian, but not on ubuntu. Blah... check before asserting wrong information ! Post fixed.

Award winning code Me and Yuwei had a fun day at hhhmcr (#hhhmcr) and even managed to put together a prototype that won the first prize \o/ We played with the gmp24 dataset kindly extracted from Twitter by Michael Brunton-Spall of the Guardian into a convenient JSON dataset. The idea was to find ways of making it easier to look at the data and making sense of it. This is the story of what we did, including the code we wrote. The original dataset has several JSON files, so the first task was to put them all together:

#!/usr/bin/python
# Merge the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import os
res = []
for f in os.listdir("."):
    if not f.startswith("gmp24"): continue
    data = open(f).read().strip()
    if data == "[]": continue
    parsed = simplejson.loads(data)
    res.extend(parsed)
print simplejson.dumps(res)

The results however were not ordered by date, as GMP had to use several accounts to twit because Twitter was putting Greather Manchester Police into jail for generating too much traffic. There would be quite a bit to write about that, but let's stick to our work. Here is code to sort the JSON data by time:

#!/usr/bin/python
# Sort the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import sys
import datetime as dt
all_recs = simplejson.load(sys.stdin)
all_recs.sort(key=lambda x: dt.datetime.strptime(x["created_at"], "%a %b %d %H:%M:%S +0000 %Y"))
simplejson.dump(all_recs, sys.stdout)

I then wanted to play with Tf-idf for extracting the most important words of every tweet:

#!/usr/bin/python
# tfifd - Annotate JSON elements with Tf-idf extracted keywords
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import sys, math
import simplejson
import re
# Read all the twits
records = simplejson.load(sys.stdin)
# All the twits by ID
byid = dict(((x["id"], x) for x in records))
# Stopwords we ignore
stopwords = set(["by", "it", "and", "of", "in", "a", "to"])
# Tokenising engine
re_num = re.compile(r"^\d+$")
re_word = re.compile(r"(\w+)")
def tokenise(tweet):
    "Extract tokens from a tweet"
    for tok in tweet["text"].split():
        tok = tok.strip().lower()
        if re_num.match(tok): continue
        mo = re_word.match(tok)
        if not mo: continue
        if mo.group(1) in stopwords: continue
        yield mo.group(1)
# Extract tokens from tweets
tokenised = dict(((x["id"], list(tokenise(x))) for x in records))
# Aggregate token counts
aggregated =  
for d in byid.iterkeys():
    for t in tokenised[d]:
        if t in aggregated:
            aggregated[t] += 1
        else:
            aggregated[t] = 1
def tfidf(doc, tok):
    "Compute TFIDF score of a token in a document"
    return doc.count(tok) * math.log(float(len(byid)) / aggregated[tok])
# Annotate tweets with keywords
res = []
for name, tweet in byid.iteritems():
    doc = tokenised[name]
    keywords = sorted(set(doc), key=lambda tok: tfidf(doc, tok), reverse=True)[:5]
    tweet["keywords"] = keywords
    res.append(tweet)
simplejson.dump(res, sys.stdout)

I thought this was producing a nice summary of every tweet but nobody was particularly interested, so we moved on to adding categories to tweet. Thanks to Yuwei who put together some useful keyword sets, we managed to annotate each tweet with a place name (i.e. "Stockport"), a social place name (i.e. "pub", "bank") and a social category (i.e. "man", "woman", "landlord"...) The code is simple; the biggest work in it was the dictionary of keywords:

#!/usr/bin/python
# categorise - Annotate JSON elements with categories
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
# Copyright (C) 2010  Yuwei Lin <yuwei@ylin.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import sys, math
import simplejson
import re
# Electoral wards from http://en.wikipedia.org/wiki/List_of_electoral_wards_in_Greater_Manchester
placenames = ["Altrincham", "Sale West",
"Altrincham", "Ashton upon Mersey", "Bowdon", "Broadheath", "Hale Barns", "Hale Central", "St Mary", "Timperley", "Village",
"Ashton-under-Lyne",
"Ashton Hurst", "Ashton St Michael", "Ashton Waterloo", "Droylsden East", "Droylsden West", "Failsworth East", "Failsworth West", "St Peter",
"Blackley", "Broughton",
"Broughton", "Charlestown", "Cheetham", "Crumpsall", "Harpurhey", "Higher Blackley", "Kersal",
"Bolton North East",
"Astley Bridge", "Bradshaw", "Breightmet", "Bromley Cross", "Crompton", "Halliwell", "Tonge with the Haulgh",
"Bolton South East",
"Farnworth", "Great Lever", "Harper Green", "Hulton", "Kearsley", "Little Lever", "Darcy Lever", "Rumworth",
"Bolton West",
"Atherton", "Heaton", "Lostock", "Horwich", "Blackrod", "Horwich North East", "Smithills", "Westhoughton North", "Chew Moor", "Westhoughton South",
"Bury North",
"Church", "East", "Elton", "Moorside", "North Manor", "Ramsbottom", "Redvales", "Tottington",
"Bury South",
"Besses", "Holyrood", "Pilkington Park", "Radcliffe East", "Radcliffe North", "Radcliffe West", "St Mary", "Sedgley", "Unsworth",
"Cheadle",
"Bramhall North", "Bramhall South", "Cheadle", "Gatley", "Cheadle Hulme North", "Cheadle Hulme South", "Heald Green", "Stepping Hill",
"Denton", "Reddish",
"Audenshaw", "Denton North East", "Denton South", "Denton West", "Dukinfield", "Reddish North", "Reddish South",
"Hazel Grove",
"Bredbury", "Woodley", "Bredbury Green", "Romiley", "Hazel Grove", "Marple North", "Marple South", "Offerton",
"Heywood", "Middleton",
"Bamford", "Castleton", "East Middleton", "Hopwood Hall", "Norden", "North Heywood", "North Middleton", "South Middleton", "West Heywood", "West Middleton",
"Leigh",
"Astley Mosley Common", "Atherleigh", "Golborne", "Lowton West", "Leigh East", "Leigh South", "Leigh West", "Lowton East", "Tyldesley",
"Makerfield",
"Abram", "Ashton", "Bryn", "Hindley", "Hindley Green", "Orrell", "Winstanley", "Worsley Mesnes",
"Manchester Central",
"Ancoats", "Clayton", "Ardwick", "Bradford", "City Centre", "Hulme", "Miles Platting", "Newton Heath", "Moss Side", "Moston",
"Manchester", "Gorton",
"Fallowfield", "Gorton North", "Gorton South", "Levenshulme", "Longsight", "Rusholme", "Whalley Range",
"Manchester", "Withington",
"Burnage", "Chorlton", "Chorlton Park", "Didsbury East", "Didsbury West", "Old Moat", "Withington",
"Oldham East", "Saddleworth",
"Alexandra", "Crompton", "Saddleworth North", "Saddleworth South", "Saddleworth West", "Lees", "St James", "St Mary", "Shaw", "Waterhead",
"Oldham West", "Royton",
"Chadderton Central", "Chadderton North", "Chadderton South", "Coldhurst", "Hollinwood", "Medlock Vale", "Royton North", "Royton South", "Werneth",
"Rochdale",
"Balderstone", "Kirkholt", "Central Rochdale", "Healey", "Kingsway", "Littleborough Lakeside", "Milkstone", "Deeplish", "Milnrow", "Newhey", "Smallbridge", "Firgrove", "Spotland", "Falinge", "Wardle", "West Littleborough",
"Salford", "Eccles",
"Claremont", "Eccles", "Irwell Riverside", "Langworthy", "Ordsall", "Pendlebury", "Swinton North", "Swinton South", "Weaste", "Seedley",
"Stalybridge", "Hyde",
"Dukinfield Stalybridge", "Hyde Godley", "Hyde Newton", "Hyde Werneth", "Longdendale", "Mossley", "Stalybridge North", "Stalybridge South",
"Stockport",
"Brinnington", "Central", "Davenport", "Cale Green", "Edgeley", "Cheadle Heath", "Heatons North", "Heatons South", "Manor",
"Stretford", "Urmston",
"Bucklow-St Martins", "Clifford", "Davyhulme East", "Davyhulme West", "Flixton", "Gorse Hill", "Longford", "Stretford", "Urmston",
"Wigan",
"Aspull New Springs Whelley", "Douglas", "Ince", "Pemberton", "Shevington with Lower Ground", "Standish with Langtree", "Wigan Central", "Wigan West",
"Worsley", "Eccles South",
"Barton", "Boothstown", "Ellenbrook", "Cadishead", "Irlam", "Little Hulton", "Walkden North", "Walkden South", "Winton", "Worsley",
"Wythenshawe", "Sale East",
"Baguley", "Brooklands", "Northenden", "Priory", "Sale Moor", "Sharston", "Woodhouse Park"]
# Manual coding from Yuwei
placenames.extend(["City centre", "Tameside", "Oldham", "Bury", "Bolton",
"Trafford", "Pendleton", "New Moston", "Denton", "Eccles", "Leigh", "Benchill",
"Prestwich", "Sale", "Kearsley", ])
placenames.extend(["Trafford", "Bolton", "Stockport", "Levenshulme", "Gorton",
"Tameside", "Blackley", "City centre", "Airport", "South Manchester",
"Rochdale", "Chorlton", "Uppermill", "Castleton", "Stalybridge", "Ashton",
"Chadderton", "Bury", "Ancoats", "Whalley Range", "West Yorkshire",
"Fallowfield", "New Moston", "Denton", "Stretford", "Eccles", "Pendleton",
"Leigh", "Altrincham", "Sale", "Prestwich", "Kearsley", "Hulme", "Withington",
"Moss Side", "Milnrow", "outskirt of Manchester City Centre", "Newton Heath",
"Wythenshawe", "Mancunian Way", "M60", "A6", "Droylesden", "M56", "Timperley",
"Higher Ince", "Clayton", "Higher Blackley", "Lowton", "Droylsden",
"Partington", "Cheetham Hill", "Benchill", "Longsight", "Didsbury",
"Westhoughton"])
# Social categories from Yuwei
soccat = ["man", "woman", "men", "women", "youth", "teenager", "elderly",
"patient", "taxi driver", "neighbour", "male", "tenant", "landlord", "child",
"children", "immigrant", "female", "workmen", "boy", "girl", "foster parents",
"next of kin"]
for i in range(100):
    soccat.append("%d-year-old" % i)
    soccat.append("%d-years-old" % i)
# Types of social locations from Yuwei
socloc = ["car park", "park", "pub", "club", "shop", "premises", "bus stop",
"property", "credit card", "supermarket", "garden", "phone box", "theatre",
"toilet", "building site", "Crown court", "hard shoulder", "telephone kiosk",
"hotel", "restaurant", "cafe", "petrol station", "bank", "school",
"university"]
extras =   "placename": placenames, "soccat": soccat, "socloc": socloc  
# Normalise keyword lists
for k, v in extras.iteritems():
    # Remove duplicates
    v = list(set(v))
    # Sort by length
    v.sort(key=lambda x:len(x), reverse=True)
# Add keywords
def add_categories(tweet):
    text = tweet["text"].lower()
    for field, categories in extras.iteritems():
        for cat in categories:
            if cat.lower() in text:
                tweet[field] = cat
                break
    return tweet
# Read all the twits
records = (add_categories(x) for x in simplejson.load(sys.stdin))
simplejson.dump(list(records), sys.stdout)

All these scripts form a nice processing chain: each script takes a list of JSON records, adds some bit and passes it on. In order to see what we have so far, here is a simple script to convert the JSON twits to CSV so they can be viewed in a spreadsheet:

#!/usr/bin/python
# Convert the JSON twits to CSV
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import sys
import csv
rows = ["id", "created_at", "text", "keywords", "placename"]
writer = csv.writer(sys.stdout)
for rec in simplejson.load(sys.stdin):
    rec["keywords"] = " ".join(rec["keywords"])
    rec["placename"] = rec.get("placename", "")
    writer.writerow([rec[row] for row in rows])

At this point we were coming up with lots of questions: "were there more reports on women or men?", "which place had most incidents?", "what were the incidents involving animals?"... Time to bring Xapian into play. This script reads all the JSON tweets and builds a Xapian index with them:

#!/usr/bin/python
# toxapian - Index JSON tweets in Xapian
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import simplejson
import sys
import os, os.path
import xapian
DBNAME = sys.argv[1]
db = xapian.WritableDatabase(DBNAME, xapian.DB_CREATE_OR_OPEN)
stemmer = xapian.Stem("english")
indexer = xapian.TermGenerator()
indexer.set_stemmer(stemmer)
indexer.set_database(db)
data = simplejson.load(sys.stdin)
for rec in data:
    doc = xapian.Document()
    doc.set_data(str(rec["id"]))
    indexer.set_document(doc)
    indexer.index_text_without_positions(rec["text"])
    # Index categories as categories
    if "placename" in rec:
        doc.add_boolean_term("XP" + rec["placename"].lower())
    if "soccat" in rec:
        doc.add_boolean_term("XS" + rec["soccat"].lower())
    if "socloc" in rec:
        doc.add_boolean_term("XL" + rec["socloc"].lower())
    db.add_document(doc)
db.flush()
# Also save the whole dataset so we know where to find it later if we want to
# show the details of an entry
simplejson.dump(data, open(os.path.join(DBNAME, "all.json"), "w"))

And this is a simple command line tool to query to the database:

#!/usr/bin/python
# xgrep - Command line tool to query the GMP24 tweet Xapian database
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import simplejson
import sys
import os, os.path
import xapian
DBNAME = sys.argv[1]
db = xapian.Database(DBNAME)
stem = xapian.Stem("english")
qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
qp.add_boolean_prefix("place", "XP")
qp.add_boolean_prefix("soc", "XS")
qp.add_boolean_prefix("loc", "XL")
query = qp.parse_query(sys.argv[2],
    xapian.QueryParser.FLAG_BOOLEAN  
    xapian.QueryParser.FLAG_LOVEHATE  
    xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE  
    xapian.QueryParser.FLAG_WILDCARD  
    xapian.QueryParser.FLAG_PURE_NOT  
    xapian.QueryParser.FLAG_SPELLING_CORRECTION  
    xapian.QueryParser.FLAG_AUTO_SYNONYMS)
enquire = xapian.Enquire(db)
enquire.set_query(query)
count = 40
matches = enquire.get_mset(0, count)
estimated = matches.get_matches_estimated()
print "%d/%d results" % (matches.size(), estimated)
data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))
for m in matches:
    rec = data[m.document.get_data()]
    print rec["text"]
print "%d/%d results" % (matches.size(), matches.get_matches_estimated())
total = db.get_doccount()
estimated = matches.get_matches_estimated()
print "%d results over %d documents, %d%%" % (estimated, total, estimated * 100 / total)

Neat! Now that we have a proper index that supports all sort of cool things, like stemming, tag clouds, full text search with complex queries, lookup of similar documents, suggest keywords and so on, it was just fair to put together a web service to share it with other people at the event. It helped that I had already written similar code for apt-xapian-index and dde before. Here is the server, quickly built on bottle. The very last line starts the server and it is where you can configure the listening interface and port.

#!/usr/bin/python
# xserve - Make the GMP24 tweet Xapian database available on the web
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import bottle
from bottle import route, post
from cStringIO import StringIO
import cPickle as pickle
import simplejson
import sys
import os, os.path
import xapian
import urllib
import math
bottle.debug(True)
DBNAME = sys.argv[1]
QUERYLOG = os.path.join(DBNAME, "queries.txt")
data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))
prefixes =   "place": "XP", "soc": "XS", "loc": "XL"  
prefix_desc =   "place": "Place name", "soc": "Social category", "loc": "Social location"  
db = xapian.Database(DBNAME)
stem = xapian.Stem("english")
qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
for k, v in prefixes.iteritems():
    qp.add_boolean_prefix(k, v)
def make_query(qstring):
    return qp.parse_query(qstring,
        xapian.QueryParser.FLAG_BOOLEAN  
        xapian.QueryParser.FLAG_LOVEHATE  
        xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE  
        xapian.QueryParser.FLAG_WILDCARD  
        xapian.QueryParser.FLAG_PURE_NOT  
        xapian.QueryParser.FLAG_SPELLING_CORRECTION  
        xapian.QueryParser.FLAG_AUTO_SYNONYMS)
@route("/")
def index():
    query = urllib.unquote_plus(bottle.request.GET.get("q", ""))
    out = StringIO()
    print >>out, '''
<html>
<head>
<title>Query</title>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
<script type="text/javascript">
$(function() 
    $("#queryfield")[0].focus()
 )
</script>
</head>
<body>
<h1>Search</h1>
<form method="POST" action="/query">
Keywords: <input type="text" name="query" value="%s" id="queryfield">
<input type="submit">
<a href="http://xapian.org/docs/queryparser.html">Help</a>
</form>''' % query
    print >>out, '''
<p>Example: "car place:wigan"</p>

<p>Available prefixes:</p>

<ul>
'''
    for pfx in prefixes.keys():
        print >>out, "<li><a href='/catinfo/%s'>%s - %s</a></li>" % (pfx, pfx, prefix_desc[pfx])
    print >>out, '''
</ul>
'''
    oldqueries = []
    if os.path.exists(QUERYLOG):
        total = db.get_doccount()
        fd = open(QUERYLOG, "r")
        while True:
            try:
                q = pickle.load(fd)
            except EOFError:
                break
            oldqueries.append(q)
        fd.close()
        def print_query(q):
            count = q["count"]
            print >>out, "<li><a href='/query?query=%s'>%s (%d/%d %.2f%%)</a></li>" % (urllib.quote_plus(q["q"]), q["q"], count, total, count * 100.0 / total)
        print >>out, "<p>Last 10 queries:</p><ul>"
        for q in oldqueries[:-10:-1]:
            print_query(q)
        print >>out, "</ul>"
        # Remove duplicates
        oldqueries = dict(((x["q"], x) for x in oldqueries)).values()
        print >>out, "<table>"
        print >>out, "<tr><th>10 queries with most results</th><th>10 queries with least results</th></tr>"
        print >>out, "<tr><td>"
        print >>out, "<ul>"
        oldqueries.sort(key=lambda x:x["count"], reverse=True)
        for q in oldqueries[:10]:
            print_query(q)
        print >>out, "</ul>"
        print >>out, "</td><td>"
        print >>out, "<ul>"
        nonempty = [x for x in oldqueries if x["count"] > 0]
        nonempty.sort(key=lambda x:x["count"])
        for q in nonempty[:10]:
            print_query(q)
        print >>out, "</ul>"
        print >>out, "</td></tr>"
        print >>out, "</table>"
    print >>out, '''
</body>
</html>'''
    return out.getvalue()
@route("/query")
@route("/query/")
@post("/query")
@post("/query/")
def query():
    query = bottle.request.POST.get("query", bottle.request.GET.get("query", ""))
    enquire = xapian.Enquire(db)
    enquire.set_query(make_query(query))
    count = 40
    matches = enquire.get_mset(0, count)
    estimated = matches.get_matches_estimated()
    total = db.get_doccount()
    out = StringIO()
    print >>out, '''
<html>
<head><title>Results</title></head>
<body>
<h1>Results for "<b>%s</b>"</h1>
''' % query
    if estimated == 0:
        print >>out, "No results found."
    else:
        # Give as results the first 30 documents; also use them as the key
        # ones to use to compute relevant terms
        rset = xapian.RSet()
        for m in enquire.get_mset(0, 30):
            rset.add_document(m.document.get_docid())
        # Compute the tag cloud
        class NonTagFilter(xapian.ExpandDecider):
            def __call__(self, term):
                return not term[0].isupper() and not term[0].isdigit()
        cloud = []
        maxscore = None
        for res in enquire.get_eset(40, rset, NonTagFilter()):
            # Normalise the score in the interval [0, 1]
            weight = math.log(res.weight)
            if maxscore == None: maxscore = weight
            tag = res.term
            cloud.append([tag, float(weight) / maxscore])
        max_weight = cloud[0][1]
        min_weight = cloud[-1][1]
        cloud.sort(key=lambda x:x[0])
        def mklink(query, term):
            return "/query?query=%s" % urllib.quote_plus(query + " and " + term)
        print >>out, "<h2>Tag cloud</h2>"
        print >>out, "<blockquote>"
        for term, weight in cloud:
            size = 100 + 100.0 * (weight - min_weight) / (max_weight - min_weight)
            print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(query, term), size, term)
        print >>out, "</blockquote>"
        print >>out, "<h2>Results</h2>"
        print >>out, "<p><a href='/'>Search again</a></p>"
        print >>out, "<p>%d results over %d documents, %.2f%%</p>" % (estimated, total, estimated * 100.0 / total)
        print >>out, "<p>%d/%d results</p>" % (matches.size(), estimated)
        print >>out, "<ul>"
        for m in matches:
            rec = data[m.document.get_data()]
            print >>out, "<li><a href='/item/%s'>%s</a></li>" % (rec["id"], rec["text"])
        print >>out, "</ul>"
        fd = open(QUERYLOG, "a")
        qinfo = dict(q=query, count=estimated)
        pickle.dump(qinfo, fd)
        fd.close()
    print >>out, '''
<a href="/">Search again</a>

</body>
</html>'''
    return out.getvalue()
@route("/item/:id")
@route("/item/:id/")
def show(id):
    rec = data[id]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Result %s</title></head>
<body>
<h1>Raw JSON record for twit %s</h1>
<pre>''' % (rec["id"], rec["id"])
    print >>out, simplejson.dumps(rec, indent=" ")
    print >>out, '''
</pre>
</body>
</html>'''
    return out.getvalue()
@route("/catinfo/:name")
@route("/catinfo/:name/")
def catinfo(name):
    prefix = prefixes[name]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Values for %s</title></head>
<body>
''' % name
    terms = [(x.term[len(prefix):], db.get_termfreq(x.term)) for x in db.allterms(prefix)]
    terms.sort(key=lambda x:x[1], reverse=True)
    freq_min = terms[0][1]
    freq_max = terms[-1][1]
    def mklink(name, term):
        return "/query?query=%s" % urllib.quote_plus(name + ":" + term)
    # Build tag cloud
    print >>out, "<h1>Tag cloud</h1>"
    print >>out, "<blockquote>"
    for term, freq in sorted(terms[:20], key=lambda x:x[0]):
        size = 100 + 100.0 * (freq - freq_min) / (freq_max - freq_min)
        print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(name, term), size, term)
    print >>out, "</blockquote>"
    print >>out, "<h1>All terms</h1>"
    print >>out, "<table>"
    print >>out, "<tr><th>Occurrences</th><th>Name</th></tr>"
    for term, freq in terms:
        print >>out, "<tr><td>%d</td><td><a href='/query?query=%s'>%s</a></td></tr>" % (freq, urllib.quote_plus(name + ":" + term), term)
    print >>out, "</table>"
    print >>out, '''
</body>
</html>'''
    return out.getvalue()
# Change here for bind host and port
bottle.run(host="0.0.0.0", port=8024)

...and then we presented our work and ended up winning the contest. This was the story of how we wrote this set of award winning code.

I love titles with plenty of buzz-words :) After a bit of google-ing I finally found the right combination of tools that fit my needs. Luckily they are all packaged for debian and well documented. Auto-configure your network is an old topic. There are plenty of articles and howtos. Moreover the gnome NetworkManager already solves many problems alone. But since I don't use NetworkManager I tried to find a solution that works well with wicd. The "right" combination for me so far is to use guessnet and ifplugd to handle my wired connection and use Wicd for the wireless network. In theory, you can go for a fully automated solution using guessnet and wpa_supplicant, but I find much better to hide all the complexity of wpa_supplicant and just use a simple - and unobtrusive - UI to handle my wireless. First thing is to convince Wicd to leave alone my wired connection. This is pretty easy since Wicd wont touch your wired interface if it finds an entry in /etc/network/interfaces for it. Moreover to selecting always switch to a wired connection when available will make Wicd to leave alone you routing tables even if there is a wireless connection available that is marked as automatically connect to this network. Second you need to install guessnet and ifplugd I've noticed that the default settings for ifplugd work pretty well except for the -q option that sometimes lives my interface configured even if I've been disconnected for quite a while. Using ifplugd default, I re-obtain an address every time the cable is plugged-in, maybe causing a small delay in some occasion, but I prefer waiting that de-configuring the interface by hand. Guessnet needs a tiny bit of work. The debian package has already nice defaults that you can copy directly in your /etc/network/interfaces. These will make sure you avoid long timeouts and you leave the interface configured even if no cable is connected. Now you have to add one entry per network that you want to configure: For example to use dhcp on your home network (and of course run any if-up / if-down script as you like). Or : to set a static address at work. Guessnet is nice and well integrated with ifupdown. Very flexible and documented. I had however the impression that from time to time the documentation I found on the net was not in sync with the latest release... I've also noticed that a component of guessnet is not in the package anymore. Guessnet-scan is a small program that will output a stanza to be used with ifupdown based on values sniffed on the wire. This is a kinda of extreme / hackish way to get connected but i reckon a nice tool to have. Hopefully it will make it back in the package sometimes soon... ps: first post on debian planet !

With the scheduled R release of version 2.12.0 this morning, we have just uploaded version 0.8.7 of Rcpp to CRAN; Debian will follow shortly once the autobuilders have processed R 2.12.0 This Rcpp release depends on R 2.12.0 as two things have changed. First, we play along with change in R concerning the ordering of inheritance for time classes. But secondly, and more importantly, we support in Rcpp the corresponding change R itself which brings the new ReferenceClasses. Here is corresponding bit from R's NEWS file for R 2.12.0:

    o A facility for defining reference-based S4 classes (in the OOP
      style of Java, C++, etc.) has been added experimentally to
      package methods; see ?ReferenceClasses.
[...]
    o An experimental new programming model has been added to package
      methods for reference (OOP-style) classes and methods.  See
      ?ReferenceClasses.

This was made possible in large part by code committed by John Chambers (whom we had welcomed recently as a co-author to Rcpp) building on the changes he made to R 2.12.0 itself, as well on the work Romain had done with 'Rcpp Modules'. The R help page for ReferenceClasses carries a reference (bad pun) to Rcpp 0.8.7 so these two releases do go together. This should be a lot of fun over the next little while: S3, S4, and now ReferenceClasses. We also made a number of internal changes some of which leads to speed-ups and internal improvement. The NEWS entry follows below:

0.8.7   2010-10-15
    o   As of this version, Rcpp depends on R 2.12 or greater as it interfaces 
        the new reference classes (see below) and also reflects the POSIXt class
        reordering both of which appeared with R version 2.12.0
    o   new Rcpp::Reference class, that allows internal manipulation of R 2.12.0
        reference classes. The class exposes a constructor that takes the name
        of the target reference class and a field(string) method that implements
        the proxy pattern to get/set reference fields using callbacks to the 
        R operators "$" and "$<-" in order to preserve the R-level encapsulation
    o   the R side of the preceding item allows methods to be written
        in R as per ?ReferenceClasses, accessing fields by name and
        assigning them using "<<-".  Classes extracted from modules
        are R reference classes.  They can be subclassed in R, and/or R methods
        can be defined using the $methods(...) mechanism.
    o   internal performance improvements for Rcpp sugar as well as an added
        'noNA()' wrapper to omit tests for NA values -- see the included
        examples in inst/examples/convolveBenchmarks for the speedups
    o   more internal performance gains with Functions and Environments

As always, even fuller details are in Rcpp Changelog page and the Rcpp page which also leads to the downloads, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page

Five different people asked or commented, in reply to my previous journal post about pkg-config files, why I didn't just have Autoconf generate the pkg-config configuration file directly. One could use the same template that I proposed in the previous post, but rather than adding the sed fragment to Makefile.am, instead add libfoo.pc to AC_CONFIG_FILES and let configure and config.status generate it directly. This is simpler, but it has two problems:

• Autoconf variables may depend on other Autoconf variables (in other words, the value of the variable may contain another unexpanded variable), and Autoconf won't expand all those variables. This means that you need to list each variable and expand it in the pkg-config configuration file so that pkg-config can expand all the variables, but which variables you need isn't well-defined and is to some extent an implementation detail (and has changed in the past). The above solution using a Makefile.am fragment will cause make to expand all the variables for you.
• Running configure with one prefix setting and then overriding that setting when running make should build software using the overridden prefix. This isn't necessary very often, but there are times when it's very useful, particularly when rebuilding parts of the tree. Generating the pkg-config configuration file from the Makefile ensures that this works properly.

I therefore prefer this approach, even though it's a bit more complex. I think it's more robust against both unusual ways in which people want to build my source packages and any future changes to Autoconf. Note, for example, that it's possible and sometimes useful to configure a package with a command like:

    ./configure --libdir='$ datadir '

(admittedly, more common to do that the other way arond) but that would fail with a pkg-config configuration file generated directly from Autoconf unless I'd anticipated that usage and added both datadir and datarootdir to the pkg-config template even though I wasn't using either. I've updated my build system coding style page to include this rationale.

Today, I prepared a patch to reprepro to add support for debian packages using Lzip compression. A patched reprepro of squeeze for amd64 and i386 can be fetched from here. An example package using Lzip compression, lzip itself, is here (for this example, the package is build as 3.0 native on purpose). Hopefully the patch will be merged so lzip compressed binary and source packages can be handled with reprepro.

Paternity leave. I had a baby boy last month, and took the month off work to be with my new boy full time. A newborn is a big deal, he needs all the attention I can give. Actually, he needs more than that. I am back from that vacation, but still, I expect my life style in general to change a little bit more from now on. It will be centred around the baby.

After having dealt with the removal of obsolete conffiles, I ll now explain what you should do when a configuration file managed by dpkg must be renamed. The problem Let s suppose that version 1.2 of the software stopped providing /etc/foo.conf. Instead it provides /etc/bar.conf because the configuration file got renamed. If you do nothing special, the new conffile will be installed with the default configuration, and the old one will stay around. Any customization made by the administrator are lost in the process (in fact they are not lost, they are still in foo.conf but they are unused). Of course, you could do mv /etc/foo.conf /etc/bar.conf in the pre-installation script. But that s not satisfactory: it will generate a spurious conffile prompt that the end-user will not understand. The solution In the preinst script, you have to verify if the old conffile has been modified by the administrator. If yes, you want to keep the file around. Otherwise you know you will be able to ditch it once the upgrade is over, and you rename it to /etc/foo.conf.dpkg-remove to remember this fact. In the postinst script, you remove /etc/foo.conf.dpkg-remove. If the old conffile (/etc/foo.conf) still exists, it s because it was modified by the administrator. You make a backup of the new conffile in /etc/bar.conf.dpkg-dist and rename the old one into /etc/bar.conf. In the postrm, when called to abort an upgrade, you move /etc/foo.conf.dpkg-remove back to its original name. In practice, use dpkg-maintscript-helper dpkg-maintscript-helper can automate all those tasks. You just have to put the following snippet in the maintainer scripts (postinst, postrm, preinst):

if dpkg-maintscript-helper supports mv_conffile 2>/dev/null; then
    dpkg-maintscript-helper mv_conffile /etc/foo.conf /etc/bar.conf 1.1-3 -- "$@"
fi

In this example, I assumed that version 1.1-3 was the last version of the package that contained /etc/foo.conf (i.e. the last version released before 1.2-1 was packaged). You can avoid the preliminary test if you pre-depend on dpkg (>= 1.15.7.2) or if enough time has passed to assume that everybody has a newer version anyway. You can learn all the details in dpkg-maintscript-helper s manual page. Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email (just check Send me blog updates ). No comment Support my work

Bruce Schneier wrote an insightful post about why designing products for wiretapping is a bad idea [1]. It seems that large parts of the Internet will be easy to tap (for both governments and criminals) in the near future unless something is done. The bad results of criminal use will outweigh any benefits of government use. Sam Watkins wrote an informative post about Android security [2]. Among other things any application can read all stored data including all photos, that s got to be a problem for anyone who photographs themself naked Rebecca Saxe gave an interesting TED talk about how brains make moral judgements [3]. Somehow she managed to speak about the Theory of Mind without mentioning Autism once. The Guardian has an amusing article by Cory Doctorow about security policies in banks [4]. He advocates promoting statistical literacy (or at least not promoting a lack of it) as a sound government policy. He also suggests allowing regulators to fine banks that get it wrong. Steven Johnson gave an interesting TED talk about Where Good Ideas Come From [5]. It s a bit slow at the start but gets good at the end. Adam Grosser gave an interesting TED talk about a fridge that was designed for use in Africa [6]. The core of the Absorption Refrigerator is designed to be heated in a pot of water in a cooking fire and it can then keep food cool for 12 hours. It s a pity that they couldn t design it to work on solar power to avoid the fuel use for the cooking fire. Josh Silver gave an interesting TED talk about liquid filled spectacles [7]. The glasses are shipped with a syringe filled with liquid at each side that is used to inflate the lenses to the desired refractive index. The wearer can just adjust the syringes until they get to the right magnification, as there are separate syringes the glasses work well with people who s eyes aren t identical (which is most people). Once the syringes are at the right spots the user can tighten some screws to prevent further transfer of liquid and cut the syringes off to give glasses that aren t overly heavy but which can t be adjusted any more, I guess that a natural extension to this would be to allow the syringes to be re-attached so that the user could adjust them every year to match declining vision. One thing that this wouldn t do is counter for Astigmatism (where the lens of the eye doesn t focus light to a point), but I guess they could make lenses to deal with a few common varieties of Astigmatism so that most people who have that problem can get a reasonable approximation. The current best effort is to make the glasses cost $19, which is 19 days salary for some of the poorest people in the world. Glasses in Australia cost up to $650 for a pair (or a more common cost of $200 or about $100 after Medicare) which would be about one day s salary. Eben Bayer gave an inspiring TED talk about one of the ways that mushrooms can save the planet [8]. He has designed molds that can be filled with Pasteurised organic waste (seed husks etc) and then seeded with fungal spores. The fungus then grows mycelium (thin fungal root fibers) through the organic waste making it into a solid structure which fits the shape of the mold. This is currently being used to replace poly-styrene foam for packaging and can apparently be used for making tiles that are fire retardant and sound proof for constructing buildings. The main benefits of the material are that it can be cheaply made without petrochemicals and that it is bio-degradable, I m not sure how the bio-degradable part would work with constructing buildings maybe they would just replace the panels every few years. Annie Lennox gave a TED talk about her Sing foundation to care for women and children who are affected by AIDS [9]. She describes the effect of AIDS in Africa as Genocide. Robert Ballard gave a very informative TED talk about exploring the oceans [10]. This was one of the most informative TED talks I ve seen and Robert is also one of the most enthusiastic speakers I ve seen, it s really worth watching! We really need more money spent on exploring the oceans. Jessa Gamble gave an interesting TED talk which suggests that the best thing to do is to go to bed at about sunset and then have a couple of hours of relaxing time during the middle of the night [11]. Apparently the subjects of body-block experiments who live for a month in a bunker without natural light or access to a clock get better sleep in this manner than they ever had in their life and feel fully awake for the first time. World Changing is a blog that has a lot of interesting articles about climate change and related issues [12]. It s worth a read. Cynthia Schneider gave an interesting TED talk about how reality competition TV is affecting reality [13]. Shows that are derived from the American Idol concept are driving a resurgence in some traditional forms of performance art while also promoting equality among other things it s apparent that winning is more important than misogyny. The Ritual of the Calling of an Engineer is an interesting concept [14]. I think it would be good to have something similar for Computer Science. Benjamin Mako Hill wrote an interesting and insightful essay about Piracy and Free Software [15].

• [1] http://www.schneier.com/blog/archives/2010/09/wiretapping_the.html
• [2] http://sswam.wordpress.com/2010/09/25/android-security-model-is-insufficient-and-it-needs-a-firewall/
• [3] http://www.ted.com/talks/rebecca_saxe_how_brains_make_moral_judgments.html
• [4] http://www.guardian.co.uk/technology/2010/sep/21/password-security-cory-doctorow
• [5] http://www.ted.com/talks/steven_johnson_where_good_ideas_come_from.html
• [6] http://www.ted.com/talks/lang/eng/adam_grosser_and_his_sustainable_fridge.html
• [7] http://www.ted.com/talks/lang/eng/josh_silver_demos_adjustable_liquid_filled_eyeglasses.html
• [8] http://www.ted.com/talks/download/video/9786/talk/971
• [9] http://www.ted.com/talks/annie_lennox_why_i_am_an_hiv_aids_activist.html
• [10] http://www.ted.com/talks/lang/eng/robert_ballard_on_exploring_the_oceans.html
• [11] http://www.ted.com/talks/jessa_gamble_how_to_sleep.html
• [12] http://www.worldchanging.com/
• [13] http://www.ted.com/talks/cynthia_schneider_the_surprising_spread_of_idol_tv.html
• [14] http://en.wikipedia.org/wiki/The_Ritual_of_the_Calling_of_an_Engineer
• [15] http://mako.cc/copyrighteous/20101011-00

A little while back, as part of a discussion on debian-devel, I spent some time bad-mouthing pkg-config, and then later realized (with the help of an IRC conversation) that I was doing so unfairly. So as penance here is some documentation for how to add a pkg-config configuration file to a typical Automake project. This information is also in my build system coding style and will be maintained there (so if you see this some time after I wrote it, check that page for more up-to-date information). pkg-config is a tool to provide the flags required to compile or link against a shared library. It's a replacement for complex Autoconf probes and for all of the incompatible and occasionally buggy *-config scripts. To install a pkg-config configuration file, provide a template pkg-config configuration in a file named after the library but with a .pc.in extension. Here is an example:

    prefix=@prefix@
    exec_prefix=@exec_prefix@
    includedir=@includedir@
    libdir=@libdir@
    Name: <library>
    Description: <short description>
    URL: <upstream url>
    Version: @PACKAGE_VERSION@
    Cflags: -I$ includedir 
    Libs: -L$ libdir  -l<library>
    Libs.private: @KRB5_LDFLAGS@ @KRB5_LIBS@

Replace <library> with the official name of the library (which isn't necessarily the library name with lib removed). This is only informational, may contain spaces, and isn't used to locate the file. Its filename is used for that. Replace <short description> with a short description of the purpose of the library. Often the short description of the package from the start of the README file works well here. Replace <upstream url> with the URL of the package's distribution page. Normally, all Cflags needs to contain is what's indicated above, which lets the compiler find the header files if installed in a non-standard location. For Libs, note that pkg-config distinguishes between the shared libraries that must be linked against on a system with proper transitive dependency support and the libraries that must be linked for a static build or on a system with poor support for shared libraries. For a well-designed shared library, one should rarely need anything in Libs other than what's listed above. Libs.private, on the other hand, needs all the LDFLAGS and LIBS of everything the library itself links with external to this package. In this example, the shared library is linked with the Kerberos libraries and therefore needs the Kerberos LDFLAGS and LIBS; modify as needed for your example. If the dependencies of the shared library also use pkg-config, just list the other pkg-config file names in Requires or Requires.private instead of using Libs or Libs.private for them. Finally, add the Automake code to build and install the pkg-config configuration file. It should look like this:

    pkgconfigdir = $(libdir)/pkgconfig
    nodist_pkgconfig_DATA = lib/libfoo.pc
    lib/libfoo.pc: $(srcdir)/lib/libfoo.pc.in
            sed -e 's![@]prefix[@]!$(prefix)!g' \
                -e 's![@]exec_prefix[@]!$(exec_prefix)!g' \
                -e 's![@]includedir[@]!$(includedir)!g' \
                -e 's![@]libdir[@]!$(libdir)!g' \
                -e 's![@]PACKAGE_VERSION[@]!$(PACKAGE_VERSION)!g' \
                -e 's![@]KRB5_LDFLAGS[@]!$(KRB5_LDFLAGS)!g' \
                -e 's![@]KRB5_LIBS[@]!$(KRB5_LIBS)!g' \
                $(srcdir)/lib/libfoo.pc.in > $@

Replace libfoo with the actual name of the library and change its path in the source package as needed. Note the pattern above and use it for any Autoconf substitution variables that you need to replace in the pkg-config template. The above example adds KRB5_LDFLAGS and KRB5_LIBS to the variables; you may have other ones you need, particularly to construct Libs.private. The above installs the pkg-config configuration file under $(libdir)/pkgconfig. $(datadir)/pkgconfig is appropriate instead if one is certain that the flags required to link with this library will never vary by architecture, but that's not the case for most libraries due to Libs.private. Note that the Automake documentation recommends using comma as the sed delimiter for doing this sort of trick, but LDFLAGS frequently contains a comma since it's the separator for parts of a linker-specific option. I use an exclamation point instead.

Today, I prepared a patch to dpkg to add support using Lzip compression. A patched dpkg of squeeze for amd64 and i386 can be fetched from here. An example package using Lzip compression, lzip itself, is here (for this example, the package is build as 3.0 native on purpose). Hopefully the patch will be merged so we can make use of lzip compressed binary and source packages in Debian.

The recent CVE-2010-2961 mountall vulnerability got a nice write-up by xorl today. I ve seen a few public exploits for it, but those that I ve seen, including the one in xorl s post, miss a rather important point: udev events can be triggered by regular users without any hardware fiddling. While the bug that kept udev from running inotify correctly on the /dev/.udev/rules.d directory during initial boot kept this vulnerability exposure pretty well minimized, the fact that udev events can be triggered at will made it pretty bad too. If udev had already been restarted, an attacker didn t have to wait at all, nor have physical access to the system. While it is generally understood that udev events are related to hardware, it s important to keep in mind that it also sends events on module loads, and module loads can happen on demand from unprivileged users. For example, say you want to send an X.25 packet, when you call socket(AF_X25, SOCK_STREAM), the kernel will go load net-pf-9, which modules.alias lists as the x25 module. And once loaded, udev sends a module event. (Which, by the way, should serve as a reminder to people to block module loading if you can.) So, as I mentioned, here s yet another exploit for the mountall vulnerability: mountall-CVE-2010-2961.py. It writes to the vulnerable udev rule file and then attempts to trigger udev immediately by walking a list of possible socket() AF_* types.

Today, I launched the contest for prediction of the day Debian bugs #600000 and #1000000 will be reported. Have you bet already? (update: I mistakenly published an old version of this call for bet, yesterday. Sorry for this)

Yesterday, the Application Bridging for Federated Authentication working group was approved in the IETF. This working group s charter includes the IETF technologies needed by Project Moonshot. The group will meet at IETF 79 in Beijing this November. Meanwhile, at last month s Moonshot meeting in Copenhagen, an initial version of the technology was demonstrated. We re still working through some of the administrative details needed before we can release the code for public review. There have been several exciting discussions both on the Moonshot implementation list and on the ABFAB list over the past few weeks.

The bad news is that you won t be able to use ZFS as your root filesystem in Debian Squeeze with the official installer. The blocker is missing support in GNU Parted. Unfortunately the patch I sent in August wasn t integrated in time for the freeze (and still isn t, but there s no hurry now, it ll hopefully be there for Wheezy). The good news is that all fixes required for the installed system (including ZFS userland utilities and GRUB) made it and are already in Squeeze. As a result, the unofficial installer I built in September will (unless something weird happens) continue to work during Squeeze life cycle. Note that although the installer itself is modified, it installs a pristine Debian GNU/kFreeBSD system from official packages. Hope this is helpful to you if you plan on deploying ZFS in your organization and the other approaches don t fit well enough with your needs.