Another week has passed, where is Squeeze? Sadly, I did even less than the week before, but here is the log anyways:

• #599782 pyopencl: FTBFS: ImportError: No module named pyopencl
  the build process imports the just built module to generate the docs but the build-path guessing in debian/rules is broken, after some testing Jakub has posted the right snippet needed. RT has to decide whether maintainer can upload a slightly updated upstream version with the fix, or the on in the archive (with the fix too ;))
• #598202 update-manager-gnome: will not start
  this one is from the last week neither me nor the submitter can reproduce it any longer, thus closed
• #599523 update-manager-gnome: wants to downgrade packages without any notice
  submitters pinning seems broken, waiting for a reply

That s all. Sorry, I had a busy week.

The PSAS team planned to have their LV2.3 airframe updates ready for BALLS this year, but RL interfered and they weren t ready to go on time. A quick bit of rescheduling and we decided to fly over the weekend of the 16-17 of October. Tsolo and family welcomed us warmly as we arrived on Saturday afternoon after driving over from Portland. They ve got a new puppy who is also cute and friendly. We managed to get the launch tower and flight operations set up before dark, cooked dinner and spent the evening prepping the airframe. It was cold overnight, but the light cloud cover that had been there when we arrived was nearly gone by morning. I called in to open the waiver and was asked to hold until noon, which suited us just fine. The rocket finally made it onto the rail around 1:30 and we managed to convince the launch controller to actually light the motor at 2:44:39 (according to the GPS data). Flying on a CTI N2850 blue-streak, LV2.3 hit about 380m/s and reached 4851m. This was the first flight with the new fin system, you can see how that works in the video here: <iframe frameborder="0" height="480" src="http://player.vimeo.com/video/15972443" width="640"></iframe>Portland State Aerospace LV2.3 flight with roll control from Keith Packard on Vimeo. The roll control system worked as designed, executing a programmed sequence of maneuvers during ascent. You can see the canard fins moving back and forth in the video, controlling the roll of the airframe. Yes, the fins really do spin separately from the airframe. You can see how that was constructed here: PSAS LV2.3 Spin Can Live telemetry and stored flight data, consisting of GPS coordinates along with barometry altimeter and accelerometer information, was collected using the TeleMetrum flight computer. The airframe was recovered about 800m downrange. Thanks to Portland State University for their support in hosting PSAS and Oregon Rocketry for letting us fly at their launch site.

State of X Server 1.9 The push of X Server 1.9 mentioned previously continued past weekend, with an upload of 1.9.1 rc2 this time, which is likely to become 1.9.1 in a few days. To keep it company in experimental, almost all X video drivers were uploaded as well, with a few exceptions:

• intel: We would like the current version in sid to migrate to squeeze now that a suitable kernel ( 2.6.32-24) is available there. Additional patches from experimental or from upstream might be added to sid then. For now, I kept 2.13 (built against X Server 1.7) in experimental, since I asked users a couple of times to try 2.13 from there.
• nouveau and openchrome: There were entirely new codebases in experimental for both of them, so I didn t touch anything there.
• nv: We plan to drop it entirely in favor of nouveau.

If my package manager isn t lying, the xserver-xorg-video-all meta package pulling everything could almost be installed, if one were to ignore those drivers. The same goes for xserver-xorg-input-all: almost alright, except for xserver-xorg-input-wacom, which is maintained outside debian-x. Autobuilding X packages So we have mostly this situation: 1.7 in sid, 1.9 in experimental. But we may want to build experimental packages (new upstream releases, or packages with patches cherry-picked from upstream) against either of them, without interfering with what is in the archive. We could ask users to try some patches. Actually, we do that already, and while many of them are usually responsive and willing to do so, not everyone knows how to build a package, or has time to. That s why I spent some time setting up http://autobuild.ikibiki.org/, where I can push source packages in the autobuild-unstable or autobuild-experimental suites, so that they get built on amd64 and on i386. It supports multiple versions in a given suite, which may help narrowing down when a fix appeared. That s part of a let s try to stick as closely as possible to upstream plan. We regularly get blamed for not doing so, but there are several reasons (or excuses? You decide.) for that:

• Spending time stabilizing stuff is the established standard in Debian, with that stuff called freeze. This means that rushing for the very last upstream releases isn t usually a good idea.
• X is some kind of a beast. Many libraries, many drivers, and a server. That means plenty of time to spend. And we re only a handful (and that includes Ubuntu folks who are helping us preparing stuff in git, thanks guys).

So if anyone thinks we re willingly holding back any kind of move towards newer upstream releases or the like, rest assured that s not the case. There s just too few of us to closely track upstream (which we do by reporting bugs and trying to cherry-pick patches anyway) in addition to preparing stuff for the next stable release. Now that things are settling down, that sounds feasible somehow, but remember: your help is more than welcome. Speaking of those Ubuntu guys, Robert Hooker is currently updating a bunch of drivers in experimental, yay.

Currently, the main product of the Debian project is its stable release^[1]. Those release come out approximately every 18-24 months. This article gives a short overview of the process leading to the next stable release. Creating a new distribution Immediately after a stable release, a new distribution is created in the Debian archive. Its initial content is a copy of the (just released) stable distribution. Its codename is decided by the release managers and there s a tradition of picking a character s name from the Toy Story movie. As an example, the wheezy distribution will be created once squeeze (aka Debian 6.0) is out. For simplicity there s a generic name to refer to the distribution used to prepare the next stable release: it s testing. In the Debian archive, testing is just a symbolic link pointing to the right directory (squeeze currently). Updating packages, working on release goals During most of the cycle, developers work on packaging new upstream versions and implementing release goals. They upload their packages in the unstable distribution. From there packages trickle to the testing distribution once they satisfy some quality checks: they must not have new release-critical bugs, they must have been built on all architectures that were previously supported, they must not break any dependency in testing, and they must have spent at least 10 days in unstable. This minimal period ensures the package has been tested and gives enough time to users to file bugs if the package is suffering from problems. If the problems are deemed release-critical, they will block the migration of the package to testing. During this part of the release cycle, the main work of the release team is ensuring that updated packages flow from unstable to testing. It can be a tricky task: package dependencies frequently tie packages together so that they can only migrate to testing together. If only one of the tied packages is not ready (for example if a new revision has been uploaded and has not spent 10 days in unstable yet), then none of them can migrate. Stabilizing, polishing, fixing release-critical issues The constant churn of new packages makes it very difficult to build a very polished release. That s why, at some point, release managers freeze the testing distribution: automatic updates are stopped and they vet every single update made to testing. They have strong requirements, the goal is to only allow updates fixing release critical bugs, or those which are low-risk and bring significant value to the user experience (like new translations, updated documentation, etc.). During freeze, some packages are also removed because the current upstream version can t be supported for the lifetime of the stable release. The freeze tends to slow down the pace of changes in unstable. Many maintainers opt to push new upstream versions in experimental instead so that if they need to update their packages in testing, they can still do it through unstable. This procedure is recommended by the release managers because it means that updates that they unblock have been tested as usual. It s not the case for updates uploaded directly to testing (through testing-proposed-updates). This behavior is rather annoying for the bleeding-edge users that use testing or unstable like a rolling release. Release time Once release managers are satisfied of the quality of the new distribution, some last minute work is needed, like generating the CD images. In the Debian archive, the release is made official by pointing the stable symbolic link to the new distribution (and the oldstable one to the previous distribution). Now it s party time, the cycle is over, and a new one can start. ^[1] The Constantly Usable Testing project aims to make testing a first-class product like stable but with a very different update policy.

Follow me on Identi.ca, Twitter and Facebook.

3 comments Support my work

Award winning code Me and Yuwei had a fun day at hhhmcr (#hhhmcr) and even managed to put together a prototype that won the first prize \o/ We played with the gmp24 dataset kindly extracted from Twitter by Michael Brunton-Spall of the Guardian into a convenient JSON dataset. The idea was to find ways of making it easier to look at the data and making sense of it. This is the story of what we did, including the code we wrote. The original dataset has several JSON files, so the first task was to put them all together:

#!/usr/bin/python
# Merge the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import os
res = []
for f in os.listdir("."):
    if not f.startswith("gmp24"): continue
    data = open(f).read().strip()
    if data == "[]": continue
    parsed = simplejson.loads(data)
    res.extend(parsed)
print simplejson.dumps(res)

The results however were not ordered by date, as GMP had to use several accounts to twit because Twitter was putting Greather Manchester Police into jail for generating too much traffic. There would be quite a bit to write about that, but let's stick to our work. Here is code to sort the JSON data by time:

#!/usr/bin/python
# Sort the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import sys
import datetime as dt
all_recs = simplejson.load(sys.stdin)
all_recs.sort(key=lambda x: dt.datetime.strptime(x["created_at"], "%a %b %d %H:%M:%S +0000 %Y"))
simplejson.dump(all_recs, sys.stdout)

I then wanted to play with Tf-idf for extracting the most important words of every tweet:

#!/usr/bin/python
# tfifd - Annotate JSON elements with Tf-idf extracted keywords
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import sys, math
import simplejson
import re
# Read all the twits
records = simplejson.load(sys.stdin)
# All the twits by ID
byid = dict(((x["id"], x) for x in records))
# Stopwords we ignore
stopwords = set(["by", "it", "and", "of", "in", "a", "to"])
# Tokenising engine
re_num = re.compile(r"^\d+$")
re_word = re.compile(r"(\w+)")
def tokenise(tweet):
    "Extract tokens from a tweet"
    for tok in tweet["text"].split():
        tok = tok.strip().lower()
        if re_num.match(tok): continue
        mo = re_word.match(tok)
        if not mo: continue
        if mo.group(1) in stopwords: continue
        yield mo.group(1)
# Extract tokens from tweets
tokenised = dict(((x["id"], list(tokenise(x))) for x in records))
# Aggregate token counts
aggregated =  
for d in byid.iterkeys():
    for t in tokenised[d]:
        if t in aggregated:
            aggregated[t] += 1
        else:
            aggregated[t] = 1
def tfidf(doc, tok):
    "Compute TFIDF score of a token in a document"
    return doc.count(tok) * math.log(float(len(byid)) / aggregated[tok])
# Annotate tweets with keywords
res = []
for name, tweet in byid.iteritems():
    doc = tokenised[name]
    keywords = sorted(set(doc), key=lambda tok: tfidf(doc, tok), reverse=True)[:5]
    tweet["keywords"] = keywords
    res.append(tweet)
simplejson.dump(res, sys.stdout)

I thought this was producing a nice summary of every tweet but nobody was particularly interested, so we moved on to adding categories to tweet. Thanks to Yuwei who put together some useful keyword sets, we managed to annotate each tweet with a place name (i.e. "Stockport"), a social place name (i.e. "pub", "bank") and a social category (i.e. "man", "woman", "landlord"...) The code is simple; the biggest work in it was the dictionary of keywords:

#!/usr/bin/python
# categorise - Annotate JSON elements with categories
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
# Copyright (C) 2010  Yuwei Lin <yuwei@ylin.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import sys, math
import simplejson
import re
# Electoral wards from http://en.wikipedia.org/wiki/List_of_electoral_wards_in_Greater_Manchester
placenames = ["Altrincham", "Sale West",
"Altrincham", "Ashton upon Mersey", "Bowdon", "Broadheath", "Hale Barns", "Hale Central", "St Mary", "Timperley", "Village",
"Ashton-under-Lyne",
"Ashton Hurst", "Ashton St Michael", "Ashton Waterloo", "Droylsden East", "Droylsden West", "Failsworth East", "Failsworth West", "St Peter",
"Blackley", "Broughton",
"Broughton", "Charlestown", "Cheetham", "Crumpsall", "Harpurhey", "Higher Blackley", "Kersal",
"Bolton North East",
"Astley Bridge", "Bradshaw", "Breightmet", "Bromley Cross", "Crompton", "Halliwell", "Tonge with the Haulgh",
"Bolton South East",
"Farnworth", "Great Lever", "Harper Green", "Hulton", "Kearsley", "Little Lever", "Darcy Lever", "Rumworth",
"Bolton West",
"Atherton", "Heaton", "Lostock", "Horwich", "Blackrod", "Horwich North East", "Smithills", "Westhoughton North", "Chew Moor", "Westhoughton South",
"Bury North",
"Church", "East", "Elton", "Moorside", "North Manor", "Ramsbottom", "Redvales", "Tottington",
"Bury South",
"Besses", "Holyrood", "Pilkington Park", "Radcliffe East", "Radcliffe North", "Radcliffe West", "St Mary", "Sedgley", "Unsworth",
"Cheadle",
"Bramhall North", "Bramhall South", "Cheadle", "Gatley", "Cheadle Hulme North", "Cheadle Hulme South", "Heald Green", "Stepping Hill",
"Denton", "Reddish",
"Audenshaw", "Denton North East", "Denton South", "Denton West", "Dukinfield", "Reddish North", "Reddish South",
"Hazel Grove",
"Bredbury", "Woodley", "Bredbury Green", "Romiley", "Hazel Grove", "Marple North", "Marple South", "Offerton",
"Heywood", "Middleton",
"Bamford", "Castleton", "East Middleton", "Hopwood Hall", "Norden", "North Heywood", "North Middleton", "South Middleton", "West Heywood", "West Middleton",
"Leigh",
"Astley Mosley Common", "Atherleigh", "Golborne", "Lowton West", "Leigh East", "Leigh South", "Leigh West", "Lowton East", "Tyldesley",
"Makerfield",
"Abram", "Ashton", "Bryn", "Hindley", "Hindley Green", "Orrell", "Winstanley", "Worsley Mesnes",
"Manchester Central",
"Ancoats", "Clayton", "Ardwick", "Bradford", "City Centre", "Hulme", "Miles Platting", "Newton Heath", "Moss Side", "Moston",
"Manchester", "Gorton",
"Fallowfield", "Gorton North", "Gorton South", "Levenshulme", "Longsight", "Rusholme", "Whalley Range",
"Manchester", "Withington",
"Burnage", "Chorlton", "Chorlton Park", "Didsbury East", "Didsbury West", "Old Moat", "Withington",
"Oldham East", "Saddleworth",
"Alexandra", "Crompton", "Saddleworth North", "Saddleworth South", "Saddleworth West", "Lees", "St James", "St Mary", "Shaw", "Waterhead",
"Oldham West", "Royton",
"Chadderton Central", "Chadderton North", "Chadderton South", "Coldhurst", "Hollinwood", "Medlock Vale", "Royton North", "Royton South", "Werneth",
"Rochdale",
"Balderstone", "Kirkholt", "Central Rochdale", "Healey", "Kingsway", "Littleborough Lakeside", "Milkstone", "Deeplish", "Milnrow", "Newhey", "Smallbridge", "Firgrove", "Spotland", "Falinge", "Wardle", "West Littleborough",
"Salford", "Eccles",
"Claremont", "Eccles", "Irwell Riverside", "Langworthy", "Ordsall", "Pendlebury", "Swinton North", "Swinton South", "Weaste", "Seedley",
"Stalybridge", "Hyde",
"Dukinfield Stalybridge", "Hyde Godley", "Hyde Newton", "Hyde Werneth", "Longdendale", "Mossley", "Stalybridge North", "Stalybridge South",
"Stockport",
"Brinnington", "Central", "Davenport", "Cale Green", "Edgeley", "Cheadle Heath", "Heatons North", "Heatons South", "Manor",
"Stretford", "Urmston",
"Bucklow-St Martins", "Clifford", "Davyhulme East", "Davyhulme West", "Flixton", "Gorse Hill", "Longford", "Stretford", "Urmston",
"Wigan",
"Aspull New Springs Whelley", "Douglas", "Ince", "Pemberton", "Shevington with Lower Ground", "Standish with Langtree", "Wigan Central", "Wigan West",
"Worsley", "Eccles South",
"Barton", "Boothstown", "Ellenbrook", "Cadishead", "Irlam", "Little Hulton", "Walkden North", "Walkden South", "Winton", "Worsley",
"Wythenshawe", "Sale East",
"Baguley", "Brooklands", "Northenden", "Priory", "Sale Moor", "Sharston", "Woodhouse Park"]
# Manual coding from Yuwei
placenames.extend(["City centre", "Tameside", "Oldham", "Bury", "Bolton",
"Trafford", "Pendleton", "New Moston", "Denton", "Eccles", "Leigh", "Benchill",
"Prestwich", "Sale", "Kearsley", ])
placenames.extend(["Trafford", "Bolton", "Stockport", "Levenshulme", "Gorton",
"Tameside", "Blackley", "City centre", "Airport", "South Manchester",
"Rochdale", "Chorlton", "Uppermill", "Castleton", "Stalybridge", "Ashton",
"Chadderton", "Bury", "Ancoats", "Whalley Range", "West Yorkshire",
"Fallowfield", "New Moston", "Denton", "Stretford", "Eccles", "Pendleton",
"Leigh", "Altrincham", "Sale", "Prestwich", "Kearsley", "Hulme", "Withington",
"Moss Side", "Milnrow", "outskirt of Manchester City Centre", "Newton Heath",
"Wythenshawe", "Mancunian Way", "M60", "A6", "Droylesden", "M56", "Timperley",
"Higher Ince", "Clayton", "Higher Blackley", "Lowton", "Droylsden",
"Partington", "Cheetham Hill", "Benchill", "Longsight", "Didsbury",
"Westhoughton"])
# Social categories from Yuwei
soccat = ["man", "woman", "men", "women", "youth", "teenager", "elderly",
"patient", "taxi driver", "neighbour", "male", "tenant", "landlord", "child",
"children", "immigrant", "female", "workmen", "boy", "girl", "foster parents",
"next of kin"]
for i in range(100):
    soccat.append("%d-year-old" % i)
    soccat.append("%d-years-old" % i)
# Types of social locations from Yuwei
socloc = ["car park", "park", "pub", "club", "shop", "premises", "bus stop",
"property", "credit card", "supermarket", "garden", "phone box", "theatre",
"toilet", "building site", "Crown court", "hard shoulder", "telephone kiosk",
"hotel", "restaurant", "cafe", "petrol station", "bank", "school",
"university"]
extras =   "placename": placenames, "soccat": soccat, "socloc": socloc  
# Normalise keyword lists
for k, v in extras.iteritems():
    # Remove duplicates
    v = list(set(v))
    # Sort by length
    v.sort(key=lambda x:len(x), reverse=True)
# Add keywords
def add_categories(tweet):
    text = tweet["text"].lower()
    for field, categories in extras.iteritems():
        for cat in categories:
            if cat.lower() in text:
                tweet[field] = cat
                break
    return tweet
# Read all the twits
records = (add_categories(x) for x in simplejson.load(sys.stdin))
simplejson.dump(list(records), sys.stdout)

All these scripts form a nice processing chain: each script takes a list of JSON records, adds some bit and passes it on. In order to see what we have so far, here is a simple script to convert the JSON twits to CSV so they can be viewed in a spreadsheet:

#!/usr/bin/python
# Convert the JSON twits to CSV
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import sys
import csv
rows = ["id", "created_at", "text", "keywords", "placename"]
writer = csv.writer(sys.stdout)
for rec in simplejson.load(sys.stdin):
    rec["keywords"] = " ".join(rec["keywords"])
    rec["placename"] = rec.get("placename", "")
    writer.writerow([rec[row] for row in rows])

At this point we were coming up with lots of questions: "were there more reports on women or men?", "which place had most incidents?", "what were the incidents involving animals?"... Time to bring Xapian into play. This script reads all the JSON tweets and builds a Xapian index with them:

#!/usr/bin/python
# toxapian - Index JSON tweets in Xapian
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import simplejson
import sys
import os, os.path
import xapian
DBNAME = sys.argv[1]
db = xapian.WritableDatabase(DBNAME, xapian.DB_CREATE_OR_OPEN)
stemmer = xapian.Stem("english")
indexer = xapian.TermGenerator()
indexer.set_stemmer(stemmer)
indexer.set_database(db)
data = simplejson.load(sys.stdin)
for rec in data:
    doc = xapian.Document()
    doc.set_data(str(rec["id"]))
    indexer.set_document(doc)
    indexer.index_text_without_positions(rec["text"])
    # Index categories as categories
    if "placename" in rec:
        doc.add_boolean_term("XP" + rec["placename"].lower())
    if "soccat" in rec:
        doc.add_boolean_term("XS" + rec["soccat"].lower())
    if "socloc" in rec:
        doc.add_boolean_term("XL" + rec["socloc"].lower())
    db.add_document(doc)
db.flush()
# Also save the whole dataset so we know where to find it later if we want to
# show the details of an entry
simplejson.dump(data, open(os.path.join(DBNAME, "all.json"), "w"))

And this is a simple command line tool to query to the database:

#!/usr/bin/python
# xgrep - Command line tool to query the GMP24 tweet Xapian database
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import simplejson
import sys
import os, os.path
import xapian
DBNAME = sys.argv[1]
db = xapian.Database(DBNAME)
stem = xapian.Stem("english")
qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
qp.add_boolean_prefix("place", "XP")
qp.add_boolean_prefix("soc", "XS")
qp.add_boolean_prefix("loc", "XL")
query = qp.parse_query(sys.argv[2],
    xapian.QueryParser.FLAG_BOOLEAN  
    xapian.QueryParser.FLAG_LOVEHATE  
    xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE  
    xapian.QueryParser.FLAG_WILDCARD  
    xapian.QueryParser.FLAG_PURE_NOT  
    xapian.QueryParser.FLAG_SPELLING_CORRECTION  
    xapian.QueryParser.FLAG_AUTO_SYNONYMS)
enquire = xapian.Enquire(db)
enquire.set_query(query)
count = 40
matches = enquire.get_mset(0, count)
estimated = matches.get_matches_estimated()
print "%d/%d results" % (matches.size(), estimated)
data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))
for m in matches:
    rec = data[m.document.get_data()]
    print rec["text"]
print "%d/%d results" % (matches.size(), matches.get_matches_estimated())
total = db.get_doccount()
estimated = matches.get_matches_estimated()
print "%d results over %d documents, %d%%" % (estimated, total, estimated * 100 / total)

Neat! Now that we have a proper index that supports all sort of cool things, like stemming, tag clouds, full text search with complex queries, lookup of similar documents, suggest keywords and so on, it was just fair to put together a web service to share it with other people at the event. It helped that I had already written similar code for apt-xapian-index and dde before. Here is the server, quickly built on bottle. The very last line starts the server and it is where you can configure the listening interface and port.

#!/usr/bin/python
# xserve - Make the GMP24 tweet Xapian database available on the web
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import bottle
from bottle import route, post
from cStringIO import StringIO
import cPickle as pickle
import simplejson
import sys
import os, os.path
import xapian
import urllib
import math
bottle.debug(True)
DBNAME = sys.argv[1]
QUERYLOG = os.path.join(DBNAME, "queries.txt")
data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))
prefixes =   "place": "XP", "soc": "XS", "loc": "XL"  
prefix_desc =   "place": "Place name", "soc": "Social category", "loc": "Social location"  
db = xapian.Database(DBNAME)
stem = xapian.Stem("english")
qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
for k, v in prefixes.iteritems():
    qp.add_boolean_prefix(k, v)
def make_query(qstring):
    return qp.parse_query(qstring,
        xapian.QueryParser.FLAG_BOOLEAN  
        xapian.QueryParser.FLAG_LOVEHATE  
        xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE  
        xapian.QueryParser.FLAG_WILDCARD  
        xapian.QueryParser.FLAG_PURE_NOT  
        xapian.QueryParser.FLAG_SPELLING_CORRECTION  
        xapian.QueryParser.FLAG_AUTO_SYNONYMS)
@route("/")
def index():
    query = urllib.unquote_plus(bottle.request.GET.get("q", ""))
    out = StringIO()
    print >>out, '''
<html>
<head>
<title>Query</title>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
<script type="text/javascript">
$(function() 
    $("#queryfield")[0].focus()
 )
</script>
</head>
<body>
<h1>Search</h1>
<form method="POST" action="/query">
Keywords: <input type="text" name="query" value="%s" id="queryfield">
<input type="submit">
<a href="http://xapian.org/docs/queryparser.html">Help</a>
</form>''' % query
    print >>out, '''
<p>Example: "car place:wigan"</p>

<p>Available prefixes:</p>

<ul>
'''
    for pfx in prefixes.keys():
        print >>out, "<li><a href='/catinfo/%s'>%s - %s</a></li>" % (pfx, pfx, prefix_desc[pfx])
    print >>out, '''
</ul>
'''
    oldqueries = []
    if os.path.exists(QUERYLOG):
        total = db.get_doccount()
        fd = open(QUERYLOG, "r")
        while True:
            try:
                q = pickle.load(fd)
            except EOFError:
                break
            oldqueries.append(q)
        fd.close()
        def print_query(q):
            count = q["count"]
            print >>out, "<li><a href='/query?query=%s'>%s (%d/%d %.2f%%)</a></li>" % (urllib.quote_plus(q["q"]), q["q"], count, total, count * 100.0 / total)
        print >>out, "<p>Last 10 queries:</p><ul>"
        for q in oldqueries[:-10:-1]:
            print_query(q)
        print >>out, "</ul>"
        # Remove duplicates
        oldqueries = dict(((x["q"], x) for x in oldqueries)).values()
        print >>out, "<table>"
        print >>out, "<tr><th>10 queries with most results</th><th>10 queries with least results</th></tr>"
        print >>out, "<tr><td>"
        print >>out, "<ul>"
        oldqueries.sort(key=lambda x:x["count"], reverse=True)
        for q in oldqueries[:10]:
            print_query(q)
        print >>out, "</ul>"
        print >>out, "</td><td>"
        print >>out, "<ul>"
        nonempty = [x for x in oldqueries if x["count"] > 0]
        nonempty.sort(key=lambda x:x["count"])
        for q in nonempty[:10]:
            print_query(q)
        print >>out, "</ul>"
        print >>out, "</td></tr>"
        print >>out, "</table>"
    print >>out, '''
</body>
</html>'''
    return out.getvalue()
@route("/query")
@route("/query/")
@post("/query")
@post("/query/")
def query():
    query = bottle.request.POST.get("query", bottle.request.GET.get("query", ""))
    enquire = xapian.Enquire(db)
    enquire.set_query(make_query(query))
    count = 40
    matches = enquire.get_mset(0, count)
    estimated = matches.get_matches_estimated()
    total = db.get_doccount()
    out = StringIO()
    print >>out, '''
<html>
<head><title>Results</title></head>
<body>
<h1>Results for "<b>%s</b>"</h1>
''' % query
    if estimated == 0:
        print >>out, "No results found."
    else:
        # Give as results the first 30 documents; also use them as the key
        # ones to use to compute relevant terms
        rset = xapian.RSet()
        for m in enquire.get_mset(0, 30):
            rset.add_document(m.document.get_docid())
        # Compute the tag cloud
        class NonTagFilter(xapian.ExpandDecider):
            def __call__(self, term):
                return not term[0].isupper() and not term[0].isdigit()
        cloud = []
        maxscore = None
        for res in enquire.get_eset(40, rset, NonTagFilter()):
            # Normalise the score in the interval [0, 1]
            weight = math.log(res.weight)
            if maxscore == None: maxscore = weight
            tag = res.term
            cloud.append([tag, float(weight) / maxscore])
        max_weight = cloud[0][1]
        min_weight = cloud[-1][1]
        cloud.sort(key=lambda x:x[0])
        def mklink(query, term):
            return "/query?query=%s" % urllib.quote_plus(query + " and " + term)
        print >>out, "<h2>Tag cloud</h2>"
        print >>out, "<blockquote>"
        for term, weight in cloud:
            size = 100 + 100.0 * (weight - min_weight) / (max_weight - min_weight)
            print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(query, term), size, term)
        print >>out, "</blockquote>"
        print >>out, "<h2>Results</h2>"
        print >>out, "<p><a href='/'>Search again</a></p>"
        print >>out, "<p>%d results over %d documents, %.2f%%</p>" % (estimated, total, estimated * 100.0 / total)
        print >>out, "<p>%d/%d results</p>" % (matches.size(), estimated)
        print >>out, "<ul>"
        for m in matches:
            rec = data[m.document.get_data()]
            print >>out, "<li><a href='/item/%s'>%s</a></li>" % (rec["id"], rec["text"])
        print >>out, "</ul>"
        fd = open(QUERYLOG, "a")
        qinfo = dict(q=query, count=estimated)
        pickle.dump(qinfo, fd)
        fd.close()
    print >>out, '''
<a href="/">Search again</a>

</body>
</html>'''
    return out.getvalue()
@route("/item/:id")
@route("/item/:id/")
def show(id):
    rec = data[id]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Result %s</title></head>
<body>
<h1>Raw JSON record for twit %s</h1>
<pre>''' % (rec["id"], rec["id"])
    print >>out, simplejson.dumps(rec, indent=" ")
    print >>out, '''
</pre>
</body>
</html>'''
    return out.getvalue()
@route("/catinfo/:name")
@route("/catinfo/:name/")
def catinfo(name):
    prefix = prefixes[name]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Values for %s</title></head>
<body>
''' % name
    terms = [(x.term[len(prefix):], db.get_termfreq(x.term)) for x in db.allterms(prefix)]
    terms.sort(key=lambda x:x[1], reverse=True)
    freq_min = terms[0][1]
    freq_max = terms[-1][1]
    def mklink(name, term):
        return "/query?query=%s" % urllib.quote_plus(name + ":" + term)
    # Build tag cloud
    print >>out, "<h1>Tag cloud</h1>"
    print >>out, "<blockquote>"
    for term, freq in sorted(terms[:20], key=lambda x:x[0]):
        size = 100 + 100.0 * (freq - freq_min) / (freq_max - freq_min)
        print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(name, term), size, term)
    print >>out, "</blockquote>"
    print >>out, "<h1>All terms</h1>"
    print >>out, "<table>"
    print >>out, "<tr><th>Occurrences</th><th>Name</th></tr>"
    for term, freq in terms:
        print >>out, "<tr><td>%d</td><td><a href='/query?query=%s'>%s</a></td></tr>" % (freq, urllib.quote_plus(name + ":" + term), term)
    print >>out, "</table>"
    print >>out, '''
</body>
</html>'''
    return out.getvalue()
# Change here for bind host and port
bottle.run(host="0.0.0.0", port=8024)

...and then we presented our work and ended up winning the contest. This was the story of how we wrote this set of award winning code.

I love titles with plenty of buzz-words :) After a bit of google-ing I finally found the right combination of tools that fit my needs. Luckily they are all packaged for debian and well documented. Auto-configure your network is an old topic. There are plenty of articles and howtos. Moreover the gnome NetworkManager already solves many problems alone. But since I don't use NetworkManager I tried to find a solution that works well with wicd. The "right" combination for me so far is to use guessnet and ifplugd to handle my wired connection and use Wicd for the wireless network. In theory, you can go for a fully automated solution using guessnet and wpa_supplicant, but I find much better to hide all the complexity of wpa_supplicant and just use a simple - and unobtrusive - UI to handle my wireless. First thing is to convince Wicd to leave alone my wired connection. This is pretty easy since Wicd wont touch your wired interface if it finds an entry in /etc/network/interfaces for it. Moreover to selecting always switch to a wired connection when available will make Wicd to leave alone you routing tables even if there is a wireless connection available that is marked as automatically connect to this network. Second you need to install guessnet and ifplugd I've noticed that the default settings for ifplugd work pretty well except for the -q option that sometimes lives my interface configured even if I've been disconnected for quite a while. Using ifplugd default, I re-obtain an address every time the cable is plugged-in, maybe causing a small delay in some occasion, but I prefer waiting that de-configuring the interface by hand. Guessnet needs a tiny bit of work. The debian package has already nice defaults that you can copy directly in your /etc/network/interfaces. These will make sure you avoid long timeouts and you leave the interface configured even if no cable is connected. Now you have to add one entry per network that you want to configure: For example to use dhcp on your home network (and of course run any if-up / if-down script as you like). Or : to set a static address at work. Guessnet is nice and well integrated with ifupdown. Very flexible and documented. I had however the impression that from time to time the documentation I found on the net was not in sync with the latest release... I've also noticed that a component of guessnet is not in the package anymore. Guessnet-scan is a small program that will output a stanza to be used with ifupdown based on values sniffed on the wire. This is a kinda of extreme / hackish way to get connected but i reckon a nice tool to have. Hopefully it will make it back in the package sometimes soon... ps: first post on debian planet !

The bug webinterface of the Universal Debian Database currently knows about the following release critical bugs:

In Total:	732
Affecting Squeeze:	327
Squeeze only:	84
Remaining to be fixed in Squeeze:	243

Of these 243 bugs, the following tags are set:

Pending in Squeeze:	22
Patched in Squeeze:	51
Duplicates in Squeeze:	25
Can be fixed in a security Update:	26
Contrib or non-free in Squeeze:	12
Otherwise fixed in Squeeze:	26

Ignoring all the above (multiple tags possible), 130 bugs need to be fixed by Debian Contributors to get Debian 6.0 Squeeze released. However, with the view of the Release Managers, 183 need to be dealt with for the release to happen. Please see Interpreting the release critical bug statistics for an explanation of the different numbers.

I wrote the following as a foreword for the great German book Open Source Projektmanagement that Michael Prokop wrote. Unfortunately the foreword didn't manage it in time into the book because of various circumstances. As I'm not a person to throw away already written material, here is it for your reading pleasure. Maybe it makes you consider obtaining a copy of the German language book for yourself!

The Whole is more than the Sum of the Parts When I first met Mika at a Linux conference some years ago I quickly noticed that he had potential. The way he asked questions and tackled issues did impress me. We did meet again at various events, joined forces in various projects and knew that the result would be good. Back in the year 2004 the only real live CD was Knoppix. It always was too sluggish for me because I never really liked KDE and also OpenOffice.org was too bloated for me. I was working since a year at an Internet provider, felt comfortable in the shell and was missing the tools on Knoppix which I used daily. Often enough the frustration with the status quo starts the best projects. And so I had the idea spinning around to start a sysadmin live CD. There was just one problem here: Exactly at the same time Mika had already started such a project. And I knew one thing: it would had been a lost race to compete with him in creating the better live CD. So I joined his team and helped with the best of my knowledge. The name choice alone showed that he was the right person for the job: Grml an expression of the frustration that even he felt which spoke directly to the heart of so many. But it wasn't just the chosen name that showed that Mika was the right person for the job of the project leader. It's always the sum of the parts, no matter how minor they might look. He quickly managed a first release and in light of that also created an event which was fitting for the release of the first version: the OS04 an Open Source event which managed to get Jon Maddog Hall as keynote speaker. Mika managed through his welcoming way to lure more people into helping out. The team grew over the years, further regular releases increased the fan base, not only through the creative release names which are surely one of the parts that helped create the success. He always was open to suggestions for new tools that helped to extend the project. The lived openness did lead among other things to the case that other live CDs started to use the grml build system. JUXlala 2.0 (the system for preschool kids) is just one example. Why do I write primarily about Grml, one could get subliminal advertising probably cheaper? I do it because Grml is an extremely good example of the experience revealed in this book. And even though every project is different, it is still exactly the sum of the parts that leads to the success of a project. And exactly these parts are covered in this book and can be considered for the project at hand and get implemented accordingly. Good luck whether in large or small.
Rhonda

/debian permanent link Comments: 0

After having dealt with the removal of obsolete conffiles, I ll now explain what you should do when a configuration file managed by dpkg must be renamed. The problem Let s suppose that version 1.2 of the software stopped providing /etc/foo.conf. Instead it provides /etc/bar.conf because the configuration file got renamed. If you do nothing special, the new conffile will be installed with the default configuration, and the old one will stay around. Any customization made by the administrator are lost in the process (in fact they are not lost, they are still in foo.conf but they are unused). Of course, you could do mv /etc/foo.conf /etc/bar.conf in the pre-installation script. But that s not satisfactory: it will generate a spurious conffile prompt that the end-user will not understand. The solution In the preinst script, you have to verify if the old conffile has been modified by the administrator. If yes, you want to keep the file around. Otherwise you know you will be able to ditch it once the upgrade is over, and you rename it to /etc/foo.conf.dpkg-remove to remember this fact. In the postinst script, you remove /etc/foo.conf.dpkg-remove. If the old conffile (/etc/foo.conf) still exists, it s because it was modified by the administrator. You make a backup of the new conffile in /etc/bar.conf.dpkg-dist and rename the old one into /etc/bar.conf. In the postrm, when called to abort an upgrade, you move /etc/foo.conf.dpkg-remove back to its original name. In practice, use dpkg-maintscript-helper dpkg-maintscript-helper can automate all those tasks. You just have to put the following snippet in the maintainer scripts (postinst, postrm, preinst):

if dpkg-maintscript-helper supports mv_conffile 2>/dev/null; then
    dpkg-maintscript-helper mv_conffile /etc/foo.conf /etc/bar.conf 1.1-3 -- "$@"
fi

In this example, I assumed that version 1.1-3 was the last version of the package that contained /etc/foo.conf (i.e. the last version released before 1.2-1 was packaged). You can avoid the preliminary test if you pre-depend on dpkg (>= 1.15.7.2) or if enough time has passed to assume that everybody has a newer version anyway. You can learn all the details in dpkg-maintscript-helper s manual page. Found it useful? Be sure to not miss other packaging tips (or lessons), click here to subscribe to my free newsletter and get new articles by email (just check Send me blog updates ). No comment Support my work

Bruce Schneier wrote an insightful post about why designing products for wiretapping is a bad idea [1]. It seems that large parts of the Internet will be easy to tap (for both governments and criminals) in the near future unless something is done. The bad results of criminal use will outweigh any benefits of government use. Sam Watkins wrote an informative post about Android security [2]. Among other things any application can read all stored data including all photos, that s got to be a problem for anyone who photographs themself naked Rebecca Saxe gave an interesting TED talk about how brains make moral judgements [3]. Somehow she managed to speak about the Theory of Mind without mentioning Autism once. The Guardian has an amusing article by Cory Doctorow about security policies in banks [4]. He advocates promoting statistical literacy (or at least not promoting a lack of it) as a sound government policy. He also suggests allowing regulators to fine banks that get it wrong. Steven Johnson gave an interesting TED talk about Where Good Ideas Come From [5]. It s a bit slow at the start but gets good at the end. Adam Grosser gave an interesting TED talk about a fridge that was designed for use in Africa [6]. The core of the Absorption Refrigerator is designed to be heated in a pot of water in a cooking fire and it can then keep food cool for 12 hours. It s a pity that they couldn t design it to work on solar power to avoid the fuel use for the cooking fire. Josh Silver gave an interesting TED talk about liquid filled spectacles [7]. The glasses are shipped with a syringe filled with liquid at each side that is used to inflate the lenses to the desired refractive index. The wearer can just adjust the syringes until they get to the right magnification, as there are separate syringes the glasses work well with people who s eyes aren t identical (which is most people). Once the syringes are at the right spots the user can tighten some screws to prevent further transfer of liquid and cut the syringes off to give glasses that aren t overly heavy but which can t be adjusted any more, I guess that a natural extension to this would be to allow the syringes to be re-attached so that the user could adjust them every year to match declining vision. One thing that this wouldn t do is counter for Astigmatism (where the lens of the eye doesn t focus light to a point), but I guess they could make lenses to deal with a few common varieties of Astigmatism so that most people who have that problem can get a reasonable approximation. The current best effort is to make the glasses cost $19, which is 19 days salary for some of the poorest people in the world. Glasses in Australia cost up to $650 for a pair (or a more common cost of $200 or about $100 after Medicare) which would be about one day s salary. Eben Bayer gave an inspiring TED talk about one of the ways that mushrooms can save the planet [8]. He has designed molds that can be filled with Pasteurised organic waste (seed husks etc) and then seeded with fungal spores. The fungus then grows mycelium (thin fungal root fibers) through the organic waste making it into a solid structure which fits the shape of the mold. This is currently being used to replace poly-styrene foam for packaging and can apparently be used for making tiles that are fire retardant and sound proof for constructing buildings. The main benefits of the material are that it can be cheaply made without petrochemicals and that it is bio-degradable, I m not sure how the bio-degradable part would work with constructing buildings maybe they would just replace the panels every few years. Annie Lennox gave a TED talk about her Sing foundation to care for women and children who are affected by AIDS [9]. She describes the effect of AIDS in Africa as Genocide. Robert Ballard gave a very informative TED talk about exploring the oceans [10]. This was one of the most informative TED talks I ve seen and Robert is also one of the most enthusiastic speakers I ve seen, it s really worth watching! We really need more money spent on exploring the oceans. Jessa Gamble gave an interesting TED talk which suggests that the best thing to do is to go to bed at about sunset and then have a couple of hours of relaxing time during the middle of the night [11]. Apparently the subjects of body-block experiments who live for a month in a bunker without natural light or access to a clock get better sleep in this manner than they ever had in their life and feel fully awake for the first time. World Changing is a blog that has a lot of interesting articles about climate change and related issues [12]. It s worth a read. Cynthia Schneider gave an interesting TED talk about how reality competition TV is affecting reality [13]. Shows that are derived from the American Idol concept are driving a resurgence in some traditional forms of performance art while also promoting equality among other things it s apparent that winning is more important than misogyny. The Ritual of the Calling of an Engineer is an interesting concept [14]. I think it would be good to have something similar for Computer Science. Benjamin Mako Hill wrote an interesting and insightful essay about Piracy and Free Software [15].

• [1] http://www.schneier.com/blog/archives/2010/09/wiretapping_the.html
• [2] http://sswam.wordpress.com/2010/09/25/android-security-model-is-insufficient-and-it-needs-a-firewall/
• [3] http://www.ted.com/talks/rebecca_saxe_how_brains_make_moral_judgments.html
• [4] http://www.guardian.co.uk/technology/2010/sep/21/password-security-cory-doctorow
• [5] http://www.ted.com/talks/steven_johnson_where_good_ideas_come_from.html
• [6] http://www.ted.com/talks/lang/eng/adam_grosser_and_his_sustainable_fridge.html
• [7] http://www.ted.com/talks/lang/eng/josh_silver_demos_adjustable_liquid_filled_eyeglasses.html
• [8] http://www.ted.com/talks/download/video/9786/talk/971
• [9] http://www.ted.com/talks/annie_lennox_why_i_am_an_hiv_aids_activist.html
• [10] http://www.ted.com/talks/lang/eng/robert_ballard_on_exploring_the_oceans.html
• [11] http://www.ted.com/talks/jessa_gamble_how_to_sleep.html
• [12] http://www.worldchanging.com/
• [13] http://www.ted.com/talks/cynthia_schneider_the_surprising_spread_of_idol_tv.html
• [14] http://en.wikipedia.org/wiki/The_Ritual_of_the_Calling_of_an_Engineer
• [15] http://mako.cc/copyrighteous/20101011-00

How many bugs can be filed at once? 55 bugs in a row, so that the desired bug number is reached. It took some time to prepare all those reports, which eventually resulted in 55 successive bug numbers, on top of last-reported #599945 at the time, ta-da! I love it when a plan comes together! Just as a reminder, the main idea behind those bug reports is to get as many FTBFSes as possible reported, as previous mentioned. How many X packages can be uploaded between two dinstall runs? 14 packages. Now that things are settling down a bit on the squeeze front, I managed to have a look at X Server 1.9 (more precisely, 1.9.0.901 aka 1.9.1 rc1), which had already been worked on by Christopher James Halse Rogers (thanks!). Uploaded source packages include:

• xorg-server: the X server itself. Selective users want xserver-xorg-core and xserver-common from that one.
• xorg: a set of Debian-specific meta-packages. xserver-xorg is mostly what s wanted from that one.
• xserver-xorg-video-fbdev: a FrameBuffer-based video driver.
• xserver-xorg-video-vesa: the well-known generic video driver.
• All supported input drivers for which debian-x@ is listed in Maintainer. That means xserver-xorg-input- acecad, aiptek, elographics, evdev, joystick, mouse, penmount, synaptics, vmmouse, void.

So, what about the major (ati/radeon, intel, and nouveau) drivers? More on that later; if you want to test 1.9 with those, pick xserver-xorg-dev from experimental and build your driver against it. Hopefully only a few days should pass before you get an update on that topic. How many times did this blog go haywire? Possibly none, even though it was recently reworked so that http://ikibiki.org/ and http://blog.ikibiki.org/ are distinct. Some d-i or xorg tags may be added to older posts when time allows, but those shouldn t pop up in planet again.

How many bugs can be filed at once? 55 bugs in a row, so that the desired bug number is reached. It took some time to prepare all those reports, which eventually resulted in 55 successive bug numbers, on top of last-reported #599945 at the time, ta-da! I love it when a plan comes together! Just as a reminder, the main idea behind those bug reports is to get as many FTBFSes as possible reported, as previous mentioned. How many X packages can be uploaded between two dinstall runs? 14 packages. Now that things are settling down a bit on the squeeze front, I managed to have a look at X Server 1.9 (more precisely, 1.9.0.901 aka 1.9.1 rc1), which had already been worked on by Christopher James Halse Rogers (thanks!). Uploaded source packages include:

• xorg-server: the X server itself. Selective users want xserver-xorg-core and xserver-common from that one.
• xorg: a set of Debian-specific meta-packages. xserver-xorg is mostly what s wanted from that one.
• xserver-xorg-video-fbdev: a FrameBuffer-based video driver.
• xserver-xorg-video-vesa: the well-known generic video driver.
• All supported input drivers for which debian-x@ is listed in Maintainer. That means xserver-xorg-input- acecad, aiptek, elographics, evdev, joystick, mouse, penmount, synaptics, vmmouse, void.

So, what about the major (ati/radeon, intel, and nouveau) drivers? More on that later; if you want to test 1.9 with those, pick xserver-xorg-dev from experimental and build your driver against it. Hopefully only a few days should pass before you get an update on that topic. How many times did this blog go haywire? Possibly none, even though it was recently reworked so that http://ikibiki.org/ and http://blog.ikibiki.org/ are distinct. Some d-i or xorg tags may be added to older posts when time allows, but those shouldn t pop up in planet again.

As Roozbeh noted, Unicode 6.0 came out yesterday. Somewhat fittingly (given it having been National Coming Out Day), it adds
U+1F46C TWO MEN HOLDING HANDS and
U+1F46D TWO WOMEN HOLDING HANDS

Unfortunately most people will probably be looking at small boxes with numbers in there for a while yet, which is probably a wonderful metaphor but I'm not quite sure for what.

I love Unicode. I love that it lets me produce correct typography without having to resort to . I love that it allows for hilarity like Zalgo and years of entertainment in searching for the limits of the LCA conference badge rendering. I'm thrilled that a poorly managed introduction of IDNs let me get away with things like www. .net, and that nobody's got around to taking it away from me yet. And U+1F64C PERSON RAISING BOTH HANDS IN CELEBRATION is going to be a core part of my IRC character set the moment I get a compose map that includes compose+\+o+/. But I also love little things like having the typographical freedom to express whatever kind of relationship I choose to.

Well, providing that it's between two adults. Maybe 6.1 should add combining variants?

In the real world, David does not always win over Goliath. That's sad, but it's true. Particularly in the case of a big corporation using its wealth, influence, and minions to stomp down invividuals, to prevent them from reducing those all-important profits. A corporation must profit, by any means necessary. Profit is more important than people. A corporation has exactly as little ethics, morals, conscience, and good behavior as it can get away with. It all pretty much rests the officers, and their personal balance between greed and being good people. While most CEOs and managers and other decision makers in a corporation aren't going to do big evil, say, have people killed, they might not mind being a little bit nasty occasionally. If you have a large corporation, with a culture of profit-worshipping, you'll have a large number of people willing to be a little bit nasty, and that amounts to a lot of nastiness. That's all a long prelude to this: earlier this year, the PTFS corporation bought Liblime, a competitor. They did this to get a controlling interest on Koha, the free software librarary mangament system. Much to their surprise, you can't buy a healthy free software project like that. The community of users and developers of Koha has resisted PTFS's strongarm tactics. PTFS wants control of the code base, version control systems, mailing lists, domains, web pages, wikis, and decision making, but the community doesn't let them. Right now they are attacking Nicole Engard, the documentation manager for the Koha community, spreading lies about her. Nicole is one of the Koha community members who has spoken against PTFS's actions. She also works for a PTFS competitor, another sin against PTFS. This is an unfortunate situation for Nicole, for Koha, and for free software in general. If corporations like PTFS manage to silence their critics by nastiness, then nastiness is a working strategy and will be used more in the future. We need to squash this every time it happens, so it does not spread. However, for PTFS to lose, Nicole will have to have the courage to stand up against even more aggression. In the old world, before the Internet, this would have been quite hard. She would have had to get newspapers interested in her case. PTFS would almost certainly have been able to prevent that, by using their PR muscle to conduct a smear campaign against her. However, with the Internet, it's easier. Anyone can step up and help defend Nicole. That's what this is. I've never actually met Nicole, but in my brief involvement with the Koha community, she was always helpful and professional. She has, personally, helped Koha and free software spread farther than it would otherwise have. She does good. Like pretty much everyone in the Koha community, she is considerate, polite, and just to a fault, even when dealing with those attacking her. PTFS, on the other hand, has done only one honorable thing that I am aware of: they finally released their version of Koha. Every other action from PTFS seems to be an attempt to gain an upper hand by underhanded stratagems. Luckily, they seem to be failing. Let's see if giving more publicity to the attack on Nicole will make it, too, fail. See Nicole's post about this for more information.

Let s join gregora and zack (and others, too lazy to search for links/names) and fix some RC bugs in Debian!So what happened last week?

• passenger (# 599024): passenger-doc: Package is empty patch sent on 06.10.2010, lucas has uploaded it on the 07.10.2010 (with my name in Changed-By *g*)
• cpu (#598173): cpu: FTBFS: Unable to locate package cracklib2-dev NMU prepared and uploaded to DELAYED/10 on 06.10.2010 (10 because cpu has 2 important bugs I did not fix)
• update-manager-gnome (#598202): update-manager-gnome: will not start seems to be an issue with lsb_release, but I cannot reproduce it, asked submitter for more info
• ktoon (#599303): ktoon: KToon crashes with Signal 11 cannot reproduce either, asked submitter for more info

What does that mean? Two bugs are (almost) closed, two need more work. And this leads me to a feature I miss from http://udd.debian.org/bugs.cgi: a comment field as http://bts.turmzimmer.net/ has it. Will ping lucas about this later today.Oh, and on a side-note, I joined http://ask.debian.net let s help our users :)

Debian keyring maintainers did a batch of key updates in Debian keyring this week-end. As a consequence, I just uploaded samba 3.5.6 (which got release managers' approval for a freeze exceptions: thanks, guys) signed by this new key. It's now time to do a proper key transition (first step: learn how to do this appropriately by looking at others practices).

After having switched from Courier to Dovecot half a year ago I thought it was finally time to look into Sieve. For many years now I have maintained a procmail script to filter my incoming e-mail into numerous subfolders. Two features are important to me: For mailing lists, folders are created automatically based on the list names, with some adjustments for certain domains, such as putting all Debian lists into a folder of their own. And my procmail script was parsing the a dump of my evolution address book to put mail from people I know in a special folder. I did not know that at the first point is actually possible in Sieve but it is, using the variables extensions. And then the second was not hard to fix either, using this small and crude bash script, which creates a Sieve script recognizing these addresses, which is called from my main Sieve script:

#!/bin/bash
(
echo 'require ["fileinto"];'
echo 'if address ["From"] ['
/usr/lib/evolution/*/evolution-addressbook-export  
 perl -n -e 'print "\"$1\",\n" if /.*:(.*@[a-z0-9.]*)\s*$/'
 echo '"Justadummy"'
echo '  ]  '
echo ' fileinto "INBOX.Bekannte"; '
echo '  '
) > /tmp/bekannte
sieve-connect --user mail@joachim-breitner.de \
    --server 10.254.1.101 --port 2000 --notlsverify \
    --localsieve /tmp/bekannte --remotesieve bekannte  \
    --upload

I set up ManageSieve to be able to upload the scripts using sieve-connect. And finally I am able to do something that I have missed in all the years: Using the imap4flag extension, I can mark some mail automatically as read. This is useful for mail that is actually generated by me using program besides my e-mail client, such as bug reports sent by reportbug or patches submitted using darcs. I might even enable this feature for mailing list mails that I have sent.

After having switched from Courier to Dovecot half a year ago I thought it was finally time to look into Sieve. For many years now I have maintained a procmail script to filter my incoming e-mail into numerous subfolders. Two features are important to me: For mailing lists, folders are created automatically based on the list names, with some adjustments for certain domains, such as putting all Debian lists into a folder of their own. And my procmail script was parsing the a dump of my evolution address book to put mail from people I know in a special folder. I did not know that at the first point is actually possible in Sieve but it is, using the variables extensions. And then the second was not hard to fix either, using this small and crude bash script, which creates a Sieve script recognizing these addresses, which is called from my main Sieve script:

#!/bin/bash
(
echo 'require ["fileinto"];'
echo 'if address ["From"] ['
/usr/lib/evolution/*/evolution-addressbook-export  
 perl -n -e 'print "\"$1\",\n" if /.*:(.*@[a-z0-9.]*)\s*$/'
 echo '"Justadummy"'
echo '  ]  '
echo ' fileinto "INBOX.Bekannte"; '
echo '  '
) > /tmp/bekannte
sieve-connect --user mail@joachim-breitner.de \
    --server 10.254.1.101 --port 2000 --notlsverify \
    --localsieve /tmp/bekannte --remotesieve bekannte  \
    --upload

I set up ManageSieve to be able to upload the scripts using sieve-connect. And finally I am able to do something that I have missed in all the years: Using the imap4flag extension, I can mark some mail automatically as read. This is useful for mail that is actually generated by me using program besides my e-mail client, such as bug reports sent by reportbug or patches submitted using darcs. I might even enable this feature for mailing list mails that I have sent.

#593401