The following contributors got their Debian Developer accounts in the last two months:
Just as I did for 2020, I won't publically disclose exactly how many books I read in 2021, but they evidently provoked enough thoughts that felt it worth splitting my yearly writeup into separate posts. I will reveal, however, that I got through more books than the previous year, and, like before, I enjoyed the books I read this year even more in comparison as well.
How much of this is due to refining my own preferences over time, and how much can be ascribed to feeling less pressure to read particular books? It s impossible to say, and the question is complicated further by the fact I found many of the classics I read well worth of their entry into the dreaded canon.
But enough of the throat-clearing. In today's post I'll be looking at my favourite books filed under memoir and biography, in no particular order.
Books that just missed the cut here include: Bernard Crick's celebrated 1980 biography of George Orwell, if nothing else because it was a pleasure to read; Hilary Mantel's exhilaratingly bitter early memoir, Giving up the Ghost (2003); and Patricia Lockwood's hilarious Priestdaddy (2017). I also had a soft spot for Tim Kreider's We Learn Nothing (2012) as well, despite not knowing anything about the author in advance, likely a sign of good writing. The strangest book in this category I read was definitely Michelle Zauner's Crying in H Mart. Based on a highly-recommended 2018 essay in the New Yorker, its rich broth of genuine yearning for a departed mother made my eyebrows raise numerous times when I encountered inadvertent extra details about Zauner's relationships.
Beethoven: A Life in Nine Pieces (2020) Laura Tunbridge Whilst it might immediately present itself as a clickbait conceit, organising an overarching narrative around just nine compositions by Beethoven turns out to be an elegant way of saying something fresh about this grizzled old bear. Some of Beethoven's most famous compositions are naturally included in the nine (eg. the Eroica and the Hammerklavier piano sonata), but the book raises itself above conventional Beethoven fare when it highlights, for instance, his Septet, Op. 20, an early work that is virtually nobody's favourite Beethoven piece today. The insight here is that it was widely popular in its time, played again and again around Vienna for the rest of his life. No doubt many contemporary authors can relate to this inability to escape being artistically haunted by an earlier runaway success. The easiest way to say something interesting about Beethoven in the twenty-first century is to talk about the myth of Beethoven instead. Or, as Tunbridge implies, perhaps that should really be 'Beethoven' in leaden quotation marks, given so much about what we think we know about the man is a quasi-fictional construction. Take Anton Schindler, Beethoven's first biographer and occasional amanuensis, who destroyed and fabricated details about Beethoven's life, casting himself in a favourable light and exaggerating his influence with the composer. Only a few decades later, the idea of a 'heroic' German was to be politically useful as well; the Anglosphere often need reminding that Germany did not exist as a nation-state prior to 1871, so it should be unsurprising to us that the late nineteenth-century saw a determined attempt to create a uniquely 'German' culture ex nihilo. (And the less we say about Immortal Beloved the better, even though I treasure that film.) Nevertheless, Tunbridge cuts through Beethoven's substantial legacy using surgical precision that not only avoids feeling like it is settling a score, but it also does so in a way that is unlikely to completely alienate anyone emotionally dedicated to some already-established idea of the man to bring forth the tediously predictable sentiment that Beethoven has 'gone woke'. With Alex Ross on the cult of Wagner, it seems that books about the 'myth of X' are somewhat in vogue right now. And this pattern within classical music might fit into some broader trend of deconstruction in popular non-fiction too, especially when we consider the numerous contemporary books on the long hangover of the Civil Rights era (Robin DiAngelo's White Fragility, etc.), the multifarious ghosts of Empire (Akala's Natives, Sathnam Sanghera's Empireland, etc.) or even the 'transmogrification' of George Orwell into myth. But regardless of its place in some wider canon, A Life in Nine Pieces is beautifully printed in hardback form (worth acquiring for that very reason alone), and it is one of the rare good books about classical music that can be recommended to both the connoisseur and the layperson alike.
Sea State (2021) Tabitha Lasley In her mid-30s and jerking herself out of a terrible relationship, Tabitha Lasley left London and put all her savings into a six-month lease on a flat within a questionable neighbourhood in Aberdeen, Scotland. She left to make good on a lukewarm idea for a book about oil rigs and the kinds of men who work on them: I wanted to see what men were like with no women around, she claims. The result is Sea State, a forthright examination of the life of North Sea oil riggers, and an unsparing portrayal of loneliness, masculinity, female desire and the decline of industry in Britain. (It might almost be said that Sea State is an update of a sort to George Orwell's visit to the mines in the North of England.) As bracing as the North Sea air, Sea State spoke to me on multiple levels but I found it additionally interesting to compare and contrast with Julian Barnes' The Man with Red Coat (see below). Women writers are rarely thought to be using fiction for higher purposes: it is assumed that, unlike men, whatever women commit to paper is confessional without any hint of artfulness. Indeed, it seems to me that the reaction against the decades-old genre of autofiction only really took hold when it became the domain of millennial women. (By contrast, as a 75-year-old male writer with a firmly established reputation in the literary establishment, Julian Barnes is allowed wide latitude in what he does with his sources and his writing can be imbued with supremely confident airs as a result.) Furthermore, women are rarely allowed metaphor or exaggeration for dramatic effect, and they certainly aren t permitted to emphasise darker parts in order to explore them... hence some of the transgressive gratification of reading Sea State. Sea State is admittedly not a work of autofiction, but the sense that you are reading about an author writing a book is pleasantly unavoidable throughout. It frequently returns to the topic of oil workers who live multiple lives, and Lasley admits to living two lives herself: she may be in love but she's also on assignment, and a lot of the pleasure in this candid and remarkably accessible book lies in the way these states become slowly inseparable.
Twilight of Democracy (2020) Anne Applebaum For the uninitiated, Anne Applebaum is a staff writer for The Atlantic magazine who won a Pulitzer-prize for her 2004 book on the Soviet Gulag system. Her latest book, however, Twilight of Democracy is part memoir and part political analysis and discusses the democratic decline and the rise of right-wing populism. This, according to Applebaum, displays distinctly authoritarian tendencies, and who am I to disagree? Applebaum does this through three main case studies (Poland, the United Kingdom and the United States), but the book also touches on Hungary as well. The strongest feature of this engaging book is that Appelbaum's analysis focuses on the intellectual classes and how they provide significant justification for a descent into authoritarianism. This is always an important point to be remembered, especially as much of the folk understanding of the rise of authoritarian regimes tends to place exaggerated responsibility on the ordinary and everyday citizen: the blame placed on the working-class in the Weimar Republic or the scorn heaped upon 'white trash' of the contemporary Rust Belt, for example. Applebaum is uniquely poised to discuss these intellectuals because, well, she actually knows a lot of them personally. Or at least, she used to know them. Indeed, the narrative of the book revolves around two parties she hosted, both in the same house in northwest Poland. The first party, on 31 December 1999, was attended by friends from around the Western world, but most of the guests were Poles from the broad anti-communist alliance. They all agreed about democracy, the rule of law and the route to prosperity whilst toasting in the new millennium. (I found it amusing to realise that War and Peace also starts with a party.) But nearly two decades later, many of the attendees have ended up as supporters of the problematic 'Law and Justice' party which currently governs the country. Applebaum would now cross the road to avoid them, and they would do the same to her, let alone behave themselves at a cordial reception. The result of this autobiographical detail is that by personalising the argument, Applebaum avoids the trap of making too much of high-minded abstract argument for 'democracy', and additionally makes her book compellingly spicy too. Yet the strongest part of this book is also its weakest. By individualising the argument, it often feels that Applebaum is settling a number of personal scores. She might be very well justified in doing this, but at times it feels like the reader has walked in halfway through some personal argument and is being asked to judge who is in the right. Furthermore, Applebaum's account of contemporary British politics sometimes deviates into the cartoonish: nothing was egregiously incorrect in any of her summations, but her explanation of the Brexit referendum result didn't read as completely sound. Nevertheless, this lively and entertaining book that can be read with profit, even if you disagree with significant portions of it, and its highly-personal approach makes it a refreshing change from similar contemporary political analysis (eg. David Runciman's How Democracy Ends) which reaches for that more 'objective' line.
The Man in the Red Coat (2019) Julian Barnes As rich as the eponymous red coat that adorns his cover, Julian Barnes quasi-biography of French gynaecologist Samuel-Jean Pozzi (1846 1918) is at once illuminating, perplexing and downright hilarious. Yet even that short description is rather misleading, for this book evades classification all manner number of ways. For instance, it is unclear that, with the biographer's narrative voice so obviously manifest, it is even a biography in the useful sense of the word. After all, doesn't the implied pact between author and reader require the biographer to at least pretend that they are hiding from the reader? Perhaps this is just what happens when an author of very fine fiction turns his hand to non-fiction history, and, if so, it represents a deeper incursion into enemy territory after his 1984 metafictional Flaubert's Parrot. Indeed, upon encountering an intriguing mystery in Pozzi's life crying out for a solution, Barnes baldly turns to the reader, winks and states: These matters could, of course, be solved in a novel. Well, quite. Perhaps Barnes' broader point is that, given that's impossible for the author to completely melt into air, why not simply put down your cards and have a bit of fun whilst you're at it? If there's any biography that makes the case for a rambling and lightly polemical treatment, then it is this one. Speaking of having fun, however, two qualities you do not expect in a typical biography is simply how witty they can be, as well as it having something of the whiff of the thriller about it. A bullet might be mentioned in an early chapter, but given the name and history of Monsieur Pozzi is not widely known, one is unlikely to learn how he lived his final years until the closing chapters. (Or what happened to that turtle.) Humour is primarily incorporated into the book in two main ways: first, by explicitly citing the various wits of the day ( What is a vice? Merely a taste you don t share. etc.), but perhaps more powerful is the gentle ironies, bon mots and observations in Barnes' entirely unflappable prose style, along with the satire implicit in him writing this moreish pseudo-biography to begin with. The opening page, with its steadfast refusal to even choose where to begin, is somewhat characteristic of Barnes' method, so if you don't enjoy the first few pages then you are unlikely to like the rest. (Indeed, the whole enterprise may be something of an acquired taste. Like Campari.) For me, though, I was left wryly grinning and often couldn't wait to turn the page. Indeed, at times it reminded me of a being at a dinner party with an extremely charming guest at the very peak of his form as a wit and raconteur, delighting the party with his rambling yet well-informed discursive on his topic de jour. A significant book, and a book of significance.
| Publisher: | Tor |
| Copyright: | May 2019 |
| ISBN: | 1-250-19551-9 |
| Format: | Kindle |
| Pages: | 528 |
The thought has barely formed when a different shape looms over him, grinning widely enough to show every tooth in its head. They are even, white, and perfect, and yet he somehow can't stop himself from thinking there's something wrong with them, that they're mismatched, that this assortment of teeth was never meant to share a single jaw, a single terrible smile.This isn't effective. This is telling the reader how they're supposed to feel about the thing you're describing, without doing the work of writing a description that makes them feel that way. (Also, you may see what I mean by overwrought.) That leads me to my next complaint: the villains. My problem is not so much with Leigh, who I thought was an adequate monster, if a bit single-note. There's some thought and depth behind her arguments with Reed, a few hints of her own motives that were more convincing for not being fully shown. The descriptions of how dangerous she is were reasonably effective. She's a good villain for this type of dark fantasy story where the world is dangerous and full of terrors (and reminded me of some of the villains from McGuire's October Daye series). Reed, though, is a storytelling train wreck. The Big Bad of the novel is the least interesting character in it. He is a stuffed tailcoat full of malicious incompetence who is only dangerous because the author proclaims him to be. It only adds insult to injury that he kills off a far more nuanced and creative villain before the novel starts, replacing her ambiguous goals with Snidely Whiplash mustache-twirling. The reader has to suffer through extended scenes focused on him as he brags, monologues, and obsesses over his eventual victory without an ounce of nuance or subtlety. Worse is the dynamic between him and Leigh, which is only one symptom of the problem with Middlegame that made me the most angry: the degree to this book oozes patriarchy. Every man in this book, including the supposed hero, orders around the women, who are forced in various ways to obey. This is the most obvious between Leigh and Reed, but it's the most toxic, if generally more subtle, between Roger and Dodger. Dodger is great. I had absolutely no trouble identifying with and rooting for her as a character. The nasty things that McGuire does to her over the course of the book (and wow does that never let up) made me like her more when she tenaciously refuses to give up. Dodger is the math component of the Doctrine of Ethos, and early in the book I thought McGuire handled that well, particularly given how difficult it is to write a preternatural genius. Towards the end of this book, her math sadly turns into a very non-mathematical magic (more on this in a moment), but her character holds all the way through. It felt like she carved her personality out of this story through sheer force of will and clung to it despite the plot. I wanted to rescue her from this novel and put her into a better book, such as the one in which her college friends (who are great; McGuire is very good at female friendships when she writes them) stage an intervention, kick a few people out of her life, and convince her to trust them. Unfortunately, Dodger is, by authorial fiat, half of a bound pair, and the other half of that pair is Roger, who is the sort of nice guy everyone likes and thinks is sweet and charming until he turns into an emotional trap door right when you need him the most and dumps you into the ocean to drown. And then somehow makes you do all the work of helping him feel better about his betrayal. The most egregious (and most patriarchal) thing Roger does in this book is late in the book and a fairly substantial spoiler, so I can't rant about that properly. But even before that, Roger keeps doing the the same damn emotional abandonment trick, and the book is heavily invested into justifying it and making excuses for him. Excuses that, I should note, are not made for Dodger; her failings are due to her mistakes and weaknesses, whereas Roger's are natural reactions to outside forces. I got very, very tired of this, and I'm upset by how little awareness the narrative voice showed for how dysfunctional and abusive this relationship is. The solution is always for Dodger to reunite with Roger; it's built into the structure of the story. I have a weakness for the soul-bound pair, in part from reading a lot of Mercedes Lackey at an impressionable age, but one of the dangerous pitfalls of the concept is that the characters then have to have an almost flawless relationship. If not, it can turn abusive very quickly, since the characters by definition cannot leave each other. It's essentially coercive, so as soon as the relationship shows a dark side, the author needs to be extremely careful. McGuire was not. There is an attempted partial patch, late in the book, for the patriarchal structure. One of the characters complains about it, and another says that the gender of the language and math pairs is random and went either way in other pairs. Given that both of the pairs that we meet in this story have the same male-dominant gender dynamic, what I took from this is that McGuire realized there was a problem but wasn't able to fix it. (I'm also reminded of David R. Henry's old line that it's never a good sign when the characters start complaining about the plot.) The structural problems are all the more frustrating because I think there were ways out of them. Roger is supposedly the embodiment of language, not that you'd be able to tell from most scenes in this novel. For reasons that I do not understand, McGuire expressed that as a love of words: lexicography, translation, and synonyms. This makes no sense to me. Those are some of the more structured and rules-based (and hence mathematical) parts of language. If Roger had instead been focused on stories collecting them, telling them, and understanding why and how they're told he would have had a clearer contrast with Dodger. More importantly, it would have solved the plot problem that McGuire solved with a nasty bit of patriarchy. So much could have been done with Dodger building a structure of math around Roger's story-based expansion of the possible, and it would have grounded Dodger's mathematics in something more interesting than symbolic magic. To me, it's such an obvious lost opportunity. I'm still upset about this book. McGuire does a lovely bit of world-building with Asphodel Baker, what little we see of her. I found the hidden alchemical war against her work by L. Frank Baum delightful, and enjoyed every excerpt from the fictional Over the Woodward Wall scattered throughout Middlegame. But a problem with inventing a fictional book to excerpt in a real novel is that the reader may decide that the fictional book sounds a lot better than the book they're reading, and start wishing they could just read that book instead. That was certainly the case for me. I'm sad that Over the Woodward Wall doesn't exist, and am mostly infuriated by Middlegame. Dodger and Erin deserved to live in a better book. Should you want to read this anyway (and I do know people who liked it), serious content warning for self-harm. Rating: 4 out of 10
| Series: | Species Imperative #2 |
| Publisher: | DAW |
| Copyright: | 2005 |
| ISBN: | 0-7564-0260-3 |
| Format: | Hardcover |
| Pages: | 453 |
| Series: | Imperial Radch #2 |
| Publisher: | Orbit |
| Copyright: | October 2014 |
| ISBN: | 0-316-24665-4 |
| Format: | Trade paperback |
| Pages: | 354 |
(Photo credit: Obey Arthur Liu; originally on Picasa, license.)
Three weekends ago, I participated in a Debian bug squashing party. It was more fun than I had guessed!
The event worked: we squashed bugs. Geoffrey Thomas (geofft) organized it as an event for MIT's student computing group, SIPB. In this post, I'll review the good parts and the bad. I'll conclude with beaming photos of my two mentees and talk about the bugs they fixed.
So, the good:
Jessica McKellar is a software engineer at 
Noah Swartz is a recent graduate of Case Western Reserve University where he studied Mathematics and played Magic. He is an intern at the MIT Media Lab where he contributes to DoppelLab in Joe Paradiso's Responsive Environments group. This was definitely his first direct contribution to Debian. It was also one of the most intense command-line experiences he has had so far. Noah wasn't originally planning to come, but we were having lunch together before the hackathon, and I convinced him to join us.
Noah fixed #625177, a fails-to-build-from-source (FTBFS) bug in nslint. The problem was that "-Wl" was instead written in all lowercase in the debian/rules file, as "-wl". Noah fixed that, making sure the package properly built in pbuilder, and then spent some quality time with lintian figuring out the right way to write a debian/changelog.
That's a wrap! We'll hopefully have one again in a few months, and before that, I hope to write up a guide so that we run things even more smoothly next time.
Jo bought me a voucher last Christmas to spend some time on a race
track, driving an Aston Martin and a Lamborghini. Finally we got
around to using it last week. Jo is just as much a petrol-head as me,
so I surprised her by buying the same experience voucher for her -
call it a wedding present. Good call - she didn't stop grinning all
day, even on the long drive home!
The Aston Martin was a DB9 Volante (i.e. the convertible). Lovely
car, but a little softer than I'd expected (maybe due to it being the
convertible). I preferred the Lamborghini Gallardo, pictured here in a
nice subtle colour. *grin*
The nice folks doing the official photos at the track also took
some extra photos of us sitting in another supercar - an Audi R8 V10
convertible. We've got those photos coming on CD in the post, and I'll
post one when they land.
#!/usr/bin/python # Merge the JSON data # (C) 2010 Enrico Zini <enrico@enricozini.org> # License: WTFPL version 2 (http://sam.zoy.org/wtfpl/) import simplejson import os res = [] for f in os.listdir("."): if not f.startswith("gmp24"): continue data = open(f).read().strip() if data == "[]": continue parsed = simplejson.loads(data) res.extend(parsed) print simplejson.dumps(res)The results however were not ordered by date, as GMP had to use several accounts to twit because Twitter was putting Greather Manchester Police into jail for generating too much traffic. There would be quite a bit to write about that, but let's stick to our work. Here is code to sort the JSON data by time:
#!/usr/bin/python # Sort the JSON data # (C) 2010 Enrico Zini <enrico@enricozini.org> # License: WTFPL version 2 (http://sam.zoy.org/wtfpl/) import simplejson import sys import datetime as dt all_recs = simplejson.load(sys.stdin) all_recs.sort(key=lambda x: dt.datetime.strptime(x["created_at"], "%a %b %d %H:%M:%S +0000 %Y")) simplejson.dump(all_recs, sys.stdout)I then wanted to play with Tf-idf for extracting the most important words of every tweet:
#!/usr/bin/python # tfifd - Annotate JSON elements with Tf-idf extracted keywords # # Copyright (C) 2010 Enrico Zini <enrico@enricozini.org> # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. import sys, math import simplejson import re # Read all the twits records = simplejson.load(sys.stdin) # All the twits by ID byid = dict(((x["id"], x) for x in records)) # Stopwords we ignore stopwords = set(["by", "it", "and", "of", "in", "a", "to"]) # Tokenising engine re_num = re.compile(r"^\d+$") re_word = re.compile(r"(\w+)") def tokenise(tweet): "Extract tokens from a tweet" for tok in tweet["text"].split(): tok = tok.strip().lower() if re_num.match(tok): continue mo = re_word.match(tok) if not mo: continue if mo.group(1) in stopwords: continue yield mo.group(1) # Extract tokens from tweets tokenised = dict(((x["id"], list(tokenise(x))) for x in records)) # Aggregate token counts aggregated = for d in byid.iterkeys(): for t in tokenised[d]: if t in aggregated: aggregated[t] += 1 else: aggregated[t] = 1 def tfidf(doc, tok): "Compute TFIDF score of a token in a document" return doc.count(tok) * math.log(float(len(byid)) / aggregated[tok]) # Annotate tweets with keywords res = [] for name, tweet in byid.iteritems(): doc = tokenised[name] keywords = sorted(set(doc), key=lambda tok: tfidf(doc, tok), reverse=True)[:5] tweet["keywords"] = keywords res.append(tweet) simplejson.dump(res, sys.stdout)I thought this was producing a nice summary of every tweet but nobody was particularly interested, so we moved on to adding categories to tweet. Thanks to Yuwei who put together some useful keyword sets, we managed to annotate each tweet with a place name (i.e. "Stockport"), a social place name (i.e. "pub", "bank") and a social category (i.e. "man", "woman", "landlord"...) The code is simple; the biggest work in it was the dictionary of keywords:
#!/usr/bin/python # categorise - Annotate JSON elements with categories # # Copyright (C) 2010 Enrico Zini <enrico@enricozini.org> # Copyright (C) 2010 Yuwei Lin <yuwei@ylin.org> # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. import sys, math import simplejson import re # Electoral wards from http://en.wikipedia.org/wiki/List_of_electoral_wards_in_Greater_Manchester placenames = ["Altrincham", "Sale West", "Altrincham", "Ashton upon Mersey", "Bowdon", "Broadheath", "Hale Barns", "Hale Central", "St Mary", "Timperley", "Village", "Ashton-under-Lyne", "Ashton Hurst", "Ashton St Michael", "Ashton Waterloo", "Droylsden East", "Droylsden West", "Failsworth East", "Failsworth West", "St Peter", "Blackley", "Broughton", "Broughton", "Charlestown", "Cheetham", "Crumpsall", "Harpurhey", "Higher Blackley", "Kersal", "Bolton North East", "Astley Bridge", "Bradshaw", "Breightmet", "Bromley Cross", "Crompton", "Halliwell", "Tonge with the Haulgh", "Bolton South East", "Farnworth", "Great Lever", "Harper Green", "Hulton", "Kearsley", "Little Lever", "Darcy Lever", "Rumworth", "Bolton West", "Atherton", "Heaton", "Lostock", "Horwich", "Blackrod", "Horwich North East", "Smithills", "Westhoughton North", "Chew Moor", "Westhoughton South", "Bury North", "Church", "East", "Elton", "Moorside", "North Manor", "Ramsbottom", "Redvales", "Tottington", "Bury South", "Besses", "Holyrood", "Pilkington Park", "Radcliffe East", "Radcliffe North", "Radcliffe West", "St Mary", "Sedgley", "Unsworth", "Cheadle", "Bramhall North", "Bramhall South", "Cheadle", "Gatley", "Cheadle Hulme North", "Cheadle Hulme South", "Heald Green", "Stepping Hill", "Denton", "Reddish", "Audenshaw", "Denton North East", "Denton South", "Denton West", "Dukinfield", "Reddish North", "Reddish South", "Hazel Grove", "Bredbury", "Woodley", "Bredbury Green", "Romiley", "Hazel Grove", "Marple North", "Marple South", "Offerton", "Heywood", "Middleton", "Bamford", "Castleton", "East Middleton", "Hopwood Hall", "Norden", "North Heywood", "North Middleton", "South Middleton", "West Heywood", "West Middleton", "Leigh", "Astley Mosley Common", "Atherleigh", "Golborne", "Lowton West", "Leigh East", "Leigh South", "Leigh West", "Lowton East", "Tyldesley", "Makerfield", "Abram", "Ashton", "Bryn", "Hindley", "Hindley Green", "Orrell", "Winstanley", "Worsley Mesnes", "Manchester Central", "Ancoats", "Clayton", "Ardwick", "Bradford", "City Centre", "Hulme", "Miles Platting", "Newton Heath", "Moss Side", "Moston", "Manchester", "Gorton", "Fallowfield", "Gorton North", "Gorton South", "Levenshulme", "Longsight", "Rusholme", "Whalley Range", "Manchester", "Withington", "Burnage", "Chorlton", "Chorlton Park", "Didsbury East", "Didsbury West", "Old Moat", "Withington", "Oldham East", "Saddleworth", "Alexandra", "Crompton", "Saddleworth North", "Saddleworth South", "Saddleworth West", "Lees", "St James", "St Mary", "Shaw", "Waterhead", "Oldham West", "Royton", "Chadderton Central", "Chadderton North", "Chadderton South", "Coldhurst", "Hollinwood", "Medlock Vale", "Royton North", "Royton South", "Werneth", "Rochdale", "Balderstone", "Kirkholt", "Central Rochdale", "Healey", "Kingsway", "Littleborough Lakeside", "Milkstone", "Deeplish", "Milnrow", "Newhey", "Smallbridge", "Firgrove", "Spotland", "Falinge", "Wardle", "West Littleborough", "Salford", "Eccles", "Claremont", "Eccles", "Irwell Riverside", "Langworthy", "Ordsall", "Pendlebury", "Swinton North", "Swinton South", "Weaste", "Seedley", "Stalybridge", "Hyde", "Dukinfield Stalybridge", "Hyde Godley", "Hyde Newton", "Hyde Werneth", "Longdendale", "Mossley", "Stalybridge North", "Stalybridge South", "Stockport", "Brinnington", "Central", "Davenport", "Cale Green", "Edgeley", "Cheadle Heath", "Heatons North", "Heatons South", "Manor", "Stretford", "Urmston", "Bucklow-St Martins", "Clifford", "Davyhulme East", "Davyhulme West", "Flixton", "Gorse Hill", "Longford", "Stretford", "Urmston", "Wigan", "Aspull New Springs Whelley", "Douglas", "Ince", "Pemberton", "Shevington with Lower Ground", "Standish with Langtree", "Wigan Central", "Wigan West", "Worsley", "Eccles South", "Barton", "Boothstown", "Ellenbrook", "Cadishead", "Irlam", "Little Hulton", "Walkden North", "Walkden South", "Winton", "Worsley", "Wythenshawe", "Sale East", "Baguley", "Brooklands", "Northenden", "Priory", "Sale Moor", "Sharston", "Woodhouse Park"] # Manual coding from Yuwei placenames.extend(["City centre", "Tameside", "Oldham", "Bury", "Bolton", "Trafford", "Pendleton", "New Moston", "Denton", "Eccles", "Leigh", "Benchill", "Prestwich", "Sale", "Kearsley", ]) placenames.extend(["Trafford", "Bolton", "Stockport", "Levenshulme", "Gorton", "Tameside", "Blackley", "City centre", "Airport", "South Manchester", "Rochdale", "Chorlton", "Uppermill", "Castleton", "Stalybridge", "Ashton", "Chadderton", "Bury", "Ancoats", "Whalley Range", "West Yorkshire", "Fallowfield", "New Moston", "Denton", "Stretford", "Eccles", "Pendleton", "Leigh", "Altrincham", "Sale", "Prestwich", "Kearsley", "Hulme", "Withington", "Moss Side", "Milnrow", "outskirt of Manchester City Centre", "Newton Heath", "Wythenshawe", "Mancunian Way", "M60", "A6", "Droylesden", "M56", "Timperley", "Higher Ince", "Clayton", "Higher Blackley", "Lowton", "Droylsden", "Partington", "Cheetham Hill", "Benchill", "Longsight", "Didsbury", "Westhoughton"]) # Social categories from Yuwei soccat = ["man", "woman", "men", "women", "youth", "teenager", "elderly", "patient", "taxi driver", "neighbour", "male", "tenant", "landlord", "child", "children", "immigrant", "female", "workmen", "boy", "girl", "foster parents", "next of kin"] for i in range(100): soccat.append("%d-year-old" % i) soccat.append("%d-years-old" % i) # Types of social locations from Yuwei socloc = ["car park", "park", "pub", "club", "shop", "premises", "bus stop", "property", "credit card", "supermarket", "garden", "phone box", "theatre", "toilet", "building site", "Crown court", "hard shoulder", "telephone kiosk", "hotel", "restaurant", "cafe", "petrol station", "bank", "school", "university"] extras = "placename": placenames, "soccat": soccat, "socloc": socloc # Normalise keyword lists for k, v in extras.iteritems(): # Remove duplicates v = list(set(v)) # Sort by length v.sort(key=lambda x:len(x), reverse=True) # Add keywords def add_categories(tweet): text = tweet["text"].lower() for field, categories in extras.iteritems(): for cat in categories: if cat.lower() in text: tweet[field] = cat break return tweet # Read all the twits records = (add_categories(x) for x in simplejson.load(sys.stdin)) simplejson.dump(list(records), sys.stdout)All these scripts form a nice processing chain: each script takes a list of JSON records, adds some bit and passes it on. In order to see what we have so far, here is a simple script to convert the JSON twits to CSV so they can be viewed in a spreadsheet:
#!/usr/bin/python # Convert the JSON twits to CSV # (C) 2010 Enrico Zini <enrico@enricozini.org> # License: WTFPL version 2 (http://sam.zoy.org/wtfpl/) import simplejson import sys import csv rows = ["id", "created_at", "text", "keywords", "placename"] writer = csv.writer(sys.stdout) for rec in simplejson.load(sys.stdin): rec["keywords"] = " ".join(rec["keywords"]) rec["placename"] = rec.get("placename", "") writer.writerow([rec[row] for row in rows])At this point we were coming up with lots of questions: "were there more reports on women or men?", "which place had most incidents?", "what were the incidents involving animals?"... Time to bring Xapian into play. This script reads all the JSON tweets and builds a Xapian index with them:
#!/usr/bin/python # toxapian - Index JSON tweets in Xapian # # Copyright (C) 2010 Enrico Zini <enrico@enricozini.org> # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. import simplejson import sys import os, os.path import xapian DBNAME = sys.argv[1] db = xapian.WritableDatabase(DBNAME, xapian.DB_CREATE_OR_OPEN) stemmer = xapian.Stem("english") indexer = xapian.TermGenerator() indexer.set_stemmer(stemmer) indexer.set_database(db) data = simplejson.load(sys.stdin) for rec in data: doc = xapian.Document() doc.set_data(str(rec["id"])) indexer.set_document(doc) indexer.index_text_without_positions(rec["text"]) # Index categories as categories if "placename" in rec: doc.add_boolean_term("XP" + rec["placename"].lower()) if "soccat" in rec: doc.add_boolean_term("XS" + rec["soccat"].lower()) if "socloc" in rec: doc.add_boolean_term("XL" + rec["socloc"].lower()) db.add_document(doc) db.flush() # Also save the whole dataset so we know where to find it later if we want to # show the details of an entry simplejson.dump(data, open(os.path.join(DBNAME, "all.json"), "w"))And this is a simple command line tool to query to the database:
#!/usr/bin/python # xgrep - Command line tool to query the GMP24 tweet Xapian database # # Copyright (C) 2010 Enrico Zini <enrico@enricozini.org> # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. import simplejson import sys import os, os.path import xapian DBNAME = sys.argv[1] db = xapian.Database(DBNAME) stem = xapian.Stem("english") qp = xapian.QueryParser() qp.set_default_op(xapian.Query.OP_AND) qp.set_database(db) qp.set_stemmer(stem) qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME) qp.add_boolean_prefix("place", "XP") qp.add_boolean_prefix("soc", "XS") qp.add_boolean_prefix("loc", "XL") query = qp.parse_query(sys.argv[2], xapian.QueryParser.FLAG_BOOLEAN xapian.QueryParser.FLAG_LOVEHATE xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE xapian.QueryParser.FLAG_WILDCARD xapian.QueryParser.FLAG_PURE_NOT xapian.QueryParser.FLAG_SPELLING_CORRECTION xapian.QueryParser.FLAG_AUTO_SYNONYMS) enquire = xapian.Enquire(db) enquire.set_query(query) count = 40 matches = enquire.get_mset(0, count) estimated = matches.get_matches_estimated() print "%d/%d results" % (matches.size(), estimated) data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json")))) for m in matches: rec = data[m.document.get_data()] print rec["text"] print "%d/%d results" % (matches.size(), matches.get_matches_estimated()) total = db.get_doccount() estimated = matches.get_matches_estimated() print "%d results over %d documents, %d%%" % (estimated, total, estimated * 100 / total)Neat! Now that we have a proper index that supports all sort of cool things, like stemming, tag clouds, full text search with complex queries, lookup of similar documents, suggest keywords and so on, it was just fair to put together a web service to share it with other people at the event. It helped that I had already written similar code for apt-xapian-index and dde before. Here is the server, quickly built on bottle. The very last line starts the server and it is where you can configure the listening interface and port.
#!/usr/bin/python # xserve - Make the GMP24 tweet Xapian database available on the web # # Copyright (C) 2010 Enrico Zini <enrico@enricozini.org> # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. import bottle from bottle import route, post from cStringIO import StringIO import cPickle as pickle import simplejson import sys import os, os.path import xapian import urllib import math bottle.debug(True) DBNAME = sys.argv[1] QUERYLOG = os.path.join(DBNAME, "queries.txt") data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json")))) prefixes = "place": "XP", "soc": "XS", "loc": "XL" prefix_desc = "place": "Place name", "soc": "Social category", "loc": "Social location" db = xapian.Database(DBNAME) stem = xapian.Stem("english") qp = xapian.QueryParser() qp.set_default_op(xapian.Query.OP_AND) qp.set_database(db) qp.set_stemmer(stem) qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME) for k, v in prefixes.iteritems(): qp.add_boolean_prefix(k, v) def make_query(qstring): return qp.parse_query(qstring, xapian.QueryParser.FLAG_BOOLEAN xapian.QueryParser.FLAG_LOVEHATE xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE xapian.QueryParser.FLAG_WILDCARD xapian.QueryParser.FLAG_PURE_NOT xapian.QueryParser.FLAG_SPELLING_CORRECTION xapian.QueryParser.FLAG_AUTO_SYNONYMS) @route("/") def index(): query = urllib.unquote_plus(bottle.request.GET.get("q", "")) out = StringIO() print >>out, ''' <html> <head> <title>Query</title> <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script> <script type="text/javascript"> $(function() $("#queryfield")[0].focus() ) </script> </head> <body> <h1>Search</h1> <form method="POST" action="/query"> Keywords: <input type="text" name="query" value="%s" id="queryfield"> <input type="submit"> <a href="http://xapian.org/docs/queryparser.html">Help</a> </form>''' % query print >>out, ''' <p>Example: "car place:wigan"</p> <p>Available prefixes:</p> <ul> ''' for pfx in prefixes.keys(): print >>out, "<li><a href='/catinfo/%s'>%s - %s</a></li>" % (pfx, pfx, prefix_desc[pfx]) print >>out, ''' </ul> ''' oldqueries = [] if os.path.exists(QUERYLOG): total = db.get_doccount() fd = open(QUERYLOG, "r") while True: try: q = pickle.load(fd) except EOFError: break oldqueries.append(q) fd.close() def print_query(q): count = q["count"] print >>out, "<li><a href='/query?query=%s'>%s (%d/%d %.2f%%)</a></li>" % (urllib.quote_plus(q["q"]), q["q"], count, total, count * 100.0 / total) print >>out, "<p>Last 10 queries:</p><ul>" for q in oldqueries[:-10:-1]: print_query(q) print >>out, "</ul>" # Remove duplicates oldqueries = dict(((x["q"], x) for x in oldqueries)).values() print >>out, "<table>" print >>out, "<tr><th>10 queries with most results</th><th>10 queries with least results</th></tr>" print >>out, "<tr><td>" print >>out, "<ul>" oldqueries.sort(key=lambda x:x["count"], reverse=True) for q in oldqueries[:10]: print_query(q) print >>out, "</ul>" print >>out, "</td><td>" print >>out, "<ul>" nonempty = [x for x in oldqueries if x["count"] > 0] nonempty.sort(key=lambda x:x["count"]) for q in nonempty[:10]: print_query(q) print >>out, "</ul>" print >>out, "</td></tr>" print >>out, "</table>" print >>out, ''' </body> </html>''' return out.getvalue() @route("/query") @route("/query/") @post("/query") @post("/query/") def query(): query = bottle.request.POST.get("query", bottle.request.GET.get("query", "")) enquire = xapian.Enquire(db) enquire.set_query(make_query(query)) count = 40 matches = enquire.get_mset(0, count) estimated = matches.get_matches_estimated() total = db.get_doccount() out = StringIO() print >>out, ''' <html> <head><title>Results</title></head> <body> <h1>Results for "<b>%s</b>"</h1> ''' % query if estimated == 0: print >>out, "No results found." else: # Give as results the first 30 documents; also use them as the key # ones to use to compute relevant terms rset = xapian.RSet() for m in enquire.get_mset(0, 30): rset.add_document(m.document.get_docid()) # Compute the tag cloud class NonTagFilter(xapian.ExpandDecider): def __call__(self, term): return not term[0].isupper() and not term[0].isdigit() cloud = [] maxscore = None for res in enquire.get_eset(40, rset, NonTagFilter()): # Normalise the score in the interval [0, 1] weight = math.log(res.weight) if maxscore == None: maxscore = weight tag = res.term cloud.append([tag, float(weight) / maxscore]) max_weight = cloud[0][1] min_weight = cloud[-1][1] cloud.sort(key=lambda x:x[0]) def mklink(query, term): return "/query?query=%s" % urllib.quote_plus(query + " and " + term) print >>out, "<h2>Tag cloud</h2>" print >>out, "<blockquote>" for term, weight in cloud: size = 100 + 100.0 * (weight - min_weight) / (max_weight - min_weight) print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(query, term), size, term) print >>out, "</blockquote>" print >>out, "<h2>Results</h2>" print >>out, "<p><a href='/'>Search again</a></p>" print >>out, "<p>%d results over %d documents, %.2f%%</p>" % (estimated, total, estimated * 100.0 / total) print >>out, "<p>%d/%d results</p>" % (matches.size(), estimated) print >>out, "<ul>" for m in matches: rec = data[m.document.get_data()] print >>out, "<li><a href='/item/%s'>%s</a></li>" % (rec["id"], rec["text"]) print >>out, "</ul>" fd = open(QUERYLOG, "a") qinfo = dict(q=query, count=estimated) pickle.dump(qinfo, fd) fd.close() print >>out, ''' <a href="/">Search again</a> </body> </html>''' return out.getvalue() @route("/item/:id") @route("/item/:id/") def show(id): rec = data[id] out = StringIO() print >>out, ''' <html> <head><title>Result %s</title></head> <body> <h1>Raw JSON record for twit %s</h1> <pre>''' % (rec["id"], rec["id"]) print >>out, simplejson.dumps(rec, indent=" ") print >>out, ''' </pre> </body> </html>''' return out.getvalue() @route("/catinfo/:name") @route("/catinfo/:name/") def catinfo(name): prefix = prefixes[name] out = StringIO() print >>out, ''' <html> <head><title>Values for %s</title></head> <body> ''' % name terms = [(x.term[len(prefix):], db.get_termfreq(x.term)) for x in db.allterms(prefix)] terms.sort(key=lambda x:x[1], reverse=True) freq_min = terms[0][1] freq_max = terms[-1][1] def mklink(name, term): return "/query?query=%s" % urllib.quote_plus(name + ":" + term) # Build tag cloud print >>out, "<h1>Tag cloud</h1>" print >>out, "<blockquote>" for term, freq in sorted(terms[:20], key=lambda x:x[0]): size = 100 + 100.0 * (freq - freq_min) / (freq_max - freq_min) print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(name, term), size, term) print >>out, "</blockquote>" print >>out, "<h1>All terms</h1>" print >>out, "<table>" print >>out, "<tr><th>Occurrences</th><th>Name</th></tr>" for term, freq in terms: print >>out, "<tr><td>%d</td><td><a href='/query?query=%s'>%s</a></td></tr>" % (freq, urllib.quote_plus(name + ":" + term), term) print >>out, "</table>" print >>out, ''' </body> </html>''' return out.getvalue() # Change here for bind host and port bottle.run(host="0.0.0.0", port=8024)...and then we presented our work and ended up winning the contest. This was the story of how we wrote this set of award winning code.
On Thursday it was my birthday, and I fulfilled a promise I made to
myself many years ago. I've always been a car nut,
and when I was about 6 years old and had posters of Lamborghinis on my
bedroom walls I promised myself that by the time I reached 35 I would
have one. That would be long enough in the future that it
might happen, but I'd still be young enough to still
be able to appreciate it. Well, as it got to the beginning of this
year I realised I was running out of time and my bank balance didn't
quite look large enough to buy a Lamborghini. So, I cheated a little
and I hired one for the day on Thursday.
More pictures here.
I had one car booked a few weeks ago, but unfortunately the company
involved had to let me down as the car developed problems a couple of
days before I was due to have it and needed to go away for
repairs. I'm not going to complain too much about them, as it wasn't
their fault and they did their level best to make things right:
offering a different car, a refund or a later replacement hire. But
dammit! I wanted the Lamborghini and I wanted it on my
birthday! I quickly rang around on Wednesday to find somebody else,
and thankfully the folks at Signature Car Hire were
very helpful, sorting out a car with just a few hours'
notice. Yay!
When they guys turned up on Thursday morning to deliver the car to
my office, I couldn't stop myself grinning like the 6-year old boy
whose promise I was keeping. Here was a gorgeous car, and it was mine
for the day!!! I planned a route for the day to take in some nice
driving roads and, importantly, some tunnels. I've watched Top Gear on
the TV and I read Autocar every week, and they had both said that
taking a fast sports car through a tunnel with the roof down was a
good thing to do. I now know that's an understatement!
As a whole, the car was awesome: the acceleration, the handling,
the brakes were all wonderful. But, as I found out on that day, the
noise from the engine was just astonishing. I went through
several tunnels that day, heading down past Baldock to Hatfield and
back, and at full throttle I could not help but laugh out loud at the
music coming from the 5-litre V10 orchestra just inches behind
me. It's an experience I'm never going to forget. grin.
I also gave short rides to quite a few of the people at work; oddly
I had lots of friends that day... :-)
I'd also planned other things for the day. I bought myself a new camera in
advance to make sure I got some nice photos of the day. Then at the
end of the day I went out to the pub
with a huge group of friends and I was forced to drink lots of
beer. It was horrible, horrible I tell you.
Ok, so at work, pretty much the whole company uses Windows, of some sort. The web team, however, are pretty adamant that they use the best tools available for them, and we use Ubuntu because of this (because it s the easiest option that we all have the same distro, and as long as it s Linux, we don t mind)
Anyway, at the moment we re working through the process of moving all our internal business applications from Visual Basic.NET (urgh!) to PHP + a web based app. The Product Owner for this is one of the bosses, who currently uses Windows Vista.
He has, however, seen us all using Ubuntu and it s got him interested in using it. He s said that he wants to try using Ubuntu, and see whether he can get his work done on that, and slowly transition to it, as obviously, he needs to be able to use Windows for the .NET stuff (which sadly, doesn t work well under WINE).
So, we set him up a Virtual Machine, and he got to grips with what he was doing, great. Except for the fact that he had to allocate memory to it, switching back and forth between windows and Ubuntu was a pain. He eventually gave up on this idea, and got a spare machine, a spare monitor, and set it up next to his machine.
Still, however, it was a bit of a pain, he d have to switch position on his desk, start using a different keyboard and mouse, and i was all a bit of hassle.
Recently, we ve gotten a big screen (which isn t actually that big!) for the Web Team to monitor the servers, our order download service, etc etc. We were showing our manager some of the stuff we could do with it, and he asked the question So, how do you control it - to which my response was to move my mouse off of the edge of my screen, and onto the big screen. So is that a third monitor? he asked. No I replied, it s another machine.
Enter Synergy
So this is the point where I explained to our manager exactly what synergy is. Trying to explain it however, is a bit of a hassle, so I ll try my best here. Synergy is an application that allows you to control another machine s keyboard and mouse from your own PC. It s a bit of a mix between a KVM switch, and Remote Desktop, but it doesn t require extra hardware, and you dont have to relay the video across the network to show it on your screen too.
At this point, our boss got a glint in his eye
Does it work in Windows? he asked. At this point, I didn t realise that he had a seperate machine for Ubuntu. I told him it did anyway, but I wasn t too sure whether it worked with Vista
Lets give it a go anyway
So we went over to his machine, and I installed quicksynergy on his ubuntu box, while he downloaded the Windows installer for Synergy. 5 minutes later, he was grinning like a maniac as he moved his mouse from one machine to another.
Now he s happily using Ubuntu without any hassle, it s just a third screen to him - with different stuff on it.
I think the moral here is, there are different ways to get people to be able to use Linux, and Synergy is one of those great tools that enable people to use Linux without having all the hassle that comes with switching over.
I think also, a quote from m manager sums it up.
This is one of the best things I ve seen all year. I thought it d take forever to setup, but it was so quick. I can be so much more productive nowI hope that this will be one of the turning points for my manager, and we ll have another convert by the end of the year
Real men would design a custom ASIC and write a new FORTH system optimized for Reddit, complete with custom developent environment, web server, etc. It would multiply the development time-to-1.0 by about 2-4x, but the entire system would be smaller than the ethernet plug, would handle a saturated gigabit pipe with sub centisecond latency, never crash, and use less power than your toaster. Everyone would be too busy saying "WTF is FORTH?" to bitch about you not using Ruby on Rails. Then you'd lose to the guy who did it in PHP (*turn* *puke*), because it's not about the technology, stupid.
