Search Results: "ed"

18 October 2010

Benjamin Mako Hill: Redefining "Realistic"

When talking about free culture or free software, many people suggest that they would love to support free models, but that they don't see how to make it all work. Until they have an alternate model in front of them, they cannot bring themselves to argue for a more ethical alternative. I disagree with this approach. Instead, I say, "this is the world I want to live in and, even though I don't know exactly how to get to there from here, I'm going to refuse to settle for anything short of this ideal." Most people dismiss such thinking as "impractical" and "unrealistic." I think most people are being unimaginative. Robot jockeys are one recent illustrative reason, among many, that I feel comfortable taking this position. Some background is necessary for those that are unfamiliar with the example. A decade ago, several Gulf emirates used thousands of young boys from Sudan and South Asia as jockeys for camel racing. Human rights groups campaigned against the practice and suggested that these boys were at sometimes held as slaves and intentionally underfed to keep their weight low. Despite criticism, camel racers resisted moving away from young boys as jockeys. If they moved to heavier adults instead of young boys, they reasoned, the camels would be much slower. Of course, they were right. But they were being unimaginative in the alternatives they were considering. As the young jockeys became a increasingly unjustifiable public relations disaster for the states that supported it, law-makers in several Gulf states gave in to calls from UNICEF and others and created laws to outlaw the practice. Within three years of UAE passing strict laws against child jockeys, Swiss engineers, funded by racers desperate for an alternative, had created the first robotic camel jockey. Within several years these jockeys were lighter, cheaper, more responsive to the owner, and well on their way to being more effective than any young boy. When forced, by law and by an ethical prerogative, to come up with an alternative to young boys, racers created a solution that was superior, along nearly every axis, to the system they had fought to keep. Although the costs to society of proprietary software cannot be compared to slavery and abuse, the basic same pattern of solution-seeking can be seen in the example of free software. Early free software advocates suggested that most programmers would likely need to take a paycut. As it turned out, vibrant and successful economic models to support free software have supported a large and growing free software industry. But we have free software business models only because a small group of principled individuals refused to settle for what they knew, came up with creative ethical business models that "just might work," and put their own paychecks on the line to try them out. As open source has shown, some of these creative solutions offered models superior to what we had before. In the world of software development, free software redefined "practical" and "realistic." One can think of solving human problems as like searching for the highest point in hilly terrain in thick fog. It's easy to get stuck on the top of the first little hill you walk up (i.e., a local maximum) and then conclude you can never do better. If we refuse to compromise and force ourselves to leave that first little hill, chances are pretty good we'll find a "higher" peak. Of course, it is also possible that we will find the global maximum or the best possible solution to a given problem. In those cases, any change will mean a sacrifice. But when dealing with most most social and legal dilemmas, there are enough variables involved that this seems very unlikely. Indeed, most big problems can be thought of as having many interacting dimensions -- and only some of these will be ethical concerns. In other words, most social problems are more like the problem of child camel jockeys than they are like trying to transcend the laws of physics. Business models and laws for the regulation of technology and knowledge are extremely complicated human creations. Do we really think we cannot create ethical systems to compensate cultural creators that are at least as good as what we have now? If we never force ourselves to be "impractical" and "unrealistic", we will never find out.

17 October 2010

Christian Perrier: [life] [running] 3h40'20"

One more personal record broken for my 6th marathon (including those I ran alone). This time, it was a small marathon, the Seine-Eure marathon, in Normandy, between Paris and Rouen. But a small marathon is small only by number of runners (here, 500, plus about 160 teams of Ekiden runners -relay marathon-). It still has 42.195km. Despite a non optimal preparation (too many runs in September and no run over 25km since June), it went fairly well and I could benefit from the optimal conditions: ultra-flat race, few turns, few runners, goo dweather, though very cold for the season. Again, I started too fast (around 5'/km) and had trouble finishing with pace down to 5'30. But I'm still very happy for breaking my PB for the 4th time in a row with my results now being: Next marathon will be Paris marathon, on April 11th 2011. Target: 3h35 For french-speaking readers: full report on the Kikourou website.

Neil Williams: Migrating debconf settings

I've been trying to work out how to handle (or whether to handle) the vestigial debconf values from emdebian-tools which was replaced in Debian some months ago. I tried to transfer the postrm script to libemdebian-tools-perl which itself is currently a transitional package. Turns out that when debconf tries to purge libemdebian-tools-perl, it doesn't purge the old emdebian-tools settings (it looks for libemdebian-tools-perl.templates instead of emdebian-tools.templates) - unless those were themselves setup by libemdebian-tools-perl. i.e. users with the original emdebian-tools setup will likely have debconf settings that are not purged by removing the package to which those settings should have migrated.

I have reconstructed a bare templates file locally and a simple script that can purge those values and it works on my system - I'm just not sure if this should be implemented in what is now merely a transitional package (libemdebian-tools-perl) which exists merely to support migrating to the current emdebian-crush package.

It's only three fields - single word values - and debconf settings are only a cache after all. Comments are still disabled on my blog but my email address isn't hard to find. If people really think it's worth writing code to "catch-up" with these redundant settings, I can do something after Squeeze. What I don't want to do is reintroduce debconf templates which are no longer used, merely to remove old settings left behind from the first migration.

So, lazyweb, what's the feeling? Is this worth doing anything about?

Gerfried Fuchs: Debian at openSUSE conference

Within the next few days there will be the openSUSE conference held in Nuremberg/Germany. The topic of the conference was set to Collaboration Across Borders and along that theme they invited other distributions to submit talks. Given my involvement with the Derivatives Frontdesk zack asked me whether I'd like to submit a talk proposal. I usually have a hard time to refuse such questions this will actually happen. My talk will be in the Distributions Track on Thursday afternoon, the topic is Debian The Project and its Resources. The resources that I will cover are those that can be very useful to other distributions too, and last Thursday I tested the talk at our local Debienna Meeting, gathered further ideas and feedback to cover in the talk. Thanks to you, guys! Unfortunately contrary to what we are used to in Debian (extraordinary thanks to the Video Team!!!) there will be no streaming or recording of the talks from what I was told, so you have either to hurry to be there, or be satisfied with the slides I'll put up after the talk. It's the best I can offer. See you around!

/debian permanent link Comments: 5 Flattr this

Dirk Eddelbuettel: RPostgreSQL 0.1-7

After a somewhat long hiatus, RPostgreSQL version 0.1-7 has now been released to CRAN. RPostgreSQL connects R to PostgreSQL database systems using the standard DBI interface. This version fixes a number of issues that had been compiled in the issue tracker on the project site at Google Code. Tomoaki Nishiyama, who joined our small development group for his package a few weeks ago, was instrumental in a number of these fixes, with assistance from Joe Conway. The relevant NEWS file entry follows below:
Version 0.1-7 -- 2010-10-17
    o   Several potential buffer overruns were fixed
    o   dbWriteTable now writes a data.frame to database through a network
        connection rather than a temporary file. Note that row_names may be
        changed in future releases.  Also, passing in filenames instead of
        data.frame is not supported at this time. 
    o   When no host is specified, a connection to the PostgreSQL server 
        is made via UNIX domain socket (just like psql does)
    o   Table and column names are case sensitive, and identifiers are escaped
        or quoted appropriately, so that any form of table/column names can be
        created, searched, or removed, including upper-, lower- and mixed-case.
    o   nullOk in dbColumnInfo has a return value of NA when the column does
        not correspond to a column in the table. The utility of nullOk is
        doubtful but not removed at this time.
    o   Correct Windows getpid() declaration (with thanks to Brian D. Ripley)
    o   A call of as.POSIXct() with a time format string wrongly passed to TZ
        has been corrected; this should help with intra-day timestamps (with
        thanks to Steve Eick)
    o   Usage of tmpdir has been improved on similarly to Linux (with thanks
        to Robert McGehee)
More information is on the my RPostgreSQL page, and on project site at Google Code.

David Welton: When sunny gets blue, I get traffic

This is definitely a "captain obvious" observation, but it's interesting to see just how much bad weather here in the Veneto is correlated with traffic to my real-time weather site, http://www.meteo-veneto.net The precipitation and insolation numbers for the month of September - as well as the radar images for the site itself - come from http://www.arpa.veneto.it/ The numbers are all normalized so as to make them easy to compare. The cloudy/dark numbers are the inverse of the insolation for the month, and could probably be fiddled with to be a bit more accurate - it's obvious that the number increases as the month goes on, with less sunlight.

16 October 2010

Tim Retout: Erm... more bugs.

I went shopping today, and inadvertently stole a very nice cardigan-like thing from a multinational clothing retailer. Enough anecdotes. Yesterday: Today: Looking at the weekly summary reports, we squashed a net total of 31 RC bugs last week. Of course, it's possible they'll get more difficult as we go along... but I'm hoping to see all the easy ones closed before the BSP two weeks from now.

David Paleino: How to: database-less local map rendering with Mapnik and OSM

Here's a quick howto for local map rendering, using python, mapnik and data from OpenStreetMap. This tutorial shows how to create a stylesheet for mapnik from scratch, if you're looking for how to tweak OpenStreetMap's mapnik rendering, I plan to write a tutorial on PostGIS-driven rendering at a later time. This post is not for those who want to make a local slippy map, nor those who want to render different areas of big dumps (think of big national dumps). We'll render some features of a residential area of a city with buildings mapped. We'll get a small dump of the area for this, using the XAPI. First, you need to install mapnik. On Debian-like systems, it should be as easy as:
# apt-get install python-mapnik
Then, you need some python code to use the mapnik library. Let's call it render.py:
#!/usr/bin/python
import mapnik
###
# Configuration
###
style = 'style.xml'
output = 'output.svg'
width = 1280
height = 800
bbox = '12.58664,37.65759,12.5945,37.66325'
###
# Don't touch below!
###
bbox = bbox.split(',')
ll = mapnik.Coord(float(bbox[0]), float(bbox[1]))
tr = mapnik.Coord(float(bbox[2]), float(bbox[3]))
mymap = mapnik.Map(width, height)
mapnik.load_map(mymap, style)
map_bbox = mapnik.Envelope(ll, tr)
mymap.zoom_to_box(map_bbox)
mapnik.render_to_file(mymap, output)
For the purposes of this post, we're going to create a 1280x800 map -- adjust that to suit your needs -- and we specified a bounding box corresponding to that area. That's composed as west,south,east,north coordinates. Be sure to get them right :-). Now we need to write a Mapnik stylesheet, we'll call it style.xml. That's a XML document, you can find the documentation at the official website. The skeleton of a stylesheet is as follows:
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE Map>
<Map bgcolor="lightblue" srs="+proj=latlong +datum=WGS84">
<Style name="...">
...
</Style>
<Layer name="...">
 <Datasource>
  ...
 </Datasource>
</Layer>
</Map>
The stylesheet has an arbitrary number of Style elements and an arbitrary number of Layer elements. The former class of elements define the appearance of the features, while the latter defines where to get data from. Each layer has a Datasource child: here's where you say where to get your data from. We're going to use the osm input plugin, which lets you parse the dump directly. This has some drawbacks: But works great for small dumps. In this tutorial, we want to pretty render a residential area, so we first need to render the road network. This will mainly be made of highway=residential roads. Each style can have one or more Rule element. These define what elements that particular style should apply to. Each rule can have different rendering settings. The given dump has different roads: residential, unclassified, tertiary and secondary. Let's add a rule for each. Here's the first of them:
<Rule>
 <Filter>[highway] = 'residential'</Filter>
 <LineSymbolizer>
  <CssParameter name="stroke">#ffffff</CssParameter>
  <CssParameter name="stroke-width">5</CssParameter>
  <CssParameter name="stroke-linejoin">round</CssParameter>
  <CssParameter name="stroke-linecap">round</CssParameter>
 </LineSymbolizer>
 <TextSymbolizer name="name" face_name="DejaVu Sans Book" size="8" fill="#000" halo_radius="1" spacing="300" placement="line" />
</Rule>
It is quite straightforward: you define what elements the rule should apply to, and then add some symbolizer. For a detailed description of each symbolizer, please read the documentation. The example dump also has a railway and some steps. To render them, we use the stroke-dasharray property:
<Rule>
 <Filter>[highway] = 'steps'</Filter>
 <LineSymbolizer>
  <CssParameter name="stroke">#ff0000</CssParameter>
  <CssParameter name="stroke-width">5.0</CssParameter>
  <CssParameter name="stroke-dasharray">2,1</CssParameter>
 </LineSymbolizer>
</Rule>
<Rule>
 <Filter>[railway] = 'rail'</Filter>
 <LineSymbolizer>
  <CssParameter name="stroke">#222222</CssParameter>
  <CssParameter name="stroke-width">3</CssParameter>
  <CssParameter name="stroke-linejoin">round</CssParameter>
 </LineSymbolizer>
 <LineSymbolizer>
  <CssParameter name="stroke">white</CssParameter>
 <CssParameter name="stroke-width">1</CssParameter>
  <CssParameter name="stroke-linejoin">round</CssParameter>
  <CssParameter name="stroke-dasharray">10,20</CssParameter>
 </LineSymbolizer>
</Rule>
Also here, it is quite simple. The values given to stroke-dasharray determine the size of the dashes, and they can be alternated. So, for the steps we have 2px of #ff0000 and 1px of trasparency, while for the rail we have 10px of white and 20px of #222222. What if we want to add some markers to the map? Easy, just use another symbolizer, called PointSymbolizer:
<Rule>
 <Filter>[highway] = 'stop'</Filter>
 <PointSymbolizer width='10' height='22' file='stop.png' type='png' allow_overlap='true' opacity='1.0' />
</Rule>
Here we mark stop signals with a pre-rendered stop.png (only png and tiff are currently supported, and svg support is only in the development version). What to do next? Well, we can make this map really nice. We'll render buildings with a pseudo-3D effect, which, however, does not reflect real heights -- but it's rather nice to see. How to achieve this? BuildingSymbolizer is the answer.
<Rule>
 <Filter>[building] &lt;&gt; ''</Filter>
 <BuildingSymbolizer>
  <CssParameter name="fill">#00ff00</CssParameter>
  <CssParameter name="fill-opacity">0.4</CssParameter>
  <CssParameter name="height">0.00007</CssParameter>
 </BuildingSymbolizer>
</Rule>
Here we just draw building=*. Regarding the height, you need to play a bit -- it took a while before I found a good value of it. One thing missing in this stylesheet, is the rendering of coastlines. Since we don't have a coastline here, I'm skipping it, but the easy way is: use another rule. This is really just a trick; OpenStreetMap renders the coastline in a different way respect to other features, that's why I'm using this trick. I'm not posting the whole stylesheet, but you can download it. You can also get the script, the OSM dump and the stop.png image. Now, you should be able to just do:
$ ./render.py
and you will have your map in output.svg :-) Here's what you would get if you use the files provided above (click to see the original SVG version):

Thorsten Glaser: Debian/m68k re-bootstrapping

I m currently working on something which will eventually amount to a re-bootstrapping of sorts of the Debian/m68k (Linux) port patches to the Linux kernel, gcc, etc. are prepared (some have been accepted into upstream or the packages already). I will probably have more, once the compile processes finish, anyway (even emulated, it s slow). I think that, once I get past that TLS (thread-local storage, needed by eglibc) migration, I will try to find out a list of packages needed for debootstrap (AFAICT: all packages marked Essential, or of priority important, and all marked Build-Essential (for the *-builder variant), and their dependencies (although I ll substitute sysv-rc with file-rc, which is better and needs less deps)) and pull arch:all from sid, then build the rest myself using a consistent snapshot of sid possibly with patches going to unreleased. Then I can use cowbuilder to make cleaner packages, which can eventually be uploaded (once I get enough to get a buildd running kernel, bootloader, etc) binNMUs are way to go here I suppose. I will only upload once it s self-hosted, installable (seen by edos-debcheck), clean, etc. (i.e. I ll rebuild all binaries) and probably keep a bootstrap repo around (until m68k caught up) so that unstable (possibly amended by unreleased for a while) will not again become uninstallable, e.g. if arch:all packages change their dependencies (Python, gcc-defaults are some I ve seen). That bootstrap repository is needed anyway because debootstrap can t install from two separate repositories (unstable+unreleased for example). Progress is slow because I try to keep as close to official packages as possible, refuse to cross-compile, and try to produce uploadable if possible packages all the time. Getting patches into packages, so that I can build from unstable, instead of debian-ports.org unreleased, has proven time-consuming and occasionally frustrating as well. Although I would like to thank the people who helped me on the way already. (I am not naming any in fear of forgetting some, but you know who you are ) They are among the Debian (gcc, kernel, m68k) and Linux-68k crowd. (Why does genattrtab in gcc-4.4 take 3 hours when it took less than half an hour in gcc-4.3 anyway?) I m also still working on mksh and some Python ISO hacks for mika and some minor stuff, and further cleaning up MirBSD.

Dirk Eddelbuettel: RcppArmadillo 0.2.8

Version 0.2.8 of RcppArmadillo is now on CRAN. This updates the included Armadillo template library to version 0.9.90 which Conrad released two few days ago. RcppArmadillo makes it easy to write highly efficient and highly readable C++ code for linear algebra (based on Armadillo) in R extensions (using Rcpp for the interface). The short NEWS file extract follows, also containing Conrad's entry for 0.9.90::
    o   Upgraded to Armadillo 0.9.90 "Water Dragon":
         * Added unsafe_col()
         * Speedups and bugfixes in lu()
         * Minimisation of pedantic compiler warnings
    o   Switched NEWS and ChangeLog between inst/ and the top-level directory
        so that NEWS (this file) gets installed with the package
More information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

Bastian Venthur: Introducing python-popcon

Python-popcon is a small Python library which allows to query Debian s popcon database. The usage is very trivial:
>>> import popcon
>>> popcon.package('python-popcon')
'python-popcon': 2 >>> popcon.package('icedove', 'iceweasel')
'icedove': 12140, 'iceweasel': 45666
>>> popcon.package('foobarbaz')

You can call the package method with an arbitrary number of package names and it will return a dictionary with package name : popcon value mappings for all packages found. If a package is not found in the popcon database it will not be in the resulting dictionary or if it was the only package in the method call, the method will return an empty dictionary. The popcon value is the number of installations according to Debian s Popularity Contest service. There is also a second method which gives some more information
>>> popcon.package_raw('icedove', 'iceweasel')
'icedove': [8065, 2195, 1879, 1], iceweasel : [27857, 10681, 7120, 8]
It returns a list of [vote, old, recent, no-files] for the given package. How does it work? Upon a query, popcon downloads http://popcon.debian.org/all-popcon-results.txt.gz, extracts and parses the file and returns the desired information. Since downloading and extraction this file is expensive, it saves the file and tries to re-use it is not older than 7 days. So while the first call of one of the above methods can take a few seconds, the result will appear almost instantaneously for the next week until the cache file expired. That s it, no rocket science but a convenient way to get popcon information from within python. The package python-popcon is available in Debian/Sid for a few days.

Ingo Juergensmann: Upgrading to Squeeze - and suddenly CGI doesn't work anymore

Before I upgraded from Lenny to Squeeze, all my CGI scripts were working properly. The CGI scripts has been executed by Apache and resulted in rendered webpage. After the upgrade to Squeeze all those CGI scripts stopped been executed but instead started to get displayed as a plain text. Common to all those CGI scripts is that they have .phtml as suffix, but the bang path/shebang consists of "#!/usr/bin/python". As an example you can have a look at Buildd.Net and its scripts like this one. The section in Apaches config looks like this:
<Directory /home/builddnet/unstable/WWW/cgi/>
Options -Indexes +ExecCGI +FollowSymLinks
AddHandler cgi-script .cgi .sh .pl .py .phtml
Order allow,deny
Allow from all
</Directory>
So, I would expect that the CGI script handler will get executed when loading a *.phtml file and that the shebang would be honoured. Funny enough: when I rename that script to *.cgi it works. I haven't figured out yet, what causes this beahviour or what have changed during the upgrade - and how to revert to the old behaviour. So, dear lazyweb, can you give me some hints and pointers?

Siegfried Gevatter: Debious A dubious Debian packaging GUI

Just some little (unfinished) concept mockup. Seeing that much of it still ends up as a text box with syntax highlighting it d probably make sense to implement it as a gedit plugin.

Balsamiq source XML Related posts:

  1. One week with Debian
  2. Packaging: Test-building your packages
  3. Arreglant paquets que no compilen: necpp

8 comments
Siegfried-Angel Gevatter Pujals, 2010. Permalink License Post tags: , ,

Pietro Abate: hidden ssh service via tor

We live in a nat-ed / firewall-ed world. Almost all DSL providers don't give public IPs and when the do, they are often behind a draconian firewall. In this context having an emergency remote shell, despite not fast and not public is very handy. A simple way to solve this problem it to create a hidden server on the tor network and the access the shell from anywhere in the world without caring of change of IPs, routing, dns or anything else. On debian you can just install tor from the official repository. Since Tor is not available in ubuntu (but it is available on debian), we need to get it directly from the tor website. There is a nice write-up on the ubuntu site : https://help.ubuntu.com/community/Tor . And these are the details on the tor website. So we add this to apt.sources
deb http://deb.torproject.org/torproject.org lucid main
and the we aptitude install tor .The package will install and run by default the tor daemon. Next step is to edit /etc/tor/torrc to add the proxy server.
HiddenServiceDir /var/lib/tor/ssh/
HiddenServicePort 22 127.0.0.1:22
Remember also to install the openssh-server server if you don't have it already. And this is it. In the directory /var/lib/tor/ssh/ you will find a file with the hostname on the TOR ring that you have to use to connect to you new hidden server. On the client side we need to aptitude install connect-proxy. It's a simple tool to tunnel ssh through a socks5 connection. Now you are ready to test. In your ~/.ssh/config you can simply add something like
Host *.onion
ProxyCommand connect -R remote -5 -S 127.0.0.1:9050 %h %p
and then ssh-way youronionhost.onion server . The connection will be veeeeery slow since you are going through different layers of encryption and indirection. You should also check the hostid of your server before connecting and dropping in a pub-key as you should never trust your friendly TOR providers (US govt, Chinese gvt, Iranian govt, etc ...). For emergency is actually pretty handy. For anything else if will make you die of boredom ...

Update Ahhh . It seems I've assumed that since TOR was not available on ubuntu, this was the same on debian. Tor is definitely available on debian, but not on ubuntu. Blah... check before asserting wrong information ! Post fixed.

Martin-&#201;ric Racine: Batch-editing EXIF data to add copyrights?

I've recently become more serious about my photographic hobby and it dawned onto me that manually editing each JPEG to add my copyright was entirely the wrong approach. Thus, I was wondering if anybody would know of a Free Software tool that can batch edit the EXIF data in JPEG images? What I'd like to accomplish is simple: Preferences go for a tool that is already packaged for Debian or Ubuntu but, worst comes, I could package the software myself.

Vincent Sanders: Coming to terms

Yesterday was not a great day. One of the family's cats died.

Molly just after she arrived 26th September 2000
Molly, for that was her name, came to us the week Melodie and I moved into our first house together. September 15th 2000, in the middle of the fuel protests which were raging at the time, we hired a van and moved in.

Most people would have considered that enough for one week! We happened to be in Halifax on the Sunday afternoon at the supermarket and decided to visit the RSPCA because somehow we decided what our new home needed was a cat.

On that day a decade ago we saw Molly and Lucy, a pair of cats who needed a new home as their previous owners had a baby who was allergic to them. I still recall them skulking at the back of their cage in the cattery, Lucy who was reluctant to come down of a perch to be petted and Molly looking generally unhappy. No-one else wanted this misfit pair because they were already both over 5 years old, the warden despairing of ever finding them a home.

Of course they were obviously the right animals for us! ;-) So by the end of the week we had taken them home. Molly immediately showed how things were going to be by shredding her way out of the RSPCA provided double walled cardboard carrier. To the day she died she detested being placed inside a carrier, funny that.

Melodie and Molly in the snow 28th December 2000
By the end of the year, when the snow came, the cats ruled the house and we were all happy together.

Little did we realise that soon in the summer of 2001 there would be another arrival to the family.

Molly with Melodie and Alex 2001
The arrival of our first child in June 2001 was a complete change in all our lives but we all managed to settle back into a routine. Although the banishment of the cats from upstairs remained a point of disagreement for a long time.

As Alex grew up the cats learned that feeding times for Alex could result in all manner of things falling from above. Soon Alex was mobile which resulted in different lessons on cats being sharp objects if not treated with respect. In February 2003 our family grew once more with the arrival of Joshua. The cats, now used to infants, took this in their stride.

Molly in Febuary 2004 hiding on a windowsill
While still about day to day the cats are not captured on camera so often form this point on.

Over the forthcoming months they ensured they were out of reach of the newborn and the precocious toddler.

By the time of Joshuas first birthday in 2004 Molly had taken to hiding "out of the way" as much as possible but remained as affectionate as ever.

Molly in a box March 2007
As the children grew up and life progressed Molly became ever more at home and developed the odd aggravating trait like taking clean washing off airers and dragging it out the window, through the cat flap and upstairs so she could sleep on it!

She still enjoyed participating in claiming boxes and defending them vigorously though!

The kids took on the job of feeding the cats which made their bond closer ensuring they were greeted by happy cats sitting lookout as they came home from school each day.

Molly continued to be a good companion and an infuriating self centred animal like all good cats.
Molly asleep in the kitchen sink
Then on Tuesday evening our Neighbour knocked on our front door. They gave us the news that one of our cats had been run over and taken to the Vet. I immediately went to the back door and called and shook the treat box.

Lucy came running, but molly did not. We rang the vets who confirmed they had molly (we had them electronically tagged back in Halifax) and that they advised we did not come and see her until they had chance to asses her and deal with the shock.

As the days progressed her prognosis improved and then sank. She had a broken jaw, broken teeth, a dislocated hip, extensive bruising and something was definitely wrong with her kidney function.

Thursday evening the whole family went to see her and she purred and seemed happy to see us, she looked as bad as I feared though and somehow I knew there and then that this was probably goodbye. The children stroked her and petted her for a while and we left with an odd sadness, oh and to the sound of molly trying to gouge the veterinary nurse.

Yesterday she was due to have surgery to fix her jaw and hip...Alas when she was anaesthetised and x-rayed once more it became evident the hip was not just dislocated but her femur was fractured and there was additional damage. So just after midday came the call to ask what we wanted to do. The vet could attempt the repairs but due to her age and the other complications it was probably futile. So with heavy heart we agreed the best thing was not to revive her and she died a short time later.

I shall miss her morning greetings, her demands for attention, her sleeping in odd places and her companionship. I keep calling for her at meal times forgetting she will never come. I think Lucy is upset too, after spending their lives together her friend is gone and I cannot explain that to her.

So that is the end of Molly, a good cat.


15 October 2010

Enrico Zini: Award winning code

Award winning code Me and Yuwei had a fun day at hhhmcr (#hhhmcr) and even managed to put together a prototype that won the first prize \o/ We played with the gmp24 dataset kindly extracted from Twitter by Michael Brunton-Spall of the Guardian into a convenient JSON dataset. The idea was to find ways of making it easier to look at the data and making sense of it. This is the story of what we did, including the code we wrote. The original dataset has several JSON files, so the first task was to put them all together:
#!/usr/bin/python
# Merge the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import os
res = []
for f in os.listdir("."):
    if not f.startswith("gmp24"): continue
    data = open(f).read().strip()
    if data == "[]": continue
    parsed = simplejson.loads(data)
    res.extend(parsed)
print simplejson.dumps(res)
The results however were not ordered by date, as GMP had to use several accounts to twit because Twitter was putting Greather Manchester Police into jail for generating too much traffic. There would be quite a bit to write about that, but let's stick to our work. Here is code to sort the JSON data by time:
#!/usr/bin/python
# Sort the JSON data
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import sys
import datetime as dt
all_recs = simplejson.load(sys.stdin)
all_recs.sort(key=lambda x: dt.datetime.strptime(x["created_at"], "%a %b %d %H:%M:%S +0000 %Y"))
simplejson.dump(all_recs, sys.stdout)
I then wanted to play with Tf-idf for extracting the most important words of every tweet:
#!/usr/bin/python
# tfifd - Annotate JSON elements with Tf-idf extracted keywords
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import sys, math
import simplejson
import re
# Read all the twits
records = simplejson.load(sys.stdin)
# All the twits by ID
byid = dict(((x["id"], x) for x in records))
# Stopwords we ignore
stopwords = set(["by", "it", "and", "of", "in", "a", "to"])
# Tokenising engine
re_num = re.compile(r"^\d+$")
re_word = re.compile(r"(\w+)")
def tokenise(tweet):
    "Extract tokens from a tweet"
    for tok in tweet["text"].split():
        tok = tok.strip().lower()
        if re_num.match(tok): continue
        mo = re_word.match(tok)
        if not mo: continue
        if mo.group(1) in stopwords: continue
        yield mo.group(1)
# Extract tokens from tweets
tokenised = dict(((x["id"], list(tokenise(x))) for x in records))
# Aggregate token counts
aggregated =  
for d in byid.iterkeys():
    for t in tokenised[d]:
        if t in aggregated:
            aggregated[t] += 1
        else:
            aggregated[t] = 1
def tfidf(doc, tok):
    "Compute TFIDF score of a token in a document"
    return doc.count(tok) * math.log(float(len(byid)) / aggregated[tok])
# Annotate tweets with keywords
res = []
for name, tweet in byid.iteritems():
    doc = tokenised[name]
    keywords = sorted(set(doc), key=lambda tok: tfidf(doc, tok), reverse=True)[:5]
    tweet["keywords"] = keywords
    res.append(tweet)
simplejson.dump(res, sys.stdout)
I thought this was producing a nice summary of every tweet but nobody was particularly interested, so we moved on to adding categories to tweet. Thanks to Yuwei who put together some useful keyword sets, we managed to annotate each tweet with a place name (i.e. "Stockport"), a social place name (i.e. "pub", "bank") and a social category (i.e. "man", "woman", "landlord"...) The code is simple; the biggest work in it was the dictionary of keywords:
#!/usr/bin/python
# categorise - Annotate JSON elements with categories
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
# Copyright (C) 2010  Yuwei Lin <yuwei@ylin.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import sys, math
import simplejson
import re
# Electoral wards from http://en.wikipedia.org/wiki/List_of_electoral_wards_in_Greater_Manchester
placenames = ["Altrincham", "Sale West",
"Altrincham", "Ashton upon Mersey", "Bowdon", "Broadheath", "Hale Barns", "Hale Central", "St Mary", "Timperley", "Village",
"Ashton-under-Lyne",
"Ashton Hurst", "Ashton St Michael", "Ashton Waterloo", "Droylsden East", "Droylsden West", "Failsworth East", "Failsworth West", "St Peter",
"Blackley", "Broughton",
"Broughton", "Charlestown", "Cheetham", "Crumpsall", "Harpurhey", "Higher Blackley", "Kersal",
"Bolton North East",
"Astley Bridge", "Bradshaw", "Breightmet", "Bromley Cross", "Crompton", "Halliwell", "Tonge with the Haulgh",
"Bolton South East",
"Farnworth", "Great Lever", "Harper Green", "Hulton", "Kearsley", "Little Lever", "Darcy Lever", "Rumworth",
"Bolton West",
"Atherton", "Heaton", "Lostock", "Horwich", "Blackrod", "Horwich North East", "Smithills", "Westhoughton North", "Chew Moor", "Westhoughton South",
"Bury North",
"Church", "East", "Elton", "Moorside", "North Manor", "Ramsbottom", "Redvales", "Tottington",
"Bury South",
"Besses", "Holyrood", "Pilkington Park", "Radcliffe East", "Radcliffe North", "Radcliffe West", "St Mary", "Sedgley", "Unsworth",
"Cheadle",
"Bramhall North", "Bramhall South", "Cheadle", "Gatley", "Cheadle Hulme North", "Cheadle Hulme South", "Heald Green", "Stepping Hill",
"Denton", "Reddish",
"Audenshaw", "Denton North East", "Denton South", "Denton West", "Dukinfield", "Reddish North", "Reddish South",
"Hazel Grove",
"Bredbury", "Woodley", "Bredbury Green", "Romiley", "Hazel Grove", "Marple North", "Marple South", "Offerton",
"Heywood", "Middleton",
"Bamford", "Castleton", "East Middleton", "Hopwood Hall", "Norden", "North Heywood", "North Middleton", "South Middleton", "West Heywood", "West Middleton",
"Leigh",
"Astley Mosley Common", "Atherleigh", "Golborne", "Lowton West", "Leigh East", "Leigh South", "Leigh West", "Lowton East", "Tyldesley",
"Makerfield",
"Abram", "Ashton", "Bryn", "Hindley", "Hindley Green", "Orrell", "Winstanley", "Worsley Mesnes",
"Manchester Central",
"Ancoats", "Clayton", "Ardwick", "Bradford", "City Centre", "Hulme", "Miles Platting", "Newton Heath", "Moss Side", "Moston",
"Manchester", "Gorton",
"Fallowfield", "Gorton North", "Gorton South", "Levenshulme", "Longsight", "Rusholme", "Whalley Range",
"Manchester", "Withington",
"Burnage", "Chorlton", "Chorlton Park", "Didsbury East", "Didsbury West", "Old Moat", "Withington",
"Oldham East", "Saddleworth",
"Alexandra", "Crompton", "Saddleworth North", "Saddleworth South", "Saddleworth West", "Lees", "St James", "St Mary", "Shaw", "Waterhead",
"Oldham West", "Royton",
"Chadderton Central", "Chadderton North", "Chadderton South", "Coldhurst", "Hollinwood", "Medlock Vale", "Royton North", "Royton South", "Werneth",
"Rochdale",
"Balderstone", "Kirkholt", "Central Rochdale", "Healey", "Kingsway", "Littleborough Lakeside", "Milkstone", "Deeplish", "Milnrow", "Newhey", "Smallbridge", "Firgrove", "Spotland", "Falinge", "Wardle", "West Littleborough",
"Salford", "Eccles",
"Claremont", "Eccles", "Irwell Riverside", "Langworthy", "Ordsall", "Pendlebury", "Swinton North", "Swinton South", "Weaste", "Seedley",
"Stalybridge", "Hyde",
"Dukinfield Stalybridge", "Hyde Godley", "Hyde Newton", "Hyde Werneth", "Longdendale", "Mossley", "Stalybridge North", "Stalybridge South",
"Stockport",
"Brinnington", "Central", "Davenport", "Cale Green", "Edgeley", "Cheadle Heath", "Heatons North", "Heatons South", "Manor",
"Stretford", "Urmston",
"Bucklow-St Martins", "Clifford", "Davyhulme East", "Davyhulme West", "Flixton", "Gorse Hill", "Longford", "Stretford", "Urmston",
"Wigan",
"Aspull New Springs Whelley", "Douglas", "Ince", "Pemberton", "Shevington with Lower Ground", "Standish with Langtree", "Wigan Central", "Wigan West",
"Worsley", "Eccles South",
"Barton", "Boothstown", "Ellenbrook", "Cadishead", "Irlam", "Little Hulton", "Walkden North", "Walkden South", "Winton", "Worsley",
"Wythenshawe", "Sale East",
"Baguley", "Brooklands", "Northenden", "Priory", "Sale Moor", "Sharston", "Woodhouse Park"]
# Manual coding from Yuwei
placenames.extend(["City centre", "Tameside", "Oldham", "Bury", "Bolton",
"Trafford", "Pendleton", "New Moston", "Denton", "Eccles", "Leigh", "Benchill",
"Prestwich", "Sale", "Kearsley", ])
placenames.extend(["Trafford", "Bolton", "Stockport", "Levenshulme", "Gorton",
"Tameside", "Blackley", "City centre", "Airport", "South Manchester",
"Rochdale", "Chorlton", "Uppermill", "Castleton", "Stalybridge", "Ashton",
"Chadderton", "Bury", "Ancoats", "Whalley Range", "West Yorkshire",
"Fallowfield", "New Moston", "Denton", "Stretford", "Eccles", "Pendleton",
"Leigh", "Altrincham", "Sale", "Prestwich", "Kearsley", "Hulme", "Withington",
"Moss Side", "Milnrow", "outskirt of Manchester City Centre", "Newton Heath",
"Wythenshawe", "Mancunian Way", "M60", "A6", "Droylesden", "M56", "Timperley",
"Higher Ince", "Clayton", "Higher Blackley", "Lowton", "Droylsden",
"Partington", "Cheetham Hill", "Benchill", "Longsight", "Didsbury",
"Westhoughton"])
# Social categories from Yuwei
soccat = ["man", "woman", "men", "women", "youth", "teenager", "elderly",
"patient", "taxi driver", "neighbour", "male", "tenant", "landlord", "child",
"children", "immigrant", "female", "workmen", "boy", "girl", "foster parents",
"next of kin"]
for i in range(100):
    soccat.append("%d-year-old" % i)
    soccat.append("%d-years-old" % i)
# Types of social locations from Yuwei
socloc = ["car park", "park", "pub", "club", "shop", "premises", "bus stop",
"property", "credit card", "supermarket", "garden", "phone box", "theatre",
"toilet", "building site", "Crown court", "hard shoulder", "telephone kiosk",
"hotel", "restaurant", "cafe", "petrol station", "bank", "school",
"university"]
extras =   "placename": placenames, "soccat": soccat, "socloc": socloc  
# Normalise keyword lists
for k, v in extras.iteritems():
    # Remove duplicates
    v = list(set(v))
    # Sort by length
    v.sort(key=lambda x:len(x), reverse=True)
# Add keywords
def add_categories(tweet):
    text = tweet["text"].lower()
    for field, categories in extras.iteritems():
        for cat in categories:
            if cat.lower() in text:
                tweet[field] = cat
                break
    return tweet
# Read all the twits
records = (add_categories(x) for x in simplejson.load(sys.stdin))
simplejson.dump(list(records), sys.stdout)
All these scripts form a nice processing chain: each script takes a list of JSON records, adds some bit and passes it on. In order to see what we have so far, here is a simple script to convert the JSON twits to CSV so they can be viewed in a spreadsheet:
#!/usr/bin/python
# Convert the JSON twits to CSV
# (C) 2010 Enrico Zini <enrico@enricozini.org>
# License: WTFPL version 2 (http://sam.zoy.org/wtfpl/)
import simplejson
import sys
import csv
rows = ["id", "created_at", "text", "keywords", "placename"]
writer = csv.writer(sys.stdout)
for rec in simplejson.load(sys.stdin):
    rec["keywords"] = " ".join(rec["keywords"])
    rec["placename"] = rec.get("placename", "")
    writer.writerow([rec[row] for row in rows])
At this point we were coming up with lots of questions: "were there more reports on women or men?", "which place had most incidents?", "what were the incidents involving animals?"... Time to bring Xapian into play. This script reads all the JSON tweets and builds a Xapian index with them:
#!/usr/bin/python
# toxapian - Index JSON tweets in Xapian
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import simplejson
import sys
import os, os.path
import xapian
DBNAME = sys.argv[1]
db = xapian.WritableDatabase(DBNAME, xapian.DB_CREATE_OR_OPEN)
stemmer = xapian.Stem("english")
indexer = xapian.TermGenerator()
indexer.set_stemmer(stemmer)
indexer.set_database(db)
data = simplejson.load(sys.stdin)
for rec in data:
    doc = xapian.Document()
    doc.set_data(str(rec["id"]))
    indexer.set_document(doc)
    indexer.index_text_without_positions(rec["text"])
    # Index categories as categories
    if "placename" in rec:
        doc.add_boolean_term("XP" + rec["placename"].lower())
    if "soccat" in rec:
        doc.add_boolean_term("XS" + rec["soccat"].lower())
    if "socloc" in rec:
        doc.add_boolean_term("XL" + rec["socloc"].lower())
    db.add_document(doc)
db.flush()
# Also save the whole dataset so we know where to find it later if we want to
# show the details of an entry
simplejson.dump(data, open(os.path.join(DBNAME, "all.json"), "w"))
And this is a simple command line tool to query to the database:
#!/usr/bin/python
# xgrep - Command line tool to query the GMP24 tweet Xapian database
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import simplejson
import sys
import os, os.path
import xapian
DBNAME = sys.argv[1]
db = xapian.Database(DBNAME)
stem = xapian.Stem("english")
qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
qp.add_boolean_prefix("place", "XP")
qp.add_boolean_prefix("soc", "XS")
qp.add_boolean_prefix("loc", "XL")
query = qp.parse_query(sys.argv[2],
    xapian.QueryParser.FLAG_BOOLEAN  
    xapian.QueryParser.FLAG_LOVEHATE  
    xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE  
    xapian.QueryParser.FLAG_WILDCARD  
    xapian.QueryParser.FLAG_PURE_NOT  
    xapian.QueryParser.FLAG_SPELLING_CORRECTION  
    xapian.QueryParser.FLAG_AUTO_SYNONYMS)
enquire = xapian.Enquire(db)
enquire.set_query(query)
count = 40
matches = enquire.get_mset(0, count)
estimated = matches.get_matches_estimated()
print "%d/%d results" % (matches.size(), estimated)
data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))
for m in matches:
    rec = data[m.document.get_data()]
    print rec["text"]
print "%d/%d results" % (matches.size(), matches.get_matches_estimated())
total = db.get_doccount()
estimated = matches.get_matches_estimated()
print "%d results over %d documents, %d%%" % (estimated, total, estimated * 100 / total)
Neat! Now that we have a proper index that supports all sort of cool things, like stemming, tag clouds, full text search with complex queries, lookup of similar documents, suggest keywords and so on, it was just fair to put together a web service to share it with other people at the event. It helped that I had already written similar code for apt-xapian-index and dde before. Here is the server, quickly built on bottle. The very last line starts the server and it is where you can configure the listening interface and port.
#!/usr/bin/python
# xserve - Make the GMP24 tweet Xapian database available on the web
#
# Copyright (C) 2010  Enrico Zini <enrico@enricozini.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
import bottle
from bottle import route, post
from cStringIO import StringIO
import cPickle as pickle
import simplejson
import sys
import os, os.path
import xapian
import urllib
import math
bottle.debug(True)
DBNAME = sys.argv[1]
QUERYLOG = os.path.join(DBNAME, "queries.txt")
data = dict((str(x["id"]), x) for x in simplejson.load(open(os.path.join(DBNAME, "all.json"))))
prefixes =   "place": "XP", "soc": "XS", "loc": "XL"  
prefix_desc =   "place": "Place name", "soc": "Social category", "loc": "Social location"  
db = xapian.Database(DBNAME)
stem = xapian.Stem("english")
qp = xapian.QueryParser()
qp.set_default_op(xapian.Query.OP_AND)
qp.set_database(db)
qp.set_stemmer(stem)
qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
for k, v in prefixes.iteritems():
    qp.add_boolean_prefix(k, v)
def make_query(qstring):
    return qp.parse_query(qstring,
        xapian.QueryParser.FLAG_BOOLEAN  
        xapian.QueryParser.FLAG_LOVEHATE  
        xapian.QueryParser.FLAG_BOOLEAN_ANY_CASE  
        xapian.QueryParser.FLAG_WILDCARD  
        xapian.QueryParser.FLAG_PURE_NOT  
        xapian.QueryParser.FLAG_SPELLING_CORRECTION  
        xapian.QueryParser.FLAG_AUTO_SYNONYMS)
@route("/")
def index():
    query = urllib.unquote_plus(bottle.request.GET.get("q", ""))
    out = StringIO()
    print >>out, '''
<html>
<head>
<title>Query</title>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
<script type="text/javascript">
$(function() 
    $("#queryfield")[0].focus()
 )
</script>
</head>
<body>
<h1>Search</h1>
<form method="POST" action="/query">
Keywords: <input type="text" name="query" value="%s" id="queryfield">
<input type="submit">
<a href="http://xapian.org/docs/queryparser.html">Help</a>
</form>''' % query
    print >>out, '''
<p>Example: "car place:wigan"</p>

<p>Available prefixes:</p>

<ul>
'''
    for pfx in prefixes.keys():
        print >>out, "<li><a href='/catinfo/%s'>%s - %s</a></li>" % (pfx, pfx, prefix_desc[pfx])
    print >>out, '''
</ul>
'''
    oldqueries = []
    if os.path.exists(QUERYLOG):
        total = db.get_doccount()
        fd = open(QUERYLOG, "r")
        while True:
            try:
                q = pickle.load(fd)
            except EOFError:
                break
            oldqueries.append(q)
        fd.close()
        def print_query(q):
            count = q["count"]
            print >>out, "<li><a href='/query?query=%s'>%s (%d/%d %.2f%%)</a></li>" % (urllib.quote_plus(q["q"]), q["q"], count, total, count * 100.0 / total)
        print >>out, "<p>Last 10 queries:</p><ul>"
        for q in oldqueries[:-10:-1]:
            print_query(q)
        print >>out, "</ul>"
        # Remove duplicates
        oldqueries = dict(((x["q"], x) for x in oldqueries)).values()
        print >>out, "<table>"
        print >>out, "<tr><th>10 queries with most results</th><th>10 queries with least results</th></tr>"
        print >>out, "<tr><td>"
        print >>out, "<ul>"
        oldqueries.sort(key=lambda x:x["count"], reverse=True)
        for q in oldqueries[:10]:
            print_query(q)
        print >>out, "</ul>"
        print >>out, "</td><td>"
        print >>out, "<ul>"
        nonempty = [x for x in oldqueries if x["count"] > 0]
        nonempty.sort(key=lambda x:x["count"])
        for q in nonempty[:10]:
            print_query(q)
        print >>out, "</ul>"
        print >>out, "</td></tr>"
        print >>out, "</table>"
    print >>out, '''
</body>
</html>'''
    return out.getvalue()
@route("/query")
@route("/query/")
@post("/query")
@post("/query/")
def query():
    query = bottle.request.POST.get("query", bottle.request.GET.get("query", ""))
    enquire = xapian.Enquire(db)
    enquire.set_query(make_query(query))
    count = 40
    matches = enquire.get_mset(0, count)
    estimated = matches.get_matches_estimated()
    total = db.get_doccount()
    out = StringIO()
    print >>out, '''
<html>
<head><title>Results</title></head>
<body>
<h1>Results for "<b>%s</b>"</h1>
''' % query
    if estimated == 0:
        print >>out, "No results found."
    else:
        # Give as results the first 30 documents; also use them as the key
        # ones to use to compute relevant terms
        rset = xapian.RSet()
        for m in enquire.get_mset(0, 30):
            rset.add_document(m.document.get_docid())
        # Compute the tag cloud
        class NonTagFilter(xapian.ExpandDecider):
            def __call__(self, term):
                return not term[0].isupper() and not term[0].isdigit()
        cloud = []
        maxscore = None
        for res in enquire.get_eset(40, rset, NonTagFilter()):
            # Normalise the score in the interval [0, 1]
            weight = math.log(res.weight)
            if maxscore == None: maxscore = weight
            tag = res.term
            cloud.append([tag, float(weight) / maxscore])
        max_weight = cloud[0][1]
        min_weight = cloud[-1][1]
        cloud.sort(key=lambda x:x[0])
        def mklink(query, term):
            return "/query?query=%s" % urllib.quote_plus(query + " and " + term)
        print >>out, "<h2>Tag cloud</h2>"
        print >>out, "<blockquote>"
        for term, weight in cloud:
            size = 100 + 100.0 * (weight - min_weight) / (max_weight - min_weight)
            print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(query, term), size, term)
        print >>out, "</blockquote>"
        print >>out, "<h2>Results</h2>"
        print >>out, "<p><a href='/'>Search again</a></p>"
        print >>out, "<p>%d results over %d documents, %.2f%%</p>" % (estimated, total, estimated * 100.0 / total)
        print >>out, "<p>%d/%d results</p>" % (matches.size(), estimated)
        print >>out, "<ul>"
        for m in matches:
            rec = data[m.document.get_data()]
            print >>out, "<li><a href='/item/%s'>%s</a></li>" % (rec["id"], rec["text"])
        print >>out, "</ul>"
        fd = open(QUERYLOG, "a")
        qinfo = dict(q=query, count=estimated)
        pickle.dump(qinfo, fd)
        fd.close()
    print >>out, '''
<a href="/">Search again</a>

</body>
</html>'''
    return out.getvalue()
@route("/item/:id")
@route("/item/:id/")
def show(id):
    rec = data[id]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Result %s</title></head>
<body>
<h1>Raw JSON record for twit %s</h1>
<pre>''' % (rec["id"], rec["id"])
    print >>out, simplejson.dumps(rec, indent=" ")
    print >>out, '''
</pre>
</body>
</html>'''
    return out.getvalue()
@route("/catinfo/:name")
@route("/catinfo/:name/")
def catinfo(name):
    prefix = prefixes[name]
    out = StringIO()
    print >>out, '''
<html>
<head><title>Values for %s</title></head>
<body>
''' % name
    terms = [(x.term[len(prefix):], db.get_termfreq(x.term)) for x in db.allterms(prefix)]
    terms.sort(key=lambda x:x[1], reverse=True)
    freq_min = terms[0][1]
    freq_max = terms[-1][1]
    def mklink(name, term):
        return "/query?query=%s" % urllib.quote_plus(name + ":" + term)
    # Build tag cloud
    print >>out, "<h1>Tag cloud</h1>"
    print >>out, "<blockquote>"
    for term, freq in sorted(terms[:20], key=lambda x:x[0]):
        size = 100 + 100.0 * (freq - freq_min) / (freq_max - freq_min)
        print >>out, "<a href='%s' style='font-size:%d%%; color:brown;'>%s</a>" % (mklink(name, term), size, term)
    print >>out, "</blockquote>"
    print >>out, "<h1>All terms</h1>"
    print >>out, "<table>"
    print >>out, "<tr><th>Occurrences</th><th>Name</th></tr>"
    for term, freq in terms:
        print >>out, "<tr><td>%d</td><td><a href='/query?query=%s'>%s</a></td></tr>" % (freq, urllib.quote_plus(name + ":" + term), term)
    print >>out, "</table>"
    print >>out, '''
</body>
</html>'''
    return out.getvalue()
# Change here for bind host and port
bottle.run(host="0.0.0.0", port=8024)
...and then we presented our work and ended up winning the contest. This was the story of how we wrote this set of award winning code.

Joey Hess: the carrot that time forgot

found in an overlooked corner of my raised bed

Lior Kaplan: Recognizing apt-get output in a youtube movie

It seems that during a daily upgrade for packages in Debian unstable gave me the ability to recognize the apt-get output with just glimpsing on screen. That was exactly what happened to me when I watched Bits in pieces on you tube. In one of the scenes there s a text running up on the screen, and new rows are added at the bottom. After about two seconds after the scene I had to jump back and pause to verify if I spotted the output correctly. And it seems I was right. You re welcome to watch the movie, and notice the output at 3:06. I wonder what else does Debian puts in my subconscious
Filed under: Debian GNU/Linux

Wouter Verhelst: Samba

Warning: rant ahead Someone decided that doing a Windows-specific protocol would be a good idea. Whoever this person is, I hope he rots in hell. But that's not what this post is about. Someone else decided that doing a free software implementation of that Windows-specific protocol would be a good idea. Hence samba was born. And all was good. For the next few years, Samba kept implementing more and more features of that particular Windows-specific protocol, which kept getting changed and renamed over the years. Originally, samba could just export files, and you could specify whether users could or could not write to your files. Perhaps it already had passwords at that point not sure. Later, usernames were added, and the ability to join a Windows domain, and then another type of Windows domain, and eventually a number of other differently deficient difficulties. As these things were implemented, the ways of configuring samba to properly do all these things was made more and more complex, to the point where it now no longer seems possible to install samba without major repercussions on the 'plain' Linux installation. A bit like having an apache requiring you to add apache-specific usernames through an apache-specific NSS module (winbind anyone?) just so you could export ACLs. Oh, and forget about retaining your existing usernames. Now I'm not sure whether this is related to a) me having tried winbind at some point in the past, deciding that I didn't like the effects it had, and removing it again (with as a result that it left some traces on the system), b) the system being joined to an ADS domain which might have some special requirements, or c) this particular system having a bit of a weird history that might have fucked up some of the internal state files of samba; but at any rate, it sucks. It seems to me that somehow, over the years, samba has lost the ability to just function as a minor something that just happens to be one service on a machine much like apache on my laptop and now requires one to do a complete overhaul of the entire system. That's just stupid; I am not going to migrate users to their winbind counterparts, thankyouverymuch. Stupid ******** Of course, though unlikely, it's not impossible that I've missed something. But at any rate, even if that is the case, the hoops one needs to jump through to get to the above situations are far too numerous and complex.

Next.

Previous.