Search Results: "ema"

5 November 2012

Vincent Sanders: Another NetSurf Developer Workshop

The NetSurf Developers - Bringing you alternative internet meme delivery since 2002
The weekend saw some of the NetSurf developers gather once again to do battle against the Red Queen...no, wait. I mean plan and write code.

The successful event was once again kindly hosted by Collabora at their Cambridge office, the facilities remain as good as at the previous event and provided a comfortable environment in which to work.

Four developers managed to attend from around the UK John-Mark Bell, Vincent Sanders, Daniel Silverstone and Rob Kendrick. The turnout was a little lower than previous events, however Michael Drake managed to assist with some bug fixes remotely.

From Friday to Sunday afternoon we each managed to put in 24 hours of work consisting of almost 90 commits changing over 100 files and removing 10,000 lines of code.

Whiteboard with the 3.0 release tasks
The main theme of the event was consolidation ready for the 3.0 release. While not a glamorous target this is a major release for the project which we want to achieve soon, this requires the completion of numerous ongoing tasks (like finally removing the dependency of libxml) and fixing as many reported bugs as possible.

The Major things we achieved were:

Once the final usage of libXML is removed (from the hotlist code) NetSurf will become substantially smaller on several platforms as we will not have two separate parser implementations linked.

One big feature of the 3.0 release which was not originally on the roadmap is that of JavaScript. We initially intend to provide two builds of with and without the JavaScript interpreter compiled into NetSurf. This allows those on very small systems to continue to run NetSurf and those users who want a richer web experience to enjoy it.

Although JavaScript is a major new feature which is frequently requested the initial implementation will be incomplete. To complete the feature several thousand JavaScript to DOM interface binding functions must still be written. In addition it should be noted that we have yet to add dynamic layout or extensions such as Canvas or WebGL

We had a Raspberry Pi which was used to run RISC OS so we could examine the operation on this system.

The 2.9 edition as shipped works OK but is lacking the substantial progress made in the last six months.

The CI system build was run on the system and works well so we are reassured the 3.0 edition will provide an even better browsing experience.

The event was a great success overall and we are now very close to our 3.0 release which we hope to produce before Christmas.

4 November 2012

Russ Allbery: Review: Set This House in Order

Review: Set This House in Order, by Matt Ruff
Publisher: Perennial
Copyright: 2003
Printing: 2004
ISBN: 0-06-095485-X
Format: Trade paperback
Pages: 479
A textbook case history would say that Andy Gage has disassociative identity disorder (referred to here as the more common although medically inaccurate "multiple personality disorder" for reasons of clarity and to match casual speech, according to the author). However, that's a completely inadequate description. A better description would be that Andy Gage's body is inhabited by a large set of different personalities, who take turns being in control. The original Andy Gage is dead, shattered by horrific childhood abuse. Aaron became the dominant personality and gave structure to the internal geography of the mind they all share, with the help of extensive counselling. But Aaron was exhausted by being the public face and maintaining control of the shared body. Hence, Andrew. Andrew Gage is the first-person narrator of half of this book. He is twenty-eight, the same age as the body, but he's only existed for two years. He considers Aaron his father, since Aaron called him out of the lake inside their internal metal geography. His purpose is to run the body and be the public face of the family. He is by turns capable and naive, shy on personal experience, but able to get the help of the others. The one heard from most frequently is Adam, a rude and mischevious teenager who also has an uncanny ability to read other people, and who frequently watches through the body's eyes from what the family calls the "pulpit" and gives Andrew advice. Andrew works for Julie Sivik, a woman he met the day he quit his job at a big-box store because he couldn't explain his behavior to his boss without discussing being a multiple. Julie is a wholly impractical, slightly unethical, well-meaning force of chaos who serves as the thread that draws the plot of the book together. She's running a startup on ingenuity and borrowed money that is attempting to create a comprehensive and customizable virtual reality world. She's the one who finds Penny and hires her, in part for her skills, and in part to push Penny and Andrew together. Penny Driver has the same condition as Andrew, but Penny is only vaguely aware of it. The world for her is full of blackouts, lost time, waking up in strange and sometimes horrific situations, and attempting to structure her life based on notes left for herself that she didn't write. When she meets Andrew, she's barely making it through life; thankfully, one of her personalities is remarkably good at computers, and that has been enough to make a living despite the chaos. But Julie is the unintentional catalyst for her losing that job, the best job she's ever had, leaving her with no real alternative but to take the job with Julie's company. There are a few things that are important to say up-front about this book. First, neither I nor the author are actually multiples (or, put another way, neither of us have DID), and writing about atypical mental states is fraught with peril. It's very easy to misrepresent other people's experiences, and multiple personalities in particular are frequent material for dramas of all sorts. There's something people find inherently fascinating about the idea, and it's easy for that fascination to run away with reality and distort the lived experience. I recommend reading Ruff's FAQ about the book, which spells out what his goals are and what approach he took. I did some quick searches and didn't uncover a lot of anger or dislike of the book among either people who have DID or psychiatrists, and it seems true to the descriptions of internal reality I've heard from people who are multiples. But despite being more respectful of real experience than most fictional treatments of DID, this is still a work of fiction written by someone without first-hand experience, and it's worth keeping that in mind. Second, DID is an intensely controversial diagnosis among psychiatrists. There are, generally speaking, two major controversies. One is over whether or not DID can actually be caused by traumatic experiences such as childhood abuse, or whether it is a creation of psychiatrists, a mental model that they've imposed on their patients or convinced their patients to believe. Set This House in Order comes down firmly on the first side of that controversy: that DID can be created by abuse and is not a creation of psychotherapy. (So, I should note, does every multiple that I've ever talked to or read.) Second, there is significant controversy over treatment, with some psychiatrists believing that integration of the personalities into one is the only true treatment, and others believing that multiples can remain stable and functional as multiple personalities. Set This House in Order leans strongly towards the view that multiples can be stable as multiples and that while integration may be appropriate for some, it's not for all. Both of these viewpoints are presented very convincingly by the narrative in the book, and both are consistent with the beliefs of a lot of multiples, but it's worth remembering that these are active and often quite heated controversies and a novel is not evidence. It's good to be aware of both of those points, and to read Put This House in Order as something of a what-if, where much more is clear about DID than is actually clear in our world. But once one has done that, one is in for an excellent book. I loved this book from start to finish. (Well, okay, I felt some surprise at the nature of the ending, but it grew on me after further consideration.) It's simply a brilliant novel, with a little bit of everything: very good characterization, messy love affairs, the satisfying tension of someone overcoming great odds to put themselves together with the help of friends, a startlingly original perspective of the world, some excellent surprises, and some absolutely wonderful characters. Andrew is a good choice of narrator: he brings an earnest calm to a story that could otherwise be quite chaotic and provides a great vantage point for the reader. And I found Penny (and particularly some of her other souls) impossible not to like. Set This House in Order is a tense book in places, and it's a book where people's lives break down before they can get better. It's also a book about rather horrific abuse, and while nearly all of the specifics happen off-stage, imagination fills in some nasty details. But Ruff manages to write it with a structure and with enough calm assurance that I felt just reassured enough without taking away the tension. That's a tricky balance; few books do it better. And the supporting characters are a delight. One might think, with all the complexity happening for both Andrew and Penny, that Ruff wouldn't have room in the narrative for other characters, but one would be wrong. There are very few saints and demons here, just a lot of complex people, with their own problems and their own perspectives, who are generally trying to do the best they can from day to day. There are some clear-cut villains in the past Ruff wisely makes no attempt to add shades of grey to abusers but even there one can see some clear signs of untreated severe mental illness and chaotic lives rather than mustache-twirling pre-meditated villainy. This is, in short, a book full of people, in all their messy complications, which is what makes it so delightful. I'm not going to say anything more about the plot, or even the various personalities, since discovering the direction of the book was so much of the fun. I will say that, despite the subject matter and some of the events, the book is surprisingly gentle. One of the things about it that I loved the most was the emotional tone Ruff gave to being a multiple: the sense of a squabbling but tight-knit family looking out for each other. I can't judge the reality of this impression, but it makes for a wonderful story. Highly recommended. The best book I've read so far this year. Rating: 10 out of 10

Paul Wise: Remembering the dead

Last week was el D a de los Muertos (the Day of the Dead), a Mexican holiday where people gather to remember their dead. Today I would like to remember <predator>. pred was a core member of the cat@lyst collective, a group of folks providing computers, Internet and other tech resources to the activists of Sydney, Australia. I knew pred as cat@lyst's hardware geek, he was always scrounging computers from roadsides, coaxing them back to life and setting them to work supporting cat@lyst. I only knew pred through email and just missed out on meeting him due to the cancer that eventually killed him. pred wrote extensively during his life and of his experiences while dying of cancer. After he died, some friends compiled some of his writing into a book, simply named pred.txt. His full writings also happen to have been archived on the net. Through the book and his writings I came to learn that he was much more than just a hardware geek and gain some more insight into who he was. pred was a geek, activist, anarchist, urban explorer, scientist in the area of molecular genetics, cancer victim and all-around awesome dude. I still miss him.

2 November 2012

Neil Williams: Introducing pyBit - Buildd Integration Toolkit

pyBit - cross-platform package building using AMQP


Message queues provide a simple way to create a distributed, cross-platform buildd toolkit to build packages using a collection of buildds, direct from various VCS clients. pyBit is intended to support rapidly evolving software collections and can support multiple VCS frontends and multiple build backends. Cross building is expected to be supported for some backends. The initial backend uses dpkg for Debian with subversion providing the source and sbuild doing the actual build.

pyBit includes support for cancelling selected builds and using multiple buildd clients per architecture, per platform and per suite.

Hooks are available or in development for subversion and git, other VCS hooks can be added. A RESTful web API provides live build reports and can generate build jobs for specific packages using particular VCS branches on selected architectures to support re-building packages at any point in the development process. Build history is stored using postgresql.

Other buildd systems can rebuild long lists of packages or build lots of binary packages from relatively slow moving source packages. PyBit exists to handle much more rapid software development across a wide range of platforms, VCS inputs and architectures. Buildd clients which are under-used can be tasked with building multiple suites or adding cross-build support. Buildd clients which are over-utilised are easily identified and adding new machines to an existing architecture / platform / suite pool should be easy. Hook activation automatically cancels any ongoing build for the same architecture, platform and suite to avoid wasting time on an interim version.

The emphasis in pyBit is to have fast builds with redundant clients, reliable reporting using a flexible and intuitive frontend. To this end, there is no need for a source package to be uploaded. Depending on the VCS hook in use, builds can happen every time a particular file is changed (e.g. debian/changelog for a native Debian package) or at every push (for a distributed VCS) or whatever is appropriate for a particular software collection. Builds are checked for available build-dependencies and packages re-queued if the build-dependencies are not yet met.

In the longer term, it may well be possible to use more than one server / database combination to support more builds and more platforms.

So far, we've got to a working model and tagged 0.1.0 as the first downloadable release. There is a lot more to do, especially adding more documentation, more VCS hooks, support for more VCS methods on the buildd clients and more buildd client scripts for platforms other than Debian. (The git hook and git source client are expected to let pyBit be self-buildable but a certain amount of configuration will be required for the server and each client which makes it not-quite self-hosting.)

pyBit concentrates on preparing collections of binary packages which can be used to build others, rather than trying to rebuild everything every time - this allows more rapid upstream software development and encourages modular, re-usable software. pyBit will also support rebuilding specific versions, architectures, suites or platforms via the RESTful web API. Access to this frontend can be controlled through any of the standard methods.

Components are loosely coupled via JSON encoded messages sent using rabbitMQ and curl. A new client can be added at any time and it will simply pick up buildd jobs from the relevant queue.

Development has now moved to Github and if anyone wants to look at new clients and new hooks, just contact the team by email or via IRC (#pybit on irc.oftc.net). Current development is based on Debian Squeeze 6.0.6 with backports and also on Debian Wheezy - more testing is welcome. Patches are also very welcome, pyBit is licensed under GPL2+.

There is no intrinsic reason why pyBit could not support any buildd platform capable of taking source from a known location and building a set of binaries. Packages are added to the queues whenever the hook is activated, so adding new packages to the collection is simply a case of triggering the existing hook.

Another interesting challenge for pyBit would be to trigger a hook on a new source code repository and just let it work through every package until everything is built. That probably won't work with the current 0.1.0 code but if that is what you'd like to work on, join the team.

Erich Schubert: DBSCAN and OPTICS clustering

DBSCAN [wikipedia] and OPTICS [wikipedia] are two of the most well-known density based clustering algorithms. You can read more on them in the Wikipedia articles linked above.
An interesting property of density based clustering is that these algorithms do not assume clusters to have a particular shape. Furthermore, the algorithms allow "noise" objects that do not belong to any of the clusters. K-means for examples partitions the data space in Voronoi cells (some people claim it produces spherical clusters - that is incorrect). See Wikipedia for the true shape of K-means clusters and an example that canot be clustered by K-means. Internal measures for cluster evaluation also usually assume the clusters to be well-separated spheres (and do not allow noise/outlier objects) - not surprisingly, as we tend to experiment with artificial data generated by a number of Gaussian distributions.
The key parameter to DBSCAN and OPTICS is the "minPts" parameter. It roughly controls the minimum size of a cluster. If you set it too low, everything will become clusters (OPTICS with minPts=2 degenerates to a type of single link clustering). If you set it too high, at some point there won't be any clusters anymore, only noise. However, the parameter usually is not hard to choose. If you for example expect clusters to typically have 100 objects, I'd start with a value of 10 or 20. If your clusters are expected to have 10000 objects, then maybe start experimenting with 500.
The more difficult parameter for DBSCAN is the radius. In some cases, it will be very obvious. Say you are clustering users on a map. Then you might know that a good radius is 1 km. Or 10 km. Whatever makes sense for your particular application. In other cases, the parameter will not be obvious, or you might need multiple values. That is when OPTICS comes into play.
OPTICS is based on a very clever idea: instead of fixing MinPts and the Radius, we only fix minpts, and plot the radius at which an object would be considered dense by DBSCAN. In order to sort the objects on this plot, we process them in a priority heap, so that nearby objects are nearby in the plot. this image on Wikipedia shows an example for such a plot.
OPTICS comes at a cost compared to DBSCAN. Largely because of the priority heap, but also as the nearest neighbor queries are more complicated than the radius queries of DBSCAN. So it will be slower, but you no longer need to set the parameter epsilon. However, OPTICS won't produce a strict partitioning. Primarily it produces this plot, and in many situations you will actually want to visually inspect the plot. There are some methods to extract a hierarchical partitioning out of this plot, based on detecting "steep" areas.
The open source ELKI data mining framework (package "elki" in Debian and Ubuntu) has a very fast and flexible implementation of both algorithms. I've benchmarked this against GNU R ("fpc" package") and Weka, and the difference is enormous. ELKI without index support runs in roughly 11 minutes, with index down to 2 minutes for DBSCAN and 3 minutes for OPTICS. Weka takes 11 hours and GNU R/fpc takes 100 minutes (DBSCAN, no OPTICS available). And the implementation of OPTICS in Weka is not even complete (it does not support proper cluster extraction from the plot). Many of the other OPTICS implementations you can find with Google (e.g. in Python or MATLAB) seem to be based on this Weka version ...
ELKI is open source. So if you want to peek at the code, here are direct links: DBSCAN.java, OPTICS.java.
Some part of the code may be a bit confusing at first. The "Parameterizer" classes serve the purpose of allowing automatic UI generation, for example. So there is quite a bit of meta code involved.
Plus, ELKI is quite extensively optimized. For example, it does not use Java Collections much anymore. Java Iterators, for example, require returning an object on next();. The C++ style iterators used by ELKI can have multiple values, and primitive values.
for(DBIDIter id = relation.iterDBIDs(); id.valid(); id.advance())
is a typical for loop in ELKI, iterating over all objects of a relation, but the whole loop requires creating (and GC'ing) a single object. And actually, this is as literal as a for loop can get.
ModifiableDBIDs processedIDs = DBIDUtil.newHashSet(size);
is another example. Essentially, this is like a HashSet<DBID>. Except that it is a lot faster, because the object IDs do not need to live a Java objects, but can internally be stored more efficiently (the only currently available implementation of the DBID layer uses primitive integers).
Java advocates always accuse you of premature optimization when you avoid creating objects for primitives. Yet, in all my benchmarking, I have seen this continuously to have a major impact how many objects you allocate. At least when it is inside a loop that is heavily used. Java collections with boxed primitives just eat a lot of memory, and the memory management overhead does often make a huge difference. Which is why libraries such as Trove (which ELKI uses a lot) exist. Because memory usage does make a difference.
(Avoiding boxing/unboxing systematically in ELKI yielded approximately a 4x speedup. But obviously, ELKI involves a lot of numerical computations.)

1 November 2012

Michal &#268;iha&#345;: Weblate brings source strings checks and review

Weblate 1.3 will bring several new features. One of recently implemented parts is source strings review. Translators quite often find some problems with original string. Be it typo, wrong wording or simply the message is hard to understand. In Weblate 1.3 they will have an option to push feedback to developers. The feedback is stored within Weblate and optionally can be sent to configured email. In addition to this, there are now checks for source strings as well, which might help you to improve quality of them. New checks are of course covered in the documentation. You can already try current Weblate on http://demo.weblate.org/, though there are still some things which need polishing.

Filed under: English Weblate 0 comments Flattr this!

Antoine Beaupr : My short computing history

This is a long story. I wanted to document which tools I used on my desktop, but then realized it was difficult to do this without an historical context. This therefore, became a history of my computer usage and programming habits.

Before the desktop: vic 20 and apple My first contacts with computers were when my uncle showed me his Apple computers at his home. He was working at UQAM and had access to all sorts of crazy computers. My memory is vague, but I believe he showed me a turtle moving around the screen, which I suspect was Logo, and some rabbit game which I was quite fond of. Later, I hooked up with a Vic-20 (similar to a Commodore 64) and typed in my first programs. Back then, you would painstakingly transcribe the program from a book into the screen, type RUN at the end and really hope that you didn't make a typo. You could also read and write data off audio tape with an external tape drive, which was really amazing. To put things into perspective, the Vic-20 would plug into your television for a display, and the computer was "in the keyboard". Old school shit. The language was BASIC, which was awful. Later, we got luxurious and had the privilege of a Macintosh Plus computer. 3000$ plus the printer, that was quite something. Still, that modern system could run Word, had games, and a 9" black and white screen. But the best part, and where I really started to learn to program, is with Hypercard, the crazy software written by Bill Atkinson that became the inspiration for wikis, the Myst computer game and, to a certain extent, popularized the concept of hyperlinking, which is the basic principle of the World Wide Web.

The early years: afterstep and FreeBSD But my real computer life all started with FreeBSD 2.2.x (probably 2.2.5), around 1997, when I installed my first FLOSS operating system. This was just amazing: source code for everything, down to the compilers themselves, I was amazed, since I was coming from a Mac (still had that Mac Plus lying around) and Windows (95!) world. Back then, X11 (XFree86, it was called) was a fantasy. Something I would compile with the FreeBSD ports system and sometimes would be able to run. Window managers were some weird things I wasn't sure how to use, but I figured out twm (yuck!) but was still amazed. At the university, we had SGI workstations (remember the Indigo machines?) on which I started setting up the Afterstep window manager, which I used for a long time. The web browser was Netscape Communicator 4, and lo and behold, it also did email, although I mostly used Pine over a modem terminal to the university. Internet access at home was spotty and was dependent on students compiling discreetly copies of slirp which were allowing us, in a really clunky way, to run a PPP session over those terminal lines. That actually got my account suspended because I messed up my sendmail configuration and was sending emails that appeared to be coming as , which freaked out the sysadmins that be over there. The hardware was a Pentium 166Mhz with 32MB of ram and 20GB of disk space, which was a huge improvement over the 1MB of ram and 20MB of (external!) hard disk that the Mac Plus was proud of. By then, I started programming in more languages: I learned the basics of shell scripting, C programming and I had learned Pascal in a college class.

The real freedom: Debian and... enlightenment? I then switched to Debian, thanks to my friend scyrma, which showed me the power of dselect which, for a FreeBSD user, was really impressive. Imagine that: you could just install software and all dependencies without compiling anything! ;) I don't exactly remember when that was, but it was probably Woody. I probably still used afterstep, but at that time, changing window managers was like changing t-shirts and we would waste countless hours fiddling around with borders and background images and other silly things. I remember running Enlightenment and writing software for it, for example, but not much else. Back then, email was still over pine, but a new web browser was rearing its ugly head, Mozilla, and was quickly trimmed down to Phoenix, shortly lived as Mozilla Firebird, only to be finally renamed Mozilla Firefox, and was unleashed as a stable release to an unsuspecting Internet Explorer and Microsoft-dominated world. Throughout this chaos, I also used various Mozilla derivatives like Galeon since all of this was pretty unstable and heavy for my poor computers. This was probably around 2002 to 2005. By then, I had formal classes on programming languages, was forced to learn Java, discovered C++, and learned more C as the university assumed that you knew that language in the data structures and algorithm class, which some students really didn't find funny at all. I also learned some Prolog and Haskell back then. Through my experiences with free software, I discovered various eclectic programming languages like TCL, and kept an eye out for other languages like lisp (thanks to Emacs), Forth (thanks to the FreeBSD boot loader), HTML/CSS/Javascript (no thanks to the web). There are probably others I am forgetting here, voluntarily or not.

The years of the desktops: rox, sawfish and gnome Another stable increment on my desktop was the use of the sawfish window manager and the rox desktop, mostly because of the awesome Rox file manager that had no equal in the rest of the free software world. Gnome was an atrocity and KDE even more so: bloated and unstable. Eventually, however, KDE and Gnome not only matured, but simply took over the whole of the graphical environment psyche of the free world. People nowadays don't even know what a window manager is and assume that you now have a "desktop environment", which, for me, is a real desk with a bunch of papers, a radio transceiver and maybe a computer with a screen lying around. So I did use KDE, and then Gnome. I switched from KDE to Gnome because I felt that Gnome left me more space to run my own stuff without starting crazy daemons to preload everything. At this point, Firefox was the new thing and basically stable enough for daily use. This lasted until around 2009, I guess. On the programming side, I had picked up Perl because I had to learn it to work at Ericsson (don't ask) and learned PHP to pay my rent, which I consider one of the biggest professional mistakes I have made so far in that story. It is around that time that Koumbit was founded by me and other friends.

The tiles fall on the gnomes: awesome desktop and sysadmin programming At some point, friends (mostly comar and millette) introduced me to wmii as the best thing since sliced bread. I banged my head against it with great pains, but I really enjoyed the hands off approach to window management. Tiling window managers really got a hold on me there, and in fact we can say on the whole desktop, with recent Gnome 3 taking up some tiling concepts itself. I have since then switched to the Awesome Window Manager which is probably the worst name you can use for software (try googling for "Awesome crash" to get an idea). I have switched from Firefox to Chromium, after having repeated crashes from Firefox on a site that Chrome would load fine. I was expecting Chrome to crash only one tab, because of its design, and it turned out to not crash at all, and I have since given up on Firefox totally. I have also given up on desktop environments. While I have given them a chance in quick setups and still use them when forced to, I find them too bloated and single-minded for my needs. I want control over my desktop, and those tools work very hard to remove control and hide it behind user interfaces that treat me like a brain-dead dog. Finally, I have kept on programming in PHP for work, because that's what we do over there (unfortunately). Mostly because of Drupal and Aegir, in which I was and still am deeply involved. But I am quite fond of Python for my personnal project and still use shell scripting to get myself out of trouble or automate data processing from time to time. I have a rule though: once a program hits 100 lines of shell scripting, it needs to be rewritten, usually in Perl because the syntax is closer. During the years 2009 to 2012, I have started to work more and more on system administration however. In 2010, I became a Debian Developer and a lot of my work at Koumbit revolves around system administration, optimisation and maintenance. I work oncall and sometimes get called during the night to nurse servers (which is why I am up writing this right now).

The future of programming I have become seriously disgruntled with programming. I have seen the emergence of the web, where programmers felt it was relevant to rewrite 30 years of programs into the web framework (webmail? wtf?), and now we are seeing the same thing happen with mobile devices, the tracking devices they call phones and "pads", we should rather say. And again they are trying to make people pay for trivial software. I am tired of seeing our community reinvent the wheel every 10 years because of infatuated egos and marketing sprints. We are wasting great minds on this, a lot of energy and precious time. I would rather maintain 5 year old imperfect software than coordinate a 5 year project to rewrite that software into another imperfect software. That is what is happening right now, and it's going nowhere. So here's the current state of affairs, some perls, some tiles, but mostly system administration and less programming. Through all this time, the common pattern is that I have learned to solve problems and document my work, which are key concepts of the job of a system administrator, which is what I really am now. Somehow, that popped into my life like a natural fit. Maybe, one day, I will retire and become a shepherd in far away mountains, but I will always have in my mind the strong desire to fix shit up and make things right with the world. Maybe get involved in a project like Open Source Ecology and especially their Global Village Construction Set to make sure humankind survives all that stupidity.

Rog rio Brito: Another meme: The Geek test

I was with some spare time a few days ago and I took one of those stupid tests that are so popular in the Internet. Well, here are the results, saying that I am, indeed, a Geek, which does not surprise me that much. :)
NerdTests.com says I'm a Cool Nerd King.  Click here to take the Nerd Test, get geeky images and jokes, and write on the nerd forum!
I did expect my Math score to be higher than everything else (I'm only in the 90th percentile with respect to this), while I am high on the Computer side of things (in the 97th percentile). I guess that I should devote myself to more Mathematics and writing less code. :)

30 October 2012

Nathan Handler: Debian Screencast Challenge

Today in #debian-mentors, a brief discussion about screencasts came up. Some people felt that a series of short screencasts might be a nice addition to the existing Debian Developer documentation. After seeing the popularity of of Ubuntu's various video series over the years, I tend to agree that video documentation can be a nice way to supplement traditional text-based documentation. No, this is not designed to replace the existing documentation; it is merely designed to provide an alternative method for people to learn the material. From past experience, I know that if I were to start making screencasts by myself, I would end up making one or two before getting bored and moving on to a new project. That is why I am posing a challenge to the Debian community: I will make one screencast for every screencast made by someone in the Debian community. These screencasts can and should be short (only a couple of minutes in length) and focus on accomplishing one small task or explaining a single concept related to Debian Development. I discovered a Debian Screencasts page on the Debian wiki that links to Ubuntu's guide to creating screencasts. While I have done some traditional documentation work in the past, I have never really tried doing screencasts. As a result, this will be a bit of a learning experience for me at first. If you are interested in taking me up on this challenge, simply send me an email (nhandler [at] ubuntu [dot] com) or poke me on IRC (nhandler@ freenode,oftc ). This will allow us to coordinate and plan the screencasts. As I mentioned earlier, I have no experience creating screencasts, so if you are familar with the Debian Development processes and think this challenge sounds interesting, feel free to contact me.

Rapha&#235;l Hertzog: Auto Mounting Windows Shares in GNOME with Gigolo and gvfs-fuse

The traditional way to mount Windows (or Samba) shares involves hardcoding the credentials in a plain-text file and some /etc/fstab entry to mount it automatically at boot time. If you don t want to store a plain-text copy of your password, you re bound to mount your shares interactively. This is clearly sub-optimal and I thought that there must be a better way to handle this in the context of a GNOME desktop (which already supports connecting to such shares in the file browser). So I looked for a solution and after a bit of googling I found one. Start by installing a few packages:
$ sudo apt-get install gigolo gvfs-fuse
Launch Gigolo, setup your shares as bookmarks and mark them as Auto-Connect . During (first) connection, you will be prompted for your password and you have the possibility to store it in the GNOME keyring (and here it s encrypted, not in plain-text!). You should also configure the Gigolo preferences so that it starts minimized in the system tray (because we re going to run it on session startup):
The last step is to ensure that Gigolo is executed at the start of each GNOME session. Unfortunately this GNOME feature is no longer accessible from the control center so you have to execute gnome-session-properties manually (from a terminal or the command line accessible via Alt+F2). Click on Add to add a new startup program: You re done! The main limitation is that those shares are not real mounts, instead they are available within GNOME s virtual file system (GVFS). If you use only 100% GNOME application, then it s not a problem but otherwise it s pretty annoying. You can t cd in those shares from a terminal for example. There s a workaround though, it s called gvfs-fuse and you installed it right at the start of this HOWTO. This service hooks into GVFS and exports all the virtual filesystem(s) in a real fuse-based mount that is automatically setup in ~/.gvfs/. However for this to work, the user must be in the fuse group. So you should run something like this:
$ sudo adduser $USER fuse
By the way, I haven t found a way to use a non-hidden directory so if you want this directory to be more visible, I suggest that you create a symlink pointing to it.
Do you want to read more HOWTO like this one? Click here to subscribe to my free newsletter, you can opt to receive future articles by email.

No comment Liked this article? Click here. My blog is Flattr-enabled.

Craig Small: procps-ng 3.3.5

Are you using procps 3.3.4? My suggestion is you don t! An API change slipped through which mean if you tried to run procps 3.3.5 on libproc 3.3.4 it did strange things. One of those strange things is ps ax crashes. Top looked rather interesting as well. This in turn upsets the boot sequence around mounting drives so you get a horrible mess. The problem is the testsuite didn t catch it because we re not testing the API yet and the binaries use the libtool hack to use the library in the build directory. Lucky for me I only updated manually the binaries and not the library so I saw it early. procps 3.3.5 was released tonight and it has the SONAME bump ,so we are not at libprocps1 not libprocps0.

29 October 2012

Yves-Alexis Perez: Switching to OpenPGP smartcard

A friend of mine recently reminded me of the OpenPGP smartcard v2, and told me that it was perfectly able to handle 4096 bit RSA keys (provided you have GnuPG v2.0.18+). I had the opportunity to play with one a little, and notice it was super easy to use it for ssh authentication, especially since I already use gpg-agent as my ssh-agent (it should be easy to use a purely software authentication key as ssh key with GnuPG 2.1). So I decided to buy two of them and try to switch my main key (0x71ef0ba8) to it. The cards arrived this weekend, and I was able to play with it a little. I didn't log every command I typed, but it was pretty easy, in the end. What I decided to do was to use one smartcard for every day usage, and one only for key signing. So basically, I would generate three (signing, encryption, authentication) subkeys, put them on smartcard 1, then put the primary key on smartcard 2. Then erase the private parts, and only keep them on smartcards. In case it interests people, here the somehow detailed steps. Note that everywhere 'gpg' means 'gpg2' on Debian, we really need GnuPG v2 for correct smartcard handling. You'd better use gpg-agent too, although it doesn't seem mandatory.
  1. make a backup! As we're gonna play with private parts (!), it's always a good idea to have backups. And it'll be useful to have one later, in case there's a problem with the smartcards. You can do a copy of your complete ~/.gnupg folder, but I simply did:
    corsac@scapa: umask 066
    corsac@scapa: gpg -o 71ef0ba8.gpg --export-secret-keys 71ef0ba8
    
  2. Add three subkeys. Skip this is you already have subkeys (you usually already have an encryption subkey, but I wanted to switch to a new one too) --expert is needed in order to chose capabilities.
    corsac@scapa: gpg --expert --edit-key 71ef0ba8
    gpg> addkey
    Please select what kind of key you want:
       (3) DSA (sign only)
       (4) RSA (sign only)
       (5) Elgamal (encrypt only)
       (6) RSA (encrypt only)
       (7) DSA (set your own capabilities)
       (8) RSA (set your own capabilities)
    Your selection? 8
    Possible actions for a RSA key: Sign Encrypt Authenticate 
    Current allowed actions: Sign Encrypt 
       (S) Toggle the sign capability
       (E) Toggle the encrypt capability
       (A) Toggle the authenticate capability
       (Q) Finished
    Your selection? e
    Possible actions for a RSA key: Sign Encrypt Authenticate 
    Current allowed actions: Sign 
       (S) Toggle the sign capability
       (E) Toggle the encrypt capability
       (A) Toggle the authenticate capability
       (Q) Finished
    Your selection? Q
    RSA keys may be between 1024 and 4096 bits long.
    What keysize do you want? (2048) 
    Requested keysize is 2048 bits
    Please specify how long the key should be valid.
             0 = key does not expire
            = key expires in n days
          w = key expires in n weeks
          m = key expires in n months
          y = key expires in n years
    Key is valid for? (0) 1y
    Key expires at dim. 27 oct. 2013 20:38:44 CET
    Is this correct? (y/N) y
    Really create? (y/N) y
    We need to generate a lot of random bytes. It is a good idea to perform
    some other action (type on the keyboard, move the mouse, utilize the
    disks) during the prime generation; this gives the random number
    generator a better chance to gain enough entropy.
    
    Repeat this for encryption and authentication subkeys. Then save and send the key to keyservers
    gpg> save
    corsac@scapa: gpg --send-keys 71ef0ba8
    
  3. Next, we'll switch to the smartcard part. I use a Gemalto PC ExpressCard reader which is perfectly recognized under Debian. You just need few tools:
    root@scapa: ~# apt-get install pcscd scdaemon
    
    Plug the reader, insert the card, make sure it's detected:
    corsac@scapa: gpg --card-status
    Application ID ...: D2760001240102000005000016A10000
    Version ..........: 2.0
    Manufacturer .....: ZeitControl
    ...
    
    You can edit various parameter (name etc.) and change the PINs using gpg:
    corsac@scapa: gpg --change-pin
    corsac@scapa: gpg --card-edit
    
  4. Then we'll put the subkeys in the first smartcard. It might be a good idea to export again the private keys for backups.
    corsac@scapa: gpg -o 71ef0ba8.gpg --export-secret-keys 71ef0ba8
    
  5. We'll now use the keytocard gpg command to move the private parts on the smartcard:
    corsac@scapa: gpg --edit-key 71ef0ba8
    gpg> key 1 # select encryption subkey
    gpg> keytocard
    gpg> key 2 # select signature subkey
    gpg> keytocard
    gpg> key 3 # select authentication subkey
    gpg> keytocard
    gpg> save
    
    A quick check on the card now reveals that it's populated:
    corsac@scapa: gpg --card-status
    Application ID ...: D2760001240102000005000016A10000
    Version ..........: 2.0
    Manufacturer .....: ZeitControl
    Serial number ....: 000016A1
    Name of cardholder: Yves-Alexis Perez
    Language prefs ...: fr
    Sex ..............: unspecified
    URL of public key : http://www.corsac.net/71ef0ba8.asc
    Login data .......: corsac
    Signature PIN ....: forced
    Key attributes ...: 2048R 2048R 2048R
    Max. PIN lengths .: 32 32 32
    PIN retry counter : 3 3 3
    Signature counter : 7
    Signature key ....: 9745 B022 7323 81FE 9E7E  AFF5 6DDB 53F2 A675 C0A5
          created ....: 2012-10-27 11:24:07
    Encryption key....: F7E0 078F EA1A 5F23 92E0  20B3 A83A D136 D98D 0D9F
          created ....: 2012-10-27 11:27:01
    Authentication key: 8CFD D478 AB4A 16F8 F0EC  CD33 24E2 3B5C CC0E 273D
          created ....: 2012-10-17 14:29:18
    General key info..: pub  2048R/A675C0A5 2012-10-27 Yves-Alexis Perez 
    sec>  4096R/71EF0BA8  created: 2009-05-06  expires: never     
                          card-no: 0005 000016A2
    ssb   4096g/36E31BD8  created: 2009-05-06  expires: never     
    ssb>  2048R/CC0E273D  created: 2012-10-17  expires: 2013-10-27
                          card-no: 0005 000016A1
    ssb>  2048R/A675C0A5  created: 2012-10-27  expires: 2013-10-27
                          card-no: 0005 000016A1
    ssb>  2048R/D98D0D9F  created: 2012-10-27  expires: 2013-10-27
                          card-no: 0005 000016A1
    
  6. At that point, the private part is replaced by a stub in the secret keyring, so when you export them, you only export stubs which you can then use anywhere without actually giving your private key. So now is a good idea to export the subkeys so you can import them on other boxes:
    corsac@scapa: gpg -o 71ef0ba8-subkeys.gpg --export-secret-subkeys 71ef0ba8
    
    Note that only the subkeys private parts have been moved to the card, not the primary one, so you're still able to sign keys. Here, you have multiple choices. You can simply erase the private key (and later re-import the stubs) and use the offline copy made above when you need to sign another key.
  7. What I did is something else. I've put the primary key on my second OpenPGP smartcard. That way, I won't lose it, it'll be kept safely in my house, but still be on a hardware token where it won't come out. The procedure for doing is so is exactly the same as above. First take a backup (in case you didn't do it first, do it now since after the keytocard command you won't have a backup of your primary key and there'll be no way to extract it from the smartcard. Then put the new smartcard in the reader, edit the key (don't select a subkey) and run the keytocard command. After that, running gpg --export-secret-keys will export the stub and not the private part of your primary key.
In the end, it seems that everything is running fine. Only issue is that scdaemon is sometime not behaving nicely (especially after a card change or or suspend/resume cycle). I didn't yet report a bug but you might want to kill it in case it's stuck. You can also use the authentication subkey for ssh logins. When the card is inserted, the authentication subkey appears automatically (through the magic of gpg-agent):

corsac@scapa: ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EA... cardno:0005000016A1 And now you can add it to your various authorized_keys and use the smartcard for SSH.

Matthew Garrett: Ted Ts'o is a rape apologist and why this matters

(This post contains some discussion of rape and sexual assault but does not go into any specifics)

There was a brief controversy at Linux.conf.au back in 2011. The final keynote speaker gave a compelling presentation on online privacy, including some slides containing sexualised imagery. This was against the terms of the conference policies, and resulted in an apology from the conference organisers and the speaker. The situation was unfortunate but well handled, and that should have been the end of it.

Afterwards, there was some pushback on the conference mailing list. Concerns were raised about the policy being overly restrictive and the potential for it to be used to stifle speech that influential groups disagreed with. I don't agree with these arguments, but discussion of why policies have been implemented is completely natural and provides an opportunity for a community to determine what its expected standards are.

And then Ted Ts'o effectively called rape victims liars[1]. At first I assumed that this was just some sort of horrific failure to understand the implications of what he was saying, so I emailed him to check. The reply I got drew a pretty clear distinction between the case of a drunk college student raping another drunk college student in their room and the case of knifepoint rape in a dark park. You know, the difference between accidental rape and rape rape. The difference between the one any of us might have done and the one that only bad people do. Legitimate rape and the "rape" that those feminists talk about. The distinction that lets rapists convince themselves that they didn't really rape anyone because they weren't holding a knife at the time.

Ted Ts'o argues that only a small percentage of rape really counts as what people think of as rape. Ted Ts'o is a rape apologist.

There's an ongoing scandal in the UK at the moment. A well known DJ, Jimmy Savile, died last year. He grew up in a working class family, but through hard work and natural talent was one of the most significant figures in promoting pop music in the UK in the 50s and 60s, and worked in various parts of the BBC for the best part of 30 years. He spent significant amounts of time raising money for charity, and it's estimated that he raised over 40 million for various causes. Since his death, around 300 people have accused him of sexually abusing them. The BBC is desperately trying to explain why it cancelled an expose shortly before it aired. Multiple people who worked there at the time claim that everyone knew he was involved in indecent activities, but saying anything would risk both their career and the charities that depended on his fundraising. Nobody said anything, and he was allegedly free to continue his abuse.

Ted Ts'o is a significant figure in the Linux kernel community. He has expressed abhorrent beliefs that damage that community. Condemnation was limited to a mailing list with limited readership, meaning, effectively, that nobody said anything. Last week the Ada Initiative published a blog post pointing out the damage that did, and I realised that my effective silence was not only helping to alienate 50% of the population from involving themselves with Linux, it was also implicitly supporting my community leadership. I was giving the impression that I was basically fine with our community leaders telling people that it wasn't really rape if you were both drunk enough. I was increasing the chances of members of our community being sexually assaulted. Silence is endorsement. Saying nothing is not ok.

In the absence of an apology and explanation from Ted, I'll be interacting with him to the bare minimum that I'm compelled to as a result of my job. I won't be attending any Linux Foundation events he's involved in organising. If I'm running any events, I won't be inviting him. At a time when we're finally making progress in making our community more open and supportive, we don't need leaders who undermine that work. Support organisations who encourage that progress, not the people who help drag us back.

Footnotes

[1]The original archive has vanished. I've put up a copy of the relevant thread here. Throughout, Ted states that he's actually arguing against the idea that women need to be frightened of sexual assault, and not against the definition of rape. Except saying things like This one does a pretty good job of taking apart the Koss / Ms. Magazine study, which is the source for the "1 in 4" number. For example, it points out that over half of those cases were ones where undergraduates were plied with alcohol, and did not otherwise involve using physical force or other forms of coercion is difficult to read in any way other than "Half of the people you're counting as having been raped haven't really been raped", and favourably referring to an article that asserts that the rate of false rape reports is probably close to 50% is pretty strong support for the idea that many rape victims are liars.

(Update 2012/10/30: Adam Williamson suggests in this comment that this mail is a better example of Ted's behaviour - there's some explicit victim blaming and a lot of "Is that rape" questioning with the obvious implication that the answer should be "no". Ted Ts'o is a victim blaming rape apologist.)

(Update 2012/11/05: It's been suggested that I haven't been sufficiently clear about which of Ted's statements justify my claims. So, here we go.

In this mail, Ted links to and endorses this article. He explicitly links to it because of its treatment of rape statistics. Quoting directly from that article:
the rate of false reports is at least 9 percent and probably closer to 50 percent
Ted explicitly endorses an article that claims that a significant percentage of reported rapes are false. The study that generated that figure is held in poor regard by other researchers in the field - Australian police figures indicate that 2.1% of rape accusations were classified as false. Ted asserts that he was trying to argue against poor use of statistics, so it's a fair assumption that he agrees with the alternative statistics that he's citing. Ted believes that many rape victims are making false accusations. Ted believes that many rape victims are liars.

Again in this mail, Ted argues against a claimed figure that 1 in 4 women have been sexually assaulted. One of his arguments is that Also found in the Koss study, although not widely reported, was the statistic that of the women whom she classified as being raped (although 73% refused to self-classify the event as rape), 46% of them had subsequent sex with the reported assailant. Ted disagrees with a statistic because some rape victims subsequently have sex with the reported assailant. This means that Ted believes that this indicates that they were not really raped. Ted is a rape apologist.)

comment count unavailable comments

Wouter Verhelst: Dear lazyweb,

I am looking for a replacement for IlohaMail, since I just noticed it stopped working reliably:
[Mon Oct 29 18:57:18 2012] [error] [client 195.144.91.202] PHP Warning:  strcasecmp() expects parameter 1 to be string, array given in /usr/share/IlohaMail/include/mime.inc on line 87, referer: [...]
This happens when trying to view an email. When that happens, the headers are parsed correctly, but the body isn't shown, and one has to instead click on the "view source" link to see anything at all. There's actually a patch for this bug gathering dust in the BTS; that, plus the fact that it has not seen any upstream activity since 2005, and that the last maintainer upload into Debian was in 2008, makes me conclude that IlohaMail is probably dead. Therefore, I'm looking for a replacement. Specifically, I'm looking for: Does something like that exist? If so, pointers would be greatly appreciated.

28 October 2012

Leo 'costela' Antunes: Silly Wal(l Cloc)k

Inspired by this clock, which made the rounds a few weeks back, and fueled by a bet to see who could make it first (hi Fabs!), ladies and gentleman, I m honored to introduce to you, the Silly Wal(l Cloc)k:

The finished clock This is a prototype version, done without any fancy precision tools and whatnot, so you can easily see that it s not the most high quality product you ll ever get your hands on. If anyone wants to reproduce this at home, here s how it was done: First get a hold of some cheap wall clock like this one: The original cheap clock Some things to think about when buying this beauty *cough*:

Disassembling this specific cheap model was pretty straightforward and didn t take more than a minute to get to juicy parts:

Clock parts: actual clock, cover tack and hands As a base image I used this and traced it to produce this SVG with the legs in different layers and a marker for the holes. For this prototype, it would probably have been ok to use the original photo directly, but having the vector version ready means it s easy to use a 3d printer and laser-cutter to make the next version of the hands and face, which will definitely look and behave a lot better. Next step is printing the three layers body and two legs. Since this specific clock is wider than an A4 sheet, I had to use some white cardboard as background (the clock s own box hurray for resourcefulness) and painstakingly cut the printed body s outline, to avoid the slight tone difference between cardboard and paper. If you use the provided SVG and a laser-cutter, you d hopefully avoid this sort of issue. I didn t have any black cardboard lying around, so I had to print the legs, cut the outline again and strengthen them with some thin cardboard from tea boxes (taken directly from the trash hurray for resourcefulness again). Again something that 3d printing will hopefully side-step. It s important to use light materials for the legs, since most clock mechanisms don t have enough force to move heavy hands, which would cause the clock to quickly loose precision. Depending on the width of the clock s axis, a paper hole puncher might be the ideal tool to make holes in the legs. Since the prototype was largely made with paper, there was no point in going for precision when making the holes, as there s no way to fit the legs tightly to the clock s axis. For this, the original hands were cut in length and used as a base on which to super-glue the legs, so the original holes could be used. After these few easy steps, it s only a matter of reassembling the clock and voil !

Silly Wal(l Clock)k

Steve Langasek: SecureBoot in Ubuntu 12.10

The 12.10 release is the first version of Ubuntu that supports Secure Boot out of the box. In what is largely an accident of release timing, from what I can tell (and please correct me if I'm wrong), this actually makes Ubuntu 12.10 the first general release of any OS to support Secure Boot. (Windows 8 of course is also now available; and I'm sure Matthew Garrett, who has been a welcome collaborator throughout this process, has everything in good order for the upcoming Fedora 18 release.) That's certainly something of a bittersweet achievement. I'm proud of the work we've done to ensure Ubuntu will continue to work out of the box on the consumer hardware of the future; in spite of the predictable accusations on the blogowebs that we've sold out, I sleep well at night knowing that this was the pragmatic decision to make, maximizing users' freedom to use their hardware. All the same, I worry about what the landscape is going to look like in a few years' time. The Ubuntu first-stage EFI bootloader is signed by Microsoft, but the key that is used for signing is one that's recommended by Microsoft, not one that's required by the Windows 8 certification requirements. Will all hardware include this key in practice? The Windows 8 requirements also say that every machine must allow the user to disable Secure Boot. Will manufacturers get this right, and will users be able to make use of it in the event the manufacture didn't include the Microsoft-recommended key? Only time will tell. But I do think the Linux community is going to have to remain engaged on this for some time to come, and possibly hold OEMs' feet to the fire for shipping hardware that will only work with Windows 8. But that's for the future. For now, we have a technical solution in 12.10 that solves the parts of the problem that we can solve. This first release gives us preliminary support for booting on Secure Boot, but there's more work to be done to provide a full solution that's sustainable over the long term. We'll be discussing some of that this week at UDS in Copenhagen. And as part of our committment to enabling new hardware on the current LTS release, we will be backporting this work for inclusion in 12.04.2. It remains to be decided how Debian will approach the Secure Boot question. At DebConf 12, many people seemed to consider it a foregone conclusion that Debian would never agree to include binaries in main signed by third-party keys. I don't think that should be a given; I think allowing third-party signatures in main for hardware compatibility is consistent with Debian's principles, and refusing to make Debian compatible with this hardware out of the box does nothing to advance user freedom. I hope to see frank discussion post-wheezy about keeping Debian relevant on consumer hardware of the future.

Iustin Pop: Open Source collaboration, 2012

I'm interrupting my blog silence (which was due to the fact that I didn't have anything interesting to say) to note two things: first, but less important, spammers are still active as ever - I had ~2K spam comments and another ~2K in pending moderation. And second, and the actual subject of this post, is how easy collaboration has become. In the last year or so (where ~so means since I last looked), GitHub has introduced a very nice collaboration work-flow: This is soo easy (what, 3-4 clicks?) that I expect it will increase trivial fix-ups against Open Source projects hosted on GitHub or other services with similar capabilities. I did in the past week two or three such trivial fixes myself; this is not much note that I'm not talking about code fixes; but still, it is something that I think will definitely improve many, many projects. The same way that Wikis made it easy to improve documentation sites, this could make it easier for end-users to improve upstream's project documentation, when stored in a VCS as opposed to a Wiki. Yay! On the other hand, I still have a few patches sent months ago to an upstream email address, since they don't have a issue tracker (not even speaking about a place to submit merge requests). These will probably languish and disappear, as opposed to visible/public "hey, this is a patch to fix tiny typo in File.hs!". Yes, I like this a lot. Now, if only all projects would have a public VCS and issue tracker :) PS: yes, I'm talking here about GitHub, which is not Open Source software itself. I regard that as a (temporary) inconvenience, which dies not detract from the significant advantage it brings to the table.

27 October 2012

Russ Allbery: Review: Devil Take the Hindmost

Review: Devil Take the Hindmost, by Edward Chancellor
Publisher: Plume
Copyright: 1999
Printing: June 2000
ISBN: 0-452-28180-6
Format: Trade paperback
Pages: 349
Subtitled A History of Financial Speculation, Devil Take the Hindmost is exactly that: a history, with chapters on notable speculative bubbles or surges going back to the 1630s. The first significant bubble described is, of course, the notorious tulip speculation in Holland, but Chancellor is more thorough than any previous history I've read. There are chapters on the early stock market schemes, the South Sea bubble, 1820s emerging markets, the railway mania of 1845, the Gilded Age, the late 1970s and early 1980s, and the Japanese real estate bubble of the 1980s, as well as the mandatory look at the stock market crash of 1929. Each is thorough and detailed; indeed, perhaps slightly too detailed, as the clear narrative picture can occasionally groan under the weight of names and capsule biographies. The first thing one should realize about this book is that, despite the apparent currency of the topic, it was written in 1999. That means it predates both the subprime real estate collapse and the dot.com boom, as well as some more minor but possibly relevant events such as the Enron electricity market manipulation. Japan is the most recent bubble that Chancellor can cover; one can only speculate as to what he would make of the 2008 financial crash. The other focus to be aware of is the omission of sovereign debt. Chancellor is exclusively focused on private markets, usually stock markets, and (apart from the chapter on Japan) Western markets, mostly in the US and the UK. Governments play a role in some of the early speculative bubbles (most notably the South Sea bubble), but Chancellor's topic is private speculative frenzies, not the risks and effects of government borrowing. Interrogation of the role of government is mostly limited to cursory discussions of the impact of regulation, mostly in the negative. More on that in a moment. This is a history as opposed to either a polemic or a prescription. Chancellor does not set out to explain how to curb financial speculation. Insofar as he expresses an opinion on it, he presents speculation as a force of unbridled freedom in natural tension with regulation and financial conservatism and sees a cyclical pattern of speculative frenzy, collapse, retrenchment, and then renewed speculation. Speculation, as presented here, is more akin to a force of nature than, as seen in most other books on the topic, a phenomenon of greed that should be limited or prevented. Indeed, Chancellor seems skeptical of the entire project of preventing or limiting speculation, and presents considerable reason for the reader to become skeptical as well. The history he presents is one of speculative manias adapting to and circumventing every control that governments and markets have attempted to place on them. When certain practices are outlawed (and some modern complex financial practices have surprisingly long histories), people just find loopholes or alternative approaches through which to accomplish the same thing. Or the speculative mania takes an unexpected new form. Nor do the manias seem to be decreasing in severity as more regulation, custom, and restrictions attempt to blunt them. Even without adding the financial collapse of 2008, the impacts of the speculative frenzies documented here seem, if anything, to be growing. This is, therefore, a relatively straightforward history, but it's an entertaining one, filling in details that I'd not previously been aware of. The complexity of financial schemes as early as the 1600s was particularly eye-opening. There's a tendency to believe that complex derivative schemes are an artifact of modern financial markets, but they've been a part of speculation for as long as Chancellor goes back. Computers may have added layers to the mathematics, but they haven't added as much to the legal strategems and levels of indirection as one might think. Chancellor emphasizes that throughout: speculation involves a lot of complexity, not necessarily to deceive but to construct situations in which it's in the interest (at least short term) of all involved parties to continue and escalate the speculation. While Chancellor doesn't provide much encouragement to the reader hoping to find ways to curb speculation, he does show some commonalities that provide a bit of a handle on how speculation starts. The one that lept out at me was leverage. Every speculative bubble documented in Devil Take the Hindmost involves borrowing money to reinvest. The mechanism keeps changing, from straightforward loans during the tulip craze to much more complex constructions in modern finance, primarily in response to regulation that attempted to reduce leverage but only banned one specific technique. But whatever method is used, at the bottom is always some scheme for purchasing an asset using debt, magnifying both positive and negative price swings. Some variation of buying stocks on margin is particularly common; Chancellor documents how it was reinvented despite the limits on straight margin purchases instituted after the Great Depression. There is no recommendation here for any regulatory or legal action, but it's obvious from the repetition of history that limiting leverage would be the place to focus attention. I enjoyed this history, but it can be dry, and at times is a bit overstuffed with names, unimportant details, and microbiographies. It's also unfortunate that Chancellor missed two of the most significant (and two very different) speculative bubbles through the timing of his work. It would have been interesting to see them through the lens of his history. What we have is still worth reading, but also somewhat frustrating in Chancellor's refusal to draw or at least underline general conclusions. Devil Take the Hindmost is long on history but short on analysis, and while I appreciate the historian's focus in a source of data, it left me somewhat unsatisfied. Rating: 6 out of 10

26 October 2012

Colin Watson: Automatic installability checking

I've just finished deploying automatic installability checking for Ubuntu's development release, which is more or less equivalent to the way that uploads are promoted from Debian unstable to testing. See my ubuntu-devel post and my ubuntu-devel-announce post for details. This now means that we'll be opening the archive for general development once glibc 2.16 packages are ready. I'm very excited about this because it's something I've wanted to do for a long, long time. In fact, back in 2004 when I had my very first telephone conversation with a certain spaceman about this crazy Debian-based project he wanted me to work on, I remember talking about Debian's testing migration system and some ways I thought it could be improved. I don't remember the details of that conversation any more and what I just deployed may well bear very little resemblance to it, but it should transform the extent to which our development release is continuously usable. The next step is to hook in autopkgtest results. This will allow us to do a degree of automatic testing of reverse-dependencies when we upgrade low-level libraries.

Matthew Palmer: The e-mail PDA

I ve been a wannabe GTD afficionado for some years. I ve wanted to do it, but managing lists has always been something that has too much friction, overhead, or whatever. Finally, though, I think I might have found a way to manage lists that works. My use-case isn t unique, although I will concede I m perhaps being more dogmatic than most. I want something that: My previous attempt was a tool I called tagnote it was a vim-outliner file full of hierarchically organised outliner entries, with tags inlined. It was a neat idea, but it wasn t smooth to add/browse/delete items, and didn t work with my phone at all (trying to use vim for any length of time on a bottom-of-the-range Android phone would kill me). The current iteration, as the title of this post suggests, is a list manager that entirely uses e-mail. It really is a perfect symbiosis: So what have I got, exactly? It s fairly straightforward: That last point is the one I m really happy I achieved. I ve always been a fan of hide it until you need it , but my previous system didn t let me do that. Now, though, I have a separate list called tickler, and all the items in there have an X-Tickle header, which specifies the date I want to see them. Each night a cronjob runs through the tickler and moves anything for today into the INBOX. An X-Tickle-Repeat header lets me have things that repeat over and over again. So in short, using entirely open-source tools and a couple of hours of my time doing things I enjoy anyway (shell scripts! woo!), I ve now got a list manager that doesn t get in my way more than it absolutely has to. We ll see how long I last this time before I feel the urge to improve my lists again.

Next.

Previous.