After 27 hours of travel, I'm finally back from my first Linux.conf.au. This was a great conference! Some of my favorite highlights:

• Out for dinner the first night, my whole table started spontaneously talking about Haskell, including details of IO management frameworks like conduit and pipes, and a new one that's waiting in the wings. That just doesn't happen in the real world. A lot of us continued to do Haskell stuff in the hallway track, although for some reason there were no Haskell talks on the official schedule. Maybe it's time for a Haskell miniconf at LCA?
• Meeting Josh, Jamie, Sarah, Brendan, Craig, and others I've worked with online but never encountered IRL. Also reconnecting with old friends (some I'd not seen in 13 years) and finding new ones.
• The speaker's dinner in a revolving restaurant overlooking Canberra. Leave it to a restaurant full of geeks to invent an asychronous communications medium in such a setting. (Notes attached to windows to be read and answered by tables as they rotated by.)
• Meeting quite a lot of git-annex users, Kickstarter backers, and people interested in using git-annex. Thanks to everyone who came up to me for a chat.
• The evening pickup board game sessions. Especially my wiping out three other Tsuro players at once by forcing them all to end on the same square. ;) These were where I felt the most at home in Australia.
• Robert Llewellyn dropping in and mingling with fans of Red Dwarf and Scrapheap Challenge. One of the very few actors I could possibly fanboy on, and LCA not only somehow got him, but constructed an atmosphere that allowed for this photo of us.

---

My git-annex talk went ok, despite technical problems with the video output, which the video team did a great job of working around. The first demo went well, and the Q&A was excellent. The demo of the git-annex assistant webapp was marred by, apparently, a bad build of the webapp -- which I ironically used rather than my usual development build because I assumed it'd be less likely to fail. At least I may have a way to reproduce that hang somewhat reliably now, so I can get on with debugging it. I will be redoing a demo of the webapp as a screencast.

---

Here are some of the best talks I attended. (I have quite a lot more queued up to watch still, and had to cut several due to space.) Click titles for videos, or browse all the videos.

• Git for Ages 4 and Up
  Schwern has found a new, excellent way to explain git. I felt rather bad for using up a seat, especially once people were kicked out when the room was filled over capacity. But I enjoyed every minute of it. (Also has the best speaker intro ever. "Schwern once typed git pull --hard and pulled GitHub's sever room across the street.")

  BTW, I must confess: I left the red apple on teacher's desk.

• Radia Perlman's keynote
  Network protocol design and poetry from one of the quiet heros of our field. I knew Spanning Tree Protocol was used in ethernet, but it just works, so like many I never paid much attention to it. This talk felt like the best lectures, where you're learning from a master on multiple levels at once. Well done work often becomes an unremarked part of the landscape, which I sometimes find unrewarding, so it was great to have Radia give some perspective on what that's like over the course of decades.
• Lightning Talks
  A 90 second time limit really helps. Too many conferences have 5 minute talks, which is less exciting. If you still find them boring, skip forward to 13:50, where pjf does two talks in 90 seconds! (If a 20 second talk on depression is too .. manic, there's an encore at the end.)
• The IPocalypse 20 months later
  A reality check, with real data. Very important stuff here. We need to work to avoid this worst case scenario, and we also need to design around it.
• REPENT!!! FOR THE END OF THE UNIX EPOCH IS NIGH!!!
  Wildest talk beginning I've seen since RMS put the hard drive halo on his head. And to an important point: Any programs that deal with dates 25 years in the future already need to be fixed today to deal with the epoch rollover. This got me digging around in Haskell date libraries, to make sure they're ok.
• Building Persona: Federated and Privacy Sensitive Identity for the Web
  This talk and some previous conversation with Francois have convinced me that Persona (AKA Browserid) has a design that can succeed. I will be adding Persona login support to ikiwiki.
• Beyond Alt Text: What Every Project Should Know About Accessibility
  I missed the first half due to giving my talk, but the second half was full of rather a lot of excellent information, some of which I'd only guessed at before.
• Git: Not Just for Source Code Anymore
  Good overview of the new ways to use git. Also kept giving examples from my body of work, which is some nice ego stroking, thanks Josh. ;-)

It was interesting to read recently from Martin F. Krafft a botnet-like configuration management proposal. Professionally I've used CFEngine, which in version 2.x, supported a bare minimum of primitives, along with a distribution systme to control access to a central server. Using thse minimal primitives you could do almost anything:

• Copy files, and restart services that depend upon them.
• Make minor edits to files. (Appending lines not present, replacing lines you no longer wanted, etc)
• Installing / Removing packages.
• More ..

Now I have my mini cluster (and even before that when I had 3-5 machines) it was time to look around for something for myself. I didn't like the overhead of puppet, and many of the other systems. Similarly I didn't want to mess around with weird configuration systems. From CFEngine I'd learned that using only a few simple primitives would be sufficient to manage many machines provided you could wrap them in a real language - for control flow, loops, conditionals, etc. What more natural choice was there than perl, the sysadmin army-knife? To that end slaughter was born:

• Download polices (i.e. rules) to apply from a central machine using nothing more complex than HTTP.
• Entirely client-driven, and scheduled via cron.

Over time it evolved so that HTTP wasn't the only transport. Now you can fetch your policies, and the files you might serve, via git, hg, rsync, http, and more. Today I've added one final addition, and now it is possible to distribute "modules" alongside policies and files. Modules are nothing more than perl modules, so they can be as portable as you are careful. I envisage writing a couple of sample modules; for example one allowing you to list available sites in Apache, disable the live ones, enable/disable mod_rewrite, etc. These modules will be decoupled from the policies, and will thus be shareable. Anyway , I'm always curious to learn about configuration management systems but I think that even though I've reinvented the wheel I've done so usefully. The DSL that other systems use can be fiddly and annoying - using a real language at the core of the system seems like a good win. There are systems layered upon SSH, such as fabric, ansible, etc, and that was almost a route I went down - but ultimately I prefer the notion of client-pull to server-push, although it is possible in the future we'll launche a mini-daemon to allow a central host/hosts to initial a run.

I lost my will roll against placing another book order a couple of weeks ago. There are just too many people writing too many fascinating things that I want to read! Elizabeth Bear Shoggoths in Bloom (collection)
David Graeber Debt: The First 5,000 Years (non-fiction)
Eric Hobsbawm The Age of Revolution 1789 1848 (non-fiction)
China Mi ville Kraken (sff)
John McPhee Annals of the Former World (non-fiction)
C.E. Murphy The Queen's Bastard (sff)
C.E. Murphy The Pretender's Crown (sff)
Steven Pinker Words and Rules (non-fiction)
David Roodman Due Diligence (non-fiction)
Rebecca Skloot The Immortal Life of Henrietta Lacks (non-fiction)
Richard G. Wilkinson & Kate Pickett The Spirit Level (non-fiction)
Fumi Yoshinaga oku: The Inner Chambers #1 (graphic novel) As you can see, mostly non-fiction. Which is, of course, the type of book that takes the longest to read. I'm so smart about managing a backlog. The Pinker was recommended by one of my favorite Teaching Company courses (Myths, Lies, and Half-Truths of Language Usage by John McWhorter), and I was interested enough in linguistics to want to read something a bit more intensive. John McPhee's Annals of the Former World is about geology, something I've wanted to know more about for a while, and won the Pulitzer for general non-fiction. It's also huge. It might take me a while to get to that. Of the fiction, I read the first Murphy a long time back, but it was a borrowed copy, and I want to re-read it before reading the second volume in the duology. I remember really enjoying it, but don't remember anything about what happened. oku is manga that won the Tiptree award, so that sounded too interesting to pass up. And since I own just about everything else Elizabeth Bear has published, I figured I should keep the streak alive, even if I'm a bit behind on reading it.

Today short before ending business hours I was noticed that there is a problem with a server system (domU). Login with unprivileged user was possible but using "su" didn't worked, also login in as root via privkey failed. Fortunately I was able to connect via xen console and login via tty. Looking into the bash history the reason revealed quickly:

4979 2013-02-01 15:03:39 cd /var/www/test.org/data/galleries/
4980 2013-02-01 15:03:43 chown www-data:www-data -R /var/www/test.org/data/galleries/
4981 2013-02-01 15:04:36 ls -la
4982 2013-02-01 15:04:46 ls -la
4983 2013-02-01 15:04:54 chown www-data:www-data -R /*
4984 2013-02-01 15:07:42 chown www-data:www-data -R /var/www/test.org/data/galleries/
4985 2013-02-01 15:36:55 chown www-data:www-data -R /var/www/test.org/data/galleries/

This made my day (and maybe parts of the rest of the weekend). For recovery our 1st Level mounted the domU-fs on the dom0 to '/tmp/recover' and did:

2131  2013-02-01 21:29:28 cd /tmp/recover
[...]
2142  2013-02-01 21:31:17 rm -r lib64/

The experienced reader may see the problem:

# ls -lad lib64
lrwxrwxrwx 1 root root 4 Jun 28  2011 /lib64 -> /lib

So also the dom0 was knocked out ... what a funny evening (and maybe night). Maybe our staff looked similar like here.

As most people reading this have probably seen, xkcd found a particularly entertaining Wikipedia argument. (It is all true. There are at least 40,000 words arguing over whether to capitalize an "i" in a page title. I went and looked last night and spent several hours following the conversation.) This is, of course, funny, and I'm sure lots of people went away smiling at an excellent example of bikeshed painting on the Internet. But I think it's rather more serious than it appears to be. In fact, I would go so far as to say that it is a visible manifestation of a deeply-rooted problem that is the core of why I personally refuse to contribute to Wikipedia in any way that requires me to invest non-trivial time or become a member of that community. To explain that statement requires a bit of background. One of the very first services I found when I joined the Internet in 1993 was Usenet. Originally, I talked about comics collecting and then Magic: The Gathering, but I eventually ended up involved in Usenet governance. I spent more than ten years deeply invested in Usenet hierarchy administration; roughly, the process of agreeing on names for and sometimes moderators for global Usenet discussion groups, as well as all the supporting infrastructure for Usenet news server administrators to agree upon and share articles in those groups. Usenet newsgroup creation was, throughout that period, a consensus-driven process. There was a voting system that was used to measure consensus, but the voting system was largely designed to ensure that rough consensus existed (it was heavily skewed towards not changing anything). We took great pride in that fact. I'm sure that one could find many articles from me proudly championing the concept of a consensus-based decision-making process against many alternatives. That means I was also there when the toxic flaws at the heart of a consensus-based process completely destroyed it and, as a side effect, destroyed the community of people who had formed around it. I will never again invest my time and energy in a community that operates solely via a consensus process unless that community is small enough that consensus can actually work (which means not much more than twenty people). Consensus decision-making in large groups destroys communities and hurts people. These sorts of extended arguments on Wikipedia are not rare anomolies. They happen with some frequency; just by going through the debate xkcd references and looking at pointers, you can easily find others (Sega Genesis, cat ownership vs. companionship, endless notability debates... there are many). They look funny from outside, but from inside they're hugely demoralizing. Even when they don't get heated and spill over into personal animosity (and if they don't do that, you're lucky), they consume astonishingly vast quantities of time exactly not the resource that you want to be squandering in volunteer projects. And they do so without giving back anything emotionally. I know these arguments very well. One of the most contentious and angry arguments in the history of Usenet newsgroup creation was the argument over whether, when breaking up a group into multiple more focused groups, to rename the original group to end in *.misc or leave its existing name. This argument went on for years and led to deep, lingering hatred between the participants. It's the same sort of thing. These arguments happen in the way that they happen because there is toxic waste at the center of Wikipedia's governance process. And that toxic waste hurts people, wastes their time, and destroys their willingness to volunteer. It undermines community, which is sad and ironic because the whole point of consensus decision-making is to create, support, and empower community. But it doesn't work. It doesn't work for two very simple but very closely-related reasons: it's impossible to ever end an argument, and there is no incentive for anyone to shut up. As a result, it is nearly impossible to get the decision-making to cohere into something that is timely, consistent, unambiguous, and final. These are not optional requirements of a decision-making process. Without them, people will lose faith in the decision-making process itself, and some people will learn that, as long as they never stop talking about something, they can prolong the debate indefinitely and possibly eventually get their way. Now, of course, I'm deeply involved and invested in Debian. And on the surface, it may look like Debian also uses a consensus-based decision-making process. But while Debian decision-making has some problems, it's much healthier for a few reasons that are useful to examine:

1. Individual packages in Debian are not managed by consensus, but rather fall under the authority of the package maintainer, who is either a single person or a small team. (Consensus works in small teams; it's great if the number of people involved is five or less.) This drastically limits the number of decisions that are subject to a large-scale consensus process. Nearly all decisions in Debian are made directly by the person doing the work, and only in exceptional circumstances can some larger process be invoked. This, on the surface, sounds like it could be similar to Wikipedia, but note that appeals to a larger process are much rarer in Debian than they are in Wikipedia. There is a strong bias in favor of letting the person working on the package do their job however they want to do it. This is more social than structural, but it's a core value (and we should be very careful about changing it). As a result, Debian operates less by mass consensus and more as a federation of semi-autonomous fiefdoms. This sounds worse: it sounds authoritarian and non-democratic. But it's significantly more functional, and it doesn't waste people's time.
2. On appeal, and for broader questions that necessarily cut across multiple packages, there are several people in Debian who are perceived as having clear authority to make timely and relatively final decisions, who are almost never overruled, and who can therefore effectively end arguments. These include core teams like ftp-master or the release team, formal appeal bodies like the Technical Committee, and the Debian Project Leader. Not all of this ability is written in black and white in the governance process, but that's not the important part; the important part is that it works in practice to provide timely, consistent, unambiguous, and final decisions most of the time. (Some parts work better than others, and we do struggle with timely, but I've seen much, much worse.)
3. There is a well-established ultimate appeal (a General Resolution), the ultimate appeal process is very specific and concrete, there is little or no ambiguity or human interpretation involved in analyzing the results, it is time-limited, and it's almost universally respected. Everyone abides by the result of a GR if matters get to that point, and while it's theoretically possible to propose another GR immediately to overturn the first one, this doesn't happen in practice. There's also substantial pushback against invoking the GR process for anything that isn't really important.

We still occasionally devolve into endless discussion, usually on some technical topic that cuts across the distribution and usually when the options haven't cohered enough for any authority to make a useful decision (often because it's too early for it to be possible to make a final decision). Those cases, such as the recurring systemd vs. upstart discussion, are still quite frustrating. But they're also relatively rare and largely ignorable because they have to be about just the right sort of issue. Wikipedia has these sorts of arguments all the time, and they're much less ignorable, because consensus is invoked for every minor dispute on any page in the entire project. You have to participate in endless consensus process or risk having your work undone, reverted, or removed. We would all like all decisions to be made by consensus, since we don't want anyone in a volunteer project to lose (and because we all love to convince other people we're right). And for small teams, consensus is wonderful. But that's precisely because governance in small teams is very easy; humans are naturally adapted for decision-making in small teams and have a huge variety of natural tools, tendencies, and abilities devoted to making those sorts of decisions. You rarely need any formal decision-making process; most decisions will just happen. You need a decision-making process when enough people are involved that mutual social pressure can't bring a dispute to resolution, and when you're in that situation, consensus is a horrible process. Due to my past experiences, I'm almost certainly at the extreme in dislike of consensus processes. Most people are not going to refuse to participate in communities solely on that basis. But be aware that 40,000 words of discussion about something that an outsider will not consider particularly important is not an accidental phenomenon. It's not something that just happens randomly on the Internet. It's an effect, and symptom, of a governance process, or lack thereof, and it's something that you can do something about if you want to.

Like many people I'll be going to FOSDEM again this year. I'll be giving a talk on Saturday at 1800 in the Virtualisation devroom on Evolving Xen Paravirtualization. I'll also be manning the Xen.org booth at least some of the time but hopefully I'll get to see some of the other talks too. Hope to see everyone there!

Dear Lazyweb, can anyone recommend a good Thunderbird extension which allows for synchronizing the address book with Google mail? So far I tried Google Contacts, but something went wrong with the syncing and some contacts where deleted on both sides. To avoid this problem, one can use Google Contacts in read-only mode (it will only fetch contacts from Google, but never write to it) but then you have to import new Thunderbird contacts to Google mail manually. Google introduced CardDav in December 2012, which allows for syncing of contacts, but since Thunderbird s development is apparently on hold this is probably not gonna be supported out-of-the-box. There are some other extensions for Thunderbird, but since synchronization is hard and a lot more complicated than: replace newer version with older one I m looking for something mature and well tested. Before someone suggests it, I know Evolution has this feature built-in. I gave it a try last week and found so many other grave bugs with the calendar and newsgroups that Evolution is simply unfit for my needs. I really like Thunderbird and want to stick with it for a few more years until I have to look for something else. Yours truly, Basti

The War on Time Whoosh! I ve been incredibly quiet on my blog for the last 2-3 months. It s been a crazy time but I ll catch up and explain everything over the next few entries. Firstly, I d like to get out a few details about the last Ubuntu Developer Summit that took place in Copenhagen, Denmark in October. I m usually really good at getting my blog post out by the end of UDS or a day or two after, but this time it just flew by so incredibly fast for me that I couldn t keep up. It was a bit shorter than usual at 4 days, as apposed to the usual 5. The reason I heard for that was that people commented in previous post-UDS surveys that 5 days were too long, which is especially understandable for Canonical staff who are often in sprints (away from home) for the week before the UDS as well. I think the shorter period works well, it might need a bit more fine-tuning, I think the summary session at the end wasn t that useful because, like me, there wasn t enough time for people to process the vast amount of data generated during UDS and give nice summaries on it. Overall, it was a great get-together of people who care about Ubuntu and also many areas of interest outside of Ubuntu. Copenhagen, Denmark I didn t take many photos this UDS, my camera is broken and only takes blurry pics (not my fault I swear!). So I just ended up taking a few pictures with my phone. Go tag yourself on Google+ if you were there. One of the first interesting things I saw when arriving in Copenhagen was the hotel we stayed in. The origami-like design reminded me of the design of the Quantal Quetzel logo that is used for the current stable Ubuntu release. The Road ahead for Edubuntu to 14.04 and beyond St phane previously posted about the vision we share for Edubuntu 14.04 and beyond, this was what was mostly discussed during UDS and how we ll approach those goals for the 13.04 release. This release will mostly focus on the Edubuntu Server aspect. If everything works out, you will be able to use the standard Edubuntu DVD to also install an Edubuntu Server system that will act as a Linux container host as well as an Active Directory compatible directory server using Samba 4. The catch with Samba 4 is that it doesn t have many administration tools for Linux yet. St phane has started work on a web interface for Edubuntu server that looks quite nice already. I m supposed to do some CSS work on it, but I have to say it looks really nice already, it s based on the MAAS service theme and St phane did some colour changes and fixes on it already. From the Edubuntu installer, you ll be able to choose whether this machine should act as a domain server, or whether you would like to join an existing domain. Since Edubuntu Server is highly compatible with Microsoft Active Directory, the installer will connect to it regardless of whether it s a Windows Domain or Edubuntu Domain. This should make it really easy for administrators in schools with mixed environments and where complete infrastructure migrations are planned. You will be able to connect to the same domain whether you re using Edubuntu on thin clients, desktops or tablets and everything is controllable using the Epoptes administration tool. Many people are asking whether this is planned for Ubuntu / Ubuntu Server as well, since this could be incredibly useful in other organisations who have a domain infrastructure. It s currently meant to be easily rebrandable and the aim is to have it available as a general solution for Ubuntu once all the pieces work together. Empowering Ubuntu Flavours This cycle, Ubuntu is making some changes to the release schedule. One of the biggest changes made this cycle is that the alpha and beta releases are being dropped for the main Ubunut product. This session was about establishing how much divergence and changes the Ubuntu Flavours (Ubuntu Studio, Mythbuntu, Kubuntu, Lubuntu and Edubuntu) could have from the main release cycle. Edubuntu and Kubuntu decided to be a bit more conservative and maintain the snapshot releases. For Edubuntu it has certainly helped so far in identifying and finding some early bugs and I m already glad that we did that. Mythbuntu is also a notable exception since it will now only do LTS releases. We re tempted to change Edubuntu s official policy that the LTS releases are the main releases and treat the releases in between more like technology previews for the next LTS. It s already not such a far stretch from the truth, but we ll need to properly review and communicate that at some point. Valve at UDS and Steam for Linux One of the first plenaries was from Valve where Drew Bliss talked about Steam on Linux. Steam is one of the most popular publishing and distribution systems for games and up until recently it has only been available on Windows and Mac. Valve (the company behind Steam and many popular games such as Half Life and Portal) are actively working on porting games to run natively on Linux as well. Some people have asked me what I think about it, since the system is essentially using a free software platform to promote a lot of non-free software. My views on this is pretty simple, I think it s an overwhelmingly good thing for Linux desktop adoption and it s been proven to be a good thing for people who don t even play games. Since the announcement from Valve, Nvidia has already doubled perfomance in many cases for its Linux drivers. AMD, who have been slacking on Linux support the last few years have beefed up their support drastically with the announcement of new drivers that were released earlier this month. This new collection of AMD drivers also adds support for a range of cards where the drivers were completely discontinued, giving new life to many older laptops and machines which would be destined for the dumpster otherwise. This benefits not only gamers, but everyone from an average office worker who wants snappy office suite performance and fast web browsing to designers who work with graphics, videos and computer aided design. Also, it means that many home users who prefer Linux-based systems would no longer need to dual-boot to Windows or OS X for their games. While Steam will actively be promoting non-free software, it more than makes up for that by the enablement it does for the free software eco-system. I think anyone who disagrees with that is somewhat of a purist and should be more willing to make compromises in order to make progress. Ubuntu Release Changes Last week, there was a lot of media noise stating that Ubuntu will no longer do releases and will become a rolling release except for the LTS releases. This is certainly not the case, at least not any time soon. One meme that I ve noticed increasingly over the last UDSs was that there s an increasing desire to improve the LTS releases and using the usual Ubuntu releases more and more for experimentation purposes. I think there s more and more consensus that the current 6 month cycle isn t really optimal and that there must be a better way to get Ubuntu to the masses, it s just the details of what the better way is that leaves a lot to be figured out. There s a desire between developers to provide better support (better SRUs and backports) for the LTS releases to make it easier for people to stick with it and still have access to new features and hardware support. Having less versions between LTS releases will certainly make that easier. In my opinion it will probably take at least another 2 cycles worth of looking at all the factors from different angles and getting feedback from all the stakeholders before a good plan will have formed for the future of Ubuntu releases. I m glad to see that there is so much enthusiastic discussion around this and I m eager to see how Ubuntu s releases will continue to evolve. Lightning Talks Lightning talks are a lot like punk-rock songs. When it s good, it s really, really amazingly good and fun. When it s bad, at least it will be over soon :) Unfortunately, since it s been a few months since the UDS, I can t remember all the details of the lightning talks, but one thing that I find worth mentioning is that they re not just awesome for the topic they aim to produce (for example, the one lightning talks session I attended was on the topic of Tests in your software ), but since they are more demo-like than presentation-like, you get to learn a lot of neat tricks and cool things that you didn t know before. Every few minutes someone would do something and I d hear someone say something like Awesome! I didn t know you could do that with apt-daemon! . It s fun and educational and I hope lightning talks will continue to be a tradition at future UDSs. Social Stefano Rivera (fellow MOTU, Debianista, Capetonian, Clugger) wins the prize for person I ve seen in the most countries in one year. In 2012, I saw him in Cape Town for Scaleconf, Managua during Debconf, Oakland for a previous UDS and Copenhagen for this UDS. Sometimes when I look at silly little statistics like that I realise what a great adventure the year was! Between the meet n greet, an evening of lightning talks and the closing party (which was viking themed and pretty awesome) there was just one free evening left. I used it to gather with the Debian folk who were at UDS. It was great to see how many Debian people were attending, I think we had around a dozen or so people at the dinner and there were even more who couldn t make it since they work for Canonical or Linaro and had to attend team dinners the same evening. It was as usual, great to put some more faces to names and get to know some people better. It was also great to have a UDS with many strong technical community folk present who is willing to engage in discussion. There were still a few people who felt missing but it was less than at some previous UDSs. I also discovered my face on a few puzzles! They were a *great* idea, I saw a few people come and go to work on them during the week, they seem to have acted as good menial activities for people to fix their brains when they got fried during sessions :) Overall, this was a good and punchy UDS. I ll probably not make the next one in Oakland due to many changes in my life currently taking place (although I will remotely participate), but will probably make the one later this year, especially if it s in Europe. I ll also make a point of live-blogging a bit more, it s just so hard remembering all the details a few months after the fact. Thanks to everyone who contributed their piece in making it a great week!

Well, today was another day of sleeping, zoning, and having no brain for programming (and no willpower for that or exercise, which is even more frustrating), so you all get more musing about missions and work. Tomorrow, I will re-engage my normal schedule and turn the television off again, because seriously, two days of recovery should be enough even for a whole afternoon of meetings. Perhaps the best concept in the scrum methodology for agile project management is the concept of the product backlog and the product owner. For those who aren't familiar, a scrum project has, as one of its inputs, a prioritized list of stories that the team will implement. During each planning session (generally between once a week and once a month), the development team estimates and takes from the front of the product backlog the top-priority stories that can be completed during that period of time. The product owner is responsible for building the list of pending development stories based on business needs and keeping them sorted in priority order so that the work of the team follows the goals of the larger organization. But the product owner does not get to say how difficult the stories are; the development team estimates the stories, and the product owner may change order based on those estimates (to do simpler things earlier, for example). We've been doing scrum to some extent or another for quite some time now, and while there are a variety of things that I like about it, I think this is the best part. And I think it has very useful concepts for management in general. The product backlog is effectively a problem stream of the sort that I talked about in my last couple of posts. It's a set of problems that need to be solved, that matter to other people. And, if the team is doing scrum properly, the problems are presented in a form that describe a complete unit of desired functionality with its requirements and constraints, but without assuming any sort of implementation. The problem is the domain of the product owner; the solution is the domain of the development team. I am increasingly convinced that my quality of life at work, and that of many other people doing similar sorts of development and project work, would be drastically improved if managers considered this role the core of their jobs. (Ironically, at least in our initial experiences with scrum, it was quite rare for regular managers to be product owners; instead, individual line staff or project managers act as product owners. I think this speaks to how confused a lot of managers are about their roles in a technical development organization. This seems to be improving.) The core of the sort of work that I do is a combination of ongoing maintenance and solving new problems. Both of those can be viewed as a sequence of (possibly recurring) stories; I know this because that's largely how I do my own time management. Apart from HR and staffing, the core role of a manager is almost exactly what scrum calls "backlog grooming": communicating with everyone who has input into what their group is doing, finding out what their problems are, translating that into a list of things that the organization needs to have done, prioritizing them, breaking them into manageable chunks, ensuring they are fully specified and actionable (read: well-written stories), and communicating their completion back to other stakeholders (and then possibly iterating). This lept out at me when I started thinking about our larger strategic vision. That strategic vision is a sort of product backlog: a set of development stories (or, in this case, epics that would be broken down into a large number of smaller stories). But most strategic plans have glaring flaws if they're evaluated by the standards of scrum product backlogs. They're normally written as marketing statements and aimed at the clients or customers rather than at the staff. From the staff perspective, they're often hopelessly vague, not concrete, actionable epics. Often, they are almost entirely composed of grand, conceptual epics that any scrum team would immediately reject as too large and nebulous to even discuss. And often they're undefined but still scheduled: statements that the organization will definitely complete some underspecified thing within the next year or two. Scrum rightfully rejects any form of scheduling for unspecified stories and epics. Additional vagueness and sketchiness is tolerated the farther the story is from the current iteration, so they can be put into the future backlog, but scrum makes no guarantees about when they'll get done until they've been sized and you can apply a reasonable velocity estimate. If you want to put a date on something, you have to make it concrete. This is exactly the problem that I have with long-range strategic plans. We already know this about technology development: long-range strategic plans are feel-good guesses, half-truths, and lies that we tell other people because we're expected to do so, but which no one really believes. The chance that we can predict anything about the shape of projects and rate of delivery of those projects three years from now is laughable, and certainly none of the work that would be required to make real estimates has normally been done for those sorts of long-term strategic projects. There are a lot of scrum advocates who would go farther than I in asking for a complete revolution of how technical projects are planned. I'm not sure how realistic that is, or how well the whole scrum process would work when rolling up work across dozens, if not hundreds, of scrum-sized teams. But in the specific area of the role of managers in a development organization, I think we could learn a lot from this product backlog concept. Right now, there is a constant tension between managers who feel like they need to provide some sort of visionary leadership and guidance (the cult of Steve Jobs) and line staff who feel like managers don't understand enough about specific technology to lead or guide anything. But if you ask a technical developer whether, instead, it would be useful for managers to provide a well-structured, priority-ordered, and clean problem stream from the larger organization so that the developer can trust the problem stream (and not have to go talk to everyone themselves) but have control over the choice of implementation within the described constraints, I think you would find a great deal of enthusiasm. As a further bonus, scrum has a lot of advice about what that problem stream should look like. I think some of the story structuring is overblown (for example, I can't stand the story phrasing structures, the "AS A... I WANT TO... SO THAT..." shackles), but one thing it does correctly emphasize is the difference between problem and solution. The purpose of the story is to describe the outcome from the perspective of the person who will use the resulting technology. Describing the constraints on the solution, in terms of cost or integrations, is actively encouraged. Describing the implementation is verboten; it's up to the project team to choose the implementation that satisfies the constraints. If managers in general would pick up even that simple distinction, I think it would greatly improve the engagement, excitement, and joy that developers could take in their work. There's very little that's more frustrating than to be given an interesting problem in the guise of a half-designed solution and be told that one is not permitted to apply any of one's expertise in elegance, robustness, or durability to replacing the half-baked partial solution. Or even be allowed to know the actual motivating problem, instead of the half-digested form that the manager is willing to pass on.

After this announcement about the future of GNOME in Debian, I received several requests to look at SolusOS and their Consort fork of GNOME. On the paper, this doesn t look like a too bad idea, since they only forked gnome-panel and metacity, the two key components that are no longer maintained upstream. Still, starting a plain fork at the very moment when the former maintainer decided to give the key to anyone who wanted to, looked like a very destructive way of acting. Even if the fork gains momentum (which remains speculative), the amount of effort to rename packages makes you think twice before such a switch. I decided to go talk about it with them on IRC nevertheless. Mind you, they work on a GNOME fork and a Debian derivative, but deliberately use their own IRC server (you ll soon understand why). Just in case you would want to cooperate with SolusOS, you d have to use their infrastructure. It is an euphemism to say that the conversation didn t go well. This could have ended there, but thankfully for your already widening eyes, Ikey (the charming person I had the opportunity to discuss with) made the log public, in an attempt at public shaming that would soon gather his followers, chanting out loud how the revolutionary SolusOS would quickly replace every other Linux distribution. (As a side note, of the hundreds of such claims that were made public in the last 10 years, only one came true, and it was from a billionnaire who hired dozens of developers. Just saying.) Then you can understand why the specific IRC server: it is to K-line people who don t behave with enough deference to Ikey, because kickban is not enough. This log and the claims made in the comments look very ironical, if only for presenting a person working in a team of 10+ on the most popular desktop on one of the most popular desktop distributions, as a beggar. Even more ironical when it is the person who spent the most time on making GNOME Classic available and working nicely for wheezy. So when a dozen kids go chanting that Consort is so much better than GNOME and will replace it, while it just re-does the work I already did, let me at least smile. So what is SolusOS, after all? If you want to see where this story is going, there s a place that tells you: the consortium and consort-panel repositories. Here you will find interesting things:

• the work started from a tarball, instead of forking the git repositories (just in case you wanted to pull some of their patches);
• there is only one active contributor, who happens to be the founder of SolusOS, and the same person who K-lines people on his IRC channel;
• he seems to commits stuff randomly, without any cleanup, and with a lot of 4-letter words;
• most commits are here to rename everything to the Consort brand;
• even in debian/changelog, he retroactively renamed every mention of the former package names;
• some changes were made prior to creating the repository, so they are completely lost in history;
• among those changes I noted the reversion of a GNOME 3.0 change, which is to require Alt+right-click to edit a panel (and anyone who thinks this is a bug has never had to manage a help desk where disappearing notification areas were in the Top 5 incidents);
• nevermind the change was done in a hackish way and even mentions that in a comment, while the upstream commit is still available in git history;
• this also includes committing directly Debian-specific patches without a further look at why they hadn t been pushed upstream.
• For example, he pushed the menus renaming, which is a very Debuntu-specific stuff which will yield empty menus on other distributions.
• Or a bugfix that will only build with Debian s modified libwnck.

I ve stopped looking there. We re facing an ego issue, but not something that can improve the desktop for our users. In the end, I m sorry for those who genuinely thought we could do better by sharing some maintenance load with SolusOS, but it doesn t look like a big loss anyway.

This post is about the port bootstrap build ordering tool (naming suggestions welcome) which was started as a Debian Google Summer of Code project in 2012 and continued to be developed afterwards. Sources are available through gitorious. In the end of November 2012, I managed to put down an approximation algorithm to the feedback arc set problem which allowed to break the dependency graph into a directed acyclic graph with only few removed build dependencies. I wrote about this effort on our mailinglist but didnt mention it here because it was still too much of a proof-of-concept. Later, in January 2013, I mentioned the result of this algorithm in an email wookey and me wrote to debian-devel mailinglist. Many things happened since November 2012:

Processing pipeline instead of monolithic tools The tools I developed so far tried to accomplish everything by themselves, reusing functionality implemented in a central library. Therefor, if one wanted to try out even trivial new things, it mostly meant to hack some OCaml code. Pietro Abate suggested to instead develop smaller tools which could work independently of each other, would only execute one algorithm each and could easily be connected together in different ways to achieve different effects. This switch is now done and all functionality from the old tools is moved into a new toolset. The exchange format between the tools is either plain text files in deb822 control format (Packages/Sources files) or a dependency graph. The dependency graph is currently marshalled by OCaml but future versions should work with just passing a GraphML (an XML graph format) representation around. This new way of doing things seems close to the UNIX philosophy (each program does one thing well, data is stored as text, every program is a filter). For example the deb822 control output can easily be manipulated using grep-dctrl(1) and there exist many tools which can read, analyze and manipulate GraphML. It is a big improvement over the old, monolithic tools which did not allow to manipulate any intermediate result by external, existing tools. Currently, a shell script (native.sh) will execute all tools in a meaningful order, connecting them together correctly. The same tools will be used for a future cross.sh but they will be connected differently. I wrote about a first proposal of what the individual tools should do and how they should be connected in this email from which I also linked a confusing overview of the pipeline. This overview has recently been improved to be even more confusing and the current version of the lower half (the native part) now looks like this: Solid arrows represent a flow of binary packages, dottend arrows represent a flow of source packages, ovals represent a set of packages, boxes with rounded corners represent set operations, rectangular boxes represent filters. There is only one input to a filter, which is the arrow connected to the top of the box. Outgoing arrows from the bottom represent the filtered input. Ingoing arrows to either side are arguments to the filter and control how the filter behaves depending on the algorithm. I will explain this better once the pipeline proved to be less in flux. The pipeline is currently executed like this by native.sh.

Two new ways to break dependency cycles have been discovered So far, we knew of three ways to formally break dependency cycles:

• remove build dependencies through build profiles
• find out that the build dependency is only used to build arch:all packages and therefor put it into Build-Depends-Indep
• cross compile some source packages

Two new methods can be added to the above:

• if a dependency relationship is not strong (pdf), then a different installationset can be chosen
• if a depended upon package is Multi-Arch:foreign then a binary package of an existing architecture might be able to satisfy the dependency.

Dependency graph definition changed Back in September when I was visiting irill, Pietro found a flaw in how the dependency graph used to be generated. He supplied a new definition of the dependency graph which does away with the problem he found. After fixing some small issues with his code, I changed all the existing algorithms to use the new graph definition. The old graph is as of today removed from the repository. Thanks to Pietro for supplying the new graph definition - I must still admit that my OCaml foo is not strong enough to have come up with his code.

Added complexity for profile built source packages As mentioned in the introduction, wookey and me addressed the debian-devel list with a proposal on necessary changes for an automated bootstrapping of Debian. During the discussiong, two important things came up which are to be considered by the dependency graph algorithms:

• profile built source packages may not create all binary packages
• profile built source packages may need additional build dependencies

Both things make it necessary to alter the dependency graph during the generation of a feedback arc set and feedback vertex set beyond the simple removal of edges. Luckily, the developed approximation algorithms can be extended to support such changes in the graph.

Different feedback arc set algorithms The initially developed feedback arc set algorithm is well suited to discover build dependency edges which should be dropped. It performs far worse when creating the final build order because it only considers edges by itself and not how many edges of a source package can be dropped by profile building it. The adjusted algorithm for generating a build order is more of a feedback vertex set algorithm because instead of greedily finding the edge with most cycles through it, it greedily finds the source package which would break most cycles if it was profile built.

Generating a build order with less profile built source packages After implementing all the features above I now feel more confident to publish the current status of the tools to a wider audience. The following test shows a run of the aforementioned ./native.sh shell script. Its final output is a list of source packages which have to be profile built and a build order. Using the resulting build order, starting from a minimal build system (essential:yes, build-essential and debhelper), all source packages will be compiled which are needed to compile all binary packages in the system. The result will therefor be a list of source and binary packages which fulfill the following property:

• all binary packages can be built from the available source packages
• all source packages can be built with the available binary packages

I called this a "reduceded distribution" in earlier posts. The interesting property of this specific selection is, that it contains the biggest problem set of Debian when bootstrapping it: a 900 to 1000 nodes big strongly connected component. Here is a visualization of the problem: Source packages do not yet come with build profiles and the cross build situation can not yet be analyzed, so the following assumptions were made:

• the minimal build system can be cross compiled from nothing
• the reduced build dependencies harvested from Gentoo and supplied by Daniel Schepler, Patrick McDermott, Thorsten Glaser and Wookey are correct
• the current list of weak build dependencies can be dropped from any source package
• even when profile built, all source packages build all binary packages
• the 14 forcefully broken build dependencies do not significantly change the output in real life

The last point is about 14 build dependencies which were decided to be broken by the feedback arc set algorithm but for which other data sources did not indicate that they are actually breakable. Those 14 are dynamically generated by native.sh. If above assumptions should not be too far from the actual situation, then not more than 73 source packages have to be modified to bootstrap a reduced distribution. This reduced distribution even includes dependency-wise "big" packages like webkit, metacity, iceweasel, network-manager, tracker, gnome-panel, evolution-data-server, kde-runtime, libav and nautilus. By changing one line in native.sh one could easily develop a build order which generates gnome-desktop or really any given (meta-)package selection. All of native.sh takes only 80 seconds to execute on my system (Core i5, 2.5 GHz, singlethreaded). Here is the final build order which creates 2044 binary packages from 613 source packages.

1. nspr, libio-pty-perl, libmcrypt, unzip, libdbi-perl, cdparanoia, libelf, c-ares, liblocale-gettext-perl, libibverbs, numactl, ilmbase, tbb, check, libogg, libatomic-ops, libnl($), orc($), libaio, tcl8.4, kmod, libgsm, lame, opencore-amr, tcl8.5, exuberant-ctags, mhash, libtext-iconv-perl, libutempter, pciutils, gperf, hspell, recode, tcp-wrappers, fdupes, chrpath, libbsd, zip, procps, wireless-tools, cpufrequtils, ed, libjpeg8, hesiod, pax, less, dietlibc, netkit-telnet, psmisc, docbook-to-man, libhtml-parser-perl, libonig, opensp($), libterm-size-perl, linux86, libxmltok, db-defaults, java-common, sharutils, libgpg-error, hardening-wrapper, cvsps, p11-kit, libyaml, diffstat, m4
2. openexr, enca, help2man, speex, libvorbis, libid3tag, patch, openjade1.3, openjade, expat, fakeroot, libgcrypt11($), ustr, sysvinit, netcat, libirman, html2text, libmad, pth, clucene-core, libdaemon($), texinfo, popt, net-tools, tar, libsigsegv, gmp, patchutils($), dirac, cunit, bridge-utils, expect, libgc, nettle, elfutils, jade, bison
3. sed, indent, findutils, fastjar, cpio, chicken, bzip2, aspell, realpath, dctrl-tools, rsync, ctdb, pkg-config($), libarchive, gpgme1.0, exempi, pump, re2c, klibc, gzip, gawk, flex-old, original-awk, mawk, libtasn1-3($), flex($), libtool
4. libcap2, mksh, readline6, libcdio, libpipeline, libcroco, schroedinger, desktop-file-utils, eina, fribidi, libusb, binfmt-support, silgraphite2.0, atk1.0($), perl, gnutls26($), netcat-openbsd, ossp-uuid, gsl, libnfnetlink, sg3-utils, jbigkit, lua5.1, unixodbc, sqlite, wayland, radvd, open-iscsi, libpcap, linux-atm, gdbm, id3lib3.8.3, vo-aacenc, vo-amrwbenc, fam, faad2, hunspell, dpkg, tslib, libart-lgpl, libidl, dh-exec, giflib, openslp-dfsg, ppl($), xutils-dev, blcr, bc, time, libdatrie, libpthread-stubs, guile-1.8, libev, attr, libsigc++-2.0, pixman, libpng, libssh2, sqlite3, acpica-unix, acl, a52dec
5. json-c, cloog-ppl, libverto, glibmm2.4, rtmpdump, libdbd-sqlite3-perl, nss, freetds, slang2, libpciaccess, iptables, e2fsprogs, libnetfilter-conntrack, bash, libthai, python2.6($), libice($), libpaper, libfontenc($), libxau, libxdmcp($), openldap($), cyrus-sasl2($), openssl, python2.7($)
6. psutils, libevent, stunnel4, libnet-ssleay-perl, libffi, readline5, file, libvoikko, gamin, libieee1284, build-essential, libcap-ng($), lcms($), keyutils, libxml2, libxml++2.6, gcc-4.6($), binutils, dbus($), libsm($), libxslt, doxygen($)
7. liblqr, rarian, xmlto, policykit-1($), libxml-parser-perl, tdb, devscripts, eet, libasyncns, libusbx, icu, linux, libmng, shadow, xmlstarlet, tidy, gavl, flac, dbus-glib($), libxcb, apr, krb5, alsa-lib
8. usbutils, enchant, neon27, uw-imap, libsndfile, yasm, xcb-util, gconf($), shared-mime-info($), audiofile, ijs($), jbig2dec, libx11($)
9. esound, freetype, libxkbfile, xvidcore, xcb-util-image, udev($), libxfixes, libxext($), libxt, libxrender
10. tk8.4, startup-notification, libatasmart, fontconfig, libxp, libdmx, libvdpau, libdrm, libxres, directfb, libxv, libxxf86dga, libxxf86vm, pcsc-lite, libxss, libxcomposite, libxcursor, libxdamage, libxi($), libxinerama, libxrandr, libxmu($), libxpm, libxfont($)
11. xfonts-utils, libxvmc, xauth, libxtst($), libxaw($)
12. pmake, corosync, x11-xserver-utils, x11-xkb-utils, coreutils, xft, nas, cairo
13. libedit, cairomm, tk8.5, openais, pango1.0($)
14. ocaml, blt, ruby1.8, firebird2.5, heimdal, lvm2($)
15. cvs($), python-stdlib-extensions, parted, llvm-2.9, ruby1.9.1, findlib, qt4-x11($)
16. xen, mesa, audit($), avahi($)
17. x11-utils, xorg-server, freeglut, libva
18. jasper, tiff3, python3.2($)
19. openjpeg, qt-assistant-compat, v4l-utils, qca2, jinja2, markupsafe, lcms2, sip4, imlib2, netpbm-free, cracklib2, cups($), postgresql-9.1
20. py3cairo, pycairo, pam, libgnomecups, gobject-introspection($)
21. gdk-pixbuf, libgnomeprint, gnome-menus, gsettings-desktop-schemas, pangomm, consolekit, vala-0.16($), colord($), atkmm1.6
22. libgee, gtk+3.0($), gtk+2.0($)
23. gtkmm2.4, poppler($), openssh, libglade2, libiodbc2, gcr($), libwmf, systemd($), gcj-4.7, java-atk-wrapper($)
24. torque, ecj($), vala-0.14, gnome-keyring($), gcc-defaults, gnome-vfs($)
25. libidn, libgnome-keyring($), openmpi
26. mpi-defaults, dnsmasq, wget, lynx-cur, ghostscript, curl
27. fftw3, gnupg, libquvi, xmlrpc-c, raptor, liboauth, groff, fftw, boost1.49, apt
28. boost-defaults, libsamplerate, cmake, python-apt
29. qjson, qtzeitgeist, libssh, qimageblitz, pkg-kde-tools, libical, dwarves-dfsg, automoc, attica, yajl, source-highlight, pygobject, pygobject-2, mysql-5.5($)
30. libdbd-mysql-perl, polkit-qt-1, libdbusmenu-qt, raptor2, dbus-python, apr-util
31. rasqal, serf, subversion($), apache2
32. git, redland
33. xz-utils, util-linux, rpm, man-db, make-dfsg, libvisual, cryptsetup, libgd2, gstreamer0.10
34. mscgen, texlive-bin
35. dvipng, luatex
36. libconfig, transfig, augeas, blas, libcaca, autogen, libdbi, linuxdoc-tools, gdb, gpm
37. ncurses, python-numpy($), rrdtool, w3m, iproute, gcc-4.7, libtheora($), gcc-4.4
38. libraw1394, base-passwd, lm-sensors, netcf, eglibc, gst-plugins-base0.10
39. libiec61883, qtwebkit, libvirt, libdc1394-22, net-snmp, jack-audio-connection-kit($), bluez
40. redhat-cluster, gvfs($), pulseaudio($)
41. phonon, libsdl1.2, openjdk-6($)
42. phonon-backend-gstreamer($), gettext, libbluray, db, swi-prolog, qdbm, swig2.0($)
43. highlight, libselinux, talloc, libhdate, libftdi, libplist, python-qt4, libprelude, libsemanage, php5
44. samba, usbmuxd, libvpx, lirc, bsdmainutils, libiptcdata, libgtop2, libgsf, telepathy-glib, libwnck3, libnotify, libunique3, gnome-desktop3, gmime, glib2.0, json-glib, libgnomecanvas, libcanberra, orbit2, udisks, d-conf, libgusb
45. libgnomeprintui, libimobiledevice, nautilus($), libbonobo, librsvg
46. evas, wxwidgets2.8, upower, gnome-disk-utility, libgnome
47. ecore, libbonoboui
48. libgnomeui
49. graphviz
50. exiv2, libexif, lapack, soprano, libnl3, dbus-c++
51. atlas, libffado, graphicsmagick, libgphoto2, network-manager
52. pygtk, jackd2, sane-backends, djvulibre
53. libav($), gpac($), ntrack, python-imaging, imagemagick, dia
54. x264($), matplotlib, iceweasel
55. strigi, opencv, libproxy($), ffms2
56. kde4libs, frei0r, glib-networking
57. kde-baseapps, kate, libsoup2.4
58. geoclue, kde-runtime, totem-pl-parser, libgweather, librest, libgdata
59. webkit
60. zenity, gnome-online-accounts
61. metacity, evolution-data-server
62. gnome-panel
63. tracker

The final recompilation of profile built source packages is omitted. Source packages marked with a ($) are selected to be profile built. All source packages listed in the same line can be built in parallel as they do not depend upon each other. This order looks convincing as it first compiles a multitude of source packages which have no or only few build dependencies lacking. Later steps allow fewer source packages to be compiled in parallel. The amount of needed build dependencies is highest in the source packages that are built last.

I've been lucky in life. I've known several people who've killed themselves, but I've been friendly with them rather than friends with them. Their deaths have saddened me, but haven't left giant gaping holes that I have trouble imagining filling. I've also been lucky in that I'm not especially predisposed towards depression, and even luckier that the one serious episode I had ended well. That luck means I'm not qualified to speak authoritatively on the topic, but this seems like a good time to share. Most of 2002 wasn't great for me. Many of my friends had graduated and moved away. My PhD, which was what I'd spent the past few years of my life working towards, was going horribly wrong. The girl I liked didn't like me. Lots of individual small things that didn't fundamentally matter, but which build up into overall feelings of isolation and failure at life. I was spending an increasing number of days not leaving bed and not talking to people. It wasn't that I couldn't have fun - socialising was still pleasurable, and I didn't actively avoid people, but it was always tempered with the knowledge that this was temporary and I'd be returning to the unhappiness afterwards. I couldn't see any sequence of events that would turn my life around and restore my happiness. This was how life was, and this was how life was going to be. I'd irrevocably fucked up, and this was my future. Looking back, what strikes me is how reasonable this seemed. I could point at specific things that were making me unhappy, and nobody could argue that it was unreasonable to be unhappy about them. There wasn't any point in speaking to people about it, because what would they be able to do except agree that I was justifiably unhappy? I thought about suicide. Not seriously, because overall life still seemed worthwhile, but I could conceive of a level of further unhappiness that would make it seem like the best choice. I don't think it would ever have occured to me to speak to someone about it first. It seemed like it would be the same argument - I'm justifiably unhappy, I already feel like I'm letting my friends down, what could they do other than tell me that my feelings are wrong or make me feel even more guilty? So, when I saw exhortions for people to speak to someone if they felt suicidal, it seemed like they were talking to people who hadn't thought this through as well as me. It felt like I'd thought this all through carefully and rationally and come to a completely logical decision. If changes in circumstances and further consideration made it seem like suicide was the better choice, that would be because it was the better choice. Maybe other people weren't thinking about this as logically as I was. Maybe they'd have their minds changed by speaking to a friend or a professional. I wouldn't. Of course, with hindsight I was rationalising the way I already felt rather than making entirely rational decisions. I could have rationalised myself to death even though there were (in my case) straightforward ways to make my life better. In any case, I've no idea how close I ever got to that point. Things were at their worst in August - by September I had a new job and new house, and things just got better from there. In the end, the friends I was convinced could do nothing for me ended up giving me the opportunity to find gainful employment and made sure I had somewhere to live. Without them, things might have been different. As it was, I spent less than nine months depressed and it was still the most hellish experience of my life. The thought of returning to that state is terrifying. I was lucky. I might not be again. There's no terribly good moral here. If I'd known more about depression beforehand, I might have been able to identify what was happening to me and seek professional help. Other than that, I didn't learn anything about how to avoid or deal with depression. The experience didn't make me a better person. I've no advice for people in the same situation. The only thing I gained from it all was the realisation that if I'd felt any worse and knew that this was what I faced for the rest of my life, death might not have been the worst choice I had. Depression is a huge social problem and we deal with it badly. We refuse to talk about it, and when we do talk about it we mostly limit ourselves to platitudes about how things will get better or placing the blame on depressed people for not wanting to talk to those around them. Sometimes it doesn't get better. Sometimes talking to those around you will make things worse. People need to be aware of what the effects of depression are and get better at identifying it in others, rather than assuming that they'll be able to ask for help themselves. Society as a whole needs to be better at ensuring that professional support is there for people who need it. And, unless we can make massive improvements in our understanding of the causes of depression and effective mechanisms for countering it, we need to accept that it will cost us friends. Let's redirect the anger we feel at their choice to avoid a lifetime of misery into anger at the society that still hasn't done everything it can to help them.

comments

I recently added a formal interface to testrepository to enable cross-machine scaling of test runs. As testrepository is still a static scheduler, this isn t perfect, but its quite a minimal interface, which makes it easy to implement. I will likely evolve it in reaction to feedback and experience. In the long term I d love to have a super generic tool that matches that interface, so the project VCS copy of .testr.conf can just call out to it. However I don t yet have that, but I do have a simple by-hand implementation that I use to run nova s tests across my personal laptop, desktop and work laptop. Testr models this by assuming each test running process can be mapped to a single instance id (which could be a chroot, vm, cloud instances, ) and then running one or more commands in the instance, before disposing of it. This by hand implementation consists of 4 things:

1. A tiny script to rsync my source directory to the relevant places before I run tests. (This takes <2seconds on my home wifi).
2. A script to allocate instance ids (I just use ints)
3. A script to discard them
4. And a script to copy tempfiles onto the target machine and run a given command.

I do my testing in lxc containers, because I like my primary environment to be free of project-specific quirks and workarounds. lxc is not needed though, if you don t want it. So, to set this up for yourself:

1. on each host, make an lxc container (e.g. following) http://wiki.openstack.org/DependsOnUbuntu
2. start them all (lxc-start -n nova -d)
3. Make SSH config entries for the lxc containers, so you can get at them remotely. (make sure your host * rules are at the end of the file otherwise the master overrides won t work [and you might not notice for some time...]):

   Host desktop-nova.lxc
   # lxc addresses may be present on localhost too, so namespace the control
   # path to avoid connecting to the wrong container.
     ControlPath ~/.ssh/master-lxc-%r@%h:%p
     hostname 10.0.3.19
     ProxyCommand ssh 192.168.1.106 nc -q0 %h %p
   Host hplaptop-nova.lxc
   # lxc addresses may be present on localhost too, so namespace the control
   # path to avoid connecting to the wrong container.
     ControlPath ~/.ssh/master-lxc-%r@%h:%p
     hostname 10.0.3.244
     ProxyCommand ssh 192.168.1.116 nc -q0 %h %p

4. make a script to copy your nova source tree to each test location. I called mine sync

   #!/bin/bash           
   cd $(dirname $0)
   echo syncing in $(pwd) 
   (rsync -a . desktop-nova.lxc:source/openstack/nova --delete-after && echo dell done) &
   (rsync -a . hplaptop-nova.lxc:source/openstack/nova --delete-after && echo hp done)

5. Make sure you have the base directory on each location

   ssh desktop-nova.lxc mkdir -p source/openstack
   ssh hplaptop-nova.lxc mkdir -p source/openstack

6. Sync your code over.

   ./sync

7. And check tests run by running a few.

   ssh hplaptop-nova.lxc "cd source/openstack/nova && ./run_tests.sh compute"
   ssh hplaptop-nova.lxc "cd source/openstack/nova && ./run_tests.sh compute"

   This will check the test environment: we re not going to be running tests on each node via run-tests or even testr (because it gets immediately meta), but if this fails, later attempts won t work. Your test virtualenv is inside the source tree, so it is copied implicitly by the sync.
8. Decide what concurrency you want. For me, I picked 12: I have a desktop i7 with 4 cores, and two laptops with 2 cores each, and hyperthreads are on on all of them I m going to set a concurrency figure of 12 between the cores (8) and threads (16) counts, and possibly balance it more in future. A higher number assumes less contention between ALU s and other elements of the core pipeline, and I expect quite some contention because most of nova s unittests are CPU bound not I/O. If the test servers are not busy, I can always raise it later.
9. Create scripts to create / dispose / execute logical worker threads.
10. Creation. I call this instance-provision and all it does is find the lowest ints not currently allocated and return them.

    #!/usr/bin/env python
    import os.path
    import sys
    if not os.path.isdir('.instances'):
        os.mkdir('.instances')
    running_ids = os.listdir('.instances')
    count = int(sys.argv[1])
    top = count + len(running_ids)
    ids = [str(i) for i in range(top)]
    new = set(ids) - set(running_ids)
    for id in new:
        file('.instances/%s' % id, 'w').close()
    print(' '.join(new))

11. Disposal is easy: remove the file marking the instance as in-use.

    #!/bin/bash
    echo freeing $@
    cd .instances
    rm $@

12. Execution is a little trickier. We need to run some commands locally, and other ones by copying in temp files that testr has setup to the machine sshing to the remote machine, cd ing to the right directory, sourcing the virtual env, and finally running the command.

    #!/bin/bash
    instance="$(($1 % 4))"
    case $instance in
    [0]) node=
         local="true"
         ;;
    [1]) node=hplaptop-nova.lxc
         local=""
         ;;
    [2-3]) node=desktop-nova.lxc
         local=""
         ;;
    *)   echo "Unknown instance $instance" >&2
         exit 1
         ;;
    esac
    shift
    files=
    # accumulate files to copy
    while [ "--" != "$1" ]; do 
    files="$files $1"
    shift ; done 
    shift   
    if [ -n "$files" -a -z "$local" ]; then
        echo copying $files to node.
        for f in $files; do
            rsync $f $node:$(dirname $f) ;
        done
    fi  
    if [ -n "$local" ]; then
        eval $@
    else
        echo ssh to $node
        ssh $node "cd source/openstack/nova && . .venv/bin/activate && $@"
    fi

13. Finally, tell testr how to use this. (Don t commit this change to nova, as it would break other people). Add this to your .testr.conf.

    test_run_concurrency=echo 12
    instance_provision=./instance-provision $INSTANCE_COUNT
    instance_execute=./instance-execute $INSTANCE_ID $FILES -- $COMMAND
    instance_dispose=./instance-dispose $INSTANCE_IDS

Now, when you run testr run parallel, it will run across your machines. Just do a ./sync before running tests to get the code out there. It is possible to wrap all of this up via automation (or to include just-in-time provisioned cloud instances), but I like the results of still rough scripts here it strikes a good balance between effort, reliability and performance.

Review: Asimov's Science Fiction, April/May 2011

Editor:	Sheila Williams
Issue:	Volume 35, No. 4 & 5
ISSN:	1065-2698
Pages:	192

Williams's editorial this issue is about the tendency of SF to take a rose-colored view of the world, which on the surface seems odd given the tendency of recent SF towards dystopia. But she makes a good point that the portrayal of the past is rose-colored, linking that into the current steampunk trend. She doesn't take the argument quite as far as I'd like, but I'm glad to see editorials raising points like this. I'm inclined to think that a lot of the rose-colored frame of the past is because few of us want to read about real historic conditions at any length, even for edification, because the stench and discomfort isn't fun to read about. Silverberg's column is another discussion of programmatic plot generators, which mostly makes the point that plot ideas are the easy part of writing. James Gunn contributes an extended biography of Isaac Asimov that probably won't be new to long-time genre readers but may fill in some details (although it politely sticks to mostly flattering material). Spinrad's book review column is one of his better ones; it looks at two novels by China Mi ville and two by Ian McDonald and explores differences in world-building. Spinrad predictably makes the case in favor of science fiction with rules and against the New Weird, but the discussion along the way was worth reading. "The Day the Wires Came Down" by Alexander Jablokov: Speaking of steampunk, here's an example. There is even an airship, although the primary technological focus is suspended street cars. Jablokov postulates a city-wide transportation network of suspended carriages called telpher cars, along with a city built around the telpher cables: stores on roofs, windows displaying merchandise to passing cars, and even a history of heated competition and dirty tricks between competing telpher networks. The story is set, as the title would indicate, on the last day of the network. It's being shut down for cost, with some hints that progress is destroying something precious. There is a plot here, revolving around some mysteries of the history of the telpher network and the roles of several people in that history. But the story is primarily a celebration of old technology. It's a rail fan's story recast with a steampunk technology, featuring the same mix of fascination with mechanics and a sense that the intricate details are falling out of common knowledge (and perhaps usefulness). As a story, it's a bit slow-moving, but I enjoyed the elegiac tone. (7) "An Empty House with Many Doors" by Michael Swanwick: This is a very short story, more of an emotional profile, involving a man's reaction to the death of his wife. Oh, and parallel universes. It's sort of the inverse of Niven's classic "All the Myriad Ways." Similar to Niven's story, I found the idea vaguely interesting but the conclusion and emotional reaction unbelievable and alien. (5) "The Homecoming" by Mike Resnick: Resnick tends to yank on the heart-strings rather sharply in his stories, so I knew roughly what to expect when a father comes home to find his son is visiting. A son who, rather against his father's wishes, has been significantly altered to be able to live with aliens. Throw in a mother with serious dementia, and you can probably predict what Resnick does with this. Still, most of the story is a two-sided conversation, and I thought he succeeded in doing justice to both sides, even though one of them was destined to lose. (6) "North Shore Friday" by Nick Mamatas: Illegal Greek immigrants, a family-run system for getting them married off before the INS catch them, government psi probes and eavesdropping on thoughts, joint projects between computer and religion departments, secret government experiments, and even ghosts... this story is a complex mess, with numerous thoughts stuck into small boxes and scattered through the surface story. It's one of those stories where figuring out what's going on, and even how to read the story in a sensible way, is much of the fun. If you find that fun, that is; if not, it will probably be frustrating. I wished there was a bit more plot, but there's something delightful about how much stuff Mamatas packs into it. (6) "Clockworks" by William Preston: This is a prequel to Preston's earlier "Helping Them Take the Old Man Down". Like that story, it's primarily a pulp adventure, but layered with another level of analysis and thoughtfulness that tries to embed the pulp adventure in our understanding of human behavior and the nature of the world, although this one stays a bit more pulp than its predecessor. As with Preston's other story, we don't get directly in the head of the Old Man (here, just called the man, but identifiable from clues in both stories as Doc Savage); instead, the protagonist is a former villain named Simon Lukic who the man hopes to have fixed by operating on his brain. The undercurrent that lies beneath a more typical pulp adventure is the question of whether Lukic is actually healed. I think there was a bit too much daring-do and human perfection, but it's a perfectly servicable pulp story with some depth. (6) "The Fnoor Hen" by Rudy Rucker: If you've read any of Rucker's work before, you probably know what to expect: a mind-boggling blizzard of mathematically-inspired technobabble that turns into vaguely coherent surrealism. (You can probably tell that I'm not much of a fan, although the clear good humor in these stories makes it hard to dislike them too much.) There's a mutated chicken and some sort of alternate mathematical space and then something that seems like magic... I'd be lying if I said that I followed this story. If you like Rucker, this seems like the sort of thing that you'd like. (4) "Smoke City" by Christopher Barzak: At the start of this story, I thought it was going to be an emotional parable about immigration. The progatonist lives two lives: one in our world, and one in the Smoke City of industry, a world of hard labor, pollution, and little reward, with families in both. But nearly all of the story is set within Smoke City, and the parable turns out to be a caustic indictment of industry and its exploitation of labor. I kind of wish Barzak hadn't used rape as a metaphor, but when the captains of industry show up, I can't argue with how deeply and accurately the story shoves in the knife. There isn't much subtlety here, but it's still one of the better stories in this issue. (7) "A Response from EST17" by Tom Purdom: I'm very happy to see Purdom's writing appearing regularly. His stories are always quiet and matter-of-fact, and at first seem to miss emotional zest, but they almost always grow on me. He lets the reader fill in their own emotional reactions to events, and there's always a lot going on. This story is a first-contact story, except that the "humans" here are not human at all. They're automated probes sent by two separate human civilizations, with different programming and different governance algorithms, and they quickly start competing negotiations. The aliens they've discovered similarly have factions, who start talking to the different probes in an elaborate dance of gathering information without giving too much away. The twist is that this pattern has replayed itself many times in the past, and information itself can be a weapon. I enjoyed this one from start to finish. (7) "The One That Got Away" by Esther M. Friesner: Friesner is best known, at least to me, for humorous fantasy, and this story is advertised as such from early on. The first-person protagonist is a prostitute in a seaside town. She's bemused to finally be invited over by a sailor who's been eyeing her all evening, but that sailor has something else in mind than normal business. For much of this story, the fantasy element is unclear; when it finally comes, it was an amusing twist. (7) "The Flow and Dream" by Jack Skillingstead: This is a mildly interesting variation on the old SF story of hibernating humans (on a generation ship or elsewhere) waking up to a transformed world. Here, it's not a ship, it's a planet, and the hiberation was to wait for terraforming rather than for transit. The twist comes from an excessively literal computer and the fun of putting together the pieces. Sadly, the story trails off at the end without much new to say. (5) "Becalmed" by Kristine Kathryn Rusch: "Becalmed" takes place immediately before "Becoming One with the Ghosts" and explains the incident that created the situation explored in that story. The first-person protagonist of "Becalmed" is a linguist, an expert in learning alien languages so that the Fleet can understand the civilizations that it runs across. But something went horribly wrong at their last stop, something that she's largely suppressed, and now she's confined to quarters and possibly in deep trouble. As is the ship; they're in foldspace, and they have been for days. "Becalmed" is structed like a mystery, centered around recovering the protagonist's memories. It's also a bit of a legal procedural; the ship is trying to determine what to do with her and to what degree she's responsible. But the heart of the story is a linguistic and cultural puzzle. This is another great SF story from Rusch, whose name on a cover will make me eager to start reading a new magazine. I love both angles on the universe she's built, but I think I like the Fleet even better than the divers. The Fleet captures some of the magic of the original Star Trek, but with much more mature characters, more believable situations, and a more sensible and nuanced version of the Prime Directive. Rusch writes substantial, interesting plots that hold my interest. I'd love to see more like this. (8) Rating: 7 out of 10

My wife likes watching DVDs a lot, which is why she borrows many from the public library. The problem is that many of those were handled by people who have no clue whatsoever about the proper handling of DVDs, which means that they are in a state ranging from moderately scratched in the best cases to properly scarred in the worst ones... xine or VLC fail miserably to read such DVDs. What's worse is that failures happen right in the middle of DVDs, which is quite frustrating !That's why I've started quite some time ago to write a small C wrapper around libdvdread to first copy the DVD files onto the disk and then read them from there. It was also an easy solution to copy files to a driveless netbook to watch the DVD there. With time, this project evolved into a neat C++ command-line utility that's able to keep track of badly read sectors and come back to them later.Today is the 0.0 release of dvdcopy. You can get the tarball there. It follows the standard autoconf dance:

~ ./configure
~ make
~ make install

Using it is as simple as running:

~ dvdcopy /dev/dvd MyMovieDirectory

If everything went fine, you're set, and you can open the directory created with vlc or xine:

~ xine dvd:// pwd /MyMovieDirectory

(yes, xine needs the full path, I haven't checked with VLC, but from what I remember, you can open DVD directories directly from a menu).If you're not that lucky and dvdcopy found some bad sectors, try reading again using

~ dvdcopy --second-pass /dev/dvd MyMovieDirectory

I found that in many cases, one or two second passes get all the sectors back. While that wasn't really the intended purpose, the fact that most copy protection schemes (in addition to CSS) are implemented using bad sectors make dvdcopy quite resistant to them.I have no plan of making a Debian package for it at the time being, but if there is demand for that, I may reconsider. Enjoy !

I've been reading "The Millionaire Next Door" and have so far found it to be a pleasant book with a good message: don't waste your money on silly things and appearance (fancy suits, fancy cars, expensive boats, etc...), save what you do earn consistently and constantly, invest wisely, and so on. Wikipedia has a good summary:
http://en.wikipedia.org/wiki/The_Millionaire_Next_Door One of the things I like about it is that it focuses on "ordinary" wealthy people, those with a million or more in the bank, but not the Warren Buffets or Bill Gates types that are extreme statistical outliers. There are plenty of people in the US who have done well by themselves by slowly but surely putting together enough money to be financially independent, without, however, being in the spotlight. As the book says, these are the kind of people who maybe own a local chain of businesses doing something fairly ordinary, but doing it well enough to succeed. They may very well not live in a fancy house, nor drive an expensive car, or otherwise outwardly draw much attention to themselves. The world of software does not revolve around "dressing for success" (you noticed?), but we do tend to focus on the "big winners". Gates, Jobs, Zuckerberg, Larry & Sergey, Larry Ellison, and so on are the stars of the show. Of course, the economics of software being what they are, instances of winner-take-all markets with one big fish and a lot of also-rans are not uncommon. However, that is not the only story, and I think it'd be interesting to know more about those in our industry who have accumulated significant wealth, yet are not the guys with more money than they could possibly ever spend on things that aren't, say, country-sized chunks of real-estate. I'm guessing they'd fall into these categories:

• Highly paid workers who have consistently saved over the years. There are examples in the aforementioned book about people with relatively low salaries who happened to be very frugal and invest well (and have had some luck in their investments too). These people would probably tend to be older, as it takes a while to save up that kind of money, and since this industry is so young with so much turnover, I would not think there would be a lot of people out there like this, but who knows, maybe there are a bunch of IBMers with this kind of story.
• Those who got in on the right IPO, like Google or Facebook or something like that. These events not only generate billions for those at the top of the heap, but for the right person at the right place at the right time, can mean significant wealth even without being in the upper echelons of the company. My suspicion is that this kind of IPO, where everyone cashes out, is not common enough to have a lot of people in this category, but who knows, maybe it adds up over the years.
• Those who own or started software firms that do something that's not very visible, but nonetheless dominates some particular niche. This is where I'd guess most of them would be, but I certainly have no data or even anecdotes to back this up.

It'd be very interesting to gather some actual data on this, although I'm not in a position to do so myself - I wouldn't even really know where to start. As I age, I think the third category has begun to seem appealing in many ways - I'm simply not cut out for the Big Company life, and I'm not interested in living in Silicon Valley and going "all in" on the latest startup - I already did that, and while it was fun and I don't regret it, it's not the kind of thing I'd want to do now that I'm married and have kids. Incidentally, this more relaxed, under the radar approach is exactly what is expoused in one of my favorite books of the past few years, Start Small, Stay Small. Edit : I finished reading the book and reviewed it here: http://davids-book-reviews.blogspot.com/

spin is the program that I use to generate all of my web pages, including this journal, from a bespoke macro language. I've been slowly evolving it since 1999 and really do intend to turn it into a proper Perl module one of these days.... My web site was one of the few places left that I was still using Subversion, and I finally decided to do something about that during this vacation. The main reason why it wasn't trivial was that I was using Subversion Id strings to get a more accurate last change date for files rather than just using file timestamps. (Nit-picky, I know, but if I'm not allowed to be nit-picky on my own web site, where can I be?) This release recognizes when the source directory is a Git working tree and supports determining the last modification date from git log instead (by way of the Git::Repository Perl module). I also updated all my various (not released) helper scripts for maintaining the site to use Git rather than Subversion, which mostly involved changing svn to git and removing some fiddly bits that Git doesn't require. Of course, this means that I also removed the \id thread commands on all my pages, which in turn means that I just updated the last modification date of every file, making it somewhat hard to test since every thread-generated page on my site has now been modified today. *heh* You can get the latest release of spin from my web tools distribution page. I also released version 1.44 of my release utility, which I use to help automate some of the work of generating software releases. The change there is to use Git rather than Subversion to commit changes to the .versions file that drives some of the software-related parts of my web site. You can get the latest version of release from my scripts page.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1,SHA512
I am transitioning GPG keys from an old 1024-bit DSA key to a new
4096-bit RSA key.  The old key will continue to be valid for some
time, but I prefer all new correspondance to be encrypted to the new
key, and will be making all signatures going forward with the new key.
This transition document is signed with both keys to validate the
transition.
If you have signed my old key, I would appreciate signatures on my new
key as well, provided that your signing policy permits that without
re-authenticating me.
The old key, which I am transitioning away from, is:
  pub   1024D/DFCA34A3 2007-10-07 [expires: 2013-10-24]
    Key fingerprint = FE23 BE62 DF18 72FB C58D  D637 FFAE 0427 DFCA 34A3
The new key, to which I am transitioning, is:
  pub   4096R/95206DD3 2013-01-02 [expires: 2016-01-02]
    Key fingerprint = DF0B FDFF 4A4D DA01 7944  1B8F 6906 4B01 9520 6DD3
To fetch the full new key from a public key server using GnuPG, run:
  gpg --keyserver keys.gnupg.net --recv-key 95206DD3
If you have already validated my old key, you can then validate that
the new key is signed by my old key:
  gpg --check-sigs 95206DD3
If you then want to sign my new key, a simple and safe way to do that
is by using caff (shipped in Debian as part of the "signing-party"
package) as follows:
  caff 95206DD3
Please contact me via e-mail at <richih.mailinglist@gmail.com>
if you have any questions about this document or this transition.
    Richard Hartmann
    richih.mailinglist@gmail.com
    2013-01-03
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
iEYEARECAAYFAlDk27gACgkQ/64EJ9/KNKMGDgCeLyQy6yhEu3OB8UarJ6LYOfVY
ihgAnRSbYMrVxK+28souMHiLx4y0rSQ+iQIcBAEBCgAGBQJQ5Nu4AAoJEGkGSwGV
IG3TcbYP/je8mu+MCIiNNY8w/gN+/9S8gEGWd2p9LeN71xpCuSpBVSevAxxStV9R
2XRTfI8tf/nk27y+4073IscUTPaBkb2IhvIfCnFmOtZjP9slS2JMHOylqH3KLulQ
Cm/IbpKCAwTmQtRMzFxKoRjNPJDCVzZYpoD9r7TmIrs+iKtDQ/w/zQmtwEFWncYK
2JJqVEd+qN9REpWlKDDifcUDPGiugsICelAaUF8VMvp2ouVxQTGbz53IdAgvrX9G
vM+BAfel3W73nfF3L+08vgad0BGGMnEWcfcZ6r+DKu2WXgC8AXT0Y32sD8TiGJSd
JrWMqwec1iOd6lfOpFDdhSCFyMl5r1NiOLw5iNsETHYjysW2N1wZCicmF/8ukm/y
EpIXf3z8vtLrofGLN1jTC34Jugts8C5NNRil195Hkim36t2F97OJswNLhxt7nQ41
vhyXaBK3kwOZ1nVp6xM/yMnQzi5eVO1W/BE8Wr27O6pe3KqEx0Lv8zBzQjj4ly64
shyNRY3gzxs/KZ3p/3nrSn1AYOXIuCM3nILh89qQ/ynotrqYmpFkUorUSSOvraoQ
LN5IPH0wh8F2eDUCui2fGDEmnQ5S5mlHnIZmL3gkPVGLhNaoFQh0R8UwUO6HIxMQ
cTxv5TGL1YWtIvl0bvlEXK6MA4g0ZBXk15JZkWUD84zycr5cb79d
=DFzt
-----END PGP SIGNATURE-----

Review: City of Diamond, by Jane Emerson

Publisher:	DAW
Copyright:	March 1996
ISBN:	0-88677-704-6
Format:	Mass market
Pages:	624

The setting is deep into the star-faring future of humanity. Humans have settled many other worlds, thanks largely to technology from a race named the Curosa. There is a thriving network of settlement based around a sort of warp conduit network. But, standing apart from most human settlements, are three huge ships: the City of Opal, the City of Pearl, and the City of Diamond. Their inhabitants are devout members of the Redemptionist faith, a merging of Christian (primarily Catholic) belief and Curosa religion. Their ships are gifts from the now-departed Curosa, featuring drives unknown to the rest of humanity that allows them to ignore the normal transport network. They are meant to witness and spread their religion, but now, six centuries later, they primarily trade. Much of this is not apparent at the start. City of Diamond opens with Adrian Mercati, chosen heir of the current dying Protector, dodging people in what seems to be an oddly-organized city. Impudent, occasionally flippant, utterly charming, and in love with danger, he inherits the position of Protector shortly after we meet him. Emerson slowly reveals more of the nature of the ship while Adrian is shocking (and charming) the nobility and the church, and consorting with demons. Or, at least, one demon: Tal, an entirely human-appearing creature except for his eyes, who the Redemptionists claim is born without a soul. Slowly, it becomes clear that he's actually the offspring of an alien and a human, a mating of species whose children cannot socialize as human and behave in ways that seem like human sociopaths. Tal is no exception, but his interests align with Adrian's for the time. To those two viewpoint characters, Emerson adds four more: a low-rank administrator named Spider who works for Tal after Tal saved him from death by recycling, Adrian's politically-arranged bride-to-be from the rival (and even more religious) ship City of Opal, a guard from a French-descended slum on City of Opal, and a dangerously honorable and literal mercenary from an off-shoot culture of humanity that's now only a legend. This is a book that tends to sprawl. There's quite a lot going on, starting with the negotiations between City of Diamond and City of Opal to put to rest the remnants of a civil war and leading to the search for a legendary Curosa artifact on the planet they're both visiting. All of the characters have their own agendas, their own history, and their own goals, and are pursuing them in parallel and occasionally in opposition. Emerson takes the time to let all of this build. Despite that, there is very little in this book that drags or bores, and that's due primarily to the strength of the characters. Every one of the viewpoint characters is unique, bringing a different perspective to the story, and every one of them changes and grow over the course of the story. They also play off of each other in delightful ways. This is a book that makes full use of the fact that three characters make seven character interactions. For example, Tal by himself is logical but just slightly askew, Tal seen by Spider is uncomfortable and worrisome, Tal and Adrian is a dance of mutual amusement and respect, and Tal and Keylinn finding points of commonality is just delightful. I did wish that there was a bit more Keylinn and Tal and a bit less of Spider, but once Emerson built her world, introduced her characters, and got the story moving, there aren't many weak pieces. Part of the fun is figuring out the world background. There are almost no infodumps here; Emerson just starts telling stories of characters in the surroundings in which they find themselves, and the reader is left to piece things together from background description, the occasional explanation from one character to another, and parallels with long-standing SF tropes. It's not a puzzle book, but it is a book of where the reader is dropped as an alien into a fully-developed culture. In both that way and in the use of nobility, religion, and formal hierarchy against a starfaring backdrop, City of Diamond reminds me of the sort of classic space opera that led up to Star Wars but isn't seen as much these days. It's also funny. I was unsurprised to discover that Emerson is a pseudonym for Doris Egan, who (in addition to writing some other SF) is an occasional writer and producer for House. The humor isn't entirely obvious at the start, although Adrian frequently provokes a grin from the start of the book, but Tal's perceptive but obliviously blunt comments become more and more fun as the book progresses, and both Iolanthe (Adrian's bride) and Keylinn (the mercenary, sort of) are a delight. I want to see more of both of them, but sadly the book ends just as they (particularly Iolanthe) are hitting their strides. And that's the biggest problem with City of Diamond: it's a failed (or at least postponed) trilogy. As with Walter Jon Williams's Metropolitan and City on Fire, there were supposed to be more books, but apparently either the publisher lost interest or other projects took precedence. The story does come to something of a conclusion, but most of the major story arcs aren't resolved. Emerson is still bringing more guns onto the set late in the book. None of the villains truly get their just desserts, nor do most of the character arcs fully resolve, although friendships and more have formed satisfyingly by the end of the book. There is obviously far more to both these characters and this universe, including an entire ship that was never visited and an alien whose presence should prove startling. This is a great book: warm, funny, full of interesting characters, and with a nicely complex background universe. If it were complete in itself, or if the sequels had materialized, I would recommend it unreservedly. As is, you will have to weigh your dislike of unfinished stories against its stand-alone merits. But I enjoyed it a great deal. Rating: 7 out of 10

I'm thankfully in a place where I can afford to give money to charity, and I've slowly built up a set of charities that I give money to. But I've never been that systematic about it. I've had a vague target for how much I want to give, I add new charities when they seem neat, I'll rarely drop a charity when it doesn't seem to be effective, and I probably spread my money out too much. Recently, I followed a chain of blog references and discovered GiveWell, a site that's devoted to doing more in-depth analysis of charities than the basic financial analysis done by sites like Charity Navigator. In particular, they do literature surveys to try to dig into the effectiveness of particular interventions and take a closer look at how effectively a charity can use new donations. After reading quite a bit on their site and some related sites, I sat down yesterday and did a comprehensive review of my goals and principles and then put together a new plan. Some of that may be interesting to other people, so I'll write up that process here. This will be somewhat US-centric, since charities vary a lot by tax regime, but I think some of the general principles will still apply elsewhere. (But most of my specific examples will be US charities.) First, a few basic principles about charitable giving:

1. Charity goals cannot be evaluated uniformly. Different types of organizations need different criteria. I've found that charities divide generally into ones that are trying to accomplish a specific task and ones that are trying to create political change. For the former, small and quiet organizations are often the best. For the latter, the charity has to get to a certain size before it's part of the conversation, and that involves a different set of tradeoffs.
2. When you care about efficiency (number of lives saved or quality of life improvement per dollar spent), don't give money to organizations you've heard about via means other than word of mouth or third-party evaluations. This is for a simple reason: if the charity has successfully made you aware of it, that means they're advertising themselves to you. Advertising generally costs quite a bit of money. That's money they're not spending on whatever they're trying to accomplish. Excellent examples are many of the health research and education charities, such as the American Heart Association and the American Lung Association, or the massive disaster relief organizations like the American Red Cross. These are not efficient organizations. If you're trying to get the most impact for your dollar, this isn't a good place to spend it. As a special case of this, please don't ever give money to any organization that produces TV commercials showing starving children in Africa. If the commercial makes you want to give money, that's great; just don't give it to the organization on TV. TV commercials are extremely expensive, and those organizations are notorious for being some of the least efficient charities in existence (in some cases bordering on scams). You'll be lucky if 50% of your money actually goes to Africa, and the money that is spent probably isn't spent wisely.
3. If your donation gets you a membership that comes with a glossy magazine, remember that means that the organization is spending your money producing that magazine. This is counted as program funding, not administrative costs, in sites like Charity Navigator because the charity probably counts the magazine as part of their educational mission. If you want to subscribe to the magazine and enjoy it, there's not necessarily anything wrong with this. But if you were giving them money to do something else, be aware that money is being used to produce the magazine (and the Christmas cards and the mailing labels and the tote bag) instead of doing things that you might consider program funding. For example, I stopped giving money to the Audubon Foundation because I want my environmental giving to go directly to protecting the environment, not towards printing calendars and making a glossy magazine full of bird pictures. As a counter-example, I consider the Southern Poverty Law Center to be one of the most effective US political charities, despite the fact that they produce a glossy magazine. That's because a large part of their mission is investigative reporting and research, and that magazine is how they publish that research. In this case, I both want to read the magazine and consider the magazine something that I'm happy to fund.
4. Efficiency isn't everything. For some types of charities, the charity has to be huge in order to be effective. This is particularly true of political charities. For example, the ACLU is not a particularly efficient charity and has several fund-raising practices that I dislike. However, what the ACLU does requires that they have extensive media access and respect from legislators and the judicial system. It cannot be done effectively by a smaller and more efficient organization. Similar principles apply to a lot of political and environmental charities, and to some international aid charities such as Doctors without Borders (M decins Sans Fronti res).

If you're looking to increase the efficiency and effectiveness of your giving according to objective measures, I highly recommend reading through GiveWell's site. They recommend charities that avoid all the pitfalls above. You've probably never heard of them because they don't spend their money on advertising and marketing. However, keep in mind that efficiency isn't everything. With those principles in mind, and after doing a lot of reading on GiveWell, I redid my charity plan by dividing my giving up into five categories with different goals. Services. There are some things that I use that I think should be paid for by tax dollars but aren't, and are instead funded by charitable donations. I don't consider this charity in the same sense as the rest of this list. Rather, I'm paying for what I use plus some extra so that other people who can't afford it can use the service for free. Efficiency doesn't matter as much because there's usually only one organization supporting the thing that I personally use. Examples include my local PBS (public broadcasting) station, the non-profit that works with city government to support the local library system, and the trust that helps maintain the state parks where I go on vacation. Education. This is a special category for me since I regularly give money to the community college I attended. I know the region and the college and have some idea of what they need and what matters to the students there, so I can make specific choices. I give all the money in this category to the same place. Free software. Here, I know that my donations won't be used as efficiently in terms of quality of life as they could be if sent somewhere else, but this is my community and I want to support it because it's my community. Here, as with education, I have a lot more data and can pick and choose places that I think can use my money. Examples here are the Ada Initiative and the Free Software Foundation. Politics. This is the hardest area for efficiency, since it's so difficult to measure effectiveness in any organization where much of the goal of the organization is to persuade politicians or the general public. I try to measure effectiveness by how many concrete actions the organization takes (rather than press releases), which is easiest for organizations like the ACLU or the EFF that file lawsuits or legal documents and requests. As mentioned above, I think the standout in this area is the Southern Poverty Law Center, but political charities are so targeted that I give to multiple charities to cover my range of interests. I've found environmental charities particularly difficult to evaluate, but the one that appears the most effective to me so far is the Environmental Defense Fund, so that's where I concentrate my environmental money. I used to give to small political charities but have given up on that as basically useless, at least for national politics. I think the charity has to be of a minimum size to get anyone in government to listen to them. Poverty and Health. This is where efficiency matters much more than size. GiveWell's analysis is very interesting here, and sadly shows that most interventions either don't work or at least can't be proven to work. Here, I abandoned nearly all of my traditional large charities and have gone almost entirely with GiveWell's recommendations, with two exceptions. I still cycle money through Kiva (a microfinance support charity) because I have a bit of an emotional attachment to it, although microfinance is looking increasingly suspect as an effective charity method and I may yet drop them. And I still give money to Doctors without Borders because, despite being less efficient, I think their size and history has won them an international credibility that lets them get into areas that smaller, more efficient charities wouldn't be able to help. One of GiveWell's recommendations is my new favorite charity: GiveDirectly. All aid charities run a serious risk of having a colonialist bent, where rich countries come into poor countries and build things or tell them how to do things following the priorities of the rich countries. (This is one of the reasons why I prefer medical charities, since they're less susceptible to this.) GiveDirectly identifies the poorest people in a region (using a very transparent process) and transfers money to them directly using the M-Pesa cell system to spend however they choose, with no strings attached other than some due diligence to protect against fraud. This is refreshingly non-paternalistic and makes me far more comfortable than typical aid projects. Originally, I was going to assign money into those buckets and then break the buckets down further, but I ended up not quite doing that. The services bucket is more driven by the services I use than by an even allocation. Of the rest, it goes approximately three shares to poverty and health, two to politics, and one each to education and free software. I think I've managed to eliminate most of the places where I'm giving money to multiple organizations that do the same thing, but I'm still spreading my money rather widely and could stand to focus it more. That's hard, though; there are a lot of different things that I care about, and political charities in particular aren't very interchangeable. I'm currently not giving locally very much, which might be worth changing. The easiest way to do so would be to find a good local food bank and start donating to them (giving cash, not food). For a more comprehensive list of charities I support, see my charities link page.