Search Results: "xam"

28 December 2012

Richard Hartmann: Release Critical Bug report for Week 52

The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the squeeze cycle ?
Week Squeeze Wheezy Diff
43 284 (213+71) 468 (332+136) +184 (+119/+65)
44 261 (201+60) 408 (265+143) +147 (+64/+83)
45 261 (205+56) 425 (291+134) +164 (+86/+78)
46 271 (200+71) 401 (258+143) +130 (+58/+72)
47 283 (209+74) 366 (221+145) +83 (+12/+71)
48 256 (177+79) 378 (230+148) +122 (+53/+69)
49 256 (180+76) 360 (216+155) +104 (+36/+79)
50 204 (148+56) 339 (195+144) +135 (+47/+90)
51 178 (124+54) 323 (190+133) +145 (+66/+79)
52 115 (78+37) 289 (190+99) +174 (+112/+62)
1 93 (60+33)
2 82 (46+36)
3 25 (15+10)
4 14 (8+6)
5 2 (0+2)
6 release!
Graphical overview of bug stats thanks to azhag:

Petter Reinholdtsen: A Christmas present for Skolelinux / Debian Edu

I was happy to discover a few days ago that the Skolelinux / Debian Edu project also this year received a Christmas present from Another Agency in Trondheim. NOK 1000,- showed up on our donation account December 24th. I want to express our thanks for this very welcome present. As the Debian Edu / Skolelinux project is very short on funding these days, and thus lack the money to do regular developer gatherings, this donation was most welcome. One developer gathering cost around NOK 15 000,-, so we need quite a lot more to keep the development pace we want. Thus, I hope their example this year is followed by many others. :) The public list of donors can be found on the donation page for the project, which also contain instructions if you want to donate to the project.

Russ Allbery: Review: Perl Best Practices

Review: Perl Best Practices, by Damian Conway
Publisher: O'Reilly
Copyright: July 2005
Printing: April 2011
ISBN: 0-596-00173-8
Format: Trade paperback
Pages: 492
I had a rather bad first reaction to this book or, rather, the existence of this book. Some time ago, I started hearing about a new program named perlcritic that served as a lint program for Perl and decided to give it a try. I was taken aback by the flood of diagnostics, but even more taken aback by the fact that most of them were apparently explained only by a reference to a book that I didn't own. It struck me wrong in an open source project and gave me a negative reaction towards the book, which lingered for years. That was exacerbated by discovering that it recommended using use English in some circumstances, something that I consider to be a horrible idea. So, when a coworker recommended it and suggested I really should read it, I was dubious. I'm glad I gave it a chance, since it turns out that nearly all of my objections were based on misunderstandings or false information. (Well, except for the use English part, but more on that in a moment.) Perl Best Practices does provide a lot of advice, explaining that flood of diagnostics, but perlcritic also warns about the same thing repeatedly, inflating a handful of issues into a lot of messages. Most of the recommendations in Perl Best Practices were things I was doing already; most of the rest are entirely sensible. Also, perlcritic does come with complete explanations of its diagnostics once you know where they're hidden (the perldoc of Perl::Critic::Policy modules whose names you can get at by insisting perlcritic divulge them). And, even more delightfully, once I got past my objections, I discovered that Perl Best Practices is one of the best books on coding style that I've read. Coding style is a notoriously difficult subject, since nearly everyone is convinced their personal style is the best (or at least that they don't want to change). Conway acknowledges that up-front, and acknowledges that some things are just arbitrary and consistency with any rule set is better than following some specific rule set. He then spends surprisingly little time on the sorts of things that dominate most coding style discussions (indentation, spacing, commenting, etc.). Instead, this is primarily a book about how to write efficient and transparent Perl code, one that focuses much more on the content and structure of that code than on how it is indented. The best part is that there are no undefended rules here. Perl Best Practices contains 256 separate guidelines, some quite general ("Provide an optimal interface, rather than a minimal one") and some quite specific ("Pass constructor arguments as labeled values, using a hash reference"), but every one of them has a supporting argument and specific reasons behind it. Conway backs up every recommendation with code samples and practical scenarios showing the pluses and minuses of different approaches. This is a book unconcerned with being an authority; rather, it reads like optional advice from a good mentor, complete with insightful discussions on each point raised. You may come away from it disagreeing with Conway's specific recommendation (and I did, in several places), but you'll almost certainly be better-informed about the problem for having read it. Plus, this book is just fun to read. Admittedly, I love this sort of thing; I spend lots of time thinking about coding style in any language, and a lot more time refactoring coding style when working on my own projects to try to achieve clarity and simplicity. But even without sharing my love of the material, I think this book will be a delight for any practicing Perl programmer who hasn't yet read it. Conway clearly has a wealth of experience and years in the trenches of working with other people's Perl code. The examples he draws on are brief, comprehensible, and illuminating. He also gave me a tour of a variety of helper modules and libraries that I didn't know existed. List::Util is the most prominant example; this incredibly helpful supply of list manipulation functions that I've always half-known I wanted but never bothered to write is now part of Perl core (as of 5.7.3), so there's no reason to avoid it. You too can stop using grep when you actually mean first, or rewriting max in ad hoc ways. That does raise two of the rare flaws in this book, though. One is that I disagree (partially) with Conway's willingness to recommend non-core modules. He is a particular champion of the Readonly module over either normal file global variables or use constant, for very good reasons. But making Readonly a prerequisite for all other Perl code that one releases seems a bit annoying (speaking as someone who used to maintain local Perl installations). Maybe this isn't as big of a deal in these days of ubiquitous Linux distributions and willingness to use the CPAN shell, but I still dislike releasing code that depends on a half-dozen (or more) other CPAN modules for reasons of coding convenience and minor readability rather than core functionality. (His advocacy of Sub::Install raised similar objections.) The second flaw is not something Conway could have done anything about: this book is from 2005, and it's now 2012. Now, the Perl ecosystem is now quite mature, so that hampers the book much less than it would a corresponding book about Ruby or even Python. I also aim for compatibility with the oldest thing out there I know someone is still running, so I'm still hesitant to require anything newer than Perl 5.8. But still, this book was written seven years ago, and Perl is now at 5.14. In the meantime, we've seen core language features like given/when added, the addition of state variables, the need for substantial reworking of how Unicode I/O is handled, the addition of the new // operator, and the new package syntax, among many other things. The Perl taught here is a bit dated, and I would love to hear Conway's opinion about when (or if) to use say. Maybe someday we'll get a second edition. There are two other points that I do have to argue with. One is that Conway does advocate, in a limited way, use English. I could almost go along with his recommendation to use it only for those variables that one has to look up, except that in nearly every case there are other options and one can take the much better approach of simply not using those variables at all. But he then writes the remaining examples in the book using $EVAL_ERROR rather than $@, and I'm sorry, but no. One could not construct a more frustrating annoyance for an experienced Perl programmer than sending them on a futile hunt for where the global variable $EVAL_ERROR is defined. One can debate the merits of Larry Wall's original decision to use variables like $_, @_, $?, $!, and $?, but for better or ill this is Perl, and marginal readability gains for inexperienced programmers do not warrant confusing experienced programmers. This is particularly true when the use English variables are completely indistinguishable from normal global variables, and not infrequently look like a global variable that a programmer might choose. The second is more of an amusement that a real disagreement. Conway rightfully points out that Perl OO, and particularly the way that Perl stores object instance data, is awkward, and none of the commonly-used techniques are particularly satisfying. He then introduces a completely new way of doing instance data that, while having some advantages, has a very strange syntax and is utterly unlike how any OO code one finds in the wild is written. While the discussion of the problem that he's trying to solve is still interesting, I think the solutions he proposes will be more of a curiosity than a style that most programmers will adopt. But, those disagreements aside, I thoroughly enjoyed this book. I found it so engrossing that I kept reading it straight through a long train ride rather than switching to fiction halfway, like I do with most programming books. I've since recommended it to several other coworkers, and we're in the process of revising our internal Perl coding style based heavily on it. It's the sort of book that I expect to re-read several times to refresh my memory of Conway's analysis. If you've been avoiding this one due to misconceptions like I had, or because you think it might be too dry, or even just because you're afraid it's outdated, I strongly recommend it. If you're worried about the price of a printed (or electronic) book on this topic and you do Perl programming as part of your job, see if your employer will buy you a copy as training. It's worth it. I've been writing Perl professionally for more than fifteen years, and I learned a ton from this book. (And if it inspires you to change your coding style, take a look at perlcritic. It requires some configuration, and getting started is quite frustrating, but it reliably detects a surprisingly high percentage of the issues discussed here and can help a great deal in making the coding style shift stick until you can build new habits.) Rating: 9 out of 10

27 December 2012

Russ Allbery: Charity review

I'm thankfully in a place where I can afford to give money to charity, and I've slowly built up a set of charities that I give money to. But I've never been that systematic about it. I've had a vague target for how much I want to give, I add new charities when they seem neat, I'll rarely drop a charity when it doesn't seem to be effective, and I probably spread my money out too much. Recently, I followed a chain of blog references and discovered GiveWell, a site that's devoted to doing more in-depth analysis of charities than the basic financial analysis done by sites like Charity Navigator. In particular, they do literature surveys to try to dig into the effectiveness of particular interventions and take a closer look at how effectively a charity can use new donations. After reading quite a bit on their site and some related sites, I sat down yesterday and did a comprehensive review of my goals and principles and then put together a new plan. Some of that may be interesting to other people, so I'll write up that process here. This will be somewhat US-centric, since charities vary a lot by tax regime, but I think some of the general principles will still apply elsewhere. (But most of my specific examples will be US charities.) First, a few basic principles about charitable giving:
  1. Charity goals cannot be evaluated uniformly. Different types of organizations need different criteria. I've found that charities divide generally into ones that are trying to accomplish a specific task and ones that are trying to create political change. For the former, small and quiet organizations are often the best. For the latter, the charity has to get to a certain size before it's part of the conversation, and that involves a different set of tradeoffs.
  2. When you care about efficiency (number of lives saved or quality of life improvement per dollar spent), don't give money to organizations you've heard about via means other than word of mouth or third-party evaluations. This is for a simple reason: if the charity has successfully made you aware of it, that means they're advertising themselves to you. Advertising generally costs quite a bit of money. That's money they're not spending on whatever they're trying to accomplish. Excellent examples are many of the health research and education charities, such as the American Heart Association and the American Lung Association, or the massive disaster relief organizations like the American Red Cross. These are not efficient organizations. If you're trying to get the most impact for your dollar, this isn't a good place to spend it. As a special case of this, please don't ever give money to any organization that produces TV commercials showing starving children in Africa. If the commercial makes you want to give money, that's great; just don't give it to the organization on TV. TV commercials are extremely expensive, and those organizations are notorious for being some of the least efficient charities in existence (in some cases bordering on scams). You'll be lucky if 50% of your money actually goes to Africa, and the money that is spent probably isn't spent wisely.
  3. If your donation gets you a membership that comes with a glossy magazine, remember that means that the organization is spending your money producing that magazine. This is counted as program funding, not administrative costs, in sites like Charity Navigator because the charity probably counts the magazine as part of their educational mission. If you want to subscribe to the magazine and enjoy it, there's not necessarily anything wrong with this. But if you were giving them money to do something else, be aware that money is being used to produce the magazine (and the Christmas cards and the mailing labels and the tote bag) instead of doing things that you might consider program funding. For example, I stopped giving money to the Audubon Foundation because I want my environmental giving to go directly to protecting the environment, not towards printing calendars and making a glossy magazine full of bird pictures. As a counter-example, I consider the Southern Poverty Law Center to be one of the most effective US political charities, despite the fact that they produce a glossy magazine. That's because a large part of their mission is investigative reporting and research, and that magazine is how they publish that research. In this case, I both want to read the magazine and consider the magazine something that I'm happy to fund.
  4. Efficiency isn't everything. For some types of charities, the charity has to be huge in order to be effective. This is particularly true of political charities. For example, the ACLU is not a particularly efficient charity and has several fund-raising practices that I dislike. However, what the ACLU does requires that they have extensive media access and respect from legislators and the judicial system. It cannot be done effectively by a smaller and more efficient organization. Similar principles apply to a lot of political and environmental charities, and to some international aid charities such as Doctors without Borders (M decins Sans Fronti res).
If you're looking to increase the efficiency and effectiveness of your giving according to objective measures, I highly recommend reading through GiveWell's site. They recommend charities that avoid all the pitfalls above. You've probably never heard of them because they don't spend their money on advertising and marketing. However, keep in mind that efficiency isn't everything. With those principles in mind, and after doing a lot of reading on GiveWell, I redid my charity plan by dividing my giving up into five categories with different goals. Services. There are some things that I use that I think should be paid for by tax dollars but aren't, and are instead funded by charitable donations. I don't consider this charity in the same sense as the rest of this list. Rather, I'm paying for what I use plus some extra so that other people who can't afford it can use the service for free. Efficiency doesn't matter as much because there's usually only one organization supporting the thing that I personally use. Examples include my local PBS (public broadcasting) station, the non-profit that works with city government to support the local library system, and the trust that helps maintain the state parks where I go on vacation. Education. This is a special category for me since I regularly give money to the community college I attended. I know the region and the college and have some idea of what they need and what matters to the students there, so I can make specific choices. I give all the money in this category to the same place. Free software. Here, I know that my donations won't be used as efficiently in terms of quality of life as they could be if sent somewhere else, but this is my community and I want to support it because it's my community. Here, as with education, I have a lot more data and can pick and choose places that I think can use my money. Examples here are the Ada Initiative and the Free Software Foundation. Politics. This is the hardest area for efficiency, since it's so difficult to measure effectiveness in any organization where much of the goal of the organization is to persuade politicians or the general public. I try to measure effectiveness by how many concrete actions the organization takes (rather than press releases), which is easiest for organizations like the ACLU or the EFF that file lawsuits or legal documents and requests. As mentioned above, I think the standout in this area is the Southern Poverty Law Center, but political charities are so targeted that I give to multiple charities to cover my range of interests. I've found environmental charities particularly difficult to evaluate, but the one that appears the most effective to me so far is the Environmental Defense Fund, so that's where I concentrate my environmental money. I used to give to small political charities but have given up on that as basically useless, at least for national politics. I think the charity has to be of a minimum size to get anyone in government to listen to them. Poverty and Health. This is where efficiency matters much more than size. GiveWell's analysis is very interesting here, and sadly shows that most interventions either don't work or at least can't be proven to work. Here, I abandoned nearly all of my traditional large charities and have gone almost entirely with GiveWell's recommendations, with two exceptions. I still cycle money through Kiva (a microfinance support charity) because I have a bit of an emotional attachment to it, although microfinance is looking increasingly suspect as an effective charity method and I may yet drop them. And I still give money to Doctors without Borders because, despite being less efficient, I think their size and history has won them an international credibility that lets them get into areas that smaller, more efficient charities wouldn't be able to help. One of GiveWell's recommendations is my new favorite charity: GiveDirectly. All aid charities run a serious risk of having a colonialist bent, where rich countries come into poor countries and build things or tell them how to do things following the priorities of the rich countries. (This is one of the reasons why I prefer medical charities, since they're less susceptible to this.) GiveDirectly identifies the poorest people in a region (using a very transparent process) and transfers money to them directly using the M-Pesa cell system to spend however they choose, with no strings attached other than some due diligence to protect against fraud. This is refreshingly non-paternalistic and makes me far more comfortable than typical aid projects. Originally, I was going to assign money into those buckets and then break the buckets down further, but I ended up not quite doing that. The services bucket is more driven by the services I use than by an even allocation. Of the rest, it goes approximately three shares to poverty and health, two to politics, and one each to education and free software. I think I've managed to eliminate most of the places where I'm giving money to multiple organizations that do the same thing, but I'm still spreading my money rather widely and could stand to focus it more. That's hard, though; there are a lot of different things that I care about, and political charities in particular aren't very interchangeable. I'm currently not giving locally very much, which might be worth changing. The easiest way to do so would be to find a good local food bank and start donating to them (giving cash, not food). For a more comprehensive list of charities I support, see my charities link page.

Russ Allbery: Review: Science Fiction: The 101 Best Novels: 1985 2010

Review: Science Fiction: The 101 Best Novels: 1985 2010, by Damien Broderick & Paul Di Filippo
Publisher: Nonstop
Copyright: 2012
ISBN: 1-933065-39-7
Format: Trade paperback
Pages: 288
I like book reviews and lists of best novels, as a follower of my reviews probably noticed, so I couldn't resist when this collection made my radar. A follow-up to the earlier Science Fiction: The 101 Best Novels: 1949 1985 by David Pringle (which I have not read), it is a collection of short (two to three pages, generally) reviews of 101 recent SF novels. The date spread is fairly balanced: at least two novels from each year under consideration are featured, and no year gets more than seven or eight. The authors also clearly tried to cover the range of what falls under the science fiction genre, from alternate history through space opera and including novels normally marketed as mainstream, so the result should contain something to everyone's taste. With those characteristics, you may suspect that the "best" part of the title is a bit questionable, and you would be right. "101 of the better novels" would be a more accurate description. While most Hugo and Nebula winners are included here, the section is at times eclectic. But it's eclectic in the spirit of broad inclusiveness: Jumper, Temeraire, or The Hunger Games would normally not be included on this sort of list because they're too popular or "light," but they're here alongside more obscure books (at least for SF readers) like Galatea 2.2, Distance Haze, or My Dirty Little Book of Stolen Time. I doubt anyone will seriously argue that this selection should have replaced the Hugo or Nebula short lists, but one doesn't read review collections like this only to hear about books one already knows about. Those books one has either already read or already chosen not to read. The wide-ranging selection makes it likely that something here will be new to most readers. The authors, Damien Broderick and Paul Di Filippo, are known reviewers in the SF world, and I've read reviews from both of them before. I bought this book largely on the strength of Broderick's name, since I've usually enjoyed his contributions to The New York Review of Science Fiction. Paul Di Filippo was more of a gamble; he's one of the regular reviewers for Asimov's Science Fiction and not one of my favorites. But what I usually disliked about his columns was their focus on obscure small-press titles, graphic novels, and slipstream, so I was hoping that an SF review collection aiming towards the genre mainstream would be more to my taste. The result is mixed. There are things about this selection, and about the reviews, that I enjoyed, and there are other things I found quite annoying. First, the selection. I could (and will in a moment) talk about good and bad selections, but I also have a good statistical metric for at least the alignment of the authors' taste with mine. Of the 107 novels reviewed here (in several case, duologies and trilogies are given single entries), I've read 39, or a little over a third. (Note that I've read every Nebula or Hugo winner in that time period and most of the Hugo nominees, so that will give you a good feel for how broad-ranging this selection is, and how far afield of the normal award slates it goes.) My average rating for those 39 books was 7.49 (including four perfect 10s). By comparison, my average rating for Hugo winners is 6.68 and for Nebula winners is 7.10. There was one 4 (The White Queen) and one 5 (Red Mars), and in both cases I can see why they're here. Of the rest of the books I read I rated them all at least at 6 out of 10. At least among the books I've read, this seems to be a solid selection. Sometimes the details of those selections are odd, though. For example, the authors make an effort to limit the number of selections for each author, a wise choice since they're clearly going by diversity. But if one is operating within that limitation, choosing Ammonite over Slow River for Nicola Griffith, or particularly Ventus over Lady of Mazes or even Permanence for Karl Schroeder, is baffling. There were several similar places where I thought the selection for an author was obscure, minor, or just missed obvious alternatives. Perhaps this was to fill in one of the other breadth criteria, such as balancing number of novels per year or attempting to cover each subgenre. Also, if one is going to divide science fiction and fantasy and try to cover only the science fiction (a division that I think is quite difficult, which is why I don't do it, but it does have the merit of narrowing the field), including The Falling Woman is quite strange. It's a solid book, to be sure, and a Nebula winner. It is also quite straightforward contemporary fantasy involving ghosts and Mayan mythology, without a hint of science-fictional content. Making the protagonists archeologists and scientists doesn't make the book science fiction. The authors try to defend this (unpersuasively to me), but it wasn't the only instance here where I thought their line between science fiction and fantasy was a bit off. That said, there are a lot of great selections here, including books that I love but that aren't frequently picked for this sort of list (The Fortunate Fall, The Time Traveler's Wife, or China Mountain Zhang, for example). It's great to see underappreciated authors like Linda Nagata, Joan Slonczewski, and Karl Schroeder featured. But, of course, one doesn't buy this sort of book just for the list, if for no other reason than that lists aren't copyrightable and one can easily find the complete list of reviewed novels on the Internet (just one example that turned up in a search). Rather, one reads this sort of book for the reviews. And that's where this book moves onto more questionable ground. First, while I realize that everyone has different thresholds for what they consider spoilers and most professional reviewers are more cavalier about them than I am, Broderick and Di Filippo cross any line that I consider reasonable. Most of the reviews are okay, if skirting the limits, but in several places they give away key reveals of books or discuss plot twists right up to, or even including, the ending. The combined review of The Sparrow and Children of God is particularly egregious, containing unambiguous, book-destroying spoilers for The Sparrow. Giving away the ending of Ammonite is only slightly less bad. And those are just two examples I remember. This is not okay. The whole point of this sort of collection is to expose the reader to books they've not yet read but may want to. Proceeding to spoil the book for them in the course of the review is perverse. This alone would make me hesitant to recommend this collection. Second, quite a few of the reviews in this book are, for lack of a better term, emotionally overreaching. Here's an excerpt of a review picked at random (As She Climbed Across the Table by Jonathan Lethem) that will hopefully illustrate:
Lethem's beautifully balanced, metaphorically rich prose propels this blackly jolly fable to a surprising yet satisfying conclusion. By book's end, a sense that the author had accomplished his takeoff taxiing and was now fully in flight for more cosmopolitan cities pervades the pages.
What's "beautifully balanced prose"? Could you recognize it? Does that phrase communicate anything to you other than that the authors liked the book? The whole review collection is written with adjectives and metaphors like this, and after a while it all seems a bit much. It felt like the authors were straining for ways to describe how important or significant the books are and, in the process, lost sight of the basic goal of conveying information about the book. It feels overwrought rather than informative. Even if one is reviewing the books that one considers the pinnacle of achievement in science fiction, a conversational tone with concrete examples and specifics communicates more than impressive but slippery terms like "metaphorically rich." Lest I sound entirely negative, one thing that I did appreciate is that the reviews go to some effort (particularly for their short length) to put the work in the broader context of the field and within the author's oeuvre. Often there's some discussion of previous and subsequent work or related books, and the reviews that feature books from larger series provide good explanations for why those particular books were singled out. Sometimes the number of dangling references was frustrating; authors of these sorts of collections need to remember that most readers will not be as widely read, and reviewing books largely by comparison to other books runs the risk of missing the reader's knowledge entirely. But the reviews convey a real sense of SF as a broad conversation and provide a sense of the breadth and variety of themes and subgenres available. This is one of the fun explorations that this sort of catalog lets the authors and reader do together. Another, more minor, touch that I appreciated was the cover art. Each review leads with an image of the reviewed book's cover (alas, only in black and white for obvious printing reasons). But rather than taking the obvious approach of using the covers of the first releases, or the covers from a particular country, they're chosen from all of the world-wide editions in all their delightful variety. Typical artistic styles for book covers vary drastically between countries, and getting to see a sample of artwork from different markets is a treat. I want to recommend this book. It casts a much broader net than most collections of its kind and provides some needed attention to smaller corners of the genre. I was impressed by the book list before I bought it, and (with the inevitable quibbles) am even more impressed now that I've read it. Broderick and Di Filippo go out of their way to broaden the reader's horizon and open up new avenues for reading, which is one of the best things a review collection can do. But when reviewers don't avoid spoilers, I just can't recommend their work. For me, this is a cardinal sin. Combine that with a writing style that was occasionally overblown and overwritten and the merits don't quite overcome the flaws. I'm glad I read it; it got me excited about reading many books I've already purchased but not gotten to, and I got from it another slew of books to add to my to-purchase list. But I had to read it uncomfortably and lightly, constantly prepared to jump past a review that was too revealing. Rating: 6 out of 10

26 December 2012

Gunnar Wolf: Brazilian electronic booths hacked in a real election Surprised, anyone?

Doing my regular news scan, I stumbled across this: Hacker reveals in Rio how he rigged an election (in Portuguese; you can try the Google-translated version). Why am I reposting this? Because, even after the reported studies by Diego Aranha and the information disclosure exploited by Sergio Freitas, Brazil is still portrayed as the biggest example on how electronic voting can be 100% secure and tamper-proof. Well, in this case, Rangel (his full name ahs not yet been disclosed), a 19 year old hacker, not only demonstrated how elections could be rigged, but admitted on doing so together with a small group, and even pointed at who was benefitted from this. Rangel's attack was done during the transmission phase After ~50% of the electoral results had been sent over the Oi network. And yes, the provider will most likely close the hole that was pointed at, but this basically shows (again!) that no system can be 100% tamperproof, and that the more electronic devices are trusted for fundamental democratic processes, the more we as a society will be open to such attacks. The security-minded among us will not doubt even for a second that, as this attack was crafted, new attacks will continue to be developed. And while up to some years ago the attack surface was quite smaller (i.e. booths didn't have a communications phase, just stored the votes, and communication was done by personal means), earlier booths have been breached as well. And so will future booths be breached. So, the news of this attack are indeed very relevant for the field. The presentation I am quoting was held around two weeks ago And December will surely dillute attention from this topic. Anyway, I will look for further details on the mechanism that was used, as well as to the process that follows in the TSE (Supreme Electoral Tribunal). I hope we have news to talk about soon!

25 December 2012

Russ Allbery: Review: A Paradise Built in Hell

Review: A Paradise Built in Hell, by Rebecca Solnit
Publisher: Penguin
Copyright: 2009
Printing: 2010
ISBN: 0-14-311807-2
Format: Trade paperback
Pages: 319
A Paradise Built in Hell is a book with an agenda. Solnit's goal is to convince the reader that nearly everything we're shown in movies and popular culture about human behavior during disasters is wrong. Not only is destructive mass panic unusual verging on nonexistent, spontaneous cooperation and acts of startling courage and resourcefulness are commonplace. The heroes that we hear about afterwards are more the norm than the exception. Disasters are far more likely to bring out the best in people than the worst. They break down barriers and form spontaneous human communities that survivors remember for a lifetime. Rather than natural disasters leading inevitably to an exacerbating human crisis in the absence of strong authority, Solnit argues that they are an amazing human opportunity, and that the most negative and destructive human behavior during crises is not from untrained victims but from the panic of the elites who are supposedly responsible for protecting the public. I think her analysis and presentation is both deeply appealing and deeply flawed. The flaws involve extrapolation and generalization beyond the applicability of her research, and I'll say more about that in a moment. But the appeal is still surprising and satisfying, and I think there's a lot of truth in this book despite its one-sided presentation. I like to tell people, half-jokingly, that I'm an anarchosyndicalist. It's my political alliegance of the heart: the political structure that I know would never work, but that matches the world I want to live in. (I suspect much of the belief in libertarianism is of this form.) Now I have a book to point people at when they ask what the appeal of anarchosyndicalism is. The sense of spontaneous community, of bonding, of reaching out to other people to do something necessary, together, that Solnit describes here in the aftermath of a disaster matches that vision of a world without imposed economic authority, where people spontaneously collaborate to solve problems. A Paradise Built in Hell is structured by disaster, providing a tour of major disasters and their aftermath interspersed with extended musings on what the human reactions mean and how disaster communities functioned. The major disasters discussed are the 1906 San Francisco earthquake, the Halifax explosion, the 1985 Mexico City earthquake, 9/11, and Hurricane Katrina. Solnit moves through them in chronological order, closing the book with Katrina, which had (at least for me) the most infuriating and uplifting mixes of community bonding and elite panic. But both of those factors are present from the beginning in the 1906 earthquake: spontaneous human organization that (from multiple first-hand accounts) transformed people's lives mixed with a military reaction and elite panic that possibly did more damage to the city than the earthquake did. Elite panic is the dark side of Solnit's celebration of human altruism and spontaneous community organization. The primary picture she paints is of ordinary people responding with great courage and creativity and finding, in their reaction, a transformative experience and a sense of purpose that often becomes the most powerful and positive experience of their life. But this is at odds with a deeply-entrenched elite belief in mass panic, in the necessity of keeping information from the public to suppress their reaction, and in the need to re-establish "order." And with that, Solnit is absolutely scathing, and with some justification. Along with detailed accounts of these major disasters, Solnit also gives the reader a tour of the research literature on disasters, research that has shown that the panic and chaos that is at the center of practically every disaster movie ever made is essentially a myth. Mass panic in disasters is not only rare, it's close to non-existent. As you might expect, the handling of Katrina by both the government and by neighboring white communities is the subject of the harshest attacks. The material on Katrina presented here is enough to make anyone want to throw out the current playbook on how we react to disasters (and lest one think Solnit is unfair, her research matches very closely with other research and journalism on Katrina that I've seen from a wide variety of sources). But Solnit shows that this is a pattern: the combination of our media-driven belief in how people will react to disasters combined with the elite need to re-establish control plays out in many major disasters, and almost always negatively. Even in 9/11, the official response in some cases hampered and undermined an already-effective unofficial response. This is very interesting and has concrete implications for public policy around disasters. But where Solnit moves onto thinner anarchosyndicalist ice is the story that she tells about disasters as dramatic upheavals of the established social order that could point to a revolution in how we interact with each other. Solnit is deeply inspired by human altruism during disasters, and her enthusiasm is somewhat contageous, but it was useful to read this book shortly after Bruce Schneier's Liars and Outliers. I think Solnit, in her understandable eagerness to find a way to extend disaster behavior to general life, does not understand the relationships between what Schneier terms moral, reputational, and institutional pressures. And that's the core of why we need to both enable spontaneous human response in disasters and have organized first responders and supporting infrastructure, and why disaster communities will break down when extended beyond the disaster. When I first mentioned A Paradise Built in Hell to my mother, her first reaction was dubiousness due to her memory of the 1977 New York City blackout. There was a disaster that resulted in widespread looting and destruction, rather than the positive reaction Solnit describes. Solnit does have a cogent argument for why looting is poorly analyzed (we don't, for example, distinguish between breaking into stores to get desperately needed supplies and doing so for personal gain) and should not be a priority in a life-threatening disaster. But it's not completely convincing; in some disasters, particularly ones like the 1977 power outage that were not particularly life-threatening, the looting has been the most destructive part of the disaster. Schneier provides what I think is the missing piece, both to this and to why the wonderful behavior during disasters doesn't last into everyday life. During a life-threatening disaster, moral pressures are massively increased. Nearly all of us have self-images and moral beliefs that provide very strong incentives to help others and collaborate in the face of life-threatening emergencies, incentives that override other motives and lead to the sort of behavior that Solnit celebrates. But moral pressures have inherent limitations. That moral reaction applies most strongly to small groups (such as bands of survivors), and will start to break down as larger society re-establishes itself. And disasters, like a power outage, that are not immediately life-threatening will not provoke the same intensification of moral pressures: we don't have strong moral beliefs and internal stories about staying quietly at home during power outages the way that we do for saving people from burning buildings or pulling them out of rubble. Once larger communities have reformed and that short-term emergency reaction has dissipated, the need for institutional pressures (police, law, and formal authority structures) will return. I wish Solnit had been able to read Schneier's book before writing hers. I think his analysis structure provides a healthy dose of realism and perspective to her desire for a self-organizing communal world, and might have blunted her enthusiastic but somewhat unrealistic hopes for finding some key to a revolution in human organization in the heart of disasters. But if A Paradise Built in Hell is unlikely to lead to a radical change in how we organize societies, it should at least provoke a radical rethinking of some of our unnecessary pessimism about our fellow humans. The evidence is substantial and compelling: crisis brings out the best in people, not the worst, and the ordinary people who happen to be around are capable of becoming some of the most amazing disaster response teams that we can imagine. This has numerous mundane implications for social policy. We should consider simple training for the general public, for example, to strengthen that reaction, should allow for it in legal structures and training of first responders (around looting, for instance), and should do everything we can to kill the destructive myth that people cannot be trusted in disasters or that mass panic will be as much of a threat as the disaster. But the best part of this book is that it provides a concrete, well-documented, and well-defended reason to let go of some of our cynicism and to extend a bit of trust to our fellow humans. There sadly aren't earth-shaking implications for everyday society, but there is a wealth of evidence that shows that we're better people in the crunch than we believe we are, or that we expect of others. And the more that we can embrace the dissolution of boundaries and social hierarchies in the middle of disasters, the more we are likely to be surprised by deep human connection and a sense of shared, focused purpose. Solnit is a bit long-winded with her agenda, but that bright hope has stuck with me. Recommended. Rating: 8 out of 10

23 December 2012

Benjamin Mako Hill: The Cost of Collaboration for Code and Art

This post was written with Andr s Monroy-Hern ndez for the Follow the Crowd Research Blog. The post is a summary of a paper forthcoming in Computer-Supported Cooperative Work 2013. You read also read the full paper: The Cost of Collaboration for Code and Art: Evidence from Remixing. It is part of a series of papers I have written with Monroy-Hern ndez using data from Scratch. You can find the others on my academic website.
Does collaboration result in higher quality creative works than individuals working alone? Is working in groups better for functional works like code than for creative works like art? Although these questions lie at the heart of conversations about collaborative production on the Internet and peer production, it can be hard to find research settings where you can compare across both individual and group work and across both code and art. We set out to tackle these questions in the context of a very large remixing community.

Example of a remix in the Scratch online community, and the project it is based off. The orange arrows indicate pieces which were present in the original and reused in the remix.

Remixing platforms provide an ideal setting to answer these questions. Most support the sharing, and collaborative rating, of both individually and collaboratively authored creative works. They also frequently combine code with artistic media like sound and graphics. We know that that increased collaboration often leads to higher quality products. For example, studies of Wikipedia have suggested that vandalism is detected and removed within minutes, and that high quality articles in Wikipedia, by several measures, tend to be produced by more collaboration. That said, we also know that collaborative work is not always better for example, that brainstorming results in less good ideas when done in groups. We attempt to answer this broad question, asked many times before, in the context of remixing: Which is the better description, the wisdom of crowds or too many cooks spoil the broth ? That, fundamentally, forms our paper s first research question: Are remixes, on average, higher quality than single-authored works? A number of critics of peer production, and some fans, have suggested that mass collaboration on the Internet might work much better for certain kinds of works. The argument is that free software and Wikipedia can be built by a crowd because they are functional. But more creative works like music, a novel, or a drawing might benefit less, or even be hurt by, participation by a crowd. Our second research question tries to get at this possibility: Are code-intensive remixes, higher quality than media-intensive remixes? We try to answers to these questions using a detailed dataset from Scratch a large online remixing community where young people build, share, and collaborate on interactive animations and video games. The community was built to support users of the Scratch programming environment: a desktop application with functionality similar to Flash created by the Lifelong Kindergarten Group at the MIT Media Lab. Scratch is designed to allow users to build projects by integrating images, music, sound and other media with programming code. Scratch is used by more than a million, mostly young, users. Measuring quality is tricky and we acknowledge that there are many ways to do it. In the paper, we rely most heavily a measure of peer ratings in Scratch called loveits very similar to likes on Facebook. We find similar results with several other metrics and we control for the number of views a project receives. In answering our first research question, we find that remixes are, on average, rated as being of lower quality than works of single authorship. This finding was surprising to us but holds up across a number of alternative tests and robustness checks. In answering our second question, we find rough support for the common wisdom that remixing tends to be more effective for functional works than for artistic media. The more code-intensive a project is, on average, the closer the gap is between a remix and a work of single authorship. But the more media-intensive a project is, the bigger the gap. You can see the relationships that our model predicts in the graph below.

Two plots of estimated values for prototypical projects showing the predicted number of loveits using our estimates. In the left panel, the x-axis varies number of blocks while holding media intensity at the sample median. The right panel varies the number of media elements while holding the number of blocks at the sample median. Ranges for each are from 0 to the 90th percentile.

Both of us are supporters and advocates of remixing. As a result, we were initially a little troubled by our result in this paper. We think the finding suggests an important limit to the broadest claims of the benefit of collaboration in remixing and peer production. That said, we also reject the blind repetition of the mantra that collaboration is always better for every definition of better, and for every type of work. We think it s crucial to learn and understand the limitations and challenges associated with remixing and we re optimistic that this work can influence the design of social media and collaboration systems to help remixing and peer production thrive. For more, see our full paper, The Cost of Collaboration for Code and Art: Evidence from Remixing.

Cyril Brulebois: d-i hacking recipe #3: Debugging debconf

Problem #3: How to find out what s going on with debconf? Slightly longer version: As reported in #679327, the grub prompt during the installation process is entitled Configuring man-db , which isn t exactly great. Given man-db is a known dpkg trigger, I suspected this would be the reason for the broken title, and here s how I debugged it. For starters, using d-i basically means asking a lot of questions, which is implemented using the debconf mechanism. One should note that two implementations are available:
  1. the historic implementation, in Perl: debconf
  2. the C reimplementation: cdebconf
d-i uses cdebconf udebs, but joy begins when packages get installed in /target: debconf is used there. We ll see why that matters. Recipe #3: Set DEBCONF_DEBUG=developer and read logs!
  1. Both implementations support the DEBCONF_DEBUG environment variable. The installer supports passing that as a kernel parameter; example from the syslinux prompt: select the Graphical install entry, press Tab , remove -- quiet and put DEBCONF_DEBUG=developer there instead.
  2. Let s look at /var/log/syslog; there s a tail -f on it in VT4, but one can run grep and friends on it in VT2 or VT3. There, one can see what happens at the debconf level (the debconf protocol is documented in debconf-devel(7)). Tiny excerpt (timestamps removed for clarity): 
    frontend: --> GET preseed/early_command
frontend: <-- 0
 
frontend: --> GET debian-installer/framebuffer
frontend: <-- 0 true
 
debconf: --> GET debconf/language
debconf: <-- 0 en
debconf: --> SET debconf/language en
debconf: Setting debconf/language to en
debconf: <-- 0 value set
debconf: --> GET debconf/priority
debconf: <-- 0 high
debconf: --> GET rescue/enable
debconf: <-- 0 false
  3. The interesting point was titles, right? So let s look at the last TITLE occurrences: 
    debconf: --> SETTITLE debian-installer/main-menu-title
debconf: --> SETTITLE debian-installer/grub-installer/title
in-target: debconf (developer): ----> TITLE Configuration de man-db
debconf: --> TITLE Configuration de man-db
in-target: debconf (developer): <-- TITLE Configuration de man-db
in-target: debconf (developer): ----> TITLE Configuration de man-db
in-target: debconf (developer): ----> TITLE Configuration de grub-pc
debconf: --> TITLE Configuration de grub-pc
in-target: debconf (developer): <-- TITLE Configuration de grub-pc
in-target: debconf (developer): ----> TITLE Configuration de grub-pc
in-target: debconf (developer): ----> TITLE Configuration de man-db
debconf: --> TITLE Configuration de man-db
in-target: debconf (developer): <-- TITLE Configuration de man-db
in-target: debconf (developer): ----> TITLE Configuration de man-db
in-target: debconf (developer): ----> TITLE Configuration de grub-pc
debconf: --> TITLE Configuration de grub-pc
in-target: debconf (developer): <-- TITLE Configuration de grub-pc
in-target: debconf (developer): ----> TITLE Configuration de grub-pc
debconf: --> SETTITLE debian-installer/grub-installer/title
    Bad news: the last one is about grub, so the title should be correct, right?
  4. At this point, it might be the Gtk frontend behaving strangely. It s shipped in cdebconf-gtk-udeb, built from the cdebconf source. Let s look:
    • src/modules/frontend/gtk/ui.c has cdebconf_gtk_update_frontend_title(), which does what the function name suggests.
    • src/modules/frontend/gtk/di.c has cdebconf_gtk_di_run_dialog(), which calls the first function.
    • src/modules/frontend/gtk/progress.c has cdebconf_gtk_show_progress() (also calls the first function), which comes with the following comments: 
      /** Show the progress widgets.
  *
  * This will actually add the widgets to the corresponding containers.
  * The main title saved when starting the PROGRESS operation will be restored
  * from the value saved when START was called.  This is needed when GO is
  * called during a PROGRESS operation.
    Now that s something! Even if the last entry is about grub, some previous value could be restored and could be the reason for the bad behaviour (that is easily confirmed by adding some debug_printf() calls on fe->title in cdebconf, thanks to an extra "debug.h" include). Maybe that s what needs fixing!
  5. Now it s time to take a step back. A d-i environment is nice, but testing things can easily become a burden. Taking a random wheezy or sid chroot, and installing/removing a tiny package shipping a manpage should be enough to run into the man-db trigger. Exporting DEBCONF_DEBUG=developer there too, let s either install or remove x11-apps (the trigger does the same job, and the debconf trace is the same), here s the output: 
    Processing triggers for man-db ...
debconf (developer): frontend started
debconf (developer): frontend running, package name is man-db
debconf (developer): starting /var/lib/dpkg/info/man-db.config configure /usr/share/man
debconf (developer): <-- VERSION 2.0
debconf (developer): --> 0 2.0
debconf (developer): <-- INPUT medium man-db/install-setuid
debconf (developer): --> 30 question skipped
debconf (developer): <-- GO
debconf (developer): --> 0 ok
debconf (developer): starting /var/lib/dpkg/info/man-db.postinst triggered /usr/share/man
debconf (developer): <-- VERSION 2.0
debconf (developer): --> 0 2.0
debconf (developer): <-- GET man-db/auto-update
debconf (developer): --> 0 true
    Even if one didn t know about the cdebconf/debconf thing, the frontend started bit in the d-i syslog would have been the key: it s only found in debconf, in the frontend script. That one has several ways to determine the package being acted on, and it finally calls: 
    debug developer => "frontend running, package name is $package";
$frontend->default_title($package) if length $package;
    Tada, that s likely to be the issue.
  6. Tweaking that frontend script, it s easy to make sure that the code path for this use case is: 
    elsif ($ARGV[0]=~m!^.*/(.*?)\.(?:postinst postrm prerm)$!)  
        $package=$1;
    Since $ARGV[1] is the action being passed to the maintainer script, one can intercept the triggered action, and avoid emitting a title update in this case; I ve therefore reassigned #679327 to the debconf package, and proposed a patch implementing that, which Joey Hess seems to find reasonable.
In my message to the bug report I wrote I actually tested it in a d-i environment, and made sure the issue had gone away. How I did that will likely be described in the next d-i hacking recipe.

21 December 2012

Richard Hartmann: Release Critical Bug report for Week 51

The gaffe Please note that the historic data had a bug which is fixed and the correct value highlighted below. On the plus side, this means that the delta is not as large as I feared. On the negative side, the gap between the Squeeze and Wheezy release cycles keeps widening slowly but steadily. The gist The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the squeeze cycle ?
Week Squeeze Wheezy Diff
43 284 (213+71) 468 (332+136) +184 (+119/+65)
44 261 (201+60) 408 (265+143) +147 (+64/+83)
45 261 (205+56) 425 (291+134) +164 (+86/+78)
46 271 (200+71) 401 (258+143) +130 (+58/+72)
47 283 (209+74) 366 (221+145) +83 (+12/+71)
48 256 (177+79) 378 (230+148) +122 (+53/+69)
49 256 (180+76) 360 (216+155) +104 (+36/+79)
50 204 (148+56) 339 (195+144) +135 (+47/+90)
51 178 (124+54) 323 (190+133) +145 (+66/+79)
52 115 (78+37)
1 93 (60+33)
2 82 (46+36)
3 25 (15+10)
4 14 (8+6)
5 2 (0+2)
6 release!
Graphical overview of bug stats thanks to azhag:

19 December 2012

Paul Tagliamonte: The most horrifying thing i've done -- or: hy

Hang tight folks, this is a trippy one: As many have noted in my most recent post, I ve taken a shine to lisps hosted in Python. While clojure-py is badass, it s not fit for using from python. I hit some problems in the way it handles atomic variable access (well atomic , it s actually not ;) ), so I ve decided to give it a shot myself. That s right. I wrote a Lisp. A very basic, unmacroed lisp, but a lisp. So, let s take a look a trivial hy file pair The first file (really_lisp.hy), is as follows: 
; vim: tabstop=2 expandtab shiftwidth=2 softtabstop=2 filetype=lisp
; Copyright (c) Paul Tagliamonte, in sofar as any of this is at all
;  copyrightable.
(def hello "World!")
(def square (fn [arg]
  (* arg arg)))
(def factorial (fn [n]
  (if (<= n 1)
   1
   (* n (factorial (- n 1))))))
(def fib (fn [n]
  (if (< n 2)
    n
    (+ (fib (- n 1)) (fib (- n 2))))))
And, the example Python script example.py is as follows: 
import hy.lang.importer
import really_lisp
print really_lisp.hello
# "World!"
print really_lisp.square(4)
# 16
print really_lisp.factorial(10)
# 3628800
print [really_lisp.fib(x) for x in range(1, 20)]
# [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144,
#        233, 377, 610, 987, 1597, 2584, 4181]
Please note two awesome things firstly, we re importing a non-python script into a Python module. Awesome. Secondly, we re able to access symbols natively, as well as functions. Let s also demonstrate the other way (just hython this time): 
; vim: tabstop=2 expandtab shiftwidth=2 softtabstop=2 filetype=lisp
; Copyright (c) Paul Tagliamonte, in sofar as any of this is at all
;  copyrightable.
(puts "Hello, World!")
; "Hello, World!"
(import ["os"
         "sys"])
(puts (os.path.abspath "."))
; "/home/tag/tmp"
(puts sys.argv)
; ['/home/tag/.virtualenvs/hy/bin/hython', 'test.hy']
(sys.stdout.write "Fuggn' test\n")
; "Fuggn' test"
Naturally, not to be outdone, I decided to amp it up. Django in Lisp While insane, the big stuff looks like this: I ve replaced urls.py with urls.hy: 
; vim: tabstop=2 expandtab shiftwidth=2 softtabstop=2 filetype=lisp
(import ["django.conf.urls"])
(def urlpatterns (django.conf.urls.patterns ""
  (django.conf.urls.url "^$" "djlisp.hellisp.views.home")
))
And, views.py with views.hy: 
; vim: tabstop=2 expandtab shiftwidth=2 softtabstop=2 filetype=lisp
(import ["django.shortcuts"])
(def home (fn [request]
  (django.shortcuts.render_to_response "home.html"  )))
And lastly, the settings.hy file. I m not going to paste it because it s huge. But, it works!

18 December 2012

Petter Reinholdtsen: Ledger - double-entry accounting using text based storage format

A few days ago I came across a blog post from Joey Hess describing ledger and hledger, a text based system for double-entry accounting. I found it interesting, as I am involved with several organizations where accounting is an issue, and I have not really become too friendly with the different web based systems we use. I find it hard to find what I look for in the menus and even harder try to get sensible data out of the systems. Ledger seem different. The accounting data is kept in text files that can be stored in a version control system, and there are at least five different implementations able to read the format. An example entry look like this, and is simple enough that it will be trivial to generate entries based on CVS files fetched from the bank: 
2004-05-27 Book Store
      Expenses:Books                 $20.00
      Liabilities:Visa
The concept seemed interesting enough for me to check it out and look for others using it. I found blog posts from Christine Spang, Pete Keen, Andrew Cantino and Ronald Ip describing how they use it, as well as a post from Bradley M. Kuhn at the Software Freedom Conservancy. All seemed like good recommendations fitting my need. The ledger package is available in Debian Squeeze, while the hledger package only is available in Debian Sid. As I use Squeeze, ledger seemed the best choice to get started. To get some real data to test on, I wrote a web scraper for LODO, the accounting system used by the NUUG association, and started to play with the data set. I'm not really deeply into accounting, but I am able to get a simple balance and accounting status for example using the "ledger balance" command. But I will have to gather more experience before I know if the ledger way is a good fit for the organisations I am involved in.

Russ Allbery: Review: Liars and Outliers

Review: Liars and Outliers, by Bruce Schneier
Publisher: John Wiley
Copyright: 2012
ISBN: 1-118-14330-2
Format: Hardcover
Pages: 285
One of the perils of buying a book written by a blogger one reads regularly is that the book may be little more than a rehashing of their blog, with insufficient original material to warrant the time investment. Sometimes it's still nice to support them financially, but it may not make sense to read the book. I've been following Schneier's blog for years (as should anyone with an interest in security), including through the entire process of writing Liars and Outliers, and was a bit worried that might be the case here. Thankfully, I can reassure any other worried potential readers that is not the case. This is substantial new material establishing a firm framework for thinking about incentives and controls in any society or organization. Liars and Outliers talks about security mechanisms, but it's not, at its core, a book about security. Rather, it's a book about incentives, order, and how order is established. It's a comprehensive reductionist analysis of how societies create predictability and compliance to allow us to trust other people who we have never met before and will never meet again. It's a unique (at least in my experience) combination of anthropology, sociology, security analysis, and political science. Schneier cuts across fields in an idiosyncratic but illuminating way that reminded me of (on an entirely separate topic) Jane Jacobs. This is not a prescriptive book, nor is a collection of answers, solutions, or even deep analysis of particular problems. Rather, it's an attempt to construct a general framework for analyzing societal dilemmas: conflicts between individual desires and social good, how those conflicts are resolved, and how societies can weigh the scales and influence the statistical outcomes. The closest Schneier comes to telling the reader how to solve problems is a checklist, at the end of the book, for designing effective societal pressures. Its primary contribution is vocabulary and structure. It also passes one of my litmus tests for any book about human behavior: Schneier complicates, broadens, deepens, and expands understanding, and points out complex interactions and complex feedback in effects we're inclined to consider simple, rather than simplifying or eliminating human complexity. One point I found refreshing about this book is that Schneier is scrupulous in refusing to define either society or individuals as good or bad, to the point of carefully defining terminology used in all of the social dilemmas. Following societal rules is called compliance; not following those rules is called defection. In some cases, defection is morally correct (Schneier's most frequent example is in the civil rights struggles of the 1960s in the United States). In all cases, social pressures are tools, which can be used to encourage compliance with moral or immoral systems, and which are deployed by totalitarian dictatorships and utopian communes alike. Schneier explicitly puts out of scope for this book the questions of how societal goals should be determined, how they change, and whether any given societal rule or interest is moral or immoral. He focuses, rather, only on the mechanisms, with a primary goal of informing and deepening debates over how best to encourage behavior that societies want to encourage and discourage behavior societies want to discourage. He also emphasizes discourage, as opposed to eliminate. Early on, Schneier shows some of the results of game theory, as well as basic common sense, that indicate that no healthy society can totally eliminate defection. Not only would that stifle valuable and important reform, such as changes in civil rights, but the degree of pressure required is immense. Defectors are natural and will always exist, and are sometimes valuable and necessary. Rather, the goal of a society is to reduce defectors to a level where most people can ignore their existence most of the time, a state that leads to the level of risk and trust required to have a functioning and healthy society. The word society, similarly, is intentionally broad, and can refer to just about any collection of people, from a circle of friends to a corporation, institution, or country. However, as Schneier points out early in the book, small societies rarely need much in the way of formal pressure and appear almost magically self-governing. That's the first property that he disassembles, resulting in a general classification of societal pressures into four categories: moral, reputational, institutional, and security systems. The last is an odd category that's partly orthagonal to the other three. Moral pressure is internalized conscience: the normal tendency of nearly everyone to follow their own moral code, a code that's at least partly constructed and certainly heavily influenced by the surrounding society. Reputational pressure is, in a sense, externalized morality: it's the informal reactions of others around one to one's past actions. Included in reputational pressure is shunning of every kind, from cutting off a friendship to boycotts against corporations, but it also includes confrontation from another member of one's society, the more subtle effects of our individual desires to be liked and respected, and all the various aspects of "face", honor, and respect within a community. Small communities are frequently self-governing, in Schneier's model, because they're small enough that moral and reputational pressures are sufficient and no other pressures are required. We're so used to applying moral and reputational pressure to other humans almost unconsciously that we sometimes don't even notice its existence, leading to that "magical" self-governing property. But Schneier puts pressures in a sequence: pressures that work extremely well with small groups often don't scale. Moral pressure works best with small groups and reputational pressure with somewhat larger groups, but when societies scale beyond the limits of reputational pressure when, for example, one frequently interacts with people whose reputations are unknown to you and whose subsequent opinion of you will not be relevant institutional pressure is required to force compliance. Institutional pressure is the sort of pressure that we all tend to think of first when we look for ways to enforce rules: laws, policies, contracts, and other codes of behavior that carry with them formal punishments and some enforcement mechanism. But even in societies so large that institutional pressures are frequently required, such as whole countries, moral and reputational pressures still exist and are extremely important. One of Schneier's most interesting points is his analysis of how institutional pressures can paradoxically undermine reputational and moral pressures, resulting in more defection than if the institutional pressure hadn't existed. This is just the basic framework of Schneier's analysis, hopefully giving you a feel for the structure of the book. He goes much deeper into the complicated interactions between the various levels of pressure, and then dives into an extensive look at competing societal dilemmas: cases where there is more than one society in play simultaneously, possibly demanding contradictory actions. Liars and Outliers also includes a wonderful analysis of organizational entities within the same framework, including their much-different reactions to moral, reputational, and institutional pressures. One of the most cogent analyses of the difficulties of regulating both corporations and governmental institutions falls out of that analysis, once one looks at them in light of Schneier's basic framework. Pressures quickly become complex and multi-layered, and human reactions to pressures are frequently counter-intuitive. Schneier draws extensively on game theory to show that some counter-intuitive responses are actually emergent properties of logical analysis of the situation, but that others are more uniquely human and have little or nothing to do with a mathematical cost-benefit analysis. I haven't even mentioned his discussion of security systems, and how they can extend moral, reputational, and institutional pressures, as well as add a new type of pressure (making a defecting action impossible) that scales even better than institutional pressures. Liars and Outliers has all of the supporting infrastructure you would expect in a scholarly book: notes, extensive references, and a good index. I suspect it will end up being used as at least additional reading in college classes. The notes are, unfortunately, end notes, making the full context of the book much harder to read than was necessary, but at least Schneier does separate the notes from the references so that one doesn't chase notes for further explanation and find a simple citation. As with any book like this, one always wishes it could end in a simple prescription to fix everything, but of course it doesn't. But that's also a measure of a good scholarly work. Human and organizational motivations are complex and tricky, and any framework for analyzing them needs to be able to represent that complexity. Schneier here has constructed a very powerful one, one that I started using in discussions before I'd even finished the book. Perhaps the most valuable contribution of Liars and Outliers to public discussion is clear terminology and categories, which can be of great help in finding the core components of a problem. Liars and Outliers can be slow going, particularly early on when Schneier is still defining terms and setting up the background of his analysis. One can get a bit tired of the analysis matrices of societal dilemmas. But stick with it through the groundwork, since the analyses of competing societal dilemmas and of the impact of societal pressures on organizations are exceptional. Highly recommended, particularly for anyone who is designing or implementing societal pressures: managers, political activists, or anyone in a security-related field. Rating: 8 out of 10

17 December 2012

Russell Coker: Using BTRFS

I ve just installed BTRFS on some systems that matter to me. It is still regarded as experimental but Oracle supports it with their kernel so it can t be too bad and it s almost guaranteed that anything other than BTRFS or ZFS will lose data if you run as many systems as I do and I run lots of systems that don t have enough RAM for ZFS (4G isn t enough in my tests). So I have to use BTRFS. BTRFS and Virtual Machines I m running BTRFS for the DomUs on a virtual server which has 4G of RAM (and thus can t run ZFS). The way I have done this is to use ext4 on Linux software RAID-1 for the root filesystem on the Dom0 and use BTRFS for the rest. For BTRFS and virtual machines there seem to be two good options given that I want BTRFS to use it s own RAID-1 so that it can correct errors from a corrupted disk. One is to use a single BTRFS filesystem with RAID-1 for all the storage and then have each VM use a file on that big BTRFS filesystem for all it s storage. The other option is to have each virtual machine run BTRFS RAID-1. I ve created two LVM Volume Groups (VGs) named diska and diskb, each DomU has a Logical Volume (LV) from each VG and runs BTRFS. So if a disk becomes corrupt the DomU will have to figure out what the problem is and fix it. #!/bin/bash
for n in $(xm list cut -f1 -d\ egrep -v ^Name\ ^Domain-0) ; do
echo $n
ssh $n "btrfs scrub start -B -d /"
done I use the above script in a cron job from the Dom0 to scrub the BTRFS filesystems in the DomUs. I use the -B option so that I will receive email about any errors and so that there won t be multiple DomUs scrubbing at the same time (which would be really bad for performance). BTRFS and Workstations The first workstation installs of BTRFS that I did were similar to installations of Ext3/4 in that I had multiple filesystems on LVM block devices. This caused all the usual problems of filesystem sizes and also significantly hurt performance (sync seems to perform very badly on a BTRFS filesystem and it gets really bad with lots of BTRFS filesystems). BTRFS allows using subvolumes for snapshots and it s designed to handle large filesystems so there s no reason to have more than one filesystem IMHO. It seems to me that the only benefit in using multiple BTRFS filesystems on a system is if you want to use different RAID options. I presume that eventually the BTRFS developers will support different RAID options on a per-subvolume basis (they seem to want to copy all ZFS features). I would like to be able to configure /home to use 3 copies of all data and metadata on a workstation that only has a single disk. Currently I have some workstations using BTRFS with a BTRFS RAID-1 configuration for /home and a regular non-RAID configuration for everything else. But now it seems that this is a bad idea, I would be better off just using a single copy of all data on workstations (as I did for everything on workstations for the previous 15 years of running Linux desktop systems) and make backups that are frequent enough to not have a great risk. BTRFS and Servers One server that I run is primarily used as an NFS server and as a workstation. I have a pair of 3TB SATA disks in a BTRFS RAID-1 configuration mounted as /big and with subvolumes under /big for the various NFS exports. The system also has a 120G Intel SSD for /boot (Ext4) and the root filesystem which is BTRFS and also includes /home. The SSD gives really good read performance which is largely independent of what is done with the disks so booting and workstation use are very fast even when cron jobs are hitting the file server hard. The system has used a RAID-1 array of 1TB SATA disks for all it s storage ever since 1TB disks were big. So moving to a single storage device for /home is a decrease in theoretical reliability (in addition to the fact that a SSD might be less reliable than a traditional disk). The next thing that I am going to do is to install cron jobs that backup the root filesystem to something under /big. The server in question isn t used for anything that requires high uptime, so if the SSD dies entirely and I need to replace it with another boot device then it will be really annoying but it won t be a great problem. Snapshot Backups One of the most important uses of backups is to recover from basic user mistakes such as deleting the wrong file. To deal with this I wrote some scripts to create backups from a cron job. I put the snapshots of a subvolume under a subvolume named backup . A common use is to have everything on the root filesystem, /home as a subvolume, /home/backup as another subvolume, and then subvolumes for backups such as /home/backup/2012-12-17, /home/backup/2012-12-17:00:15, and /home/backup/2012-12-17:00:30. I make /home/backup world readable so every user can access their own backups without involving me, of course this means that if they make a mistake related to security then I would have to help them correct it but I don t expect my users to deal with security issues, if they accidentally grant inappropriate access to their files then I will be the one to notice and correct it. Here is a script I name btrfs-make-snapshot which has an optional first parameter -d to cause it do just display the btrfs commands it would run and not actually do anything. The second parameter is either minutes or days depending on whether you want to create a snapshot on a short interval (I use 15 minutes) or a daily snapshot. All other parameters are paths for subvolumes that are to be backed up: #!/bin/bash
set -e

# usage:
# btrfs-make-snapshot [-d] minutes days paths
# example:
# btrfs-make-snapshot minutes /home /mail

if [ "$1" == "-d" ]; then
BTRFS="echo btrfs"
shift
else
BTRFS=/sbin/btrfs
fi

if [ "$1" == "minutes" ]; then
DATE=$(date +%Y-%m-%d:%H:%M)
else
DATE=$(date +%Y-%m-%d)
fi
shift

for n in $* ; do
$BTRFS subvol snapshot -r $n $n/backup/$DATE
done Here is a script I name btrfs-remove-snapshots which removes old snapshots to free space. It has an optional first parameter -d to cause it do just display the btrfs commands it would run and not actually do anything. The next parameters are the number of minute based and day based snapshots to keep (I am currently experimenting with 100 100 for /home to keep 15 minute snapshots for 25 hours and daily snapshots for 100 days). After that is a list of filesystems to remove snapshots from. The removal will be from under the backup subvolume of the path in question. #!/bin/bash
set -e

# usage:
# btrfs-remove-snapshots [-d] MINSNAPS DAYSNAPS paths
# example:
# btrfs-remove-snapshots 100 100 /home /mail

if [ "$1" == "-d" ]; then
BTRFS="echo btrfs"
shift
else
BTRFS=/sbin/btrfs
fi

MINSNAPS=$1
shift
DAYSNAPS=$1
shift

for DIR in $* ; do
BASE=$(echo $DIR cut -c 2-200)
for n in $(btrfs subvol list $DIR grep $BASE/backup/.*: head -n -$MINSNAPS sed -e "s/^.* //"); do
$BTRFS subvol delete /$n
done
for n in $(btrfs subvol list $DIR grep $BASE/backup/ grep -v : head -n -$MINSNAPS sed -e "s/^.* //"); do
$BTRFS subvol delete /$n
done
done A Warning The Debian/Wheezy kernel (based on the upstream kernel 3.2.32) doesn t seem to cope well when you run out of space by making snapshots. I have a filesystem that I am still trying to recover after doing that. I ve just been buying larger storage devices for systems while migrating them to BTRFS, so I should be able to avoid running out of disk space again until I can upgrade to a kernel that fixes such bugs.
Related posts:
  1. BTRFS and ZFS as Layering Violations LWN has an interesting article comparing recent developments in the...
  2. Starting with BTRFS Based on my investigation of RAID reliability [1] I have...
  3. ZFS vs BTRFS on Cheap Dell Servers I previously wrote about my first experiences with BTRFS [1]....

16 December 2012

Joachim Breitner: Circle Packing

For an upcoming introductory Haskell talk of mine (next Tuesday in Karlsruhe please come and join if you will) I was looking into data visualization as a example application. With libraries like gloss, getting some nice vector graphics up and running within one talk is quite possible. For some reason, I want to arrange multiple circles of varying size so that they do not overlap and use as little space as possible. My first approach was to use general purpose optimizers such as cmaes, Numeric.GSL.Minimization from the hmatrix package and force-layout. I especially liked the interface of cmaes, which is able to minimize a function of, say, type [(String, Double, Double)] -> Double by automatically finding the values of type Double in the input, as described by Takayuki Muranushi. That would have looked great in the talk. Unfortunately, none of these libraries gave sufficient good results in a reasonable amount of time for this problem. I then reviewed some of the academic literature on the circle packing problem and there were some algorithms proposed, but no simple one and none of the papers had a concise pseudo-code description that I could transfer to Haskell. So eventually I set out to implement a relatively dump and slow greedy algorithm myself: I place one circle after another, starting with the largest, and among the valid positions where a new circle touches two already placed circles, I choose the positions closest to the center of mass. The result, available in the circle-packing package, looks surprisingly well, here visualized using the diagrams library (See the documentation of Optimisation.CirclePacking for the code used to generate that image):
Now showing just this image is not a very good way to demonstrate my code. A few years ago, I might have created a CGI script that would dynamically generate images based on your input. But not in 2012: Since my code is pure Haskell without fancy features like type classes, I can use the fay compiler to generate JavaScript code from it. Add a little Haskell code to interact with the HTML5 canvas and now you can interactively try out circle-packing in your browser. Oh, and while I am talking about neat tricks: You can put vector graphics in your haddock documentation, as I have done for Optimisation.CirclePacking, using the syntax <<<data:image/svg+xml;base64,PD94bWwgdmV...c3ZnPg==>>.

14 December 2012

Richard Hartmann: Release Critical Bug report for Week 50

The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the squeeze cycle ?
Week Squeeze Wheezy Diff
43 284 (213+71) 468 (332+136) +184 (+119/+65)
44 261 (201+60) 408 (265+143) +147 (+64/+83)
45 261 (205+56) 425 (291+134) +164 (+86/+78)
46 271 (200+71) 401 (258+143) +130 (+58/+72)
47 283 (209+74) 366 (221+145) +83 (+12/+71)
48 256 (177+79) 378 (230+148) +122 (+53/+69)
49 256 (180+76) 360 (216+155) +104 (+36/+79)
50 204 (148+56) 339 (195+144) +135 (+47/+90)
51 124 (124+54)
52 115 (78+37)
1 93 (60+33)
2 82 (46+36)
3 25 (15+10)
4 14 (8+6)
5 2 (0+2)
6 release!
Graphical overview of bug stats thanks to azhag:

Paul Tagliamonte: Scripting Python in Clojure

Yeah, wait, what? No, really! I took a look through clojure-py, and found out something pretty rad. In order to get clojurepy scripts to load other clojurepy, it had to shim out and add a new importer for clojure scripts. What this really means, is that you can import Clojure from Python, which is pretty sweet. I abused the internals here, and added it to dput-ng as a quick hack. Basically, I just had to import clojure.main, and let the main function set up the sys shim. I would set it up in dput, but I d have to import that module anyway. Remember, when doing this, the namespace ((ns ...)) needs to match the filename. So, I present the first Clojure hook for dput-ng: 
; Copyright (c) Paul R. Tagliamonte <paultag@debian.org>, 2012, under the
; terms of dput-ng itsself.
(ns clojtest
  (:require dput.core
            dput.exceptions))
(defn log [x]  ; for debug output
  (.debug dput.core/logger x))
(defn dput-checker [changes profile interface]
  (cond (>= (-> changes (.get "maintainer") (.find "arno@debian.org")) 0)
    (throw (dput.exceptions/HookException. "Maintainer's Arno. Aborting upload"))
  :else
    (log "Nah, it's not arno, we're good")))
This, of course, checks if the maintainer is Arno, and throws a fit if it is. Gist is over on gist.github. After, Algernon added his own checker, which throws a fit if you have a package in the B-Ds that you aught to not use (such as dpatch) 
; Copyright (c) Gergely Nagy <algernon@debian.org>, 2012, under the
; terms of dput-ng itself.
(ns bd-blacklist
  (:require dput.core
            dput.exceptions
            dput.dsc))
(defn prune-build-deps
  "Prune a string representation of the build-depends so that only a
  list of packages remain."
  [bd-string]
  (map #(first (-> % (.strip) (.split " "))) (.. bd-string (split ","))))
(defn has-blacklisted?
  "Given a dsc file and a blacklist, check if any of the
  build-depencencies are in that list. Throws an error if there are
  matches."
  [dsc-file blacklist]
  (let [dsc (dput.dsc/parse_dsc_file dsc-file)
        build-deps (prune-build-deps (.. dsc (get "build-depends")))]
    (if-let [bad-bd (some blacklist build-deps)]
      (throw (dput.exceptions/HookException. (str "Blacklisted build-dependency found: " bad-bd)))
      (-> dput.core/logger (.trace "Build-Dependencies do not have anything on the blacklist")))))
(defn blacklist-checker
  "Checks whether the dsc has blacklisted build-dependencies, ignores
  the check when no dsc is to be found."
  [changes profile interface]
  (if-let [dsc-file (.. changes (get_dsc))]
    (has-blacklisted? dsc-file (set (get profile "bd-blacklist")))
    (-> dput.core/logger (.trace "No .dsc found, build-dependencies cannot be checked"))))
You can check that out in the git tree Hacks welcome!

13 December 2012

Andrea Veri: The future is Cloudy

Have you ever heard someone talking extensively about Cloud Computing or generally Clouds? and have you ever noticed the fact many people (even the ones who present themselves as experts) don t really understand what a Cloud is at all? That happened to me multiple times and one of the most common misunderstandings is many see the Cloud as something being on the internet. Many companies add a little logo representing a cloud on their frontpage and without a single change on their infrastructure (but surely with a price increment) they start calling their products as being on the Cloud. Given the lack of knowledge about this specific topic people tend to buy the product presented as being on the Cloud without understanding what they really bought. But what Cloud Computing really means? it took several years and more than fifteen drafts to the National Institute of Standards and Technology s (NIST) to find a definition. The final accepted proposal:

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

The above definition requires a few more clarifications specifically when it comes to understand where should we focus on while checking for a Cloud Computing solution. A few key points:
  1. On-demand self-service: every consumer will be able to unilaterally provision multiple computing capabilities like server time, storage, bandwidth, dedicated RAM or CPU without requiring any sort of human interaction from their respective Cloud providers.
  2. Rapid elasticity and scalability: all the computing capabilities outlined above can be elastically provisioned and released depending on how much demand my company will have in a specific period of time. Suppose the X company is launching a new product today and it expects a very large number of customers. The X company will add more resources to their Cloud for the very first days (where they suppose the load to be very high) and then it ll scale the resources back as they were before. Elasticity and scalability permit the X company to improve and enhance their infrastructure when they need it with an huge saving in monetary terms.
  3. Broad network access: capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, tablets, laptops, and workstations).
  4. Measured service: Cloud systems allow maximum transparency between the provider and the consumer, the usage of all the resources is monitored, controlled, and reported. The consumer knows how much will spend, when and in how long.
  5. Resource pooling: each provider s computing resources are pooled to serve multiple consumers at the same time. The consumer has no control or knownledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).
  6. Resources price: when buying a Cloud service make sure the cost for two units of RAM, storage, CPU, bandwidth, server time is exactly the double of the price of one unit of the same capability. An example, if a provider offers you one hour of bandwitdh for 1 Euro, the price of two hours will have to be 2 Euros.
Another common error I usually hear is people feeling Cloud Computing just as a place to put their files online as a backup or for sharing them with co-workers and friends. That is just one of the available Cloud features , specifically the Cloud Storage , where typical examples are companies like Dropbox, Spideroak, Google Drive, iCloud and so on. But let s make a little note about the other three features :
  1. Infrastructure as a Service (IaaS): the capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. In this specific case the consumer has still no control or management over the underlying Cloud infrastructure but has control over operating systems, storage, and deployed applications. A customer will be able to add and destroy virtual machines (VMs), install an operating system on them based on custom kickstart files and eventually manage selected networking components like firewalls, hosted domains, accounts.
  2. Platform as a Service (PaaS). the capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages, libraries, services, and tools (like Mysql + PHP + PhpMyAdmin or Ruby on Rails) supported by the provider. In this specific case the consumer has still no control or management over the Cloud infrastructure itself (servers, OSs, storage, bandiwitdh etc.) but has control over the deployed applications and configuration settings for the application-hosting environment.
  3. Software as a Service (SaaS): the capability provided to the consumer is to use the provider s applications running on a Cloud infrastructure. The applications are accessible through various client devices, such as a browser, a mobile phone or a program interface. The consumer doesn t not manage nor control the Cloud infrastructure (servers, OSs, storage, bandwidth, etc.) that allows the applications to run. Even the provided applications aren t customizable by the consumer, which should rely on limited configuration settings.
The Cloud Computing technology is reasonably the future but can we trust Cloud providers? Are we sure that no one will ever have access to our files except us? and what about governments interested in acquiring a specific customer data hosted on the Cloud? I always suggest to read deeply both the Privacy Policy and Terms of Use of a certain service before signing in especially when it comes to choose a Cloud storage provider. Many providers have the same aspect, they seem to provide the same resources, the same amount of storage for the same price but legally they may present different problems, and that is the case of Spideroak vs Dropbox. Quoting from the Dropbox s Privacy Policy:
Compliance with Laws and Law Enforcement Requests; Protection of DropBox s Rights. We may disclose to parties outside Dropbox files stored in your Dropbox and information about you that we collect when we have a good faith belief that disclosure is reasonably necessary to (a) comply with a law, regulation or compulsory legal request; (b) protect the safety of any person from death or serious bodily injury; (c) prevent fraud or abuse of DropBox or its users; or (d) to protect Dropbox s property rights. If we provide your Dropbox files to a law enforcement agency as set forth above, we will remove Dropbox s encryption from the files before providing them to law enforcement. However, Dropbox will not be able to decrypt any files that you encrypted prior to storing them on Dropbox.
It s evident that Dropbox employees can access your data or be forced by legal process to turn over your data unencrypted. On the other side, Spideroak on its latest update to its Privacy Policy states that data stored on their Cloud is encrypted and inaccessible without user s key, which is stored locally on user s computers. And what about the latest research paper, titled Cloud Computing in Higher Education and Research Institutions and the USA Patriot Act written by the legal experts of the University of Amsterdam s Institute for Information Law stating the anti-terror Patriot Act could be theoretically used by U.S. law enforcement to bypass strict European privacy laws to acquire citizen data within the European Union without their consensus? The only requirement for the data acquisition is the provider being an U.S company or an European company conducting systematic business in the U.S. For example an Italian company storing their documents (protected by the European privacy laws and under the general Italian jurisdiction) on a provider based in Europe but conducting systematic business in the United States, could be forced by U.S. law enforcement to transfer data to the U.S. territory for inspection by law enforcement agencies. Does someone really care about the privacy of companies, consumers and users at all? or better does privacy exists at all for the millions of the people that connect to the internet every day?

12 December 2012

Vincent Sanders: What man's mind can create, man's character can control.

I have a project that required me to programmatically control power to several devices. I have done this before using a Velleman vm8090 board which is relatively easy to control. However they are relatively expensive.

I turned to ebay and found a similar module at a substantially reduced cost. Upon receipt however I discovered that instead of being a simple serial USB interface it presented USB HID and the Debian system I was running it on has loaded the hiddev driver for me but it did not implement any of the standard HID Usage Pages leaving me with no way to control the device.
I did the obligatory 
sudo lsusb -d 12bf:ff03 -vvv

Bus 003 Device 019: ID 12bf:ff03  
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         8
  idVendor           0x12bf 
  idProduct          0xff03 
  bcdDevice            1.00
  iManufacturer           1 Matrix Multimedia Ltd.
  iProduct                2 Flowcode USB HID
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           41
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower               50mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      0 No Subclass
      bInterfaceProtocol      0 None
      iInterface              0 
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.10
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      54
         Report Descriptors: 
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval               5
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval               5
Device Status:     0x0980
  (Bus Powered)
This simply showed me what I already knew and surprised me that lsusb did not dump HID report descriptor items. Some searching revealed that teh device had to be unbound so lussb could access the descriptor.

Thus a simple 
echo -n 3-1.1.4:1.0   sudo dd of=/sys/bus/usb/drivers/usbhid/unbind
resulted in lussb dumping the descriptor items:

Item(Global): Usage Page, data= [ 0xa0 0xff ] 65440
                (null)
Item(Local ): Usage, data= [ 0x01 ] 1
                (null)
Item(Main  ): Collection, data= [ 0x01 ] 1
                Application
Item(Local ): Usage, data= [ 0x02 ] 2
                (null)
Item(Main  ): Collection, data= [ 0x00 ] 0
                Physical
Item(Global): Usage Page, data= [ 0xa1 0xff ] 65441
                (null)
Item(Local ): Usage, data= [ 0x03 ] 3
                (null)
Item(Local ): Usage, data= [ 0x04 ] 4
                (null)
Item(Global): Logical Minimum, data= [ 0x00 ] 0
Item(Global): Logical Maximum, data= [ 0xff 0x00 ] 255
Item(Global): Physical Minimum, data= [ 0x00 ] 0
Item(Global): Physical Maximum, data= [ 0xff ] 255
Item(Global): Report Size, data= [ 0x08 ] 8
Item(Global): Report Count, data= [ 0x08 ] 8
Item(Main  ): Input, data= [ 0x02 ] 2
                Data Variable Absolute No_Wrap Linear
                Preferred_State No_Null_Position Non_Volatile Bitfield
Item(Local ): Usage, data= [ 0x05 ] 5
                (null)
Item(Local ): Usage, data= [ 0x06 ] 6
                (null)
Item(Global): Logical Minimum, data= [ 0x00 ] 0
Item(Global): Logical Maximum, data= [ 0xff 0x00 ] 255
Item(Global): Physical Minimum, data= [ 0x00 ] 0
Item(Global): Physical Maximum, data= [ 0xff ] 255
Item(Global): Report Size, data= [ 0x08 ] 8
Item(Global): Report Count, data= [ 0x08 ] 8
Item(Main  ): Output, data= [ 0x02 ] 2
                Data Variable Absolute No_Wrap Linear
                Preferred_State No_Null_Position Non_Volatile Bitfield
Item(Main  ): End Collection, data=none
Item(Main  ): End Collection, data=none

By consulting the device class definitions document I determined the device was using the "Vendor defined" Usage page (0xff00 to 0xffff) so I would definitely have to write a program to control the device.

Linux provides a really easy interface to deal with HID devices called hiddev (gosh, such adventurous naming) which I already had to unbind to get my descriptors decoded so I am fairly sure it works ;-)

The kernel documentation and header for hiddev provide the absolute basic mechanics of the interface but no example code or guidance. The obligatory web search turned up very little and even that had to be retrieved from the internet archive. So It seems I would be forced to work it through myself.

It seems the hiddev interface is orientated around HID devices generating reports which the program is expected to read. Numerous ioctl() are provided so the program can obtain the descriptor information necessary to control and process the received reports.

However in this case we need to be able to send reports to the device, all the descriptor information revealed was that there were eight (Item Report Count = 8) values with eight bits each (Item Report Size = 8) with logical and physical values representing the whole range of the octets.

Fortunately the seller provided a website with some control programs and even source. After some time rummaging through the Visual Basic program I finally found (in FrmMain.vb:2989) that the eight bytes were largely unused and the first was simply a bitmask of the eight relays coil status, set for energised clear for off. With bit 0 controlling relay labelled 1 through to bit 7 for relay 8.

To send a report to a HID device the hiddev interface uses the HIDIOCSREPORT ioctl where the report data is first set using HIDIOCSUSAGE .

The HIDIOCSUSAGE ioctl is passed a hiddev_usage_ref structure which must be initialised with information about the report descriptor identifier (constructed from the Usage Page and Usage as set by the items in the descriptor), the index of the item (named usage) we wish to set in the report (in this case the first which is 0) and the value we actually want to set.

After a great deal of debugging the final program is very short indeed but does the job, my main problem now is that if I switch too many (more than one) relays at once the whole device resets. The scope says the supply rails are behaving very badly when this happens, looks like I need to add a load of capacitance to the power well to stabilise it during the switching events.

Oh and add in the fact Relay 1 LED doest work unless you push on it and I do wonder about the wisdom of the economy in this case. Though yet again Linux makes the software side easy.

Raphael Geissert: A bashism a week: $RANDOM numbers

Commonly used to sleep a random amount of time or to create unique temporary file names, $RANDOM is one of those bashisms that you are best avoiding it altogether.

It is not uncommon to see scripts generating a "unique" temporary file name with code that goes like: tempf="/tmp/foo.$RANDOM", or tempf="/tmp/foo.$$.$RANDOM".

Under some shells the "unique" temporary file name will be "/tmp/foo." for the first example code. So much for randomness, right?

Even if you go around it by defining $RANDOM to the output of cksum after reading some bytes from /dev/urandom, please: don't do that. Use the mktemp command instead.
When creating temporary files there's more than just generating a file name. Just don't do it on your own: use mktemp. Really, use it, the list of those who weren't using mktemp (or similar) is large enough as it is.

Don't even dare to mention the linux kernel-based protection against symlink attacks. There's no excuse for not using mktemp.

Tip: If you are going to use multiple temporary files, create a temporary directory instead. Use mktemp -d.
Tip: Don't reuse a temporary file's name, even if you unlink/remove it. Generate a new one with mktemp.
Tip: Reusing also means doing things like tmp="$(mktemp)"; some_command > "$tmp.stdout" 2> "$tmp.stderr"
Tip: Even if $RANDOM is not empty, don't use it. It could have been exported as an environment variable. Again, just use mktemp.

For the remaining cases where you may want a pseudo random number such as for sleeping a random number of seconds: you can use something as simple as $$. Use shell arithmetic to adjust it as needed: use the modulo operator, multiply it, etc.

If you think you need something more "random" than the process' id, then you should probably not be using $RANDOM in the first place.

Next.

Previous.