Search Results: "beh"

5 January 2013

Olly Betts: Empty Strings in C++

Gaurav (one of Xapian's GSoC students this year) queried my advice that it's better not to explicitly initialise a std::string in C++ like this:
std::string s = "";
but instead to write:
std::string s;
It could be argued that the former makes it clearer that the string is initialised (since C++ does inherit C's behaviour of not implicitly initialising variables of fundamental types, such as int and double). But objects aren't left uninitialised - the default constructor gets called (or if there isn't one, the code won't compile). The downside is that you get quite a lot more code from the first version than the second. Perhaps compilers will grow smarter and in future both the above will compile to the same code, but that's not true today. Here's a simple bit of test code:
#include <string>
std::string f()  
#ifdef EXPLICIT_INIT
    std::string s = "";
#else
    std::string s;
#endif
    return s;
 
And I'll compile it with GCC (g++ (Debian 4.7.2-4) 4.7.2) on x86-64 to produce assembler output:
$ g++ -O2 -S s.cc -o s1.s
$ g++ -DEXPLICIT_INIT -O2 -S s.cc -o s2.s
$ diff s1.s s2.s
1a2,4
>   .section        .rodata.str1.1,"aMS",@progbits,1
> .LC0:
>   .string ""
9,10c12,25
<   movq    %rdi, %rax
<   movq    $_ZNSs4_Rep20_S_empty_rep_storageE+24, (%rdi)
---
>   pushq   %rbx
>   .cfi_def_cfa_offset 16
>   .cfi_offset 3, -16
>   movl    $.LC0, %esi
>   movq    %rdi, %rbx
>   subq    $16, %rsp
>   .cfi_def_cfa_offset 32
>   leaq    15(%rsp), %rdx
>   call    _ZNSsC1EPKcRKSaIcE
>   addq    $16, %rsp
>   .cfi_def_cfa_offset 16
>   movq    %rbx, %rax
>   popq    %rbx
>   .cfi_def_cfa_offset 8
(As an aside, I wouldn't recommend spending a lot of time digging into the assembler your compiler produces, but if there's more than one equivalent way to write code for a common case (as here) and you want to know which is most efficient, it can be informative to see what code is produced for each). You don't really need to know x86-64 assembler to grasp what's happening above, which is lucky, since I don't really know x86-64 assembler. In the first hunk, we get an empty literal string being added to the rodata (read-only data) section; and in the second, instead of two instructions which copy a standard empty string representation, we get much more elaborate code, including a function call to _ZNSsC1EPKcRKSaIcE - the C++ mangled name for the std::string from const char * constructor:
$ c++filt _ZNSsC1EPKcRKSaIcE
std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&)
I also tested the example above with clang (Debian clang version 3.0-6 (tags/RELEASE_30/final) (based on LLVM 3.0)) and the resultant code is fairly similar for each case. The overhead for doing this once isn't going to matter, but if it happens every time you declare a std::string variable, the cumulative effects may be measurable. And as well as taking longer to execute, the larger code will cause greater CPU cache pressure.

4 January 2013

Stefano Zacchiroli: bits from the DPL for December 2012

Happy new year, Debian! To celebrate, here are some freshly posted, bits from the DPL for December 2012.
Dear Project Members, happy new year! Here goes another report of DPL activities, this time for December 2012. This issue of the DPL-monthly is skinnier than usual: during the past month I've been struck by the catastrophe also known as "family holiday season", enjoying a solid 10 day break from computer-related activities. Talks Assets DPL helpers Collaboration with the outer world That's all for last year, enjoy the new one, which will soon see a new Debian release out of the door. And to make it happen sooner, let's go back fix RC bugs! Cheers.
PS the day-to-day activity log for December 2012 is available at the usual place master:/srv/leader/news/bits-from-the-DPL.txt.201212

2 January 2013

Russ Allbery: podlators 2.5.0

podlators is the package that contains Pod::Man and Pod::Text and the pod2man and pod2text driver scripts, which convert Perl's POD documentation format into text and *roff (for man pages) output. This release is a "clearing the decks" release that rolls up various smallish feature requests and bug fixes that have been pending for a while. I want to do the same reworking to podlators that I did to Term::ANSIColor, but it's going to take longer since it's quite a bit more code. (And Term::ANSIColor took me about two days.) I didn't want to block the various pending issues on that work, so this tries to clean up all the open bugs I was aware of other than sorting out the defaults for Unicode output. (That's a much larger problem and will probably be fixed in a major release.) The primary change is that pod2man and pod2text now die by default if the POD contains syntax errors. This will hopefully flush out numerous POD documents around the world that have been carrying POD ERRORS sections around forever. I considered just warning by default but not failing, since this will also cause CPAN builds to fail, and I may fall back to that if there are too many complaints. But usually fixing the POD syntax is straightforward, and normally parsing programs will fail on syntax errors. Note that the defaults for the modules, Pod::Man and Pod::Text, continue to be to add a POD ERRORS section, since aborting inside a module isn't very friendly to the caller. There is a new option for both the modules and the driver scripts to set the error handling behavior, which can be set to die, report to standard error but continue, add a POD ERRORS section, or ignore errors entirely. Also in this release is a new nourls option that suppresses the URLs in text and *roff output for L<> codes that have anchor text, which can declutter some POD texts, and fixes for X<> formatting codes containing whitespace and small-caps handling of paragraphs consisting entirely of all-caps text. You can get the latest release from the podlators distribution page.

Raphael Geissert: A bashism a week: read

Whether for interacting with the caller, for reading the output of some command, or a file descriptor in general, the read shell command can be found in many scripts.

Unless you stick to the POSIX:2001-required "read variable_name", possibly with the -r option, you should expect problems.


dash, for instance, supports prompts but nothing else.

Remember, if you rely on any non-standard behaviour or feature make sure you document it and, if feasible, check for it at run-time.

1 January 2013

Russ Allbery: Review: City of Diamond

Review: City of Diamond, by Jane Emerson
Publisher: DAW
Copyright: March 1996
ISBN: 0-88677-704-6
Format: Mass market
Pages: 624
The setting is deep into the star-faring future of humanity. Humans have settled many other worlds, thanks largely to technology from a race named the Curosa. There is a thriving network of settlement based around a sort of warp conduit network. But, standing apart from most human settlements, are three huge ships: the City of Opal, the City of Pearl, and the City of Diamond. Their inhabitants are devout members of the Redemptionist faith, a merging of Christian (primarily Catholic) belief and Curosa religion. Their ships are gifts from the now-departed Curosa, featuring drives unknown to the rest of humanity that allows them to ignore the normal transport network. They are meant to witness and spread their religion, but now, six centuries later, they primarily trade. Much of this is not apparent at the start. City of Diamond opens with Adrian Mercati, chosen heir of the current dying Protector, dodging people in what seems to be an oddly-organized city. Impudent, occasionally flippant, utterly charming, and in love with danger, he inherits the position of Protector shortly after we meet him. Emerson slowly reveals more of the nature of the ship while Adrian is shocking (and charming) the nobility and the church, and consorting with demons. Or, at least, one demon: Tal, an entirely human-appearing creature except for his eyes, who the Redemptionists claim is born without a soul. Slowly, it becomes clear that he's actually the offspring of an alien and a human, a mating of species whose children cannot socialize as human and behave in ways that seem like human sociopaths. Tal is no exception, but his interests align with Adrian's for the time. To those two viewpoint characters, Emerson adds four more: a low-rank administrator named Spider who works for Tal after Tal saved him from death by recycling, Adrian's politically-arranged bride-to-be from the rival (and even more religious) ship City of Opal, a guard from a French-descended slum on City of Opal, and a dangerously honorable and literal mercenary from an off-shoot culture of humanity that's now only a legend. This is a book that tends to sprawl. There's quite a lot going on, starting with the negotiations between City of Diamond and City of Opal to put to rest the remnants of a civil war and leading to the search for a legendary Curosa artifact on the planet they're both visiting. All of the characters have their own agendas, their own history, and their own goals, and are pursuing them in parallel and occasionally in opposition. Emerson takes the time to let all of this build. Despite that, there is very little in this book that drags or bores, and that's due primarily to the strength of the characters. Every one of the viewpoint characters is unique, bringing a different perspective to the story, and every one of them changes and grow over the course of the story. They also play off of each other in delightful ways. This is a book that makes full use of the fact that three characters make seven character interactions. For example, Tal by himself is logical but just slightly askew, Tal seen by Spider is uncomfortable and worrisome, Tal and Adrian is a dance of mutual amusement and respect, and Tal and Keylinn finding points of commonality is just delightful. I did wish that there was a bit more Keylinn and Tal and a bit less of Spider, but once Emerson built her world, introduced her characters, and got the story moving, there aren't many weak pieces. Part of the fun is figuring out the world background. There are almost no infodumps here; Emerson just starts telling stories of characters in the surroundings in which they find themselves, and the reader is left to piece things together from background description, the occasional explanation from one character to another, and parallels with long-standing SF tropes. It's not a puzzle book, but it is a book of where the reader is dropped as an alien into a fully-developed culture. In both that way and in the use of nobility, religion, and formal hierarchy against a starfaring backdrop, City of Diamond reminds me of the sort of classic space opera that led up to Star Wars but isn't seen as much these days. It's also funny. I was unsurprised to discover that Emerson is a pseudonym for Doris Egan, who (in addition to writing some other SF) is an occasional writer and producer for House. The humor isn't entirely obvious at the start, although Adrian frequently provokes a grin from the start of the book, but Tal's perceptive but obliviously blunt comments become more and more fun as the book progresses, and both Iolanthe (Adrian's bride) and Keylinn (the mercenary, sort of) are a delight. I want to see more of both of them, but sadly the book ends just as they (particularly Iolanthe) are hitting their strides. And that's the biggest problem with City of Diamond: it's a failed (or at least postponed) trilogy. As with Walter Jon Williams's Metropolitan and City on Fire, there were supposed to be more books, but apparently either the publisher lost interest or other projects took precedence. The story does come to something of a conclusion, but most of the major story arcs aren't resolved. Emerson is still bringing more guns onto the set late in the book. None of the villains truly get their just desserts, nor do most of the character arcs fully resolve, although friendships and more have formed satisfyingly by the end of the book. There is obviously far more to both these characters and this universe, including an entire ship that was never visited and an alien whose presence should prove startling. This is a great book: warm, funny, full of interesting characters, and with a nicely complex background universe. If it were complete in itself, or if the sequels had materialized, I would recommend it unreservedly. As is, you will have to weigh your dislike of unfinished stories against its stand-alone merits. But I enjoyed it a great deal. Rating: 7 out of 10

30 December 2012

Iustin Pop: Interesting tool of the day: ghc-gc-tune

Courtesy of a recent Google+ post/Stack overflow answer, I stumbled upon ghc-gc-tune, a simple but nice tool which generates interesting graphs for Haskell programs. What is does is quite trivial: iterate over a range of arena and heap sizes, and run a specified program with those RTS options, then generate a graph comparing the performance (by default cpu time) across the combinations of the values. Note that for newer GHC versions, you'll need to link with -rtsopts to allow for the -A/-H custom sizes. The reason I mention it is that the graphs it generates can be quite interesting. For one of the Ganeti programs, hspace, run with the command line ./hspace --simu p,20,1t,96g,16,1 --disk-template drbd, it generates this graph: runtime graph with trivial usage This picture, if I read it correctly, says that this program is actually well behaved (well, +RTS -s says "3 MB total memory in use" with default options), and that the optimum sizes actually relate to the (L1? L2?) cache size. But note that the maximum difference is only about ~1.6 . By changing the parameters to hspace and make it allocate more memory (./hspace --simu p,20,1t,128g,256,32 --disk-template plain --tiered=1g,128m,1, ~52MB reported by +RTS -s), the graph changes significantly: runtime graph with 50mb usage Now we have somewhat the opposite situation: very small arena sizes are detrimental (and by a big factor, 4.5 ), large arena/heap sizes are OK-ish, and the sweet spot is around 2-4MB arena size with heap sizes up to 4MB. Now maybe these particular examples were not very elightening (and they were definitely not well-conducted tests, etc.), but they should allow some intuition into how the program behaves. Plus, the tool can also generate other plots, for example peak memory usage.

28 December 2012

Russ Allbery: Review: Perl Best Practices

Review: Perl Best Practices, by Damian Conway
Publisher: O'Reilly
Copyright: July 2005
Printing: April 2011
ISBN: 0-596-00173-8
Format: Trade paperback
Pages: 492
I had a rather bad first reaction to this book or, rather, the existence of this book. Some time ago, I started hearing about a new program named perlcritic that served as a lint program for Perl and decided to give it a try. I was taken aback by the flood of diagnostics, but even more taken aback by the fact that most of them were apparently explained only by a reference to a book that I didn't own. It struck me wrong in an open source project and gave me a negative reaction towards the book, which lingered for years. That was exacerbated by discovering that it recommended using use English in some circumstances, something that I consider to be a horrible idea. So, when a coworker recommended it and suggested I really should read it, I was dubious. I'm glad I gave it a chance, since it turns out that nearly all of my objections were based on misunderstandings or false information. (Well, except for the use English part, but more on that in a moment.) Perl Best Practices does provide a lot of advice, explaining that flood of diagnostics, but perlcritic also warns about the same thing repeatedly, inflating a handful of issues into a lot of messages. Most of the recommendations in Perl Best Practices were things I was doing already; most of the rest are entirely sensible. Also, perlcritic does come with complete explanations of its diagnostics once you know where they're hidden (the perldoc of Perl::Critic::Policy modules whose names you can get at by insisting perlcritic divulge them). And, even more delightfully, once I got past my objections, I discovered that Perl Best Practices is one of the best books on coding style that I've read. Coding style is a notoriously difficult subject, since nearly everyone is convinced their personal style is the best (or at least that they don't want to change). Conway acknowledges that up-front, and acknowledges that some things are just arbitrary and consistency with any rule set is better than following some specific rule set. He then spends surprisingly little time on the sorts of things that dominate most coding style discussions (indentation, spacing, commenting, etc.). Instead, this is primarily a book about how to write efficient and transparent Perl code, one that focuses much more on the content and structure of that code than on how it is indented. The best part is that there are no undefended rules here. Perl Best Practices contains 256 separate guidelines, some quite general ("Provide an optimal interface, rather than a minimal one") and some quite specific ("Pass constructor arguments as labeled values, using a hash reference"), but every one of them has a supporting argument and specific reasons behind it. Conway backs up every recommendation with code samples and practical scenarios showing the pluses and minuses of different approaches. This is a book unconcerned with being an authority; rather, it reads like optional advice from a good mentor, complete with insightful discussions on each point raised. You may come away from it disagreeing with Conway's specific recommendation (and I did, in several places), but you'll almost certainly be better-informed about the problem for having read it. Plus, this book is just fun to read. Admittedly, I love this sort of thing; I spend lots of time thinking about coding style in any language, and a lot more time refactoring coding style when working on my own projects to try to achieve clarity and simplicity. But even without sharing my love of the material, I think this book will be a delight for any practicing Perl programmer who hasn't yet read it. Conway clearly has a wealth of experience and years in the trenches of working with other people's Perl code. The examples he draws on are brief, comprehensible, and illuminating. He also gave me a tour of a variety of helper modules and libraries that I didn't know existed. List::Util is the most prominant example; this incredibly helpful supply of list manipulation functions that I've always half-known I wanted but never bothered to write is now part of Perl core (as of 5.7.3), so there's no reason to avoid it. You too can stop using grep when you actually mean first, or rewriting max in ad hoc ways. That does raise two of the rare flaws in this book, though. One is that I disagree (partially) with Conway's willingness to recommend non-core modules. He is a particular champion of the Readonly module over either normal file global variables or use constant, for very good reasons. But making Readonly a prerequisite for all other Perl code that one releases seems a bit annoying (speaking as someone who used to maintain local Perl installations). Maybe this isn't as big of a deal in these days of ubiquitous Linux distributions and willingness to use the CPAN shell, but I still dislike releasing code that depends on a half-dozen (or more) other CPAN modules for reasons of coding convenience and minor readability rather than core functionality. (His advocacy of Sub::Install raised similar objections.) The second flaw is not something Conway could have done anything about: this book is from 2005, and it's now 2012. Now, the Perl ecosystem is now quite mature, so that hampers the book much less than it would a corresponding book about Ruby or even Python. I also aim for compatibility with the oldest thing out there I know someone is still running, so I'm still hesitant to require anything newer than Perl 5.8. But still, this book was written seven years ago, and Perl is now at 5.14. In the meantime, we've seen core language features like given/when added, the addition of state variables, the need for substantial reworking of how Unicode I/O is handled, the addition of the new // operator, and the new package syntax, among many other things. The Perl taught here is a bit dated, and I would love to hear Conway's opinion about when (or if) to use say. Maybe someday we'll get a second edition. There are two other points that I do have to argue with. One is that Conway does advocate, in a limited way, use English. I could almost go along with his recommendation to use it only for those variables that one has to look up, except that in nearly every case there are other options and one can take the much better approach of simply not using those variables at all. But he then writes the remaining examples in the book using $EVAL_ERROR rather than $@, and I'm sorry, but no. One could not construct a more frustrating annoyance for an experienced Perl programmer than sending them on a futile hunt for where the global variable $EVAL_ERROR is defined. One can debate the merits of Larry Wall's original decision to use variables like $_, @_, $?, $!, and $?, but for better or ill this is Perl, and marginal readability gains for inexperienced programmers do not warrant confusing experienced programmers. This is particularly true when the use English variables are completely indistinguishable from normal global variables, and not infrequently look like a global variable that a programmer might choose. The second is more of an amusement that a real disagreement. Conway rightfully points out that Perl OO, and particularly the way that Perl stores object instance data, is awkward, and none of the commonly-used techniques are particularly satisfying. He then introduces a completely new way of doing instance data that, while having some advantages, has a very strange syntax and is utterly unlike how any OO code one finds in the wild is written. While the discussion of the problem that he's trying to solve is still interesting, I think the solutions he proposes will be more of a curiosity than a style that most programmers will adopt. But, those disagreements aside, I thoroughly enjoyed this book. I found it so engrossing that I kept reading it straight through a long train ride rather than switching to fiction halfway, like I do with most programming books. I've since recommended it to several other coworkers, and we're in the process of revising our internal Perl coding style based heavily on it. It's the sort of book that I expect to re-read several times to refresh my memory of Conway's analysis. If you've been avoiding this one due to misconceptions like I had, or because you think it might be too dry, or even just because you're afraid it's outdated, I strongly recommend it. If you're worried about the price of a printed (or electronic) book on this topic and you do Perl programming as part of your job, see if your employer will buy you a copy as training. It's worth it. I've been writing Perl professionally for more than fifteen years, and I learned a ton from this book. (And if it inspires you to change your coding style, take a look at perlcritic. It requires some configuration, and getting started is quite frustrating, but it reliably detects a surprisingly high percentage of the issues discussed here and can help a great deal in making the coding style shift stick until you can build new habits.) Rating: 9 out of 10

27 December 2012

Jamie McClelland: Live video streaming using vp8enc and webm on Debian

The vp8 video codec and webm container format (which are alternates for theora/ogg and h264/flv) are getting lots of attention, especially as more web browsers are supporting html5 - which includes a new video tag that plays video directly in a web page, rather than requiring a plugin. With Google fully behind these new formats (and converting YouTube videos) and even skype using vp8, it seems as though it could supplant h264/flash as the default video codec and container. After years under the boot of the proprietary flash video, this change could usher in a lot of exciting developments. While the shift in formats is also significant for on-demand video, I'm focusing on live streaming in this blog. There are a lot of tools needed to successfully stream a live video. Enough of them support vp8 and webm to create a live video stream, however, not all of the right versions of these tools have landed in Debian. Without any patches or changes, if you try to send a live video stream to icecast using vp8 encoding and webm, you'll get something like this:
0 jamie@animal:~$ gst-launch v4l2src ! video/x-raw,width=320,height=240 ! vp8enc ! webmmux ! shout2send ip=icecast.server port=8000 password=secret mount=/test.webm
WARNING: erroneous pipeline: could not link webmmux0 to shout2send0
1 jamie@animal:~$
I've documented the steps for patching/upgrading gstreamer and icecast2 so you can get a live video stream using vp8 encoding and the webmc container that originates on a Debian wheezy machine using gstreamer and streams to a Debian squeeze server running icecast2. On the workstation, you'll need libshout3 version 2.3.0 or higher, which is at the moment available in experimental and installable without bringing in any dependencies. In addition, you will need to bring in gstreamer1.0 from sid and you will need to rebuild gstreamer1.0-plugins-good after libshout-dev has been installed from experimental. So, begin by adding experimental and sid to your sources list by creating the files /etc/apt/sources.list.d/experimental.list and /etc/apt/sources/lists.d/sid.list with the respective contents:
deb http://mirror.cc.columbia.edu/debian experimental main
and
deb http://mirror.cc.columbia.edu/debian sid main 
deb-src http://mirror.cc.columbia.edu/debian sid main
In order to avoid upgrading your entire machine to sid (experimental packages won't upgrade by default), you'll also need to add the file /etc/apt/preferences.d/sid:
Package: *
Pin: release n=sid
Pin-Priority: 200
Then update and install:
apt-get update
apt-get install -t experimental libshout3
apt-get install gir1.2-clutter-gst-1.0 gir1.2-gst-plugins-base-1.0 gir1.2-gstreamer-1.0 gstreamer0.10-ffmpeg  gstreamer1.0-libav:amd64 gstreamer1.0-plugins-bad:amd64 gstreamer1.0-plugins-base:amd64 gstreamer1.0-plugins-base-apps gstreamer1.0-plugins-good:amd64 gstreamer1.0-pulseaudio:amd64 gstreamer1.0-tools gstreamer1.0-x:amd64 libclutter-gst-1.0-0:amd64 libgstreamer-plugins-bad1.0-0:amd64 libgstreamer-plugins-base1.0-0:amd64 libgstreamer-plugins-base1.0 libgstreamer1.0-0:amd64 libgstreamer1.0
Fortunately, the version of gstreamer1.0-plugins-good in sid includes a fix for a bug patched on November 31, 2012 that allows the use of the webm container. However, for it to work properly gstreamer1.0-plugins-good has to be built against libshout version 2.3.0 or higher (which is in experimental, not sid). Therefore, you have install libshout-dev from experimental and then rebuild gstreamer1.0-plugins-good.
sudo apt-get build-dep gstreamer1.0-plugins-good
sudo apt-get -t experimental install libshout-dev
apt-get source gstreamer1.0-plugins-good
cd gst-plugins-good1.0-1.0.4
dch -v 1.0.4-1+webmmux
fakeroot debian/rules binary
sudo dpkg -i ../gstreamer1.0-plugins-good_1.0.4-1+webmmux_amd64.deb
Now if you try the command, things should work on the client side. When streaming to an icecast2 server running squeeze, I don't get any errors, however, when opening the streaming page via the icecast URL, there is a play button, which can be clicked, but no picture is shown. That's because the version of Icecast2 in squeeze doesn't support either vp8 or webm. On the server side, you will need icecast2 version 2.4beta or higher. This version hasn't landed anywhere in Debian as of this writing. However, you can build it from source by following these steps (the patches I'm removing have all been applied to the new upstream version).
apt-get source icecast2
wget http://downloads.xiph.org/releases/icecast/icecast-2.4-beta.tar.gz
tar -xzvf icecast-2.4-beta.tar.gz
cp -r icecast2-2.3.2/debian icecast-2.3.99.0/
cd icecast-2.3.99.0/
rm debian/patches/ 1001_autotools_avoid_debian-subdir.patch,1002_add_missing_xspf_file,1003_fix_memory_leak.patch,series 
dch -v 2.3.99.0-1
fakeroot debian/rules binary
When you install it, be sure to replace the icecast2.xml configuration file and update it with your server-specific customizations. Hopefully this version of icecast2 will make it to experimental soon. And now... you can successfully stream via:
gst-launch-1.0 v4l2src ! video/x-raw,width=320,height=240 ! vp8enc ! webmmux !  shout2send ip=icecast.server port=8000 password=secret mount=/test.webm
If you want full audio and video, here's a more complete pipeline: gst-launch-1.0 v4l2src ! queue ! video/x-raw,width=320,height=240 ! tee name=tscreen ! queue ! autovideosink tscreen. ! queue ! videorate ! video/x-raw,framerate=25/2 ! queue ! vp8enc ! queue ! webmmux name=mux pulsesrc ! queue ! audioconvert ! vorbisenc ! queue ! queue ! mux. mux. ! queue ! shout2send ip=hobo port=8000 mount=jamie.webm password=secret Unfortunately, since I'm not sure how to deal with the python2.7 vs python3 dependecies, my elegant mobi python script that creates this pipeline for you using python-gtk as been reduced to a crude bash script. Another item on the todo list is opus audio encoding. gstreamer doesn't seem able to use opus in the webm container (although it seems to work with ogg). In the course of this research I found directions for getting gstreamer-1.0 installed on debian independent of packages installed on the system. If you follow the directions above, these steps should not be necessary. I'm including in case you need the latest sources for another reason. Extra steps I needed were:

25 December 2012

Cyril Brulebois: X-mas gift

I ve been working on some autotesting features for d-i over the past few weeks, and I ve started lagging behind on my debian-boot@ reading accordingly, by several hundreds of mails. Today s gift to myself: no more unread/unanswered mails in that mailbox! Even if I ve prepared things for a possible d-i wheezy rc1 release around Christmas, several things weren t ready, most notably: grub-installer. Hopefully #696615 will be fixed soonish, probably by asking where to write when /target(/boot) isn t on (hd0). Wanted! Speaking of grub-installer, if anyone can reproduce #681227, I m all ears! Wouter has a workaround for it (only in sid for now), but I d very much like to track down the root cause for that one. It is supposedly reproducible with wheezy beta 4 images and with weekly builds (they use components from testing), all of those are available from the debian-installer website.

Russ Allbery: Review: A Paradise Built in Hell

Review: A Paradise Built in Hell, by Rebecca Solnit
Publisher: Penguin
Copyright: 2009
Printing: 2010
ISBN: 0-14-311807-2
Format: Trade paperback
Pages: 319
A Paradise Built in Hell is a book with an agenda. Solnit's goal is to convince the reader that nearly everything we're shown in movies and popular culture about human behavior during disasters is wrong. Not only is destructive mass panic unusual verging on nonexistent, spontaneous cooperation and acts of startling courage and resourcefulness are commonplace. The heroes that we hear about afterwards are more the norm than the exception. Disasters are far more likely to bring out the best in people than the worst. They break down barriers and form spontaneous human communities that survivors remember for a lifetime. Rather than natural disasters leading inevitably to an exacerbating human crisis in the absence of strong authority, Solnit argues that they are an amazing human opportunity, and that the most negative and destructive human behavior during crises is not from untrained victims but from the panic of the elites who are supposedly responsible for protecting the public. I think her analysis and presentation is both deeply appealing and deeply flawed. The flaws involve extrapolation and generalization beyond the applicability of her research, and I'll say more about that in a moment. But the appeal is still surprising and satisfying, and I think there's a lot of truth in this book despite its one-sided presentation. I like to tell people, half-jokingly, that I'm an anarchosyndicalist. It's my political alliegance of the heart: the political structure that I know would never work, but that matches the world I want to live in. (I suspect much of the belief in libertarianism is of this form.) Now I have a book to point people at when they ask what the appeal of anarchosyndicalism is. The sense of spontaneous community, of bonding, of reaching out to other people to do something necessary, together, that Solnit describes here in the aftermath of a disaster matches that vision of a world without imposed economic authority, where people spontaneously collaborate to solve problems. A Paradise Built in Hell is structured by disaster, providing a tour of major disasters and their aftermath interspersed with extended musings on what the human reactions mean and how disaster communities functioned. The major disasters discussed are the 1906 San Francisco earthquake, the Halifax explosion, the 1985 Mexico City earthquake, 9/11, and Hurricane Katrina. Solnit moves through them in chronological order, closing the book with Katrina, which had (at least for me) the most infuriating and uplifting mixes of community bonding and elite panic. But both of those factors are present from the beginning in the 1906 earthquake: spontaneous human organization that (from multiple first-hand accounts) transformed people's lives mixed with a military reaction and elite panic that possibly did more damage to the city than the earthquake did. Elite panic is the dark side of Solnit's celebration of human altruism and spontaneous community organization. The primary picture she paints is of ordinary people responding with great courage and creativity and finding, in their reaction, a transformative experience and a sense of purpose that often becomes the most powerful and positive experience of their life. But this is at odds with a deeply-entrenched elite belief in mass panic, in the necessity of keeping information from the public to suppress their reaction, and in the need to re-establish "order." And with that, Solnit is absolutely scathing, and with some justification. Along with detailed accounts of these major disasters, Solnit also gives the reader a tour of the research literature on disasters, research that has shown that the panic and chaos that is at the center of practically every disaster movie ever made is essentially a myth. Mass panic in disasters is not only rare, it's close to non-existent. As you might expect, the handling of Katrina by both the government and by neighboring white communities is the subject of the harshest attacks. The material on Katrina presented here is enough to make anyone want to throw out the current playbook on how we react to disasters (and lest one think Solnit is unfair, her research matches very closely with other research and journalism on Katrina that I've seen from a wide variety of sources). But Solnit shows that this is a pattern: the combination of our media-driven belief in how people will react to disasters combined with the elite need to re-establish control plays out in many major disasters, and almost always negatively. Even in 9/11, the official response in some cases hampered and undermined an already-effective unofficial response. This is very interesting and has concrete implications for public policy around disasters. But where Solnit moves onto thinner anarchosyndicalist ice is the story that she tells about disasters as dramatic upheavals of the established social order that could point to a revolution in how we interact with each other. Solnit is deeply inspired by human altruism during disasters, and her enthusiasm is somewhat contageous, but it was useful to read this book shortly after Bruce Schneier's Liars and Outliers. I think Solnit, in her understandable eagerness to find a way to extend disaster behavior to general life, does not understand the relationships between what Schneier terms moral, reputational, and institutional pressures. And that's the core of why we need to both enable spontaneous human response in disasters and have organized first responders and supporting infrastructure, and why disaster communities will break down when extended beyond the disaster. When I first mentioned A Paradise Built in Hell to my mother, her first reaction was dubiousness due to her memory of the 1977 New York City blackout. There was a disaster that resulted in widespread looting and destruction, rather than the positive reaction Solnit describes. Solnit does have a cogent argument for why looting is poorly analyzed (we don't, for example, distinguish between breaking into stores to get desperately needed supplies and doing so for personal gain) and should not be a priority in a life-threatening disaster. But it's not completely convincing; in some disasters, particularly ones like the 1977 power outage that were not particularly life-threatening, the looting has been the most destructive part of the disaster. Schneier provides what I think is the missing piece, both to this and to why the wonderful behavior during disasters doesn't last into everyday life. During a life-threatening disaster, moral pressures are massively increased. Nearly all of us have self-images and moral beliefs that provide very strong incentives to help others and collaborate in the face of life-threatening emergencies, incentives that override other motives and lead to the sort of behavior that Solnit celebrates. But moral pressures have inherent limitations. That moral reaction applies most strongly to small groups (such as bands of survivors), and will start to break down as larger society re-establishes itself. And disasters, like a power outage, that are not immediately life-threatening will not provoke the same intensification of moral pressures: we don't have strong moral beliefs and internal stories about staying quietly at home during power outages the way that we do for saving people from burning buildings or pulling them out of rubble. Once larger communities have reformed and that short-term emergency reaction has dissipated, the need for institutional pressures (police, law, and formal authority structures) will return. I wish Solnit had been able to read Schneier's book before writing hers. I think his analysis structure provides a healthy dose of realism and perspective to her desire for a self-organizing communal world, and might have blunted her enthusiastic but somewhat unrealistic hopes for finding some key to a revolution in human organization in the heart of disasters. But if A Paradise Built in Hell is unlikely to lead to a radical change in how we organize societies, it should at least provoke a radical rethinking of some of our unnecessary pessimism about our fellow humans. The evidence is substantial and compelling: crisis brings out the best in people, not the worst, and the ordinary people who happen to be around are capable of becoming some of the most amazing disaster response teams that we can imagine. This has numerous mundane implications for social policy. We should consider simple training for the general public, for example, to strengthen that reaction, should allow for it in legal structures and training of first responders (around looting, for instance), and should do everything we can to kill the destructive myth that people cannot be trusted in disasters or that mass panic will be as much of a threat as the disaster. But the best part of this book is that it provides a concrete, well-documented, and well-defended reason to let go of some of our cynicism and to extend a bit of trust to our fellow humans. There sadly aren't earth-shaking implications for everyday society, but there is a wealth of evidence that shows that we're better people in the crunch than we believe we are, or that we expect of others. And the more that we can embrace the dissolution of boundaries and social hierarchies in the middle of disasters, the more we are likely to be surprised by deep human connection and a sense of shared, focused purpose. Solnit is a bit long-winded with her agenda, but that bright hope has stuck with me. Recommended. Rating: 8 out of 10

24 December 2012

Wouter Verhelst: m68k: back from the dead

It's been a few years coming, but the day has finally arrived. Contrary to some rumours which I've had to debunk over the years, the m68k port did not go into limbo because it was kicked out of the archive; instead, it did because recent versions of glibc require support for thread-local storage, a feature that wasn't available on m68k, and nobody with the required time, willingness, and skill set could be found to implement it. This changed a few years back, when some people wrote the required support, because they were paid to do so in order to make recent Linux run on ColdFire processors again. Since ColdFire and m68k processors are sufficiently similar, that meant the technical problem was solved. However, by that time we'd fallen so far behind that essentially, we needed to rebootstrap the port all over again. Doing that is nontrivial, and most of the m68k porters team just didn't have the time or willingness anymore to work on this; and for a while, it seemed like the m68k port was well and truly dead. But then, two things happened. The first thing that happened was called "ARAnyM", short for "Atari Running on Any Machine". An atari emulator, it started supporting running Linux at some point, allowing us to work on the port while on the road, using a laptop something not possible beforehand. While there were some bugs in the emulation of early versions of aranym, these were all fairly quickly fixed, and before long we could run Debian/m68k in the emulator. However, that was still the etch-m68k port on debian-ports.org, some time after Debian etch wasn't even supported anymore; this still required work. The second "thing" that happened was called "Thorsten Glaser". A fresh DD and maintainer of the "mksh" shell, he wanted to make sure his shell would work on all Debian architectures, including the ones on debian-ports.org. When he found out that the m68k port was in a pretty bad shape, he did not, like many before him, shrug and move on; instead, he took it upon himself to start compiling things, just so he could compile his shell. How's that for dedication. Anyway, fast-forward two years, and we're now, finally, at a point where I spent most of today getting buildd up and running on arrakis, one of our oldest buildd hosts. There were some issues while trying that, but it looks like everything is in working order now; and the first package to start building on the first functioning buildd host since a long time (the build of which is still ongoing at the time of this writing) is "babl". Of course, with only one buildd, we'll have no hope whatsoever of ever catching up. But that can be remedied easily by using more than one buildd host... Update: It built! Party!

23 December 2012

Cyril Brulebois: d-i hacking recipe #3: Debugging debconf

Problem #3: How to find out what s going on with debconf? Slightly longer version: As reported in #679327, the grub prompt during the installation process is entitled Configuring man-db , which isn t exactly great. Given man-db is a known dpkg trigger, I suspected this would be the reason for the broken title, and here s how I debugged it. For starters, using d-i basically means asking a lot of questions, which is implemented using the debconf mechanism. One should note that two implementations are available:
  1. the historic implementation, in Perl: debconf
  2. the C reimplementation: cdebconf
d-i uses cdebconf udebs, but joy begins when packages get installed in /target: debconf is used there. We ll see why that matters. Recipe #3: Set DEBCONF_DEBUG=developer and read logs!
  1. Both implementations support the DEBCONF_DEBUG environment variable. The installer supports passing that as a kernel parameter; example from the syslinux prompt: select the Graphical install entry, press Tab , remove -- quiet and put DEBCONF_DEBUG=developer there instead.
  2. Let s look at /var/log/syslog; there s a tail -f on it in VT4, but one can run grep and friends on it in VT2 or VT3. There, one can see what happens at the debconf level (the debconf protocol is documented in debconf-devel(7)). Tiny excerpt (timestamps removed for clarity):
    frontend: --> GET preseed/early_command
    frontend: <-- 0
     
    frontend: --> GET debian-installer/framebuffer
    frontend: <-- 0 true
     
    debconf: --> GET debconf/language
    debconf: <-- 0 en
    debconf: --> SET debconf/language en
    debconf: Setting debconf/language to en
    debconf: <-- 0 value set
    debconf: --> GET debconf/priority
    debconf: <-- 0 high
    debconf: --> GET rescue/enable
    debconf: <-- 0 false
    
  3. The interesting point was titles, right? So let s look at the last TITLE occurrences:
    debconf: --> SETTITLE debian-installer/main-menu-title
    debconf: --> SETTITLE debian-installer/grub-installer/title
    in-target: debconf (developer): ----> TITLE Configuration de man-db
    debconf: --> TITLE Configuration de man-db
    in-target: debconf (developer): <-- TITLE Configuration de man-db
    in-target: debconf (developer): ----> TITLE Configuration de man-db
    in-target: debconf (developer): ----> TITLE Configuration de grub-pc
    debconf: --> TITLE Configuration de grub-pc
    in-target: debconf (developer): <-- TITLE Configuration de grub-pc
    in-target: debconf (developer): ----> TITLE Configuration de grub-pc
    in-target: debconf (developer): ----> TITLE Configuration de man-db
    debconf: --> TITLE Configuration de man-db
    in-target: debconf (developer): <-- TITLE Configuration de man-db
    in-target: debconf (developer): ----> TITLE Configuration de man-db
    in-target: debconf (developer): ----> TITLE Configuration de grub-pc
    debconf: --> TITLE Configuration de grub-pc
    in-target: debconf (developer): <-- TITLE Configuration de grub-pc
    in-target: debconf (developer): ----> TITLE Configuration de grub-pc
    debconf: --> SETTITLE debian-installer/grub-installer/title
    
    Bad news: the last one is about grub, so the title should be correct, right?
  4. At this point, it might be the Gtk frontend behaving strangely. It s shipped in cdebconf-gtk-udeb, built from the cdebconf source. Let s look:
    • src/modules/frontend/gtk/ui.c has cdebconf_gtk_update_frontend_title(), which does what the function name suggests.
    • src/modules/frontend/gtk/di.c has cdebconf_gtk_di_run_dialog(), which calls the first function.
    • src/modules/frontend/gtk/progress.c has cdebconf_gtk_show_progress() (also calls the first function), which comes with the following comments:
      /** Show the progress widgets.
        *
        * This will actually add the widgets to the corresponding containers.
        * The main title saved when starting the PROGRESS operation will be restored
        * from the value saved when START was called.  This is needed when GO is
        * called during a PROGRESS operation.
      
    Now that s something! Even if the last entry is about grub, some previous value could be restored and could be the reason for the bad behaviour (that is easily confirmed by adding some debug_printf() calls on fe->title in cdebconf, thanks to an extra "debug.h" include). Maybe that s what needs fixing!
  5. Now it s time to take a step back. A d-i environment is nice, but testing things can easily become a burden. Taking a random wheezy or sid chroot, and installing/removing a tiny package shipping a manpage should be enough to run into the man-db trigger. Exporting DEBCONF_DEBUG=developer there too, let s either install or remove x11-apps (the trigger does the same job, and the debconf trace is the same), here s the output:
    Processing triggers for man-db ...
    debconf (developer): frontend started
    debconf (developer): frontend running, package name is man-db
    debconf (developer): starting /var/lib/dpkg/info/man-db.config configure /usr/share/man
    debconf (developer): <-- VERSION 2.0
    debconf (developer): --> 0 2.0
    debconf (developer): <-- INPUT medium man-db/install-setuid
    debconf (developer): --> 30 question skipped
    debconf (developer): <-- GO
    debconf (developer): --> 0 ok
    debconf (developer): starting /var/lib/dpkg/info/man-db.postinst triggered /usr/share/man
    debconf (developer): <-- VERSION 2.0
    debconf (developer): --> 0 2.0
    debconf (developer): <-- GET man-db/auto-update
    debconf (developer): --> 0 true
    
    Even if one didn t know about the cdebconf/debconf thing, the frontend started bit in the d-i syslog would have been the key: it s only found in debconf, in the frontend script. That one has several ways to determine the package being acted on, and it finally calls:
    debug developer => "frontend running, package name is $package";
    $frontend->default_title($package) if length $package;
    
    Tada, that s likely to be the issue.
  6. Tweaking that frontend script, it s easy to make sure that the code path for this use case is:
    elsif ($ARGV[0]=~m!^.*/(.*?)\.(?:postinst postrm prerm)$!)  
            $package=$1;
     
    
    Since $ARGV[1] is the action being passed to the maintainer script, one can intercept the triggered action, and avoid emitting a title update in this case; I ve therefore reassigned #679327 to the debconf package, and proposed a patch implementing that, which Joey Hess seems to find reasonable.
In my message to the bug report I wrote I actually tested it in a d-i environment, and made sure the issue had gone away. How I did that will likely be described in the next d-i hacking recipe.

20 December 2012

Wouter Verhelst: Installing Debian to the network with NBD

As I've blogged about before, wheezy will be the first release since I've started maintaining NBD for Debian to feature NBD support in the installer. That is, you'll be able to install wheezy to the network, simply by adding "modules=partman-nbd" to the kernel command line: obligatory screenshot While I'm happy and, in fact, a bit proud about that fact, there are still a few minor issues that make the whole process a bit less smooth than it could be: In my plan to take over the world of network storage, I plan to work on that last point during the jessy release cycle, at least for x86 architectures; I envision a system which would mount /boot on NFS, and write pxelinux.cfg files, where the name of this config file would be configurable (so it can be configured to only boot the system on which the file was written, or to boot any system on the local network). If done well, it will also fix the point about installing the bootloader to local hard disk when doing a network installation, since we'll have a "install netbootloader" or some such option.

Joachim Breitner: GHCi integration for GHC.HeapView

Given the very positive feedback for Dennis Felsing s tool ghc-vis, which visualizes the heap representation of a Haskell value, including all the gory details such as thunks, values retained by thunks indirections, sharing etc, I saw the need to provide this information also directly in GHCi, without having to load a graphics library or opening extra libraries. So I added the required features (traversing the heap and pretty-printing the results) to my ghc-heap-view package and, also following ghc-vis s lead, added a ghci file that, when loaded, provides you with a :printHeap command. Here you can see it in action: A plain value
Prelude> :script /home/jojo/.cabal/share/ghc-heap-view-0.4.0.0/ghci
Prelude> let x = [1..10]
Prelude> x
[1,2,3,4,5,6,7,8,9,10]
Prelude> :printHeap x
_bh [S# 1,S# 2,S# 3,S# 4,S# 5,S# 6,S# 7,S# 8,S# 9,S# 10]
Note that the tools shows us that the list is a list of S# constructors, and also that it is still hidden behind a blackhole. After running System.Mem.performGC, this disappears. A thunk
Prelude> let x = Just (1 + 1)
Prelude> :printHeap x
Just _bco
Prelude> x
Just 2
Prelude> System.Mem.performGC
Prelude> :printHeap x
Just (S# 2)
Here, we see how the calculation was deferred until forced by showing the value of x. The name _bco stands for a bytecode object as used by the interpreter. Getting useful information from them is a bit harder than for compiled thunks, so for more accurate results put the code in a Haskell source file, compile it and use the GHC.HeapView API to print the interesting parts. A partial application
Prelude> let a = "hi"
Prelude> let partial = (a ++)
Prelude> partial ""
"hi"
Prelude> System.Mem.performGC
Prelude> let x = (a, partial)
Prelude> :printHeap x
let x1 = "hi"
in (x1,_fun x1)
The information <emph>which</emph> function is called there (++ in this case) is lost at runtime, but we still see that the second element of the tuple is a partial application of <emph>some</emph> value to the first element. A cyclic structure
Prelude> let s = "ho"
Prelude> let x = cycle s
Prelude> length (take 100 (show x))
100
Prelude> System.Mem.performGC
Prelude> :printHeap x
let x0 = C# 'h' : C# 'o' : x0
in x0
Prelude> let y = map Data.Char.toUpper x
Prelude> length (take 100 (show y))
100
Prelude> :printHeap y
C# 'H' : C# 'O' : C# 'H' : C# 'O' : C# 'H' : C# 'O' : C# 'H' : C# 'O' : C# 'H' : C# 'O' : _bh (C# 'H' : C# 'O' : C# 'H' : C# 'O' : C# 'H' : C# 'O' : C# 'H' : C# 'O' : ... : ...)
The cyclic, tying-the-knot structure of cycle is very visible. But can also see how easily it is broken, in this case by mapping a function over the list. Mutual recursion
Prelude> let  x = 'H' : y ; y = 'o' : x  
Prelude> length (show (take 10 x, take 10 y))  seq  return ()
Prelude> System.Mem.performGC
Prelude> :printHeap (x,y)
let x1 = C# 'H' : x3
    x3 = C# 'o' : x1
in (x1,x3)
If you want to look at multiple variables at once, just pass a tuple to printHeap In the hope that this will be a useful tool for you, I uploaded version 0.4.0.0 of ghc-heap-view to hackage.

Francois Marier: Keeping GMail in a separate browser profile

I wanted to be able to use the GMail web interface on my work machine, but for privacy reasons, I prefer not to be logged into my Google Account on my main browser. Here's how I make use of a somewhat hidden Firefox feature to move GMail to a separate browser profile. Creating a separate profile The idea behing browser profiles is simple: each profile has separate history, settings, bookmarks, cookies, etc. To create a new one, simply start Firefox with this option:
firefox -ProfileManager
to display a dialog which allows you to create new profiles:
Once you've created a new "GMail" profile, you can start it up from the profile manager or directly from the command-line:
firefox -no-remote -P GMail
(The -no-remote option ensures that a new browser process is created for it.) To make this easier, I put the command above in a tiny gmail shell script that lives in my ~/bin/ directory. I can use it to start my "GMail browser" by simply typing gmail. Tuning privacy settings for 2-step authentication While I initially kept that browser profile in private browsing mode, this was forcing me to enter my 2-factor authentication credentials every time I started the browser. So to avoid having to use Google Authenticator (or its Firefox OS cousin) every day, I ended up switching to custom privacy settings and enabling all cookies:
It turns out however that there is a Firefox extension which can selectively delete unwanted cookies while keeping useful ones. Once that add-on is installed and the browser restarted, simply add accounts.google.com to the whitelist and set it to clear cookies when the browser is closed:
Then log into GMail and tick the "Trust this computer" checkbox at the 2-factor prompt:
With these settings, your browsing history will be cleared and you will be logged out of GMail every time you close your browser but will still be able to skip the 2-factor step on that device.

18 December 2012

Raphael Geissert: Nicer, but stricter

Lately I've been working on making the redirector nicer to the mirrors and to some potential users. More specifically, those behind a caching proxy.

The redirector is now nicer to traditional web proxies by redirecting to objects that are known not to change with a "moved permanently" code (HTTP status code 301.) This applies to files in the pool/ directory and ".pdiff" files, among others.
Previously, a traditional caching web proxy would sometimes end up with multiple copies of the same object, fetched from different mirrors; and the redirection would not be cached at all. With this change, this is no longer the case.

Using a caching proxy that is aware of the Debian repository design is still more likely to yield better results, however: If my memory serves correctly, apt-cacher has the ability of updating the Packages, Sources, and similar files with the ".pdiff"s on the server side. Apt-Cacher-NG apparently can use debdelta, and so on.
Check my blog post about one APT caching proxy not being efficient for some comments related to those tools.

Another recent change is that mirrors that can't be used by the redirector will no longer be monitored as often as the other mirrors. For instance, if a mirror doesn't generate a trace file (used for monitoring) then the redirector will gradually limit the rate at which the mirror is checked.
This rate-limiting mechanism applies to different kinds of errors, and should reduce the amount of wasted time and bandwidth while still allowing automatic-detection of mirrors that recover.


Projection of a rate-limited mirror over six weeks. The mirror would have to fail in every attempt for that to happen.
N.b. there's a bump in the scale.

The rate limiter applies an initial exception to allow temporary errors to not affect the use of the mirror by the redirector. After that exception, it is pretty much linear. However, that chart doesn't really reflect the effect of the rate limiter, so put in comparison with the normal checking behaviour:


Comparison of the two behaviours over an 8 weeks period using a logarithmic scale.
Nice chart colours by Libreoffice.

The code to detect mirrors that don't perform a two-stages sync that I talked about in a previous post has not yet been integrated as the current implementation would be too expensive on the mirrors to just add it as-is.

While tracking down problems exposed to users, I decided to take a stricter approach as to what mirrors are used by the redirector. Suffice to say that the remaining mirrors using the obsolete anonftpsync are going to be ignored entirely. ftpsync has been around for a few years now and it is the standard tool.
Whether you are mirroring Debian, Raspbian, Ubuntu, or any other Debian-like packages repository, ftpsync is the right tool to use.

Most of the issues I've been discovering, and sometimes working around, affect direct users of the mirrors and are not related to the http.debian.net redirector. When not detected beforehand they happen to be exposed by the redirector, but like I said, I plan to be stricter in order to increase the redirector's reliability. Once a strict and reliable foundation is built, more workarounds might see their way in to better use the available resources.

That's it for now. The road is long, the challenge is great, and being an observer in an uncontrolled environment makes it even more interesting.

Russ Allbery: Review: Liars and Outliers

Review: Liars and Outliers, by Bruce Schneier
Publisher: John Wiley
Copyright: 2012
ISBN: 1-118-14330-2
Format: Hardcover
Pages: 285
One of the perils of buying a book written by a blogger one reads regularly is that the book may be little more than a rehashing of their blog, with insufficient original material to warrant the time investment. Sometimes it's still nice to support them financially, but it may not make sense to read the book. I've been following Schneier's blog for years (as should anyone with an interest in security), including through the entire process of writing Liars and Outliers, and was a bit worried that might be the case here. Thankfully, I can reassure any other worried potential readers that is not the case. This is substantial new material establishing a firm framework for thinking about incentives and controls in any society or organization. Liars and Outliers talks about security mechanisms, but it's not, at its core, a book about security. Rather, it's a book about incentives, order, and how order is established. It's a comprehensive reductionist analysis of how societies create predictability and compliance to allow us to trust other people who we have never met before and will never meet again. It's a unique (at least in my experience) combination of anthropology, sociology, security analysis, and political science. Schneier cuts across fields in an idiosyncratic but illuminating way that reminded me of (on an entirely separate topic) Jane Jacobs. This is not a prescriptive book, nor is a collection of answers, solutions, or even deep analysis of particular problems. Rather, it's an attempt to construct a general framework for analyzing societal dilemmas: conflicts between individual desires and social good, how those conflicts are resolved, and how societies can weigh the scales and influence the statistical outcomes. The closest Schneier comes to telling the reader how to solve problems is a checklist, at the end of the book, for designing effective societal pressures. Its primary contribution is vocabulary and structure. It also passes one of my litmus tests for any book about human behavior: Schneier complicates, broadens, deepens, and expands understanding, and points out complex interactions and complex feedback in effects we're inclined to consider simple, rather than simplifying or eliminating human complexity. One point I found refreshing about this book is that Schneier is scrupulous in refusing to define either society or individuals as good or bad, to the point of carefully defining terminology used in all of the social dilemmas. Following societal rules is called compliance; not following those rules is called defection. In some cases, defection is morally correct (Schneier's most frequent example is in the civil rights struggles of the 1960s in the United States). In all cases, social pressures are tools, which can be used to encourage compliance with moral or immoral systems, and which are deployed by totalitarian dictatorships and utopian communes alike. Schneier explicitly puts out of scope for this book the questions of how societal goals should be determined, how they change, and whether any given societal rule or interest is moral or immoral. He focuses, rather, only on the mechanisms, with a primary goal of informing and deepening debates over how best to encourage behavior that societies want to encourage and discourage behavior societies want to discourage. He also emphasizes discourage, as opposed to eliminate. Early on, Schneier shows some of the results of game theory, as well as basic common sense, that indicate that no healthy society can totally eliminate defection. Not only would that stifle valuable and important reform, such as changes in civil rights, but the degree of pressure required is immense. Defectors are natural and will always exist, and are sometimes valuable and necessary. Rather, the goal of a society is to reduce defectors to a level where most people can ignore their existence most of the time, a state that leads to the level of risk and trust required to have a functioning and healthy society. The word society, similarly, is intentionally broad, and can refer to just about any collection of people, from a circle of friends to a corporation, institution, or country. However, as Schneier points out early in the book, small societies rarely need much in the way of formal pressure and appear almost magically self-governing. That's the first property that he disassembles, resulting in a general classification of societal pressures into four categories: moral, reputational, institutional, and security systems. The last is an odd category that's partly orthagonal to the other three. Moral pressure is internalized conscience: the normal tendency of nearly everyone to follow their own moral code, a code that's at least partly constructed and certainly heavily influenced by the surrounding society. Reputational pressure is, in a sense, externalized morality: it's the informal reactions of others around one to one's past actions. Included in reputational pressure is shunning of every kind, from cutting off a friendship to boycotts against corporations, but it also includes confrontation from another member of one's society, the more subtle effects of our individual desires to be liked and respected, and all the various aspects of "face", honor, and respect within a community. Small communities are frequently self-governing, in Schneier's model, because they're small enough that moral and reputational pressures are sufficient and no other pressures are required. We're so used to applying moral and reputational pressure to other humans almost unconsciously that we sometimes don't even notice its existence, leading to that "magical" self-governing property. But Schneier puts pressures in a sequence: pressures that work extremely well with small groups often don't scale. Moral pressure works best with small groups and reputational pressure with somewhat larger groups, but when societies scale beyond the limits of reputational pressure when, for example, one frequently interacts with people whose reputations are unknown to you and whose subsequent opinion of you will not be relevant institutional pressure is required to force compliance. Institutional pressure is the sort of pressure that we all tend to think of first when we look for ways to enforce rules: laws, policies, contracts, and other codes of behavior that carry with them formal punishments and some enforcement mechanism. But even in societies so large that institutional pressures are frequently required, such as whole countries, moral and reputational pressures still exist and are extremely important. One of Schneier's most interesting points is his analysis of how institutional pressures can paradoxically undermine reputational and moral pressures, resulting in more defection than if the institutional pressure hadn't existed. This is just the basic framework of Schneier's analysis, hopefully giving you a feel for the structure of the book. He goes much deeper into the complicated interactions between the various levels of pressure, and then dives into an extensive look at competing societal dilemmas: cases where there is more than one society in play simultaneously, possibly demanding contradictory actions. Liars and Outliers also includes a wonderful analysis of organizational entities within the same framework, including their much-different reactions to moral, reputational, and institutional pressures. One of the most cogent analyses of the difficulties of regulating both corporations and governmental institutions falls out of that analysis, once one looks at them in light of Schneier's basic framework. Pressures quickly become complex and multi-layered, and human reactions to pressures are frequently counter-intuitive. Schneier draws extensively on game theory to show that some counter-intuitive responses are actually emergent properties of logical analysis of the situation, but that others are more uniquely human and have little or nothing to do with a mathematical cost-benefit analysis. I haven't even mentioned his discussion of security systems, and how they can extend moral, reputational, and institutional pressures, as well as add a new type of pressure (making a defecting action impossible) that scales even better than institutional pressures. Liars and Outliers has all of the supporting infrastructure you would expect in a scholarly book: notes, extensive references, and a good index. I suspect it will end up being used as at least additional reading in college classes. The notes are, unfortunately, end notes, making the full context of the book much harder to read than was necessary, but at least Schneier does separate the notes from the references so that one doesn't chase notes for further explanation and find a simple citation. As with any book like this, one always wishes it could end in a simple prescription to fix everything, but of course it doesn't. But that's also a measure of a good scholarly work. Human and organizational motivations are complex and tricky, and any framework for analyzing them needs to be able to represent that complexity. Schneier here has constructed a very powerful one, one that I started using in discussions before I'd even finished the book. Perhaps the most valuable contribution of Liars and Outliers to public discussion is clear terminology and categories, which can be of great help in finding the core components of a problem. Liars and Outliers can be slow going, particularly early on when Schneier is still defining terms and setting up the background of his analysis. One can get a bit tired of the analysis matrices of societal dilemmas. But stick with it through the groundwork, since the analyses of competing societal dilemmas and of the impact of societal pressures on organizations are exceptional. Highly recommended, particularly for anyone who is designing or implementing societal pressures: managers, political activists, or anyone in a security-related field. Rating: 8 out of 10

17 December 2012

Dominique Dumont: Be wary of optimised for devices

Hello This story began with linphone having a weird behavior: every now and then, the mouse pointer would become stuck: the pointer moves, but clikcking on it always activate the same widget. Since this always happened in the middle of phone conference done for work, this was infuriating. Long story short, I finally found that my plantronics USB headset was responsible for this weird behavior. This headset is connected to the USB bus and is seen by the computer as a USB sound card. The bug was triggered by pressing the mute button provided by the headset. Suspicious of this usb device, I used lsusb to find the features provided by the heaset: $ lsusb -v -s 1:3 2>&1 grep InterfaceClass
bInterfaceClass 1 Audio
bInterfaceClass 1 Audio
bInterfaceClass 1 Audio
bInterfaceClass 1 Audio
bInterfaceClass 1 Audio
bInterfaceClass 3 Human Interface Device
Audio devices were expected, but why a HID device ? This device is a headset, not a keyboard or a mouse Testing further, I also saw some number poping up on my screen whenever I pressed the the mute button. The only way to get the mouse back was to unplug the headset. A quick google search gave a solution to setup X11 to ignore the headset. Problem solved. But this did not answer the question regarding the HID device. Then I got a hint in the form of a sticker glued on the headset USB plug: Optimized for Microsoft Lync . This page gave the answer: an optimised for Lync device provides mute/unmute across PC and device . I can only guess that when the mute button is pressed, some data is sent from the HID interface. Unfortunately, the window manager does not like to be on the receiving end of this data. The moral of this story is: optimized for something actually means not standard .
ok. There s a bug somewhere. The comments of this blog have convinced me that I went too far with the moral of this story. It s now overstriked instead of plainly remvoed so the comments still make sense. Thanks everybody for the constructive comments.
All the best
Tagged: wtf

14 December 2012

Roland Mas: Back to space

A long time ago, I spent many many hours playing Frontier (Elite 2) on the family Atari ST. It was nice-looking (oh, those sunrises on distant planets!), playable, varied, had ample room for progression, very different possible roles, a huge universe of which nobody could ever hope of exploring more than a tiny fraction, and it was basically the first game I encountered with no set goals. You are a spaceship pilot in a galaxy colonised left and right, and it's up to you to decide what you want the game to be. Pirate, headhunter, mercenary, trader, taxi, explorer, there's no end goal to achieve except what you fix for yourself. That was, for me as a youngster, a mind-opening experience, which I never felt as strongly since (with the possible exception of the Creatures game, but I barely tried it). Then came modernity, the PC, and the First Encounters sequel (Elite 3). It added moderate complexity to the gameplay, some sort of political evolution among the factions and a kind of plot so the galaxy was no longer quite so static, but the end results were still a bit disappointing. It felt like Frontier with a few not-exactly-overwhelming textures stuck onto the ships, a loose plot arc, and a third faction, the Alliance of Independent Systems, beyond the Federation and the Empire. Not that exciting, although the name of the Alliance's capital star system stayed in my mind. So, like many others, I came to wait for Elite 4. That wait started in 1995, more than seventeen years ago. During that time, Duke Nukem Forever was announced, promised, developed, postponed, abandoned, found again, developed again, and it was even actually released in 2011, while Elite 4 went into the same kind of hell only without the actual release (yet). At some point I started trying various substitutes to help pass time. There was Parsec which was a nice-looking space combat simulator, multiplayer and all, but never felt like finished. Development slowed down to a thin trickle, and it seems like it only recently started again as Open Parsec. Hopefully it won't wither off again, but since it's only focused on combat, it's not exactly the same spirit as Elite. There's Oolite, but it looks and feels very much like a lightly-modernised Frontier. Better textures, sure, but the UI still looks like it got taken from the times where games mentioned they required an EGA/VGA video card. There's Vega Strike. It's very promising too, and I did try it at some point, but it was crashing too often for my tastes. Development seems to have slowed down, but not stopped; maybe it's time I gave it another go. I just recently found out about Pioneer, and I admit I'm excited. Really excited. It feels like like Frontier felt at the time. It's not finished yet (the on-planet cities in particular look a bit weird), but from the alpha I tried and the videos on the website it's really impressive. I'm going to keep an eye on that, knowing I could spend quite some time in there even before it's quite finished, if only to fly a spaceship in the canyons of Europa with Jupiter rising in the distance. I'll let you know if I find a black monolith. But the main event triggering this blog post is that apparently Elite 4 is not dead (yet). Much better, it's under heavy development, under the Elite Dangerous name, and the current state seems to be rather good already. And Frontier Developments, the company behind it, opened a Kickstarter campaign to fund the project. The screenshots and the videos look impressive, and the interviews of the lead developer imply that the gameplay will be awesome. The galaxy will apparently be really dynamic, and the player's actions could really influence its geopolitical (galactopolitical?) evolution, the profitability of trade routes, the prevalence and repartition of space pirates in various sectors, and so on. Combine with a multi-player mode, and this could become the greatest game ever made (for my personal taste at least). So I'm very much hoping the funding campaign reaches its target. The release date is set to March 2014. It's going to be a long wait, fraught with incertitudes. But then again, Duke Nukem Forever was eventually released, wasn't it? [No, I never played the original Elite. I'm not that old.]

12 December 2012

Vincent Sanders: What man's mind can create, man's character can control.

I have a project that required me to programmatically control power to several devices. I have done this before using a Velleman vm8090 board which is relatively easy to control. However they are relatively expensive.

I turned to ebay and found a similar module at a substantially reduced cost. Upon receipt however I discovered that instead of being a simple serial USB interface it presented USB HID and the Debian system I was running it on has loaded the hiddev driver for me but it did not implement any of the standard HID Usage Pages leaving me with no way to control the device.
I did the obligatory
sudo lsusb -d 12bf:ff03 -vvv

Bus 003 Device 019: ID 12bf:ff03  
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 1.10
bDeviceClass 0 (Defined at Interface level)
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 8
idVendor 0x12bf
idProduct 0xff03
bcdDevice 1.00
iManufacturer 1 Matrix Multimedia Ltd.
iProduct 2 Flowcode USB HID
iSerial 0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 41
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0x80
(Bus Powered)
MaxPower 50mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 2
bInterfaceClass 3 Human Interface Device
bInterfaceSubClass 0 No Subclass
bInterfaceProtocol 0 None
iInterface 0
HID Device Descriptor:
bLength 9
bDescriptorType 33
bcdHID 1.10
bCountryCode 0 Not supported
bNumDescriptors 1
bDescriptorType 34 Report
wDescriptorLength 54
Report Descriptors:
** UNAVAILABLE **
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0008 1x 8 bytes
bInterval 5
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x01 EP 1 OUT
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0008 1x 8 bytes
bInterval 5
Device Status: 0x0980
(Bus Powered)
This simply showed me what I already knew and surprised me that lsusb did not dump HID report descriptor items. Some searching revealed that teh device had to be unbound so lussb could access the descriptor.

Thus a simple
echo -n 3-1.1.4:1.0   sudo dd of=/sys/bus/usb/drivers/usbhid/unbind
resulted in lussb dumping the descriptor items:

Item(Global): Usage Page, data= [ 0xa0 0xff ] 65440
(null)
Item(Local ): Usage, data= [ 0x01 ] 1
(null)
Item(Main ): Collection, data= [ 0x01 ] 1
Application
Item(Local ): Usage, data= [ 0x02 ] 2
(null)
Item(Main ): Collection, data= [ 0x00 ] 0
Physical
Item(Global): Usage Page, data= [ 0xa1 0xff ] 65441
(null)
Item(Local ): Usage, data= [ 0x03 ] 3
(null)
Item(Local ): Usage, data= [ 0x04 ] 4
(null)
Item(Global): Logical Minimum, data= [ 0x00 ] 0
Item(Global): Logical Maximum, data= [ 0xff 0x00 ] 255
Item(Global): Physical Minimum, data= [ 0x00 ] 0
Item(Global): Physical Maximum, data= [ 0xff ] 255
Item(Global): Report Size, data= [ 0x08 ] 8
Item(Global): Report Count, data= [ 0x08 ] 8
Item(Main ): Input, data= [ 0x02 ] 2
Data Variable Absolute No_Wrap Linear
Preferred_State No_Null_Position Non_Volatile Bitfield
Item(Local ): Usage, data= [ 0x05 ] 5
(null)
Item(Local ): Usage, data= [ 0x06 ] 6
(null)
Item(Global): Logical Minimum, data= [ 0x00 ] 0
Item(Global): Logical Maximum, data= [ 0xff 0x00 ] 255
Item(Global): Physical Minimum, data= [ 0x00 ] 0
Item(Global): Physical Maximum, data= [ 0xff ] 255
Item(Global): Report Size, data= [ 0x08 ] 8
Item(Global): Report Count, data= [ 0x08 ] 8
Item(Main ): Output, data= [ 0x02 ] 2
Data Variable Absolute No_Wrap Linear
Preferred_State No_Null_Position Non_Volatile Bitfield
Item(Main ): End Collection, data=none
Item(Main ): End Collection, data=none

By consulting the device class definitions document I determined the device was using the "Vendor defined" Usage page (0xff00 to 0xffff) so I would definitely have to write a program to control the device.

Linux provides a really easy interface to deal with HID devices called hiddev (gosh, such adventurous naming) which I already had to unbind to get my descriptors decoded so I am fairly sure it works ;-)

The kernel documentation and header for hiddev provide the absolute basic mechanics of the interface but no example code or guidance. The obligatory web search turned up very little and even that had to be retrieved from the internet archive. So It seems I would be forced to work it through myself.

It seems the hiddev interface is orientated around HID devices generating reports which the program is expected to read. Numerous ioctl() are provided so the program can obtain the descriptor information necessary to control and process the received reports.

However in this case we need to be able to send reports to the device, all the descriptor information revealed was that there were eight (Item Report Count = 8) values with eight bits each (Item Report Size = 8) with logical and physical values representing the whole range of the octets.

Fortunately the seller provided a website with some control programs and even source. After some time rummaging through the Visual Basic program I finally found (in FrmMain.vb:2989) that the eight bytes were largely unused and the first was simply a bitmask of the eight relays coil status, set for energised clear for off. With bit 0 controlling relay labelled 1 through to bit 7 for relay 8.

To send a report to a HID device the hiddev interface uses the HIDIOCSREPORT ioctl where the report data is first set using HIDIOCSUSAGE .

The HIDIOCSUSAGE ioctl is passed a hiddev_usage_ref structure which must be initialised with information about the report descriptor identifier (constructed from the Usage Page and Usage as set by the items in the descriptor), the index of the item (named usage) we wish to set in the report (in this case the first which is 0) and the value we actually want to set.

After a great deal of debugging the final program is very short indeed but does the job, my main problem now is that if I switch too many (more than one) relays at once the whole device resets. The scope says the supply rails are behaving very badly when this happens, looks like I need to add a load of capacitance to the power well to stabilise it during the switching events.

Oh and add in the fact Relay 1 LED doest work unless you push on it and I do wonder about the wisdom of the economy in this case. Though yet again Linux makes the software side easy.

Next.

Previous.