Review: Science Fiction: The 101 Best Novels: 1985 2010, by Damien Broderick & Paul Di Filippo

Publisher:	Nonstop
Copyright:	2012
ISBN:	1-933065-39-7
Format:	Trade paperback
Pages:	288

I like book reviews and lists of best novels, as a follower of my reviews probably noticed, so I couldn't resist when this collection made my radar. A follow-up to the earlier Science Fiction: The 101 Best Novels: 1949 1985 by David Pringle (which I have not read), it is a collection of short (two to three pages, generally) reviews of 101 recent SF novels. The date spread is fairly balanced: at least two novels from each year under consideration are featured, and no year gets more than seven or eight. The authors also clearly tried to cover the range of what falls under the science fiction genre, from alternate history through space opera and including novels normally marketed as mainstream, so the result should contain something to everyone's taste. With those characteristics, you may suspect that the "best" part of the title is a bit questionable, and you would be right. "101 of the better novels" would be a more accurate description. While most Hugo and Nebula winners are included here, the section is at times eclectic. But it's eclectic in the spirit of broad inclusiveness: Jumper, Temeraire, or The Hunger Games would normally not be included on this sort of list because they're too popular or "light," but they're here alongside more obscure books (at least for SF readers) like Galatea 2.2, Distance Haze, or My Dirty Little Book of Stolen Time. I doubt anyone will seriously argue that this selection should have replaced the Hugo or Nebula short lists, but one doesn't read review collections like this only to hear about books one already knows about. Those books one has either already read or already chosen not to read. The wide-ranging selection makes it likely that something here will be new to most readers. The authors, Damien Broderick and Paul Di Filippo, are known reviewers in the SF world, and I've read reviews from both of them before. I bought this book largely on the strength of Broderick's name, since I've usually enjoyed his contributions to The New York Review of Science Fiction. Paul Di Filippo was more of a gamble; he's one of the regular reviewers for Asimov's Science Fiction and not one of my favorites. But what I usually disliked about his columns was their focus on obscure small-press titles, graphic novels, and slipstream, so I was hoping that an SF review collection aiming towards the genre mainstream would be more to my taste. The result is mixed. There are things about this selection, and about the reviews, that I enjoyed, and there are other things I found quite annoying. First, the selection. I could (and will in a moment) talk about good and bad selections, but I also have a good statistical metric for at least the alignment of the authors' taste with mine. Of the 107 novels reviewed here (in several case, duologies and trilogies are given single entries), I've read 39, or a little over a third. (Note that I've read every Nebula or Hugo winner in that time period and most of the Hugo nominees, so that will give you a good feel for how broad-ranging this selection is, and how far afield of the normal award slates it goes.) My average rating for those 39 books was 7.49 (including four perfect 10s). By comparison, my average rating for Hugo winners is 6.68 and for Nebula winners is 7.10. There was one 4 (The White Queen) and one 5 (Red Mars), and in both cases I can see why they're here. Of the rest of the books I read I rated them all at least at 6 out of 10. At least among the books I've read, this seems to be a solid selection. Sometimes the details of those selections are odd, though. For example, the authors make an effort to limit the number of selections for each author, a wise choice since they're clearly going by diversity. But if one is operating within that limitation, choosing Ammonite over Slow River for Nicola Griffith, or particularly Ventus over Lady of Mazes or even Permanence for Karl Schroeder, is baffling. There were several similar places where I thought the selection for an author was obscure, minor, or just missed obvious alternatives. Perhaps this was to fill in one of the other breadth criteria, such as balancing number of novels per year or attempting to cover each subgenre. Also, if one is going to divide science fiction and fantasy and try to cover only the science fiction (a division that I think is quite difficult, which is why I don't do it, but it does have the merit of narrowing the field), including The Falling Woman is quite strange. It's a solid book, to be sure, and a Nebula winner. It is also quite straightforward contemporary fantasy involving ghosts and Mayan mythology, without a hint of science-fictional content. Making the protagonists archeologists and scientists doesn't make the book science fiction. The authors try to defend this (unpersuasively to me), but it wasn't the only instance here where I thought their line between science fiction and fantasy was a bit off. That said, there are a lot of great selections here, including books that I love but that aren't frequently picked for this sort of list (The Fortunate Fall, The Time Traveler's Wife, or China Mountain Zhang, for example). It's great to see underappreciated authors like Linda Nagata, Joan Slonczewski, and Karl Schroeder featured. But, of course, one doesn't buy this sort of book just for the list, if for no other reason than that lists aren't copyrightable and one can easily find the complete list of reviewed novels on the Internet (just one example that turned up in a search). Rather, one reads this sort of book for the reviews. And that's where this book moves onto more questionable ground. First, while I realize that everyone has different thresholds for what they consider spoilers and most professional reviewers are more cavalier about them than I am, Broderick and Di Filippo cross any line that I consider reasonable. Most of the reviews are okay, if skirting the limits, but in several places they give away key reveals of books or discuss plot twists right up to, or even including, the ending. The combined review of The Sparrow and Children of God is particularly egregious, containing unambiguous, book-destroying spoilers for The Sparrow. Giving away the ending of Ammonite is only slightly less bad. And those are just two examples I remember. This is not okay. The whole point of this sort of collection is to expose the reader to books they've not yet read but may want to. Proceeding to spoil the book for them in the course of the review is perverse. This alone would make me hesitant to recommend this collection. Second, quite a few of the reviews in this book are, for lack of a better term, emotionally overreaching. Here's an excerpt of a review picked at random (As She Climbed Across the Table by Jonathan Lethem) that will hopefully illustrate:

Lethem's beautifully balanced, metaphorically rich prose propels this blackly jolly fable to a surprising yet satisfying conclusion. By book's end, a sense that the author had accomplished his takeoff taxiing and was now fully in flight for more cosmopolitan cities pervades the pages.

What's "beautifully balanced prose"? Could you recognize it? Does that phrase communicate anything to you other than that the authors liked the book? The whole review collection is written with adjectives and metaphors like this, and after a while it all seems a bit much. It felt like the authors were straining for ways to describe how important or significant the books are and, in the process, lost sight of the basic goal of conveying information about the book. It feels overwrought rather than informative. Even if one is reviewing the books that one considers the pinnacle of achievement in science fiction, a conversational tone with concrete examples and specifics communicates more than impressive but slippery terms like "metaphorically rich." Lest I sound entirely negative, one thing that I did appreciate is that the reviews go to some effort (particularly for their short length) to put the work in the broader context of the field and within the author's oeuvre. Often there's some discussion of previous and subsequent work or related books, and the reviews that feature books from larger series provide good explanations for why those particular books were singled out. Sometimes the number of dangling references was frustrating; authors of these sorts of collections need to remember that most readers will not be as widely read, and reviewing books largely by comparison to other books runs the risk of missing the reader's knowledge entirely. But the reviews convey a real sense of SF as a broad conversation and provide a sense of the breadth and variety of themes and subgenres available. This is one of the fun explorations that this sort of catalog lets the authors and reader do together. Another, more minor, touch that I appreciated was the cover art. Each review leads with an image of the reviewed book's cover (alas, only in black and white for obvious printing reasons). But rather than taking the obvious approach of using the covers of the first releases, or the covers from a particular country, they're chosen from all of the world-wide editions in all their delightful variety. Typical artistic styles for book covers vary drastically between countries, and getting to see a sample of artwork from different markets is a treat. I want to recommend this book. It casts a much broader net than most collections of its kind and provides some needed attention to smaller corners of the genre. I was impressed by the book list before I bought it, and (with the inevitable quibbles) am even more impressed now that I've read it. Broderick and Di Filippo go out of their way to broaden the reader's horizon and open up new avenues for reading, which is one of the best things a review collection can do. But when reviewers don't avoid spoilers, I just can't recommend their work. For me, this is a cardinal sin. Combine that with a writing style that was occasionally overblown and overwritten and the merits don't quite overcome the flaws. I'm glad I read it; it got me excited about reading many books I've already purchased but not gotten to, and I got from it another slew of books to add to my to-purchase list. But I had to read it uncomfortably and lightly, constantly prepared to jump past a review that was too revealing. Rating: 6 out of 10

One of the nicest developments in the cloud arena is the increasing adoption of standards, this of course will impact on maturity and market confidence on such technologies. Amazon, as one of the pioneers, made a good choice on their offering design by making their API implementation public. Now, vendors such as Eucalyptus private/hybrid cloud offering and many other providers can leverage and build upon the specs to offer compatible services without the hassle for their customers to learn a new technology/tool. I ve bare-metal servers siting on Connectria s data-center. A couple of months ago I ve learned about their new cloud storage offering. Since I m working a lot on cloud lately I checked the service. It s was nice to learn they are not re-inventing the wheel but instead implementing Amazon s Simple Storage Service (S3) de-facto standard for cloud storage. Currently there are many S3-compatible tools available both FLOSS and freeware/closed source (Connectria provides a nice list of them). I ve been using s3tools, which is already available in the Debian archive, to interact with S3-compatible services. Usage is pretty straightforward. For my use case I intend to store copies of certain files on my provider s S3 service. I ll need to create buckets to be able to store files. For the ones not very familiar with S3 terminology buckets can be seen as containers or folders (in the desktop paradigm). First thing to do is configure your keys and credentials for accessing S3 from your provider. I do recommend you to use the --configure option to create the $HOME/.s3cfg file because it will put all the available options for a standard S3 service, leaving you the work of just tweaking them based on your needs. You can go and create the file all by yourself if you prefer, of course.

$ sudo aptitude install s3cmd
$ s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3
Access Key:
...

You ll be required to enter the access key and the secret key. You ll be asked for a encryption password (use only if you plan to use this feature). Finally, the software will test the configuration against Amazon s service. Since this is not our case it will fail. Tell the configuration instead to not retry configuration and say Y to Save configuration. Now, edit the $HOME/.s3cfg file and set the address for your private/third-party S3 provider. This is done here:
host_base = s3.amazonaws.com
host_bucket = %(bucket)s.s3.amazonaws.com
Change s3.amazonaws.com to your provider s address and the host_bucket configuration also. For my case I had to use:
host_base = rs1.connectria.com
host_bucket = %(bucket)s.rs1.connectria.com
Now, save the file and test the service by listing the available buckets (of course there is none yet).

$ s3cmd ls

If you don t get an error then the tool is properly configured. Now you can create buckets, put files, list, etc.

$ s3cmd mb s3://testbucket
Bucket 's3://testbucket/' created
$ s3cmd put testfile.txt s3://testbucket
testfile.txt -> s3://testbucket/testfile.txt [1 of 1]
8331 of 8331 100% in 1s 7.48 kB/s done
$ s3cmd ls s3://testbucket
2012-12-26 22:09 8331 s3://testbucket/testfile.txt

Doing my regular news scan, I stumbled across this: Hacker reveals in Rio how he rigged an election (in Portuguese; you can try the Google-translated version). Why am I reposting this? Because, even after the reported studies by Diego Aranha and the information disclosure exploited by Sergio Freitas, Brazil is still portrayed as the biggest example on how electronic voting can be 100% secure and tamper-proof. Well, in this case, Rangel (his full name ahs not yet been disclosed), a 19 year old hacker, not only demonstrated how elections could be rigged, but admitted on doing so together with a small group, and even pointed at who was benefitted from this. Rangel's attack was done during the transmission phase After ~50% of the electoral results had been sent over the Oi network. And yes, the provider will most likely close the hole that was pointed at, but this basically shows (again!) that no system can be 100% tamperproof, and that the more electronic devices are trusted for fundamental democratic processes, the more we as a society will be open to such attacks. The security-minded among us will not doubt even for a second that, as this attack was crafted, new attacks will continue to be developed. And while up to some years ago the attack surface was quite smaller (i.e. booths didn't have a communications phase, just stored the votes, and communication was done by personal means), earlier booths have been breached as well. And so will future booths be breached. So, the news of this attack are indeed very relevant for the field. The presentation I am quoting was held around two weeks ago And December will surely dillute attention from this topic. Anyway, I will look for further details on the mechanism that was used, as well as to the process that follows in the TSE (Supreme Electoral Tribunal). I hope we have news to talk about soon!

git-pbuilder is a shell script suitable for use as the builder command in git-buildpackage that simplifies using pbuilder, cowbuilder, or qemubuilder to build packages in a chroot. It's now part of the git-buildpackage Debian package, but I maintain it somewhat separately. This version supports a new environment variable, GIT_PBUILDER_OUTPUT_DIR, that can be set to override the default output location of ".." (the parent directory). It also changes the script to be a bash script and uses arrays to handle the options passed to the builder command so that shell metacharacters in the GIT_PBUILDER_OPTIONS environment variable will hopefully be handled correctly. As part of that change, it also tries to escape any options that will be passed to --debbuildopts to hopefully protect them from the additional layer of shell parsing that they'll be subject to. Thanks to Guido G nther and paul cannon for the patches. I hate having to make the script bash-specific, but trying to get the shell expansion right is otherwise a huge headache. That probably means that it's outlived the usefulness of the original implementation language (which was shell only because this started life as a quick hack). I should probably take the time to rewrite it in either Perl (which I know best) or Python (which is the language the rest of git-buildpackage is written in). But that's for another day. (Here's an excellent case in point for why one shouldn't embed language-specific extensions in the name of an executable that's part of a user interface or configuration interface.) You can get the latest version from my scripts page.

Review: A Paradise Built in Hell, by Rebecca Solnit

Publisher:	Penguin
Copyright:	2009
Printing:	2010
ISBN:	0-14-311807-2
Format:	Trade paperback
Pages:	319

A Paradise Built in Hell is a book with an agenda. Solnit's goal is to convince the reader that nearly everything we're shown in movies and popular culture about human behavior during disasters is wrong. Not only is destructive mass panic unusual verging on nonexistent, spontaneous cooperation and acts of startling courage and resourcefulness are commonplace. The heroes that we hear about afterwards are more the norm than the exception. Disasters are far more likely to bring out the best in people than the worst. They break down barriers and form spontaneous human communities that survivors remember for a lifetime. Rather than natural disasters leading inevitably to an exacerbating human crisis in the absence of strong authority, Solnit argues that they are an amazing human opportunity, and that the most negative and destructive human behavior during crises is not from untrained victims but from the panic of the elites who are supposedly responsible for protecting the public. I think her analysis and presentation is both deeply appealing and deeply flawed. The flaws involve extrapolation and generalization beyond the applicability of her research, and I'll say more about that in a moment. But the appeal is still surprising and satisfying, and I think there's a lot of truth in this book despite its one-sided presentation. I like to tell people, half-jokingly, that I'm an anarchosyndicalist. It's my political alliegance of the heart: the political structure that I know would never work, but that matches the world I want to live in. (I suspect much of the belief in libertarianism is of this form.) Now I have a book to point people at when they ask what the appeal of anarchosyndicalism is. The sense of spontaneous community, of bonding, of reaching out to other people to do something necessary, together, that Solnit describes here in the aftermath of a disaster matches that vision of a world without imposed economic authority, where people spontaneously collaborate to solve problems. A Paradise Built in Hell is structured by disaster, providing a tour of major disasters and their aftermath interspersed with extended musings on what the human reactions mean and how disaster communities functioned. The major disasters discussed are the 1906 San Francisco earthquake, the Halifax explosion, the 1985 Mexico City earthquake, 9/11, and Hurricane Katrina. Solnit moves through them in chronological order, closing the book with Katrina, which had (at least for me) the most infuriating and uplifting mixes of community bonding and elite panic. But both of those factors are present from the beginning in the 1906 earthquake: spontaneous human organization that (from multiple first-hand accounts) transformed people's lives mixed with a military reaction and elite panic that possibly did more damage to the city than the earthquake did. Elite panic is the dark side of Solnit's celebration of human altruism and spontaneous community organization. The primary picture she paints is of ordinary people responding with great courage and creativity and finding, in their reaction, a transformative experience and a sense of purpose that often becomes the most powerful and positive experience of their life. But this is at odds with a deeply-entrenched elite belief in mass panic, in the necessity of keeping information from the public to suppress their reaction, and in the need to re-establish "order." And with that, Solnit is absolutely scathing, and with some justification. Along with detailed accounts of these major disasters, Solnit also gives the reader a tour of the research literature on disasters, research that has shown that the panic and chaos that is at the center of practically every disaster movie ever made is essentially a myth. Mass panic in disasters is not only rare, it's close to non-existent. As you might expect, the handling of Katrina by both the government and by neighboring white communities is the subject of the harshest attacks. The material on Katrina presented here is enough to make anyone want to throw out the current playbook on how we react to disasters (and lest one think Solnit is unfair, her research matches very closely with other research and journalism on Katrina that I've seen from a wide variety of sources). But Solnit shows that this is a pattern: the combination of our media-driven belief in how people will react to disasters combined with the elite need to re-establish control plays out in many major disasters, and almost always negatively. Even in 9/11, the official response in some cases hampered and undermined an already-effective unofficial response. This is very interesting and has concrete implications for public policy around disasters. But where Solnit moves onto thinner anarchosyndicalist ice is the story that she tells about disasters as dramatic upheavals of the established social order that could point to a revolution in how we interact with each other. Solnit is deeply inspired by human altruism during disasters, and her enthusiasm is somewhat contageous, but it was useful to read this book shortly after Bruce Schneier's Liars and Outliers. I think Solnit, in her understandable eagerness to find a way to extend disaster behavior to general life, does not understand the relationships between what Schneier terms moral, reputational, and institutional pressures. And that's the core of why we need to both enable spontaneous human response in disasters and have organized first responders and supporting infrastructure, and why disaster communities will break down when extended beyond the disaster. When I first mentioned A Paradise Built in Hell to my mother, her first reaction was dubiousness due to her memory of the 1977 New York City blackout. There was a disaster that resulted in widespread looting and destruction, rather than the positive reaction Solnit describes. Solnit does have a cogent argument for why looting is poorly analyzed (we don't, for example, distinguish between breaking into stores to get desperately needed supplies and doing so for personal gain) and should not be a priority in a life-threatening disaster. But it's not completely convincing; in some disasters, particularly ones like the 1977 power outage that were not particularly life-threatening, the looting has been the most destructive part of the disaster. Schneier provides what I think is the missing piece, both to this and to why the wonderful behavior during disasters doesn't last into everyday life. During a life-threatening disaster, moral pressures are massively increased. Nearly all of us have self-images and moral beliefs that provide very strong incentives to help others and collaborate in the face of life-threatening emergencies, incentives that override other motives and lead to the sort of behavior that Solnit celebrates. But moral pressures have inherent limitations. That moral reaction applies most strongly to small groups (such as bands of survivors), and will start to break down as larger society re-establishes itself. And disasters, like a power outage, that are not immediately life-threatening will not provoke the same intensification of moral pressures: we don't have strong moral beliefs and internal stories about staying quietly at home during power outages the way that we do for saving people from burning buildings or pulling them out of rubble. Once larger communities have reformed and that short-term emergency reaction has dissipated, the need for institutional pressures (police, law, and formal authority structures) will return. I wish Solnit had been able to read Schneier's book before writing hers. I think his analysis structure provides a healthy dose of realism and perspective to her desire for a self-organizing communal world, and might have blunted her enthusiastic but somewhat unrealistic hopes for finding some key to a revolution in human organization in the heart of disasters. But if A Paradise Built in Hell is unlikely to lead to a radical change in how we organize societies, it should at least provoke a radical rethinking of some of our unnecessary pessimism about our fellow humans. The evidence is substantial and compelling: crisis brings out the best in people, not the worst, and the ordinary people who happen to be around are capable of becoming some of the most amazing disaster response teams that we can imagine. This has numerous mundane implications for social policy. We should consider simple training for the general public, for example, to strengthen that reaction, should allow for it in legal structures and training of first responders (around looting, for instance), and should do everything we can to kill the destructive myth that people cannot be trusted in disasters or that mass panic will be as much of a threat as the disaster. But the best part of this book is that it provides a concrete, well-documented, and well-defended reason to let go of some of our cynicism and to extend a bit of trust to our fellow humans. There sadly aren't earth-shaking implications for everyday society, but there is a wealth of evidence that shows that we're better people in the crunch than we believe we are, or that we expect of others. And the more that we can embrace the dissolution of boundaries and social hierarchies in the middle of disasters, the more we are likely to be surprised by deep human connection and a sense of shared, focused purpose. Solnit is a bit long-winded with her agenda, but that bright hope has stuck with me. Recommended. Rating: 8 out of 10

This post was written with Andr s Monroy-Hern ndez for the Follow the Crowd Research Blog. The post is a summary of a paper forthcoming in Computer-Supported Cooperative Work 2013. You read also read the full paper: The Cost of Collaboration for Code and Art: Evidence from Remixing. It is part of a series of papers I have written with Monroy-Hern ndez using data from Scratch. You can find the others on my academic website.

Does collaboration result in higher quality creative works than individuals working alone? Is working in groups better for functional works like code than for creative works like art? Although these questions lie at the heart of conversations about collaborative production on the Internet and peer production, it can be hard to find research settings where you can compare across both individual and group work and across both code and art. We set out to tackle these questions in the context of a very large remixing community. Remixing platforms provide an ideal setting to answer these questions. Most support the sharing, and collaborative rating, of both individually and collaboratively authored creative works. They also frequently combine code with artistic media like sound and graphics. We know that that increased collaboration often leads to higher quality products. For example, studies of Wikipedia have suggested that vandalism is detected and removed within minutes, and that high quality articles in Wikipedia, by several measures, tend to be produced by more collaboration. That said, we also know that collaborative work is not always better for example, that brainstorming results in less good ideas when done in groups. We attempt to answer this broad question, asked many times before, in the context of remixing: Which is the better description, the wisdom of crowds or too many cooks spoil the broth ? That, fundamentally, forms our paper s first research question: Are remixes, on average, higher quality than single-authored works? A number of critics of peer production, and some fans, have suggested that mass collaboration on the Internet might work much better for certain kinds of works. The argument is that free software and Wikipedia can be built by a crowd because they are functional. But more creative works like music, a novel, or a drawing might benefit less, or even be hurt by, participation by a crowd. Our second research question tries to get at this possibility: Are code-intensive remixes, higher quality than media-intensive remixes? We try to answers to these questions using a detailed dataset from Scratch a large online remixing community where young people build, share, and collaborate on interactive animations and video games. The community was built to support users of the Scratch programming environment: a desktop application with functionality similar to Flash created by the Lifelong Kindergarten Group at the MIT Media Lab. Scratch is designed to allow users to build projects by integrating images, music, sound and other media with programming code. Scratch is used by more than a million, mostly young, users. Measuring quality is tricky and we acknowledge that there are many ways to do it. In the paper, we rely most heavily a measure of peer ratings in Scratch called loveits very similar to likes on Facebook. We find similar results with several other metrics and we control for the number of views a project receives. In answering our first research question, we find that remixes are, on average, rated as being of lower quality than works of single authorship. This finding was surprising to us but holds up across a number of alternative tests and robustness checks. In answering our second question, we find rough support for the common wisdom that remixing tends to be more effective for functional works than for artistic media. The more code-intensive a project is, on average, the closer the gap is between a remix and a work of single authorship. But the more media-intensive a project is, the bigger the gap. You can see the relationships that our model predicts in the graph below. Both of us are supporters and advocates of remixing. As a result, we were initially a little troubled by our result in this paper. We think the finding suggests an important limit to the broadest claims of the benefit of collaboration in remixing and peer production. That said, we also reject the blind repetition of the mantra that collaboration is always better for every definition of better, and for every type of work. We think it s crucial to learn and understand the limitations and challenges associated with remixing and we re optimistic that this work can influence the design of social media and collaboration systems to help remixing and peer production thrive. For more, see our full paper, The Cost of Collaboration for Code and Art: Evidence from Remixing.

Jacob (and Oliver too, somewhat) has taken quite the interest in Christmas music this year. Perhaps it s singing in the choir at school, or perhaps it s just him getting older, but in any case, Christmas music fascinates him. And no song more than Jingle Bells. I have recordings of it by several artists in different styles, and he has his favorites and often wants to hear them again and again. The other night at supper, he said, Dad, can you sing Jingle Bells in German? Kind of a random question. I normally try to keep books and devices of all sorts away from the table, but my son had just asked me to sing. In German. I don t believe that had ever happened before, so I wasn t going to say no! I got my tablet, pulled up Google, and found some German lyrics. Not exactly a translation, but it fit the tune, so that s what counts. So I started singing, and when I got to the chorus, and sang Oh, Glockenklang, Glockenklang both boys bust up laughing. They thought Glockenklang was a hilarious word, and loved to hear it. Oliver requests I sing Glockenklang every so often now. He has this eager anticipation when he does it, as if he knows I m going to be hilarious so much so that he almost starts laughing before I even say a word. Then yesterday at breakfast, Jacob requested more German songs. I finally got a couple of hymn books (one of which, Gesangbuch mit Noten, really is a German songbook). I sang some songs in German for the boys, while they enjoyed their blueberry crepes. Sadly Stille Nacht and O du fr hliche did not prove as hilarious as Glockenklang, but they got impatient as I looked through the idnex between each song, saying Dad, just sing more German! Don t look at those pages! So I guess it was a hit. I m not sure where this sudden fascination with German music came from, but it appears to be leaving me hungry. While driving around, Oliver requested I sing Glockenklang in the car. I said yes, despite not remembering even one German word to that song except for, well, Glockenklang. So I just filled in with some random German words I do know. Fortunately, they didn t seem to have memorized the rest of it either. The other day, we went to the annual Christmas concert at the local high school. It was perfect for the boys various styles of music, an interesting old auditorium, nobody was going to care if they weren t perfectly quiet, and the price was right, too. The band played first, and both boys sat there, paying great attention, soaking it all in. They loved the cymbal crashes and Jacob seemed to dream of playing the chimes and bells. At one point, I whispered to Jacob that I could feel the timpani on the wood floor in the auditorium, and he tried it, and made his Hmm, I just learned something interesting! reaction. As is tradition, the concert concluded with singing the Hallelujah Chorus and anyone in the audience that wants to sing is invited on stage to join the choir. I remember being in high school for that concert, and after practicing it in school, the great fun of being joined by many powerful voices from the community all around me on the risers. So this time, while my parents stayed with the boys, I was one of the many that went up to join the choir. It was no less thrilling all these years later. Ending with Forever and ever, Hallelujah, Hallelujah. Hallelujah! , and feeling it, right then, put a huge smile on my face.

As you may know, Debian 7.0 (codename 'wheezy') includes Linux 3.2. I'm maintaining the 3.2.y stable series at kernel.org which will collect bug fixes and some new hardware support as nominated by kernel developers and distributors, following the usual stable kernel rules. The Debian package has some additional features not present in mainline Linux 3.2. Union filesystem Debian Live needs a union filesystem to allow writing files on top of the read-only live system image. To support this, we include Aufs developed by Junjiro R. Okajima. I hope that Linux will gain union mount or union filesystem support in time for the next release, but currently it doesn't take much work on our part to integrate. Security hardening

• Magic SysRq functions are restricted by default. This can be changed through sysctl kernel.sysrq.
• Network protocols AF_IEEE802154, AF_RDS and AF_DECnet must be explicitly loaded; they cannot be loaded just by creating a socket.
• The default permissions on the debugfs root are 700, i.e. accessible only to root. (This change has also been made in Linux 3.7.)
• The symlink and hard link restrictions from Linux 3.6 (and previously grsecurity, Openwall and Ubuntu) are included and enabled by default. This can be changed through sysctls fs.protected_symlinks and fs.protected_hardlinks.
• The procfs hidepid/hidegid feature from Linux 3.4 is included, though not enabled by default. When enabled, this prevents users from reading the state of each others' processes through procfs, which can in some cases be security-sensitive.

Bufferbloat I backported two big pieces of the solution to bufferbloat:

• Byte Queue Limits (BQL) allows network drivers to reduce unnecessary buffering in their hardware transmit queues. This has been implemented in the bnx2, bnx2x, e1000e, forcedeth, igb, ixgbe, sfc, skge and tg3 drivers in Linux 3.3 onwards. Documentation is in the commit that added it, but you're unlikely to need it.
• The CoDel queue management algorithm can reduce unnecessary buffering in software transmit queues. FQ CoDel adds an element of 'fairness' between multiple network flows. These were implemented as the queue disciplines codel and fq_codel in Linux 3.5. They can be selected using the tc command, e.g.:
  tc qdisc replace dev eth0 root fq_codel

Problem #3: How to find out what s going on with debconf? Slightly longer version: As reported in #679327, the grub prompt during the installation process is entitled Configuring man-db , which isn t exactly great. Given man-db is a known dpkg trigger, I suspected this would be the reason for the broken title, and here s how I debugged it. For starters, using d-i basically means asking a lot of questions, which is implemented using the debconf mechanism. One should note that two implementations are available:

1. the historic implementation, in Perl: debconf
2. the C reimplementation: cdebconf

d-i uses cdebconf udebs, but joy begins when packages get installed in /target: debconf is used there. We ll see why that matters. Recipe #3: Set DEBCONF_DEBUG=developer and read logs!

1. Both implementations support the DEBCONF_DEBUG environment variable. The installer supports passing that as a kernel parameter; example from the syslinux prompt: select the Graphical install entry, press Tab , remove -- quiet and put DEBCONF_DEBUG=developer there instead.
2. Let s look at /var/log/syslog; there s a tail -f on it in VT4, but one can run grep and friends on it in VT2 or VT3. There, one can see what happens at the debconf level (the debconf protocol is documented in debconf-devel(7)). Tiny excerpt (timestamps removed for clarity):

   frontend: --> GET preseed/early_command
   frontend: <-- 0
    
   frontend: --> GET debian-installer/framebuffer
   frontend: <-- 0 true
    
   debconf: --> GET debconf/language
   debconf: <-- 0 en
   debconf: --> SET debconf/language en
   debconf: Setting debconf/language to en
   debconf: <-- 0 value set
   debconf: --> GET debconf/priority
   debconf: <-- 0 high
   debconf: --> GET rescue/enable
   debconf: <-- 0 false
   

3. The interesting point was titles, right? So let s look at the last TITLE occurrences:

   debconf: --> SETTITLE debian-installer/main-menu-title
   debconf: --> SETTITLE debian-installer/grub-installer/title
   in-target: debconf (developer): ----> TITLE Configuration de man-db
   debconf: --> TITLE Configuration de man-db
   in-target: debconf (developer): <-- TITLE Configuration de man-db
   in-target: debconf (developer): ----> TITLE Configuration de man-db
   in-target: debconf (developer): ----> TITLE Configuration de grub-pc
   debconf: --> TITLE Configuration de grub-pc
   in-target: debconf (developer): <-- TITLE Configuration de grub-pc
   in-target: debconf (developer): ----> TITLE Configuration de grub-pc
   in-target: debconf (developer): ----> TITLE Configuration de man-db
   debconf: --> TITLE Configuration de man-db
   in-target: debconf (developer): <-- TITLE Configuration de man-db
   in-target: debconf (developer): ----> TITLE Configuration de man-db
   in-target: debconf (developer): ----> TITLE Configuration de grub-pc
   debconf: --> TITLE Configuration de grub-pc
   in-target: debconf (developer): <-- TITLE Configuration de grub-pc
   in-target: debconf (developer): ----> TITLE Configuration de grub-pc
   debconf: --> SETTITLE debian-installer/grub-installer/title
   

   Bad news: the last one is about grub, so the title should be correct, right?
4. At this point, it might be the Gtk frontend behaving strangely. It s shipped in cdebconf-gtk-udeb, built from the cdebconf source. Let s look:
   • src/modules/frontend/gtk/ui.c has cdebconf_gtk_update_frontend_title(), which does what the function name suggests.
   • src/modules/frontend/gtk/di.c has cdebconf_gtk_di_run_dialog(), which calls the first function.
   • src/modules/frontend/gtk/progress.c has cdebconf_gtk_show_progress() (also calls the first function), which comes with the following comments:

     /** Show the progress widgets.
       *
       * This will actually add the widgets to the corresponding containers.
       * The main title saved when starting the PROGRESS operation will be restored
       * from the value saved when START was called.  This is needed when GO is
       * called during a PROGRESS operation.
     

   Now that s something! Even if the last entry is about grub, some previous value could be restored and could be the reason for the bad behaviour (that is easily confirmed by adding some debug_printf() calls on fe->title in cdebconf, thanks to an extra "debug.h" include). Maybe that s what needs fixing!
5. Now it s time to take a step back. A d-i environment is nice, but testing things can easily become a burden. Taking a random wheezy or sid chroot, and installing/removing a tiny package shipping a manpage should be enough to run into the man-db trigger. Exporting DEBCONF_DEBUG=developer there too, let s either install or remove x11-apps (the trigger does the same job, and the debconf trace is the same), here s the output:

   Processing triggers for man-db ...
   debconf (developer): frontend started
   debconf (developer): frontend running, package name is man-db
   debconf (developer): starting /var/lib/dpkg/info/man-db.config configure /usr/share/man
   debconf (developer): <-- VERSION 2.0
   debconf (developer): --> 0 2.0
   debconf (developer): <-- INPUT medium man-db/install-setuid
   debconf (developer): --> 30 question skipped
   debconf (developer): <-- GO
   debconf (developer): --> 0 ok
   debconf (developer): starting /var/lib/dpkg/info/man-db.postinst triggered /usr/share/man
   debconf (developer): <-- VERSION 2.0
   debconf (developer): --> 0 2.0
   debconf (developer): <-- GET man-db/auto-update
   debconf (developer): --> 0 true
   

   Even if one didn t know about the cdebconf/debconf thing, the frontend started bit in the d-i syslog would have been the key: it s only found in debconf, in the frontend script. That one has several ways to determine the package being acted on, and it finally calls:

   debug developer => "frontend running, package name is $package";
   $frontend->default_title($package) if length $package;
   

   Tada, that s likely to be the issue.
6. Tweaking that frontend script, it s easy to make sure that the code path for this use case is:

   elsif ($ARGV[0]=~m!^.*/(.*?)\.(?:postinst postrm prerm)$!)  
           $package=$1;
    
   

   Since $ARGV[1] is the action being passed to the maintainer script, one can intercept the triggered action, and avoid emitting a title update in this case; I ve therefore reassigned #679327 to the debconf package, and proposed a patch implementing that, which Joey Hess seems to find reasonable.

In my message to the bug report I wrote I actually tested it in a d-i environment, and made sure the issue had gone away. How I did that will likely be described in the next d-i hacking recipe.

Care2 has an interesting article about people being offended by a picture of a poor Afro-American boy using an iPad [1]. It seems that people object to poor people having what is in many ways the greatest educational tool available because it can also be seen as a status symbol. I wonder if one of the cheaper Android tablets (at 1/3 the price of an iPad or less) would get the same response. Dvice has an inspiring article by Evan Ackerman about an experiment where a large quantity of Motorolla Xoom tablets were delivered to Ethiopian kids who couldn t read and didn t know English [2]. The kids learned printed English and learned to hack the devices to enable all the hardware features! I don t expect that kids in first-world countries would get the same benefits as kids in countries such as the US and Australia are distracted by TV, games consoles, and other forms of entertainment that aren t readily available in Ethiopian villages. But the potential benefits for widespread deployment of tablets in the poorer parts of first-world countries seems significant. I think that the government of every first-world country should give a voucher to every resident who is more than two years old for a $100 discount off the list price of a tablet. If someone is happy with a cheap Android tablet then $100 will cover the prices. If they want something better then they can pay more. I m happy for wealthy people to get a $100 discount on the price of one of the more expensive tablets, it seems that means testing such government subsidies just causes political issues without saving much money. Apart from the obvious educational benefits for people who own tablets (which means adults as well as children) there are also benefits in having everyone own a tablet. If everyone owned a tablet then the resale price of a low-end tablet would approach zero and the resale price of even more desirable tablets such as the iPad would drop a lot. This would reduce the incentive to steal tablets and allow people to use them in situations where there is less safety. If most people felt safe to use a tablet computer on public transport and started doing educational things instead of reading newspapers then it would improve the overall education of the population. Newspapers generally aren t educational and in the case of News Corp publications and other tabloids you can become less well informed by reading them I believe that on average newspapers have a negative educational value in Australia. The cost of giving a $100 tablet voucher to every resident of Australia would be about $2,200,000,000 (plus administrative costs) which seems like a lot of money. But when compared to the costs of building infrastructure it s not significant. I ve just read some government announcements regarding transport budget which includes items such as $6,500,000 to determine the best route for a new rail line. So if we were to get 350 new rail lines (and we need more than this) then merely determining the routes would cost as much as getting a tablet for everyone in the country!

• [1] http://www.care2.com/causes/why-is-the-us-so-insulted-by-welfare-queens.html
• [2] http://dvice.com/archives/2012/10/ethiopian-kids.php

A couple of weeks ago, I walked in to a nice, sit-down restaurant, with a smile on my face. It s the kind of restaurant with folded cloth napkins on the tables. Table for three, please as Jacob and Oliver were with me. This much isn t unusual. I have periodically taken them out to eat for quite some time, and they enjoy it. But there were a few unusual things about this particular day. I suppose the main one is that they had just been doing this. Yes, painting your own face can be a lot of fun. And also serious business. The boys and I were in Santa Fe, NM on a train trip. It had been a year since their last train trip, and that s longer than they are typically used to. I d taken Jacob on a train trip with just the two of us before, but this was the first trip with just the two boys and me. And one of the places we visited was the excellent Santa Fe Children s Museum. It may be the best children s museum I ve ever seen. Not the largest, or the flashiest, but that s part of the reason I say best . They had chimes (and many other percussive instruments to produce different pitches, including mounted hubcaps and varying length wooden planks). They had a great magnets table with washers and nuts, so children can build their own bridges, stairs, etc. using magnetism. A giant bubble table, tunnels to crawl in outside, etc. A great place. And, apparently, the thing they were really known for I did not know this in advance is the paint your own face station. Jacob and Oliver really got into it. Oliver informed me he was a lion and I heard ROAR! ROAR! periodically all afternoon. Jacob asked me to help paint a J, and the spirals, on his cheeks. After some careful thought, he informed me that he was spiral man . Next to the paint your own face area was a clean your own face area. Most kids were being helped to clean their own face by their parents on their way out. Jacob and Oliver protested that plan, so I figured, if they want to enjoy painted faces all day, why not? And this, of course, led into lunch with self-painted faces. Nobody at the restaurant commented, but the owner had a huge grin when he saw them. (It was a Mediterranean place, and I m sure the owner would have commented had there not been a language barrier.) Incidentally, the boys became quite the fans of souvlaki. Later, as we walked around Santa Fe Plaza and another museum, they drew smiles all over the place. Several kind people asked them, Did you enjoy the children s museum? Yes, everyone in Santa Fe seemed to know precisely where kids with painted faces had been that day. Santa Fe is an amazing and beautiful city. It was warm and friendly, and the architecture and layout was fun to see and pedestrian-friendly. We walked past the Cathedral Basilica of St. Francis of Assisi several times, and went in once. For some reason I could never fully explain, I could often smell their incense even a block or two away. It added to the crisp wintry feel of the plaza. The point of the trip wasn t Santa Fe, though. It was Jacob and Oliver on Amtrak, which is the thing they were really most excited about of course. They were excited as usual, and despite the fact that the train comes through this area only at around 3AM, were plenty excited to be on the train. And, in fact, didn t fall asleep again until about 5 due to the excitement (though they did an excellent job of being quiet). Of course, 6AM was morning so they were wide awake by then. Jacob had been planning what he d eat on the train for days already, and had announced he would be having French toast for breakfast and pizza at lunch. He was a bit disappointed to see that French toast wasn t on the menu this time, but pancakes saved the day. While waiting for the dining car to open at 6:30, we went to the lounge car for awhile. I had brought along various things for them to do on the train, of course, and among them was a notebook and some markers. Jacob loved drawing suns and stars, and sometimes writing short notes. He gave notes to several friendly people that happened to be visiting with us on the train. Oliver enjoyed it too, but he was more intrigued by the cheap set of multi-colored post-it notes. There were two happy, and somewhat tired, boys getting off that train in the middle of the night when we returned.

I have been dealing with an interesting forking issue at work. It happens to involve Perl, but don't let that put you off. So, suppose you need to perform an I/O-bound task that is eminently parallelizable (in our case, generating and sending lots of emails). You have learnt from previous such attempts, and broken out Parallel::Iterator from CPAN to give you easy fork()ing goodness. Forking can be very memory-efficient, at least under the Linux kernel, because pages are shared between the parent and the children via a copy-on-write system. Further suppose that you want to generate and share a large data structure between the children, so that you can iterate over it. Copy-on-write pages, should be cheap, right?

my $large_array_ref = get_data();
my $iter = iterate( sub  
    my $i = $_[1];
    my $element = $large_array_ref->[$i];
    ...
 , [0..1000000] );

Sadly, when you run your program, it gobbles up memory until the OOM killer steps in. Our first problem was that the system malloc implementation was less good for this particular task than Perl's built-in malloc. Not a problem, we were using perlbrew anyway, so a quick few experimental rebuilds later and this was solved. More interesting was the slow, 60MB/s leak that we saw after that. There were no circular references, and everything was going out of scope at the end of the function, so what was happening? Recall that Perl uses reference counting to track memory allocation. In the children, because we took a reference to an element of the large shared data structure, we were effectively writing to the relevant page in memory, so it would get copied. Over time, as we iterated through the entire structure, the children would end up copying almost every page! This would double our memory costs. (We confirmed the diagnosis using 'smem', incidentally. Very useful.) The copy-on-write semantics of fork() do not play well with reference-counted interpreted languages such as Perl or CPython. Apparently a similar issue occurs with some mark-and-sweep garbage-collection implementations - but Ruby 2.0 is reputed to be COW-friendly. All was not lost, however - we just needed to avoid taking any references! Implement a deep copy that does not involve saving any intermediate variables along the way. This can be a bit long-winded, but it works.

my $large_array_ref = get_data();
my $iter = iterate( sub  
    my $i = $_[1];
    my %clone;
    $clone id   = $large_array_ref->[$i] id ;
    $clone foo  = $large_array_ref->[$i] foo ;
    ...
 , [0..1000000] );

This could be improved if we wrote an XS CPAN module that cloned data structures without incrementing any reference counts - I presume this is possible. We tried the most common deep-copy modules from CPAN, but have not yet found one that avoids reference counting. This same problem almost certainly shows up when using the Apache prefork MPM and mod_perl - even read-only global variables can become unshared. I would be very interested to learn of any other approaches people have found to solve this sort of problem - do email me.

Hi, I am interested in your opionions! If you would setup today an new server for a (business) project, would you use Debian Squeeze or Wheezy? Personal and already in most business cases we have decided for Wheezy, because the pros are:

• Enhanced hardening
• More up to date technologies and scripting languages
• Longer security support, because..
• .. you do not have to dist-upgrade within the next year(?)

The cons:

• It is not stable yet

What is your (business) opionion about it?

My new Samsung Galaxy S3 arrived a couple of days ago. Kogan added to the weirdness in their shipping that I documented in a previous post [1] by shipping the two phones on consecutive days with separate invoices for $789 each (as opposed to the $449 that I paid). This means that if customs were to track this down they could bust me for import duties (which apply whenever you order more than $1000 of electronics from another country) as multiple shipments in a small period of time are aggregated for tax purposes. It also means that it would be easy to defraud the tax office or an employer about the cost of the devices if I didn t blog about it Phone and Case Review To install the Samsung Galaxy S3 in the power case I bought from Kogan the back of the S3 has to be removed. One annoying implication of this is that when using the phone in it s case I will either have to carry the back of the phone with me or have no option of removing the case during the day. The phone has to slide in to the case from a small angle and if it s removed carelessly there seems to be a risk of breaking the power connector on the case. The above photos show the kick-stand on the power case in use, I haven t yet had an opportunity to use the stand for anything other than photography and as it s made of thin plastic there s a reasonable chance that it will break off before I ever use it properly. One major deficiency of the power case is that it has no protection for the screen. The typical gel case will extend by about 2mm above the screen which means that you can place the phone screen-down on a table without a risk of scratching the screen. Also if a phone in a gel case is dropped on a hard surface it seems likely that the chance of the screen hitting a bump and cracking will be lower although I m not aware of any research regarding how well this theory works in practice. This combined with the fact that the power case is smooth and difficult to grip makes it quite unsuitable for regular use. I am concerned that the added mass of the power case would increase the risk of serious phone damage if it s dropped screen-down. I plan to buy a gel case ASAP which I will use most of the time. I will swap the gel case for the power case on the occasions when I expect to spend an extended period of time away from home or an office where I can charge my phone. The power case really should have been designed with a flange that extends above the level of the screen on all sites. It also shouldn t be smooth on the outside. Phone Review The Galaxy S3 equals the Galaxy S for the title of the most slippery phone I ve ever owned. This combined with the fact that the Galaxy S3 is slightly thinner and also 10% wider and longer makes it a more difficult phone to hold. It s a phone that really needs a case with a lot of grip. The camera is reasonable given the fact that any device that is 8.6mm thick will have a very small focal length and also a very small CCD for receiving light. The laws of physics just don t allow something the size of the Galaxy S3 to have a half-decent camera. My ideal mobile phone would be about the size of the Galaxy S3 with the power case, it would use the extra thickness for a big battery as well as a much longer focal length for the camera. I d like a phone to be about 16mm thick over most of it s area with a section that s 20mm for the camera. The burst shot mode (which allows you to take 20 pictures rapidly) is good for capturing certain types of relatively slow movement (EG someone s mouth moving when they talk). But it s no good at all if you try to pan the camera while doing burst mode. When I first started using the phone it gave me lots of hints about using various functions. This wasn t particularly useful for me as most things are obvious to anyone who has much experience with computers and the non-obvious things are probably going to be forgotten if you are doing something else when the help system prompts you. It will be useful for my sister in law who s getting the other phone, but the down-side of this is that I can t configure her phone before delivering it. She needs to use it first to see all the tips. The Galaxy S3 has a screen that s 4.8 diagonally which is slightly larger than the 4.0 screens of my previous Android phones (Sony Ericsson Xperia X10 and Samsung Galaxy S). That gives 44% more screen area which is a good thing. Unfortunately many apps seem to display the same amount of data, I had hoped that 44% more area and 140% more pixels would result in most apps displaying more data by default. Fortunately ConnectBot (my SSH client of choice) gives me 60 rows and 120 columns on the Galaxy S3 as opposed to the mere 31 rows and 80 columns on the Xperia X10. I don t use ConnectBot that often (a phone is a really bad ssh client no matter how you do it), but when I do it s really important and the extra space matters. The down-side of the larger screen for me is that the distance from the center of the palm of my hand to the end of my thumb is slightly less than 4.8 . This means that I can t hold the Galaxy S3 with one corner in the palm of my hand and still reach the far corner of the screen. So I am forced to hold the sides of the phone for one-handed use which means that it would be easier for it to slip out of my hand. A further complication is that the settings and back buttons are very close to the edge of the phone and can be hit by accident very easily. Hopefully a gel case will alleviate that. But I have never owned a hand-held device which was so difficult to hold in a stock configuration. According to AndroSensor (one of my favorite demo programs for showing useless things that a phone can do) the Galaxy S3 lacks support for measuring humidity and external temperature but it has all the other sensors. I think I can survive without knowing how hot and humid my pockets are, but it would still be nice to have every possible feature. The default partitioning of the phone storage is to have all internal storage used for applications and none used for a VFAT filesystem accessible over USB. It s nice that I won t run out of application storage space (a problem I ve had with every other Android device) but it does mean that I ll have to buy a micro SD card before I can use the phone properly. Micro SD storage costs about $1.50 per gig in the size range from 8G to 32G, it s not particularly expensive but it s annoying to have the phone not be fully usable right from the start. Without using a SD card I have to use gphoto2 to download pictures from the phone so it s fully usable in every way. But the cost of the power case, the gel case I m about to buy, and an SD card make the phone less of a bargain. The phone has more RAM, a faster CPU, and Android 4.1. These are all really good things, but nothing worth writing about in a review. Conclusions The power case is very poorly designed, I probably wouldn t buy it if I knew then what I know now. For one-handed phone use by someone like me with longer than average fingers a 4.0 phone is about the largest possible. This doesn t mean that phones should stay small, but it does mean that Apple weren t being entirely silly when they released all the 3.5 iPhones. Presumably a large portion of the population would find a 3.5 phone to be as usable as a 4.0 phone is for me. It now seems to me that as a 4.8 screen is going to be impractical for one-hand use by almost everyone the phone designers should concentrate more on larger phones similar to the Samsung Galaxy Note. I think that the 4.8 screen size may be ideal for some young children who could use it the way an adult might use a 7 tablet an Android phone seems to be a relatively common present for children in the 6-12 age range. Generally I think that no-one should ever buy a new phone or tablet without trying one of the same size. While I am happy with the S3 it really doesn t work in the way I expected. Having 2.9* as many characters on screen for ConnectBot is a massive benefit. If I ever write another magazine article on a hand-held device the extra size and resolution will really help if I found a suitable editor (please make suggestions in the comments). For the WordPress client to write long blog posts on it then the screen could potentially be reasonably useful, but so far it seems that it s more suited to reviewing posts and comments than to writing posts. Generally the Samsung Galaxy S3 is a good phone and I think it s great value for money when it s less than $500.

• [1] http://etbe.coker.com.au/2012/12/11/dependencies-in-online-ordering/

Given the very positive feedback for Dennis Felsing s tool ghc-vis, which visualizes the heap representation of a Haskell value, including all the gory details such as thunks, values retained by thunks indirections, sharing etc, I saw the need to provide this information also directly in GHCi, without having to load a graphics library or opening extra libraries. So I added the required features (traversing the heap and pretty-printing the results) to my ghc-heap-view package and, also following ghc-vis s lead, added a ghci file that, when loaded, provides you with a :printHeap command. Here you can see it in action: A plain value

Prelude> :script /home/jojo/.cabal/share/ghc-heap-view-0.4.0.0/ghci
Prelude> let x = [1..10]
Prelude> x
[1,2,3,4,5,6,7,8,9,10]
Prelude> :printHeap x
_bh [S# 1,S# 2,S# 3,S# 4,S# 5,S# 6,S# 7,S# 8,S# 9,S# 10]

Note that the tools shows us that the list is a list of S# constructors, and also that it is still hidden behind a blackhole. After running System.Mem.performGC, this disappears. A thunk

Prelude> let x = Just (1 + 1)
Prelude> :printHeap x
Just _bco
Prelude> x
Just 2
Prelude> System.Mem.performGC
Prelude> :printHeap x
Just (S# 2)

Here, we see how the calculation was deferred until forced by showing the value of x. The name _bco stands for a bytecode object as used by the interpreter. Getting useful information from them is a bit harder than for compiled thunks, so for more accurate results put the code in a Haskell source file, compile it and use the GHC.HeapView API to print the interesting parts. A partial application

Prelude> let a = "hi"
Prelude> let partial = (a ++)
Prelude> partial ""
"hi"
Prelude> System.Mem.performGC
Prelude> let x = (a, partial)
Prelude> :printHeap x
let x1 = "hi"
in (x1,_fun x1)

The information <emph>which</emph> function is called there (++ in this case) is lost at runtime, but we still see that the second element of the tuple is a partial application of <emph>some</emph> value to the first element. A cyclic structure

Prelude> let s = "ho"
Prelude> let x = cycle s
Prelude> length (take 100 (show x))
100
Prelude> System.Mem.performGC
Prelude> :printHeap x
let x0 = C# 'h' : C# 'o' : x0
in x0
Prelude> let y = map Data.Char.toUpper x
Prelude> length (take 100 (show y))
100
Prelude> :printHeap y
C# 'H' : C# 'O' : C# 'H' : C# 'O' : C# 'H' : C# 'O' : C# 'H' : C# 'O' : C# 'H' : C# 'O' : _bh (C# 'H' : C# 'O' : C# 'H' : C# 'O' : C# 'H' : C# 'O' : C# 'H' : C# 'O' : ... : ...)

The cyclic, tying-the-knot structure of cycle is very visible. But can also see how easily it is broken, in this case by mapping a function over the list. Mutual recursion

Prelude> let  x = 'H' : y ; y = 'o' : x  
Prelude> length (show (take 10 x, take 10 y))  seq  return ()
Prelude> System.Mem.performGC
Prelude> :printHeap (x,y)
let x1 = C# 'H' : x3
    x3 = C# 'o' : x1
in (x1,x3)

If you want to look at multiple variables at once, just pass a tuple to printHeap In the hope that this will be a useful tool for you, I uploaded version 0.4.0.0 of ghc-heap-view to hackage.

WebAuth is a site-wide web authentication system with single sign-on support. We use it extensively at Stanford for most of our user-facing web applications that need to authenticate people. This is the culmination of the last four months of work, so it's rather nice to finally get it out the door. Even if not everything I wanted to get done is done. That's always how it is with software releases. This release fixes a bug in the encoding of delegated Kerberos credentials when Heimdal is used as the Kerberos library. If you're building WebAuth against Heimdal, upgrade any mod_webauth modules that receive delegated credentials before upgrading mod_webkdc on the WebKDC. Otherwise, the flags of delegated tickets won't be sent correctly. The major new feature in this version is support for authorization identities separate from authentication identities. This can be used for testing, for administrators to proxy into an application as another user, or in other cases where there's a mismatch between the canonical concept of users and the capabilities of an application. The WebLogin server and WebKDC now support an ACL file that controls what alternative identities users can assert to specific web sites, and there is a new flow on the WebLogin side to set or change identities. All of this is disbled by default in each component. There is a new WebKDC directive to enable this support and specify the identity ACL file, and there is a new directive in mod_webauth to tell it to trust the authorization identity. There is a new environment variable that will be set to the authorization identity (trusted or not), and REMOTE_USER will be set to the authorization identity only if it is trusted. WEBAUTH_USER will always remain the authentication identity, so both identities can be logged and web applications can distinguish. To address one of the edge cases required by this support, there's a new mod_webkdc Apache directive, WebKdcLoginTimeLimit, which controls how long a multi-step login process can take, and also how recent a login has to be to contribute its authentication factors to the session factors for an authentication. This is also now used for WebAuthForceLogin, which means that forced logins won't require re-authentication if they're within the WebKdcLoginTimeLimit interval of the last login. This allows that feature to work properly in conjunction with authorization identities and with some multifactor authentication methods. Also in this release, optional replay detection and rate limiting of failed logins has been added to the WebLogin server. Either or both can be enabled in the configuration file. This support requires a memcached server (shared across any pool of WebLogin servers) be available to store the necessary state. There are new parameters in the error template to handle the error messages generated by these features. The WebLogin server also has a few more minor improvements: single sign-on cookies are now set even when displaying error pages if any are available, fixing some looping issues with some scenarios around restricting users to authenticate to specific sites, and the @REMUSER_REALMS setting has been broken into two settings so that its two properties can be changed independently. The old setting is still supported for backward compatibility. Multiple bugs in Kerberos ticket encoding have been fixed, some that were introduced in 4.3.0 and some that have been present since the first days of Heimdal support. Other fixes include the mapping of WebKDC error codes to names in WebLogin (which previously resulted in Perl warnings in the error log) and the missing documentation for the WebAuthRequireSSL directive. Finally, the thing that I'm the happiest about in this release is that I finished my complete refactoring of the libwebauth library. Most of the low-level interfaces are gone in favor of higher-level manipulation of WebAuth protocol objects. The last pieces of code that used the old token encoding system have been replaced with the new data-driven encoder. And the last parts of the library have been converted to APR, so there is no longer a mix of traditional memory management and APR pools. This comes with lots of simplification of the API and removal of old cruft from the public headers. You can get the latest release from the official WebAuth distribution site or from my WebAuth distribution pages.

This release is a rollup of a variety of miscellaneous features and bug fixes that have accumulated since September. It takes the first few steps towards the general Perl coding style revision that I'm working on, but only a few. I've yet to move some generic Perl test frameworks into this package (hopefully soon). In the new feature department, there's a new portability wrapper around sys/statvfs.h that converts statvfs code to statfs code for older systems, and a lot of enhancements to the portable/apr.h header to fix up some more constants for APR 0.9 compatibility. In the bug fix department, the Kerberos and GSS-API probes now probe for Heimdal's libroken by looking for a function that it has provided since at least 0.4 instead of one that's new in 1.3, fixing some portability issues to BSD systems. The Kerberos header probes also now check for headers using file existence checks instead of the compiler in some cases to fix problems with detecting the wrong header when using one of several different Kerberos library installations on a system. In the test suite department, this release updates the POD and POD spelling checks to be generic and use more modern modules and techniques, and adds a new set of generic Perl tests that will be useful for any of my packages that contain Perl code. There's also a new valgrind suppression file for assisting with valgrind testing in packages that use Kerberos. You can get the latest version from the rra-c-util distribution page.

Lately I've been working on making the redirector nicer to the mirrors and to some potential users. More specifically, those behind a caching proxy.

The redirector is now nicer to traditional web proxies by redirecting to objects that are known not to change with a "moved permanently" code (HTTP status code 301.) This applies to files in the pool/ directory and ".pdiff" files, among others.
Previously, a traditional caching web proxy would sometimes end up with multiple copies of the same object, fetched from different mirrors; and the redirection would not be cached at all. With this change, this is no longer the case.

Using a caching proxy that is aware of the Debian repository design is still more likely to yield better results, however: If my memory serves correctly, apt-cacher has the ability of updating the Packages, Sources, and similar files with the ".pdiff"s on the server side. Apt-Cacher-NG apparently can use debdelta, and so on.
Check my blog post about one APT caching proxy not being efficient for some comments related to those tools.

Another recent change is that mirrors that can't be used by the redirector will no longer be monitored as often as the other mirrors. For instance, if a mirror doesn't generate a trace file (used for monitoring) then the redirector will gradually limit the rate at which the mirror is checked.
This rate-limiting mechanism applies to different kinds of errors, and should reduce the amount of wasted time and bandwidth while still allowing automatic-detection of mirrors that recover.


The rate limiter applies an initial exception to allow temporary errors to not affect the use of the mirror by the redirector. After that exception, it is pretty much linear. However, that chart doesn't really reflect the effect of the rate limiter, so put in comparison with the normal checking behaviour:


The code to detect mirrors that don't perform a two-stages sync that I talked about in a previous post has not yet been integrated as the current implementation would be too expensive on the mirrors to just add it as-is.

While tracking down problems exposed to users, I decided to take a stricter approach as to what mirrors are used by the redirector. Suffice to say that the remaining mirrors using the obsolete anonftpsync are going to be ignored entirely. ftpsync has been around for a few years now and it is the standard tool.
Whether you are mirroring Debian, Raspbian, Ubuntu, or any other Debian-like packages repository, ftpsync is the right tool to use.

Most of the issues I've been discovering, and sometimes working around, affect direct users of the mirrors and are not related to the http.debian.net redirector. When not detected beforehand they happen to be exposed by the redirector, but like I said, I plan to be stricter in order to increase the redirector's reliability. Once a strict and reliable foundation is built, more workarounds might see their way in to better use the available resources.

That's it for now. The road is long, the challenge is great, and being an observer in an uncontrolled environment makes it even more interesting.

Review: Liars and Outliers, by Bruce Schneier

Publisher:	John Wiley
Copyright:	2012
ISBN:	1-118-14330-2
Format:	Hardcover
Pages:	285

One of the perils of buying a book written by a blogger one reads regularly is that the book may be little more than a rehashing of their blog, with insufficient original material to warrant the time investment. Sometimes it's still nice to support them financially, but it may not make sense to read the book. I've been following Schneier's blog for years (as should anyone with an interest in security), including through the entire process of writing Liars and Outliers, and was a bit worried that might be the case here. Thankfully, I can reassure any other worried potential readers that is not the case. This is substantial new material establishing a firm framework for thinking about incentives and controls in any society or organization. Liars and Outliers talks about security mechanisms, but it's not, at its core, a book about security. Rather, it's a book about incentives, order, and how order is established. It's a comprehensive reductionist analysis of how societies create predictability and compliance to allow us to trust other people who we have never met before and will never meet again. It's a unique (at least in my experience) combination of anthropology, sociology, security analysis, and political science. Schneier cuts across fields in an idiosyncratic but illuminating way that reminded me of (on an entirely separate topic) Jane Jacobs. This is not a prescriptive book, nor is a collection of answers, solutions, or even deep analysis of particular problems. Rather, it's an attempt to construct a general framework for analyzing societal dilemmas: conflicts between individual desires and social good, how those conflicts are resolved, and how societies can weigh the scales and influence the statistical outcomes. The closest Schneier comes to telling the reader how to solve problems is a checklist, at the end of the book, for designing effective societal pressures. Its primary contribution is vocabulary and structure. It also passes one of my litmus tests for any book about human behavior: Schneier complicates, broadens, deepens, and expands understanding, and points out complex interactions and complex feedback in effects we're inclined to consider simple, rather than simplifying or eliminating human complexity. One point I found refreshing about this book is that Schneier is scrupulous in refusing to define either society or individuals as good or bad, to the point of carefully defining terminology used in all of the social dilemmas. Following societal rules is called compliance; not following those rules is called defection. In some cases, defection is morally correct (Schneier's most frequent example is in the civil rights struggles of the 1960s in the United States). In all cases, social pressures are tools, which can be used to encourage compliance with moral or immoral systems, and which are deployed by totalitarian dictatorships and utopian communes alike. Schneier explicitly puts out of scope for this book the questions of how societal goals should be determined, how they change, and whether any given societal rule or interest is moral or immoral. He focuses, rather, only on the mechanisms, with a primary goal of informing and deepening debates over how best to encourage behavior that societies want to encourage and discourage behavior societies want to discourage. He also emphasizes discourage, as opposed to eliminate. Early on, Schneier shows some of the results of game theory, as well as basic common sense, that indicate that no healthy society can totally eliminate defection. Not only would that stifle valuable and important reform, such as changes in civil rights, but the degree of pressure required is immense. Defectors are natural and will always exist, and are sometimes valuable and necessary. Rather, the goal of a society is to reduce defectors to a level where most people can ignore their existence most of the time, a state that leads to the level of risk and trust required to have a functioning and healthy society. The word society, similarly, is intentionally broad, and can refer to just about any collection of people, from a circle of friends to a corporation, institution, or country. However, as Schneier points out early in the book, small societies rarely need much in the way of formal pressure and appear almost magically self-governing. That's the first property that he disassembles, resulting in a general classification of societal pressures into four categories: moral, reputational, institutional, and security systems. The last is an odd category that's partly orthagonal to the other three. Moral pressure is internalized conscience: the normal tendency of nearly everyone to follow their own moral code, a code that's at least partly constructed and certainly heavily influenced by the surrounding society. Reputational pressure is, in a sense, externalized morality: it's the informal reactions of others around one to one's past actions. Included in reputational pressure is shunning of every kind, from cutting off a friendship to boycotts against corporations, but it also includes confrontation from another member of one's society, the more subtle effects of our individual desires to be liked and respected, and all the various aspects of "face", honor, and respect within a community. Small communities are frequently self-governing, in Schneier's model, because they're small enough that moral and reputational pressures are sufficient and no other pressures are required. We're so used to applying moral and reputational pressure to other humans almost unconsciously that we sometimes don't even notice its existence, leading to that "magical" self-governing property. But Schneier puts pressures in a sequence: pressures that work extremely well with small groups often don't scale. Moral pressure works best with small groups and reputational pressure with somewhat larger groups, but when societies scale beyond the limits of reputational pressure when, for example, one frequently interacts with people whose reputations are unknown to you and whose subsequent opinion of you will not be relevant institutional pressure is required to force compliance. Institutional pressure is the sort of pressure that we all tend to think of first when we look for ways to enforce rules: laws, policies, contracts, and other codes of behavior that carry with them formal punishments and some enforcement mechanism. But even in societies so large that institutional pressures are frequently required, such as whole countries, moral and reputational pressures still exist and are extremely important. One of Schneier's most interesting points is his analysis of how institutional pressures can paradoxically undermine reputational and moral pressures, resulting in more defection than if the institutional pressure hadn't existed. This is just the basic framework of Schneier's analysis, hopefully giving you a feel for the structure of the book. He goes much deeper into the complicated interactions between the various levels of pressure, and then dives into an extensive look at competing societal dilemmas: cases where there is more than one society in play simultaneously, possibly demanding contradictory actions. Liars and Outliers also includes a wonderful analysis of organizational entities within the same framework, including their much-different reactions to moral, reputational, and institutional pressures. One of the most cogent analyses of the difficulties of regulating both corporations and governmental institutions falls out of that analysis, once one looks at them in light of Schneier's basic framework. Pressures quickly become complex and multi-layered, and human reactions to pressures are frequently counter-intuitive. Schneier draws extensively on game theory to show that some counter-intuitive responses are actually emergent properties of logical analysis of the situation, but that others are more uniquely human and have little or nothing to do with a mathematical cost-benefit analysis. I haven't even mentioned his discussion of security systems, and how they can extend moral, reputational, and institutional pressures, as well as add a new type of pressure (making a defecting action impossible) that scales even better than institutional pressures. Liars and Outliers has all of the supporting infrastructure you would expect in a scholarly book: notes, extensive references, and a good index. I suspect it will end up being used as at least additional reading in college classes. The notes are, unfortunately, end notes, making the full context of the book much harder to read than was necessary, but at least Schneier does separate the notes from the references so that one doesn't chase notes for further explanation and find a simple citation. As with any book like this, one always wishes it could end in a simple prescription to fix everything, but of course it doesn't. But that's also a measure of a good scholarly work. Human and organizational motivations are complex and tricky, and any framework for analyzing them needs to be able to represent that complexity. Schneier here has constructed a very powerful one, one that I started using in discussions before I'd even finished the book. Perhaps the most valuable contribution of Liars and Outliers to public discussion is clear terminology and categories, which can be of great help in finding the core components of a problem. Liars and Outliers can be slow going, particularly early on when Schneier is still defining terms and setting up the background of his analysis. One can get a bit tired of the analysis matrices of societal dilemmas. But stick with it through the groundwork, since the analyses of competing societal dilemmas and of the impact of societal pressures on organizations are exceptional. Highly recommended, particularly for anyone who is designing or implementing societal pressures: managers, political activists, or anyone in a security-related field. Rating: 8 out of 10

With apologies to Ken Thompson, this is a list derived from the wafflings residents of a certain UK IRC channel. The conversation happened several years ago but I was reminded of it again today from a conversation on that same channel.

No names have been kept to project the guilty.


/bin/bush - a shell that steals and lies?
/bin/cash - displays adverts before each prompt
/bin/crash - Microsoft shell
/bin/mash - requires libsausage.so
/bin/hush - terse output
/bin/irish - only shell that finds jerkcity funny
/bin/rush - ber-optimised for speed, but might not actually work correctly
/bin/flash - proprietary shell that displays adverts and cartoons and hangs periodically
/bin/welsh - whinges about people not using it, but then steals features from other shells in order to actually make sense
/bin/rehash - never actually runs any program, just uses markov chains to construct output from stuff it has seen before.
/bin/wash - will remove lint in your scripts if you let it
/bin/parish - prays for the successful exit of every command
/bin/punish - symlink to /bin/csh
/bin/sheepish - apologises when $? is not zero
/bin/diminsh - decrements all result codes by one
/bin/lavish - and you thought bashisms were bad...
/bin/brainwash - once you've tried it, you'll believe it's the only shell in existence
/bin/hoggish - written in Java
/bin/reversepolish - arguments order different in go must
/bin/ganesh - no subprocess dares exit zero for fear of being removed from the system
/bin/roguish - every day is April 1st as far as it is concerned
/bin/macintosh - it's shiny, has *loads* of things you think you can poke at, yet it only actually responds to a single key on the keyboard and has no useful features for fear of confusing the user
/bin/lush - garbles output from processes like a drunkard
/bin/snobbish - you're not good enough to use it
/bin/thrush - it itches
/bin/vanquish - kills processes mercilessly
/bin/tush - pert
/bin/fetish -> /bin/zsh
/bin/skirmish - multi-user shell
/bin/whiplash - gets invoked if you break too hard
/bin/newsflash - BREAKING NEWS pid 1234 terminated
/bin/mulish - DJB does Shell
/bin/hsilop - arguments order different in go must
/bin/whitewash - government approved
/bin/trish - for tri-state hardware
/bin/jdsh - it sucks...
/bin/flesh - optimised for viewing p0rn