Search Results: "ron"

25 December 2012

Petter Reinholdtsen: How to backport bitcoin-qt version 0.7.2-2 to Debian Squeeze

Let me start by wishing you all marry Christmas and a happy new year! I hope next year will prove to be a good year. Bitcoin, the digital decentralised "currency" that allow people to transfer bitcoins between each other with minimal overhead, is a very interesting experiment. And as I wrote a few days ago, the bitcoin situation in Debian is about to improve a bit. The new debian source package (version 0.7.2-2) was uploaded yesterday, and is waiting in the NEW queue for one of the ftpmasters to approve the new bitcoin-qt package name. And thanks to the great work of Jonas and the rest of the bitcoin team in Debian, you can easily test the package in Debian Squeeze using the following steps to get a set of working packages:
git clone git://git.debian.org/git/collab-maint/bitcoin
cd bitcoin
DEB_MAINTAINER_MODE=1 DEB_BUILD_OPTIONS=noupnp fakeroot debian/rules clean
DEB_BUILD_OPTIONS=noupnp git-buildpackage --git-ignore-new
You might have to install some build dependencies as well. The list of commands should give you two packages, bitcoind and bitcoin-qt, ready for use in a Squeeze environment. Note that the client will download the complete set of bitcoin "blocks", which need around 5.6 GiB of data on my machine at the moment. Make sure your ~/.bitcoin/ directory have lots of spare room if you want to download all the blocks. The client will warn if the disk is getting full, so there is not really a problem if you got too little room, but you will not be able to get all the features out of the client. As usual, if you use bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Russ Allbery: Review: A Paradise Built in Hell

Review: A Paradise Built in Hell, by Rebecca Solnit
Publisher: Penguin
Copyright: 2009
Printing: 2010
ISBN: 0-14-311807-2
Format: Trade paperback
Pages: 319
A Paradise Built in Hell is a book with an agenda. Solnit's goal is to convince the reader that nearly everything we're shown in movies and popular culture about human behavior during disasters is wrong. Not only is destructive mass panic unusual verging on nonexistent, spontaneous cooperation and acts of startling courage and resourcefulness are commonplace. The heroes that we hear about afterwards are more the norm than the exception. Disasters are far more likely to bring out the best in people than the worst. They break down barriers and form spontaneous human communities that survivors remember for a lifetime. Rather than natural disasters leading inevitably to an exacerbating human crisis in the absence of strong authority, Solnit argues that they are an amazing human opportunity, and that the most negative and destructive human behavior during crises is not from untrained victims but from the panic of the elites who are supposedly responsible for protecting the public. I think her analysis and presentation is both deeply appealing and deeply flawed. The flaws involve extrapolation and generalization beyond the applicability of her research, and I'll say more about that in a moment. But the appeal is still surprising and satisfying, and I think there's a lot of truth in this book despite its one-sided presentation. I like to tell people, half-jokingly, that I'm an anarchosyndicalist. It's my political alliegance of the heart: the political structure that I know would never work, but that matches the world I want to live in. (I suspect much of the belief in libertarianism is of this form.) Now I have a book to point people at when they ask what the appeal of anarchosyndicalism is. The sense of spontaneous community, of bonding, of reaching out to other people to do something necessary, together, that Solnit describes here in the aftermath of a disaster matches that vision of a world without imposed economic authority, where people spontaneously collaborate to solve problems. A Paradise Built in Hell is structured by disaster, providing a tour of major disasters and their aftermath interspersed with extended musings on what the human reactions mean and how disaster communities functioned. The major disasters discussed are the 1906 San Francisco earthquake, the Halifax explosion, the 1985 Mexico City earthquake, 9/11, and Hurricane Katrina. Solnit moves through them in chronological order, closing the book with Katrina, which had (at least for me) the most infuriating and uplifting mixes of community bonding and elite panic. But both of those factors are present from the beginning in the 1906 earthquake: spontaneous human organization that (from multiple first-hand accounts) transformed people's lives mixed with a military reaction and elite panic that possibly did more damage to the city than the earthquake did. Elite panic is the dark side of Solnit's celebration of human altruism and spontaneous community organization. The primary picture she paints is of ordinary people responding with great courage and creativity and finding, in their reaction, a transformative experience and a sense of purpose that often becomes the most powerful and positive experience of their life. But this is at odds with a deeply-entrenched elite belief in mass panic, in the necessity of keeping information from the public to suppress their reaction, and in the need to re-establish "order." And with that, Solnit is absolutely scathing, and with some justification. Along with detailed accounts of these major disasters, Solnit also gives the reader a tour of the research literature on disasters, research that has shown that the panic and chaos that is at the center of practically every disaster movie ever made is essentially a myth. Mass panic in disasters is not only rare, it's close to non-existent. As you might expect, the handling of Katrina by both the government and by neighboring white communities is the subject of the harshest attacks. The material on Katrina presented here is enough to make anyone want to throw out the current playbook on how we react to disasters (and lest one think Solnit is unfair, her research matches very closely with other research and journalism on Katrina that I've seen from a wide variety of sources). But Solnit shows that this is a pattern: the combination of our media-driven belief in how people will react to disasters combined with the elite need to re-establish control plays out in many major disasters, and almost always negatively. Even in 9/11, the official response in some cases hampered and undermined an already-effective unofficial response. This is very interesting and has concrete implications for public policy around disasters. But where Solnit moves onto thinner anarchosyndicalist ice is the story that she tells about disasters as dramatic upheavals of the established social order that could point to a revolution in how we interact with each other. Solnit is deeply inspired by human altruism during disasters, and her enthusiasm is somewhat contageous, but it was useful to read this book shortly after Bruce Schneier's Liars and Outliers. I think Solnit, in her understandable eagerness to find a way to extend disaster behavior to general life, does not understand the relationships between what Schneier terms moral, reputational, and institutional pressures. And that's the core of why we need to both enable spontaneous human response in disasters and have organized first responders and supporting infrastructure, and why disaster communities will break down when extended beyond the disaster. When I first mentioned A Paradise Built in Hell to my mother, her first reaction was dubiousness due to her memory of the 1977 New York City blackout. There was a disaster that resulted in widespread looting and destruction, rather than the positive reaction Solnit describes. Solnit does have a cogent argument for why looting is poorly analyzed (we don't, for example, distinguish between breaking into stores to get desperately needed supplies and doing so for personal gain) and should not be a priority in a life-threatening disaster. But it's not completely convincing; in some disasters, particularly ones like the 1977 power outage that were not particularly life-threatening, the looting has been the most destructive part of the disaster. Schneier provides what I think is the missing piece, both to this and to why the wonderful behavior during disasters doesn't last into everyday life. During a life-threatening disaster, moral pressures are massively increased. Nearly all of us have self-images and moral beliefs that provide very strong incentives to help others and collaborate in the face of life-threatening emergencies, incentives that override other motives and lead to the sort of behavior that Solnit celebrates. But moral pressures have inherent limitations. That moral reaction applies most strongly to small groups (such as bands of survivors), and will start to break down as larger society re-establishes itself. And disasters, like a power outage, that are not immediately life-threatening will not provoke the same intensification of moral pressures: we don't have strong moral beliefs and internal stories about staying quietly at home during power outages the way that we do for saving people from burning buildings or pulling them out of rubble. Once larger communities have reformed and that short-term emergency reaction has dissipated, the need for institutional pressures (police, law, and formal authority structures) will return. I wish Solnit had been able to read Schneier's book before writing hers. I think his analysis structure provides a healthy dose of realism and perspective to her desire for a self-organizing communal world, and might have blunted her enthusiastic but somewhat unrealistic hopes for finding some key to a revolution in human organization in the heart of disasters. But if A Paradise Built in Hell is unlikely to lead to a radical change in how we organize societies, it should at least provoke a radical rethinking of some of our unnecessary pessimism about our fellow humans. The evidence is substantial and compelling: crisis brings out the best in people, not the worst, and the ordinary people who happen to be around are capable of becoming some of the most amazing disaster response teams that we can imagine. This has numerous mundane implications for social policy. We should consider simple training for the general public, for example, to strengthen that reaction, should allow for it in legal structures and training of first responders (around looting, for instance), and should do everything we can to kill the destructive myth that people cannot be trusted in disasters or that mass panic will be as much of a threat as the disaster. But the best part of this book is that it provides a concrete, well-documented, and well-defended reason to let go of some of our cynicism and to extend a bit of trust to our fellow humans. There sadly aren't earth-shaking implications for everyday society, but there is a wealth of evidence that shows that we're better people in the crunch than we believe we are, or that we expect of others. And the more that we can embrace the dissolution of boundaries and social hierarchies in the middle of disasters, the more we are likely to be surprised by deep human connection and a sense of shared, focused purpose. Solnit is a bit long-winded with her agenda, but that bright hope has stuck with me. Recommended. Rating: 8 out of 10

24 December 2012

Bastian Blank: Relaying e-mail over unreliable connections

Introduction I still prefer to handle all my mails at home. I got backup set-up on my own and have anything in direct access. This network is connected to the internet via an end-user DSL connection without a static IP-address and without any useful SLA. However relaying mails over such an unreliable connection is still an unsolved problem. There exists a lot of solutions for this problem. I'm currently building a new e-mail setup, so I tried to collect all the used and maybe possible solutions for the relay problem. I will show some of them.
Don't do it The easiest solution is to just don't try to collect mails at home. This solution is somewhat against the rules, but I know people who prefers it. Access to the mails is usually done with IMAP and a copy is made if needed. This is not really a solution, but it works for many people.
SMTP SMTP is the mail protocol used in the internet. It is used for all public mail exchanges. By itself it can't be used to submit mails to remote destinations With some preparations it can be used to relay mail over unreliable connections. There are three different turn commands in SMTP that can be used to start the mail flow. Two of them will be
VPN or dynamic DNS with TLS with or without extended turn (ETRN) Using SMTP themself is an easy solution. It can relay mails without much hasle But it needs either a fixed or authenticated connection. SMTP can use a fixed connection. This is usually provided by some sort of VPN. The VPN can be encrypted, but does not need to. This allow the MTA to connect to the downstream server if it is available. The other solution is to authenticate the downstream server. Authentication is available via TLS and X.509 certificates. It still needs some way to find it, but with dynamic DNS this is no real problem. Both variants can be combined with the extended turn command. Extended turn allows to request a queue flush for a domain. It can be used to request mails only to be delivered if the downstream server is available at all. This reduces the load on the MTA.
Authenticated turn (ATRN) On-demand mail relay is a rarely supported ESMTP feature. The Authenticated turn command effectively reverses the SMTP connection and allows, after authentication, the flow of mails from the server to the client via standard SMTP. There exists some stand-alone implementations, but no widely used MTA includes an implementation.
POP3/IMAP and fetchmail/getmail All e-mails can be delivered to mailboxes and retrieved by the end-users system and re-injected into the delivery system. Both fetchmail and getmail are able to retrieve mail from POP3 and IMAP servers. They are either directly delivered via a MDA like procmail or maildrop. Otherwise they are submitted via the sendmail interface or SMTP and delivered by the MTA. Neither POP3 nor IMAP have support for meta data like the real recipient or sender.
Mailbox per address The mails for each address are delivered to its own mailbox. This allows proper identification of the recipient address. It still have no real idea of the sender address. Because it needs to poll one mailbox per address, this highers the resources needed on both sides dramatically.
Multi-drop mailbox The mails for all addresses are delivered into one mailbox. The original recipient must be saved in a custom header to allow this information to be restored. Only one mailbox needs to be polled for mail addresses.
UUCP One of the oldest transports used for mail is UUCP. It effectively copies a file to a different system and pipes it into a program. UUCP can be used to transport mails in various ways.
rmail Each mail is copied verbatim to the client. It saves the sender address in form of a "From" pseudo header in the mail itself and supplies the recipient on the command-line. So it have access to both sender and recipient address.
Batched SMTP Batched SMTP transfers old-style SMTP transactions over UUCP The MTA (Exim supports this) or a special tool (in the bsmtpd package) writes this files and after a given time or size they are dispatched via UUCP to the remote system. The bsmtpd packages was removed from Debian some years ago.
Dovecot synchronization Dovecot support bi-directional synchronization of mailbox contents. It holds all data on both sides. The internal log is used to merge changes done on both sides, so it should not loose any data. This synchronization can be used to work with the data on both sides (via Dovecot of cause) or create backups. It needs shell access to the user owning the data on both sides.
Conclusions There is no one size fits all solution for this problem. If you admin the remote system, you can implement any of this solutions If it is managed by someone else you need good luck. Almost all solutions does not support current SMTP features. The one I'm really missing is DSN, aka configurable delivery notices. POP3 and IMAP handle already delivered mail and have no need for this. All UUCP variants does not handle it, because they are much older anyway. Only SMTP itself supports all of its features. I still use rmail over UUCP for my mails at home and it works flawless. UUCP itself runs over SSH, it can compress data on the fly and authenticate using private keys.

Russ Allbery: Review: Cry Wolf

Review: Cry Wolf, by Patricia Briggs
Series: Alpha and Omega #2
Publisher: Ace
Copyright: August 2008
ISBN: 0-441-01615-4
Format: Mass market
Pages: 310
Cry Wolf is a sequel to the novella Alpha and Omega. This was originally included in the anthology On the Prowl, which I had no particular interest in, but it's now available as a standalone novella if you have an ebook reader. I recommend reading it first if you have any interest in this series; it's not particularly expensive, and piecing together what happened in it while reading the beginning of Cry Wolf is not particularly enjoyable. There are some mild spoilers for Alpha and Omega here, so if you're planning on reading the novella first, you may want to do that before reading on. Anna is an Omega, an extremely rare form of werewolf who does not fall into the normal pack hierarchy. With normal werewolves, either the wolf is dominant or is submissive, and the most dominant wolf leads the pack and can force their will on the other pack members. Omegas are strange. Normal pack dominance doesn't really work on them, but neither are they dominant themselves. At the start of Cry Wolf, Anna has very little idea what any of this means. She's been rescued from a horribly abusive situation and is being brought to Montana by Charles, the enforcer and son of the leader of all North American werewolves. And, apparently, her mate, according to both of their wolf natures, but she's not entirely sure what that means either. She doesn't have a lot of time to find out, since the local pack is immediately threatened by a rogue werewolf and she's pulled into both pack politics and that hunt. Cry Wolf, despite being the first full book of the Alpha and Omega series, is set in the same world as Briggs's Mercy Thompson series. (At a guess, somewhere around Iron Kissed.) This is not just a vague link within the universe, unfortunately. The characters here refer frequently to Mercy and previous events in her series in that way that implies the reader should have read them, without explaining fully what happened. If, like me, you're more interested in this series than the Mercy Thompson series (I preferred it because there are no vampires and I'm sick of vampires), be aware that starting here will feel like starting in the middle of a series. This will be even worse if you don't read Alpha and Omega first, making the first section of the book feel like constant namedropping when you don't know any of the references. The story, otherwise, is basically hurt/comfort with a side of coming of age. Whether you will enjoy this will depend on whether you're in the mood for a fairly predictable story that hits those buttons. Anna is a mess, thrown into survival mode after years of abuse and hating what she's become. (She is, actually, somewhat less of a mess than I think she should be given what happened to her, although Briggs has both her Omega nature and her wolf as outs there.) Charles wants to comfort her and isn't sure how. She misreads things, he misreads things, there's lots of missed communication... you've probably read this story before and have a pretty good idea how it will go. They are somewhat less stupid about their communication problems than the typical novel of this type, but they still don't use words very well. The rest of the plot felt like mostly an excuse to throw Anna and Charles together doing something external to their relationship and to give Anna a chance to use her unique talents. It takes a bit to get going, since most of the early book is introductions and setting up all the connections to the Mercy Thompson stories, but there is some meat to it, including fleshing out the mythology of werewolves in Briggs's world. (And some passable descriptions of Montana wilds.) I'm not a big fan of how power struggles and combat seem binary and very hierarchical, but that seems to be the way that Briggs differentiates her werewolves, and it does form a sharp contrast with Anna's ability. I think Omegas could have been even more subtle than they are here, but the interplay between the conflict-driven approach of everyone else and Anna's inherent calm can be fun. The biggest problem I had with Cry Wolf, apart from the general predictability of the story and relationships, is that the hurt/comfort angle never felt adequately resolved. Hurting a character very badly and then having others offer comfort is a very common story technique that brings with it some reader expectations in terms of emotional payoff. The point at which the comfort finally works can be very empowering and very affecting. I don't think Briggs hit the payoff as well as she could have. Anna recovers, but she does so rather more easily than felt correct, and without the emotional punch that I was waiting for. It seems odd to complain about this sort of urban fantasy novel having too little angst, and maybe I was just hoping for a story other than the one I was reading, but I wanted a story that dug a bit deeper into Anna's psyche. I think the best word for Cry Wolf is okay. If this is the story you're in the mood for, it delivers that story. Briggs's werewolves are mildly interesting, and the Omega concept is a good one, although not as thoroughly or convincingly explored as I would have liked. I would have preferred more intensity in the hurt/comfort plot and a more difficult healing, but I still enjoyed the book. Followed by Hunting Ground. Rating: 6 out of 10

23 December 2012

Benjamin Mako Hill: The Cost of Collaboration for Code and Art

This post was written with Andr s Monroy-Hern ndez for the Follow the Crowd Research Blog. The post is a summary of a paper forthcoming in Computer-Supported Cooperative Work 2013. You read also read the full paper: The Cost of Collaboration for Code and Art: Evidence from Remixing. It is part of a series of papers I have written with Monroy-Hern ndez using data from Scratch. You can find the others on my academic website.
Does collaboration result in higher quality creative works than individuals working alone? Is working in groups better for functional works like code than for creative works like art? Although these questions lie at the heart of conversations about collaborative production on the Internet and peer production, it can be hard to find research settings where you can compare across both individual and group work and across both code and art. We set out to tackle these questions in the context of a very large remixing community.

Example of a remix in the Scratch online community, and the project it is based off. The orange arrows indicate pieces which were present in the original and reused in the remix.

Remixing platforms provide an ideal setting to answer these questions. Most support the sharing, and collaborative rating, of both individually and collaboratively authored creative works. They also frequently combine code with artistic media like sound and graphics. We know that that increased collaboration often leads to higher quality products. For example, studies of Wikipedia have suggested that vandalism is detected and removed within minutes, and that high quality articles in Wikipedia, by several measures, tend to be produced by more collaboration. That said, we also know that collaborative work is not always better for example, that brainstorming results in less good ideas when done in groups. We attempt to answer this broad question, asked many times before, in the context of remixing: Which is the better description, the wisdom of crowds or too many cooks spoil the broth ? That, fundamentally, forms our paper s first research question: Are remixes, on average, higher quality than single-authored works? A number of critics of peer production, and some fans, have suggested that mass collaboration on the Internet might work much better for certain kinds of works. The argument is that free software and Wikipedia can be built by a crowd because they are functional. But more creative works like music, a novel, or a drawing might benefit less, or even be hurt by, participation by a crowd. Our second research question tries to get at this possibility: Are code-intensive remixes, higher quality than media-intensive remixes? We try to answers to these questions using a detailed dataset from Scratch a large online remixing community where young people build, share, and collaborate on interactive animations and video games. The community was built to support users of the Scratch programming environment: a desktop application with functionality similar to Flash created by the Lifelong Kindergarten Group at the MIT Media Lab. Scratch is designed to allow users to build projects by integrating images, music, sound and other media with programming code. Scratch is used by more than a million, mostly young, users. Measuring quality is tricky and we acknowledge that there are many ways to do it. In the paper, we rely most heavily a measure of peer ratings in Scratch called loveits very similar to likes on Facebook. We find similar results with several other metrics and we control for the number of views a project receives. In answering our first research question, we find that remixes are, on average, rated as being of lower quality than works of single authorship. This finding was surprising to us but holds up across a number of alternative tests and robustness checks. In answering our second question, we find rough support for the common wisdom that remixing tends to be more effective for functional works than for artistic media. The more code-intensive a project is, on average, the closer the gap is between a remix and a work of single authorship. But the more media-intensive a project is, the bigger the gap. You can see the relationships that our model predicts in the graph below.

Two plots of estimated values for prototypical projects showing the predicted number of loveits using our estimates. In the left panel, the x-axis varies number of blocks while holding media intensity at the sample median. The right panel varies the number of media elements while holding the number of blocks at the sample median. Ranges for each are from 0 to the 90th percentile.

Both of us are supporters and advocates of remixing. As a result, we were initially a little troubled by our result in this paper. We think the finding suggests an important limit to the broadest claims of the benefit of collaboration in remixing and peer production. That said, we also reject the blind repetition of the mantra that collaboration is always better for every definition of better, and for every type of work. We think it s crucial to learn and understand the limitations and challenges associated with remixing and we re optimistic that this work can influence the design of social media and collaboration systems to help remixing and peer production thrive. For more, see our full paper, The Cost of Collaboration for Code and Art: Evidence from Remixing.

Russ Allbery: Review: Alpha and Omega

Review: Alpha and Omega, by Patricia Briggs
Series: Alpha and Omega #1
Publisher: Berkley
Copyright: 2007
ISBN: 1-101-41379-4
Format: Kindle
Pages: 112
Alpha and Omega is a novella set in Briggs's Mercy Thompson universe, an urban fantasy world that started with Moon Called. There's a limited edition hardcover version which is, at this point, ridiculously expensive, but it's available relatively inexpensively on the Kindle. (The page count in the sidebar may be high, since that's from the hardcover version; the Kindle version doesn't have page numbers.) It's the prequel novella to the novel Cry Wolf. You don't have to have read any of the Mercy Thompson novels to follow the story; it stands alone fairly well. I actually read Cry Wolf first, but (as I was warned) that proved to be a mistake, so I'm publishing my reviews in the "correct" reading order. Unusually for a niche novella leading into a regular series, Cry Wolf leans heavily on Alpha and Omega and works best if you've read it first. Anna is a werewolf, but not a very happy one. She's been a werewolf for three years, cutting herself off from everything before at the insistance of her pack, and she's at the bottom of the pecking order: abused, mistreated, and ordered about (although some of the abuse has gotten mildly better). That makes it startling that, when she sees a photograph in the paper of a missing teenager that she recognizes, she's willing to call the Marrok, the ruler of all wolves in North America, to report her pack's possible involvement. The Marrok sends his enforcer, Charles, to investigate. As soon as he arrives, he can tell that far more is wrong than anyone had realized. Anna's pack is completely dysfunctional, and Anna herself... well, that's the point of the story. Alpha and Omega has one of the more appealing qualities of novellas: a clear narrative drive that goes from point A to point B without many sidetracks or digressions. It's told in alternating tight third person perspectives, a traditional choice for this sort of story. Very little here will surprise an urban fantasy reader. It's the hurt/comfort story that one expects from the early going, but it's one that moves right along and hits most of the right notes. The biggest drawback for me was that I'm not horribly fond of Briggs's werewolf mythology. There are some parts that I like, such as Anna herself and the role that she discovers, but there's rather a lot of characters wrestling with their instinctive reactions or finding pack hierarchy enforced via means that appear to be magic. Given that the sort of absolute dominance hierarchy portrayed here is now considered dubious or refuted even in wild wolves, and given that I have a knee-jerk dislike of simplified, absolute structures, that part of Briggs's world-building makes my teeth itch. But, if you can get past that, and if you're not looking for any complex plotting (it is just a novella, after all), this isn't a bad way to spend an afternoon. It's comfort reading: things go roughly as you would expect them to go, and you can have the emotional satisfaction of seeing Anna's growth with very little risk. Followed by Cry Wolf. Rating: 7 out of 10

Cyril Brulebois: d-i hacking recipe #3: Debugging debconf

Problem #3: How to find out what s going on with debconf? Slightly longer version: As reported in #679327, the grub prompt during the installation process is entitled Configuring man-db , which isn t exactly great. Given man-db is a known dpkg trigger, I suspected this would be the reason for the broken title, and here s how I debugged it. For starters, using d-i basically means asking a lot of questions, which is implemented using the debconf mechanism. One should note that two implementations are available:
  1. the historic implementation, in Perl: debconf
  2. the C reimplementation: cdebconf
d-i uses cdebconf udebs, but joy begins when packages get installed in /target: debconf is used there. We ll see why that matters. Recipe #3: Set DEBCONF_DEBUG=developer and read logs!
  1. Both implementations support the DEBCONF_DEBUG environment variable. The installer supports passing that as a kernel parameter; example from the syslinux prompt: select the Graphical install entry, press Tab , remove -- quiet and put DEBCONF_DEBUG=developer there instead.
  2. Let s look at /var/log/syslog; there s a tail -f on it in VT4, but one can run grep and friends on it in VT2 or VT3. There, one can see what happens at the debconf level (the debconf protocol is documented in debconf-devel(7)). Tiny excerpt (timestamps removed for clarity):
    frontend: --> GET preseed/early_command
    frontend: <-- 0
     
    frontend: --> GET debian-installer/framebuffer
    frontend: <-- 0 true
     
    debconf: --> GET debconf/language
    debconf: <-- 0 en
    debconf: --> SET debconf/language en
    debconf: Setting debconf/language to en
    debconf: <-- 0 value set
    debconf: --> GET debconf/priority
    debconf: <-- 0 high
    debconf: --> GET rescue/enable
    debconf: <-- 0 false
    
  3. The interesting point was titles, right? So let s look at the last TITLE occurrences:
    debconf: --> SETTITLE debian-installer/main-menu-title
    debconf: --> SETTITLE debian-installer/grub-installer/title
    in-target: debconf (developer): ----> TITLE Configuration de man-db
    debconf: --> TITLE Configuration de man-db
    in-target: debconf (developer): <-- TITLE Configuration de man-db
    in-target: debconf (developer): ----> TITLE Configuration de man-db
    in-target: debconf (developer): ----> TITLE Configuration de grub-pc
    debconf: --> TITLE Configuration de grub-pc
    in-target: debconf (developer): <-- TITLE Configuration de grub-pc
    in-target: debconf (developer): ----> TITLE Configuration de grub-pc
    in-target: debconf (developer): ----> TITLE Configuration de man-db
    debconf: --> TITLE Configuration de man-db
    in-target: debconf (developer): <-- TITLE Configuration de man-db
    in-target: debconf (developer): ----> TITLE Configuration de man-db
    in-target: debconf (developer): ----> TITLE Configuration de grub-pc
    debconf: --> TITLE Configuration de grub-pc
    in-target: debconf (developer): <-- TITLE Configuration de grub-pc
    in-target: debconf (developer): ----> TITLE Configuration de grub-pc
    debconf: --> SETTITLE debian-installer/grub-installer/title
    
    Bad news: the last one is about grub, so the title should be correct, right?
  4. At this point, it might be the Gtk frontend behaving strangely. It s shipped in cdebconf-gtk-udeb, built from the cdebconf source. Let s look:
    • src/modules/frontend/gtk/ui.c has cdebconf_gtk_update_frontend_title(), which does what the function name suggests.
    • src/modules/frontend/gtk/di.c has cdebconf_gtk_di_run_dialog(), which calls the first function.
    • src/modules/frontend/gtk/progress.c has cdebconf_gtk_show_progress() (also calls the first function), which comes with the following comments:
      /** Show the progress widgets.
        *
        * This will actually add the widgets to the corresponding containers.
        * The main title saved when starting the PROGRESS operation will be restored
        * from the value saved when START was called.  This is needed when GO is
        * called during a PROGRESS operation.
      
    Now that s something! Even if the last entry is about grub, some previous value could be restored and could be the reason for the bad behaviour (that is easily confirmed by adding some debug_printf() calls on fe->title in cdebconf, thanks to an extra "debug.h" include). Maybe that s what needs fixing!
  5. Now it s time to take a step back. A d-i environment is nice, but testing things can easily become a burden. Taking a random wheezy or sid chroot, and installing/removing a tiny package shipping a manpage should be enough to run into the man-db trigger. Exporting DEBCONF_DEBUG=developer there too, let s either install or remove x11-apps (the trigger does the same job, and the debconf trace is the same), here s the output:
    Processing triggers for man-db ...
    debconf (developer): frontend started
    debconf (developer): frontend running, package name is man-db
    debconf (developer): starting /var/lib/dpkg/info/man-db.config configure /usr/share/man
    debconf (developer): <-- VERSION 2.0
    debconf (developer): --> 0 2.0
    debconf (developer): <-- INPUT medium man-db/install-setuid
    debconf (developer): --> 30 question skipped
    debconf (developer): <-- GO
    debconf (developer): --> 0 ok
    debconf (developer): starting /var/lib/dpkg/info/man-db.postinst triggered /usr/share/man
    debconf (developer): <-- VERSION 2.0
    debconf (developer): --> 0 2.0
    debconf (developer): <-- GET man-db/auto-update
    debconf (developer): --> 0 true
    
    Even if one didn t know about the cdebconf/debconf thing, the frontend started bit in the d-i syslog would have been the key: it s only found in debconf, in the frontend script. That one has several ways to determine the package being acted on, and it finally calls:
    debug developer => "frontend running, package name is $package";
    $frontend->default_title($package) if length $package;
    
    Tada, that s likely to be the issue.
  6. Tweaking that frontend script, it s easy to make sure that the code path for this use case is:
    elsif ($ARGV[0]=~m!^.*/(.*?)\.(?:postinst postrm prerm)$!)  
            $package=$1;
     
    
    Since $ARGV[1] is the action being passed to the maintainer script, one can intercept the triggered action, and avoid emitting a title update in this case; I ve therefore reassigned #679327 to the debconf package, and proposed a patch implementing that, which Joey Hess seems to find reasonable.
In my message to the bug report I wrote I actually tested it in a d-i environment, and made sure the issue had gone away. How I did that will likely be described in the next d-i hacking recipe.

22 December 2012

Dirk Eddelbuettel: Patricia Barber at Unity Temple

Forgot to blog about this (though I posted on Google+ about it), but we went to a neat little concert in the neighborhood on October 6. Chicago's own Patricia Barber came to the wonderful Unity Temple by Frank Lloyd Wright (more on the temple at Wikipedia and the foundation). This closed the 2012 series of the Unity Temple concerts and the setting was a little, well, weird. Look like mostly concert subscribers, with the commensurate age brackets, and not too many jazz folks. Not sure how many Patricia Barber, along with Larry Kohut on bass, won over, but I enjoyed it. I had secured front row (!!) seats in what is already an wonderfully intimate setting for such a duet concert. See the Google+ link for a YouTube recording by her.

Russ Allbery: Review: Regenesis

Review: Regenesis, by C.J. Cherryh
Series: Cyteen #2
Publisher: DAW
Copyright: January 2009
ISBN: 0-7564-0592-0
Format: Mass market
Pages: 682
Regenesis is a direct sequel to Cyteen, picking up very shortly after the end of that book and featuring all of the same characters. It would be absolutely pointless to read this book without first reading Cyteen; all of the emotional resonance and world-building that make Regensis work are done there, and you will almost certainly know whether you want to read it after reading the first book. Besides, Cyteen is one of the best SF novels ever written and not the novel to skip. Because this is such a direct sequel, it's impossible to provide a good description of Regenesis without spoiling at least characters and general plot developments from Cyteen. So stop reading here if you've not yet read the previous book. I've had this book for a while, and re-read Cyteen in anticipation of reading it, but I've been nervous about it. One of the best parts of Cyteen is that Cherryh didn't belabor the ending, and I wasn't sure what part of the plot could be reasonably extended. Making me more nervous was the back-cover text that framed the novel as an investigation of who actually killed the first Ari, a question that was fairly firmly in the past by the end of Cyteen and that neither I nor the characters had much interest in answering. Cyteen was also a magical blend of sympathetic characters, taut tension, complex plotting, and wonderful catharsis, the sort of lightning in a bottle that can rarely be caught twice. I need not have worried. If someone had told me that Regenesis was another 700 pages of my favorite section of Cyteen, I would have been dubious. But that's exactly what it is. And the characters only care about Ari's murderer because it comes up, fairly late in the novel, as a clue in another problem. Ari, and Justin, are back in the safe laboratory environment of Reseune, safe now that politics are not trying to kill or control them. Yanni has taken over administration. There is a general truce, and even some deeper agreement. Everyone can take a breath and relax, albeit with the presence of Justin's father Jordan as an ongoing irritant and hostile factor. But broader Union politics are not as stable: there is an election in progress for the Defense councilor that may break the tenuous majority in favor of Reseune and the Science Directorate, Yanni is working out a compromise to gain more support by turning a terraforming project loose on a remote world, and Ari isn't sure she likes that Eversnow idea at all. Then, just as things seem to be turning Reseune's way, politics and interpersonal relationships abruptly deteriorate, tensions with Jordan sharply worsen, and there may be moles in Reseune's iron-clad security. Navigating the crisis while keeping her chosen family safe will once again tax all of Ari's abilities. The third section of Cyteen, where Ari finally has the tools to take fate into her own hands and starts playing everyone off against each other, is one of my favorite sections of any book. If it was yours as well, Regenesis is another 700 pages of exactly that. As an extension and revisiting, it does lose a bit of immediacy and surprise from the original, and Regenesis is much less concerned with the larger questions of azi society, the nature of thought and personality, loyalty and authority, and the best model for the development of human civilization. It's somewhat less far-thinking and more of a political thriller. But it's a political thriller that recaptures much of the drama and tension of Cyteen and is full of exceptionally smart and paranoid people thinking through all angles of a problem, working fast on their feet, and successfully navigating tricky and treacherous political landscapes. And, like Cyteen but unlike others of Cherryh's novels I've read, it's a novel about empowerment, about seizing control of one's surroundings and effectively using all of the capability and leverage at one's fingertips. That gives it a thrilling catharsis that's almost as good as Cyteen. It's also, like its predecessor, a surprisingly authoritarian novel. I think it's in that, more than anything else in these books, that one sees the impact of the azi. Regenesis makes it clear that the story is set, not in a typical society, but inside a sort of corporation, with an essentially hierarchical governance structure. There are other SF novels set within corporations (Solitaire comes to mind), but normally they follow peons or at best mid-level personnel or field agents, or otherwise take the viewpoint of the employees or the exploited. When they follow the corporate leaders, the focus usually isn't down inside the organization, but out into the world, with the corporation as silent resources on which the protagonist can draw. Regenesis is much more of a balance. It's about decisions of the future of humanity that characters feel they can make undemocratically (in part because they or their predecessors have effectively engineered the opinions of the democratic population), but it's also about how one manages a top-down organization. And, most vitally, how one secures it. Reseune is, as in the previous novel, a paranoid's suspicions come true; everyone is out to get everyone else, or at least might be, and the level of omnipresent security and threat forces a close parsing of alliances and motivations that elevates loyalty to the greatest virtue. In Cyteen, we had long enough with Ari to see the basic shape of her personality and her slight divergences from her predecessor, but her actions are mostly driven by necessity. Regenesis gives us more of a picture of what she's like when her actions aren't forced, and here I think Cherryh manages a masterpiece of subtle characterization. Ari has diverged substantially from her predecessor without necessarily realizing it, but in ways that are firmly grounded in the differences she found or created between her life and the first Ari's. Specifically, she has friends, confidents, and a community, which combined with past trauma has made her fiercely, powerfully protective, and it's that protective instinct that weaves the plot together. It's probably the most significant way in which so much of the events of Cyteen and Regenesis are driven by people's varying reactions to trauma. If you, like me, loved the last third of Cyteen, read this, because Regenesis is more of exactly that. Cherryh finds new politics, new challenges, and a new and original plot within the same world and with the same characters, but it has the same feel of maneuvering, analysis, and decisive action. You will, as with Cyteen have to be comfortable with pages of internal monologue from really smart people thinking through all sides of a problem. If you didn't like that in the previous book, avoid this one; if you loved it, here's the sequel you didn't know you were waiting for. Rating: 9 out of 10

21 December 2012

Wouter Verhelst: BSP postmortem

This post is a bit late, but still interesting: Last weekend, I held a Bug Squashing Party at my company's offices in Mechlin, Belgium. This is the first time I've attended, let alone hosted, such an event; so I'm not that experienced in figuring out what I can and cannot do with other people's packages yet. As a result, our success rate was a bit lower than I'd hoped for. Still we closed two bugs, figured out that one more bug required just some binNMUs, that one should probably be tagged wheezy-ignore (as it was tagged squeeze-ignore too, and hasn't seen updates since then), and touched three more bugs. Having said that, I did have a bit of a hidden agenda, in that I've been wanting to build a stronger Debian community in Belgium; we are ranked fairly highly on the Debian Developer per capita list, but us Belgian DDs never meet up, in contrast to DDs from three of the four countries that surround Belgium. In that, I did have some success, too; some local people showed up who'd never (directly) contributed to Debian before. While I'm not silly enough to think that just showing up to a BSP once makes you suddenly an active member of a community, it's still a good first step. Unfortunately, however, I did not manage to get any other active Belgian DDs to show up. Apart from myself, only Dutch DD Joost Van Baal and DD Emeritus Joost Damad were present. If I can't find a way to improve on that, I'm not sure this BSPing will have a long life in Mechelen. At any rate, though, I did have a lot of fun doing this. Surely that, if nothing else, counts as "success".

20 December 2012

Russell Coker: Samsung Galaxy S3 First Review with Power Case

My new Samsung Galaxy S3 arrived a couple of days ago. Kogan added to the weirdness in their shipping that I documented in a previous post [1] by shipping the two phones on consecutive days with separate invoices for $789 each (as opposed to the $449 that I paid). This means that if customs were to track this down they could bust me for import duties (which apply whenever you order more than $1000 of electronics from another country) as multiple shipments in a small period of time are aggregated for tax purposes. It also means that it would be easy to defraud the tax office or an employer about the cost of the devices if I didn t blog about it Phone and Case Review back of power case with S3 installed next to back plastic of S3S3 and case next to each other To install the Samsung Galaxy S3 in the power case I bought from Kogan the back of the S3 has to be removed. One annoying implication of this is that when using the phone in it s case I will either have to carry the back of the phone with me or have no option of removing the case during the day. The phone has to slide in to the case from a small angle and if it s removed carelessly there seems to be a risk of breaking the power connector on the case. S3 on power case standedge view of S3 in power case stand The above photos show the kick-stand on the power case in use, I haven t yet had an opportunity to use the stand for anything other than photography and as it s made of thin plastic there s a reasonable chance that it will break off before I ever use it properly. One major deficiency of the power case is that it has no protection for the screen. The typical gel case will extend by about 2mm above the screen which means that you can place the phone screen-down on a table without a risk of scratching the screen. Also if a phone in a gel case is dropped on a hard surface it seems likely that the chance of the screen hitting a bump and cracking will be lower although I m not aware of any research regarding how well this theory works in practice. This combined with the fact that the power case is smooth and difficult to grip makes it quite unsuitable for regular use. I am concerned that the added mass of the power case would increase the risk of serious phone damage if it s dropped screen-down. I plan to buy a gel case ASAP which I will use most of the time. I will swap the gel case for the power case on the occasions when I expect to spend an extended period of time away from home or an office where I can charge my phone. The power case really should have been designed with a flange that extends above the level of the screen on all sites. It also shouldn t be smooth on the outside. Phone Review The Galaxy S3 equals the Galaxy S for the title of the most slippery phone I ve ever owned. This combined with the fact that the Galaxy S3 is slightly thinner and also 10% wider and longer makes it a more difficult phone to hold. It s a phone that really needs a case with a lot of grip. The camera is reasonable given the fact that any device that is 8.6mm thick will have a very small focal length and also a very small CCD for receiving light. The laws of physics just don t allow something the size of the Galaxy S3 to have a half-decent camera. My ideal mobile phone would be about the size of the Galaxy S3 with the power case, it would use the extra thickness for a big battery as well as a much longer focal length for the camera. I d like a phone to be about 16mm thick over most of it s area with a section that s 20mm for the camera. The burst shot mode (which allows you to take 20 pictures rapidly) is good for capturing certain types of relatively slow movement (EG someone s mouth moving when they talk). But it s no good at all if you try to pan the camera while doing burst mode. When I first started using the phone it gave me lots of hints about using various functions. This wasn t particularly useful for me as most things are obvious to anyone who has much experience with computers and the non-obvious things are probably going to be forgotten if you are doing something else when the help system prompts you. It will be useful for my sister in law who s getting the other phone, but the down-side of this is that I can t configure her phone before delivering it. She needs to use it first to see all the tips. The Galaxy S3 has a screen that s 4.8 diagonally which is slightly larger than the 4.0 screens of my previous Android phones (Sony Ericsson Xperia X10 and Samsung Galaxy S). That gives 44% more screen area which is a good thing. Unfortunately many apps seem to display the same amount of data, I had hoped that 44% more area and 140% more pixels would result in most apps displaying more data by default. Fortunately ConnectBot (my SSH client of choice) gives me 60 rows and 120 columns on the Galaxy S3 as opposed to the mere 31 rows and 80 columns on the Xperia X10. I don t use ConnectBot that often (a phone is a really bad ssh client no matter how you do it), but when I do it s really important and the extra space matters. The down-side of the larger screen for me is that the distance from the center of the palm of my hand to the end of my thumb is slightly less than 4.8 . This means that I can t hold the Galaxy S3 with one corner in the palm of my hand and still reach the far corner of the screen. So I am forced to hold the sides of the phone for one-handed use which means that it would be easier for it to slip out of my hand. A further complication is that the settings and back buttons are very close to the edge of the phone and can be hit by accident very easily. Hopefully a gel case will alleviate that. But I have never owned a hand-held device which was so difficult to hold in a stock configuration. According to AndroSensor (one of my favorite demo programs for showing useless things that a phone can do) the Galaxy S3 lacks support for measuring humidity and external temperature but it has all the other sensors. I think I can survive without knowing how hot and humid my pockets are, but it would still be nice to have every possible feature. The default partitioning of the phone storage is to have all internal storage used for applications and none used for a VFAT filesystem accessible over USB. It s nice that I won t run out of application storage space (a problem I ve had with every other Android device) but it does mean that I ll have to buy a micro SD card before I can use the phone properly. Micro SD storage costs about $1.50 per gig in the size range from 8G to 32G, it s not particularly expensive but it s annoying to have the phone not be fully usable right from the start. Without using a SD card I have to use gphoto2 to download pictures from the phone so it s fully usable in every way. But the cost of the power case, the gel case I m about to buy, and an SD card make the phone less of a bargain. The phone has more RAM, a faster CPU, and Android 4.1. These are all really good things, but nothing worth writing about in a review. Conclusions The power case is very poorly designed, I probably wouldn t buy it if I knew then what I know now. For one-handed phone use by someone like me with longer than average fingers a 4.0 phone is about the largest possible. This doesn t mean that phones should stay small, but it does mean that Apple weren t being entirely silly when they released all the 3.5 iPhones. Presumably a large portion of the population would find a 3.5 phone to be as usable as a 4.0 phone is for me. It now seems to me that as a 4.8 screen is going to be impractical for one-hand use by almost everyone the phone designers should concentrate more on larger phones similar to the Samsung Galaxy Note. I think that the 4.8 screen size may be ideal for some young children who could use it the way an adult might use a 7 tablet an Android phone seems to be a relatively common present for children in the 6-12 age range. Generally I think that no-one should ever buy a new phone or tablet without trying one of the same size. While I am happy with the S3 it really doesn t work in the way I expected. Having 2.9* as many characters on screen for ConnectBot is a massive benefit. If I ever write another magazine article on a hand-held device the extra size and resolution will really help if I found a suitable editor (please make suggestions in the comments). For the WordPress client to write long blog posts on it then the screen could potentially be reasonably useful, but so far it seems that it s more suited to reviewing posts and comments than to writing posts. Generally the Samsung Galaxy S3 is a good phone and I think it s great value for money when it s less than $500.

Russ Allbery: WebAuth 4.4.0

WebAuth is a site-wide web authentication system with single sign-on support. We use it extensively at Stanford for most of our user-facing web applications that need to authenticate people. This is the culmination of the last four months of work, so it's rather nice to finally get it out the door. Even if not everything I wanted to get done is done. That's always how it is with software releases. This release fixes a bug in the encoding of delegated Kerberos credentials when Heimdal is used as the Kerberos library. If you're building WebAuth against Heimdal, upgrade any mod_webauth modules that receive delegated credentials before upgrading mod_webkdc on the WebKDC. Otherwise, the flags of delegated tickets won't be sent correctly. The major new feature in this version is support for authorization identities separate from authentication identities. This can be used for testing, for administrators to proxy into an application as another user, or in other cases where there's a mismatch between the canonical concept of users and the capabilities of an application. The WebLogin server and WebKDC now support an ACL file that controls what alternative identities users can assert to specific web sites, and there is a new flow on the WebLogin side to set or change identities. All of this is disbled by default in each component. There is a new WebKDC directive to enable this support and specify the identity ACL file, and there is a new directive in mod_webauth to tell it to trust the authorization identity. There is a new environment variable that will be set to the authorization identity (trusted or not), and REMOTE_USER will be set to the authorization identity only if it is trusted. WEBAUTH_USER will always remain the authentication identity, so both identities can be logged and web applications can distinguish. To address one of the edge cases required by this support, there's a new mod_webkdc Apache directive, WebKdcLoginTimeLimit, which controls how long a multi-step login process can take, and also how recent a login has to be to contribute its authentication factors to the session factors for an authentication. This is also now used for WebAuthForceLogin, which means that forced logins won't require re-authentication if they're within the WebKdcLoginTimeLimit interval of the last login. This allows that feature to work properly in conjunction with authorization identities and with some multifactor authentication methods. Also in this release, optional replay detection and rate limiting of failed logins has been added to the WebLogin server. Either or both can be enabled in the configuration file. This support requires a memcached server (shared across any pool of WebLogin servers) be available to store the necessary state. There are new parameters in the error template to handle the error messages generated by these features. The WebLogin server also has a few more minor improvements: single sign-on cookies are now set even when displaying error pages if any are available, fixing some looping issues with some scenarios around restricting users to authenticate to specific sites, and the @REMUSER_REALMS setting has been broken into two settings so that its two properties can be changed independently. The old setting is still supported for backward compatibility. Multiple bugs in Kerberos ticket encoding have been fixed, some that were introduced in 4.3.0 and some that have been present since the first days of Heimdal support. Other fixes include the mapping of WebKDC error codes to names in WebLogin (which previously resulted in Perl warnings in the error log) and the missing documentation for the WebAuthRequireSSL directive. Finally, the thing that I'm the happiest about in this release is that I finished my complete refactoring of the libwebauth library. Most of the low-level interfaces are gone in favor of higher-level manipulation of WebAuth protocol objects. The last pieces of code that used the old token encoding system have been replaced with the new data-driven encoder. And the last parts of the library have been converted to APR, so there is no longer a mix of traditional memory management and APR pools. This comes with lots of simplification of the API and removal of old cruft from the public headers. You can get the latest release from the official WebAuth distribution site or from my WebAuth distribution pages.

Russ Allbery: rra-c-util 4.7

This release is a rollup of a variety of miscellaneous features and bug fixes that have accumulated since September. It takes the first few steps towards the general Perl coding style revision that I'm working on, but only a few. I've yet to move some generic Perl test frameworks into this package (hopefully soon). In the new feature department, there's a new portability wrapper around sys/statvfs.h that converts statvfs code to statfs code for older systems, and a lot of enhancements to the portable/apr.h header to fix up some more constants for APR 0.9 compatibility. In the bug fix department, the Kerberos and GSS-API probes now probe for Heimdal's libroken by looking for a function that it has provided since at least 0.4 instead of one that's new in 1.3, fixing some portability issues to BSD systems. The Kerberos header probes also now check for headers using file existence checks instead of the compiler in some cases to fix problems with detecting the wrong header when using one of several different Kerberos library installations on a system. In the test suite department, this release updates the POD and POD spelling checks to be generic and use more modern modules and techniques, and adds a new set of generic Perl tests that will be useful for any of my packages that contain Perl code. There's also a new valgrind suppression file for assisting with valgrind testing in packages that use Kerberos. You can get the latest version from the rra-c-util distribution page.

18 December 2012

Petter Reinholdtsen: Ledger - double-entry accounting using text based storage format

A few days ago I came across a blog post from Joey Hess describing ledger and hledger, a text based system for double-entry accounting. I found it interesting, as I am involved with several organizations where accounting is an issue, and I have not really become too friendly with the different web based systems we use. I find it hard to find what I look for in the menus and even harder try to get sensible data out of the systems. Ledger seem different. The accounting data is kept in text files that can be stored in a version control system, and there are at least five different implementations able to read the format. An example entry look like this, and is simple enough that it will be trivial to generate entries based on CVS files fetched from the bank:
2004-05-27 Book Store
      Expenses:Books                 $20.00
      Liabilities:Visa
The concept seemed interesting enough for me to check it out and look for others using it. I found blog posts from Christine Spang, Pete Keen, Andrew Cantino and Ronald Ip describing how they use it, as well as a post from Bradley M. Kuhn at the Software Freedom Conservancy. All seemed like good recommendations fitting my need. The ledger package is available in Debian Squeeze, while the hledger package only is available in Debian Sid. As I use Squeeze, ledger seemed the best choice to get started. To get some real data to test on, I wrote a web scraper for LODO, the accounting system used by the NUUG association, and started to play with the data set. I'm not really deeply into accounting, but I am able to get a simple balance and accounting status for example using the "ledger balance" command. But I will have to gather more experience before I know if the ledger way is a good fit for the organisations I am involved in.

Raphael Geissert: Nicer, but stricter

Lately I've been working on making the redirector nicer to the mirrors and to some potential users. More specifically, those behind a caching proxy.

The redirector is now nicer to traditional web proxies by redirecting to objects that are known not to change with a "moved permanently" code (HTTP status code 301.) This applies to files in the pool/ directory and ".pdiff" files, among others.
Previously, a traditional caching web proxy would sometimes end up with multiple copies of the same object, fetched from different mirrors; and the redirection would not be cached at all. With this change, this is no longer the case.

Using a caching proxy that is aware of the Debian repository design is still more likely to yield better results, however: If my memory serves correctly, apt-cacher has the ability of updating the Packages, Sources, and similar files with the ".pdiff"s on the server side. Apt-Cacher-NG apparently can use debdelta, and so on.
Check my blog post about one APT caching proxy not being efficient for some comments related to those tools.

Another recent change is that mirrors that can't be used by the redirector will no longer be monitored as often as the other mirrors. For instance, if a mirror doesn't generate a trace file (used for monitoring) then the redirector will gradually limit the rate at which the mirror is checked.
This rate-limiting mechanism applies to different kinds of errors, and should reduce the amount of wasted time and bandwidth while still allowing automatic-detection of mirrors that recover.


Projection of a rate-limited mirror over six weeks. The mirror would have to fail in every attempt for that to happen.
N.b. there's a bump in the scale.

The rate limiter applies an initial exception to allow temporary errors to not affect the use of the mirror by the redirector. After that exception, it is pretty much linear. However, that chart doesn't really reflect the effect of the rate limiter, so put in comparison with the normal checking behaviour:


Comparison of the two behaviours over an 8 weeks period using a logarithmic scale.
Nice chart colours by Libreoffice.

The code to detect mirrors that don't perform a two-stages sync that I talked about in a previous post has not yet been integrated as the current implementation would be too expensive on the mirrors to just add it as-is.

While tracking down problems exposed to users, I decided to take a stricter approach as to what mirrors are used by the redirector. Suffice to say that the remaining mirrors using the obsolete anonftpsync are going to be ignored entirely. ftpsync has been around for a few years now and it is the standard tool.
Whether you are mirroring Debian, Raspbian, Ubuntu, or any other Debian-like packages repository, ftpsync is the right tool to use.

Most of the issues I've been discovering, and sometimes working around, affect direct users of the mirrors and are not related to the http.debian.net redirector. When not detected beforehand they happen to be exposed by the redirector, but like I said, I plan to be stricter in order to increase the redirector's reliability. Once a strict and reliable foundation is built, more workarounds might see their way in to better use the available resources.

That's it for now. The road is long, the challenge is great, and being an observer in an uncontrolled environment makes it even more interesting.

Russ Allbery: Review: Liars and Outliers

Review: Liars and Outliers, by Bruce Schneier
Publisher: John Wiley
Copyright: 2012
ISBN: 1-118-14330-2
Format: Hardcover
Pages: 285
One of the perils of buying a book written by a blogger one reads regularly is that the book may be little more than a rehashing of their blog, with insufficient original material to warrant the time investment. Sometimes it's still nice to support them financially, but it may not make sense to read the book. I've been following Schneier's blog for years (as should anyone with an interest in security), including through the entire process of writing Liars and Outliers, and was a bit worried that might be the case here. Thankfully, I can reassure any other worried potential readers that is not the case. This is substantial new material establishing a firm framework for thinking about incentives and controls in any society or organization. Liars and Outliers talks about security mechanisms, but it's not, at its core, a book about security. Rather, it's a book about incentives, order, and how order is established. It's a comprehensive reductionist analysis of how societies create predictability and compliance to allow us to trust other people who we have never met before and will never meet again. It's a unique (at least in my experience) combination of anthropology, sociology, security analysis, and political science. Schneier cuts across fields in an idiosyncratic but illuminating way that reminded me of (on an entirely separate topic) Jane Jacobs. This is not a prescriptive book, nor is a collection of answers, solutions, or even deep analysis of particular problems. Rather, it's an attempt to construct a general framework for analyzing societal dilemmas: conflicts between individual desires and social good, how those conflicts are resolved, and how societies can weigh the scales and influence the statistical outcomes. The closest Schneier comes to telling the reader how to solve problems is a checklist, at the end of the book, for designing effective societal pressures. Its primary contribution is vocabulary and structure. It also passes one of my litmus tests for any book about human behavior: Schneier complicates, broadens, deepens, and expands understanding, and points out complex interactions and complex feedback in effects we're inclined to consider simple, rather than simplifying or eliminating human complexity. One point I found refreshing about this book is that Schneier is scrupulous in refusing to define either society or individuals as good or bad, to the point of carefully defining terminology used in all of the social dilemmas. Following societal rules is called compliance; not following those rules is called defection. In some cases, defection is morally correct (Schneier's most frequent example is in the civil rights struggles of the 1960s in the United States). In all cases, social pressures are tools, which can be used to encourage compliance with moral or immoral systems, and which are deployed by totalitarian dictatorships and utopian communes alike. Schneier explicitly puts out of scope for this book the questions of how societal goals should be determined, how they change, and whether any given societal rule or interest is moral or immoral. He focuses, rather, only on the mechanisms, with a primary goal of informing and deepening debates over how best to encourage behavior that societies want to encourage and discourage behavior societies want to discourage. He also emphasizes discourage, as opposed to eliminate. Early on, Schneier shows some of the results of game theory, as well as basic common sense, that indicate that no healthy society can totally eliminate defection. Not only would that stifle valuable and important reform, such as changes in civil rights, but the degree of pressure required is immense. Defectors are natural and will always exist, and are sometimes valuable and necessary. Rather, the goal of a society is to reduce defectors to a level where most people can ignore their existence most of the time, a state that leads to the level of risk and trust required to have a functioning and healthy society. The word society, similarly, is intentionally broad, and can refer to just about any collection of people, from a circle of friends to a corporation, institution, or country. However, as Schneier points out early in the book, small societies rarely need much in the way of formal pressure and appear almost magically self-governing. That's the first property that he disassembles, resulting in a general classification of societal pressures into four categories: moral, reputational, institutional, and security systems. The last is an odd category that's partly orthagonal to the other three. Moral pressure is internalized conscience: the normal tendency of nearly everyone to follow their own moral code, a code that's at least partly constructed and certainly heavily influenced by the surrounding society. Reputational pressure is, in a sense, externalized morality: it's the informal reactions of others around one to one's past actions. Included in reputational pressure is shunning of every kind, from cutting off a friendship to boycotts against corporations, but it also includes confrontation from another member of one's society, the more subtle effects of our individual desires to be liked and respected, and all the various aspects of "face", honor, and respect within a community. Small communities are frequently self-governing, in Schneier's model, because they're small enough that moral and reputational pressures are sufficient and no other pressures are required. We're so used to applying moral and reputational pressure to other humans almost unconsciously that we sometimes don't even notice its existence, leading to that "magical" self-governing property. But Schneier puts pressures in a sequence: pressures that work extremely well with small groups often don't scale. Moral pressure works best with small groups and reputational pressure with somewhat larger groups, but when societies scale beyond the limits of reputational pressure when, for example, one frequently interacts with people whose reputations are unknown to you and whose subsequent opinion of you will not be relevant institutional pressure is required to force compliance. Institutional pressure is the sort of pressure that we all tend to think of first when we look for ways to enforce rules: laws, policies, contracts, and other codes of behavior that carry with them formal punishments and some enforcement mechanism. But even in societies so large that institutional pressures are frequently required, such as whole countries, moral and reputational pressures still exist and are extremely important. One of Schneier's most interesting points is his analysis of how institutional pressures can paradoxically undermine reputational and moral pressures, resulting in more defection than if the institutional pressure hadn't existed. This is just the basic framework of Schneier's analysis, hopefully giving you a feel for the structure of the book. He goes much deeper into the complicated interactions between the various levels of pressure, and then dives into an extensive look at competing societal dilemmas: cases where there is more than one society in play simultaneously, possibly demanding contradictory actions. Liars and Outliers also includes a wonderful analysis of organizational entities within the same framework, including their much-different reactions to moral, reputational, and institutional pressures. One of the most cogent analyses of the difficulties of regulating both corporations and governmental institutions falls out of that analysis, once one looks at them in light of Schneier's basic framework. Pressures quickly become complex and multi-layered, and human reactions to pressures are frequently counter-intuitive. Schneier draws extensively on game theory to show that some counter-intuitive responses are actually emergent properties of logical analysis of the situation, but that others are more uniquely human and have little or nothing to do with a mathematical cost-benefit analysis. I haven't even mentioned his discussion of security systems, and how they can extend moral, reputational, and institutional pressures, as well as add a new type of pressure (making a defecting action impossible) that scales even better than institutional pressures. Liars and Outliers has all of the supporting infrastructure you would expect in a scholarly book: notes, extensive references, and a good index. I suspect it will end up being used as at least additional reading in college classes. The notes are, unfortunately, end notes, making the full context of the book much harder to read than was necessary, but at least Schneier does separate the notes from the references so that one doesn't chase notes for further explanation and find a simple citation. As with any book like this, one always wishes it could end in a simple prescription to fix everything, but of course it doesn't. But that's also a measure of a good scholarly work. Human and organizational motivations are complex and tricky, and any framework for analyzing them needs to be able to represent that complexity. Schneier here has constructed a very powerful one, one that I started using in discussions before I'd even finished the book. Perhaps the most valuable contribution of Liars and Outliers to public discussion is clear terminology and categories, which can be of great help in finding the core components of a problem. Liars and Outliers can be slow going, particularly early on when Schneier is still defining terms and setting up the background of his analysis. One can get a bit tired of the analysis matrices of societal dilemmas. But stick with it through the groundwork, since the analyses of competing societal dilemmas and of the impact of societal pressures on organizations are exceptional. Highly recommended, particularly for anyone who is designing or implementing societal pressures: managers, political activists, or anyone in a security-related field. Rating: 8 out of 10

Dirk Eddelbuettel: R / Finance 2013 Call for Papers

The text below just went out to r-sig-finance along with updates to the R/Finance website and its Call for Papers page.
Call for Papers: R/Finance 2013: Applied Finance with R
May 17 and 18, 2013
University of Illinois, Chicago, IL, USA
The fifth annual R/Finance conference for applied finance using R will be held on May 17 and 18, 2013 in Chicago, IL, USA at the University of Illinois at Chicago. The conference is expected to cover topics including portfolio management, time series analysis, advanced risk tools, high-performance computing, market microstructure, and econometrics. All will be discussed within the context of using R as a primary tool for financial risk management, portfolio construction, and trading. Over the past four years, R/Finance has included attendees from around the world. It featured presentations from prominent academics and practitioners, and we anticipate another exciting line-up for 2013. We invite you to submit complete papers in pdf format for consideration. We will also consider one-page abstracts (in txt or pdf format) although more complete papers are preferred. We welcome submissions for full talks and abbreviated "lightning talks". Both academic and practitioner proposals related to R are encouraged. Presenters are strongly encouraged to provide working R code to accompany the presentation/paper. Data sets should also be made public for the purposes of reproducibility (though we realize this may be limited due to contracts with data vendors). Preference may be given to presenters who have released R packages. The conference will award two (or more) $1000 prizes for best papers. A submission must be a full paper to be eligible for a best paper award. Extended abstracts, even if a full paper is provided by conference time, are not eligible for a best paper award. Financial assistance for travel and accommodation may be available to presenters at the discretion of the conference committee. Requests for assistance should be made at the time of submission. Please send submissions to: committee at RinFinance.com. The submission deadline is February 15, 2013. Submitters will be notified of acceptance via email by February 28, 2013. Notification of whether a presentation will be a long presentation or a lightning talk will also be made at that time. Additional details will be announced at this website as they become available. Information on previous year's presenters and their presentations are also at the conference website. For the program committee:
Gib Bassett, Peter Carl, Dirk Eddelbuettel, Brian Peterson,
Dale Rosenthal, Jeffrey Ryan, Joshua Ulrich
So see you in Chicago in May!

17 December 2012

Lars Wirzenius: Obnam 1.3 (backup software) and other releases

I've just pushed out the release files for Obnam version 1.3, my backup application, as well as Larch, my B-tree library, and cliapp, my Python framework for command line applications. They are available via my home page (http://liw.fi/). Since Debian is frozen, I am not uploading packages to Debian, but .deb files are available from my personal apt repository for the intrepid. (I will be uploading to Debian again after the freeze. I am afraid I'm too lazy to upload to experimental, or do backports. Help is welcome!) From the Obnam NEWS file: Bug fixes for Obnam: NEWS for Larch: NEWS for cliapp:

Ian Campbell: Foreign Chroots with schroot and qemu

Since I've been working on qcontrol I've had need of a way to build packages in an ARM environment. Since all my native ARM hardware is "production" (at least so far as something can be production in a home environment) I prefer to avoid installing development tools on them. When I started looking into this all my ARM systems ran from relatively small flash devices which also ruled out the possibility of a development chroot. After search the web a bit I discovered that it is possible to use qemu-user-static together with [[!binfmt-support ]] to create cross architecture chroots which can be entered mostly transparently. It is also possible to combine this with schroot and sbuild which makes the day to day use pretty much the same as a native or biarch chroot. Since I've now had to rediscover how to do this twice (once for Sid and then again when I realized I also needed a Wheezy environment) I thought I should write it down for the benefit of my future self. First things first lets install the various packages which we need. As well as debootstrap to create the chroot we need qemu-user-static and binfmt-support to allow us to run ARM binaries:
$ sudo apt-get install debootstrap qemu-user-static binfmt-support
I'm using schroot's lvm-snapshot mode so we need an LVM volume, formatted with a filesystem and mounted in a temporary location:
$ sudo lvcreate -L8G -n sbuild-wheezy-armel /dev/VG
$ sudo mkfs.ext3 /dev/VG/sbuild-wheezy-armel
$ sudo mount /dev/VG/sbuild-wheezy-armel /mnt/
Next we need to create the chroot. For this we use debootstrap in foreign mode which does a 2 stage setup, firstly by using the --foreign option:
$ sudo debootstrap --variant=buildd --include=fakeroot,build-essential \
        --arch=armel --foreign \
        wheezy /mnt http://<DEBIAN-MIRROR>/debian/
Now we (temporarily) copy the qemu-arm-static binary into the chroot (at the place where binfmt-support expects it), change into the chroot and run debootstrap's second stage to finish the basic chroot setup:
$ sudo cp /usr/bin/qemu-arm-static /mnt/usr/bin/
$ sudo chroot /mnt
(chroot)# ./debootstrap/debootstrap --second-stage
(chroot)# exit
Now that we have a basic chroot in place it needs a little bit of tailoring. I've found the easiest way to do this is to use sbuild-createchroot and then tweak the result:
$ sudo sbuild-createchroot --arch=armel --foreign --setup-only \
    wheezy /mnt http://<DEBIAN-MIRROR>/debian/
If we were building a native chroot then we could have skipped the debootstrap stuff and let sbuild-createchroot handle things. Unfortunately this doesn't seem to play nicely with --foreign. Now we can remove the qemu-arm-static binary from the chroot. schroot will automatically bind mount the version from the host into the chroot which ensures it remains up to date. We can also unmount the chroot since we will let schroot manage that from here on:
$ sudo rm /mnt/usr/bin/qemu-arm-static
$ sudo umount /mnt
Next we need to tweak the resulting schroot/sbuild configuration since sbuild-createchroot assumes the use of schroot's directory mode. First remove the symlink to /mnt which it created:
$ sudo rm /etc/sbuild/chroot/wheezy-armel-sbuild
Lastly we need to edit /etc/schroot/chroot.d/wheezy-armel-sbuild-<SUFFIX> (where is randomly generated) and reconfigure it to use lvm-snapshot mode, by apply the following changes:
 [wheezy-armel-sbuild]
-type=directory
-directory=/mnt
+type=lvm-snapshot
+device=/dev/VG/sbuild-wheezy-armel
+lvm-snapshot-options=--size 2G
 description=Debian wheezy/armel autobuilder
 groups=root,sbuild
 root-groups=root,sbuild
 profile=sbuild
That's it, we now have a chroot which we can trivially invoke via sbuild to build armel packages:
$ sbuild --arch=armel --dist=wheezy <... other options...>
We can also use sbuild-update to keep the chroot up to date:
$ sudo sbuild-update -udcar wheezy-armel
You can also use schroot to login to the chroot directly, although for that case you'll likely want to create a different chroot using schroot's directory or block-device mode instead lvm-snapshot and drop some of the sbuild specific setup. I expect I'll blog about this configuration in some more detail at some point in the future since I'll want to document it to give a more convenient build environment for Xen on ARM than the build farm which we have at work.

Dominique Dumont: Be wary of optimised for devices

Hello This story began with linphone having a weird behavior: every now and then, the mouse pointer would become stuck: the pointer moves, but clikcking on it always activate the same widget. Since this always happened in the middle of phone conference done for work, this was infuriating. Long story short, I finally found that my plantronics USB headset was responsible for this weird behavior. This headset is connected to the USB bus and is seen by the computer as a USB sound card. The bug was triggered by pressing the mute button provided by the headset. Suspicious of this usb device, I used lsusb to find the features provided by the heaset: $ lsusb -v -s 1:3 2>&1 grep InterfaceClass
bInterfaceClass 1 Audio
bInterfaceClass 1 Audio
bInterfaceClass 1 Audio
bInterfaceClass 1 Audio
bInterfaceClass 1 Audio
bInterfaceClass 3 Human Interface Device
Audio devices were expected, but why a HID device ? This device is a headset, not a keyboard or a mouse Testing further, I also saw some number poping up on my screen whenever I pressed the the mute button. The only way to get the mouse back was to unplug the headset. A quick google search gave a solution to setup X11 to ignore the headset. Problem solved. But this did not answer the question regarding the HID device. Then I got a hint in the form of a sticker glued on the headset USB plug: Optimized for Microsoft Lync . This page gave the answer: an optimised for Lync device provides mute/unmute across PC and device . I can only guess that when the mute button is pressed, some data is sent from the HID interface. Unfortunately, the window manager does not like to be on the receiving end of this data. The moral of this story is: optimized for something actually means not standard .
ok. There s a bug somewhere. The comments of this blog have convinced me that I went too far with the moral of this story. It s now overstriked instead of plainly remvoed so the comments still make sense. Thanks everybody for the constructive comments.
All the best
Tagged: wtf

Next.

Previous.