We currently use completely different methods and tools of building test beds and running tests for Debian vs. Click packages, for normal uploads vs. CI airline landings vs. upstream project merge proposal testing, and keep lots of knowledge about Click package test metadata external and not easily accessible/discoverable. Today I released autopkgtest 3.0 (and 3.0.1 with a few minor updates) which is a major milestone in unifying how we run package tests both locally and in production CI. The goals of this are:

• Keep all test metadata, such as test dependencies, commands to run the test etc., in the project/package source itself instead of external. We have had that for a long time for Debian packages with DEP-8 and debian/tests/control, but not yet for Ubuntu s Click packages.
• Use the same tools for Debian and Click packages to simplify what developers have to know about and to reduce the amount of test infrastructure code to maintain.
• Use the exact same testbeds and test runners in production CI than what developers use locally, so that you can reproduce and investigate failures.
• Re-use the existing autopkgtest capabilities for using various kinds of testbeds, and conversely, making all new testbed types immediately available to all package formats.
• Stop putting tests into the Ubuntu archive as packages (such as mediaplayer-app-autopilot). This just adds packaging and archive space overhead and also makes updating tests a lot harder and taking longer than it should.

So, let s dive into the new features! New runner: adt-virt-ssh We want to run tests on real hardware such as a laptop of a particular brand with a particular graphics card, or an Ubuntu phone. We also want to restructure our current CI machinery to run tests on a real OpenStack cloud and gradually get rid of our hand-maintained QA lab with its test machines. While these use cases seem rather different, they both have in common that there is an already existing machine which is pretty much only accessible with ssh. Once you have an ssh connection, they look pretty much the same, you just need different initial setup (like fiddling with adb, calling nova boot, etc.) to prepare them. So the new adt-virt-ssh runner factorizes all the common bits such as communicating with adt-run, auto-detecting sudo availability, doing SSH connection sharing etc., and delegates the target specific bits to a setup script . E. g. we could specify --setup-script ssh-setup-nova or --setup-script ssh-setup-adb which would then get called with open at the appropriate time by adt-run; it calls the nova commands to create a VM, or run a few adb commands to install/start ssh and install the public key. Then autopkgtest does its thing, and eventually calls the script with cleanup again. The actual protocol is a bit more involved (see manpage), but that s the general idea. autopkgtest now ships readymade scripts for these two use cases. So you could e. g. run the libpng tests in a temporary cloud VM:

# if you don't have one, create it with "nova keypair-create"
$ nova keypair-list
[...]
  pitti   9f:31:cf:78:50:4f:42:04:7a:87:d7:2a:75:5e:46:56  
# find a suitable image
$ nova image-list 
[...]
  ca2e362c-62c9-4c0d-82a6-5d6a37fcb251   Ubuntu Server 14.04 LTS (amd64 20140607.1) - Partner Image                           ACTIVE    
$ nova flavor-list 
[...]
  100   standard.xsmall    1024        10     10                 1       1.0           N/A        
# now run the tests: please be patient, this takes a few mins!
$ adt-run libpng --setup-commands="apt-get update" --- ssh -s /usr/share/autopkgtest/ssh-setup/nova -- \
   -f standard.xsmall -i ca2e362c-62c9-4c0d-82a6-5d6a37fcb251 -k pitti
[...]
adt-run [16:23:16]: test build:  - - - - - - - - - - results - - - - - - - - - -
build                PASS
adt-run: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ tests done.

Please see man adt-virt-ssh for details how to use it and how to write setup scripts. There is also a commented /usr/share/autopkgtest/ssh-setup/SKELETON template for writing your own for your use cases. You can also not use any setup script and just specify user and host name as options, but please remember that the ssh runner cannot clean up after itself, so never use this on important machines which you can t reset/reinstall! Test dependency installation without apt/root Ubuntu phones with system images have a read-only file system where you can t install test dependencies with apt. A similar case is using the null runner without root. When apt-get install is not available, autopkgtest now has a reduced fallback mode: it downloads the required test dependencies, unpacks them into a temporary directory, and runs the tests with $PATH, $PYTHONPATH, $GI_TYPELIB_PATH, etc. pointing to the unpacked temp dir. Of course this only works for packages which are relocatable in that way, i. e. libraries, Python modules, or command line tools; it will totally fail for things which look for config files, plugins etc. in hardcoded directory paths. But it s good enough for the purposes of Click package testing such as installing autopilot, libautopilot-qt etc. Click package support autopkgtest now recognizes click source directories and *.click package arguments, and introduces a new test metadata specification syntax in a click package manifest. This is similar in spirit and capabilities to DEP-8 debian/tests/control, except that it s using JSON:

    "x-test":  
        "unit": "tests/unittests",
        "smoke":  
            "path": "tests/smoketest",
            "depends": ["shunit2", "moreutils"],
            "restrictions": ["allow-stderr"]
         ,
        "another":  
            "command": "echo hello > /tmp/world.txt"
         
     

For convenience, there is also some magic to make running autopilot tests particularly simple. E. g. our existing click packages usually specify something like

    "x-test":  
        "autopilot": "ubuntu_calculator_app"
     

which is enough to do what I mean , i. e. implicitly add the autopilot test depends and run autopilot with the specified test module name. You can specify your own dependencies and/or commands, and restrictions etc., of course. So with this, and the previous support for non-apt test dependencies and the ssh runner, we can put all this together to run the tests for e. g. the Ubuntu calculator app on the phone:

$ bzr branch lp:ubuntu-calculator-app
# built straight from that branch; TODO: where is the official" download URL?
$ wget http://people.canonical.com/~pitti/tmp/com.ubuntu.calculator_1.3.283_all.click
$ adt-run ubuntu-calculator-app/ com.ubuntu.calculator_1.3.283_all.click --- \
      ssh -s /usr/share/autopkgtest/ssh-setup/adb
[..]
Traceback (most recent call last):
  File "/tmp/adt-run.KfY5bG/tree/tests/autopilot/ubuntu_calculator_app/tests/test_simple_page.py", line 93, in test_divide_with_infinity_length_result_number
    self._assert_result("0.33333333")
  File "/tmp/adt-run.KfY5bG/tree/tests/autopilot/ubuntu_calculator_app/tests/test_simple_page.py", line 63, in _assert_result
    self.main_view.get_result, Eventually(Equals(expected_result)))
  File "/usr/lib/python3/dist-packages/testtools/testcase.py", line 406, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: After 10.0 seconds test failed: '0.33333333' != '0.3'
Ran 33 tests in 295.586s
FAILED (failures=1)

Note that the current adb ssh setup script deals with some things like applying the autopilot click AppArmor hooks and disabling screen dimming, but it does not do the first-time setup (connecting to network, doing the gesture intro) and unlocking the screen. These are still on the TODO list, but I need to find out how to do these properly. Help appreciated! Click app tests in schroot/containers But, that s not the only thing you can do! autopkgtest has all these other runners, so why not try and run them in a schroot or container? To emulate the environment of an Ubuntu Touch session I wrote a --setup-commands script:

adt-run --setup-commands /usr/share/autopkgtest/setup-commands/ubuntu-touch-session \
    ubuntu-calculator-app/ com.ubuntu.calculator_1.3.283_all.click --- schroot utopic

This will actually work in the sense of running (and succeeding) the autopilot tests, but it will fail due to a lot of libust[11345/11358]: Error: Error opening shm /lttng-ust-wait... warnings on stderr. I don t know what these mean, just that I also see them on the phone itself occasionally. I also wrote another setup-commands script which emulates read-only apt , so that you can test the unpack only fallback. So you could prepare a container with click and the App framework preinstalled (so that it doesn t always take ages to install them), starting from a standard adt-build-lxc container:

$ sudo lxc-clone -o adt-utopic -n click
$ sudo lxc-start -n click
  # run "sudo apt-get install click ubuntu-sdk-libs ubuntu-app-launch-tools" there
  # then "sudo powerdown"
# current apparmor profile doesn't allow remounting something read-only
$ echo "lxc.aa_profile = unconfined"   sudo tee -a /var/lib/lxc/click/config

Now that container has enough stuff preinstalled to be reasonably fast to set up, and the remaining test dependencies (mostly autopilot) work fine with the unpack/$*_PATH fallback:

$ adt-run --setup-commands /usr/share/autopkgtest/setup-commands/ubuntu-touch-session \
          --setup-commands /usr/share/autopkgtest/setup-commands/ro-apt \
          ubuntu-calculator-app/ com.ubuntu.calculator_1.3.283_all.click \
          --- lxc -es click

This will successfully run all the tests, and provided you have apt-cacher-ng installed, it only takes a few seconds to set up. This might be a nice thing to do on merge proposals, if you don t have an actual phone at hand, or don t want to clutter it up. autopkgtest 3.0.1 will be available in Utopic tomorrow (through autosyncs). If you can t wait to try it out, download it from my people.c.c page . Feedback appreciated!

LAVA has arrived in Debian unstable. No need for third party repositories (unless you want to), a simple apt install lava-server. What happens from that point isn t so simple. I ve made the single instance installation as straightforward as I could but there is a lot more to do to get LAVA working in a useful manner. First, a little history to explain some of the hysterical raisins which may appear later. Validation So, you ve made a change to the code, good. It compiles, yay! Before you ship it, does it pass the unit tests? Right, now you have an improved patch which does what you want and keeps the unit tests running. Validation is all about asking that awkward question: does your change work? Does it introduce side effects on other systems / services in the same code? Does it break programs which use services exported by the code? LAVA is one step along the road to system testing, starting at the bottom. Automation Well you could do all that yourself. Write the image to the device yourself, apply power yourself, connect to serial, copy test scripts to the booted image and pull the results off, somehow. Much better if this is automated maybe every commit or every push or as often as the tests can be run on the number of devices available. LAVA Linaro Automated Validation Architecture. Linaro is a not-for-profit building the future of the Linux kernel on ARM devices. Lava was built for and by Linaro, so the initial focus is clearly on validating the Linux kernel on ARM devices. There are likely to be gotchas in the code for those wanting to use LAVA for other kernels Linaro can t support lots of different kernels, but if there are changes which make it easier to use other kernels without impacting on validation of Linux, those would likely be accepted. The experience with using LAVA is all with ARM but if there is interest in extending LAVA to work with devices of other architectures, again, patches would be welcome. The development of the packaging to make LAVA suitable for Debian has also meant that LAVA will run on hardware other than x86. e.g. armv7.com. I m running a mini lab at home based around an arndale board, I ve also got a mini lab at work based on a cubie2. Requirements for such setups are described in the documentation and in previous entries on this blog (principally you will need SATA, lots of RAM and as many CPU cores as you can find. If you want to run LAVA on other architectures, go ahead and let us know if there are issues. Available versions The versions uploaded to Debian will (from now on) be production releases. The same code as is running on http://validation.linaro.org/ development builds and test builds are supported using helpers in the lava-dev package. Releases to unstable will automatically migrate into Ubuntu Unicorn. I ll continue building interim versions at the former locations on http://people.linaro.org/~neil.williams/, including builds for Ubuntu Trusty 14.04LTS as well as providing packages for Jessie until the uwsgi package can migrate. LAVA is looking to work with anyone who can prepare and maintain packages for other distributions like Fedora. Bugs LAVA is migrating from Launchpad bugs to http://bugs.linaro.org which is a bugzilla interface. Now that LAVA is also in Debian, anyone is welcome to use the Debian BTS which does not require any login or account setup. The maintainers (including me) will then forward those bugs as appropriate. Documentation The immediate task is to update the documentation in the lava-server-doc package (and the Debian wiki) to reflect the availability of LAVA from Debian unstable and how to choose which release of LAVA to use in your systems. However, there is a large amount of documentation already available click the Help link in the menu bar of any current LAVA instance. As with many projects, the docs have been written by the development team. If there are things which are unclear or if sections need to be moved around to make it easier for people new to LAVA to pick it up, please file bugs. Hardware It isn t easy to run a LAVA lab, there is a lot more to it than simply installing lava-server. LAVA would not be possible without the lab team and any other LAVA instance is going to need the same level of excellence in system administration, device support and cooperation. I ve covered a little bit of that in previous entries on this blog about my home lab but that is nothing compared to the work required to provide a working lab like the one in Cambridge. Access to that lab is restricted to people within Linaro but LAVA is also reaching out to the community and if there are tests or hardware you want to see within a LAVA instance, not necessarily in the main lab, then talk to us. Contact Bug reports are preferable but you can also find LAVA people on #linaro-lava on OFTC or contact us on the linaro-validation mailing list. Future There is a lot more work to do on LAVA yet. There are assumptions about partition layout within images (hysterical raisins) and issues with unused software being required for remote worker installations. Both are now part of the next major development work within LAVA. DebConf14 There is lot more to talk about with LAVA if you are attending DebConf14 then there will be talks on LAVA and plenty of time to answer questions.

John Scalzi wrote an insightful post about the utility of blog comments with the way the Internet works nowadays [1]. He starts out focusing on hate comments that could reasonably be described as terrorism (death threats with the aim of preventing people writing about politics meet any reasonable definition of terrorism ). Terrorists on the Internet are a significant problem but it s one that doesn t get much attention as it generally only affects people who aren t straight-acting white men. Blogging About Technology One corner case that John doesn t seem to consider is that of writing about technology. Issues related to programming often aren t related to politics and are often testable so comments will be based on things that have been shown to work rather than stuff people invent or want to believe. I ve received many useful and educational comments on my technical posts with little hostility. Even getting a snarky comment is rare when writing a strictly technical blog post. The comments problem for technology blogging is spam. I ve been using the WordPress plugin Block Spam by Math [2] (which is obsolete but still works) for years. Initially it stopped almost all spam, but now I m getting at least 20 spam comments a day. Extremely Popular Blogs The comments section of a blog is sometimes described as a conversation . When a blog post gets comments from less than 10 people it is possible for them to have something that resembles a conversation with the author that is of benefit to other readers and doesn t take excessive amounts of time for the author. When a blog is very popular and every post gets comments from 50+ people it s not really possible. So a traditional blog comment section seems to work best when the blog is primarily read by a small well connected group of people who sometimes comment and some casual readers who never comment (but sometimes find value in the comments of others). Discussions of blog comment systems usually include a reference to a post written by someone who disabled comments on their blog and found it to be a good thing, it always seems that the person who writes such a post has a large and varied audience who s comments would take a lot of time to moderate. John followed the usual form in this regard by linking to a reasonably popular SF author who would presumably have a lot of fans with good net access. I m not going to criticise anyone for disabling comments when their blog becomes really popular, but any advice that they have to offer about such things won t apply to the vast majority of blogs. Due to the long-tail effect the small blogs would probably comprise the majority of all comments so in terms of the way the blog environment works I don t think it makes much difference when the small minority of very popular blogs disable comments. The vast majority of blogs that I regularly read only have a small number of comments. One thing that should be noted is that getting a lot of readers shouldn t be the only factor for writing a successful blog. For example some of my blog posts about SE Linux are aimed at a small audience of Linux programmers and have an even smaller number of people who are qualified to comment. When I write a post that can only receive comments other than please explain more because I don t understand from a few dozen people that doesn t make it any less important. Sometimes the few dozen people who know a topic well need to work together to educate the few thousand who can implement the ideas for the benefit of millions of users of the software. Disabling Comments on Contentious Posts One interesting method John uses is to disable comments early when posting about contentious issues. It s a general practice when running a blog to disable comments on posts after a certain period of time (3 months to 1 year seem to be common time limits for comments). This means that the moderators can concentrate on recent posts and not be bothered with spam bots hitting ancient posts as the interest in writing legitimate comments on an old post is vanishingly small. John has a practice of disabling comments after a couple of days when the comments start to lose quality. No matter how contentious the issue is I m not likely to get the 400+ comments that John gets. But the idea of closing comments quickly still has some merit for my blog and other blogs with less traffic. Not Accepting Comments While Asleep John has a practice of closing comments while he s asleep to avoid allowing a troll to get 8 hours of viewing for a nasty comment. The most immediate down-side to that is that it inconveniences people who don t want to wait 8 hours to comment and prioritises comments from people in the same time zone, this makes me think of Cory Doctorow s novel Eastern Standard Tribe (which is available for free download and I highly recommend reading it) [3]. It seems that a better solution to that problem would be to have a team of moderators to watch things 24*7 which is what a lot of popular blogs that allow comments do. The WordPress capabilities model doesn t support granting a user no special privileges other than moderating comments [4], as WordPress is the most popular self-hosted blog software this limits the possibilities for people moderating comments on other people s blogs. No variation of this would work for me. I have lots of things that require my ongoing attention and don t want to add my blog to the list. If I have other things to work on for a few days I want to just not bother with my blog. This means that my blog needs to be able to run on autopilot for days at a time however I do monitor my blog closely after publishing a post that is likely to attract nasty comments. One extra problem that I have is that the Android client for WordPress has problems in synchronising comments. Using a Forum for Comments Popular Planet installations such as Planet Debian and Planet Linux Australia syndicate more than a few blogs that have comments disabled. A forum installation for such a Planet would be useful to allow people to comment on all posts and also support bloggers who are thinking of disabling comments. While the use of a forum for blog comments has been proven to work well for Boing Boing forums have their own issues of spam and anti-social behavior. Debian already has a forum [5], if a section of that was devoted to discussing blog posts from Planet Debian then it shouldn t make much of an increase to the work of the forum administrators while providing a benefit to the community. Also if the Debian forum had such a section it would probably attract use from more Debian Developers, I would use that forum if it was a place to comment on blogs that don t have a comment section and I might also comment on other forum discussions. It would be good if there was a forum for discussing Linux in Australia. I m not volunteering to run it but I would help out if someone else wants to be the main sysadmin and I can offer free hosting.

• [1] http://whatever.scalzi.com/2013/08/28/various-thoughts-on-online-comments/
• [2] http://wordpress.org/plugins/block-spam-by-math/
• [3] http://craphound.com/est/?page_id=1574
• [4] http://codex.wordpress.org/Roles_and_Capabilities
• [5] http://forums.debian.net/

Russ Albery wrote an insightful blog post about trust, computer security, and training programmers [1]. He makes a good case that social problems in our community decrease the availability of skilled people to write and audit security code. The Lawfare blog has an insightful article by Dan Geer about Heartbleed as a Metaphor [2]. He makes some good points about security and design, ways of potentially solving some flaws and problems with the various solutions. Eben Moglen wrote an insightful article for The Guardian about the way that the NSA spying is a direct threat to democracy [3] The TED blog has an interesting interview with Kitra Cahana about her work living with and photographing nomads in the US [4]. I was surprised to learn that there s an active nomad community in the US based on the culture that started in the Great Depression. Apparently people are using Youtube to learn about nomad culture before joining. Dave Johnson wrote an interesting Salon article about why CEOs make 300* as much money as workers [5]. Note that actually contributing to the financial success of the company is not one of the reasons. Maia Szalavitz wrote an interesting Slate article about Autism and Anorexia [6]. Apparently some people on the Autism Spectrum are mis-diagnosed with Anorexia due to food intolerance. Groups of four professors have applied for the job of president and vice-chancellor of the University of Alberta [7]. While it was a joke to apply in that way, 1/4 of the university president s salary is greater than the salary of a professor and the university would get a team of 4 people to do the job so it would really make sense to hire them. Of course the university could just pay a more reasonable salary for the president and hire an extra 3 professors. But the same argument applies for lots of highly paid jobs. Is a CEO who gets paid $10M per annim really going to do a better job than a team of 100 people who are paid $100K? Joel on Software wrote an insightful article explaining why hiring 1/200 applicants doesn t mean you hire the top 0.5% of workers [8]. He suggests that the best employees almost never apply through regular channels so an intern program is the only way to get a chance of hiring the best people. Chaotic Idealism has an interesting article on some of the bogus claims about autism and violence [9]. Salon has an interesting articleby Lindsay Abrams about the way the food industry in the US lobbies for laws to prevent employees from reporting animal cruelty or contamination of the food supply and how drones will now be used for investigative journalism [10]. Jacobin Mag has an interesting article by Geoff Shullenberger about the Voluntariat , the people who volunteer their time to help commercial organisations [11]. I don t object to people voluntarily helping companies, but when they are exploited or when the company also requires voluntary help from the government it becomes a problem. We need some legislation about this. Laura Hudson wrote an insightful article about how Riot Games solved their online abuse problem [12]. There are ideas in this that can apply to all online communities. Matt LeMay wrote an interesting article for Medium titled What (Else) Can Men Do? Grow The Fuck Up [13]. It s a general commentary on the treatment of women in geek communities (and most other places). Foz Meadows wrote an insightful analysis of the attempts of bigots to influence science-fiction [14]. If I had more spare time I d read some of the books by bigoted authors on the Sad Puppy Slate (from a library of course) and see if they lack talent in the same way that Orson Scott Card does. Racialicious has an interesting article by Phenderson Djeli Clark about the horrible racism and bigotry of H.P. Lovecraft [15]. I have only read two HP Lovecraft stories, one was mediocre and the other (The Horroe at Red Hook) was quite poor largely due to his inability to write about non-white people. Grace Wyler wrote an insightful article for Vice magazine about the right-wing terrorists in the US killing cops [16]. Paul Rosenberg wrote an interesting and amusing (for people outside the US) article about the gun crazies in the US [17]. Maybe the gun crazies should have a loaded assault rifles and tequila party to follow up on their gun appreciation day . A US TV show made a 4 minute clip of some of the stupid things that Tony Abbott has done [18]. Tony is almost as stupid as Dubya.

• [1] http://www.eyrie.org/~eagle/journal/2014-04/003.html
• [2] http://www.lawfareblog.com/2014/04/heartbleed-as-metaphor/
• [3] http://tinyurl.com/kcocf7o
• [4] http://tinyurl.com/kysyz77
• [5] http://tinyurl.com/qyt9qf8
• [6] http://tinyurl.com/m95dv8q
• [7] http://tinyurl.com/ka27qyg
• [8] http://www.joelonsoftware.com/items/2005/01/27.html
• [9] http://chaoticidealism.livejournal.com/134561.html
• [10] http://tinyurl.com/nv3ygzm
• [11] https://www.jacobinmag.com/2014/05/the-rise-of-the-voluntariat/
• [12] http://www.wired.com/2014/05/fighting-online-harassment/
• [13] http://tinyurl.com/phrq937
• [14] http://tinyurl.com/kye63mw
• [15] http://tinyurl.com/me2aj5a
• [16] http://www.vice.com/read/whats-behind-all-the-right-wing-cop-shootings
• [17] http://tinyurl.com/pynqmls
• [18] http://tinyurl.com/khkav88

Review: The Knowledge, by Lewis Dartnell

Publisher:	Penguin
Copyright:	2014
ISBN:	0-698-15165-8
Format:	Kindle
Pages:	328

The cover pitch for The Knowledge is that technological civilization has collapsed. A war, an asteroid, a pandemic, or some other catastrophic event has ended life as we know it. Survivors have recovered enough to stabilize the population, but all the industry and infrastructure that supports the modern world has been crippled and the knowledge lost. What do they need to know to rebuild the world? (The subtitle, How to Rebuild Our World from Scratch, is deceptive. Dartnell spends a lot of time talking about what could be scavenged from the ruins after an event like a pandemic that leaves buildings and tools intact.) This brief description may bring several incorrect ideas to mind, so a few clarifications are in order. First, this is not a book about how to survive an apocalypse, or how to stabilize society afterwards. Dartnell skips over anything related to basic survival or politics, and this book would only be of limited interest to preppers. Rather, it is focused on the key concepts and key processes that underlie much of what makes up our civilization, and how those processes can be bootstrapped and recreated, hopefully much more quickly than the thousands of years they took to develop the first time around. Second, as much as I think Dartnell would have liked it to be, The Knowledge is not an instruction manual. There's just too much involved in modern machinery, metallurgy, chemistry, and other necessary science and engineering. What it is instead is a tour of the most important ideas, basic processes, and approaches that can lead to creation of an industrial economy. This means it's also a tour of the most critical technology and discoveries that we rely on for day-to-day living. Each idea presented here is a bare sketch, and would require extensive research and experimentation to put into practice. The Knowledge is not a scientific encyclopedia. Rather, it's a collection of signposts and suggested directions along with some suggestions about sequencing, with special emphasis on the ideas that were unintuitive or surprising, or which took humans much longer to stumble across than would have been necessary. I will say up front that I am quite dubious about the stated goal of this book. I'm not sure if it would be of significant assistance to people who found themselves in the circumstances that Dartnell postulates, and I'm also not sure it would be as difficult as Dartnell supposes to reconstruct these details from a good library, although the point is well-taken that most modern industrial techniques would be unavailable to a civilization starting from scratch and many historic techniques are no longer practiced and therefore not well-documented. But this book does not have to succeed at its stated goal to be worth reading. The Knowledge provides a fascinating insight into the way our current capabilities developed, other paths that development could have followed, and what building blocks are vital for the industry that we rely on. If you are like me, the first thing The Knowledge will remind you of is Minecraft, particularly with some of the mods (such as TerraFirmaCraft) that add more realistic detail to the technological bootstrapping. It's a treasure trove for anyone who might be working on that sort of mod or game. Dartnell tries to cover as much of the spectrum of technology as possible, from farming through medicine, from basic materials to industrial chemistry, and from electrical power through transportation. Each chapter necessarily covers only the highlights, the most vital technologies and the early priorities, and many of those are relatively vague pointers. But that's enough for a fascinating orientation. It's also full of memorable trivia: the huge variety of uses to which wood can be put, the difficulty in re-establishing production of industrial acids, the shelf life of gasoline, the global seed vault within the Arctic Circle, and the importance of a long-threaded screw for a lathe, just to name a few examples. In short, The Knowledge may or may not be a practical aid in rebuilding the world, but it's a lot of fun to read and full of tidbits that are fun to think about or pass on to others. It's also a great orientation in applied technology. The Knowledge is particularly fascinating if you have any interest in the simplified recreation of technological and industrial processes, such as in Minecraft and similar games. For authors of Minecraft mods who are aiming for realism, it might even be required reading. I'd love to see more mods based heavily on the approaches described in this book. The Knowledge also conveys, better than any book or course I've seen, an intuitive feel for the amazing complexity and ingenuity underlying industrial technology. I pre-ordered a copy of this book as soon as I heard about it, and I was not disappointed. Despite my certainty that I would be one of the early casualties in any general social collapse, I thoroughly enjoyed reading it, and would like to read it again at some point. Recommended, albeit with the caveat that I do not have the background required to verify any of the facts in this book, and am assuming that Dartnell has done his research properly. Rating: 9 out of 10

As previously mentioned I've been working on git-based DNS hosting for a while now. The site was launched for real last Sunday, and since that time I've received enough paying customers to cover all the costs, which is nice. Today the site was "relaunched", by which I mean almost nothing has changed, except the site looks completely different - since the templates/pages/content is all wrapped up in Bootstrap now, rather than my ropy home-made table-based layout. This week coming I'll be slowly making some of the implementation available as free-software, having made a start by publishing CGI::Application::Plugin::AB - a module designed for very simple A/B testing. I don't think there will be too much interest in most of the code, but one piece I'm reasonably happy with is my webhook-receiver. Webhooks are at the core of how my service is implemented:

• You create a repository to host your DNS records.
• You configure a webhook to be invoked when pushing to that repository.
• The webhook will then receive updates, and magically update your DNS.

Because the webhook must respond quickly, otherwise github/bitbucket/whatever will believe you've timed out and report an error, you can't do much work on the back-end. Instead I've written a component that listens for incoming HTTP POSTS, parses the body to determine which repository that came from, and then enqueues the data for later processing. A different process will be constantly polling the job-queue (which in my case is Redis, but could be beanstalkd, or similar. Hell even use MySQL if you're a masochist) and actually do the necessary magic. Most of the webhook processor is trivial, but handling different services (github, bitbucket, etc) while pretending they're all the same is hard. So my little toy-service will be released next week and might be useful to others. ObRandom: New pricing unveiled for users of a single zone - which is a case I never imagined I'd need to cover. Yay for A/B testing :)

Jim has been busy as part of the procps-ng team that looks after top. Basically all the changes you find in top from around 2.7 or so are by him. Not satisfied enough with fixing top, making it faster and showing more fields, he has given us CPU and memory graphs. He also thinks I don t have enough colours (or as he would put it colors) on my top output so I ve posted what the new top looks like for me so you can see the graphs and he can see my colours. I think it is both colourful and useful addition. The colours have been available for a while now and the graphs will appear in the next upstream release of procps-ng.

DNSSEC is one of those things that have been on the I should poke at that sometime list, but today I decided to actually get in and play around with it. Turns out it s pretty trivially easy to setup, but I got bitten by a few gotchas during testing, that I thought worthwhile to document. Since I like to explain things, a lot of this article isn t how to so much as how does .

Configuring The Resolver First off, you need a resolver¹ that you can trust, and which implements DNSSEC validation. In theory, you could use your ISP s resolvers (if they re DNSSEC-enabled), or even Google s public resolvers (8.8.8.8 and 8.8.4.4), but it s really not recommended how much do you trust those, really? More importantly, how much do you trust the network path between them and you? Frankly, running a local on-host caching resolver is trivial, consumes next-to-no resources, and is highly recommended. It s what I did. I used unbound, finally casting free the venerable dnscache, which has served me well for many years. I chose unbound because it fits the do one small thing, and do it well philosophy that I cherish (unlike BIND), and because it s what all the cool kids appear to be using these days. Installing it was as simple as apt-get install unbound it s available in pretty much every distro, I believe. Out of the box, unbound in Debian wheezy comes configured for DNSSEC validation. Other methods of installation might not be so amenable, so just make sure your unbound.conf liiks like this, so that DNSSEC is enabled:

server:
    auto-trust-anchor-file: "/var/lib/unbound/root.key"
    val-log-level: 2

The val-log-level line isn t strictly required, but I found it useful to have unbound log info on validation failures to syslog. Season to taste. You also have to configure your system to actually use your new, shiny local resolver. Editing /etc/resolv.conf would, in an ideal world, work Just Fine, but between Magical Unhelpful Pixies (I m looking at you, NetworkManager) and DHCP, it often isn t that easy. Ultimately, though, you should configure your system so that the only resolver in /etc/resolv.conf is always 127.0.0.1. Anything else is a security hole waiting to happen. If you re using DHCP and a recent version of ISC dhclient, adding supersede domain-name-servers 127.0.0.1; to dhclient.conf should do the needful.

Weighing the (Trust) Anchor Whatever local resolver you choose, you need to tell it about trust anchors . This is a key (or keys) which are trusted to sign all the other data that the resolver will find. Like the rest of the DNS, DNSSEC is hierarchical, and you need to know who to start with before you can start. To resolve a name using DNS, you ask the root servers for the answer to a question (say, what is the address of www.example.com? ). The root server doesn t know the answer, but it knows who to ask, and it tells you where to go (typically, go ask the DNS servers for .com, which are X, Y, and Z ). Then you ask those servers, and they probably don t know, but they know who to ask, and they tell you to go ask those servers. This continues until you finally get a server which says, oh, yes, I know the answer to that! and hands over the information you re after. In DNSSEC, you get digital signatures that prove that the answers you get from each DNS server you ask are legit, and not forged. So when you ask the root servers, what is the address of www.example.com? , they answer, I don t know, but go ask DNS servers at addresses X, Y, and Z and by the way, you can trust this answer is legit because here s this signature you can check . To verify a signature is legit, you need to know that the key used to make the signature is trustworthy, and it wasn t just any ol key that made it similar to how an ink scribble on a page doesn t mean anything unless you know what the signature you re looking at should look like. DNSSEC has two ways to make sure you know who to trust. Firstly, those trust anchors I mentioned three paragraphs ago. These are the keys that the root servers use to sign the responses they send out. You tell unbound (or your local DNSSEC-enabled resolver of choice), if you see a signature provided by this trust anchor key, you can trust it to be legit . The second way to trust is key is to have someone you already trust say, oh, you can trust this key, too . This is done by providing more data in the answer that you get from the root servers. So the full answer you get from the root servers when you ask, what is the address of www.example.com? looks like this:

I don t know, but you should ask the DNS servers at addresses X, Y, and Z. You can trust their answers because they ll sign their responses with key K. Finally, you can trust that I m legit because of this signature I m providing you, generated from key T.

Assuming that T is the trust-anchor key you ve already told unbound to trust, it can then trust all of the information it got from that first question. When it then goes to DNS server X and asks, Hey, do you know the address of www.example.com? , that server can send back its own response saying:

Gee, shucks, I don t know, but DNS servers at P and Q can help you. You can trust them because they ll sign responses with key S. And just to show you I m on the level, check out this nifty signature I made, signed with key K.

Since a server we trust told us to trust key K, and then we got a response signed by key K that told us to trust key S, when we then go and ask server P and get a response signed by S, we re still demonstrably secure. Neat setup, right? Anyway, now you know all about trust anchors. How do you set it up? For unbound, it s capable of priming the trust anchor itself. It does this by retrieving a copy of the trust anchor from [a well-known HTTPS URL]((https://data.iana.org/root-anchors/root-anchors.xml), which provides a reasonable measure of security the trust anchor file should be safe as long as that HTTPS server is trustworthy. For the properly paranoid, you should determine an alternate means of verifying that you have the correct root key (a web search for DNSSEC root key attestation would probably be a good start).

Testing It s always handy to know that something s working as you expect. For something that s important for security, that goes double. So, how do you test that your DNSSEC-capable resolver is, indeed, working as designed? Well, firstly, you can test that the resolver, by itself, is working. This is simple: do a query for a zone known to have broken DNSSEC. Thankfully, some nice people (Comcast, specifically) have already set this up. You can test a resolver for DNSSEC-correctness with this command:

dig @127.0.0.1 www.dnssec-failed.org

If dig gives you back an IP address, something is broken. The correct, my DNSSEC works! response should be a status: SERVFAIL response. That s all well and good, but having a resolver that works isn t much good if nothing s actually using it. Your system configuration has probably got some other resolvers configured into it (such as those provided by a DHCP server), and you ll want to make sure those aren t getting used². The trivial test is to visit http://www.dnssec-failed.org in your browser. If that comes back with a page, then something s still wrong. A more user-friendly test of your DNSSEC configuration is provided by http://dnssectest.sidnlabs.nl/test.php, which will display a page either way, and give you a tick or a cross depending on whether you re DNSSEC-enabled or not. One thing to be wary of when you re doing browser tests is caching. I spent far too long trying to work out why unbound wasn t working , when in actual fact it was working fine, it s just that both my browser and local caching proxy (squid) were caching DNS requests³. So, even after I d correctly setup unbound, visiting the test sites was providing me with erroneous results. Fixing up the browser was simple enough. In Chromium, visiting chrome://net-internals#dns will show you what s being cached, what resolv.conf settings are being used, and allow you to clear the browser s resolver cache. Squid was similarly simple, although it took me a while to figure out that it was causing me grief (software that just works , being so rare, tends to slip your mind, and so I often forget that squid is actually doing me good). Restarting squid has the appropriate result (clears the DNS cache), but in the end I decided that having another layer of caching wasn t going to help significantly when I ve got an on-host DNS cache already, in unbound. So, I just turned the knobs for DNS caching in squid right down:

negative_dns_ttl 1 second
positive_dns_ttl 1 second

And voila! All was well.

In summary DNSSEC is cool. It s easy to get going with, and over time it will provide more and more practical benefits (I am a DANE fanboi). Get started with it now, and you ll be ahead of the curve.

---

1. A resolver is a program which takes a name, does all the DNS queries required to get the data associated with that name, and return it to whoever wanted to know. This can be a library which links to a client application, or it can be a separate program which receives DNS requests itself. There are usually both types of resolvers in use for a typical Linux system to resolve names. On every machine, in every program, there is the so-called stub resolver , which is part of libc, and does nothing except create DNS requests and send them to the addresses listed in resolv.conf. Also present, although not always running on every machine (typically your ISP or network admin has some running for everyone to share) are one or more recursive resolvers , which are separate programs which take DNS requests, do all of the queries themselves, and return the result to the stub resolver for presentation to the client program which made the request in the first place. You will see this material again.
2. You can configure dhclient to use your local resolver instead of the DHCP-provided ones by simply setting:

   supersede domain-name-servers 127.0.0.1;
   

   in your dhclient.conf file.
3. It also didn t help that I hadn t correctly knobbled dhclient the first time around, and so even when I thought I was only resolving against my local unbound server, it was still actually using the DHCP-provided resolvers

One which ends in tears, I m afraid. A week or so ago, I got an MacBook Air 13 (MacBook Air 6-2), to take with me when I head out to local hack sessions, and when I travel out of state for short lengths of time. My current Thinkpad T520i is an amazing machine, and will remain my daily driver for a while to come. After getting it, I unboxed the machine, and put in a Debian install USB key. I wiped OSX (without even booting into it) and put Debian on the drive. To my dismay, it didn t boot up after install. Using the recovery mode of the disk, I chrooted in and attempted an upgrade (to ensure I had an up-to-date GRUB). I couldn t dist-upgrade, the terminal went white. After C-c ing the terminal, I saw something about systemd s sysvinit shim, so I manually did a purge of sysvinit and install of systemd. I hear this has been resolved now. Thanks :) My machine still wasn t booting, so I checked around. Turns out I needed to copy over the GRUB EFI blob to a different location to allow it to boot. After doing that, I could boot Debian whilst holding Alt. After booting, things went great! Until I shut my lid. After that point, my screen was either 0% bright (absolutely dark) or 100% (facemeltingly bright). I saw mba6x_bl, which claims to fix it, and has many happy users. If you re in a similar suggestion, I highly suggest looking at it. Unfortunately, this caused my machine to black out on boot and I couldn t see the screen at all anymore. A rescue disk later, and I was back. Annoyingly, the bootup noise s volume is stored in NVRAM, which gets written out by OSX when it shuts down. After consulting the inimitableMatthew Garrett, I was told the NVRAM can be hacked about with by mounting efivarfs on /sys/firmware/efi/efivars type efivarfs. Win! After hitting enter, I got a kernel panic and some notes about hardware assertions failing. That s when I returned my MacBook and got a Dell XPS 13.
This is a problem, folks. If I can t figure out how to make this work, how can we expect our users? Such a shame the most hardware on the planet gets no support, locking it s users into even less freedom.

So, I have a SEN account (it's part of the PSN), I have 2 videos with SEN, I have a broken PS3 so I can no deactivate video (you can only do that from the console itself, yes, really)... and the response from SEN has been abysmal, specifically:

As we take the security of SEN accounts very seriously, we are unable to provide support on this matter by e-mail as we will need you to answer some security questions before we can investigate this further. We need you to phone us in order to verify your account details because we're not allowed to verify details via e-mail.

I mean, seriously, they're going to verify my details over the phone better than over e-mail how exactly? All the contact details are tied to my e-mail account, I have logged in to their control panel and renamed the broken PS3 to "Broken PS3", I have given them the serial number of the PS3, and yet they insist that I need to call them, because apparently they're fucking stupid. I'm damned glad that I only ever got 2 videos from SEN, both of which I own on DVD now anyways, this kind of idiotic tie in to a system is badly wrong. So, you phone the number... and now you get stuck with hold music for ever... oh, yeah, great customer service here guys. I mean, seriously, WTF. OK - 10 minutes on the phone, and still being told "One of our advisors will be with you shortly". I get the feeling that I'll just be writing off the 2 videos that I no longer have access to. I'm damned glad that I didn't decide to buy more content from that - at least you can reset the games entitlement once every six months without jumping through all these hoops (you have to reactivate each console that you still want to use, but hey).

Review: The Emergency Sasquatch Ordinance, by Kevin Underhill

Publisher:	American Bar Association
Copyright:	2013
ISBN:	1-62722-269-3
Format:	Trade paperback
Pages:	334

First, if you have not read Lowering the Bar, you should do so. There aren't many blogs where I will start reading, end up reading numerous posts out loud to anyone in the vicinity, and then find myself systematically reading the entire backlog (which, in this case, is substantial). Kevin Underhill is a lawyer with civil libertarian leanings, a wicked sense of humor, and a knack for finding the most absurd legal stories. He varies between weary incredulity and caustic sarcasm, almost always manages to make me smile, and often makes me laugh out loud. The Emergency Sasquatch Ordinance, and Other Real Laws that Human Beings Have Actually Dreamed Up, Enacted, and Sometimes Even Enforced is, as one might expect, more of the same sort of thing that one gets from Lowering the Bar. However, unlike a lot of books released by long-standing bloggers, it is not a collection of previously published material. This is an entirely new collection of legal absurdities, drawing on history, US federal, state, and local law, and some non-US laws, and complete with Underhill's trademark commentary. Long-time readers of Lowering the Bar have seen several posts about this sort of thing before, which also means there's a great sample up on the web. See posts tagged "Law (Dumb)" for a preview of the sort of thing you'll get, although as mentioned the material in the book is original. My one complaint about the book version of this post type is that I think the blog posts tend to be longer and contain a bit more analysis. Underhill gets in about one punch line per law in the book, and I frequently wished for a somewhat longer analysis. The blog posts tend to get funnier and more biting once he works up a head of steam on a topic. The book is, alas, a bit uniform in its style. It's composed of a large number of "chapters," each of which is usually a page or two and covers a single law. Those are divided into ancient, pre-modern, US federal, US state, US city, and non-US laws, but with considerably funnier section headings than those. Each chapter has a title that's usually a brief summary of the law (often funny), an introduction, a quote (or several), and a punch line. There's not much variation, which makes it good bathroom reading you can pick it up anywhere, read a few pages, and put it back down without fear of losing context but lacks variety when read in large gulps. I find Lowering the Bar itself funnier than this book. I think that's because the blog posts are longer, more varied, and mix weird laws with caustic political commentary and entertaining stories of the extremely ill-advised things people do around, within, and against the law. The odd laws are entertaining; the stories of defendants, the accounts of weird legal proceedings, and Underhill's exasperated sarcasm about legal and political stupidities are even funnier. If you had to choose between the book and the blog, I'd go with the blog. That said, you don't have to choose, and if you've been reading the blog for a while, more material is always welcome. A book also has the substantial advantage of being a way to throw some money at a blogger as a reward for hours of entertainment, with a bonus of getting an enjoyable physical object in return. I certainly don't regret the purchase, and I've already had an opportunity to mention Millard Fillmore's State of the Union address on guano. I'm not sure if that means the book made me smarter, but I do believe it made me more entertaining. TL,DR version: Read Lowering the Bar. If you like that, there's this book. I think you can figure out the rest from there. Rating: 7 out of 10

While cooperatives fortnight is mostly a celebration of how well cooperatives are doing in the UK, this year is tinged with sadness for me because it sees Downham Food Coop stop trading. This Friday and Saturday will be their last market stall, 9til 1 on the Town Square, aka Clock or Pump square. As you can see, the downturn has hit the market hard and I guess being the last stall left outside the market square (see picture: it used to have neighbouring stalls!) was just too much. The coop cites shortage of volunteers and trading downturn as reasons for closure. But if you re near Downham today or tomorrow morning, please take advantage of this last chance to buy some great products in West Norfolk!

Review: Due Diligence, by David Roodman

Publisher:	Center for Global Development
Copyright:	2012
ISBN:	1-933286-48-2
Format:	Trade paperback
Pages:	335

Due Diligence: An Impertinent Inquiry into Microfinance is a detailed analysis of the effects, philosophy, approaches, and merits of microfinance as an institution and as, increasingly, a charitable focus of the developed world. The author, David Roodman, was a senior fellow at the Center for Global Development at the time of writing this book, and is an expert in microfinance and economic development. This book starts as an overview and a history of the spectrum of microfinance institutions (something I found very valuable, since I knew very little about microfinance prior to Kiva's popularity), and then attempts a critical, data-driven analysis of whether, to what extent, and by which measures microfinance is successful. The most common aspect of microfinance, made popular and prominent by Kiva among others, is microcredit: small loans to people in developing countries. The underlying theory of microcredit is that lack of financial services handicaps prospective entrepreneurs and business owners in the third world, preventing them from growing businesses, funding expansion, and moving out of poverty. By extending relatively small amounts of credit to these people, or so the theory goes, they can better smooth out the uncertainties of day-to-day life and get the necessary capital to expand small local business. Microcredit is not the only type of microfinance, though, and Roodman's careful discussion of that point was one of the most informative parts of this book for me. It can also come in the form of microsavings tiny bank accounts or small-scale savings plans or microinsurance. Roodman makes a very interesting argument that these three forms of microfinance form two opposing continuums. Microcredit is the easiest to provide and microinsurance is the hardest, for many reasons he discusses including some that were surprising to me, but microinsurance is arguably the most helpful and beneficial to the individual poor and microcredit is the least. The opening history and survey does an excellent job showing the range of techniques and approaches used in microfinance, and showing the challenges of providing it. I had some vague understanding of the logistical challenges in serving areas far away from traditional infrastructure. I had much less understanding of the fascinating array of techniques in use, both inside and outside what's become the microfinance industry. Some of these techniques are brilliant examples of human economic creativity and skill at balancing incentives with risk protection. There are a much wider variety of approaches and instruments available than the individual or group loans that one typically sees on a site like Kiva, all with different trade-offs in group size, flexibility, rigidity, risk, and complexity. This book is well worth reading just to get a feel for the huge array of financial arrangements possible outside of the traditional banking structures that most readers of this review are used to. Another question Roodman answers thoroughly is why microfinance is so focused on microcredit. My vague unease with that was strengthened by reading David Graeber's Debt, which among many other things makes clear some of the social perils and power imbalances in lending. Due Diligence both crystallized that discomfort and explained why alternatives are difficult. Both microsavings and microinsurance necessarily involve considerably more regulation (which has both advantages and complexity), since they involve taking and safeguarding people's money and promising service in the future. Lenders require less trust; lenders instead extend trust and assume that risk themselves, and are in a position to do so. Microinsurance is also harder to explain and harder to target, although it is potentially the most helpful financial instrument when well-targeted. And microcredit can absorb huge influxes of capital from the outside, lending it out in a region, while both microsavings and microinsurance are necessarily built on the money already in the local economy and are inherently more resistant to external involvement. (I'm of the opinion this is a significant feature, but it makes it far harder for them to catch on as a charitable trend.) The heart of the book is a hard look at whether microfinance, and particularly microcredit, works. Roodman attempts this analysis for three different definitions of working: lifting people out of poverty (the most commonly claimed success), giving people more freedom and control over their lives even if their poverty level does not change in absolute terms, and developing the local economy. Sadly for charities like Kiva, microcredit does surprisingly poorly on the first two metrics. There is still little clear data, and the degree to which data from one region is applicable to a different region is highly debatable, but what evidence we have does not support the theory that microcredit lifts people out of poverty. In fact, the more intense the microcredit, the more likely it appears to be that credit pathologies common in the developed world, such as taking out loans from one organization to pay off loans from another, spirals of debt that is unlikely to ever be repaid, and aggressive debt collection methods, turn up in microcredit as well. The vision of the entrepreneur who succeeds through new-found access to capital is largely a myth. Indeed, the current packaging and presentation of loans to distant charities in developed countries means that even the supposed purpose of the loans is dubious. Loans are almost always presented as being for some sort of business activity, since that's what external charities want to fund, but money is fungible and the funds are more often used for all the things people normally need to use funds for: weddings, funerals, food for the family in hard times, paying off other loans to relatives, and other daily expenses. On the empowerment front, microcredit fairs little better, and here I particularly appreciated Roodman's clear-eyed and honest portrayal of common microcredit practices and their benefits for the creditors. For example, microcredit makes much of loaning to women, and presents this as a story of female empowerment. Roodman argues that there is some reality behind that, but it's also clear that microcredit organizations loan to women because women are more likely to repay not due to any inherent gender trait, but because women are easier to intimidate and shame into repaying. Similarly, the much-vaunted group loans, usually presented as a way for the community to support each other and teach each other financial skills, are most common in the poorest regions because they represent an outsourcing of costs. The main risk to the lender is that poor people have no such thing as credit histories, and detailed investigations of someone's trustworthiness is expensive. Groups are jointly liable and therefore will police themselves and check each other's reliability at no cost to the lender. Groups will also use intimidation tactics and even violence against members who don't or can't repay their share, without the lender having to get involved or have any responsibility for those tactics. Roodman takes a philosophical tone on this topic, noting several times that services for the very poor are generally of low quality and finance is no exception. I'm less inclined to be philosophical; after reading this book, I find it very hard to support group loans. This leaves economic development as the most compelling argument left for microcredit. This is weak tea for people like myself who wanted to believe in the transformative power of microcredit, but Roodman does a solid job explaining why it's still important. There is a great deal to be said for the perspective that only industrialization actually lifts people out of poverty on any substantial scale, and therefore the most useful type of aid is aid that somehow contributes towards industrialization. Building a native financial industry could arguably be such a step. And there seems to be some real merit and lasting effect to having a richer and more capable set of local financial institutions. However, here, it's not at all clear that external money helps. In fact, there's quite a bit of evidence that external money hurts, since it pushes microfinance institutions to find ways of using that money and pleasing external donors instead of building their infrastructure and capabilities as financial institutions. The earlier institutions, such as the Grameen Bank, seem to have done a better job at being local institutions woven into the local economy than have the later organizations that are largely supported by western donors. Roodman makes a good case for the usefulness of microfinance as a part of economic development, but it comes with a strong argument against too much external funding, and particularly against the practice of developed world donors funding specific loans. I came away from this book feeling like I had a much better understanding of the core problems and techniques of financial services for the poor, and with a strong desire to see more microsavings than microcredit. The two are closer in nature than I had expected, and Roodman's analysis of the similarities was fascinating, but microsavings seems less vulnerable to some of the credit pathologies. There is also some statistical evidence that microsavings does provide some modest reductions in poverty, while the evidence for microcredit is much less compelling. Sadly, from the perspective of a resident of the developed world such as myself who would very much like to help, neither microsavings nor effective microcredit seems likely to benefit from my contributions, and particularly not in the form of a site like Kiva. The end of this book is a long appendix that reprints multiple blog posts on related topics, including additional discussion of the problems with the Kiva model. Effectively, Kiva is a way for people like myself to subsidize microcredit organizations. It's not at all clear that subsidies are needed, and there's some reason to believe that they encourage negative and destructive behavior. The way they are marketed involves a certain amount of lying, or at least distortion, on the part of multiple actors in the microcredit picture. And they're a distraction from microsavings, and from microcredit based on loaning out savings from within the same local economy, both of which seem to have considerably more positive properties. Due Diligence is a book with a somewhat narrow target audience. It will appeal primarily to people with an interest in economics, particularly development economics, or who are evaluating microfinance charities or analyzing their support of groups like Kiva. But within that context, this is an excellent book, and I recommend it to your attention. Roodman is clear, thorough, and fair within my ability to measure that, and while no one book can be the final word on a topic, he adds considerable data and thoughtful analysis. If you are a supporter of Kiva, like I was, I fear you'll find this book rather discouraging. But it's better to have the information than not, and I found it a good reminder of the inherent drawbacks of loans and debt. There is, sadly, nothing magical about microcredit that removes those problems. Rating: 9 out of 10

I am struck by the fragmentation in communication mechanisms. Let's look at how I have communicated with my friends in the past few days:

• Phone call
  Tried and tested, though I tend to avoid them. I've made some deliberate calls to sort out immediate plans, and at least one accidental call caused by user error which resulted in talking to someone it was good to hear from.
• Text message
  Again, reasonably tried and tested. I miss the inability to use Google Voice when I'm in the UK; I'd much rather read and compose text messages from my web browser when I'm near a computer than type them on my phone, even if it does have a keyboard.
• Email
  One of my favourite methods of communication. Suitable for quick messages or longer screeds. I can throw links in and expect you to be able to click them. I can put lots of detail so that everything is covered easily. I can confuse you by quoting correctly. I guess while I do read email on my phone I'm less likely to reply there as I'm always a bit embarrassed how the clients cope with replies.
• IRC
  Like, I suspect, many readers of my blog posts, I'm still a daily user of IRC. There are friends I keep in touch with mostly via this method. It's great. It's like Twitter for old people and much better in many ways.
• Skype
  This started out as a work thing. It was the way in which the Belfast office communicated with the US, it become the way the Belfast office communicated with each other and when I moved on it was the way in which I kept in contact with a group of people I consider good friends. It's great for calls (I feel bad saying that, but it's an idea executed well across multiple platforms and any other VOIP stuff I've played with has been much more of a hassle), but the one to one and group chat functionality is pretty spot on as well. Also has the advantage that I can turn it off and mostly not end up with work queries.
• Google Hangouts
  I actually quite like these. They work on my phone, I can poke them from a web browser, I can dump more than just text into them. IRC is better in some ways, but I do like the additional flexibility I get from a Hangout. It doesn't play well with people who haven't drunk the Google koolaid, which is the main reason I haven't managed to convince the Skype group chat group to move it over here.
• Facebook messenger
  I hate this. On the face of it there's not a lot of difference between it and Hangouts, but the app wants more and more privileges, I'm less likely to be logged into Facebook (e.g. I avoid it at work, whereas there are good reasons I'd be logged into my Google account there, though less so since the demise of Reader) and I don't think it's as nicely implemented. However there are a few people who it's easiest to get hold of via this method. And there's a certain amount of mesmerisation by the floaty wee faces it invokes on my phone.

While some of these work better for me than others really what I'd like is to use fewer of them, and I can't see that happening any time soon. I don't want to have to run a handful of different messaging apps on my phone. I also don't want to be limited to only using my laptop or my phone for something - I'd much prefer to be able to pickup the phone, laptop or tablet depending on what I'm up to and have my full range of communication available. Some of these things can be aggregated together, but that will then lose some of the advantages. And I'm sure that even if I got rid of one or two of the above there'd be something to fill the gap along shortly (I have, for example, so far completely avoided WhatsApp).

Review: A People's History of the Supreme Court, by Peter Irons

Publisher:	Penguin
Copyright:	1999, 2006
Printing:	2006
ISBN:	0-14-303738-2
Format:	Trade paperback
Pages:	553

I first encountered Peter Irons via his Teaching Company course, The History of the Supreme Court. I listened to that and enjoyed it, and would recommend it as an excellent overview. When I later ran across this book, I was excited: I usually prefer books to lectures on topics that can benefit from greater depth, and A People's History of the United States is one of my favorite history books. A book that talked about the Supreme Court from a similar bottom-up approach was appealing. Unfortunately, I think the title oversells this book. It is a history of the Supreme Court, and, as with Zinn's book, it carries its bias openly (if not as forcefully or compellingly as Zinn). It is a legal history concerned with individual rights, and Irons openly expresses his opinions of the merits of certain decisions. But it's not the full-throated cry for justice and for the voice of the general population that Zinn's book was, nor did I think it entirely delivered on the introductory promise to look at the people and situations underlying the case. It does provide more background and biographical sketch of the litigants in famous cases than some other histories, but those sketches are generally brief and focused on the background and relevant facts for the case. This is, in short, a history of the Supreme Court, with some attention paid to the people involved and open comfort with an authorial viewpoint, but without the eye-opening and convention-defying freshness of Zinn's refusal to point the camera at the usual suspects. It also largely duplicates Irons's course for the Teaching Company. That's only really a problem for those, like me, who listened to that course already and came to this book looking for additional material. If you haven't listened to the course, this is a reasonable medium in which to provide the same content, although you do miss the audio recordings of actual Supreme Court argument in some of the most famous cases. But after having listened or read both, I think I prefer the course. It felt a bit more focused; the book is not padded, but the additional material is also not horribly memorable. That said, as a history, I found this a solid book. Irons opens with a surprisingly detailed look at the constitutional convention and the debates over the wording of the US Constitution, focusing on those sections that will become the heart of later controversy. Some of them were known to be controversial at the time and were discussed and argued at great length, such as anything related to slavery and the role of a bill of rights. But others, including many sections at the heart of modern controversies, were barely considered. Despite being a bit off-topic for the book, I found this section very interesting and am now wanting to seek out a good, full history of the convention. Irons's history from there is primarily chronological, although he does shift the order slightly to group some major cases into themes. He doesn't quite provide the biographies of each Supreme Court justice that he discussed in the introduction, but he comes close, along with discussion of the surrounding politics of the nomination and the political climate in which they were sitting. There's a bit of an overview of the types of cases each court saw, although not as much as I would have liked. Most of the history is, as one might expect, focused on more detailed histories of the major cases. Here, I was sometimes left satisfied and sometimes left a bit annoyed. The discussion of the Japanese internment cases is excellent, as you might expect given Irons's personal role in getting them overturned. The discussion of the segregation cases is also excellent; in general, I found Section V the strongest part of this book. Irons also does a good job throughout in showing how clearly political the court was and has always been, and the degree to which many major decisions were pre-decided by politics rather than reasoned legal judgment. Where I think he fails, however, is that much of this book has little sense of narrative arc. This is admittedly difficult in a history of a body that takes on widely varying and almost random cases, but there are some clear narratives that run through judicial thinking, and I don't think Irons does a good enough job telling those stories for the readers. For example, I remembered the evolution of interpretation of the 14th Amendment from freedom of contract to the keystone of application of the Bill of Rights to the states as a compelling and coherent story from his course, and here it felt scattered and less clear. In general, I liked the earlier sections of this book better than the later, with Section V on the Warren court as the best and last strong section. Beyond that point in the history, it felt like Irons was running out of steam, and it was harder and harder to see an underlying structure to the book. He would describe a few cases, in a factual but somewhat dry manner, make a few comments on the merits of the decision that felt more superficial than earlier sections of the book, and then move on to another case that might be largely unrelated. Recent judicial history is fascinating, but I don't think this is the book for that. Irons is much stronger in the 19th and first half of the 20th centuries; beyond that, I prefer Jeffrey Toobin's The Nine, which also has far deeper and more interesting biographies of recent justices. This is not a bad history of the Supreme Court. If you're looking for one, I can recommend it. But if you're flexible about format, I recommend Irons's course more. I think it's less dry, better presented, and flows better, and I don't feel like you'll miss much in the transformation of this book into an 18 hour course. It's also remarkable to hear the actual voices of the lawyers and justices in some of the landmark cases of the middle of the 20th century. But if you specifically want a book, this does the job. Do also read The Nine, though. It's a good complement to Irons's straight history, showing much more of the inner workings, political maneuverings, and day-to-day struggles of the justices. Rating: 7 out of 10

I have begun to push some of the Emacs Lisp Packages I have been working on over the last years to GNU ELPA, the Emacs Lisp Package Archive. That means you can use "M-x list-packages RET" to install them in GNU Emacs 24.

Since first posting my idea for an SSL co-op a couple of weeks ago, I ve gotten some positive feedback from people, and further thinking and research has convinced me that it is feasible to at least attempt it. As a result, I d like to announce the public unveiling of The SSL Co-op. It is intended to be a commercial, not-for-profit¹ organisation that issues widely-trusted certificates to members, for their use or for resale. Eventually, I d like the co-op to be a root CA in its own right, with its certificate trusted by all the browsers and other X.509-using applications out there, but that isn t something that s achievable immediately. At this stage, the co-op hasn t been formed, and I m looking for expressions of interest from individuals and organisations who would be interested in becoming members. If you fit that description, I d really appreciate it if you could fill out a short survey so I can get a better idea of what sort of scale the co-op will be operating at initially. This is the first step towards an interesting future, where there is more choice of provider for online identity verification. Exciting times.

---

type=AVC msg=audit(1403622580.061:96): avc: denied write for pid=1331 comm="mysqld_safe" name="/" dev="dm-0" ino=256 scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=dir
type=SYSCALL msg=audit(1403622580.061:96): arch=c000003e syscall=269 success=yes exit=0 a0=ffffffffffffff9c a1=7f5e09bfe798 a2=2 a3=2 items=0 ppid=1109 pid=1331 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="mysqld_safe" exe="/bin/dash" subj=system_u:system_r:mysqld_safe_t:s0 key=(null) For a long time (probably years) I ve been seeing messages like the above in the log from auditd (/var/log/audit/audit.log) when starting mysqld. I haven t fixed it because the amount of work exceeded the benefit, it s just a couple of lines logged at every system boot. But today I decided to fix it. The first step was to find out what was going on, I ran a test system in permissive mode and noticed that there were no attempts to create a file (that would have been easy to fix). Then I needed to discover which system call was triggering this. The syscall number is 269, the file linux/x86_64/syscallent.h in the strace source shows that 269 is the system call faccessat. faccessat(2) and access(2) are annoying cases, they do all the permission checks for access but don t involve doing the operation so when a program uses those system calls but for some reason doesn t perform the operation in question (in this case writing to the root directory) then we just get a log entry but nothing happening to examine. A quick look at the shell script didn t make the problem obvious, note that this is probably obvious to people who are more skilled at shell scripting than me but it s probably good for me to describe how to solve these problems every step of the way. So the next step was to use gdb. Here is the start of my gdb session: # gdb /bin/sh
[skipped]
Reading symbols from /bin/dash (no debugging symbols found) done.
(gdb) b faccessat
Breakpoint 1 at 0 3960
(gdb) r -x /usr/bin/mysqld_safe
[lots skipped]
+ test -r /usr/my.cnf
Breakpoint 1, 0x00007ffff7b0c7e0 in faccessat ()
from /lib/x86_64-linux-gnu/libc.so.6 After running gdb on /bin/sh (which is a symlink to /bin/dash) I used the b command to set a breakpoint on the function faccessat (which is a library call from glibc that calls the system call sys_faccessat()). A breakpoint means that program execution will stop when the function is called. I run the shell script with -x as the first parameter to instruct the shell to show me the shell commands that are run so I can match shell commands to system calls. The above output shows the first call to faccessat() which isn t interesting (it s testing for read access). I then ran the c command in gdb to continue execution and did so a few times until I found something interesting. + test -w / -o root = root
Breakpoint 1, 0x00007ffff7b0c7e0 in faccessat ()
from /lib/x86_64-linux-gnu/libc.so.6 Above is the interesting part of the gdb output. It shows that the offending shell command is test -w / . I filed Debian bug #752593 [1] with a patch to fix this problem. I also filed a wishlist bug against strace asking for an easier way to discover the name of a syscall [2].

• [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=752593
• [2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=752595

Twenty years ago, I was sixteen and in high school. The school at the time, if memory serves right, was the "Kunsthumaniora voor muziek en woord van het gemeenschapsonderwijs" in Brussels (which translates approximately to "art high school for music and word of the communal educational branch"... that's not entirely right, but it's late and I'm too tired to look for a dictionary). I was taking drama classes there. No, I'm not making this up. Except maybe the sixteen bit I might be off by one or two years. Hey, I did a lot of things during high school. Stop looking at me like that. At one point in time during my stint in drama, a group of educational interns the sort of people who would be teaching some sort of stuff somewhere after their graduation cooperated with our school to come up with a whole day of classes around one subject. I'll never forget the title: "Can art rescue the democracy?" If that sounds pompous and silly, that's because it was. However, I was at art school, fer crying out loud, so I drank it up like it was cool-aid. Which it wasn't. It was worse. At the end of the day (literally, that is), the educational interns had booked some hot shot art person for a debate. I've since completely forgotten his name. He must've been not that hot shot after all, since I never even once read anything about him in the next few years. Of course I can't exclude the most recent decade, not remembering him and all, but whatever. I also don't remember whether he was a hot shot art critic, a hot shot artist, or just some random hot shot person who writes about art, but doesn't actually do it himself. Whatever. One of the main topics during the whole day was the point about how artsy people find it extremely difficult to define what art actually is. I mean, it's all they do all day, but they can't come up with a decent definition of the damn thing. During the afternoon break, just before the debate with this maybe-sortof-semi hot shot art person, I walk around the playground and think about the whole thing. And come up with some personal definition of art. My definition. As the time of the debate comes up, the hot shot art person sits in the front of the gym behind some table, and the whole school (literally) is sitting in chairs in the rest of the gym. Some questions are asked. Many of those are just shot down. At some point, I raise my finger when it's asked if anyone has further questions. I walk up to the microphone. I ask him: "Could we maybe define art as that thing that, though it might be easy to reproduce, in no case is easy to produce?" He sits (he never got up, really). He thinks. I stand, and wait. After a few seconds of this, he answers. He seems impressed. His reply is something along the lines of "that's not a perfect definition, but it's pretty good. There's a lot to be said for that, and I urge you to write about art when you grow older". I never followed his advice. I got a bit more interested in art, but quickly found out that art consists of one group of people who spend their time doing things other people find pretty, and another group of people who spend their time doing things that makes other people "think", whatever that is. They may not have a brain, they may be silly as hell, but they still want to "think". It's not for me, it's never been. Art, that is -- not the think bit. That is something I don't mind doing. Don't get me wrong. I still like art. I like going to museums from time to time; I like the performing arts. I mean, I play the flute. Not the piano, not the guitar, not the flipping drums, the flute. Which I like, for what it's worth. But if the intent is to make people think, there are better ways to do that. If I want to make people think, I'm not going to make some obscure object that may or may not have a message, in the hope that a millionaire with no better use for his money would buy it just to make his friends jealous, after which he's going to put it in a safe for a few years so he can sell it at a higher value. Without thinking about it. If I want to make people think, I'm not going to write a play or piece of music that's so obscure it will make people all confused, so they can fill their evening afterwards drinking cocktails at a reception, claiming it was all nice and thought-provoking, quoting little parts of it to people they've never met, just so they can make their social status look more than it actually is. Good thing I never finished drama school, I suppose. No, if I want to make people think, I'll try doing so where it actually matters. Like, say, in politics. Not that I have any political ambitions, mind you. But I think the answer to that question of twenty years ago should be a firm "no." Art cannot rescue democracy. Not if they don't have a lot of interesting things to say to anyone but themselves. Maybe the reverse is true, though; maybe democracy can rescue art. Not that I care much. Why is all this relevant today? A few days ago, as I was driving somewhere, there was some show on the radio relating to the current exploits of the Belgian national soccer team. There are a lot of them these days. Radio shows about that subject, that is not Belgian national soccer teams. I suppose having a lot of competition makes it hard to find a new angle to come up with, and still keep things interesting. I also suppose having a lot of stuff going on about that squad gets people annoyed if they're not the least bit interested. I suppose that could be a new angle. Presumably that radio host supposed the same, because he'd been looking for, and asking questions of, people who didn't like soccer and who weren't going to watch the match. Most of them said they weren't interested and added one or two words about what they were going to be doing instead. One of them said that "they" would be better off spending money on "art and culture", rather than on soccer. No, I don't know who "they" were, he didn't say. Never mind that, let me go on now. I don't know who this dude was; they the radio people didn't say. He sounded like someone between 50 and 65, and had a somewhat tenor-y voice. I suspect he had two kids, and a mercedes. Yes, I just made that up. The part about the kids. And the car. No, it doesn't matter. But it's still likely. He sounded like that sort of guy. Whoever this dude was, though, I'd like to just say one thing: Dude, you're an idiot. There's a time and a place for everything. The time and place for art is "everywhen", and "in any random art gallery, opera house, or theatre, out of the public eye". Not because the rest of us doesn't want to deal with art, but because artsy people like it that way. They like to feel all pompous and important, and therefore use difficult words. Words that nobody except those along with them in their ivory tower like to use. Words that don't actually mean anything. But in doing so, they make this Art thing uninteresting to look at for people who don't care about their pompous and silly words. And strengthen the walls of their ivory tower. Only to complain later on that nobody ever shows up at art galleries, and that the really good ones keep going out of business. The time and place for the world championship soccer is "once every four years", and "everywhere". Not because soccer people want to annoy you although, yes, I'll grant you that the KBVB has gone a little overboard with the merchandising this time around but rather because it's so simple. You kick the ball, and you hit the goal. There, done. Everyone can do it. Yes, true, there are some pompous people talking about it on TV, too. And yes, true, some people are better at it than others. But nobody claims you can't do soccer unless you're part of the "in" club. There are plenty of people who claim you can't do art unless you are. And if everyone can do it, then everyone can understand it. If you can understand it, it's easier to enjoy it. This is why so few people enjoy cricket or baseball outside of the few countries where it's popular. That's also why so few people enjoy art: because you make it so difficult. And people just don't care. They want to be entertained. I'm not saying that soccer is the best sport in the world, or that watching it is the most entertaining thing one can do. It isn't. In fact, beyond the national team, I'm not really following it all that well myself. If through some weird spacial anomaly the world championship would suddenly cease to exist and I would be the only person alive remembering it, I don't think I'd spend a lot of time trying to get it back. But don't compare it to art. Because, well, in the grand scheme of things, neither of those two really matters all that much. That is all.