We currently use completely different methods and tools of building test beds and running tests for Debian vs. Click packages, for normal uploads vs. CI airline landings vs. upstream project merge proposal testing, and keep lots of knowledge about Click package test metadata external and not easily accessible/discoverable. Today I released autopkgtest 3.0 (and 3.0.1 with a few minor updates) which is a major milestone in unifying how we run package tests both locally and in production CI. The goals of this are:

• Keep all test metadata, such as test dependencies, commands to run the test etc., in the project/package source itself instead of external. We have had that for a long time for Debian packages with DEP-8 and debian/tests/control, but not yet for Ubuntu s Click packages.
• Use the same tools for Debian and Click packages to simplify what developers have to know about and to reduce the amount of test infrastructure code to maintain.
• Use the exact same testbeds and test runners in production CI than what developers use locally, so that you can reproduce and investigate failures.
• Re-use the existing autopkgtest capabilities for using various kinds of testbeds, and conversely, making all new testbed types immediately available to all package formats.
• Stop putting tests into the Ubuntu archive as packages (such as mediaplayer-app-autopilot). This just adds packaging and archive space overhead and also makes updating tests a lot harder and taking longer than it should.

So, let s dive into the new features! New runner: adt-virt-ssh We want to run tests on real hardware such as a laptop of a particular brand with a particular graphics card, or an Ubuntu phone. We also want to restructure our current CI machinery to run tests on a real OpenStack cloud and gradually get rid of our hand-maintained QA lab with its test machines. While these use cases seem rather different, they both have in common that there is an already existing machine which is pretty much only accessible with ssh. Once you have an ssh connection, they look pretty much the same, you just need different initial setup (like fiddling with adb, calling nova boot, etc.) to prepare them. So the new adt-virt-ssh runner factorizes all the common bits such as communicating with adt-run, auto-detecting sudo availability, doing SSH connection sharing etc., and delegates the target specific bits to a setup script . E. g. we could specify --setup-script ssh-setup-nova or --setup-script ssh-setup-adb which would then get called with open at the appropriate time by adt-run; it calls the nova commands to create a VM, or run a few adb commands to install/start ssh and install the public key. Then autopkgtest does its thing, and eventually calls the script with cleanup again. The actual protocol is a bit more involved (see manpage), but that s the general idea. autopkgtest now ships readymade scripts for these two use cases. So you could e. g. run the libpng tests in a temporary cloud VM:

# if you don't have one, create it with "nova keypair-create"
$ nova keypair-list
[...]
  pitti   9f:31:cf:78:50:4f:42:04:7a:87:d7:2a:75:5e:46:56  
# find a suitable image
$ nova image-list 
[...]
  ca2e362c-62c9-4c0d-82a6-5d6a37fcb251   Ubuntu Server 14.04 LTS (amd64 20140607.1) - Partner Image                           ACTIVE    
$ nova flavor-list 
[...]
  100   standard.xsmall    1024        10     10                 1       1.0           N/A        
# now run the tests: please be patient, this takes a few mins!
$ adt-run libpng --setup-commands="apt-get update" --- ssh -s /usr/share/autopkgtest/ssh-setup/nova -- \
   -f standard.xsmall -i ca2e362c-62c9-4c0d-82a6-5d6a37fcb251 -k pitti
[...]
adt-run [16:23:16]: test build:  - - - - - - - - - - results - - - - - - - - - -
build                PASS
adt-run: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ tests done.

Please see man adt-virt-ssh for details how to use it and how to write setup scripts. There is also a commented /usr/share/autopkgtest/ssh-setup/SKELETON template for writing your own for your use cases. You can also not use any setup script and just specify user and host name as options, but please remember that the ssh runner cannot clean up after itself, so never use this on important machines which you can t reset/reinstall! Test dependency installation without apt/root Ubuntu phones with system images have a read-only file system where you can t install test dependencies with apt. A similar case is using the null runner without root. When apt-get install is not available, autopkgtest now has a reduced fallback mode: it downloads the required test dependencies, unpacks them into a temporary directory, and runs the tests with $PATH, $PYTHONPATH, $GI_TYPELIB_PATH, etc. pointing to the unpacked temp dir. Of course this only works for packages which are relocatable in that way, i. e. libraries, Python modules, or command line tools; it will totally fail for things which look for config files, plugins etc. in hardcoded directory paths. But it s good enough for the purposes of Click package testing such as installing autopilot, libautopilot-qt etc. Click package support autopkgtest now recognizes click source directories and *.click package arguments, and introduces a new test metadata specification syntax in a click package manifest. This is similar in spirit and capabilities to DEP-8 debian/tests/control, except that it s using JSON:

    "x-test":  
        "unit": "tests/unittests",
        "smoke":  
            "path": "tests/smoketest",
            "depends": ["shunit2", "moreutils"],
            "restrictions": ["allow-stderr"]
         ,
        "another":  
            "command": "echo hello > /tmp/world.txt"
         
     

For convenience, there is also some magic to make running autopilot tests particularly simple. E. g. our existing click packages usually specify something like

    "x-test":  
        "autopilot": "ubuntu_calculator_app"
     

which is enough to do what I mean , i. e. implicitly add the autopilot test depends and run autopilot with the specified test module name. You can specify your own dependencies and/or commands, and restrictions etc., of course. So with this, and the previous support for non-apt test dependencies and the ssh runner, we can put all this together to run the tests for e. g. the Ubuntu calculator app on the phone:

$ bzr branch lp:ubuntu-calculator-app
# built straight from that branch; TODO: where is the official" download URL?
$ wget http://people.canonical.com/~pitti/tmp/com.ubuntu.calculator_1.3.283_all.click
$ adt-run ubuntu-calculator-app/ com.ubuntu.calculator_1.3.283_all.click --- \
      ssh -s /usr/share/autopkgtest/ssh-setup/adb
[..]
Traceback (most recent call last):
  File "/tmp/adt-run.KfY5bG/tree/tests/autopilot/ubuntu_calculator_app/tests/test_simple_page.py", line 93, in test_divide_with_infinity_length_result_number
    self._assert_result("0.33333333")
  File "/tmp/adt-run.KfY5bG/tree/tests/autopilot/ubuntu_calculator_app/tests/test_simple_page.py", line 63, in _assert_result
    self.main_view.get_result, Eventually(Equals(expected_result)))
  File "/usr/lib/python3/dist-packages/testtools/testcase.py", line 406, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: After 10.0 seconds test failed: '0.33333333' != '0.3'
Ran 33 tests in 295.586s
FAILED (failures=1)

Note that the current adb ssh setup script deals with some things like applying the autopilot click AppArmor hooks and disabling screen dimming, but it does not do the first-time setup (connecting to network, doing the gesture intro) and unlocking the screen. These are still on the TODO list, but I need to find out how to do these properly. Help appreciated! Click app tests in schroot/containers But, that s not the only thing you can do! autopkgtest has all these other runners, so why not try and run them in a schroot or container? To emulate the environment of an Ubuntu Touch session I wrote a --setup-commands script:

adt-run --setup-commands /usr/share/autopkgtest/setup-commands/ubuntu-touch-session \
    ubuntu-calculator-app/ com.ubuntu.calculator_1.3.283_all.click --- schroot utopic

This will actually work in the sense of running (and succeeding) the autopilot tests, but it will fail due to a lot of libust[11345/11358]: Error: Error opening shm /lttng-ust-wait... warnings on stderr. I don t know what these mean, just that I also see them on the phone itself occasionally. I also wrote another setup-commands script which emulates read-only apt , so that you can test the unpack only fallback. So you could prepare a container with click and the App framework preinstalled (so that it doesn t always take ages to install them), starting from a standard adt-build-lxc container:

$ sudo lxc-clone -o adt-utopic -n click
$ sudo lxc-start -n click
  # run "sudo apt-get install click ubuntu-sdk-libs ubuntu-app-launch-tools" there
  # then "sudo powerdown"
# current apparmor profile doesn't allow remounting something read-only
$ echo "lxc.aa_profile = unconfined"   sudo tee -a /var/lib/lxc/click/config

Now that container has enough stuff preinstalled to be reasonably fast to set up, and the remaining test dependencies (mostly autopilot) work fine with the unpack/$*_PATH fallback:

$ adt-run --setup-commands /usr/share/autopkgtest/setup-commands/ubuntu-touch-session \
          --setup-commands /usr/share/autopkgtest/setup-commands/ro-apt \
          ubuntu-calculator-app/ com.ubuntu.calculator_1.3.283_all.click \
          --- lxc -es click

This will successfully run all the tests, and provided you have apt-cacher-ng installed, it only takes a few seconds to set up. This might be a nice thing to do on merge proposals, if you don t have an actual phone at hand, or don t want to clutter it up. autopkgtest 3.0.1 will be available in Utopic tomorrow (through autosyncs). If you can t wait to try it out, download it from my people.c.c page . Feedback appreciated!

Review: Ancillary Justice, by Ann Leckie

Series:	Imperial Radch #1
Publisher:	Orbit
Copyright:	2013
ISBN:	0-316-24662-X
Format:	Trade paperback
Pages:	416

As Ancillary Justice opens, Breq is on an icy planet trying to track down a person who has gone rather thoroughly to ground. As her search continues, the reader slowly learns the context of that search, which is only the latest step of a long and quixotic exercise in private determination. She was set on her path by events twenty years previous, told in interleaved flashbacks, that have left her greatly diminished and thrust outside of any of the context of her over thousand years of life. Breq was Justice of Toren, a starship and AI that possessed thousands of ancillaries, human bodies slaved to her mind. Now only this one remains, and nothing exists of her former life other than this goal. It's always hard to write a review of a book that I loved this much. I want to find some way to grab the reader and shake them and say "you have to read this!" without giving away any of the delicious details. That's particularly difficult with Ancillary Justice, since one of the delights of this book is the slow unfolding of not only the plot but the background and motivations behind the plot. There is so much beneath the surface of Breq's methodical intent, and discovering all the nuances is utterly delightful. This is also a book that is very deeply concerned with identity, and which does one of the best jobs I've seen in science fiction of showing a non-human first-person protagonist: close enough to human to permit identification and comprehension, but far enough away for delightful sparks of unexpected insight or thought-provoking difference. Breq is a ship, and AIs are central to this story, so the comparison that comes immediately to mind is with Iain M. Banks. Leckie's role for ship AIs is much different than the structure of Banks's Culture, but I think this is still an apt comparison. Like Banks, Leckie is writing large-scale space opera dominated by a meddling empire that follows some familiar human patterns but not others. Banks's Culture takes a less direct approach with its meddling; the Radch has a more Roman attitude towards preventative conquest and citizenship. But both deal with large-scale issues of politics, culture, and conquest. Both also write excellent AIs, but I think Leckie is more successful than Banks at giving her AIs inhuman properties and creating a sense of eerie alienness that sometimes fades almost entirely and sometimes comes sharply to the surface. The first-person perspective helps considerably there. But where Ancillary Justice truly shined for me is in the interpersonal relationships, and in the insight they provided into character and motive. At the start of the book, Breq finds a drug addict dying in the cold and rescues her, bringing her along on her search for lack of a better option. Breq's relationship with Seivarden is complex, difficult for both of them, badly asymmetric at the start, and develops into something brilliant. At the start of the book, Seivarden is easy to dislike, and Breq's tone is refreshingly bracing and forthright. But through the course of the book Seivarden grows into something much more, in a way that I found both believable and incredibly compelling. And Leckie does this without falling into any of the typical relationship patterns, without introducing artificial romance (or, indeed, any romance at all, which is an excellent choice here), and without compromising the personalities of either character. It's masterfully done. One of the most amazing things about this book to me is that it's a first novel. I never would have guessed that from reading it. It's beautifully paced, the characterization is deep and compelling, and Leckie avoids any sign of the typical first-novel problem of stuffing the book with too much Stuff. Ancillary Justice is a capable and confident novel that builds a compelling world and even more compelling characters. I liked it more than I like most of the Culture novels, which is saying quite a lot, but Leckie offers much of the same scope with deeper and more personal characterization and a tighter plot. I haven't yet remarked on one aspect of this book that every other review seems to remark on: its treatment of gender. The Radch do not recognize or care about gender distinctions, and therefore Breq struggles throughout the book with proper gender labeling in much the same way that a native English speaker tends to struggle with grammatical gender when learning a Romance language, except with more social consequences. Leckie has chosen to represent this in the novel by having Breq refer to everyone uniformly as "she." This has its pluses and minuses: it still supports a binary gender concept where a gender-neutral pronoun might not, but given all the negative reaction this book got just for using "she," a gender-neutral pronoun might be a bridge too far for a lot of readers. I thought it created a nice bit of alienation, a way of forcing the reader to look at gender markers from the outside and a way to point out how arbitrary many of them are. It was also interesting to see how surprised I was at various points in the book when it became obvious that some person Breq had been calling "she" in the first-person narration turned out, from story context, to probably be male. That said, I think this part of the book is overblown in reviews. It's often the first thing people mention, but while it was a nice side bit of world-building, I don't think it's that central to the story. I'm particularly baffled by the handful of people who complained about it, since it's not intrusive and it quickly fades into the background apart from occasional necessary shifts of mental image. (It does create the impression of a world containing only women, but I found that a nice change from the more common impression in space opera of a world containing only men.) Ancillary Justice has already won the Nebula and Arthur C. Clarke awards and tied for the BSFA award for best novel, and I'm happy to report that it deserves all of those. I haven't yet read all of the other Hugo nominees, but it's hard to imagine a world in which it won't top my ballot. This is a fantastic novel, by far the best thing I've read so far this year. I'm delighted that it's the first book of a trilogy, since I'm not done with either the world or the characters yet, but it stands well on its own and reaches a satisfying conclusion. I recommend it to everyone, but particularly to anyone who likes Banks, intelligent ships, or who is looking for thoughtful and complex space opera. Followed by Ancillary Sword. Rating: 10 out of 10

Review: The Wilding, by C.S. Friedman

Series:	In Conquest Born #2
Publisher:	DAW
Copyright:	2004
Printing:	July 2005
ISBN:	0-7564-0202-6
Format:	Mass market
Pages:	551

In Conquest Born is a rather odd space opera, featuring an interstellar war between two great empires that culminates in an intensely personal struggle between leaders of the two sides. It has one of the most intensely misogynistic societies in science fiction (the villains, yes, but weirdly ambiguous villains), genetic breeding programs for telepaths, and a host of unlikable but oddly fascinating characters. I thought it was weaker on re-read than I remembered it being originally, but it's still singular enough in memory that I wanted to read the sequel. The Wilding picks up in the same universe a century and a half later. I expected more of a direct connection to the plot of In Conquest Born, but I think this book could be read on its own. (This is good, given how the plot of In Conquest Born tends to slip out of my memory, despite how unique the book is.) The Azeans and the Braxins are, of course, still at war, an on-again, off-again affair that neither side seems likely to ever win. But the Azeans have forbidden psychic powers and driven all psychics underground, if they still exist at all. They are supposedly too unstable to be allowed in the carefully constructed Azean society. Tathas is Braxin, but not from the ruling Braxan tribe. He is caught practicing old rituals of his native tribe and escapes sentence of death by going on the Wilding, an ancient practice of leaving Braxin culture to find new genetic material and bring it back to strengthen their species. Or, put in less euphemistic terms, to find and rape a woman and bring back the resulting child, although apparently retrieving a genetic sample would also work. Did I mention the misogynistic society part? Still there. Zara, the other primary protagonist, is an Azean mediator who, as the book is opening, starts developing psychic powers. This is a rather serious problem that gets her suspended from her job. Her check of her own genetic records (the old-school SF obsession with eugenics is also still there) leads to the discovery that she's an identical twin, whose sister was kidnapped as an infant under mysterious circumstances and apparently by underground psychics. Zara decides to go in search of her. As with In Conquest Born, this book is full of deeply unpleasant societies and world views. Also as with In Conquest Born, I don't think Friedman intends any of them to be held up as examples of correct politics or behavior. What makes both of these books unusual is that neither she nor the characters seem to be making any attempt to construct a third alternative that would be better. The secret society of psychics has a different appeal, but it's not clear the characters would want to be part of it either, and it's quite possible that they're still responsible for a lot of the crap that's going on in the galaxy. The book feels like it ought to be political, as these societies are clearly broken and deeply abusive (the Braxins more obviously than the Azeans, but they're both creepy), but it mostly isn't. Instead, The Wilding is focused on individual people trying to make their way in this world. Those people come from varying strata of these societies. Zara and Tathas are there throughout; other characters crop up briefly with their own viewpoints and then often go away again or are killed. I think Friedman is going for a story of raw emotion and a tight focus on individual actions, with little in the way of clean morality or broader morals. And yet... the final act of the protagonists in the book is a political act that seems aimed at making the universe a better place. In a way, this is a very cynical book: there is a tremendous amount of manipulation and political maneuvering, and it's hard to see any cause here that one can fully support. This is all rather similar to In Conquest Born, but what held the previous book together was its backbone of obsessive vengeance and the spectacle of two intelligent people from utterly different societies locked in all-out emotional and physical combat. The Wilding doesn't have that. It has characters with strong motives and strong emotions, but none of them are the sort of epic, transcendent emotions that propel mythic heroes at each other. This is a more prosaic and personal book, which means the drama isn't strong and sweeping enough to make the reader forget the deep unpleasantness that's going on around them. The Wilding feels less like a clash of titans and more like personal quests for identity against a universe full of awful people and worse systems. It also retains, in the Braxin, aggressive male-dominant sexuality just this side of a Gor novel, but here that feels seedier and even more squirm-inducing than it did in In Conquest Born. The Wilding is even more ambiguous about the ethics of that sexual and misogynistic abusiveness; Tathas partly gives it up, but this is presented more as an accommodation to another culture than as a real engagement with the inherent abusiveness of Braxin society. I still have no idea whether and to whom I would recommend In Conquest Born. I'm less torn about The Wilding: don't bother. It's a lesser work in all respects, it shares some of the problems with too many characters and an occasionally incoherent plot, and I had even more trouble with the horrible societies Friedman creates than I did in the previous book. And it doesn't have the same counterbalance of epic, uncontrolled emotions and characters who seem to transcend the story they're in. Instead, it feels more like a story of damaged people in an awful universe who are meddling through as best they can while being manipulated by the people with real power. It got more depressing the more I thought about it, and the ending doesn't salvage the story. Rating: 4 out of 10

I stared at Noodles' Emptiness, where I found a short rant on the currently most used forms of communication. No, into the most socially-useful forms of communication. No, into what works best for him. And, as each person's experience is unique, I won't try to correct him Noodles knows himself much, much, much, much better than I do. But some people have wondered recently (i.e. at conferences I have been at) why I give such an atypical use to social networks (...a term which I still hold to be grossly misused, but that's a topic for a different rant...One that's been had too many times). So, although my blog is syndicated at Planet Debian, and I know a good deal of readers come from there, this post is targetted at the rest of the world population: Those that don't understand why many among us prefer other ways of communication. Noodles mentions seven forms of communication he uses, arguably sorted by their nowadayness, low to high: Phone call, text (SMS) message, email, IRC, Skype, Google Hangouts and Facebook messenger. Among those, I strongly dislike two: Phone call and Skype (or any voice-based service, FWIW). I do most of my communication while multitasking, usually at work. I enjoy the quasi-real-timeliness of IRC and the instant messengers, but much more, I like the ability to delay an answer for seconds or minutes without it breaking the rules of engagement. Second, if the ordering is based on what I found, the reason for my little rant should become obvious: We had kept a great job so far building interoperable technology.1 Up until now, you could say drop me a mail , and no matter if you had your mail with GMail and I insisted on self-hosting my gwolf.org, as long as our communications adhered to simple and basic standards, we would be perfectly able to communicate. Skype is a bit of a special case here: They did build a great solution, ~ten years ago, when decent-quality VoIP was nowhere to be found. They have kept their algorithm and mechanisms propietary, and deliberately don't operate with others. And, all in all, there is a case for them remaining closed. But Google Hangouts and Facebook Messenger do piss me off. More the first than the second. Both arrived to the instant messenger scene long after the experimentation and early stages, so they both took Jabber / XMPP, a well tried and tested protocol made with interoperability and federability in mind. And... They closed it, so they can control their whole walled garden. So, yes, I do use them both Because in both there are many people which I value, and which I want to be in touch with. And no, I won't try to change the ways of everybody so they can talk with me on IRC. I'm an asshole, but everything has its limits! :) Of course, I can (still?) use them from my trusty old multiprotocol chat manager, Pidgin. And it's so much better to have one small program gobbling up moderate heaps of memory than to be connected to the Facebook site, spending much more memory and CPU cycles via Javascript in a Web browser. Bah. PS- Interestingly, he left out the face-to-face communication.I am quite an anchorite in my daily life, but I still think it's worth at least a mention ;-) So, Noodles: Thanks for the excuse to let me vent a rant ;-)

• 1. Interestingly, a counterexample came up on me. One I do not remember, but I have seen printed information that make me believe it: Back in the 1940s/1950s, Mexico (Mexico City only perhaps?) had two parallel phone networks. If I'm not mistaken, one was Ericsson and the other was AT&T. Businesses often gave you both of their numbers in their ads, because you could not call one network from the other. And now that seems so backwards and unbelievable!

While not required by Debian policy, the patch tagging guidelines (aka DEP-3) recommends to add a structured header to source package patches. Long story short, patches should begin with something like :

Description: tweak lcdproc config for debian
patch LCDd.conf to:
* use syslog instead of stderr to show message
* run LCDd as root (to read /dev/lcd* file)
.
The latter could be done better by tweaking udev rule
Author: dod
Applied-Upstream: NA

Making sure that the DEP-3 recommendation is respected may not be fun. Do not despair: once libconfig-model-dpkg-perl is installed on your system, you can check whether your patches respect the DEP3 recommendation with the following command: $ cme check dpkg-patches
loading data
checking data
check done You can also check individual patches with: cme check dpkg-patch tweak-conf Note that bash auto-completion is provided only from version 2.049 of libconfig-model-dpkg-perl package. If editing the header of your patches with your favorite editor does not strike you as fun, you can also use cme graphical editor provided by libconfig-model-tkui-perl. Once this package is installed, you can run: cme edit dpkg-patches You will get this editor to update your patches: This editor will show you the available parameters and the associated documentation to let you provide relevant information. Last but not least, patch header check is also performed when you check your whole package with cme check dpkg All the best
Tagged: Config::Model, debian, package, patch

This was another good week. This week I focused on cleaning up the merge implementation as well as learning about Debian packaging. As far as cleaning up the implementation, I got a few tests written to test the merge function. Additionally I rewrote the merge command code (the code responsible for the kdb merge command, it depends on the implementation of merge I wrote last week). Now the kdb merge command takes in a parent key for each keyset and creates a keyset containing all the keys under the parent key. It then invokes the KeysetMerge function I wrote on those keysets and saves the returned keyset to the key database. Most of the week I spent learning about Debian packaging. I was able to get some hand-on experience by creating a package for the augeas plugin for Elektra. I modified the control file to include the libaugeas0 dependency and adding in a description of the plugin. Also I created a new libelektra-augeas4.install file to install the share library to the correct location on the system. Then, I tried building the source using git-buildpackage but unfortunately it failed on packaging the plugin because the libelektra-augeas.so file was never built. It turned out that the augeas plugin had to be added to ElektraCache.cmake under plugins_dep (plugins with dependencies) so that the plugin would be built on make. So I fixed that and ran buildpackage again and voila! It worked! I successfully make my first Debian package! The other thing I looked into was how configuration files get updated during a package upgrade under Debian. It turns out, most package define their configurations as conffiles . These conffiles do not get overwritten during package upgrade if the user has edited them. Basically, during an upgrade, dpkg calculates a checksum on each conffile and if it has changed, prompts the user what to do. Instead, the goal of our project is to attempt an automatic merge of the conffile and if successful, used the merge version. If the merge is unsuccessful dpkg will prompt the user as usual. I need to figure out how to change dpkg s behavior during an upgrade with a conffile so I can build a prototype of a package upgrade merging a file. Lastly, this week I looked into adding tab completion to the kdb tool. Currently kdb does not try to complete commands using tab completion in bash. It turns out adding tab completion isn t that difficult. You basically just need to write rules for how tab completion should work for your program and add it to /etc/bash_completion.d/ I wrote a basic tab completion script for kdb that will complete the commands for kdb. It follows here: #Proof of concept bash completion for kdb. You must copy this file to /etc/bash_completion.d/kdb and load it with ". /etc/bash_completion.d/kdb"
_kdb()

local cur prev opts
COMPREPLY=()
cur="$ COMP_WORDS[COMP_CWORD] "
prev="$ COMP_WORDS[COMP_CWORD-1] "
opts="check convert cp export file fstab get getmeta import info ls lsmeta merge mount mv rm set setmeta sget shell test umount vset" if [[ $ cur == * ]] ; then
COMPREPLY=( $(compgen -W "$ opts " -- $ cur ) )
return 0
fi

complete -F _kdb kdb Its just a proof on concept but its a really neat feature and surprisingly easy to implement. One of the other students working on Elektra, Felix, is also working on tab completion. I think we will collaborate and have this feature implemented for kdb very soon. For next week I have a few goals I would like to accomplish. First of all I will have a working prototype of a package being upgraded with automatic conffile merging using kdb merge. Second, I want to continue working on the merge implementation to make it cleaner and work on more sophisticated keysets. I also need to add better error handling to tell if a merge was successful or not, and if not why. Third, I want to work on some of the lintan errors for the deb package of Elektra 0.8.6 and try to get some of those fixed to move closer toward release. Lastly I want to updated the kdb man page for Debian to make it more informative for the shell tool. I was able to get all my goals done for this week except the man page and getting some of the lintian errors fixed up. Overall though it was a very productive week. Until next week,
Ian S. Donnelly

According to the Free Software Foundation, the Debian distribution is not a Free System Distribution. While I, as a Debian Developer, disagree with that point of view, it should be obvious that the opinion of an organization that engages in politics (or, at the very least, in political lobbying) and of which I am not a member should not be my concern. However, it has long irked me the wrong way that these claims exist. A few weeks ago, I found the System distribution commitments page on Wikipedia. This does change the situation somewhat; I cannot say that a page explaining an opinion of the FSF is wrong; but if that opinion is copied nearly verbatim to a website which claims neutrality as one of its "Core content policies", then the situation is significantly different. When I originally found that page, it was immediately obvious where the content came from, but there was no reference listed, so I added a "citation needed" tag, the first time I ever did so on Wikipedia. Now, a few weeks later, a reference has been added, and the "citation needed" tag removed. However, as expected, the only reference which has been added is the highly opinionated document on the FSF's website, which is hardly neutral. I've added another tag (this time a "which" tag, asking for more clarification) and added a Talk page listing my concerns with the article. But I can't really explain in Wikipedia why I think the whole premise is wrong; that's what a blog is for. So, here goes. The FSF lists two reasons why they do not endorse Debian:

1. The non-free and contrib repositories
2. An unsubstantiated claim that "the installer in some cases recommends [...] non-free firmware files for the peripherals on the machine."

Let's have a look at them, each in turn.

Non-free and contrib It is true that Debian provides non-free software in separate repositories, and that software in these repositories does not comply with either the Debian Free Software Guidelines (DFSG) or the FSF's Free Software Definition. However, the non-free and contrib repositories are fairly small:

wouter@carillon:/var/lib/apt/lists$ awk -F':' 'BEGIN total=0 /^Size/ total+=$2 END print total ' ftp.be.debian.org_debian_dists_sid_non-free_binary-amd64_Packages
4398812934

That is, the non-free repository contains just over 4G of software (after installation). Contrib is even smaller:

wouter@carillon:/var/lib/apt/lists$ awk -F':' 'BEGIN total=0 /^Size/ total+=$2 END print total ' ftp.be.debian.org_debian_dists_sid_contrib_binary-amd64_Packages 
306578636

While, combined, this is approximately the same size as some other distributions, free and non-free together, it pales in comparison with the "main" repository:

wouter@carillon:/var/lib/apt/lists$ awk -F':' 'BEGIN total=0 /^Size/ total+=$2 END print total ' ftp.be.debian.org_debian_dists_sid_main_binary-amd64_Packages
60025033622

i.e., the "main" repository contains just shy of 60G of software. Of course, whether non-free is large or not should not matter at all; the fact that it exists in the first place is a problem for free software advocates. But of those 4 gigabytes, what exactly is the non-free repository used for? For that, the Debian popularity contest has a page specifically for the non-free section. Looking at this page, and ignoring the packages that are listed as "Not in sid" (which either means they were installed from third-party suppliers and therefore cannot be attributed to Debian, or were only available in older distributions so should not be held against the Debian of today anymore), we see three main uses of non-free:

1. Various implementations of non-free algorithms (unrar, libfaac0, etc)
2. A lot of packages containing non-free firmware
3. A number of packages containing non-free documentation

The first package which does not fit any of those descriptions is iozone3, currently on the 64th position of the popularity contest (including packages "Not in sid"). The first item in this list is something which most free software advocates will try to avoid. After all, for all of those algorithms, alternatives exist, and so there is no real reason why one would want to use such software, other than "someone sent me a file encoded according to that algorithm, and I want to decode it". Debian gives you the option to do so, but a valid case could be made for not doing so. The second item in that list is really an alternate version of "the installer recommends non-free software", so I'll come back on that in the next section. The 3rd item in the above list deserves some special attention. Just over 10 years ago, Debian clarified its position where it clearly specified that documentation and programs (as both are interpreted as "software") should follow the same set of guidelines regarding freedom. The FSF disagrees with this position, and chose to release some of its documentation under the GFDL with invariant sections, which Debian decided is a non-free license in 2006. Since then, such GNU documentation would no longer be available were Debian to ditch the non-free repository.

Recommending non-free firmware As I said above, the FSF's page claims that the Debian installer recommends non-free firmware "in some cases", without going into much detail. Special care should be taken in understanding the word "recommend" here; wiktionary defines it as follows:

recommend (third-person singular simple present recommends, present participle recommending, simple past and past participle recommended)

1. (transitive) To bestow commendation on; to represent favourably; to suggest, endorse or encourage as an appropriate choice.

The board recommends Philips, given his ample experience in similar positions.

1. (transitive) To make acceptable; to attract favor to.
2. (transitive) To advise, propose, counsel favorably

The therapist recommends resting the mind and exercising the body.

1. (transitive, archaic) To commit, confide to another's care, confidence or acceptance, with favoring representations

A medieval oblate's parents recommended the boy for life to God and the monastery

However, according to the FSF, merely showing a firmware filename in unencrypted form counts as a recommendation. I've always found that use of the word "recommendation" to be dubious, at best. Ignoring that part, we can make the following observations: The debian-installer module ("udeb") which is responsible for configuring apt is called apt-mirror-setup (code in the apt-setup git repository). The actual debconf templates which allow the user to enable the non-free or contrib repositories are the apt-setup/non-free and apt-setup/contrib templates, and they look like this:

Template: apt-setup/non-free
Type: boolean
Default: false
# :sl1:
_Description: Use non-free software?
 Some non-free software has been made to work with Debian. Though this
 software is not at all a part of Debian, standard Debian tools can be used
 to install it. This software has varying licenses which may prevent you
 from using, modifying, or sharing it.
 .
 Please choose whether you want to have it available anyway.
Template: apt-setup/contrib
Type: boolean
Default: false
# :sl1:
_Description: Use contrib software?
 Some additional software has been made to work with Debian. Though this
 software is free, it depends on non-free software for its operation. This
 software is not a part of Debian, but standard Debian tools can be
 used to install it.
 .
 Please choose whether you want this software to be made available to you.

Note the lines Default: false, which imply that the default answer to these questions is to disable the non-free and contrib repositories. A question which defaults to false isn't a recommendation, nor do I read the text as any form of recommendation to install these repositories (quite the contrary, in fact). More important than whether the default is to enable or disable these repositories is whether this question is even shown to the user in normal installations. Whether this is the case isn't decided by any template; instead, it is decided by the code that calls the templates. That code currently looks like this:

STATE=1
while true; do
        case "$STATE" in
            1)
                db_input low apt-setup/non-free   true
                ;;
            2)
                # If they chose not to use non-free, ask about
                # contrib, with a default of no. If they chose to
                # use non-free, they get contrib too..
                db_get apt-setup/non-free
                if [ "$RET" = false ]; then
                        db_fget apt-setup/contrib seen
                        if [ "$RET" = false ]; then
                                db_set apt-setup/contrib false
                        fi
                        db_input low apt-setup/contrib   true
                else
                        db_fget apt-setup/contrib seen
                        if [ "$RET" = false ]; then
                                db_set apt-setup/contrib true
                        fi
                fi
                ;;
            *)
                break
                ;;
        esac
        if db_go; then
                STATE=$(($STATE + 1))
        else
                STATE=$(($STATE - 1))
        fi
done

Note the "low" in the db_input lines; it means the question is only shown at low priority, i.e., in "expert" mode. The result is that, unless you enable expert mode, you won't even see the question about the non-free repository, and it will be disabled by default. There won't even be any mention of it in your sources.list file. Of course, the above isn't about firmware files, which is what the FSF website claims. So, there is a second part to this story. If you run a system with hardware for which there is a non-free firmware package in the non-free repository, and you provided firmware for this hardware to the installer through various means, then the hw-detect module comes into play. This has code which currently looks like this:

# install cached firmware debs
if [ -d /var/cache/firmware ]; then
    for deb in /var/cache/firmware/*.deb; do
            if [ -f "$deb" ]; then
                    cp -a "$deb" /target/tmp
                    # TODO debconf passthrough
                    if ! in-target dpkg -i "/tmp/$(basename "$deb")"; then
                            # dpkg failed, force removal of package
                            in-target dpkg --force-depends --remove "$(deb_package "$deb")"   true
                    fi
                    rm -f "/target/tmp/$deb"
                    need_nonfree=1
            fi
    done
fi
if [ "$need_nonfree" ]; then
    db_set apt-setup/non-free true
fi

As can bee seen, if any firmware .deb files exist under /var/cache/firmware in the installer environment, then these will be installed on the target system and default answer to the apt-setup/non-free question will be changed to enable the non-free repository. How do such packages end up there? Well, that's done by the same hw-detect udeb, in a different script. The code there is a bit too long to quote in this already far too long blog post, but it boils down to:

• If the installer detects missing firmware, the user is asked whether (s)he wants to install firmware, with the following template:

  Template: hw-detect/load_firmware
  Type: boolean
  Default: true
  # :sl2:
  _Description: Load missing firmware from removable media?
   Some of your hardware needs non-free firmware files to operate. The
   firmware can be loaded from removable media, such as a USB stick or
   floppy.
   .
   The missing firmware files are: $ FILES 
   .
   If you have such media available now, insert it, and continue.
  

• If the user agreed to doing so (and to me, the above template does not read as a recommendation), any firmware .deb files that are provided to the installer are tested for the missing firmware files.
• If any of those files are found, these .deb files are copied to the /var/cache/firmware directory, and the firmware files are made available to the running kernel.

Conclusion In my opinion, while some of the claims on the FSF website are correct, others are at the very least misleading. Debian does not recommend the use of non-free software; and while providing non-free firmware to the installer isn't too difficult, it isn't documented very well; and as such installing a system with non-free enabled isn't something you can easily do by accident. Debian and the FSF clearly disagree on certain things, and this disagreement, not any policies by the Debian project, is the main reason why the FSF won't encourage Debian as a free distribution. I don't think the Debian project will easily agree to get rid of the non-free repository while the FSF still produces non-free documentation or while non-free firmware is still a fact of life in this world. However, I do think the following should happen:

1. The FSF should relicense its non-free documentation so it becomes truly free.
2. Once that has happened, Debian should rename its non-free repository to a "hardware-support" repository (or something along those lines), and remove everything that does not exist solely for the support of proprietary hardware (i.e., non-free drivers, non-free firmware, etc).
3. The installer could be clearer on the fact that if non-free firmware is found to be in use for the system, the user is using software that cannot be properly supported by the Debian project.

While the above wouldn't necessarily bring the opinions of the FSF and Debian 100% in line, it would at least bring their positions closer to eachother. Unfortunately, however, I doubt that this will happen; and that's a shame, because in the end, both Debian and the FSF want the same thing: a world where Free Software is the norm, not the exception.

Update

• The first version of this text said that providing firmware is not very easy. As was pointed out to me, it isn't too hard, but it's not very well documented and not part of the 'regular' installer images.
• Add contrib too, when we compare non-free and main
• Clarify when GFDL is non-free

Wheezy for industrial software development I'm helping with setting up a wheezy-based toolchain for industrial automation. The basic requirements are: live-build, C++11, Qt 5.3, and a frozen internal wheezy mirror. debmirror A good part of a day's work was lost because of #749734 and possibly #628779. Mirror rebuild is still ongoing, and fingers crossed. This is Italy, and you can't simply download 21Gb of debs just to see how it goes. C++11 Stable toolchains for C++11 now exist and have gained fast adoption. It makes sense, since given what is in C++11 it is unthinkable to start a new C++ project with the old standard nowadays. C++11 is supported by g++ 4.8+ or clang 3.3+. None of them is available on wheezy or wheezy-backports. Backports exist of g++ 4.8 only for Ubuntu 12.04, but they are uninstallable on wheezy due at least to a different libc6. I tried rebuilding g++4.8 on wheezy but quickly gave up. clang 3.3 has a build dependency on g++ 4.8. LOL. However, LLVM provides an APT repository with their most recent compiler, and it works, too. C++11 problem solved! Qt 5.3 Qt 5.3 is needed because of the range of platforms it can target. There is no wheezy backport that I can find. I cannot simply get it from Qt's Download page and install it, since we need it packaged, to build live ISOs with it. I'm attempting to backport the packages from experimental to wheezy. Here are its build dependencies: libxcb-1.10 (needed by qt5) Building this is reasonably straightforward. libxkbcommon 0.4.0 (needed by qt5) The version from jessie builds fine on wheezy, provided you remove --fail-missing from the dh_install invocation. libicu 52.1 (needed by harfbuzz) The jessie packages build on wheezy, provided that mentions of clang are deleted from source/configure.ac, since it fails to build with clang 3.5 (the one currently available for wheezy on llvm.org). libharfbuzz-dev Backporting this is a bloodbath: the Debian packages from jessie depend on a forest of gobject hipsterisms of doom, all unavailable on wheezy. I gave up. qt 5.3 qtbase-opensource-src-5.3.0+dfsg can be made to build with an embedded version of harfbuzz, with just this change:

diff -Naur a/debian/control a/debian/control
--- a/debian/control    2014-05-20 18:48:27.000000000 +0200
+++ b/debian/control    2014-05-29 17:45:31.037215786 +0200
@@ -28,7 +28,6 @@
                libgstreamer-plugins-base0.10-dev,
                libgstreamer0.10-dev,
                libgtk2.0-dev,
-               libharfbuzz-dev,
                libicu-dev,
                libjpeg-dev,
                libmysqlclient-dev,
diff -Naur a/debian/rules b/debian/rules
--- a/debian/rules  2014-05-18 01:56:37.000000000 +0200
+++ b/debian/rules  2014-05-29 17:45:25.738634371 +0200
@@ -108,7 +108,6 @@
                -plugin-sql-tds \
                -system-sqlite \
                -platform $(platform_arg) \
-               -system-harfbuzz \
                -system-zlib \
                -system-libpng \
                -system-libjpeg \

(thanks Lisandro Dami n Nicanor P rez Meyer for helping me there!) There are probably going to be further steps in the Qt5 toolchain. Actually, let's try prebuilt binaries The next day with a fresh mind we realised that it is preferable to reduce our tampering with the original wheezy to a minimum. Our current plan is to use wheezy's original Qt and Qt-using packages, and use Qt's prebuilt binaries in /opt for all our custom software. We run Qt's installer, tarred the result, and wrapped it in a Debian package like this:

$ cat debian/rules
#!/usr/bin/make -f
QT_VERSION = 5.3
%:
    dh $@
override_dh_auto_build:
    dh_auto_build
    sed -re 's/@QT_VERSION@/$(QT_VERSION)/g' debian-rules.inc.in > debian-rules.inc
override_dh_auto_install:
    dh_auto_install
    # Download and untar the prebuild Qt5 binaries
    install -d -o root -g root -m 0755 debian/our-qt5-sdk/opt/Qt
    curl http://localserver/Qt$(QT_VERSION).tar.xz   xz -d   tar -C debian/our-qt5-sdk/opt -xf -
    # Move the runtime part to our-qt5
    install -d -o root -g root -m 0755 debian/our-qt5/opt/Qt
    mv debian/our-qt5-sdk/opt/Qt/$(QT_VERSION) debian/our-qt5/opt/Qt/
    # Makes dpkg-shlibdeps work on packages built with Qt from /opt
    # Hack. Don't try this at home. Don't ever do this unless you
    # know what you are doing. This voids your warranty. If you
    # know what you are doing, you won't do this.
    find debian/our-qt5/opt/Qt/$(QT_VERSION)/gcc_64/lib -maxdepth 1 -type f -name "lib*.so*" \
          sed -re 's,^.+/(lib[^.]+)\.so.+$$,\1 5 our-qt5 (>= $(QT_VERSION)),' > debian/our-qt5.shlibs
$ cat debian-rules.inc.in
export PATH := /opt/Qt/@QT_VERSION@/gcc_64/bin:$(PATH)
export QMAKESPEC=/opt/Qt/@QT_VERSION@/gcc_64/mkspecs/linux-clang/

To build one of our packages using Qt5.3 and clang, we just add this to its debian/rules:

include /usr/share/our-qt5/debian-rules.inc

Wrap up We got the dependencies sorted. Hopefully the mirror will rebuild itself tonight and tomorrow we can resume working on our custom live system.

Five years ago, I opened The Eye of the World, the first book in the Wheel of Time series, and read:

Lord of the Morning, he said, I have come for you.

On the island, the air shimmered and coalesced. The black-clad man stood staring at the fiery mountain rising out of the plain. His face twisted in rage and contempt. You cannot escape so easily, Dragon. It is not done between us. It will not be done until the end of time. Then he was gone, and the mountain and the island stood alone. Waiting.

And then, as often, Robert Jordan began books citing some of the prophecy he invented:

And it came to pass in those days, as it had come before and would come again, that the Dark lay heavy on the land and weighed down the hearts of men, and the green things failed, and hope died. And men cried out to the Creator, saying, O Light of the Heavens, Light of the World, let the Promised One be born of the mountain, according to the prophecies, as he was in ages past and will be in ages to come. Let the Prince of the Morning sing to the land that green things will grow and the valleys give forth lambs. Let the arm of the Lord of the Dawn shelter us from the Dark, and the great sword of justice defend us. Let the Dragon ride again on the winds of time. from Charal Drianaan to Calamon,
The Cycle of the Dragon.
Author unknown, the Fourth Age

And then, the grand opening to each of the books in the 11,000-page series, something like this:

The Wheel of Time turns, and Ages come and pass, leaving memories that become legend. Legend fades to myth, and even myth is long forgotten when the Age that gave it birth comes again. In one Age, called the Third Age by some, an Age yet to come, an Age long past, a wind rose in the Mountains of Mist. The wind was not the beginning. There are neither beginnings nor endings to the turning of the Wheel of Time. But it was a beginning. Born below the ever cloud-capped peaks that gave the mountains their name, the wind blew east, out across the Sand Hills, once the shore of a great ocean, before the Breaking of the World. Down it flailed into the Two Rivers, into the tangled forest called the Westwood, and beat at two men walking with a cart and horse down the rock-strewn track called the Quarry Road.

And yesterday, 11,000 pages later, I have completed the series. Really, how could I have put down a book with such a majestic beginning as that? It hinted at the epic that was to come, the amazing literature I was going to read. I didn t read it straight through; I enjoyed books like War and Peace along the way. But what a trip the Wheel of Time series was. The characters were so detailed, so deep, the setting so elaborate, the plots so grand and also so small everyone from servants to queens were well-represented in the books. Not every section was a page-turner; in fact, around the 4th or 5th book, the series started to plod for a thousand pages or two, reading more like a history book than a novel. Sometimes the prejudices and stereotypes of the characters, particularly the sexism that both the men and women displayed, were frustrating to read. I never quite figured out of this was brilliant writing (making the reader feel frustrated at the plodding pace of the book when it was describing characters being frustrated at the plodding pace of events), or if it could have been trimmed down by 2/3 in some places. But really, the number of brilliant endings, laugh-out-loud situations, and page-turning suspense that somehow managed to build over hundreds, and then even thousands of pages, outweighed the flaws.

Politics is a foggy mire full of snakes. - The Eye of the World

Ah. Furlan washed his hands in one direction, then rubbed them in the other. Ah. Forgive me, Master Andra, but Lady Alys is a fierce sounding woman. Only with those who displease her, Lan said blandly. Her bite is far worse than her bark. Ah. Ah. Ah. Your rooms are this way. . . - The Dragon Reborn

One of the marks of truly good writing is that I leave a book or series with a sense of sadness or nostalgia, leaving characters that seem almost like friends. I felt that after finishing the last book in the series, A Memory of Light. The ending was deep, and satisfying, but ultimately left me wanting even more even more than 11,000 pages, somehow.

Elienda and Briain woke from the dream. She might have been speaking of the weather rather than two deaths among women she knew. We all must wake eventually. - Knife of Dreams

I was often touched by the way the Aiel referred to each other. A desert people, their fond farewall was may you always find water and shade. A fitting metaphor for anyone s life, I think. And they referred to their spouse as shade of my heart. Jordan touched on quite a few themes of religion and philosophy. In that quote above, he has the Aiel referring to death as waking from a dream. An intriguing metaphor, isn t it? He even writes conflicts of philosophy, religion, and culture into the series. I was particularly impressed with his treatment of theodicy and the problem of evil (the argument and religious response relating to the presence of evil in the world in the face of a loving God.) It is the most impressive treatment I have seen in fiction, and does a better job of advancing the theodistic argument than many philosophers have, I suspect. It is hard to really go into that one here because of the length and potential for spoilers. I can, however, comment on his treatment of existential nihilism the theory that life and the universe is without meaning, and the sometimes-related comment that what does anything matter then anyway? Here are a couple of quotes:

In Maradon, I saw what had been done to men who followed me. I saw Light in them, Min. Defying the Dark One no matter the length of his shadow. We will live, that defiance said. We will love and we will hope. And I saw him trying so hard to destroy that. He knows that if he could break them, it would mean something. Something much more than Maradon. Breaking the spirit of men he thirsts for that. He struck far harder than he otherwise would have because he wanted to break my spirit. His voice grew softer and he opened his eyes, looking down at her. And so I stood against him. - Towers of Midnight

What if I think it s all meaningless? he demanded with the loud voice of a king. What if I don t want it to keep turning? We live our lives by the blood of others! And those others become forgotten. What good is it if everything we know will fade? Great deeds or great tragedies, neither means anything! They will become legends, then those legends will be forgotten, then it will all start over again!

Why? Rand thought with wonder. Because each time we live, we get to love again.

It felt relaxing to stare out at that distant light, so welcoming and noble. Storms will soon come, it seemed to say. But for now, I am here. I am here. - The Gathering Storm

I unfortunately can t put in in its proper context due to spoilers, but it is a pivotal turning point in the series, relating to the preservation of the world. By giving what we might normally call a characteristic or behavior evil an almost-human voice and character, it can be discussed in interesting ways.

We are always more afraid than we wish to be, but we can always be braver than we expect. Hold on to your heart, and the Aes Sedai cannot harm what is really you, your heart. They are not nearly so far above us as we believed. May you always find water and shade, Egwene. And always remember your heart. - Lord of Chaos

Eben is dead. Would you want to forget your pain if you lost that hulking giant of yours? Have your feelings for him cut away like some spoiled chunk of flesh in an otherwise good roast? - The Gathering Storm

These quotes are examples of characters in the series dealing with difficult situations. In one, a character is yelled at for offering to use her special powers to numb the pain of another person over a death. It brings up a question that those of us that have known pain might find interesting: would we really have wanted our pain erased? And suggests perhaps not. That there is a benefit to it. That we get through it by embracing it by embracing it while still being true to our heart.

He had four rules concerning action and information. Never make a plan without knowing as much as you can of the enemy. Never be afraid to change your plans when you receive new information. Never believe you know everything. And never wait to know everything. - Lord of Chaos

Sometimes the series treats us to logical thinkers, whether they are generals or innkeepers, and gives us little tidbits like this one.

Androl brushed off his hands, smiling. Children were so adaptable. Before them, centuries of tradition, terror and superstition could melt away like butter left too long in the sun. - Towers of Midnight

And sometimes, things to chew over. Isn t that an interesting comment on prejudice? Children aren t born prejudiced; we can see how easily they make friends with anyone. It is sad that we teach them to be, isn t it? Wheel of Time is a fine story, expertly woven. But it is not just a story. It is true, in the sense that it illuminates truths, just like Romeo and Juliet can. In it, I see lessons and echoes of the civil rights era, comments on what makes good leaders, characters wrestling with decisions without adequate data, people putting on a mask of emotions, sacrifice, and the overall theme of the entire series: the power of love and compassion. The inspiration of the series for me well, it is there, but it may take a few more days to figure it out. I think it has something to do with this: The wind rose high and free, to soar in an open sky with no clouds.

2013 is nearly all finished up and so I thought I'd spend a little time writing up what was noteable in the last twelve months. When I did so I found an unfinished draft from the year before. It would be a shame for it to go to waste, so here it is. 2012 was an interesting year in many respects with personal highs and lows. Every year I see a lots of "round-up"-style blog posts on the web, titled things like "2012 in music", which attempt to summarize the highlights of the year in that particular context. Here's JWZ's effort, for example. Often they are prefixed with statements like "2012 was a strong year for music" or whatever. For me, 2012 was not a particularly great year. I discovered quite a lot of stuff that I love that was new to me, but not new in any other sense. In Music, there were a bunch of come-back albums that made the headlines. I picked up both of Orbital's Wonky and Brian Eno's Lux (debatably a comeback: his first ambient record since 1983, his first solo effort since 2005, but his fourth collaborative effort on Warp in the naughties). I've enjoyed them both, but I've already forgotten Wonky and I still haven't fully embraced Lux (and On Land has not been knocked from the top spot when I want to listen to ambience.) There was also Throbbing Gristle's (or X-TG) final effort, a semi/post-TG, partly posthumous double-album swan song effort which, even more than Lux, I still haven't fully digested. In all honesty I think it was eclipsed by the surprise one-off release of a live recording of a TG side project featuring Nik Void of Factory Floor: Carter Tutti Void's Transverse, which is excellent. Ostensibly a four-track release, there's a studio excerpt V4 studio (Slap 1) which is available from (at least) Amazon. There's also a much more obscure fifth "unreleased" track cruX which I managed to "buy" from one of the web shops for zero cost. The other big musical surprise for me last year was Beth Jeans Houghton and the Hooves of Destiny: Yours Truly, Cellophane Nose. I knew nothing of BJH, although it turns out I've heard some of her singles repeatedly on Radio 6, but her band's guitarist Ed Blazey and his partner lived in the flat below me briefly. In that time I managed to get to the pub with him just once, but he kindly gave me a copy of their album on 12" afterwards. It reminds me a bit of Goldfrapp circa "Seventh Tree": I really like it and I'm looking forward to whatever they do next. Reznor's How To Destroy Angels squeezed out An Omen EP which failed to set my world on fire as a coherent collection, despite a few strong songs individually. In movies, sadly once again I'd say most of the things I recall seeing would be "also rans". Prometheus was a disappointment, although I will probably rewatch it in 2D at least once. The final Batman was fun although not groundbreaking to me and it didn't surpass Ledger's efforts in The Dark Knight. Inception remains my favourite Nolan by a long shot. Looper is perhaps the stand-out, not least because it came from nowhere and I managed to avoid any hype. In games, I moaned about having moaning about too many games, most of which are much older than 2012. I started Borderlands 2 after enjoying Borderlands (disqualified on age grounds) but to this day haven't persued it much further. I mostly played the two similar meta-games: The Playstation Plus download free games in a fixed time period and the more sporadic but bountiful humble bundle whack-a-mole. More on these another time. In reading, as is typical I mostly read stuff that was not written in 2012. Of that which was, Charles Stross's The Apocalypse Codex was an improvement over The Fuller Memorandum which I did not enjoy much, but in general I'm finding I much prefer Stross's older work to his newer; David Byrne's How Music Works was my first (and currently last) Google Books ebook purchase, and I read it entirely on a Nexus 7. I thoroughly enjoyed the book but the experience has not made a convert of me away from paper. He leans heavily on his own experiences which is inevitable but fortunately they are wide and numerous. Iain Banks' Stonemouth was an enjoyable romp around a fictional Scottish town (one which, I am reliably informed, is incredibly realistical rendered). One of his "mainstream" novels, It avoided a particular plot pattern that I've grown to dread with Banks, much to my suprise (and pleasure). Finally, the stand-out pleasant surprise novel of the year was Pratchett and Baxter's The Long Earth. With a plot device not unlike Banks' Transition or Stross's Family Trade series, the pair managed to write a journey-book capturing the sense-of-wonder that these multiverse plots are good for. (Or perhaps I have a weakness for them). It's hard to find the lines between Baxter and Pratchett's writing, but the debatably-reincarnated Tibetan Monk-cum-Artificial Intelligence 'Lobsang' must surely be Pratchett's. Pratchett managed to squeeze out another non-Discworld novel (Dodger) as well as a long-overdue short story collection, although I haven't read either of them yet. On to 2013's write-up...

This post is mostly a sum-up of the Wikipedia page History of Australia, with some content taken from History of the British Empire. Both texts are available under the Creative Commons Attribution-ShareAlike License. I do not seem to be able to learn about a new topic without taking notes: in this case I have decided to publish my work, hoping that someone will find it useful. Some very important themes such as the Gold Rush and Australian History during the World Wars have been impudently ignored.

I recently bought a new Lenovo X1 Carbon. It is the new second-generation, type 20A7 laptop, based on Intel s Haswell microarchiteture with the adaptive keyboard. It is the version released in 2014. I also ordered the Thinkpad OneLink Dock which I have returned for the OneLink Pro Dock which I have not yet received. The system is still very new, challenging, and different, but seems to support GNU/Linux reasonably well if you are willing to run a bleeding edge version and/or patch your kernel and if you are not afraid to spend an afternoon or two tweaking things. What follows are my installation notes for Debian testing (jessie) when I installed it in early May 2014. My general impressions about the laptop as a GNU/Linux system and overall are at the end of this write-up.

I had a really productive day today actually working on Debian, as planned for my Mondays. I'm still working through my list of packages, trying to get them all vaguely up to date for jessie. It's mostly just addressing Lintian issues that mostly revolve around old standards versions, with the occasional new upstream release. I've also been doing bug triage where the bugs aren't overwhelming. Today I made uploads for pssh (a new upstream release), pymetrics (mostly just a rebuild), rcs-blame (mostly just a rebuild) and simpleproxy (mostly just a rebuild). I need to revisit simpleproxy, because I'm having problems convincing the resulting binary to be linked correctly for relro. It's weird, because I'm doing everything I'm supposed to be doing and I can see all the other hardening flags being passed in except for this one. I really like simplifying down debian/rules using dh, that really makes things readable. You can see the useful stuff without losing it in all the boilerplate. For some reason I was never a fan of CDBS, but I'm quite liking dh. I think it's because it's less opaque than CDBS.

For some years LVM (the Linux Logical Volume Manager) has been used in most Linux systems. LVM allows one or more storage devices (either disks, partitions, or RAID sets) to be assigned to a Volume Group (VG) some of which can then allocated to a Logical Volume (LVs) which are equivalent to any other block device, a VG can have many LVs. One of the significant features of LVM is that you can create snapshots of a LV. One common use is to have multiple snapshots of a LV for online backups and another is to make a snapshot of a filesystem before making a backup to external storage, the snapshot is unchanging so there s no problem of inconsistencies due to backing up a changing data set. When you create a snapshot it will have the same filesystem label and UUID so you should always mount a LVM device by it s name (which will be /dev/$VGNAME/$LVNAME). One of the problems with the ReiserFS filesystem was that there was no way to know whether a block of storage was a data block, a metadata block, or unused. A reiserfsck --rebuild-tree would find any blocks that appeared to be metadata and treat them as such, deleted files would reappear and file contents which matched metadata (such as a file containing an image of a ReiserFS filesystem) would be treated as metadata. One of the impacts of this was that a hostile user could create a file which would create a SUID root program if the sysadmin ran a --rebuild-tree operation. BTRFS solves the problem of filesystem images by using a filesystem specific UUID in every metadata block. One impact of this is that if you want to duplicate a BTRFS filesystem image and use both copies on the same system you need to regenerate all the checksums of metadata blocks with the new UUID. The way BTRFS works is that filesystems are identified by UUID so having multiple block devices with the same UUID causes the kernel to get confused. Making an LVM snapshot really isn t a good idea in this situation. It s possible to change BTRFS kernel code to avoid some of the problems of duplicate block devices and it s most likely that something will be done about it in future. But it still seems like a bad idea to use LVM with BTRFS. The most common use of LVM is to divide the storage of a single disk or RAID array for the use of multiple filesystems. Each filesystem can be enlarged (through extending the LV and making the filesystem use the space) and snapshots can be taken. With BTRFS you can use subvolumes for the snapshots and the best use of BTRFS (IMHO) is to give it all the storage that s available so there is no need to enlarge a filesystem in typical use. BTRFS supports quotas on subvolumes which aren t really usable yet but in the future will remove the need to create multiple filesystems to control disk space use. An important but less common use of LVM is to migrate a live filesystem to a new disk or RAID array, but this can be done by BTRFS too by adding a new partition or disk to a filesystem and then removing the old one. It doesn t seem that LVM offers any benefits when you use BTRFS. When I first experimented with BTRFS I used LVM but I didn t find any benefit in using LVM and it was only a matter of luck that I didn t use a snapshot and break things. Snapshots of BTRFS Filesystems One reason for creating a snapshot of a filesystem (as opposed to a snapshot of a subvolume) is for making backups of virtual machines without support from inside the virtual machine (EG running an old RHEL5 virtual machine that doesn t have the BTRFS utilities). Another is for running training on virtual servers where you want to create one copy of the filesystem for each student. To solve both these problems I am currently using files in a BTRFS subvolume. The BTRFS kernel code won t touch those files unless I create a loop device so I can only create a loop device for one file at a time. One tip for doing this, don t use names such as /xenstore/vm1 for the files containing filesystem images, use names such as /xenstore/vm1-root. If you try to create a virtual machine named vm1 then Xen will look for a file named vm1 in the current directory before looking in /etc/xen and tries to use a filesystem image as a Xen configuration file. It would be nice if there was a path for Xen configuration files that either didn t include the current directory or included it at the end of the list. Including the current directory in the path is a DOS mistake that should have gone away a long time ago. Psychology and Block Devices ZFS has a similar design to BTRFS in many ways and has some similar issues. But one benefit for ZFS is that it manages block devices in a zpool , first you create a zpool with the block devices and after that you can create ZFS filesystems or ZVOL block devices. I think that most sysadmins would regard a zpool as something similar to LVM (which may or may not be correct depending on how you look at it) and immediately rule out the possibility of running a zpool on LVM. BTRFS looks like a regular Unix filesystem in many ways, you can have a single block device that you mount with the usual mount command. The fact that BTRFS can support multiple block devices in a RAID configuration isn t so obvious and the fact that it implements equivalents to most LVM functionality probably isn t known to most people when they start using it. The most obvious way to start using BTRFS is to use it just like an Ext3/4 filesystem on an LV, and to use LVM snapshots to backup data, this is made even more likely by the fact that there is a program to convert a ext2/3/4 filesystem to BTRFS. This seems likely to cause data loss.

Republished by Slate. Translations available in French (Fran ais), Spanish (Espa ol), Chinese ( ) For almost 15 years, I have run my own email server which I use for all of my non-work correspondence. I do so to keep autonomy, control, and privacy over my email and so that no big company has copies of all of my personal email. A few years ago, I was surprised to find out that my friend Peter Eckersley a very privacy conscious person who is Technology Projects Director at the EFF used Gmail. I asked him why he would willingly give Google copies of all his email. Peter pointed out that if all of your friends use Gmail, Google has your email anyway. Any time I email somebody who uses Gmail and anytime they email me Google has that email. Since our conversation, I have often wondered just how much of my email Google really has. This weekend, I wrote a small program to go through all the email I have kept in my personal inbox since April 2004 (when Gmail was started) to find out. One challenge with answering the question is that many people, like Peter, use Gmail to read, compose, and send email but they configure Gmail to send email from a non-gmail.com From address. To catch these, my program looks through each message s headers that record which computers handled the message on its way to my server and to pick out messages that have traveled through google.com, gmail.com, or googlemail.com. Although I usually filter them, my personal mailbox contains emails sent through a number of mailing lists. Since these mailing lists often hide the true provenance of a message, I exclude all messages that are marked as coming from lists using the (usually invisible) Precedence header. The following graph shows the numbers of emails in my personal inbox each week in red and the subset from Google in blue. Because the number of emails I receive week-to-week tends to vary quite a bit, I ve included a LOESS smoother which shows a moving average over several weeks. From eyeballing the graph, the answer to seems to be that, although it varies, about a third of the email in my inbox comes from Google! Keep in mind that this is all of my personal email and includes automatic and computer generated mail from banks and retailers, etc. Although it is true that Google doesn t have these messages, it suggests that the proportion of my truly personal email that comes via Google is probably much higher. I would also like to know how much of the email I send goes to Google. I can do this by looking at emails in my inbox that I have replied to. This works if I am willing to assume that if I reply to an email sent from Google, it ends up back at Google. In some ways, doing this addresses the problem with the emails from retailers and banks since I am very unlikely to reply to those emails. In this sense, it also reflects a measure of more truly personal email. I ve broken down the proportions of emails I received that come from Google in the graph below for all email (top) and for emails I have replied to (bottom). In the graphs, the size of the dots represents the total number of emails counted to make that proportion. Once again, I ve included the LOESS moving average. The answer is surprisingly large. Despite the fact that I spend hundreds of dollars a year and hours of work to host my own email server, Google has about half of my personal email! Last year, Google delivered 57% of the emails in my inbox that I replied to. They have delivered more than a third of all the email I ve replied to every year since 2006 and more than half since 2010. On the upside, there is some indication that the proportion is going down. So far this year, only 51% of the emails I ve replied to arrived from Google. The numbers are higher than I imagined and reflect somewhat depressing news. They show how it s complicated to think about privacy and autonomy for communication between parties. I m not sure what to do except encourage others to consider, in the wake of the Snowden revelations and everything else, whether you really want Google to have all your email. And half of mine. If you want to run the analysis on your own, you re welcome to the Python and R code I used to produce the numbers and graphs.