Search Results: "ron"

17 November 2013

Charles Plessy: The source of a package developed in a Git repository is that repository.

In his blog, Lars puts words on my feelings of weariness in these situations where we are asked much efforts to make our packages modifiable by contributors who can not or want not to use Git. Most of the core Debian tools, dpkg, debhelper, litnian, debian-installer and many others are developed in Git repositories. Somebody serious about contributing to Debian can hardly avoid Git. It is time for our tools and methods to evolve and take advantage of Git. This requires to say no thank you to contributions based on source packages downloaded from our FTP server, instead of being cloned from the Git repository where the package is actually developed, and this calls the question of what is the real source of the binary packages, but this is a question that is becoming unavoidable.

16 November 2013

Lars Wirzenius: Debian: developing it wrong

This is an essay form of a talk I have given today at the Cambridge Mini-debconf. The talk was videoed, so will presumably show up in the Debconf video archive eventually. Abstract Debian has a long and illustrious history. However, some of the things we do perhaps no longer make as much sense as they used to do. It's a new millennium, and we might find better ways of doing things. Things that used to be difficult might now be easy, if we dare look at things from a fresh perspective. I have been doing that at work, for the Baserock project, and this talk is a compilation of observations based on what I've learnt, concentrating on things that affect the development workflow of package maintainers. Introduction and background I have been a Debian developer since August, 1996 (modulo a couple of retirements), and have used it a little bit longer. I have done a variety of things for Debian, from maintaining PGP 2 packages to writing piuparts to blathering excessively on the mailing lists. My day job is to develop the Baserock system at Codethink. Baserock is a set of tools and workflows for developing embedded and appliance Linux systems. If you squint, it looks a bit like a source-based Linux distribution. I have worked on Baserock since September, 2011. Some of Baserock's design has been influenced by my experience with Debian. With Baserock I have the chance to fix all the things that are wrong in Debian, and this talk is me giving back to Debian by pointing out some the things I feel should be fixed. I don't have solutions for these problems: this is a bug report, not a patch. It's also perhaps a bit of a rant. I am specifically concentrating here on technical and tooling issues that affect the development process of Debian. I excluding social issues. I am also not trying to get Debian to switch to Baserock. Baserock is targeting embedded and appliance systems, and makes simplifying assumptions based on those targets, which Debian does not get to do. I am pointing out problems, and I am outlining solutions as implemented in Baserock, when I think the concept carries over well to Debian. Build tools should be intelligent, packaging should be dumb In Debian, the fundamental tool for compiling the upstream code and assembling a binary package is dpkg-buildpackage. It uses debian/rules, which is a Makefile with targets with specific names and semantics. By executing the right targets in the right order, dpkg-buildpackage tells the source package to build the binary package. On the one hand, this is a nice design, because it abstracts away the large variety of upstream build systems into one API for dpkg-buildpackage to use. On the other hand, it puts all the intelligence for how packages are built into the packages. Making packaging intelligent, rather than the build tool, means packagers need to do more work, and there's more work to do when, say, the Debian policy changes, or when there are other changes that affect a large number of packages. If packaging is intelligent, then every package needs changes. If the build tool is intelligent, then you change the build tool and re-build everything. In Baserock we put as much intelligence as we can into the Morph tool, which drives the build process. It turns out that today, unlike in 1995, most upstream projects use one of a handful well-known build systems: autotools, cpan, cmake, Python distutils, etc. With just a little extra logic in Morph we avoid having any delta against upstream in Baserock. This doesn't work for quite all upstream projects, of course, but we've spent the 2% (two, not twenty) of effort that solves 80% of the problem. In recent years, the dh approach to packaging has made a lot of packages be able to have only a minimal, 3-line debian/rules file. This is excellent. Wouldn't it be nice if even that wasn't needed? It'd save thousands of files in source packages across the archive, at least. It would be easy to: if the file is missing, have dpkg-buildpackage assume the 3-line version by default. Getting rid of a single file is, of course, not a particularly big win. The big win is the change in mindset: rather than dealing with all new issues in development by adding yet more fields to debian/control and more optional, competing tooling outside the core toolset, if you improve the tools everyone uses, then everyone's packages gets better. The goal should, in my opinion, be that for the large number of packages where upstream uses a well-known, well-behaved build system, and uses it in a reasonably sensible way, the Debian source package should not require anything added to make the package build. There will still be a need to add some stuff, such as the debian/copyright file, to make a good Debian package, but just getting the package to build should require nothing extra. (Side note: wouldn't it be nice if there was a well-known, widely used way to mark up copyright information so that debian/copyright could be constructed and updated automatically?) Configuration file handling on upgrades: add ucf to dpkg already In the 1990s, dpkg had excellent handling of configuration files and merging local changes with changes from the new package version, but it was excellent only because it tried to do that at all, and mostly nothing else did. It hasn't changed much, since, and it's not excellent on any absolute scale. We have the ucf tool, which can do a better job, but requires to be added to each package wanting to use it. Why don't we make dpkg smarter, instead? If ucf is not not good enough to be merged wholesale into dpkg, let's write something better. Making every package maintainer use ucf manually is just wasteful. This is not the kind of thing that should be done for each package separately: the package manager should be smart so that packaging can be stupid. The goal should be that dpkg is smart enough in its configuration file handling that having the package do it is a very rare special case. Clean building shouldn't be hard The basic tool for building a package is dpkg-buildpackage. It is ever so slightly cumbersome to use, so there are some wrappers, most importantly debuild. However, if you're making a build intended to be uploaded to the Debian archive, you should be doing a clean build. This means having to learn, configure, and use yet more tools. A clean build is important: security updates, development, debugging, quality control, porting, and user support become more difficult if we don't know how a package was built, and can't reproduce the build. It gets harder to keep build dependencies correct, making it harder for everyone to build things. Luckily, Debian has solved the clean build problem. Unluckily, it has solved it multiple times, creating the problem of having to choose the clean building approach you want to use. The default way of building is not clean, so then you have to remember to use the non-standard way. You also get to spend time maintaining the clean build environments yourself, since that doesn't seem to be fully automated. None of this is hard, as such, but it's extra friction in the development workflow. The primary approaches for clean building in Debian seem to be pbuilder, cowbuilder, and sbuild. I happen to use pbuilder myself, because it's what's been around the longest, but I make no claim of having made an informed choice. That is part of the problem here: why should I have to spend the effort to become informed to make a choice well? Why is the default way of building not clean? Don't say performance: Morph sets up a clean staging area in seconds, and does not offer you a choice of not doing so. It's a chroot, with everything but the build tree hardlinked from cached, unpacked build dependencies, and protected using read-only bind mounts. What's more, this approach avoids having to maintain the pbuilder base tarballs or sbuild chroots manually. It's all automatic, and up to date, for every build. Furthermore, the staging area contains only the specified build dependencies, and not anything else, meaning a build fails if a build dependency is missing, rather than succeeding because it happens to be in the default set of packages the build tool installs. Mass building shouldn't be hard Suppose you want to try out a large-scale change in Debian. It might be trying out a new version of GCC, or using llvm's clang as the default C compiler, or updating glibc, or doing a large library transition such as a new version of GTK+, or trying a new dpkg version, or even something more exploratory such as changing the default optimisation flags for all packages to see what breaks. All of these will require you to at least rebuild all affected packages. Ideally you'd test the built packages as well, but let's concentrate on the building for now. Here's what you do. You make the changes you want to try out, and build those packages. You create a dedicated APT repository, and upload your packages there. You configure your build environment to favour that APT repository. You make a list of all reverse build dependencies of the packages you changed. You write a script to build all of those, preferably so that if you change A, and rebuild B, then you also rebuild C which build depends on B. Each rebuilt package you also upload to your APT repository. You keep track of the build log, and success or failure, of each build. The people in Debian who do this kind of stuff regularly presumably have tools for doing it. It shouldn't be a rare, special thing, though. If my package has reverse build dependencies, I should at least consider testing building when I'm making changes. Otherwise, it might take years until the reverse build dependencies are rebuilt, and the problem is only found then, making it harder to fix. To be fair, building a lot of packages takes a lot of resources. It's not feasible to rebuild everything in Debian every time there's any change to, say, eglibc. However, it's feasible to do it, for large subsets of the archive, without huge hardware investments. One VCS to rule them all In 1995 there was really only one relevant version control system: CVS. It was not a great tool. In 2000, another contender existed: Subversion. It fixed some problems in CVS, but still wasn't a great tool. In 2005, there was a great upheaval and distributed version control systems started to become mainstream. There were a large handful of them. In 2010, it was becoming pretty clear that git had won. It's ugly, but it's powerful. I'm not going to debate the relative merits of different version control systems. Until recently, I was a Bazaar boy, and all of my personal projects were kept in Bazaar. (I have recently switched everything to git.) There are, however, strong benefits from everyone using the same system. Developers don't need to learn a dozen version control systems. Tools that operate on many repositories are easier to write and maintain. Workflows become simpler if one system can be assumed. Debian has a strong historical tendency to choose every option. This is sometimes a good thing, and sometimes a bad thing. For keeping source packages in version control I believe it is to be a bad thing. The status quo is that a Debian source package may not be in version control at all, or it might be in any version control system. This is acceptable when everyone only ever needs to maintain their own packages. However, in a distribution the size of Debian, that is not the case. NMUs, security support, archive wide transitions, and other situations arise when I might need to change yours, or you might need to change mine. We try to work around this by having a complicated source package format, using quilt to maintain patches to upstream semi-manually in a debian/patches directory. This is an awkward workflow. It's a workflow that trips up those that are not used to it. (I know quilt is a patch management system, not a version control system. I think git does it much better anyway.) It would be oh so much easier if everyone kept their source packages in the same, real version control system. I don't even care what it is, as long as it is powerful enough to handle the use cases we have. Imagine a world where every Debian source package is kept, for argument's sake, git, and everyone also uses the same layout and roughly the same workflow to maintain it. What would this mean? It would mean that if you want to inspect the history of your package, you know how to do that. If you want to merge in some bugfix from upstream code, you know how to do that, without having to figure out which of the several source package formats are in use. It would make feasible the development of more powerful, more higher level tooling. For example, it would allow Debian to have what we call system branches in Baserock. In Debian we have stable, testing, unstable, and experimental. We may get something like Ubuntu's PPAs, or perhaps an improved version of those. These are very poor versions of system branches, just like quilt is a poor way to manage patches and deltas against upstream. For example, you can upload an experimental version of gccto experimental, but then nobody else can upload another experimental version. You can set up your own PPA for this, but you'll still be affected by all the uploads to unstable while you're working. A Baserock system branch is a branch of the entire system, or the entire distribution in the case of Debian. It is isolated from other system branches. In branch in an individual repository is a well-known concept. A system branch is conceptually like branching every repository in the distribution at once. The actual implementation is more efficient, of course. This would be possible to implement without standardising on one version control system, but it would be much harder to implement, and would have to live with the lowest common denominator for features. CVS and Subversion, for example, don't really to merges, where Bazaar, Mercurial, and git do. Possible does not mean feasible. Any work you do in a system branch is isolated. Your work doesn't affect others, and theirs doesn't affect yours, until a merge happens. This is a simple, but very powerful tool. Cheap system branches, and powerful merging, makes it possible to do experiments safely, with little fuss. Combine that with being able to build everything cleanly and quickly, and get into a situation where there's no need to have make technical decisions based on arguments on mailing lists, and instead they can be done by choosing by looking at working code. I don't know how this could be implemented in Debian, but think about it. If Debian could have this, it might many archive-scale changes easier. debian/rules clean: really? One of the silliest things we require of packages is that they have a debian/rules clean rule that cleans up after a build perfectly so that we can do repeated builds in the same source tree. Let's just use git clean -fdx instead. This is a problem that is superbly well suited for automation. There is no point whatsover making packagers do any manual work for this. Large scale semi-mechanical changes require too much effort About a decade ago, we decided to follow a new version of the Filesystem Hierarchy Standard and transition from /usr/doc to /usr/share/doc. This was an almost entirely mechanical change: in many cases, a mere rebuild would fix it, and in almost every other case it was just a minor tweak to the packaging. A one-line change. It took us seven years to do this. Seven years. Think about it. In a recent discussion about building R binary data files from source at package build time it was suggested that we take 2-3 release cycles to get this done. That's four to six years. Think about it. These are not isolated cases. Every time we need to make a change that affects more than a small handful of packages, it becomes a major undertaking. Most of the time all the people involved in this are agreeable to the change, and welcome it. The change takes long because it requires co-ordinating a large number of people. Sometimes people are just busy. Sometimes they've left the project, but haven't properly orphaned their packages. Waiting for a timeout for answers about packages drags the process longer. Mechanical changes, or semi-mechanical ones, which are very easy and very quick to do should not take years. They should take an evening, no more. There's no end of changes we might want to do like this. In 2005 and 2006 I filed a few hundred bugs from failed piuparts runs. They're still being fixed, even when the fix is simple, such as adding a postrm script to remove, when the package is purged, a configuration file created by postinst, or starting the service with invoke-rc.d rather than running an init.d script directly. Mostly one-line changes. There's more mechanical changes that might happen. For example, changing the value of the VCS-Browser field when the Debian version control server changes domain names. It's not just about simple, mechanical changes, either. Transitions of important library packages, for example, which require changes to reverse dependencies due to a changed API are another source of pain. What should be a few evenings of build fixes can drag out to months of co-ordination. This is caused partly by technical issues, and partly due to social issues. The main social issue is that we have quite a strong ownership of packages, and NMUs are to be done carefully, and only in some cases. This is both good and bad, and I won't discuss that aspect now. The technical issues are that our tools are primarily intended for maintaining individual packages, rather than all of them together, making it harder to make even simple changes in large number of packages. In addition to easy mass building and system branches, as outlined above, large-scale changes would require testing tools so that you don't build the new package versions, but also test things automatically. Essentially, CI at the distribution level. Conclusion I've listed above a small variety of problems I see in the Debian development processes and tools. They're not the important part of this talk. The important part is that it we, the Debian developers, should look at our tools, and workflows, critically, and improve them when we can. Even small improvements are very useful, when they affect each of our twenty thousand source packages. The important change I argue for here is one in mindset, rather than any specific problem in any tool. We need a mindset of constant, incremental improvement for our tools.

Russ Allbery: Review: Blue Remembered Earth

Review: Blue Remembered Earth, by Alastair Reynolds
Series: Poseidon's Children #1
Publisher: Ace
Copyright: January 2012
Printing: June 2012
ISBN: 0-441-02071-2
Format: Hardcover
Pages: 505
Geoffrey Akinya is one heir to the vast Akinya business empire, which straddles the solar system and has created vast riches from exploration and mining. He wants nothing to do with it. His passion is elephants: long and methodical study of wild herds in Africa, including a slow and careful investigation into their thought processes. His obnoxious and superior cousins have a passion for business and finance and can have the running of the company for all he cares. But Geoffrey's grandmother Eunice, the family matriarch and driving force behind much of their business expansion, has died. Her ashes are brought back to Africa from the Winter Palace in orbit around the Moon where she lived out her final days and scattered in a family ceremony. And, shortly thereafter, Geoffrey's cousins convince him to investigate a safe-deposit box left on the Moon by Eunice to ensure that it doesn't contain anything damaging to the family. What it contains is the first step in a puzzle. Despite himself, Geoffrey and his sister Sunday an artist and family black sheep who lives in a region of the Moon that is one of the sole holdouts against the ubiquitous monitoring common in the rest of the inhabited solar system are slowly pulled into unraveling that puzzle. This leads them into an uneasy alliance with one of the major political forces in the solar system and some startling revelations about their grandmother's actual plans. Blue Remembered Earth is another entry in the currently-popular sub-genre of solar system SF. Interstellar travel is still a dream, but a combination of improved technology, suspended animation, and heavy use of robots and robotic factories has let humans slowly expand into more of the solar system. On Earth, powers have risen and fallen, and the leap into the solar system coincided with a surging and powerful Africa. Extensive undersea colonization has established a new international consortium of sea-floor countries, and Earth politics have realigned into a diplomatic and business struggle between surface and oceanic nations. Unusually for this sort of fiction, most characters of significance in this story are African. The United States and Europe have quietly disappeared in the way that all non-US countries tend to in most other SF. Reynolds doesn't do very much with this setup (and the characters don't seem distinctly African to me). It's just there as the unremarked normal. I liked that. I kind of wish more of African culture had come through in the story, but there are enough SF novels with an assumed and unremarked white (and usually US) future that an assumed and unremarked black African culture is a nice change of pace. I picked up this book in part because it was described as a better 2312. That's a fairly accurate summary (although only helpful to those who have read Kim Stanley Robinson's novel). The scope and setting are similar, although Reynolds's future cuts back on the scope of off-world human colonization, relies more heavily on robotics, and keeps the power base on Earth. Both books are structured as a mystery, although in Blue Remembered Earth that mystery is deliberately planted for the characters. Reynolds is nowhere near as good at set pieces as Robinson, sadly. There's nothing here at the level of some of the scenes on Mercury in 2312. But he is much better at plotting and characterization, and Blue Remembered Earth tells a coherent story with increasing tension and a satisfying ending. Reynolds here uses an interesting strategy for spinning a mystery out of pieces of knowledge and discovery: set the discovery process prior to the novel, use protagonists who were unaware of it, and turn the details into a mystery that they have to investigate. This is an interesting way to address the problem that most discovery processes are slow and not particularly dramatic. It is artificial Eunice creates the plot for the novel but that worked for me. By the end of the book, I think I understand her reasons for doing so, and both Geoffrey and Sunday have good reasons to not be aware of any of the details before they start following the clues that Eunice leaves behind. The mystery of course sends Geoffrey and Sunday on a bit of a tour of the solar system as well as a tour of the past of their family business and the politics of the present. Here, Reynolds draws on an old SF idea: a conflict between groups that want to mechanize space exploration and groups that want to adapt humans for space. Long-time SF readers will be familiar with this approach, and I don't think Reynolds does anything startlingly new with it, but I thought it was a well-executed side plot. And, for once, an SF author realizes that Earth's oceans are as compelling of a place for that conflict to play out as outer space. There are, of course, significant mysteries to be uncovered, but it takes much of the book before those revelations start (even if one of them is almost spoiled in the dust jacket summary). The story up until that point is driven by character, particularly Geoffrey's stubborn annoyance at the rest of his family and his complete lack of interest in being the protagonist of a novel. I should probably warn here that many of the Amazon reviews disliked this book because of the characters, particularly Geoffrey. Apparently he comes across to some as whiny, resentful, and annoying. I, however, didn't have this reaction at all, possibly in part because I wholeheartedly share his opinion of both big business and his cousins. I thoroughly enjoyed reading an SF novel whose main characters are not driven either typical capitalist motives or a desire for personal exploration, but instead are pulled into the plot through a sense of loyalty and an inability to get out of the way of the problem. I also liked the way Reynolds manages to broaden and deepen the reader's appreciation for the characters as the story unfolds, and even salvages some characters I thought were unsalvageable. The last third of the book features some excellent plot twists and some very nicely-handled revelations. This is apparently the first book of a trilogy, and I had a suspicion at the end of the book that sequels were coming, but I think it stands on its own quite well. Most of the questions raised by the plot are resolved by the end of the book, and those that remain are of a size that other books have left unanswered intentionally. Despite not reaching the same moments of awe-inspiring beauty, and despite more workman-like descriptions, the stronger plot and more interesting characters do make Blue Remembered Earth a better 2312 for me. Given that 2312 won a Nebula, that's saying something (although I don't agree with that award). It's a bit deliberate in its pacing, and the plot built around a constructed puzzle can feel a bit artificial, but I think it's a solid and satisfying near-space SF novel. Followed by On the Steel Breeze. Rating: 7 out of 10

14 November 2013

Russ Allbery: Review: Water Logic

Review: Water Logic, by Laurie J. Marks
Series: Elemental Logic #3
Publisher: Small Beer Press
Copyright: 2007
ISBN: 1-931520-23-2
Format: Trade paperback
Pages: 332
Water Logic is the third book in the Elemental Logic series, starting with Fire Logic. As with Earth Logic, it's best to read all three books in close temporal proximity, since Water Logic offers little recap of what happened in the previous books and relies on a large cast of characters who are not reintroduced. It's also hard to review without spoilers for the previous books. Earth Logic ended with a monumental shift in Shaftal politics, and half of Water Logic is deeply concerned with the implications of that shift and where Shaftal should go from here. Karis and Zanja are, of course, deeply involved, but much of the political story is told from the perspective of the character introduced in Earth Logic, and from the perspective of another earth blood, Meriseth. Water bloods haven't been seen much so far in the series. Water magic is about repeated patterns and the malleability of time, and the plot of Water Logic eventually entangles Zanja in events far in the past of Shaftal. This proceeds in parallel with more of Marks's excellent analysis of the meaning of war and peace, diplomacy, cooperation, multiculturalism, adaptation, and empathy. The two sides of the book do to some extent reinforce each other, but the connections are tricky and less obvious. Parts of this book are excellent. Wars are complicated and powerful, and don't end easily. The mutual incomprehension and mutual (if intermittant) struggle for understanding between the natives of Shaftal and the Sainnites is beautifully presented. I particularly loved Mariseth's approach throughout the book. Earth bloods are perhaps Marks's best invention, and seeing another (besides Karis) at work, with the same stubborn patience but with different tactics and different levels of power, was a delight. I feel like reading, and thinking about, Seth's decisions makes me a better person. As with the previous books, though, parts of Water Logic are a bit boring, or at least don't seem to coherently move in a conclusive direction. Zanja has long journeys across the landscape as well as several adventures after she's grabbed by water logic that seem only distantly connected to the rest of the story. Marks does pull this mostly together in the end, and I could detect other thematic linkages while mulling the book over after finishing it, but the book feels a bit disjointed while reading it. And there are some elements, such as the cowdogs, that are neat bits of world-building but that seem tossed in just because they're neat and don't feel strongly connected to the plot. The other complaint I have, which would probably have been lessened by reading the books closer together, is that the cast is a bit too large. All of Karis's extended family are, of course, still around, which I found more confusing than helpful. Named and important characters keep popping in and then disappearing, with no introduction and no help for the reader who doesn't remember the exact details of their roles from previous books. It's a bit frustrating, and it makes the book feel a bit cluttered, although it does a great job of showing the dynamics of a large and comfortable family. (Marks shows the dynamics of family webs with multiple adults better than any other fantasy author I know.) Karis is a bit more sidelined in this book, casting her in the role of force of nature, and I think that works well. It took me a bit to warm to Zanja again, but by the end of the book I was greatly enjoying her sections. But the star, for me, is Seth. Even though some of her segments should have been the boring parts of the book, since they were mostly concerned with quitodian issues, they were the parts I wanted to read about. I was very touched by the ending, even if I think a bit too much authorial hand-waving was required. There is material left for a sequel at the end of Water Logic, including a villain who has yet to fully appear, but it's currently not clear whether Air Logic will ever materialize. But, as with Earth Logic, Water Logic provides a satisfying ending and leaves the world in a stable enough place that I didn't feel cheated out of the end of the story. I think Fire Logic remains the strongest book of this series, but Water Logic is very much worth reading if you've read the previous books. Unfortunately, the books apparently didn't sell well, so this book is from a small press instead of Tor. You may have to order it on-line instead of finding it a local bookstore. But the whole series is very much worth reading. It's a refreshingly original take on fantasy and is much more thoughtful about human relationships and politics than most fantasy works. Rating: 7 out of 10

13 November 2013

Andrea Veri: Configuring DNSSEC on your personal domain

Today I ll be working out how to properly configure DNSSEC on a BIND9 installation, I ll also make sure to give you all the needed instructions to properly verify if a specific domain is being correctly covered by DNSSEC itself. In addition to that a few more details will be provided about adding the relevant SSHFP s entries on your DNS zone files to be able to automatically verify the authenticity of your domain when connecting to it with SSH avoiding any possible MITM attack. First of all, let s create the Zone Signing Key (ZSK) which is the key that will be responsible to sign any other record on the zone file which is not a DNSKEY record itself:
dnssec-keygen -a RSASHA1 -b 1024 -n ZONE gnome.org


Note: the dnssec-keygen binary file should be part of bind97 (RHEL 5) or bind (RHEL6) package according to yum whatprovides: RHEL 5:
32:bind97-9.7.0-17.P2.el5_9.2.x86_64 : The Berkeley Internet Name Domain (BIND) DNS (Domain Name System) server
Repo : rhel-x86_64-server-5
Matched from:
Filename : /usr/sbin/dnssec-keygen


RHEL 6:
32:bind-9.8.2-0.17.rc1.el6.3.x86_64 : The Berkeley Internet Name Domain (BIND) DNS (Domain Name System) server
Repo : rhel-x86_64-server-6
Matched from:
Filename : /usr/sbin/dnssec-keygen


Then, create the Key Signing Key (KSK), which will be used to sign all the DNSKEY records:
dnssec-keygen -a RSASHA1 -b 2048 -n ZONE -f KSK gnome.org


Creating the above keys can take several minutes, when done copy the public keys to the zone file this way:
cat Kgnome.org*.key >> gnome.org


When done you can clean out the useless bits from the zone file and just leave the DNSKEY records (which are not commented out as you will notice) An additional and cleaner way of accomplishing the above is to use the INCLUDE rule on the zone file itself as follows:
$INCLUDE /srv/dnssec-keys/Kgnome.org+005+12345.key
$INCLUDE /srv/dnssec-keys/Kgnome.org+005+67890.key


Choosing which method to use is really up to you. Once that is done you can go ahead and sign the zone file. As of myself I m making use of the do-domain script taken from the Fedora Infrastructure Team s repositories. If you are going to use it yourself, make sure to adjust all the relevant variables to match your setup, especially keyspath, region_zones, template_zone, signed_zones and AREA. The do-domain script also checks your zone file through named-checkzone before signing it.
/me while editing the do-domains script with the preview of gnome-code-assistance!

/me while editing the do-domains script with the preview of gnome-code-assistance!

If instead you don t want to use the script above, you can sign the zone file manually in the following way:
dnssec-signzone -K /path/to/your/dnssec/keys -e +3024000 -N INCREMENT gnome.org


By default, the above command will append .signed to the file name, you can modify that behaviour by appending the -f flag to the dnssec-signzone call. The -N INCREMENT will increment the serial number automatically making use of the RFC 1982 arithmetics while the -e flag will extend the zone signature end date from the default 30 days to 35. (this way we can safely run a monthly cron job that will sign the zone file automatically) You can make use of the following script to achieve the above:
#!/bin/sh
SIGNZONE="/usr/sbin/dnssec-signzone"
DNSSEC_KEYS="/srv/dnssec-keys"
NAMEDCHROOT="/var/named/chroot"
ZONEFILES="gnome.org"
cd $NAMEDCHROOT
for ZONE in $ZONEFILES; do
$SIGNZONE -K $DNSSEC_KEYS -e +3024000 -f $ZONE.signed -N INCREMENT $ZONE
done
/sbin/service named reload


Once the zone file has been signed just make sure to include it on named.conf and restart named:
zone "gnome.org"  
file "gnome.org.signed";
 ;


When you re done with that we should be moving ahead adding a DS record for our domain at our domain registrar. My example is taken from the known gandi.net registrar.
gandi

Gandi s DNSSEC interface

Select KSK (257) and (RSA/SHA-1) on the dropdown list and paste your public key on the box. You will find the public key you need on one of the Kgnome.org*.key files, you should look for the DNSKEY 257 entry as dig DNSKEY gnome.org shows:
;; ANSWER SECTION:
gnome.org. 888 IN DNSKEY 257 3 5 AwEAAbRD7AymDFuKc2iXta7HXZMleMkUMwjOZTsn4f75ZUp0of8TJdlU DtFtqifEBnFcGJU5r+ZVvkBKQ0qDTTjayL54Nz56XGGoIBj6XxbG8Es+ VbZCg0RsetDk5EsxLst0egrvOXga27jbsJ+7Me3D5Xp1bkBnQMrXEXQ9 C43QfO2KUWJVljo1Bii3fTfnHSLRUsbRn8Puz+orK71qxs3G9mgGR6rm n91brkpfmHKr3S9Rbxq8iDRWDPiCaWkI7qfASdFk4TLV0gSVlA3OxyW9 TCkPZStZ5r/WRW2jhUY/kjHERQd4qX5dHAuYrjJSV99P6FfCFXoJ3ty5 s3fl1RZaTo8=


Once that is done you should have a fully covered DNSSEC domain, you can verify that this way:
dig . DNSKEY   grep -Ev '^($ ;)' > root.keys
dig +sigchase +trusted-key=./root.keys gnome.org. A   cat -n


The result:
105 ;; WE HAVE MATERIAL, WE NOW DO VALIDATION
106 ;; VERIFYING DS RRset for org. with DNSKEY:59085: success
107 ;; OK We found DNSKEY (or more) to validate the RRset
108 ;; Ok, find a Trusted Key in the DNSKEY RRset: 19036
109 ;; VERIFYING DNSKEY RRset for . with DNSKEY:19036: success
110
111 ;; Ok this DNSKEY is a Trusted Key, DNSSEC validation is ok: SUCCESS

Bonus content: Adding SSHFP entries for your domain and verifying them You can retrieve the SSHFP entries for a specific host with the following command:
ssh-keygen -r $(hostname --fqdn) -f /etc/ssh/ssh_host_rsa_key.pub


When retrieved just add the SSHFP entry on the zone file for your domain and verify it:
ssh -oVerifyHostKeyDNS=yes -v subdomain.gnome.org


Or directly add the above parameter into your /etc/ssh/ssh_config file this way:
VerifyHostKeyDNS=yes


And run ssh -v subdomain.gnome.org , the result you should receive:
debug1: Server host key: RSA 00:39:fd:1a:a4:2c:6b:28:b8:2e:95:31:c2:90:72:03
debug1: found 1 secure fingerprints in DNS
debug1: matching host key fingerprint found in DNS
debug1: ssh_rsa_verify: signature correct

That s it! Enjoy!

Joachim Breitner: Constructing a list in a Monad

In Haskell, lists are ubiquitous. In real-world Haskell, monads are ubiquitous. So it should be an easy task to fill a list with values obtained in a monad, i.e. a list of n random numbers (in the Rand monad), or getting some input from the user, and storing it in a list, until the user chooses to finish the input. The task For now, assume a function getInput :: IO Int; we want to run it 10000 and then have the list of the values returned. Sounds simple and straight forward, but it is actually really hard to get the run-time behaviour that we naively and optimistically expect. The expectation In an imperative setting, we would have a simple counting loop. In the loop, we run getInput, allocate a new cell for the linked list, and append it to the list. Clearly, this
  • consumes no stack space and
  • at the end of the loop, the list is already fully evaluated in memory, ready to be used.
Can we achieve that with Haskell? Let s look at a few alternatives (complete code available)
Attempt 1: The accumulator I ll start with the probably worst attempt. Unfortunately, from what I have seen from correcting Haskell exams, this is what beginners would write, especially if they have the above imperative algorithm in mind, and know about the convert loops to recursion with accumulators approach to solving Haskell problems:
accumAppend :: Int -> IO [Int]
accumAppend n = go n []
  where
    go 0 l = return l
    go n l = do  x <- getInput; go (n-1) (l++[x]) 
To check the space usage of this, I pass -Kn to the GHC runtime and see what limit is just enough to let this program calculate accumAppend 10000 and fully evaluate the resulting list. It turns out that we need 328336 bytes of stack space. So what about the second requirement; that the list is fully evaluated? Experienced Haskellers see it immediatelly: The accumulator is not used in the loop, so each iteration adds a thunk around the previous value, that would add the next entry to the end. Only when the list is actually used, these are run, each traversing the whole list, causing quadratic cost. We can verify this using ghc-vis:
Clearly (and not surprisingly) this is not the best solution. Attempt 2: The accumulator (forced) Lets try to fix the second issue, by making sure that the list is always fully evaluated. We import Control.DeepSeq and write
accumAppendSeq :: Int -> IO [Int]
accumAppendSeq n = go n []
  where
    go 0 l = return l
    go n l = do  x <- getInput ; go (n-1) $!! (l++[x]) 
As the accumulator gets fully evaluated before being passed to the next invocation of go, the result will not contain thunks. But note that this has also fixed the stack consumption: The code now runs the smallest setting for -K, 8 bytes. This works as both go and deepseq are tail-recursive. But this is still not the solution we are looking for, as the quadradic cost caused by using ++ is still there. Attempt 3: The accumulator (reversed) Since we know that l++[x] is expensive, but x:l is cheap, we can simply use this to update the accumulator. This has the slightly annoying consequence that the entries of the list are in the reverse order, but we can fix that in the base case:
accumReverse :: Int -> IO [Int]
accumReverse n = go n []
  where
    go 0 l = return (reverse l)
    go n l = do  x <- getInput ; go (n-1) (x:l) 
And indeed, we get noticably faster code that still runs in 8 bytes of stack. Unfortunately, we still don t construct the final list directly. Attempt 4: Naive recursion Maybe we can do without an accumulator. The most naive such code would be
listRecurse :: Int -> IO [Int]
listRecurse n = go n
  where
    go 0 = return []
    go n = do  x <- getInput; r <- go (n-1); return (x:r) 
and I d expect that most experienced Haskellers would write that code (if they are asked not to use a library function). And indeed, this code does create the final list directly. But it requires a large stack or 164616 bytes. The reason is that this is no longer tail recursive: After calling go again we need to take the result and prepend x Attempt 5: library functions Maybe we should simply not worry about the implementation, and use library functions? Clearly, the authors of such libraries have found the best solution. So how does
listReplicateM :: Int -> IO [Int]
listReplicateM n = Control.Monad.replicateM n getInput
fare? Unfortunately, not better than the naive recursion. In fact, the stack space requirement is the same 164616 bytes. The same holds when using special libraries for generating and consuming list, e.g. the Streams module from the vector package:
listStreams :: Int -> IO [Int]
listStreams n = MStream.toList $ MStream.replicateM n getInput
Attempt 6: Difference lists It is time to dig deeper into our box of tricks. What if we have lists with constant time Snoc (a.k.a. appending a new element)? Then Attempt 3 would not require reversing the list. Such lists are provided by difference lists; there are libraries for that, but we can simply implement them using lambdas:
accumDList :: Int -> IO [Int]
accumDList n = go n id
  where
    go 0 l = return (l [])
    go n l = do  x <- getInput; go (n-1) (l . (x:)) 
Indeed, no stack is required (8 bytes are enough). But we are cheating here: We are still not constructing the final list, but rather we combine functions that append elements to lists. So when accumDList returns, what we have in memory, is a thunk and sequence of partial function applications in the wrong order, and evaluating the thunk does basically the same thing as reverse in Attempt 3. It might be a bit faster, is still not satisfying enough:
Attempt 7: Unsafe functions Even deepter in the box of tricks, somewhere in the grit and dust where one usually should not reach, there is a function that can help us out here: unsafeInterleaveIO. This allow us to modify the naive recursion in a way that first creates the Cons-cell, and then continues the loop:
listUnsafeInterleave :: Int -> IO [Int]
listUnsafeInterleave n = do  l <- go n; return $!! l 
  where
    go 0 = return []
    go n = do x <- getInput
              r <- unsafeInterleaveIO (go (n-1))
              return (x:r)
The stack consumption is 8 bytes. It is worth noting that the all to deepseq (in $!!) will actually drive the evaluation of go. Also, it guarantees that the unsafe effects of unsafeInterleaveIO are confined to the execution of listUnsafeInterleave, i.e. are actually safe. Run time statistics I ran criterion over the functions (generating lists of length 1000), and found these results:
  • accumAppend: 6.5 ms
  • accumAppendSeq: 8.2 ms
  • accumReverse: 14.0 us
  • listRecurse: 8.0 us (same as listReplicateM and listStreams)
  • accumDList: 12.0 us
  • listUnsafeInterleave: 174.0 us
Conclusion There are many ways to write a list-constructing list in Haskell, none are perfect they either fill the heap, or do not construct the list directly (usually requiring a second pass through it), or rely on IO-specific unsafe functions. The naive recursion performs the best, but may cause a stack overflow (note, though that from GHC 7.8, the stack will be unlimited). The next best option seems to be the difference lists
I m still wondering: What would be required from Haskell, GHC or the monads in question to have a fully satisfactory solution here, i.e. one that is as fast as the naive recursion, as stack efficient as the difference list solution, and allocates no thunks, only list cells?

Raphael Geissert: A bashism a week: heredocs

One great feature of POSIX shells is the so-called heredoc. They are even available in languages such as Perl, PHP, and Ruby.

So where is the bashism?

It's in the implementation. What odd thing do you see below?


$ strace -fqe open bash -c 'cat <<EOF
foo
EOF' 2>&1 grep -v /lib
open("/etc/ld.so.cache", O_RDONLY O_CLOEXEC) = 3
open("/dev/tty", O_RDWR O_NONBLOCK O_LARGEFILE) = 3
open("/proc/meminfo", O_RDONLY O_CLOEXEC) = 3
[pid 6696] open("/tmp/sh-thd-1384296303", O_WRONLY O_CREAT O_EXCL O_TRUNC O_LARGEFILE, 0600) = 3
[pid 6696] open("/tmp/sh-thd-1384296303", O_RDONLY O_LARGEFILE) = 4
[pid 6696] open("/etc/ld.so.cache", O_RDONLY O_CLOEXEC) = 3
foo
--- SIGCHLD (Child exited) @ 0 (0) ---


Yes, it uses temporary files!

So do ksh, pdksh, mksh, posh and possible other shells. Busybox's sh and dash do not use temporary files, though:


$ strace -fqe open dash -c 'cat <<EOF
foo
EOF' 2>&1 grep -v /lib
open("/etc/ld.so.cache", O_RDONLY O_CLOEXEC) = 3
[pid 6767] open("/etc/ld.so.cache", O_RDONLY O_CLOEXEC) = 3
foo
--- SIGCHLD (Child exited) @ 0 (0) ---


Next time you want data to never hit a hard disk, beware that heredocuments and herestrings are best avoided.

12 November 2013

Russell Coker: Bad Advertising of Protoplast

Carpal Tunnel Syndrome In early 2010 I wrote about Carpal Tunnel Syndrome, I gave a picture of the wrist brace and a description of the process of creating it [1]. A few months later I wrote a follow-up about how the wrist brace had helped me, at the time I was anticipating wearing it at night for the rest of my life [2]. But my wrist has continued to improve. Nowadays I never wear the wrist brace and hardly even think of it. I am a little more careful about posture than I was before, but mostly it has no impact on my life. I think this counts as a full recovery. At the time I was mildly interested in buying some thermo-plastic of that type to make things, but the only source I found was Jaycar which sold 100 grams for $14.95, Jaycar prices have now dropped to $11.50 for 100 grams or $89.50 for a kilo but it s still expensive [3]. Protoplast When looking at the sites that link to my blog I saw a site advertising Protoplast which is a type of thermo-plastic that can be used for making things [4]. Protoplast have more reasonable prices, E30 ($AU42.75) for a kilo and their web site offers better prices for anyone who purchases 10kg or more (good for hackerspaces etc). As they post no further than Belgium they aren t an option for me but they do seem to make a good offer, better than Jaycar offers in Australia. Protoplast is probably a good product, but I think that their advert is misleading. While it is probably possible to make a wrist-brace similar to mine with Protoplast it s a fact that my wrist brace was not made with it. My wrist brace started out as a flat sheet of plastic which could be formed to the final product without being melted. Converting a packet of plastic granules (as sold by Protoplast or Jaycar) to a sheet to even start work on such a wrist brace would take some work. To make a sheet of plastic out of beads sold by Jaycar or Protoplast you would ideally completely melt the plastic, if you just softened the plastic then it wouldn t be as strong as using a pre-made sheet. During the time that I wore the wrist brace I probably repaired it more than 30 times, this wasn t particularly difficult to do, I poured boiling water over it to soften the part that cracked until it became tacky and then pushed it together again. But the result was significantly weaker than the original due to the difficulty in welding the broken sections together. I ended up cutting some strips from less important parts of the wrist brace (the part nearest to my elbow) to reinforce the weakest part which was between my thumb and fore-finger. That section ended up being about three times thicker than it was originally while also not being as strong as it was originally. I also had difficulty in making it flat so it wasn t nearly as comfortable as the original version. It seems rather misleading to include the picture of my wrist brace on a site advertising plastic which wasn t used to make it and which probably couldn t be used to make something that looks as good with the same ratio of strength to mass. But with a suitable disclaimer indicating that the user probably can t get the same result it would be OK to give it as an example of what could be done. I can imagine someone using a product like Protoplast to make a wrist brace that can hold a mobile phone, that can be worn with a watch, or has other features that aren t offered by medical staff. License Issues The license for my blog doesn t permit people to just copy pictures, it s a Creative Commons Attribution-Noncommercial-Share Alike based license [5]. I am interested in supporting companies that make interesting products and anything that can help people with medical issues is also of interest. So I wouldn t rule out the possibility of a free license for a picture from my blog to be used in the way that they used it (with a link back to the original blog post). But I would want a description that informs readers that it s not an item made with their product, merely something similar that can be used for inspiration.

11 November 2013

Simon Josefsson: Using Replicant on Samsung Galaxy S III

For the last half-year I have used CyanogenMod on an Nexus 4 as my main phone. Recently the touch functionality stopped working on parts of the display, and the glass on the back has started to crack. It seems modern phones are not built to last. For comparison, before the N4 I used a Nokia N900 for around 3 years without any hardware damages (in my drawer now, still working). A few weeks ago I started looking for a replacement. My experience with CyanogenMod had been good, but the number of proprietary blobs on the N4 concerned me. Finding something better wasn t easy though, so I m documenting my experience here. My requirements were, briefly, that I wanted a phone that I could buy locally that had a free software community around it that produced a stable environment. I have modest requirements for things I wouldn t give up on: telephony, data connection (3G), email (IMAP+SMTP), chat (XMPP), and a web browser. I like the philosophy and openness around the Firefox OS but the more I have read about it, it seems unlikely that it would deliver what I need today. In particular none of the devices capable of running Firefox OS appealed to me, and the state of email reading seemed unclear. I m sure I ll revisit Firefox OS as an alternative for me in the future.

As I had been happy with CyanogenMod, but concerned about its freeness, it felt natural to move on and test the more free software friendly project Replicant. Replicant only supports a small number of devices. After talking with people in the #replicant IRC channel, it seemed the Samsung S3 would be a decent choice for me. The Samsung S2 would have worked as well, but it cost almost as much as the S3 where I looked. Despite the large number of Samsung S3 devices out there, it seems the prices even for used devices are high (around 2500 SEK in Sweden, ~380 USD). I ended up buying a brand new one for 3200 SEK (~500 USD) which felt expensive, especially after recalling the recent $199 sale for Nexus 4. Noticing that brand new Nexus 4 devices are still over 3000 SEK in Sweden comforted me a bit. I would have preferred a more robust phone, like the CAT B15, but the state of free software OSes on them seem unclear and I wanted something stable. So, enough about the background, let s get started. Building and installing Replicant on the device was straight forward. I followed the Replicant Samsung S3 Build instructions to build my own images. The only issue I had was that I had not set JAVA_HOME and the defaults were bad; make sure to set JAVA_HOME before building. I built everything on my Lenovo X201 running Debian Wheezy, with OpenJDK 6 as the Java implementation. Installing the newly built firmware was easy, I just followed the installation process documentation. I made sure to take a clockworkmod backup to an external SD card before wiping the old system. To get a really clean new device, I also re-formated /sdcard inside clockworkmod; I noticed there were some traces left of the old system there. I spent about one week testing various configurations before settling on something I could use daily. A fair amount of time was spent looking into backup and restore options for Android devices. My idea was that I would take a backup of the apps I ran on the N4 and transfer them to the S3. The Android Debug Bridge (adb) has a backup/restore command, however it (intentionally) ignores apps marked as allowBackup=false which a number of apps has. It doesn t seem possible to override that settings so much for the freedom to backup your own device. I then discovered oandbackup. It can backup your entire system, saving each app (together with associated data) into a separate directory, for simple review and inspection before restore. You can do batch backups and batch restore. I couldn t get it to automatically restore things, though, which would be neat for really automated re-installations (there is an open issue about this feature). After noticing that some apps did not like being moved from the N4 (running Android 4.2) to the S3 (running 4.0), I ended up installing most apps from scratch on a freshly installed Replicant. I use oandbackup to the external SD card so that I can quickly restore my phone. For backup/restore of SMS/MMS and Call Log, I use SMS Backup+ against my own IMAP server. Camera pictures are synced manually using adb when I am connected to my laptop. There is a number of apps that deserve to be mentioned because they are what I use on a daily basis. All of them come via the free software market F-Droid. For email (IMAP/SMTP), I use K-9 Mail which is feature rich but still easy to use. For chat, I use Xabber. I use NewsBlur s free software app to read RSS flows. For two-factor authentication, I use Google Authenticator. I haven t evaluated different PDF viewers, but the first one I tried (APV PDF Viewer) has worked fine so far. Handling a a synchronized address book and calendar deserve its own blog post because it is a challenging topic, but briefly, I m currently using a combination of aCal and DAVdroid. Finally, since Replicant is still work in progress, some words about stability and notes on what doesn t work. This is probably the most interesting part if you are considering running Replicant on an S3 yourself. Overall system stability is flawless, I hadn t had any crash or problem with the fundamental functionality (telephony, 3G, Camera). People have said graphics feels a bit laggy, but I cannot compare with the stock ROM and it doesn t get in the way of daily use. First some notes about non-free aspects:
  • Bluetooth doesn t work by default. After installing /system/vendor/firmware/bcm4334.hcd (MD5 b6207104da0ca4a0b1da66448af7ed4c) pairing and testing with a Bluetooth headset worked fine.
  • Front camera doesn t work by default. After installing /system/vendor/firmware/fimc_is_fw.bin (MD5 52eeaf0889bc9206860075cd9b7f80b9and) and /system/vendor/firmware/setfile.bin (MD5 0e6fdeb378fb154c39fd508ae28eaf2a) it works. The extensions are *.bin but I don t believe this code is executed on the main CPU.
  • GPS doesn t work by default. After installing /system/bin/gpsd (MD5 6757ed2e2a283259e67c62e6b2b9cfef), /system/lib/libsecril-client.so (MD5 a836df0f947d2aa2ef20dcb213317517), /system/lib/hw/gps.exynos4.so (MD5 1ea1d67f297dd52d59d40dbad9b02a0a) it works. This is code that runs on the main CPU, and even more alarming, it embeds a copy of OpenSSL and talks to various online servers for A-GPS, and is thus a likely (and probably not too challenging) attack vector for anyone wanting to remotely exploit any phone.
  • Wifi doesn t work, and I haven t gotten this to work. There is a list of non-free S3 firmware on the Replicant wiki however my stock ROM did not ship with those files. I don t believe any of the blobs run on the main CPU.
  • NFC doesn t work, and I haven t gotten it to work. It seems the infrastructure for NFC is missing in Replicant 4.0, it doesn t even expose the NFC hardware permission capability. This is quite unfortunate for me, since I daily work with YubiKey NEO and would have preferred to replace Google Authenticator with the YubiOATH that uses the NEO for OATH secret storage.
Some other observations:
  • Panorama mode in the Camera crashes; see issue about it.
  • There is a number of smaller graphical issues. I believe these are related to the EGL but haven t understood the details. What I ve noticed are the following issues. The task switcher doesn t show mini screenshots of all running apps (the screenshots are just black). ZXing is not able to QR decode images (I m told this is because Replicant uses a RGB color plane instead of the required YUV color plane). Video playback in the gallery is laggy to the point of being unusable. Video playback on Youtube in the default web browser works in full screen (not laggy), but not embedded in the webpage.
  • MTP has been a bit unreliable, my main laptop is able to import photos, but another laptop (also running Debian Wheezy) just stalls when talking to it. This may be a host issue, I have experience similar issues with a Nexus 7 2nd generation device.
I am quite happy with the setup so far, and I will continue to use it as my primary phone. flattr this!

Matthew Palmer: Timezones are not optional information

In high school, I had a science teacher who would mark your answers wrong if you forgot to include units. What does that mean? , he would write, 17.2 elephants? The point he was trying to get across was that a bare number, without the relevant units, wasn t precise enough to be useful. Also, carrying the units with you helped to cross-check your work if you got a numeric answer, but the units were garbage (seconds per kilogram, for instance, when you were trying to find an acceleration), then you could be pretty sure you d made a mistake somewhere. Fast forward to today, and I m currently working with a database containing a pile of timestamps. Without timezones. Where the timestamps being inserted are in local time. Thankfully, so far, this particular system has always been run in one timezone, so I ve only got one timezone to deal with, but the potential ramifications of systems with different timezones inserting data into this database terrify me. The naive answer is to just store everything in UTC and be done with it. I m not particularly averse to that solution, as long as it s very clear to everyone what s going on. The correct answer, though, I think, is to always keep timezone information with your timestamps otherwise, you ll never know whether it s 0830 in elephants

10 November 2013

Steve Kemp: So I have a new project

Recently I decided to set myself a big photography challenge. The three options which I discussed with a couple of people were:
  • Photograph the front of every pub in the nearby area city-centre.
  • Photograph ever plaque, monument, and statue in the city-centre.
  • Photograph every gravestone and memorial bench in the city centre.
Ultimately I decided pubs would be most fun. Not least because you could do it every year or two, to see what changes occurred. To make it more useful I decided to not only take the pictures, but to collect, and share, the meta-data too:
  • Lat/Longditude GPS for each pub.
  • Contact details for each pub.
  • etc.
Today I spent an hour walking up Easter road, and down Leith Walk. I shot the outside of about 20 pubs, and then fiddled with the layout and organization of the images. I'm reasonably happy with the result, but it remains obvious that I'm not a designer. The data-set use to generate the site - which is perhaps the most interesting/useful part of the whole exercise to other people - is available online too: All the data, even the images, is stored on github for collaboration purposes. I'm not sure if folk will join in, but I can probably manage a few of the major thoroughfares every weekend indefinitely it will only take a couple of days to get "city-wide coverage", then the rest is gravy.

9 November 2013

Carl Chenet: Brebis 0.8, the fully automated backup checker, released

Follow me also on Twitter The new version of Brebis "Claquette" 0.8 was published two days ago. Brebis is a fully automated backup checker. Brebis does not create backups. It only performs controls on them. Brebis is able to detect a corrupted archive compressed with different formats (tar, gzip, bzip2, lzma and zip), automatically checking the integrity of your archives and lots of parameters of the files inside these archives or inside a tree of files and provide detailed reports if unexpected states are detected. Check out the list of the supported features. What s new? The main features of this new version are:
  • new parameter sha512 in the Brebis configuration file to verify the authenticity of the list of files.
  • support for a GPG signed configuration file of Brebis, allowing to verify the authenticity of the backup checking configuration.
The official archive of Brebis 0.8 is available in Debian Sid or from sources at http://brebisproject.org/attachments/download/9/brebis-0.8.tar.gz The documentation has improved and a full example of how to secure you backup checking process is now available. Feedback about Brebis What do you think about the Brebis project ? We at the Brebis Project welcome any feedback about Brebis. Feel free to comment on this blog, to subscribe to the Brebis-users mailing list, by Twitter or email me directly at carl.chenet@brebisproject.org
Official website: http://www.brebisproject.org
Mailing-list: http://lists.sourceforge.net/lists/listinfo/brebis-users

Russ Allbery: Review: Fantasy & Science Fiction, July/August 2011

Review: Fantasy & Science Fiction, July/August 2011
Editor: Gordon van Gelder
Issue: Volume 121, No. 1 & 2
ISSN: 1095-8258
Pages: 258
Nothing of particular interest in the book reviews in this issue, although I was entertained to see Charles de Lint review a collection of the Prince Valiant strip. I have memories of that strip being one of the most boring works of art created by mankind. De Lint, of course, quite likes it. Our tastes seem to be very disjoint, although I have to admit that I've not read it collected and it may be more coherent and more interesting in that format. The science column in this issue, by Paul Doherty and Pat Murphy, deserves special mention. It's on roshambo (rock-paper-scissors), human difficulties with randomness, and strategy in roshambo competitions. It's also the first essay I've read that clearly explains how there can be strategy to roshambo, and why that strategy is worth studying. Even though the primary topic of the column is randomness, it's worth reading from the perspective of strategy in competitive human vs. human games. "Bronsky's Dates with Death" by Peter David: Anyone familiar with PAD's body of work will immediately expect a humorous story with some deeper thoughtful bits, and that's exactly what this is. Bronsky, the title character, is a man almost incapable of saying exactly what's on his mind, and what's on his mind is his eventual death. Not that it bothers him that much; he's just thinking a lot about it. But his incessant discussion of it certainly bothers the people around him. This leads to a few entertaining exchanges with his family, and then to more entertaining exchanges with Death. Or Deaths, as there appear to be several different kinds. I found the exact metaphysics a bit confused, but the ending was still touching and a bit funny. (7) "The Way It Works Out and All" by Peter S. Beagle: This is a reprint of a special fund-raising story about Avram Davidson, so a lot of it was lost on me given that I know almost nothing about Davidson and have yet to get to any of his novels I own. But even without that background, it's a diverting story of hidden and parallel worlds and unexpected explorations. There isn't all that much in the way of a plot, but it's a nice bit of characterization set against a fun SF twist. (6) "Less Stately Mansions" by Rob Chilson: This is a story about conservatism in life, about a farmer staying on his farm and resisting change, and about nostalgia, but I liked it much better than I normally like stories with those themes. It's set against a future world in which climate change is making life increasingly untenable. Humans are migrating into space colonies of various types, but Jacob refuses. This frustrates some parts of the family who want a piece of the substantial cash-out he's being offered for his farm, which of course makes Jacob even more stubborn. It's more of an elegy than a story, but I think it captures a particular stubborn mood, and a conscious decision to go with what one knows even if it doesn't have a long future, quite well. (7) "The Ants of Flanders" by Robert Reed: This is the novella of the issue, and, as you might expect from the author, it's thoughtful, meaty, and satisfying. At the start of the book, the planet is visited by an extraterrestrial ship (or ships it's not entirely clear at first). One of the people near one landing is Bloch, a huge teenager who has an odd lack of natural fear. He stays near the center of the story as Reed slowly develops a cosmology and a galactic political background that makes it clear humans may be incidental to everything that's happening. I liked this. It's a touch depressing in spots, and Bloch is a strange protagonist, but the cosmology is not the normal SF background and sparks some thoughts about how a status quo would be maintained by powers that don't care much about individual lives. The interlude with the leopard is nicely done, even if its significance is inobvious at first. (7) "Hair" by Joan Aiken: This is one of those weird Gothic horror stories about creepy families and half-explained supernatural events that some people love and that do nothing for me. (3) "The Witch of Corinth" by Steven Saylor: This is straight historical fantasy, featuring a Roman and his Greek tutor (heroes, apparently, of a series of historical mysteries) visiting the ruins of Corinth and encountering some bloody and dangerous local conflicts. It's slow and atmospheric, carried along by good characterization and description of ruins. It's not that much of a mystery the characters don't figure things out as much as stick around until the answer becomes obvious but it kept me entertained throughout a sizable story. Numerous elements of the story appear to be fantasy and then get other explanations, but there is a fantasy twist to the ending. (6) "Sir Morgravain Speaks of Night Dragons and Other Things" by Richard Bowes: This odd story is set among the knights of King Arthur on Avalon, where they sleep (mostly), awaiting their call to aid Britain again. Most of the story is told as one-sided dialogue from Morgravain, interspersed with some italic narration. Again, not much of a plot; the story, as such, is figuring out what Morgravain is doing and the point of his interactions with the other knights. I thought it was slight and oddly pointless, but I may have just missed the point. (4) "Someone Like You" by Michael Alexander: A time travel story, but one that's less about time travel per se than about examining and speculating about the alternate paths childhood could take and whether those changes produce different people. It takes some time to figure out what's going on, during which Alexander fills in the protagonist's past and the murder mystery that drives the tale. The time travel mechanism is blatantly hand-waved, making this more of a fantasy than an SF story, which matches the emphasis on emotion and psychology. It's not a bad story, and I think I see where the author was going with the slowly-constructed central conflict, but I still found it hard to take the conflict that seriously. One of the problems with time travel is that undermining of causality also undermines finality of decisions and consequences in ways that can rob stories of their punch. (6) "The Ramshead Algorithm" by KJ Kabza: The first-person protagonist is a well-respected and experienced fixer in a world of chaos and dimensional connection, a world with physics and inhabitants very much unlike ours. But the story doesn't spend much time there; his connection to his home earth is threatened, and he returns to try to stabilize it, which leads to the reader discovering that his family considers him a worthless appendage on a wealthy business family whose sole purpose in life is to stay out of their way. And his portal is rooted in a hedge maze that his father intends to demolish. This is one of those stories that gets more interestingly complex the deeper one gets into it. Ramshead's family is badly screwed up along multiple axes, but not without some hope of redemption. He's desperate and ineffective in his home universe, but more confident and capable when it comes to dealing with dimensional portal problems (although he still seems very young and relies on tools given to him by others). And there's always more going on than it first appears, and not as few obvious villains as it first appears. Good stuff, although I would have liked to understand more about Ramshead's world. (7) Rating: 7 out of 10

5 November 2013

Russ Allbery: Review: Asimov's, August 2011

Review: Asimov's Science Fiction, August 2011
Editor: Sheila Williams
Issue: Volume 35, No. 8
ISSN: 1065-2698
Pages: 112
The editorial this issue is about the Dell Magazines Award, so newsy without a lot of content (particularly since we don't get to read the winners). Silverberg's column is much more interesting, focusing on the complex rules of the honor code of the Albanian highlands and pointing out that there's more complexity and strangeness in things we can find on Earth than in a lot of science fiction. Even within the rules of real human cultures. James Patrick Kelly's column this issue is an interesting summary of what graduates of the Clarion writing workshop learned, in their own words. It's a nice barrage of quick writing tips, and an interesting view of what people take away from a writing workshop. The book review column this issue is Peter Heck's normal workman-like job. "The End of the Line" by Robert Silverberg: Someday, I really should read Lord Valentine's Castle. As you might guess from this comment, this is another Majipoor story. This one follows an official who is part of an advance party for the Coronal Lord, the ruler (of sorts). He's decided, as part of those duties, to take the opportunity to learn more about the aboriginal people of Majipoor: the somewhat mysterious metamorphs, or Piurivar. I'm not that familiar with the history of Majipoor, since all I've read is this story and one other, none of the novels. Apparently, knowing that the official in question is named Stiamot will place this story for more familiar readers. For the unfamiliar, such as myself, there's a lot of politics here and what's clearly background for a major event in the world, but as a standalone story it's a bit unsatisfying (and grim). It's not that clear why things had to turn out the way they did, and the characters seem largely without agency. It's well-written, but mostly a story for fans of the series, I think. (6) "Corn Teeth" by Melanie Tem: This is a tight third-person story about a human child raised by alien foster parents, and it's deeply disturbing. Not because of the fostering, which appears wonderful and loving, but because it's a train wreck sort of story: the reader can see the horrible coming and can't do anything about it (and it turns out even worse than one might expect). It's also a story built around failure of communication and failure of empathy, and has a monumentally depressing ending, the kind that leaves scars. I'm sure all of this is entirely intentional; it seems very well-written. But I really didn't want this much horrible misunderstanding and depressing hopelessness in my reading. (2) "Watch Bees" by Philip Brewer: I rather liked this story even if the protagonist is an awful person. The story is set in a future of bioengineered insects and hard economic times, and it features a man who is supposedly working his way from farm to farm to get back home. What he's actually after is more complicated and is closely related to the defense systems that keep intruders off of farms. You might guess some of the rest from the title. It's a story about understanding layered defense systems, and about economic warfare. I didn't care much for any of the characters, but the story is well-plotted and kept me interested in seeing what would happen next. (6) "For I Have Lain Me Down on the Stones of Loneliness and I'll Not Be Back Again" by Michael Swanwick: By Swanwick, so it's a little odd, but I found this story surprisingly moving and ambiguous. It's about an American of Irish descent, a trip to Ireland, and a love affair with a fierce Irish nationalist, all set against an SF background of an Earth conquered by benevolent aliens. It's angry, uncertain, fanatical, and realistic by turns, and left me with profound mixed feelings. I think it does a good job capturing in a brief story the emotional complexities of what it means to give onself to a cause. (7) "We Were Wonder Scouts" by Will Ludwigsen: This is a short and odd story about a variant of the Boy Scouts founded by a man who is a little too obsessed with the paranormal, and an outing in the woods that turns rather creepy. It's a type of story that I'm not very fond of: one that twists the delight of discovery into something dark and mundane. I suppose you could call it horror; it's more horror than fantasy, at least. Anyway, not my thing. (3) "Pairs" by Zachary Jernigan: This story, on the other hand, isn't as dark as it seems like it should be from the setup. Humans have been conquered and enslaved by more powerful alien races, and now human souls are a profitable business. The protagonist is a person who has been embodied in a spaceship, and who works with (and monitors) another largely insane embodied person as they work as couriers, carrying souls to their buyers. But neither of them are as fully under control as they might appear, which is the point of the story. There is no grand tale of redemption, and the price is high, but I found the psychological portrayal oddly satisfying and faintly hopeful, and I was intrigued by the world background. (7) "Paradise is a Walled Garden" by Lisa Goldstein: The cover story, this is by far the best story of the issue. It's steampunk, set in a world where Muslim civilization was not pressed back by Christianity and continues to thrive into the reign of Queen Elizabeth as, among other things, makers of automata. A girl has managed to get herself a job in a British factory by posing as a boy and is the first to sound the alarm when the automata that do most of the work go strangely (and violently) haywire. That leads to her being assigned to the subsequent delegation to Al-Andulus (Muslim Spain, if you're not familiar with that name from history). The protagonist is the best part of this story. She's thoughtful, resourceful, and delights in learning things, something that she's rarely had the opportunity to do. She's also utterly unintimidated. In Al-Andulus, she thrives, despite the contempt of the leader of the expedition and some dangerous intrigues around the source of the anomalous behavior. I won't spoil the ending, but it's a delight, leaving the reader with a lot of hope for her future. I also liked the portrayal of the Muslim world, which is engrossed in its own business and has its own advantages and disadvantages, but is at least open to and focused on learning and technological development. The contrast with the superstitious British delegation is both pointed and historically grounded in Muslim relations with Europe around the point of divergence of Goldstein's world. (8) Rating: 7 out of 10

3 November 2013

Russ Allbery: Review: The Walls of Westernfort

Review: The Walls of Westernfort, by Jane Fletcher
Series: Celaeno #2
Publisher: Bold Strokes
Copyright: 2004
Printing: March 2005
ISBN: 1-60282-349-9
Format: Kindle
Pages: 320
While this is the first book of the Celaeno series that I've read, it turns out to be the second book in the series. This is somewhat difficult to determine from sources on-line since the first three books in the series were published in the same year, but it's fairly obvious from reading synopses after having finished one of the books. I don't think it does any great damage to the story to read it out of order (indeed, it preserves the suspense around some plot elements), but you may want to seek out The Temple at Landfall first. Natasha Ionadis is a member of the Guard. The military services are divided into three parts in her world: the Rangers, who defend cities against bandits and others in the countryside; the Militia, who maintain law and order in the cities; and the Guard, who are responsible for guarding the temples and protecting the rituals and teachings of the state religion. She is a devout worshiper of Celaeno and delighted to be part of the Guard. But, as this book opens, she gets a very unexpected mission for a young private: a suicide mission into the heart of a heretic group in the wilds, heretics who have successfully defied the official religion and are in danger of setting up a self-sustaining community. The Walls of Westernfort has the surface trappings of a fantasy and does include some magic, but it slowly develops into more of a character story with a medieval (or, more accurately, medieval-inspired) backdrop. At the start, it appears that this mission will be an opportunity for Natasha to find the mentors and companions that she's never had before, but then her encounter with the heretics turns her world upside down. They're not the monsters that she had imagined, although their beliefs are still outrageous to her. One heretic in particular affects her deeply and introduces a strong lesbian romance element to the plot that dominates the rest of the book. I enjoyed the slow emergence of one aspect of the setting (that I'm about to spoil, so skip to the next paragraph if you don't want to hear it, although it would already be spoiled by reading the predecessor book or most of the blurbs). This is a world devoid of men; all the characters are women. This isn't part of the plot; it's just the unremarked normal. Fletcher tosses in a hand-waving explanation about halfway through, but it's only discussed in passing and never mentioned again. I really enjoy this technique, and enjoyed watching how long it took in the book before I noticed. There are so many books out there where men are the unremarked normal that it's a refreshing change to read the opposite (and notice how little it really matters), although it does require throwing in implausible magic to solve the problem of procreation. (That's apparently more the topic of The Temple at Landfall.) Plot-wise, The Walls of Westernfort is not complicated. The shape of the plot after Natasha's first encounter with the heretics is not likely to surprise any reader. But it's a simple plot well-executed, with distinct and fleshed-out characters and steady helpings of people being thoughtful and reasonable. The basic story carries plenty of natural conflict; Fletcher doesn't need to add much additional tension and wisely chooses not to, keeping the focus tightly on Natasha and her conflict between duty and conscience. I liked both of the people involved in the romance and could believe they'd fall in love, and I also liked that the romance was significant but not decisive. It's part of what helps Natasha expand her view of the world, but one could see how she would have made the right decision in the end even if the romance had not been present. The story does force some significant secret-keeping, but it doesn't draw cheap tension from mundane failure to communicate. All of the communication gaps here are believable and honest, driven by the characters' backgrounds, and absolutely necessary. I'm hypersensitive to romance plots based on communication failure, and I wasn't even grumbling. The series this book reminds me the most of is another lesbian romance in a fantasy-style setting: Catherine M. Wilson's When Women Were Warriors. The Walls of Westernfort is not as deep or philosophical, it's a bit more black and white in its morality, and I don't think it's quite as good of a book, but for me it had a similar tone of good people trying to do the right thing and finding one's way through ethical mazes. It's a bit slow to get started, but by the time Natasha's critical decision point arrived, I was fully invested in the story and had a hard time putting it down. Recommended if you're in the mood for a relatively light faux-medieval fantasy romance. I'm not rushing out to grab the other books in the series, but they went on my want list and I'll probably get them at some point. Followed by Rangers at Roadsend in the publication order sense, but apparently it's a prequel. Rating: 7 out of 10

2 November 2013

Marco d'Itri: New PGP key

Since my current PGP key is a 1024 bits DSA key generated in 1998, I decided that it is time to replace it with a stronger one: there are legitimate concerns that breaking 1024 bits DSA is well within the reach of major governments. I have been holding out for the last year waiting for GnuPG 2.1, which will support elliptic curves cryptography, but I recently concluded that adopting ECC now would not be a good idea: Red Hat still does not fully support it due to unspecified patent concerns and there is no consensus in the cryptanalists community about the continued strength of (some?) ECC algorithms. So I created three fancy keys: a 4096 bits main key for offline storage, which hopefully will be strong enough for a long time, and two 3072 bits subkeys for everyday use. I have published a formal key transition statement and I will appreciate if people who have signed my old key will also sign the new one. What follows are the instructions that I used to generate these PGP keys. They follow the current best practices and only reference modern software. While the GnuPG defaults are usually appropriate, I think that it is a good idea to use a stronger hash for the key signatures of very long-lived keys. I could not find a simple way to "upgrade" the algorithm of key self signatures.
echo 'cert-digest-algo SHA256' >> ~/.gnupg/gpg.conf
First, generate a RSA/4096 sign only key, which will be your master key and may be stored offline. Then add to it two RSA/3072 subkeys (one sign only and one encrypt only):
# generate a RSA/4096 sign only key
gpg2 --gen-key
# add two RSA/3072 subkeys (sign only and encrypt only)
gpg2 --edit-key 8DC968B0
Since GnuPG lacks a command to remove the master secret key while keeping its secret subkeys, you need to delete the complete secret keys and then re-import only the subkeys:
gpg2 --export-secret-keys 8DC968B0 > backup.secret
gpg2 --export-secret-subkeys 8DC968B0 > backup.subkeys
gpg2 --delete-secret-key 8DC968B0
gpg2 --import backup.subkeys
Then you can import again the complete keys in a different secret keyring, which can be stored offline:
mkdir ~/.gnupg/master/
gpg2 --no-default-keyring \
  --keyring ~/.gnupg/pubring.gpg \
  --secret-keyring ~/.gnupg/master/secring.gpg \
  --import backup.secret
Now you can move ~/.gnupg/master/ to a USB stick. You are supposed to protect the master secret key with a strong passphrase, so there is no point in using block level encryption on the removable media. Since you are only using the master key to sign other keys, it only needs to be configured as the second keyring in ~/.caffrc:
$CONFIG 'secret-keyring'  = $ENV HOME  . '/.gnupg/master/secring.gpg';
It is also a good idea to have an hard copy backup of your keys, since the lifetime of USB sticks should not be trusted too much:
paperkey -v --output printable.txt --secret-key backup.secret
a2ps -2 --no-header -o printable.ps printable.txt
Some references that I used:

Dominique Dumont: About LWP::UserAgent, https and proxy setup

Hello The last few weeks, I ve been banging my head to use uscan, Pithub and JIRA::Client::Automated behind a corporate firewall. They are all written in Perl and use LWP::UserAgent to fetch information from Internet. At $work, using a proxy is mandatory to connect to Internet. But LWP::UserAgent https connection does not work through a proxy. This bug was reported 10 years ago but is still not fixed. Here s my plan to fix (or work-around) this issue, at least for Debian. Before going further, let s step back to explain briefly what is https and how proxies work. https is not a protocol by itself. https is plain http over a socket encrypted with SSL (aka TLS). To create a https connection, the agent must first setup the SSL socket with the server. Then the agent uses http protocol to communicate with the server. All the traffic is encrypted on agent side and decrypted on server side by the SSL layer. When creating a connection through a proxy, things get a little more complicated. Plain http requests (like GET, POST and so on ) are sent to the proxy as if the proxy was the http server. Then, the proxy forwards the request to the actual server. From what I ve read, most proxy servers refuse to plainly forward encrypted data to a web server. First a negotiation to create a tunnel towards the web server must be done by sending a http CONNECT request to the proxy server. The encrypted socket is then set up between the user and the web server through the proxy. Once this encrypted tunnel is set up, the usual http communication can be done. Let s go back to LWP::UserAgent. To create a connection over SSL, LWP::UserAgent will use a SSL library to setup the socket. This SSL library can be IO::Socket::SSL or Net::SSL. Direct https connection with LWP::UserAgent works fine with either library. When trying to setup a https connection through a proxy, LWP::UserAgent will try to use the proxy like a regular http proxy without going through the CONNECT phase. This does not work. Net::SSL is able to setup correctly an SSL socket through a proxy provided the proxy server is specified in an https_proxy environment variable. But LWP::UserAgent will also try to (badly) setup the proxy when this variable is set. One can find various work-around to play with %ENV to bypass LWP::UserAgent. So I ve proposed a patch to make LWP::UserAgent skip the https_proxy variable. Then Net::SSL can handle the connection to the proxy. I hope this blog will push upstream to either apply this path or tell me why it s a bad idea (which is possible, I ve some trouble to believe that a 10 years old bug can be fixed by such a simple patch) [ Edited ] I was right to worry. Turns out that relying on Net::SSL is a bad idea: SSL certificates are not checked correctly with this library. For more details, see https://github.com/libwww-perl/libwww-perl/pull/51 and https://bugzilla.redhat.com/show_bug.cgi?id=705044 . I m going to push JGMYERS set of patches on libwww-perl repositories Anyway, this patch works provided:
  • Net::SSL is loaded by the application
  • the proxy is setup with https_proxy environment variable
  • LWP user agent is created this way:
    my $ua = LWP::UserAgent->new(
        env_proxy => 1,
        ssl_opts =>   verify_hostname => 0 
    );
    The verify_hostname parameter is used by IO::Socket::SSL.
    LWP::UserAgent bails out if Net::SSL is used while this parameter is not set to 0.
Even though this patch is probably not ideal, it does fix issues with uscan so I ll push it to Debian. On the other hand, the patch does not work if https_proxy is specified by calling:
$ua->proxy( https => ...);
JGMYERS has proposed a set of patches to fix this problem. But these patches are more invasive and I do not dare to push them to Debian. Last but not least, this blog avoids the issue of host name verification or peer authentication. Net::SSL and Crypt::SSLeay are able to perform such a verification, but I did not verify this point. Next step is to provide simple patches to let uscan, Pithub and JIRA::Client::Automated correctly connect through proxies without jumping through hoops. All the best
Tagged: debian, https_proxy, Perl

Robert Collins: Learning is hard

I feel like I m taking a big personal risk writing this, even though I know the internet is large and probably no-one will read this :-) . So, dear reader, please be gentle. As we grow as people, as developers, as professionals some lessons are are hard to learn (e.g. you have to keep trying and trying to learn the task), and some are hard to experience (they might still be hard to learn, but just being there is hard itself ) I want to talk about a particular lesson I started learning in late 2008/early 2009 while I was at Canonical sadly one of those that was hard to experience. At the time I was one of the core developers on Bazaar, and I was feeling pretty happy about our progress, how bzr was developing, features, community etc. There was a bunch of pressure on to succeed in the marketplace, but that was ok, challenges bring out the stubborn in me :) . There was one glitch though we d been having a bunch of contentious code reviews, and my manager (Martin Pool) was chatting to me about them. I was as far as I could tell doing precisely the right thing from a peer review perspective: I was safeguarding the project, preventing changes that didn t fit properly, or that reduced key aspects- performance, usability from landing until they were fixed. However, the folk on the other side of the review were feeling frustrated, that nothing they could do would fix it, and generally very unhappy. Reviews and design discussions would grind to a halt, and they felt I was the cause. [They were right]. And here was the thing I simply couldn t understand the issue. I was doing my job; I wasn t angry at the people submitting code; I wasn t hostile; I wasn t attacking them (but I was being shall we say frank about the work being submitted). I remember saying to Martin one day look, I just don t get it can you show me what I said wrong? and he couldn t. Canonical has a 360 review system every 6 months / year (it changed over time) you review your peers, subordinate(s) and manager(s), and they review you. Imagine my surprise I was used to getting very positive reports with some constructive suggestions when I scored low on a bunch of the inter-personal metrics in the review. Martin explained that it was the reviews thing folk were genuinely unhappy, even as they commended me on my technical merits. Further to that, he said that I really needed to stop worrying about technical improvement and focus on this inter-personal stuff. Two really important things happened around this time. Firstly, Steve Alexander, who was one of my managers-once-removed at the time, reached out to me and suggested I read a book Getting out of the box and that we might have a chat about the issue after I had read it. I did so, and we chatted. That book gave me a language and viewpoint for thinking about the problem. It didn t solve it, but it meant that I got it , which I hadn t before. So then the second thing happened we had a company all hands and I got to chat with Claire Davis (head of HR at Canonical at the time) about what was going on. To this day the sheer embarrassment I felt when she told me that the broad perception of me amongst other teams managers was and I paraphrase a longer, more nuance conversation here technically fantastic but very scary to have on the team will disrupt and cause trouble . So, at this point about 6 months had passed, I knew what I wanted I wanted folk to want to work with me, to find my presence beneficial and positive on both technical and team aspects. I already knew then that what I seek is technical challenges: I crave novelty, new challenges, new problems. Once things become easy, it call all too easily slip into tedium. So at that time my reasoning was somewhat selfish: how was I to get challenges if no-one wanted to work with me except in extremis? I spent the next year working on myself as much as specific projects: learning more and more about how to play well with others. In June 2010 I got a performance review I could be proud of again I was in no way perfect, but I d made massive strides. This journey had also made huge improvements to my personal life a lot of stress between Lynne and I had gone away. Shortly after that I was invited to apply for a new role within Canonical as Technical Architect for Launchpad and Francis Lacoste told me that it was only due to my improved ability to play well with others that I was even considered. I literally could not have done the job 18 months before. I got the job, and I think I did pretty well in fact I was awarded an internal Spotlight on Success award for what we (it was a whole Launchpad team team effort) achieved while I was in that role. So, what did I change/learn? There s just a couple of key changes I needed to make in myself, but a) they aren t sticky: if I get overly tired, ye old terrible Robert can leak out, and b) there s actually a /lot/ of learnable skills in this area, much of which is derived lots of practice and critical self review is a good thing. The main thing I learnt was that I was Selfish. Yes capital S. For instance, in a discussion about adding working tree filter to bzr, I would focus on the impact/risk on me-and-things-I-directly-care-about: would it make my life harder, would it make bzr slower, was there anything that could go wrong. And I would spend only a little time thinking about what the proposer needed: they needed support and assistance making their idea reach the standards the bzr community had agreed on. The net effect of my behaviours was that I was a class A asshole when it came to getting proposals into a code base. The key things I had to change were:
  1. I need to think about the needs of the person I m speaking to *and not my own*. [Thats not to say you should ignore your needs, but you shouldn't dwell on them: if they are critical, your brain will prompt you].
  2. There s always a reason people do things: if it doesn t make sense, ask them! [The crucial conversations books have some useful modelling here on how and why people do things, and on how-and-why conversations and confrontations go bad and how to fix them.]
Ok so this is all interesting and so forth, but why the blog post? Firstly, I want to thank four folk who were particularly instrumental in helping me learn this lesson: Martin, Steve, Claire and of course my wife Lynne I owe you all an unmeasurable debt for your support and assistance. Secondly, I realised today that while I ve apologised one on one to particular folk who I knew I d made life hard for, I d never really made a widespread apology. So here it is: I spent many years as an ass, and while I didn t mean to be one, intent doesn t actually count here actions do. I m sorry for making your life hell in the past, and I hope I m doing better now. Lastly, if I m an ass to you now, I m sorry, I m probably regressing to old habits because I m too tired something I try to avoid, but it s not always possible. Please tell me, and I will go get some sleep then come and apologise to you, and try to do better in future. Less-assily-yrs,
Rob

31 October 2013

Daniel Pocock: init systems: what do users want?

The debate about Debian init system replacement continues and many developers have given their opinions. However, what do the users want? What type of features will make it easier for users to adopt and promote Debian and free software in general? Could this even help bring improvements to other distributions too? Some of the possibilities I've previously raised for discussion (and they are not specific to Debian by any means):
  • Better integration with tools for service monitoring (for example, so Nagios can automatically detect all the services on a system and the dependency relationship between them, avoiding any manual configuration of the monitoring system)
  • Better integration with tools for service management (for example, so Nagios can more easily restart a service without having to provide some action script developed by hand)
  • Better support for dependencies in clustering and distributed environments (e.g. Apache on host A should only start after MySQL is running on host B or host C)
  • Support for multiple instances of a service, each with separate configuration files on the same host
Some of the systems up for discussion go some of the way to addressing these types of requirements - none of them go all the way. What other requirements do people have? Should Debian wait a little longer and let the different solutions compete to offer more of these things before any decision is made? My own feeling is that Debian should not commit to something that will involve any loss in current features or support for ports and we should use this opportunity to find the best possible set of features for the future.

30 October 2013

Joey Hess: license monads

This could have been a blog post about toothbrushes for monkeys, which I seem to have dreamed about this morning. But then I was vaguely listening to the FAIFCast in the car, and I came up with something even more esoteric to write about! License monads would allow separating parts of code that are under different licenses, in a rigorous fashion. For example, we might have two functions in different license monads:
foo :: String -> GPL String
bar :: Char -> BSD Char
Perhaps foo needs to use bar to calculate its value. It can only do so if there's a way to lift code in the BSD monad into the GPL monad. Which we can legally write, since the BSD license is upwards-compatable with the GPL:
liftGPL :: BSD a -> GPL a
On the other hand, there should be no way provided to lift the GPL monad into the BSD monad. So bar cannot be written using code from foo, which would violate foo's GPL license. Perhaps the reason I am thinking about this is that the other day I found myself refactoring some code out of the git-annex webapp (which is AGPL licensed) and into the git-annex assistant (which is GPL licensed). Which meant I had to relicense that code. Luckily that was easy to do legally speaking, since I am the only author of the git-annex webapp so far, and own the whole license of it. (Actually, it's only around 3 thousand lines of code, and another thousand of html.) It also turned out to be easy to do the refactoring, technically speaking because looking at the code, I realized I had accidentially written it in the wrong monad; all the functions were in the webapp's Handler monad, but all of them used liftAnnex to actually do their work in the Annex monad. If that had not been the case, I would not have been able to refactor the code, at least not without entirely rewriting it. It's as if I had accidentially written:
foo :: String -> GPL String
foo = mapM (liftGPL . bar)
Which can be generalized to:
foo :: String -> BSD String
foo = mapM bar
I don't think that license monads can be realistically used in the current world, because lawyers and math often don't mix well. Lawyers have, after all, in the past written laws re-defining pi to be 3. Still, license monads are an interesting way to think about things for myself. They capture how my code and licenses are structured and allow me to reason about it on a more granular level than the licences of individual files. (I have a vague feeling someone else may have written about this idea before? Perhaps I should have been blogging about monkey toothbrushes after all...)

Next.

Previous.