• Facebook Like
• Google Plus One
• Linkedin Share Button
• Tweet Widget

In a way similar to my previous post about persisting graphs in PostGreSQL in PHP, here s similar examples for Python and MySQL. Note that you may need to adjust the MySQL table creation templates if you re running librdf version 1.0.16 (see this bug for details). The first example will load data from a file, and store the parsed model to the RDBMS, while the second example, will reload the model and issue queries on it. These examples don t do much, but hopefully, they ll help you get started if you re interested in using redland/librdf. One potential benefit can be interoperability with other languages like PHP, for instance if you re running harvesters in Python that will save facts in a triple store, which is then accessed by a PHP application that will publish a Web interface.

I'll soon be disposing of the Nissan Leaf that we leased a few months ago, so I thought it a useful exercise to write about my experiences with it. I am not a car man. I am a gadget man. For me, driving is a means to an end, and I'm much more interested in what I call the "cabin" experience, than the "driving" experience, so this is going to slanted much more that way. That said, I found the Nissan Leaf a fun car to drive, from my limited experiences of having fun driving cars. I liked how responsive it was when you put your foot down. It has two modes of operation, "D" and "Eco". I've actually taken to driving it in "Eco" mode most of the time, as it squeezes more range out of the batteries, but occasionally I'll pop it back into "D" to have a bit of fun. The main difference between the two modes, from a driving perspective, is it seems to limit the responsiveness of the accelerator. In "Eco" mode it feels more like you're driving in molasses, whereas in "D" mode, when you put your foot down, it responses instantly. "D" is great for dragging people off at the lights. It's a very zippy little car in "D" mode. It feels lighter. I've noticed that it has a bit of a penchant for over steering. Or maybe that's just my driving. If I have floored it a bit to take a right turn into oncoming traffic, I've noticed slight over steering. That's about all the driving type "real car stuff" I'll talk about. Now to the driver's "cabin experience". It's absolutely fabulous. I love sitting in the driver's seat of this car. Firstly, the seat itself is heated (in fact all of them are). As is the steering wheel. Nissan has gone to great lengths to allow you to avoid needing to run the car's heating system to heat the car, as doing so immediately drops at least 10 miles off the range. Unfortunately I found the windscreen had a tendency to fog up in the winter rainy periods, so I'd have to intermittently fire up the air conditioning to defog the windscreen. Of course, in the summer months, you're going to want to run the AC to cool down, so the range hit in that situation is unavoidable. I've only had this car from late Autumn until late Winter so far, so that hasn't been an issue I've had to contend with. The dashboard is all digital, and looks relatively high tech, which appeals to my inner geek. It's a dashboard. It tells you stuff. The stuff you'd expect it to tell you. Enough said. The audio system is nice. It supports Bluetooth audio, so one can be streaming Pandora from one's phone, through the sound system, for example. Or listening to audio from the phone. There's also a USB port, and it will play various audio files from USB storage. I found the way it chose to sort the files on a USB stick to be slightly surprising though. I haven't invested the time to figure out how I should be naming files so that they play in the order I expect them to play. The ability to play from USB storage compensates nicely for the fact that it only has a single CD player. (We have a 6 disc stacker in our 2006 Prius). The car also came with a 3 month free trial of Sirius XM satellite radio. This was fun. The only dance music FM station in the Bay Area has a weak signal in Mountain View, and I hate dance music with static, whereas there was an excellent electronic music station that I could listen to in glorious high definition. As long as I wasn't driving under a bridge. There's no way I'd pay money for satellite radio, but it was a fun gimmick to try out. The navigation system is really, really good. I haven't bothered using Google Maps on my phone at all. It gives such good spoken directions, that you don't even need to have the map on the screen. It names all the streets. I couldn't figure out a way to get distances in metric. The telematics service, Carwings, is probably my favourite feature. This is what really makes it feel like a car of the future. Through the companion Android application, I can view the charging status (if it's plugged in) or just check the available range (if it's not plugged in). From a web browser, I can plan a route, and push the route to the vehicle's navigation system. If the car is plugged in, I can also remotely turn on the vehicle's climate control system, pre-warming or cooling the car. It's a little thing, but the door unlocking annoyed me a little bit. I'm used to the Prius, where if you unlock the boot (that's trunk, Americans), or the front passenger door, all the doors unlocked. This was a convenient way of unlocking the car for multiple people as you approached it. With the Leaf, unlocking the boot only unlocks the boot. Unlocking the front passenger door only unlocks that door. It requires a second unlock action to unlock all the doors. I've found this to be slightly cumbersome when trying to unlock the car for everyone all at once, quickly (like when it's raining). Another minor annoyance is the headlights. I've gotten into the habit of driving with the headlights on all the time, because I believe it's safer. In the Prius, one could just leave the headlights switched to the "on" position, and they'd turn off whenever the driver's door was opened after the car was switched off. If you try that in the Leaf, the car beeps at you to warn you you've left the headlights on. It has an "auto" mode, where the car will choose when to turn the headlights on, based on ambient light conditions. In that case, when you turn the car off, it'll leave the headlights on for a configurable period of time and then turn them off. This is actually slightly unsettling, because it makes you think you've left your headlights on. The default timeout was quite long as well, something like 60 seconds. The way multiple Bluetooth phones are handled is just as annoying in the Leaf as it is in the Prius, which disappoints me, given 6 years have passed. The way I'd like to see multiple phones handled is the car chooses to pair with whichever phone is in range, or if multiple phones are in range, it asks or uses the last one it paired with. In reality, it tries to pair with whatever it paired with last time, and one has to press far too many buttons to switch it to one of the other phones it knows about. Range anxiety is definitely something of a concern. It can be managed by using the GPS navigation for long or otherwise anxiety-inducing trips, and then one can compare the "miles remaining" on the GPS with the "miles remaining" on the battery range, and reassure oneself that they will indeed make it to where they're trying to go. The worst case I had was getting to within 5 miles of empty. The car started complaining at me. The charging infrastructure in the Bay Area is pretty good. There are plenty of charging stations in San Francisco and San Jose. I'm spoiled in that I have free charging available at work (including a building at the end of my street, so I never bothered with getting a home charger installed). I've almost never had to pay for charging, so it's been great while gas prices have been on the rise. The car's navigation system knows about some charging stations, so you can plan a route with the charging stations it knows about in mind. The only problem is it doesn't know if the charging stations are in use. If you use the ChargePoint Android app, you can see if the charging stations are in use, but then you have to do this cumbersome dance to find an available charging station and plug the address into the vehicle's navigation system. Of course, what can then happen is in the time you're driving to the charging station, someone else starts using it. I actually got bitten by this yesterday. Would I buy a Leaf again? Not as my sole car. It makes a perfect second/commuter car, but as a primary vehicle, it's too limited by its range. They're also ridiculously expensive in Australia, and Brisbane has absolutely no charging infrastructure.

Boat Again, technology.. I am on a boat in international waters somewhere in between Stockholm and Tallinn and there's a (sluggish) satellite uplink so I can push out this post. We really do live in the future.. Beta git annex assistant runs on Android now; get it, break it, report it. Bugs The UDD bugs interface currently knows about the following release critical bugs:

• In Total: 795
  • Affecting wheezy: 206 That's the number we need to get down to zero before the release. They can be split in two big categories:
    • Affecting wheezy and unstable: 144 Those need someone to find a fix, or to finish the work to upload a fix to unstable:
      • 33 bugs are tagged 'patch'. Please help by reviewing the patches, and (if you are a DD) by uploading them.
      • 9 bugs are marked as done, but still affect unstable. This can happen due to missing builds on some architectures, for example. Help investigate!
      • 103 bugs are neither tagged patch, nor marked done. Help make a first step towards resolution!
    • Affecting wheezy only: 62 Those are already fixed in unstable, but the fix still needs to migrate to wheezy. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.
      • 24 bugs are in packages that are unblocked by the release team.
      • 38 bugs are in packages that are not unblocked.

How do we compare to the Squeeze release cycle?

Week	Squeeze	Wheezy	Diff
43	284 (213+71)	468 (332+136)	+184 (+119/+65)
44	261 (201+60)	408 (265+143)	+147 (+64/+83)
45	261 (205+56)	425 (291+134)	+164 (+86/+78)
46	271 (200+71)	401 (258+143)	+130 (+58/+72)
47	283 (209+74)	366 (221+145)	+83 (+12/+71)
48	256 (177+79)	378 (230+148)	+122 (+53/+69)
49	256 (180+76)	360 (216+155)	+104 (+36/+79)
50	204 (148+56)	339 (195+144)	+135 (+47/+90)
51	178 (124+54)	323 (190+133)	+145 (+66/+79)
52	115 (78+37)	289 (190+99)	+174 (+112/+62)
1	93 (60+33)	287 (171+116)	+194 (+111/+83)
2	82 (46+36)	271 (162+109)	+189 (+116/+73)
3	25 (15+10)	249 (165+84)	+224 (+150/+74)
4	14 (8+6)	244 (176+68)	+230 (+168/+62)
5	2 (0+2)	224 (132+92)	+222 (+132/+90)
7	release+1	194 (128+66)	+194 (+128/+66)
8	release+2	206 (144+62)	+206 (+144/+62)

Graphical overview of bug stats thanks to azhag:

I finished delivering all my Kickstarter rewards at the end of the year. This is an overview of how that went, both financially and in general.

my Kickstarter pie (not including taxes)

While the Kickstarter was under way, several friends warned me that I might end up spending a lot of the money for rewards, or even shipping, and come out a loser. It's happened to some people, but I avoided it. Most of the pie went to its intended purpose.

USB key arrives in Amsterdam

I kept shipping cost low by shipping everything by US postal service, including Air Mail for international shipping. This was particularly important for the stickers (which cost $1.05 to ship internationally). But I also shipped USB keys in regular mail envelopes, protected by bubble wrap, which worked very well and avoided the bother of shipping packages. The USPS will be annoyed at you for a rigid letter and add a non-machinable surcharge, but it's still a nice savings.

expenses

I spent more on rewards than on transaction fees, but the fees are still pretty large. Being dinged a second time by Amazon is the worst part. I have not been able to work out exactly what formula Kickstarter uses to determine its fee per pledge. It does not seem to be a simple percentage of the pledge. For example, they seem to have charged $0.25 per $10 pledge, but $25 for a $500 pledge. I wanted to solve this, but I'd have to match up all the pledges and fees manually to do it.

gross income by reward type

This chart is slightly innacurrate, because it puts any money pledged, beyond the amount needed to get a reward, into the "intangibles" category, despite the reward being probably responsible for that money being pledged. (The intangibles also includes people who did not ask for a reward, and several categories of rewards not involving shipping matter around.) But, the surprise for me is how large a peice the T-shirts are responsible for. It was my least favorite reward, and a low volume one, but I made out pretty well on it. However, I'd still try to avoid putting T-shirts on Kickstarter again. It's hard to do a good design (I didn't, really); they're expensive, and were by far the most annoying thing to ship. Also, I was not happy with the countries Cafe Press sourced their shirts from; I've been to Honduras and talked with people who have relatives in las machinas.

gross and net income by reward type (excluding shipping)

In contrast, the stickers had an amazing margin; they're so inexpensive to print that I printed up two kinds and included multiple with every other reward I mailed. I still have hundreds left over, too.. All the online print shops I tried have very annoying interfaces to upload artwork though. I had to do quite a bit of math to render TIFF files with appropriate DPI and margins. The USB keys were my favorite reward. I got them from USB Memory Direct, who gave me quite a nice deal. I was very happy that I was able to send them a SVG file of my artwork, so I didn't need to worry about lacking resolution for the laser engraving. And it came out looking great. The best part was when their sales guy Mike actually did a minor alteration of the artwork, to better fit on the key, when I, being overloaded with Kickstarter stuff asked him too. A bit above and beyond. There was an issue with their Chinese manufacturer's quality control of the 16 gb drives, but they were willing to send me replacement for all the ones I found problems with.

---

All told I spent probably 3 full days stuffing and shipping envelopes, and probably spent a week working on Kickstarter reward fullfillment. As work-related overhead goes, that's not bad. Maybe someone considering a Kickstarter will find this information useful somehow. Oh well, back to work. :)

A few years ago I was doing research comparing how large software distributions handled shared object libraries, and studied Debian, FreeBSD, and Ubuntu. Extracting data about Debian packages was easy thanks to Peter Palfrader s snapshot.debian.org service, which provides a machine-usable interface to Debian s package history. FreeBSD s data is equally accessible, albeit in a less pleasant format: their ports tree was stored in CVS until July 2012. One could easily rsync a copy of the ports tree s CVS repository to a local machine to analyze the data. This left you with a local tree full of ,v files, each corresponding to the history of a given file with at that location. I needed to extract all kinds of data from a tree full of these files, such as what revisions contained lines matching a regex, when these revisions were checked in, any tags associated with it, etc. To make things easier, it also helped to know the line numbers of the matching lines. Hence the birth of rcsgrep. rcsgrep is a Python script that makes use of Paul McGuire s fabulous pyparsing library. It allows you to search a RCS file (the ,v file format used by RCS and CVS to store revision history) using a Python regex, and the output format is customizable to allow printing only certain kinds of information, such as the revision number, the line number, the matching line, the line s author, the date it appeared, any tags associated with the line, and (useful when running over a large number of files) the file name. To make machine parsing (using AWK of course) easier, you can also specify the column separator. For example, I entered the lines The quick brown , fox jumped over , the lazy dog. Woof! into the file abc, checking in the changes after each line. The invocation ./rcsgrep -s ' ' -f rlLda '.*' abc,v, with spaces for column separation, and format options r is for revision, l for line number, L for line contents, d for date, and a for author, outputs:

1.3    1    The quick brown    2013.02.20.14.24.09    ryan
1.3    2    jumped over the    2013.02.20.14.24.09    ryan
1.3    3    lazy dog. Woof!    2013.02.20.14.24.09    ryan
1.2    1    The quick brown    2013.02.20.14.23.48    ryan
1.2    2    jumped over the    2013.02.20.14.23.48    ryan
1.1    1    The quick brown    2013.02.20.14.23.25    ryan

I m particularly proud about my grep() function in rcsfile.py, which goes through each revision, tracking additions and deletions, but only keeping the lines matching the regex in memory. In any case, rcsgrep is licensed under the ISC license and can be found on github. Addendum: I learned after the fact that O Reilly s UNIX Power Tools offers something similar by the same name, except that it is runs several processes, such as co, grep and sed, as opposed to a single Python script.

I just released umockdev 0.2. The big new feature of this release is support for evdev ioctls. I. e. you can now record what e. g. X.org is doing to touchpads, touch screens, etc.:

  $ umockdev-record /dev/input/event15 > /tmp/touchpad.umockdev
  # umockdev-record -i /tmp/touchpad.ioctl /dev/input/event15 -- Xorg -logfile /dev/null

and load that back into a testbed with X.org using the dummy driver:

  cat <<EOF > xorg-dummy.conf
  Section "Device"
        Identifier "test"
        Driver "dummy"
  EndSection
  EOF
  $ umockdev-run -l /tmp/touchpad.umockdev -i /dev/input/event15=/tmp/touchpad.ioctl -- \
       Xorg -config xorg-dummy.conf -logfile /tmp/X.log :5

Then e. g. DISPLAY=:5 xinput will recognize the simulated device. Note that Xvfb won t work as that does not use udev for device discovery, but only adds the XTest virtual devices and nothing else, so you need to use the real X.org with the dummy driver to run this as a normal user. This enables easier debugging of new kinds of input devices, as well as writing tests for handling multiple touchscreens/monitors, integration tests of Wacom devices, and so on. This release now also works with older automakes and Vala 0.16, so that you can use this from Ubuntu 12.04 LTS. The daily PPA now also has packages for that. Attention: This version does not work any more with recorded ioctl files from version 0.1. More detailled list of changes:

• umockdev-run: Fix running of child program to keep stdin.
• preload: Fix resolution of /dev and /sys
• ioctl_tree: Fix endless loop when the first encountered ioctl was unknown
• preload: Support opening a /dev node multiple times for ioctl emulation (issue #3)
• Fix parallel build (issue #2)
• Support xz compressed ioctl files in umockdev_testbed_load_ioctl().
• Add example umockdev and ioctl files for a gphoto camera and an MTP capable mobile phone.
• Fix building with automake 1.11.3 and Vala 0.16.
• Generalize ioctl recording and emulation for ioctls with simple structs, i. e. no pointer fields. This makes it much easier to add more ioctls in the future.
• Store return values of ioctls in records, as they are not always 0 (like EVIOCGBIT)
• Add support for ioctl ranges (like EVIOCGABS) and ioctls with variable length (like EVIOCGBIT).
• Add all reading evdev ioctls, for recording and mocking input devices like touch pads, touch screens, or keyboards. (issue #1)

A few weeks ago, a new version of lldpd, a 802.1AB (aka LLDP) implementation for various Unices, has been released. LLDP is an industry standard protocol designed to supplant proprietary Link-Layer protocols such as EDP or CDP. The goal of LLDP is to provide an inter-vendor compatible mechanism to deliver Link-Layer notifications to adjacent network devices. In short, LLDP allows you to know exactly on which port is a server (and reciprocally). To illustrate its use, I have made a xkcd-like strip: If you would like more information about lldpd, please have a look at its new dedicated website. This blog post is an insight of various technical changes that have affected lldpd since its latest major release one year ago. Lots of C stuff ahead!

Version & changelog UPDATED: Guillem Jover told me how he met the same goals for libbsd :

1. Save the version from git into .dist-version and use this file if it exists. This allows one to rebuild ./configure from the published tarball without losing the version. This also handles Thorsten Glaser s critic.
2. Include CHANGELOG in DISTCLEANFILES variable.

Since this is a better solution, I have adopted the appropriate line of codes from libbsd. The two following sections are partly technically outdated.

Automated version In configure.ac, I was previously using a static version number that I had to increase when releasing:

AC_INIT([lldpd], [0.5.7], [bernat@luffy.cx])

Since the information is present in the git tree, this seems a bit redundant (and easy to forget). Taking the version from the git tree is easy:

AC_INIT([lldpd],
        [m4_esyscmd_s([git describe --tags --always --match [0-9]* 2> /dev/null   date +%F])],
        [bernat@luffy.cx])

If the head of the git tree is tagged, you get the exact tag (0.7.1 for example). If it is not, you get the nearest one, the number of commits since it and part of the current hash (0.7.1-29-g2909519 for example). The drawback of this approach is that if you rebuild configure from the released tarball, you don t have the git tree and the version will be a date. Just don t do that.

Automated changelog Generating the changelog from git is a common practice. I had some difficulties to make it right. Here is my attempt (I am using automake):

dist_doc_DATA = README.md NEWS ChangeLog
.PHONY: $(distdir)/ChangeLog
dist-hook: $(distdir)/ChangeLog
$(distdir)/ChangeLog:
        $(AM_V_GEN)if test -d $(top_srcdir)/.git; then \
          prev=$$(git describe --tags --always --match [0-9]* 2> /dev/null) ; \
          for tag in $$(git tag   grep -E '^[0-9]+(\.[0-9]+) 1, $$'   sort -rn); do \
            if [ x"$$prev" = x ]; then prev=$$tag ; fi ; \
            if [ x"$$prev" = x"$$tag" ]; then continue; fi ; \
            echo "$$prev [$$(git log $$prev -1 --pretty=format:'%ai')]:" ; \
            echo "" ; \
            git log --pretty=' - [%h] %s (%an)' $$tag..$$prev ; \
            echo "" ; \
            prev=$$tag ; \
          done > $@ ; \
        else \
          touch $@ ; \
        fi
ChangeLog:
        touch $@

Changelog entries are grouped by version. Since it is a bit verbose, I still maintain a NEWS file with important changes.

Core

C99 I have recently read 21st Century C which has some good bits and also handles the ecosystem around C. I have definitively adopted designated initializers in my coding style. Being a GCC extension since a long time, this is not a major compatibility problem. Without designated initializers:

struct netlink_req req;
struct iovec iov;
struct sockaddr_nl peer;
struct msghdr rtnl_msg;
memset(&req, 0, sizeof(req));
memset(&iov, 0, sizeof(iov));
memset(&peer, 0, sizeof(peer));
memset(&rtnl_msg, 0, sizeof(rtnl_msg));
req.hdr.nlmsg_len = NLMSG_LENGTH(sizeof(struct rtgenmsg));
req.hdr.nlmsg_type = RTM_GETLINK;
req.hdr.nlmsg_flags = NLM_F_REQUEST   NLM_F_DUMP;
req.hdr.nlmsg_seq = 1;
req.hdr.nlmsg_pid = getpid();
req.gen.rtgen_family = AF_PACKET;
iov.iov_base = &req;
iov.iov_len = req.hdr.nlmsg_len;
peer.nl_family = AF_NETLINK;
rtnl_msg.msg_iov = &iov;
rtnl_msg.msg_iovlen = 1;
rtnl_msg.msg_name = &peer;
rtnl_msg.msg_namelen = sizeof(struct sockaddr_nl);

With designated initializers:

struct netlink_req req =  
    .hdr =  
        .nlmsg_len = NLMSG_LENGTH(sizeof(struct rtgenmsg)),
        .nlmsg_type = RTM_GETLINK,
        .nlmsg_flags = NLM_F_REQUEST   NLM_F_DUMP,
        .nlmsg_seq = 1,
        .nlmsg_pid = getpid()  ,
    .gen =   .rtgen_family = AF_PACKET  
 ;
struct iovec iov =  
    .iov_base = &req,
    .iov_len = req.hdr.nlmsg_len
 ;
struct sockaddr_nl peer =   .nl_family = AF_NETLINK  ;
struct msghdr rtnl_msg =  
    .msg_iov = &iov,
    .msg_iovlen = 1,
    .msg_name = &peer,
    .msg_namelen = sizeof(struct sockaddr_nl)
 ;

Logging Logging in lldpd was not extensive. Usually, when receiving a bug report, I asked the reporter to add some additional printf() calls to determine where the problem was. This was clearly suboptimal. Therefore, I have added many log_debug() calls with the ability to filter out some of them. For example, to debug interface discovery, one can run lldpd with lldpd -ddd -D interface. Moreover, I have added colors when logging to a terminal. This may seem pointless but it is now far easier to spot warning messages from debug ones.

libevent In lldpd 0.5.7, I was using my own select()-based event loop. It worked but I didn t want to grow a full-featured event loop inside lldpd. Therefore, I switched to libevent. The minimal required version of libevent is 2.0.5. A convenient way to check the changes in API is to use Upstream Tracker, a website tracking API and ABI changes for various libraries. This version of libevent is not available in many stable distributions. For example, Debian Squeeze or Ubuntu Lucid only have 1.4.13. I am also trying to keep compatibility with very old distributions, like RHEL 2, which does not have a packaged libevent at all. For some users, it may be a burden to compile additional libraries. Therefore, I have included libevent source code in lldpd source tree (as a git submodule) and I am only using it if no suitable system libevent is available. Have a look at m4/libevent.m4 and src/daemon/Makefile.am to see how this is done.

Client

Serialization lldpctl is a client querying lldpd to display discovered neighbors. The communication is done through an Unix socket. Each structure to be serialized over this socket should be described with a string. For example:

#define STRUCT_LLDPD_DOT3_MACPHY "(bbww)"
struct lldpd_dot3_macphy  
        u_int8_t                 autoneg_support;
        u_int8_t                 autoneg_enabled;
        u_int16_t                autoneg_advertised;
        u_int16_t                mau_type;
 ;

I did not want to use stuff like Protocol Buffers because I didn t want to copy the existing structures to other structures before serialization (and the other way after deserialization). However, the serializer in lldpd did not allow to handle reference to other structures, lists or circular references. I have written another one which works by annotating a structure with some macros:

struct lldpd_chassis  
    TAILQ_ENTRY(lldpd_chassis) c_entries;
    u_int16_t        c_index;
    u_int8_t         c_protocol;
    u_int8_t         c_id_subtype;
    char            *c_id;
    int              c_id_len;
    char            *c_name;
    char            *c_descr;
    u_int16_t        c_cap_available;
    u_int16_t        c_cap_enabled;
    u_int16_t        c_ttl;
    TAILQ_HEAD(, lldpd_mgmt) c_mgmt;
 ;
MARSHAL_BEGIN(lldpd_chassis)
MARSHAL_TQE  (lldpd_chassis, c_entries)
MARSHAL_FSTR (lldpd_chassis, c_id, c_id_len)
MARSHAL_STR  (lldpd_chassis, c_name)
MARSHAL_STR  (lldpd_chassis, c_descr)
MARSHAL_SUBTQ(lldpd_chassis, lldpd_mgmt, c_mgmt)
MARSHAL_END;

Only pointers need to be annotated. The remaining of the structure can be serialized with just memcpy()¹. I think there is still room for improvement. It should be possible to add annotations inside the structure and avoid some duplication. Or maybe, using a C parser? Or using the AST output from LLVM?

Library In lldpd 0.5.7, there are two possible entry points to interact with the daemon:

1. Through SNMP support. Only information available in LLDP-MIB are exported. Therefore, implementation-specific values are not available. Moreover, SNMP support is currently read-only.
2. Through lldpctl. Thanks to a contribution from Andreas Hofmeister, the output can be requested to be formatted as an XML document.

Integration of lldpd into a network stack was therefore limited to one of those two channels. As an exemple, you can have a look at how Vyatta made the integration using the second solution. To provide a more robust solution, I have added a shared library, liblldpctl, with a stable and well-defined API. lldpctl is now using it. I have followed those directions²:

• Consistent naming (all exported symbols are prefixed by lldpctl_). No pollution of the global namespace.
• Consistent return codes (on errors, all functions returning pointers are returning NULL, all functions returning integers are returning -1).
• Reentrant and thread-safe. No global variables.
• One well-documented include file.
• Reduce the use of boilerplate code. Don t segfault on NULL, accept integer input as string, provide easy iterators,
• Asynchronous API for input/output. The library delegates reading and writing by calling user-provided functions. Those functions can yield their effects. In this case, the user has to callback the library when data is available for reading or writing. It is therefore possible to integrate the library with any existing event-loop. A thin synchronous layer is provided on top of this API.
• Opaque types with accessor functions.

Accessing bits of information is done through atoms which are opaque containers of type lldpctl_atom_t. From an atom, you can extract some properties as integers, strings, buffers or other atoms. The list of ports is an atom. A port in this list is also an atom. The list of VLAN present on this port is an atom, as well as each VLAN in this list. The VLAN name is a NULL-terminated string living in the scope of an atom. Accessing a property is done by a handful of functions, like lldpctl_atom_get_str(), using a specific key. For example, here is how to display the list of VLAN assuming you have one port as an atom:

vlans = lldpctl_atom_get(port, lldpctl_k_port_vlans);
lldpctl_atom_foreach(vlans, vlan)  
    vid = lldpctl_atom_get_int(vlan,
                               lldpctl_k_vlan_id));
    name = lldpctl_atom_get_str(vlan,
                                lldpctl_k_vlan_name));
    if (vid && name)
        printf("VLAN %d: %s\n", vid, name);
 
lldpctl_atom_dec_ref(vlans);

Internally, an atom is typed and reference counted. The size of the API is greatly limited thanks to this concept. There are currently more than one hundred pieces of information that can be retrieved from lldpd. Ultimately, the library will also enable the full configuration of lldpd. Currently, many aspects can only be configured through command-line flags. The use of the library does not replace lldpctl which will still be available and be the primary client of the library.

CLI Having a configuration file was requested since a long time. I didn t want to include a parser in lldpd: I am trying to keep it small. It was already possible to configure lldpd through lldpctl. Locations, network policies and power policies were the three items that could be configured this way. So, the next step was to enable lldpctl to read a configuration file, parse it and send the result to lldpd. As a bonus, why not provide a full CLI accepting the same statements with inline help and completion?

Parsing & completion Because of completion, it is difficult to use a YACC generated parser. Instead, I define a tree where each node accepts a word. A node is defined with this function:

struct cmd_node *commands_new(
    struct cmd_node *,
    const char *,
    const char *,
    int(*validate)(struct cmd_env*, void *),
    int(*execute)(struct lldpctl_conn_t*, struct writer*,
        struct cmd_env*, void *),
    void *);

A node is defined by:

• its parent,
• an optional accepted static token,
• an help string,
• an optional validation function and
• an optional function to execute if the current token is accepted.

When walking the tree, we maintain an environment which is both a key-value store and a stack of positions in the tree. The validation function can check the environment to see if we are in the right context (we want to accept the keyword foo only once, for example). The execution function can add the current token as a value in the environment but it can also pop the current position in the tree to resume walk from a previous node. As an example, see how nodes for configuration of a coordinate-based location are registered:

/* Our root node */
struct cmd_node *configure_medloc_coord = commands_new(
    configure_medlocation,
    "coordinate", "MED location coordinate configuration",
    NULL, NULL, NULL);
/* The exit node.
   The validate function will check if we have both
   latitude and longitude. */
commands_new(configure_medloc_coord,
    NEWLINE, "Configure MED location coordinates",
    cmd_check_env, cmd_medlocation_coordinate,
    "latitude,longitude");
/* Store latitude. Once stored, we pop two positions
   to go back to the "root" node. The user can only
   enter latitude once. */
commands_new(
    commands_new(
        configure_medloc_coord,
        "latitude", "Specify latitude",
        cmd_check_no_env, NULL, "latitude"),
    NULL, "Latitude as xx.yyyyN or xx.yyyyS",
    NULL, cmd_store_env_value_and_pop2, "latitude");
/* Same thing for longitude */
commands_new(
    commands_new(
        configure_medloc_coord,
        "longitude", "Specify longitude",
        cmd_check_no_env, NULL, "longitude"),
    NULL, "Longitude as xx.yyyyE or xx.yyyyW",
    NULL, cmd_store_env_value_and_pop2, "longitude");

The definition of all commands is still a bit verbose but the system is simple enough yet powerful enough to cover all needed cases.

Readline When faced with a CLI, we usually expect some perks like completion, history handling and help. The most used library to provide such features is the GNU Readline Library. Because this is a GPL library, I have first searched an alternative. There are several of them:

• NetBSD Editline library (libedit).
• Autotool port of the NetBSD Editline library (also libedit).
• Debian version of the Editline library (libeditline).
• linenoise, a small and minimal readline library.
• Many others.

From an API point of view, the first three libraries support the GNU Readline API. They also have a common native API. Moreover, this native API also handles tokenization. Therefore, I have developed the first version of the CLI with this API³. Unfortunately, I noticed later this library is not very common in the Linux world and is not available in RHEL. Since I have used the native API, it was not possible to fallback to the GNU Readline library. So, let s switch! Thanks to the appropriate macro from the Autoconf Archive (with small modifications), the compilation and linking differences between the libraries are taken care of. Because GNU Readline library does not come with a tokenizer, I had to write one myself. The API is also badly documented and it is difficult to know which symbol is available in which version. I have limited myself to:

• readline(), addhistory(),
• rl_insert_text(),
• rl_forced_update_display(),
• rl_bind_key()
• rl_line_buffer and rl_point.

Unfortunately, the various libedit libraries have a noop for rl_bind_key(). Therefore, completion and online help is not available with them. I have noticed that most BSD come with GNU Readline library preinstalled, so it could be considered as a system library. Nonetheless, linking with libedit to avoid licensing issues is possible and help can be obtained by prefixing the command with help.

OS specific support

Netlink on Linux Previously, the list of interfaces was retrieved through getifaddrs(). lldpd is now using directly Netlink on Linux. This is not a big change since the GNU C Library already uses it to implement getifaddrs() and additional information, like VLAN, are still retrieved through ioctl() or sysfs. However, lldpd now gets notified when a change happens and update all interfaces in the next second. Like many other projects, I have written my own Netlink implementation instead of using libnl, a nice collection of libraries providing everything you need to query the kernel through Netlink, including some advanced bits. Why?

1. The latest version of libnl is still young and its availability in major distributions is scarce. It is not available in Debian Squeeze but will be available in Debian Wheezy. Like libevent, I could circumvent this problem by shipping the library with lldpd and use it when there is not system alternative. But
2. libnl is licensed under LGPL 2.1. This makes static linking difficult because the license is quite shaddy about static linking being derivative work or not. It is believed that it is authorized under the same provisions as in LGPL 3 which handles the case explicitely. This has been a problem with many projects. For example, OGRE has added an exception for static linking in version 1.6 and switched to MIT license in version 1.7.

I had a short discussion with Thomas Graf about this issue and he seems willing to add a similar exception. This may take some time, but once this is done, I will happily switch to libnl and retrieve more stuff from Netlink.

BSD support Until version 0.7, lldpd was Linux-only. The rewrite to use Netlink was the occasion to abstract interfaces and to port to other OS. The first port was for Debian GNU/kFreeBSD, then for FreeBSD, OpenBSD and NetBSD. They all share the same source code:

• getifaddrs() to get the list of interfaces,
• bpf(4) to attach to an interface to receive and send packets,
• PF_ROUTE socket to be notified when a change happens.

Each BSD has its own ioctl() to retrieve VLAN, bridging and bonding bits but they are quite similar. The code was usually adapted from ifconfig.c. The BSD ports have the same functionalities than the Linux port, except for NetBSD which lacks support for LLDP-MED inventory since I didn t find a simple way to retrieve DMI related information. They also offer greater security by filtering packets sent. Moreover, OpenBSD allows to lock the filters set on the socket:

/* Install write filter (optional) */
if (ioctl(fd, BIOCSETWF, (caddr_t)&fprog) < 0)  
    rc = errno;
    log_info("privsep", "unable to setup write BPF filter for %s",
        name);
    goto end;
 
/* Lock interface */
if (ioctl(fd, BIOCLOCK, (caddr_t)&enable) < 0)  
    rc = errno;
    log_info("privsep", "unable to lock BPF interface %s",
        name);
    goto end;
 

This is a very nice feature. lldpd is using a privileged process to open the raw socket. The socket is then transmitted to an unprivileged process. Without this feature, the unprivileged process can remove the BPF filters. I have ported the ability to lock a socket filter program to Linux. However, I still have to add a write filter.

OS X support Once FreeBSD was supported, supporting OS X seemed easy. I got sponsored by xcloud.me which provided a virtual Mac server. Making lldpd work with OS X took only two days, including a full hour to guess how to get Apple Xcode without providing a credit card. To help people installing lldpd on OS X, I have also written a lldpd formula for Homebrew which seems to be the most popular package manager for OS X.

Upstart and systemd support Many distributions propose upstart and systemd as a replacement or an alternative for the classic SysV init. Like most daemons, lldpd detaches itself from the terminal and run in the background, by forking twice, once it is ready (for lldpd, this just means we have setup the control socket). While both upstart and systemd can accommodate daemons that behave like this, it is recommended to not fork. How to advertise readiness in this case? With upstart, lldpd will send itself the SIGSTOP signal. upstart will detect this, resume lldpd with SIGCONT and assume it is ready. The code to support upstart is therefore quite simple. Instead of calling daemon(), do this:

const char *upstartjob = getenv("UPSTART_JOB");
if (!(upstartjob && !strcmp(upstartjob, "lldpd")))
    return 0;
log_debug("main", "running with upstart, don't fork but stop");
raise(SIGSTOP);

The job configuration file looks like this:

# lldpd - LLDP daemon
description "LLDP daemon"
start on net-device-up IFACE=lo
stop on runlevel [06]
expect stop
respawn
script
  . /etc/default/lldpd
  exec lldpd $DAEMON_ARGS
end script

systemd provides a socket to achieve the same goal. An application is expected to write READY=1 to the socket when it is ready. With the provided library, this is just a matter of calling sd_notify("READY=1\n"). Since sd_notify() has less than 30 lines of code, I have rewritten it to avoid an external dependency. The appropriate unit file is:

[Unit]
Description=LLDP daemon
Documentation=man:lldpd(8)
[Service]
Type=notify
NotifyAccess=main
EnvironmentFile=-/etc/default/lldpd
ExecStart=/usr/sbin/lldpd $DAEMON_ARGS
Restart=on-failure
[Install]
WantedBy=multi-user.target

OS include files Linux-specific include files were a major pain in previous versions of lldpd. The problems range from missing header files (like linux/if_bonding.h) to the use of kernel-only types. Those headers have a difficult history. They were first shipped with the C library but were rarely synced and almost always outdated. They were then extracted from kernel version with almost no change and lagged behind the kernel version used by the released distribution⁴. Today, the problem is acknowledged and is being solved by both the distributions which extract the headers from the packaged kernel and by kernel developers with a separation of kernel-only headers from user-space API headers. However, we still need to handle legacy. A good case is linux/ethtool.h:

• It can just be absent.
• It can use u8, u16 types which are kernel-only types. To work around this issue, type munging can be setup.
• It can miss some definition, like SPEED_10000. In this case, you either define the missing bits and find yourself with a long copy of the original header interleaved with #ifdef or conditionally use each symbol. The latest solution is a burden by itself but it also hinders some functionalities that can be available in the running kernel.

The easy solution to all this mess is to just include the appropriate kernel headers into the source tree of the project. Thanks to Google ripping them for its Bionic C library, we know that copying kernel headers into a program does not create a derivative work.

---

1. Therefore, the use of u_int16_t and u_int8_t types is a left-over of the previous serializer where the size of all members was important.
2. For more comprehensive guidelines, be sure to check Writing a C library.
3. Tokenization is not the only advantage of libedit native API. The API is also cleaner, does not have a global state and has a better documentation. All the implementations are also BSD licensed.
4. For example, in Debian Sarge, the Linux kernel was a 2.6.8 (2004) while the kernel headers were extracted from some pre-2.6 kernel.

DRI3000 Even Better Direct Rendering This all started with the presentation that Eric Anholt and I did at the 2012 X developers conference, and subsequently wrote about in my DRI-Next posting. That discussion sketched out the goals of changing the existing DRI2-based direct rendering infrastructure. Last month, I gave a more detailed presentation at Linux.conf.au 2013 (the best free software conference in the world). That presentation was recorded, so you can watch it online. Or, you can read Nathan Willis summary at lwn.net. That presentation contained a lot more details about the specific techniques that will be used to implement the new system, in particular it included some initial indications of what kind of performance benefits the overall system might be able to produce. I sat down today and wrote down an initial protocol definition for two new extensions (because two extensions are always better than one). Together, these are designed to provide complete support for direct rendering APIs like OpenGL and offer a better alternative to DRI2. The DRI3 extension Dave Airlie and Eric Anholt refused to let me call either actual extension DRI3000, so the new direct rendering extension is called DRI3. It uses POSIX file descriptor passing to share kernel objects between the X server and the application. DRI3 is a very small extension in three requests:

1. Open. Returns a file descriptor for a direct rendering device along with the name of the driver for a particular API (OpenGL, Video, etc).
2. PixmapFromBuffer. Takes a kernel buffer object (Linux uses DMA-BUF) and creates a pixmap that references it. Any place a Pixmap can be used in the X protocol, you can now talk about a DMA-BUF object. This allows an application to do direct rendering, and then pass a reference to those results directly to the X server.
3. BufferFromPixmap. This takes an existing pixmap and returns a file descriptor for the underlying kernel buffer object. This is needed for the GL Texture from Pixmap extension.

For OpenGL, the plan is to create all of the buffer objects on the client side, then pass the back buffer to the X server for display on the screen. By creating pixmaps, we avoid needing new object types in the X server and can use existing X apis that take pixmaps for these objects. The Swap extension Once you ve got direct rendered content in a Pixmap, you ll want to display it on the screen. You could simply use CopyArea from the pixmap to a window, but that isn t synchronzied to the vertical retrace signal. And, the semantics of the CopyArea operation precludes us from swapping the underlying buffers around, making it more expensive than strictly necessary. The Swap extension fills those needs. Because the DRI3 extension provides an X pixmap reference to the direct rendered content, the Swap extension doesn t need any new object types for its operation. Instead, it talks strictly about core X objects, using X pixmaps as the source of the new data and X drawables as the destination. The core of the Swap extension is one request SwapRegion. This request moves pixels from a pixmap to a drawable. It uses an X fixes Region object to specify the area of the destination being painted, and an offset within the source pixmap to align the two areas. A bunch of data are included in the reply from the SwapRegion request. First, you get a 64-bit sequence number identifying the swap itself. Then, you get a suggested geometry for the next source pixmap. Using the suggested geometry may result in performance improvements from the techniques described in the LCA talk above. The last bit of data included in the SwapRegion reply is a list of pixmaps which were used as source operands to earlier SwapRegion requests to the same drawable. Each pixmap is listed along with the 64-bit sequence number associated with an earlier SwapRegion operation which resulted in the contents which the pixmap now contains. Ok, so that sounds really confusing. Some examples are probably necessary.

• If the SwapRegion operation was implemented by copying data out of the source pixmap into the destination drawable, then the idle swap count will be equal to the swap count from this SwapRegion operation.
• If the SwapRegion operation was implemented by swapping the destination contents with the source contents, then the idle swap count will be equal to the previous swap count on the destination drawable.

I m hoping you ll be able to tell that in both cases, the idle swap count tries to name the swap sequence at which time the destination drawable contained the contents currently in the pixmap. Note that even if the SwapRegion is implemented as a Copy operation, the provided source pixmap may not be included in the idle list as the copy may be delayed to meet the synchronization requirements specfied by the client. Finally, if you want to throttle rendering based upon when frames appear on the screen, Swap offers an event that can be delivered to the drawable after the operation actually takes place. Because the Swap extension needs to supply all of the OpenGL SwapBuffers semantics (including a multiplicity of OpenGL extensions related to that), I ve stolen a handful of DRI2 requests to provide the necessary bits for that:

1. SwapGetMSC
2. SwapWaitMSC
3. SwapWaitSBC

These work just like the DRI2 requests of the same names. Current State of the Extensions Both of these extensions have an initial protocol specification written down and stored in git:

1. DRI3 protocol
2. Swap protocol

A new Rcpp master class is scheduled for March 9 in New York. The format will an updated version of the one-day workshops I have given at the University of Rochester in 2010, in San Franciso in 2011 (organised by Revolution Analytics) and at the UseR! conference in 2012. The style will be hands-on, with numerous concrete examples and solid coverage of most aspects of Rcpp and related packages. As before, about six hours of instruction, split into four sessions of around ninety minutes focussing (loosely) on motivation/intro, core parts, extensions and applications. This should leave ample time for informal discussions and Q+A---as well as for lunch and coffee breaks---for a total of eight hours in the classroom. This is being put together in New York with the help of Jared Lander, and we will have some technical assistance from RStudio in order to use their EC2 farm for exercises with Rcpp. Registrations details are available here; information about other Rcpp events is also available. Feel free to contact me or Jared at our usual email addresses with questions.

A new minor release RQuantLib 0.3.10 is now on CRAN and in Debian. RQuantLib combines (some of) the quantitative analytics of QuantLib with the R statistical computing environment and language. The discount curve building code in QuantLib has shown some overly large numerical instabilities. We have used the same example parameters (taken from the Swap example in QuantLib) for years; it currently fails to solve for a rate at some term further out the curve. So I made the decistion to disable this just in the examples in order to not upset the CRAN testing framework. The examples now use a flat curve instead. I also updated one function to silence some new warnings from R-devel about symbols from another packages's namespace (in this case rgl, and it is just for surface plots, a purely cosmetic function). Thanks to CRANberries, there is also a diff to the previous release 0.3.9. Full changelog details, examples and more details about this package are at my RQuantLib page.

Server management on daily basis in a growing environment could be a challenge. Most of the tasks is here done with ClusterSSH in help with a homebrew mail notification. Unfortunately this workflow is not scaling well for 200+ systems. I had a look into apt-dater but this seems not to fit our workflow. FAI looks a bit overengineered at the first glance. Maybe Puppet is an option? There seems to be many extensions for it, unfortunately ruby is needed on every single node. Is there a CTO-compatible overview about puppet, any other alternatives? ;) Update: Many thanks for the hints from Andr Lu s Lopes, Stig Sandbeck Mathisen, Phil Miller, Steve Kemp, Andrew Latham, Bob Proulx and Natxo Asenjo

• Chef
  • Packaged in Debian
  • Upstream Debian packages (no recent one)
• CFEngine3
  • No ruby/perl/python dependencies
  • Light, fast
  • (A bit) harder to understand
  • Packaged in Debian
  • Upstream Debian packages
• Salt
  • Very young
  • Packaged in Debian
• Ansible
  • Very flexible
  • Easy to understand
  • Relies on ssh
  • Open ITP
  • PPA
• MCollective
  • Framework for building server orchestration and parallel job execution
  • Could extend for example Puppet or Chef
  • Packaged in Debian
  • Upstream Debian packages (no recent one)
• Foreman
  • Puppet dashboard and node classifier
  • Open ITP
  • Upstream Debian packages

Old but conceptual still valid: www.infrastructures.org, for example Bootstrapping an Infrastructure Other thoughts: push- or pull-infrastructure? Software footprint (on nodes and your central instance)! Hints, rants and comments could be send to 'blog - at - waja - dot - info' or via @blogwajainfo.

Garmin Montana notes Sometimes, I get waaay too excited about gadgets, and I have to share it. So, sorry, here it goes For some reason that is not very important, I was looking for a better solution for GPSes with maps. Side-note: if you ask why Android is not good enough , here's the story: last year, knowing that Google Maps supports caching (finally), I upgraded Maps, preloaded London, and went on a trip. Only to exit the Victoria underground station and to stare ~20 minutes in light rain at my phone, wondering why it doesn't get a fix. I gave up in the end, and used the cached maps for getting around. Later I learned the reason: Maps was not broken, but without data, Android can't get a lock easily; and more-over, the recent Maps update without GPS active even once afterwards, to do whatever it needed, meant I couldn't get a lock until back in Switzerland. Or, alternatively: while on a trip to Japan, it turns out that Google Maps caching is not supported there, probably because of rights over mapping data. So I had location, but only temporary cached maps (from the wifi in the hotels). Due to this kind of problems makes I consider Android maps a toy, so I wanted a GPS device with proper maps (on the device, not in the cloud), and with good GPS capabilities (no fancy A-GPS or anything, just plain good standalone GPS). Having had good experiences with the sport line of Garmin (but which doesn't have maps, so it's out of the question), I looked at Garmin. Many people complained about some things (same all over the place):

• newer devices less tested, software with lots of bugs
• dumbing down of devices, ultra-simplification of the UI

The latter is something that jumped at me from many of the blogs/fora that I read; I like customisable power tools , not one-button mice. The other thing that I didn't really like was that most devices are very focused on a niche: the n vi line on automotive, the Zumo line on motorbikes, and the hand-held lines with some blurring of hiking/geocaching/etc. So no one-size fits all device Features! Customisation! That is, until I found out the Montana series. My my, I was blown. Both by the price (I couldn't believe how expensive it was, without maps!) and by the features. As it turns out, the Montana has different profiles , and it can act:

• as a hand-held device for hiking/geocaching (yay!)
• as a bicycle-mounted device for biking through the woods (yay!)
• as a motorcycle-mounted device (hmmm )
• as a car-mounted device, for automotive use, including guidance, both TTS and non-TTS voices, lane assist, speed limits, etc.

That is for flexibility. As for performance/features:

• it has a Transflective liquid crystal display screen, so it works best (as opposed to worst) in bright sun, and of course also works in the dark, via its back light
• the touchscreen is resistive, so glove-friendly: tested it with my regular gloves on and with my biking gloves (which are very clumsy), and it works like a charm
• it is water-proof (IPX7); that means I can keep it in the open in the rain, and not worry about dropping it in the snow
• it has both a (replaceable) LiIon battery (many hours of runtime, > 10) and support for powering via 3 AA batteries (> 20 hours), which coupled with some Eneloops is awesome
• GPS sync is awesome; from hard poweroff, it syncs in ~3 seconds I would say (exited building, powered on, was already synced by the time I entered the satellite menu); even in-house, it gets around 2 satellites, and sometimes manages to get a lock

Now for the downsides. Being a multi-functional device, it's not as good as a dedicated one:

• for handheld use, it's a bit heavy; I personally don't mind, but it's something that has to be said: ~300 g
• for motorcycle use, it's worse than the Zumo: no bluetooth compatibility, so you can't have it talk to you in the helmet; but I don't have a motorcycle, so
• for automotive use, it lacks many things compared to a top-of-the-live n vi: no voice recognition, no junction view, no birds-eye view, etc.; but I don't have a car, so

But the small (for me) draw-backs aside, it's definitely not a dumbed-down device. It's the first device that I have which I feel it a bit too customisable . From completely changeable menus, to profiles that remember their maps selection, to many dashboards, to almost-programmable shortcuts (e.g. a shortcut to change profile, set destination to a given way point, reset the trip log, and turn track logging on) a bit overwhelming at first. Add the 3-axis compass, barometric altimeter, ANT+ support and you get a very powerful device. Software As to the software, yes it has some bugs - mainly that the device crashes sometimes (abruptly) if you play too much with the routes (didn't happen to me yet on the road , only when testing it at home). On the PC-side, Garmin's BaseCamp is nice, but doesn't hold a candle to Google Maps, for example, in terms of UI; I don't mean shininess, but simply how well the UI works. The most annoying thing is how it doesn't scale detail appropriately with the zoom level. But it works, and I can plan hikes offline, and upload both tracks and routes to it (this is important: it seems other recent Garmins can't replay tracks, only routes). And I can load many maps, and toggle them at will, etc. The many maps feature is not to be overlooked. I could have, for example, standard Garmin maps loaded, but just in case they're insuficiently detailed or out of date (hey, Navteq is slow ), I can also have Open Street Maps from two days ago. The routing is then done on the currently active maps, so you are not restricted to a single mapping provider. And of course you can also have custom POIs (Point of interest) on it. Bycicle use The bicycle mount is solid enough, so that on a rough roads it's held firmly. From my ~20 minute test this morning, a few things to note:

• it wanted me to take bigger paths, rather than the trails I like to take; probably I haven't customised it well enough yet
• recalculating a route (when it realises you insist on taking another path than it planned) is very fast
• whenever you approach a change-of-direction, it beeps (see below) and turns on the back light (temporarily), so visibility is improved
• you can actually use the touchscreen with winter gloves (of course, when stopped or at very very low speed)

Voice output By default, the unit can only beep, but if coupled via a 3.5" jack to headphones or via the automotive mount, it can properly speak; quality varies with voice, and whether it is TTS or not. I managed to find one voice (non-TTS) that sounds really well, but of course with less information, and one TTS voice that sounds OK-ish but gives full info. By full info , I mean the following: the behaviour of the various TTS voices is not consistent. For example, on a simulated route, various voices were saying the following:

• non-TTS: keep ahead (voice A), or keep-ahead, then keep right (voice B)
• voice C, TTS: in 94 kilometers, exit right
• voice D, TTS: in 94 kilometers, take exit 14 on the right
• voice E, TTS: in 94 kilometers, take exit 14 on the right in the direction Landquart, then keep right

I don't know why these differences, or if they are consistent across routes. But I'll keep with voice E for now Summary So, sorry again for sounding like an ad (I don't have any relation with Garmin the company), but I find this device awesome - powerful, rugged, and working (so far) very well. I'm glad that there still exist companies who don't believe they know better than their customers, and instead offer you with a tool that you can make it work just like you want. Or almost Garmin's tag line: Big and Tough Goes anywhere, everywhere . I tend to agree

Technology Technology is... weird. I am in a cabbin in the middle of Lapland an hour away from the next city to speak of. GPS

N 66 15.958
E 24 59.981

look it up. We haven't had power for the last 60 minutes, the electrical heaters are out and we are trying to keep the main room warm with a wood stove while the wood-burning sauna above is ready to receive us the minute I hI push this post. The cabbin has become noticiably colder already, supply of logs small enough to actually burn is limited, we don't have an axe, but we have a car so we can leave if need be. The next house is about six kilometers away, no idea if they have power or not; the woods are dark all round other than for snowshine. Yet, I am sitting here, LED lamp lighting the room, GPS telling me what location to write into this post, laptop running on battery, cell phone with EGPRS supplying me with bugstats plus picture (even though I had to do most of it by hand as LWP::Simple barfs on the timeout) and allowing me to push to ikiwiki. As I said, technology is... weird. Bugology The UDD bugs interface currently knows about the following release critical bugs:

• In Total: 786
  • Affecting wheezy: 194 That's the number we need to get down to zero before the release. They can be split in two big categories:
    • Affecting wheezy and unstable: 128 Those need someone to find a fix, or to finish the work to upload a fix to unstable:
      • 34 bugs are tagged 'patch'. Please help by reviewing the patches, and (if you are a DD) by uploading them.
      • 12 bugs are marked as done, but still affect unstable. This can happen due to missing builds on some architectures, for example. Help investigate!
      • 90 bugs are neither tagged patch, nor marked done. Help make a first step towards resolution!
    • Affecting wheezy only: 66 Those are already fixed in unstable, but the fix still needs to migrate to wheezy. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.
      • 40 bugs are in packages that are unblocked by the release team.
      • 26 bugs are in packages that are not unblocked.

How do we compare to the Squeeze release cycle?

Week	Squeeze	Wheezy	Diff
43	284 (213+71)	468 (332+136)	+184 (+119/+65)
44	261 (201+60)	408 (265+143)	+147 (+64/+83)
45	261 (205+56)	425 (291+134)	+164 (+86/+78)
46	271 (200+71)	401 (258+143)	+130 (+58/+72)
47	283 (209+74)	366 (221+145)	+83 (+12/+71)
48	256 (177+79)	378 (230+148)	+122 (+53/+69)
49	256 (180+76)	360 (216+155)	+104 (+36/+79)
50	204 (148+56)	339 (195+144)	+135 (+47/+90)
51	178 (124+54)	323 (190+133)	+145 (+66/+79)
52	115 (78+37)	289 (190+99)	+174 (+112/+62)
1	93 (60+33)	287 (171+116)	+194 (+111/+83)
2	82 (46+36)	271 (162+109)	+189 (+116/+73)
3	25 (15+10)	249 (165+84)	+224 (+150/+74)
4	14 (8+6)	244 (176+68)	+230 (+168/+62)
5	2 (0+2)	224 (132+92)	+222 (+132/+90)
6	release!	212 (129+83)	+212 (+129/+83)
7	release+1	194 (128+66)	+194 (+128/+66)

Graphical overview of bug stats thanks to azhag:

To convert video files between formats I use Makefiles, this means I can run make -j2 on my dual-core server to get both cores going at once. avconv uses 8 threads for it s computation and I ve seen it take up to 190% CPU time for brief periods of time, but overall it seems to average a lot less, if nothing else then running two copies at once allows one to calculate while the other is waiting for disk IO. Here is a basic Makefile to generate a subdirectory full of mp4 files from a directory full of flv files. I used to use this to convert my Youtube music archive for my Android devices until I installed MX Player which can play every type of video file you can imagine [1]. I ll probably encounter some situation where this script becomes necessary again so I keep it around. It s also a very simple example of how to run a batch conversion of video files. MP4S:=$(shell for n in *.flv ; do echo $$n sed -e s/^/mp4\\// -e s/flv$$/mp4/ ; done)

all: $(MP4S)

mp4/%.mp4: %.flv
avconv -i $< -strict experimental -b $$(~/bin/video-encoding-rate $<) $@ > /dev/null Here is a more complex Makefile. I use it on my directory of big videos (more than 1280*720 resolution) and scales them down for my favorite Android devices (Samsung Galaxy S3, Samsung Galaxy S, and Sony Ericsson Xperia X10). My Galaxy S3 can t play a FullHD version of Gangnam Style without going slow so I need to do this even for the fastest phones. This makefile generates three subdirectories of mp4 files for the three devices. S3MP4S:=$(shell for n in *.mp4 ; do echo $$n sed -e s/^/s3\\// -e s/.mp4$$/-s3.mp4/ -e s/.flv$$/-s3.mp4/; done)
XPERIAMP4S:=$(shell for n in *.mp4 ; do echo $$n sed -e s/^/xperiax10\\// -e s/.mp4$$/-xperiax10.mp4/ -e s/.flv$$/-xperiax10.mp4/; done)
SMP4S:=$(shell for n in *.mp4 ; do echo $$n sed -e s/^/galaxys\\// -e s/.mp4$$/-galaxys.mp4/ -e s/.flv$$/-galaxys.mp4/; done)

all: $(S3MP4S) $(XPERIAMP4S) $(SMP4S)

s3/%-s3.mp4: %.mp4
avconv -i $< -strict experimental -s $(shell ~/bin/video-scale-resolution 1280 720 $<) $@ > /dev/null

galaxys/%-galaxys.mp4: %.mp4
echo avconv -i $< -strict experimental -s $(shell ~/bin/video-scale-resolution 800 480 $<) $@ > /dev/null

xperiax10/%-xperiax10.mp4: %.mp4
echo avconv -i $< -strict experimental -s $(shell ~/bin/video-scale-resolution 854 480 $<) $@ > /dev/null The following script is used by the above Makefile to determine the resolution to use. Some Youtube videos have unusual combinations of width and height (Linkin Park seems to like doing this) so I scale them down so it fits the phone in one dimension and the other dimension is scaled appropriately. This requires a script from the Mplayer package and expects it to be in the location that it s used in the Debian package, for distributions other than Debian a minor change will be required. #!/bin/bash
set -e
OUT_VIDEO_WIDTH=$1
OUT_VIDEO_HEIGHT=$2

eval $(/usr/share/mplayer/midentify.sh $3)
XMULT=$(echo $ID_VIDEO_WIDTH*100/$OUT_VIDEO_WIDTH bc)
YMULT=$(echo $ID_VIDEO_HEIGHT*100/$OUT_VIDEO_HEIGHT bc)
if [ $XMULT -gt $YMULT ]; then
NEWX=$OUT_VIDEO_WIDTH
NEWY=$(echo $OUT_VIDEO_WIDTH*$ID_VIDEO_HEIGHT/$ID_VIDEO_WIDTH/2*2 bc)
else
NEWX=$(echo $OUT_VIDEO_HEIGHT*$ID_VIDEO_WIDTH/$ID_VIDEO_HEIGHT/2*2 bc)
NEWY=$OUT_VIDEO_HEIGHT
fi
echo $ NEWX x$ NEWY Note that I can t preserve TAB characters in a blog post. So those Makefiles won t work until you replace strings of 8 spaces with a TAB character.

• [1] https://play.google.com/store/apps/details?id=com.mxtech.videoplayer.ad

So as previously mentioned I gave a talk at FOSDEM this year on the future of paravirtualisation under Xen. I thought the talk was reasonably well attended despite being at six o'clock in the evening. In the end I underran my half hour slot but fortunately people had plenty of interesting questions! I've posted my slides on my xenbits space. I spent a fair chunk of the rest of the time on the Xen.org booth, which is hard work but it is always nice to speak to users and other developers. I also managed to catch up with various friends from other projects and managed to meet a few folks who I've only ever met in email before. I didn't get to see as many talks as I wanted (for example I missed pretty much all of the ARM track :-() due to clashes with my stints on the booth, my own talk, and talks clashing with each other. Still it was all in all a worthwhile, hopefully I'll be able to make it again next year! Next up: LinaroConnect in Hong Kong!

Subunit is seven and a half years old now Conrad Parker and I first sketched it up at a CodeCon camping and coding, a brilliant combination in mid 2005.

revno: 1
committer: Robert Collins <robertc@robertcollins.net>
timestamp: Sat 2005-08-27 15:01:20 +1000
message:  design up a protocol with kfish

It has proved remarkably resilient as a protocol the basic nature hasn t changed at all, even though we ve added tags, timestamps, support for attachments of arbitrary size. However a growing number of irritations have been building up with it. I think it is time to design another iteration of the protocol, one that will retain the positive qualities of the current protocol, while helping it become suitable for the next 7 years. Ideally we can keep compatibility and make it possible for a single stream to be represented in any format. Existing design The existing design is a mostly human readable line orientated protocol that can be sniffed out from the regular output of make or other build systems. Binary attachments are done using HTTP chunking, and the parser has to maintain state about the current test, tags, timing data and test progression [a simple stack of progress counters]. How to arrange subunit output is undefined, how to select tests to run is undefined. This makes writing a parser quite easy, and the tagging and timestamp facility allow multiplexing streams from two or more concurrent test runs into one with good fidelity but also requires that state be buffered until the end of a test, as two tests cannot be executing at once. Dealing with debuggers The initial protocol was intended to support dropping into a debugger just pass each line read through to stdout, and connect stdin to the test process, and voila, you have a working debugger connection. This works, but the current line based parsers make using it tedious the line buffered nature of it makes feedback on what has been typed fiddly, and stdout tends to be buffered, leading to an inability to see print statements and the like. All in-principle fixable, right ? When running two or more test processes, which test process should stdin be connected to? What if two or more drop into a debugger at once? What is being typed to which process is more luck than anything else. We ve added some idioms in testrepository that control test execution by a similar but different format one test per line to list tests, and have runners permit listing and selecting by a list. This works well, but the inconsistency with subunit itself is a little annoying you need two parsers, and two output formats. Good points The current protocol is extremely easy to implement for emitters, and the arbitrary attachments and tagging features have worked extremely well. There is a comprehensive Python parser which maps everything into Python unittest API calls (an extended version of the standard, with good backwards compatibility). Pain points The debugging support was a total failure, and the way the parser depraminates it s toys when a test process corrupts an outcome line is extremely frustrating. (other tests execute but the parser sees them as non-subunit chatter and passes the lines on through stdout). Dealing with concurrency The original design didn t cater for concurrency. There are three concurrency issues the corruption issue (see below for more detail) and multiplexing. Consider two levels of nested concurrency: A supervisor process such as testrepository starts 2 (or more but 2 is sufficient to reason about the issue) subsidiary worker processes (I1 and I2), each of which starts 2 subsidiary processes of their own (W1, W2, W3, W4). Each of the 4 leaf processes is outputting subunit which gets multiplexed in the 2 intermediary processes, and then again in the supervisor. Why would there be two layers? A concrete example is using testrepository to coordinate test runs on multiple machines at once, with each machine running a local testrepository to broker tests amongst the local CPUs. This could be done with 4 separate ssh sessions and no intermediaries, but that only removes a fraction of the issues. What issues? Well, consider some stdout chatter that W1 outputs. That will get passed to I1 and from there to the supervisor and captured. But there is nothing marking the chatter as belonging to W1: there is no way to tell where it came from. If W1 happened to fail, and there was a diagnostic message printed, we ve lost information. Or at best muddled it all up. Secondly, buffering imagine that a test on W1 hangs. I1 will know that W1 is running a test, but has no way to tell the supervisor (and thus the user) that this is the case, without writing to stdout [and causing a *lot* of noise if that happens a lot]. We could have I1 write to stdout only if W1 s test is taking more than 5 seconds or something but this is a workaround for a limitation of the protocol. Adding to the confusion, the clock on W1 and W3 may be very skewed, so timestamps for everything have to be carefully synchronised by the multiplexer. Thirdly, scheduling if W1/W2 are on a faster machine than W3/W4 then a partition of equal-timed tests onto each machine will lead one idle before the other finishes. It would be nice to be able to pass tests to run to the faster machine when it goes idle, rather than having to start a new runner each time. Lastly, what to do when W1 and W2 both wait for user input on stdin (e.g. passphrases, debugger input, $other). Naively connecting stdin to all processes doesn t work well. A GUI supervisor could connect a separate fd to each of I1 and I2, but that doesn t help when it is W1 and W2 reading from stdin. So additional requirement over baseline subunit:

1. make it possible for stdout and stderr output to be captured from W1 and routed through I1 to the supervisor without losing its origin. It might be chatter from a noisy test, or it might be build output. Either way, the user probably will benefit if we can capture it and show it to them later when they review the test run. The supervisor should probably show it immediately as well the protocol doesn t need to care about that, just make it possible.
2. make it possible to pass information about tests that have not completed through one subunit stream while they are still incomplete.
3. make it possible (but optional) to pass tests to run to a running process that accepts subunit.
4. make it possible to route stdin to a specific currently process like W1. This and point 3 suggest that we need a bidirectional protocol rather than the solely unidirectional protocol we have today. I don t know of a reliable portable way to tell when some process is seeking such input, so that will be up to the user I think. (e.g. printing (pdb) to stdout might be a sufficiently good indicator.)

Dealing with corruption Consider the following subunit fragment:

test: foo
starting serversuccess:foo

This is a classic example of corruption: the test foo started a server and helpfully wrote to stdout explaining that it did that, but missed the newline. As a result the success message for the test wasn t printed on a line of its own, and the subunit parser will believe that foo never completed. Every subsequent test is then ignored. This is usually easy to identify and fix, but its a head-scratcher when it happens. Another way it can happen is when a build tool like make runs tests in parallel, and they output subunit onto the same stdout file handle. A third way is when a build tool like make runs two separate test scripts serially, and the first one starts a test but errors hard and doesn t finish it. That looks like:

test: foo
test: bar
success: bar

One way that this sort of corruption can be mitigated is to put subunit on it s own file descriptor, but this has several caveats: it is harder to tunnel through things like ssh and it doesn t solve the failing test script case. I think it is unreasonable to require a protocol where arbitrary interleaving of bytes between different test runner streams will work so the make -j2 case can be ignored at the wire level though we should create a simple way to safely mux the output from such tests when the execute. The root of the issue is that a dropped update leaves bad state in the parser and it never recovers. So some way to recover, or less state to carry in the parser, would neatly solve things. I favour reducing parser state as that should shift stateful complexity onto end nodes / complex processors, rather than being carried by every node in the transmission path. Dependencies Various suggestions have been made JSON, Protobufs, etc A key design goal of the first subunit was a low barrier to entry. We keep that by being backward compatible, but being easy to work with for the new revision is also a worthy goal. High level proposal A packetised length prefixed binary protocol, with each packet containing a small signature, length, routing code, a binary timestamp in UTC, a set of UTF8 tags (active only, no negative tags), a content tag one of (estimate + number, stdin, stdout, stderr, test- + test id), test status (one of exists/inprogress/xfail/xsuccess/success/fail/skip), an attachment name, mime type, a last-block marker and a block of bytes. The content tags:

• estimate the stream is reporting how many tests are expected to run. It affects everything with the same routing code only, and replaces (doesn t adjust) any current estimate for that routing code. A estimate packet of 0 can be used to say that a routing target has shutdown and cannot run more tests. Routing codes can be used by a subunit aware runner to separate out separate threads in a single process, or even just separate TestSuite objects within a single test run (though doing so means that they will need to process subunit and strip packets on stdin. This supercedes the stack of progress indicators that current subunit has. estimates cannot have test status or attachments.
• stdin/stdout/stderr: a packet of data for one of these streams. The routing code identifies the test process that the data came from/should go to in the tree of test workers. These packets cannot have test status but should have a non-empty attachment block.
• test- + testid: a packet of data for a single test. test status may be included, as may attachment name, mime type, last-block and binary data.

Test status values are pretty ordinary. Exists is used to indicate a test that can be run when listing tests, and inprogress is used to report a test that has started but not necessarily completed. Attachment names must be unique per routing code + testid. So how does this line up? Interleaving and recovery We could dispense with interleaving and say the streams are wholly binary, or we can say that packets can start either after a \n or directly after another packet. If we say that binary-only is the approach to take, it would be possible to write a filter that would apply the newline heuristic (or even look for packet headers at every byte offset. I think mandating adjacent to a packet or after \n is a small concession to make and will avoid tools like testrepository forcing users to always configure a heuristic filter. Non-subunit content can be wrapped in subunit for forwarding (the I1 in W1->I1->Supervisor chain would do the wrapping). This won t eliminate corruption but it will localise it and permit the stream to recover: the test that was corrupted will show up as incomplete, or with incomplete attachment data. listing Test listing would emit many small non-timestamped packets. It may be useful to have a wrapper packet for bulk amounts of fine grained data like listing is, or for multiplexers with many input streams that will often have multiple data packets available to write at once. Selecting tests to run Same as for listing while passing regexes down to the test runner to select groups of tests is a valid use case, thats not something subunit needs to worry about : if the selection is not the result of the supervisor selecting by test id, then it is known at the start of the test run and can just be a command line parameter to the backend : subunit is relevant for passing instructions to a runner mid-execution. Because the supervisor cannot just hand out some tests and wait for the thing it ran to report that it can accept incremental tests on stdin, supervisor processes will need to be informed about that out of band. Debugging Debugging is straight forward . The parser can read the first 4 or so bytes of a packet one at a time to determine if it is a packet or a line of stdout/stderr, and then either read to end of line, or the binary length of the packet. So, we combine a few things; non-subunit output should be wrapped and presented to the user. Subunit that is being multiplexed and forwarded should prepend a routing code to the packet (e.g. I1 would append 1 or 2 to select which of W1/W2 the content came from, and then forward the packet. S would append 1 or 2 to indicate I1/I2 the routing code is a path through the tree of forwarding processes). The UI the user is using needs to supply some means to switch where stdin is attached. And stdin input should be routed via stdin packets. When there is no routing code left, the packet should be entirely unwrapped and presented as raw bytes to the process in question. Multiplexing Very straight forward unwrap the outer layer of the packet, add or adjust the routing code, serialise a header + adjusted length + rest of packet as-is. No buffering is needed, so the supervisor can show in-progress tests (and how long they have been running for). Parsing / representation in Python or other languages The parser should be very simple to write. Once parsed, this will be fundamentally different to the existing Python TestCase->TestResult API that is in used today. However it should be easy to write two adapters: old-style <-> this new-style. old-style -> new-style is useful for running existing tests suites and generating subunit, because thats way the subunit generation is transparent. new-style->old-style is useful for using existing test reporting facilities (such as junitxml or html TestResult objects) with subunit streams. Importantly though, a new TestResult style that supports the features of this protocol would enable some useful features for regular Python test suites:

• Concurrent tests (e.g. in multiprocessing) wouldn t need multiplexers and special adapters a regular single testresult with a simple mutex around it would be able to handle concurrent execution of tests, and show hung tests etc.
• The routing of input to a particular debugger instance also applies to a simple python process running tests via multiprocessing, so the routing feature would help there.
• The listing facility and incrementally running tests would be useful too I think we could go to running tests concurrently with test collection happening, but this would apply to other parts of unittest than just the TestResult

The API might be something like:

class StreamingResult(object):
    def startTestRun(self):
        pass
    def stopTestRun(self):
        pass
    def estimate(self, count, route_code=None):
        pass
    def stdin(self, bytes, route_code=None):
        pass
    def stdout(self, bytes, route_code=None):
        pass
    def test(self, test_id, status, attachment_name=None, attachment_mime=None, attachment_eof=None, attachment_bytes=None):
        pass

This would support just-in-time debugging by wiring up pdb to the stdin/stdout handlers of the result object, rather than actual stdin/stdout of the process a simple matter once written. Alternatively, the test runner could replace sys.stdin/stdout etc with thunk file-like objects, which might be a good idea anyway to capture spurious output happening during a test run. That would permit pdb to Just Work (even if the test process is internally running concurrent tests.. until it has two pdb objects running concurrently Generation new streams Should be very easy in anything except shell. For shell, we can have a command line tool that when invoked outputs a subunit stream for one instruction. E.g. test foo completed + some attachments or test foo starting .

From time to time it occurs that two people answer a mail in the same way where one would do closing an unblock request, for example. When this almost happened on debian-release the other day I amused myself by dreaming up an SMTP header that would prevent such embarrassment. I wasn t being serious in the slightest, but nevertheless X-RaceProtection was born (and it turns out at least one resident of a certain IRC channel thought I was).

X-RaceProtection can be a message identifier or the simple value yes and is intended to prevent duplicate replies to, for example, mailing lists. When set as a mail ID, list software should silently drop the message being delivered if the identified message has already received a reply that is, another message quoting that ID in In-Reply-To. If X-RaceProtection is simply yes , the mail ID of In-Reply-To for the message being delivered is used, providing a shortcut.

This means you can set X-RaceProtection when replying to a mail where there is a chance of collision. If someone beat you to it, there is no embarrassment at your mail arriving with a later timestamp. If someone fancies implementing this for smartlist/debbugs, please be my guest!

---

X-RaceProtection: yes is a post from: jwiltshire.org.uk Flattr

Brett points at a Guardian article on Terry Deary and his opinion that libraries are no longer relevant. In fact, Mr. Deary goes further: he thinks libraries are destroying the book business by preventing authors, publishers, and bookshops from getting paid. From the article:

As one of the most popular library authors his books were borrowed more than 500,000 times during 2011/12 Deary will have received the maximum amount possible for a writer from the Public Lending Right scheme, which gives authors 6.2p every time one of their books is borrowed, up to a cap of 6,600. "If I sold the book I'd get 30p per book. I get six grand, and I should be getting 180,000. But never mind my selfish author perception what about the bookshops? The libraries are doing nothing for the book industry. They give nothing back, whereas bookshops are selling the book, and the author and the publisher get paid, which is as it should be. What other entertainment do we expect to get for free?" he asked.

He's making several of the same false claims that people in the media business always make whenever they see people enjoying any kind of cultural products without paying for them.

1. Every loan from a library is not a lost sale. More than 500 thousand loans does not mean 500 thousand lost sales. When borrowing a book is free, people will do it even if they aren't sure if they're going to like a book. The cheapest book by Mr. Deary I Amazon UK is 4.27 when sold as new by amazon (I'll ignored used books for the moment). The threshold for buying a book at that price is much, much higher than borrowing on for free.
2. Letting people have stuff for free does not automatically mean they won't pay for other stuff. Indeed, they may well pay for the very stuff they got for free. See the several experiments to let customers set their own price for music, books, or computer games, or the success Cory Doctorow has letting people download e-books of his novels for free, and then letting people donate one or more hardcopies of the book to schools and libraries. This is not a model that will work well for everyone, but it proves the point: charging people money for each copy is not necessary to make good profit. In fact, studies of "Internet pirates", or people who download music and movies (and books) more or less illegally from the Internet, show that such people tend to spend more money on music and movies (and books!) than the average person. That's not something the publishers want to talk about, or hear. (I am talking here about real research, not the propganda material that lobbying organisations produce.)
3. Why would people buy a book in the first place? They need to know they'll like it, but more importantly, they need to like reading in the first place. See point 1: if books cost any amount of money, the threshold for getting one is high. If you don't have easy, convenient access to books when you're a child, you're unlikely to be a reader of books when you're an adult. Public libraries are the perfect way to give people access to a wide variety of literature. In fact, they're about the most perfect marketing ploy in the history of media: the libraries pay for the books, and they pay each time they loan out a book, and every time a new future customer is brainwashed that much more into getting more stuff in the future. Libraries are not just for children, of course. Adult readers use them as well, and they're just as important as children, as far as the marketing ploy is concerned.

It's a bit sad that bookshops are dying out. Public libraries have, however, nothing to do with it. Online booksellers are usually cheaper, and have vastly larger catalogues, and that's a much more likely culprit. Also, bookshops have no intrinsic right to exist: they're middle-men who make a living by making it more convenient for customers to get what they want, and if they get out-competed, that's business life. Now, I neglected above to consider the possibility of buying used books. The reason for that is that media companies, and their lobbying organisations, would like to make selling of used books, music recordings, movies, and computer games not happen anymore. For example, Sony, and files patents for, stopping people from playing used games on its game consoles, even without Internet access. Mr. Deary seems to be following the MPAA and RIAA checklist on how to talk about those evil people who enjoy something without paying for it every time. It can't be long before he starts asking for a ban on selling used books. Let's hope, however, that reality and facts and sense win this time, and that we can let Mr. Deary live alone in his little horrible hallucination.