By now I am a long time happy user of tmux instead of screen. I started using it somewhere in 2011 and by now I only have one usage of screen left over (direct ttyS0 access for one serial). As usual I have a custom config for it, and if you are interested there is a link for you: tmux.conf. But the more interesting part and why I blog is that I also wrote a little helper around tmux. Ingeniously called tm. And a colleague said I should let other people use it too and place it somewhere accessible. So, err, here it is. Or better, the usage output of it, the actual script has a link somewhere below.

---

tmux helper by Joerg Jaspert <joerg@ganneff.de>
Call as: tm CMD [host]...
CMD is one of
 ls          List running sessions
 s           Open ssh session to host
 ms          Open multi ssh sessions to hosts, synchronizing input
 $anything   Either plain tmux session with name of $anything or
             session according to TMDIR file
TMDIR file:
Each file in $TMDIR defines a tmux session.
Filename is the commandline option $anything
Content is defined as:
  First line: Session name
  Second line: extra tmux options
  Any following line: A hostname to open a shell with in the normal
  ssh syntax. (ie [user@]hostname)
Environment variables recognized by this script:
TMPDIR - Where tmux stores its session information
         DEFAULT: If unset: /tmp
TMSORT - Should ms sort the hostnames, so it always opens the same
         session, no matter in which order hostnames are presented
         DEFAULT: true
TMOPTS - Extra options to give to the tmux call
         Note that this ONLY affects the final tmux call to attach
         to the session, not to the earlier ones creating it
         DEFAULT: -2
TMDIR  - Where are session information files stored
         DEFAULT: $ HOME /.tmux.d
TMWIN  - Where does your tmux starts numbering its windows?
         This script tries to find the information in your config,
         but as it only checks $HOME/.tmux.conf it might fail.
         So if your window numbers start at anything different to 0,
         like mine do at 1, then you can set TMWIN to 1

---

It is mainly useful for using tmux as a ssh-multiplexer and replacement for clusterssh though its happy to do any other tmux session too. As the help text may already have told you, you can use it to list existing tmux sessions (oh, wow!). Honestly, that one is there because tm ls is shorter than tmux ls More interestingly, tm s HOSTNAME opens a tmux session with the first window being a ssh shell to HOSTNAME. Should you later type tm s HOSTNAME again, it will attach to the existing session. But the real kicker for me (and colleagues) actually is the ms subcommand of it. ms being short for multi-session, it is my way of no-longer using clusterssh. Basically you type tm ms HOST1 user@HOST2 and it opens a session with one tmux window consisting of two panes. Showing you host1 and host2 at the same time. And the input to both is synchronized, that is - all your input is send to both of them. You aren t limited to just two hosts, do as many as you wish. Actually, I have no idea if tmux has an upper limit, but I did already have more than 100 hosts open at once. The major limitation there is not tmux - but simply the available screenspace, as all hosts are shown in ONE window! (The one advantage of clusterssh, it uses one xterm per host) This is quite helpful in all kinds of administration work involving clusters - imaging installing a package on 5 hosts of a cluster, you usually want it on all of them - or any kind of situations where one wants to run one command on many systems. As I m already said, I m lazy. And so there is a way to easily define multisessions that one uses regulary, by creating files inside the directory behind $TMDIR, usually $ HOME /.tmux.d. They are simple text files and let you easily define sessions you can then (re)open with a simple tm whateveritsname. And if you want the whole script by now: tm

Review: The Hare with Amber Eyes, by Edmund de Waal

Publisher:	Farrar, Straus and Giroux
Copyright:	2010
ISBN:	0-374-10597-9
Format:	Kindle
Pages:	351

A netsuke ( ) is a miniature sculpture, often made from ivory or boxwood, generally rounded and no larger than a matchbox. They were a part of Japanese clothing starting from the 17th century: traditional garments had no pockets, but people needed a way to carry small objects around with them, so they wore pouches or small boxes attached to a cord. The cord could then be passed under the sash of the garment and be held in place there by a netsuke attached to the end of the cord, since the netsuke was too large to slip under the sash. The hare with amber eyes is one of 264 netsuke. It, and all the rest, belonged to Edmund de Waal's uncle Iggie. They sat in a vitrine in his home in Japan, where de Waal went for a two-year scholarship and was working on a book on japonisme: the passionate and creative Western misunderstanding of Japan. When Iggie later died in 1991, de Waal inherited the collection. This book is his attempt to tell their history. The Hare with Amber Eyes falls into the genre of family memoir or biography, but it's one told from an unusual angle. Edmund de Waal is a potter by profession: an artist who creates physical objects that then pass from his studio out into the world. The physicality of the netsuke their occupation of time and space and their passsage through the world fascinates him, and he tries to express that fascination to the reader and uses it as a focal point for the memoir. He tells, here, the story of his own family, but he does so via the netsuke, following them from their purchase in Paris in 1871 through one branch of the family and then another until they reach Iggie and, finally, him. Along the way, he tries to put them into context. At first, the context is primarily artistic, and the early parts of this book are something of a tour of late 19th century artistic culture in Paris. But, as the story continues, the connection becomes more emotional and the context becomes more complicated. Anyone who is a fan of the TV program Antiques Roadshow or any similar effort to trace the history of specific objects will see the immediate appeal in this presentation. Antiques are fascinating in their mute path through lives and history; attempting to reconstruct the rooms they've sat in and the conversations they've witnessed is something I've always found intriguing. But it's also very difficult to do. Most of our objects, and most owners of objects, simply don't have that sort of documentation. I was wondering, at the start of this book, how de Waal was going to manage to construct his history. The answer is that his family is, or at least was, famous enough to have letters, papers, and history in libraries throughout Europe. That's what makes the parts of the book before Iggie's living memory possible. It's a bit disappointing, if unsurprising; de Waal doesn't have any special technique to find hidden history, and most objects could never have this sort of history written about them. The first owner of the netsuke in his family is Charles Ephrussi, a critic, historian, and art historian who was a patron of the Impressionists and who was important enough in Parisian art culture that he is the inspiration of a character in Marcel Proust's Remembrance of Things Past. This means that quite a lot of this book is about people who moved through social circles and lived in ways that I find quite foreign, which made it hard to invest too much in the first part of the book. The Ephrussi family was originally from Odessa and made a fortune as grain merchants some time prior to the start of the history, and by the time de Waal opens their story, they have vast inherited wealth. There is no sign in the book that Charles needs to work in any way. The first part of the book paints a picture of the culture of art patronage partly via descriptions of Charles's rooms and personal collection, to which the netsuke were added during the surge in interest in Japanese art in Europe produced by the opening of Japan during the Meiji period. It's interesting, but for me in a distant and intellectual way. (People with a deeper interest in Impressionist art or other late 19th century art may have a different reaction.) De Waal writes at length at the start of this book about avoiding sentimentality and trying to take an honest and immersive look at the history of the netsuke. The emotional intensity that he brings to this goal seemed a bit odd, since little in the early part of the book seems to tempt towards sentimentality. But then the Dreyfus Affair enters Charles's story, followed by his gift of the vitrine and the netsuke to a cousin in Austria as a marriage gift, and I started to realize the implications of following a possession of a prominent Jewish family in Europe during the first half of the twentieth century. This is the sort of non-fiction book that you can spoil, so I won't give a detailed description of the rest of the book. But there is a specific event in the history of the netsuke that de Waal tells that would be the story most people would start with, and I now understand the sentimentality that de Waal was trying to avoid. It's a great story, but one of the things I like about this book is that de Waal puts it into a broader context and complicates the story. He includes the long periods of time when the netsuke were mostly ignored, he ties them into the history of western obsession with (and exploitation of) Japanese art, and he shows how the sturdy nature of the netsuke, meant for physical handling rather than display, led to them creating the emotional connection that made them important to the family. It's a book that deals with anti-Semitism with accuracy and specifics without being either lurid or overwhelming. It's a book that traces a line through one of the most overwritten and well-trod areas of history and manages to stay on its own path. It's a book that made me angry, and made me cry, and then made me step back and think about those emotions in a broader context and think about the benefits of leaving those emotions behind. I've never read a book quite like this before. It's slow in places (particularly if one is not very interested in art history), utterly compelling in others, and carefully avoids any simple themes or grand stories. It's entangled with some of the darkest events of recent history, but de Waal stays tightly focused on the family through the lens of the netsuke, which is valuable for keeping the story from drifting into well-worn and less unique material. I was not as completely taken by this book as the review that got me to buy it (it's an excellent review, better than this one). But I'm glad that I read it, and I feel like I now know more about the world. Rating: 7 out of 10

A few weeks ago, a new version of lldpd, a 802.1AB (aka LLDP) implementation for various Unices, has been released. LLDP is an industry standard protocol designed to supplant proprietary Link-Layer protocols such as EDP or CDP. The goal of LLDP is to provide an inter-vendor compatible mechanism to deliver Link-Layer notifications to adjacent network devices. In short, LLDP allows you to know exactly on which port is a server (and reciprocally). To illustrate its use, I have made a xkcd-like strip: If you would like more information about lldpd, please have a look at its new dedicated website. This blog post is an insight of various technical changes that have affected lldpd since its latest major release one year ago. Lots of C stuff ahead!

Version & changelog UPDATED: Guillem Jover told me how he met the same goals for libbsd :

1. Save the version from git into .dist-version and use this file if it exists. This allows one to rebuild ./configure from the published tarball without losing the version. This also handles Thorsten Glaser s critic.
2. Include CHANGELOG in DISTCLEANFILES variable.

Since this is a better solution, I have adopted the appropriate line of codes from libbsd. The two following sections are partly technically outdated.

Automated version In configure.ac, I was previously using a static version number that I had to increase when releasing:

AC_INIT([lldpd], [0.5.7], [bernat@luffy.cx])

Since the information is present in the git tree, this seems a bit redundant (and easy to forget). Taking the version from the git tree is easy:

AC_INIT([lldpd],
        [m4_esyscmd_s([git describe --tags --always --match [0-9]* 2> /dev/null   date +%F])],
        [bernat@luffy.cx])

If the head of the git tree is tagged, you get the exact tag (0.7.1 for example). If it is not, you get the nearest one, the number of commits since it and part of the current hash (0.7.1-29-g2909519 for example). The drawback of this approach is that if you rebuild configure from the released tarball, you don t have the git tree and the version will be a date. Just don t do that.

Automated changelog Generating the changelog from git is a common practice. I had some difficulties to make it right. Here is my attempt (I am using automake):

dist_doc_DATA = README.md NEWS ChangeLog
.PHONY: $(distdir)/ChangeLog
dist-hook: $(distdir)/ChangeLog
$(distdir)/ChangeLog:
        $(AM_V_GEN)if test -d $(top_srcdir)/.git; then \
          prev=$$(git describe --tags --always --match [0-9]* 2> /dev/null) ; \
          for tag in $$(git tag   grep -E '^[0-9]+(\.[0-9]+) 1, $$'   sort -rn); do \
            if [ x"$$prev" = x ]; then prev=$$tag ; fi ; \
            if [ x"$$prev" = x"$$tag" ]; then continue; fi ; \
            echo "$$prev [$$(git log $$prev -1 --pretty=format:'%ai')]:" ; \
            echo "" ; \
            git log --pretty=' - [%h] %s (%an)' $$tag..$$prev ; \
            echo "" ; \
            prev=$$tag ; \
          done > $@ ; \
        else \
          touch $@ ; \
        fi
ChangeLog:
        touch $@

Changelog entries are grouped by version. Since it is a bit verbose, I still maintain a NEWS file with important changes.

Core

C99 I have recently read 21st Century C which has some good bits and also handles the ecosystem around C. I have definitively adopted designated initializers in my coding style. Being a GCC extension since a long time, this is not a major compatibility problem. Without designated initializers:

struct netlink_req req;
struct iovec iov;
struct sockaddr_nl peer;
struct msghdr rtnl_msg;
memset(&req, 0, sizeof(req));
memset(&iov, 0, sizeof(iov));
memset(&peer, 0, sizeof(peer));
memset(&rtnl_msg, 0, sizeof(rtnl_msg));
req.hdr.nlmsg_len = NLMSG_LENGTH(sizeof(struct rtgenmsg));
req.hdr.nlmsg_type = RTM_GETLINK;
req.hdr.nlmsg_flags = NLM_F_REQUEST   NLM_F_DUMP;
req.hdr.nlmsg_seq = 1;
req.hdr.nlmsg_pid = getpid();
req.gen.rtgen_family = AF_PACKET;
iov.iov_base = &req;
iov.iov_len = req.hdr.nlmsg_len;
peer.nl_family = AF_NETLINK;
rtnl_msg.msg_iov = &iov;
rtnl_msg.msg_iovlen = 1;
rtnl_msg.msg_name = &peer;
rtnl_msg.msg_namelen = sizeof(struct sockaddr_nl);

With designated initializers:

struct netlink_req req =  
    .hdr =  
        .nlmsg_len = NLMSG_LENGTH(sizeof(struct rtgenmsg)),
        .nlmsg_type = RTM_GETLINK,
        .nlmsg_flags = NLM_F_REQUEST   NLM_F_DUMP,
        .nlmsg_seq = 1,
        .nlmsg_pid = getpid()  ,
    .gen =   .rtgen_family = AF_PACKET  
 ;
struct iovec iov =  
    .iov_base = &req,
    .iov_len = req.hdr.nlmsg_len
 ;
struct sockaddr_nl peer =   .nl_family = AF_NETLINK  ;
struct msghdr rtnl_msg =  
    .msg_iov = &iov,
    .msg_iovlen = 1,
    .msg_name = &peer,
    .msg_namelen = sizeof(struct sockaddr_nl)
 ;

Logging Logging in lldpd was not extensive. Usually, when receiving a bug report, I asked the reporter to add some additional printf() calls to determine where the problem was. This was clearly suboptimal. Therefore, I have added many log_debug() calls with the ability to filter out some of them. For example, to debug interface discovery, one can run lldpd with lldpd -ddd -D interface. Moreover, I have added colors when logging to a terminal. This may seem pointless but it is now far easier to spot warning messages from debug ones.

libevent In lldpd 0.5.7, I was using my own select()-based event loop. It worked but I didn t want to grow a full-featured event loop inside lldpd. Therefore, I switched to libevent. The minimal required version of libevent is 2.0.5. A convenient way to check the changes in API is to use Upstream Tracker, a website tracking API and ABI changes for various libraries. This version of libevent is not available in many stable distributions. For example, Debian Squeeze or Ubuntu Lucid only have 1.4.13. I am also trying to keep compatibility with very old distributions, like RHEL 2, which does not have a packaged libevent at all. For some users, it may be a burden to compile additional libraries. Therefore, I have included libevent source code in lldpd source tree (as a git submodule) and I am only using it if no suitable system libevent is available. Have a look at m4/libevent.m4 and src/daemon/Makefile.am to see how this is done.

Client

Serialization lldpctl is a client querying lldpd to display discovered neighbors. The communication is done through an Unix socket. Each structure to be serialized over this socket should be described with a string. For example:

#define STRUCT_LLDPD_DOT3_MACPHY "(bbww)"
struct lldpd_dot3_macphy  
        u_int8_t                 autoneg_support;
        u_int8_t                 autoneg_enabled;
        u_int16_t                autoneg_advertised;
        u_int16_t                mau_type;
 ;

I did not want to use stuff like Protocol Buffers because I didn t want to copy the existing structures to other structures before serialization (and the other way after deserialization). However, the serializer in lldpd did not allow to handle reference to other structures, lists or circular references. I have written another one which works by annotating a structure with some macros:

struct lldpd_chassis  
    TAILQ_ENTRY(lldpd_chassis) c_entries;
    u_int16_t        c_index;
    u_int8_t         c_protocol;
    u_int8_t         c_id_subtype;
    char            *c_id;
    int              c_id_len;
    char            *c_name;
    char            *c_descr;
    u_int16_t        c_cap_available;
    u_int16_t        c_cap_enabled;
    u_int16_t        c_ttl;
    TAILQ_HEAD(, lldpd_mgmt) c_mgmt;
 ;
MARSHAL_BEGIN(lldpd_chassis)
MARSHAL_TQE  (lldpd_chassis, c_entries)
MARSHAL_FSTR (lldpd_chassis, c_id, c_id_len)
MARSHAL_STR  (lldpd_chassis, c_name)
MARSHAL_STR  (lldpd_chassis, c_descr)
MARSHAL_SUBTQ(lldpd_chassis, lldpd_mgmt, c_mgmt)
MARSHAL_END;

Only pointers need to be annotated. The remaining of the structure can be serialized with just memcpy()¹. I think there is still room for improvement. It should be possible to add annotations inside the structure and avoid some duplication. Or maybe, using a C parser? Or using the AST output from LLVM?

Library In lldpd 0.5.7, there are two possible entry points to interact with the daemon:

1. Through SNMP support. Only information available in LLDP-MIB are exported. Therefore, implementation-specific values are not available. Moreover, SNMP support is currently read-only.
2. Through lldpctl. Thanks to a contribution from Andreas Hofmeister, the output can be requested to be formatted as an XML document.

Integration of lldpd into a network stack was therefore limited to one of those two channels. As an exemple, you can have a look at how Vyatta made the integration using the second solution. To provide a more robust solution, I have added a shared library, liblldpctl, with a stable and well-defined API. lldpctl is now using it. I have followed those directions²:

• Consistent naming (all exported symbols are prefixed by lldpctl_). No pollution of the global namespace.
• Consistent return codes (on errors, all functions returning pointers are returning NULL, all functions returning integers are returning -1).
• Reentrant and thread-safe. No global variables.
• One well-documented include file.
• Reduce the use of boilerplate code. Don t segfault on NULL, accept integer input as string, provide easy iterators,
• Asynchronous API for input/output. The library delegates reading and writing by calling user-provided functions. Those functions can yield their effects. In this case, the user has to callback the library when data is available for reading or writing. It is therefore possible to integrate the library with any existing event-loop. A thin synchronous layer is provided on top of this API.
• Opaque types with accessor functions.

Accessing bits of information is done through atoms which are opaque containers of type lldpctl_atom_t. From an atom, you can extract some properties as integers, strings, buffers or other atoms. The list of ports is an atom. A port in this list is also an atom. The list of VLAN present on this port is an atom, as well as each VLAN in this list. The VLAN name is a NULL-terminated string living in the scope of an atom. Accessing a property is done by a handful of functions, like lldpctl_atom_get_str(), using a specific key. For example, here is how to display the list of VLAN assuming you have one port as an atom:

vlans = lldpctl_atom_get(port, lldpctl_k_port_vlans);
lldpctl_atom_foreach(vlans, vlan)  
    vid = lldpctl_atom_get_int(vlan,
                               lldpctl_k_vlan_id));
    name = lldpctl_atom_get_str(vlan,
                                lldpctl_k_vlan_name));
    if (vid && name)
        printf("VLAN %d: %s\n", vid, name);
 
lldpctl_atom_dec_ref(vlans);

Internally, an atom is typed and reference counted. The size of the API is greatly limited thanks to this concept. There are currently more than one hundred pieces of information that can be retrieved from lldpd. Ultimately, the library will also enable the full configuration of lldpd. Currently, many aspects can only be configured through command-line flags. The use of the library does not replace lldpctl which will still be available and be the primary client of the library.

CLI Having a configuration file was requested since a long time. I didn t want to include a parser in lldpd: I am trying to keep it small. It was already possible to configure lldpd through lldpctl. Locations, network policies and power policies were the three items that could be configured this way. So, the next step was to enable lldpctl to read a configuration file, parse it and send the result to lldpd. As a bonus, why not provide a full CLI accepting the same statements with inline help and completion?

Parsing & completion Because of completion, it is difficult to use a YACC generated parser. Instead, I define a tree where each node accepts a word. A node is defined with this function:

struct cmd_node *commands_new(
    struct cmd_node *,
    const char *,
    const char *,
    int(*validate)(struct cmd_env*, void *),
    int(*execute)(struct lldpctl_conn_t*, struct writer*,
        struct cmd_env*, void *),
    void *);

A node is defined by:

• its parent,
• an optional accepted static token,
• an help string,
• an optional validation function and
• an optional function to execute if the current token is accepted.

When walking the tree, we maintain an environment which is both a key-value store and a stack of positions in the tree. The validation function can check the environment to see if we are in the right context (we want to accept the keyword foo only once, for example). The execution function can add the current token as a value in the environment but it can also pop the current position in the tree to resume walk from a previous node. As an example, see how nodes for configuration of a coordinate-based location are registered:

/* Our root node */
struct cmd_node *configure_medloc_coord = commands_new(
    configure_medlocation,
    "coordinate", "MED location coordinate configuration",
    NULL, NULL, NULL);
/* The exit node.
   The validate function will check if we have both
   latitude and longitude. */
commands_new(configure_medloc_coord,
    NEWLINE, "Configure MED location coordinates",
    cmd_check_env, cmd_medlocation_coordinate,
    "latitude,longitude");
/* Store latitude. Once stored, we pop two positions
   to go back to the "root" node. The user can only
   enter latitude once. */
commands_new(
    commands_new(
        configure_medloc_coord,
        "latitude", "Specify latitude",
        cmd_check_no_env, NULL, "latitude"),
    NULL, "Latitude as xx.yyyyN or xx.yyyyS",
    NULL, cmd_store_env_value_and_pop2, "latitude");
/* Same thing for longitude */
commands_new(
    commands_new(
        configure_medloc_coord,
        "longitude", "Specify longitude",
        cmd_check_no_env, NULL, "longitude"),
    NULL, "Longitude as xx.yyyyE or xx.yyyyW",
    NULL, cmd_store_env_value_and_pop2, "longitude");

The definition of all commands is still a bit verbose but the system is simple enough yet powerful enough to cover all needed cases.

Readline When faced with a CLI, we usually expect some perks like completion, history handling and help. The most used library to provide such features is the GNU Readline Library. Because this is a GPL library, I have first searched an alternative. There are several of them:

• NetBSD Editline library (libedit).
• Autotool port of the NetBSD Editline library (also libedit).
• Debian version of the Editline library (libeditline).
• linenoise, a small and minimal readline library.
• Many others.

From an API point of view, the first three libraries support the GNU Readline API. They also have a common native API. Moreover, this native API also handles tokenization. Therefore, I have developed the first version of the CLI with this API³. Unfortunately, I noticed later this library is not very common in the Linux world and is not available in RHEL. Since I have used the native API, it was not possible to fallback to the GNU Readline library. So, let s switch! Thanks to the appropriate macro from the Autoconf Archive (with small modifications), the compilation and linking differences between the libraries are taken care of. Because GNU Readline library does not come with a tokenizer, I had to write one myself. The API is also badly documented and it is difficult to know which symbol is available in which version. I have limited myself to:

• readline(), addhistory(),
• rl_insert_text(),
• rl_forced_update_display(),
• rl_bind_key()
• rl_line_buffer and rl_point.

Unfortunately, the various libedit libraries have a noop for rl_bind_key(). Therefore, completion and online help is not available with them. I have noticed that most BSD come with GNU Readline library preinstalled, so it could be considered as a system library. Nonetheless, linking with libedit to avoid licensing issues is possible and help can be obtained by prefixing the command with help.

OS specific support

Netlink on Linux Previously, the list of interfaces was retrieved through getifaddrs(). lldpd is now using directly Netlink on Linux. This is not a big change since the GNU C Library already uses it to implement getifaddrs() and additional information, like VLAN, are still retrieved through ioctl() or sysfs. However, lldpd now gets notified when a change happens and update all interfaces in the next second. Like many other projects, I have written my own Netlink implementation instead of using libnl, a nice collection of libraries providing everything you need to query the kernel through Netlink, including some advanced bits. Why?

1. The latest version of libnl is still young and its availability in major distributions is scarce. It is not available in Debian Squeeze but will be available in Debian Wheezy. Like libevent, I could circumvent this problem by shipping the library with lldpd and use it when there is not system alternative. But
2. libnl is licensed under LGPL 2.1. This makes static linking difficult because the license is quite shaddy about static linking being derivative work or not. It is believed that it is authorized under the same provisions as in LGPL 3 which handles the case explicitely. This has been a problem with many projects. For example, OGRE has added an exception for static linking in version 1.6 and switched to MIT license in version 1.7.

I had a short discussion with Thomas Graf about this issue and he seems willing to add a similar exception. This may take some time, but once this is done, I will happily switch to libnl and retrieve more stuff from Netlink.

BSD support Until version 0.7, lldpd was Linux-only. The rewrite to use Netlink was the occasion to abstract interfaces and to port to other OS. The first port was for Debian GNU/kFreeBSD, then for FreeBSD, OpenBSD and NetBSD. They all share the same source code:

• getifaddrs() to get the list of interfaces,
• bpf(4) to attach to an interface to receive and send packets,
• PF_ROUTE socket to be notified when a change happens.

Each BSD has its own ioctl() to retrieve VLAN, bridging and bonding bits but they are quite similar. The code was usually adapted from ifconfig.c. The BSD ports have the same functionalities than the Linux port, except for NetBSD which lacks support for LLDP-MED inventory since I didn t find a simple way to retrieve DMI related information. They also offer greater security by filtering packets sent. Moreover, OpenBSD allows to lock the filters set on the socket:

/* Install write filter (optional) */
if (ioctl(fd, BIOCSETWF, (caddr_t)&fprog) < 0)  
    rc = errno;
    log_info("privsep", "unable to setup write BPF filter for %s",
        name);
    goto end;
 
/* Lock interface */
if (ioctl(fd, BIOCLOCK, (caddr_t)&enable) < 0)  
    rc = errno;
    log_info("privsep", "unable to lock BPF interface %s",
        name);
    goto end;
 

This is a very nice feature. lldpd is using a privileged process to open the raw socket. The socket is then transmitted to an unprivileged process. Without this feature, the unprivileged process can remove the BPF filters. I have ported the ability to lock a socket filter program to Linux. However, I still have to add a write filter.

OS X support Once FreeBSD was supported, supporting OS X seemed easy. I got sponsored by xcloud.me which provided a virtual Mac server. Making lldpd work with OS X took only two days, including a full hour to guess how to get Apple Xcode without providing a credit card. To help people installing lldpd on OS X, I have also written a lldpd formula for Homebrew which seems to be the most popular package manager for OS X.

Upstart and systemd support Many distributions propose upstart and systemd as a replacement or an alternative for the classic SysV init. Like most daemons, lldpd detaches itself from the terminal and run in the background, by forking twice, once it is ready (for lldpd, this just means we have setup the control socket). While both upstart and systemd can accommodate daemons that behave like this, it is recommended to not fork. How to advertise readiness in this case? With upstart, lldpd will send itself the SIGSTOP signal. upstart will detect this, resume lldpd with SIGCONT and assume it is ready. The code to support upstart is therefore quite simple. Instead of calling daemon(), do this:

const char *upstartjob = getenv("UPSTART_JOB");
if (!(upstartjob && !strcmp(upstartjob, "lldpd")))
    return 0;
log_debug("main", "running with upstart, don't fork but stop");
raise(SIGSTOP);

The job configuration file looks like this:

# lldpd - LLDP daemon
description "LLDP daemon"
start on net-device-up IFACE=lo
stop on runlevel [06]
expect stop
respawn
script
  . /etc/default/lldpd
  exec lldpd $DAEMON_ARGS
end script

systemd provides a socket to achieve the same goal. An application is expected to write READY=1 to the socket when it is ready. With the provided library, this is just a matter of calling sd_notify("READY=1\n"). Since sd_notify() has less than 30 lines of code, I have rewritten it to avoid an external dependency. The appropriate unit file is:

[Unit]
Description=LLDP daemon
Documentation=man:lldpd(8)
[Service]
Type=notify
NotifyAccess=main
EnvironmentFile=-/etc/default/lldpd
ExecStart=/usr/sbin/lldpd $DAEMON_ARGS
Restart=on-failure
[Install]
WantedBy=multi-user.target

OS include files Linux-specific include files were a major pain in previous versions of lldpd. The problems range from missing header files (like linux/if_bonding.h) to the use of kernel-only types. Those headers have a difficult history. They were first shipped with the C library but were rarely synced and almost always outdated. They were then extracted from kernel version with almost no change and lagged behind the kernel version used by the released distribution⁴. Today, the problem is acknowledged and is being solved by both the distributions which extract the headers from the packaged kernel and by kernel developers with a separation of kernel-only headers from user-space API headers. However, we still need to handle legacy. A good case is linux/ethtool.h:

• It can just be absent.
• It can use u8, u16 types which are kernel-only types. To work around this issue, type munging can be setup.
• It can miss some definition, like SPEED_10000. In this case, you either define the missing bits and find yourself with a long copy of the original header interleaved with #ifdef or conditionally use each symbol. The latest solution is a burden by itself but it also hinders some functionalities that can be available in the running kernel.

The easy solution to all this mess is to just include the appropriate kernel headers into the source tree of the project. Thanks to Google ripping them for its Bionic C library, we know that copying kernel headers into a program does not create a derivative work.

---

1. Therefore, the use of u_int16_t and u_int8_t types is a left-over of the previous serializer where the size of all members was important.
2. For more comprehensive guidelines, be sure to check Writing a C library.
3. Tokenization is not the only advantage of libedit native API. The API is also cleaner, does not have a global state and has a better documentation. All the implementations are also BSD licensed.
4. For example, in Debian Sarge, the Linux kernel was a 2.6.8 (2004) while the kernel headers were extracted from some pre-2.6 kernel.

Attachment	Size
Tripe W - Entrevista: no al voto electronico	9.97 MB

Of course, as happens sadly often, the scope creeps.. Additional pain points Zope s test runner runs things that are not tests, but which users want to know about layers . At the moment these are reported as individual tests, but this is problematic in a couple of ways. Firstly, the same test runs on multiple backend runners, so timing and stats get more complex. Secondly, if a layer fails to setup or teardown, tools like testrepository that have watched the stream will think a test failed, and on the next run try to explicitly run that test but that test doesn t really exist, so it won t run [unless an actual test that needs the layer is being run]. Openstack uses python coverage to gather coverage statistics during test runs. Each worker running tests needs to gather and return such statistics. The current subunit protocol has no space to hand this around, without it pretending to be a test [see a pattern here?]. And that has the same negative side effect test runners like testrepository will try to run that test . While testrepository doesn t want to know about coverage itself, it would be nice to be able to pass everything around and have a local hook handle the aggregation of that data. The way TAP is reflected into subunit today is to mangle each tap test into a subunit test , but for full benefits subunit tests have a higher bar they are individually addressable and runnable. So a TAP test script is much more equivalent to a subunit test. A similar concept is landing in Python s unittest soon subtests which will give very lightweight additional assertions within a larger test concept. Many C test runners that emit individual tests as simple assertions have this property as well there may be 5 or 10 executables each with dozens of assertions, but only the executables are individually addressable there is no way to run just one assertion from an executable as a test . It would be nice to avoid the friction that currently exists when dealing with that situation. Minimum requirements to support these Layers can be supported via timestamped stdout output, or fake tests. Neither is compelling, as the former requires special casing in subunit processors to data mine it, and the latter confuses test runners. A way to record something that is structured like a test (has an id the layer, an outcome in progress / ok / failed, and attachment data for showing failure details) but isn t a test would allow the data to flow around without causing confusion in the system. TAP support could change to just show the entire output as progress on one test and then fail or not at the end. This would result in a cognitive mismatch for folk from the TAP world, as TAP runners report each assertion as a test , and this would be hidden from subunit. Having a way to record something that is associated with an actual test, and has a name, status, attachment content for the TAP comments field that would let subunit processors report both the addressable tests (each TAP script) and the individual items, but know that only the overall scripts are runnable. Python subtests could use a unique test for each subtest, but that has the same issue has layers. Python will ensure a top level test errors if a subtest errors, so strictly speaking we probably don t need an associated-with concept, but we do need to be able to say that a test-like thing happened that isn t actually addressable. Coverage information could be about a single test, or even a subtest, or it could be about the entire work undertaken by the test process. I don t think we need a single standardised format for Coverage data (though that might be an excellent project for someone to undertake). It is also possible to overthink things . We have the idea of arbitrary attachments for tests. Perhaps arbitrary attachments outside of test scope would be better than specifying stdout/stderr as specific things. On the other hand stdout and stderr are well known things. Proposal version 2 A packetised length prefixed binary protocol, with each packet containing a small signature, length, routing code, a binary timestamp in UTC, a set of UTF8 tags (active only, no negative tags), a content tag one of (estimate + number, stdin, stdout, stderr, file, test), test-id, runnable, test-status (one of exists/inprogress/xfail/xsuccess/success/fail/skip), an attachment name, mime type, a last-block marker and a block of bytes. The std/stdout/stderr content tags are gone, replaced with file. The names stdin,stdout,stderr can be placed in the attachment name field to signal those well known files, and any other files that the test process wants to hand over can be simply embedded. Processors that don t expect them can just pass them on. Runnable is a boolean, indicating whether this packet is describing a test that can be executed deliberately (vs an individual TAP assertion, Python sub-test etc). This permits describing things like zope layers which are top level test-like things (they start, stop and can error) though they cannot be run.. and it doesn t explicitly model the setup/teardown aspect that they have. Should we do that? Testid is for identifying tests. With the runnable flag to indicate whether a test really is a test, subtests can just be namespaced by the generator reporters can choose whether to be naive and report every test , or whether to use simple string prefix-with-non-character-seperator to infer child elements. Impact on Python API If we change the API to:

class TestInfo(object):
    id = unicode
    status = ('exists', 'inprogress', 'xfail', 'xsuccess', 'success', 'fail', 'error', 'skip')
    runnable = boolean
class StreamingResult(object):
    def startTestRun(self):
        pass
    def stopTestRun(self):
        passs
    def estimate(self, count, route_code=None, timestamp=None):
        pass
    def file(self, name, bytes, eof=False, mime=None, test_info=None, route_code=None, timestamp=None):
        """Inform the result about the contents of an attachment."""
    def status(self, test_info, route_code=None, timestamp=None):
        """Inform the result about a test status with no attached data."""

This would permit the full semantics of a subunit stream to be represented I think, while being a narrow interface that should be easy to implement. Please provide feedback! I ll probably start implementing this soon.

Subunit is seven and a half years old now Conrad Parker and I first sketched it up at a CodeCon camping and coding, a brilliant combination in mid 2005.

revno: 1
committer: Robert Collins <robertc@robertcollins.net>
timestamp: Sat 2005-08-27 15:01:20 +1000
message:  design up a protocol with kfish

It has proved remarkably resilient as a protocol the basic nature hasn t changed at all, even though we ve added tags, timestamps, support for attachments of arbitrary size. However a growing number of irritations have been building up with it. I think it is time to design another iteration of the protocol, one that will retain the positive qualities of the current protocol, while helping it become suitable for the next 7 years. Ideally we can keep compatibility and make it possible for a single stream to be represented in any format. Existing design The existing design is a mostly human readable line orientated protocol that can be sniffed out from the regular output of make or other build systems. Binary attachments are done using HTTP chunking, and the parser has to maintain state about the current test, tags, timing data and test progression [a simple stack of progress counters]. How to arrange subunit output is undefined, how to select tests to run is undefined. This makes writing a parser quite easy, and the tagging and timestamp facility allow multiplexing streams from two or more concurrent test runs into one with good fidelity but also requires that state be buffered until the end of a test, as two tests cannot be executing at once. Dealing with debuggers The initial protocol was intended to support dropping into a debugger just pass each line read through to stdout, and connect stdin to the test process, and voila, you have a working debugger connection. This works, but the current line based parsers make using it tedious the line buffered nature of it makes feedback on what has been typed fiddly, and stdout tends to be buffered, leading to an inability to see print statements and the like. All in-principle fixable, right ? When running two or more test processes, which test process should stdin be connected to? What if two or more drop into a debugger at once? What is being typed to which process is more luck than anything else. We ve added some idioms in testrepository that control test execution by a similar but different format one test per line to list tests, and have runners permit listing and selecting by a list. This works well, but the inconsistency with subunit itself is a little annoying you need two parsers, and two output formats. Good points The current protocol is extremely easy to implement for emitters, and the arbitrary attachments and tagging features have worked extremely well. There is a comprehensive Python parser which maps everything into Python unittest API calls (an extended version of the standard, with good backwards compatibility). Pain points The debugging support was a total failure, and the way the parser depraminates it s toys when a test process corrupts an outcome line is extremely frustrating. (other tests execute but the parser sees them as non-subunit chatter and passes the lines on through stdout). Dealing with concurrency The original design didn t cater for concurrency. There are three concurrency issues the corruption issue (see below for more detail) and multiplexing. Consider two levels of nested concurrency: A supervisor process such as testrepository starts 2 (or more but 2 is sufficient to reason about the issue) subsidiary worker processes (I1 and I2), each of which starts 2 subsidiary processes of their own (W1, W2, W3, W4). Each of the 4 leaf processes is outputting subunit which gets multiplexed in the 2 intermediary processes, and then again in the supervisor. Why would there be two layers? A concrete example is using testrepository to coordinate test runs on multiple machines at once, with each machine running a local testrepository to broker tests amongst the local CPUs. This could be done with 4 separate ssh sessions and no intermediaries, but that only removes a fraction of the issues. What issues? Well, consider some stdout chatter that W1 outputs. That will get passed to I1 and from there to the supervisor and captured. But there is nothing marking the chatter as belonging to W1: there is no way to tell where it came from. If W1 happened to fail, and there was a diagnostic message printed, we ve lost information. Or at best muddled it all up. Secondly, buffering imagine that a test on W1 hangs. I1 will know that W1 is running a test, but has no way to tell the supervisor (and thus the user) that this is the case, without writing to stdout [and causing a *lot* of noise if that happens a lot]. We could have I1 write to stdout only if W1 s test is taking more than 5 seconds or something but this is a workaround for a limitation of the protocol. Adding to the confusion, the clock on W1 and W3 may be very skewed, so timestamps for everything have to be carefully synchronised by the multiplexer. Thirdly, scheduling if W1/W2 are on a faster machine than W3/W4 then a partition of equal-timed tests onto each machine will lead one idle before the other finishes. It would be nice to be able to pass tests to run to the faster machine when it goes idle, rather than having to start a new runner each time. Lastly, what to do when W1 and W2 both wait for user input on stdin (e.g. passphrases, debugger input, $other). Naively connecting stdin to all processes doesn t work well. A GUI supervisor could connect a separate fd to each of I1 and I2, but that doesn t help when it is W1 and W2 reading from stdin. So additional requirement over baseline subunit:

1. make it possible for stdout and stderr output to be captured from W1 and routed through I1 to the supervisor without losing its origin. It might be chatter from a noisy test, or it might be build output. Either way, the user probably will benefit if we can capture it and show it to them later when they review the test run. The supervisor should probably show it immediately as well the protocol doesn t need to care about that, just make it possible.
2. make it possible to pass information about tests that have not completed through one subunit stream while they are still incomplete.
3. make it possible (but optional) to pass tests to run to a running process that accepts subunit.
4. make it possible to route stdin to a specific currently process like W1. This and point 3 suggest that we need a bidirectional protocol rather than the solely unidirectional protocol we have today. I don t know of a reliable portable way to tell when some process is seeking such input, so that will be up to the user I think. (e.g. printing (pdb) to stdout might be a sufficiently good indicator.)

Dealing with corruption Consider the following subunit fragment:

test: foo
starting serversuccess:foo

This is a classic example of corruption: the test foo started a server and helpfully wrote to stdout explaining that it did that, but missed the newline. As a result the success message for the test wasn t printed on a line of its own, and the subunit parser will believe that foo never completed. Every subsequent test is then ignored. This is usually easy to identify and fix, but its a head-scratcher when it happens. Another way it can happen is when a build tool like make runs tests in parallel, and they output subunit onto the same stdout file handle. A third way is when a build tool like make runs two separate test scripts serially, and the first one starts a test but errors hard and doesn t finish it. That looks like:

test: foo
test: bar
success: bar

One way that this sort of corruption can be mitigated is to put subunit on it s own file descriptor, but this has several caveats: it is harder to tunnel through things like ssh and it doesn t solve the failing test script case. I think it is unreasonable to require a protocol where arbitrary interleaving of bytes between different test runner streams will work so the make -j2 case can be ignored at the wire level though we should create a simple way to safely mux the output from such tests when the execute. The root of the issue is that a dropped update leaves bad state in the parser and it never recovers. So some way to recover, or less state to carry in the parser, would neatly solve things. I favour reducing parser state as that should shift stateful complexity onto end nodes / complex processors, rather than being carried by every node in the transmission path. Dependencies Various suggestions have been made JSON, Protobufs, etc A key design goal of the first subunit was a low barrier to entry. We keep that by being backward compatible, but being easy to work with for the new revision is also a worthy goal. High level proposal A packetised length prefixed binary protocol, with each packet containing a small signature, length, routing code, a binary timestamp in UTC, a set of UTF8 tags (active only, no negative tags), a content tag one of (estimate + number, stdin, stdout, stderr, test- + test id), test status (one of exists/inprogress/xfail/xsuccess/success/fail/skip), an attachment name, mime type, a last-block marker and a block of bytes. The content tags:

• estimate the stream is reporting how many tests are expected to run. It affects everything with the same routing code only, and replaces (doesn t adjust) any current estimate for that routing code. A estimate packet of 0 can be used to say that a routing target has shutdown and cannot run more tests. Routing codes can be used by a subunit aware runner to separate out separate threads in a single process, or even just separate TestSuite objects within a single test run (though doing so means that they will need to process subunit and strip packets on stdin. This supercedes the stack of progress indicators that current subunit has. estimates cannot have test status or attachments.
• stdin/stdout/stderr: a packet of data for one of these streams. The routing code identifies the test process that the data came from/should go to in the tree of test workers. These packets cannot have test status but should have a non-empty attachment block.
• test- + testid: a packet of data for a single test. test status may be included, as may attachment name, mime type, last-block and binary data.

Test status values are pretty ordinary. Exists is used to indicate a test that can be run when listing tests, and inprogress is used to report a test that has started but not necessarily completed. Attachment names must be unique per routing code + testid. So how does this line up? Interleaving and recovery We could dispense with interleaving and say the streams are wholly binary, or we can say that packets can start either after a \n or directly after another packet. If we say that binary-only is the approach to take, it would be possible to write a filter that would apply the newline heuristic (or even look for packet headers at every byte offset. I think mandating adjacent to a packet or after \n is a small concession to make and will avoid tools like testrepository forcing users to always configure a heuristic filter. Non-subunit content can be wrapped in subunit for forwarding (the I1 in W1->I1->Supervisor chain would do the wrapping). This won t eliminate corruption but it will localise it and permit the stream to recover: the test that was corrupted will show up as incomplete, or with incomplete attachment data. listing Test listing would emit many small non-timestamped packets. It may be useful to have a wrapper packet for bulk amounts of fine grained data like listing is, or for multiplexers with many input streams that will often have multiple data packets available to write at once. Selecting tests to run Same as for listing while passing regexes down to the test runner to select groups of tests is a valid use case, thats not something subunit needs to worry about : if the selection is not the result of the supervisor selecting by test id, then it is known at the start of the test run and can just be a command line parameter to the backend : subunit is relevant for passing instructions to a runner mid-execution. Because the supervisor cannot just hand out some tests and wait for the thing it ran to report that it can accept incremental tests on stdin, supervisor processes will need to be informed about that out of band. Debugging Debugging is straight forward . The parser can read the first 4 or so bytes of a packet one at a time to determine if it is a packet or a line of stdout/stderr, and then either read to end of line, or the binary length of the packet. So, we combine a few things; non-subunit output should be wrapped and presented to the user. Subunit that is being multiplexed and forwarded should prepend a routing code to the packet (e.g. I1 would append 1 or 2 to select which of W1/W2 the content came from, and then forward the packet. S would append 1 or 2 to indicate I1/I2 the routing code is a path through the tree of forwarding processes). The UI the user is using needs to supply some means to switch where stdin is attached. And stdin input should be routed via stdin packets. When there is no routing code left, the packet should be entirely unwrapped and presented as raw bytes to the process in question. Multiplexing Very straight forward unwrap the outer layer of the packet, add or adjust the routing code, serialise a header + adjusted length + rest of packet as-is. No buffering is needed, so the supervisor can show in-progress tests (and how long they have been running for). Parsing / representation in Python or other languages The parser should be very simple to write. Once parsed, this will be fundamentally different to the existing Python TestCase->TestResult API that is in used today. However it should be easy to write two adapters: old-style <-> this new-style. old-style -> new-style is useful for running existing tests suites and generating subunit, because thats way the subunit generation is transparent. new-style->old-style is useful for using existing test reporting facilities (such as junitxml or html TestResult objects) with subunit streams. Importantly though, a new TestResult style that supports the features of this protocol would enable some useful features for regular Python test suites:

• Concurrent tests (e.g. in multiprocessing) wouldn t need multiplexers and special adapters a regular single testresult with a simple mutex around it would be able to handle concurrent execution of tests, and show hung tests etc.
• The routing of input to a particular debugger instance also applies to a simple python process running tests via multiprocessing, so the routing feature would help there.
• The listing facility and incrementally running tests would be useful too I think we could go to running tests concurrently with test collection happening, but this would apply to other parts of unittest than just the TestResult

The API might be something like:

class StreamingResult(object):
    def startTestRun(self):
        pass
    def stopTestRun(self):
        pass
    def estimate(self, count, route_code=None):
        pass
    def stdin(self, bytes, route_code=None):
        pass
    def stdout(self, bytes, route_code=None):
        pass
    def test(self, test_id, status, attachment_name=None, attachment_mime=None, attachment_eof=None, attachment_bytes=None):
        pass

This would support just-in-time debugging by wiring up pdb to the stdin/stdout handlers of the result object, rather than actual stdin/stdout of the process a simple matter once written. Alternatively, the test runner could replace sys.stdin/stdout etc with thunk file-like objects, which might be a good idea anyway to capture spurious output happening during a test run. That would permit pdb to Just Work (even if the test process is internally running concurrent tests.. until it has two pdb objects running concurrently Generation new streams Should be very easy in anything except shell. For shell, we can have a command line tool that when invoked outputs a subunit stream for one instruction. E.g. test foo completed + some attachments or test foo starting .

At FOSDEM I was talking even two times about Debian Med. The talk in the morning was attached to the FOSS for scientists track and was only a short overview about the use scientists can make from Debian Med. As usual the slides are available from my web page.
The main talk was in the evening in the Cross Distro Devroom and it was even video recorded.
It was fun to be at FOSDEM for the first time and meet a lot of Debian people and discuss new ideas with them.

As I did not find a nice explanation of Debian version numbers to point people at, here some random collection of information about: All our packages have a version. For the package managers to know which to replace with which, those versions needs an ordering. As version orderings are like opinions (everyone has one), none of them would match any single one chosen for our tools to implement. So maintainers of Debian packages sometimes have to translate those versions into something the Debian tools understand. But first let's start with some basics: A Debian version string is of the form: [Epoch:]Upstream-Version[-Debian-Revision] To make this form unique, the Upstream-Version may not contain an colon if there is no epoch and not contain a minus, if there is no Debian-Revision. The Epoch must be an integer (so no colons allowed). And the Debian-Revision may not contain a minus sign (so the Debian-Revision is everything right of the right-most minus sign, or empty if there is no such sign). Two versions are compared by comparing all three parts. If the epochs differ, the biggest epoch wins. With same epochs, the biggest upstream version wins. With same epochs and same upstream versions, the biggest revision wins. Comparing first the upstream version and then the revision is the only sensible thing to do, but it can have counter-intuitive effects if you try to compare versions with minus signs as Debian versions:

$ dpkg --compare-versions '1-2' '<<' '1-1-1' && echo true   echo false
true
$ dpkg --compare-versions '1-2-1' '<<' '1-1-1-1' && echo true   echo false
false

To compare two version parts (Upstream-Version or Debian-Revision), the string is split into pairs of digits and non digits. Consecutive digits are treated as a number and compared numerrically. Non-digit parts are compared just like ASCII strings with the exception that letters are sorted before non-letters and the tilde is treated specially (see below). So 3pl12 and 3pl3s are slit into (3, 'pl', 12, '') and (3, 'pl', 3, 's') and the first is the larger version. Comparing digits as characters makes not sense at least at the beginning of the string (otherweise version 10.1 would be smaller than 9.3). For digits later in the string there are two different version schemes competing here: There is GNU style 0.9.0 followed by 0.10.0 and decimal fractions like 0.11 < 0.9. Here a version comparison algorithm has to choose one and the one chosen by dpkg is both the one supporting the GNU numbering and also the one easier supporting the other scheme: Imagine one software going 0.8 0.9 0.10 and one going 1.1 1.15 1.2. With out versioning scheme the first just works, while the second has to be translated into 1.1 1.15 1.20 to still be monotonic. The other way around, we would have to translate the first form to 0.08 0.09 0.10, or better 0.008 0.009 0.010 as we do not know how big those numbers will be, i.e. one would have to know beforehand where the numbers will end up, while adding zeros as needed for our scheme can be done with only knowing the previous numbers. Another decision to be taken is how to treat non-numbers. The way dpkg did this was assuming adding stuff to the end increases numbers. This has the advantage to not needing to special case dots, as say 0.9.6.9 will be bigger than 0.9.6 naturally. I think back then this decision was also easier as usually anything attached was making the version bigger and one often saw versions like 3.3.bl.3 to denote some patches done atop of 3.3 in the 3th revision. But this scheme has the disadvantage that version schemes like 1.0rc1 1.0rc2 1.0 do not map naturally. The classic way to work arround this is to translate that into 1.0rc1 1.0rc2 1.0.0 which works because the dot is a non-letter (it also works with 1.0-rc1 and 1.0+rc1 as the dot has a bigger ASCII number than minus or plus). The new way is the specially treated tilde character. This character was added some years ago to sort before anything else, including an empty string. This means that 1.0~rc1 is less than 1.0:

dpkg --compare-versions '1.0~rc1-1' '<<' '1.0-1' && echo true   echo false

This scheme is especially useful if you want to create a package sorting before a package already there, as you for example do want with backports (as a user having a backport installed upgrading to the next distribution should get the backport replaced with the actual package). That's why backport usually having versions like 1.0-2~bpo60+1. Here 1.0-2 is the version of the un-backported version; bpo60 is a note that this is backported to Debian 6 (AKA squeeze) and the +1 is the number of the backport in case there are multiple tries necessary. (Note the use of the plus sign as the minus sign is not allowed in revisions and would make the part before a part of the upstream version). Now, when to use which technique?

• Usually try to stick to the upstream version if it is sane. (With the exception of data based versions, as experience shows upstreams later change their mind, so a version of 20130210 is best translated to 0.20130210 in order to have a future 1.0 sort higher).
• If there are some suffixes in the version that denote a "before", prefix them with a tilde.
• Compare a version to your last version and put zeros or .0 at the end (or the ends of parts) where necessary to increase them.

Some common examples:

• You want to package 2.0 but you need to repack the .orig.tar because it contains a file with some non-free license (for example no license at all). Then you can call it 2.0~dfsg-1. If the file is later relicensed, you can do a 2.0-2 with the original file which sorts higher.
• You need to repack 2.0 but it was already in the archive. then you can do 2.0+dfsg-1 as this is higher. (using the plus has the advantage that you can later use the 2.0.0 trick if the file is relicensed, while the older 2.0.dfsg-1 made this impossible). (One can also use -dfsg-1 here, but + might be easier to parse for humans).
• You package a git snaphost of a version 2.0 not yet released, then you can use 2.0~gitXYZ-1
• You package a git snapshot of upstream doing some works on top of 1.9, then you can use 1.9+gitXYZ-1

In a previous entry I wrote about the technology of additive manufacture and the use in printing three dimensional objects.

It is now almost two years since I built my own 3D printer and I keep getting asked by colleagues and friends about the technology and often what printer to buy.

I will answer the purchase question first and then describe my experiences which have lead to the conclusion in another post. This may seem a bit backwards but the explanation is long and is not necessary if you are happy to learn from my mistakes.

Of all the options available right now, and there are many, I would choose a Mendal90 kit from Chris Palmer. The complete kit including everything to build the machine is 499 plus shipping. If I could afford it this is what I would buy myself to replace my current machine.

This is a Fused deposition modelling (FDM) printer similar to my Prusa Reprap but better engineered to produce repeatable results without the numerous issues of the other models. In Europe I would also recommend faberdashery as a materials source as their product is first rate every time.

Yes the kit requires some assembly but the commonly available commercial printers either cost many times more to deliver equivalent results, use an SLS or other print strategy requiring very expensive consumables or are from a company with dubious track record with the community.

If forced to recommend one, the 3DTouch from Bits From Bytes is not awful, but really do not be afraid of the kit, you will learn more about how it all fits together and save lots of money for your materials.

One thing anyone buying a 3D printer right now should understand is that this technology is nowhere near as polished as the 2D equivalent. With the exception of the SLS systems like shapeways and the like use (and have price tags to match) The output will have clear "layering" and some objects simply cannot be created using the FDM process.

I guess what I am saying is do not expect a thousand pound machine to produce output that looks like that of a hundred thousand pound printer. To be clear you will not be printing complex moving machines on an FDM process more simple things that need assembly.

Having said that I have some pretty good results my favourite has to be the working recorder though, I might have said the whistles except my sons have them and they are way too loud.

You will spend a lot of time designing your things in 3D CAD packages and fair warning they all SUCK and I mean really, really badly. Add to that all the rest of the tools in the workflow are also iffy and I do wonder how anyone every gets anything printed.

My (open source) workflow is:

• 3D CAD with OpenSCAD
• STL file
• meshlab to clean up the STL
• slic3r to create a toolpath
• Gcode file
• printrun to run the gcode on the printer.

Which is probably a case of "least bad" tool selection though I warn you now that OpenSCAD is effectively a bad editor (I wish I could use emacs) for a 3D solid macro language with visualisation attached and definitely not a graphical tool.

After 27 hours of travel, I'm finally back from my first Linux.conf.au. This was a great conference! Some of my favorite highlights:

• Out for dinner the first night, my whole table started spontaneously talking about Haskell, including details of IO management frameworks like conduit and pipes, and a new one that's waiting in the wings. That just doesn't happen in the real world. A lot of us continued to do Haskell stuff in the hallway track, although for some reason there were no Haskell talks on the official schedule. Maybe it's time for a Haskell miniconf at LCA?
• Meeting Josh, Jamie, Sarah, Brendan, Craig, and others I've worked with online but never encountered IRL. Also reconnecting with old friends (some I'd not seen in 13 years) and finding new ones.
• The speaker's dinner in a revolving restaurant overlooking Canberra. Leave it to a restaurant full of geeks to invent an asychronous communications medium in such a setting. (Notes attached to windows to be read and answered by tables as they rotated by.)
• Meeting quite a lot of git-annex users, Kickstarter backers, and people interested in using git-annex. Thanks to everyone who came up to me for a chat.
• The evening pickup board game sessions. Especially my wiping out three other Tsuro players at once by forcing them all to end on the same square. ;) These were where I felt the most at home in Australia.
• Robert Llewellyn dropping in and mingling with fans of Red Dwarf and Scrapheap Challenge. One of the very few actors I could possibly fanboy on, and LCA not only somehow got him, but constructed an atmosphere that allowed for this photo of us.

---

My git-annex talk went ok, despite technical problems with the video output, which the video team did a great job of working around. The first demo went well, and the Q&A was excellent. The demo of the git-annex assistant webapp was marred by, apparently, a bad build of the webapp -- which I ironically used rather than my usual development build because I assumed it'd be less likely to fail. At least I may have a way to reproduce that hang somewhat reliably now, so I can get on with debugging it. I will be redoing a demo of the webapp as a screencast.

---

Here are some of the best talks I attended. (I have quite a lot more queued up to watch still, and had to cut several due to space.) Click titles for videos, or browse all the videos.

• Git for Ages 4 and Up
  Schwern has found a new, excellent way to explain git. I felt rather bad for using up a seat, especially once people were kicked out when the room was filled over capacity. But I enjoyed every minute of it. (Also has the best speaker intro ever. "Schwern once typed git pull --hard and pulled GitHub's sever room across the street.")

  BTW, I must confess: I left the red apple on teacher's desk.

• Radia Perlman's keynote
  Network protocol design and poetry from one of the quiet heros of our field. I knew Spanning Tree Protocol was used in ethernet, but it just works, so like many I never paid much attention to it. This talk felt like the best lectures, where you're learning from a master on multiple levels at once. Well done work often becomes an unremarked part of the landscape, which I sometimes find unrewarding, so it was great to have Radia give some perspective on what that's like over the course of decades.
• Lightning Talks
  A 90 second time limit really helps. Too many conferences have 5 minute talks, which is less exciting. If you still find them boring, skip forward to 13:50, where pjf does two talks in 90 seconds! (If a 20 second talk on depression is too .. manic, there's an encore at the end.)
• The IPocalypse 20 months later
  A reality check, with real data. Very important stuff here. We need to work to avoid this worst case scenario, and we also need to design around it.
• REPENT!!! FOR THE END OF THE UNIX EPOCH IS NIGH!!!
  Wildest talk beginning I've seen since RMS put the hard drive halo on his head. And to an important point: Any programs that deal with dates 25 years in the future already need to be fixed today to deal with the epoch rollover. This got me digging around in Haskell date libraries, to make sure they're ok.
• Building Persona: Federated and Privacy Sensitive Identity for the Web
  This talk and some previous conversation with Francois have convinced me that Persona (AKA Browserid) has a design that can succeed. I will be adding Persona login support to ikiwiki.
• Beyond Alt Text: What Every Project Should Know About Accessibility
  I missed the first half due to giving my talk, but the second half was full of rather a lot of excellent information, some of which I'd only guessed at before.
• Git: Not Just for Source Code Anymore
  Good overview of the new ways to use git. Also kept giving examples from my body of work, which is some nice ego stroking, thanks Josh. ;-)

The previous article was introducing basic streaming principles based on Icecast. The issue with this, of course, is lag and overhead of HTTP-based connexions. In this article we introduce RTP-based streaming system, (unfortunately) based on Pulseaudio and multicast. (Update: there was a rant here about Pulseaudio (PA) that I have moved out of this post because it's not what I was aiming to talk about. Those wanting to answer that troll are welcome to join the flamewar here.)

A word on multicast I took the opportunity of trying out Multicast (it's actually IP multicast, to be more precise), since I wanted to have only minimal configuration to perform on the client side. Plus, since the RTP frames are raw 44khz 16 bit audio, the traffic is actually quite heavy, 1.4mbps, to be more precise (punch in 44kHz * 16bit/Hz * 2 in the excellent Qalculate to see why), so I didn't want bandwidth to explode as more listeners joined in. As a comparison, also note that the Icecast Vorbis stream is around 64 or 128kbps depending on quality. Multicast allows packets to be sent only once for multiple listeners on the same network. It's actually quite cool, but doesn't work very well over the wider internet, for reasons that are still unclear to me. I presume that the fact there is virtual monopoly over traditional broadcasting systems (e.g. television and radio) means that multicast hasn't really kicked in yet... I hope this will make sense over the Montreal mesh, although the quality would probably need to be dialed down, or the audio streamed encoded instead of be sent as raw samples.

So why PA? VLC/pipe failures That aside, it seems that to stream through RTP, I have no choice. I have tried to use VLC as a pipe to stream data out of MPD as RTP, but somehow this config snippet just failed:

audio_output  
   type            "pipe"
   name            "RTP"
   command         "cvlc -q - --sout '#rtp dst=239.72.72.144,port=5004 '"
   encoder         "vorbis"                # optional, vorbis or lame
   format          "44100:16:2"
 

I don't know why - the multicast port wouldn't open properly and MPD would even hang when restarting. It seems the pipe plugin is somewhat broken. Note that the vlc command above actually works fine to stream audio content out through multicast, which is a good way to test if multicast works in the first place...

Shoving PA down MPD's throat Instead, I had to go to the clunky PA-way (also know as "my way or the highway"). The first step was to configure the default.pa to enable RTP streaming. I copied over the default /etc/pulse/default.pa file into ~mpd/.pulse/default.pa and uncommented the following lines:

# just to get the PA announced
load-module module-zeroconf-publish
# we need a null sink, no clue why, but without this glue, it fails
load-module module-null-sink sink_name=rtp format=s16be channels=2 rate=44100 sink_properties="device.description='RTP Multicast Sink'"
# port is specified otherwise it is random (?!)
# loop=1 is to broadcast locally too
# other useful setting: destination (defaults to 224.0.0.56)
load-module module-rtp-send source=rtp.monitor port=5004 loop=1

See the rtp-send module documentation for more information here. Then we need MPD to connect to that sink:

audio_output  
    type            "pulse"
    name            "Pulsaudio RTP"
    sink            "rtp"
 

Also, now that we got into the PA kitchen sink, we are kind of forced to use it for the local sound card output too, so we'll add this as a sink to:

audio_output  
    type            "pulse"
    name            "Pulseaudio sound card"
    sink            "alsa_output.pci-0000_00_1b.0.analog-stereo"
 

(Take a moment to admire the nice sink name. Yours may be different, have fun finding it in the output of pacmd list-sinks, but that works only once PA is started!) So now you need to restart MPD for those changes to take effect. Note that if the mpd user has already started PA, you're in luck: it won't restart it by itself, and you need to:

service mpd stop
killall pulseaudio
service mpd start

... yep. Basically because pulseaudio just detaches completely from the parent process when spawned automatically. Lots of fun.

Testing and listening Of course, you can listen to the stream using... Pulseaudio. I assume this would magically show up on the network thanks to avahi, if you run Pulseaudio on (say) your laptop. Since I still want to stay away from it, I would rather use a straight commandline tool to listen to the stream. According to this page, there aren't that many clients that could listen to such a funky stream. I chose VLC, since its commandline options actually make sense. Here's the magic mantra to repeat every night before you go to bed:

cvlc rtp://@224.0.0.56:5004

Yes, that's all. Drop the c if you want a GUI. With mplayer, it's more complicated - you need to specify a bitmask using a hexadecimal string (fun stuff):

mplayer -demuxer rawaudio -rawaudio format=0x20776172 rtp://224.0.0.56:5004

Yes, good old 0x20776172...

Results! We want results! All this work reduced the lag from 15 seconds to around 1 second. It's still present, and confusing to listen to, but maybe more acceptable. It seems most of the lag is in the vlc client buffering, when I enable debugging, I see:

[0x963c7bc] main input debug: Stream buffering done (1001 ms in 976 ms)

I was able to reduce that delay significantly by using the --rtp-caching option of vlc, as such:

cvlc --rtp-caching=48  rtp://@224.0.0.56:5004

The value is in miliseconds. Anything below 48ms introduces audible artifacts (probably buffer underruns). With that setting, there's still a noticeable delay, especially with isolated sounds that are naturally more noticeable. With more harmonious music, it sounds more like an echo. I have also tried to fiddle with similar settings with mplayer and simply failed to get any results. --no-cache simply drops samples like crazy. --cache 64 is the lowest I could push it down to, and it still has noticeable delay, which I couldn't measure, but I would assume would be around 400ms at least. Also note that contrarily to the previous setup I had, based on Icecast, VLC will automatically resume streaming from the RTP stream as it is stopped or started, which is also very nice - as it allows me to disable remote RTP streams easily and harmlessly.

Troubled areas Speaking of insane nonsense, here are a few pitfalls to avoid. Take care to disable the ALSA output, if it's not yet commented out, because MPD will freak out with the following error message:

Feb 03 20:25 : output: "Sound card" [alsa] failed to play: Resource temporarily unavailable

I have not been able to find a way to make PA let go of the audio sound card. I would have prefered MPD to open ALSA directly if PA wasn't needed (I need it only for RTP streaming), but it seems that PA assumes you'll want to play stuff locally. I have tried commenting out the following in default.pa:

load-module module-udev-detect

Seems like it's not enough. Also note that PA is designed with "dumb users" in mind (read: you need a GUI to operate it), so in the context of MPD, it's really annoying to debug things unless you can actually fire up a X11 session as the MPD user (which strikes me as a crazy idea) to configure PA properly with paprefs and all the graphical gizmos. I just gave up and went with PA all the way, and just added the RTP sink through the commandlines. Other guides can show you screenshots to clikety your way through inane graphical interfaces to do the above, if you are into that kind of stuff. With PA, you will want to get familiar with the pacmd and pactl commands. Why there are two different commands to basically do the same thing is completely beyond me.

Reference and more reading I got a lot of information in the Pulseaudio MPD wiki page and the Pulseaudio user documentation. The VLC manual about RTP streaming (server) (client) was also useful for debugging. The Pulseaudio Network documentation has a set of recipes for using pulseaudio for everything. This Ask Ubuntu question explains how to clikety your way through paprefs to essentially do the above.

Thanks to all the people at LCA2013 in Canberra; it was a blast! So good to see old friends and chat freely on what s hot and happening. Radia (known for STP, TRILL), Sir Tim (the web) and old friend Bdale (Debian, SPI, Freedom Box) were inspiring. As was Robert Llewellyn (Kryten, Red Dwarf), who was a complete pleasure he wandered back and talked for a while with the volunteer video crew. Hats off to Pia for organising the TBL tour, to Mary Gardner for being awarded the Rusty Wrench, and to the team from PLUG (Euan, Jason, Leon, Luke) who stepped up to help with the video team and to Paul who graciously accepted the help. Next up LCA2014 Perth! Y all come back now.. it s been a decade.

It's not too obvious how to test email receive handlers in Appengine. The important observation is that the handlers take HTTP POSTs with multipart/form-data encoded data. In python you can build an email to be handled with the following code:

from email.message import Message 

def test_email(self):
  body = Message()
  body.add_header('to', 'test-unknown@other-app.com')
  body.add_header('from', 'test@app.com')
  body.add_header('Content-Type', 'multipart/alternative', boundary=self.boundary)
  text = Message()
  text['content-type'] = 'text/plain'
  text.set_payload('I am I! Don Quixote!  The man of La Mancha!')
  body.attach(text)

  post(payload=body.as_string())

For some time I ve been wondering how the wire size for power supplies limits the power. So I ve done some quick calculations to determine if it s a problem. The first type that is of interest are the Inverters that are used to convert 12VDC to 240VAC (mains power) to allow electric devices to be operated in a car. I ve seen some reports from dissatisfied user about Inverters not supplying as much power as expected and I ve had problems with my 150W Inverter not always supplying my Thinkpad (which definitely doesn t draw 150W). The second type is phone chargers as charging a phone in a reasonable amount of time is always a problem. Inverter Rating Fine Print vs Laptop PSU My Thinkpad Power Supply claims Efficiency Level IV which according to the US EPA document describing the efficiency marking protocol for external power supplies [1] means that it is at least 85% efficient when supplying 50W+. The peak output of the PSU is 4.5A at 20V which is 90W peak output, 90/0.85 == 106W power drawn. One would hope that wouldn t be a problem from a 150W PSU. But the fine print on the PSU says that it can provide 110W continuously and 150W for 10 minutes. So according to my calculations I m within 4W of overloading the PSU if my Thinkpad uses full power. It also says that it is designed for 13.8V input. I have no idea how the performance of the Inverter changes as the supply Voltage changes between the 12.6V that a 6 cell lead-acid battery is designed to provide and the 13.8V charge from the car alternator. But I have had occasions when my Inverter stopped working correctly presumably due to being unable to supply as much current as my Thinkpad draws. As an aside I measured the Voltage in my car (with the engine off) at 12.85V from the cigarette lighter socket and 13.02V directly from the battery. I wonder if there is some sort of overload protection on the cigarette lighter which has a side effect of reducing the Voltage. Resistance in wires reduces the Voltage, but all Voltage meters are designed to have a high resistance to prevent that from being an issue. If anyone has an explanation for the 0.17Volt drop then please write a comment! Can a Car Provide 130W from the Cigarette Lighter socket? If the Inverter is also 85% efficient (and it might be less as it has no indication of efficiency on the box) then when supplying 110W it would draw 110/0.85 == 129.4W (I ll round it up to 130W). The power in Watts is equal to the Voltage multiplied by the current in Amps (W=V*I). Therefore I=W/V so if the car battery was at 12.85V then 130W/12.85V == 10.12A will flow. The current that goes through a circuit is equal to the Voltage divided by the resistance (see the Wikipedia page on Ohm s law for more information). This also means that the resistance equals the Voltage divided by the current. 12.85V/10.12A == 1.27 Ohms. Note that this is the resistance of the entire circuit, all the wires going to the battery, the circuitry inside the Inverter, and the internal resistance of the battery. The Inverter s cable is 1M long (2 meters of wire) and each wire is about 3.5mm in diameter including the insulation which means that the copper wire is probably equivalent to a single core conductor that is about 1mm in diameter. According to one of the online guides to resistance [2] wire that is 1.02mm in diameter will have a resistance of 0.02 Ohms per meter which gives a resistance of 0.04 Ohms. 0.04 Ohms is 3% of the total resistance of the circuit which doesn t seem like it will be a real problem. In practice I ve noticed that the connector gets extremely hot when it s in use while the cable doesn t get warm enough to notice. I suspect that the quality of the connector limits the power that is available but I don t have an easy way of measuring this. Inverters that are rated at 300W are designed to attach directly to the battery. An Inverter that is rated at 300W would draw 300W/0.85 == 352W from the battery. That needs 352W/13.02V == 27.04A and therefore a circuit resistance of 13.02V/27.04A == 0.48 Ohms total resistance. I wonder whether dirt on the battery terminals would give a significant portion of that. Phone Charging I ve also been wondering about why mobile phones take so long to charge, and now I ve finally done the calculations. The latest standard for mobile phones is to use USB for charging. The Wikipedia page about USB says that the standard is for USB 2.0 to supply up to 500mA at 5V +-5%. That means 0.5A*5V == 2.5W +- 5%. If we assume that the internal power supply in a phone is also 85% efficient then that means 2.5*0.85 == 2.125W going to the battery. My Samsung Galaxy S3 has a battery which is rated at 7.98Wh. According to the Wikipedia page about Lithium Ion batteries the charge/discharge efficiency is 80% to 90% I ll assume that it s 85% for further calculations. If the battery in the phone is 85% efficient and the phone is doing nothing but charging then the charge time for a regular USB port would be 7.98Wh/0.85/2.125W == 4.42 hours (4 hours 25 minutes) of charge time. That probably means something closer to 5 hours to totally charge the phone while it s running. There are dedicated charging ports for USB which can supply up to 1.5A. The 3rd party charger which came with my phone was rated at 1A and would hopefully be capable of completely charging the phone in less than 3 hours (but in practice isn t). It s interesting to note that MacBooks expose the amount of current drawn from a USB port with a GUI, so it should be possible to measure a phone charge rate by connecting it to a MacBook (which is cheaper than cutting up a phone cable). My old Samsung Galaxy S has a battery which is rated at 5.55Wh, by the same calculations it would take slightly more than 3 hours to charge on a standard USB port or 1.5 hours on my newest USB charger. In practice it has never got anywhere close to that, I presume that the phone is designed to draw less than 500mA. Phone Cable Resistance The charger that came with my Galaxy S has a cable that is about 1.75M long, the cable is flat and measures just over 1mm thick and about 2mm wide. Presumably the wire is equivalent to a single core that s about 0.4mm in diameter thus giving it a resistance of about 0.134 Ohm per meter, or 1.75*2*0.134 == 0.469 Ohm for the cable. The charger is rated at 0.7A. To supply 0.7A at 5V the resistance would be 5V/0.7A == 7.143 Ohm so about 6.6% of the total resistance of the circuit would be in the wire from the charger to the phone. The charger that came with my Galaxy S3 has a round cable that s just over 3mm thick and about 90cm long. If each wire in the cable is equivalent to a solid wire that is 0.912mm in diameter then it would be 0.0264 Ohm per meter of wire or 0.9*2*0.0264 == 0.0475 Ohm. The total circuit resistance would be 5V/1A == 5 Ohm. So 0.0475 Ohm is less than 1% of the circuit resistance. Voltage Drop The Voltage across a part of a circuit is proportional to the resistance (see the Wikipedia page on Series and Parallel Circuits for a good explanation). Basically this means that if 1% of the resistance of a circuit is in the wire then 1% of the Voltage drop will also be in the wire, so if we have a 5V supply with my Galaxy S3 cable then each of the two wires in the cable will have a difference of about 0.025V between the ends and the phone will receive a supply of 4.95V, the difference isn t something that is worth worrying about. But the cable from my Galaxy S has a resistance equivalent to 6.6% of the circuit resistance which means that the theoretical charge time will be 6% longer than it might be or 6% more current will be drawn from the mains than should be needed. Conclusion The charger that came with my Samsung Galaxy S isn t much good. Wasting 6.6% of the power in the wire is unreasonable. Phones keep getting more power hungry and batteries keep getting larger. There are third party phone batteries and external batteries that are charged by USB which have more than twice the capacity of the stock phone batteries this means more than twice the charge time. This problem will keep getting worse. The problem of a phone in active use drawing more power than the charger can provide (and running out of battery while on the charger) seems likely to stay with us. So while an Android phone has the potential to be a great little embedded server it seems that hacking the power supply is going to be a required first step for realising that potential. The decision to make 5V the USB power standard was reasonable at the time as it was the voltage used for most things on the motherboard. The decision to use USB as the phone charging standard was also reasonable, it allows phones to be charged anywhere. The combination of those two decisions isn t good for the user. If a higher Voltage such as 12V was used then 5* the power could be supplied through the same wires at the same level of efficiency. It would be really good if cars came with built in Inverters and supplied 240VAC or 110VAC depending on the region they were manufactured for. It s becoming a fairly common feature to have a cigarette lighter port in the car boot as well as at least two ports inside the car. When a car has three sockets and only one device to actually light cigarettes (which I suspect is only provided to fill an empty socket) it s very obvious that people want to connect random devices. Also having USB charging ports inside the car would be a really good idea (one for each seat would be good for Ingress).

• [1] http://tinyurl.com/cd6jpcv
• [2] http://www.daycounter.com/Calculators/AWG.phtml

I've been frustrated by my inability to schedule appointments directly from the org-agenda view, for quite some time, but I think I've found a way. If like me you're using org-mode to manage your appointments, you're probably using SCHEDULED TODO items that can be entered in your system using org-capture templates. A typical scenario is I'm asked by a colleague (face to face or on the phone) when I'm available to schedule a meeting or some other activity. I'm then firing an agenda view, and browse through the dates available... then agreeing on an empty slot, I'm capturing the meeting... but then I have to C-c C-s to add the SCHEDULED marker and set the correct date, although I was previously on that exact date in the org-agenda view The solution seems to be with the 'k c' command, i.e. the capture org-agenda-action in the agenda view. From the docs :

k runs the command org-agenda-action, which is an interactive compiled
c     Call  org-capture' with cursor date as the default date

Exactly what I need : it will prompt me for a capture template, which I just have to set to a template containing "SCHEDULED: %t". Excerpt from my .emacs :

(custom-set-variables
...
'(org-capture-templates (quote (("t" "Todo" entry (file+headline "~/org/newgtd.org" "TACHES") "* TODO %^ Brief description  %^g
 %?   
    Added: %U
%i
%a") ("n" "Notes" entry (file+datetree "~/org/journal.org") "* %^ topic  %T %^g
%i%?
%a") ("d" "Scheduled appointment" entry (file+headline "~/org/newgtd.org" "TACHES") "* TODO %^ Brief description  %^g
 %?   
    SCHEDULED: %t
%i
%a"))))

Tadaaa ! it works

Yesterday, I wrote about the modalias values provided by the Linux kernel following my hope for better dongle support in Debian. Using this knowledge, I have tested how modalias values attached to package names can be used to map packages to hardware. This allow the system to look up and suggest relevant packages when I plug in some new hardware into my machine, and replace discover and discover-data as the database used to map hardware to packages. I create a modaliases file with entries like the following, containing package name, kernel module name (if relevant, otherwise the package name) and globs matching the relevant hardware modalias.

Package: package-name
Modaliases: module(modaliasglob, modaliasglob, modaliasglob)

It is fairly trivial to write code to find the relevant packages for a given modalias value using this file. An entry like this would suggest the video and picture application cheese for many USB web cameras (interface bus class 0E01):

Package: cheese
Modaliases: cheese(usb:v*p*d*dc*dsc*dp*ic0Eisc01ip*)

An entry like this would suggest the pcmciautils package when a CardBus bridge (bus class 0607) PCI device is present:

Package: pcmciautils
Modaliases: pcmciautils(pci:v*d*sv*sd*bc06sc07i*)

An entry like this would suggest the package colorhug-client when plugging in a ColorHug with USB IDs 04D8:F8DA:

Package: colorhug-client
Modaliases: colorhug-client(usb:v04D8pF8DAd*)

I believe the format is compatible with the format of the Packages file in the Debian archive. Ubuntu already uses their Packages file to store their mappings from packages to hardware. By adding a XB-Modaliases: header in debian/control, any .deb can announce the hardware it support in a way my prototype understand. This allow those publishing packages in an APT source outside the Debian archive as well as those backporting packages to make sure the hardware mapping are included in the package meta information. I've tested such header in the pymissile package, and its modalias mapping is working as it should with my prototype. It even made it to Ubuntu Raring. To test if it was possible to look up supported hardware using only the shell tools available in the Debian installer, I wrote a shell implementation of the lookup code. The idea is to create files for each modalias and let the shell do the matching. Please check out and try the hw-support-lookup shell script. It run without any extra dependencies and fetch the hardware mappings from the Debian archive and the subversion repository where I currently work on my prototype. When I use it on a machine with a yubikey inserted, it suggest to install yubikey-personalization:

% ./hw-support-lookup
yubikey-personalization
%

When I run it on my Thinkpad X40 with a PCMCIA/CardBus slot, it propose to install the pcmciautils package:

% ./hw-support-lookup
pcmciautils
%

If you know of any hardware-package mapping that should be added to my database, please tell me about it. It could be possible to generate several of the mappings between packages and hardware. One source would be to look at packages with kernel modules, ie packages with *.ko files in /lib/modules/, and extract their modalias information. Another would be to look at packages with udev rules, ie packages with files in /lib/udev/rules.d/, and extract their vendor/model information to generate a modalias matching rule. I have not tested any of these to see if it work. If you want to help implementing a system to let us propose what packages to install when new hardware is plugged into a Debian machine, please send me an email or talk to me on #debian-devel.

release is the tool that I use to manage software releases. This version adds support for signing a tarball with multiple PGP keys at once. For WebAuth, we have an official signing key, but I also wanted to add my personal signature. The minimum supported version of Python was also bumped to 2.5 (see below for why). You can get the latest version from my scripts page. Normally, I wouldn't announce this to everyone, since it's a fairly minor change, except I had a question and a quick Python note. The question: how does one get GnuPG to verify multiple signatures? When I run GnuPG on the resulting detached *.asc file created by this new code, I get the following:

gpg: WARNING: multiple signatures detected.  Only the first will be checked.
gpg: Signature made Thu 10 Jan 2013 03:58:43 PM PST using RSA key ID 5736DE75
gpg: Good signature from "Russ Allbery <rra@stanford.edu>"
gpg:                 aka "Russ Allbery <rra@debian.org>"
gpg:                 aka "Russ Allbery <eagle@windlord.stanford.edu>"
gpg:                 aka "Russ Allbery <eagle@eyrie.org>"

That's nice, I suppose, but it's not as helpful as it could be. Why not verify both signatures and report all of the results? I searched both the gpg man page and the Internet for some option I'm missing, without any luck. (Searching Google for that exact message amusingly turns up lots of verbatim output from Debian repository verification and almost nothing else.) If anyone knows, send me email (rra@stanford.edu or any other convenient address), and I'll update this post with the answer. UPDATE: gpg can currently only verify multiple signatures if the signatures are from keys with the same class and digest. I was attempting this with a 2048-bit RSA key and a 1024-bit DSA key, which gpg does not support. (Yet another reason to replace the WebAuth signing key with a newly-generated key.) Thanks to Philip Martin and Bernhard R. Link for the information! The note is that, way back when I was first learning Python, one of the things that bothered me the most was the lack of safe program execution. The best available was os.spawnv(), which left something to be desired. If you too were bothered by this but don't program in Python enough to keep track of the current state of the art, there's now a subprocess module that has much saner interfaces that work properly, do reasonable things, and avoid the shell. It was added in Python 2.4, with some nicer interfaces added in Python 2.5, so it's fairly safe to use at this point.