Second part; bullet points to save time

• For the first time in a long time, I used something that comes "from Nokia": Nokia_n tires with spikes. Nokia_n means "from Nokia", i.e. a former business unit which Nokia sold and allowed the new owner to keep the modified name. I don't expect to use anything else from Nokia any time soon, which is a pity imo.
• Roads are often plowed, but a lot of ice still remains. This is not a problem because of the spikes; they, in turn, will be a problem for non-icy roads as they turn tarmac into gravel and potholes... At least while I was there, most streets, both within and without cities, were covered with either snow or ice.
• Cars moving snow are not purpose-built, but re-use existing machinery. Plows tend to be smaller dump trucks (~20-30 tons?) while actual shovelling is done by tractors and small diggers.
• Plow drivers expect you to make way. In narrow streets, they will not slow down even if their shovel extends well onto your lane. You get out of the way, and quickly, or your car is scrap.
• Finland is a nation of tailgaters. Distance between cars driving at around 80 km/h on icy roads averages at about 10-20 meters.
• One reason why tailgating is somewhat understandable, if still highly dangerous, is that a single slow driver can accumulate literally a dozen or more car behind them as there's almost no chance to overtake on those wavy single-lane roads. I witnessed a car driving at a constant 50 km/h in a 80 km/h zone. As they refused to stop at the side to let others pass, this was... infuriating.
• Overtaking is generally done at the lowest speed delta possible without crashing into oncoming traffic. This ensures that only one car per overtaking opportunity gets to overtake.
• Licorice all the things. Licorice ice cream. Yay.
• There are a lot of body building products in Finnish super markets. Even the smallest stores always have their own section.
• Lactose and gluten free food is a Big Thing.
• Foreign loanwords always have at least one letter doubled; tack an "i" at the end and you're good to go.
• Finns are fiercely proud of their own country and its achievements. Most if not all of them sport a Nokia phone. Try and find a door hinge not made by Fiskars.
• Angry. Birds. Everything. There is even Angry Birds coffee beans on sale.
• Keys are half-round, of a system I have never seen before, and tend to jam in the locks.
• Self-locking doors are not opened automagically by depressing the inner handle; you are forced to turn an extra knob. The same system is used for doors you lock manually. Said knob needs to be turned clock-wise to open and vice versa to close. The German system follows standard screws (clock-wise to tighten/close) and thus is the exact opposite. If you are used to the exact opposite of a system like that, you get to run into doors a lot.
• Energy seems to be free or nearly so. Doors of many heated buildings just stay open all the time. Lights burn continuously. Both cabins I stayed at were heated with electric stoves of all things.
• Drafty (thus non-insulated) windows are apparently common enough that you can buy plastic tape marketed specifically to tape over drafty windows.

Taking advantage of a new hard drive, I decided to reinstall my Ubuntu (Precise 12.04.2) laptop from scratch so that I could easily enable full-disk encryption (a requirement for Mozilla work laptops). Reinstalling and reconfiguring everything takes a bit of time though, so here's the procedure I followed to keep the configuration to a minimum.

Install Ubuntu/Debian on the new drive While full-disk encryption is built into the graphical installer as of Ubuntu 12.10, it's only available through the alternate install CD in 12.04 and earlier. Using that CD, install Ubuntu on the new drive, making sure you select the "Encrypted LVM" option when you get to the partitioning step. (The procedure is the same if you use a Debian CD.) To make things easy, the first user you create should match exactly the one from your previous installation.

Copy your home directory Once the OS is installed on the new drive, plug the old one back in (in an external enclosure if you need to) and mount its root partition as a read-only drive on /mnt. Then log in as root (not just using sudo, actually login using the root user) and copy your home directory on the new drive:

rm -rf /home/*
cp -a /mnt/home/* /home/

Then you should be able to log in as your regular user.

Reinstall all packages The next step is to reinstall all of the packages you had installed on the old OS. But first of all, let's avoid having to answer all of the debconf questions we've already answered in the past:

rm -rf /var/cache/debconf
cp -a /mnt/var/cache/debconf /var/cache/

and set the debconf priority to the one you usually use (medium in my case):

dpkg-reconfigure debconf

Next, make sure you have access to all of the necessary repositories:

cp -a /mnt/etc/apt/sources.list /etc/apt/
cp -a /mnt/etc/apt/sources.list.d/* /etc/apt/sources.list.d/
apt-get update

Getting and setting the list of installed packages To get a list of the packages that were installed on the old drive, use dpkg on the old install:

mount -o remount,rw /mnt
chroot /mnt
dpkg --get-selections > /packages
exit
mount -o remount,ro -r /mnt

Use that list on the new install to reinstall everything:

dpkg --set-selections < /mnt/packages
apt-get dselect-upgrade

Selectively copy configuration files over Finally, once all packages are installed, you can selectively copy the config files from the old drive (in /mnt/etc) to the new (/etc/). In particular, make sure you include these ones:

cp -a /mnt/etc/alternatives/ /mnt/etc/default/ /etc/

(I chose not to just overwrite all config files with the old ones because I wanted to get rid of any cruft that had accumulated there and suspected that there might be some slight differences due to the fresh install of the distro.)

Reboot Once that's done, you should really give that box a restart to ensure that all service are using the right config files, but otherwise, that's it. Your personal data and config are back, all of your packages are installed and configured the way they were, and everything is fully encrypted!

A friend who s a long-time iPhone user just asked for my advice about whether to get a Samsung Galaxy S3, a Samsung Galaxy Note 2, or a iPhone 5. Advantages for Android I think that liberty should be the first consideration, I ve previously written about how Android phones won t necessarily give you as much freedom as you desire if you buy on the basis of price and features [1]. But even the least free Android options are way better than the iOS (iPhone and iPad) environment. This isn t necessarily a big deal for my friend, like most of the population he usually just wants things to work being able to hack them isn t such an issue. However unlike most of the population he does make a reasonable portion of his income from software development and it could be that he will have a contract for developing an app on a mobile device in which case the freedom to tinker on Android will help him. He could use an iPhone for his personal use and develop on an Android platform for his clients, but generally it s more efficient if your personal use of technology is similar to that of your clients. The Nexus devices are very good for liberty and they also have nice hardware at a low price, I ve just got a Nexus 4 for my wife and it s very nice. The next issue is that of hardware standards, I ve previously written about the potential for developing a standard form factor for Android phones [2] although this doesn t seem likely to be implemented in the near future. The wide range of Android hardware means that the range of cases etc on the market is rather small. But the advantage of the wide range is that with an Android phone you can have a device that s bigger, smaller, cheaper, cuter, or faster than an iPhone. There are Android devices which have a higher resolution, more RAM, more storage (if you include SD storage), or has other benefits over an iPhone. For whatever reasonable range of specs appeal to you you can probably find a device to match. I ve previously written about the way the ideal size for devices depends on your hand size and your preferred manner of gripping the device [3], so the lack of size range in Apple devices is not just a limitation on personal choice but also a failure to properly support people with different size hands. Depending on the preferred manner of gripping a phone the iPhone 5 is either too big for an average woman or too small for a tall man. The Google Play store apparently has more applications than the iPhone/iPad App Market. This difference can be expected to increase now that the Samsung Galaxy S3 is outselling the iPhone 5. Comparing the number of unit sales of the iPhone vs Android phones is no longer interesting, comparing Samsung to Apple is the interesting thing. Advantages for the iPhone By all accounts it s quite an easy process to backup and restore all iPhone settings. You can expect that after losing an iPhone you can just connect the new one to your PC and have it work in exactly the same way after all the data is transferred. Trying to do such things on Android is merely difficult if you have root access to your phone and the source and destination phones are of exactly the same make and model. But if you have different versions of the phone or if you don t have root access then it may be impossible. I welcome comments from anyone who knows of good solutions to this problem. The iPhone achieved a reasonable share of the smart-phone market before Android really started going well so there are a lot of people who are used to the iPhone. Simply by being unfamiliar Android will be a more difficult option for people who have used the iPhone such as my friend. But it is possible to learn other systems. Generally I think that this may be a big issue for people who use Macs for all their other computing. But if the only Apple product you use is an iPhone then switching to Android shouldn t be a big deal. Update: One feature of the iPhone that is very important to my friend is the ability to add arbitrary tags in the contacts. In addition to name, address, phone number, etc he wants to add arbitrary notes related to his business. While he could put that sort of thing into the Notes field in Android he would rather have several fields with his own names. Android 4.1.x definitely doesn t have this and I can t test Android 4.2.x at this time. Is there any way of doing such things on Android? Conclusion It seems to me that Android devices are better in every way apart from backup, restore, and general management. If I was about to buy 100 phones then I d probably consider the iPhone (not necessarily buy but definitely consider). But for a single user I definitely recommend Android devices. The Android devices which seem good at the moment are the Galaxy S3 (which I m using now), the Nexus 4 (which is really good apart from being unable to change the battery or add more storage), and the Galaxy Note 2 (which is about the biggest phone available). One of the things that my friend wants to do is to use a phone instead of a tablet or laptop. I think that the Galaxy Note 2 is the only option for him.

• [1] http://etbe.coker.com.au/2012/05/06/liberty-mobile-phones/
• [2] http://etbe.coker.com.au/2012/01/04/standardising-android/
• [3] http://etbe.coker.com.au/2013/02/11/phone-tablet-sizes/

For a while, it looked to me like my bicycle's rear-light was somewhat dim. I supposed that the batteries might be getting low, but I was not sure. So, given that I have a multimeter, and that the internets give ~20% decrease in voltage as reasonable limit for a non-recheargable battery that is not "dead", I set to measure. The batteries are type N Battery (which were not as common as I hoped, at least not as common as either AA, AAA, D or C, all of which I could find at the nearby store, but not N). Battery "marked" voltage: 1.5V. New batteries (after I replaced them): 1.451V (good enough). My old batteries 1.041V. Ouch! That's ~30% decrease. No wonder I could stare in the light without problems With the new batteries in, it's a bit painful to look directly into the small light. In hindsight, 3 years of light (pun intended) use is more than reasonable. So note to self: test batteries more often, at least at the beginning of the cycling season.

Sometimes, you have to use ZIP archives, or worse, RAR archives (curse them!), with one significant annoyance: zip, unzip, rar and unrar use a rather uncommon command line convention, compared to the usual tar, cpio and pax.This is where bsdtar and bsdcpio come handy: these two equivalent tools from FreeBSD do not directly implement any archive format, relying on libarchive to do that instead. That allows you to do thinks like:

% bsdtar -tf crap.rar
% bsdtar -xf crap.rar
% bsdtar --format zip -cf stuff.zip stuff

Too bad it does not have a -a option to automatically select the archive format as GNU tar does.

I finished delivering all my Kickstarter rewards at the end of the year. This is an overview of how that went, both financially and in general.

my Kickstarter pie (not including taxes)

While the Kickstarter was under way, several friends warned me that I might end up spending a lot of the money for rewards, or even shipping, and come out a loser. It's happened to some people, but I avoided it. Most of the pie went to its intended purpose.

USB key arrives in Amsterdam

I kept shipping cost low by shipping everything by US postal service, including Air Mail for international shipping. This was particularly important for the stickers (which cost $1.05 to ship internationally). But I also shipped USB keys in regular mail envelopes, protected by bubble wrap, which worked very well and avoided the bother of shipping packages. The USPS will be annoyed at you for a rigid letter and add a non-machinable surcharge, but it's still a nice savings.

expenses

I spent more on rewards than on transaction fees, but the fees are still pretty large. Being dinged a second time by Amazon is the worst part. I have not been able to work out exactly what formula Kickstarter uses to determine its fee per pledge. It does not seem to be a simple percentage of the pledge. For example, they seem to have charged $0.25 per $10 pledge, but $25 for a $500 pledge. I wanted to solve this, but I'd have to match up all the pledges and fees manually to do it.

gross income by reward type

This chart is slightly innacurrate, because it puts any money pledged, beyond the amount needed to get a reward, into the "intangibles" category, despite the reward being probably responsible for that money being pledged. (The intangibles also includes people who did not ask for a reward, and several categories of rewards not involving shipping matter around.) But, the surprise for me is how large a peice the T-shirts are responsible for. It was my least favorite reward, and a low volume one, but I made out pretty well on it. However, I'd still try to avoid putting T-shirts on Kickstarter again. It's hard to do a good design (I didn't, really); they're expensive, and were by far the most annoying thing to ship. Also, I was not happy with the countries Cafe Press sourced their shirts from; I've been to Honduras and talked with people who have relatives in las machinas.

gross and net income by reward type (excluding shipping)

In contrast, the stickers had an amazing margin; they're so inexpensive to print that I printed up two kinds and included multiple with every other reward I mailed. I still have hundreds left over, too.. All the online print shops I tried have very annoying interfaces to upload artwork though. I had to do quite a bit of math to render TIFF files with appropriate DPI and margins. The USB keys were my favorite reward. I got them from USB Memory Direct, who gave me quite a nice deal. I was very happy that I was able to send them a SVG file of my artwork, so I didn't need to worry about lacking resolution for the laser engraving. And it came out looking great. The best part was when their sales guy Mike actually did a minor alteration of the artwork, to better fit on the key, when I, being overloaded with Kickstarter stuff asked him too. A bit above and beyond. There was an issue with their Chinese manufacturer's quality control of the 16 gb drives, but they were willing to send me replacement for all the ones I found problems with.

---

All told I spent probably 3 full days stuffing and shipping envelopes, and probably spent a week working on Kickstarter reward fullfillment. As work-related overhead goes, that's not bad. Maybe someone considering a Kickstarter will find this information useful somehow. Oh well, back to work. :)

For once, I think we are going to have a funny DPL campaign thanks to Francesca who brought to us a DPL Game :) So here is my little list of fantastic four:

• Gregor Herrmann
• Lucas Nussbaum
• Russ Allbery
• Enrico Zini (sorry, you've been on my list since the beginning).

(in random order)

Each one of them has super powers (lot of energy, wisdom, impartiality, ability to sort out complex situations) and can do a great DPL!

My fantastic four, please consider running for DPL!

On Planet Debian, Vincent Bernat wrote:

Excuse me This is totally inacceptable. Regenerating files like aclocal.m4 and Makefile.in (for automake), configure (for autoconf), and the likes is one of the absolute duties of a software package. Things will break sooner or later if people do not do that. Additionally, generated files must be remakable from the distfile, so do not break this! May I suggest, constructively, an alternative? (People rightfully, I must admit complain I m just ranting too much.)
When making a release from git, write the git describe output into a file. Then, use that file instead of trying to run the git executable if .git/. is not a directory ( test -d .git/. ). Do not call git, because, in packages, it s either not installed or/and also undesired. Couldn t comment on your blog, but felt strongly enough about this I took the effort of writing a full post of my own. (But thanks for the book recommendation.)

A few weeks ago, a new version of lldpd, a 802.1AB (aka LLDP) implementation for various Unices, has been released. LLDP is an industry standard protocol designed to supplant proprietary Link-Layer protocols such as EDP or CDP. The goal of LLDP is to provide an inter-vendor compatible mechanism to deliver Link-Layer notifications to adjacent network devices. In short, LLDP allows you to know exactly on which port is a server (and reciprocally). To illustrate its use, I have made a xkcd-like strip: If you would like more information about lldpd, please have a look at its new dedicated website. This blog post is an insight of various technical changes that have affected lldpd since its latest major release one year ago. Lots of C stuff ahead!

Version & changelog UPDATED: Guillem Jover told me how he met the same goals for libbsd :

1. Save the version from git into .dist-version and use this file if it exists. This allows one to rebuild ./configure from the published tarball without losing the version. This also handles Thorsten Glaser s critic.
2. Include CHANGELOG in DISTCLEANFILES variable.

Since this is a better solution, I have adopted the appropriate line of codes from libbsd. The two following sections are partly technically outdated.

Automated version In configure.ac, I was previously using a static version number that I had to increase when releasing:

AC_INIT([lldpd], [0.5.7], [bernat@luffy.cx])

Since the information is present in the git tree, this seems a bit redundant (and easy to forget). Taking the version from the git tree is easy:

AC_INIT([lldpd],
        [m4_esyscmd_s([git describe --tags --always --match [0-9]* 2> /dev/null   date +%F])],
        [bernat@luffy.cx])

If the head of the git tree is tagged, you get the exact tag (0.7.1 for example). If it is not, you get the nearest one, the number of commits since it and part of the current hash (0.7.1-29-g2909519 for example). The drawback of this approach is that if you rebuild configure from the released tarball, you don t have the git tree and the version will be a date. Just don t do that.

Automated changelog Generating the changelog from git is a common practice. I had some difficulties to make it right. Here is my attempt (I am using automake):

dist_doc_DATA = README.md NEWS ChangeLog
.PHONY: $(distdir)/ChangeLog
dist-hook: $(distdir)/ChangeLog
$(distdir)/ChangeLog:
        $(AM_V_GEN)if test -d $(top_srcdir)/.git; then \
          prev=$$(git describe --tags --always --match [0-9]* 2> /dev/null) ; \
          for tag in $$(git tag   grep -E '^[0-9]+(\.[0-9]+) 1, $$'   sort -rn); do \
            if [ x"$$prev" = x ]; then prev=$$tag ; fi ; \
            if [ x"$$prev" = x"$$tag" ]; then continue; fi ; \
            echo "$$prev [$$(git log $$prev -1 --pretty=format:'%ai')]:" ; \
            echo "" ; \
            git log --pretty=' - [%h] %s (%an)' $$tag..$$prev ; \
            echo "" ; \
            prev=$$tag ; \
          done > $@ ; \
        else \
          touch $@ ; \
        fi
ChangeLog:
        touch $@

Changelog entries are grouped by version. Since it is a bit verbose, I still maintain a NEWS file with important changes.

Core

C99 I have recently read 21st Century C which has some good bits and also handles the ecosystem around C. I have definitively adopted designated initializers in my coding style. Being a GCC extension since a long time, this is not a major compatibility problem. Without designated initializers:

struct netlink_req req;
struct iovec iov;
struct sockaddr_nl peer;
struct msghdr rtnl_msg;
memset(&req, 0, sizeof(req));
memset(&iov, 0, sizeof(iov));
memset(&peer, 0, sizeof(peer));
memset(&rtnl_msg, 0, sizeof(rtnl_msg));
req.hdr.nlmsg_len = NLMSG_LENGTH(sizeof(struct rtgenmsg));
req.hdr.nlmsg_type = RTM_GETLINK;
req.hdr.nlmsg_flags = NLM_F_REQUEST   NLM_F_DUMP;
req.hdr.nlmsg_seq = 1;
req.hdr.nlmsg_pid = getpid();
req.gen.rtgen_family = AF_PACKET;
iov.iov_base = &req;
iov.iov_len = req.hdr.nlmsg_len;
peer.nl_family = AF_NETLINK;
rtnl_msg.msg_iov = &iov;
rtnl_msg.msg_iovlen = 1;
rtnl_msg.msg_name = &peer;
rtnl_msg.msg_namelen = sizeof(struct sockaddr_nl);

With designated initializers:

struct netlink_req req =  
    .hdr =  
        .nlmsg_len = NLMSG_LENGTH(sizeof(struct rtgenmsg)),
        .nlmsg_type = RTM_GETLINK,
        .nlmsg_flags = NLM_F_REQUEST   NLM_F_DUMP,
        .nlmsg_seq = 1,
        .nlmsg_pid = getpid()  ,
    .gen =   .rtgen_family = AF_PACKET  
 ;
struct iovec iov =  
    .iov_base = &req,
    .iov_len = req.hdr.nlmsg_len
 ;
struct sockaddr_nl peer =   .nl_family = AF_NETLINK  ;
struct msghdr rtnl_msg =  
    .msg_iov = &iov,
    .msg_iovlen = 1,
    .msg_name = &peer,
    .msg_namelen = sizeof(struct sockaddr_nl)
 ;

Logging Logging in lldpd was not extensive. Usually, when receiving a bug report, I asked the reporter to add some additional printf() calls to determine where the problem was. This was clearly suboptimal. Therefore, I have added many log_debug() calls with the ability to filter out some of them. For example, to debug interface discovery, one can run lldpd with lldpd -ddd -D interface. Moreover, I have added colors when logging to a terminal. This may seem pointless but it is now far easier to spot warning messages from debug ones.

libevent In lldpd 0.5.7, I was using my own select()-based event loop. It worked but I didn t want to grow a full-featured event loop inside lldpd. Therefore, I switched to libevent. The minimal required version of libevent is 2.0.5. A convenient way to check the changes in API is to use Upstream Tracker, a website tracking API and ABI changes for various libraries. This version of libevent is not available in many stable distributions. For example, Debian Squeeze or Ubuntu Lucid only have 1.4.13. I am also trying to keep compatibility with very old distributions, like RHEL 2, which does not have a packaged libevent at all. For some users, it may be a burden to compile additional libraries. Therefore, I have included libevent source code in lldpd source tree (as a git submodule) and I am only using it if no suitable system libevent is available. Have a look at m4/libevent.m4 and src/daemon/Makefile.am to see how this is done.

Client

Serialization lldpctl is a client querying lldpd to display discovered neighbors. The communication is done through an Unix socket. Each structure to be serialized over this socket should be described with a string. For example:

#define STRUCT_LLDPD_DOT3_MACPHY "(bbww)"
struct lldpd_dot3_macphy  
        u_int8_t                 autoneg_support;
        u_int8_t                 autoneg_enabled;
        u_int16_t                autoneg_advertised;
        u_int16_t                mau_type;
 ;

I did not want to use stuff like Protocol Buffers because I didn t want to copy the existing structures to other structures before serialization (and the other way after deserialization). However, the serializer in lldpd did not allow to handle reference to other structures, lists or circular references. I have written another one which works by annotating a structure with some macros:

struct lldpd_chassis  
    TAILQ_ENTRY(lldpd_chassis) c_entries;
    u_int16_t        c_index;
    u_int8_t         c_protocol;
    u_int8_t         c_id_subtype;
    char            *c_id;
    int              c_id_len;
    char            *c_name;
    char            *c_descr;
    u_int16_t        c_cap_available;
    u_int16_t        c_cap_enabled;
    u_int16_t        c_ttl;
    TAILQ_HEAD(, lldpd_mgmt) c_mgmt;
 ;
MARSHAL_BEGIN(lldpd_chassis)
MARSHAL_TQE  (lldpd_chassis, c_entries)
MARSHAL_FSTR (lldpd_chassis, c_id, c_id_len)
MARSHAL_STR  (lldpd_chassis, c_name)
MARSHAL_STR  (lldpd_chassis, c_descr)
MARSHAL_SUBTQ(lldpd_chassis, lldpd_mgmt, c_mgmt)
MARSHAL_END;

Only pointers need to be annotated. The remaining of the structure can be serialized with just memcpy()¹. I think there is still room for improvement. It should be possible to add annotations inside the structure and avoid some duplication. Or maybe, using a C parser? Or using the AST output from LLVM?

Library In lldpd 0.5.7, there are two possible entry points to interact with the daemon:

1. Through SNMP support. Only information available in LLDP-MIB are exported. Therefore, implementation-specific values are not available. Moreover, SNMP support is currently read-only.
2. Through lldpctl. Thanks to a contribution from Andreas Hofmeister, the output can be requested to be formatted as an XML document.

Integration of lldpd into a network stack was therefore limited to one of those two channels. As an exemple, you can have a look at how Vyatta made the integration using the second solution. To provide a more robust solution, I have added a shared library, liblldpctl, with a stable and well-defined API. lldpctl is now using it. I have followed those directions²:

• Consistent naming (all exported symbols are prefixed by lldpctl_). No pollution of the global namespace.
• Consistent return codes (on errors, all functions returning pointers are returning NULL, all functions returning integers are returning -1).
• Reentrant and thread-safe. No global variables.
• One well-documented include file.
• Reduce the use of boilerplate code. Don t segfault on NULL, accept integer input as string, provide easy iterators,
• Asynchronous API for input/output. The library delegates reading and writing by calling user-provided functions. Those functions can yield their effects. In this case, the user has to callback the library when data is available for reading or writing. It is therefore possible to integrate the library with any existing event-loop. A thin synchronous layer is provided on top of this API.
• Opaque types with accessor functions.

Accessing bits of information is done through atoms which are opaque containers of type lldpctl_atom_t. From an atom, you can extract some properties as integers, strings, buffers or other atoms. The list of ports is an atom. A port in this list is also an atom. The list of VLAN present on this port is an atom, as well as each VLAN in this list. The VLAN name is a NULL-terminated string living in the scope of an atom. Accessing a property is done by a handful of functions, like lldpctl_atom_get_str(), using a specific key. For example, here is how to display the list of VLAN assuming you have one port as an atom:

vlans = lldpctl_atom_get(port, lldpctl_k_port_vlans);
lldpctl_atom_foreach(vlans, vlan)  
    vid = lldpctl_atom_get_int(vlan,
                               lldpctl_k_vlan_id));
    name = lldpctl_atom_get_str(vlan,
                                lldpctl_k_vlan_name));
    if (vid && name)
        printf("VLAN %d: %s\n", vid, name);
 
lldpctl_atom_dec_ref(vlans);

Internally, an atom is typed and reference counted. The size of the API is greatly limited thanks to this concept. There are currently more than one hundred pieces of information that can be retrieved from lldpd. Ultimately, the library will also enable the full configuration of lldpd. Currently, many aspects can only be configured through command-line flags. The use of the library does not replace lldpctl which will still be available and be the primary client of the library.

CLI Having a configuration file was requested since a long time. I didn t want to include a parser in lldpd: I am trying to keep it small. It was already possible to configure lldpd through lldpctl. Locations, network policies and power policies were the three items that could be configured this way. So, the next step was to enable lldpctl to read a configuration file, parse it and send the result to lldpd. As a bonus, why not provide a full CLI accepting the same statements with inline help and completion?

Parsing & completion Because of completion, it is difficult to use a YACC generated parser. Instead, I define a tree where each node accepts a word. A node is defined with this function:

struct cmd_node *commands_new(
    struct cmd_node *,
    const char *,
    const char *,
    int(*validate)(struct cmd_env*, void *),
    int(*execute)(struct lldpctl_conn_t*, struct writer*,
        struct cmd_env*, void *),
    void *);

A node is defined by:

• its parent,
• an optional accepted static token,
• an help string,
• an optional validation function and
• an optional function to execute if the current token is accepted.

When walking the tree, we maintain an environment which is both a key-value store and a stack of positions in the tree. The validation function can check the environment to see if we are in the right context (we want to accept the keyword foo only once, for example). The execution function can add the current token as a value in the environment but it can also pop the current position in the tree to resume walk from a previous node. As an example, see how nodes for configuration of a coordinate-based location are registered:

/* Our root node */
struct cmd_node *configure_medloc_coord = commands_new(
    configure_medlocation,
    "coordinate", "MED location coordinate configuration",
    NULL, NULL, NULL);
/* The exit node.
   The validate function will check if we have both
   latitude and longitude. */
commands_new(configure_medloc_coord,
    NEWLINE, "Configure MED location coordinates",
    cmd_check_env, cmd_medlocation_coordinate,
    "latitude,longitude");
/* Store latitude. Once stored, we pop two positions
   to go back to the "root" node. The user can only
   enter latitude once. */
commands_new(
    commands_new(
        configure_medloc_coord,
        "latitude", "Specify latitude",
        cmd_check_no_env, NULL, "latitude"),
    NULL, "Latitude as xx.yyyyN or xx.yyyyS",
    NULL, cmd_store_env_value_and_pop2, "latitude");
/* Same thing for longitude */
commands_new(
    commands_new(
        configure_medloc_coord,
        "longitude", "Specify longitude",
        cmd_check_no_env, NULL, "longitude"),
    NULL, "Longitude as xx.yyyyE or xx.yyyyW",
    NULL, cmd_store_env_value_and_pop2, "longitude");

The definition of all commands is still a bit verbose but the system is simple enough yet powerful enough to cover all needed cases.

Readline When faced with a CLI, we usually expect some perks like completion, history handling and help. The most used library to provide such features is the GNU Readline Library. Because this is a GPL library, I have first searched an alternative. There are several of them:

• NetBSD Editline library (libedit).
• Autotool port of the NetBSD Editline library (also libedit).
• Debian version of the Editline library (libeditline).
• linenoise, a small and minimal readline library.
• Many others.

From an API point of view, the first three libraries support the GNU Readline API. They also have a common native API. Moreover, this native API also handles tokenization. Therefore, I have developed the first version of the CLI with this API³. Unfortunately, I noticed later this library is not very common in the Linux world and is not available in RHEL. Since I have used the native API, it was not possible to fallback to the GNU Readline library. So, let s switch! Thanks to the appropriate macro from the Autoconf Archive (with small modifications), the compilation and linking differences between the libraries are taken care of. Because GNU Readline library does not come with a tokenizer, I had to write one myself. The API is also badly documented and it is difficult to know which symbol is available in which version. I have limited myself to:

• readline(), addhistory(),
• rl_insert_text(),
• rl_forced_update_display(),
• rl_bind_key()
• rl_line_buffer and rl_point.

Unfortunately, the various libedit libraries have a noop for rl_bind_key(). Therefore, completion and online help is not available with them. I have noticed that most BSD come with GNU Readline library preinstalled, so it could be considered as a system library. Nonetheless, linking with libedit to avoid licensing issues is possible and help can be obtained by prefixing the command with help.

OS specific support

Netlink on Linux Previously, the list of interfaces was retrieved through getifaddrs(). lldpd is now using directly Netlink on Linux. This is not a big change since the GNU C Library already uses it to implement getifaddrs() and additional information, like VLAN, are still retrieved through ioctl() or sysfs. However, lldpd now gets notified when a change happens and update all interfaces in the next second. Like many other projects, I have written my own Netlink implementation instead of using libnl, a nice collection of libraries providing everything you need to query the kernel through Netlink, including some advanced bits. Why?

1. The latest version of libnl is still young and its availability in major distributions is scarce. It is not available in Debian Squeeze but will be available in Debian Wheezy. Like libevent, I could circumvent this problem by shipping the library with lldpd and use it when there is not system alternative. But
2. libnl is licensed under LGPL 2.1. This makes static linking difficult because the license is quite shaddy about static linking being derivative work or not. It is believed that it is authorized under the same provisions as in LGPL 3 which handles the case explicitely. This has been a problem with many projects. For example, OGRE has added an exception for static linking in version 1.6 and switched to MIT license in version 1.7.

I had a short discussion with Thomas Graf about this issue and he seems willing to add a similar exception. This may take some time, but once this is done, I will happily switch to libnl and retrieve more stuff from Netlink.

BSD support Until version 0.7, lldpd was Linux-only. The rewrite to use Netlink was the occasion to abstract interfaces and to port to other OS. The first port was for Debian GNU/kFreeBSD, then for FreeBSD, OpenBSD and NetBSD. They all share the same source code:

• getifaddrs() to get the list of interfaces,
• bpf(4) to attach to an interface to receive and send packets,
• PF_ROUTE socket to be notified when a change happens.

Each BSD has its own ioctl() to retrieve VLAN, bridging and bonding bits but they are quite similar. The code was usually adapted from ifconfig.c. The BSD ports have the same functionalities than the Linux port, except for NetBSD which lacks support for LLDP-MED inventory since I didn t find a simple way to retrieve DMI related information. They also offer greater security by filtering packets sent. Moreover, OpenBSD allows to lock the filters set on the socket:

/* Install write filter (optional) */
if (ioctl(fd, BIOCSETWF, (caddr_t)&fprog) < 0)  
    rc = errno;
    log_info("privsep", "unable to setup write BPF filter for %s",
        name);
    goto end;
 
/* Lock interface */
if (ioctl(fd, BIOCLOCK, (caddr_t)&enable) < 0)  
    rc = errno;
    log_info("privsep", "unable to lock BPF interface %s",
        name);
    goto end;
 

This is a very nice feature. lldpd is using a privileged process to open the raw socket. The socket is then transmitted to an unprivileged process. Without this feature, the unprivileged process can remove the BPF filters. I have ported the ability to lock a socket filter program to Linux. However, I still have to add a write filter.

OS X support Once FreeBSD was supported, supporting OS X seemed easy. I got sponsored by xcloud.me which provided a virtual Mac server. Making lldpd work with OS X took only two days, including a full hour to guess how to get Apple Xcode without providing a credit card. To help people installing lldpd on OS X, I have also written a lldpd formula for Homebrew which seems to be the most popular package manager for OS X.

Upstart and systemd support Many distributions propose upstart and systemd as a replacement or an alternative for the classic SysV init. Like most daemons, lldpd detaches itself from the terminal and run in the background, by forking twice, once it is ready (for lldpd, this just means we have setup the control socket). While both upstart and systemd can accommodate daemons that behave like this, it is recommended to not fork. How to advertise readiness in this case? With upstart, lldpd will send itself the SIGSTOP signal. upstart will detect this, resume lldpd with SIGCONT and assume it is ready. The code to support upstart is therefore quite simple. Instead of calling daemon(), do this:

const char *upstartjob = getenv("UPSTART_JOB");
if (!(upstartjob && !strcmp(upstartjob, "lldpd")))
    return 0;
log_debug("main", "running with upstart, don't fork but stop");
raise(SIGSTOP);

The job configuration file looks like this:

# lldpd - LLDP daemon
description "LLDP daemon"
start on net-device-up IFACE=lo
stop on runlevel [06]
expect stop
respawn
script
  . /etc/default/lldpd
  exec lldpd $DAEMON_ARGS
end script

systemd provides a socket to achieve the same goal. An application is expected to write READY=1 to the socket when it is ready. With the provided library, this is just a matter of calling sd_notify("READY=1\n"). Since sd_notify() has less than 30 lines of code, I have rewritten it to avoid an external dependency. The appropriate unit file is:

[Unit]
Description=LLDP daemon
Documentation=man:lldpd(8)
[Service]
Type=notify
NotifyAccess=main
EnvironmentFile=-/etc/default/lldpd
ExecStart=/usr/sbin/lldpd $DAEMON_ARGS
Restart=on-failure
[Install]
WantedBy=multi-user.target

OS include files Linux-specific include files were a major pain in previous versions of lldpd. The problems range from missing header files (like linux/if_bonding.h) to the use of kernel-only types. Those headers have a difficult history. They were first shipped with the C library but were rarely synced and almost always outdated. They were then extracted from kernel version with almost no change and lagged behind the kernel version used by the released distribution⁴. Today, the problem is acknowledged and is being solved by both the distributions which extract the headers from the packaged kernel and by kernel developers with a separation of kernel-only headers from user-space API headers. However, we still need to handle legacy. A good case is linux/ethtool.h:

• It can just be absent.
• It can use u8, u16 types which are kernel-only types. To work around this issue, type munging can be setup.
• It can miss some definition, like SPEED_10000. In this case, you either define the missing bits and find yourself with a long copy of the original header interleaved with #ifdef or conditionally use each symbol. The latest solution is a burden by itself but it also hinders some functionalities that can be available in the running kernel.

The easy solution to all this mess is to just include the appropriate kernel headers into the source tree of the project. Thanks to Google ripping them for its Bionic C library, we know that copying kernel headers into a program does not create a derivative work.

---

1. Therefore, the use of u_int16_t and u_int8_t types is a left-over of the previous serializer where the size of all members was important.
2. For more comprehensive guidelines, be sure to check Writing a C library.
3. Tokenization is not the only advantage of libedit native API. The API is also cleaner, does not have a global state and has a better documentation. All the implementations are also BSD licensed.
4. For example, in Debian Sarge, the Linux kernel was a 2.6.8 (2004) while the kernel headers were extracted from some pre-2.6 kernel.

DRI3000 Even Better Direct Rendering This all started with the presentation that Eric Anholt and I did at the 2012 X developers conference, and subsequently wrote about in my DRI-Next posting. That discussion sketched out the goals of changing the existing DRI2-based direct rendering infrastructure. Last month, I gave a more detailed presentation at Linux.conf.au 2013 (the best free software conference in the world). That presentation was recorded, so you can watch it online. Or, you can read Nathan Willis summary at lwn.net. That presentation contained a lot more details about the specific techniques that will be used to implement the new system, in particular it included some initial indications of what kind of performance benefits the overall system might be able to produce. I sat down today and wrote down an initial protocol definition for two new extensions (because two extensions are always better than one). Together, these are designed to provide complete support for direct rendering APIs like OpenGL and offer a better alternative to DRI2. The DRI3 extension Dave Airlie and Eric Anholt refused to let me call either actual extension DRI3000, so the new direct rendering extension is called DRI3. It uses POSIX file descriptor passing to share kernel objects between the X server and the application. DRI3 is a very small extension in three requests:

1. Open. Returns a file descriptor for a direct rendering device along with the name of the driver for a particular API (OpenGL, Video, etc).
2. PixmapFromBuffer. Takes a kernel buffer object (Linux uses DMA-BUF) and creates a pixmap that references it. Any place a Pixmap can be used in the X protocol, you can now talk about a DMA-BUF object. This allows an application to do direct rendering, and then pass a reference to those results directly to the X server.
3. BufferFromPixmap. This takes an existing pixmap and returns a file descriptor for the underlying kernel buffer object. This is needed for the GL Texture from Pixmap extension.

For OpenGL, the plan is to create all of the buffer objects on the client side, then pass the back buffer to the X server for display on the screen. By creating pixmaps, we avoid needing new object types in the X server and can use existing X apis that take pixmaps for these objects. The Swap extension Once you ve got direct rendered content in a Pixmap, you ll want to display it on the screen. You could simply use CopyArea from the pixmap to a window, but that isn t synchronzied to the vertical retrace signal. And, the semantics of the CopyArea operation precludes us from swapping the underlying buffers around, making it more expensive than strictly necessary. The Swap extension fills those needs. Because the DRI3 extension provides an X pixmap reference to the direct rendered content, the Swap extension doesn t need any new object types for its operation. Instead, it talks strictly about core X objects, using X pixmaps as the source of the new data and X drawables as the destination. The core of the Swap extension is one request SwapRegion. This request moves pixels from a pixmap to a drawable. It uses an X fixes Region object to specify the area of the destination being painted, and an offset within the source pixmap to align the two areas. A bunch of data are included in the reply from the SwapRegion request. First, you get a 64-bit sequence number identifying the swap itself. Then, you get a suggested geometry for the next source pixmap. Using the suggested geometry may result in performance improvements from the techniques described in the LCA talk above. The last bit of data included in the SwapRegion reply is a list of pixmaps which were used as source operands to earlier SwapRegion requests to the same drawable. Each pixmap is listed along with the 64-bit sequence number associated with an earlier SwapRegion operation which resulted in the contents which the pixmap now contains. Ok, so that sounds really confusing. Some examples are probably necessary.

• If the SwapRegion operation was implemented by copying data out of the source pixmap into the destination drawable, then the idle swap count will be equal to the swap count from this SwapRegion operation.
• If the SwapRegion operation was implemented by swapping the destination contents with the source contents, then the idle swap count will be equal to the previous swap count on the destination drawable.

I m hoping you ll be able to tell that in both cases, the idle swap count tries to name the swap sequence at which time the destination drawable contained the contents currently in the pixmap. Note that even if the SwapRegion is implemented as a Copy operation, the provided source pixmap may not be included in the idle list as the copy may be delayed to meet the synchronization requirements specfied by the client. Finally, if you want to throttle rendering based upon when frames appear on the screen, Swap offers an event that can be delivered to the drawable after the operation actually takes place. Because the Swap extension needs to supply all of the OpenGL SwapBuffers semantics (including a multiplicity of OpenGL extensions related to that), I ve stolen a handful of DRI2 requests to provide the necessary bits for that:

1. SwapGetMSC
2. SwapWaitMSC
3. SwapWaitSBC

These work just like the DRI2 requests of the same names. Current State of the Extensions Both of these extensions have an initial protocol specification written down and stored in git:

1. DRI3 protocol
2. Swap protocol

Ten years ago, this very day, my first Debian package entered the Debian unstable repository. It was an addon for Mozilla Composer, Daniel Glazman s Cascades. On the same day, my second Debian package entered the Debian unstable repository as well. It was an addon for Mozilla Browser, Checky. A few days later, my third Debian package entered Debian unstable. It was an addon for Mozilla Browser, Diggler. Do you see a pattern? They are now abandoned software, although I made Checky and Diggler live a little longer (and I m actually considering reviving Diggler) but they had their importance in my journey, and are part of the reason why I am where I am now. My journey on the web started with NCSA Mosaic on VAX/VMS, then continued with Netscape Navigator, Netscape Communicator and Mozilla Suite on Linux. That s where I was ten years ago, sailing between Galeon (a browser using the Mozilla engine) and Mozilla Suite, and filing some layout bugs. Ten years ago, there was a new kid on the block. It used to be called Phoenix, it had just changed its name to Firebird. Eventually, it changed again for Firefox. You may have heard about it. Because Firebird was so much nicer than the browser in the Mozilla Suite, I started using its Debian package, and wanted my packaged addons to work with it. So I contacted Eric Dorland, Phoenix/Firebird package maintainer at the time, and got the addons working. I then ended up fixing a bunch of packaging issues. This is how I got involved in Firefox packaging for Debian, and what eventually led me to work for Mozilla.

My last two blog posts were largely about the needs of subunit, but a key result of any protocol is how easy working with it in a high level language is. In the weekend and evenings I ve done an implementation of a new set of classes StreamResult and friends that provides:

• Adaption to and from the existing TestResult APIs (the 2.6 and below API, 2.7 API, and the testtools extended API).
• Multiplexing multiple streams together.
• Adding timing data to a stream if it is absent.
• Summarising a stream.
• Copying a stream to multiple outputs
• A split out API for instructing a test run to stop.
• A simple test-at-a-time stream processor that makes it easy to just deal with tests rather than the innate complexities of an event based interface.

So far the code has been uniformly simple to write. I started with an API that included an estimate function, which I ve since removed I don t believe the complexity is justified; enumeration is not significantly more expensive than counting, and runners that want to be efficient can either not enumerate or remember the enumeration from prior runs. The documentation in the linked pull request is a good place to start to get a handle on the API; I d love feedback. Next steps for me are to do a subunit protocol revision that maps to the Python API, both parser and generator and see how it feels. One wrinkle there is that the reason for doing this is to fix intrinsic limits in the existing protocol so doing forward and backward wire protocol compatibility would defeat the point. However we can make the output side explicitly choose a protocol version, and if we can autodetect the protocol version in the parser, even if we cannot handle mixed streams we can get the benefits of the new protocol once data has been detected. That said, I think we can start without autodetection during prototyping, and add it later. Without autodetection, programs like TestRepository will need configuration options to control what protocol variant to expect. This could be done by requiring this new protocol and providing a stream filter that can be deployed when needed.

Thanks for the group hug! Francesca started a DPL game and I've been mentioned a few times, by people I like deeply. Thank you! However I don't intend to run, and I hope I won't disappoint those who nominated me by saying so. But I don't think of it in terms of letting people down: I can't let anyone down since I never mentioned I'd like to run in the first place. Rather, I like to think that I've just received a wonderful group hug, and hey, wow, come here and let me hug you back! <3 And let me hug some more:

• Gregor Herrmann, I'm in a constant state of awe for what you have done with the Perl team. If you think that some of what you did could also be done as a DPL, please go for it!
• Bdale Garbee, I didn't think there could be such a thing as a reliable source of common sense until I got to know you. And you've been DPL and quite a good deal of other things and definitely know the drill. And you can't say you're busy with your day job. The last thing I want is for you to get bored after your retirement! :P
• Paul Tagliamonte, you seem to bring an incredible energy in everything you get involved with. Let's admit it: Zack's been a perfect DPL in many aspects, he has been incredibly, inhumanly smooth, but he was a bit boring. Zack has left Debian as a perfectly working train steaming on professionally towards awesomeness. Let's give it colours! Let's give it excitement! Let's give it creativity, and silliness! We need someone to cheer Christian up, and I if you can't do it, I can't think of anyone else who can!

I cannot think of a fourth DPL candidate right now and I don't want to postpone this post indefinitely. Think about it this way: you three are so good I can't think of a fourth one right now There are actually lots of people I admire in Debian. I tried to name a few without thinking, but I wasn't thinking so I lost count as soon as I ran out of fingers. I know however that many enjoy to stay out of the spotlight and keep their fun focused on a few specific things. I am one of those myself. Oh dear, FOSDEM was too short, can I have DebConf soon? In the meantime, let's have some fun with the DPL campaign.

Linux is a great platform for dealing with all kinds of different file systems, partition tables etc. But one of the few annoying situations when working in IT forensics are Microsoft Windows dynamic disks, AKA LDM (Logical Disk Manager). Thanks to libldm s ldmtool this is no longer true. A short demonstration from a real-life IT forensics investigation (actual IDs/data randomized for obvious reasons):

# ldmtool
ldm> scan /dev/sdc*
[
  "1bad5bbc-a4b5-42e1-8823-001014b00003"
]
ldm> show diskgroup 1bad5bbc-a4b5-42e1-8823-001014b00003
 
  "name" : "FOOBAR-Dg0",
  "guid" : "1bad5bbc-a4b5-42e1-8823-001014b00003",
  "volumes" : [
    "Volume1"
  ],
  "disks" : [
    "Disk1",
    "Disk2"
  ]
 
ldm> show volume 1bad5bbc-a4b5-42e1-8823-001014b00003 Volume1
 
  "name" : "Volume1",
  "type" : "striped",
  "size" : 3907039232,
  "chunk-size" : 128,
  "hint" : "D:",
  "partitions" : [
    "Disk1-01",
    "Disk2-01"
  ]
 
ldm> show partition 1bad5bbc-a4b5-42e1-8823-001014b00003 Disk1-01
 
  "name" : "Disk1-01",
  "start" : 1985,
  "size" : 1953519616,
  "disk" : "Disk1"
 
ldm> create all
Unable to create volume Volume1 in disk group 1bad5bbc-a4b5-42e1-8823-001014b00003: Disk Disk2 required by striped volume Volume1 is missing
[
]
ldm> scan /dev/sdd*
[
  "1bad5bbc-a4b5-42e1-8823-001014b00003"
]
ldm> create all
[
  "ldm_vol_FOOBAR-Dg0_Volume1"
]
ldm>

The just created device mapper device then can be handled as usual:

# dmsetup ls   grep ldm
ldm_vol_FOOBAR-Dg0_Volume1        (254:4)
# mount /dev/mapper/ldm_vol_FOOBAR-Dg0_Volume1 /mnt/whatever

ldmtool just hit Debian unstable (and I intend to ship the tool with the upcoming version of Grml-Forensic).